TALIS 2018 Design

Methodology
Quantitative Study
TOP
Method(s)
  • Overall approach to data collection: self-administered cross-sectional survey, with questionnaires administered online, in paper form, or both
  • Specification: trend study
TOP
Target population

Core study

ISCED Level 2 teachers and their principals, excluding adult education and schools solely devoted to children with special needs

International options

ISCED Level 1 and ISCED Level 3 teachers and their principals with the above exclusions

Teachers teaching the modal grade of fifteen-year-old students and their principals in schools selected to participate in PISA 2018

TOP
Sample design
Stratified two-stage sample design for each ISCED level

Stratification on basis of nationally relevant criteria (e.g., geography, source of funding, language of instruction)

First stage: sampling of schools

  • Selected systematically with probability proportional to school size; file sorted by school size within stratum
  • Samples for field trial (FT) and main study collection (MS) selected at same time to avoid reuse of schools across phases
  • Replacement schools identified at the time of sample selection (one for each of the schools selected for FT, and two for each school selected for the MS)
  • In a few countries, the number of schools is too small to allow for the efficient use of systematic selection of 200 schools and their replacements; in such situations, simple random sampling, stratified by school size, may be more appropriate.

Second stage: sampling of teachers within schools

Simple random sampling of individuals 

Same design applied to each target population that a country chooses to survey (ISCED Level 1 and ISCED Level 3)

Samples coordinated across ISCED levels – if required – to minimize risks of selecting the same school in more than one survey.

Overlap control was applied for the TALIS-PISA link option, resulting in sampling adaptations in smaller countries, moving from PPS systematic to equal probability systematic sampling, in order to preserve some of the larger schools for PISA.

    TOP
    Sample size

    Per country and ISCED level:

    • Nominally 200 schools; may be reduced if the total number of schools is quite small
      • The principal of a selected school was invited to participate.
      • A nominal simple random sample of 20 ISCED Level 2 teachers: all are selected if fewer than 20 are available; suggestion was to sample all if between 20 and 30 are available, selection of 20 if at least 30 are available.
    • Nominally 200 principals and 4,000 teachers from the ISCED level of interest
    • Nominally 150 schools and 3,000 teachers from schools participating in PISA 2018
    TOP
    Data collection techniques and instruments
      • Two questionnaires, one for principals and one for teachers, were administered.
      • Questionnaires were subdivided into sections with section headers outlining the content areas of the survey, then developed and assembled in MS Word format for paper administration and transferred to IEA’s Online Survey System for online administration.
      • Automatic routing with filter questions was applied to the online questionnaire version.
      TOP
      Techniques
      • questionnaire
      TOP
      Languages
      Languages used
      • The principal questionnaire and the teacher questionnaire were administered in 26 different languages plus four local languages in Spain.
      • The most common languages:
        • English (9 countries/sub-national entities)
        • Spanish (5 countries)
        • Arabic (3 countries)
        • Russian (3 countries)
        • French (2 countries/sub-national entities)
        • Dutch (2 countries)
        • Portuguese (2 countries)
        • Swedish (2 countries)
      TOP
      Translation procedures
      • Reviews of the TALIS 2018 survey instruments, at several stages of instrument preparation, focused on approving national modifications to questionnaires and cover letters, language translations, questionnaire layouts, online data collection (ODC) versions, and codebooks.
      • Instruments were provided in an English-language and a translated French-language source version.
      • Instruments were first adapted and translated in the pilot study.
      • Instruments were then adapted using national adaptation forms (NAFs) by the national project managers (NPMs).
      • Adaptations were verified and approved by the International Study Center (ISC), then translated, internally reviewed, and revised by NPMs, and then checked by professional translation verifiers coordinated by IEA Amsterdam.
      • NPMs implemented translation verifiers’ feedback and adjusted the layout of the national instruments as necessary.
      • The ISC verified and approved the layout of the paper instruments and provided the Online Survey System (OSS); implementation of the instruments in the online delivery system was performed by NPMs.
      • The ISC verified and approved the final online instruments using the OSS.
      TOP
      Quality control of operations

      Measures during data collection

      • Development of technical standards
      • Quality control implementation at various levels and stages during instrument production, administration, and data entry and processing
      • Standardized international quality control program of school and national center visits by international quality control monitors (IQCMs) for the main survey, organized and overseen by IEA Amsterdam
      • An additional national quality control program of school visits during the field trial and main survey provided by NPMs
      • A quality control manual template provided by the TALIS International Consortium that countries/sub-national entities could adapt to their needs and use as a basis for training the national quality control monitors (NQCMs)
      • Information from these activities was augmented by NPM responses to an on-line survey activities questionnaire (SAQ) conducted after administration of the main survey; the questionnaire asked NPMs about their respective national centers’ implementation of the TALIS procedures at each stage of the project. The outcomes of the national quality control program were also reported in the survey activities questionnaire.

       

      Measures during data processing and cleaning

      • Signed confidentiality agreements were obtained from all project staff of all TALIS participants.
      • All countries/sub-national entities received training in how to perform manual data entry according to the rules and standards outlined in the TALIS 2013 Main Survey Manual for Data Managers (OECD, 2012j) and the TALIS 2013 Technical Standards (OECD, 2012k) and in using IEA’s Data Management Expert (DME) software. In the case of paper-based administration, countries were required to perform double entry of a sampled set of teacher and principal questionnaires to ensure high-quality data entry as an important quality control measure.
      • The teacher (IDTEACH) and school identifiers (IDSCHOOL) were scrambled and therefore did not match those used during data collection; however, the structural link between the school and teacher level (the variable IDSCHOOL in the teacher file and the first four digits of any IDTEACH) was maintained. For each country, unique matching tables were created and made available to authorized individuals.
      • Variables used purely for the stratification of the teacher sample, i.e., birth year (ITBIRTHY), gender (GENDER) and the teachers’ main subject domain (SUBJDOMAIN) were removed; only the gender (TT2G01) and categorized age (TCHAGEGR, see below) variables as collected in the questionnaire were retained.
      • Variables used purely for stratification of schools were removed (IDSTRATE and IDSTRATI) to avoid the identification of geographical or organizational groups. Because the stratum information is mostly of interest for national-level analysis, it was, of course, made available to the country concerned. Experience shows that researchers from other countries might also wish to conduct analysis by stratification, in which case the stratification variables have to be requested directly from the country.
      • Information used in the calculation of final sample and replicate weights was removed (school level: WGTFAC1 and WGTADJ1; teacher level: WGTFAC1, WGTADJ1, WGTFAC2, WGTADJ2, WGTADJ3, and WGTADJ4), as these could allow identification of stratification cells.
      • Replication zone and unit variables (BRRSZONE, BRRSREP, BRRTZONE, and BRRTREP), which could allow for indirect identification of schools, were also dropped from public-use micro-data.
      • Some answers to numeric/count questions that increase the probability of identifying schools and teachers were categorized. This applies to questions about the principals’ and teachers’ ages (PQ-02 and TQ-02) and school enrollment (PQ- 16). The original variables TC3G02, TT3G02 and TC3G16 therefore have been replaced by the categorized variables PRAGEGR, TCHAGEGR and NENRSTUD.
      TOP
      Sources - Technical documentation
      TOP