TALIS 2013 Design
- Overall approach to data collection
- Self-administered cross-sectional survey, with questionnaires being administered online or in paper form or both
- Trend study
ISCED Level 2 teachers and their principals, excluding adult education and schools solely devoted to children with special needs
ISCED Level 1 and ISCED Level 3 teachers and their principals, with above exclusions
Stratified two-stage sample design for each ISCED level
Stratification on basis of nationally relevant criteria (e.g., geography, source of funding, language of instruction)
First stage: sampling of schools
- Selected systematically with probability proportional to school size; file sorted by school size within stratum
- Samples for field trial (FT) and main study collection (MS) selected at same time to avoid reuse of schools across phases
- Replacement schools identified at the time of sample selection (one for each of the schools selected for FT, and two for each school selected for the MS)
- In a few countries, the number of schools is too small to allow for the efficient use of systematic selection of 200 schools and their replacements; in such situations, simple random sampling, stratified on school size, may be more appropriate.
Second stage: sampling of individuals
- Simple random sampling of individuals
Same design applied to each target population a country chooses to survey (ISCED Level 1 and ISCED Level 3).
Samples coordinated across ISCED levels – if required – to minimize risks of selecting the same school in more than one survey.
Per country and ISCED level
- Nominally 200 schools; may be reduced if the total number of schools is quite small
- The principal of a selected school participates by default.
- A nominal simple random sample of 20 ISCED Level 2 teachers: all are selected if fewer than 20 are available; suggestion to sample all if between 20 and 30 are available, selection of 20 if at least 30 are available.
- Nominally, 200 principals and 4,000 teachers from the ISCED level of interest
- Two questionnaires, one for principals and one for teachers, were administered.
- Questionnaires were subdivided into sections with section headers outlining the content areas of the survey, then developed and assembled in MS Word format for paper administration and transferred to IEA’s Online Survey System for online administration.
- Automatic routing with filter questions was applied to the online questionnaire version.
- The principal questionnaire, the teacher questionnaire, and the mathematics teacher questionnaire for the TALIS-PISA Link option (= study instrument) were administered in 26 different languages plus four local languages in Spain.
- The most common languages:
- English (7 countries)
- Spanish (3 countries)
- Arabic (2 countries)
- Dutch (2 countries)
- Portuguese (2 countries)
- Swedish (2 countries)
- Reviews of the TALIS 2013 survey instruments, at several stages of instrument preparation, focused on approving national modifications to questionnaires and cover letters, language translations, questionnaire layouts, online data collection (ODC) versions, and codebooks.
- Instruments were provided in an English-language and a translated French-language source version.
- Instruments were first adapted and translated in the pilot study.
- Instruments were then adapted using national adaptation forms (NAFs) by the national project managers (NPMs).
- Adaptations were verified and approved by the International Study Center (ISC), then translated, internally reviewed, and revised by NPMs, and then checked by professional translation verifiers coordinated by IEA Amsterdam.
- NPMs implemented translation verifiers’ feedback and adjusted the layout of the national instruments as necessary.
- The ISC verified and approved the layout of the paper instruments and provided the Online Survey System (OSS); implementation of the instruments in the online delivery system was performed by NPMs.
- The ISC verified and approved the final online instruments using the OSS.
Measures during data collection
- Development of technical standards
- Quality control implementation at various levels and stages during instrument production, administration, and data entry and processing
- Standardized international quality control program of school and national center visits by international quality control monitors (IQCMs) for the main survey, organized and overseen by IEA Amsterdam
- An additional national quality control program of school visits during the field trial and main survey provided by NPMs
- A quality control manual template provided by the TALIS International Consortium that countries could adapt to their needs and use as a basis for training the national quality control monitors (NQCMs)
- Information from these activities was augmented by NPM responses to an on-line survey activities questionnaire (SAQ) conducted after administration of the main survey; questionnaire asked NPMs about their respective national centers’ implementation of the TALIS procedures at each stage of the project. The outcomes of the national quality control program were also reported in the survey activities questionnaire.
Measures during data processing and cleaning
- Signed confidentiality agreements were obtained from all project staff of all TALIS participants.
- All countries received training in how to perform manual data entry according to the rules and standards outlined in the TALIS 2013 Main Survey Manual for Data Managers (OECD, 2012j) and the TALIS 2013 Technical Standards (OECD, 2012k) and in using IEA’s Data Management Expert (DME) software. As an important quality control measure, countries were required to perform double entry of a sampled set of teacher and principal questionnaires to ensure high-quality data entry.
- The teacher (IDTEACH) and school identifiers (IDSCHOOL) were scrambled and therefore did not match those used during data collection; however, the structural link between the school and teacher level (the variable IDSCHOOL in the teacher file and the first four digits of any IDTEACH) was maintained. For each country, unique matching tables were created and made available to authorized individuals.
- Variables used purely for the stratification of the teacher sample, i.e., birth year (ITBIRTHY) and gender (ITSEX), were removed; only the gender (TT2G01) and age (TT2G02) variables as collected in the questionnaire were retained.
- Variables used purely for stratification of schools were removed (IDSTRATE and IDSTRATI) to avoid the identification of geographical or organizational groups. Because the stratum information is mostly of interest for national-level analysis, it was, of course, made available to the country concerned. Experience shows that researchers from other countries might also wish to conduct analysis by stratification, in which case the stratification variables have to be requested directly from the country.
- Information used in the calculation of final sample and replicate weights was removed (school level: WGTFAC1 and WGTADJ1; teacher level: WGTFAC1, WGTADJ1, WGTFAC2, WGTADJ2, WGTADJ3, and WGTADJ4), as these could allow identification of stratification cells.
- Replication zone and unit variables (BRRSZONE, BRRSREP, BRRTZONE, and BRRTREP), which could allow for indirect identification of schools, were also dropped from public-use micro-data.
- Data for Question 47 (TT2G47A-J) in the teacher questionnaire were removed at OECD request.