TIMSS 2023 Design

Methodology
  • International large-scale sample-based assessment of student achievement and survey of the educational context
  • Monitors trends in student achievement since 1995 by reporting results from successive cycles on a common achievement scale
  • Predominantly quantitative, with qualitative information presented in descriptive country chapters in the TIMSS 2023 encyclopedia
TOP
Method(s)
  • Overall data collection approach:
    • Proctored assessment of student achievement and student questionnaire
    • Self-administered surveys for parents, teachers, and school principals
  • Specification
    • Cross-sectional
    • Some trend reporting possible with previous cycles
TOP
Target population
  • Grade 4: All students enrolled in the grade that represented four years of schooling, counting from the first year of ISCED Level 1, provided that the mean age at the time of testing was at least 9.5 years.
  • Grade 8: All students enrolled in the grade that represented eight years of schooling, counting from the first year of ISCED Level 1, provided that the mean age at the time of testing was at least 13.5 years.
  • In countries where Grade 4 or Grade 8 students were expected to find the TIMSS assessments too difficult, Grade 6 or Grade 9, respectively, were defined as the target populations.
TOP
Sample design
Stratified two-stage cluster sampling design 

First stage: sampling schools

  • Selection probability proportional to the size of the school (PPS)
  • Optional: stratification of schools according to (demographic) variables of interest (e.g., school type or source of funding, level of urbanization, region of the country), either explicit or implicit
  • Random-start fixed-interval systematic sampling
  • Schools sampled at the same time for field trial and main data collection
  • For each sampled school, two replacement schools were assigned where possible

 

Second stage (students): sampling classes

Within schools agreeing to participate:

  • Systematic random sampling was used to select one or more classes from the sampled school in the target grade.
  • Each class had an equal selection probability within the school.
  • If a selected class was smaller than half of the average class size, a pseudo-class was created with other classes.
  • All students within a selected class were asked to participate.

 

General notes

  • Sampling of schools was conducted by RTI and the sampling team at IEA Hamburg.
  • Sampling procedures within the center/school were carried out by the national study centers, using the Within-school Sampling Software for Windows (WinW3S) provided by IEA.
  • All sampling activities were monitored and documented by RTI and IEA Hamburg staff.
TOP
Sample size

Intended per country and target grade

  • Approx. 4,000 assessed students
  • A minimum of 150 schools (in countries with fewer than 150 schools, all available schools were included)
  • In each sampled school, selection of one class
  • Required effective sample size: minimum of 400 students

 

Total achieved

  • Grade 4: Approximately 360,000 students in 12,000 schools from 65 education systems (59 countries and 6 benchmarking entities) participated.
  • Grade 8: Approximately 300,000 students in 9,000 schools from 47 education systems (44 countries and 3 benchmarking entities) participated.
TOP
Data collection techniques and instruments

Large-scale assessments of student achievement

  • TIMSS uses a matrix sampling approach that involves packaging the entire assessment pool of mathematics and science items at each grade level into a set of student achievement booklets or test forms (a booklet equivalent in the context of computer-based assessments).
  • TIMSS 2023 completed the transition to a fully digital assessment
    • Available in Grade 4 and Grade 8
      • Mathematics and science, Grade 4: 14 test forms
      • Mathematics and science, Grade 8: 14 test forms
    • The 14 mathematics and 14 science item blocks at each grade were arranged into 14 forms with two mathematics and two science blocks each.
    • Group adaptive design
      • Available in digital format only
      • Three levels of item block difficulty (difficult, medium, and easy) were combined into two levels of test form difficulty (i.e., each student being assigned a block of easy and a block of medium items, or a block of medium and a block of difficult items)
      • Each country administered the entire assessment with the balance of more difficult and less difficult booklets varying with the mathematics and science achievement level of the students in the country.
    • Each student was randomly assigned a test form, consisting of two mathematics item blocks and two science item blocks. 

 

  • paperTIMSS
    • Administered in a small number of countries for whom the fully digital TIMSS may not be logistically feasible
    • Regular paperTIMSS
      • Included only the secured trend items from the TIMSS 2019 paper assessment (and will provide only overall mathematics and science achievement results)
      • Available in the two grades
        • Mathematics and science, Grade 4: 8 booklets
        • Mathematics and science, Grade 8: 8 booklets
    • Less difficult paperTIMSS
      • Included only the secured trend items from the TIMSS 2019 fourth grade less difficult mathematics assessment
      • Mathematics, Grade 4: 8 booklets

 

  • For the first time, TIMSS 2023 offered a longitudinal option in which the same students that participate in TIMSS 2023 could take the same assessment one year later. This optional extension provided a more granular look at education progress over the course of one year.

 

Background questionnaires

  • Student questionnaire
  • Teacher questionnaire
  • School questionnaire
  • Early Learning Survey (also called the “home questionnaire”), grade 4 only
  • Curriculum questionnaire (completed by National Research Coordinators)

 

Descriptive encyclopedia chapters

Each country and benchmarking participant prepares a chapter describing the structure of the country’s education system, the mathematics and science curricula in the primary and secondary grades, the teacher education requirements, the types of examinations and assessments employed, policies associated with their mathematics and science curricula, school organizational approaches, and instructional practices.

TOP
Languages
  • At the fourth grade, the assessment instruments were translated into 92 language versions and the context questionnaires into 96 language versions.
  • At the eighth grade, the assessment instruments were translated into 66 language versions and the context questionnaires into 65 language versions.
  • The most common languages:
    • English (21 countries and 5 benchmarking entities)
    • Arabic (11 countries and 3 benchmarking entities)
    • French (5 countries and 2 benchmarking entities)
TOP
Translation procedures
  • All participating countries follow a set of standardized procedures for translating and adapting their national assessment instruments and any related directions or administrative screens.
  • IEA Amsterdam manages the translation verification process, which involves external verifiers comparing each national version to the international text and providing detailed feedback for the National Research Coordinators (NRCs) to improve the accuracy and comparability of their national translations.
  • Verifiers at the TIMSS & PIRLS International Study Center conduct layout and adaptations verification of national instruments, which includes checking that all national instruments match the international format and that any national adaptations to the 
TOP
Quality control of operations

Measures during data collection

  • In each TIMSS assessment cycle, standardized assessment materials and survey operations procedures are developed and adapted from previous cycles so that the data collection meets the highest standards.
  • As part of the quality assurance program, International Quality Control Monitors (IQCMs) are selected and appointed in each TIMSS country to implement the program by visiting a sample of participating schools to observe the TIMSS administration. 

 

Measures during data processing and cleaning

  • IEA Hamburg is responsible for checking the data submitted by each country,
    • applying standardized data cleaning rules to verify the accuracy and consistency of the data, and
    • documenting any deviations from the international file structure.
  • This process also includes
    • importing and cleaning the data collected by the TIMSS Player that delivers the digital assessment to students,
    • transferring student response data that require human scoring and student response data for automated scoring into their corresponding systems, and
    • importing item scores for further processing.
TOP
Sources - Technical documentation
TOP
Other sources
TOP