TIMSS 2015 Design

Methodology
  • International large-scale sample survey of student achievement and educational context
  • Monitoring of trends by reporting results from successive cycles on a common achievement scale
  • Predominantly quantitative, with qualitative information presented in descriptive country chapters in the TIMSS encyclopedia
TOP
Method(s)
  • Overall data collection approach:
    • Proctored assessment of student achievement
    • Self-administered surveys for students, parents, teachers, and school principals
TOP
Target population
  • School children in grades representing 4 and 8 years of formal schooling in the participating countries
TOP
Sample design
Stratified two-stage cluster sample design

First stage: sampling schools

  • Probability proportional to the size of the school (PPS)
  • Stratifying schools (optional):
    • According to important (demographic) variables (e.g., region of the country, school type or source of funding, level of urbanization)
    • Can take two forms: explicit or implicit stratification
  • Random-start fixed-interval systematic sampling
  • Schools sampled at the same time for field test and main data collection
  • Sampling of two replacement schools for each school sampled (main data collection only)

 

Second stage: Sampling classes within schools

  • Only after sampled schools have agreed to participate in the study
  • One or more intact classes from the target grade of each school selected using systematic random sampling

 

General notes

  • School sampling conducted by Statistics Canada, class sampling by participating countries
  • All sampling activities monitored and documented by Statistics Canada and IEA Hamburg staff
TOP
Sample size

​​​​Per country and target grade

  • School sample: minimum 150–200 schools
  • Student sample: about 4,500 students

 

In total

Approximately 300,000 students per grade

TOP
Data collection techniques and instruments

Student assessments in mathematics and science

  • Written format
  • Two types of questions
    • Multiple-choice (at least half of the total number of points)
    • Constructed-response
  • Achievement items
    • A total of 28 item blocks per grade (14 with mathematics items, 14 with science)
      • Items per block
        • Grade 4: 10–14 items in each block
        • Grade 8: 12–18 items in each block
    • A total of 14 test booklets per grade – 4 blocks (2 mathematics, 2 science, in counterbalanced order) per booklet
      • Items per assessment
        • Grade 4: approx. 170 items in each of mathematics and science
        • Grade 8: approx. 200 items in each of mathematics and science
    • Linking mechanisms between booklets and between cycles
      • New items in 2015: 12 blocks (6 mathematics and 6 science)
      • The other 8 blocks (for each subject) are trend items (unreleased items) from TIMSS 2011 and 2007.
  • Matrix sampling of items (rotated test booklet design)

 

Background questionnaires

  • Student questionnaire in print format modular design in Grade 4
  • Teacher questionnaire to be completed by the mathematics and science teachers of assessed classes, print or online; modular design in Grade 4
  • School questionnaire to be completed by the principal of each school sampled in print or online format
  • Home questionnaire to be completed by students’ parents or guardians in print or online format

 

Curriculum questionnaires

  • To be completed by NRCs in each participating entity
  • Modular design
  • Online format

 

Descriptive encyclopedia chapters

  • One chapter for each participating entity
  • Written by experts from ministries of education, research institutes, or institutions of higher education (in countries and benchmarking entities) based on an internationally agreed-upon outline
TOP
Techniques
  • achievement or student test
  • questionnaire
TOP
Languages
  • Assessment instruments administered in 43 languages
  • Instruments (assessments of achievement and student questionnaires) administered in two or more languages in 22 countries and five benchmarking entities
  • The most common languages: English (21 countries), Arabic (10 countries)
TOP
Translation procedures
  • International version of all assessment instruments developed in English by TIMSS & PIRLS International Study Center.
  • Instruments then translated by participating countries into their languages of instruction.
  • Translations verified by independent linguistic and assessment experts in order to ensure equivalence with the international version.
TOP
Quality control of operations

During data collection

  • National research coordinator (NRC) in each participating country responsible for data collection
  • Standardized survey operations procedures: step-by-step documentation of all operational activities provided with manuals
  • Full-scale field test of all instruments and operational procedures (in each participating country and benchmarking entity)
  • Provision of software tools for supporting activities (e.g., sampling and tracking classes and students, administering school and teacher questionnaires, documenting scoring reliability, creating and checking data files)
  • Training of NRCs and their staff, school coordinators, test administrators, etc.
  • School visits conducted by international quality control monitors (IQCMs) during test administration (15 schools per grade and country)
  • National quality control program
  • Survey activities questionnaire (SAQ) to be completed by NRCs

 

During data processing and cleaning

  • Testing of all data cleaning programs with simulated data sets
  • Material receipt database
  • Standardized cleaning process
  • National adaptation database
  • Repetition of data cleaning and comparison of new data sets with preceding versions
  • Identification of irregularities in data patterns and correction of data errors
TOP
Sources - Technical documentation
TOP
Other sources
TOP