TIMSS 2011 Design

Methodology
  • International large-scale sample survey of student achievement and the educational context
  • Monitors trends by reporting results from successive cycles on a common achievement scale
  • Predominantly quantitative, with qualitative information presented in descriptive country chapters in the TIMSS encyclopedia
TOP
Method(s)
  • Overall data collection approach
    • Proctored assessment of student achievement and
    • Self-administered surveys for students, parents, teachers, and school principals
TOP
Target population
  • School children in grades representing 4 and 8 years of formal schooling, in participating countries
  • In those countries where fourth-grade or eighth-grade students were expected to find the TIMSS assessments too difficult: Grades 6 and 9, respectively
TOP
Sample design
Stratified two-stage cluster sample design

First stage: sampling schools

  • Probability proportional to the size of the school (PPS)
  • Stratifying schools (optional):
    • According to important (demographic) variables (e.g., region of the country, school type or source of funding, level of urbanization)
    • Can take two forms: explicit or implicit stratification
  • Random-start fixed-interval systematic sampling
  • Schools sampled at the same time for field test and main data collection
  • Sampling of two replacement schools for each school sampled (main data collection only)

 

Second stage: sampling classes within schools

  • Only after sampled schools agree to participate in the study
  • One or more intact classes from the target grade of each school, selected using systematic random sampling

 

General notes

  • School sampling conducted by Statistics Canada, class sampling by participating countries
  • All sampling activities monitored and documented by Statistics Canada and IEA Hamburg staff
  • Most of the countries that participated in TIMSS 2011 at Grade 4 also participated in PIRLS 2011 with the same samples of students
TOP
Sample size

Per country and target grade

  • School sample: minimum 150-200 schools
  • Student sample: about 4,500 students

 

In total

Approximately 300,000 students per grade

TOP
Data collection techniques and instruments

Student assessments in mathematics and science

  • Written format
  • Two types of questions
    • Multiple-choice (at least half of the total number of points)
    • Constructed-response
  • Achievement items
    • 28 item blocks each grade (14 with mathematics items, 14 with science)
      • Items per block
        • Grade 4: 10-14 items in each block
        • Grade 8: 12-18 items in each block
    • 14 Test booklets each grade – 4 blocks (2 mathematics, 2 science in counterbalanced order) each booklet
      • Items per assessment
        • Grade 4: approx. 170 items in each of mathematics and science
        • Grade 8: approx. 200 items in each of mathematics and science
    • Linking mechanisms between booklets and between cycles
      • New items in 2011: 12 blocks (6 mathematics and 6 science)
      • The other 8 blocks (for each subject) are trend items (unreleased items) from TIMSS 2007 and 2003
  • Matrix sampling of items (rotated test booklet design)

                                              

Background questionnaires

  • Student questionnaire, in print format; modular design in Grade 4
  • Teacher questionnaire to be completed by the mathematics and science teachers of assessed classes, print or online; modular design in Grade 4
  • School questionnaire to be completed by the principal of each school sampled, in print or online format
  • Home questionnaire to be completed by students’ parents or guardians, in print format (only in countries that administered both TIMSS and PIRLS in 2011 to the same sample of students in Grade 4)

 

Curriculum questionnaires

  • To be completed by national research coordinator in each participating entity
  • Modular design
  • Online format

 

Descriptive encyclopedia chapters

  • One chapter for each participating entity
  • Written by experts from ministries of education, research institutes, or institutions of higher education (in countries and benchmarking entities) based on an internationally agreed outline
TOP
Languages
  • Assessment instruments administered in 45 languages
  • Instruments (assessments of achievement and student questionnaires) administered in two or more languages in 21 countries and five benchmarking entities  
  • Most common languages: English (19 countries) and Arabic (13)
TOP
Translation procedures
  • International version of all assessment instruments developed in English by TIMSS & PIRLS International Study Center
  • Translated by participating countries into their languages of instruction
  • Translations verified by independent linguistic and assessment experts in order to ensure equivalence with the international version
TOP
Quality control of operations

During data collection

  • National Research Coordinator (NRC) in each participating country responsible for data collection
  • Standardized survey operations procedures: step-by-step documentation of all operational activities provided with manuals
  • Full-scale field test of all instruments and operational procedures (in each participating country and benchmarking entity)
  • Provision of software tools for supporting activities (e.g., sampling and tracking classes and students, administering school and teacher questionnaires, documenting scoring reliability, creating and checking data files)
  • Training of national research coordinators and their staff, school coordinators, test administrators, etc.
  • School visits conducted by international quality control monitors (IQCMs) during test administration (15 schools per grade and country)
  • National quality control program
  • Survey activities questionnaire (SAQ) to be completed by NRCs

 

During data processing and cleaning

  • Testing of all data cleaning programs with simulated data sets
  • Material Receipt Database
  • National Adaptation Database
  • Standardized cleaning process
  • Repetition of data cleaning and comparison of new data sets with preceding versions
  • Identification of irregularities in data patterns and correction of data errors
TOP
Sources - Technical documentation
TOP
Other sources
TOP