TIMSS Advanced 2015 Design

Methodology
  • International large-scale sample survey of student achievement and educational context
  • Trends monitored by reporting results from successive cycles on a common achievement scale
  • Predominantly quantitative, with qualitative information provided by NRCs presented in parts of the TIMSS Advanced 2015 Results in Advanced Mathematics and Physics report
Method(s)
  • Overall data collection approach:
    • Proctored assessment of student achievement
    • Self-administered surveys for students, teachers, school principals, and TIMSS Advanced NRCs
Target population

Students in their final year of secondary school enrolled in advanced mathematics and physics programs or tracks

The decision as to which mathematics or physics courses should be included in defining the target population was determined separately by each participating country. In general, the courses included were those taken by the most advanced students, typically those students planning further study in mathematics or physics at university or other institutes of higher education.

Sample design
Stratified two-stage cluster sample design

First stage: Sampling secondary schools

  • In countries where the number of schools in the population greatly exceeded the number required in the sample, schools were sampled with probability proportional to the size of the school (PPS).
  • In countries where the number of schools from which to select samples was relatively small, schools were sampled with equal probability.
  • Stratifying schools (optional):
    • According to the types of populations — schools with advanced mathematics only, physics only, or both
    • According to important (demographic) variables (e.g., region of the country, school type or source of funding, language of instruction)
    • Can take two forms: explicit or implicit stratification
  • Using random-start fixed-interval systematic sampling
  • Sampling schools for field test and main data collection at the same time
  • Sampling of two replacement schools for each school sampled (main data collection only)

 

Second stage: Sampling classes within schools

  • Only after the sample school has agreed to participate in the study
  • One or more intact classes from the target grade of each school, selected using systematic random sampling – except in the United States, where students were randomly sampled directly from groups according to whether they were in the advanced mathematics and/or physics population(s)

 

General note

  • School sampling conducted by Statistics Canada; class and student sampling by the participating countries
Sample size

Per country

  • School sample: 150 schools intended, but actual number varied by country
  • Class sample: 1 or 2 classes per school
  • Student sample: 3,000–4,000 students

 

In total

  • Approximately 32,000 students were sampled in advanced mathematics
  • Approximately 24,000 students were sampled in physics
Data collection techniques and instruments

Student assessments in advanced mathematics and physics

  • Written format
  • Two types of questions
    • Multiple-choice (at least half of the total number of points)
    • Constructed-response
  • Achievement items
    • 18 item blocks (9 with advanced mathematics, 9 with physics) with approx. 10 items per block
    • 6 Advanced mathematics booklets, 6 physics booklets; 3 blocks per booklet
      • Approx. 100 items per assessment in advanced mathematics
      • Approx. 100 items in physics
    • Linking mechanisms between booklets and between cycles
      • New items in 2015: 12 blocks (6 advanced mathematics and 6 physics)
      • The remaining 3 blocks (for each subject) are trend items from TIMSS Advanced 2008
  • Matrix sampling of items (rotated test booklet design)

 

Background questionnaires

  • Student questionnaires, in print format
  • Teacher questionnaires to be completed by the advanced mathematics and physics teachers of the assessed classes, in print or online format
  • School questionnaire to be completed by the principal of each school sampled, in print or online format

 

Curriculum questionnaires

  • To be completed by NRC in each participating entity
  • Modular design
  • Online format
Techniques
  • achievement test
  • questionnaire
Languages
  • Assessment instruments administered in 8 languages
  • The most common languages: English (2 countries), French (2 countries)
Translation procedures
  • Development of an English-language international version of all assessment instruments by TIMSS & PIRLS International Study Center
  • Translation of international version by participating countries into their languages of instruction
  • Verification of translations by independent linguistic and assessment experts in order to ensure equivalence with the international version
Quality control of operations

During data collection

  • National research coordinator (NRC) in each participating country responsible for data collection
  • Standardized survey operation procedures: step-by-step documentation of all operational activities provided with manuals
  • Full-scale field test of all instruments and operational procedures (in each participating country)
  • Provision of software tools for supporting activities (e.g., sampling and tracking classes and students, administering school and teacher questionnaires, documenting scoring reliability, creating and checking data files)
  • Training of NRCs and their staff, school coordinators, test administrators, etc.
  • School visits by international quality control monitors (IQCMs) during test administration (24 schools per country – 12 for advanced mathematics and 12 for physics )
  • National quality control program
  • Survey activities questionnaire (SAQ) to be completed by NRCs

 

During data processing and cleaning

  • Testing of all data cleaning programs with simulated data sets
  • Material receipt database
  • National adaptation database
  • Standardized cleaning process
  • Repetition of data cleaning and comparison of new data sets with preceding versions
  • Identification of irregularities in data patterns and correction of data errors