ICCS 2022 Design

Methodology
  • International large-scale sample survey of students, teachers, and school principals
  • Student achievement test allowing monitoring of trends across successive cycles
  • Predominantly quantitative, with qualitative information presented in descriptive country chapters
TOP
Method(s)
  • Overall approach to data collection
    • Proctored assessment of student achievement
    • Self-administered surveys for students, teachers, and school principals
  • Specification
    • Cross-sectional
    • Some trend reporting possible with previous cycles
TOP
Target population

Students

  • The student target population consisted of all students enrolled in the grade that represented eight years of schooling, counting from the first year of ISCED Level 1, provided that the mean age at the time of testing was at least 13.5 years.
  • Students older than 17 years were not included in the target population.
  • In countries where the average age of students in Grade 8 was less than 13.5 years, Grade 9 was defined as the target population.

 

Teachers

All teachers teaching regular school subjects to students of the target grade (regardless of the subject or the number of hours taught) during the ICCS testing period who had been employed at the school since the beginning of the school year.

 

Schools

  • The population for the ICCS school survey comprised schools at which target grade students were enrolled.
  • Principals of sampled schools were asked to complete the school questionnaire.
TOP
Sample design
Stratified two-stage cluster sampling design - optimized for the student population

Schools selected at the first stage for the student population were also considered sampled for the teacher and school populations.

 

First stage: sampling schools

  • Selection probability proportional to the size of the school (PPS)
  • Optional: stratification of schools according to (demographic) variables of interest (e.g., school type or source of funding, level of urbanization, region of the country), either explicit or implicit
  • Random-start fixed-interval systematic sampling
  • Schools sampled at the same time for field trial and main data collection
  • For each sampled school, two replacement schools were assigned where possible

 

Second stage (students): sampling classes

Within schools agreeing to participate:

  • Systematic random sampling was used to select one or more classes from the sampled school in the target grade.
  • Each class had an equal selection probability within the school.
  • If a selected class was smaller than half of the average class size, a pseudo-class was created with other classes.
  • All students within a selected class were asked to participate

 

Second stage: sampling teachers

Within schools agreeing to participate:

  • Selection of 15–20 teachers from all eligible teachers using systematic random sampling
  • Optional: oversampling of teachers of civic and citizenship-related subjects

 

Sampling for the ICCS 2022 Bridge Study

  • Countries transitioning from the paper-based assessment in the previous cycle to the computer-based assessment for the student survey were required to also select an equivalent, smaller bridge sample.
  • The samples of schools for the core and the ICCS Bridge Study were drawn simultaneously:
    • In a first step, a relatively large sample of schools was drawn.
    • Then, sampled schools were randomly allocated to participate either in the core survey, the ICCS 2022 Bridge Study, or both (if applicable).
  • An additional step was required for countries that chose to have an overlap between the core survey and the Bridge Study school samples:
    • In such cases, some schools were selected to participate in both studies.
    • The sampled classes were randomly allocated to participate in the core survey or in the Bridge Study.

 

General notes

  • Sampling of schools was conducted by the sampling team at IEA Hamburg.
  • Sampling procedures within the center/school were carried out by the national study centers, using the Within-school Sampling Software for Windows (WinW3S) provided by IEA.
TOP
Sample size
INTERNATIONAL Core STUDY

Intended per country

  • Approx. 3,000 assessed students
    • Bridging study: an additional sample of at least 1,500 tested students
  • Approx. 2,000 participating teachers
  • A minimum of 150 schools (in countries with fewer than 150 schools, all available schools were included)
  • In each sampled school, selection of one class and 15 target grade teachers (or all if the number of target grade teachers was less than or equal to 20)
  • Required effective sample size: minimum of 400 students


Total achieved

Approximately 80,000 students, 50,000 teachers, and 3,500 schools (i.e., school principals) from 24 education systems (22 countries and two benchmarking entities) participated.

 
Regional parts - European

Approximately 65,000 students and 40,000 teachers from 18 European countries and two benchmarking participants



Regional parts - Latin American

Approximately 10,000 students from two Latin American countries
 

TOP
Data collection techniques and instruments

Student Civic Knowledge Test

  • Paper-based format (PBA) or computer-based (online or offline) format (CBA)
  • Conventional items (CBA and PBA)
    • Multiple-choice
    • Constructed-response
  • Computer-enhanced items (CBA only)
    • Drag and drop
    • Dynamic interactive
  • Test booklets
    • PBA: 11 booklets with 3 clusters each (with items that were common across modes)
      • 11 conventional item clusters
      • 55 trend items
      • 66 newly developed test items
    • CBA: 14 booklets with 3 clusters each
      • 11 conventional item clusters and three computer-enhanced clusters
      • 55 trend items
      • 66 newly developed test items
      • 20 computer-enhanced items
    • Bridging study (paper-based): 8 booklets with 3 clusters each
      • Eight of the 11 conventional item clusters
    • Linking mechanism between booklets and cycles
      • 29 items from ICCS 2016
      • 26 items from ICCS 2009
  • Test administration with matrix-sampling of items, i.e., rotated test booklet design

 

Student questionnaires

  • Student questionnaire, paper-based format or computer-based (online or USB) format
  • Student regional questionnaire (Europe, Latin America), print format or computer-based (online or USB) format

 

School and teacher questionnaires

  • Teacher questionnaire to be completed by a target grade teacher, paper-based or online format
  • School questionnaire to be completed by the principal of each sampled school, paper-based or online format

 

National context questionnaires

  • To be completed by the national research coordinator in each participating entity
  • Online format
TOP
Languages
  • Administration of the Student Civic Knowledge Test in 30 different languages
  • The most common languages
    • Russian (3 countries)
    • English (2 countries)
    • Polish (2 countries)
    • Spanish (2 countries)
TOP
Translation procedures
  • Development of an international version of all instruments in English by the ICCS 2022 International Study Center at ACER
  • Translation into applicable languages of instruction by participating entities
  • Translation verification by linguistic and assessment experts to ensure equivalence with the international version
TOP
Quality control of operations

Measures during data collection

  • Participants were responsible for data collection within their own respective territories.
  • Standardized survey operation procedures: step-by-step documentation of all operational activities provided with manuals
  • Full-scale field test of all instruments and operational procedures, (in each participating country and benchmarking entity)
  • Provision of software tools for supporting activities (e.g., sampling and tracking classes and students, administering school and teacher questionnaires, documenting scoring reliability, creating and checking data files)
  • Training, e.g., for national research coordinators (NRCs) and their staff, school coordinators, and test administrators
  • School visits conducted by international quality observers (IQOs) during test administration (15 school visits required. In countries who were transitioning from PBA in 2016 to CBA in 2022, IQOs were required to visit three additional schools to observe the paper booklet administration as part of the bridging study.)
  • National quality control program
  • Survey activities questionnaire (SAQ) completed by NRCs

 

Measures during data processing and cleaning

  • Testing of all data cleaning programs with simulated data sets
  • Data and documents receipt database maintained by IEA Hamburg
  • Standardized cleaning process and documentation of any necessary data recoding
  • National adaptation database
  • Repetition of data cleaning and comparison of the new data sets with the preceding versions
  • Finally, identification of irregularities in data patterns and correction of data errors
TOP
Sources - Technical documentation
TOP
Other sources
TOP