ICCS 2009 Design

Methodology
Quantitative Study
TOP
Method(s)
  • Overall approach to data collection
    • Assessment and self-administered surveys
  • Specification
    • Cross-sectional
TOP
Target population

Students

  • The grade representing 8 years of formal schooling in participating entities
  • The target was Grade 8 if the average age was at least 13.5 years.
  • If the average age in Grade 8 was below 13.5, the target grade became Grade 9.
  • Students older than 17 years were not part of the target population.

 

Teachers

Teachers of the target grade comprised the second target population of the survey.

TOP
Sample design
Two-stage cluster sample

School sampling

  • PPS (probability proportional to size of the school)
  • Stratification of schools
    • Explicit strata
    • Implicit strata
  • Selection of the school samples using random-start fixed-interval systematic sampling
  • Simultaneous sampling of schools for field test and main data collection
  • For each sampled school, two replacement schools were assigned.

 

Class sampling

  • Systematic random sampling was used to select classes within the sampled school from the target grade.
  • Each class had an equal selection probability within the school.
  • If a selected class was smaller than half of the average class size, a pseudo-class was created with other classes.
  • All students within a selected class were asked to participate
TOP
Sample size

Per country

  • Minimum of 400 students required for effective sample size
  • Expected: approx. 3,000 students per country
  • Intended: a minimum of 150 schools and one class in each school
  • In countries with less than 150 schools, all available schools were included.

 

Overall

Approximately 140,000 students and 62,000 teachers from over 5,300 schools in 38 countries

TOP
Data collection techniques and instruments

Student tests of civic knowledge

  • Written format
  • Two types of questions
    • Multiple-choice
    • Constructed-response (6 questions)
  • Test booklets
    • 7 booklets with 3 item clusters each
    • Items
      • Numbers
        • 80 items
      • Items per cluster
        • 10−11 items in 6 clusters
        • 17 items in the CIVED cluster
    • Linking mechanisms between booklets and between cycles
      • 63 new items in 2009
      • 17 items from CIVED 1999
  • Test administration with matrix-sampling of items, i.e., rotated test booklet design

 

Background questionnaires

  • Student questionnaire, print format
  • Student regional questionnaire (Europe, Latin America, Asia), print format
  • Teacher questionnaire to be completed by a target grade teacher, print or online
  • School questionnaire to be completed by the principal of each sampled school, print or online format

 

National context questionnaires

  • To be completed by the national research coordinator (NRC) in each participating entity
  • Online format
TOP
Techniques
  • achievement test
  • questionnaire
TOP
Languages
  • The ICCS instruments were administered in 31 languages.
  • The most common languages
    • Spanish (7 countries)
    • English (4 countries)
TOP
Translation procedures
  • The international study center (ISC) developed an international English-language version of the ICCS assessment and questionnaires.
  • These materials were subsequently translated and adapted by countries to their languages of instruction.
  • Because all countries participating in the Latin American module required Spanish as the assessment language, the instrument was developed in this language for use and adaptation by countries participating in this module.
  • The IEA Secretariat enlisted the assistance of an independent translation company to verify the translations for each country.
TOP
Quality control of operations

Measures during data collection

  • All national centers received guidelines on survey operations procedures for each stage of the assessment and material setting out procedures for quality control.
  • An international quality control program was developed to document data collection activities around the world.
  • The NRCs in each country were required to nominate an international quality control monitor (IQCM), who was then hired and trained by the IEA Secretariat.
  • School visits were conducted by the IQCMs during test administration (15 schools per country)
  • Countries were asked to conduct their own quality control procedures in 10 percent of the sampled schools. To assist them, the international team also provided each country with the National Quality Control Monitor Manual modified to suit the national system that was used for training the observers.

 

Measures during data processing and cleaning

Data quality control included the following steps:

  • Thorough testing of all data-cleaning programs
  • Registration of all incoming data and documents in a database
  • Compliance with strict data-cleaning rules
  • Documentation of all systematic data recodings that applied to all countries
  • Logging, in a recoding file, of all “manual” corrections to a country’s data files
  • Replication, on completion of data-cleaning for each country, of the entire process, from the beginning
  • Close cooperation with the national centers throughout the various stages of the cleaning process

The program-based data cleaning consisted of the following steps:

  • Documentation and structure check
  • Identification variable (ID) cleaning
  • Linkage cleaning
  • Resolution of inconsistencies in questionnaire data
TOP
Sources - Technical documentation
TOP
Other sources
TOP