LaNA 2023 Linking Study Design

Methodology

International large-scale sample-based assessment of student achievement and survey of the educational context

TOP
Method(s)
  • Overall approach to data collection: international large-scale assessment of student achievement and survey of the educational context
  • Specification: cross-sectional project design; paper-based assessment of student achievement combined with context questionnaires
TOP
Target population
  • Students enrolled in the grade that represented four, five, or six years of schooling, counting from the first year of ISCED Level 1.
  • LaNA targets students at the end of primary education (typically Grades 4–6, depending on the structure of the national education system and average student proficiency). For the Link countries, the most suitable grade was determined in agreement with each participating country based on evidence from the practice administration on average student achievement levels, resulting in implementation in Grades 5 or 6.
TOP
Sample design
Stratified two-stage cluster sampling design 

First stage: sampling schools

  • Selection probability proportional to the size of the school (PPS)
  • Optional: stratification of schools according to (demographic) variables of interest (e.g., school type or source of funding, level of urbanization, region of the country), either explicit or implicit
  • Random-start fixed-interval systematic sampling
  • Schools sampled at the same time for field trial and main data collection
  • For each sampled school, two replacement schools were assigned where possible

 

Second stage (students): sampling classes

Within schools agreeing to participate:

  • One or more intact classes in the target grade were randomly selected from the sampled school.
  • Each class had an equal selection probability within the school.
  • If a selected class was smaller than half of the average class size, a pseudo-class was created with other classes.
  • In very large classes (e.g., 100 or more students), a random subsample of students may have been selected for assessment, depending on country implementation.
TOP
Sample size

Intended per country

  • At least 4,000 assessed students
  • At least 100 schools
  • In each sampled school, selection of one class 

 

Achieved (main data collection)

  • Per country
    • Burkina Faso: 100 schools, 4,916 students (Grade 6)
    • Egypt: 104 schools, 5,424 students (Grade 5)
    • Nigeria: 98 schools, 4,232 students (Grade 6)
    • Pakistan: 360 schools, 8,744 students (Grade 5)
    • Palestinian National Authority (Westbank): 96 schools, 2,744 students (Grade 5)
    • Senegal: 116 schools, 4,636 students (Grade 6)
  • Total: 874 schools and 30,696 assessed students across the 6 countries.
TOP
Data collection techniques and instruments

Achievement test

  • Paper-and-pencil, multiple-choice items only; rotated booklet design
  • The assessment design comprised eight booklets (four LaNA booklets and four Linking booklets), and each student was randomly assigned one booklet
    • LaNA booklets (4 booklets)
      • The LaNA booklets comprised 80 mathematics items organized into four blocks of 20 items each, with each booklet containing two mathematics blocks. 
      • In addition, five reading passages were included in the design, each followed by 10–11 items, with each booklet containing two passages.
      • Testing time: 40 minutes per subject (mathematics and reading), total 80 minutes
    • Linking booklets (4 booklets)
      • A total of 52 items from TIMSS 2019 and 68 items from PIRLS 2021 (including open-ended constructed-response items) were used to establish the psychometric link between LaNA and the TIMSS and PIRLS scales.

 

Context questionnaires

  • Student context questionnaire (30 minutes)
  • School context questionnaire (completed by school principal)
TOP
Languages
  • Administration of achievement booklets and context questionnaires in the national language(s) of instruction in each of the 6 participating countries
  • Languages used in LaNA 2023
    • Arabic: Egypt, Palestinian National Authority (Westbank)
    • French: Burkina Faso, Senegal
    • English: Nigeria, Pakistan
    • Urdu: Pakistan
    • Sindhi: Pakistan
TOP
Translation procedures

All participating countries followed standardized procedures for translating and adapting the national assessment instruments:

  • Countries received the international (English) instrument versions for translation.
  • Desired adaptations were first entered in National Adaptation Forms (NAFs) and reviewed/approved by IEA Hamburg before translation.
  • After adaptation verification, countries translated instruments into their national languages.
  • NAFs and translations were submitted to IEA Amsterdam, which employed independent translators to verify all national translations.
  • Following translation verification, instruments were assembled into printable PDFs. For some countries, IEA Hamburg produced national PDFs; others did so in-country.
  • A layout verification step by IEA Hamburg assured conformance with international standards for all main data collection instruments.
TOP
Quality control of operations

Measures during data collection

  • Detailed Survey Operations Procedures (SOP) units provided to National Research Coordinators (NRCs) by IEA Hamburg, covering all stages from sampling through data submission.
  • Training for NRCs and test administrators at two online international NRC meetings prior to the main data collection.
  • Two international scoring training sessions (one for the practice administration, one for main data collection) to ensure consistent scoring of constructed-response linking items.
  • W3ST (Within-School Sampling and Student Tracking) Macro used for standardized within-school sampling and instrument assignment.
  • Standardized test administration procedures including school coordinator and test administrator manuals; timing enforced by test administrators.
  • Within-country reliability scoring: Two independent scorers evaluated a random sample of student responses to constructed-response items in the linking booklets to document inter-rater reliability.
  • Survey Activities Questionnaire (SAQ) completed by all six NRCs as a quality assurance measure to document field experiences.
  • Online version of Test Administration Forms completed by test administrators (MS Forms) to document quality control of the test administration procedures.

 

Measures during data processing and cleaning

  • Data entered using IEA Data Management Expert (IEA DME) software; built-in validation checks identified inconsistencies at the point of entry.
  • Double-entry verification: Data entry staff independently re-entered at least 5% of records from each instrument type; acceptable error rates were ≤1% for questionnaire files and ≤0.1% for achievement and reliability scoring files.
  • IEA Hamburg applied standardized international data cleaning procedures: structure checks, ID checks, linkage checks, and context data cleaning, documented in country-specific cleaning reports.
  • Iterative cleaning cycle: All cleaning steps were repeated upon completion to detect any inadvertent errors introduced during the process.
  • NRCs reviewed interim data products and collaborated with IEA Hamburg to resolve queries throughout the cleaning process.
  • TIMSS & PIRLS International Study Center conducted operational psychometric analyses and produced achievement scores, context questionnaire scores, derived variables, new basic benchmarks, and benchmark results; IEA Hamburg Sampling Unit calculated sampling weights, population coverage, and participation rates.
TOP
Sources - Technical documentation
TOP