PIAAC Design

Methodology
Quantitative Study
Method(s)
  • Overall approach to data collection: computer-based or paper-based assessment and personal interview survey
  • Specification:  trend study
Target population

Adults

  • Intended: non-institutionalized individuals ages 16 to 65 who are residents in the participating country/economy
  • Countries/economies were permitted to extend the target population.
  • The target population covered all residents regardless of citizenship, nationality, and language.
  • Adults in institutional collective dwelling units, such as prisons, hospitals, nursing homes, military barracks, and military bases, were not included.
Sample design
  • A probability-based sample, which should be representative of the target population of the country/economy
  • The following PIAAC standards were communicated to participating countries/economies. A typical sample design consists of:
    • A stratified multistage clustered area sample for geographically large countries/economies
    • Less clustering and fewer stages of sampling for geographically small countries/economies
  • Each country/economy was permitted to select a sample design in line with the PIAAC standards.
  • National sample designs as decided by countries/economies

 

One-stage sample designs

  • With or without explicit stratification

 

Two-stage stratified probability-proportionate-to-size (PPS) designs

  • First stage: primary sampling units (PSUs)
  • Second stage: persons to be sampled in each unit

 

Three-stage stratified PPS designs

  • First stage: PSUs
  • Second stage: dwelling units (DUs)
  • Third stage: persons from the housing unit

 

Four-stage stratified PPS designs

  • First stage: PSUs
  • Second stage: secondary selection units (SSUs)
  • Third stage: DUs
  • Fourth stage: persons from the housing unit
Sample size

Per country/economy

  • Round 1 (determined by the optional components of the psychometric assessments)
    • Min. of 5,000 individuals for the component problem-solving (only); for the main language of the country
    • Otherwise 4,500; for the main language of the country
  • Round 2 (standard minimum sample size)
    • Min 5,000 individuals in each participating country/economy; for the main language

 

In total

Approximately 250,000 adults in 33 countries/economies

Data collection techniques and instruments

Technique 1: Individual interview (BQ)

  • Administered in a computer-assisted personal interview (CAPI) format by the interviewer
  • 258 variables

 

Technique 2: Core tests (ICT core and domain-based core part)

  • Computer self-assessment under supervision of the interviewer
  • Sets of core items used in selecting the paper or computer path for the respondents

 

Technique 3:  Literacy, numeracy, and problem-solving achievement tests

  • CBA (computer-based assessment) format under supervision of the interviewer; multistage adaptive design
  • Respondents with no (or extremely limited) experience with the use of computers were given a pencil-and-paper version of the literacy and numeracy components.
  • 341 booklets: 4 paper-based booklets and 337 computer-based booklets/paths
  • 204 items: literacy (95 items), numeracy (95items), problem solving (14 items)
  • Short, open-ended response formats on paper or computer-based open-response formats (e.g., highlighting the correct phrase or word)

 

Technique 4: Reading components achievement test

  • Respondents with poor literacy and numeracy skills were directed to the reading components test, which was taken in pencil-and-paper format only.
  • Interviewers timed the completion of the reading components tasks using the above computer application.
  • 100 items
Techniques
  • achievement test
  • individual interview
Languages
  • Administration of the background questionnaire in 34 languages
  • Administration of assessment instruments in 31 languages
Translation procedures
  • Development of an international version of all assessment instruments in English
  • Translation into applicable languages of instruction by participating entities (national centers); guidance and assistance provided to countries throughout process by PIAAC Consortium, by developing and conducting linguistic quality assurance (LQA) and linguistic quality control (LQC) processes.
  • Translation verification by linguistic and assessment experts in order to ensure equivalence to the international version
Quality control of operations

Measures during data collection

  • General note: Participants were responsible for data collection within their own territories.
  • Provision of a set of technical standards and guidelines (TSG) for survey/operations/data collection.
  • Provision of software tools for supporting activities
  • Quality control monitoring with continuous support to countries in accordance with the TSG (standardized forms and reports to be completed at each step of the data collection, regular communication, etc.)
  • Training program produced by the Consortium (data management training, etc.)

 

Measures during data processing and cleaning, using the Statistical Analysis Software (SAS)

  • Standardized cleaning process at the national level (using IEA DME software)
  • Recoding, cleaning, and editing at the national level as discussed with and approved by the Consortium
  • Data processing at the international level by the Consortium
  • Repetition of data cleaning and comparison of new data sets with preceding version
  • Iterative process of data review and correction within the Consortium, the participating countries, and the OECD Secretariat.