SEA-PLM 2019 Design

Methodology
Quantitative Study
TOP
Method(s)

Overall approach to data collection

  • Proctored assessment of student achievement
  • Self-administered questionnaires for students, parents, teachers, and school principals
TOP
Target population

All children enrolled in the grade that represents 5 years of schooling counting from the first year of ISCED Level 1

TOP
Sample design
Stratified two-stage cluster sampling design

First Stage: Sampling schools

  • Schools were selected systematically with selection probability proportional to the number of enrolled Grade 5 children from the targeted population.
  • A minimum of 150 schools were sampled from each participating country.

 

Second Stage: Sampling classes within schools

  • One Grade 5 class was selected randomly within each sampled school.
  • All children of the selected class were sampled.
TOP
Sample size

Per country (achieved)

  • Between 149 and 233 schools
  • One Grade 5 class per school
  • Between 4,800 and more than 6,000 students

Across all countries: more than 31,000 students

TOP
Data collection techniques and instruments

Student assessment in mathematics, reading, and writing

  • Paper-and-pencil tests
  • Types of questions
    • Selected-response questions (four-option multiple-choice questions; in the mathematics and reading tests)
    • Constructed-response questions, with short responses (a number or solution in the mathematics test and a word or up to one or two sentences in the reading test)
    • Writing questions (in the writing test only; with responses ranging in length from a single word to multiple paragraphs)
  • Rotated booklet design
    • Eighteen different test booklets administered randomly in each sampled classroom and school
    • Each child had 1 hour to complete the test (2 out of the 3 domains in each booklet).

 

Global citizenship questionnaire

  • For students, it was administered as a separate module to the student questionnaire and without any time constraints.
  • For teachers, the respective questions were integrated in the background questionnaire.
  • For both
    • Paper-and-pencil
    • Likert-style item response format

 

Contextual questionnaires

Four questionnaires, all in paper-and-pencil format:

  • Student questionnaire
    • To be completed by students after the test
    • 30 minutes for completion of the questionnaire
  • Parent questionnaire: to be completed by the parents of participating children
  • School questionnaire: to be completed by school principals
  • Teacher questionnaire: to be completed by Grade 5 teachers
TOP
Techniques
  • achievement test
  • questionnaire
TOP
Languages

Assessment instruments administered in 8 different languages:

  • Khmer: Cambodia
  • Lao language: Lao PDR
  • Chinese, Malay, Tamil: Malaysia
  • Myanmar language: Myanmar
  • English: Philippines
  • Vietnamese: Viet Nam
TOP
Translation procedures
  • Source versions of all SEA-PLM 2019 materials were prepared in English.
  • English-language source materials were separately translated by two different translators into the target language.
  • The two translations were reconciled into a single national version by a third person.
  • The quality of the instruments and translations were pre-tested during a field trial in each participating country.
  • The process was monitored and approved by each country and verified by an external body in collaboration with the National Team based on standardized procedures.
TOP
Quality control of operations

Measures during data collection

  • In each country, the National Team was responsible for managing the data-collection operations and all activities conducted at the national level, such as printing, coding, and data entry.
  • The National Teams prepared and implemented the survey and supervised all operations and contributors using SEA-PLM guidelines and national resources.
  • The quality of the instruments was pre-tested during a field trial in each participating country on children from a sample of schools representing different national contexts.
  • People involved in the survey signed a confidentiality agreement to restrict access to SEA-PLM materials.
  • All administrators were trained by the National Team prior to operations, following SEA-PLM 2019 standardized materials, manuals, and procedures.
  • ACER technical experts supported National Teams in all operations by delivering in-country and remote assistance.
  • Countries reported these operations to the SEA-PLM Secretariat for memory and quality assurance.

 

Measures during data processing and cleaning

  • Technical standards were defined and agreed upon by participating countries and experts.
  • Standards guided technical activities, with consideration given to international conventions for large-scale assessments and the scope of SEA-PLM as a first round and as a regional assessment.
  • The need for consistency, precision and generalizability of the data collected was recognized by participating countries and stakeholders.
  • The Independent Technical Advisory Group of external experts was consulted to adjudicate the final database against the technical standards established for SEA-PLM 2019.
  • All countries reached the expected technical standards.
TOP
Sources - Technical documentation
TOP
Other sources
TOP