PASEC 2019 Design

Methodology
Quantitative Study
TOP
Method(s)
  • Overall approach to data collection: proctored assessment and self-administrated questionnaires, supported by the PASEC Coordination Center
  • For Grade 2: one-to-one administration of both the test and the questionnaire
  • Specification: trend study
TOP
Target population

Students

All students formally enrolled in the early (Grade 2) and late (Grades 5 or 6) stages of a primary school (formal education) where the official language of instruction is the language used for the test, regardless of the type of school (public, private, community, etc.) and its location (rural/urban)

 

Teachers

Teachers in the surveyed schools

TOP
Sample design

STRATIFIED THREE-STRAGE CLUSTER SAMPLE DESING

First stage: school sampling

  • Stratification of schools by important variables (type of school, location)
  • Probability proportional to size (PPS): systematic sampling proportional to the total number of students in early (2nd grade) and late (5th/ 6th grade) primary education.
  • For each sampled school, sampling of one or two replacement schools
  • Sampling of different schools for the field trial (20 schools in each country)

 

Second stage: sampling classes within schools 

  • One class from the target grades of each school selected by random sampling
  • Carried out during school data collection operations by test administrators

 

Third stage: sampling students within sampled classes 

  • For Grade 2, selection of 16 students in each sampled class 
  • For Grade 6, selection of 25 students in each sampled class
  • Carried out during school data collection operations by test administrators
TOP
Sample size

INTENDED

  • Per country, Grade 6
    • At least 180 schools
    • One class in each school
    • 25 students from each sampled late primary class (approx. 4,500 students)
  • Per country, Grade 2
    • At least 90 schools from the Grade 6 sample
    • One class in each school
    • 16 students from each sampled early primary class (approx. 1,440 students)

All participating countries could extend their sample in order to improve subnational analysis.

 

ACHIEVED

Students

  • Grade 6: Across countries, approximately 63,000 students
  • Grade 2: Across countries, approximately 22,000 students

Teachers

Across countries, more than 19,100 teachers

TOP
Data collection techniques and instruments

Technique 1: student achievement test and background questionnaire for Grade 2

  • One-to-one administration of both the test and the questionnaire by test administrators over four mornings 
  • About 30 minutes per discipline (language, math) and for the questionnaire per student
  • Test specifics
    • One common booklet for all students
    • Test administrators were instructed to deliver instructions and questions word for word
    • Pupils answered questions orally with very short answers
    • Two mornings per discipline
  • Contextual questionnaire
    • One booklet 
    • Administered together with the test
    • Test administrators supported the completion of the questionnaire, reading questions aloud and translating them into the native language of students, if needed and possible, and completing the questionnaire based on each student's answers.

 

Technique 2: student achievement test and background questionnaire for Grade 6

  • Pencil and paper
  • Collective administration
  • Contextual questionnaire
    • Written questionnaire
    • Implementation on the first day of the study at the school
    • Test administrators supported the completion of the questionnaire, reading questions and response procedures aloud, and ensured that all students completed the questionnaire at the same time.
  • Test specifics
    • The reading and mathematics tests were taken on the second and third day 
      • Lasting up to 2 hours each
      • With a 10-minute break after 1 hour
    • For each discipline, use of four booklets with a rotating design
      • Each student administered one booklet with two blocks in reading and two blocks in math
      • The two blocks for one discipline were administered on the same day
      • One hour maximum per block

 

Technique 3: Teacher test and questionnaire

  • Teacher test
    • Paper and pencil
    • Administration of the entire teacher survey (tests and contextual questionnaire) in each school in one morning on the fourth day of the study
    • Four booklets, copies of which were handed out at random according to the principles of the rotating design
    • Multiple choice question (MCQ) format
  • Teacher/Class questionnaire
    • To be completed by the teachers of the sampled class on the day of the test
    • Test administrators could, if necessary, answer teachers’ questions and check that all the questions had been answered

 

Technique 4: Principal/School questionnaire

  • Completed by the principal of the school
  • Could be administered over the four days of the test
  • Test administrators could, if necessary, answer the principal’s questions and check that all the questions had been answered.
  • The second part of this questionnaire was completed with the administrator, who checked the principal’s answers against the documents and equipment available in the classroom and school.

 

Technique 5: Document analysis

  • One chapter for the international report 
    • To describe the context
    • Written by the coordination center, based on national documents and data
  • One chapter for each national report 
    • To describe the context
    • Written by the countries based on their national documents and data
TOP
Techniques
  • documents
  • questionnaire
  • achievement or student test
TOP
Languages

Assessment instruments administered in the following languages:

  • French, most countries
  • In addition/alternatively, in the following languages
    • English (Cameroon)
    • Arabic (Chad)
    • Kirundi (Burundi)
    • Malagasy (Madagascar)
TOP
Translation procedures
  • Test developed in French.
  • Translation and adaptation by a group of translators from participating countries with guidance and support provided by an international specialized agency
  • Independent verification and final validation by each country
TOP
Quality control of operations

Measures during data collection

  • Countries were responsible for data collection
  • Standardized survey operation procedures: step-by-step documentation of all operational activities provided with manuals
  • International training for national PASEC team at each step
  • International support and training for national PASEC teams and test administrators (e.g., about coding, data entry, and cleaning)
  • National quality control program
  • International quality control program

 

Measures during data processing and cleaning

After data collection

  • Standardized storage of instruments, with a manual of procedures made available for the national teams; verification of procedures on a country-by-country basis by PASEC
  • Training of the team on data entry, with a manual of procedures made available
  • Data entry controls incorporated (program, forms) 
  • Double data entry required for a percentage of the instruments (10% to 15%, depending on the type of instrument); data matching in order to estimate data entry errors; data entry repeated if necessary

Data cleaning

  • First phase of data cleaning at the PASEC center by means of various verification programs to check for inconsistencies, missing values, outlet values, etc.
  • Second phase of cleansing by national teams, as necessary, in the respective countries 

 

Measures during data analysis and report writing

  • PASEC (with expert support on some aspects) established estimates of student scores and contextual indices, followed by statistical analysis of the data; these steps were checked and validated by the PASEC Scientific Committee.
  • The writing of the international report as well as the national reports was also checked and validated by the Scientific Committee.
TOP
Sources - Technical documentation
TOP
Other sources
TOP