ICILS 2018 Design

Methodology
Quantitative Study
TOP
Method(s)
ICILS 2018
  • Overall approach to data collection
    • Proctored assessment and self-administered surveys
  • Specification
    • Cross-sectional
    • Some trend reporting possible with previous cycles

 

ICILS Teacher Panel 2020

  • Overall approach to data collection: Self-administered surveys

  • Specification: Longitudinal

TOP
Target population
ICILS 2018
  • Students: The ICILS target population comprises students in their eighth year of schooling. In most education systems, the eighth year of schooling is Grade 8, provided that the average age of students are 13.5 years or above. In education systems where the average age in Grade 8 is below 13.5, Grade 9 is defined as the ICILS target population.
  • Teachers: The population for the ICILS teacher survey was defined as all teachers teaching regular school subjects to the students in the target grade at each school sampled.
  • Schools: The population for the ICILS school survey comprised schools at which target grade students were enrolled.
  • ICILS also administered separate questionnaires to principals and designated ICT-coordinators in each school.

 

ICILS Teacher Panel 2020
  • Students: The panel study did not survey students.
  • Teachers: All teachers eligible for ICILS 2018 except those who left the profession.
  • Schools: All schools eligible for ICILS 2018 that were still operating in 2020.
  • The panel study also administered separate questionnaires to principals and designated ICT-coordinators in each school.
TOP
Sample design
ICILS 2018: Multistage (two stage) cluster sampling

First Stage (Schools)

  • Random selection of schools with a probability proportional to size as measured by the number of students enrolled in a school.
  • Each country was advised for a minimum sample size of 150 schools except some small education systems where all schools were included in the survey.

Second Stage (Students)

  • Twenty students were randomly sampled among all the enrolled students in the target grade.
  • All students were invited to participate for schools with fewer than 20 students in a targeted grade.

Second Stage (Teachers)

  • Fifteen teachers were selected randomly from all teachers teaching in the target grade at each sampled school.
  • In schools with fewer than 20 teachers, all teachers were invited to participate in the survey.

 

ICILS Teacher Panel 2020

The Teacher Panel is based on the existing random samples of schools and teachers from the 2018 ICILS survey in the three participating countries:

  • As far as teachers are concerned, the goal was to identify the exact same individuals
  • In contrast, for principals and ICT-coordinators, the questionnaires focused on school‐level details, so that finding the same individuals from the 2018 study was not necessary.

However, the data are subject to a certain degree of uncertainty because

  • Not all schools and teachers responding in 2020 participated in the 2018 survey.
  • Not all schools and teachers responding in 2018 participated in the repeated survey in 2020.
TOP
Sample size
Per Country (intended)
  • School Sample: 150-261 schools (minimum sample = 150, except for two smaller systems, for which the achieved samples were 35 and 110 schools, respectively)
  • Student Sample: 20 per school (minimum)
  • Teacher Sample: 15 per school (minimum)

 

Total (achieved)

ICILS 2018

  • Students: around 46,000
  • Teachers: around 26,000
  • Schools: around 2,200

ICILS Teacher Panel 2020

In the 3 countries and participating in both study years (2018 and 2020)

  • Teachers: more than 2150
  • Schools: more than 260
TOP
Data collection techniques and instruments
ICILS 2018

Student CIL and CT assessments  

  • The tests were embedded within modules. In total, there were five 30 minutes CIL modules and two 25 minutes CT modules.
  • Each student completed 2 out of 5 CIL modules. The order and selection of the CIL modules were assigned randomly.
  • In countries participating in the CT option, each student completed the two 25-minute modules following completion of the CIL test and student questionnaire. The CT module order was balanced across students.

 

Student questionnaire

  • Following the CIL assessment, each student completed a 30-minute student questionnaire.
  • The student questionnaire was used to investigate student engagement with ICT and included questions related to experience and use of ICT, their attitudes towards the use of computers and ICT and background characteristics.

 

Questionnaires for teachers, ICT coordinators, and principals

  • ICILS 2018 also included a teacher questionnaire, which was completed by 15 randomly selected Grade 8 teachers in each sampled school.
  • The questions were related to familiarity of teachers with ICT, their use of ICT in educational activities, their perceptions of ICT in schools, learning to use ICT in teaching, and their background characteristics.
  • The ICT coordinators also had to complete a 15-minute questionnaire, which included questions related to the resources of ICT in school, ICT use in school, ICT technical support, and provisions for professional development in ICT.
  • The principal of respective schools completed a 15-minute questionnaire answering questions related to school characteristics, policies, procedures, and priorities for ICT.

 

National coordinator questionnaires

  • National research coordinators (NRCs) collected data from experts in a national context survey (NCS).
  • The survey was used for gathering information about the structure of the education system and systematic descriptions of policy and practice in the use of ICT in school education.

 

ICILS Teacher Panel 2020

Teacher questionnaire

  • Expected to take 40 minutes to complete,
  • Compared to 30 minutes for the ICILS 2018 study.

 

School principal questionnaire

  • Expected to take 20 minutes,
  • Compared to 15 minutes in the ICILS 2018 study.

 

ICT coordinator questionnaire

  • Expected to take 18 minutes,
  • Compared to 15 minutes for the 2018 study.

 

All questionnaires were administered online, using the same version of the IEA Online Survey System (OSS) as in ICILS 2018.

TOP
Techniques
  • achievement test
  • questionnaire
TOP
Languages
ICILS 2018
  • Administration of assessment instruments and questionnaires in 17 different languages
  • The most common languages
    • English (2 countries)
    • Spanish (2 countries)
    • German (2 countries)
    • Russian (2 countries)

 

ICILS Teacher Panel 2020

Administration of questionnaires in 4 different languages:

  • Danish (in Denmark)
  • Finnish and Swedish (in Finland)
  • Spanish (in Uruguay)
TOP
Translation procedures
  • Development of an international version of all assessment instruments in English by the ICILS International Study Center
  • Translation into applicable languages of instruction by national research coordinators (NRCs)
  • Translation verification by linguistic and assessment experts in order to ensure equivalence with the international version
TOP
Quality control of operations

Measures during data collection

  • Participants were responsible for data collection within their own respective territories.
  • Standardized survey operation procedures: step-by-step documentation of all operational activities provided with the operation manuals
  • Full-scale field test of all instruments and operational procedures (in each participating country and entity)
  • Provision of software tools for supporting activities (e.g., sampling and tracking classes and students; administering school and teacher questionnaires; documenting scoring reliability; creating, and checking data files)
  • Training, e.g., for national research coordinators (NRCs) and their staff, for school coordinators, and test administrators
  • School visits conducted by international quality observers (IQOs) during test administration (at 15 schools per grade and country)
  • Training, e.g., for IQOs for procedures related to arranging visits to schools, interviewing school coordinator and test administrator, and reporting of these activities
  • Documentation of data collection activities by IQOs to determine whether the ICILS assessment was administered in compliance with the standardized procedures
  • National quality control program by national study centers that involved national quality officers (NQOs) to visit 10 percent of the sampled schools (minimum 15 schools) during testing
  • Survey activities questionnaire (SAQ) completed by NRCs

 

 Measures during data processing and cleaning

  • Testing of all data cleaning programs with simulated data sets
  • Registering all incoming data and documents in a specific database recording the date of arrival
  • All systematic data recoding were documented.
  • Iterative data cleaning process on each national dataset until all data were consistent and comparable
  • Standardized cleaning process
  • Repetition of data cleaning and comparison of the new data sets with the preceding version
  • Finally, identification of irregularities in data patterns and correction.
TOP
Sources - Technical documentation
TOP
Other sources
TOP