ICILS 2018 Results
Scale Creation
Computer and information literacy scale
The ICILS CIL reporting scale was established for ICILS 2013, with a mean of 500 (the average CIL scale score across countries in 2013) and a standard deviation of 100 for the equally weighted national samples.
In 2018, three of the five test modules were the same as those used in ICILS 2013, and two modules were new for ICILS 2018.
Prior to scaling of the 2018 data, an extensive analysis of scaling properties was carried out that included reviews of missing values, test coverage, assessment of item fit, differential item functioning by gender, and cross-national measurement equivalence.
The ICILS test items were scaled using item response modeling with the (one-parameter) Rasch model.
- The CIL scale was derived from student responses to the 81 test questions and large tasks (which corresponded to a total of 102 score points).
- Most questions and tasks corresponded to a single item each; however, each ICILS large task was scored against a set of criteria (each criterion with its own unique set of scores) relating to the properties of the task. Each large-task assessment criterion was, therefore, also an item in ICILS.
Item parameters for CIL were obtained from a joint data file that included response data from both ICILS 2013 and ICILS 2018. This joint calibration methodology is also applied in IEA TIMSS, PIRLS, and ICCS studies.
Additionally, plausible values as ability estimates were generated with full conditioning in order to take all student-level and between-school differences into account.
Countries were equally weighted within each ICILS cycle for the CIL calibration, and all items were included (except for items that were deleted nationally or internationally following the adjudication process).
Four proficiency levels were established for ICILS 2013 and, following the equating procedure, student achievement in ICILS 2018 could be reported against the same described proficiency levels. The descriptions of the levels were updated to accommodate the new test item material developed for ICILS 2018.
Computational thinking scale
As for the CIL scale, prior to scaling the CT data, extensive analysis of scaling properties was carried out that included reviews of missing values, test coverage, assessment of item fit, differential item functioning by gender, and cross-national measurement equivalence.
The CT test items were scaled using item response modeling with the (one-parameter) Rasch model.
- The CT scale was derived from student responses to the 18 discrete tasks and questions (which corresponded to a total of 39 score points).
- Most questions and tasks corresponded to a single item each; however, the visual coding tasks were scored according to both the correctness of the solution (i.e., the degree to which the executed code completed the specified actions) and the efficiency of the code solution (measured by the number of code blocks used in a solution).
The final CT reporting scale was set to a metric with a mean of 500 (the ICILS average CT score) and a standard deviation of 100 for the equally weighted national samples.
Three proficiency regions were established for CT, thereby providing test item locations on the CT achievement scale and allowing a description of these regions complete with example test items.
List of Achievement Scales
The computer and information literacy scale
The computational thinking scale
Scale Creation
Two general types of indices could be distinguished, both of which derived from the ICILS questionnaires:
Simple indices
- They were constructed through arithmetical transformation or simple recoding.
- For example, ratios between ICT and students or an index of immigration background based on information about the country of birth of students and their parents.
Scale indices
- They were derived from the scaling of items, a process typically achieved by using item response modeling of dichotomous or Likert-type items.
- Item response modeling (applying the Rasch partial credit model) provided an adequate tool for deriving 10 international student questionnaire scales, nine teacher questionnaire scales, and seven school questionnaire scales.
- A composite index reflecting socioeconomic background was derived using principal component analysis of three home background indicators, namely, parental occupation, parental education, and home literacy resources.
- Generally, the scales used in ICILS had sound psychometric properties, such as high reliability.
- Confirmatory factor analyses showed satisfactory model fit for the measurement models underpinning the scaling of the questionnaire data.
Only scale indices are reported below.
List of Background Scales
Student questionnaire
Students’ general engagement with ICT
- Frequency of use of general ICT applications
- Frequency of use of specialist ICT applications
- Frequency of use of ICT for social communication
- Frequency of use of ICT for exchanging information
- Frequency of use of ICT for accessing content from the internet
Student engagement with ICT for school-related purposes
- Frequency of use of ICT for study purposes
- Frequency of use of general applications in class
- Frequency of use of specialist applications in class
Extent of student learning about ICT at school
- Extent to which students learned about CIL tasks at school
- Extent to which students learned about CT tasks at school
Students’ ICT self-efficacy
- ICT self-efficacy regarding the use of general applications
- ICT self-efficacy regarding the use of specialist applications
- Attitudes to ICT futures
Students’ perceptions of ICT
- Perceptions of positive effects of ICT on society
- Perceptions of negative effects of ICT on society
- Perceptions of personal futures with ICT
Teacher questionnaire
Teachers’ ICT self-efficacy
Teachers’ emphasis on developing ICT skills and coding skills
- Teachers’ emphasis on developing ICT capabilities in class
- Teacher emphasis of teaching CT-related tasks
Teachers’ use of ICT for class activities
Teachers’ use of ICT for teaching practices
Teachers’ use of ICT tools in class
- Teachers' use of digital learning tools
- Teachers' use of general utility software
Teachers’ perceptions of ICT resources and teacher collaboration
- Teachers' perceptions of the availability of computer resources at school
- Teachers' perceptions of the collaboration between teachers when using ICT
Teachers’ reports on ICT-related professional learning
- Teacher participation in structured learning professional development related to ICT
- Teacher participation in reciprocal learning professional development related to ICT
Teachers’ perceptions of positive outcomes of using ICT for teaching and learning
Teachers’ perceptions of negative outcomes of using ICT for teaching and learning
School questionnaires
Principals’ use of ICT
- Principals’ use of ICT for general school-related activities
- Principals’ use of ICT for school-related communication activities
Principals’ views on using ICT
Principals’ reports on expected ICT knowledge and skills of teachers
- Principals' reports on expectations of ICT use by teachers
- Principals' reports on expectations for teacher collaboration using ICT
Principals’ reports on priorities for ICT use at schools
- Principals’ views of priorities for facilitating use of ICT - hardware
- Principals’ views of priorities for facilitating use of ICT - support
ICT coordinators’ reports on the availability of digital resources at school
ICT coordinators’ reports on hindrances to the use of ICT for teaching and learning at school
- ICT coordinators reports on computer resource hindrances
- ICT coordinators reports on pedagogical resource hindrances
Computer and information literacy (CIL)
Variations in students’ CIL
- Students’ CIL varied more within countries than across countries.
- Variations across countries
- The difference between the highest and lowest average CIL scores across countries was 157 scale points.
- Denmark had the highest average CIL score of 553 points, and Kazakhstan had the lowest average CIL score of 395 points.
- Variations within countries
- Differences in the within-country student score distributions tended to be larger in countries with lower average achievement than in countries with higher average achievement scores.
- The difference between the lowest five percent and the highest ninety-five percent of students’ CIL scores within countries varied between 216 scale points in Denmark (lowest within-country variation) and 347 scale points in Kazakhstan (highest within-country variation).
Students’ proficiency in CIL
- The CIL achievement scale represented students’ proficiency at four levels with increasing sophistication in CIL.
- Students with CIL achievement scores at Level 1 demonstrated a functional working knowledge of computers as tools.
- Students at Level 2 could use computers to complete management tasks and gather information under direct instruction.
- Students at Level 3 demonstrated the ability to work independently on computers to gather information and complete management tasks.
- Students at Level 4 demonstrated control of computers to search for information and create information products.
- In all countries, 80 percent of students achieved scores that placed them within CIL Levels 1, 2, and 3, with majority of the students reporting a Level 2 CIL proficiency across all countries.
- The differences in average CIL achievement scores in the three countries that met the necessary sample participation requirements for trend analysis in ICILS 2013 and 2018 were small and not statistically significant.
Personal and social backgrounds related to students’ CIL
- Gender
- Female students demonstrated higher CIL achievement than male students.
- The average CIL scores of female students were statistically significantly higher than that of male students in 10 out of 13 countries and benchmarking participants.
- Socioeconomic status
- In all countries, students belonging to high socioeconomic status (denoted by high parental occupation, high parental education, and high number of books in the home) groups performed significantly higher on the CIL achievement scale than those belonging to lower socioeconomic groups.
- The socioeconomic background is a consistent positive predictor of CIL across countries, however, the effect sizes vary across countries.
- Immigrant and language background
- In 9 out of 13 countries and benchmarking participants, students from non-immigrant families performed significantly better than students from immigrant families.
- In 10 of 13 countries and benchmarking participants, students who identified the language of the ICILS test as the language they spoke at home had statistically higher CIL scores than those who reported speaking another language at home.
- Computer use outside school
- All participating entities indicated that students who reported having two or more computers at home had statistically significantly higher CIL scores than students who reported having less than two computers at home.
- In 12 out of 13 ICILS 2018 participants, students who reported having used computers for five or more years had statistically significantly higher CIL scale scores than those who reported having computer use experience of less than five years.
Computational thinking (CT)
Variations in students’ CT
- Students’ CT varied more within countries than across countries.
- Variations across countries
- The difference between the highest and lowest average CT scores across the eight participating countries and one benchmarking participant that completed the ICILS 2018 CT option was 76 scale points
- Korea had the highest average CT score of 536 points and Luxembourg had the lowest average CT score of 460 points.
- Variations within countries
- The difference between the lowest five percent and the highest ninety-five percent of students’ CT scores within countries varied between 266 scale points in Portugal (lowest within-country variation) and 371 scale points in Korea (highest within-country variation).
Students’ proficiency in CT
- The CT achievement scale represented students’ proficiency across three regions with increasing sophistication in CT:
- At the lower region of the scale, students could demonstrate a functional working knowledge of computation as input and output, which meant that they could record data from observed outputs and solve simple coding problems.
- At the middle region of the scale, students understood that computation could be enabled as practical solutions to real-world problems. They demonstrated the ability to plan solutions associating inputs with outputs and could implement solutions to complex coding using non-linear logic.
- At the upper region of the scale, students viewed computation as a generalizable problem-solving framework, infer the relationship between observed inputs and outputs to evaluate solutions, implement sophisticated solutions to complex coding problems using logic.
Personal and social backgrounds related to students’ CT
- Gender
- Across all countries, the average CT scale scores of male students were statistically significantly higher than that of female students. Female gender tended to be negatively related to CT scores.
- However, statistically significant differences in the average CT scale scores between female and male students were found in only two countries. In one of those countries the difference was in favor of female students and in the other it was in favor of male students.
- Socioeconomic status
- In all participating countries, students in the high socioeconomic status groups performed significantly higher on the CT scale than those belonging to the lower socioeconomic groups.
- Socioeconomic background is a consistent positive predictor of CT across countries; however, the effect sizes vary across countries.
- Immigrant and language background
- Immigrant and language background were found to be associated with student CT achievement.
- In six out of seven countries, students from non-immigrant families performed significantly better than students from immigrant families.
- In five of seven countries, students who identified the language of the ICILS test as the language they spoke at home had statistically higher CT scores than those who reported speaking another language at home.
- Computer use outside school
- All participating entities indicated that students who reported having two or more computers at home had statistically significantly higher CT scores than students who reported to have less than two computers at home.
- In all countries, students who reported having used computers for five or more years had statistically significantly higher CT scale scores than those who reported having computer use experience of less than five years.
Association between CIL and CT
- Student CT achievement was strongly associated with student CIL achievement.
- The correlation between students’ CIL and CT scale scores was found to be 0.82 on average, across all countries.
- This correlation varied between 0.74 and 0.89 across all countries.
Engagement with information and communications technology (ICT)
Student’s use of ICT
- Students’ daily use of ICT and experience with computers were consistent positive predictors of both CIL and CT.
- About less than half of the Grade 8 students had been using computers for five or more years.
- Seven out of 10 Grade 8 students used ICT daily outside of school, but only 1 in 5 students used it for school-related purposes.
- School-related use of ICT most often involved Internet searching and document production. About one quarter of the students used ICT weekly to collaborate with other students or organize their time and work, and to prepare reports and essays.
- Four out of 5 students indicated confidence in their ability to use ICT for basic tasks such as search for information, insert an image into a document, and complete a school assignment.
- While there was little difference between the confidence of male and female students in using ICT, male students expressed greater confidence in the use of specialist ICT applications.
- Confidence in using general ICT applications was associated with measured CIL and CT, but confidence in using specialist ICT applications was not.
- Male students expressed greater expectations of using ICT for work or study in the future than their female counterparts.
Teachers’ use of ICT
- On average across countries, more than two-thirds of teachers had at least five years of experience with using ICT during lessons or with their preparation.
- While majority of the teachers across countries reported they had confidence in the use of ICT, they were found to be a little hesitant on the use of online ICT engagement platforms and management systems.
- Older teachers were found to be more reluctant in the general use of ICT.
- While many teachers understood the positive outcomes of the use of ICT, some showed concerns about it having its potential negative effects.
- There were considerable differences across countries in the availability of ICT at schools, the extent of teacher collaboration, and conditions for professional learning.
- Teachers placed some or strong emphasis on developing skills related to computer and information literacy and computational thinking.