ICILS 2023 Results

Achievement and test scales
Scale Creation

Computer and information literacy scale

The ICILS CIL reporting scale was established for ICILS 2013, with a mean of 500 (the average CIL scale score across countries in 2013) and a standard deviation of 100 for the equally weighted national samples.

In 2023, two of the CIL modules were already used in 2013 and 2018, two were used in 2018 and three are new for ICILS2023.

Prior to scaling of the 2023 data, an extensive analysis of scaling properties was carried out that included reviews of missing values, test coverage, assessment of item fit, differential item functioning by gender, and cross-national measurement equivalence.

The ICILS test items were scaled using item response modeling with the (one-parameter) Rasch model.

  • The CIL scale was derived from student responses to the 104 test questions and large tasks (which corresponded to a total of 157 score points).
  • Most questions and tasks corresponded to a single item each; however, each ICILS large task was scored against a set of criteria (each criterion with its own unique set of scores) relating to the properties of the task. Each large-task assessment criterion was, therefore, also an item in ICILS.

Item parameters for CIL were obtained from a joint data file that included response data from both ICILS 2018 and ICILS 2023. This joint calibration methodology is also applied in IEA TIMSSPIRLS, and ICCS studies.

Additionally, plausible values as ability estimates were generated with full conditioning in order to take all student-level and between-school differences into account.

Countries were equally weighted within each ICILS cycle for the CIL calibration, and all items were included (except for items that were deleted nationally or internationally following the adjudication process).

Four proficiency levels were established for ICILS 2013 and, following the equating procedure, student achievement in ICILS 2023 could be reported against the same described proficiency levels. The descriptions of the levels were updated to accommodate the new test item material developed for ICILS 2023.

 

Computational thinking scale

As for the CIL scale, prior to scaling the CT data, extensive analysis of scaling properties was carried out that included reviews of missing values, test coverage, assessment of item fit, differential item functioning by gender, and cross-national measurement equivalence.

The CT test items were scaled using item response modeling with the (one-parameter) Rasch model. In 2023, two of the CT modules were already used in 2018 and had some tasks replaced for 2023, and two modules are new for ICILS2023.

  • The CT scale was derived from student responses to the 31 tasks and questions (which corresponded to a total of 65 score points).
  • Most questions and tasks corresponded to a single item each; however, the visual coding tasks were scored according to both the correctness of the solution (i.e., the degree to which the executed code completed the specified actions) and the efficiency of the code solution (measured by the number of code blocks used in a solution).

The final CT reporting scale was set to a metric with a mean of 500 (the ICILS average CT score) and a standard deviation of 100 for the equally weighted national samples.

Four proficiency levels were established for ICILS 2023 for CT, thereby providing test item locations on the CT achievement scale and allowing a description of these levels with example test items.

 

List of Achievement Scales

The computer and information literacy scale

The computational thinking scale

Questionnaire and background scales
Scale Creation

Two general types of indices could be distinguished, both of which derived from the ICILS questionnaires:

Simple indices

  • They were constructed through arithmetical transformation or simple recoding.
  • For example, ratios between ICT and students or an index of immigration background based on information about the country of birth of students and their parents.

 

Scale indices

  • They were derived from the scaling of items, a process typically achieved by using item response modeling of dichotomous or Likert-type items.
  • Item response modeling (applying the Rasch partial credit model) provided an adequate tool for deriving 13 international student questionnaire scales, 20 teacher questionnaire scales, and 16 school questionnaire scales.
  • A (national and international) composite index reflecting socioeconomic background was derived using principal component analysis of three home background indicators, namely, parental occupation, parental education, and home literacy resources.
  • Generally, the scales used in ICILS had sound psychometric properties, such as high reliability.

Only scale indices are reported below.

 

List of Background Scales

Student questionnaire

  • Learning about internet related tasks at school
  • Learning about internet related tasks outside of school
  • Academic-media multitasking
  • ICT self-efficacy regarding the use of general applications
  • ICT self-efficacy regarding the use of specialist applications
  • Students' perceptions of ICT - positive beliefs about ICT and society
  • Students' perceptions of ICT - negative beliefs about ICT and society
  • Students' perceptions of ICT - expectations for future use of ICT
  • Students' perceptions of ICT - learning with and use of ICT
  • Learning about CT in class
  • Learning about safe and responsible ICT use at school
  • Students' use of general ICT applications in class
  • Students' use of specialist ICT applications in class

 

Teacher questionnaire

  • Emphasis on ICT use in initial teacher education
  • Collaboration between teachers in using ICT
  • Availability of computer resources at school
  • Negative views on using ICT in teaching and learning
  • Positive views on using ICT in teaching and learning
  • Use of digital learning tools
  • Use of general utility software
  • Teachers ICT self-efficacy
  • Teachers' use of ICT for classroom activities - basic tasks
  • Teachers' use of ICT for classroom activities - advanced tasks
  • Teacher emphasis on CT related tasks in class
  • Emphasis on ICT capabilities in class
  • Teacher epistemological beliefs-embodied cognition
  • Teacher epistemological beliefs-cognitivism
  • Teacher epistemological beliefs-constructivist
  • Teacher centered teaching approach
  • Inquiry-based teaching approach
  • Experiential/scenario-based teaching approach
  • Teacher participation in professional learning activities
  • Teacher perceptions of need for professional learning

 

School questionnaires

  • Principals' views on using ICT for educational outcomes
  • Principals' reports on expectations of ICT use by teachers
  • Principals' reports on expectations for teacher collaboration using ICT
  • Principals' reports on priorities for facilitating use of ICT
  • Principals' use of ICT for general school-related activities
  • Principals' use of ICT for school-related communication activities
  • Principals' views on negative impact of ChatGPT on students' learning
  • Principals' views on positive impact of ChatGPT on students' learning
  • Principals' views on increased workload as a consequence of ChatGPT
  • Principals' views on negative consequences of ChatGPT on teacher work
  • Principals' views on positive consequences of ChatGPT on teacher work
  • ICT coordinators reports on computer resource hindrances to the use of ICT in teaching and learning
  • ICT coordinators reports on pedagogical resource hindrances to the use of ICT in teaching and learning
  • ICT coordinators reports on availability of ICT resources at school
  • ICT coordinators reports on availability of ICT resources at school - tools
  • ICT coordinators reports on availability of ICT resources at school - utility software
Overview of key study results

Computer and information literacy (CIL)

Variations in students’ CIL

  • Students’ CIL varied considerably both within and across countries.
  • Across countries, the range of average student CIL achievement scores was more than 220 scale score points, from 319 (Azerbaijan) to 540 scale score points (Rep. of Korea. 
  • Within most countries, the difference between the lowest performing students (bottom 10%) and the highest performing students (top 10%) was also more than 200 CIL scale score points.

 

Students’ proficiency in CIL

  • Many students demonstrated only basic CIL competence (below CIL Level 2). Students below CIL Level 2 generally require explicit step‐by‐step instructions to perform simple CIL actions associated with information location and communication in the digital environment.
    • On average across countries, nearly half of students’ CIL achievement was below Level 2. 
    • In some countries, more than three quarters of students’ CIL achievement was below CIL Level 2.
    • In the highest performing countries between a quarter to a third of students’ CIL achievement was below Level 2.
  • Trends in CIL proficiency
    • Students’ CIL achievement was typically lower in 2023 than in 2018 and 2013, in countries with comparable data across the cycles. 
    • There were some individual exceptions to this general pattern.

 

Computational thinking (CT)

Variations in students’ CT

  • Students’ CT varied both within and across countries. In comparison to CIL, students’ CT varied considerably more within countries than across countries.
  • Across countries, the range of average student achievement scores on the CT scale was 127 scale points, extending from 421 scale score points (Uruguay) to 548 scale score points (Chinese Taipei).
  • In most countries, the difference between the lowest performing students (bottom 10%) and the highest performing students (top 10%) was more than 270 scale score points.

 

Students’ proficiency in CT

  • On average across countries, the distribution of student CT achievement scores was centered around Level 2 on the CT achievement scale
    • 37% of students achieved scores that placed them within Level 2, with 34% of student scores below Level 2 and 29% of student scores above Level 2.
    • In 19 countries, the highest percentage of students had CT achievement scores at Level 2. No country had the highest percentage of students scoring below Level 1 or above Level 3.
  • Trends in CT proficiency
    • Students’ average CT did not change significantly between 2018 and 2023 in five of seven countries.
    • However, it increased significantly in one country and decreased significantly in two countries between 2018 and 2023.

 

Personal and social backgrounds related to students’ CIL and CT

  • Students from higher socioeconomic backgrounds (SES) consistently outperformed lower SES peers in both CIL and CT.
  • Students who spoke the test language at home performed better than those who did not.
  • Students with an immigrant background had lower average achievement than their non-immigrant peers.
  • The availability of computers at home was a positive predictor of CIL and CT in most countries but the relationship weakened after controlling for personal and social background likely indicating the close link between SES and device availability.

 

Gender differences 

  • CIL
    • Female students outperformed male students in every participating education system in CIL.
    • This is consistent with findings from previous ICILS cycles (2013, 2018) and suggests that female students, on average, have stronger competencies in evaluating and using digital information effectively.
  • CT
    • In contrast, male students slightly outperformed female students in CT.
    • The gender gap in CT was smaller than in CIL, and differences varied across education systems.

 

Association between CIL and CT

  • Moderate to strong correlation
    • ICILS 2023 found a moderate-to-strong positive correlation between CIL and CT achievement.
    • Students who performed well in CIL also tended to score higher in CT, suggesting overlapping cognitive skills.
  • Variation across education systems
    • While the association between CIL and CT was positive across countries, the strength of this relationship varied.
    • In some systems, explicit CT instruction led to stronger CT performance, even among students with lower CIL skills.
  • Influence of digital exposure and curriculum
    • Students with more exposure to ICT in learning environments tended to show stronger competencies in both CIL and CT.
    • Countries with integrated curricula for digital literacy and computational thinking exhibited a stronger association.

 

Engagement with Information and Communications Technology (ICT

Student’s use of ICT

  • Outside and inside school
    • 47% of students reported using ICT daily outside school for school-related activities.
    • Only 33% of students reported using ICT daily in school for learning purposes.
    • Students reported learning more about internet-related topics outside school than in structured classroom settings.
  • Variations by SES
    • Students from higher SES backgrounds reported more frequent ICT use for educational purposes than their lower SES peers.
    • Lower SES students were more likely to use ICT for entertainment rather than learning, highlighting a digital divide in educational technology use.
  • Gender differences in ICT engagement
    • Female students engaged more with communication and information-related digital tasks, while male students showed slightly higher engagement with programming and technical aspects of ICT.
Sources - Report(s) of results