ICILS 2023 OUTCOME MEASURES

Assessment domain(s)
  • Computer literacy
  • Information literacy
  • Computational thinking
TOP
Achievement and test scales
Scale Creation

Computer and information literacy scale

The ICILS CIL reporting scale was established for ICILS 2013, with a mean of 500 (the average CIL scale score across countries in 2013) and a standard deviation of 100 for the equally weighted national samples.

In 2023, two of the CIL modules were already used in 2013 and 2018, two were used in 2018 and three are new for ICILS2023.

Prior to scaling of the 2023 data, an extensive analysis of scaling properties was carried out that included reviews of missing values, test coverage, assessment of item fit, differential item functioning by gender, and cross-national measurement equivalence.

The ICILS test items were scaled using item response modeling with the (one-parameter) Rasch model.

  • The CIL scale was derived from student responses to the 104 test questions and large tasks (which corresponded to a total of 157 score points).
  • Most questions and tasks corresponded to a single item each; however, each ICILS large task was scored against a set of criteria (each criterion with its own unique set of scores) relating to the properties of the task. Each large-task assessment criterion was, therefore, also an item in ICILS.

Item parameters for CIL were obtained from a joint data file that included response data from both ICILS 2018 and ICILS 2023. This joint calibration methodology is also applied in IEA TIMSS, PIRLS, and ICCS studies.

Additionally, plausible values as ability estimates were generated with full conditioning in order to take all student-level and between-school differences into account.

Countries were equally weighted within each ICILS cycle for the CIL calibration, and all items were included (except for items that were deleted nationally or internationally following the adjudication process).

Four proficiency levels were established for ICILS 2013 and, following the equating procedure, student achievement in ICILS 2023 could be reported against the same described proficiency levels. The descriptions of the levels were updated to accommodate the new test item material developed for ICILS 2023.

 

Computational thinking scale

As for the CIL scale, prior to scaling the CT data, extensive analysis of scaling properties was carried out that included reviews of missing values, test coverage, assessment of item fit, differential item functioning by gender, and cross-national measurement equivalence.

The CT test items were scaled using item response modeling with the (one-parameter) Rasch model. In 2023, two of the CT modules were already used in 2018 and had some tasks replaced for 2023, and two modules are new for ICILS2023.

  • The CT scale was derived from student responses to the 31 tasks and questions (which corresponded to a total of 65 score points).
  • Most questions and tasks corresponded to a single item each; however, the visual coding tasks were scored according to both the correctness of the solution (i.e., the degree to which the executed code completed the specified actions) and the efficiency of the code solution (measured by the number of code blocks used in a solution).

The final CT reporting scale was set to a metric with a mean of 500 (the ICILS average CT score) and a standard deviation of 100 for the equally weighted national samples.

Four proficiency levels were established for ICILS 2023 for CT, thereby providing test item locations on the CT achievement scale and allowing a description of these levels with example test items.

 

List of Achievement Scales

The computer and information literacy scale

The computational thinking scale

TOP
Questionnaire and background scales
Scale Creation

Two general types of indices could be distinguished, both of which derived from the ICILS questionnaires:

Simple indices

  • They were constructed through arithmetical transformation or simple recoding.
  • For example, ratios between ICT and students or an index of immigration background based on information about the country of birth of students and their parents.

 

Scale indices

  • They were derived from the scaling of items, a process typically achieved by using item response modeling of dichotomous or Likert-type items.
  • Item response modeling (applying the Rasch partial credit model) provided an adequate tool for deriving 13 international student questionnaire scales, 20 teacher questionnaire scales, and 16 school questionnaire scales.
  • A (national and international) composite index reflecting socioeconomic background was derived using principal component analysis of three home background indicators, namely, parental occupation, parental education, and home literacy resources.
  • Generally, the scales used in ICILS had sound psychometric properties, such as high reliability.

Only scale indices are reported below.

 

List of Background Scales

Student questionnaire

  • Learning about internet related tasks at school
  • Learning about internet related tasks outside of school
  • Academic-media multitasking
  • ICT self-efficacy regarding the use of general applications
  • ICT self-efficacy regarding the use of specialist applications
  • Students' perceptions of ICT - positive beliefs about ICT and society
  • Students' perceptions of ICT - negative beliefs about ICT and society
  • Students' perceptions of ICT - expectations for future use of ICT
  • Students' perceptions of ICT - learning with and use of ICT
  • Learning about CT in class
  • Learning about safe and responsible ICT use at school
  • Students' use of general ICT applications in class
  • Students' use of specialist ICT applications in class

 

Teacher questionnaire

  • Emphasis on ICT use in initial teacher education
  • Collaboration between teachers in using ICT
  • Availability of computer resources at school
  • Negative views on using ICT in teaching and learning
  • Positive views on using ICT in teaching and learning
  • Use of digital learning tools
  • Use of general utility software
  • Teachers ICT self-efficacy
  • Teachers' use of ICT for classroom activities - basic tasks
  • Teachers' use of ICT for classroom activities - advanced tasks
  • Teacher emphasis on CT related tasks in class
  • Emphasis on ICT capabilities in class
  • Teacher epistemological beliefs-embodied cognition
  • Teacher epistemological beliefs-cognitivism
  • Teacher epistemological beliefs-constructivist
  • Teacher centered teaching approach
  • Inquiry-based teaching approach
  • Experiential/scenario-based teaching approach
  • Teacher participation in professional learning activities
  • Teacher perceptions of need for professional learning

 

School questionnaires

  • Principals' views on using ICT for educational outcomes
  • Principals' reports on expectations of ICT use by teachers
  • Principals' reports on expectations for teacher collaboration using ICT
  • Principals' reports on priorities for facilitating use of ICT
  • Principals' use of ICT for general school-related activities
  • Principals' use of ICT for school-related communication activities
  • Principals' views on negative impact of ChatGPT on students' learning
  • Principals' views on positive impact of ChatGPT on students' learning
  • Principals' views on increased workload as a consequence of ChatGPT
  • Principals' views on negative consequences of ChatGPT on teacher work
  • Principals' views on positive consequences of ChatGPT on teacher work
  • ICT coordinators reports on computer resource hindrances to the use of ICT in teaching and learning
  • ICT coordinators reports on pedagogical resource hindrances to the use of ICT in teaching and learning
  • ICT coordinators reports on availability of ICT resources at school
  • ICT coordinators reports on availability of ICT resources at school - tools
  • ICT coordinators reports on availability of ICT resources at school - utility software
TOP