TEDS-M 2008 Design

Methodology
Quantitative Study
Method(s)
  • Overall approach to data collection
    • Survey
  • Specification
    • Cross-sectional
Target population

Institutions

Tertiary or post-secondary schools, colleges, or universities offering one or more structured “opportunities to learn” (i.e. a program or programs) on a regular and frequent basis to future teachers as part of a route of teacher preparation

 

Future primary and lower secondary teachers

All members of a teacher preparation route in their last year of training enrolled in an institution offering formal opportunities to learn to teach mathematics, explicitly intended to prepare individuals qualified to teach mathematics in any of Grades 1 to 8 (ISCED Levels 1 and 2)

 

Educators

All persons with regular, repeated responsibility to teach future teachers one of the compulsory courses of their program at any year of the program

Sample design

TEDS-M targeted four distinct populations (institutions, educators, primary and secondary future teachers); thus four distinct sampling plans were designed and implemented.

Where required, institutions to be included were selected by systematic random sampling within explicit strata.

  • If reliable measures of size for the institutions were available, probability proportional to size (PPS) sampling was used.
  • Otherwise, or if an institution was so small that a census of individuals within the institution was anticipated, institutions were sampled with equal probabilities.
  • When implicit stratification was used, institutions were sorted by implicit stratum and a measure of size prior to sampling.

Within institutions, simple random samples of session groups or individual future teachers were selected. Further, systematic random samples of mathematics educators or mathematics pedagogy educators and general pedagogy educators were selected.

Sample size

Per country and target level (primary vs. lower secondary)

  • Institution sample: 50 (or all)
  • Future teachers: 30 per program per institution (or all)
  • Educators: 30 per type (or all)

 

Achieved sample size

Depending on the circumstances of the respective countries, the achieved sample sizes ranged between:

  • 1 and 78 institutions
  • 36 and 2,266 future primary teachers
  • 53 and 2,141 future lower secondary teachers
  • 43 and 1,212 educators
Data collection techniques and instruments

Future Teacher assessment in mathematics content knowledge and mathematics pedagogy content knowledge

  • Written format
  • Two types of questions
    • Multiple-choice (at least half of the total number of points)
    • Constructed-response
  • Primary Future Teacher booklets
    • 5 blocks of items
    • 5 booklets with 2 blocks each
    • 50 items per participant
  • Lower-secondary Future Teacher booklets
    • 3 blocks of items
    • 3 booklets with 2 blocks each
    • 40 items per participant
  • Matrix sampling of items (rotated test booklet design)
  • Additional questions on the future teacher’s:
    • beliefs
    • opportunities to learn
    • background and demographics

 

Context questionnaires

  • Educator questionnaire to be completed by educators in the teacher preparation programs; print format
  • Institutional program questionnaire to be completed by institution’s designated TEDS-M coordinator together with experts knowledgeable about the teacher education program of that specific institution; print format
  • Route questionnaire to be completed by TEDS-M NRCs; electronic format

 

Descriptive encyclopedia chapters

  • One chapter for each participating entity
  • Based on an outline by experts from the ministry of education, research institutes, or institutions of higher education in the participating countries or benchmarking entities
  • Format: electronic

 

Policy report

Reports by country experts following guidelines set by the TEDS-M international team

 

Curriculum study

  • Additional analysis:
    • Primary and secondary mathematics curricula
    • Curricula for mathematics teacher preparation
  • Although the in-depth analysis of these documents was a national option, all countries completed it as a precursor to the development of the assessments in order to determine the balance of the areas or domains to be measured.

 

Cost study

Secondary analysis of national survey data and/or national statistics data

Techniques
  • achievement test
  • documents
  • questionnaire
Languages
  • The TEDS-M instruments were administered in 12 different languages.
  • The most common language was English (6 out of 17 countries)
    Translation procedures
    • Development of an international version of all assessment instruments and questionnaires in English by the TEDS-M International Study Center
    • Translation into applicable languages of instruction by participating entities
    • Verification by linguistic and assessment experts in order to ensure equivalence with the international version
    • A second verification phase by the TEDS-M International Study Center
    Quality control of operations

    During data collection

    • Participants responsible for data collection within their respective territories
    • Standardized survey operation procedures: step-by-step documentation of all operational activities provided with manuals
    • Full-scale field test of all instruments and operational procedures (in each participating country and entity)
    • Provision of software tools for supporting activities (e.g., sampling and tracking future teachers and educators, administering future teacher booklets and educator questionnaires, documenting scoring reliability, creating and checking data files)
    • Training of national research coordinators (NRCs) and their staff, institution coordinators, survey administrators, etc.
    • Visits by international quality control monitors (IQCMs) during test administration to at least 10% (but no fewer than 3) of the institutions per country, resulting in observations in 85 institutions in total and visits to 3–8 institutions per country
    • National quality control program
    • Survey activities questionnaire (SAQ) to be completed by NRCs

     

    During data processing and cleaning – using Statistical Analysis Software (SAS)

    • Testing of all data cleaning programs with simulated data sets
    • Material receipt database
    • National adaptation database
    • Standardized cleaning process
    • Repetition of data cleaning and comparison of new data sets with preceding versions
    • Identification of irregularities in data patterns and correction of data errors