- Mathematics Problem Analysis Team
- Mathematics Quality Analysis Team
- Problem Implementation Analysis Team
- Text Analysis Team
- Teacher Questionnaire Coding Team

Mathematics Code Development Team

An international team was assembled to develop codes to apply to the TIMSS 1999 Video Study mathematics data. The team consisted of country associates (bilingual representatives from each country) and was directed by a mathematics education researcher. (The mathematics team did not include a representative from Japan because Japanese mathematics lessons were not filmed as part of the TIMSS 1999 video data collection.) The mathematics code development team was responsible for creating and overseeing the coding process, and for managing the international video coding team.

The mathematics code development team worked closely with two advisory groups: a group of national research coordinators representing each of the countries in the study, and a steering committee consisting of five North American mathematics education researchers.

International Video Coding Team

Members of the international video coding team represented all of the participating countries. They were fluently bilingual so they could watch the lessons in their original language, and not rely heavily on the English transcript. In almost all cases, coders were born and raised in the country whose lessons they coded.

Coders in the international video coding team applied 45 codes in seven coding passes. Most of the codes in the first few passes were coverage codes, and segmented the entire lesson into meaningful chunks that later could be studied in more detail. In Pass 1, coders marked the beginning and end of the lesson, and then divided the lesson into periods of public and private interaction. Passes 2 and 3 involved dividing the lesson into periods of time when mathematical problems were and were not worked on. Additionally, coders had to note the beginning and ending time of each problem, and write down the problem statement and problem solution.

The fourth pass was comprised of occurrence codes for specific events that might occur during the lesson, such as outside interruptions, goal statements, and lesson summaries. In the fifth and sixth passes numerous questions were asked about each mathematical problem that had been identified. For example, was the problem connected to the real world, how many solutions were presented publicly, and was the problem worked on or discussed by the class for more than 45 seconds.

Pass 6 also involved a series of questions about periods of time marked as private interaction, such as what kind of problems were students assigned to work on, and did they work individually or in groups. Another set of codes in Pass 6 explored whether particular resources were used during the lesson, such as computers and calculators. Finally, in Pass 7, coders divided each lesson into segments according to their purpose- addressing previously learned content, introducing new content, or practicing and applying new content.

Team members also created a lesson table for each video, which combined information from a number of codes. For example, the lesson tables noted when each mathematical problem began and ended, and included a description of the problem and the solution. These tables served a number of purposes: they acted as quick reference guides to each lesson, they were used in the development process for later codes, and they enabled problems to be further coded by specialist coding teams.

Coders established initial reliability on all codes in a coding pass prior to their implementation. After they finished coding approximately half of their assigned set of lessons (in most cases about 40-50 lessons), coders established midpoint reliability. The minimum acceptable reliability score for each code was 85 percent.

Specialist Coding Teams

The majority of codes presented in this report were applied to the video data by members of the international video coding team, who were cultural insiders and fluent in the language of the lessons they coded. However, not all of them were experts in mathematics or teaching. Therefore, several specialist coding teams with different areas of expertise were employed to create and apply special codes regarding the mathematical nature of the content, the pedagogy, and the discourse.

Mathematics Problem Analysis Team

The mathematics problem analysis team was comprised of individuals with expertise in mathematics and mathematics education. They developed and applied a series of codes to all of the mathematical problems in the videotaped lessons, using lesson tables prepared by the international video coding team.

The mathematics problem analysis team constructed a comprehensive, detailed, and structured list of mathematical topics covered in eighth grade in all participating countries. Each problem marked in a lesson was connected to a topic on the list. In addition, the group also coded the procedural complexity of each problem, the relationship among problems, and identified application problems.

The members of this team computed initial reliability as well as reliability after approximately two-thirds of the lessons had been coded. The percent agreement was above 85 percent for each code at both time points.

Mathematics Quality Analysis Team

A second specialist team possessed special expertise in mathematics and teaching mathematics at the post-secondary level. The same team previously was commissioned to develop and apply codes for the TIMSS 1995 Video Study. The mathematics quality analysis team reviewed a randomly selected subset of 20 lessons from each country except Japan. Japan was not included because the team already had analyzed a sub-sample of the Japanese lessons as part of the TIMSS 1995 Video Study.

Specially trained members of the international video coding team created expanded lesson tables for each lesson in this subset. Furthermore, the tables were "country-blind," with all indicators that might reveal the country removed. The mathematics quality analysis team worked solely from these written records, and had no access to the video data.

The mathematics quality analysis team created and applied a coding scheme that focused on mathematical reasoning, mathematical coherence, the nature and level of mathematical content, and the overall quality of the mathematics in the lessons. The scheme was reviewed by mathematics experts in each country and then revised based on the feedback received. The team applied their coding scheme by studying the written records of the lessons and reaching consensus about each judgment.

Problem Implementation Analysis Team

The problem implementation analysis team analyzed a subset of mathematical problems and examined 1) the types of mathematical processes implied by the problem statement and 2) the types of mathematical processes that were publicly addressed when solving the problem.

Using the video data, translated transcripts, and the same lesson tables provided to the mathematics problem analysis group, the problem implementation analysis team analyzed only those problems that were publicly completed during the videotaped lesson. Problems had to be publicly completed in order for the group to code for problem implementation. Furthermore, the group did not analyze data from Switzerland, since most of the Swiss transcripts were not translated into English.

Reliability of at least 85 percent was achieved for all countries.

Text Analysis Team

The text analysis team used all portions of the mathematics lesson transcripts designated as public interaction to conduct various discourse analyses. The team utilized specially designed computer software for these quantitative analyses of classroom talk. Because of resource limitations, computer-assisted analyses were applied to English translations of lesson transcripts.

Teacher Questionnaire Coding Team

The teacher questionnaire coding team included individuals who had helped develop the questionnaire and who had expertise in mathematics education. They developed quantitative codes in order to categorize teachers' responses to the open-ended questions. Teachers' answers to these questions were translated into English by individuals who were bilingual in English and one of the other relevant languages. Coding of the data then was carried out using the English translations. An 85% within country, inter-rater reliability criterion was used.

For more information on the codes applied by the teacher questionnaire coding team, download this document: add link to a pdf of the Teacher Questionnaire Coding Manual