TIMSS 2015

National Quality Control Monitor Manual

Instructions for the NRC

This TIMSS 2015 National Quality Control Monitor Manual is provided to assist you in
implementing your national quality control program. This manual documents the guidelines to
follow when observing test administration in 10% of the participating schools. This manual is
generalized to meet the needs of both TIMSS and TIMSS Numeracy; it should be adapted
according to your countrys participation and other national aspects of your country. You may use
the provided TIMSS 2015 Classroom Observation Record (applies to TIMSS Numeracy as well)
provided by the TIMSS & PIRLS International Study Center or prepare your own.
This manual can be used as the basis for training your National Quality Control Monitors
(NQCMs). In addition to the contents of this manual, the training should include basic
information about the study, assessment instruments, and survey operations, with particular
attention to the purpose and use of different tracking forms and the Test Administration Form.
Guidelines for selecting National Quality Control Monitors and the basic information that the
monitors should collect during their site visits are provided in the TIMSS 2015 Survey
Operations Procedures Unit 5.
It is very likely that one NQCM will not be able to visit all testing sessions in the selected
schools (especially if the testing window is very short or your country is large) and you will need
to select additional NQCMs to visit schools and observe the testing sessions. In the case of a
large country, NQCMs should come from different states, regions, or districts of the country to
cover the range of school locations and environments.
Since all the testing materials are highly confidential, all NQCMs should sign a confidentiality
agreement stating that they are responsible for keeping their materials secure at all times and for
returning them to the national center after completing their tasks. Duplication or distribution of
the testing materials other than for the sole purpose of the study is not allowed.

Table of Contents
Introduction..............................................................................................................................1
1

NQCM Materials............................................................................................................2

Your Role and Responsibilities as a NQCM................................................................2

Selecting Schools for Classroom Observation.............................................................4

Visiting Schools...............................................................................................................5
4.1.
Arranging for Your Visit to the School................................................6
4.2.
Arriving at the School........................................................................7
4.3.
The Classroom Observation Record..................................................7
4.4.
Observing the TIMSS Testing Sessions..............................................8
4.5.
Roles and Responsibilities of the School Coordinator and the Test
Administrator....................................................................................8
4.6.
Interviewing the School Coordinator.................................................9
4.7.
Returning Your Materials to the National Center...............................9

Appendix: Classroom Observation Tracking Form............................................................11

TIMSS 2015 NQCM Manual

Introduction
Thank you for agreeing to be a National Quality Control Monitor (NQCM) for the IEAs Trends
in International Mathematics and Science Study (TIMSS 2015).
The International Association for the Evaluation of Educational Achievement (IEA) is an
independent international cooperative of national research institutions and government agencies
that has been conducting studies of cross-national achievement since 1959.
Internationally recognized as a valid and reliable measure of student achievement in mathematics
and science at the fourth and eighth grades, IEAs TIMSS has reported on international
achievement trends every four years since 1995. TIMSS 2015 is the sixth assessment, enabling
participating countries to chart the progress of their education systems over a 20-year period. It is
designed to measure and interpret differences in national educational systems in order to help
improve the teaching and learning of mathematics and science worldwide. As a new initiative to
extend the options that IEA provides for matching the TIMSS mathematics assessment to a
countrys educational development in primary school, TIMSS Numeracy is a less difficult
version of the TIMSS fourth grade assessment that concentrates on measuring childrens
numeracy learning outcomes.
More than 60 countries are participating in TIMSS 2015. Malaysia is among the countries
participating in this important project. In each participating country, a sample of schools has been
selected. Within each of the sampled schools, at least one class has been selected to participate.
Considerable effort has been made in developing standardized materials and procedures so that
the data collected in each country will be comparable to the greatest possible extent. To further
ensure the quality of the data, a national quality control program has been developed to
document data collection activities. The National Quality Control Monitors, including you, will
be making visits on the day of testing to at least 10% of the participating schools (per grade
level) to observe the actual TIMSS testing sessions. You will not interfere in the testing in any
way. Your responsibility is only to observe whether the standardized procedures are followed.

Purpose and Organization of This Manual

This manual provides you, as a NQCM, with information about your role and responsibilities in
this project. This manual is intended to supplement the training that you are or will be receiving
from the national center.
The manual is organized in four sections:
Section 1 lists the materials that you will need to complete your tasks and indicates when you
will receive them.
Section 2 defines your role and responsibilities.
Section 3 describes procedures for selecting schools for observation.
Section 4 describes procedures and tasks for visiting the selected schools.

TIMSS 2015 NQCM Manual

1 NQCM Materials
During or after the training, you will receive the following materials that you will need to
complete your tasks:

NQCM manual, which explains the procedures you will follow;

TIMSS 2015 Survey Operations Procedures Unit 4 and Unit 5;

School Coordinator Manual;

Test Administrator Manual(s); and

A copy of the Classroom Observation Tracking Form.

Once you (in cooperation with the national center) have selected schools for your observations,
you will receive the following materials:

Copies of the Classroom Observation Records to use during your school visits;

A copy of the completed Class Listing Form, Student Tracking Form, and the Teacher
Tracking Form for each school/class you will be visiting; and

The address of each school you will be visiting and the name and telephone number of
the School Coordinator.

2 Your Role and Responsibilities as a NQCM

As an international study measuring student achievement in mathematics and science, TIMSS
expects that all data collection activities are conducted to the highest standards and follow the
agreed upon international procedures. Your role as a National Quality Control Monitor is to help
document data collection activities in your country to ensure that the standard, international data
collection procedures were followed at all times.
To accomplish this, you will visit a sample of schools as a representative of your countrys
national center to observe the data collection. During your visits to each of the schools, you will
observe an actual testing session and use the Classroom Observation Record provided by the
national center to record your observations. You also will interview the School Coordinator.
Prior to visiting schools, you will be trained by your countrys national center to learn about the
project in your country and your responsibilities as a National Quality Control Monitor.

TIMSS 2015 NQCM Manual

In summary, your responsibilities will include the following:

Become thoroughly familiar with your role and responsibilities as a monitor.

Become thoroughly familiar with the testing procedures, as described in the manuals
provided. In particular, you should familiarize yourself with the Test Administrator
Manual, because it explains the procedures for the testing sessions you will be observing.

Work with the national center to select schools for your observations.

Ensure that you have the completed Class Listing Form and Teacher Tracking Form for
each school you are visiting, as well as the Student Tracking Form for each of the
classrooms you will be observing. As part of your duties, you will need to check this
documentation with the School Coordinator and/or Test Administrator after the testing
session.

Contact each school that you have selected about five working days before the testing
session is scheduled. You will need to make arrangements with the School Coordinator to
meet him or her on the testing day and to schedule an interview time.

Arrange to arrive at the school approximately one hour before the testing session.

For each testing session, you will complete a Classroom Observation Record.

After the testing session, check with the Test Administrator to update information on your
copy of the Student Tracking Form.

During your school visit, you will interview the School Coordinator. The details of the
interview will be recorded in the last section of the Classroom Observation Record.

Upon completion of your school visit, attach the corresponding Class Listing Form,
Teacher Tracking Form, and Student Tracking Form to the completed Classroom
Observation Record for that school. Keep these documents together until you complete
all of your school visits.

Upon completion of all of your school visits, return your finalized Classroom
Observation Records to the national center along with the Class Listing Forms, Student
Tracking Forms, and Teacher Tracking Forms.

TIMSS 2015 NQCM Manual

3 Selecting Schools for Classroom Observation

The task is to select 10% of participating schools, plus three extra schools for each target grade.
The extra schools should be selected as replacements if you have difficulty contacting the
original selections.
It is possible that you will not be able to visit all 10% of the participating schools. In this event,
additional NQCMs will visit schools and observe the testing sessions in your country, allowing
you to visit a portion of the schools selected for quality control observations.
The selection of schools to be visited should be carried out jointly with the national center. As
much as possible, the schools to be visited should be chosen using a random selection process.
However, such a selection often is subject to a number of practical constraints. For example, the
schools assigned to you should be within a reachable distance for you (as specified by the
national center), taking into account the testing times given.

Procedures for Selecting the Schools

First, obtain the list of participating schools. Considering any constraints for selecting schools for
observation, you should then prepare a list of eligible schools. The following steps describe the
procedure for selecting the schools:

Sort the schools by their location and remove the ones that are not eligible based on
practical constraints previously determined by the national center;
Prepare a list of all remaining eligible schools; and
Sort schools on the list by the testing date.

Once the list of eligible schools is prepared, use the following steps to select the schools for
observation:
1. Calculate, how many schools you will need to visit. Since your aim is to select 10% of
schools, divide the number of participating schools (not just eligible schools) by 10 and
round to the nearest whole number. This is the number of schools you (shared with other
NQCMs, if applicable) will need to visit. For example, if there are 150 participating
schools, you will need to visit 15 schools.
2. Number the schools on your list of eligible schools.
3. Compute the sampling interval. This number is the number of eligible schools divided by
the number of schools you will need to visit. Round the result to a whole number. For
example, if there are 135 eligible schools and you need to visit 15 of them, the sampling
interval is 9.

TIMSS 2015 NQCM Manual

4. Pick an initial random number between 1 and 10 to select your first school from your list
of eligible schools. For example, if your initial number is 3, select the third school on the
list.
5. Start with your first selected school, and mark off every Nth school on the list, where N
corresponds to your sampling interval calculated in step 3.
6. Print a copy of the Classroom Observation Tracking Form from the appendix of this
manual. Document the schools name and address, the Class IDs, and the scheduled
testing date(s).
7. Repeat the above process until you have selected the number of schools needed, as
determined in step 1.
8. Remember to select at least three extra schools for each grade as possible replacements.
9. Finally, if any of your selected schools has more than one class participating in the
corresponding study/grade level, select one class to observe.

4 Visiting Schools
Your job as a NQCM will be devoted to conducting on-site classroom observations of the TIMSS
and/or TIMSS Numeracy assessment administration. For the schools you have selected for your
observations, you will need to observe the achievement testing sessions, the administration of the
Student Questionnaire, and the distribution of the Early Learning Survey (Home Questionnaire
for TIMSS Numeracy and TIMSS at the fourth grade), and interview the School Coordinator.
In many ways, your role during the school visits is a challenging one. Without being intrusive or
disruptive to the study, you will need to observe the testing session, discuss administration
practices with the School Coordinator, and keep record of how TIMSS procedures are handled in
the school. It is difficult to provide specific instructions on how to conduct these tasks
effectively. Much will depend on the situation in the schools, and you will need flexibility to
adjust to these varying circumstances. Generally, however, the majority of the School
Coordinators will understand the importance of your visits and be cooperative.
Being well prepared when you arrive at the school will help you adjust to any circumstance you
may encounter within a school. Prior to your school visits, you should be thoroughly familiar
with the testing procedures as described in the School Coordinator Manual and the Test
Administrator Manual.
On the day of your school visit, you will gather information about several aspects of the data
collection which will involve watching an actual testing session and interviewing the School
Coordinator.

TIMSS 2015 NQCM Manual

Here are some guidelines for approaching your role as a TIMSS representative:

Be on time!
Your interaction with the school staff members should consistently reflect an appreciation
of their efforts. By having a professional manner, you will indicate to the schools staff
that the project is important and worthy of their time.
Be sure to observe all school rules and protocols (e.g., obtaining a pass if needed, etc.).
Be thoroughly prepared and know what information you need to gather. In this way, you
will be efficient and minimize disruption.
Be friendly and personable. Developing a good relationship with the school staff
members will make the job easier for everyone.

It is very important that you strictly remain an observer in the visited schools and do not actively
participate in the testing sessions.

4.1. Arranging for Your Visit to the School

You will need to contact the School Coordinator in each of the schools and schedule your visits
at least five working days before the scheduled testing date. You should call and introduce
yourself to the School Coordinator, explain your role in the study, and provide the School
Coordinator with an overview of your responsibilities.
This call is important to ensure that the school has advance notice and will be able to
accommodate your visit. If you do not make contact five days before the first testing day, double
your efforts on the next day. If you cannot reach them by the end of the third day, depending on
your situation, you may need to contact the national center for additional assistance.
Before making the call, review the information about the school that you obtained from the
national center. Also, determine whether you will need directions to the school from the School
Coordinator. Finally, it is best to have your TIMSS materials available for reference if the School
Coordinator has any questions about the quality assurance activities and your role as a NQCM.
During the call, you will need to:
1. Introduce yourself to the School Coordinator. Explain that, in your role as a NQCM, you
have been asked to observe testing sessions in a random sample of schools in order to
document the activities.
2. Explain that, in addition to observing the TIMSS testing sessions, you also will need to
conduct a brief interview with the School Coordinator to ask his or her opinion about the
testing procedures. Schedule this interview for (no more than 30 minutes) after the testing
session.
3. Explain that you also must double check the booklet and questionnaire labels and
compare them with the records in the Student Tracking Forms.

TIMSS 2015 NQCM Manual

4. Verify the information you obtained from the national center about the testing session,
including the schools address, the testing date, and the testing time. Explain that you will
be arriving at the school about one hour before the testing session.
5. Ask the School Coordinator whether he or she will be administering the test or if there is
a separate Test Administrator. If there is a separate Test Administrator, ask for the name
of that person and about the best time to call and introduce yourself.
If you receive questions about the organization of TIMSS or detect an annoyance with the
rigidity of the procedures, it may be because the situation is very different from School
Coordinators past classroom testing experiences. Explain that the TIMSS assessments are
administered to thousands of students in more than 60 countries. Therefore, to understand the
assessment results, it is important for the participants to use uniform procedures and document
how those procedures were implemented in their different countries.

4.2. Arriving at the School

It is very important that you arrive at the school up to one hour before the scheduled testing
session. A late arrival will more than likely be a distraction and an annoyance to the School
Coordinator and/or the Test Administrator as they prepare for the testing session. To guard
against a late arrival, we urge you to review a map of the area and obtain detailed directions from
the School Coordinator during your scheduled call.
If the School Coordinator and/or Test Administrator are confused about your role, convey the
following reasons for your visit:

You are there simply to observe a testing session, which requires you to review forms,
testing materials, and procedures.
You are looking forward to learning more about the testing procedures and hearing the
schools opinions on all matters related to TIMSS.

4.3. The Classroom Observation Record

During each of the school visits, you will document the information you are gathering in the
provided Classroom Observation Record. The Classroom Observation Record includes sections
on assessment administration, Student Questionnaire administration and, if applicable,
distribution of the Early Learning Survey, and a section on the structure of your interview with
the School Coordinator.
You will need the Student Tracking Form for the class you are observing and the completed
Class Listing Form for the school you are visiting in order to complete each Classroom
Observation Record.
You should familiarize yourself with the record before your first observation to allow enough
time to address any matters that are unclear. Contact the national center if you need any
assistance with the record.

TIMSS 2015 NQCM Manual

4.4. Observing the TIMSS Testing Sessions

When observing a testing session, you will need to do the following:
1. Bring the Student Tracking Form and Classroom Observation Record.
2. Leave enough time before the session begins to review the student assessment
instruments.
3. Document timing of the session (have an accurate watch with a seconds hand or a timer
with you).
4. During the testing session, devote full attention to completing the corresponding part of
the Classroom Observation Record. Keep notes about any aspects of the session that
should be documented. Being familiar with the record in advance is crucial.
5. After the testing session, check your copy of the Student Tracking Form, and update
absences against the Student Tracking Form used by the Test Administrator.

4.5. Roles and Responsibilities of the School Coordinator and the

Test Administrator
The specific duties of the School Coordinators are described in the School Coordinator Manual.
Similarly, the duties of the Test Administrators are described in the Test Administrator Manual,
including the administration scripts, the timing instructions, and the directions for managing the
testing sessions. It is very important to familiarize yourself with each of these manuals, as they
will provide detailed information you will need about the school responsibilities and activities.
The following is a short summary:
The School Coordinator provides the national center with information on selecting the classes
to be tested, helps to determine testing dates and times, ensures the correct distribution,
completion, and collection of the administrative forms and survey instruments, and returns the
materials to the national center. The School Coordinator is responsible for keeping all the
materials secure at all times.
The Test Administrator is responsible for conducting the testing session(s), which includes the
following tasks:

Distributing the achievement test booklets, Student Questionnaires, and Early Learning
Surveys (if applicable) to the correct students and documenting student participation;
Administering the assessment and Student Questionnaire in accordance with the
instructions in the Test Administrator Manual;
Ensuring the correct timing of the testing session; and
Maintaining the security of test instruments for the duration of the test administration
(from the time of receiving the materials from the School Coordinator until returning
them back to the School Coordinator).

TIMSS 2015 NQCM Manual

4.6. Interviewing the School Coordinator

As previously noted, part of the school visit involves interviewing the School Coordinator. The
interview should take place after the testing session, and should be no longer than 30 minutes.
The questions for this interview are in the last section of the Classroom Observation Record.
Refer to the Class Listing Form when you interview the School Coordinator. Ask the School
Coordinator if he or she is aware of other classrooms in that grade that are not on the list, any
students in the grade who are not in any of the listed classes, or any students in the grade who are
in more than one of the listed classes.
Again, be sure to conduct your visit with minimum disruption to the school. Make the School
Coordinator aware of your schedule, arrangements for your arrival, and plans for conducting the
interview in advance.

4.7. Returning Your Materials to the National Center

Upon completion of your school visits, you have accomplished your tasks as a National Quality
Control Monitor. Return all your NQCM materials to the national center as soon as possible. The
returned materials should include all your completed Classroom Observation Records, each with
its corresponding Class Listing Form, Student Tracking Form, and Teacher Tracking Form. The
national center will expect these materials within one week of your last observed testing session.

Appendix: Classroom Observation

Tracking Form

TIMSS 2015: Classroom Observation Tracking Form

Grade

Name and
Address of School

CLASS
ID

Date of
Observation

Completed
Classroom
Observation
Record?

Yes ___
No ___

Yes ___
No ___

Yes ___
No ___

Yes ___
No ___

Yes ___
No ___

Yes ___
No ___

Yes ___
No ___

Yes ___
No ___

Yes ___
No ___

10

Yes ___
No ___
Use additional pages if necessary

Signature:

Date: