Teaching and Teacher Education 65 (2017) 48e60

Contents lists available at ScienceDirect

Teaching and Teacher Education

journal homepage: www.elsevier.com/locate/tate

Subjectivity of teacher judgments: Exploring student characteristics

that influence teacher judgments of student ability
Kane Meissel*, Frauke Meyer, Esther S. Yao, Christine M. Rubie-Davies
University of Auckland, Faculty of Education and Social Work, Private Bag 92601, Symonds St, Auckland, New Zealand

h i g h l i g h t s

Explored alignment of standardized achievement results with teacher judgments.

Marginalized students received lower judgments after controlling for achievement.

Classroom and school achievement composition inversely related to teacher judgments.

Robust moderation of teacher judgments needed, both within and between schools.

Professional development may assist teachers to make fair and consistent judgments.

a r t i c l e i n f o a b s t r a c t

Article history: Teacher judgments of student achievement are increasingly used for high-stakes decision-making,
Received 22 June 2016 making it imperative that judgments be as fair and reliable as possible. Using a large national database
Received in revised form from New Zealand, we explored the relation between psychometrically designed standardized
16 January 2017
achievement results and teacher judgments in reading (N ¼ 4771 students) and writing (N ¼ 11,765
Accepted 25 February 2017
Available online 22 March 2017
students) using hierarchical linear modelling. Our findings indicated that judgments were systematically
lower for marginalized learners after controlling for standardized achievement differences. Additionally,
classroom and school achievement composition were inversely related to teacher judgments. These
Keywords:
Teacher judgments
discrepancies are concerning, with important implications for equitable educational opportunities.
Standardized testing © 2017 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND
Student achievement license (http://creativecommons.org/licenses/by-nc-nd/4.0/).
Social justice

1. Introduction higher quality learning opportunities than students judged less

The ability of teachers to accurately gauge student achievement
is considered an important aspect of teachers' professional
competence, as teacher judgments are often the primary source of
information about student achievement (Ready & Wright, 2011;
Südkamp, Kaiser, & Mo € ller, 2012; Südkamp, Kaiser, & Mo € ller,
2014). Teacher judgments are determinations made by teachers
about students' current achievement (see Section 2 for more
detail), and can impact teachers' ongoing instructional decision-
making within the classroom, including instructional pace, level
of support, and level of task difficulty (Alvidrez & Weinstein, 1999;
Clark & Peterson, 1986; Hoge & Coladarci, 1989). For example,
students judged to be more capable are more likely to receive
* Corresponding author. University of Auckland, Faculty of Education and Social
Work, Private Bag 92601, Symonds St, Auckland 1150, New Zealand.
E-mail address: k.meissel@auckland.ac.nz (K. Meissel).
able (Clark & Peterson, 1986; Rubie-Davies, 2014; Rubie-Davies,
Hattie, & Hamilton, 2006; Sharpley & Edgar, 1986). Furthermore,
teacher judgments have implications for placement decisions in
programs or ability groups, grade retention, and ultimately for
students' future academic pathways (Begeny, Eckert, Montarello, &
Storie, 2008; Begeny, Krouse, Brown, & Mann, 2011; Francis et al.,
2016; Harlen, 2005; Parsons & Hallam, 2014; Wiliam &
Bartholomew, 2004).
Internationally, much research has focused on teacher judgment
alignment, mainly investigating the relations between teacher
judgments and measured student performance. Reviews of this
body of research have shown broad agreement between judgments
and standardized assessments on average (r ¼ 0.63, Südkamp et al.,
2012), but the relations have been vastly inconsistent with a wide
range of correlations reported (0.03 to 0.92; Hoge & Coladarci,
1989; Südkamp et al., 2012). Südkamp et al. (2012) noted that
teacher judgments showed higher correlations with measured

http://dx.doi.org/10.1016/j.tate.2017.02.021
0742-051X/© 2017 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).
 K. Meissel et al. / Teaching and Teacher Education 65 (2017) 48e60
achievement when teachers were informed about what measure
their judgment was being compared with. Correlations were also
higher when judgments and measures addressed the same domain
or aspect of ability. Other test characteristics such as the number of
points on the judgment scale did not impact the degree of align-
ment between judgments and measured student achievement.
Within the New Zealand context, overall teacher judgments
(OTJs) were introduced as a specific achievement measure in 2010,
and are assessed in relation to expected curriculum standards in
reading, writing, and mathematics. These judgments are commonly
referred to as National Standards (NS), and are intended to reflect a
student's achievement in relation to the standard expected of
students at the same year level nationally. Teachers are asked to
consider a range of data, including observations of student learning,
learning conversations, and formal assessments such as standard-
ized achievement tests, to reach a decision on whether a student
meets the demands of the New Zealand curriculum (Ministry of
Education, 2011). Determination of whether a student meets the
standard is up to the teacher, with no mandate with respect to
which of these forms of evidence is utilized, nor the degree of
weighting of specific types of data. The judgment, however, should
focus solely on a student's achievement at that point in time and
should not include construct irrelevant information such as a stu-
dent's behavior or perceived potential ability.
A perfect correlation is unlikely and arguably undesirable since
the two measures should be used for different purposes e stan-
dardized assessments often focus on a specific aspect of a student's
learning whereas teacher judgments should take into account a
number of aspects of a student's achievement within a whole
subject area. Nonetheless, while previous research has investigated
the relation between judgments and standardized achievement,
the properties of teacher judgments and what informs these de-
cisions remain relatively unexplored. The question remains
whether lower correlations simply reflect differences in the nature
of the assessments, or whether there are construct irrelevant fac-
tors which influence teachers when they make judgments about
student performance. For example, although neither ethnicity nor
special needs status should affect a judgment about a student's
achievement, previous research has indicated that such factors
might indeed influence teachers' judgments (see e.g., Glock, Krolak-
Schwerdt, & Pit-ten Cate, 2015; Martínez, Stecher, & Borko, 2009;
Ready & Wright, 2011). Although discrepancies in teacher judg-
ments are to be expected given that there is random error in all
assessments of student performance, systematic differences
relating to specific subgroups would suggest a degree of bias.
Alignment between standardized tests and teacher judgments
may also be affected by the inherently different interpretive ap-
proaches (Hattie & Brown, 2003; Hattie et al., 2003). Standardized
tests are specifically designed to maximize reliability and consis-
tency across students, classrooms, schools, and regions. In contrast,
individual teachers typically make evaluations of student perfor-
mance in relation to local (class or school) level evidence. That is,
although teacher judgments may be defined as criterion-
referenced, judgments are likely to be influenced by normative
evaluations, such as how well each student is performing in rela-
tion to other students within the teacher's class (Angoff, 1974).
Despite considerable work investigating the properties of
teacher judgments, the majority of these studies have been con-
ducted in a North American context. Notable exceptions include the
earlier studies by Doherty and Conolly (1985) and Sharpley and
Edgar (1986), which were undertaken in Australia and the UK
respectively, as well as the more recent research undertaken in
Germany by Kaiser, Retelsdorf, Südkamp, and Mo € ller (2013).
The current study extends previous work investigating the na-
ture of teacher judgments in several ways. It most closely aligns
49
with the work of Ready and Wright (2011), but draws on a sample
of older students (approximately 9e13 years old) in both reading
and writing. Ready and Wright’s (2011) study focused on kinder-
garten students and research has shown that the alignment be-
tween teacher judgments and measured achievement can differ
across grade levels, highlighting the need for further study with
respect to older students (Südkamp et al., 2014). In addition,
teachers in Ready and Wright’s (2011) study had no access to
standardized assessment results of students, whereas teachers in
the current study had access to each students' standardized
achievement results, and were advised by the New Zealand Min-
istry of Education that this was a source of evidence that could be
drawn on when making judgments about students' achievement.
The meta-analysis by Südkamp et al. (2012) indicated that research
has yet to examine the way in which teacher judgments are affected
by knowledge of standardized assessment results prior to making a
holistic judgment about students' achievement within a learning
domain. Furthermore, relatively few studies have utilized data
collected as part of regular school routine. The current study uses
teacher judgments and standardized achievement results collected
in actual classroom contexts where data collection was not an
imposed measure for schools.
Previous research has frequently focused on relatively small,
localized samples of students; the average sample size of the 75
studies reported in a recent meta-analysis by Südkamp et al. (2012)
was 518 students. The current study drew on data from a large-
scale teacher professional development project with almost 5000
students represented in reading, and around 12,000 additional
students for writing. Since teacher judgments are inherently likely
to violate statistical assumptions of independence because one
teacher determines the judgments for all students in his/her class,
we employed three-level hierarchical linear modelling in the ana-
lyses with students nested within classrooms and nested within
schools. This enabled the inherent clustering of the data to be
accounted for.
Furthermore, the majority of teacher judgment studies have not
attended to between-group differences with respect to student
characteristics. The extent to which students' characteristics in-
fluence teachers' overall judgments of achievement remains largely
inconclusive. Due to the importance of equitable educational op-
portunities, this is a key focus of the current study.
2. Review of teacher judgment literature
The following sections provide a review of the extant literature
on teacher judgments e their alignment with standardized
achievement results and the impact of student characteristics and
school composition on these judgments. Due to the overlap of
teacher expectations and teacher judgments, the review begins
with a brief discussion of this issue.
2.1. Teacher expectations and teacher judgments
Teacher expectations and teacher judgments are similar in that
both represent subjective teacher estimates about student
achievement. They mainly differ in that expectations are typically
predictions about future achievement while judgments are a cur-
rent estimate of a student's performance. The latter are mostly
made in circumstances in which the teacher has taught the student
for some time and therefore can take into account a range of in-
formation. In contrast, teacher expectations focus on expected
improvement or performance over a future time period, and are
predictions of the possible academic progression of a student rather
than an assessment of their existing skills and knowledge (see for
example Rubie-Davies, Peterson, Sibley, & Rosenthal, 2015).
50 K. Meissel et al. / Teaching and Teacher Education 65 (2017) 48e60
Südkamp et al. (2014) point out that research considering group
differences based on student characteristics has largely focused on
teacher expectations rather than teacher judgments. Teacher
expectation research has concluded that teacher expectations
about student abilities are subject to bias related to the students'
ethnicity, socioeconomic status (SES), gender, and special needs,
and English for Speakers of Other Languages (ESOL) status
(McKown & Weinstein, 2008; Rubie-Davies et al., 2012; Südkamp
et al., 2012). However, authors in this area have noted that stu-
dents' actual achievement has in earlier studies often not been
controlled for, limiting the power of such studies (Jussim & Harber,
2005; Rubie-Davies et al., 2012).
Teacher expectation researchers also note that negatively biased
expectations are likely to be problematic for reasons beyond simple
prejudice. It is argued that when teachers underestimate students'
current achievement level, they are likely to plan lower level
learning opportunities for those students (Rubie-Davies et al.,
2006). In turn, this directly affects how much students learn,
because these differential learning opportunities accumulate over
time and ultimately reduce students' life chances (Rubie-Davies,
2014). Teacher judgments seem likely to be subject to similar bia-
ses, but since teacher judgments are more frequently formalized for
high stakes decisions such as students' placements in ability groups
or admission to a particular educational track, the ramifications of
any bias in these judgments are particularly serious.
2.2. Teacher judgments and the influence of student characteristics
Research focusing on teacher judgments has tended to focus
more strongly on the overall degree of correspondence between
teacher judgments and measured student achievement, and less on
differences in these judgments for specific student groups. Teacher
judgment research considering the influence of student charac-
teristics on those judgments remains largely inconclusive. Previous
reviews and meta-analyses indicate difficulties with the aggrega-
tion of results as information on student characteristics has been
scarce or reported in aggregated ways (e.g., Hoge & Coladarci, 1989;
Ready & Wright, 2011; Südkamp et al., 2014). Furthermore, studies
considering student characteristics have often used teacher judg-
ments to validate rating scales and measures for certain domains
and groups of students (e.g., Lembke, Foegen, Whittaker, &
Hampton, 2008; Li, Pfeiffer, Petscher, Kumtepe, & Mo, 2008;
Methe, Hintze, & Floyd, 2008), rather than to examine discrep-
ancies in teacher judgments for specific learner groups. However,
several student characteristics warrant examination in terms of
whether these characteristics inform or influence teachers' judg-
ments, including gender, ethnicity, socioeconomic background, and
students' special needs or ESOL status.
Martínez et al. (2009) found that the gap between minority and
non-minority kindergarten students' academic abilities was judged
to be smaller by teachers than suggested by standardized test re-
sults in mathematics. The authors concluded that teachers may
have been compensating for perceived inequities. In contrast,
Ready and Wright (2011) concluded that teacher judgments for
kindergarten students from low SES or minority backgrounds were
negatively biased in the domain of literacy, whereas Feinberg and
Shapiro (2003) found no effect of student ethnicity on teacher
judgments.
Previous research has generally indicated that student gender
has no significant effect on teachers' judgments (Doherty & Conolly,
1985; Hecht & Greenfield, 2002; Helwig, Anderson, & Tindal, 2001;
Hoge & Butcher, 1984; Hoge & Coladarci, 1989; Sharpley & Edgar,
1986). However, Ready and Wright (2011) found a negative bias
in teacher judgments for boys in the domain of literacy, while
Martínez et al. (2009) showed that teachers marked girls higher on
a criterion-referenced scale in mathematics than suggested by
standardized achievement test results. Other studies have found
that teachers were indirectly influenced by student gender when
judging academic skills (Bennett, Gottesman, Rock, & Cerullo, 1993;
Beswick, Willms, & Sloat, 2005). Foremost, perceptions of students'
behavior affected teachers' judgments in these North American
studies; boys were often rated lower in academic or literacy skills
because their behavior was perceived to be more problematic than
that of girls. A few studies have indicated that behavioral factors
such as student engagement and motivation can influence teacher
judgments (Benner & Mistry, 2007; Dompnier, Pansu, & Bressoux,
2006; Kaiser et al., 2013).
Few studies have examined the effect that English language
learner or special needs status has on teachers' judgments, but
research thus far has indicated negative relations. Hurwitz, Elliott,
and Braden (2007) argued that teachers consistently under-
estimated the performance of students with special needs status,
whereas Martínez et al. (2009) found that teachers marked stu-
dents with English language learner or special needs status lower in
mathematics than standardized achievement test results
suggested.
2.3. Methodological differences in studies of teacher judgments
As described in the previous section, studies of teacher judg-
ments have typically not considered group differences based on
student characteristics. Where group differences have been
considered, objective measures of student achievement have
seldom been employed as a means of assessing teacher judgment
discrepancies for specific subgroups, thereby neglecting to control
for differences in student achievement between these groups. In
general, these studies have followed a design where teachers have
judged students' current ability on a scale provided by the re-
searchers (e.g., Kaiser et al., 2013; Ready & Wright, 2011).
In addition, data have seldom been collected within the usual
classroom context, comparing between group differences in
teacher judgments of their own students. This is important because
when teachers make judgments of students in experimental
studies, the descriptions they read are not of their own students.
Therefore, it is not possible to determine whether the responses of
teachers in experimental studies would be the same as in natu-
ralistic studies where they know their students well and interact
with them daily (Rubie-Davies, 2014).
The studies by Kaiser et al. (2013) and Ready and Wright (2011)
provide the most comprehensive naturalistic studies to date in
actual classroom settings. In total, we could only identify three
studies that had used naturalistic data of teacher judgments made
in the classroom context while also examining standardized
achievement results (Kaiser et al., 2013; Martínez et al., 2009;
Ready & Wright, 2011). Kaiser et al. (2013) reported on three
studies examining the reciprocal relationship between students'
reading achievement, engagement, and teacher judgments. The
first study drew on data from 52 teachers and 1135 students taking
part in a German literacy development project, whereas the other
two studies utilized simulated classrooms to gather experimental
data. The first study, which is of most relevance to the current study
since it collected data in the field, found a moderate correlation of
r ¼ 0.49 between teacher judgments and student achievement. The
authors pointed out that the low level of alignment could stem
from teachers not being informed about the achievement measure
with which their judgment was compared. The study used a
reading test from the German PIRLS study and teacher responses to
two four-point scales in regard to students' ability in literacy. Thus,
although teachers knew the students they were rating, their
judgments were made in regard to two specific items of
 K. Meissel et al. / Teaching and Teacher Education 65 (2017) 48e60
achievement without the use of recent standardized achievement
data, and were uninformed in relation to the comparison measure.
Besides the low correlation between teacher judgments and actual
achievement, the authors reported an effect of student engagement
on teacher judgments of student achievement and vice versa. The
study did not examine differences in judgments in regard to stu-
dent characteristics.
The studies by Martínez et al. (2009) and Ready and Wright
(2011) both used data from the Early Childhood Longitudinal Sur-
vey, which followed 22,000 children in the United States from
kindergarten through to fifth grade. Martínez et al. (2009) analysed
teacher judgments and math achievement whereas Ready and
Wright (2011) focused on the literacy domain. Their analytic sam-
ples included around 10,000 students each. Martínez et al. (2009)
concluded that teachers perceived smaller achievement gaps than
indicated by the standardized assessments for female, minority and
students from low SES backgrounds. They argued that this could
reflect a bias within the tests, or a deliberate effort by teachers to
compensate for disadvantages faced by these student groups.
However, achievement was not specifically controlled for. In direct
contrast, Ready and Wright (2011) controlled for measured student
achievement and concluded that a negative bias appeared to be
evident. Although roughly half of the group differences were
accounted for by between group differences indicated by the
standardized assessments, the remaining differences indicated
systematic biases. Teachers in the study had overestimated girls'
performance and underestimated Black, Asian, and Hispanic stu-
dents, and especially students from a low socioeconomic back-
ground (over half a standard deviation) in their judgments. The
authors indicated concerns that their results suggested a system-
atic bias among teachers about already marginalized learners,
potentially exacerbating existing disparities.
2.4. Teacher judgments, classroom, and school composition effects
Teacher judgments are made within the context of individual
classrooms nested within schools. Martínez et al. (2009) and Ready
and Wright (2011) took this nestedness into consideration using
hierarchical linear modelling and both found a high variation be-
tween classrooms and less variation between schools. Whereas
Martínez et al. (2009) examined overall differences between
classrooms with respect to teachers' practice, Ready and Wright
(2011) employed different classroom and school level variables in
regard to teacher and student characteristics. At the classroom
level, variables included classroom composition (e.g., classroom
average SES and literacy ability, and high minority proportion) and
teacher characteristics (e.g., teacher ethnicity, educational attain-
ment, and experience). Ready and Wright (2011) concluded that
classroom composition had a stronger effect on teacher judgments
than teacher characteristics. Teachers in higher-achieving and
higher-SES classrooms overestimated students' literacy abilities,
even after controlling for child-level SES and measured achieve-
ment, and underestimated students' skills in lower achieving and
socioeconomically disadvantaged classrooms. In contrast, teacher
characteristics were unrelated to the degree of alignment between
teacher judgments and measured achievement once the authors
controlled for student characteristics, except for beginning teachers
who tended to overestimate students' ability. School level mea-
sures included school average SES, school location (e.g. urban, ru-
ral), school sector (e.g., private, public), and school size. Ready and
Wright (2011) found that teachers in urban schools tended to un-
derestimate students' ability, whereas teachers in small schools
tended to overestimate students' literacy skills.
51
In conclusion, it has been argued that teacher judgments allow
for a more holistic understanding of student achievement, since
teachers' daily interactions with students provide them with a
richer picture of what students know and can do than tests alone
(Chamberlain, 2010; Südkamp et al., 2014). However, previous
research has indicated that teacher judgments might be biased on
the basis of student characteristics and influenced by classroom
and school contexts. Little research has examined teacher judg-
ments in naturalistic settings (see e.g., Hopkins, George, &
Williams, 1985; Meisels, Bickel, Nicholson, Xue, & Atkins-Burnett,
2001), and even less research has considered specific student
group characteristics, classroom, and school level factors within a
naturalistic setting (see Ready & Wright, 2011). Systematic bias
such as that signaled by Ready and Wright (2011), could serve to
exacerbate and perpetuate achievement gaps, in opposition to the
notion of equality of opportunities (de Boer, Bosker, & van der Werf,
2010). However, Ready and Wright’s (2011) study focused on
kindergarten students and research has shown that the alignment
between teacher judgments and measured achievement can differ
across grade levels, highlighting the need for further study with
respect to older students (Südkamp et al., 2014).
3. The current study
The following section provides background detail about the New
Zealand context and the project from which the data were ob-
tained, the use of teacher judgments within this context, as well as
a description of the aims and purpose of this study.
3.1. The New Zealand context
Students in New Zealand attend primary school from Years 1e8
(~5e13 years old), while secondary schools cater for students in
Years 9e13. New Zealand schools and teachers are highly autono-
mous relative to educational systems globally (Hanushek, Link, &
Woessmann, 2013; Wylie, Cosslett, & Burgon, 2016). Since 1989,
individual schools have been independent, self-managing admin-
istrative units. Each school is managed by a board of trustees made
up predominantly by parents, a school staff member, a current
student (at the secondary school level), as well as the principal.
Boards are elected every three years, and are responsible for the
overall running and performance of the school. Staffing numbers
(including teacher aides and special education staffing) are deter-
mined nationally, with funding allocated by the Ministry of Edu-
cation, but each board is individually responsible for hiring staff,
allocating operational funding and ensuring the budgets are met
(Wylie et al., 2016). Schools use a national curriculum which pro-
vides a common framework of learning areas, values, and key
competencies but schools have the flexibility to design and adapt
the curriculum to their particular school community.
New Zealand's population is approximately 4.5 million, with
85% of the population residing in urban areas. The school-age
population comprises 52% New Zealand (NZ) European, 24%
Ma ori, 10% Pasifika, and 11% Asian students (Statistics New Zealand,
2016). M aori are the indigenous group; Pasifika are those with
Pacific Island ancestry (e.g., Samoa, Tonga, Cook Islands); and Asian
students are those from North Asia to the Indian sub-continent.
New Zealand's educational system has been described as a high
performance, but low equity system. While the highest achieving
students excel, and mean performance is comparatively high,
achievement data typically show large disparities, with particular
groups increasingly over-represented in the lowest quartile of the
distribution (OECD, 2005, 2013; Ogle et al., 2003). Within New
52 K. Meissel et al. / Teaching and Teacher Education 65 (2017) 48e60
Zealand, the largest disparities exist for Ma ori and Pasifika stu-
dents, and those attending schools with low socio-economic
catchment areas.
The introduction of overall teacher judgments (OTJs) in 2010
was, ostensibly, intended as a way to measure and hopefully reduce
these disparities. Performance is judged ‘holistically’ based on the
evidence deemed appropriate for each student by the student's
teacher (Ministry of Education, 2010). There has been considerable
commentary for and against the policy, with advocates arguing that
holistic judgments retain breadth in the curriculum, while oppo-
nents raise concerns about the potential for bias and unreliability
(Courtney, 2010; Eames, 2010; Ministry of Education, 2011, 2010;
€
OZerk & Whitehead, 2012; Smith, Anderson, & Blanch, 2016;
Thrupp, 2013). These judgments have important implications for
students' schooling. While students in New Zealand are not held
back, within class ability grouping is an entrenched practice,
despite research suggesting it contributes to a perpetuation of
disparities (Schmidt, Burroughs, Zoido, & Houang, 2015; Wilson,
Madjar, & McNaughton, 2016). Since ability groups are deter-
mined by teachers' judgments, reliable and fair determinations are
essential. In addition, aggregated National Standards (NS) data are
published on the Ministry of Education website (Ministry of
Education, 2016) and media agencies provide NS school league
tables (Fairfax New Zealand Limited, 2012, 2016). Despite the
importance of these judgments, minimal empirical research has
been conducted to assess the properties of the measure to date.
3.2. The professional learning project
This study draws on data collected in a large national profes-
sional learning and development project, called the Consortium for
Professional Learning (CPL; for complete details see cpl.org.nz). The
project was funded by the New Zealand Ministry of Education, and
aims to ensure that the professional learning produced measurable
gains in student achievement, with a particular focus on improving
equity. Improvement was evaluated in terms of gains in both
standardized achievement and OTJ results, but there was no focus
on ensuring alignment of standardized results with OTJs. Since New
Zealand schools are self-governing, participation within the project
was voluntary, meaning that schools chose whether to “opt-in”.
Due to the project's improvement focus, the majority of schools
choosing to take part did so out of a self-perceived need to improve
student achievement results. This self-selection resulted in an over-
representation of schools with a low SES profile, as well as an over-
representation of Pasifika students relative to the national student
profile. Despite this over-representation, a complete range of
school types participated within the project.
3.3. Purpose of this study
The current study examined the relations between teacher
judgments and psychometrically designed standardized achieve-
ment test results in reading and writing. As previously noted, these
measures are extremely unlikely to align perfectly, as both have
inherent error, while teacher judgments are intended to evaluate a
student's achievement more holistically than a single test (Ministry
of Education, 2012). Determining whether there are systematic
discrepancies between teacher judgments and standardized
achievement tests is, however, important since any discrepancies
would suggest that evidence unrelated to actual achievement dif-
ferences was playing a role in teachers' judgments. As New Zealand
teachers are required to report OTJs for each student in Years 1
through 8 (aged approximately 5e13 years old) for reading, writing,
and mathematics at the end of each year (see Section 4.2.1 for full
details), the educational system provides a naturalistic context for
studying teacher judgments and their alignment with standardized
achievement measures. New Zealand schools also typically use
standardized assessments, and teachers are advised that these as-
sessments can be used to inform teacher judgments, but there is no
requirement to do so.
The overarching research question was whether there were
systematic differences in the judgments that teachers made about
the attainment of students belonging to priority learner groups,
which were not explained by differences indicated by standardized
achievement. Priority learner groups in New Zealand are defined as
those traditionally marginalized/underserved within the New
Zealand education system (i.e., males, Maori and Pasifika, students
with special needs, and students for whom English is their second
language (ESOL)). In addition, of interest was whether there were
contextual factors that mitigated or exacerbated any systematic
discrepancies detected within teacher judgments. Thus the relation
between teacher judgments and student characteristics was
examined by taking into account differences in standardized
achievement results.
The specific research questions examined in this study were as
follows:
1. What is the correlation between standardized achievement re-
sults and teacher judgments of student achievement?
2. After accounting for standardized achievement differences, are
there residual differences in teacher judgments of student
achievement that vary systematically by student characteristics
(i.e., gender, ethnicity, ESOL status, and special needs status)?
3. After accounting for standardized achievement differences, are
there residual differences in teacher judgments of student
achievement that vary systematically by contextual factors (i.e.,
classroom and school achievement composition, ethnic
composition, school socioeconomic profile, school size, and
region)?
4. Method
4.1. Participants
This study utilized reading and writing achievement data
collected at the end of the 2012 and 2013 academic years as part of
an ongoing large-scale professional development project operating
across New Zealand. We selected all students from Years 4e8
(approximately 8e13 years old; Grades 3 to 7) who had an end of
year standardized achievement score from the same subject
domain, in addition to their OTJ. The OTJs (see Section 4.2.1 for full
description) determined at the end of the year represented teacher
judgments after having worked with each student for a complete
year. Data for students in Years 1e3 (aged approximately 5e8; K to
G2) were unable to be included since most schools did not provide
standardized assessment data for younger students. The data
specified which classroom students were in, but did not provide
teacher-level information. The final sample for reading comprised
of 4771 students nested within 194 classrooms attending 44
schools, and for writing, 11,765 students nested within 561 class-
rooms across 105 schools. Note that while the majority of schools
provided data for only one domain, around 20% of schools (n ¼ 26)
chose to provide data for both reading and writing, introducing
some overlap across the two samples. Hence, the data are not
strictly independent. Participating schools tended to be in lower
socioeconomic catchment areas and have an overrepresentation of
Pasifika students compared to the national student population.
Table 1 presents the demographic information for the reading and
writing samples in more detail.
Within both subject domains, more than half the schools
 K. Meissel et al. / Teaching and Teacher Education 65 (2017) 48e60 53

Table 1
Student-level demographic characteristics by subject domain.

Characteristic Reading Writing New Zealand schools

(N ¼ 4771) (N ¼ 11,765)

n % n % %

Gender Male 2386 50 5995 51 51

Female 2385 50 5770 49 49

Ethnicity NZ European 999 21 3602 31 52

NZ Maori 1327 28 3107 26 24
Pasifika 1730 36 3431 29 10
Other 715 15 1625 14 14

Year level 4 685 14 2483 21 21

5 747 16 2604 22 20
6 694 15 2365 20 19
7 1496 31 2148 18 22
8 1149 24 2165 18 19

ESOL 327 7 976 8 e

Special needs 147 3 776 7 e

Decile group Low 3673 77 7062 60 24

Mid 764 16 3136 27 38
High 334 7 1567 13 38

(reading: n ¼ 28; writing: n ¼ 62) were situated in low SES areas
(decile1 rating 1e3), around a third in mid SES areas (decile rating
4e7; reading: n ¼ 12; writing: n ¼ 31), and comparatively few in
more affluent zones (decile rating 8e10; reading: n ¼ 4; writing:
n ¼ 12). Geographically, participating schools were mainly located
in the North Island of New Zealand (reading: n ¼ 42; writing:
n ¼ 92). A much smaller number of schools were from the South
Island (reading: n ¼ 2; writing: n ¼ 13). The majority of the pop-
ulation lives in the North Island (~75%), and is comparatively less
affluent than the South Island; 91% of low SES schools are in the
North Island. The average school roll was around 250 students,
although this varied considerably (reading: M ¼ 247.12,
SD ¼ 239.55; writing: M ¼ 244.64, SD ¼ 203.11). On average, just
over half the students in each school were of either NZ M aori or
Pasifika descent (reading: 60%; writing: 56%), although these fig-
ures are higher in the northern region. Because the sample is drawn
from a professional development project into which schools opt
based on their own assessment of need, the sample is more heavily
weighted toward schools with lower achievement and SES profiles.
However, the diversity of the sample means that a complete range
of students and schools remains represented.
4.2. Measures
4.2.1. Dependent variable
OTJ was entered as the dependent variable. OTJs for reading and
writing are each made on a four-point curriculum-referenced scale.
The scale is intended to reflect a student's achievement in relation
to the standard expected of their year level nationally: well below
standard (coded as 1), below standard (coded as 2), at standard
(coded as 3), and above standard (coded as 4). The standards are
aligned to the New Zealand curriculum. Teachers are expected to
determine the appropriate OTJ for each student using their own
professional judgment of best fit, but are provided with suggestions
of possible sources of achievement evidence; for example,
1
A school decile indicates the extent to which a school intakes students from low
socioeconomic communities. Decile 1 schools are the 10% of schools with the
highest proportion e and decile 10 schools the 10% with the lowest proportion e of
students from low socio-economic communities. Decile is used primarily for
funding purposes; schools with poorer students receive more funding than those
with students from more affluent backgrounds.
observations of student learning, conversations with students,
classroom tests, and standardized achievement results (Ministry of
Education, 2011). Thus, OTJs are complex judgments which argu-
ably encapsulate a broader range of student ability than what a
single achievement measure can indicate. OTJs are a Ministry of
Education requirement for all schools working with students in
Years 1e8 (5e13 year olds), and are made at the end of each year of
schooling. Unfortunately, no research has yet been conducted to
investigate the reliability or validity of OTJs, so these metrics are not
available.
4.2.2. Standardized achievement scores
Most schools in the teacher professional development project
conducted a standardized achievement test towards the end of the
school year, close to when the OTJs were made. In most cases,
teachers had access to the results of these tests prior to determining
the appropriate OTJ for each student. Schools chose to use either
the Assessment Tools for Teaching and Learning (e-asTTle) or the
Progressive Achievement Tests (PATs). Both of these tests were
constructed specifically for the New Zealand educational context
using item-response theory (Brown, 2013; Darr, McDowall, Ferral,
Twist, & Watson, 2008; Darr, Neill, Stephanou, & Ferral, 2006).
The use of item-response theory ensures that achievement is
measured on a common scale irrespective of the specific items used
within each test. However, while this ensures equivalence across
different forms of the same test, it does not necessarily follow that
PAT and e-asTTle are equivalent. Therefore, the alignment between
OTJs and each test was assessed separately to determine whether
the relation differed. These tests are both curriculum-referenced
(allowing comparison against the curriculum expectations for
students at each year level) and norm-referenced (allowing com-
parison against typical achievement nationally). The reliability of e-
asTTle was reported to be a ¼ 0.96 (Ministry of Education & NZCER,
2012), and the reliability of PAT Reading Comprehension is a ¼ 0.90
(Darr et al., 2006, 2008).
Each standardized test was scored on its own equal-interval
common scale. Because older students should typically be ex-
pected to achieve higher scores, the end-of-school-year norm for
students' respective year levels was subtracted from their actual
score to eliminate confounding maturational effects. To standardize
the e-asTTle and PAT scores for reading and writing, students' test
scores relative to the norm were divided by the sample standard
54 K. Meissel et al. / Teaching and Teacher Education 65 (2017) 48e60
deviation of the corresponding test. This placed all achievement
scores on the same scale regardless of the test administered.
The absolute values of skewness (reading: 0.25, SE ¼ 0.04;
writing: 1.06, SE ¼ 0.02) and kurtosis (reading: 0.86, SE ¼ 0.07;
writing: 3.39, SE ¼ 0.05) for the standardized scores were below 2
and 7, respectively, and thus were well within the typically
acceptable range for normality (Kim, 2013; Kline, 2005). However,
as the Kolmogorov-Smirnov test showed statistically significant
deviation from normality, Markov chain Monte Carlo (MCMC)
estimation was used for the Hierarchical Linear Models (HLMs) to
mitigate this departure (Gill, 2002). The negatively skewed distri-
bution is likely a reflection of the over-representation of low decile
schools and minority ethnic groups in the sample, as these groups
typically have lower achievement (Hattie, 2008).
4.2.3. Student characteristics
Student-level characteristics were dummy-coded and incorpo-
rated in the HLMs as Level 1 predictors. Ethnicity was coded with
New Zealand European as the reference group; and Ma ori (indig-
enous New Zealanders), Pasifika (Pacific Islanders), and “other”
ethnicities (e.g., Asian, Middle Eastern, Latin American, and African)
as comparison groups. Other demographic variables were coded as
binary dummy variables, including students' gender (male ¼ 0,
female ¼ 1), ESOL status (no ¼ 0, yes ¼ 1), and special needs status
(no ¼ 0, yes ¼ 1).
4.2.4. Classroom characteristics
Classroom achievement composition, measured using the
average standardized achievement of the students in each class-
room, was examined as a Level 2 variable. As student-level stan-
dardized achievement scores had already been standardized, there
was no need to center the aggregate measure. We did not have
access to other classroom-level data such as teacher gender or
ethnicity.
4.2.5. School characteristics
Contextual effects were also examined by including school-level
characteristics at Level 3 in the HLMs. These characteristics
included: the school's decile band (low [1e3], mid [4e7] or high
[8e10] decile); school region (central south, southern, or north-
ern); school size (the number of students in the school roll); the
percentage of minority students attending each school (‘minority’
was defined as students who identified as M aori and/or Pasifika;
students of ‘other’ ethnicities were not included as there is typically
no achievement gap between this group and New Zealand Euro-
peans; Satherley, 2006); and school achievement composition
(average standardized achievement of the school). Decile band and
school region were entered into the models as polytomous dummy-
coded variables in the same way as explained for ethnicity, with
low decile and the Northern region as the reference groups. These
groups were selected as the reference since both represented the
largest proportion of students. School size and minority percentage
were grand-mean centered to improve interpretability, but
retained the same scale (Kreft, de Leeuw, & Aiken, 1995). School
achievement composition was not centered as student-level stan-
dardized achievement scores had already been standardized.
4.3. Analytic approach
We utilized three-level HLMs (Raudenbush & Bryk, 2002;
Woltman, Feldstain, MacKay, & Rocchi, 2012), with students nes-
ted in classrooms nested within schools. These models account for
clustering in the data and were necessary since OTJs are made by
classroom teachers and thus may vary in interpretation across
classrooms (Ready & Wright, 2011). Hierarchical linear modelling is
an extension of conventional regression, and therefore estimates
the degree to which predictor variables relate to differential out-
comes. As with regression, these models establish relations be-
tween variables and cannot explain causality. The HLMs were
estimated using MLwiN 2.26 with MCMC estimation, since MCMC
tends to outperform likelihood methods (e.g., maximum likeli-
hood) when data are non-normal (Gill, 2002).
We conducted separate HLMs for each time point for reading
and writing (i.e., Reading 2012, 2013; Writing 2012; and Writing
2013). Each model was built in the same way. First, the uncondi-
tional model was specified as:
Yijk ¼ g000 þ u00k þ r0jk þ eijk (1)
where Y is the OTJ for student i in classroom j of school k, g000 is the
grand mean, u00k is the variance at the school level, r0jk the variance
at the classroom level, and eijk the variance at the student level. The
unconditional model allows variance partitioning, which provides
an estimate of the degree of variance at each of the levels within the
model. If the unconditional model shows non-significant variance
at the higher levels within the specified hierarchy, conventional
regression may be sufficient, though some authors argue the hi-
erarchy should be specified even for low levels of clustering (e.g.,
Dorman, 2008).
Next, standardized achievement scores were added as a Level 1
predictor to account for student achievement as measured by a
standardized test. Thereafter, the models were built iteratively;
student-, classroom-, and school-level predictors were examined
separately, then included in the full model, in order to investigate
the variables which explained a significant amount of the variance
in OTJs after standardized achievement was accounted for. Factors
can be entered allowing only the intercept term to vary at each
level, or the individual slopes. It made theoretical sense to allow the
relation between standardized achievement and OTJs to vary across
classrooms so this was investigated. Interaction effects were also
explored to determine whether discrepancies were compounded
for students who were members of more than one priority learner
group, such as Pasifika who were also English language learners.
Parameter estimates are reported in terms of standard deviation
units (SDU). Note that interpretation of SDU differences is similar to
Cohen's d effect size (1988), in that both provide an indicator of
difference in terms of standard deviations. However, Cohen's d is
typically calculated as a single-level bivariate comparison, so the
magnitude of SDU differences in a multilevel framework tend to be
smaller and need to be interpreted with an awareness of what has
been taken into account.
5. Results
In our preliminary analyses for reading and writing, we built
separate models for each collection phase (2012 or 2013) and
standardized tool (e-asTTle or PAT). There were no significant dif-
ferences in the parameter estimates across the models, so the data
were aggregated into a single reading dataset and single writing
dataset, then re-analysed using the same approach to simplify
interpretation. The results of these models are presented below.
5.1. Variance partitioning
In the unconditional models (see Table 2), the majority of the
variability in OTJs was at the student level for both reading (75%)
and writing (78%). This is unsurprising given that students' aca-
demic ability had not yet been accounted for. The remainder of the
variance was partitioned fairly evenly between the classroom and
school levels (12% and 10%, respectively, for reading; and 13% and
 K. Meissel et al. / Teaching and Teacher Education 65 (2017) 48e60 55

Table 2
Variance decomposition from the unconditional models for reading and writing.

Level Reading Writing

Absolute variance % variance Absolute variance % variance

School (level 3), u00k 0.11*** 12% 0.09*** 11%

Classroom (level 2), r0jk 0.12*** 13% 0.10*** 12%
Student (level 1), eijk 0.70*** 75% 0.66*** 78%
Deviance (2*loglikelihood) 11822.34 28506.66

***p < 0.001.

12%, respectively, for writing). The moderate and significant clus- standard’ (coded 2) and ‘at standard’ (coded 3; M ¼ 2.69 in reading;
tering at both the classroom and school levels indicated the ne- M ¼ 2.57 in writing), but were lower for writing than for reading,
cessity of a three-level HLM for both domains. suggesting a degree of discrepancy between the standardized
measures and the OTJs. That is, teachers considered reading
5.2. Descriptive results achievement to be closer to the standard than writing achievement,
while the standardized test results suggested the opposite.
Descriptive statistics for standardized achievement scores and The correlation between the two achievement measures was
OTJs are shown in Table 3. As students' test scores have had the slightly greater than 0.70 for both reading and writing, both overall
appropriate year level norm subtracted, with the result divided by at the student-level, and as an average across schools. However,
the standard deviation of the achievement test, the standardized there was considerable variation across schools; school-level cor-
achievement scores reflect students' achievement relative to the relations ranged between 0.50 and 0.94 for reading, and 0.07
national norm in terms of SDU. and 0.94 for writing. For both reading and writing, the correlation
The average standardized achievement scores for reading and was negative in two schools. Negative correlations are surprising
writing were both below the national norm, with the reading since the two measures should fundamentally assess the same
sample further from the national norm (0.42 SDU; equivalent to domain; regardless of the fact the measures assess different aspects
approximately one academic year) than the writing sample (0.19 of achievement.
SDU). The average OTJs for both domains were between ‘below
5.3. Standardized achievement
Table 3
Descriptive statistics and correlation of standardized achievement and OTJ by Standardized achievement scores were added to the uncondi-
domain. tional model as a Level 1 predictor. As shown in the ‘Standardized
Achievement’ columns in Table 4, there was a positive and statis-
Reading (SD) Writing (SD)
tically significant relationship between standardized achievement
Standardized achievementa 0.42 (1.04) 0.19 (0.99) and OTJ, such that on average an increase of one standard deviation
OTJ 2.69 (0.97) 2.57 (0.92)
in standardized achievement score was associated with a 0.61 in-
Overall correlation (rs) 0.73*** 0.72*** crease in OTJ for reading, and 0.67 increase in OTJ for writing. The
Average school-level correlation (rs) 0.68*** (0.24) 0.72*** (0.16)
inclusion of individual standardized achievement explained half of
Range of school-level correlations (rs) 0.5 to 0.94 0.07 to 0.94
the student-level variance in OTJs in writing, and 42% of the
***p < 0.001.
a
student-level variance in reading. It also reduced much of the
Shows achievement scores relative to the norm in standard deviation units.
variance at the school level for writing (43%) and at the classroom
level for reading (59%). Of the residual variance, the proportion at

Table 4
Three-level HLM estimates for the standardized achievement, student characteristics, and final models for reading and writing.

Parameter Standardized achievement Student characteristics Final

Reading Writing Reading Writing Reading Writing

Fixed effects
Intercept 3.01(0.05)*** 2.74(0.03)*** 3.08(0.06)*** 2.75(0.03)*** 3.02(0.06)*** 2.70(0.03)***
Standardized achievement 0.61(0.01)*** 0.67(0.01)*** 0.58(0.01)*** 0.64(0.01)*** 0.58(0.01)*** 0.65(0.01)***
Female 0.05(0.02)** 0.10(0.01)*** 0.06(0.02)** 0.10(0.01)***
Maori 0.10(0.03)** 0.08(0.02)*** 0.10(0.03)** 0.09(0.02)***
Pasifika 0.15(0.03)*** 0.10(0.02)*** 0.16(0.03)*** 0.10(0.02)***
Other 0.10(0.04)** 0.01(0.02) 0.11(0.04)** 0.01(0.02)
ESOL 0.14(0.04)*** 0.12(0.02)*** 0.13(0.04)** 0.12(0.02)***
Special needs 0.55(0.06)*** 0.24(0.03)*** 0.53(0.06)*** 0.23(0.03)***
Classroom average achievement 0.09(0.05) 0.16(0.03)***
School average achievement 0.28(0.11)** 0.14(0.06)*
Random effects
School 0.09 [17%] 0.05 [11%] 0.09 [17%] 0.04 [9%] 0.08 [15%] 0.05 [11%]
Classroom 0.05 [9%] 0.08 [17%] 0.05 [9%] 0.07 [16%] 0.05 [10%] 0.07 [16%]
Student 0.40 [74%] 0.33 [72%] 0.39 [74%] 0.32 [74%] 0.39 [75%] 0.32 [73%]
Model summary
Deviance (2*loglikelihood) 9210.29 20259.46 9095.39 20034.54 9099.81 20014.00
Number of estimated parameters 5 5 11 11 13 13

Note. (Standard errors); [% residual variance]. *p < 0.05. **p < 0.01. ***p < 0.001.
56 K. Meissel et al. / Teaching and Teacher Education 65 (2017) 48e60
each level remained relatively stable, with 72e75% of the variance
at the student level in each model. Allowing the standardized
achievement slope to vary did not significantly improve model fit
suggesting a fairly consistent relation between standardized
achievement and OTJs across classrooms (p > 0.05 for reading and
writing); thus, this slope was estimated as a fixed effect in all
subsequent analyses.
5.4. Student characteristics
At the student level, we explored whether there were system-
atic differences in the OTJs assigned to priority learners, after
controlling for differences in standardized achievement scores. As
shown in the ‘Student Characteristics’ columns of Table 4, these
student characteristics were all significant predictors of OTJs even
after differences in standardized achievement were accounted for.
To quantify the magnitude of the under or overestimate relative to
standardized achievement, effect sizes in SDU based on the HLM
parameter estimates are provided in parentheses. Specifically, for
both reading and writing, even when the standardized achieve-
ment evidence was the same, females typically received signifi-
cantly higher OTJs than males (SDU ¼ 0.06 for reading; SDU ¼ 0.11
for writing), Maori (SDU ¼ 0.10 for reading; SDU ¼ 0.09 for
writing) and Pasifika (SDU ¼ 0.16 for reading; SDU ¼ 0.10 for
writing) students received significantly lower OTJs than NZ Euro-
pean students, and ESOL students (SDU ¼ 0.14 for reading;
SDU ¼ 0.13 for writing) and those with special needs
(SDU ¼ 0.57 for reading; SDU ¼ 0.26 for writing) received
markedly lower OTJs than those not in these categories. The degree
of discrepancy was typically similar for reading and writing, except
for students with special needs, whereby the discrepancy was
much larger in reading than for writing.
Many students belong to more than one of these learner groups,
so we also explored interaction effects between these groups. For
example, whereas OTJs were typically lower for boys and M aori
than suggested by the standardized achievement results, we
wanted to determine whether being a M aori boy led to an even
greater discrepancy. We found no significant interaction between
gender and ethnicity for either domain. Interactions including ESOL
or special needs status could not be explored because compara-
tively few students were identified as being in these learner groups
when delineated by gender or ethnicity.
5.5. Achievement composition
Classroom and school achievement composition were added
into the model containing standardized achievement and student
characteristics to test whether mean achievement in the classroom
and school explained additional variance in OTJs. As shown in the
two rightmost columns of Table 4, school achievement composition
had a significant, inverse relationship with OTJs; that is, after
controlling for individual standardized achievement and student
characteristics, when the average achievement of a school was
comparatively high, the OTJs made in that school were typically
lower for a student with the same standardized achievement as a
student in a school with lower average achievement. This effect was
stronger in reading (SDU ¼ 0.29) than in writing (SDU ¼ 0.16). In
writing, however, there was also an inverse compositional effect
relating to achievement at the classroom level, such that a student
in a classroom with comparatively high average achievement was
typically assigned a lower OTJ than a student with equivalent
standardized achievement in a classroom with average achieve-
ment (SDU ¼ 0.17). This classroom effect was not significant in
reading (SDU ¼ 0.10). Incorporation of the achievement composi-
tion variables only very minimally altered the existing standardized
achievement and student characteristic parameters.
5.6. School characteristics
We also investigated the degree to which additional school-
level contextual factors explained the variance in OTJs after con-
trolling for standardized achievement, student-level characteris-
tics, and achievement composition. These contextual factors
included the socioeconomic profile of the school (low, mid, or high
decile), the region where the school was located, the school size,
and the proportion of minority students attending the school. None
of the contextual factors investigated was found to be significant. In
the New Zealand context, the socioeconomic profile of the school
has frequently been found to be significantly associated with
achievement, but in the current study school SES did not explain
additional variance, suggesting that controlling for standardized
achievement differences already encapsulated variability associ-
ated with decile, and also that average levels of bias were similar
irrespective of the socioeconomic profile of the school. Interaction
effects between student and school characteristics were also
explored. None of the interaction effects explained a significant
proportion of the variance in OTJs after the inclusion of student-
level predictors, further emphasizing the consistency of these re-
sults across contexts.
6. Discussion
The current study explored the relations between standardized
tests and teachers' judgments about student achievement in
reading and writing. The first research question investigated the
correlation between the two measures. It was expected that the
correlation would be high since, in contrast with much of the
previous research, teachers had access to the standardized
achievement results and could use these if desired. However, cor-
relations were only marginally stronger than those previously re-
ported. An extensive meta-analysis by Südkamp et al. (2012)
reported average correlations of 0.63, with a wide range of corre-
lations reported in individual studies (0.03 to 0.92). The overall
correlation in the current study was stronger (rs ¼ 0.72e0.73), but
also with a wide range of correlations in individual schools. In the
current study, more than two-thirds of reading schools, and almost
three-quarters of writing schools had correlations below 0.8, indi-
cating that in most schools teachers did not rely on the test as the
sole source of evidence for determining the OTJ for each student.
This suggests that, provided possible bias is addressed and reli-
ability improved, OTJs could provide useful information encapsu-
lating much more than an individual test score.
The second research question investigated whether there were
residual differences in teachers' judgments of student achievement
as a function of specific student characteristics, after controlling for
standardized achievement differences. The results indicated that
there were significant differences in the OTJs made by teachers
about students of different groups, with priority learners being
systematically assigned lower OTJs even when standardized
achievement was the same. The effect size of these differences may
at first appear small. For example, Hattie (2008) indicated that an
intervention effect size of less than 0.4 would be considered less
than average. However, Hattie further notes that maturation typi-
cally accounts for an effect size of approximately 0.25 per year,
suggesting interventions with an effect of 0.15 are sizable if
maturation has been deducted. Effect sizes are also typically re-
ported as bivariate comparisons e effects are typically attenuated
when more than one covariate is included (e.g., SES and ethnicity
within a single model; individually each is much larger than when
both are estimated together due to confounding). The effect sizes
 K. Meissel et al. / Teaching and Teacher Education 65 (2017) 48e60
reported in the current study represent the unique variance
explained by each variable after controlling for achievement dif-
ferences and any confounding, and yet remain comparable to the
intervention effect reported by Hattie, suggesting these effects
remain important.
Our third question was whether there were contextual factors
such as the achievement context of the classroom or school. It was
found that aggregate SES of the school had no significant effect on
students' OTJs after controlling for standardized achievement, nor
did school location, roll size, or the proportion of students from
minority backgrounds. However, the aggregate achievement profile
of the school did affect teachers' judgments in both reading and
writing, with the same standardized achievement result typically
resulting in a lower overall teacher judgment in schools with high
average achievement. In writing, there was also an achievement
context effect at the classroom, with students again typically
receiving lower OTJs if their classroom had higher average
achievement. This is an important finding because it indicates that
teacher judgments are context-specific, suggesting that while
teachers are being asked to make these judgments against specific
standards, judgments are at least partially norm-referenced. In
other words, if a teacher is working with a class of students in
which all students are above the standard, the teacher's knowledge
of differential ability within the classroom may well result in a
tendency to report OTJs that attempt to reflect this differentiation.
This ‘localization’ of judgments points to the complexities inherent
in the creation of teacher judgments that are comparable at a na-
tional level, and suggests the need for improved means of aligning
expectations across contexts. However, the contextual effects do
not explain why results are lower for priority learners e the in-
clusion of the compositional effects had little effect on the
parameter estimates for these groups.
6.1. Possible explanations and response
Internationally, standardized tests are often argued to disad-
vantage cultural minorities (e.g., Erwin & Worrell, 2012; Kim &
Zabelina, 2015; McGrady & Reynolds, 2013), but to date, no
research has quantitatively determined presence or absence of
cultural bias within the standardized tests used in the current
study, despite discussion both for (e.g., Fothergill & Taylor-
Jorgensen, n.d.; Haitana, 2007; May, n.d.) and against (e.g.
Keegan, Brown, & Hattie, 2014) the notion of bias within stan-
dardized tests used in New Zealand. If standardized tests are indeed
culturally biased, the misalignment between teacher judgments
and standardized results for priority learners is especially con-
cerning since teachers' judgments typically suggested considerably
larger achievement gaps for priority learners than potentially
biased tests did.
It is worth noting that these relative differences were very much
in line with typical societal understandings of which groups of
students do well in New Zealand schools, suggesting that these
results may be partly explained by a societal-level bias against
certain learner groups, particularly given the consistency of the
results across contexts. Previous research has suggested that cul-
tural biases might be present in the teaching workforce, with recent
research highlighting the unconscious bias persisting against Maori
within the educational system (Bishop & Berryman, 2006; Blank,
Houkamau, & Kingi, 2016; Peterson, Rubie-Davies, Osborne, &
Sibley, 2016). In New Zealand as well as elsewhere, the majority
of teachers (79% in NZ) belong to the cultural majority group
(Chubbuck & Zembylas, 2016; Education Counts, 2005). Chubbuck
and Zembylas (2016) report that teachers' bias or deficit views in
regard to diversity were persistently found in research examining
pre-service teacher dispositions.
57
It could be argued that some students may perform compara-
tively well within the relatively narrow bounds of a standardized
assessment, while still having lower performance across the full
curriculum. Arguably, only the classroom teacher would have the
necessary insight to be able to draw on a wide enough range of
evidence to determine achievement across the curriculum. The
possibility that OTJs draw on wider but concurrently different ev-
idence than what is captured by a standardized test provides a
justifiable reason for discrepancies between an individual OTJ and
the achievement level indicated by the standardized test e and
indeed, previous research has shown that primary teachers in New
Zealand often consider standardized tests to be assessments of
surface learning (Brown, 2009). However, if priority learners sys-
tematically have lower performance across the full curriculum even
when they are able to perform equally well on standardized
achievement tests, this would provide further evidence that these
students are underserved by the current educational system, and
suggests that teachers' judgments may be informed by construct
irrelevant information. Therefore, an understanding of the source
of these discrepancies is imperative.
It is possible that the discrepancies reported are at least partly
explained by different behavior among students within the priority
learner groups, especially among those with special needs (see e.g.
Chang & Sue, 2003), though this would be a particularly concerning
finding: behavior is not a construct relevant component of OTJs, nor
should it be. Unfortunately, as behavioral indicators were not
available, this possibility could not be explored.
OTJs may also be partly informed by differential teacher ex-
pectations. Expectation research has typically focused on teachers'
initial impressions of students, whereas judgment research usually
looks at teachers' evaluations of student achievement later in the
academic year. There is a substantial body of evidence both inter-
nationally, and in the New Zealand context, that indicates that
teachers typically have lower expectations of marginalized learners
even after controlling for achievement (Rubie-Davies et al., 2012).
However, the teacher judgment research has mostly focused on the
alignment of teachers' judgments with achievement measures,
rather than the specific implications for different groups of stu-
dents, so it has been less clear whether teacher beliefs about
different groups of students might play a role in the judgments they
make about students' achievement levels. It is plausible that
teachers' initial impressions of students, which are influenced by
the learner group of the student, may be sufficiently enduring that
they impact on teachers' judgments, even after working with the
student for some time.
Students evaluated to be less able than their peers are typically
given fewer learning opportunities, and are generally provided
with more restricted learning experiences, contributing to a self-
fulfilling prophecy of underachievement (Rubie-Davies, 2010).
Subjectivity in teacher judgments risks exacerbating these effects
further, since each subsequent teacher receives the OTJ from the
previous classroom teacher. Although prior judgment could pro-
vide useful information about a student's performance that sub-
sequent teachers can use to inform their teaching, any inherent bias
in the OTJ being provided to the student's new teacher risks a
compounding effect (Rubie-Davies et al., 2014), reducing expecta-
tions even further.
6.2. Implications for teaching and teacher education
In the absence of a coherent and robust moderation process,
dependable and consistent teacher judgments are unlikely
(Raphael, Au, & Goldman, 2009). It should be noted that these re-
sults reflect a system during early implementation, with facilitators
reporting that moderation was variable, and, in some schools
58 K. Meissel et al. / Teaching and Teacher Education 65 (2017) 48e60
absent. Therefore, one possible response would be improved
moderation, for example, by facilitating discussions within and
between schools about what evidence would be considered
necessary for a student to meet each of the OTJ levels. In addition,
we believe that clear guidelines should be provided to all schools
and teachers about how to moderate effectively. Effective moder-
ation hinges on sharing and discussing the evidence within a high
trust environment (Wyatt-Smith, Klenowski, & Gunn, 2010), which
remains the exception rather than the norm. The Ministry of Edu-
cation already provides moderation guidelines (see http://
assessment.tki.org.nz/Moderation), but the compositional effects
found regarding average achievement in each school suggested
that many teachers and schools were making judgments in relation
to the achievement within their own class. Considering moderation
was very limited at the time these data were collected, it could be
argued that these results represent ‘unfiltered’ teacher perceptions.
If teacher bias is the cause of the differences found in the current
study, as moderation processes improve and a framework around
the setting of OTJs is developed further, these differences should
reduce, or disappear. Indeed, in 2015 the Ministry of Education
released an optional tool called the Progress and Consistency Tool
(PaCT; https://pactinfo.education.govt.nz/), which provides a
framework to support teachers to make more consistent judg-
ments, though uptake has been comparatively low, with around
15% of schools nationally opting in during the first year (Gerritsen,
2016). However, if teacher bias is indeed the cause of the results
described in this paper, more robust processes that ensure
construct irrelevant information is reduced from OTJs would be
insufficient in and of itself, as it would not address the underlying
bias.
Biases are seen as deep, cognitive and emotional responses that
people have and it is argued that cultural competency training
focusing on differences might be of little effect (Blank et al., 2016).
To change someone's dispositions has been described as a “life-long
journey of transformation” (Nieto, 2000, p. 183). Recent studies of
interventions to develop social justice dispositions in pre-service
teachers continue to show mixed results (Chubbuck & Zembylas,
2016). Lai et al. (2014) conducted a meta-analysis of mainly US-
based studies, showing that only eight of seventeen interventions
were effective at reducing bias. Effective studies usually used a form
of counter-stereotype interventions in which participants were
primed to pair positive characteristics with groups that were usu-
ally subject to bias. The researchers also found that increasing
participants' critical thinking and moral reasoning tended to help to
reduce bias. In New Zealand some previous research has indicated
that teacher expectations about specific subgroups can be changed
via targeted professional learning interventions (Rubie-Davies
et al., 2015). However, the consistency of bias across contexts
suggests that professional development should not be restricted to
specific contexts and it is unclear whether increased teacher ex-
pectations would actually translate to reduced bias in teacher
judgments. Initial teacher education programs would provide a
potential avenue for wider delivery, but there is clearly a need for
further research in this area.
6.3. Limitations
Although we believe that these results paint a clear picture of an
underlying systematic bias within the New Zealand education
system, there are some limitations and alternative explanations
that should be noted. In particular, the lack of previous research
about OTJs meant that reliability and validity statistics were not
available, making it difficult to ascertain precisely what OTJs mea-
sure. The current study provides some insight into factors that have
an effect on OTJs, but more research is needed. Although the sample
is large, especially for the New Zealand context, it is not a repre-
sentative sample, since the data were collected as part of a pro-
fessional learning initiative. In addition, the priority learner
categories used are the standard categories used by the Ministry of
Education, but may mask certain underlying patterns for subgroups
within these categories. For example, the category Pasifika is an
umbrella term that includes a wide range of backgrounds (NZ born
of Pacific descent, as well as those born in the Pacific islands; e.g.,
Tonga, New Caledonia, Samoa, Fiji [Indian and indigenous], among
others). The use of an aggregate, school-level SES indicator is
another limitation, but since New Zealand schools do not collect
individual-level SES information, we were restricted to the school-
level indicator.
6.4. Conclusions
Our results indicate that priority learners received systemati-
cally lower teacher judgments than other students in 2012 and
2013, even when their standardized achievement was the same.
However, it is not possible to be certain about the cause of these
differences in teacher judgments for specific student groups
without further research. One possible explanation is that teachers
are biased against priority learners. However, it could also be that
the standardized achievement tests are positively biased, or that
the narrow focus of standardized tests measures an aspect of
achievement distinct from that measured by teacher judgments.
There may also be other possible causes that we have not yet
considered. Future research should investigate alternative expla-
nations for these results to develop a better understanding of how
teachers make OTJs, and why these judgments indicate larger
achievement gaps than standardized achievement tests. Removing
bias from teacher judgments should be a priority within education.
All students deserve equitable educational opportunities and the
elimination of teacher bias would be one way to reduce the
perpetuation of the social structure and enable all students to have
the opportunity to succeed within the educational system. Ac-
knowledgements We would like to acknowledge the Consortium
for Professional Learning (Evaluation Associates and The University
of Auckland’s Team Solutions) who made this study possible by
trusting us to explore and develop an understanding of the data
gathered during their professional learning project. The dedication
of every facilitator is hugely admired, and the provision of consis-
tently high quality data greatly appreciated. We also acknowledge
the New Zealand Ministry of Education for funding the work of the
Consortium for Professional Learning.
Acknowledgements
We would like to acknowledge the Consortium for Professional
Learning (Evaluation Associates and The University of Auckland's
Team Solutions) who made this study possible by trusting us to
explore and develop an understanding of the data gathered during
their professional learning project. The dedication of every facili-
tator is hugely admired, and the provision of consistently high
quality data greatly appreciated. We also acknowledge the New
Zealand Ministry of Education for funding the work of the Con-
sortium for Professional Learning.
References
Alvidrez, J., & Weinstein, R. S. (1999). Early teacher perceptions and later student
academic achievement. Journal of Educational Psychology, 91(4), 731e746.
http://dx.doi.org/10.1037/0022-0663.91.4.731.
Angoff, W. H. (1974). Criterion-referencing, norm-referencing and the SAT. The
College Board Review, 92, 3e5, 21.
Begeny, J. C., Eckert, T. L., Montarello, S. A., & Storie, M. S. (2008). Teachers'
 K. Meissel et al. / Teaching and Teacher Education 65 (2017) 48e60
perceptions of students' reading abilities: An examination of the relationship
between teachers' judgments and students' performance across a continuum of
rating methods. School Psychology Quarterly, 23(1), 43.
Begeny, J. C., Krouse, H. E., Brown, K. G., & Mann, C. M. (2011). Teacher judgments of
students' reading abilities across a continuum of rating methods and achieve-
ment measures. School Psychology Review, 40(1), 23e38.
Benner, A. D., & Mistry, R. S. (2007). Congruence of mother and teacher educational
expectations and low-income youth's academic competence. Journal of Educa-
tional Psychology, 99(1), 140e153. http://dx.doi.org/10.1037/0022-0663.99.1.140.
Bennett, R. E., Gottesman, R. L., Rock, D. A., & Cerullo, F. (1993). Influence of behavior
perceptions and gender on teachers' judgments of students' academic skill.
Journal of Educational Psychology, 85(2), 347e356. http://dx.doi.org/10.1037/
0022-0663.85.2.347.
Beswick, J. F., Willms, J. D., & Sloat, E. A. (2005). A comparative study of teacher
ratings of emergent literacy skills and student performance on a standardized
measure. Education, 126, 116e137.
Bishop, R., & Berryman, M. (2006). Culture speaks: Cultural relationships & classroom
learning. Wellington, NZ: Huia.
Blank, A., Houkamau, C., & Kingi, H. (2016). Unconscious bias and education: A
comparative study of Ma ori and African American students. Retrieved from http://
apo.org.au/resource/unconscious-bias-and-education-comparative-study-
maori-and-african-american-students.
de Boer, H., Bosker, R. J., & van der Werf, M. P. C. (2010). Sustainability of teacher
expectation bias effects on long-term student performance. Journal of Educa-
tional Psychology, 102(1), 168e179. http://dx.doi.org/10.1037/a0017289.
Brown, G. (2013). asTTle - a national testing system for formative assessment: How
the national testing policy ended up helping schools and teachers. In M. Lai, &
S. Kushner (Eds.), A national developmental and negotiated approach to school
and curriculum evaluation (pp. 39e56). Bingley, UK: Emerald Group Publishing.
Brown, G. T. L. (2009). Teachers' self-reported assessment practices and concep-
tions: Using structural equation modelling to examine measurement and
structural models. In T. Teo, & M. S. Kline (Eds.), Structural equation modelling in
educational research: Concepts and applications (pp. 243e266). Rotterdam, NL:
Sense Publishers.
Chamberlain, M. (2010). Blueprint for national standards. Retrieved from New Zea-
land Education Gazette http://www.edgazette.govt.nz/Articles/Article.aspx?
ArticleId¼8187.
Chang, D. F., & Sue, S. (2003). The effects of race and problem type on teachers'
assessments of student behavior. Journal of Consulting and Clinical Psychology,
71(2), 235e242. http://dx.doi.org/10.1037/0022-006X.71.2.235.
Chubbuck, S. M., & Zembylas, M. (2016). Social justice and teacher education:
Context, theory, and practice. In John Loughran, & Mary Lynn_Hamilton (Eds.),
International handbook of teacher education (Vol. 2, pp. 463e501). Singapore:
Springer.
Clark, C. M., & Peterson, P. L. (1986). Teachers' thought processes. In M. C. Wittrock
(Ed.), Third handbook of research on teaching (pp. 255e296). New York, NY:
Macmillan.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Hillsdale, NJ:
Laurence Erlbaum.
Courtney, B. (2010). National standards: A parent's perspective. New Zealand Journal
of Teachers' Work, 7, 8e14.
Darr, C., McDowall, S., Ferral, H., Twist, J., & Watson, V. (2008). Progressive
achievement Test: Reading - teacher manual. Wellington, NZ: New Zealand
Council for Educational Research.
Darr, C., Neill, A., Stephanou, A., & Ferral, H. (2006). Progressive achievement Test:
Mathematics e teacher manual. Wellington, NZ: New Zealand Council for
Educational Research.
Doherty, J., & Conolly, M. (1985). How accurately can primary school teachers
predict the scores of their pupils in standardized tests of attainment? A study of
some non-cognitive factors that influence specific judgements. Educational
Studies, 11(1), 41e60. http://dx.doi.org/10.1080/0305569850110105.
Dompnier, B., Pansu, P., & Bressoux, P. (2006). An integrative model of scholastic
judgments: Pupils' characteristics, class context, halo effect and internal attri-
butions. European Journal of Psychology of Education, 21(2), 119e133. http://
dx.doi.org/10.1007/BF03173572.
Dorman, J. P. (2008). The effect of clustering on statistical tests: An illustration using
classroom environment data. Educational Psychology, 28(5), 583e595.
Eames, D. (2010). National standards policy: How parents mark it. Retrieved from
nzherald.co.nz http://www.nzherald.co.nz/nz/news/article.cfm?c_
id¼1&objectid¼10624503.
Education Counts. (2005). Teacher census 2004. Retrieved 24/11/2016, from www.
educationcounts.govt.nz/publications/schooling/teacher_census.
Erwin, J. O., & Worrell, F. C. (2012). Assessment practices and the underrepresen-
tation of minority students in gifted and talented education. Journal of Psy-
choeducational Assessment, 30(1), 74e87.
Fairfax New Zealand Limited. (2012). How New Zealand schools rate. Retrieved from
stuff.co.nz http://www.stuff.co.nz/national/education/7715044/How-New-
Zealand-schools-rate.
Fairfax New Zealand Limited. (2016). School report. Retrieved from stuff.co.nz http://
www.stuff.co.nz/national/education/school-report.
Feinberg, A. B., & Shapiro, E. S. (2003). Accuracy of teacher judgments in predicting
oral reading fluency. School Psychology Quarterly, 18(1), 52e65.
Fothergill, J., & Taylor-Jorgensen, T. (n.d.). Assessment in New Zealand primary
schools. Retrieved June 2016, from https://sites.google.com/site/ttjjmf/home.
Francis, B., Archer, L., Hodgen, J., Pepper, D., Taylor, B., & Travers, M.-C. (2016).
59
Exploring the relative lack of impact of research on ‘ability grouping’ in En-
gland: A discourse analytic account. Cambridge Journal of Education, 1e17.
http://dx.doi.org/10.1080/0305764X.2015.1093095.
Gerritsen, J. (2016). School groups wary of national standards computer system. Radio
New Zealand. Retrieved from www.radionz.co.nz/news/national/295621/
school-groups-wary-of-national-standards-computer-system.
Gill, J. (2002). Bayesian methods: A social and behavioral sciences approach. Boca
Raton, Fla: Chapman and Hall.
Glock, S., Krolak-Schwerdt, S., & Pit-ten Cate, I. M. (2015). Are school placement
recommendations accurate? The effect of students' ethnicity on teachers'
judgments and recognition memory. European Journal of Psychology of Educa-
tion, 30(2), 169e188. http://dx.doi.org/10.1007/s10212-014-0237-2.
Haitana, T. N. (2007). Testing Tamariki: How suitable is the PPVT-III?. Master’s
Dissertation. Hamilton, NZ: University of Waikato.
Hanushek, E. A., Link, S., & Woessmann, L. (2013). Does school autonomy make
sense everywhere? Panel estimates from PISA. Journal of Development Eco-
nomics, 104, 212e232.
Harlen, W. (2005). Trusting teachers' judgement: Research evidence of the reli-
ability and validity of teachers' assessment used for summative purposes.
Research Papers in Education, 20(3), 245e270. http://dx.doi.org/10.1080/
02671520500193744.
Hattie, J. A. C. (2008). Narrow the gap, fix the tail, or close the curves: The power of
words. In C. Rubie, & C. Rawlinson (Eds.), Challenging thinking about teaching
and learning. Nova Science.
Hattie, J. A., & Brown, G. T. L. (2003). Standard setting for asTTle reading: A comparison
of methods. asTTle Tech. Rep. #21. University of Auckland/Ministry of Education.
Hattie, J. A. C., Brown, G. T. L., Keegan, P., Irving, S. E., MacKay, A. J., Sutherland, T.,
et al. (2003). Validation evidence of asTTle reading assessment Results: Norms and
criteria. AsTTle Tech. Rep. 22. University of Auckland/Ministry of Education.
Hecht, S. A., & Greenfield, D. B. (2002). Explaining the predictive accuracy of teacher
judgments of their students' reading achievement: The role of gender, class-
room behavior, and emergent literacy skills in a longitudinal sample of children
exposed to poverty. Reading and Writing, 15(7e8), 789e809. http://dx.doi.org/
10.1023/A:1020985701556.
Helwig, R., Anderson, L., & Tindal, G. (2001). Influence of elementary student gender
on teachers' perceptions of mathematics achievement. The Journal of Educa-
tional Research, 95(2), 93e102. http://dx.doi.org/10.1080/00220670109596577.
Hoge, R. D., & Butcher, R. (1984). Analysis of teacher judgments of pupil achieve-
ment levels. Journal of Educational Psychology, 76(5), 777e781. http://dx.doi.org/
10.1037/0022-0663.76.5.777.
Hoge, R. D., & Coladarci, T. (1989). Teacher-based judgments of academic achieve-
ment: A review of literature. Review of Educational Research, 59(3), 297e313.
http://dx.doi.org/10.3102/00346543059003297.
Hopkins, K. D., George, C. A., & Williams, D. D. (1985). The concurrent validity of
standardized achievement tests by content area using teachers' ratings as
criteria. Journal of Educational Measurement, 22, 177e182.
Hurwitz, J. T., Elliott, S. N., & Braden, J. P. (2007). The influence of test familiarity and
student disability status upon teachers' judgments of students' test perfor-
mance. School Psychology Quarterly, 22(2), 115.
Jussim, L., & Harber, K. D. (2005). Teacher expectations and self-fulfilling prophe-
cies: Knowns and unknowns, resolved and unresolved controversies. Person-
ality and Social Psychology Review, 9(2), 131e155. http://dx.doi.org/10.1207/
s15327957pspr0902_3.
Kaiser, J., Retelsdorf, J., Südkamp, A., & Mo €ller, J. (2013). Achievement and engage-
ment: How student characteristics influence teacher judgments. Learning and
Instruction, 28, 73e84. http://dx.doi.org/10.1016/j.learninstruc.2013.06.001.
Keegan, P., Brown, G. T., & Hattie, J. A. (2014). 4 A psychometric view of sociocultural
factors in test validity: The development of standardised test materials for
Ma ori medium schools in New Zealand/Aotearoa. In S. Phillipson, K. Y. L. Ku, &
S. N. Phillipson (Eds.), Constructing achievement: A sociocultural perspective (pp.
42e54). London: Routledge.
Kim, H. Y. (2013). Statistical notes for clinical researchers: Assessing normal dis-
tribution using skewness and kurtosis. Restorative Dentistry & Endodontics,
38(1), 52e54. http://dx.doi.org/10.5395/rde.2013.38.1.52.
Kim, K. H., & Zabelina, D. (2015). Cultural bias in assessment: Can creativity
assessment help? The International Journal of Critical Pedagogy, 6(2).
Kline, T. J. (2005). Psychological testing: A practical approach to design and evaluation.
Thousand Oaks, CA: Sage.
Kreft, I. G. G., de Leeuw, J., & Aiken, L. S. (1995). The effect of different forms of
centering in Hierarchical Linear Models. Multivariate Behavioral Research, 30,
1e21.
Lai, C. K., Marini, M., Lehr, S. A., Cerruti, C., Shin, J. L., Joy-Gaba, J. A., … Nosek, B. A.
(2014). Reducing implicit racial preferences: I. A comparative investigation of 17
interventions. Journal of Experimental Psychology: General, 143(4), 1765e1785.
http://dx.doi.org/10.1037/a0036260.
Lembke, E. S., Foegen, A., Whittaker, T. A., & Hampton, D. (2008). Establishing
technically adequate measures of progress in early numeracy. Assessment for
Effective Intervention, 33(4), 206e214. http://dx.doi.org/10.1177/
1534508407313479.
Li, H., Pfeiffer, S. I., Petscher, Y., Kumtepe, A. T., & Mo, G. (2008). Validation of the
gifted rating scaleseschool form in China. Gifted Child Quarterly, 52(2), 160e169.
http://dx.doi.org/10.1177/0016986208315802.
Martínez, J. F., Stecher, B., & Borko, H. (2009). Classroom assessment practices,
teacher judgments, and student achievement in mathematics: Evidence from
the ECLS. Educational Assessment, 14(2), 78e102. http://dx.doi.org/10.1080/
60 K. Meissel et al. / Teaching and Teacher Education 65 (2017) 48e60
10627190903039429.
May, S. (n.d.). Assessment: What are the cultural issues in relation to Pasifika, Asian,
ESOL, immigrant and refugee learners? University of Waikato. Retrieved June
2016, from http://assessment.tki.org.nz/Media/Files/May-S.-Assessment-what-
are-the-cultural-issues-in-relation-to-Pasifika-Asian-ESOL-immigrant-and-
refugee-learners-University-of-Waikato.
McGrady, P. B., & Reynolds, J. R. (2013). Racial mismatch in the classroom beyond
Black-white differences. Sociology of Education, 86(1), 3e17.
McKown, C., & Weinstein, R. S. (2008). Teacher expectations, classroom context, and
the achievement gap. Journal of School Psychology, 46(3), 235e261.
Meisels, S. J., Bickel, D. D., Nicholson, J., Xue, Y., & Atkins-Burnett, S. (2001). Trusting
teachers' judgments: A validity study of a curriculum-embedded performance
assessment in kindergarten to grade 3. American Educational Research Journal,
38(1), 73e95. http://dx.doi.org/10.3102/00028312038001073.
Methe, S. A., Hintze, J. M., & Floyd, R. G. (2008). Validation and decision accuracy of
early numeracy skill indicators. School Psychology Review, 37(3), 359e373.
Ministry of Education. (2010). National standards. Retrieved from TKI: Te Kete
Ipurangi http://nzcurriculum.tki.org.nz/National-Standards.
Ministry of Education. (2011). Overall teacher judgment. Retrieved from www.
nzcurriculum.tki.org.nz/National-Standards/Key-information/Fact-sheets/
Overall-teacher-judgment/ http://www.nzcurriculum.tki.org.nz/National-
Standards/Key-information/Fact-sheets/Overall-teacher-judgment/.
Ministry of Education. (2012). National Standards: Key information. Retrieved from
http://www.nzcurriculum.tki.org.nz/National-Standards/.
Ministry of Education, & NZCER. (2012). e-asTTle technical manual. Wellington, NZ:
Ministry of Education.
Ministry of Education. (2016). Find a school. Retrieved from Education Counts
https://www.educationcounts.govt.nz/find-school.
Nieto, S. (2000). Placing equity front and center some thoughts on transforming
teacher education for a new century. Journal of Teacher Education, 51(3),
180e187.
Ogle, L., Sen, A., Pahlke, E., Jocelyn, L., Kastberg, D., Roey, S., & Williams, T. (2003).
International comparisons in fourth grade reading literacy: Findings from the
Progress in International Reading Literacy Study (PIRLS) of 2001 (NCES 2003-073).
Washington, DC: U.S. Government Printing Office.
Organisation for Economic Co-operation and Development (OECD). (2005). PISA
2003 Technical Report. Paris, France: OECD Publications.
Organisation for Economic Co-operation and Development (OECD). (2013). PISA
2012 results: Excellence through equity: Giving every student the chance to succeed
(Vol. 2). Paris, France: OECD Publications. http://dx.doi.org/10.1787/
9789264201132-en.
€
OZerk, K., & Whitehead, D. (2012). The impact of national standards assessment in
New Zealand, and national testing protocols in Norway on indigenous
schooling. International Electronic Journal of Elementary Education, 4(3), 545.
Parsons, S., & Hallam, S. (2014). The impact of streaming on attainment at age
seven: Evidence from the Millennium Cohort Study. Oxford Review of Education,
40(5), 567e589. http://dx.doi.org/10.1080/03054985.2014.959911.
Peterson, E. R., Rubie-Davies, C. M., Osborne, D., & Sibley, C. (2016). Implicit and
explicit teacher expectations: Relations with ethnicity and achievement.
Learning and Instruction, 42, 123e140. http://dx.doi.org/10.1016/
j.learninstruc.2016.01.010.
Raphael, T. E., Au, K. H., & Goldman, S. R. (2009). Whole school instructional
improvement through the standards-based change process. In J. Hoffman, &
Y. Goodman (Eds.), Changing literacies for changing times (pp. 198e229). London:
Routledge.
Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and
data analysis methods (2nd ed.). Thousand Oaks: Sage Publications.
Ready, D. D., & Wright, D. L. (2011). Accuracy and inaccuracy in teachers' percep-
tions of young children's cognitive abilities: The role of child background and
classroom context. American Educational Research Journal, 48(2), 335e360.
http://dx.doi.org/10.3102/0002831210374874.
Rubie-Davies, C. M. (2010). Teacher expectations and perceptions of student attri-
butes: Is there a relationship? British Journal of Educational Psychology, 80(1),
121e135.
Rubie-Davies, C. (2014). Becoming a high expectation teacher: Raising the bar. New
York, NY: Routledge.
Rubie-Davies, C. M., Hattie, J., & Hamilton, R. (2006). Expecting the best for stu-
dents: Teacher expectations and academic outcomes. British Journal of Educa-
tional Psychology, 76(3), 429e444.
Rubie-Davies, C. M., Peterson, E. R., Flint, A., Garrett, L., McDonald, L. M., Watson, P.,
& O'Neill, H. (2012). Ethnicity and teacher expectations in New Zealand. Pro-
cedia-Social and Behavioral Sciences, 69, 256e261.
Rubie-Davies, C. M., Peterson, E. R., Sibley, C. G., & Rosenthal, R. (2015). A teacher
expectation intervention: Modelling the practices of high expectation teachers.
Contemporary Educational Psychology, 40, 72e85.
Rubie-Davies, C. M., Weinstein, R. S., Huang, F. L., Gregory, A., Cowan, P. A., &
Cowan, C. P. (2014). Successive teacher expectation effects across the early
school years. Journal of Applied Developmental Psychology, 35(3), 181e191.
Satherley, P. (2006). Student outcome overview 2001e2005: Research findings on
student achievement in reading, writing and mathematics in New Zealand schools.
Wellington, NZ: Ministry of Education, Research Division.
Schmidt, W., Burroughs, N., Zoido, P., & Houang, R. (2015). The role of schooling in
perpetuating educational inequality: An international perspective. Educational
Researcher, 44(7), 371e386. http://dx.doi.org/10.3102/0013189X15603982.
Sharpley, C. F., & Edgar, E. (1986). Teachers' ratings vs standardized tests: An
empirical investigation of agreement between two indices of achievement.
Psychology in the Schools, 23(1), 106e111. http://dx.doi.org/10.1002/1520-
6807(198601)23:1<106::AID-PITS2310230117>3.0.CO;2-C.
Smith, L. A., Anderson, V., & Blanch, K. (2016). Five beginning teachers' reflections
on enacting New Zealand's national standards. Teaching and Teacher Education,
54, 107e116.
Statistics New Zealand. (2016). Time series data for student numbers. Retrieved 23/
11/2016, from Education Counts www.educationcounts.govt.nz/statistics/
schooling/student-numbers/6028.
Südkamp, A., Kaiser, J., & Mo €ller, J. (2012). Accuracy of teachers' judgments of
students' academic achievement: A meta-analysis. Journal of Educational Psy-
chology, 104(3), 743e762. http://dx.doi.org/10.1037/a0027627.
Südkamp, A., Kaiser, J., & Mo €ller, J. (2014). Teachers' judgments of students' aca-
demic achievement. In S. Krolak-Schwerdt, S. Glock, & M. Bo € hmer (Eds.),
Teachers' professional development: Assessment, training, and learning (pp. 5e25).
Rotterdam, NL: Sense Publishers.
Thrupp, M. (2013). National standards for student achievement: Is New Zealand's
idiosyncratic approach any better. Australian Journal of Language and Literacy,
36(2), 99e110.
Wiliam, D., & Bartholomew, H. (2004). It's not which school but which set you’re in
that matters: The influence of ability grouping practices on student progress in
mathematics. British Educational Research Journal, 30(2), 279e293. http://
dx.doi.org/10.1080/0141192042000195245.
Wilson, A., Madjar, I., & McNaughton, S. (2016). Opportunity to learn about disci-
plinary literacy in senior secondary English classrooms in New Zealand. The
Curriculum Journal, 27(2), 204e228. http://dx.doi.org/10.1080/
09585176.2015.1134339.
Woltman, H., Feldstain, A., MacKay, J. C., & Rocchi, M. (2012). An introduction to
hierarchical linear modeling. Tutorials in Quantitative Methods for Psychology,
8(1), 52e69.
Wyatt-Smith, C., Klenowski, V., & Gunn, S. (2010). The centrality of teachers'
judgment practice in assessment: A study of standards in moderation. Assess-
ment in Education: Principles, Policy & Practice, 17, 59e75.
Wylie, C., Cosslett, G., & Burgon, J. (2016). New Zealand principals: Autonomy at a
cost. In A decade of research on school principals (pp. 269e290). Springer In-
ternational Publishing.