Construct-Irrelevant Variance

in High-Stakes Testing
Thomas M. Haladyna, Arizona State University West
Steven M. Downing, University of Illinois at Chicago

developing abilities (Messick, 1984),

and learned abilities (Sternberg, 1998).
There are many threats to validity in high-stakes achievement Each cognitive ability involves contex
testing. One major threat is construct-irrelevant variance (CIV). tualized mental models, schemas, or
This article defines ClV in the con text of the contemporary, unitary frarn1s, and complex performance that
view of validity and presents logical arguments, hypotheses, 111:1y have multiple correct pathways that
and documentation for a variety of CIV sources that commonly depend on knowledge and skills. These
abilities are slow growing. They are
threaten interpretations of test scores. A more thorough study difficult to teach and learn. The item
of ClV is recommended. format with greatest fidelity to mea
sure this kind of construct is perfor
Keywords: construct-irrelevant variance, high-stakes testing, validity mance. Messick (1984) referred to this
measurement approach as construct
referenced because the performance
test focuses on the ability itself and not
C urrently, achievement testing can
be characterized as driven by con
tent standards that affect the planning
Part 1: Validity
The most fundamental. step in valida
on the domain of knowledge and skills
that support it. Mislevy (1996) traced
tion is defining the construct. Cronbach the history of achievement testing and
and delivery of instruction and the de and Meehl ( 1955) called this construct forecasted that cognitive psychology will
sign of student assessments. The align formulation. Two kinds of achievement lead us in a direction that merges learn
ment of content standards, instruction, constructs seem represented in state ing theory and test theory. For our pur
,md the t(\Sts used to evaluate student ancl national content standards. poses both kinds of achievement con
l,arning are commonly held paradigms The first kind of achievement con structs are SUS('Pptible to CIV, hut. as we
in education. Most states either have struct can be envisioned as a large will see in different ways.
in place or are developing comprehen domain of knowledge and skills, some According to the Standards for
sive assessment systems that have these times called declarative and procedural Edu.rational and Psychological TesHng
fr:atures. knowledge. Any achievement test is in- (Strwdards) (American Educational
In many states, test scores can have 1 ended to be a representative sample Rcscarch Association [AERA], Ameri
several high-stakes uses. For instance, from that domain (Haertel & Calfee, can Psychological Association [APA], &
students must pass tests to graduate 1983). Although test specifications guide National Council on Measurement in
from high school or to be promoted the design of these tests, the sample of Education [NCME], 1999): "Validity
to the next grade. Schools are evalu this domain is usually small. Each stu
ated based on test scores and annual dent's test score is intended to show
progress. Low-performing schools may status in this domain. This type of
be subject to intervention. Teachers and achievement construct can be viewed
school leaders may be evaluated based Thomas M. Haladyna is Professor r!f'
as traditional. Tests of domains of knowl Educational Psychology, College of Teacher
on student test performance, and their edge and skills are consistent with the Education and Leadership, Arizona State
employment and pay may be affected by 'criterion-referenced" movement of the U,1,ivenntv West, FAE 240 South, 4701 West
this evaluation. 1970s and 1980s. Multiple-choice (MC) Thunderbird Road, PO Box 37100, Phoeni:x,
This article addresses a serious prob formats work well to measure this kind AZ 85069-7100, thomas. haladyna@asu.edu.
lem in high-stakes testing: construct of achievement construct. He specializes in developing and evaluating
irrelevant variance (CIV). Part 1 reviews A second kind of achievement con testing programs and in item develop
the contemporary view of validity, with struct focuses on a cognitive abil:itu, ment and validation.
its ,mphasis on construct validity. In such as reading, writing, or mathemati Steve11 JI Downing is Associate Professor
Part 2, CIV is defined. In Part 3, a tax cal prnll!Pm solving. We can conceive of of Medi(/// Fr/!ication, Unh)(lnity of Illinois
onomy is presented that organizes this ability as consisting of a domain of at Chi,e,1.1;,1, n ,//ege ofMedicine, Department
ofMedical Education (MC 591), 8088. Wood
sources of CIV. Evidence is presented complex tasks. The theoretical ratio Street, Chicago, IL 60612; sdowning@uic.
of CIV's extensiveness in high-stakes nale for any cognitive ability comes edu. He specializes in test development
testing. Some of these sources of Cl\ from cognitive psychology. Other terms and ps,lfchrnnr:tric issues.fin achievement
have rnceived more research attention used to signify a cognitive ability in and credrmlialing examinations in the
than others. clude fluid abilities (Lohman, 1993), profess lo n s.

Spring 2004 17
refers to the degree to which evidence
and theory support the interpretations
of test scores entailed by proposed uses
of tests" (p. 9). Validation is an inves
tigative process by which we (a) create
a plausible argument regarding a de
sired interpretation or use of test scores,
(b) collect and organize validity evi
dence bearing on this argument, and
(c) evaluate the argument and the ev
idence concerning the validity of the
interpretation. Kane (1992) described
the process for establishing a plausible
argument and criteria we might use to
evaluate this argument. Kane (2002)
also described some nuances of de
scriptive and policy-based interpreta
tions in high-stakes student achieve
ment testing. He argued that current
validation does not go far enough to
justify the full array of interpretations
and uses.
As the stakes for testing increase, the
need for validity evidence also increases
( Linn, 2002). The quest for validity evi
dence can be very complex. This evi
dence will likely consist of documenta
tion of procedures in test development,
administration, scoring and report
ing, and empirical studies (Downing &
Haladyna, 1996; Haladyna, 2002a). The
Standards (AERA, APA, & NCME, 1999)
present categories of validity evidence
that include content, cognitive pro
cesses, internal structure of item re
sponses or ratings of performance,
reliability, relations of test scores to
other variables, and consequences. Es
says by Messick ( 1995a, 1995b) also pro
vide suggestions for types of validity ev
idence and their importance.
Although we may view validation as a
process for strengthening an argument
about the validity of a particular inter
pretation or test score use, Cronbach
(1988) noted that validation should
also include the testing of alternate
hypotheses concerning the validity of
an interpretation. Crooks, Kane, and
Cohen (1996) provided a comprehen
sive model for the study of threats to
validity. They identified eight linked
inferences and argued that a weakness
in any link weakens the entire chain.
As we acquire and evaluate validity ev
idence, we may conclude that some ev
idence is weak or negative. By elimi
nating or reducing these threats to
validity, we can increase our confi
dence that a desired test score inter
pretation or use is more valid.
Several writers have proposed ways of
organizing and describing threats to va-
18
lidity ( e.g., Crooks, et al. 1996; Messick,
1984, 1989). At least five major threats
to validity stand out: construct under
representation arising form poorly con
ceptualized or inadequately operational
ized constructs, faulty logic of the causal
inference regarding test scores, negative
consequences of test score interpreta
tions and uses, lack of reproducibility of
test scores, and CIV. Although all of
these threats deserve attention in vali
dation research, this article concen
trates on CIV.
Part 2: Construct-Irrelevant
Variance
The major point here is that educa
tional achievement tests, at best,
reflect not only the psychological con
structs of knowledge and skills that
are intended to be measured, but in
variably a number of contaminants.
These adulterating influences include
a variety of other psychological and
situational factors that technically
constitute either construct-irrelevant
test difficulty or construct-irrelevant
contamination in score interpreta
tion. (Messick, 1984, p. 216)
CIV is error variance that arises from
systematic error. A good way to think
about systematic error is to compare it
with random error. If we were to write
the linear model representing what we
know about random and systematic er
rors, the model would be:
y = t +er+ es,
where y is the observed test score for
any student, t is the true score, en
is random error, and es is systematic
error due to CIV. Lord and Novick
(1968, pp. 43-44) developed this idea
and presented systematic error as a
redefined true score that is essentially
biased.
Random error is the difference be
tween any observed and correspond
ing true score for each examinee. Both
classical and generalizability theory
(Brennan, 2001) present methods to
study random error. Random error can
be large or small, positive or negative.
We never know the size ofrandom error
for any examinee. Reliability is the ratio
of true-score variance and observed
score variance for a set of test scores.
Random error and the observed scores
are random variables, whereas the true
score is a constant (Lord & Novick,
1968, p. 35). Random error is uncorre-
Educational Measurement: Issues and Practice
lated with true and observed scores. The
expected value of random error across a
set of test scores is zero.
Systematic error is not random, but
group- or person specific. Construct
irrelevant easiness refers to a contami
nating influence on test scores that
tends to systematically increase test
scores for a specific examinee or a group
of examinees; construct-irrelevant dif
ficulty does the opposite. It systemati
cally decreases test scores for a specific
examinee or a group of examinees. Lord
and Novick (1968, p. 43) discussed sys
tematic error as an undesirable change
in true score. The change is caused by a
variable that is unrelated to the con
struct being measured. Thus, the change
in test score is construct irrelevant.
Although random error is variable from
examinee to examinee, systematic error
is not. It is predictable. It also manifests
in two types.
The first type of systematic error is
constant error for all members of a pa;-
ticular group. This kind of error is char
acterized by members of a specific group
having systematic over- or underestima
tion of their true scores. A good example
of this type of constant error is rater
severity in a performance test of a cogni
tive ability. Two raters are consis
tently harsh. Student papers evaluated
by these two raters will likely result in
systematic error that lowers their test
scores. The group being scored by these
two harsh raters gets lower scores than
they deserve. Test form difficulty is
another example of group-specific CIV.
Those taking the more difficult test form
will have underestimated scores unless
test score equating is carried out.
The second type of systematic error is
the over- or underestimation of individ
ual examinee scores due to a CIV source
that affects examinees differentially.
Messick (1989) cited reading ability
on subject-matter tests as an example
of this kind of CIV. If performance
on a test of a construct that is not
reading comprehension is strongly de
pendent on one's level of reading com
prehension, then reading comprehen
sion is construct-irrelevant, because
the definition of achievement does
not include reading comprehension. For
instance, two students having equal
science achievement may differ in their
test performance because one is a better
reader than the other. It is reading com
prehension that differentiates these two
students, not science achievement. By
increasing the reading comprehension
demand on this science test, reading
comprehension potentially contami
nates the measure of science achieve
ment and its subsequent interpretation
or use. By lowering the reading demand
on this test, CIV's threat due to reading
comprehension is lessened. Other exam
ples of person-specific CIV include moti
vation to perform on a test, test anxiety,
and fatigue. TheStandards (AERA, APA,
& NCME, 1999) also give some examples
of potential sources of CIV, such as read
ing comprehension, item formats, anxi
ety, and test administration conditions.
Both Messick (1989) and Lord and
Novick (1968) offer examples. This arti
cle extends and expands these discus
sions of sources of CIV.
Contrasts and Comparisons of
Random and Systemati,c Error
Some contrasts and comparisons be
tween random and systematic error may
further clarify this definition of CIV.
While random error is uncorrelated to
true and observed scores, systematic
error is correlated to both true and ob
served scores. This is true because both
individuals and groups are either af
fected or unaffected by CIV. The un
affected individuals or groups have
observed scores that are closer to true
scores. Although the expected value of
random error is zero, if CIV is present,
then the expected value of systematic
error is a non-zero value. Systematic
error is quantifiable. The larger this error
variance, the more serious the threat
to validity. Lord and Novick (1968) con
tended that if the estimation of system
atic error is impossible, then test scores
may not be validly interpreted and,
thus, be of no use to the test sponsor.
Estimating CJV and Adjusting Test
Scores to Eliminate CJV
The estimation of systematic error can
be accomplished by using the general
linear model. The test score serves
as the dependent variable, and a CIV
source serves as the independent vari
able. The percentage of accounted vari
ance provides a measure of effect size
for the source of CIV. With individual
specific error, the product-moment
correlation provides this index. With
group-specific error, R2 provides this
index. Analysis of variance also provides
a basis for computing the effect size as
well as the appropriate statistical test
of significance.
There are many computer programs
for the adjustment and elimination of
Spring 2004
some sources of CIV arising from MC
or performance tests. These include
ConQuest (Adams, Wu,& Wilson, 1998),
Facets (Linacre & Wright, 2004),
Parscale (Muraki & Bock, 2003), and
Rumm (Andrich, Lyne, Sheridan, &
Luo, 1997). Generalizability theory
(Brennan, 2001) offers a useful way to
study the precision of test scores and
sources of random error, but it also pro
vides a basis for studying CIV as well.
However, not all sources of CIV are so
easily identifiable, and adjustment to
reduce or eliminate its influence will be
a considerable challenge to measure
ment specialists.
Related Ideas in the
Measurement Literature
Messick (1989) devoted only several
paragraphs to CIV in his influential essay
on validity. Lord and Novick (1968) rec
ognized this problem and incorporated it
into their discussion of true scores. The
Standards (AERA, APA, & NCME, 1999,
p. 10) identifies only six standards bear
ing on CIV, and provides a brief discus
sion. The six standards and their points
of discussion are:
Standard 7.1-subgroup differ
ences
Standard 7.2-the validity of in
ferences when CIV is present
Standard 7.3-differential item
functioning, one source of CIV
Standard 7.10-the importance of
studying systematic error among groups
where no differences are believed to
exist
Standard 12.19-the interpreta
tion of test results when CIV may be
present
Standard 13.18-concerns of CIV
in computerized testing.
These standards are not very specific as
to the many sources of CIV identified in
this article. Considering the seriousness
of the CIV threat to validity for high
stakes test score interpretations and
uses, it is important to expand our un
derstanding of CIV. Chapter 7 of the
Standards (AERA, APA, & NCME, 1999)
addresses fairness in testing. As pointed
out in that chapter, there is no common
meaning or technical definition for fair
ness. Fairness has four connotations:
(a) all tests should be free from bias,
(b) all examinees are treated equitably,
(c) the test scores should be equal for
groups thought to be equal, and (d) stu
dents being tested have equal and fair
opportunities to learn before being
tested and before we make decisions
based on these test scores.
Cole and Moss ( 1989) described a du
ality with the term bias. The first conno
tation of bias involves social justice and
equal treatment of students, such as is
embodied in the idea of opportunity to
learn. The second connotation is a more
technical issue that relates to the test as
a cause of differences between groups
believed to be equal in achievement.
This second meaning seems to resem
ble what we think of as CIV. In fact,
the Standards (AERA, APA, & NCME,
1999, p. 76) stated, "The term bias in
tests and testing refers to construct
irrelevant components that result in
systematically higher or lower scores
for identifiable groups of examinees."
Neither the Standards nor Cole and
Moss gave us detailed, specific infor
mation about categories of CIV.
Part 3: A Proposed Taxonomy for
Studying CIV and Some Evidence
We begin this section by providing a
simple taxonomy for classifying vari
ables that produce CIV and then pro
vide logical arguments, hypotheses, and
empirical evidence for each source.
The extensiveness of CIV weakens the
validity of interpretations and uses of
test scores in high-stakes accountabil
ity systems. Our purpose is to present
some documentation of sources of CIV
as a serious threat to validity and call
for needed research or point out where
research is being conducted.
The documentation has many ori
gins. While we rely on scholarly reports
as a primary source, secondary sources
include newspaper articles, and other
non-scholarly periodicals. One of the
most productive secondary sources is
the World Wide Web. Another useful
secondary source is an in-house research
report, such as those from the extensive
collection of the Educational Testing
Service. States and testing organiza
tions often provide such information in
their archives and on Web pages. We
believe the mix of evidence for each
source of CIV lends credibility to the
claims made about the extensiveness of
CIV in high-stakes achievement testing.
Occasionally, we suggest viable research
hypotheses to explore a particular source
of CIV and indicate what research needs
to be done.
Table 1 presents the taxonomy in four
broad areas. In the three columns to the
right, we name the type of achievement
19
Table 1. A Taxonomy for the Study of Systematic Errors Associated with CIV
Category Instances Construct Need for Research 1 Type
Uniformity and 1 . Whether or not students Both Adequate Group
types of test get test preparation
preparation 2. The extensiveness of Both Adequate Group
test preparation
3. Unethical test preparation Domain Needed Group
Test development, Test Development
administration, 1. Item quality Domain Needed Group
and scoring 2. Test item format Both Adequate Group
3. Differential item functioning Both Abundant Group
Test Administration
1 . Location of test site Both Needed Group
2. Altering the administration Both Needed Group
3. Participation and exclusion Both Needed Group
4. Computer-based testing Domain Needed Group
5. Calculators in testing Domain Needed Group
Test Scoring
1. Scoring errors Domain Inadequate Group
2. Sanitizing answer sheet Domain Inadequate Group
3. Test form comparability Domain Inadequate Group
4. Rater severity and prompt choice Ability Adequate Group
5. Accuracy of passing scores Both Needed Group
Students 1 The influence of verbal
abilities on test performance Both Needed Ind.
2. Test anxiety, motivation,
and fatigue Both Needed Ind.
3. Accommodations for special
student populations Both Needed Ind.
Cheating 1. Institutional Both Needed Group
2. Individual Both Needed Group
1
Rating Scale: Abundant research exists; research on this topic is adequate; more research is needed. Ind. = individual.

construct associated with the source
of CIV (either domain, ability, or both),
give a subjective appraisal of the ade
quacy of research (abundant, adequate,
or needed), and identify the type of CIV
error (individual or group specific).
Uniformity and Types of
Test Preparation
As noted previously, AERA (2000) has
provided a useful set of guidelines
regarding high-stakes testing that in
cludes advice about alignment of con
tent and cognitive processes, instruc
tion, and assessment. These guidelines
also address opportunity to learn and
the providing of remedial opportuni
ties. After assuring that these guide
lines have been met, we should also
consider the issue of uniform and ethi
cal test preparation. Most testing spe
cialists recommend test preparation
(e.g., Nitko, 2001, chapter 14). Two
guiding principles in test preparation
are (a) no test preparation should vio
late ethical standards of our profession
and (b) increases in test scores should
be correlated with a corresponding in
crease in student learning (Popham,
1991).
There are many aspects to test prepa
ration, including (a) giving advice to
parents, (b) instructing students based
on the curriculum represented by the
test, (c) providing examples of different
test item formats, (d) motivating stu
dents to do their best, and (e) teaching
testwiseness-test-taking strategies that
include efficient time use, error avoid
ance, informed guessing, and deductive
reasoning.
Whether or not students received
test preparation can be a source of CIV.
If some students in a reportable unit of
analysis, such as a school or school dis
trict, have received test preparation
and another group of these students
has not, how does this difference in test
preparation affect the validity of test
score interpretations? Differences in
performance might not be attributable
to sound curriculum design and appro-
priate and effective instruction, but to
the fact that some students received
test preparation and others did not.
A second type of CIV associated with
test preparation is its extensiveness.
There should be some evidence that all
students received uniform test prepara
tion. For instance, Nolen, Haladyna, and
Haas (1992) reported considerable vari
ation in the amount of test preparation
by teachers in one state. Lomax, West,
Harmon, Viator, and Madaus (1995) pro
vided evidence of excessive test prepara
tion with educationally disadvantaged
students.
A third type of CIV involves unethical
types of test preparation. In an article
on preparing for a performance test,
Mehrens, Popham, and Ryan (1998)
offered a set of guidelines that seems ap
plicable to all high-stakes tests. Their
first guideline has to do with criterion
performance being task- or domain spe
cific. Their second guideline is that if
the criterion performance is domain
specific, we should not teach to the ex-

20 Educational Measurement: Issues and Practice

tent that the inference from the test
score to the domain of knowledge or
skills or a cognitive ability is inaccurate.
By focusing test preparation on a subset
of the domain that happens to include
items appearing on the test, the test
scores would be higher than deserved.
If, however, the achievement construct
is an ability, such as writing, it would be
misleading to the consumers of test
score information to teach writing in
only the one genre that is assessed in a
high-stakes test. These examples are in
stances of construct-irrelevant easiness,
and may also be related to construct
under representation. Specifically, un
ethical test preparation might include
(a) developing a curriculum based on
test content instead of following the
state's content standards, (b) present
ing items that are similar or identical in
content and format to those on the test,
or (c) using published instructional
materials that focus on exactly what the
specific test measures or instructional
practices directly aimed at the specific
content of the test, pejoratively called
narrowing the curriculum. All these
practices can increase test scores with
out materially affecting the broader do
main that the test samples. Studies at
test to different problems associated
with test preparation (e.g., Herman &
Golan, 1993; Mehrens & Kaminski,
1989; Nolen et al., 1992). Mehrens and
Kaminski voiced the concern that if test
preparation is unethical, then the public
is led to believe that students achieved
more than they really did achieve.
Although test preparation should be
a standardized practice in all class
rooms, the variations in test preparation
and the use of unethical test prepara
tion constitute sources of CIV. We need
more research on the unethical types of
test preparation and better documenta
tion in all student achievement testing
programs of the role of test preparation.
All students should receive ethical test
preparation and the extensiveness of
this ethical test preparation should be
uniform.
Test Development, Administration,
and Test Scoring
Item quality. While the principles
of writing effective MC test items are
documented in textbooks and research
(Haladyna, Downing, & Rodriguez,
2002), the writing of performance test
items has little scientific or research
basis. What effect, if any, does the use
of poorly crafted items have on item
Spring 2004
and test characteristics? A study by
Downing (2002a) showed that flawed
MC test items were about 7 percentage
points more difficult than non-flawed
items measuring the same content.
Lower achieving students had greater
difficulty with these flawed items than
did higher achieving students. Although
this problem may be more common with
teacher-made tests, it points to item
quality as a potential CIV threat to valid
ity. Most professionally developed, large
scale tests are systematically reviewed
and edited to remove such CIV-inducing
items, but locally developed tests may
have a greater tendency to exhibit this
source of CIV (Downing, 2002b).
Test item format. Although research
suggests that item format may not intro
duce CIV for boys and girls, the threat is
omnipresent that some formats tend
to advantage some groups of students
while other formats do not. For example,
Beller and Gafni (2000) found a gender
by-format interaction in 2 different
years of the International Assessment of
Educational Progress that essentially re
versed itself. Upon closer examination of
this interaction, they found that difficult
performance items were more difficult
for girls than for boys. Thus, under some
circumstances, performance differed as
a function of both the item format and
item difficulty. DeMars (1998) studied
the consequences of performance under
varying item format conditions by gen
der and ethnicity. Gender-by-format
interactions were reported. Rodriguez
(2002) reviewed research on format dif
ferences and found, typically, that format
is not construct irrelevant. However,
Martinez (1999) provided evidence that
occasionally item format can be con
struct irrelevant. Test item format con
tinues to be a fruitful area of study of CIV.
Differential item functioning. An
item shows differential item functioning
(DIF) if the probability of a correct re
sponse depends on group membership
and the groups are assumed to be equal
with respect to the achievement con
struct that the test measures. Standard
7.3 specifically provides for research on
DIF and the taking of appropriate ac
tion to eliminate DIF. Methods for the
study of DIF are numerous. Holland
and Wainer (1993) provided a thorough
treatment of this source of CIV for MC
tests. Penfield and Lam (2000) reviewed
methods of study of DIF with perfor
mance formats and made recommenda-
tions for new methodology for the study
of DIF. Thus, DIF as a source of CIV
applies to both kinds of achievement
constructs, domain-based and abilities.
Research on DIF seems abundant, and
the analysis of test items for DIF seems
to be standard practice in high-quality
testing programs.
Test Administration
Location of the testing site. Adverse
testing conditions may be a source of
CIV. The location in which a test is
given is not necessarily standardized.
The classroom may be the most natural
environment, but when students are
displaced from their classrooms to take
a large-scale test, are the results the
same as those from comparable students
taking the test in their own classroom?
Is there an age-location interaction?
Younger students may perform better in
their own classroom, but as they ma
ture, they might be relocated for testing
without affecting their performance.
Research studies are much needed to
evaluate the effect of specific testing en
vironments on test performance.
Altming the administration. We can
extend the administration time, hop
ing that this extended time will help
students who need more time. Scores
might be higher than comparable groups
where administration time was not ex
tended. To what extent do such non
standardized time extensions occur?
What is the effect of this alteration on
test scores? Nolen et al. (1992) reported
that 8% of elementary and 3% of sec
ondary teachers altered the administra
tion time. Wodtke, Harper, Schommer,
and Brunelli (1989) concluded after
their observation of 10 kindergarten
classrooms that standardized tests were
administered in such a non-standardized
way as to render them incomparable.
They also stated that non-standardized
administration may result in confusion,
anxiety, behavioral resistance, nega
tive attitudes, and other problems that
caused these researchers to wonder why
we even bother to test. Research on test
administration, more frequent monitor
ing, and independent supervision of test
ing are needed to ensure that altered
administration does not contribute CIV.
Parl'icipation and exclusion. The
percentage of participation in a school,
school district, or state can contribute
CIV. By excluding a group of low-scoring
students, we can raise average test
21
scores and by so doing misrepresent the
achievement of a class, school, or even
a school district. Thus, differences
in group performance may not be based
on actual achievement differences but
on who was sampled and excluded.
The problem is more serious when
one considers the recent policy where
schools are labeled asfailing as called
for in the new Elementary and Secondary
Education Act-No Child Left Behind
(http://www.ed.gov/nclb/landing.j html).
A report from theNat'ions Report Card
(NCES, 2002) for NAEP Science shows
participation rates by states range
considerably from national averages.
A National Assessment of Educational
Progress (NAEP) report showed partici
pation rates for students with disabilities
can vary by state from 2.6% to 6.7%. Given
that these students tend to be low scor
ing, greater fluctuations in participation
can contribute sizably to CIV (Grissmer,
Flanagan, Kawata, & Williamson, 2000).
Large disparities in participation rates
for students with disabilities have also
been observed (Erickson, Ysseldyke, &
Thurlow, 1996). They stated that such
variability in participation rates may be
due to the need for accountability and
achieving high test scores. Erickson,
et al. concluded:
Such variability prohibits valid com
parisons between states, and prevents
policy-relevant findings to be drawn
about how students with disabilities
are benefitting from their educational
experiences.
Without a doubt, there is an urgent need
to ensure through policies and proce
dures that standardization exists in test
participation and exclusion. Variations
in these rates directly contribute to CIV
when comparisons are made within any
unit of analysis. Policies that provide
clear guidelines regarding participation
and exclusion coupled with research
and documentation of uniform practices
would help alleviate this threat to valid
interpretations of achievement scores
for schools, school districts, and state.
Computer-based testing. We would
not offer computer-based testing ( CBT)
to any student if we thought the results
would be lower than those obtained by
paper-and-pencil administration. There
is increasing use of CBT, but less fre
quently do we see documented evidence
of the equivalence of CBT and paper
and-pencil testing. Huff and Sireci
(2001) raised several important CIV is-
22
sues related to CBT, such as student
proficiency in taking a computerized
test, computer platform familiarity, user
interface, speededness, and test anxi
ety. They also noted the potential of in
correct estimates of student scores due
to problems with scoring algorithms.
Another potential problem with com
puterized adaptive testing is the heavy
demand on mid-difficult items that pro
vide maximum information. Since these
items are the most frequently used, these
items quickly become overexposed,
which is another source of CIV. The po
tential threats of CIV in the CBT envi
ronment have only begun to be explored
at this time. Indeed, Standard 12.19 of
the Standards (AERA, APA, & NCME,
1999) provides specific warning about
the dangers of CIV related to computer
ized testing. Besides research reports
addressing these problems, technical
reports on such testing programs offer
an opportunity to document that CBT
does not contribute CIV.
Calculators in testing. The role of
calculators in testing has been an ac
tive research topic in item development
and test design (Haladyna, 2004). The
plausible hypothesis is that students
who have calculators have an added ad
vantage over those without calculators
in mathematics tests and in other con
tent that may require calculation. A re
cent report in The Nation :S- Report Card
(NCES, 2002) presented results from
the 2000 NAEP showing an interaction of
grade level with calculator usage. More
frequent use of calculators was corre
lated with lower scores in grade four, but
the opposite was true at grades eight
and 12. Also, some item types seem
more susceptible to better performance
by using calculators. Thus, calculator
usage seems associated with CIV and
the type of item being offered. The use
of calculators would seem to enhance
testing of many types of achievement by
providing a higher fidelity experience..
At the same time, the use of calculators
must not be permitted to increase CIV.
Thus, research is constantly needed to
address each new application involving
calculators or any other technological
innovation.
Test Scoring
Scoring errors. Standard 11.10 reads,
"Test users should be alert to the pos
sibility of scoring errors; they should
arrange for rescoring if individual
scores or aggregated data suggest the
Educational Measurement: Issues and Practice
need for it" (AERA, APA, & NCME, 1999,
p. 115). Indeed, an epidemic of scor
ing errors has arisen throughout the
United States. For example, in Min
nesota, 47,000 students received in
correct scores, leading to serious nega
tive consequences for these students
and to subsequent lawsuits (Henriques
& Steinberg, 2001). More than 20 states
have been affected by scoring errors.
In Arizona, 12,000 students received
incorrect scores due to an error in
the scoring key (Bowman, 2001). In
Washington, 204,000 essays had to be
rescored. Scoring errors or delays also
occurred in California, Florida, Georgia,
Indiana, Mississippi, New York City,
Nevada, North Carolina, South Carolina,
Tennessee, and Wisconsin. In the Edu
cation Week on the Web Archives (2004),
there are 8 listings for scoring error in
cidents. In high-stakes testing, espe
cially where critical pass-fail decisions
are made, we need stronger, more inde
pendent assurance of score accuracy
and additional documentation of extra
scrutiny in scoring.
Sanitizing answer sheets. "Cleaning
up answer sheets" is a practice that is
recommended. For instance the National
Association of Test Directors (2004) pro
vides specific examples of how answer
sheets should be sanitized: "Erase
all stray marks, darken light marks,
and clean up incomplete erasures."
Volunteer parents may be asked to
"clean up" answer sheets before scor
ing. For example, they might make
incomplete erasures more thorough,
since double-marked items are scanned
as incorrect. That some schools and dis
tricts may sanitize answer sheets while
other schools and school districts intro
duce potential CIV. The solution to this
validity threat is to have all answer
sheets sanitized as is recommended by
nearly all test scoring services. This
threat to validity is primed for studies
that explore the frequency of sanitizing
and its consequences on test scores.
Testform comparability. The equat
ing of test forms is a standard practice
in testing programs. There are many
methods for adjusting test scores so that
one test form is no more difficult or easy
than any other test form. However, it is
possible that errors can occur in equat
ing studies. Although research on equat
ing methods is active and important, we
have few mechanisms and little docu
mentation for ensuring that equating is
done accurately. Without adequate qual
ity control and technical reports that
thoroughly document these procedures,
such errors can go undetected and in
troduce CIV.
A related problem is test-score drift.
From the time of first administration of
a new test, test scores tend to increase
for comparable groups as a function of
test age. It would be easy and presump
tuous to conclude that these increases
are due to improved learning. However,
this growth should be validated by com
parison to growth on another standard
ized achievement test, such as the NAEP.
In a study by Linn, Graue, and Sanders
(1990), comparisons of publishers' stan
dardized achievement test results with
NAEP test results suggested that test
scores may depend on non-achievement
factors, such as item exposure. Test
score drift may be related to unethical
test preparation previously discussed or
to other methods used to increase scores
for some examinees in a construct
irrelevant way. The problem of drift im
plies that we have continuing problems
with the accuracy of equating.
Rater severity and prompt choice.
The threat of CIV when equating per
formance tests principally comes from
rater effects, particularly rater severity,
and the issue of prompt choice in writing
assessments. Linn, Baker, and Dunbar
(1991, p. 8) stated, "The training and
calibration of raters is critical in this
regard." The practice of letting students
choose writing prompts assumes that
prompt choice does not contribute to
CIV. Testing specialists have argued that
prompt choice indeed introduces CIV
(Linn, Betebenner, & Wheeler, 1998;
Wainer & Thissen, 1994). Fortunately,
we have an extensive literature on rater
severity and a growing literature on the
problem of prompt choice (Engelhard,
2002). However, this research litera
ture has not yet affected practice of
scoring performance tests and remov
ing CIV due to rater severity, other
rater effects, and prompt choice.
Accuracy of passing scores. The es
tablishment of a passing score for a pass
fail decision or the setting of multiple
points on a test score scale for setting
benchmarks, (as in the NAEP) is usually
a judgmental process involving subject
matter experts (SME). The threat of CIV
is that the SME group, which recom
mends a passing score or set of bench
marks, may not be representative of
Spring 2004
the population of content experts from
which a committee was convened.
Another dimension to this problem
arises when a jurisdiction applies stan
dards in non-uniform ways. States vary
in the way they identify and label "failing
schools." While Michigan scores above
the national average on the NAEP, it re
ported the most failing schools (40%),
while states like Arkansas and Wyoming
reported no failing schools. Thus, we
have a clear indication that CIV is
present in standard setting. Rothstein
(September 8, 2002) reported that the
different standards employed in states
to designate schools as failing has re
sulted in some states engaging in more
busing so that students can change
schools.
These differences in passing stan
dards and standard-setting outcomes
cause us to question the extent to
which the method of standard setting
may contribute CIV. A confounding
factor is the type of achievement test
used in a state and its connectedness
to instruction. Based on the disparities
among states on how failing schools are
identified and labeled, one is inclined to
conclude that standards are indeed very
arbitrary. Research is sorely needed into
the validity of standards. Part of this va
lidity research should examine the con
sistency of judges who set standards
and potential bias introduced by panels
of judges who may differ from the pop
ulation of potential judges. Consistency
of ratings is another issue within this
need for research on standard setting.
Students
Compared with others sources of CIV,
students potentially provide the most se
rious CIV threat to validity. In these in
stances, students comprise individual
specific CIV versus the group-specific
CIV.
The Influence of Verbal Abilities
on Test Perfonnance
Another important source of CIV may be
the demand for verbal abilities needed in
the measurement of another ability. By
verbal abilities, we mean reading, writ
ing, speaking, and listening. Ryan and
Demark (2002) hypothesized four types
of achievement constructs:
1. A construct, such as writing,
where the highest fidelity mea
surement technique is a perfor
mance test.
2. A construct where verbal abilities
are very important in the perfor
mance test. A good example of this
would be an advanced placement
test that requires reading a pas
sage, exercising critical thinking,
and writing ability.
3. A construct where verbal abilities
are used but are not deemed cru
cial in the performance task. This
construct might involve a domain
of knowledge and skills with a
low cognitive demand. Minimal
reading and writing skills may be
needed to perform adequately on
a test of this kind of construct.
4. A construct where there is lit
tle demand for reading and writ
ing abilities. This construct might
involve computation, symbolic
representation associated with
chemistry or physics, or one of the
performing arts.
To what extent do deficits in read
ing interfere with performance on an
achievement test? Research involving
students with limited English proficiency
(LEP) by Abedi, Lord, Hofstetter, and
Baker (2000) showed that vocabulary
has a powerful influence on test perfor
mance. They experimented with NAEP
test items, using simplified vocabulary
to improve students' test performance
by allowing students to use a glossary
and permitting students extra testing
time. Fitzgerald (1995) characterized
students with LEP as slow readers,
whose test performance is obviously im
paired with time limits. Garcia (1991)
showed that LEP students may have
trouble with the familiarity of main
stream American topics, contributing
to their lower performance. Thus, we
have increasing evidence that reading
comprehension plays a vital role in test
performance. Students with LEP are
probably most affected by deficits in
reading comprehension. The measure
ment of achievement in other subject
matter may be hopelessly contaminated
by their deficiency in reading compre
hension. However, this problem is not
limited to LEP students. Non-LEP stu
dents can also have low reading compre
hension. The treatment of this threat to
validity should include a careful con
sideration of the definition of the con
struct and the role that verbal ability
plays in this definition. For the mea
surement of some abilities, reading and
writing at a high level are necessary,
and for other abilities, the demand for
these verbal abilities is less important.
23
With so many students being deficient
in these verbal abilities, the threat of
CIV in these challenging performance
tests suggests that research on this
problem is very much needed.
Test Anxiety, Motivation, and Fatigue
We know that test anxiety can increase
test performance but more generally
lowers test performance. In a meta
analysis of 562 studies, the pattern of
student performance in relation to test
anxiety is unmistakable and conclusive
(Hembree, 1988). Test anxiety can be
pernicious in three ways. First, it af
flicts many test takers. Test anxiety
is estimated to include about 25% of
the general population (Hill & Wigfield,
1984). Second, test anxiety can be exac
erbated or reduced by imposing certain
conditions on the examinees. Hancock
(2001) provided experimental evidence
in a study with college students that an
evaluative threat can increase test anx
iety. Zahar ( 1998) also provided com
plex experimental evidence that dispo
sition to anxiety and the high-stakes
situation contribute to test anxiety.
Third, test anxiety can have conse
quences. For example, Thornton (2001)
reported that teachers in training in
Great Britain have been so intimidated
by teacher testing that they are drop
ping out of their teacher education pro
grams and making alternative plans. As
we can see, not only is test anxiety a
powerful source of CIV, it also affects
students and preservice teachers.
The motivational level of students
may affect test score performance, no
matter the achievement level of the
student. The manifestation of low mo
tivation may be non-compliance with
the test-taking protocol. Students may
seriously underperform, make random
marks on the answer sheets, omit an
swers, or not finish the test. The fre
quency of omitted responses and items
not reached are signals of low motiva
tion and non-compliance. Paris, Lawton,
Turner, and Roth (1991) found that
younger students take large-scale tests
more seriously than older students.
Schools and school districts take very
different approaches to motivating stu
dents to perform on these tests. Tactics
include threats, parties, prizes, awards,
and pep rallies. Whether the tactic is
positive or negative, knowing the ex
tensiveness of these practices and the
degree of the influence of each of these
motivational tactics is important. If the
24
motivational strategies work, then test
scores containCIV, because not all stu
dents or schools receive uniform motiva
tional stimulation from school leaders.
What may be accounting for differences
among schools or school districts might
not be real learning, but more effective
motivational techniques. Although these
motivational techniques are desirable,
these techniques should be uniformly
applied to ensure that motivation does
not become a source ofCIV.
While there is no research to report
about fatigue in testing, we hypothe
size that young students may be more
susceptible to fatigue in long testing
situations than older students, and the
conditions for test administration may
interact with different types of stu
dents. The effects of fatigue are not well
understood or studied, but should we be
concerned with the energy level of stu
dents as they take long, high-stakes
tests? Or is fatigue not a factor in test
performance? A related area of concern
is the extent to which students eat be
fore testing and are allowed breaks and
snacks during a long testing day.
Although we have a promising emerg
ing science of person-fit analysis (Meijer
& Sjitsma, 1995), we do not routinely
study item response patterns of students
to find out if students' response patterns
suggest anxiety, poor motivation, or
fatigue. Some students are plodders
who work slowly and correctly but do
not finish tests in the allotted time.
Studies of examinee fit ought to be
routine in large-scale, high-stakes as
sessments, and evidence supporting
any of these student sources of CIV
should invalidate the scores or cause us
to look for reasons for underperfor
mance other than inadequate learning.
Another aspect of this problem is non
response, items omitted or not reached
(Koretz, Lewis, Skewes-Cox, & Burstein,
1993). The frequency of omitted and
not-reached items should signal poten
tial problems with test anxiety, motiva
tion, fatigue, or timing. Yet, there is
surprisingly little research on threats to
validity.
Unique Problems of
Special Populations
Keeping in mind the admonitions of
Messick (1984) that CIV can contami
nate both interpretations of test scores
and implications we make from knowl
edge of test scores, we confront the
unique problems associated with four
Educational Measurement: Issues and Practice
often co-mingled populations: students
with disabilities, LEP students, stu
dents living in poverty, and students liv
ing in cultural isolation.
Students with Di<;abilities or LEP
Federal guidelines and the new Stand
ards (AERA, APA, & NCME, 1999) give
considerable attention to the necessity
of altering the administration conditions
or the test itself to eliminate a disabil
ity as a source of CIV. As discussed pre
viously, reading comprehension may be
a serious source of CIV. With LEP stu
dents, this type ofCIVis likely to occur.
Chapters 9 and 10 of the Standards
provide considerable discussion and
offer many standards bearing on what
is needed to eliminate CIV when testing
students with disabilities and students
with LEP. Policies of excluding these
students from assessments vary not
only within classrooms and schools, but
also across school districts and states.
Federal law requires that students with
disabilities be included in assessments,
but the law does not explain which ac
commodations are acceptable or spec
ify the criteria for accommodation. If
such accommodations are carried out
uniformly in all school districts and
states, then differences in performance
will not be due to this source of CIV.
Until we have full participation and
more uniformity in the way accommo
dations are offered, comparisons of per
formance of students with disabilities
and LEP among units of analysis such
as classrooms, schools, and states can
not be considered reasonable or validly
interpreted.
Students Living in Cultural Isolation
The measurement of achievement of
students living in culturally homoge
neous, isolated communities can be
affected in many ways. For instance,
students living on Native American
reservations have a variety of charac
teristics that work against effective test
performance (Haladyna, 2002b). These
students, too, need accommodations in
testing and, in some circumstances, al
ternative assessments. The same case
might be made for racial or ethnic com
munities that live in isolation from the
rest of society. While test scores may
traditionally be low for these groups, the
lack or failure of accommodations and
modifications in assessments might ac
count for some of this low performance.
The threat of CIV for these populations
is similar to that of students with disabil-
ities and with LEP. Research on the mo
tivation and preparedness for achieve
ment tests for these special population
students can reveal much about how
they learn and why their performance is
lower than expected. Such studies can
benefit from methods where students
are actually interviewed after the test
or "think-aloud" during a test-taking
session.
Cheating
Any deception committed to misrepre
sent a group's or a student's level of
achievement is cheating. Institutional
cheating is a deception used to mis
represent student achievement for a
class, school, school district, or state.
Some examples of institutional cheat
ing include teachers getting an advance
copy of the test and planning some
lessons based on specific items or pro
viding students "practice" with actual,
secure test items (Popham, 2003); read
ing answers to students during the test
or helping students select correct an
swers; giving hints to correct answers in
the classroom during the test or chang
ing wrong answers to right answers after
the test is given. Unintentionally ex
cluding low-achieving students from a
test may raise the average score for a
unit of analysis, but intentionally ex
cluding such students can also be con
sidered cheating. This problem was a
topic of discussion over a decade ago
(Cannell, 1989). One of the most publi
cized cheating scandals occurred in the
New Orleans public schools (Meitrodt
& Nabonne, 1997). A variety of score
boosting techniques were used that in
cluded exclusion oflow-scoring students,
unethical test preparation, early distri
bution of tests and subsequent teaching
of the test, and unusual test administra
tion arrangements. We have witnessed
outbreaks of cheating in Austin, Texas;
Chicago, Illinois; New York City; Reston,
Virginia; and Rhode Island (Hoff, 2000).
Cizek (1999, pp. 62-69) provided exten
sive evidence of institutional cheating in
schools and school districts. The trust
worthiness of student achievement data
should be questioned, particularlywhen
large gains are observed for a school or
school district, as in the New Orleans
scandal. Like other sources, institutional
cheating in a school or school district
level introduces CIV that is difficult to
detect and document.
Individual cheating is also a veryseri
ous source of CIV. We have many exam-
Spring 2004
ples. The Educational Testing Service
(ETS) conducted an investigation of
student cheating on one of its tests that
led to the arrest of 61 students who were
accused of fraudulent test taking (Li,
2002). This type of cheating involved hir
ing professional test takers as substi
tutes. There have been recent incidents
of electronic devices used to copy test
items or pirate test items. In China,
64 students had their scores invalidated
for copying from other sources in a per
formance test for the Graduate Record
Examination (Taipei Times, 2002). Two
students from Columbia University were
arrested for using high-tech transmit
ters and walkie-talkies to cheat on
the Graduate Record Examination
(Chronicle ofHigher Education, 2002).
This small sample of incidents shows
that CIV from individual student cheat
ing is a worldwide problem.
The volume by Cizek (1999) is a
milestone in the study and documen
tation of cheating as a source of CIV.
Nonetheless, more research is needed
into the motivation for institutional
cheating and its extensiveness partic
ularly within high-stakes accountabil
ity testing programs.
Conclusion
Every interpretation or use of an
achievement test score in a high-stakes
environment is vulnerable to many va
lidity threats, such as inadequate con
struct definition, construct under repre
sentation, illogical reasoning regarding
the causes of student learning, negative
consequences of test score uses, and
low reliability of test scores. CIV is one
of these threats to validity. We have
shown that CIV has many sources, and
evidence has been presented of its ex
tensiveness. We have argued that CIV
needs increased attention, especially in
high-stakes testing programs.
Researchers should systematically
address understudied sources of CIV.
Identification and assessment of the se
riousness of each source in high-stakes
testing programs is the first step. This
research should build a robust litera
ture that provides a clear picture of the
seriousness of this threat to validity.
Documentation is needed to assure
the public that each specific threat to
validity is not serious for the particular
score interpretations proposed. In in
stances where this documentation
shows that a threat is serious, we need
to take appropriate action to study the
CIV threat, and then eliminate or re
duce it. A primary type of documenta
tion is the annual technical report. This
report should contain references to va
lidity evidence and its relationship to
the appropriate standards (AERA, APA,
& NCME, 1999) and detail specific CIV
threats and their management.
Sponsors of state and local high
stakes achievement testing programs
and test companies that develop these
tests should work together to consider
the seriousness of all threats to validity,
including CIV. A shared responsibility
and honest examination of these CIV
threats are likely to improve the effec
tiveness and increase the validity evi
dence for these testing programs.
References
Abedi, J., Lord, C., Hofstetter, C., & Baker, E.
(2000). Impact of accommodation strate
gies on English language learners' test
performance. Educational Measurement:
Issues and Practice, 19(3), 16-26.
Adams, R., Wu, M., & Wilson, M. ( 1998). Con
Quest [ Computer Program J. Camberwell,
Victoria, Australia: Australian Council
for Educational Research.
American Educational Research Asso
ciation (2000). Position statement of the
American Educational Research Asso
ciation concerning high-stakes testing
in pre K-12 education. Educational Re
searcher, 29(8), 24-25.
American Educational Research Associa
tion, American Psychological Associa
tion, & National Council on Measurement
in Education. (1999). Standards for
educational and psychological testing.
Washington, DC: American Educational
Research Association.
Andrich, D., Lyne, A., Sheridan, B, & Luo, G.
(1997). RUMM 2010: A windows pro
gramfor Rasch unidimensional models
for measurement I Computer Progi:am].
Duncraig, Western Australia: Murdoch
University-Social Measurement Labora
tory.
Beller, M., & Gafni, N. (2000). Can item for
mat (multiple-choice vs. open-ended)
account for gender differences in mathe
matics achievement? Sex roles, 42(),
1-22.
Bowman, D. H. (2001, December 12).
Arizona reports scoring errors on state
exam. Education Week on the Web. Re
trieved March 17, 2004, from http://www.
edweek.org/ew/newstory.cfm?slug=l5ca
ps.h21
Brennan, R. L. (2001). Generalizability
theory. New York: Springer.
Cannell, J. J. (1989). How public educa
tors cheat on standardized achieve
ment tests. Albuquerque, NM: Friends
for Education.
25
Chronicle of Higher Education (2002, Nov
ember 21). Two students arrested for
alleged high tech cheating on the GRE.
Retrieved March 17, 2004, from http://
chronicle.com/free/2002/11/2002112102t.
htm
Cizek, G. J. (1999). Cheating on tests: How
to do it, detect it, and prevent it. Mahwah,
NJ: Erlbaum.
Cole, N. S., & Moss, P. A. (1989). Bias in test
use. In R. L. Linn (Ed.),Educationalmea
surement (3rd ed., pp. 201-220). New
York: American Council on Education and
Macmillan.
Cronbach, L. J. (1988). Five perspectives on
validity argument. In H. Wainer & H. I.
Braun (Eds.), Test validity (pp. 3-17).
Hillsdale, NJ: Erlbaum.
Cronbach, L. J., & Meehl, P. E. (1955).
Construct validity in psychological tests.
Psychological Bulletin, 52, 281-302.
Crooks, T. J., Kane, M. T., & Cohen, A. S.
(1996). Threats to valid use ofassessment.
Assessment in Education, 3(3), 265-285.
DeMars, C. E. (1998). Gender differences in
mathematics and science on a high school
proficiency exam: The role of response for
mat.Applied Measurement in Education,
11 (3), 279-299.
Downing, S. M. (2002a). Construct-irrele
vant variance and flawed test questions:
Do multiple-choice item writing princi
ples make any difference? Academic
Medicine, 77(10), S103-104.
Downing, S. M. (2002b). Threats to the
validity of locally developed multiple
choice tests in medical education:
Construct-irrelevant variance and con
struct underrepresentation. Advances in
Health Sciences Education, 7, 235-241.
Downing, S. M., & Haladyna, T. M. (1996).
Model for evaluating high-stakes testing
programs: Why the fox should not guard
the chicken coop. Educational Mea
surement: Issues and Practice, 15 (1),
5-12.
Education Week on the Web Archives.
(2004). Scoring errors. Retrieved March
17, 2004, from http://www.edweek.org/
search/
Engelhard, Jr., G. E. (2002). Monitoring
raters in performance assessments. In G.
Tindal & T. M. Haladyna (Eds.), Large
scale (J};Sessrnentfor all student: Validity,
technical adequacy, and implementation
(pp. 261-287). Mahwah, NJ: Erlbaum.
Erickson,R. N., Ysseldyke, J. E., & Thurlow,
M. L. (1996). Neglected numerators,
drifting denominators, and fractured
fractions: Determining participation rates
for students with disabilities in state
wide assessment programs (Synthesis
Report No. 23). Minneapolis: University
of Minnesota, National Center on Educa
tional Outcomes. Retrieved March 17,
2004, from http:/!education. umn.edu/
NCEO/OnlinePubs/Synthesis23.html
Fitzgerald, J. (1995). English-as-a-second
language learners' cognitive reading
26
processes: A review of research in the
United States. Review of Educational
Research, 65, 145-190.
Garcia, G. E. (1991). Factors influencing
the English reading test performance
of Spanish-speaking Hispanic children.
Reading Research Quarterly, 26, 371-391.
Grissmer, D. W., Flanagan, A. E., Kawata,
J. H., & Williamson, S. (2000). Improving
student achievement: What state NAEF
t;est scores tell us. Santa Monica, CA: Rand
Corporation.
Haertel, E., & Calfee, R. (1983). School
achievement: Thinking about what to test.
Journal of Educational Measurement,
20(2), 119-131.
Haladyna, T. M. (2002a). Supporting docu
mentation: Assuring more valid test score
interpretations and uses. In G. Tindal &
T. M. Haladyna (Eds.), Large-scale (1};
sessmentfor all students: Validity, tech
nical adequacy, and implementation
(pp. 89-108). Mahwah, NJ: Erlbaum.
Haladyna, T. M. (2002b). Standard'ized
achievement testing: Validity and ac
countability. Boston: Allyn & Bacon.
Haladyna, T. M. (2004). Developing and
validating multiple-choice test items
(3rd ed.). Mahwah, NJ: Erlbaum.
Haladyna, T. M.,Downing, S. M., & Rodriguez,
M. C. (2002). A review of multiple-choice
item-writing guidelines for classroom
assessment. Applied Measurement in
Education, 15(3), 309-334.
Hancock, D.R. (2001). Effects of test anxi
ety and evaluative threat on students'
achievement. Journal of Educational
Research, 94(5), 284-290.
Hembree, R. (1988). Correlates, causes,
effects, and treatment of test anxiety.
Review of Educational Research, 58,
47-77.
Henriques, D. B., & Steinberg, J. (2001,
May 20). Right answer, wrong score:
Test flaws take toll. New York Times.
Retrieved March 17, 2004, from http:
//www.nytimes.com/2001/05/20/business/
20EXAM.html?ex=1079672400&en=fbce
f3a39c75ddbd&ei=5070
Herman, J., & Golan, S. (1993). The ef
fects of testing on teaching and schools.
Educational Measurement: Issues and
Practice, 12, 20-25, 41.
Hill, K., & Wigfield, A. (1984). Test anxiety:
A major educational problem and what
can be done about it. The Elementary
School Journal, 85, 105-126.
Hoff, D. J. (2000, June 21) As stakes rise
definition of cheating blurs. Education
Week, 19(41), 1.
Holland, P. W., & Wainer, H. (Eds.). (1993).
Dlfferential item functioning. Mahwah,
NJ: Erlbaum.
Huff,K. L.,&Sireci,S. (2001). Validity issues
in computer-based testing. Educational
Measurement: Issues and Practices,
20(3), 16-25.
Kane, M. T. (1992). An argument-based ap
proach to validity. Psychological Bulletin,
112, 527-535.
Educational Measurement: Issues and Practice
Kane, M. T. (2002). Validating high-stakes
testing programs. Educational Measure
ment: Issues and Practices, 21 (1), 31-41.
Koretz, D., Lewis, E., Skewes-Cox, T., &
Burstein, L. (1993). Omitted and not
reached items in mathematics in the
1990 National Assessment ofEducational
Progress (CRE: Technical Report 347).
Los Angeles, CA: Center for Research
on Evaluation, Standards, and Student
Testing.
Li, K. (2003, March 5). Fraudelent TOEFL
takers face possible deportation. Daily
Princetonian. Retrieved on March 17,
2004, from http://www.dailyprinceton
ian.com/archives/2003/03/05/news/7516.
shtml
Linacre, J. M., & Wright, B. D. (2004).
FACETS: Computer program for many
faceted Rasch measurement. [ Computer
Software]. Chicago: MESA Press.
Linn, R. L. (2002). Validation of the uses
and interpretations of results of state as
sessment and accountability systems. In
G. Tindal & T. Haladyna (Eds.), Large
scale assessment programs jor all stu
dents: Development, implementation,
and analy. Mahwah, NJ: Erlbaum.
Linn, R. L., Baker, E. L., & Dunbar, S. B.
(1991). Complex performance assess
ment: Expectations and validation cri
teria. Educational Researcher, 20 ( 8),
15-21.
Linn, R. L., Betebenner, D. W., & Wheeler,
K. S. (1998). Problem choice by test tak
ers: Implications for comparabuity and
construct validity (CSE Technical Report
485). Boulder: University of Colorado at
Boulder, Center for Research on Eval
uation, Standards and Student Testing.
Linn, R. L., Graue, M. E., & Sanders, N. M.
(1990). Comparing state and district re
sults to national norms: The validity of
claims that "Everyone is above average."
Educational Measurement: Issues and
Practice, 9(3), 5-14.
Lohman, D. F. (1993). Teaching and testing
to develop fluid abilities. Educational
Researcher, 22(1), 12-23.
Lomax, R. G., West, M. M., Harmon, M. C.,
Viator, K. A.,& Madaus, G. F. (1995). The
impact of mandated standardized testing
on minority students. Journal of Negro
Education, 64, 171-185.
Lord, F. M., & Novick, M. R. (1968).
Statistical theories of mental test scores.
Reading, MA: Addison Wesley.
Martinez, M. E. (1999). Cognition and the
questions of test item format. Educational
Psychology, 34( 4), 207-218.
Mehrens, W. A., & Kaminski, J. (1989).
Methods for improving a standardized
test scores: Fruitful, fruitless, or fraudu
lent? Educational Measurement: Issues
and Practices, 8, 14-22.
Mehrens, W. A., Popham, W. J., & Ryan,
J. R. (1998). How to prepare students for
performance assessments. Educational
Measurement: Issues and Practice, 17,
18-22.
Meijer, R.R., & Sjitsma, K. (1995). Detection
of aberrant item score patterns: A re
view of recent developments. Applied
Measurement inEducation, 8(3), 261-
272.
Meitrodt, J., & Nabonne, R. ( 1997). Scores,
testing practices raise suspicions of ex
perts. New Orleans Times-Picayune
Special Report. Retrieved March 17,
2004, from http://www.nola.com/speced/
toogood/rnain.html
Messick, S. (1984). The psychology of edu
cational measurement. Journal ofEdu
catwnal Measurement, 21, 215-237.
Messick, S. (1989). Validity. In R. L.
Linn (Ed.), Educational measurement
(3rd ed., pp. 13-104). New York: American
Council on Education and Macmillan.
Messick, S. (1995a). Validity of psychological
assessment: Validation of inferences from
persons' responses and performances as
scientific inquiry into score meaning.
American Psychologist, 50, 741-749.
Messick, S. (1995b). Standards of validity
and the validity of standards in perfor
mance assessment. Educational Measure
ment: Issues and Practice, 14 ( 4), 5-8.
Mislevy, R. J. (1996). Test theory recon
ceived. Journal of Educational Mea
surement, 33(4), 379-416.
Muraki, E., &Bock, R. D. (2003). PARSCALE:
IRT based test scoring and item analy
sis for graded open-ended exercises and
performance tests. Version 3.1 plus.
[ Computer Software]. Chicago: Scientific
Software.
National Association of Test Directors
(2004). Cleaning up answer sheets. 7.
Retrieved March 17, 2004, from http://
Spring 2004
www.natd.org/Case_3_Cherry_Creek_
part_E.PDF
National Center of Educational Statis
tics. (2002). The nation's report card:
Mathematics highlights (NCES 2001-
518). Washington, DC: U.S. Department
of Education: Office of Educational Re
search and Improvement. Retrieved
March 17, 2004, from (http://nces.ed.gov/
nationsreportcard/)
Nitko, A. J. (2001). Educational assessment
ofstudents (3rd ed.). Upper Saddle River:
NJ: Merrill/Prentice Hall.
Nolen, S. B., Haladyna, T. M., & Haas, N. S.
(1992). Uses and abuses of achievement
test scores.Educational Measurement:
Issues and Practices, 11, 9-15.
Paris, S. G., Lawton, T. A., Turner, J. C., &
Roth, J. L. (1991). A developmental per
spective on standardized achievement
testing.Educational Researcher, 20 (1) ,
2-7.
Penfield, R. D., & Lam, R. C. M. (2000).
Assessing differential item functioning in
perfo rmance assessment: Review and
recommendations. Educatwnal Measure
ment: Issues and Practice, 19(5), 5-15.
Popham, W. J. (1991). Appropriateness of
teachers' test-preparation practices. Edu
cational Measurement: Issues and Prac
tices, 10(4), 12-16.
Popham, W. J. (2003). Seeking redemption
for our psychometric sins. Educational
Measurement: Issues and Practices,
22(1), 45--48.
Rodriguez, M. (2002). Choosing an item for
mat. InG. Tindal&T. M. Haladyna (Eds.),
Large-scale assessment programs for all
students: Validity, technical adequacy,
and implementatwn 'issues (pp. 211-
229). Mahwah, NJ: Erlbaum.
Rothstein, R. (2002, September 18). How
U.S. punishes states with higher stan
dards. The New York Times. Retrieved
March 17, 2004, from httpJ/www.ny
times.com/2002/09/18
Ryan, J.M., & Demark, S. (2002). Variation
in achievement scores related to gender,
item format and content area tested. In
G. A. Tindal & T. M. Haladyna (Eds.),
Large-scale assessment programs for all
students: Development, implementation,
and analysis (pp. 67-88). Mahwah, NJ:
Erlbaum.
Sternberg, R. J. (1998). Abilities are forms
of developing expertise. Educational
Researcher, 27(3), 11-20.
Taipei Times (2002, August 9). GRE cheat
ing probe uncovers Asian internet sites.
Retrieved March 17, 2004, from http://
www.taipeitimes.com/News/front/archiv
es/2002/08/09/159537
Thornton, K. (2001, May 4). Test pressures
forces trainees to quit. The Times Edu
catwnal Supplement, N. 4427.
Wainer, H., & Thissen, D. (1994). On exami
nee choice in educational testing. &view
of EducationalResearch, 64(1), 159-195.
Wodtke, K. H., Harper, F., Schommer, M., &
Brunelli, P. (1989). How standardized is
school testing? An exploratory study of
standardized group testing in kinder
garten. Educational Evaluation and
Policy Analysis, 11 (3), 223-235.
Zohar, D. (1998) An additive model of test
anxiety: Role of exam-specific expecta
tions. Journal ofEducational Psychology,
90(2), 330-340.
27