Alonzo2009 PDF

Developing and Assessing
a Force and Motion Learning

Progression
ALICIA C. ALONZO
Department of Teaching and Learning, University of Iowa, Iowa City,
IA 52242-1529, USA
JEFFREY T. STEEDLE
School of Education, Stanford University, Stanford, CA 94305, USA
Received 14 December 2007; revised 20 May 2008, 31 May 2008; accepted 6 June 2008
DOI 10.1002/sce.20303
Published online 2 September 2008 in Wiley InterScience (www.interscience.wiley.com).
ABSTRACT: Learning progressions are ordered descriptions of students’ understanding

of a given concept. In this paper, we describe the iterative process of developing a force
and motion learning progression and associated assessment items. We report on a pair of
studies designed to explore the diagnosis of students’ learning progression levels. First,
we compare the use of ordered multiple-choice (OMC) and open-ended (OE) items for
assessing students relative to the learning progression. OMC items appear to provide
more precise diagnoses of students’ learning progression levels and to be more valid,
eliciting students’ conceptions more similarly to cognitive interviews. Second, we explore
evidence bearing on two challenges concerning reliability and validity of level diagnoses:
the consistency with which students respond to items set in different contexts and the ways
in which students interpret and use language in responding to items. As predicted, students
do not respond consistently to similar problems set in different contexts. Although the
language used in OMC items generally seems to reflect student thinking, misinterpretation
of the language in items may lead to inaccurate diagnoses for a subset of students. Both
issues are less problematic for classroom applications than for use of learning progressions
in large-scale testing. C 2008 Wiley Periodicals, Inc. Sci Ed 93:389 – 421, 2009
Earlier versions of this paper were presented at the 2007 annual meeting of the American Educational
Research Association (Chicago, IL) and at the 2007 biennial meeting of the European Science Education
Research Association (Malmö, Sweden).
Correspondence to: Alicia C. Alonzo; e-mail: alicia-alonzo@uiowa.edu
Jeffrey T. Steedle’s current affiliation is Council for Aid to Education, New York, NY 10016.
The full set of force and motion items are available by contacting the first author.
Any opinions, findings, conclusions, or recommendations expressed in this paper are those of the
authors. They do not necessarily represent the official views, opinions, or policy of the National Science
Foundation.

C 2008 Wiley Periodicals, Inc.
390 ALONZO AND STEEDLE
INTRODUCTION
Learning progressions are ordered descriptions of students’ understanding of a given
concept. They represent a promising framework for developing meaningful assessments,
allowing both large-scale and classroom-based assessments to be grounded in models of
how understanding develops in a given domain. The use of cognitive models to inform
assessment is one of the main recommendations of the National Research Council (NRC)
volume Knowing What Students Know (NRC, 2001). In addition, learning progressions are
explicitly endorsed by the NRC in Systems for State Science Assessments (NRC, 2006)
and Taking Science to School (NRC, 2007) and by the National Assessment Governing
Board (NAGB) in the framework and item specifications for the 2009 National Assessment
of Educational Progress (NAEP) science test (NAGB, 2006a, 2006b). In the classroom,
learning progressions may help teachers use knowledge of students’ understanding of a
given concept to make instructional decisions.
Learning progressions grew out of earlier work by Mark Wilson and colleagues (e.g.,
Roberts, Wilson, & Draney, 1997) with “construct maps” whose defining features are (1) “a
coherent and substantive definition for the content of the construct” and (2) “an underlying
continuum” (Wilson, 2005, p. 26). Learning progressions rely upon cognitive science
research on how students learn a particular concept to describe a path (or set of paths)
that students might take in moving from novice to expert understanding. This is consistent
with the NRC’s call for the “central role of a model of cognition and learning” (2001,
p. 6) in the assessment design process. Learning progressions are typically based upon
research describing students’ ideas at various ages (e.g., Driver, Guesne, & Tiberghien,
1985), but learning progressions must be “partly hypothetical or inferential, since long-
term longitudinal accounts of learning by individual students do not exist” (NAGB, 2006a,
p. 152). Learning is complex, and it is likely that students will take different pathways
in coming to understand a given idea, as instruction, prior knowledge, and individual
differences play out in the learning process (NAGB, 2006a). The nature of learning and our
limited knowledge of exactly how student understanding develops pose special challenges
for the development of learning progressions.
To validate learning progressions and to use them as assessment tools in both large-scale
and classroom contexts, items must be developed to efficiently assess students’ learning
progression levels. The cognitive science research that provides the foundation for learning
progressions typically relies upon clinical interviews to assess student understanding (e.g.,
Bell, Osborne, & Tasker, 1985), but this approach is not feasible in nonresearch settings.
Briggs, Alonzo, Schwab, and Wilson (2006) demonstrated the use of “ordered multiple-
choice” (OMC) items to assess students’ levels on a learning progression. In such items,
each response option corresponds to a level of the learning progression so that a student’s
responses to a set of items could be used to characterize his or her understanding of the
target concept. Both this learning-progression-specific format and other more traditional
item formats pose unique development challenges. It is difficult to write items that provide
opportunities for students to respond at multiple levels of a learning progression (Anderson,
Alonzo, Smith, & Wilson, 2007). Open-ended (OE) items must be carefully crafted to elicit
complete responses without telling students what should be included. Multiple-choice
items must be written in such a way that the highest level is not indicated by the use of
“science-y” terminology not present in lower level options. In addition, Steedle (2006)
found that students performed differently on multiple-choice and short-answer items tar-
geting understanding of the same underlying progression, which points to a need for better
understanding of the affordances of different item types for assessing students’ learning
progression levels.
Science Education
FORCE AND MOTION LEARNING PROGRESSION 391
Thus, while learning progressions hold great promise, they also pose a number of chal-
lenges. This paper proposes a learning progression for force and motion, focusing on the
case of one-dimensional forces and resulting motion. Through this example, we seek to
illuminate the process of developing a learning progression and to explore some of the
challenges involved in designing items to elicit evidence about students’ learning progres-
sion levels, not to discourage the use of learning progressions, but rather to encourage more
reflective conversations about this approach. In particular, we describe the development of
the learning progression and address two questions about the design of items to diagnose
students’ learning progression levels:
1. What are the advantages and disadvantages of the OMC item format for diagnosing
students’ understanding with respect to the force and motion learning progression?
2. What are some challenges associated with developing items to diagnose students’
understanding with respect to the force and motion learning progression?
Before turning to the specifics of the learning progression and its associated items, the
paper starts with an orientation to more general ideas about learning progressions, which
will be illustrated with the specific example of the force and motion learning progression.
The methods section that follows describes (1) the process of developing and revising
the force and motion learning progression and associated items and (2) the evaluation of
attempts to assess students’ levels on the learning progression in two separate, but related,
studies. Results of these two studies are described in the next section, followed by some
concluding comments, which draw implications of this work for the development of other
learning progressions and associated items.
CONSIDERATION OF LEARNING PROGRESSIONS

AND THEIR DEVELOPMENT
What Do We Mean by “Learning Progressions”?
Learning progressions are rapidly gaining popularity in the science education community,
but it is clear that a shared definition of this concept is lacking. The “learning progression”
label has been applied to a variety of descriptions, which vary in terms of what is being
portrayed—whether content, conceptions, or performances—and in their breadth and grain
size. Indeed, in describing the use of learning progressions for the 2009 NAEP science
assessment, the item specifications document (NAGB, 2006a) includes examples of quite
different types of learning progressions, leaving some ambiguity as to what is expected.
In Taking Science to School, the NRC (2007) defines learning progressions as descriptions
of thinking “over a broad span of time (e.g., 6–8 years)” (p. 219). The learning progres-
sions commissioned by the National Academy of Sciences—matter and atomic-molecular
theory (Smith, Wiser, Anderson, & Krajcik, 2006)1 and evolution (Catley, Lehrer, & Reiser,
2004)—are good examples of learning progressions with this large scope. These learning
progressions are necessarily complicated, including constellations of concepts that con-
tribute to mastery of a single big idea.
1
The original commissioned paper is Smith, C., Wiser, M., Anderson, C. W., & Krajcik, J. (2004,
October). Implications of research on children’s learning for assessment: Matter and atomic molecu-
lar theory. Paper commissioned by the National Academy of Sciences Committee on Test Design for
K-12 Science Achievement. Retrieved March 23, 2007, from http://www7.nationalacademies.org/bota/
Big%20Idea%20Team %20AMT.pdf.
Science Education
Broad learning progressions may be thought to comprise more fine-grained descriptions

of student thinking—over either a smaller span of time or a smaller slice of content—such
that a broad learning progression may consist of a number of smaller progressions. For
example, the floating and sinking learning progression proposed by the Stanford Education
Assessment Laboratory (2003) was designed to track student understanding over the course
of a single middle school curriculum unit. However, this learning progression might be
thought to detail student thinking over a portion of a larger learning progression, which
could extend to describe younger students’ understanding of the prerequisite concepts of
mass/weight and volume, as well as older students’ understanding of the molecular basis of
density. As another example, the matter and atomic-molecular theory learning progression
(Smith et al., 2006) contains six big ideas. One might provide a more detailed accounting
of students’ development with respect to each of these big ideas through smaller learning
progressions. These smaller, more detailed learning progressions have much in common
with Wilson’s (2005) notion of construct maps. Wilson (2008) recently proposed several
potential relationships between construct maps and learning progressions, including a one-
to-one mapping (i.e., construct maps represent learning progressions), as well as models
in which a single learning progression comprises multiple construct maps. Regardless
of how they are labeled, these smaller learning progressions (or construct maps) may
ultimately provide the detail needed for teachers to track student thinking over the course
of instructional units.
In some ways, variations in breadth and grain size are reflected in differences in what
is considered to progress in various instantiations of “learning progressions.” Consistent
with the NRC (2007), most learning progressions provide descriptions of student thinking.
However, the larger the scope of a learning progression, the less feasible it may be to
provide detailed descriptions of student thinking. There are certainly examples in the
literature of “learning progressions,” which define progressions of content, rather than of
student thinking (e.g., Roseman, Caldwell, Gogos, & Kurth, 2006), as well as definitions of
learning progressions (e.g., Popham, 2007) supporting this practice. Recently, Smith et al.
(2006) called for learning progressions to contain not just descriptions of student thinking
but also the ways in which that understanding is enacted in “learning performances” (p. 9).
And recently proposed learning progressions focus on students’ development of particular
scientific skills (e.g., modeling) and, thus, include descriptions of students’ performance of
these skills (Lehrer et al., 2008; Reiser et al., 2008).
The learning progression proposed in this paper focuses on student thinking over a
relatively short period—what might be expected as students move through a curriculum unit
representing their first formal study of force and motion. This is appropriate for its intended
purpose of diagnosing student understanding of this construct to support instructional
decisions. As will be described below, at each level of the learning progression, we attempt
to lay out the thinking that students at that level could be expected to exhibit, including
both the correct ideas that can be carried to the next level and the misconceptions that will
need to be revised before students can reach the next level.
How Can Learning Progressions Be Developed?

In part because we lack longitudinal data tracking how student understanding of sci-
ence concepts evolves in different curricular environments, the development of a learning
progression is necessarily an iterative process. We learn more about patterns in students’
thinking as we attempt to assess their understanding, which in turn influences the learning
progression itself. Thus, development of the learning progression is part of an iterative pro-
cess of revision, involving changes to both the learning progression and its associated items.
Science Education
The process of developing a learning progression is detailed by Briggs et al. (2006). To

inform the development of learning progressions in other areas of the science curriculum,
this process will be laid out generally here and then illustrated through discussion of the
force and motion learning progression later in this paper.
The development process starts with a definition of the construct of the learning progres-
sion. For example, standards documents or cognitive science research may be used to define
what students can be expected to understand about a scientific concept. This understanding
defines the top level of the learning progression. Lower levels of the learning progression
can be defined through a review of the research literature on students’ understanding of
the identified construct, including “misconceptions,” as well as productive ideas that may
support development of full understanding. The definition of levels involves both laying
out an ordered progression for the ideas identified in the research literature and grouping
similar sets of ideas together into a single level. It is important to note that this is typically
a logical, rather than an evidence-driven process. Depending upon the construct of the
learning progression, there may be more or less information about how the ideas at a given
level “hang together,” but research has typically focused on describing single ideas, rather
than the relationships between them. Thus, the learning progression represents a hypothesis
about student thinking, rather than a description. As such, it expresses a current idea about
how student understanding develops, which can—and should—be revised in response to
new information about student thinking.
Starting with a preliminary learning progression, sets of items can be developed to assess
students’ levels on the progression. Data obtained from administering these items to students
can inform not only item revision but also consideration of the learning progression itself.
Student thinking, as revealed by their responses to the items, may lead to new ideas being
added to the learning progression or even to a reorganization of the ideas it contains. This
iterative process will be illustrated below, using the force and motion learning progression.
What Challenges Might We Expect in Assessing Students’ Levels

on a Learning Progression?
Cognitive science research has attempted not only to uncover students’ ideas but also
to characterize the nature of those ideas. These findings impact the development and use
of learning progressions. In particular, two areas of cognitive science research seem par-
ticularly important to the force and motion learning progression. First is the consistency
with which students respond to problem situations in which the same underlying principles
apply. Second is the role of language, both language students use to describe their under-
standing and the ways in which they interpret the language used in items designed to assess
that understanding. We will explore each of these areas briefly below.
Consistency of Student Responses. Students’ inconsistency in approaching physics

problems addressing the same underlying principles is well documented. Chi, Feltovich,
and Glaser (1981) demonstrated that experts tend to categorize problems according to
underlying principles, but novices tend to use surface features of the problems instead.
Thus, problems that look very similar to a physicist seem completely different from the
point of view of a novice student. Finegold and Gorsky (1991) asked students about the
forces acting on six different objects. They found that, while it was possible to identify 11
frameworks used by more than two thirds of college and advanced high school students
to describe forces acting on objects in motion, for objects at rest, “specific rules exist
for specific situations: a force law for objects at rest on surfaces, for objects suspended
Science Education
from strings, etc.” (p. 103). Halloun and Hestenes (1985) found that while college students
applied some mixture of three theories of motion, almost none applied the same theory
across different problem situations.
Constructing a learning progression and assessing students’ levels on that continuum
is even more complicated than assessing understanding of a single idea because learning
progressions typically group together related ideas to form a single level describing student
understanding. There is even less reason to expect consistency across items addressing
different concepts, even those united by the same underlying physics principle. For example,
while Newton’s first law can account for the behavior of both objects at rest and objects in
motion, Finegold and Gorsky (1991) found little relationship between students’ frameworks
for these two conditions. When they attempted to identify frameworks that could account
for students’ descriptions of the forces acting on both objects at rest and in motion, there
were “almost as many models as there were students” (p. 109).
Students’ Use of Language. Another challenge is the potential lack of shared under-
standing of scientific words that are integral to the concepts under consideration. For
example, “force” is a necessary word for many item contexts related to the force and mo-
tion learning progression. Students may hold a different interpretation of this word from the
very specific meaning used by physicists, in part because the “scientific” word also has a
range of everyday meanings (Halloun & Hestenes, 1985). Ioannides and Vosniadou (2001)
identified four well-defined, internally consistent meanings for the word “force,” held by
children from 4 to 15 years of age. None of these is consistent with the physicist’s meaning.
In their study, students’ meanings for the word “force” varied with age. Younger children
viewed force as an internal property of all objects, whereas older children interpreted this
word to mean “an internal property of objects that move, as the result of an agent pushing or
pulling them” (p. 2). They termed the latter conception “acquired force.” Thus, it seems that
students’ ideas about the word “force” are likely changing along with their ideas about the
relationship between force and motion. Complicating the assessment of students’ learning
progression levels is the possibility that their conceptions of the word “force” may influence
the way in which they interpret and respond to items using this word.
METHODS
Development and Revision of the Force and Motion Learning
Progression and Associated Items
We now describe the evolution of the force and motion learning progression to its current
form, focusing on the iterative nature of its development.
Development of the Preliminary Force and Motion Learning Progression and

Associated Items. The preliminary force and motion learning progression was devel-
oped using the process summarized above. The top level of the learning progression was
defined by the understanding of force and motion expected of eighth-grade students in
the National Science Education Standards (NRC, 1996) and the American Association
for the Advancement of Science (AAAS) Benchmarks for Science Literacy (1993). These
understandings are similar and can be summarized by two points from the National Science
Education Standards (NSES):
• An object that is not being subjected to a force will continue to move at a constant
speed and in a straight line.
Science Education
TABLE 1
Common Student Conceptions About Force and Motion
Conception References Documenting Conception
Forces are caused by living or active things Halloun and Hestenes (1985) and Minstrell
(n.d.)
Forces can only be caused by something Halloun and Hestenes (1985) and Minstrell
touching an object (n.d.)
Weight, motion, activity, and strength are Ioannides and Vosniadou (2001)
important in determining an object’s
force
Force is a property of a single object, Reiner, Slotta, Chi, and Resnick (2000)
rather than a feature of the interaction
between two objects
A moving object has a force within it, Reiner et al. (2000), Halloun and Hestenes
which keeps it going (1985), Ioannides and Vosniadou (2001),
and McCloskey (1983)
A moving object stops when its force is Halloun and Hestenes (1985), McCloskey
used up, which happens naturally over (1983), and Trumper and Gorsky (1996)
time
If there is motion, there is a force acting Clement (1982) and Ioannides and
Vosniadou (2001)
If there is no motion, then there is no force Halloun and Hestenes (1985), Gilbert and
acting Watts (1983), and Minstrell (n.d.)
A “preventer” force is required to keep an Minstrell (n.d.)
object at rest
Forces cause motion in the direction of diSessa (1983)
the force
If there is a force, then there is motion Champagne, Klopfer, and Anderson (1980)
If there is not a force, either there is no Champagne et al. (1980)
motion or the motion is decreasing
When an object is moving, there is a (net) Gilbert and Watts (1983), Minstrell (n.d.), and
force in the direction of its motion Trumper and Gorsky (1996)
Motion is proportional to the force acting Champagne et al. (1980)
If there is no net force, an object slows Halloun and Hestenes (1985)
down
A constant speed results from a constant diSessa (1983), Champagne et al. (1980),
(net) force Gilbert and Watts (1983), Halloun and
Hestenes (1985), and Minstrell (n.d.)
Constant acceleration requires a Champagne et al. (1980), Halloun and
constantly changing force Hestenes (1985), and Minstrell (n.d.)
A force cannot keep accelerating an Halloun and Hestenes (1985)
object indefinitely
• If more than one force acts on an object along a straight line, then the forces will
reinforce or cancel one another, depending on their direction and magnitude. Un-
balanced forces will cause changes in the speed or direction of an object’s motion.
(p. 154)
Lower levels of the learning progression were defined and ordered, based upon research
literature reporting students’ ideas about force and motion (detailed in Table 1) and (occa-
sionally) the relative difficulty of these ideas, as well as a logical consideration of proximity
Science Education
to the top level of the learning progression. Much of the research literature focuses on stu-
dents’ conceptions about individual ideas that make up the force and motion construct, but
not on the relationship between their thinking about different ideas. The force and motion
learning progression includes two very closely related ideas: the conditions under which
an object is in motion and the conditions under which an object is at rest. It is an open
question as to whether student understanding of the two situations develops at the same
time or whether one condition is easier for students to understand and thus might fall at
a lower level of the learning progression. Indeed, Finegold and Gorsky (1991) provided
some evidence that these two ideas may not be related in students’ minds in any predictable
way. As a hypothesis about how student understanding develops, the preliminary learning
progression (Table 2) includes full understanding of objects at rest but some misconceptions
about moving objects at Level 4. Students are not hypothesized to reach full understanding
of moving objects until Level 5. This conjecture must be tested before the learning pro-
gression can be considered a valid representation of developing understanding of force and
motion.
Following creation of this preliminary learning progression, a set of nine OMC items
was written to assess students’ learning progression levels, often relying upon canonical
questions (those widely used as examples in instruction and research on students’ ideas).
The OMC options were carefully designed to match the levels of the learning progression
and were expected to represent the way a student might respond to an open-ended version
of the item.
Data Sources. Three separate studies have been conducted to date to refine the force
and motion learning progression and evaluate our ability to assess students with associated
items. An overview of these studies can be found in Table 3.
Pilot Study. The pilot study was designed as a preliminary investigation of the OMC
item type (as reported by Briggs et al., 2006). As part of this study, 112 eighth-grade
students answered both OMC and OE versions of items about force and motion and several
other areas of science. Each student answered seven OMC items and seven OE items. For a
given student, the latter were, in general, not OE versions of the OMC items that he or she
answered. The nine items assessing students’ understanding relative to the force and motion
learning progression were distributed across five different forms, such that all students took
at least one OMC and one OE item in this area, with each student answering a total of 2–4
force and motion items. Between 22 and 27 students responded to each item.
Study 1. Study 1 was designed specifically to obtain information about the force and
motion learning progression. In particular, the pilot study data did not allow us to compare
students’ responses to OE and OMC versions of the same items. In addition, students’
responses to the OE items in the pilot study revealed that some students may not have had
much exposure to ideas about force and motion. Thus, we chose to situate Study 1 in a
classroom in which students had just completed a conceptually focused unit on this topic.
We expected these students to have greater access to the ideas represented in the force and
motion learning progression.
Subjects. Forty-four seventh-grade students participated in this study directly following
their completion of a pilot version of the Full Option Science System (FOSS) force and
motion unit (FOSS, 2003). The FOSS unit provided students with an inquiry-based intro-
duction to force and motion concepts. The unit consisted of a series of investigations, in
which students developed ideas about motion (including ways to measure and represent
velocity and acceleration) and the relationship between force and motion.
Science Education
TABLE 2
Initial Version of the Force and Motion Learning Progression
Level Description
5 Student understands that the net force applied to an object is proportional
to its resulting acceleration (change in speed or direction), and that this
force may not be in the direction of motion. Student understands forces
as an interaction between two objects.
4 Student understands that an object is stationary either because there are
no forces acting on it or because there is no net force acting on it.
However, student may have misconceptions related to a belief that the
applied force is proportional to an object’s speed or motion (rather than
its acceleration). Student can use phrases such as “equal and opposite
reaction” to justify the existence of no net forces but may not understand
this as an interaction.
Common Errors:
• Motion is proportional to the force acting.
• A constant speed results from a constant force.
• Confusion between speed/velocity and acceleration.
3 Student recognizes that forces are not contained within moving objects;
however, student believes that motion implies a force in the direction of
motion and that nonmotion implies no force.
Common Errors:
• Forces are associated only with movement.
• Forces are viewed as causing things to move but not causing things
to stop.
• If there is motion, there is a force acting.
• If there is no motion, then there is no force acting.
• There cannot be a force without motion.
• When an object is moving, there is a force in the direction of its
motion.
2 Student recognizes that forces can be caused by nonliving things; however,
student may believe that forces reside in within moving objects.
Common Errors:
• A moving object has a force within it that keeps it going.
• A moving object stops when its force is used up.
1 Student understands forces as a push or pull, but believes that only living or
supernatural things can cause forces.
Common Errors:
• Forces are caused by living things.
• Forces are associated with physical activity or muscular strength.
• Weight, motion, activity, and strength are important in determining
an object’s force.
0 No evidence or way off-track
The students attended a middle class suburban middle school and came from two classes,
taught by the same teacher. The force and motion unit occurred at the end of the school
year. While their seventh-grade curriculum consisted primarily of inquiry-based FOSS
curriculum units, students experienced more textbook-based instruction as sixth graders.
Force and Motion Items. This study examined student performance on five OMC items,
designed to assess students’ levels on the force and motion learning progression. These
Science Education
TABLE 3
Overview of Studies Used to Collect Data for Revising and Evaluating the
Force and Motion Learning Progression and Its Associated Items
Study Itemsa Students Testing Conditions
Pilot n=9 n = 112 Each student answered at least one OMC item and
at least one OE item.
1 n=5 n = 44 Each student answered OMC and OE items, with
counterbalancing for order.
2 n = 17 n = 64 Fifty-two students answered OE items, then OMC
items. Twelve students participated in
“think-aloud” interviews while answering either
OMC or OE items.
a
The values refer to the number of OMC items. In all three studies, each OMC item had
a corresponding OE version, which was also administered to students.
were selected from those used in the pilot study. Using the classification scheme shown in
the final expanded version of the learning progression (Table 4), these items represented
all four item types, with two items of the no motion type. OE versions of the five items
were developed, using an identical stem when possible. Students answered all items of both
formats, with counterbalancing for order: 23 took OMC first, 21 took OE first. The OE
and OMC items were in two separate test booklets, and students turned in one test booklet
before receiving the second one.
Study 2. Study 2 was designed to answer the research questions addressed in this
paper. It expanded upon Study 1, through the inclusion of additional items. In addition, an
interview component provided the opportunity for deeper exploration of student thinking
with respect to the force and motion learning progression. This study was purposefully
situated in a different instructional context from Study 1, to prevent the learning progression
from being too heavily influenced by a single approach to teaching force and motion.
Subjects. Sixty-four students from a middle class rural high school participated in this
study, directly following a unit on force and motion in their physical science classes.
Students were drawn from “Physical Science” (a year-long course, typically taken by ninth
graders) and “Physical Science 2” (the second year of a 2-year-long course, typically
taken by tenth graders). Students participating in the study were in grades 9–12 (52 ninth
graders, 9 tenth graders, 2 eleventh graders, and 1 twelfth grader). The students were taught
by two teachers at the same high school. Sixteen students were taught by one teacher
and 48 students by the other. In both classrooms and both courses, students experienced a
traditional, textbook-based curriculum, using the physical science textbook from the Holt
Science Spectrum series (2001). The force and motion unit emphasized students’ knowledge
of terms and formulas and their ability to perform calculations, such as those involving the
equation F = ma. This was accomplished through the textbook chapter, in-class lectures,
and completion of worksheets, consisting primarily of calculation problems. In addition,
students performed one laboratory exercise, in which they applied different amounts of force
to a cart and measured its acceleration. The textbook included three pages about forces,
introducing the terms “balanced forces,” “unbalanced forces,” “friction,” “air resistance,”
and “gravity.” Students also completed a worksheet focused on Newton’s laws of motion.
Twelve students were selected to participate in interviews as they completed the items.
The physical science teachers rated each student according to his or her ability to com-
municate with adults and understanding of force and motion concepts. Stratified random
Science Education
sampling by gender and the teachers’ rating of understanding was used to select students
from those who agreed to be interviewed, with preference given to students the teachers
felt would be able to express their thinking in an interview setting. The interview sample
consisted of six boys and six girls. Two of these students had been rated by their teacher as
having low understanding, five as having medium understanding, and five as having high
understanding of force and motion concepts.
Force and Motion Items. Study 2 examined student performance on 17 OMC items
designed to assess students’ levels on a revised version of the force and motion learning
progression. Using the classification scheme shown in Table 4, these items represented
all four item types: three force items, three no force items, five motion items, and six no
motion items. OE versions of the 17 items were developed, using an identical stem when
possible. Fifty-two students (those not participating in the interview) answered all items of
both formats, first the OE items and then the OMC items. The OE and OMC items were in
two separate test booklets, and students turned in the OE test booklet before receiving the
OMC one. Students completed the tests in their regular physical science classes.
Owing to the additional time required for the interviews, students who were interviewed
were randomly assigned to answer questions from either the OMC or the OE test booklet.
Students were interviewed individually in a small conference room in the school’s media
center.
Interviews. All interviews were conducted by the first author. After a brief introduction to
the idea of “thinking aloud” (Ericsson & Simon, 1993), the interviewer completed a practice
problem while thinking aloud, and then the student was asked to do the same on a different
practice problem. There were separate OMC and OE practice items; students listened to and
practiced with the same item format they would be asked to complete during the interview.
Students were instructed to take the test exactly as they would if they were not thinking
aloud, but to say out loud anything they were thinking. Students then answered the items
while thinking aloud. The interviewer had no interaction with the students as they completed
the test booklets, except to remind them to keep talking if they fell silent. If a student asked
a question, such as how to interpret an item, the interviewer explained that he or she should
just answer the question as in a regular testing situation.
After the student completed the test booklet, the interviewer went back over the test,
asking him or her to talk about each item. This part of the interview took the form of a
more traditional clinical interview, in which the goal was to understand students’ responses
to the items. In some cases, students were clear enough during their think-aloud that no
additional questioning about a particular item was required. However, for most items,
follow-up questions were needed to fully understand how the student was interpreting and
thinking about the item. These questions were a combination of general requests for the
student to explain his or her thinking and specific questions about student speech during
the think-aloud portion of the interview. The interviews lasted between 24 and 43 minutes.
The think-aloud portion varied from 10 to 22 minutes, and the clinical interview portion
varied from 9 to 23 minutes.
Revision of the Force and Motion Learning Progression and Associated Items
Pilot Study. The force and motion learning progression and its associated items were
first revised using data from the pilot study, which had several purposes:
1. to obtain basic information about the functioning of the items to inform revisions;
2. to revise the OMC options to better reflect observed responses to the OE versions of
the items; and
Science Education
Figure 1. Revisions to a force and motion OMC item.
3. to test the learning progression in terms of its ability to be used as a scoring guide
for students’ OE responses.
The items were substantially revised in light of student responses. For example, the item
shown in Figure 1(a) asked students to identify the forces acting on a stone after it was
thrown straight up into the air. One of the original OMC options read, “A force inside the
stone is keeping it moving upward.” Only one student (of 23 who responded to this item)
Science Education
chose this option, even though, based on the research literature, we would expect this to be
a popular idea for eighth-grade students. Examination of the OE responses revealed that
students’ writing seemed more consistent with the stone “carrying the force from Derek’s
hand,” so the option was revised to reflect the students’ language, as shown in Figure 1(b).
Ninety-four percent of the students’ OE responses could be scored using the learning
progression, which gave us some confidence that the learning progression and its associated
items reasonably reflected the range of student thinking.2 At this point, revisions to the
learning progression could be made based upon the unscorable responses or new ideas
revealed in students’ OE responses; however, in this case, no revisions were identified.
Study 1. Five items from the pilot study were revised and selected for use in Study
1, which was designed to collect OE and OMC responses from the same students. These
results significantly influenced the structure of the learning progression.
Examination of students’ responses to the OE versions of the items, as well as reex-
amination of the original OMC items, revealed a significant flaw in the original learning
progression (Table 2). In Level 2, we had identified the common error, “A moving object
has a force within it which keeps it going.” This misconception was expected to be resolved
at Level 3, where we had stated that the “student recognizes that forces are not contained
within moving objects; however, student believes that motion implies a force in the direction
of motion.” It was clear from the OE responses that many students who held ideas consis-
tent with Level 3 (and even Level 4) believed that the force causing the motion was being
carried along with the object. When we reexamined the OMC options for the item shown
in Figure 1(a), we realized that there was no difference between the Level 2 and Level 3
options. A student who chose B, one of the Level 3 responses (“only the force from Derek’s
hand is acting on the stone”), would also have to agree with C, a Level 2 response (“a force
inside the stone is keeping it moving upward”). This could account for the relatively low
number of students selecting C in the pilot study. This is also consistent with Ioannides
and Vosniadou’s (2001) notion of “acquired force,” as well as Buridan’s impetus concept:
“A mover, while moving a body, impresses on it a certain impetus, a certain power capable
of moving this body in the direction in which the mover set it going, whether upwards,
downwards, sideways, or in a circle” (as quoted in Halloun & Hestenes, 1985, p. 1057).
Careful consideration of students’ OE responses seemed to indicate a different orga-
nization of ideas in the learning progression. Students who believed that the force from
Derek’s hand was still acting on the stone as it was traveling up into the air seemed to have
ideas consistent with two quite different conceptions of the relationship between force and
motion. Some students, who identified the force from Derek’s hand as the only one acting
on the stone (consistent with option B), seemed to believe that the only force acting on a
moving object could be one in the direction of its motion. Other students, who identified
the force from Derek’s hand and gravity as acting on the stone (consistent with option D),
seemed to believe that, while there needs to be a force in the direction of an object’s motion,
other forces could also be acting on the object. Both of these ideas could be held by a student
with the physicist’s definition of force or by a student with a view of force more like what
physicists would call momentum, a quantity related to the speed of a moving object that
decreases as an object slows down. Thus, we collapsed Levels 2 and 3 and added sublevel
“A” to this level and the next higher one (originally Level 4, now Level 3). Students at a
given level (e.g., Level 2) and students at the corresponding sublevel A (e.g., Level 2A)
share the same underlying idea about the relationship between force and motion. Students
at Levels 2 and 3 have a more conventional understanding of “force,” whereas students at
2
This result can provide information only about whether the learning progression captures the range of
student thinking; nothing can be inferred as to whether this is actually how that thinking develops.
Science Education
Levels 2A and 3A have the “impetus view” of force. This revision is reflected in the current
version of the learning progression, shown in Table 4.
Once we had a new version of the learning progression, the existing OMC items had to be
revisited to ensure that the options corresponded to levels of the new learning progression.
Additional revisions were made to the items, based upon analysis of students’ OE responses.
For example, some students mentioned air resistance as a force acting on the stone in the
item in Figure 1. As shown in Figure 1(c), the stem was rewritten to instruct students to
ignore air resistance. Also, students described the force as being from “Derek’s throw” (or
“Derek’s push”), rather than from “Derek’s hand,” as worded in the original options. This
change can also be seen in Figure 1(c).
With the new learning progression, we began to develop more items to assess the student
understanding it describes. This process led to further revisions of the learning progression.
In particular, as shown in Table 4, we identified four item types:
• Force: Situations in which a force is acting, and students are asked about the resulting
motion.
• No Force: Situations in which there is no net force acting, and students are asked
about the resulting motion.
• Motion: Situations in which an object is moving, and students are asked about the
force(s) acting on the object.
• No Motion: Situations in which an object is at rest, and students are asked about the
force(s) acting on the object.
We specified how a student at each level would be expected to respond to items of each
type. This was useful in writing items and also, later, in scoring them.
Study 2. The learning progression has been further revised, in response to the data
collected in Study 2. Several students’ OE responses revealed that they did not believe it
was possible for an object to move in a frictionless environment. This misconception was
probed in one student’s interview and was determined to be a correct interpretation of his
thinking. Thus, this misconception was added to Level 1 of the learning progression, as
reflected in the current version in Table 4. Comparison of students’ OMC responses to
their OE responses and to the discussion surrounding these items in the interviews was
valuable in rethinking options and other aspects of the OMC items. For example, several
items, including the one shown in Figure 1, contained a Level 1 option, “There are no
forces acting on the [object] because nothing is touching it.” As students discussed these
options during the interviews, they clearly found them to be implausible, because they knew
that gravity does not touch the objects it acts upon. In many cases, such as the example
in Figure 1(d), these options could be rewritten without the qualifier “because nothing is
touching it” without losing the essence of the option. Other items were revised to more
accurately reflect the language used by students in their OE responses and interviews, as
well as to capture the thinking revealed through the interviews. The revised items will
be tested with another set of students, to continue the process of revising the learning
progression and associated items.
Study of the Force and Motion Learning Progression

and Associated Items
Data collected in Studies 1 and 2 were used to address the research questions identified
above.
Science Education
TABLE 4
Current Expanded Version of the Force and Motion Learning Progression
Level Description Force No Force Motion No Motion
4 Student understands that the net force applied If there is a nonzero If there is no net force If an object is If an object is not
to an object is proportional to its resulting net force acting on acting upon an accelerating, a moving, the net
Science Education
acceleration (change in speed or direction) an object, it will object, it will move nonzero net force force acting
and that this force may not be in the direction accelerate. with constant velocity. is acting upon it. If upon the object
of motion. an object is is zero.
moving with
constant velocity,
no net force is
acting upon it.
3 Student understands that an object is stationary If there is a nonzero If there is no net force If an object is moving If an object is not
either because there are no forces acting on it net force acting on acting upon an with constant moving, the net
or because there is no net force an object, it will object, it is either velocity, a nonzero force acting
acting on it. move with slowing down or net force is acting upon the object
Student has a partial understanding of forces constant velocity. stopped. upon it. If an is zero.
acting on moving objects. 3A: The zero net force object is slowing
• Student recognizes that objects may be could result from down, no net force
moving even when no forces are being opposing forces is acting upon it.
applied; however, the student does not coming into balance
believe that objects can continue moving (e.g., through one
at a constant speed without an applied force dissipating).
force.
• Student recognizes that there may be
forces acting on an object that are not in
the direction of its motion. However, he
or she believes that an object cannot be
FORCE AND MOTION LEARNING PROGRESSION
moving at a constant speed in a direction

in which a force is not being applied.
403
(Continued)
404
TABLE 4
Continued
• Student believes that the object’s speed

(rather than its acceleration) is
proportional to the net force in the
direction of its motion.
Common Errors:
• An object’s speed and direction are
proportional to the nonzero net force
acting on it.
ALONZO AND STEEDLE
◦ 3A: An object may come to rest when

opposing forces (e.g., the force which put
the object into motion initially and gravity)
come into balance.
• A constant force causes constant speed.
• Without an applied force, all objects will slow
down and eventually come to rest.
2 Student believes that motion implies a force in If a force is acting If no force is acting If an object is If an object is not
the direction of motion and that nonmotion upon an object, it upon an object, it is moving, a force is moving, no
implies no force. Conversely, student believes is moving. not moving. acting upon it. force is acting
that force implies motion in the direction of the 2A: The force acting upon it.
force. on an object could
be the initial force
(which is carried
with the object and
may dissipate over
time).
(Continued)
Science Education
TABLE 4
Continued
Common Errors:
Science Education
• If there is no motion, there are no forces
acting.
• When an object is moving, there is a force in
the direction of its motion.
◦ 2A: This motion could be the force that
put the object into motion initially.
◦ 2A: The object may come to rest because
the force it carries with it has been used
up.
1 Student understands force as a push or pull that If a force is acting on
may or may not involve motion. an object, it is
Common Errors: moving unless the
• Forces are caused by living things. object is
• Force is an internal property of objects immovable.
related to their weight. (There is a force
on all objects that is not due to gravity or
because of their motion.)
• Forces prevent the natural movement of
objects (i.e., gravity prevents objects from
flying off into space).
• Objects cannot move in the absence of
friction.
FORCE AND MOTION LEARNING PROGRESSION
0 Way off-track
405
Scoring and Analysis of Student Item Responses

Scoring Student Item Responses. Study 1. Although items had been developed ac-
cording to the original learning progression (Table 2), the OMC options were recoded
and all OE responses were scored according to the new learning progression (Table 4).
To prepare for scoring the OE responses, two raters (the authors) practiced independently
scoring OE responses from the pilot study and discussed results. Scoring of data for this
study was carried out in four phases to continually monitor interrater agreement. These
phases included the data for 8, 12, 12, and 13 students, for a total of 45 students, since
one student took only the OE test booklet. Both raters scored all student responses, with
an average interrater agreement of 84%, and discrepancies were discussed to agree upon a
final score for each response.
Study 2. Twenty percent of the OE responses to each item were randomly selected to be
used as “training” for scoring the OE responses. Two raters (the authors) practiced scoring
independently and then discussed results to agree upon a final score for each response.
During the training, it became apparent that, in addition to the levels of the learning
progression, another code was required to indicate cases in which the student response to
an OE item did not answer the question. This “no score” code will be discussed in more
detail in the results section.
As in Study 1, scoring was completed in phases. Items were divided into four groups
so that items addressing the same underlying principle were scored together. Each group
was further divided, so that no more than 40% of the items were scored in any given
phase. This allowed the raters to independently code and then discuss results periodically
to continually monitor interrater agreement. Both raters scored all student responses, with
an average interrater agreement of 86%, and discrepancies were discussed to agree upon a
final score for each response.
Comparing Responses to OMC and OE Versions of the Items. With OMC and OE
item responses coded using learning progression levels, the item responses for each student
served as evidence of his or her learning progression level such that a student’s performance
across the set of items (in either the OMC or OE format) provided a diagnosis of learning
progression level. Before such diagnoses could be made, however, two data manipulations
were required. The first addressed the fact that some OE responses were coded with multiple
learning progression levels (e.g., 2/3 or 2A/3). This occurred when a student’s OE response
was consistent with the understanding at two different levels of the learning progression,
without adequate information about his or her thinking to distinguish between the two.
To resolve this issue, these responses were recoded with only one level. Decisions were
based on evidence from students’ OMC responses. For example, 19 students provided a
Level 2A/3 response to OE1. Of those students, nine selected a Level 2A response to the
corresponding OMC item (OMC2), and only three students selected the Level 3 response.
For this reason, 2A/3 responses for OE1 were recoded as 2A. The second manipulation
involved collapsing Levels 2 and 2A and Levels 3 and 3A to convert the partially ordered
data to a fully ordered form. With each response coded by only one of five ordered learning
progression levels (0, 1, 2, 3, or 4), it was then possible to compute students’ learning
progression levels.
Two learning progression diagnoses were made for each student who answered both
sets of items: one based on OMC responses and the other based on OE responses. Stu-
dents’ learning progression diagnoses were calculated as the mean of all of their responses
to the set of items of a given format. These diagnoses were used to examine whether
the two item formats yielded similar pictures of students’ understanding of force and
motion.
Science Education
The standard error of measurement (SEM) was then estimated to determine the precision
with which students’ learning progression levels could be diagnosed. Larger SEMs indicate
less certainty as to students’ actual learning progression levels. Thus, to use a set of items
to diagnose students’ levels on the learning progression, the SEM should be as small as
possible.
In addition, the reliability (Cronbach’s α) of each learning progression level was estimated
using data that were dichotomized for that level. For example, to calculate the reliability
for Level 2, each Level 2 response was coded as 1, and all other responses were coded as 0.
Since many items had no students with Level 0 or 1 responses, reliability was computed only
for Levels 2, 3, and 4. One item in Study 1 and four items in Study 2 had a maximum score
of Level 3; to compute the Level 4 reliability, Level 3 responses to those items were coded
as 1. Thus, the Level 4 reliability is equivalent to the reliability obtained by scoring the
items dichotomously (i.e., correct or incorrect, like typical multiple-choice items). Overall
reliability was calculated for both the set of OE items and the set of OMC items. This is
equivalent to the reliability obtained by treating item responses as polytomous (i.e., like
typical open-ended responses scored using a rubric). These values may have been slightly
underestimated due to the restricted range of the items with a maximum score of Level 3.
Statistical packages commonly exclude students with missing data when estimating
reliability because such students would have unexpectedly low total scores. This procedure
is reasonable when sample sizes are large and missing data are rare. Unfortunately, these
assumptions do not hold for the data reported here. For example, in Study 2, 53 of 58 students
had at least one missing OE response, mostly due to the “no score” code. To correct for
this, a student’s missing responses were imputed using his or her average item score. This
correction had very small effects for all calculations of reliability except for OE items in
Study 2, for which it reduced the reliability estimate.
To further explore the relationship between OMC and OE versions of the items, two
item-level analyses were also performed. First, responses to OE and OMC versions of each
item, as coded with the learning progression levels they expressed, were cross-tabulated.
The cross-tabulation for the second item in Figure 2 is shown in Table 5. Each table (5 in
Study 1 and 17 in Study 2) was studied to determine whether students provided similar
responses to the OE and OMC versions of the items. Counts shown in the cells of each
table indicated the number of students who provided a particular pair of responses to the
two versions of the item. If students tended to respond similarly to both item formats, one
would expect relatively large counts in the diagonal cells. Relatively large off-diagonal
counts served as evidence that the different item formats were not eliciting the same types
of responses from students. Second, coded OE and OMC responses were used to compute
a Spearman rank-order correlation coefficient for each item pair.
Examining the Consistency of Student Responses. The calculations of reliability de-

scribed above were used as a measure of the consistency of student responses at a given
learning progression level and to the items overall. In Study 2, reliability was also estimated
for the four types of items identified in the expanded learning progression (Table 4): force,
no force, motion, and no motion. The groups did not have the same number of items, so
the Spearman–Brown prophecy formula was used to estimate the reliability as though each
group had six items (the size of the largest group).
In addition, in Study 2, there were some pairs of items that shared not only the same item
type (e.g., no motion), but which were essentially the same problem, set in two different
contexts. An example of such a pair is shown in Figure 2. For these items, responses to the
OMC versions of each item, as coded by the learning progression levels they expressed,
were cross-tabulated. If students tended to answer the two questions similarly, one would
Science Education
Figure 2. Two OMC items that assess the same content in two different contexts.
expect relatively large counts in the diagonal cells. Relatively large off-diagonal counts
served as evidence that the two items were not eliciting the same types of responses from
students.
Using Qualitative Analysis of Open-Ended Responses to Gain Insight into Quantitative
Findings. In Study 2, students’ answers to the OE items, as well as transcripts of the
12 interviews, were examined to further explore the way that students were responding to
the items. Four different analyses were conducted with these data. Items of the same type
(force, no force, motion, no motion) were analyzed together to look for patterns across
items of the same type.
First, cross-tabulations of OE and OMC versions of the same items were inspected to
identify off-diagonal cells containing relatively large numbers of students, as compared to
the numbers in the diagonal cells. All available data were then used to try to make sense of
the off-diagonal cells. OE responses provided by students in a given cell were examined,
along with the OMC options they selected. Students’ responses to the OMC and OE items
in the interviews were also studied for evidence that could help to explain responses in the
off-diagonal cells.
Second, for a particular item, interview transcripts for those students responding to the
OMC version of the item were read. Each student’s response during the think-aloud portion
of the interview (while he or she was actually taking the test) was compared to his or her
Science Education
TABLE 5
Cross-Tabulations for OMC7/OE15 (Second Item in Figure 2) with Diagonal
Cells Shaded
OMC7 Response Levels
OE15 Response Levels 1 2 3
0 1 0 0
1 6 0 4
2 2 13 6
3 3 5 12
response during the clinical interview. This allowed us to compare the OMC results to what
might have been obtained from these students in a clinical interview setting.
Third, all interview transcripts were read, looking for instances in which students’ use
and interpretation of language differed from that intended by the items. Data for students
taking the OMC version of the items typically included explanations for why they did not
pick certain options, either as part of their process of answering the item or in response
to the interviewer’s questions after completing the test. These responses often included
comments on the language used in the options and provided an opportunity to explore how
students were interpreting the items. Interviews with students answering both item formats
included multiple opportunities to explore students’ use of language. For example, for the
item shown in Figure 1(c), the interviewer checked with students to be sure that they really
meant that the force from Derek’s throw was still acting on the stone at point A and not just
that Derek’s throw was what started the stone moving in the first place.
Finally, cross-tabulations of pairs of similar items were inspected to identify off-diagonal
cells containing relatively large numbers of students, as compared to the numbers in the
diagonal cells. Interview data and results from the previous three analyses of individual
items were considered in looking for evidence that would help explain the responses in the
off-diagonal cells.
RESULTS
Comparing OMC and OE Versions of the Items
The comparison between student responses on the OMC and OE versions of the items
is complicated by the fact that different ranges of responses are possible for the two item
formats. While the OE versions allow students to provide answers at any level of the
learning progression,3 the OMC versions restrict responses to levels corresponding to the
options provided. The OMC items present a truncated range of levels, in which the lowest
levels of the learning progression are not typically represented in the options. Implications
of this difference between the OMC and OE versions of the items are discussed below.
Diagnosis of Learning Progression Levels. We first consider the diagnosis of students’

levels on the force and motion learning progression provided by the two item formats. On
average, students received slightly higher scores on the OMC items than on the OE items in
3
As described above, some items had a maximum score of Level 3; thus, for both the OE and OMC
versions of the items, a Level 4 response was not possible.
Science Education
TABLE 6
Standard Errors of Measurement and Reliabilities of Learning Progression
Levels and Overall Item Sets (Study 1)
OE OMC
Level SEM α SEM α
2 0.199 .459 (.743) 0.191 .445 (.732)
3 0.209 .249 (.530) 0.198 .279 (.568)
4 0.156 .378 (.674) 0.158 .299 (.592)
Overall 0.361 .472 (.752) 0.299 .318 (.613)
Note: SEMs are expressed in the average score metric (on a 0 – 1 scale for the level
reliabilities and a 0 – 4 scale for the overall reliability). For comparison to Study 2, the
Spearman–Brown prophecy formula was used to calculate expected values for a 17-item
test (shown in parentheses).
both studies. These differences are not practically significant and are not surprising, given
that OE items provide students the opportunity to respond at lower levels of the learning
progression than do the OMC items. In Study 1, students who took both item formats (n =
44) responded at a mean level of 2.61 (SD = 0.36) to the OMC items and a mean level of
2.55 (SD = 0.50) to the OE items. In Study 2, the mean response level of students who took
both versions of the items (n = 52) was 2.86 (SD = 0.23) for the OMC versions and 2.64
(SD = 0.38) for the OE versions. The correlation between students’ OMC and OE learning
progression level diagnoses was .59 in Study 1 and .44 in Study 2.
Practically speaking, individual students received similar diagnoses based upon the OMC
and OE items. In Study 1, the mean absolute difference between OMC and OE diagnoses
was just over one quarter of a level (0.26). Further examination of these results revealed
one outlier, a student who provided “way off track” answers to 4 of the 5 OE items, thus
receiving scores of 0 on these items and an OE diagnosis 1.8 levels below her OMC
diagnosis. This student was excluded from further consideration of the results. For all
other students, the maximum absolute difference between OMC and OE diagnoses was
1.0 levels, with a mean absolute difference of 0.23. Considering half of a level to be the
smallest difference of any practical significance, only four students (9%) had a practically
significant absolute difference between their two diagnoses
Similar results were observed in Study 2. The mean absolute difference between OMC
and OE diagnoses was 0.3 levels. Again, there was one outlier, a student who did not answer
half of the OE items and provided “way off track” responses to 2 of the 9 items he did
answer. His OMC responses were 1.55 levels above those for the OE items he answered.
For the other students, the maximum absolute difference between OMC and OE diagnoses
was 0.84, with a mean absolute difference of 0.27. Eight students (14%) had a practically
significant difference between their OMC and OE diagnoses (an absolute difference greater
than 0.5 levels).
Tables 6 and 7 show the standard errors of measurement estimated for Study 1 and
Study 2, respectively. Although SEMs are provided for Levels 2–4 (to correspond with
the reliability calculations, discussed below), to diagnose students’ learning progression
levels, the important metric is the overall SEM for a set of items. In both Studies 1 and
2, the SEMs for OMC and OE items are comparable, with slightly higher values obtained
for the OE items as compared to the OMC items. Thus, we can be slightly more certain
about students’ learning progression level diagnoses when using the set of OMC items, as
compared to the corresponding set of OE items.
Science Education
TABLE 7
Standard Errors of Measurement and Reliabilities of Learning Progression
Levels and Overall Item Sets (Study 2)
OE OMC
Level SEM α SEM α
2 0.092 .551 0.092 .568
3 0.095 .613 0.106 .417
4 0.097 .600 0.102 .500
Overall 0.193 .735 0.162 .523
Note: SEMs are expressed in the average score metric (on a 0 – 1 scale for the level
reliabilities and a 0 – 4 scale for the overall reliability.
Tables 6 and 7 also show the reliabilities calculated for Studies 1 and 2, respectively. The
reliabilities for the two studies are comparable when the values from Study 1 are adjusted
using the Spearman–Brown prophecy formula. The reliability coefficients indicate that
students applied ideas slightly more consistently in the OE items. However, this result
should not be interpreted to mean that OE items provide a better diagnosis of students’
learning progression levels than do the OMC items. Reliability can be formulated as
σE2
1− ,
σX2
where σE2 is the error variance (=SEM2 ) and σX2 is the observed score variance. Because
there is greater variation in possible responses to the OE items, the observed score variance
for the OE items (σ̂X2 = 40.62 in Study 2) is much greater than that for the OMC items
(σ̂X2 = 15.90 in Study 2), whereas the error variance for the OE items (σ̂E2 = 10.76 in
Study 2) is relatively close to that for the OMC items (σ̂E2 = 7.58 in Study 2). Thus, the
difference in reliability between the two item formats can be thought of as an artifact of the
greater range of response options for the OE items, as compared to the OMC items.
Responses to Individual Items. We now compare student performance on the OMC

and OE formats for individual items. In Study 2, comparison of OE and OMC responses
was hindered by the relatively large proportion of “no scores” assigned to students’ OE
responses on some of the items. For 2 of the 17 items more than 50% of the responses
to the OE version were coded as “no score”: OE8 (an item about the forces acting on an
accelerating rocket sled, 65%) and OE16 (an item about the motion of a rocket propelled
by a constant force, 56%).
In both studies, on average, about half of the students received the same score on both
versions of a given item (54% in Study 1 and 51% in Study 2, considering only the OE
responses that could be scored). The average rank-order correlations between students’
scores on the OE and OMC versions of each item were modest, around .3. In Study 1, the
rank-order correlations ranged from .28 to .44, with an average of .35. There was a greater
range of rank-order correlations in Study 2: −.19 to .64, with an average of .29.
Analysis of data from Study 2 indicated that, across all four item types, ambiguity
in students’ OE responses can be considered to account for many of the differences in
students’ scores on the two item formats. Vague writing was scored using a best guess
Science Education
as to the student’s meaning; examination of the cross-tabulations and associated OE and

OMC responses led to the identification of a number of instances in which the best guess
seemed to over- or underestimate a student’s level on the learning progression. This can
be illustrated using two examples from the no force items. In these examples, a force is
applied to an object and then removed. Students are asked to describe the motion of the
object after the force has been removed.
First, consider an example (OMC2/OE1) in which students’ levels on the learning pro-
gression seems to have been overestimated by their OE responses. In this item, students
are asked to describe the motion of a puck traveling on a frictionless surface after a force
has been applied and then removed. Students whose responses included a vague statement
that the puck would travel at a constant speed or that it would keep going were scored at
Level 4. We interpreted these statements to represent the Level 4 understanding that an
object moving at a constant speed with no forces acting on it will continue to move at a
constant speed. However, examination of the OMC options chosen by these students, as
well as questioning during the interviews, revealed that students did not necessarily think
that the puck would keep going forever. Many thought that it would eventually slow down,
a Level 3 response.
Second, consider an example (OMC4/OE4) in which students’ levels on the learning
progression seems to have been underestimated by their OE responses. In this item, students
are asked to describe the motion of a car on an icy road after a force has been applied and
then removed. Students who stated that the car would stop when the force was removed were
scored at a Level 2. We interpreted these statements to represent the Level 2 understanding
that an object cannot be moving without a force acting in the direction of its motion,
but examination of the OMC options chosen by these students revealed that they did not
necessarily think that the car would stop moving immediately after the force was removed.
Rather, they thought that the car would gradually slow down to a stop, a Level 3 response.
This interpretation was corroborated in an interview with a student (S044) whose OE
response in the test booklet was, “It would not be moving because there is no force acting
on the car” (scored at Level 2).
In the force and no force items, there was the particular difficulty that students’ responses
(understandably) often did not include an explanation for the motion they described. Since
the same motion could be predicted using two different underlying conceptions of the rela-
tionship between force and motion, it was often impossible to distinguish between levels of
the learning progression using students’ OE responses. For example, in a no force problem,
students may believe that an object is moving at a constant speed when an applied force is
removed either because no forces are required to keep an object moving at a constant speed
(Level 4) or because the force applied to the object is acting on it as it moves (Level 2A).
In several of the motion items, students’ OE and OMC answers seemed to differ because
the OMC options alerted them to additional forces to consider. For example, in response
to the OE version of the item shown in Figure 1(c), students often listed a single force,
either gravity (Level 4) or the force from Derek’s throw (Level 2A). After reading the OMC
options, 3 of the 6 students who responded to the OE version at Level 4 and 18 of the
32 students who responded to the OE version at Level 2A chose the Level 3A OMC option
(both gravity and the force from Derek’s throw). These results should not be interpreted
to mean that all students held a Level 3A response and just failed to provide a complete
response to the OE item. Twelve of the thirty-two students who indicated in the OE item
that only the force from Derek’s throw was acting on the stone as it traveled up did not
add gravity to their answer in the OMC version, sticking with their original Level 2A
conception. Interview results support these findings. Students who listed a single force in
response to either the OMC or OE version of this item and other similar items were asked
Science Education
whether the other force (either gravity or a force from Derek’s throw) was also acting. While
some students decided that the additional force was acting, others did not. For example,
some students did not think that gravity could be acting, since the stone was moving up
“away from gravity” (S051).
In the no motion items (and one of the motion items), some discrepancy between OE
and OMC results seems attributable to students’ failure to recognize their conception in
the corresponding OMC option. In the interviews, the no motion items seemed to require
the least amount of thought on the part of students, who readily identified that there were
balanced forces acting on an object at rest. The textbook included a passage on “balanced
forces,” and several students explained that similar examples had been discussed in one
teacher’s classes. Indeed, 36 students provided a Level 3 response to the OE version of
OMC1 (shown in Figure 2). However, six of these students did not select the Level 3 option
in the OMC item. Closer examination of these students’ OE responses reveals that they
wrote vague statements like, “the forces are balanced” or “the forces are equal.” (Other
students’ responses more clearly explained that gravity was pulling down and the table was
pushing up.) Thus, it seems that some of the students were not able to recognize the idea
of balanced forces in answer choice D. In this case, it seems as if the OE response may
overestimate these students’ level on the learning progression, since their conception of
“balanced forces” does not seem to be very well developed. The students are able to recall
the term “balanced forces” in response to a situation in which an object is at rest, but they
may not actually know which forces are balanced. This appears to be one way in which
students’ responses to the learning progression items were influenced by the curriculum
they had experienced.
Comparing OMC Responses and Clinical Interviews

For the six students who were interviewed while completing the OMC test booklet,
82% of their answers match the conceptions revealed through the clinical interview. The
different item types yielded various levels of consistency between the clinical interviews
and the OMC responses, from 100% for the force items to 73% for the motion items. There
seem to be two categories of instances in which students’ choice of OMC option did not
match the understandings expressed in the clinical interviews, each accounting for exactly
half of the mismatches. First, some students were ambivalent or confused about the correct
answer choice. Both their think-aloud and clinical interviews revealed multiple (sometimes
conflicting) conceptions, whereas their OMC answer could, necessarily, reflect only one of
these ideas. Of these responses, 67% belong to one student, S124, who frequently changed
his mind about the correct option, even before any interview questions had been asked.
Second, there were some genuine discrepancies between the understanding revealed in
the think-alouds and clinical interviews. S130 accounts for 56% of these cases, due to
an interpretation of the Level 2A options that differed significantly from that intended
(discussed in more detail below).
Consistency of Student Responses

As expected, examinees did not always apply the same scientific ideas consistently across
different problem contexts. When adjusted to reflect a set of 17 items, the reliabilities shown
in Tables 6 and 7 are all moderate. For both item formats in Study 1 and the OMC item format
in Study 2, the lowest reliabilities were obtained for Level 3. It is possible that students
using Level 3 ideas are in a transitional phase between novice and expert understanding in
which they provide answers from a variety of learning progression levels.
Science Education
TABLE 8
Standard Errors of Measurement and Reliability of Item Types, Corrected
Using Spearman–Brown Prophecy Formula (Study 2)
OE OMC
Item Type Number of Items SEM α SEM α
Force 3 0.339 .601 (.751) 0.260 .178 (.302)
No force 3 0.594 .754 (.860) 0.586 .308 (.471)
Motion 5 0.603 .740 (.774) 0.352 .176 (.204)
No motion 6 0.487 .493 0.513 .384
Note: For comparison across item types, the Spearman–Brown prophecy formula was
used to calculate expected reliabilities for a six-item test (shown in parentheses). SEMs are
expressed in the average score metric (on a 0 – 4 scale).
For Study 2, the overall reliabilities of items of the four types addressed by the force
and motion learning progression (force, no force, motion, no motion) are shown in Table 8.
Similar calculations were not possible using the data from Study 1, since there was only
one item per item type. In the OMC item format, the motion items exhibited the lowest
reliability. Although the fewest different item contexts were included in this category, they
addressed both accelerating and constant motion. By definition, other item types addressed
either accelerating or constant motion. In addition, the no force category contained items
with a similar structure, all addressing situations in which a force had been applied and
was then removed. This may help to explain the higher reliabilities observed for this item
type (at least for the OMC items). In the OE item format, the no motion items exhibited
the lowest reliability. The interviews provide evidence that some of the variation in student
responses may be attributable to varying familiarity with the item contexts represented in
the items. Interviews revealed that one item context used in a no motion item had been
discussed in class by one of the teachers; thus, it represented a familiar context for these
students. In contrast, interviewed students expressed confusion about (and a variety of
interpretations for) the situation described in another no motion item.
In Study 2, there were five pairs of items that asked essentially the same question, but
in two different contexts (e.g., the pair of items in Figure 2). For each pair, at least 50%
of the responses were the same across the two items; on average, over the five item pairs,
59% of the responses were the same. Careful reading of students’ interview responses to
these items revealed three patterns: (1) recognizing the items as the same and responding
accordingly; (2) recognizing the items as the same, yet responding differently; and (3) not
recognizing the items as the same at all. Students in the second category tended to confuse
themselves, by drawing connections across both similar and different items. For example,
students sometimes discussed the force and no force items interchangeably, not recognizing
that in the former, a constant force was being applied, whereas in the latter, a force had been
applied but was no longer acting. Or, they commented that a given item was like one they
had already responded to, but did not seem to use this information in selecting a response
option.
The problem context seemed to have a significant influence on the way in which students
in the third category responded to the items. Not surprisingly, students responded differently
to contexts that had been discussed in class than they did to more novel situations. Students
in one teacher’s classes talked explicitly in the interview about the example of a book
on a table that he had discussed with them, seemingly in some detail. A relatively high
proportion of students (69%) selected the Level 3 option (the highest level) for the OMC
Science Education
item about a box on a table (the first item in Figure 2). However, 40% of these students
did not select the Level 3 response in a similar item (the second item in Figure 2), seeming
to indicate that they had difficulty transferring their knowledge to a new situation. This
appears to be another example of the curriculum influencing the assessment of students’
learning progression levels.
Role of Language in Student Responses

The way in which most students used the word “force” in responding to the learning
progression items matched the thinking revealed through probing during the clinical inter-
views. Even so, there were several issues identified with students’ use and interpretation of
language that may impact the evaluation of their responses.
Most significantly, S130, whose understanding of force and motion concepts expressed
during the clinical interview was relatively high, had a dramatically different interpretation
for “forces acting” on an object than was intended, particularly in the Level 2A and Level
3A options. An excerpt from S130’s explanation of his response to OMC2, the first such
item he encountered, illustrates this point. This is a no force item, in which a girl has hit a
puck and it is now traveling across a frictionless surface. S130 chose the Level 2A option,
“The speed is constant because of the force from Amelia hitting the puck.” This excerpt
begins as the student is explaining how he selected this option over the Level 4 option,
“The speed is constant because there is no net force acting on the puck.”
S130: So that’s why, ‘cause this one says there is no net force, and there has
to be a net force. That’s why I chose A [Level 2A option] instead of
B [Level 4 option].
I: And so why does there have to be a net force, then?
S130: Because if it’s not moving, and then it starts moving, there’d have to
be a force acting on it.
I: Okay. So she, she hits, she hits the puck.
S130: Yeah.
I: And then she’s, she’s not hitting it anymore, what’s happening to it?
S130: The inertia. Or like the, yeah, that’s it. Inertia.
I: Okay, so something’s carrying it along still.
S130: Yeah. From the original force.
I: Okay. And so is that force still acting on it?
S130: Is the force still acting on it? . . . Probably not.
He further clarified his thinking, in response to another item (OMC6):
S130: It’s not like actually touching it anymore, to like act on it, but it’s
like, the . . . it’s like, how you interpret the word act. ‘Cause, ‘cause
it’s not like touching it, so it’s like literally acting on it, but it’s like,
the consequences of it acting on it here [when the force was being
applied] are, are make, are determining how it’s doing stuff [as it’s
traveling].
S080 provided a similar explanation for his Level 2A response to the OE version of
another item (OMC5). Thus, it might be tempting to conclude that students do not ac-
tually believe that moving objects carry a force with them. However, a number of other
students clearly verified that they did hold beliefs consistent with Level 2A of the learning
progression. The challenge is figuring out how to distinguish between these two groups
Science Education
of students in an OMC format, without the benefit of probing questions to determine how
students are really thinking about “forces acting.”
Another issue arose with respect to the force items. In these items, a constant force
was being applied to an object, and students were asked to describe the object’s motion.
Preliminary evidence from students’ OE responses seemed to indicate that some students
were interpreting these questions as if the force was acting on the object only momentarily.
(This issue also arose in the version of items used in Study 1 and revisions were made,
which were intended to clarify that the force was still acting on the object.) Interview
data corroborated the OE results, indicating that some students were still interpreting the
problem situation incorrectly. This is a potential problem, since a student with a Level 4
conception, who interprets the item as a no force item, would produce a Level 3 answer,
thus resulting in a misdiagnosis of his or her level of understanding.
Finally, the use of the word “net force,” a phrase often necessary in Levels 3 and 4 of
the learning progression, seemed unfamiliar to some students in Study 2. In the clinical
interview, S047 explained that she had not chosen options containing this phrase because
she did not know what that was. In response to one item (OMC2), she seemed to interpret
this as a really big force, claiming that the object would stop right away if a net force were
applied to it.
DISCUSSION
Item Format
OMC items seem better suited for assessing students’ levels on the force and motion
learning progression, as compared to OE items. OMC items are clearly much easier to
score, but more importantly, they provide a more unambiguous and slightly more precise
indicator of student thinking. For the most part, discrepancies between students’ responses
to the OMC and OE versions of the items could be attributed to difficulty in eliciting
sufficiently detailed explanations in student responses to the OE versions. Many of the
“off-diagonal” cells in the cross-tabulations discussed above were due to our difficulty in
interpreting vague student responses to the OE versions of the items.
More than 10% of the student responses could not be scored for four of the OE items, and
more than 50% of the responses could not be scored for two items. There are at least three
explanations for the relatively high numbers of students responding in a manner that did
not answer the question. First, three items (OE8, OE12, and OE16) dealt in some way with
rockets, a topic that had just been discussed by the teacher whose classes three quarters of the
students attended. The interview students explained that the example of a rocket had been
used to illustrate Newton’s third law. Thus, students’ “no score” answers to these questions
tended to focus on Newton’s third law, rather than responding to what the questions were
asking about the rockets’ motion. For example, after some introductory material, OE16
asked, “While the engines are on, how will the rocket move? Please describe the rocket’s
motion with as much detail as possible.” In response to this prompt, Student 051 wrote,
“The engine will put flams [sic] out the back in one direction and the rocket will go in the
other direction.” This seems to be another example of students’ responses to the learning
progression items being influenced by the curriculum they had just experienced. Second,
the prompts for these items may have been poorly written, leading to confusion about the
type of response being sought. For example, in OE16, students may have interpreted “how
will the rocket move?” to mean “how is it possible for the rocket to move?”, ignoring the
request to describe the rocket’s motion. Third, it just may not be possible to write OE
prompts to elicit responses including sufficiently detailed explanations of the phenomena.
Science Education
Clinical interviews clearly provide the most unequivocal picture of student thinking, but
the OMC items seemed to do a reasonably good job of estimating the student understanding
expressed in the clinical interviews. In some ways, the multiple-choice options can be
considered to serve a similar function to the questioning in a clinical interview, probing
student thinking in preidentified areas. One could argue that a student’s reading of the
options in an OMC item is, to a certain extent, an approximation of part of what happens
in a clinical interview. For example, consider the item in Figure 1(c). As described above,
cross-tabulations for this item revealed that a number of students identified only gravity or
only the force from Derek’s hand in the OE version of the item, but these same students
chose option C (that both forces were acting) when responding to the OMC version of
the item. Presumably, these students considered the second force after reading the options,
and, indeed, the pattern of the clinical interviews was similar. If a student responding to
the OE version of the item in the interview stated that only the force from Derek’s hand
was acting on the stone, the interviewer would ask whether that meant that gravity was not
acting on the stone. In the clinical interview, the interviewer does something similar to the
OMC options by presenting students with some possibilities and getting their reactions.
Students’ think-aloud responses support this idea, as most students considered each option
individually, testing its plausibility against their own thinking. Since the OMC options
represent conceptions that students are known to hold, and thus are ideas likely to be
explored in a clinical interview, they can serve this purpose in a way not possible with
traditional multiple-choice items.
Consistency of Student Responses

Students’ inconsistency in responding to problems with the same underlying physics
content is certainly a challenge to our ability to place them at a given level of the learning
progression. Diagnosis of students’ learning progression levels may depend upon the prob-
lem contexts of items used to assess their understanding. Even in items that were basically
the same, students’ responses seemed to be influenced by the problem context and, thus,
on average, only about 60% of the responses were consistent across the two contexts.
Because students do not respond consistently across problem contexts, a learning pro-
gression and related assessment items that accurately reflect student thinking in different
contexts may not produce a single, reliable diagnosis of a student’s level on that progres-
sion. Considering the different proposed uses for learning progressions, it seems as if this
is a much greater problem for standardized testing situations. In a classroom setting, a
teacher could profitably use information about his or her students’ consistency to examine
issues of transfer, helping students to see the same underlying principles applying in a
variety of situations. This is consistent with Finegold and Gorsky’s (1991) call for strate-
gies “which develop abilities to apply general rules to particular instances” (p. 110). In the
classroom, the focus can be on the reasoning students are applying in different situations,
and the consistency with which students respond can provide valuable information about
their developing understanding.
Role of Language in Student Responses

Although the OMC items seem to fairly accurately reflect student thinking, there is a
danger that we will overidentify students as holding the “A” level conceptions. Again, a
classroom teacher could easily remedy this by checking in with his or her students to
find out whether they really believe a force is being carried along with a moving object.
However, this luxury is not available in standardized testing situations, and no “easy fix”
to this problem has been identified.
Science Education
CONCLUSIONS
Interest in learning progressions is increasing rapidly. While there are certainly good rea-
sons to believe that learning progressions have the potential to transform science teaching,
learning, and assessment, caution is advised. The development of a learning progression
and its use in assessing student understanding are challenging endeavors. The learning
progression is, at best, a hypothesis about how student understanding develops, which will
need to be continually modified in response to additional data. Thus, it is important to
recognize the tentative nature of these conjectures about the ways in which student thinking
develops.
While attempts were made to reduce the curriculum dependence of the force and mo-
tion learning progression through the collection of responses from students experiencing
two contrasting approaches to the teaching of this topic, it is still possible that students
experiencing an alternative curriculum may provide a different view of the way in which
understanding develops. The studies reported in this paper were useful in preliminary re-
visions to the learning progression, but longitudinal studies of students as they actually
develop understanding of force and motion would be required to fully validate this learning
progression. Ideally, such studies would involve students learning about force and motion
in different instructional contexts, to examine the curriculum dependence of the proposed
learning progression. In addition, such studies would need to be sensitive to the extent to
which students’ responses to the items are curriculum dependent. At a minimum, items
assessing understanding of the same concept in different contexts would be required.
To develop learning progressions in other areas, there does not seem to be any shortcut
that would eliminate the need for iterative cycles, in which both the learning progression and
its associated items are revised based upon students’ item responses. Longitudinal studies
of student learning in different curricular contexts are needed to fully validate any learning
progression. All of this work takes time but is required to have confidence that the learning
progression is a valid representation of the way in which student thinking develops.
This paper highlights two challenges with respect to assessing students’ learning pro-
gression levels. Although these were explored with respect to the force and motion learning
progression, similar issues might be encountered in assessment involving other learning
progressions.
First, students’ lack of consistency in responding to different item contexts impacts our
ability to obtain reliable estimates of their learning progression levels. As discussed above,
this is less problematic for classroom assessments, since teachers may use information
about students’ differing performance across contexts to enhance instruction. However, for
both classroom and large-scale assessment contexts, students’ lack of consistency points
to the need to assess student understanding across multiple contexts, as it is not safe to
assume a response at a given level in one context will translate to a response at the same
level in another context. This is particularly important as curriculum may influence the
relative difficulty of different item contexts, and a skewed picture of student understanding
may result if only familiar contexts are used to assess student understanding.
Second, concerns about students’ interpretation and use of language raise validity issues.
If understanding of words such as “force” change as students move through the learning
progression, it may be difficult to write items that have the same meaning for students at
different levels. Fortunately, although some students were found to interpret the force and
motion items differently than was intended, most students provided responses to the items
that seemed consistent with those revealed in their interviews. Thus, it does not appear that
the items were incapable of capturing student thinking at particular levels of the learning
progression. However, this would need to be tested for any learning progression in which
Science Education
students may have different interpretations of words, depending upon the level of their
understanding. For example, if one were interested in assessing students’ understanding
with respect to a learning progression about plant nutrition, the way in which students
interpret the word “food” at lower levels of the progression (as dirt or other substances that
the plants “eat”) might differ significantly from the interpretation of this word at higher
levels (as glucose, which plants produce through photosynthesis). Interviews such as those
conducted in Study 2 would be necessary to verify that items capture the thinking of students
with different interpretations of words such as “force” or “food.”
Despite these challenges, OMC items appear to be a promising approach for assessing
students’ learning progression levels. OMC results were consistent with those obtained
in clinical interviews, and sets of these items exhibited relatively low SEMs and modest
reliabilities. By narrowing the range of possible responses and providing students with a
set of conceptions to consider, these items appear to capture student thinking relative to the
learning progression reported here. Thus, this would appear to be a fruitful approach for
others attempting to assess students’ levels on a learning progression.
This paper illustrated the process of iteratively revising a learning progression and
its associated items, evaluated the OMC item type, and explored two challenges with
respect to assessing students’ learning progression levels. Caution is certainly warranted,
and we urge those engaged in work on learning progressions to undertake studies such
as those described here, as well as longitudinal studies, to better represent and assess
student thinking. Careful development, revision, and testing of learning progressions and
associated items are necessary for these tools to achieve their promised potential. Learning
progressions have been proposed for use in both large-scale and classroom assessments.
In both cases, they may provide more detailed information about student thinking than
more traditional models of assessment. This detailed information is particularly important
in the classroom, where it can be used as the first step in a formative assessment process,
to impact instructional decisions and provide feedback to students, ultimately improving
student learning.
Initial development of this learning progression and associated items was conducted at the University
of California, Berkeley, in collaboration with Derek Briggs, Cheryl Schwab, and Mark Wilson. This
work was supported with funding from the National Science Foundation (#REC-0087848, Research
in Standards-Based Science Assessment), as part of a collaboration between the BEAR Center and
WestEd. Support for subsequent research presented in this paper was provided by the National Science
Foundation, through a Center for Learning and Teaching grant to CAESL (Center for Assessment and
Evaluation of Student Learning), and the University of Iowa College of Education Research Fund.
The authors are grateful to Derek Briggs for his thoughtful suggestions for improving earlier versions
of the manuscript.
REFERENCES
American Association for the Advancement of Science. (1993). Benchmarks for science literacy. New York:
Oxford University Press.
Anderson, C. W., Alonzo, A. C., Smith, C., & Wilson, M. (2007, August). NAEP pilot learning progression
framework. Report to the National Assessment Governing Board.
Bell, B., Osborne, R., & Tasker, R. (1985). Appendix A: Finding out what children think. In R. Osborne &
P. Freyberg (Eds.), Learning in science: The implications of children’s science (pp. 151 – 165). Auckland,
New Zealand: Heinemann.
Briggs, D. C., Alonzo, A. C., Schwab, C., & Wilson, M. (2006). Diagnostic assessment with ordered multiple-
choice items. Educational Assessment, 11, 33 – 63.
Science Education
Catley, K., Lehrer, R., & Reiser, B. (2004). Tracing a prospective learning progression for developing understanding
of evolution. Paper commissioned by the National Academy of Sciences Committee on Test Design for K-12
Science Achievement. Retrieved March 23, 2007, from http://www7.nationalacademies.org/bota/Evolution.pdf
Champagne, A., Klopfer, L. E., & Anderson, J. H. (1980). Factors influencing the learning of classical mechanics.
American Journal of Physics, 48, 1074 – 1079.
Chi, M. T. H., Feltovich, P. J., & Glaser, R. (1981). Characterization and representation of physics problems by
experts and novices. Cognitive Science, 5, 121 – 152.
Clement, J. (1982). Students’ preconceptions in introductory mechanics. American Journal of Physics, 50, 66 – 71.
diSessa, A. A. (1983). Phenomenology and the evolution of intuition. In D. Gentner & A. L. Stevens (Eds.),
Mental models (pp. 15 – 33). Hillsdale, NJ: Erlbaum.
Driver, R., Guesne, E., & Tiberghien, A. (1985). Children’s ideas in science. Milton Keynes, England: Open
University Press.
Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: Verbal reports as data. Cambridge, MA: MIT Press.
Finegold, M., & Gorksy, P. (1991). Students’ concepts of force as applied to related physical systems: A search
for consistency. International Journal of Science Education, 13, 97 – 113.
Full Option Science System. (2003, Spring). Force and motion teacher’s guide (Trial ed.). Berkeley, CA: Lawrence
Hall of Science.
Gilbert, J., & Watts, M. (1983). Misconceptions and alternative conceptions: Changing perspectives in science
education. Studies in Science Education, 10, 61 – 98.
Halloun, I. A., & Hestenes, D. (1985). Common sense concepts about motion. American Journal of Physics, 53,
1056 – 1065.
Holt science spectrum: A physical approach. (2001). Austin, TX: Holt, Rinehart & Winston.
Ioannides, C., & Vosniadou, S. (2001). The changing meanings of force: From coherence to fragmentation.
Cognitive Science Quarterly, 2(1), 5 – 62. Retrieved October 30, 2006, from http://www.cs.phs.uoa.gr/el/staff/
vosniadou/force.pdf
Lehrer, R., Schauble, L., Lucas, D., Henrie, A., Taylor, R., & Knapp, N. (2008, March). Development of models
and modeling in the life sciences. In A. C. Alonzo & A. W. Gotwals (Chairs), Diverse perspectives on the
development, assessment, and validation of learning progressions in science. Symposium conducted at the
annual meeting of the American Educational Research Association, New York.
McCloskey, M. (1983). Naive theories of motion. In D. Gentner & A. L. Stevens (Eds.), Mental models (pp. 299 –
324). Hillsdale, NJ: Erlbaum.
Minstrell, J. (n.d.). Facets of students’ thinking. Retrieved October 27, 2006, from http://depts.washington.edu/
huntlab/diagnoser/facetcode.html
National Assessment Governing Board. (2006a). Science assessment and item specifications for the 2009 National
Assessment of Educational Progress (Prepublication ed.). Retrieved July 11, 2006, from http://www.nagb.org/
pubs/naep science specs 2009.doc
National Assessment Governing Board. (2006b). Science framework for the 2009 National Assessment
of Educational Progress (Prepublication ed.). Retrieved July 11, 2006, from http://www.nagb.org/pubs/
naep science framework 2009.doc
National Research Council. (1996). National Science Education Standards. Washington, DC: National Academy
Press.
National Research Council. (2001). Knowing what students know: The science and design of educational assess-
ment. Washington, DC: The National Academies Press.
National Research Council. (2006). Systems for state science assessment. Washington, DC: The National
Academies Press.
National Research Council. (2007). Taking science to school. Washington, DC: The National Academies Press.
Popham, W. J. (2007). The lowdown on learning progressions. Educational Leadership, 64(7), 83 – 84.
Reiner, M., Slotta, J. D., Chi, M. T. H., & Resnick, L. B. (2000). Naive physics reasoning: A commitment to
substance-based conceptions. Cognition and Instruction, 18, 1 – 34.
Reiser, B. J., Schwarz, C., Shwartz, Y., Kenyon, L., Fortus, D., Krajcik, J., et al. (2008, March). MoDeLS:
Articulating a learning progression for scientific modeling. In A. C. Alonzo & A. W. Gotwals (Chairs), Diverse
perspectives on the development, assessment, and validation of learning progressions in science. Symposium
conducted at the annual meeting of the American Educational Research Association, New York.
Roberts, L., Wilson, M., & Draney, K. (1997, June). The SEPUP assessment system: An overview. BEAR Report
Series, SA-97-1. University of California, Berkeley.
Roseman, J. E., Caldwell, A., Gogos, A., & Kurth, L. (2006, April). Mapping a coherent learning progression
for the molecular basis of heredity. Paper presented at the annual meeting of the National Association for
Research in Science Teaching, San Francisco, CA. Retrieved March 23, 2007, from http://www.project2061.org/
publications/articles/papers/narst2006.pdf
Science Education
Smith, C. L., Wiser, M., Anderson, C. W., & Krajcik, J. (2006). Implications of research on children’s learning
for standards and assessment: A proposed learning progression for matter and the atomic molecular theory.
Measurement: Interdisciplinary Research and Perspectives, 14, 1 – 98.
Stanford Education Assessment Laboratory. (2003). FAST teacher’s guide to the reflective lessons Ver 6 22.
Unpublished manuscript, Stanford University, Stanford, CA.
Steedle, J. T. (2006, April). Seeking evidence supporting assumptions underlying the measurement of progress
variable levels. Paper presented at the annual meeting of the National Council on Measurement in Education,
San Francisco, CA.
Trumper, R., & Gorsky, P. (1996). A cross-college age study about physics students’ conceptions of force in
pre-service training for high school teachers. Physics Education, 31, 227 – 235.
Wilson, M. (2005). Constructing measures: An item response modeling approach. Mahwah, NJ: Erlbaum.
Wilson, M. (2008, March). Measuring progressions. In A. C. Alonzo & A. W. Gotwals (Chairs), Diverse per-
spectives on the development, assessment, and validation of learning progressions in science. Symposium
conducted at the annual meeting of the American Educational Research Association, New York. Retrieved
March 30, 2008, from http://myweb.uiowa.edu/alonzo/aera2008.html
Science Education

Alonzo2009 PDF

Загружено:

Сведения о документе

Оригинальное название

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

Alonzo2009 PDF

Загружено:

Авторское право:

Доступные форматы

Developing and Assessing

a Force and Motion Learning

ABSTRACT: Learning progressions are ordered descriptions of students’ understanding

CONSIDERATION OF LEARNING PROGRESSIONS

Broad learning progressions may be thought to comprise more fine-grained descriptions

How Can Learning Progressions Be Developed?

The process of developing a learning progression is detailed by Briggs et al. (2006). To

What Challenges Might We Expect in Assessing Students’ Levels

Consistency of Student Responses. Students’ inconsistency in approaching physics

Development of the Preliminary Force and Motion Learning Progression and

Figure 1. Revisions to a force and motion OMC item.

Study of the Force and Motion Learning Progression

moving at a constant speed in a direction

• Student believes that the object’s speed

◦ 3A: An object may come to rest when

Scoring and Analysis of Student Item Responses

Examining the Consistency of Student Responses. The calculations of reliability de-

Diagnosis of Learning Progression Levels. We first consider the diagnosis of students’

Responses to Individual Items. We now compare student performance on the OMC

as to the student’s meaning; examination of the cross-tabulations and associated OE and

Comparing OMC Responses and Clinical Interviews

Consistency of Student Responses

Role of Language in Student Responses

Consistency of Student Responses

Role of Language in Student Responses

Вам также может понравиться