Once you know what theyve learned, what do you do next?

Designing curriculum and assessment for growth

Dylan Wiliam Institute of Education, University of London www.dylanwiliam.net Presentation to MDSE/MARCES conference; University of Maryland, College Park, MD; October 2006

Outline
Education reform in England and Wales Designing an assessment system to support learning Age-independent levels of achievement Distribution of achievement over time Applications to curriculum specification

A familiar story
Education Reform Act (1988)
An early attempt to use markets to reform education
Choice Diversity Standardization Information

Key features of ERA

Basic curriculum:
Religious education (!) Core subjects (English, Math, Science) Non-core subjects (7 in all)

Four key stages (5-7, 7-11, 11-14, 14-16) Core subjects assessed at end of each key stage Other subjects assessed at some key stages

Task Group on Assessment and Testing (TGAT)

To advise the Secretary of State on the practical considerations which should govern all assessment including testing of attainment at age (approximately) 7, 11, 14 and 16, within a national curriculum; including
the marking scale or scales and kinds of assessment including testing to be used, the need to differentiate so that assessment can promote learning across a range of abilities, the relative roles of informative and of diagnostic assessment, the uses to which the results of assessment should be put, the moderation requirements needed to secure credibility for assessments, and the publication and other services needed to support the system

with a view to securing assessment and testing arrangements which are simple to administer, understandable by all in and outside the education service, costeffective, and supportive of learning in schools.

Task Group on Assessment and Testing (TGAT)

Basic choice
Age-dependent
benchmark assessments at each age-point

Age-independent
linked system of achievement levels across ages

Crucial factors
Technical feasibility Impact on students

Age-dependent levels
Simple to understand Familiar Significant negative impact on student motivation Encourages a notion of ability as fixed rather than incremental

Age-independent levels
In psychology
Piaget (Shayer et al., 1976; Shayer & Wylam, 1978) Pascual-Leone Case SOLO (Biggs & Collis, 1982) Van Hiele CSMS (Hart, 1981)

In Education (or math education at least!)

The Dalton Plan (Parkhurst, 1922) Kent Mathematics Project (Banks, 1991) Secondary Mathematics Individualised Learning Experiment Graded Assessment in Mathematics (Brown, 1992)

Preliminary evidence
6099 + 1 = ? (Foxman et al., 1980)
Correctly answered by some 7-year-olds Incorrectly answered by some 14-year-olds

The seven year gap (Cockcroft, 1981) Progression in measuring (Simon et al., 1995) Spread of achievement in an age cohort apparently much greater than generally assumed

CSMS (Hart, 1981)

Achievement in Decimals by age
100 90 80 70

Proportion

60 50 40 30 20 10 0 1 2 3 4 5 6 Level achieved

Age Age Age Age

12 13 14 14

Sequential tests of educational progress (ETS, 1957)

Annual growth in school attainment (STEP)
1

Annual increase (standard deviations)

0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 5 6 7 8 9 10 Grade 11 12 13 14 15 Reading Writing Listening Soc. Stud. Science Math

Sensitivity to instruction

1 year

Distribution of attainment on an item highly sensitive to instruction

Sensitivity to instruction (2)

1 year

Distribution of attainment on an item moderately sensitive to instruction

Sensitivity to instruction (3)

1 year

Distribution of attainment on an item relatively insensitive to instruction

Sensitivity to instruction (4)

Test IQ-type test (insensitive) NAEP TIMSS Sensitivity index 0 6 8

ETS STEP tests (1957)

Completely sensitive test

8
100

Insensitivity to instruction
Artifact or reality?
Influenced by test construction procedures Influenced by approaches to curriculum Dimensions of progression
Reasoning power Curriculum exposure Maturity

Nature of hierarchies
Hierarchies are partly arbitrary
Division can precede multiplication Integration can precede differentiation

Hierarchies are partly psychological

Some learning sequences appear inevitable
Writing Number skills

Graded Assessment in Mathematics

Intended for all school students, aged 11 to 16 Design requirement: all students should be able to increase by one level per year Upper levels designed to be equivalent to existing national examinations

Proportion of age cohort

ITBS language usage test

Percentile

Grade equivalent

A very simple model

Achievement age is normally distributed about chronological age, with a standard deviation proportional to the chronological age Constant of proportionality varies from around one-sixth to one-half, depending on the kind of curriculum and assessment

Standardized tests
8 7
J J B J

CAT:Q CAT:NV

T1L1 M T1L2 T2L1

6 5 4 3 2 1 0 7 8 J F 7 H 7
B

B B H M M M H J M H B B H 7 F 7 7 7

B H M F M F

H CAT:V F TGAT

T2L2 7 T9L1 T9L2

NFER DH CSMS(M)

9 10 11 12 13 14 15 16 Age (years)

WSRT

The TGAT model

Stage
1 2 3 4

Ages
5-7 7-11 11-14 14-16

Levels
1-3 2-6 3-8 4-10

Curriculum development
Curriculum developers forced to focus on What develops? Models of curriculum
Grade-based models (France, Germany) Social promotion (England, Japan, Sweden) Hybrid models (USA)

Models of differentiation
Same goals, same curriculum, different speeds Same goals, different curriculum Different goals

Models of progression
Good in math, design technology OK in language arts, science Poor in history

Dimensions of progression
Mathematics: reasoning power Science: curriculum exposure English: maturity

Hierarchies in science
1. 2.
3. 4. 5. 6. 7. 8. 9. 10.

Know that light comes from different sources Know that light passes through some materials and not others, and that when it does not, shadows may be formed Know that light can be made to change direction, and that shiny surfaces can form images Know that light travels in straight lines, and this can be used to explain the formation of shadows Understand how light is reflected Understand how prisms and lenses refract and disperse light Be able to describe how simple optical devices work Understand refraction as an effect of differences of velocities in different media
Understand the processes of dispersion, interference, diffraction and polarisation of light

Strengths
Forces a focus on progression in big ideas rather than coverage Supports incremental, rather than entity view of ability Supports strong value-added inferences

Weaknesses
Some subjects fit the model better than others Some (accepted) models of curriculum become non-viable Requires careful articulation between curriculum, standards, and assessment May focus on aspects relatively insensitive to instruction