AAAL 2012 Annual Conference

Diagnostic Testing of ESL Reading Skills:

Relating Test-taking Strategies to
R di Abilities
Reading
Abiliti

Alan Urmston
Hong
g Kong
g Polytechnic
y
University
y

DELTA is
an online English language proficiency assessment system
designed specifically for the Hong Kong academic context
which can

diagnose students
d
d
strengths
h and
d weaknesses
k
track and report on students language gains
be linked to curriculum/teaching/materials
help students to plan their language learning
inform curriculum development

DELTA Track and Component Skills Profile

It is common to assert that diagnostic tests are

i
intended
d d to probe
b the
h strengths
h and
d weaknesses
k
off
learners, but there is virtually no description,
much less discussion of what the underlying
constructs might be that should be operationalized
in valid diagnostic
g
tests (p.28).
p
(Alderson, 2007)

Validating the DELTA:

Th reading
The
di test

Jang (2009) identifies two key issues in the

diagnostic assessment of reading comprehension:
the characteristics of reading skill profiles
and the usefulness of the resulting diagnostic
information for students and teachers
teachers (p.39),
(p 39)
reiterated by Alderson (2007) who suggests that
central to diagnosis must be the provision of usable
feedback either to the learners themselves or to the
diagnoser (p.30).

The DELTA reading subskills

Drawing on taxonomies of reading subskills (Munby, 1978; Weir, 1983;
Alderson & Lukmani, 1989; Jang, 2009), knowledge of the Hong Kong tertiarylevel English language learning and teaching context,
context the DELTA Team initially
identified eight (8) reading subskills to be tested in the DELTA:

Identifying
y g specific
p
information
Interpreting a word or phrase as used by the writer
Understanding main ideas and supporting ideas
Understanding information and making an inference
Understanding an argument made by the writer
Interpreting an attitude or intention of the writer
Understanding grammatical relationships of words or phrases across a text
Identifying text type

The DELTA reading

g component
p
consists of

four testlets (reading text + test items)

increasing difficulty level (e.g. easy, medium, difficult)
items of calibrated difficulty level (Winsteps)
reading texts of the type (topic, theme, genre) that
undergraduate students would be expected to read (both
academic and non-academic)

Component Diagnostic Report: Reading

What a test of reading tests is not simply what its

constructors say iit tests, nor what
h a set off jjudges
d
considers it to test. It must surely and crucially relate
to what happens inside a test-taker
test-takerss head when he
or she responds to an item (p.478).
(Alderson 1990)
(Alderson,

Test-taker stimulated recalls

Post-test interviews were conducted with 10
students who had taken the DELTA a few days
y
previously.
Students were asked to describe the strategies they
had used to answer the questions.
In some cases students did live think-aloud while
answering reading questions.
Strategies used were identified and coded using the
coding
di scheme
h
d
devised
i d by
b C
Cohen
h and
dU
Upton
t ((2007).
)

Identifying specific information

(Ability to locate and recognize information in one spot/in more
than one spot with or without rephrasing within specific context.)
Text level: Easy (DELTA 1-65)
1 65)
Although the cities that the Maya
built were beautiful and the people
worked hard to build them, usually,
only the priests lived in the cities and
other people lived in small villages in
the forests. They lived in small huts
with no windows with the walls made
of poles covered with dried mud,
mud and
the roof made of grass or leaves.

Item
Even though the workers built the cities,
________
A. they were reserved only for priests.*
B. the ordinary people didn't like the
cities.
C. it took them a long time to construct
the cities.
D. people
l thought
h
h they
h were not
comfortable enough.

Identifying specific information

S2. Annie (average ability):
I read the q
question and the options
p
and then I scanned through
g the
paragraph and I found the word 'priests'. I noticed that the first
option has the same word and so I chose A.

S3.
S3 Joyce (high ability):
I read the whole passage quickly and so I knew the answer should be
A. Its an easy passage so I understand very well.

S5. Bill (average ability):

I read the whole passage. I understand it easily coz the sentences are
short and the words are easy to understand. I eliminated the other
options as I knew they dont make sense.

C
Common
strategies
i ffor Identifying
d
if i specific
ifi information
i f
i
Code

Strategy description

T5

Reads the question and then reads the passage/portion to look for clues to the answer,
either before or while considering options.

R7

Reads a portion of the passage rapidly looking for specific information.

information

T24

Selects options through elimination of other option(s) as unreasonable based on

paragraph/overall passage meaning.

R5

Reads the whole passage rapidly.

TW1

Uses the process of elimination (i.e., selecting an option even though it is not
understood out of a vague sense that the other options couldn
understood,
couldntt be correct).
correct)

R4

Reads the whole passage carefully.

TW3

Selects the option because it appears to have a word or phrase from the passage in it
possibly
ibl a key
k word.
d

R6

Reads a portion of the passage carefully.

T22

Selects options
p
through
g vocabulary,
y, sentence,, pparagraph,
g p , or ppassage
g overall meaning
g
(depending on item type).

So, for Identifying specific information (and for

the other subskills)
students use a combination of reading
reading, test management
and test wiseness strategies to answer the items.

So how do reading strategies translate into

reading
di skills
kill or subskills?
b kill ?

Reading strategies are deliberate, goal-directed

attempts to control and modify the readers efforts to
decode text
text, understand words
words, and construct meanings
of text. Reading skills are automatic actions that result
in decoding and comprehension with speed, efficiency,
and fluency and usually occur without awareness of the
components or control involved (p.368).
Afflerbach et al (2008)

Just reads from the start and

straight to the end; reading for
no purpose,
purpose [she] wouldn't stop
or pause. If it's something [she]
can't understand then [she'll]
stop once in a while to check
what [shes] read.

Reads all the questions

first then reads the
relevant part of the text.

DELTA Reading Text Comparison

DELTA Reading
R di P
Person-Item
It
M
Map

More-proficient Readers

G
R
O
W
T
H

Less-proficient Readers

DELTA Reading
R di P
Person-Item
It
M
Map

more proficient readers

what the more

proficient readers
can do

less proficient readers

The challenge
to provide our learners with usable
information to help them know what
they can and cannot do now
now.

Component Diagnostic Report: Reading (revised)

Tentative descriptions of proficiency

Proficient readers (DELTA > 110) can:
Identify specific information in feature articles on less
familiar topics
Understand the main and supporting ideas in academic
articles in fields outside of their course of study
Interpret the
h attitude
d or intention of
f a writer in fictional
f
l
or biographical texts

The challenge to predict growth in our

learners.
learners
DELTATRACK

200
D 180
E
L
160
T
A
140
M
E 120
A
S
100
U
R
E 80

113

120

118

125

60
08/2011
DELTAMeasure

08/2012

08/2013

PredictedMeasure

Graduation
TargetMeasure

The challenge
to help our
g
learners to grow
as readers.

ICOSA
Reading Requirement

References
Afflerbach,P.,Pearson,P.D.&Paris,S.G.(2008).ClarifyingDifferencesbetweenReadingSkills
Afflerbach
P Pearson P D & Paris S G (2008) Clarifying Differences between Reading Skills
andReadingStrategies.TheReadingTeacher,(61)5,InternationalReadingAssociation
Alderson,J.C.(1990).TestingReadingComprehensionSkills(Part2):GettingStudentstoTalk
about Taking a Reading Test (A Pilot Study) Reading in a Foreign Language (7)1 pp 465
aboutTakingaReadingTest(APilotStudy).ReadinginaForeignLanguage(7)1,pp.465
503.
Alderson,J.C.(2007).TheChallengeofDiagnosticTesting:DoWeKnowWhatWeAre
Measuring? In J Fox M Wesche D Bayliss L Cheng C Turner & C Doe (Eds ) Language
Measuring?InJ.Fox,M.Wesche,D.Bayliss,L.Cheng,C.Turner&C.Doe(Eds.)Language
TestingReconsidered.Ottawa:UniversityofOttawaPress,pp.2140.
Alderson,J.C.&Lukmani,Y.(1989).CognitionandReading:CognitiveLevelsasEmbeddedin
Test Questions Reading in a Foreign Language (5)2 pp 253 270
TestQuestions.ReadinginaForeignLanguage(5)2,pp.253270.
Jang,E.E.(2009).DemystifyingaQMatrixforMakingDiagnosticInferencesaboutL2Reading
Skills.LanguageAssessmentQuarterly,(6),pp.210238.
Munby J L (1978) Communicative Syllabus Design Cambridge University Press
Munby,J.L.(1978).CommunicativeSyllabusDesign.CambridgeUniversityPress
Weir,C.J.(1983).Identifyingthelanguageproblemsofoverseasstudentsintertiaryeducationin
theU.K. UnpublishedPhDthesis,InstituteofEducation,UniversityofLondon.

For more information on DELTA

http://gslpa.polyu.edu.hk/eng/delta web/
http://gslpa.polyu.edu.hk/eng/delta_web/