Language Assessment – for Item Writers in Ha Tay

LANGUAGE ASSESSMENT PRINCIPLES

AND TIPS AND TECHNIQUES FOR DESIGNING
MULTIPLE CHOICE QUESTIONS
(for items writers in Ha Tay’s high schools )

PART A
LANGUAGE ASSESSMENT PRINCIPLES: AN OVERVIEW
Assessment Can…
 increase motivation by serving as milestones of student progress.
 aid in the reinforcement and retention of information
 confirm areas of strength and pinpoint areas needing further work.
 provide a sense of periodic closure to modules within a curriculum.
 promote student autonomy by encouraging student self-evaluation of their progress.
 encourage learners to set goals for themselves
 aid in evaluating teaching effectiveness
Brown, D.H. (2004). Language assessment: Principles and classroom
practices. Longman.

Curriculum and Assessment

“Assessment is an integral part of the entire curriculum cycle, not something tacked on as an
afterthought to teaching.”
Coombe, C., Folse, K., & Hubley, N. (2007). Assessing English Language
Learners. Univ. of Michigan Press.

Evaluation: All the factors that influence the teaching/learning process

Assessment: A variety of ways of collecting information on a learner‟s ability or
achievement
Testing: One type of assessment: a formal, systematic way to gather information
about students‟ behavior

What is a test? A measurement of a person‟s

ability
knowledge
(specific) competence

“A test is a method of measuring a person’s ability or knowledge in a given domain”

Brown, D.H. (2001). Teaching by principles. Longman.

Ha Tay Department of Education and Training, April 2008 1

Language Assessment – for Item Writers in Ha Tay

Types of Tests
placement to place students at appropriate level of instruction within a program
diagnostic to identify students‟ strengths and weaknesses
progress to provide information about mastery of the course materials
achievement to evaluate students‟ attainment of course outcomes at end of course
standardized to measure students‟ proficiency using international standards
aptitude to measure capacity for learning a language

Traditional Assessment Alternative Assessment

 One-shot standardized exams
 Timed, multiple-choice format
 Decontextualized test items
 Scores suffice for feedback
 Norm-referenced scores
 Focus on the “right” answer
 Summative
 Oriented to the product
 Non-interactive performance
 Fosters extrinsic motivation
 Formal, indirect, traditional
 Objective
 Focused on linguistic/logical-
mathematical intelligence
Adapted from Armstrong (1994) and
Bailey (1998) in Brown, D.H. (2001).
 Continuous long-term assessment
 Untimed, free-response format
 Contextualized, integrative,
communicative tasks
 Interactive feedback
 Criterion-referenced scores
 Formative
 Oriented to process
 Interactive performance
 Fosters intrinsic motivation
 Informal, direct, authentic
 Subjective
 Computer-based/computer-adaptive
 Inclusive of multiple intelligences
(spatial, musical, bodily-kinesthetic,
interpersonal, intrapersonal)

Guiding Principles of Assessment

 Usefulness
 Validity: face, content, construct, concurrent, consequential, predictive
 Reliability
 Practicality
 Positive washback
 Authenticity
 Transparency
 Security
 Interactivity
Washback
 the effect of testing on teaching and learning(including curriculum and materials)
 best kind of assessment has a positive effect on how well students learn
 „good‟ washback relates the test content and tasks to the target environment of
language use
 „bad‟ washback can lead to „teaching to the test‟ > loss of focus on learning

Assessment of English in Vietnam

English can be assessed in the: classroom, school, region, country
AND/OR
worldwide > international benchmarking

Ha Tay Department of Education and Training, April 2008 2

Language Assessment – for Item Writers in Ha Tay

Local Focus or International Benchmark

Should assessment:
… use an international proficiency test? … be linked to a specific course? … be
discipline-specific? … focus on the assumed future study needs of students? … be
linked to student standards and teacher standards?
Which English?
 Does international benchmarking presuppose one form of English as „correct‟?
 Do international benchmarked tests „own‟ English?
 Which variety(ies) of English is/are acceptable?

Standardized Tests
 Standard objectives or criteria held constant across one form of the test to another
 Designed to apply to a broad band of competencies not necessarily linked to a
particular curriculum
 Produced through extensive research and development
 Dictates standard procedures for administration and scoring
 Norm-reference: places test-takers on a continuum across a range of scores;
differentiates test-takers by relative ranking

High Stakes Testing Issues

 Despite high probabilities of accurately assessing test-takers, what about the minority
who are not fairly assessed?
 Should more types of performance be included to get a more comprehensive picture
of the test-taker (speaking, writing)?
 Does the high-stakes test negatively impact learners‟ lives?
 Is teaching driven by the test?
 Does standardized testing promote a specific cultural, political, or social agenda?

Ongoing Issues: the Vietnamese Situation

 Combining graduation and university entrance exams
 Addition of „communicative‟ elements into tests
 The relationship between the curriculum and the test
 The match of textbooks to test content
 Teacher training and testing
 Discrete items or linked items in test content
 Longer test vs. shorter test
 Standard test administrative procedures
 Technology in testing
 Others????

Assessment of Reading

Microskills for Reading Comprehension Macroskills for Reading Comprehension

 Recognition of letters, words, and
spelling
 Ability to retain „chunks‟ of language
in short-term memory
 Ability to process written text at an
efficient speed
 Processing of vocabulary and
 Recognition of rhetorical forms of
written discourse
 Recognition of the purpose and
communicative function of a text
 Making inferences by using
background knowledge
 Inferring connections and relationships

Ha Tay Department of Education and Training, April 2008 3

Language Assessment – for Item Writers in Ha Tay

grammatical patterns within a text

 Differentiation of word classes (noun,
verb etc.)
 Recognition of alternate meanings and
grammatical forms
 Recognition and utilization of textual
cohesive devices which indicate
relationships between ideas
 Distinction of literal from implied
meanings
 Appropriate interpretation of cultural
references
 Use of reading strategies to extract
meaning from text
 skimming, scanning, guessing,
activating background knowledge,
detecting discourse markers

Characteristics of Intensive Reading Strategies for Intensive Reading

Comprehension
 Emphasis on the quality of reading  Preview and make predictions
 Short texts  Identify the purpose in reading
 Difficult texts (i + 1)  Use graphemic rules and patterns to aid
 Usually authentic texts in bottom-up decoding
 Many tasks to complete after reading  Use efficient silent reading techniques
 Reading is slow and careful for relatively rapid comprehension
 Need to understand the text in detail  Skim the text for main idea
 Primarily for language study and  Scan the text for specific information
reading skills‟ development  Use semantic mapping or clustering
 Teacher-driven  Guess when uncertain (vocabulary,
 Teacher selects texts inference, discourse relationship)
 Teacher often intervenes in the reading  Analyze vocabulary: word part, roots,
process context
 Teacher determines where and when to  Distinguish between literal and implied
read meanings
 Reading is usually done in the  Capitalize on discourse markers to
classroom process relationships and organization

Davidson, P. (2005)

Bottom-up and Top-down Processing

“Research has shown that reading is only incidentally visual. More information is contributed
by the reader than by the print on the page. That is, readers understand what they read
because they are able to take the stimulus beyond its graphic representation and assign it
membership to an appropriate group of concepts already stored in their memories… Skill in
reading depends on the efficient interaction between linguistic knowledge and knowledge of
the world. Clarke, M. & Silberstein, S. (1977)
Activating Prior Knowledge Activating Activities
(schema/schemata)
 Facilitates comprehension  Pre-reading discussion
 Unlocks vocabulary before reading  Semantic maps
 Makes readers more independent  Visuals: pictures, photographs, charts
 Expectations of rhetorical patterns
 Predictions/hypotheses

Ha Tay Department of Education and Training, April 2008 4

Language Assessment – for Item Writers in Ha Tay

The Role of Vocabulary in Reading

Research Studies
 Levine & Reeves (1990) cite “lack of adequate vocabulary… as one of the obstacles
to text comprehension” and point out that “frequent and numerous recycling should
be…emphasized”
 Grabe (1991) cites the important role of vocabulary as a predictor of overall reading
ability
 Nation (1990) emphasizes “ a systematic and principled approach to vocabulary by
both the teacher and the learners”

Extensive Reading = Extensive Vocabulary

 Anderson (1996): the average 12th grader knows about 80,000 words in L1; children
learn up to 4,000-6,000 new words each year
 Miller (1977): children between 6-8 pick up an average of 14 new words a day
 Nagy & Herman (1987): children acquire 3,000 words a year between grades 3-12

Five-Step Guessing Strategy

How successful is guessing?
1. Look at the unknown word and decide its part of speech.
2. Look at the context (clause, sentence) What word goes with what?
3. Look at the relationship with other sentences. Are there signal words?
4. Guess the word.
5. Check that the guess is correct.

L1 Reading Rates Carver, R. P. (1990)

Reading Process Processing Components Target Words Per

Minute
Scanning Lexical accessing 600
Skimming Semantic encoding 450
“Rauding” Sentence integrating 300
Learning Idea remembering 200
Memorizing Fact rehearsing 138

Reading Rate Goal Slow Reader Problems

For intermediate learners: These activities slow down reading:
 Pronouncing every word while reading
 Moving the lips while reading silently
200 wpm + 70% comprehension  Using a finger or pen to follow the
words
 Translating
 Reading each word individually instead
of in chunks
 Looking up all or some unknown words

Ha Tay Department of Education and Training, April 2008 5

Language Assessment – for Item Writers in Ha Tay

Reading Assessment
 A reading log of material read
 Reading rate chart
 Reading portfolio
 Reading reports/journals
 Oral reports
 Comprehension questions in a variety of formats
 Quizzes/tests

Designing Assessments for Reading Comprehension

Cloze Multiple Choice

The ability to fill in gaps in a written text Most common reading testing technique
using linguistic expectations, background Tests according to specifications, e.g.
knowledge, and strategic competence  Main idea
 A minimum of 2 paragraphs  Vocabulary in context
 Fixed-ratio deletion (every 7th word)  Expressions/idioms/phrases in context
 Rational deletion (choosing deletions)  Inference
 Scoring: exact word or appropriate  Grammatical features
word  Details
 Multiple-choice format  Unstated details
 Variations: C-test; cloze-elide  Supporting ideas
procedure  Computer-based additions: clicking on
word or picture, adding a sentence
Short Answer Editing
 Test-takers construct own answers  Texts of 200-300 words
 Range of answers might be acceptable  Introduction of errors in numbered
sentences
 Multiple-choice format
 May test writing more than reading

Sequencing Information Transfer

 Overall global understanding of a text reading>graphic OR graphic > writing,
 Use of cohesive devices speaking
 Useful for relatively short texts Learners need to:
 comprehend conventions of graphic
types
 comprehend labels, headings, numbers,
symbols
 understand relationships among
elements of graphic
 make inferences

Multiple Choice Question Types for Reading

Main idea  What is the topic of this passage?

 What is the main idea expressed in this passage?
 Which title best reflects the main idea of the passage?

Factual Questions  According to the passage, why did…

Ha Tay Department of Education and Training, April 2008 6

Language Assessment – for Item Writers in Ha Tay

 According to the information in paragraph 2, where did…

 Which of the following is true, according to the author?

Negative factual  The author mentions all of the following in the passage
questions EXCEPT …
 In the passage, the author does NOT provide a specific
example of …
 Which of the following is LEAST likely?
 According to the information in paragraph 3, which of the
following is NOT …

Vocabulary  Which of the following could best be substituted for the word
questions _____ in line 9?
 The expression “_____” in line __ could best be replaced by …
 The word “_____” in line ___ is closest in meaning to which of
the following?

Inference  It can be inferred from the passage that …

Questions  It is probable that…
 It can be concluded from the information in paragraph 2 that…
 In paragraph 4, the author implies that…
 Which of the following can be inferred from the passage?
 From the information in paragraph 4, it can be inferred that…
 In the last paragraph, the author suggests that…

Author‟s purpose  Why does the author mention _____ in paragraph 2?

 Why does the author give details about _____?
 The author refers to _____ to indicate that…
 The author‟s main purpose in paragraph 3 is to …
 The author refers to _____ to indicate that …

Author‟s attitude  What is the author‟s opinion of _____?

 Which of the following most accurately reflects the author‟s
opinion of _____?

Sentence  Which of the following best expresses the essential information

restatement in the sentence below?

Reference  The word _____ in line 4 refers to …

questions  The pronoun “it” in line 5 refers to …
 The phrase _____ in paragraph 4 refers to…
 The word ____ is a reference to …

Assessment of Vocabulary
Workshop Explorations
 What is a word?
 Does vocabulary consist of single words or as larger lexical items?
 What does it mean to know a lexical item?
Ha Tay Department of Education and Training, April 2008 7
Language Assessment – for Item Writers in Ha Tay

 What should be measured in a vocabulary test?

 How should vocabulary be measured?
 What is better – discrete-item testing or contextualized-item testing?
 And more…

What is a Word?
 Single words: chair
 Phrasal verbs: put on
 Short phrases: as a matter of fact
 Longer phrases: it has come to my attention that…
 idioms: out to lunch

Larger Lexical Items

 Phrasal verbs: move out, put up with, put off
 Compound nouns: personal computer, Applied Linguistics
 Idioms: let the cat out of the bag, raining cats and dogs
 Collocations: a piercing scream, analyze data
 Lexical phrases
Polywords: short fixed phrases
 at any rate, so to speak, as a matter of fact
Institutionalized expressions: proverbs, formulaic utterances
 once upon a time, long time no see
Semi-fixed expressions: basic frame + options
 a day/month/year ago, yours sincerely/truly, as far as I know/can tell/
am aware
Sentence builders: framework for complete sentence with slots for whole
ideas
 I think that…, not only… but also…, I regret to inform you that…
Characteristics of lexical phrases
 Relatively fixed in form
 The meanings of the individual words do not provide the meaning for
the phrase
 Familiar expressions for everyday communication
 Pragmatic function

What Does it Mean to Know a Word? Components of Word Knowledge

Spoken Form
 (R) What does the word sound like?
 (P) How is the word pronounced?
Written Form
 (R) What does the word look like?
 (P) How is the word written and spelled?
Grammatical Patterns
 (R) In what patterns does the word occur?
 (P) In what patterns must we use the word?
Collocations
 (R) What words or types of words can be expected before or after the word?
 (P) What words or types of words must we use with this word?
Frequency

Ha Tay Department of Education and Training, April 2008 8

Language Assessment – for Item Writers in Ha Tay

 (R) How common is the word?

 (P) how often should the word be used?
Appropriateness
 (R) where would we expect to meet this word?
 (P) Where can this word be used?
Concept
 (R) What does the word mean?
 (P) What word should be used to express this meaning?
Associations
 (R) What other words does this word make us think of?
 (P) What other words does this word make us use instead of this one?
Nation, 1990: 31

Four Stages of Knowing a Word

Stage 1: I never saw it before.
Stage 2: I’ve heard of it, but I don’t know what it means.
Stage 3: I recognize it in context – it has something to do with…
Stage 4: I know it.
(Stage 5: I can distinguish this word from others that are closely related to it.)

Dale, 1965: 898

What is Vocabulary Ability?
 The context of vocabulary use
generational usage
language variations
everyday usage/specialized terminology
 Vocabulary knowledge and fundamental processes
vocabulary size
knowledge of word characteristics
lexical storage
fundamental vocabulary processes
 Metacognitive strategies for vocabulary use
strategic competence (paraphrasing, avoiding, language switching, appealing
to authority, using general terms)

Types of Vocabulary Testing

I. Matching items
Next to each word, write the letter of its meaning.

a. Gas that we breathe

1. Region ____ b. Part of a country
2. Status ____ c. Smallest part of living things
3. Cell ____ d. Useful liquid
4. Oxygen ____ e. Set of bones in the
5. Skeleton ____ f. Exciting event
g. Position in relation to others

II Matching format
Find the word which fits in each sentence and write it in the blank at the end.

Ha Tay Department of Education and Training, April 2008 9

Language Assessment – for Item Writers in Ha Tay

a. violent
1. Someone who is not worried about life is _____. b. secure
2. A metal that is not processed is _____ c. crude
3. An illness that never gets better is _____ d. hostile
4. Clothes that fit close to your body are _____ e. tight
5. Someone who hits other people hard is _____ f. chronic
g. parallel

III. Blank-filling/completion
Write one suitable word in each blank.

a. Modern planes fly at an a__________ of 35,000 feet.

b. Many diamond mines are l_________ in South Africa.
c. Near the Equator, there is not much seasonal v_________ in temperature.

IV. Multiple-choice matching

Last week while Jack and Linda were having lunch in a café, thieves (1)broke down the front
door of their house, went inside, and (2)broke into their safe. Now they‟re (3)broke.
_____ a. poor
_____ b. made into two or more pieces
_____ c. entered to steal something
_____ d. entered by force

V. Sentence-writing
Write a sentence to show you know what the word means and how it is used.
a. vegetation
b. involve
c. twist
d. starve
e. principal

VI. Testing depth of processing

To Interpret
1. Write two sentences: A and B. In each sentence, use the two words given.
A. interpret experiment
B. interpret language
2. Write three words than can fit in the blank.
to interpret a(n) i __________
ii __________
iii __________
3. Write the correct ending for each word:
a. Someone who interprets is an interpret____.
b. Something that can be interpreted is interpret_____.
c. Someone who interprets gives an interpret_____.

VII. Multiple-choice Vocabulary Items

Little ongoing research available, except for TOEFL
 Difficult to construct; require field-testing, analysis, refinement

Ha Tay Department of Education and Training, April 2008 10

Language Assessment – for Item Writers in Ha Tay

 Learner may know an alternate meaning of the word, but not the one sought
 25% chance of guessing correctly
 Items may test knowledge of distractors instead of exact meaning of target word
 Wrong answer may result from lack of understanding of syntax
 Permits only limited sampling of total vocabulary knowledge
 BUT: convenient to administer; well-established procedures to analyze them
Weshe & Paribakht, 1996: 17

Assessment of Pronunciation
Possible Pronunciation Difficulties for Vietnamese Speakers

 Omission of word-final voiceless stop consonants: /p/, /t/, /k/: bite > bi
 Difficulty in differentiating between voiced and voiceless stops in word-final
position: /b/ and /p/, /d/ and /t/, /g/ and /k/: cup > cub, pat > pad, back > bag
 Omission of word-final consonants /f/, /v/, /th/, /s/, /z/, /sh/, /z/
 The boys always pass the garage on their way home > the boy alway pa(t) the gara
on their way home.
 Tendency to delete one or more consonants from a consonant cluster: first > firt,
street > stheet
 Substitution of /n/ for /l/: children > chinren
 Difficulty in pronouncing the voiceless and voiced /th/:
 this > zis, think > tink
 Substitution of /s/ for /sh/: delicious > deli/s/ious
 Tendency to substitute /sh/ for /ch/:
 church > shursh, child > shild
 Substitution of word initial /p/ for /b/ or /f/:
 pen > ben/fen, pig > big/fig
 Some difficulty differentiating between vowels:
 bed/bad, hut/hot
 Multi-syllabic words tend to lack the correct stress patterns:
 grandfather, embassy
 Lack of linking in connected speech:
 I‟ve gotta go > I have got to go.
 Difficulty pronouncing contractions with consonant clusters: I would‟ve gone if…
 Difficulty with English intonation patterns

Problem Consonants Problem Vowels

t /iy/ (beat)
k /I/ (bit)
b /ey/ (bait)
d /ɛ/ (bet)
g /uw/ (boot)
th
z /ʊ/ (book)
s /æ/ (bat)
v
f
sh

Ha Tay Department of Education and Training, April 2008 11

Language Assessment – for Item Writers in Ha Tay

ch

General Rules for Word Stress

Longer, louder, higher in pitch

 90% of two-syllable nouns have stress on the first syllable: REcord, SURvey
 60% of two-syllable verbs have the stress on the second syllable: reCORD,
surVEY
 The first part of a noun compound is stressed: GREENhouse, SUNshine
 The preposition part of a phrasal verb is stressed: watch OUT, get UP
 The suffixes „er‟ and „or‟ are not stressed
 The first syllable after a prefix is usually stressed: unHEALTHy, proPOSal
 BUT nouns: FOREcast, OUTlook, OVercoat, UNderdog, UPkeep
 The stress patterns of multisyllabic words need to be learned

Assessment of Grammar

Goals of this section:

 To review the components of grammatical knowledge, ability, and performance
 To explore grammar test tasks
 To review types of multiple choice grammar questions
 To design multiple choice questions and answers to test specific grammar points

What is Grammar? Form-based, Form and use-based perspective, and Communication-

based perspective

Grammatical Form Grammatical Meaning Pragmatic meaning

Phonological/ Phonological/ Contextual meanings

graphological forms graphological meanings
Lexical forms Lexical meanings Sociolinguistic meanings

Morphosyntactic forms Morphosyntactic meanings Sociocultural meanings

Cohesive forms Cohesive meanings Psychological meanings

Information management Information management Rhetorical meanings

forms meanings
Interactional forms Interactional meanings

Low to high context Low to high context High context

Ha Tay Department of Education and Training, April 2008 12

Language Assessment – for Item Writers in Ha Tay

Grammatical form Grammatical Pragmatic meaning

meaning
F. Straighten your F. Directive order F. Directive order F. Expression of power
room
D. OK D. Agreement to do D. Agreement to do as D. Acceptance of
as ordered ordered power relationship
F. How can you live F. Seeking factual F. Request for action F. Expression of
like this? information criticism, disgust,
rudeness.
D. Okay, I‟ll clean D. Acknowledgment D. Agreement to do D. Acknowledgment of
it up. of request and something problem and
agreement to do agreement to
what‟s needed maintain harmony
and obey parent

Grammar Test Tasks

Selected-response tasks Limited production tasks Extended-production tasks
Multiple-choice Gap-filling Essays, summaries
True/false Cloze Dialogs, interviews
Matching Short answer Role-plays, simulations
Discrimination activities Dictation Stories, reports
Word lists activities Information transfer Information-gay
Grammaticality judgment Dialog completion Problem-solving
activities Decision making
Noticing activities

PART B:
TIPS AND TECHNIQUES FOR DESIGNING MULTIPLE CHOICE QUESTIONS

Start with Test Specifications

Addresses curriculum objectives in terms of performance (can do)

Outline of the test
Skills and topics to be included
Item types and tasks
Elicitation and response formats for items
Number of items in each section
Time allocation and plan for scoring

Advantages of Multiple Choice Items

good discrimination among levels of ability
high reliability
valid indicators of overall language ability
highly practical: convenient to administer
well-established procedures to analyze items
writing ability does not affect performance

Ha Tay Department of Education and Training, April 2008 13

Language Assessment – for Item Writers in Ha Tay

cost-effective format - easy scoring and grading

familiarity of students with MCQs

Disadvantages of Multiple Choice Items

tendency to treat words as independent linguistic units
difficulty in construction
field-testing, analysis, refinement required
25% chance of guessing correctly
items may test knowledge of distractors
wrong answer may result from lack of understanding of syntax (English)
permits only limited sampling of total knowledge
tests only recognition knowledge
restricts what can be tested
washback may be harmful
cheating may be facilitated

Terminology

Every multiple choice item has a stem, which presents a stimulus and several options or
alternatives to choose from. One of these options, the key, is the correct response. The
others serve as distractors

Multiple choice questions

Each test item should be designed to measure a specific objective

The question or task should be clear, testing only one concept
The answer should involve interpretation of the material, not just background knowledge
A context should be provided for the items
The items should not show an insensitive/unfair bias
The number of response options should be the same in each question
Options should generally not include: None of the above; both a. and b.
Ambiguity in the answers must be avoided; the intended answer should clearly be the
only correct one.
The options should all be similar in length and level of difficulty
Both the stem and options should be stated as simply and directly as possible
Writing illogical or nonsensical distractors should be avoided
Extraneous clues should not influence the answer
The correct answers should appear in all positions (A, B, C, D)
Words should not be repeated in the answers
A negative „stem‟ should be avoided – make it positive
The questions should not be dependent on previous items
Appropriate punctuation should be used.

For more materials and resources, please visit the following emails:
dayhoctienganh@gmail.com
dayhoctienganh@yahoo.co.uk

Ha Tay Department of Education and Training, April 2008 14