The Dos and Donts of writing MCQs (multiple choice questions) version 4 1

_________________________________________________________________________________________________

1. Strengths and limitations of MCQs (Zimmaro, 2004:11) Strengths: 1. Achievement of learning outcomes from simple to complex can assesse. 2. Highly structured and clear tasks are provided. 3. A broad sample of achievement can be assessed. 4. Incorrect alternatives provide diagnostic information. 5. Scores are less influenced by guessing than true-false items. 6. Scores are more reliable than subjectively scored items (e.g. essays). 7. Scoring is easy, objective, and reliable. 8. Item analysis can reveal how difficult each item was and how well it discriminated between the strong and weaker students in the class 9. Achievement can be compared from class to class and year to year 10. Can cover a lot of material very efficiently (about one item per minute of testing time for straightforward questions). 11. Items can be written so that students must discriminate among options that vary in degree of correctness. 12. Avoids the absolute judgments found in True-False tests. Limitations: 1. Constructing good items is time consuming. 2. It is frequently difficult to find plausible distractors. 3. Can be ineffective for assessming some types of problem solving and the ability to organize and express ideas. 4. Real-world problem solving differs a different process is involved in proposing a solution versus selecting a solution from a set of alternatives. 5. Scores can be influenced by reading ability. 6. There is a lack of feedback on individual thought processes it is difficult to determine why individual students selected incorrect responses. 7. Students can sometimes read more into the question than was intended. 8. Often focus on testing factual information and fails to test higher levels of cognitive thinking. 9. Sometimes there is more than one defensible correct answer. 10. They place a high degree of dependence the instructors writing ability. 11. Does not provide an assessment of writing ability. 12. May encourage guessing. 2. Parts of a multiple choice question (Bull & Mckenna, 2002)

A traditional multiple choice question (or item) is one in which a student chooses one answer from a number of choices supplied. A multiple choice question consists of a:

stem - the text of the question options - the choices provided after the stem (these include the key and the distractors) the key - the correct answer in the list of options distracters - the incorrect answers in the list of options

Note that this version does not go into statistical analysis of MCQs

3. Some examples of dos and donts (Bull & Mckenna, 2002, Kehoe, 1995, Zimmaro, 2004) Begin writing items well ahead of the time when they will be used this allows time for revision and peer review. Before writing the stem, identify the single idea to be tested by that item. This should be about an important aspect of the content area and not with trivia. In general, the stem should not pose more than one problem, although the solution to that problem may require more than one step. Be sure that each item is independent of all other items (i.e. a hint to an answer should not be unintentionally embedded in another item). Design each item/question so that it can be answered by 60-65% of the student cohort (Zimmaro, 2004:15) 3.1 Writing Stems

(i) Present a single, definite statement or direct question to be completed or answered by one of the several given choices A. original stem B. improved stem

Polysaccharide Polysaccharides of the plant cell wall are synthesized a. are made up of thousands of smaller units mainly in the called monosaccharides a. endoplasmic reticulum b. are NOT found in the aloe vera leaf b. cytosol c. are created during photosynthesis c. plasma membrane d. can be described by the chemical formula: CHHOH d. Golgi complex

In Example A, there is no sense from the stem what the question is asking. Example B more clearly identifies the question and offers the student a set of homogeneous choices.

(ii)

Avoid unnecessary and irrelevant material in the stem. It should be clear and unambiguous
B. improved stem

A. original stem:

Paul Muldoon, an Irish postmodern poet who uses Paul Muldoon uses which poetic genre in "Why experimental and playful language, uses which Brownlee Left"? poetic genre in "Why Brownlee Left"? a. sonnet a. sonnet b. elegy b. elegy c. narrative poem c. narrative poem d. dramatic monologue d. dramatic monologue e. haiku e. haiku

Example A contains material irrelevant to the question. This sort of material should not be used to make the answer less obvious. This tends to place too much importance on reading comprehension as a determiner of the correct option (iii) Use clear, straightforward language in the stem of the item.

Questions that are constructed using complex or imprecise wording may become a test of reading comprehension rather than an assessment of whether the student knows the subject matter.
A. original stem B. improved stem

As the level of fertility approaches its nadir, what is A major decline in fertility in a developing nation is the most likely ramification for the citizenry of a likely to produce a developing nation? a. decrease in the workforce participation rate a. a decrease in the workforce participation of women rate of women b. dispersing effect on population concentration b. a dispersing effect on population c. downward trend in the youth dependency concentration ratio c. a downward trend in the youth dependency d. broader base in the population pyramid ratio e.an e. increased infant mortality rate d. a broader base in the population pyramid e. an increased infant mortality rate

(iv) Use negatives sparingly in the stem. If negatives must be used, capitalize, underscore, embolden or otherwise highlight them. Negatives include except, only
A. original stem Which one of the following is not a symptom of osteoporosis? a. b. c. d. decreased bone density frequent bone fractures raised body temperature lower back pain B. improved stem Which one of the following is a symptom of osteoporosis? a. b. c. d. decreased bone density raised body temperature hair loss painful joints

Negatives in the stem usually require that the answer be a false statement. Because students are likely in the habit of searching for true statements, this may introduce an unwanted bias.

(v) Put as much of the question in the stem as possible, rather than duplicating material in each of the options.
A. original stem Theorists of pluralism have asserted which of the following? a. The maintenance of democracy requires a large middle class. b. The maintenance of democracy requires autonomous centres of contervailing power. c. The maintenance of democracy requires the existence of a multiplicity of religious groups. d. The maintenance of democracy requires a predominantly urban population. e. The maintenance of democracy requires the separation of governmental powers. B. improved stem Theorists of pluralism have asserted that the maintenance of democracy requires a. b. c. d. e. a large middle class autonomous centres of contervailing power existence of a multiplicity of religious groups a predominantly urban population separation of governmental powers

Another example: If the point of an item is to associate a term with its definition, the preferred format would be to present the definition in the stem and several terms as options, rather than to present the term in the stem and several definitions as options.

(vi) Avoid irrelevant clues to the correct option in the stem. Grammatical construction, for example, may lead students to reject options which are grammatically incorrect as the stem is stated. Perhaps more common and subtle, though, is the problem of common elements in the stem and in the answer. Consider the following item:
What led to the formation of the States Rights Party? a. The level of federal taxation b. The demand of states for the right to make their own laws c. The industrialization of the South d. The corruption of federal legislators on the issue of state taxation

One does not need to know U.S. history in order to be attracted to the answer, b.

3.2

Writing distractors

(Zimmaro D. 2004, Bull & Mckenna, 2002, Kehoe, 1995, Nitko, 2001, Parkes) This is more difficult than writing stems. Theyre called distracters because they are strategically designed to attract examinees who havent completely mastered the content and skills. This isn't tricky or deceptive or unfair. It is because the goal of testing is to find out who has learned the content and can apply skills and who has not, perhaps along a continuum between the two. Students who mastered the material should recognize the key (correct answer) and those who havent should not. (Parkes) (i) Decide on how many distractors to write
4

According to Nitko (2001) there is no magic number that you should use. A 1987 study by Owen & Freeman suggests that three choices are sufficient. Clearly, the higher the number of distracters, the less likely it is for the correct answer to be chosen through guessing (providing all alternatives are of equal difficulty) (Bull & Mckenna, 2002). Be satisfied with three or four well constructed options. Generally, the minimal improvement to the item due to that hard-to-come-by fifth option is not worth the effort to construct it (Kehoe, 1995). (iii) Follow these hints to avoid test validity problems

1. Try to write items in which there is one and only one correct or clearly is the best answer and one on which experts would agree. 2. Be sure wrong answer choices (distractors) are at least plausible. For example, a distractor can be correct but not answer the question. However, the distractor must not be so close to the correct answer that it confuses students who really do know the answer. 3. Incorporate common student misunderstandings or errors in distractors. 4. The position of the correct answer should vary randomly from item to item. After the options are written, vary the location of the answer on as random a basis as possible. A convenient method is to flip two (or three) coins at a time where each possible Head-Tail combination is associated with a particular location for the answer. Students should be informed that the locations are randomized. (Testwise students know that for some instructors the first option is rarely the answer.) 5. Avoid overlapping alternatives. For example, in the original form of this item, if either of the first two alternatives is correct, C is also correct.)
Original 1. During what age period is thumb-sucking likely to produce the greatest psychological trauma? A. Infancy B. Preschool period C. Before adolescence D. During adolescence E. After adolescence Revised 1. During what age period is thumb-sucking likely to produce the greatest psychological trauma? A. From birth to 2 years old B. From 2 years to 5 years old C. From 5 years to 12 years old D. From 12 years to 20 years old

6. The length of the response options should be about the same within each item (preferably short). Adherence to this rule avoids some of the more common sources of biased cueing. For example, we sometimes find ourselves increasing the length and specificity of the answer (relative to distractors) in order to insure its truthfulness. This, however, becomes an easy-to-spot clue for the testwise student. The number of students choosing a distractor should depend only on deficits in the content area which the item targets and should not depend on cue biases or reading comprehension differences in favour of the distractor

7. There should be no grammatical clues to the correct answer.

Original 1. Albert Eisenstein was a: A. Anthropologist. B. Astronomer. C. Chemist. D. Mathematician Revised 1. Who was Albert Einstein? A. An anthropologist. B. An Astronomer. C. A chemist D. A mathematician

8. Avoid excessive use of negatives and/or double negatives and words such as always, never, and all. 9. Avoid the use of All of the above, both a. and e. above, and None of the above in the response alternatives, when students are asked to choose the best answer. In the case of All of the above, students only need to have partial information in order to answer the question. Students need to know that only two of the options are correct (in a four or more option question) to determine that All of the above is the correct answer choice. Conversely, students only need to eliminate one answer choice as implausible in order to eliminate All of the above as an answer choice. Similarly, with None of the above, when used as the correct answer choice, information is gained about students ability to detect incorrect answers. However, the item does not reveal if students know the correct answer to the question. 4. Reviewing the MCQs: guidelines (Cohen & Wollack, 2000)

Cohen and Wollack recommend these for reviewing individual questions/items before students sit the test. 1. Consider the item as a whole and whether it measures knowledge or a skill component which is worthwhile and appropriate for the examinees who will be tested there is a markedly better way to test what this item tests it is of the appropriate level of difficulty for the examinees who will be tested. 2. Consider the stem and whether it presents a clearly defined problem or task to the examinee contains unnecessary information could be worded more simply, clearly or concisely. 3. Consider the alternatives and whether they are parallel in structure they fit logically and grammatically with the stem they could be worded more simply, clearly or concisely any are so inclusive that they logically eliminate another more restricted option from being a possible answer. 4. Consider the key and whether it is the best answer among the set of options for the item actually answers the question posed in the stem is too obvious relative to the other alternatives (i.e., should be shortened, lengthened, given greater numbers of details, made less concrete).

5. Consider the distractors and whether there is any way you could justify one or more as an acceptable correct answer they are plausible enough to be attractive to examinees who are misinformed or ill-prepared any one calls attention to the key (e.g., no distractor should merely state the reverse of the key or resemble the key very closely unless another pair of choices is similarly parallel or involves opposites). 5. Preparing Your Students for Taking Multiple-Choice Tests (Dewey, 1998 in Zimmaro, 2004, Parkes) 1. Remind students of the learning outcomes and give them the exam map/blueprint so that they are not forced to guess what will be on a test. Exam questions cover the all the important ideas in the course. 2. Let students write multiple-choice items for revision purposes (not for the actual test). This puts them "behind the scenes" and helps them identify material that might be on the test and how it might be asked. It also allows you to gauge their depth of understanding and what issues/ topics they are focusing on (which may not align with yours). (Parkes) 3. Give students practice in doing MCQs so they learn strategies for answering them and to manage the time spent on each question. The test items are then drawn from but do not copy the practice questions. The practice questions can be done in the first 5 minutes of a lecture and then corrected or offered online with immediate feedback re the correct answer (and why a chosen answer was incorrect). 4. Defeat the lazy students rules of thumb for taking MCQ tests. Inform students that they WONT work because YOU know about these rules of thumb and have dealt with them in designing questions. (Dewey 1998) Students rules of thumb (test-wise strategies) or how to avoid studying (and what to do about it)
a. Pick the longest answer make sure the longest answer is only correct a part of the time try to make options equal length b. When in doubt pick c make sure the correct answer choice letter varies c. Never pick an answer which uses the word always or never in it make sure this option is correct part of the time or avoid using always and never in the option choices d. If there are two answers which express opposites, pick one or the other and ignore other alternatives sometimes offer opposites when neither is correct or offer two pairs of opposites e. If in doubt, guess use five alternatives instead of three or four to reduce guessing f. Pick the scientific-sounding answer use scientific sounding jargon in wrong answers g. Dont pick an answer which is too simple or obvious sometimes make the simple, obvious answer the correct one h. Pick a word which you remember was related to the topic when creating the distractors use terminology from the same area of the text as the right answer, but in distractors use those words incorrectly so the wrong answers are definitely wrong

References
Bull, J. & McKenna, C. (2002). Computer Assisted Assessment Centre. Retrieved 8 February 2009 from http://www.caacentre.ac.uk/resources/objective_tests/index.shtml Brown, G. & Pendlebury, M. (1992). Assessing Active Learning. Sheffield: CVCP, USDU. Cohen, A., & Wollack, J. (2000). Handbook on test development: Helpful tips for creating reliable and valid classroom tests. Madison, WI: University of Wisconsin, Center for Placement Testing. Retrieved 13 October, 2003 from http://testing.wisc.edu/Handbook%20on%20Test%20Construction.pdf Dewey, R. A. (1998, January 20). Writing multiple choice items which require comprehension. Retrieved November 3, 2003 from http://www.psywww.com/selfquiz/aboutq.htm Kehoe, J. (1995) Writing multiple-choice test items. Practical Assessment, Research & Evaluation, 4(9). Retrieved July 29, 2008 from http://PAREonline.net/getvn.asp?v=4&n=9 Nitko, A. J. (2001). Educational assessment of students. (3rd Ed.). Columbus, OH: Merrill Prentice Hall. Owen, S. & Freeman, R. (1987). What's wrong with three option multiple items? In Educational & Psychological Measurement (47), 513-22. Parkes, J. Multiple Choice Test. Retrieved 20 September 2005 from http://www.flaguide.org/cat/mutiplechoicetest/multiple_choice_test7.php Zimmaro D. (2004). Writing Good Multiple-Choice Exams, Measurement and Evaluation Center: University of Texas, Austin