Module

6 Assessment of Student Learning

Module 6
ITEM ANALYIS

Objectives

Discuss the importance of item analysis in test development

Compute difficulty index and discrimination index of test items
Interpret item analysis data

Introduction

The preceding two lessons dealt with the different methods of assessment that

can be used to assess varying learning targets. After writing the items, a test

cannot be considered good unless it is tried out because the ultimate judge of the

test is the user. More often than not, teachers prepare, administer, and score a test,

return the papers to their students, possibly discuss the test, and then either file or

discard the test. Ebel (1965) believes that one of the common mistakes of

teachers is that they do not check on the effectiveness of the their test because of

the following reasons: a) teachers feel that test analysis is too time consuming, b)

they are not aware of the methods of analyzing tests, and 3) they do not always

understand the importance of accurate evaluate evaluation.

Item analysis

Page 1 of 13
Module
6 Assessment of Student Learning

Remember that tests are used for instructional purposes and other important

decisions, it is therefore important to conduct item analysis. Item analysis is the

process of evaluating the quality of the items and of the tests as a whole. Item

analysis investigates the performance of items considered individually in relation

to the remaining items on the test (Thompson & Leviton, 1985). It is the process

of examining the pupils responses to each test item.

The item analysis procedure provides the following information:

The difficulty of the item

The discriminating power of the item
The effectiveness of the distracters

The key aim of the item analysis is to improve the whole test and

eventually increase the reliability and reliability of the test. The tools to test the

validity and reliability of the test include item difficulty, item discrimination, and

effectiveness of distracters. There are many methods that can be used for item

analysis. One method is the U L Index Method (Stocklein, 1957). The steps

are:

1. Score the papers and rank them from highest to lowest according to
total score.
2. Separate the top 27% and the bottom 27% of the papers.
3. Tally the responses made to each test item by each individual in the
upper 27% group.
4. Tally the responses made to each test item by each individual in the
lower 27% group.

Page 2 of 13
Module
6 Assessment of Student Learning

5. Compute the percentage of the upper group that got the item right and
call it U.
6. Compute the percentage of the lower group that got the item right and
call it L.
7. Average U and L percentage and the result is the difficulty index of
the item.
8. Subtract the L from the U percentage and the result is the
discrimination index of the item

Item difficulty/difficulty index

Difficulty index, denoted by p, is simply the percentage of the students taking the

test who answered the item correctly. It can be interpreted as how easy or how difficult

an item is. The larger the percentage getting an item right, the easier the item. The

higher the difficulty index, the easier the item is understood to be. For example, an item

answered correctly by 85% of the examinees will have an item difficulty of .85, whereas

an item answered by 50% of the examinees would have a lower item difficulty of .5. it is

usually best when difficulty index is around .5, for it provides maximum differentiation.

The best difficulty index, however, is halfway between the lowest and highest expected

score. When all of the items are extremely difficult, the great majority of the test scores

will be very low. When all items are extremely easy, most test scores will be extremely

high. We do not want items that are too easy or too difficult.

Item discrimination/discrimination index

Page 3 of 13
Module
6 Assessment of Student Learning

A good item discriminates between those who do well on the test and those who

do poorly (Susan Matlock-hetzel, 1997). The discriminative power of a test item,

denoted by D refers to the degree of the ability of an item to distinguish those who know

from those who do not know. It can be measured by comparing the number of people

with high test scores who answered that item correctly with the number of people with

low scores who answered the same item correctly. The top and bottom 27% are used for

analysis because 27% has shown that this value will maximize differences in the normal

distribution while providing enough cases for analysis (Wiersma and Jurs (1990). Other

methods use 30% while others use 50% (no middle group).

The higher discrimination index, the better the item because such a value

indicates that the item discriminates in favor of the upper group, which should get more

items correct. If more students in the lower group get an item correct than in the upper

group, the item will have a negative D value and is probably flawed. Similarly, if there

are equal number of students from the upper group and lower group, the item cannot

discriminate because D value is 0.

Computing for the difficulty index and discrimination index of an item is a lot

easier than interpreting it. Remember that the purpose of the analysis is to determine

which are good items; that is there are items to be retained or accepted, rejected or

discarded, and revised. The table below will serve as guide in interpreting the results of

the item analysis.

Table for interpreting difficulty index (p)

Page 4 of 13
Module
6 Assessment of Student Learning

Range of Difficulty Index (p) Interpretation

0 - .20 Very difficult
0.21 0.40 Difficult
0.41 0.60 Moderately difficult
0.61 0.80 Easy
0.81 1.00 Very easy

Table for interpreting discrimination index (D)

Range of Discrimination Index (D) Interpretation

-1.0 - -.06 Questionable
-0.59 - -0.20 Not discriminating
-0.21 - 0.20 Moderately discriminating
0.21 - 0.60 Discriminating
0.61 - 1.00 Very discriminating

In interpreting the results, we have to consider not only whether how easy or how

difficult the item is, but also its ability to discriminate students who know and those who

do not know the answer. In other words, both the p values and D values are taken into

consideration. The decision rule is to retain or accept the items that are not so easy or too

difficult, and at the same time can discriminate bright from poor students.

After interpreting the difficulty and discrimination indices, the table below will help us

what to do with the test item

Difficulty Index Discrimination Index Suggested Action

Not discriminating Discard/Reject
Moderately discriminating May need revision
Difficult
Discriminating/Very Accept/Retain
discriminating
Not discriminating Needs revision
Moderately discriminating May need revision

Page 5 of 13
Module
6 Assessment of Student Learning

Moderately difficult Discriminating/Very Accept/Retain

discriminating
Not discriminating Discard/Reject
Moderately discriminating Needs Revision
Easy
Discriminating/Very Needs Revision
discriminating

Note that whether an item is easy, moderately difficult or difficult, there are

always three categories of discrimination indices that can be obtained, that also leads to

three different actions. For example, an easy item could either be not discriminating,

moderately discriminating or discriminating, and then it could be rejected, revised or

retained.

However, care and caution musts be followed in using the table in interpreting the

results of an item analysis. Judgment of the test constructor sis very important. For

example, what will be done with an item that is easy and not discriminating. Using the

table, we should reject the item. But there will be an instance when that kind of item can

be revised. When? When that particular item is the only item left to test a very important

concept. So, we have no choice but to revise or improve it. On the other hand, what will

be done with an item that is moderately difficult and discriminating? Normally that item

should be retained because it has good indices. But there will be instance when that kind

of item may be discarded or rejected. That will happen if there are already enough items

to test the particular concept or skill that it assesses.

The table below shows a sample result of an item analysis, illustrating the steps of

the U-L Index method.. Study the results and focus on the interpretation of the test item.

Page 6 of 13
Module
6 Assessment of Student Learning

Table showing a summary of the Item Analysis

Ite Uppe Lowe U L p Interpretation D Interpretation Action

m r r
No. 27% 27%
1 14 12 .88 .75 .82 Very easy .13 Moderately Revise
discriminating
2 10 6 .63 .38 .51 Moderately .25 Discriminating Retain
difficult
3 11 7 .69 .44 .57 Moderately .25 Discriminating Retain
difficult
4 9 2 .56 .13 .35 Difficult .43 Discriminating Retain
5 12 6 .75 .38 .57 Moderately .37 Discriminating Retain
difficult
6 6 14 .38 .88 .63 Easy -.50 Questionable Discard
7 13 4 .81 .25 .53 Moderately .56 Discriminating Retain
difficult
8 3 10 .19 .63 .41 Moderately -.44 Questionable Discard
difficult
9 13 12 .81 .75 .78 Easy .06 Moderately Revise
discriminating
10 8 6 .50 .38 .44 Moderately .12 Moderately Revise
difficult discriminating
Number of students tested - 60

Step 1. The papers have been scored and arranged from highest to lowest

Step 2. There were 60 students who took the test, so 27% of 60 is 16. Therefore, there
are 16 students from the upper group and another 16 students for the lower
group. The middle group (28 students) will not be included in the analysis.

Step 3. The number of students from the upper group was tallied. For example, out of 16
students, 14 students got item 1 correctly, 10 students for item 2, and so forth.

Step 4. The number of students from the lower group was tallied. For example, out of 16
students, 12 students got item 1 correctly, 6 students for item 2, and so forth.

Page 7 of 13
Module
6 Assessment of Student Learning

Step 5. The percentage of the students from the upper group is obtained by dividing the
number of students who got the item right and the total number of students; that is
14/16 = .88, or 88%., 10/16 = .63 or 63 % for item 2, and so forth. The results are
placed under Column U (upper group)

Step 6. The same procedure like in step 5 was performed. For example, 12/16 = .75 0r
755 for item 1, and so forth. Results are placed under Column L (lower group)

Step 7. The difficulty index (p) was computed by getting the average of U and L
percentages. For example in item 1 U = .88, L = .75. Therefore, p = (U+L)/2 =
(.88+.75)/2 = .82. This means that 82 % of the students got item 1 correctly.

Step 8. The discrimination index (D) is obtained by subtracting L from U, that is D = U

L, or D = .88 - .75 = .13

So for item number 1, the difficulty index is .82 and the discrimination index is .

13. Using the tables presented earlier, a difficulty index of .82 is said to be very easy, and

a discrimination index of .13 is moderately discriminating. The item therefore needs

revision. The item was too easy because there were many students, even from the lower

group who got the item correctly.

Item Distractor Analysis

An item distractor analysis is performed to review items judged to be problematic

during difficulty and discrimination analyses, to evaluate the plausibility of distracters or

incorrect options, and to identify areas in which instruction needs to be revised. For

example, item number 1 in the previous item analysis needs revision. Why? Because the

Page 8 of 13
Module
6 Assessment of Student Learning

item is very easy, though it can moderately discriminate. In selection-type of test, like the

multiple choice, an item may be very easy or very difficult because of the options or

alternatives.

To perform distractor analysis, there is a need to compare the selected responses

of students in the upper and lower group. Creating a distractor analysis table will help

summarize this information. Look at the example below:

Item Number Options

A B C D
Upper 0 14** 2 0
Lower 0 12** 4 0
1
Upper 13** 1 1 1
Lower 12** 2 2 0
9
Upper 8** 8 0 0

10
Lower 6** 10 0 0
N = 16 ** correct answer/key

For example, item number has p value of .82 and D value is .13. The item is easy

but discriminating, which means that the item needs revision. If we will revise the item,

we might look as well at the distracters. Not one of the respondents choose A and D.

This may have two explanations. The distracters are impossible or illogical to them. You

may have included distracters that are obviously wrong. However, if you really think that

A and D represent logical alternatives to the correct answer, then it is more probable that

Page 9 of 13
Module
6 Assessment of Student Learning

B reflects common misconception held by large portion of the class. This requires

revising your initial instruction and includes information on how to avoid the problem.

The same thing applies to item number 9. For item number 10, the item needs revision as

well. Though the item can moderately because there were more students from the upper

group than the lower group who got the correct answer, the same number of students

from the upper group chose a distracter as the correct answer (option B). Similarly,

looking at the lower group, it seems that option B is the correct answer for item 10.

In summary, item analysis provides relevant information about the individual

items of the test, and the test as a whole. Always remember that you can maximize the

ability of the test to help you assess students learning by making sure that you construct

high quality test items.

Activity 1
Page 10 of 13
Module
6 Assessment of Student Learning

A. Complete the table below.

Ite Uppe Lowe U L p Interpretation D Interpretation Action

m r r
No. 27% 27%
1 20 18
2 19 12
3 17 11
4 10 20
5 21 11
6 9 2
7 24 14
8 18 13
9 9 19
10 22 15
11 26 24
12 25 13
13 6 3
14 23 12
15 11 19
Number of pupils tested = 10

Activity 2
A. Answer the following questions.
1. When do we obtain a difficulty index of 0.0?; 1.0?
2. When do we get a negative discrimination index?; positive discrimination index?;

zero discrimination index?

3. If the item has a perfect discrimination index, what is the difficulty index of that

item?
4. In as far as item analysis is involved, what is a good item?

B. Analysis of Distractors

Page 11 of 13
Module
6 Assessment of Student Learning

Below shows the students answers in 5 items. Study the information and

in each item, indicate (under remarks row) whether each option is a good, poor or

fair option. Figure with asterisk indicate the correct response.

Item 1 A B C D
Upper 10* 4 1 1
Lower 7 0 4 5
Remarks

Item 2 A B C D
Upper 5* 5 5 1
Lower 16 0 0 0
Remarks

Item 3 A B C D
Upper 2 3 4 7*
Lower 8 2 5 1
Remarks

Item 4 A B C D
Upper 2 8* 1 5
Lower 5 5 1 5
Remarks

Item 5 A B C D
Upper 16* 0 0 0
Lower 16 0 0 0
Remarks

Page 12 of 13
Module
6 Assessment of Student Learning

Activity 3

Share insights gained from the lesson (2-3 short paragraphs)

Page 13 of 13