Steps in Making Item Analysis

________________________________________________________________

1. Score each answer sheet, write score total on the corner Obviously have to do this anyway 2. Sort the pile into rank order from top to bottom score ( 1 minute, 30 seconds tops) 3. If normal class of 30 students, divide class in half Same number in top and bottom group; Toss middle paper if odd number (put aside) 4. Take top pile, count number of students who responded to each alternative Fast way is simply to sort piles into A, B, C, D // or true/false or type of error you get for short answer, fill-in-the-blanks Or set up on spread sheet if youre familiar with computers

ITEM ANALYSIS FORM TEACHER CONSTRUCTED TESTS CLASS SIZE = 30 ITEM UPPER LOWER DIFFERENCE D TOTAL DIFFICULTY 1.A 0 *B 4 C 1 D 1 O *= Keyed Answer Repeat for lower group

ITEM ANALYSIS FORM TEACHER CONSTRUCTED TESTS CLASS SIZE = 30 ITEM UPPER LOWER DIFFERENCE D TOTAL DIFFICULTY 1.A 0 *B 4 2 C 1 D 1 O *= Keyed Answer This is the time consuming part but not bad, can do it while watching TV, because youre just sorting piles

________________________________________________________________ THREE POSSIBLE SHORT CUTS HERE (STEP 4) (A) If you have a large sample of around 100 or more, you can cut down the sample you work with Take top 27% ( 27 out of 100 ); bottom 27% ( so only dealing with 54, not all 100) Put middle 46 aside for the moment Larger the sample, more accurate, but have to trade off against labour; using top 1/3 or so is probably good enough by the time you get to 100; -- 27% magic figure statisticians tell us to use Id use halves at 30, but you could just use a sample of top 10 and bottom 10 if youre pressed for time But it means a single student changes stats by 10% Trading off speed for accuracy But Id rather have you doing ten and ten than nothing (B) Second short cut, if you have access to photocopier (budgets) Photocopy answer sheets; cut off identifying info (cant use if handwriting is distinctive) Colour code high and low groups dab of marker pen color Distribute randomly to students in your class so they dont know whose answer sheet they have Get them to raise their hands For #6, how many have A on blue sheet? How many have B; how many C For #6, how many have A on red sheet Some reservations because they can screw you up if they dont take it seriously Another version of this would be to hire kid who cuts your lawn to do the counting, provided youve removed all identifying information I actually did this for a bunch of teachers at one high school in Edmonton when I was in university for pocket money (C) Third shortcut, If you cant use separate answer sheet, sometimes faster to type than to sort SAMPLE OF TYPING FORMAT FOR ITEM ANALYSIS ITEM # 1 2 3 4 5 6 7 8 9 10 KEY T F T F T A D C A B

STUDENT Kay Jane John T T T F F A D D A C T T T F T A D C A D F F T F T A D C A B

Type name; then T or F, or A, B, C, D == all left hand on typewriter, leaving right hand free to turn pages ( from Sax)

If you have a computer program some kicking around will give you all stats you need, plus bunches more you dont automatically after this stage

OVERHEAD: SAMPLE ITEM ANALYSIS FOR CLASS OF 30 ( PAGE #1) (IN TEXT) 5. Subtract the number of students in lower group who got question right from number of high group students who got it right Quite possible to get a negative number ITEM ANALYSIS FORM TEACHER CONSTRUCTED TESTS CLASS SIZE = 30 ITEM UPPER LOWER DIFFERENCE D TOTAL DIFFICULTY 1.A 0 *B 4 2 2 C 1 D 1 O *= Keyed Answer 6. Divide the difference by number of students in upper or lower group In this case, divide by 15 This gives you the discrimination index (D) ITEM ANALYSIS FORM TEACHER CONSTRUCTED TESTS CLASS SIZE = 30 ITEM UPPER LOWER DIFFERENCE D TOTAL DIFFICULTY 1.A 0 *B 4 C 1 D 1 O *= Keyed Answer 7. Total number who got it right ITEM ANALYSIS FORM TEACHER CONSTRUCTED TESTS CLASS SIZE = 30 ITEM UPPER LOWER DIFFERENCE D TOTAL DIFFICULTY 1.A 0 *B 4 2 2 0.33 6 C 1 D 1 O *= Keyed Answer 8. If you have a large class and were only using the 1/3 sample for top and bottom groups, then you have to NOW count number of middle group who got each question right ( not each alternative this time, just right answers) 9. Sample Form Class Size = 100. If class of 30, upper and lower half, no other column here

10. Divide total by total number of students Difficulty = ( proportion who got it right (p) )

ITEM ANALYSIS FORM TEACHER CONSTRUCTED TESTS CLASS SIZE = 30 ITEM UPPER LOWER DIFFERENCE D TOTAL DIFFICULTY 1.A 0 *B 4 2 2 0.333 6 .42 C 1 D 1 O *= Keyed Answer 11. You will NOTE the complete lack of complicated statistics counting, adding, dividing no tricky formulas required for this Not going to worry about corrected point biserials etc. One of the advantages of using fixed number of alternatives

Sample Multiple Choice Item Analysis in Assessment of Student Learning II

EXAM 3 NUMBER OF STUDENTS IN ITEM ANALYSIS = 30 NO. OF ITEMS: 24

The RESPONSE TABLE below contains the percentages of students who selected each option to each item. The percentage selecting the correct option is repeated in the KEY - % column. The ITEM EFFECT column contains biserial coefficients. Negative values indicate ineffective items; some students with high tests cores missed these items. A + sign identifies an incorrect option selected by students with a higher average test score than those who selected the correct option.

ITEM NO. OMIT % 1 2 3 4 5 6 7 8 0 0 0 0 0 0 0 0

RESPONSE TABLE FORM A A % 100 0 13 13 3 13 53 10 B % 0 0 73 17+ 97 0 47 70 C % 0 100 0 60 0 60 0 10 D % 0 0 13 10 0 27 0 10 E % 0 0 0 0 0 0 0 0 A C B C B D A B

ITEM KEY - % EFFECT

100 100 73 60 97 27 53 70

0.00 0.00 0.14 0.03 0.16 -0.07 -0.16 0.60

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 0

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 7

63 0 0 67 0 10 7 10 77 10 7 3 3 97 0 0

17 80 23 3 3 63 80 17 7 3 10 0 0 0 0 7

17 20 0 23 87 0 13 0 0 3 63 10 93 3 97 87 0

3 0 77 7 10+ 27 0 73 17 83 20 87 3 0 3

0 0 0D 0 0 0 0 0 0 0 0 0 0 0 0 D 87

A B 77 A C B B D A D C D C A C

63 80 0.29 67 87 63 80 73 77 83 63 87 93 97 97 0.22

0.23 0.69

0.70 -0.02 0.51 0.54 0.27 0.60 0.14 0.48 0.36 0.36 0.46 0.46

Distribution of items by % CORRECT: % Correct 0-20 (very difficult) 21-60 (difficult) No. of items 0 3

Distribution of items by ITEM EFFECT: Biserial Coefficient Negative (ineffective) .00-.20 (low effectiveness) .21-.40 (medium effectiveness) .41-1.00 (high effectiveness) No. of items 3 6 6 9

61-90 (moderately difficult) 15 91-100 (easy) 6

The RELIABILITY of scores for this form is 0.659

The reliability coefficient above is a Kuder-Richardson Formula 20 value. K-R 20 values range from .00 to 1.00. Higher values indicate greater measurement precision. Scores on tests with K-R 20 values below .50 are poor indicators of which students are strong and which are weak on the information tested.