Grading Rubrics

Caroline R.H. Wiley and Darrell Sabers

Department of Educational Psychology

Assessment Coordinating Council - February 11, 2009 Tucson, AZ

1
 Overview
Why use rubrics?
Psychometric considerations
Types of scoring systems
Rubric construction and examples
The Rubric for Rubrics (ETS, 2006)

Assessment Coordinating Council - February 11, 2009 Tucson, AZ

2
2
 Why Use Rubrics?
Mechanism for judging the quality of student
work (Arter & Chappuis, 2006)
High-stakes nature of grades
Clarity
Advantageous for diagnosing learning or
summarizing learning (Assessment for learning vs.
Assessment of learning)
Reduce subjectivity as much as possible

Assessment Coordinating Council - February 11, 2009 Tucson, AZ

3
3
 How Do I know If I Need a Rubric?

If you administer...
Essay tests
Papers (research, theoretical, etc.)
Constructed response items (paragraphs, a
few sentences, 1-2 words)
Performance assessments (speeches, debates,
presentations, skits, skills (car repair, medical
procedures, scientific procedures, etc.)

Assessment Coordinating Council - February 11, 2009 Tucson, AZ

4
4
 Psychometric Considerations
Content validity
Learning objectives
Bluffing
Absence of bias
Halo effect
Consistency
With others
With yourself

Assessment Coordinating Council - February 11, 2009 Tucson, AZ

5
5
 Types of Scoring Systems
Checklists
Rating scales
Analytic rubrics
Holistic rubrics

Assessment Coordinating Council - February 11, 2009 Tucson, AZ

6
6
 Checklists

Gronlund & Waugh (2009)

Assessment Coordinating Council - February 11, 2009 Tucson, AZ

7
7
 Checklists

McMillan (2007)

Assessment Coordinating Council - February 11, 2009 Tucson, AZ

8
8
 Rating Scales

Reynolds, Livingtson, & Wilson (2006)

Assessment Coordinating Council - February 11, 2009 Tucson, AZ
9
9
 Analytic Rubric

McMillan (2007)
Assessment Coordinating Council - February 11, 2009 Tucson, AZ
10
10
 Analytic Rubric

McMillan (2007)

Assessment Coordinating Council - February 11, 2009 Tucson, AZ

11
11
 Holistic Rubric

McMillan (2007)

Assessment Coordinating Council - February 11, 2009 Tucson, AZ

12
12
 Generic Rubric Construction
Define learning objectives
Determine purpose
Procedures vs. final product
Determine assessment purpose
Assessment for learning vs. Assessment of learning
Determine best type to use
Depends on all of the above
If using analytic, determine components and weights
Determine number of levels to use
3 to 7
Determine criteria for the levels
Representative of desired learning objectives
Parallel
Eliminate double-penalties

Assessment Coordinating Council - February 11, 2009 Tucson, AZ

13
13
 Example
Task: Using a thermometer
Excelling
(20-25 points)
Preparation Grasps non-bulb end
10% and wipes thermometer
downward.
Administration Leaves thermometer in
10% mouth for 3 minutes,
removes thermometer
grasping non-bulb end,
and reads temperature
to the nearest tenth of
a degree.
Clean-up Correctly records
5% temperature on patients
record and cleans
thermometer with an
alcohol wipe.
Creativity Is very creative in the
40% overall process of
measuring temperature.
Assessment Coordinating Council - February 11, 2009
Proficient Below Proficient
(15-19 points) (1-14 points)
Either does not grasp Is not creative
non-bulb end OR does in grasping non-bulb
not wipe thermometer end.
downward.
Only does 2 of the 3 Does not leave
actions described in thermometer in mouth
the Excelling for 3 minutes.
category.
Records temperature on Doesnt record temperature
patients record, but on patients record AND
does so sloppily OR does not clean thermometer
cleans thermometer with at all.
non-alcohol wipes.
A little creative. Not creative at all.
Student has own style No originality present.
of taking temperature.
Tucson, AZ
14
14
Stevens & Levi (2005)

Assessment Coordinating Council - February 11, 2009 Tucson, AZ

15
15
 Stevens & Levi (2005)
Assessment Coordinating Council - February 11, 2009 Tucson, AZ
16
16
 Watch Out For Weights
Using different number of
points in each component
leads to misleading scores
Checklist: 5/17 = 29%
Persuasiveness: 3/17 = 18%
Delivery: 3/17 = 18%
Sensitivity: 3/17 = 18%
Holistic rating: 3/17 = 18%

Double-penalty
Holistic rating?

McMillan (2007)
Assessment Coordinating Council - February 11, 2009 Tucson, AZ
17
17
 The Rubric For Rubrics
ETS (2006)
Criterion 1: Coverage/Organization
1A: Covers the right content
1B: Criteria are well organized
1C: Number of levels fits targets and uses
Criterion 2: Clarity
2A: Levels defined well
2B: Levels parallel

Assessment Coordinating Council - February 11, 2009 Tucson, AZ

18
18
 Using the Rubric for Rubrics:
GRE Analyze an Argument
GENERAL DESCRIPTIONS:
Score 6: A cogent, well-articulated critique of the
argument and conveys meaning skillfully.
Score 5: A generally thoughtful, well-developed critique
of the argument and conveys meaning clearly.
Score 4: A competent critique of the argument and
conveys meaning adequately.
Score 3: Some competence in its analysis of the issue
and in conveying meaning but is obviously flawed.
Score 2: Serious weaknesses in analytical writing.
Score 1: Fundamental deficiencies in analytical writing.
Assessment Coordinating Council - February 11, 2009 Tucson, AZ
19
19
 Summary
Choose the appropriate scoring method based on the
purpose
Look for rubrics made elsewhere and adapt as necessary
Your criteria should reflect your learning objectives
And should be weighted appropriately
Criteria should be clearly defined and parallel across
levels
Check for double-penalty criteria
Number of levels should be reasonably distinguishable
Check that scores reflect knowledge appropriately
Develop the rubric as you write the task
Physically organize the rubric to be efficient and clear
Take it for a dry run

Assessment Coordinating Council - February 11, 2009 Tucson, AZ

20
20
 References
Arter, J., & Chappuis, J. (2006). Creating & recognizing quality rubrics.
Educational Testing Service. Upper Saddle River, NJ: Pearson.
Gronlund, N. E., & Waugh, C. K. (2009). Assessment of student achievement.
(9th ed.). Upper Saddle River, NJ: Pearson.
McMillan, J. H. (2007). Classroom assessment: Principles and practice for effective
standards-based instruction (4th ed.). Boston: Allyn and Bacon
Reynolds, C. R., Livingston, R., & Wilson,V. (2006). Measurement and
assessment in education. Upper Saddle River, NJ: Prentice Hall.
Stevens, D. D., & Levi, J. A. (2005). Introduction to rubrics. Sterling,VA: Stylus
Publishing, LLC.

Assessment Coordinating Council - February 11, 2009 Tucson, AZ

21
21