Effective grading (and teaching) through the use of rubrics & use of rubrics for assessment

Sarah Murnen Kenyon College

Personal Background

Although a psychologist, no particular background in this topic As Kenyons Assessment Coordinator from 2002-2006 I learned that:

Some faculty resent the assessment process and see it as extra burden Departments with faculty who feel burdened by assessment dont get much out of it Departments that find a way to integrate assessment with what they already do benefit from the process they discuss what students are learning and change their teaching methods, assignments, and sometimes their entire curricula to help students learn

Benefits of Ohio-5 Project

To help improve student learning (and faculty teaching) by encouraging the use of rubrics

(Many other benefits to the use of rubrics)

To specifically encourage a focus on creativity and critical thinking, important aspects of a liberal education To use rubrics for assessing critical thinking and creativity To show faculty how all of the work they put into grading can be used for assessment purposes

Background Information

Presentation by Dr. Douglas Eder on Primary Trait Analysis Presentation by Barbara Walvoord on the use of rubrics for effective grading Walvoord, B. E., P& Anderson, J. A. (1998). Effective Grading: A tool for learning and assessment. JosseyBass: San Francisco, CA. Stevens & Levi (2005). Introduction to rubrics: An assessment tool to save grading time, convey effective feedback, and promote student learning. Stylus Publishing: Sterling, Virginia.

Our primary concern is student learning

Good practice in undergraduate education (Walvoord, 1998, p. 15):

Encourages student-faculty contact Encourages cooperation among students Encourages active learning Gives prompt feedback Emphasizes the time that students devote to the task Communicates high expectations Respects diverse talents and ways of learning

Walvoord asks (1998, p. 15):

How many of these principles of good practice in some way involve the grading system in your class, the tests and assignments on which that system is based, and your ways of communicating with students about their work and their grades?

Walvoords Argument for Establishing Clear Criteria and Standards for Grading

Saves time in grading process Allows you to make the process consistent and fair Helps you explain to students what you expect Shows you what to teach Identifies essential relationship between discipline information and processes Help students evaluate their own and each others work Saves you from having to explain your criteria to students after they have handed in their work, as a way of justifying the grades they are contesting Helps student peers give each other constructive feedback on plans and drafts Helps team teachers or teaching assistants grade student papers consistently Helps teachers of sequenced courses communicate with each other about standards and criteria Form the basis for departmental or institutional assessment

How to Establish Clear Criteria: Primary Trait Analysis

Developed to score essays on the National Assessment of Educational Progress by LloydJones in1977 Creates a scoring rubric that can be used to assess any student performance Is assignment specific Can be used for grading For this project, we will develop rubrics using PTA to measure the critical thinking and creativity we are encouraging in our students

Why take the time to do PTA for grading?

Makes grading more consistent and fair Saves time in the grading process once the rubric is developed Can diagnose students strengths and weaknesses very specifically in order to teach more effectively Can track changes in student performance

Process of PTA (Eder)

Identify the primary traits essential or central components of the discipline to be learned by the student Build a scale for scoring the students performance on the trait Evaluate the students performance against those criteria

Key Stages in Constructing a Rubric (Stevens & Levi)

1.

Reflecting

What do we want from our students? Why did we create the assignment? What happened the last time we used the assignment? What are our expectations for the assignment?

2.

3.

4.

Listing Focus on particular details of the assignment and what specific learning objectives we hope to see completed (Sometimes helps to imagine the best and the worst performance on the assignment) Grouping and Labeling the goals together Organize the results of our reflections in Stages 1 and 2, grouping similar expectations together in what will probably become the dimensions of the rubric Application Transfer groupings to a rubric grid

Example: Research Article Critique

1.

Reflecting

Want students to learn to be critical evaluators of psychological research The research article critique should start them thinking critically, will follow up with class discussion on each critique, and use multiple critiques throughout the semester to help develop their skills I expect students will move from description to analysis

2.

Listing

I have seen students move through Blooms taxonomy in a semester using this assignment: Most can show knowledge and comprehension at the beginning of the course, move to application, analysis, synthesis, and evaluation (hopefully). I want to try to capture this process. see the questions that follow See the rubric that follows

3. 4.

Grouping and Labeling the goals together

Application Transfer groupings to a rubric grid

Research Article Critique Assignment

For each article analysis you are to answer the following questions:

What is the primary question posed by the study? Is there a hypothesis stated? If so, what is it? What is the theoretical explanation for the proposed hypothesis? Briefly describe the way the independent and dependent variable(s) were manipulated or measured How do the results of the study affect the originally posed hypothesis (or purpose of study)? Two strengths of the study? Two weaknesses of the study? What is a logical extension of the study? Briefly describe a study you could conduct to extend the research

Sample Rubric Designed for Assessment

ACC = Accomplished, AVG = average, DEV = developing, BEG = beginning
Understands primary purpose of research Clearly labels the independent and dependent variable without extraneous material provided and shows understanding of theoretical basis of study ACC AVG DEV BEG

Describes measurement issues clearly Hones in on how the variables are manipulated/measured without providing extraneous material Understands significance and implication of the results E.g., does research represent a methodological improvement, novel hypothesis, important test of a theory, etc.

ACC

AVG

DEV

BEG

ACC

AVG

DEV

BEG

Chooses relevant and important strengths of study Discusses strengths that are very relevant to judging the internal validity of the study (or possibly external validity) Chooses relevant and important weaknesses of study Discusses weaknesses that are very relevant to judging the internal validity of the study (or possibly external validity)

ACC

AVG

DEV

BEG

ACC

AVG

DEV

BEG

Proposed study an important extension of the research E.g., methodological improvement, advancement of theory, tests an alternative explanation, etc. Creativity e.g., Unusual association of ideas in responses, proposed study is novel and represents a significant advancement

ACC

AVG

DEV

BEG

ACC

AVG

DEV

BEG

Rubric Used for Grading weight each component

5% Understands primary purpose of research Clearly labels the independent and dependent variable without extraneous material provided and shows understanding of theoretical basis of study ACC AVG DEV BEG

5% Describes measurement issues clearly Hones in on how the variables are manipulated/measured without providing extraneous material 10% Understands significance and implication of the results E.g., does research represent a methodological improvement, novel hypothesis, important test of a theory, etc.

ACC

AVG

DEV

BEG

ACC

AVG

DEV

BEG

25% Chooses relevant and important strengths of study Discusses strengths that are very relevant to judging the internal validity of the study (or possibly external validity) 25% Chooses relevant and important weaknesses of study Discusses weaknesses that are very relevant to judging the internal validity of the study (or possibly external validity)

ACC

AVG

DEV

BEG

ACC

AVG

DEV

BEG

25% Proposed study an important extension of the research E.g., methodological improvement, advancement of theory, tests an alternative explanation, etc. 5% Creativity e.g., Unusual association of ideas in responses, proposed study is novel and represents a significant advancement

ACC

AVG

DEV

BEG

ACC

AVG

DEV

BEG

Describing the level of performance on the rubric

Anywhere from 2 to 5+ levels of performance on each trait Terms used to describe level of performance might be positive and active to encourage motivation in students (suggestions from Stevens & Levi):

Mastery, partial mastery, progressing, emerging high level, middle level, beginning level Sophisticated, competent, partly competent, not yet competent Exemplary, proficient, marginal, unacceptable Advanced, intermediate high, intermediate, novice Distinguished, proficient, intermediate, novice Accomplished, average, developing, beginning

Using Rubrics for Assessment

We all need to attend to the issue of assessment, and Walvoord argues that we can make use of what we already do in the grading process

American Association for Higher Educations Principles of Good Practice for Assessing Student Learning

Answer questions that people care about Lead directly to improvement in teaching and learning Be embedded in the context of learning Take place repeatedly over time

The rubrics we use for grading can be used for assessment

If we as faculty do not make our learning goals, tests, criteria, and standards explicit and understandable to legislatures, boards, accrediting agencies, and other audiences in ways that meet their needs and concerns, we face the very real possibility that some of the control we currently exercise in the classrooms will be taken away from us. We must deal with assessment, but we need not construct a parallel assessment structure that ignores the assessment we already conduct. (Walvoord, 1998, p. 5).

How to turn this into assessment

Paper 1 Paper 2 1 1 1 1 Paper 3 2 2 3 3 Total 7 7 7 6

Prof A Creative1 Creative2 Critical1 Critical2

Prof B Creative1 Creative2 Critical1 Critical2 Prof C Creative1 Creative2 Critical1 Critical2 Prof D Creative1 Creative2 Critical1 Critical2 Total

4 4 3 2

3 4 3 2

2 1 1 1

2 3 3 3

7 8 7 6

4 3 2 2

1 2 2 1

2 3 3 3

7 8 7 6

4 4 3 3 50

2 2 2 2 23

3 3 3 4 45

9 9 8 9

How have faculty been influenced by this process?

I was able to clarify my expectations for the course by articulating the purpose behind each graded exercise. This helped dramatically in easing anxieties about performance and final grades. Students seemed to appreciate understanding the logic behind the assignments and seeing that each focused on building and assessing a particular skill, rather than merely providing me with another grade. Best of all, Im looking forward to sharing the process of developing PTA-based rubrics with my students. Specifically, students will work in groups to identify their own PTA-based critical thinking rubrics before they embark on writing a particular essay. I can hardly think of a better way to develop the students meta-cognition., i.e., how they think about their own thinking! Yes, weve been disappointed with some aspects of student performance (for example synthetic ability) and using the rubric has helped us communicate better with students, and were thinking about ways to improve courses to help teach the skills needed to succeed on this senior exercise objective. I have learned a great deal, and I think that my teaching has been positively influenced by this important work. I can also say that my research and scholarship have been impacted by these foci, as well.

Go to Website for Rubric Samples:

http://www.wooster.edu/teagle/ default.html