The Development of the Peer Assistance and Review Form (PARF)

Carlo Magno De La Salle University, Manila

Assessing Teacher Performance

Students assess teachers *A peer (fellow teacher/supervisor) assessing the teacher External personnel assessing the teacher

Peer Review of Teaching

When a teacher is observed by another teacher for a specific purpose, it is called in literature as peer assistance and review (Goldstein, 2004) peer review (Kerchner & Koppich, 1993), teacher peer coaching (Bruce & Ross, 2008)

What peer review of teaching?

involving teachers in the summative [also formative] evaluation of other teachers (Goldstein, 2004, p. 397) evaluating ones peers which allow the assessment of ones teaching by another person who has similar experience and goals a structured approach for building a community in which pairs of teachers of similar experience and competence observe each other teach, establish improvement goals, develop strategies to implement goals, observe one another during the revised teaching, and provide specific feedback (Bruce & Ross, 2008)

Purpose
Hiring of teachers Clinical supervision Modeling for new teachers Promotion Rehiring

Common uses of Teacher Performance Results

Developmental Approach
Teachers performance from peer reviews should be conceptualized with the aim of helping teachers attain success in their teaching rather than point out mistakes of teachers (Oakland & Hambleton, 2006; Stiggins, 2008). It is described as a constructive process where the peer aims to provide assistance to a less experienced teacher in improving their instruction and handling of students.

Assistance component of the PARF

Clinical supervision process Pop-in visit formal visit clinical supervision Several visits to help teachers improve their teaching Academic coaches Phases of observation: pre-observation, post observation, and post-observation

Conceptual Framework
Learner-centered principles
perspectives that allow the teachers ability to facilitate the learners in their learning, the learning in the programs, and other processes that involves the learner

Danielsons Components of Professional Practice

identified aspects of the teachers responsibilities that have been documented through empirical studies and theoretical research promoting improved student learning.

Conceptual Framework
Danielsons Components of Professional Practice
Planning and preparation Classroom environment Instruction Professional responsibility

The LC and components of professional practice fits well together in a model (Magno & Sembrano, 2010).

Method
Search for Content Domain
FGD to assess the suitability of the items for teaches
applicability for the faculty relevance in their teaching phrased accordingly produced consistent meanings for different users

Method
Rubric Construction (analytical rubric)
Domain 1: Planning and Preparation 1a. Demonstrating knowledge of content and pedagogy 1b. Demonstrating knowledge of students 1c. Selecting instructional goals 1d. Demonstrating knowledge of resources 1e. Designing coherent instruction 1f. Assessing student learning
Domain 3: Instruction 3a. Communicating clearly and accurately 3b. Using questioning and discussion techniques 3c. Engaging students in learning 3d. Providing feedback to students 3e. Demonstrating flexibility and responsiveness

Domain 2: The Classroom Environment 2a. Creating an environment of respect and rapport 2b. Establishing a culture for learning 2c. Managing classroom procedures 2d. Managing student behavior 2e. Organizing physical space

Domain 4: Professional Responsibilities 4a. Reflecting on teaching 4b. Maintaining accurate records 4c. Communicating with families 4d. Contributing to the school and district 4e. Growing and developing professionally 4f. Showing professionalism

Layers of constructs
Quality Domain component Subcomponent Chance Quantity Standards

4 point scale: exemplary, successful, limited, poor Exhibits were provided for each subcomponent The activities can be done as pre-observation, during observation, post-observation

Item Review
FGD Review checklist External reviewers

Pretesting
Initial pretesting
Developed a manual how to use the PRPF and how to conduct proper observation Orientation for raters and ratees 2 raters are assigned for a ratee (primary and secondary raters) 183 ratees were completed

Final pretesting
175 ratees were completed Same procedure

PARF results (Reliability)

Initial pretesting (N=183, 89 items) Internal consistency for raters: .98 and .97
Planning and Preparation: .94 & .93 Classroom Environment: .93 & .92 Instruction: .94 & .92 Professional responsibility: .93 & .91

Final pretesting (N=175, 59 items) Internal consistency for raters: .97 and .96
Planning and Preparation: .94 & .93 Classroom Environment: .88 & .88 Instruction: .93 & .90 Professional responsibility: .92 & .76

PARF results (Rater agreement)

Initial pretesting (N=183, 89 items) =.47, p<.05 Primary raters, r of four domains: r=.73 to r=.88, p<.05 Secondary Rater, r of four raters: r=.64 to r=.83, p<.05. Concordance of Primary and secondary across 4 domains: .30-.45 Final pretesting (N=175, 59 items) =.45, p<.05 Primary raters, r of four domains: r=.69 to r=.65, p<.05 Secondary Rater, r of four raters: r=.64 to r=.87, p<.05. Concordance of Primary and secondary across 4 domains: .26-.49

PARF results (step function)

Initial pretesting (N=183, 89 items) Appropriate Threshold categories
Primary rater: -3.86, -1.20, -.36, and 4.24 Second rater: -3.26, -2.01, .81, and 4.07 There are only 9 misfitting items in the Rasch Model

Final pretesting (N=175, 59 items) Appropriate Threshold categories

primary rater are, -3.79, 1.95, .96 and 4.35, secondary rater, -3.90, -2.25, .32 and 3.60. There are only 6 misfitting items

PARF results (GRM)

Initial pretesting (N=183, 89 items) Final pretesting (N=175, 59 items)

Figure 1. Test Information Function of PRPF for the Primary Raters

Figure 3. Test Information Function of PRPF for the Primary Raters

Figure 2. Test Information Function of the PRPF of the Secondary Rater

Figure 4. Test Information Function of the PRPF of the Secondary Rater

Item Maps
Item Maps

Figure 6. Item Map of the PRPF for the Primary Rater

Figure 7. Item Map of the PRPF for the Secondary Rater

PARF results (Factorial Validity)

Initial pretesting (N=183, 89 items) A four factors structure (3 parcels each factor) 2=8829.23, df=3734, PGI=.97, Bentler-Bonnett Normed Fit Index=.96, Bentler-Bonnett Non-Normed Fit Index=.96.

PARF results (Construct Validity)

2=169.56, df=98, standardized RMS=.02, PGI=.95, GFI=.90, Bentler-Bonett NFI=.97, Bentler CFI=.99, Bollens Rho=.96.

Discussion
Consistent results were obtained in the preliminary and final pilot testing in terms of the internal consistencies, concordance of raters, convergence of factors, and factor structure. Theoretical construction of Danielsons Components of Professional Practice and Learner-centeredness. Very few items turn out to be bad fit. Precision of measurement

Problems:
Majority of scores given were very high Low discriminating power of the items Recommendations:
High cut off scores Include a wide array of faculty in the assessment Strict and objective raters

Contituous Project
Versions following the same framework:
Students rating for teachers (rating scale) Screening tool for teacher applicants (checklist) Community Service classes (Ratings scale) Culinary Laboratory classes (rubric) Seminar classes (rating scale) Faculty Advisers (rating scale)