Validation of a School Reform Rubric

Susan Gracia RIC Faculty Research Conference April 27, 2007

CSR

The objective of the federal Comprehensive School Reform (CSR) Program is to improve student achievement by supporting the implementation of comprehensive school reforms based on scientifically based research and effective practices so that all children, especially those in low-performing, high-poverty schools, can meet challenging state content and academic achievement standards (CSR Program Guidance, July 26, 2002). Specifically, comprehensive school reform is a means to improve student achievement through reorganizing and revitalizing the entire school, rather than implementing isolated programs.

CSR Components

CSR schools must plan a comprehensive program, as defined by these 11 eleven components: Effective, research-based, replicable methods and strategies Comprehensive design with aligned components Professional development Measurable goals and benchmarks Support within the school Support for principals and staff Parental and community involvement External technical support and assistance Evaluation strategies Coordination of resources Scientifically-based evidence of improved student achievement

CSR Evaluation

Implementation/progress of each component needs to be documented A number of instruments exist Rubrics, surveys, etc. purport to be able to identify stages of implementation of each component

E.g., CSR Implementation Continuum

Present Study

CSR Implementation Continuum was modified into a series of survey scales 153 teachers in CSR schools rated their perceptions of CSR implementation in their schools along the 11 CSR components

Research Questions

Do survey scales fit the Rasch model? If so:

What is the nature of the continuum of CSR implementation (for each component)? Does the continuum of CSR implementation match the CSR Implementation Continuum rubric? How can measurement of CSR implementation be improved?

Rasch model

People and items form a hierarchy of less than/more than on a continuum (variable) People are more likely to agree with easy (to endorse) items than difficult (to endorse) items Respondents perceiving high CSR implementation are more likely to endorse all items than those perceiving low implementation Probability of providing responses defines an order of respondents and items

Rasch model

Parameters are neither sample nor test dependent Total scores have interpretable meaning Ordinal data is converted to interval data Missing data is not problematic

Model (PD Scale)

CSR implementation data fit the model Fit statistics were well within defined bounds Item reliability = .97

How precisely items can be located on the latent variable The Rasch equivalent of the KR-20 or Cronbach Alpha "test reliability" statistic

Person reliability = .86

Item and person separation indices are 5.63 and 2.48

Items have sufficient breadth and persons are welldiscriminated.

Variable Map

A yardstickresulting from data Reveals the operational definition of what the CSR PD scale is measuring. The variable map allows one to see how items spread out over the continuum of CSR implementation and how respondents distribute on the CSR implementation variable.

People reporting highest implementation

-1

People reporting lowest implementation

-2

-3

Respondents MAP OF Items <more>|<rare> XXXX + | Most | difficult | items to XX | | endorse + | | XXXXX T| X | | XXXX + | | XXXXX | XXXXXX | S| XXXXX + | XXX | XXXXXXXXXXX | XXX | | XXXX + XXXX | MANY PD OPPS AVAILABLE TO PARENTS/COMMUNITY XXXXXXX M| XXXXXXXXXX |T XXXXXXX | XXXXXXXXX | XXXXXXXXXX + STAFF RECEIVE FEEDBACK ON STRATEGY IMPLEMENTATION XXXX |S XXXX | XXX | XXXX | PD PLAN RELATED TO SCHOOL GOALS/STAFF INPUT STAFF REGULARLY ATTENDS PD & WORKGROUPS XX S| PD CONTINUOUS, DIRECTED TOWARD COMMON GOALS SCHOOL DEVELOPS INTERNAL CAPACITY FOR PD XXXXXXX +M XXX | PD IS ONGOING PROCESS BASED ON STAFF NEEDS XXX | PD OPPS FOCUS ON FACULTY NEEDS RE: STRATEGIES SCHOOL HAS GENL PLAN FOR PD SCHOOL PROVIDES ONSITE PD & TIME FOR REFLECTION XX | | SOME PD OPPS AVAILABLE ONSITE XX |S XX + | PD OPPS RELATED TO GOALS STAFF GENERALLY ATTENDS PD X T| |T | | + Easiest items | | to endorse | X | | + <less>|<frequ>

Correspondence with original CSR Implementation Continuum

If expectations of what is being measured are based on good theory, the results shown in the empirical map should correspond to the conceptual one

Respondents MAP OF Items <more>|<rare> 6 XXXX + | | | XX | | 5 + | | XXXXX T| X | | 4 XXXX + | | XXXXX | XXXXXX | S| 3 XXXXX + | XXX | XXXXXXXXXXX | XXX | | 2 XXXX + XXXX | Level XXXXXXX M| XXXXXXXXXX |T XXXXXXX | XXXXXXXXX | 1 XXXXXXXXXX + Level XXXX |S XXXX | XXX | XXXX | Level XX S| Level 0 XXXXXXX +M XXX | Level XXX | Level XX | | Level XX |S -1 XX + | Level X T| |T | | -2 + | | | X | | -3 + <less>|<frequ>

2 5 4 1 2 1

Level 4 Level 5 Level 3 Level 3

Level 3

Results

Implementation as laid out in CSR Implementation Continuum rubric does not match implementation continuum as revealed through survey data Similar results for all 11 scales/CSR components CSR Implementation Continuum should be revised to reflect empirical data

REVISED CSR PROFESSIONAL DEVELOPMENT IMPLEMENTATION RUBRIC 1

Professional development opportunities at my school are related to school, district, or state goals. Staff, in general, attends professional development activities.

2
Some professional development opportunities are available onsite. The school provides some onsite professional development opportunities as well as time for reflective practice. The school has a general plan for professional development. Professional development opportunities focus on the needs of faculty to implement improvement strategies. Professional development is an ongoing process based on staff needs to implement improvement strategies.

3
Professional development is continuous and selfdirected towards the achievement of collectively developed school wide improvement goals. The school develops internal capacity to provide onsite professional development to its faculty and sponsors small grade level or subject matter work groups of teachers for reflective practice. Staff regularly participates in professional development activities and in small grade level or subject matter work groups to reflect on practice. The professional development plan is related to school wide improvement goals and is the result of staff input.

4
Instructional staff receive feedback on the implementation of improvement strategies.

5
Many of these professional development services are available to parents and community members.