Академический Документы
Профессиональный Документы
Культура Документы
Program Evaluation
Nazanin Mohajeri-Nelson, Ph.D. and Tina Negley, M.A.
Office of Data, Program Evaluation, and Reporting (DPER)
February 4, 2016
Introduction to
Program Evaluation
What is Program Evaluation?
Process to Determine if an intended outcome is
reached using empirical methods and evidence.
Process for judging the worth of some intervention,
strategy, or service.
3
Need for Local Evaluation
4
Purpose of Program Evaluation:
What Should Be Learned from the Results?
5
Theory of Action
8
PE process always starts with the question(s)…
Purpose
– Evaluation Question(s)
Data Identification
Data -Identify goals & measurable objectives of project being
Collection evaluated
& Storage -Determine if currently available data can be used to
address evaluation question; If not, create data
collection (collect and store data)
Data Preparation
-Gather & merge data from various sources
-Clean data
14
Sections of Logic Models
Source: http://meera.snre.umich.edu/sites/all/images/step-2-logic-model-big.png
15
Narrow the Focus
Evaluation Purpose
Why are we evaluating? What do we hope to learn?
Do our evaluation questions meet this purpose?
Evaluation Needs
What resources (time, capacity) do we have to conduct the
evaluation? Is that enough to conduct the type and extent of
evaluation we need?
Is this something we can do internally or should we contract out?
How quickly do we need the results/findings?
Evaluation Plans
16
The Evaluation Plan
Designing the Plan :
Will D e pe n d on Q ue s t ion , D at a Av a il a ble, & P u r p o se of
t h e Ev alu at ion
Evaluation Plan
Include evaluator as early as possible in the planning
process.
Numerous evaluation methods ~ two commonly used
Treatment v. comparison group
Pre, mid-, and post-program comparison
Sample plan
Program Goals Target Outcomes Steps to Achieve Analysis and Action Results Person Time Frame
the Outcome Plan Responsible
Teachers who Treatment teachers, Math PD will be Compare the average Program Coordinator Fall the year following
attend math on average, will have provided as planned post-test score of In collaboration with the summer
professional a 5% higher score in the proposal attending teachers to Program Evaluator professional
development will than the comparison non-attending development
have higher scores group teachers from same
on a math content school
knowledge Treatment teachers
assessment will increase CK by Compare pre-test
10 points on pre- to score of attending
post-assessment teachers to their post-
test scores
18
Sample Template
Source:
http://www.stepstoolkit.org/index.php?option=com_courses&task=ShowModule&Module
=8&type=T&CatId=54&Itemid=142&lang=en
19
CDE Evaluation Plan Template
Program: Evaluat 20122013 - Team FTE NMN Report Preliminary September 15, 2014
ion 20132014 devoted to Projected Analyses
Period: this program Timeline
Coordina Segmen evaluation: EY/T Final Analyses December 31, 2014
tor: ts: N and Report
20
Broad Evaluation Steps and/or more specific Data to be used Analysis Method Comments/
Question questions Notes
What is the overall impact of 1. Identify all SS students and their resident 1. Reading 1. Calculate the
Summer School participation on schools. a. CBLA percentage of students
Student Academic Performance? 2. Comparison students: i. 2010-2011 that moved up in PL
a. If the school/district already ii. 2011-2012 from 20102011 to
identified comparison students, b. CSAP PL, SGP 20112012 for SS and
then include those in the i. 2010-2011 comparison students
analyses. ii. 2011-2012 2. Calculate the MGP for
b. If school/district did not 2. Math SS and comparison
include comparison students, a. CSAP PL, SGP students
create comparison groups i. 2010-2011
i. take SASIDS of all ii. 2011-2012
students in the same
grades as those
students that
participated from
each resident school;
ii. Remove SS students
from list;
iii. Select a random
sample of students
who were eligible for
participating in SS
but did not with
similar demographics
to those students that
did participate
21
Find and Use
the Right Data
Purpose
– Evaluation Question(s)
Data Identification
Data -Identify goals & measurable objectives of project being
Collection evaluated
& Storage -Determine if currently available data can be used to
address evaluation question; If not, create data
collection (collect and store data)
Data Preparation
-Gather & merge data from various sources
-Clean data
24
Example
Source: http://meera.snre.umich.edu/sites/all/images/logic-model-big.png
25
Data Identification
26
Data Preparation and
Analysis
Purpose
– Evaluation Question(s)
Data Identification
Data -Identify goals & measurable objectives of project being
Collection evaluated
& Storage -Determine if currently available data can be used to
address evaluation question; If not, create data
collection (collect and store data)
Data Preparation
-Gather & merge data from various sources
-Clean data
30
Services received
Program type
Format and location
Contracted hours completed
Example
NEP or LEP (WIDA ACCESS)
SES Data Used
Assessment data
TCAP proficiency
TCAP growth (MGP)
DRA-2
ACCESS proficiency
ACCESS growth (MGP)
31
Treatment v. Comparison Group approach
Treatment group (received SES)
Participated in SES, enrolled in eligible
school
Example Two years of assessment data available
Completed all tutoring or at least 20 hours
SES Evaluation
Components Comparison group
Randomized sample of students who did not
participate, same schools
Proportional distribution based on grade and
proficiency
Compare the percent of students receiving SES
that increased at least one proficiency level to
the performance of the comparison students
Compare the MGP of students receiving SES to
the MGP of comparison students
32
Disseminate the Findings
Purpose
– Evaluation Question(s)
Data Identification
Data -Identify goals & measurable objectives of project being
Collection evaluated
& Storage -Determine if currently available data can be used to
address evaluation question; If not, create data
collection (collect and store data)
Data Preparation
-Gather & merge data from various sources
-Clean data
Purpose of Dissemination
Accountability?
Justification?
Program Improvement?
Targeted Audience
Disseminating findings
Theory of Action
Logic Model
Methods
Results
Conclusions and Recommendations (based on purpose)
35
Create Your Report
For the Intended Users
Methodology ~ Can It Be Replicated?
Describe each data point used
The collection method and process
Describe the possible ranges and results
How many responded to survey?
What were the response options?
What was the distribution of responses?
Time frame covered by evaluation
Analysis plan and techniques to be used
Program Reach
Describe the targeted recipients versus the actual participants
Describe the frequency, intensity, duration of program
36
Create Your Report
For the Intended Users
Results ~ What Did the Data Show?
Describe the analyses conducted
Describe or even better show the descriptive data
Tables, graphs, charts
Data ranges, averages, medians
Describe or even better show the results
Changes in scores
Differences between groups
Variance
Rankings
Noteworthy trends
Quotes that exemplify the results
37
Create Your Report
For the Intended Users
Conclusions ~ Would Another Evaluator Reach the Same?
Findings ~ describe what was learned
Include what was expected and what was unexpected / were any of the
findings surprising?
Describe limitations in data, methods, or design
Conclusions drawn based on findings
Did the program have the impact that had been hoped for? In what
way?
Lessons learned from the evaluation
Next steps and further investigations
38
CDE’s Evaluations
39
Internal
Overall program effectiveness
SES – how we
share results Academic and linguistic performance of
treatment groups compared to comparison
groups
Best practices
Location of services
Format (group vs. individual; online)
Hours of services completed
Effective providers/programs
External
Effective providers/programs
Best practices
40
Questions?
Contact
Nazanin Mohajeri-Nelson, Ph.D.
Mohajeri-nelson_n@cde.state.co.us
Tina Negley
Negley_t@cde.state.co.us
41
We would appreciate your honest feedback
https://docs.google.com/forms/d/1sMhYz61gXdE
nSwWR3DD0Ddv1ArRSqm12bRPJBqioBOg/viewfo
rm?c=0&w=1
Thank you!
42