The 2010 User Friendly

Handbook for Project

Evaluation
Chapter 4
The Evaluation Process

THE EVALUATION
PROCESS:
CARRYING OUT
THE STUDY AND
REPORTING

Conducting the Data

Collection

When conducting data collection there are several important steps to

take to ensure proper collection.

- Step 1: Obtain necessary clearances and permissions

- Step 2: Consider needs and sensitivities of respondents
- Step 3: Ensure proper training of data collectors
- Step 4: Collect data from as many members of sample as possible
- Step 5: Cause as little disruption as possibly to the ongoing effort

Step 1: Obtain Clearances and

Permissions
- Some school systems have pre-determined procedures for collecting data
about students, teachers, and projects.

- Parental permission is frequently required for children to participate in

data collection.
- Offer to share what is learned from the data once the evaluation is
completed.
- This can help gain the cooperation of the participants needed.

Step 2:
Consider the
Needs of the
Respondent
s

- Be open and honest about

the evaluation and its results
with the participants in the
evaluation
- Guarantee anonymity when
the evaluation allows.
- There are certain instances
when anonymity is not an
option.

- Access the American

Evaluation Associations
guiding principles on this
topic for more useful tips in
reference to data collection.

Step 3: Ensure Adequate

Training
is Received
- Data collectors should
be adequately trained prior to evaluation
- This training should include educating the collectors on the culture of the are.a where
data will be collected.
- Being knowledgeable about the culture where the evaluation will take place will lead to
more effective interactions where understanding can be maximized.

- Data collectors should be trained to hold no bias.

- Inexperience collectors often influence respondents with their own perceptions.

- Data collectors should be consistent throughout their collection and

between other data collectors.
- The process of data collection should be consistent.

Step 4: Obtain Data on Most of

Sample
Size process is dependent on who is selected and the
- Validity of the evaluation
success in obtaining as much data as possible from the selected group.
- A response rate of 70% is considered a high quality response rate.

- Seek out nonresponders

- Send out multiple surveys and follow-up after initial contacts.

- Follow-up with Evaluation Drop-outs

- This will allow for a more complete picture of the data.

Step 5: Do
Not Be
Disruptive

- The process of data gathering

should be as little of a
disruption as possible.
- Be sensitive to the time
constraints of participants
and the project.
- Be flexible and ready to
handle changes in the
evaluation.

Analyzing
the Data

Check the raw data and

prepare them for analysis.
Conduct initial analysis based
on the evaluation plan.
Conduct additional analyses
based on the initial results.
Integrate and synthesize
findings.

Step 1: Viewing and Analyzing

Raw Data
Check data for responses that may be out of the ordinary
Ex: Data that shows a person choosing more than 1 answer, always choosing
the third alternative on a multiple-choice test, reporting amounts of time that
add up to more than 100 percent, giving inconsistent answers, etc.

Eliminate this data

Step 2: Conduct Initial Data

Analysis
Start with the set of analyses that seemed originally to be of interest.
Code and Enter Data into computer as needed.
Be sure to follow the procedures that are currently in place.
Analysts still need to be careful, that the data sets they are using
meet the assumptions of the technique being used.
Using an incorrect technique can result in invalidation of the whole
evaluation project.

Step 3: Conduct Additional

It is veryAnalyses
likely

that the initial

analyses will
raise as many
questions as they
answer.

Step 3:
Conduct
Additional
Analyses

Conduct a second set of

analyses to further
address questions held
during the first analysis.
Look for emerging patterns
of data that suggest other
interesting avenues to
explore.
Most likely you will acquire
findings that were not
anticipated.
Be sure to use proper
techniques for qualitative
data analysis.

Step 4: Integrate and

Synthesize
Findings
Present the results
of the
At times, there are not clear or
varied analyses.

Integrate the separate

analyses into an overall
picture.
Develop conclusions regarding
what the data shows.

consistent findings.

If this occurs, the findings must

be allowed to stand as they
are, unresolved.

Report the
Findings

Background
Evaluation Study Questions
Evaluation Procedures

This stage requires a collection

of data and findings. The
report includes six key
components.

Data Analysis
Findings
Conclusions

Background
The Background is the foundation of the project. The following
information is reported:

Problem or needs
Literature review
Stakeholders
Participants
Project objectives
Project activities and components
Project longevity and location
Resources
Projects measurable outcomes

Evaluation Study
Questions
This section of the report describes the questions that
the study addressed. As relevant, it also points out
some important questions that could not be addressed
because of factors such as time, resources, or
inadequacy of available data collection techniques.

Evaluation Procedure:

This section
describes the groups who participated and the
data collected.
The following questions are answered.
Groups:
Who are they?
How were they selected?
How do they represent the population?
Are they volunteers or chosen?
Were comparison groups included?

The following information describes the data

and the instruments used (A matrix or table
can be useful):
Data collected through reports such as
GPA (grade point average).
Information obtained in questionnaires
and interviews.
Data identified through observation.
Extant data such as budgets.

Data Analysis:
This section describes the
techniques used to analyze the
data that were collected. It
describes the various stages of
analysis that were implemented
and the checks that were carried
out to make sure that the data
were free of as many confounding
factors as possible.

Findings
This section gives the results of
the study.

The evaluation study question are

stated again.
Each question is addressed and
results are given (including
inconclusive results).
Visuals to support the data.
A summary of the findings including
the outcome, answered questions,
and unanswered questions.

Conclusion:
The conclusion section reports the findings with more broadbased and summative statements. These statements must
relate the findings of the projects evaluation questions and
relate to the overall goals of the program. Sometimes the
conclusion section goes a step further and includes
recommendations.

Additional Sections
These additional sections may be included
in a report:
An abstract or summary.
An executive summary or detailed
overview.

How do you Develop a Report?

Combine background information, goals, objectives, and results.
Present the information in a variety of forms to appeal to all readers.
Combine technical data, visuals, and nontechnical summaries.
Utilize technical tools and instruments for an attention-getting report.
The finding and conclusion sections are important.
Mention, but spend little time, on inconclusive information.
Ask colleagues and stakeholders to read and review the report before
submission.

Disseminating the
Information
(Dispersed throughout)

Step 1: Identifying the

audience
for
dissemination
Idealy audiences should be determined simultaneously with project design.
Develop a dissemination report.
Language of the report should fit the audience.
A distinction between statistical and practical importance may be too
much to provide to high level decision makers. Instead, only findings that
are of practical importance should be presented.

The Need to
Know
Step
2: Development
of the dissemination
Customize
the dissemination.
Taylor the dissemination to various groups.
What do the various groups need to know?
What is the best manner for communicating the information?
Package the information for others to use.
Include support materials.

Step 3:
Potential
The funding source(s).Audiences
Potential funding source(s).
Others involved with similar projects or areas of research.
Community members directly involved in the project.
Members of the business or political community.

Discussion Question
- In what ways is this process of conducting data
collection, analyzing the data collected, and
disseminating those findings similar to actions you
take in the classroom when evaluating students?