W H AT D O W E M E A N BY C O L L E C T I N G DATA ?

W H AT D O W E M E A N BY A N A LY Z I N G DATA ?
W H Y S H O U L D YO U C O L L E C T A N D A N A LY Z E DATA F O R
YO U R E VA LU AT I O N ?
W H E N A N D BY W H O M S H O U L D DATA B E C O L L E C T E D
A N D A N A LY Z E D ?
H O W D O YO U C O L L E C T A N D A N A LY Z E DATA ?
In previous sections of this chapter, weve discussed studying the issue,
deciding on a research design, and creating an observational system for
gathering information for your evaluation. Now its time to collect your data
and analyze it figuring out what it means so that you can use it to draw
some conclusions about your work. In this section, well examine how to do
just that.

WHAT DO WE MEAN BY COLLECTING DATA?

Essentially, collecting data means putting your design for collecting
information into operation. Youve decided how youre going to get
information whether by direct observation, interviews, surveys,
experiments and testing, or other methods and now you and/or other
observers have to implement your plan. Theres a bit more to collecting data,
however. If you are conducting observations, for example, youll have to
define what youre observing and arrange to make observations at the right
times, so you actually observe what you need to. Youll have to record the
observations in appropriate ways and organize them so theyre optimally
useful.
Recording and organizing data may take different forms, depending on the
kind of information youre collecting. The way you collect your data should
relate to how youre planning to analyze and use it. Regardless of what
method you decide to use, recording should be done concurrent with data
collection if possible, or soon afterwards, so that nothing gets lost and
memory doesnt fade.
Some of the things you might do with the information you collect
include:

Gathering together information from all sources and observations

Making photocopies of all recording forms, records, audio or video

recordings, and any other collected materials, to guard against loss,
accidental erasure, or other problems

Entering narratives, numbers, and other information into a computer

program, where they can be arranged and/or worked on in various
ways

Performing any mathematical or similar operations needed to get

quantitative information ready for analysis. These might, for instance,
include entering numerical observations into a chart, table, or
spreadsheet, or figuring the mean (average), median (midpoint),
and/or mode (most frequently occurring) of a set of numbers.

Transcribing (making an exact, word-for-word text version of) the

contents of audio or video recordings

Coding data (translating data, particularly qualitative data that isnt

expressed in numbers, into a form that allows it to be processed by a
specific software program or subjected to statistical analysis)

Organizing data in ways that make them easier to work with. How you
do this will depend on your research design and your evaluation
questions. You might group observations by the dependent variable
(indicator of success) they relate to, by individuals or groups of
participants, by time, by activity, etc. You might also want to group
observations in several different ways, so that you can study
interactions among different variables.

There are two kinds of variables in research. An independent variable (the

intervention) is a condition implemented by the researcher or community to
see if it will create change and improvement. This could be a program,
method, system, or other action. A dependent variable is what may change
as a result of the independent variable or intervention. A dependent
variablecould be a behavior, outcome, or other condition. A smoking
cessation program, for example, is an independent variable that may change
group members smoking behavior, the primary dependent variable.

WHAT DO WE MEAN BY ANALYZING DATA?

Analyzing information involves examining it in ways that reveal the

relationships, patterns, trends, etc. that can be found within it. That may
mean subjecting it to statistical operations that can tell you not only what
kinds of relationships seem to exist among variables, but also to what level
you can trust the answers youre getting. It may mean comparing your
information to that from other groups (a control or comparison group,
statewide figures, etc.), to help draw some conclusions from the data. The
point, in terms of your evaluation, is to get an accurate assessment in order
to better understand your work and its effects on those youre concerned
with, or in order to better understand the overall situation.
There are two kinds of data youre apt to be working with, although not all
evaluations will necessarily include both. Quantitative data refer to the
information that is collected as, or can be translated into, numbers, which
can then be displayed and analyzed mathematically. Qualitative data are
collected as descriptions, anecdotes, opinions, quotes, interpretations, etc.,
and are generally either not able to be reduced to numbers, or are
considered more valuable or informative if left as narratives. As you might
expect, quantitative and qualitative information needs to be analyzed
differently.

Q U A N T I TAT I V E DATA
Quantitative data are typically collected directly as numbers. Some
examples include:

The frequency (rate, duration) of specific behaviors or conditions

Test scores (e.g., scores/levels of knowledge, skill, etc.)

Survey results (e.g., reported behavior, or outcomes to environmental

conditions; ratings of satisfaction, stress, etc.)

Numbers or percentages of people with certain characteristics in a

population (diagnosed with diabetes, unemployed, Spanish-speaking,
under age 14, grade of school completed, etc.)

Data can also be collected in forms other than numbers, and turned into
quantitative data for analysis. Researchers can count the number of times
an event is documented in interviews or records, for instance, or assign

numbers to the levels of intensity of an observed event or behavior. For

instance, community initiatives often want to document the amount and
intensity of environmental changes they bring about the new programs and
policies that result from their efforts. Whether or not this kind of translation
is necessary or useful depends on the nature of what youre observing and
on the kinds of questions your evaluation is meant to answer.
Quantitative data is usually subjected to statistical procedures such as
calculating the mean or average number of times an event or behavior
occurs (per day, month, year). These operations, because numbers are
hard data and not interpretation, can give definitive, or nearly definitive,
answers to different questions. Various kinds of quantitative analysis can
indicate changes in a dependent variable related to frequency, duration,
timing (when particular things happen), intensity, level, etc. They can allow
you to compare those changes to one another, to changes in another
variable, or to changes in another population. They might be able to tell you,
at a particular degree of reliability, whether those changes are likely to have
been caused by your intervention or program, or by another factor, known or
unknown. And they can identify relationships among different variables,
which may or may not mean that one causes another.

Q U A L I TAT I V E DATA
Unlike numbers or hard data, qualitative information tends to be soft,
meaning it cant always be reduced to something definite. That is in some
ways a weakness, but its also a strength. A number may tell you how well a
student did on a test; the look on her face after seeing her grade, however,
may tell you even more about the effect of that result on her. That look cant
be translated to a number, nor can a teachers knowledge of that students
history, progress, and experience, all of which go into the teachers
interpretation of that look. And that interpretation may be far more valuable
in helping that student succeed than knowing her grade or numerical score
on the test.
Qualitative data can sometimes be changed into numbers, usually by
counting the number of times specific things occur in the course of
observations or interviews, or by assigning numbers or ratings to dimensions
(e.g., importance, satisfaction, ease of use).
The challenges of translating qualitative into quantitative data have to do
with the human factor. Even if most people agree on what 1 (lowest) or 5

(highest) means in regard to rating satisfaction with a program, ratings of

2, 3, and 4 may be very different for different people. Furthermore, the
numbers say nothing about why people reported the way they did. One may
dislike the program because of the content, the facilitator, the time of day,
etc. The same may be true when youre counting instances of the mention of
an event, such as the onset of a new policy or program in a community
based on interviews or archival records. Where one person might see a
change in program he considers important another may omit it due to
perceived unimportance.
Qualitative data can sometimes tell you things that quantitative data cant.
It may reveal why certain methods are working or not working, whether part
of what youre doing conflicts with participants culture, what participants
see as important, etc. It may also show you patterns in behavior, physical
or social environment, or other factors that the numbers in your
quantitative data dont, and occasionally even identify variables that
researchers werent aware of.
It is often helpful to collect both quantitative and qualitative information.
Quantitative analysis is considered to be objective without any human bias
attached to it because it depends on the comparison of numbers according
to mathematical computations. Analysis of qualitative data is generally
accomplished by methods more subjective dependent on peoples opinions,
knowledge, assumptions, and inferences (and therefore biases) than that of
quantitative data. The identification of patterns, the interpretation of
peoples statements or otherCOMMUNICATION , the spotting of trends all of
these can be influenced by the way the researcher sees the world. Be
aware, however, that quantitative analysis is influenced by a number of
subjective factors as well. What the researcher chooses to measure, the
accuracy of the observations, and the way the research is structured to ask
only particular questions can all influence the results, as can the researchers
understanding and interpretation of the subsequent analyses.

WHY SHOULD YOU COLLECT AND ANALYZE DATA

FOR YOUR EVALUATION?
Part of the answer here is that not every organization particularly small
community-based or non-governmental ones will necessarily have
extensive resources to conduct a formal evaluation. They may have to be
content with less formal evaluations, which can still be extremely helpful in

providing direction for a program or intervention. An informal evaluation will

involve some data gathering and analysis. This data collection and
sensemaking is critical to an initiative and its future success, and has a
number of advantages.

The data can show whether there was any significant change in
the dependent variable(s) you hoped to influence. Collecting and
analyzing data helps you see whether your intervention brought about
the desired results

The term significance has a specific meaning when youre discussing

statistics. The level of significance of a statistical result is the level of
confidence you can have in the answer you get. Generally, researchers dont
consider a result significant unless it shows at least a 95% certainty that its
correct (called the .05 level of significance, since theres a 5% chance that
its wrong). The level of significance is built into the statistical formulas:
once you get a mathematical result, a table (or the software youre using)
will tell you the level of significance.
Thus, if data analysis finds that the independent variable (the intervention)
influenced the dependent variable at the .05 level of significance, it means
theres a 95% probability or likelihood that your program or intervention had
the desired effect. The .05 level is generally considered a reasonable result,
and the .01 level (99% probability) is considered about as close to certainty
as you are likely to get. A 95% level of certainty doesnt mean that the
program works on 95% of participants, or that it will work 95% of the time. It
means that theres only a 5% possibility that it isnt actually whats
influencing the dependent variable(s) and causing the changes that it seems
to be associated with.

They can uncover factors that may be associated with changes

in the dependent variable(s). Data analyses may help discover
unexpected influences; for instance, that the effort was twice as large
for those participants who also were a part of a support group. This can
be used to identify key aspects of implementation.

They can show connections between or among various factors

that may have an effect on the results of your evaluation. Some
types of statistical procedures look for connections (correlations is
the research term) among variables. Certain dependent variables may
change when others do. These changes may be similar i.e., both
variables increase or decrease (e.g., as childrens proficiency at

reading increases, the amount of reading they do also increases). Or

the opposite may be observed i.e. the two variables change in
opposite directions (as the amount of exercise they engage in
increases, peoples weight decreases). Correlations dont mean that
one variable causes another, or that they both have the same cause,
but they can provide valuable information about associations to expect
in an evaluation.

They can help shed light on the reasons that your work was
effective or, perhaps, less effective than youd hoped. By
combining quantitative and qualitative analysis, you can often
determine not only what worked or didnt, but why. The effect of
cultural issues, how well methods are used, the appropriateness of
your approach for the population these as well as other factors that
influence success can be highlighted by careful data collection and
analysis. This knowledge gives you a basis for adapting and changing
what you do to make it more likely youll achieve the desired outcomes
in the future.

They can provide you with credible evidence to show

stakeholders that your program is successful, or that youve
uncovered, and are addressing limitations. Stakeholders, such as
funders and community boards, want to know their investments are
well spent. Showing evidence of intermediate outcomes (e.g. new
programs and policies) and longer-term outcomes (e.g., improvements
in education or health indicators) is becoming increasingly important to
receiving and retaining funding.

Their use shows that youre serious about evaluation and

about improving your work. Being a good trustee or steward of
community investment includes regular review of data regarding
progress and improvement.

They can show the field what youre learning, and thus pave
the way for others to implement successful methods and
approaches. In that way, youll be helping to improve community
efforts and, ultimately, quality of life for people who benefit.

WHEN AND BY WHOM SHOULD DATA BE

COLLECTED AND ANALYZED?
As far as data collection goes, the when part of this question is relatively
simple: data collection should start no later than when you begin your work
or before you begin in order to establish a baseline or starting point and
continue throughout. Ideally, you should collect data for a period of time
before you start your program or intervention in order to determine if there
are any trends in the data before the onset of the intervention. Additionally,
in order to gauge your programs longer-term effects, you should collect
follow-up data for a period of time following the conclusion of the program.
The timing of analysis can be looked at in at least two ways: One is that its
best to analyze your information when youve collected all of it, so you can
look at it as a whole. The other is that if you analyze it as you go along, youll
be able to adjust your thinking about what information you actually need,
and to adjust your program to respond to the information youre getting.
Which of these approaches you take depends on your research purposes. If
youre more concerned with a summative evaluation finding out whether
your approach was effective, you might be more inclined toward the first. If
youre oriented toward improvement a formative evaluation we
recommend gathering information along the way. Both approaches are
legitimate, but ongoing data collection and review can particularly lead to
improvements in your work.
The who question can be more complex. If youre reasonably familiar with
statistics and statistical procedures, and you have the resources in time,
money, and personnel, its likely that youll do a somewhat formal study,
using standard statistical tests. (Theres a great deal of software both for
sale and free or open-source available to help you.)
If thats not the case, you have some choices:

You can hire or find a volunteer outside evaluator, such as from a

nearby college or university, to take care of data collection and/or
analysis for you.

You can conduct a less formal evaluation. Your results may not be as
sophisticated as if you subjected them to rigorous statistical
procedures, but they can still tell you a lot about your program. Just

the numbers the number of dropouts (and when most dropped out),
for instance, or the characteristics of the people you serve can give
you important and usable information.

You can try to learn enough about statistics and statistical software to
conduct a formal evaluation yourself. (Take a course, for example.)

You can collect the data and then send it off to someone a university
program, a friendly statistician or researcher, or someone you hire to
process it for you.

You can collect and rely largely on qualitative data. Whether this is an
option depends to a large extent on what your program is about. You
wouldnt want to conduct a formal evaluation of effectiveness of a new
medication using only qualitative data, but you might be able to draw
some reasonable conclusions about use or compliance patterns from
qualitative information.

If possible, use a randomized or closely matched control group for

comparison. If your control is properly structured, you can draw some
fairly reliable conclusions simply by comparing its results to those of
your intervention group. Again, these results wont be as reliable as if
the comparison were made using statistical procedures, but they can
point you in the right direction. Its fairly easy to tell whether or not
theres a major difference between the numbers for the two or more
groups. If 95% of the students in your class passed the test, and only
60% of those in a similar but uninstructed control group did, you can
be pretty sure that your class made a difference in some way, although
you may not be able to tell exactly what it was that mattered. By the
same token, if 72% of your students passed and 70% of the control
group did as well, it seems pretty clear that your instruction had
essentially no effect, if the groups were starting from approximately
the same place.

Who should actually collect and analyze data also depends on the form of
your evaluation. If youre doing a participatory evaluation, much of the data
collection - and analyzing - will be done by community members or program
participants themselves. If youre conducting an evaluation in which the

observation is specialized, the data collectors may be staff members,

professionals, highly trained volunteers, or others with specific skills or
training (graduate students, for example). Analysis also could be
accomplished by a participatory process. Even where complicated statistical
procedures are necessary, participants and/or community members might be
involved in sorting out what those results actually mean once the math is
done and the results are in. Another way analysis can be accomplished is by
professionals or other trained individuals, depending upon the nature of the
data to be analyzed, the methods of analysis, and the level of sophistication
aimed at in the conclusions.

HOW DO YOU COLLECT AND ANALYZE DATA?

Whether your evaluation includes formal or informal research procedures,
youll still have to collect and analyze data, and there are some basic steps
you can take to do so.

I M P L E M E N T YO U R M E A S U R E M E N T S Y S T E M
We've previously discussed designing an observational system to
gather information. Now its time to put that system in place.

Clearly define and describe what measurements or observations are

needed. The definition and description should be clear enough to
enable observers to agree on what theyre observing and reliably
record data in the same way.

Select and train observers. Particularly if this is part of a participatory

process, observers need training to know what to record; to recognize
key behaviors, events, and conditions; and to reach an acceptable
level of inter-rater reliability (agreement among observers).

Conduct observations at the appropriate times for the appropriate

period of time. This may include reviewing archival material;
conducting interviews, surveys, or focus groups; engaging in direct
observation; etc.

Record data in the agreed-upon ways. These may include pencil and
paper, computer (using a laptop or handheld device in the field,
entering numbers into a program, etc.), audio or video, journals, etc.

O R G A N I Z E T H E DATA YO U V E C O L L E C T E D
How you do this depends on what youre planning to do with it, and
on what youre interested in.

Enter any necessary data into the computer. This may mean simply
typing comments, descriptions, etc., into a word processing program,
or entering various kinds of information (possibly including audio and
video) into a database, spreadsheet, a GIS (Geographic Information
Systems) program, or some other type of software or file.

Transcribe any audio- or videotapes. This makes them easier to work

with and copy, and allows the opportunity to clarify any hard-tounderstand passages of speech.

Score any tests and record the scores appropriately.

Sort your information in ways appropriate to your interest. This may

include sorting by category of observation, by event, by place, by
individual, by group, by the time of observation, or by a combination or
some other standard.

When possible, necessary, and appropriate, transform qualitative into

quantitative data. This might involve, for example, counting the
number of times specific issues were mentioned in interviews, or how
often certain behaviors were observed.

C O N D U C T DATA G RA P H I N G , V I S U A L I N S P E C T I O N ,
S TAT I S T I C A L A N A LY S I S , O R O T H E R O P E RAT I O N S O N T H E
DATA A S A P P R O P R I AT E
Weve referred several times to statistical procedures that you can apply to
quantitative data. If you have the right numbers, you can find out a great
deal about whether your program is causing or contributing to change and
improvement, what that change is, whether there are any expected or
unexpected connections among variables, how your group compares to
another youre measuring, etc.
There are other excellent possibilities for analysis besides statistical
procedures, however. A few include:

Simple counting, graphing and visual inspection of frequency or rates

of behavior, events, etc., over time.

Using visual inspection of patterns over time to identify

discontinuities (marked increases, decreases) in the measures over
time (sessions, weeks, months).

Calculating the mean (average), median (midpoint), and/or mode

(most frequent) of a series of measurements or observations. What
was the average blood pressure, for instance, of people who exercised
30 minutes a day at least five days a week, as opposed to that of
people who exercised two days a week or less?

Using qualitative interviews, conversations, and participant

observation to observe (and track changes in) the people or
situation. Journals can be particularly revealing in this area because
they record peoples experiences and reflections over time.

Finding patterns in qualitative data. If many people refer to similar

problems or barriers, these may be important in understanding the
issue, determining what works or doesnt work and why, or more.

Comparing actual results to previously determined goals or

benchmarks. One measure of success might be meeting a goal for
planning or program implementation, for example.

TA K E N O T E O F A N Y S I G N I F I C A N T O R I N T E R E S T I N G R E S U LT S
Depending on the nature of your research, results may be statistically
significant (the 95% or better certainty that we discussed earlier), or simply
important or unusual. They may or may not be socially significant (i.e., large
enough to solve the problem).
There are a number of different kinds of results you might be
looking for.

Differences within people or groups. If you have repeated

measurements for individuals/groups over time, we can see if there are
marked increases/decreases in the (frequency, rate) of behavior
(events, etc.) following introduction of the program or intervention.

When the effects are seen when and only when the intervention is
introduced and if the intervention is staggered (delayed) across
people or groups this increases our confidence that the intervention,
and not something else, is producing the observed effects.

Differences between or among two or more groups. If you have one or

more randomized control groups in a formal study (groups that are
drawn at random from the same population as the group in your
program, but are not getting the same program or intervention, or are
getting none at all), then the statistical significance of differences
between or among the groups should tell you whether your program
has any more influence on the dependent variable(s) than whats
experienced by the other groups.

Results that show statistically significant changes. With or without a

control or comparison group, many statistical procedures can tell you
whether changes in dependent variables are truly significant (or not
likely due to chance). These results may say nothing about the causes
of the change (or they may, depending on how youve structured your
evaluation), but they do tell you whats happening, and give you a
place to start.

Correlations. Correlation means that there are connections between or

among two or more variables. Correlations can sometimes point to
important relationships you might not have predicted. Sometimes they
can shed light on the issue itself, and sometimes on the effects of a
groups cultural practices. In some cases, they can highlight potential
causes of an issue or condition, and thus pave the way for future
interventions.

Correlation between variables doesnt tell you that one necessarily causes
the other, but simply that changes in one have a relationship to changes in
the other. Among American teenagers, for instance, there is probably a fairly
high correlation between an increase in body size and an understanding of
algebra. This is not because one causes the other, but rather the result of
the fact that American schools tend to begin teaching algebra in the seventh,
eighth, or ninth grades, a time when many 12-, 13-, and 14-year-olds are
naturally experiencing a growth spurt.

On the other hand, correlations can reveal important connections. A very

high correlation between, for instance, the use of a particular medication and
the onset of depression might lead to the withdrawal of that medication, or
at least a study of its side effects, and increased awareness and caution
among doctors who prescribe it. A very high correlation between gang
membership and having a parent with a substance abuse problem may not
reveal a direct cause-and-effect relationship, but may tell you something
important about who is more at risk for substance abuse.

Patterns. In both quantitative and qualitative information, patterns

often emerge: certain health conditions seem to cluster in particular
geographical areas; people from a particular group behave in similar
ways; etc. These patterns may not be specifically what you were
looking for or expected to find, but they may either be important in
themselves or shed light on the areas youre interested in. In some
cases, you may need to subject them to statistical procedures
(regression analysis, for example) to see if, in fact, theyre random, or
if they constitute actual patterns.

Obvious important findings. Whether as a result of statistical analysis,

or of examination of your data and application of logic, some findings
may stand out. If 70% of a group of overweight participants in a
healthy eating and physical activity program lowered their weight and
blood pressure significantly, compared to only 20% of a similar group
not in the program, you can probably assume that program may have
been effective. If theres no change whatsoever in education
outcomes after two years of your education program, then youre
either running an ineffective program, or youre simply not reaching
those who are most likely to have poorer outcomes (which can also be
interpreted to mean youre running an ineffective program.)

Not all important findings will necessarily tell you whether your program
worked, or what is the most effective method. It might be obvious from your
data collection, for instance, that, while violence or roadway injuries may not
be seen as a problem citywide, they are much higher in one or more
particular areas, or that the rates of diabetes are markedly higher for
particular groups or those living in areas with greater disparities of income. If
you have the resources, its wise to look at the results of your research in a
number of different ways, both to find out how to improve your program, and
to learn what else you might do to affect the issue.

I N T E R P R E T T H E R E S U LT S
Once youve organized your results and run them through whatever
statistical or other analysis youve planned for, its time to figure out what
they mean for your evaluation. Probably the most common question that
evaluation research is directed toward is whether the program being
evaluated works or makes a difference. In research terms, that often
translates to What were the effects of the independent variable (the
program, intervention, etc.) on the dependent variable(s) (the behavior,
conditions, or other factors it was meant to change)? There are a number of
possible answers to this question:

Your program had exactly the effects on the dependent variable(s) you
expected and hoped it would. Statistics or other analysis showed clear
positive effects at a high level of significance for the people in your
program and if you used a multiple-group design none, or far fewer,
of the same effects for a similar control group and/or for a group that
received a different intervention with the same purpose. Your early
childhood education program, for instance, greatly increased
development outcomes for children in the community, and also
contributed to an increase in the percentage of children succeeding in
school.

Your program had no effect. Your program produced no significant

results on the dependent variable, whether alone or compared to other
groups. This would mean no change as a result of your program or
intervention.

Your program had a negative effect. For instance, intimate partner

violence increased (or at least appeared to) as a result of your
intervention. (It is relatively common for reported events, such as
violence or injury, to increase when the intervention results in
improved surveillance and ease of reporting).

Your program had the effects you hoped for and other effects as well.
o These effects might be positive. Your youth violence prevention
program, for instance, might have resulted in greatly reduced
violence among teens, and might also have resulted in

significantly improved academic performance for the kids

involved.
o These effects might be neutral. The same youth violence
prevention program might somehow result in youth watching TV
more often after school.
o These effects might be negative. (These effects are usually called
unintended consequences.) Youth violence might decrease
significantly, but the incidence of teen pregnancies or alcohol
consumption among youth in the program might increase
significantly at the same time.
o These effects might be multiple, or mixed.For instance, a
program to reduce HIV/AIDS might lower rates of unprotected
sex but might also increase conflict and instances of partner
violence. Your program had no effect or a negative effect and
other effects as well. As with programs with positive effects,
these might be positive, neutral, or negative; single or multiple;
or consistent or mixed.
If your analysis gives you a clear indication that what youre doing is
accomplishing your purposes, interpretation is relatively simple: You should
keep doing it, while trying out ways to make it even more effective, or while
aiming at other related issues as well.
As we discuss elsewhere in the Community Tool Box, good programs are
dynamic -- constantly striving to improve, rather than assuming that what
theyre doing is as good as it can be.
If your analysis shows that your program is ineffective or negative, however
or, for that matter, if a positive analysis leaves you wondering how to make
your successful efforts still more successful interpretation becomes more
complex. Are you using an absolutely wrong approach? Are you using an
approach that could be effective, but is poorly implement? Is there a
particular contributing factor youre failing to take into account? Are there
barriers to success of culture, experience, personal characteristics,
systematic discrimination present in the population from which participants
are drawn? Are there particular components or elements you can change to

make your program more effective, or should you start again from scratch?
What should you address to make a good program better?
Careful and insightful interpretation of your data may allow you to answer
questions like these. You may be able to use correlations, for instance, to
generate hypotheses about your results. If positive or negative changes in
particular variables are consistently associated with positive or negative
changes in other variables, the two may be connected. (The word may is
important here. The two may be connected, but they may not, or both may
be related to a third variable that youre not aware of or that you consider
trivial.) Such a connection can point the way toward a factor (e.g., access to
support) that is causing the changes in both variables, and that must be
addressed to make your program successful. Correlations may also indicate
patterns in your data, or may lead to an unexpected way of looking at the
issue youre addressing.
You can often use qualitative data to understand the meaning of an
intervention, and peoples reactions to the results.The observation that
participants are continually suffering from a variety of health problems may
be traced, through qualitative data, to nutrition problems (due either to
poverty or ignorance) or to lack of access to health services, or to cultural
restrictions (some Muslim women may be unwilling or unable because of
family prohibition to accept care and treatment from male doctors, for
example).
Once you have organized your data, both statistical results and anything that
cant be analyzed statistically need to be analyzed logically. This may not
give you convincing information but it will almost undoubtedly give you some
ideas to follow up on, and some indications of connections and avenues you
might not yet have considered. It will also show you some additional results
people reacting differently than before to the program, for example. The
numbers can tell you whether there is change, but they cant always tell you
what causes it or why (although they sometimes can), or why some people
benefit while others dont. Those are often matters for logical analysis, or
critical thinking.
Analyzing and interpreting the data youve collected brings you, in a sense,
back to the beginning. You can use the information youve gained to adjust
and improve your program or intervention, evaluate it again, and use that

information to adjust and improve it further, for as long as it runs. You have
to keep up the process to ensure that youre doing the best work you can
and encouraging changes in individuals, systems, and policies that make for
a better and healthier community.
You have to become a cultural detective to understand your initiative, and, in
some ways, every evaluation is an anthropological study.

IN SUMMARY
The heart of evaluation research is gathering information about the program
or intervention youre evaluating and analyzing it to determine what it tells
you about the effectiveness of what youre doing, as well as about how you
can maintain and improve that effectiveness.
Collecting quantitative data information expressed in numbers and
subjecting it to a visual inspection or formal statistical analysis can tell you
whether your work is having the desired effect, and may be able to tell you
why or why not as well. It can also highlight connections (correlations)
among variables, and call attention to factors you may not have considered.
Collecting and analyzing qualitative data interviews, descriptions of
environmental factors, or events, and circumstances can provide insight
into how participants experience the issue youre addressing, what barriers
and advantages they experience, and what you might change or add to
improve what you do.
Once youve gained the knowledge that your information provides, its time
to start the process again. Use what youve learned to continue to evaluate
what you do by collecting and analyzing data, and continually improve your
program.