DESIGNING IMPACT EVALUATIONS :

Assessing Jamaica's PATH Program

IEDM No. 23

WORD COUNTS : 978

Three evaluation designs are compared, to evaluate PATH Program, Design I (Different in Different design) is comparing
change in outcomes for participants to change in outcomes for comparison group and control statistically for a host of
household and individual characteristic. Design II,( Regression Discontinuity Design) comparing outcome for PATH
participant to outcome for comparison group and control statically for a host of household and individual characteristics
including the eligibility score and Design III (Randomized Control Trial) randomly select from group of applicant whose
scored just above the threshold and who will not be offered to participate in the program. The comparison of each
strength and benefit of the three design are as follow :
DESIGN I (D in D ) DESIGN II (RDD) DESIGN III (RCT)


The scientific quality of the design

Strengths

. The method has comparing PATH
participants with group of
household who are poor and have
similar observable characteristic.
2. Evaluation does not require large
samples, because only measuring
the change in outcome for
comparison group and target
group.
3. Can ignore unobservable
characteristics that do not change
over time

Strengths

. Can create valid treatment and
control groups. Treatment group
in this case applicant below the
eligibility threshold and the control
group were group above the
eligibility threshold.
Strengths

. Evaluation use large amount of
sample, which can produce better
analysis
2. Randomized method will reduce
the bias from the researcher and
guarantee validity.

Weaknesses
. Evaluation assumes that the
difference in trends due to the
treatment alone, without consider
Weaknesses
. Sample requirements may not be
acceptable
2. Requires a large sample around
Weaknesses
. Requires a large number of sample
from participant target and
comparison group. Large sample
other factors.
2. Evaluation assumes that the trend
in a chosen comparison group
represents the trend in a "true
counterfactual

the cutoff.
3. The impact could be "local, i.e.,
the estimate cannot be generalized
to units whose index values lie
further away from the cutoff
mean extra work, extra time and
extra effort.




The Political feasibility of implementing the design


Strengths
. The sample representing
population of policy interest. /
program (PATH)

Strengths
. Explain the different in cut off area
clearly.
Strengths
. Easy to explain, few assumptions
or economic wizardry
2. Understandable for many parties


Weaknesses
. Unobservable characteristic in
target group sample may not
similar with the control group
sample.

Weaknesses
. May need large sample to prove
evaluation result.

Weaknesses:
. Survey taking time
2. Require careful preparation
3. Require alot of work to do
The logistical implications of the design ( Finding from evaluation are available in timely manner for
policymaker)


Strengths
. Evaluation result enable policy
maker to take action promptly

Strengths
. Evaluation inform policy maker the
different of those who received and
another group that did not receive
treatment


Strengths
. Valid comparison between those
who receive the program and those
who did not receive program.

Weaknesses
. Difficult to justify objectively that
program contribute to community
progress


Financial implications of the design, in particular if it involve more resources that those already
budgeted

Strengths
. Evaluation method, may require
less financial resources, because
using less sample.

- Strengths
. n this case PATH, financial will be
covered by world bank.
- Weaknesses
. Requires more financial resource
to gather the sample more the
cutoff area.
Weaknesses
. Requires huge financial resources
to gather data sample, from target
group and control group.







MEMO
TO : Ministry of Labor and Social Security.
FROM : Director of the Social and Manpower Planning Division
Date : March , 2003
RE : Comparison of Evaluation Design for PATH Program.

After carefully examine three designs given by the consultant firm and make short
assessment of each evaluation design. recommended to Ministry of Labor and
Social Security to use design no 3,in evaluating the effectiveness and efficiency of
social safety net program, through conditional cash transfer program.
Design number 3 has several advantages because of the availability of baseline
data of program participant. t was determined randomly in summer 2002,just
before the program start in non pilot area, in my opinion, the design has ability to
measure condition before intervention and after intervention and also make a valid
comparison between those who receive the program and those who did not receive
program. Since the sample were selected randomly this design also reduce the
bias.
believe evaluation design number 3 able to represent some population policy
interest to comparison group with, because it utilize large sample, 200 sample from
target and 200 sample from control group. Certainly this design has potential to
produce evaluation result which represent both group and meet the scientific
requirement.
This design also using few assumption and easy to understand by many parties.
More over this design able to justify the evaluation result because it give valid
comparison between group with treatment and with-out treatment.
The evaluation design has meet four important factors.
a. The scientific quality of the design, by employ large amount of sample which
selected randomly. This will reduce the bias and guarantee external validity.
b. The Political feasibility of implementing the design, it easy to explain, because
use few assumptions and understandable for many parties.
c. From the logistical implications of the design, this design give valid
comparison between group with treatment and with out the treatment.
d. n term of financial, although the design requires huge financial resource to
gather large sample, but its result able to describe the link of social
assistance with human capital accumulation.
Therefore as a Director of the Social and Manpower Planning Division, recommend
to the Minister of Labour and Social Security select design number 3 to evaluate
PATH Program.
-
Your Sincerely
Director of the Social and Manpower Planning Division