Treatment evaluation

© All Rights Reserved

Просмотров: 0

Treatment evaluation

© All Rights Reserved

- 6 Causal Inference Technical
- Manual on Sectoral and Barangay Planning, Programming, and Budgeting System
- Lewis on Causation
- A Computational Foundation for the Study of Cognition
- r481
- Shields 1998
- 5-regression.pdf
- Jerry Stannard-The Role of Categories in Historical Explanation
- Cause and Effect Graphing
- Analysis of Relationship Between Learning Achievement of Tafsir Al Quran and Arabic Learning Interests
- The Impact of Earnings Extremity(1)
- jeas_0118_6679.pdf
- weinstein
- a
- Hammer.explication
- QMM5100 Multiple Regression HW
- Simulating the Concentration of Some Heavy Metals in Mista Ali River Mining Pond in Jos Nigeria
- PANORAMICA INDIGENAS
- selcuk.pdf
- Wahlberg - 2014 - Elder-Vass on the Causal Power of Social Structure

Вы находитесь на странице: 1из 29

Program Evaluation

Katja Kaufmann

Winter 2012/13

Bocconi University

Organization of the Course

What is expected of you?

1. Read theory and applied papers with (*) in syllabus,

2. Present papers of your choice in class (with +)

20 min presentation (question and contribution of the paper,

methodology, results, conclusion and critical assessment)

3. Problem sets

AND more importantly, course should help you for your future

research in terms of analyzing papers of others and in applying the

learned methods to address research questions of interest

Outline

Section 1: Introduction

1. Program Evaluation: Motivation and Questions that can be

addressed

2. Challenge: The Problem of Causal Inference

3. Counterfactual Framework

4. Different Approaches

Motivation: Why learn “Program Evaluation”

Approaches?

Learn to evaluate the impact of “programs/policies”

Helps you to address interesting and policy-relevant questions from

many different areas

Examples:

• Education: Effect of an additional year of schooling on earnings (“Returns to

education”); Effect of school inputs or class size on test scores; Effect of

watching TV on test scores, Effect of fellowships on college enrollment

• Labor: Effect of unemployment insurance on duration of unemployment,

Effect of minimum wages or job training on employment

• Development: Effect of antipoverty programs such as conditional cash

transfer programs on children’s education and health, effect of microfinance

programs on level and volatility of income

• Health: Effect of smoking/drinking on health; Effect of an advertising

campaign to cut smoking, effect of increasing minimum drinking age on

traffic deaths

Challenge: The Problem of Causal Inference

job training program on earnings?” requires answering

counterfactual questions:

1. How would individuals who participated in a program have fared in

the absence of the program?

2. How would those who were not exposed to the program have fared in

the presence of the program?

• Problem: never observe counterfactual outcomes, as we can not

observe a person in two different states of the world.

introduced later on today, right now giving intuition)

Problem of Causal Inference: Counterfactual

Outcomes are unobserved

1. Outcome of people who are in program with people who are

not in the program, e.g. compare earnings of training

participants to those of non-participants?

Omitted variables (eg people who are more motivated to work

might decide to join the program, but because of the motivation

they would have higher earnings anyways even without program)

Self-selection into treatment (e.g. job training, of those who

expect to benefit most from such a program)

Change of other factors that affect outcomes, for example,

macroeconomic shocks such as business cycles, seasonal

differences, natural process like aging

Goal of the literature on Program Evaluation:

How to find a good comparison group to make up for not knowing

counterfactual outcomes

• Many different approaches that rely on different assumptions and that have

different data requirements (e.g. single cross-section, repeated cross-

section, panel data)

• depends on the context what is a good approach and what approach is

“available”.

Understand the identifying assumptions needed to justify the application

of different estimators (and learn to apply the methods).

Discuss for each approach and do a comprehensive comparison at the end

of the course.

Program Evaluation and Regression

Framework

Example: OLS Regression

usually interested in estimating causal (i.e. “ceteris paribus”) effects

(causation versus correlation)

dE(Y|X)/dX=b average causal effect

1) Internal validity: Does an approach provide credible estimates of these

effects for the population and setting under study?

2) External validity: When does an approach provide credible estimates of

these effects that can be generalized from the given population and

setting to other populations and settings (e.g. legal, policy and

physical environments).

Internal validity

effects are valid for the population being studied

• Five threats to internal validity

1. Omitted variable bias

2. Sample selection bias

3. Simultaneous causality bias (or reversed causality)

4. Wrong functional form

5. Measurement error (“Errors-in-variables” bias)

assumption

1. Omitted variable bias

• Coefficients of interest:

An omitted variable bias arises if the omitted variable is both (i) a

determinant of y and (ii) correlated with at least one included regressor

Evaluate the conditions for an upward or downward bias from the equation

model that is correlated with the omitted variable

Coefficient on that other variable can not be interpreted as “causal”

(example: ability bias)

• include variable if measurable (or proxy)

• use panel data (solves problem when there are time-constant individual effects)

• use instrumental variable regression

• run a randomized experiment

2. Sample selection bias

the population, BUT sometimes sample “selects itself”

• Sample selection bias arises when a selection process

– Influences the availability of data and

– That process is related to the dependent variable.

Induces correlation between regressor and error term (again violates

conditional mean independence)

2. Sample selection bias

and years of education

– Problem: factors that determine whether someone works are quite

similar to the factors that determine how much that person earns when

employed the fact that someone has a job suggests that person has

a high U and furthermore this error term could be correlated with the

included regressors (educ)

• Potential solutions to sample selection bias

– Randomized controlled experiment

– Construct a model of the sample selection problem and estimate that

model (Heckman’s sample selection model (1979) which was cited in

his 2000 Nobel)

3. Simultaneous causality bias (or reversed

causality)

health causes low wages?

• More general simultaneous equations: Price and quantity are

determined jointly from demand and supply equation

3. Simultaneous causality bias (or reversed

causality)

• Economic theory provides us with not one, but two causal equations:

OLS of y on x will not yield consistent estimates of

Intuition: a high value of u leads to a high value of y, which in turn leads

to a high value of x x and u will be correlated

Formally:

even if E(uv) =0, E(yu) is generally not equal to zero (see first equation)

• Solutions to simultaneous causality bias:

– Randomized controlled experiment: because x is chosen at random by

the experimenter, there is no feedback from the outcome variable to y

(assuming perfect compliance)

– Use instrumental variables regression to estimate the causal effect of

interest (use the variation in x that is exogenous, e.g. supply shifters

such as bad weather which change supply but do not affect demand)

– Develop and estimate a complete model of both directions of causality

(idea behind large macro models, very difficult in practice).

Framework of potential outcomes

(Rubin’s causal model)

– Potential outcome without treatment

– Potential outcome with treatment

Treatment Effect:

for each individual, but only one of the two outcomes is observed

• Observed outcome:

Framework of potential outcomes

(Rubin’s causal model)

• If individual is treated:

– is observed,

– is a counterfactual

– is observed,

– is a counterfactual

Parameters of interest

• Average treatment effect (ATE):

Parameters of interest

Other parameters of interest

• Proportion of people benefiting from the program

relevant for policies as these are the people who will benefit

people at different quantiles of the income distribution)

Model for outcomes with and without

treatment

• Model:

• Observed outcome:

When are ATE and TTE the same?

• ATE:

• TTE:

• Parameters are the same if

– (A1) U1=U0

homogenous effect (conditional on X): g(X)

Y=a+g(X)*D+U0

Note: usually assumed in regression framework, problems of

internal validity as discussed before (e.g. omitted variable bias)

When are ATE and TTE the same?

• ATE:

• TTE:

• Parameters are the same if

– (A2)

Effects can be heterogeneous, but choice of treatment D is

independent of U1-U0 (i.e. ex-post heterogeneity, but not acted

on ex-ante)

This is for example the case in a randomized experiment with

full compliance: E(Y1-Y0|D=1)=E(Y1-Y0), as (U1-U0) indep of D

Goal of the literature on Program Evaluation

How to find a “good” comparison group to make up for not knowing

counterfactual outcomes

Illustration:

- Identification problem:

we observe E(Y0|D=0), E(Y1|D=1), but not the counterfact. E(Y0|D=1), E(Y1|D=0)

TTE = E(Y1|D=1) - E(Y0|D=1) (Problem: second term is unobserved)

E(Y0|D=1)=E(Y0|D=0)=E(Y0),

i.e. no selectivity based on outcome in untreated state

(violated for example, if those who face negative income shocks enter job training

programs. Works for randomized experiment)

Substitute unobserved second term with observed E(Y0|D=0)

Selection problems

The “naïve” estimator (difference in observed means) is then a

biased estimator for TTE

E(Y1|D=1) - E(Y0|D=0)

= [E(Y1|D=1) - E(Y0|D=1)] + [E(Y0|D=1) - E(Y0|D=0)]

= TTE + Bias

Bias is the difference between (average) counterfactual Y0 in both

populations (treated and untreated)

Why is this bias likely?

Selection Rule: D=1 if Y1 - Y0 > C

Then in general:

E(Y0 | D=1)=E(Y0 | Y0 < Y1-C) those who chose treatment

not equal to

E(Y0 | D=0)=E(Y0 | Y0 > Y1-C) those who chose not to be treated

- Comparative advantages in terms of Y1 - Y0

- Simple example: participants have smaller Y0, thus larger potential gain

(think of job training example, “Ashenfelter” dip)

- Heterogeneity in costs

2) Administrative rule

- “Cream-skimming”: they choose “the best”, i.e.

E(Y0 | D=1) > E(Y0 | D=0) overestimate treatment effect

- They put the weakest kids in smaller classes, i.e.

E(Y0 | D=1) < E(Y0 | D=0) underestimate treatment effect (e.g. of

class size)

Different approaches of Program Evaluation

1. Selection on observables (unconfoundedness

assumption): observe all X that affect participation

decision and outcome

• Matching

• Diff-in-Diff: very specific form of selection on unobservables is

allowed, that is based on fixed (i.e. time-constant) effects

• Regression discontinuity: no selection on unobservables around

the discontinuity

2. Selection on unobservables

• Control function approach

• Instrumental variable estimation: find variable that is correlated

with treatment participation decision, but does not affect outcome

(“exclusion restriction”, as excluded from outcome eqn)

- 6 Causal Inference TechnicalЗагружено:Swati Sucharita Das
- Manual on Sectoral and Barangay Planning, Programming, and Budgeting SystemЗагружено:Carl
- Lewis on CausationЗагружено:Scott Walsh
- A Computational Foundation for the Study of CognitionЗагружено:gusorh
- r481Загружено:Regino Barranquero Cardeñosa
- Shields 1998Загружено:Muhammad Arfan
- 5-regression.pdfЗагружено:hawk91
- Jerry Stannard-The Role of Categories in Historical ExplanationЗагружено:jf_90
- Cause and Effect GraphingЗагружено:Anonymous KUw5kEG2KT
- Analysis of Relationship Between Learning Achievement of Tafsir Al Quran and Arabic Learning InterestsЗагружено:IJSTR Research Publication
- The Impact of Earnings Extremity(1)Загружено:Recca Damayanti
- jeas_0118_6679.pdfЗагружено:Javier
- weinsteinЗагружено:api-363619955
- aЗагружено:Nurfaizi
- Hammer.explicationЗагружено:ianverstegen
- QMM5100 Multiple Regression HWЗагружено:manjinderchabba
- Simulating the Concentration of Some Heavy Metals in Mista Ali River Mining Pond in Jos NigeriaЗагружено:IJSTR Research Publication
- PANORAMICA INDIGENASЗагружено:Joel Rojas
- selcuk.pdfЗагружено:Anonymous 6tnaPFbe
- Wahlberg - 2014 - Elder-Vass on the Causal Power of Social StructureЗагружено:xxxxdadad
- Current and Probable Past Wildlife Fatality Hotspots on the Thousand Islands Parkway, Ontario, CanadaЗагружено:GRUNDOPUNK
- June 2014 (IAL) MA - S1 EdexcelЗагружено:AmeenIbrahim
- darrow-cssh-2001-43-04Загружено:Nicolau
- Causality & Explanation in the SciencesЗагружено:limuvi
- Statistics Revision 2 SolutionsЗагружено:Nicholas Teh
- Regression TestingЗагружено:Rahul Kumar
- CAST HandbookЗагружено:Vijay Bantu
- Cph 14Загружено:Sulistyono
- Reading ExamЗагружено:Jessa Mae A. Tribunalo

- Carsten Sudhoff FinalЗагружено:Moonarak
- Cass 10 Year PlanЗагружено:smcaagency
- Assignment 3Загружено:arafat
- 13 1 reviewЗагружено:api-251319721
- Tool Swot Pest AnalysisЗагружено:sterling
- The Arab Spring of LiesЗагружено:Julian
- 1 - Introduction to EconomicsЗагружено:Ishraq Ifaz
- Do Minimum Wage Hikes Raise US Long Term Unemployment Evidence Using State Minimum Wage RatesЗагружено:stampit
- GGItЗагружено:Jerry B Cruz
- ec521Загружено:Sudha Sivadas
- Impact of DownsizingЗагружено:Sulaman Sadiq
- Chapter 21_conflicts Between Macroeconomic GoalsЗагружено:ohyeajcrocks
- 0. EU438 NotesЗагружено:Sinemis Temel
- Youth and Inequality: Time to support youth as agents of their own futureЗагружено:Oxfam
- Morris County Data Book - Rev 2009 JanuaryЗагружено:Morris County NJ
- Essays Micro and Macro Up to May 2015Загружено:Suresh Madhavan
- notes7Загружено:Amit Jha
- 100 Q&A of GCE O'Level Economics 2281Загружено:Hassan Asghar
- Indian Villages Weakness StrengthЗагружено:Biswajit Behera
- 17140 Young Greeks and the Debt CrisisЗагружено:Patrícia Rodrigues
- Department of Labor: PR-94Загружено:Department of Labor
- Youth problemЗагружено:Tranceformat
- Ch 03 SampleЗагружено:Karen Yung
- Regulation 883 2004 ECЗагружено:LeaLeacska
- Accomplishments of the Last Two RegimesЗагружено:Eleni Vamvakari
- Indian EconomyЗагружено:sparkle_suru
- Business CycleЗагружено:Hlengiwe Phewa
- (517768512) 100 Q&A of Cambridge O'Level Economics by ALIMЗагружено:AbhaySingh
- Arnett 2006 Emerging Adulthood in EuropeЗагружено:Yulka Valiova
- UCSC Midterm 2Загружено:ايهاب غزالة