Practical Meta-Analysis

These lectures are based on the book

Practical Meta-Analysis
by David B. Wilson & Mark W. Lipsey

Outline of Meta-Analysis
Topics covered will include:
What is meta-analysis ?
Meta-analysis issues:

Problem definition and basic concepts

Finding eligible studies, criterion for inclusion
Coding issues, key variables, etc.
Effect sizes (ES) and computational issues
Combining effect sizes using meta-analysis
Publication bias
Interpretation of results
Evaluating the quality of a meta-analysis

Outline of Meta-Analysis
Topics covered will include:

Effect Sizes (ES) these are combined using

meta-analysis
Conducting a meta-analysis how do we
combine effect sizes from multiple studies to
obtain an aggregate effect size.

What is Meta-analysis?
Meta-analysis focuses on the aggregation
the findings of from multiple research
studies.
These studies must produce quantitative
findings from empirical research.
In order combine the study results we
need a standard measure of effect size (ES)
that can be calculated for each study and
then combined.

What is Meta-analysis?
Meta-analysis essentially takes a weighted
average of the effect sizes from each of the
studies.
The weights take into account sample sizes
and variability of the results from each study.
We also may need to account for random
differences between studies due to variations
in procedures, settings, populations sampled,
etc.

Example: Effectiveness of Correctional

Boot-Camps for Prisoners
32 unique studies were found, that reported on
43 independent boot-camp/comparison samples
looking recidivism as the response.
For each study, the odds ratio for recidivism was
calculated looking at the benefit of the
prisoner boot-camps.
Random differences between the differences in
the study designs, prisoner populations,
protocols, etc. were accounted for.

Forest Plot from a Meta-Analysis oft

Correctional Boot-Camps (Wilson et al.)
F
avors C
om
parison
H
arer &K
lein-S
affran, 1996
Jones &R
oss, 1997
F
l. D
ept. of JJ (S
tuart C
o.), 1997
F
l. D
ept. of JJ (P
olk C
o., B
oys), 1997
Jones (F
Y
97), 1998
Jones (F
Y
94-95), 1998
M
ackenzie&S
ouryal (Illinois), 1994
M
ackenzie&S
ouryal (Louisiana), 1994
Jones (F
Y
91-93), 1998
M
ackenzie&S
ouryal (F
lorida), 1994
Jones (F
Y
96), 1998
M
arcus-M
endoza(M
en), 1995
M
ackenzie, et al. 1997
P
enn. D
ept. of C
orrections, 2001
F
low
ers, C
arr, &R
uback 1991
B
ureauof D
ataandR
esearch, 1996
M
ackenzie&S
ouryal (O
klahom
a), 1994
T
3A
ssociates, 2000
M
ackenzie&S
ouryal (N
ewY
ork), 1994
P
eters, 1996b
C
am
p&S
andhu, 1995
M
ackenzie&S
ouryal (S
.C
., N
ew
), 1994
Jones, 1996
N
YD
C
S(88-96R
eleases), 2000
M
arcus-M
endoza(W
om
en), 1995
A
ustin, Jones, &B
olyard, 1993
B
urns &V
ito, 1995
P
eters, 1996a
F
l. D
ept. of JJ (B
ay C
o.), 1997
N
YD
C
S(96-97R
eleases), 2000
N
YD
C
S(97-98R
eleases), 2000
F
l. D
ept. of JJ (P
inellas C
o.), 1996
F
l. D
ept. of JJ (M
anateeC
o.), 1996
C
AD
ept. of theY
outhA
uthority, 1997
B
oyles, B
okenkam
p, &M
adura, 1996
M
ackenzie&S
ouryal (S
.C
., O
ld), 1994
F
l. D
ept. of JJ (P
olk C
o., G
irls), 1997
Jones, 1997
T
hom
as &P
eters, 1996
W
right &M
ays, 1998
M
ackenzie&S
ouryal (G
eorgia), 1994
O
verall M
eanO
dds-R
atio

F
avors B
ootcam
p

The Great Debate & Historical Background

1952: Hans J. Eysenck concluded that there

were no favorable effects of psychotherapy,
starting a raging debate.
20 years of evaluation research and hundreds
of studies failed to resolve the debate
1978: To prove Eysenck wrong, Gene V. Glass
statistically aggregated the findings of 375
psychotherapy outcome studies.
Glass (and colleague Smith) concluded that
psychotherapy did indeed work
Glass called his method meta-analysis

The Emergence of Meta-Analysis

The ideas behind meta-analysis predate

Glass work by several decades
Karl Pearson (1904)

averaged correlations for studies of the effectiveness of

inoculation for typhoid fever

R. A. Fisher (1944)

When a number of quite independent tests of significance

have been made, it sometimes happens that although few or
none can be claimed individually as significant, yet the
aggregate gives an impression that the probabilities are on the
whole lower than would often have been obtained by chance.
Source of the idea of cumulating probability values

The Emergence of Meta-analysis

Ideas behind meta-analysis predate Glass

work by several decades.
W. G. Cochran (1953)

Discusses a method of averaging means across

independent studies
Laid-out much of the statistical foundation that
modern meta-analysis is built upon (e.g., Inverse
variance weighting and homogeneity testing)

The Logic of Meta-analysis

Traditional methods of review focus on statistical

significance testing
Significance testing is not well suited to this task
Highly dependent on sample size
Null finding does not carry the same weight as a
significant finding

significant effect is a strong conclusion

non-significant effect is a weak conclusion

Meta-analysis focuses on the direction and magnitude

of the effects across studies, not statistical significance

Isnt this what we are interested in anyway?

Direction and magnitude are represented by the effect size

When Can You Do Meta-Analysis?

Meta-analysis is applicable to collections of

research that

Are empirical, rather than theoretical

Produce quantitative results, rather than
qualitative findings
Examine the same constructs and relationships
Have findings that can be configured in a
comparable statistical form (e.g., as effect sizes,
correlation coefficients, odds-ratios, proportions)
Are comparable given the question at hand

Forms of Research Findings

Suitable to Meta-analysis

Central tendency research

Prevalence rates

Pre-post contrasts
Growth rates

Group contrasts
Experimentally created groups

Comparison of outcomes between treatment and

comparison groups

Naturally occurring groups

Comparison of spatial abilities between boys and girls

Rates of morbidity among high and low risk groups

Forms of Research Findings

Suitable to Meta-analysis

Association between variables

Measurement research

E.g. Validity generalization

Individual differences research

E.g. Correlation between personality constructs

Effect Size: The Key to Meta-Analysis

The effect size (ES) makes meta-analysis

possible

It is the dependent variable in a metaanalysis

It standardizes findings across studies such
that they can be directly compared and
aggregated.

Effect Size: The Key to Meta-Analysis

Any standardized index can be an effect

size (e.g., standardized mean difference,
correlation coefficient, odds-ratio) as long as
it meets the following:

It is comparable across studies (generally

requires standardization)
Represents the magnitude and direction of the
relationship of interest
It is independent of sample size

Different meta-analyses may use different

effect size indices.

The Replication Continuum

Pure
Replications

Conceptual
Replications

You must be able to argue that the collection of studies you are
meta-analyzing examine the same relationship. This may be at
a broad level of abstraction, such as the relationship between a
risk factor and disease incidence or between a medical therapy
and outcome of interest. Alternatively it may be at a narrow
level of abstraction and represent pure replications.

The closer to pure replications your collection of studies, the

easier it is to argue comparability.

Which Studies to Include?

It is critical to have an explicit inclusion and

exclusion criteria. The broader the research
domain, the more detailed they tend to become.

May need to refine criteria as you interact with the

literature.
Components of a detailed criteria for inclusion:

distinguishing features
research respondents
key variables
research methods
cultural and linguistic range
time frame
publication types

Methodological Quality Dilemma

Include or exclude low quality studies?

The findings of all studies are potentially in error

(methodological quality is a continuum, not a
dichotomy, i.e. bad vs. good)
Being too restrictive may restrict ability to generalize
Being too inclusive may weaken the confidence that
can be placed in the findings.
Methodological quality is often in the eye-of-thebeholder.
You must strike a balance that is appropriate to your
research question.

Searching Far and Wide

The we only included published studies

because they have been peer-reviewed
argument.
Significant findings are more likely to be
published than non-significant findings.
This is a major source of bias!
Critical to try to identify and retrieve all
studies that meet your eligibility criteria

Searching Far and Wide

Potential sources for identification of documents:

Computerized bibliographic databases

(WSU Databases PubMed, Proquest, OVID, CINAHL, etc.)

Google internet search engine

Authors working in the research domain (email a
relevant Listserv?)
Conference programs and proceedings
Dissertations
Review articles
Hand searching relevant journals
Government reports, bibliographies, clearinghouses

A Note About Computerized

Bibliographies

Rapidly changing area

Get to know your local librarian!
Searching one or two databases is
generally inadequate.
Use wild cards (e.g., random? will find
random, randomization, and randomize)
Throw a wide net; filter down with a
manual reading of the abstracts

Strengths of Meta-Analysis

Imposes a discipline on the process of summing

up research findings
Represents findings in a more differentiated and
sophisticated manner than conventional reviews
Capable of finding relationships across studies
that are obscured in other approaches
Protects against over-interpreting differences
across studies
Can handle a large numbers of studies (this
would overwhelm traditional approaches to
review)

Weaknesses of Meta-Analysis

Requires a good deal of effort

Mechanical aspects dont lend themselves
to capturing more qualitative distinctions
between studies
Apples and oranges criticism
Most meta-analyses include blemished
studies to one degree or another (e.g., a
randomized design with attrition)

Weaknesses of Meta-Analysis

Selection bias poses a continual threat

Negative and null finding studies that
you were unable to find.

Outcomes for which there were

negative or null findings that were not
reported.