This article was downloaded by: [UZH Hauptbibliothek / Zentralbibliothek Zürich]

On: 10 September 2013, At: 13:24

Publisher: Routledge
Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House,
37-41 Mortimer Street, London W1T 3JH, UK

Evidence-Based Communication Assessment and

Intervention
Publication details, including instructions for authors and subscription information:
http://www.tandfonline.com/loi/tebc20

A review of meta-analyses of single-subject

experimental designs: Methodological issues and
practice
a a
S. Natasha Beretvas & Hyewon Chung
a
Department of Educational Psychology, University of Texas at Austin, USA
Published online: 15 Nov 2008.

To cite this article: S. Natasha Beretvas & Hyewon Chung (2008) A review of meta-analyses of single-subject experimental
designs: Methodological issues and practice, Evidence-Based Communication Assessment and Intervention, 2:3, 129-141, DOI:
10.1080/17489530802446302

To link to this article: http://dx.doi.org/10.1080/17489530802446302

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) contained
in the publications on our platform. However, Taylor & Francis, our agents, and our licensors make no
representations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of the
Content. Any opinions and views expressed in this publication are the opinions and views of the authors, and
are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon and
should be independently verified with primary sources of information. Taylor and Francis shall not be liable for
any losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoever
or howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use of
the Content.
This article may be used for research, teaching, and private study purposes. Any substantial or systematic
reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any
form to anyone is expressly forbidden. Terms & Conditions of access and use can be found at http://
www.tandfonline.com/page/terms-and-conditions
 129 META-ANALYSES OF SINGLE-SUBJECT EXPERIMENTAL DESIGNS

A review of meta-analyses of single-subject experimental

designs: Methodological issues and practice
S. Natasha Beretvas & Hyewon Chung, Department of Educational Psychology, University of Texas at Austin, USA
...............................................................................................................................................

Abstract
Several metrics have been suggested for summarizing results from single-subject experimental designs. This study briefly
reviews the most commonly used metrics, noting their methodological limitations. This study also includes a synthesis of
recent meta-analyses, describing which metrics were used and how meta-analysts handled dependence in the form of
multiple treatments, outcomes, and participants per study. Guidelines for future methodological research and for single-
subject experimental design meta-analysts are provided.

Keywords: Meta–analysis, effect sizes, single–subject experimental–designs, methodology

Downloaded by [UZH Hauptbibliothek / Zentralbibliothek Zürich] at 13:24 10 September 2013

INTRODUCTION discerned from SSED results. And for the results

Meta-analytic procedures afford researchers
a method to quantitatively synthesize past research
results, thereby providing evidence to support best
practice (Glass, 1976; Hedges & Olkin, 1985; Hunter
& Schmidt, 1990). While these meta-analytic meth-
ods work well for synthesizing the results of studies
with large sample sizes (large-n), there is still no
consensus concerning how best to summarize results
from single-subject experimental design (SSED)
studies. This is a problem because a considerable
amount of educational and psychological research
has made use of SSEDs (Galassi & Gersh, 1993).
Indeed, SSEDs are frequently employed in educa-
tional research designed to assess a treatment’s
effect on special populations, such as individuals
with autism and related developmental disabilities.
Typically in pre–post large-n intervention studies,
the focus is on change in outcome level between pre-
test and post-test. While this is part of the interest
with SSED studies, trends within and across baseline
and treatment phases are simultaneously considered
and are perhaps the most important aspect of the
data to consider when evaluating the results from
such studies. Numerical descriptors of these trends
are difficult to estimate when the data describe
a treatment’s effect on an individual (Crosbie, 1993)
and when the number of repeated measures is as
small as is commonly found in educational SSED
research (Busk & Marascuilo, 1988; Huitema, 1985).
For this reason, visual analysis of graphed data is
typically employed in SSED studies to assess
a treatment’s effect.
A perusal of a plot of results should clearly
identify whether a treatment effect can be
For correspondence: S. Natasha Beretvas. E-mail: tasha.beretvas@
utexas.edu
Source of funding: Preparation of this article was supported by a grant from
the Institute of Education Sciences, U.S. Department of Education. However,
the opinions expressed do not express the opinions of this agency.
of a visual analysis of data to support a treatment’s
effect, usually the magnitude of the effect must be
sufficiently large that associated practical and
clinical significance can be assumed. However,
visual inspection of results involves the considera-
tion of several dimensions along which data might
vary, including mean shift (change in average
behavior in baseline versus treatment), slope
change (change in trend between baseline and
treatment), and variability (of data points around
the general trend). Parsonson and Baer (1992)
provide a detailed summary of research conducted
to investigate the relationship between character-
istics of graphs and the inferences resulting from
visual interpretations of the results. Unfortunately,
some of this research has indicated that inferences
based on visual analysis are not very reliable.
Statistical summaries of results provide an alter-
native to visual inspection.
There are several justifications for using statistical
methods to summarize results from SSEDs. Beyond
the parsimony associated with use of quantitative
meta-analysis, the current climate of evidence-based
practice also heralds a renewed focus on methods
used to meta-analyze SSEDs results. A statistical
summary of results also allows a potentially more
objective summary of studies’ results through the
use of meta-analytic procedures. Use of meta-
analysis encourages generalization of findings
across individuals. It also permits exploration of
any differences identified in study results.
In addition, statistical description of study results
allows for the identification of a treatment effect,
despite potentially unstable baseline data, and of
small treatment effects that might not be visible
graphically (Nourbackhsh & Ottenbacher, 1994).
Lastly, the potential lack of reliability noted for
interpretation of data using visual analysis could
be reduced with the use of statistical summaries.

ISSN 1748–9539 print/ISSN 1748–9547 online ß 2008 Informa Healthcare USA, Inc.
DOI: 10.1080/17489530802446302
 130 META-ANALYSES OF SINGLE-SUBJECT EXPERIMENTAL DESIGNS
The interpretation of results associated with an
appropriately conducted quantitative descriptor will
be more consistent.
Several quantitative descriptors have been derived
as options to describe a treatment’s effect in SSEDs.
These descriptors include single-indicator summaries
such as various versions of a standardized difference
in phase (treatment and baseline) means (Busk &
Serlin, 1992) and the percentage of non-overlapping
data (PND; Scruggs, Mastropieri, & Casto, 1987).
Others have suggested using single R2-change
indicators that simultaneously describe change in
Downloaded by [UZH Hauptbibliothek / Zentralbibliothek Zürich] at 13:24 10 September 2013
the outcome’s level and slope as a result of treatment
(Center, Skiba, & Casey, 1985–1986; Faith, Allison, &
Gorman, 1996). Other researchers have suggested
a pair of R2-change indicators to describe a treatment
effect with one indicator describing change in the
level of an outcome between baseline and treatment
(e.g. Crosbie, 1995; Tryon, 1982), with the other
indicator describing change in trend as a result of the
treatment being introduced (e.g. Beretvas & Chung,
2007; Crosbie, 1995). Finally, some researchers have
proposed a multi-level approach to the meta-analysis
of SSEDs (van den Noortgate & Onghena, in press).
Despite the need to quantitatively synthesize results
from SSEDs, there is little consensus in the field
about what qualifies as an appropriate descriptor.
Methodological critiques of each of the descriptors
have been performed, and no single descriptor has
yet been established as best. In spite of these
criticisms, researchers continue to use several of
these descriptors. The current paper provides a brief
description of some of the more commonly used
effect sizes metrics, emphasizing some of the
problems associated with each metric. In addition,
a summary of recent applied meta-analyses of single
subject designs will be provided, along with some
directions for future research in this area.
SINGLE INDICATOR EFFECT-SIZE METRICS
Standardized mean difference
The standardized mean difference used in SSEDs is
calculated on the basis of the effect size of the same
name used for large-n studies that compares the
difference in means of two independent groups at
a single time point. The basic formula for the large-n
standardized mean difference is estimated using
_ Y T  Y C
¼ ð1Þ
S (Hershberger, Wallace, Green, & Marquis, 1999)
_
where  is an estimate of the difference between the
mean outcome score of the treatment group, Y T , and
the mean of the control group, Y C . This difference is
standardized by dividing by an estimate of the
population standard deviation, s, of outcome scores
within the populations. The corresponding standard-
ized mean difference that is frequently used as
a metric for SSEDs is
_ Y B  Y A ð2Þ
SMD ¼
S
where the sample mean, Y i , is calculated on the basis
of the mean of the outcome values in phase i (A or B)
for a single individual, whereas the sample mean in
Equation 1 is calculated on the basis of the outcome
scores for a sample of individuals. What further
_ _
distinguishes  from SMD is the calculation of the
standard deviation. In Equation 2, s is the sample
standard deviation for the individual’s outcome
scores (calculated using data just from the baseline
phase or pooled across the A and B phases).
There will be less variability in measures on
a single individual over time, S , than in measures
on multiple individuals at a single time point, s. The
reason for this is that an individual’s scores will
probably be similar to his or her score at an earlier
point in time, whereas if two independent
individuals had been randomly selected, then there
should be no relationship between their scores. The
autocorrelation introduced in SSED time-series
data results in a violation of the assumption of
independence that is made with most large-n
group-comparison designs. Given the fundamental
differences between s and S , the metric estimates in
Equations 1 and 2 should not be considered as
measured on the same metric. The reduced varia-
bility in repeated measures on an individual
_
(S ) will
inflate the scale
_
of the resulting SMD as compared
with that of . Thus, if the effect-size estimate in
Equation 2 is used to describe results from SSEDs,
they should not be combined with effect-size
estimates from large-n studies. In addition, it
should not be assumed
_
that the variance typically
associated with  (see, for example, Cooper &
Hedges, 1995; Hedges & Olkin, 1985) is associated
_
with SMD . Mostly as a result of the potential
autocorrelation resulting from repeated measures _
on an individual, the sampling distribution of SMD
is unclear and, thus, the relevant variance to
associate with the metric’s index is also unclear.
_
The standardized mean difference effect size,
SMD , appears to be quite commonly used in meta-
analyses of SSED research (see, for example, Busk &
Serlin, 1992; Faith et al., 1996). Other researchers
have encouraged calculating an effect-size estimate
similar to that in Equation 2, except that the means
and standard deviations would be based on only the
last three time points in each phase. Use of only the
 131 META-ANALYSES OF SINGLE-SUBJECT EXPERIMENTAL DESIGNS

last three time points per phase might be preferred as 9

8
providing a more valid measure of baseline data once
7
the pattern in the outcome measure has stabilized;
6
however, all of the criticisms mentioned above with

Outcome
_ 5
regard to SMD also apply to effect size. 4
Apart
_
from the
_
caveat about the different metrics 3
of SMD versus , it should be emphasized that use of 2
this metric descriptor would only make sense when 1
no trend was evident in the baseline or treatment 0
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14
phases (see Figure 1 for an example). However, there
might be a trend in the pattern of behavior either Time

within one or within both phases and, thus, the

Downloaded by [UZH Hauptbibliothek / Zentralbibliothek Zürich] at 13:24 10 September 2013

Figure 1. AB design, outcome level shift (increase) with no trend

mean value will not be very representative of within baseline and treatment phases.
a treatment’s effect. This metric will only represent
the difference in the average outcome levels in each
phase. No trend might be evident in baseline but the
8
treatment might change the outcome level and
introduce a trend (for example, see Figure 2). 7

Alternatively, there might already be a slight trend 6

in baseline (reflecting a natural development in the 5

Outcome
outcome over time). The intervention might raise the 4
level and convert the trend to a steeper slope 3
supporting more rapid growth in the outcome (see 2
Figure 3). In any of the scenarios in which there is
1
a trend1, a change in the average level in the
0
outcome of each phase will not capture the treat- 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14
ment’s impact on the trend. It is also possible that Time
there might be a trend in baseline and that the
treatment has absolutely no effect on either trend or Figure 2. AB design, outcome level shift (increase) with no trend in
level (for example, see Figure 4). Summarizing the baseline and trend (increasing behavior) introduced in treatment
results of a study for which the pattern depicted in phase.
_
Figure 4 applies using SMD will lead to (false)
30
detection of a treatment’s effect even though the
change in level solely resulted from natural 25
development.
20
Outcome

Percentage of non-overlapping data 15

Scruggs et al. (1987) introduced the PND as an index 10

of a treatment’s effect. The PND provides a non-
parametric descriptor of the overlap between the 5
data in the treatment versus the baseline phases. To
0
describe results in an AB design for a treatment 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14
designed to increase a behavior, the PND is Time
calculated by tallying the percentage of data points
in the treatment phase that exceed the highest point Figure 3. AB design, outcome level shift (increase) with slight,
reached in the baseline phase. (If a treatment is positive trend in baseline and change (increase) in trend
introduced in treatment phase.
designed to decrease a behavior then the percent of
points in the treatment phase that are lower than the
lowest data point in the baseline phase are tallied). with parametric descriptors of effect size. And
The PND does not require the assumptions of while exact values for data points are needed for
normality, linearity, and homogeneity associated other metrics, this is not the case for PND calcula-
tion. Extensive description of problems and benefits
..................................................................... associated with use of the PND are provided
1
All trends have been depicted, here, as positive. It is of course feasible that
trends could be negative (reflecting a reduction in the behavioral outcome). elsewhere (see, for example, the entire second issue
The same conclusions apply for such scenarios. of Remedial and Special Education, 1987, Volume 8).
 132
9 and X2 are hypothesized to predict Y, then the
8 following regression model could be tested:
7
6
Outcome
5
4
3
2 normality) is that of the independence of the
1 residual terms, the eis. This assumption is approxi-
0 mately met in large-n studies and, thus, the standard
Downloaded by [UZH Hauptbibliothek / Zentralbibliothek Zürich] at 13:24 10 September 2013
0 1 2 3 4 5 6 7 8 9
Time
Figure 4. AB design, positive trend in baseline and treatment
phases and no treatment effect (on level or trend).
Despite the caveats associated with its use, the PND
is one of the most commonly used descriptors of
treatment effectiveness in the research syntheses of
SSEDs in education and special education (see
Schlosser, Lee, & Wendt, in this issue).
Several other non-overlapping metrics have been
developed since the introduction of the PND,
including the percentage of all non-overlapping
data (Parker, Hagan-Burke, & Vannest, 2007), the
percentage of data points exceeding the median (Ma,
2006), the improvement rate difference (Parker &
Hagan-Burke, 2007), the mean baseline reduction
(Lundervold & Bourland 1988), among others. The
benefit of these metrics includes that they are simple
to use and unaffected by potential autocorrelation.
Unfortunately, however, several associated problems
have been noted. One of the problems with these
non-parametric indices has to do with the unknown
sampling distributions associated with each of them.
This seriously compromises the validity of statistical
tests conducted using these indices. Additional
criticisms, including the potential impact of floor or
ceiling effects and orthogonal slope changes have
also been noted. Another common criticism of these
non-parametric descriptors is that their values can be
confounded in the presence of a trend in the data.
Additional procedures have been recommended to
handle assessment of a treatment’s effect using
regression-based procedures that model the possible
changes in level and trend of the outcome. Before
describing the different procedures, one of the
fundamental assumptions of multiple regression
will be reviewed. In addition, its possible violation
in SSEDs will be discussed along with the associated
effect on statistical results.
With ordinary least squares (OLS) regression
analyses for large-n studies, the outcome for
individual i, Yi, is modeled as a function of the
relevant predictors. For example if two predictors, X1
META-ANALYSES OF SINGLE-SUBJECT EXPERIMENTAL DESIGNS
Yi ¼ 0 þ 1 X1i þ 2 X2i þ . . . ei ð3Þ
Estimation of the relevant regression coefficients
(the s) in Equation 3 is unbiased and efficient when
the relevant model assumptions are met. One of the
primary assumptions (beyond homoscedasticity and
errors estimated using OLS can be assumed to be
10 11 12 13 14
accurate. However, when this assumption is violated,
then estimation can no longer be considered efficient
and the standard error estimates cannot be assumed
accurate.
If residuals are (what is termed) positively
autocorrelated2, as might be possible with time-
series data (namely, data measured over time for an
individual), then the standard error estimates will be
biased underestimates (see for example, Pankratz,
1983; Crosbie, 1993; Busk & Marascuilo, 1992;
Gorman & Allison, 1996). Various researchers
(Jones, Weinrott, & Vaught, 1978; Huitema, 1985;
Busk & Marascuilo, 1988, and others) have argued
whether there is autocorrelation in the errors of
SSED data. Regardless, as recommended by Busk
and Marascuilo (1992), ‘‘single-case researchers
should analyze their data not assuming indepen-
dence of observations’’ (p. 165).
Huitema and McKean (1998) demonstrated that
appropriate modeling of the trends in SSED datasets
can greatly reduce the autocorrelation in residuals.
This should be a goal for researchers interested in
describing trends in SSED studies because, if
autocorrelation is not sufficiently well explained
away with predictors, the autoregressive models that
are needed to appropriately model the patterns are
particularly complex. The use of autoregressive
integrated moving average (ARIMA) time-series
models as popularized by Box and Jenkins (1976)
has been advocated for the more formally termed
‘interrupted time series’ (ITS) data that are com-
monly encountered in SSEDs (Glass, Willson,
& Gottman, 1975). However, ARIMA models are
complicated and function well only with a number of
data points per phase that is much higher (i.e. 50 or
more; Box & Jenkins, 1976) than is typically
encountered in educational SSED research
(Hartmann et al., 1980).
Huitema (1985) examined 881 experiments pub-
lished in the Journal of Applied Behavior Analysis
.....................................................................
2
If residuals are negatively autocorrelated, then standard error estimates will
be inflated with overly conservative Type I error rates.
 133 META-ANALYSES OF SINGLE-SUBJECT EXPERIMENTAL DESIGNS
between 1968 and 1977 and found the modal
number of data points per phase to be 3–4, with
a median of 5. Busk and Marascuilo (1988)
summarized this information for articles published
in the same journal from 1975 to 1985 and found
that the vast majority of SSED analyses involved
fewer than 30 data points for the baseline and
intervention phases (85% and 73%, respectively). Not
only are model parameters poorly estimated in the
presence of autocorrelation, but the autocorrelation
(in the residuals) is also poorly estimated with small
data sets.
Downloaded by [UZH Hauptbibliothek / Zentralbibliothek Zürich] at 13:24 10 September 2013
Given the evident problems with estimating
autocorrelation, Huitema and McKean’s (1998)
suggestion to reduce autocorrelation with appropri-
ate model specification holds great merit. Several
regression models have been suggested to explain
patterns exhibited in two-phase data to explore the
existence of potential treatment effects. It should be
remembered that one of the strongest criticisms
applied to the non-regression
_
effect-size estimates
(such as the PND and SMD ) is that they do not
reflect the possible impact of a linear trend. In the
presence of the simplest kind of trend, a treatment
can affect both level and trend and, thus, the
relevant model needs to take this into consideration.
SINGLE METRICS BASED ON CHANGE IN R2
As a first reaction to this same concern, Gorsuch
(1983) had suggested calculating an effect-size
estimate that was a function of the change in R2
introduced by adding time as a predictor of the
outcome, Yt. A similar procedure was suggested by
White, Rusch, Kazdin and Hartmann (1989). White
et al.’s suggestion was to calculate an effect-size
estimate using the following formula:
YB0  YA0
ESY 0 ¼ ð4Þ
SY 0
where Yi0 is the outcome score on the last day of
phase i predicted using the linear regression of Y on
Time using data from phase i, and SY 0 is the pooled
within-phase standard deviation estimate. The stan-
dard deviation is calculated for each phase i using:
qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
 
SY 0 ¼ Si 1  rYt 2 ð5Þ
i calculated based on the F-ratio statistic testing the
where si is the conventional standard-deviation
estimate of scores calculated within phase i (for
each of phase A and B), and rYt represents the
correlation between Y and time. The resulting two
values for SYi0 are pooled together to obtain SY 0 (in
Equation 4). Given the non-independence of the
values on Y, it seems necessary to correct the
standard deviation describing the variability of the
scores in each phase. A form of the correction used
by White et al. (see Equation 5), although using r not
r2, is applied to standard deviations in related-
samples test statistics in large-n meta-analyses (see
Morris & DeShon, 2002, for example) to correct for
correlated data. Yet, given that the relationship
between time and the outcome has already been
modeled in each phase with the regression of Y on t,
use of this correction does not seem appropriate.
Since these ideas were introduced, several authors
have suggested use of a piecewise regression model
to describe ITS data (such as is found in AB designs
and their extensions). The parameterizations of the
piecewise regression model were designed to provide
parameters describing potential changes in level and
slope upon introduction of the treatment. Center
et al. (1985–1986) suggested use of a metric based on
a change in R2 (R2) for two piecewise regression
models. The full piecewise regression model’s
parameterization,
Yt ¼ 0 þ 1 Tt þ 2 Dt þ 3 ðTt  n1 ÞDt þ et ð6Þ
was designed to provide two regression coefficients
(2 and 3) that described the change in the level
of and the trend in the outcome from baseline to
treatment. (Note that variable Tt is used to identify
the time point of the outcome, Yt, at time t, and Dt is
a dummy-coded variable used to identify whether
the outcome was measured in the baseline (D ¼ 0) or
the treatment (D ¼ 1) phase). A restricted piecewise
regression model, which did not include the inter-
action (change in slope) nor the phase (D) (change
in level) predictors,
Yt ¼ 0 þ 1 Tt þ et ð7Þ
would also be estimated. The change in R2 for the
two models (in Equations 6 and 7) is calculated and
converted into an effect size that describes the effect
of an intervention on both the slope and intercept.
Specifically, the effect size, F(2,df), represents
a measure of the change in the proportion of
variance in the outcome explained (i.e. the R2)
with the simultaneous addition of the change in
intercept and the change in trend parameters (from
Equation 6 to Equation 7). The effect size is
R2 of the full and restricted models:
ðR2full  R2rest Þ=ðdfrest  dffull Þ
F¼ ð8Þ
ð1  R2full Þ=ðn  dffull  1Þ
where df represents the degrees of freedom.
In Equation 8, the ‘rest’ subscript refers to the
‘restricted’ regression model in Equation 7. The ‘full’
 134 META-ANALYSES OF SINGLE-SUBJECT EXPERIMENTAL DESIGNS

subscript refers to the ‘full’ regression model in
Equation 6 that includes the change in level and
linear growth coefficients. This means that the
numerator of Equation 8 provides the amount of
variability explained by the addition of the change in
level and change in trend coefficients. The associated
effect size, F(2,df), is a function of this F-ratio and is
designed to correct for the F-ratio being based on
two df (Faith et al. 1996):
sffiffiffiffiffiffiffiffiffiffiffiffi
_ 2F
Fð2, df Þ ¼ 2
dfError
Downloaded by [UZH Hauptbibliothek / Zentralbibliothek Zürich] at 13:24 10 September 2013
Still other metrics have been suggested, although
problems with these have also been identified. (For
example, Blumberg, 1984; Crosbie, 1995; Gorman
& Allison, 1996, all of whom have pointed out
serious problems with the C statistic of Tryon,
1982). Crosbie (1993; 1995) developed a procedure
designed to analyze ITS data such as is encoun-
tered in SSEDs. Crosbie wrote a program that uses
an alternative estimate of the autocorrelation
designed to correct for its bias with small sample
sizes. The program, ITSACORR, provides two
ð9Þ
indices. It includes tests of the change in intercept

between baseline and treatment phases and of the

This resulting F(2,df) essentially describes the mag-
nitude of the change in level and trend introduced
by the treatment.
Faith et al. (1996) suggested a slight modification
to Center et al.’s (1985–1986) suggested metric;
however, there are two problems with what was
suggested. A fundamental problem is that the
piecewise regression used by both Center et al.
(1985–1986) and Faith et al. (1996) is better
parameterized using Huitema and McKean’s (2000)
model. In this model, [Tt  (n1 þ 1)]Dt is used
instead of Tt  n1 (in Equation 6):
Yt ¼ 0 þ 1 Tt þ 2 Dt þ 3 ½Tt  ðn1 þ 1ÞDt þ et :
ð10Þ
In this model, coefficient 0 provides an estimate of
the baseline phase’s intercept (value of the outcome
when T ¼ 0 and D ¼ 0). Coefficient 1 provides an
estimate of the linear trend in the outcome during
the baseline phase (i.e. the change in Y for a change
of 1 in T ). 2 provides an estimate of the difference
in the intercept predicted based on the linear trend
in the intervention data and the intercept predicted
for the first intervention time point (when
T ¼ n1 þ 1) based on the baseline data. The 3
coefficient (of the interaction term) provides an
estimate of the change in the slope for the treatment
data versus the baseline data. The 2 and 3
coefficients thus provide valuable information that
can be used to describe a treatment’s effect on the
level and slope, respectively.
Another fundamental problem with Center et al.’s
(1985–1986) and Faith et al.’s (1996) metric is that
the use of a single effect-size estimate to describe
a treatment’s effect on both level and slope does not
seem optimal. It would seem more appropriate to
calculate a metric for an intervention’s effect on
a slope and a metric to describe the effect on the level
of the outcome. Lastly, if any of the models used
do not sufficiently explain autocorrelation in resi-
duals, then this could negatively impact the resulting
metrics as accurate estimates of effect size.
change in slope between the two phases while
modeling potential autocorrelation. On the basis of
the results of a simulation study, Crosbie found
evidence supporting the usefulness of ITSACORR
for estimating treatment effects for two-phase,
short ITS data sets (1993). More recently, however,
Huitema (2004) described some potential problems
with the setting of parameters for the design
matrix assumed in ITSACORR. The set of problems
evolve whenever there might be a trend during the
baseline phase. The comparison of intercepts under
ITSACORR involves a comparison of the predicted
measure at the first baseline time point with the
predicted value for the first measure in the
treatment phase. When there is a trend during
baseline, then these intercepts would be expected
to be different regardless of whether there is
a treatment effect or not. Another problem
is that the value of the intercept predicted for
the treatment phase will be a function of the
autocorrelation. When no trends exist in the
baseline data, then ITSACORR has been found to
function well.
Several other researchers have also explored the
use of a pair of indices to describe a treatment’s
effect on slope and on level (Beretvas & Chung, 2007;
van den Noortgate & Onghena, 2003). Beretvas and
Chung used Center et al.’s (1985–1986) R2 metric
idea paired with Huitema and McKean’s (2000)
piecewise regression equation (see Equation 10)
formulation to derive their pair of effect sizes.
Specifically, for their effect size describing the
effect of an intervention on the level of an outcome,
the authors calculated the R2 for the full piecewise
regression equation (Equation 10) versus the follow-
ing restricted regression equation:
Yt ¼ 0 þ 1 Tt þ 2 ½Tt  ðn1 þ 1ÞDt þ et ð11Þ
which included no change-in-level coefficient. The
associated F-ratio statistic testing the significance
of this R2 was then calculated using Equation 8.
The effect size describing the treatment’s effect on
 135 META-ANALYSES OF SINGLE-SUBJECT EXPERIMENTAL DESIGNS
the level was calculated by substituting the resulting
F-ratio into the following Equation:
sffiffiffiffiffiffiffiffiffiffiffiffi
_ F
F,Level ¼ 2 ð12Þ
dfError
Note that, because there is only one parameter added
to the full (Equation 10) over the restricted model
(Equation 11), the df associated with this F-ratio is
only 1 (see Equation 12 versus Equation 9).
The metric describing the effect of the intervention
on the slope is calculated in a similar way. The
Downloaded by [UZH Hauptbibliothek / Zentralbibliothek Zürich] at 13:24 10 September 2013
difference is calculated in the R2 for the full
piecewise regression model in Equation 10 versus
the following restricted piecewise regression model:
Yt ¼ 0 þ 1 Tt þ 2 Dt þ et ð13Þ
which assumed no change-in-slope as a result of the
intervention. The F-ratio testing the change in R2 is
calculated using Equation 8 and the metric describ-
ing the effect of the intervention on the slope is
calculated in the same way as in Equation 12:
sffiffiffiffiffiffiffiffiffiffiffiffi
_ F
F,Slope ¼ 2 : ð14Þ
dfError
Beretvas and Chung (2007) recommend testing for
autocorrelation in the residuals (remaining once
the full model was estimated). If significant auto-
correlation is found, the authors recommend using
auto-regression to estimate the relevant regression
equations (Equations 10, 11, and 13). Otherwise, OLS
estimation could be used. The authors conducted
a simulation study to assess the functioning of their
metric and found that it worked well for scenarios in
which the model fully explained the autocorrelation.
The metric worked less well in scenarios possibly
typical of some SSEDs with small data sets, in which
there was residual autocorrelation even when it was
modeled using auto-regression.
Van den Noortgate and Onghena (2003) suggested
calculating two metrics to describe an intervention’s
effect on the level and linear growth in an outcome.
They encouraged estimating the full regression
model in Equation 10 using OLS and suggested
standardizing the change in level and slope coeffi-
cients (i.e. 2 and 3) by dividing each by the square
root of the mean squared error. The authors suggest
synthesizing the resulting standardized effect-size
estimates using multivariate multilevel modeling
(while correcting the covariance matrix for the two
effect sizes by dividing its elements by the mean
squared error. This procedure addresses several of
the concerns associated with some of the other
metrics. However, the procedure was applied only to
a real data set and was not empirically evaluated.
In addition, as with Beretvas and Chung’s (2007)
metrics, potential autocorrelation in residuals could
affect the precision and accuracy of these metrics
when used with small data sets. Thus, when
calculating these metrics, the full piecewise regres-
sion model (in Equation 10) could be estimated by
modeling the potential autocorrelation in residuals.
It is anticipated, however, that with the small data
sets typically encountered in SSED studies, estima-
tion will still be problematic.
There is clearly a lack of consensus about the best
metric to use to summarize SSED results. And, as is
evident from this review of methodological studies’
discussions of these metrics, there are problems
associated with each of them. However, applied
meta-analyses are still being conducted on SSED
results. It is important to see which metrics are most
commonly used by practitioners despite their asso-
ciated methodological caveats. This could help
inform future research into refinements for the
relevant metrics. Given the plethora of metrics that
have been suggested for use in describing SSED
results, a survey of relatively recent SSED meta-
analyses was conducted. A description of the survey
and its results is described below.
SSED-STUDY META-ANALYSES NARRATIVE REVIEW
RESULTS
The PsycINFO, MEDLINE and ERIC databases were
each searched using the keywords ‘meta-analysis’,
‘review’, or ‘synthesis’ paired with ‘single-case’,
‘single-subject’, or ‘PND’ for the years 1985 to
2005. Applied meta-analyses were included that
incorporated computation of some form of quanti-
tative effect size for single-n studies. In addition to
the studies identified in the search, relevant applied
meta-analyses known to the authors were also
surveyed. Only those studies that clearly defined
the type of metric calculated were included in the
review. If an applied meta-analysis was unclear
about any of the other meta-analytic steps, it was
still included.
Metrics used
The database searches led to the identification of 279
studies in PsycINFO, 65 in ERIC, and 405 in
MEDLINE. Redundancy across databases was
removed and the remaining subset of studies was
assessed to identify which of them involved applied
meta-analyses that synthesized results from SSEDs
using clearly described quantitative metrics. The
resulting 21 studies were supplemented with
another 4 applied single-n meta-analyses that met
 136 META-ANALYSES OF SINGLE-SUBJECT EXPERIMENTAL DESIGNS
criteria for inclusion. Thus, a total of 25 meta-
analyses were summarized.
The most popular metric used was the PND index,
and was commonly used along with the percent of
zero data points (PZD). The PND and the PZD were
used in 12 of the 25 meta-analyses. The next most
popular
_
metric, used in 7 of the studies, was
the SMD , in various versions. This metric was
typically obtained by dividing the difference in the
outcome means of the selected AB phases by the
standard deviation pooled across the data points in
the baseline phase. Three studies used a version of
Downloaded by [UZH Hauptbibliothek / Zentralbibliothek Zürich] at 13:24 10 September 2013
the mean baseline reduction procedure, and three
other studies used results from analyses using
Crosbie’s ITSACORR software. Two studies used
metrics based on an incorrectly specified piecewise
regression model.
Of the 25 meta-analyses, 7 used multiple indices to
describe their results. Of these 7, however, 4 used
PZD with PND, which tend to provide very similar
summaries. Campbell (2003) and Bonner and
Barnett (2004) used a version of the standardized
mean difference and the PND to assess treatments’
effectiveness. Maughan, Christiansen and Jenson
(2005) standardized
_
the results from ITSACORR and
also used SMD . Marquis et al. (2000) used three
effect sizes, including one _
very like the mean
baseline reduction, the SMD , and a regression-
based effect size.
Results of this survey indicated that most SSED
meta-analysts are using the simplest indicators to
synthesize results (i.e. PND and the standardized
mean difference). Given the typically small data sets
involved in these meta-analyses, only simple metrics
should really be used. However, use of these
particular indices has been criticized as noted above.
In addition to investigating the types of metrics
that were used to describe studies’ results, other
steps in the meta-analytic process were also briefly
summarized. The foundation of most SSEDs entails
a comparison between results in a baseline phase
with results in an intervention phase. Nevertheless,
the simple AB design in and of itself is not used due
to resulting validity threats (see, for example,
Kazdin, 1982). Instead, more complex designs,
such as multiple baseline, reversal, alternating
treatment designs, and the like, are used.
Interpretation of the results from these more
complex designs, however, still focus on the pattern
in an intervention phase compared with the pattern
in the relevant baseline phase. The problem is, of
course, that there are frequently multiple treatment
phases and even multiple baseline phases. In an
ABAB design, for example, a researcher could
calculate two indices to describe the treatment’s
effect on a single subject. (One index would describe
the effect from the first baseline to the first
treatment phase and the other would describe the
effect for the second AB phase). How are these two
indices then used in the meta-analysis? Are they
treated as independent data points? They should not
be treated as independent because they represent the
treatment’s effect on the same person. In a design
including the following sequence of phases, ABC, are
two effect sizes calculated; one describing results in
phase B versus the baseline phase (A) and another
effect size for the pattern of outcome scores in phase
C versus the same baseline phase? How do meta-
analysts handle these two dependent metrics?
Typically, a primary study does not just investigate
the effect of an intervention on a single subject.
In a multiple-baseline study, for example, the
researcher might assess a treatment’s effectiveness
for three participants who are part of the same
multiple-baseline study. This would lead to the
calculation of three outcomes. How are these three
metrics used in an ensuing meta-analysis? Are they
treated as independent? It seems likely that there is
some degree of dependence among these three
metrics, given the commonality in the participants’
experiences in their being assessed by the same
researcher, possibly being treated by the same
caregiver, and their assessment taking place in the
same setting.
It is also possible that the intervention’s effect is
not evaluated just for a single outcome. Quite
frequently, multiple measures are used to assess
the effect of an intervention. If, for example, two
kinds of social behaviors are tallied for a single
participant, then two outcomes could be calculated
for the participant. These metrics cannot be assumed
to be independent, because they describe the same
participant and the measures being assessed are
probably correlated.
Little methodological research addresses how
SSED meta-analysts should handle multiple treat-
ments, participants, or outcomes. In other words, it
is unclear how the dependence of outcomes within
each primary study is handled. It seems important
to consider how to handle these issues and
a preliminary step involves assessing how they are
currently being handled.
Techniques used to deal with the dependence of
outcomes yielded by the same metric
Given the lack of methodological focus on the issue
of dependence of multiple outcomes yielded by the
same metric within studies, the majority of meta-
analyses were not very clear about how this
issue was dealt with. Of the 25 meta-analyses
that were reviewed, 10 did not make explicit
 137 META-ANALYSES OF SINGLE-SUBJECT EXPERIMENTAL DESIGNS
how the inevitable multiple-treatment dependence
was handled. Some seemed to have treated the
multiple outcomes per primary study as indepen-
dent, but it was not completely clear. The most
commonly described method (in 7 meta-analyses)
for handling multiple-treatment dependence was to
average the multiple indices (one per treatment
phase versus preceding baseline phase) together.
_
This meant taking a simple mean of SMD s (e.g.
Swanson & Sachse-Lee, 2000) or using the mean or
median of the PNDs (e.g. Algozzine, Browder, &
Karvonen, 2001; Schlosser & Lee, 2000). In six of the
Downloaded by [UZH Hauptbibliothek / Zentralbibliothek Zürich] at 13:24 10 September 2013
meta-analyses, the multiple-treatment outcomes per
study were treated as independent, thereby ignoring
the possible dependence. In one study (Skiba, Casey,
& Center, 1985–1986), only the outcome for the first
treatment phase was used to counter possible carry-
over effects. And one study used the outcome
associated with the largest dosage treatment
(Allison, Faith, & Franklin, 1995).
The techniques used to handle multiple measures
per study were also reviewed. The majority of the
studies (n ¼ 13) summarized results separately for
each of the related outcomes. For example, Skiba
et al. (1985–1986) presented metrics for each of the
following different, but related, types of behavior:
withdrawn, noncompliant, management problem,
off-task, appropriate and social interaction beha-
viors. In eight of the meta-analyses it was very
unclear how the multiple related outcome depen-
dence was handled. Four studies calculated simple
averages across the multiple measures per
participant.
When multiple participants per study were
encountered, a similar set of techniques were used
to handle the resulting within-study dependencies.
Almost half of the studies (n ¼ 12) ignored the
dependence and treated each participant’s outcome
as independent. Four of the studies calculated
a single outcome for each study by aggregating the
participants’ outcomes. Three of the studies (Carey &
Matyas, 2005; Scholz & Ott, 2000; Wurthmann,
Klieser, & Lehmann, 1996) used meta-analytic
techniques to summarize the results gathered in
their study and treated each participant’s results as
independent. For example, Scholz and Ott (2000)
synthesized the 21 p values from ITSACORR analyses
of 21 participants’ data. In six studies, how the
dependence resulting from multiple participants per
study was handled was not clearly described.
In terms of the analyses conducted using the
resulting outcomes, the majority of meta-analyses
reported average (mean or median) metric values
across the set of metrics (either per study or per
participant as noted above). Sample or study char-
acteristics were explored as moderators of outcomes
in several studies (n ¼ 6), using either descriptive or
inferential statistics. All studies included
a table listing the outcomes by study (and participant,
treatment, and outcome, as mentioned above).
DISCUSSION
As outlined in the introduction, there are multiple
metrics that have been introduced and suggested for
use with SSEDs. Unfortunately, there are methodo-
logical issues associated with the majority of these
metrics. These issues are founded in the complex
time-series nature of SSEDs and in the inevitable
developmental trajectories of outcomes measured
over time. A critical weakness of many of the metrics
currently used in applied meta-analyses is that only
a single metric, rather than multiple metrics, is used
to describe a treatment’s effect. Development leads
to possible trends over time. If there is a trend in an
outcome even without intervention, then the level of
the outcome will inevitably change at a later (e.g.
intervention-phase) time point. For example,
a treatment might only be considered clinically
significant if it produces immediate change in the
dependent variable. For such studies, a summary of
differences in means between baseline and treat-
ment could be appropriate. More commonly, it is
recognized that the effect of a treatment might entail
more gradual improvement. For example, in devel-
oping communication behaviors in children with
autism, treatment is expected to be associated with
a gradual, positively accelerating trend. In this
scenario, not only is the level of the outcome
expected to change with the introduction of the
treatment, so is the growth, or slope, of the outcome.
Thus, a descriptor other than a single mean-shift
metric would be needed to describe the treatment’s
effect. A single number could not effectively convey
a change in both level and slope.
Other treatments might be designed to change
outcomes that already naturally increase (or
decrease) over time during baseline. Such treatments
might be designed to accelerate (or decelerate) the
trend in the outcome that already exists without
intervention. To identify the effectiveness of these
kinds of treatments, again, a single metric cannot
convey the expected change in level and trend. The
current paper has focused solely on the potential for
linear trends. There is, of course, a possibility of
curvilinear trends for certain outcomes and espe-
cially of asymptotic trends (Shadish & Rindskopf,
2007). This adds further complexity to the models
needed to describe an intervention’s effect, and is an
important area of continued research. It seems that
more specific guidelines that outline which metric to
 138 META-ANALYSES OF SINGLE-SUBJECT EXPERIMENTAL DESIGNS
use for which kind of treatment-effect investigation
are needed. In addition, while there are a host of
possible alternatives, the emphasis needs to be on
matching the method with the anticipated trends in
the data.
The other challenge to the identification of an
appropriate metric for summarizing SSED results is
autocorrelation. Autocorrelation changes the vari-
ability of the standardized metrics associated with
SSEDs over what would be expected with large-n
designs. Traditional tests of autocorrelation have
been found to be biased when used with data sets as
Downloaded by [UZH Hauptbibliothek / Zentralbibliothek Zürich] at 13:24 10 September 2013
small as those typically encountered in the SSED
literature. More recent research has identified some
modified test statistics that perform well in identify-
ing lag-one autocorrelation for small data sets
(Huitema & McKean, 2000; Riviello & Beretvas,
2008). It is hoped that future research will evolve
from better identification of, to potential correction
for, autocorrelation. This could then lead to metric
formulation that controls for autocorrelation result-
ing from repeated measures of individuals.
Additional challenges to the meta-analysis of
results from SSEDs have also been noted and
surveyed in the current study. A primary challenge
involves the inevitable dependence of metrics within
studies being synthesized. As noted above, this
dependence can result for individual participants’
data in the following ways: (a) from calculating
multiple outcomes with repeated use of single-
baseline-phase data (e.g. for a design incorporating
a pattern such as ‘ABC’); (b) from multiple treat-
ments per study (e.g. a design including the ‘‘ABAC’’
pattern); or (c) from multiple outcome measures per
participant. Multiple dependent outcomes could be
associated with a primary study’s results because the
study involves multiple participants (e.g. multiple-
baseline designs). From the results of the narrative
review, it seems clear that a number of techniques
are being used to handle this dependence. These
techniques match the various techniques used to
handle multiple dependent outcomes per study in
large-n meta-analyses. While several ad hoc techni-
ques are used with large-n meta-analyses (including
averaging together each study’s effect sizes, selecting
‘best’ single effect sizes for each study, etc.), multi-
variate pooling through generalized least squares
estimation (Becker, 1992) is also frequently used.
Unfortunately, generalized least squares cannot yet
be used with SSED results, because of the unknown
sampling distributions of most of the SSED metrics.
This again underscores the importance of identifying
SSED metrics and their associated sampling dis-
tributions. This could then lead to better-founded
techniques for handling within-study dependence.
Identification of optimal metrics and their sampling
distributions will also provide accurate variance
estimates that could then be used in the weighting
of SSED metrics for calculating pooled estimates.
CAVEATS AND RECOMMENDATIONS FOR SSED META-
ANALYSTS
In the culture of evidence-based practice and
accountability, many researchers and practitioners
are turning to meta-analysis to provide the relevant
evidence for best practice. The meta-analysis tradi-
tion was founded originally to summarize treatment
effects from large-n studies most typically involving
a comparison of groups at a single time point. As the
use of SSED studies increased, meta-analytic
researchers tried to impose the effect sizes used
with large-n studies on results from single-n studies.
Unfortunately, while SSED studies typically involve
a comparison of outcomes scores for an individual
under treatment with his or her scores during
a baseline phase, the pattern of these scores is
assessed over time for an individual. An effect size
needed to describe changes in an individual over
time will not have the same metric as an effect size
comparing groups of individuals at a single time
point. In fact, only for the simplest pattern of change
anticipated with a treatment (in which there is no
trend during baseline and treatment and the treat-
ment is designed only to change the outcome’s
level), could there be some correspondence between
large-n and SSED summaries. Both designs could
then be used to detect a change in outcome level.
However, even for this simplest scenario, the
variability underlying estimates, and thus the result-
ing metrics of the associated effect sizes, will differ
as a result of the studies’ designs. This subtle
difference between large-n and SSED studies lies at
the root of what complicates the formulation of
a useful effect size for use with single-n data. And,
more importantly, researchers should not quantita-
tively synthesize results from large-n and single-n
studies (Kavale, Mathur, Forness, Quinn, &
Rutherford, 2000). Separate large-n and SSED
syntheses should be conducted.
Using data from an applied meta-analysis of
school-based interventions’ effect on communication
skills, Wendt (2008) and Beretvas, Chung,
Machalicek and Riviello (2008) compared various
non-parametric and parametric effect-size estimates,
respectively. Different inferences about the interven-
tion’s effectiveness were made on the basis of
the metric that was used to describe the studies’
results. This does not seem surprising, given the
differing sources of the criticisms of SSED metrics
(e.g. autocorrelation, trend in baseline, need for
 139 META-ANALYSES OF SINGLE-SUBJECT EXPERIMENTAL DESIGNS
multiple metrics). The authors of both papers found
that, when the effect-size-based inferences did not
converge, it was possible to identify the source of the
divergence from the primary study’s data or design.
This strongly supports triangulation of metrics
through meta-analysts’ use of multiple SSED metrics
until some optimal metric has been derived. When
divergent inferences result, the meta-analyst is
strongly encouraged to identify the source of the
differences by close examination of each primary
study.
Finally, no mention has been made of criteria that
Downloaded by [UZH Hauptbibliothek / Zentralbibliothek Zürich] at 13:24 10 September 2013
could be used to categorize metric values mimicking
Cohen’s (1977) ‘small’, ‘moderate’ and ‘large’ cut-
offs. Until a metric (and sampling distribution) is
derived that better matches the pattern of SSED data
being synthesized, numerical cut-offs are perhaps
less important than the consideration of clinical
significance (Shogren, Faggella-Luby, Bae, &
Wehmeyer, 2004).
It is hoped that methodological research focused
on synthesizing results from SSEDs will continue to
evolve, and that some of these challenges will
eventually be overcome. In the meantime, careful
descriptive summaries of studies’ results can and
should be conducted.
REFERENCES
References used in the narrative literature review are marked with
an asterisk.
*Algozzine, B., Browder, D., & Karvonen, M. (2001). Effects of
interventions to promote self-determination for individuals
with disabilities. Review of Educational Research, 71, 219–277.
*Allison, D. B., Faith, M. S., & Franklin, R. D. (1995). Antecedent
exercise in the treatment of disruptive behavior: A meta-
analytic review. Clinical Psychology: Science and Practice, 2,
279–304.
Becker, B. J. (1992). Using results from replicated studies to
estimate linear models. Journal of Educational Statistics, 17,
341–362.
Beretvas, S. N., & Chung, H. (2007, May) R-squared change effect size
estimates for single-subject meta-analyses. Paper presented at the
7th Annual International Campbell Collaboration Colloquium
in London, England.
Beretvas, S. N., Chung, H., Machalicek, W. A., & Riviello, C. (2008,
May). Computation of regression-based effect size measures. Paper
presented at the 8th Annual International Campbell
Collaboration Colloquium in Vancouver, Canada.
Blumberg, C. J. (1984). Comments on ‘‘A simplified time-series
analysis for evaluating treatment interventions’’. Journal of
Applied Behavior Analysis, 17, 539–542.
 141–154.
Bonner, M., & Barnett, D. W. (2004). Intervention-based school
psychology services: Training for child-level accountability;
preparing for program-level accountability. Journal of School
Psychology, 42, 23–43.
Box, G. E. P., & Jenkins, J. M. (1976). Time series analysis:
Forecasting and control (2nd ed.). San Francisco, CA: Holden–
Day.
Busk, P. L., & Marascuilo, L. A. (1988). Autocorrelation
in single-subject research: A counterargument to the
myth of no autocorrelation. Behavioral Assessment, 10,
229–242.
Busk, P. L., & Marascuilo, L. A. (1992). Statistical analysis in
single-case research: Issues, procedures, and recommendations,
with applications to multiple behaviors. In T. R. Kratochwill, &
J. R. Levin (Eds.), Single-case research design and analysis: New
directions for psychology and education (pp. 159–185). Hillsdale, NJ:
Erlbaum.
Campbell, J. M. (2003). Efficacy of behavioral interventions for
reducing problem behavior in persons with autism: A quanti-
tative synthesis of single-subject research. Research in
Developmental Disabilities, 24, 120–138.

Carey, L. M., & Matyas, T. A. (2005). Training of somatosensory
discrimination after stroke: Facilitation of stimulus general-
ization. American Journal of Physical and Medical Rehabilitation, 84,
428–42.
Center, B. A., Skiba, R. J., & Casey, A. (1985–1986).
A methodology for the quantitative synthesis of intra-
subject design research. Journal of Special Education, 19,
387–400.
Cohen, J. (1977). Statistical power analysis for the behavioral sciences
(rev ed.). Hillsdale, NJ: Erblaum.
Cooper, H., & Hedges, L. V. (Eds.)., (1994). The handbook of research
synthesis. New York, NY: Russell Sage Foundation.
Crosbie, J. (1993). Interrupted time-series analysis with single-
subject data. Journal of Consulting and Clinical Psychology, 61,
966–974.
Crosbie, J. (1995). Interrupted time-series analysis with short
series: Why it is problematic; how it can be improved.
In J. M. Gottman (Ed.), The analysis of change (pp. 361–395).
Mahwah, NJ: Erlbaum.

Didden, R., Duker, P. C., & Korzilius, H. (1997). Meta-analytic
study on treatment effectiveness for problem behaviors with
individuals who have mental retardation. American Journal on
Mental Retardation, 101, 387–399.

DuPaul, G. J., & Eckert, T. L. (1997). The effects of school-based
interventions for attention deficit hyperactivity disorder: A
meta-analysis. School Psychology Review, 26, 5–27.
Faith, M. S., Allison, D. B., & Gorman, B. S. (1996). Meta-analysis
of single-case research. In D. R. Franklin, D. B. Allison, &
B. S. Gorman (Eds.), Design and analysis of single-case research
(pp. 245–277). Hillsdale, NJ: Erlbaum.
Galassi, J. P., & Gersh, T. L. (1993). Myths, misconceptions
and missed opportunity: Single-case designs and counsel-
ing psychology. Journal of Counseling Psychology, 40,
525–531.
Glass, G. V. (1976). Primary, secondary, and meta-analysis of
research. Educational Researcher, 5, 3–8.
Glass, G. V., Willson, V. W., & Gottman, J. M. (1975). Design and
analysis of time-series experiments. Boulder: Colorado Associated
University Press.
Gorman, B. S., & Allison, D. B. (1996). Statistical alternatives for
single-case designs. In D. R. Franklin, D. B. Allison, &
B. S. Gorman (Eds.), Design and analysis of single-case research
(pp. 159–214). Hillsdale, NJ: Erlbaum.
Gorsuch, R. L. (1983). Three methods for analyzing limited
time-series (N of 1) data. Behavioral Assessment, 5,
Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis.
New York: Academic Press.
Hershberger, S. L., Wallace, D. D., Green, S. G., & Marquis, J. G.
(1999). Meta-analysis of single-case designs. In R.
H. Hoyle (Ed.), Statistical strategies for small sample research
(pp. 107–132). Thousand Oaks, CA: Sage.
 140 META-ANALYSES OF SINGLE-SUBJECT EXPERIMENTAL DESIGNS
Huitema, B. E. (1985). Autocorrelation in applied behavior
analysis: A myth. Behavioral Analysis, 7, 107–110.
Huitema, B. E. (2004). Analysis of interrupted time-series
experiments using ITSE: A critique. Understanding statistics, 3,
27–46.
Huitema, B. E., & McKean, J. W. (1998). Irrelevant autocorrela-
tion in least-squares intervention models. Psychological Methods,
3, 104–116.
Huitema, B. E., & McKean, J. W. (2000). Design specification
issues in time-series intervention models. Educational and
Psychological Measurement, 60, 38–58.
Hunter, J. E., & Schmidt, F. L. (1990). Methods of meta-analysis:
Correcting error and bias in research findings. Newbury Park, CA:
Sage Publications.
Downloaded by [UZH Hauptbibliothek / Zentralbibliothek Zürich] at 13:24 10 September 2013
Jones, R. R., Weinrott, M., & Vaught, R. S. (1978). Effects of serial
dependency on the agreement between visual and statistical
inference. Journal of Applied Behavior Analysis, 11, 277–283.

Kahng, S., Iwata, B. A., & Lewin, A. B. (2002). Behavioral
treatment of self-injury, 1964 to 2000. American Journal on
Mental Retardation, 107, 212–221.
Kavale, K. A., Mathur, S. R., Forness, S. R., Quinn, M. M.,
& Rutherford, R. B. (2000). Right reason in the integration of
group and single-subject research in behavioral disorders.
Behavioral Disorders, 25, 142–157.
Kazdin, A. E. (1982). Single-case research designs: Methods for clinical
and applied settings. New York, NY: Oxford University Press.
Lundervold, D., & Bourland, G. (1988). Quantitative analysis of
treatment of aggression, self-injury, and property destruction.
Behavior Modification, 12, 590–617.
Ma, H. (2006). An alternative method for quantifying synthesis of
single-subject research: Percent of data points exceeding the
median. Behavior Modification, 30, 598–617.

Marquis, J. G., Horner, R. H., Carr, E. G., Turnbull, A. P.,
Thompson, M., Behrens, G. A., et al. (2000). A meta-analysis of
positive behavior support. In R. M. Gersten, E. P. Schiller, &
S. Vaughn (Eds.), Contemporary special education research: Syntheses
of the knowledge base on critical instructional issues (pp. 137–178).
Mahway, NJ: Erlbaum.

Mastropieri, M. A., & Scruggs, T. E. (1985–86). Early intervention
for socially withdrawn children. Journal of Special Education, 19,
429–441.

Mathur, S. R., Kavale, K. A., Quinn, M. M., Forness, S. R.,
& Rutherford, R. B. (1998). Social skills interventions with
students with emotional and behavioral problems: A quantita-
tive synthesis of single-subject research. Behavioral Disorders, 23,
193–201.

Maughan, D. R., Christiansen, E., & Jenson, W. R. (2005).
Behavioral parent training as a treatment for externalizing
behaviors and disruptive behavior disorders: A meta-analysis.
School Psychology Review, 34, 267–286.
Morris, S. B., & DeShon, R. P. (2002). Combining effect size
estimates in meta-analysis with repeated measures and
independent-groups designs. Psychological Methods, 7, 105–125.
Nourbakhsh, M. R., & Ottenbacher, K. J. (1994). The statistical
analysis of single-subject data: A comparative examination.
Physical Therapy, 74, 768–776.
Pankratz, A. (1983). Forecasting with univariate Box–Jenkins models:
Concepts and cases. New York: Wiley.
Parker, R. I., & Hagan-Burke, S. (2007). Single-case research
results as clinical outcomes. Journal of School Psychology, 45,
637–653.
Parker, R. I., Hagan-Burke, S., & Vannest, K. (2007). Percentage of
all non-overlapping data (PAND): An alternative to PND.
Journal of Special Education, 40, 194–204.
Parsonson, B. S., & Baer, D. M. (1992). The visual analysis of data,
and current research into the stimuli controlling it.
In T. R. Kratochwill, & J. R. Levin (Eds.), Single-case research
design and analysis: new directions for psychology and education
(pp. 15–40). Hillsdale, NJ: Erlbaum.
Riviello, C., & Beretvas, S. N. (2008). Detecting lag-one autocorrelation
in interrupted time series designs with small sample sizes. Manuscript
submitted for publication.

Schlosser, R. W., & Lee, D. L. (2000). Promoting general-
ization and maintenance in augmentative and alternative
communication: A meta-analysis of 20 years of effectiveness
research. Augmentative and Alternative Communication, 16,
208–226.
Schlosser, R. W., Lee, D., & Wendt, O. (in press). The
percentage of non-overlapping data (PND): A systematic
review of reporting characteristics in systematic reviews and
meta-analyses. Evidence-Based Communication Assessment and
Intervention.

Scholz, O. B., & Ott, R. (2000). Effect and course of tape-based
hypnotherapy in subjects suffering from insomnia. Australian
Journal of Clinical Hypnotherapy and Hypnosis, 21, 96–114.

Scotti, J. R., Evans, I. M., Meyer, L. H., & Walker, P. (1991).
A meta-analysis of intervention research with problem
behavior: Treatment validity and standards of practice.
American Journal on Mental Retardation, 96, 233–256.
Scruggs, T. E., Mastropieri, M. A., & Casto, G. (1987). The
quantitative synthesis of single-subject research: Methodology
and validation. Remedial and Special Education, 8, 24–33.

Scruggs, T. E., Mastropieri, M. A., Cook, S. B., & Escobar, C.
(1986). Early intervention for children with conduct disorders:
A quantitative synthesis of single-subject research. Behavioral
Disorders, 11, 260–71.

Scruggs, T. E., Mastropieri, M. A., Forness, S. R., & Kavale, K. A.
(1988). Early language intervention: A quantitative synthesis
of single-subject research. Journal of Special Education, 22,
259–83.

Scruggs, T. E., Mastropieri, M. A., & McEwen, I. (1988). Early
intervention for developmental functioning: A quantitative
synthesis of single-subject research. Journal of the Division for
Early Childhood, 12, 359–67.
Shadish, W. R., & Rindskopf, D. M. (2007). Methods for evidence-
based practice: Quantitative synthesis of single-subject designs.
In G. Julnes, & D. J. Rog (Eds.), Informing federal policies on
evaluation method: Building the evidence base for method choice in
government sponsored evaluation (pp. 95–109). San Francisco:
Jossey–Bass.

Shogren, K. A., Faggella-Luby, M. N., Bae, S. J., & Wehmeyer, M. L.
(2004). The effect of choice-making as an intervention for
problem behavior: A meta-analysis. Journal of Positive Behavior
Interventions, 6, 228–237.

Skiba, R. J., Casey, A., & Center, B. A. (1985–86). Nonaversive
procedures in the treatment of classroom behavior problems.
Journal of Special Education, 19, 459–481.

Swanson, H. L., & Sachse-Lee, C. (2000). A meta-analysis of
single-subject-design intervention research for students with
LD. Journal of Learning Disabilities, 33, 114–136.

Swanson, H. L., O’Shaughnessy, T. E., & McMahon, C. M. (1998).
A selective synthesis of single subject design intervention
research on students with learning disabilities. Advances in
Learning and Behavioral Disabilities, 12, 79–126.
Tryon, W. W. (1982). A simplified time-series analysis for
evaluating treatment interventions. Journal of Applied Behavior
Analysis, 15, 423–429.
van den Noortgate, W., & Onghena, P. (2003). Hierarchical linear
models for the quantitative integration of effect sizes in single-
case research. Behavior Research Methods, Instruments & Computers,
35, 1–10.
van den Noortgate, W., & Onghena, P. (in press). A multi-level
analysis of single-subject studies. Evidence-Based Communication
Assessment and Intervention.
 141 META-ANALYSES OF SINGLE-SUBJECT EXPERIMENTAL DESIGNS

Wendt, O. (2008, May). Computation of non-regression-based effect Wurthmann, C., Klieser, E., Lehmann, E., & Krauth, J. (1996).
size metrics. Paper presented at the 8th Annual Single-subject experiments to determine individually differen-
International Campbell Collaboration Colloquium in tial effects of anxiolytics in generalized anxiety disorder.
Vancouver, Canada. Neuropsychobiology, 33, 196–201.
White, O. R., Rusch, F. R., Kazdin, A. E., Xin, Y. P., Grasso, E., Dipipi-Hoy, C. M., & Jitendra, A. (2005). The
& Hartmann, D. P. (1989). Applications of meta- effects of purchasing skill instruction for individuals with
analysis in individual subject research. Behavioral developmental disabilities: A meta-analysis. Exceptional Children,
Assessment, 11, 281–296. 71, 379–400.
Downloaded by [UZH Hauptbibliothek / Zentralbibliothek Zürich] at 13:24 10 September 2013