Lecture Notes 7: Bootstrap: continued

Maria Ponomareva
Econ 9615
1 Alternative resampling procedures
There is a number of alternative ways to estimate F
0
consistently. The m out of n boot-
strap and subsampling methods require weaker assumptions for consistency than traditional
nonparametric or parametric bootstrap.
1.1 Jackknife
The ordinary jackknife estimator is given as follows. Let T
n,(i)
be computed from the sample
where the ith observation is deleted. Then the jackknife estimator is
T
n,J
=
1
n
n

i=1
T
n,(i)
Bias correction with jackknife: let T
n
=

and let p
i
= n

(n1)

(i)
. Then the jackknife
bias=adjusted estimator is

=
1
n
n

i=1
p
i
In this case

is unbiased (for linear

).
Jackknife variance estimator for the bias-adjusted estimator:
v
J
=
1
n(n 1)
n

i=1
(p
i

)(p
i

Advantages: faster and easier than bootstrap. Use to adjust for bias and calculate
variance.
1
1.2 Wild bootstrap
Wild bootstrap is a method that is based on residuals. Suppose that T
n
=
1
n
n

i=1
g
n
(X
i
).
1. Choose a distribution Q with E
Q
(Z) = 0 and E
Q
(Z
2
) = 1.
2. Draw iid sample {Z
W
i
: 1 i n} from this distribution. Let
T
W
n
=
n

i=1
Z
W
i
_
g
n
(X
i
) n
1
T
n
_
3. Estimate G
n
(, F
n
) by EDF of T
W
n
.
Common distribution Q used in the literature is a two-point distribution with
P
_
Q =
1 +

5
2
_
=

5 1
2

5
P
_
Q =
1

5
2
_
=

5 + 1
2

5
Advantages: generalizes traditional bootstrap; linear regression: heteroskedasticity.
1.3 m out of n bootstrap
Here bootstrap sample is obtained by drawing with replacement m < n observations from
the sample {X
i
: 1 i n}. Therefore, the m out of n bootstrap estimator of G
n
(, F
0
)
is G
m
(, F
n
). For consistency of this resampling method, we need m , n , and
m/n 0. If this is the case, then the random sampling error of F
n
as an estimator of F
0
is smaller that the random sampling error of F
mn
as an estimator of F
n
. This makes the m
out of n bootstrap less sensitive to the behavior of G
n
(, F) in the neighborhood of F
0
.
Advantages: requires weaker assumptions for consistency.
1.4 Subsampling
Let

n
=

n
(X
1
, . . . , X
n
) be a consistent estimator of . Let T
n
= (n)(

n
) be a statistic
for testing hypothesis about . Here (n) is a sequence of numbers such that if G
n
(, F
0
) is
the distribution of T
n
, then
G
n
(, F
0
) G

(, F
0
),
2
where G

(, F
0
) is nondegenerate.
Let {X
i
j
: j = 1, . . . , m} be a kthe subset of size m < n from the data {X
i
: i = 1, . . . , n}
and let

m,k
=

m
({X
i
j
: j = 1, . . . , m}). There are total N
mn
=
_
n
m
_
possible subsamples.
Then the subsampling distribution is
G
nm
(u) =
1
N
mn
Nmn

k=1
T
_
(m)(

m,k

n
) u
_
Note that G
m
(, F
0
) is the exact sampling distribution of (m)(

m
). Then if n is large
but m/n is small, then sampling distributions of (m)(

n
) and (m)(

m
) are close.
So, if m goes to innity, then G
nm
() and G

(, F
0
) are close.
The only major requirement for subsampling to work: the limit G

(, F
0
) exists and is
non-degenerate.
2 OLS
Consider the following linear model:
Y
i
= X

i
+ e
i
, where E[e
i
|X
i
] = 0
Nonparametric bootstrap resamples observations (Y

i
, Xi

) from joint empirical distribution

function (EDF), which implies that
Y

i
= X

+ e

i
, where E

[e

i
X

i
] = 0
but in general
E

[e

i
|X

i
] = 0
That is, the bootstrap distribution does not impose mean independence assumption, and as
the result it is an inecient estimator of the true distribution. One way to impose mean
independence on the bootstrap world is to impose a stronger assumption of independence
and use a parametric bootstrap. Let
e
i
= Y
i
X

3
Then we can sample bootstrap errors e

i
randomly with replacement from OLS residuals
{ e
i
, 1 i n}. We can either treat regressors as xed, so that X

i
= X
i
, or we can sample
them independently from {X
i
: 1 i n} and generate the bootstrap sample as:
Y

i
= X

+ e

i
However, this approach rules out heteroskedasticity in the bootstrap world. To impose only
mean independence in the bootstrap world (and therefore to allow for possible heteroskedas-
ticity), we can use wild bootstrap procedure to generate bootstrap samples:
P
_
e

i
=
1 +

5
2
e
i
_
=

5 1
2

5
P
_
e

i
=
1

5
2
e
i
_
=

5 + 1
2

5
3 Bootstrap for Dependent Data
We need to make sure that our bootstrap procedure preserves the properties of the original
data generating process that generated our data.
3.1 Time series models
Suppose that we have the following model:
X
t
= + X
t1
+ U
t
, X
0
= 0
where {U
t
: 1 t T} are independent and identically distributed. Let
T
and

T
be

T-consistent, asymptotically normal estimators of and . Dene

U
t
= X
t

T

T
X
t1
The the bootstrap sample can be generated in the following way:
1. Draw a random sample {U

t
: 1 t T} with replacement from the set of residuals
{

U
t
: 1 t T};
2. Set X

0
= 0. Then X

t
=
T
+

T
X

t1
+ U

t
.
4
This bootstrap procedure preserves dependence structure between {X

t
}. One can also spec-
ify a parametric distribution of U
t
(e.g. U
t
N(0,
2
)), estimate with s
T
and sample
bootstrap residuals from N(0, s
2
T
) distribution.
Now suppose that instead of the linear regression model we have a GMM model:
E[X
t
|X
1
, . . . , X
t1
] = m(X
t1
, )
In this case, we cannot reduce the data generating process to independent sampling from
some probability distribution. That is, in this general setting we cannot expect that the
generalized residuals e
t
= X
t
m(X
t1
, ) are independent (let alone identically distributed).
In this case, the bootstrap can be implemented by dividing the data into blocks and then
sampling those blocks randomly with replacement. The blocks may be either overlapping or
non-overlapping.
Block bootstrap: given the data {X
t
: t = 1, . . . , T}, we construct the bootstrap
sample in the following way
1. Pick some l < t (length of the block).
2. (a) For non-overlapping blocks: block 1 is {X
t
: t = 1, . . . , l}, block 2 is {X
l+j
: j =
1, . . . , l},..., block k is {X
l(k1)+j
: j = 1, . . . , l}.
(b) For overlapping blocks with overlap of length q < l, block 1 is {X
t
: t = 1, . . . , l},
block 2 is {X
lq+j
: j = 1, . . . , l},..., block k is {X
l(kq)+j
: j = 1, . . . , l}.
3. Sample blocks randomly with replacement and generate bootstrap sample {X

t
: t =
1, . . . , T} by glueing blocks together in the orders they were sampled.
One can also use overlapping blocks with lengths that are sampled randomly from a geo-
metrical distribution. Then the block bootstrap procedure will enforce stationarity in the
bootstrap world.
Block length: the block length must increase with T for block bootstrap to be consis-
tent. The optimal rate at which l grows with T depends on what is being estimated. Hall et
al. (1995) showed that if the parameters of interest are bias and variance, then the optimal
length is l = O(T
1/3
). Foe a one-sided test, the optimal length is l = O(T
1/4
) and for a
two-sided test the optimal length is l = O(T
1/5
).
5
3.1.1 Special case: unit root
Lets go back to the linear autoregression model
X
t
=
0
X
t1
+ U
t
Then OLS estimator of is

=
T

t=1
X
t
X
t1
T

t=1
X
2
t1
If U
t
N(0, 1), then the t-statistic for testing hypotheses about is
=
_
T

t=1
X
2
t1
_
1/2
(

0
)
and its distribution depends on
0
in a discontinuous way (discontinuity is at
0
= 1).
3.2 Panel data models
Panel data contain both cross-sectional and time dimensions. Data are usually assumed to
be independent along the cross-section dimension (units), but not along the time dimension.
Consider the following general linear panel data model:
Y
it
= X

it
+
i
+
t
+ e
it
Here
i
is unobserved heterogeneity across units that is constant over time (personal unob-
served characteristics like ability) and
t
is unobserved heterogeneity across time (common
time shocks).
Consider the following three ways to construct the bootstrap sample based on the data
{(Y
it
, X
it
)}:
1. Cross-sectional bootstrap: resample only individual randomly with replacement.
That is, sample from {Z
n
} randomly with replacement, where Z
i
= {(Y
it
, X
it
) : t =
1, . . . , T}.
2. Block bootstrap: x individuals and resample only along time dimension using block
bootstrap. That is, for each i sample blocks from {(Y
it
, X
it
) : t = 1, . . . , T} and glue
those blocks together.
6
3. Double bootstrap: combination of the two. For example, start by resampling only
individuals and obtain intermediate data sample {(Y

it
, X

it
)}. Then using this interme-
diate sample, resample along the time dimension by block bootstrap and obtain nal
bootstrap sample {(Y

it
, X

it
)}.
In many cases, cross-sectional and block bootstrap may be inconsistent for panel data mod-
els. Cross-sectional bootstrap may be inconsistent in the presence of time shocks and serial
correlation in regressors (it underestimates the variance), and block bootstrap may be in-
consistent in the presence of individual-specic heterogeneity (it also underestimates the
variance since we do not sample
i
s) Double bootstrap takes into account both time shocks
and unobserved heterogeneity
i
that is constant over time.
4 Summary
1. If asymptotic approximation G

(, F) is continuous in F in a neighborhood of F
0
,
then both bootstrap and asymptotic approximations are valid. In general, bootstrap
approximation is valid under weaker assumptions than needed for asymptotic approx-
imation.
2. If statistic of interest is asymptotically pivotal (i.e. G

(, F
0
) does not depend on F
0
,
then use bootstrap - it will provide asymptotic renement (except when the SFM is
violated, e.e LAD estimator). If statistic is not asymptotically pivotal, then there is a
reason for using bootstrap when it is hard to estimate G

(, F
0
) (e.g. LAD estimator,
trimmed OLS estimator).
3. Bootstrap sampling process must mimic the sampling process that generated the data
(wild bootstrap for heteroskedasticity, block bootstrap for time series data, double
bootstrap for panel data). That is, assumptions made in the model must be satised
in the bootstrap world (no less, but also no more, or the bootstrap variance will be
too small).
4. If using bootstrap to test a hypothesis, do not forget to enforce this hypothesis in the
bootstrap world (recenter!).
5. For panel data, use double bootstrap.
6. If everything else fails, use subsampling (just make sure that the limiting distribution
is nondegenerate).
7
References
[1] Hansen, B. (2010), Econometrics, manuscript, UW-M (link).
[2] Horowitz, J. (2001), The Bootstrap, Handbook of Econometrics, vol. 5, ch. 52, pp.
3159-3228 (link).
[3] Mammen, E. (1992), When Does Bootstrap Work?, Lecture Notes in Statistics,
Springer-Verlag.
8