MA585 Time Series Analysis Lecture 2

Time Series Analysis: Model Building
Autoregressive Moving Average Processes
Ashis Gangopadhyay
Boston University
Spring 2016
Ashis Gangopadhyay (Boston University) ARMA Models Spring 2016 1 / 49

Linear Process
Denition
Let et be WN (0, 2 ). A process Xt is called a linear process if it can be
represented as
Xt = + et + 1 et 1 + 2 et 2 + ....

= + j et j
j =0
where we assume 0 = 1 and j=0 2j < .

Realizations from Linear Processes

Stationarity of Linear Process
We now investigate if a linear process is stationary.

Mean of the process:
t = E (Xt ) =
a constant.
Variance of the process:

Var (Xt ) = 2 2j
j =0
a constant.

Covariance of the process:
!

Cov (Xt , Xt +h ) = Cov j et j, k et + h k
j =0 k =0

= j k Cov (et j , et + h k )
j =0 k =0

= j k E ( et j et + h k )
j =0 k =0

= 2
j j +h
j =0
the last step follows from the fact that
2 if k = j + h
E ( et j et + h k) =
0 otherwise
Thus the autocovariance depends only on h. However, for the process to
be stationary, we need the autocovariance function to be nite for all h.
To see this, note
j(h)j = jCov (Xt , Xt +h )j

q
Var (Xt )Var (Xt +h )

= 2 2j
j =0
<
follows from the assumption on the parameters of the linear process.

Thus we conclude that the process is stationary if

2j < .
j =0

Backshift (or Lag) Operator
A backshift operator transforms an observation in a time series to the

previous one, i.e.,
BXt = Xt 1
It follows that
B k Xt = Xt k
Thus we can express the linear process in terms of the backshift operator
as follows
Xt = + (B )et
where (B ) = j=0 j B j , 0 = 1. Note that backshift operator is also
called a Lag Operator and denoted by L.

Autoregressive (AR) Process
Denition
An autoregressive process of order p [denoted by AR (p )] is given by
(Xt ) 1 (Xt 1 ) 2 (Xt 2 ) ... p (Xt p ) = et
where E (Xt ) = is a constant and et WN (0, 2 ).
Remark
Without loss of generality, we can assume that = 0 and rewrite the
AR (p ) process as Xt 1 Xt 1 2 Xt 2 ... p Xt p = et
The process can be expressed in terms of the backshift operator as
(B )Xt = et where (B ) = 1 1 B 2 B 2 .... p B p .

AR(1) Process: Condition for Stationarity
We will now discuss the su cient condition for stationarity of an AR (p )

process. First, lets consider a simpler AR (1) model given by
Xt Xt 1 = et . Under what condition this process is stationary? Note
that (by taking advantage of the recursive relation) we can express an
AR (1) process as
Xt = Xt 1 + et
= (Xt 2 + et 1 ) + et
= 2 Xt 2 + et 1 + et
...................
= et + et 1 + 2 et 2 + ....

AR(p) Process: Condition for Stationarity
Thus Xt is a linear process given by

Xt = j et j (1)
j =0
From the condition for stationarity of a linear process discussed earlier, it

follows that the AR (1) process is stationary if

2j < (2)
j =0
However, the condition (2) holds if jj < 1. Thus an AR (1) process is

stationary if the absolute value of the coe cient is less than 1.

Remark
There is an alternative way the relation (1) can be established. This
approach is particularly useful for higher order AR models. First note that
an AR (1) model can be expressed as (1 B )Xt = et . However, our
objective is to write Xt as Xt = (B )et , where (B ) = j=0 j B j ,
0 = 1.
Thus for the two representations of Xt to be equal, we need
(1 B )(B ) = 1
i.e,
(1 B )(1 + 1 B + 2 B 2 + ....) = 1
Solving this identity for j we get j = j , for j = 0, 1, 2, ...

What can we say about stationarity of an AR (1) process if jj 1? First
consider the case = 1. Under this condition, the AR (1) process is
given by Xt = Xt 1 + et , which is a random walk process and it is
non-stationary (Why?).
Now consider the case jj > 1, and write the AR (1) process as
Xt +1 = Xt + et +1 . We can rewrite the process as
1 1
Xt = Xt +1 et + 1

As shown in (1), we can express the process as
j
1
Xt =
et + j
j =1
and this process is stationary as j=1 ( 1 )2j < .Thus, we have concluded
that the AR (1) process is stationary as long as 6= 1. Lets think through
this a bit more carefully. Do we really want to consider AR (1) processes
for all 6= 1 for modeling a real time series data?
Causality of AR process
Denition
An AR (p ) process is called causal if it can be expressed as
Xt = et + 1 et 1 + 2 et 2 + ....
We will now discuss the condition for causality of an AR (p ) process.

First, lets consider an AR (1) process. It follows from (1) that the process
is causal if jj < 1. However, lets express the condition in another way.
First, note that we can write an AR (1) process as (1 B )Xt = et ,
where B is the backshift operator. The equation
1 B = 0
is called the characteristic equation of the AR (1) process. In this

equation, we can treat B as a variable, and we can solve the equation in
B. Thus, the solution of the equation is given by B = 1 . It is clear that
an equivalent condition for causality for an AR (1) process is that the root
of the characteristic equation lie outside the unit interval, i.e., jB j > 1.
Causality of AR Process
Now lets consider a zero mean AR (p ) process
Xt 1 Xt 1 2 Xt 2 .... p Xt p = et
That is
(1 1 B 2 B 2 .... p B p )Xt = et
Thus the characteristic equation of the AR (p ) process is given by
1 1 B 2 B 2 .... p B p = 0
Fact
An AR (p ) process is causal if all roots of its characteristic equation lie
outside the unit circle.

Example
Consider the AR (2) process given by
Xt 1.1Xt 1 + 0.3Xt 2 = et
Is this process causal?

The characteristic equation is given by
1 1.1B + 0.3B 2 = 0
The solutions of the quadratic equation are 1.67 and 2.0. Thus, both
roots lie outside the unit interval. Hence the process is causal.

Example
Consider the AR (2) process
Xt 1.8Xt 1 + 0.81Xt 2 = et
Is the process causal?

1 1.8B + 0.81B 2 = 0
has solution given by 1.1 with multiplicity 2. Thus the process is causal.

Example
Consider the process
Xt 1.9Xt 1 + 1.1Xt 2 0.2Xt 3 = et

1 1.9B + 1.1B 2 0.2B 3 = 0

The roots of the characteristic equation are 2.0, 2.5 and 1.0. Thus one of
the roots does not lie outside the unit interval. Hence the process is not
causal.

Example
Consider the process
Xt Xt 1 + 0.81Xt 2 = et

The roots of the characteristic equation
1 B + 0.81B 2 = 0
are (0.62 0.93i ); i.e., the roots are complex. However, the complex
modulus of the roots are given by
q
(0.62)2 + (0.93)2 = 1.12 > 1
Thus the roots lie outside the unit circle. Hence the process is causal.

Remark
The condition for causality of AR(2) process can also be expressed in the
following way: An AR(2) process Xt 1 Xt 1 2 Xt 2 = et is causal if
2 + 1 < 1
2 1 < 1
1 < 2 < 1
i.e, if the coe cients 1 and 2 belongs to the triangular region of R 2

dened by the three inequalities.
Example
Is the process Xt Xt 1 + 0.81Xt 2 = et causal? Note that 1 = 1 and
2 = 0.81. Thus 2 + 1 = 0.19 < 1, 2 1 = 1.81 < 1 and
1 < 2 < 1. Thus the process is causal.
ACF and PACF of Causal AR Process
Lets rst consider the special case of a zero mean AR (1) process
Xt Xt 1 = et and derive the ACF of the process. Recall that the
process is causal if jj < 1. Thus for any h 1, the autocovariance
function of the process is given by
(h) = E (Xt +h Xt )
= E [(Xt +h 1 + et +h )Xt ]
= E (Xt +h 1 Xt ) + E (et +h Xt )
= (h 1)
since et +h is uncorrelated with Xt , for all h 1. Taking advantage of the
recursive relation, we get
(h ) = h (0)
Dividing both sides by (0), we get the ACF of the process
(h) = h , for h 0.
ACF and PACF of Causal AR Process
Remark
Note that in this approach, we never actually compute the
(0) = Var (Xt ). What is (0) for the AR (1) process?
Lets now consider the PACF of the process. Note

11 = (1) =
1 (1) 1
(1) (2) 2
22 = = =0
1 (1) 1
(1) 1 1
In general, we can show
if h = 1
hh =
0 if h 2
ACF and PACF of AR(1) Process
The ACF of AR(1) process decays to zero exponentially, whereas the

PACF cuts-o to 0 after lag 1.

Consider a zero mean causal AR (2) process Xt 1 Xt 1 2 Xt 2 = et
To derive the ACF of the process, we proceed in an analogous manner.
For h 1,
(h) = E (Xt +h Xt )
= E [(1 Xt +h 1 + 2 Xt +h 2 + et +h )Xt ]
= 1 (h 1) + 2 (h 2)
since E (et +h Xt ) = 0, for h 1. Thus the autocovariance function is the

solution of the second order dierence equation given by
(h ) 1 (h 1) 2 (h 2) = 0
Dividing both sides by (0), we get
(h ) 1 (h 1) 2 (h 2) = 0
Note that if h = 1
(1) 1 (0) 2 ( 1) = 0
or , (1 2 ) (1) = 1
1
0r , (1) =
1 2
Next, substituting h = 2 in the dierence equation and dividing by (0),

we get
(2) = 1 (1) + 2
21 + 2 22
or , (2) =
1 2
We can continue in this manner substituting h = 3, 4, ...recursively solving

for (h).
Remark
A more e cient approach to derive ACF of an AR(2) process is to solve
the dierence equation. Suppose B1 and B2 be the solutions of the
characteristic equation associated with the AR (2) process. If the two
solutions are distinct, then the general solution of the dierence equation
is given by
(h) = c1 B1 h + c2 B2 h
and if the two solutions are equal, i.e., B1 = B2 = B, i.e., the solution of
the characteristic equation is B with multiplicity 2, then the solution of
the dierence equation is given by
h
( h ) = ( c1 + c2 h ) B
where c1 and c2 are constants that can be evaluated from the initial
2 + 2
conditions (1) = 1 1 and (2) = 1 1 2 2 .
2 2

Implication of general solution of the dierence equation
Since the process is causal, the roots of the characteristic lie outside the
unit circle, which implies that the autocorrelation decays to zero
exponentially, or, if the roots are complex, as damped sine wave.
Now lets consider the PACF of an AR (2) process. Note that

1
11 = (1) =
1 2
(2) (1)2
22 =
1 (1)2
21 +2 22 1 2
1 2 1 2
= 2
1
1 1 2
= 2
1 (1) (1)
(1) 1 (2)
(2) (1) (3)
33 =
1 (1) (2)
(1) 1 (1)
(2) (1) 1
1 (1) 1 + 2 (1)
(1) 1 1 (1) + 2
(2) (1) 1 (2) + 2 (1)
=
1 (1) (2)
(1) 1 (1)
(2) (1) 1
= 0
Similarly we can show that the PACF of AR (2) process is 0 for all other h..
Thus the PACF of an AR (2) process is given by
8
>
< 1 12 for h = 1
hh = for h = 2
>
: 02 for h 3
Remark
To summarize, the ACF of an AR(2) process decays to zero exponentially
or as damped sine wave, while the PACF cuts-o to zero after lag 2.
We will skip the derivation of ACF and PACF of the general AR (p )

process, which follows essentially in the same manner. However, the
characteristics of ACF and PACF that we have discussed in the special
cases AR (1) and AR (2) hold for general AR (p ) process; i.e., the ACF of
an AR (p ) process decays to zero exponentially or as damped sine wave,
while the PACF cuts-o to zero after lag p.
Examples of ACF and PACF of AR(2) Process

Moving Average Process
Denition
A moving average process of order q (denoted by MA(q )) is given by
Xt = + et + 1 et 1 + 2 et 1 + .... + q et q
where E (Xt ) = and et WN (0, 2 ).

As before, without loss of generality we can assume that = 0.
Is a MA(q ) process stationary?

Invertibility of Moving Average Process
Denition
A MA(q ) process is called invertible if it can be expressed as an innite
order AR process given by
Xt 1 Xt 1 2 Xt 2 ...... = et
Fact
A MA(q ) process is invertible if all roots of its characteristic equation
1 + 1 B + 2 B 2 + ..... + q B q = 0
lie outside the unit circle.

Example
Consider the MA(1) process
Xt = et 0.2et 1
Is this process invertible?

The root of the characteristic equation
1 0.2B = 0
1
is B = 0.2 = 5. Thus jB j > 1; hence the process is invertible.

Example
Xt = et + 0.2et 1 1.2et 2
1 + 0.2B 1.2B 2 = 0
with roots 0.833 and 1. Thus the process is not invertible.

Example
Xt = et 0.7et 1 + 0.6et 2
1 0.7B + 0.6B 2 = 0
with
p roots 0.58 1.15i. The complex modulus is given by
(0.58)2 + (1.15)2 > 1. Thus the process is invertible.

Example
Xt = et + 0.8et 1 0.15et 2
The roots of the characteristic equation are 1.04 and 6.38. Thus the
process is invertible.

ACF and PACF of Moving Average Process
First consider a MA(1) process
Xt = et + et 1
It is easy to check that the ACF of the process is given by

8
< 1 for h = 0

(h ) = 2 for h = 1
: 1 +
0 for h 2

The PACF of the process is given by

11 = (1) =
1 + 2
(2) (1)2 2 2 (1 2 )
22 = = =
1 (1)2 1 + 2 + 4 1 6
(1)3 3 (1 2 )
33 = =
1 2 (1)2 1 8
In general, for h 1,
( 1 )h +1 h ( 1 2 )
hh =
1 2 (h +1 )
which decays to zero exponentially. Thus, we have shown that the ACF of
a invertible MA(1) process cuts-o after lag 1, while the PACF decays to
zero exponentially.
Remark
Invertibility condition imposes a uniqueness restriction on a MA process.
To understand this, consider the two MA(1) processes given by
Xt = et + 0.4et 1
and
Xt = et + 2.5et 1
It is easy to see that both processes have the same ACF

8
< 0 for h = 0
(h ) = 0.34 for h = 1
:
0 for h 2
However, only the rst process is invertible. Thus there is a unique

correspondence between an invertible MA process and its ACF.
Now lets consider an invertible MA(2) process
Xt = et + 1 et 1 + 2 et 2
The ACF of the process is easy to nd, and is given by

8
> 1 for h = 0
>
>
< 1 (12+ 2 2) for h = 1
(h ) = 1 + 1 + 2
>
>
2
for h = 2
> 2
: 1 + 1 + 2
2
0 for h 3
Thus ACF cuts-o after lag 2. A formula for PACF of the process is
di cult to derive. But it can be shown that the PACF decays to zero
exponentially or as a damped sine wave depending on whether the roots of
the characteristic equation is real or complex.
In general, the ACF of an invertible MA(q) process cuts-o to zero after q

lags and PACF decays to zero exponentially or as a damped sinusoid.

Autoregressive Moving Average (ARMA) Process
Denition
An autoregressive moving average process of order (p, q ) (denoted by
ARMA(p, q )) is given by
(Xt ) 1 (Xt 1 ) .. p (Xt p ) = et + 1 et 1 + .. + q et q
where E (Xt ) = and et WN (0, 2 ).

As before, without loss of generality we can assume = 0.
Remark
Note that ARMA(p,q) model can also be expressed in the backshift
operator form as (B )(Xt ) = (B )et , where
(B ) = 1 1 B 2 B 2 .... p B p and
(B ) = 1 + 1 B + 2 B 2 + .... + q B q .

Denition
An ARMA(p, q ) process is causal if the roots of the characteristic equation
1 1 B 2 B 2 .... p B p = 0
lie outside the unit circle. In addition an ARMA(p, q ) process is invertible

if the roots of the characteristic equation
1 + 1 B + 2 B 2 + .... + q B q = 0
lie outside the unit circle.

Example
Consider the ARMA(2,1) process given by
Xt 1.2Xt 1 + 0.2Xt 2 = et + 0.2et 1
The roots of thecharacteristic equation 1 1.2B + 0.2B 2 = 0 are 5 and 1.

Thus one of the roots does not fall outside the unit interval, so the process
in non-causal. However, the process is invertible as the root of the
equation 1 + 0.2B = 0 fall outside the unit interval.
Remark
An implicit assumption in the denition of ARMA(p, q ) process is that the
characteristic equations (B ) = 0 and (B ) = 0 do not have a common
root. Why is this assumption important? Consider the following ARMA
process Xt + 0.6Xt 1 0.16Xt 2 = et + 0.8et 1 . What is the order of
this ARMA process?
ACF and PACF of ARMA(1,1) Process
Lets consider a causal and invertible ARMA(1, 1) process
Xt Xt 1 = et + et 1
and derive the ACF of the process. First note that we can express the
process as an innite order MA process given by
Xt = et + 1 et 1 + 2 et 2 + ...
where
j = j 1
( + ) for j = 1, 2, ...
Thus it follows
E (et Xt ) = 2
and
E ( et 1 Xt ) = 1 2

Now, as before
(h) = E (Xt +h Xt )
= E ((Xt +h 1 + et +h + et +h 1 ) Xt )
= (h 1) + E (et +h Xt ) + E (et +h 1 Xt )
Thus, for h = 0
(0) = (1) + (1 + 1 )2 (*)
for h = 1
(1) = (0) + 2 (**)
and for h 2
(h) = (h 1) (***)

Solving (*) and (**)
!
2 1 + 2 + 2
(0) =
1 2
and
( + )(1 + )
(1) = 2
1 2
Also, it follows from (***) that for h 2,
(h ) = h 1
(1)
h 1
or , (h) = (1)
diving both sides by (0).
Thus, we have the ACF of ARMA(1, 1) process given by
8
>
< 1 for h = 0
(+ )(1 + )
(h ) = 1 +2 + 2
for h = 1
>
: h 1
(1) for h 2
ACF and PACF of ARMA(p,q) Process
This shows that the ACF of an ARMA(1, 1) decays to zero exponentially

after lag 1, while there is an irregular component at lag 1 that is not
consistent with the exponential decay.
The derivation of the ACF of PACF of a general ARMA(p,q) process is

technically challenging and we will omit the details. However, recall that
for an AR process the ACF decays to 0 and the PACF cuts o to 0 after a
nite number of lags, whereas for an MA process the ACF cuts o to 0
after a nite number of lags and the PACF decays to 0. It should be clear
that since an ARMA process contains both an AR component and an MA
component, both the ACF and PACF of an ARMA process should decay to
0.

Examples of ACF and PACF of ARMA Process

Summary
Thus we can summarize the behavior of the ACF and PACF of

ARMA(p, q ) process as follows:
Model ACF PACF

AR (p ) Exponential decay Cuts-o after lag p
MA(q ) Cuts-o after lag q Exponential decay
ARMA(p, q ) Exponential decay Exponential decay

MA585 Time Series Analysis Lecture 2

Загружено:

Сведения о документе

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

MA585 Time Series Analysis Lecture 2

Загружено:

Авторское право:

Доступные форматы

Time Series Analysis: Model Building

Autoregressive Moving Average Processes

Ashis Gangopadhyay (Boston University) ARMA Models Spring 2016 1 / 49

where we assume 0 = 1 and j=0 2j < .

Ashis Gangopadhyay (Boston University) ARMA Models Spring 2016 2 / 49

Ashis Gangopadhyay (Boston University) ARMA Models Spring 2016 3 / 49

We now investigate if a linear process is stationary.

Ashis Gangopadhyay (Boston University) ARMA Models Spring 2016 4 / 49

the last step follows from the fact that

j(h)j = jCov (Xt , Xt +h )j

follows from the assumption on the parameters of the linear process.

Ashis Gangopadhyay (Boston University) ARMA Models Spring 2016 6 / 49

A backshift operator transforms an observation in a time series to the

Ashis Gangopadhyay (Boston University) ARMA Models Spring 2016 7 / 49

(Xt ) 1 (Xt 1 ) 2 (Xt 2 ) ... p (Xt p ) = et

where E (Xt ) = is a constant and et WN (0, 2 ).

Ashis Gangopadhyay (Boston University) ARMA Models Spring 2016 8 / 49

We will now discuss the su cient condition for stationarity of an AR (p )

Ashis Gangopadhyay (Boston University) ARMA Models Spring 2016 9 / 49

Thus Xt is a linear process given by

From the condition for stationarity of a linear process discussed earlier, it

However, the condition (2) holds if jj < 1. Thus an AR (1) process is

Ashis Gangopadhyay (Boston University) ARMA Models Spring 2016 10 / 49

Ashis Gangopadhyay (Boston University) ARMA Models Spring 2016 11 / 49

We will now discuss the condition for causality of an AR (p ) process.

is called the characteristic equation of the AR (1) process. In this

Now lets consider a zero mean AR (p ) process

Ashis Gangopadhyay (Boston University) ARMA Models Spring 2016 14 / 49

Is this process causal?

Ashis Gangopadhyay (Boston University) ARMA Models Spring 2016 15 / 49

Is the process causal?

Ashis Gangopadhyay (Boston University) ARMA Models Spring 2016 16 / 49

Xt 1.9Xt 1 + 1.1Xt 2 0.2Xt 3 = et

Is the process causal?

1 1.9B + 1.1B 2 0.2B 3 = 0

Ashis Gangopadhyay (Boston University) ARMA Models Spring 2016 17 / 49

Is the process causal?

Ashis Gangopadhyay (Boston University) ARMA Models Spring 2016 18 / 49

i.e, if the coe cients 1 and 2 belongs to the triangular region of R 2

Lets now consider the PACF of the process. Note

The ACF of AR(1) process decays to zero exponentially, whereas the

Ashis Gangopadhyay (Boston University) ARMA Models Spring 2016 22 / 49

since E (et +h Xt ) = 0, for h 1. Thus the autocovariance function is the

Next, substituting h = 2 in the dierence equation and dividing by (0),

We can continue in this manner substituting h = 3, 4, ...recursively solving

Ashis Gangopadhyay (Boston University) ARMA Models Spring 2016 25 / 49

Now lets consider the PACF of an AR (2) process. Note that

We will skip the derivation of ACF and PACF of the general AR (p )

Ashis Gangopadhyay (Boston University) ARMA Models Spring 2016 29 / 49

where E (Xt ) = and et WN (0, 2 ).

Is a MA(q ) process stationary?

Ashis Gangopadhyay (Boston University) ARMA Models Spring 2016 30 / 49

lie outside the unit circle.

Ashis Gangopadhyay (Boston University) ARMA Models Spring 2016 31 / 49

Is this process invertible?

Ashis Gangopadhyay (Boston University) ARMA Models Spring 2016 32 / 49

The characteristic equation is given by

with roots 0.833 and 1. Thus the process is not invertible.

Ashis Gangopadhyay (Boston University) ARMA Models Spring 2016 33 / 49

The characteristic equation is given by

Ashis Gangopadhyay (Boston University) ARMA Models Spring 2016 34 / 49

Ashis Gangopadhyay (Boston University) ARMA Models Spring 2016 35 / 49

First consider a MA(1) process

It is easy to check that the ACF of the process is given by