Markov Chain

Stochastic process (discrete time):

{X1, X2, ..., }
Markov chain
Consider a discrete time stochastic process with
discrete space. Xn {0, 1, 2, ...}.
Markovian property
P {Xn+1 = j | Xn = i, Xn1 = in1, ..., X0 = i0}
= P {Xn+1 = j | Xn = i} = Pi,j
Pi,j is the transition probability: the probability of
making a transition from i to j.
Transition probability matrix

P0,0 P0,1 P0,2

P1,0 P1,1 P1,2

.
.
.
.

.
.
.
P= .

Pi,0 Pi,1 Pi,2

..
..
..
..

Example
Suppose whether it will rain tomorrow depends on
past weather condition ony through whether it rains
today. Consider the stochastic process {Xn, n = 1, 2, ...}

0 rain on day n
Xn =
1 not rain on day n
P (Xn+1|Xn, Xn1, ..., X1) = P (Xn+1 | Xn)
State space {0, 1}.
Transition matrix:

P0,0 P0,1
P1,0 P1,1

P0,0 = P (tomorrow rain|today rain) = . Then P0,1 =

1 .
P1,0 = P (tomorrow rain|today not rain) = . Then
P1,1 = 1 .

Transition matrix:

1
1
2

Transforming into a Markov Chain

Suppose whether it will rain tomorrow depends on
whether it rained today and yesterday.
P (Xn+1|Xn, Xn1, ..., X1) = P (Xn+1|Xn, Xn1). The
process is not a first order Markov chain.
Define Yn:

1
Yn =
2

Xn
Xn
Xn
Xn

= 0, Xn1 = 0
= 0, Xn1 = 1
= 1, Xn1 = 0
= 1, Xn1 = 1

RR
NR
RN
NN

P (Yn+1|Yn, Yn1, ...)

= P (Xn+1, Xn|Xn, Xn1, ...)
= P (Xn+1, Xn|Xn, Xn1)
= P (Yn+1|Yn)
{Yn, n = 1, 2, ...} is a Markov chain.

P0,0 P0,1 P0,2 P0,3

P1,0 P1,1 P1,2 P1,3

P2,0 P2,1 P2,2 P2,3

P3,0 P3,1 P3,2 P3,3
3

 P0,1 = P (Yn+1 = 1|Yn = 0) = P (Xn+1 = 0, Xn =

1|Xn = 0, Xn1 = 0) = 0.
P0,3 = P (Yn+1 = 3|Yn = 0) = P (Xn+1 = 1, Xn =
1|Xn = 0, Xn1 = 0) = 0.
Similarly, P1,1 = P1,3 = 0, P2,0 = P2,2 = 0, P3,0 =
P3,2 = 0.
Transition matrix

P0,0 0 P0,2 0
P0,0 0 1 P0,0
0
P1,0 0 P1,2 0 P1,0 0 1 P1,0

0 P2,1 0 P2,3 0 P2,1

0
1 P2,1
0 P3,1 0 P3,3
0 P3,1
0
1 P3,1

The Markov chain is specified by P0,0, P1,0, P2,1, P3,1.

1. P0,0 = P (tomorrow will rain|today rain, yesterday rain).

2. P1,0 = P (tomorrow will rain|today rain, yesterday not rain).
3. P2,1 = P (tomorrow will rain|today not rain, yesterday rain).
4. P3,1 = P (tomorrow will rain|today not rain, yesterday not rain).

Chapman-Kolmogorov Equations
Transition after nth steps:
n
= P (Xn+m = j | Xm = i).
Pi,j

Chapman-Kolmogorov Equations:
n+m
Pi,j
=

n
m
Pi,k
Pk,j
,

k=0

n, m 0 for all i, j.

Proof (by Total probability formula):

n+m
Pi,j
= P (Xn+m = j|X0 = i)

X
=
P (Xn+m = j, Xn = k|X0 = i)

k=0

X
k=0

P (Xn = k|X0 = i)

P (Xn+m = j|Xn = k, X0 = i)

X
n
m
=
Pi,k
Pk,j
k=0

 n-step transition matrix:

n
n
n
P0,0 P0,1
P0,2
n
n
n
P1,0
P
P
1,1
1,2
.
(n)
.
.
P =
.n .n .n
Pi,0 Pi,1 Pi,2
..
..
..

Chapman-Kolmogorov Equations:
P(n+m) = P(n) P(m),

..

..

P(n) = Pn.

Weather example:

 

1
0.7 0.3
P=
=
1
0.4 0.6
Find P (rain on Tuesday | rain on Sunday) and
P (rain on Tuesday and rain on Wednesday | rain on Sunday).
Solution:

P(2)

2
P (rain on Tuesday | rain on Sunday) = P0,0

 

0.7 0.3
0.7 0.3
= PP=

0.4 0.6
0.4 0.6


0.61 0.39
=
0.52 0.48
P (rain on Tuesday | rain on Sunday) = 0.61

=
=
=
=
=

P (rain on Tuesday and rain on Wednesday | rain on Sunday)

P (Xn = 0, Xn+1 = 0 | Xn2 = 0)
P (Xn = 0|Xn2 = 0)P (Xn+1 = 0|Xn = 0, Xn2 = 0)
P (Xn = 0|Xn2 = 0)P (Xn+1 = 0|Xn = 0)
2
P0,0
P0,0
0.61 0.7 = 0.427

Classification of States
n
>
Accessible: State j is accessible from state i if Pi,j
0 for some n 0.

i j.
Equivalent to: P (ever enter j |start in i) > 0.
P (ever enter j |start in i)
= P (
n=0{Xn = j}|X0 = i)

P (Xn = j | X0 = i)
=

n=0

n
Pi,j

n=0

n
= 0 for all n, P (ever enter j |start in i) =
Hence if Pi,j
0. On the other hand,

P (ever enter j |start in i)

= P (
n=0 {Xn = j}|X0 = i)
P ({Xn = j}|X0 = i) for any n
n
= Pi,j
.
n
> 0 for some n, P (ever enter j |start in i)
If Pi,j
n
> 0.
Pi,j
Examples
8

 Communicate: State i and j communicate if they are

accessible from each other.
i j.
Properties:
0
= P (X0 = i|X0 = i) = 1. Any state i
1. Pi,i
communicates with itself.
2. If i j, then j i.
3. If i j and j k, then i k.
Proof:

n
n
>0
i j = Pi,j
> 0 and Pj,i

m
m
>0
j k = Pj,k
> 0 and Pk,j

X
n+m
m
=
Pi,k
Pi,ln Pl,k
Chapman-Kolmogorov Eq.
l=0
n
Pi,j

>
> 0

m
Pj,k

Similarly, we can show

k.

n +m
Pk,i

> 0. Hence i

 Class: Two states that communciate are said to be

in the same class. A class is a subset of states that
communicate with each other.
Different classes do NOT overlap.
Classes form a partition of states.
Irreducible: A Markov chain is irreducible if there is
only one class.
Consider the Markov chain with transition probability matrix:
1 1
2 2 0
P = 12 41 14
0 31 23
The MC is irreducible.
MC with transition probability matrix:

1 1
0 0
2 2
1 1 0 0
2 2

P=
1 1 1 1
4 4 4 4
0 0 0 1
Three classes: {0, 1}, {2}, {3}.

10

Recurrent and Transient States

fi: probability that starting in state i, the MC will ever
reenter state i.
Recurrent: If fi = 1, state i is recurrent.
A recurrent states will be visited infinitely many
times by the process starting from i.
Transient: If fi < 1, state i is transient.

Starting from i, the MC will be in state i for exactly n times (including the starting state) is
fin1(1 fi) , n = 1, 2, ...
This is a geometric distribution with parameter 1
fi. The expected number of times spent in state i
is 1/(1 fi).

11

 A state is recurrent if and only if the expected number

of time periods that the process is in state i, starting
from state i, is infinite.
Recurrent E(number of visits to i|X0 = i) =
Transient E(number of visits to i|X0 = i) <
Compute E(number of visits to i|X0 = i). Define

1 if Xn = i
In =
0 if Xn 6= i
P
Then the number of visits to i is n=0 In.
E(number of visits to i|X0 = i)

X
=
E(In|X0 = i)
=
=
=

n=0

n=0

n=0

P (In = 1|X0 = i)
P (Xn = i|X0 = i)
n
Pi,i

n=0

12

 Proposition 4.1: State

P i is nrecurrent if
, and transient if n=0 Pi,i < .

n
n=0 Pi,i

Corollary 4.2: If state i is recurrent and state i communicates with state j, then state j is recurrent.
Corollary 4.3: A finite state Markov chain cannot
have all transient states.
For any irreducible and finite-state Markov chain,
all states are recurrent.

13

 Consider a MC with

1 1
2 2

0
1
P=
0
0

0
0 0 0

1 0 0
1 0 0

The MC is irreducible and finite state, hence all states

are recurrent.
Consider a MC with

1
2
1
2

1
2
1
2

0 0 0

0 0 0

1 1

P=0 0 2 2 0

0 0 1 1 0
2 2
1
1 1
4 4 0 0 2

Three classes: {0, 1}, {2, 3}, {4}. State 0, 1, 2, 3 are

recurrent and state 4 is transient.

14

Random Walk
A Markov chain with state space i = 0, 1, 2, ....
Transition probability: Pi,i+1 = p = 1 Pi,i1.

At every step, move either 1 step forward or 1 step

backward.
Example: a gambler either wins a dollar or loses a
dollar at every game. Xn is the number of dollars he
has when starting the nth game.
ji
ji
For any i < j, Pi,j
= pji > 0, Pj,i
= (1 p)ji >
0. The MC is irreducible.

Hence, either all the states are transient or all the

states are recurrent.

15

 Under which condition are the states transient or recurrent?

Consider State 0.


recurrent
n
P0,0
=
finite transient
n=1

m
Only for even m, P0,0
> 0.


(2n)!
2n
2n
n
n
(p(1 p))n
P0,0 =
p (1 p) =
n
n!n!
n = 1, 2, 3, ...

By Stirlings approximation
n! n

2n
P0,0

e
2

n+1/2 n

(4p(1p))n

.
n

When p = 1/2, 4p(1 p) = 1.

X
(4p(1 p))n X 1

,
=
n
n
n=0
n=0

The summation diverges. Hence, all the states are

recurrent.
P (4p(1p))n

When p 6= 1/2, 4p(1 p) < 1.

n=0
n
converges. All the states are transient.
16

Limiting Probabilities
Weather example
P=

0.7 0.3
0.4 0.6




0.5749 0.4251
0.5668 0.4332


0.572 0.428
P(8) = P(4)P(4) =
0.570 0.430
P(4) = P4 =

P(4) and P(8) are close. The rows in P(8) are close.
Limiting probabilities?

17

n
= 0 whenever n is not
Period d: For state i, if Pi,i
divisible by d and d is the largest integer with this
property, d is the period of state i.

Period d is the greatest common divisor of all the

m
> 0.
m such that Pi,i
Aperiodic: State i is aperiodic if its period is 1.

Positive recurrent: If a state i is recurrent and the expected time until the process returns to state i is finite.
If i j and i is positive recurrent, then j is positive recurrent.
For a finite-state MC, a recurrent state is also positive recurrent.
A finite-state irreducible MC contains all positive
recurrent states.
Ergodic: A positive recurrent and aperiodic state is
an ergodic state.
A Markov chain is ergodic if all its states are ergodic.

18

 Theorem 4.1: For an irreducible ergodic Markov chain,

n
exists and is independent of i. Let j =
limn Pi,j
n
, j 0, then j is the unique nonnegative
limn Pi,j
solution of

X
j =
iPi,j j 0

i=0

j = 1 .

j=0

The Weather Example:



1
P=
1
The MC is irreducible and ergodic.
0 + 1 = 1
0 = 0P0,0 + 1P1,0 = 0 + 1
1 = 0P0,1 + 1P1,1 = 0(1 ) + 1(1 )
Solve the linear equations, 0 =

19

,
1+

1 =

1
.
1+

Gamblers Ruin Problem

At each play, a gambler either wins a unit with probability p or loses a unite with probability q = 1 p.
Suppose the gambler starts with i units, what is the
probability that the gamblers fortune will reach N
before reaching 0 (broke)?
Solution:

Let Xt be the number of units the gambler has at

time t. {Xt; t = 0, 1, 2, ...} is a Markov chain.
Transition probabilities:
P0,0 = PN,N = 1
Pi,i+1 = p, Pi,i1 = 1 p = q .

Denote the probability that starting from i units,

the gamblers fortune will reach N before reaching
0 by Pi, i = 0, 1, ..., N .
Condition on the result of the first game and apply
the total probability formula:
Pi = pPi+1 + qPi1 .

20

 Changing forms:
(p + q)Pi = pPi+1 + qPi1
q(Pi Pi1) = p(Pi+1 Pi)
q
Pi+1 Pi = (Pi Pi1)
p
Recursion:
P0 = 0
q
q
P2 P1 = (P1 P0) = P1
p
p
 2
q
q
P3 P2 = (P2 P1) =
P1
p
p
..
 i1
q
q
P1
Pi Pi1 = (Pi1 Pi2) =
p
p
Add up the equations:

q i1
q
Pi = P1[1 + + + ( ) ]
p
p

1( pq )i

P1 1 q if pq 6= 1
p
=
iP1
if q = 1
p

We know PN = 1
1( pq )N

P1 1 q if
p
PN =
N P1
if
21

q
p
q
p

6= 1
=1

 Hence,
P1 =

1 pq

1( pq )N

1/N

if p 6= 1/2

if p = 1/2

In summary,
Pi =
Note, if N ,
Pi =

1( pq )i

1( pq )N

i/N

if p 6= 1/2

if p = 1/2

1 ( pq )i if p > 1/2
0
if p 1/2

When p > 1/2, there is a positive probability that the

gambler will win infinitely many units.
When p 1/2, the gambler will surely go broke
(with probability 1) if not stop at a finite fortune (assuming the opponent is infinitely rich).

22