EE132B-HW Set #5 UCLA Winter 2018 Prof.

Izhak Rubin

Problem 1
Consider a Markov chain X = {Xk , k = 0, 1, . . . } with state space S = {a, b, c}
and the transition probability matrix as follows:
 1 1 1

 43 2 4
1
P =  4 0 4 .

1 2
3 3
0

Compute
h i
(a) The stationary distribution π = π(a) π(b) π(c) .

(b) P (X1 = b, X2 = c, X3 = a, X4 = c, X5 = a, X6 = c, X7 = b | X0 = c).

(c) P (Xk+2 = c | Xk = a).

(d) If the initial condition is given by

h i h i
2 1 2
π0 = π0 (a) π0 (b) π0 (c) = 5 5 5
,

calculate P (X1 = a, X2 = b, X3 = a).

Ans:

(a)
We calculate the stationary distribution by using the following equations:
X
π = πP ; π(i) = 1.
i∈S
h i
4 16 1
This set of equations yields π = 9 45 5
.

(b)
Let A0 = {X0 = c}, A1 = {X1 = b}, A2 = {X2 = c}, A3 = {X3 = a}, A4 = {X4 =
c}, A5 = {X5 = a}, A6 = {X6 = c}, and A7 = {X7 = b}. Then we have:
6
\ 7
Y k−1
\ 7
Y
P( Ai ) = P (Ak | Ai ) = P (Ak | Ak−1 )
i=1 k=1 i=1 k=1
= p(c, b)p(b, c)p(c, a)p(a, c)p(c, a)p(a, c)p(c, b)
2 1 1 1 1 1 2
= × × × × × ×
3 4 3 4 3 4 3
1
= .
1296

1
EE132B-HW Set #5 UCLA Winter 2018 Prof. Izhak Rubin

(c)
Due to the time-homogeneous property, we have
3
P (Xk+2 = c | Xk = a) = P (2) (a, c) =
X
P (a, k)p(k, c) = .
k∈S 16

(d)

P (X1 = a, X2 = b, X3 = a)
X
= P (X1 = a, X2 = b, X3 = a, X0 = n)
n∈S
X
= P (X3 = a | X2 = b)P (X2 = b | X1 = a)P (X1 = a | X0 = n)P (X0 = n)
n∈S
3 1 1 2 3 1 1 2
 
= × × × + × + ×
4 2 4 5 4 5 3 5
23
= .
160

Problem 2
Consider a Markov Chain with state space S = {0, 1} and the following transition
probability matrix
" #
 1−
P= .
1− 

For each of the following three cases, find the stationary distribution and the steady
state distribution.
(a) 0 <  < 1
(b)  = 0
(c)  = 1
Ans:

(a)
The stationary distribution [π(0), π(1)] can be found by solving

(
π(0) = π(0) + (1 − )π(1)
π(0) + π(1) = 1
1
We get the stationary distribution π(0) = π(1) = 2

2
EE132B-HW Set #5 UCLA Winter 2018 Prof. Izhak Rubin

To find the steady state distribution, we need to compute lim P(k) (i, j), the proba-
k→∞
bility that in k steps a process transitions from state i to state j. A general relationship
between P(k) (j) and P(k+1) (j) for j = [0, 1] is as follows:
P(k+1) (0) = P(k) (0) + P(k) (1)(1 − )





P(k+1) (1) = P(k) (0)(1 − ) + P(k) (1)
P(k+1) (0) + P(k+1) (1) = 1



P(k+1) (0) = P(k) (0) + (1 − P(k) (0))(1 − ) (1)

= P(k) (0)(2 − 1) + (1 − ) (2)
Expressing P(k+1) (0) in terms of the probability at initial state i = 0 P(0) (0),
P(k) (0) = P(0) (0)(2 − 1)k + (1 − )((2 − 1) + (2 − 1)2 + .....) (3)
As k → ∞ , P(0) (0)(2−1)k → 0, while the rest of the terms form the infinite sum and
1
is equal to (1 − ) 1−(2−1) = 12 . lim P(k) (0, 0) = 1/2. Similarly, lim P(k) (1, 0) = 1/2 ,
k→∞ k→∞
lim P(k) (0, 1) = 1/2 , lim P(k) (1, 1) = 1/2 .
k→∞ k→∞
1 1
Since lim P(k) (i, j) = 2
∀i, j, we conclude that π(0) = π(1) = 2
is also a steady
k→∞
state distribution.

(b)
For  = 1, the stationary distribution is also π(0) = π(1) = 12 as we have computed
(n)
in (a). But
" this # is not a steady state distribution since P
# distribution
" oscillates
0 1 1 0
between and . Therefore, the steady state distribution doesn’t exist.
1 0 0 1

(c)
The solution to
(
π(0) = π(0) + (1 − )π(1)
π(0) + π(1) = 1
is not unique for  = 1. Therefore, the stationary distribution doesn’t exist. This
implies that the steady state distribution doesn’t exist, either.

Problem 3
The probability that Coin A and Coin B come up with head are 0.6 and 0.5,
respectively. Alice repeats the following process: She flips one coin until a tail comes
up. The coin is put aside and she starts to flip another coin.
(a) Find the portion of flips that coin A is used.
(b) If Alice starts with Coin A, find the probability that Coin B is used on the 5th
flip?

3
EE132B-HW Set #5 UCLA Winter 2018 Prof. Izhak Rubin

(a)
Consider the Markov Chain X = {Xk , k = 1, 2, ...}, where Xk ∈ {A, B} is the
coin that Alice uses in the kth toss. The transition probability matrix is written as:
" #
0.6 0.4
P = .
0.5 0.5
The stationary distribution π = [π(a)π(b)] is found by solving
X
π = πP ; π(i) = 1.
i∈S

Hence, the portion of time that coin A is used is equal to π(a) = 95 .

(b)
It is equal to P (4) (1, 2) = 0.4444. Note that
" #
(4) 4 0.5556 0.4444
P =P = .
0.5555 0.4445

Problem 4
Bob has r umbrellas. He commutes between his home and office everyday. If it
is rainy when he leaves home/office, he takes one umbrella with him. Otherwise, he
doesn’t take any umbrella. The probability that it is rainy when he leaves home/office
is p, independent of the past. Let q = 1 − p
(a) Find the transition probability matrix of the Markov Chain, whose state is the
number of umbrellas that Bob has in his current location.
(b) Show that the stationary distribution is given by:

q
 r + q, i = 0



π(i) = 1
, i = 1, ..., r



r+q


(c) Find the fraction of time that Bob gets wet.

(b) If Bob has 3 umbrellas, find the value of p to maximize the probability that he
gets wet.

(a)





1 i = 0, j = r

p i > 0, j = r − i + 1


p(i, j) =


 q i > 0, j = r − i



0 o.w.

4
EE132B-HW Set #5 UCLA Winter 2018 Prof. Izhak Rubin

(b)
P
The solution must satisfy π = πP ; i∈S π(0) = 1. Check the following equality
holds for the provided solution:

π(0) = π(r)q
π(i) = π(r − i)q + π(r − i + 1)p, i = 1...r − 1
π(r) = π(0) + π(1)p

(c)
pq
It is equal to pπ(0) = r+q
.

(d)

d p(1 − p) (4 − p)(1 − 2p) + p(1 − p) p2 − 8p + 4

 
= == =0
dp 3 + (1 − p) (4 − p)2 (4 − p)2
√
We find p∗ = 8− 48
2
= 0.55