Math 182

Exercise 1
1. A Markov chain has transition matrix
1 2 1 6 1 3
1 2 0 1 2

3 4 1 4 0
a) Find 2=0 Pr{2 = | 0 = }.
1
b) Given the initial distribution 0 = 1 = 2 = . Find Pr{1 2 }.
3
c) Find the stationary probabilities of the Markov chain.
2. A Markov chain with states {0,1,2} has the transition probability matrix
34 14 0

2
1
0
3
3

14 14 12
Find
a) the probability that starting from state 1 the chain ends up in a different state
after two steps;
b) the stationary probabilities of the Markov chain.
3. On any given Saturday, Mr. Fit either goes biking or swimming but not both. If
he bikes on a particular Saturday, then he will swim the succeeding Saturday with
probability 0.8. On the other hand, if he swim on a particular Saturday then he is
as likely to swim the succeeding Saturday as he is to bike.
a) Specify the transition matrix.
b) Given that Mr. Fit biked today, a Saturday, what is the probability that he will
go swimming two Saturdays from today?
c) In the long run, what fraction of Saturdays is Mr. Fit expected go biking?
4. A taxicab driver moves between the airport A and two hotels B and C according
to the following rules. If he is at the airport, he will be at one of the two hotels
next with equal probability. If at a hotel then he returns to the airport with
probability 3/4 and goes to the other hotel with probability 1/4.
a) Find the transition matrix for the chain.
b) Suppose the driver begins at the airport at time 0. Find the probability for each
of his three possible locations at time 2 and the probability he is at hotel B at
time 3.

5. Let represent the outcome during the nth roll of a fair dice. Define to be
the maximum outcome obtained so far after the nth roll, i.e.,
= max{1 , , }.
a) Write down the transition matrix of the Markov chain.
b) Represent the transition matrix by a directed graph.
c) State whether each state is recurrent or transient.
6. A Markov chain has transition matrix
0 0.2 0.8 0
0.3 0.1 0 0.6

0.5 0
0 0.5

0
0
1
0
a) Draw the graph of this chain and find its communication classes.
b) Which state(s) is (are) recurrent and transient?
7. A Markov chain on states {0,1,2,3,4,5} has transition matrix
13 0 23 0 0 0

3
1
0 4 0 4 0 0
23 0 13 0 0 0

1
4
0
0
0
0

5
5
1 1 0 0 1 1
4
4
4 4
1
1
1
1
1
1
6 6 6 6 6 6
Find all communication classes.
8. An individual either drives his car or walks going from his home to his office in
the morning and from his office to his home in the afternoon. He uses the
following strategy: If it is raining in the morning, then he drives the car, provided
it is at home to be taken. Similarly, if it is raining in the afternoon and his car is at
the office, then he drives the car home. He walks on any morning or afternoon
that it is not raining or the car is not where he is. Assume that, independent of the
past, it rains during successive mornings and afternoons with constant probability
p. In the long run, on what fraction of days does our man walk in the rain?