Академический Документы
Профессиональный Документы
Культура Документы
! " ! !
! " ! !
! "
! !! !"
" "! ""
! "
! !! !"
" "! ""
#
!
!
!
!
!
!
!
!
!
"
#
$
$
$
$
$
$
$
$
%
&
=
' ' '
' ' '
' ' '
= = =
= = =
= = =
!
"
"! !"
!
"
" !"
!
"
" !"
!
"
"! "
!
"
" "
!
"
" "
!
"
"! "
!
"
" "
!
"
" "
# # # # # #
# # # # # #
# # # # # #
! !
"
!
!
!
"
!
" "
!
! "
!
!
!
" !
!
! !
#
! " ! !
!
1herefore, ls Lhe $!'
#*
elemenL of !
2
(lf you counL Lhe rsL row and column as 0)
( ) !
!"
#
( )
!
=
=
!
"
"# $" $#
% % %
!
"
Chapman-Kolmogorov
Equations
28
16.3 Chapman-ko|mogorov Lquanons
! So, lf (=1 Lhen
! And lf (=)-1, Lhen
! Whlch lndlcaLe:
! and ls Lhe $,'
th
elemenL of !
)
! and
( ) ( )
!
=
"
=
!
"
#
"$ %"
#
%$
& & &
!
"
( ) ( )
!
=
"
=
!
"
"#
$
%"
$
%#
& & &
!
"
( ) ( )
( )
1
1
n n
n n
n
n
!
!
= " = " =
# =
P P P P P P
P P
( ) !
"#
$
( )
n m
n m
+
= ! P P P
29
1he Weather Lxamp|e
! Cuesuon: lf you sLarL on a dry day ( $=0 ),
Lhe probablllLy LhaL lL wlll be dry ( '=0 ) 3
days laLer ls?
!
"
#
$
%
&
=
!
"
#
$
%
&
!
"
#
$
%
&
=
!" # $ %! # $
!& # $ %' # $
& # $ ' # $
! # $ " # $
& # $ ' # $
! # $ " # $
!
!
30
! When you do noL know Lhe sLaLe of Lhe Markov
chaln aL ume #=0 (we know Lhe probablllues of
belng ln dlerenL sLaLes, lnsLead)
! uene +
$
= probablllLy LhaL Lhe chaln ls ln sLaLe $ aL
Lhe ume 0, ln oLher words, ,("
0
=$) = +
$
" vecLor += [+
1
, +
2
,.+
-
] ls Lhe |n|na| probab|||ty
d|str|bunon for Lhe Markov chaln.
! robablllLy of belng ln sLaLe ' aL ume )
! Lxample?
Uncond|nona| State robab|||nes
( )
( ) !
"#
$ "
"
" !
% & # ' (
!
=
=
= =
!
31
16.4 C|ass|hcanon of States |n a Markov Cha|n
! A path from sLaLe i Lo sLaLe j ls a sequence of
Lransluons LhaL beglns ln i and ends ln j, such LhaL
each Lransluon ln Lhe sequence has a posluve
probablllLy of occurrlng.
! A sLaLe j ls access|b|e from sLaLe i lf Lhere ls a paLh
leadlng from i Lo j.
! ldenufy all posslble paLhs of
"
"
"
0 1 2 3 4
0 .4 .6 0 0 0
1 .5 .5 0 0 0
2 0 0 .3 .7 0
3 0 0 .5 .4 .1
4 0 0 0 .8 .2
P
! "
# $
# $
# $ =
# $
# $
# $
% &
! 1wo sLaLes i and j are sald Lo commun|cate lf j ls
reachable from i, and i ls reachable from j.
! Lxample:
! A seL of sLaLes ./ ln a Markov chaln ls ln a separaLe
c|ass lf no sLaLe ouLslde of ./ ls reachable from any
sLaLe ln ./.
Identify the classes in matrix P:
#
#
#
! lf all sLaLes ln Lhe chaln communlcaLe, Lhe Markov
chaln ls sald Lo be |rreduc|b|e
0 1 2 3 4
0 .4 .6 0 0 0
1 .5 .5 0 0 0
2 0 0 .3 .7 0
3 0 0 .5 .4 .1
4 0 0 0 .8 .2
P
! "
# $
# $
# $ =
# $
# $
# $
% &
! A sLaLe $ ls a trans|ent state lf Lhere exlsLs a sLaLe '
LhaL ls accesslble from $, buL Lhe sLaLe $ ls noL
accesslble from sLaLe '.
" 1here ls a chance LhaL Lhe process wlll never reLurn Lo
Lhls sLaLe agaln, upon enLerlng Lhe sLaLe.
! A sLaLe $ ls a recurrent state lf aL some fuLure
polnL Lhe sysLem wlll reLurn Lo sLaLe $
! noLe: a sLaLe ls elLher LranslenL or recurrenL
! A sLaLe , ls an absorb|ng state once Lhe sysLem
enLers sLaLe ', wlll never leave. 1hls means LhaL
!
''
=1.
"
34
Gamb|er's ku|n
! ldenufy Lhe sLaLes' Lype:
!
!
!
!
!
!
!
!
"
#
$
$
$
$
$
$
%
&
=
1 0 0 0 0
60 . 0 0 40 . 0 0 0
0 60 . 0 0 40 . 0 0
0 0 60 . 0 0 40 . 0
0 0 0 0 1
4 $
3 $
2 $
1 $
0 $
4 $ 3 $ 2 $ 1 $ 0 $
P
33
1rans|ent State Lxamp|e
! ldenufy Lype of sLaLes
!
!
!
!
!
"
#
$
$
$
%
&
=
!" # $ %" # $ $
"$ # $ "$ # $ $
&$ # $ '$ # $ ($ # $
!
&
$
!
" $ %
36
More dehn|nons: er|od|c|ty
! A recurrent sLaLe $ ls per|od|c wlLh perlod 0 > 1 lf 0 ls Lhe
smallesL number such LhaL all paLhs leadlng from sLaLe $ back
Lo sLaLe $ have a lengLh LhaL ls a muluple of 0.
! l.e: for all values of n oLher Lhan k, 2k, 3k, (k>1 ls
smallesL lnLeger wlLh Lhls properLy).
! lf a recurrenL sLaLe ls noL perlodlc, lL ls called aper|od|c.
!
!
!
"
#
$
$
$
%
&
=
0 0 1
1 0 0
0 1 0
P
!
!
!
"
#
$
$
$
%
&
=
!" # $ %" # $ $
"$ # $ "$ # $ $
&$ # $ '$ # $ ($ # $
!
&
$
!
( )
0
n
ii
p =
37
Lrgod|c Markov Cha|n
! A nlLe-sLaLe lrreduclble Markov Chaln where
all sLaLes are recurrenL and aperlodlc ls sald Lo
be ergod|c
!
!
!
"
#
$
$
$
%
&
=
Rain Dry
Rain
Dry
P
40 . 60 .
20 . 80 .
!
!
!
!
!
!
"
#
$
$
$
$
$
$
%
&
=
1 0 0 0 0
60 . 0 0 40 . 0 0 0
0 60 . 0 0 40 . 0 0
0 0 60 . 0 0 40 . 0
0 0 0 0 1
4 $
3 $
2 $
1 $
0 $
4 $ 3 $ 2 $ 1 $ 0 $
P
38
16.S Long kun ropernes of Markov Cha|ns
! Steady-state probab|||nes are used Lo descrlbe
Lhe long-run behavlor of a Markov chaln
! ln some far o fuLure ume, whaL ls Lhe
probablllLy LhaL lf l observe Lhe sysLem l wlll
nd lL ln sLaLe '?
! ln oLher words, nd for a very large )
! &'() *+ )', -'(.(-),.*+)*- /0 )',+,
1./2(2*3*)*,+4
( ) !
"#
$
39
Steady-State robab|||nes
! 1heorem: LeL " be Lhe Lransluon maLrlx for an
-*% sLaLe Lrgodlc chaln. 1hen Lhere exlsLs a
vecLor
such LhaL (long-Lerm fuLure sLaLe ls lndependenL of currenL sLaLe)
lim
n
n
P
!"
=
[ ]
0 1 M
! ! ! = ! !
0 1
0 1
0 1
M
M
M
! ! !
! ! !
! ! !
" #
$ %
$ %
$ %
$ %
$ %
& '
!
!
" " "
!
! 1hls Lheorem Lells us LhaL for any lnlual sLaLe $,
! 1he vecLor " = [1
2
1
3
. 1
%
] ls oen called Lhe steady-
state d|str|bunon, or equ|||br|um d|str|bunon, for Lhe
Markov chaln.
! Lach 1
'
represenLs Lhe long-run probablllLy of arrlvlng
aL sLaLe ' ln perlod 8lC ), regardless of where you
sLarLed!
! 1he behavlor of a Markov chaln before Lhe sLeady
sLaLe ls reached ls oen called trans|ent behav|or
( )
!
"
#!
"
$ ! =
" #
!"#
41
8ack to the Weather Lxamp|e
3
0.8 0.2 0.76 0.24 0.752 0.248
0.6 0.4 0.72 0.28 0.744 0.256
! " ! " ! "
= =
# $ # $ # $
% & % & % &
P
4
0.752 0.248 0.80 0.20
0.744 0.256 0.60 0.40
! " ! "
=
# $ # $
% & % &
P
5
0.75 0.25 0.80 0.20
0.749 0.251 0.60 0.40
! " ! "
=
# $ # $
% & % &
P
6
0.75 0.25 0.80 0.20
0.75 0.25 0.60 0.40
! " ! "
= =
# $ # $
% & % &
P
!
"
#
$
%
&
=
!
"
#
$
%
&
!
"
#
$
%
&
=
!" # $ %! # $
!& # $ %' # $
& # $ ' # $
! # $ " # $
& # $ ' # $
! # $ " # $
!
!
42
I|nd|ng the Steady State robab|||nes
! 8ack Lo Lhe Chapman-kolmogorov Lquauons:
! 1herefore, when ) geLs really large( )
! Slnce we have Lo be ln some sLaLe aL any polnL
ln ume, ln long-Lerm, we have
( ) ( ) 1
0
, 0,... ; ,
M
n n
ij ik kj
k
p i j p M p
!
=
= =
"
( )
!
"
#!
"
$ ! =
" #
!"#
43
8ack to the Weather Lxamp|e
!
"
=
!
=
!
"
"
" and
State 0 = Dry and State 1 = Rain
1
3
2
0.8 0.2
0.6 0.4
! "
=
# $
% &
P
0
0,...,
M
j i ij
i
p
j M
! !
=
=
=
"
44
3
2
43
Monopo|y Lxamp|e
! Assumlng LhaL lf you land ln [all, you have Lo
sLay ln [all for 3 Lurns or unul you roll doubles,
Lhe long Lerm probablllues of each space on
Lhe Monopoly board has been calculaLed
! Ash and 8lshop (1972)
! hup://www.bewersdor-onllne.de/amonopoly/
1 MedlLerranean Ave 0.0237
9 ConnecucuL Ave 0.0237
3 8aluc Ave 0.0241
37 ark lace 0.0243
6 CrlenLal Ave 0.0233
8 vermonL Ave 0.0238
13 SLaLe Ave 0.0238
34 ennsylvanla Ave 0.0279
14 vlrglnla Ave 0.0288
29 Marvln Cardens 0.0289
32 norLh Carollna Ave 0.0294
39 8oardwalk 0.0293
27 venLnor Ave 0.0299
31 aclc Ave 0.0300
26 ALlanuc Ave 0.0301
11 SL Charles lace 0.0304
23 lndlana Ave 0.0303
21 kenLucky Ave 0.0310
16 SL !ames lace 0.0318
19 new ?ork Ave 0.0334
18 1ennessee Ave 0.0333
24 llllnols Ave 0.0333
47
48
1wo more th|ngs
! lf l and [ are recurrenL sLaLes from dlerenL
classes:
! lf a sLaLe ' ls LranslenL, Lhen
! 1he probablllLy LhaL aL some fuLure ume, ), you
are leavlng sLaLe ' ls equal Lo Lhe probablllLy
LhaL aL some fuLure ume you are enLerlng sLaLe
'!
( )
! "#$ =
! "
!
"#
!
$
( )
0 for all
n
ij
p n =
49
Lxpected Cost]keward
! lf l know Lhe long run probablllues, l can
calculaLe long run expecLed average cosL (or
reward) per unlL ume:
45'6 = cosL (reward) for belng ln sLaLe j=0,,M
"
#
= sLaLe of Lhe sysLem aL ume L
average cost = ( ) ( ) ( )
1 1
1
lim lim Pr
n n
t t
n n
t t
E C X X j C j
n
!" !"
= =
# $ # $
= = =
% & % &
' ( ' (
) )
( )
0
M
j
j
C j !
=
"
( ) ( )
1
lim Pr
n
t
n
t
X j C j
!"
=
= = # $
% &
' ( ) ( )
1
lim Pr
n
t
n
t
X j C j
!"
=
# $
= =
% &
'
30
Lxamp|e
! WeaLher Lxample: Assume LhaL on average
sales aL an ouLdoor markeL are $300 on a
ralny day, and $2000 when lL doesn'L raln.
WhaL's Lhe average sales level per day? lf Lhe
markeL runs for 30 days nexL year, whaL are
Lhe LoLal expecLed sales?
0 1
0.25, 0.75 ! ! = =
31
Lxpected Cost]keward for Comp|ex
rob|ems
! WhaL lf Lhe cosL depends noL only on Lhe
sLaLe, buL some oLher varlable?
" Cn ralny days, sales are $300 lf no crulse shlp ls ln
porL, $1300 lf Lhere ls a crulse shlp ln porL
" Cn dry days, sales are expecLed Lo be $2000 lf no
crulse shlp ls ln porL, $4000 lf Lhere ls one
! 1he probablllLy LhaL on any random day Lhere
ls a crulse shlp ln porL ls 0.30
32
More comp|ex average costs
5
#
: a random varlable lndlcaung presence of a
crulse shlp
llnd Lhe values for 0(') rsL, Lhen compuLe Lhe
average cosL per unlL ume
( ) ( )
0 0
Expected Sale (daily) = ,
M M
j t t j
j j
E C X j Y k j ! !
= =
= = " #
$ %
& &
33
8ack to the prob|em
x
L
:
C(x
L
,?
L
): 8aln ury
?
L
: no-Shlp 300 2000
Shlp 1300 4000