Академический Документы
Профессиональный Документы
Культура Документы
STOCHASTIC PROCESS
Introduction to
STOCHASTIC PROCESS
A.K. Basu
0
Alpha Science International Ltd.
Harrow, U.K.
Professor A.K. Basu
Editor of Stochastic Modelling and Applications
Department of Statistics, Calcutta University
Kolkata-700 019, India
www.alphasci.com
ISBN 978-1-84265-105-6
Printed in India
Preface
A.K. B asu
Abbreviations and Notations
F*(s) f e~stclF
(x)L.T. of r.v. X or d.f.
Jo
[ x m { x ,} Stochastic process in continuous time
C(.,; C(.s, 0 Covariance function between X(t) and X(s)
Pij Transition probability from state i to j
m Number of occurrances of an event by time t (upto epoch /)
C(z) Autocavariance generating function of {X,}
P(k)\ pk Correlation function of X, at lag k
P Utilization factor
A => B A implies B
Contents
Preface v
Abbreviations and Notations Vll
1. Introduction 1
1.1 Notion of Stochastic Processes 1
1.2 Different Types of Stochastic Processes 2
1.3 An Introduction to Stationary Processes 2
Exercises and Complements 5
3. Random Walks 51
3.1 Different Types of Random Walks 51
3.2 Sequential Analysis 58
3.3 Wald’s Equation and Wald’s Identity 59
3.4 Fluctuation Theory 62
Exercises and Complements 62
4. Renewal Theory 65
4.1 Introduction 65
4.2 Renewal Equation 66
4.3 Renewal Theorems 70
4.4 Central Limit Theorem for Renewal Theory 74
4.5 Delayed and Equilibrium Renewal Processes 75
4.6 Residual and Excess Life Times 78
4.7 Renewal Reward Process 80
4.8 Replacement Policies Connected with Renewal Theory 82
Exercises and Complements 84
x Contents
5. Branching Process 87
5.1 Introduction and History of Branching Process 87
5.2 Properties of Generating Functions 88
5.3 Probability of Extinction 91
5.4 Distribution of Total Number of Progeny 96
5.5 Continuous Parameter (Markov) Branching Process 98
Exercises and Complements 105
Appendices
I. Sample Function of Brownian Motion 259
II. Second Order Stochastic Analysis 27/
III. Stochastic Integral (Ito Integral) 287
IV. Some Important Martingale Theorems 299
Definition
A stochastic process is a family of random variables {Xt}, where t takes values
in the index set T (sometimes called a parameter set or a time set).
The values of Xt are called the state space and will be denoted by S.
If T is countable then the stochastic process is called a stochastic sequence (or
discrete parameter stochastic process). If S is countable then the stochastic
process is called a discrete state {space) process.
If S is a subset of the real line the stochastic process is called a real valued
process.
If T takes continuously uncountable number
of values like (0, °o) or (-°o, «?) the stochastic
process is called a continuous time process. To
emphasize its dependence on t and sample point
w, we shall denote the stochastic process by
X(t, w), te 7\ w e Q. i.e. for each we Q, Xt = X(r,
w) is a function of t.
This graph is known as the “typical sample function” or “realization of the
stochastic process” X (t, w).
2 Introduction to Stochastic Process
i.e. E[Xtn+l \Xtn ,. . . , XtQ] = Xtn a.s. for all choices of the partition (1.1), then
{Xt, t E T) is called a Martingale process.
6. Stationary process
If the joint distribution of (X,1+th, . . . , Xt +h) are the same for all h > 0 and
function Cs f = E(XtXs) depends only on the difference | t-s | for all r, s e T. Note
that in our definition we have taken a zero mean stochastic process.
Ergodicity
The behavior in which sample averages formed from a process converge to some
underlying parameter of the process is termed ergodic. To make inference about
the underlying laws governing an ergodic process, one need not observe separate
independent replications of entire processes or sample paths. Instead, one need
only observe a single realization of the process, but over a sufficiently long span
of time. Thus, it is an important practical problem to determine conditions that
lead to a stationary process being ergodic. The theory of stationary processes has
a prime goal the clarification of ergodic behavior and the prediction problem for
processes falling in the wide range of extremeties.
In covariance stationary process usually the added condition that E(Xr) does
not depend on t is imposed. But it should be noted that in order for a stochastic
process with E{X}) < °° to be covariance stationary it is not necessary that
its mean function m(t) = E(Xt) be a constant. Consider the example: X(t) =
cos + T(r), where Y(t) = N(t + L) - N(t), {N(t), t > 0} be a Poisson process
Example 1 (Gaussian process). If the joint distribution of (X lx, . . . , Xtn) for all
t.\, ... ,tnE T is multivariate normal, then {Xt, t E T] is a Gaussian process. Such
type of process appears in e.g. physics and atmospheric turbulence theory. If a
Gaussian process is wide sense stationary then it is strictly stationary. Now the
multivariate normal distribution of (X t],. . . , X,^) is completely determined by
its mean vector (EXtx, . . . , EXtn) and covarience-matrix (Ctj ti) = (EXtj Xt j )
/, j = I, 2, , n. If a Gaussian process is covariance stationary then
E(X~ ) < oo, E(X~. ) < oo and Ctj tj = Cf/ is a function of | t, - t/ |.
Example 2 Let [Xlv n > 1} be uncorrelated random variables with mean 0 and
variance 1.
Then C„.m = Cov (X„ X J = E(X„ X J
0 if n * m
1 if n = m
So {Xn, n> 1} is covariance stationary.
If moreover, Xns are normally identically distributed then {Xn, n > 1} is strictly
stationary.
A process which is not stationary is called evolutionary.
Example 3 Poisson process
P[Xt = k\ - exp (-Xt)(Xt)kl k\, X > 0, k = 0, 1, . . . with the mean function
m{t) = Xt, Var(X,) = Xt are functions of t and is evolutionary. Also note that
Cov(X,, Xs) = min (r, s) in not a function of {t - s) only.
Example 4 Let X(t) = A cos (wt) + B sin(wr), t > 0, where A and B are uncorrelated
random variables each with mean 0 and variance 1 and w > 0 (constant). Here
EX(t) = 0,
E(X(t) ■X(s)) = cos (wf) • cos (ws) + sin (ws) • sin (wr)
= cos ((s - t) • w), and E{X\t)) = 1.
Therefore the process is covariance stationary.
N I—
Example 5 Let Zt = Z Ak el6kt, / = V -l , where 02, . . . , 0N are real
constants, Ak's are uncorrelated with mean 0 and Var (A,-) = a ] . Here E(Zt) =
m(t) = 0 and
= S 2 E(Aj Ak ) ei{0kS~6jt)
7=1k=1
yV
= Z crA
2 eiGk {s~n
k=1
Introduction 5
N
Var ( Z t ) = X err.
k= 1
Therefore, the process is stationary in the wide-sense.
Example 6 Let Xn = X a* £n_k, where £,’s are uncorrelated with mean zero
k=0
and variance a 2 < and | a \ < 1.
= if | a | < l.
1- Cl“
® i ’ L “ 11 J • • • 0 ( 0/1 ’ K - 1n- 1) •
6 Introduction to Stochastic Process
Exercise 1.5 Prove that every continuous parameter Stochastic process with independent
increments is a Markov process.
Exercise 1.6 Let T be a nonnegative discrete random variable. Prove that T has a
geometric distribution iff
P[T > x + y\T > a] - P[T > y] for all integers a, y > 0.
Exercise 1.7 Let T be a non-negative continuous random variable. Prove that T has an
exponential distribution iff P[T > x + y|7’ > x] = P[T > y].
Exercise 1.8 Let 7' be a nonnegative random variable such that X(T) is a stochastic
process and for a fixed value of T, say /, X(t) has a gamma density
P + k fp + k- r
T= p p (1 - p ) ,kp > 0
a V k ,
1 >p= 1 - q > 0 , k = 0,1,2___
then X{T) also has a gamma density. Derive its parameters,
(c) Prove that if X(T) has a gamma distribution, i.e.
E[e -vviT) 1_ where ft > 0, A > fj. > 0, then conversely, T has a negative
n+ s
binomial distribution. Determine its parameters.
Exercise 1.9 The random variables X and Y have the following properties: X is positive
i.e. P(X > 0) = 1, with continuous density function/(x) and Y \ X has a uniform distribution
on [0, X].
Prove that if Y and X - Y are independently distributed, then
/ (a ) = a2xe~a\ x > 0, a > 0.
Exercise 1.10 Let U and Vbe independent A^O, 1) r.vs. Let T be the circumference of
the unit circle T = [0, 2k] and Y(t) = U sin t + V cos tZ{t) = - U cos / + V sin /. Then show
that X{t) = (Y(t), Z(/)), t e T is a strictly stationary process.
Exercise 1.11 Let X(t) = A cos A t + B sin A/, where A and B are uncorrelated random
variables with common mean zero and common variance o-2. Decide whether X(t) is
covariance stationary.
Exercise 1.12 Let {X,} and {Y,} be two sequences of r.vs. on a probability space with
E(Xj) = E{Yj) = 0, Var(X,) = Var(T,) = cr2 < E(X,Xj) = E(Yft) = E(X,Yj) = 0 for all i *
j and EX\Yx = 0 for all /. Let {Z(/), t > 0} be a stochastic process defined by
n
Z(t) = S {Xj cos (Ay /) + Yj sin (Ay f)}
Determine its covariance function. Is the process second order stationary? Is the
process strictly stationary? Justify your answer.
Introduction 7
Exercise 1.13 Let X(/) = sin wt, where w is uniformly distributed on [0, 2k ] (i) Show
that {X(/), / = 1, 2,. . .) is covarience stationary but is not strictly stationary (ii) show that
(X(/), 1 > 0} is neither covariance stationary nor strictly stationary.
Exercise 1.14 Let Xn = cos n U (n > 1), where U is uniformly distributed on \-n, n).
Show that {X,,} is covariance stationary but not strictly stationary.
Exercise 1.15 (Random Binary Noise) A stochastic process X(t) is called random binary
noise if X(t) = Y(t - u), where Y is a process taking value ± 1 with probability 1/2 on
successive intervals of fixed length T and such that the r.v.’s Y(t) are independent for
values of t lying in non overlapping intervals and U is the random shift of time such that
U is uniformly distributed on (0, T) and is independent of Y(t). Show that X{t) is wide
sense stationary.
Exercise 1.16 (Random Telegraph Signal Process)
Let X(t) be a stochastic process defined by X(t) = Y(-\)N(t\ t > 0 where N(t), t > 0 is a
Poisson process with parameter Aand Y is a r.v. independent of the process N{t), i > 0 with
probabilities specified by P(Y = ± 1) = 1/2. Show that X(t), t > 0 is a covariance stationary
process.
Exercise 1.17 (Model of a Germination Process)
Let 7 / T2, . . . , Tn be the germination times of N seeds planted at time t = 0 and they are
assumed to be i.i.d. with common unknown d.f. H(t) = P(T> t) with H{0) = 0 and //(°°)
= 1 - p, where 0 < p < 1 be the probability that a seed may fail to germinate. Also assume
that the average proportion of germination in a small interval (S, S + AS) is approximately
A > 0. Find the probability distribution of Xt, the number of seeds that will germinate in
(0, /).
Exercise 1.18 (FKG inequality)
Let Xh X2, • • • , Xn be i.i.d. r.v.’s having Bemouli distribution with parameter p. Let/be
increasing such that /(x) < / ( y) whenever < y, for each i. Let x = (xu . . . xn).
K € zS Pk = L
Proof
P[Xq = i0>X\ —i» • • • »Xn = in]
- P[Xn = in I Xn-\ - Xn-2 - in-2» • • • ♦X ] = i‘i . . . X0 = i0]
P\Xn- 1 - *n-b Xn-2 = in-2* • . • , X\ = i\, Xq = io\
—P\Xn —in| Xjj.j — —f/2-b • • •» *o —i0]
= P i n-\in P i n- j i n-\ ^ X n-2 ~ in-2> • • • » ^0 = *()]
= /Vi/„ Pin- 2in-\ • • •P1V2 Pio*i (by induction).
Definition 2.1 A vector u = (ui, u2, . • • , un) is called a probability vector if the
components are non-negative and their sum is one.
Discrete Time Markov Chain 9
' 0 1 0
1/2 1/6 1/3 is a stochastic matrix.
vl/3 2/3 0
A transition matrix P of a M.C. is a
Definition 2.3 A stochastic matrix A is said to be regular if all the entries of
some power Am is positive (m is a positive integer), e.g.
r 1/2 1/2")
A= ff i °l l is regular, since A2 =
,1/4
) 1/4,
Problems
1. Suppose P is a stochastic matrix, then show that Pn is also a stochastic
matrix for all n> 1.
2. If P n is stochastic, is P stochastic?
3. Show that 1 is an eigenvalue if A is a stochastic matrix, i.e.
| A / - A | = 0=> A= 1.
Consider a sequence of trials with possible outcomes E u E * . . . , Ek . . . To
the pairs of outcomes (Ep Ek) we can associate some numbers (i.e. conditional
probabilities) p]k. The {Ek} are referred to as the possible states of the system.
Instead of saying that the nth trial results in Ek one says that the nth step leads
to Ek or that Ek is entered at the nth step.
We shall denote by P j ^ the probability of transition from Ej to Ek in exactly
n steps i.e. the conditional probability of entering Ek at the nth step from Ey This
is the sum of all the probabilities of all possible paths Ej —> Ejx—» . . . Ejnl —» Ek
of length n starting at Ej. and ending at _Ek
In particular, p ^ =p jk.
Theorem 2.2 (Chapman-Kolmogorov equation)
„(*)_ V n{n~X)n
P ‘J ~ kes Plk Pkj
Proof P<p=P[Xa = j \ X 0 = i]
= 2 P[Xn =J I = k, X0 =—k |
k ES
= kes
2 <v )P//,-
* | ) = kes
2 KJ ik
PkjP t l
Corollary For
10 Introduction to Stochastic Process
Note Actually P-"} are the elements of Pn, where P = (ptj) and the corollary
express the matrix multiplication rule Pm+n - PmPn.
To be consistant we define
p.^0) = P[X() =j \ X0 = i] = Sij (Kronecker’s delta).
Transition probabilities of a non-homogeneous (non-stationary) M.C. {X„} also
satisfies Chapman-Kolmogorov equation given by
_ p(m,m+\)p(m+l,m+2) p {n -l,n )
= 2 P(Xn = j | = k) P(X„_, = 11 x m = i )
- T n (w,n-l)
kes Pik Pk j
qp i for7 = 0, 1, 2,. . . , i + n - 1
Here PJ for; =7 + n
0 otherwise
P if j = i + l
PiJ = P(Xn = j \ X n_x = i) = q if; = 0
0 otherwise.
f\ 0 0 0 ... 0 0 0
q 0 p 0 ... 0 0 0
0 <7 0 p ... 0 0 0
0 0 0 0 ... q 0 p
^0 0 0 0 ... 0 0 1
From each of the ‘interior’ states Ex, E2, . . . , Ek_x transitions are possible to
the right and left neighbour with pii+x = p, pLi_x - q. However, no transition is
possible from either E0 or Ek to any other state.
The system may move from one state to another, but once E0 or Ek is reached
the system stays there permanently.
q p 0 0 ... 0 0
q 0 p 0 ... 0 0 0
0 q 0 p ... 0 0 0
0 0 0 0 ... q 0 p
0 0 0 0 ... 0 q pj
all his money his adversary returns it, so that the game can continue forever i.e.
a random walk in a finite interval such that whenever the particle is at point 1 it
has probability p of moving to position 2 and probability q to stay. The system
is in state Ek, if the capitals are k and R - k + 1 respectively.
Taking / = m + n,
i —» k and similarly k —» i => / <-» k.
By Lemma 2.1, i.e. 3 = u (C(/), where C(i) = {j e 3 | i <-» y} is called a
communicating class, i.e. the class of essential states is partitioned into disjoint
equivalent classes (communicating classes).
Definition 2.5
A Markov chain is called irreducible (or ergodic) if there is only one communicating
class, i.e. if all states communicate with each other or every state can be reached
from every other state.
Definition 2.6
A subset C of S is called closed (or transient) if it is impossible to leave C in one
step i.e. p^ = 0 for all i e C and all j £ C.
Discrete Time Markov Chain 13
By induction p - ^ = 0 if i e C and j £ C.
<ii) i = ( v M= ~ p :;" * ^ c
Hence, the communicating classes are {5}, {3, 8}, {1, 4, 9}, {2} and {6, 7}.
Example 2.2 A man tosses a fair coin until 3 heads occur in a row. Let Xn
denote the largest string of heads ending at the nth trial.
Then with positive probabilities Xn = 0,1, 2, 3. This is a Markov chain with
state space 0, 1, 2, 3.
The transition matrix is given by
1/2 1/2 0
0 1
1/2 0 1/2 0
1/2 0 0 1/2
0 0 0 1,
3 is an absorbing state. Therefore, M.C. is not irreducible. The communicating
classes are {0, 1, 2} and {3}.
Proof p O0 =P[Xn = j \ X 0 = i]
V [X
P n= j x m= j , XmA ± j .X, * y |X0 = i]
m=1 A Bm c
=i P[ABm\C]
m= 1
p (n) v P(ABmC)P(BmC
)
Hence
" «-i P(C)P(BmC)
= 2 F(i4 | BmQP(Bm | C)
m= l
= m
i P(Xn = y | X m= y ) 4 m)
n
_ y D(n~m) fin)
m=\ FJl
^ oo \
_ X . V n(n~m) „n~m Am) m
- °>j + n% Pa s hj s
V /
Write fn = E / (n) = the probability that starting from i the system will ever
■ n- 1 •'
pass through j. Hence, 0 <f tj < 1.
Definition 2.8 A state i e 5 of a Markov chain is said to be transient if f u < 1
and recurrent (Persistent) iff u = 1. If f l} = 1, then {/^n)} is a proper probability
distribution and we shall refer to it as the first passage time distribution of j
1 n
lim — X ak = L.
z-+\^ n k=o
For proofs of 2 and 3 see the book of Karlin (page 46, Introduction of Stochastic
process) and proofs of 1,4 and 5 see the book of Tichmarsh-Theory of Functions.
Let n - 0, 1, 2, . . . .
1. (Lebesgue) Dominated Convergence Theorem
If (i) lim anm exists for every m
n—
(ii) | anm | < bm (independent of n) for all m > 0
(iii) m=0
£ bm
< °°
3. Fubinis Theorem
In order that X X = X X
n=0 m=0 m=0 n=0
it is sufficient that at least one of the following conditions is satisfied:
(i) anm > 0 for all n. m
(ii) X X | a „ < oo
n=0 m=0 1
(iii) X X < oo
m - 0 n=0
1 - Fu(s)
Also F'f 1) = lim —-— -— (if i is recurrent, Fu{1) =f it = 1)
Proof Suppose i e S and j <-> /, i.e. p \ p > 0, for some N and p^f*) > 0 for some
1. Let i be transient
By repeated applications of Chapman-Kolmogorov equation
P\ r +M^ p \ P p ^ p T = e p ^ \ e > 0
(N+n+M) > 0 £
I p<w+"+M) > I /><"> and 00 > s /4 n) > 2
n= 0 n=0r j j n=0r " 0 “ n=0
( N+n+M)
> 0 I p[n) = 00 (/ is recurrent by Theorem 2.5 part (ii))
Jo'7"
3. Let i be recurrent
We had proved earlier
p (N+M+n) > 0 p (n)
p(N+M+n)> e p (n)
and
Also I
' n+m ^ P)nj)sn- n=0
SN+M p ]"+M+n)sn>e Z p[n)s n for 0 < « < 1.
JJ n=0r "
where P;j is) is the G.F. of [p]]') and Pjj( s ) is the G.F. of
-4 -3-2-1 0 1 2 3 4
0 if n is odd
pin) _ r2m\
r 00 “ p mq m if n is even (n = 2m)
20 Introduction to Stochastic Process
p(2n) _ 1 f 2") 2 n\
r 00 ~ (1/3) In (2.4)
10 <i+j<n _i\j\ ( n- i
u
f 2 n\ n\
<C (1/3)" where Cn = max (2.5)
" 2 2" 0 <j<n i \ j\ (n - i - j )!
since I -t-77----------77(1/3)"= 1
0 <i+j<n ll J\ (tl - I - J)\
We shall show that for large n the value of Cn is attained for i = j ~ n/ 3. Let
■ • n■ i •
io, io maximize /------:----sub ect to 0 < / + / < « .
i!j! ( n - i - j ) !
Discrete Time Markov Chain 21
Then
n\ n\
j o '■Oo - 1)! ( n - j 0 - ‘o + 1)! " j o ! ! (« -7o - <o)!’
n! n!
j o ! Oo + 1)! (n - Jo - io ~1)! " i o ! h ! (« - Jo - <o)!
P
/l! /i!
and
(Jo + !)! *o! (« - Jo - io ~ 1)! jo'-*o'-(n ~ Jo ~ io)'-
which lead to n - j 0 - 1 < 2/0 < n - j 0 + 1 and n - i0 - 1 < 2j0 < n - /0 + 1-
Hence for large n, /0 ~ rc/3 and y0 ~ n/3.
Putting i - j - n/3 in (2.5), we obtain from (2.4)
p(2n) , n\ ( 2n\
r00
(n/3)\(n/3)'.(n/3)\22n3n \ n J
3V3
i,l n„ 3/2
2 n 3/2
(using Stirling’s formula for factorials).
Thus, Z ^ P0(02n) < oo. Hence, ‘0’ is a transient state. Since the M.C. is
irreducible, the Random walk is always transient.
Example. 2.3 (Success Chains or Runs)
Consider a M.C. with possible states Zs0, E2, • • • with transition probability
matrix
r Po <7o 0
Pi 0 <7i 0
Pi 0 0 <72 0
V •• • • V
Let k represent the ‘age’ of the system. When the system reaches k the aging
process continues with probability qh but with probability p k it rejuvenates and
starts afresh with age zero. The successive passages through the state E0 again
represent a recurren event and the probability that a recurrence time equal k is
again given by the product q^q\q2 • • • cIk-\Pk-\- This is actually a sequence of
trials in which at the Azth trial the system is in state k if the last failure occurred
at the trial number n - k (i.e. index k equals the length of uninterrupted block of
successes ending at the nth trial). By convention we take 0 < pt < 1 which implies
that the chain is irreducible. The nature of the transition matrix P shows that a
Tirst return at the nth trial can occur only through the sequence E0 —>Ex E2
• • • —» En_j —> £0, and so for n > 1,
22 Introduction to Stochastic Process
/oo = n=\
2 /do0 = n=l
2 (q0qi . . . qn- 2 Pn-\)
= Z f/;_2(since lim Un = 0)
fc=l A
2—>°°
= t/_, + 2 t/* = 1 + 2 Uk = 1 + 2
1 *=0 * A:=0 £=0 i=0
<7, n
Hence, M.C. is positive recurrent iff Z Uk <°
fc=0
< ) >° -
Thus d(0 is a divisor of {« > 1 : > 0}. Since d(j) is the largest of such
divisors, d(i) < d(j). Hence, by symmetry d(j) < d(i).
Hence d(i) = d(j). Therefore having a period d is a class property.
Note If pa > 0, then d{i) = 1 and this implies that a sufficient condition for an
irreducible M.C. to be aperiodic is that p n > 0 for some i e S. Hence a queueing
chain is aperiodic.
Theorem 2.7 Limit Theorem (for diagonal elements)
Let j be any state in a M.C. As n —> «>.
(i) if j is transient, then p ^ —> 0
Corollary 2.3 Let a M.C. be irreducible and aperiodic. Then j is transient iff
and this can be written as yn = Z f m un_m (the discrete renewal equation). Thus
m- 1
Proof of Corollary 2.3 follows from Theorem 2.5 and Theorem 2.7.
^77 —» 0 => —
JX ■ = 0 => ^UjJ = oo i.e. / is null recurrent iff r JJ —» 0
p jj} 7tj > 0 => jUj < oo i.e. positive recurrent iff —>/r;- > 0.
n
Proof i is recurrent => X p[n>= ©o .
n=l
n ,
(i) Now i is positive iff p t = X = lim —----- - , . < »
r /i=l Ml- (1 - s)Pu(s)
1
i.e. if lim — X p \k) > 0 (by Cesaro-Tauber Theorem).
Moo n. k=1 w 7
Conversely by Cesaro-Abel Theorem,
if /M
lim
oo —
fj >0
ie. — (1 -s)Pu
1
=> lim — X p-k) = 0(by Cesaro-Tauber Theorem)
moo n k=o^u J '
1
Conversely, lim — X =0
n-)oo ft £=0 "
=> lim (1 - s)Pu(s) = 0(by Cesaro-Abel Theorem)
1
■V — > -
if e - i ’ l '
and («) — OO .
? P0Q} - ? ^00
then fa
Corollary 2.4 Let a M.C. be irreducible and aperiodic then lim p j j ] exists for
all i and /, but is independent of /.
Proof If j is transient or null, the result is obvious from Theorem 2.8. If j is
positive, then this can be proved if f f = 1.
Lemma 2.5 If j is recurrent and j —> /, then i is recurrent and
L
Proof Assume j * /, for otherwise there is nothing to prove.
Since f f > 0, there exists n0 such that p ^ o) > 0
and p^ff = 0 for 0 < m < n0 (2.6)
Since p ^ o) > 0, we can find states /j, . . . , ino x such that p Jn . . -PjinQ ] > 0
and none of the states iu . . . , /no i equal j or /, for if one of them did equal j
or /, it would be possible to go from j to i with positive probability in fewer then
n0 steps in contradiction to (2.6).
Suppose f f < 1. Then a M.C. starting from i has positive probability 1 - f f of
never hitting) and that implies it has positive probability . . . p in (1 - f f )
of visiting the states . . . , /„01, i successively in the first n0 steps and never
returning to j after n0 steps. But if this happens then the M.C. never returns to j
at any time n > 1 and that contradicts the fact that j is recurrent. So f f = 1. Since
f f = 1, there exists n x such that p \f^ > 0. Now p \ff+n+n° ^ > p \fl)p ^ p ^ o) and
( 2 . 8)
m=n'+1
For (i) and (ii) when j is transient or null recurrent take n so large that p^n
77
m)
< £ for all 0 < m < n < n (by Theorem 2.7). Now from (2.7) and (2.8), we have
bm p<?> - - f 2 / / m> c £.
n— 7 p , m-1 7
Now take then ri —> then
=i k
X( Z v kPki)PlJ
Then n is called the steady state distribution of the M.C. with transition matrix
(iii) n .i = %s n kPkj
Moreover, (ii) and (iii) determine [7ih i e S) completely.
(Note that in this case the steady state probabilities or the long run distribution
is identical with the stationary distribution of the chain).
Proof (i) follows from Theorem 2.8 and the lemma thereafter.
(ii) nj = — > 0 (since j is positive recurrent /Uj < °o). Suppose is a
subset of state space S with exactly M states.
Now
2 P ^ P k j |^ = P u +l)(Chapman-Kolmogorov)
Let
n oo then Z n kpk j < 7i j .
keSM * -7 7
Discrete Time Markov Chain 29
Then letting
M —» oo, we get Z 7ikp kj < n j . (2.13)
jeS
]tsn Jp? = Is ,!
= h { h n >pkk) pki- £ s n
Now 1 (Since|,/eS?^ ”) = D
Z n k= Z/cg5ZjeS^kPkV
7
£eS
L ir
C * t c C .'c C * A, /
xi = Z Z-XtPkj Pj<
jeS X j P >‘ j£S keS
Note that condition (iii) of Theorem 2.9 can be written in a matrix equation
7i{P —/) = 0 where 7t= Tb, . . .). This matrix P-I is such that its off-diagonal
elements are non-negative and the diagonal elements are strictly negative. In
case of finite chain with k-states, from the irreducibility condition it follows that
the matrix P- 1 is of rank In this case n - (tzu /z^, . . . n k) can be easily
obtained from the homogeneous equations n(P - 1) = 0 and the normalizing
k
condition X n, = 1.
/=i
We state the following theorem without proof.
Theorem 2.10 (Limit theorem for periodic chain)
Let the M.C. be irreducible, positive recurrent having a stationary distribution. If
the chain is periodic with period d, then for each pair /, j of states in S there is
an integer r, 0 < r < d such that p \^ = 0 unless md + r - n for some non-negative
integer m
(md+r) _
7li d__
mlim
—»oo p)j•' /V
Note
(i) if i = j, r = 0, Theorem 2.10 is proved before (see Theorem 2.7).
(ii) if r = 0, d = 1, Theorem 2.10 is proved before (see Theorem 2.8).
It is extremely useful to calculate higher order transition probabilities. We
shall illustrate this with a simple example. In general, Pn is difficult to compute
except for 2-state M.C.
Example 2.4.
0 1
0fI- p p \
Consider the M.C. with P =
H <7 I - Q)
Denote the initial distribution by /r0 = (n Q(0), n Q( 1)), row vector nn - (P(Xn =
0), P(Xn = 1)) represents the marginal distribution of Xn.
We compute p ^ ) = Po(Xn = 0), the conditional probability of Xn = 0 given
that X0 = 0. Let tt0(0) = P(X0 = 0).
For a stationary chain
Poi = P(Xn+1 = 1 I = 0) = p , p l0 =) = q,
Poo = I - p , Pn = I - q,P\o = *r0(l) = P(-X^ = 1) = 1 - ^o(0).
Po(Xn+l =0) = P0(Xn = 0)p00 + P0((2.17)
P0(Xn = 0)(1 - p ) + P0(Xn =l ) q = (l
(2.17) is a first order difference equation. The solution is
p \l} = — ----- (1 - p - q ) n- 2 — ,
^ 10 p +q y P +q
and = — — + (1 - p - q ) n— q—,
11 p +q
1 ~q p~ ’ P - p~
SX
1
1
Thus, Pn =
p + q _q p_ p +q -q q _
1 q p \0 < p + q < 2
—» as n —> «> if \
p +q q p p < 1, q < 1
This is the steady state distribution and also a stationary distribution. Hence
the marginal distribution of Xn is given by
q
P(Xn=0) = - ^ + (l
p +q p +q
Therefore P q
x2= = p +q
p +q
Let us make a few commets about existence of steady state distribution of M.C.
1. Even if a M.C. is irreducible the steady state distribution may not always
exist (since the irreducible chain may be periodic).
2. If the M.C. has more than one closed sets then the steady state probabilities
(i.e. the fixed probability vector n) do not exist.
3. A M.C. is called regular if there exists a positive integer m such that
every element of Pm except those relating to transient states is strictly
positive.
32 Introduction to Stochastic Process
"0.6 0 0.4"
Exercise 2.5. P = 0.3 0.5 0.2 Here state 2 is transient.
v0.7 0 0.3y
f
7 0 4A
11 11
and pn —> 7 0 ± as n —» oo
11 11
7 0 4_
111 llj
(f r0
r Po 0
Q\ 0 P\ 0.
0 <72 r2 Pi 0.. .
0 0 <73 r3 Pi 0
^ • J
' 0 1 0 0 ••A
q\ 0 p\ 0
X = (x0, x u x 2, • • •) = (x0, x l, x 2, • • •) (2.18)
0 <?2 0 P2
Define yi = - L, y o = l > i = U 2 , 3,
Then yi - 1 1 - q\ p\
y 1= Uq\>y\= i + ^<72 or y2 =
q2 q\Q2 q\q2
P\P2 P\P2 - ■Pn- \
y3= >yn = >0 for all n = 1,2,
q\ci2q?> <7 l <72 • • •
(by assumption that all p, g’s are > 0).
By Theorems 2.9 and 2.11, the non-homogeneous random walk is positive
recurrent if 0 < £ a , < i.e. £ y, < ©o i.e. iff
o ' \ Jl
$ P \ P 2 • • -/V-l
£ ---------------- < oo.
n=1 Q\Q2 • - - <]n
1 if y = 0
Set y.i = P0Pl-..Pj-> i f . > L
Similarly, when d < ©o, then the unique stationary distribution is given by
As a particular case of the last example consider the Ehrenfest chain. The
transition matrix is given by
i/d if 7 = i ~ 1
P ij = 1 - i/d if j = i + 1 and S = {0, 1,2, . . . , J}
0 otherwise
such chain arises in exchange of heat or gas molecules between two isolated
bodies. Consider two boxes labelled 1 and 2 and d balls labelled 1, 2, 3, . . . d.
An integer is chosen at random. Then ball labelled by the chosen integer is
removed from a box and put in the other box. Trials are repeated. Let Xn be the
number of balls in box 1 after n trials.
Then Xn is a M.C. on S = {0, 1, 2, 3}, d = 3.
0 1 0 0>
1/3 0 2/3 0
0 2/3 0 1/3
0 0 1 0/
This i$ an irreducible birth and death chain with
Discrete Time Markov Chain 35
r 41 0 4
4
0 3
pn ^ 4 when n is even and
7 0 4
» 7
0 4 0 4
3
pn ^ 4 when n is odd and n is large.
0 4 0 4
4 0 4
Again from Theorems 2.9 and 2.11 the unique stationary distribution is given by
13 3 1
4’ 8’ 8’ 8
Note If some r, > 0, then pu = rx> 0 and the chain is aperiodic. In particular
Modified Ehrenfest chain is aperiodic. Suppose rt = 0 for all /, i.e. we have a
36 Introduction to Stochastic Process
non-homogeneous random walk. Then in one transition the state of the chain
changes either from an odd numbered state to an even numbered or from an even
numbered state to an odd numbered. In particular, the chain can return to its
initial state only after an even number of transitions. Hence, the period of the
chain is 2 or multiple of 2. Since = p0qx > 0, the chain is periodic with
period 2 and so the Ehrenfest chain is periodic with period 2.
Example 2.6 Consider the Markov chain with transition matrix
' 0 1 0
p = 1/2 0 1/2
v0 1 0
"1/2 0 1/2
P2 = 0 1 0
v1/2 0 1/2
Theorem 2.12 The M.C. is transient iff x t = ZqPijXj has a solution for i * 0,
which is bounded and non-constant i.e. all x,’s are not equal.
Theorem 2.13 The M.C. is positive recurrent if x t > XQPijx j has a solution
such that-Xj -> ©o as i -> oo (see Chung’s book on Markov Chains with Stationary
Transition Probabilities).
Jim p \fp - 0 for each j), which is absured and hence not all states in a finite
M.C. are transient. Consider the subchain C\ formed by a closed set of null
recurrent states. Then Z p \p = a (say) > 0. Letting j i —><», 0 = a > 0 which
is also absurd. So there cannot be any null recurrent state in a finite M.C.
Theorem 2(b). An irreducible M.C. having a finite number of states is positive
recurrent.
Proof By previous theorem, there is no null recurrent state and not all states are
38 Introduction to Stochastic Process
transient. Suppose there is one transient state. Then all states are transient by
Solidarity Theorem. Hence, all states are positive recurrent.
Exercise 2.6 If a finite M.C. is irreducible, aperiodic and has doubly stochastic
transition matrix, then show that lim p f(.n) = 1Ik, where k is the number of states
n —>oo
in the chain.
Pn • • •Pu
= 1
P2\ P22 • • ■
•Pik
= \ f lim ptf> = 1
_ j i=l n —>°° lJ
KPk\ • • -Pkk
1 1 . . ,. 1
0 1 0
P = 1/6 1/2 1/3
0 2/3 1/3
1 1 1'
6 2 3
P2 = 1 23 5 > 0, so P is regular.
12 36 18
1 5 1
9 9 3
1
6y=* y = 6x
1 , 2 or 6x + 3y = 4z = 6y
or 2 y + 3 Z= y
1 1 y + z = 3z
3y+ r = z
by;r=(A’^’^)-0bviously
i 6 3
10 10 10
i 6 3
10 10 10
i 6 3
10 10 10
1 0 0"
1 1 1
4 2 4
I 8 9
18 18 18
Under these conditions : (1) what is the probability that the machine will be in
state i (i = 1, 2, 3) after n time period? (2) What is the probability that the
machine is in state i when n becomes infinitely large (i.e. the steady state probability
or long run behaviour of the machine)? (It is given that the machine is initially
in need of adjustment).
( N \ k N N
pk 2 A, A,- = 2 A.*A; = 2 M U M ' . ( 2 . 21 )
m ' 1 i=1 ' i=l 1
Since the latent vectors are determined uniquely only upto a multiplicative
constant, we have chosen them such that U' Vt = 1. From (2.21) one can get any
power of P knowing A,’s and A,-’s.
Example 2.7 We shall illustrate the last method with the help of Exercise 2.8 of
Section 2.7.
1 0 0"
1 1 1
In our problem, P = 4 2 4 with characteristic equation
1 8 9
18 18 18
1- A 0 0
1 1
IA = 0 or (1 - A ) A2 - A + 0.
4 2 4 36
=
1 8 9
18 18 18 ‘
1- A 0 0
I 1 ~0i ~
4 5a. 4 = 0 we get the characteristic vector U\ =
_L JL 9 ,
_c\ _
18 18 T8Al
1
o
o
U= 3 3 and r 1= —
24
-4 4_
So
0
'1 0 0] p* 0 0
Pk = UDkU~l= 1 3 3 6 4 -3
1 -4 4j 0 4 3
0
0 0
1 - rk
3s_k n 3sk
or 8 2 8
■ rk sk 2 sk rk
2 8 8 2
k
'5 V
where rk =+ 1 ’sH ,6
Now rk —> 0 and sk —> 0 as k ►oo.
1 0 0
Therefore Pk —» 1 0 0
1 0 0
This shows that irrespective of the initial distribution (state), the machine will
be in state 1 (irrepairably broken). So state 1 is an absorbing or a trapping state.
Exercise 2.9. Derman (1963) designed a model for making the Maintenance or
Replacement decisions.
At the beginning of time period State of equipment at the end of time period
Operating properly Need of Adjustment
Operating Properly 0.90 0.10
Need of Adjustment 0.01 0.99
0.90 0.10'
Here P =
0.10 0.99 *
Suppose the equipment starts out working properly in period zero. Calculate
the unconditional probabilities of the state of the equipments and the conditional
probabilities after time period n.
Solution The initial distribution a(0) = (1, 0),
42 Introduction to Stochastic Process
f. 90
am = a (f>)P= (1,0) = (0.90,0.10) and
v-01 .99
^0.8110 0.1890A
a <2) = a <I)P = (a (0>P 2) = ( 1,0) = (0.8110,0.1890)
_0.0189 0.9811
In general a(n> = a<0)P". We shall calculate Pn by another method.
0.9- A 0.10
= 0 => A2 - 1.89 A + 0.89 = 0.
0.01 0.99
By Caley-Hamilton Theorem,
P2 - 1.89 P + 0.89 / = 0 =* P2 = 1.89 P - 0.89 /
=> P2 = 1.89 P2 - 0.89 P = 1.89(1.89 P - 0.89 I) - 0.89 P
= 2.6821 -1.6821 /.
P
Similarly, any power of P can be calculated in this manner.
(c) Method of Generating Functions
Since 1 is always a latent root of a stochastic matrix P. (/ - tP) is non-singular
for 0 < t < 1.
Hence (/ - AP)~l exists. But since (tP)n —> 0,
/ + tP + P'P2 + . . . = ( / - tP)~x. Therefore, p \yn) is the coefficient of t n in the
( ij)th element of (I -tP)~l, which is the generating function of {p^n)}- Suppose
S= {0, 1, 2, 3} and
0 0 0)
<7 p 0 0
P= ,p + q= l,
0 q p 0
0 q p.
(1 - O '1 0 0 0
tq 1 0 0
(1 - 0 ( 1 - t p ) 1 - tp
(i-tpy' = (tq)2 tq 1
0
(l - 0(1 - tp)2 (l -
t
p)2 1
(f<7)2 (*<?)2 _ _ _1_ _
(1 - r)(l - tp)2 (1 - tp)2 (1 - t p ) 2 1 - tp
Discrete Time Markov Chain 43
Pjk(s) = Fjlc(s)Pkk( s ) U * k ) }
and PjJ(s) = FjJ( s ) P j j ( s ) + l j
Hence we can determine Pjk(s) and Pkk(s ) if we know Fjk(s) and Fjj(s).
1 2 3 4
1 "0.0 0.0 1.0 0.0
2 1.0 0.0 0.0 0.0
3 0.3 0.7 0.0 0.0
4 0.6 0.2 0.2 0.0
Exercise 2.2 A service agency assigns its jobs to a particular worker in the following
way. The maximum number of jobs assigned to him, in addition to one he is working on
at any time is N(> 1). If he can not finish the assigned jobs on a given day, he starts with
the remaining ones the following day. However, if any time of the day he finishes all the
jobs assigned to him, he returns to his own work and becomes unavailable for any more
agency jobs for that day. Let pj be the probability that j(> 0) new jobs arrive during a
service period. Let Xn be the number of jobs assigned to him at the end of the nth service.
Under what conditions is {X,,} a Markov chain? Determine its transition probability
martix and classify its states.
Exercise 2.3 (Bartky's sampling inspection scheme). In a sampling inspection procedure
successive sampling of size N is taken. If in the initial sample the number of defective is
zero, the lot is accepted. If the number of defective exceeds a predetermined number a,
the lot is rejected. From the second sample onward one defective per sample is allowed.
Thus after n such samples, the lot will be accepted if the total number of defectives is
^ n and rejected if the number of defectives is > a + n. Let Xk = number of defectives out
of N - One at the &th sample. Let Sn = Total excess number of defectives in the lot. Find
the distribution of Xk. Show that {£„} is a Markov chain. Find the transition matrix for
{£„} in terms of distribution of Xk. Also classify the states of {5,,}.
44 Introduction to Stochastic Process
Exercise 2.4 A Simple Waiting Model (Queueing). In a simple queueing model a server
serves one customer (if any) at time instant 0, 1 ,2 ,.... Let be the number of customers
arrive in the time interval (n, n + 1) and we assume {£,„ n > 0} is a sequence of i.i.d.
nonnegative integer valued r.v.’s with P(%o = k) = pk, Zq pk = 1 and there is a waiting
room for at most m customers (including the customer being served). Let Xn be the
number of customers present at time n, including the one being served. Show that Xn is
a Markov chain with states 0, 1, . . ., m. Find its transition matrix in terms of {pk }q .
Exercise 2.5 (Storage Model). Consider a dam which can hold at most m units of water.
Assume that during day n a quantity T]n of units of water flow into the dam, n > 0, where
{r)n, n > 0} is a sequence of i.i.d. nonnegative integer valued r.vs. with P(rj0 = k) = pk,
E pk - 1. Any overflow is lost with no possibility of recovery. At the end of any day
k =0
when the dam is not dry, one unit of water is released. Show that (X '}, the content of
the dam at the beginning of day n and {X''}, the content of the dam at the end of day n
are Markov chains on appropriate state spaces. Find the transition matrix of { X'', n > 0}
in terms of (pk )q.
(Note: {X', n > 0) is a M.C. identical to that with the preceding Queueing example
with states space (0, 1, . . . , m) and hence it is omitted for any further discussion.)
Exercise 2.6 A Markov chain remains Markov if the time is reversed, i.e.
PQ^n ~ hi I Xn+\ — in+1, • • •, Xn+k — in+k) —P(Xn — itj | X,I+1 — in+\)
Exercise 2.7 Show that the eigenvalues of a stochastic matrix are in absolute value at
most equal to 1.
Exercise 2.8 (A model for the spread of a contagious disease in a small community). Let
m > 0 be the number of individuals getting infected at time m (i.e. the number of
infectious individuals at time m + 1) and Xn, n > 0, the number of susceptibles still
uninfected at time n. Let us assume that the chance of contact between two individuals be
0 < p = 1 - q < 1 and the contacts to be independent. Show that {X„} is a Markov chain
with states space {0, . . . . , r] when the initial number of uninfected susceptibles is r. Find
its transition matrix. If r be the duration of epidemic defined as smallest integer r such
that £t = 0, express the number of cases in the epidemic in terms of XT(XT_! = XT).
Exercise 2.9 Show that in a Markov chain, if the present is specified, then the past is
independent of the future in the following sense. If k < m < n then
P(X„ = i„, x* = ik
Ix m = i
Exercise 2.12 Prove that in a finite Markov chain a class C of states is recurrent iff it
is closed.
Exercise 2.13 (a) If a Markov chain starts in state i then show that the probability of
returning to i at least r times equals f t .
(b) If a Markov Chain starts in state i then show that the probability that state j occurs
at least r times equals fij[fjjY'X-
(c) Calculate the probabilities of returning to i infinitely often when f u < 1 and when
fa = 1•
Exercise 2.14 Let u(i) be the number of visits to the state i and P,[A] is the conditional
probability of A starting from the state i. Then show that
(a) P,[v(j) = n] =fU[fijr lU-fjln> 1
where fj is the conditional probability of ever visiting j starting from /.
f ki
(b) Pk[vU) °<°] = 1 and n(k, i)=
1 Jii
where p{k, /) is the expected number of visit to i starting from k when i is transient.
(c) Hence show that if i is a transient state, then the M.C. visits i only finite number
of times and the expected number of visits is also finite.
Exercise 2.15 In a Markov chain whatever be the states i and j, prove the Doeblin’s
formula
m
(1)
Exercise 2.16 Let {Yn, n > 1} be a sequence of i.i.d. r.v.s having discrete uniform
distribution on {0, 1, 2, . . . k}. Let Xn = max {Tj, . . . , T„}, n > 1. Show that {X,,} is a
Markov chain (M.C.) and obtain its one-step transition probability matrix. Discuss about
the states of this M.C.
Exercise 2.17 Consider the polya urn scheme with a white and b black balls in a box.
Each time one ball is drawn and is replaced together with an additional ball of the same
colour. The procedure is repeated several times. Let Sn equals the total number of white
balls in the box at the end of the rcth step. Show that S0, ..., Sn, ... is a nonhomogeneous
Markov chain with transition probabilities given by
i _ a+b+n- i
n Pi,j+1 a + b + n nPl1 a + b + n ’ „p,j = o i f / * i , ; + i .
Exercise 2.18 If Xn = max j K,. Y,. . . . Yn] where K, denotes the number on the face
turning up in the ith toss of a die with faces 1,2,..., 6. Show that {X,,} is a Markov chain.
Obtain its transition matrix.
Exercise 2.19 Inputs at the jth period, Xr into a dam of finite capacity k follows a
geometric distribution P(Xj = k) = qpk(k = 6, 1, . . .).
There is unit release unless the dam is empty. Show that the contents of the dam {Yn)
at any time n form a Markov chain. Write down its transition probability matrix.
46 Introduction to Stochastic Process
Exercise 2.20 Let {AT„} be a M.C. on {1, 2, . . . M}. The conditional distribution of X„+1
given Xn = j is the discrete uniform distribution on [M - j + 1, . . . , M). Obtain the
stationary distribution.
Exercise 2.21 Consider a Markov chain having state space {0, 1,2} and the following
transition probability matrix P.
0.0 1.0 0.0
0.5 0.0 0.5
0.0 1.0 0.0
Identify the recurrent states of the chain.
Exercise 2.22 Consider a communication system which transmit the two digits 0 and 1
through several stages. Let {Xn, n > 1} be the digit leaving the nth stage of the system and
X0, the digit entering the first stage leaving the 0th stage. At each stage there is a constant
probability q that the digit which enters will be transmitted unchanged, and probability p
otherwise, p + q = 1.
Find the rc-step transition matrix of the two-state Markov chain and the steady state
probabilities if there is any. Also find the probability that the digit entering the first stage
is 0 given the digit leaving the rath stage is 0 when the initial distribution is given by
P(X0 = 0) = a = \ - p ( X 0 =l).
Exercise 2.23 Let pxy denotes the probability that the Markov chain starting at state x
will be in state y at some positive time; also N(y) denotes the number of times n > 1 that
the chain is in state y. Show that P[N(y) > 2 \ X 0 = x] = pxypyr X0 being the initial state.
Extend your arguments to establish that in general P(N(y) > m \ X0= x) = pxy pyy~\ ra > 1.
Hence sho,Wthat if y is a transient state, then for every initial state x, E[N(y) | X0 = x] < «>.
Now argue to deduce that a Markov chain having a finite state space must have atleast
one recurrent state.
Exercise 2.24 Deduce that (with usual notation)
(n+m)
f JJm)p(n)<p..
rjj r JJ + (1 - p T )
Exercise 2.25 On a genetic inbreading experiment with single pair of Allelomorphos A
and a, the progeny of a mating Aa x Aa results in 50% heteroztgotes Aa where the
proportion of hetrozygotes being reduced by one-half at each generation. Find the transition
matrix of various genotypes and the proportion of hetrozygotes at rth generation and in
the long run.
Exercise 2.26 Consider a dam for water storage with Capacity k units, k being a positive
integer. Let there be independently identically distributed integer-valued daily inputs to
the dam with probability generating function g, Zy. Any overflow is lost. If the dam
is not empty, m > 1 units is released at the end of each day. The dam contains Zn units at
the end of nth day. Show that Z„ is a Markov chain and write down the transition matrix.
When is the chain irreducible? Find also the transition matrix when the input has a
geometric distribution.
Exercise 2.27 Modelling of soil erosion effect and modelling of a rock crunching process
(.Kolmogorov 1941).
Let {Yn}T be the sequence of r.v.s representing annual yields of a given crop in a crop-
producing area A when it is not affected by erosion. Let Rn be the percentage reduction
of annual yield due to soil erosion in the nth year and Un - R J 100. Then show that when
Discrete Time Markov Chain 47
A is subject to soil erosion the resulting annual yield in the nth year is given by
n n
Xn = Yn n Z,, Z, = 1 - Ui. Incase {Z„}[° arei.i.d. and independent of {Yn}~, Ln = n Z,
i=l i=1
is called “loss-rate function”. Show that {L„}T is a Markov chain.
Exercise 2.28 N black balls and N white balls are placed in two urns so that each urn
contains N balls. At each step one ball is selected at random from each urn and the two
balls interchange. The state of the system is the number of white balls in the urns.
Determine the transition matrix of the Markov chain. Find its steady state probabilities
and show that these do not depend on the initial distribution of black and white balls in
the two urns.
Exercise 2.29 Consider two urns A and B containing a total of N balls. Assume that at
time t there are exactly k balls in A.
(a)_A time t + 1 urn A is chosen with probability k/N or B is chosen with probability
N It
—— —. Then a ball is selected from A with probability p or from B with probability q and
placed in the previously chosen urn. Determine the transition matrix for this Markov
chain.
(b) Now assume that at time t + 1 a ball and an urn are chosen with probability
depending on the contents of the urn, i.e. both a ball and the urn A are chosen with
aj _ im
probability k/N or both a ball and urn B are chosen with probability ———. Determine
the transition matrix of the Markov chain with states represented by the contents of A.
(c) Determine the equivalence class in (a) and (b).
Exercise 2.30 Every stochastic n x n matrix corresponds to a Markov chain for which
it is the one step transition matrix. Similarly every n x n two-step transition matrix of a
Markov chain is a stochastic matrix but the converse is not true. Give a counter example.
Exercise 2.31 If P is the one-step transition matrix of a finite aperiodic irreducible
Markov chain, prove that for some positive integer n all terms of Pn are positive.
Exercise 2.32 If j is a transient state prove that for all i e S, 2 p\^ < °°.
Exercise 2.33 If an infinite irreducible Markov chain has a doubly stochastic transition
matrix, prove that the chain can not be positive recurrent.
Exercise 2.34 Suppose the quantities {nip i j • e 5} are defined as the “Cesaro means”
nn' = lim III imX-1 1P\p.
n m=l ‘j
Exercise 2.35 Show that if a finite irreducible Markov chain has a doubly stochastic
transition matrix, then all the stationary probabilities are equal.
48 Introduction to Stochastic Process
Exercise 2.36 Let /V, be the number of returns to state i, starting at /. If / is transient, then
prove that P(N, = r) = (1 -fu)(fuY. Also find E(Nj).
Exercise 2.37 A positive recurrent irreducible Markov chain is given.
Let /i/; = £ f yn). Show that
<d> Z / j V - f j * +1
fkk
(e) If/** < 1 then jZ p^0 = } •
(b) If the Markov chain is irreducible and aperiodic then show that , = n—
-
lim
>°° J
(c) Assume that the Markov chain has a stationary distribution {/r7}. Evaluate p(.n) if
ak = nk, Explain.
Exercise 2.40 Consider a Markov chain on S = {0, 1, 2, . . .} with transition matrix
Show that the Markov chain is irreducible, aperiodic and positive recurrent. Also
prove that Kj = e~Xlq{Xlq)ilj\ is the stationary distribution.
Exercise 2.41 Assume state {0} of a Markov chain is positive recurrent. Take {0} to be
the initial state of the Markov chain. Let {Wn, n > 1} denote successive recurrence times
which are independent and identically distributed r.vs. with finite mean and with a generating
function G(t) = YPiW^ = /:), 111< 1. Define Yn as the time of the last visit to state {0}
before time n.
1 -G(t)
Show that £ r Z\oxJP{Y,,=j) =
n=0 (1 - / ) ( 1 - G ( x t ) )
Exercise 2.42 Consider a discrete time Markov chain with states 0, 1, . . . , N whose
transition matrix has elements
Discrete Time Markov Chain 49
Hi if i= / - 1 i, j =0, 1,2__ N
A, if j= / + 1
1 - Hi - A, if;
0 otherwise
Suppose that ^ = Ao = = XN = 0, and all other ^,’s and A,’s are positive and that
initial state of the process is k. Determine the absorption probabilities at 0 and N.
Exercise 2.43 Consider an irreducible positive recurrent Markov chain with initial state
X0 = i. Let Nn(i) be the number of visits to state i in the first n trials. Prove that
E(Nn(i)) = 1
n—>°° n Hi ’
Exercise 2.44 Consider an irreducible Markov chain. For some state; call it 0. Prove that
if fjo > a > 0 for every i * 0 then the chain is recurrent, where f i0 is the probability of
reaching state 0 from state i.
Exercise 2.45
(a) Let a finite state Markov chain be given. Prove that j e S is transient iff there exists
a state k e S such that j —>k but k + j.
(b) Give a counter example to (a) if we drop the “finiteness” of the Markov chain.
Exercise 2.46 Let a Markov chain be given with only absorbing states (E) and transient
states (Ec = S - E).
(a) Show that transition matrix P can be put in the form
E E(
E (I
EC\ R
(b) If i j e Ec, let Ny be the mean number of times the chain is in state j, starting at
i. Prove that N = ^ Q (k) where Q = Q(0) is defined in (a).
(c) Define By = P {The Markov chain is absorbed at j \ X0 = /} for j e E, i e Ec. If
E * (p(null set), show that B - NR (with B = (By)).
(1
(d) Derive from this nlim
-* o o
Pn = D
Exercise 2.47 Prove that if the one-step transition matrix P of a discrete time Markov
chain is symmetric, so is the n-step transition matrix P(n) for all n > 1. Moreover this
Markov chain cannot be positive recurrent (except for finite chain).
Exercise 2.48 Let M be an irreducible, aperiodic positive recurrent discrete time Markov
chain. Assume that the initial distribution coincides with stationary distribution. For all /,
j e S and integer m > 1, define (R{m\ = Rjtm) = P(Xn = i \Xm=j).
(a) Show that this chain is also positive recurrent, and determine its stationary distribution.
Exercise 2.49 (Markov-Bernoulli chain, Wang (1981))
Consider a sequence of dependent trials such that each trial has only two outcomes
denoted by ‘1’ and ‘O’ and dependency is connected by a simple Markov chain model
having transition probabilities
50 Introduction to Stochastic Process
0 1
Of \ - ( \ - c ) p 0 -C)p \
, 0 < p < 1, 0 < c < 1
1 [(1 -c)(l - p) (1 - c ) p + cj
with initial distribution px - P(XQ= 1) = p = 1 - P(X0 = 0).
Find the marginal probabilities pn = P(Xn = 1). If Sn = Sx + . . . + Xn, show that
E(Sn) = npvar(S„)
, =np(1 -p ) + 2p(l -p ) - c2 / / , ~ j ■If as n “ and
p —>0 such that np —>A, then show that E(Sn) —» A and var (S„) —>A + -p— •
Exercise 2.50 Let P = (/fy) be a transition matrix of an irreducible Markov chain and
suppose that P is idempotent Markov matrix, i.e. P2= P. Prove that p^ = pjj for all i,j, e 5
and that the chain is aperiodic.
Exercise 2.51 (Ergodic theorem in discrete time). Let Xn be an irreducible M.C. Let
n- 1
pi be the mean recurrence time of state i. Let Vt{n) = 2 I\Xk=i\ numt>er of visits
to i upto n - 1 and /b e any bounded function on S. Show that:
(a) ri~lVi(n) ■—> p~x as n —»
1 " '1
(b) if Pi < °o for all /, then —= ^ f ( x k) —> X f(i)/pi a s ^ ~
Exercise 2.52 (Estimating transition matrix). Let {X*, 0 < /:< rc) be a M.C. with initial
distribution a, = P(X0 = /). The log-likelihood function L(P) = log (ccXQpXQX{. . . pXn_lXn)•
Show that:
(a) L(P) = log a*0+ 2 /Vy log ptj where is the number of transitions from i to j,
(b) Viewed as a function of pijt L(P) is maximal when Pij = Pij = N^/2
(c) If X is irreducible and ergodic then ptj —>pi} a.s. as n
3
Random Walks
3.0 Introduction
Let [Xn, n = 0, 1, 2, 3, . . .} be a sequence of independent discrete random
variables taking integral values only and Sn = X x + X2 . . . + Xn(n = 0, 1, 2, . . .).
Then the sequence {S„} is a M.C. whose transition probabilities are given by,
{m)Pij = P(Sm+1 = j | = i) = P(Xm+l = j - /), i j = . . . , -2, -1, 0, 1, 2, . . .
(non-homogeneous random walk).
The chain represents a Random walk of a particle along a straight line, the
magnitude of ‘jump’ at time n being given by the random variable Xn. If X0 is
denotes the initial position of a particle then its position after n jumps (at time n)
is given by Sn. When Xn's are also indentically distributed, = Pj_t where pj
= P{Xn = j). We have then a homogeneous Random walk (RW). Such Random
walks occur in fluctuation theory (sums of discrete or continuous random variables).
In classical RW, P{Xn = +1) = /?, p(Xn = -1) = q = 1 - p.
= 0 if n is odd.
and n n/2
Poo = 2 (pq)
v J
f n ) p(n-i)/2 (j (n+i)/2
K( n - l)/2.
Probability of visiting ‘O’ from i ever,
fio = Z //0"} satisfies difference equations
1 if p < q
fiO ~
(q/pY if p > q.
P if / = i + 1
Pu <17 if j = i - 1 and i = 2, 3, . . . , s
0, otherwise
1 if 7 = 1
and i = 1 and 5.
0 if j * 1
More explicitly the transition matrix is given by
" 1 0 0 0 ■
q 0 p 0 0
0 <7 0 p 0
_0 0 0 0 1
Random Walks 53
8 = ------ !------ =
1 - (q/py 1 - ( q/ pY
(qlpY ~ [qlpT
if P * <7
Hence P(x) (q/pY - 1 (3.1)
l - f i f p =q
The last expression follows from the fact that if — = > 1, then
P
lim p( jc) = 1 - - (by L’ Hospital’s Rule).
Now suppose the gambler is playing against a person with infinite capital, i.e.
s -» 00 (a random walk with one absorbing barrier) then from (3.1) we get
1 if q > p
lim p(x) =
S — >°o
(q/ p) x if q <p
Let us investigate the effect of changing stakes.
If the amount of money held by two players are doubled, then
( pi a)s —( picj)s x
Then p(s) = ----------------------- (replacing p in (3.1) by q and x by s - x)
(p/qY- 1
Evidently, p(0) + p(s) = 1.
Let Dx be the expected number of steps (time) required for absorption to one
of the two absorbing states (i.e. the expected duration of the game). Then Dx
satisfies the difference equation
Dx = pDx+x + qDx_i -f 1 if 0 < x < s (3.2)
with boundary conditions D0 = 0, Ds = 0.
This follows from the fact that if the first trial results in success the game
continues as if the initial position has been x + 1. Conditional expectation of the
duration assuming success at the first trial is Dx+X+ 1. More formally
E(DX | first trial is a success) = Dx+X + 1.
Then Dx = (Dv+1 + 1) p + (Dx_x + 1)q - pDx+l + qDx_x + p + q with boundary
conditions: D0 = 0 = Ds.
A formal solution is
Dr = (p * q) with Ax = p Ax+1 + q Ax -\
q -p
(where Av is the difference of two solutions of (3.2)).
To get the general solution we proceed as follows.
Boundary conditions give A + B = 0 and A + B(q/p)s = -
q -p
Solving A and B we get, Dx = + A + B( q/ p) x
q -p
x s 1 ~ (qtpY
q-p q —P \ —(q I p) s
x(s - x) if p = q = j
which is the expected duration of the game in the gambler’s ruin problem,
(d) Random Walk with a Reflecting Barrier
P,.M= P’ Pi,,-i = q for 1 > 1, = q-
Here we imagine a barrier placed at -1/2 such that every time the particle
moves to the left from 0, it is reflected at the barrier and returns to ‘O’. The chain
is irreducible if 0 < p < 1. To classify its states, consider the system of equations
y, = X Pj, v,. Then we get
7=i
yi = pyi+\ + qyi-\ 0 ^ l)
i.e. p(yl+\ - yt) = q{yt - y,_i) O' > 2), y, = py2-
Therefore by iteration we obtain
y/+1 - y, = y\(qip)1, / ^ i
and yf- - vj = y x{(q/p) + (q!p)2 + • • . + ( q lp T X}
Random Walks 55
d -<?)
71q — ^1 ~ ^0
<7
Proceeding successively Uj = (p/g)7 7r0(y > 0),
where 7i0 {1 + (p/q) + (p/q)2 + . . . } = 1. (3.3)
If p = g, the series diverges and consequently 7r0 = 0 and ^ = 0 (y > 0) so that
stationary distribution does not exist. Thus, if p = q, the states are null recurrent.
If p < q, by (3.3) 7T0 = 1 - p/q > 0, and is the stationary distribution (the states
are positive recurrent).
(e) Random Walk with two Reflecting Barriers
If in addition to the barrier at -1/2, there is a second reflecting barrier at a + —,
so that paa = p , pa = q, {pu /+1 = p , pu. { = q, i = 1, . . . , a - 1, p w = q,
p0] = p ). We have a finite chain on {0, 1, 2, . . . , a}. In this case stationary
distribution exists and is in fact the geometric distribution of the previous case
truncated at a, i.e.
= i - pig (1 - p/q) (p /q )J
y = 1, . . .
1 - (p /q )a+l 1- ( plq
In gambling terminology Sn, n > 1 is the Peter’s accumulated gain at the end
of the nlh trial if Peter and Paul play for unit stakes. Now consider the event
A „ = { S ,< 0 , S2 < 0, . . . , S n_ !< 0 , Sn = l} .
Thus, the nih trial is the first to render Peter’s accumulated gain positive. The
event An is called first visit to +1 or the index n is the passage time through 1 in
random walk terminology.
Let (pn = P(A„). Define (fo = 0, 0i = p.
If the event holds for n > 1, then S\ = -1 and there exists a smallest integer v
< n such that Sv = 0. The outcome of the first n trials may be described as
follows: (1) At the first trial Peter looses an unit amount. (2) It takes v - 1 further
trials for Peter to reestablish the initial situation. (3) It takes exactly n - v further
trials for Peter to attain a positive net gain. These events depend on non-overlapping
blocks of trials and are therefore mutually independent, (2) and (3) have probabilities
0i-i and 0w_y.
56 Introduction to Stochastic Process
Now the event An occurs iff the events (1) to (3) occcur for some v < n.
Summing over all possible 1 < v < n - 1, we get
n- 1
2 L 0, Sn = q {s S 0„_i s" 1 }
n —2 \n-2
= 0» j " |
= <7* n=2
2 {0, <P„-2 + ■■■+ 01 } Sn 1
i 00
H ) H (4 m )^£ ^<.2*-!
2
=J- E
2<y *=i
z' n *-i
( - 1)
Therefore 0 2 * -l “ (4pq)k and 02* = 0, k > 1.
2q
v*/
l-jp-gj \p lq if
Now 2 L0„ = <P(1) = -
n =0 2q 1 if p > q
P (Sn remains negative forever)
\q - p
= 1 -2 if p < q
n=\ i q
o \f p > q
Random Walks 57
Therefore, P(Sn will sooner or later become positive) = 1 if p > q and E (number
ifp = q = 1\2
of trials preceding the first positive sum Sn) = & ' (I) = }
—L~
p-q if rp > Hq
(
' 2 (4 pqs2)" = (1 - 4 2\~J
= n2- 0 (-i)»
F~(s) = Z / 2- s 2n = sq S <h.n-\ s 2n -\
n=\ n- 1
i
2^2
1 - ( 1 - 4 pqs1)
= qs <P(s) = qs
2 qs
By symmetry, F*(s) = F~(s) and hence
= 1- in general j
F( D = n=1 % fl n
= 1 - (1 - 4 p q ) 1= 1- - |
2q if p > q
2p if p < q
1 if p = q= 4-
1 - F(s)
if q=
\-s 2
As both U and F are power series in s2 this relation differs only notationally from
1 -P(s)
Q(s) =
1 -5
^ 2 n = fln+ 2 + /2 n + 4 + • • • •
that is when p = the probability that S2n = 0 equals the probability that the
2n sums S\, S2, . . . S2w are different from zero.
Hence E ( N k) = £ n k
n- 1
m -1 oo
= E n l P[jV = n] + S P[N = n]
n= 1 ti-m
=£ I. P[N = n ] E ( X l \ N
n= 1 i=l
=£ £ P[N= n] £(X,
i = 1 n=i
(interchanging the order of summation)
= l p [ N > i ] E ( X i ) = E( Xi) E( N) .
60 Introduction to Stochastic Process
Let Nn = min (A, n). Now let Nn N monotonically, it follows from the
Monotone convergence theorem that
ENn —> E(N) a s/i-^ o o (3.7)
Since {(£„ - njd)2 - n a 2, n > 1} is a martingale (prove it).
We can apply optional sampling theorem to obtain (see Appendix iv)
E(SNn - np)2=
Now let m > n. Since martingales have orthogonal increments we have, by
(3.7) and (3.8),
E(SNm ( SN
n-pNn))2 = - E(SNn
= cr ( E N -m ENn) —» 0 as >
that is SNfj - fuiNn converges in L2 as n —» ©o.
However, since we already know that SNn- juNn —> SN - jiN as n —> it
follows that
E (S n„~ pNn)2 —> E(Sn - fiN)2 as n —> <»,
which together with (3.7) and (3.8), completes the proof.
£ -1 etixdFn{x)
= Z ,z " J e0 x dFn(x- 1
n=1
= (Did) |
\-:b
"=I J-b
= z0 ( 0 ) £ z"-' f e°>dFn_x
= 1 + [ Z 0 ( 0 ) - 1] F(z, 0) (3.10)
62 Introduction to Stochastic Process
E(eesN[<p(6)TN) =1 + lim - 1] 6)
z->[0(e)rl
= 1.
(<?/?)*
(s) - A" (s)
where AjO) and A2(s) are given by
Random Walks 63
l + ( l - 4 pqs2)m , , N l - ( l - 4
Ai (s)
-------- 2 sJ ' {S) = 2 --------
Also show that
Exercise 3.7 Let Xu X2, . . . be i.i.d. rvs taking values in the integres Z = {. . -2, -1,
n
0, 1,2,...} and having finite mean. Show that the M.C. {£„, n > 1} given by Sn = £ Xf
is transient if E(X{) * 0.
Exercise 3.8 Consider a symmetric random walk Sn with S0 = 0. Let T = min {n> 1 :
Sn = 0} be the time of first return of the walk to its starting point. Show that
4.1 Introduction
Let Xn,n = 1, 2,. . be the nonnegative i.i.d r.v.s with Sn = Xx + .. . + Xn, n > 1,
Sq = 0. F is the common d.f. of X and assume P(Xn = 0) < 1. Define N(t) =
sup {n | Sn < t}. The process {N(t), t > 0} is called the Renewal Process.
To fix our ideas X, can be taken to represent the life time of the machines
being replaced. The first machine is installed at time t = 0 and is replaced
instantaneously at time t = X x. The replaced machine is again replaced at time
t = X x + X2, and so on. If we write Sn = X x + . . . + Xn, the partial sum Sn can be
interpreted to be the time at which the nth replacement is made. N(t) is the
largest value of n for which Sn < t. In other words N(t) is the number of renewals
that would have occurred at time t. The Renewal Theory, in a sense, is a special
case of a Random Walk with absorbing barrier. We are sampling the Xt until Sn
shoots the barrier at time t and N{t) + 1 is the sample size when we stop. Hence
the Renewal Theory is also linked with Sequential Analysis in statistics.
{N(t), t e (0, <»)} is called the Renewal Counting Process. We can also write
N(t) = max{« | Sn < r}.
We want to find P[N(t) = n] given F. To compute this we proceed as follows:
F(t - u)dF(u)
o
= F* F{t)= F® (0, • • •
Define
-r Jo
P[t - u < Sn < t]dF(u) (since Sn is independent of Xn+1)
=
Jo
f {/r(n >(/) - F (n)(t - m)JJF( m) (by (4.1))
(c) //(/) = F(/) + I //(/ - «)dF(w) , the so-called integral equation of Renewal
Jo
Theory (Renewal equation).
(d) {N(t)y te [0, °o)} is completely determined by H(t).
Since nk < ed°n for large n and all k = 1, 2, 3, . . . , ENk(t) < °° for all k - 1,
fR e (0 )< lo g i j .
T
H(t) = E N (t)= =-
7=1 J P
We have derived before the integral equation and showed that the H (t) = 2 F {n)(t)
n=1
is a solution of (4.1). Now we have to show that the solution is unique.
Let V(t) be a solution to (4.1), which is bounded for all finite t. Then,
V(t) = F(t)+ f
Jo
V(t -u)dF(u) for all t
= F il)(t) + f
Jo
dF(u)[F(t u) + V(t - u-
/) + f dF (u ) f
Jo Ju
= F ° \ t ) + F (2\t) + f0
and proceeding in similar manner, we find that
I* V
(t- G)dF(n)( G<) supV(u)F(n) 0 as -> °o an
Jo
V(t) = X F{n)(t) = H(t).
n—
1
This proves the uniqueness of H(t) = X F ("}(r) as a solution of (4.1). The
n= 1
Renewal (Integral) equation can be generalized as follows:
where G and F are known and V is unknown. Equation (4.3) is called the Renewal
Type equation. An unique solution of V(t) exists in terms of G and F as can be
seen from the following.
(4.5)
o
Proof Taking Laplace-Stieltjes Transform of (4.4) we get
or
= Y - F * t y where v*(i) =
and
= F*(s)I [F*(j)]"”'
n=1
= F*W/[1 -F*(s)] if F*(s) < 1.
Inverting the Laplace transform, we get
e - A,(A?)"
Hence F (n\t) - F<n+'\t) = )=
n!
i.e. the Renewal process generated by exponential random variables is a Poisson
process, or in other words, Poisson process is a Renewal process with Exponential
Interarrival times.
In this case, H(t) = E(N(t)) = Xt - / ■ for all t > 0, i.e.
E(X)
H(t)
= A, where A 1 = E(X).
If Q(t) > 0 and non increasing and fJo Q(t)dt < °° then
Proof Assume that X2, . . . and i.i.d. with 0 < ju < °o.
and for all 0 < £ < ju, (/u - e)n < Sn < (ju + e)n for all large n.
nt < SN(tn)+} <(ju+ £) (N(tn) + 1). (4.6)
Now N(t) = max{/:: Sk < t] => SN{t) < t, SN(t)+\ > 1. Replace t by nt to get
nt > SN(tn) > (ju - e) N(tn) for all large n. (4.7)
From (4.6) and (4.7)
t N(tn) + 1
- —-— + — for t > 0 and large n. Hence
jl + £ n ju - £ n &
N(tn)
—> — a.s. as h and t > 0,
P
N(tn)
i.e. -j as t —> oo. Putting nt = r* we get the result.
tn
f A if jc„ > A
Define = (truncated r.v.)
[0 otherwise
Then X'n are i.i.d. r.v.s and X'n < Xn a.s. for all n = 1, 2, . . . .
Define S'„ = t X'k and yV'(f) = max {£: 5^ < r}.
k=\
Then S'n < Sn and N'(t) > N(t).
X'
Let P[Xn > A] = p > 0. Then is a Bernoulli random variable with probability
of success p. If t is an integer then N'(t + 1) - N'(t) is the number of times, the
partial sums take on the value t + 1.
Also N' ( t ) = %
j=o
.( N (t) N \t)Y
<E = 0(1) as t—» oo, because
72 Introduction to Stochastic Process
AT(/)
— + —— = 0(1) as t —> oo, 0 < < 1. Thisimplies
P" P “'
Therefore |fW(0
—-—, t > 11 is (uniformly integrable) u.i. by the following result.
Proof f
J \ X n \>k
f
\ X n\<
]-dP X l d P Z T E ( X ^ ) < f
K J KK
i
0 as k —» if n > n0(4.8)
since f Q(t)dt = f 1I h d t - 1
Jo Jo
Renewal Theory 73
r //(r+ h) - H{t) 1
i.e. lim --------- ;-----------= — => Blackwell s Theorem.
h ju
Proof of Key Renewal Theorem
Let I'
Jo
Q ( t - x ) d H ( x ) = f + / 2,
f
fit/ 2
where f = I Q(t - x)dH(x) and /2 = I Q( t - x ) d H( x ) .
Jo JJ //t/2
2
Since Q(t) is non-increasing and non-negative, we have
fit/2
0 < /, < Q(t/2) dH(x) = Q(t!2) H( t f 2) (since //(0) = 0)
Jo
= 1 j G 0 / 2 ) J| (4.10)
{ t2
! J'
and 0<Q- h Z 2(«/i) < hQ{0) < £ if we choose 0 < h < sJQ(0) (4.12)
M=1
If [r/2/i] = /V0 then
hit) =f i t ) + f hit - r ) f i r ) d z
Jo
which again is a Renewal type integral equation. Such an argument is called the
Renewal argument. We have seen _> 1 a s i f n > 0 (Doob’s Renewal
t fi''
Theorem) and
r Hit) r EiNit)) 1
lim ------ = lim -------------= — as t—> «>
t t ii
(from Elementary Renewal Theorem).
£(/V(r)) = 0 ^ j
We can prove that as t —> ©o. 2
V(/V(r)) = C 2 _ where a 2 = VarX,.
I*
(see Ex. 4.12)
N(t) - t/fi i rx i 2
Central limit Theorem 4.4 lim P - j L r f , 1 “ dU-
*J(ta2//u3) V2 n
Renewal Theory 75
Sn - nlju
lim P
/ —>o°
e - L 2du
because we note that t > n/x for x < 0 and t < nfi for x > 0 in the two solutions
of the above equation.
From CLT of i.i.d r.v.s,
t - nix
lim P[Sn > t] = lim p\ H— -x
«fno
-~u2
du
n - t/jx _
Now equation (4.17) shows
V(«T2/ / / 3) " ' • Theref° re’
lim P <X L2
^(rcr2/ ^ 3)
J o F
is called the Equilibrium (Stationary) Renewal Process.
76 Introduction to Stochastic Process
H(t) 1
------------- >— as » oo, wherex dF(x).
t— u
t n
Hence, the question is whether there is any Delayed Renewal Process for
which HD(t) = tip? Also we know that for Poisson process, where the underlying
variables are exponential, H(t) = tip.
A|S0 ( 4 '2 0 )
where f e*(s) is the Laplace Transform off e and/* (5) is the Laplace Transform
of/w here F'(x) =f(x).
Equation (4.20) follows from the following argument: Integrating by parts
Laplace transform of d.f. F(jc) is (F(jc))* =f*(s)Js. By multiplicative property of
Laplace transform
wt* ( o =
If Sn and S'n denote the nth renewal epochs of N(t) and Ne(t), respectively,
then for n > 1
= £(AUO) = ^ i f f
i.e. F- - —<(>) F(
F(x)
In other words, a unit is NBU (NWU) if the conditional survival probability
of a unit at age x is less (greater) than the corresponding survival probability of
a new unit.
Definition 4.4 A unit is said to be New Better (Worse) than used in Expectation
NBUE (NWUE) iff
1 ~ F(x)
dx < (>) p.
1 - F(t)
In other words, a unit is NBUE (NWUE) if the conditional expectation of the
residual life of a unit at age t is less (greater) than the expectation of the life-time
of a new unit.
The above result may be restated as
P[Ne(t) >n]> P[N(t) > n], t > 0, n = 0, 1, 2, . . .
as well as H(t) < tip iff distribution F involving the Renewal process is NBUE.
Note If F belongs to NWUE class then H(t) > j j - 1 for t > 0.
P{t)
■r P [ Y ( t ) > x \ X l = Y]dF(y).
•'o f
P ( t ) = l - F ( t + x)+ \ [1 - F(t + x - y ) ] d H ( y ) (4.26)
Now suppose that F is not lattice. Then applying the key Renewal theorem
with Q(t) = 1 - F(t + jc), we get as t —>
or lim P[Y(t)
t—>oo
< X ]= 1 - i f [1 [1
- F ( y ) ] d y = r±
PJr P'JO
[1 - F( t - y ) ] d H( y ) if x < t
P[Z(t )< x] =
1 if x > t
(since Z{t) = t - SN(t) < x Sm > t - x < 0 if x > t, so Z(t) < x is a sure event)
and if F is not lattice then,
also holds for the modified Renewal process regardless of initial distribution G.
a Renewal Reward process. Now Y(t) = I gives the total reward carried (or
n- 1
cost incurred) by time t and {Y(t), t > 0} is called the renewal reward process
generated by {Xn, Yn}.
Theorem 4.7 Suppose E(X) < «>, E | Y | < °°, then
(a) Y(t)it -> E(Y)/E(X) as t -> oo a.s.
(b) E(Y(t)/t) -> E(Y)/E(X) a s t-^ o o .
Renewal Theory 81
Proof
N( t )
--------- 1
E(Y(t)) = E 2, K„ = E
1
l " =1 /
=« + 1 )^ 0 0 - £ ( W i )
= ( / / ( / ) + 1) E(Y) ~ E(YN:t)+l).
Hence
EY(t) = j/(Q + 1 _ £ (* W i)
t t 1 ' /
Thus, it is enough to prove E(YN(t)+llt) —> 0 as t ©o.
Let 5 (0 = E O W i ) = f ^ ( ^ ( , ) +. I X, = x)dF(x)
Jo
Now i i f
[£(^1 | Xj = x) if x > t.
,
82 Introduction to Stochastic Process
E \ Y } |= f
Jo
E[ | Y\ | |X, =x] dF( x ) <oo,
Hence
t
wi+r
? Jo
h(t - x)
dH(x) + r
J l- T
i
1
h(i- dH(x)
H(t)
sin ce ■—^ — as t —^ °°
t P
—> e !\x as t -> oo (ju > 0)
H(t-T)
—> ~ as t —> 00
t
(1 - F{x)dx
fJo
= S [ 1 - F (T,\' I
n=0
J
( l- F ( ,)H (putting x = t - nT)
0
f u - F(x)]dxlF(T), since
■ Jo
1
, F(T) < 1.
hoj N ( r )
E , since = 0.
T lim nT + r
Exercise 4.3 (a) If the random lifetime of an item has d.f. F(x), what is the mean remaining
life of an item of age xl
(b) Find the mean total life time of an item when the d.f. F(x) = 1 - e ~ x > 0, A> 0.
Hence show that the mean total life is approximately 2 times the mean life when the
renewal process has been in operation for a long time.
Exercise 4.4 Let [Xk, k > 1} be a renewal process, Xk has distribution F{t), N(t) is the
renewal counting function, and H(t) = E(N(t)) is the renewal function.
(a) lim H(t) = oo
t—>00
1. Show that
(b) for every t >0 ,0 < H(t) <^ 1
1 - F(t)
2. Let H2(t) = E(N\t))
(b) Apply Smith’s key renewal theorem to the left hand side of this equality to prove
that if Xk has finite second moment ^2 then
lim Pi ~2jU2
t—>°° 2(x1
(c) Interpret this result graphically.
Exercise 4.5 Show that ENk(t) < is finite for all k > 1 and that
Exercise 4.6 Derive the alternative formula for Residual life time T(/) in a renewal
process
p t+ X
by arguing that Y(t) < x means that there is at least one renewal point in the interval
(r, t + x]. If the largest such point is Sn, then
P(Y(t) n= l
<x)= £ Pit 5„+1).
Exercise 4.7 Prove that Residual life time Y(t) and spent life time Z(t) have joint
distribution given by
P(Y(t) > jc, Z(t) < z) = f [1 - F(t + x - y)]dH(y) for t > 0 and 0 < z. < t.
j t-z
Exercise 4.8 (Uniform Renewal Process) Let Vh V2, . be a sequence of i.i.d. r.v.s with
EV, = 0, EVi2 = (T2.
Assume also that {Tn} and { } are independent and Fx is the distribution function of
7j. Define a stochastic process {X(/)> / ^ 0} as follows.
If 0 < / < Tx let X(t) = Vlf if + . . . + Tn_x < t < Tx + . . . + Tn let X(t) = Vn+X
(n = 1,2,...). Show that {X(0} is stationary with covariance function K{t) = cr2(l - Fx{t)).
Also assume that F{ = — I (1 - F(x)dx and m is the expectation of F, the common d.f.
of r 2, r 3, ______ m°
Exercise 4.9 Suppose that EV{ < for some r > 1 and the renewal process is arithmatic.
Let Y(t) = SN(t+l) - t be the residual lifetime. Show that
(a) E(Y(t))r ^ ^ — EX| as t —^ °°
rfi 1
2 2
(b) E(Y(t)) —> ° ■ + ^ as M «, where EXx - ju and var (X{) = <72 <
Exercise 4.10 We say that the random walk (or renewal process) is arithmetic if the
common distribution function F has its support on {0, ± d, ± 2d, . . .} (or {0, d, 2d, . . .})
for some d > 0. The largest d with this property is called the span. A random walk or
renewal process which is arithmetic with span d is also called d-arithmetic.
n
If the renewal process is d-arithmetic, then show that un = /„ + Z un_kf k where
k =0
Exercise 4.11 Let 0 < fx = EXx < Then show that for renewal counting process N(t),
E(N(t)/i)r -» — as -t > °°
Exercise 4.12 Suppose that 0 < \i = EXx and a2 = Var(Xx) < «>. Show that
(a) E(N(t)) = t/fx + 0( V7) as / -> «
(b) Var (N(t)) = — - + as -> «>.
Exercise 4.13 Let Yt = SN(t)+l - t be the excess life at time t in a renewal process and
Z, = / - SN(t) be the spent life.
Then show that
Exercise 4.14 (Replacement policy) If f(x) is a p.d.f. associated with a lifetime d.f.
F(x), the hazard rate is r(x) =f(x)/[l-F(x)]. Suppose T, is the cost of the /th replacement
under age replacement policy where a planned replacement at age T costs Cx dollars,
while a failure replaced at time x < T costs C2 dollars. Show that the long run mean cost
per unit time is given by
0(7) = Ct[l - F(T)] + C2F(T) / [1 - F(x)]dx and also show that the
replacement age T* that minimizes 0(7) must satisfy
r C
r(r* )J [1 - F(x)]dx - F(T*) =Ci _‘c ^ .
Exercise 4.15 Let Nx(t) and (N2(t) be two independent renewal processes with the same
irfteroccurrence distribution F with mean i. Let N(t) = Nx(t) + N2(t). If N(t) is also a
renewal process, then show that A)(/), N2(t) and A(/) are all Poisson.
5
Branching Process
(i) 0„+1(z) = k=
Z0 m B+i = *]z*
= l P [ X n = l][0(z)]1
= 0„(0(z)).
Also 0 o(z) = P[X0 = l]z = z for n = 0
0,(z)= Z P[Xo = /]0(z) = 0(z)
1=0
= mn+'0E
(X) = w"+1, since />(X0 = 1)
Note If we assume / ’(Xq = n0) =1, then EX„ = n0m".
We also get from 0 „ +,(z) = 0(0„(z)),
m n- l0 " ( l ) = 1) + Km2n-(,,-])
Adding both sides of the above equations, we get
0 " +i (1 ) = mnK+ Kmn+] + ... + Km2n
- 1 +
wj ”+* —1
= mnK --------— ifm * 1.
m- 1
90 Introduction to Stochastic Process
am” if An * 1.
i(1 ) = 0J,'(1) + K
0"+ (from (5.6), putting = 1)
If m = 1, 0+
n\' 1( ) = (n + 1)K.
This is a particular case of Wald’s Identity for second moment and follows
from (i)
Now Var(Xn+1) = E(Xn) Var(^) + ( E ^ f Var(X„)
= m V + nr Var(X„) (5.7)
This is a first order difference equation.
A particular solution of (5.7) is
= 0(7T).
Now 0(z) = z=> 7t is positive and hence 7Tis the positive root of 0(z) = z-
Assume that S0 is another positive root of 0 ( z ) = z. Then q0 = 0 < s0. Assume
qn < an^ which implies
qn+\ < s0 (v <7n+1 = 0 ( ^ ) < 0(so) = s0)
Now 0 „ (0) = P[Xn=0]=qn <1.
92 Introduction to Stochastic Process
Example 5.1 P[% = 0]= ju > 0, P[£ = 2] = A > 0, P[£ = 1]= 1 - ju - A > 0.
/ '( z ) = - + P2 + 2 p 3z + .. .
"7^
m Case 2 m < 1
Then j8 = 1. Hence
i
f'(z) < 0 in z for 0 < z < 1 orf(z) 4.
/(0 +) = o o /(l)= 1.
t Hence, there does not exist 0 < a < 1
such th at/(a ) = 1 and hence z = 1 is
0 the only root of 0(z) = z.
Theorem 5.2 If m > 1, then with probability n, the population become extinct
and with probability 1- n, the population explodes. If m < 1, then with probability I ,
the population becomes extinct. [Regardless of the actual value of the mean
EXx = m > 1, the probability that the nth generation will consist of any positive
finite number of individuals tends to zero as n —»©o, with probability of extinction
tending to n. In these circumstances, we say that Xn —>0 as n —> with probability
1 - n\.
Proof qn = P[Xn = 0],
Therefore, lim 0 n+\(z) = lim 0 ( 0 n (z)) = 0(lim 0 n (z)) and hence y= 0(y).
ft—>°° n—>°°
If 7t < y < 1, then /(y ) < 1
is a contradiction to 0 (y) = y.
If 0 < y < 7T, 0 (y ) > y and again a
contradiction arises.
Therefore lim 0 n(z) = tt, i.e. 0 n(z)
f t—» °o
1 - Pi"
- d - /» r ) + p " z
Hence qn - P (X n = 0) = 1 - p[7—> 1 as n —> <».
E(T) = £ npl Vo 2 1
m= 1 n=l
__ 1 Po 1
°(1-Pi)2 p% Po
Example 5.3 It is frequently assumed that the problem of a mutant gene having
k distinct discordants {k = 0, 1, 2,. ..) is governed by a Poisson distribution with
mean A = 1.
Then 0(z) = ez~x and n - 1, because
0(z)= £ = e -A (l-:)
*=o a:!
0(z) = z has the explicit solution for A < 1 though for A > 1, the extinction is
proved with probability 0 < n< 1 (iff A > 1). If A = 2, then z = e2(z~l) has a root
0.2 < 1 and 7T= 0, 2.
The graph of 0(z) in 0 < z < 1 between (0, e~k) and (1, 1) 0(z) is convex and
that the curve y = 0(z) always lies above y = z when A = 1. There is no other root
of z = 0(z) except 1 in [0, 1].
Hence the probability of extinction is 1.
Wn = — ,n = 0, 1 ,2 ,...
mn
Claim W„ is a Martingale, i.e.
E ( W n+r\W ,n Wn_u W0) = Wn a.s. for r = 0, 1, 2, . . .
=— — Wn a.s.
m
96 Introduction to Stochastic Process
EXl m n- ' ( m n - 1) 2
Also = m 2n + ------ -----;---- - a 2
m In m- 1
= 1+ (1 - m~n) —» 1 + if m ^ 1
m(m - 1) m(m - 1)
= coefficient of sk in 0(sGn_\(s)).
Branching Process 97
Proof For 0 < s < 1, R\{s) = s0{s) < s ( v 0 (j) < 1).
R2(s) = ^0(/?j(j)) < s0(s) = Rfs). Assuming Rm(s) < Rm_\(s), we get
pk = bck \ k = 1 ,2 ,.. . , 0 1
= 1- bz
1 - cz
Let pijit) denote the conditional probability that the system consists of j
objects at the instant t + T if there were i objects at the instant r. To solve the
problems that arises in the theory of branching processes, we shall find it convenient
to use the method of generating functions. We, therefore, introduce the generating
function G/(z, t) of the distributions for j = 0, 1, 2, . . .
Since the individuals act independently, the probabilities pik(t) for the fixed i
correspond to the distribution of i independent r.v.s all having the same distribution
determined by Gi(z, t).
Thus, we have
G,(z, 0 = [G,(z,/)]' (5.17)
The formula (5.17) characterizes and distinguishes branching process from other
continuous time Markov chains. In view of the time homogenity, the Chapman-
Kolmogorov equation takes the form
Pik (t + ?) = ^ Pii (0 Pi k (r )- We shall now obtain a continuous time analogue
7=0
of the relation (5.1) satisfied by a Galton-Watson process.
Lemma 5.3 G,-(z, t + t) = [G(z, t + t)]1= [G(G(z, t), T)Vi i - 0, |z| < 1 and in
particular,
G ( z ,f + r ) = G(G(z, 0, t ). (5.18)
Proof Using (5.16) and Chapman-Kolmogorov equation, we get
[G(z, f + t ] 1 = S pu (t + r ) z 7, i > 0
r=0
= Iz^ £ P ik ( t ) p k j ( T )
7=0 k=Q J
= 2 p ,* (o f 2 p jty (r)z y)
*=o V/=° 7
(interchanging the order of summation by Fubini’s Theorem)
Theorem 5.6 Let X(t) be a Markov branching process with infinitesimal transition
probabilities defined by (5.19) and (5.20). Then the generating function of the
probabilities obeys the equations
^ = «(G) (5.21)
and = (5-22)
= - A,G(z, 0 + 2 A *G * (z , 0 , ( 5 .2 5 )
Ot k=0
k*\
where we set G(z, t) = Gj(z, /). Finally we obtain the following non-linear
equation
3G (z,/)
= m(G(z, t)) with initial condition G(z, 0) = z. (5.26)
Or
Branching Process 101
Suppose that the conditions under which the second system of Kolmogorov
forward differential equations may be applied are satisfied. From the definition
of the branching process it follows :
Pkk(t) = (1 - k xt f + o(t) = 1 - + o(t),
k-i(tP)= *(1 - Xxt)k~xX0t + o(t) = kX0t + o(t),
Pk,k-j(t)= o{t),j > 2
Pk.k+jiO = k(l - Axt)Aj+xt + o(t) = kk J+\t + o(t), j > 1.
From which it follows (in the notation of Section 6.1) that
*7jj ~ = A>. Q j j - k = 0» k ^2,
Qj\j+k = j^k+1’ j — 1•
G(z, 0 = V (5.31)
dz
where y/(t)is the inverse of the function t - <j>(z) = fJo u(z)‘
This solution coincides with (5.30).
102 Introduction to Stochastic Process
(1 - P)2[\ - eKq
G(z,t) = 1 2 (5.33)
1 _/?-V<9-p) n=\ IP-
1 - e,iq~p) (5.34)
P\o(t) =
1 - p - le,(l’-,,)
„ , [1 - g '(< 7 -p )l" -l
P\n (0 =(1 - P) [ 0 _ e H q - p ) ] n +\
,n>
------ 1
---------- (5.35)
Jo p (l - z ) 2
1
P 1-z
z , = v(t ) =1-1 +1pt' (5.36)
(5.38)
(1 + pf)”+l
The following asymptotic relationships as t —>°o are immediate consequences
of formulae (5.34) to (5.38): If q < p, then p 10(f) —> 1. P\n(t) —> 0 for n > 1 as
Branching Process 103
t —» oo. If q < p, the branching process degenerates with probability 1, i.e. all
objects eventually disappear. If q > p, the probability of degeneration of the
process is equal to j8 = plq < 1. However, if the objects do not disappear the
number of them increases without bound in time, because
i N
P(X(t) > N\X(t) > 0) = 1 ------ — Z 1 for every N.
Pio(0*=i
Let us look at the question of asymptotic behavior as t —><*>of a branching
process in the general case. In what follows we shall need the moments of X(t).
Thus Afj (r) = lim 0] (z, 0 = lim exp fJo u \G )dt = e m' (5.41)
d<h(z,t)
= u ' ( G ) h ( z , t ) + u " (G )tf (z, t),
di
d<t>k(zt) (5.43)
= u'(G)<j>k(z,t) + ii/k(z,t).
dt
where \pk(z, t) is a polynomial in ^(z, t), ty2(z, t),... y<t>k-\(z, t) and the coefficients
depend on m"(G), .. ., u{k)(G). The initial conditions are given by 0*(z, 0) = 0 for
k > 2. The solution of equation (5.43) is of the form
( tUl- (e mi‘ if *0
m,
(5.44)
( m2t if m{ = 0
The following theorem illustrates under what conditions the branching process
become extinct.
Theorem 5.7 The probability of degeneration of a branching process coincides
with the smallest non-negative root of the equation u(x) = 0. If u ( 1) = mx =
Now p 10(0 cannot be equal to jc0 for all t > 0, because if Pio(^o) were equal to
x0 for t0 > 0, then by virtue of uniqueness of the solution of equation (5.45) we
would have p\o(t) = jc0, which is impossible. Since the limit q = lim p 10 (t) < 1
t —»oo
exists, it follows from equation (5.45) that lim p{o(t) = u(q) also exists by
continuity of u and this implies u(q) = 0. Otherwise the quantity
Exercise 5.4 Let c\n = P{Xn = 0) = 0„(O), where Qn is the nth iterate of the
probability generating function 0(5) = £ pksk. Prove that qn = 0w_i(po)» n> \ .
k- 0
Exercise 5.5 Let Xn denote the number of offsprings in the nth generation of
a discrete time branching process. The function (pn(s) is the nth iterate of the
probability generating function 0(5), m = EXx, a 2 = Var(Xj) and P(XQ= 1) = 1.
1. Assume n > I. Show that the bivariate generating function of Xt and Xn is
£ £ P i x , = k, X„=j] ukx ’ = ft
k=07=0
2. Prove that E(XnXt) - mn~’E(Xf ) if n > I.
3. Find Rnh the correlation coefficient for Xn and Xh and discuss the behavior
of lim Rni for fixed values of /.
n—
>°°
Exercise 5.6 Show that 0(z), the generating function for each of the cases (i)
Poisson, (ii) binomial (iii) negative binomial the offspring distribution satisfies
a differential equation of the form (c + dz) 0'(z) = a + bQiz), where a, b, c and
d are constants. Conversely, if such a differential equation holds for 0(z), show
that the following recursive formula applies:
if P(Xn = k + 1) > 0, then, for k > 0 and n > 1
(c + dqn_i)P(X„ —k + i) = aP(Xn. , = k - l ) + j
Exercise 5.7 Let m > 1. Assume that X0 = 1 and 0 < EXf < °o. Show that the
sequence Xn/EXn is an a.s. convergent martingale.
Exercise 5.8 At time 0 a blood culture starts with one red cell. At the end of
one minute, the red cell dies and is replaced by one of the combinations with
probabilities as indicated: 2 red cells with probability 1/4, 1 red cell and 1 white
cell with probability 2/3, 2 white cells with probability 1/12. Each red cell lives
for one minute and gives birth to off springs in the same way as the parent cell.
Each white cell lives for one minute and dies without reproducing. Assume that
individual cells behave independently. What is the probability that the entire
culture dies out eventually?
Exercise 5.9 Let En be the event that extinction occurs at sometime after n and
Yn = Xn/(na2), a 2 > 0. Show that if m = 1 and 0"(z) < 00 then
P(Yn < y | En) -> 1 - €“2^, as n - ) 00
Exercise 5.10 (Branching immigration) Each generation of a branching process
(single ancestor) is augmented by a random number of immigrations who are
indistinguishable from the other member of the population. Suppose that number
of immigrants are independent of each other and of the past history of the
process, each such number having p.g.f. H{z). Show that 0w+1(z) = 0n((p(z)) H(z)-
6
Continuous Time Discrete State
Markov Processes
Now define p t f = P [X (0 i]
= P[X(t + s) = j \ X ( s ) = i)
(by stationarity) as the transition function of a continuous parameter Markov
process. Let (X(r), t e R+] be a Markov process with countable state space S and
tanstition probability function p ^ t ), / yj E 5, t e R+.
We assume the following axioms about transition function Pij(t)\
1. 0 < py{t) < 1
2. ^>s Pij(t) = 1 for all / = 1, 2, . . . and t > 0.
3. Chapman-Kolmogorov equation (or semigroup property)
Pu (t + h) = Zsp ik( t)pk(j
4. p,j(0) = S,j
5. lim Pij (t ) = <5<y (makes Markov process standard or regular)
(4) P(0) = I
(5) lim P(t) = I.
do
Suppose now P(s) for 0 < s < his known. For each t we can find an integer
n such that nh <t < (n +1 )h. T h e n
P(t) = P(nh + t - nh) =P{nh) P(t - nh) (From (3))
= P(h + h + . . . + h) P(t - nh)
n times
lira^ = lim W M
A'-lO ^ A-iO $
H n)r'° (b)
Pait) = Zp iji tln )P j
^ Pu Pu
1 - Pu( t )
-Pu (0) = Jim
t
By semigroup property pu(t + s) > pu(t) pu (s).
Taking log on both sides and writing - log pu{t) = 0(/)
110 Introduction to Stochastic Process
(this is well defined since pu(t) > 0 for t > 0), we get
0(t + s)< 0(t) + 0(s) (6.3)
Since 0 < pn(t) < 1, 0(f) > 0.
iLet♦ qu = sup ------
0 ( 0 . Then
tu 0n <
^ qu <
^ <*>.
t> o t
If qlt < oo, there exists f0 > 0 such that 0(fo)/fo > qi{ - £.
By (6.3)
0(fo) < 0(nt) + 0( 8) < 0((n - l)f) + 0(f) + 0 (5 ) < <n0(t) + 0 (5 )
if f0 = nt + 5, 0 < 5 < f, n = 1, 2, . . .
Hence 0 ( * o ) ^ /i0 (O + 0 ( 3 )
{o 10
0 (5 )
=^0(t)/t +
to to
m 0 (f) 0 (5 )
Therefore qu - e< lim (6.4)
rro h ' t + t0
nf
But f —> 0 =>------> 1 and 0 (5) —» 0 (by axiom (5) and pn(5) —> 1 as 5 —>0).
to
nt_ 0 ( f ) 0 (5 )
Therefore lim 0 (f)/f = lim (6.5)
t —>0 / —>0 L
f0 f fo
Now
f->0 f /->0 0 (f) f ™
(since 0(f) —> 0 as f —> 0, by L-Hospital’s rule we get the last equation).
Continuous Time Discrete State Markov Processes 111
(b) Let c E (1/2, 1) be fixed. Then by lemma (6.2), there exists an 5 > 0 sufficiently
small so that pH(s) > c and pt]{s) > c. Let 0 < s < nt < <5. Consider a stationary
discrete parameter M.C. {Xn }q with state space S such that
P(Xn+I = j \ X n = i)
Then for all n>
1, we have
Point) = P(X„ = j \ X 0 =i) > P(Xl = j, X„=j \ Xo i)
Since each term on the right corresponds to a possible way of going from i to
j in H-steps (relative to (py-(f)) ) and these paths are mutually exculsive but not
necessarily exhaustive.
On the other hand,
P(X{ *j , . . . ,X r_! * j , X r = i \ X 0 =i)
= P(X, *j , Xr_2* j, Xri) - *j,
*j , Xr_, = y, X r= iIX0
= P{Xx *y, . . . , Xr_3 * j, X, = I X0 = i) - *y, . . . , Xr_3
^ y\ Xf_2 = y, Xr = i|Xq = i) —PCX] ^ y , ..
X. = / 1X0 = 0.
and so on. Continuing in similar way we obtain
P(Xl * j , . . . , X r_l * j , X r = i \ X 0 =l )
P( Xr = i \ X 0 = i ) - l P( Xx* j ,...,X * _ , *y,X*
k<r
= / | > 2c - 1.
Consequently,
Pij(nt) > (2c - 1)npij(t). Let h < 8, t < 8, and n = [h/t],
where [_*] is the interger part of jc; then, Pij(t)lt < Pij([h/t]t)/{[h/t]t0(2c - 1)}.
Letting, t —> 0+, we obtain
lim sup (Pij(t)/t ) < Py(/i) {/ic(2c - 1)} < ©o for all r > 0.
Therefore,
lbn sup (p i; (f)/r) < Hm in f(/^ (/i)/{/i(2c - l)c},
which means that the limit qtj = lim (Pij(t)lt) < ©o exists.
112 Introduction to Stochastic Process
N
(c) We shall prove <7,,<°° for a finite state process. Now 1 = /?,,(/i) + Z Pu(h ),
7 =1 7
j*i
N
where N is the number of states, or 1 - (h) = Z p u (h).
7*‘ 7
Dividing by h and letting h —>0 and by part (a)
N
A, if / = / + 1
qu = + Mi and qij = 0 if j * i - 1 or i + 1, i = 0, 1, 2,. . .
Pi if j = i - 1
= 2. pik(s)pkj (?) - [1 - (t
„ . V Pik ( s ) P k j ( 0 ~
Claim hm Z =Z (?) (6.7)
. . Z P ik(s) , v
hm inf Z — -— />*, (?) > Z < 7 ,^ (?) ( 6 . 8)
viO k*i S J k±i J
N
Take N > k,then ^ p ik{s)pk](?) = ^ p lk(s)pkj (l) +
(?)
- ^ 0 Pik(s)Pkj(t)+
N f N >
= k2= 0 Pik i s) P k j (?) + 1 - A:=l
Z Pik (s)
V
/V
= £ 0 Pik(s)Pkj(t) + 1 - / > „ (•*) - z
2 : P«* ( i ) -
Hence Z kjJ)< £
Pjk^lP Pu^Pkj + 1 ~ P »(s) _
5 ~~ k * i S S k*i S
TV, f r V ik(s)jt ) £
P / N £
Therefore hm sup Z ------- -------- < Z <7,*p*; (r) + qn - Z qtk
v io J k±i
A“ sup* ? .~ —
1 P kjM i& qikP kjif)
( 6 . 10)
.. Z Pik(s) ., , , , .. ■ t Z P i j ( S') , ,
lim sup Z. — -— < Z jQ)^ lim inf
ikP
.vio 5 J k*i J S
I oo OO
= I Pik(s)[Pkj(t)-
and hence the result (ii) can be proved similarly as (i) after dividing by t and
letting t -> 0.
Since the scope and validity of the forward equation is rather involved we are
not discussing any more details of the proof.
( hj* } \
<
aU ~ r •
\-<hi if* =JJ
The important problem is, however, to construct transition probabilities p^t)
having prescribed infinitesimal matrix A satisfying atJ > 0 for all /, j e S, i = j
and
-an = 2 atj (6.11)
= U A (I),
0
where a (0 =
for all i e S, where a'"’ are the elements of An, and A = (a,;j-
Proof of (6.16) By (6.14), (6.11) and by induction on n, we have
| a-p | < a (2a )n~x, for i,j e S', n > 1 (6.17)
Thus, the series on right side of (6.16) converges to an analytic function /i,y(r),
say, absolutely for all t. Hence term-by-term differentiation of this series for
lift) one can easily verifies Kolmogorov’s backward equation with initial conditions
p,;/(0) = 0 for i * j and pn(0) = 1.
To verify the Chapman-Kolmogorov equations, note that
= etA<?A = P(t)P(s)
116 Introduction to Stochastic Process
with initial conditions P, (0) = I <5(,- = 1 for i e S. Since Pt (t) = 1 for, all t > 0,
j eS J
all i e S clearly satisfy these equations, we have Pt (t) = 1 for all t by uniqueness
of such solutions. Solution (6.17) have been shown to satisfy all conditions for
being transition functions except for non-negativity which will follow from the
following more general result.
Theorem 6.3 Given any infinitesimal transition matrix
Pi' (t ) = ’L a i kPkJ(t),i,jeS(6.18)
e Xi'PikU) = S ik + 2 f ,
J*' Jo
-A q A0 0 0 0 \
Mi —(Aj + P i) Ai 0 0
0 M2 -(M2 + a 2 ) A2 0
v : : : : J
where A, and P p i - 0, 1, 2, ... ,7 = 1, 2, 3 ... are the birth and death rates,
respectively.
Definition 2 (Karlin) A birth and death process is a continuous time Markov
process on the state space S = {0, 1, 2, . . .} with stationary transition functions
P i j ( t ) = P(X(t + 5)) = 7 | X (5) = i) and such that
(i)PlJ(0) = S0
(ii)Pu+\(h) = Xjn + o(h) for i > 0 and h i 0
(iii)Pu-\(h) = pth -v o(h) for i > 0 and h i 0
(iv) Pn(h) = 1 - (At- + Pi) h + o(h) for i > 0 and hiO and
P i j ( h ) =o(h) for7 * 1 + 1, i - 1, i as h i 0 and all i > 1.
(v) A/ > 0 for i > 0 and p t > 0 for / > 1 and p$ = 0 are the birth and death rates,
respectively. A, and p( are also called parameters of birth and death processes.
Note These two definitions can be proved to be identical.
The backward Kolmogorov equation will be given by
Pij ( 0 = MiPi-ij(0 - (Ay + Pi )pu (t) + A,/?/+1 (0 , i > 1
and p',y(/) = - A0p„y ( 0 + A0p i;-(O with initial conditions p,y(0) = <5^.
The forward Kolmogorov equation for birth and death processes are
Pi j ( 0 = Ay-l Pi ,7-1 ( 0 - (Ay + Pj )Pij (t) + /iy+iPj-,y+i (0 ,7 ^ 1
where £ is over all k * i - 1, i, i + 1. Using axioms (ii), (iii) and (iv) we obtain
k
* *
^ Pi k ( h) pkl (t) < I . p ik(h) = 1
k k
= 1 - [1 - (A, + \xi)h + o(h) + iijh + o(h) + Xfi + o(h)]
= o{h) as h —> 0.
Hence ptf t + h) = h p ^ / t ) + (1 - (A, + ^,)/i) + A,/i + o(h).
Transposing the term py(f) to the left hand side and dividing the equation by
h, we obtain (after letting /i —> 0 the backward Kolmogorov’s equation)
Pij(t)= -iP
\j(t) (A, + pdpijit) + Xi pMJ (t
^ = 2 , p'N+n(t)zN+n=I., [-P(N+n)pN+n(t)+P(N + n-
Ot n=0 n=0
= -p %A N + n) pN+n(t)zN+n+ p l ( N + n - l ) PN+n_i(t)zN+n
n-0 n=0
Continuous Time Discrete State Markov Processes 119
,SG a 2 SG
8z
Now u ( j c , y , z ) ^ + v(x,y, z ) ^ = 0 ( x , y , z)
or ^ = j3z(l - z) => Pt = c2 -
Elemenating one of the constants Ci and c2 we get the general solution given
N
y/(u) . Substituting u - g & we get
G(t, z) = (6.25)
1 - z ( l
To see why the last equality holds, define An = jx j^ /j = /,/: = 0, 1 ,... nj.
i 00
Then An Tin n and An = lim A = n An = {X(^) = i for all s in [0, /] which is
n—>°° n=\
a rational multiple of t] = {Tt > r)^ c A.
Since the sample paths are step functions, if a sample path is not in {Tx > t]
then there occurs a jump to state y, different from /, at some r0(0 < t0 <t). The
case t0 = t may be excluded, since it is not in A. Because each sample path is
right-continuous step function, there is a time point tx > t0 such that X(s) = j for
tQ<s< ti. Since there is some 5 of the form s = tk/n < t in every nondegenerate
interval, it follows that X(^) = j for some s of the form (k/n)t < t\ this implies that
this sample path is not in An and hence, not in A. Therefore, {Tx > i ) d A .
Hence {Tx> t] = A.
f n- t ^ (t\
F(t) =limX(t) = i x ------ = ; , . . . , X - = i, X(0) = i
n—
>°° V « J Vn J
x P X(0) = i
n- 1
= lim P X(t) = i /,. . . | X(0) = i
n—>°° n
(by Markov property)
-in
= lim ---- qH + o
n—>°° n
Continuous Time Discrete State Markov Processes 121
[1 - + o( + [pn+l(t)A
[1 - + o(At + )] ) (« > !)
Neglecting the terms of o(At)
pnN(t + At) = [1 - (A„(r) + pn(t))At]p+ A„_i N(t)
+ Hn+i(t)Atp„+lN(t) + (n > 1).
Since ‘0’ is an absorbing state, p^{t)= 0 and we get
p0N(t + At) = (1 - Ao(t)At)pON(t) +
Taking limits as At —» 0, we get
S P nN (t)
St :
(n > 1) (6.26)
SN
Pt()
O
^ ---- = -Ao(t)POiV(0 + (6.27)
+ I ' Zn p „ - w ( t ) A n - ! ( t ) +
n=0 n=0
G (z,t)= f
1- z f
Jo
A(r)eA(t)dr
Continuous Time Discrete State Markov Processes 123
r r l
Letting a(t) = 1 - + eh w X{T)dr and /3(f) = 1 - eA(T)[ - a it)], the
Jo
G.F. can be written as
ait) +[1 - « ( f ) - m u
Giz, t) = (6.31)
1 - /j(0z
Let us first calculate pnit) = P[Xit) = n | X(0) = 1] in the simpler case when
An(t) = nA(t) and p„it)= njuit) and population start with one memb
Then the Kolmogorov’s equation becomes
dPnif)
=in + l)p it) p n+1it)+in-l)Ait)pn-iit)-niMt)+Pit))Pnit), « ^ 1
dt
The fundamental property of this Recatti equation is that the general solution
Hence
/ i ( 0 ) + / 2(0) z h -/■
G(z,t) = / 2 - Z /4 g\
zh -f\ gs(t) + zgi(t)
/ 3( 0 ) + /4(0)
h
where g2,g2 and g4are functions of t.
On expansion, we get
Po(t) =a(t) and pn(t)= {1- Po((t)}1 - (« ^ D
where a (t) and fi(t) are functions of t. Thus for generalized birth death process
the population size at any time t is distributed in a geometric distribution with a
modified zero term.
In the generalized birth and death process starting with N individuals the
probability of extinction is given by
1N
1
PON (0 “
eA<'> + J A ( r ) e A(T)d r
Now
eA(t) + f
Jo
A ( r) e A(r)d r = eAU) + f
Jo
p(z)e*{T)dz - f
Jo
e A(T)[u(z) - A(z)]dz
= eA(0 + f
Jo
p( z) e * {T)d z - [eA(T)Y0 = 1 + f
Jo
/ u (z) e^T)dz.
f
Jo
jn(z)eMT)dz
Hence, Pon(0 =
1+ p ( z ) e MT)dz
Jo
We observe that chance of extinction tends to one as t —» °o iff
lim f
'-*°° Jo
p ( z ) e A{T)dz = °o.
Since lim I jit exp [lit2/2 - Xr\dr = «>, the extinction is certain in Arley
'->°° Jo
process. There are four important special forms for birth and death rates. If
Xn = a, a constant we have immigration at rate a. If Xn = nX, we have simple
linear birth process with birth rate X per individual. If jin = /? and = 0, we
have emigration at rate /}. If ^ = n/i we have linear death process with rate fi
per individual.
(6.32)
jx( 1 - e {X- ^ ‘ )
a(t) = and P(t) = (X/ii)a(t)
The general solution pnN{t) is the coefficient of zn in the expansion of G(z, t).
(6.33)
f-A A 'N +i- r
Since | fi(t)z | < 1 and = (-D '
' J V ' >
AA N + i-V
[1 - p(t)z]-N= I (-l y m t u y = z [P(t)Yz‘ (6.34)
1=0
^ *i / )=0
1 if p >A
lim pon V) = (6.35)
( p /A ) N <A
126 Introduction to Stochastic Process
1 if ju>A
lim G(0, t) =
/—»°o
(6.36)
(ju/A)N if (jl < A
Equation (6.35) shows that it is certain that the population dies out when the
death rate exceeds the birth rate, but when the birth rate is greater than the death
rate, the probability of eventual extinction is equal to (plA)N. From (6.36), since
the limiting P.G.F. is constant, the probability that the population will increase
without bound is 1 - (jn/A)N. The relative magnitude of A and ju also influences
the asymptotic value of the expectation and variance of X(t).
The moments can be calculated for linear birth and death processes by taking
derivative of G (z, t) at z = 1. We get
m(t) = EX(t) = n \ — ~ = ,
1
0 if H > X
Hence lim m(t) = N if X = n
/ — >oo
OO if A > n
if /i > A
and lim
/—>oo
a 2{X{t=
if ju < A
When A = jl we have an interesting case in which though the probability of
extinction tends to one, yet the expected population size tends to N. Most population
will eventually become extinct while a few others will attain huge sizes, so that
the average size is N.
~ ( z - l ) ( A z - / i ) |^ = y (z -l)G
or G(z, = C,(Az -
Az - /x ,(M-A)f
From (6.37) and (6.38) G = 0 (Az - p) y/A for some unknown
V
z “- A1 /
function 0 to be determined from initial conditions.
Initial conditions give G (z, 0) = 1.
Az - p
Hence (Az - n) -ya = 1.
z- 1
„ - Az - /LI . 0- p
Put 6 = ------ — i.e. z = ■, then we have
z- 1 e -A
y/A
0(0) = e ^ a -jo ^ e -A )-^
(A - /x)/(Ae(A- ^ -
where P= •(6.40)
(1 + At) * if A = /x
(applying L’ Hospital’s rule)
and q = I - p .
From (6.39) we see that X(t) has a negative binomial distribution. As
(6.40) shows that p -» 0, <7 —> 1 if A > p, while p —> 1 - A/p, g —> A/p if A < p.
128 Introduction to Stochastic Process
Therefore, if A > ju, no limiting distribution exists for X(t) as t —» <*>, i.e.
X(t) —> oo with probability one. If X < ju we find from (6.39) that
7
Therefore, lim m(t) = ------ r- if A * u.
t->oo // - A
These formulae relate to the stable distribution of population size which
immigration can just maintain against the excess of jU over A. It is clear from
(6.39) that the distribution will still be negative binomial in form for every finite
value of t when fj. = 0, the process being one of immigration and reproduction
only. On the other hand, when A = 0 so that the process is one of immigration and
mortality (or emmigration) only the distribution assume a Poisson form.
aPo(t)
+p =—[ j+ ( pa+ojS( 0J ) - ~ r ) c - <a+/,)'
a + fi = -a A+ P^ + - ^ ■ (a+/’,'
if po(0) = 1.
Exercise 6.3 (Continuation of Exercise 6.2) Let ptj(ij = 0, 1) be the expected length
of time in (0, t) that the Markov process [X (/), t >0} with two states {0,1} spends in state
j, having started at X(0) = /(/= !, 2) initially. Then show that PijiiJ = 0, 1) are given by
a [1 _ e-a]
Poo(0 = pt/(d+ P) +
(a + P)2
Continuous Time Discrete State Markov Processes 129
at a
M o i(0
(a+P) (a+p)2
Pt P
Pio(0 (a+p) (a + P)2
at P [1
Pu(t) a+P)
( H (a+P)2
Exercise 6.4 (Contagion or Accident Proneness) Suppose that the probability of an
accident occuring in tto / + Atis
previous to time t, for v. Ay> 0. Show that this is a pure birth process on the set of integers
S = {0 , 1, 2, . . .) which satisfies the forward equations
Pon (0 = -(u + nX)p0n (t) + (v +(n-l)A)p0i„_l (
Poo(t) = -vpoo(t)
and the transition probabilities of ) are given by
- j - v/X^
P j t U ) = e ~ (v*j x v («rA' ( 2)
V k-j ,
In particular show that
v(v + A)... (v + nX) (j _ )w+1
PVn+\ ( 0 “ (3 )
(n+ l)!An+1
Exercise 6.5 (Branching Birth and Death Process) Consider a collection of particles
which act independently in giving rise to succeeding generation of particles. Suppose that
each particles, from the time it appears, waits a random length of time having an exponential
distribution with parameter 6 and the splits into two identical particles with probability p
and disappears with probability 1 - p. Let X(t)y 0 < / < «> be the number of particles
present at time /. This branching process is a birth and death process. Find the birth and
death rates.
Exercise 6.6 In exercise 6.5 suppose that new particles immigrate into the system at
random times from a Poisson process with parameter A and then give rise to succeeding
generations as described in the previous exercise.
Find the birth and death rates of this birth and death process.
Exercise 6.7 (Application of Birth and Death Process in a Telephone Traffic Problem)
Consider a telephone exchange where the number of available lines is large that for all
practicle purposes it can be considered infinite. Assume that Y(t), the number of lines in
use at time t follows a birth and death process with P(Y(t +h) = i+\ \ Y(t) = /) = Xh + o(h),
t > 0, h > 0 and i > 1.
Also P(Y(t + h) = i~ 1 | Y(t) = 0 = Pih + o(h) as h -> 0
(In the first case we have assumed that a call will occur in (r, t + h) is independent of
the number of busy lines at time t): Asume also that p, = ip. Find the probability generating
function (P.G.F.) of Y(t), P(Y(t) = k) and E(Y(t)).
Exercise 6.8 Consider the Feller-Arley process (linear growth process) which is a
hirth and death process with birth and death rates A„ = nX and pn = np respectively for all
n ^ 1 and initial condition P(X(0) = n) = 6nl.
130 Introduction to Stochastic Process
(a) Show that the generating function G(z, t) of the size distribution at time t satisfies
the partial differential equation
dG 2xdG
- f o- = [ n - ( k + n ) z + k z 1} - ^ .
(b) Show that the forward equations have the unique solution given by
2
j=k
Pi i (t ) = 1 iffj - k^ 1/Aj = for all finite k. (1)
that X(t) —>oo a.s. as t —>«> iff £ —- < oo. Hence show that a Yule process or a linear
*=° Xk
growth process (i.e. Xk = kX) does not move out to infinity.
Exercise 6.12 (a) Show that for a stationary distribution to exist in a birth and death
Continuous Time Discrete State Markov Processes 131
00 A A A
process it is necessary and sufficient that the series £ —— (pk > 0, k > 1)
*=1 P\Pl - Pic
converges (A*’s and p*’s have their usual meanings).
(b) (The servicing of lathes) Suppose that m lathes are serviced by a crew of a repairmen.
When a lathe fails to function properly it is repaired immediately unless all of the repairmen
are working on lathes that have already failed in which case the lathe must await repair.
The following assumptions are made about functioning of a lathe.
(1) The probability of getting out of order during an interval of time (/, t + At) is
indpependent of t and is equal to A(A/) = XAt + o(At), independently of the “history” of
its operation. Similarly, if a lathe is being repaired, the probability of its being putback
into operation during an interval of time (t, t + At) is equal to p(A/) = pAt + o(A/) and
is independent of the nature of its work and its length of service upto the instant /. Let Ek
denote the event that at a given instant the number of lathes being repaired or awaiting
repair is equal to k subsequent removal of a single lathe from service denotes transition
to the state e m , and completion of the repair of one of the lathes indicates transition into
the state Ek_x. Show that this is a birth and death process with finitely many states E0,
. . . , Em. Find also its stationary distribution.
Exercise 6.13 Let M be a discrete time a periodic Markov chain with state space
S = {0, 1, 2,...}, transition matrix (/?,-,) and initial distribution {a„ i e 5}. Let Abe a fixed
positive real number and s any real number of [0, «>). Define for all i, j e S
~pii { s ) = l Qe - ^ a s ) " l n \ p \
1. Show that (pn (5)) is a transition matrix of a continuous time Markov chain
with state space S, and initial distribution [ah i s 5}. [Note : Only check the conditions
for transition functions of a continuous parameter Markov process. Do not prove actual
existence of M.]
2. Evaluate the infinitesimal generator A of M in terms of the original matrix P of M.
Is M conservative?
3. Prove the following statements:
(a) M is irreducible iff M is irreducible.
(b) State j e S is positive recurrent iff j s S.
(c) State j e S is transient iff j s 5 is transient.
Exercise 6.14 Continuation of Exercise 6.13.4. Assume that M is irreducible and positive
recurrent with {Kpj s 5} as stationary distribution and there exists a constant p, 0 < p < 1
such that for all n > 0
Show that a Markov chain with Sup | | < K <°° is positive iff there exists a
i
solution {X.,i
i * e 5} tfor /=0
2 jt.-a.-.-
' ’J = 0 for which 0 < E/=oIx.1 'I1< «>.
♦♦ Exercise 6.15 Let P(t) be the transition matrix of a continuous time Marikov chain
with infinitesimal generator A given by
-a p Y
Y -a P , where p + y - a > 0.
P Y -a)
p= — and l/2(0-y)V 3.
2. We put for all t > 0
P(t) = R + S exp(-(p + ia)t) + T exp (-(p - io)t)
where R, S and T are 3 x 3 matrices with (possibly) complex entries.
Use the following facts to compute /?, S and T.
(a) A is symmetric; (b) P{t) is real for all t > 0;
(c) P(0) = /: (d) P'(0) = A\ (e) Refer to Exercise 6.14. Show that all entries of the
matrix R are 1/3.
3. Using the above results, indicate how it is possible for an infinite number of
different continuous time Markov Chains, to have a common skeleton chain.
4. Use the above results to prove: If the one-step transition matrix of a descrete time
markov chain is of the form
Q
P where p + q + r = 1, p * 1, then there exist infinitely many
V? r P)
solutions to Elfving’s problem.
♦♦Exercise 6.16 Consider a generalised birth and death process with k n(t) = wA/, A> 0
and p n(t) = njxt, p> 0 for all n > 0. Also pn(0) = Sin-
(a) Solve this birth and death process by exhibiting an explicit expression for pn(t) =
P(X(t) = n | X(0) = 1).
(b) Compute the mean population size, and evaluate its limit as t —><».
(c) If T denotes the time till absorption, then show that P(T <t)= p0(t). Compute E(T)
if A = 0.
Exercise 6.17 Show that Poisson processes are the only renewal processes which are
Markov Chains.
7
Poisson Process
= 0 *,2 ( « ) - 0 x,1(iO
Let / ( t ) = 0 ( m). Then/(r) satisfies the following functional equation (known
as Cauchy equation)
#a i = i
n=0
[--W 0 + 4 v i(0 ]z "
=>4JZ = A (Z - l ) d t ^ G ( t , z ) = c.eXiz-'}l.
Cr
The boundary condition 1 = G(0, z) gives c = 0. Hence
G(t,z) = e - Xt 2 n\
e~h (Ar)"
Therefore P[Nt = n] = pn (t) =
n\
Axiomatic Definition 7.3 It is a continuous parameter stochastic process
satisfying the conditions:
(1) P[N0 = 0] = 1
(2) {Nt, f eT] has independent increments
(3) {Nt, t e f ) has stationary increments
(4) P[Nt > 0] > 0 for all / > 0
P[Nh > 2] _
(5) lim
hi o 1]
or
(4') P[Nh =1] = Ah + o(h), h1 0, A > 0
(50 P[Nh 2>] = o(A) ash 0i
Conditions (4) and (5) or (4') and (5') are known as regularity or oderly properties
of Poisson process.
Definition 7.4 Poisson process is a Renewal process with interarrival times as
exponential distributions (see Renewal theory).
Let Nt denote the number of occurrences upto time t.
Poisson Process 135
Theorem 7.1 Under the axioms (postulates) (1) to (5) Nt has Poisson distribution
with mean function Af, i.e. pn(t) is given by the Poisson law
e x\A t)n/n\ n= 0, 1, 2,. . .
Proof Let pn(t) = P(Nt = rt|Af0 = 0) = P(Nt = n) (by axiom 1). Actually
P(Nt+h - Nt = y\Nt = jc) is independent of x. Axioms (4') and (5') => p0(t) =
1 - Xt + o(t). Consider pn(t + h) for n > 0.
The n events by epoch t + h can happen in the following mutually exclusive
ways A x, 4 2, 4 3, . . . , An.
For n > 1
Ai = {n events occur by epoch t and no event occurs between t and t + h)
we have P{AX) = P[Nt = n\P[Nt+h - N t = 0|A, = n]
= Pn(t)PoW = PnW 1 o(
A2 = {(w - 1) events occur by epoch t and 1 event occurs between t and t + h]
P(A2) = P[N,= n - 1]P[AU - 1|AT, = 1]
= Pn-\U)P\(h) = pn-\(t){kh) +
4 3 = {(n - 2 ) events occur by time t and 2 events occur between t and t + h]
P(Al) = Pn~2 (t){o(h)} = o(h)
and so on for A4, A5, . . . , A n i.e. P(Ak) = o(h) for 4 < k < n.
Therefore pn(t + h) = pn(t)( 1 - A/i) + pn_\(t)A,h + o(h) for n > 1
nr Pn(t + h) - Pn(t) o(/l)
or ^---------- - - Ap n + Ap„-\ (t) + — —
Taking
lim h-> Oyieldsp^(t) = -Ap+ 1 ... (7.1)
as the Kolmogorov’s differential equation for Poisson process.
For n= 0, p0(t + h) = p0(t)p0(h) = p0(t)( 1 - Ah) + o{h).
Po(t + h) o(h)
Therefore -Ap{t) +
h
Talcing limit h —> 0. we get
p'0{t) = - A p 0{t) (7.2)
(c) H(t)^ = Z F (n) ( o j , the renewal function in this case is given by At.
Proof
(a) See Theorem 7.5. (b) This is trivial.
(c) Note that P(Nt < n - 1) = P(Wn > 0 = 1 - P(Wn < t)
n -\
= X e * (At)k/ k \(integrating by parts)
k=0
Therefore, P(N, = n) = e \ f o )n/n\
= FU + x ) ~ [1 - F(t + x - u)]dH(u) if 0
Jo
Poisson Process 137
1
F(t) - f
Jo
[1 - F{t - u)]dH(u) if x <t
1 if x > 1
Proof
Let Si < S2 < . • . < Sn be positive renewal points with Sn be the largest such
points. Then
[T(/) < x] = [At least one renewal point fall in the interval (f, t + x]].
P{t < Sn < t + x < Sn+1) = J P[Sn+i ~Sn > t + x - u]dF(n) (m),
Therefore
(2) Y{t) and Z(t) are independent iff {N(t), t eT] is a Poisson process
(see Exercise 7.9).
“ (x/2)2r+n
where ln{x) = 2
r=o r !|n + + 1
is the modified Bessel function of order n(n > - 1).
The Poisson type of occurrences are called purely random events and the
poisson process is called a purely (or completely) random process the occurrence
is equally likely to happen anywhere in [0, T] given that only one occurrence has
taken place in that interval. We state this by the following theorem.
Theorem 7.4 Given that only one occurrence of a Poisson process Nt has
occurred by epoch T, the distribution of the time interval T, in [0, T) in which it
occurred is uniform in [0, T], i.e.
(Ae-A>dr)(g- ^ r -'> _ dt
( e~XT))T
{X T
Note It may be said that a Poisson process distributes points at random over
the infinite interval [0, <*>] in the same way as the uniform distribution distributes
points at random over a finite interval [a, b].
Interarrival time
With a Poisson process {Nt), the number of occurrences of an event E by epoch
t, there is associated a random variable—the interarrival time X between two
successive occurrences of E.
Theorem 7.5 The interarrival time between two successive occurrences of a
Poisson process {N,} having parameter A has a negative exponential distribution
with mean 1/A.
Proof Let X be the r.v. representing the interval between two successive occurrences
of {jV,} with P[X < jc] = F(x) be its distribution function. Let E{ and Ei+X be the
successive events and Et occurred at time tr Then
P\X > x] = P[Ei+1 did not occur in (th t{ = jc) given that E, occurred at instant fj
= P[Ei+{ did not occur in (rt tt + jc)| Nti = i]
= P [no occurrence takes place in an interval (th tt + jc) of length jc | N tj = /].
= P [ N t i +x - N ti = 0 \ N t . = i]
n- 1
+ 2 P N\ | - N(0)
j =o
= 0,N\ — fi) - 4 — I ^ 2
n- 1 ^ M ^2 f
Now the second sum < Z e~Xjtl,n 1 - e n - e n A —
7=0 n
V
. 0 („-2 ) =0ri
i - e~**1n \n
n-I / 2, \
First sum < S e(-*jt\in-h\in) \ —1_ (i (by L’Hospital’s rule and
j= o V. nJ
n- 1
= s *(-- fy titn ) 9- * * \ I n A t2
-i)
>=o 1- e
P( lim sup ( r n+i - Tn) > a) > lim sup P ( r n+1- r n > a) > exp (-Aa).
n —>oo n —>oo
This implies 7 ( lim = <») > exp (-Aa) for every a > 0.
In particular it follows that 7(lim Tn = °o) = 1.
Now we give an alternative proof of Theorem 7.6(a) based on renewal
argumant.
Proof of (a) Let 0 < Sx < S2 < S3 . . . are the renewal points (arrival points) of
the Poisson process. We prove the theorem by induction. For 71? the theorem has
already been proved in Theorem 7.5. Now calculate the joint distribution of Tj
and 7Y After conditioning on Tx = x and taking expectation, we get
P(TX< u, T2 > v) = f
Jo
P{S2 > v + ;c|7i = x)he~*Jcdx
*+i
Hence P(S*+1 > 2 xj | 7} = x t , 1 < i < k)
fk+1 >
=Pu Z Xi 1 ~ N = P{W(**+i) = 0}
i \ 1 )1 J J
(By stationarity and independence of increment of Poisson process N(x)) = e **k+l.
Evaluating the multiple integral, we get
P(Tj < uj, 1 <j < k, Tk+i > xk+l)= - .. (1 - e ~ ^ )
Therefore Tx, T2, . . . , Tk+l are i.i.d. exponential r.v.s.
Therefore
Under initial condition y/^z, 0) = 1 and therefore yf(z, 0 = exp [m(0 [z - 1]],
= y/(z,t) Y m j {-
+ -£
n n =1
Then we can show (as in the case of ordinary Poisson process) that there
exists a constant u > 0 such that
Therefore V x ( t ) W = evt{*r(u)~l]
where (pY(u) is the common characteristic function of {Yn}.
Thus we have shown that {X(0, t> 0} has independent stationary increments
with the above characteristic function for any t > 0.
Note If Y is integer-valued then the compound Poisson process is a generalised
Poisson process.
Compound Poisson process arises in bulk queues where customers arrive or
are served in groups. An important application arises in “collective risk theory”.
Suppose that claims against a company occur in accordance with a Poisson
process with mean Xt and the individual claims X-s are i.i.d. with distribution
{P*}. Then X(t) represents the total claims at time t. If A represents initial reserve
and c the rate of increase of the reserves in the absence of claims, then the total
reserve at time t is A + ct - X{t) and negative reserve implies “ruin”. The
moments of X(t) are given by Wald’s identity as
E(X(t)) = E m ) ) E ( X x) = Xt{EXx\
Var (X(t)) = E(N(t) ) Var(X,) + E 2(X,) Var = At EX,2 (EX, = 0)
e~MU
a linear function of independent exponential variates and hence Z(l), Z(2), . . . , Z(n)
is a Markov chain (cf. Renyi, 1953).
As a general case, let X(1) < X(2) < . . . < be the order statistics from a
strictly increasing distribution function F(t).
The transformation Uk = F(Xk) converts into £/(r), r = 1, 2, . . . n, the order
statistics from uniform /?[0, 1] distribution. Let Z* = -log £/*.
F[Z* < x] = P[-log £/*<*] = P[t/* > *-*]
= P[F(Xk) > = P[Xk > F~\e~x)]
= 1 - F(F~\e-x)) 1 - e~x
i.e. Zk s are exponential random variables.
Hence, Z(r) = log f/(n_r+d are order statistics from exponential distribution.
Yi , >2 + Yr
= F - 1 [exp - . . . + (7.11)
n n- 1 n - r+r
Since T/s are independent, hence by (7.11) X(„_r+1) and Tr+1 are independent.
Therefore, by (7.12), X(n), . . . , X(1) form a Markov chain and hence
W o . X(2), . . . , X(n)} is also a Markov chain.
From this the following theorem follows.
Theorem 7.9 For random sample of n from a continuous distribution function
the conditional distribution of X(i) given X(r) = xr(s . r) is just the distribution of
(s - r)th order statistics in a sample of (n - r) drawn from the parent distribution
truncated on the left at x = xr
Proof
nl
( n - r ) l ( r - 1)!
Proof The density of fcth order statistic X(k) from uniform [0, t] distribution is
given by
n\
( n - k ) l ( k - 1)!
Now from Theorem 7.8 given Nt = n arrival times are distributed as order
statistics from uniform [0, t] distribution.
Therefore, Wr is distributed as r th order statistics from uniform [0, t\ distribution.
Theorem 7.10 Let {Nt, t e T} be a Poisson process with intensity A. Let s eT,
s < t E T, k< n.
n-k
Then P[NS k\N t = n] =
P[NS = k,N, = n]
Proof P[NS = k\N, = n] =
PIN,=n].
P[ Nt - N s = n - k , N s = k]
PIN, = n]
P [ N , - N s = n - k ] P [ N s =k] . . . .
-----------------------^ -------------- (due to independent increment)
(t - s) n-k ok
yh tn
that each arrival is “registered” with probability p, independent of other arrivals. Let
{Y(t), / > 0} be the process of registered arrivals. Prove that Y(t) is a Poisson process
with parameter Xp.
Exercise 7.3 Suppose g(t) is the conditional failure rate of an article at time /, given that
it has not failed upto time r, i.e. P (failure in time (t, t + h)/no failure upto time r} = g(t)h +
o(h) as h I 0. Assume that g(f) is positive and continuous on (0, °o). Find an expression
for F{t) = P (failure at some time r, r< t) in terms of g(t).
Exercise 7.4 Let X(t) and Y(t) be two independent Poisson processes with intensity
parameters Aand p respectively. Let W and W* be the times of two successive occurrences
of Poisson events of the processes X{t), i.e. W < W*, X(t) = X(W) for W < t < W* and
X(W*) = X(W) + 1. Define N = Y(W*) - Y(W). Find the probability distribution of N.
XU )
Exercise 7.5 Let Y(t) = Z %n be a compound Poisson process. Then show that
n= 1
(a) the process Y(t) has independent increments, (b) The characteristic functional 0y(/) of
Yi() is given by 0y(o(u) = exp [Af(0|(w) - 1)] where 0|(w) is the common characteristic
function of {£„} and Ais the parameter of the poisson events, (c) If E(%2) < °° then show
that E(Y(t)) = XtE(£) and Var (Y(t)) = XtE£2.
Exercise 7.6 Consider a Poisson process with rate A(> 0). Let Txbe the time of occurrance
of the first event and let N(TX) denote the number of events in the next Tx units of time.
Show that EiN(T{)Ti) = 2IX and find the variance of N(Ti)Tx.
Exercise 7.7 Let X(t), - oo </<«>, be a Poisson process with rate A. Let Y(t) = X(t + 1)
- X(t), - oo < t < oo. Find the mean and covariance function of Y(t) process and show that
it is a second order stationary process.
Exercise 7.8 Let X(t) be a Poisson process with parameter Aand Tabe the first time X(t)
reaches a > 0. Calculate E(Ta), Var (Ta) and also m.g.f. of Ta. Can you guess what type of
random variable will be Tal
Exercise 7.9 Prove that if the sum of two independent renewal processes is a Poisson
process, then both renewal processes must be Poisson processes.
Exercise 7.10 A telephone exchange has m channels. Calls arrive in the pattern of a
Poisson process with parameter A; they are accepted if there is an empty channel, otherwise
they are lost. The duration of each call is a r.v. where distribution function is exponential
with parameter p. The life times of separate calls are independent r.v.s. Find the stationary
probabilities of the number of busy channels.
Exercise 7.11 Show that if the interarrival distribution function F(x) of a renewal
process Y(t) is continuous and if the expected residual life time E(Y(t)) is a finite constant
for t > 0, then show that the renewal process is a Poisson process.
Exercise 7.12 In a factory, accidents happen according to a Poisson process with intensity
A. The times that the victims stay in the hospital are independent random variables with
common distribution function F(t). Let X(t) be the number of persons in the hospital at
time t € (0, oo).
(a) Show that P(X(t) = k) = e-XaU){Xa(t)]klk\, = 0, 1, 2___
where a(t) = f
Jo
[\-F(x)]dx
(b) Derive the distribution of tlim X(t) if F(t) has a finite first moment.
—>°o
(c) Compute P(X(t) = k) if all the victims stay in the hospital for a constant length of
time L.
8
Continuous Time and Continuous
State Markov Process
(iv) P(0) = /, i.e. Pij(0) = Sjj, the Kronecker’s delta, and I is the Identity Matrix.
We shall develop an analogous theory in the continuous state case. To avoid
many measure theoretic difficulties let us define transition distribution function
as
F (y, s\ x, t) - P[X( t ) < x | X (s) = y], s < t, s, t E T.
To avoid many pathological difficulties we shall assume that the transition
distribution function of a continuous time Markov process satisfies the following
axioms:
(i) F ( y , s \ x , t ) > 0
(ii) F(y, s ’, - 0 0 , t) = 0, F{y, s\ + 00, t) = 1.
(iii) Chapman-Kolmogorov equations: For every t, s and t satisfying s < r < t ,
we have
Definition 8.1 A Markov process that satisfies axioms (i) to (v) is called a
Diffusion process.
Special Case (When density function exists)
(Chapman-Kolmogorov equation)
(iv) /(y, t\ x, t) = <5(x - y), Dirac delta function,
Informal Proof
Consider the n-step transition probabilities in a simple random walk P(Sm+n = j
| Sm= i) = P(X{ + ... + Xn- j - i \S 0 = 0) = P(Sn= j - i), whereXf-’s arei.i.d. Bernoulli
random variables taking values 1 and -1 with P(X{ = 1) = p and q = 1 - p.
0 if n + j - i is odd n-j+i
Then p( n + j- i) / 2 q 2
Pr = if n + j - i is even
(n + j -
C (Ac)2 a
E [Total displacement] ~ t
D D
= 2 Ct ~ a(x, 0 = m(0, the mean function
Var [displacement] ~ 4t ^ - C 2 At j — ■
± Ax, t) = v(x, t) ± A3
^ + (Ax)2 ■+ ( 8. 6)
dx 2 dx 2
By (8.3), (8.4) and (8.5) and letting 0, we get
^ i £ )= d |! £_2C ^,
dt ddx
154 Introduction to Stochastic Process
Since the random walk and the Brownian motion (W(r)} have independent
increments, the joint ch. f.s of Xn(tx, . . . , Xn(tk) is
exp \i S u jXn (t A
7= 1
k
= E exp z Uj(Xn(t 2 ) ~ X n( ti ))
7=2
J —As;
/
-JJ \z- y \> S
[F(z,s\x,t)~ F(y,s;x,t)]dFz(y ,s-As;z,s) + I
J|.iz-si<5
Kz-
°y
+ 2 (z - y)2| ^ £ F
(y,s-,X
,t) o((z ~ yf)]dFz(y, - As; z, s)
~ /i + h-
156 Introduction to Stochastic Process
as a ( y , s ) ^ F ( y , s \ x , t ) + - b ( y Js ) —- £ F ( y , s \ x , t )
(x -y)/D ( t- s )
= —i — f exp(-w2 ! 2 )du (8.7)
■J2n J_,
Obviously (i) F(y, s; x, 0 > 0; (ii) F(y, x; = 0 and F(y, s\ 1.
f0 if x < y
(iii) F(y,s;x=
[0 if* > y
From condition (iv) of Wiener process, the transition density exists and equals
1 -(x-y)1!\2D(t-s)\
f(y,s\x,t) =
(2 J tD { t- s )) m
and satisfies Chapman-Kolmogorov equation
= lim -j-f L—
t|Jy-x|>
A SJlFDiAT)
-M2/2
= lim I du y
AI-+0 At V2 nJ\ujDAi\>8 V d ZF
= nm ± J i r e 14 12 du (by symmetry)
At->0 At v 71 JSi
S/fDAt
(vb) lim
fcl
. (y-*)
^jlnDAt
1 ~ - ( y - x ) 2 l(2 D A t)
dy
= 0 (by symmetry)
= a (x , 0-
158 Introduction to Stochastic Process
lim 1___ 1 _ f u2
£ "2/2 du(puty - x = u-yfDAt)
At-^o At
^ 4 l n J|„|<s| / d I7
= D = b(xt)
Note (a) If the Diffusion process is homogeneous, then t) and t) are
independent of t.
(b) If the Diffusion process is additive (i.e. given X(r0) = x() the increment
X(t) - X(t0) depends on t and t0 and not on a0) then a{x, t) and t) are
independent of x. For example Wiener process is additive.
=P w i l l < ill
aJUt oJUt
= & ( y ! o 4 t ),where <t> is the c.d.f. of 1).
Hence, E(Y(t)) = 0 and Var(K(/)) = t2a 2lt = a 2t. In fact X(1 and X(Uu) - X(1
are independent for 1 It< 1 lu.The independe
by showing that they are uncorrelated.
E(uX(l/u)[tX(l/t) - uX(Uu)}
= l/)EX
(utX (1It)- u2X2 (Mu)) = - u\Uu) = 0
(since E(X(s)X(t)) = min U. /)). Hence for 0 <
E(exp [id{tX(l/t) - mX(1/m)}])
= E exp {i0((r - u) X(1 It)- u[X(l/u) - X(l/r)])}
Continuous Time and Continuous State Markov Process 159
= exp { - 6 ll 2 (t - u)}.
Hence Y(t) - Y(u) has Gaussian distribution.
(4) (a) X(0, t < 0 is a Wiener Process (follows by symmetry).
(b) (Origin change and strong Markov property). {X(t + r) - X(t), t > 0, T> 0
may be a stopping time} is a Wiener process
Let Ta = inf {r: X(0 > a) be a random variable and in fact a stoping time
(continuous valued). Then [Ta, a e R+] is a stochastic process.
Lemma 8.1 Let (X(r), 0 < t < T) be a Wiener Process with X(0) = 0 a.s. and
P = 0. Then for any a > 0,
160 Introduction to Stochastic Process
\X(t) \ i t < T a
Xa(t) =
[a - [X(t) - a ] = 2 a - X(t) if t > Ta
i.e. the new Stochastic process Xa{t) gives for t > Ta the reflection of X(t) about
the line x - a.
By symmetry, X(t) and X0(t) have same probability distribution.
[M(T)> a] = [M(T) > a, X(T) >a]U[M (T ) > a, X(T) <a]U[M(T)> a, X(T) = a]
Note that Xa(T) < a, since X(T) > a. Because the probability law of the path
for t > Ta given X(Ta) = a, is symmetrical with respect to the values x > a and
x < a and is independent of the history prior to time Ta. From reflection principle
to every sample path X(t) for which X(T) > a, there exists two sample points such
that M (T ) > a and vice versa i.e. to every sample path X(t) for which M{T) > a
there corresponds two sample paths X(T) > a and X(T) < a with equal probability.
P[M(T) >a] = P[M(T) > a, X(T) > a] + P[M(T) > a, X(T) < a]
(Since P[M(T ) > a, X(T) = a] < P[X(T) = a] = 0, the other two are mapped onto
one another by reflection about the line x = a after the time Ta)
= 2P[M(T) > a, X(T) > a] = 2P[X(T) > a].
Note Tm a> 0 is a.s. right continuous but may not be continuous. P[M{t) > a] =
2P[X(t) > a] with ju = 0, X(0) = 0 and a = 1.
Continuous Time and Continuous State Markov Process 161
f
V27Z? "J„a
e x p (-* 2 1 2 t)dx
=~i= f exp
x/2 n J ai41 p u tV7
= 2 1 - 4> a
vV7y
M OO MOO
= {1 - (2/7T),/2 I y-i\!2)Z2 fa
JO Ja/Jx
(At ) 2 . V ar(4 F (0 ) „ „
= lim —r— + lim ------- = 2D.
At-^O At A t-* 0 At
Since these limits exist, the stochastic process V(t) is a diffusion process and
its transition density function g(v0, v, t) satisfies the forward Kolmogorov equation.
More specifically let g(v, t)dv = P[v < V{t) < v + dv | V(0) = v0].
Hence, the Kolmogorov’s forward equation is
dh
(vg) + D ( 8 . 8)
dv dv 2
with initial conditions:
g(v, 0) = d(v - u0), Dirac delta function.
ei6u j ^ ( v g ) d v J ei6v
e‘ev°
Therefore 0 (0 ,0 ) = ei9v° = exp(- ( 8 . 12)
1 - U v / 0 0)2
= - ---- e
7— 1 dv, i.e. the distribution of velocity
V27T (To
is normal with mean zero and variance Dip. In Maxwell Boltzman distribution
(kinetic theory of gases), <Tq = where ATis the Boltzman constant, T the
temperature and m the mass of the particle.
Continuous Time and Continuous State Markov Process 165
P[v < V(t) <v + dv | V(0) = v0] P[vo < V(t) < v 0 + dv 0 ]dv0
- [
.If I l f
o A <nn J -I ffo I dv o
^/(2?rcr2 (f)) J(27ral)
2 rr^ i| &
<y{t)
f (2^<r2( f) )f (2^ cTo) r dv o
1 -~(l;/CT0)2 I , V,
e 2 dv0 =go(v)dv0
*j2 n o 0
which is known as the Maxwell distribution of velocity of gases (<7q = £>//?).
Hence, if v 0 is Maxwell Boltzman then E(V(t)) = 0, Var(V(r)) = cr^ and
Cov (V(f), V(j)) = a 0V ^ ' - vl.
Thus, V(r) is covariance stationary.
But K(r) = —dx where X is the displacement. Hence
= lim
At->0 "a f Q(y) f ( x 0 ,T;y,t+
At
Aty t)
dy
= HZ-T t {j d y Q ( y ) ^ f(x
-r"Cl
Q ( x ) f ( x 0 , r ; y , 0 dy j* (by Chapman-Kolmogorov)
166 Introduction to Stochastic Process
; GOO if
•J —o
/(x,
is uniformly continuous]
<2(x) being continuous on [a, 6], it is bounded by diffusion condition v(a) and
hence we get
Thus,
I
l.y-*l><5
/( x , t \ y , t + At)dy = o(At), as At —> 0
/ = lim
At-*°>
d x f ( x 0, r ; x , o j j
Expanding Q(y) in the neighbourhood of x, we get
Using (8.14) and diffusion conditions (a), (b) and (c), we get
1 |y-*|£<s
f ( x , t ; y ,t +At)Qiy)dy - Qix)
1|y - A |< 5
f i x , r,y, t + At)[Q(x) + (y - x)Q'(x) + ^ i y - x ) 2 Q"(x)
+ jQ "(x) f
*\y-x\<d
[(y - x ) 2 + o(y - x )2]/(x , t\y, t + At)dy
I
J|;y-Jt|< 8 |y-x|<3
J |v -.
f
f i x , t \ y , t +At)dy/ (=x ,1V-, f, + At)dy = 1 -
[
( a f ) Q \ x ) d x = [ a f Q ) ba
•b
r
"a
-(af)dx
Q(x)-^(af)dx
(8.16)
-]> *
(since Q(a) = Q(b)= 0)
= - j b Q(x) (bf)dx
= - -H-(bf)Q]ba + f Q(x) ^ - ( b t f d x
(bf)dx (8.17)
_ r w
Since / Q(x)
.L dt
r
* -o o
^ + ± { a f ) - l / 2 £ - (bf ^ Q i x ) d X = 0.
Since the last expression in {•} is a continuous function and Q(x) is arbitrary,
fv . ±
l r + i < « / ) - i / 2 - W ) = o.
dt'dx J '" ” dx 2
n n n-1 n - l n -l
Now 2 2 OijXiXj = 2 Z <7y.x,.r, + xj?<7„„ + 2 x n Z cr,,,*, (8.5.3)
/=1 ; =1 <=1 7=1 ;=1
Then
n -1
exp(-l/2cr„„.^ - Z a inXi)
1= 1
r
*n-i) = n -l
exp(-l/2crnAI^„ —x n ^ OinXi)dxn
V —o i- 1
n-l
exp <- 1 / 2 <7*, ■^n ^ (Gin ^Gnn)Xi
i=1
l
n-l 2 '
f " e x p ■ - 1 / 2 ( 7 „„ ■^n "A" ^ (^Ln ! Gnn )■*■*
J -o o i=l
I 1
n-l
x C exp j -l/2cr„ + Z ( CTin (8.5.4)
/=!
endent of x u . .., jc„_i. Equation (8.5.4) is clearly a
/?—
1
normal p.d.f. with mean - 2 (o in! o nn)Xi.
1=1
Continuous Time and Continuous State Markov Process 169
Now if X(t) is a Markov process, it follows from (8.5.2) that for any s<t from 7,
E(X(t)\X(s) = x) = - o X2/o22x (with n = 2).
-1
r(.s, 5) - r ( j , t)
Now a, are elements of Z 1=
-r(/,j) r(r,r)
1 T( / , / ) - r ( j , o '
T ( ^ , 5 ) r ( / , r ) - r 2( 5, 0 - r ( j , 0 r ( i , i )
-|2
Obviously /( x „ | = C. exp I - - - <7„ An ~ _ An-1
r ( ^ _ i »tn) ^ N
X{tn) - *('*) =0
s jx
[ uj - U
rr//"~1
'/i-h ,/
V "lJ\ x(/"-i)lX(f1) = ■*■•• • •’*('«->) =
Ij(z)=S ( z/ 2)2k+Jk
[/ ],1
J k=0
We shall prove the case k = 2.
Use the polar coordinates by defining
R(t) = JX
)2 ( 0 + * ! ( / ) , 0(f) = tan"1
Since Brownian motion is Markov,
= + t) <y\X{tn)P[R(t) < y
where 0 < t0 < . .. < t„ , t> 0, and x , . . . , are arbitrary points in the plane,
~0 ~ n- 1
x = (jcj, x2). Then
P[R(t) < y | X(0) = x]
(r sin - x \ ) 2 + (r cos - x 2) 2
■ffJTi
0 0
exp d 6 rdr
It
where transformations U\ - r cos 0 and u2 = r sin 0 gives dux du2 = rdrdO. Since
(r sin 0 - x x) 2 + (r cos 6 - jc2)2 = r2 + 2r(jcj sin 0 + jc2 cos 0) + || x ||2
where ||x\\ = x 2 + x \ ,
exp n m i cos 0 ^ 0 ,
■I 2/
Since cos (y/+ 0) is periodic over the interval [0, 2 n],
172 Introduction to Stochastic Process
Hence P ( R ( t ) < y \ r 2=
X(0) = x) +J*
l MI^2 exp
2 nh dr
2t
= f / , M r)dr (when 2)
Jo
Therefore
=Jof , r)dr
This verifies Markov property of R(t) and its transition density when k = 2.
Similarly, the general case can be proved by induction.
(c) Show that {X(0, -°° < t <°°) is a Gaussian process iff A has a Rayleigh distribution,
that is, A has a density/* given by
f a( a)= S - exp(-l/(2cr2a 2)). 0.
(J
Is {X(t), ~oo < / < 00} a Markov process?
Exercise 8.7 Determine the Covariance function of U(t) = e~* W(e2t), t > 0 where W(r)
is the standard Brownian motion.
(1) Trend
A smooth long-term movement covering a number of years reflecting the general
tendency of the series. Growth curves are good examples of time series exhibiting
trend. Naturally, characteristics of the process depend very much on the time of
observation. One method of eliminating the trend factor from a time series is the
method of moving averages which is well known and we shall not discuss it in
some detail in this book.
We would, however, consider only those oscillatory series which have a constant
amplitude. When the trend and cyclical component are eliminated from a time
series we are left with a series which apart from the disturbance term fluctuates
round some constant value.
t —>°o
(ii) From (i) it is clear that the joint distribution of a finite number of points
in the index set do not involve the points themselves, instead it is a function of
the distance between the points,
(iii) t \ 9 . . ., tn are not necessarily consecutive.
(iv) If the second order moment E(X,2) < <», then
<E(X?)E(Xt+h+s- X l+h) 2
= E ( X 2+h+s)+ -
o E2
< e 2 by taking r? < ^ q)-
= £ ( ( ( A - A ) + ( f t - f t ) 0 ( ( A - A ) + ( f t - f t ) ( * + * )))
= cru + cr12 r + cr120 + h) + <r22 + h)
is not a constant for all t.
Hence it is not covariance stationary.
Result (3) Autocovariance function is an even function of h, i.e.
y(h) = y (- h).
Proof y(h) - Cov (X„ X/+/f) = Cov(X ^, Xt>) = y (- h) where t - t' - h.
0 if k > q
1 if* = 0
q-k
Also P(k) = £0 PiPi+k
if k= 1,2,
I Pf
1=0 1
or
E(Xt ) = n i P i = 0 i f ,?o^'2 <
and j=0 ^
(these conditions are
Var(X,) = <72 i p ? ,?0 1/?' 1<
satisfied if - 1 < a < 1).
Time Series Analysis 181
E(£t ) = /a,
K(£,) = cr2 and
Cov (£t , £s ) = 0 if t * s
Take X0 = 0. Now
Xi = ex
x ‘ - Si
Then E(Xt) = t/a and V(Xt) = t o 2.
Hence the process is not second order stationary.
However, the first order difference of {X,} is a stationary process, since £t =
Xi - XM, with expectation jll and variance = a 2 (independent of t). For example,
share price on tih day = Share price on (t - l)th day + error.
182 Introduction to Stochastic Process
1 m ( ni ^ i m
E - E — Z X, =E E - AC the ensemble average and
"I /=! [ ,1j '=1 j m j=\ M7)
Var — 2 X,(./) £1
tn j =i m
1 —
If the number of realizations is large enough then — Z X( j\-^> JUas m —> °q,
m J=i
Ergodicity of p
Consider a discrete stationary time series {X,} with E(Xt) = p for all t. Then the
_ 1 n
series is mean ergodic if the sequence of sample means Xn = — E X„ n = 1,
2, . . . converges in q.m. to p.
Theorem 9.1 A stationary time series {X,} whose covariance function as
~ = -1 E
y(h) —> 0 as h —> °° is ergodic for p. Hence we can suggest that X„ n X,
n t=\
is an optimal estimate of p.
Proof To show that Xn ^ we are to show that lim E( Xn - p ) 2 = 0-
n —>oo
- n n
Now Var ( Xfl) = Var E x, = — Z I Cov
n /=i
X,-)
n~ i=i i'=\
I n
Z Cov , ,X,-)- V Z Var (A",).
(X
t'=l n ~ i=l
Time Series Analysis 183
Since Var (X,) = cr2, second term of right hand side —> 0 as n —» ©o.
VI
Now 4 -i 2 Co - £
n “ r=l /'=! AZ“ /=! t t'=\
1 '- 1
Z 7 Z Y(h) since y(/i) = y (- h) and — < 1
n /=i t h=0
i n 1 r’ 1 1 '- 1
Z - 2 y(h)
n ,=i t h=0
+^n /=yv+1
z 7 Z y(A) ,(n > AO.
t h=o
1 /_1
< ■£ !- for r > jV since 7 2 y(A)
2 n * h—Q
Now take n> N, so that the second term < -■. Also for n > N, first term < .
Hence Var(X„) = E(Xn - ji)2 -> 0 =» Xn ju.
Corollary 9.1 A stationary time series with absolutely summable covariance
function is ergodic for mean. Also
Proof Z | y(/z) | < oo y(A) —> 0 as h —> <*>. Ergodicity follows from the
h
theorem.
Also we have
n V a r ( X „ ) = in j=\
i /=i
i y ( r - y ) = ni- h=-(n-1
"l ) |)
= I (1 - \h\ln)S y(A)
l/J
(by Kronecker’s lemma lim Z — | y(/z)| = 0.)
«—>°° /i w
An alternative expression of lim zzVar(X„) given by the following corollary
n— >°°
should be studied after section 9.2.4.
**Corollary 9.2 If the spectral density of a stationary time series X, is continuous,
then lim zzVar(X„) = 27r/(0), where /(A) is the spectral density of Xt (to be
/?—>00
defined in section 9.2.4).
Proof Since /(A) is a continuous periodic function on [-n, n] the Cesaro
184 Introduction to Stochastic Process
1 * *”
sum C„(A) = — 2 Z (ak cos kX + bk sin kX) + a 0l2 of the Fourier series of
H r-2 lc=1
f(X ) —>f(X) uniformly in X e [- /r, tt].
Therefore
n r- 1 1/
= lim 7(0) + - Z I 7(*) note . i r cos kX f(X ) dX = 7(/:)
A r=2 *=1 j v
= lim «Var(Xn)
n—>oo
[r,, t2]. The integral J X} d t, if it converges, is called the total energy o f the
1 CT ,
ser/es {X,, - oo < t < o°}. The limit lim I X fd t is known as the average
T—>°° " * J _ t-
power of the random process. If the stochastic process is complex valued, we
write | X, |2 in place of X}.
In many situations time series are simulated by the sum of harmonics with
given frequencies and random amplitudes and phases. In other words, one considers
processes of the form
n
Xt = Z ak cos (ukt+ Qk) (9.2.1)
k=\
where uk are given numbers, amplitudes ak s and phases 0k s are random quantities.
The probabilistic structure of this process is completely determined by the joint
distribution of the r.v.s ak and 6k for k = 1, 2,. . ., n. If a second order process X,
is the sum of n harmonic oscillations with amplitudes | ak | and frequencies
uk, then all the frequencies {uk}, for k = 1, 2,. .., n, considered as a set of points
on the real line, is called the spectrum of the process {X,}. In many cases it is
convenient to consider complex-valued processes of oscillatory nature
n
Z, = k=1
I (9.2.2)
2T | Tr I Z* I2 dt = J 7^ i z y*f
186 Introduction to Stochastic Process
sin ^ - n , )
= ^ \Y k \2 + 2 YkYr
T(uk - u r) ' K
k=l *,r=l
1 f7 "
Therefore lim — \Z t \2 dt= 2 y? a.s. (9.2.5)
T— k= 1
Thus the average power of the oscillatory stochastic process Z, is equal to the
sum of the average power of the harmonic components of the process.
From (9.2.4) we see that if 0 is a point of the spectrum, then
lim
T~>°° 1
y f
J- T
=
This shows that F(u) is equal to the average power of the harmonic components
Time Series Analysis 187
of the process Z, (whose frequencies are less than or equal to u) in a unit time.
It completely characterizes both the average power of each harmonic component
of the process Z, and the overall average power of the harmonic components of
the process, whose frequencies lie in a given interval. Thus
c j = F(uk) - F(uk - o). Z C 2k = F(u2) - F(UX).
u\<uk<u2
By means of spectal function, the covariance function of the process Z, can be
written in the form
(9.2.8)
1 for k = 0
P* = for k = 1, 2 ,.. ., q
0 k >q
k if /: = 0, 1,
Then q+ 1 q
Pk =
if k > q
=E 2 a JE,_k_j
l ' =° J { ■ J
t t+k
oT
7
M
= £ 2 a l+k (where / = t i, m = t + k - j)
[ = -oo m=-«*>
t
= cr2a* Z a 2(t l) (only those terms which satisfy / = m contribute)
/= - o o
= o 2a 2 a 2'
i=0
= - a 2).
Therefore, pk = a k.
Time Series Analysis 189
a ± *s](a2 + 4b)
Case (i) rx and r2 are real, i.e. are real, i.e. if a2 + 4b > 0.
A j and A2 are found as follows :
Since p0 = 1 = A x + A2. From first Yule-Walker equation, we have
Pi = aPo + bp-1 = a + bp_x = a + bpx
Therefore p x =
1 - b’
Hence Ay =
1 - b -~ 'r22 i /VM
K ri ~ r2) andA2 = l-A ,
------
For real equal roots of the characteristic equation rx = r2 = r and
,2 n
0 as &—><*>.
« =^ * TTT^ ^
The correlogram is oscillatory and decaying in character.
Case (ii) rx, r2 are complex conjugates if a2 + 4b < 0.
Write r} = p (cos 0 + i sin 6) and r2 = p (cos 6 - i sin 6).
190 Introduction to Stochastic Process
*
If II
II
II
A = 1, Pi = p [cos 0 + 0 * sin 0]
* = -1, p_! = [cos 0 - 0 * sin 0]
1 - P~2 cos 6 1 —p 2
R* = ------ —cot 0 = cot y/ (say)
1 + p 2 sin 0 1+p
Now Pk = Pk [cos + cot y/ sin 0/:]
/ sin (0& + yO
k s‘n (Ok + iff) ^Q
since <c
p sin y/ v
sin y/
rk
0
lag k —►
( f) ARM A
The correlogram of a MA(q) process is easy to recognise as it “cuts o ff’ at large
q whereas AR(p) process is a mixture of damped exponential and sinusoids and
dies out slowly (or attenuates). The correlogram of a mixed ARMA model also
generally attenuates rather than cuts off.
Example Show that the infinite MA (oo) process {X,} defined by
Xt = £t + C(£t_i + £,_2 + . . .)
where £,’s are white noise with E£t = 0, Var £t = a 2 and c is a constant, is a non-
stationary process. Also show that the series of 1st differences {Yt} defined by
Yt = X, - XM is a first order MA(1) process and also 2nd order stationary. Find
the autocovariance function of { } .
Solution Var (Xt) = cr2[l + c2X 1] is infinite, obviously Xt is not stationary.
Yt =£t + c{£t. x + . . . ) - £ t_ x -c (£,_2 + . . .)
= £t + ( C - 1) £t_x
is MA (1) process with
Time Series Analysis 193
with a 0 = j80 = 1 and {£,}’s are uncorrelated (0, a 2) r.v.s. We assume that the
characteristic equation
z p + a \Z p~x + . . . + ap = 0
has all the roots less than one in absolute value.
(i) Then Xt has the MA (°o) representation
Xt = £ Vj£t_j where v0 = Oq = 1, ^ = A ~
7=0
v2 = P z - C h - a2v x, . . .
min( 7, p )
Vj = P j - X -iif /'< 4
jV
(X
/=1
min(7,7>)
Vj = - X a jV H if/' > q
X ,= X w, 2 P i E ,- h = 2 V r E,-r (9.3.4)
;=i l'=o J r=0
Q P p ( oo \
S p ,e ,_ i =2 a.j Xt-j = 2 (Xj X V r Et_ j _ r
j=0 7=0
n
1=0
o
P
w |
X C t j £ t_ j _ r
ii
7=0
and obtain v fs by equating coefficients of £t_j, j = 0, 1, 2, . . . .
(ii) Equation (9.3.2) can be written as
d>P(B)Xt = y/q(B)et, (9.3.5)
where QP(B) and y/q(B) are the polynomials in Backshift operator B. Since
E(etXt_k) = 0, multiplying (9.3.5) by Xt_k(k > 0) and taking expectation on both
sides
<l>p(B)pk = 0 (k > q)
and so, X &iPk-i = 0. Now a 0 = 1 and the last equation is a homogeneous
i=0
p
difference equation of order p . Hence = X (Xr Zr ( k > q ) .
r=\
\o 2
lim o l = (Jo = \ (considering two extreme situations)
[0
(i) If residual variance a 2 > 0 then it is useless to consider the regression of
X, on {Xt_q, XM_{, ...} . Such a process {X,} is then called indeterministic, since
the process cannot be used for forecasting purposes (MA, AR, ARMA, satisfy
this property). Then linear regression on the remote past is useless for prediction
purpose.
(ii) If residual variance is 0 then it can be used for forecasting purposes and
hence is deterministic.
Theorem 9.2 Wold Decomposition Theorem (1930)
Any discrete stationary time series {X,} can be expressed as the sum of two
uncorrelated processes, one purely deterministic Vt and the other purely
indeterministic process Ut. Further, Ut may be represented as a infinite moving
average MA (<*>), i.e.
Ut = 0
£ t +
+ 2 -2 +
x£ t- \ • • •
& £ t
where e/s are uncorrelated. The proof is given in Appendix 11(B), p.284.
for any discrete stationary time series {X,} satisfying £ | y(k) | < ©o. Therefore
dF(X)
/(A ) = spectral density exists in this case and (9.4.2) becomes y(k) =
dX
I cos (Ak)f(X) dX.
Jo
The autocovariance function and the spectral density function are equivalent
in representing underlying time series. In some situations, the autocovariance
function is seen to be useful in determining the underlying structure of the time
series while in some situations the spectral density function /(A) is seen to be
more useful. Fourier Analysis of spectral density/(A) plays an important role.
k
It can be shown that the finite Fourier series sk = -■ + 2 (ar cos rt + br sin rt)
2 r=\
converges to /(/) as k —> except at points of discontinuities where it converges to
Fourier seires. Let us recall the definition “ A stochastic process {X;} is said to
be mean-square continuous if lim Var(Xr+/, - Xt) = 0.”
h->0
Theorem 9.5 (Bochner-Khintchine) If the function ph is the correlation function
of a mean-square continuous stationary time series with index set T = (- °°),
then it is representable in the form
(1) Ph =
£ e i(0h dF(co) where F(co) is a d.f.
If the index set T is discrete, i.e. 7 = {0, ±1, ± 2 ,...} then we have a theorem
due to Herglotz.
Let the correlation function pk of a stationary time series be absolute summable
(i.e. X | pk | < <*>). Then it is representable in the form
k
(2) pk = f
* -7 1
eM dF(A)
(ii) /(A) * 0,
(iii) f
J -7 1
/(A) dX = 1
(iv) /(A) is an even function of A.
Proof From Fourier Analysis
1
g(A) = T + X p* cos LA
^ *=i
Hence Z Z pm_n [cos (mX) cos {qX) + sin (mX) sin (qX)]
m —1 <7= 1
n n
= S S pm. qcos (m - q ) A > 0
m- 1<7=1
m- q = 0 n times
= 1 —> n - 1 times
Note
= n - 1 —> 1 times
Letting m - q = k, we have
*=-(«-1) V n )
Now {pkcos k }Ais absolutely summable ( E |cos (kA) | < oo) and hen
£
Kronecker’s lemma
(" - 1) . . .
lim Z *1— I pk cos (kX) = 0.
n-+oo *__(W_1) n
(n - 1,1 00
Thus, lim Z - — '— I-p* cos (&A) = Z pk cos (kX) = 2g(X) > 0
£=-(<i-l) ^ k=-oo
Now pk are the Fourier coefficients of g shows that
A = [ # U ) cos
J -7 1
g(X) 1,
(iv) The function /(A) is an even function since it is the uniform limit of a
sum of even functions (cosines).
The stationary process {X,} itself has a spectral representation (Cramer 1942),
Time Series Analysis 199
where Z(A) (- °o < A < °o) is a stochastic process with finite second moments has
orthogonal increments such that E \ Z(Z\) - Z(A2) |2 < 00 and
^ ( - o o ,r ) i f r = / ? 1
also Z(/) =
/i[- 7T, t] if 7 = [- /r, /r].
1 + 2 2 pk cos (A:A)
k=1
1 if* = 0
P
X' = e, + pe,_u pk = if &= ±1
l+P2
0 otherwise
1 cr2 [1 + 2/3cos A +
« 1+ p2
2P 2P
and /(0) = 1+ J in ) 1-
n 1 + P2 n 1 + P2
f /(0 ) = 0
If ft = - 1, then ^ 9
} /W =
The shape of the spectrum depends on the value of (3. When /3 > 0 the power is
concentrated at low frequencies giving what is called a low frequency spectrum,
/U ) = ^ 1 + 2 Z a k cos (Xk)
k= l
In
1+ Z a k (eiXk + e~iXk)
*=i
2;r
1 + Z (a V A)* + Z (ae-/A)*
*=i *=i
51 ae iX ae - i X
1+
2/r 1 - ae iX 1 - ae -iX
a2 1 + a 2 - a (e lX + e lX) + a e lX - a 2 + ae~iX - a 2
2n 1 + a 2 - a (e iX +e~iX)
0
5 1 1 - or
In 1 + a 2 - 2acos a
G2 if 5 = 1
E( XsXt )
0 if s * t
Here Yk = 0 if k * 0
= a 1 if k = 0.
So,/(A) becomes, o 2! 2 n , - n < X < n.
Autocovariance generating functions are very useful for finding spectral densities
of many time series models.
Define C(z) = 2 ykzk, where z is such that C(z) is convergent and C(z) is called
k=-oo
autocovariance generating function.
m = ^ 2rn.*=_,
2 2n y ke~i=
V ar(X ,)
S(e~a ).
2n
Theorem 9.6 Let Yt be a linear process, i.e.
Cy(z) = h(z)h(z
and f y(X = h(e,A)h(e ' )A(A)
= 2 z*-{2 2 a
*=-~ j=o /=o
oo oo
= 2 2 2 a > a , z - j+lY ( k
7=0 /=0 k=-°°
( oo ^ / \(
£ a{z l £ ajz j £ y(s) zA L setting j = k + y - 1
1=0
/ V"
= K e a )h(e~a ) f M )
Exercise 9.4 MA(^) Xt = £ aj£t^ Var(£,) = o2, where et is a white noise. Then
7=0
A (A ) = f e /l(e 'A)/l(e' U)
<7 > f <1 \
2 a ,e iAi 2 a re-"*
2;r
O
II
O
II
2 n n-k
7T— £ £ cos(&A)
^ 7T k = - q r=0
[/ - r = k or / - k = r].
Since spectral density is a real function of A only cosine terms appear.
Time Series Analysis 203
Z (X}Xt_i = £,, a 0 = 1
7=0
Assume that roots of the equation z p + cc]z p~] + . . . + a p = 0 all lie within the
p
unit circle. Applying Theorem 9.6 to £t = Z (XjXt_j, spectral density of {£,}
y=o
f />
-ikX
/.(A ) = Z OLji Z A (A),
;=0 k=0
f P
-1
Hence /,(A ) = Z a ,e ° A Z -ikX
W=0 J V*=0
Particular case p = 2
_ ___________________ 1___________________
l + a \ + a \ + 2 a j(l + a 2) cos X + l a 2 cos 2A
Assume roots of + ajZ ^"1+. .. + a p = 0 all lie within the unit circle. Putting
n
Zt = Z p r£t_r and applying Theorem 9.6 we get
r=0
Z' <1
/ za ) = f i f r ^ Z -j'.vA
Z/T ^ r=o s= 0
11=0 J
Equating the two expressions we get
<7 ( <7
£ P ,eia 2 p te -z.vA
—2 I r=0 s= 0
/,<A) = f j > 7 -
( P
Z a re irX Z a se~isA
r=0 v=o
204 Introduction to Stochastic Process
/ U ) = 2“ | e-"xp,dt
=27F £ C""A
a f 1 _ J _ N) _
2n y b - i X . b + i A) n (b 2 + A 2)
is a Cauchy type density.
The /(A ) has maximum at zero, remains
/(A) constant when A is small compared to b and then
falls off slowly. In Figure 1, pt in fact falls off
exponentially with |t | and can already be con
sidered virtually zero when the distance from the
origin is only a few multiples of b~l. The b~x
-2-10 12
Figure 2 characterizes the time needed for any correlation
between X(s) and X{s + t) to die out.
/(A ) = I e ~ i kXYk-
2 n k=
We are interested in suggesting some estimate of the spectral density/(A) on the
basis of n observations from {X,}. Given any finite realization, a covariance
stationary time series based on n observations can be represented as
Time Series Analysis 205
a0 = X, ak = — Z Xt cos (Okt ,
n t=\
bk = — Z X , sin k = 1, 2, . . . , m.
w r=i
Any function/(0 defined on n integers can be expressed in a finite Fourier
series as
L(n)
f ( t ) = Z (ak cos cokt + bk sin (Dkt), t - 1,. . . , n
*=0
2nk
where cok = — , ab bk are suitably chosen and L(n) is the largest integer not
exceeding -j.
'/ (
Un)
{ak cos cok 1^ ^ bk sin (Ok \^
/(2)
= Z + (9.6.1)
k=0
Kak cos (Okn bk sin cokn
\h*)j
fm '
\ =pxai + ... +pnan,
v /("),
* • • • Gik+i -
n
*<■N
*
0 if***'
n
3. £ sin (cokt) sin (cok't) n il if* = * ' *
t=1
n
II
II
n
4. £ cos (ay) sin (cok't) = 0 for all (oh cok'.
t~\
we can easily check that the above sine, cosine functions appearing in (9.6.1) are
orthogonal functions. Thus a set of n functions defined on n integers can be
obtained such that they form an orthogonal basis in ^-dimensional space. Thus
for suitable choices of ah bk we can always represent any function/(f) defined
on n integers in the form (9.6.1).
To obtain ak, bk we apply the usual regression technique (least square), i.e.
minimize
n f n) 2
£ \ f ( t ) - £ (ak cos cokt + bk sin cokt) with respect to ak and bk s.
/=1 k=0
9
a* = — £ f ( t ) cos cokt, k = 0, 1, 2 , . . . , L(n)
We get n '=' (9.6.2)
6* = — Z / ( / ) sin A: = 0, 1 , . . . , L(n)
n t=1
If n is even. L (« ) = f
2 n
= - % f ( t ) cos (nt)
bL{n) = \ £ / ( f ) s i n ( w ) = 0
- — (a 2 + b2) + . . . + -■ ( a 2 + b^)
Time Series Analysis 207
ANOVA Table
Source d.f. SS
Total n Total SS
(n \ 2"
2 Xt cos (okt Z Xt sin cokt
n V=' )
2
2 Xte - it(0k , k = 0, 1, • . . , tn
n t=\
1
2 71n /=l
The periodogram appears to be a natural way of estimating the spectral density
function. But we shall see that for a process with a continuous spectrum, it
provides a poor estimate and need to be modified.
Under normal autocovariance stationary time series N(0, cr^) we have
ak = — Z Xt cos a)kt - N
n /=1
2<t 2
and bk = — 2 X, sin (Okt AM 0 , for k *
n ,=i *
A 2 n
= —t— X {cos cokt sin cokt]
n L t=l
2<j 2 n
= —r— X sin 2a)kt = 0 [since X,’s are uncorrelated].
/=i
Therefore, ak, ^ ’s are independent (since being linear combinations of normal
r.v.s a*, bk are normal).
X Xte~ita) X Xr elt'a
t=\ /'=1
X X XtXr e-i(t~r)a)
/=! /'=1
Now EXt = 0. Therefore
= - 2 Z Y .- r e - ^
n t=\ t '= i
n-1
= — X y h e lh(° (n - \h\) [yt. is an even function]
n h=-(n-\)
n- 1
=2 Z yh e~ihu> - 2 I ■ihc^o
h=-(n-\) h--{n-\) n
Time Series Analysis 209
t - f = 0 —» n times,
= 1 —> n - 1 times,
= —> n - 2 times and
= n - 1 —> 1 times]
n- 1 n- 1
Z 1A I Yhe -ihw < 2 -ihco |
lA im e
&=-(«-1) n *=-(#!-1) n
AJ-1
<2 Jy*.
/?=0 «
z
By applying the Kronecker’s lemma (since autocovariances are absolutely
summable), we get
lim Z \A1 y
n—>°° / i = - ( a 2- 1) n
V a r ( / ^ ( c u )) = - p - y * 0.
A n1
Bartlett (1966) even showed that
lim Var(/;(fl)» = - p y * 0
47TZ
for stationary process with continuous spectrum. Thus l*n {(D) is not a consistent
estimate of the spectrum f{(D).
We can use
Ink
If (0 cannot be expressed as — , then the regression associated with (9.6.3)
can be computed and the test be constructed by usual regression technique.
- 2 I„((0 In(L)>
tn t=\
where In(L) is the largest periodogram ordinate in a sample of m periodogram
ordinates each with 2 d.f. Fisher (1929, Proceedings of Royal Society, Series A)
demonstrated that for g > 0,
r i , i / m' I(COj)
T
p — ^m > 8 (1 - g j ) m = rn max
II
= - I I (X ,-X )c o sft)„(r-.s)
n t=\ s—l
(n-l) M—
^ _ _
=2 Z C* cos where= and Q = Z ( Xt - X ) ( X t+k- X ) / n
k = -(n -l) t=\
n-1
ln(0)) = J ^ l n(0)p)= 1 Un + 2 Z Ck COS (0 pk (9.6.5)
2n k=\
where {bk} are a set of weights called the Lag Window and M(n) is called the
truncation point. Here we note that the values of Ck for M < k < n are no longer
used, while that of Ck for k < M are weighted by bk.
In order to use the above estimator we shall have to choose a suitable lag
window and a suitable truncation point. Two lag windows generally used in
practice are
212 Introduction to Stochastic Process
(a) Tukey-Window
bk = j ( l + cos n k / M) , k =1,2,
9.7 Forecasting
Suppose we have an observed time series X2, . . . , Xn. The forecasting
problem is to estimate XN+i or more generally XN+h (h is a positive integer). The
prediction of XN+h made at time N of the value h steps ahead will be denoted by
X(N, h). This integer /i is called the /ead time. There are many forecasting
procedures which can be classified into three broad categories,
(a) Subjective: Forecasts can be subjective using judgement, intuition,
commercial knowledge and any other relevant information procedures of this
type (e.g. Delphi method) but they will not be discussed here.
(b) Univariate: These are forecasts entirely based on past observations of a
given series, by fitting a model to the data and extrapolating. We shall discuss
only two types.
(c) Multivariate: Such forecasts can be made by taking observations of other
variables into account. For example, sales may depend on stocks. Regression
models are of this type, as are econometric models. The use of a “leading
indicator” also comes into this category. Multivariate models are sometimes
called causal or prediction models. We shall not discuss this but only two of
several forecasting procedures available for time series data.
Xt = p + a E £j -f et.
j< t
first few value of rk. Values of P, Q are selected by examining the values of rk at
k = 12, 24 . . . (where seasonal period is 12), e.g. r 12 is large but r24 is small
suggests one seasonal moving average term. So take p = 0, Q = 1.
After finding reasonable seasonal ARMA model, L.S. estimators of the
parameters are obtained by minimizing residual S.S. in a similar way to that
proposed for the ordinary ARMA model. In case of seasonal model, it is advisable
to estimate initial valuers of at and Wt by back-forecasting (or back casting)
rather than set them equal to zero (see Box-Jenkins section 9.2.4). If the model
contains a seasonal moving average parameter which is close to one, several
cycles of forward and backward iteration may be needed. The adequacy of the
fitted model should be checked by what Box and Jenkins call “diagnostic-checking”.
This consists of examining the residuals from the fitted model to see if there is
any evidence of non-randomness. The correlogram of the residuals is calculated
and we can then see how many coefficients are significantly different from zero
and whether any further terms are indicated in the ARIMA model. If the fitted
model appears to be inadequate, then alternative ARIMA models may be tried
until satisfactory one is found.
When a satisfactory model is found, forecasts may readily be computed.
Given data upto time N these forecasts will involve the observations and fitted
residuals upto and including time N. They rely on the fact that the best forecast
of all future residuals is simple zero (i.e. aN+\, aN+2, • • • = 0), e.g. consider the
seasonal ARIMA model example
= P(ax + en+i e A | Xn = x)
= P(en+1 e A - ax | Xn = x) = F(A - a X n)
where F is the common d.f. of en and the initial distribution given by the distribution
of XQ. Without loss of generality we take X0 = 0 a.s. Now Xn has the stationary
representation
? <°
a +
M
i x ^ x , =
X
p , 2- , /
v ml t 1 1 ) V "1 ) / . k 1 )
( n } f * "y
or a n - a - E etx t_\ 2 * 2,
k'=1 J / i ,=i j
We shall study the asymptotic optimal properties of a .
Var(X„) = E a 2(*“1}, it follows that lim Var(X„) < iff | a | < 1 since
k=l o
(1 - a 2n)/( 1 - a 2) if * 1
Var(X„)
n if a = 1
Proof We shall prove the lemma only in the stable case | a | < 1. The proofs of
the other cases are quite involved and hence are omitted. It is enough to prove
that Wn -£> cf2 > 0. By equation (9.8.3) we have
fn-l 2"
1 £ * , 2- , = 1 £ 2 + ( £ 2 + CtE j)2 + . . . +
« i=i ' n
l*=i J
2 £ a 2(*-D + e 2 % a 2(k-l) + _ + e2
1 k= 1 2 *=1
= ---------- r---- ) 0 a s n - ) o o .
n(l - a 2)
Hence the second term -£» 0. The first term is equal to
I 1 - a 2(n~l) 1 - a 2^ )
+ £^ . . + 1 - « 2 _2
n 1- a2 1- a 2 1- a 2
i n-1 i
= 1 E £2 - 1 t£2a 2("-') + • • + a 2£n2_,]
n (l - a 2 ) i=i ' n (l - a 2)
The second term has mean zero and variance —»0 as » whereas the first
term by the Kolm ogorov’s strong law for i.i.d. r.v.s tends to
2
£(£?)/( 1 - a 2) = ———r a.s. Therefore, by Slutsky’s Theorem as [(1 - a 2)/
1 - cr
n]Vn -£> g 2 > 0 as n —>
Hence lim V~l = 0 a.s. and Vn > kQ> 0 a.s. for large n.
n
Let $8n = <7(£i, . . . , £n). Consider Zn = 2 j£t
>t=i
218 Introduction to Stochastic Process
EZ- = E 2 v r * X l +2 Z E(Vk\v;}_xXk_,X
k=\ \<k<k'<n
=Z E { V ^ X 2_ { E { e 2 \ 3
k- 1
Also Var(X,) = — - cr2 and Var {Anl j n ) ------——r- for large n. Let
1- cr I - a1
us state the following Martingale CLT (due to Billingsley 1968).
n
Let Sn = Z Yt be a martingale and Yt’s be stationary with finite (common)
/-i
variance.
Then 5„/(Var(5„))1/2 = SnlV ^(Var(y,))1/2 A /V(0, 1) as
4
X,_i£,. Then Var(X,_i£,)----- ——- . By Slutsky’s theorem (a„ - a) g(n, a)
1 - a2
N(0, 1).
Again V y 2 ( a n- a) =Vn(A„)(V„/n)~y2has the same asymptotic distribution
as Vn A„/ct( 1 - a 2)~U2.This shows that
= (2 n) nl2exp | - j Ax) j,
= (X'BX)/(X'CX) (9.8.8)
220 Introduction to Stochastic Process
where X' -(X\, X2,. ■. , Xn) and Bnxn is a symmetric matrix with
diagonal except for the «th element, which is zero, -1 for the first line parallel
to the diagonal and zeros elsewhere, and
1 0 0 O'
0 1 0 0
0 0 1 0
0 0 0 0 0 1 0
0 0 0 0 0 0 0
-n/2
= (2 n )
If exp \ - | — | (x'Ax) +u(x'Bx) + v(x'Cx) d x x... d x n
-m/2
1/2
= 1 n A, = 1H -
(9.8.10)
J=i
Settingp - 1 + a 2 - 2v + 2 au, q = - {a + u), then writting Hn for the H which
depends on n, we obtain by expansion (of Hn = An- 2uBn - 2vCn) the difference
equation
= pHn-i - q2Hn_2, (9.8.11)
with initial values = 1 and H2 = p - q2- Let z\ and z 2 be the roots the
characteristic equation of the difference equation (9.8.11) z 2 - pz + 1 = 0 so that
z u z2 = 1 / 2 p ± 1 /2 ( p 2 - 4 q 2) m (9.8.12)
1-
H„ = 1 - *2
Z\
Hence •*2 (9.8.13)
^1-^2 1 Z2 - Z \
Substituting (9.8.13) in (9.8.10) we get the m.g.f. of the r.v.s for each n.
0i 0? 1/2
Mn(Ou 0 2) = m = 1Hn
-
(9.8.14)
g(n, a ) ’ g2(„, a )
0 2 + 202
z\ = 1 - + 0(n~1/2)
0 2 + 2 a 26 2
z 2 = a 2 + 2a 0^(1 - a 2)/n]m + + 0 (« -i/z)
Case 2. |a |> 1
(0 2 + 202) ( a 2 - 1)
Zj —1 + + 0 (| a I-3")
a In
2a 0 , (a* - 1) _ + 0(| a
| a |" a 2"
S u b s t i t u t i n g t h e s e v a l u e s in ( 9 . 8 . 1 4 ) a n d s i m p l i f y i n g , w e g e t
X'BnX
= nlim P - x ■* ^ < o
—>°o g (n ,a ) g 2( n , a )
X'BnX X'CnX
where (U,V)
g(n, a y , a)
(from Lemma 9.3 and Theorem 9.8).
222 Introduction to Stochastic Process
The ch.f. of (UyV) is given by (9.8.16) and hence by Cramer (1937) Theorem
for ratio of two r.v.s and the joint ch.f., we get the density of U/V as follows.
Case 1. |a |< 1
30
F'(x)=f(x) = — r (v, - xv) dv
2 nlI V —ex 3u
=_L r jdt
2 nii( + t l + 2 itx)m
*“ J _______ t + ix_______
( (1 + x 2)^/l + + 2 J
___________ 1 - (ix/T)__________
~(1 + x 2)(- l)([l +T 2 - 2iT]/T 2 ) ' 12
1 2 1 1
1 + jc2 71i + x2
and the autocorrelation p(k) = ap(k - 1), k > 2. Hence show that autocorrelations of X,
obey p{k)= a*~‘p( 1), k>2 and p(l) = + "•
1+ 2
Exercise 9.2 If X, and T, are two independent Moving average processes MA(qx) and
MA(q2) respectively. Then show that Z, = X, + T, is a MA(q), where <7< max (ql%q2).
Exercise 9.3 If Xt and Yt are two independent ARMA (px, qx) and ARMA (p2, q2)
processes respectively. Then show that Z, = X, + Yt is also a ARMA (p, q) process, where
p < px + p 2 and q < max {px + q2, p2 + qx).
Exercise 9.4 (a) If Xt is an ARMA (1, 1) process given by Xt = 4iXt_x + 6 et_x +
| 0 | < 1 and et is a WN (0, cr2) then express X, as an infinite linear combination of
{£t, £t~i» &t-2*• • •}•
(b) If | 6 | < 1 then express £t as an infinite linear combination of {Xt, XM, . . . Xt_h
Exercise 9.5 (a) If {X,} is zero mean second order stationary nondeterministic time
series, then using Wold’s representation theorem show that fc-step optimal mean square
predictor is given by Xt+k\t = £ Wuket-i> where £ \j/j < 00 and e/s are i.i.d. random
1 i=o 0
variables. Also show that the minimum value of the mean square error is given by
*-1
cr2 £ ipj. (b) In case of AR(1) process X, = oOCt_x + £„ | a \ < 1, show that the optimal
mean/=0square predictor is given by Xt+k\t = orXv
k In case of MA(1) process X, = £t + &£t-X,
| (5 | < 1, show that the optimum one-step ahead predictor is given by
x l+tu= px, - 8I 2X
Exercise 9.6 Show that the real valued function K(h) defined on the set of integers Z,
1 if/r = 0
K(h) = pifh = ± 1
0 otherwise
is an autocoveriance function iff | p \ < 1/2.
Exercise 9.7 Let {St, t = 0, 1, 2 ,...} be the random walk with constant drift pydefined
by S0 = 0 and
St = p + St_x + Xt t = 1, 2, . . .
where Xb X2, ... are i.i.d. r.v.s with mean 0 and variance or2. Compute the mean of St and
the autocovariance function of the process {S,}. Show that {AS, = (1 - B)St} is stationary
and compute its mean and autocovariance function.
Exercise 9.8 Show that in order for an AR(2) process X, = 01X,_1 + <j)2Xt_2 +£„ £t ~
WV(0, a2) to have an infinite moving average representation, the parameters (0b 02)
may not lie in the triangular region determined by the intersection of the three regions 02
+ 0i < 1, < 1 and [ | < 1. Show that 0, = P(1)(1 ~ P(2)) <t>2 =
1 - P 2(1) 1 - P 2(D
Also prove that p 2(l) < l/2(p(2) + 1).
Exercise 9.9 Let X, = 6Xt_x + £„ t 1, 0, 1, ... be an AR(1) process where £t are
_ °~21^
i-i.d. /V(0, cr2) and | 6 \ < 1. Show that autocovariance function is given by y (k) =
1 -0 2
224 Introduction to Stochastic Process
the spectral density is given by /(A) = | 1- 6e a |2. Also show that the log-likelihood
Oi-dimensional p.d.f.)
Ln = log p{xx< . . . , xn) of Xh . . . , Xn is - 1/2 {n log {2kg2)- log (1 - Q2)
n
+ ct- 2 [ ( \ - o 2) x f + Z { Xj -eXj_x)2}).
7= 2
P Q
Exercise 9.10 Let {X,} be an ARMA (p, q) process given by Z a rXt_r = Z /3y£,_y,
r=0 s=0
Oq * 0, Pq * 0 and {£,} is a \V7V(0, ct2). Assume that the roots of the AR and MA
polynomials are within the unit disc. Show that the variance of the prediction error is E
| Xt - Xt |2 = |V « o |2 <?2 and Xt = E(Xt \ X,, s < t - 1). Also show that
1 - | 0, |2("+1) l-IO iP
n log {2 k g 2 ) + log
l - |0 i |
+ er~2
i w- i - 1e, |2(n+1)
2 0.x,
/U ) = - 0, |2.
** Exercise 9.12 Let X, be a MA(2) process given by Xt = et- 2et_x + £r_2where £/s are
as in Exercise 9.11. Show that the spectral density
/(A ) = f i | l
( n \2 r « » ^1
+ (2 + n) Zj Yj + (2 - n - n2) S /7 ? |
\7= J I '- 1 2=1
j~1
where = X„ = X, + (1 - 0,) Z X*,;' = 2 ,...,
Exercise 9.14 (Explosive AR(1) process) Consider the Stochastic process generated
by Xt - aXt_x + et where | a | > 1 and et is HW(0, o2). Show that this may be written as
a B(oTlB~l - 1) Xt = et and hence that Xt = orlB~l(oTlB' 1 - l)-1£f where B is the backshift
operator and Brx is the forward shift operator. Deduce that the original process can be
modelled in terms of future £,’s rather than past ones.
Exercise 9.15 Suppose e2, ... are independent r.vs. all having the same mean p and
variance cr2. Define Xx = eh
Xn = aXn_i + en for n > 2,
n
where - 1 < a < 1. Show that n~l Z Xt pl( 1 - a).
i=i
Exercise 9.16 Let X0 = 0.
_ n
Define X„ as in exercise 9.15 and X„ = n~' Z X,. Prove that
a 2 12), if « = - 1.
What happens when a = 1?
**Exercise 9.17 Let Xh X2, . . . , be i.i.d. r.v.s with mean and variance <72.
_ n
(a) Find the joint asymptotic distribution of Xn = n~x Z Xj and sample autocovariance
226 Introduction to Stochastic Process
y(l) = n~l Z (Xf - X„)(X,+] - Xn) [Hint: Find the asymptotic distribution of aXn + bZn
i=1 n
for all real a and b, where Z„ = n~] Z X,X/+1. Then apply Cramer-Wold’s device and
_ 1=1
express y (1) = Zn ~ X„ .
(b) Now assume also ju = 0 and EX ?< 00 ■Define the sample autocorrelation of
lag 1 based on first (n + 1) observations as
n ( » 'i
p( 1) = n-' J , ( X , - X M ) - X i
i=1 /
/ K
n -'lX ? - X f
1 Z
(Note that the limiting distribution of p(l) does not depend on the assumption jj, = 0)
(i) Let Zf = (X„Xf,XiXi+ly. Show that -
V P3 oN
01
where = <T2 and Z = P3 Q
1
0 0
, o. 1° 0 * 4J
where F(A) is the spectral d.f. Deduce that Xn is mean square consistent (m.s) iff F(0)
i "-1
- F(0-) = 0 and show that y(0) {F(0) - F(0 -)} = Jim —^ y(k).
Queueing Theory
10.1 Introduction
Congestion is a natural phenomenon in real systems. A service facility gets
congested if there are more people than the server can possibly handle. A footpath
(side-walks) gets congested when there are too many pedestrians using the footpath
at the same time. Queueing for milk, kerosin oil etc are examples of queueing
system. In all these situations the uncertainties related to the system characteristics,
such as arrival of customers, time needed for service, the processes occurring in
the analysis of these systems are better represented by Stochastic Processes. “A
system consisting of a servicing facility, a process of arrival of customers who
wish to be served by the facility and the process of service, is called a queueing
system”. The four common characteristics of such systems are:
3. Queue discipline All other factors regarding the rules of conduct of the
queue can be pooled under this heading. In this context, rules as first come first
served (FCFS) or first come first out (FIFO), Last come first served and Random
selection for service are important. In many situation “Priority” disciplines need
to be introduced so as to make the system more realistic.
3. Number of queues When there is only one server in the system, specifications
of the above three factors give a complete description. However, in problems of
net-work flows and job-shop scheduling, one has to deal with more than one
server in series and/or in parallel.
This book will mainly deal with (i) negative exponential service times
228 Introduction to Stochastic Process
distribution, (ii) basic queue discipline will always be assumed to be first come
first served and (iii) one server.
It is convenient to simplify the description of a queueing system by a notational
representation (designed by Kendall in 1951). This can be written as :
Input distribution/service time distribution/number of servers in that order.
Some standard notations used are G (or GI if input) for an arbitrary distribution.
M for Poisson (if arrivals) or negative exponential distributions (also called
Markovian) (for interarrival or service times); D for constant length of time (for
interarrival or service times), and Ek for the Erlang distribution (Gamma with
one of the parameter, k an integer). For instance, suppose the arrivals are Poisson,
service times are negative exponential, and there are 5-servers, then the system
is denoted by M/MIs. In this notation, M /Mbo, MIDI 1, are M/G/l, M/G/l/R etc.
same as M/G/l with fourth descriptor R denoting that the system has a limited
holding capacity R.
Queueing Process
The following r.v.s which arise in the study of Queueing theory provide important
measures of performances and effectiveness of a stochastic Queueing system.
I. The number of customers Q(t) at time t waiting in the queue including
the one being served, if any, is also called Queue length.
II. Busy period 7(7,) (initiated by i customers) which means the duration of
the interval from the moment the service commences with arrival of an unit at an
empty counter to the moment the server becomes free for the first time.
III. Waiting time W(t) in the Queue i.e. the duration of time a unit has to
spend in the Queue. Also the waiting time Wn of the nth arrival.
IV. The vartual waiting time W'(t) i.e. the interval of time a unit would have
to wait in the Queue, were it to arrive at the instant t. Q(t), the number in the
system at time t and its probability distribution, given by
are both time dependent. For a complete description of the queueing process we
need to consider transient or time-dependent solutions. It is often difficult to
obtain such solutions. In many practical situations one need to know the behaviour
in steady state i.e. when the system reaches an equilibrium state after in operation
for a pretty long time. It is easier to find n n = lim Pn(t) provided the limit
exist. It is necessary to know the condition for the existence of the limit in the
first place when the limit exists, it is said that the system has reached equilibrium
or steady state and the problem then boils down to finding the steady state solutions.
There are certain useful statements and relationship in Queueing theory which
hold under fairly general conditions. Though rigorous mathematical proofs of
such relations are somewhat complicated, intutive and heuristic proofs are simple
enough and have been known for a long time (see Morse, 1947). It has been
argued in (Krakowski- 1973) also by Stidham (1974) that conservation methods
could very well applied to supply proofs of some of these relations. Conservation
Queueing Theory 229
f 1 if n = 0
with initial conditions pn (0) = \
0 if/i ^ 0
Since for irreducible (i.e. when all states communicate) and positive recurrent
(starting in any state, the mean time to return to that state is finite) Markov
process lim pn(t) exists, it is independent of the time parameter and
t—
Pn(i) —^ 0 as t —> °°.
Writing lim p n(t) = n n and setting lim p'n(t) = 0,
/—
»oo t—
>°°
t 4. 2 AqA] . . . A„.!
Also X Tt • — 1 —A7Tft —
y=o ; u n+! P\Pi • • Pn
It should be noted that pn is a nonzero solution only if
eo A A
1+ X - — —— < oo i.e. limiting distribution of Q(t) as t —> <» exists
A I + 1 ^iyU2 • • • Pn
only if
i + £ A0A i. . . x n_\ ^
( 10. 1. 1)
PoP\ " P n
By this method we can analyze the Q(t) processes of some simple queueing
systems and solve realistic examples based on them. But there are other complex
realistic models which cannot be solved by this method.
00 (
dF(x) = jE (1 - p)p"e~^x
= (1 - p) + f A(1 - p ) e - (»-X)>dy
Jo
= (1 - p ) + ^ 1 - p ) r ( p - A r ^ - ^ d y
p - A JQ
= (1 - p) + p[l - e^ ~ X)x]
- i _ pe- ^ x
It is the probability that the waiting time of the customer in the system is no
more than x. If one wants to consider the total delay of the customer in the
system (time spent in the system of Sojurn time), then the length (Queueing
time) of service has to be added to the waiting time.
Let G(x) be the probability that the total time spent by the customer in the
system is no nore than x.
Queueing Theory 233
Then we have G(x) = P[W + V < jc], where V is the service time which is
negative exponentially distributed with density jxe~fW. Hence
-f (I - ,,
pe~^vdv e-H{\-p)Xe-H(\-p)ve-nvdv
- rJ o Jo
=i _ + e-Mn-PU[e-Ax _
= 1 _ £-/41-P)*
£ (1 V + V ) = — i — - , A E ( W + V ) = 1
1*0- P Y H (l-p) 1
but EQ = So EQ = AE (W + V) .
when arrival process is Poisson W„ and W(t) have the same form.
Therefore LQ= XWQand E(Q) = XE(W+ V) i.e. Little’s (1961) form
where W = W(°o).
G*(9,z)= [
(0
J e~0 lG(t, z)dt (10.2.7)
=- z‘ +0G*(0, (10.2.9)
Multiplying both sides of (10.2.6) by e~01 and integrating over (0 < < °°), we
obtain
9zG*(9, z) - zM = [A
-M( 1 - z)Pio(9),
z,+1 - i w ~ z ) p h ( e )
or G*(9,z) ( 10.2 . 10)
(A + n + 9)z
The denominator on the right hand side of (10.2.10) has two zeros, namely
A + ju+ 9 —^(A + + 0 )2 - 4
1 = 1 (0 ) = ( 10.2. 11)
2A
Queueing Theory 235
A + p + 9+ ^/(A + p + 0 ) 2 - 4A p
r?= tl(9) = ( 10.2 . 12)
2A
where the square root is taken so that its real part is positive. Clearly
zM ( l - z ) g M / ( l - g )
G *(0,z) (10.2.15)
A(z - Z)(ri - z)
for the transform of the transition probabilities of the process Q(t). This is due
to Bailey (1954). The probabilities Pij(t) can be obtained by inverting transform
(10.2.15). Alternately p{j (t)'s can be obtained explicity by combinatorial method
(due to Champernowne (1956)).
where /. (jc) =
J
Z ( x/ 2 )2n+j /[n\(n + y)!] is the modified Bessel function of
n=0
index j.
e - ( ) ' p -j/2 (j 2 )
= P~jkj(t).
When A = p, this gives k_j(t) = kfit) (p = 1) and X(t) then is a symmetric
stochastic process.
The P.G.F. of X(t) is given by
A' n
*0(0 = X M O = f>
J )——oo
2Z7
0>.m
Hrr>0 ?-m<j THnl (t)t+m (10.2.18)
X(t)
X(0)-----------
_____ I-------
o ----------------------------------------------------------------------
The process Q(t) shares the same properties as X(t), with the important difference
that Q(t) has an impenetrable barrier at 0, such that when Q(t) reaches zero, it
remains at zero until process X(t) takes the first upward jump (of unit magnitude),
and at this point Q (t) = 1. The process Q(t) then repeats itself. It can be proved
that
Q(t) = max{ sup [X(t) - X(r)], Q(0) + X(t)} (10.2.20)
0<r<f
This representation is typical of many of the processes which will be encountered
henceforth and its regorous proof depends on storage process (due to Pyke and
Gani (I960)) and additive process (separable infinitely divisible process)
Q(t) = max [X(t) - a(t), Q(0) + X(t)] (10.2.21)
£ PiJ (o = £ (r) + £ (o + (i - p) .£ p ; z C
7=0 7 7= 1 7=0 7=0 u=—oo
e-(4*-J7i)2t pj/2
as r —> oo.
2(7T/)1/2(A p )1/4
" co, , 8)
0 if p > 1
Therefore as (-> » for all 7 (10.2.29)
1 if p < 1
By (10.2.26), (10.2.29), (10.2.23) and (10.2.24), we get
238 Introduction to Stochastic Process
0 if p > 1
lim pij(t) = (10.2.30)
t —>°o J
(1 - p ) p i if p < 1
This implies that independently of its initial length /, the queue grows unduly
long at the end of an indefinitely long period of time if p > 1, whereas if p < 1,
the Queue-length attains statistical equilibrium and tends to have the Geometric
distribution. The limiting distribution is stationary in the sense that if the initial
Queue-length 2(0) has this distribution, so does this Queue-length Q(t) for t > 0.
For we have
Pi0 (t)= £
1+ 1
°p*i(9) = [ e e‘
Jo Jo
°pij(t)dt = f e e,kJ+
e e'kH ( t ) di(t)dt
t-p '
r£Ujp_ - <gf )
if J ^ i
M n - ^)
By (10.2.19) °p* (0) = (10.2.35)
- i d l if j i
Mri - €)
where r) are given by (10.2.11) and (10.2.12).
The distribution of busy period Tt is given by
dG,(t) = °pn(t) pdt (10.2.36)
= i/tk_i(t) dt (after simplification)
= gfa) dt which is the density function of Tt(i > 1, t > 0).
(i = 1 is due to Kendall (1951) and i >1 due to Good (1948)
The Laplace transform of this p.d.f. is given by
g * (0 )= f
Jo
e-d,gi{t)dt=
Jo
f (10.2.37)
£<(£•-> - n"1)
- M n-(, 10.2.35)
fl if p < 1
P(Ti < oo) = lim g *( 0 ) =
e~^° [p 1 if p > 1
Here for each of the c channels use i.i.d exponential service time distribution
with mean rate p. If n (< c) channels, are busy, the number of services completed
in the whole system is a Poisson process with mean np and thus the time between
two successive service completions is exponential with mean 1/(np) whereas if
n (> c) channels are busy, the time between two successive service completions
is exponential with mean 1!cp.
Pnn+\(dt) P(Q(t + dt) = n+ 1 | Q(t) = n) = Adt + o(dt)
Pnn-\(dt) p ndi + o{dt)
Pnnidt) 1 -(A + pn)dt + o(dt)
Pnmidt) P(Q(t + dt) = m | Q (t) = n) = o(dt) if m * n - 1, n, n + 1,
(10.3.1)
np if 0 < n < c
where Pn (10.3.2)
cju if n > c
Thus Q(t) is a B.D. process with constant arrival (birth) rate Am= A and state
dependent service (death) rates as given above.
Substituting (10.3.1) in Chapman-Kolmogorov equation and proceeding as
the model MIM/l we find the forward Kolmogorov equations of the process to
be
Using (10.3.2) and writingpn{t) = P[Q(t) = n \ Q(0) = 0], i.e. forward Kolmogorov
equation of birth and death process becomes
P o (t) = -A /?0 (O + P P \(t)
(10.3.3)
c-l n c -l
n : 1 = 1 + 2 (A/p)" /n! + I = I + — I (p /c )« .
0 n=l ^ c c\cn~c n=l rt! c! n=0 ^
For existence of steady state solutions, the series Zq (p/c)n must converge
and for this to happen the relative traffic intensity pic must be less than 1.
c -l nn c
Then = 2 ^ +
0 n~o n\ ' c!(l - pic)'
Also nn = (p/c)n~cnc for n > c.
Note
(1) nn satisfy the following recurrence relations
[1! n p n n_x, n = 1, 2,. . . , c - 1
]
[He p7tn_i ,n = c ,c + 1,. . .
(2) The probability that an arriving unit has to wait is given by
nn0p n
E(Q) = X n n n = X 7i0 n / n \p n + X
n=0 n *=0 U ^ #i=c+l c!cn_C
c-l oo
= p n 0 X l/n\ pn + n 0 /c \( p) c X (n + c){p!c)
n-v 1
c-l
pic (
= pn0 2 l/n !p " + n 0 lc \ p c
_ ( l - p / c ) 2 (1 -
_ „ , c p ^ o
H
1
to
1
Let 0 ' be the number of customers actually waiting in the long run. Then
Q' - Max (0, Q - c).
Therefore P(Q' =0) = P( Q < c ) = n 0 Z V j ' - P i
If all the servers are busy, then the intervals of time between successive
departures are independent r.vs. having a negative exponential distribution with
mean \/c\i and the total time until the (j - c + l)th departure is the (j - c + 1)-
fold convolution of this distribution and hence
o° e-c»x (CLLx)n~C
e Cfix(cjux)
dF(x) = P(x < W < X + dx) = E 7T„ ---- 7-------rr---- cfxdx
ai=o (n - c)\
{ c (c - 1)! + c (c - 2 ) ! + '
.. + l / c p \ n 0
Queueing Theory 243
Again the probability that a particular channel is busy at any time (in the long
run) is the traffic intensity p.
10.4.1 The system with service in phases: The Queue model MIEkl 1
In this system customers arrive in a Poisson process with mean At, and the
service time V has the Erlangian distribution
244 Introduction to Stochastic Process
Taking Laplace transforms with respect to time on both sides, we then get
,i+i k p ( \ - z ) p l ( 9 )
G * U ,0 ) = (| z 1 < 1, Re(0) > 0) (10.4.2)
( 6 + A + k p ) z - k j L i - A , z &+i
In the region | z | = 1, Re(0) > 0, we have
| (0+ A + kp)z \ = \ 0 + X + k p \ > X + k p > \ XzM + kp |,
so that, by Rouche’s theorem, the denominator on the right hand side of (10.4.2),
has only one zero in the unit circle. Let £ = £ ( 6 ) denote this zero. The arguments
used in case of Ml MIX Queue then give
Queueing Theory 245
= p(z), say.
We invert the ztransform to find the distribution of the number of ph
the system.
M l - z)
In case k = 1, P ( z ) = -p - -+- A z L - (Af *+ ■ (since
p)z 1 + p z 2 -(1 + p)z
p = XIp) - -■■■-■—-— - = -j——■— which is the G.F. of geometric
(1 - z)(l - pz) 1 - pz 5
distribution pk = (I - p) p k, k > 0. (P(l) = 1 (10.4.3) and L’Hospital rule gives
kp n 0
P{ D = 1 n o = l - A/p = 1 - p if p < 1.
kp - kk'
Expand P(z) by partial fraction expansion and then invert each term by
inspection. Denominator of (10.4.3) is equal to (1 - z ) [kp-X{z + z 2 + . . . + z*)].
If z\, Z2* • • • Zk are the remaining zeros of the denominator assuming distinct
when (p < 1), the denominator = kp{ 1 - z)(l -!z\) . . . (1 - zJzk).
Then P(z) = 7 ------ 7— -— --------7— r. By Partial fraction expansion
(1 - z/zi) . .. (1 - z!zk )
pj =d - p) 2 Atz ; j j = 1 , 2 , . . . , *
which is a weighted sum of geometric distribution [expanding (10.4.4) in power
series in z, convergent in the region | z | < z,-, | Z; | > 1. Z/ are the roots of the
equation z* + z*~l + . . . + z = plk and Xz\ . Z2 • • . zk = £p(-l)*+1
246 Introduction to Stochastic Process
P(Z) = (1 - p )SA
, (z/z,- ) ' , | z | < min | 1/w, |
k
= (1 - p)/tf (1 - zm/ ) where mt = l/z„ | ra;- | < 1].
Let W be the waiting time of a customer who joins the system after an
indefinitely long period of time. If the queue is empty, W = 0; but if the system
contains j > 1 phases then W is the time to complete these j phases.
Therefore P(W = 0) = 1 - p, P(x < W < x + dx)
G * (s)= e~ su d G ( v ) (1 0 .5 .1 )
Jo
L et tn, n = 1, 2 , 3 , . . . (t0 = 0 ) b e th e nth d e p a r tu r e e p o c h , i .e . t h e in s ta n t at
w h ic h t h e nth u n it c o m p l e t e s h is s e r v i c e a n d l e a v e s t h e s y s t e m . T h e s e p o i n t s tn
a re t h e r e g e n e r a t io n p o in t s o f t h e p r o c e s s [ Q ( t) } , Q (0 b e i n g t h e n u m b e r o f
c u s t o m e r s in th e s y s t e m . T h e s e q u e n c e o f p o in t s { t n } f o r m s a r e n e w a l p r o c e s s .
Q (tn + 0 ) , t h e n u m b e r in t h e s y s t e m im m e d i a t e l y a fte r t h e n th d e p a r t u r e h a s a
d e n u m b e r a b le s t a t e s p a c e 0 , 1, 2 , . . . W r ite Q (tn + 0 ) = <2„, n = 0 , 1, 2 , . . . a n d
th e n { Q n} is a M .C . ( w h ic h w i l l b e d e m o n s t r a t e d s h o r t ly ) .
L et X n b e th e r.v. g i v i n g t h e n u m b e r o f c u s t o m e r s th a t a r r iv e d u r in g t h e
s e r v i c e t i m e o f t h e n th c u s t o m e r w h o s e s e r v i c e t im e is d e n o t e d b y Vn.
f 1 i f jc > 0
» (* ) =
0 i f jc < 0
N o w t h e s e r v i c e t i m e s o f a ll th e c u s t o m e r s h a v e t h e s a m e d is t r ib u t io n s o th a t
X ni X f o r a ll n > 1.
T hen w e h ave P { X - r| s e r v i c e t im e o f a c u s t o m e r is i>}
-Xv U v y _ r
P(Xn = r \ V n) P ( v < V n < v + dv)
r‘ 'Jo
MOO -Xv (Au)r
and so kr = P(X - r ) - ‘ dG {v),r= 0 , 1 , 2 , . . . ( 1 0 .5 .2 )
Jo
g i v e s t h e n u m b e r o f a r r iv a ls d u r in g t h e s e r v i c e t im e o f a c u s t o m e r . T h e tr a n s it io n
p r o b a b ilit ie s p tj - P ( Q n+\ = j \ Q„ = /) are g iv e n b y
kj-i+i, i> 1 ,j - 1
Pu
0 i> 1 J ( 1 0 .5 .3 )
Poj = P ij = 0
E q u a t io n ( 1 0 . 5 . 3 ) s h o w s th a t [ Q n, n 0> } i s a M .C . w it h
*0 *2 *3
*0 *2 *3
0 *0 *1 *2
0 0 *0 *1
248 In tro du ctio n to S to ch a stic P ro ce ss
A s e v e r y s t a t e c a n b e r e a c h e d fr o m e v e r y o t h e r o t h e r s t a t e , t h e M .C . { Q n} is
ir r e d u c ib le a n d a l s o s i n c e p ti > 0 , t h e c h a in is a p e r io d ic . It c a n b e s h o w n th a t i f
A
th e tr a ffic in t e n s it y p = ~ < 1, th e M .C . is r e c u r r e n t, N o n n u ll a n d h e n c e E r g o d ic .
B y E r g o d ic t h e o r e m o f M .C . t h e lim i t i n g p r o b a b i l it i e s n ,• = lim
j n — >oo
p l\J , y = 0 , 1,
2 , 3 , . . . e x i s t a n d a re in d e p e n d e n t o f t h e in it ia l s t a t e i. T h e p r o b a b ilit ie s
(Aw)'
T hen £ (* ) = 2
7=0
s 7'
fJ o ■</G(w)j (1 0 .5 .5 )
r
Jo
)vd G ( u ) = G * ( A - A j ) ( | 5 | < 1) (1 0 .5 .6 )
, {l-K'(\)}(\-s)K(s)
K(s) —s
= ^ — — - ( 0 < P = ^ ,(1)< (1 0 .5 .7 )
_ (1 - p ) ( l - s ) G * ( A - As)
G * (A - A ^) - 5
w h ic h is t h e P o l l a c z e k ( 1 9 3 0 ) - K h i n t c h i n e ( 1 9 3 2 ) ( P .K .) f o r m u la .
Note
( 1 ) In t h e M/G/ l Q u e u e b y e x t r a c t in g a p r o c e s s a t d e p a r tu r e p o in t s tn w e get
an a s s o c i a t e d p r o c e s s Q n = Q (tn + 0 ) w h ic h is a M .C . H e n c e t h e t r a n s it io n
p r o b a b ilit ie s p i; o f th e M .C . g i v e d e p a r tu r e p o in t p r o b a b ilit ie s o f t h e Q u e u e -
10.5.3 Examples
Example 1 The Model M/EjJl: Here the service time Vis Erlang (&) with p.d.f.
(fik)k v k le
dF(V) = g(v) = , 0 < v < oo
( * - 1)!
juk
Its Laplace Transform is G*(s) = y
From (10.5.6) and (10.5.7) we get
-k
fik p(l - s)
K (s) = G*(A - As) = = 1+
A - As + ilk
(1 - P ) ( l ~ s )
where p = A/p and n(s) =
1 - s ( 1 + p (l - s)/k )k *
Expanding in powers of s and comparing coefficients of sn on both sides, we
get n,j, and equivalently pn.
We can find the moments of pn from n(s). This model could also be considered
as non-birth and death Markovian, by considering that the service comprises of
k exponential phases each with mean 1/(kp) and the Chapman-Kolmogorov
equations can be written and solution obtained there from.
Special case: Ml MU Queue: putting k = 1, we get (for MlMl I Queue),
(1 - P ) ( l ~ s ) 1-p
7t(s) =
(1 - s - sp( 1 - s)) 1 - sp'
-r
where W*(0) is the L.T. of W(t). Now
e - A,(1 - s ) d W ( t ) = (10.5.10)
(1 - p)(cc/A)G
Hence IV* = , where a = A- As
G * (a ) - (1 -
_ «(1 - p ) G * ( a ) ._ *(1 - p ) G * ( s )
(10.5.11)
a-A{\- G * ( a ) ) ’ ° r (5) j - A(1 - G*(j))
This connects the Laplace transform of the distribution of the service time V
and waiting time W in the system.
G*<*)(0) = J ^ . G* ( 5 ) U = (-1)*
ds
From (10.5.12)
E ( W ) = E (W q+ V) = 2 ( 1 - ' p ) E ( y 2 ) + ^
V a r ( ) = — A_ — j [4(1 - p ) E ( V 3) + 3A{£(1/2)}2 ]
(0 < k < k - 1), then the Queue length is Q(t) = y, thus Q(t) = 6 i(0
k
The successive stages enter the system at a Poisson rate kX, whereas the
departure of each customer removes k stages from it. Thus Q\(t) is a denumerable
Markov process whose transitions during (/, t + dt] are given by
P u(dt) = 1 - kX dt + o(dt) (0 < i < k - 1),
P u(dt) = 1 ~(kX + jU) dt + o(dt) (i > k),
Pii+i(dt) = kXdt + o(dt),
252 Introduction to Stochastic Process
From these equations, an expression for the transform G*(Z, G) can then be
derived exactly as M \E k\\ Model. However, Jackson and Vickols (1956) obtain
the limiting probability distribution u} of Q\(t) by solving these equations after
setting p'ij(t) = 0. Thus u- s satisfy the equations.
kAu0 -
, , (10.6.3)
kAuj = kAUj_{ + puj+k (1 < j < k - 1)
k-\
= kAzU(z) + U(z) - I uj z j
which gives
(1 - z k ) ' L u Jz }
U(z) = (10.6.4)
k p z M - (1 + k p ) z k + 1
[1 - kd + U2k(k +l)d 2 + . . . ] - (1 - k p
” 1+ 1+
= i , ‘ ( e . - w * - < „ v p < i l.
1 + kp
Queueing Theory 253
A[ k pz M - (1 + k p ) z k + 1] = l)(z - z 0 ) S (10.6.7)
XzJ
U(z) = A—----- . The condition U( 1) = 1 gives A = (z0 - 1)/k, so that finally
Zq - z
( k-1
1 - if p
UM = } I z^ ( |z |< D
v'=° 1 - Z&0
( k- 1 N\
1*' O-Sfo) Z A z% b )J
7=0
k-1 j oo k- 1
1 - if n
I z] Z + Z Z if r
0 u=0 0 j=k D=00
*-l
= 1/Jk Z z '( l - if£ +1) + 1/Jfc Z z ' if£ ’*+1(l - i f J)
7—0 7-^
k
= l/Jfc Z z-'d - if£ +1) + 1/;t ? zy if£ '* +1(l - i f o) (by 10.6.6)
'/=o ' ' u ' ./=*
Hence, the limiting distribution of Qj(r) is given by
i/*(i - i f ^ 1) (o<y<*-i)
p ( l - i f 0) i f ^ 0 ’z k ) .
Using the relation Qx{t) - [Q\(t)/k], we get the limiting distribution of the
Queue-length
254 Introduction to Stochastic Process
(by (10.6.6)),
jk+k-\
Vj = £ uv= p ( l o(H) (j z 1).
i-i
s M
\ M '
J-l
M!
7T0 = pn + Z
n ) ! s i s n-j Pn
II
"=-v (A/ -
o
3
v»J
When 5 = 1, we get
Ml
7r„ =
(M -^y.p n °
Ml
and Uq - 1+ ■p 2 + . . . + M lp M
(M- 1)\P + (M - 2)1'
The probability that the number of machines in working order is r (< Af) is
given by nM_r.
Two measures of effectiveness for the system can be defined as follows:
Machine availability = 1 - E (N)/M
nn M
Operative utilization = Z — - . + Z n n.
n- 0 S /i=a+1
Clearly, the machine availability represents the fraction of total running time
on all machines and the operative utilization represents the fraction of time any
operative would be working. When s = 1, the operative utilization has the simple
form 1 - 7T0.
Other measures of effectiveness of the system are coefficient of loss for
machines, defined as
Average number of machines actually waiting
Total number of machines
and
^ . -, Average number of idle operatives
Coefficient of loss for operatives = ----------- :-^ --------
Total number of operatives
256 Introduction to Stochastic Process
Then the P.G.F. of X is C(s) = 2 with mean arrival rate A£X = AC'(1),
k=1
XEX
and the traffic intensity p = ■.......= A C '(l)/p. Assuming steady state solution
exists, we get
0 = - Ap0 + Xpx (10.8.1)
Therefore P d - s)p0
P(s) =
- s) - As(l - C(s))'
To get p0 we use P(l) = 1. By L’Hospital’s rule
-PPo
1 = t!£!i p (si = -* j + ac'(1) or p 0 = 1 - A
M(l-s)(l-p)
Therefore P(s) =
p d - s ) - A s ( i -C (s)Y
Particular case: M/MIl Since cx = 1, ck = 0, k > 1 and P(s) = (1 - p)/(l - ps).
Queueing Theory 257
distributed with mean y. The shop always has enough of a work backlog to ensures that
all machines in operating condition will be working. The repair time for each repairman
is exponentially distributed with a mean —. Find the average number of machines operational
at any given time, the expected “downtime” of a machine requires repair, and the expected
percent idle time of each repairman.
Exercise 10.7 Show that the two expressions for the effective mean arrival rate, in the
M/M\/k model are equal, that is, A' (1 - nn) = p(L - Lq).
Exercise 10.8 In the M/G/\ queueing system, let Qn be the number of customers left
behind in the system by the nth departing customer. Using the fact that in the steady-state
the expectation E(Qn+l) = E(Qn) and E(Q*+l) = E{Ql), deduce that the steady-state
expectation E(Q) is given by
p 2(l + C2 )
E(Q) = p +
2(1+p) ’
where Cv is the coefficient of variation of service times, and p is the traffic intensity.
Exercise 10.9 (Takacs integral equation) In Af/G/1 Queueing system, show that the busy
period distribution function G{x) is given by
GO) = f £ {(kv)ne-Xuln')Gw (
Jo n=0
where V(v) is the service time distribution and G('"’ denotes the /t-fold convolution of
G(jc) with itself. Hence deduce the functional equation
G*(s) = V*(s + A - AG*(s))
where G* (s) and V* (5 ) are the Laplace-Stietijes transforms of G(jc) and V(v), respectively.
Also verify the expression for G*(s) obtained in the text for the MIM/\ system.
Exercise 10.10 Show that, for a MIG/1 queue, the sequence of times at which the server
passes from being busy to being free constitutes a renewal process.
APPENDIX I
Sample Function of Brownian
Motion
need not be in J?in general, unless T is countable. This then implies that functions
supX(r) and inf X (0 may not be ^-measurable because they involve uncountable
te T te T
for {X(/), t e /}, / an interval of R we ask the question whether almost every
sample function is Lebesgue integrable on /. Using Fubini’s Theorem
= W(i) a.s. for any t > s. A process having this property is called a continuous
parameter martingale. For discrete parameter martingale the reader may refer
An introduction to measure theory and probability, A. Basu Prentice Hall, India.
For Brownian motion we define a continuous parameter martingale a little more
generally as follows.
Definition 1.3 Let (X(r), t e T] be a stochastic process, and let {JF{t), t e T]
be an increasing family of a-algebras such that for each /, X(t) is J^t) - measurable.
[X(t), J^lt), t e T} is said to be a martingale if i > s implies E(X(t) | s)) = X(s)
a.s. and E(X(t)) < ©o. The process is said to be a submartingale (supermartingale)
if the last equality is replaced by > (respectively, <).
For any Brownion motion t > 0}, we can take 3T(t) to be the smallest
cr-algebra with respect to which {X(s), s < t] is a martingale. To prove this
merely note that for t > s,
E(X(t) | j r (s)) = E{(X(s) + X(t) - X(s)) | j r {s)]
To prove theorem 1.1, let us first state and prove an extremely useful condition
due to Kolmogorov which guarantees almost sure sample continuity of a separable
process, (see Definition 1.1, Appendix I).
Theorem 1.2 (Kolmogorov) Let (X(r), t e T] be a separable process and let T
be a finite interval. If there exists a, /3, c > 0 such that E | X(t + h) - X(t) \a <
c hl+p then
sup | X(t) - X(s) | -> 0 a.s. as h 0
t,seS
\t-s\<2n
Pi I X(, + *) - m i a e) ^ w f ., o.
£a
Hence P(ZV > (l/2u)r) < 2 Z ' P( | X((* + l)/2°) - X(k/2U) \ > (l/2 u) r)
< c2u(l/2 u) l+5 = c2-lSu
Since X 2 - ^ < °o We get X P{ZV >(
1/2" )r ) <
u=0 & u=0
By Borel-Cantell Lemma, there exists N((0) almost surely finite such that
Zv(co) < \!2vr for all v > N(a>) and
lim X Z „ = 0 a .s . (A.I.l)
n—
>oo v=n+1
If | t - s | < 2"”, then we can find k such that 0 < 2" and 11 - k/2n \ < 2~",
\ s - k/ 2n \< 2~n.
If t e S= 2{k/", ^ = 0, 1.2" - 1; n = 0, 1, 2 ,
t must be of the form
m
k/2n ±X 2~i( tv = 0,1)
Thus,
n+m oo
|X ( 0 - X ( * / 2 " ) |< s z„<
u=«+l
X z„
u=n+l
N (n)
= Z [(VT(^n )) - W(t[n
_l))2 - (; - ;<">)]
N (n)
ESn= 0, E S 2= 2 ^)- W
V
W
K
E
N (n)
=2 2 (t(kn) - t (kn_ \)2 < 2A n( b - a )
Since 2 A„ <
we have for £ > 0,
n=l
2 P ( |S „ |> £ ) < o o .
n
2n
lim Z a.s. and in L2,
n —»«> /c = I
z
*=1
max
1<j<2n
The numerator on the right converges a.s. to t, while the denominator tends to
zero because Brownian motion is uniformly continuous over bounded intervals.
Therefore
2n
1
lim Z w{— t
1
Fix p > 0 and suppose that W (t) has derivative W'(s), | VT'(.s) | < /3, at some
point s e [0,1]; then there is an n0 such that for n> n0
\W(t) - W(j) \ < 2 P \ t - s \ \ f \ t - s \ < 2!n, (A.I.3)
Let W(.) denote functions on [0, 1].
An = {W(.); there exists s such that
| W(t) - W ( s ) \ < 2 P \ t - s \ , i f \ t - s \ < 2In]
Then An T A c {set of all sample paths of W(t) on [0, 1] having a derivative
at any point which is less than P in absolute value}.
If (A.I.3) holds, and let k be the largest integer such that kin < s, then the
following is implied
k +1 k +1 k -l
Y„ = max W\ w\ W -W\ W\ ^ \ - W \ <6f3/n
n n n
Therefore, if
Bn = {W( )\ at least one Yk < 6pin].
w\ ± |- w\
k- 1
= n 3 -™Ll2dx
=n iy = /w)
4lm
Therefore P(A) = lim P(An) < lim P(Bn ) = 0, which implies the theorem.
n —>oo n —>oo
X(t)
lim sup . ■■■■■:= = 1 = 1 (A.1.4)
t-*°° <y]2t log log t
X(t)
and lim sup = 1 = 1 (A.1.5)
0 <yj2t log log (1/0
sup v■/.-:= = sup . .-= a.s. for any sequence {tn} which
te(a,b) f 2 t log log t n ^ 2 t n log log tn
is dense in (a, b). So the supermums are r.v.s.
To prove (A.I.4) it is sufficient to prove that for every e > 0,
r X(t)
lim sup . ==T > 1 + e = 0 (A.I.6)
^ 2 / log log
/-> o o
and
P(Ak ) =-^2/(3ra*) f
"(l+e/2^2ak
_______ exp
log logrt*
___________ y
< J 2 l { n a k)
* ( \ + e / 2 ) y j 2 a lc log log a k (l + e / 2 ) ^ 2 a k log log a k
x exp (~y2f(2ak)) [ dy
< (1 + el) ^2? log log f-y/(2a* log log a* )/(2a*~' log log a*"1).
Choose a > 1 such that
1/2
log k + log log a
(1 + £/2)V^z < 1+ for all* > 2 .
log ( / : - ! ) + log log a 4
[Note: If we had taken a < 1 but sufficiently close to 1 we could have proved in
the same manner the first half of the local law, namely
X(t)
lim sup —======= > 1 + e = 0 for every e > 0.]
/->o ^J2t log log t
To prove (A.I.6), let us take a > 1 (but this time very large).
Let Bk= [X(ak)- X(ak~') (>1 - ell) V2 a *log log a* ],
Since X(t) has independent increments, the events Bk are independent. Since
X(ak) - X(ak~')d ( 0, ak - ak~'),
N
r ^ 2 A o g \ o g ~ (1 - r 2) log log a k
Appendix I: Sample Function of Brownian Motion 267
( 1/2
lo g a)r2+1
Hence, P(Bk )> k~P, where 0 < j8= ( r 2 + l)/2 < 1. Therefore,
2n
V J
E P{Bk) = °o and by Borel’s zero-one law P(Bk i.o.) = 1. Since the distribution
k
of process (X(/), t > 0} is synmetric, we obtain from (A.1.6)
P[-X(ak *) < (1 + 5) ^ 2 a k 1 log log a k 1 for all large k\ = 1 for any S > 0.
Hence P[X(ak) > (1 - e/2) ^ 2 a k log log a k - (1 + 5) ^ 2 a k~{ log log a k~{
for infinitely many k] = 1.
But
(1 - e/2) ^ 2 a k log log a k - (1 - 5) ^ 2 a k~x log log a k~]
1 + 5 logloga k- 1
where A = -< e/2 if a is large for all k > 2.
-sfa V log log <
*(«*)
lim sup ■ ------------------ > 1- e = 1
or
yj2ak log log a k
Since
X( ak ) i:____
lim sup > 1- e lim sup . X(t)—-— > 1 - e
k-+°° ^ 2 a k log log a k £-»«> ^\2t log log t
(A.1.7) is proved.
Note: If one select 0 < a < 1, with a close to zero, then similarly we can prove
X(t)
lim sup . > 1 - e = 1. (A.I.8)
/->o ^]2t log log | 1/f |
Corollary 1.5.1 The function X(t) has (a.s.) arbitrarily large zeros.
Proof The r.v.s {-X(0} are again Brownian motion to which the law of iterated
logaritham (A.I.6) applies. Hence (A.I.6) holds for -X(t) as well as X(t). This
268 Introduction to Stochastic Process
means that X(t) has both positive and negative values for arbitrarily large t, and
continuity does the rest.
Corollary 1.5.2 For each t0, the function X(t) - X(t0) has (a.s.) a sequence of
zeros converging to t0 from above. In particular t0 = 0 is not an isolated root of
m -
Proof follows from local law of iterated logarithm for Brownian motion.
for every a e [0, rj/y]. In particular, the paths of X are Holder Continuous of
order a. For each positive integer m, let Dmbe the set sm= {2~mi}, i is an integer
in the interval [0, 2m). The set D = u Dm is the set of dyadic numbers. Let
further Ambe the set of pairs (5 , t) in Dmsuch that 11- s | = 2~m\ there are 2msuch
pairs. Now set Kt = sup | X(s) - X(t) |. By assumption
(M)eAt
E[ Kj ] < S E\ X( s ) - X( t ) \ Y < 2 i - C 2 ' l(]^ )= c 2 iri
1 (s .t)e A j 1 1
Appendix I: Sample Function of Brownian Motion 269
+ i-m
.2 )]
Here series are actually finite sums. It follows that
Then, we have
Ma < sup{2(w+1)a sup \ X ( t ) - X ( s ) \ \ s , t e D , s * t ]
m \t-s\< 2~ m
For y< 1, the same arguments holds to EM a instead of (EM a )1/y. It follows,
in particular that for almost every co, X is uniformly continuous on D and it
makes sense to set
X(r, co) = lim X(jc, co)
s-*t
seD
By Fatou’s lemma and the hypothesis, X(t) = X(f) a.s. and X is clearly the
desired modification.
Now back to Brownian motion (increments being normal), for every p > 0,
E | W(t) - W(s) \2p = Cp \ t - s
So Brownian motion has a modification which is Holder continuous of order
a for a < (p - \)l2p - 1/2 - 1/2/7. Since p can be taken arbitrarily large, the result
follows.
Note Condition A.1.10 is the same condition which appears in theorem A.1.2 to
prove much weaker property of Brownian motion.
Corollary 1.7.1 The Brownian paths are a.s. nowhere locally holder continuous
of order a for a > 1/2.
Proof If | W(t, co) - W(s, co) | < C | t - s \a for 0 < 5 , / < 1 and a > 1/2 then
270 Introduction to Stochastic Process
? )- W
O
C (t,,, (0))2 < C2 sup I tM - |2“-' (A.I.10)
1 /
Now by theorem A.I.3 and the note after that
if sup | A„ | —> 0 where {An} = {/rt+1 - tn] is a subdivision of [a, b],
n
Z( W(tM ) - W(ti ))2 (b - a) a.s. as -» °o. (A.1.11)
i
Hence the right hand side of (A.I.ll) would converge to 0 as n —> if W(t)
is locally Holder continuous which is contradictory. Note Theorem A.I.7 and its
corollary leave open the case a = 1/2. It can be proved by the law of iterated
logarithm for Brownian motion (Theorem A.1.5) that the Brownian paths are not
Holder continuous of order 1/2.
APPENDIX II
Second Order Stochastic Analysis
Theorem II.2 Let (X(r), t e T ) be a second order process. If X(t) has a derivative
at all t E T, then the derivatives
d 3 3^
T(r, s), T(r, s), T(r, s) of the covariance function T(f, s) exist
ot os otos
and are finite on T x T.
Proof If X(t) has the derivative X(t) (q.m.) at t eT, Then
X(s + u) X
- (s)^
= lim E(X(t) (from A.II.2)
u—^O
(A.II.4)
= EX(t)X{s),
shows that the existence of the partial derivative ^dr ( t , /') depends on that of
the derivative of X(t). We shall need the concept of the second generalized
derivative of F(t, t'), defined as the limit (if it exists) as /i, h' —» 0 of the quotient
s, t + s') -ru sr
+ r t)}; (a.ii.6)
SS
the limit of which (if it exists) as > 0 is the second generalized derivative
of F(t, t') at (/, t).
Sf l = (A.II.7)
Then (AIL 10) often admits the interpretation of a linear system with input
X(r), output Y(t), and impulse response h(t, s).
276 Introduction to Stochastic Process
X ( t ) - X ( t 0) = f Y(s)ds, t E T
then the process Y(t) is called the derivative of X(t) and is denoted by X'(f), t E
T. Thus
r , u . , ) = ^ 7 r , (, . o = ^ r.(.-«) = - r .
Orthogonal Expansions
As on page 275 the Hilbert space generated by the class of second order r.v.s
X(t), t e T will be denoted by %x = X- A family ^ o f elements (random Variables)
in Xx is said to be orthonormal (0 - AO family if any two distinct elements Y and
Z of & satisfy
Since {X(t),te T) is q.m. continuous, we have for every 7, X(t) = lim q.m. X( t n).
n—
><»
If follows that every element in Xx is a linear combination of {X(7), t e T' } or
the q.m. limit of a sequence of such linear combinations. In short, the countable
family {X{, t E T ) is dense in Xx- It follows that every O-N family in %x is at most
countable. For a q.m. continuous process {X(7), t e T}, let {Z;, j*> 1} be an
O-N family in %x. If Z e Xx^ then
Suppose that {Z;, j > 1} is a given complete O-N family in Xx and set
0j(t) = (X(t),Zj). (A.II. 19)
Then from (A.II. 18) we get
since 0r(O’s are the cross covariance function of the q.m. continuous X(t) and the
r.v. Zr are continuous functions.
Further, the set of functions {Qj(t),j > 1} is also linearly independent, i.e. for
every n
r ( r , 5 ) = 2 0/ ( 7) 0/ ( 5) 7, 5 E T.
7= 1
Then by Karhunen general representation theorem (1947) there exists complete
O-N family of r.vs. {Z;, y > 1} in Xx such that
Appendix II: Second Order Stochastic Analysis 279
X(t,co)= E <pj(t)Zj{(o),t e T .
7=1
Thus (A.II.20) and (A.II.21) represent each other. The representation (A.11.20)
permit uncountable set of r.v.s {X(0, t e T] to be presented by a countable
number of orthonormal r.vs. {0,(0}.
Though it is usually difficult to express the r.v.s {Z; } explicitly in terms of
T(f, s) = E 0 (00/ (s) for each (t, s) e [a, 6] x [a, &] (A.II.25)
f
Ja
T(/, 5)0y (5)^/5 = A7 (00/ (0, a < t < b (A.11.27)
The integral equations of the form (A.11.27) are well known and we shall now
summarize some of the elementary facts about integral equation of the form
(A.11.27) which are necessary for Loeve-Karhunen expansion of second order
process.
f
* a
T(f, s)(j>{s)ds = A0(/), a < t < b (A.II.28)
where fa, b] is finite closed interval and T(f, s) is continuous on [a, b] x [a, b].
The A * 0 for which there exists 0 satisfying both (A.11.28) and the condition
rb
I |0 (/)|2 dt < °o, is called an e/gen va/we of the integral equation. The
J a
corresponding 0 is called an eigen function. Let us list the important facts.
1. Any eigen value of (A.11.28) must be real and positive. The fact that A is
real follows from the Hermition symmetry T(L s) = F(s, t). The fact that A is
positive follows from the non-negative definitness of T .
2. There is at least one eigen value for (A.II.28), if T is not identically zero.
The largest eigen value Ao is given by
fib fib
A0 = max J J T ( t , s ) Q ( s ) Q ( . t ) d s d t , (A.II.29)
- 1/2
where ||0|| = |0 (O |2 dt
Note: This fact is not easily proved. It depends on both the nonnegative definiteness
of T and its continuity (see Taylor 1961, p 334-336).
3. Let 0O(t) denote the normalized eigen function corresponding to Ao, then
0o(0 is continuous on [a, b], since
i ^
0o (t) = —— f V(t, s)0o(s)ds, a < t < b (A.11.30)
A 0 Ja
fib fib
~E X ( s ) M t ) X(T)dT + E 0o(f)0oU) X{T)<j>0 (u)drdu
"ra Ja Ja
Appendix II: Second Order Stochastic Analysis 281
r _
0, (00/ ( =
"a r
* |/ 0(O|2 * = 0.
f
«a
T(f, s)(j>{s)ds = A0 (f), a < t < b (A.11.41)
Proof (a) First note that (A.II.4A) is well difined, because X(t) being q.m.
continuous and hence 0„(f) is continuous on [a, b] by result (3).
rb _
Now 4^ k Ak - [ X{t)Qk(t)dt,
E(X(t)Ak) E(X(t)X(s))4>k(s)ds,
!)=vt 1
= —r= = f r(f, s)<t>k(s)ds = -j= = Ak0k(t) =
Ja
=£=0
i V * r * ( ') f c ( o < A T = k2=0 * * lfc (o i2.
IV. s ) = t)X
(X
E (j)) = £ Ak<k ( f ) 4 (j)
k=0
= lim E( Xn(t)Xn(s)\
or I s<p(s)ds + t
Jo I 0 (s)ds = A0(r)
Proof Let X(t) denote the projection of X(t) on Xt-i 0-e- the linear subspace of
% spanned by the r.vs. X(s), 5 < t - 1 i.e. limits of Cauchy sequences of linear
combinations of XM, Xt_2, . . . and the all such linear combinations. In fact X{t)
is the linear least square predictor of X(t) given X(t-\), X(t-2), . . .). Define the
process {£,} by et = X(t) - X(t). Then £tXXt-\ i-e* £t orthogonal to every
element of %t-v For anY PaiL £r} with, say s < t. We have et±Xs c Xt-b an<^
€s e Xs (since both X(t) and X(s) are in x s)-
Hence es and et are orthogonal i.e. E{es £t) = 0 for s * t. Now consider the
projection of X(t) on Xt (£X the subspace spanned by £„ Et_1 ,.. and this projection
can be written as
Appendix II: Second Order Stochastic Analysis 285
E[X(t)e,] = <r}c,-,)-
For s > t,
V(t) e X, and es± x , ( " e,e
Hence £s is orthogonal to V(t) for all s, t, and f/(s) is orthogonal to F(r) for
all v, r,f/(.y) being linear combination of es for all s.
Therefore || X (f)||2 = || U(t) ||2 + 1| V(t) ||2
and we then obtain
‘remote past” and can be predicted with zero prediction variance. To establish
uniqueness we observed from
X(t) = £ + Vt (A.II.53)
.,=0
286 Introduction to Stochastic Process
that if {£,} and {Vt} are any sequences satisfying (A.11.53) and having the desired
properties proved above, then Xt-\ c sp {es , 5 < t - 1} © sp {Ve, s < t - 1}
from which it follows, {£,} being white noise (orthogonal) and E(etVs) = 0,
that et is orthogonal to Xt-v Projecting each side of (A.II.53) onto %t-\ an<^
subtracting the resulting equation from (A.II.53), we find that the process {£,}
must satisfy £, = X, - X, = X, - PXt-\ X, • By taking inner products of each side
of (A.11.53) with et_j we see that cj must also satisfy (A.II.51). Finally, if (A.II.53)
is to hold, it is obviously necessary that Vt must be defined as in (A.II.52).
Note
1. For a general stationary discrete parameter process {X,} (whose spectrum
need not necessarily be continuous) let us consider the sequence of prediction
variances (J2 = E[Xt+m - Xt+m]2, m > 1. By stationarity of X ,, a 2 does not
depend on t. Clearly %s c %t, s < t and hence a 2 < c \ < . .. < cr^, < . . .
(obviously we would expect that variances to increase as we predict further
ahead).
If a 2 > 0 the process is called regular or nondeterministic where as if crj2 = 0
process is called singular or deterministic. If a 2 > 0, it follows that <7% > 0 for
all m. On the other hand if a 2 > 0 then X/+1 is its projection on Xt i.e. Xt+i= Xt
for all t and all the Xt spaces are same. Thus, Xt+m is its own projection on Xt f°r
all m > 1 and hence 0 = <72 = a 2 = . . . a 2 = . . . We therefore see that the
{ a 2} are either all positive (corresponding to X, regular) or all zero (corresponding
to X, singular). In the later case X, can be predicted “prefectly” i.e. with zero
prediction variance, from its past values.
2. If X -00 = {0}, then Vt = 0 in Wold’s decomposition and X, = Ut is “purely
nondeterministic”. If o j = 0, then Ut - 0, and Xt - Vt is then a deterministic
process.
Appendix III
Stochastic Integral (Ito Integral)
Introduction
In recent years, it has become apparent that physical systems, classically modelled
by deterministic differential equations, can be more satisfactorily modelled by
certain stochastic counterparts if random effects in the physical phenomena as
well as measuring devices are to be taken into account. In this context, an
ordinary differential equation - ^ - = /( r ,* ) would be replaced by a random
differential
dX(t)
= F (f,X (0 , Y(t)) (A.III.l)
dt
where Y(t) represent some stochastic input process explicitly. It is often seen that
it is not possible to interpret (A.III.l) as an ordinary differential equation along
each sample path. For example
dX(t)
= f ( t , X( t ) ) + g((t,X(t))N(t) (A.III.2)
dt
with N(t) being a Gaussion white noise process. Equation (A.III.2) has been
popular in the engineering literature especially, since Gaussion white noise
approximates the effect of the superposition of a large number of small random
disturbances, a situation encountered in engineering systems. However, the
irregularity of the sample paths of N makes (A.III.2) intractable mathematically.
Just as a solution of (A.III.l) satisfies the deterministically integral equation
X(t) = X(r0) + I f ( s , X( s)) ds a solution of (A.III.2) should be a solution of
J'o
the random equation
but unfortunately the last integral in (A.III.3) cannot be defined in any meaningful
way. To deal with this difficulty, the integral in question is replaced by an integral
of the form
where W is the Wiener process and with the motivation being that, at least
formally, W(t) = N(t) and so dW(t) = N(t)dt. However, even equation (A.II.3)
with the second integral replaced by (A.III.4) is not unquestionable. The natural
interpretation of (A.III.4) would be as a Stieltjes integral along sample paths.
But this is not possible, since the sample paths of the Wiener process are not of
bounded variation. Furthermore, if different choices of the T/ are made from the
subinterval [/M, rj of the partition t0 < tx < t2 < . . . < tn = t, the natural
approximating sums
=0 (A.III.6)
where 0 < a = t (nn) < . . . < t (nn) - b is a sequence of partitions of the interval
l a , b l A t [ n) = t[n) - t [ % d n = max At[n\ and A W {k n) = W{t[n)) - W (t[n„\)
results in the stochastic integral (by taking q.m. limit as dn —> 0)
Appendix III: Stochastic Integral (Ito Integral) 289
A W ( t [ n) ) + (1 - X) W( t {kn] ) (A.III.l 1)
f
*a
W(t)dW(t) = \!2{[W 2 (b)- W 2 {a)} - (b - a)}
and it can be shown that the integral interpreted in the Ito sense satisfies
\
(a) W(t )dW(t) - 0 and
f E\ f ( t ) \2 dt <oo (A.III.14)
~a
Denote by £the class of random step functions satisfying (A.III.12), (A.III. 13)
and (A.III. 14).
The stochastic Integral is now defined as follows:
1. If a = t0 < tx < . . . < tn = b be partition points of the interval [a, b] are
independent of (O and
satisfy conditions (A.III.13) and (A.III. 14), then we define the stochastic
Integral by
2. If/satisfy (A.III.13) and (A.III.14), then we shall show that three exists a
sequence of random-step functions [ f n( c o , r)} satisfying (A.III.13) and (A.III.14)
such that
cb
It will then follow that I /„ ((0 , t ) d W ( c o , t) converges in mean square as —>
Ja
and this q.m. limit is the same as any sequence of random step functions which
satisfy (A.III.17).
Therefore, we define
b *b
f(w,t)dW (CO, t ) = lim /„ in q.m., (A.III.18)
J n —>°o J
"a * a
IIf-nI2I = f \f(.,t)2I
E -> Oasn
Appendix III: Stochastic Integral (Ito Integral) 291
Proof (a) Suppose E[f(., t ) f ( . ys)] is continuous on [a, b] x [a, b], that i s ,/
is q.m. continuous on [a, b], then an approximating sequence of random step
functions {/„} can be constructed by partitioning [a, b], sampling /(g>, f) at
partition points tfn), defining f n (a), t) = /(cu, tjn)), t ^ < t < t^ , and refining
the partitions to zero (taking max (tj*j - t (p ) —> 0 as n -» oo). Since f is q.m.
continuous, E |/(., t) -/„(., t) |2 —» 0 as n —» °ofor every r e [a, 6]. By dominated
+ ) -/ (f)
h \2 d <4E J |/(0
Define the function 07(r) = [jt] / j j an integer > where [ ] denotes the “greatest
integer” function.
Since Qj(t) —> t as j —» «>, (A.III.23) can be replaced by
E f |/(5 + Qj (/)) ~ f ( s + t) \2ds —> 0 as7 —» «> for any fixed r (A.III.24)
J —00
J - 0 0 J a- 1
f U + 0 /(0 ) - / O + 0 -r•'rj-i J-
If ( s + 0 ,(0 )
f E \ f(s + 0jn( t ) ) - f ( s +t) |2d t - ^ 0 as j n —> °°, for almost all 5 (A.III.26)
Jo- 1
f E \ f ( s + <l>jn ( t - s ) ) - f ( t ) \ 2 d t ^ 0 (A.III.28)
Jo
/( /„ ) = S /„ . - , t)n))] (A.III.29)
E\ /(/„ ) |2 = ? 2 E ( f f at A* W(">)
J
If k>
j , then Ai W(") is independent of ) while )is
J^,(n(-measurable. Hence E i ^ j ^ A j W ^ A t W ^ ) = 0 if k ± j
and E \ I ( f n ) \ 2 = l E [ \ f nj \2 ( A j W ^ ) 2]
= Z E \ f n.\ 2 E[(AJ W ^ ) 2 |
Appendix III: Stochastic Integral (Ito Integral) 293
Cb
Z E \ f nj\2 ( t ) H - t (; ) )= \ (A.III.30)
J Ja
Cb
E|K f m+n) - / ( / „ ) |2 = E| n(., 0 - / „ (.
fm
+
Ja
<2 [b E \fm+n(-.,t) / ( . , r ) | 2 dr + 2 f £ | / „ ( . , * ) - / ( . , r ) |2 dr
•'rt Ja
—> 0 as n —> oo (A.III.31)
Hence {/(/„)} is a mutually q.m. convergent sequence, and there exists a
second-order r.v. 1(f) such that E \ I ( fn) - 1(f) |2 —>0 as n —> <».
(c) Suppose {/„)} and {/„'} are random step functions such that
|| f n -/II —>0 as at -> °o and \\f' - f \ \ -> Oasn oo.
Then, II/. || < V 2 (||/. - / | | 2 + ||/ n' - / | | 2)1/2 -> 0 asn -> ~
Therefore, E\ l ( f „) - „I(f ') |2 —>0 as n » °° and
—
Then f
J a
f ( . , tt)dW(t
) dW( t )) f
"c ■ "ra E(fg)dt (A.III.32)
= ± [ E \ I ( f + g ) \ 2- E \ I ( f - g ) \ 2]
+ E[E\ I ( - i f + g) I2 - g) I2(A.II
E \ I ( f ) \ 2 =E\ / ( / - / „ ) + /( /„ ) |2
= E | / ( / „ ) |2 “ )])
+ £| / ( f - / ,) |2.
Since E\ I ( f - /„) |2 —> 0 as n —)we have
E \ I ( f ) \ 2 = lim £ | /( /„ ) |2 = lim f
E\ f ( . , t ) |
Now we want to define definite Ito Integral and in this vain let us state and
prove the following theorem.
Theorem III.3 Let {W(r), ^ } be a Brownian motion and let/satisfy (A.III.13)
and (A.III. 14). Define a process {X„ a < t < b) by
Then for each n, {Xn(co, t), a < t < b] is a martinglae and hence
E[Xt - Xs | = £[*(., f) - X„(., t) \ j r \ ~ E[X(., s) - X„(., s) \
Since E\ X(., t) - X„(., t) |2 —> 0 as n —> «>, applying Jenson’s inequality for
conditional expectation, E[X (., f) - X„(., r)] - E [X(., s) - X„(., s)] —> 0 in q.m.
as n —»
Hence, £ [X ,-X J | ^ ] = 0 a.s.
Appendix III: Stochastic Integral (Ito Integral) 295
Ito Formula
Our aim here is to give a feel for simple ITO formula and not to give the most
general formula. In usual calculus we know that for a continuously differentiable
function/(r) on (0, <»).
m=Jorf'W*+m
or /(0 -/(0 )=
Jo
f
f\s )d s .
f(w (t))-f(w m =
Jo
f r (w(s))dw(s)?
The answer is no, but some interesting and useful facts can be proved. Why does
not the formula work in ordinary calculus? This is because of Taylor’s expansion
which says f ( t + At) - f(t) =f' (t) At + f "(t) (At) 2 and (A t) 2 term is too small
and can be ignored. To derive the calculus formula we assume that/has continuous
second derivatives (i.e ./e C2) and £ e (t,t + At). Take a sequence of partitions
n „ of [0, t] with || n „ || = ()max_i (tj+{ - t j ) -> 0, n „ ={0=f0<fi< . • .</*= t}.
Of course the points tk depend on n.
Then
n- 1
fU)- /( 0 ) = .2 [ f ( t M ) - f ( t , )] (Talescopic sum)
n-1 n-1
= 2 /'(< , )[//+i ~ t i ] + 2 1/ 2 / " ( |; )(fI+, - ti )2 (by Taylor’s expansion)
1=0 1=0
As n —>oo, the first term —> I / ' ( s)ds by definition of Riemann integral and
Jo
n-1
| Second term | < 1/2 C 2 (f/+1 - t{)2, where C = sup \f"(s) |
1=0 0<s£t
^ 1/2 Ct || Iln || —> 0 as n —> °°.
For Brownian motion on the average (W(t + At) - W(t) ) 2 behaves like At (not
like (Af)2) and hence can not be ignored.
296 Introduction to Stochastic Process
Ito formula (1) Let/be a real valued function with continuous second derivatives.
Then
0)) + J o
f(W
= f ' ( W( s ) ) dW + ^ J ' f"(W(s))ds.
Remark Left side is the process whose value at (a), t) is f(W(co, t)) and f( W (0))
n m t ) ) - / ( w ( o )) = 2 [ / ( w ^ V ) - /( w ( r ,(n)))]
where lies between points W(aj, /■'” ) and VV( . t-" /). Note that first term I
to the r.v. I f ' ( W(s) dW( s). This is obtained under weaker condition
Jo
I |/(ft), t) \2dt < 00 a.s. instead of the condition I E|/(ft), 0 |2 dt< 00. Here
Ja Ja
the stochastic integral is defined by
fib fib
I f ( a) , t ) dW( co, t ) = lim I f n (ft), t)dW(a), t) in probability sense,
*J a n— >00 I
where f n(co, t) is defined by
Appendix 111:Stochastic Integral (lto Integral) 297
f IH<o, 0 |2 d/ < n
fn («0, t) = /(© ,O if *a
0 otherwise
This can be proved as follows :
Cb
Now /( /„ ) = I f n (co, t)dW(co, t) is jointly measurable, for fixed co it is
•b
measurable and f E\ f n (co, t) |2 dt < oo, so that I(fn) is well defined. Now,
rb
for any co such that I | /(co, t) | 2 dt < min (m, n), we have
"a
sup \ f m(CO,t) - f n 1(0, t)I = 0
and that implies
"a r
It follows that for every £ > 0,
.o-r "a
f m(co,t)dW (co,t).
M-l
2 / " ( W ( / , ))n v (ri+1) - W(/, )]2 (A.III.38)
i=0
n-\
The first term of (A.III.38) in modulus < cn (co) Z [W(tM ) -
i=0
W(t t )]2, where
cn (co) = 1/2 sup sup |/" (W (fl ) ) - / " ( * ) |, between W(co, t{) and W(co, ti+x).
i x
Note that by continuity of/"(jc) and W(t) paths it is easy to see that cn (co) —> 0
n- 1
a.s. as || fl„ || -» 0 and Z [ W(ti+{) - W(t{)]2 -4 t in q.m. The product
/=0
n-\
Cn(CO) I . m t ^ - W i t t ) ] 2 - ^ > 0 .
i=0
1 n ~ l
nd term of (A.III.38) = j Z r ( W ( t , ) ) { l
n- 1
+j x n m t ' M t M - t , ] (A.III.39)
298 Introduction to Stochastic Process
The first term of (A.III.39) in modulus < C\(co) " l ( W /i+1) - W(t, ) ) 2 t
1=0
I ’ f"(W(co, s))ds in probability. Now for each fixed t the two random variables
Jo
on either side of ITO formula (1) are equal a.s. So the two processes are equal
a.s. because both sides are obviously continuous path processes.
ITO formula (2) Let / (x, t) be a real valued function on R x (0, 00) with
continuous second partial derivative with respect to x and has continuous first
derivative with respect to t.
, f' 3 f ( W ( s ) , s ) J, 1 f ' 3
Jo — 3 7 — ^ 5 + 2 j 0 ------- ------------
Proof Proof is exactly the same as formula (1)—only use Taylor expansions for
functions of two variables and a little more messy computation than before.
Hence the details of proof is left to the reader.
APPENDIX IV
Some Important Martingale
Theorems
Proof E(XnIl=
k,T) = E{ E[XnIlT=k]]}
= E[I[T=k]E(Xn \ j r ) ] (<)
= E[I[T=k]Xk].
= Zo E[XkI[T=kl] + E[XnI[T>n]]
= £(*„)•
For a martingale, EXn = EX0, which completes the proof of (A.IV.2) in this
case.
For a supermartingale we have to prove ) > E(XTa„). We first assume
£ | Xn| < oo, for all n >1.
n
Define X* = 2 [Xk - E( Xk \ ^ _ i)} .T h e n {X*yJ% n > 0} is a martingale
with X0 = 0. Then by (A.2)
TAn
0 = E[X*TAn] = E £ { X k ~ E ( X k|^_,)}
TAn
>E Z1 {x, - x,.,}
k-
= £[X Ta„] - E ( X 0)
The general case can be obtained by truncation considering the new super-
. , * f X„ if X„ < c
martingale Xn = < , where c is a constant.
[ c i f Xn >c
E [| W |] > E[ WI (T<n)] = i E [ \ W \ \ T [T = j]
oo
= [E\ W\ ].
Appendix IV: Some Important Martingale Theorems 301
Like discrete parameter case we obtain the optional sampling theorem for
continuous parameter martingale.
Note: Martingale Convergence theorem A.IV.3 is valid under much less restrictive
condition sup E | Xn | < «> (see J.L. Doob).
Solution to Exercises
Chapter 1
cu f i r u
Hence Fx(u) = I 1 • dv + I —dv = ~ 1 - log *j , 0 < u < 4
Jo Ju/4 V 4 L 4-
36
Exercise 1.2 P(X P(X = f c | w < A < 4 + du) P(u < A < u + dw)
Jo
= f e- ^
Jo * ! V/t
4 = u " - 'e -udV un =V F T T = (1/2)*+".
jc!
pp y (ui - ^P y - y =£ ± f-£ _ V
y! U -p j (*-y)!
\y
= £-r- t —
>! U
f y 2-P)x=o a -
= i— (Ap)y,y = 0 , 1 , 2 , . . .
( n
Exercise 1.4 The joint Ch.f. of Xtr . .. Xtn = E exp / ?
7=1 7
( n \ f
Exercise 1.5 Let /0 < < . . . < * „ < fn+, e T, the index set and let A be an
arbitrary subset of the real line. Let {an} denote the set consisting of the single
point an
= E[X,n+1 e A \ X tll= a H]
but E [0 -“ ( n |
p+k- r ( Ap
E[ e ~sx(T
)] = £ (iT—f ° ^ Pp (1 - P ) k =
k=o yA + s j ^ Ap + s
(by (1)) ( 2)
So fJo e~udF(t)=
H + X[ e ula- 1]
(by (2))
P
pe -u/a
where 0 < p = -j- < 1.
1 - qe -u/a
= f" f ( x ) d x + |f - f ( x ) d x
Jo Jv
0 if y < 0
Hence fr(y) = J ^f-dx ify > 0 ( 1)
P(X - Y < u) = f
J U
P [ X - Y < u \ x =x]dx
= f J-u xf ( x ) dx.
if u < 0
Similarly fx-M) = fix) (2 )
f dx if u > 0
dy (from (1))
■f^fr e"ydy dx
e',x- 1
= if f ( x ) dx (by Fubini Theorem) (4)
" Jo
Solution to Exercises 307
fix)
So it (fry (t) + i(j)y(t) - fJo
Jo
[xeltx] dx (differenting the expression in (4)).
Exercise 1.10 EY(t) = EZ(t) = 0, and since sin2 t + cos2 t = 1 for all t e T,
E Y 2(t) = E Z 2{t) = 1. Since Y{t) and Z{t) have a joint normal distribution, it is
enough to show that E{Y(t)Z{t)) = 0. Y{t) and Z(t) are independent normal having
zero mean and variance unity for all t E T. Therefore X{t) has same distribution
as X(t + h) for any h. Here the bivariate process X(f) = (Y(t), Z(t)) is strictly
stationary.
Exercise 1.11 EX(t) = 0, Cov (X(t), X(s)) = E(X(t)X(s))
= E(A2) cos Xt cos (As) + E(B2) sin (Xt) sin (As)
+ E(AB) cos Xt sin (A^) + E(AB) cos (As) sin (Xt)
= <j2 (cos (Xt) cos (Xs) + sin (Xt) sin (As)) = cr2 cos X(t - s)
Obviously (X(r)} is a covariance stationary process.
In
E{X(t + j)X(j)} = ^ Jq sin co(t + s) sin cos dco
_ j_ c2n
[cos cot - cos co(t + 2s)]dco
Jo
In In
1 1 . 1
y- - sin cot sin co(t + 25)
4n t t +2s
= 0 i f * , j = 1, 2, 3, . . .
308 Introduction to Stochastic Process
IT if n = 0
E(Xn) = £(cos nU) = \
r
E(XnXn+v) = I cos nu cos (n + v) udu
* -n
r
= [cos (2n + v) u + cos uu] du
Jo
= 0 if n > 1.
Therefore, {Xn} is covariance stationary.
Similarly calculate the third moment to determine
E(XnX n+v Xn+v+[) depends on n.
Hence {XJ is not strictly stationary.
Exercise 1.15 Now Y(t) = Yn for (n - 1)T < t < nT
Where Yn)s are i.i.d. with P{Yn= ± 1) = 1/2
Since U and Y(t) are independent.
E(Y(t - U ) \ U = u) = E(Y(t- u)) = 0.
Consequently, m(t) = EX(t) = E[Y(t - u) | U - u] = 0
Therefore Cov (X(s), X(/)) = E(X(^) X(t)).
The time axis divided into successive intervals, it follows from the independence
assumption on the r.v.s Yn that Cov (X(j), X(t)) = 0 if | t - s | > T. Let | t - s | >
T. Since U is uniform in (O, 7), it follows that E(X(s), X(0) = 0 and consequently
E(X(s)X(t)) = E(E[X(s)X(t) | u]] = 0. Now first \ t - s \ < T and 5 = nT. Then t and
s will lie in the same interval if U + [t - s | < T that i s U < r - | r - 5 | and this
event occurs with probability (T - | t - s | )/T. Therefore
f N\
P(Xt -
k
To find H(t), partition (0, t] into n subintervals of equal length A* = tin. By
assumption, the average number of germination in (0, At] = N(l - p)XAt. So the
number of nongerminating seeds at time Aris N{\ - p) (1 - AAt). Thus the number
of nondegerminating seeds at time t is approximately N{\ - p)( 1 - AAO”. Hence
for sufficiently large rc,
P(T< t) - {1 - (l-AAt)"l(l - P) = { l - d - A f / n r i d - p )
By letting At —> 0, we obtain
' N^
H(t) =(1 - p)(l - e~and [(1 - p)(l - e~Xl)]k
'/
,-A/\ 1n-k
[i-d -rtd -n i
Chapter 2
Exercise 2.1 From the directed graph we see that the state space decomposes
into two classes C\ - {1, 2, 3} and C2 = {4, 5}. Also for i e Cj and j E C2 we
see that i -/> j or j -A i. Therefore C\ and C2 are closed sets of states.
310 Introduction to Stochastic Process
1.0 ( 2)
Exercise 2.2 State space of the M.C. is given by S = {0, 1, 2 , . . . N}. *„+1 = 0
if Xn = 0, Xn+l = min {N, Xn + Jn+X- 1} if Xn > 0, where Jn+l = number of new
jobs arriving during nth service period. Then the transition matrix in its cannonical
form is given by
1 0 0 0 0
Po Pi Pi Pn-i Pn
0 Po Pi PN-2 Pn-i
0 0 0 Po P\
f N "
P(Xk = j ) = Pj = p j+'( 1 - P ) N-J~'.
J + 1y
Obviously the conditional distribution of Sn+X given S2, . . ., Sn is a Markov
chain with states space {0, 1, . . . , a, “> a'} and the transition matrix is given
by
1 0 0 0 0 0 0
P- 1 Po Pi • Pi 0 0 0
P=
0 0 0 0 P-i Po 1 -Pi- 1
0 0 0 . 0 0 0 1
States “ 0” and “ > are absorbing and other states are transient.
Exercise 2.4 Let Xn be the number of customers present at time n. Since the
Solution to Exercises 311
P(X„+i = j I Xn - i) = P(^n = j + l - i )
= Pj-M i f f - 1 < y '< /M - 1
For 1 < j < m, Pij = P(x„> m + 1 - y) = Pm+H ify = m
0 otherwise
0 ' 0 1 2 m- 1 m
1 Po P\ Pi Pm-\ Pm
2 Po P\ Pi ••• Pm~\ Pm
0 Po P\ Pm-1 Pm-\
m- 1 0 0 P\ Pi
m 0 0 Po P\
are calculated like Exercise 2.4 (Queueing). Hence the transition matrix is given
by
312 Introduction to Stochastic Process
0 1 2 m- 1 m
0 > 0 + Pi P2 P3 Pm-1 pm
1 Po Pi Pi Pm-2 Pi 1
P= : 0 Po Pi Pm-3 Pm-2
m- 1 0 0 0 Pi Pi
m 0 0 0 Po Pi
Exercise 2.6
= i’n-1 1 ^ = 1 A^n+1 = 'n+1' • • .)
“ ifi+\ ’ Xn+2~ in+2» ••• |^/2—*n» Xn-l = ln-OP(Xn —im Xn-\ —ifi—\)
P(Xn+\ — Xn+2 = in+2»•••> I Xn —in)P(Xn —in)
= = V i | Xn = /„) [since PCA^j = /n_b Xn+2 = /„+2, • • . |
Xn = Xn_x = i„_!) = P(X„+1 = /„+!, Xn+2 = Jw+2, • • • I Xn = /„)].
Exercise 2.7
fcxl
f n r n
i i
pkxk ,&xl
= A= 1
V 1 V
=2 \ I=> |A I 2
ju |m7 I < 2 |m, I => |A I < 1.
2=1 7 =1 2=1
Exercise 2.8
Clearly X0 = r,X n = 4 +1 + X„+1, n > 0 . . . (1)
The probability that among i susceptables k of them (k < i) have contacts with
infectious individuals equals p kq l k (q = 1 - p). Hence
Solution to Exercises 313
. p ifi>j
P(L+i = i - j \ X „ i) = i w ;
0 if i < j
0 1 2 0 k
1 0 0 0
p q 0 0 0
P2 2 pq 42 0 0
( r\
pr r pqrX p r 2q 2 .. qr
v2.
The duration of the epidemic is the smallest number r such that ^r - 0 i.e. the
moment t when there are no further infectives i.e. the smallest number T such
that Xr_i = Xr using the equality X„ = £n+1 + Xn+l. Hence the number of cases in
the epidemic is §) + X0 - XT= §) + r - Xr
ft m = \
lim — X P(2)
»oo nk m=2
Since lim Pnk+l/nk = 0 = lim P/nk (elements of powers of Pare bounded by 1).
k—
By (1) and (2) A = AP = PA (3)
Exercise 2.12 Suppose that C is recurrent but is not closed, then there exists
i e C such that £ p tJ < 1 and hence there exists k £ C such that pik > 0, i.e.
i —>k. State i being recurrent we have i <-> k, hence k e C. This is a contradiction.
Hence C is closed. Conversely, suppose that C be a closed class of states. Since
the state space of a finite M.C. contains at least one recurrent state, the sub-
Markov chain with state space C has atleast one recurrent state. But recurrence
is a class property, hence C is a recurrent class.
= 2 = = [/„]'"
*>1
(b) Proof is entirely similar to that of (a) using Ti;- instead of T{ when
Tij = min {n > 1; Xn = j given that X0 = i }.
(c) Let A = {The Markov chain returns to i infinitely often}
= n {The Markov chain returns to i at least r times}
r= l
B y (a )
f l if/,, = 1
E x e r c is e 2 .1 4 (a ) F ir s t p r o v e th a t ( b y in d u c t io n o n n)
P I[ v ( j ) > n ] = f iJ[fj j 1r (1 )
- Pi[v(j)>n]~P,[v+ 1]
- fi jU ij V A 1 -/,./]• (2)
= ^ r r r L = 1 > f / y < 1 a n d A , = 1.
1 Jij
(c ) O b v io u s fro m (b ).
E x e r c is e 2 .1 5
f o r a ll ra' < m. T h e la s t i n e q u a l it i e s h o ld b e c a u s e
1+ t p ™ * 2 P)?~k) = 1 + 2 "
n—\
f m-m'+k \
An-k)
and 2 (/<*>) 2 > 2
*=1 n=£ JJ &=1 (^ } £ ^
Solution to Exercises 317
f m -m ' \ ( m’ \
1+ L
n= k
V
m
Dividing both sides of (1) by 1 + Z p-n) and letting first m —> °° and then
n=1
m —> «> yields the result.
The last part is trivial and is omitted.
Exercise 2.16
Pa = for i = 0, 1,. . ., k
k+ 1
i + 1 •r • ■
P" = I 7 T ,fy<1
= 0 if j > i
From the relation Xn = Xn+l + (Yn - Xn_\)+it is easy to see that Xn is a Markov
chain. Since Pkk- U k is an absorbing state.
From the transition matrix
1 1 1 1 3
k+ 1 k+ 1 k+ 1 ’ ' k+1
0 2 1 1
k+1 k+ 1 " k+1
0 0 3 1
k+ 1 " ' k1
0 0
0 0 1 ,
It is clear that j —>k but k -/> j for j < k and the chain is finite. Hence all the states
0, 1, 1 are transient and k is positive recurrent.
Sincepu > 0 for all i = 0, 1 ,2 , . . . , k all the states {0, 1, 2 , . . . , k] have period
one though the chain is not aperiodic (since not irreducible).
Exercise 2.18 The states space of the M.C. is {1, 2, 3, 4, 5, 6}. The transition
matrix is
1 ( 6l 1 1 1 1 n
62 6 6 6 6
2 0 1 1 1 1
6 6 6 6 6
3 0 0 2 1 26 26
6 6
4 0 0 0 4 2 26
6 6
5 0 0 0 0 5 2
6 6
6 0 0 0 0 b
We obtain, for example, the third row of the matrix as follows. Suppose the
system is in state 3, i.e. the maximum of the numbers occurring on the nth toss
is 3. Then the system remains in state 3 if a 1,2 or 3 occurs on the (n + l)th toss;
hence p 33 = - On the other hand, the system moves to state 4,5 or 6 respectively,
if a 4, 5 or 6 occurs on the (n + 1) the toss; hence /?34 = p35 = p36 = ^ The system
can never move to 1 or 2 since a 3 has occurred on one of the tosses; hence
p 31 = p32 = 0. The other rows are obtained similarly. The matrix is not regular
since state 6 is absorbing and there is a 1 on the main diagonal in row 6.
Exercise 2.19 (See Exercise 2.26)
Let Yn be the content of the dam at time n and Xn is the input at the nth period.
Then
yn
+ 1= min (*. y
n+-1
Hence, clearly {Yn} is a Markov chain. The transition matrix is^
- 1 0 0 q P
This is a finite irreducible chain and the chain is positive recurrent. Hence a
stationary distribution exist.
The stationary distribution is given by
(7Tj, 7thy . . .
» n N t)
' 0 0 0 0 0
0 1 1
0
1 !
0 0 0 0
2 2
0 0 0 0 0
I I i
X
3 3 3
1 1 1 1
0
M- 1 M- 1 M- 1 M- 1
1 _1_
I m M )
' K\ '
n2
1= jj, n M. x= Kn2 =
distribution is given by
Exercise 2.21
f 0 1 0
p = 1/2 0 1/2
v0 1 0
1/2 0 1/2
0 1 0
1/2 0 1/2
In general P2n= P2, P2n+l = P, so that p i2n) > 0 , p (2n+l) = 0 for each
states are periodic with period 2.
We find that f l\l) = O ,//^ = 1 so that f u = S f^\n) = 1, i.e. the state 1 is
recurrent. Since the chain is irreducible, the other states 0 and 2 are also recurrent.
Being a finite chain all the states are positive recurrent.
Exercise 2.22 Here {Xn} is a homogeneous two-state Markov chain with
p ^
transition matrix P = ( Q . It can be shown by induction or otherwise
VP <7 )
that
' 1/2 12
/ '
and lim P m =
m—>°° 1/2 1/2 ’
and P( Xm = 0 , X 0= 1) = (1 -« )P ,(0m)-
By Bayes Theorem
P(xo =o I Xm =0)
« /4 m)
(m)
“Pm ) + (! ~ a)P 10
a[1/2 + 1 /2 (9 - p ) m}_____________
_
a{ 1/2 + 1/2(9 - p )m
} + (1 - a){ 1/2 - 1/2(9 - P)m >
a {l + (9 - p ) " }
1 + ( 2 a - l ) ( p - q ) m'
Solution to Exercises 321
Exercise 2.23 Let /v(z), z e S, denote the indicator function of the set {y },
defined by
N( y ) = 2 / , (
n= 1 y
Now P(N(y) > 1 | X0 = jc) = pxy. Let Ty be the first passage time to the state y.
By Markov property P(Xn+l = xn+u,Xn+m = xn+m =
= pXn Xn+l , . . . p Xn+m_\ ,xn+m- Hence the probability that a Markov chain starting at
x first visity at time m and next visit yn units of time later is
P(Ty = m \ X 0 = x) P(Ty= n \ X 0 = y).
= 2 P(Ty =m
| X0 +
m=1 7 n=\ J ' •'
Now E ( I , ( Xn) \ X 0 =x ) = P( Xn = y \ X 0 =x ) =.
= J , mPxyPyV 0 - Pyy )
1
Z mt n
m=1 (1 - o 2'
we conclude that
Px
G( x, y) = < oo if 0 < pyy < 1, i.e. y is a transient state.
1 -Pv
Hence by (3)
Hence the Markov chain must have at least one recurrent state.
Exercise 2.24 By Chapman-Kolmogorov equation
.(n+m) _ y (m) (n)
jj j t s ? Jj P j j
I -""I ( 1)
- p ' - ’r ) ? *p'"j' h . p»
-= Pn(W>J n) p\T = i
JJ JJ + Pj j ( l ■p(r
J£S
+ d -p\V)-
Exercise 2.25
Parent aa x aa aa x Aa Aa x Aa AA x aa
mating AA x AA AA x Aa
Progeny
mating
aax aa
AAxAA 1 ■
1/4 1/8 0
aa x Aa)
0 1/2 1/2 0
AA x Aa]
Aa x Aa 0 1/4 1/4 1
AA x aa 0 0 1/8 0
Solution to Exercises 323
0 1 2 k- m- 1 k- m
0 Gm Sm+ 1 8m+2 •* 8k- 1 K
1 Gm-i 8m Sm+ 1 •• 8k-2 hk~\
P= m G0 81 82 8k-m -\ hk-m
m+ 1 0 &o 8 1 • 8k-m -2 h k -m -1
k- m 0 0 0 8 m -\ hm
324 Introduction to Stochastic Process
II
0 8o 8k-3 hk-2
k- 1 0 0 So hx
where ht = S g ,. Let us assume that gf > 0 for all j, the chain is then irreducible
j= i J
and aperiodic, so that the stationary distribution n t{i = 0, 1, 1) exists
where n l = lim P(Zn = i). The /r, is the unique solution of the equations
n—>°o
k-l
n j = 2 7T, Pij (j = 0, 1, 1) together with
' i-o J
^0 + + • • • + ^ -1 = I-
In case of geometric input = j) = gp7 = gr 0 < q < 1, p = 1 - qj =
0, 1, 2, . . . when m = 1, the transition matrix is given by
0 1 2 k-2 * -1
0 q + qp CIP2 qp3 ■■ w k-' Pk
1 q qp qp2. qpk~2 Pk- X
P=2 0 q qp ■• <7P*-3 Pk-2
k- 1 0 0 9 p
Exercise 2.27 Loss suffered in the first year is Y • /?/100 = Y{ • Ux. Thus X x =
Ki - YxU{ = Y\( 1 - f/j). Loss suffered in the second year is K2(l - Ui )R2I 100 =
K2(l - UY)U2. Therefore X2 = K2(l - Ux) - Y2( 1 - UY)U2 = Yx( 1 - Ux) (1 - U2).
n
Hence by induction on n, the crop yield in the nth year is Xn = Yn n Zh Z, =
i=i
n
1 - Ui and Xn = YnLn. Since Ln = n Z, = Ln_xZn, Ln is obviously a Markov chain.
/=i
Exercise 2.28 Let pjk = P{/: white balls in first urn after n + 1 interchanges | j
white balls in first urn after n interchanges}.
Note that if there are j white balls in the first urn after n interchanges then
except for j - I J , J + 1 states other states are impossible after n + 1 interchanges.
Therefore pjj_x = P (Replacing a white ball in the first urn by a black ball in
the 2nd urn) = j/N •jIN = j 2/N2.
Solution to Exercises 325
N~j
=j / N -
N N
Pjj+l = P (Replacing a black ball in the first urn by a white ball in the 2nd urn)
_ N -j N -j
N N { N )
if k = j - \, j =
Pjl > 0 for all /, j e S so that there are no transient states, the chain IS
i
irreducible and aperiodic finite chain.
N N
Ttj can be solved by solving the equations £ itiPij - Kj and £q 7tj = 1.
(Nl)4
The solution is given by n } = ---- r-------------=--------.
J U!)2((N -y )!)2(2
Exercise 2.29
(if N)q if k = i + 1
4r p +^ -i-q if k = i,i= 1 ,2 , 1
(a) Pik = N - i p if = i - 1
N
0 otherwise
i2/ N2+ ( N - i ) 2/ N 2
if
i(N -- ii) .,CI . . . . , . , ^ A,
(b) ------t— if k = i + 1, i - 1 i = 1, 2 ,.. . N
N2
0 otherwise
Exercise 2.30 In fact we shall show that a 2 x 2 stochastic matrix is the two-
step transition matrix of a Markov chain iff the sum of its diagonal terms is
greater than or equal to 1.
1- a 1- a
Let P = P and A be two stochastic
\-q <7 1-/3
matrices such that P = A2. We shall prove that p + q > 1.
a 2 + (1 - a)(l
(1 - a ) ( a + =1 • • ■( 2)
if p + q > 1.
( 1 - P ) ( a + P ) = 1 - <7 • • • (3)
p 2 + (I - a)(l - P) = q . .. (4)
(2 - p q- ) a = 1- q± - q ) 2 + (2 - - + p - l - p 2 )
= 1 - q ±(1 - p) - 1
Since p + q - 1 > 0, this solution is real. Choose the positive root, then
1- q (1 + <? - 1
> 0.
2 -P~q 2-p-q
1-p + (1 - + <7 - 1
Similarly >0.
2-p-q + 2-p-q
2nd proof Let P and A be as above. Now if then the eigen values of P
and A2 coincide. Since A and P are stochastic, 1 is an eigen value of P and A. The
other eigen value of P is p + q - 1 and that of A is a + 0 - 1 . So p + q - l =
(a + 0 - l)2 > 0. Now since 1 > 0 and p + q - I > 0, P = A2 has a solution
A = ^fP (all eigen values are non-negative).
Exercise 2.31 1st Solution Since the M.C. is finite and irreducible, it is positive
recurrent. Since it is aperiodic p ^ —» 7Tj > 0 as n —> ©o. Hence there exists a
N(i, j) such that > 0 for all n > N (/, j). Since the M.C. is finite,
N = max N(i •j) < ©o.
iJ
Solution to Exercises 327
Second Solution If p-"0 > 0, then p{m+n(i(7)) > q for all sufficiently large
positive integer n and where d(j) is the period of state j. Now same type of
arguments as the first solution applies.
OO OO fl oo oo
Exercise 2.33 Since P is doubly stochastic 1 = Z p~n) for all n > 1 (by
induction). Fix A > 0.
oo yv
Then 1 = Z p,(.n) > Z p.(.n) > N min p \ S i n c e chain is irreducible,
i= 0riJ 1=0r i J 0 <i<Nr i J
0 < lim [ min p -^] does not depend on j and is < for all a. Take N —»
0</</V 7 A
then lim p- w) = 0 exists for all i j E S and hence the chain is either transient or
rt—
>oo •'
null recurrent.
F y(z)d-z)
= lim —J---- „ . .
h i- 1 - (z)
and this limit exists always (may be zero) where F,y(z) is the generating function
(n ) ]
of {/,.
(b) By Fatau’s lemma
( C ) z /r
k
ikP kj =Z
k
lim - V (m)
lim
n—
2_
n m= 1 k
Exercise 2.35 Let 0, 1, 2, . . . , A are the states of the M.C. and the M.C. is
irreducible. Therefore the chain is positive recurrent and hence has a stationary
=I P[N, = r]f<n) = (1 - f u K f „ y
n n
= ( 1 - f a ) fa fa = 0 - fu ) f a +'( "
r(«)
< » + »/-;■ *" = /« * , 5. < ’ = / « - ? » kj
Solution to Exercises 329
f ‘J = L
Second Solution Given a statey, the mean first passage times {jUy, i * j ] can
be regarded as mean absorption times in a new Markov chain obtained from the
original Markov chain by makingy an absorbing chain. If P = (P,-,) is the transition
matrix of the original M.C., define a new transition matrix P ' = (p ki) as follows:
p'ki = 1 if k = y, i = y, = 0 if k = y, i *y, p'ki = p*/ if * * y \«is any state. The
resulting chain contains a single absorbing statey, while all other states are non
recurrent since they are inessential. The behaviour of the new chain before
absorption is the same as behaviour of the original chain before visiting state y
for the first time. In particular, the mean first passage time from i to y in the
original chain is the same as the mean time to absorption.
Exercise 2.38
n n
< f jk for all n > 0.
Therefore to
f i n) _ An) y r i m) (n-m)
Jjk -Pjk- mt 0 Pkk
Hence /* * !y p c)
Combining (1) and (2) we get (a).
(b) If j —> k, then p ^ > 0 for some n. Hence by (1)
Alternately, if f jk > 0, then by (2) X p ^ > 0 => for some n, p ^ > 0. Soj —» k.
(c) Similarly, k —>y iff > 0. Soy <->k iff fjk f kj > 0.
s.
m=l p r + i
330 Introduction to Stochastic Process
Exercise 2.39
= 2 a t = 1.
jeS
(b) By dominated convergence theorem
= 7i j Z a , = n j ,
1 ieS 1 J
Hence
Therefore the n-step unconditional probabilities are the same for all n > 1 and
the M.C. is “Stationary” i.e. the joint distribution of {Xt[+h , Xt2+h ,. . ., Xtn+h }
for positive integers tx < t2 . . . < tn is independent of h, if h is a positive integer.
Now p00 = e~x > 0, p0i = e~xk > 0 if 0 < p, q < \ and 0 < A < °°
Similarly every other pjk > 0 for all j , k e S.
Hence, the M.C. is irreducible.
Since p ]k > 0, p {jk > 0,. . . > 0 for all n > 1. So the M.C. is aperiodic.
Since 0 < ^ | xk | < °o, the M.C. is positive recurrent. Moreover xk > 0 and
k
„-x 2 p nqj-nA k-n /(k - n
* (z ) = *?o 7=0 n=0
_ e - M \ - z ) ( \ + p + p 2 + ...+ p n ) g \ \ + p n (Z - 1)1.
Therefore, Xj = e~Xlq(X/q)j/j\
is a solution of the given system which is accidently also a stationary distribution.
Hence the M.C. is positive recurrent.
Exercise 2.41
where
Poo(*t) = [1 - G C tf)]" 1 ( 2)
1 - G(t)
and Z qmt m =
m=0v
Z0 1 - n=0
2 00
tm =
1- r
(3)
= ' m-0
\ ~ m=\
2 + 1.
= z z , p[ s m=j ] ( x t y
m = 0 7=0
= z z (xtyp[sm =n
w=0 7=1
oo n 1 _ f2(t\
Therefore, Z f" Z ^ /’[ f „ = i l = — ... y * . . .
«=o 7=0 " ^ (1 - r)(l - G(-tt))
where A = ^ Po = 1-
Summing over (2) gives a x =(pj + . . . + pN_x) (1 -
Solution to Exercises 333
AM
f N- 1 \ f AM \ -i 2 pi - 1
7=0
or «i = Oi
/—I N- 1
I'-1 / V 7
.2 A
7=0
f am
z pj 1-1
7=1
From (2) OCi - 1 = TV-1
- 1 2 pj
7=0 ^
2 P,7
j=0
7=1
2o a
(X; = 1
TV-1 (v p 0 = 1)
2 P7
7=0 ^
AM /A M
Putting ( = k , a k = Z Pj / S pi
J=k J / 7=0
E [N„m i
-E Z Z„ —> 1 1
n m=1 n Vu Vi
Second Solution Define a renewal process as follows: Let {Wn, n > 1} be the
inter recurrence times to state i. By Markov property {Wn, n > 1} are i.i.d. non
negative r.v.s with distribution P[Wn = k\ = / j k). Now since i is positive recurrent,
Let A^l) be the renewal counting function for state /, that is i.e. A^(i') =
max [m : WO + . . . + Wm < n]. Then by discrete renewal Theorem
E( Nn(i)) _ j _
lim
n—>°o Vi’
Exercise 2.44 First Solution Assume that the M.C. is transient and put
p*j=T^p(j \ Since j is transient, p* < °o.
334 Introduction to Stochastic Process
By Problem 2.38
l-/oo > 0 since the M.C. is irreducible.
2. fko ^ Pko by 2.38 (a) for k*.0
3- Pko =fkoPoo by 2.38 (e) and 2.38 (d), and = > °-
Let k * 0. Then 0 < cc<fk0 < p*k0 (by 2)
Since p^0 = ----- — (by 3) > 0 and / 00 < 1 by assumption, we can choose
1 ~/oo
n so large that [/oo ]” < •
Zj kOPoO
Hence 0 < a < f k0 [fooT p oo < ol!2 which is a contradiction. Therefore, the
M.C. is recurrent.
Second Solution
l2io
Qoo = —f~
> 0 since Qi0 > 0 and f i0 > 0. (3)
JiO
Since Qoo is either 0 or 1, by (3) Q00 = 1 => 0 is recurrent and the chain being
irreducible, it is recurrent.
Exercise 2.45 (a) Assume that j is transient. Let S* be the Communicating class
of j. If there is no k such that j -/» k and k +j then all states of S are in S* so that
S is positive recurrent (since the chain is finite). Since j is transient, there will be
a k such that j -A k and k + j.
Solution to Exercises 335
Conversely if j —> k and k —>j, then > 0 for some n and p ^ = 0 for
all m.
Hence 1 - fn = P(Xm * j for all m > 0 | X0 = j) > > 0 for some n and some
k * j. Hence fy < 1, i.t.j is transient.
Exercise 2.46
Solution (a) If j e S is absorbing, then pjj = 1 and pjt = 0 for all i * j. Since S
is denumerable, record S such that the absorbing states are numbered 1 , 2 , . . . .
The remaining states are supposed to be transient. Clearly
E Ec
(I 0^| E
P=
Q )E C’
1 if X* =7
Then let Zk
0 if Xk * j
An)
(c) = X f \ p l) X X,c Pik Pkj ( 1)
o=0 o=0
An)
= PiP + \ kj• Hence
iP == X pik
keE' keE1
(/ 0 '
(d) =>
C, vfl(/ + Q + . . . + <2n-1) GB,
f / 0" f7 0N
—> =
kRN oy 0,
since RN = # and Q* —» 0 (since j is transient if j G £ c ).
336 Introduction to Stochastic Process
=L PikPkj = L PkiPjk
Hence, P{2) is symmetric. Similarly, by induction on n P (n) is symmetric for all
n > 1.
Now 2 pa = 2 p n = 1 so that P is douldy stochastic. If the M.C. is irreducible
kGS j GS
and infinite then it is not positive (by Ex. 2.33). Since ptj > 0 => pJt = ptj > 0. So
the M.C. consists of closed classes only. Every closed class can be treated as a
separate M.C. So if every closed class is infinite then by previous argument the
original M.C. can not be positive. But if the M.C. consists of finite number of
states only then there will be no transient state, not all states are null recurrent
and therefore there will be some positive states. But if the M.C. is irreducible
and has finite number of states then it will be positive recurrent and same thing
is true for every commucating class.
Exercise 2.48 Let the initial distribution be {nh i e 5}, then by Ex. 2.13.
(1) 71j = Z 7 1 , for all n> 1, (2) p\n) = P( Xn = = 7th
ieS J 1 J
Hence
Hence /?/z is a transition matrix and since the initial distribution is given, /?(m)
is the m-step transition matrix of a discrete time Markov Chain.
Note Let N be any large integer. Define Yn = XN_n for n = 0, 1, 2, . . . , N then
{Yn, n = 0, 1,. . . , /V} is a Markov chain with transition matrix R. It is called the
reverse or dual chain of {X„}.
(b) Since the M.C. is irreducible, so is its dual. Moreover
Thus the chain is positive and the stationary distribution is given by the initial
distribution {7r„ i e S}.
Solution to Exercises 337
Exercise 2.49
Pn = Pn-i(0 - c)p + c) + qn_j((l - c)p), where </„_! = 1 - p n_x
= <7Vi + 0 - <?)P-
Therefore /?„ = Ac""1/?! + ^ = Ac""1/? + p , n > 1 (since /?! = p).
From initial condition p = Ap + p so that A = 0. Therefore
pn = p = P(X0 = 1) for all n > 1.
Therefore E(Xn) = p, Var (Xn) = p - p2 = p(\ - p).
E(Xn_\Xn)= 1.1 ((1 - c)p + c)p = ((1 - c)p + c)p = +cp
and Cov (Xn- 1, Xn) = 2(p - 2 + cp) cp
Similarly Cov (X„_2, X„) = c2p (1
and in general
Cov (X„_*, Xn)=ckp(\ - Pf
or n
= p d - p ) T ^ \ ( n - l ) - c-
n
Hence, Var(S„) = 2 Var(X*) + 2 2 Cov(X,,X*)
k =1 j< k
cjn - 1)
= np(l- p) + 2p(l - p) - c 2( 1 - c"-')/(l - c)
1- c
Exercise 2.50
i n
First Solution IfP = />2then p (jln) = /?,, for all n > 1. Hence /?,, = — Z p (hm) —> tt,
z/ 7 7 n m=o lJ7
1 n
and pij = — Z^/?^”0 —» ;r7 (since the chain is irreducible). So p{j = pjj = Ky
Assume p0o = 0, then pi0 = 0 for all * e S and hence {0} can not be reached,
so the M.C. is not irreducible.
Hence p0o > 0 and {0} is aperiodic. Since the chain is irreducible, all states
are aperiodic.
Second Solution If P2 = P, the p^n) = p i: for all i e S and hence lim pjfn) = pjj.
n—>°°
Assume j as transient or null recurrent then pl} - 0 for all i and j and so P is not
a stochastic matrix. Hence j is positive and the chain being irreducible, the M.C.
is positive recurrent. Hence p \ p —> n } and p {^ > 0. So the M.C. is aperiodic
and p l} = Pjj.
338 Introduction to Stochastic Process
Chapter 3
Exercise 3.1
K \(n ) = P7h(n
- 1 ) + q 7 l0 (n - 1)5,,i (1
TC^n) = <5„0, na{n) = 0 where <5„„ i= 1,0 are Kronecker’s deltas. Multiplying both
sides of the above equations and summing over all values of n we obtain
Since (2) is a linear difference equation with one variable k, taking Gk(s) = k k(s)
as a trial solution we find from (2)
A* = PsAk+l + ^A*"1
or /xvA2 - A -i- gs = 0. The roots of the quadratic equation are
0 1 f
Since 1 - 4pqs~ > 0 or s < — ==■, we assume here 0 < 5 < — ■== for two real
2 ^p q 2j p q
roots A](5 ) and A2(s).
The general solution is given by
A,""*( 5 ) - A?"* ( 5 )
Gk (s) —( q / p Y (5)
A." (s) - A? (s)
( a —k\
z [0(^)]r {l ~ ( - l ) r }
r=0
= (q/p)k (2ps)k
z
r=0
[0(S) Y { i - ( - [ ) ' }
V' y
( a - k - 1)/2 ' a-
S [0U )]2;
1=0
Gk(s) = (c,/p)k(2ps)k if a - and a are odd.
( a - 1)/2 f a
2 [<K*)]2'
2j + 1,
(n -k-2)/2 ra-
2 W * )]2'
7=0 V27 + ly if a - k and a are even.
(2/«)* (a-2)/2 ( a N
2 [«>(5)]2;
2 7 + l.
G(5)
Now G* (s) = where G(s) and VG) are polynomials in 5 with degree of
V(J)
G(s) does not exceed that of V(s) by more than one. So we assume the degree of
U(s) is less than the degree of V(s) by m (say). Let the equation V(s) = 0 has m
distinct roots sh s2, . . . , sfn. Write Gk(s) in partial fraction as
Pm
Gk (s) - P\/(s\ - s) + + ...+
v^2 “ s ) (Sm ~ s )
with coefficients pk are given by
(«) = -^7■
/J+l + . . . + P'
n+1 (7)
7 = 0, 1------ ,a.
2( pq) U2 cos | — j
sin(a - k)6 .
So Pi = lim ( 2 — / a T (s i ~ 5)
sin (a0) J
n k(n) = 2n p i n - M q W 2
f
•'o
cos'7 sin (nx) sin ( k nx ) dx
p ( n - \ - k ) / 2 q ( n + k )/2
k/n
(n - k)/2
Exercise 3.2 If the particle starts at a, absorption occurs at time 0 with probability
1, and if it starts at any other point, absorption cannot occur at time 0. So
1 if 7 = a
( 1)
0 if j * a
Since the random walk takes positive jump or negative or no jump at all at the
first step
( j = - b +1 ,___
The boundary conditions are given by
= 1 a n d /';", = 0, 1, 2, . . . (3)
(2) is a difference equation of the first order in n and second order in j with
initial conditions and boundary conditions (3).
Multiplying equation (2) by sn and summing over n - 1, 2, . . . we obtain
Fj„(s) = s[ pFJ+[a(s) + (1 - p -q)Fjn(s) + qFH a(s)] (4)
with boundary conditions
Faa(s) = 1 and F_M (s) = 0 (5)
(4) is a difference equation in one variable j.
Substituting a trial solution Fja(s) = A'(s) we get from (4)
Solution to Exercises 341
w S 1 , X 1 - 5(1 - p - q)± [1 - j ( l
A , ( s ) , A 2 ( j ) = ------------------------------------------ y ]Ts-------------------------------------------
Assuming j is real and positive and the function under the radical sign is also
positive leads to
______ 1_______
[ l - i ( l - p - <7)12 > 4 or 0 < s <
i - (4 p - V?)2
The general solution of (4) is given by
Fja(s) = A[Xl(s)]J + B[X1(s)]J
where A and B are functions of 5 and are to be determined from the boundary
conditions (5).
[ A, ( j ) ] ^ - [ A 2 ( j ) ] ^
This gives ( 8)
u , ( 5 ) r ft - [A2(5)]fl+i
putting j = 0 in (8)
[*,(*)]* - [ A 2 (J)]*
[ A ^ ^ r " - [ x 2( s ) r +b
Exercise 3.3 Since the jumps are independent, the particle’s position at time n
depends only on its position at time (n - 1) and the nth jump. During the interval
(n - 1, n), the state j can be reached by one of three mutually exclusive and
exhaustive ways, viz. a jump of + 1, 0 o r-1 from states j - 1J J + 1, respectively.
= E ( e e°SN)= 1.
and E(eeSN | 5.v > a . N = n, e0SN-' = pea8) = e"ep E ^eex” \ e xn6 > 1 ) (2)
P)
Solution to E x e rc is es 343
E* < ) if 0> 0 (3 )
N o w to p r o v e th a t W a ld 's id e n t it y c a n b e d i f f e r e n t ia t e d u n d e r th e e x p e c t a t io n
s i g n a n y n u m b e r o f t im e s w it h r e s p e c t to 0 at a n y v a lu e 0 in t h e d o m a in
| 0 ( 0 ) | > 1, it is s u f f ic i e n t to s h o w th e f o l l o w i n g t w o r e s u lt s h o ld .
fo r a ll v a l u e s o f 0 in / a n d E (H (S N, N ) ) < °°.
N o w e x i s t e n c e o f 0 ( 0 ) fo r a ll 0 i m p l i e s t h e e x i s t e n c e o f a ll d e r i v a t i v e o f 0 ( 0 )
( D u e o f a n a ly t ic it y o f 0 ( 0 ) ) . T h is i m p l i e s r e s u lt 1.
the type N ri eeSN [0(0)]“** Result 2 is proved if we can show that any given
integers r, and r2 there exists a function Hnr2{SN, AO > \S% N n ees" [0(0)]"* |
for all 0 e / and E{Hrxn (SN, N)} < <».
where 0O> | 0 | in /. Let 0! > 0O. Then for c > Max (a, b) we have
+ £ ( e - Xw01 | N= (7)
Hence by (7), (3) and (4) we obtain
344 Introduction to Stochastic Process
S in c e E ( N r) < °° fo r a ll in t e g e r s r > 0 ( b y T h e o r e m 3 .1 ) t h e R e s u lt 2 h o ld s .
D i f f e r e n t i a t i n g th e W a ld ’s id e n t it y a t 6 = 0 , t h e d e r iv a t iv e o f e SN° [<j)(0)]~N
at 0 = 0 is g iv e n b y
d' (0)N
Sn ~ J ^ W = Sn ~ E ( X ) N (8)
T a k in g e x p e c t a t io n w e g e t E (SN) = E (N )E ( X ) and h en ce
FS k,
E(N) = ~E{X)
D i f f e r e n t ia t in g W a ld ’s id e n tit y t w i c e u n d e r th e e x p e c t a t io n s ig n at 6- 0 , w e get
E(X) =
S i n c e 0 ( 0 ) = 1, 0 '( 0 ) = 0 a n d 0"(O ) = E X 2, p u tt in g 0 = 0 , (9 ) b e c o m e s
S 2 - N ( I ) " ( 0 ) = S 2n - n e x 2 .
T a k in g e x p e c t a t io n E [ S j j - N E ( X 2)] = 0.
T h e r e fo r e E N = E ( S N2 ) / E ( X 2 ) if EX = 0.
Chapter 4
Exercise 4.1
L e t C i b e t h e fir st t im e a c u s t o m e r m a k e s a p u r c h a s e a n d 0 < S x < S2 < . . .
b e th e arrival p o in ts (r e n e w a l p o in ts ) o f th e c u s t o m e r s . T h e n , b y th e to ta l p r o b a b ility
r u le
Exercise 4.2 F r o m e x e r c is e 4 .6
Differentiating under the integral sign and making change of variable along
with substitution r = x + t - y yields
Exercise 4.3
(a)E(X-x\X>x) = i; [1 - F ( t ) ) d t
1 - F(x)
Exercise 4.4
Fit)
To prove H(t) < ----- =rr-r if F(t) = 1 for t > 0, then the result is obvious. So let
I - t it)
Fit) < 1 for all t > 0.
(since F ( r ) t)
(b) H*H( t ) = f Z
: F(n) (t - u) d Z {F(n) (0} (by Fubini’s theorem)
Jo n n= 1
OO OO /• t oo oo
= E (2 k - 1) F(k){t) = /F (r )
k=1
0 = lim
/->°°
\ ju - f
Jo
xdF( x) = lim f
J/
xdF( x)
/• o
/(*) =J Jo
e~stdF(t) = e~s'|F
(t)o + J
Jo
f e~s,F{t)dt (3)
F (X 2) ju2
2/u
= li — — + -— — = Laplace-Stieltjes transform of
jus jus v J
[H - t/ju + F*] (by 3 (b)).
By uniqueness theorem of Laplace-Stieltjes transform
(1 - F*) * H = H(t) - t/ju + F*(t)
F*(t) = -
F Jo
f (1 - F(jc))djc = - f (1 - F(r - «))rf(r - w) (put jc = r - u)
J/
= tin — — (4)
F** G= — * (1 - F)*G = - L *
n
So F* * H = F* * G + F* * t i n = — * F
V
or (1 - F*) * H = H - F * * H - H - t / n * F = H - t l n + F*(by (4))
lim {//(f) -
; -> ° o jJ. J Q
tin }+ Hm Jh_
iF -
f Q(t)dt =
Vi - 2 / t 2
or Mm [ H i t ) - t i n } = -^V
2 /^ 2 n2
(c) The above relation means that
LLi
H{t) tends to its asymptotic value t/ju + — z 2 ± l = / / / I + C
2 n-
Solution to Exercises 349
Proof of 4(b)
Define A(t) = H(t) + 1 - j i xt
= E[N( t )+l ]-t / j u (1)
Since E(N(t) + 1) < <*>, by Wald’s equation
MO = P l{E[Sm+l ] - t }
= n~'E( Y(t)) where = - t,
the excess life time.
By the renewal argument, conditioning on the time X{ = x o f the first renewal
we shall get a renewal type equation for A(t). Distinguishing two cases
x-t if .r >
we get E[K(f)|X, = * ] =
fiY ( t - x) if
-r
(x - t)dF(x) + ju I A(t - x) dF(x)
Jo
f (2)
Now f
Jt
(x-t)dF(x)= f
Jo
ydF (t + y) =
Jo
f {1 - F ( t + y)}dy
MOO MOO
dF(u) ( 3)
Jo Jt+y
is a decreasing function of t.
Since the integrand is nonnegative, by Fubini theorem interchanging the
orders of integration we get
/•o o I moo | moo moo moo
[1 - F ( t + y ) ] d y \ d t = \dF(u)dydt
Jo [Jo J Jo Jf Jf+y
moo moo f m u-t 1 /•= » /too
= J J |J dy\dF(y)dt= J J (
= [
Jo Jo
f (u - t )dt dF( u) = 1/2
Jo
f l/2(cr2 +(4)
350 Introduction to Stochastic Process
Since (x - t)dF (x) is bounded, the renewal type of equation (2) has an
Jo
unique solution given by
fiA(t)
f ■f
(x - t)dF(x) + jJ. I A (t - x) dH(x), H( x ) = S F (k) (/).
Jo k~ 1
T T 9 9
<7~ + JU- , <7~ - jU^
2 jU2 ~ 2 "^
I n* [ F <n) ( t ) - P(n+1)(r)]
n= l
= I [ ( n - l ) ‘ - n f ] F (,,tl)(l) (1)
n—1
by rearrangement of terms provided we can prove that the above sum is convergent.
If F(x) is degenerate at a point c > 0, then both sides of (1) are clearly finite
for every fixed t. If F(x) is not degenerate, then there is a y > 0 such that 0 < F(y)
< 1. Hence for every m> 1, F (m\my) < 1 because 1 - F^m)(my) = P(Sm > my) >
P(Xj >y , 1 < j < m ) = [\ ~F(y)]m >0-
Furthermore, for any z > 0,
i.e.
F(n)(z) =
r F(k)(z - x ) d F (n-k>( x ) <
Exercise 4.6
/•/+A'
P(t +x >
Sn > r, S„tl > / +x) = J P(Sn+i + | S„= v)
/• t +.V f t+ .X
^ oo ^
[1 - F(t + x -y )]dH (y) Since Z F{n) (y) = H (y) and 0 < t < v
> n=\
Exercise 4.7 Starting with total probability iaw, we first note that P(Z(t) < z,
N(t) = 0) = 0 for z < t.
Hence, P(Y(t) > x, Z(t) <z) = Z P(Y(t) > x, Z(t) < z, N(t) = n)
n=\
= £f
/I = I J
P ( Y ( t ) > x , Z ( t ) < z , N ( t ) = n \ S n =y)dP{S(1)
Note that conditional probabilities are equal to zero if v £ (t - z, r), while for
y e (t - z, t), the meaning of such conditional probabilities is that the process
N(t) has no points in the interval (v, t + a), i.e. Sn+] - Sn > t + x - y. Therefore by
(1)
OO »t
P(Y(t)>x,Z(t)<z)=Z [1 - F ( t + x - y ) ] d F $
H-l J[_~
[1 - F(t + x ~ y)]dH(y)
Exercise 4.8
Exercise 4.9
termwise. Let H (t) = Z F{n) (t) be the renewal function. Deviding the range
n—\
of integration into [0, t) and [/, <*>], after an elementary computation we get
E Y r' = E ( ( X x - t ) +y - 1 + f
Jo
(5),
A OO
, t > 1 >is uniformly integrable for all r > 0. Let N(t) + 1 = v(t) = min
{n: S„ > t). So it is enough to show that , t > 11* is uniformly integrable
(u.i.) for all r > 0.
Solution to Exercises 353
Consider v(t + 5), t, s > 0. In order to reach the level t + s we must reach the
level t. When this has been done the process starts afresh. Since Su(t) > t the
remaining distance of the process to climb is at most equal to 5, and thus, the
required number of steps to achive this is majorized by a random variable distributed
as
v(s), i.e. v(t + s) < u(t) + min {k - v(t): Sk - Sv{t) > 5}
£U) < _ + 1)
/ “ 1 '
Therefore from (3) we get
Exercise 4.12
N(t) l
(a) Since ----- - a.s. — as -* ( 1)
t H
and
N(t) —> — as —» oo ( 2)
N(n) - nljuY
show that
^ J , n > 1 [ is uniformly integrable. Hence for r > 2,
(2) implies
354 Introduction to Stochastic Process
N(t) - t!ju
, t > 1 \ is uniformly integrable.
V7
(b) can be proved in a similar way.
Exercise 4.13 Conditioning on the time X] = x of first renewal we get
1 if x > t +y
P(Yt > y \ X ] =x) = 0 if t + y > x > t
Av(t - x) if t > x > 0
= 1 - F(t + y) + f A ^ t - x)dF{x)
Jo
is a renewal type equation and the solution is given by
•'O
I 1 - F(t + y)}dt = I
t
{1 - F(y)}dx < oo and {1 - F(t + y)} is
Iim P(Y,
/ — > oo t —> ° °
> y) = lim
i; = 1 - F(y)}dy,y > 0
Now the set of events {Yt > x and Zt > y } is equivalent to {Yt_y > x + y}.
Therefore
lim P(Zt > y, Yt > x) = lim P(Yt_v > x + y)
In particular,
= /i-' r
•'.V
;i - F ( z ) } d ( Z).
Then by the law of total probability we get the renewal type equation
[1 - F(x)]dx.
E (min (.Xh T)) -
Jo
f
We have the long run mean cost per unit is given by
Ct [l -F (Q ] + C2F ( r )
6(T) =
|Jo [1 - F(x)]dx
f cT
[1 -F U )]& =0
ll
which yields the result.
356 Introduction to Stochastic Process
Exercise 4.15 Let U(t) be the interoccurrence distribution for N(t). Then
1 - U(x) = P(N(x) = 0) = P(Ni(x) = 0, 0) = [1 - F(x)]2 (1)
Let K|(r), Y2(t) and Y(t) be the excess life at time t for the processes Nu N2 and
N respectively. Then, because the processes N\ and N2 and N respectively. Then,
because the processes N] and N2 are composed and we must have
Y(t) = min {Y2(t), Y2(t)}
and P(Y(t) > x) = [PiYiit) > x)}2
Letting / —» °° and by exercise 4.14, we obtain
[1
if ( 2)
where
-f Jo
[1 - U(y)] dy, the mean of U(y).
- [ l - F W ] 2 = ill-t/(K )] = ^ -
j; [1 - F ( y ) ] d y \ [ l
i.e. 2v r
1 - F(x) [1 - F ( y ) ] d y (3)
n 2J . v
2 v c(x)
(4)
dx
Since G(0) = 1, the solution of differential equation (4) gives
Hence all of the Nx(t), N2(t) and N(t) are Poisson, because of the fact that sum
of two Poisson processes is a Poisson process.
Chapter 5
Exercise 5.1
f (s) = 1 - p ( l - s)^ is a Probability generating function iff the coefficients in the
expansion o f / (5 ) are nonnegative and sum to 1, since (3> 0 ,/ ( l ) = 1.
f \
( P )
By the binomial expansion f ( s ) = 1 - p I M ) "
n- 0
V J
( f P '
\
1
II
^3
( ~ s ) n
n= 1
V )
Solution to Exercises 357
R\ n -\
Also = P ( P - l ) . . .(P -n+ l)/n'.= (-D
n n\
and (1 - (5)(2 - fi) . . . (n - 1 - P) > 0, since (3 < 1
' p\ t 1X2„
Coefficient of s" is - (-I)” = p — ^— P(l - f 3 ) ...( n - 1- /?) > 0, since
/? > 0, for n > 2.
P
Also for n - 1 we have - p (-1) = pP > 0. H e n c e /(s) is a probability
u
generating function. To compute the iterates, we use induction.
For n = 1,/,(s) = /( s ) = \ - p(l- s / .
Assume for n= m f m(s) =1 - p l+P1(1 - s)^m, then
= i - P{p« +p+- +r ~
'i(1 _ _ s ) p”+'
= lim
yV—
Z= i E( Xn ) - limN —Z (EX i) " = lim S #»"
>oon - \
h N —>°° n — 1
.V+l
= lim m - tn , since a?i < 1.
1- m 1- m
Exercise 5.3
/ (5) = as2 + bs + c, a > 0, b > 0, c > 0 and a + b + c = 1.
Hence m = f '(I) = 2a + b = 2a + (I - a - c ) = a - c+ 1. (1)
Now only root of / (5) = 5 are s = 1 and s = * ~ ^ ~ ^ = ~
Since the extinction probability is less than 1, it is d a , for otherwise from (1)
m - 1 > 0 => a > c => a/c> 1 => da < 1. Therefore, s - 1 cannot be the probability
of extinction and d must be d a .
= k=0
Z P[X/1 = ifc]MA & P V * 1
358 Introduction to Stochastic Process
= 2 I P[Xm+i= j \ X m P[Xm
=
(by Markov property)
= l p [ X m = l \ X 0 = k]
„ d2(jh (u,x)
1 E ix - x ' ' — l f c j 7 - 1 - '- -
Now, 90, „ (u, x)l du |„=1 = 0'[w0n_, U)]0n-/(*) |u=)= 0'[0„_, (*)]0h-/
90/,„(«.Jc) .
Hence [u=l. v=l = 0 "d )[0 n- /( ^ ) ] ^ - /( ^ ) ^ - /( ^ ) U i
9 udx
+ 0'[0„_/(*)]0UU ) |*=i
CovjX^Xt) a 2m n- l ( ml - 1){m - 1)
VVar(X/2)Var(X/) (m - 1)cr2 ^]mn+l~2 (mn - l)(m/ - 1)
= *Jlln if /w = 1
—> 0 as n —» oo
Thus, /?„/ —» 0 if m < 1 as n —> If w > 1, then mn —> °°, m~n —> 0 and hence
Rni —■> V1 - ni 1 .
Exercise 5.6 (i) For Poisson distribution with parameter Athe generating function
0(z) = <?A(z"l) satisfies the difference equation A0(z) = 0'(z); (ii) Similarly for
Binomial distribution with generating function 0(z) = (q + pz)" where parameters
are a? and p satisfies np0(z) = (q +pz)(p'(z)\and (iii) negative Binomial distribution
with (Parameters n and p) with generating function </>(z) = (pz)n/(i - qz)n also
satisfies np$(z) = (1 - qz)<S>\z). All of them are desired form of differential
equation. Now from the relation <j>n(z) = 0(0„_i(z)) differentiating with respect to
z, we get <p'(z)
P(Xn+{ I )=
k+
= (0) (3)
or 52 1 1 n
T "3 n -0
or 3s2 - 4j + 1 = 0
or (3 5 - 1)(5- 1) = 0
or 5 = 1 or 1/3
Hence probability that the blood culture will eventually die out is 1/3.
Chapter 6
Exercise 6.1
For / > 0, P„(X(t) =1) = l/2pM(0 + l/2p
and Pn(X(t) =1) =1/2.
By Chapman-Kolmogorov’s equation
P\,\{t + s) =P\,i(s)pul(t) + pi,_i(i)p-i,i(0 (2
From (2), we get the functional equation
a(t + s) =a(s)a(t) + (1 - a(s)
= 2a(s)a(t) + 1 - afs) - a(t) (3)
Put a(t) = 1/2[1 + h(t)].Then equation (3) becomes
h(t + s) = h(s) h(t) (4)
This is the Cauchy-functional equation whose only continuous solution is
h(t) =<rA', A > 0
From (4) and (5) we obtain that a(t) =(1 + e~^
Exercise 6.2 This is a two state birth and death process. The Chapman-
Kolmogorov equation for a homogenous M.C. is given by
Pil(t + h) = p ik (0 pkj(h), k = 0, 1
dpoo(t)
= ~a Poo(0 + ( 1)
dt
Since we have
PooM + Poi(0 - 1- (2 )
equation (1) becomes
<fpoo (0 = P - (a + (t)
dt
(3)
Solution of equation (3) subject to the initial condition poo(O) = 1 >s given by
- P r+(Po(0)-----
a + P a+ P
_P _
+ e~(a+P)' if p0(0) = 1.
a +P
Exercise 6.3
Poo(0 = a + [
+P a +P
IT if X(t) =
j given X(0) =
Let fy (0 =
lo if X(t) * gj iven X(0) = I
Pj
+ al(a+ p ) 2[1 - e-<a+P)‘]
a +P
362 Introduction to Stochastic Process
In a similar way the other functions Po\(t), /Li\o(t) and yUu (r) can be determined.
A ) 2 ( 0 = e T-(.v+
2X
)t j e ( v +2X
)u(V etp
R- W ( l
U + A) f '
_ g -(u+2A )/ I
(e 2 Xu „X u
eMi) du
Jo
w+2A)» + A ) J g 2h 1 <?A' - 1
2/1
i 2 £ ± i ) €- „ [ , _ e -A,|2
2a
We shall prove (3) by induction on n. Assume that
1,(0+ 1 ) . . . ( » ■ > ( » - i w e. „ (1 _ e. „ . hold
Pon(t) = (4)
n! An
Then,
P™ (0 = -(u + (n + l)A)p„n+i (r) + (i> + nA)pon (f) yields
Solution to Exercises 363
Xt i \ n+ 1
(( X \ \ ndx
j 1 I (* ~ 0 " +1 1 (eM - 1)
4 j; - i) - - r
A n+1
Ji A 1
Exercise 6.5 Consider a branching process starting out with i particles. Let
be the times until these particles split apart or disappear. Then
. . . , each has exponential distribution with parameter 0and hence X\ = min
(£b . . . , £;) has an exponential distribution with parameter g, = iG. Whichever
particle acts first has probability p of splitting into two particles and probability
1 - p of disappearing. Let p/; (f) be the probability that starting at i X(t) jumps to
state j after waiting a random length of time. Thus for i > 1
Exercise 6.6 Suppose there are initially j particles present. Let (£b . . . , be
the times at which these particles split apart or disappear and let rj be the first
time a new particle enters the system. We interpret the description of the systems
as implying that r] is independent of £b £2, . . . , r\ are independent exponentially
distributed random variables having respective parameters 0, 0, . . ., 0, A. Thus
Tj = min (£b . . . , rj) is exponentially distributed with parameter
« ■ 7 5 T T 1(l - ■ ^ 1>') ■- 7 5 T T + 7 5 T T P-
Hence we conclude that the process X(t) has birth and death rates
X q- j P j , J+\ = jOp +A and Hj = q,pu y_, =j 9( l - p).
364 Introduction to Stochastic Process
[Note that equation (1) follows from some standard properties of independent
exponentially distributed r.vs: If . . . , be independent r.v.s having exponential
distributions with respective parameters . . . ,0 n. Then £(d = min (£b . . . , £„)
has an exponential distribution with parameter 6\ + . . . + 6n and P(t;k =
Q
min (£h . . . , £„)) = -2------- , k = 1, 2,. . ., n. These results can easily be
■+ 0n
verified in the following manner.
P(min(^, . . . , & ) > * ) = P(£i > x, . . . , £n > x) = > x) . . ./>(£„ > x) =
.. r =6?-(0i+ Hence £(1) is exponentially distributed. Set
rjk = min (%j : j * k). Then T]k has an exponential distribution with parameter
fik = Z 6j, and and rjk are independent.
Thus = min , 4 )) = />(& ^ Vk)
= f e ke -(Sk+Mxdx = -s - ^ = -s----- ^ s
J0 Ok + Pk 0\ + - - . + 6 n
Po(t) =- Ap0(r) + p p x(
and p'j ( t )= XpH (t) - (A +jp)pj(t) + + j )
dg(C«)
— ^ ------ /l(l - u) ,—
9g(^ C«) , ,1, - u)g(t, u)
— = —A(
is the initial condition. Solving this by Lagrange method (Recatti) and using the
initial condition, we obtain
g(t, « ) = { ! - ( ! - »'e)u N
) exp { - £ (1 - n)(l - e~>*) J
Note that the first term on the right of (3) is the RG.F. of Binomial distribution
with p = exp(-jur), whereas the second term is the RG.F. of the Poisson distribution
with mean a (t) = (1 - Therefore Y(t) = Y0(t) + Y{(t), where Y0(t) is a
Binomial component and Y\(t) is the Poisson component and they are also
independent for all / > 0.
Expanding the P.G.F., we get
min (N,t)f AT a y * - e
Pj(t) = exp {-AJp(l *“)} I
=0 ks a . (T a j!
for j > 0 and
E(Y(t)) = Ne-* + £ ( 1 - e - f ) .
P
This formula was first time obtained by Palm.
Exercise 6.8
(a) The system of difference-differential equation in this case is
p'n (r) = -n(A + p)p„(r) + A(/i - 1 ( )+ + l)p„+l (?), > 1(1)
Po( t ) = ppx(t),pn(0) = <5„, (2 )
There = 2 npn(t)zn~]
dz "=i
+ a? n2,
=\ (n + 1)/Vn ( ?)z" + p\ (?)
- (A + ^ )z ^ + Az2^ p + / J r p
az dz dz
2i dG
= [p - (A + p)z + Az2]
dz
= (z - l)O z - M) G(Z, 0) = z (3)
is a partial differential equation of Lagrange type.
366 Introduction to Stochastic Process
i) + Z(A - )
Hence G(z, r) =
( A e ^ ' - / t ) + Az(l - e ^ ' )
1- _ ; f 1- 1
Put £ (0 = ju (4)
Po(0 = £ (0
if A * (7)
Pn u ) =(i - $(0)(i - r 1, « > i
A?
Po(0 = 1 + At
and if A = ( 8)
(A O " -1
Pn (0 = n>1
(1 + At)n+1
(d) if A = Jta nd if
p,p0(t) = T * ^ =^
(e) From the definition of |(t) and r/(t) we see that
Solution to Exercises 367
E if ju < A
and
lim r\(t) = A
1 i f jU > A
f e i f A * p an d
£ ( * (0 ) =
[ 1 if jli = A
1} if A * p
Var (X(r)) = A - p
2 A/ i f A = jU
Alternative derivation
Multiplying both sides of (1) by n and summing over n = 1, 2, 3,. . . and adding
= m2(0 + m(t).
= {m2(-t) p i( ( ) | - {m(r) -p i
= "i2<0 - '"(0
oo
and I (0 = m'(t)
n=\
0 if A < p
lim E(X(t)) = 1 if X - p
°o if A > p
^ U L ea-n)>{ea - ^ t _ i} j
A — /LI
2 Xt if A = p
oo \iX>JJL
Hence lim Var(X(t)) = oo if A = p
t—>°o
0 if A < p
f P{{ t ) d t = \ a-i(o)(i
J0 Jo
= (^-A)3
I
o - Ae(A-">' ]2
[H
Putting v=n -Xe{A we obtain
l/A if A > p
i; Pi({) dt = Up if A > //
l/A if A > p
< oo.
The last case is obtained-either by continuity from the cases A > p and A < p or
v2
by calculating J | 1 - 1 +XtXt dt = l/A (from (8)) < oo. Thus {1, 2, 3, . . .}
'H - ^ L
Pij(O = n Xk I exp (- if > i
J \ k=i k=i
- exp (-Xjt) if j = i
- Xkt
x Z
* = ' A ' " ( I - *)((+ 1 (-1)1 ...(j-
fj-i^
_ e ( 1-
y j-
Since p^(0’s are transition functions, S(t, /, °o) = 2 /?,*(*) < 1. Then either
k-i
S(t, /, 7) = 1 or < 1 for all f.
Using forward equation we obtain
=- .jP
A i(?) for j > i ( 1)
Let cr((9, /,y) be the Laplace transform of S(f, /,7). Noting that S(0, i,j) = 1,
it follows from (1) and integrating by parts that
9(7(9, i j ) -1 = - Aj 71(9, i, j)
J Xk 1 . We see that
By previous exercise 7r(0, /, j ) = n
k=i 0 + Xk Xj
n-i
n 1+ e if k=i
Z l/Xk < 00
k>i
Jim Ajiz(9, i,j)
if kZ= i 1/A* = 00
Hence
(i) lim 0<7(0, /, j ) < 1 if the series Z 1fXk < 00
7—>°° k>i
Solution to Exercises 371
t implies lim
;-> °o
0 If°° e~dtS(t, i, j) dt = 1 or < 1 . Hence lim S(t, i j ) = Z
J 0
00
pik(t) =
*= *
iO
¥n(0) = n 1 - -t— and clearly (from the theory of infinite product)
k=i J
lim y/n (0)
n—4 oo
= 0 if Z AIs.1
K
= ©©. Therefore, as n - > ©o, w - > oo a.s.
**Exercise 6.11 Let rk be the duration of occupancy of the £th state. If rk are
positive, by definition of Markov process they are independent with distribution
function P(rk < x) = 1 - exp [- Xkx\ and Erk = 1!Xk. Define r k = rk if xk < 1 and
are nonnegative r.v.s, Exk2 < E( xk) and therefore convergence of the series
Z E(xl) implies convergence of the series kZ= i Exk2 and hence of the series
k=i
l P ( T k >1)= Z (1)
k=i k=0
and £
k= 1 k=0
E(
k=0
t I) = £ e ' Xk + £ (1
Hence ^Zq 1/A^ = ©©but the series (1) converges, then the series (2) diverges.
Thus in the case of divergence of the series 1/Xk, one of the series (1) or (2)
diverges and consequently Z■xk diverges with probability one. For linear process
Exercise 6.12 (a) From Kolmogorov’s forward equations for birth and death
process we get for the unconditional probabilities pt{t) the system of equations
372 Introduction to Stochastic Process
~^oPo + P\P\= 0-
Suppose that p* > 0 for k > 1. Then pj = (A0/pi)po and by induction, we find
AqAi . . . AqAj . . . A*_!
Pit = - p0, and Z p* = Po 1 + E
P 1 P 2 • • - Pit ' *=°J P 1P 2 • • Pk
AqAj . . . Aj^_i
From part (a) by virtue of equations pk = ------------------p0, k = 1, . . . , m
P 1P 2 • • •Pk
^0 • • • ^k-\ ^
and E p k = po 1 + Z the stationary distribution is given by
k=o K ™ *=1 P\ . . . P*
1) (s + l)( A
Pit =
s £-.S u .
\* m
k ( k - l ) . . . ( s + l)( A_
Po = 1 + Z - + 2
k=0 J i = .t + l P
**Exercise 6.13
_ £ £ g - A ( J + / ) i ^ £ ) ^ _ . £ ^ £ ) ^ L n (n+m)
n'. /n!
(by Fubini and Chapman Kolmogorov)
! rn
_ g-A(j+l) J“ ejk_
o(,)
• 2 (As)" (A/)'-"
1-0 l\ /i=0
A
= S P1
* [A (j + /)]' -Pik (s + /).
\ av
or
I ^ii ~ ~ Q ii = Xpa ’ * ~ J
Now e~^ — p i{}n) > 0 for all n > 0, A > 0 and 5 > 0, so that
n\ lJ
Therefore i —>j in M .
Conversely, if i —>j in M then there is some s > 0 such that Py (s) > 0.
Now
374 Introduction to Stochastic Process
e - Xt (Ay)"
and />'"1 > 0 for all n > 0, some 0 and A > 0
and there will be at least one n > 1 such that
e - Av(As)"
p '"'' > 0 =>p-(n)
'; " ' > 0 for some 1. So —»y in
n!
(b) and (c) We have to prove
(i) Urn p0 (.s) = Urn and (ii) J p,; (s) =jj- p\p
Proof of (i) Let p-n)—» coexists since the M.C. is aperiodic), then | Py(s) - a,
-Av
(As)"
< X | p)n - a j | and for every e > 0, there exists N(e) such that
n=0 n\
e -^ ik sY
for n > N | p {tJn] - a] \< ell. Since —> 0 as j —» oo, we can choose
r°° °o p {n) p -
Proof of (ii)
f/) J (5) ds = Zq ■J e”'1' (As)” ds (by Fubini Theorem)
= 1 £ n<n>
A n=0^/7
From (i) and (ii) we get (b) and (c) since j is transient in M iff
**Exercise 6.14
4. Let M be irreducible and positive with stationary distribution {7ry, k e S}.
Let M be null. Then p\.n)< p n even though p (/° -> 0. For°o= Z p ^ }< Z p"
1 ' n n
1 < oo.
1- p
AT> sup | au | < 00 and P a stochastic matrix since P > 0, 2 p i; = -p-Z al} + 1
Exercise 6.13, and the definition of P. Hence M and M are the same by
Chung’s theorem. Hence M is positive <=> M is positive <=> M is positive by
Foster’s Theorem (By Exercise 2.36) ( Z j c = x} has a solution such that
0< Zi | jc,-1 < 00 (by definition of P) <=> Zi jc, = x; has a solution such
- f +
that 0 < X |jt,-1 < °° <=> %XiOjj = 0 has a solution such that 0 < X | | < °°)
This proves 5 of Exercise 6.14.
Exercise 6.15
-(a + A ) p y
1. | A —A/ | = y - ( a + A) = 0 or
P y -(a + A )
Pi 3(0 = P 2 l ( 0 = P 3 2 ( 0
recurrent (If not obvious, use skeleton chains to reduce it to discrete time).
Hence lim P(t) = R. By Exercise 2.14 every row of R satisfies Xjc,ai7 = 0.
t ~.»°o I J
3
cos (err) + [S" - T") sin (ot) + i[S" + T"] cos (at) + i[T' - 5 '] sin (at)
(0 )
Su = V = 1/3
rv p 2p - 3a . 3
j,/// —~ j>t a// _—— + —- — 2------ r 0, since p =
2a 3a 6a
S' = V = - 1/6 if/ * j
P // T">// _
*5 // + l ~ ~ 1ii+ 1 ~
2cr 3(7 = ^ 4— - 1/3 V372 ( < 7 = i(jB - y ) V 3
All continuous time Markov chain with infinitesimal generator of the given
form and with P+ r constant have the same skeleton (X(/i), X(2h), X(3h), ...},
P-r
where h is defined above.
Since we look for solution of Elfving’s problem, we need that
p q r
/>(!) = r p q ,p * 1 (6)
q r p
e p cos a - 3 P - 1
Hence (7)
e~p sin <7= (p + 2q - l)
, r- p + 2q - \
so that (7= V 3 — -------- :— .
j p —1
This determines infinitely many <7’s if we can find p also. If p can be determined
then every continuous time Markov chain with that p and one of the possible a
values is a solution to Elfving’s problem. Now from (7)
e~2p = e~2p (cos2 <7 + sin2 o)
= 3 ( p 2 + q2 + pq - p1
The right hand side is minimal for p = q = r = 1/3. This is excluded. Hence
e~2p > 0. But g > 0 , so p < p + q\ r > 0 so p < I - q. Moreover one of these
inequalities is strict since otherwise q = O.p = 1.
Exercise 6.16
(a) is completely similar to Feller-Arley process.
(b) Following Feller-Arley the P.G.F. of {pn (0)o
!-£ (? )
= ^ = e ^ " 2'2
1 - r(?)
0 if Ji > 0
Hence E (X (? ))^ 1 if A = fj.
00 if jc > 0, // = 0
1—e = 1 - e - f /2.
When A = 0 ,p 0 (O|A=0 =
f V
I11 + Jo
Chapter 7
Exercise 7.1
1st Solution Let X(t) be the number of shocks upto time ?. Hence
Axk-'e~Xx
P[T< ?] = I e - A'(A?) " /n\ = f dx (i)
n=k Jo Tk
(integrating by parts).
k
Xk t k~ - 1 „ - A /
2nd Solution
Let Xt be the interarrival time till the ith shock, then P[T < t] = P S xt< t
t-1
Third Solution
Let T = Tk. Then by induction on k,
Exercise 7.2
Exercise 7.4 Let r = W* - W. Then r is the interval time and P(r < s) =
1 - e-Av. Now P(N = k) = P[y(lV*) -Y(W) = k]
380 Introduction to Stochastic Process
'o
e (fis)klk\ Xe ds =
K- Jo
J ds
Exercise 7.5
(a) Let 0 < /0 < / ] < . . . < tm. Then
Now, {£,} are the i.i.d., X(t) being a Poisson process, X(tk) - and
- X(tk_2) are independent and Y(tk) - Y{tk_\) and Y(tk_i) - Y(tk_2) are also
independent.
Exercise 7.6 Since the interarrival times Tu T2, ... in a Poisson process are
i.i.d. with common distribution exponential with parameter A, £(^(7]) | Tx - t )
= Ar. Hence
£ W ,) T ,) = £ [r,£ (V(7,) | 7-,)]
= 2/A.
Var (/vcrWi) = £(M7",)£|)2 - £2 W , ) r , )
if 11 - s | < 1
PrC M ) =
if | / - s | > 1
Therefore Y(t), - oo < f< o o isa second order stationary process having mean
A and covariance function
p Y(t) decreases linearly to zero as 111increases from 0 to 1 and remains zero for
all larger values of 11 1.
E(U(Ta)) = E(Y2(T„)) - = 0) = 0
E(Y2(Ta)] = XE(Ta) = a( by ( l ) ) (2)
Also
Y(Tn) = X(Ta)- XTaandE[Y2(Ta)}= E(X(T„) = E[XE(Ta)
(••• X(T„) = a, E(T„) = a/X => X(T„) = XE(Ta). Therefore
E(Y2(T„)] = X2E(Ta- E ( T a)]2
=> a - X2Var ( T„) =>
Again V(t) = exp [- (t )+ Ar(l - e-0)], r > 0 is a Martingale.
X
G
Hence
£ [T O ] = E(
exp [- GX(T„) + Arfl(l - e - % = E(V(0)) =
=> 1= E{exp [- 0X(Tn)e] xp (- a T a)} where 1 - e-0)
1 = e-0a E[exp (- ccTa)] £[exp (- )] = e"e.
£[exp ( -a])=
a +X
Ta has a gamma distribution with parameters a and X.
Exercise 7.9 Let N(t) = Nx(t) + N2(t), A, + A2, A, > 0, > 0 be a Poisson
process, where [N{(t)} and {;V2(/)) are independent renewal processes.
k\
Then e~Xl (Xt)klk\= I
m=0
A 1m ^2
l k-m
e~XtIT [n J =■
P[yVl(f) = 7,11 P[Nl(t)
Putting k - m = n we get
Exercise 7.10 This problem can be looked upon as a birth and death process
with birth rates and death rates
Solution to Exercises 383
1r r* if r
=
1)1 LJ o
I fie px dx
Jl J h
+ o{h)
We know that lim pn(t) = n n exists and are independent of initial conditions
t->0o
pio(0) = 1 and /?„(0) = 0, n * i. Also lim p'n (t) = 0.
Either we can derive explicit solutions of the differential equations (1) and (2)
which is known from D.G. Kendall’s result or just taking limits t —> °° in (1) and
(2) and putting p'n (t ) = 0. We get
Ano = p,7i\
(A + np)nn + Ann_x + (n + l)pnn+\
Let an = nfiUn - Ann_\. Then from (4) we get
an+x = a for all n > 1. So a x - a2 = ...
But from (3) we get a x = 0 and an = 0 for all n > 1.
Therefore, from (4) we get nfinn = Ann_x for ft > 1 or n n - — • - 7 l n-X(jU* 0 ).
P
By iteration we get
r a y /• m m
n - I — I /ft! 7r0for all n > 1. But 1 = ^ n k = /r0 ^ 1!
384 Introduction to Stochastic Process
Therefore, 1
/T0 =
IT
So the stationary probabilities are given by
ju) n\
ft = 0, 1
X | , ,,
*=o I ju) k\
Exercise 7.11
Let EY(t)
-rJo
P(Y(t) > x) dx - I < oo ( 1)
- f
Jo
P(Y(t-y)> x)dF (y) +
J/ +JK f
where for y < /. Since the origin is shifted to y, so that point t becomes t - y, while
for y > r, we observe that
P(Y(t) = y) = 0 if r < y < f + x and = 1 if y > t + x.
Substituting this recursive formula in (1) we obtain
/[1 -fW ] [1 - F ( y ) \ d y (2 )
■ f
Since F(x) is continuous, the right hand side of (2) is differentiable;
differentiation and the solution of the resulting differential equation give the
exponential distribution, since F(0) = 0. We know that N(t) is Poisson iff F(x) is
exponential. In fact,
oo > /
■a P(Y(y - t ) > x ) d F ( y ) d x +
JO Jr+.v
dF(y) dx
= f / dF(y) +
+ If [1 - F ( y ) ] d y (y = t + x)
Jo Jr
= lF(t)+\
I [1 (F( 0) = 0)
[0 if r < L
(C) F
(t=
(l if t>L
c \t if t < L
Therefore a(t) = [1 - F(x)]dx =
Jo u if t>
Chapter 8
Exercise 8.1 Since Brownian motion X(t) is an independent stationary increment
process to derive the joint density of X(t{), X(t2), . . • , X(tn) (tx < t2 < • . • < tn)
subject to the condition X(0) = 0 a.s., we only have to know the probability
density of Zx - X(rj) = x x, of Z2 = X(t2) - X(t{) = x2 - x x, etc. and finally that of
Zn = X(tn) - X(tn_x). Since the Jacobian of the linear transformation is unity,
making change of variables the joint density is given by
386 Introduction to Stochastic Process
Now consider the case n = 3 with the condition X(tx) = X(r3) = 0 and more
specifically suppose that t\ = 0, t3 = 1, t2 = t(0 < t < 1). By (1) the joint density
of X(t) and X(l) is
/ U ,0 ) 2njt{r^7C
Xp ( 2r(l- / ) ) '
The marginal density of X(l) = Y at the point 0 is
fy (0)=L i ^k^T e
)xp(2^1-0) =vfe
Therefore, the conditional density of X(t) given X(l) = X(0) = 0 is given by
fix I X(0) = X(l) = 0) =f(x, 0)/fy(0) = - .......- L -
J 2 n t( l - t )
Exercise 8.2 This can be reduced to the preceding exercise 8.1 as follows.
Since X(t) has stationary increments, the r.v. X(t) subjected to the conditions
X(ti) = A and X(t2) = #, has the same density as the r.v. A + X(t - t{) under the
conditions X(0) = 0 and X(t2 - t\) = B - A. This clearly has the same density as
the r.v. A + X(t - tx) + —— — (B - A) under the conditions X(0) = 0 and
h ~h
X(t2 —t j) = 0.
Exercise 8.3 (a) We must compute the probability p that the particle will cross
zero level at some point in the time interval (a, b). Let px denote the probability
that the particle located at A at time a will cross the zero level in the time interval
(a, b).
Then p = f pzp0[\X(a)\ = MdA
Jo
where Pq(A) is the conditional probability of A given X(0) = 0.
Now Pa = P{ min X{t) < 0 | X(a) = A}
a<t<b
= P{ min (t) <- A |
X
a<t <b
(by spatial homogeneity of Brownian motion)
= P{ min X(t) < A | X(a) = 0)}
a<t <b
(by symmetry)
^a<t<b-a ~^ (by temporal homogeneity of X(t))
= P0[Tx < b - a ]
Solution to Exercises 387
where T\ is the first passage time across the boundary A of Brownian motion.
Hence
-T (2na)-'l2e - A /2aP0[T
Ml
b-a
= 2n 1tan 1
cos 0 - Z sin 0 1
f a .b(Zcos 0, Zsin 6 = 1/2 7rZ<rl/2(z2 >
sin 6 Z cos 6
r pin
l/2n exp iA Z cos (27rf* + 2 /rr +0 )
I Jo *=1
p 2n{ r+1)
= Ej 1 1 exp
1 /A Z uk cos (2 /rf* + y/) dyf\ (\\f- 2nt + 6)
[ 2 n \' 2nx *=1
Solution to Exercises 389
r-"
=E exp iA 2 uk cos (2ntk + y/) dW\
& Jo k=\
( n ^
= E i2 uk (v X(0 = A cos (2nt + yr)
*=' y
So by uniqueness theorem of characteristic function (X(r)} is stationary.
pin
(b) E(X(t)) = E\A( \l2 n
/ .2*
rJ
Jo
cos (2m + 0 ) d 0 | = E(A).
= ——
—zr~exp (- a 2! 2 o 2), a > 0,
2n o 1
which is the Raleigh density.
{X(r), - oo < t < oo} is not a Markov Process because for t > 1/4
E(X(t) | X(0) = y, X(l/4) = - z) = y cos (2 m) - z sin (2m)
which depends on y, contrary to the Markov property.
Exercise 8.7
E(U(t)U(s)) = e~'e~sE(X(e2')X(e2s)},t > j
= e~‘e~s min (e2‘, e2s) = e~‘e~'e2s = e~,+s = />5
Therefore E(t/(r) { /( j )) = e - 1 ' - ' 1.
390 Introduction to Stochastic Process
Exercise 8.8
EX2(t) = E
f
Jo
W(s)ds =E
Jo' W(v)dv\
E[W(u)W(v)]du + | J
■III
=2 r r )W(v)du] dv = 2J du
| J > dv
u
Jo Jo
where Sn = {k/2n, k = 0, 1,. . 2n - 1}. For fixed W, let rb t2, .. . tN be the points
of Sn with tx < t2 < . . . < tN. Let T((d) be the first i (if any) such that
| W(th (d) | > £, and write
N
EW2(1) = 2 P(T= k)E(W2{1) 1 7
k=I tesn
N
E(W2( 1) I max I X(t) \ < e) > I P(T= k)E(W2( 1) I T= k)
te s n k=\
+ 2E[W(tk)(W(l) - W(tk))2 1
Because the event T = kdepends only on W(t\), W(t2), ■■ W(tk),
W[W(tk) ( W ( l ) - W ( t k))T=k]
= E[W(tk)V
[(\ , - W(tk) | W(t,),. .
E
(by martingale property of Brownian motion, see Appendix I).
Solution to Exercises 391
Exercise 8.10 Let Yk = W(k) - W(k - 1). Then Yk are independent /V(0, 1)
variables, X(n) = Yl + . . . + Yn is the sum of i.i.d. r.vs. Since EYk = 0, by strong
law of large numbers for i.i.d. r.v.s, X(n)/n —> 0 a.s. a sn -)< » .
The first term —> 0 a.s. Since Zk has the same distribution as
Therefore lim = 0 a s-
E[e~m x ) ] = f
Jo
e’ 0' dP[T(x) <t] = exp [- jcJ2Q]
Therefore 0(m) = f
Jo
exp (-1/2 u2t) dP[T(x) < t]
This implies that the first passage time r.v. T(x\ + x2) for reaching x } + x2 > 0
when W(0) = 0 a.s. is the same as the distribution of the sum of T(xx) and T(jc2),
where T(xx) and T(x2) are independent, X\ > 0, x2 > 0, i.e. {T(jc), ;t > 0} has
stationary independent increaments and it is nondecreasing. From this it follows
that
Y(x) = W2(T( x ))
also has stationary independent increments.
f2m -
= i -<t> , m > 0, m > x.
l~ v rj
where <p(x) is the c.d.f. of N(0, 1). Differentiating with respect to x and then with
respect to m, changing the sign and using the relation dldx(j>(x) = -0(jc), we get
the joint p.d.f. of M(t) and W{t)
d__d j _ f 2m - x d f 2 m - jc ^
dm dx V > dm
2m - x 2 2m - x
* V7 ~ 7 T
Solution to Exercises 393
where Q(x) is the p.d.f. of /V(0, 1). Joint density of M and W is given by
Therefore
E [T a /i] = E[W2(T a w)] < (| a |2 + Z>)2
0 = E(W(T)) a P [ W ( T ) bP[W(T) = b]
= a(1- p ) + bp.
Therefore
P ~\a\ +b
E(Y(T)) = 0 implies E(Tab] = E= a \ l - p ) +
= a2■ +b = \a\b.
a|+b |a | + b
By changing the origin and scale we get the result for Wiener process with
variance cr2 and starting position W(0) = x.
Exercise 8.15 Let Z(t) = exp {X(W(t)) - (A/i + 1/2 A2cr2)r} is a martingale,
where W(t) is a wiener process with drift ji * 0 and variance parameter a2. If
Ao = - 2/i/o2. Then Z$(t) = exp (A2(W(/))} is a martingale. Let Tab be the first
time the process reaches a < 0 or b < 0. Then by optional stopping theorem (see
Appendix IV) 1 = E(Z0(Tab)) = P(W(Tab) = a) exp ( V ) + P(W(Tab) = b) exp (A^),
and
1 - exp (A0 a)
P(W(Tab) = b) = , where A0 = - 2 f il e
exp (A0 6 ) - exp (A0 a)
394 Introduction to Stochastic Process
changing the origin to the case W(0) = jc. We get the result
Now left hand side becomes the probability that the process ever reaches b,
i.e. the probability that the maximum of the process ever exceeds b. Thus, for
m -b-x,
**Exercise 8.16 Now Y(t)= exp {AW<t) - \l7?}t) is a martingale for a standard
Brownian motion W(t). Also W(T) = a + bT, a > 0, b > 0. By optional stopping
theorem £[exp {\W{T) - 1/2 X2T}] = 1. Therefore
£[exp {Ma + bT) - 1/2 A2 T}] = 1.
Putting 6 = A- 1/2 A2, yields E[exp {6T + Aa}] = 1 .
Hence £[exp (GT)] = e~
Now to get the value of A, we have A2 + 20 - 2Xb = 0 and
X = b± -fi - 20
We require A> 0, which implies
A= 26> if d = b 2/2.
Chapter 9
Exercise 9.1
The autocovariance generating function for ARM A (1,1) process is given by
C ( Z) = a 2 a + f c -1
j B z ) ( l + ) = c t 2 (1 + + f c + f c - 1 )
f(A) = C ( z ) = X
J=~<J
Exercise 9.3 Let B be the Backshift operator. Then <plpi(B)Xt = 6i(n(B)eu and
<kn (B)Y, = 02n(B)e2„ where <pi<n(z), <P\q](z) and <t>2 q2(z) are
polynomials in z of degree P\,P2^ \ y ch respectively and eu and £%are independent,
Zero means white noise processes.
Since Zt = Xt + Yt, it follows that
0lp](B),4>2p2(B) Zt = 02p2(fl) 4>]pi( B ) = X t Hh 4>lpi(B) 02p2(B)Yt
Exercise 9.4 For ARM A (1,1) model equate the coefficients of the right and
left sides of (1 - 0 - B) \i/(B) = 1 + QB, where y/(B) is a function of back shift
operator B given by y/(B) = iff jB j . From the constant term \f/0 = 1, the
coefficient of B gives - 0 4- y/l = 0, so y/] = 6 + 0; equating the coefficients of
B2 gives -0iff\ + iff2 = 0, so ¥2 = $V\ = 0(0 + 0). and so on, to give
396 Introduction to Stochastic Process
( 1)
Since we have the knowledge of the entire past Xh Xt_u . . . of the series at
time r, the linear predictor of Xt+k k > 0 Xt+k\t is a linear combination of the
known past values, viz.
(2)
k- 1
= .2 V , £ , +k-i + 2
/= 0 z = A: 7=0
M *-l
= I y/jE,+k-i+ 2 - *,•
/=0 z=A:
8 i ~ V i+ b
— V i+ k £f -i
Solution to Exercises 397
E[(Xl+
k- X , +*„)2 ] = (X 2 2 V?.
(b) For the AR(1) process X, = a XM + £1? | a | < 1
Xt = Z a j e . i so that w = a j
1 i=0 1J TJ
We find the &-step predictor has
gj = a j+
k,given X,+t|, = jS a 2+*e,-, = a* 2q a*e H = a*X,
For A/A(l) process X, = £, + /fcM, | j8 | < 1, the optimum one step ahead
predictor is
£, = X, - 0eM ,~8(X,_,
X
= j - =..
= X, - PX,_1 + P2 -
X Xj_3 + . . .
Substituting this in (3) we get one-step predictor as
X,+1|, = PX, - j32X„, + - 3+ • • •
Exercise 9.6 Let Xt = 6 £M + £, be a MA{\) process where {£,} are i.i.d. r.v.s
with zero mean and finite variance <r2.
(1 + 0 2) if /t = 0
Then Cov (X/+/,, X,) = 0cr2 if h = ± 1
0 if | h | > 1
If | p | > 1/2 then k{h) is the autocovariance function of the above MA( 1)
process with a2 = (1 + 0 2) ' 1 and 0 = (2p)"1(1 ± ^(1 - 4 p 2)).
which shows that k(h) is not non-negative definite and therefore by Bochner’s
Theorem it is not an autocovariance function. If p < - (1/2), the same argument
using n-component vector a = (1, 1, 1, . . .)' again shows that k(h) is not non
negative definite.
398 Introduction to Stochastic Process
Exercise 9.7
S\ = [i + S$ + X \ = /a + X\
E(S\) = juy E(St) = p + £(Sm ) = tju
Cov(S,, 5,+/,) = E(St - tju)(St+h - t + h ju) = Var (5, - tju) = fcr2
Thus, S, is not second order stationary,
Let Yt = VS,. Then VS, = (1 - - SM = p + Xt
E(VSt) = ju, E(Yt+h) = ju for all h = . . . , -1, 0, 1, 2, . . .
f or2 if h=0
Cov(K„ T,+„) = £(Y„ X,+/>) =
[0 if | A | * 0
Obviously Yt - ju + Xt is a (i.i.d.) stationary process and also a covariance
stationary process.
02 + 0i < 1
02 - 01 < 1 ( 1)
-1 < 02 < 1
X,I = i=
£Q 0*8'^
l l and Xr+*
[+K = /=0
I 0' et+k*
I.
f U ) = ^ \ l - 6 e - a \2.
Since {eb e2, . . . , £n) is distributed as multivariate normal with zero mean
vector and covariance matrix
"(1 - 0 2)- 1 a2 0 . . .0
0 a2 0.. .0
E=
0 0 a 2. . .0
0 0.. . . . . a 2
iii = G 2n
1 - G2
n
and x'lr' x a 2 (1 - 0 2) X2 +
Therefore
Ln = -1/2 [n log(27K72) - log( 1 - 6)2
+ or “ (i - 9 2) x,2 + I -
7= 2
P <7
**Exercise 9.10 Let {X,} be a ARMA (/?, q) process E a rXt_r = r=0
E f3s et_s,
r-0
Oq, /30 * 0 and {£,} is a white noise with £(£,) = 0 and {£,} are i.i.d. r.v.s with
2
s=0
£ £ 2 = cr2. Then the spectral density of X{ is given by •—
p
2
r=0
a rzr
s = e~lX. Let = 38 (Xt, t < n) denote the closed linear space (closed in the sense
of mean square convergence) generated by linear forms in the Xt's, t < n. Now
I a{e-iX ) |2 = n (e a,
M=1
= n, 12 , 12 n | i - e- ‘V |2 | « p|2
Since for | | > 1, the log(l - z^1) expansion is valid and hence
and hence Po
ao
= 2 ^exp J log/(A)<M.J.
= n - '( A i/( l- a ) ) S (1 - a 2)
7=1
= n~'(n/(l - a))(n - a (1 - a")/(l - a))
+ 2E(X„)(EX„ - / i / ( 1 - a )) -> 0.
or Vn((l - a ) X n - ju) = - AO - a X n/ J n .
n
E( X„ /J n) E
n a n-j 14n ->
=
7=1
or
B f i E { X 2n
jl {\Xnj\> e Bn))
402 Introduction to Stochastic Process
= b; 2 2 j 2{(£j - h )2
/ [(e,
i= l J
— n __ _ n
** Exercise9.17 Let Z„ = n"1 2 X, XM since aXn + bZn = n~' 2 X,(a + bXM).
i= l i=1
We let T, = Xj(a + bXi+l). Then Y2, . . . is a stationary 1-dependent sequence
with
Now aV«(X - n) + b ^ ( Z „ n )2 aX +
^ (72 2a 2n
(X, Z) ~ N2(0, 2), 2 =
\2 a 2n <J4 +4o2n 2j
which implies 4n(X - Z
n, rt - n 2) W2 (0, 2) •
Now y (l) = Z„ - X2. Apply Cramer’s (delta method) Theorem using the
(b) (i) Let (/, = X,2 and V, = X,X,+I. Then W, = aX, + bUj + cV, is a 1-dependent
sequence with mean EW, = b o 2. ThenE= era2 + 2abf.li, + b2fiA +
EWjWi+i = /ro"\ so that Var (VV,) = a2o 2 + + + c2r f -
aXn + bUn + cVn - 2bo ) N(0, 202 + 2aZ>p3
b2(pA- o4
) + c2o 4).
and V^(X„ , U n- o 2,Vn) ^ > /V3(0, £),
^ o2 ^3 0 '
v 0 0 *4 J
Therefore U
„_-o_2,Vn)-^> N}(0, £).
(ii) p ( 1) = g(X„, Un, V„), where £(x, m, v) = (v - x)2/(u - x2). We have
g(0, o 2, 0) = 0, and hence g ( x , u , v ) = ((2jc(u - w), -( - at2),
we have g(0, cr2,0) = (0, 0, 1/ct2). Hence by Cramer’s theorem (delta method),
V n(p( 1) - 0) o 4t o 4)=
Chapter 10
Exercise 10.1 This is a Queueing system with mean arrival and service rates
A and ju, respectively, where no more than k units are allowed to enter the system
at any time, we set
Then 7T01 in the generalized Birth and Death model in the text is given by
\ _ pk+l
s = 7Tq] = 1 + p + p 2 + .. . + p* = —— — where p = A/p is the traffic
intensity. Since s is finite, we conclude that the steady state exists for all values
of p. Then nn= (1 - p )p ”/(l - p*+1), p * 1, = 0, 1, . . . ,When p = 1
= ik!2f p = 1.
404 Introduction to Stochastic Process
= k ( k - l)/{2(k+ 1)} if p = L
Exercise 10.2
Here Xn = A if 0 < n < k
= 0 ifn> k
fin = n/u if n < c
= cfi if c < n < k
Writing Xlcp = p, we have
nn = /Zo cnff/n\ \i 0 < n < c
= 7l§c(pPIn! if c < n < k
i i m n ^0 = 1
n= 0 n \ \ jU ° «=r cn~Cc\ I P
AV
and hence 5= = Z (cp)n/n\ + Z —-— f —
« = r C n~ r c ! IP,
r-1
= Z (cp)n/n! + — - Z p"
n=0 Cl n=0
= S (cpr/n! + ^ i ^ l ifp3tl
n=0 cl 1-p
r-1
z
= Z ^ +^ a - c + l )
M=o ni cl
ifp = 1
Note that the steady state exists for all values of p. Now, the expected number
in the queue (waiting) is
Lq= ^ - C) 7T„
n=r+l
= {^o(cp)‘7c!} Z (
n=c+\
k-c
= {7T0(cp)<;p / c !} Z np n - \
n=\
Solution to Exercises 405
= { T t o i c p Y pic'.}^ [ ( 1 - p * - f+ ,) /l - p ] if p ^ 1
- Q;-;C P / P [1 - P*~'+l - (1 - p ) ( * - c + 1 if p ^1
c!(l - p ) “
The average number being served or the mean number of busy channels is
<-] k c-1 k
Ls = Z nn„ +Z cn„= Zn Zc n 0ccp n/c\
n=0 n-c n=0 n=r+l
r r k
= Z { (cp )'"'/(H - 1)!} + —
p=l : M
2C: p=r+1
" -'
= c p TTottfo1 - (c p ) c /c\ p
= c p ( l - ^ ) = — (1 - A*)
L - Lq + Lv = Lq + —■(1 - 7T*)
U n&t+ o(At)]nn
= lim k- 1
A f-> 0 X [AwAf + o(At)]nn + 0 .nk
n-0
[A„ + 0 (Aa)/(Aa)]tt„
lim k-1
Af —>0 X [A„Ar + o(A/)/Af]7Tn
a?=0
Aw7Tw A7T„ 7T „
k-1
(n < k - 1) ( 1)
1 - 7Ti
X An7Tn A X 7Tn
p=0 p=0
406 Introduction to Stochastic Process
<7n + WJO)
f ' ycincx)'
qn
ii
n-c Jo (n ■ - c) 1
k-j JUC |3
1
= 2 Qn + WAO)
n-c
i J, ( n - c)\
k- 1 n-c (pct)-i e ~ ^
7^
Qn - z 2 :I + WJO)
II
, v V (net)i e
i - 2 <7 ,, 2 -----------n----------
n=c j-c J1
An7Tn
In MIMIC/oo,qn = by leting k —> °° in (1)
Z An7Tn
n=0
?l7l„
= 7in(n> 1)
A Z un
n=0
= 0 for n > c
Thus, the steady state exists for all values of A//x and
7in = {XlfX)kln\7^ (0 < n < c).
The probability that all the channels are busy in the steady state is given by
(Xlju)c/c\
71r
1+ {X! fi) + (A//i)2/ 2! + . .. + ( X / j u ) c lc\
Solution to Exercises 407
N \/( N - n )\( AV
— ^— [ ttJ *<> (0- w<c)
n„ =
N \/(N - n)\ ( V \ n
n 0 (c < n < N )
cn~cc\
C —1 i y n n!
Ur} = 2 - 1 + 2
n=0
V “ V
n=c
\ n J cn Cc\ I I1
This is a finite source queue. A typical application o f this model is that of
servicing of N machines, where arrivals corresponds to breakdown of machines
and the repairmen are the servers. The service times are assumed to be exponentially
distributed with mean IIfi.
Exercise 10.6
c- 1 'AP a ' i " 'AP
* i n\
E—
n=0
n n —7Tq Z Al
n=0 y , UJ + Z n\
n -c \y j c n~cc\ UJ J
c~l fAT| A Y \ 1 $ N\ n\ X Y
- n 0 2j n — + -r Z
n=0 c! n=c
n) cn~cc\
"c-l n N 'AP
where n 0x = z ( N \ fA] +— z n\
fiVl
, n . UJ c\ n=0 y , UJ _
n=c c n~cc\
N N c -1
N
Lq = Z (n - c) n n = E n„- 1 - 2 nn c
n -0 n =0 n- 0 n=Q n=0
c-l
= L - c + X (c - n)7tn
n- 0
408 Introduction to Stochastic Process
AA
Hence Lq = L - c + /r0 Z (c - n)
n=0
If the system has n units in it, there are N - n possible arrival units outside
the system, each with a mean arrival rate A, so the mean arrival to the system is
(N - n) A.
Hence the effective mean arrival rate,
N N N
A' = Z (N - n )A n n = AM Z n n - A Z az7T„
n=0 n=0 n=0
= A M - A L = A(Af - L).
E ( W) = 1—- L ) a"d
Hence the expected downtime is L/A(M - L). The average percent idle time of
each server is
\ rj } ( N )( AX '
100 7l0 1 + 1/c
n-0
V " J ’ J)
p d - p *)
=p
i - p k+[
Exercise 10.8 Let Xn be the number of arrivals during the service time of the
nth customer. Then in the M/G/l system we have Qn+] = Qn - H{Qn) + Xn+h
where H(x) is the heavyside’s function, given by
H(jc)= 1 if jc > 0
= 0 if jc< 0.
Solution to Exercises 409
G*(j)= f e~sxdG(x).
Jo
Let V be the service time with d.f. F(u) = P(V < v),
F*(s) = f e~svdF(v).
Jo
Let G{n) be the Ai-fold convolution of G with itself and A(t) denotes the number
of arrivals in [0, t]. By conditioning on the duration of the service time V of the
initial customer initiating the busy period and on the number A of arrivals during
his service tipie, we get
G(x) =e~**V(x) 4 £
"=' Jo
f P [ T < x \ V = v , A ( v ) = n}dV(v)P{A(v) = n}.
The first term corresponds to the case that none arrives during the service
time of the initial customer (and then v = x), let TLbe the total service time of the
ith “descendent” and all its other customer initied by him. Then T,’s are i.i.d. r.v.s
with same distribution as T. We have
G * ( s ) = f esxdG(x)
Jo
and changing the order of integration (on the r.h.s.) and after simplification, we
get
G*(s) =P* (s +A - AG*(j)).
References
1. Anderson, T.W. The Statistical Analysis of Time Series, John Wiley, New York
(1971).
2. Andre, D. (1887).
3. Arthey, K.B. and Ney, P. Branching Processes, Springer-Verlag, Berlin (1972).
4. Bachliar, L. Theorie de la speculation. Ann. Sci. Ecole Norm Sup (3), 21-86
(1900).
5a. Bailey, N.T.J. The Elements of Stochastic Proceses. John Wiley, New York (1964).
5b. Bailey, N.T.J Mathematical Theory of Epidemics, Giriffin and Co., London.
6. Barlow, R.E. and Proschan, F. Statistical Theory of Relability and Life Testing,
Holt, Rinehart, New York (1975).
7. Bartlett, M.S. An introduction to stochastic process, Camb. Univ. Press, London
(1954).
8. Basu, A.K. An introduction to measure theoretic probability, Prentice Hall, New
Delhi (1999).
9. Bharucha-Ried, A.T. Elements of the theory of Markov processes and their
applications, McGraw-Hill Book Co, Inc, New York (1960).
10. Bhatt, U.N. Elements of Applied Stochastic Processes. John Wiley, New York
(1984).
11. Bhattacharyya, R.N. and Waymire, E. Stochastic Processes with Application,
John Wiley, New York (1990).
12. Billingsley, P. Convergence of Probability measure, 2nd ed, John Wiley New
York (1999).
13. Blackwell, D. Extension of renewal theorem, Pacific J. Math, 3, 315-320 (1953).
14. Blumenthal, R.M. and Gatoor, R.K. Markov Processes and Potential Theory,
Academic Press, New York (1960).
15. Bloomfield, P. Fourier Analysis of Time Series: An introduction, John Wiley,
New York (1976).
16. Bochner, S. Stochastic Processes, Ann. Math, 48, 1014-1061 (1947).
17. Box, G.E.P. and Jenkins, G.M. time series Analysis: Forecasting and control,
Holdey Dey, Sanfrancisco (1996).
18. Blanc - Lapierre, A. and Fortet, R. Theorie des Fonetions Aleatories, Masson et.
Cie, Paris (1953).
19. Brieman, L. Probability, Addison-Wesley, Reading. Massachusetts (1998).
20. Champemowne, D.C. An elementary method of solution of the Queueing problem
with a single server and a constant parameter. J.R.S.S. B, 18, 125-128(1956).
21. Chatfield, C. The analysis of Time Series: An Introduction, 5th ed. Chapman and
Hall, London (1989).
22. Christer, A.H. Refined Asymptotic costs for renewal reward processes. J. Opns.
Res. Soc., 29, 577-583 (1978).
23. Chow, Y.S. and Tiecher, H. Probability Theory, Springer Verlag, 3rd ed. (1998).
24. Chung, K.L. Markov Chains with Stationary Tranisition Probabilities, Springer
Verlag, 2nd ed. Berlin (1967).
412 Introduction to stochastic process
25. Chung, K.L. and Ornstein, D. On the Recurrence of Sums of Random Variables,
Bull. Am. Math. Soc., 5, 332-334 (1954).
26. Cox, D.R. Renewal Theory. Methuen, London (1962).
27. Cox, D.R. and Miller, H.D. The Theory of Stochastic Processes, Methuen, London
(1977).
28. Cramer, H. and Leadbetter, M.R. Stationary and Related Stochastic Processes,
John Wiley, New York (1967).
29. Doeblin, W. Sur les proprietis asymptotiques de movement rigis par certains
types des chaines simples. Bull. Math. Soc. Roum. Sci, 39, No. 1, 57-115, No.
2, 3-61 (1936).
30. Doob, J.L. Stochastic processes, John Wiley, New York (1953).
31. Doob, J.L. Renewal theory from the point of view of the theory of probability.
Trans. Amer. Math. Soc. 63, 422-438 (1948).
32. Dvoretzky, A, Erdos, P. and Katutani, S. Nonincrease everywhere of the Brownion
motion process. Proc. 4th Berkley. Symp. On Math. Stat. and Prob. vol. II, 103—
116 (1961).
33. Erdos, P, Feller, W., and Pollard, H. A Property of power series with positive
coefficients, Bull. Amer. Math. Soc. Vol-55, 201-204 (1949).
34. Fabens, A.T. The solution of Queueing and Inventory models by Semi-Markov
processes, J.R.S.S. B, 23, 113-127 (1961).
35. Feller, W. On the integral equation of renewal theory, Ann. Math. Statistics, 12,
243-267 (1941).
36. Feller, W. An introduction to probability theory and its applications, vol. I & II,
John Wiley, New York (1968, 71).
37. Fisher, R.A. Tests of significance in harmonic Analysis. Proc. Roy. Soc, Ser. A,
125, 54-59.
38. Folkes, J.L. and Chikara, R.S. The Inverse Gaussian Distribution and its Statistical
Application—a review, J.R.S.S.B. 40, 263-275(1978).
39. Foster, F.G. On Stochastic Matrices Associated with Certin Queueing Processes.
Ann. Math. Statist., 24, 355-360 (1953).
40. Fuchs, W.H.J. and Chung, K.L. On the distribution of values of sums of random
variables. Mem. Am. Math. soc. No. 6, 1-12 (1951).
41. Fuller, W.A. Introduction to Statistical Time Series, John Wiley, 2nd ed, New
York (1996).
42. Galton, F. and Watson, H.W. On the Probability of extinction of families. J. of
Anthr. Inst., 6, 138-144 (1874).
43. Gikhman, 1.1, and Skorohod, A.V. Introduction to the theory of random functions,
Saunders, Philadelphia (1969).
44. Granger, C.W.J. and Newbold, P. Forecasting Economic Time Series. Academic
Press, New York (1977).
45. Grenander, V. and Rosenblatt, M. Statistical Analysis of Stotionary Time Series.
John Wiley, New York (1957).
46. Grimmet, G. and Stirzakar, D. Probability and Random Processes, Oxford University
Press, 3rd. ed. (2001).
47. Good, I.J. The Number of Individuals in a cascade process, proc. Camb. Phil.
Soc., 45, 360-363 (1948).
48. Gross, D. and Harris, C.M. Fundamentals of Queueing theory. 2nd ed. John
Wiley, New York (1985).
49. Harris, T.E. The theory of Branching Processes, Springer-Verlag, Berlin (1963).
50. Hannan, E.J. Time Series Analysis. Methuen, London (1960).
51. Hannan, E.J. Multiple Time Series. John Wiley, New York (1970).
References 413
52. Isifescu, M. and Tautu, P. Stochastic Processes and Applications in Biology and
Medicine, I, Springer-Verlag, Berlin (1973).
53. Ito. K. Stochastic integral. Proc. Imp. Acad. Tokyo 20, 519-524 (1944).
54. Jagers, P. Branching Process with Biological Applications, John Wiley, New York
(1975).
55. Kalkutani, S. Ergodic theorems and the Markov Process with a stable distribution,
proc. Acad. Tokyo, 16, 49-54 (1940).
56. Karakowski, M. Conservation methods in queueing Theory. Rev.Fran d’auto.
Infor. Rech. Oper (R.A.l.R.O) 7e annee, vol. I, 3-20 (1973).
57. Karhunen, K. Uber Lineare Method Methoden in der Wahrschein lich Keit
srechnung. Ann. Acad. Sci. Fenn. A. 37 (1947).
58. Karlin, S. and Taylor H.M. A first course in stochastic processes, Academic
Press, New York, 2nd ed. (1975).
59. Keilson, J. and Kooharian, A. On time dependent Queueing processes. Ann.
Math. Statist, 31, 104-112 (1960).
60. Kemeny, J.G. and Snell, J.L. Finite Markov Chains, Academic Press, New York
(1966).
61. Kendall, D.G. Some Problems in the Theory of Queues, J.R.S.S. B, 13, 151-185
(1951).
62. Kendall, D.G. On the generalized birth and death processes, Ann Math, Statist.,
19, 1-15.
63. Khintchine, A.Y. Correlation theory of stationary random processes. Usp. Math.
Nauk, 5,42-51 (1938).
64. Kingman, J.F.C. The single server queue in heavy traffic, proc. Camb. Phil. Soc.,
57, 902-904 (1961).
65. Kleinrock, L. Queueing Systems, Vol. 1 & II, John Wiley, New York (1975).
66. Kolmogorov, A.N. Interpolation and extrapolation of Stationary random sequences,
IZV. Akad. Nauk. SSSR Ser. Math. 5, 3-14 (1941).
67. Kolmogorov, A.N. On the problem of differencetiability of transition probabilities
of time-homogenous Markov Processes with Countable number of states. Uchenye
Zapiski Markovskovo Gos. 148 (Mathematika 4), 53-59 (in Russian).
68. Loeve, M, Probability Theory. Vol I & II, Springer verlag New York (1982).
69. Levy, P Theorie de l ’addition des variables aleatories, Paris (1937).
70. Levy, P. processus stochastiques et mouvement Brownien, Paris (1948).
71. Lotka, J.A. Stochastic convergence. Academic Press (1937).
72. Markov, A.A. Calclus of Probability (Russian) 4th ed. Moscow (1924).
73. Medhi, J. Stochastic Processes, New Age International Pub. New Delhi, 2nd ed
(1994).
74. Miller, H.D. A Generalization of Wald’s identity with Applications to Random
Walks. Ann. Math. Statist., 32, 549-560 (1961).
75. Morse, PM. Queues, Inventories and Maintenance. John Wiley, New York (1958).
76. Neuts, M.F. A general class of bulk queues with Poisson input. Ann. Math.
Statist., 38, 759-770 ( 1967).
77. Omstein, L.S. and Uhlenbeck, G.E. on the theory of Brownian motion. Physical
Review, 36.93-112 (1930). [Reprinted in selected papers on Noise and Stochastic
Processes, (Ed. Wase, N.) Dover, New York (1954)].
78. Palm, C. (1943) Intensitatschwarkungen in Femsprechovorkehr Eriesson technics,
44, 1-189, Stocholm: Tele fonaktiebolaget I.M. Ericeson. I.M. Ericeson.
79. Parzen, E. Stochastic Processes, SIAM, Philadelphia (1999).
80. Prabhu, N.U. Queues and Inventories, John Wiley, New York (1965).
81. Prabhu, N.U. Stochastic Processes, Macmillan, New York (1965).
414 Introduction to stochastic process
82. Priestley, M.B. Spectral analysis and time Series, Vol. I & II, Academic Press,
New York (1981).
83. Quenouille, M.H. Approximate tests for correlation in time Series. J. Roy. Stat.
Soc. B, 11, 68-84 (1957).
84. Resnik, S.I. Adventures in stochastic Processes. Birkhairser, Boston (1999).
85. Rolski, T., schmidli, H., Schmidt, V., Tengels, J. Stochastic Processes for Insurance
and Finance, Wiley, New York (1998).
86. Rosenblatt, M. Some regression problems in time series Analysis. Proc. 3rd.
Berkeley. Symp. On Math. Statist. Prob., Univ. of California Press, Berkeley
165-186 (1956).
87. Ross, S. Stochastic Processes. 2nd ed. John Wiley, New York (2000).
88. Slutsky, E. Sur Les Fonctions Eventuelles Continues, Integrates et Drivables
Dans le sens stochastique. C.R. Acad. Sci. Paris, 187, 878-880 (1928).
89. Smith, W.L. Renewal Theory and its Ramifications, J. Roy. Statist. Soc., B 20,
243-302 (1958).
90. Smoluchowlski, M. V. Molekularthearetische studen uber Umkehn thermodynamisch
irreversilder vorgange and uber wideskehn abnormale zustance, S.B. Akad. Wiss
Wien (2a): 124-339 (1915).
91. Spitzer, F. Principles of random walk. D. Van Nonstrand, Co. Princeton (1964).
92. Stidham, S. A last work on L = W, Opns. Res. 22, 417-421 (1974).
93. Stratonovich, R.L. A new representation for stochastic Integrals and equation,
Bull. Moscow Univ. Math, and Mech. Ser. No: 1, 3-12 (in Russian) (1964)
Translation in J. SIAM CONTROL, 4, 362-371 (1965).
94. Takacs, L. Investigation of wating time problems by reduction to Markov processes,
Act. Math. Hung., 6, 101-129 (1955).
95. Tichmmarsh, E.C. Theory of Functions, Griffin, London (1936).
96. Wald, A. Sequential Analysis, John wiley, New York (1945).
97. Wiener, N. Extrapolation, interpolation and smoothing of stationary time series;
with Engineering Applications. Cambridge - New York (1949).
98. Wold, H. A study in the Analysis of stationary Time series, 2nd ed. Almqvist &
Wiksells, Stockholm (1954).
99. Wong, E. Stochastic Processes in Information and Dynamical systems. McGraw-
Hill, New York (1971).
100. Yule, G.D. A Mathematical Theory of Evolution, Phil. Trans. Roy. Soc., London
B, 213, 21-87.
Subject Index
ISBN cl?fl-l-fl4EtjS-lDS-b
©
Alpha Science International Ltd.
Introduction to
Stochastic Process i Introduction to
Stochastic Process
Professor A.K. Basu
Editor of Stochastic Modelling and Applications
Department of Statistics, Calcutta University
Kolkata-700 019, India
ISBN t»?fi-l-642tS-lDS-b
©
Alpha Science In