A FIRST COURSE IN STOCHASTIC PROCESSES

SECOND EDITION

SAMUEL KARLIN
STANFORD UNIVERSITY AND THE WEIZMANN INSTITUTE OF SCIENCE

HOWARD M. TAYLOR
CORNELL UNIVERSITY

ACADEMIC PRESS

New York

San Francisco

London

A Subsidiary of Harcourt Brace Jovanovich, Publishers

CONTENTS

Preface .. .. Preface to First Edition

. ..

.. ..

.. ..

.. ..

.. ..

xi xv

Chapter 1 ELEMENTS OF STOCHASTIC PROCESSES 1. Review of Basic Terminology and Properties of Random Variables and Distribution Functions .. .. .. .. 2. Two Simple Examples of Stochastic Processes .. .. 3. Classification of General Stochastic Processes .. .. 4. Denning a Stochastic Process .. .. .. .. Elementary Problems .. .. .. .. Problems .. .. .. .. .. Notes .. .. .. .. .. .. References .. .. .. .. ..

.. .. .. .. .. .. .. ..

1 20 26 32 33 36 44 44

Chapter 2 MARKOV CHAINS 1. Definitions .. .. .. 2. Examples of Markov Chains .. .. 3. Transition Probability Matrices of a Markov Chain .. .. .. .. .. .. .. .. .. 45 47 58

vi 4. 5. 6. 7. Classification of States of a Markov Chain . . Recurrence .. .. . . Examples of Recurrent Markov Chains .. More on Recurrence . . .. .. Elementary Problems .. .. Problems .. .. .. Notes .. .. .. .. References .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..

CONTENTS .. .. .. .. .. .. .. 59 62 67 72 73 77 79 80

Chapter 3 THE BASIC LIMIT THEOREM OF MARKOV CHAINS AND APPLICATIONS

1. 2. 3. 4. 5. 6. 7. Discrete Renewal Equation Proof of Theorem 1.1 Absorption Probabilities Criteria for Recurrence A Queueing Example Another Queueing Model Random Walk .. Elementary Problems Problems .. Notes .. .. Reference .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. 81 87 89 94 96 102 106 108 112 116 116

Chapter 4 CLASSICAL EXAMPLES OF CONTINUOUS TIME MARKOV CHAINS

1. 2. 3. 4. 5. 6. 7. 8. General Pure Birth Processes and Poisson Processes More about Poisson Processes .. .. A Counter Model .. .. .. Birth and Death Processes .. .. Differential Equations of Birth and Death Processes Examples of Birth and Death Processes .. Birth and Death Processes with Absorbing States Finite State Continuous Time Markov Chains Elementary Problems .. .. Problems .. .. .. Notes .. .. .. .. References .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. 117 123 128 131 135 137 145 150 152 158 165 166

CONTENTS

vii

Chapter 5 RENEWAL PROCESSES

1. 2. 3. 4. 5. 6. 7. 8. 9. Definition of a Renewal Process and Related Concepts Some Examples of Renewal Processes .. .. More on Some Special Renewal Processes . . .. Renewal Equations and the Elementary Renewal Theorem The Renewal Theorem .. .. .. Applications of the Renewal Theorem .. .. Generalizations and Variations on Renewal Processes . . More Elaborate Applications of Renewal Theory .. Superposition of Renewal Processes .. .. Elementary Problems .. .. .. Problems .. .. .. .. Reference .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. 167 170 173 181 189 192 197 212 221 228 230 237

Chapter 6 MARTINGALES
1. Preliminary Definitions and Examples .. .. .. 2. Supermartingales and Submartingales .. .. .. 3. The Optional Sampling Theorem .. .. .. 4. Some Applications of the Optional Sampling Theorem . . .. 5. Martingale Convergence Theorems .. .. .. 6. Applications and Extensions of the Martingale Convergence Theorems 7. Martingales with Respect to <r-Fields .. .. .. 8. Other Martingales .. .. .. .. .. Elementary Problems .. .. .. .. Problems .. .. .. .. .. Notes .. .. .. .. .. .x. References .. .. .. .. .. .. .. ., .. .. .. .. .. .. .. .. .. 238 248 253 263 278 287 297 313 325 330 339 339

Chapter 7 BROWNIAN MOTION

1. 2. 3. 4. 5. 6. 7. Background Material .. .. .. Joint Probabilities for Brownian Motion .. .. Continuity of Paths and the Maximum Variables .. Variations and Extensions .. .. .. Computing Some Functionals of Brownian Motion by Martingale Multidimensional Brownian Motion .. .. Brownian Paths .. .. .. .. .. .. .. .. Methods .. .. .. .. .. .. .. .. .. 340 343 345 351 357 365 371

viii
Elementary Problems Problems .. Notes .. .. References . . .. .. .. . . .. .. ... .. .. .. .. .. .. .. .. ..

CONTENTS
.. .. .. .. 383 386 391 391

Chapter 8 BRANCHING PROCESSES

1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11.

Discrete Time Branching Processes .. .. Generating Function Relations for Branching Processes Extinction Probabilities .. .. .. Examples .. .. .. .. Two-Type Branching Processes .. .. .. Multi-Type Branching Processes .. .. Continuous Time Branching Processes .. .. Extinction Probabilities for Continuous Time Branching Processes Limit Theorems for Continuous Time Branching Processes Two-Type Continuous Time Branching Process .. Branching Processes with General Variable Lifetime .. Elementary Problems .. .. .. Problems .. .. .. .. Notes .. .. .. .. .. Reference .. .. .. ..

.. .. .. .. .. .. .. .. .. .. .. .. .. .. ..

392 394 396 400 404 411 412 416 419 424 431 436 . . 4 3 8 .. 442 .. 442

.. .. .. .. .. .. .. .. .. .. .. ..

Chapter 9 STATIONARY PROCESSES

1. 2. 3. 4. 5. 6. 7. 8. 9. 10. Definitions and Examples .. .. Mean Square Distance .. .. Mean Square Error Prediction .. .. Prediction of Covariance Stationary Processes Ergodic Theory and Stationary Processes .. Applications of Ergodic Theory .. Spectral Analysis of Covariance Stationary Processes Gaussian Systems .. .. .. Stationary Point Processes .. .. The Level-Crossing Problem .. .. Elementary Problems .. .. Problems .. .. .. Notes .. .. .. .. References .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . .. .. .. .. .. 443 451 461 470 474 489 502 510 516 519 524 527 534 535

CONTENTS

ix

Appendix REVIEW OF MATRIX ANALYSIS 1. The Spectral Theorem .. .. 2. The Frobenius Theory of Positive Matrices . . .. .. .. .. .. .. 536 542

Index

..

..

..

..

553