2.8. Stochastic Processes A stochastic process {X(t),t T } is a collection of random variables.

That is, for each

t T, X(t) is a random variable. The index t is often interpreted as time and, as a result, we refer to X(t) as the state of the process at time t . For example, X(t)might equal the total number of customers that have entered a supermarket by time t or the number of customers in the supermarket at time t ; or the total amount of sales that have been recorded in the market by time t; etc.
The set T is called the index set of the process. When T is a countable set the stochastic process is said to be a discrete-time process. If T is an interval of the real line, the stochastic process is said to be a continuous-time process. For instance, {Xn, n = 0, 1, . . .} is a discrete-time stochastic process indexed by the nonnegative integers; while {X(t), t > 0} is a continuous-time stochastic process indexed by the nonnegative real numbers. The state space of a stochastic process is defined as the set of all possible values that the random variables X(t) can assume. Thus, a stochastic process is a family of random variables that describes the evolution through time of some (physical) process. One can see much of stochastic processes in the Sec 2.8, Introduction to probability Models, Sheldon Ross.

1. Three white and three black balls are distributed in two urns in such a way that each contains three balls. We say that the system is in state i, i = 0, 1, 2, 3, if the first urn contains i white balls. At each step, we draw one ball from each urn and place the ball drawn from the first urn into the second, and conversely with the ball from the second urn. Let Xn denote the state of the system after the nth step. Explain why {Xn, n = 0, 1, 2,...} is a Markov chain and calculate its transition probability matrix. 2. In the Dark Ages, Harvard, Dartmouth, and Yale admitted only male students. Assume that, at
that time, 80 percent of the sons of Harvard men went to Harvard and the rest went to Yale, 40 percent of the sons of Yale men went to Yale, and the rest split evenly between Harvard and Dartmouth; and of the sons of Dartmouth men, 70 percent went to Dartmouth, 20 percent to Harvard, and 10 percent to Yale. We form a Markov chain with transition matrix

3. Modify Example 11.6 by assuming that the son of a Harvard man always went to Harvard. The transition matrix is now

4. (Ehrenfest Model) The following is a special case of a model, called the Ehrenfest model,that has been used to explain diffusion of gases. We have two urns that, between them, contain four balls. At each step, one of the four balls is chosen at random and moved from the urn that it is in into the other urn. We choose, as states, the number of balls in the First urn. The transition matrix is then

5. 6. 7. 8. 9. 10. 11. 1. 2. 3. 4. 5. 6. 7.