Академический Документы
Профессиональный Документы
Культура Документы
Tim Carroll
Sheiva Sajadpour
Jesus (Jesse) Gonzalez
19 February 2012
Using the textbook by Wayne Winston, Operations Research Applications, explain Chapter
17.6, learn the examples, and perform problems 4 and 5.
Table of Contents
Introduction to Absorbing Chains:........................................................................................ 3
Absorbing Markov Chains.............................................................................................. 4
Canonical Form............................................................................................................ 5
The Fundamental Matrix................................................................................................. 6
Absorption Probabilities................................................................................................. 6
Input Output Chart:....................................................................................................... 7
Properties of Transient and Absorbing states........................................................................8
Example 1: An Accounting Firm.......................................................................................... 8
Problem Statement........................................................................................................ 8
Assumptions.............................................................................................................. 10
Solution................................................................................................................... 10
Summary:................................................................................................................. 12
Example 2: Law Firm..................................................................................................... 14
Problem Statement...................................................................................................... 14
Assumptions.............................................................................................................. 14
Solution................................................................................................................... 15
Summary:................................................................................................................. 16
Problem 4: Telemarketing................................................................................................ 18
Problem Statement...................................................................................................... 18
Assumptions.............................................................................................................. 19
Solution................................................................................................................... 19
Answers................................................................................................................... 20
Problem 5: Machine Production......................................................................................... 23
Problem Statement...................................................................................................... 23
Assumptions:............................................................................................................. 24
Solution:.................................................................................................................. 24
Answers................................................................................................................... 26
Appendix A: Complete Transition Matrix Tables....................................................................28
Appendix A-1: Accounting Firm..................................................................................... 28
Appendix A-2: Law Firm.............................................................................................. 29
Appendix A-3: Telemarketing........................................................................................ 30
1
Appendix A-4: Machine Production.................................................................................31
Appendix B: Matrix and Gauss - Jordan Brief Review.............................................................32
Appendix B-1: Matrix Addition, Subtraction, and Multiplication.............................................32
Appendix B-2: Using Gauss to find the Inverse...................................................................33
Appendix C: Work Shown from all Problems........................................................................34
2
Parisays comments in red
In different areas of our everyday life such as stock market, finance, accounting, education and
marketing we might need to understand how a random variable changes over time. This study
Markov chain is a discrete-time process. This means assuming we have a set of states:
The process can start in one of these states and move to another state. Each move is called a step.
Each step has a probability of its own. If the chain is currently in state si, then it moves to state sj
at the next step with a probability shown by pij , and this probability does not depend upon
which states the chain was in before the current. The probabilities pij are called transition
probabilities. The probabilities can be shown in a matrix called transition matrix. In the P
transition matrix below pij is the probability of being in state Si at step n + 1 given that the
3
p11
p12 p1 r p 21
p22 p 2 r ..
P= ..
pr 1
[ pr 2 prr ]
Also, Qi is the probability of the chain at time 0. The entries in the matrix P are nonnegative and
sum of each row equals to 1. For a Markov chain with a transition matrix P, pnij gives the
probability that the Markov chain, starting in state si, will be in state sj after n step.
A state Si of a Markov chain is called an absorbing state if, once the Markov chains enter the
state, it is impossible to leave that state. Therefore the probability of leaving that state would be
4
pii =1
zero and it is shown as . A Markov chain is absorbing if it has at least one absorbing
In an absorbing Markov chain, a state which is not absorbing is called transient. In the
hypothetical example below, 0 and 4 are in the absorbing state with the probability of 1.
In an absorbing Markov chain if the beginning state is absorbing, we will stay at that state and if
Canonical Form
Transition matrix of an absorbing Markov chain follows a Canonical form, which means that the
5
I: is an r-by-r identity matrix
R: is a nonzero t-by-r matrix, giving transition probabilities from transient to absorbing states
Theorem: For an absorbing Markov chain the matrix I - Q has an inverse N and N = I + Q + Q^2
1
For an absorbing Markov chain P, the matrix N = ( I Q) is called the fundamental matrix
for P. The entry nij of N gives the expected number of times that the process is in the transient
In practicality, this matrix contains actual values such as time, or money, as shown in Example 1
and 2 respectfully.
Absorption Probabilities
If B is a t-by-r matrix with entries bij and bij is the probability that an absorbing chain will be
absorbed in the absorbing state sj if it starts in the transient state si. Then:
B = NR
In practicality, these do not have any units as they are the probabilities that any given transition
6
Input Output Chart:
The following chart shows the basic needs to perform calculations for an Absorbing Chain:
The inputs that one requires before performing any calculations are shown on the left side. One
can find these inputs as stated above, or as shown in the following the examples. (Appendix A
The outputs one is required to find is the Matrix I-Q, the inverse of such matrix, and
As stated before, Absorbing states cannot revert to any other state at any time. This means that
the chance of the state remaining the same is equal to 1, or 100%. While this is the only true
1. Transient States may only Progress forward until reaching an Absorbing State
2. Transitent States may skip other transient states and go directly into an Absorbing State
3. Transient states may revert back to previous transient states before reaching Absorbing
7
4. Transient States have the chance to stay the same, however, the chance of this being true
We will now explore this subject in more practicality by going over several examples in detail.
Problem Statement
Suppose a firm assumes that an account is uncollectable if the account is more than three
months overdue. Then at the beginning of each month, each account may be classified into one
8
For example, if an account is two months overdue at the beginning of a month, there is a 40%
chance that at the beginning of next month, the account will not be paid up (and therefore be
three months overdue) and a 60% chance that the account will be paid up. To simplify our
example, we assume that after three months, a debt is either collected or written off as a bad debt.
Once a debt is paid up or written off as a bad debt, the account is closed, and no further
debt?
3. If the firms sales average $100,000 per month, how much money per year will go
uncollected?
Assumptions
All New debts move forward as they grow older and cannot backtrack. For example, a
New Debt becomes One Month Overdue but cannot become a New Debt again.
Once a debt has been paid off, any new debt by the same customer is listed as New
Debt.
Once a debt is written off as a Bad Debt, the debt cannot be paid by the customer.
Once a debt is paid or deemed bad, the account closes and nothing else can happen.
Bad Debt and Paid are absorbing states
New Debt and the months overdue until the third month are transient states.
Solution
1. From the problem statement, we can draw a Transition Diagram, shown in figure 1:
9
Figure 1: Shows the transition diagram along with the likeliness of that transient state
going to the next until the absorbing stages are reached at the far left.
2. We defined four transient states (New, 1 Month, 2 Months, and 3 Months), and
two absorbing states (Paid and Bad Debt). The four transient states correspond to t1,
t2, t3, and t4 respectfully. Also, the absorbing states correspond to a1 and a2.
3. We can then make a Transition Matrix as a second means of outlining the given
[ ]
0 .6 0 0 .4 0
0 0 .5 0 .5 0
0 0 0 .4 .6 0
P=
0 0 0 0 .7 .3
0 0 0 0 1 0
0 0 0 0 0 1
[ ]
0 .6 0 0 .4 0
0 0 .5 0 .5 0
0 0 0 .4 .6 0
P=
0 0 0 0 .7 .3
0 0 0 0 1 0
0 0 0 0 0 1
10
4. From the Transition Matrix we can find the Q, R, I and 0 matrices.
[ ] [ ]
0 .6 0 0 .4 0
Q=
0 0 .5 0
0 0 0 .4 and R=
.5 0
.6 0 and [ ]
I= 1 0
0 1
0 0 0 0 .7 .3
5. We then subtract the Q matrix from the I matrix, or the Identity Matrix, and then
[ ]
1 .6 0 0
0 1 .5 0
I Q=
0 0 1 .4
0 0 0 1
Then
[ ]
1 .60 .30 .12
1 0 1 .50 .20
(I Q) =
0 0 1 .40
0 0 0 1
6. We then multiply that matrix by the R matrix to find the final matrix
[ ]
.964 .036
1 .940 .060
( I Q) R=
.880 .120
.700 .300
11
Summary:
absorbed in absorbing state aj is the ijth element of the matrix (I-Q)-1R) Since we want to
know the likeliness of a new debt is paid before the next month, we can take the transient
period, New which is t1, and the absorbing state, Paid which is a1, and see that :
t1 and a1 => 11th (ijth) element
If we look at Figure 1.b , we can see that the <1,1> (in terms <i,j> from the top-left) is
equal to .964
[ ]
.964 .036
( I Q) R= .940
1 .060
.880 .120
.700 .300
Figure 1.b
2. What is the probability that a one-month-overdue account will eventually become a bad
debt?
Recall that we defined 1 Month as t2, and Bad Debt as a2. Following the same
method above and referring to the previous figure 1.b we see that:
1
<2,2> of ( I Q) R = .060
3. If the firms sales average $100,000 per month, how much money per year will go
uncollected?
From question 1, if .964 of debts are collected before the first month, we can say that:
1 - .964= .036
12
This means that 3.6% of all debts are uncollected. Since the firm averages $100,000
($100,000/month)(12Months/year) = 1,200,000/year
Then
The next example we will go over involves a chance that the transient state will be unchanged as
Transient States have the chance to stay the same, however, the chance of this being true is
Problem Statement
The law firm of Mason and Burger employs three types of lawyers: junior lawyers, senior
lawyers, and partners. During a given year, there is a .15 probability that a junior lawyer will be
promoted to senior lawyer and a .05 probability that he or she will leave the firm. Also, there is
a .20 probability that a senior lawyer will be promoted to partner and a .10 probability that he or
she will leave the firm. There is a .05 probability that a partner will leave the firm. The firm
13
1. What is the average length of time that a newly hired junior lawyer spends working for
the firm?
2. What is the probability that a junior lawyer makes it to partner?
3. What is the average length of time that a partner spends with the firm (as a partner)?
Assumptions
Solution
Figure X\2: Unlike the problem before, we see that each transient period has a probability
to reoccur and go to the next state. The absorbing states, on the left, have 100%, or 1, of
reoccurring again.
2. Again, we will define all the transient states, Junior, Senior, and Partner as, t1, t2,
and t3 respectfully. The absorbing states, Leave N Partner and Leave as Partner will
[ ]
.80 .15 0 .05 0
0 .70 .20 .10 0
P= 0 0 .95 0 .05
0 0 0 1 0
0 0 0 0 1
14
4. From the Transition Matrix we can find the Q, R, I and 0 matrices.
[ ] [ ]
.80 .15 0 .05 0
Q= 0 .70 .20
0 0 .95
And R= .10 0
0 .05
And [ ]
I= 1 0
0 1
5. Subtract the Q matrix from the I matrix, or the Identity Matrix, and then take the
[ ]
.20 .15 0
I Q= 0 .30 .20
0 0 .05
Then
[ ]
5 2.5 10
1
( I Q) = 0 10/3 40/3
0 0 20
6. Finally, multiply that matrix by the R matrix to find the final matrix
[ ]
.50 .50
( I Q)1 R= 1/3 2/3
0 1
Summary:
1. What is the average length of time that a newly hired junior lawyer spends working for
the firm?
15
Recall from statistics that the average is also known as expected value, hence,
Expected time = the sum of the time a junior lawyer stays with the firm, even as he or she
promotes.
[ ]
5 2.5 10
1
( I Q) = 0 10/3 40/3
0 0 20
Figure 2b.
The top left row, <1,1> through <1,3>, are 5, 2.5, and 10. Hence:
3. What is the average length of time that a partner spends with the firm (as a partner)?
Once again we are dealing with time in a certain transient period, hence we look at the figure
from part 1. Partner is t3, and we want to know the time spent as partner. This is:
<3,3> =20years.
Now we will explore some other cases that involve transition states that can go between
16
Problem 4: Telemarketing
Problem Statement
Absorbing Markov chains are used in marketing to model the probability that a customer who is
contacted by telephone will eventually buy a product. Consider a prospective customer who has
never been called about purchasing a product. After one call, there is a 60% chance that the
customer will express a low degree of interest in the product, a 30% chance of a high degree of
interest, and a 10% chance the customer will be deleted from the companys list of prospective
customers. Consider a customer who currently expresses a low degree of interest in the product.
After another call, there is a 30% chance that the customer will purchase the product; a 20%
chance the person will be deleted from the list, a 30% chance that the customer will still possess
a low degree of interest, and a 20% chance that the customer will express a high degree of
interest. Consider a customer who currently expresses a high degree of interest in the product.
After another call, there is a 50% chance that the customer will have purchased the product, a
40% chance that the customer will still have a high degree of interest, and a 10% chance that the
a. What is the probability that a new prospective customer will eventually purchase the product?
b. What is the probability that a low-interest prospective customer will ever be deleted from the
list?
17
c. On the average, how many times will a new prospective customer be called before either
Assumptions
All given information (percentages, level of interest, etc.) is correct and accurate.
The customers level of interest can truly be categorized into low or high levels.
Any customer that is deleted remains deleted forever.
Once the product is purchased, the customer is no longer called.
Solution
1. Our first step to solving this problem is drawing out a Transition Diagram.
Figure 3: Now the transition diagram features paths that can revert to previous states.
2. We will follow the same steps as the previous problems and define three transient states
(new prospective, low interest and high interest customers), and two absorbing states
(deletion, purchase). As before the transient states are t1, t2, and t3. The absorbing states
[ ] [ ] [ ]
0 .6 .3 0 .1 1 0 0
Q= 0 .3 .2 R= .3 .2 I= 0 1 0
0 .1 .4 .5 0 0 0 1
5. We then subtract the Q matrix from the I matrix and then take the inverse of that
[ ]
1 .975 .825
[ ]
1 .6 .3 1
I Q= 0 .7 .2 I Q = 0 1.5 .5
0 .1 .6 0 .25 1.75
6. We then multiply that matrix by the R matrix to find the final matrix.
[ ]
.705 .295
I Q 1 R= .7 .3
.95 .05
Answers
a. What is the probability that a new prospective customer will eventually purchase the product?
19
For this answer we know that:
Thus, the probability that a new customer will eventually purchase the product is element 11 of
(I-Q)-1R matrix:
<1,1> =.705
b. What is the probability that a low-interest prospective customer will ever be deleted from the
list?
So, we know the probability that a low-interest prospective customer will ever be deleted from
<2,2>=.3 or 30%
c. On the average, how many times will a new prospective customer be called before either
20
We know that the average number of times a new prospective customer will be called
before either purchasing the product or being deleted from the list is equal to the summation of
all the expected times for the transient states starting at t1. In other words:
= (I-Q)-112=.975+
= (I-Q)-113=.825
Therefore, the total expected number of calls a new customer receives before being absorbed is
21
Problem 5: Machine Production
Problem Statement
Each week, the number of acceptable-quality units of a drug that are processed by a machine is
observed: 100, 50100, 150, 0 (indicating that the machine was broken during the week). Given
last weeks observation, the probability distribution of next weeks observation is as follows,
figure 4.
>100 50 to 100 1 to 50 0
>100 0.8 0.1 0.05 0.05
50 to 100 0.1 0.6 0.1 0.2
1 to 50 0.1 0.1 0.5 0.3
0 0 0 0 1
For example, if we observe a week in which more than 100 units are produced, then there is a .10
chance that during the next week 50100 units are produced.
a Suppose last week the machine produced 200 units. On average, how many weeks will elapse
b Suppose last week the machine produced 50 units. On average, how many weeks will elapse
Assumptions:
All given information (percentages, output levels, etc.) is correct and accurate.
Broken machines will remain broken for the remainder of the week
22
Solution:
1. From the problem statement, we can draw a Transition Diagram, shown in figure 5.
Figure 5: in this diagram the transitions can skip states back and forth until reaching the
absorbing state
2. This problem only has three transient states and one absorbing state. Again, we will
define the three transient states as t1, t2, and t3 for >100, 50-100, and 1-50
[ ]
.8 .1 .05 .05
.1 .6 .1 .2
P=
.1 .1 .5 .3
0 0 0 1
4. The matrix is smaller than usual, however, this does not change the process, we can find
23
[ ] [] [ ]
.8 .1 .05 .05 1 0 0
Q= .1 .6 .1 R= .2 I= 0 1 0
.1 .1 .5 .3 0 0 1
5. We then subtract the Q matrix from the I matrix, or the Identity Matrix, and then
[ ]
6.441 1.864 1.017
[ ]
.2 .1 .05 1
I Q= .1 .4 I Q = 2.039 3.221 .847
.1
.1 .1 .5 1.695 1.017 2.373
Note: since we are only interested in the time that the machine will spend in that transient period,
24
Answers
a Suppose last week the machine produced 200 units. On average, how many weeks will elapse
We know that the average number of weeks elapsed before the machine breaks down is
equal to the summation of all the expected times for the transient states starting at t1 or over 100
units of production.
In other words:
Expected number of weeks elapsed while producing >100 units = (I-Q)-111=6.441+ Expected
number of weeks elapsed while producing 50 to 100 units = (I-Q)-112=1.864+ Expected number
Therefore, the total expected number of calls a new customer receives before being absorbed is:
b Suppose last week the machine produced 50 units. On average, how many weeks will elapse
We know that the average number of weeks elapsed before the machine breaks down is
equal to the summation of all the expected times for the transient states starting at t3 or up to 50
units of production.
In other words:
25
Expected number of weeks elapsed while producing >100 units = (I-Q)-131=1.695+ Expected
number of weeks elapsed while producing 50 to 100 units = (I-Q)-132=1.017+ Expected number
Therefore, the total expected number of calls a new customer receives before being absorbed is
26
Appendix A: Complete Transition Matrix Tables
This appendix is meant for detailed Transition Matrices and is meant to be used as a visual on the
Q, R, I, and 0 sub-matrices.
1 2 3
1 2 3
For the purpose of simplification and communication, we have substituted the actual name for
J Junior
S Senior
P Partner
LNP Leave as Non Partner
LAP Leave as Partner
J S P LNP LAP
J 0.8 0.15 0 0.05 0
S 0 0.7 0.2 0.1 0
P 0 0 0.95 0 0.05
LNP 0 0 0 1 0
LAP 0 0 0 0 1
J S P LNP LAP
J 0.8 0.15 0 0.05 0
S 0 0.7 0.2 0.1 0
P 0 0 0.95 0 0.05
LNP 0 0 0 1 0
LAP 0 0 0 0 1
For the purposes of simplification and communications, the following aliases have been added:
N Never Called
28
L Low interest
H - High Interest
P Purchased
D Deleted
N L H P D
N 0 0.6 0.3 0 0.1
L 0 0.3 0.2 0.3 0.2
H 0 0.1 0.4 0.5 0
P 0 0 0 1 0
D 0 0 0 0 1
N L H P D
N 0 0.6 0.3 0 0.1
L 0 0.3 0.2 0.3 0.2
H 0 0.1 0.4 0.5 0
P 0 0 0 1 0
D 0 0 0 0 1
50 to
>100 1 to 50 0
100
>100 0.8 0.1 0.05 0.05
50 to
0.1 0.6 0.1 0.2
100
1 to 50 0.1 0.1 0.5 0.3
0 0 0 0 1
>100 50 to 1 to 50 0
29
100
>100 0.8 0.1 0.05 0.05
50 to
0.1 0.6 0.1 0.2
100
1 to 50 0.1 0.1 0.5 0.3
0 0 0 0 1
This is a quick review of the math that is required to work the problems in this report. This will
not go into deep detail, and an appropriate Math lecture should be consulted for further
1. In order to perform a Matrix addition or subtraction, we must have a matrix of the same
dimensions.
c d[ ] [ ]
A= a b B= e f
g h
[
C= a+e b+ f
c + g d+ h ]
The same rules follow for subtraction
2. For Multiplication of two matrices, the matrix being multiplied must have the same
the same method, however, if A remained the same and B added another column or row,
it would be impractical.
Using the Gauss Jordan method is one of many ways to find the inverse of a Matrix.
We must use a Matrix in question, A, and the Identity matrix, I, such that:
[ ] [ ]
A= 4 3 I = 1 0
0 2 0 1
Note that I will be a pattern following a slanted line of ones no matter the size.
A= [ 4 3 1 0
0 2 0 1 ]
Then we can perform any operation to each individual row in the left and right sides as shown:
31
So
[
A1= 1 0 2 3
0 1 0 .5 ]
Which is the inverse.
A similar approaches and operations are used for all problems, as such; we will only show the
resource that assisted us in finding the inverse without having to use the full Gauss Jordan
method.
C-1
[ ]
0 .6 0 0
Q= 0 0 .5 0 I = 1 0
0 0 0 .4 0 1 [ ]
0 0 0 0
Since I is an identity matrix, more rows and columns can be added to match
dimensions:
New I:
32
[ ]
1 0 0 0
I= 0 1 0 0
0 0 1 0
0 0 0 1
Then I
[ ][ ]
1 0 0 0 0 .6 0 0
0 1 0 0 0 0 .5 0
I Q=
0 0 1 0 0 0 0 .4
0 0 0 1 0 0 0 0
[ ]
1 .6 0 0
0 1 .5 0
I Q=
0 0 1 .4
0 0 0 1
[ ]
1 .6 0 0 1 0 0 0
0 1 .5 0 0 1 0 0
0 0 1 .4 0 0 1 0
0 0 0 1 0 0 0 1
The above is how one would set up the problem using the Gauss Jordan method, however, we
program such as MATLAB to find and perform all calculations please refer to a MatLab video or
33
[ ]
1 .60 .30 .12
(I Q)1 = 0 1 .50 .20
0 0 1 .40
0 0 0 1
Then multiply by R by adding the products of rows and columns (see Appendix B):
[ ][ ][ ]
1 .60 .30 .12 .4 0 .964 .036
0 1 .50 .20 .5 0 .940 .060
x =
0 0 1 .40 .6 0 .880 .120
0 0 0 1 .7 .3 .700 .300
34