The effectiveness of risk management: measuring
what didn't happen
Keywords
Risk management, Measurement,
Project management
Abstract
Two of the most common reasons slippage on large software engineering
for not implementing a risk
management program are cost
and benefit. This paper focuses
on whether the benefits of
intervention can be shown to
justify the costs. A confounding
factor is that the acts of
intervention during a risk
management program may alter
the outcome in ways we cannot
separate and therefore cannot
cost out. A second confounding
factor is response bias ± the
tendency of individuals
consistently to underestimate or process of analyzing and addressing risk. The
overestimate risk, resulting in
interventions that may be
ineffective or excessively
wasteful. The authors
demonstrate that signal detection phase, risks are analyzed for likelihood and
theory (SDT) can be used to
analyze data collected during a
risk management program to
disambiguate the confounding
effects of intervention and
response bias. SDT can produce
an unbiased estimate of percent
correct for a risk management
program. Furthermore, this
unbiased estimator allows
comparison of results from one
program to another.
Management Decision
38/4 [2000] 293±300
# MCB University Press
[ISSN 0025-1747]
John F. McGrew
Pacific Bell, San Ramon, California, USA
John G. Bilotta
Charles Schwab & Co., Inc., San Francisco, California, USA
Introduction
Development of software is an inherently
risky business. The probability of schedule
projects is nearly 100 per cent (Jones, 1994).
The probability of cost overruns exceeds 50
per cent, while the probability of outright
failure is about 10 per cent (Jones, 1994). One
in ten large system efforts in the USA is
never finished or delivered (Putnam and
Myers, 1992). Any effort on the scale of
software development involving significant
risk factors requires that risks be actively
managed if the effort is to succeed.
Project risk management is the overall
process has three major components:
assessment, mitigation, and contingency
planning (Charette, 1989). In the assessment
impact on project objectives. The mitigation
phase generates action plans to minimize the
risks. As the project progresses, the
effectiveness of the mitigation plans is
reviewed and adjustments are made as
necessary. Finally, contingency plans are
developed to offset the consequences of
failure should the risk mitigation plans fail
(Bilotta, 1995). An effective program of risk
management is an ongoing process of
assessment, intervention and fallback
planning (Boehm, 1989). Yet the
implementation of risk management
programs on software development projects
is rare in commercial business (Jones, 1994).
The reasons for not implementing risk
management include cost, benefits, and
expertise. The most common rationalizations
are that the project is too small (or too large)
to justify the time and expense of a review;
that the benefits cannot be determined and,
therefore, the costs are assumed to outweigh
The current issue and full text archive of this journal is available at
http://www.emerald-library.com
the benefits; and that the effort is unlikely to
uncover anything that is not already well
known to everyone involved in the project.
Nearly every project risk is known to at least
one person on the team. Success results from
assuring that someone owns the
responsibility for addressing each risk. The
issue is to identify, document, and manage
risks ± not just to know about them. This
paper is focused on the first two concerns
which come down to whether the benefits of
risk management justify the costs ± a
question that can only be answered if we can
measure the effectiveness of a risk
management program.
It is difficult to argue against the objection
that the benefits of risk management cannot
be demonstrated. If a team assesses the risks
associated with a project and intervenes to
minimize them and the project succeeds, was
the intervention program successful ± or
would the project have succeeded anyway?
The act of intervention may have altered the
outcome in ways we cannot separate out ± or
it may not have altered it at all. The act of
intervention confounds any effort to quantify
the effectiveness of a risk management
program.
A second confounding factor is response
bias. We can best understand the impact of
response bias by examining two extreme
hypothetical cases: Projects A and B. The
managers of Projects A and B are both
anxious to minimize their risks. They each
hold risk assessment and planning sessions
to identify risks and to create risk mitigation
and contingency plans.
The team members of Project A are very
confident about their ability to deliver on all
tasks, they see little or no risks associated
with any of their commitments. Rather than
assessing each activity objectively by
reference to its actual risks, the team is nay-
saying, that is, minimizing the assessment of
risk across the board. Consequently, their
analysis generates few, if any, mitigation or
contingency plans. The result is that, as
[ 293 ]
John F. McGrew and problems do arise, the impacts are greater releases. The measure of effectiveness is
John G. Bilotta than expected and the team is unprepared to based on an analysis that contrasts the
The effectiveness of risk deal with them. In contrast, the team results of the risk assessment, mitigation,
management: measuring what
didn't happen members of Project B see significant and contingency planning efforts with the
Management Decision problems hidden in every task, even the actual outcome of the project in terms of the
38/4 [2000] 293±300 simplest routine work. They err on the side tasks that failed and the tasks that did not.
of yea-saying. In their minds, every task is There are two prerequisites for such an
overflowing with risk that must be analysis. The first is that the risk assessment,
controlled. As a result, their analysis mitigation, and contingency data be tracked.
generates dozens or hundreds of action items The second is that an independent audit or
for staff members to implement at post-implementation review of the project be
considerable cost in terms of people and conducted in a way that allows the actual
resources with, proportionately, little in the outcome of each task (in terms of its degree of
way of payback. failure) to be assessed. This information can
Assuming that both projects have then be organized into two contingency
moderate, and roughly equal, amounts of tables (examples with hypothetical data for
risk, it is not too difficult to imagine their an imaginary project are provided by Tables
respective outcomes. Project A demonstrated I and II).
a strong negative response bias by The rows of Table I separate the tasks into
consistently underestimating the risk two groups: those which the team estimated
associated with the project and, to be of high risk and those which it
consequently, underpreparing for the estimated to be of low risk. The rows of Table
consequences. Project B, on the other hand, II also separate the tasks into two groups:
demonstrated a strong positive response bias those which the team judged to require
by overestimating the risks and, therefore, intervention and those which required no
overpreparing for the consequences. intervention. Similarly, the columns in both
What will the outcome be? Project A, which Tables I and II separate the tasks into two
underestimated the risks, will experience groups: those found, after completion of the
significant problems in delivering its product project, to have failed and those found not to
and will dismiss risk assessment as have failed.
ineffective because it did not prevent the If we look at the second column of each
problems from occurring. Project B, which table (Tables I and II), we can see the nature
overestimated the risk, will likely avoid most of the analytical problem facing us. At the
of the real problems in delivering its product end of the project, 36 of the tasks were
and all of the imaginary ones as well. It is classified as having been completed
likely that it too will dismiss the value of risk successfully, that is, they did not fail. Of
management since it was so costly in terms of
people and resources yet the project was Table I
successful anyway. Contingency table for the outcome of a
Ideally, we want to optimize the value of a hypothetical risk assessment program
risk management program. To achieve that
goal, we must ask whether it is possible,
Tasks Tasks
that that did Row
based on an evaluation of a team's risk
failed not fail totals
management efforts relative to the actual
outcome of the project, to determine just how Tasks estimated 21 4 25
skillful the team was in correctly identifying to be high risk
significant risks, dismissing insignificant Tasks estimated 9 32 41
risks, and intervening to head off risks. In to be low risk
short, can we, from the data, determine a Column totals 30 36 66
team's skill in risk assessment and risk
mitigation while controlling for response
bias? Can we provide an unbiased estimate of Table II
the effectiveness of risk management? Put Contingency table for the outcome of a
another way, can we measure what we may hypothetical risk intervention program
have caused to not happen? Tasks Tasks
This paper discusses techniques for that that did Row
measuring and understanding the failed not fail totals
effectiveness of risk management programs
while controlling for response bias. Our Tasks intervened 19 12 31
research data focus on risk programs that Tasks not 11 24 35
were implemented for two software intervened
development teams over several product Column totals 30 36 66

[ 294 ]
John F. McGrew and those, 32 were judged to be of low risk (cf., the
John G. Bilotta lower right hand cell of Table I) so no
The effectiveness of risk intervention was planned for 24 of them (cf.,
management: measuring what
didn't happen the same cell of Table II). On the other hand,
Management Decision the team did intervene in the four tasks
38/4 [2000] 293±300 judged to be of high risk, taking steps, either
through risk mitigation or contingency
planning, to minimize the risk. The core
analytical problem is that the team
intervened on behalf of four tasks based on
its estimate of the risk inherent in each task.
Consequently, it is not possible to say how
many of those four tasks succeeded because
the team intervened as opposed to the
number that would have succeeded even if
the team had not intervened. After all, it also
intervened on eight other low risk tasks that
were successful (the team intervened in a
total of 12 tasks as shown in Table II, but only
four of the 12 tasks were assessed to be of
high risk in Table I). This is an instance of a
positive response bias, that is, seeing
problems where they don't exist.
In the same way, if we look at the first
column of each table, we see that the team
estimated 21 of the 30 tasks that failed to be of
high risk but intervened on only 19 ± yet all
30 tasks failed. Is this a skill problem on the
part of the team in intervening on the correct
tasks, or is it an example of a negative
response bias? If an intervened task fails, did
the intervention prevent a worse failure or
did it actually cause the failure? If the task
succeeds, did the intervention cause the
success or would it have succeeded anyway?
In other words, we need a method to
disambiguate the interaction of risk and
intervention. We need a tool to extract from
the risk assessment and intervention
contingency tables any bias toward yea-
saying or nay-saying if we are accurately to
estimate the effectiveness of the overall risk
management program.
At the end of any risk management
program, a project team will be unable to say
much about its effectiveness. It will know
how many tasks failed and how many did not
± they will have some estimate of its overall
success rate ± but it will not know whether its
intervention had an impact on the tasks that
succeeded or whether their success or failure
was owing to an overly positive or negative
response bias. This paper will show that,
using the tools of signal detection analysis,
the skill of the team, that is, the effectiveness
of the team's intervention program in
mitigating and controlling risk, can be
estimated while controlling for response
bias. Understanding response bias provides
the means to assess, and through repeated
use, to optimize the value of a risk
management program.
Applying signal detection theory to
the problem
Signal detection theory (SDT) was developed
in the communications industry by Abraham
Wald (Swets, 1996) to provide a tool for
determining whether a signal could be
accurately separated from background noise.
The technique was adapted by Swets and
Tanner (Swets, 1996) for use in psychophysics
to separate a subject's skill from the response
bias when detecting a signal. Since that time,
SDT has been used extensively to describe
the interaction of a person's sensitivity and
bias in perceptual, auditory, and other
sensory tasks (Macmillan and Creelman,
1991; Swets, 1996). It has also been extended
beyond sensory discrimination tasks to
describe the interaction between a person's
understanding of the world and the world as
it actually exists (McGrew and Jones, 1976;
McGrew, 1983, 1994). Used in this way, it
provides a method to separate an individual's
ability to understand the realities of a
situation from his/her biases about the
situation. In the context of risk management,
it can separate a team's ability to accurately
detect and intervene in risks from its
tendency to nay-say or yea-say.
The advantage of SDT over traditional
inferential and descriptive statistics is that,
instead of comparing a sample distribution to
a theoretical distribution, it compares two
sample distributions directly. The ability to
work directly with sample distributions
makes SDT an ideal tool for assessing the
success or failure of a risk management
program because there is no theoretical
distribution for risk. The comparison of
sample distributions is configured as a
contingency table. The contingency table
contrasts the perception of the world with the
world as it really is. Table III shows the
contingency table layout for a typical SDT
analysis. In Table III, A and B represent the
column marginals which sum to 1; C and D
represent the row marginals which do not
necessarily sum to 1.
The row labels in Table III (event reported
and event not reported) represent a
perception of the world. The column labels
Table III
Contingency table for a typical signal
detection analysis
Event Event did Row
occurred not occur totals
Event reported Hit False alarm C=1
Event not Incorrect Correct D=1
reported Rejection Rejection
Column totals A=1 B=1
[ 295 ]
John F. McGrew and (event occurred and event did not occur)
John G. Bilotta represent the real world. In the case of a risk
The effectiveness of risk management program, using Table II as an
management: measuring what
didn't happen example, we can interpret row ``task
Management Decision intervened'' as event reported and ``task not
38/4 [2000] 293±300 intervened'' as event not reported. We can,
also in Table II, interpret the column ``task
failed'' as event occurred and ``task did not
fail'' as event did not occur. The row and
column headings of Table I can be
interpreted in the same way.
The upper lefthand cell of Table III (labeled
``hit'') represents the correct perceptions of
true events (event reported and event
occurred). The lower righthand cell of Table
III (labeled ``correct rejections'') represents
the correct perceptions of false events (event
not reported and event did not occur).
Together these two cells represent the only
situations in which the team has correctly
discerned the nature of the events it has
examined or in which it has intervened. That
is, they have correctly called true events true
and false events false. In the context of risk
management, we can say the team has
correctly identified high risks that require
intervention and low risks that do not.
In contrast, the other diagonal of Table III
represents those items the team has
incorrectly perceived. That is, these are
events which are false but which the team
has judged to be true (false alarms ± upper
righthand cell) or events which are true and
which the team has judged to be false
(incorrect rejections ± lower lefthand cell).
Owing to the confounding effect of
intervention, however, the only two cells of
Table III that offer the possibility of an
unambiguous interpretation fall along the
bottom row (the incorrect rejections and the
correct rejections). In the top row, the truth
of a task (that is, whether it failed) is partly
due to the actions of the risk management
team. If you intervene to mitigate a risk and
it does occurs (that is, the task fails), you
don't know whether it would have failed
anyway or if the intervention increased or
decreased the degree of failure. The same
problem exists for an intervention for which
the risk does not occur (that is, the task does
not fail). You don't know whether the
intervention succeeded or if the risk simply
did not materialize. Interpretation of the top
row is confounded by the act of intervention
and the human propensity toward response
bias. However, in the bottom row, it is clear
when a risk occurs and you do not intervene
(incorrect rejections) and it is also clear
when you do not intervene and a risk does
not occur (correct rejection).
Traditionally, SDT uses the hit and false
alarm rates (the top row of Tables I, II and III)
[ 296 ]
for its analysis. One can just as appropriately
use the correct and incorrect rejections in the
bottom row to perform the analysis owing to
the symmetry of the theory. The structure of
an SDT contingency table is such that we can
infer the correct values for the top row of the
table from the bottom row by virtue of the
fact that the columns must sum to 1. In this
analysis, we use the incorrect and correct
rejection rates to determine the skill and bias
of the team because they provide a direct and
unambiguous measure of the effectiveness of
the team's risk management program. The
outcome would be the same if we used the
inferred values. However, the inferred values
are confounded with the actions of the team
in an unknowable way.
SDT provides a model for understanding
the contingency table summarizing the
outcome of a risk management program in
spite of the analytical complications
introduced by the team's acts of intervention.
An SDT analysis of the contingency tables for
risk assessment and risk intervention allows
us to determine three critical metrics about
the effectiveness of a risk management
program: it provides an estimate of the
team's skill, of its response bias, and an
unbiased estimate of the team's percent-
correct, an easily understood indicator of
effectiveness. We could, alternatively, take
the very simple approach of adding up the
hits and correct rejections (all of the team's
correct responses) and dividing by the total
number of tasks to determine a simple
percent-correct metric from the contingency
table. The straight percent-correct metric,
however, is easily distorted by response bias
which can cause a gross over or
underestimate of the true percent correct. In
our empirical data, we will show just such a
large effect for one of the project teams.
Method
The data presented here were collected from
two different project teams across three
product releases. The first team, Project T,
implemented a risk management program
during the final months of the project's
overall schedule. The team consisted of
almost 200 management and technical
professionals working over a period of five
years (Bilotta, 1995). The project management
team requested a risk program be set up to
address concerns about the readiness of the
project. This was the team's first attempt at
formal risk management.
Fifty-two tasks remaining in the project
schedule were reviewed once a month for six
months by the key technical staff under the
John F. McGrew and direction of one of the authors. Likeliness of Table V
John G. Bilotta failure and impact of failure were estimated Risk intervention contingency table for
The effectiveness of risk by the team for each task and were used to
management: measuring what Project T
didn't happen assess risk according to a process that was
first documented by the Air Force (US Air Tasks Tasks
Management Decision
38/4 [2000] 293±300 Force, 1988). Separate sessions were held to that that did Row
develop action plans to mitigate identified failed not fail totals
risks or to develop contingency plans to Tasks intervened 16 15 31
minimize the impact of failure. Tasks not
After the application had been successfully intervened 9 12 21
implemented, a quality assurance team from Column totals 25 27 52
outside the project performed a post-
conversion review, covering all aspects of the Table VI
development effort. They provided an Risk assessment contingency table for
independent assessment of the root causes of Project N, Phase I
success and failure that could be linked back
Tasks Tasks
to project activities, including the 52 tasks
that that did Row
monitored under the risk management
failed not fail totals
program (Bilotta, 1995). This, plus the
original risk assessment, mitigation, and Tasks estimated
contingency planning data provided the to be high risk 3 5 8
material to construct risk assessment and Tasks estimated
risk intervention contingency tables for the to be low risk 1 6 7
team. Column totals 4 11 15
The same methodology was used to
establish a risk management program for a
second project, Project N, being developed in Table VII
multiple phases by a small team of a dozen Risk intervention contingency table for
technical staff. Risk assessments, facilitated Project N, Phase I
by the authors, were done once for each of Tasks Tasks
two separate project phases. Again, follow up that that did Row
assessments of success and failure for failed not fail totals
individual tasks were conducted by a quality
Tasks intervened 3 7 10
assurance manager.
Tasks not
To begin the SDT analysis for risk
intervened 1 4 5
assessment, the team's assessment of risk for
Column totals 4 11 15
each task (high or low) was matched with the
task's corresponding outcome (failed or did
not fail) ± see Tables IV, VI and VIII. The Table VIII
same procedure was followed to analyze risk Risk assessment contingency table for
intervention: the team's action relative to Project N, Phase II
each task (intervened or did not intervene) Tasks Tasks
was matched with the task's outcome (again, that that did Row
failed or did not fail) see Tables V, VII, and failed not fail totals
IX. In this way, each task was placed in one of
four categories corresponding to the four Tasks estimated
cells of the contingency table: correct to be high risk 1 0 1
rejections, incorrect rejections, hits, and Tasks estimated
false alarms. to be low risk 0 27 27
Column totals 1 27 28
Table IV
Risk assessment contingency table for Table IX
Project T Risk intervention contingency table for
Project N, Phase II
Tasks Tasks
that that did Row Tasks Tasks
failed not fail totals that that did Row
failed not fail totals
Tasks estimated
to be high risk 18 3 21 Tasks intervened 0 11 11
Tasks estimated Tasks not
to be low risk 7 24 31 intervened 1 16 17
Column totals 25 27 52 Column totals 1 27 28

[ 297 ]
John F. McGrew and A correct rejection is a task which did not
John G. Bilotta fail and which, during assessment, was
The effectiveness of risk judged to be of low risk or which, during
management: measuring what
didn't happen intervention, was not acted upon. An
Management Decision incorrect rejection is a task which did fail
38/4 [2000] 293±300 during implementation but which, during
assessment, was judged to be of low risk or
which, during intervention, was not acted
upon. The upper lefthand cell of the upper
row contains the risk estimates and
interventions that were successful (hits)
while the upper righthand cell contains the
risk estimates and interventions that were
not (false alarms). The reciprocal
relationship between the cells in the upper
and lower rows is made clear in Equations 1
and 2 where H stands for hits, FA for false
alarms, IR for incorrect rejections, and CR
for correct rejections. A is the column total
for ``event occurred'' (task failed) and B is the
column total for ``event did not occur'' (task
did not fail). The relationships in Equations 1
and 2 are what enable us to estimate the
effect of events that did not happen.
1 = (H/A) + (IR/A) (1)
1 = (FA/B) + (CR/B) (2)
The primary measures derived from a signal
detection analysis are d', which is a measure
of the observer's sensitivity or skill, and c
which is a measure of the observer's
response bias. The statistic d' is the distance,
in standard deviation units, between the
mean of the distribution of tasks that failed
and the mean of the distribution of tasks that
did not. d' can be interpreted in the same way
as differences between standard normal
scores (or z-scores). A d' of 0 indicates a
complete overlap of the sample distributions.
A d' of 1.65 indicates a distance between the
means of the sample distributions of 1.65
standard normal scores or about a 5 per cent
overlap of the distributions, while a d' of 3
indicates an overlap of no more than 0.3 per
cent.
The statistic c is the acceptance cutoff point
for choosing between the two sample
distributions. Its value, positive or negative,
reflects the shift in the team's criterion for
accepting false alarms and incorrect
rejections, the two types of errors that can be
made. As c shifts, the team is adjusting its
level of acceptance of incorrect rejections or
false alarms. When c = 0, the team has been
able to minimize the acceptance of both. c
translates to observer bias, or the tendency to
yea-say or nay-say. Both risk assessment and
risk intervention decisions are considered
unbiased when c = 0.
Because c indexes the tendency to say no,
when c > 0 the bias is towards nay-saying or
denying that the risk exists. When c < 0 the
bias is towards yea-saying or seeing risk
[ 298 ]
where it doesn't exist. The range of c runs
from 0 to any value, positive or negative,
although it is unusual for it to be beyond ‹2.
It is obvious that the closer c is to 0, the more
efficient the risk management program
because response bias is not leading the team
toward underestimating or overestimating
the risk. When c = 0, risk intervention
actions are taken when required, but only
when required. Both d' and c are based on a
comparison of the hit and false alarm rates
or, alternatively, the correct rejections and
the incorrect rejections, as in our case.
Other measures can also be derived from
an SDT analysis including p(c), the raw (but
biased) percent correct described earlier and
p(c)unb, an unbiased estimate of the percent
correct. The statistic p(c) is the sum of the
hits and the correct rejections divided by the
total number of tasks in all the cells. p(c)
yields a biased estimate of the true percent-
correct because the underlying sample
distributions may be skewed. The statistic
p(c)unb yields an unbiased estimate of the true
percent-correct and is calculated from d'
which is a measure of skill that is
independent of bias.
Equations 3, 4, 5, and 6 are used to calculate
d', c, p(c), and p(c)unb respectively.
d' = z(H/A) ± z(FA/B) (3)
c = ±0.5[z(H/A) + z(FA/B)] (4)
p(c) = (H + CR)/(A + B) (5)
p(c)unb = d'/2) (6)
In equations (3) and (4), z is the standard
normal variate expressed in standard
deviations. In equation (5),  is the standard
normal cumulative probability.
Results
The risk management program for Project T
focused on the 52 most critical tasks
remaining in the project schedule. Ninety-
two risk mitigation action items and 23
contingency plans were created to offset the
risk associated with 33 of the 52 tasks. The
risk mitigation action items were never
implemented for two of the tasks, leaving 31
tasks for which intervention was planned
and taken. Twenty-nine of the 52 tasks were
estimated to be of low risk and no
intervention was planned or taken.
Table IV is the contingency table for
Project T's risk assessment data. It compares
the project team's estimates of risk for the 52
tasks with the post-implementation review
team's determination of the actual degree of
failure or success for the same tasks.
Table V is the contingency table for Project
T's risk intervention data. It contrasts the
John F. McGrew and project team's intervention actions for the 52 during risk assessment but underestimated
John G. Bilotta tasks with the post-implementation review the risk during intervention. In contrast,
The effectiveness of risk team's determination of the degree of failure
management: measuring what Team T tended to minimize the risk during
didn't happen or success for the same tasks. Table V makes assessment but to overreact to the risk
Management Decision clear that the team was able to control the during intervention. Bias results of this type
38/4 [2000] 293±300 risk associated with only 15 of the 31 tasks for are an indicator of systemic problems within
which it intervened ± the other 16 tasks in an organization and a zeitgeist in which
which the team intervened failed. On the teams publicly deny problems but scramble
other hand, the risk management team was behind the scenes to correct them. This is
successful in predicting that 12 of the 27 tasks frequently a consequence of a ``no negative
which did not fail did not require feedback'' culture.
intervention. Although p(c), the raw percent correct
Tables VI and VII summarize the shown in the third column of Table X, has the
assessment and intervention results of the
advantage of being more familiar to most
risk management program established by
people than d', it is only accurate in
Project N for Phase I of its project. Fifteen
situations where there is no bias in the
critical tasks were reviewed by the team and
observer's judgments. So long as c is near
the outcome, at the end of Phase I, was
zero, the difference in interpretation between
assessed by a quality assurance manager.
p(c)unb and p(c) is small, but if there is bias,
Tables VIII and IX summarize the
assessment and intervention results of whether positive or negative, the difference
Project N's risk management program for between p(c)unb and p(c) will grow. For
Phase II of its project. Twenty-eight critical instance, a comparison of the raw and
tasks were reviewed by the team and again unbiased percent correct show that if
the outcome was assessed by a quality ±0.30 < c < +0.30, then the raw percent correct
assurance manager. is fairly accurate. If bias is greater than ‹0.30,
The values of d' and c were calculated for the raw percent correct underestimates the
each contingency table using Equations 3 and actual percent correct. Team N, Phase I, had
4. The raw and unbiased estimates of the a p(c) of 47 per cent for its risk intervention
percent correct were calculated using efforts but a p(c)unb of 62 per cent ± a 15 per
Equations 5 and 6. The results for Project T cent underestimation. Team N, Phase II, had
and Project N, Phases I and II, are a p(c) of 57 per cent for risk intervention but a
summarized in Table X. p(c)unb of 99 per cent ± a 42 per cent
The d' values in Table X show that both underestimation. This is not surprising
teams were effective at assessing risk. Team T given that the bias is 0.45 in the first case and
had a d' of 1.81 in risk assessment and Team N, 3.15 in the second. In contrast, all three teams
Phase I, had a d' of 0.80 and in Phase II a d' of show little difference between p(c) and p(c)unb
12. Both teams were less effective at risk for risk assessment where their bias scores
intervention. Team T had a d' of 0.37. Team N, all fall at approximately ‹0.30. Thus, it can be
Phase I, had a risk intervention of 0.30 and in seen that p(c) can give seriously erroneous
Phase II a d' of 5.70. The assessment and estimates of a risk management program's
intervention d' values for Team N show a effectiveness.
strong learning effect from Phase I to Phase II. Team T made only one pass at risk
The c values in Table X indicate that Team management so we can say nothing about its
N overestimated the risk during both ability to learn from experience. However,
assessment and intervention for Phase I. In Team N shows clearly that this technique is
Phase II, it provided an unbiased estimate most valuable in its repetition. By using risk
management and SDT analysis from one
Table X
phase to the next, the team is able to build on
Skill and bias estimates for Project T and Project N, Phases I and II
its past experiences to improve its future
p(c) p(c)unb performance ± and the metrics d', c, and
d' c ``raw % ``Unbiased p(c)unb provide the means to quantify and
``Skill'' ``Bias'' correct'' % correct'' demonstrate for management the level of
Project T improvement and value of a risk
Risk assessment 1.81 0.32 81 82 management program.
Risk intervention 0.37 ±0.34 54 57
Project N, Phase I
Risk assessment 0.80 ±0.20 60 66 Discussion
Risk intervention 0.30 ±0.45 47 62
Is it possible to measure what you may have
Project N, Phase II
prevented from happening? The answer is
Risk assessment 12.00 0.00 100 100
yes. Ultimately, the effectiveness of a team's
Risk intervention 5.70 3.15 57 99
risk management program is probably best
[ 299 ]
John F. McGrew and represented by the effectiveness of its
John G. Bilotta intervention strategy. In our opening
The effectiveness of risk discussion, a team following a strategy of
management: measuring what
didn't happen intervention to minimize risk has no way of
Management Decision judging the effectiveness of its efforts. It is
38/4 [2000] 293±300 unable to do so because it is not possible to
separate the set of all successful tasks into
those that owe their success to intervention
and those that do not. Signal detection
theory, however, enables us to separate them
out by looking at the correct and incorrect
rejections. The prerequisites to doing this are
that potential risks be identified and tracked,
and an independent ``after-action'' review of
the results be done.
When arranged in contingency tables and
subjected to SDT analysis, the data can be
used to extract estimates of a team's skill and
bias in assessing and intervening in risks.
SDT also provides a means to calculate an
unbiased estimate of the percent correct, a
direct indicator of effectiveness. The
unbiased percent correct p(c)unb is derived
from the value of d' and has the advantage of
being more easily and intuitively understood
by those less familiar with SDT, especially if
percent correct is a measure being used to
report the effectiveness of risk management
programs. In the context of risk management,
either d' or p(c)unb is a valuable and unbiased
estimator of effectiveness.
The positive news is that even teams
inexperienced in risk management are able
to assign risk estimates with a remarkable
degree of accuracy. The risk assessment team
for Project T had never participated in a risk
management program and was able to
identify correctly the risk associated with 82
per cent of the tasks. On the other hand,
controlling risk through intervention is more
difficult to do. Although Project T could
identify the correct level of risk 82 per cent of
the time, it was able to mitigate only 57 per
cent of the identified risk. Team N increased
its ability to assess risk between Phase I,
where p(c)unb = 66 per cent, and Phase II,
where p(c)unb = 100 per cent. Its risk
intervention skill improved from 62 per cent
in Phase I to 99 per cent in Phase II.
We have no way of knowing if the
effectiveness ratings for risk assessment and
mitigation for the projects in this study are
above or below average relative to other
Application questions
1 What is your organisation's risk
management plan? Are there any areas
which you think might need addressing
based on the authors' arguments/
discussions?
[ 300 ]
software development projects. For the
project teams in this study, we have some
results indicating that teams inexperienced
in risk management can be very effective in
managing risk if only by their own reports to
the authors of their satisfaction with the
results of the program. An accumulation of
data from a number of projects will help
establish the true range of risk control.
Because p(c)unb is monotonically related to d',
either d' or p(c)unb can be used to support the
direct comparison of results across widely
different projects.
References
Bilotta, J.G. (1995), ``A study in risk
management'', presented at the Bay Area
Software Process Improvement Network, 18
January.
Boehm, B. (1989), Software Risk Management,
IEEE Computer Society Press, Los Alamitos,
CA.
Charette, R.N. (1989), Software Engineering Risk
Analysis and Management, McGraw-Hill, New
York, NY.
Jones, C. (1994), Assessment & Control of Software
Risks, Prentice-Hall, Englewood Cliffs, NJ.
Macmillan, N.A. and Creelman, C.D. (1991),
Detection Theory: A User's Guide, Cambridge
University Press, New York, NY.
McGrew, J.F. (1983), ``Human performance
modeling using the signal detection
paradigm'', presented to the Human Factors
Society, Norfolk, VA.
McGrew, J.F. (1994), ``Measuring the success of
the SEI model using signal detection theory:
an exploratory evaluation'', presented to the
Software Engineering Process Group
National Meeting, Dallas, TX, 27 April.
McGrew, J.F. and Jones, W. (1976), ``Likelihood
ratio shift as an indicator of motivational
shift in multiple choice test items'', presented
to the annual meeting of the Midwestern
Association of Behavior Analysis, Chicago,
IL, 4 May.
Putnam, L.H. and Myers, W. (1992), Measures for
Excellence, Prentice-Hall, Englewood Cliffs,
NJ.
Swets, J.A. (1996), Signal Detection Theory and
ROC Analysis in Psychology and Diagnostics:
Collected Papers, Lawrence Erlbaum
Associates, Mahwah, NJ.
US Air Force (1988), Software Risk Abatement,
AFCS/AFLC Pamphlet 800-45.
2 Which of your projects do you feel are the
most high risk? Do you think this idea
could help you?