Understanding Uncertainty

Definitions and Tools for Risk

Analysts

Goals for this Lecture

Understand sources and types of
uncertainty
Evaluate quantitative and qualitative
representations of uncertainty
Distributional forms

Explore some typologies

Consider how people interpret
uncertainty

Overview
Uncertainty is unavoidable
Uncertainty is information!
Where to do further research
Worst case scenarios

Uncertainty is chronically understated

Generally satisfied with statistical analysis
Henrion and Fischhoff (1986), estimates of speed
of light (see Kammen and Hassenzahl 1999 page
125)

Measured speed of light (km/sec)

300000

299830

Expected value
with standard error

299950

299820

Recommended value
with reported
uncertainty

299900

299810

299850

299800

299800

299790

299750

299780

299700

299770

299650

299760

299600

299750
1870

1880

1890

Year of experiment

From Henrion and Fischhoff

(1986)

1900

1984
value

1900

1910

1920

1930
Year of experiment

1940

1950

1960

1970

Some thoughts
Research efforts in risk analysis should
be viewed as tools for understanding
uncertainties, not necessarily for
reducing them. (Finkel, 1990)
Probability does not exist (Morgan and
Henrion, 1990)
Either you die from cause X or you dont

Uncertainty Analysis Should

Be Thorough
It should include quantitative measures
of uncertainty
It should include qualitative discussion
of uncertainty
Doing this well takes time and effort
(and is not always rewarded)

Quantitative Measures
Display the data set
Use descriptive statistics
Moments, ranges, etc

Apply sensitivity analysis

Eschew spurious precision!

Use of Point Estimates?

introducing confidence intervalsto
deal with uncertainty may be
pointlessunless the policy analyst
eliminates the point estimate itself.
A specific number has a vividness and
simplicity which makes it an inevitable
focus of policy debate.
Camerer and Kunreuther, 1989

Point Estimate or not?

+100%

increas
e

Change
in risk

reduction

-100%

do
nothing

Figure 10-6 from SWRI page 258

Option

Qualitative discussion
Discuss known sources of error
Consider plausible sources of error
Evaluate the importance of uncertainty

Challenges to ignoring
uncertainty
Regarding a report on ozone depletion no
attempt was made to estimate the systematic
errors in evaluating rates or omission of
chemical processes. Without such estimates,
decision makers are free to make their own
judgments ranging from uncritical acceptance
of the current models to complete skepticism
as to their having any likelihood of being
correct. Morgan and Henrion 1990.

Historical Mistakes
Rasmussen Report (WASH 1400) on reactor safety
Significant portions retracted by US government
Still referenced

Inhaber report on nuclear safer than other energy

technologies
Major source (Holdren) responded not so

Chauncy Starr
Voluntary/involuntary
Risk as f(benefit)
Truebut not precisely known

Tengs et al?

Briggs and Sculpher (1995)

Cost-effectiveness analyses from
medical literature
Public Health origins of CEA

Incomplete or inadequate attention to

uncertainty analysis in 86%

Starr (1969) interpretation

10

-3

Voluntary

10

-4

-5

-6

10

-7

10

-8

-9

10

-10

R~B

General aviation

10

P f [Fatalities/person hr. exposure]

10

10

10

Commercial
aviation

Hunting, skiing,
smoking
Railroads

Motor
vehicles

Average P
due to disease

Involuntary
R~B

Electric
power
R = Risk
B = Benefit

Natural
disasters

-11

10
100

200

500
Average annual benefit/person involved [dollars]

1000

2000

5000

10000

Otway and Cohen (1975)

Interpretation

-3

10

-4

-5

-6

10

R~B

1.8

General aviation

10

P f [Fatalities/person hr. exposure]

10
10

-7

10

-8

-9

10

10

Hunting, skiing,
smoking

Commercial
aviation

Voluntary

Motor
vehicles

Average P
to disease

due

Railroads

-10

R~B

Involuntary

6.3

Electric
power

R = Risk
B = Benefit

Natural
disasters

-11

10
100

200

500
Average annual benefit/person involved [dollars]

1000

2000

5000

10000

Example: Amitraz on Pear

Orchards
EPA decided to ban Amitraz for use on
pear orchards (US EPA 1979)
Point estimate generated for Cost
effectiveness
$2.6 million per life-year saved (Tengs et al
1995)

Expected Value of Ban

Does Amitraz control pests?
If not, ban has no economic implications
cpyls $0

Is Amitraz a carcinogen?
If not, ban has major economic implications
cpyls $

E(cpyls) $2.6 million

E(cpyls) = uniform($0, $)
Hassenzahl (2004)

Estimators
Because we cant always get the data
we want, we need to estimate data
We can use
Frequencies
Distributions
Curve-fitting

Frequency example: 500 people

495 have 10 toes

2 have 12 toes
1 has 9 toes
1 has 5 toes
1 has 0 toes
In next 1000: how many will have 6
toes?

Distributions as estimators
We often use DISTRIBUTIONAL
FORMS to approximate data sets
We then estimate missing or future
values using the distribution
Extrapolate beyond data set
Interpolate within data set

Example: 500 people

feet
0
0.5
1
1.5
2
2.5
3
3.5

number
0
0
0
0
0
0
1
1

feet
4
4.5
5
5.5
6
6.5
7
7.5

number
12
48
145
203
78
10
2
0

Triangular? Normal?

Normal: Characteristics: Mean = 5.6

feet, Standard Deviation = 1.3 feet
Forecast: Normal
10,000 Trials

Frequency Chart

9,921 Displayed

.022

218

.016

163.5

.011

109

.005

54.5

.000

0
2.21

3.89

5.58

7.27

8.96

Triangular? Normal?

Triangular? Normal?

A Good Estimator Is

Consistent
Unbiased
Efficient
Sufficient
Robust
Practical

Depicting Uncertainty:
Distributions
No consensus on how to depict
ignorance!
More later under Monte Carlo Analysis

Key forms
Uniform / rectangular
Triangular
Normal, lognormal
Many others!

Decision
When should we hold the Founders
Day parade?
Assume we want to avoid tornadoes

Kammen and Hassenzahl 1999

Uniform
0.003

Probability of tornado

0.002

0.001

Jan

Feb

Mar

Apr

May

Jun

Jul

Aug

Sep

Oct

Nov

Month

From SWRI page 106. Figure 3.5 Uniform distribution for daily
p(tornado) as described in problem 3-7a.

Dec

Kammen and Hassenzahl 1999

Rectangular
0.012
0.010
0.008

Probability of tornado

0.006
0.004
0.002

Jan

Feb

Mar

Apr

May

Jun

Jul

Aug

Sep

Oct

Nov

Month

From SWRI page 106. Figure 3-6 Rectangular distribution for

daily p(tornado) as described in problem 3-7a.

Dec

Kammen and Hassenzahl 1999

Triangular

0.025

0.020

Probability of tornado

0.015

0.010

0.005
0
Jun

Jul

Aug

Sep

Month

From SWRI page 107. Figure 3-7 Triangular distribution for

daily p(tornado) as described in problem 3-7a.

Oct

Policy Decision
Radon is found in homes across the
country
We might worry at 100 pCi/liter
We cant measure all the homes in the
country, but we have a decent sized
sample
It matters how we model that
distribution

Normal Distribution
1
4
1
2
1
0

Number of homes

8
6
4
2
0

3
Radon level (pCi/liter)

Kammen and Hassenzahl 1999

Lognormal
20

15

Houses (%)

10

>8

0
0

Nero et al 1987

4
222

Rn (pCi/liter)

Exponential
100

USAless6
pwr(1.25) %

10

pwr(1.75) %
LnNrm3.5%

% of houses with concentration > x

0.1
0.01
0.001
0.0001

Goble and Socolow 1990

10

100

Rn Concentration

1000

Fitting distributions to data sets

Compare data to predictions
Least squares
Maximum Likelihood

We will explore these in context

Binomial for animal toxicology data

Breaking Down Uncertainty

Useful typologies for thinking about
uncertainty
Cant always reduce uncertainty
Typologies have
Internal overlaps
Missing pieces (no perfect typology)

Four typologies
Finkel
Parameter Uncertainty
Model Uncertainty
Decision-rule Uncertainty

Smithson
non-quantifiable/holistic
aspects
uncertainty as one
component of
ignorance.

Boholm
Situates Uncertainty
as the non-calculable
part of risk
appropriate coping
strategies:
faith
precaution and
avoidance

Typology (after Morgan and

Henrion 1990)
1. Random error and
statistical variation
2. Systematic error
and subjective
judgment
3. Linguistic
imprecision
4. Variability

5. Randomness and
unpredictability
6. Expert Uncertainty
7. Approximation
8. Model uncertainty
Normative
Uncertainty

Random error/statistical
variation
We have a well defined set of tools
These can be misleading!
Often the ONLY thing that is done
Often doneand ignored

Z-scores, Chi-squared, p-values

Meaning of 95% confidence interval?

Random Error: Clusters

Systematic error and

subjective judgment
Example: speed of light, or energy
predictions
Chronically understated
Useful approach: bounding

200

Systematic Error: Predicted Year

2000 Energy Use

175
150
125

Quadrillion Btu Per Year

100
75
50
25
0
1972

1974

Goldemberg et al 1987

1976

1978
Year of Publication

1980

1982

Linguistic imprecision
Inconsistencies in language and usage
can lead to problems
Is beyond a reasonable doubt 95%?
What does that mean?
Rain is likelyare you from Las Vegas
or Bangladesh?
A few thousand deaths

Variability
Also called dispersion
Get the right population!
Describing variability can be a
challenge
Monte Carlo analysis is a useful tool

Variability
Height of individuals
Deterministic
element
Random element

Susceptibility to
disease
Predisposition
Known
Unknown

Life-history and habit

Theoretical models
Empirical data
Who are we worried
about?
Average person
Most susceptible
subset

Randomness and
unpredictability
Inherent randomness is irreducible!
Practical limitations and chaos

Expert uncertainty
Multiple interpretations of a single data
set
Norms of analysis
Motivational bias (decision stakes,
reputation)

Expert Uncertainty
Economists Conception
Limited resources
Resource substitution
Adaptability

Ecologists Conception
Stable systems
Long term impact of disruption

Climate Change: Economists

versus Ecologists
25
90th percentile
50th percentile

20

10th percentile

15

Loss of gross world product

10

-5
14

17

16

Individual respondents' answers

Nordhaus 1994

11

15

12

18

13

10

EMFs and Expertise

Biomechanists / physicists
Impossibility theorems

Epidemiologists
Correlation and proposed causation

Toxicologists
Extrapolation

Approximation
Never have complete data
There is a tradeoff between efficient
computation and resolution or precision
Sensitivity analysis

Significant figures are important

Model uncertainty
Getting the right model
Does the model explain the data?
Is the model consistent with theory?

Getting the model right

Is the model properly stated?
Is the math done correctly

Other uncertainties (drawn from the

typology above)

Model Uncertainty
100

USAless6
pwr(1.25) %

10

pwr(1.75) %
LnNrm3.5%

% of houses with concentration > x

0.1
0.01
0.001
0.0001

Goble and Socolow 1990

10

100

Rn Concentration

1000

Normative Uncertainty
Often not asked: what is important to
us?
Arguments about technical information
mask the true issues
Leads to vitriol and claim of ignorance
and antiscientific attitudes

Normative UC and YMP

Gore, Bush: let the science decide
Secretary Abraham technically suitable
site
LV residents technically unsuitable site
Have we defined suitability?

Interpreting Uncertainty
Very limited information on how people
interpret uncertainty
Possible links
Uncertainty and credibility
Uncertainty and trust

References
Briggs, A. and M. Sculpher (1995) Sensitivity analysis
in economic evaluation: a review of published
studies. Health Economics 4: 355-371.
Camerer, C. F. and Kunreuther, H. (1989) Decision
Processes for Low Probability Events: Policy
Implications, J. Policy Analysis and Management 8
(4), 565 - 592.
Goble, R. and Socolow, R. (1990), High Radon
Houses: Implications for Epidemiology and Risk
Assessment, Cented Research Report No. 5 (Clark
University, Worcester, MA).

References
Goldemberg, J,. Johansson, T., Reddy, A and
Williams, R (1987). Energy for a Sustainable
World. Washington DC: World Resources
Institute.
Hassenzahl, D. M. (2004). The effect of
uncertainty on risk rationalizing decisions.
Journal of Risk Research.
Henrion, Max and Fischhoff, Baruch (1986)
Assessing uncertainty in physical constants,
American J. of Physics,54, 791 - 798.

References
Nero, A. V., Schwehr, M. B., Nazaroff, W.W. and
Revzan, K.L. (1986) Distribution of Airborne Radon222 Concentrations in U.S. Homes, Science (234)
992 - 997.
Nordhaus, W. D. (1994) Expert opinion on climate
change, American Scientist, 82, 45 - 51.
Otway, H. and J. J. Cohen (1975). Revealed
Preferences: Comments on the Starr Benefit-Risk
Relationships. Laxenburg, Austria, International
Institute for Applied Systems Analysis.

References
Starr, C. (1969). "Social benefit versus
techological risk." Science 165: 1232-1238.
Tengs, T. O., M. E. Adams, et al. (1995).
"Five-Hundred Life-Saving Interventions and
Their Cost-Effectiveness." Risk Analysis
15(3): 369 - 390.
US EPA (1979) Determination pursuant to
40CFR162.11(a)(5) concluding the rebuttable
presumption against registration of pesticide
products containing amitraz. Federal Register
48: 2678-83.

Additional readings
Kammen and Hassenzahl, 1999. Should We
Risk It? Exploring Environmental, Health and
Technology Problem Solving, Princeton
University Press, Princeton NJ.
Finkel, Adam, 1990. Confronting Uncertainty in
Risk Management (Resources for the Future,
Washington DC)
Morgan, M. Granger and Max Henrion (1990).
Uncertainty Cambridge University Press, NY
NY.