Technological Forecasting & Social Change

70 (2003) 797 – 817

Correspondence to what? Coherence to what?

What is good scenario-based decision making?
Clare Harries*
Centre for Decision Research, Leeds University Business School, Leeds LS2 9JT, UK

Received 28 February 2003; received in revised form 3 March 2003; accepted 4 March 2003

Abstract

This paper provides a framework for the evaluation of scenario planning and other strategic
decision making methods or techniques. If scenario planning is useful, we should be teaching it in
schools and we as individuals should be using it to cope with the uncertainty inherent in modern life. A
prerequisite to this is the need to identify why, where and how (in what way) scenario planning and
other methods or techniques are useful. Here, I review evaluations of scenario planning. Taking a
Brunswikian perspective, I highlight the issues that have failed to be addressed in this evaluation. I
demonstrate that there are many ways in which scenario planning could be useful other than those that
have appeared in previous discussions. These multiple routes are dependent upon the interaction
between the individual organisation, the environment in which they are operating and the method
being followed.
D 2003 Elsevier Science Inc. All rights reserved.

Keywords: Scenario-based decision making; Scenario planning; Decision making

1. Introduction

Scenario-based decision making is widespread in business and organisations. Several

guidelines as to its practice are available, and accounts of its use abound with positive
messages. If scenario planning is a useful strategic decision making method, we should teach

* Tel.: +44-113-3432634; fax: +44-113-3434465.

E-mail address: ch@lubs.leeds.ac.uk (C. Harries).

0040-1625/03/$ – see front matter D 2003 Elsevier Science Inc. All rights reserved.
doi:10.1016/S0040-1625(03)00023-4
798 C. Harries / Technological Forecasting & Social Change 70 (2003) 797–817

it in schools, health professionals should use it to encourage patients to think strategically

about their health, families could use it in planning their lifestyles and we as individuals
should use it to cope with the uncertainty inherent in modern life. Unfortunately, we do not
know where, when and how scenario-based decision making is useful: there is little empirical
evaluation of it. The aim of this paper is to explore what evidence exists that evaluates
scenario-based decision making and to examine what could be done to evaluate scenario-
based decision making.
This paper falls into four sections. In Section 1, I briefly identify and characterise scenario-
based decision making. In Section 2, I introduce different taxonomies of evaluation. In
Section 3, the main body of the paper, I consider the case studies, empirical studies and
theoretical arguments evaluating scenario planning. I highlight the issues that have failed to
be addressed in current evaluations. In Section 4, I demonstrate that there are many ways in
which scenario planning could be useful other than those appearing in previous discussions.
Different mechanisms can each lead to successful strategic decision making. Understanding
the value of scenario-based decision making requires an understanding of the interaction
between the actions, goals and knowledge of the individual organisation and the environment
in which they are operating.

2. What is scenario-based decision making?

Scenario use in organisational planning and decision making involves creation of

descriptions of alternative future realities. The emphasis of these scenarios and the methods
and techniques by which they are derived cover a diverse set of behaviours [1,2] (see Ref. [3]
for a contrast between scenario methods) that combine qualitative and quantitative and
subjective and objective methodologies to different degrees. For example, Thomas et al. [4]
describe the development of scenarios for the steel industry that are based on a combination
of the results of mathematical models of consumption of (canned) moist cat and dog food (a
predictor of demand for steel) with a qualitative survey of experts’ opinions. In contrast, Islei
et al. [5] used a systematic but nonmathematical approach to elicit scenarios on the future of
process engineering in the pharmaceutical industry and followed this with statistical
judgmental analysis to evaluate the resulting scenarios and with the purpose of identifying
a consensus expected scenario. (There was none.) Therefore, companies purporting to use
scenario planning, or scenario analysis, or scenarios may be referring to any of a variety of
methods. The number of these potential scenario methods is increasing, as researchers and
consultants from different backgrounds use their particular expertise to create new variations
(e.g., Refs. [2,6,7]).
In this paper, I focus on the scenario planning method described clearly by Goodwin and
Wright [8] and Schoemaker [9]. The historical development of scenario planning by
companies such as Shell, Rank and IBM has been well documented (see, e.g., Refs. [10–
14]). Scenario planning has been used in a wide range of business and not-for-profit
organisation sectors. These include, for example, the agrifood industry [15], water and
computer industries [16], health service and pharmaceutical industries (e.g., Ref. [17]),
 C. Harries / Technological Forecasting & Social Change 70 (2003) 797–817 799

Table 1
Contrast between scenario-based decision making, (probabilistic) forecast-based decision making and decision
analysis
Scenario-based decision making
A few extreme plausible future worlds identified.
Future worlds identified independently of own actions.
Future worlds described causally and labelled like stories.
Choose strategic decision that is a winner with all
possible scenarios. Reject strategic decisions that do
not prove to be robust.
Contrast
One future world identified (F) or a complete
set of future worlds identified with associated
probabilities (DA).
Future world potentially used as a target (F)
or future worlds seen in context of actions
and chance nodes (DA).
Future identified in terms of each variable or
event rather than described as complex patterns.
Choose strategic decision best for (to achieve)
expected future world or that gives the highest
expected utility.

community forestry [18], local government [3] and those concerned with greenhouse gas
emissions [19].
The scenario planning method can be characterised thus: The drivers of change in an
industry or domain are identified and the (two) key uncertainties involved are picked out. The
extremes of these are combined and the resulting scenarios are described. Scenario planning
is the process of generating causal story-like scenarios against which a strategy can be tested.
In scenario-based decision making, strategic decisions are tested for robustness against a
series of scenarios describing possible/plausible future worlds.
Table 1 contrasts features of this process with those of decision analysis and forecast-based
decision making. The differing emphasis on plausibility and probability of future worlds
means that the influence of unlikely events is no stronger than that of likely ones in scenario-
based decision making, but more likely events have greater impact on decision making based
on forecasting or on decision analysis. The consideration of the future world from the starting
point of the decision in decision analysis and from the starting point of forecasts of individual
variables in forecasting-based decision making leads to an underemphasis on the result of
interactions between less controllable variables. The emphasis on causal, story-like construc-
tion of future worlds in scenario planning means that creative causal thinking is encouraged
compared with the evaluative, probabilistic thinking inherent in decision analysis and
forecasting-based decision making. Each of these features has potential implications for
the nature and quality of decision making. We shall discuss some of these later.

3. Taxonomies of evaluation

We have indicated the type of scenario-based decision making process in which we are
interested and some of the ways in which this differs from decision analytic and forecasting
approaches. In this section, I discuss evaluation methods and identify the dimensional space
within which scenario-based decision making could be evaluated.
800 C. Harries / Technological Forecasting & Social Change 70 (2003) 797–817

Researchers and practitioners have outlined taxonomies for the evaluation of judgment and
decision making (e.g., Refs. [20–24]; see also [25,26]). Purvis [23] extends the operational
research distinction between model verification and validation to methods. ‘‘This devel-
opment was undertaken in the context of it being an extension of the (British) Ministry of
Defence (MOD)’s existing fitness-for-purpose assessment framework for operational analysis
(OA) capabilities, which hitherto focussed predominantly on verification and validation of
models’’ (Purvis, 2003, personal communication). Method verification is a measure of the
degree to which the method has been correctly implemented. Verification that a method or
technique has been implemented is clearly a prerequisite to its validation or evaluation,
especially when other nominally similar methods exist, as they do in scenario planning.
Several dimensions emerge from these taxonomies of evaluation. (a) Decision making can
be assessed intuitively (subjective measures of satisfaction) or objectively (weak and strong
effectiveness) (see, e.g., Refs. [20,21]). (b) The process or outcome can be compared with that
of another set of rules (e.g., Ref. [20]). (c) The method can be evaluated in terms of (i) its
coherence or logical reasoning from basic axioms, and the truth of those axioms (the
method’s underlying assumptions), or in terms of measures of effectiveness (subjective or
objective) exhibited in (ii) simulations, (iii) experiments or (iv) real-world decision making
[20,21]. (d) Finally, Hammond’s [25,26] dichotomy between coherence and correspondence
metatheories of judgment and decision making contrasts effectiveness measures based on
actual behaviour and outcomes with theoretical and axiomatic evaluations. Comparison with
another set of rules and a theoretical evaluation in terms of coherence or logical reasoning fall
into coherence modes. Evaluations that focus on success in implementation and comparison
with actual outcomes (measured subjectively or objectively) might be labelled correspond-
ence modes.
Fig. 1 puts some of the above dimensions onto an evaluation space for one decision
making method or technique. The x-axis is divided into subjective measures and objective
measures. The z-axis is divided into the theoretical and practical evaluation methods or
techniques that vary in terms of prior knowledge or assumptions about the decision
environment, the individual and the interaction between them. A theoretical evaluation (a
coherence evaluation) and a simulation make assumptions about the decision making
behaviour and their interaction with the environment. An experiment makes assumptions
about the environment (but measures behaviour). An examination of real-world behaviour
arguably makes the fewest assumptions about the environment and about individual
behaviour. I shall return to the importance of this prior knowledge later.

3.1. Knowing what ‘‘good’’ is

Each of the evaluations of any one method or technique, shown in the x–z plane of Fig. 1,
can be carried out with respect to different references shown on the y-axis. A method or
technique may be useful in terms of carrying out its aims or the decision maker’s aims (see
Ref. [22]). Analogously, a pair of pliers is designed to grip small objects and bend wire but is
also useful as a pendulum weight where one is needed [27]. As we shall see later in this
article, this distinction between intended purposes and potential usefulness is less than clear in
 C. Harries / Technological Forecasting & Social Change 70 (2003) 797–817 801

Fig. 1. Evaluation space for decision making techniques or methods.

implementation of strategic decision making methods or techniques: benefits that emerge in

use can become intended benefits and part of the rationale behind the method.
There are currently three ways in which scenario-based decision making has been
evaluated: case studies, empirical studies and theoretical analysis. Case studies tend to fall
into area A on Fig. 1: measures are often subjective reports of real-world decision making,
implicitly relating to the decision maker’s goals. Empirical studies fall into area B: objective
measures of real-world decision making. The goals are again those of the decision maker.
Theoretical analysis falls in area C of Fig. 1: analysis is in terms of the method or technique’s
goals, from objective or subjective perspectives. I shall outline each of these evaluations
below.

4. Real-world evaluations of scenario-based decision making

4.1. Case reports of scenario-based decision making

A number of commentators (e.g., Ref. [16]) have noted the fact that evaluations of the
benefits of use of scenarios seem to be confined to single case reports. Ringland [11] cites a
number of successes:

‘‘Scenario planning helped Electrolux spot new consumer markets. Scenario planning
helped Pacific Gas and Electric prepare for the earthquake in California. Scenario planning
helped Shell anticipate the fall of communism in Russia and its effect on natural gas prices.
802 C. Harries / Technological Forecasting & Social Change 70 (2003) 797–817

Scenario planning helped Austrian insurance company Erste Allemeine Versicherung

anticipate the fall of the Berlin Wall and enter new markets in central Europe. Scenario
planning helped wiring and cable supplier KRONE develop 200 new product ideas.’’

In such reports, companies credit their ability to conceptualise unlikely situations or

combinations of events, and their ability to act (in those situations), to scenario planning.
Success is seen in several different forms. These successes and these forms become rationales
for use of the method. For example, scenario planning is implicated in improving prediction
(Erste Allemeine Versicherung) and increased creativity (Electrolux and KRONE). On the
surface, very few case studies describe less ‘‘successful’’ scenario planning (see Ref. [28] for
an exception). At best, some commentators report difficulties in breaching the distance
between formation of potential scenarios and putting plans and actions into place as a result of
them [13]. Thus, organisational failure is associated with failure to take onboard the scenario-
based decision making, a low score on verification rather than validation of scenario-based
decision making.
There are several obvious problems with evaluation on the basis of case studies. Firstly, the
number of reports of usage of scenario planning probably reflects a small percentage of the
actual instantiations of its use: companies in a business or organisational environment in
which (sustained) competitive advantage is seen as achievement will not trip over themselves
to reveal their business practices. In an environment in which an organisation’s image
creation is emphasised, they are even less likely to trip over themselves to reveal unsuccessful
business practices. The positive skew of these reports is likely to reflect a biased sample.
Secondly, much of the success is self-reported. Unless satisfaction with the process is one of
the goals of the decision maker, such a measure is unlikely to have a mapable relationship to
objective measures of success. Many researchers have demonstrated the mismatch between
verbal reports and behaviour (see, e.g., Refs. [29,30]), and Phelps et al. [16] demonstrated this
in relation to companies’ reports of the efficacy of scenario-based methods.
Thirdly, in both successful reports of use of scenario planning and unsuccessful reports, the
causal relationships between team personalities, environmental pressure, decision making
processes, organisational structure and organisational success are difficult to disentangle.
Indeed, Covin et al. [31] illustrate the inherent relationship between decision making style,
organicity of the organisation and technological nature of the industry. The complexity of the
situation can be seen in an article by Hodgkinson and Wright [28] in which they describe the
difficulties of facilitating scenario planning in a management team operating under stress.
Both the team and the CEO demonstrated dysfunctional behaviours such as bolstering and
hypervigilance and were reluctant to be involved in the scenario planning process and the
company has done badly. The problems in facilitation of scenario-based decision making in
this situation may in part have been because of negative attitudes and dysfunctional
personalities and may also have been because many of the scenarios will have been negative.
Schoemaker [32] reports that negative scenarios are given less credence than positive ones.
In theory, a sample of case studies is a sample. Unfortunately, as things are, it will be a
biased sample, reported with insufficient information about participants, process, goals,
objective outcome measures and context. Were more organisations prepared to outline the
 C. Harries / Technological Forecasting & Social Change 70 (2003) 797–817 803

methods they use, in theory, it would be possible via metaanalysis to disentangle the role of
environmental factors, situational factors, organisational characteristics and impact of the
components of the strategic decision making method on performance.

4.2. Empirical evaluations of scenario-based decision making

Single case studies are reports of the weak effectiveness of scenario-based decision
making. There is no real measure of validation: there is nothing to compare with, no measure
of what would have occurred had scenario-based decision making not been carried out and no
systematic data collection across organisations. Larger-scale studies theoretically allow the
possibility to examine the interplay between the situational, environmental and organisational
factors and the strategic decision making method or technique. An extensive search of the
peer-reviewed literature reveals only one study that examines the strategic decision making
policies of several organisations in order to evaluate the efficacy of scenario planning. Phelps
et al. [16] reported that water companies choosing to carry out scenario-based decision
making and tending to be larger tended to have worse water quality and perform worse in
terms of water pressure and interruptions to flow, without difference in ROCE, and that IT
companies using scenario planning had greater growth in profit, and ROCE, as well as
moderately better performance on a number of customer-related measures. They conclude
that use of such methods ‘‘allows companies to adapt their strategies to maximise their
financial returns while viewing customer service as a secondary goal.’’
Although it is an interesting analysis, there are four problems with the approach taken by
Phelps et al. Firstly, the questions asked in order to classify firms as using scenario planning or
not (the verification measure) would not necessarily be those used by all researchers. For
example, in the first study, the extent of scenario planning use was scored higher if a company
reported generating more scenarios, if they reported using probability assessments for
scenarios, if they reported greater use of computer modelling or if they report greater numbers
of people involved. None of these would be associated with greater scenario planning style
thinking as outlined by, for example, Schoemaker [32], van der Heijden [12] or Wright [33].
Secondly, although they controlled for company size (larger companies tended to be more
likely to be using scenarios) both in analyses and in sampling, this is an evaluation of
companies using scenario-based decision making, not of the process itself: we do not know
whether the success-prone characteristics of a firm caused it to use scenario planning (or have
time to do scenario planning) or whether the use of scenario planning independently
contributed to success. One way to examine this would be to track each organisational
performance prior to their use of scenario planning as well as after use of such a method. The
size of the firm might have influenced performance two ways: larger firms may have been in
stronger positions, but smaller firms may have been able to adapt to changing environments
more quickly. The effects of no other characteristics of the firm or its situation are analysed.
Thirdly, differences in organisations’ goals were not explicitly measured and different patterns
of firm performance may reflect equal goal achievement. Such a possibility is suggested by
Phelps et al. themselves when they suggest that water companies choosing to carry out
scenario-based decision making may not prioritise customer experience or water wastage.
804 C. Harries / Technological Forecasting & Social Change 70 (2003) 797–817

The fourth issue is also related to the measurement of performance. Strategic decision
making occurs over the long term. Phelps et al. [16] report the cumulative performance over a
period of equally weighted time periods. However, avoidance of probability measures in
scenario-based decision making reflects a deliberate lack of concern for average performance
that is not reflected in other decision making methods. In scenario-based strategic decision
making, goal achievement during unlikely periods counts for as much as goal achievement
during (longer) predictable periods. In a business situation, it may count for more: this may be
just the opportunity to create a competitive advantage. Phelps et al. did not analyse
performance in terms of the predictability and impact of the events in each time period.
Such an analysis would be difficult to carry out.

4.2.1. Summary of real-world evaluations

Realistic evaluations of scenario planning and correspondence measures leave us wanting.
Case studies repeatedly report positive weak effectiveness—subjective correspondence but a
little more. The empirical study highlights the number of factors that need to be taken into
account in order to assess the value of scenarios. Both approaches lack systematic verification
measures that are evidence that the same type of scenario-based decision making has been
correctly carried out or that allows us to identify the features applied in the process.

5. Theoretical arguments for scenario-based decision making

Next, we shall consider some of the rationales for scenario-based decision making a
coherence evaluation. It should become clear that by themselves theoretical evaluations and
rationales for scenario planning are unsatisfactory. They are rhetorical accounts of how they
operate. By itself, any one account may appear coherent and convincing, but other potential
accounts develop as understanding of humans and their environment increases. It is not clear
that we can guarantee that a theoretical account is sufficient and complete.
van der Heijden [12] describes five objectives of scenario planning: development of robust
strategies, better understanding of the future, better perception of patterns and change,
transmission of management ideas through use of these scenarios throughout the organisation
and leadership. Importantly, these are objectives that emerged during the decades over which
Shell was developing its scenario use. Elsewhere, the importance of adaptability has been
emphasised. For example, again, according to van der Heijden [12], ‘‘the ultimate purpose of
the scenario planner is to create a more adaptive organisation, which recognises change and
uncertainty and uses it to its advantage.’’ Numerous publications that discuss scenario
planning tend to do so favourably and they bring up these and other rationales.
As well as objectives, theoretical mechanisms by which scenario planning might or might
not be useful (and might meet objectives) have emerged and been rejected as it has been
enacted. Benefits in use of scenario-based decision making might be expected because of, for
example, the roles of probabilities, prediction, mental models, perception of change, causal
reasoning and creative thinking or because of changes in metacognition. As we shall see
below, not all of these mechanisms are advocated as important by scenario planners.
 C. Harries / Technological Forecasting & Social Change 70 (2003) 797–817 805

5.1. The role of probabilities

The role of probabilities in improving scenario-based decision making may occur in

several forms. Firstly, the use of scenarios may lead to improved tacit probability estimates.
Consideration of different scenarios has been shown to debias probability judgments by
increasing consideration for counterfactuals and counterarguments (though see Ref. [34] for a
fuller exploration). For example, Shoemaker [32] found that both generation and considera-
tion of scenarios increased the width of participants’ confidence intervals, thereby decreasing
overconfidence. Indeed, the increase in perceived probability associated with elaboration of
scenarios has been mooted as a means of improving acceptance of forecasts [35,36]. Thus, an
argument here is that scenarios will lead to a more accurate implicit understanding of the
likelihood of different outcomes.
Secondly, Wright [37] has argued that avoidance of probability estimates may lead to
avoidance of their associated biases and better decision making. Even in risky tasks, where
probabilities can be calculated, judgments of probability are notoriously prone to biases such
as overconfidence, neglect of base rate, inversion fallacy, etc. [38,39] (see Refs. [40,41] for a
discussion of the mechanisms underlying this and Refs. [42,43] for a discussion on their
impact on strategic decision making). Although it is a matter of debate (see http://
www.gbn.org/public/gbnstory/articles/ex _probabiliti.htm), one of the characteristics of scen-
ario-based decision making, in contrast to much of forecasting and decision analysis, is that
probabilities (and their associated biases) are omitted.
A third and related perspective on the role of probabilities suggests that any probabilities
(not just those derived from human estimations) are meaningless. Rather than in a risky
world, it is often argued that much strategic decision making occurs in an uncertain,
constantly changing world (e.g., Ref. [44]), where calculation of probabilities is impossible.
A strategic decision making method that does not attempt this activity will be advantageous.

5.2. The role of prediction

The role of the predictability of the world in rendering assessments of probability

redundant leads on to a second rationale for scenario planning. Just as avoidance of
probability judgments avoids the biases associated with these, according to Wright [33],
one of the benefits of scenario planning is that it ‘‘downplays’’ managers’ inability to predict
the future. Ayres [45] argues that although we have the capability to sense the warning signals
of physical discontinuities (such as volcano eruptions), we do not have such a forecasting
mechanism to sense discontinuities in the social and economic world (such as an oil crisis). A
similar opinion is espoused by Sherden [46]. The perspective of a scenario planner is that
there is no one forecastable future [13].
We shall explore this argument more in Section 5.4. However, although the proponents of
scenario planning emphasise the importance of not predicting but being prepared for a range
of possibilities, the fact is that case studies often uphold foresight as an example of successful
application of scenario planning (see earlier). If scenario planning is seen as development of a
portfolio of future worlds (alongside a portfolio of future strategies), highlighting successful
806 C. Harries / Technological Forecasting & Social Change 70 (2003) 797–817

predictions makes sense. Statistically speaking, a set of scenarios picked at random would
be more likely to encompass what does actually come to pass than would a single scenario
also picked at random. Anticipation of plausible but unlikely scenarios and planning
accordingly, where other companies have not, would give an organisation a substantial
competitive edge.

5.3. The payoff matrix from scenario planning

The payoff matrix associated with scenario predictions is also consistent with an advantage
for consideration of a portfolio of futures rather than just one. The penalties for false positives
(forecasting events that do not happen) lie in the resources spent on strategies that would not
otherwise have been developed. Although is likely that this is something we could do in
abundance and do well, there is little advantage of true negatives (not forecasting events that
do not happen) in predicting the world. What counts are hits (predicting what happens), and
what potentially costs most are misses (failing to foresee what happens).

5.4. The role of mental models and prepared minds

The potential role of these multiple scenarios in including one that may come to pass is
somewhat contradictory to the idea that the usefulness of scenario planning does not rest in
their ability to contain the actual future. Although the above arguments are feasible
mechanisms by which scenario use might be advantageous, the vital issue is missed that
the scenarios developed in scenario planning are not picked at random and as a portfolio they
are selective. Rather, the focus is on combinations of extremes of uncontrollable variables
that, although plausible, are arguably unlikely to happen. It is not that one of the handful of
scenarios will turn out to have been the correct forecast but rather that between them they
should reveal the underlying working of and interactions between uncertainties.
According to Wack [13], early versions of scenario planning at Shell, which were designed
to cover the range of possible futures, incited much interest but were not taken onboard in
terms of planning actions. Wack’s argument is that for scenarios to be beneficial they must
effectively encourage people to challenge their mental maps of the world and challenge the
organisation’s mental map. Only then can plans be put into action to respond to changes in
terms of these uncertainties. Warren [1] found that the creation of cognitive maps of scenarios
showed up inherent inconsistencies and problems with scenarios illustrating ‘‘conventional
wisdom.’’ Such mechanisms lie at the heart of understanding the role of strategic planning as
organisational learning (see Ref. [47] for a historical perspective on the different roles of
strategic planning). From this perspective, evaluation of the effectiveness of scenario
planning requires measurement of learning and knowledge.

5.5. The role of causal and story-like thinking

It has been argued that a causal storytelling mode is an advantage of scenario planning [8].
There is a body of work that suggests that causal, reason-based thinking occurs unsolicited in
 C. Harries / Technological Forecasting & Social Change 70 (2003) 797–817 807

judgment and decision making. For example, differences in ability to find reasons for
decisions can account for and create differences in choice [48,49]. The role of accountability
is arguably key in judgment and decision making and the use of stories allows us to account
for judgements and decisions made (see Ref. [50]).
Models of the world occur in all sorts of formats (including options, probabilities and
utilities or causal maps or stories). That causal, story-like accounts are a more natural way of
making judgments and decisions may mean that framing the future thus allows members of a
decision making team to communicate with each other, engage in the task and may be better
able to spot incongruence in conventional understanding [1]. Following this line, Fox and
Das [51] have used arguments as the basis of knowledge storage, reasoning and explanations
in computer-based medical decision aids. However, although it seems a more naturally
occurring form of communicating about decision making and may engage decision makers,
causal mental models are not without their potential biases [42].

5.6. The role of the perception of change

One of the reasons for developing a better mental model of the world is to increase
people’s ability to perceive the different worlds as they come into being and to perceive
patterns that are contradictory to the assumptions of their current strategy. In making
judgmental forecasts, people are relatively poor at distinguishing noise (random variations)
from underlying changes (e.g., Ref. [52]). Andreassen and Kraus [53] found that participants’
ability to extrapolate from a time series depends upon the salience of the change information.
In theory, the formation of scenarios will make potential changes more salient and will make
it easier for the decision maker to distinguish noise from a different but distinguishable world
pattern.

5.7. The role of creative thinking

Several authors have emphasised the importance of encouragement of creative minds and
creative thinking in strategic planning. For example, Beinhocker and Kaplan [44] identified
innovative development of strategies as the second important role of strategic planning (after
developing prepared minds). Creativity is often seen as a personality trait alongside other
aspects of cognitive style (see, e.g., Ref. [54]). However, it is also possible that different
processes change a person’s capacity for creative thought on a particular task—different
processes change a person’s cognitive style as if it were a state. The instruction to think of
plausible future worlds and to hold several different worlds in mind may be just such a
process to encourage creative thought and divergent thinking (see Ref. [55]; cf. Ref. [56]).
This encouragement of creative, divergent thinking about the future may act positively in
several ways. Differences may be encouraged across team members counteracting the widely
discussed negative effects of groupthink [12]. It may also free up any one individual or the
team to constructively explore apparently inconsistent and incoherent ideas. Finally, of
course, this encouragement of creative divergent thinking with respect to scenarios may be all
that is required to increase both quality (in terms of innovativeness) and quantity of strategies.
808 C. Harries / Technological Forecasting & Social Change 70 (2003) 797–817

Such a creativity mechanism is suggested in comparing managers’ use of art appreciation to

scenario planning [57]. Case studies indicate that scenario planning has led to creativity in
strategy development but the extent and nature of change in creativity with use of scenarios is
still to be measured.
Note that for the above seven rationales to work, an organisation must be able to develop
and explore scenarios fearlessly and meaningfully, and the skill of a decision facilitator is key.
Although the effect may be mitigated with causal and narrative reasoning, if left to their own
devices, people shy away from exploration of unlikely and negative futures. For example,
Dougherty et al. [34] found that although people appeared to think up multiple scenarios
automatically, they were quick to dismiss unlikely ones to focus on the few most plausible.
Negative scenarios also naturally have less credence than positive ones [9] (see also the
discussion on desirability bias in [58]). If potential scenarios can be believed, they can
potentially be learnt from planned for and compared with emerging realities.

5.8. The role of metacognition

The role of metacognition rarely appears in discussions of scenario-based decision making

but may well underlie its use. Here are two mechanisms. Firstly, in engaging in scenario-
based decision making, managers consciously address issues of the way they process
information, solve problem and make decisions. Although such process-oriented metacog-
nitive activity is reported to be of little use in complex problem solving [59,60], it has
generally been shown to enhance performance in problem solving [61] and may improve
strategic decision making (see also Ref. [62]).
Secondly, an emphasis on consideration of the multiple states of the world and the strategy
independently of each other as well as in combination could lead to a more accurate model of
an organisation’s efficacy. Although we have discussed the importance of models of how
uncertainties in the world interact, Warren [1] argues that for a scenario to be useful it must
contain the strategies both of the organisation and of its competitors. When Beinhocker and
Kaplan[44] reiterate the importance of the role of strategic planning in developing ‘‘prepared
minds’’ they emphasise an understanding of ‘‘the business, its strategy, and the assumptions
behind that strategy.’’ In other words, an understanding of the place and scope of the
organisation, within its world, is key to strategic decision making. Humans demonstrate an
illusion of control both in terms of being able to predict the future and in terms of being able
to control it [63].
Forecasting starts from the point of separate but predictable variables in the world and is
likely to encourage both of these illusions. Decision analysis focuses the decision maker at
first on their choices then moves to consider chance events. As such, it is also likely to
overemphasise the role of the decision maker in relation to multiple interacting uncontrollable
uncertainties. In scenario planning, scenarios are built initially independent of the organ-
isation; then, the organisations strategies are tested for robustness against them. This and the
emphasis on an unpredictable world may lead to a more accurate understanding of an
organisation’s efficacy. Of course, a more accurate, less optimistic perception of self-efficacy
may not be beneficial and may lead to reduced effort and reduced performance [64]. On the
 C. Harries / Technological Forecasting & Social Change 70 (2003) 797–817 809

other hand, realisation of the limits of current strategies may lead to development of new,
more robust strategies (and this will in turn increase self-efficacy judgements).

5.9. The role of adaptivity

Adaptivity has been discussed in relation to both the behavioural and the cognitive aspects
of an organisation’s scenario-based decision making. One of the major tenets of scenario
planning is that it is a useful basis to test the robustness of plans of action. The robustness of
strategy, or ability to survive in different worlds, is one type of adaptivity. In Section 5.7, we
discussed the potential role played by creativity encouraged in scenario planning to identify
more and more innovative strategies. Beinhocker [65] discusses the maintenance of multiple
strategies by an organisation in evolutionary terms: strategies are equivalent to gene
combinations within a population. Those that are fit flourish to different degrees at different
times, with some dying out and new strains appearing. The ability to change quickly to
alternative (preplanned) strategies is also a type of adaptivity.
Adaptivity also comes into play respect to mental models. As discussed earlier, scenario
development plays a key role in challenging current understanding and improving mental
models of the mechanisms underlying the organisation’s world, and this allows early
perception of states that do not match the assumptions of current strategies. However,
underneath this is the implication that, once shifted, managers will be open to adapt their
models of the world again, given evidence of changes in the world that do not match their
developed understanding.

5.10. Conclusions are theoretical evaluations

I have discussed several ways in which scenario-based decision making might be useful.
There are others that I have not discussed. For example, that scenario-based decision making
has general advantages of being a novel process for managers—encouraging engagement and
attention. Also, for example, scenario planning has the general advantages of being a formal
process in that it can be monitored; as a result, learning can occur. Also, for example, scenario
planning, like decision analysis, includes information processing aids that allow humans, who
have limited information processing capacity, to process more information about a complex
world.
In the early days of decision analysis, the rationale for its expected good performance was
that its use would maximise the decision maker’s expected utility. Later experimental work
suggested that drawing up a decision analysis is beneficial because it helps the decision
maker take all the information in a complex situation into account: the method was shown to
have no effect on simple decisions [66]. Another account suggests that rather than just
revealing the decision maker’s preferences to the facilitator, assessment of utility also reveals
and makes concrete the decision maker’s preferences to themselves [21]. With scenario
planning, as with decision analysis, even if the evaluation of one rationale proves it to be
coherent, such a discussion is unlikely to capture all relevant issues and all relevant
arguments.
810 C. Harries / Technological Forecasting & Social Change 70 (2003) 797–817

Although it is potentially regressive,1 theoretical evaluation of a decision method or

technique can be carried out using decision aiding methods or techniques. Goodwin and
Wright [67] and van der Heijden [12] emphasise the role of scenarios as ‘‘wind tunnels’’ in
which to test strategies. The strategy of use of scenario-based decision making could be
evaluated using scenarios. Simply listing pros and cons, a technique described by Benjamin
Franklin writing to his friend Joseph Priestly in 1772 (see Ref. [68] for an account of this) is
also use of a decision technique to evaluate a decision technique. The earlier mention of
probability and the potential payoffs of predictions takes something of an expected utility
framework. It is hard to avoid theoretical assumptions within a theoretical evaluation. Their
compatibility with the theoretical assumptions of the method being evaluated need to be
monitored.

6. Charted areas in the evaluation space

Therefore, what can we conclude about the evaluation space of scenario-based decision
making? Although Fig. 1 included a number of areas of potential evaluation, the research and
ideas discussed above fall into only three areas. Each of these has been seen to be insufficient
to draw conclusions about how and whether scenario-based decision making is beneficial.
Real-world evaluations lack measures of verification, are subject to biased sampling, rely on
invalidated reports, do not explicitly define the reference goal or measure the reference goal
inappropriately in terms of the method and are unable to distinguish between the effects of
organisation, method and environment. Theoretical evaluations, as suggested above, are still
emerging and have the potential to be theoretically inappropriate. Experimental work appears
within a theoretical evaluation but controlled experimental evaluations of scenario-based
decision making per se examining either objective or subjective measures of performance
have been elusive. Similarly, simulations evaluating scenario-based decision making are as
yet under development.2

6.1. Comparing and using alternative methods and techniques

So far, in this paper, I have discussed evaluations of scenario-based decision making per
se. One aim of evaluation is to facilitate informed choice of or between methods in terms of
their relative advantages in specific situations. Between-method comparisons also potentially
enable assessment of decision making components (such as those in Table 1). Summaries of
the development of methods, examples and pros and cons each reported separately but
juxtaposed underlie development of method logbooks [23] and of the Millennium Project
Futures Research Methodology report [69]. Theoretical discussions directly comparing
scenario planning with other methods have focussed on paired comparisons (e.g., Refs.

1
The choice of a technique with which to evaluate scenario-based decision making of course leads to an
infinite regression.
2
Theo Stewart, 2002, personal communication.
 C. Harries / Technological Forecasting & Social Change 70 (2003) 797–817 811

[37,70]; see Ref. [71] for an exception). Empirical comparisons between scenario planning
and other specified methods or techniques are elusive but those between methods or
techniques other than scenario planning tend to compare a pair at a time (e.g., Ref. [72]).
Comparing methods in real-world decision making is likely to become harder rather than
easier, as methods that are in some ways complimentary are merged. For example, academic
practitioners have promoted the formalisation of robustness testing in scenario planning via
the use of decision aids such as SMART [8] and MCDA.2
In the absence of direct comparisons, differences between methods and techniques could
emerge from a comparison of results on an evaluation space. Both correspondence and
coherence comparisons will be difficult. The theoretical assumptions about the nature of the
environment (such as its ability to be predicted) differ and need to be kept explicit. Aims of
the methods may also differ (for example, in terms of focus on average performance vs.
performance in extraordinary times). Although they may evolve, these also need to be kept
explicit. The need to take account of differences in the organisation, and environment applies
as much for between-method comparisons as for examinations of single methods. However,
as will be seen shortly, a comparison between methods and techniques in terms of the ability
of the method to achieve the decision maker’s aims is potentially the most difficult
comparison of all.
Including the rationales discussed above, Fig. 2 indicates a number of different strategic
decision making routes that could potentially lead to success or achievement. The arrows
indicate a positive association. For example, a benign environment will lead to a good result
regardless of the decision making process. For example, the ability to think of several
scenarios is positively associated with the ability to predict the actual world. Increasing the
ability to form a probabilistic model is associated with increase in ability to predict the world.
Increasing the ability to think of multiple scenarios will (according to the creativity argument
made earlier) increase ability to think of multiple strategies. In practical terms, an organisation
could potentially go down any of these paths to achieve their goals. If an organisation has the
potential to go down more of these paths, it can bypass weak spots. For example, failure to
foresee world events can be made up for by ability to perceive contraindications to the model
(perception of the environment) and to act on these quickly. Interactions between the
individual and the environment can also be seen in the connections between achievement
(the probability of outcomes matching goals) and environmental benignness and predictabil-
ity: potentially, organisations can make decisions that will place themselves in a more benign
environment or a more predictable one. These are just some of the possible routes.
Changing method or technique or changing environment will also change the goals of the
decision maker. A considerable body of work suggests that our preferences evolve as we go
through the process of making decisions (see, e.g., Ref. [73]). They are affected by
differences in presentation and in process. For example, achievement is seen relative to a
reference point [74] and will be affected by the general wealth of the environment and by
others’ achievement. Preference reversals have been seen to occur when objects are evaluated
simultaneously compared with when they are evaluated one at a time. One explanation for
this is that attributes that are harder to assess come into play when the relative level on a
particular object can be compared with that on another but are discounted when the objects
812 C. Harries / Technological Forecasting & Social Change 70 (2003) 797–817

Fig. 2. Paths to good decision making.

are considered alone [75]. A similar effect of process may well occur when choosing between
strategies simultaneously (as in decision analysis) or sequentially (as in scenario analysis,
where each option is evaluated for robustness one at a time). With such a difference, goals
that are less measurable will be relatively discounted in a scenario planning process compared
with a process that compares strategies with each other. Indeed, Phelps et al. [16] report such
a pattern. They argue that their results suggest a change in emphasis on criteria (financial over
customer related) with use of scenarios.
Perhaps the job of the evaluator is to identify the decision maker’s fundamental stable
goals against which performance should be evaluated. Even such fundamental goals are hard
to pin down. For example, most papers in strategic management uphold competitive
advantage as a fundamental goal. According to Tsoukas and Knudsen [76], ‘‘most researchers
agree that the chief purpose of corporate strategy is the creation of sustainable competitive
advantage.’’ However, although researchers may agree, individual companies, seen as
extremely successful, do not appear to seek competitive advantage per se. When given the
opportunity, they take their competitors out of the game altogether—leaving them with no
competition and therefore no competitive advantage. Competitive advantage is simply a
vicarious route to maximum performance—maximum meaningful returns and maximum
efficiency. Arguably, there are no goals, child protection and happiness included, which a
 C. Harries / Technological Forecasting & Social Change 70 (2003) 797–817 813

human cannot, in extreme circumstances, rationalise over with some otherwise less pressing
goals. A key to success then is to have a portfolio of goals: one will be successful no matter
what our interaction with the environment brings. As yet, this rationale for the usefulness of
scenarios has not been made.
In sum, just as the contribution of criteria to preferences changes with process, the decision
maker’s goals’ contribution to their consideration of options are likely to be similarly affected
and similarly evolving. The goals emerging during the process of scenario-based decision
making may not be the same as those of emerging from other types of strategic decision
making. Just as creativity and divergent thinking encourages adaptivity of mental models,
perhaps adaptivity of goals is encouraged via use of scenario planning. The perfect decision
maker is someone who can adapt their goals to fit the outcome of any decision.
Egon Brunswik, a psychologist working in the 1920s and 1930s, emphasised the
importance of focussing on individuals and their interaction with the environment in studying
human behaviour (see Ref. [77]). I have emphasised the importance of the environment and
the individual (organisation) in constructing theoretical, simulation and experimental
evaluations and in interpreting real-world evaluations above. The effects of their interaction
should also be considered in evaluation. Organisations make decisions that put themselves
into environments that are more or less predictable or that are more or less wealthy. For
example, by encouraging others to be innovative, creative (and unpredictable) and being so
themselves, they may create the conditions under which the method they use is designed to
flourish.
Brunswik also emphasised the importance of vicarious functioning. In coining this phrase,
he emphasised the potential for individuals to substitute cues for each other in making a
judgment or to use different means to reach the same end. Arguments behind scenario
planning and strategic decision making tap into the rhetoric of evolution. The world is an
ever-changing place, and in order to survive, organisations have to be adaptive. Vicarious
functioning is a different type of adaptive technique. Simply put, different mechanisms can
each lead to successful strategic decision making and the possibilities of these mechanisms
are striking. The concept of vicarious functioning and of the interaction between an individual
(organisation) and the environment are key to our understanding evaluation.

6.2. Practical implications

Given all the potential mechanisms by which it might be advantageous, scenario planning
seems plausibly so. However, there are still no concrete conclusions as to how and whether it
is. The focus of this paper has been on evaluation. However, in discussing some of the
numerous routes by which scenario planning might be of practical benefit, I have highlighted
the possibility that a successful decision maker will have several decision making mecha-
nisms in their repertoire and will be able to improvise around any of these given different
environmental and situational constraints. Within this framework, a practically minded
decision maker might identify the best route for a given set of circumstances. A practically
minded decision maker who wishes to learn from their experience will document all aspects
of it, successful and unsuccessful.
814 C. Harries / Technological Forecasting & Social Change 70 (2003) 797–817

7. Conclusions

In this paper, I have reviewed case studies, empirical studies and theoretical arguments
evaluating scenario-based decision making. Real-world evaluations lack measures of veri-
fication, are subject to biased sampling, rely on invalidated reports, do not explicitly define
the reference goal or measure the reference goal inappropriately in terms of the method and
are unable to distinguish between the effects of organisation, method and environment.
Theoretical evaluations are constantly evolving rationales about which completeness and
sufficiency are difficult to assess. Such coherence and correspondence evaluations are reliant
on each other to develop understanding of environment characteristics and human character-
istics given that environment and given that method. Evaluations of a decision making
method need to take this into account, to take emerging understandings of the method’s
mechanisms into account and particularly to take into account the full implications of the
potential effects of human’s adaptability.

Acknowledgements

Versions of this paper were presented at the Centre for Decision Research Seminar Series,
University of Leeds 2001–2002, at the 22nd International Symposium of Forecasters in
Dublin, at IFORS 2002 in Edinburgh, at the 8th BDRM in Chicago and at the OR Society
Strategy Subgroup Workshop on Scenario Planning in Sheffield. I am grateful to those at
these meetings and who subsequently have directed me to related but citation exclusive
literatures. Unless indicated otherwise, however, what appears above is my own exploration.

References

[1] K. Warren, Exploring competitive futures using cognitive mapping, Long Range Plan. 28 (5) (1995) 1 – 9.
[2] B. Sapio, SEARCH (Scenario evaluation and analysis through repeated cross impact handling): a new
method for scenario analysis with an application to the Videotel service in Italy, Int. J. Forecast. 11
(1995) 113 – 131.
[3] G. Cairns, G. Wright, R. Bradfield, K. van der Heijden, G. Burt, Exploring e-government futures through the
application of scenario planning, Technol. Forecast. Soc. Change 70 (2003) in press.
[4] M.C. Thomas, M. Evans, R.L. Davies, Forecasting and scenario planning in the steel packaging industry:
case of tinplate petfood cans, Ironmak. Steelmak. 28 (6) (1999) 463 – 469.
[5] G. Islei, G. Lockett, P. Naudé, Judgmental modelling as an aid to scenario planning and analysis, Omega Int.
J. Manage. Sci. 27 (1999) 61 – 73.
[6] P. McBurney, S. Parsons, Formalizing scenario analysis, Department of Computer Science, University of
Liverpool, Technical Report, 2002.
[7] P. McBurney, S. Parsons, Ensemble theory: arguing across and within scenarios, Presented at Probing the
Future: Developing Organizational Foresight in the Knowledge Economy, 2002, July 11 – 13.
[8] P. Goodwin, G. Wright, Decision Analysis for Management Judgment, 2nd ed., Wiley, Chichester, UK, 1998.
[9] P.J.H. Schoemaker, Profiting from Uncertainty: Strategies for Succeeding No Matter What the Future Brings,
Free Press, Simon and Schuster Inc., New York, 2002.
[10] M. Godet, F. Roubelat, Scenario planning: an open future, Technol. Forecast. Soc. Change 65 (2000) 1 – 2.
 C. Harries / Technological Forecasting & Social Change 70 (2003) 797–817 815

[11] G. Ringland, Scenario Planning: Managing for the Future, Wiley, Chichester, UK, 1997.
[12] K. van der Heijden, Scenarios: The Art of Strategic Conversation, Wiley, Chichester, UK, 1996, Vols.
Chichester, UK.
[13] P. Wack, Scenarios: uncharted waters ahead, Harv. Bus. Rev. (1985) 73 – 89 (September – October).
[14] P. Wack, Scenarios: shooting the rapids, Harv. Bus. Rev. (1985) 139 – 150 (November – December).
[15] B. LaFourcade, P. Chapuy, Scenarios and actors’ strategies: the case of the agri-foodstuff sector, Technol.
Forecast. Soc. Change 65 (2000) 67 – 80.
[16] R. Phelps, C. Chan, S.C. Kapsalis, Does scenario planning affect performance? Two exploratory studies,
J. Bus. Res. 51 (1999) 223 – 232.
[17] R.F. Schreuder, Health scenarios and policy making: lessons from the Netherlands, Futures 27 (1995)
953 – 958.
[18] E. Wollenberg, D. Edmunds, L. Buck, Using scenarios to make decisions about the future: anticipatory
learning for the adaptive co-management of community forests, Landsc. Urban Plan. 47 (2000) 65 – 77.
[19] N. Nakićenović, Greenhouse gas emissions scenarios, Technol. Forecast. Soc. Change 65 (2000) 149 – 166.
[20] R. Hastie, Problems for judgment and decision making, Annu. Rev. Psychol. 52 (2001) 653 – 683.
[21] R.T. Clemen, Does Decision Analysis Work? Manuscript available at: http://faculty.fuqua.duke.edu/
 clemen/bio/Effective.pdf, 1999.
[22] M. Pidd, Tools for Thinking: Modelling in Management Science, Wiley, Chichester, UK, 1996.
[23] M. Purvis, Methods and log books, London, UK, HMSO, Crown Copyright dstl/CPO1382, Presented at the
Operational Research Society Annual Meeting, 2001.
[24] M. Landry, C. Banville, M. Oral, Model legitimisation in operational research, Eur. J. Oper. Res. 92 (1996)
443 – 457.
[25] K.R. Hammond, How convergence of research paradigms can improve research on diagnostic judgment,
Med. Decis. Mak. 16 (1996) 281 – 287.
[26] K.R. Hammond, Human Judgment and Social Policy: Irreducible Uncertainty, Inevitable Error, Unavoidable
Injustice, Oxford Univ. Press, New York, USA, 1996.
[27] N. Maier, Reasoning in humans: II. The solution of a problem and its appearance in consciousness, J. Comp.
Psychol. 12 (1931) 181 – 194.
[28] G.P. Hodgkinson, G. Wright, Confronting strategic inertia in a top management team: learning from failure,
Organ. Stud. 23 (6) (2002) 949 – 977.
[29] R.E. Nisbett, T.D. Wilson, Telling more than we know: verbal reports on mental processes, Psychol. Rev. 84
(1977) 231 – 258.
[30] P.A. White, Knowing more about what we can tell: ‘‘Introspective access’’ and causal report accuracy 10
years later, Br. J. Psychol. 79 (1988) 13 – 45.
[31] J.G. Covin, D.P. Slevin, M.B. Heeley, Strategic decision making in an intuitive vs. technocratic mode:
structural and environmental considerations, J. Bus. Res. 52 (1) (2001) 51 – 67.
[32] P.J.H. Schoemaker, Multiple scenario development: its conceptual and behavioral foundation, Strategic
Manage. J. 14 (1993) 193 – 213.
[33] G. Wright, Strategic Decision Making: A Best Practice Blueprint, Wiley, Chichester, UK, 2001.
[34] M.R.P. Dougherty, C.F. Gettys, R.P. Thomas, The role of mental simulation in judgments of likelihood,
Organ. Behav. Hum. Decis. Processes 70 (2) (1997) 135 – 148.
[35] W.L. Gregory, A. Duran, Scenarios and acceptance of forecasts, in: J.S. Armstrong (Ed.), Principles of
Forecasting: A Handbook for Researchers and Practitioners, Kluwer Academic Publishing, Holland, 2001,
pp. 519 – 540.
[36] J.S. Armstrong, Strategic planning and forecasting fundamentals, in: K. Albert (Ed.), The Strategic Manage-
ment Handbook, McGraw-Hill, New York, 1983, pp. 2-1 – 2-32.
[37] G. Wright, Scenario thinking versus subjective probability forecasting, Technol. Forecast. Soc. Change 61
(1999) 81.
[38] D. Kahneman, P. Slovic, A. Tversky, Judgment Under Uncertainty: Heuristics and Biases, Cambridge Univ.
Press, Cambridge, UK, 1982.
816 C. Harries / Technological Forecasting & Social Change 70 (2003) 797–817

[39] T. Gilovich, D. Griffin, D. Kahneman, Heuristics and Biases: The Psychology of Intuitive Judgment, Cam-
bridge Univ. Press, Cambridge, UK, 2002.
[40] G. Gigerenzer, U. Hoffrage, How to improve Bayesian reasoning without instruction: frequency formats,
Psychol. Rev. 102 (1995) 684 – 704.
[41] U. Hoffrage, G. Gigerenzer, S. Krauss, L. Martignon, Representation facilitates reasoning: what natural
frequencies are and what they are not, Cognition 84 (2002) 343 – 352.
[42] J.H. Barnes, Cognitive biases and their impact on strategic planning, Strategic Manage. J. 5 (1984) 129 – 137.
[43] C.R. Schwenk, Strategic decision making, J. Manage. 21 (3) (1995) 471 – 493.
[44] E.D. Beinhocker, S. Kaplan, Tired of strategic planning? McKinsey Q. Risk Resilience 2 (2002) 49 – 57.
[45] R.U. Ayres, On forecasting discontinuities, Technol. Forecast. Soc. Change 65 (2000) 81 – 97.
[46] W.A. Sherden, The Fortune Sellers: The Big Business of Buying and Selling Predictions, Wiley, New York,
1998.
[47] A. Pettigrew, H. Thomas, R. Whittington, The Handbook of Strategy and Management, Sage Publications,
London, 2001.
[48] E. Shafir, I. Simonson, A. Tversky, Reason-based choice, Cognition 49 (1998) 11 – 36.
[49] D.W. Green, A. McClelland, L. Muckli, C. Simmons, Arguments and deontic decisions, Acta Psychol. 101
(1999) 27 – 47.
[50] P.E. Tetlock, The impact of accountability on judgment and choice: toward a social contingency model, Adv.
Exp. Soc. Psychol. 25 (1992) 331 – 376.
[51] J. Fox, S. Das, Safe and Sound: Artificial Intelligence in Hazardous Applications, MIT Press, Cambridge,
MA, 2000.
[52] M. O’Connor, W. Remus, K. Griggs, Judgmental forecasting in times of change, Int. J. Forecast. 9 (1993)
163 – 172.
[53] P.B. Andreassen, S.J. Kraus, Judgmental extrapolation and the salience of change, J. Forecast. 9 (1990)
347 – 372.
[54] J. Hayes, C.W. Allinson, Cognitive style and its relevance for management practice, Br. J. Manage. 5 (1994)
53 – 71.
[55] L. Hudson, Contrary Imaginations, Methuen and Co., London, 1966.
[56] L.A. O’Hara, R.J. Sternberg, It doesn’t hurt to ask: effects of instructions to be creative, practical, or
analytical on essay-writing performance and their interaction with students’ thinking styles, Creat. Res. J.
13 (2) (2002) 197 – 210.
[57] D. Sumpf, Changing the perception of managers by art, Paper given at the European Academy of Manage-
ment 2nd Annual Conference on Innovative Research in Management, Stockholm, http://www.sses.com/
public/events/euram/complete_tracks/management_play/sumpf.pdf, 2002.
[58] R.A. Olsen, Desirability bias among professional investment managers: some evidence from experts,
J. Behav. Decis. Mak. 10 (1997) 65 – 72.
[59] W. Putz-Osterloh, Selbstreflexionen, Testintelligenz und interindividuelle Unterschiede bei der Bewaltigung
komplexer Probleme, Sprache Kognit. 4 (1993) 203 – 216.
[60] W. Putz-Osterloh, Uber Determinanten komplexer Problemloseleistungen und Moglichkeiten zu ihrer Erfas-
sung, Sprache Kognit. 2 (1995) 100 – 116.
[61] B. Berardi-Coletta, R.L. Dominowski, L.S. Buyer, E.R. Rellinger, Metacognition and problem solving: a
process oriented approach, J. Exp. Psychol. Learn. Mem. Cognit. 21 (1995) 205 – 223.
[62] J. McMackin, P. Slovic, When does explicit justification impair decision making? Appl. Cognit. Psychol. 14
(2000) 527 – 541.
[63] P. Presson, V. Benassi, Illusion of control: a meta-analytic review: 3, J. Soc. Behav. Pers. 11 (1996) 493 – 510.
[64] D.N. Stone, Overconfidence in initial self-efficacy judgments: effects on decision processes and perform-
ance, Organ. Behav. Hum. Decis. Processes 59 (1994) 452 – 474.
[65] E.D. Beinhocker, On the origin of strategies, McKinsey Q. Strategy 4 (1999) 47 – 57.
[66] P.E. Politser, Do medical decision analyses’ largest gains grow from the smallest trees? J. Behav. Decis. Mak.
4 (1991) 121 – 138.
 C. Harries / Technological Forecasting & Social Change 70 (2003) 797–817 817

[67] P. Goodwin, G. Wright, Enhancing strategy evaluation in scenario planning: a role for decision analysis,
J. Manage. Stud. 38 (2001) 292 – 304.
[68] A. Pearman, Two hundred and twenty-three years of decision research in Leeds: any progress? Professorial
Inaugural Speech, Available at http://www.leeds.ac.uk/decision-research/Papers/PearmanSpeech.pdf, 1993.
[69] J.C.E. Glenn, Futures Research Methodology, Version 1, American Council for the United Nations Univer-
sity: The Millennium Project, 1999.
[70] K. van der Heijden, Scenarios and forecasting: two perspectives, Technol. Forecast. Soc. Change 65 (2000)
31 – 36.
[71] J.C. Glenn, Introduction to the Futures Research Methodology Series, in: J.C.E. Glenn (Ed.), Futures
Research Methodology, Version 1, American Council for the United Nations University: The Millennium
Project, 1999.
[72] J.L. Corner, J.T. Buchanan, Capturing decision maker preference: experimental comparison of decision
analysis and MCDM techniques, Eur. J. Oper. Res. 98 (1997) 85 – 97.
[73] J. Payne, J. Bettman, E. Johnson, The Adaptive Decision Maker, Cambridge Univ. Press, Cambridge, UK,
1993.
[74] D. Kahneman, A. Tversky, Prospect theory: an analysis of decision under risk, Econometrica 47 (2) (1979)
263 – 291.
[75] C.K. Hsee, G.F. Loewenstein, S. Blount, M.H. Bazerman, Preference reversals between joint and separate
evaluations of options: a review and theoretical analysis, Psychol. Bull. 125 (1999) 576 – 590.
[76] H. Tsoukas, C. Knudsen, The conduct of strategy research, in: A. Pettigrew, H. Thomas, R. Whittington
(Eds.), Handbook of Strategy and Management, Sage, London, 2001, pp. 411 – 435.
[77] K.R. Hammond, T.R. Stewart, The Essential Brunswik, Oxford Univ. Press, New York, 2001.

Clare Harries is a university research fellow in Risk, Innovation and Strategic Decision Making at Leeds
University Business School and is part of the Centre for Decision Research. Her research interests lie in the
cognitive and metacognitive mechanisms involved in strategic decision making, judgmental forecasting, medical
decision making and advice taking and relationship between judgment and decision making.