PROCEEDINGS of the HUMAN FACTORS AND ERGONOMICS SOCIETY 47th ANNUAL MEETING—2003 449

SITUATION AWARENESS APPLICATIONS TO EXECUTIVE DASHBOARD DESIGN

Marc L. Resnick
Florida International University
Miami, FL
resnickm@fiu.edu

With the advent of mass customization, global financial markets, and Internet-based business models,
executive decision making has an increased need for speed, scope, and accuracy. To meet these needs, IT
vendors have created executive dashboards, systems that display visualizations of critical data via
interfaces that pull from corporate data warehouses. While the visualization techniques they use take
advantage of the latest technology and may support complex data analysis, these systems often fall short
because they do not match the schema of the decision maker nor do they support situation assessment or
awareness. A design process that considers a more naturalistic decision making perspective and uses
cognitive engineering techniques such as ecological interface design would provide a significant
improvement to the design of executive dashboard interfaces. However there are several challenges to
applying current cognitive engineering methods to the executive decision making domain. This paper will
present the executive decision making domain, illustrate how cognitive engineering principles are
necessary for the design of executive dashboards, and present some methodological challenges to applying
these principles.

INTRODUCTION come to a distressing conclusion. Often, the cost of

Naturalistic decision making (NDM) has been
studied in a variety of domains. Some involve the use of
complex interfaces, such as air traffic control (Rodgers,
Mogford and Strauch, 2000), nuclear power plants
(Roth, 2001) and commercial and military pilots
(Amalberti and Deblon, 1992). Some involve very little
technology, such as fireground commanders (Klein,
2000a). But regardless of the technological differences,
these domains have many things in common, including
complexity, uncertainty, significant consequences,
competing goals, and time constraints (Zsambok and
Klein, 1997).
An additional domain that shares many of these
characteristics is executive decision making. The speed
and accuracy with which business executives’ decisions
must be made has increased substantially over the past
decade. Reasons for this acceleration vary for each
industry, but include a decrease in the typical product
development cycle, integration of processes such as
production and development into enterprise systems, a
shift to Internet-based business models, and the
sensitivity of the stock market to fluctuations in a
company’s financial position. The similarities between
executive decision making and other NDM domains
suggest that similar techniques can be used to evaluate
the decision making processes and design support
systems and interfaces to accommodate them.
Sutcliffe and Weber (2003) provide a detailed
analysis of the knowledge needed by executives and
acquiring knowledge exceeds its value toward executive
decision making. The implications of raw data are
seldom obvious and need to be interpreted within the
context of collection circumstances and possible uses.
They find that the key requirement of executive
information systems is to provide a clear and consistent
framework from which the executive can create a
schema of the environment. As with pilots, fireground
commanders, and other complex decision makers,
identifying the general nature of the problem is key to
selecting the proper approach.
Sutcliffe and Weber (2003) directly correlated
decision making with macro indicators of business
performance and found that the accuracy of the
executive’s knowledge of particular details is less
important than having a structural schema that
maximizes the executive’s situation awareness and his or
her ability to improvise and adjust quickly. Becoming
“lost” in the raw data was often worse than not having
the data in the first place. This contrasts with the
approach of many executive dashboard designers which
is to maximize the amount of data users can access.
Errors on the opposite end of the spectrum are also
common. Bonabeau (2003) reports that executives
frequently use intuitive decision making styles. He
suggests that this stems from the natural pattern
recognition capabilities of the human information
processing system. The lack of specific training in
decision making reduces the ability of the executive to
recognize and overcome these processes. However, in
450 PROCEEDINGS of the HUMAN FACTORS AND ERGONOMICS SOCIETY 47th ANNUAL MEETING—2003

the complex environment of business decisions, intuition businesses (Kaplan and Norton, 2001). Each quadrant is
is frequently wrong. Notorious cases of successful divided into components , the number of which depends
intuition (such as Jack Welch of General Electric Corp.) on the particular strategy selected by the company.
often mask the more numerous cases of failures of Certain parameters are immediately visible, based on the
intuition in the business environment. Premature color of the display, and within each component there
commitment to a particular understanding of the are a number of indicators. Additionally, more detailed
environment narrows an executive’s thinlung and data, historical analysis and other advanced features are
reduces the alternatives that will be considered. available by drilling down into each component.
Ecological models can be used to identify the The kinds of decisions executive dashboards are
information sources that should be considered for each designed to support vary widely in scope and depth.
type of decision environment and focus the user more Some decisions focus on details from just one function
directly on the most diagnostic sources. This can while others require integration of data from several
increase the likelihood that natural pattern recognition functions. Hung (2003) also includes trend analysis and
processes and anchoring bias the user towards to the multi-dimensional analysis as typical capabilities of an
correct interpretation. executive dashboard; thus the interface must support
integrating substantial quantities of data in complex
ways. However, the decisions are not the same for each
EXECUTIVE DASHBOARDS
business. Depending on the strategy, industry, market
position and other factors, the information required to
In order to assist executives in making these
make decisions will vary. In order to support this
complex decisions, a new kind of computer system has
considerable breadth of requirements, executive
been developed - the executive dashboard. Executive
dashboards must be customizable for various types of
dashboards are systems that allow company executives
users and tasks.
to view key business facts, providing a circumspective
Current designs often have powerful data analysis
view that is designed to support effective decision
functionality, but are not based on cognitive engineering
making. Functionality can include completely
principles, either regarding the information content that
configurable displays based on ERP-connected data
is provided, how this information is organized for the
warehouses and analytic modules. Executive dashboards
user, or the design of specific displays and navigation
are intended to provide managers and executives with
controls. They often are used by executives with a great
the business intelligence they need to benchmark and
deal of business knowledge, but perhaps very little
improve business processes, motivate employees,
experience with executive dashboards or other decision-
increase organizational learning, and improve decision-
support technologies (Hung, 2003). And with the high
making to keep pace with or out-perform the
turnover among corporate executives that has become a
competition (Gledhill, 2002).
rule rather than a trend, experience with a particular
As with traditional dashboards for automobiles and
implementation of an executive dashboard and
aircraft, executive dashboards are composed of several
familiarity with specific data structures cannot be
gauges that represent various aspects of the system to be
assumed. However current executive dashboard designs
managed. Representations can include current status,
do not always support the prospective users’ task
history, rates of change, warnings and indicators, and
specific needs.
other formats depending on the data type and source.
The depiction of priority can be accomplished using
SITUATION AWARENESS
typical screen design techniques such as location, size
and color.
It would be much more effective if executive
However, unlike typical dashboards, the decisions
dashboards were developed to maximize the situation
that must be supported are much more complex and the
awareness (Endsley, 2000a) of the executive user. It is
amount of data that must be accessible can be enormous.
critical that the interface draws attention to the
Data from functions such as finance, operations,
parameters that are most influential in the current
marketing, sales, distribution, competitive positioning,
situation, facilitates the integration of these data into an
and others may be needed for certain kinds of high level
assessment of the business’ current situation, and assists
decisions, and thus must all be available to the executive
in the prediction of what is likely to occur next. These
in an intuitive and intelligible fashion.
correspond to the three levels of situation awareness
For example, Figure 1 presents a hierarchical
described by Endsley (2000b) as perception,
dashboard that is divided into quadrants based on the
comprehension, and prediction. Considering the
Balanced Scorecard method commonly used to manage
tremendous amount of data that is contained within an
 PROCEEDINGS of the HUMAN FACTORS AND ERGONOMICS SOCIETY 47th ANNUAL MEETING—2003
Figure 1. Sample screen from an executive dashboard (Qualitech Solutions, Inc.)
executive dashboard, the interface must support a
filtration process whereby executives’ attention is led to
the appropriate data. One serious implication of this
need is for the system to help the executive limit his or
her consideration to those data that are most relevant
(Pew, 2000).
Wickens’ (2000) use of SA to refer to a ‘continuous
extraction’ of information from the environment is
perhaps the most relevant to the executive decision
making domain. But while the research cited by
Wickens generally refers to tasks that take minutes and
hours, the executive is extracting information over
months and years and must maintain a consistent schema
of the competitive environment that evolves from some
events but persists despite extreme shocks from others.
For example, the situation awareness of executives
at major book retailers such as Barnes and Noble
regarding their competitive landscape went through
major changes when Amazon.com entered the market.
Amazon priced its books well below the prices found at
bricks and mortar stores and made a much wider
selection of books available. However the relationships
451
that the retailers had with its distributors, publishers,
franchisees, employees and other partners did not
change. During the period between 1998 and 2003, the
relative performance of Amazon and online retailing in
general went through many stages. As this new data
became available, the implications for bricks and mortar
retailers needed to be constantly reevaluated.
The extraction of information from data sources is
often driven by the user’s goals and expectations
(Endsley, 200 1). Considering the extended durations of
executive decision making situations, expectations are
more prevalent and more strongly ingrained into the
executive’s schema, thus creating more opportunity for
bias when the expectations are unfounded. Any support
system must help executives identify these biases and
perhaps overcome them. This is particularly relevant
when a potentially large shift in the market occurs.
For example, when Amazon began selling books
over the Internet, the traditional retailers were forced to
deal with this completely new model of competition.
However, the ingrained marketing channels, distribution,
vendor relationships, inventory control and other
 452 PROCEEDINGS of the HUMAN FACTORS AND ERGONOMICS SOCIETY 47th ANNUAL MEETING—2003
business processes they had were not going to disappear
either from reality or from the strong decision making
schema that executives had developed over thousands of
repetitions (Zsambok, 1997). Executive dashboards can
help executives make better strategic decisions by
focusing attention on those factors that are relevant to
each kind of decision - some of which must be radically
reconfigured in the new Internet-based market while
others should not.
SA METHODS FOR EXECUTIVE DASHBOARD
DEVELOPMENT
Unfortunately, many of the methods currently used
to evaluate SA in other domains are not feasible in the
executive decision making domain, both from content
and methodological perspectives. Companies are very
different from each other in terms of strategic goals,
organizational structures, operational processes, and the
products and services they offer. Within each company,
there are only a few individuals who are responsible for
executive level decisions. Even within each functional
area (ie. marketing), there are only a few people who are
involved in the highest level strategic decisions.
Therefore creating an evaluation scenario with a large
enough sample population to collect meaningful data is
rarely possible. Whereas 747 pilots interact with very
similar challenges using similar interfaces regardless of
what airline they work for, this cannot be assumed for
marketing executives selecting a brand strategy.
Therefore new techniques that can be used with much
smaller sample sizes must be developed. Pritchett and
Hansman (2000) describe the advantages and
disadvantages of several SA methods, but they do not
consider the sample sizes required.
A second difference is the timing of the decision
making process. Although in many cases the actual
decision must be made quickly, the development of the
situation awareness occurs over months and years.
Events that occurred in past times may not only
contribute to general domain knowledge but also may be
directly relevant to the current decision.
c
-- Again considering the domain of book retailing, the
,JILranceof Amazon into the market affected pricing and
delivery, but the marketing of each genre of book to
particular user groups did not change. Selecting an
online marketing strategy for each user group requires
combining the new and old information into one
integrated schema.
This is problematic because running simulations that
take months to complete is not a feasible analysis
method. Direct measurement methods such as those
described in Endsley (2000~)would not be feasible.
Perhaps critical incident reports (Klein 2000b) and other
post-hoc methods (Rodgers, Mogford, and Strauch,
2000) are the most appropriate.
The personalities of the executive users also may
challenge the use of SA methods in ways that are not as
problematic in traditional SA domains. Subjective
methods require participants to reflect on how confident
they are in their SA (Jones, 2000). However executives
may be less able to introspect their SA than pilots and
commanders, in part because of the personalities that
tend to obtain executive level positions and in part
because of an unwillingness to concede a lack of
awareness. The long duration of each decision unit may
also create challenges. SA may vary greatly between the
beginning and end of a single decision, both in reality
and in subjective perception.
NEW SA METHODS
As a result, there is a need to extrapolate the findings
of SA research from other areas to gain insight into
executive dashboard design. SA knowledge can be
drawn from past research into the design of systems and
displays for pilots and air traffic controllers, training of
individuals and teams, and others to design executive
dashboards that facilitate the development of SA for
business executives. But this extrapolation must be done
carefully by experts in both SA and the business domain
to insure that it is applied appropriately.
Reising and Sanderson (2002) describe a
methodology using ecological design to create an
interface for an industrial decision support system that
accurately maps the semantics of the applied
environment directly to the display. Similar techniques
can be used specifically for the creation of executive
dashboards.
Testing of executive dashboard systems to evaluate
their support of SA must also evolve to fit the needs of
the business domain. One cannot test a generic
implementation of an executive dashboard across several
companies because the interface must be significantly
customized for each one in order to be relevant. The
effectiveness of these systems is limited out of context.
Therefore more of a case study method that evaluates
qualitative data must be used. Effective methods do not
currently exist and must be developed specifically for
this domain.
CONCLUSION
The executive decision malung domain is a rich
environment for cognitive engineering because of the
complexity of the decision making environment and the
importance of these decisions to the performance of the
 PROCEEDINGS of the HUMAN FACTORS AND ERGONOMICS SOCIETY 47th ANNUAL MEETING—2003
company and often to national economies. However the
current architecture of executive dashboards fails to take
advantage of what cognitive engineering can provide.
Cognitive engineering principles such as situation
assessment and awareness are critical for successful
executive decision making and can be used to improve
executive dashboard design. But SA research methods
must be modified to investigate executive decision
making effectively. Executive decision making is a new
and challenging domain for SA research and practice.
REFERENCES
Amalberti R. and Deblon F. (1992). Cognitive Modeling of
fighter aircraft process control: a step towards an intelligent
on-board assistance system. International Journal of Man-
Machine Studies. 36,639-671.
Bonabeau E. (2003). Don't trust your gut. Harvard Business
Review, May, 116-122.
Endsley M.R. (2000a). Theoretical underpinnings of situation
awareness: A critical review. In M.R. Endsley and D.J.
Garland (eds). Situation Awareness Analysis and
Measurement. Lawrence Erlbaum Associates: Mahwah, NJ.
Endsley M. R. (2000b). Situation Assessment Analysis and
Measurement. Lawrence Erlbaum Associates: Mahwah, NJ.
Endsley M.R. (2000~).Direct measurement of situation
awareness: validity and use of SAGAT. In M.R. Endsley
and D.J. Garland (eds). Situation Awareness Analysis and
Measurement. Lawrence Erlbaum Associates: Mahwah, NJ.
Endsley M.R. (2001) Designing for situation awareness in
complex systems. Proceedings of the Second International
Worhhop of Symbiosis of Humans, Artifacts and
Environments, Kyoto, Japan.
Gledhill J. (2002). Create value with IT investments: how to
generate a healthy ROI across the enterprise Food
Processing, Sept 2002 v63 i9 p76-81.
Hung S.Y. (2003). Expert versus novice use of the executive
support systems: an empirical study. Infirmation &
Management, 40, 3, 177-189.
Jones D.G. (2000). Subjective measures of situation
awareness. In M.R. Endsley and D.J. Garland (eds).
Situation Awareness Analysis and Measurement. Lawrence
Erlbaum Associates: Mahwah, NJ.
453
Kaplan R.S. and Norton D.P. (2001). The Strategy-Focused
Organization. Harvard Business School Press: Cambridge,
MA.
Klein G. (2000a). Sources of Power. MIT Press: Cambridge,
MA.
Klein G. (2000b). Analysis of situation awareness from
critical incident reports. In M.R. Endsley and D.J. Garland
(eds). Situation Awareness Analysis and Measurement.
Lawrence Erlbaum Associates: Mahwah, NJ.
Pew R.W. (2000). The state of situation awareness
measurement: heading toward the next century. In M.R.
Endsley and D.J. Garland (eds). Situation Awareness
Analvsis and Measurement. Lawrence Erlbaum Associates:
Mahwah, NJ.
Pritchett A.R. and Hansman R.J. (2000). Use of testable
response for performance-based measurement of situation
awareness. In M.R. Endsley and D.J. Garland (eds).
Situation Awareness Analysis and Measurement. Lawrence
Erlbaum Associates: Mahwah, NJ.
Reising D.V.C. and Sanderson P.M. (2002). Ecological
Interface Design for Pasteurizer 11: A process description of
semantic mapping. Human Factors, 44,2,222-241.
Rodgers M.D., Mogford R.H. and Strauch B. (2000). Post hoc
assessment of situation awareness in air traffic control
incidents and major aircraft accidents. In M.R. Endsley and
D.J. Garland (eds). Situation Awareness Analysis and
Measurement. Lawrence Erlbaum Associates: Mahwah, NJ.
Roth E.M. (2001). Situation Assessment Panel. Presented at
the Human Factors and Ergonomics Society 45thAnnual
Meeting. Human Factors and Ergonomics Society, Santa
Monica, CA.
Sutcliffe K.M. and Weber K. (2003). The high cost of accurate
knowledge. Harvard Business Review, May, 75-82.
Wickens C.D. (2000). The trade-off of design for routine and
unexpected performance: implications of situation
awareness. In M.R. Endsley and D.J. Garland (eds).
Situation Awareness Analysis and Measurement. Lawrence
Erlbaum Associates: Mahwah, NJ.
Zsambok C.E. (1997) Naturalistic decision making:
Where are we now? In Naturalistic Decision Making.
C.E. Zsambok and G. Klein (eds). Lawrence Erlbaum
Associates: Mahwah, NJ.
Zsambok C.E. and Klein G. (1997). Naturalistic Decision
Making. Lawrence Erlbaum Associates: Mahwah, NJ.