Correspondence

Coming to Grips with

Measures of Effectiveness
Noel Sproles

Systems Engineering and Evaluation Centre, University of South Australia, The Levels Campus, Mawson Lakes, South
Australia, 5095
COMING TO GRIPS WITH MEASURES OF EFFECTIVENESS

Received July 14, 1999; accepted November 24, 1999

ABSTRACT
Although “Measures of Effectiveness” (MOEs) are referred to in systems engineering docu-
ments such as EIA/IS-632, there is no universally accepted meaning for the term. This paper
provides an overview of this important topic by suggesting a definition of MOEs. The role of
MOEs as the standards for identifying successful solutions is examined. They are shown to be
“mission” or “purpose” oriented and not concerned with the internal details of the solutions
per se. A clear distinction is drawn between the MOEs as established by the stakeholder and
the quantified MOE, or figure of merit, established as a result of the analysis and evaluation
of particular solutions. MOEs are seen to be the precursor to, hence the “engines” of, test and
evaluation and of criteria. A study of the roles of other forms of “measures” enables a clearer
understanding of the role and nature of MOEs to be established. © 2000 John Wiley & Sons,
Inc. Syst Eng 3: 50–58, 2000

1. INTRODUCTION MOEs are a vital part of the SE process. They are an

The interim standard, EIA/IS-632 [ Electronic Indus-
tries Association, 1994], included Measure(s) of Effec-
tiveness (MOEs) as part of the “process input” to the
Systems Engineering Process Model. While most of the
other inputs appear to be self-explanatory, no explana-
tion or definition of MOEs is offered by the interim
standard. A literature search will provide a plethora of
definitions for MOEs, none of which are universally
accepted. To add to the confusion, there is a tendency
to interchange other terms such as Measure of Perform-
ance (MOPs) and Metrics with MOEs.
important management tool not only for the stakehold-
ers who initiate the process input but for the systems
engineers who must provide a solution for the stake-
holders. It is important that there be an awareness of
what is meant by the term MOE and how it can assist
in the selection of the correct solution to the stakehold-
ers need. This paper will provide an overview of the
topic of MOEs by establishing the stakeholders preemi-
nent position in the process. It will suggest a definition
of MOEs as the standard by which the stakeholders
establish if a solution will meet the need. A discussion
of the various forms that an MOE can take and the other
types of measures used will serve to further clarify the
role and nature of MOEs.

© 2000 John Wiley & Sons, Inc.

50
 COMING TO GRIPS WITH MEASURES OF EFFECTIVENESS
2. REVIEW OF THE LITERATURE
Roche and Watts [1991: 198] trace the origin of the
acronym MOE to Morse and Kimball who used the term
“measures of effectiveness” in 1946. Morse and Kim-
ball are regarded as the founders of operations research
for their work with the U.S. Navy (USN) during WWII.
They carried out extensive studies of the effectiveness
of weapons systems and strategies during that period
using data gained from actual operations. Given this
long history, it is surprising that so little scholarly work
has been written regarding MOEs. While there are
numerous mentions of MOEs in various documents and
even attempts at defining MOEs, there is not what could
be called a body of literature on the topic. In most
instances, MOEs are referred to in the context of pro-
cedures and legislation peculiar to a particular organi-
zation. In some cases, MOEs are even referred to by
different names.
It would be expected that the management literature
would address this topic because MOEs are an element
of planning and control, which are two of the functions
of management [Dale, 1969]. Authors in business man-
agement do refer to MOEs, although they do not call
them by this name. Instead they use terms such as
“customer satisfaction” or “customer value” [Nau-
mann, 1995]. Recognition is given to the necessity of
getting close to the customer, of knowing customer
requirements, and even of knowing the relative impor-
tance of the customer’s decision criteria. However, the
business management literature does not address how
the customer arrives at these values or criteria (MOEs)
in the first place. Other business writers, such as Dem-
ing [1982], concentrate on quality control and take
customer value criteria as a given. Given this accep-
tance of the importance of customer values to manage-
ment and the exhortation to “move close to the
customer” in order to ascertain these values, the lack
of scholarly works on how to arrive at them repre-
sents a significant gap in the literature. A formal
process of establishing customer values (MOEs)
would seem to be a logical step in a process of
improved communications between the customer and
the management of providing a solution meeting
those values.
In what may loosely be termed the engineering
literature, several authors were found who referred to
MOEs, although their references were brief and made
in the context of other topics. Keeney and Raiffa
[1976] in writing on decision analysis do devote a
chapter to “Objectives and Attributes.” The “attrib-
utes” in this instance are MOEs, although such a
connection between “attributes” and MOEs, as op-
posed to “properties,” is disputed. The chapter is
51
mainly devoted to the topic of objectives, although there
is some sound discussion of MOEs supported by clear
examples. However, the discussion is of a general na-
ture and, as Keeney et al. have noted, it represents a brief
set of guidelines only. Stevens [1986] devotes just two
pages to MOEs in his text on Test and Evaluation
(T&E). It too is a valuable source, albeit brief and
without much supporting explanation. The emphasis
placed by Stevens [1986] on the use of MOEs is in the
particular case of developing T&E programs for items
of military equipment. Broader issues and uses for
MOEs are not addressed. The largely procedural docu-
ments on equipment acquisition for the U.S. Depart-
ment of Defense (DoD) also refer to MOEs, particularly
in relation to the higher levels of the process of acquir-
ing major items, without explaining how MOEs are to
be formulated. The U.S. Federal Transit Administration
[FTA, 1996] applies MOEs in a study of urban transit
systems and provides a useful demonstration into the
use of MOEs but does not explain what they are. The
U.S. Navy provides a definition for MOEs as “…those
indicators of how well process objectives are being
met” [COMNAVSURFLANT, 1994]. This definition
refers to the role of MOEs in telling “how well” a
solution meets the stakeholder’s requirements or in
telling if success has been achieved or not, but does not
make it clear as to what exactly constitutes “indica-
tors.” The U.S. Defense Systems Management Col-
lege [DSMC, 1996] also provides a definition of
MOEs, but it is quite specifically directed at the
capabilities of military items and is not appropriate
for a generic definition. Linas and Waltz (in Hall
[1992: 278]), when discussing data fusion systems,
provide a useful definition of MOEs as does Lei-
bowitz [1988: 127–130].
In the absence of a body of literature, information
relevant to MOEs must be sought by reference to
authors in a disparate group of disciplines. These do
not, however, represent a body of work on the topic
of MOEs per se. Extended reading of works in a
range of disciplines is necessary in order to develop
a model for the formulation and use of MOEs, for
example, into cognitive psychology for the manner
in which humans resolve problems or into decision
analysis for the resolution of subjective issues. In the
process of completing these readings, it becomes appar-
ent that MOEs are not the exclusive domain of just the
engineering disciplines. MOEs are equally applicable
to all disciplines and are a manifestation of good man-
agement practices. As such, MOEs may owe more to
management than to any other discipline.
52 SPROLES
3. WHAT ARE MEASURES OF
EFFECTIVENESS?
3.1. Criteria for Success
Given that satisfying a need presents a problem to be
solved, there is general agreement in the literature that
the three elements necessary for a solution are problem
definition, problem resolution (i.e., generation of can-
didate solutions), and analysis or T&E of the solutions.
In the case of a simple problem, each stage could be
performed by the same entity, but in more complex
problems these entities are most likely to be separate.
Many models for this process have been produced,
typical of which is Finkelstein and Finkelstein’s [1983]
“general model” for solving design problems. This
model was generated from: “…the broad agreement
that exists in the literature on the elements of such a
process…” [Finkelstein and Finkelstein, 1983: 216].
This process, illustrated at in Figure 1, is iterative and
comprises five steps initiated by a needs statement of
some kind. The iteration between phases, where the
problem definition and possible solutions are defined
and redefined by systematic analysis, is a feature of all
such models and is reflected in the SE process model of
EIA/IS-632 [Electronic Industries Association, 1994].
Figure 1. Stages of the design process—after Finkelstein
and Finkelstein (1983).
“Information gathering and organization” and
“value models,” as illustrated in Figure 1, may be
grouped together as “problem definition” and represent
the initial phases of the System Life Cycle of EIA/IS-
632 [Electronic industries Association, 1994]. It is the
“value model” which formulates “…the criteria which
arise from the requirement and by which [the] candidate
design may be evaluated” [Finkelstein and Finkelstein,
1983: 216]. MOEs are intended to form a part of this
area by ensuring that the evaluation process establishes
if the solution meets the need.
3.2. No Universal Definition
MOEs are a little-recognized part of this process of
defining a problem or need, and the seeking and selec-
tion of solutions. Keeney and Raiffa [1976: 32] state
that: “…there are no universal definitions for the terms
… measures of effectiveness….” Yet, of the various
definitions attempted of MOEs to be found in the litera-
ture, a thread of general agreement can be found run-
ning through them indicating that:
• MOEs represent the viewpoint of the stakehold-
ers, i.e., “those who have a right to impose re-
quirements on a solution”;
• MOEs assist in making the right choice by indi-
cating “how well” a solution meets the stakehold-
ers need; and
• MOEs should be able to be quantified in some
manner.
It is the concept of MOEs being a part of the process
of indicating “how well” something is done or how well
some entity is prepared to do something which pervades
most definitions of the term. MOEs are seen as the
“standards” against which the “goodness” of a solution
may be assessed. A study of the meanings of the sepa-
rate terms “measure” and “effectiveness” will establish
that the term “measures of effectiveness” elegantly
incorporates these notions
3.3. Measure
In English, “measure” has many meanings and nuances
to those meanings as well as being both a noun and a
transitive verb. It is only as a noun that it is used in the
situations of interest, for example, “…the measure of
effectiveness is...” or “...to establish a measure of effec-
tiveness for this situation, etc.” The various definitions
of MOEs show that, when associated with “MOEs,” the
term “measure” tends to be given two separate mean-
ings. The more common meaning is that of measure as
a standard; the other meaning is that of measure as a
quantity.
 COMING TO GRIPS WITH MEASURES OF EFFECTIVENESS
3.4. Effectiveness
The Oxford English Dictionary [1989] defines effective
as: “...concerned in the production of an event or con-
dition...” and “...concerned with, or having the function
of, carrying into effect, execution or accomplishing.”
This is confirmed by The Cambridge International
Dictionary of English [1995] which illustrates the use
of “effective” by saying: “Something can be described
as effective if it produces the results that it was intended
to: ‘It is a very effective cure for a headache.’” Further
analysis shows that an MOE is the measure of the
property of “being effective,” i.e., of “producing the
intended results” or “as an indicator of how well some-
thing is done.” Because an MOE represents the views
of the stakeholders, the “intended results” refer to how
well the solution meets the stakeholders needs. MOEs
are therefore “mission-oriented” standards directed at
the solution’s purpose rather than at the solution per se.
4. STAKEHOLDERS—FORMULATORS OF
MEASURES OF EFFECTIVENESS
Establishing exactly who are the stakeholders is not an
easy task. Eastern societies make a great effort to ensure
that all those who have a reason to be involved are
consulted in defining the user requirements. In Indone-
sia, the Javanese term “musyawarrah” is used to de-
scribe this mutual consultative process. In Western
societies, attempts to formalize this concept are seen in
the literature covering “soft systems” with writers such
as Jackson and Checkland being prominent. The more
positivistic Western approach is called “hard systems”
and is sometimes considered to be a subset of the “soft
systems” approach. Whatever process is used to estab-
lish who the stakeholders are and their user require-
ments, the output must be considered “hard” else the
users will have failed to establish their requirements to
the degree that others can devise practical solutions to
their need.
Stakeholders are therefore defined as those who,
either through “hard” or “soft” methodology, have dem-
onstrated their need and willingness to be involved in
seeking a solution. They will want to be able to answer
the question “How will I recognize if a solution meets
my needs?” or “How will I recognize success?” The
term “stakeholders” is preferred to “customers” if only
because a stakeholder may not necessarily be an end
user, which is the connotation usually associated with
“customer.” For example, many people, such as neigh-
bors or local governments, may have a right to impose
requirements on a proposed high rise development,
although they have no intention of using the completed
development. They will never be the “customers” of the
53
development. Procedures such as the preparation of
environmental impact statements recognize the exist-
ence of a wider range of involved parties other than
“customers.”
It is the prerogative of the stakeholders to determine
the MOEs, for the final choice is ultimately their deci-
sion and responsibility. It is the stakeholders who must
determine whether a solution will, or will not, meet the
specified need. This is reflected in the interim standard
by including MOEs in the process input phase as part
of the ‘givens’ to be manipulated in the Systems Engi-
neering (SE) process model. An MOE is a standard by
which the stakeholders will judge success and so must
be the responsibility of the stakeholders.
4.1. User Requirements
Stevens et al. [1998: 22] draws a distinction between
user and system requirements. User requirements refer
to what the system (or solution) will provide the stake-
holders while the system requirements impose “...re-
quirements on an abstract model of the final system.”
Aslasken [1996: 48] divides requirements into four
categories of:
• functional requirements which relate to the pur-
pose or reason for the solution;
• performance requirements establishing how well
the functions are to be performed;
• commercial requirements detailing economic
matters such as local content and cost; and
• boundary conditions detailing color, meeting of
standards, etc.
MOEs have their genesis in the user requirements of
Stevens et al. [1998] for these requirements are con-
cerned with establishing if a candidate solution will
meet the stakeholders’ needs. The functional require-
ments described by Aslasken (1996) fall within the
definition of user requirements because they too are
concerned with the purpose or reason for the solution,
i.e., the need.. On the other hand, MOEs are not con-
cerned with the other categories of requirements de-
tailed in Aslasken [1996] because, important as they
are, they do not dictate the function of the solution. To
establish if a candidate solution to a need successfully
meets that need, a stakeholder must first establish if it
is functionally successful.
While there may be room for compromise on the
boundary conditions or commercial requirements, the
functional requirements, if properly established, must
be met; otherwise the proposed solution will not do
what is required of it. Determining if functional require-
ments are met is achieved by putting the solution into
54 SPROLES
its operational role and testing it in that role. If it is not
feasible to test it in its operational role, then it will be
necessary to test it using alternative means such as by
the use of probable scenarios or in simulations. Such
testing is not needed for boundary conditions or com-
mercial requirements.
4.2. MOEs Should Not Dictate a Solution
The user requirements refer to the function of a solution
able to meet the stakeholders’ need. Ideally, they should
be stated so as not to constrain the solution provider to
any particular course of action. By specifying require-
ments based solely on the need rather than a solution,
the stakeholder leaves open the possibility of creative
solutions or solutions which the stakeholder may not
have previously considered. Even when a stakeholder
decides the nature of a solution, for example, it will be
an aircraft as opposed to a ship, the user requirements
should still be based on the need to be met and not the
performance requirements of the solution. The former
enable MOEs to be formulated; the latter enable Meas-
ures of Performance (MOPs) to be formulated.
4.3. A Definition
A definition of MOEs incorporating these concepts
may be formulated as:
Measures of Effectiveness are standards against
which the capability of a solution to meet the needs of
a problem may be judged. The standards are specific
properties which any potential solution must exhibit to
some extent. MOEs are independent of any solution and
specify neither performance nor criteria.
4.4. Examples of Measures of Effectiveness
Rich and Janos [1998: 108] provide a graphic example
of an MOE with the use of the F-117 stealth fighter in
Desert Storm. The stakeholders’ need was the classical
military requirement to achieve surprise, and stealth
technology applied to an aircraft was a possible solu-
tion. The crews made the observation that, after the
initial raid over Baghdad, “No one had suffered as much
as a hit or even a near miss.” The MOE which they had
applied for the effectiveness of stealth technology could
be interpreted as “the probability of being hit by hostile
fire.” Their reasoning was that if stealth worked, the
aircraft could not be seen and so could not be hit.
4.5. Statements Not Questions
Each MOE is presented as a statement and not a ques-
tion, for they represent a standard, or yardstick, against
which a solution may be judged. From this it follows
that they are able to form the basis of a test and evalu-
ation program for a number of candidate solutions.
5. QUANTIFIED OR NOT?
As already noted in the discussion of the term “meas-
ure,” it is undecided whether MOEs are a quantified
entity or not. This question can be a major source of
confusion, especially to the novice practitioner. For
example, when the statements “probability of success”
and “the probability of success is 58%” are used, which
is the MOE? Is it “probability of success” or is it
“58%”? Is it a statement or is it a number? Instances can
be found in the literature where both are called MOEs.
This confusion arises due to the use of the term MOEs
for each of what Pinker, Samuel, and Batcher [1995: 8]
refer to as the four activities usually associated with
MOEs:
• the selection of an MOE;
• the quantification of the MOE;
• the evaluation of the MOE;
• the combining of MOE values as the basis of an
overall decision.
Of these four activities, the selection of MOEs is the
stakeholders responsibility, as has been noted, and
forms part of the process input to the SE model: It is in
effect one of the “givens” from which the SE will
develop a solution. The remaining three activities result
from the implementation of the SE process model
through the T&E procedures carried out for validation
and verification during the System Life Cycle and are
largely out of the stakeholders control. They are a result
of the SE process.
The unquantified statement, which expresses the
stakeholders views of the properties which any success-
ful solution to the need must possess, is a concept; it is
associated only with the stakeholders; and it gives a
direction for the verification and validation phase on
which tests can be planned. It is the unquantified state-
ment that leads to the establishment of criteria and the
tests to establish the existence of the properties. When
quantified, it is no longer a concept but fact. It is the
resultant of the tests dictated by the MOEs; it is peculiar
to only one solution; it becomes the purview of the likes
of mathematicians and operational analysts using tech-
niques and processes such as multiattribute analysis and
Saaty’s Analytic Hierarchy Process [Saaty, 1980]. The
quantity obtained as a result of MOEs and T&E is now
something to be evaluated against a set of criteria. While
both the unquantified and quantified entities still refer
to effectiveness, they are using “measure” in a different
 COMING TO GRIPS WITH MEASURES OF EFFECTIVENESS
manner. When used in the unquantified statement,
“measure” is intended to have the meaning of “measure
as a standard.” On the other hand, when quantified,
“measure” is being used with the meaning of “measure
as a quantity.” Because the very nature of the unquan-
tified statement and the quantified value are so different,
they each warrant separate identities.
Dockery [1985: 25] refers to MOEs, when quanti-
fied, as “effectiveness measures.” Pinker, Samuel, and
Batcher [1995: 8] refer to a quantified MOE as a “figure
of merit” which “...reflects the degree to which a desired
goal was attained.” The adoption of the term “figure of
merit” for the quantified MOE is suggested, for it is both
an elegant description of the nature of the number and
a means to make the distinction clear. It is also sug-
gested that the term “MOE” be restricted to the unquan-
tified statement made by the stakeholders with the
meaning that MOEs establish the standards by which a
successful solution is judged. It is in this sense that the
term MOE is used in the interim standard which refers
to “...customer (stakeholders) provided measures of
effectiveness” (EIA/IS-632 [Electronic Industries As-
sociation, 1994: 9]) and it is in this sense that it is used
in this paper.
5.1. Examples of Figures of Merit
Table I shows examples of Figures of Merit (FOMs),
based on an MOE of “number of residents displaced by
site acquisition (based on) 1981 census.” This MOE is
one of a set for the selection of a site for the Second
Sydney Airport [Kinhill Stearns, 1985: 128]. Each fig-
ure of merit shows the number of residents who would
have to be displaced from each site if the airport was
built on that particular site.
A figure of merit is unique to a particular candidate
solution, which in this example is a possible airport site.
They can only be established after an analysis, or T&E,
of each solution. The figures are meaningless unless
cross-referenced to the initiating MOE or standard. The
manner of expression of the example MOE referring to
the F-117 fighter and that of the figures of merit shown
in Table I highlight the fundamental differences be-
tween these two entities.
Table I. Rank Order of Sites Based on the MOE for Displacement of Residents
Site Bringelly Badgerys Creek
FOM 4250 1350
55
6. FORMULATING MEASURES OF
EFFECTIVENESS
In order to formulate an MOE, it is necessary to estab-
lish from the user requirements what properties are
essential for a successful solution to possess in the form
of critical issues. It means asking the question “If a
candidate solution cannot do this, would I reject it?” If
the answer is “yes,” then it is a critical issue and the
basis for an MOE. If not, it is a “nice to have” or one of
the other categories of requirements described by
Aslasken [1996] and should be categorized as such. In
the example of an MOE applied to the F-117 stealth
fighter, a critical issue could have been “Will the enemy
be able to detect the aircraft sufficiently well to enable
it to be destroyed or damaged?” If the answer is ulti-
mately shown to be “yes,” then stealth technology
would be no better than current technology, and it
would be a failure and unacceptable as a solution. If
“no,” then it had passed at least one hurdle.
This approach has a singular advantage. By limiting
MOEs to that which is essential, to the “show stoppers,”
it is probable that the number of MOEs for any one
project will be small. It will enable the T&E effort to
focus on what is essential and save resources by elimi-
nating nonessential testing. If it is decided to test non-
critical properties, then it can be done in the clear
understanding that it is separate from establishing if the
candidate solution meets the stakeholders’ needs. What-
ever measures are established as the basis for this testing
or evaluation, they should not be called MOEs as this
would be a misnomer.
Formulating the actual MOE is a cognitive process
and is dependent on very human characteristics, espe-
cially that of lateral thinking. Because of this, different
people may produce different MOEs for the same set of
user requirements. An example of this can be seen from
the need to clean an urban storm-water system. One
possible MOE is the state of the frog population in the
inland waterways affected by the storm-water system.
A water system with a low level of pollution means that
the frogs are healthier and their population will in-
crease. Another possible MOE is the state of sea grasses
near the storm-water outfalls. The more chemicals and
nutrients in the outfalls, the more the sea grasses die off.
To further illustrate this point, an example of an MOE
Holsworthy Goulbourn
1230 20
56 SPROLES
for stealth technology aircraft has already been pro-
vided. Yet another MOE for this project could have
related to the probability of bats being able to detect the
aircraft. Rich and Janos [1998: 106] noted the large
number of dead bats in the F-117 hangars each morning
as a result of the bats flying into the parked F-117s after
their sonar failed to detect the aircraft! It is therefore
possible that a range of MOEs could be formulated for
any project and the only limit is the human imagination.
7. THE “ENGINE” OF TEST AND
EVALUATION AND CRITERIA
7.1. Test and Evaluation–Analyzing
Possible Solutions
T&E, or what Finkelstein and Finkelstein (1983) refer
to as “analysis of candidate designs,” is the central
element of the iterative process of finding a satisfactory
solution to a problem. The Georgia Institute of Technol-
ogy [ACTE, 1996] defines T&E as: “A scientific and
systematic process to support the assessment of the
quality or ‘goodness’ of a product with known confi-
dence.”
While T&E may be used to assess the “quality and
goodness” of an entity, that which constitutes “quality
and goodness” needs to be established before T&E can
commence. It would be pointless to hand an item to the
test agency and say “Test and evaluate this for me to see
how well it meets my needs” without also providing
some indication as to the goal of the test. MOEs enable
the stakeholders to provide the T&E agency with the
information necessary to establish what is quality or
what is the “goodness” of a choice: MOEs enable the
test agency to plan the test program and so they may be
considered the “engines” of T&E.
7.2. Criteria
Criteria bind the properties specified in MOEs. They
provide the limits within which the figures of merit for
a successful solution must lie. The MOEs identify the
properties to be tested: Criteria identify the limits within
which a successful solution will lay. An orderly process
will demand that MOEs be formulated before criteria, and
so, in this sense, MOEs are also the “engine” of criteria.
While criteria are needed for the evaluation process,
they are specified separately from the MOEs. By sepa-
rating criteria from MOEs, it is possible to reuse MOEs
when technology changes, for it is possible for MOEs
to have a long life and even for libraries of proven
MOEs to be established. Keeping criteria separate from
MOEs ensures that they are not tied to any particular set
of solutions or level of technology.
8. MEASURES OF PERFORMANCE
8.1. A Common Term
The term “Measure of Performance” (MOP) is one
which is often found in the literature and is repeatedly
associated with MOEs. Like much of the terminology
allied with this subject, there is no universally accepted
meaning for the term leading to confusion about its use.
Cases can be found where the terms MOE and MOP are
interchanged and even indiscriminately interchanged
within the text of the same article! There are important
distinctions between the two terms which need to be
recognized and maintained. A study of these distinc-
tions will serve not only to illustrate the source of
confusion but will also serve to enhance an under-
standing of the roles of MOEs.
8.2. Performance
Like “effectiveness,” “performance” is a noun but is the
name of an action. The Oxford English Dictionary
[1989] defines performance as “the action of perform-
ing, or something performed. The carrying out of a
command, duty, purpose, promise, etc.; execution, dis-
charge, fulfilment” and notes that it applies to the
“...capabilities of a machine or device, now esp. those
of a motor vehicle or aircraft measured under test and
expressed in a specification.” The Chambers English
Dictionary [1988] has a similar definition and states
that performance is a noun and is “...the act of perform-
ing; a carrying out of something; something done...an
act or action: the power or capability of a machine (esp.
a motor vehicle) to perform.” Cambridge [1995] de-
fines performance as “the performance of a person or
machine is how well they do a piece of work or activity.
‘These cars have a reputation for poor performance.’”
In defining the verb “to perform,” Cambridge (1995)
states: “If something such as a machine performs well
or badly, it operates satisfactorily or does not operate
satisfactorily. ‘The equipment performed well during
the tests.’”
8.3. Relationship between Measures of
Effectiveness and of Performance
The distinction between “effectiveness” and “perform-
ance” shows that MOEs and MOPs are formulated from
different viewpoints. An MOE refers to the effective-
ness of a solution and is independent of any particular
solution; an MOP refers to the actual performance of an
entity. The relationship between the two types of meas-
ures can be described using the analogy of the relation-
ship between effectiveness and efficiency.
Effectiveness is how well something does its job. Effi-
 COMING TO GRIPS WITH MEASURES OF EFFECTIVENESS
ciency is how well something does what it is doing.
Therefore, efficiency can be high while effectiveness is
low or even zero. Something can be done well even
though it is the wrong job which is being done. An MOE
will indicate a property which a potential solution must
possess in order to meet a need: An MOP will tell what
something is capable of doing, even if this is not neces-
sarily what the stakeholders want it to do.
The difference between “effectiveness” and “per-
formance” as applied to a solution to a need is that
“effectiveness” is a quality of fitness for service or of
producing the results for which it was intended. “Per-
formance” is the quality of “doing something,” and
“doing something” does not necessarily indicate fitness
for service. Cambridge [1995] says that performance is
how well a person or machine does a piece of work or
activity. It does not require that this “piece of work or
activity” be anyway connected with the quality of
something being fit for service or of it achieving the
results for which it was intended. An MOE is a standard
against which we judge how well we achieved what we
intended or wanted to achieve; an MOP is a measure of
what was achieved. The qualities of “effectiveness” and
of “performance” are two separate entities in that the
MOE refers to the stakeholders intention whereas the
MOP is concerned with actual performance which may
be quite divorced from the stakeholders intentions.
Another view is expressed by Simon [1984], who
proposes the division of the environment into an outer
and an inner environment, an idea shared by Champy
[1995]. Essentially they suggest that the stakeholders
view an artifact from the outside and are not necessarily
aware or concerned with what is inside the artifact. This
is akin to the notion that the stakeholders, hence the
MOEs, are concerned with the ‘mission” or purpose of
the solutions and not necessarily the solutions per se.
This justifies the view that stakeholders should avoid
dictating the solution and leave it to those responsible
for providing the solution. MOEs relate to how well any
from a range of solutions may meet a need; MOPs are
concerned with what a particular solution does regard-
less of its intended purpose.
To further illustrate the difference between an MOE
and an MOP, consider a project involving the cleaning
of a river system. It may be decided to base the MOE
on the property of the presence of wildlife, for example,
“...furthest distance upstream fish of type X have been
positively identified,” such as was used for the cleaning
of the River Thames in England. It is not dependent on
any one solution. It is predicated on the basis that there
will be fish in the river if the river is clean. It does not
define “clean” in terms of the amount of rubbish re-
moved, etc., yet it provides a definitive standard for
cleanliness of a waterway.
57
Measures such as amount of suspended solids re-
moved, percentage reduction in heavy metals, reduction
in volume of industrial liquid waste, etc., are MOPs as
they relate to what is achieved in fact. These may be of
use to those involved in the process of cleaning the river,
but, for those instigating the clean-up, the ability of
wildlife to survive in the river is seen as a better indica-
tor and better represents their objective. Telling the
stakeholders that Y tons of rubbish has been removed
will not necessarily achieve the same result. The re-
moval of rubbish alone may not clean the river and, in
any event, how much rubbish would it be necessary to
remove? The actions of the fish (the MOEs) are “the
proof of the pudding” and tell how well the project
succeeded. The amount of rubbish removed, etc. (the
MOPs) only tells what work was done and is not nec-
essarily relevant to the purpose of the project. It may
have been necessary, but it was incidental to the mission.
9. METRICS
The term “metrics” is another term which is sometimes
found in the literature. Various definitions of “metrics”
have been made by professional organizations such as
by the IEEE (IEEE Std 1220 [IEEE, 1994]) which
defines “metric” as: “A measure of the extent or degree
to which a product possesses and exhibits a quality, a
property or an attribute.” However, there does not seem
to be equivalent definitions in main-stream dictionaries.
It appears that accepted usage for the term “metrics” in
the wider world is reserved for standards of measures
as defined by the SI system of weights and measures,
i.e., “pertaining to the metre or to the system of measures
and weights originally based upon it” [Macquarie, 1985].
”Metrics” is commonly associated with software
engineering, where it seems to be used most often as an
MOP. Instances can, however, be found where it is also
used as an MOE. The use of the term “metrics” illus-
trates the confusion which can be caused when there is
no agreed terminology and method of usage, leading in
turn to a proliferation of terms. Lewis [1996] states the
case succinctly when he says that many consultants use
the term “metrics” rather that measures and that:
They probably do so for the same reason that we call
records “rows” or “tuples” when they are in a relational
database. That is to say, for no good reason.
10. CONCLUSION
MOEs are a management tool by which stakeholders
can determine if a solution will meet their needs. They
present a guide to the systems engineer as to the essen-
58 SPROLES

tial properties which any solution must possess. Be-
cause of a lack of agreed definitions, it is possible to
confuse MOEs with other measures unless it is realized
that MOEs are mission or purpose oriented and are a
reflection of the stakeholders standards for success.
REFERENCES
ACTE, Course Notes for “Principles of Test and Evaluation,”
Australian Centre for Test and Evaluation, University of
South Australia, Adelaide, 1996.
E.W. Aslasken, The changing nature of engineering,
McGraw-Hill, Sydney, Australia, 1996.
Cambridge International Dictionary of English, Cambridge
University Press, Cambridge, 1995.
Chambers English Dictionary, Cambridge University Press,
Cambridge, 1988 (reprint).
J. Champy, Reengineering management, HarperCollins, Lon-
don, 1995.
COMNAVSURFLANT, COMNAVSURFLANT NOTICE
9080 MAINTENANCE RESOURCE MANAGEMENT
SYSTEM (MRMS), http://www.spear.navy.mil/
MRMS/mrms_i.htm, 1994.
E. Dale, Management: Theory and practice, McGraw-Hill,
New York, 1969.
W.E. Deming, Quality, productivity and competitive position,
Massachusetts Institute of Technology, Center for Ad-
vanced Engineering Study, Cambridge, MA, 1982.
J.T. Dockery, Why not fuzzy measures of effectiveness? ITEA
J VIII(2) (1987), 24–27.
DSMC, 1996, http://137.244.38.241/library/glossary/app-
b/m.txt
Electronic Industries Association, EIA/IS-632, EIA Interim
Standard, Engineering Department, Washington, DC, De-
cember 1994.
L. Finkelstein and A.C.W. Finkelstein, Review of design
methodology, IEEE Proc 130(Pt. A, No. 4) (1983), 216.
FTA, Revised measures for assessing major investments: A
discussion draft, http://www.fta.dot.gov/fta/library/pol-
icy/draft/INTRO.HTM, 1996.
D.L. Hall, Mathematical techniques in multi-sensor data fu-
sion, Artech House, Dedham, MA, 1992.
Institute of Electrical and Electronic Engineers (IEEE), Std
1220-1994, IEEE Trial-Use Standard for Application and
Management of the Systems Engineering Process, IEEE,
New York, 1994.
R.L. Keeney and H. Raiffa, Decisions with multiple objectives:
Preferences and value tradeoffs, Wiley, New York, 1976.
Kinhill Stearns, Second Sydney Airport Site Selection Pro-
gramme: draft environmental impact statement, prepared
for the Department of Aviation, Kinhill Stearns, Sydney,
April 1985.
M.L. Leibowitz, Metaphysical considerations involved in
choosing a measure of effectiveness, Oper Res J Oper Res
Soc Am 6(1) (1988), 127–130.
B. Lewis, Measure for measure: For better performance re-
sults, call it what it is, IS Survival Guide, http://www.in-
foworld.com/pageone/opinions/lewis/rl021996.htm,
February 19, 1996.
Macquarie, Macquarie Library Pty. Ltd., Sydney, Australia,
1985.
E. Naumann, Creating customer value: The path to sustain-
able competitive advantage, International Thomson Pub-
lishing, 1995.
Oxford English Dictionary, 2nd ed., Vol. V, Clarendon Press,
Oxford, 1989.
A. Pinker, A.H. Samuel, and R. Batcher, On measures of
effectiveness, Phalanx J Military Oper Res Soc, 28 (De-
cember 1995), 8–12.
J. Preece, Y. Rogers, H. Sharp, D. Benyon, S. Holland, and T.
Carey, Human–computer interaction, The Open Univer-
sity, Addison-Wesley, UK, 1994.
B.R. Rich and L. Janos, Skunk works, Warner Books, Lon-
don, 1998 (reprint).
J.G. Roche and B.D. Watts, Choosing analytic measures, J
Strategic Stud 14 (June 1991), 165–209.
T.L. Saaty, The analytic hierarchy process, McGraw-Hill,
Singapore, 1980.
H.A. Simon, The sciences of the artificial, 2nd ed., 3rd reprint,
The MIT Press, Cambridge, MA, 1984.
R. Stevens, P. Brook, K. Jackson, and S. Arnold, Systems
engineering, Prentice Hall Europe, 1998.
R. T. Stevens, Operational test & evaluation, Krieger, Florida,
1986.

Noel Sproles is a former Australian Army officer and graduate surveyor who served for 23 years in
Australia, Papua New Guinea, Indonesia, and South Vietnam in map-making and precision surveys. He
studied the British acquisition process for military equipment at the Royal Military College of Science in
the United Kingdom and spent five years with the Department of Defence in Canberra preparing
requirements documents. He left the Army to follow a new career in warehousing and distribution before
satisfying an old ambition to return to University. He gained a First Class Honours degree in Information
Management and a scholarship to complete a Ph.D. at the Systems Engineering and Evaluation Centre
(SEEC), a research group of the University of South Australia in Adelaide. He successfully completed his
Ph.D. in 1999 with his thesis titled “Measures of Effectiveness: The Standards for Success.” He has had
several articles published on the topic of MOEs both in Australia and the USA and has made presentations
based on his Ph.D. thesis in Australia as well as the UK and USA. He is currently applying his work to a
project at the Defence Science and Technology Oroganisation (DSTO) in Adelaide.