Economic evaluation of new therapies in critical illness

Michael T. Coughlin, MA; Derek C. Angus, MD, MPH

The recent Food and Drug Administration approval of dro-
trecogin alfa (activated) and the potential of several other new
therapies may represent the beginning of a breakthrough in the
management of critical illness in the intensive care unit. How-
ever, their use in clinical practice will likely be dependent on
a rigorous appraisal not only of their effects, but also of their
costs. Novel therapies can no longer be judged simply by their
effectiveness in treating illness, but must also be evaluated on
an institutional and societal level on the basis of their cost.
These considerations have important implications for the prac-
ticing intensivist, who will need to better understand the
conduct and design of economic evaluations, including their
strengths and weaknesses. In this article, we review the ra-
tionale behind economic evaluations of new therapies and the
alternative economic approaches available. We then discuss in
more detail the elements contained in a cost-effectiveness
analysis, the preferred approach to pharmacoeconomic eval-
uation today. (Crit Care Med 2003; 31[Suppl.]:S7–S16)
KEY WORDS: pharmacoeconomics; cost-effectiveness; critical
care; intensive care unit; outcomes research; new therapies;
sepsis

T he care of critically ill patients
in a modern intensive care
unit (ICU) results in a large
societal burden in terms of
both manpower and monetary cost. The
high cost of critical care can largely be
attributed to high overhead costs (e.g.,
need for experienced staff and expensive
equipment), high resource utilization
(pharmaceuticals, laboratory tests, and
imaging procedures), and high demand
for ICU services. With the continued in-
crease in healthcare costs, there is an
increasing need to establish whether new
therapies are not only effective, but also
cost-effective. Although this is true
throughout medicine, the issue of cost-
effectiveness is especially important in
critical care medicine. ICU costs in the
United States exceed $150 billion, repre-
senting up to one third of all hospital
costs (1). Furthermore, attempts to re-
duce ICU costs by other mechanisms,
such as reduction in length of stay, have
proved to be difficult (2).
The concern over the financial effect
of new therapies in the ICU is so intense
that scrutiny begins even before therapies
are approved by the Food and Drug Ad-
ministration. Before ever gaining ap-
From the Clinical Research, Investigation, and Sys-
tems Modeling of Acute Illness (CRISMA) Laboratory,
Department of Critical Care Medicine, University of
Pittsburgh, Pittsburgh, PA.
Copyright © 2003 by Lippincott Williams & Wilkins
DOI: 10.1097/01.CCM.0000045033.69843.FD
proval, the antiendotoxin monoclonal an-
tibody HA-1A stimulated considerable
furor and debate—not only in the medi-
cal literature, but also in the national
media (3–7)— over its anticipated cost.
Currently, the Food and Drug Adminis-
tration does not explicitly consider cost
when evaluating new therapies. However,
the high anticipated costs of new biologic
agents considered in the treatment of
sepsis and severe infections have placed
pressure on the agency. It is perhaps as a
consequence of this pressure that many
recent antisepsis biologic therapies have
been burdened with proving their ability
to decrease mortality to gain Food and
Drug Administration approval (8 –12).
This burden is greater than that faced by
many less expensive therapies (e.g., anti-
biotics).
Even if the Food and Drug Adminis-
tration approves a new therapy, access to
the therapy is increasingly restricted by
pharmacy and therapeutic committees.
These committees are given the respon-
sibility of reviewing hospital formularies
to determine the value, or cost-effective-
ness, of new products. With the forma-
tion of large hospital networks and man-
aged care organizations, such review is
becoming increasingly centralized, and
the physician is gradually losing control
over what drugs are available for the
treatment of patients. Furthermore, the
decision-making process by pharmacy
and therapeutic committees is often not
driven by in-depth economic analyses of
the likely effect of new therapies but,
rather, is made simply on the basis of
anticipated drug acquisition costs. Drug
acquisition costs are important, but these
must also be combined with data on effi-
cacy and hospital costs to evaluate the
total economic effect of new therapies.
However, although access to new ther-
apies is being tied increasingly to cost, and
not simply effect, physicians continue to be
skeptical and suspicious of cost analyses. In
part, skepticism is prompted by the variable
quality of earlier cost analyses. However,
skepticism is also due to the lack of famil-
iarity many physicians have with the gen-
eral principles of health economics. Efforts
by the US Public Health Service have at-
tempted to set standards for the conduct of
cost-effectiveness analyses in medicine (13–
15). More recently, the American Thoracic
Society (ATS) established specific guide-
lines for the conduct of cost-effectiveness
analyses in critical care on the basis of the
US Public Health Service recommenda-
tions (16). These standards are likely to
greatly improve the rigor of future cost-
effectiveness studies. However, if clinicians
do not embrace the principles of cost-
effectiveness assessment, it is likely that
such studies will continue to be viewed
simply as ammunition for nonclinician ad-
ministrators in a war to restrict physician
autonomy and control clinical decision-
making.
In this review, we cover many of the
principal aspects of cost-effectiveness
analyses and consider how such studies

Crit Care Med 2003 Vol. 31, No. 1 (Suppl.) S7

ought to be conducted on future thera-
pies. Throughout this work, we will use a
theoretical newly developed sepsis ther-
apy as a working example. The goal of
this review is to encourage all of us to
familiarize ourselves with cost analyses
and to consider cost-effectiveness analy-
sis (CEA) as simply another tool in our
evaluation toolset, to sit alongside case
reports, animal studies, and randomized
controlled trials (RCTs) of new drug ther-
apies. For an in-depth discussion of eco-
nomic analysis in health care, the reader
is referred to the text by Gold et al (17).
ECONOMIC QUESTIONS IN
HEALTH CARE
One can distill all of health economics
down to two main questions. The first
asks, “Is a given therapy worth using
when compared with alternatives?” For
example, what is the worth of a new an-
tisepsis agent that has recently been dem-
onstrated in a large RCT to have some
beneficial effect? We can consider that
worth represents some trade-off of cost
and benefit (or effect).
The second question is broader and
asks, “Should a portion of available
healthcare resources be allocated to a
given therapy or program?” This is
largely a social policy issue and requires
considering the worth of new programs
or therapies not only within a given dis-
ease, but also in comparison with other
therapies in other diseases. For example,
although a new therapy to fight sepsis
might be deemed worthwhile in the treat-
ment of sepsis, a state Medicaid agency
might be forced to compare its value with
that of a hepatitis B vaccination program
for newborns or influenza vaccinations
for the elderly. These types of choices
become increasingly frequent in situa-
tions in which healthcare resources are
limited and must be equitably distributed
among multiple programs.
In other words, we would wish to
know of a new sepsis agent not only
whether it is cost-effective with respect to
standard management for sepsis, but also
whether we can afford it as a nation.
There are a variety of different analytic
approaches that address all or some part
of these questions. Although the ap-
proaches seem similar and have similar
names, there are key differences. To un-
derstand the strengths and weaknesses of
each approach, we must first describe the
various methodologies used to evaluate
costs in health care. There are essentially
four types of cost analysis: cost minimi-
zation, cost benefit, cost-effectiveness,
and cost utility (Table 1). The fourth,
cost-utility, is best viewed as a special
case of cost-effectiveness. In addition,
there are situations in which a CEA can
produce a cost-minimization statement.
Cost-Minimization Analysis
The cost studies perhaps most familiar
to clinicians are focused solely on how
much a drug costs to put on the phar-
macy shelf. These studies are tradition-
ally called cost-minimization studies.
They are essentially drug acquisition cost

Table 1. Types of cost analyses

Denominator,
Numerator, Outcome
Type of Study Costs or Benefit Comment

Cost minimization Dollars None Antibiotic therapy for ICU patients at low No estimate of consequences on other
risk of nosocomial pneumonia healthcare costs; clinical outcomes are
Singh et al.44 (2001) assumed to be equivalent (i.e., no
Drug acquisition costs for a 3-day course difference in subsequent pneumonia rate
of ciprofloxin are $9,520 cheaper than or mortality) even though a formal
average acquisition costs for equivalence study was not conducted
unregulated antibiotic prescription
Cost benefita Dollars Dollars Use of an aminoglycoside dose- A key advantage is that all costs and effects
monitoring program for burn are expressed in monetary units (dollars),
patients with Gram-negative sepsis facilitating assessment of worth; however,
45
Bootman et al. (1979) the key concern is that converting clinical
effects, such as lives lost (or gained), into
The dose-monitoring program led
dollar amounts is controversial, somewhat
to $8.70 savings per dollar
arbitrary, and biased toward saving the
spent
lives of those with greater earning capacity
Cost effectiveness Dollars Specific measure of Thrombolysis for acute myocardial Assesses change in both costs and effects but
effectiveness (lives infarction avoids controversy of converting clinical
saved) Mark et al.46 (1995) outcomes into dollar values; it is not clear
Tissue plasminogen activator costs an whether “lives saved” are equivalent to
additional $32,678 per additional life other lives saved by other therapies in
saved when compared with other diseases
streptokinase
Cost utilitya Dollars A common utility Prophylaxis against recurrence of peptic Cost per QALY allows comparison to other
metric (quality esophageal strictures therapies used in other diseases; this is
adjusted life years) Stal et al.47 (1998) now the recommended approach.
Omeprazole costs an additional $49,600
per additional QALY when compared
with ranitidine

ICU, intensive care unit; QALY, quality-adjusted life years.

a
Because cost benefit and cost utility analyses produce a common metric, they can be used to compare studies that evaluate different outcomes.
Reproduced with permission from Angus et al (16).

S8 Crit Care Med 2003 Vol. 31, No. 1 (Suppl.)

studies and are conducted frequently by
hospital pharmacy departments. When
different products are compared (e.g.,
two sulfonamides), each product is as-
sumed to have equal efficacy and effect on
all other aspects of treatment (although
this may or may not be true). Effects such
as shortened length of stay, reduced need
for other therapies, and improved quality
of life after illness are not considered in
cost-minimization analyses. The pre-
ferred product is simply the one that
costs the hospital less money per unit of
treatment (e.g., per day of therapy or per
dose).
In addition, a cost-minimization anal-
ysis can result when a formal CEA (see
below) with sophisticated assessment of
all potential changes in costs and effects
between two programs demonstrates no
difference in effect. Nonetheless, in the
situation where there is no difference in
effect, there may be significant differ-
ences in cost between the programs. This
result does not produce a cost-effective-
ness ratio (because one would be dividing
the change in costs by zero), but it does
allow for accurate assessment of the true
differences in cost between two programs
with comparable treatment effects. This
explicit evaluation of costs and effects, as
opposed to an assumption of no differ-
ence in effect, is in contrast to traditional
cost-minimization studies performed in
the past.
Cost-Benefit Analysis
The term cost-benefit is frequently
confused with cost-effectiveness (18). In
fact, a cost-benefit analysis is a very spe-
cific analysis, rarely conducted today,
which expresses all costs and effects in
monetary units. This means that a dollar
value must be placed on all effects. For
example, a life saved must be converted
into a financial benefit. This conversion
of life into monetary terms can be prob-
lematic and unintuitive. After conversion
of all effects into monetary units, one
then adds up all the costs (expressed in
dollars) and subtracts them from all the
benefits (effects, expressed in dollars). If
the final total is negative, the costs out-
weigh the benefits, and vice versa. Al-
though the final output is attractive in its
simplicity, the manipulations required to
convert all effects into dollar values are
inevitably controversial. Because of the
controversy concerning the values of ef-
fects, this type of analysis has largely
Crit Care Med 2003 Vol. 31, No. 1 (Suppl.)
fallen out of favor for pharmacoeconomic
evaluations.
Cost-Effectiveness Analysis
Perhaps the most misused phrase is
the term cost-effectiveness. For example,
many confuse the term with cost saving
and effective. In fact, cost-effectiveness is
simply a ratio of the net change in costs
(dollars) associated with two different
programs or therapies divided by the net
change in effects (health outcome). The
denominator represents the gain in
health (life-years gained, number of addi-
tional survivors, or cases of disease avert-
ed), whereas the numerator reflects the
cost (in dollars) of affecting the gain in
health. Because the units used are differ-
ent for the numerator and denominator,
the typical cost-effectiveness expression
will take the form such as cost (dollars)
per year of life saved. Cost-effectiveness
analyses are the dominant form of cost
analysis today and were endorsed by both
the US Public Health Service Panel on
Cost-effectiveness in Health and Medicine
(PCEHM) and more recently by the ATS
as the principal method by which to as-
sess the costs and effects of healthcare
programs and therapies (14, 16).
Deciding whether a therapy is cost-
effective is a subjective interpretation of
the cost-effectiveness ratio. In other
words, if $100,000 per year of life gained
is deemed the cutoff for effectiveness,
then a new therapy with a cost-effective-
ness ratio of $82,000 per year of life
gained is viewed as cost-effective. Al-
though there is no absolute cutoff, there
is general consensus that a level some-
where between $50,000 and $100,000 per
year of life gained is deemed acceptable in
Figure 1. Simple decision tree comparing outcome for patients treated with a new therapy vs. standard
care. To calibrate the tree, we must estimate 1) the probability for a given patient to live or die, given
whether he or she received the new therapy or not, and 2) the average costs associated with each of
the four branches.
the United States today. Needless to say,
however, the way in which both costs and
effects are calculated can have profound
effects on the resulting ratio, and herein
lies much of the controversy over CEA.
For reasons that will become clear
later in this work, the typical CEA will
require the collection of a significant
amount of information on costs and ef-
fects, much of which may be gathered
from widely differing sources. Interpreta-
tion of these different pieces of informa-
tion is often difficult, and a decision anal-
ysis model is usually constructed that
mimics the key clinical decisions and
events. This model can most easily be
represented by a tree in which each
branch point is calibrated with a proba-
bility of occurrence and a cost. At its
simplest, the tree will contain only
branches for treatment allocation (e.g.,
new therapy or standard) and outcome
(e.g., alive or dead). To calibrate such a
tree, we would need to know only the
probability of living or dying, depending
on whether a given patient received the
new therapy or not, and the average cost
of care for survivors and nonsurvivors in
the two treatment arms (Fig. 1).
Alternatively, we may be interested in
understanding the key events that drive
either morbidity or cost (e.g., mechanical
ventilation or hemodialysis). This could
be important for a variety of reasons;
there may be evidence that the study pop-
ulation has a far lower rate of mechanical
ventilation than is expected for septic pa-
tients in general. Therefore, the extent to
which differences in cost are the result of
the number of patients undergoing me-
chanical ventilation may be important
when estimating the cost-effectiveness of
S9
the new therapy in the real world. Simi-
larly, need for hemodialysis can be a sig-
nificant driver of costs. A new therapy
that reduces the need for mechanical
ventilation or hemodialysis may be ex-
pensive, but the cost of the therapy can
be offset by the reduced need for support-
ive care, and the therapy would hence be
deemed cost-effective by CEA. Con-
versely, cost-benefit analysis would see
only the expense of the new therapy with-
out considering the reduction in support-
ive care. However, the addition of branch
points for mechanical ventilation and he-
modialysis expands the model dramati-
cally, creating 16 branches to consider,
and for each, we must know a patient’s
likelihood of entering such an arm and
the average costs (Fig. 2).
Special Case of CEA
Cost-Utility Analysis. A cost utility
analysis is a form of CEA in which the
effects are converted into common units
of utility. Typically, this approach might
involve adjusting the number of years of
survival for the quality of survival, where
a person living for 1 yr with a quality of
life score of 80% would be awarded 0.8
yrs of quality-adjusted survival. The ad-
vantage of this approach is that it allows
comparison of different programs in dif-
ferent diseases. For example, using the
number of quality-adjusted life years (as
opposed simply to the number of lives
saved), we can perhaps more equitably
compare our sepsis therapy for critically
ill adults with the hepatitis B vaccination
program in newborns.
Methodologic Considerations in
CEA
Good CEA design requires consider-
ation of many elements to both ade-
quately explore the relationship between
costs and effects and to determine the
robustness of the conclusions and the
comparability of the results with those of
other studies. These elements are out-
lined in Table 2 with reference to both
the PCEHM and ATS Guidelines and are
discussed individually in more detail be-
low.
Perspective
The costs considered in a CEA can
vary depending on whose perspective is
taken. For example, consider the issue of
early hospital discharge after childbirth.
S10
Figure 2. Decision tree comparing outcomes for patients treated with a new therapy vs. standard care
that incorporates the potential to undergo mechanical ventilation and receive hemodialysis. To
calibrate the tree, we must estimate the probabilities and average costs for 16 separate trees. MV,
mechanical ventilation; HD, hemodialysis.
The cost from the hospital’s or managed
care organization’s perspective may be
reduced by early discharge. However,
from a societal perspective, the cost sav-
ings of the healthcare facility or organi-
zation may be offset by additional costs to
the patient, such as extra time off work
for the husband who must stay home to
care for the new mother. Previously con-
ducted cost studies have often been ham-
pered by a lack of consistent perspective
either within or among studies. The lat-
ter problem hampers comparison of re-
sults across studies, whereas the former
threatens the validity of the study itself.
The PCEHM and ATS recommend adop-
tion of the societal perspective when cost-
effectiveness studies are conducted.
Outcomes (Effects)
This is an exceedingly difficult prob-
lem for CEA for a variety of reasons. First,
information on outcomes usually comes
from RCTs, which often do not reflect the
actual clinical practice of medicine. Con-
versely, the implications of a CEA are
intended for real-world practice. In other
words, a cost-effectiveness ratio is in-
tended to capture the expected relation-
ship between the costs incurred and the
effects gained in actual practice. An RCT
is usually designed to maximize the like-
lihood of finding an effect. As such, an
RCT can represent a rather rarefied situ-
ation, which is quite distinct from the
real world. For example, only specific pa-
tients may be selected, the dosage and
timing of therapy will likely be optimized,
and other aspects of care may be stan-
dardized and carefully controlled. The ef-
fect size generated under such rigorous
situations is termed a therapy’s efficacy
(or maximal effect). In the real world, the
effect of a new therapy is likely diluted by
less appropriate patient selection,
changes in dosing and timing, and in-
creased heterogeneity in other aspects of
care. The effect of a new therapy under
these real-world conditions is termed a
therapy’s effectiveness. The more RCTs
are refined, the further removed they are
from the reality of using a therapy in
clinical practice (19). Thus, the relation-
ship between cost and effect in some
RCTs becomes increasingly distorted.
A cost analysis conducted by using the
effect size generated from an RCT might
better be termed a cost-efficacy study
rather than a cost-effectiveness study.
However, there are no clear guides on
how to reduce the bias introduced by
using efficacy data instead of effectiveness
data. One possibility is to consider adding
an open-label, open-enrollment arm to
Crit Care Med 2003 Vol. 31, No. 1 (Suppl.)
Table 2. Methodologic considerations in cost-effectiveness analysis

Methodologic Problems PCEHM Second ATS Workshop on Outcomes Research

Recommendations
Aspect Individual CEA Comparing CEAs ICU-Specific (Rationale) Position Comment

Perspective Not defined Different Societal (ethical, Agree May be instances when
pragmatic) provider perspective is
useful
Outcomes, (effects) Data are inadequate or Different outcomes Long-term follow-up QALYs (pragmatic, Agree Require better natural
difficult to evaluate is rare conventional) history of ICU conditions
and modeling or longer
follow-up; other outcomes
may be useful depending
on perspective
Best-designed, Agree Consider modeling reduced
least biased efficacy in sensitivity
source analysis
(pragmatic)
Costs Data are inadequate or Different costs Only hospital costs Costs to include: Agree Standard approach to
difficult to evaluate are usually healthcare measuring these costs not
measured; no services, patient yet developed; estimating
international time, caregiving, units of resource use and
standard non–health multiplying by standard
impacts costs probably most
(theoretical) practical approach
currently; detail with
which resource use is
tracked should be tailored
to nature of intervention
and likely effects on costs
Include or exclude Include costs of
other disease other diseases
costs and test in (too hard to
SA (theoretical, disentangle)
pragmatic, user
needs, and
accounting)
Comparators, (standard Choice distorts results — Determining standard Existing practice Agree
care) often difficult (conventional)
If existing practice Agree Many existing ICU practices
is suspect, may be ineffective or
consider best- cost-ineffective; therefore,
available, viable consider comparison to
low-cost, or “do “best practice” rather
nothing” than standard practice
(conventional)
Discounting Inadequate Different rates Not usually done Discount costs and Agree
representation of effects to
the effect of time present value
(theoretical)
Use a 3% discount Agree
rate (theoretical,
pragmatic)
Uncertainty Inadequate — Not usually done SA essential; Agree Multiway sensitivity analyses
representation of multiway SA probably essential given
uncertainty on preferred (user high likelihood that
results needs) several key assumptions
will be necessary to
generate reference case
from critical care trials
Reporting — Not standard Reference case Agree But, also present “data-rich”
(user needs) case
Compare to Agree
available ratios
(user needs)
Journal and Agree Also file (e.g., on Internet)
technical report intended analysis plan
(user needs) prior to unblinding when
concurrent with
randomized clinical trial

CEA, cost-effectiveness analysis; ICU, intensive care unit; PCEHM, Panel on Cost-effectiveness in Healthcare and Medicine; ATS, American Thoracic
Society; QALY, quality-adjusted life years; SA, sensitivity analysis.
Reproduced with permission from Angus et al (16).

Crit Care Med 2003 Vol. 31, No. 1 (Suppl.) S11

RCTs on which a CEA is being conducted
(20). However, this presents many logis-
tic and ethical problems. The more ac-
cepted alternative is to expose the cost
model to varying estimates of reduced
effect from those seen in the RCT, during
sensitivity analysis (see below).
Another problem encountered when
determining effect, or outcome, is that
the outcome measure evaluated in the
RCT may not be directly relevant in the
cost analysis. The PCEHM recommends,
and the ATS agrees, that quality-adjusted
life years be used as the units of effect, or
utility. However, most RCTs in critical
care use short-term (day 28 or hospital)
mortality as the primary end point, and
still others use indices such as organ fail-
ure–free days as outcome measures (21).
Although short-term survival likely cor-
relates with long-term quality-adjusted
survival, the relationship is not explicitly
clear. Whether there is any relationship
between organ failure–free days and long-
term quality of life is even less clear. In
fact, a recent study by Clermont et al.
(22) showed that patients who develop
acute organ dysfunction are at risk for
poor long-term quality of life but that the
risk is largely due to poor baseline health
status and not directly to organ failure
during critical illness.
This problem is only slowly changing.
Although the PCEHM recommends long-
term outcome, a National Institutes of
Health–sponsored workshop on sepsis
studies recommended day 28 mortality
(21). More recently, the UK MRC work-
shop still maintained that day 28 mortal-
ity was an appropriate primary end point
but recommended follow-up to ⱖ90 days
and, whenever possible, to ⱖ6 months
(23). Furthermore, recent successful tri-
als in sepsis reported mortality at widely
varying time points—28 days (drotreco-
gin alfa [activated]) (24), 28 days and 1 yr
(steroids) (25), and 60 days (early goal-
directed therapy) (26)—and a recent
study of acute respiratory distress syn-
drome (low tidal volume) (27) reported
mortality to 180 days. Proponents of
short-term outcome state that longer fol-
low-up is too expensive and not necessar-
ily related to the therapy being studied.
Advocates of longer follow-up state that
short-term survival, of indeterminate
quality of life, and possibly with death a
short time thereafter, is of little societal
relevance (17). They further argue that
the ability to prioritize healthcare spend-
ing on the basis of value requires that we
compare the long-term value of alterna-
S12
tive programs for alternative disease pro-
cesses. Many healthcare programs are ad-
ministered, and/or have effects lasting
over a long period of time, making long-
term follow-up of patients enrolled in
these types of programs essential.
There is currently relatively little
long-term follow-up information on ICU
patients. However, the available evidence
does suggest that there is considerable
mortality and morbidity beyond hospital
discharge, supporting the notion that we
should consider longer follow-up (28 –
30). Quartin et al. (28) showed that con-
tinuing mortality occurs in sepsis pa-
tients for many months after discharge
from the hospital. Studies exploring qual-
ity of life after ICU care have yielded con-
flicting results, but certainly several sug-
gest a considerable diminution of quality
of life, which seems to be sustained over
time.
Thus, until more evidence is available,
studies of new therapies in the treatment
of sepsis on which CEAs are intended
should have some mechanism (e.g., a
subset study or parallel cohort) to incor-
porate mortality follow-up for 6 –12
months with an accompanying quality of
life assessment.
Costs
Which Costs Should We Include?
Which costs should be included? Debates
over this subject can be so contentious as
to resemble debates over whether to give
colloids or crystalloids to hypotensive pa-
tients. The subject is further complicated
by economic terms such as direct vs. in-
direct costs and tangible vs. intangible
costs. We will attempt to avoid using too
many accounting terms and to suggest
alternative ways to understand this issue.
Let us go back to the cost-effective-
ness ratio. It is a ratio of net costs divided
by net effects. Thus, regardless of
whether the costs of any given element
seem important, if they are distributed
equally in both comparison groups, the
net difference will be zero, and we there-
fore need not worry about them. Alterna-
tively stated, we need consider only those
costs that we believe to be relevant and
likely to differ between the treatment and
control groups. As an example, the
PCEHM believed that the intangible costs
of pain and suffering were relevant costs
that should be measured in CEA, but we
have never measured such costs in any
critical-care CEA. Therefore, if a new
therapy is unlikely to cause either more
or less pain, then we can continue to
ignore such intangible costs, even al-
though they are considered relevant. The
caveat here is that we have now made an
important assumption—no difference in
pain—which may or may not be true.
We have of course glossed over the
term relevant. Which costs are relevant?
All costs to society could be considered
relevant when conducting a CEA from
the societal perspective. Using this per-
spective, one could argue that the costs of
lost wages while a patient is sick are rel-
evant. In response to this issue, the
PCEHM recognized that there are no cor-
rect answers. However, to promote stan-
dardization of CEA methodologies, they
recommend inclusion of all health-
related costs, and the ATS concurs that
this is the current best approach. This
includes intangible costs of pain and suf-
fering and travel costs. They also recom-
mended including opportunity costs and
suggested that lost wages, not only as a
postdischarge consequence of the illness,
but also during hospitalization, represent
an example of an opportunity cost. Direct
application of these guidelines to critical
care is not easy. However, one way to
consider them is to think about a health-
care system without the new therapy vs. a
health care system with the new therapy.
We then need to include all possible cost
elements that could differ between these
two healthcare worlds.
How Should Costs Be Included? Not
all costs included in a CEA are necessarily
measured empirically. The CEA is a
model often calibrated by estimates.
Some of these estimates come from mea-
surements. For example, the estimate of
differences in the mortality rate between
a drug and placebo often is derived from
the effect size in an RCT. Other estimates
can be based on expert opinion or on
some combination of measurement and
opinion. For example, the cost of the ac-
tual therapy is usually unknown because
the therapy is often not yet approved, and
no pricing decision has yet been made by
the company that manufactures the ther-
apy. One is therefore forced to estimate
on the basis of an educated best guess,
perhaps with knowledge of the prelimi-
nary pricing from the company. Although
one might be alarmed at this notion of
educated guesswork, it is important to
appreciate that estimates can be wildly
erroneous yet have only a minimal effect
on the cost-effectiveness ratio. To test
how sensitive a CEA ratio is to various
estimates in the cost model, the com-
Crit Care Med 2003 Vol. 31, No. 1 (Suppl.)
pleted CEA model is exposed to a rigorous
sensitivity analysis (see below).
In this way, we can decide to include
many costs in a CEA, yet measure specif-
ically only some portion of that total. As
long as these estimated costs have little
effect on the overall final CEA conclu-
sions, the strategy regarding which costs
to measure and which to estimate can be
considered robust.
For How Long Do We Measure Costs?
When the cost of a therapy is computed,
the duration of the costs attributed to
therapy must also be considered. For ex-
ample, if our new therapy allows more
people to leave the ICU but causes a
higher prevalence of renal failure requir-
ing long-term dialysis, should all the
costs of dialysis be attributed to that ther-
apy? The answer is yes.
Although most intensivists do not ac-
cept the concept of blaming the therapy
received in the ICU for incurring long-
term cost, it is difficult to argue to the
contrary. In producing a survivor, one
must also take responsibility for the cost
of maintaining survival, which means fol-
lowing these costs for a significant length
of time. Furthermore, if chronic renal
failure leads to a lower quality of life, the
new therapy will be doubly penalized,
both for the cost of the dialysis program
and for the decrement in effect (reduced
quality-adjusted survival).
How to Measure Costs? For those costs
that we choose to measure, we must decide
what represents true cost. When we con-
sider hospital costs, as an example, true
costs are generally assumed to be those
generated by formal cost-accounting mech-
anisms. For a complete blood count, the
costs include the wage rate for and time
spent by the employee who drew the blood,
the cost of the tube, some tiny amortized
fraction of the cost of the equipment upon
which the test is run, and so on. Needless to
say, detailed information such as this is
rarely available as part of a CEA. Another
approach is to collect hospital charges and
adjust them by the hospital- or depart-
ment-specific cost/charge ratios. The rela-
tionship between charges and costs has
long been a source of skepticism for physi-
cians. However, recent work by Shwartz et
al. (32), comparing department-specific
cost/charge ratio–adjusted charges with es-
timates generated from a formal cost-
accounting system, found good correlation
when assessing patients in groups. Agree-
ment was much poorer when comparing
individual patients and when using hospi-
tal-specific ratios. CEAs rely on average
Crit Care Med 2003 Vol. 31, No. 1 (Suppl.)
grouped estimates of costs, and therefore
department-specific cost/charge ratios
seem adequate for estimating hospital
costs.
Other proxy measures of cost, such as
the Therapeutic Intervention Scoring
System or length of stay, can also be used
(33, 34). As stated previously, their value
will depend on how sensitive the conclu-
sions are to variations in the relationship
between these proxy measures and true
costs.
Defining Standard Care
(Comparators)
When comparing a new therapy, the
choice of comparator, or standard ther-
apy, is also critical. For example, the cost-
effectiveness ratio of a 1-yr cervical can-
cer screening program is quite different
when compared with 2- or 3-yr programs.
Similarly, a tissue thromboplastin activa-
tor has a different cost-effectiveness ratio
when compared with standard acute
myocardial infarction therapy with no
thrombolytic therapy as opposed to stan-
dard therapy with streptokinase. The
PCEHM recommends that the control
therapy used for comparative purposes be
the least expensive available standard
therapy. However, this view is currently
changing in the field of critical care. For
example, in the treatment of sepsis,
should standard care include early goal-
directed therapy, steroids, or drotrecogin
alfa (activated), even although these may
be expensive? If so, do we consider all
treatments, or just one, to be standard
therapy? The ATS guidelines recommend
that standard practice is not always best
practice and that best practice should be
the comparator of choice in critical care.
Discounting (Time)
Discounting costs due to time is an-
other important factor to consider when
conducting a CEA. When we borrow
money, we must pay it back with interest.
This is because money is worth more now
than it will be later. Therefore, $10 is
more valuable now than $10 delivered at
a rate of $1 per year for the next 10 yrs.
Thus, to pay back $10 that we just re-
ceived over the next 10 yrs, we would be
required to pay back ⬎$1 per year. Be-
cause world economic growth is occur-
ring at approximately 3%, the PCEHM
has recommended that all costs be dis-
counted at a 3% rate per annum, and the
ATS agrees with this recommendation.
But what about effects: should they
also be discounted? Are ten people living
for 1 yr more valuable than one person
living for 10 yrs? Although this issue may
seem inhumane, consideration of this
point is crucial. Discounting costs with-
out discounting effects will incur the
Keeler-Cretin procrastination paradox,
wherein we would forever favor health-
care programs that take place some time
in the future (35). This situation would
have us forever putting off until tomor-
row that which could be done today.
Therefore, we also discount effects at 3%,
the same rate as costs.
Robustness and Uncertainty
When we perform a RCT, our primary
conclusion is a statement of effect: did
the new therapy change the outcome of
interest? Although it is highly likely that
the outcome rates will be different (rarely
would the mortality rates in both arms be
identical), we rely on statistical signifi-
cance to tell us whether the observed
difference is due to a true effect of the
therapy and not to chance alone. We tra-
ditionally infer statistical significance
when the p value is ⬍.05. In other words,
we are 95% certain that the observed
difference did not arise by chance alone.
If we are interested only in the single
dimension effect, then we care only about
which therapy arm is better, not how
much better.
It is important to appreciate, however,
that the p value does not confirm the
magnitude of effect. Consider an example
in which a new antisepsis strategy ther-
apy has a mortality rate of 35%, as op-
posed to a placebo rate of 40%, with a p ⬍
.05. This does not mean that 5 lives are
saved per 100 persons treated. Rather, it
tells us that our best estimate is that five
lives are saved. If we presume a binomial
distribution around the mortality rates,
we can generate confidence intervals
around the two estimates. These confi-
dence intervals might now tell us that
new therapy saves from 1 to 9 lives per
100 persons treated, but they cannot tell
us where the true value falls within that
range. The p value simply confirms the
likelihood that lives are saved by the new
therapy, not how many lives.
In CEA, however, we must quantitate
the magnitude of effect (and cost) so that
we can generate a ratio. The general prin-
ciple is to first take our best point esti-
mates of cost and effect to generate a base
case. Thereafter, we vary all our measures
S13
and estimates across their range of prob-
abilities (e.g., 95% confidence interval) to
determine the extent to which the cost-
effectiveness ratio varies. This is a sensi-
tivity analysis (SA) and can be performed
either with one or multiple variables si-
multaneously. In one respect, the sensi-
tivity analysis can be considered analo-
gous to the p value in that it allows us to
explore the robustness of our conclu-
sions. In other words, if, despite varying
several or all variables across their sto-
chastic distributions, there is minimal
change in the final ratio, then one can
have considerable confidence in the cost-
effectiveness ratio estimate.
Another aspect of the sensitivity anal-
ysis is that it can be used to determine
which model estimates must be the most
accurate. For example, the cost-effective-
ness ratio may be exquisitely sensitive to
the estimate of ICU costs but relatively
insensitive to the expected costs of post-
discharge healthcare resource use. In this
situation, one might need to measure
ICU costs very carefully yet rely only on
approximate estimates of postdischarge
resource use. A comprehensive sensitivity
analysis can, in fact, be considered more
powerful than a p value in that it can be
used to graphically show all of the uncer-
tainties inherent in the underlying as-
sumptions of the CEA model.
Reporting and the PCEHM
Reference Case
The PCEHM also recommended that
all future CEAs produce a reference case
in which the cost-effectiveness ratio is
generated by a standardized approach to
estimating and measuring each of the
important elements of the CEA (Table 2).
This includes the perspective chosen, de-
termination of costs and effects, study
time horizon, and measurement of un-
certainty and sensitivity analyses. Use of
this standardized approach allows for
comparison of CEA results across studies.
Comparison of reference cases between
CEAs allows us to make inferences about
the cost-effectiveness of a new sepsis
therapy vs. a therapy used to fight breast
cancer. It allows us to compare apples
with oranges and not just apples with
apples. Figure 3 shows comparisons of
costs for a variety of treatments in differ-
ent diseases. These include both inter-
ventions against specific disease states
(e.g., myocardial infarction and stroke)
(36, 37) and interventions designed to
prevent injury or illness (e.g., airbags)
S14
Figure 3. League table showing the range of cost effectiveness ratios for a variety of medical or
preventive interventions (36 – 43). The vertical lines show the 20,000/quality-adjusted life years (QALY)
and 100,000/QALY levels. Shading of the horizontal lines shows level of cost effectiveness: good (light
gray), marginal (dark gray), and poor (black). The range of values within an intervention indicates
differences in conditions or assumptions included in the model. tPA, tissue plasminogen activator;
AMI, acute myocardial infarction; CABG, coronary artery bypass graft; 2V, two vessel.
(38). The ATS guidelines also recommend sult in a prioritized list of community
the generation of a reference case and benefits for given cost outlays. However,
further recommend the presentation of a although CEA can inform us about where
data-rich case from the results of the to spend money to improve utility, it can-
RCT. This case would be generated by not inform about how much money
using a minimal number of model as- should be spent improving health care
sumptions and the maximum amount of overall.
available data from the trial. The overall goal of CEAs is to supply
decision makers with information that
CEA AND HEALTHCARE POLICY can be used to choose between medical
care options when all options are not
Decision making based on the results financially feasible. If monies are unlim-
of a CEA is founded on the idea of social ited, the relevant question becomes
utilitarianism. This value is in turn based “what treatment options minimize pa-
on the assumptions that a) good is deter- tient morbidity and mortality?” and CEAs
mined by consequences at the commu- are unnecessary. If funds are limited, the
nity level, these consequences being the question becomes “what is the best val-
sum of individual utilities (health and ue?” and CEAs can help us to answer this
happiness); b) all utilities are equal question. This was the overall conclusion
within the metric used to measure them; of the ATS. The workshop further recom-
and c) loss of benefit to some is balanced mended that critical care researchers use
by benefit to others. As a simple example, the PCEHM guidelines and conduct ref-
consider the decision to fund a childhood erence case CEAs as part of the evaluation
immunization program rather than a of ICU interventions; these CEAs will al-
radical chemotherapy program to treat a low more informative health care policy
rare cancer. This decision assumes that in the care of the critically ill.
spending resources on immunizations
will maximize the community’s utility CONCLUSIONS
more than money spent on treating a rare
cancer. Social utilitarianism acts to max- The conduct of a rigorous CEA is
imize the health and happiness (utility) of clearly challenging. There are many
the community; consequently, utility methodologic complexities to consider
leads to the maximum efficiency in the within the study, and the analysis is likely
use of healthcare resources for the com- to be highly dependent on the availability
munity’s benefit. CEA is designed to re- and quality of information from other
Crit Care Med 2003 Vol. 31, No. 1 (Suppl.)

C
onsider cost-effec-
tiveness analysis
as simply another
tool in our evaluation toolset
of new drug therapies.
studies (e.g., a phase III RCT). Much of
the information required to conduct a
thorough CEA may in fact be missing,
and therefore the analysis will require
sophisticated modeling techniques to ex-
plore the effects of various assumptions
and uncertainties. For the reader to be
confident in the results of a CEA, the
analysis not only must be robust, but
must also be perceived as such. Accom-
plishing this requires clear reporting on
the part of the analyst and commitment
on the part of the reader to embrace
rigorously conducted cost-effectiveness
analyses as a legitimate approach to de-
termining the value of new therapies in
the treatment of critical illness.
REFERENCES
1. Halpern NA, Bettes L, Greenstein R: Federal
and nationwide intensive care units and
healthcare costs: 1986 –1992. Crit Care Med
1994; 22:2001–2007
2. Angus DC, Linde-Zwirble WT, Sirio CA, et al:
The effect of managed care on ICU length of
stay: Implications for Medicare. JAMA 1996;
276:1075–1082
3. Ziegler EJ, Fisher CJ Jr, Sprung CL, et al:
Treatment of gram-negative bacteremia and
septic shock with HA-1A human monoclonal
antibody against endotoxin: A randomized,
double-blind, placebo-controlled trial.
N Engl J Med 1991; 324:429 – 436
4. Warren HS, Danner RL, Munford RS: Anti-
endotoxin monoclonal antibodies. N Engl
J Med 1992; 326:1153–1157
5. Wenzel RP: Anti-endotoxin monoclonal anti-
bodies: A second look. N Engl J Med 1992;
326:1151–1153
6. Winslow R: Centocor drug is said to need
more testing. Wall Street Journal April 21,
1992; A3
7. Winslow R: Dose of reality: Costly biotech
drugs will pose hard choices for US health-
care. Wall Street Journal November 5, 1991;
A1
8. Abraham E, Wunderink RG, Silverman H, et
al: Efficacy and safety of monoclonal anti-
body to human tumor necrosis factor alpha
in patients with sepsis syndrome: A random-
Crit Care Med 2003 Vol. 31, No. 1 (Suppl.)
ized, controlled, double-blind, multicenter
clinical trial. JAMA 1995; 273:934 –941
9. Bone RC, Balk RA, Fein AM, et al: A second
large controlled clinical study of E5, a mono-
clonal antibody to endotoxin: Results of a
prospective, multicenter, randomized, con-
trolled trial. Crit Care Med 1995; 23:
994 –1006
10. Dhainaut JF, Tenaillon A, Le Tulzo Y, et al:
Platelet-activating factor receptor antagonist
BN 52021 in the treatment of severe sepsis: A
randomized, double-blind, placebo-con-
trolled, multicenter clinical trial. Crit Care
Med 1994; 22:1720 –1728
11. Fein AM, Bernard GR, Criner GJ, et al: Treat-
ment of severe systemic inflammatory re-
sponse syndrome and sepsis with a novel
bradykinin antagonist, deltibant (CP-0127):
Results of a randomized, double-blind, pla-
cebo-controlled trial. CP-0127 SIRS and Sep-
sis Study Group. JAMA 1997; 277:482– 487
12. Fisher CJ Jr, Dhainaut JF, Opal SM, et al:
Recombinant human interleukin 1 receptor
antagonist in the treatment of patients with
sepsis syndrome: Results from a randomized,
double-blind, placebo-controlled trial. JAMA
1994; 271:1836 –1843
13. Russell LB, Gold MR, Siegel JE, et al: The
role of cost-effectiveness analysis in health
and medicine: Panel on Cost-Effectiveness in
Health and Medicine. JAMA 1996; 276:
1172–1177
14. Weinstein MC, Siegel JE, Gold MR, et al:
Recommendations of the Panel on Cost-
Effectiveness in Health and Medicine. JAMA
1996; 276:1253–1258
15. Siegel JE, Weinstein MC, Russell LB, et al:
Recommendations for reporting cost-effec-
tiveness analyses. JAMA 1996; 276:
1339 –1341
16. Angus DC, Rubenfeld GD, Roberts MS, et al:
Understanding costs and cost-effectiveness in
critical care: Report from the Second Amer-
ican Thoracic Society Workshop on Out-
comes Research. Am J Respir Crit Care Med
2002; 165:540 –550
17. Gold MR, Russell LB, Seigel JE, et al: Cost-
effectiveness in Health and Medicine. New
York, Oxford University Press, 1996
18. Doubilet P, Weinstein MC, McNeil BJ: Use
and misuse of the term “cost effective” in
medicine. N Engl J Med 1986; 314:253–256
19. Linden PK, Angus DC, Chelluri L, et al: The
influence of clinical study design on cost-
effectiveness projections for the treatment of
gram-negative sepsis with human anti-
endotoxin antibody. J Crit Care 1995; 10:
154 –164
20. Freemantle N, Drummond M: Should clini-
cal trials with concurrent economic analyses
be blinded? JAMA 1997; 277:63– 64
21. Dellinger RP: From the bench to the bedside:
The future of sepsis research. Executive sum-
mary of an American College of Chest Phy-
sicians, National Institute of Allergy and In-
fectious Disease, and National Heart, Lung,
and Blood Institute Workshop. Chest 1997;
111:744 –753
22. Clermont G, Angus DC, Linde-Zwirble WT, et
al: Does acute organ dysfunction predict pa-
tient-centered outcomes? Chest 2002; 121:
1963–1971
23. Cohen J, Guyatt G, Bernard GR, et al: New
strategies for clinical trials in patients with
sepsis and septic shock. Crit Care Med 2001;
29:880 – 886
24. Bernard GR, Vincent JL, Laterre PF, et al:
Efficacy and safety of recombinant human
activated protein C for severe sepsis. N Engl
J Med 2001; 344:699 –709
25. Annane D, Sebille V, Charpentier C, et al:
Effect of treatment with low doses of hydro-
cortisone and fludrocortisone on mortality in
patients with septic shock. JAMA 2002; 288:
862– 871
26. Rivers E, Nguyen B, Havstad S, et al: Early
goal-directed therapy in the treatment of se-
vere sepsis and septic shock. N Engl J Med
2001; 345:1368 –1377
27. Ventilation with lower tidal volumes as com-
pared with traditional tidal volumes for acute
lung injury and the acute respiratory distress
syndrome: The Acute Respiratory Distress
Syndrome Network. N Engl J Med 2000; 342:
1301–1308
28. Quartin AA, Schein RM, Kett DH, et al: Mag-
nitude and duration of the effect of sepsis on
survival. JAMA 1997; 277:1058 –1063
29. Angus DC, Carlet J: Surviving intensive care:
A report from the 2002 Brussels Roundtable.
Intensive Care Med, In Press
30. Angus DC, Carlet J (Eds): Surviving Intensive
Care: Update in Intensive Care and Emer-
gency Medicine No. 39. Berlin, Springer-
Verlag, 2002
31. Chelluri L, Grenvik AN, Silverman M: Inten-
sive care for critically ill elderly: Mortality,
costs, and quality of life. Review of the liter-
ature. Arch Intern Med 1995; 155:1013–1022
32. Shwartz M, Young DW, Siegrist R: The ratio
of costs to charges: How good a basis for
estimating costs? Inquiry 1995; 32:476 – 481
33. Keene AR, Cullen DJ: Therapeutic Interven-
tion Scoring System: Update 1983. Crit Care
Med 1983; 11:1–3
34. Rapoport J, Teres D, Lemeshow S, et al: A
method for assessing the clinical perfor-
mance and cost-effectiveness of intensive
care units: A multicenter inception cohort
study. Crit Care Med 1994; 22:1385–1391
35. Keeler EB, Cretin S: Discounting of life-
saving and other nonmonetary effects.
Manage Science 1983; 29:300 –306
36. Kalish SC, Gurwitz J, Krumholz HM, et al: A
cost-effectiveness model of thrombolytic
therapy for acute myocardial infarction.
J Gen Intern Med 1995; 10:321–330
37. Gage BF, Cardinalli AB, Albers GW, et al:
Cost-effectiveness of warfarin and aspirin for
prophylaxis of stroke in patients with non-
valvular atrial fibrillation. JAMA 1995; 274:
1839 –1845
38. Graham JD, Thompson KM, Goldie SJ, et al:
The cost-effectiveness of air bags by seating
position. JAMA 1997; 278:1418 –1425
39. Stinnett AA, Mittleman MA, Weinstein MC:
S15
 The cost-effectiveness of dietary and pharma-
cologic therapy for cholesterol reduction in
adults. In: Cost-effectiveness in Health and
Medicine. Gold MR, Siegel JE, Russell LB, et
al (Eds). New York, Oxford University Press,
1996, pp 349 –391
40. Hunink MG, Wong JB, Donaldson MC, et al:
Revascularization for femoropopliteal dis-
ease: A decision and cost-effectiveness anal-
ysis. JAMA 1995; 274:165–171
41. Pliskin JS, Stason WB, Weinstein MC, et
al: Coronary artery bypass graft surgery:
Clinical decision making and cost-effec-
tiveness analysis. Med Decis Making 1981;
1:10 –28
S16
42. Weinstein MC, Stason WB: Cost-effectiveness
of coronary artery bypass surgery. Circula-
tion 1982; 66:III56 –III66
43. Owens DK, Sanders GD, Harris RA, et al:
Cost-effectiveness of implantable cardio-
verter defibrillators relative to amiodarone
for prevention of sudden cardiac death. Ann
Intern Med 1997; 126:1–12
44. Singh N, Rogers P, Atwood CW, et al: Short-
course empiric antibiotic therapy for patients
with pulmonary infiltrates in the intensive
care unit: A proposed solution for indiscrim-
inate antibiotic prescription. Am J Respir
Crit Care Med 2000; 162:505–511
45. Bootman JL, Wertheimer AI, Zaske D, et al:
Individualizing gentamicin dosage regimens
in burn patients with gram-negative septice-
mia: A cost-benefit analysis. J Pharm Sci
1979; 68:267–272
46. Mark DB, Hlatky MA, Califf RM, et al: Cost
effectiveness of thrombolytic therapy with
tissue plasminogen activator as compared
with streptokinase for acute myocardial
infarction. N Engl J Med 1995; 332:
1418 –1424
47. Stal JM, Gregor JC, Preiksaitis HG, et al: A
cost-utility analysis comparing omeprazole
with ranitidine in the maintenance therapy of
peptic esophageal stricture. Can J Gastroen-
terol 1998; 12:43– 49
Crit Care Med 2003 Vol. 31, No. 1 (Suppl.)