Volume 13 Number 5 2010

VA L U E I N H E A LT H

A Review of Quantitative RiskBenet Methodologies for

Assessing Drug Safety and EfcacyReport of the ISPOR
RiskBenet Management Working Group
vhe_725

657..666

Jeff J. Guo, PhD,1 Swapnil Pandey, MS,2 John Doyle, PhD,3,4 Boyang Bian, MS,1 Yvonne Lis, PhD,4
Dennis W. Raisch, PhD5
1

University of Cincinnati Health Academic Center, College of Pharmacy, Cincinnati, OH, USA; 2Kendle International Inc., Cincinnati, OH, USA;
Department of Epidemiology, Mailman School of Public Health, Columbia University, New York, NY, USA; 4Centre for Socioeconomic
Research, Welsh School of Pharmacy, Cardiff, UK; 5University of New Mexico, College of Pharmacy, Albuquerque, NM, USA
3

A B S T R AC T
Objective: Although regulatory authorities evaluate the risks and benets
of any new drug therapy during the new drug-approval process, quantitative riskbenet assessment (RBA) is not typically performed, nor is it
presented in a consistent and integrated framework when it is used. Our
purpose is to identify and describe published quantitative RBA methods
for pharmaceuticals.
Methods: Using MEDLINE and other Internet-based search engines, a
systematic literature review was performed to identify quantitative methodologies for RBA. These distinct RBA approaches were summarized to
highlight the implications of their differences for the pharmaceutical
industry and regulatory agencies.
Results: Theoretical models, parameters, and key features were reviewed
and compared for the 12 quantitative RBA methods identied in the
literature, including the Quantitative Framework for Risk and Benet
Assessment, benet-less-risk analysis, the quality-adjusted time without
symptoms and toxicity, number needed to treat (NNT), and number

needed to harm and their relative-value-adjusted versions, minimum clinical efcacy, incremental net health benet, the riskbenet plane (RBP),
the probabilistic simulation method, multicriteria decision analysis
(MCDA), the riskbenet contour (RBC), and the stated preference
method (SPM). Whereas some approaches (e.g., NNT) rely on subjective
weighting schemes or nonstatistical assessments, other methods (e.g., RBP,
MCDA, RBC, and SPM) assess joint distributions of benet and risk.
Conclusions: Several quantitative RBA methods are available that could
be used to help lessen concern over subjective drug assessments and to help
guide authorities toward more objective and transparent decision-making.
When evaluating a new drug therapy, we recommend the use of multiple
RBA approaches across different therapeutic indications and treatment
populations in order to bound the riskbenet prole.
Keywords: drug safety, incremental riskbenet ratio, multicriteria decision analysis, number needed to treat, riskbenet assessment, riskbenet
plane, stated preference method.

Introduction

for proactive intervention in health-care settings, which could

save lives, reduce litigation, and lead to improved patient safety,
better health outcomes, and lower overall health-care costs
[812].
In Europe, part of the mandate of the Committee for Medicinal Products for Human Use (CHMP) is to assess risks and
benets of authorized medicines on behalf of the European Medicines Agency (EMEA). In 2007, the CHMP revised its guidance
and included quantitative RBA in the regulatory agenda with the
publication of a report examining the potential value of existing
benetrisk models and methods [13,14]. Although no specic
method was recommended, several RBA features were noted as
being of value, including 1) all important benets and medically
serious risks are identied; and 2) the risks and benets are
weighted according to their relative importance and the strength
of the evidence available. It was also decided that a comprehensive review of available quantitative methods for RBA relevant to
the CHMP was required to explore further development of tailored methodologies. The EMEA recently created the European
Network of Centres for Pharmacoepidemiology and Pharmacovigilance, which is in the process of developing an algorithm to
articulate safety and benet proles for pharmaceutical products.
Regulatory agencies use various methods to discover rare
toxic events. These include review of data from randomized
controlled clinical trials, observational epidemiological studies
(case-control, cohort, and cross-sectional analyses), drug-use
surveys, automated databases linking drugs and disease, spontaneous reporting (passive surveillance, such as the FDAs

In the past decade, over a dozen high-prole brand-name drugs,

including rofecoxib, troglitazone, cisapride, and cerivastatin,
were withdrawn from the market [1]. The US Food and Drug
Administration (FDA) established a Drug Safety and Risk Management Division, which is charged with evaluating the safety,
efcacy, and abuse potential of drugs, as well as risk management
and risk communication. In March 2005, the FDA issued risk
management guidance for the pharmaceutical industry, which
included three separate guidelines: Premarketing Risk Assessment, Development and Use of Risk Minimization Action Plans,
and Good Pharmacovigilance Practices and Pharmacoepidemiologic Assessment [25]. In 2007, Title IX of the FDA Amendments Act gave the FDA the authority to require companies to
develop and implement a risk evaluation and mitigation strategy
for specied prescription drugs. In this era of renewed focus on
drug safety, the FDA has called for more creative approaches to
conceptualizing, measuring, and applying riskbenet assessment (RBA) techniques to develop and improve a systematic
RBA approach throughout the life cycle of a pharmaceutical
product [69]. Appropriate RBA can provide useful information
Address correspondence to: Jeff J. Guo, Pharmacoeconomics & Pharmacoepidemiology, University of Cincinnati Medical Center College of Pharmacy, 3225 Eden Ave., Cincinnati, OH 45267-0004, USA. E-mail:
jeff.guo@uc.edu
10.1111/j.1524-4733.2010.00725.x

2010, International Society for Pharmacoeconomics and Outcomes Research (ISPOR)

1098-3015/10/657

657666

657

Guo et al.

658
MedWatch program), and established patient registries [15,16].
When a regulatory authority such as the FDA or the EMEA
obtains information about potentially signicant drug toxicity, it
thoroughly reviews the original new drug application data that
were used for the initial approval. Concurrently, a formal analysis is initiated by reviewing all the spontaneous reports available
from the FDA Adverse Events Reporting System, the United
Kingdoms Yellow Card Scheme, and the World Health Organizations Uppsala Monitoring Centre. Additionally, the sponsor of
a pharmaceutical product is required to review its safety database
and report directly to the regulators. In some circumstances,
further postmarketing studies are designed to answer specic
questions about toxicity. In the process of reviewing risks and
benets, a regulatory authority typically seeks input from advisory committees that review safety and efcacy data and make
recommendations [8,9,17]. There is a structured process for convening the appropriate advisory committee. The authority prepares a set of questions involving product safety, efcacy,
study design, results interpretation, and riskbenet proles
for committee discussion. After review of the advisory committees recommendations, the authority determines a course of
action, which may include changes to labeling, direct correspondence to health professionals, or removal of the drug from the
market.
This traditional process does not produce an explicit, consistent, transparent, and aggregate quantication of the risks and
benets and lacks clarity pertaining to the role of specic factors
in the recommendations. Although some quantitative measures,
such as quality-adjusted life-years (QALYs) and number needed
to treat (NNT) [11,1721], discussed later in this article, are used
by regulators, there is lack of standardized and validated quantitative methodology. Challenges to developing such a methodology include heterogeneity and multiplicity of risks and benets,
uncertainty regarding attribution of risks and benets to a particular treatment, and the temporality and paucity of drugexposure and outcome data [1823]. Furthermore, the
traditional process does not allow for systematic reassessment of
risks and benets over time. Although an innovative drug may
initially possess advantageous riskbenet ratios versus older
drugs, these ratios often change over time, as occurred, for
example, with COX-II inhibitors.
Although none of the major regulatory agencies has a clear
benchmark for what constitutes an acceptable level of risk, nevertheless, the pharmaceutical industry is functioning in an era of
increased risk assessment, which requires proactive drug-safety
analysis and additional commitment to safety. Several dozen
risk-management programs or registries have been established by
major pharmaceutical companies in the United States, including
iPLEDGE to prevent exposure to isotretinoin during pregnancy, the clozapine patient registry to prevent agranulocytosis,
an alosetron prescribing program for reducing the risk of severe
gastrointestinal adverse events, and the STEPS program to
prevent exposure to thalidomide during pregnancy [15,24,25].
Moreover, various RBA methods have been utilized for clinical
decision-making in the following drug categories: oral contraceptives, antipsychotics, antihyperlipidemia medications, cancer
chemotherapy, iron-chelation, and antihypertensives, among
others [1719,2636].
Our study objectives are to review and compare published
quantitative RBA methodologies employed by regulatory agencies and/or the pharmaceutical industry. These comparisons may
help disclose unique characteristics of the RBA techniques that
may be more applicable to a specic drug evaluation scenario or
a specic therapeutic indication. This useful information can be
used to inform public and private RBA designers.

Methods
An extensive literature search was rst performed to identify
qualitative and quantitative approaches to RBA in drug evaluations. The MEDLINE and Cochrane Library databases were used
as well as other Internet-based search engines to nd scientic
publications, books, and academic conference proceedings.
Country-specic drug regulatory Internet Web sites were also
searched. Key words included risk, benet, riskbenet assessment, postmarketing surveillance, drug safety surveillance,
adverse drug reactions (ADRs), and pharmacovigilance. Our
inclusion criteria were 1) the publications were in English; and 2)
they focused on pharmaceutical or drug safety surveillance and
provided risk and/or benet assessment of drugs. Excluded were
conference programs, educational catalogues, announcements in
trade journals, veterinary-medicine publications, animal studies,
ADR case reports, abstracts without specic information on risk
or benet assessment, and general discussions related to risk or
benet assessment for medical devices or biotechnology.
The perspective on RBA was not limited to that of any one
stakeholder. Although both qualitative and quantitative RBA
studies were retrieved from the literature, we reviewed only those
using quantitative methods.

Results
Based on the key words, over 12,000 papers were identied.
Using our inclusion and exclusion criteria, over 300 abstracts
and Web site summaries related to RBA for drug safety were
identied and reviewed. Further selection of only quantitative
RBAs resulted in 59 articles, which were used as the basis for this
article. Figure 1 summarizes the process of study selection.
Because of limited published information regarding net clinical
benet analysis, the principle of threes, and net-benet-adjustedfor-utility analysis [10,21,37], we excluded these methods.
Remaining were 12 quantitative RBA methodologies, which are
individually described below and summarized in Table 1. They
are described roughly in chronological order.

Quantitative Framework for RiskBenet

Assessment (QFRBA)
Standard notions of risk and benet are well documented in the
literature as part of a basic quantitative framework (QFRBA).
The QFRBA attempts to quantify these notions directly, and
more recent RBA techniques build on these same ideas.
Risk refers to a comprehensive set of all possible adverse drug
events (ADEs) and a set of probabilities associated with these
adverse outcomes. The basic quantitative expression of risk is the
incidence of adverse events, i.e., the number of new ADEs in a
dened population over a specic period of time divided by the
population at risk for the adverse event over this same time
period [27,28,38,39]. Relative risk (RR) takes into account differences in exposure to the drug and is dened as the ratio
between the proportions of exposed individuals who experience
an ADE divided by the proportion of unexposed individuals who
experience an ADE [3841]. For example, whereas osteoporotic
individuals are more likely to experience a fracture, there is some
likelihood that people without osteoporosis will fracture as well,
and this fact is accounted for in the RR denominator. Attributable risk (AR), otherwise known as risk difference, uses the same
raw information as in the computation of RR but measures the
absolute difference between the risk of ADEs between those
exposed to the drug and those who were not and then divides this
difference by the absolute difference between the size of the two

659

RiskBenet Assessments for Drug Safety

Over 12,000 articles
searched using key words*

*Key words: Risk, benefit,

risk-benefit assessment, drug
safety surveillance, adverse drug
reactions, postmarketing
surveillance, pharmacovigilance

2000 5000 articles

Limited in English, human study, drug safety

300 400 articles

Further exclusions: ADR data-mining, signal
detection, effectiveness, case report.

Search engines: Web-based

MEDLINE & Cochrane plus Intranet
search for country-specific drug
regulatory.

Inclusion criteria: English, drug

intervention, quantitative risk
and/or benefit assessments.

Figure 1 Flowchart of inclusion and exclusion

criteria for identifying published riskbenet
assessment methods.

59 articles
Final list of published quantitative
risk-benefit assessments

groups [38,40,42]. The population-based attributable risk (PAR)

is an extension of AR that uses total population data. PAR is
particularly important for public-health decision-makers.
The benet associated with a medication might be its ability
to reduce either an ADE or another adverse event associated with
the disease itself. In this case, relative risk reduction (RRR) is
useful and is dened as the ratio between the proportion of
exposed individuals who experience a decline in adverse events
divided by the proportion of unexposed individuals who experience such a benet [27,28,38,40,43]. For instance, again in the
case of osteoporosis, RRR rates are computed for different antiosteoporosis medications based on the number of fractures
averted by the treatments. Similarly, attributable risk reducation,
otherwise known as absolute risk reduction (ARR), represents
the decline in adverse events between exposed and unexposed
groups [27,28,38,43]. However, benet does not always mean
absence of adverse events. From clinical trials, benet measurements might include clinically relevant efcacy parameters such
as specic biomarkers, surrogates, and putative surrogate end
points [4446]. From observational studies, benet measurements could include the patient medication and treatment adherence rate or population-based treatment effectiveness, among
many other possibilities.
The QFBRA methodology is widely used in drug safety surveillance by regulatory agencies and by the pharmaceutical
industry. It has a long history and a solid foundation. The
QFRBA does not, however, provide a methodology for combining risks and benets into a single value that could potentially
be used to compare riskbenet proles between alternative
therapies.

Benet-Less-Risk Analysis (BLRA)

For BLRA, risk and benet relationships are presented as risks
subtracted from benets using weights assigned to one of ve
benetrisk outcome categories: 1) efcacy without side effects;
2) efcacy with side effects; 3) no efcacy and no side effects; 4)
no efcacy with side effects; and 5) side effects leading to withdrawal from an ongoing treatment or intolerable side effects
[47,48]. The risk-adjusted benet measure for each individual is
obtained from the difference between an aggregate benet score
and an aggregate risk score.
Each patients efcacy experience (benet) of a therapy is
represented by a binary response variable; 1 signies that a

Exclusion criteria: nonacademic

or trade information, nonhuman
studies, case reports, no
assessment.

therapy response is obtained, and 0 means that no response is

achieved. The patients side effect experience (risk) from ve
different body functions is represented by a value ranging from
0.0 to 1.0, where the value of 1.0 represents the worst safety
experience and 0.0 means no safety concern. Notably, these
categories are primarily qualitative, but weights can be chosen by
individuals to reect the relative importance of the ve categories. A mathematical model is derived from these weights and
categories to calculate the observed benet-less-risk score for an
individual. If applied in a clinical trial comparing different treatments, statistical signicance tests can be performed to compare
treatments. The interpretation of the results remains focused on
the individual.
The weighting method used is subject to differences in interpretation between individuals. There may be a question regarding the validity of these assessments because the methods used to
present the relative weights may affect the outcome. The conversion from a qualitative assessment into a quantitative value may
also be subject to bias. The advantage of BLRA is that it provides
a structure for combining risks and benets into a single measure.

Quality-Adjusted Time Without Symptoms and

Toxicity (Q-TWiST)
Another riskbenet measure is Time Without Symptoms and
Toxicity (TWiST). Time lost due to ADEs or toxicity is subtracted from time gained through treatment. A quality-adjusted
version of TWiST, known as Q-TWiST, converts time into
QALYs; the QALys lost due to ADEs or toxicity are subtracted
from QALys gained from treatment [26,4951]. The benet is
measured by drug-attributed gain in QALYs. Cumulative risks of
toxicities and disease progression are calculated to obtain drugattributed loss of QALYs (risk). Q-TWiST compares the relative
therapeutic value of treatments based on the patients experience
within the context of clinical outcomes related to a disease and its
treatment. This method assumes that patients progress through a
series of health states with varying quality of life. This RBA
method has been used for riskbenet analysis of cancer treatments [50,52,53], as well as other therapies and transplantation
procedures [5456]. By weighting the durations of health states
by their respective utility values, a patient arrives at a single end
point reecting the duration of survival and the quality of life.
Using survival analysis techniques, the mean duration of each

Guo et al.

660
Table 1 Summary of twelve quantitative riskbenet assessment techniques
Seq. No.

Title and quantitative

approach

Parameters for assessment

Quantitative Framework for

Risk and Benet
Assessment (QFRBA)
[27,28,3840,4246]

Risk and benet are dened and quantied separately;

Risk focuses on adverse events or outcomes, dened as relative
risk, attributable risk, population attributable risk.
Benet focuses on risk differences (i.e., relative risk reduction,
absolute risk reduction).

Benet-less-risk analysis
(BLRA) [47,48]

Intensity scores are used to compare severity and frequency of

adverse drug events (ADEs) and assigned for each patient.
Data on observed benet from the treatment are required.
Proportionality constant determines how much penalty the ADEs
offset benet measure.

Quality-adjusted Time
Without Symptoms and
Toxicity (Q-TWiST)
[4957]

Benet measured as drug-attributed gain in quality-adjusted

life-years (QALYs);
Risk measured as drug-attributed loss of QALY
Compare gain versus loss of QALY.

Number needed to treat

(NNT) and number
needed to harm (NNH)
[27,28,5860]

Benet measure: number of persons treated (NNT) to avoid one

person developing disease of interest (absolute risk reduction,
Relative risk reduction);
Risk measure: number of persons treated when one person
experiences ADE (NNH);
Ratio of NNT and NNH.

Relative value adjusted

number needed to treat
(RV-NNT) [27,28]

Expands NNT to incorporate relative utility values (RV) based upon

patient preferences;
RV-NNH can be also be determined;
Ratio of RV-NNT and RV-NNH.

Minimum clinical efcacy

(MCE) [27,28,61]

Benet stated as efcacy difference between new treatment and

conventional treatment or placebo;
Harm stated as probability of AE (risk) in patients receiving new
treatment vs. conventional treatment or placebo;
Relative utility values may be considered.

Incremental net health

benet (INHB) [17,62]

Risk measured as decrease in QALY;

Benet measured as improvement in QALY;
INHB as relative gain or loss of QALYs due to treatment versus
usual care or placebo.

Riskbenet plane (RBP) and

riskbenet acceptability
threshold (RBAT) [18,64]

Risk measured as relative probability of risk of AEs between

treatment and control groups;
Benet measured as relative probability response between
treatment and control groups;
Visual application of riskbenet comparisons.

Probabilistic simulation
methods (PSM) and
Monte Carlo simulation
(MCS) [18,6264]

Average difference in the probability of risk and benet for the new
therapy relative to conventional therapy;
Incremental riskbenet ratio (IRBR).

10

Multicriteria decision analysis

(MCDA) [45,46]

Benet measured as clinically relevant end points from clinical trials;

Risk measured as incidence of ADE, discontinuation rate, drug
interactions, and other risk criteria;
Decision tree model is developed to incorporate all key risks and
benets;
Relative weights are of risks and benets are assigned.

11

Riskbenet Contour (RBC)

[66]

Probability of potential benet of treatment such as an increased

survival rate;
Probability of potential risk due to severe ADE or drug toxicity.

12

Stated preference method

(SPM) or maximum
acceptable risk (MAR)
[6777]

Relative utility values for therapeutic treatment alternatives;

Vector of attribute levels for treatment options;
Benetrisk trade-off preferences estimated by probability of severe
AEs versus benet in terms of treatment success;
Patient surveys required to provide data regarding value of benet
versus negative impact of risk.

Theoretical model and key features

Theoretically sound quantitative method;
Probability driven assessment for risk of adverse events and benets of
improved outcomes;
Relatively simple calculation;
Often used for drug safety surveillance in industry and regulatory
agencies.
Simple empirical method with sound theoretical basis;
Differences between treatments can be statistically analyzed (t-test or
ANOVA);
Requires subjective rankings of ADE intensity scores;
Patient preferences are incorporated using a discounting process;
Subjective ranking for ADEs and proportionality a potential threat to
internal validity;
Useful for comparing one drug treatment to a placebo or alternative
treatment using clinical trial data.
Statistical method can be conducted to compare alternative treatments;
QALYs incorporate patients preference measurement changes over
time;
Validity of QALY measurements and differences between techniques are
potential concerns;
Widely used riskbenet assessment technique in clinical oncology.
Well-dened quantitative framework;
Simple calculation for RBA used for comparing treatment and control
groups;
Lack of strong statistical properties;
Riskbenet relation can be compared directly by NNT to NNH ratio;
NNT should be lower than NNH for the drug to be valuable in term of
risk benet ratio;
Widely used for RBA in different therapy areas;
Difcult to incorporate more than one outcome (ADE or benet
reduction) simultaneously;
Relative value of ADE versus benet not considered.
Well-dened quantitative framework with relatively simple calculation;
RV measures involve patients preference for specic ADEs or avoidance
of negative clinical outcomes;
Similarly to NNT/NNH ratio, the RV-NNT must be higher than the
RV-NNH for the drug to be valuable.
Quantitative framework for integrating riskbenet data into a single
decision rule;
New treatment is warranted over conventional treatment if efcacy
exceeds probability of AE;
More than one type of AEs can be considered;
Lack of strong statistical properties;
Does not incorporate uncertainty in the benet or risk measurements;
Applied primarily for cardiovascular treatments.
Theoretically sound modeling method;
Riskbenet relation presented as an incremental difference;
QALY data from clinical trials are required;
Statistical variance in the estimates of both risks and benets can be
calculated;
Potential application in clinical and regulatory decision-making.
Well-dened theoretical model;
Two-dimensional plot with benet measurement on x-axis, risk
measurement on y-axis;
An acceptable threshold of relative riskbenet ratio can be plotted to
visually compare with other treatments;
Often used for explaining the phenomenon of drug safety surveillance.
Framework applies incremental riskbenet ratio;
Joint density of benet and risk scatter plot can be presented with a
riskbenet acceptability curve;
Involves simulation modeling of uncertainty around the incremental
riskbenet differences;
Statistical support incorporated using nonparametric bootstrap method
and MCS.
A decision tree describes clinical outcomes in a hierarchical manner;
Decision-making tool incorporates evaluation of both drugs risks and
benets;
Relative scores for alternative treatments can be calculated based on
modeling;
Data extraction from clinical trials are critical for internal validity;
Missing data and uncertainty can be addressed.
Implication for clinical and regulatory policy decision makings.
Graphical depiction of both benet (x-axis) and risk (y-axis)
measurements on a two-dimensional graph;
Riskbenet contours for comparative treatments are plotted as a set of
nonlinear curves;
For each individual patient, RBC scores can be identied with condence
intervals;
Useful tool for clinical making decisions.
SPM/MAR is based on hedonic-utility principles, and uses similar
techniques to contingent valuation;
Patients preferences for benetrisk trade-offs are incorporated.

661

RiskBenet Assessments for Drug Safety

disease state and the mean quality-adjusted survival time can be
estimated [5257].
As for BLRA, Q-TWiST provides a summary measure incorporating both benets and risks over time. Quality-adjusted survival is an understandable concept and can help individual
patients make informed treatment decisions. As with any QALYbased methodology, concerns regarding the validity and application of QALYs in a broad decision-making context are inherent
with the method. Additionally, the type of QALY-measurement
technique may inuence the results.

NNT and Number Needed to Harm (NNH)

NNT is dened as the number of patients who need to be treated
in order to prevent one more occurrence of a disease
[27,28,39,5860]. NNT can be calculated by taking the reciprocal of the ARR, and, as such, is dependent upon the incidence of
the disease for calculating ARR. Treatment is warranted if NNT
is low enough so that benets accrue at a sufciently low number
of patients before ADEs occur [27,28,5860].
NNH is calculated similarly for ADEs related to a specic
therapy, thus describing how many patients will be treated before
a patient will experience an ADE [27,28,5860]. Both NNT and
NNH have been used widely for RBA across different therapeutic
areas [29,30,33,34,58,59]. A comparison between NNT and
NNH can be used as a very basic comparison of benet versus
risk for a population of patients who may benet from the
treatment. Indeed, a riskbenet ratio equal to NNH/NNT can
be calculated between treatment and control groups. If the ratio
is greater than 1 (i.e., NNH/NNT > 1 or NNT < NNH), then
fewer patients need to be treated to achieve benet than will be
treated to have one additional occurrence of an ADE [27,28].
Rather than compare NNT and NNH explicitly, NNT can be
adjusted to account for ADE potential. Adverse-event-adjusted
NNT is dened as the number of patients who need to be treated
to prevent one additional disease outcome after adjusting for the
potential risks (e.g., ADEs) of the treatment, that is, after allowing for the probability of incurring a treatment-related adverse
event [27,28].
Both NNT and NNH are widely used in clinical practice
because they are simple decision-making tools for clinicians.
Indeed, these tools currently have a ubiquitous role in medical
decision-making. However, NNT and NNH by themselves do
not account for the patients value of the benet or the value of
the harm, respectively. In addition, it is cognitively complex to
incorporate several benets and harms simultaneously, which
commonly exists in clinical situations.

Relative-Value-Adjusted- (RV) NNT and RV-NNH

In an attempt to account for patient preferences, both the NNT
and NNH measures have been revised to incorporate patients
relative utility values, in this way correcting for one of the problems mentioned above. Adding a patients relative utility values
to the riskbenet evaluation should make this RBA method
more robust [27,28]. Relative utility values are obtained using
either the standard-gamble method or the time-trade-off
approach. A metric, RV, is calculated from a numeric representation of patients preferences for specic outcomes. Specically,
RV = (1 - utility of adverse event)/(utility of improvement using
a specic treatment), where the value of 1 represents the value of
perfect health. RV can then be interpreted as a quantication of
the overall value of patients preferences for avoiding an ADE
relative to avoiding the disease of interest or target event.
NNH adjusted for relative value can also be calculated similarly to RV-NNT. Thus, a riskbenet ratio (RV-NNH/NNT) can

be also calculated between treatment and control comparison

groups. A favorable riskbenet outcome is obtained when
RV-NNH/NNT > 1. This technique overcomes one of the weaknesses of NNH and NNT. It may also be possible to incorporate
several benets and risks simultaneously using this technique.

Minimum Clinical Efcacy (MCE)

The MCE of a treatment is dened as the minimal clinical efcacy
required by the treatment in comparison with a standard treatment, after taking into account the efcacy of the standard treatment, the adverse event proles associated with both the
standard treatment and the treatment under consideration, and
the risk of the disease of interest associated with no treatment
[27,28,61]. A broader denition of benet as the efcacy difference between two treatments (the new treatment versus the standard treatment) is calculated for RRR for the disease of interest.
Risk of an adverse event among persons treated with the new
treatment versus the standard treatment can also be calculated
[27,28,61]. A new treatment is warranted over the standard
treatment if the difference in risk is less than the efcacy difference. MCE seeks to improve clinical care through a quantitative
comparison of the potential benet against the potential risk of a
particular treatment. MCE takes into account not only the benets and harms of the new and standard treatments but also the
natural characteristics of the disease in the general population,
represented by an untreated group.
The MCE method seeks to nd the minimal therapeutic
benet at which a treatment is worth administering, which can be
used as a yardstick for the acceptance of a new treatment alternative. The details required to balance the ADE proles as well as
efcacy impact can be extensive. This method is similar to the
RV-NNT model [27,28], which cannot easily handle uncertainty
in the measurement of benets or risks.

Incremental Net Health Benet (INHB)

Riskbenet differentials can be expressed as either ratios or
differences, although the latter are more mathematically manageable if the units of measurement are the same. The INHB accomplishes this through the application of QALYs. The INHB of a
treatment in comparison with standard treatments is calculated
as the incremental difference between effectiveness and risk
changes; i.e., INHB = (E2 - E1) - (R2 - R1), where E denotes
effectiveness, and R is risk [17]. Both risk and benet must be
measured using QALYs. A favorable riskbenet balance exists
when (E2 - E1) > (R2 - R1). That is, the expected QALY gains as
a result of efcacy must exceed the expected losses from risk
(ADEs) in order for the net health benet to be positive [17,62].
Some of the literature mentioned that the benets and risks for
INHB could be measured using value-adjusted life-years as
opposed to QALYs [63].
The INHB approach is a theoretically sound modeling
method with strong potential for usefulness in clinical and regulatory decision-making. As with Q-TWiST, the potential problems with this technique rely upon the inherent difculties with
the use of QALYs as outcome measures. Furthermore, these
measurements must be accrued over time to gather sufcient
data. There may be benets or risks that occur after the time of
original data collection.

RiskBenet Plane (RBP) and RiskBenet Acceptability

Threshold (RBAT)
For this method, the benetrisk ratio can be interpreted as the
increase in the expected number of patients who will benet for

Guo et al.

662
NW
Experimental Therapy
Dominates

NE
High Risk
More Benefit

Risk-benefit
acceptability threshold
determined by the
slope of the line
Risk measurement can be
incidence or frequency or
severity of ADEs

(-)

Benefit measurement
can be incidence of
benefit or product
efficacy and responder
rate

Benefit

(+)

Point defined by the

observed difference in
risk and difference in
benefit

SW
Low Risk
Less Benefit

Figure 2 Hypothetical model for riskbenet

plan and acceptable riskbenet threshold for
riskbenet assessment. Revised gure from
Shaffer et al. [64] BMC Med Res Methodology
2006;6:48.

SE
Control Therapy
Dominates

(-)

Risk

(+)

each additional ADE that occursfrom new treatment rather

than usual care [64]. R = (PE - PC)/(qE - qC), where the probabilities of benet from the experimental treatment and in the control
arm are PE and PC, respectively, and the probabilities of risk from
the experimental treatment and in the control arm are qE and qC,
respectively [17,63].
Figure 2 shows a hypothetical model of the RBP, which is a
two-dimensional plot with benet and risk on the two axes,
including four quadrants NE, SE, NW, and SW [17,64]. The risk
measurement can be incidence of ADEs or frequency of ADE. If
the risk is on the x-axis, the risk for the new therapy increases
from left to right. The benet measurement can be incidence of
benet or product of efcacy and responder rate. The benet on
the y-axis increases from bottom to top. Hence, riskbenet
ratios in NW depicts that experimental therapy dominates
because of this treatment option with low risk and high benet.
In the SE quadrant, the active treatment option has higher risks
and lower benets (with a high riskbenet ratio), and the
control therapy is said to dominate. The remaining two quadrants involve high risk and more benet in SW, and less risk and
less benet in NE. An appropriate riskbenetRBAT will be
determined in RBP plot, which is indicated by a slope of line that
crosses over the SW and NE quadrants [15,62].
This method is a well-dened hypothetical model and offers a
visual tool to make complex comparisons between risk and

benet. Some ambiguity is associated with collapsing benets

and risks into single measures. Because of two-dimensional
model for RBP, it is unclear how one might incorporate multiple
dimensions of risks and benets. It may be useful to use these
types of graphical representations in patient decision-making
regarding alternative treatments.

Probabilistic Simulation Methods (PSMs) and Monte

Carlo Simulation (MCS)
Similar to the above RBP model, the average difference in the
probability of achieving a benet with the new therapy relative to
conventional therapy can be plotted on the x-axis (DB), and the
average difference in the probability of risk for the new therapy
can be plotted on the y-axis (DR). Both axes therefore range from
-1 to 1, with 0 at the origin. Then, four quadrants are labeled
with points of the compass NE, SE, NW, and SW [17,62,64].
Because the benet increases from left to right along the x-axis,
positive values (to the right of the vertical axis) represent greater
benets with the new treatment. Similarly, positive Y-coordinates
indicate a greater probability of the risk for the new treatment.
Figure 3 demonstrates a hypothetical model of PSM. Using
differences in the probability of achieving benet and risk, the
incremental riskbenet ratio related to the new therapy can be
dened as the incremental probability of an ADE (DR) with a

*UHDWHU5LVN

<D[LV$YHUDJH
GLIIHUHQFHLQWKH
SUREDELOLW\RIULVNIRU
WKHQHZWKHUDS\5

;D[LV$YHUDJH
GLIIHUHQFHLQWKH
SUREDELOLW\RI
DFKLHYLQJDEHQHILW
ZLWKWKHQHZWKHUDS\
UHODWLYHWR
FRQYHQWLRQDOWKHUDS\
%

1:
5
/HVV%HQHILWV
*UHDWHUSUREDELOLW\RI5LVN

1(
*UHDWHUEHQHILWV
*UHDWHUSUREDELOLW\RI5LVN

%

/HVV%HQHILW

6:
/HVV%HQHILWV
/HVVSUREDELOLW\RI5LVN



6(
*UHDWHU%HQHILWV
/HVVSUREDELOLW\RI5LVN

/HVV5LVN

Figure 3 Hypothetical model of probabilistic

simulation methods and Monte Carlo simulation
for riskbenet assessment. Revised gure from
Lynd LD, et al. [18] J Clin Epidemiology
2004;57:795803.

663

RiskBenet Assessments for Drug Safety

new therapy relative to conventional treatment divided by the
incremental probability of a benecial effect (DB) [18]. The y-axis
represents average difference in the probability of risk for the
new therapy versus conventional treatment.
There are two potential methods for quantifying the joint
density of the uncertainty around the risks and benets, depending on the accessibility of patient-level data [18]. If data are
available, a nonparametric bootstrap sample of data can be
selected repeatedly; if original data are not available, a simulation can be run using information on the distributions t to the
data. From the simulations or bootstrap estimates, the incremental riskbenet pairs can be plotted on the RBPriskbenet. For
example, in a randomized, double-blind, controlled clinical trial,
low-dose unfractionated heparin was compared with a low
molecular weight heparin, enoxaparin, for the prophylaxis of
venous thromboembolism following major trauma [65]. In a
previous clinical study, a MCS was applied to compare the efcacy and safety of administering anticoagulants to trauma
patients who are already at an elevated risk of bleeding [18].
RBP permits the estimation of the joint density of risks and
benets with their associated uncertainty and facilitates estimation of the probability that a therapy is net-benecial. MCS can
be used to compare drugs for both efcacy and safety. These tools
rely upon more complex statistical techniques to describe relationships between treatments similar to RBP. The same limitations exist, but the statistical modeling provides a greater level of
condence in the ndings, and provides the data used to generate
the results are valid and are collected reliably.

Multicriteria Decision Analysis (MCDA)

MCDA is a decision tool aimed at supporting decision-makers
who are faced with making numerous risk and benet evaluations. The risk can be measured by incidence of ADEs, discontinuation rate due to ADEs, and other risk factors such as
potential drug interactions, off-label use leading to safety
hazards, and safety issues observed in preclinical safety studies
[45,46]. The benet involves clinically relevant end points from
clinical trials and other benet criteria.
Using a decision value tree, a riskbenet ratio for a specic
drug therapy can be evaluated systematically. Both benet and
risk criteria can be split into multiple criteria in case of different
primary end points, relevant subgroups, and relevant interactions
[45,46]. Benet criteria can be split into one or more pivotal
trials and other benet criteria. Each pivotal trial may include
efcacy related to primary end points in overall subjects and in
relevant subgroups, as well as efcacy nonprimary end points.
Other benet criteria may include efcacy from nonpivotal trials
and anticipated patient compliance in clinical practice. Risk criteria are split into adverse effects, other risk criteria, and potential for nondemonstrated additional risks. Each adverse effect
involves its incidence and discontinuation rate. Other risk criteria
involve safety in subgroups, interaction with other drugs and
food, and potential for off-label use leading to safety hazards.
Potential for nondemonstrated additional risk involves long-term
exposure safety prole, safety issues observed in preclinical
studies, and safety issues observed in the same pharmacological
class [45,46].
MCDA can handle missing data and uncertainty using appropriate modeling and relative weights. There are several key steps
for MCDA. First, the decision context (tree) needs to be established with dened options. Clear objectives and criteria should
be identied for assessing the consequences for each option with
high-level and low-level rates of occurrence in the hierarchy.
Then, the expected performance scores of each option against the

criteria of consequences should be assessed. Finally, weighted

scores at each level in the hierarchy and overall weighted scores
can be calculated. The result can be examined with sensitivity
analysis [45,46].
MCDA is a relatively new model that takes in account multiple criteria, judgment, and uncertainty. Although the MCDA
model can be customized by adding or changing benet and risk
criteria, the process may be too burdensome for limited evaluations. Data extraction from clinical trials is critical for the internal validity assessment of the MCDA technique.

RiskBenet Contour (RBC)

The RBC is another relatively new method that provides a twodimensional graph showing both the probability of benet from
treatment based, for example, on the survival rate and the probability of drug toxicity or ADEs (the risk) [66]. The degree of
drug benet is captured along the x-axis, and the degree of drug
risk is measured along the y-axis. By nding out from each
patient the amount of risk he or she is willing to accept to obtain
a certain benet, a set of individual RBCriskbenets can be
determined. For example, the RBC can show whether a patient is
willing to accept a 40% probability of acute toxicity as long as
there is a 5% improvement in the chance of survival [66].
The RBC approach is a way of formalizing the riskbenet
trade-off for patients and, as such, can be a useful tool for
patients and physicians. It can take information from clinical
trials in order to present the riskbenet trade-off visually.
However, as unique to each patient, the, RBC method does not
have an intrinsic riskbenet threshold and does not allow for
multiple risks and multiple benets from a drug therapy.

Stated Preference Method (SPM) or Maximum

Acceptable Risk (MAR)
SPM or MAR is based on hedonic-utility principles and therapeutic treatment options (commodities) over which consumers
make choices. Consumer choices can be considered as a random
utility function specied as Uj = Vj + ej with Vj = Xjb [67], where
Vj is the determinate part of the utility function for treatment j,
Xj is a vector of attribute levels for treatment j, b is a vector of
attribute parameters, and ej is a random error. Benetrisk tradeoff preferences can be estimated based on consumer experience
or probability of adverse events (AEs). Patients preferences can
be collected from survey questionnaires and interview techniques
such as contingent valuation techniques. The current best practice standard requires participants to make trade-offs between
choices using discrete choice experiments. Bestworst scaling
methods are also being developed and may become more wisely
used in the future. Using SPM or MAR, the riskbenet trade-off
can be calculated as the increase in risk of AEs that reduces the
patients satisfaction scores between two treatment options. This
RBA method has been applied to several therapeutic areas such
as Crohn, HIV, type 2 diabetes, multiple sclerosis treatments, and
vasomotor symptom relief [6877].
SPM or MAR is theoretically sound and uses similar techniques to contingent valuation. It requires the collection of
patients treatment preference, which allows the evaluation of
benetrisk trade-offs. The method can demonstrate the patients
willingness to accept risks to the benets of controlling disease
symptoms.

Discussion
Multiple scientic methods that quantify benets and risks of
medical treatments are available and can be used to reduce the

Guo et al.

664
current subjective onus on regulatory agencies, thus guiding them
to more objective and transparent decisions regarding drug efcacy and safety. Each assessment method requires different data
and has its unique characteristics and features, as well as
strengths and weaknesses. The reviewed methodologies for RBA
are representative of technical advances in the eld of quantitative analysis. So far, they have been used primarily for research
demonstrations and to showcase their scientic and mathematical feasibility. They have not yet been systematically adopted by
regulatory agencies or by the pharmaceutical industry. The
reviewed RBA methods might be helpful for making decisions at
the population level. In addition, they are clearly valuable in
providing information for individual patients and their physicians in disease treatment decisions.
In trying to develop a consistent RBA strategy, Lynd (Lynd L,
unpublished presentation) suggested that desirable features
include 1) universal applicability across medications; 2) inclusiveness (taking account of all risks and benets); 3) patient
sensitivity (person-specic preference weightings); 4) straightforward interpretability; 5) exibility; 6) adaptability; 7) ability to
incorporate uncertainty; 8) integrability with economic evaluations of medications; and 9) ability to be consistent with a
quantitative riskbenet threshold. Some methods like INHB,
RBP, PSM, MCDA, SPM, and RBC seem to be exible and easier
to adopt for certain clinical situations than others. Although
some have argued that INHB is the best RBA because it meets all
nine criteria, it is too early to conclude that any single method is
suitable for all purposes [2,810,21,66,78]. From ongoing academic activities at both the FDA and the EMEA, three RBA
approaches have received the most attention: INHB using healthoutcomes modeling, MCDA, and SPM (comments from an
expert reviewer which is based on communication with the FDA
and the EMEA).
Because RBP and PSM provide health-care decision-makers
with a functional relationship between patient preference and
probability of net benet, they may be two promising methods
for forming a standardized approach to modeling riskbenet
relationships. Signicant research, however, is required to determine probability distributions for incremental risks and benets.
Further validation using multiple data sets and various simulation techniques may be required to generate comparative results.
On the other hand, BLRA may be most appropriate for comparisons between treatments in the same therapeutic class because it
takes both benet and risk into consideration as long as the risks
and benets are available in similar qualitative measurements.
Nevertheless, BLRA does not clearly delineate benet and risk,
and interpretation is very complicated.
We note that both RV-NNT and MCE analysis rely heavily
upon efcacy or effectiveness data, adverse event rates, and
patient preference (utility) data. Patients must specify their relative preferences for an outcome given potential risks. These data
may be difcult to obtain. Furthermore, NNT has been criticized
for its lack of strong statistical properties. When the ARR value
gets closer to 0, NNT will be nearly innity and therefore hard to
interpret [27,28]. MCE analysis allows for determining the
worth of a new treatment relative to an older one, given not only
the potential risks of adverse events and benets that may be
gained, but also taking into account the risk of disease without
any treatment. However, statistical properties for MCE are
unstudied, and it is not clear whether a valid condence interval
can be created around an MCE value [27,28].
The QALY is often used to measure disease burden and
provides the foundation for cost-effectiveness analysis despite the
difculties in measuring QALYs lost or gained through adverse
events or successful therapy, respectively. QALYs are used for

both INHB and Q-TWiST; benet is measured by expected

health improvements (QALY gains), whereas risk is measured by
adverse health impacts (QALY losses). QALY estimates for
known potential side effects may be inferred from knowledge of
the products mechanism of action or signals from clinical trials.
Information about rare events may be nonexistent when a drug is
rst marketed before wider exposure to the drug by patients.
Presumably, one could rely on broad historical experience in
order to include an explicit but very small probability for a rare
event [17,62]. In the absence of obtainable utilities from the
literature or other studies, primary data collection may be
required to determine these values.
We are not suggesting that any or all of the quantitative RBA
approaches reviewed here should replace the existing decisionmaking process of either clinicians or regulatory agencies; rather,
they should serve as supplemental tools to inform such decisionmaking. Quantitative RBA methodologies may be particularly
benecial in an era of limited health-care resources for making
appropriate decisions about pharmaceutical products. All
reviewed RBA methods have limitations, predominantly in their
data requirements, statistical properties, and the availability of
valid patient preference measures (utilities). In many situations,
we are limited to retrospective analysis of historical data and
existing databases.

Limitations and Future Research

The present study was limited to review of existing, published
scientic RBA techniques by using a Web-based literature review
of peer-reviewed journal articles found through PubMed/
MEDLINE, the Cochrane Library, the FDA repository, and
other Internet-based search engines. Many other riskbenet
approaches based on specic clinical parameters or specic perspectives were not included [7984]. There may also be other
quantitative approaches that regulators have not formalized or
published. Further research is warranted to ensure the robustness
of these latter measures and to dene the appropriate context in
which they should be applied. Our review of RBA methods is
consistent with the observation that technical development of
riskbenet analysis is far from what is needed in the contemporary regulatory environment of pharmaceutical product
development.

Conclusion
Several scientically sound, quantitative RBA methodologies can
be used to reduce the current subjective onus on the regulatory
agencies and help guide them toward making more objective,
transparent, and evidenced-based decisions regarding drug risks
and benets. Each quantitative method has its unique advantages
and disadvantages based on data requirements and statistical
properties. Whereas some methods rely on subjective weighting
schemes or simple statistical assessments (e.g., NNT and NNH),
other methods are used for obtaining joint distributions of
benet and risk (e.g., RBP, PSM, MCDA, SPM, and RBC) and
have a sound mathematical basis. Methods that incorporate the
patients risk tolerance and preference for health states may
represent a promising channel of study in this area. Numerous
RBA methodologies have been proposed, but there are a limited
number of empirical applications of these techniques, and there is
no consensus among regulators necessary for dening a clear
gold standard. More testing, and particularly head-to-head comparisons, is needed if these approaches are to receive serious
consideration. When evaluating any new health-care technology,
we recommend the use of multiple RBA approaches across

RiskBenet Assessments for Drug Safety

different therapeutic indications and treatment populations to
bound the riskbenet prole.

Acknowledgements
The authors would like to thank both Pamela C. Heaton and
Azhar Choudhry for their suggestions and advices for the early
research report. We would like to thank Dr. Christina ML.
Kelton for her language edits. We recognize the efforts of Ashish
Parekh, Shital Kamble, Anthony Lockett, Kara Suter, and many
committee members from the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) RiskBenet
Management Working Group for their helpful comments.
Source of nancial support: The authors did not receive any funding for
this study. This study was presented as a research workshop at the ISPOR
Annual Meeting, Orlando, FL, USA, May 2009.

References
1 FDA. Drug safety annual report to the nation 2006. Available
from: http://www.fda.gov/cder/ [Accessed October 31, 2007].
2 FDA. FDA issues nal risk minimization guidance 2005. Available from: http://www.fda.gov/bbs/topics/news/2005 [Accessed
October 31, 2007].
3 FDA. Premarketing risk assessment guidance 2005. Available
from: http://www.fda.gov/cder/guidance [Accessed October 31,
2007].
4 FDA. Development and use of risk minimization action plans
2005.
Available
from:
http://www.fda.gov/cder/guidance
[Accessed October 31, 2007].
5 FDA. Good pharmacovigilance practices and pharmacoepidemiologic assessment 2005. Available from: http://www.fda.gov/cder/
guidance [Accessed October 31, 2007].
6 Callreus T. The precautionary principle and pharmaceutical risk
management. Drug Saf 2005;28:46571.
7 FDA. FDA Amendment Act 2007. Available from: http://www.
fda.gov/oc/initiatives/HR3580.pdf [Accessed January 2, 2009].
8 Strom BL. How the US drug safety system should be changed.
JAMA 2006;295:20725.
9 Ray WA, Stein CM. Reform of drug regulationbeyond an independent drug-safety board. N Engl J Med 2006;354:194201.
10 Schmitt S, Lis Y, Carter M. Benet risk assessment during
research and development. In: Schmitt S, ed. Risk-Based Compliance Handbook (Chapter 7). Baltimore, MD: PDA/DHI Publisher, 2008.
11 Okie S. Safety in numbersmonitoring risk in approved drugs. N
Engl J Med 2005;352:11736.
12 Veatch RM. Benet risk assessment: what patients can know that
scientists cannot. Drug Info J 1993;27:10219.
13 Bahri P, Tsintis P, Waller PC. Regulatory pharmacovigilance in
the EU. In: Mann R, Andrews E, eds. Pharmacovigilance (2nd ed.)
(Chapter 14). Hoboken, NJ: Wiley, 2007.
14 European Medicine Agency (EMEA) CHMP. Reection paper on
benetrisk assessment in the context of the evaluation of marketing authorization applications of medicincal products for
human use. Available from: http://www.emea.europa.eu/pdfs/
human/brmethods/1540407en.pdf [Accessed March 19, 2008].
15 Guo JJ, Goehring E, Jones JK. The story of cisapride and its
withdraw from market: a case study. In: Hartzema AG, Tilson
HH, Chan AK, eds. Pharmacoepidemiology and Therapeutic Risk
Management (Chapter 29). Cincinnati, OH: Harvey Whitney
Books, 2008.
16 Gould AL. Practical pharmacovilance analysis and strategies.
Pharmacoepidemiol Drug Saf 2003;12:55974.
17 Garrison LP, Towse A, Bresnahan BW. Assessing a structured,
quantitative health outcomes approach to drug riskbenet
analysis. Health Aff 2007;26:68495.
18 Lynd LD, OBrien BJ. Advances in riskbenet evaluation using
probabilistic simulation methods: an application to the prophylaxis of deep vein thrombosis. J Clin Epidemiol 2004;57:795803.

665
19 Sutton AJ, Cooper NJ, Abrams KR, et al. A Bayesian approach to
evaluating net clinical benet allowed for parameter uncertainty.
J Clin Epidemiol 2005;58:2640.
20 Moynihan R, Bero L, Ross-Degnan L, et al. Coverage by the news
media of the benets and risks of medications. N Engl J Med
2000;342:164550.
21 Holden WL. Benetrisk analysis: a brief review and proposed
quantitative approaches. Drug Saf 2003;26:85362.
22 Arlett P. Risk benet assessments. Pharm Physician 2001;12:12
17.
23 Edwards R, Wiholm B-E, Martinez C. Concepts in riskbenet
assessment. A simple merit analysis of a medicine? Drug Saf
1996;15:8995.
24 FDA. Historical information on Natalizumab (marketed as
Tysabri) 2001. Available from: http://www.fda.gov/cder/drug/
infopage/natalizumab/historical.html [Accessed October 31,
2007].
25 FDA. FDA approves lotronex for IBS in women. February 10,
2000. Available from: http://womenshealth.about.com/library/
weekly/aa021000a.htm [Accessed on May 20, 2008].
26 Gelber RD, Goldhirsch A, Cavalli F. Quality-of-life-adjusted
evaluation of adjuvant therpies for operable breast cancer. The
international breast cancer study group. Ann Intern Med
1991;114:6218.
27 Holden WL, Juhaeri J, Dai W. Benetrisk analysis: a proposal
using quantitative methods. Pharmacoepidemiol Drug Saf
2003;12:61116.
28 Holden WL, Juhaeri J, Dai W. Benetrisk analysis: examples
using quantitative methods. Pharmacoepidemiol Drug Saf
2003;12:6937.
29 Segal ES, Valette C, Oster L, et al. Risk management strategies in
the post-marketing period: safety experience with the US and
European boasentan surveillance programmes. Drug Saf 2005;
28:97180.
30 Yildiz BO. Oral contraceptives in polycystic ovary syndrome:
riskbenet assessment. Semin Reprod Med 2008;26:11120.
31 Tranulis C, Skalli L, Labonde P, et al. Benets and risks of antipsychotic polypharmacy: an evidence-based review of the literature. Drug Saf 2008;31:720.
32 Lea R, Whorwell PJ. Benetrisk assessment of tegaserod in irritable bowel syndrome. Drug Saf 2004;27:22942.
33 Lyman GH. Risk assessment in oncology clinical practice: from
risk factors to risk models. Oncology 2003;17(Suppl. 11):S8
13.
34 Chuck SK, Penzak SR. Riskbenet of HMG-CoA reducatase
inhibitors in the treatment of HIV protease inhibitor-related
hyperlipidaemia. Expert Opin Drug Saf 2002;1:517.
35 Porter JB. A riskbenet assessment of iron-chelation therapy.
Drug Saf 1997;17:40721.
36 Burrell LM. A riskbenet assessment of losartan potassium in
the treatment of hypertension. Drug Saf 1997;16:5665.
37 Cohen TJ, Neumann PJ. Whats more dangerous, your aspirin or
your car? Thinking rationally about drug risks and benets.
Health Aff 2007;26:63646.
38 Bjornson DC. Interpretation of drug risk and benet: individual
and population perspectives. Ann Pharmacother 2004;38:6949.
39 Schechtman E. Odds ratio, relative risk, absolute risk reduction,
and the number needed to treatwhich of these should we use?
Value Health 2002;5:4316.
40 Rothman KJ, Greenland S. Introduction to Categorical Statistics.
Modern Epidemiology (2nd ed.). Philadelphia, PA: LippincottRaven, 1998.
41 Davies HTO, Crombie IK, Tavakoli M. When can odds ratios
mislead? BMJ 1998;316:98991.
42 Nuevo J, Melnikov J, Chang D. Reporting number needed to treat
and absolute risk reduction in randomized controlled trials.
JAMA 2002;287:281314.
43 Califf RM. Benet assessment of therapeutic products: the
Centers for Education and Research on Therapeutics. Pharmacoepidemiol Drug Saf 2007;16:516.
44 Writing Group for the Womens Health Initiative Investigators. Risks and benets of estrogen plus progestin in healthy

Guo et al.

666

45

46

47
48

49

50
51

52

53

54

55

56

57

58

59
60

61

62

63

postmenopausal women-principal results from the Womens

Health Initiative randomized controlled trial. JAMA 2002;288:
32133.
Mussen F, Salek S, Walker S. A quantitative approach to benet
risk assessment of medicinespart 1: the development of a new
model using multi-criteria decision analysis. Pharmacoepidemiol
Drug Saf 2007;16:S215.
Mussen F, Salek S, Walker S. A quantitative approach to benet
risk assessment of medicinespart 2: the practical application
of a new model. Pharmacoepidemiol Drug Saf 2007;16:S16
41.
Chuang-Stein C. A new proposal for benet-less-risk analysis in
clinical trials. Control Clin Trials 1994;15:143.
Tallarida RJ, Murray RB, Eiben C. A scale for assessing the
severity of diseases and adverse drug reactions. Application to
drug benet and risk. Clin Pharmacol Ther 1979;25:38190.
Gelber RD, Goldhirsh A. A new endpoint for the assessment of
adjuvant therapy in postmenopausal women with operable breast
cancer. J Clin Oncol 1986;4:17729.
Gelber RD, Goldhirsch A, Cole BF. Evaluation of effectivenss:
Q-TwiST. Cancer Treat Rev 1993;19(Suppl. A):S7384.
Gelber RD, Goldhirsch A, Cole BF, et al. A quality-adjusted time
without symptoms or toxicity (Q-TWiST) analysis of adjuvant
radiation therapy and chemotherapy for resectable rectal cancer.
J Natl Cancer Inst 1996;88:103945.
Sherrill B, Amonkar MM, Stein S, et al. Q-TWiST analysis of
lapatinib combined with capecitabine for the treatment of metastatic breast cancer. Br J Cancer 2008;99:71115.
Goldhirsch A, Gelber RD, Simes RJ, et al. Costs and benets of
adjuvant therapy in breast cancer: a quality-adjusted survival
analysis. J Clin Oncol 1989;7:3644.
Hynes N, Sultan S. A prospective clinical, economic, and qualityof-life analysis comparing endovascular aneurysm repair (EVAR),
open repair, and best medical treatment in high-risk patients with
abdominal aortic aneurysms suitable for EVAR. J Endovasc Ther
2007;14:76376.
Mounier N, Haioun C, Cole BF, et al. Quality of life-adjusted
survival analysis of high-dose therapy with autologous bone
marrow transplantation versus sequential chemotherapy for
patients with aggressive lymphoma in rst complete remission.
Groupe dEtude les Lymphomes de lAdulte (GELA). Blood
2000;95:368792.
Agarwala SS, Kirkwood JM. Adjuvant interferon treatment
for melanoma. Hematol Oncol Clin North Am 1998;12:823
33.
Cole BF, Gelber RD, Gelber S, Mukhopadhyay PA. A qualityadjusted survival model (Q-TWiST) for advanced stage cancer. J
Biopharm Stat 2004;14:11124.
Laupacis A, Sackett D, Roberts R. An assessment of clinically
useful measures of the consequences of treatment. N Engl J Med
1988;318:172833.
Cook D, Sackett D. The number needed to treat: a clinically
useful measure of treatment effect. BMJ 1995;310:4524.
Chatellier G, Zapletal E, Lemaitre D, et al. The number needed to
treat: a clinically useful mammogram in its proper context. BMJ
1996;312:4269.
Djulbegovic B, Hozo I, Fields K, Sullivan D. High-dose chemotherapy in the adjuvant treatment of breast cancer: benet/risk
analysis. Cancer Control 1998;5:394405.
Ponce RA, Bartell SM, Wong EY, et al. Use of quality-adjusted
life year weights with dose-response models for public health
decisions: a case study of the risks and benets of sh consumption. Risk Anal 2000;20:52942.
Lynd LD, Najafzadeh M, Colley L, et al. Using the incremental
net benet framework for quantitative benetrisk analysis in
regulatory decision-makinga case study of alosetron in irritable

64

65

66

67

68

69

70

71

72

73

74

75
76

77

78
79

80
81

82

83

84

bowel syndrome. Value Health 10 September 2009 [Epub ahead

of print].
Shaffer ML, Watterberg KL. Joint distribution approaches to
simultaneously quantifying benet and risk. BMC Med Res
Methodol 2006;6:48.
Geerts WH, Jay RM, Code KI, et al. A comparison of low-dose
heparin with low-molecular-weight heparin as prophylaxis
against venous thromboembolism after major trauma. N Engl J
Med 1996;335:7017.
Shakespeare TP, Gebski VJ, Veness MJ, Simes J. Improving interpretation of clinical studies by use of condence levels, clinical
signicance curves, and riskbenet contours. Lancet 2001;357:
134953.
Johnson FR, Ozdemir S, Manseld C, et al. Are adult patients
more tolerant of treatment risks than parents of juvenile patients?
Risk Anal 2009;29:12136.
Hauber AB, Mohamed AF, Johnson FR, Falvey H. Treatment
preferences and medication adherence of people with Type 2
diabetes using oral glucose-lowering agents. Diabet Med
2009;26:41624.
Hauber AB, Mohamed AF, Watson ME, et al. Benets, risk, and
uncertainty: preferences of antiretroviral-nave African Americans
for HIV treatments. AIDS Patient Care STDS 2009;23:2934.
Johnson FR, Van Houtven G, Ozdemir S, et al. Multiple sclerosis
patients benetrisk preferences: serious adverse event risks
versus treatment efcacy. J Neurol 2009;256:55462.
Johnson FR, Ozdemir S, Hauber B, Kauf TL. Womens willingness to accept perceived risks for vasomotor symptom relief. J
Womens Health 2007;16:102840.
Bryan S, Martin B, Robert S, Alison G. Magnetic resonance
imaging for the investigation of knee injuries: an investigation of
preferences. Health Econ 1998;7:595603.
Ryan M, McIntosh E, Shackley P. Methodological issues in the
application of conjoint analysis in health care. Health Econ
1998;7:3738.
Johnson FR, Banzhaf M, Desvousges W. Willingness to pay for
improved respiratory and cardiovascular health: a multiple-format
stated-preference approach. Health Econ 2000;9:295317.
Gan TJ, Lubarsky DA, Flood EM, et al. Patient preferences for
acute pain treatment. Br J Anaesth 2004;92:6818.
Johnson FR, Ozdemir S, Manseld C, et al. Crohns disease
patients riskbenet preferences: serious adverse event risks
versus treatment efcacy. Gastroenterology 2007;133:76979.
Hauber AB, Johnson FR, Fillit H, et al. Older Americans risk
benet preferences for modifying the course of Alzheimer disease.
Alzheimer Dis Assoc Disord 2009;23:2332.
Lasagna L. Balancing risks versus benets in drug therapy decisions. Clin Ther 1998;20:C729.
Bass R. Riskbenet decisions in product license applications. In:
Walker SR, Asscher AW, eds. Medicines and Risk/Benet Decisions. Lancaster: MTP Press Limited, 1987.
Beckmann J. Basic aspects of riskbenet analysis. Semin Thromb
Hemost 1999;25:8995.
Brendorp B, Pedersen OD, Torp-Pedersen C, et al. A benetrisk
assessment of class III antiarrhythmic agents. Drug Saf 2002;
25:84765.
Mancini GBJ, Schulzer M. Reporting risks and benets of therapy
by use of the concepts of unqualied success and unmitigated
failure; applications to highly cited trials in cardiovascular medicine. Circulation 1999;99:37783.
Luijn HEM, Musschenga AW, Keus RB, et al. Assessment of the
risk/benet ratio of phase II cancer clinical trials by Institutional
Review Board (IRB) members. Ann Oncol 2002;13:130713.
Burkell J. What are the chances? Evaluating risk and benet
information in consumer health materials. J Med Libr Assoc
2004;92:2008.