VALUE IN HEALTH 20 (2017) 909918

Available online at www.sciencedirect.com

journal homepage: www.elsevier.com/locate/jval

A Review of Empirical Analyses of Disinvestment Initiatives

James D. Chambers, PhD1,*, Mark N. Salem, BA1, Brittany N. DCruz, BA1, Prasun Subedi, PhD2,
Sachin J. Kamal-Bahl, PhD2, Peter J. Neumann, ScD1
1
Center for the Evaluation of Value and Risk in Health, Institute for Clinical Research and Health Policy Studies, Tufts Medical Center,
Boston, MA, USA; 2Global Health and Value, Innovation Center, Pzer Inc., New York, NY, USA

AB STR A CT

Background: Disinvesting in low-value health care services provides
opportunities for investment in higher value care and thus an
increase in health care efciency. Objectives: To identify interna-
tional experience with disinvestment initiatives and to review empir-
ical analyses of disinvestment initiatives. Methods: We performed a
literature search using the PubMed database to identify international
experience with disinvestment initiatives. We also reviewed empirical
analyses of disinvestment initiatives. Results: We identied 26
unique disinvestment initiatives implemented across 11 countries.
Nineteen addressed multiple intervention types, six addressed only
drugs, and one addressed only devices. We reviewed 18 empirical
analyses of disinvestment initiatives: 7 reported that the initiative
was successful, 8 reported that the initiative was unsuccessful, and 3
reported that ndings were mixed; that is, the study considered
multiple services and reported a decrease in the use of some but
not others. Thirty-seven low-value services were evaluated across the
18 empirical analyses, for 14 (38%) of which the disinvestment
initiative led to a decline in use. Six of the seven studies that reported
the disinvestment initiative to be successful included an attempt to
promote the disinvestment initiative among participating clinicians.
Conclusions: The success of disinvestment initiatives has been
mixed, with fewer than half the identied empirical studies reporting
that use of the low-value service was reduced. Our ndings suggest
that promotion of the disinvestment initiative among clinicians is a
key component to the success of the disinvestment initiative.
Keywords: disinvestment, health care efciency, low-value care,
resource allocation.
Copyright & 2017, International Society for Pharmacoeconomics and
Outcomes Research (ISPOR). Published by Elsevier Inc.

cost-effective practices. Historically, health care systems have relied

Introduction
Health technology assessment (HTA) agencies have conventionally
evaluated the effectiveness and cost-effectiveness of new technol-
ogies to determine whether a health care system should invest in
them [1]. Nevertheless, observers argue that by focusing efforts on
assessing new technologies, rather than on broader efciency
questions, HTA agencies have contributed to increases in health
care expenditures and pressure on health care budgets [2,3].
Health care systems across the globe are increasingly recogniz-
ing that to control health care costs it is necessary to complement
judicious investment in new health care technology with strategies
to reduce the use of unnecessary, ineffective, inefcient, or harmful
care [4]. These strategies, commonly referred to as disinvestment
initiatives, have been dened as the partial or complete with-
drawal of health resources from existing health care practices,
procedures, technologies, or drugs that are deemed to deliver little
or no health gain for their cost [5]. Reducing spending on low-value
care provides opportunities for a health care system to invest in
higher value care and thus to increase health care efciency.
The introduction of new technologies will increase health care
system efciency only if the introduced technologies displace less
on passive disinvestmentor natural attritionto reduce use of
low-value or inappropriate care [4]. In other words, policymakers
have assumed that in response to the introduction of new and
effective technologies, clinicians will modify their practice and stop
using, or use less frequently, existing less effective or cost-effective
alternatives. There are a range of levers that exist to guide the use
for new and existing technologiesincluding clinical guidelines and
clinical support toolsbut there is a growing consensus that
existing approaches are insufcient, and that hurdles such as the
entrenchment of low-value services in clinical practice and other
biases hinder the reduction of ineffective, inefcient, unnecessary,
or harmful care [69].
Health care systems in various countries have experimented
with a range of active disinvestment initiatives to address low-value
services. Among these initiatives is the National Institute for Health
and Care Excellences (NICEs) do not do list of low-value inter-
ventions in the United Kingdom and the Choosing Wisely Cam-
paign that has been implemented in the United States and other
countries to encourage a reduction in the use of wasteful or
unnecessary medical tests, procedures, and treatments [4,10,11].
This experimentation has not led, however, to a single widely

* Address correspondence to: James D. Chambers, Center for the Evaluation of Value and Risk in Health, Tufts Medical Center, 800
Washington Street No. 063, Boston, MA 02127.
E-mail: jchambers@tuftsmedicalcenter.org.
1098-3015$36.00 see front matter Copyright & 2017, International Society for Pharmacoeconomics and Outcomes Research (ISPOR).
Published by Elsevier Inc.
http://dx.doi.org/10.1016/j.jval.2017.03.015
910 VALUE IN HEALTH 20 (2017) 909918
accepted or validated approach, or to a consensus on what leads to
a successful disinvestment initiative. We examined the success of
disinvestment strategies by reviewing empirical evaluations of
disinvestment initiatives implemented by health care systems
across the globe. Our study had two objectives: 1) to sample the
literature to identify and describe disinvestment initiatives imple-
mented by health care systems in various countries and 2) to review
empirical evaluations of disinvestment initiatives to examine their
success in reducing the use of low-value services.
Methods
Study Objective 1: Identifying International Disinvestment
Initiatives
We searched the PubMed database to sample the literature for
studies that described disinvestment initiatives published through
May 5, 2016, using the following search terms: disinvestment,
decommission, delist, health technology reassessment, low-
value, marginal value, reallocation, resource allocation, and
resource management. We limited our search to studies in
English and did not restrict study inclusion on the basis of
publication date. We used the same search terms to search on
Internet search engines. We also searched the gray literature
including government documents and academic working papers
for pertinent studies.
We included studies that described or evaluated national
disinvestment initiatives that addressed any type of health care
service, including drugs, medical devices, diagnostic imaging and
screening tests, surgical procedures, and so on. We excluded
strategies that were not national disinvestment initiatives, for
example, those limited to individual hospitals. We also excluded
studies that described program budgeting and marginal analysis
programs because although this technique considers simultane-
ous investment and disinvestment of health care interventions
or programs, it did not meet our denition of a disinvestment
initiative.
Study Objective 2: Reviewing Empirical Evaluations of
Disinvestment Initiatives
We reviewed the studies that met the criteria as outlined in study
objective 1 to determine those that were empirical analyses of
disinvestment initiatives, that is, those that compared the use of
a low-value service before and after the implementation of the
disinvestment initiative. We evaluated each empirical study that
met our inclusion criteria. First, we determined what disinvest-
ment initiative was being evaluated and its setting. Second, we
reported the low-value service or intervention to which the
disinvestment initiative pertained. Third, we reported details on
the studys design, including, sample size, the inclusion of a
control group, study duration, and the inclusion of a phase-in
period, that is, a period of time to allow physicians to become
aware of the disinvestment initiative before assessing its impact
on prescribing. Fourth, we determined the success of the dis-
investment initiative. We considered the initiative to be success-
ful if the study found it to have resulted in a decline in the use of
the low-value service attributable to the disinvestment initiative
relative to the baseline use of the service (pre-initiative). We
reported the statistical signicance of the study ndings when
presented in the study. We considered the initiative to be
unsuccessful if it did not result in a decline in the use of the
low-value service attributable to the disinvestment initiative. We
considered the success of the initiative to be mixed if the study
considered multiple low-value services and reported a decrease
in the use of some services but not others.
Results
Study Objective 1: Identifying International Experience with
Disinvestment Initiatives
We identied 26 unique disinvestment initiatives implemented
from 1984 to 2014 (Table 1). Eighteen initiatives (69%) were
established in 2005 or more recently, including 10 (38%) estab-
lished in 2010 or more recently. Nineteen initiatives addressed
multiple intervention types, six addressed only drugs, and one
addressed only devices.
One initiative, the Choosing Wisely Campaign, has been
implemented in six countries. Australia (seven initiatives), the
United Kingdom (six initiatives), and New Zealand (three initia-
tives) have implemented the most disinvestment initiatives.
Twenty-ve initiatives were implemented in a single country.
Study Objective 2: Reviewing Empirical Evaluations of
Disinvestment Initiatives
We identied 18 empirical evaluations of disinvestment initia-
tives (Table 2). Nine evaluated Choosing Wisely Campaign rec-
ommendations, eight evaluated disinvestment initiatives
implemented by NICE in the United Kingdom, and one evaluated
a US Preventive Services Task Force (USPSTF) grade D recom-
mendation (a recommendation against the service).
Choosing Wisely Campaign
In 2012, the American Board of Internal Medicine launched the
Choosing Wisely Campaign to help providers and patients engage
in conversations about avoiding the use of unnecessary treat-
ments, tests, and procedures [10]. The campaign was developed
in conjunction with 20 medical specialty professional organiza-
tions, each of which generated a list of ve interventions that
they considered to be overused in their eld.
NICE initiatives
NICE is an executive nondepartmental public body of the Depart-
ment of Health in the United Kingdom. Among its roles is to
provide recommendations on care that should or should not be
used in the National Health Service [12]. NICEs recommendations
regarding low-value care can originate in various sources includ-
ing clinical guidelines and in HTAs that are informed through
review of clinical and cost-effectiveness evidence and input from
stakeholders [13,14].
US Preventive Services Task Force
The USPSTF is an independent panel of experts in primary care
and prevention [15]. The task force reviews evidence of an
interventions effectiveness to develop evidence-based recom-
mendations for clinical preventive services intended for use by
health care professionals and patients. It uses a grading scheme
to summarize its recommendations [16]. The task force uses a
grade D to recommend against a service in which there is
moderate or high certainty that the service has no net benet.
Success of Disinvestment Initiatives
Seven (39%) studies reported that the disinvestment initiative
was successful, eight (44%) reported that the initiative was
unsuccessful, and three (17%) reported that the success of the
initiative was mixed. In all, the 18 studies evaluated 37 low-value
services, for 14 (38%) of which the disinvestment initiative led to
a decline in use.
 VALUE IN HEALTH 20 (2017) 909918 911

Table 1 Identied disinvestment initiatives.

Country Disinvestment initiative Year Included interventions

Australia Life Saving Drugs Program Review 2014 Drugs

Selecting the most appropriate and relevant tests and 2013 Multiple intervention types
treatments care
New technology program reallocation 2012 Multiple intervention types
Pharmaceutical Benets Scheme Postmarket Surveillance 2011 Drugs
Medicare Benets Schedule Review Taskforce (previously 2009 Devices
named Quality Framework)
Southern Health Sustainability in Healthcare by Allocating 2007 Multiple intervention types
Resources Effectively
National Medicines Policy MedicineWise 1998 Multiple intervention types
Canada Ontario Reassessment Framework 2011 Multiple intervention types
Atlantic Common Drug Review 2010 Drugs
Denmark Danish Centre for Health Technology Assessment Pilot 2004 Multiple intervention types
France Haute Autorit de Sant Comprhensive Drug Review 20002004 Drugs
International Choosing Wisely (Australia, Canada, England, Japan, the 2012 Multiple intervention types
Netherlands, United States)
New Zealand National Health Committee Reprioritization Program 2011 Multiple intervention types
Auckland District Health Board Clinical Practice Committee 2005 Multiple intervention types
Technology Review
Pharmaceutical Management Agency Yearly Review 1993 Drugs
Scotland Scotland Disinvestment Project 2004 Multiple intervention types
Spain Guideline for Not Funding Health Technologies Methodology 2010 Multiple intervention types
PriTec Prioritization Tool 2008 Multiple intervention types
United Kingdom Wales Prudent HealthCare 2013 Multiple intervention types
National Institute for Health and Care Excellence Do Not Do 2007 Multiple intervention types
List Policy
National Institute for Health and Care Excellence Delist Pilot 2006 Multiple intervention types
National Institute for Health and Care Excellence 2006 Multiple intervention types
recommendation reminders
The British Medical Journals Too Much Medicine Campaign 2002 Multiple intervention types
National Institute for Clinical Excellence clinical guidelines 1999 Multiple intervention types
United States Veterans Health Administration Comprehensive Review 1996 Drugs
US Preventive Services Task Force grade D recommendations 1984 Multiple intervention types

Five of the nine empirical studies that evaluated Choosing
Wisely recommendations reported the disinvestment initiative to
be successful, two reported the initiative to be unsuccessful, and
two reported the success of the initiative to be mixed. Nineteen
interventions were examined in the nine empirical studies that
evaluated Choosing Wisely recommendations, for nine (47%) of
which the disinvestment initiative led to a decline in use.
Two of the eight empirical studies that evaluated disinvest-
ment initiatives promulgated by NICE reported the disinvestment
initiative to be successful, ve reported the initiative to be
unsuccessful, and one reported the success of the initiative to
be mixed. Seventeen interventions were examined in the nine
empirical studies that evaluated initiatives promulgated by NICE,
for ve (47%) of which the disinvestment initiative led to a
decline in use.
The study that evaluated the USPSTF grade D recommenda-
tion reported that it did not reduce the use of the low-value
service (routine mammography screening in women younger
than 50 years).
reporting the success of the initiatives to be mixed had a median
follow-up of 11 months (range 615 months).
Four of the 18 reviewed studies included a phase-in period, of
which 3 (75%) reported that the initiative was successful. Of the
14 studies without a phase-in period, 4 (29%) reported the
initiative to be successful, 7 (50%) reported the initiative to be
unsuccessful, and 3 (21%) reported mixed success.
Two of the 18 reviewed studies included control groups, of
which 1 reported the initiative to be successful. Of the 16 studies
without a control group, 6 reported the initiative to be successful,
8 reported the initiative to be unsuccessful, and 2 reported the
success of the initiative to be mixed.
Seven of the 18 reviewed studies included a formal attempt to
promote the disinvestment initiative, of which 6 reported the
initiative to be successful and 1 reported that the initiative was a
mixed success. One of the 11 studies without a formal attempt to
promote the disinvestment initiative reported the initiative to be
successful, 8 reported the initiative to be unsuccessful, and 2
reported mixed success.

Study Characteristics
Seventeen studies reported the follow-up period postimplemen-
tation of the disinvestment initiative: the median period was 12
months (range 172 months). Studies reporting initiatives to be
successful had a median follow-up of 6 months (range 112
months), studies reporting initiatives to be unsuccessful had a
median follow-up of 24 months (range 372 months), and studies
Discussion
We identied widespread experimentation of disinvestment ini-
tiatives, with 26 initiatives implemented in 11 countries. More than
one-third of these initiatives were established in the last 5 years,
suggesting a growing recognition of the need to reduce the use of
unnecessary, ineffective, inefcient, or possibly harmful care [4].
 912
Table 2 Empirical studies evaluating disinvestment initiatives.
Author Country Low-value Control Data source(s) Sample size Time period Phase-in Promotion of the Details of the Success
intervention(s) period disinvestment outcome of the
(date of guidance/ initiative disinvestment
recommendation) initiative

Choosing Wisely
Corson United Common None Electronic 13,583 patients Pre-initiative: 1 mo Multifaceted QI Total no. of common Yes
et al., 2015 States laboratory medical Baseline 7,824 10 mo initiative, including: laboratory tests
[24] tests: CBC, basic record Intervention 5,759 Wash-in: 1 mo  Monthly email ordered per patient-
metabolic panels, Postinitiative: 7 mo  Monthly reporting day, 2.06 vs. 1.76
nutrition panel, of provider behavior (P o 0.01)
comprehensive CBC labs (P o 0.01)
metabolic panel Basic metabolic
(February 2013) panel (P o 0.01)
Nutritional panel
(P 0.01)
Comprehensive
metabolic panel
(P o 0.01)
Baseline: 9 y* Extended fraction

VALUE IN HEALTH 20 (2017) 909918

Gebhardt United Radiation therapy None Medical records 12,678 None Online clinical Yes
et al., 2015 States with 410 fraction treatment Follow-up: 1 y pathway system regimens (410)
[25] regimens for courses (decision support overall from
pain relief from tool) 18.6% to 9.7%
bone metastases (P o 0.0001)
(September 2013)
Kost et al., United 1. Imaging for low-back None Chart review 1,089 patient Pre-initiative: None Either a 1-h seminar 1. No change in Mixed
2015 [26] States pain within the charts 6 mo or an online webinar low-back pain
rst 6 wk Postinitiative: reviewing the imaging (P 0.99)
2. Antibiotic prescribing 6 mo recommendations 2. in prescribing
for mild-moderate antibiotics for
sinusitis sinusitis (P o 0.01)
3. Pap tests on women 3. No change in pap
o21 y old or have tests in women
had hysterectomy o21 y (P 0.55)
for noncancerous 4. No change in ECGs
disease for low-risk patients
4. Annual (P 0.96)
electrocardiogram
screenings for 5. in DXA screening
low-risk patients (P 0.049)
5. DXA scanning for
osteoporosis
in women o65 y or
men o70 y
with no risk factors
(April 2012)
Lasser United DXA screening for None Electronic 42,320 patients Pre-initiative: None None Average provider No
et al., 2016 States osteoporosis in medical Baseline 20,256 9 mo DXA order rate per
[27] women o65 y records Intervention Postinitiative: patient (2.6% pre to
with no risk 22,064 9 mo 2.0% post) (P 0.75)
factors (April 2012)
Lesuis Netherlands Antinuclear None Laboratory and 11,939 tests pre- Pre-initiative: None 1-h education and No. of antinuclear Yes
et al., 2016 antibody testing administrative initiative 6,514 12 mo feedback group antibody testing
[28] for new patients hospital Postinitiative Postinitiative: session. 1-h booster Center 1 and 2
with rheumatology databases 5,425 12 mo session held 6 mo (0.37 to 0.11)
(February 2013) after intervention (P o 0.01)
Center 3 (0.45 to 0.30)
(P o 0.01)
continued on next page
Leverenz United Common laboratory None Hospital laboratory Not reported Not reported None  Distribution of an % patients receiving Yes
et al., 2015 States tests: daily CBC or database evidence-based ier CBC or basic
[29] basic metabolic  Presentation of data metabolic panel
panels (February at house staff each day (statistical
2013) conferences signicance not
 Weekly email reported)
reminder/feedback
 Monthly report on
rate of laboratory
tests
Rosenberg United 1. Imaging for None Claims data 25 million Pre-initiative: None None 1. use of imaging for Mixed
et al., 2015 States uncomplicated members 27 mo headache (P o 0.001)
[30] headaches Postinitiative: 2. cardiac imaging
2. Imaging for cardiac 15 mo (P o 0.001)
conditions for patients 3. No change in
without history of imaging for low-back
cardiac conditions pain (P 0.71)
3. Imaging for low-back
4. No change in use of
pain without red-ag
preoperative chest
conditions
x-rays (P 0.70)
4. Preoperative chest x-
rays with unremarkable 5. in HPV testing in
history and physical younger women

VALUE IN HEALTH 20 (2017) 909918

examination results (P o 0.001)
5. HPV testing for women 6. No change in use of
o30 y antibiotics for sinusitis
6. Antibiotic use for acute (P 0.16)
sinusitis 7. in use of NSAIDs
7. NSAID for hypertension, (P o 0.001)
heart failure, or chronic
kidney disease (April
2012)
Schondelmeyer United CPOx for hospitalized None Chart review 617 patients Baseline: 3 mo None  Education in median time per Yes
et al., 2015 States children with (intervention Follow-up: 3 mo  Checklist used week on CPOx
[39] respiratory illness unit 455; during nurse (baseline 10.7 h to
(February 2013) control hand-off 3.1 h) (statistical
unit 162)  Discontinuation signicance
criteria not reported)
incorporated
into order set
Simos Canada Imaging for None Medical records 200 patients Pre-initiative: 1 y 4 mo None No change in % of patients No
et al., 2014 metastatic Wash-in period: who received imaging
[31] disease in 4 mo for distant metastases
asymptomatic Postinitiative: not keeping with guide-
patients with 3 mo lines (77% pre to 75%
early-stage breast post) (statistical signi-
cancer (April 2012) cance not reported)

NICE initiatives
Chamberlain United 1. Endometrial None National hospital 1. 137,028 Preguidance: 6 y None None 1. endometrial No
et al., 2013 Kingdom biopsies in the episode statistics admissions Postguidance: biopsies
[32] luteal phase to for female 6y (APC 6.0% [95% CI 3.6%
investigate female infertility to 8.4%] up to 2003; APC
infertility (February 2. 9,399 1.5% [95% CI 4.3%
2004) consultant to 7.7%] up to 2007;
2. Varicocele operations episodes for and APC 12.8% [95% CI
to treat infertility in male infertility 1.0% to 26.0%] up to
men (February 2004) 2010)
2. No change in
varicocele procedures
(APC 0.5% [95% CI

913
2.3% to 1.3%])
continued on next page
 914
Table 2 continued

Author Country Low-value Control Data source(s) Sample size Time period Phase-in Promotion of the Details of the Success
intervention(s) period disinvestment outcome of the
(date of guidance/ initiative disinvestment
recommendation) initiative

3. Cesarean sections in 3. 5,546 births 3. planned cesarean

women with sections in women
hepatitis (April 2004) with hepatitis,
compared with women
without (APC 4.0% [95%
CI 2.7% to 5.3%]
up to 2001; APC 0.6%
[95% CI 2.8% to 1.8%]
up to 2004; and APC
1.3% [95% CI 0.8%
to 1.8%] up to 2010)
Coronini- United 1. Spinal surgery for Coronary National hospital 5,248,808 Pre-initiative: 8 y None None 1. No change in spinal Mixed
Cronberg Kingdom lower back pain revascu- episode statistics procedures Postinitiative: 1 y surgery
et al., 2014 2. Myringotomy to larization; 2. myringotomies

VALUE IN HEALTH 20 (2017) 909918

[33] relieve eardrum cholecys- (by 11.4%) (P o 0.05)
pressure tectomy 3. in hernia
3. Inguinal hernia repair repair (P o 0.05)
4. Cataract removal 4. in cataract
5. Primary hip removal (by 4.8%)
replacement (P o 0.05)
5. No change in hip
6. Hysterectomy
replacement
for heavy
6. in hysterectomies
menstrual bleeding
(by 10.7%) (P o 0.05)
Dietrich, United 21 drugs subject None National 10 million Pre-initiative: 4 y None None from 10 million items No
2009 [34] Kingdom to negative prescribing low-value Postinitiative: 1 y dispensed in 2000 to
and restricting data items dispensed 22 million in 2004
NICE technology in 2000; 22 million (statistical signicance
appraisal (20012004) dispensed in 2004 not reported)
Hassan United SXR and observation/ CT scan Patient 1,130 patients Baseline: 1 mo 1 mo North Tyneside Hope Hospital Yes
et al., 2005 Kingdom admission as the rst records with head Wash-in period: General Hospital (signicance
[35] investigation in head injury 1 mo intensive ED not reported):
injury management Follow-up: 1 mo education and  in SXR use
(June 2003) promotion exercise (37% to 4%)
Hope Hospital  in CT % scan
disseminated (higher value)
ED protocol  in admissions
(9% to 4%)
North Tyneside
General Hospital:
 in CT scans
(1.4% to 9%)
 in SXR use (19% to
0.57%)
 in admissions
(7% to 5%) (statistical
signicance not
reported)
Okagbue United ECT for nonlife- None Patient records 1,071 patients Pre-initiative: 10 y None None No early effect No
et al. [19] Kingdom threatening Postinitiative: of the guidance
illness (May 2003) 2y was found in the
rate of ECT usage
continued on next page
Ryan United 1. Extraction of None CHKS hospital 1. 14,545 Pre-initiative: None None No. of impacted No
et al., 2004 Kingdom pathology-free episode database impacted 3y teeth extractions
[36] impacted third wisdom teeth Postinitiative: (effect of NICE
molars (March extractions 2y guidance not
2000) 2. 12,604 signicant,
2. Noncemented hip P 0.8287)
hip procedures
prostheses in noncemented
(April 2000) total hip
replacements
(5.8%, P 0.9124)
Sheldon United 1. Extraction of None Case notes, 6,308 usable Pre-initiative: None None 1. in the number of No
et al. [40] Kingdom pathology-free survey, patient 5y extractions, although
impacted third interviews audit forms Postinitiative: no evidence that
molars (April 2000) 2y decline was due to
2. Laparoscopic NICE guidance ( 8.9
surgery for treatment extractions per month,
of colorectal cancer 95% CI 57.5 to 39.6)
(December 2000) 2. No change in
3. Laparoscopic surgery percentage of cases of
for primary inguinal colorectal cancer
hernia (January 2001) treated with
laparoscopic surgery

VALUE IN HEALTH 20 (2017) 909918

(0.1% from 1998 to
2001)
3. No change in
percentage of hernia
prepares treated
laparoscopically
(95% CI 5.48 to 6.08)
Thornhill United Antibiotic prophylaxis None National 13,019 Pre-initiative: 13 mo None in mean prescriptions Yes
et al., 2011 Kingdom (prevention of prescribing prescriptions 12 mo (78.6%)
[37] infective endocarditis) data Postinitiative: in prescribing of
for dental procedures 12 mo antibiotic prophylaxis
(March 2008) by dentists (79.9%,
P o 0.001)

USPSTF
Dehkordy United Routine None Population-level Not reported Comparison 12 mo None None rate of screening No
et al., 2015 States mammography data from the (population-level before USPSTF initiation at age
[38] screening in Behavioral Risk data used) recommendation 40 y (P 0.012)
women o50 y Factor
(November 2009) Surveillance
System (self-
report data)

APC, annual percentage change; CBC, complete blood cell count; CI, condence interval; CT, computed tomography; CPOx, continuous pulse oximetry; DXA, dual-energy x-ray absorptiometry;
ECG, electrocardiogram; ECT, electroconvulsive therapy; ED, emergency department; HPV, human papilloma virus; NICE, National Institute of Health and Care Excellence; NSAID, nonsteroidal
anti-inammatory drug; QI, quality improvement; SXR, skull x-ray; USPSTF, US Preventive Services Task Force.
*
There were three time periods in this study: 5 y of baseline data were collected for an initial clinical pathway; 4 y of additional baseline data were collected for a modied clinical pathway that
included peer review; and postinitiative follow-up data were collected 1 y after the pathway was modied to reect the Choosing Wisely guideline.

Time period of study was reported in yearly quarters. Pre-initiative data were collected for nine quarters (assumed to be 27 mo) and postinitiative data were collected for ve quarters (assumed
to be 15 mo).

Study compared physicians who had received an intervention to encourage adherence to the guideline to a control group of physicians (those who did not receive the intervention).

P values were reported for other results including for the change in the number of at least one imaging test for distant metastatic disease (% of total) from 83% prerecommendation and 86%
postrecommendation (P 0.70).

915
916 VALUE IN HEALTH 20 (2017) 909918
Nevertheless, despite these efforts, we found few empirical anal-
yses of disinvestment initiatives. Reasons for this are unclear.
Possibly, insufcient time has passed for researchers to have
performed and published empirical analyses. It may also be that
the agencies that implemented the initiatives have not published
the evaluations they have performed.
The empirical studies we reviewed reported that the studied
disinvestment initiatives had varied success in reducing the use
of the low-value service(s). Fewer than half of the studies (39%)
reported that the disinvestment initiative was successful and the
remaining reported the initiatives to be either unsuccessful (44%)
or of mixed success (17%). This nding was consistent when we
considered low-value services individually; of the 37 low-value
services studied, the disinvestment initiative reduced the use of
only 14 (38%).
Although our sample size was insufcient to draw rm
conclusions on the necessary components of a successful dis-
investment initiative or the types of intervention most amenable
to disinvestment, our study provides an insight into why some
disinvestment initiatives were successful while others were not.
Primarily, our ndings suggest that the incorporation of
accompanying strategies to promote the disinvestment initia-
tives among participating practitioners is key to their success. Six
of the seven studies that promoted the initiative were reported to
be successful. In contrast, only 1 of the 11 studies that did not
include promotion was reported to be unsuccessful. The
approaches to promote initiatives and encourage adherence
included clinician education, email reminders, feedback on
recent prescribing behavior, and online clinical decision support
tools. Leverenz et al. report a study in which multiple promo-
tional activities were used to support a disinvestment initiative
that successfully reduced the use of common laboratory tests in a
university hospital. The promotional activities included the dis-
tribution of a yer before the start of the initiative, a presentation
to participating physicians, a weekly email providing feedback on
individual-level prescribing behavior, and a monthly email iden-
tifying the clinical team that ordered the fewest common
laboratory tests. In a second study that included multiple promo-
tional activities, Schondelmeyer et al. [39] reported a disinvest-
ment initiative that successfully reduced the use of continuous
pulse oximetry for hospitalized children with respiratory illness.
The promotional activities included staff education at initiative
outset, communication between clinical team members through
a nursing hand-off tool, feedback on the use of continuous pulse
oximetry at biweekly and monthly meetings, and weekly email
updates to compare the performance of participating clinical
teams. Although our study does not indicate a specic best
practice, it does suggest that promotional efforts that comple-
ment disinvestment initiatives are essential.
Four low-value services were evaluated by two separate
empirical studies and provide further indication of the inuence
of promotion. For two of these low-value servicesbone densi-
tometry screening for osteoporosis in women younger than 65
years and antibiotics for treatment of acute sinusitisone of the
empirical studies included promotion of the disinvestment ini-
tiative to participating clinicians, whereas the other did not; in
both instances, the study that included promotion was successful
and the one without promotion was not. Both studies that
evaluated initiatives to decrease the number of common labo-
ratory tests ordered included some promotion of the disinvest-
ment initiative and reported the initiative to be successful. Both
studies that evaluated initiatives to decrease extractions of
pathology-free molars did not include promotion of the disin-
vestment initiative and reported the initiative to be unsuccessful.
With centralized approaches for assessing the effectiveness
and cost-effectiveness of interventions and the ability to more
readily implement policies across the health care system, it is
asserted that single-payer health care systems such as the
National Health Service in the United Kingdom may be best
positioned to implement disinvestment initiatives [17]. Our nd-
ings do not support this assertion. Only two of the eight
disinvestment studies that evaluated NICE initiatives in the
United Kingdom were reported to be successful. In contrast, four
of the eight studies that evaluated disinvestment initiatives in
the United States were reported to be successful, despite the
countrys fragmented health care system [18]. Nevertheless, the
small number of studies in our review prevents rm conclusions
to be drawn regarding whether single-payer health care systems
are best positioned to successfully implement disinvestment
initiatives.
Study Limitations
Our study has a number of limitations. First, because PubMed
was the only database we searched, this study should be
considered a sampling of the literature rather than a systematic
review. Furthermore, because we relied on studies in English
when searching the PubMed database and the gray literature, we
may have introduced language bias into the study and thus may
not have comprehensively accounted for all examples of dis-
investment initiatives. Second, the reviewed empirical studies
varied with respect to study design, which limited our ability to
compare them. For instance, we found that four studies included
a study phase-in period, three of which were successful. In
comparison, 4 of the 14 studies without a phase-in period were
successful. Third, only 2 of the 18 studies included a control
group. The lack of a control group raises concerns that an
observed change in behavior may be due to environmental
inuences or factors other than the disinvestment initiative.
Fourth, the included studies were performed over different
durations. Indeed, the authors of one study suggested that a
longer study duration was required to fully understand the
initiatives long-term effect [19]. Fifth, the reviewed studies relied
on different data sources, including electronic medical records,
physician surveys, chart reviews, and claims data. Sixth, we
excluded disinvestment strategies implemented locally. Seventh,
although we examined whether the disinvestment initiative
resulted in a reduction in the use of the low-value intervention,
we did not judge whether the magnitude of the reduction was
meaningful. Last, because most of the empirical studies eval-
uated either NICE or Choosing Wisely Campaign initiatives, the
generalizability of our ndings regarding the success of disinvest-
ment initiatives may be limited.
Challenges Associated with Disinvestment Strategies
Our ndings are consistent with a number of well-documented
reported challenges of disinvestment initiatives [8,20]. For
instance, researchers suggest that cultural and attitudinal bar-
riers are important hurdles, and that gaining physician accept-
ance is integral to an initiatives success [7,2123]. Two studies
supported this assertion by attributing the success of the initia-
tive in part to a clinical setting with a receptive culture, and one
study attributed success in part to gaining physician acceptance
[24,39,40]. Researchers also point to the importance of adequate
resources to support disinvestment initiatives [1,41]. Our ndings
strongly support this assertion.
We were unable to assess other reported challenges including
consumer resistance and problems identifying low-value services
[4143]. Disinvestment initiatives may be less likely to be suc-
cessful for services that are deeply entrenched in care, although
we were unable to judge such matters from the information
available.
 VALUE IN HEALTH 20 (2017) 909918
Looking Forward
How best to allocate scarce health care resources in the face of
constrained budgets is a global challenge. Successfully disinvest-
ing in low-value care is necessary to generate the nances
required to pay for emerging innovations. It is estimated that
as much as one-third of health care spending is wasteful, and
disinvesting in low-value care offers great potential to generate
much needed health care dollars [44].
Health care systems implementing disinvestment initiatives
will look to others experiences for lessons. Our study is the rst
to review empirical evaluations of disinvestment initiatives and
provides insights into why some succeed. We found that simply
promulgating information regarding which interventions are
low-value services may be insufcient to lead to a meaningful
reduction in their use and that additional approaches to promote
the initiative among participating clinicians should be an integral
part of any policy.
More research is necessary to better understand the compo-
nents of a successful disinvestment initiative. For instance,
researchers should examine how long it takes for disinvestment
initiatives to have an effect and to explore reasons for our nding
that the studies in our sample with a shorter median follow-up
were more likely to report that the disinvestment initiative was
successful. Researchers should also examine disinvestment strat-
egies implemented locally, for example, at individual hospitals,
because these may offer lessons for successful initiatives at the
national level. Research that scrutinizes disinvestment strategies
implemented locally would have particular relevance to the US
health care system, in which, because of its fragmented struc-
ture, national-level disinvestment initiatives are challenging to
implement [18]. We were unable to draw conclusions as to what
types of interventions, that is, drugs, medical procedures, and so
forth, are most amenable to disinvestment initiatives. Knowledge
of the interventions for which disinvestment would yield the
greatest savings would be valuable. Although our ndings sug-
gest the need for promotion of the disinvestment initiative to
encourage adherence to it, the best approach to achieving this is
unclear and in need of study.
Along with the judicious addition of new health care tech-
nologies to a health care system, disinvestment of existing low-
value services should be part of a coordinated set of approaches
to help manage scarce resources. Disinvestment initiatives
should continue to support other approaches to help guide the
use of technology, including clinical guidelines and clinical
decision support tools. Furthermore, demand-side approaches
(e.g., patient cost sharing and value-based insurance design),
supply-side approaches (e.g., pay for performance and
performance-based arrangements), and policies that incorporate
behavioral economics (e.g., physician peer comparison) should all
be considered as tools for health care systems to manage their
scarce resources [7,17,4549].
Conclusions
Disinvestment initiatives are increasingly being implemented in
health care systems around the globe to reduce the use of low-
value care. In this sampling of the literature we found that few
disinvestment initiatives have been empirically studied, and for
those that have, success has been mixed. Our ndings suggest
that promotion of the disinvestment initiative among participat-
ing practitioners is key to the success of the disinvestment
initiative.
Source of nancial support: Financial support for this study
was provided by Pzer Inc. Publication was not contingent on
917
Pzer Inc.s approval and the remarks expressed are those of the
authors and do not necessarily reect the views of others.
R EF E R EN C ES
[1] Henshall C, Schuller T, Mardhani-Bayne L. Using health technology
assessment to support optimal use of technologies in current practice:
the challenge of disinvestment. Int J Technol Assess Health Care
2012;28:20310.
[2] Bryan S, Mitton C, Donaldson C. Breaking the addiction to technology
adoption. Health Econ 2014;23:37983.
[3] Scotland G, Bryan S. Why do health economists promote technology
adoption rather than the search for efciency? A proposal for a change
in our approach to economic evaluation in health care. Med Decis
Making 2017;37:13947.
[4] Parkinson B, Sermet C, Clement F, et al. Disinvestment and value-based
purchasing strategies for pharmaceuticals: an international review.
Pharmacoeconomics 2015;33:90524.
[5] Elshaug AG, Hiller JE, Tunis SR, Moss JR. Challenges in Australian policy
processes for disinvestment from existing, ineffective health care
practices. Aust New Zealand Health Policy 2007;4:23.
[6] Williams I, McIver S, Moore D, Bryan S. The use of economic
evaluations in NHS decision-making: a review and empirical
investigation. Health Technol Assess 2008;12: iii, ix -iii, 175.
[7] Gonzales R, Cattamanchi A. Changing clinician behavior when less is
more. JAMA Intern Med 2015;175:19212.
[8] Cooper C, Starkey K. Disinvestment in health care. BMJ 2010;340:c1413.
[9] Watkins K, Wood H, Schneider CR, Clifford R. Effectiveness of
implementation strategies for clinical guidelines to community
pharmacy: a systematic review. Implement Sci 2015;10:151.
[10] Choosing Wisely. Choosing Wisely Campaign. Available from: http://
www.choosingwisely.org/. [Accessed January 13, 2017].
[11] National Institute of Health and Care Excellence. Do not do list. Available
from: https://www.nice.org.uk/savingsandproductivity/collection?
page=1&pagesize=2000&type=do%20not%20do. [Accessed January 13, 2017].
[12] National Institute for Health and Care Excellence. What we do
Available from: https://www.nice.org.uk/about/what-we-do. [Accessed
January 13, 2017].
[13] National Institute for Health and Care Excellence. Developing NICE
guidelines: the manual. Available from: https://www.nice.org.uk/
process/pmg20/chapter/introduction-and-overview#key-principles-
for-developing-guidelines. [Accessed January 13, 2017].
[14] National Institute of Health and Care Excellence. Guide to the
processes of technology appraisal. Available from: https://www.nice.
org.uk/process/pmg19/chapter/acknowledgements. [Accessed January
13, 2017].
[15] US Preventive Services Task Force. About the USPSTF. Available from:
https://www.uspreventiveservicestaskforce.org/Page/Name/
about-the-uspstf. [Accessed January 13, 2017].
[16] US Preventive Services Task Force. USPSTFgrade denitions.
Available from: https://www.uspreventiveservicestaskforce.org/Page/
Name/grade-denitions. [Accessed January 13, 2017].
[17] Garner S, Littlejohns P. Disinvestment from low value clinical
interventions: NICEly done? BMJ 2011;343:d4519.
[18] Cebul RD, Rebitzer JB, Taylor LJ, Votruba ME. Organizational
fragmentation and care quality in the U.S. healthcare system. J Econ
Perspect 2008;22:93113.
[19] Okagbue N, McIntosh A, Gardner M, Scott AI. The rate of usage of
electroconvulsive therapy in the city of Edinburgh, 19932005. J ECT
2008;24:22931.
[20] Health Policy Advisory Committee on Technology. Disinvestment in
Australia and New Zealand. Available from: https://www.health.qld.
gov.au/__data/assets/pdf_le/0024/439611/disinvestment-report.pdf.
[Accessed January 13, 2017].
[21] Morden NE, Colla CH, Sequist TD, Rosenthal MB. Choosing wiselythe
politics and economics of labeling low-value services. N Engl J Med
2014;370:58992.
[22] Wolfson D, Santa J, Slass L. Engaging physicians and consumers in
conversations about treatment overuse and waste: a short history of
the Choosing Wisely Campaign. Acad Med 2014;89:9905.
[23] Rooshenas L, Owen-Smith A, Hollingworth W, et al. I wont call it
rationing : an ethnographic study of healthcare disinvestment in
theory and practice. Soc Sci Med 2015;128:27381.
[24] Corson AH, Fan VS, White T, et al. A multifaceted hospitalist quality
improvement intervention: decreased frequency of common labs. J
Hosp Med 2015;10:3905.
[25] Gebhardt BJ, Rajagopalan MS, Gill BS, et al. Impact of dynamic changes
to a bone metastases pathway in a large, integrated, National Cancer
918 VALUE IN HEALTH 20 (2017) 909918
Institute-designated comprehensive cancer center network. Pract
Radiat Oncol. 2015;5:398405.
[26] Kost A, Genao I, Lee JW, Smith SR. Clinical decisions made in primary
care clinics before and after Choosing Wisely. J Am Board Fam Med
2015;28:4714.
[27] Lasser EC, Pfoh ER, Chang HY, et al. Has Choosing Wisely(R) affected
rates of dual-energy X-ray absorptiometry use? Osteoporos Int.
2016;27:23116.
[28] Lesuis N, Hulscher ME, Piek E, et al. Choosing Wisely in daily practice:
An intervention study on Antinuclear Antibody testing by
rheumatologists. Arthritis Care Res (Hoboken) 2016;68:5629.
[29] Leverenz D, Iams W, Heck J, Brady D. Who is going to make the wise
choice? J Hosp Med 2015;10:5446.
[30] Rosenberg A, Agiro A, Gottlieb M, et al. Early trends among seven
recommendations from the Choosing Wisely campaign. JAMA Intern
Med 2015;175:191320.
[31] Simos D, Hutton B, Clemons M. Are physicians Choosing Wisely when
imaging for distant metastases in women with operable breast cancer?
J Oncol Pract 2015;11:628.
[32] Chamberlain CA, Martin RM, Busby J, et al. Trends in procedures for
infertility and caesarean sections: was NICE disinvestment guidance
implemented? NICE recommendation reminders. BMC Public Health
2013;13:112.
[33] Coronini-Cronberg S, Bixby H, Laverty AA, et al. English National Health
Services savings plan may have helped reduce the use of three low-
value procedures. Health Aff (Millwood) 2015;34:3819.
[34] Dietrich ES. Effects of the National Institute for Health and Clinical
Excellences technology appraisals on prescribing and net ingredient
costs of drugs in the National Health Service in England. Int J Technol
Assess Health Care 2009;25:26271.
[35] Hassan Z, Smith M, Littlewood S, et al. Head injuries: a study evaluating
the impact of the NICE head injury guidelines. Emerg Med J
2005;22:8459.
[36] Ryan J, Piercy J, James P. Assessment of NICE guidance on two surgical
procedures. Lancet. 2004;363:15256.
[37] Thornhill MH, Dayer MJ, Forde JM, et al. Impact of the NICE guideline
recommending cessation of antibiotic prophylaxis for prevention of
infective endocarditis: before and after study. BMJ. 2011;342:d2392.
[38] Dehkordy SF, Hall KS, Roach AL, et al. Trends in breast cancer
screening: Impact of U.S. Preventive Services Task Force
recommendations. Am J Prev Med 2015;49:41922.
[39] Schondelmeyer AC, Simmons JM, Statile AM, et al. Using quality
improvement to reduce continuous pulse oximetry use in children with
wheezing. Pediatrics 2015;135:e104451.
[40] Sheldon TA, Cullum N, Dawson D, et al. Whats the evidence that NICE
guidance has been implemented? Results from a national evaluation
using time series analysis, audit of patients notes, and interviews. BMJ
2004;329:999.
[41] Howard DH, Gross CP. Producing evidence to reduce low-value care.
JAMA Intern Med 2015;175:18934.
[42] Santa JS. Communicating information about what not to do to
consumers. BMC Med Inform Decis Mak 2013;13(Suppl. 3):S2.
[43] Elshaug AG, Moss JR, Littlejohns P, et al. Identifying existing health care
services that do not provide value for money. Med J Aust
2009;190:26973.
[44] Berwick DM, Hackbarth AD. Eliminating waste in US health care. JAMA
2012;307:15136.
[45] Colla CH. Swimming against the currentwhat might work to reduce
low-value care? N Engl J Med 2014;371:12803.
[46] Neumann PJ, Chambers JD, Simon F, Meckley LM. Risk-sharing
arrangements that link payment for drugs to health outcomes are
proving hard to implement. Health Aff (Millwood) 2011;30:232937.
[47] Fendrick AM, Smith DG, Chernew ME, Shah SN. A benet-based copay
for prescription drugs: patient contribution based on total benets, not
drug acquisition cost. Am J Manag Care 2001;7:8617.
[48] Blumenthal-Barby JS. Choosing wisely to reduce low-value care: a
conceptual and ethical analysis. J Med Philos 2013;38:55980.
[49] Admon AJ, Cooke CR. Will Choosing Wiselys improve quality and
lower costs of care for patients with critical illness? Ann Am Thorac
Soc 2014;11:8237.