Editorial
The Science and Art of Deduction: Complex Systematic Overviews
Like all other arts, the science of deduction and analysis is
one which can only be acquired by long and patient stu-
dy . . . let the inquirer begin by mastering the more elementary
problems.
—Sherlock Holmes, A Study in Scarlet
O rganizations that develop clinical practice guidelines
increasingly use systematic reviews to evaluate the
benefits and harms associated with health care interven-
tions (1). Well-executed systematic reviews efficiently syn-
thesize evidence to answer focused clinical questions and
can provide a strong foundation for clinical recommenda-
tions. Traditionally, systematic reviews build this founda-
tion with evidence from individual studies. However, An-
nals has witnessed an increase in the frequency of “complex
systematic overviews,” a term that we use to describe sys-
tematic reviews that use previous systematic reviews to con-
struct an evidence base. Although the incorporation of past
reviews into more recent syntheses can improve efficiency,
complex systematic overviews present challenges. We de-
scribe those challenges and provide suggestions for over-
coming them.
Why have complex systematic overviews become more
common? First, as the number of published systematic re-
views grows, so does the chance that those who set out to
systematically review a topic will find previous reviews on
the same or closely related topics. In addition, practice
guidelines often require answers to not only one but several
linked questions. A common example concerns screening
interventions for which direct clinical trials that randomly
assigned participants to a screening or control group are
lacking. In such cases, guideline developers typically seek
answers to a sequence of interrelated questions: How prev-
alent is the condition? What is its natural history? Is a good
screening test available? What are the benefits (and harms)
of early detection and treatment? When systematic over-
views address interlocking questions, it is valuable for re-
viewers to tie the questions together in an analytic frame-
work; this has become the standard practice for full
evidence reviews commissioned by the U.S. Preventive Ser-
vices Task Force (2). Another example is when those who
are developing treatment recommendations need to answer
questions about the effectiveness and safety of a variety of
treatment options. When previous systematic reviews that
address these component questions are available, the use of
these reviews rather than going back to the individual,
original studies presents an attractive strategy for efficiency.
Examples of complex systematic overviews are the
background reviews on nonpharmacologic and pharmaco-
logic treatment of low back pain that were published in the
2 October 2007 issue in support of guidelines from the
American College of Physicians and the American Pain
Society (3, 4). In the current issue, correspondence about
these reviews critiques the use of previous reviews as source
786 © 2008 American College of Physicians
Annals of Internal Medicine
material and points to the perils that one faces when in-
corporating existing systematic reviews into a complex
overview (5–7). An article by Whitlock and colleagues de-
scribes how some reviewers at the Agency for Healthcare
Quality and Research Evidence-based Practice Centers
have begun to manage the many complexities encountered
when considering whether to incorporate existing system-
atic reviews into a complex overview (8). As one reads this
series of material, it becomes clear that authors of system-
atic overviews must pay the same attention to the princi-
ples of comprehensiveness, transparency, and objectivity
when assembling systematic reviews into a complex system-
atic overview as they do when assembling individual stud-
ies into systematic reviews.
Whitlock and colleagues break their process into the
following steps: 1) locate existing reviews, 2) assess their
relevance, 3) assess their quality, 4) determine whether and
how to use reviews versus going back to the component
studies, and 5) report the methods and results of included
reviews (8). In large part, these steps echo those recom-
mended when searching for and assembling evidence from
individual studies. However, because the building blocks
become increasingly unwieldy as one moves from individ-
ual studies to systematic reviews, special care is necessary to
avoid drawing misleading conclusions from the evidence.
The sheer volume and complexity of available system-
atic reviews require reviewers to adopt carefully targeted
search strategies that maximize the yield of well-executed
systematic reviews on a given topic. Once reviews have
been located, reviewers must decide whether and how to
incorporate these reviews into their complex overview.
This requires consideration of congruence, overlap, and
clinical and statistical heterogeneity among studies in-
cluded in the selected systematic reviews and their impli-
cations for addressing the key questions to which the re-
viewers or guideline developers seek answers. A systematic
review’s inclusion/exclusion criteria must be congruent
with that of the complex overview with respect to study
type, methodological quality, populations represented, and
definitions of interventions and outcomes. Instruments for
assessing the quality of a systematic review, such as the
Oxman–Guyatt (9) or AMSTAR (10) instruments, can
save a great deal of time and effort in assessing the quality
of systematic review reporting. However, scores generated
by these and other rating tools, although useful for screen-
ing candidate systematic reviews, are relatively uninforma-
tive indicators of the quality and relevance of a systematic
review’s component studies with respect to the specific
question(s) under consideration in the complex systematic
overview (11). Furthermore, reliance on these scales makes
it too easy to present readers with simple quality judg-
ments, such as “evidence from a set of high-quality system-
atic reviews suggest that,” without providing them with the

necessary string of inferences upon which the final judg-
ment about the strength of evidence is based. Developers
and consumers of complex overviews should carefully heed
this advice from one of the masters of the art of deduction
in evaluating the veracity of such claims.
It is not really difficult to construct a series of simple
inferences, each dependent upon its predecessor and each sim-
ple in itself. If, after doing so, one simply knocks out all the
central inferences and presents one’s audience with the start-
ing-point and the conclusion, one may produce a startling,
though possibly a meretricious, effect.
—Sherlock Holmes, “The Dancing Men”
Sherlock Holmes cautions all of us about how easy it is
to create the appearance of quality or rigor in an argument
that in essence lacks substance. As systematic reviews be-
come mainstream vehicles for summarizing evidence in the
medical literature, the plethora of systematic reviews in-
cludes multiple reviews on the same topic. Multiple sys-
tematic reviews may represent a series of reviews on a given
question, in which the later reviews are updates of older
reviews that incorporate some or all of the studies included
in the earlier reviews. It is important to account for both
the deletion of older studies and the addition of newer
studies when deciding whether and how to incorporate
evidence from a series of systematic reviews into a complex
overview. As the volume of systematic reviews increases, we
also expect to see growing heterogeneity among reviews
published on the same topic. This heterogeneity is attrib-
utable to subtle—and not so subtle— differences in inclu-
sion/exclusion criteria, definitions of interventions and
outcomes, study types, and study quality. Critical scrutiny
of the component studies is unavoidable when these differ-
ences lead to conflicting or equivocal conclusions (12), and
sometimes a completely new review is warranted (11).
It is of the highest importance in the art of deduction to be
able to recognize, out of a number of facts, which are inciden-
tal and which are vital.
—Sherlock Holmes, “The Reigate Squires”
At Annals, we take this dictum to heart in our assess-
ment of the quality of any review article submitted for
publication. However, because the risks for mistakes in
deduction become higher as one moves away from primary
evidence, we are learning to be particularly alert to the
underlying principles of comprehensiveness, transparency,
and objectivity when considering complex systematic over-
views. First, we look for assurance that the reviewers have
comprehensively searched all pertinent data sources by us-
ing criteria that minimize the chances that they might have
missed a relevant previous review. Second, we consider
whether the reviewers adequately assess the need to update
the existing reviews (13). Third, when multiple overlap-
ping reviews are used, we require an adequate assessment of
the overlap in included studies and explication of hetero-
geneity among the subsets of studies between as well as
www.annals.org
Complex Systematic Overviews Editorial
Table. Guidance for Reviewers Preparing Complex
Systematic Overviews
Describe in detail the methods for searching and selecting previous
reviews.
Describe the criteria used to determine the need to update an existing
review.
Identify overlap in primary studies included in systematic reviews and
explain sources of differences in the primary studies covered by included
reviews.
Evaluate the quality of included systematic reviews by describing individual
methodological features. Avoid using numerical scales to summarize
review quality.
Provide a clear rationale for not “starting from scratch” and systematically
reviewing the component primary studies.
within the systematic reviews cited in a complex overview.
Fourth, just as we expect reviewers of individual studies to
evaluate the quality of those studies by using explicit crite-
ria, we expect those who review previous reviews to criti-
cally evaluate those evidence blocks. We frown upon reli-
ance on numerical scales to rate the quality of systematic
reviews that are included in complex overviews because
these scales say more about the quality of the reporting
than about the quality of the conduct of the appraisal and
synthesis tasks used to develop the review. Rather, review-
ers should focus on and communicate to readers the critical
features that affect their assessment of the strength and
credibility of the information used to represent the benefits
or harms associated with a health care intervention. A table
that displays the item-level evaluation of methodological
features for each of the included systematic reviews can be
an invaluable component of the report. Finally, reviewers
should provide a clear rationale for using previous reviews
instead of starting from scratch with individual component
studies and alert readers to the potential shortcomings of
this approach. We believe that increased attention to these
principles (Table) will enhance the science and art of trans-
lating complex systematic overviews into evidence-based
clinical recommendations.
Circumstantial evidence is a very tricky thing . . . It may
seem to point very straight to one thing, but if you shift your own
point of view a little, you may find it pointing in a equally
uncompromising manner to something entirely different.
—Sherlock Holmes, “The Boscombe Valley Mystery”
John E. Cornell, PhD
Associate Editor
Christine Laine, MD, MPH
Senior Deputy Editor
Potential Financial Conflicts of Interest: None disclosed.
Corresponding Author: Christine Laine, MD, MPH, American College
of Physicians, 190 N. Independence Mall West, Philadelphia, PA 19106.
Ann Intern Med. 2008;148:786-788.
20 May 2008 Annals of Internal Medicine Volume 148 • Number 10 787
Editorial Complex Systematic Overviews
References
1. Silagy CA, Stead LF, Lancaster T. Use of systematic reviews in clinical practice
guidelines: case study of smoking cessation. BMJ. 2001;323:833-6. [PMID:
11597966]
2. Guirguis-Blake J, Calonge N, Miller T, Siu A, Teutsch S, Whitlock E.
Current processes of the U.S. Preventive Services Task Force: refining evidence-
based recommendation development. Ann Intern Med. 2007;147:117-22.
[PMID: 17576998]
3. Chou R, Huffman, LH. Nonpharmacologic therapies for acute and chronic
low back pain: a review of the evidence for an American Pain Society/American
College of Physicians clinical practice guideline. Ann Intern Med. 2007;147:492-
504. [PMID: 17909210]
4. Chou R, Huffman LH. Medications for acute and chronic low back pain: a
review of the evidence for an American Pain Society/American College of Physi-
cians clinical practice guideline. Ann Intern Med. 2007;147:505-14. [PMID:
17909211]
5. Bjordal JM, Klovning A, Lopes-Martins RA, Roland PD, Joensen J, Slørdal
L. Overviews and systematic reviews on low back pain [Letter]. Ann Intern Med.
2008;148:779-80.
6. Ernst E. Overviews and systematic reviews on low back pain [Letter]. Ann
Intern Med. 2008;148:780-1.
788 20 May 2008 Annals of Internal Medicine Volume 148 • Number 10
7. Chou R, Shekelle P, Qaseem A, Owens DK. Overviews and systematic re-
views on low back pain [Letter]. Ann Intern Med. 2008;148:781-2.
8. Whitlock EP, Lin JS, Chou R, Shekelle P, Robinson KA. Using existing
systematic reviews in complex systematic reviews. Ann Intern Med. 2008;148:
776-82.
9. Oxman AD, Guyatt GH. Validation of an index of the quality of review
articles. J Clin Epidemiol. 1991;44:1271-8. [PMID: 1834807]
10. Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, et al.
Development of AMSTAR: a measurement tool to assess the methodological
quality of systematic reviews. BMC Med Res Methodol. 2007;7:10. [PMID:
17302989]
11. Cook DJ, Greengold NL, Ellrodt AG, Weingarten SR. The relation be-
tween systematic reviews and practice guidelines. Ann Intern Med. 1997;127:
210-6. [PMID: 9245227]
12. Biondi-Zoccai GG, Lotrionte M, Abbate A, Testa L, Remigi E, Burzotta F,
et al. Compliance with QUOROM and quality of reporting of overlapping
meta-analyses on the role of acetylcysteine in the prevention of contrast associated
nephropathy: case study. BMJ. 2006;332:202-9. [PMID: 16415336]
13. Moher D, Tsertsvadze A, Tricco AC, Eccles M, Grimshaw J, Sampson M,
et al. When and how to update systematic reviews. Cochrane Database Syst Rev.
2008:MR000023. [PMID: 18254126]
www.annals.org