Systematic Review Appraisal Sheet

SYSTEMATIC REVIEW: Are the results of the review valid? What question ( IC!" did the s#ste$ati% review address?
What is best? The main question being addressed should be clearly stated. The exposure, such as a therapy or diagnostic test, and the outcome(s) o interest will o ten be expressed in terms o a simple relationship. This paper# $es ( 'omment# %o ( &nclear ( Where do I find the information? The Title, Abstract or inal paragraph o the !ntroduction should clearly state the question. ! you still cannot ascertain what the ocused question is a ter reading these sections, search or another paper"

& ' Is it unli(el# that i$)ortant* relevant studies were $issed?

What is best? The starting point or comprehensive search or all relevant studies is the ma(or bibliographic databases (e.g., )edline, 'ochrane, *)+AS*, etc) but should also include a search o re erence lists rom relevant studies, and contact with experts, particularly to inquire about unpublished studies. The search should not be limited to *nglish language only. The search strategy should include both )*S, terms and text words. This paper# $es ( %o ( &nclear ( 'omment# Where do I find the information? The )ethods section should describe the search strategy, including the terms used, in some detail. The Results section will outline the number o titles and abstracts reviewed, the number o ull-text studies retrieved, and the number o studies excluded together with the reasons or exclusion. This in ormation may be presented in a igure or low chart.

A ' Were the %riteria used to sele%t arti%les for in%lusion a))ro)riate?
What is best? The inclusion or exclusion o studies in a systematic review should be clearly de ined a priori. The eligibility criteria used should speci y the patients, interventions or exposures and outcomes o interest. !n many cases the type o study design will also be a .ey component o the eligibility criteria. This paper# $es ( 'omment# A - /ere the included studies su iciently valid or the type o question as.ed0 What is best? Where do I find the information? The article should describe how the quality of each study The Methods section should describe the assessment of was assessed using predetermined quality criteria quality and the criteria used. The Results section should appropriate to the type of clinical question (e.g., pro ide information on the quality of the indi idual studies. randomization, blinding and completeness of follow-up) This paper# $es ( 'omment# T - /ere the results similar rom study to study0 What is best? !deally, the results of the different studies should be similar or homogeneous. !f heterogeneity exists the authors may estimate whether the differences are significant (chi-square test). "ossible reasons for the heterogeneity should be explored. This paper# $es ( 'omment# %o ( &nclear ( Where do I find the information? The Results section should state whether the results are heterogeneous and discuss possible reasons. The forest plot should show the results of the chi-square test for heterogeneity and if discuss reasons for heterogeneity, if present. %o ( &nclear ( %o ( &nclear ( Where do I find the information? The Methods section should describe in detail the inclusion and exclusion criteria. Normally, this will include the study design.

&niversity o 1x ord, 2334

Systematic Review Appraisal Sheet /hat were the results0 ,ow are the results presented0 # systematic re iew pro ides a summary of the data from the results of a number of indi idual studies. !f the results of the indi idual studies are similar, a statistical method (called meta-analysis) is used to combine the results from the indi idual studies and an o erall summary estimate is calculated. The meta-analysis gi es weighted alues to each of the indi idual studies according to their size. The indi idual results of the studies need to be expressed in a standard way, such as relati e ris$, odds ratio or mean difference between the groups. %esults are traditionally displayed in a figure, li$e the one below, called a forest plot.

The forest plot depicted above represents a meta-analysis of 5 trials that assessed the effects of a hypothetical treatment on mortality. Individual studies are represented by a black square and a horizontal line, which corresponds to the point estimate and 95 confidence interval of the odds ratio. The size of the black square reflects the wei!ht of the study in the meta-analysis. The solid vertical line corresponds to "no effect# of treatment - an odds ratio of $.%. &hen the confidence interval includes $ it indicates that the result is not si!nificant at conventional levels '()%.%5*. The diamond at the bottom represents the combined or pooled odds ratio of all 5 trials with its 95 confidence interval. In this case, it shows that the treatment reduces mortality by +, '-. %.// 95 0I %.5/ to %.12*. 3otice that the diamond does not overlap the "no effect# line 'the confidence interval doesn#t include $* so we can be assured that the pooled -. is statistically si!nificant. The test for overall effect also indicates statistical si!nificance 'p4%.%%%$*.

Exploring heterogeneity
5etero!eneity can be assessed usin! the 6eyeball7 test or more formally with statistical tests, such as the 0ochran 8 test. &ith the 6eyeball7 test one looks for overlap of the confidence intervals of the trials with the summary estimate. In the e9ample above note that the dotted line runnin! vertically throu!h the combined odds ratio crosses the horizontal lines of all the individual studies indicatin! that the studies are homo!enous. 5etero!eneity can also be assessed usin! the 0ochran chi-square '0ochran 8*. If 0ochran 8 is statistically si!nificant there is definite hetero!eneity. If 0ochran 8 is not statistically si!nificant but the ratio of 0ochran 8 and the de!rees of freedom '8:df* is ) $ there is possible hetero!eneity. If 0ochran 8 is not statistically si!nificant and 8:df is 4 $ then hetero!eneity is very unlikely. In the e9ample above 8:df is 4$ '%.9;:,< %.;+* and the p-value is not si!nificant '%.9;* indicatin! no hetero!eneity. Note: The le el of significance for &ochran ' is often set at (.) due to the low power of the test to detect heterogeneity.

&niversity o 1x ord, 2334