Literature Evaluation

HNS)
resident
ENT-HN Studies
Maharajgunj

Dr.Ramesh Parajuli
MS(ORL1styear
GMSMA of
TUTH, IOM,

Reasons for Surgeons to Read

Medical Literature:
1.Improve Patient Care
2.Learn about Research
3.Educate Peers and Students about Clinical Care

To ensure Best Practice and Treatment for

Patients

Doctor must :

Access Medical Information

Acquire New Knowledge
Achieve Information Mastery in their field

Without systematic approach to identify & critically appraise

clinical research, doctor might become dependant on
inappropriate or outdated information

Keeping Medical Informations up to

date

Past:
Reading few journals every month
Occasional conferences

Present:
Internet
Informations- available to everyone

Over 10,000 new articles per week are archived

by the National Library of Medicine, and at least
twice
that many are published worldwide. (Rudmik
LR,2008)

More than 2 millions/year articles

published in the biomedical press
(scott Brown 7th edn)

Most published articles are not translatable

to clinical practice, or
their methods are not clinically sound

Publication Bias:Articles with Positive findings: Easier to get

published
Articles with Negative findings: Author/Editors

Publication Bias Biased sample of article

 As many as 30% of journal articles may contain

errors.
(Victor A F,2003)

Among health professionals, medical literature is

the preferred source of new knowledge
Medical informations doubles every 5 years
(Castillo D L)

To keep up with available clinical evidence,

todays dedicated physician would have to
read 19 articles everyday, of which only15%
would provide information of practical
relevance. (Diana L C,2008)

Journal Article
Rigid structure
Extracting information easier

Format:
1.Abstract
2.Intorduction
3.Methods
4.Results
5.Discussion
6.Conclusion

1.Abstractoverview of

What the research is about

What it did
How it did
What is found
What those results mean

2.Introduction

Review of previous research

Rationale for the research
What the research is aiming to do

3.Method:

Methods used in the Research

4.Result :
Describe the result found
5.Discussion:
What the results actually mean
6.Conclusion:
Results of several experiments presented and
discussed, Implications of the research
Recommendations about further research or
policy and practice in the relevant area

Knowledge of the Structure and Function

of the Sections of Journal Article: Reduces the time to locate information
Improves efficiency of reading
Reduces the time to read the article

 Reading Articles Uncritically

Acceptance of Text and Arguments
Flawed, Biased and Subjectively written

Efficient Reading is not the only skill we need

Critical Reading

Critical Reading:
Suspending the Judgment on a text until
1.Have understood the message being put forward
2.Evaluated the evidence supporting that message
3.Evaluated the writers perspective

Critical Reading Checklist

To avoid reading poor quality literature and

mastering irrelevant information Systematic
Strategy
A skill that a busy otolaryngologist can learn
Critical reading check list for critical reading

Critical Reading for

Research Articles
Are the limitations of the procedures clear?
Is the methodology valid? (eg.sample size, sampling
method)
Are the results consistent with the objectives?
Are the claims the author makes about his or her
own research internally consistent?
Are the diagrams clear to the reader?

 Critical Thinking/Critical
Analysis:
the intellectually disciplined process of actively and

skillfully conceptualising, applying, analysing,

synthesising and/or evaluating informations gathered
from ,or generalised by, observation, experience,
reflection, reasoning or communication ,as a guide to
belief or action (or argument)
(Scriven & Paul
2001)

What critical analysis is NOT:

1.A
2.A
3.A
4.A

summary of a text
descriptive comment on the text
rehash of the theory learnt in a class
statement of ones own unsupported views

 Harm to pts can be about:Diagnosis:Therapy:Prognosis:-

To prevent the harm to the pts:-most up to date

knowledge available

 Evidence Based Medicine-evidence-based

medicine is the conscientious, explicit and
judicious use of current best evidence in making
decisions about the care of individual patients
(Sackett et al 1996 )

CONSCIENTIOUS
- To do more good than harm
- To do right things in right way
- Carefulness and attention

EXPLICIT
- Openness and transparency
-To evaluate and criticize
the methods used
- Doctor- Patient relationship

JUDICIOUS
- To evaluate and weigh
the evidence of the work
-To consider Benefit &
balance it against harm
Current Best Evidence

Evidence Based Practice (EBP)

Begins and ends with patients
Integrating individual clinical expertise with the
best available external clinical evidence from
systematic research
Application of the knowledge to the prevention,
diagnosis or management of the disease

Evidence Based Medicine practice of medicine

with an emphasis relying on the medical
literature for clinical decision making

EBM approach Improves ability to evaluate clinical literature

Enhances life long learning skills in day to day medical
practice
Keeping up to date
Continuing medical education (CME)

Purpose of EBM

To propagate the evidence but not to replace

individuals clinical experience

One shouldnt expect clinical practice to be

100%evidence based-better patient care shouldnt
be compromised by not counting on evidence
when it is readily available by different sources.

 EBM is key component of modern medical

practice
EBM is necessary but not sufficient
Otolaryngologists should acquire the
fundamentals skills of searching, appraising,
and synthesizing, required to practice EBM
and should do so with the same diligence they
apply to learning surgical skills

Critical Appraisal one aspect of EBM

The ability to apply the principals of analysis
To identify those studies which are unbiased and valid.

Critical Appraisal of Research Studies

helps to:1.Improve patient care
2.Improve teaching
3.Become a better researcher
4.Prepare for journal club
5.Gain respect among the peers

Practising Evidence Based

Health Care:EBM process involves 5 steps
1.Ask clinical question:-formulation of P I C O
question
2.Search literature
3.Appraisal (critically evaluate the literature)
4.Share the knowledge
5.Apply to the patient

 Forming Answerable Questions:

1.Background questions:-understand the problem in general

2.Foreground questions:-decision making questions

 The PICO Format for Foreground

Questions
1.P=Patient and Problem (population-kids, women,
patients)
2.I=Intervention( test)
3.C=Comparison intervention (Control group)
4.O=Outcome

To avoid reading poor quality literature and mastering

irrelevant information Systematic Approach

1.Retrieve
2.Review
3.Reject
4.Read

 5s model for Evidence-Based

Health Care Decisions
(Hynes R B; ACP J Club 2006;145:A8)

To answer a particular question: evidence Based

Research Resources
1.The cochrane collaboration
2.National library of medicine-gateway
3.Open clinical
4.bestBETs
Selection of medical database:
Type of question
Ease of use for a particular problem
Need to limit the search for highest quality studies

 Selecting an Article: Filtering Process

1.Primary Survey (initial evaluation and brief
overview)
(A) Analyse the Title :
(B) Review the list of authors:
(C) Read the summary or abstract beginning
with conclusion:

If the conclusion if valid ,important to me?

Results if true, how useful they are?
Is the primary outcome measured important for me?
Do the interventions make sense?
Can the information be generalised to my patients?

Secondary Survey:
1.Introduction :

Problem under study, context of the study, reasons for

conducting
study

Importance of the topic, what is known and what is unknown

about
the topic

Specific questions (objective, goal of the study, hypothesis) to

be
evaluated

Study sample, primary outcome & intervention being

evaluated

Method design

Conclusion shouldnt extend beyond the stated objective

2.Methods:
Research design-descriptive or comparative study
Single or multicenter
3. Study Sample:
How were the subjects and controls selected?
Are the inclusion & and exclusion criteria sufficiently
clear to describe the target population?
4.Treatment Allocation:
Randomization
Masking (blinding)

5.Outcomes:
primary outcome-all studies

secondary outcome-some studies

How it was measured?
was the measurement free of bias ?
How reproducible were the results?
How to standardize measurements & to minimize interobserver variability?

6.Statistical Analysis:
Statistical Tests

7.Results:
Tables and

How many

How many

How many

Figures,
pts eligible for the study?
enrolled?
completed?

8.Discussion :

Authors interpretation of clinical relevance

Comparison with previous studies, similarities &
differences
Limitation of the study
Suggest new directions for appropriate study

9.Conclusion:
Must consistent with the study objective
Justified by the study results
Shouldnt over generalize the results of the study

 If there is not concealment of

Randomizationmay exaggerate efficacy of the
treatment by as much as 30% more
If there is no Blinding may exaggerate the
effectiveness of the treatment by 15%
If <80% pts Followed Upresults become
meaningless

Types of Study by Content:

1.Evaluation of a new therapy
2.Evaluation of a new diagnostic test
3.Determinnation of the etiology of a condition
4.Prediction of the outcome
5.Natural course of a condition

Levels of
evidence

Level 1 : 1a-SR(homogeneity) of RCTs

1b-Individual RCT
1c-All or None
Level 2 :2a-SR(with homogeneity) of Cohort studies
2b-Individual Cohort Study (including low
quality RCT; eg
<80% follow up)
2c-Outcomes research; ecological studies
Level 3 3a-SR(with homogeneity) of case control studies
3b-Individual case-control study
Level 4 - Case Series(& poor quality cohort & case
control studies)
Level 5 - Expert opinion

Grades of
Recommendation
Grade A- consistent level 1 studies
Grade B- consistent level 2 or 3 studies
or extrapolations from level 1 studies
Grade C- level 4 studies or
extrapolations from level 2 or 3 studies
Grade D- level 5 evidence or
troublingly inconsistent or inconclusive
studies of any level

once the trial or trials have been

found that seems to answer the
questions check three things:
1.Is the study valid?
2.What are the results?
3.Can these be applied to my patients?

(I) Appraising the Diagnostic

Articles:
1.Is the study valid?
Did the authors answer the questions?
1.What were the characteristics of the groups?
2.Is it clear how the test was carried out?
3.Is the test result reproducible?
4.Was the reference standard (gold standard) appropriate?
5.Were the reference standard & the diagnostic test
interpreted blind and independently of each other?
6.Was the reference standard applied to all patients?
7.Was the test evaluated on an appropriate spectrum of
patients?

2. What were the results?

Are the sensitivity/specificity and/or likelihood ratios
presented?
Could the results have occurred by chance?
Are there confidence limits?

Likelihood Ratio : Likelihood Ratio (+ve results)=

sensitivity
(100% - specificity)
Likelihood Ratio (-ve results)= (100% - sensitivity)
specificity
If LR > 10 + ve
< 1 - ve

3. Will the Results help my Patients?

1.Is the diagnostic test available, affordable, accurate
& precise in my setting?
2.Are the results applicable to my pts? Do my pts have
a similar mix of disease severity & competing
conditions?
3.Will the results change my case management ?will
the information gain be sufficient to change a clinical
decision (rule in or rule out)?
4.Will patients be better off as a result of performing
the test?

(II) Appraising Therapy Articles

1.IS THE STUDY VALID?
Did the authors answer the questions?
1.What were the characteristics of the pts?
2.Were the groups similar at the start of the trial?
3.Aside from the experimental treatment, Were the groups treated
equally?
4.What was the treatment & what was it compared against
placebo?
5.Was randomization used?
6.Were all pts followed up at its conclusion?
7.Wre they analyzed in the groups to which they were
randomized?
8.Were blinding used?
9.Was the length of study appropriate?

2.WHAT WERE THE RESULTS?

Event rates-control event rate (CER), experimental event rate
(EER)
Relative risk
CER-EER
Relative risk reduction (RRR) = -----------------CER
Absolute risk reduction (ARR) = CER- EER
Number needed to treat (NNT) =

1
------ARR

3.WILL THE RESULTS HELP MY

PATIENTS?
1.Can the results be applied to my patient care?
Would my pts eligible for the study?
Are there any reasons why the results shouldnt be
applied to my pts?
2.Were all clinically important outcomes consider?
3.Are the likely treatment benefits worth the potential
harm and costs?

(III) Appraising the Systematic

Review
1.IS THE REVIEW VALID?
Did the authors answer the questions?
1.What databases & other sources did the authors of this review
search?
2.What were their eligibility criteria (inclusion & exclusion)? do
these seems appropriate?
3.Was there independent data extraction of the result by the
reviewers (then compared later)?
4.Is there a description of the quality of each trial?
5.Were the results consistent from study to study
(homogeneous)?

(2)WHAT Were The Results?

If the results of the studies have been
combined, was it reasonable to do so?
consider whether:
the results of each study are clearly displayed?
the results were similar from study to study?

How are the results presented and what

is the main result?
How the results are expressed (eg. odds ratio, relative risk etc)

How precise are these results?

3 .CAN I APPLY EVIDENCE FROM THIS

SYSTAMATIC REVIEW IN CARING FOR MY PTS?
Whether the population sample different from my
population ?
Whether my local setting differs ?
Were all clinically important outcomes considered?
Should policy or practice change as a result of the
evidence contained in this review?
Whether any benefit reported outweighs any harm/or
cost

(IV) Critical Appraisal Skills for the

Cohort Study
1.ARE THE RESULTS VALID?
Did the study address a clearly focused issue?
Did the authors use an appropriate method to
answer their question?
Was the cohort recruited in an acceptable way?
Was the exposure accurately measured to minimize
bias?
Was the outcome accurately measured to minimize
bias?

Have the authors identified all important

confounding factors?

Was

was the follow up of subjects long enough?

2.WHAT ARE THE RESULTS?

What are the results of this study?

How precise are the results?

Do I believe the results?

the follow up of subjects complete enough?

3.WILL THE RESULTS HELP ME

LOCALLY?
Can the results be applied to the local population?
Do the results of this study fit with other available
evidence?

(V) Appraisal of a Case Control Study

1.ARE THE RESULTS OF THE STUDY VALID?
1.Did the study address a clearly focused issue?
2.Did the authors use an appropriate method to answer their
question?
3.Were the cases recruited in an acceptable way?
4.Were the controls selected in an acceptable way?
5.Was the exposure accurately measured to minimize bias?
6.(A)what confounding factors have the authors accounted for?
(B) Have the authors taken account of the potential
confounding factors in the design and/or in their analysis?

2.WHAT ARE THE RESULTS?

7.What are the results of this study?
8.How precise are the results?
how precise is the estimate of risk?
9.Do I believe the results?

3.WILL THE RESULTS HELP ME

LOCALLY?
10.Can the results be applied to the local population?
11.Do the results of this study fit with other available
evidence?

REFERENCES:
1.Scott-Browns Otorhinolaryngology, Head & Neck Surgery(7 th edition)
2.Critical appraisal skills programme (CASP)-making sense of evidence
public health resource unit;england(2006)
3.Evaluating the literature (emedicine)
4.Unilearing-University of wollongong, australia
5.Evidence based medicine (EBM)-what, why and how(KUMJ 2003)
6CEBM-EBM tools
7.Assesing the medical literature: let the buyer beware
8.Knoweledge managemet:how to keep up with the literature
9.Literature review(st.kates libraries guides)
10.How to evaluate the literature(stanford university)
11.Guidelines for evaluating writing about literature
(john jay college of criminal justice)
12.Critical analysis-so what does that mean(university of bradford)
13.Evaluating the medical literature(clista clantin 2009)
14.Literature evaluation (hussain al awami)
15.Introduction to critical appraisal(jason curtis-NHS)

16.How to read a paper? (BMJ)

17.Crafting the literature review (massey university;2006)
18.Argument based medical ethics
19.Assessment of the critical appraisal (american journal of
surgery,2004)
20.Teaching critical appraisal skills to medical students in
obstetrics and gynecololgy (university of california OBGYN
department)
21.A simple method for evaluating clinical literature (JF Robert)
22.Evidence based medicine literature evaluation questions
(washington state university,2006)
23.Evidence based medicine capitol conference 2007

Thank you