D r.

E r i c a
Ro e b b e l e n

CRITICAL APPRAISAL

January 16,
2 0 15
B a n d a Ac e h

WHAT IS CRITICAL APPRAISAL?

The process of systematically examining research
evidence to assess validity, results and relevance
before using it to inform a decision
Why do you think this is important?

WHAT TYPES OF EVIDENCE DO WE

ENCOUNTER?
Research articles
Expert opinion from our seniors, from staff
physicians
Case reports
Anecdotes or stories
Popular media

REVIEW TYPES OF STUDY DESIGN

Observational or experimental
Case-control study
Randomized Control Trial

Qualitative
Studies

Cohort Study

Meta-analysis

ASSESSING EVIDENCE
What types of studies give us the strongest evidence?

ASSESSING EVIDENCE

APPROACH TO CRITICAL APPRAISAL

When reading a study, always ask yourself these 4
questions:
Who is the POPULATION or PATIENT
CHARACTERISTICS?
What is the INTERVENTION?
What is the COMPARISON or CONTROL group?
What is the OUTCOME being measured?
Remember PICO
These are also the steps to take to formulate a
research question

APPROACH TO CRITICAL APPRAISAL

Once you know the basics of the study (the PICO),
you need to ask these 3 questions:
1. Are the results of the study valid?
2. What are the results?
3. Will the results help me in caring for my patients?

ASSESSING VALIDITY

What is validity?
The degree to which the
outcome observed in the study
can be attributed to the
intervention.

ASSESSING VALIDITY
5 questions to ask about the validity of
primary studies:
1. Were the patients randomized?
2. Was the follow-up of patients suffi ciently
long enough and complete?
3. Were all patients analyzed in the groups to
which they were randomized? (intention to
treat analysis)
4. Were the groups treated equally except for
the intervention?
5. Were the patients and clinicians kept blind to
treatment? (was the study double
blinded?)

ANALYSIS
Per-Protocol Analysis (PP)
Strategy of analysis in which only patients
who complete the entire study are counted
towards the results
Intention to Treat Analysis (ITT)
When groups are analyzed exactly as they
existed upon randomization (i.e. using data
from all patients, including those who did not
complete the study)

ASSESSING VALIDITY
Some more questions to consider:
Were the groups similar at the start of the trial?
(think about gender, comorbidities, age, etc.)
Were the appropriate and valid exposure and
outcome measures obtained?
Were outcome assessors aware of group
allocation?
Was the study conducted in an ethical way?
Who funded the study?
Do the authors have any confl icts of interest?

ASSESSING VALIDITY
Internal Validity
the degree to which the findings of the
sample truly represent the findings in the
study population
External Validity
degree to which the results of the study can
be generalized to other situations or
populations

APPROACH TO CRITICAL APPRAISAL

Once you know the basics of the study (the PICO),
you need to ask these 3 questions:
1. Are the results of the study valid?
2. What are the results?
3. Will the results help me in caring for my patients?

CRITICAL APPRAISAL OF RESULTS

What was the impact of the treatment
results?
How precise was the estimate of the
treatment eff ect?
What were the confi dence intervals and
power of the study?

A VERY QUICK STATISTICS REVIEW

Confi dence Interval:
provides a range of values within which the
true population result (e.g. the mean) lies
frequently reported as 95% CI (e.g. one can
be 95% certain that the true value is within
this data range)
bounded by the upper and lower confidence
limits
A wider confidence interval implies more
variance than a tighter confidence interval

A VERY QUICK STATISTICS REVIEW

Power
the probability of finding a specified
difference to be statistically significant at a
given p-value
power increases with an increase in sample
size
the probability of a true positive result

EFFECTIVENESS OF INTERVENTIONS
Relative Risk Reduction (RRR)
Absolute Risk Reduction (ARR)
Number Needed to Treat (NNT)
Number Needed to Harm (NNH)

EFFECTIVENESS OF A TEST
SPIN: Specificity use a SPecific test to rule IN a
hypothesis. Specific tests have very few false
positives.
SNOUT: Sensitivity use a SENsitive test to rule
OUT a hypothesis. Sensitive tests have few false
negatives.

APPROACH TO CRITICAL APPRAISAL

Once you know the basics of the study (the PICO),
you need to ask these 3 questions:
1. Are the results of the study valid?
2. What are the results?
3. Will the results help me in caring for my patients?

WILL THE RESULTS HELP ME IN

CARING FOR MY PATIENTS?
are the results clinically signifi cant?
can I apply the results to my patient
population?
were all clinically important outcomes
considered?
are the likely treatment benefi ts worth the
potential harm and costs?