Critical

Appraisal:
Prognosis
Studies

Sayu Aryantari Putri Thanaya

Learning Objectives

– Introduction to prognostic studies

– Research designs that are specific to prognostic studies: cohort and case-control
studies
– Overview of cohort & case-control studies:
Purpose, study design, procedure, advantages & disadvantages, outcome & interpretation
– Critical appraisal tools
– Critical appraisal of cohort and case-control studies
– Critical appraisal exercise – appraising a cohort/case-control study
Introduction to prognostic
studies
– Prognostic questions may be about the impact of a disease or event on a patient’s
long-term outcome. For example, a patient may ask, “Will I be able to ski after back
surgery?” or “When can I expect to go back to work?”
– Many factors influence the answers to prognostic questions such as the severity of
the patient’s problem, gender, age, home environment, and co-morbidities.
– Valid prognostic studies can assist in answering these types of questions, and they
assist in weighing the various factors that may contribute to specific outcomes.
Introduction to prognostic
studies
– The research designs most typically applied to prognostic questions are
observational studies that used associations between or among variables.
– Prognostic studies are studies that examine selected predictive variables or risk
factors and assess their influence on the outcome of a disease.
– These typically include cohort and case-control designs. Cohort and case control
designs are commonly used in epidemiological research in which very large groups
of subjects may be followed over long periods of time.
– The Framingham Heart Study began as a study of the health of a cohort of people in
1948 and later included two generations of cohorts.
Cohort Studies
Cohort studies

– In a cohort study, a group of subjects who are likely to develop a certain condition or
outcome is followed into the future (prospectively) for a sufficient length of time to
observe if the subjects develop the condition.
– Cohort studies can provide data concerning the timing of the development of the
outcome within the group and assist in defining possible causal factors in developing the
condition. The factors, or risks, for a particular outcome are identified, and the effect is
observed.
– Goal: To determine whether there is an association between a factor and development
of a disease.
– A cohort study is useful for estimating the risk of disease, the incidence rate and/or
relative risks.
Cohort Study: Procedure

http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2998589/
 Cohort Study: Procedure

Define Study Participants ①

PRESENT
EXPOSED NON-
②
EXPOSED
③
④
DISEASE DISEASE DISEASE DISEASE
FUTURE DEVELOPS DOSE NOT DEVELOPS DOSE NOT
DEVELOP DEVELOP
Cohort Study: Procedure

① Define eligible Study Participants

② The investigator selects a group of EXPOSED individuals
and a group of NON EXPOSED individuals
③ Follow up both groups
④ Compare incidence of disease
Types of cohort studies

Prospective
Retrospective
 Types of Cohort studies
NOW

EXPOSE OUTCOME

Collect Measure
Information Outcome

EXPOSE OUTCOME

Time
Prospective cohort study

 Investigator
 Starts the study (from the beginning) with the identification of the population and
the exposure status (exposed/not exposed groups)
 Follows them (over time) for the development of disease
 Takes a relatively long time to complete the study (as long as the length of the study)
Retrospective cohort study

 Investigator
 Uses existing data collected in the past to identify the population and the exposure status
(exposed/not exposed groups)
 Determines at present the (development) status of disease
 Investigator spends a relatively short time to:
 Assemble study population (and the exposed/not exposed groups) from past data
 Determine disease status at the present time (no future follow-up)
Cohort: Strengths & Weaknesses
Strengths
Establish incidence (risk) directly
Multiple outcomes
Study of rare exposure
Weaknesses
Not good for rare disease
Not good for diseases that take a
long time to develop
 Risk measurement in Cohort Study
DISEASE DISEASE
PRESENT NOT
PRESENT

EXPOSED a b

Follow-up
NOT c d
EXPOSED

Risk in EXPOSED a/(a+d)

Relative Risk (RR)= =
Risk in NOT EXPOSED c/(c+d)
 Interpretation of Relative Risk (RR)

Relative Risk (RR) Interpretation

=1 No association

>1 Risk in EXPOSED greater than Risk in NOT

EXPOSED
(Positive association; Risk Factor)
<1 Risk in EXPOSED less than Risk in NOT EXPOSED
(Negative association; Protective Factor)

95% confidence interval (CI); A 95% probability which the interval includes the true relative risk (RR)

If 95% CI range includes “1”, it is not statistically significant since it could be either a risk factor (RR>1) or a
protective factor (RR<1). If 95 % CI range is greater than 1, the exposure is a significant risk factor (RR>1) with a
probability of higher than 95%.
Case-control studies
Case-control study

– A case-control study is designed to help determine if an exposure is associated with

an outcome.
– It is always retrospective because it starts with an outcome then traces back to
investigate exposures.
– Case control studies are conducted after an outcome of interest has occurred. The
factors that contributed to the outcome are studied in a group that has the outcome
(case group) and compared to a group that does not have the outcome of interest.
Case-control study: Procedure
 Case-control study: Procedure

②
PAST NOT NOT
EXPOSED EXPOSED EXPOSED EXPOSED

HAVE THE DO NOT

① DISEASE HAVE THE
PRESENT DISEASE

“CASES” “CONTROLS”
Case-control Study: Procedure

① Identification of CASES and CONTROLS (present)

② Measurement of exposure and determination of
EXPOSED or NON EXPOSED (Past)
③ Expected findings if the exposure is associated with
DISEASE
Why case-control?

– Compared to prospective cohort studies, case-control study tends

to be less costly and shorter in duration. In several situations they
have greater statistical power than cohort studies, which must
often wait for a 'sufficient‘ number of disease events (target
disease) to accrue.
Case-control: Strengths &
Weaknesses
Strength
Good for diseases with long latency
Good for rare diseases
Can determine multiple exposures
Faster results
Weakness
Can’t establish estimate of risk directly
nor determine prevalence
Can only study one disease
More prone to bias
Advantage and Disadvantage of Case-
Control Study compared to Cohort study

Cohort Study Case-control Study

Measures Incidence Rate, Relative Risk (RR) Odds Ratio (OR) only
Cost Expensive Inexpensive
Study term Long term Short term
Sample size Need large sample Powerful with small sample
cases
Exposure Good for rare exposure Limited to rare exposure
Disease Poor potential for rare Good for rare disease
Possible for several diseases Only one disease
Causal Potentially strong Potentially less strong
Risk assessment in Case-control Study

CASES (with CONTROLS

Follow-up
DISEASE) (Without DISEASE)
EXPOSED a b
NON-EXPOSED c d

Odds = prevalence / (1 – prevalence)

Odds ratio = (Odds in cases) / (Odds in controls)
(a/c) / (b/d) = ad / bc
 Interpretation of Odds Ratio

Odds Ratio (OR) Interpretation

=1 No association

≥1 Risk Factor

≤1 Protective Factor

– 95% confidence interval (CI); A 95% probability which the interval includes the true odds ratio
(OR)
– If 95% CI range includes “1”, it is not statistically significant since it could be either a risk factor
(OR≧1) or a protective factor (OR≦1). If 95 % CI range is greater than 1, the exposure is a
significant risk factor (OR≧1) with a probability of higher than 95%.
Which study design would you
choose?

An Indonesian neurologist has a hypothesis that high coffee

consumption in youth may be associated with Pick disease in middle
or elder age.

An Indonesian clinician has a research question whether ratio of LDL

and HDL cholesterol (L/H ratio) is associated with cardiovascular
events among Vietnamese diabetic patients.
Critical Appraisal
Critical Appraisal

– Critical appraisal is the process of carefully and systematically

examining research evidence to judge its trustworthiness, its value
and relevance in a particular context.
Critical Appraisal Tools

– Joanna Briggs Institute (JBI) Critical Appraisal Tools

http://joannabriggs.org/research/critical-appraisal-tools.html
– Critical Appraisal Skills Program (CASP) Critical Appraisal Tools
https://casp-uk.net/casp-tools-checklists/
– Centre for Evidence-Based Medicine (CEBM) Critical Appraisal Tools
https://www.cebm.net/2014/06/critical-appraisal/
Critical Appraisal
CASP

How to use this appraisal tool: Three broad issues need to be considered when
appraising a cohort study:
– Are the results of the study valid? (Section A)
– What are the results? (Section B)
– Will the results help locally? (Section C)
 Cohort Study Critical Appraisal

PICO
– Descriptions of participants to determine if patients within and across groups have similar characteristics in relation to
exposure (e.g. risk factor under investigation). The two groups selected for comparison should be as similar as possible
in all characteristics except for their exposure status, relevant to the study in question. The authors should provide
clear inclusion and exclusion criteria that they developed prior to recruitment of the study participants.
– The participants should be free of the outcomes of interest at the start of the study. Refer to the ‘methods’ section in
the paper for this information, which is usually found in descriptions of participant/sample recruitment, definitions of
variables, and/or inclusion/exclusion criteria.
– Mention or describe how the exposures were measured. The exposure measures should be clearly defined and described in
detail. This will enable reviewers to assess whether or not the participants received the exposure of interest.
– Clearly describe the method of measurement of exposure.
– Assess validity of exposure measurement
– Assess reliability of exposure measurement to check repeatability of measurements of the exposures. These usually include intra-
observer reliability and inter-observer reliability.
– Read the methods section of the paper. If for e.g. lung cancer is
assessed based on existing definitions or diagnostic criteria, then
the answer to this question is likely to be yes.
– Determine if the measurement tools used were validated
instruments  validity.
– It is ideal if less objective outcomes are assessed blindly, that is,
the individual determining the outcome does not know whether
the patient has a potential prognostic factor.
– Confounding has occurred where the estimated intervention exposure effect is biased by the
presence of some difference between the comparison groups (apart from the exposure
investigated/of interest).
– Typical confounders include baseline characteristics, prognostic factors, or concomitant
exposures (e.g. smoking). A confounder is a difference between the comparison groups and it
influences the direction of the study results.
– Strategies to deal with effects of confounding factors may be dealt within the study design or in data
analysis.
– By matching or stratifying sampling of participants, effects of confounding factors can be adjusted for. When
dealing with adjustment in data analysis, assess the statistics used in the study. Most will be some form of
multivariate regression analysis to account for the confounding factors measured. Look out for a
description of statistical methods as regression methods such as logistic regression are usually employed
to deal with confounding factors/variables of interest.
It is important in a cohort study that a greater percentage of people
are followed up. As a general guideline, at least 80% of patients
should be followed up. Generally a dropout rate of 5% or less is
considered insignificant. A rate of 20% or greater is considered to
significantly impact on the validity of the study.

The appropriate length of time for follow up will vary with the
nature and characteristics of the population of interest and/or
the intervention, disease or exposure.
= 1 - No association

>1 - Risk in EXPOSED greater than Risk in NON EXPOSED

(Positive association; Risk Factor)

<1 - Risk in EXPOSED less than Risk in NON EXPOSED

(Negative association; Protective Factor)
– Confidence interval - a way of expressing how certain we are about the findings from a study,
using statistics.
– 95% CI  the range of values within which you can be 95% certain that the true value lies. A 95%
probability which the interval includes the true relative risk (RR).
– Wide CI = less reliable results, lack certainty of the true effect
– Narrow CI = more precise estimate

– If 95% CI range includes “1”, it is not statistically significant since it could be either a risk factor (RR>1) or a
protective factor (RR<1). If 95 % CI range is greater than 1, the exposure is a significant risk factor (RR>1) with
a probability of higher than 95%.

– If RR is equal to 1, it means no
association between EXPOSED
and NOT EXPOSED.
– If RR is greater than 1, EXPOSE
is risk factor.
– If RR is less than 1, EXPOSE is
protective factor, like as
Vaccine.
 Physical Activity and the Risk of Stroke
RR 0.84; 95% CI 0.73-0.96
Interpretation:
Individuals with physical activity had 0.84 times the risk of stroke compared with
those with no physical activity
Or
Individuals with physical activity had a 16% reduced risk of stroke compared with
those with no physical activity
(1 - 0.84 = 0.16 = 16%)
95% CI 0.73-0.96  we are 95% confident that the relative risk of stroke in
individuals with physical activity is between 0.73 and 0.96 and 0.87.
The null value is 1. Since the 95% confidence interval does not include the null
value (RR=1), the finding is statistically significant.
References

– Tridjaja AAP, B. 2016. Telaah Kritis Makalah Prognosis. Sari Pediatri, 4(3), pp.152-4.
– Critical Appraisal Skills Program (CASP) Critical Appraisal Tools
https://casp-uk.net/casp-tools-checklists/
– Fetters, L. and Tilson, J. 2012. Evidence Based Physical Therapy. Philadelphia: F.A. Davis Company.
– Joanna Briggs Institute (JBI) Critical Appraisal Tools
http://joannabriggs.org/research/critical-appraisal-tools.html
– Leen, B. et al. 2014. Evidence-based practice: a practice manual [Online] Available at:
http://hdl.handle.net/10147/317326 [Accessed 10 February 2019].
– Song, J.W. and Chung, K.C. 2010. Observational studies: cohort and case-control studies. Plastic and
reconstructive surgery, 126(6), p.2234.
Learning Task

– Bacalah artikel jurnal berikut:

Jeong, H.G. et al. 2017. Physical activity frequency and the risk of stroke: A
Nationwide Cohort Study in Korea. Journal of the American Heart
Association, 6(9), p.e005671.
– Lakukan critical appraisal pada artikel diatas menggunakan salah satu critical
appraisal tools (JBI atau CASP).