Types of study designs:

from descriptive studies to

randomized controlled trials

Kirsten Bibbins-Domingo, PhD, MD

Assistant Professor of Medicine and of
Epidemiology and Biostatistics
University of California, San Francisco

Objectives
To understand the difference between
descriptive and analytic studies
To identify the hierarchy of study
designs, and the strengths and
weakness of each design
To be able to apply different study
designs to the same research question

Types of Studies
Descriptive Studies
Observational Analytic Studies
Cross Sectional studies
Case Control studies
Cohort studies

Experimental Studies
Randomized controlled trials

Hierarchy of Study Types

Descriptive
Case report
Case series
Survey

Analytic

Observational
Cross sectional
Case-control
Cohort studies

Experimental
Randomized
controlled trials

Strength of evidence for causality between a risk factor and outcome

Descriptive studies
Getting a lay of the land
Surveys (NHIS, MCBS)
How many men in the U.S. filled Viagra
prescriptions in 2004?

Describing a novel phenomena

Case reports or case series
Viagra-associated serous macular
detachment.
Sildenafil-associated nonarteritic anterior
ischemic optic neuropathy.

Descriptive studies
Cannot establish causal relationships
Still play an important role in describing trends
and generating hypotheses about novel
associations
The start of HIV/AIDS research
Squamous cell carcinoma in sexual partner of Kaposi
sarcoma patient. Lancet. 1982 Jan 30;1(8266):286.
New outbreak of oral tumors, malignancies and infectious
diseases strikes young male homosexuals. CDA J. 1982
Mar;10(3):39-42.
AIDS in the "gay" areas of San Francisco. Lancet. 1983
Apr 23;1(8330):923-4.

Analytic Studies
Attempt to establish a causal link between
a predictor/risk factor and an outcome.
You are doing an analytic study if you have
any of the following words in your research
question:
greater than, less than, causes, leads to,
compared with, more likely than, associated
with, related to, similar to, correlated with

Hierarchy of Study Types

Descriptive
Case report
Case series
Survey

Analytic

Observational
Cross sectional
Case-control
Cohort studies

Experimental
Randomized
controlled trials

Strength of evidence for causality between a risk factor and outcome

Research Question
Is the regular consumption of Red Bull
associated with improved academic
performance among U.S. medical students?

Rationale
functional drink designed for periods of
mental and physical exertion.
performance, concentration, memory,
reaction time, vigilance, and emotional
balance

Taurine + glucuronolactone + caffeine

Background

Alford C, Cox H, Wescott R. The effects of red bull energy drink on

human performance and mood. Amino Acids. 2001;21(2):139-50.

Warburton DM, Bersellini E, Sweeney E. An evaluation of a

caffeinated taurine drink on mood, memory and information
processing in healthy volunteers without caffeine abstinence.
Psychopharmacology (Berl). 2001 Nov;158(3):322-8.

Seidl R, Peyrl A, Nicham R, Hauser E. A taurine and caffeinecontaining drink stimulates cognitive performance and well-being.
Amino Acids. 2000;19(3-4):635-42.

Horne JA, Reyner LA. Beneficial effects of an "energy drink" given

to sleepy drivers. Amino Acids. 2001;20(1):83-9.

Kennedy DO, Scholey AB. A glucose-caffeine 'energy drink'

ameliorates subjective and performancedeficits during prolonged
cognitive demand. Appetite. 2004 Jun;42(3):331-3.

Great idea, but how do you get

started.
Interesting, novel, and relevant, but
You only have 25,000 dollars to start
investigating this question.
What is feasible?

Study Design #1
Cross-sectional study of UCSF medical students
taking USMLE Step 2
Questionnaire administered when registering
for USMLE 2
Primary predictor: self-report of >3 cans Red Bull
per week for the previous year
Covariates: Age, sex, undergraduate university, place
of birth

Outcome: Score on USMLE Step 2

Cross-sectional study: structure

Red Bull consumption

USMLE Score

time

Cross-sectional Study:
Descriptive value:
How many UCSF medical students drink Red Bull?
What is the age and sex distribution of UCSF medical
students who drink Red Bull?
Analytic value:
Is there an association between regular Red Bull
consumption and test scores among UCSF med students?
Univariate
Multivariate (controlling for confounders)

Other cross-sectional surveys:

AAMC
California Health Interview Survey (NHIS, CHIS)
National Health and Nutrition Exam Survey (NHANES)

Cross-sectional Study: Pluses

+

Prevalence (not incidence)

Fast/Inexpensive - no waiting!

No loss to follow up

Associations can be studied

Measures of association
Risk ratio
(relative risk)

Disease
Yes

No

Yes

No

Risk
Factor

A
A+B
C
C+D

Cross-sectional study: minuses

-

Cannot determine causality

Red Bull consumption

USMLE Score

time

Cross-sectional study: minuses

-

Cannot determine causality

ACE inhibitor use and hospitalization rates

among those with heart failure
Heart failure patients with a documented
DNR status and mortality

time

Cross-sectional study: minuses

-

Cannot determine causality

Cannot study rare outcomes

What if you are interested in

the rare outcome?
The association between regular Red Bull
consumption and
A perfect score on the USMLE Step 2
Graduating top 1% of the medical school
class
Acceptance into a highly selective residency
ANSWER: A Case-Control study

Study Design #2
A case-control study
Cases: 4th year med students accepted to
residency in highly selective specialty X.
Controls: 4th year med students who applied
but were not accepted.
Predictor: self-reported regular Red Bull
consumption
Additional covariates (age, sex, medical school,
undergraduate institution)

Case control studies

Investigator works backward
(from outcome to predictor)
Sample chosen on the basis of
outcome (cases), plus comparison
group (controls)

Case-control study structure

present

Red Bull consumption

YES
Red Bull consumption
NO

ACTUAL CASES
TARGET
CASES
4th year UCSF students
Medical students accepted to
who matched in highly
highly selective residencies
selective specialty X

ACTUAL
CONTROLS
TARGET
CONTROLS
th
4
year students
who to
failed
All unsuccessful
applicants
highly
to
match
in
highly
selective
selective residency programs
specialty X
time

Case control studies

Determines the strength of the
association between each
predictor variable and the
presence or absence of disease
Cannot yield estimates of
incidence or prevalence of
disease in the population (why?)
Odds Ratio is statistics

Case-control Study: pluses

+ Rare outcome/Long latent period
+ Inexpensive and efficient: may be only
feasible option
+ Establishes association (Odds ratio)
+ Useful for generating hypotheses
(multiple risk factors can be explored)

Case-control study-minuses

Causality still difficult to establish

Selection bias (appropriate controls)

Recall bias: sampling (retrospective)

Caffeine and Pancreatic cancer in the GI clinic

Abortion and risk of breast cancer in Sweden

Cannot tell about incidence or prevalence

Studies of diagnostic tests:

Sensitivity, specificity
Positive predictive value, negative predictive value

Measures of association
Disease

Yes

Sensitivity = A/A+C

Yes

No

Test

Specificity = D/B+D

PPV = A/A+B
NPV = D/C+D

No

Case-control - the house red

Rely tampons and toxic shock syndrome:
High rates of toxic shock syndrome in menstruating women
Suspected OCPs or meds for PMS
Cases: 180 women with TSS in 6 geographic areas
Controls: 180 female friends of these patients and 180
females in the same telephone code
Tampon associated with TSS (OR = 29!)
Super absorbency associated with TSS (OR 1.34 per gm
increase in absorbency)
Led to RELY brand tampons being taken off the market.

Where are we?

Preliminary results from our cross-sectional and
case-control study suggest an association
between Red Bull consumption and improved
academic performance among medical students
Whats missing? - strengthening evidence for a
causal link between Red Bull consumption and
academic performance
Use results from our previous studies to apply
for funding for a prospective cohort study!

Study design #3
Prospective cohort study of UCSF medical
students Class of 2009
All entering medical students surveyed
regarding beverage consumption and
variety of other potential covariates
Survey updated annually to record
changes in Red Bull consumption
Outcomes: USMLE Step 1 score, USMLE
Step 2 score, match in first choice
residency

Cohort studies

A cohort (follow-up, longitudinal) study is a

comparative, observational study in which
subjects are grouped by their exposure status,
i.e., whether or not the subject was exposed to
a suspected risk factor

The subjects, exposed and unexposed to the

risk factor, are followed forward in time to
determine if one or more new outcomes
(diseases) occur

Subjects should not have outcome variable on entry

No new subjects allowed in after initial recruitment

The rates of disease incidence among the

exposed and unexposed groups are

Elements of a cohort study

Selection of sample from population

Measures predictor variables in sample
Follow population for period of time
Measure outcome variable

Famous cohort studies

Framingham
Nurses Health Study
Physicians Health Study
Olmsted County, Minnesota

Prospective cohort study structure

The present

The future

Top USMLE scorers

Everyone else
time

Strengths of cohort studies

Know that predictor variable was present
before outcome variable occurred (some
evidence of causality)
Directly measure incidence of a disease
outcome
Can study multiple outcomes of a single
exposure (RR is measure of association)

Weaknesses of cohort studies

Expensive and inefficient for studying rare
outcomes
HERS vs. WHI

Often need long follow-up period or a very large

population
CARDIA

Loss to follow-up can affect validity of findings

Framingham

Other types of cohort studies

Retrospective cohort
Identification of cohort, measurement of
predictor variables, follow-up and
measurement of outcomes have all occurred
in the past
Much less costly than prospective cohorts
Investigator has minimal control over study
design

Other types of cohort studies

Nested case-control study
Case-control study embedded in a cohort study
Controls are drawn randomly from study sample

Double cohort
Used to compare two separate cohorts with
different levels of exposure to predictor variable
(e.g., occupational groups)

What type of study is this?

Among individuals with coronary disease, what is the
association between baseline levels of B-type natriuretic
peptide and subsequent risk of heart failure?
Among individuals presenting to heart failure clinic, what
is the association between self-reported symptoms and
risk of hospitalization for heart failure?
Using data from HERS (RCT of HRT in women with
coronary disease):
Determine the risk factors for developing incident heart
failure among women without heart failure at baseline.
Determine whether HRT is associated with mortality
among women with heart failure.
Determine genetic markers for development of heart
failure among black women in HERS.

Hierarchy of Study Types

Descriptive
Case report
Case series
Survey

Analytic

Observational
Cross sectional
Case-control
Cohort studies

Experimental
Randomized
controlled trials

Strength of evidence for causality between a risk factor and outcome

What distinguishes observational

studies from experiments?
Ability to control for confounding
Confounder

Predictor

Outcome

Examples:
sex (men are more likely to drink red bull and men are
more likely to match in neurosurgery)
Undergraduate institution (students from northwest school are
more likely to drink red bull and also more likely to score higher on
USMLE)

But we measured all of the

potential confounders.
In a prospective cohort study you can
(maybe) measure all potential known
confounders, but
You cant control for unanticipated or
unmeasured confounders

Study design # 4
Randomized controlled trial of daily Red Bull
consumption among entering UCSF medical
students Class 2009
Randomized to daily consumption of Red Bull
vs. daily consumption of placebo
Outcomes: USMLE Step 1 score, USMLE Step
2 score, match in first choice residency

Randomized controlled trials

Investigator controls the predictor
variable (intervention or treatment)
Major advantage over observational
studies is ability to demonstrate
causality
Randomization controls unmeasured
confounding
Only for mature research questions

Basic Trial Design

Population
Population

Sample
Sample

Treatment

Dx No Dx

Randomization
Randomization

Control
Placebo

Dx No Dx

Steps in a randomized
controlled trial
1. Select participants
high-risk for outcome (high incidence)
Likely to benefit and not be harmed
Likely to adhere

2. Measure baseline variables

3. Randomize
Eliminates baseline confounding
Types (simple, stratified, block)

Steps in a randomized
controlled trial
4. Blinding the intervention

As important as randomization
Eliminates
co intervention
biased outcome ascertainment
biased measurement of outcome

5. Follow subjects

Adherence to protocol
Lost to follow up

6. Measure outcome

Clinically important measures

Adverse events

What
What is
is Blinding?
Blinding?

Single
Single blind
blind -- participants
participants are
are not
not
aware
aware of
of treatment
treatment group
group

Double
Double blind
blind -- both
both participants
participants
and
and investigators
investigators unaware
unaware

Triple
Triple blind
blind -- various
various meanings
meanings

persons
persons who
who perform
perform tests
tests

outcome
outcome adjudicators
adjudicators

safety
safety monitoring
monitoring group
group

Why blind?: Co interventions

Unintended effective interventions
participants use other therapy or change
behavior
study staff, medical providers, family or
friends treat participants differently

Nondifferential - decreases power

Differential - causes bias

Why
Why blind?:
blind?: Biased
Biased Outcome
Outcome
Ascertainment
Ascertainment or
or adjudication
adjudication

If
If group
group assignment
assignment is
is known
known

participants
participants may
may report
report symptoms
symptoms or
or outcomes
outcomes
differently
differently

physicians
physicians or
or investigators
investigators may
may elicit
elicit symptoms
symptoms
or
or outcomes
outcomes differently
differently

Study
Study staff
staff or
or adjudicators
adjudicators may
may classify
classify similar
similar
events
events differently
differently in
in treatment
treatment groups
groups

Problematic
Problematic with
with soft
soft outcomes
outcomes

investigator
investigator judgement
judgement

participant
participant reported
reported symptoms,
symptoms, scales
scales

Analysis of randomized
controlled trial
Analyzed like cohort study with RR
Intention to treat analysis
Most conservative interpretation
Include all persons assigned to
intervention group (including those who
did not get treatment or dropped out)

Subgroup analysis
Groups identified pre-randomization

High Quality Randomized Trials

Tamper-proof randomization
Blinding of participants, study
staff, lab staff, outcome
ascertainment and adjudication
Adherence to study intervention
and protocol
Complete follow-up

A study type of every budget,

purpose andof
research
Hierarchy
Study question
Types

Descriptive
Case report
Case series
Survey

Analytic

Observational
Cross sectional
Case-control
Cohort studies

Experimental
Randomized
controlled trials

Strength of evidence for causality between a risk factor and outcome