Deciphering Data:

How to Make Sense

of the Research
Literature
Michele Walsh & Kara Haberstock Tanoue
Community Research, Evaluation and Development (CRED)
Frances McClelland Institute for Children, Youth and Families
Norton School of Family & Consumer Sciences
University of Arizona
Evaluation Group for Analysis of Data (EGAD)

Publications are
the Currency of
Academia

 Research Question
Hypotheses

The
Anatomy
of a
Scientific
Article

Methods
Sample
Variable Selection
Study Design
Results
Statistical
Significance
Effect Size
Conclusions
Jackson, S. E., Beeken, R. J., & Wardle, J.
(2014). Perceived weight discrimination
and changes in weight, waist
circumference, and weight
status.Obesity,22(12), 2485-2488.
doi:10.1002/oby.20891

What was the question the researchers were

trying to answer?
Hypothesis: testable assertion generally involving the
relationship between two or more variables

Research
Questions
and
Hypothese
s

Sample

Methods:
The
Sample

Who the
researchers have
access to
Who the
researchers are
interested in

Who did the researchers study?

The sample matters for generalizability: how well can we
apply this to a larger population?
Small sample sizes or samples restricted to a specific
population (males over the age of 70, Latina middle-aged
women, college students in Utah) may not tell us much
about the general population

Methods:
The
Sample

What are the variables being studied?

How are they defined and measured?
Independent Variable: input variable
that may explain observed variation

Methods:
Variable
Selection

Dependent Variable: outcome being

studied

Operationalization: the process of turning

variables into factors that can be measured

Methods:
Variable
Selection

How the variables are operationalized also affects

what we can infer from the study.
Were the variables objectively measured or
self-reported? Was there coding involved?
Coding: a process of categorizing data in order to
analyze it

Methods:
Variable
Selection

Control variable: (in observational studies) a

variable held constant to study the effect of the
independent variable on the dependent variable
(all things being equal)

Study design matters because it affects our

ability to infer causation

Randomized Controlled Trials (RCTs)

Longitudinal Studies
Case-control Studies

Common
Study
Designs

Study Types:

Cross-sectional Studies
Experimental*
Ecological Studies
Observational
Systematic Reviews

Multi-study*

Meta-analyses

*Experimental and Multi-study designs

are more useful for inferring causation
than observational studies

Causation or
Correlation

xkcd.com/5
52
Observational studies typically tell us about correlation. They do
not necessarily tell us about causation.

Per capita consumption of high fructose corn

syrup (US) is correlated with pedestrians killed
in collision with car, pick-up truck or van

Spurious
Relationshi
p

Correlation: 0.980767
(http://tylervigen.com/)

Confounding variable: another variable that

correlates with both the dependent and
independent variables
These variables lead to spurious
relationships where it looks like two variables
are causally related because a third one
(related to them both) has been left out

 You dont always have to know why it

may not be causal. Be wary of any
claims of causality.

Jumping
from
correlation
to cause

Some common reasons that a correlation

could look causal when its not, include:
not adjusting for confounders,
misunderstanding the mechanism, having
an unknown confounder.
A causal relationship might be reasonable
to suspect when the statistics are

Source: Rebecca Goldin (2016)

Overwhelming
Observed in many different contexts
Observed over time, with the purported
cause preceding the effect
Repeated studies show the same effect,
on large numbers of people

Study
Design

This study is observational. There is no random

assignment to experience weight discrimination.

 Statistical Significance

Results

A result is reported as statistically

significant if the authors are
asserting that the findings are not
likely to occur if there wasnt a true
relationship in the data (i.e.,
something nonrandom is going on)
P-value: the probability of obtaining a
result equal to or more unusual than the
observed result (assuming there isnt
actually a relationship)
Want to say the probability of getting this
result, if it were random process and not a
relationship generating it, is small
A significance threshold () is set in
advance. Conventionally, the threshold for
statistical significance is set at less than
1 in 20 (p <0.05 ) or 1 in 100 (p<0.01).
(Convincingness of evidence that
something non-random is going on)

Statistical Significance Significance

(Importance)
Statistical Significance Magnitude
of Effect

P-values dont tell the whole story

When p<0.05, there is still a strong probability
of a false discovery
With large datasets, the probability of finding a
statistically significant difference, even when
the effect is trivially small, is greatly increased

The
problem
with P

With small datasets, you are underpowered to

detect an effect when there is one
P-hacking: using data-mining to discover
statistically significant patterns in data before
developing a hypothesis about causality
Multiple Testing Problem: making multiple
statistical inferences at the same time increases
the likelihood of finding some statistically
significant result (can be a particular problem
across studies in a hot field)

Colquhoun, D. (2014). An investigation of the

false discovery rate and the misinterpretation of
p-values.Royal Society Open Science,1(3),
140216-140216. doi:10.1098/rsos.140216
Ioannidis, J. P. A. (2005). Why most published
research findings are false.PLoS Medicine,2(8),
0696-0701. doi:10.1371/journal.pmed.0020124

Statistical Significance Importance

Statistical Significance Magnitude of
Effect
Effect Size

Results

How big is the difference, or strength of

association, youve observed?
Does it matter if there was a difference?
(clinical or practical significance)

Studies dont always explicitly state their effect

sizes or put them in easily-understandable or
practically useful terms.
Difference in the means (if the metric makes
sense)
Cohens d
Odds Ratios
Risk Ratios
Correlations
Coefficient of determination (R 2)

Results

This study reported four statistically significant

results:
There was a statistically significant difference in
weight change between groups for all
participants (p<0.001), overweight
participants (p=0.022), and all obese
participants (p=0.006)
There was a statistically significant difference in
waist change between groups for all
participants (p=0.046)

However, we also have to remember effect size!

Results

1.66 kg=3.66 lbs

1.12 cm= 0.44
inches

The conclusion of an article is perhaps the most

important part of the piece it discusses the
limitations and the implications of the research

Conclusion
s

The conclusion of an article is perhaps the most

important part of the piece it discusses the
limitations and the implications of the research

Conclusion
s

What about
reporting on
research
articles?

Adapted from: Rebecca Goldin (2016)

The File
Drawer
Problem

xkcd.com/8
82

What the
rest of the
world sees

xkcd.com/8

Press Releases
Dont Tell the
Whole Story
Designed to get
attention by the
press

A Press
Release

Present the
results in the
rosiest terms
possible
Dont put the
results in context
of other research
Shy away from
concrete
descriptions

Abstracts Dont
Tell the Whole
Story
How were
subjects
recruited?
What was the
design of the
experiment?
What methods
were used to
analyze the
data?
What are the
weaknesses of
the
conclusions?

Abstract

None of the limitations

mentioned in the conclusion show
up in the abstract

Press
Release

The
popular
press

(Easy)
questions
to ask
while
reading a
journal
article

How were the data collected? Is

there bias? (this is especially important
in survey, opinion, and food and exercise
studies)

Are the results statistically

significant?

Are the results practically or

clinically significant? Results that
claim significance are not necessarily
going to impact our lives that much.

If a study speaks in increased risk,

what is the absolute increased risk?

Why might there be a correlation

without a causal relationship?

(Easy)
questions
to ask
while
reading a
journal
article

What is the context and strength of

the results? The abstract will tell you
the big punch, but context is everything.

How are the people in the study

recruited? In particular, would the
recruitment method itself bias the
results by involving people who might
not be typical with regard to the thing
measured? If the results dont generalize
to a broader population, they may not
be useful.

Who funded the research? This can

be especially important in food and
nutrition-related studies

Additionally, consider the journal.

 How many scientific journals are

there? PubMed is the web interface of
the National Library of Medicine. It has
just under 45,000 titles.

Scientific
Journals

How do I know if a journal is peer

reviewed? Check the journals website.
You can also check on
Ulrich's Periodicals Directory, but this
requires an institutional subscription.
Where are the authors from? Authors
list their affiliations and funding sources
at the beginning of papers.
How do I know if a journal is good?
The typical metric is citations and impact
factor. Science Citation Index (and
websites that mirror it) will give you
rankings of journals by metric (http
://wokinfo.com/ ). Not listed? Not cited!

Scientific
Journals

Impact Factors for Common

Journals:
Nature: 41.456
New England Journal of
Medicine: 55.873
The Lancet : 45.217
JAMA- Journal of the American
Medical Association: 35.289
Cell: 32.242
International Journal of
Obesity: 5.004
Obesity: 3.374

 Whats the process of publication?

Scientific
Journals

Peer reviewed publications go through a

peer-review process. If theres no review,
anything goes.
Peer review can be time-constrained and
weak. Papers are sometimes (and
increasingly) retracted.
Researchers should report their affiliations
and support (industry, government,
conflicts).

Is there value in non-peer reviewed

science at a prestigious conference?
Sure, but there has been no examination
of the quality. A poster can report
extremely preliminary and optimistic
findings.

Beyond checking the journal credential,

review papers give us an idea of the
general scientific consensus around an
issue.
How do you find reviews?
PubMed: enter your keywords and
review

Reviews:
Getting at
the
context

Cochrane Reviews: free systematic

reviews of research in health care and
health policy

Wrapping
it all up

Research is
plagued

What can a
journalist do?

Pressured to publish
Multiple testing

Write about the levels

of significance, bias,
caveats

Lack of
context/repeated
experiments

Ask the researchers

about multiple
testing. Did they
adjust for them?

Scientists dont
know how to talk to
journalists. You can
help by asking good
questions and
reiterating what you
think the person
said/sending quotes
for edits.

Write about effect size

Look for a body of
research rather than
one specific paper
Cite your sources!
Dont indicate
causation when
correlation has been
shown!

Basic
Advice for
a
Journalist
with
Limited
Time

If you have 30
minutes

If you have 60
minutes

Read the summary, the

abstract, and the
conclusion. Many writers
ignore conclusions.

Check the methods. Who

was studied? What was
measured?

The abstract will tell you

the result. It hardly ever
hints at limitations.
The conclusion or
discussion often shares
caveats. The conclusion is
typically at the end of the
paper.
Avoid most conclusions of
causality. Read as a skeptic
at all times.
Look to quantifying the
effect. What is the effect?
What does it mean for most
peoples lives?

Ask the authors questions.

Practical impact? Whats
the gap in knowledge that
remains? Generalizable to
which populations?
Weaknesses of methods?
Get another professional
opinion. What is the
quality of this study? How
does this work fit into the
body of research? Look to
local universities and
professional organizations
in the field to find experts.
Look for reviews and other
research. Has other
literature been published
on the topic? Is there
consensus?

 STATS.org

How to get
help

A collaboration between Sense About

Science USA & the American Statistical
Association
Principal aim is to increase statistical
savvy in the media
Theres a link for journalists to get help on
the website
Advisory board consists of statisticians
who are familiar with the challenges
journalists face they answer quickly!
Typically, within 2 hours.

Call on professional and charitable

organizations, and universities, to find
experts.
Free online statistics courses: Khan
Academy, ITunes U, Coursera, Stanford
Online, etc.

Special thanks to Rebecca Goldin, Ph.D.,

Director of Stats.org, George Mason
University

Thank
You!
Contact Us:
Michele Walsh (mwalsh@email.arizona.edu)
Kara Haberstock Tanoue (
kalynq@email.arizona.edu)

 Sense about Statistics:

http://www.stats.org/
PubMed:
http://www.ncbi.nlm.nih.gov/pubmed/

Helpful
Resources

Cochrane Reviews: http://

community.cochrane.org/cochrane-review
s
Understanding the problem with P: http
://fivethirtyeight.com/features/science-isn
t-broken
/
Interpreting Academic Studies: http://
journalistsresource.org/tip-sheets/resear
ch/interpreting-academic-studies-primer-m
edia

Examining
Ns

Reviews:
Getting at
the
context