Frontiers of

Computational Journalism
Columbia Journalism School
Week 6: Drawing Conclusions from Data
October 21, 2016

This class

The Shape of Randomness

Statistical evidence
Causal Models
Analysis of Competing Hypotheses

Interpretation
There may be more than one defensible interpretation
of a data set.
Our goal in this class is to rule out indefensible
interpretations.

The Shape of Randomness

Margin of Error

The probabilities of polling

If Romney is two points ahead of Obama, 49% to 47%, in a poll
with 5.5% margin of error, how likely is it that Obama is
actually leading?

Given:
R = 49%, O=47%
MOE(R) = MOE(O) = 5.5%

How likely is it that Obama is actually ahead?

Let D = R-O = 2%. This is an observed value, and if we polled the
whole population, we would see a true value D'. We want to
know probability that Obama is actually ahead, i.e. P(D' < 0)
Margin of error on D MOE(R) + MOE(D) = 11% because they
are almost completely dependent, R+O 100.
For better analysis, see
http://abcnews.go.com/images/PollingUnit/MOEFranklin.pdf
Gives MOE(D) = 10.8%

P(Obama ahead)

P(Romney ahead)

Std. dev of D MOE(D)/1.96 as MOE is quoted as 95% confidence

interval
= 5.5%.
Z-score of -D = -2%/5.5% = -0.36
P(z<-0.35) = 0.36, so 36% chance a Romney is not ahead, or about 1
in 3.

Which one is random?

One star per box less random

Two principles of randomness

1. Random data has patterns in it way more often than you
think.
2. This problem gets much more extreme when you have less
data.

Is this die loaded?

Are these two dice

loaded?

Two dice: non-uniform distribution

Is something causing cancer?

Cancer rate per county. Darker = greater incidence of cancer.

Which of these is real data?

Global temperature record

How likely is it that the temperature won't increase over next decade?

From The Signal and the Noise, Nate Silver

It is conceivable that the 14 elderly people who are reported to have

died soon after receiving the vaccination died of other causes.
Government officials in charge of the program claim that it is all a
coincidence, and point out that old people drop dead every day. The
American people have even become familiar with a new statistic:
Among every 100,000 people 65 to 75 years old, there will be nine or
ten deaths in every 24-hour period under most normal circumstances.
Even using the official statistic, it is disconcerting that three elderly
people in one clinic in Pittsburgh, all vaccinated within the same hour,
should die within a few hours thereafter. This tragedy could occur by
chance, but the fact remains that it is extremely improbable that such
a group of deaths should take place in such a peculiar cluster by pure
coincidence.
- New York Times editorial, 14 October 1976

Assuming that about 40 percent of elderly Americans were

vaccinated within the first 11 days of the program, then about 9 million
people aged 65 and older would have received the vaccine in early
October 1976. Assuming that there were 5,000 clinics nationwide, this
would have been 164 vaccinations per clinic per day. A person aged
65 or older has about a 1-in-7,000 chance of dying on any particular
day; the odds of at least three such people dying on the same day
from among a group of 164 patients are indeed very long, about
480,000 to one against. However, under our assumptions, there were
55,000 opportunities for this extremely improbable event to occur
5,000 clinics, multiplied by 11 days. The odds of this coincidence
occurring somewhere in America, therefore, were much shorteronly
about 8 to 1
- Nate Silver, The Signal and the Noise, Ch. 7 footnote 20

The Howland Will Trial

Randomization to detect insider trading

Looking at executives' trading in the week before their companies

made news, the Journal found that one of every 33 who dipped in and
out posted average returns of more than 20% (or avoided 20%
downturns) in the following week. By contrast, only one in 117 executives
who traded in an annual pattern did that well.

P-value
p(your data | null hypothesis)
Whats it good for? Whats it bad for?

Statistical Evidence

Evidence
Information that justifies a belief.
Presented with evidence E for X, we should believe X "more."
In terms of probability, P(X|E) > P(X)

A more complete theory

Compare probability of multiple alternatives.

Did the stoplight reduce accidents?

7
0 2 4 6 8

0 2 4 6 8

0 2 4 6 8

0 2 4 6 8

0 2 4 6 8

0 2 4 6 8

0 2 4 6 8

0 2 4 6 8

0 2 4 6 8

Simulated without stoplight

2
3

5
6

8
9

7
0 2 4 6 8

0 2 4 6 8

0 2 4 6 8

0 2 4 6 8

0 2 4 6 8

0 2 4 6 8

0 2 4 6 8

0 2 4 6 8

0 2 4 6 8

Simulated with a 50% effective stoplight

2

Bayes learns from evidence

Pr(H|E) = Pr(E|H) Pr(H) / Pr(E)
or

P(H|E) = Pr(E|H)/Pr(E) * Pr(H)

Posterior probability
How likely is H
given evidence E?

Likelihood
model
Probability of
seeing E
if H is true

Prior model
Evidence model How likely was
How commonly
H to begin with?
do we see E at all?

The Bayesian approach:

probability distribution over hypotheses
E.g. Is the NYPD targeting mosques for stop-and-frisk?
1

0
H0

H1

H2

Never Once or twice Routinely

*Tricky: you have to imagine a hypothesis before you can assign it a

probability.

Parameter Estimation
Computing probability for a continuum of hypotheses
P(|E) = Pr(E|)/Pr(E) * Pr()

Strength of Evidence
Can we find a p-value equivalent?
What about Bayes factor
Pr(H1|E)/Pr(H2|E)
= [Pr(E|H1)Pr(H1)/Pr(E)] / [Pr(E|H2)Pr(H2)/Pr(E)]
= Pr(E|H1)/Pr(E|H2) * Pr(H1)/Pr(H2)
Bayes Factor

Ok, but whats a significant

Bayes Factor?

Bayes Factors, Kass and Raftery

Statistical significance is usually asking the wrong question.

Does the model reproduce the data?

Causal Models

Does chocolate make you smarter?

Does marriage make women safer?

Occupational Group
Farmers, foresters, and fisherman

Smoking

Mortality
77

84

Miners and quarrymen

137

116

Gas, coke and chemical makers

117

123

94

128

Furnace, forge, foundry, and rolling mill

116

155

Electrical and electronics workers

102

101

Engineering and allied trades

111

118

Woodworkers

93

113

Leather workers

88

104

Textile workers

102

88

91

104

Food, drink, and tobacco workers

104

129

Paper and printing workers

107

86

Makers of other products

112

96

Glass and ceramics makers

Clothing workers

How correlation happens

X

Y causes X

X causes Y

Z
X

Z causes X and Y

hidden variable causes X and Y

random chance!

Guns and firearm homicides?

X

if you have a gun, you're going to use it

if it's a dangerous neighborhood, you'll buy a gun

the correlation is due to chance

Beauty and responses

X

telling a woman she's beautiful

makes her respond less
Z
X

if a woman is beautiful,
1) she'll respond less
2) people will tell her that

Beauty is a "confounding variable." The correlation

is real, but you've misunderstood the causal

Beauty and responses

X

telling a woman she's beautiful doesn't work

Z
X

if a woman is beautiful,
1) she'll respond less
2) people will tell her that

Beauty is a "confounding variable." The correlation

is real, but you've misunderstood the causal

What an experiment is:

intervene in a network of causes

Does Facebook news feed cause

people to share links?

Analysis
of Competing Hypotheses

Cognitive biases
Availability heuristic: we use examples that come to mind,
instead of statistics.
Preference for earlier information: what we learn first has a much
greater effect on our judgment.
Memory formation: whatever seems important at the time is what
gets remembered.
Confirmation bias: we seek out and give greater importance to
information that confirms our expectations.

Confirmation bias
Comes in many forms.
...unconsciously filtering information that doesn't fit
expectations.
...not looking for contrary information.
...not imagining the alternatives.

The thing about evidence...

As the amount of information increases, it gets more
likely that some information somewhere supports any
particular hypothesis.
In other words, if you go looking for confirmation, you
will find it. This is not a complete truth-finding method.

Method of competing hypotheses

Start with multiple hypotheses H0, H1, ... HN
(Remember, if you can't imagine it, you can't conclude it!)

Go looking for information that gives you the best ability to

discriminate between hypotheses.
Evidence which supports Hi is much less useful than evidence
which supports Hi much more than Hj, if the goal is to choose
a hypothesis.

Competing hypotheses, formally

Start with multiple hypotheses H0, H1, ... HN

Each is a model of what you'd expect to see P(E|Hi),

with initial probability P(Hi)

For each new piece of evidence, use Bayes' rule to

update probability on all hypotheses.
Inference result is probabilities of different hypotheses
given all evidence
{ P(H0|E), P(H1|E), ... , P(HN|E) }

A more practical approach: Triangulation

A good conclusion is one which is supported by multiple lines of
evidence from multiple methods.
Philosophy ought to imitate the successful sciences in its methods,
so far as to proceed only from tangible premises which can be
subjected to careful scrutiny, and to trust rather to the multitude
and variety of its arguments than to the conclusiveness of any one.
Its reasoning should not form a chain which is no stronger than its
weakest link, > but a cable whose fibers may be ever so slender,
provided they are sufficiently numerous and intimately connected.
-

Charles Sanders Peirce

A difficult example
NYPD performs ~600,000 street stop and frisks per year.
What sorts of conclusions could we draw from this
data? How?

Stop and Frisk Causation

Suppose you take the address of every mosque in
NYC, and discover that there are 15% more stop-andfrisks within 100m of mosques than the overall
average.
Can we conclude that the police are targeting
Muslims?