Doing Bayesian

Data

Analysis

A Tutorial with R and BUGS

John K. Kruschke Department

of

Psychological Bloomington,

& Brain Sciences IN

Indiana University

AMSTERDAM

BOSTON

i II IDI I Jtl lU;

LONDON

Nt-VV YORK

OXFORD

PARIS

SAN PIW'.O

SAN l-'RANCISCX)

ELSEVIER

SINGAPORE
an

SYDNEY

TOKYO

Ai .uk'ink: Press is

imprint

of lilscvier

Contents

CHAPTER 1

This Book's 1.1 1.2 1.3

Organization: Read

Me First!

1
1

Real People Can Read This Book

Prerequisites
The Organization of This Book 1.3.1 1.3.2 What Are the Essential Chapters? Where's the

2 3
4

Equivalent

of Traditional Test 4

X in This Book? 1.4

Gimme Feedback

(Be Polite)

6 6

1.5

Acknowledgments

Part 1

Parameters, Probability, Bayes' Rule, andR

Introduction; Models We Believe In
2.1 Models of Observations and Models of Beliefs 2.1.1 2.2 Prior and Posterior Beliefs

The Basics:

CHAPTER 2

9
10 12 12
13 13 13

Three Goals for Inference from Data 2.2.1 2.2.2 2.2.3 Estimation of Parameter Values Prediction of Data Values Model

Comparison

2.3

The R 2.3.1 2.3.2 2.3.3 2.3.4 2.3.5

Programming Language
Getting and Installing R Invoking R and Using the Command Line
A Simple Example of R in Action

14 15 15
16

Getting

Help in R
in R

1? 18 20

Programming

2.4

Exercises

CHAPTER 3

What Is This Stuff Called 3.1

Probability?

23 24
25

The Set of All Possible Events 3.1.1 Coin Flips: Why You Should Care
Outside or Inside the Head

3.2

Probability:
3.2.1

25

Outside the Head:

Long-Run Relative
25 28
.

Frequency 3.2.2 3.2.3 3.3 Inside the Head: Subjective Belief Probabilities Assign Numbers to Possibilities

29

Probability Distributions
3.3.1 3.3.2 Discrete Distributions:

30 Probability Mass
30

Continuous Distributions: Rendezvous with

Density 3.3.3
3.3.4 3.3.5

31
a

Mean and Variance of

Variance
as

Distribution

37 39 40 42
43 44 46

Uncertainty in Beliefs

Highest Density Interval (HDI) Marginal Probability

Conditional Probability

3.4

Two-Way Distributions
3.4.1 3.4.2 3.4.3

Independence
R Code for R Code for

of Attributes

3.5

R Code 3.5.1

47

Figure 3.1 Figure 3.3

47 48 49

3.5.2 3.6

Exercises

CHAPTER 4

Bayes'
4.1

Rule

51 52

Bayes'Rule
4.1.1

Derived from Definitions of Conditional

Probability
4.1.2 4.1.3
Intuited from a

53

Two-Way
as an

Discrete Table
over

54

The Denominator

Integral

Continuous Values 4.2

56 56 59

Applied
4.2.1

to Models and Data

Data Order Invariance An

4.2.2 4.3

Example

with Coin

Flipping

60

The Three Goals of Inference

4.3.1 4.3.2

63
63

Estimation of Parameter Values Prediction of Data Values Model Comparison Why Bayesian Inference Can Be Difficult Bayesian

63
64

4.3.3 4.3.4 4.3.5

67
68

Reasoning in Everyday

Life

4.4

R Code
4.4.1
R Code for

69

Figure

4.1

69
71

4.5

Exercises

Part 2

All the Fundamentals

Inferring
CHAPTER 5

Binomial

Applied to Proportion
77 78
80 81 84 85

Inferring
5.1 5.2

Binomial

Mathematical

Proportion via Exact Analysis

The Likelihood Function: Bernoulli Distribution A Description of Beliefs: The Beta Distribution 5.2.1 5.2.2

Specifying

Beta Prior

The Posterior Beta

5.3

Three Inferential Goals

5.3.1 5.3.2 5.3.3

Estimating

the Binomial

Proportion

85 87 88

Predicting Data
Model Comparison How to Do

5.4 5.5

Summary:
RCode 5.5.1

Bayesian

Inference

90 91

R Code for

Figure 5.2

91 95

5.6

Exercises
a

CHAPTER 6

Inferring
6.1

Binomial

Proportion via Grid

101 102
102 104 106 107 108

Approximation Bayes'
6.2.1 6.3 6.4 6.5 6.6 6.7 Rule for Discrete Values of o
a

6.2

Discrettzing
Estimation

Continuous Prior Density

Examples Using Discretizad Priors

Prediction of Subsequent Data Model

Comparison

Summary
RCode
6.7.1 R Code for

109
109

Figure

6.2 and the Like

109 112

6.8

Exercises via the

CHAPTER 7

Inferring a Binomial Proportion Metropolis Algorithm

7.1 A

117

Simple Case

of the Metropolis
on

Algorithm Metropolis

119

7.1.1

A Politician Stumbles

the

Algorithm
7.1.2
A

119
120

Random Walk

7.1.3 7.1.4 7.1.5 7.2

The

General Properties of Why We Care Why It Works

Random Walk

122 125

126
More

Metropolis Algorithm

Generally

129
131 132

7.2.1 7.2.2 7.3

"Burn-in," Efficiency, and Convergence Terminology: Markov Chain Monte Carlo

From the Sampled Posterior to the Three Goals 7.3.1

7.3.2

132
134 136 137

Estimation Prediction Model Comparison: Estimation of p(D)

7.3.3
7.4

MCMC in BUGS 7.4.1

7.4.2 7.4.3 Parameter Estimation with BUGS BUGS for Prediction BUGS for Model

139
140 143 145 145

Comparison

7.5

Conclusion RCode 7.6.1 R Code for

a Home-Grown

7.6

146
Metropolis
146

Algorithm
7.7 Exercises

150

CHAPTER 8

Inferring Two Binomial Proportions

Gibbs Sampling
8.1

via

155
157

Prior, Likelihood, and Posterior for Two Proportions

The Posterior via Exact Formal The Posterior via Grid

8.2 8.3 8.4

Analysis

159 163 165 165 167 173

Approximation

The Posterior via Markov Chain Monte Carlo 8.4.1

8.4.2

Metropolis Algorithm
Gibbs Sampling
It with BUGS

8.5

Doing
8.5.1

Sampling the

Prior in BUGS

174 Biases?
175

8.6 8.7 8.8

How Different Are the

Underlying

Summary
RCode
8.8.1 R Code for Grid and

177 178

Approximation (Figures 8.1

178 181
184 186

8.2) Metropolis Sampler (Figure 8.3) Sampler (Figure 8.6)

a

8.8.2 8.8.3 8.8.4 8.9

R Code for

R Code for BUGS R Code for

Plotting

Posterior Histogram

Exercises

188

CHAPTER 9

Bernoulli Likelihood with Hierarchical Prior 9.1 A Single Coin from a Single Mint 9.1.1 9.2 Posterior via Grid Approximation Coins from
a

191
192 196

Multiple
9.2.1
9.2.2

Single Mint
Approximation

200
203 206

Posterior via Grid

Posterior via Monte Carlo Sampling Outliers and Shrinkage of Individual Estimates

9.2.3

212

9.2.4 9.2.5 9.3

Case

Study: Therapeutic Touch

Mints

217 219 219

Number of Coins and Flips per Coin

Multiple Coins from Multiple

9.3.1

Independent Mints Dependent

Mints

219
224 227 228 228

9.3.2 9.3.3 9.4 9.5

Individual Differences and Meta-Analysis

Summary
R Code
9.5.1

Code for Analysis of Therapeutic-Touch

Experiment
9.5.2 Code for Analysis of Filtration-Condensation Experiment 9.6 Exercises

228

231 235

CHAPTER 10

Hierarchical 10.1 10.2

Modeling

and Model
as

Comparison
Modeling

241
241 244 244 246

Model Comparison

Hierarchical

Model Comparison in BUGS 10.2.1 10.2.2 10.2.3 A Simple

Example
Example with "Pseudopriors"

A Realistic

Some Practical Advice When

Using
253 254

Transdimensional MCMC with

Pseudopriors 10.3 10.4

Model Comparison and Nested Models Review of Hierarchical Framework for Model

Comparison
10.4.1

256 Methods for MCMC Model 257 258 259

Comparing

Comparison 10.4.2
10.5

Summary and Caveats

Exercises

CHAPTER 11

Null Hypothesis
11.1 11.1.1

Significance Testing

265 267
267

NHST for the Bias of a Coin When the Experimenter Intends to Fix N

11.1.2 11.1.3 11.1.4 11.2

When the Experimenter Intends to Fix Soul

270 272
274 274 275 275

Searching

Bayesian Analysis
NHST Analysis

Prior Knowledge about the Coin 11.2.1 11.2.2

Bayesian Analysis

11.3

Confidence Interval and Interval 11.3.1 11.3.2

Highest Density
277
277 280

NHST Confidence Interval

Bayesian

HDI

11.4

Multiple Comparisons
11.4.1 NHST Correction for Error

281 Experimentwise
282

11.4.2

Just One

Bayesian

Posterior No Matter 284

How You Look at It 11.4.3 How Bayesian Analysis Mitigates False Alarms 11.5 What
11.5.1 11.5.2
a

285

Sampling
Planning

Distribution Is Good For Experiment

286
286

an

Exploring Model Predictions (Posterior

Predictive Check)

287 288

11.6 CHAPTER 12

Exercises to

Bayesian Approaches Hypothesis

12.1 The Estimation
12.1.1

Testing

Point

("Null")
295
296

(Single Prior) Approach

a

Is a Null Value of Credible Values?

Parameter among the

297

12.1.2

Is a Null Value of Credible Values?

Difference among the 297

12.1.3

Region

of Practical

Equivalence (ROPE)

301

12.2

The Model-Comparison (Two-Prior) Approach

12.2.1
12.2.2

303
303
307 310

Are the Biases of Two Coins Equal?

Are Different
or

Groups Equal?

12.3

Estimation
12.3.1

Model

Comparison?

What Is the Probability That the Null Value

Is True?

310 311

12.3.2 12.4

Recommendations

RCode
12.4.1
R Code for

312 Figure 12.5

312
314

12.5

Exercises

CHAPTER 13

Goals, Power, and Sample Size

13.1
The Will to Power 13.1.1 13.1.2 13.1.3 13.1.4 Goals and Obstacles Power

319
320
320

321
323 325

Sample Size Other Expressions of Goals Size for

a

13.2

Sample
13.2.1

Single Coin
to

326
a

When the Goal Is

Exclude

Null Value

326 327 329

13.2.2 13.3
13.4

When the Goal Is Precision

Sample
Power:

Size for Multiple Mints

Prospective, Retrospective,
Power

and 331

Replication
13.4.1

Analysis Requires
of

Verisimilitude 333

of Simulated Data

13.5 13.6

The

Importance
Sample

Planning
a

334 335

RCode
13.6.1 13.6.2 Size for

Single

Coin

335 Mints 338 346

Power and

Sample

Size for

Multiple

13.7

Exercises

Part 3
CHAPTER 14

Applied
14.1

to the Generalized Linear Model 357

358
358

Overview of the Generalized Linear Model

The Generalized Linear Model
14.1.1

(GLM)

Predictor and Predicted Variables Scale Types: Metric, Ordinal, Nominal Linear Function of Predictor
a

14.1.2 14.1.3

359

Single

Metric 361 364

14.1.4

Additive Combination of Metric Predictors Nonadditive Interaction of Metric Predictors

14.1.5

366 368
to the

14.1.6 14.1.7

Nominal Predictors

Linking Combined Predictors

Predicted Probabilistic Prediction

373 377 379

14.1.8 14.1.9

Formal Expression of the GLM

or

14.1.10 Two

More Nominal Variables Predicting 381 383 386

Frequency
14.2 14.3 Cases of the GLM Exercises

CHAPTER 15

Metric Predicted Variable

15.1

on a

Single Group

389
390 392 395

Estimating
Likelihood
15.1.1

the Mean and Precision of a Normal

Solution by Mathematical Analysis

15.1.2 15.1.3

Approximation by
Distribution

MCMC in BUGS

Outliers and Robust Estimation: The t

397

15.1.4

When the Data Are Non-normal:

Transformations 15.2

399 403 405 407 408 409

Repeated Measures and Individual Differences

15.2.1 15.2.2 Hierarchical Model

Implementation in BUGS

15.3
15.4

Summary
RCode 15.4.1

Estimating the Mean and Precision of

Normal Likelihood

409

15.4.2

Repeated Measures: Normal Across and Normal Within 411 414

15.5

Exercises

CHAPTER 16

Metric Predicted Variable with One Metric Predictor

16.1 419 420
422 426 427

Simple Linear Regression

16.1.1
16.1.2 16.1.3

The Hierarchical Model and BUGS Code The Posterior: How Big Is the Slope? Posterior Prediction

16.2 16.3

Outliers and Robust Regression

430

Simple Linear Regression with Repeated

Measures 433 437
437

16.4

Summary
RCode
16.5.1 16.5.2

16.5

Data Generator for Height and Weight

437

BRugs: Robust Linear Regression BRugs: Simple Linear Regression with Repeated Measures

439

16.5.3

442 446

16.6

Exercises

CHAPTER 17

Metric Predicted Variable with Multiple Metric Predictors

17.1

453 454
454

Multiple Linear Regression

17.1.1
17.1.2

The Perils of Correlated Predictors The Model and BUGS Program

458

17.1.3
17.1.4 17.2

The Posterior: How Big Are the Slopes? Posterior Prediction and

460 462

Hyperpriors
Coefficients
17.2.1

Shrinkage

of Regression 463

Informative Priors, Sparse Data, and

Correlated Predictors 17.3

467 469 470 472 476 478

Multiplicative Interaction of Metric Predictors

17.3.1 17.3.2 The Hierarchical Model and BUGS Code

Interpreting

the Posterior

17.4 17.5

Which Predictors Should Be Included? RCode

17.5.1 17.5.2

Multiple Linear Regression Multiple Linear Regression with Hyperprior

on

478

Coefficients

483 488

17.6

Exercises

CHAPTER 18

Metric Predicted Variable with One Nominal Predictor

18.1

491
492 493 495 497

Bayesian Oneway ANOVA

18.1.1
18.1.2 The Hierarchical Prior

Doing

It with R and BUGS

18.1.3

A Worked Example

18.2
18.3

Multiple Comparisons Two-Group Bayesian ANOVA and

the NHSTt Test

502
506 507

18.4

RCode
18.4.1

Bayesian Oneway ANOVA

507 512

18.5 CHAPTER 19

Exercises

Metric Predicted Variable with Multiple Nominal Predictors

19.1

515
Multifactor ANOVA 516 517 519 520 522

Bayesian
19.1.1

Interaction of Nominal Predictors The Hierarchical Prior An Example in R and BUGS

19.1.2 19.1.3 19.1.4 19.1.5

Interpreting the Posterior

Noncrossover Interactions,

Rescaling,

and 528

Homogeneous
19.2

Variances

Repeated Measures, Designs

19.2.1

a.k.a,

Within-Subject
531
533

Why Use

Within-Subject Design? And

Why Not?

19.3

R Code

535

19.3.1 19.4 CHAPTER 20

Bayesian Two-Factor ANOVA

535

Exercises

544 549
550 551 553 553 555 556 556 556 557
561

Dichotomous Predicted Variable 20.1

Logistic Regression
20.1.1 20.1.2 20.1.3 20.1.4 20.1.5 20.1.6 The Model

Doing It in Rand BUGS Interpreting the Posterior

Perils of Correlated Predictors When There Are Few l's in the Data

Hyperprior
ANOVA

Across

Regression Coefficients

20.2 20.3

Interaction of Predictors in

Logistic Regression

Logistic
20.3.1

Within-Subject Designs

20.4 20.5

Summary
RCode 20.5.1 20.5.2

561 562

Logistic Regression Code Logistic

ANOVA Code

562 567

20.6

Exercises

572

CHAPTER 21

Ordinal Predicted Variable

21.1 Ordinal Probit Regression
21.1.1

575
576 576 to Ordinal y 577 578 579 580 581

What the Data Look Like

The

21.1.2
21.1.3 21.1.4

Mapping from Metric x

The Parameters and Their Priors

Standardizing for MCMC Efficiency

Posterior Prediction

21.1.5
21.2

Some Examples
21.2.1

Why Are Some Thresholds Outside

the Data? 584

21.3
21.4

Interaction
Relation to Linear and

588

Logistic Regression

588 589 594

21.5 21.6 CHAPTER 22

RCode Exercises Table

Contingency
22.1 22.1.1 22.1.2

Analysis

597

Poisson Exponential ANOVA What the Data Look Like The Exponential Link Function

598
598 599

22.1.3 22.1.4 22.2

The Poisson Likelihood The Parameters and the Hierarchical Prior

601 603 604

Examples
22.2.1 Credible Intervals
on

Cell Probabilities

605 607 608 616

22.3 22.4 22.5

Log Linear Models for

Exercises

Contingency Tables

R Code for the Poisson Exponential Model

CHAPTER 23

Tools in the Trunk

23.1

619
620 620
621 622

Reporting
23.1.1 23.1.2
23.1.3

Bayesian Analysis
Points

Essential Points

Optional Helpful

Points

23.2 23.3

MCMC Burn-in and Functions for Intervals 23.3.1

R Code for

Thinning

623

Approximating Highest Density

626

Computing Computing

HDI of a Grid 626 HDI of an MCMC 627

Approximation
23.3.2 R Code for

Sample 23.3.3 23.4 R Code for

Computing

HDI of a Function

629

Reparameterization of Probability Distributions

23.4.1 23.4.2

630
631

Examples Reparameterization
of Two Parameters

632

REFERENCES INDEX

633
639