ANOVA Single Factor ANOVA Two Factor ANOVA Normal Linear Model

STAT 3001/7301: Mathematical Statistics

Week 3 - Lecture 8

Fred Roosta

School of Mathematics and Physics

University of Queensland
fred.roosta@uq.edu.au
ANOVA Single Factor ANOVA Two Factor ANOVA Normal Linear Model

Info

Instructor (1st Half): Fred Roosta

Email: fred.roosta@uq.edu.au
Office: 67-548
Consultation :
Mondays 3-4pm
Wednesdays 2-3pm
Email for appointment
Or just drop by and knock! ,
ANOVA Single Factor ANOVA Two Factor ANOVA Normal Linear Model

Linear Regression: Recall

Recall:

Linear regression models best capture relationships between

continuous response-explanatory pairs1 .

1
Although, there are tricks to incorporate other types of data, such as
discrete/categorical, into such models....but these tricks are just that, i.e.,
tricks! and do not naturally arise.
ANOVA Single Factor ANOVA Two Factor ANOVA Normal Linear Model

Analysis of Variance (ANOVA)

ANOVA:

A more suitable/natural way to model relationships between

categorial explanatory and continuous response pairs.

These types of data often arise in factorial experiments. These are

controlled statistical experiments in which the aim is to assess how
a response variable is affected by one or more factors tested at
several levels.
ANOVA Single Factor ANOVA Two Factor ANOVA Normal Linear Model

Analysis of Variance (ANOVA)

Example: We may wish to study how food crop yield depends on

two factors: (1) pesticide, at two levels (yes and no) and (2)
fertilizer, at three levels (low, medium, and high)
ANOVA Single Factor ANOVA Two Factor ANOVA Normal Linear Model

Analysis of Variance (ANOVA)

Single Factor

Multiple Factors
ANOVA Single Factor ANOVA Two Factor ANOVA Normal Linear Model

Single Factor ANOVA

One factor with d levels: so we have d groups of observations
Observations are all independent
Observations in each level are i.i.d normally distributed
Levels can have different means
Levels have the same variance
ANOVA Single Factor ANOVA Two Factor ANOVA Normal Linear Model

Single Factor ANOVA

Definition (Single-Factor ANOVA Model)

Suppose we have d levels, each containing ni
observations/responses, i = 1, . . . d. Then the
observations/responses are modeled as

Yik = µi + ik , k = 1, . . . , ni , i = 1, . . . , d,

i.i.d
where all ik ∼ N (0, σ 2 ).
ANOVA Single Factor ANOVA Two Factor ANOVA Normal Linear Model

Single Factor ANOVA

Sometimes, we may want to introduce a new parameter, µ, which
encapsulates the overall effect, common to all levels. Denoting
αi = µ − µi as the specific effect present in each group/level, we
can write our ANOVA model as “factor effect” formulation
Yik = µ + αi + ik , k = 1, . . . , ni , i = 1, . . . , d.

Often, we define µ as
Pd Pni ! P
d
i=1 k=1 Yik ni µi
µ=E = i=1 ,
n n

in which case, we must have

d
X ni
αi = 0,
n
i=1
Pd
where n = i=1 ni is the total number of observations.
ANOVA Single Factor ANOVA Two Factor ANOVA Normal Linear Model

Single Factor ANOVA

All n observations, can equivalently be written as
X d
Yj = µ + αi 1{Yj is in level i} + j , j = 1, . . . , n,
i=1
i.i.d
and j ∼ N (0, σ 2 ).
     
Y1 1 1 0 ... 0 1
 Y2  1 1 0 ... 0  2 

 ..   ..
 
.. .. .. ..   µ   .. 
  
 .  . . . . .  . 
α1   n 
      
 Yn1  1 1 0 ... 0 1 
Yn +1  = 1  . +
    
0 1 ... 0   ..  n1 +1 

 1  
Yn +2  1 0 1 ... 0 αd n1 +2 
  
 1  
 ..   .. .. .. .. ..  | {z }  .. 
 .  . . . . . β∈R d
 . 
Yn 1 0 0 ... 1 n
| {z } | {z } | {z }
Y∈Rn X∈Rn×d ∈Rn
ANOVA Single Factor ANOVA Two Factor ANOVA Normal Linear Model

Two Factor ANOVA

Factor 1 has d1 levels

Factor 2 has d2 levels
Observations are all independent
Observations in each level are i.i.d normally distributed
Levels can have different means
All d1 × d2 levels have the same variance
ANOVA Single Factor ANOVA Two Factor ANOVA Normal Linear Model

Two Factor ANOVA

Definition (Two-Factor ANOVA Model)

Suppose we have two factors with d1 and d2 levels, and each of the
d1 × d2 levels contains nij observations/responses,
i = 1, . . . d1 , j = 1, . . . d2 . Then the observations/responses are
modeled as

Yijk = µij + ijk , k = 1, . . . , nij , i = 1, . . . , d1 , j = 1, . . . d2 ,

i.i.d
where all ijk ∼ N (0, σ 2 ).
ANOVA Single Factor ANOVA Two Factor ANOVA Normal Linear Model

Two Factor ANOVA

Similarly as before, we can write “factor effects” representation as

Yijk = µ + αi + βj + γij + ijk ,

where

µ: Effect common to all levels

αi : Effect specific to factor 1 at level i

βj : Effect specific to factor 2 at level j

γij : Effect specific to interaction of factor 1 at level i and factor

2 at level j
ANOVA Single Factor ANOVA Two Factor ANOVA Normal Linear Model

Two Factor ANOVA

For the balanced case, where all nij are the same, the default
choices are
d1 X
d2
1 X
µ= µij
d1 d2
i=1 j=1
d2
1 X
αi = µij − µ
d2
j=1
d1
1 X
βj = µij − µ
d1
i=1
γij = µij − µ − αi − βj
ANOVA Single Factor ANOVA Two Factor ANOVA Normal Linear Model

Two Factor ANOVA

For this case, it is easy to see that

d1
X d2
X
αi = βj = 0
i=1 j=1
d1
X d2
X
γij = γij = 0, ∀i, j
i=1 j=1
ANOVA Single Factor ANOVA Two Factor ANOVA Normal Linear Model

Normal Linear Model

Example

     
Y1 1 0 0 ... 0 1
 Y2  1 0 0 ... 0  2 
      
 Y3  1 0 0 ... 0  3 
    µ11  
 Y4  0 1 0 ... 0  µ12   4 
   
  
 Y5  0 1 0 ... 0  µ13   5 
   
 Y6  = 0 1 0 +  6 
  
   ... 0  µ21   
 
 ..   .. .. .. .. ..   . 
 .  . . . . .
µ22    .. 
µ23
     
Y16  0 0 0 ... 1 16 
     
Y17  0 0 0 ... 1  17 
Y18 0 0 0 ... 1 18
ANOVA Single Factor ANOVA Two Factor ANOVA Normal Linear Model

Normal Linear Model

Example (Continued: Factor Effects Model)
 
   µ  
Y1 1 1 0 1 0 0 1 0 0 0 0 0   α1 
 1
 Y2  1 1 0 1 0 0 1 0 0 0 0 0  α   2 
    2  
 Y3  1 1 0 1 0 0 1 0 0 0 0 0   β1   3 
   
  
 Y4  1 1 0 0 1 0 0 1 0 0 0 0   β2   4 
   
  
 Y5  1 1 0 0 1 0 0 1 0 0 0 0   β3   5 
   
 Y6  = 1 + 
  
   1 0 0 1 0 0 1 0 0 0 0 

γ11   6 
 ..   .. .. .. .. .. .. .. .. .. .. .. ..  γ 
   .. 
 .  . . . . . . . . . . . .   12 
  . 
 
  
Y16  1 0 1 0 0 1 0 0 0 0 0 γ
1  13   16 
   
  
Y17  1 0 1 0 0 1 0 0 0 0 0 1 γ21  17 
 
Y18 1 0 1 0 0 1 0 0 0 0 0 1 γ22  18
| {z } | {z } γ23 | {z }
Y X | {z } 
β
ANOVA Single Factor ANOVA Two Factor ANOVA Normal Linear Model

Normal Linear Model

Example (Continued: Factor Effects Model)
   
µ 1 0 0 0 0 0
 α  0 1 0 0 0 0
 1  
 α  0 −1 0 0 0 0 
2
    
 β  0 0 1 0 0 0
 1   µ
 β  0 0 0 1 0 0 α1 
 2  
 
 β3  0 0 −1 −1 0 0  β1 
  
 =  
γ11  0 0 0 0 1 0  β2 

 
γ12  0 0 0 0 0 1  γ11 
   
0 −1 −1 γ12
  
γ13  0 0 0

0 −1 0  | {z }
   
γ21  0 0 0
    β̃
γ22  0 0 0 0 0 −1
γ23 0 0 0 0 1 1
| {z } | {z }
β M
ANOVA Single Factor ANOVA Two Factor ANOVA Normal Linear Model

Normal Linear Model

Example (Continued: Factor Effects Model)

Y = |{z}
XM β̃ + 
X
e
ANOVA Single Factor ANOVA Two Factor ANOVA Normal Linear Model

Normal Linear Model

Definition (Normal Linear Model)

A random n-dimensional vector is said to follow a normal (or
Gaussian) linear model, if

Y = Xβ + ,

where X ∈ Rn×d is the design matrix, β ∈ Rd is the vector of

parameters, and  ∼ N (0, σ 2 I).

Since Y = N (Xβ, σ 2 I), we have that its pdf is

n 1 2
f (y) = (2πσ 2 )− 2 e − 2σ2 ky−Xβk .