Multiple Regression Analysis in Matrix Form

Let

y1 x 10 x 11 x 12 … x 1p
y y2 X x 20 x 21 x 22 … x 2p
= , = ,
(N × 1)
… (N × p + 1 )
… … … … …
yN x N0 x N1 x N2 … x Np

b0 e1
b b1 e e2
= , and = ,
((p + 1) × 1) … (N × 1)
…
bp eN

and then the standard multiple regression equation:

y i = b 0 x i0 + b 1 x i1 + b 2 x i2 + … + b p x ip + e i

can be written as:

y X b
= + e
(N × 1) (N × (p × 1)) ((p + 1 ) × 1) (N × 1)

The optimization problem in regression analysis is

N

∑e
2
Min S = i = e′e = ( y – Xb )′ ( y – Xb )
i=1

= y′y – b′X′y – y′Xb + b′X′Xb

= y′y – 2b′X′y + b′X′b .

S is mimimized by setting the following partial deriviative to 0:
∂S
------ = – 2X′y + 2X′Xb = 0 .
∂b
Rearranging,
X′Xb = X′y , and so
–1
b = ( X′X ) X′y