PCA:

 Numerical  Example  

Rao  Vemuri  
•  Consider  two  a:ributes  X  and  Y  
•  Each  is  sampled  three  ?mes  
! x $ ! $ ! y $ ! $
# 1 & # 1 & # 1 & # −1 &
X = # x2 & = # 0 &; Y = # y2 &=# 1 &
# & # −1 & # & # 0 &
#" x3 &% " % #" y3 &% " %

•  Combine  these  two  into  a  matrix  S  

! x y1 $ ! 1 −1 $
# 1 & # &
S = # x2 y2 & = # 0 1 &
# &
#" x3 y3 &% #" −1 0 &%
•  Compute  the  (un-­‐normalized)  covariance  
matrix,  C  
" 1 −1 % "σ σ %
" 1 0 0 % $ ' " %
2 −1 $ xx xy
T
C = S S =$ ' $ 0 1 '=$ '= '
# −1 1 −1 & $ −1 0 ' # −1 2 & $# σ yx σ yy '
&
# &
•  Compute  the  eigenvalues  and  eigenvectors  of  
C  by  solving   Cu = λu or (C − λ I)u = 0
•  If  all  the  a:ribute  vectors  are  independent,  
then  we  have  M=2  eigenvalues  and  M  =  2  
eigenvectors  
•  Because  the  Correla?on  matrix  is  real  and  
symmetric  all  the  eigenvalues  are  real  and  
posi?ve  
•  Plugging  in  the  numbers,  we  get  
" 2−λ % " u % "
−1 $ 1 ' 0 %
$ ' =$ '
# −1 2 − λ &$# u2 '& # 0 &
•  To  solve  for  the  eigenvalues,  we  use  the  
determinant  of  the  matrix  to  get  a  quadra?c  
equa?on  
2 2
(2 − λ ) −1 = λ − 4λ + 3 = 0 ⇒ λ1 = 3, λ2 =1
•     
•  Note  that  the  lambda’s  are  listed  in  
descending  order  of  magnitude  
•  To  solve  for  the  eigenvectors,  we  simply  
subs?tute  the  two  eigenvalues  into  the  matrix  
equa?on.    
•  It  is  also  general  prac?ce  to  find  the  simplest  
eigenvector  in  each  case  by  normalizing  it  so  
that  the  sum  of  the  squares  of  its  components  
equals  1.  Thus,  we  get:  
 Eigenvectors  
•  For  λ  =  1  
" 1 −1 %" u1 % "
0 %
$ '$ '=$ '
# −1 1 &$# u2 ' # 0 &
&

which  gives  u1  =  u2  

•  For  λ  =  3   " −1 −1 %"$ u1 % " 0 %
'=$
$ ' '
$ '
# −1 −1 &# u2 & # 0 &

which  gives  u1  =  -­‐u2  

 Normalized  Eigenvectors  
•  It  is  customary  to  normalize  the  eigenvectors  
•  The  normalized  eigenvectors  are  
! u $ ! 1 $ ! v $ ! 1 $
# 1
&= 1 # 1
&= 1
# &; # &
# u2 & 2 " −1 % # v2 & 2" 1 %
" % " %

•  Note  that  u  is  the  eigenvector  associated  with  

larger  eigenvalue  
 Principal  Components  
•  Now  we  are  in  a  posi?on  to  compute  the  
principal  components  of  S.    
•  The  principal  components  are  created  by  
mul?plying  the  components  of  each  
eigenvector  by  the  a:ribute  vectors  and  
summing  the  result.    
•  That  is,  for  the  two  principal  components,  P1  
and  P2,  we  can  write  
P  1  =  u1  X  +  u2  Y  ,  and  P  2  =  v1X  +  v2Y.  
 In  Matrix  Nota?on  
•  In  matrix  form,  the  principal  component  
matrix  is  the  product  of  the  a:ribute  matrix  S  
and  the  eigenvector  matrix  U:  
! x y1 $! ! 1 $
# 1 & u1 $ −1
v1 # &! 1 1 $
P = SU = # x2 y2 &# &= 1 # 0 1 &# &
# &#" u2 v2 &% 2 # −1 0 &" −1 1 %
#" x3 y3 &% " %

•  Therefore  
" 2 % " 0 %
1 $ ' 1 $ '
P1 = $ −1 '; P2 = $ 1 '
2 $ −1 ' 2 $ −1 '
# & # &
•  Note  also  that  the  principal  component  matrix  
has  the  property  that  when  it  is  mul?plied  by  
its  transpose  we  recover  the  eigenvalues  in  
diagonal  matrix  form:  
" 2 0 %
T 1 " 2 −1 −1 %$ ' " 3 0 %
P P= $ '$ −1 1 ' = $ '
2 # 0 1 1 &$ ' # 0 1 &
# −1 1 &
•  Because  the  inverse  and  transpose  of  the  
eigenvector  matrix  are  iden?cal,  we  can  write  
PU T = SUU T = SUU −1 = S ⇒
# 2 0 &# # 1 −1 &
1% ( 1 −1 & % (
S = % −1 1 (% (=% 0 1 (
2% ($ 1 1 ' % (
$ −1 −1 ' $ −1 0 '

•  These  columns  are  the  values  of  X  &  Y  

recovered!