Co-Variance Analysis:

An Introduction to “SVD”
(Singular Value Decomposition)

It is very useful in the diagnosis of weather and climate to analyze

the structures of co-variance between two fields, making use of the
singular value decomposition of the covariance matrix. This
application is introduced in this talk. We should distinguish
between the specific application and the general mathematical
technique.
The SVD analysis discussed is (somewhat) analogous to EOF
analysis

ICTP 2009 TTA (SVDs) D. Straus 1

 A quick review of the underlying math. Here A is any NxN matrix*, which is not
necessarily symmetric (and in fact in our application will not be). The singular
value decomposition is: T
A = USV
Both U and V are matrices whose columns are vectors that form an ortho-normal
set, and S is a diagonal matrix whose diagonal elements are σi.

T
€U U = I VT V = I
Some general theorems (easy to prove using the above):
AV = US
€ ATU €
= VSU T U = VS
T 2 T 2
A €AV = S V AA U = US
Thus V is the eigenvector of ATA with eigenvectors σi2.
€
(*In general A need not be a square matrix. For simplicity we assume it is.)
ICTP 2009 TTA (SVDs) D. Straus 2

€
We will use component notation also. Using the indices i and j to denote the
coordinates of the unit vectors, and k and l to denote the identity of the unit
vectors, we can see that the conditions on V and U are just orthogonality:

T
∑ ki Vil = δ kl
V
i
(k ) ( l )
∑ i k i l ∑ i Vi = δkl
V V = V
i i

€
∑ kiU il = δ kl
U
i
(k ) (l )
€ ∑U ik
U i l = ∑U U i i
= δ kl
i i

Just as EOFs provided a perfectly good basis set (or set of unit vectors) to use, so
€
do the columns of the matrices U and V.

€ ICTP 2009 TTA (SVDs) D. Straus 3

 Application to Weather and Climate
Consider two fields, such as geopotential height Z and diabatic heating H. (If
you like you can think of H instead as rainfall). Each field has its own unique
grid (not necessarily co-located), and is available at a (common) number of
times (daily, weekly, monthly, seasonally, or even daily for only winter or
summer).
The complete matrix of covariability between the two fields contains a great
deal of information. This matrix can be written as:

H i Z j = Ai j
As before, the single index i (or k) ranges over the entire grid, and the ovarbar
denotes an average over all times. Since we have assumed that A is a square
matrix, we should have the same number of grid points N for both fields; we
will see that this is not a big constraint. It is this matrix to which apply to
singular value decomposition analysis.
€
We note that square of the covariance between the two fields, summed over all
grid points of both fields, is just:
ICTP 2009 TTA (SVDs) D. Straus 4
 SqCov = ∑ ( H i Z j ) ( H i Z j ) = ∑ Ai j Ai j = ∑ A Tj i Ai j
i, j i, j i, j

The squared covariance is thus just the trace of the matrix ATA, which is
equal to the sum of the eigenvalues:

SqCov = ∑ σ k2
k
Thus each eigenvalue contributes separately to the total squared
covariance, just as each eigenvalue of the univariate (single variable)
covariance matrix eigenvalue contributes separately to the total variance
of that single variable.
€ “mode” k contributes the most to the squared covariance.
The leading

ICTP 2009 TTA (SVDs) D. Straus 5

 Introducing Component Notation, the SVD of the matrix looks like:

Aij = H i Z j = ∑U i k S k l VlTj = ∑U i k S kV j k
k,l k

One of the key points is to expand the Z field in terms of the ortho-normal set
of (left singular) vectors in V, and to expand H in terms of the (right singular)
vectors in U. The index k here labels the vector, or “mode”:
€ (k) (k) ) ) (k ) ( k )
Z j = ∑ ck V j k = ∑ c V j
H i = ∑ ckU i k = ∑ c U i
k k k k
In the standard way, the coefficients are obtained by projecting the fields
onto the ortho-normal vectors:
(l )
∑V Z j = ∑ c( k ) ∑ V j( k )V j( l ) = c( l )
j
j €k j

(l ) )(k) (k) (l ) )(l )

∑U i H i = ∑ c ∑U i U i = c
j k i

ICTP 2009 TTA (SVDs) D. Straus 6

Just as PCs provided a new set of coordinates for a single field, so too do the
coefficients:
(α ) )
(α )
c and c
for the pair of fields Z and H.

ICTP 2009 TTA (SVDs) D. Straus 7

Using these expansions, we can show that the two sets of coefficients are bi-
orthogonal:
) (α ) ( β ) (α ) (β ) (α ) (β )
c c = ∑ ∑ V j U i H i Z j = ∑ ∑ V j U i Ai j
i j i j

(β ) (α ) (α ) (α )
= ∑U U S i i = σ δα ,β
i

where we have used the property that AV=US. Note, however, that it is not
true that the coefficients c for various modes are linearly independent:

c(α )c( β ) ≠ 0 if α ≠ β
) (α ) ) ( β )
c c ≠ 0 if α ≠ β

ICTP 2009 TTA (SVDs) D. Straus 8

€
The leading SVD pattern for Z is just given by the first column of the
vector V, while the corresponding leading SVD pattern for H is just
given by the first column of the vector U. These are termed
homogeneous patterns.
Often it is of interest to examine the covariance of the Z field with the
coefficient of a particular mode of the heating field, or the covariance of
the heating field with the coefficient of the height field. These
heterogeneous patterns are given by:
)(l ) (k) (k ) )(l )
Z i c = ∑ Vi c c = Vi( l )σ l
k
) ( k ) (l )
H j c = ∑U c c = U (jl )σ l
(l ) (k)
j
k

€
ICTP 2009 TTA (SVDs) D. Straus 9
Canonical Correlation Analysis: Relationship to SVD Analysis
In the previous development, the input fields Zi and Hj were taken to be the
height and heating variables at specific grid points. However, they could
equal well be taken to be PCs of the respective fields! The set of PCs is
just as good a set of coordinates as the values at the original grid points.
The computation would then be done in the following order:
(1) Compute EOFs and PCs of the Height Field on its grid. Call the ith PC zi
(2) Compute EOFs and PCs of the Heating Field on its grid, the jth PC is hj
(3) All the formulae written down before are still valid, with Z replaced by z,
and H by h.
(4) Note that now the leading SVD pattern for Z, given by the first column of
the vector V, gives the pattern in PC coordinates. The homogeneous grid
point version of the pattern is given as (i denotes grid point, and j the PC
index):
(α ) ( k )
∑V k ei for α = 1
k
ICTP 2009 TTA (SVDs) D. Straus 10
If the SVD analysis is applied to the PC coordinates, the expression

hi z j = Ai j
gives the co-variance of the jth height PC and the ith PC of heating.
If we simply “standardize” each PC (divide each PC by its standard
deviation), then the matrix element above gives the correlation
between €the jth height PC and the ith PC of heating.
Simply following all the remaining steps as discussed before gives you
what is called “Canonical Correlation Analysis (CCA).”
- The only difference between CCA and SVD is that CCA
maximizes correlation, while SVD maximizes co-variance.
- The relationship between CCA and SVD is particularly simple if
a PC representation of the data is used.

ICTP 2009 TTA (SVDs) D. Straus 11

 WARNING!
-While SVD analysis is usually robust, CCA analysis can be very fragile
(terminology due to J. M. Wallace, University of Washington)
- To understand this, note that using PC coordinates, the results of each
analysis depends on the number of PCs (EOFs) retained.

- Robust: Increasing the number of PCs in the representation of each field in

the SVD analysis makes less and less difference to the leading SVD modes.
The analysis finds combinations of PCs that maximize co-variance. Each
additional PC represents less and less variance, and so affects the co-variance
less and less
- Fragile: Increasing the number of PCs in the representation of each field in
CCA can impact the leading modes in an erratic way. CCA finds linear
combinations of PCs for each field that maximize correlation. Adding a large
number of PCs increases the chance that some linear combination will be
highly correlated, even though the variance explained by either may be very
small.
ICTP 2009 TTA (SVDs) D. Straus 12