Introduction to Simulation - Lecture 6

Krylov-Subspace Matrix Solution Methods

Jacob White

Thanks to Deepak Ramaswamy, Michal Rewienski,

and Karen Veroy
Outline

• General Subspace Minimization Algorithm

– Review orthogonalization and projection formulas
• Generalized Conjugate Residual Algorithm
– Krylov-subspace
– Simplification in the symmetric case.
– Convergence properties
• Eigenvalue and Eigenvector Review
– Norms and Spectral Radius
– Spectral Mapping Theorem
 Arbitrary Subspace
methods

Approach to Approximately Solving Mx=b

⎧ ⎡ w01 ⎤ ⎡ wk −11 ⎤ ⎫
Pick a k- ⎪⎢ ⎥ ⎢ ⎥⎪
dimensional ⎨⎢ ⎥ ,..., ⎢ ⎥ ⎬ ≡ {w0 ,..., wk −1}
Subspace ⎪⎢ w ⎥ ⎢ w ⎥ ⎪
⎩ ⎣ 0N ⎦ ⎣ k −1N ⎦ ⎭
Approximate x as a weighted sum of {w0 ,..., wk −1}
k

k −1
⇒ x =
k
∑α w
i =0
i i

SMA-HPC ©2003 MIT

 Arbitrary Subspace Residual Minimization
methods

The residual is defined as r ≡ b − Mx k k

k −1
If x =k
∑α w
i =0
i i
k −1
⇒ r = b − Mx = b − ∑ α i Mwi
k k

i =0
Residual Minimizing idea: pick α i ' s to minimize
T
⎛ k −1
⎞ ⎛ k −1
⎞
( ) (r ) = ⎜ b − ∑ α i Mwi ⎟ ⎜ b − ∑ α i Mwi ⎟
T
k 2
r ≡ r k k
2
⎝ i =0 ⎠ ⎝ i =0 ⎠
SMA-HPC ©2003 MIT
 Arbitrary Subspace Residual Minimization
methods Computational Approach

k −1 2

Minimizing r k 2
2
= b− ∑ α Mw
i =0
i i is easy if
2

( Mw ) = 0, i ≠ j or ( Mw ) orthogonal to ( Mw )
( Mwi )
T
j i j

Create a set of vectors { p0 ,..., pk −1} such that

span { p0 ,..., pk −1} = span {w0 ,..., wk −1}
and ( Mpi ) ( Mp j ) = 0, i ≠ j
T

SMA-HPC ©2003 MIT

 Arbitrary Subspace Residual Minimization
methods Algorithm Steps

Given M , b and a set of search directions {w0 ,..., wk −1}

1) Generate p j 's by orthogonalizing Mw j ' s

( Mw ) ( Mp ) p
T
j −1
For j = 0 to k − 1 p j = w j − ∑
j i

( Mpi ) ( Mpi )
T i
i =0

2) compute the r minimizing solution x k

x =∑
k
k −1
( r ) ( Mp )
0 T
i
k −1
pi = ∑
( r ) ( Mp )
i T
i
pi
( Mpi ) ( Mpi ) ( Mpi ) ( Mpi )
T T
i =0 i =0

SMA-HPC ©2003 MIT

 Arbitrary Subspace Residual Minimization
methods Algorithm Steps by Picture

1) orthogonalize the Mwi ' s

w
p00 w11
p w
p22 w
p33

2) compute the r minimizing solution x k

M p1
r0

M p0
SMA-HPC ©2003 MIT
 Minimization Algorithm
Arbitrary Subspace
Solution Algorithm

r 0 = b − Ax 0
For j = 0 to k-1
p j = wj
For i = 0 to j-1 Orthogonalize
p j ← p j − ( Mp j ) ( Mpi ) pi
T
Search Direction
1
pj ← pj Normalize
( Mp ) ( Mp )
T
j j

x j +1
= x + (r j
) ( Mp ) p
j T
j j
Update Solution

r j +1
=r j
− ( r ) ( Mp ) M p
j T
j j
Update Residual
SMA-HPC ©2003 MIT
 Arbitrary Subspace Subspace Selection
methods Criteria

Criteria for selecting w0 ,..., wk −1

All that matters is the span {w0 ,..., wk −1}
k −1
∃ α i ' s such that b − Mx k = b − ∑ α Mw
i =0
i i is small
A b ≈ in the span {w0 ,..., wk −1} for k
−1
N
One choice, unit vectors, x k ∈ span {e1 ,..., ek }
Generates the QR algorithm if k=N
Can be terrible if k < N
SMA-HPC ©2003 MIT
 Arbitrary Subspace Subspace Selection
methods Historical Development

1 T
Consider minimizing f ( x ) = x Mx − x b
T

2
Assume M = M (symmetric) and x Mx > 0 (pos. def)
T T

∇ x f ( x ) = Mx − b ⇒ x = M −1b minimizes f

{ ( )
Pick span {w0 ,..., wk −1} = span ∇ x f x ,..., ∇ x f x
0
( k −1
)}
Steepest descent directions for f, but f is not residual
Does not extend to nonsymmetric, non pos def case

SMA-HPC ©2003 MIT

 Arbitrary Subspace Subspace Selection
methods Krylov Subspace

{ ( )
Note: span ∇ x f x 0 ,..., ∇ x f x k −1 ( )} {
= span r 0 ,..., r k −1 }
If: span {w0 ,..., wk −1} = span r ,..., r { 0 k −1
}
k −1
then r k = r 0 − ∑ i
α
i =0
Mr i

{
and span r ,..., r0 k −1
} = span {r , Mr ,..., M
0 0 k −1 0
r }
Krylov Subspace

SMA-HPC ©2003 MIT

 The Generalized Conjugate
Residual Algorithm
Krylov Methods
The kth step of GCR

αk =
( r ) ( Mp )
k T
k Determine optimal stepsize in
( Mpk ) ( Mpk ) kth search direction
T

x k +1 = x k + α k pk Update the solution

r k +1
= r − α k Mpk
k and the residual

pk +1 = r k +1
−∑
k
( Mr ) ( Mp )
k +1 T

p
j
Compute the new
orthogonalized
( Mp ) ( Mp )
T j
j =0
j j search direction
SMA-HPC ©2003 MIT
 The Generalized Conjugate
Residual Algorithm
Krylov Methods
Algorithm Cost for iter k

αk =
( r ) ( Mp )
k T
k Vector inner products, O(n)
( Mpk ) ( Mpk ) Matrix-vector product, O(n) if sparse
T

x k +1 = x k + α k pk
Vector Adds, O(n)
r k +1
= r − α k Mpk
k

pk +1 = r k +1
−∑
( Mr ) ( Mp )
k k +1 T

p
j
O(k) inner products,
( Mp ) ( Mp )
T j
j =0
j j total cost O(nk)
If M is sparse, as k (# of iters) approaches n,
total cost = O (n ) + O (2n ) + .... + O ( kn ) = O (n )
3

Better Converge Fast!

SMA-HPC ©2003 MIT
 The Generalized Conjugate
Residual Algorithm
Krylov Methods
Symmetric Case

An Amazing fact that will not be derived

k +1
If M = M then r ⊥ Mp j < k
T j

p = r −∑
(
k +1
Mr ) ( Mp )
k k +1 T

p ⇒ p =r −
j( Mr ) ( Mp )
p k +1
k +1 T
k

( Mp ) ( Mp )
k +1 k +1
( Mp ) ( Mp )
T j T k
j =0 k k
j j

Orthogonalization in one step

If k (# of iters ) Æ n, then symmetric,
sparse, GCR is O(n2 )
Better Converge Fast!
SMA-HPC ©2003 MIT
 “No-leak Example”
Krylov Methods
Nodal Formulation Insulated bar and Matrix
Incoming Heat

T (1)
T (0)
Near End Far End
Temperature Discretization Temperature

⎡ 2 −1 ⎤
⎢−1 2 ⎥ Nodal
⎢ ⎥
m ⎢ −1⎥ Equation
⎢ ⎥ Form
⎣ −1 2 ⎦

SMA-HPC ©2003 MIT

M
 “No-leak Example”
Krylov Methods
Nodal Formulation Circuit and Matrix

1 2 3 4 m −1 m

⎡ 2 −1 ⎤
⎢−1 2 ⎥
⎢ ⎥ Nodal
m ⎢ Equation
−1⎥
⎢ ⎥ Form
⎣ −1 2 ⎦
SMA-HPC ©2003 MIT M
 “leaky” Example
Krylov Methods
Nodal Formulation Conducting bar and Matrix

T (1)
T (0)
Near End Far End
Temperature Discretization Temperature

⎡2.01 −1 ⎤
⎢ −1 2.01 ⎥ Nodal
⎢ ⎥ Equation
m
⎢ −1 ⎥ Form
⎢ ⎥
⎣ −1 2.01⎦

SMA-HPC ©2003 MIT

M
 “leaky” Example
Krylov Methods
Nodal Formulation Circuit and Matrix

1 2 3 4 m −1 m

⎡2.01 −1 ⎤
⎢ −1 2.01 ⎥ Nodal
m ⎢ ⎥ Equation
⎢ −1 ⎥ Form
⎢ ⎥
⎣ −1 2.01⎦
SMA-HPC ©2003 MIT M
 GCR Performance(Random Rhs)
0
10

R Insulating
E
-1

S 10
Leaky
I
D 10
-2

U
A
L 10
-3

-4
10
0 10 20 30 40 50 60
Iteration
Plot of log(residual) versus Iteration

SMA-HPC ©2003 MIT

 GCR Performance(Rhs = -1,+1,-1,+1….)
0
10

R
E -1
10

S
I -2
10
D
U -3
10
A Insulating
L -4
10

Leaky
-5
10
0 5 10 15 20 25 30 35 40 45 50
Iteration

Plot of log(residual) versus Iteration

SMA-HPC ©2003 MIT
 Krylov Subspace Convergence Analysis
Methods Polynomial Approach

{
If span {w0 ,..., wk } = span r 0 , Mr 0 ,..., M k r 0 }
k
x k +1
= ∑α M r
i =0
i
i 0
= ξk ( M ) r 0

kth order polynomial

k
r k +1
= r − ∑αi M r = ( I − M ξk ( M )) r
0 i +1 0 0

i =0
Note: for any α 0 ≠ 0
1
{
span r , r = r − α 0 Mr
0 0 0
}= { 0
span r , Mr 0
}
SMA-HPC ©2003 MIT
 Convergence Analysis
Krylov Methods
Basic Properties

If α j ≠ 0 for all j ≤ k in GCR, then

0
{
1) span { p0 , p1 ,..., pk } = span r , Mr , ..., M r
0 k 0
}
2) x k +1
= ξ k ( M )r , ξ k is the k order
0 th

k +1 2
polynomial which minimizes r
2
3) r = b − Mx = r − M ξ k ( M )r
k +1 k +1 0 0

= ( I − M ξ k ( M ) ) r ≡℘k +1 ( M ) r
0 0

where ℘k +1 ( M ) r is the ( k + 1) order poly

0 th

minimizing r k +1 2
subject to ℘k +1 ( 0 ) =1
2
SMA-HPC ©2003 MIT
 Convergence Analysis
Krylov Methods
Optimality of GCR poly

GCR Optimality Property

k +1 2 0 2
r ≤ ℘k+1 ( M )r where ℘k+1 is any k order
th
2 2

polynomial such that ℘k+1 ( 0 ) =1

Therefore
Any polynomial which satisfies
k +1 2
the zero constraint can be used r
2
to get an upper bound on
SMA-HPC ©2003 MIT
 Eigenvalues and Basic Definitions
Vectors Review
Eigenvalues and eigenvectors of a matrix M satisfy
eigenvalue
Mui = λi ui
eigenvector
Or, λi is an eigenvalue of M if
M − λi I is singular
ui is an eigenvector of M if
( M − λi I ) ui =0
SMA-HPC ©2003 MIT
 Eigenvalues and Basic Definitions
Vectors Review Examples

⎡ 1 −1 0 0 ⎤
⎡1.1 −1⎤ ⎢ −1 1 0 0 ⎥ Eigenvalues?
⎢ −1 1.1⎥ ⎢ ⎥ Eigenvectors?
⎣ ⎦ ⎢ 0 0 1 −1⎥
⎢ ⎥
⎣ 0 0 −1 2 ⎦

⎡ M 11 0 0 ⎤
⎢M M 22 ⎥
⎢ 21 ⎥ What about a lower
⎢ 0 ⎥ triangular matrix
⎢ ⎥
⎢⎣ M N 1 M NN −1 M NN ⎥⎦
SMA-HPC ©2003 MIT
 Eigenvalues and A Simplifying
Vectors Review Assumption
Almost all NxN matrices have N linearly
independent Eigenvectors
⎡↑ ↑ ↑ ↑⎤
M ⎢
⎢u1 u2 u3
⎥
uN ⎥
⎢↓ ↓ ↓ ↓ ⎥⎦
⎣
⎡ ↑ ↑ ↑ ↑ ⎤
⎢ ⎥
= ⎢λ1u1 λ2u2 λ3u3 λN u N ⎥
⎢ ↓ ↓ ↓ ↓ ⎥⎦
⎣
The set of all eigenvalues of M is known as
the Spectrum of M
SMA-HPC ©2003 MIT
 Eigenvalues and A Simplifying
Vectors Review Assumption Continued
Almost all NxN matrices have N linearly
independent Eigenvectors
⎡λ1 0 0⎤

MU =U ⎢0
⎢
⎢⎣ 0 0
0 ⎥⎥
λN ⎥⎦

U MU = λ or M = U λU
−1 −1

Does NOT imply distinct eigenvalues, λi can equal λ j

Does NOT imply M is nonsingular
SMA-HPC ©2003 MIT
 Eigenvalues and Spectral Radius
Vectors Review
Im ( λ )

λi
Re ( λ )

The spectral Radius of M is the radius of the smallest

circle, centered at the origin, which encloses all of
M’s eigenvalues

SMA-HPC ©2003 MIT

 Eigenvalues and Heat Flow Example
Vectors Review
Incoming Heat

T (1)
T (0)

Unit Length Rod

T1 TN

+ +
- -
vs = T(0) vs = T(1)

SMA-HPC ©2003 MIT

 Eigenvalues and Heat Flow Example
Vectors Review Continued
⎡ 2 −1 0 0 ⎤
⎢ −1 2 0 ⎥
⎢ ⎥
⎢0 −1⎥
⎢ ⎥
⎣ 0 0 − 1 2 ⎦

Eigenvalues N=20

SMA-HPC ©2003 MIT

 Eigenvalues and Heat Flow Example
Vectors Review Continued
Four Eigenvectors – Which ones?

SMA-HPC ©2003 MIT

 Useful Spectral Mapping
Eigenproperties Theorem

Given a polynomial
f ( x ) = a0 + a1 x + … + a p x p

Apply the polynomial to a matrix

f ( M ) = a0 + a1M + … + a p M p

Then
spectrum ( f ( M ) ) = f ( spectrum ( M ) )

SMA-HPC ©2003 MIT

 Useful Spectral Mapping
Eigenproperties Theorem Proof

Note a property of matrix powers

MM = U λU U λU = U λ U
−1 −1 2 −1

⇒ M = Uλ U
p p −1

Apply to the polynomial of the matrix

f ( M ) = a0UU + a1U λU + … + a pU λ U
−1 −1 p −1

Factoring f ( M ) = U ( a0 I + a1λ + … + a p λ p
)U −1

Diagonal

f ( M ) U = U ( a0 I + a1λ + … + a p λ p
)
SMA-HPC ©2003 MIT
 Useful Spectral
Eigenproperties Decomposition

Decompose arbitrary x in eigencomponents

x = α1u1 + α 2u2 + … + α N u N
⎡ α1 ⎤
Compute by solving U ⎢ ⎥ = x ⇒ α = U −1 x
⎢ ⎥
⎢⎣α N ⎥⎦
Applying M to x yeilds
Mx = M (α1u1 + α 2u2 + … + α N u N )
= α1λ1u1 + α 2 λ2u2 + … + α N λN u N
SMA-HPC ©2003 MIT
 Convergence Analysis
Krylov Methods
Important Observations

1) The GCR Algorithm converges to the exact solution

in at most n steps
Proof: Let ℘n ( x ) = ( x − λ1 )( x − λ2 ) ... ( x − λn )
where λi ∈λ ( M ) .
⇒ ℘n ( M ) r 0 = 0 and therefore r n = 0
2) If M has only q distinct eigenvalues, the GCR
Algorithm converges in at most q steps
Proof: Let ℘q ( x ) = ( x − λ1 )( x − λ2 ) ... ( x − λq )
SMA-HPC ©2003 MIT
Summary

• Arbitrary Subspace Algorithm

– Orthogonalization of Search Directions
• Generalized Conjugate Residual Algorithm
– Krylov-subspace
– Simplification in the symmetric case.
– Leaky and insulating examples
• Eigenvalue and Eigenvector Review
– Spectral Mapping Theorem
• GCR limiting Cases
– Q-step guaranteed convergence