Вы находитесь на странице: 1из 33

DEFLA TION TECHNIQUES FOR AN IMPLICITLY RE STARTED

ARNOLDI ITERATION

R B LEHOUCQ y AND D C SORENSEN z

Abstract A de ation procedure is

introduced that

is designed to improve the convergence of

an implicitly restarted Arnoldi iteration

for computing a

few eigenvalues of a large matrix As the

iteration progresses the Ritz

value approximations of the

eigenvalues

of A converge at di erent rates

A numerically stable scheme

is

introduced that implicitly

de ates the

converged approximations from

the iteration We present two

forms of implicit de ation The

rst a lo cking operation decouples

converged Ritz values and associated vectors from the active part of the iteration The second

a purging operation removes unwanted but converged Ritz pairs Convergence of the iteration is

improved and a reduction in computational e ort is also achieved The de ation strategies make

it possible to compute multiple or clustered eigenvalues with a single vector restart method A

Block method is not required These schemes are analyzed with respect to numerical stability and

computational results are presented

Key words Arnoldi

method Lanczos method eigenvalues de ation implicit restarting

AMS subject classi cations F G

 

Introduction The Arnoldi method is an e cien t procedure

for approximat

ing

a subset of the eigensystem of a large sparse n n matrix A The

Arnoldi method

is

a

generalization of the Lanczos process and reduces to that method when

the matrix

A

is symmetric After k steps the algorithm produces an upper Hessenberg

matrix H k

of

order k The eigenvalues of this small matrix H k are used to approximate a subset

of

the eigenvalues of the large matrix A The matrix H k is an orthogonal projection

of

A onto a particular Krylov subspace and the eigenvalues of H

k are usually called

Ritz values or Ritz approximations

There are a number of numerical di culties with Arnoldi Lanczos methods

In a variant of this method was developed to overcome these di culties This

technique the Implicitly Restarted Arnoldi iteration ira iteration may be viewed as

a truncation of the standard implicitly shifted qr iteration This connection will be

reviewed during the course of the paper Because of this connection an ira iteration

shares a number of the qr iteration s desirable properties These include the well un

derstood de ation rules of the qr iteration These de ation techniques are extremely

important

with respect to the convergence and stability of the qr iteration De ation

rules have

contributed greatly to the emergence of the practical qr algorithm as the

method of

choice for

computing the eigen system of dense matrices In particular the

de ation rules allow

the qr iteration to compute multiple and clustered eigenvalues

This paper introduces

This iteration is designed

de ation schemes that may be used within an ira iteration

to compute a selected subset of the spectrum of A such as

the k eigenvalues of largest real part We refer to this selected subset as wanted

and the remainder of the spectrum as unwanted As the iteration progresses some

of the Ritz approximations to eigenvalues of A may converge long before the entire

This work was supported in part by ARPA U S Army ORA by the U S Department

of Energy Contracts DE FG f ER and W Eng and by the National Science

Foundation Cooperative agreement CCR

y Mathematics and Computer Science Division Argonne National Laboratory Argonne IL

lehoucq mc s anl gov

z Computational and Applied Mathematics Department Rice University Houston TX

sorensen rice edu

R B LEHOUCQ AND D C SORENSEN

set of wanted eigenvalues has been computed These converged Ritz values may be

part of the wanted or the unwanted portion of the spectrum In either case it is

desirable to de ate

the converged Ritz values and corresponding

Ritz vectors from

the unconverged portion

of the factorization If the converged Ritz

value is wanted it

is

necessary to keep it in

the subsequent Arnoldi factorizations This is called locking

If

the converged Ritz value is unwanted then it must be removed

from the current and

subsequent Arnoldi factorizations This is called purging These

notions will be made

precise during the course of the paper For the moment we note that the advantages

of a numerically stable de ation strategy include

Reduction of the working size of the desired invariant subspace

 

Preventing the e ects of the forward

instability of the Lanczos and qr

algo

rithms

The ability to determine clusters of

nearby eigenvalues without need

for a

block Arnoldi method

The fundamentals of the Arnoldi algorithm are introduced in x

as well as the

determination of Ritz value convergence The ira iteration is reviewed

in x De at

ing within the ira iteration is examined in x The de ation scheme for converged

Ritz values is presented in x The practical issues associated with our de ation

scheme are examined in x

ined in x for dealing with a

These include block generalizations

of the ideas exam

number of Ritz values simultaneously and avoiding the

use of complex arithmetic when a complex conjugate pair

of Ritz values converges

An error analysis

of the de ated process in presented in x

A brief survey of and

comparisons with

other de ation strategies is given in x An interesting connection

with the various algorithms used to

x Numerical results are presented

re order a Schur form of matrix is presented in

in x

Capital and lower case letters denote

matrices and vectors while lower case Greek

letters denote scalars The j th canonical basis vector is denoted by e

j The norms

used are the Euclidean and Frobenius ones denoted by k k and k k F

respectively

The range of a matrix A is denoted by R A

The Arnoldi Factorization Arnoldi s method is an orthogonal projec

tion

method

for approximating a subset of the eigensystem of

a general square matrix

The

method builds step by step an orthogonal basis for the Krylov space

K k A v spanfv Av A k v g

for A generated by the vector v The original algorithm in was designed to

reduce a dense matrix to upper

Hessenberg form However the method only requires

knowledge of A through

matrix

vector products and

its ultimate value as a technique

for approximating a few

eigenvalues of a large sparse

the matrix A is symmetric the procedure reduces to

matrix was soon realized When

the Lanczos method

Over a decade of

research was devoted to understanding and overcoming the nu

merical di culties

of

the

Lanczos method Development of the Arnoldi method

lagged behind due

to the

inordinate

computational and storage requirements associ

ated with the original method when

a large number of steps are

required for conver

gence Not only is more storage required for V k and H k when A is nonsymmetric

but in general more steps are required to compute the desired Ritz value approx

imations An explicitly restarted Arnoldi iteration era iteration was introduced

by Saad

to overcome these di culties The idea is based on similar ones devel

oped for

the Lanczos process by Paige

Cullum and Donath and Golub and

Underwood Karush proposed the

rst example of a re started iteration in

DEFLA TION TECHNIQUES FOR IMPLICIT RESTARTING

After k steps the Arnoldi algorithm computes a truncated factorization

AV k

V k k

H

f k

e

T

k

T

of A R n n into upper Hessenberg form where V V k I k The vector f k is the

k

residual and is orthogonal

to the columns of V k The matrix H k R k k is

an upper

Hessenberg matrix that is

the orthogonal projection of A onto R V k K k A v

The following procedure shows how the factorization is extended from length k

to k p

Algorithm

function V k p H k p f k p Arnoldi A V k H k f k k p

Input AV k V k H k f k e T with V V k I k V f k

k

T

k

T

k

Output AV k p V k p H k p f k p e T p with V p f k p

k

k

p V k p I k p V

k

T

T

For j p

k j kf k j k if k j then stop

v k j f k j j V k j V k j v k j

k

w Av k j

h k j V j w k j v T

T

k

k j w

H k j

j

k j h k j

H

k j e T

k j

k

f k j w V k j h k j v k j k j

If k then V v represents the initial vector In order to ensure that

T

V f k in nite precision arithmetic the above algorithm requires some form of

k

re orthogonalization at step see Chapter of

In exact arithmetic the algorithm continues until f k for some k n All

of the intermediate Hessenberg

matrix is said to be unreduced

matrices H j are unreduced for j k A Hessenberg

if all of its main sub diagonal elements are nonzero

The residual vanishes at

the rst step k such that dimK k A v

k and hence is

guaranteed to vanish for

some k n The following result indicates

when an exact

truncated factorization occurs This is desirable since the

for an invariant subspace and the eigenvalues of H k are a

columns of V k form a basis

subset of those of A

Theorem Let equation de ne a k step Arnoldi factorization of A

with H k unreduced Then f k if and only if v Q

k y where AQ k Q k R k with

Q

T Q k I k and R k an upper quasi triangular matrix of order k

k

Proof See Chapter

of or for a proof based on the Jordan Canonical

form

of or for a proof based on the Jordan Canonical form In Theorem the span of

In Theorem the

span of the k columns of Q k represent an invariant subspace

for A The matrix equation AQ k Q k R k is a partial real Schur decomposition

of order k for A The diagonal blocks of R k contain the eigenvalues of A The

complex conjugate pairs are in blocks of order two and the real eigenvalues are on the

diagonal of R k respectively In particular the theorem gives that if the initial vector

is a linear combination of k linearly independent eigenvectors then the k th

residual

vector

vanishes It is therefore desirable to to devise

a method that forces the

starting

vector

v to lie in the invariant subspace associated with the the wanted eigenvalues

The algorithms of this paper are appropriate when the order of A is so large

that

storage and computational requirements prohibit completion of the algorithm

that

produces V n and H n We also remark that working in nite precision arithmetic

generally removes the

possibility of the computed residual ever vanishing exactly

As the norm of f k decreases the eigenvalues of H k become better approximations

to those of A Experience indicates that kf k k rarely becomes small let alone zero

R B LEHOUCQ AND D C SORENSEN

However as the order of H k increases certain eigenvalues of H k may emerge as excel

lent estimates

to eigenvalues of

A When an eigenvalue H k is su ciently near one of

A

we will say

that convergence

occured Since the interest is in a small subset of the

eigensystem of A alternate criteria that allow

H

k y y where ky k De ne the vector x

Ritz value Then

termination for k n are needed Let

V k y to be a R itz vector and to be

kAV k y V k H k y k kAx x k

T

kf k kje y j

k

indicates that if the last component of an eigenvector for H k is

small the Ritz pair

x is an approximation to an eigenpair of A This pair is exact for a nearby

problem it is easily shown that A E x x with E e T y f k x H The ad

k

vantage

of using the Ritz estimate is to avoid explicit formation of the quantity

AV k y V k y when accessing the numerical accuracy of an approximate eigenpair

Recent work by Chatelin and Frays ee and Godet Thobie suggests that

when A is highly non normal the size of e T y

k

is not an appropriate guide for detecting

convergence If the relative departure from normality de ned by the Henrici num

ber kAA T

A T Ak F kA k F is large the matrix A is considered highly non normal

Assuming

that

A

is diagonalizable a large Henrici number implies that the

basis of

eigenvectors is ill conditioned Bennani and Braconnier compare the use of the

Ritz estimate and direct residual kAx x k in Arnoldi algorithms They suggest

normalizing the Ritz estimate by the norm of A resulting in a stopping criteria based

on the backward error The

turbation A such that the

backward error is de ned as the smallest in norm per

Ritz pair is an eigenpair for A A Scott presents

a lucid account of the many issues involved in determining stopping criteria for the

unsymmetric problem

The Implicitly Restarted Arnoldi Iteration Theorem motivates the

selection of a starting vector that will lead to the construction of an approximate

basis

for the desired invariant subspace of A The best possible starting vector would

be a linear combination of a Schur basis for the desired invariant subspace The

ira iteration iteratively restarts the Arnoldi factorization with the goal of forcing the

starting vector closer and closer to the desired invariant subspace The scheme is

called implicit because the updating of the starting vector is accomplished with an

implicitly shifted qr mechanism on H k This will allow us to update the starting

vector by working with orthogonal matrices that live in R k k rather than in R n n

The iteration starts by extending a length k Arnoldi factorization by p steps

Next p shifted qr steps are performed on H k p

The last p columns of the factor

ization are discarded resulting in a length k factorization The iteration is de

ned by

repeating the above process until convergence

As

an example suppose that p and that k represents the dimension of the

desired invariant subspace Let be a real shift and let H k I QR with Q

orthogonal and R upper triangular matrices respectively Then

A I

V k V k k

H I e

f k

 

A I V k V k QR

A I

V k Q V k Q

RQ

f

k

f

k

e

e

A

V k V k

Q Q RQ I e

f k

T

k

T

k

T

k

T

k

from

Q

Q

DEFLA TION TECHNIQUES FOR IMPLICIT RESTARTING

The matrices are updated via V RQ I and the latter ma

k

V k Q and H

k

trix remains upper Hessenberg since R is upper triangular and Q is upper Hessenberg

However equation

is not

of equation fails to

be an

quite a legitimate Arnoldi factorization The relation

Arnoldi factorization since the matrix f k e T Q has

k

a non zero k th column Partitioning the matrices in the updated equation results in

A V

k

v

k

V k

v

k

k e T k

H

k

k

h

k

f k k e T k

k

where k e T Qe k Equating the rst k columns of gives

k

Qe k and k e T

k

AV

k

V H

k

k

v

k k

k

f k

e

T

k

Performing the update f k f k and noting that V T f it

k

k v k

k

k

follows that equation

is a length k Arnoldi factorization

We now show that the

ira iteration is equivalent to forming the leading portion

of an implicitly shifted qr iteration Note that equations are valid for

k n In particular extending the factorization of equation by n k

steps gives f n and AV n V n H n de nes

a decomposition of A into

upper

Hessenberg form Let Q n R n H n I where Q n

and R n are orthogonal and

upper

triangular matrices of order n respectively Since Q and R are the leading principal

sub matrices of order k for Q n and R n respectively V n Q n R n

e

T R n e e T

relationship

Re follow Post multiplication of equation

e

V k QRe and

with

e exposes the

A I v V k Qe V n Q n e v

where e T Re v V k e and V In words the rst column of the

k

e v

updated k step factorization matrix is the

same as the

rst column of the orthogonal

matrix obtained after a complete qr step

on A with shift Thus the

ira iteration

may be viewed as a truncated version of the standard implicitly shifted

qr iteration

This idea may be extended for up to p shifts One cycle of the iteration is

pictured in Figures Application of the shifts may be performed implicitly

as in the qr algorithm If the shifts are in

complex conjugate pairs then the implicit

double shift can be used to avoid complex arithmetic

Numerous choices are possible for the selection of the p shifts One immediate

choice is to use the p

unwanted eigenvalues

of H k p In exact arithmetic the last p

o

diagonal

elements

example in

equation

of H k p are zero and

the Arnoldi factorization decouples For

when is an eigenvalue of H k The reader is

k

referred to for further information

The number of shifts to apply

at each cycle of the above iteration is problem

dependent At present there is no a priori analysis to guide the selection of p relative

to k The

only formal requirement is that p n k However computational

experience

suggests that p k is preferable If

many problems of the same type are

to be solved experimentation with p for a xed k should be undertaken This usually

decreases the required number of matrix vector operations but increases the work and

storage required to maintain the orthogonal basis vectors The optimal cross over

with respect to CPU time varies and must be determined empirically Lehoucq makes

a connection with subspace iteration in Chapter of There has been considerable

R B LEHOUCQ AND D C SORENSEN

experience with subspace iteration and this

connection may eventually shed light on

how to select p relative to k For example it is well known that performing subspace

iteration

on a subspace of dimension larger than the

number of eigenvalues required

typically

leads to improved convergence rates see the

paper of Du and Scott for

a discussion and further references

Among the several advantages an implicit updating scheme possess are

xed storage requirements

The ability to maintain a prescribed level of orthogonality for the columns of

V

since k is of modest size

Application of the matrix polynomial v A v without need to apply

matrix vector products with A

The incorporation of the well understood numerical and theoretical behavior

of the qr algorithm

This last two points warrant further discussion Quite often

the dominant cost during

Arnoldi iterations are the matrix vector products with A Thus the ira iteration

may result in a substantial reduction in time when building a length k p Arnoldi

factorization The last point is important

general purpose and reliable software for

since it allows the possibility of constructing

the large scale eigenvalue problem

De ation within an IRA iteration As the iteration progresses the Ritz

estimates decrease at di erent rates When a Ritz estimate is small enough the

corresponding Ritz value is said to have converged The converged Ritz value may

be wanted or unwanted In either case a mechanism to de ate the converged Ritz

value from the current factorization is desired Depending

on whether the converged

Ritz value is wanted or not it is useful to de ne two

types of de

ation Before we do

this it will prove helpful to illustrate how de ation

is achieved Suppose that after

m steps of the Arnoldi algorithm we have

A

V V

V V

H

e e

T

j

G

H

T

fe

m

where V R n j

H R j j for j m If is suitably small

then the factor

ization de couples in the sense that a Ritz pair y for H provides

an approximate

T

eigen pair x V y with a Ritz estimate of j e y j Setting to zero splits a nearby

j

problem exactly and setting is called de ation If is suitably small then all

the eigenvalues of H may be regarded as converged Ritz values

Locking If de ation has taken place the column vectors in V are con

sidered locked This means that subsequent implicit restarting is done on the basis

V The sub matrices e ected during implicit restarting are G H and V However

during the phase of the iteration

that extends the Arnoldi factorization from k to

k p steps all of the columns of V V participate just as if no de ation had

occurred This assures that all of the new Arnoldi basis vectors are orthogonalized

against converged Ritz vectors and prevents the introduction of spurious eigenvalues

into the subsequent iteration

T

After de ation equating the last m j columns of results in I V V AV

V H fe m T j Thus de ating V and H from the factorization de nes a new Arnoldi

T

factorization with the matrix I V V A and starting vector V e This equivalence

was noted by Saad page Moreover this provides a means to safely compute

multiple eigenvalues when they are present A block method is not

required if de a

tion and locking are used The concept of locking was introduced by Jennings and

Stewart as a de ation technique for simultaneous iteration

DEFLA TION TECHNIQUES FOR IMPLICIT RESTARTING

Purging If de ation has occurred but some of the de ated Ritz values

are unwanted then a further mechanism purging must be introduced to remove the

unwanted Ritz values and corresponding vectors from the factorization The basic

idea of purging is perhaps best explained with the

case of a single de ated Ritz value

Let j in and equate the rst columns of both sides to obtain

Av v e

V

where v V e and H Equation is an Arnoldi factorization of length

one The Ritz value has Ritz estimate j j

Equating the last m columns of results in

AV

V

G H

V

fe

T

m

Suppose that represents an unwanted

e T and equation would become

Ritz value If A were symmetric then G

A E V V H fe

T

m

where E v V e T

V e

m Arnoldi factorization for a

v T Since kE k equation de nes a length

nearby problem The unwanted Ritz pair v

may be pur ged from the factorization simply by taking V V and H H and

setting G in If A is not symmetric the m matrix

G couples v

to the rest of the basis vectors V This vector may be decoupled using

the

standard

Sylvester equation approach pages Purging then takes place

as in the

symmetric case However the new set of basis vectors must be re orthogonalized in

order to return to an

x including the case

Arnoldi factorization This procedure is developed in x and

of purging several vectors

Complications An immediate question is Do any sub diagonal elements

in the Hessenberg matrix of the factorization become negligible as an ira

iteration progresses Since a cycle of the Arnoldi iteration involves performing a

sequence

of qr steps the question is answered by considering the behavior of the qr

iteration upon upper Hessenberg matrices In exact arithmetic under the assumption

that the Hessenberg matrix is unreduced only the last sub diagonal element may be

come zero when shifting But the other sub diagonal elements may become arbitrarily

small

In

addition in exact arithmetic the purging technique would not be necessary

as the

implicit shift technique would accomplish the removal of the unwanted Ritz

pairs from the leading portion of the iteration For example using the unwanted Ritz

values as shifts accomplishes this removal

Computing in nite precision arithmetic complicates the situation A robust

implementation of

the qr algorithm sets a sub diagonal element to zero if it is in

magnitude less than some prescribed threshold and this technique is also adopted for

de ation This de ation overcomes the technical di culty associated with tiny sub

diagonals and improves the convergence of the ira iteration In addition it may be

impossible to accomplish the removal of the unwanted Ritz values from the leading

portion of the iteration due to the forward instability of the qr algorithm

The phenomena of the forward instability of the tridiagonal

qr iteration

was

initially explored by Parlett and Le They observe that while the

implicitly shifted

qr

iteration is always backward stable there are cases where severe forward instability

can occur It is possible for a qr iteration to result in a computed Hessenberg matrix

R B LEHOUCQ AND D C SORENSEN

with entries that have no signi cant digits

of the Hessenberg matrix that would have

in common with the corresponding entries

been determined in exact arithmetic The

implication is that

the computed sub diagonal entries may

not be reliable indicators

for decoupling the

Arnoldi factorization Le and Parlett s

analysis formally implies

that the computed Hessenberg matrix may lose signi cant digits when the shift used

is nearly an eigenvalue of H and the

last component of

the normalized eigenvector is

small We also mention the work of

Watkins that

investigates the transmission

of the shift during a qr step through H

Since convergence of a Ritz value is predicated upon the associated Ritz estimate

being small using shifts that are near these converged values may force the ira

iteration to undergo forward instability This indicates that it may be impossible to

lter out unwanted eigenvalues with the implicit restarting technique and this is the

motivation for developing both the locking and purging techniques Further details

may be found in Chapter of

De ating Converged Ritz Values During an Arnoldi iteration a

Ritz

value may be near an eigenvalue of A with no small elements appearing on the

sub

diagonal of H k However when a Ritz value converges it is always possible to make an

orthogonal change of basis in which the appropriate sub diagonal of H k is zero The

following result indicates how to exploit the convergence

information available in the

last row of the eigenvector matrix for H k For notational convenience all subscripts

are dropped on the Arnoldi matrices V H and f for the remainder of this section

Lemma L et Hy y where H R k k is an unreduced upper Hessenberg

matrix and R

with ky k Let W be a Householder matrix such that W y e

where sign e T y Then

T

e W e

k

T

k

T

w

where

kw k p je T y j and

k

T

W HW e e

Proof The required Householder matrix has the form

where

where

w T e

je T yj

T y e

k

W

I y e y e T

and sign e

T

e W e

k

T y A direct computation reveals that

T

k

T

w

T y T Estimating

je T yj

k

kwk yj ky e k

je T

yj q je T yj

k

je

T

je T yj

p je T yj

k

establishes the bound on kw k The nal assertion follows from

W

T HW e W T Hy

e

W T y

Wy W T W

DEFLA TION TECHNIQUES FOR IMPLICIT RESTARTING

DEFLA TION TECHNIQUES FOR IMPLICIT RESTARTING Lemma and column indicates that the last row and column

Lemma

and column

indicates that the last row and column of W di er from the last row

of I k by terms of order j e T y j The Ritz estimate will indicate when

k

it is

safe to de ate the corresponding Ritz value Rewriting as

AV W V WW T HW

f e T W

k

and using both and and partitioning we obtain

AV W V W fe k fw

h

H

T

T

T

Equation is not an Arnoldi factorization In order to return to an Arnoldi

factorization the matrix H of order k needs to be returned to upper

form and the term fw T dropped Care must be taken not to disturb the

Hessenberg

matrix f e T

k

and the rst column of W T HW To start the process we compute a Householder

matrix Y such that

Y HY

T

Gg

k e T

k

with e T Y e T The above idea is repeated resulting in Householder matrices

k

k

Y Y Y k that return H to

upper Hessenberg form De ning

Y

Y Y Y k

it follows by the construction of the Y j that e

T

k

Y

e T and

k

Y W HW Y e e

T

T

The process of computing a

Wilkinson discusses similar

similarity transformation as in equation is

not new

techniques in pages Wilkinson references

the work of Feller and Forsythe who appear to be the rst to use elementary

Householder transformations for de ation Problem of page addresses

the

case when working with upper Hessenberg matrices What appears to

be new is

the

application to the Arnoldi factorization for converged Ritz values

Since kfw T Yk kfkkY T w k kf k

Making the updates

kw k the size of kfw T

k remains unchanged

V VWY H Y T W T HW Y w T w T Y

we obtain the relation

T

AV V H f e fw

k

T

A de ated Arnoldi factorization is

term fw T

obtained from equation by discarding the

The following theorem shows that the de ated Arnoldi factorization resulting

from this scheme is an exact k step factorization of a nearby

matrix

Theorem L et an Arnoldi factorization of length k

be given by where

Hy y and p je T y j kf k kAk for some Then there exists a matrix

k

E R n n such that

A E V VH fe

T

k

R B LEHOUCQ AND D C