Approximate solution of defect

equation
If um is any approximation of u and
rm = f Aum
is its residual, then the residual equation Aem = rm is
equivalent to the original equation: By solving for the
correction em , we obtain the solution u = um + em .
c
of A, such that
If we use, however, an approximation A
c bm
A
e = rm

can be solved more easily, we obtain an iterative process

of the form
c bm
rm = f Aum, A
e = rm, um+1 = um + ebm

(m = 0, 1, 2...)

This process is obviously equivalent to the general iteration

where
c 1
Q = I (A)
A .

c
Vice versa, if Q is given, this yields an approximation A
of A
according to
c
A
= A(I Q)1 .

Scient. Comput. and Sim./SlideNr. 1

Splitting, Preconditioning
An equivalent way of constructing Q is to start with a splitting

Here

c
c m+1
A=A
R, Au
= Rum + f .
c 1
c 1
Q = (A)
R = I (A)
A .

A third, also equivalent, approach is based on the idea

of preconditioning. Here the original equation Au = f is
replaced by an equivalent equation
CAu = Cf
where C is an invertible matrix. C is called a (left)
preconditioner of A. The identification with the above
terminology is by
c 1
(A)
= C.

c 1
In other words, the inverse (A)
of any (invertible)
c
approximation A is a left preconditioner and vice versa.

Richardsons iteration for the preconditioned system (with

= 1)
um+1 = um + C(f Aum ) = (I CA)um + Cf
is equivalent to the general iteration with Q = I CA.

Scient. Comput. and Sim./SlideNr. 2

Two ways of improving the

convergence
of basic iterative methods
For any approximation uih of the solution uh, we denote the
error by eih := uh uih, and the defect (or residual) by
rhi := bh Ah uih
The defect equation

Ah eih = rhi
is equivalent to the original equation, since

uh = uih + eih.
This leads to the procedure

uih rhi = bh Ah uih Aheih = rhi uh = uih + eih .

This procedure is not a meaningful numerical process.
However, if Ah is approximated by a simpler operator
c bi
c
c1
i
exists, ebih in A
A
h eh = rh gives a new
h such that Ah
approximation
i
bi
ui+1
h := uh + eh .

The procedural formulation then looks like

c bi
i+1
i
i
i
uih rhi = bh Ah uih A
h eh = rh uh = uh + ebh .

Scient. Comput. and Sim./SlideNr. 3

Iteration matrix
The iteration operator of this method is given by
c 1
Qh = I h ( A
h ) Ah :

G(h) G(h),

where Ih denotes the identity on G(h). We have

c 1
i
b
b
ui+1
h = Qh uh + sh with sh = (A)h bh

(i = 0, 1, 2, . . .) .

For the errors, it follows that

c 1
i
i
ei+1
h = Q h eh = I h ( A h ) A h eh

(i = 0, 1, 2, . . .)

This represents a general class of iterative schemes. For

example, with
c
A
h = Dh ,

c
the Jacobi scheme is regained; with A
h the lower triangular
part of Ah we obtain Gauss-Seidel.

Scient. Comput. and Sim./SlideNr. 4

1. Coarse grid correction

The idea is to use an appropriate approximation AH of Ah
on a coarser grid H , for instance the grid with mesh size
H = 2h. This means that the defect equation is replaced by

AH : G(H ) G(H ),
exists.

i
AH ebiH = rH
.

dim G(H ) < dim G(h) and AH 1

i
As rH
and ebiH are grid functions on the coarser grid H , we
need two (linear) transfer operators

IhH : G(h) G(H ),

IHh : G(H ) G(h)

IhH is used to restrict rhi to H :

i
rH
:= IhH rhi ,

and IHh is used to interpolate (or prolongate) the correction

ebiH back to h:
ebih := IHh ebiH .

Scient. Comput. and Sim./SlideNr. 5

for Poissons equation

Smoothing iteration: lexicographic Gauss-Seidel method
Coarse grid discretization:

0 1 0
1
AH uH = 2 1 4 1 .

H
0 1 0 H
The simplest example for a restriction operator is the
injection operator
rH (P ) = IhH rh (P ) := rh (P )

for

P H h ,

A fine and a coarse grid with the injection operator are

presented:

Scient. Comput. and Sim./SlideNr. 6

Multigrid components
Prolongation operator:

2h

2h

2h

1 b

2 2h

1 b

2 2h

1 b

4 2h

e (x, y)

h b
I2h
e2h(x, y) =

[e (x, y + h) + eb2h(x, y h)]

[e (x + h, y) + eb2h(x h, y)]

[e (x + h, y + h) + eb2h(x + h, y h)

for
for
for

+e2h (x h, y + h) + eb2h(x h, y h)] for

Scient. Comput. and Sim./SlideNr. 7

 Altogether, one coarse grid correction step (calculating

i
ui+1
h from uh ) proceeds as follows:
Coarse grid correction uih ui+1
h

Compute the defect

Restrict the defect (finetocoarse transfer)
Solve exactly on H
Interpolate the correction
Compute a new approximation

The associated iteration operator is given by

Ih B h A h

with

rhi = fi Ah uih
i
= IhH rhi
rH
i
AH ebiH = rH
ebih = IHh ebiH
i
bi
ui+1
h = u h + eh

Bh = IHh AH 1IhH .

Taken on its own, the coarse grid correction process is of no

use: It is not convergent! We have
Ih IHh AH 1 IhH Ah 1 .

Scient. Comput. and Sim./SlideNr. 8

low frequency components,

visible also on 2h

high frequency components,

not visible on 2h

High frequency components cannot be corrected on a

coarse grid !
Coarse grid correction makes sense, if low frequencies are
dominating the error.
We can decompose the sum into partial sums:
p1
X

k,l=1

k,l k,l =

Xhigh

k,l k,l +

where
Xlow

k,l

k,l

and
Xhigh

k,l

Scient. Comput. and Sim./SlideNr. 9

k,l

p/21
X
k,l=1

p1
X

Xlow

k,l k,l

k,l k,l

k,l
p/2max(k,l)

k,l k,l .

Smoothing effect
Point-wise Gauss-Seidel:
1
n
n+1
n
ui,j = [h2bi,j + un+1
i1,j + ui+1,j + ui,j1 + ui,j+1 ]
4
The effect on the error en = un uh is a local averaging effect:

1
n
n+1
n
eni,j = [en+1
i1,j + ei+1,j + ei,j1 + ei,j+1 ]
4
We have found already
||en+1|| (Q)||en ||, (n ).

Analysis of the smoothing effect requires consideration of

eigenvalues/-vectors of Q, which are closely related to A. Look
at the Fourier expansion of the error:

eh(x, y) =

p1
X

k,l=1

k,l sin kx sin ly =

p1
X

k,l=1

k,l k,l

The fact that this error becomes smooth means that the high
frequency components, i.e.,

k,l sin kx sin ly with k or l large

become small after a few iterations, whereas the low frequency
components

k,l sin kx sin ly with k and l small

hardly change.

Scient. Comput. and Sim./SlideNr. 10

The two grid iteration

Correction scheme
It is necessary to combine the two processes of smoothing
and of coarse grid correction.
Consider a linear problem Ahuh = bh
(1) 1 smoothing steps
on the fine grid:
(2) computation of residuals
on the fine grid:
(3) restriction of residuals
from fine to coarse:
(4) solution of the
coarse grid problem:
(5) prolongation of corrections
from coarse to fine:
(6) adding the corrections to the
current fine grid approximation:
(7) on the fine grid:

on grid

Gh

ub h = S 1 (u0h, bh)
rh := bh Ah ub h
rH := IhH rh
A H eH = r H
eh := IHh eH
ub h = ub h + eh
u1h = S 2 (ub h, bh)

Steps (1) and (7) are pre and postsmoothing,

steps (2)...(6) form the coarse grid correction cycle.

Scient. Comput. and Sim./SlideNr. 11

3. Multigrid
since 1973
Iterative methods like Jacobi and Gauss-Seidel converge
slowly on fine grids, however, they smooth the error uh u

2.5

1.4
1.2

1.5

0.8

0.6
0.4

0.5

0.2

0
35

35

30

30

25
0

20
10

15
15

20

25
0

20

30

15

10

20

5
35

15

10

10
25

25

30

35

1.4
1.2
1
0.8
0.6
0.4
0.2
0

Smooth errors can be

approximated well on
coarser grids (with
much less grid points)

35
30
25
0

20
10

15
15

20

10
25

30

5
35

1.4
1.2
1
0.8
0.6
0.4
0.2
0
35
30
25
0

Multigrid is a O(N )- method !

Scient. Comput. and Sim./SlideNr. 12

20
10

15
15

20

10
25

30

5
35

Multigrid components
Choice of coarse grid
The choice of grid depends on the smoothness of the error.

Grid coarsening is particularly simple for structured grids.

For irregular finite volume/ finite element grids coarse grids
are chosen based on the connections in the matrix. In
this case, it is better to say that coarse matrices are
constructed.
It is possible to determine, based on matrix properties
(M -matrix, for example), where the error will be smooth
and accordingly how to coarsen algebraically (algebraic
multigrid, AMG).

Scient. Comput. and Sim./SlideNr. 13

2. Krylov subspace accleration

Basic Iterative solution Methods
We computed the iterates by the following recursion:
ui+1 = ui + B 1(b Aui ) = ui + B 1ri
Writing out the first steps of such a process we obtain:

u0 ,
u1 = u0 + (B 1r0),
u2 = u1 + (B 1r1) = u0 + B 1r0 + B 1 (b Au0 AB 1 r0),
= u0 + 2B 1r0 B 1 AB 1r0,
...

This implies that



u u + span B r , B A(B r ), . . . , (B A)
i

1 0

1 0
n

i1

1 0

(B r ) .
o

The subspace K i(A; r 0) := span r0, Ar0, . . . , Ai1 r0 is called

the Krylov-space of dimension i corresponding to matrix A
and initial residual r 0 .
ui calculated by a basic iterative method is an element of
u0 + K i(B 1A; M 1 r0).

Scient. Comput. and Sim./SlideNr. 14

Recombination of Iterants
The acceleration of a basic iterative method by iterant
recombination starts from successive approximations
u1h, u2h, . . . , um
h , from previous iterations.
In order to find an improved approximation uh, , we consider
g + 1 latest approximations
a linear combination of the m
g
umi
h , i = 0, , m,
uh, = um
h +

g) with
(assuming m m

f
m
X

i=1

i (umi
um
h
h) ,

i = 1.

For linear equations, the corresponding residual, rh, = fh

Lhuh,, is given by
rh, =

rhm

where rhmi = fh Lhumi

h .

f
m
X

i=1

i (rhmi rhm ) ,

To obtain an improved approximation uh,, parameters i are

determined such that residual rh, is minimized.

Scient. Comput. and Sim./SlideNr. 15

Recombination of Iterants
Minimize rh,, i.e.
||rhm

f
m
X

i=1

i (rhmi rhm )|| ,

with respect to the L2-norm || ||2.

This is a classical defect minimization problem. In principle,

the optimal coefficients i can be determined by a (GramSchmidt) orthonormalization process.
Here, however, it is also possible to solve the system of linear
(normal) equations

2
H
.
..

m
f

2
=
.

..

m
f

where the matrix H = (hik ) is defined by

hik = < rhmi , rhmk > < rhm , rhmi >

g k = 1, . . . , m
g ,
< rhm , rhmk > + < rhm , rhm > i = 1, . . . , m,

with the standard Euclidean inner product < ., . > and

i =< rhm , rhm > < rhm , rhmi > .

The work for solving the minimization problem is small.

Scient. Comput. and Sim./SlideNr. 16

The Chebyshev method

Suppose u1, . . . , uk have been obtained via a basic iterative
method, and we wish to determine coefficients j (k),
j = 0, . . . , k such that
k

y =

k
X

j=0

j (k)uj

is an improvement of uk .

If u0 = . . . = uk = u, then it is reasonable to insist that y k = u.

Hence we require
k
X

j=0

j (k) = 1,

Consider how to choose the j (k) so that the error y k u is

minimized. Since error e(k+1) = Qk e0 where ek = uk u. This
implies that
k

y u=

k
X

j=0

j (k)(u u) =

k
X

j=0

j (k)Qj e0.

Using the 2-norm we look for j (k) such that ky k uk2 is

minimal.

Scient. Comput. and Sim./SlideNr. 17

The Chebyshev method

To simplify this minimization we use the following inequality:
ky k uk2 kpk (Q)k2ku0 uk2
where pk (z) =

k
P

j=0

j (k)z j and pk (1) = 1.

Minimize kpk (Q)k2 for all polynomials satisfying pk (1) = 1.

Assumption that Q is symmetric with eigenvalues i that

satisfy n . . . 1 < 1.
Using these assumptions we see that

kpk (Q)k2 = max |pk (i )| max |pk ()|.

i

<<

So, to make the norm of pk (Q) small we need a polynomial

pk (z) that is small on [, ] subject to the constraint that
pk (1) = 1. This is a minimization problem of polynomials on
the real axis.

Scient. Comput. and Sim./SlideNr. 18

The Chebyshev method

The solution of this problem is obtained by Chebyshev
polynomials. These polynomials cj (z) can be generated by
the following recursion

c0(z) = 1,
c1(z) = z,
cj (z) = 2zcj1 (z) cj2 (z).

These polynomials satisfy |cj (z)| 1 on [1, 1] but grow

rapidly in magnitude outside this interval. As a consequence
the polynomial


z
ck 1 + 2


pk (z) =
1
ck 1 + 2
1
1
satisfies pk (1) = 1, since 1 + 2
= 1 + 2
, and tends to
be small on [, ]. The last property can be explained by the
fact that
z
1 for z [, ] so the
1 1 + 2

numerator is less than 1 in absolute value, whereas the

1
denominator is large in absolute value since 1 + 2
> 1.

Scient. Comput. and Sim./SlideNr. 19

The Chebyshev semi-iterative

method
This leads to
ku u0k2
 .

ky uk2 kpk (Q)k2ku uk2
1
|ck 1 + 2 |
k

Calculation of y k is costly, since all u0, . . . , uk should be kept

in memory. Furthermore, one needs to add k + 1 vectors,
which costs for k 5 more than one matrix vector product.

Using the recursion of the Chebyshev polynomials it is

possible to derive a three term recurrence among the y k .
Vectors y k can be calculated as:
y 0 = u0
solve z 0 from Bz 0 = b Ay 0 then y 1 is given by
2
z0
y 1 = y 0 + 2

solve z k from Bz k = b Ay k then y (k+1) is given by

(k+1)

4 2
=

1
2 ck 1 + 2 k

 y
1
ck+1 1 + 2

(k1)

2
z k +y (k1) .
+
2

The Chebyshev semi-iterative method associated with

By (k+1) = (B A)y k + b.

Scient. Comput. and Sim./SlideNr. 20

The semi-iterative Chebyshev

method
Theory
Note that only 4 vectors are needed in memory and the
extra work consists of the addition of 4 vectors.
Acceleration is effective with good lower and upper bounds
of and . These parameters may be difficult to obtain.
Assumption in deriving the Chebyshev acceleration: the
iteration matrix B 1 (B A) is symmetric. Thus, analysis does
not apply to the SOR iteration matrix B1 (B A). To repair
this Symmetric SOR (SSOR) is proposed. In SSOR one SOR
step is followed by a backward SOR step. In this backward
step the unknowns are updated in reversed order.
Suppose that the matrix B 1 A is symmetric and positive
definite and that the eigenvalues i are ordered as follows
0 < 1 2 . . . n. It is then possible to prove the following
theorem:
If the Chebyshev method is applied and B 1 A is symmetric
positive definite then
r

(B 1A)

1
K2
ku0 uk .
r
ky k uk2 2

2
K2(B 1A) + 1

Scient. Comput. and Sim./SlideNr. 21

The Chebyshev method

Proof Since B 1A = B 1 (B (B A)) = I B 1(B A) = I Q
we see that the eigenvalues satisfy the following relation:
or

i = 1 i

i = 1 i .

This leads to the inequality:

ku u0k2
 .

ky uk2
(1(11 ))
|ck 1 + 2 (11)(1n) |
k

So it remains to estimate the denominator. Note that

1 +
ck

2(1 (1 1))
n + 1
1 +
= ck
= ck

(1 1) (1 n)
n 1
1

The Chebyshev polynomial can also be given by


k
k )

1 (
2
2
z+ z 1 + z z 1 .
ck (z) =
2
This expression can be used to show that

ck
=

1
1
1+ n
1+ n
1

2 1 1
1 n1
n
q

1
1 k
1 1+ n +2 n

2
1 n1

>

Scient. Comput. and Sim./SlideNr. 22

u
u 1+ 1 2
u
n
t

1 n1
q k

1+ n1

q
1 n1

+
1
2

1
=

1
n

1 .
n

The Chebyshev method

The condition number K2(B 1A) is equal to
r

K2(M 1A) 1

ky k uk2 2
r

K2(M 1A) + 1

n
1 .

This leads to

ku0 uk2.

Chebyshev type methods which are applicable to a wider

range of matrices are given in the literature.

Scient. Comput. and Sim./SlideNr. 23