Numerical Methods

LINEAR ALGEBRAIC EQUATIONS

3.0Introduction

In the previous section, the value(s) of x that satisfied a single equation, fx  0 was determined. In this
section, the case of determining the values x , x , x , . . . , x
that simultaneously satisfy a set of
equations

f x , x , x , . . . , x
  0

f x , x , x , . . . , x
  0

(3.1)

f
x , x , x , . . . , x
  0

will be dealt with. Such systems can be either linear or nonlinear. This section gives emphasis on the
methods by which linear algebraic equations of the form

a x  a x    a
x
 b

a x  a x    a
x
 b

(3.2)

a
 x  a
 x    a

x
 b

can be solved. In 3.2 all a’s are constant coefficients, the b’s are constants, and n is the number of
equations.

For small number of equations (n  3), linear equations can be solved readily by simple techniques.
However, for four or more equations, solutions become arduous and computers must be utilized. Before
computers, techniques to solve linear algebraic equations were time-consuming and awkward. These
approaches placed a constraint on creativity because the methods were often difficult to implement and
understand. In this section, implementation of computer-aided methods in solving large sets of linear
algebraic equations will be discussed.

3.1 Matrices: A Review

symbol as in equation 3.3. Here, A is the shorthand notation for the matrix and a designates an
Definitions and Notations.Amatrix consists of a rectangular array of elements represented by a single

individual element of the matrix.

A horizontal set of elements is called a row, and a vertical set is called a column. The first subscript i
always designates the number of the row in which the element lies. The second subscript j designates the
column

Linear Algebraic Equations Page 1

Numerical Methods
a a a  a
a a a  a
A   
 
a
 a
 a
 a

(3.3)

The matrix in equation 3.3 has n rows and m columns and is said to have a dimension of n by m (n ! m).

Matrices with row dimension n  1 are called row vectors, while single column matrices are called column
vectors. Matrices where n  m are called square matrices. A 4 ! 4 matrix is depicted as
a a a a$
a a a a$
A  a a  a a $ 

a$ a$ a$ a$$
(3.4)

In matrix A of 3.4, the diagonal consisting of the elements a , a , a and a$$ is termed as the
principal or main diagonal of the matrix.

symmetric matrix is one where a  a for all i's and j’s. For example
There are a number of special forms of square matrices that are important and should be noted. A

5 1 2
A  %1 3 7*
2 7 8
is a 3 ! 3symmetrical matrix. A diagonal matrix is a square matrix where all elements off the main
diagonal are equal to zero, as in

a 0 0 0
0 a 0 0
A   
0 0 a 0
0 0 0 a$$

matrix is a diagonal matrix where all elements on the main diagonal are equal to 1 as in
When there are large blocks of zero’s that are needed to be written, they can be left as blank. An identity

1 0 0 0
I  0 1 0 0

0 0 1 0
0 0 0 1

The symbol I is used to denote the identitymatrix, The identity matrix has properties similar to unity. An
upper triangular matrix is one where all the elements below the main diagonal are zero, as in

Linear Algebraic Equations Page 2

Numerical Methods
a a a a$
0 a a a$
A  
0 0 a a $ 
0 0 0 a$$

A lower triangular matrix is one where all elements above the main diagonal are zero, as in

a 0 0 0
a a 0 0
A    
a  a  a 0
a$ a$ a$ a$$

A banded matrix has all elements equal to zero, with the exception of a band centered on the main
diagonal, as in

a a 0 0
a a a 0
A    
0 a  a a $
0 0 a$ a$$

The above matrix has a bandwidth of 3 and is given a special name – the tridiagonal matrix.

Operations on Matrices. The following are the rules in the operation of matrices.

1. Addition and subtraction. This can be accomplished by adding or subtracting the corresponding
elements of the matrices, as in

c  a  b

d  e / f

(3.5)

for all i's and j’s. Note that both addition and subtraction of matrices are both commutative and
associative.

2. Multiplication by a scalar quantity. The product of a scalar quantity g and a matrix A is obtained
by multiplying every element of A by g, as in
a a a a$ ga ga ga ga$
a a a a$ ga ga ga ga$
D  gA  g a a  a a $   ga  ga  ga ga $ 

a$ a$ a$ a$$ ga$ ga$ ga$ ga$$
(3.6)

3. Multiplication of two matrices. The product of two matrices is represented as C  AB where
the elements of C are defined as

Linear Algebraic Equations Page 3

Numerical Methods

c  4 a5 b5 (3.7)

56

wheren is the column dimension of A and the row dimension of B. The multiplication of two
matrices can be performed only if the first matrix has as many columns as the number of rows of
the second matrix. Figure 3.1 provides an easy way to check whether two matrices can be
multiplied.

Figure 3.1. Checking whether two matrices can be multiplied.

3 1
5 9:,
Example 3.1

With A  %8 6*and B  8 find the products C  AB and D  BA.
7 2
0 4

22 29
Answers:

C  %82 84*butD cannot be evaluated since the number of columns of B is not equal to the
28 8
number of rows of A.

Note that matrix multiplication is associative and distributive but not commutative.

4. Inverse of a matrix. Division of matrix is not a defined operation; however, multiplication of a matrix
by the inverse of another matrix is analogous to division. This is because multiplying a matrix by its
inverse results in an identity matrix, a property of division. The algorithms for computing inverse
matrix will be discussed in future sections.

5. Transpose of the matrix. It involves transforming row elements into column and column elements
into rows. If

Linear Algebraic Equations Page 4

Numerical Methods
a a a a$
a a a a$
A  
a  a  a a $ 
a$ a$ a$ a$$

Then the transpose AT will be

a a a  a$
a a a  a$
AT  a a a a$ 

a$ a$ a $ a$$

designated as trA and is computed as

6. Trace of the matrix. The trace of the matrix is the sum of the elements on its principal diagonal. It is

trA  4 a (3.8)

6

7. Augmentation. A matrix is augmented by the addition of a column or columns to the original matrix.

a a a
For example if

A  %a a a *

a  a  a

is to be augmented with a 3 ! 3 identity matrix I, this will result to

a a a 1 0 0
@
A|I  %a a a A 0 1 0*
a  a  a 0 0 1
Representing Linear Algebraic Equations in Matrix Form. Matrices provide a concise notation for
representing simultaneous linear equations. The system of equations of 3.2

a x  a x    a
x
 b

a x  a x    a
x
 b

(3.2)

a
 x  a
 x    a

x
 b

can be represented by

AX  B (3.9)

where the matrix A holds the coefficients

Linear Algebraic Equations Page 5

Numerical Methods
a a a  a

a a a  a

A   
 
a
 a
 a
 a

matrixB is a column vector of the constants

BT  b b b  b


and matrix X is a column vector of the unknowns

XT  x x x  x


In some cases, the matrix A will be augmented by matrix B, as in

a a a  a
b
a a a  a
b
A|B  @ 

@ @
C 
a
 a
 a
 a

3.2. Gauss Elimination

This section deals with simultaneous linear algebraic equations that can be generally expressed as in
Equation 3.2. The technique described here is called Gauss elimination because it involves combining
equations to eliminate unknowns.

that are appropriate for solving small (n  3) sets simultaneous equations that can be done without the aid
Solving Small Numbers of Equations. Before proceeding with computer-aided methods, several methods

of a computer will be described first. These are the graphical method, Cramer’s rule and the elimination of
the unknowns.

Graphical Method. For two equations with two unknowns, a graphical solution is obtained by plotting each
of them on a Cartesian plane and looking for their intersection.

Example 3.2

3x  2x  18
Use graphical method to solve

/x  2x  2

Excel can be used to plot the lines and points out that their intersection is x  4and x  3.
Answer:

Linear Algebraic Equations Page 6

Numerical Methods

10

5 Equation 1, x2

4 Equation 2, x2

0
0 1 2 3 4 5 6

For three simultaneous equations, each graph is a plane in a three-dimensional coordinate system and the
point where these planes intersect yields the solution of the simultaneous equations. Beyond this, the
method breaks down and has little practical value in solving simultaneous equations. The graphical method
however provides some insight as to how the systems and their solutions behave. Figure 3.2 illustrates the
point.

Figure 3.2. Graphical depiction of singular and ill-conditioned system: (a) no solution (b) infinite solution (c) ill-conditioned
system.

Linear Algebraic Equations Page 7

Numerical Methods

Determinants and Cramer’s Rule. The Cramer’s rule is another solution technique that is best suited to
small numbers of equations. Before proceeding however, the concept of determinants is introduced first.
For a three sets of equations

AX  B (3.9)

The matrix A is defined as

a a a
A  %a a a *
a  a  a

from the matrix A, as in

There is a scalar quantity that is associated with this number, called determinants, which can be formed

a a a
Aa
D   a a A
a  a  a

a a a a a a

The value of this determinant is evaluated using expansion by minors

D  a Da a D / a Da  a D  a Da  a  D


The three 2 ! 2 matrices are called minors and can be evaluated, as in the case of the first minor, as

D  a a / a a 

Example 3.3
Compute the values of the determinants of the systems given in Example 3.2 and Figures 3.2a, b and c.

For the system of Example 3.2, D  8.

Answers:

For the system of Figures 3.2a and b, D  0.

For the system of Figure 3.2c, D  /0.04

Example 3.3 showed that if a system has zero determinants, it is a singular system; hence it can either
have no solution (Figure 3.2a) or infinite number of solutions (Figure 3.2b). If the determinant is close to
zero, it can become an ill-conditioned system.

Using Cramer’s rule, x can be computed as

Linear Algebraic Equations Page 8

Numerical Methods

b a a
Ab a a A
b a  a
x 
(3.10)
D
Notice that the coefficients of x has been replaced by the constants B. The other unknowns can be
solved in a similar manner.

Example 3.4

0.3x  0.52x  x  /0.01

Use Cramer’s rule to solve

0.5x  x  1.9x  0.67

0.1x  0.3x  0.5x  /0.44

x  /14.9, x  /29.5, x  19.8

Answer:

The Elimination of Unknowns. The elimination of unknowns by combining equations is an algebraic

approach that can be illustrated for a set of two equations

a x  a x  b
a x  a x  b
(3.11)

The variable x in 3.11 can be eliminated by multiplying the first equation by a , multiplying the second
equation by a and subtracting the results. This will yield an equation which can be used to solve for x ,
as in

a b / a b
x 
a a / a a
(3.12)

This can now be used to solve for x by substitution to 3.11 as

a b / a b
x 
a a / a a
(3.13)

Note that these forms can be directly derived from the Cramer’s rule.

Example 3.5

3x  2x  18
Use elimination of unknowns to solve the system of equations given in example 3.2

/x  2x  2

The unknowns are x  4and x  3, which is consistent with the graphical solution of example 3.2.
Answer:

Linear Algebraic Equations Page 9

Numerical Methods

This method can be extended to systems with more than two or three equations. However, the numerous
calculations that are required for larger systems make the method extremely tedious to implement by hand.
Numerical methods that can be easily implemented in a computer program is described next.

Naive Gauss Elimination. The Gauss elimination is a systematic technique of forward elimination and
back substitution as described previously. Although these techniques are ideally suited for implementation
on computers, some modifications are required to avoid pitfalls, such as division by zero. The “naive”
Gauss elimination does not avoid this problem. In the next sections, these modifications will be discussed.

The Gauss elimination consists of two steps: forward elimination of unknowns and obtaining solutions
through back substitution. Figure 3.3 summarizes each step for a set of three equations with three
unknowns.

Figure 3.3. The two steps of naive Gauss elimination.

However, the steps presented can be generally applied to any set of n equations with n unknowns.
The foregoing discussions will use a set of three equations with three unknowns to detail each step.

system as shown in figure 3.3. The initial step will be to eliminate the first unknown, x from the second
Forward Elimination of Unknowns. The first phase is to reduce the set of equations to an upper triangular

through 3rd equation. Thus for

a a a b
%@a a a A b *
a  a  a b

the second row is to be generated by the formula

Linear Algebraic Equations Page 10

Numerical Methods

a
aE  a / a
a 
(3.14a)

This has the effect of making a  0. The element b becomes

a
bE  b / b
a 
(3.14b)

The system now becomes

a a a b
F@ 0 aE aE A bE G
a  a  a b

The next to be replaced is the third row, using the formula

a 
aE   a  / a
a 
(3.15a)

This will make the element a   0. The element b becomes

a 
bE  b / b
a (3.15b)

Replacing the third row with the new values makes the system

a a a b
F@ 0 aE aE A bE G
0 aE  aE bE

To make the matrix an upper triangular one, aE  must be made zero, and consequently, the third row must
be replaced using the formula

aE  E
aEE   aE  / a
aE 
(3.16a)

The element bE becomes

aE  E
bEE  bE / E b
a
(3.16b)

Thus the above system becomes

Linear Algebraic Equations Page 11

Numerical Methods

a a a b
F@ 0 aE aE A bE G
0 0 aEE bEE

and the first step is complete. For systems having larger number of equations and unknowns, the steps are
repeated until the resulting matrix becomes an upper triangular one.

Back Substitution. Since the matrix A is now modified as an upper triangular matrix AE , the system of
equations can now be written as

AE X  B

or

a x  a x  a x  b
a H x  aE x  bE
aEE x  bEE

which can be readily solved for x which is

aEE
x 
bEE
(3.17)

Substituting this to the second equation, x can be solved by

bE / aE x
x 
aEE
(3.18)

andx as

b / a x / a x
x 
a
(3.19)

Example 3.6

3x / 0.1x / 0.2x  7.85

Use Gauss elimination to solve the system of equations

0.1x  7x / 0.3x  /19.3

0.3x / 0.2x  10x  71.4
for x , x and x . Use six decimal places in the computations.

x  3, x  /2.5and x  7.
Answer:

Linear Algebraic Equations Page 12

Numerical Methods

Figure 3.4 shows the pseudocode that implements the naive Gauss elimination.

Figure 3.4. Pseudocode for Gauss elimination.

Operation Counting. The execution time of Gauss elimination depends on the amount of floating-point
operations or flops involved in the algorithm. On modern computers using math coprocessors, the time
consumed to perform addition/subtraction and multiplication/division is about the same. Therefore, totaling

computation time increases as n gets larger.

up these operations provides insights into which parts of the algorithm are most time consuming and how

on the pseudocode shown in Figure 3.4, the forward elimination part consumes 2n /3  On 
The Gauss elimination algorithm covers two operations: forward elimination and back substitution. Based

operations, where n is the number of equations and On  are terms that are a function of n and lower.
As n gets larger, the latter terms becomes negligible. Thus it would be safe to approximate that the number
of computational effort of forward elimination part of the Gauss elimination algorithm is 2n /3.

For back substitution, the number of operations based on the pseudocode of figure 3.4 is n  On,
where On are terms that are a function of n and lower. The total effort in naive Gauss elimination is

2n
 O  n   n  O  n
3
(3.20)

Linear Algebraic Equations Page 13

Numerical Methods

which approaches 2n /3  On  as n gets larger. Thus, the computational effort of Gauss elimination
comes from the forward elimination part. Thus, efforts to make the method more efficient should focus on
this step.

Pitfalls of Elimination Methods.Whereas there are many systems of equations that can be solved with
naive Gauss elimination, there are some pitfalls that must be explored before writing a general computer
program to implement the method. These are

1. Division by zero. The naive Gauss elimination uses normalization to eliminate variables. The
coefficient used to normalize a row is called the pivot element. However, if plainly used, the
possibility of division by zero exists, as in the set of equations

2x  3x  8
4x  6x  7x  /3
2x  x  6x  5

normalizing the first row with the pivot will result in the division by zero. Problems can also arise
when a coefficient is very close to zero. The technique of pivoting has been developed to partially
avoid these problems.

2. Round-off errors. Since the process involves large amount of computations, errors can propagate
through and round-off errors come into play. A check of whether the solutions satisfy the system of
equations can be helpful to check whether a substantial error has occurred.

3. Ill-conditioned systems. Well-conditioned systems are those where a small change in one or more
coefficients results in a similar small change in the solutions. Ill-conditioned systems are those
where small changes in coefficients result in large changes in the solutions. Ill-conditioned systems
can have a wide range of solutions.

Example 3.7
Solve the following system

and then repeat the calculations but with slightly modified coefficient of x on the second equation to 1.05.

For the original system, x  4and x  3. However, a slight change of coefficient leads to the solutions
Answer:

x  8and x  1. This system is an ill-conditioned system.

Ill-conditioned systems have determinants close to zero. However, there is no general rule as to
how the determinants of a system be close to zero to lead to ill-conditioning. Also, since multiplying

Linear Algebraic Equations Page 14

Numerical Methods

the system by a scalar constant changes the determinants but not the solution, this can lead to a
value of determinant that will not categorize a system as an ill-conditioned system.

Example 3.8
Evaluate the determinants of the following systems and comment on their conditioning.

and

One way to better see the relationship between system condition and the value of the determinant
is by scaling the system; that is normalizing each row so that the maximum coefficient in each row
is unity.

4. Singular systems. If the system is ill-conditioned, then there are set of equations that are almost
identical. There will also be the case that there are set of equations that are identical, which can
arise especially when working with large set of equations. Thus, degrees of freedom are lost as
one is forced to eliminate equations that are identical to another one. These systems are called
singular systems. Singular systems are characterized by a zero determinant.

Techniques for Improving Solutions.

1. Use of more significant figures. An obvious remedy for ill-conditioning and round-off error.
However, memory requirements of the programs are increased when using extended precision.

2. Pivoting. Before normalizing, it is advantageous to determine the largest available coefficient in the
column below the pivot element. The rows can then be switched so that the largest element is the
pivot element. This process is called partial pivoting. Complete pivoting involves search in the
columns as well as rows for the largest element and made to become the pivot element, however
such method puts too much complication in the algorithm and thus is rarely used. Aside from
avoiding division by zero, partial pivoting also minimizes round-off errors and a partial remedy for
ill-conditioning.

Example 3.9
Use Gauss elimination with and without partial pivoting to solve the system of equations

Compare the results using 3, 4 5, 6 and 7 significant figures in the computations. Note that the exact

Linear Algebraic Equations Page 15

Numerical Methods

solutions are x  1/3and x  2/3.

Answers:
Without conditioning, the following table summarizes the results as well as the error in each case.

But with conditioning, the results seem to improve.

General-purpose computer programs for Gauss elimination must include pivoting strategy. Figure
3.5 presents a pseudocode for partial pivoting. In this pseudocode, the rows are physically
interchanged, that is when the algorithm has searched for a larger pivot element, it switches the
rows using the second loop. For large sets, this can be time-consuming and partial pivoting can be
done in-place.

3. Scaling. Scaling minimizes round-off errors especially with systems having wide range of values.
However, scaling itself can sometimes be a source of round-off error itself. It is sometimes good to
use scaling to identify whether the system needs pivoting.

Example 3.10
For the following set of equations

do the following:

Linear Algebraic Equations Page 16

Numerical Methods

(a) solve the system using Gauss elimination.

(b) solve the system by scaling the system first and then determine if it is necessary to use pivoting.
(c) solve the system by using the original coefficients but use pivoting if necessary based on scaled

For all cases, use three significant figures. The correct answers are x  x  1.00.
coefficients.

Gauss-Jordan Elimination.The Gauss-Jordan method is a variation of Gauss elimination. The major

difference is that when an unknown variable is eliminated in the Gauss-Jordan method, it is eliminated from
all other equations rather than the subsequent ones. In addition, all rows are normalized by dividing them
with their pivot elements. Thus, the elimination process reduces the system matrix into an identity matrix
with the constants adjusted. Consequently, it is not necessary to employ back substitution as the constants
column already gives the solutions for the unknowns.

Example 3.11

3x / 0.1x / 0.2x  7.85

Use Gauss-Jordan to solve the system of example 3.6, as in

0.1x  7x / 0.3x  /19.3

0.3x / 0.2x  10x  71.4

n  On ) thus Gauss elimination is still the simplest elimination technique available.
The Gauss-Jordan technique requires more computational effort than the Gauss elimination (around

3.3. LU Decomposition

As described in the previous section, the Gauss elimination is designed to solve systems of liner algebraic
equations

AX  B (3.21)

same system matrix A but with different right-hand side constants B. Gauss elimination has most of its
Although it provides a systematic method of solving such systems, it becomes inefficient for equations with

computational effort in the forward elimination phase, and thus LU decomposition methods have been
developed to alleviate this problem of Gauss elimination.

Basically, LU decomposition separates the time-consuming elimination of matrix A from the
manipulations of the right-hand side B. Thus, once A has been decomposed, multiple right-hand side
vectors can be evaluated in an efficient manner.

Overview of LU Decomposition. In the following discussions, a naive version of the LU decomposition will
be presented. Just like Gauss elimination, pivoting can be used to avoid division by zero, but will be
addressed in a later section.

Linear Algebraic Equations Page 17

Numerical Methods

LU decomposition requires the following steps:

1. LU decomposition. A is factored or decomposed into a lower L and upper U triangular
matrices. These matrices has the property that LU  A.
2. Substitution step. This consists of two steps: forward substitution to generate a matrix D and
backward substitution to solve for X.

Figure 3.5 presents the steps of LU decomposition.

Figure 3.5. Steps of LU decomposition.

Example 3.12

3x / 0.1x / 0.2x  7.85

Use LU decomposition to solve the system of equation given in Examples 3.6 and 3.11, as in

0.1x  7x / 0.3x  /19.3

0.3x / 0.2x  10x  71.4

Figures 3.6 and 3.7 presents the pseudocode for the decomposition and substitution phases of LU
decomposition, respectively.

Linear Algebraic Equations Page 18

Numerical Methods

Figure 3.6. Decomposition phase.

Figure 3.7. Substitution phase of LU decomposition.

elimination phase of Gauss elimination (  On), but has the same flops as that of the backward
The forward elimination phase of LU decomposition requires less computational effort than the forward

M

substitution phase (  On ).

M

Matrix Inversion. Recall that the inverse of the square matrix A, denoted by AN , has the property
that, when multiplied to the original matrix results in an identity matrix I. In this section, computation of
such inverse will be demonstrated using the LU decomposition.

To calculate the inverse using LU decomposition, a right-hand side column vector constant B consisting
of one’s and zero’s as elements will be used. For a 3 ! 3 matrix, if

Linear Algebraic Equations Page 19

Numerical Methods

1
 B   %0 *
0
the resulting X is the first column of the inverse. If

1
 B   %0 *
0
the resulting X is the second column of the inverse, and so on. The following example demonstrates the
method.

Example 3.13

3 /0.1 /0.2
Employ LU decomposition to determine the matrix inverse of

A  %0.1 7 /0.3*

0.3 /0.2 10

0.33249 0.004944 0.006798

Answer:

AN  %/0.00518 0.142903 0.004183*

/0.01008 0.00271 0.09988

This can be verified by verifying that AAN  I

A pseudocodeto generate the inverse matrix using the pseudocodes of figures 3.6 and 3.7 is depicted in
figure 3.8.

Figure 3.8. Pseudocode for finding the inverse of the matrix.

Linear Algebraic Equations Page 20

Numerical Methods

Error Analysis and System Condition. The inverse of the matrix also provides a means to discern
whether systems are ill-conditioned. Three methods are available for this purpose:

1. Scale the matrix of coefficients A so that the largest element in each row is 1. Invert the scaled
matrix and if there are elements of the inverse that are several orders of magnitude greater than
one, it is likely that the system is ill-conditioned.
2. Multiply the inverse by the original coefficient matrix and assess whether the resulting matrix is
close to the identity matrix. If not, it indicates ill-conditioning.
3. Invert the inverted matrix and assess whether the resulting matrix is sufficiently close to the original
matrix. If not, it indicates again that the system is ill-conditioned.

Although these methods can indicate ill-conditioning, it would be preferable to obtain a single number that
could serve as an indicator of the problem. This can be obtained based on the mathematical concept of
vector and matrix norm.

A norm is a real-valued function that provides a measure of the size or “length” of multicomponent
mathematical entities such as vectors and matrices. For example, a vector in a Euclidean space that can
be represented by a row matrix

F  a b c

has its norm PFPQ , also referred to as the Euclidean norm of F, or the length of F as

PFPQ  Ra  b   c  (3.22)

For an n-dimensional vector X  x x x  x

, a Euclidean norm will be computed as

PXPQ  S4 x
6
(3.23)

The concept of this Euclidean norm can be extended further to a matrix A, as in

PAPQ  S4 4 a,
6 6
(3.24)

which is given a name, the Frobenius norm. Other alternatives to the Euclidean and Frobenius norms are
the uniform vector norm

Linear Algebraic Equations Page 21

Numerical Methods

PXPT  max |x |

UU

(3.25)

which takes the element with the largest absolute value as the measure of the vector size, and the uniform
matrix norm or row-sum norm,

PAPT  max 4Va, V

UU

(3.26)
6

which takes the sum of the absolute values of each elements in a row and uses the largest of such sum as
the measure of the matrix size. The choice of definition for the norm is influenced by practical
considerations. For example, the uniform-row norm is widely used because of the simplicity of calculations
and the fact that it usually provides an adequate measure of the matrix size.

The condition number of the matrix Cond A is defined as

Cond A  PAP X PAPN (3.27)

This number can be greater than or equal to one. Using this condition number, it can be shown that

PΔXP PΔAP
 Cond A X
PXP PAP
(3.28)

norm of the coefficients of A multiplied by the condition number. Hence, the condition number can be
that is, the relative error of the norm of the computed solution can be as large as the relative error of the

used to determine how much small changes in matrixA can affect the solutions X.

Evaluate the condition number of the 3 ! 3 Hilbert matrix

Example 3.14

1 1/2 1/3
A  %1/2 1/3 1/4*
1/3 1/4 1/5

The condition number is computed as Cond A  451.2. Since this value is considerably greater than 1,
Answer:

this suggests that the system is ill-conditioned.

Linear Algebraic Equations Page 22

Numerical Methods

Drill Problems 3.1

1. Implement Gauss and Gauss-Jordan elimination with partial pivoting in VBA and MathScript. Use the
algorithm to solve the following system of equations.

System A:

1 /7.5 2.2 2.2 /3.2 2.3 1 x /1.1000

\ 7 2 /2 /2 /5 /1.25 0 _ \x _ \ 27.7500 _
[ ^ [x ^ [ ^


[ /9 2.3 2.5 0 3 2 4 ^ [ ^ [ 14.5250 ^

[ 0 /3 /7 /7 /1 5.5 /1^ [x$ ^  [ 45.8000 ^
[/2.5 /5 2 2 /4.5 /1 3.5^ [x` ^ [/24.1750^
[ /8 3.75 /3.35 /3.35 0 /2.1 2 ^ [xa ^ [/20.2125^
Z /1 /2 4 4 /5 1.15 10 ] Zxb ] Z 32.2500 ]

System B:

0 10 2.5 /3 /1.1 2.7 /3 2.3 x 17.2750

\/0.5 1 /2 0 2.5 /1.9 /5 /1.5 _ \x _ \ 2.6750 _
[ ^[ ^ [ ^
[ 3 /0.75 1.2 2 /10 1 /1 /8 ^ x 2.3500 ^
[x ^ [
[ /1 /1 4 /7 /1 3 2 10 ^ [ $ ^ [/7.7500^

[ 2 0.5 /1 2 /3.1 /4 /2.3 /1 ^ [x` ^ [ 1.0750 ^
[ 3.5 2 1.5 /3.35 /1 1.5 /1 2.5 ^ [xa ^ [/6.5000^
[ /1 4 3.75 4 5 /2 4.5 /3.5^ [xb ^ [/0.6250^
Z /4 0 /2 0 2 3 /7 0 ] Zxc ] Z 30.0000 ]

System C:

3 /9 /5 /3 0 /1 10 5 10 x 10
\7 4 1 0 /1 5 1 /1 /7 _ \x _ \ 9 _
[ ^ [ 2 ^
9 8 /2 /5 5 3 /7 1 /10 [x ^
[ ^ [x ^ [ ^
[ 10 /3 4 7 /10 /6 /5 8 /5 ^ [ $ ^ [ 0 ^
[0 4 /7 /6 4 4 /6 /8 /1 ^ [x` ^  [/16^
[2 2 10 6 /4 /7 /4 7 /8 ^ [xa ^ [ 12 ^
[8 9 /2 3 0 2 3 /3 3 ^ [xb ^ [ 23 ^
[9 /7 /2 2 1 /1 4 4 0 ^ [xc ^ [ 10 ^
Z2 /3 0 /1 /2 /1 /5 /2 /4 ] Zxd ] Z/16]

2. There are two types of LU decomposition. The one demonstrated in the discussion is the Doolittle
decomposition. An alternative one is called Crout decomposition (Chapra, Art. 10.1.4 pp.281-283).
Implement both types of decomposition with partial pivoting in VBA and Mathscript and use them to
solve the systems given in Problem 1. Include in each case an algorithm which counts the number of
flops performed. Compare each.

3. Employ LU decomposition (Doolittle and Crout) to determine the inverses of the coefficient matrices
given in Problem 1. Compare the results of the program with the MINVERSE function in MS Excel
and inv()in MathScript.

Linear Algebraic Equations Page 23

Numerical Methods

4. (a) Create a 3 ! 3 Hilbert matrix. This will be the matrix A. Multiply the matrix by the column vector
x  1 1 1T, generating the matrix b. With this, implement Gauss elimination, Gauss-Jordan
elimination, and LU decomposition to solve the system Ax  b, with the matrix x unknown.
(b) Repeat Problem 4(a) using a 7 ! 7 Hilbert matrix.
(c) Repeat Problem 4(a) using a 10 ! 10 Hilbert matrix.
(d) In each of the cases, explain the discrepancies using condition numbers for the matrix. Use the
Frobenius, as well as the row-sum norms.

5. Polynomial interpolation consists of determining the unique n / 1th-order polynomial that fits n data
points. Such polynomials have the general form

fx  p x
N  p x
N    p
N x  p

where the p’s are constant coefficients. A straightforward way for computing the coefficients is to
generate n linear algebraic equations that can be solved simultaneously for the coefficients. Determine
the coefficients of a fourth-order polynomial fx  p x $  p x  p x   p$ x  p` that passes
through the following five points: 200,0.746 , 250,0.675 , 300,0.616 , 400,0.525 and
500,0.457 . Substitute each of these pairs into fx to yield a system of equations with five
unknowns (the p’s). Use Gauss elimination and LU decomposition to solve for the coefficients. Also,
determine and interpret the condition number and relate this as to how the curve fits into the given data
points. Plot the approximate curve.

3.4. Special Matrices and Gauss-Seidel

Certain matrices have special structures than can be exploited to develop efficient solution schemes. The
first part of this section discusses two of such: banded and symmetric matrices. Efficient elimination
methods are described for both.

The second part of this section turns to an alternative to elimination methods, that is, an approximate,
iterative method. The focus is on Gauss-Seidel which is particularly well suited for large number of
equations. Round-off errors will not be as great issue as in elimination methods, however, the main
disadvantage will be the possibility of divergence.

Special Matrices. A banded matrix is a square matrix that has all elements equal to zero, with the

by two parameters: the bandwidth BW and the half-bandwidth HBW as shown in figure 3.9. The two
exception of a band centered on the main diagonal. The dimensions of a banded system can be quantified

quantities are related by BW  2HBW  1.

In general, Gauss elimination or LU decomposition can be employed to solve banded equations, however,
these are inefficient because of unnecessary manipulations because of pivoting.

Linear Algebraic Equations Page 24

Numerical Methods

Figure 3.9. Bandwidth and half-bandwidth of a banded matrix.

Tridiagonal Systems.A tridiagonal system – that is, one with a bandwidth of 3 – can be expressed generally
as

(3.29)

The Thomas algorithm can be used to solve these kinds of systems. Just like the LU decomposition, it
consists of three steps: forward elimination, forward and backward substitutions. Thus, this algorithm can
also be used to determine the inverse of such matrices. Figure 3.10 shows the algorithm for Thomas
method, with example 3.15 demonstrating the algorithm.

Example 3.15
Solve the following tridiagonal system with the Thomas algorithm.

T  65.970, T  93.778, T  124.538, and T$  159.480

Answer:

Linear Algebraic Equations Page 25

Numerical Methods

Symmetric Matrices. A symmetric matrix is one where a  a for all iand j, that is, the original matrix is
Figure 3.10. Pseudocode for Thomas algorithm.

equal to its transpose. One of the most popular approaches in decomposing symmetric matrices is the
Cholesky decomposition. This algorithm is based on the fact that a symmetric matrix can be decomposed,
as in

A  LLT (3.30)

The terms of the lower triangular matrix in 3.30 can be expressed by the recurrence relations

a5 / ∑N
6 l l5
l5 
l
(3.31)

fori  1, 2, . . . , k / 1 and

5N

l55  Sa55 / 4 l5 (3.32)

6

Linear Algebraic Equations Page 26

Numerical Methods

Example 3.16

6 15 55
Apply Cholesky decomposition to the symmetric matrix

A  %15 55 225*

55 225 979

Answer:
The lower triangular matrix generated by Cholesky decomposition is

2.4495 0 0
L  %6.1237 4.1833 0 *
22.454 20.917 6.1101

Figure 3.11 shows the pseudocode for Cholesky decomposition.

Figure 3.11. Pseudocode for Cholesky decomposition.

Gauss-Seidel. Iterative or approximate methods provide an alternative to the elimination methods

described to this point. Such approaches are similar to the techniques developed to obtain the roots of a
single equation in the previous lesson on roots of equation. In this section, the Gauss-Seidel method, which
is an iterative method to solve for the unknowns of the set of linear algebraic equations, will be
demonstrated.

For conciseness, the discussion is limited to a 3 ! 3 set of equations. If the diagonal elements are all
nonzero, the first equation can be solved for x , the second for x and the third for x , which gives

Linear Algebraic Equations Page 27

Numerical Methods

b / a x / a x
x 
a

b / a x / a x
x 
a
(3.33)

b / a  x / a  x
x 
a

The iteration begins by choosing values for x. For simplicity, zero is chosen as the initial guess. The
value of x is then computed. This will be used to compute for x and the solved values will be
subsequently used to generate x . The process is then repeated using the previously computed values.
Convergence can be checked using the criterion used in previous lessons, that is

New value / Old value

el  m m ! 100%
New value
(3.34)

When all of the error of x drop below the stopping criterion, the iteration can be terminated.

Example 3.17
Use Gauss-Seidel to solve the system described in examples 3.6, 3.11 and 3.12, that is

3x / 0.1x / 0.2x  7.85

0.1x  7x / 0.3x  /19.3
0.3x / 0.2x  10x  71.4

A similar method, called Jacobi, utilizes a somewhat different tactic. Instead of using the latest available
values to solve for the next variable, it stores them first before using them in the next iteration. Figure 3.12
depicts the difference between the two. Although there are cases where the Jacobi method is useful,
Gauss-Seidel’s utilization of the best available estimates usually makes it the method of preference.

Linear Algebraic Equations Page 28

Numerical Methods

Figure 3.12. Comparison of (a) Gauss-Seidel and (b) Jacobi methods.

Convergence criterion for the Gauss-Seidel method. The Gauss-Seidel method is similar in spirit to the
fixed-point iteration described in the previous lesson. Thus, it also exhibits two fundamental problems: (1) it
was sometimes non-convergent; and (2) if it converges, it often did so very slowly. By developing
convergence criterion, one can know in advance if the Gauss-Seidel will lead to a converging solution.

The convergence criterion for Gauss-Seidel method is stated as: the diagonal element must be greater
than the off-diagonal element for each row. Figure 3.13 illustrates the possibilities for convergence of the
Gauss-Seidel method for a set of two equations

11x  13x  286

11x / 9x  99

Figure 3.13. Convergence (a) (or non-convergence (b) ) of Gauss-Seidel method.

Linear Algebraic Equations Page 29

Numerical Methods

Seidel method and is designed to enhance convergence. After each new value of x is computed, that value
Improvement of convergence using relaxation. Relaxation represents a slight modification of the Gauss-

is modified by a weighted average of the results of the previous and present iterations

x
Qs  λx
Qs  1 / λxuvw (3.35)

whereλ is a weighting factor that is assigned a value between 0 and 2.

If λ  1, the result is unmodified. If its value is between 0and 1, the values are said to be underrelaxed.

oscillations. If the λ is between 1and 2, the values are said to be overrelaxed. This is done with an implicit
This is done to make a nonconvergent system converge or to hasten convergence by dampening out

assumption that the method is convergent. This makes the convergence faster by pushing the values
closer to convergence. The choice of the relaxation factor is problem-specific.

Linear Algebraic Equations Page 30