Part I

Numerical Methods for Solvin

g Linear Systems

Outline
6.1 Linear System of Equations
6.2 Pivoting Strategies
6.3 Linear Algebra and Matrix Inversion
6.4 The Determinant of a Matrix
6.5 Matrix Factorization
6.6 Special Types of Matrices

Linear Systems of Equations

A set of n equations and n unknowns
a11 x1 a12 x2 a13 x3 ... a1 n xn b1

a21 x1 a22 x2 a23 x3 ... a2 n xn b2

..
..
.
.
an1 x1 an 2 x2 an 3 x3 ... ann xn bn
Abbreviated Form:

AX b

linear systems applications: signal processing

automatic control theory and telecommunications.

Linear Systems of Equations

A Little Background Information:
Leontief, 1973 Nobel Prize winner in Economic Science
Worked in 1949 at Harvard University
First significant use of computers to solve
what was then a large-scale mathematical model
Divided the US economy into 500 sectors, for each sector, a linear e
quation is written down to describe how the sector distributes its
output into other sectors of the economy
Linear system of 500 equations, 500 unknowns
Mark II hummed and blinked for 56 hours to produce the solution
for a reduced system of 42 equations.

Gaussian Elimination
One of the most popular techniques for solving
linear system of equations.
We have learned Gaussian Elimination
in Primary School

Gaussian Elimination
Forward Elimination
Backward/Back
Substitution
Pitfalls
Partial Pivoting

Example: Rocket Velocity

The upward velocity of a rocket is
given at three different times
Time, t
s

Velocity, v
m/s

5
8
12

106.8
177.2
279.2

The velocity data is approximated by a polynomial as:

v t a1t 2 a2 t a3 ,
Find the velocity at t=6 seconds.

5 t 12.

v t a1t 2 a2 t a3 , 5 t 12.

Assume

Results in a matrix template of the following form:

t12

t1

2
t
2
t32

t2
t3

1
1

a1
v1
a v
2
2
a3
v3

Using date from the time/velocity table, the matrix becomes:

25
64

144

1
8 1

12 1
5

a1
106 .8
a 177 .2
2

a3
279 .2

 25 5 1
64 8 1

144 12 1

106
a1
.8
a 177
.2
2

279
a3
.2

Forward Elimination: Step 1

Row1
Row2
(64)

25

yields

25
0

5
1 a1 106 .81
4 .8 1.56 a2 96 .21

144 12
1 a3 279 .2

5
1
25
0 4.8 1.56

1
144 12

106
a1
.81
a 96
.21
2

279
a3
.2

Forward Elimination: Step 2

Row1
Row3
(144)

25
yields

25
0
0

1 a1 106 .8

1.56 a2 96 .21

16 .8 4 .76 a3 336 .0
5
4 .8

5
1
25
0 4.8 1.56

0 16.8 4.76

106
a1
.8
a 96
.21
2

336
a3
.0

Forward Elimination: Step 3

Row2
Row3
( 16.8)

4.8

yields

5
1
25
0 4 .8 1.56

0
0
0 .7

a1 106 .8
a 96 .21
2

a3 0 .735

This is now ready for Back Substitution.

5
1
25
0 4.8 1.56

0
0.7
0

106
a1
.8
a 96
.21
2

0. 735
a3

Back Substitution: Step 1

Solve for a3 using the third equation
0.7a3 0.735

0 .735
a3
0 .7
a3 1.050

5
1
25
0 4.8 1.56

0
0.7
0

106
a1
.8
a 96
.21
2

0. 735
a3

a3 1.050

Back Substitution: Step 2

Solve for a2 using the second equation

4.8a2 1.56a3 96.21

96.21 1.56a3
a2
4.8

-96 .21 1.56 1.050

a2
19.70
-4 .8

5
1
25
0 4.8 1.56

0
0.7
0

106
a1
.8
a 96
.21
2

0. 735
a3

a3 1.050
a2 19.70

Back Substitution: Step 3

Solve for a1 using the first equation
25a1 5a2 a3 106.8
106.8 5a2 a3
a1
25
106.8 5 19.70 1.050
a1
0.2900
25

a3 1.050

a2 19.70

The solution vector is, therefore

a1 0.2900
a1 0.2900
a 19.70
2

a3 1.050

The polynomial that passes through the three

data points is then:
v t a1t 2 a2 t a3
0.2900t 2 19.70t 1.050, 5 t 12

and
v 6 0.2900 6 19.70 6 1.050 129.69 m/s.
2

Forward Elimination
The goal of forward elimination is to transform the
coefficient matrix into an upper triangular matrix
25 5 1 x1 106.8
64 8 1 x 177.2

144 12 1 x3 279.2

5
1
25
0 4.8 1.56

0
0
0.7

x1 106.8
x 96.21
2

x3 0.735

Forward Elimination
A set of n equations and n unknowns

a11 x1 a12 x2 a13 x3 ... a1n xn b1

a21 x1 a22 x2 a23 x3 ... a2 n xn b2

.
.
.

.
.
.

an1 x1 an 2 x2 an 3 x3 ... ann xn bn

(n-1) steps of forward elimination

Forward Elimination
Step 1

minus

a21 x1 a22 x2 a23 x3 ... a2 n xn b2

a21
a ( a11 x1 a12 x2 a13 x3 ... a1n xn b1 )
11

a21
a21
a21
a22
a12 x2 ... a2 n
a1 n xn b2
b1
a11
a11
a11

a x2 ... a xn b
'
22

'
2n

'
2

Forward Elimination
Repeat this procedure for the remaining equations to
reduce the set of equations as
a11 x1 a12 x2 a13 x3 ... a1n xn b1
'
'
a22
x2 a23
x3 ... a2' n xn b2'

'
'
a32
x2 a33
x3 ... a3' n xn b3'
.
.
.
.
.
.

'
an' 2 x2 an' 3 x3 ... ann
xn bn'

End of Step 1

Forward Elimination
a11 x1 a12 x2 a13 x3 ... a1n xn b1
'
'
a22
x2 a23
x3 ... a2' n xn b2'
"
a33
x3 ... a3"n xn b3"

.
.
.

"
an"3 x3 ... ann
xn bn"

End of Step 2

Forward Elimination
At the end of (n-1) Forward Elimination steps, the
system of equations will look like
a11 x1 a12 x2 a13 x3 ... a1n xn b1
'
'
a22
x2 a23
x3 ... a2' n xn b2'

"
a33
x3 ... a3"n xn b3"

.
.
.

n 1
ann
xn bn n1

End of Step (n-1)

Matrix Form at End of Forward Elimination

a11
0

a12
a'22
0

a13
a'23
"
a33

a1n
a'2 n

"
a3 n

(n1 )
ann

x1 b1
x b'
2 2
x3 b3"

xn bn(n-1 )

Back Substitution
Starting Equations
a11 x1 a12 x2 a13 x3 ... a1n xn b1
'
'
a22
x2 a23
x3 ... a2' n xn b2'

a x3 ... a xn b
.
.
.
"
33

"
n

n 1

"
3

n 1

ann xn bn

Back Substitution
Start with the last equation as it has only one unknown
( n 1 )
n
( n 1 )
nn

b
xn
a

bi i 1 ai ,ii11 xi 1 ai ,ii12 xi 2 ... ai ,in1 xn

xi
,
i 1
aii
i 1

xi

bi

aij i 1 x j
j i 1
i 1
ii

i n 1,...,1

Shall we dance to celebrate?

Pitfall#1. Division by Zero

10 x2 7 x3 3
6 x1 2 x2 3 x3 11
5 x1 x2 5 x3 9
0 10 7
6 2
3

5 1 5

Cant even start!

x1 3
x 11
2
x3 9

12 x1 10 x2 7 x3 15
6 x1 5 x2 3 x3 14
24 x1 x2 5 x3 28

Is division by zero an issue here?

YES!
12 10 7
6
5
3

24 1 5

x1 15
x 14
2
x3 28

12 10 7
0

0
6.5

12 21 19

x1 15
x 6.5
2
x3 2

Division by zero is a possibility at any step of forward

elimination

Pitfall#2. Large Round-off Errors

15
10 x1 45
20
3 2.249 7 x 1.751

5
1
3 x3 9

Exact Solution

x1
x
2
x3

1
1

15
10
20
3 2.249 7

5
1
3

x1
45
x 1.751
2

x3
9

Solved on a computer using 6 significant digits with

chopping

x1
0.9625
x 1.05
2

x3
0.999995

15
10
20
3 2.249 7

5
1
3

x1 45
x 1.751
2

x3 9

Solved on a computer using 5 significant digits with

chopping
x1 0.625
x 1.5
2

x3 0.99995

Is there a way to reduce the round off error?

Avoiding Pitfalls
Increase the number of significant digits
Decreases round-off error
Does not avoid division by zero
Gaussian Elimination with Partial Pivoting
Avoids division by zero
Reduces round off error

What is Different About Partial Pivoting?

At the beginning of the kth step of the forward
elimination, find the maximum of
akk , ak 1,k ,L , ank

If the maximum of the values is

a pk

in the p th row, k p n, then switch rows p and k.

Also called Maximal Column Pivoting

Matrix Form at Beginning of the 2nd Step of F

orward Elimination

a11
0

a12

a13

'
22
'
32

'
23
'
33

'
n2

'
n3

'
n4

a1 n
'
a2 n

a3' n


'
ann

x1

x2

b1
b'
2

x3 b3'


'

xn bn

Example (2nd step of FE)

6
0

14
7
4

9
0
0 17

x1
x
2

5.1 3.7 6
6
1
2

12 1 11
23
12

6
11

8
43

x3

x4
x5

5
6

9
3

Which two rows would you switch?

Example (2nd step of FE)

6 14 5.1 3.7 6
0 17 12 11 43

0
4
12 1 11

9
23 6
8
0
0 7
6
1
2

x1

5
3

x2

x3 8

x4 9
x5 6

After row interchanges

Forward Elimination
Same as direct Gauss elimination method except
that we switch rows before each of the (n-1) steps
of forward elimination.

Back Substitution
Same as direct Gauss elimination method.

Matrix Form at the Beginning of the 2nd Step

of Forward Elimination

a11
0

0

0

a12

a13

'
22
'
32

'
23
'
33

a'n 2

an' 3

an' 4

a
a

a
a

a1 n
'
a2 n

'
a3 n


'

ann

x1

x2

b1
b'
2

'

x3
b3


xn bn'

Matrix Form at End of Forward Elimination

a11
0

a12
'
a22
0

a13
'
a23
a"33

a1n

'
a2 n

a"3 n

(n1 )
0 ann

x1 b1
x b'
2 2
x3 b3"

xn bn(n-1 )

Back Substitution Starting Eqns

a11 x1 a12 x2 a13 x3 ... a1n xn b1
'
'
a22
x2 a23
x3 ... a2' n xn b2'

"
a33
x3 ... an" xn b3"

.
.
.

n 1

n 1

ann xn bn

Back Substitution
bn( n1 )
x n ( n 1 )
ann
i 1

xi

bi

aij i 1 x j
j i 1
i 1
ii

i n 1,...,1

Partial Pivoting: Example

Consider the system of equations
10 x1 7 x2 7
3 x1 2.099 x2 6 x3 3.901
5 x1 x2 5 x3 6

In matrix form
7 0 x1
7
10
3.901
3 2.099 6 x

2 =
6
5
1 5 x3

Solve using Gaussian Elimination with Partial

Pivoting using 5 significant digits with chopping.

Forward Elimination: Step 1

Examining the values of the first column
|10|, |-3|, and |5| or 10, 3, and 5
The largest absolute value is 10, which means, to
follow the rules of Partial Pivoting, we switch row1
with row1.
Performing Forward Elimination
7 0 x1 7
10
3 2.099 6 x 3.901

5
1 5 x3 6

7
0 x1 7
10
0 0.001 6 x 6.001

0
2.5
5 x3 2.5

Forward Elimination: Step 2

Examining the values of the first column
|-0.001| and |2.5| or 0.0001 and 2.5
The largest absolute value is 2.5, so row 2 is
switched with row 3
Performing the row swap
7
0 x1 7
10
0 0.001 6 x 6.001

0
2.5
5 x3 2.5

10
0
0

7
2.5

0 x1 7
5 x 2 2 .5

0.001 6 x3 6.001

 10
0

0 x1 7
2.5
5 x2 2.5

0.001 6 x3 6.001

Forward Elimination: Step 2

Performing the Forward Elimination results in:
0
10 7
0 2.5
5

0
0 6.002

x1 7
x 2. 5
2

x3 6.002

 10 7
0 2.5

0 x1 7
5 x2 2.5

6.002 x3 6.002

Back Substitution
Solving the equations through back substitution
6.002
x3
1
6.002

2.5 5 x2
x2
1
2 .5

7 7 x2 0 x3
x1
0
10

Compare the calculated and exact solution

X calculated

x1 0
x2 1

x3 1

X exact

x1 0
x2 1

x3 1

The fact that they are equal is a coincidence, but it

does illustrate the advantage of Partial Pivoting

Shall we dance now?

30.00 x1 591400 x2 591700

5.291 x1 6.130 x2 46.78

Exact solution
x1 10
x2 1

Performing forward elimination once yields

Numerical solution
30.00 x1 591400 x2 591700
104300 x2 104400

x1 10

x2 1.001

It is inadequate to consider only the largest value

in the numbers to be zeroed out in a certain
column but also the large values in the rows.

25
equation 1
equation 2

20
15

x2

10
5
0
-5
-10
-20

-15

-10

-5

for i=1:101
x1(i)=-20+(i-1)/100*40;
x2(i)=(59170030*x1(i))/591400;
x3(i)=(46.78-5.291*x1(i))/6.130;
end
plot(x1,x2, 'r-', 'LineWidth', 4)

0
x1

10

15

20

hold on
plot(x1,x3, '-', 'LineWidth', 4)
xlabel('x1')
ylabel('x2')
legend('equation 1', 'equation 2')

Scaled Partial Pivoting: also called scaled-column pi

voting, it places the element in the pivot position t
hat is largest relative to the entries in its row.
The first step is to define a scale factorsi
row as
si max aij

for each

1 j n

If, for some i , we have si 0 , then the system

has no unique solution since all entries in the ith
row are 0.
We assume that this is not the case.

 The appropriate row interchange to place zeros in

the first column is determined by choosing the leas
t integer with
p
a p1
sp

ak 1
max
1 k n s
k

and interchanging the 1st and the pth row.

The effect of scaling is to ensure that the largest
element in each row has a relative magnitude of 1
before the comparison for row interchange is
performed.

 In a similar manner, before eliminating the variable

s , we select
the smallest integer
xi
pi
with

a pi
sp

aki
max
1 k n s
k

and interchange row i and row p.

The scale factors are computed only once, at the
start of the procedure, and must also be
interchanged when row interchanges are performed.

30.00 x1 591400 x2 591700

5.291 x1 6.130 x2 46.78

s1 max{| 30.00 |, 591400} 591400

s2 max{| 5.291 |, 6.130} 6.130
a11
30.00

0.5073 10 4
s1
591400

a21 5.291

0.8631
s2
6.130

Interchanging row 1 and row 2 yields

5.291 x1 6.130 x2 46.78
30.00 x1 591400 x2 591700

Applying Gaussian elimination to the new

system produces the correct results: x1 10; x2 1.

Matrix Form at End of Forward Elimination

a11
0

a12
'
a22
0

a13
'
a23
a"33

a1n

'
a2 n

a"3 n

(n1 )
0 ann

x1 b1
x b'
2 2
x3 b3"

xn bn(n-1 )

Gauss-Jordan Method

a11
0

0
a'22
0

0
0
"
a33

(n1 )

0 ann

x1 b1
x b'
2 2
"
x3 b3

xn bn(n-1 )

This procedure circumvents the back substitution in

the Gaussian Elimination

 a11
0

a'22
0

1
0

0
1
0

0
a"33

0
0
1

0
0

(n1 )

0 ann

0
0

0


1

x1

x1
x
2
x3


xn

b1
b2'

"
b3

bn(n-1 )

b1 / a11

'
x2 b2' / a22

''

x3 b3" / a33

(n- 1 )

xn bn(n-1 ) / ann

Operations Count for Forward Elimination

Step 1

a21 x1 a22 x2 a23 x3 ... a2 n xn b2

minus

a21
a ( a11 x1 a12 x2 a13 x3 ... a1n xn b1 )
11

a21
a21
a21
a22
a12 x2 ... a2 n
a1 n xn b2
b1
a11
a11
a11

a x2 ... a xn b
'
22

How many multiplications/divisions?

additions/subtractions ?

'
2n

'
2

Forward Elimination
Repeat this procedure for the remaining equations to
reduce the set of equations as
/ a11 x1 a12 x2 a13 x3 ... a1n xn b1
/

(n)
. (n)
.
.

(n)
Total

( n 1) n

a x2 a x3 ... a xn b

( n 1)

'
'
a32
x2 a33
x3 ... a3' n xn b3'
.
.
.
.
.
.

(. n 1)
.
.

'
22

'
23

'
2n

'
2

'
an' 2 x2 an' 3 x3 ... ann
xn bn' ( n 1)

End of Step 1

Total

( n 1)( n 1)

Forward Elimination
/

a11 x1 a12 x2 a13 x3 ... a1n xn b1

'
'
a22
x2 a23
x3 ... a2' n xn b2'

(. n 1)
.
.

"
a33
x3 ... a3"n xn b3"
.
.
.

. (n)
.
.

( n 1)

"
an"3 x3 ... ann
xn bn"

(n)

Total

( n 2)( n 1)

Total

End of Step 2

( n 2) n

Forward Elimination
At the end of (n-1) Forward Elimination steps, the
system of equations will look like

a11 x1 a12 x2 a13 x3 ... a1n xn b1

'
'
a22
x2 a23
x3 ... a2' n xn b2'

a x3 ... a xn b
.
.
.
"
33

( 2)
Total

(1)( 2)

"
3n

"
3

n 1
ann
xn bn n1

End of Step (n-1)

( 3)
Total

(1)( 3)

Zeroing out

( n 1) n

column 1
.
column
2
.
.

( n 1)( n 1)
( n. 2) n

( n . 2)( n 1)

column n-1

.
.

.
.

(1)( 2)

(1)( 3)

General formula ( n i )( n i 1)
n 1

Total

( n i )( n i 2)
n 1

( n i )( n i 1)

( n i )( n i 2)

n3 n

2n 3 3n2 5 n

i 1

i 1

Operations count for Back Substitution

Starting Equations
a11 x1 a12 x2 a13 x3 ... a1n xn b1
'
'
a22
x2 a23
x3 ... a2' n xn b2'

"
a33
x3 ... an" xn b3"

.
.
.

n 1

n 1

ann xn bn

Back Substitution
/

( n 1 )
n
( n 1 )
nn

b
xn
a

(1)

bi i 1 ai ,ii11 xi 1 ai ,ii12 xi 2 ... ai ,in1 xn

xi
,
i 1
aii
(n i)
( n i 1)

i n 1,...,1

Total

n n
(n i)

2
i 1
n 1

Total

n n
1 ( n i 1)
2
i 1
n 1

F.E.

n3 n
3

n2 n
2

/
2n3 3n2 5 n
6

n2 n
B.S.
2
n3
n
n3 n2 5 n
2
n

Total
3
3
3
2
6
Conclusion: For large n, the total number of
multiplications and divisions is approximately n3 / 3 , as
is the total number of additions and subtractions, thus the
amount of computation and the time required increases
with n in proportion to n3 .

Gauss-Jordan involves more operations than Gauss

elimination
Operation counts for G-J: n3 n 2 n
2n3 3n 2 7 n

Operation counts for G-E:

3
2
6
Therefore, Gauss elimination is the simple elimination
method of preference for obtaining solutions of linear
algebraic equations.

One of the primary reasons that we have introduced the

Gauss-Jordan, however, is that it is still used in
engineering as well as in some numerical algorithms.

function x=gausselim(A,b)
% Eliminate below pivot
% Gaussian Elimination with for j=i+1:n
Partial Pivoting
m=A(j,i)/A(i,i);
n=size(A,1);
for k=i:n
A(j,k)=A(j,k)-m*A(i,k);
% Elimination
end
for i=1:n-1
b(j)=b(j)-m*b(i);
% Find largest pivot and
end
interchange rows
end
[foo,p]=max(abs(A(i:n,i)));
p=p+i-1;
A([i,p],:)=A([p,i],:);
help max
b([i,p])=b([p,i]);

% Back substitution
x=b;
for i=n:-1:1
for j=i+1:n
x(i)=x(i)-A(i,j)*x(j)
end
x(i)=x(i)/A(i,i)
end

>> A=[2 8 7; 3 7 1; 9 8 2]
A=
2 8 7
3 7 1
9 8 2
>> b=[2 3 0]
b=
2 3 0
>> gausselim(A, b)
ans =
-0.5570 0.7215 -0.3797

Homework 1-1

Numerical Analysis, 7th edition

Page 368 Q. 5 f) (to be carried out on a computer)

Part 1. do Q. 5 f) using partial pivoting, follow the steps i
n algorithm 6.2 in page 362.
Part 2. do Q. 5 f) using scaled partial pivoting, follow the s
teps in algorithm 6.3 in page 362.

Note: No chopping is needed!

Due: Monday 13:30pm, October 10, 2016