Linear Systems

Irwan Ary Dharmawan

Linear Systems

3 3 33 2 32 1 31
2 3 23 2 22 1 21
1 3 13 2 12 1 11
b x a x a x a
b x a x a x a
b x a x a x a
= + + +
= + + +
= + + +
(
(
(
(
(

=
(
(
(
(
(

(
(
(
(
(

3
2
1
3
2
1
33 32 31
23 22 21
13 12 11
b
b
b
x
x
x
a a a
a a a
a a a
Linear Systems
Solve Ax=b, where A is an nn matrix and
b is an n1 column vector
Can also talk about non-square systems where
A is mn, b is m1, and x is n1
Overdetermined if m>n:
more equations than unknowns
Underdetermined if n>m:
more unknowns than equations
Can look for best solution using least squares
Singular Systems
A is singular if some row is
linear combination of other rows
Singular systems can be underdetermined:
or inconsistent:
10 6 4
5 3 2
2 1
2 1
= +
= +
x x
x x
11 6 4
5 3 2
2 1
2 1
= +
= +
x x
x x
Inverting a Matrix
Usually not a good idea to compute x=A
-1
b
Inefficient
Prone to roundoff error
In fact, compute inverse using linear solver
Solve Ax
i
=b
i
where b
i
are columns of identity,
x
i
are columns of inverse
Many solvers can solve several R.H.S. at once
Gauss-Jordan Elimination
Fundamental operations:
1. Replace one equation with linear combination
of other equations
2. Interchange two equations
3. Re-label two variables
Combine to reduce to trivial system
Simplest variant only uses #1 operations,
but get better stability by adding
#2 (partial pivoting) or #2 and #3 (full pivoting)
Gauss-Jordan Elimination
Solve:
Only care about numbers form tableau or augmented
matrix:
13 5 4
7 3 2
2 1
2 1
= +
= +
x x
x x
(
(

13
7
5 4
3 2
Gauss-Jordan Elimination
Given:
Goal: reduce this to trivial system
and read off answer from right column
(
(

13
7
5 4
3 2
(
(

?
?
1 0
0 1
Gauss-Jordan Elimination
Basic operation 1: replace any row by
linear combination with any other row
Here, replace row1 with
1
/
2
* row1 + 0 * row2
(
(

13
7
5 4
3 2
(
(

13 5 4
1
2
7
2
3
Gauss-Jordan Elimination
Replace row2 with row2 4 * row1
Negate row2
(
(

13 5 4
1
2
7
2
3
(
(

1 1 0
1
2
7
2
3
(
(

1 1 0
1
2
7
2
3
Gauss-Jordan Elimination
Replace row1 with row1
3
/
2
* row2
Read off solution: x
1
= 2, x
2
= 1
(
(

1 1 0
1
2
7
2
3
(
(

1
2
1 0
0 1
Gauss-Jordan Elimination
For each row i:
Multiply row i by 1/a
ii
For each other row j:
Add a
ji
times row i to row j
At the end, left part of matrix is identity,
answer in right part
Can solve any number of R.H.S. simultaneously
Pivoting
Consider this system:
Immediately run into problem:
algorithm wants us to divide by zero!
More subtle version:
(
(

8
2
3 2
1 0
(
(

8
2
3 2
1 001 . 0
Pivoting
Conclusion: small diagonal elements bad
Remedy: swap in larger element from somewhere else
Partial Pivoting
Swap rows 1 and 2:
Now continue:
(
(

8
2
3 2
1 0
(
(

2
8
1 0
3 2
(
(

(
(

2
1
1 0
0 1
2
4
1 0
1
2
3
Full Pivoting
Swap largest element onto diagonal by swapping rows 1 and
2 and columns 1 and 2:
Critical: when swapping columns, must remember to swap
results!
(
(

8
2
3 2
1 0
(
(

2
8
0 1
2 3
Full Pivoting
Full pivoting more stable, but only slightly
(
(

2
8
0 1
2 3
(
(

3
2
3
8
3
2
3
2
0
1
(
(

1
2
1 0
0 1
*
Swap results
1 and 2
Operation Count
For one R.H.S., how many operations?
For each of n rows:
Do n times:
For each of n+1 columns:
One add, one multiply
Total = n
3
+n
2
multiplies, same # of adds
Asymptotic behavior: when n is large, dominated by n
3
Faster Algorithms
Our goal is an algorithm that does this in
1
/
3
n
3
operations, and does not require
all R.H.S. to be known at beginning
Before we see that, lets look at a few
special cases that are even faster
Tridiagonal Systems
Common special case:
Only main diagonal + 1 above and 1 below
(
(
(
(
(
(
(

4
3
2
1
44 43
34 33 32
23 22 21
12 11
0 0
0
0
0 0
b
b
b
b
a a
a a a
a a a
a a
Solving Tridiagonal Systems
When solving using Gauss-Jordan:
Constant # of multiplies/adds in each row
Each row only affects 2 others
(
(
(
(
(
(
(

4
3
2
1
44 43
34 33 32
23 22 21
12 11
0 0
0
0
0 0
b
b
b
b
a a
a a a
a a a
a a
Running Time
2n loops, 4 multiply/adds per loop
(assuming correct bookkeeping)
This running time has a fundamentally different dependence
on n: linear instead of cubic
Can say that tridiagonal algorithm is O(n) while
Gauss-Jordan is O(n
3
)
Big-O Notation
Informally, O(n
3
) means that the dominant term for large n is
cubic
More precisely, there exist a c and n
0
such that
running time s c n
3
if
n > n
0
This type of asymptotic analysis is often used
to characterize different algorithms
Triangular Systems
Another special case: A is lower-triangular
(
(
(
(
(
(
(

4
3
2
1
44 43 42 41
33 32 31
22 21
11
0
0 0
0 0 0
b
b
b
b
a a a a
a a a
a a
a
Triangular Systems
Solve by forward substitution
(
(
(
(
(
(
(

4
3
2
1
44 43 42 41
33 32 31
22 21
11
0
0 0
0 0 0
b
b
b
b
a a a a
a a a
a a
a
11
1
1
a
b
x =
Triangular Systems
Solve by forward substitution
(
(
(
(
(
(
(

4
3
2
1
44 43 42 41
33 32 31
22 21
11
0
0 0
0 0 0
b
b
b
b
a a a a
a a a
a a
a
22
1 21 2
2
a
x a b
x

=
Triangular Systems
Solve by forward substitution
(
(
(
(
(
(
(

4
3
2
1
44 43 42 41
33 32 31
22 21
11
0
0 0
0 0 0
b
b
b
b
a a a a
a a a
a a
a
33
2 32 1 31 3
3
a
x a x a b
x

=
Triangular Systems
If A is upper triangular, solve by backsubstitution
(
(
(
(
(
(
(

5
4
3
2
1
55
45 44
35 34 33
25 24 23 22
15 14 13 12 11
0 0 0 0
0 0 0
0 0
0
b
b
b
b
b
a
a a
a a a
a a a a
a a a a a
55
5
5
a
b
x =
Triangular Systems
If A is upper triangular, solve by backsubstitution
(
(
(
(
(
(
(

5
4
3
2
1
55
45 44
35 34 33
25 24 23 22
15 14 13 12 11
0 0 0 0
0 0 0
0 0
0
b
b
b
b
b
a
a a
a a a
a a a a
a a a a a
44
5 45 4
4
a
x a b
x

=
Triangular Systems
Both of these special cases can be solved in
O(n
2
) time
This motivates a factorization approach to solving arbitrary
systems:
Find a way of writing A as LU, where L and U are both triangular
Ax=b LUx=b Ly=b Ux=y
Time for factoring matrix dominates computation
Cholesky Decomposition
For symmetric matrices, choose U=L
T
Perform decomposition
Ax=b LL
T
x=b Ly=b L
T
x=y
(
(
(

(
(
(

(
(
(

33
32 22
31 21 11
33 32 31
22 21
11
33 23 13
23 22 12
13 12 11
0 0
0 0
0 0
l
l l
l l l
l l l
l l
l
a a a
a a a
a a a
Cholesky Decomposition
22
31 21 23
32 23 32 22 31 21
2
21 22 22 22
2
22
2
21
11
13
31 13 31 11
11
12
21 12 21 11
11 11 11
2
11
l
l l a
l a l l l l
l a l a l l
l
a
l a l l
l
a
l a l l
a l a l

= = +
= = +
= =
= =
= =
(
(
(

(
(
(

(
(
(

33
32 22
31 21 11
33 32 31
22 21
11
33 23 13
23 22 12
13 12 11
0 0
0 0
0 0
l
l l
l l l
l l l
l l
l
a a a
a a a
a a a
Cholesky Decomposition
ii
i
k
jk ik ij
ji
i
k
ik ii ii
l
l l a
l
l a l

=
=
1
1
1
1
2
(
(
(

(
(
(

(
(
(

33
32 22
31 21 11
33 32 31
22 21
11
33 23 13
23 22 12
13 12 11
0 0
0 0
0 0
l
l l
l l l
l l l
l l
l
a a a
a a a
a a a
Cholesky Decomposition
This fails if it requires taking square root of a negative
number
Need another condition on A: positive definite
For any v, v
T
A v > 0
(Equivalently, all positive eigenvalues)
Cholesky Decomposition
Running time turns out to be
1
/
6
n
3
Still cubic, but much lower constant
Result: this is preferred method for solving
symmetric positive definite systems
LU Decomposition
Again, factor A into LU, where
L is lower triangular and U is upper triangular
Last 2 steps in O(n
2
) time, so total time dominated by
decomposition
Ax=b
LUx=b
Ly=b
Ux=y
Crouts Method
More unknowns than equations!
Let all l
ii
=1
(
(
(

(
(
(

(
(
(

33
23 22
13 12 11
33 32 31
22 21
11
33 32 31
23 22 21
13 12 11
0 0
0 0
0 0
u
u u
u u u
l l l
l l
l
a a a
a a a
a a a
Crouts Method
(
(
(

(
(
(

(
(
(

33
23 22
13 12 11
32 31
21
33 32 31
23 22 21
13 12 11
0 0
0
1
0 1
0 0 1
u
u u
u u u
l l
l
a a a
a a a
a a a
22
12 31 32
32 32 22 32 12 31
12 21 22 22 22 22 12 21
12 12
11
31
31 31 11 31
11
21
21 21 11 21
11 11
u
u l a
l a u l u l
u l a u a u u l
a u
u
a
l a u l
u
a
l a u l
a u

= = +
= = +
=
= =
= =
=
Crouts Method
For i = 1..n
For j = 1..i
For j = i+1..n
(
(
(

(
(
(

(
(
(

33
23 22
13 12 11
32 31
21
33 32 31
23 22 21
13 12 11
0 0
0
1
0 1
0 0 1
u
u u
u u u
l l
l
a a a
a a a
a a a

=
=
1
1
j
k
ki jk ji ji
u l a u
ii
i
k
ki jk ji
ji
u
u l a
l

=
1
1
Crouts Method
Interesting note: # of outputs = # of inputs,
algorithm only refers to elements not output yet
Can do this in-place!
Algorithm replaces A with matrix
of l and u values, 1s are implied
Resulting matrix must be interpreted in a special way: not a regular
matrix
Can rewrite forward/backsubstitution routines to use this packed l-u
matrix
(
(
(

33 32 31
23 22 21
13 12 11
u l l
u u l
u u u
LU Decomposition
Running time is
1
/
3
n
3
Only a factor of 2 slower than symmetric case
This is the preferred general method for
solving linear equations
Pivoting very important
Partial pivoting is sufficient, and widely implemented