Solution of Linear Equations

Com S 477/577 Oct 21, 2010

We have discussed general methods for solving arbitrary equations. We also looked at the special class of polynomial equations. Another special class is given by systems of linear equations. In fact, many problems in numerical analysis can be reduced to the problem of solving systems of linear equations. We already witnessed this in cubic spline interpolation and Newtons method for systems of nonlinear equations. Other applications include the solution of ordinary or partial dierential equations (ODEs or PDEs), the eigenvalue problems of mathematical analysis, least-squares tting of data, and polynomial approximation.

Elements of Linear Algebra

Recall that a basis for a vector space is a sequence of vectors that are linearly independent and span the space. Given a linear function f : Rn Rm , and a pair of bases B2 = {d1 , d2 , . . . , dm }, B1 = {e1 , e2 , . . . , en }, for Rn , for Rm ,

we can represent f by an m n matrix A such that f (x) = Ax for any x Rn . Note that the matrix A depends on the choices of the bases B1 and B2 . The rank r of an m n matrix A is the number of independent rows. The column space of A consists of all linear combinations of the columns. The row space of A consists of all linear combination of the rows. Since each column has m components, the column space of A is a subspace of Rm . It consists of all points in Rm that are image vectors under the mapping A. Similarly, the row space is a subspace of Rn . The null space of A consists of all solutions to Ax = 0, where x Rn . Theorem 1 (Fundamental Theorem of Linear Allegra) The row space and the column space both have dimension r. The null space has dimension n r and is the orthogonal complement of the row space (in Rn ). In other words, Rn = row space null space,

n = dim(row space) + dim(null space). Consider the system of equations Ax = b. If b is not an element of the column space of A, then the system is inconsistent (or overdetermined). If b is in the column space of A and the null space of A is non-trivial, then we say that the system 1

is underdetermined. In this case, every solution x can be split into a row space component xr and a null space component xn so that Ax = A(xr + xn ) = Axr + Axn = Axr . The null space goes to zero, Axn = 0, while the row space component goes to the column space, Axr = Ax. If A is an n n square matrix, we say that A is singular i det(A) = 0 i rank(A) < n i the rows of A are not linearly independent i the columns of A are not linearly independent i the dimension of the null space of A is non-zero i A is not invertible.

LU Decomposition
P A = LDU,

An m n matrix A, where m n can be written in the form where P is an m m permutation matrix that species row interchanges, U is an m n upper-triangular matrix with 1s on the diagonal, D is an m m square diagonal matrix.

L is an m m square lower-triangular matrix with 1s on the diagonal,

1. The entries on the diagonal of D are called pivots (after the Gaussian elimination procedure). 2. When A is a square matrix, the product of the pivots is equal to det(A), where the sign is chosen if odd number of row interchanges are performed and the sign + is chosen otherwise. 3. If A is symmetric and P = I, the identity matrix, then U = LT . 4. If A is symmetric and positive denite, then U = LT and the diagonal entries of D are strictly positive.
Example 3. 1 0 1 1 = 0 1 1 1 0 0 0 1 1 0 0 1 1 0 0 0 1 1 0 0 0 1 0 0 1 ; 0

1 2

1 1

0 1

1 0 2 1

1 0 0 1

1 0

1 1

0 1

Like most other decompositions and factorizations, the LU (or LDU ) decomposition is used to simplify the solution of the system Ax = b. Suppose A is square and non-singular, solving the above system is equivalent to solving LDU x = P b. We then rst solve Ly = P b for the vector y and then solve U x = D 1 y. for the vector x. Each of the above systems can be solved easily using forward or backward substitution.

2.1

Crouts Algorithm

The LU decomposition for an n n square matrix A can be generated directly by Gaussian elimination. Nevertheless, a more ecient procedure is Crouts algorithm. In case no pivoting is needed, the algorithm yields two matrices L = {ij } and U = {ij } whose product is A = {ai j}, namely, 1 0 0 0 11 12 13 1n a11 a12 a13 a1n 21 1 0 0 0 22 23 2n a21 a22 a23 a2n 31 32 1 0 0 0 33 3n = a31 a32 a33 a3n . . . . . . . . . . . .. . . .. .. . . . . . . . . . . . . . . . . . . . . . . . . . . . n1 n2 n3 1 0 0 0 nn an1 an2 an3 ann The algorithm solves for L and U simultaneously and column by column.1 At the jth outer iteration step, it generates column j of U and then column j of L. for j = 1, 2, . . . , n do for i = 1, 2, . . . , j do ij = aij i1 ik kj k=1 for i = j + 1, j + 2, . . . , n do ij =
1 jj

aij

j1 k=1 ik kj

If you work through a few iterations of the above procedure, you will see that the s and s that occur on the right-hand side of the two equations in the procedure are already determined by the time they are needed. And every aij is used only once and never again. Together these entries
1

You can also do it row by row except row i of L has to be determined before row i of U .

are used column by column as well. For used to occupy, namely, 11 21 31 . . .

compactness, we can store ij and ij in the location aij 12 22 32 . . . 13 23 33 . . . 1n 2n 3n . .. . . . nn

n1 n2 n3

For numerical stability, pivoting should be performed in Crouts algorithm. The key point is to notice that the rst equation in the procedure for ij is exactly the same as the second equation for ij except for the division in the latter equation. This means that we can choose the largest
j1

aij

aik kj ,
k=1

i = j, . . . , n

as the diagonal element jj and switch corresponding rows in L and A.

Example 4. To illustrate on pivoting, let us carry out a few steps of Crouts algorithm on the matrix 2 7 6 5 4 8 10 3 9 6 4 2 . 5 1 3 3 In the rst step, we need to determine 11 . Which of rows 1, 2, 3, 4 would result in the largest (absolute) value of 11 ? if row 1 if row 2 if row 3 if row 2 1 11 = 2 11 = 2,

1 11 = 4 11 = 4, 1 11 = 9 11 = 9,

1 11 = 5 11 = 5.

Thus we set 11 = 9 and exchange rows 1 and 3 in A: 9 6 4 4 8 10 2 7 6 5 1 3 4 , 9 2 , 9

2 3 . 5 3 5 . 9

In the second step, we use the rst column to determine that 21 = 31 = and 41 =

Next, we let 12 = a12 = 6 and matrices L and U take the form 1 0 0 0 9 6 0 22 4 9 0 2 0 9 5 0 0 9

To determine 22 , we nd out which of rows 2, 3, 4 would result in the largest 22 value: if row 2 if row 3 if row 4 4 32 (6) + 22 = 8 22 = , 9 3 17 2 (6) + 22 = 7 22 = , 9 3 5 13 (6) + 22 = 1 22 = . 9 3

Since row 2 yields the largest absolute value of 22 , we set 22 = 32 . Using 22 , we obtain the second column 3 of L: 13 17 and 42 = . 32 = 32 32 By this time, we have 1
4 9 2 9 5 9

0 0 1 0
17 32 13 32

0 9 0 0 0 0

6
32 3

0 0

9 4 = 2 5

6 4 2 8 10 3 . 7 6 5 1 3 3

Factorization Based on Eigenvalues

Suppose the n n matrix A has n linearly independent eigenvectors, then A = S 1 S, where is a diagonal matrix whose entries are the eigenvalues of A and S is an eigenvector matrix whose columns are the eigenvectors of A. When the eigenvalues of A are all dierent, then it is automatic that the eigenvectors are independent. Therefore A can be diagonalized. Every n n matrix can be decomposed into the Jordan form, that is, A = M JM 1 , where J = J1 .. . Js , i 1 .. .

Ji =

..

, .. . 1 i .

1 i s,

with i an eigenvalue of A. Here s is the number of independent eigenvectors of A and M consists of eigenvectors and generalized eigenvectors.

QR Factorization

Suppose A is an m n matrix with independent columns (hence m n). We can factor A as A = QR, 5

Here Q with dimensions m n has the same column space as A but its columns are orthonormal vectors. In other words, QT Q = I. And R with dimensions n n is invertible and upper triangular. The rst application is the QR algorithm which repeatedly produces QR factorizations of matrices derived from A, building the eigenvalues of A in the process. The second application is in the solution of an overconstrained system Ax = b in the least squares sense. The least-squares solution x is given by x = (AT A)1 AT b, assuming that the T Q = I, so columns of A are independent. But Q x = R1 QT b. So we can obtain x by computing QT b and then using backsubstitution to solve R = QT b. This x T A = AT b. is numerically more stable than solving the system A x The QR factorization can be be computed using the Gram-Schmidt process.

4.1

The Gram-Schmidt Procedure

Given n linearly independent vectors v 1 , . . . , v n , the Gram-Schmidt procedure constructs n orthonormal vectors u1 , . . . , un such that these two sets of vectors span the same space. First, it constructs n orthogonal vectors 1 , . . . , n as follows. w1 = v 1 ,
i1

wi = v i

j=1

v T wj i wj , wT w j j

i = 2, . . . , n.

Essentially, we subtract from every new vector v i its projection in the directions w 1 / w 1 , . . . , wi1 / w i1 already set. Next, we simply perform a normalization by letting ui =
Example 5. Consider three vectors: 1 v 1 = 1 , 0

wi , wi

i = 1, . . . , n.

2 v2 = 0 , 2

and

3 v 3 = 3 . 3

Carry out the Gram-Schmidt procedure as follows 1 w1 = 1 , 0 w2 = v2

v T w1 2 w1 wT w 1 1 1 2 2 0 1 2 0 2

1 1 , 2

w3

1 1 3 6 6 3 1 1 2 6 0 2 3 1 1 . 1 1 1 1 u3 = . 3 1

Hence the orthonormal basis consists of vectors 1 1 1 1 1 , 1 , u2 = u1 = 2 6 0 2

and

To obtain the QR factorization, we rst use Gram-Schmidt to orthogonalize the columns of A. The resulting orthonormal vectors constitute the columns of Q. The matrix R is formed by keeping track of the Gram-Schmidt operations. Specically, R expresses the column of A as linear combinations of the columns of Q.
Example 6. In the last example, we notice that v 1 = w1 = 2u1 , v 2 = w1 + w2 = 2u1 + 6u2 , v 3 = 3w1 w2 + w3 = 3 2u1 6u2 + 3u3 . So the QR decomposition is given by
1 1 6 2 1 2 3 1 1 1 0 3 = 2 6 2 0 2 3 0 6

1 3 1 3 1 3

2 2 6 0 0 0

32 6 . 3

References
[1] M. Erdmann. Lecture notes for 16-811 Mathematical Fundamentals for Robotics. The Robotics Institute, Carnegie Mellon University, 1998. [2] G. Strang. Introduction to Linear Algebra. Wellesley-Cambridge Press, 1993. [3] W. H. Press, et al. Numerical Recipes in C++: The Art of Scientic Computing. Cambridge University Press, 2nd edition, 2002.