Cs221 Linear Algebra

Linear Algebra Review
CS221: Introduction to Articial Intelligence

Carlos Fernandez-Granda
10/11/2011
CS221
10/11/2011
1 / 37
Index
Vectors Matrices Matrix Decompositions Application : Image Compression
CS221
10/11/2011
2 / 37
Vectors
CS221
10/11/2011
3 / 37
Vectors
What is a vector ?
CS221
10/11/2011
4 / 37
Vectors
What is a vector ?
An ordered tuple of numbers :
u u u=
1 2
. . .
un
CS221
10/11/2011
4 / 37
Vectors
What is a vector ?
An ordered tuple of numbers :
u u u=
1 2
. . .
un
A quantity that has a magnitude and a direction.
CS221
10/11/2011
4 / 37
Vectors
Vector spaces
More formally a vector is an element of a vector space.
CS221
10/11/2011
5 / 37
Vectors
Vector spaces
More formally a vector is an element of a vector space. A vector space V is a set that contains all linear combinations of its elements :
CS221
10/11/2011
5 / 37
Vectors
Vector spaces
If vectors u and v V , then u + v V .
CS221
10/11/2011
5 / 37
Vectors
Vector spaces
If vectors u and v V , then u + v V . If vector u V , then u V for any scalar .
CS221
10/11/2011
5 / 37
Vectors
Vector spaces
If vectors u and v V , then u + v V . If vector u V , then u V for any scalar . There exists 0 V such that u + 0 = u for any u V .
CS221
10/11/2011
5 / 37
Vectors
Vector spaces
A subspace is a subset of a vector space that is also a vector space.
If vectors u and v V , then u + v V . If vector u V , then u V for any scalar . There exists 0 V such that u + 0 = u for any u V .
CS221
10/11/2011
5 / 37
Vectors
Examples
Euclidean space Rn .
CS221
10/11/2011
6 / 37
Vectors
Examples
Euclidean space Rn .
The span of any set of vectors {u1 , u2 , . . . , un }, dened as : span (u1 , u2 , . . . , un ) = {1 u1 + 2 u2 + + n un | i R}
CS221
10/11/2011
6 / 37
Vectors
Subspaces
A line through the origin in Rn (span of a vector).
CS221
10/11/2011
7 / 37
Vectors
Subspaces
A line through the origin in Rn (span of a vector).
A plane in Rn (span of two vectors).
CS221
10/11/2011
7 / 37
Vectors
Linear Independence and Basis of Vector Spaces
A vector u is linearly independent of a set of vectors {u1 , u2 , . . . , un } if it does not lie in their span.
CS221
10/11/2011
8 / 37
Vectors
A vector u is linearly independent of a set of vectors {u1 , u2 , . . . , un } if it does not lie in their span. A set of vectors is linearly independent if every vector is linearly independent of the rest.
CS221
10/11/2011
8 / 37
Vectors
A vector u is linearly independent of a set of vectors {u1 , u2 , . . . , un } if it does not lie in their span. A set of vectors is linearly independent if every vector is linearly independent of the rest. A basis of a vector space V is a linearly independent set of vectors whose span is equal to V .
CS221
10/11/2011
8 / 37
Vectors
A vector u is linearly independent of a set of vectors {u1 , u2 , . . . , un } if it does not lie in their span. A set of vectors is linearly independent if every vector is linearly independent of the rest. A basis of a vector space V is a linearly independent set of vectors whose span is equal to V . If a vector space has a basis with d vectors, its dimension is d.
CS221
10/11/2011
8 / 37
Vectors
A vector u is linearly independent of a set of vectors {u1 , u2 , . . . , un } if it does not lie in their span. A set of vectors is linearly independent if every vector is linearly independent of the rest. A basis of a vector space V is a linearly independent set of vectors whose span is equal to V . If a vector space has a basis with d vectors, its dimension is d. Any other basis of the same space will consist of exactly d vectors.
CS221
10/11/2011
8 / 37
Vectors
A vector u is linearly independent of a set of vectors {u1 , u2 , . . . , un } if it does not lie in their span. A set of vectors is linearly independent if every vector is linearly independent of the rest. A basis of a vector space V is a linearly independent set of vectors whose span is equal to V . If a vector space has a basis with d vectors, its dimension is d. Any other basis of the same space will consist of exactly d vectors. The dimension of a vector space can be innite (function spaces).
CS221
10/11/2011
8 / 37
Vectors
Norm of a Vector
A norm ||u|| measures the magnitude of a vector.
CS221
10/11/2011
9 / 37
Vectors
Norm of a Vector
A norm ||u|| measures the magnitude of a vector. Mathematically, any function from the vector space to R that satises :
CS221
10/11/2011
9 / 37
Vectors
Norm of a Vector
Homogeneity || u|| = || ||u||
CS221
10/11/2011
9 / 37
Vectors
Norm of a Vector
Homogeneity || u|| = || ||u|| Triangle inequality ||u + v|| ||u|| + ||v||
CS221
10/11/2011
9 / 37
Vectors
Norm of a Vector
Homogeneity || u|| = || ||u|| Triangle inequality ||u + v|| ||u|| + ||v|| Point separation ||u|| = 0 if and only if u = 0.
CS221
10/11/2011
9 / 37
Vectors
Norm of a Vector
Examples :
Homogeneity || u|| = || ||u|| Triangle inequality ||u + v|| ||u|| + ||v|| Point separation ||u|| = 0 if and only if u = 0.
i |ui |. 2 i ui .
Manhattan or 1 norm : ||u||1 = Euclidean or 2 norm : ||u||2 =
CS221
10/11/2011
9 / 37
Vectors
Inner Product
Inner product between u and v : u, v =
n uv. i =1 i i
CS221
10/11/2011
10 / 37
Vectors
Inner Product
Inner product between u and v : u, v = n=1 ui vi . i It is the projection of one vector onto the other.
CS221
10/11/2011
10 / 37
Vectors
Inner Product
Inner product between u and v : u, v = n=1 ui vi . i It is the projection of one vector onto the other.
Related to the Euclidean norm : u, u = ||u||2 . 2

CS221 Linear Algebra Review 10/11/2011 10 / 37
Vectors
Inner Product
The inner product is a measure of correlation between two vectors, scaled by the norms of the vectors :
CS221
10/11/2011
11 / 37
Vectors
Inner Product
If u, v > 0, u and v are aligned.
CS221
10/11/2011
11 / 37
Vectors
Inner Product
If u, v > 0, u and v are aligned. If u, v < 0, u and v are opposed.
CS221
10/11/2011
11 / 37
Vectors
Inner Product
If u, v > 0, u and v are aligned. If u, v < 0, u and v are opposed. If u, v = 0, u and v are orthogonal.
Vectors
Orthonormal Basis
The vectors in an orthonormal basis have unit Euclidean norm and are orthogonal to each other.
CS221
10/11/2011
12 / 37
Vectors
Orthonormal Basis
The vectors in an orthonormal basis have unit Euclidean norm and are orthogonal to each other. To express a vector x in an orthonormal basis, we can just take inner products.
CS221
10/11/2011
12 / 37
Vectors
Orthonormal Basis
The vectors in an orthonormal basis have unit Euclidean norm and are orthogonal to each other. To express a vector x in an orthonormal basis, we can just take inner products. Example : x = 1 b1 + 2 b2 .
CS221
10/11/2011
12 / 37
Vectors
Orthonormal Basis
The vectors in an orthonormal basis have unit Euclidean norm and are orthogonal to each other. To express a vector x in an orthonormal basis, we can just take inner products. Example : x = 1 b1 + 2 b2 . We compute x, b1 :
x, b1 = 1 b1 + 2 b2 , b1
By linearity. = 1 + 0 By orthonormality.
= 1 b1 , b1 + 2 b2 , b1
CS221
10/11/2011
12 / 37
Vectors
Orthonormal Basis
The vectors in an orthonormal basis have unit Euclidean norm and are orthogonal to each other. To express a vector x in an orthonormal basis, we can just take inner products. Example : x = 1 b1 + 2 b2 . We compute x, b1 :
x, b1 = 1 b1 + 2 b2 , b1
By linearity. = 1 + 0 By orthonormality.
= 1 b1 , b1 + 2 b2 , b1
Likewise, 2 = x, b2 .
Matrices
CS221
10/11/2011
13 / 37
Matrices
Linear Operator
A linear operator L : U V is a map from a vector space U to another vector space V that satises :
CS221
10/11/2011
14 / 37
Matrices
Linear Operator
L (u1 + u2 ) = L (u1 ) + L (u2 )
CS221
10/11/2011
14 / 37
Matrices
Linear Operator
L (u1 + u2 ) = L (u1 ) + L (u2 ) L ( u) = L (u)
. for any scalar .
CS221
10/11/2011
14 / 37
Matrices
Linear Operator
A linear operator L : U V is a map from a vector space U to another vector space V that satises : If the dimension n of U and an m n matrix A.

L (u1 + u2 ) = L (u1 ) + L (u2 ) L ( u) = L (u)
. for any scalar .

A A
m of V are nite, L can be represented by

11 21 12 22
A A A=
Am
Am
... ... ... ...
An A n
1
Amn
CS221
10/11/2011
14 / 37
Matrices
Matrix Vector Multiplication
A A Au =

11 21
A A
12 22
Am
Am
... ... ... ...
u An A nu
1 2
1 2
Amn
. . .
v v =

un
2 . . .
vm
CS221
10/11/2011
15 / 37
Matrices
A A Au =

11 21
A A
12 22
Am
Am
... ... ... ...
u An A nu
1 2
1 2
Amn
. . .
v v =

un
2 . . .
vm
CS221
10/11/2011
15 / 37
Matrices
A A Au =

11 21
A A
12 22
Am
Am
... ... ... ...
u An A nu
1 2
1 2
Amn
. . .
v v =

un
2 . . .
vm
CS221
10/11/2011
15 / 37
Matrices
A A Au =

11 21
A A
12 22
Am
Am
... ... ... ...
u An A nu
1 2
1 2
Amn
. . .
v v =

un
2 . . .
vm
CS221
10/11/2011
15 / 37
Matrices
Matrix Product
Applying two linear operators A and B denes another linear operator, which can be represented by another matrix AB.
A A AB =

11 21
A A
12 22
Am Am AB AB AB AB =
1 2 11 21
... ... ... ...

12 22
B An B A n
1 2

1 2
11 21
B B
12 22
ABm
ABm
... ...
Amn Bn . . . AB p . . . AB p ABmp
Bn
... ... ... ...
Bp B p
1 2
Bnp
Note that if A is
CS221
m n and B is n p, then AB is m p.
Linear Algebra Review 10/11/2011 16 / 37
Matrices
Matrix Product
A A AB =

11 21
A A
12 22
Am Am AB AB AB AB =
1 2 11 21
... ... ... ...

12 22
B An B A n
1 2

1 2
11 21
B B
12 22
ABm
ABm
... ...
Bn
... ... ... ...
Bp B p
1 2
Bnp
Note that if A is
CS221
Matrices
Matrix Product
A A AB =

11 21
A A
12 22
Am Am AB AB AB AB =
1 2 11 21
... ... ... ...

12 22
B An B A n
1 2

1 2
11 21
B B
12 22
ABm
ABm
... ...
Bn
... ... ... ...
Bp B p
1 2
Bnp
Note that if A is
CS221
Matrices
Matrix Product
A A AB =

11 21
A A
12 22
Am Am AB AB AB AB =
1 2 11 21
... ... ... ...

12 22
B An B A n
1 2

1 2
11 21
B B
12 22
ABm
ABm
... ...
Bn
... ... ... ...
Bp B p
1 2
Bnp
Note that if A is
CS221
Matrices
Matrix Product
A A AB =

11 21
A A
12 22
Am Am AB AB AB AB =
1 2 11 21
... ... ... ...

12 22
B An B A n
1 2

1 2
11 21
B B
12 22
ABm
ABm
... ...
Bn
... ... ... ...
Bp B p
1 2
Bnp
Note that if A is
CS221
Matrices
Matrix Product
A A AB =

11 21
A A
12 22
Am Am AB AB AB AB =
1 2 11 21
... ... ... ...

12 22
B An B A n
1 2

1 2
11 21
B B
12 22
ABm
ABm
... ...
Bn
... ... ... ...
Bp B p
1 2
Bnp
Note that if A is
CS221
Matrices
Matrix Product
A A AB =

11 21
A A
12 22
Am Am AB AB AB AB =
1 2 11 21
... ... ... ...

12 22
B An B A n
1 2

1 2
11 21
B B
12 22
ABm
ABm
... ...
Bn
... ... ... ...
Bp B p
1 2
Bnp
Note that if A is
CS221
Matrices
Matrix Product
A A AB =

11 21
A A
12 22
Am Am AB AB AB AB =
1 2 11 21
... ... ... ...

12 22
B An B A n
1 2

1 2
11 21
B B
12 22
ABm
ABm
... ...
Bn
... ... ... ...
Bp B p
1 2
Bnp
Note that if A is
CS221
Matrices
Matrix Product
A A AB =

11 21
A A
12 22
Am Am AB AB AB AB =
1 2 11 21
... ... ... ...

12 22
B An B A n
1 2

1 2
11 21
B B
12 22
ABm
ABm
... ...
Bn
... ... ... ...
Bp B p
1 2
Bnp
Note that if A is
CS221
Matrices
Matrix Product
A A AB =

11 21
A A
12 22
Am Am AB AB AB AB =
1 2 11 21
... ... ... ...

12 22
B An B A n
1 2

1 2
11 21
B B
12 22
ABm
ABm
... ...
Bn
... ... ... ...
Bp B p
1 2
Bnp
Note that if A is
CS221
Matrices
Transpose of a Matrix
The transpose of a matrix is obtained by ipping the rows and columns. A11 A21 . . . An1 A12 A22 . . . An2 AT =
... A1m A2m . . .
Anm
CS221
10/11/2011
17 / 37
Matrices
... A1m A2m . . .
Anm
Some simple properties :
CS221
10/11/2011
17 / 37
Matrices
... A1m A2m . . .
Anm

AT
T
=A
CS221
10/11/2011
17 / 37
Matrices
... A1m A2m . . .
Anm
T AT = A T (A + B) = AT + BT
CS221
10/11/2011
17 / 37
Matrices
... A1m A2m . . .
Anm
T AT = A T (A + B) = AT + BT T (AB) = BT AT
CS221
10/11/2011
17 / 37
Matrices
... A1m A2m . . .
Anm
The inner product for vectors can be represented as u, v = uT v.
T AT = A T (A + B) = AT + BT T (AB) = BT AT
CS221
10/11/2011
17 / 37
Matrices
Identity Operator
I=
1 0 ... 0 1 . . .
... 0 0 ...
0 0

CS221
10/11/2011
18 / 37
Matrices
Identity Operator
I=
1 0 ... 0 1 . . .
... 0 0 ...
0 0

For any matrix A, AI = A.
CS221
10/11/2011
18 / 37
Matrices
Identity Operator
I=
1 0 ... 0 1 . . .
... 0 0 ...
0 0

For any matrix A, AI = A. I is the identity operator for the matrix product.
CS221
10/11/2011
18 / 37
Matrices
Column Space, Row Space and Rank
Let A be an
m n matrix.
CS221
10/11/2011
19 / 37
Matrices
Let A be an
m n matrix.
:
Column space
CS221
10/11/2011
19 / 37
Matrices
Let A be an
m n matrix.
:
Column space
Span of the columns of A.
CS221
10/11/2011
19 / 37
Matrices
Let A be an
m n matrix.
:
Column space
Span of the columns of A. Linear subspace of Rm .
CS221
10/11/2011
19 / 37
Matrices
Let A be an
m n matrix.
:
Column space

:
Row space
CS221
10/11/2011
19 / 37
Matrices
Let A be an
m n matrix.
:
Column space

:
Row space
Span of the rows of A.
CS221
10/11/2011
19 / 37
Matrices
Let A be an
m n matrix.
:
Column space

:
Row space
Span of the rows of A. Linear subspace of Rn .
CS221
10/11/2011
19 / 37
Matrices
Let A be an
m n matrix.
:
Column space

:
Row space
Important fact :
The column and row spaces of any matrix have the same dimension.
CS221
10/11/2011
19 / 37
Matrices
Let A be an
m n matrix.
:
Column space

:
Row space
Important fact :
The column and row spaces of any matrix have the same dimension. This dimension is the rank of the matrix.
CS221
10/11/2011
19 / 37
Matrices
Range and Null space
Let A be an
m n matrix.
CS221
10/11/2011
20 / 37
Matrices
Let A be an Range :
m n matrix.
CS221
10/11/2011
20 / 37
Matrices
Let A be an Range :
m n matrix.
Set of vectors equal to Au for some u Rn .
CS221
10/11/2011
20 / 37
Matrices
Let A be an Range :
m n matrix.
Set of vectors equal to Au for some u Rn . Linear subspace of Rm .
CS221
10/11/2011
20 / 37
Matrices
Let A be an Range :
Null space
m n matrix.

:
CS221
10/11/2011
20 / 37
Matrices
Let A be an Range :
Null space
m n matrix.
Set of vectors equal to Au for some u Rn . Linear subspace of Rm . belongs to the null space if Au = 0.
:
CS221
10/11/2011
20 / 37
Matrices
Let A be an Range :
Null space
m n matrix.

:
belongs to the null space if Au = 0. Subspace of Rn .

u
CS221
10/11/2011
20 / 37
Matrices
Let A be an Range :
Null space
m n matrix.

:
belongs to the null space if Au = 0. Subspace of Rn . Every vector in the null space is orthogonal to the rows of A. The null space and row space of a matrix are orthogonal.
u
CS221
10/11/2011
20 / 37
Matrices
Range and Column Space
Another interpretation of the matrix vector product (Ai is column i of A) :
Au =
A1
A2
...
An
u u
2 . . .
= u1 A1 + u2 A2 + + un An
un
CS221
10/11/2011
21 / 37
Matrices
Au =
A1
A2
...
An
u u
2 . . .
= u1 A1 + u2 A2 + + un An
un
The result is a linear combination of the columns of A.
CS221
10/11/2011
21 / 37
Matrices
Au =
A1
A2
...
An
u u
2 . . .
= u1 A1 + u2 A2 + + un An
un
The result is a linear combination of the columns of A. For any matrix, the range is equal to the column space.
CS221
10/11/2011
21 / 37
Matrices
Matrix Inverse
For an
n n matrix A : rank + dim(null space) = n.
CS221
10/11/2011
22 / 37
Matrices
Matrix Inverse
For an n n matrix A : rank + dim(null space) = n. If dim(null space)=0 then A is full rank.
CS221
10/11/2011
22 / 37
Matrices
Matrix Inverse
For an n n matrix A : rank + dim(null space) = n. If dim(null space)=0 then A is full rank. In this case, the action of the matrix is invertible.
CS221
10/11/2011
22 / 37
Matrices
Matrix Inverse
For an n n matrix A : rank + dim(null space) = n. If dim(null space)=0 then A is full rank. In this case, the action of the matrix is invertible. The inversion is also linear and consequently can be represented by another matrix A1 .
CS221
10/11/2011
22 / 37
Matrices
Matrix Inverse
For an n n matrix A : rank + dim(null space) = n. If dim(null space)=0 then A is full rank. In this case, the action of the matrix is invertible. The inversion is also linear and consequently can be represented by another matrix A1 . A1 is the only matrix such that A1 A = AA1 = I.
CS221
10/11/2011
22 / 37
Matrices
Orthogonal Matrices
An orthogonal matrix U satises UT U = I.
CS221
10/11/2011
23 / 37
Matrices
Orthogonal Matrices
An orthogonal matrix U satises UT U = I. Equivalently, U has orthonormal columns.
CS221
10/11/2011
23 / 37
Matrices
Orthogonal Matrices
An orthogonal matrix U satises UT U = I. Equivalently, U has orthonormal columns. Applying an orthogonal matrix to two vectors does not change their inner product :
Uu, Uv = (Uu) Uv
= uT UT Uv = uT v = u, v
CS221
10/11/2011
23 / 37
Matrices
Orthogonal Matrices
An orthogonal matrix U satises UT U = I. Equivalently, U has orthonormal columns. Applying an orthogonal matrix to two vectors does not change their inner product :
Uu, Uv = (Uu) Uv
= uT UT Uv = uT v = u, v
Example : Matrices that represent rotations are orthogonal.
CS221
10/11/2011
23 / 37
Matrices
Trace and Determinant
The trace is the sum of the diagonal elements of a square matrix.
CS221
10/11/2011
24 / 37
Matrices
The trace is the sum of the diagonal elements of a square matrix. The determinant of a square matrix A is denoted by |A|.
CS221
10/11/2011
24 / 37
Matrices
The trace is the sum of the diagonal elements of a square matrix. The determinant of a square matrix A is denoted by |A|. ab For a 2 2 matrix A = c d :
CS221
10/11/2011
24 / 37
Matrices
|A| = ad bc
CS221
10/11/2011
24 / 37
Matrices
|A| = ad bc
The absolute value of |A| is the area of the parallelogram given by the rows of A.
CS221
10/11/2011
24 / 37
Matrices
Properties of the Determinant
The denition can be generalized to larger matrices.
CS221
10/11/2011
25 / 37
Matrices
The denition can be generalized to larger matrices. |A| = AT .
CS221
10/11/2011
25 / 37
Matrices

|AB| = |A| |B|
CS221
10/11/2011
25 / 37
Matrices

|AB| = |A| |B| |A| = 0 if and only if A is not invertible.
CS221
10/11/2011
25 / 37
Matrices

|AB| = |A| |B| |A| = 0 if and only if A is not invertible.
If A is invertible, then A1 =
|A | .
1
CS221
10/11/2011
25 / 37
Matrix Decompositions
CS221
10/11/2011
26 / 37
Eigenvalues and Eigenvectors
An eigenvalue of a square matrix A satises :

Au = u
for some vector u, which we call an eigenvector.
CS221
10/11/2011
27 / 37

Au = u
for some vector u, which we call an eigenvector. Geometrically, the operator A expands ( > 1) or contracts ( < 1) eigenvectors but does not rotate them.
CS221
10/11/2011
27 / 37

Au = u
for some vector u, which we call an eigenvector. Geometrically, the operator A expands ( > 1) or contracts ( < 1) eigenvectors but does not rotate them. If u is an eigenvector of A, it is in the null space of A I, which is consequently not invertible.
CS221
10/11/2011
27 / 37

Au = u
for some vector u, which we call an eigenvector. Geometrically, the operator A expands ( > 1) or contracts ( < 1) eigenvectors but does not rotate them. If u is an eigenvector of A, it is in the null space of A I, which is consequently not invertible. The eigenvalues of A are the roots of the equation |A I| = 0.
CS221
10/11/2011
27 / 37

Au = u
for some vector u, which we call an eigenvector. Geometrically, the operator A expands ( > 1) or contracts ( < 1) eigenvectors but does not rotate them. If u is an eigenvector of A, it is in the null space of A I, which is consequently not invertible. The eigenvalues of A are the roots of the equation |A I| = 0. Not the way eigenvalues are calculated in practice.
CS221
10/11/2011
27 / 37

Au = u
for some vector u, which we call an eigenvector. Geometrically, the operator A expands ( > 1) or contracts ( < 1) eigenvectors but does not rotate them. If u is an eigenvector of A, it is in the null space of A I, which is consequently not invertible. The eigenvalues of A are the roots of the equation |A I| = 0. Not the way eigenvalues are calculated in practice. Eigenvalues and eigenvectors can be complex valued, even if all the entries of A are real.
Eigendecomposition of a Matrix
Let A be an n n square matrix with n linearly independent eigenvectors p1 . . . pn and eigenvalues 1 . . . n .
CS221
10/11/2011
28 / 37
Let A be an n n square matrix with n linearly independent eigenvectors p1 . . . pn and eigenvalues 1 . . . n . We dene the matrices :
P= p1 . . . pn
1 0 . . . 0 2 . . . = ... 0 0 ...
0 0
n
CS221
10/11/2011
28 / 37
P= p1 . . . pn
1 0 . . . 0 2 . . . = ... 0 0 ...
A
0 0
n
satises AP = P.
CS221
10/11/2011
28 / 37
P= p1 . . . pn
1 0 . . . 0 2 . . . = ... 0 0 ...
A P
0 0
n
satises AP = P. is full rank. Inverting it yields the eigendecomposition of A :

A = P P 1
CS221
10/11/2011
28 / 37
Properties of the Eigendecomposition
Not all matrices are diagonalizable (have an eigendecomposition). Example : ( 1 1 ). 0 1
CS221
10/11/2011
29 / 37
Not all matrices are diagonalizable (have an eigendecomposition). Example : ( 1 1 ). 0 1 Trace(A) = Trace() = n=1 i . i
CS221
10/11/2011
29 / 37
Not all matrices are diagonalizable (have an eigendecomposition). Example : ( 1 1 ). 0 1 Trace(A) = Trace() = n=1 i . i |A| = || = n=1 i . i
CS221
10/11/2011
29 / 37
Not all matrices are diagonalizable (have an eigendecomposition). Example : ( 1 1 ). 0 1 Trace(A) = Trace() = n=1 i . i |A| = || = n=1 i . i The rank of A is equal to the number of nonzero eigenvalues.
CS221
10/11/2011
29 / 37
Not all matrices are diagonalizable (have an eigendecomposition). Example : ( 1 1 ). 0 1 Trace(A) = Trace() = n=1 i . i |A| = || = n=1 i . i The rank of A is equal to the number of nonzero eigenvalues. 1 If is a nonzero eigenvalue of A, is an eigenvalue of A1 with the same eigenvector.
CS221
10/11/2011
29 / 37
Not all matrices are diagonalizable (have an eigendecomposition). Example : ( 1 1 ). 0 1 Trace(A) = Trace() = n=1 i . i |A| = || = n=1 i . i The rank of A is equal to the number of nonzero eigenvalues. 1 If is a nonzero eigenvalue of A, is an eigenvalue of A1 with the same eigenvector. The eigendecomposition makes it possible to compute matrix powers eciently : m Am = PP1 = PP1 PP1 . . . PP1 = Pm P1 .
CS221
10/11/2011
29 / 37
Matlab Example
CS221
10/11/2011
30 / 37
Eigendecomposition of a Symmetric Matrix
If A = AT then A is symmetric.
CS221
10/11/2011
31 / 37
If A = AT then A is symmetric. The eigenvalues of symmetric matrices are real.
CS221
10/11/2011
31 / 37
If A = AT then A is symmetric. The eigenvalues of symmetric matrices are real. The eigenvectors of symmetric matrices are orthonormal.
CS221
10/11/2011
31 / 37
If A = AT then A is symmetric. The eigenvalues of symmetric matrices are real. The eigenvectors of symmetric matrices are orthonormal. Consequently, the eigendecomposition becomes : A = UUT for real and U orthogonal.
CS221
10/11/2011
31 / 37
If A = AT then A is symmetric. The eigenvalues of symmetric matrices are real. The eigenvectors of symmetric matrices are orthonormal. Consequently, the eigendecomposition becomes : A = UUT for real and U orthogonal. The eigenvectors of A are an orthonormal basis for the column space (and the row space, since they are equal).
CS221
10/11/2011
31 / 37
The action of a symmetric matrix on a vector Au = UUT u can be decomposed into :
CS221
10/11/2011
32 / 37

1
Projection of u onto the column space of A (multiplication by UT ).
CS221
10/11/2011
32 / 37

1 2
Projection of u onto the column space of A (multiplication by UT ). Scaling of each coecient Ui , u by the corresponding eigenvalue (multiplication by ).
CS221
10/11/2011
32 / 37

1 2 3
Projection of u onto the column space of A (multiplication by UT ). Scaling of each coecient Ui , u by the corresponding eigenvalue (multiplication by ). Linear combination of the eigenvectors scaled by the resulting coecient (multiplication by U).
CS221
10/11/2011
32 / 37

1 2 3
Summarizing :
n
Au =
i =1
i Ui , u Ui
CS221
10/11/2011
32 / 37

1 2 3
Summarizing :
n
Au =
i Ui , u Ui i =1 It would be great to generalize this to all matrices...
CS221
10/11/2011
32 / 37
Singular Value Decomposition
Every
matrix has a singular value decomposition :

A = UVT
CS221
10/11/2011
33 / 37
Every

A = UVT
A.
The columns of U are an orthonormal basis for the column space of
CS221
10/11/2011
33 / 37
Every

A = UVT
A.
The columns of U are an orthonormal basis for the column space of The columns of V are an orthonormal basis for the row space of A.
CS221
10/11/2011
33 / 37
Every

A = UVT
A.
The columns of U are an orthonormal basis for the column space of The columns of V are an orthonormal basis for the row space of A. is diagonal. The entries on its diagonal i = ii are positive real numbers, called the singular values of A.
CS221
10/11/2011
33 / 37
Every

A = UVT
A.
The columns of U are an orthonormal basis for the column space of The columns of V are an orthonormal basis for the row space of A. is diagonal. The entries on its diagonal i = ii are positive real numbers, called the singular values of A. The action of A on a vector u can be decomposed into :
n
Au =
i =1
i Vi , u Ui
CS221
10/11/2011
33 / 37
Properties of the Singular Value Decomposition
The eigenvalues of the symmetric matrix AT A are equal to the square of the singular values of A :
AT A = VUT UVT = V2 VT
CS221
10/11/2011
34 / 37
The rank of a matrix is equal to the number of nonzero singular values.
CS221
10/11/2011
34 / 37
The rank of a matrix is equal to the number of nonzero singular values. The largest singular value 1 is the solution to the optimization problem :
1 = max
x=0
||Ax||2 ||x||2
CS221
10/11/2011
34 / 37
1 = max
x=0
||Ax||2 ||x||2
It can be veried that the largest singular value satises the properties of a norm, it is called the spectral norm of the matrix.
CS221
10/11/2011
34 / 37
1 = max
x=0
||Ax||2 ||x||2
It can be veried that the largest singular value satises the properties of a norm, it is called the spectral norm of the matrix. In statistics analyzing data with the singular value decomposition is called Principal Component Analysis.
Application : Image Compression
CS221
10/11/2011
35 / 37
Compression using the Singular Value Decomposition
The singular value decomposition can be viewed as a sum of rank 1 matrices :
CS221
10/11/2011
36 / 37
If some of the singular values are very small, we can discard them.
CS221
10/11/2011
36 / 37
If some of the singular values are very small, we can discard them. This is a form of lossy compression.
CS221
10/11/2011
36 / 37
Truncating the singular value decomposition allows us to represent the matrix with less parameters.
CS221
10/11/2011
37 / 37
For a 512 512 matrix :
CS221
10/11/2011
37 / 37
Full representation : 512 512 =262 144
CS221
10/11/2011
37 / 37
Full representation : 512 512 =262 144 Rank 10 approximation : 512 10 + 10 + 512 10 =10 250
CS221
10/11/2011
37 / 37
Full representation : 512 512 =262 144 Rank 10 approximation : 512 10 + 10 + 512 10 =10 250 Rank 40 approximation : 512 40 + 40 + 512 40 =41 000
CS221
10/11/2011
37 / 37
Full representation : 512 512 =262 144 Rank 10 approximation : 512 10 + 10 + 512 10 =10 250 Rank 40 approximation : 512 40 + 40 + 512 40 =41 000 Rank 80 approximation : 512 80 + 80 + 512 80 =82 000
CS221

Cs221 Linear Algebra

Загружено:

Сведения о документе

Исходное описание:

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

Cs221 Linear Algebra

Загружено:

Авторское право:

Доступные форматы

Linear Algebra Review

CS221: Introduction to Articial Intelligence

Linear Algebra Review

Vectors Matrices Matrix Decompositions Application : Image Compression

Linear Algebra Review

Vectors Matrices Matrix Decompositions Application : Image Compression

Linear Algebra Review

Linear Algebra Review

An ordered tuple of numbers :

Linear Algebra Review

An ordered tuple of numbers :

A quantity that has a magnitude and a direction.

Linear Algebra Review

More formally a vector is an element of a vector space.

Linear Algebra Review

Linear Algebra Review

If vectors u and v V , then u + v V .

Linear Algebra Review

If vectors u and v V , then u + v V . If vector u V , then u V for any scalar .

Linear Algebra Review

Linear Algebra Review

A subspace is a subset of a vector space that is also a vector space.

Linear Algebra Review

Linear Algebra Review

The span of any set of vectors {u1 , u2 , . . . , un }, dened as : span (u1 , u2 , . . . , un ) = {1 u1 + 2 u2 + + n un | i R}

Linear Algebra Review

A line through the origin in Rn (span of a vector).

Linear Algebra Review

A line through the origin in Rn (span of a vector).

A plane in Rn (span of two vectors).

Linear Algebra Review

Linear Independence and Basis of Vector Spaces

Linear Algebra Review

Linear Independence and Basis of Vector Spaces

Linear Algebra Review

Linear Independence and Basis of Vector Spaces

Linear Algebra Review

Linear Independence and Basis of Vector Spaces

Linear Algebra Review

Linear Independence and Basis of Vector Spaces

Linear Algebra Review

Linear Independence and Basis of Vector Spaces

Linear Algebra Review

A norm ||u|| measures the magnitude of a vector.

Linear Algebra Review

Linear Algebra Review

Homogeneity || u|| = || ||u||

Linear Algebra Review

Homogeneity || u|| = || ||u|| Triangle inequality ||u + v|| ||u|| + ||v||

Linear Algebra Review

Linear Algebra Review

Manhattan or 1 norm : ||u||1 = Euclidean or 2 norm : ||u||2 =

Linear Algebra Review

Inner product between u and v : u, v =

Linear Algebra Review

Linear Algebra Review

Related to the Euclidean norm : u, u = ||u||2 . 2

Linear Algebra Review

If u, v > 0, u and v are aligned.

Linear Algebra Review

If u, v > 0, u and v are aligned. If u, v < 0, u and v are opposed.

Linear Algebra Review

Linear Algebra Review

CS221: Introduction to Articial Intelligence

The span of any set of vectors {u1 , u2 , . . . , un }, dened as : span (u1 , u2 , . . . , un ) = {1 u1 + 2 u2 + + n un | i R}

m of V are nite, L can be represented by