Linear Algebra Guide to Eigenvalues & Eigenvectors

Eigenvalues & Eigenvectors
Paweł Polak
October 17, 2019
Linear Algebra - Lec. 7
1/69
Paweł Polak (Stevens Institute of Technology) MA 232: Linear Algebra - Lec. 7
Content:
Finding Eigenvalues and Eigenvectors
Special Types of Matrices
Solving Recursive Equations
Systems of First Order Linear Differential Equations
Second Order Linear Differential Equations
Difference Equations
Stability of 2 × 2 Matrices
The Exponential of a Matrix
Symmetric Matrices and Spectral Theorem
Positive Definite Matrices 2/69

Motivation: Calculating Powers of a Matrix
Example

4 −2
Let A = . Find A100 .
−1 3
How can we do this efficiently?

1 −2
Consider the matrix P = . Observe that P is invertible
1 1
(why?), and that
−1 1 1 2
P = .
3 −1 1
Furthermore,

1 1 2 4 −2 1 −2 2 0
P −1 AP = = = D,
3 −1 1 −1 3 1 1 0 5
where D is a diagonal matrix.

3/69
Example (continued)
This is significant, because
P −1 AP = D
−1 −1
P(P AP)P = PDP −1
−1 −1
(PP )A(PP ) = PDP −1
IAI = PDP −1
A = PDP −1 ,
and so
A100 = (PDP −1 )100

= (PDP −1 )(PDP −1 )(PDP −1 ) · · · (PDP −1 )
= PD(P −1 P)D(P −1 P)D(P −1 · · · P)DP −1
= PDIDIDI · · · IDP −1
= PD 100 P −1 .
4/69
Example (continued)
Now, 100
2100

100 2 0 0
D = = .
0 5 0 5100
Therefore,
100 −1
A100 = PD P
2100

1 −2 0 1 1 2
=
1 1 0 5100 3 −1 1
1 2100 + 2 · 5100 2100 − 2 · 5100

=
3 2100 − 5100 2 · 2100 + 5100
1 2100 + 2 · 5100 2100 − 2 · 5100

=
3 2100 − 5100 2101 + 5100
5/69
Theorem
If A is an n × n matrix and P is an invertible n × n matrix such that
A = PDP −1 , then Ak = PD k P −1 for each k = 1, 2, 3, . . .
The process of finding an invertible matrix P and a diagonal matrix D so

that A = PDP −1 is referred to as diagonalizing the matrix A, and P is
called the diagonalizing matrix for A.
Questions
When is it possible to diagonalize a matrix?
How do we find a diagonalizing matrix?
Answer
Eigenvalues and eigenvectors.
6/69
Eigenvalues and Eigenvectors
Definition
Let A be an n × n matrix, λ a real number, and u 6= 0 an n-vector. If
Au = λu, then λ is an eigenvalue of A, and u is an eigenvector of A
corresponding to λ, or a λ-eigenvector.
Eigenvectors are vectors which do not change the direction when

multiplied by A.
Example

1 2 1
Let A = and u = . Then
1 2 1

1 2 1 3 1
Au = = =3 = 3u.
1 2 1 3 1

1
This means that 3 is an eigenvalue of A, and is an eigenvector of
1
A corresponding to 3 (or a 3-eigenvector of A).
7/69
Examples
Example
Projection matrices, P = A(AT A)−1 AT , have all the eigenvalues
equal to 0 or 1.
If u ∈ C (P), then Pu = u so λ = 1.
If u ∈ N(P), then Pu = 0u so λ = 0.
By Fundamental Theorem of Linear Algebra,
dim(C (P)) + dim(N(P)) = m. Hence, there are no other
eigenvalues.
The reflection matrix R = 2P − I has eigenvalues 1 and −1, and the
eigenvectors are the same as matrix P.
(Intuition: Every reflection matrix can be written as R = 2P − I ,
where P is the projection matrix on the “mirror”, i.e.,
reflection = 2(projection) − I . So λR = 2λP − 1).
What about rotation matrices? It is not so easy and intuitive.
Example: let Q be a 2 × 2 rotation
√ matrix which rotates vectors by
90o , then Q 2 u = −u. So λ = −1, i.e., the eigenvalues are
complex numbers i and −i.
8/69
What an eigenvalue and eigenvector tell us about a matrix?
Suppose that A is an n × n matrix, with eigenvalue λ and corresponding
eigenvector u. Then u 6= 0 is an n-vector, λ ∈ R, and Au = λu.
It follows that
Au − λu = 0
Au − λI u = 0
(A − λI )u = 0
Since u 6= 0, u is a nontrivial solution to the linear system with

coefficient matrix A − λI , and therefore the matrix A − λI is not
invertible. Since a matrix is invertible if and only if its determinant is not
equal to zero, it follows that
det(A − λI ) = 0.
9/69
The Characteristic Polynomial
Definition
The characteristic polynomial of an n × n matrix A is defined to be
cA (x) = det(A − xI ).
Example

4 −2
The characteristic polynomial of A = is
−1 3
cA (x) = det(A − xI )

4 −2 x 0
= det −
−1 3 0 x

4−x −2
= det
−1 3 − x
= (4 − x)(3 − x) − 2
= x 2 − 7x + 10.
10/69
Finding Eigenvalues and Eigenvectors
Theorem
Let A be an n × n matrix.
1 The eigenvalues of A are the roots of cA (x).
2 The λ-eigenvectors u are the nontrivial solutions to (A − λI )u = 0.
Procedure:
Let A be an n × n matrix.
Eigenvalues: Find λ by solving the equation
cA (x) = det(A − xI ) = 0
Eigenvectors: For each λ, find u 6= 0 by finding the basic solutions

to
(A − λI )u = 0
Check: For each pair of λ, u check that Au = λu.
11/69
Example (continued)

4 −2
For A = , we’ve found
−1 3
cA (x) = x 2 − 7x + 10 = (x − 2)(x − 5),
so A has eigenvalues λ1 = 2 and λ2 = 5.

The 2-eigenvectors of A (meaning the eigenvectors of A corresponding to
λ1 = 2) are found by solving the homogeneous system (A − 2I )u = 0.
This is the homogeneous system with coefficient matrix:

4 −2 1 0 2 −2
A − 2I = −2 = .
−1 3 0 1 −1 1
12/69
Example (continued)
Solve the system in the standard way, by putting the augmented matrix
of the system in reduced row-echelon form.

2 −2 0 1 −1 0
→ .
−1 1 0 0 0 0
The general solution is

t 1
u= =t where t ∈ R.
t 1
However, since eigenvectors are nonzero, the 2-eigenvectors of A are all

vectors
1
u=t where t ∈ R and t 6= 0.
1
13/69
Example (continued)

4 −2
To find the 5-eigenvectors of A = solve the homogeneous
−1 3
system (A − 5I )u = 0, with coefficient matrix

4 −2 1 0 −1 −2
A − 5I = −5 = .
−1 3 0 1 −1 −2

−1 −2 0 1 2 0
→ .
−1 −2 0 0 0 0
Therefore the 5-eigenvectors of A are the vectors

−2s −2
u= =s where s ∈ R and s 6= 0.
s 1
14/69
Basic Eigenvectors
Definition
A basic eigenvector of an n × n matrix A is any nonzero multiple of a
basic solution to (A − λI )u = 0, where λ is an eigenvalue of A.

4 −2
Basic eigenvectors of A =
−1 3

1
u= is is a basic eigenvector of A corresponding to the eigenvalue
1
−2
2. u = is a basic eigenvector of A corresponding to the
1
eigenvalue 5.
15/69
Eigenvalues with multiplicity greater than one
Problem
Find the characteristic polynomial and eigenvalues of the matrix
 
4 1 2
A= 0 3 −2  .
0 −1 2
Solution
 
4−x 1 2
cA (x) = det(A − xI ) = det  0 3−x −2 
0 −1 2−x
= (4 − x)[(3 − x)(2 − x) − 2]
= (4 − x)(x 2 − 5x + 4)
= −(x − 4)(x − 4)(x − 1)
= −(x − 4)2 (x − 1).
Therefore, A has eigenvalues 1 and 4, with 4 being an eigenvalue of
multiplicity two. 16/69
Definition
The multiplicity of an eigenvalue λ of A is the number of times λ occurs
as a root of cA (x).
Example
 
4 1 2
We have seen that A =  0 3 −2  has eigenvalues λ1 = 1 and
0 −1 2
λ2 = 4 of multiplicity two. To find an eigenvector of A corresponding to
λ1 = 1, solve the homogeneous system (A − I )u = 0:
   
3 1 2 0 1 0 1 0
 0 2 −2 0  →  0 1 −1 0  .
0 −1 1 0 0 0 0 0
 
−s
The general solution is u =  s  where s ∈ R. We get a basic
s
eigenvector by choosing s = 1 (in fact, any nonzero value of s gives us a
basic eigenvector).
17/69
Example (continued)
 
−1
Therefore, u =  1  is a (basic) eigenvector of A corresponding to
1
λ1 = 1.  
4 1 2
To find an eigenvector of A =  0 3 −2  corresponding the
0 −1 2
λ2 = 4, solve the system (A − 4I )u = 0:
   
0 1 2 0 0 1 2 0
 0 −1 −2 0  →  0 0 0 0  .
0 −1 −2 0 0 0 0 0
 
s
The general solution is u =  −2t  where s, t ∈ R.
t
18/69
Example (continued)
In this case, the general solution has two parameters, which leads to two
basic eigenvectors that are not scalar multiples of each other, i.e., since
     
s 1 0
u =  −2t  = s  0  + t  −2  where s, t ∈ R,
t 0 1
we obtain basic eigenvectors

   
1 0
u1 =  0  and u2 =  −2  .
0 1
We can obtain other pairs of basic 4-eigenvectors for A by taking any

nonzero scalar multiple of u1 , and any nonzero scalar multiple of u2 .
Notice that every 4-eigenvector of A is a nonzero linear combination of

basic 4-eigenvectors.
19/69
Problem
For  
3 −4 2
A= 1 −2 2 ,
1 −5 5
find cA (x), the eigenvalues of A, and basic eigenvector(s) for each
eigenvalue.
Solution

3−x −4 2 3−x −4 2

det(A − xI ) = 1
−2 − x 2 = 0
3−x x −3

1 −5 5−x 1 −5 5−x

3−x −4 −2
3 − x −2
= 0 3−x 0 = (3 − x)
1 −x
1 −5 −x
Therefore, cA (x) = (x − 3)(x 2 − 3x + 2) = (x − 3)(x − 2)(x − 1).

20/69
Solution (continued)
Since cA (x) = (x − 3)(x − 2)(x − 1), the eigenvalues of A are
λ1 = 3, λ2 = 2, and λ3 = 1. Notice that each of these eigenvalues has
multiplicity one.
To find a basic eigenvector corresponding to λ1 = 3, solve (A − 3I )u = 0.
0 − 12 0
   
0 −4 2 0 1
 1 −5 2 0  → ··· →  0 1 − 12 0 
1 −5 2 0 0 0 0 0
 1   1 
2t 2
Thus u =  1 =t 1 , t ∈ R. Choosing t = 2 gives us
2t 2
t 1
 
1
u1 =  1 
2
as a basic eigenvector corresponding to λ1 = 3.

21/69
To find a basic eigenvector corresponding to λ2 = 2, solve (2I − A)u = 0.
   
1 −4 2 0 1 0 −2 0
 1 −4 2 0  → ··· →  0 1 −1 0 
1 −5 3 0 0 0 0 0
  
2s 2
Thus u =  s  = s  1 , s ∈ R. Choosing s = 1 gives us
s 1

2
u2 =  1 
1
as an eigenvector corresponding to λ2 = 2.
22/69
Finally, to find a basic eigenvector corresponding to λ3 = 1, solve
(I − A)u = 0.
   
2 −4 2 0 1 0 −1 0
 1 −3 2 0 → ··· → 0 1
  −1 0 
1 −5 4 0 0 0 0 0

  
r 1
Thus u =  r  = r  1 , r ∈ R. Choosing r = 1 gives us
r 1

1
u3 =  1 
1
is an eigenvector corresponding to λ3 = 1.
23/69
 
3 −4 2
Summarizing, for A =  1 −2 2 , we have found three eigenvalues,
1 −5 5
and a corresponding eigenvector for each as follows.
     
1 2 1
λ1 = 3 and X1 =  1  ; λ2 = 2 and X2 =  1  ; λ3 = 1 and X3 =  1  .
2 1 1
An easy way to check your work: compute Au1 and see if you get 3u1 .
      
3 −4 2 1 3 1
Au1 =  1 −2 2   1  =  3  = 3  1  = 3u1 .
1 −5 5 2 6 2
You should check that Au2 = 2u2 and that Au3 = 1u3 = u3 ,
24/69
Eigenvalues and eigenvectors (review)
Let A be an n × n matrix
1 Compute the charasteristic polynomial of A,
cA (x) = det(A − xI ).
2 Factorize cA (x) and find its roots.

3 For each root λ of cA (x) solve the homogeneous system
(A − λI )u = 0.
(It always has a nontrivial solution.)

4 λ-eigenvectors are the (nontrivial) solutions to this system.
25/69
Example (Triangular Matrices)
Consider the matrix  
2 −1 0 3
 0 5 1 −2 
A=
 0
.
0 0 7 
0 0 0 −4
The characteristic polynomial of A is
 
2−x −1 0 3
 0 5−x 1 −2 
 = −(2−x)(5−x)x(−x−4).
cA (x) = det(A−xI ) = det 
 0 0 −x 7 
0 0 0 −4 − x
Therefore the eigenvalues of A are 2, 5, 0 and −4, exactly the entries on

the main diagonal of A.
Eigenvalues of Triangular Matrices

If A is an n × n upper triangular (or lower triangular) matrix, then the
eigenvalues of A are the entries on the main diagonal of A.
26/69
Diagonalizing Matrix
Theorem
Suppose the n × n matrix A has n linearly independent eigenvectors
u1 , . . . , un . Put them into the columns of an eigenvector matrix U. Then
U −1 AU is the eigenvalue matrix Λ
 
λ1
U −1 AU = Λ = 
 .. 
. 
λn
where Λ is a diagonal matrix with the corresponding eigenvalues on the

diagonal.
Proof.
For all eigenvectors uk and the corresponding eigenvalues λk we have Auk = λk uk , if we put all
the eigenvectors as columns of matrix U then
AU = UΛ,
where Λ is a diagonal matrix with the corresponding eigenvalues on the diagonal.Since the
eigenvectors are assumed to be linearly independent, U −1 exists, and U −1 AU = Λ. 27/69
Diagonalizing Matrix
Remark
If A = UΛU −1 , then A2 = UΛ2 U −1 .
More generally,
Remark
Ak = UΛk U −1 for any k ∈ R.
Hence, for k = −1,

Remark
A−1 = UΛ−1 U −1 .
28/69
Similar Matrices
Definition
Let A, and B be n × n matrices. Suppose there exists an invertible
matrix P such that
A = P −1 BP
Then A and B are called similar matrices.
How do similar matrices help us in spectral theory?
Theorem
Let A and B be similar matrices, so that A = P −1 BP where A, B are
n × n matrices and P is invertible. Then A and B have the same
eigenvalues.
Proof
Assume Bu = λu. Let v = P −1 u. Then
Av = (P −1 BP)P −1 u = P −1 Bu = P −1 λu = λv.
29/69
Using Similar and Elementary Matrices
Problem
Find the eigenvalues for the matrix
 
33 105 105
A =  10 28 30 
−20 −60 −62
Solution
We will use elementary matrices to simplify A before finding the
eigenvalues. Left multiply A by E , and right multiply by the inverse of E .
     
1 0 0 33 105 105 1 0 0 33 −105 105
 0 1 0  10 28 30   0 1 0  =  10 −32 30 
0 2 1 −20 −60 −62 0 −2 1 0 0 −2
Notice that the resulting matrix and A are similar matrices (with E
playing the role of P) so they have the same eigenvalues.
30/69
We do this step again, on the resulting matrix above.
     
1 −3 0 33 −105 105 1 3 0 3 0 15
 0 1 0   10 −32 30   0 1 0  =  10 −2 30  = B
0 0 1 0 0 −2 0 0 1 0 0 −2
Again by properties of similar matrices, the resulting matrix here (labeled

B) has the same eigenvalues as our original matrix A. The advantage is
that it is much simpler to find the eigenvalues of B than A.
Finding these eigenvalues follows the usual procedure and is left as an
exercise.
31/69
Nondiagonalizable Matrices
For “exceptional” matrices eigenvalues λi can be repeated, then there are

two different ways to count its multiplicty
GM Geometric Multiplicity: Count the linearly independent eigenvectors
for λ, then Geometric Multiplicity is the dimension of the null space
of A − I .
AM Algebraic Multiplicity: Count the repetitions of λ among the
eigenvalues. Look at the n roots of det(A − λI ) = 0.
Remark
(i) We always have that GM ≤ AM.
(ii) Whenever GM < AM, then the corresponding matrix A is not
diagonalizable.
32/69
Nondiagonalizable Matrices
Example

0 1
A= has det(A − λI ) = λ2
0 0
so λ = 0 is an eigenvalue of multiplicity 2. But there is only one
eigenvector (1, 0) (check it!).
These three matrices all have the same shortage of eigenvectors, and
their repeated eigenvalue is λ = 5.

5 1 6 −1 7 2
A= , A= , A=
0 5 1 4 −2 3
The AM = 2 but the GM = 1 because each A − 5I has rank r = 1.
33/69
Fibonacci Numbers
Definition
The Fibonacci sequence 0, 1, 1, 2, 3, 5, 8, 13, . . . comes from
Fk+2 = Fk+1 + Fk
Question: Find F100 .

F
Answer: Take uk = k+1 , then the Fibonacci recursion can be written
Fk
as
1 1
uk+1 = Au, where A =
1 0
So u2 = Au1 = A2 u0 ; and, with X the matrix of eigenvectors, for any k,
  (λ )k  
c1
1
..   .. 
uk = Ak u0 = X Λk X −1 u0 = X Λk c = x1 · · · xn  

.  . 
(λn )k cn
This result is exactly uk = c1 (λ1 )k x1 + . . . + cn (λn )k xn where λi and xi
are the eigenvalues, and the corresponding eigenvectors of A. 34/69
Systems of Differential Equations
Example
Note, that the derivative of e λt is λe λt . The whole point of the next few
slides is to convert constant-coefficient differential equations into linear
algebra.
du(t) du(t)
The ordinary equations dt = u and dt = λu(t) are solved by
exponentials:
du(t)
= u(t) produces u(t) = Ce t and
dt
du(t)
= λu(t) produces u(t) = Ce λt
dt
where C = u(0) is the starting value.
We just solved a 1 × 1 problem. Linear algebra moves to n × n.
35/69
Systems of Differential Equations
Consider a system of n ordinary differential equations, where the

unknown is a vector u. It starts from an initial condition u(0) which is
given. The n equations contain a square matrix A. We expect n
exponents e λt in u(t), from n λ’s:
 
u1 (0)
du
System of n equations = Au starting from the vector u(0) =  ... 
 
dt
un (0)
These differential equations are linear, i.e., if u(t) and v(t) are solutions,
so is C u(t) + Dv(t). We will need n constants like C and D to match
the n components of u(0).
Therefore, our first job is to find n “pure exponential solutions”
u = e λt x by using Ax = λx.
36/69
Example
du
Solve dt = Au, where
0 1
A=
1 0

4
starting from u(0) = This is a vector equation for u. It contains two
2
sqalar equations for the components y and z. They are “coupled
together” because the matrix A is not diagonal.

d y 0 1 y dy (t) dz(t)
= means that = z(t) and = y (t)
dt z 1 0 z dt dt
The idea of eigenvectors and eigenvalues is to combine those equations in

a way that gets back to 1 by 1 problems. The combination y + z and
y − z will do it. Add and subtract the equations
d d
(y + z) = z + y and (y − z) = −(y − z)
dt dt
So, the combination y (t) + z(t) grows like e t , because it has λ = 1; and
the combination y (t) − z(t) decays like e −t , because it has λ = −1.
37/69
Example (continued)
This matrix A has eigenvalues 1 and −1; the corresponding eigenvectors
x are (1, 1) and (1, −1). The pure exponential solutions u1 and u2 take
the form e λt x with λ1 = 1 and λ2 = −1, i.e.,

1 1
u1 (t) = e t and u2 (t) = e −t
1 −1
Notice: these u’s satisfy Au1 = u1 and Au2 = −u2 , just like x1 and x2 .
The factors e t and e −t change with time. Those factors give
du1 du2
= u1 = Au1 and = −u2 = Au2
dt dt
We have two solutions to du/dt = Au. To find all other solutions,
multiply these special solutions by any numbers C and D and add
Ce + De −t
t
1 1
Complete Solution: u(t) = Ce t + De −t =
1 −1 Ce t − De −t
with these two constants we can match the starting vector. Here,
u(0) = (4, 2), so C = 3 and D = 1.
38/69
System of Differential Equations
In order to solve a linear system of ordinary differential equations
du(t)
= Au
dt
(1) Find c1 , . . . , cn such that you can write u(0) as a combination
c1 x1 + · · · + cn xn
of the eigenvectors of A.
(2) Multiply each eigenvector xi by its growth factor e λi t .
(3) The solution is the same combination of those pure solutions e λt x:
u(t) = c1 e λ1 t x1 + · · · + cn e λn t xn
Not included: If two λ’s are equal, with only one eigenvector, another
solution is needed. It will be te λt x. Step (1) needs to diagonalize
A = X ΛX −1 : a basis of n eigenvectors.
39/69
Example
   
1 1 1 9
du 
= 0 2 1 u starting from u(0) = 7
dt
0 0 3 4
The eigenvalues are λ = 1, 2, 3 and the corresponding eigenvectors are
x1 = (1, 0, 0), x2 = (1, 1, 0), and x3 = (1, 1, 1).
(1) The vector u(0) = (9, 7, 4) is 2x1 + 3x2 + 4x3 . Thus
(c1 , c2 , c3 ) = (2, 3, 4).
(2) The factors e λt give exponential solutions e t x1 , e 2t x2 and e 3t x3 .
[(3)] The combination that starts from u(0) is
u(t) = 2e t x1 + 3e 2t x2 + 4e 3t x3
The coefficients 2, 3, 4 came from solving the linear equation
c1 x1 + c2 x2 + c3 x3 = u(0)
      
c1 1 1 1 2 9
x1 x2 x3  c2  = 0 1 1 3 7 which is X c = u(0)
c3 0 0 1 4 4
40/69
Second Order Equations
Consider a linear second order differential equation with constant
coefficients
y 00 + by 0 + ky = 0
We can turn it into a vector first order differential equation of y and y 0 .

y d 0 1
Define u = 0 , then u = Au where A = .
y dt −k −b
A is called a companion matrix to the 2nd order equation with y 00 . The
first equation is trivial (but true). The second equation connects y 00 , y 0
d
and y . So we can solve dt u = Au by eigenvalues of A:

−λ 1
A − λI = and cA (λ) = det(A − λI ) = λ2 + bλ + k = 0
−k −b − λ
The eigenvectors and the solution are

1 1 1 1
x1 = x2 = and u(t) = c1 e λ1 t + c2 e λ2 t
λ1 λ2 λ1 λ2
The first component of u(t) is y , so y = c1 e λ1 t + c2 e λ2 t

The second component of u(t) is y 0 , so y 0 = c1 λ1 e λ1 t + c2 λ2 e λ2 t 41/69
Motion Around a Circle: y 00 + y = 0

00 d y 0 1
y = −y can be written as u = Au, where u = 0 and A =
dt y −1 0
The eigenvalues of A are λ1 = i and λ2 = −i. A is antisymmetric with

eigenvectors
1 1
x1 = and x2 =
i −i
The combination that matches u(0) = (1, 0) is 12 (x1 + x2 ) So

1 1 1 1 cos(t)
u(t) = e it + e −it =
2 i 2 −i − sin(t)
where we used Euler’s formula e it = cos(t) + i sin(t), and

sin(−t) = − sin(t) and cos(−t) = cos(t).
So, indeed the vector u goes around a circle with the radius 1 because
2
ku(t)k = cos2 (t) + sin2 (t) = 1 for all t. 42/69
To display a circle on a screen, replace y 00 = −y by a difference equation.
Here are three choices using
Y (t + ∆t) − 2Y (t) + Y (t − ∆t)
Divide by (∆t)2 to approximate y 00 by one of the formulas
(F) Forward from n − 1: Yn+1 −2Y n +Yn−1
(∆t)2 = −Yn−1
Yn+1 −2Yn +Yn−1
(C) Centered at time n: (∆t)2 = −Yn
Yn+1 −2Yn +Yn−1
(B) Backward from n + 1: = −Yn+1
(∆t)2
The two step equations above reduce to 1-step systems
Un+1 = AUn
with Un = (Yn , Zn ) with Z being like y 0 above. The corresponding
matrices A are
−1 −1
1 ∆t 1 0 1 ∆t 1 −∆t
AF = AC = and AB =
−∆t 1 ∆t 1 0 1 ∆t 1
with two eigenvalues for each case given by
λF = 1 ± i∆t |λC | = 1 and λB = 1/(1 ± i∆t)
43/69
44/69
Stability of Matrices
For the solution of du/dt = Au, there is a fundamental question.

Does the solution approach u = 0 when t → ∞?
Since the complete solution is built from pure solutions e λt x
If the eigenvalue λ is real, we know exactly when e λt will approach
zero. The λ must be negative.
If the eigenvalue λ is complex number λ = r + is, the real part r
must be negative. Because e λt splits into e rt e ist , the factor e ist has
absolute value fixed at 1:
2
e ist = cos(st) + i sin(st) has e ist = cos2 (st) + sin2 (st) = 1
So the real part of λ controls the growth (r > 0) or the decay (r < 0) of
the system.
45/69
Theorem
A is stable and u(t) →0 when
all eigenvalues λ have negative real parts.
a b
The 2 × 2 matrix A = must pass the test:
c d
The trace T = a + d must be negative
The determinant D = ad − bc must be positive.
Proof.
The trace is always equal to the sum of eigenvalues, and the determinant is always equal to
the product of eigenvalues. Hence,
T < 0 ⇐⇒ λ1 + λ2 < 0 and D > 0 ⇐⇒ λ1 λ2 > 0
For the real eigenvalues, the later is equivalent with eigenvalues having the same sign,
then the former implies that the eigenvalues are negative.
In case of complex eigenvalues, they must have the form r + is and r − is. Otherwise
T and D would not be real. The determinant is automatically positive since
D = (r + is)(r − is) = r 2 + s 2 , and the trace is T = (r + is) + (r − is) = 2r . So the
negative trace means r < 0 and the matrix is stable.
46/69
Note that for a 2 × 2 matrices the characteristic polynomial can be

written as
cA (λ) = λ2 − T λ + D
√
Finding roots involves T 2 − 4D. Hence,
47/69
Direction Field with Trajectories
48/69
One way to define e x for numbers is by the infinite series
1 2 1 1
ex = 1 + x + x + x3 + x4 + . . .
2! 3! 4!
When we change x to a square matrix At, this series define the matrix
exponential e At
Definition
Matrix Exponential e At :
At 1 2 1 3 1 4
e = I + At + (At) + (At) + (At) + · · ·
2 3! 4!
Its t Derivative Ae At :
d At 1 2 1 3 2 1 4 3 At
e = A + A t + A t + A t + · · · = Ae
dt 2 3! 4!
Its Eigenvalues e λt , where (λ, x) are the eigenvalues and eigenvectors of A:
1 2 1 3 1 4 1 2 1 3 1 4
(I +At+ (At) + (At) + (At) +· · · )x = (1+λt+ (λt) + (λt) + (λt) +· · · )x
2 3! 4! 2 3! 4!
49/69
We have
d
u = Au, where u(0) is a given initial condition.
dt
We want to write the solution u(t) in a new form e At u(0).
We found u(t) = e At u(0) by diagonalizing matrix A. Assume A has n
independent eigenvectors, so it is diagonalizable. So
1 1 1
e At = I + At + (At)2 + (At)3 + (At)4 + · · ·
2 3! 4!
1 1 1
= I + (X ΛX −1 )t + (X ΛX −1 t)2 + (X ΛX −1 t)3 + (X ΛX −1 t)4 + · · ·
2 3! 4!
1 1 1
= I + (X ΛX −1 )t + X Λ2 X −1 t 2 + X Λ3 X −1 t 3 + X Λ4 X −1 t 4 + · · ·
2 3! 4!
1 1 1
= X (I + Λt + Λ2 t 2 + Λ3 t 3 + Λ4 t 4 + · · · )X −1
2 3! 4!
= Xe Λt X −1
50/69
So if A is diagonalizable (i.e., A = X ΛX −1 ), then e At is diagonalizable
too, and
e At = Xe Λt X −1
Do the multiplication to recognize that
u(t) = e At u(0) corresponds to u(t) = Xe Λt X −1 u(0)
I.e.  e λ1 t  
 c1
..   .. 
u(t) = x1 ··· xn  

.  . 
e λn t cn
where    −1
c1
 ..  
 .  = x1 ··· xn  u(0).
cn
This solution is exactly the same answer that came earlier from three
steps.
u(t) = c1 e λ1 t x1 + · · · + cn e λn t xn
51/69
Repeated Roots Case with GM < AM
Example
Consider
00 0
y − 2y + y = 0
Linear algebra reduces this second order differential equation to a vector equation for
u = (y , y 0 )
y0

d y du 0 1
0 = 0 is = Au = u
dt y 2y − y dt −1 2
A has repeated eigenvalue λ = 1 (with trace T = tr (A) = 2 and determinant

D = det(A) = 1).
The only eigenvector are multiples of x = (1, 1).
Diagonalization is NOT possible, GM < AM.
A has only one line of eigenvectors.
So we compute e At from its definition as a series:
At It (A−I )t t
e =e e = e [I + (A − I )t]
where we used the fact that for our A: (A − I )2 = 0! So

y t −1 1 y (0) t t t 0
0 = e I+ t 0 . Hence, y (t) = e y (0) − te y (0) + te y (0).
y −1 1 y (0)
52/69
Repeated Roots Case for the Rotation Matrix
Example
Consider again the motion around the circle:
00
y +y =0
Linear algebra reduces this second order differential equation to a vector equation for
u = (y , y 0 ) 0
d y y du 0 1
0 = is = Au = u
dt y −y dt −1 0
Notice that

0 1 2 −1 0 3 0 −1 4 1 0 5 4
A= A = A = A = and A =A A=A
−1 0 0 −1 1 0 0 1
So A5 , A6 , A7 , A8 will be a repeat of A, A2 , A3 , A4 ; and
1 − 12 t 2 + · · · 1 3

At 1 2 2 1 3 3 1 4 4 t − 3! t
e = I + At + A t + A t + A t + ··· =
2 3! 4! −t + 16 t 3 − · · · 1 − 12 t 2 + · · ·

0 1 At cos(t) sin(t)
So A= and e =
−1 0 − sin(t) cos(t)
A is an antisymmetric matrix (AT = −A). Its exponential e At is an orthogonal matrix. The

eigenvalues of A are i and −i. The eigenvalues of e At are e it and e −it .
53/69
Three Rules for e At & Energy of AT = −A Matrices
Remark (Three Rules)

e At always has the inverse e −At
The eigenvalues of e At are always e λt
When A is antisymmetric (AT = −A), e At is orthogonal. Hence,
Inverse = Transpose = e −At ,
i.e.,
(e At )−1 = (e At )T = e −At
Remark
Antisymmetric is the same as “skew-symmetric”. Thoes matrices have
pure imaginary eigenvalues like i and −i. Then e At has eigenvalues like
e it and e −it . Their absolute value is 1: neutral stability, pure oscilation,
energy conserved. So ku(t)k = ku(0)k, i.e., the dynamics stay on the
circle.
54/69
Direction Field with Trajectories
55/69
Symmetric Matrices
What is special about Sx = λx when S is symmetric S = S T ?
Assume that S is diagonalizable, i.e., all the eigenvectors are linearly

independent. Then
S = X ΛX −1 = S T = (X ΛX −1 )T = (X −1 )T ΛX T
Hence, we suspect that
S = ST and S = X ΛX −1
⇓
X −1 = X T and S = X ΛX T
i.e., we can pick eigenvectors such that they form a system of

orthonormal vectors: X T X = I .
56/69
Eigenvalues of Real Symmetric Matrices
Theorem
All the eigenvalues of a real symmetric matrix are real
Proof.
Suppose Sx = λx. Until we know otherwise λ can be a complex number a + ib (a and b
are real). Its complex conjugate is λ̄ = a − ib.
Similarly the components of x can be complex numbers, and switching the signs of their
imaginary parts gives x̄
The good thing is that product of conjugate numbers is the conjugate of the product,
¯
i.e., λ̄x̄ = λx
So we can take conjugates of
T T
Sx = λx leads to Sx̄ = λ̄x̄. Transpose to x̄ S = x̄ λ̄
Now take the dot product of the first equation with x̄ and the last equation with x:
T T T T
x̄ Sx = x̄ λx and also x̄ Sx = x̄ Sx.
The left sides are the same so the right sides are equal. One equation has λ, the other
has λ̄. They multiply x̄T x = kxk2 which is not zero because x is the eigenvector.
Therefore λ must be equal λ̄, and a + ib = a − ib. So b = 0, and λ = a is real.
57/69
Eigenvectors of Symmetric Matrices
Theorem
Eigenvectors of a symmetric matrix (when they correspond to different
λ’s) are always perpendicular.
Proof.
Suppose
Sx = λ1 x and Sy = λ2 y.
We are assuming here that λ1 6= λ2 . Take dot product of the first equation with y and the
second with x
T T T T T T T
Use S =S (λ1 x) y = (Sx) y = x S y = x Sy = x λ2 y
The left side is xT λ1 y, the right side is xT λ2 y. Since λ1 6= λ2 , this proves that xT y = 0. The
eigenvector x for λ1 , is perpendicular to the eigenvector y for λ2 .
58/69
Eigenvectors of Symmetric Matrices
In general, we know that a repeated eigenvalue can produce a shortage of
eigenvectors (GM < AM case). It NEVER happens for symmetric
matrices (real or not).
Remark (S = S T ⇒ GM = AM)
If S = S T , then there is enough eigenvectors to diagonalize S.
Proof.
Uses Schur’s Theorem which is another decomposition of any square matrix given below.
Schur’s S = QTQ −1 means that T = Q T SQ. The transpose is again Q T SQ. The triangular
matrix T is symmetric when S = S T .
Then T must be diagonal, so T = Λ.
This proves that S = Q −1 . The symmetric S has n orthonormal eigenvectors in Q.
Theorem (Schur’s Theorem)
Every square A factors into QTQ −1 , where T is upper triangular and Q̄ T = Q −1 . If A has real
eigenvalues, then Q and T can be chosen real: Q T Q = I .
Proof: Schur’s Theorem.

See math.mit.edu/linearalgebra
59/69
Spectral Theorem
The following theorem summarizes the results for the symmetric matrices
Theorem (Spectral Theorem)

Every symmetric matrix has the factorization
S = QΛQ T
with real eigenvalues Λ and orthonormal eigenvectors in the columns of

Q:
Symmetric diagonalization S = QΛQ −1 = QΛQ T with Q T = Q −1
Remark
Q T Q = QQ T = I
60/69
Geometry of Spectral Theorem
Geometry:
 λ qT
  
1 1
..   ..
S = QΛQ T = q1 ··· qn  
 
.  . 
λn qT
n
Since Q T Q = I , Q is an orthonormal transformation (preserves length

and engle between vectors, it only rotatest them). Hence,
S = (ROTATE)(STRETCH)(ROTATE BACK)
So every symmetric matrix is a composition of three mappings:
(i) rotation;
(ii) stretch (along the original axis); and
(iii) rotation back.
61/69
Spectral Decomposition
Spectral Theorem implies the following Spectral Decomposition

n
X
S = QΛQ T = λ1 q1 qT T
1 + · · · + λn qn qn = λj qj qT
j
j=1
where qj qT
j are n × n matrices with rank 1, and
Sqi = QΛQ T qi = λ1 q1 qT T
1 qi + · · · + λn qn qn qi = λi qi
because q1 , . . . , qn are orthonormal eigenvectors.
62/69
Complex Eigenvalues of Real Matrices
Note that for any real matrix S
Sx = λx ⇒ Sx̄ =¯x̄
For symmetric matrices λ and x turn out to be real, and those two
equations become the same.
But a NONsymmetric matrix can easily produce λ and x that are
complex. Then Ax̄ = λ̄x̄ is true but different from Ax = λx.
We get another complex eigenvalue, and a new eigenvector.
Example

cos θ − sin θ
A= has λ1 = cos θ + i sin θ and λ2 = cos θ − i sin θ
sin θ cos θ
Those eigenvalues are conjugate to each other. The eigenvectors must be

conjugate too because A is real. In fact it is easy to show that

1 1
x1 = and x2 = x̄1 = .
−i i
63/69
Eigenvalues vs. Pivots
The only connection between eigenvalues and pivots is
Remark
product of pivots = determinant = product of eigenvalues
Remark
If S = S T , then the number of positive eigenvalues = number of positive
pivots.
64/69
Positive Definite Matrices
Definition (Positive Definite Matrix)

A symmetric matrix that has all the eigenvalues positive is called positive
definite.
Goals:
find quick tests that guarantee that a given matrix is positive
definite.
explain important applications of positive definiteness.
65/69
Positive Definite Matrices
Theorem (5 Characteristics of Positive Definite Matrix)

When a symmetric matrix S has one of the following properties, then it
has all of them:
1. All n pivots of S are positive.
2. All n upper left determinants are positive (Sylvester’s criterion).
3. All n eigenvalues of S are positive.
4. xT Sx is positive except at x = 0.
5. S equals AT A for some matrix A with independent columns.
Remark
If S and T are symmetric positive definite matrices, then S + T is also a
symmetric positive definite matrix.
66/69
Positive Semidefinite Matrices
Definition (Positive Semidefinite Matrix)

A symmetric matrix that has all the eigenvalues non-negative is called
positive semidefinite.
Theorem
Positive semidefinite matrices have:
1. All n pivots of S are non-negative.
2. All n! principal minors of S are non-negative.a
3. All n eigenvalues of S are bigger or equal to zero (λi ≥ 0)
4. xT Sx ≥ 0 for all vectors x.
5. S equals AT A for some matrix A (possibly with dependent columns).
a A principal submatrix of a square matrix S is the matrix obtained by deleting any k rows
and the corresponding k columns. The determinant of a principal submatrix is called the
principal minor of S.
67/69
Important Application: Test for a (Local) Minimum
Consider a function of two variables F (x, y ).
Suppose ∂F /∂x = 0 and ∂F /∂y = 0 at the point (x, y ) = (0, 0).

Does F (x, y ) have a (local) minimum or (local) maximum at
(x, y ) = (0, 0)?
Compute the matrix of second derivatives at (x, y ) = (0, 0):
2
∂ F /∂x 2 ∂ 2 F /∂x∂y

S= 2
∂ F /∂y ∂x ∂ 2 F /∂y 2
this is a symmetric matrix, and F has:

a (local) minimum at (x, y ) = (0, 0), if S is positive definite at
(x, y ) = (0, 0); or
a (local) maximum at (x, y ) = (0, 0), if −S is positive definite at
(x, y ) = (0, 0).
For F (x, y , z) the matrix S will be 3 × 3 but the test is the same.
68/69
Important Application: Test for a (Local) Minimum
(i) Saddle point F (x, y ) = x 2 − y 2 ; (ii) Saddle point between two maxima; (iii) Monkey Saddle
Surface F (x, y ) = x 3 − 3xy 2
69/69

Linear Algebra Guide to Eigenvalues & Eigenvectors

Загружено:

Сведения о документе

Оригинальное название

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

Linear Algebra Guide to Eigenvalues & Eigenvectors

Загружено:

Авторское право:

Доступные форматы

Eigenvalues & Eigenvectors

October 17, 2019

Linear Algebra - Lec. 7

Special Types of Matrices

Solving Recursive Equations

Systems of First Order Linear Differential Equations

Second Order Linear Differential Equations

The Exponential of a Matrix

Symmetric Matrices and Spectral Theorem

Positive Definite Matrices 2/69

How can we do this efficiently?

where D is a diagonal matrix.

A100 = (PDP −1 )100

1 2100 + 2 · 5100 2100 − 2 · 5100

1 2100 + 2 · 5100 2100 − 2 · 5100

The process of finding an invertible matrix P and a diagonal matrix D so

Eigenvectors are vectors which do not change the direction when

Since u 6= 0, u is a nontrivial solution to the linear system with

Eigenvectors: For each λ, find u 6= 0 by finding the basic solutions

so A has eigenvalues λ1 = 2 and λ2 = 5.

The general solution is

However, since eigenvectors are nonzero, the 2-eigenvectors of A are all

we obtain basic eigenvectors

We can obtain other pairs of basic 4-eigenvectors for A by taking any

Notice that every 4-eigenvector of A is a nonzero linear combination of

Therefore, cA (x) = (x − 3)(x 2 − 3x + 2) = (x − 3)(x − 2)(x − 1).

as a basic eigenvector corresponding to λ1 = 3.

2 Factorize cA (x) and find its roots.

(It always has a nontrivial solution.)

Therefore the eigenvalues of A are 2, 5, 0 and −4, exactly the entries on

Eigenvalues of Triangular Matrices

where Λ is a diagonal matrix with the corresponding eigenvalues on the

Hence, for k = −1,

How do similar matrices help us in spectral theory?

Again by properties of similar matrices, the resulting matrix here (labeled

For “exceptional” matrices eigenvalues λi can be repeated, then there are

The AM = 2 but the GM = 1 because each A − 5I has rank r = 1.

Question: Find F100 .

We just solved a 1 × 1 problem. Linear algebra moves to n × n.

Consider a system of n ordinary differential equations, where the

Therefore, our first job is to find n “pure exponential solutions”

The idea of eigenvectors and eigenvalues is to combine those equations in

In order to solve a linear system of ordinary differential equations

(1) Find c1 , . . . , cn such that you can write u(0) as a combination

The first component of u(t) is y , so y = c1 e λ1 t + c2 e λ2 t

The eigenvalues of A are λ1 = i and λ2 = −i. A is antisymmetric with

where we used Euler’s formula e it = cos(t) + i sin(t), and

For the solution of du/dt = Au, there is a fundamental question.

T < 0 ⇐⇒ λ1 + λ2 < 0 and D > 0 ⇐⇒ λ1 λ2 > 0

Note that for a 2 × 2 matrices the characteristic polynomial can be

Its Eigenvalues e λt , where (λ, x) are the eigenvalues and eigenvectors of A:

A has repeated eigenvalue λ = 1 (with trace T = tr (A) = 2 and determinant

where we used the fact that for our A: (A − I )2 = 0! So

So A5 , A6 , A7 , A8 will be a repeat of A, A2 , A3 , A4 ; and

A is an antisymmetric matrix (AT = −A). Its exponential e At is an orthogonal matrix. The

Remark (Three Rules)

Inverse = Transpose = e −At ,

What is special about Sx = λx when S is symmetric S = S T ?

Assume that S is diagonalizable, i.e., all the eigenvectors are linearly

Hence, we suspect that

i.e., we can pick eigenvectors such that they form a system of

Theorem (Schur’s Theorem)

Proof: Schur’s Theorem.