E1

251 Linear and Nonlinear

Op2miza2on

Chapter 1: Background linear algebra

1.1 Vector space

A vector space (over ! ) is a set V with two operations + and satisfying
the following properties for all u,vV and c,d!:
(i ) (Additive Closure) u+vV
(Adding two vectors gives a vector )
(ii ) (Additive Commutativity) u+v=v+u
(Order of addition doesn't matter)
(iii ) (Additive Associativity) (u+v)+w = u+(v+w)
(Order of adding many vectors doesn't matter)
(iv) (Zero) There is a special vector 0V V such that u+0V =u for all u inV
(v) (Multiplicative Closure) cvV
(Scalar times a vector is a vector)
(vi ) (Additive Inverse) For every uV there exists wV such that u+w=0V .
(vii ) (Distributivity) (c+d )v=cv+dv
(Scalar multiplication distributes over addition of scalars)
(viii ) (Associativity) (cd )v=c(dv)
(ix )
(Unity) 1v=v for all vV

Subspace of a vector space V: it is a subset of V that satifies

the above axioms

1.2 The vector space ! n

n is a vector space containing vectors of the form
a1

T
a2

a=
= a1 a2 an
, where a

a
n
Addition of two vectors: addition of corresponding indvidual elements
Mutiplication by scalar: multiply the indvidual elements

Subspace in n : a subset of n satifying the axioms of vector space

Linear variety in n : A translated subspace, i.e., a subset in n that
can be expressed as s 0 + V where V is subspace in n , s 0 is vector in n
is called a linear variety.
Subspace examples: planes and lines passing through the origin
These are not subspaces: polygons and line segments even if they pass
through origin.

1.3 Linear independence of a set of vectors

A set of vectors {a1 ,a 2 ,,a n } is called a linearly independent set if and only if
no vector in the set can be expressed as a linear combination of other vectors,i.e,
there exist no j such that
a j = i j i a i

A set of vectors {a1 ,a 2 ,,a n } is called a linearly independent set if and only if
there does not exist a set of scalars {1 ,, n } with at least one non-zero element
n

such that

a
i

= 0.

i=1

1.4 Span of a LI set and basis of subspace

A subspace V is called the span of a LI set {a1 ,a 2 ,,a L } if every vector v V
L

can be expressed as v = i,v a i .

i=1

A set {a1 ,a 2 ,,a L } is called a basis of a subspace V if the vector in the set are
L

linearly independent and every vector v V can be expressed as v = i,v a i .

i=1

L is called the dimension of V.

Any LI set of n 1 vectors, {a1 ,a 2 ,,a n } will span ! n .

1.5 Non-uniqueness of basis of a subspace and uniqueness of

representation for given subspace
Fact 1.5A) For any given subspace, the basis is not unique
Fact 1.5B) For given basis {a1 ,a 2 ,,a L } of a subspace V and vector
v V, the representation of v in terms of the basis is unique.

1.6 Rank of a matrix

The rank of a matrix is the number of linearly independent columns.

Fact 1.6A:
Adding to a given column a linear combination of other columns
does not change the rank

Reading assignment:
Section 2.3 (Linear equations) of Chong and Zak.
7

1.7 Inner products and norms

For x, y ! n , the Euclean inner product is defined by
n

x, y = xi yi = x y.
i=1

It is a real valued function having the following

properties:
1) Positivity: x,x 0, x,x = 0 only if x = 0.
2) Symmetry: x, y = y,x
3) Additivity: x + y,z = x,z + y,z
4) Homogeinity: rx, y = r x, y
The Eucledean norm of vector is defined as
x=

x, y .

Norm inequalities

Fact 1.7A) Cauchy Schwartz inequality

x, y xy

Fact 1.7B) Triangle inequality

x + yx+y

1.8 Gram-Schmidt orthogonalization

L
, Gram-Schmidt orthogonalization gives
{v i }i=1
L
a set of orthogonal vectors {ui }i=1 such that span {v i } = span {ui } .

Given a set of LI vectors

Recursive orthogonalization:
u1 =

v1
.
v1
j1

uj = v j ui ,v j ui .
i=1

uj =

uj
uj

10

An application:
Given a set of LI vectors

{v i }i=1 , find a vector p such that

L

p T v i = 0 for i = 1,..., L 1, and pT v L 0.

1) Do GS orthogonalization of {v i }i=1 to get {ui }i=1
L1

L1

L1

2) Compute p = v L ui ,v L ui .
i=1

11

1.9 Linear transformations

Coordinates of a vector :
(1) For a given vector a, its ith component,
{a}i = ai is the ith coordinate with respect to its ith standard basis vector,
n

i.e., a = ai ei , where ei is the vector of all zeros except element in ith

i =1

position which is 1.
(2) a vector x is called a representation of the vector a with respect to basis
n

{b i }in=1 if a = xi b i , where xi = {x}i . xi is the coordinate of a with

i =1

respect to the basis vector b i .

3) Suppose x is a representation of a vector in {b i }in=1 and x is a
representation of the same vector in {b i }in=1 . Then
x =Tx, where T = [b1 b n ]1[b1 b n ]
12

Linear transformation:
A function L : ! n ! m is called a linear transformation if
1. L(ax) = aL(x)
2. L(x + y) = L(x) + L(y)
Matrix representation of linear transformation:
n
1) Suppose x is the representation of a vector v in ! n w.r.t to basis {a i }i=1
and
m
and let L : ! n ! m be a linear transformation. Let {b i }i=1
be a basis for

! m and

m
y be the representation of Lx w.r.t to basis {b i }i=1
. Then the linear

transformation L can be represented by y = Ax where A is m n matrix.

2a) Suppose we have an LT L : ! n ! n . Let x the representation of vector
n
v ! n w.r.t {a i }i=1
and let y the representation of vector Lv ! n w.r.t the
same basis. Then the LT can be represented by y = Ax, where A is n n matrix.

n
2b) Suppose B be the representation of the same LT w.r.t another basis {a i }i=1
.

Then TA = BT or A = T 1BT, where T = [a 1 a n ]1[a1 a n ]

13

1.10 Range space and null space of a matrix

Range space:
Given an m n matrix A, its range space R(A) is defined as
R(A) := { y : y = Ax, x ! n } .

Dim R(A) is the rank of the matrix (Number of LI columns)

Null space (the space of null vectors):
Given an m n matrix A, its null space N(A) is defined as
N(A) := {x : Ax = 0}
Dim N(A) is the nullity of the matrix.
A matrix with non-zero nullity is called a singular matrix.
R(A) is subspace in ! m and N(A) is a subspace in ! n .
14

Direct sum of subspaces :

let V and U be subspaces in ! n such that V U = {0} . Then a
subspace W is called a direct sum of V and U if for every w W ,
there exists vector v Vand u U such that w = v + u.
Notation: W = V U.
Orthogonal complement:
1) For a subspace V its orthogonal complement, V is defined as the
set of vectors u ! n satisfying uT v = 0 for all v V.
2) For any matrix A, N(A) = R(A T ).

15

Fact 1.10A:
For any subspace, V, we have V V = ! n .
Implication: Dim R(A T ) +Dim N(A) = n
Fact 1.10B:
Dim R(A) +Dim N(A) = n
Fact 1.10C:
Row rank of a matrix is equal to its column rank.

16

1.11 Determinant of a matrix

Computation:
n

det(A) = (1)i + j aij det M i, j (A) ,

i =1

M i, j (A) : matrix obtained after removing ith row and jth column.
Value of det is independent of j.

Properties:
Define det(A) = det[a1 ,a 2 ,....,a n ]
1) det[a1 ,a 2 ,....,ca j + db,...,a n ] = c det(A) + ddet[a1 ,a 2 ,....,b,...,a n ]
2)det[a1 ,a 2 ,....,a j ,a j+1 ,...,a n ] = det[a1 ,a 2 ,....,a j+1 ,a j ,...,a n ]
3) det(I) = 1
17

Other Properties:
1) Transpose operation does not change the value
of the determinant
2) Determinant of a triangular matrix is equal to the
product of its diagonal element
3) Determinant does not change with elementary row
or column operation
4) Determinant of product of matrices is equal to the
product of determinants
5) Determinant of orthogonal matrices is equal to
unity.
6) Determinant of the inverse of a matrix is equal to
the reciprocal of its determinant.
18

Fact 1.11A: Determinant of a rank deficient square matrix:

Axioms (1) and (2) imply that determinant of a rank deficient
square matrix, i.e., a matrix with linearly dependent columns
is zero.
Proof:
i) Express a column as a linear combination of other columns
ii) Use property (1) to express the determinant as weighted sum
of determinants of matrices having two identical columns
(iii) Use property (2) to deduce that determinant of a matrix having
two identical columns is zero.

19

1.12 Eigen vectors and Eigen values

Eigen vector and Eigen value:
A vector v is called an eigen vector of the matrix A, if there exists a scalar ,
called the Eigen value, such that Av = v.

Computation:
From Av = v, we get [ A I ] v = 0. Hence the eigen values are the values

of that makes [ A I ] singular. Hence the eigen values are the roots of the

equation det [ A I ] = 0. For a given eigen value i , the corresponding eigen

vectors are the null vectors of [ A i I ].
Similarity transformation:
Similarity transformation preserves eigen values
Trace identity:
For a matrix, sum of Eigen values is equal to the trace

20

Multiplicities Eigen values and Eigen vectors

1) If i a root of det [ A I ] = 0 with multiplicity mi , then the number of
i
corresponding linearly independent eigen vector {e(i j ) }nj=1
, i.e., the number of

linearly independent null vectors of [ A i I ] satisfies ni mi .

2) mi is called the algebraic multiplicity and ni is called the geometric

multiplicity.
3) For an n n matrix A, let N e be the number of eigen values. Then the
Ne

multiplicities satisfy

i=1

= n and

Ne

n.

i=1

21

1.13 Linear independence of eigen vectors

Fact 1.13A :
The eigen vectors of a matrix are linearly independent
Proof by mathematical induction:
Step 1:
Pick a Eigen vector. It is LI because it is non-zero vector.
Step 2:
Assume that any set of k Eigen vectors are LI.
Step 3:
Prove that adding an another Eigen vector does not affect
the linear independance.

22

1.14 Diagonalizability
Diagonalizability :
A matrix A is called diagonalizable if there exists matrix P such that
P1AP is a diagonal matrix.
Condition on eigen vectors:
An n n matrix is diagonalizable if it has n linearly independent eigen
vectors.

23

1.15 Eigen values and eigen vectors of a symmetric matrix

Fact 1.15A :
Eigen values of a real symmetric matrix are real.
Fact 1.15B:
An n n real symmtric matrix has n mutually orthogonal
eigen vectors.

24

1.16 Projections
Projection:
A matrix P is called a projector onto V if for all x n we have
Px V and P2 = P. Obviously, V = R(P).
Orthogonal projection:
A matrix P is called a orthogonal projector onto V if for all
x n we have Px V, P2 = P and x Px V .

Fact 1.16A:
A matrix P is called a orthogonal projector if and only if
P2 = P and PT = P.

Properties of projection matrices:

1) Eigen values are either 1 or 0
2) Projection matrices are diagonalizable
Note : Gram-Schmitt orthogonalization is based on orthogonal projection

25