ENGMAE 200A: Engineering Analysis I

Definitions, Linear Systems, Matrices, and Vector Spaces

Instructor: Dr. Ramin Bostanabad

DEFINITIONS AND NOTATION

• Matrices

• Whiteboard vs. PowerPoint notation

• Brackets notation

• Equality of matrices

• Main diagonal and vectors

• Identity, square, and sparse matrices

• Commutative and associative properties

2
MULTIPLICATION OF MATRICES
The product C = AB (in this order) of an m × n matrix A = [ajk] times an r
× p matrix B = [bjk] is defined if and only if (iff) r = n. Then, C = [cjk] has
entries:
𝑛
𝑗 = 1, ⋯ , 𝑚
𝑐𝑗𝑘 = ෍ 𝑎𝑗𝑙 𝑏𝑙𝑘 = 𝑎𝑗1 𝑏1𝑘 + 𝑎𝑗2 𝑏2𝑘 + ⋯ + 𝑎𝑗𝑛 𝑏𝑛𝑘
𝑘 = 1, ⋯ , 𝑝
𝑙=1

3
MORE DEFINITIONS
• The transpose of an m × n matrix
A = [ajk] is the n × m matrix AT
𝑎11 𝑎12 ⋯ 𝑎𝑚1
𝑎21 𝑎22 ⋯ 𝑎𝑚2
𝐀T = 𝑎𝑘𝑗 = ⋅ ⋅ ⋯ ⋅
𝑎1𝑛 𝑎2𝑛 ⋯ 𝑎𝑚𝑛

• Symmetric and skew-symmetric matrices:

Symmetric: AT = A
Skew-symmetric: AT = −A

• Scalar, triangular, and upper (lower) triangular matrices

4
LINEAR SYSTEMS OF EQUATIONS
• Linear system of m equations in n unknowns:

𝑎11 𝑥1 + ⋯ + 𝑎1𝑛 𝑥𝑛 = 𝑏1
𝑎21 𝑥1 + ⋯ + 𝑎2𝑛 𝑥𝑛 = 𝑏2
Eq. 1
⋯⋯⋯⋯⋯⋯⋯⋯
𝑎𝑚1 𝑥1 + ⋯ + 𝑎𝑚𝑛 𝑥𝑛 = 𝑏𝑚

• What are 𝑥𝑖 , 𝑎𝑖𝑗 , and 𝑏𝑖 ?

• Why called linear?
• When is Eq. 1 homogeneous/nonhomogeneous?
• What is the matrix form?
• What is the geometric interpretation?

5
SOLVING LINEAR SYSTEMS
There are a few ways. Here, we introduce Gauss elimination.
• Simple example:

𝑥1 − 𝑥2 + 𝑥3 = 0
−𝑥1 + 𝑥2 − 𝑥3 = 0
10𝑥2 + 25𝑥3 = 90
10𝑥2 + 20𝑥1 = 80

Some jargons to know:

• Triangular matrices, back substitution, partial/total pivoting
• Elementary row operations, row-equivalent systems

6
MORE EXAMPLES

3𝑥1 + 2𝑥2 + 2𝑥3 − 5𝑥4 − 8 = 0

0.6𝑥1 + 1.5𝑥2 + 1.5𝑥3 − 5.4𝑥4 = 2.7
1.2𝑥1 − 0.3𝑥2 − 0.3𝑥3 + 2.4𝑥4 = 2.1

3𝑥1 + 2𝑥2 + 𝑥3 = 3
2𝑥1 + 𝑥2 + 𝑥3 = 0
6𝑥1 + 2𝑥2 + 4𝑥3 = 6

7
ROW ECHELON FORM
• Original system of m equations in n unknowns: [A | b]
• Upon Gauss elimination: [R | f]
• Systems Ax = b and Rx = f are equivalent: if either one has a solution,
so does the other, and the solutions are identical.

r11 r12 r1n f1  • Consistent vs. inconsistent

 f2 
 r22 r2 n • Determined, overdetermined,
  underdetermined
 
 rrr rrn fr  • Number of solutions:
 fr 1  • None
  • One
 
 fm  • Infinite
8
LINEAR SYSTEMS CHARACTERIZATION
• To fully characterize the behavior of linear systems we need to first
define linear independence and matrix rank.

• Consider m vectors 𝒂1 , … , 𝒂𝑚 with equal size:

• Linear combination of these vectors:
𝑐1 𝒂1 + … + 𝑐𝑚 𝒂𝑚
where 𝑐1, 𝑐2, … , 𝑐𝑚 are any scalars.
• Consider 𝑐1 𝒂1 + … , +𝑐𝑚 𝒂𝑚 = 𝟎
• Is there a solution with at least one 𝑐𝑗 1 ≤ 𝑗 ≤ 𝑚?
Example:
𝒂1 = 3, 0, 2, 2 𝑇
𝒂2 = −6, 42, 24, 54 𝑇
𝒂3 = 21, −21, 0, −15 𝑇
9
MATRIX RANK AND ITS PROPERTIES
• Rank A: Maximum number of linearly independent row vectors of A.

𝒂1 = 3, 0, 2, 2 𝑇 3 0 2 2
𝒂2 = −6, 42, 24, 54 𝑇 → −6 42 24 54
𝒂3 = 21, −21, 0, −15 𝑇 21 −21 0 −15

• Theorem: Row-equivalent matrices have the same rank .

• Example: Finding the rank systematically

3 0 2 2
−6 42 24 54
21 −21 0 −15
10
LINKING RANK AND LINEAR DEPENDENCY
• Theorem 1: Consider p vectors that each have n components. Then
these vectors are linearly independent if the matrix formed, with these
vectors as row vectors, has rank p. However, these vectors are linearly
dependent if that matrix has rank less than p.

• Theorem 2: The rank r of a matrix A equals the maximum number of

linearly independent column vectors of A.

• Theorem 3: Consider p vectors each having n components. If n<p,

then these vectors are linearly dependent.

11
FUNDAMENTAL THEOREM FOR LINEAR SYSTEMS
• Existence: The following linear system is consistent iff A and à have
the same rank.

𝑎11 𝑥1 + 𝑎12 𝑥2 + ⋯ + 𝑎1𝑛 𝑥𝑛 = 𝑏1

𝑎21 𝑥1 + 𝑎22 𝑥2 + ⋯ + 𝑎2𝑛 𝑥𝑛 = 𝑏2
⋯⋯⋯⋯⋯⋯⋯⋯⋯⋯⋯
𝑎𝑚1 𝑥1 + 𝑎𝑚2 𝑥2 + ⋯ + 𝑎𝑚𝑛 𝑥𝑛 = 𝑏𝑚

• Uniqueness: The above system has precisely one solution iff the rank
of A and à equals n.
• Infinitely many solutions: If this common rank r is less than n.
• Gauss elimination: If solutions exist, they can all be obtained by this
method.
12
HOMOGENEOUS LINEAR SYSTEM
• The following linear system always has the trivial solution 𝑥1 = ⋯ =
𝑥𝑛 = 0.
𝑎11 𝑥1 + 𝑎12 𝑥2 + ⋯ + 𝑎1𝑛 𝑥𝑛 = 0
𝑎21 𝑥1 + 𝑎22 𝑥2 + ⋯ + 𝑎2𝑛 𝑥𝑛 = 0
⋯⋯⋯⋯⋯⋯⋯⋯⋯⋯⋯
𝑎𝑚1 𝑥1 + 𝑎𝑚2 𝑥2 + ⋯ + 𝑎𝑚𝑛 𝑥𝑛 = 0

• Nontrivial solutions exist if and only if 𝑟𝑎𝑛𝑘(𝑨) < 𝑛.

13
NONHOMOGENEOUS LINEAR SYSTEM
If a nonhomogeneous linear system is consistent, then all of its solutions
are obtained as:

𝒙 = 𝒙0 + 𝒙ℎ

where 𝒙0 is any (fixed) solution and 𝒙ℎ runs through all the solutions of
the corresponding homogeneous system.

𝑎11 𝑥1 + 𝑎12 𝑥2 + ⋯ + 𝑎1𝑛 𝑥𝑛 = 𝑏1

𝑎21 𝑥1 + 𝑎22 𝑥2 + ⋯ + 𝑎2𝑛 𝑥𝑛 = 𝑏2
⋯⋯⋯⋯⋯⋯⋯⋯⋯⋯⋯
𝑎𝑚1 𝑥1 + 𝑎𝑚2 𝑥2 + ⋯ + 𝑎𝑚𝑛 𝑥𝑛 = 𝑏𝑚
14
DETERMINANTS
Determinant of order n: The scalar associated with an n × n matrix 𝑨 =
𝑎𝑗𝑘 denoted by
a11 a12 a1n
a21 a22 a2 n
D  det A     .
  
am 1 am 2 amn

15
EXAMPLES
• Find the minors of a 3-by-3 matrix wrt 𝑎22 .
• Find the determinant of the following:

1 3 0 1 0 3 1 10 3
𝑨= 2 6 4 𝑩= 2 4 6 𝑪= 0 4 6
−1 0 2 −1 2 0 0 0 1

−1 0 2 −1 0 2 −2 0 2
𝑫= 2 6 4 𝑬=2 2 6 4 𝑭= 4 6 4
1 3 0 1 3 0 2 3 0

16
PROPERTIES OF DETERMINANTS
1. Interchange of two rows multiplies the value of the determinant by −1.
2. Addition of a multiple of a row to another row does not alter the value
of the determinant.
3. Multiplication of a row by a nonzero constant c multiplies the value of
the determinant by c.
4. Transposition does NOT change the determinant.
5. A zero row or column renders the value of a determinant zero.
6. Proportional rows or columns render the value of a determinant zero.

17
RANK IN TERMS OF DETERMINANTS
Consider an m × n matrix 𝑨 = 𝑎𝑖𝑗 :
1. 𝑨 has rank r ≥ 1 if and only if 𝑨 has an r × r submatrix with a nonzero
determinant.
2. The determinant of any square submatrix with more than r rows,
contained in 𝑨 (if such a matrix exists!) has a value equal to zero.

Furthermore, if m = n, we have:
3. An n × n square matrix A has rank n if and only if:
det A ≠ 0
18
CRAMER’S THEOREM
Solution of Linear Systems by Determinants: For the following system
when 𝑚 = 𝑛
𝑎11 𝑥1 + 𝑎12 𝑥2 + ⋯ + 𝑎1𝑛 𝑥𝑛 = 𝑏1
𝑎21 𝑥1 + 𝑎22 𝑥2 + ⋯ + 𝑎2𝑛 𝑥𝑛 = 𝑏2
⋯⋯⋯⋯⋯⋯⋯⋯⋯⋯⋯
𝑎𝑚1 𝑥1 + 𝑎𝑚2 𝑥2 + ⋯ + 𝑎𝑚𝑛 𝑥𝑛 = 𝑏𝑚

The system has precisely one solution if 𝐷 = det 𝐴 ≠ 0 and:

19
MATRIX INVERSE
The inverse of an n × n matrix A = [ajk] is denoted by A−1 and is an n × n
matrix such that:

AA−1 = A−1A = I

• If A has an inverse, then A is called a nonsingular matrix. If A has no

inverse, then A is called a singular matrix.
• If A has an inverse, the inverse is unique (HW).

20
EXISTENCE OF INVERSE
Theorem: The inverse A−1 of an n × n matrix A exists iff rank A = n, thus
iff det A ≠ 0. Hence A is nonsingular if rank A = n and is singular if rank A
< n.
Proof: …

21
MANUAL MATRIX INVERSION
We use Gauss-Jordan elimination to find 𝑨−1 :
• Make 𝑨 𝑰
• Use Gauss elimination to find 𝑼 𝑯 where 𝑼 is upper triangular
• Diagonalize 𝑼 to get 𝑰 𝑨−1

Example:

−1 1 2 −0.7 0.2 0.3

𝑨= 3 −1 1 𝑨−1 = −1.3 −0.2 0.7
−1 3 4 0.8 0.2 −0.2

22
USEFUL FORMULAS
Theorem: Inverse of 𝑨 = 𝑎𝑖𝑗
𝐶11 𝐶21 ⋯ 𝐶𝑛1
1 T 1 𝐶21 𝐶22 ⋯ 𝐶𝑛2
𝐀−1 = 𝐶𝑖𝑗 =
det𝐀 det𝐀 ⋅ ⋅ ⋯ ⋅
𝐶1𝑛 𝐶2𝑛 ⋯ 𝐶𝑛𝑛

Example:
𝑎11 𝑎12 −1
1 𝑎22 −𝑎12
𝑨= 𝑎 𝑎22 𝐀 =
21 det𝐀 −𝑎21 𝑎11

−1 1 2 𝐶 𝐶21 𝐶31
𝑨= 3 1 11
−1 1 𝐀−1 = 𝐶21 𝐶22 𝐶32
−1 3 4 10 𝐶 𝐶23 𝐶33
31
23
USEFUL FORMULAS CONTINUED
• Diagonal Matrix 𝑨 = 𝑎𝑖𝑗 , 𝑎𝑖𝑗 = 0 𝑤ℎ𝑒𝑛 𝑖 ≠ 𝑗 : Has an inverse iff 𝑎𝑖𝑗 ≠ 0 for 𝑖 = 𝑗.
𝑨−1 is diagonal with entries 1/𝑎𝑖𝑖 .
• Inverse of matrix products: (AC … PQ)−1 = Q−1P−1 … C−1A−1
• 𝑑𝑒𝑡 𝑨𝑩 = 𝑑𝑒𝑡 𝑩𝑨 = 𝑑𝑒𝑡 𝑨 𝑑𝑒𝑡(𝑩) and det 𝑨−1 = 1/ det 𝑨
• 𝑨𝑇 −1
= 𝑨−1 𝑇
and 𝑑𝑒𝑡 𝑘𝑨 −1
= 𝑘 −𝑛 det 𝑨−1
• Cancellation Laws: Let A, B, C be n × n matrices. Then:
• If rank A = n and AB = AC, then B = C.
• If rank A = n, then AB = 0 implies B = 0. Hence if AB = 0, but A ≠ 0 as well as B
≠ 0, then rank A < n and rank B < n.
• If A is singular, so are BA and AB.
24
VECTOR SPACES
• Consider a nonempty set 𝑉 of vectors with the same size, when is 𝑉 a
vector space?
1. If ∀𝒂, 𝒃 ∈ 𝑉, then 𝛼𝒂 + 𝛽𝒃 ∈ 𝑉 for 𝛼, 𝛽 ∈ 𝑅
2. If ∀𝒂, 𝒃 ∈ 𝑉, then 𝒂 + 𝒃 = 𝒃 + 𝒂, 𝒂 + 0 = 𝒂, and 𝒂 + −𝒂 = 𝟎
3. If ∀𝒂, 𝒃, 𝒄 ∈ 𝑉, then 𝒂 + 𝒃 + 𝒄 = 𝒂 + 𝒃 + 𝒄

• Dimension of V, dim V:
• Max. possible number of linearly independent vectors in V

25
BASIS AND SPAN
• A linearly independent set in V consisting of a maximum possible
number of vectors in V is called a basis for V.

• Span of 𝒂1 , … , 𝒂𝑝 : The set of all linear combinations of 𝒂1 , … , 𝒂𝑝

• 𝒂1 , … , 𝒂𝑝 are a basis for V if (1) 𝒂1 , … , 𝒂𝑝 are linearly independent, and

(2) any vector in V can be expressed as a linear combination of
𝒂1 , … , 𝒂𝑝 .

• The row/column space of a matrix A have the same dimension.

26
EXAMPLES
• What are the rank and basis for row/column space of:

9 0 1 0
𝐴= 0 0 1 0
1 1 1 1
0 0 1 0
• What is the vector space consisting of all vectors with n components (n
real numbers)?

• Is this a vector space: All vectors in 𝑅4 with 𝑣1 = 2𝑣2 = 3𝑣3 = 4𝑣4

27
INNER PRODUCT SPACES
• Inner product: 𝒂𝑇 𝒃 = 𝒂, 𝒃 = 𝒂 ∙ 𝒃 = σ𝑛𝑖=1 𝑎𝑖 𝑏𝑖
• V is called a real inner product space if:
1. If ∀𝒂, 𝒃 ∈ 𝑉, then 𝒂, 𝒃 = 𝒃, 𝒂
2. If ∀𝒂, 𝒃, 𝒄 ∈ 𝑉 and 𝑞1 , 𝑞2 ∈ 𝑅, then 𝑞1 𝒂 + 𝑞2 𝒃, 𝒄 = 𝑞1 𝒂, 𝒄 + 𝑞2 𝒃, 𝒄
3. If ∀𝒂 ∈ 𝑉
1. 𝒂, 𝒂 ≥ 0
2. 𝒂, 𝒂 = 0 iff 𝒂 = 𝟎

• Orthogonal vector: zero inner product

• Norm: 𝒂 = 𝒂, 𝒂 ≥ 0 28
SOME USEFUL RELATIONS
• Cauchy-Schwarz inequality:
| 𝒂, 𝒂 | ≤ 𝒂 𝒃
• Triangular inequality:
𝒂+𝒃 ≤ 𝒂 + 𝒃
• Parallelogram equality:
𝟐 𝟐 𝟐 𝟐
𝒂+𝒃 + 𝒂−𝒃 =2 𝒂 + 𝒃

Examples:
• N-dimensional Euclidean space
• Function space
29
LINEAR TRANSFORMATIONS
• 𝐹 is a map (transformation or operator):

X, vector space
Y, vector space
𝒙
𝐹 𝒚

• Linear F (for all 𝒙, 𝒗 in X and c scalar):

30
EXAMPLES
• X = Rn and Y = Rm. Then, y = Ax is a linear transformation of Rn into
Rm.
• Some transformations in Cartesian coordinate system:

1 0 0 1 1 0
𝑨= ,𝑩 = ,𝑪=
0 1 1 0 0 −1
−1 0 𝑎 0
𝑫= ,𝑬 =
0 −1 0 1

• Find the linear transformation taking 𝑥1 , 𝑥2 to 2𝑥1 − 5𝑥2 , 3𝑥1 + 4𝑥2 .

31
COMPOSITION OF LINEAR TRANSFORMATIONS
• 𝐹 and 𝐺 are linear transformations:

W, vector space X, vector space Y, vector space

𝒙 𝒚
𝐺 𝐹
𝒘
𝐻

• Composition: 𝐻 = 𝐹 ∘ 𝐺 = 𝐹𝐺 = 𝐹(𝐺)
• H is linear (i.e., it satisfies the two properties on slide 30).

32