Chapter 1: Linear Algebra

- Vector space
-Linear transformation
- Equivalances for invertibility matrices

Scalar
A quantity (variable), described by a single
real number
e.g. Intensity of each
voxel in an MRI scan

Linear Algebra & Matrices, MfD 2010

Vector

Not a physics vector (magnitude, direction)

EXAMPLE:

x1
x 2
VECTOR=

i.e. A column
of numbers xn

Linear Algebra & Matrices, MfD 2010

Matrices
Rectangular display of vectors in rows and columns
Can inform about the same vector intensity at different
times or different voxels at the same time
Vector is just a n x 1 matrix

x11 x12 x13

x 21 x 22 x 23

x31 x32 x33

Linear Algebra & Matrices, MfD 2010

Matrices
Matrix locations/size defined as rows x columns (R x C)

d11
D d 21
d 31
1 4 7
A 2 5 8
3 6 9
Square (3 x 3)

d12
d 22
d 32

d13
d 23
d 33

d i j : ith row, jth column

x11 x12 x13

x11 x12 x13
x 21

x
22
x
23
x11 x12 x13

x21
x
22
x
23
11 xx33
12 x13

x 21
x31
xx32
x
22
x
23
11 xx33
12 x13

x 21
x31
xx32
x
22
x
23
x 21

x31
x32 x 22
x33x 23

x31
x32 x33
x31 x32 x33
3 dimensional (3 x 3 x 5)

Linear Algebra & Matrices, MfD 2010

1 4
A 2 5
3 6
Rectangular (3 x 2)

Basic concepts
Vector in Rn is an ordered
set of n real numbers.
e.g. v = (1,6,3,4) is in R4
(1,6,3,4) is a column
vector:
as opposed to a row
vector:

m-by-n matrix is an object

with m rows and n columns,
each entry fill with a real
number:

1

6
3

4

6 3 4
1 2 8

4 78 6
9 3 2

Matrices in MATLAB
Description
Matrix(X)

Type into MATLAB

X=[1 4 7;2 5 8;3 6 9]

Meaning
1 4 7
X 2 5 8
3 6 9

;=end of a row
Reference matrix values (X(row,column))

Note the : refers to all of row or column and , is the divider between rows and columns
3rd row

X(3, :)

2nd Element of 3rd column

X(2,3)

Elements 2&3 of column 2

X( [1 2], 2)

Special types of matrix

All zeros size 3x1

zeros(3,1)

All ones size 2x2

ones(2,2)

Linear Algebra & Matrices, MfD 2010

6 9

8
4

5
0

0
0

Transposition
1
b 1
2
column

b 1 1 2
T

row

1 2 3
A 5 4 1
6 7 4

Linear Algebra & Matrices, MfD 2010

d 3 4 9
row

1 5 6
AT 2 4 7
3 1 4

T
d 4
9

column

Basic concepts
Transpose: reflect vector/matrix on line:
T

a
a b
b

a b
a c

c d
b d

Note: (Ax)T=xTAT (Well define multiplication soon)

Vector norms:
Lp norm of v = (v1,,vk) is (i |vi|p)1/p
Common norms: L1, L2
Linfinity = maxi |vi|

Length of a vector v is L2(v)

Matrix Calculations
Addition
Commutative: A+B=B+A
Associative: (A+B)+C=A+(B+C)

2
AB
2

Subtraction
- By adding a negative matrix

Linear Algebra & Matrices, MfD 2010

4 1 0 2 1 4 0 3 4

5 3 1 2 3 5 1 5 6

Scalar multiplication
Scalar * matrix = scalar multiplication

Linear Algebra & Matrices, MfD 2010

Matrix Multiplication
When A is a mxn matrix & B is a kxl matrix, AB is only possible
if n=k. The result will be an mxl matrix
Simply put, can ONLY perform A*B IF:

Number of columns in A = Number of rows in B

A1 A2 A3

B13 B14

A4 A5 A6

A7 A8 A9

B15 B16

= m x l matrix

B17 B18

A10 A11 A12

Hint:
1) If you see this message in MATLAB:
??? Error using ==> mtimes
Inner matrix dimensions must agree
Linear Algebra & Matrices, MfD 2010

-Then columns in A is not equal to rows in B

Matrix multiplication
Multiplication method:
Sum over product of respective rows and columns

Matlab does all this for you!

Simply type: C = A * B
Hints:
1) You can work out the size of the output (2x2). In MATLAB, if you pre-allocate a
matrix this size (e.g. C=zeros(2,2)) then the calculation is quicker
Linear Algebra & Matrices, MfD 2010

Matrix multiplication
Matrix multiplication is NOT commutative
i.e the order matters!
ABBA

Matrix multiplication IS associative

A(BC)=(AB)C

Matrix multiplication IS distributive

A(B+C)=AB+AC
(A+B)C=AC+BC
Linear Algebra & Matrices, MfD 2010

Identity matrix
A special matrix which plays a similar role as the number 1 in number
multiplication?

For any nxn matrix A, we have A In = In A = A

For any nxm matrix A, we have In A = A, and A Im = A (so 2 possible matrices)
If the answers always A, why use an identity matrix?
Cant divide matrices, therefore to solve may problems have to use the inverse. The
identity is important in these types of calculations.
Linear Algebra & Matrices, MfD 2010

Identity matrix

Worked
example
A I3 = A
for a 3x3 matrix:

1+0+0

0+2+0

0+0+3

4+0+0

0+5+0

0+0+6

7+0+0

0+8+0

0+0+9

In Matlab: eye(r, c) produces an r x c identity matrix

Linear Algebra & Matrices, MfD 2010

Basic concepts
Vector products:
Dot product: u v u v u
1
T

Outer product:

u1
uv v1
u2
T

v1
u 2 u1v1 u 2 v2
v2

u1v1 u1v2

v2
u 2 v1 u 2 v2

Matrices as linear transformations

5 0 1 5

0 5 1 5
0 11 1

1 0 1 1

(stretching)

(rotation)

Matrices as linear transformations

0 1 1 0

1 0 0 1

1 0 1 1

0 0 1 0
1 c x x cy

0 1 y y

(reflection)

(projection)

(shearing)

Matrices as sets of constraints

x y z 1
2x y z 2
x
1 1 1 1

y
2 1 1 z 2

Special matrices
a 0 0

0 b 0
0 0 c

c
0

diagonal

a b

0 d
0 0

0
tri-diagonal

1 0 0

0 1 0
0 0 1

e
f

a 0

b c
d e

upper-triangular

0 lower-triangular
f

I (identity matrix)

Vector spaces
Formally, a vector space is a set of vectors which is
closed under addition and multiplication by real numbers.
A subspace is a subset of a vector space which is a
vector space itself, e.g. the plane z=0 is a subspace of
R3 (It is essentially R2.).
Well be looking at Rn and subspaces of Rn
Our notion of planes in R3
may be extended to
hyperplanes in Rn (of
dimension n-1)
Note: subspaces must
include the origin (zero
vector).

Linear system & subspaces

1 0
b1

u
2 3 b2
1 3 v b

1
0 b1


u 2 v 3 b2
1
3 b

3

Linear systems define certain

subspaces
Ax = b is solvable iff b may be
written as a linear combination
of the columns of A
The set of possible vectors b
forms a subspace called the
column space of A
(1,2,1)

(0,3,3)

Linear system & subspaces

The set of solutions to Ax = 0 forms a subspace
called the null space of A.

1 0
0

u
2 3 0
1 3 v 0

1 0 1 x 0

2 3 5 y 0
1 3 4 z 0

Null space: {(0,0)}

Null space: {(c,c,-c)}

Linear independence and basis

Vectors v1,,vk are linearly independent if
c1v1++ckvk = 0 implies c1==ck=0
(2,2)
(0,1)
(1,1)

(1,0)

1 0
0

u
2 3 0
1 3 v 0

i.e. the nullspace is the origin

|
|

v1 v2
|
|

| c1 0

v3 c2 0
| c3 0

Recall nullspace contained

only (u,v)=(0,0).
i.e. the columns are linearly
independent.

Linear independence and basis

If all vectors in a vector space may be
expressed as linear combinations of v1,,vk,
then v1,,vk span the space.
2
1


2

2

0
2
0

(0,0,1)

0

2 1
0

0

2 0
1

2
.9
.3



2 1.57 .2 1.29 1
2
0
0

(.1,.2,1)

(0,1,0)
(.3,1,0)
(1,0,0)

(.9,.2,0)

.1

2 .2
1

Linear independence and basis

A basis is a set of linearly independent vectors which span the
space.
The dimension of a space is the # of degrees of freedom of the
space; it is the number of vectors in any basis for the space.
A basis is a maximal set of linearly independent vectors and a
minimal set of spanning vectors.
2
1


2

2

0
2
0

(0,0,1)

0

2 1
0

0

2 0
1

2
.9
.3



2 1.57 .2 1.29 1
2
0
0

(.1,.2,1)

(0,1,0)
(.3,1,0)
(1,0,0)

(.9,.2,0)

.1

2 .2
1

Linear independence and basis

Two vectors are orthogonal if their dot product is 0.
An orthogonal basis consists of orthogonal vectors.
An orthonormal basis consists of orthogonal vectors
of unit length.
2
1


2

2

0
2
0

(0,0,1)

0

2 1
0

0

2 0
1

2
.9
.3



2 1.57 .2 1.29 1
2
0
0

(.1,.2,1)

(0,1,0)
(.3,1,0)
(1,0,0)

(.9,.2,0)

.1

2 .2
1

About subspaces
The rank of A is the dimension of the column space of A.
It also equals the dimension of the row space of A (the
subspace of vectors which may be written as linear
combinations of the rows of A).

1 0

2 3
1 3

(1,3) = (2,3) (1,0)

Only 2 linearly independent
rows, so rank = 2.

About subspaces
Fundamental Theorem of Linear Algebra:
If A is m x n with rank r,
Column space(A) has dimension r
Nullspace(A) has dimension n-r (= nullity of A)
Row space(A) = Column space(AT) has dimension r
Left nullspace(A) = Nullspace(AT) has dimension m - r
Rank-Nullity Theorem: rank + nullity = n => prove it!
(0,0,1)

(0,1,0)
(1,0,0)

1 0

0 1
0 0

m=3
n=2
r=2

Non-square matrices
1 0

0 1
2 3

1 0 2

0 1 3

m=3
n=2
r=2
System Ax=b may
not have a solution
(x has 2 variables
but 3 constraints).

1 0
1

x1
0 1 1
2 3 x2 1

m=2
n=3
r=2
System Ax=b is
underdetermined
(x has 3 variables
and 2 constraints).

x1
1 0 2 1

x2
0 1 3 x 1
3

Basis transformations
Before talking about basis transformations,
we need to recall matrix inversion and
projections.

Matrix inversion
To solve Ax=b, we can write a closed-form solution if
we can find a matrix A-1
s.t. AA-1 =A-1A=I (identity matrix)
Then Ax=b iff x=A-1b:
x = Ix = A-1Ax = A-1b
A is non-singular iff A-1 exists iff Ax=b has a unique
solution.
Note: If A-1,B-1 exist, then (AB)-1 = B-1A-1,
and (AT)-1 = (A-1)T

Projections
(2,2,2)

b = (2,2)

(0,0,1)

(0,1,0)
(1,0,0)

a = (1,0)
2 1 0 0 2


2 0 1 0 2
0 0 0 0 2

2
aT b
c T a
a a
0

Basis transformations
We may write v=(2,2,2) in terms of an alternate basis:
2 .9 .3 .1 1.57

2 .2 1 .2 1.29
2 0 0 1 2

2 1 0 0 2


2

0
1
0

2
2 0 0 1 2

(0,0,1)

(.1,.2,1)

(0,1,0)
(.3,1,0)
(1,0,0)

(.9,.2,0)

Components of (1.57,1.29,2) are projections of v onto new basis vectors,

normalized so new v still has same length.

Basis transformations
Given vector v written in standard basis, rewrite as vQ
in terms of basis Q.
If columns of Q are orthonormal, vQ = QTv
Otherwise, vQ = (QTQ)QTv

Special matrices
Matrix A is symmetric if A = AT
A is positive definite if xTAx>0 for all non-zero x (positive semidefinite if inequality is not strict)

1 0 0 a

a b c 0 1 0 b a 2 b 2 c 2
0 0 1 c

1 0 0 a

a b c 0 1 0 b a 2 b 2 c 2
0 0 1 c

Special matrices
Matrix A is symmetric if A = AT
A is positive definite if xTAx>0 for all non-zero x (positive semidefinite if inequality is not strict)
Useful fact: Any matrix of form ATA is positive semi-definite.

To see this, xT(ATA)x = (xTAT)(Ax) = (Ax)T(Ax) 0 => prove

it!

Determinants
If det(A) = 0, then A is
singular.
If det(A) 0, then A is
invertible.
To compute:
a b
Simple example: det
c d ad bc

Matlab: det(A)

Determinants
m-by-n matrix A is rank-deficient if it has
rank r < m ( n)
Thm: rank(A) < r iff
det(A) = 0 for all t-by-t submatrices,
rtm

Eigenvalues & eigenvectors

How can we characterize matrices?
The solutions to Ax = x in the form of eigenpairs
(,x) = (eigenvalue,eigenvector) where x is non-zero
To solve this, (A I)x = 0
is an eigenvalue iff det(A I) = 0

Eigenvalues & eigenvectors

(A I)x = 0
is an eigenvalue iff det(A I) = 0
Example:
4
5
1

A 0 3 / 4 6
0
0 1 / 2

4
5
1

det( A I ) 0
3/ 4
6 (1 )(3 / 4 )(1 / 2 )
0
0
1 / 2

1, 3 / 4, 1 / 2

Eigenvalues & eigenvectors

2 0

A
0 1

Eigenvalues = 2, 1 with
eigenvectors (1,0), (0,1)

Eigenvectors of a linear transformation A are not rotated (but will be

scaled by the corresponding eigenvalue) when A is applied.
(0,1)

Av

(1,0)

(2,0)

Solving Ax=b
x + 2y + z = 0
y- z=2
x
+2z= 1
------------x + 2y + z = 0
y- z=2
-2y + z= 1
------------x + 2y + z = 0
y- z=2
- z=5

1 2 1 0

0 1 1 2
1 0 2 1

1 0
1 2

0 1 1 2
0 2 1 1

1 2 1 0

0 1 1 2
0 0 1 5

Write system of
equations in matrix
form.

Subtract first row from

last row.

Add 2 copies of second

row to last row.

Now solve by back-substitution: z = -5, y = 2-z = 7, x = -2y-z = -9

Solving Ax=b & condition numbers

Matlab: linsolve(A,b)
How stable is the solution?
If A or b are changed slightly, how much does it
effect x?
The condition number c of A measures this:
c = max/ min
Values of c near 1 are good.