L.

Vandenberghe EE133A (Spring 2017)

5. Orthogonal matrices

matrices with orthonormal columns

orthogonal matrices

tall matrices with orthonormal columns

complex matrices with orthonormal columns

5-1
 Orthonormal vectors

a collection of real m-vectors a1, a2, . . . , an is orthonormal if

the vectors have unit norm: kaik = 1

they are mutually orthogonal: aTi aj = 0 if i 6= j

Example

0 1 1
1 1
0 , 1 , 1
1 2 0 2 0

Orthogonal matrices 5-2

 Matrix with orthonormal columns

A Rmn has orthonormal columns if its Gram matrix is the identity matrix:

T
 T  
A A = a1 a2 an a1 a2 an

aT1 a1 aT1 a2 T

a1 an
aT2 a1 aT2 a2 aT2 an
= .. .. ..

..
. . . .
aTn a1 aTn a2 aTn an

1 0 0
0 1 0
= .. .. . . ..

. . . .
0 0 1

there is no standard short name for matrix with orthonormal columns

Orthogonal matrices 5-3

 Matrix-vector product

if A Rmn has orthonormal columns, then the linear function f (x) = Ax

preserves inner products:

(Ax)T (Ay) = xT AT Ay = xT y

preserves norms:

1/2
T
kAxk = (Ax) (Ax) = (xT x)1/2 = kxk

preserves distances: kAx Ayk = kx yk

preserves angles:
T T
   
(Ax) (Ay) x y
(Ax, Ay) = arccos = arccos = (x, y)
kAxkkAyk kxkkyk

Orthogonal matrices 5-4

 Left invertibility

if A Rmn has orthonormal columns, then

A is left invertible with left inverse AT : by definition

AT A = I

A has linearly independent columns (from p. 4-24 or p. 5-2):

Ax = 0 = AT Ax = x = 0

A is tall or square: m n (see page 4-13)

Orthogonal matrices 5-5

 Outline

matrices with orthonormal columns

orthogonal matrices

tall matrices with orthonormal columns

complex matrices with orthonormal columns

 Orthogonal matrix

Orthogonal matrix

a square real matrix with orthonormal columns is called orthogonal

Nonsingularity (from equivalences on page 4-14): if A is orthogonal, then

A is invertible, with inverse AT :

T

A A=I
= AAT = I
A is square

AT is also an orthogonal matrix

rows of A are orthonormal (have norm one and are mutually orthogonal)

Note: if A Rmn has orthonormal columns and m > n, then AAT 6= I

Orthogonal matrices 5-6

 Permutation matrix

let = (1, 2, . . . , n) be a permutation (reordering) of (1, 2, . . . , n)

we associate with the n n permutation matrix A

Aii = 1, Aij = 0 if j 6= i

Ax is a permutation of the elements of x: Ax = (x1 , x2 , . . . , xn )

A has exactly one element equal to 1 in each row and each column

Orthogonality: permutation matrices are orthogonal

AT A = I because A has exactly one element equal to one in each row

n 
X 1 i=j
(AT A)ij = AkiAkj =
0 otherwise
k=1

AT = A1 is the inverse permutation matrix

Orthogonal matrices 5-7

 Example

permutation on {1, 2, 3, 4}

(1, 2, 3, 4) = (2, 4, 1, 3)

corresponding permutation matrix and its inverse

0 1 0 0 0 0 1 0
0 0 0 1 1 0 0 0
A=
1
, A1 T
=A =
0 0 0 0 0 0 1
0 0 1 0 0 1 0 0

AT is permutation matrix associated with the permutation

(1, 2, 3, 4) = (3, 1, 4, 2)

Orthogonal matrices 5-8

 Plane rotation

Rotation in a plane

Ax
 
cos sin
A= x
sin cos

Rotation in a coordinate plane in Rn: for example,

cos 0 sin
A= 0 1 0
sin 0 cos

describes a rotation in the (x1, x3) plane in R3

Orthogonal matrices 5-9

 Reflector

A = I 2aaT with a a unit-norm vector (kak = 1)

a reflector matrix is symmetric and orthogonal

AT A = (I 2aaT )(I 2aaT ) = I 4aaT + 4aaT aaT = I

Ax is reflection of x through the hyperplane H = {u | aT u = 0}

x y = x (aT x)a
= (I aaT )x
H y
z = y + (y x)
= (I 2aaT )x
line through a and origin z = Ax
(see page 2-44)
Orthogonal matrices 5-10
 Product of orthogonal matrices

if A1, . . . , Ak are orthogonal matrices and of equal size, then the product

A = A1 A2 Ak

is orthogonal:

AT A = (A1A2 Ak )T (A1A2 Ak )
= ATk AT2 AT1 A1A2 Ak
= I

Orthogonal matrices 5-11

 Linear equation with orthogonal matrix

linear equation with orthogonal coefficient matrix A of size n n

Ax = b

solution is
x = A1b = AT b

can be computed in 2n2 flops by matrix-vector multiplication

cost is less than order n2 if A has special properties; for example,

permutation matrix: 0 flops

reflector (given a): order n flops
plane rotation: order 1 flops

Orthogonal matrices 5-12

 Outline

matrices with orthonormal columns

orthogonal matrices

tall matrices with orthonormal columns

complex matrices with orthonormal columns

 Tall matrix with orthonormal columns

suppose A Rmn is tall (m > n) and has orthonormal columns

AT is a left inverse of A:
AT A = I

A has no right inverse; in particular

AAT 6= I

on the next pages, we give a geometric interpretation to the matrix AAT

Orthogonal matrices 5-13

 Range

the span of a collection of vectors is the set of all their linear combinations:

span(a1, a2, . . . , an) = {x1a1 + x2a2 + + xnan | x Rn}

the range of a matrix A Rmn is the span of its column vectors:

range(A) = {Ax | x Rn}

Example

1 0 1 x 1 + x3
range( 1 1 2 ) = x1 + x2 + 2x3 | x1, x2, x3 R
0 1 1 x2 + x3

Orthogonal matrices 5-14

 Projection on range of matrix with orthonormal columns

suppose A Rmn has orthonormal columns; we show that the vector

AAT b

is the orthogonal projection of an m-vector b on range(A)

n Ax = AAT b
range(A) = {Ax | x R }

x = AT b satisfies kAx bk < kAx bk for all x 6= x

this extends the result on page 2-12 (where A = (1/kak)a)

Orthogonal matrices 5-15

 Proof

the squared distance of b to an arbitrary point Ax in range(A) is

kAx bk2 = kA(x x) + Ax bk2 (where x = AT b)

= kA(x x)k2 + kAx bk2 + 2(x x)T AT (Ax b)
= kA(x x)k2 + kAx bk2
= kx xk2 + kAx bk2
kAx bk2

with equality only if x = x

line 3 follows because AT (Ax b) = x AT b = 0

line 4 follows from AT A = I

Orthogonal matrices 5-16

 Orthogonal decomposition

the vector b is decomposed as a sum b = z + y with

z range(A), y range(A)

y = b AAT b
b

range(A) z = AAT b

such a decomposition exists and is unique for every b

b = Ax + y, AT y = 0 AT b = x, y = b AAT b

Orthogonal matrices 5-17

 Outline

matrices with orthonormal columns

orthogonal matrices

tall matrices with orthonormal columns

complex matrices with orthonormal columns

 Gram matrix

A Cmn has orthonormal columns if its Gram matrix is the identity matrix:

H
 H  
A A = a1 a2 an a1 a2 an

aH H H

1 a1 a1 a2 a1 an
aH H
2 a1 a2 a2 aH
2 an

= .. .. ..

. . .
aH H
n a1 an a2 aH
n an

1 0 0
0 1 0
= .. .. . . ..

. . . .
0 0 1

columns have unit norm: kaik2 = aH

i ai = 1

columns are mutually orthogonal: aH

i aj = 0 for i 6= j

Orthogonal matrices 5-18

 Unitary matrix

Unitary matrix

a square complex matrix with orthonormal columns is called unitary

Inverse
H

A A=I
= AAH = I
A is square

a unitary matrix is nonsingular with inverse AH

if A is unitary, then AH is unitary

Orthogonal matrices 5-19

 Discrete Fourier transform matrix

recall definition from page 3-37 (with = e 2j/n
and j = 1)

1 1 1 1

1 1 2 (n1)
2 4 2(n1)

W =
1
.. .. .. ..


. . . .
1 (n1) 2(n1) (n1)(n1)

the matrix (1/ n)W is unitary (proof on next page):

1 H 1
W W = WWH = I
n n

inverse of W is W 1 = (1/n)W H
inverse discrete Fourier transform of n-vector x is W 1x = (1/n)W H x

Orthogonal matrices 5-20

 Gram matrix of DFT matrix

we show that W H W = nI

conjugate transpose of W is

1 1 1 1
1 2 n1
2 4 2(n1)

WH =
1
.. .. .. ..


. . . .
1 n1 2(n1) (n1)(n1)

i, j element of Gram matrix is

(W H W )ij = 1 + ij + 2(ij) + + (n1)(ij)

n(ij) 1
H
(W W )ii = n, H
(W W )ij = = 0 if i 6= j
ij 1
(last step follows from n = 1)

Orthogonal matrices 5-21