Linear Algebra by Prof. R.

Vittal Rao
Lecture 1: August 05, 2010
1.1

co-ordinates of the point of intersection of the lines represented by

the two equations. Viewed from this geometric point of view, we
see that we can encounter the following situations:

Linear Systems

Historically, the subject of linear algebra has its origins in the

study of linear systems of equations. It is therefore natural that we
begin our discussions with a look at linear systems.
Example 1.1 Consider the system of equations
)
x + 3y = 6
xy =2

(1.1)

It is easy to see that x = 3, y = 1 is a solution of this system, and

that this is the only solution. We can view this geometrically as
follows:
The first equation represents a line L1 whose X and Y intercepts
are 6 and 2 respectively. The second equation represents the line
L2 whose X and Y intercepts are 2 and 2 respectively. They meet
at the point whose co-ordinates are x = 3 and y = 1.
3
2
(3, 1)
1
0
0 1 2 3 4 5 6 7
-1
-2

1. The two lines intersect at a point, in which case we get exactly

one solution given by the coordinates of the the point of intersection,
2. The two lines coincide with each other, in which case we get an
infinite number of solutions, given by the coordinates of every
point on this line, and
3. The two lines are distinct and parallel to each other, in which case
there is no solution since the parallel lines do not intersect at
any point.
The example we considered earlier corresponds to the case 1.
Example 1.2 Consider the system
x + 3y = 6
2x + 6y = 12
This corresponds to case 2.

Thus, in the example, the solution to the system is given by the

1

)
(1.2)

1.2 The Basic Questions

Example 1.3 Consider the system
x + 3y = 6
2x + 6y = 10

1.2

)
(1.3)

This corresponds to case 3.

In general, we would like to look at m equations in n unknowns.
We can write such a linear system as
a11 x1 +
a21 x1 +
..
.

a12 x2 + +
a22 x2 + +
..
.

a1n xn =
a2n xn =
..
.

b1 ,
b2 ,
..
.

am1 x1 + am2 x2 + + amn xn = bm .

In matrix notation, the same can be written as
Ax = b

(1.4)

where A = (ai j )1im,1jn is a known matrix, and for any given

b = (bi )1im we have to find x = (x j )1jn .
We have to make this a little more precise, as we have to mention
where the entries ai j of the matrix A, the entries bi of b are from
and where we are seeking the entries xi of x. For the moment, let
us say all are real or complex. We shall denote by F either R or C.
Thus, the problem can be precisely stated as
Problem 1.1 Given A Fmn , for b Fm given, to find x Fn such
that
Ax = b
is the problem of linear systems.

The Basic Questions

We have seen systems (1.2) and (1.3) which can be written in

matrix notation as
!
!
6
6
Ax =
and Ax =
12
10
!
1 3
where A =
. Even though the matrix of the coefficients
2 6
is the same, we found out that the system (1.2) has solution(s),
whereas the system (1.3) has no solution. Thus, we see that for a
given m n matrix A, the system Ax = b may have solution for
some b Rm and may not have solution for some other b Rm .
Thus, the first question we have to ask is the following:
Question 1.1 Given A Rmn , what criterion b Fm should satisfy
in order that the system Ax = b can have a solution?
We shall denote such a criterion by [C] and we call it consistency
condition(s). When b satisfies [C] we say the system is consistent.
This leads to a series of questions as given in Figure 1.1.

1.2.1

Least Square Solution

By b not satisfying [C], we mean that Ax = b has no solution. This

means that whatever x we choose in Rn , Ax will not be equal to b.
This means
b Ax , 0 for every x Rn .
We define

Eb (x) = b Ax

1.2 The Basic Questions

Does b satisfy [C]?

YES

NO (No solution exists)

There exists solution

for problem 1.1

What can be done?

How many?

Can find least

square solution
(see section 1.2.1)
How many?

Exactly One

Infinitely many

Under what
conditions

Under what
conditions

Exactly one

Infinitely many

Characterize
all solutions

Under what
conditions

Under what
conditions
Characterize all least
square solutions

Find a criterion and

pick a unique representative solution

Find the
unique solution

Find a criterion
to pick a unique
representative least
square solution

Find the unique

least square solution

Find the unique representative solution

Find the unique

representative least
square solution

Figure 1.1: Series of questions from the consistency condition(s) [C]

3

1.3 Easy Systems

as the error we get by taking x as the solution for Ax = b. Now, Eb (x)
is an m 1 matrix. We quantify this error by the sum of squares of
its entries by defining
eb (x) =

m
X

(bi (Ax)i )2 .

i=1

What we can show is that we can find xl Rn such that this error
is minimized, that is,
eb (xl ) eb (x) x Rn .

This means, we have to start from the known b and construct

xl . So, we have to construct a new system, for which inputs are
from Rm and outputs are in Rn . Let the matrix of this system be A
(should be n m).
I/P
O/P
b Rm
A
A x Rn
We must construct A in such a way that A b is the solution we
are looking for in each of the cases. Such a matrix A is called the
pseudo inverse of A.
Remark 1.1 We shall see that

Any such xl is called a least square solution.

1. When A is n n and A1 exists then A = A1 , and

1.2.2

2. When A is nn and A1 does not exist or A is mn, the pseudo

inverse A still can be constructed.

The Pseudo Inverse

We observe that in the string of questions we have raised, in each

case, ultimately, we want to find a unique vector as our suitable
solution, as the case may be. This can be viewed as follows:
Consider a system for which we input an x Rn and get an
output Ax Rm .
I/P
O/P
System Matrix,
x Rn
Ax Rm
mn
A R
When we say we want to solve Ax = b, what we are seeking
is a desired output b and we want to find which input can this
output b. The system Ax = b is consistent means that there is an
input x which will produce exactly the output b. The system is
inconsistent means that there is no input which can produce the
desired output b, but we can find input xl which will produce an
output Axl will error eb (xl ) smaller than or equal to the error for
any other input.
4

Thus, the notion of pseudo inverse is more general than that of

an inverse of a matrix.

1.3

Easy Systems

Let us now look at those systems which are relatively easy to be

analyzed. A general system Ax = b is such that the unknown are
all coupled in all the equations. Hence the system becomes easier
when there is less coupling. The system in which the unknowns
are not coupled at all is the easiest to treat. So, let us consider
a system of n equations in n unknowns in which the ith equation
involves only the ith unknown, xi . Such a system is of the form
aii xi = bi , 1 i n.

1.4 Exercises
The matrix of such a system is

D = (aii )1in =

a11
0
..
.
0

0 ...
a22 . . .
..
..
.
.
0 ...

0
0
..
.
ann

and hence an easy system. Thus, we can analyze y and hence

x = Py. So, the basic question that arises is the following:

Question 1.2 Let F = R or C. Given any n n matrix A over F, can

we find an n n matrix over F which is invertible and such that
P1 AP is a diagonal matrix?

is a diagonal matrix. Thus, a system whose coefficient matrix is

a diagonal matrix is the easiest system. We shall soon see how to
analyze such a system.

1.3.1

If such a P exists, we say that A is diagonalizable over F.

1.4

Reduction to a diagonal system

The question that arises naturally is whether we can reduce a

general system of n equations in n unknowns to a diagonal system.
Consider,
Ax = b where A is n n matrix.

Exercises

1. Discuss, geometrically, the various alternatives that may arise

for the solution of the system of the type
)
a1 x + a2 y + a3 z = b1
1 x + 2 y + 3 z = b2
where a1 , a2 , a3 , 1 , 2 , 3 , b1 , b2 R.

Let us introduce a change of variables,

y = x,
c = b

2. For the diagonal system (over R), Dx = b where

b1
2 0 0 0
0 1 0 0

, b = b2 ,
D =

b3
0 0 4 0

0 0 0 6
b4

where is an invertible n n matrix. Then, we have

x = 1 y,
b = 1 c.
Let P = 1 . Then, x = Py, b = Pc. The system Ax = b hence
becomes
P1 APy = c.
If we can choose P such that P1 AP = D, a diagonal matrix, then
the system for y is a diagonal system
Dy = c
5

discuss the solution of the system.

3. Do the same as above for

2 0 0
0 1 0

D =
0 0 0

0 0 0

0
0
0
0