Lmaster 6

Applied Mathematical Methods 1, Applied Mathematical Methods 2,
Contents I
Preliminary Background
Matrices and Linear Transformations

Applied Mathematical Methods
Operational Fundamentals of Linear Algebra
Bhaskar Dasgupta
Systems of Linear Equations
Department of Mechanical Engineering
Indian Institute of Technology
Kanpur (INDIA) Gauss Elimination Family of Methods
dasgupta@iitk.ac.in
Special Systems and Special Methods

(Pearson Education 2006, 2007)
Numerical Aspects in Linear Systems

May 13, 2008
Contents II Contents III

Eigenvalues and Eigenvectors Topics in Multivariate Calculus
Diagonalization and Similarity Transformations Vector Analysis: Curves and Surfaces
Jacobi and Givens Rotation Methods Scalar and Vector Fields
Householder Transformation and Tridiagonal Matrices Polynomial Equations
QR Decomposition Method Solution of Nonlinear Equations and Systems
Eigenvalue Problem of General Matrices Optimization: Introduction
Singular Value Decomposition Multivariate Optimization
Vector Spaces: Fundamental Concepts* Methods of Nonlinear Optimization*
Contents IV Contents V
Constrained Optimization First Order Ordinary Differential Equations
Linear and Quadratic Programming Problems* Second Order Linear Homogeneous ODEs
Interpolation and Approximation Second Order Linear Non-Homogeneous ODEs
Basic Methods of Numerical Integration Higher Order Linear ODEs
Advanced Topics in Numerical Integration* Laplace Transforms
Numerical Solution of Ordinary Differential Equations ODE Systems
ODE Solutions: Advanced Issues Stability of Dynamic Systems
Existence and Uniqueness Theory Series Solutions and Special Functions

Contents VI Contents VII

Variational Calculus*
Sturm-Liouville Theory
Epilogue
Fourier Series and Integrals
Selected References
Fourier Transforms
Minimax Approximation*
Partial Differential Equations
Analytic Functions
Integrals in the Complex Plane
Singularities of Complex Functions
Applied Mathematical Methods Preliminary Background 9, Applied Mathematical Methods Preliminary Background 10,
Outline Theme of the Course

Course Contents
Sources for More Detailed Study
Theme of the Course Theme of the Course
Course Contents
Logistic Strategy Logistic Strategy
Expected Background Expected Background
To develop a firm mathematical background necessary for graduate
studies and research
I a fast-paced recapitulation of UG mathematics
Preliminary Background I extension with supplementary advanced ideas for a mature
Theme of the Course and forward orientation
Course Contents
I exposure and highlighting of interconnections
Logistic Strategy
Expected Background To pre-empt needs of the future challenges
I trade-off between sufficient and reasonable
I target mid-spectrum majority of students
Notable beneficiaries (at two ends)

I would-be researchers in analytical/computational areas
I students who are till now somewhat afraid of mathematics
Course Contents Theme of the Course

Course Contents
Sources for More Detailed Study Theme of the Course
Course Contents
If you have the time, need and interest, then you may consult
I Applied linear algebra I individual books on individual topics;
I Multivariate calculus and vector calculus I another umbrella volume, like Kreyszig, McQuarrie, ONeil
I Numerical methods or Wylie and Barrett;
I Differential equations + + I a good book of numerical analysis or scientific computing, like
Acton, Heath, Hildebrand, Krishnamurthy and Sen, Press et
I Complex analysis
al, Stoer and Bulirsch;
I friends, in joint-study groups.
Logistic Strategy Theme of the Course

Course Contents
Logistic Strategy Theme of the Course
Course Contents
I Study in the given sequence, to the extent possible.
Tutorial Plan
I Do not read mathematics.
Chapter Selection Tutorial Chapter Selection Tutorial
I Use lots of pen and paper. 2 2,3 3 26 1,2,4,6 4
Read mathematics books and do mathematics. 3
4
2,4,5,6
1,2,4,5,7
4,5
4,5
27
28
1,2,3,4
2,5,6
3,4
6
I Exercises are must. 5
6
1,4,5 4 29
30
1,2,5,6 6
1,2,4,7 4 1,2,3,4,5 4
I Use as many methods as you can think of, certainly including 7 1,2,3,4 2 31 1,2 1(d)
8 1,2,3,4,6 4 32 1,3,5,7 7
the one which is recommended. 9 1,2,4 4 33 1,2,3,7,8 8
I Consult the Appendix after you work out the solution. Follow 10 2,3,4 4 34 1,3,5,6 5
11 2,4,5 5 35 1,3,4 3
the comments, interpretations and suggested extensions. 12 1,3 3 36 1,2,4 4
I Think. Get excited. Discuss. Bore everybody in your known 13 1,2 1 37 1 1(c)
14 2,4,5,6,7 4 38 1,2,3,4,5 5
circles. 15 6,7 7 39 2,3,4,5 4
I Not enough time to attempt all? Want a selection ? 16 2,3,4,8 8 40 1,2,4,5 4
17 1,2,3,6 6 41 1,3,6,8 8
I Program implementation is needed in algorithmic exercises. 18 1,2,3,6,7 3 42 1,3,6 6
19 1,3,4,6 6 43 2,3,4 3
I Master a programming environment. 20 1,2,3 2 44 1,2,4,7,9,10 7,10
I Use mathematical/numerical library/software. 21
22
1,2,5,7,8 7 45
46
1,2,3,4,7,9 4,9
1,2,3,4,5,6 3,4 1,2,5,7 7
Take a MATLAB tutorial session? 23 1,2,3 3 47 1,2,3,5,8,9,10 9,10
24 1,2,3,4,5,6 1 48 1,2,4,5 5
25 1,2,3,4,5 5
Expected Background Theme of the Course

Course Contents
Points to note Theme of the Course
Course Contents
I moderate background of undergraduate mathematics I Put in effort, keep pace.

I firm understanding of school mathematics and undergraduate I Stress concept as well as problem-solving.
calculus
I Follow methods diligently.
I Ensure background skills.
Take the preliminary test.
Necessary Exercises: Prerequisite problem sets ??

Grade yourself sincerely.
Prerequisite Problem Sets*
Applied Mathematical Methods Matrices and Linear Transformations 17, Applied Mathematical Methods Matrices and Linear Transformations 18,
Outline Matrices
Geometry and Algebra
Linear Transformations
Matrices Matrices
Matrix Terminology Matrix Terminology
Question: What is a matrix?

Answers:
I a rectangular array of numbers/elements ?
Matrices and Linear Transformations I a mapping f : M N F , where M = {1, 2, 3, , m},
Matrices N = {1, 2, 3, , n} and F is the set of real numbers or
Geometry and Algebra complex numbers ?
Matrix Terminology Question: What does a matrix do?
Explore: With an m n matrix A,

y1 = a11 x1 + a12 x2 + + a1n xn

y2 = a21 x1 + a22 x2 + + a2n xn
.. .. ... .. .. or Ax = y
. . . .

ym = am1 x1 + am2 x2 + + amn xn
Matrices Matrices
Geometry and Algebra Matrices
Consider these definitions: Matrix Terminology Matrix Terminology
Let vector x = [x1 x2 x3 ]T denote a point (x1 , x2 , x3 ) in
I y = f (x)
3-dimensional space in frame of reference OX 1 X2 X3 .
I y = f (x) = f (x1 , x2 , , xn )
Example: With m = 2 and n = 3,
I yk = fk (x) = fk (x1 , x2 , , xn ), k = 1, 2, , m
I y = f(x)

y1 = a11 x1 + a12 x2 + a13 x3
I y = Ax .
y2 = a21 x1 + a22 x2 + a23 x3
Further Answer:
A matrix is the definition of a linear vector function of a Plot y1 and y2 in the OY1 Y2 plane.
vector variable. A: R3 R
2
Anything deeper? X3
Y2
y

x Y1
O X2

O
X1
Domain Codomain
Caution: Matrices do not define vector functions whose components are
Figure: Linear transformation: schematic illustration
of the form
What is matrix A doing?
yk = ak0 + ak1 x1 + ak2 x2 + + akn xn .
Geometry and Algebra Matrices

Linear Transformations Matrices
Operate A on a large number of points x i R 3 .

Operating on point x in R 3 , matrix A transforms it to y in R 2 . Obtain corresponding images yi R 2 .
The linear transformation represented by A implies the totality of

Point y is the image of point x under the mapping defined by these correspondences.
matrix A.
We decide to use a different frame of reference OX 10 X20 X30 for R 3 .
[And, possibly OY10 Y20 for R 2 at the same time.]
Note domain R 3 , co-domain R 2 with reference to the figure and
verify that A : R 3 R 2 fulfils the requirements of a mapping, by Coordinates change, i.e. xi changes to x0i (and possibly yi to yi0 ).
definition. Now, we need a different matrix, say A 0 , to get back the
correspondence as y0 = A0 x0 .
A matrix gives a definition of a linear transformation
A matrix: just one description.
from one vector space to another.
Question: How to get the new matrix A 0 ?
Matrix Terminology Matrices

Points to note Matrices
I I A matrix defines a linear transformation from one vector space

I Matrix product to another.
I Transpose I Matrix representation of a linear transformation depends on
the selected bases (or frames of reference) of the source and
I Conjugate transpose
target spaces.
I Symmetric and skew-symmetric matrices
Important: Revise matrix algebra basics as necessary tools.
I Hermitian and skew-Hermitian matrices
I Determinant of a square matrix
I Inverse of a square matrix Necessary Exercises: 2,3
I Adjoint of a square matrix
I
Applied Mathematical Methods Operational Fundamentals of Linear Algebra 25, Applied Mathematical Methods Operational Fundamentals of Linear Algebra 26,
Outline Range and Null Space: Rank and Nullity

Basis
Change of Basis
Range and Null Space: Rank and Nullity
Basis
Change of Basis
Elementary Transformations Elementary Transformations
Consider A R mn as a mapping
A : R n R m, Ax = y, x R n, y R m.
Operational Fundamentals of Linear Algebra

Range and Null Space: Rank and Nullity Observations
Basis 1. Every x R n has an image y R m , but every y R m need
Change of Basis not have a pre-image in R n .
Elementary Transformations Range (or range space) as subset/subspace of
co-domain: containing images of all x R n .
2. Image of x R n in R m is unique, but pre-image of y R m

need not be.
It may be non-existent, unique or infinitely many.
Null space as subset/subspace of domain:
containing pre-images of only 0 R m .

Basis
Change of Basis
Basis Range and Null Space: Rank and Nullity
Basis
Change of Basis
n
R m
R
0
Take a set of vectors v1 , v2 , , vr in a vector space.
O A
Range ( A ) Question: Given a vector v in the vector space, can we describe it
Null ( A) as
Domain Codomain v = k1 v1 + k2 v2 + + kr vr = Vk,
Figure: Range and null space: schematic representation where V = [v1 v2 vr ] and k = [k1 k2 kr ]T ?
Answer: Not necessarily.
Question: What is the dimension of a vector space?
Linear dependence and independence: Vectors x 1 , x2 , , xr Span, denoted as < v1 , v2 , , vr >: the subspace
in a vector space are called linearly independent if described/generated by a set of vectors.
k1 x1 + k 2 x2 + + k r xr = 0 k1 = k2 = = kr = 0. Basis:
A basis of a vector space is composed of an ordered
minimal set of vectors spanning the entire space.
Range(A) = {y : y = Ax, x R n }
Null(A) = {x : x R n , Ax = 0} The basis for an n-dimensional space will have exactly n
Rank(A) = dim Range(A) members, all linearly independent.
Nullity(A) = dim Null(A)
Basis Range and Null Space: Rank and Nullity

Basis
Change of Basis
Change of Basis Range and Null Space: Rank and Nullity
Basis
Change of Basis
Orthogonal basis: {v1 , v2 , , vn } with Elementary Transformations nElementary Transformations
Suppose x represents a vector (point) in R in some basis.
vjT vk = 0 j 6= k. Question: If we change over to a new basis {c 1 , c2 , , cn }, how
does the representation of a vector change?
Orthonormal basis:

0 if j=6 k
vjT vk = jk =
1 if j =k x = x1 c1 + x2 c2 + + xn cn

Members of an orthonormal basis form an orthogonal matrix. x1
Properties of an orthogonal matrix: x2

= [c1 c2 cn ] .. .
V1 = VT or VVT = I, and .
det V = +1 or 1, xn
Natural basis: With C = [c1 c2 cn ],

1 0 0 new to old coordinates: Cx = x and
0 1 0 old to new coordinates: x = C1 x.

0 0 0
e1 = , e2 = , , en = .
.. .. .. Note: Matrix C is invertible. How?
. . .
Special case with C orthogonal:
0 0 1 orthogonal coordinate transformation.
Change of Basis Range and Null Space: Rank and Nullity

Basis
Change of Basis
Elementary Transformations Range and Null Space: Rank and Nullity
Basis
Change of Basis
Question: And, how does basis change affect the representation of

a linear transformation? Observation: Certain reorganizations of equations in a system
Consider the mapping A : R n R m, Ax = y. have no effect on the solution(s).
Change the basis of the domain through P R nn and that of the

co-domain through Q R mm . Elementary Row Transformations:
New and old vector representations are related as 1. interchange of two rows,
2. scaling of a row, and
Px = x and Qy = y. 3. addition of a scalar multiple of a row to another.
Then, Ax = y Ax = y, with
A = Q1 AP Elementary Column Transformations: Similar operations with
Special case: m = n and P = Q gives a similarity transformation columns, equivalent to a corresponding shuffling of the variables
(unknowns).
A = P1 AP
Elementary Transformations Range and Null Space: Rank and Nullity

Basis
Change of Basis
Points to note Range and Null Space: Rank and Nullity
Basis
Change of Basis
Equivalence of matrices: An elementary transformation defines
an equivalence relation between two matrices.
Reduction to normal form: I Concepts of range and null space of a linear transformation.
I Effects of change of basis on representations of vectors and
Ir 0
AN = linear transformations.
0 0
I Elementary transformations as tools to modify (simplify)
Rank invariance: Elementary transformations do not alter the systems of (simultaneous) linear equations.
rank of a matrix.
Elementary transformation as matrix multiplication:
an elementary row transformation on a matrix is Necessary Exercises: 2,4,5,6
equivalent to a pre-multiplication with an elementary
matrix, obtained through the same row transformation on
the identity matrix (of appropriate size).
Similarly, an elementary column transformation is equivalent to

post-multiplication with the corresponding elementary matrix.
Applied Mathematical Methods Systems of Linear Equations 35, Applied Mathematical Methods Systems of Linear Equations 36,
Outline Nature of Solutions

Basic Idea of Solution Methodology
Homogeneous Systems
Nature of Solutions Nature of Solutions
Homogeneous Systems
Pivoting Pivoting
Partitioning and Block Operations
Ax = b Partitioning and Block Operations
Coefficient matrix: A, augmented matrix: [A | b].

Existence of solutions or consistency:
Systems of Linear Equations
Ax = b has a solution
Nature of Solutions
Basic Idea of Solution Methodology b Range(A)
Homogeneous Systems Rank(A) = Rank([A | b])
Pivoting
Partitioning and Block Operations Uniqueness of solutions:
Rank(A) = Rank([A | b]) = n

Solution of Ax = b is unique.
Ax = 0 has only the trivial (zero) solution.
Infinite solutions: For Rank(A) = Rank([A|b]) = k < n, solution
x = x + xN , with Ax = b and xN Null(A)

Basic Idea of Solution Methodology Nature of Solutions

Homogeneous Systems
Homogeneous Systems Nature of Solutions
Homogeneous Systems
Pivoting Pivoting
Partitioning and Block Operations To solve Ax = 0 or to describe Null(A), Partitioning and Block Operations
To diagnose the non-existence of a solution, apply a series of elementary row transformations on A to reduce it

To determine the unique solution, or to the A,
To describe infinite solutions; the row-reduced echelon form or RREF.
decouple the equations using elementary transformations. Features of RREF:
1. The first non-zero entry in any row is a 1, the leading 1.
For solving Ax = b, apply suitable elementary row transformations
2. In the same column as the leading 1, other entries are zero.
on both sides, leading to
3. Non-zero entries in a lower row appear later.
Rq Rq1 R2 R1 Ax = Rq Rq1 R2 R1 b,
Variables corresponding to columns having leading 1s
or, [RA]x = Rb;
are expressed in terms of the remaining variables.

such that matrix [RA] is greatly simplified. u1
In the best case, with complete reduction, RA = I n , and u2
components of x can be read off from Rb. Solution of Ax = 0: x = z1 z2 znk

For inverting matrix A, treat AA1 = In similarly. unk

Basis of Null(A): {z1 , z2 , , znk }
Pivoting Nature of Solutions

Homogeneous Systems
Partitioning and Block Operations Nature of Solutions
Homogeneous Systems
Pivoting Pivoting
Attempt: Partitioning and Block Operations Partitioning and Block Operations
To get 1 at diagonal (or leading) position, with 0 elsewhere. Equation Ax = y can be written as
Key step: division by the diagonal (or leading) entry.

Consider x
Ik . . . . . A11 A12 A13 1 y1
x2 = ,
. . . . . A21 A22 A23 y2
x3
. . . . BIG .

A = .
. big . . . .
with x1 , x2 etc being themselves vectors (or matrices).
. . . . . .
I For a valid partitioning, block sizes should be consistent.
. . . . . .
I Elementary transformations can be applied over blocks.
Cannot divide by zero. Should not divide by .
I Block operations can be computationally economical at times.
I partial pivoting: row interchange to get big in place of I Conceptually, different blocks of contributions/equations can
I complete pivoting: row and column interchanges to get be assembled for mathematical modelling of complicated
BIG in place of coupled systems.
Complete pivoting does not give a huge advantage over partial pivoting,
but requires maintaining of variable permutation for later unscrambling.
Applied Mathematical Methods Systems of Linear Equations 41, Applied Mathematical Methods Gauss Elimination Family of Methods 42,
Points to note Nature of Solutions

Homogeneous Systems
Outline Gauss-Jordan Elimination
Gaussian Elimination with Back-Substitution
LU Decomposition
Pivoting
Partitioning and Block Operations
I Solution(s) of Ax = b may be non-existent, unique or

infinitely many.
I Complete solution can be described by composing a particular Gauss Elimination Family of Methods
solution with the null space of A. Gauss-Jordan Elimination
I Null space basis can be obtained conveniently from the Gaussian Elimination with Back-Substitution
row-reduced echelon form of A. LU Decomposition
I For a strategy of solution, pivoting is an important step.
Necessary Exercises: 1,2,4,5,7

Applied Mathematical Methods Gauss Elimination Family of Methods 43, Applied Mathematical Methods Gauss Elimination Family of Methods 44,
Gauss-Jordan Elimination Gauss-Jordan Elimination

LU Decomposition
Gauss-Jordan Elimination Gauss-Jordan Elimination
LU Decomposition
Task: Solve Ax = b1 , Ax = b2 and Ax = b3 ; find A1 and

evaluate A1 B, where A R nn and B R np . Gauss-Jordan Algorithm
Assemble C = [A b1 b2 b3 In B] R n(2n+3+p) I =1
and follow the algorithm . I For k = 1, 2, 3, , (n 1)
1. Pivot : identify l such that |clk | = max |cjk | for k j n.
Collect solutions from the result
If clk = 0, then = 0 and exit.
Else, interchange row k and row l.
C C = [In A1 b1 A1 b2 A1 b3 A1 A1 B]. 2. ckk ,
Divide row k by ckk .
Remarks: 3. Subtract cjk times row k from row j, j 6= k.
I Premature termination: matrix A singular decision? I cnn
I If you use complete pivoting, unscramble permutation. If cnn = 0, then exit.
Else, divide row n by cnn .
I Identity matrix in both C and C? Store A1 in place.
I For evaluating A1 b, do not develop A1 . default termination .
In case of non-singular A,
I Gauss-Jordan elimination an overkill? Want something
cheaper ?
This outline is for partial pivoting.

Gauss-Jordan Elimination
LU Decomposition
Gauss-Jordan Elimination
LU Decomposition
Gaussian elimination: Anatomy of the Gaussian elimination:
The process of Gaussian elimination (with no pivoting) leads to
Ax = b
U = Rq Rq1 R2 R1 A = RA.
Ax = b
0 0 0
0
a11 a12 a1n x1 b1 The steps given by
0
a22 0
a2n x2 b0 for k = 1, 2, 3, , (n 1)
2
or, .. .. .. = .. ajk
. . . . j-th row j-th row akk k-th row for
0
ann xn bn0 j = k + 1, k + 2, , n
Back-substitutions:
involve elementary matrices
xn = bn0 /ann
0
,
1 0 0 0
n
X
1 0 0

aa21 1 0 0

xi = bi aij xj for i = n 1, n 2, , 2, 1
11

aii0 Rk |k=1 = aa31
11
0 1 0 etc.
j=i +1
.. .. .. . . ..
. . . . .
Remarks
I Computational cost half compared to G-J elimination.
aa11
n1
0 0 1
I Like G-J elimination, prior knowledge of RHS needed. With L = R1 , A = LU.
LU Decomposition Gauss-Jordan Elimination

LU Decomposition
LU Decomposition
A square matrix with non-zero leading minors is LU-decomposable. Question: How to LU-decompose a given matrix?
No reference to a right-hand-side (RHS) vector!
To solve Ax = b, denote y = Ux and split as
l11 0 0 0 u11 u12 u13 u1n
l21 l22 0 0 0 u22 u23 u2n

Ax = b LUx = b l31 l32 l33 0 0 0 u33 u3n
L= and U =
Ly = b and Ux = y. .. .. .. .. .. .. .. .. .. ..
. . . . . . . . . .
ln1 ln2 ln3 lnn 0 0 0 unn
Forward substitutions:

i 1
X Elements of the product give
1
yi = bi lij yj for i = 1, 2, 3, , n;
lii i
X
j=1
lik ukj = aij for i j,
Back-substitutions: k=1
Xj
n
X and lik ukj = aij for i > j.
1
xi = yi uij xj for i = n, n 1, n 2, , 1. k=1
uii
j=i +1
n2 equations in n 2 + n unknowns: choice of n unknowns

LU Decomposition
LU Decomposition
Question: What about matrices which are not LU-decomposable?

Doolittles algorithm Question: What about pivoting?
I Choose lii = 1 Consider the non-singular matrix
I For j = 1, 2, 3, , n
P 1 0 1 2 1 0 0 u11 = 0 u12 u13
1. uij = aij ik=1 lik ukj for 1 i j
1 Pj1 3 1 2 = l21 =? 1 0 0 u22 u23 .
2. lij = ujj (aij k=1 lik ukj ) for i > j
2 1 3 l31 l32 1 0 0 u33
Evaluation proceeds in column order of the matrix (for storage) LU-decompose a permutation of its rows

u11 u12 u13 u1n 0 1 2 0 1 0 3 1 2
l21 u22 u23 3 1 2 = 1 0 0 0 1 2
u2n

2 1 3 0 0 1 2 1 3
A = l31 l32 u33 u3n

.. .. .. .. .. 0 1 0 1 0 0 3 1 2
. . . . . = 1 0 0 0 1 0 0 1 2 .
ln1 ln2 ln3 unn 0 0 1 2 1
1 0 0 1
3 3
In this PLU decomposition, permutation P is recorded in a vector.
Applied Mathematical Methods Gauss Elimination Family of Methods 51, Applied Mathematical Methods Special Systems and Special Methods 52,
Points to note Gauss-Jordan Elimination

LU Decomposition
Outline Quadratic Forms, Symmetry and Positive Definiteness
Cholesky Decomposition
Sparse Systems*
For invertible coefficient matrices, use

I Gauss-Jordan elimination for large number of RHS vectors
available all together and also for matrix inversion,
Special Systems and Special Methods
I Gaussian elimination with back-substitution for small number Quadratic Forms, Symmetry and Positive Definiteness
of RHS vectors available together, Cholesky Decomposition
I LU decomposition method to develop and maintain factors to Sparse Systems*
be used as and when RHS vectors are available.
Pivoting is almost necessary (without further special structure).
Necessary Exercises: 1,4,5
Applied Mathematical Methods Special Systems and Special Methods 53, Applied Mathematical Methods Special Systems and Special Methods 54,
Quadratic Forms, Symmetry and Positive Definiteness

Quadratic Forms, Symmetry and Positive Definiteness
Sparse Systems*
Cholesky Decomposition Quadratic Forms, Symmetry and Positive Definiteness
Sparse Systems*
Quadratic form If A R nn is symmetric and positive definite, then there exists a

n X
X n non-singular lower triangular matrix L R nn such that
q(x) = xT Ax = aij xi xj
i =1 j=1 A = LLT .
defined with respect to a symmetric matrix. Algorithm For i = 1, 2, 3, , n

q Pi 1 2
Quadratic form q(x), equivalently matrix A, is called positive I Lii = aii k=1 Lik
definite (p.d.) when Pi 1
1
Lii aji for i < j n
I Lji =
k=1 Ljk Lik
xT Ax > 0 x 6= 0
For solving Ax = b,
and positive semi-definite (p.s.d.) when Forward substitutions: Ly = b
xT Ax 0 x 6= 0. Back-substitutions: LT x = y
Sylvesters criteria: Remarks
Test of positive definiteness.
a a I
a11 0, 11 12 0, , det A 0;

a21 a22 I Stable algorithm: no pivoting necessary!
i.e. all leading minors non-negative, for p.s.d. I Economy of space and time.
Applied Mathematical Methods Special Systems and Special Methods 55, Applied Mathematical Methods Special Systems and Special Methods 56,
Sparse Systems* Quadratic Forms, Symmetry and Positive Definiteness

Sparse Systems*
Points to note Quadratic Forms, Symmetry and Positive Definiteness
Sparse Systems*
I What is a sparse matrix? I Concepts and criteria of positive definiteness and positive
I Bandedness and bandwidth semi-definiteness
I Efficient storage and processing I Cholesky decomposition method in symmetric positive definite
I Updates systems
I Sherman-Morrison formula I Nature of sparsity and its exploitation
1 T 1
(A u)(v A )
(A + uvT )1 = A1
1 + vT A1 u
I Woodbury formula Necessary Exercises: 1,2,4,7
I Conjugate gradient method
I efficiently implemented matrix-vector products
Applied Mathematical Methods Numerical Aspects in Linear Systems 57, Applied Mathematical Methods Numerical Aspects in Linear Systems 58,
Outline Norms and Condition Numbers

Ill-conditioning and Sensitivity
Rectangular Systems
Norms and Condition Numbers Norms and Condition Numbers
Rectangular Systems
Singularity-Robust Solutions
Iterative Methods
Norm of a vector: a measure of size Singularity-Robust Solutions
Iterative Methods
I Euclidean norm or 2-norm
1 q
kxk = kxk2 = x12 + x22 + + xn2 2 = xT x
Numerical Aspects in Linear Systems
Norms and Condition Numbers I The p-norm
Ill-conditioning and Sensitivity 1
Rectangular Systems kxkp = [|x1 |p + |x2 |p + + |xn |p ] p
Singularity-Robust Solutions
Iterative Methods I The 1-norm: kxk1 = |x1 | + |x2 | + + |xn |
I The -norm:
1
kxk = lim [|x1 |p + |x2 |p + + |xn |p ] p = max |xj |
p j
I Weighted norm q
kxkw = xT Wx
where weight matrix W is symmetric and positive definite.
Norms and Condition Numbers Norms and Condition Numbers

Rectangular Systems
Ill-conditioning and Sensitivity Norms and Condition Numbers
Rectangular Systems
Singularity-Robust Solutions Singularity-Robust Solutions
Iterative Methods Iterative Methods
Norm of a matrix: magnitude or scale of the transformation 0.9999x1 1.0001x2 = 1
x1 x2 = 1 +
Matrix norm (induced by a vector norm) is given by the largest 10001+1 99991
Solution: x1 = , x2 =
magnification it can produce on a vector 2 2
I sensitive to small changes in the RHS
kAxk I insensitive to error in a guess See illustration
kAk = max = max kAxk
x kxk kxk=1
For the system Ax = b, solution is x = A 1 b and
x = A1 b A1 A x
Direct consequence: kAxk kAk kxk
If the matrix A is exactly known, then
Index of closeness to singularity: Condition number kxk kbk kbk
kAk kA1 k = (A)
(A) = kAk kA1 k, 1 (A) kxk kbk kbk
If the RHS is known exactly, then
** Isotropic, well-conditioned, ill-conditioned and singular matrices kxk kAk kAk

kAk kA1 k = (A)
kxk kAk kAk
Ill-conditioning and Sensitivity Norms and Condition Numbers

Rectangular Systems
Rectangular Systems Norms and Condition Numbers
Rectangular Systems
X2
(2)
X2
Iterative Methods (2) (2b)
Consider Ax = b with A R mn and Rank(A) = n < m. Iterative Methods
(1) (1)
AT Ax = AT b x = (AT A)1 AT b
(b)
X
o X1 o X1
X
(a)
X
(a)
Square of error norm
1 1
(a) Reference system (b) Parallel shift U(x) = kAx bk2 = (Ax b)T (Ax b)
2 2
X2 X2 1 T T 1
(2) (2) (2d)
= x A Ax xT AT b + bT b
(1) (1)
2 2
(c)
Least square error solution:
o X X1
o X1
U
= AT Ax AT b = 0
x
(c) Guess validation (d) Singularity
Figure: Ill-conditioning: a geometric perspective

Pseudoinverse or Moore-Penrose inverse or left-inverse
A# = (AT A)1 AT
Rectangular Systems Norms and Condition Numbers

Rectangular Systems
Singularity-Robust Solutions Norms and Condition Numbers
Rectangular Systems
Consider Ax = b with A R mn and Rank(A) = m < n. Iterative Methods
Ill-posed problems: Tikhonov regularization Iterative Methods
Look for R m that satisfies AT = x and

I recipe for any linear system (m > n, m = n or m < n), with
AAT = b any condition!
Solution Ax = b may have conflict: form AT Ax = AT b.
x = AT = AT (AAT )1 b
AT A may be ill-conditioned: rig the system as
Consider the problem
minimize U(x) = 12 xT x subject to Ax = b. (AT A + 2 In )x = AT b
Extremum of the Lagrangian L(x, ) = 21 xT x T (Ax b) is Coefficient matrix: symmetric and positive definite!
given by The idea: Immunize the system, paying a small price.
L L Issues:
= 0, = 0 x = AT , Ax = b.
x I The choice of ?
Solution x = AT (AAT )1 b gives foot of the perpendicular on the I When m < n, computational advantage by
solution plane and the pseudoinverse
(AAT + 2 Im ) = b, x = AT
A# = AT (AAT )1
here is a right-inverse!
Iterative Methods Norms and Condition Numbers

Rectangular Systems
Points to note Norms and Condition Numbers
Rectangular Systems
Iterative Methods Iterative Methods
Jacobis iteration method:

X n
(k+1) 1 (k) I Solutions are unreliable when the coefficient matrix is
xi = bi aij xj for i = 1, 2, 3, , n. ill-conditioned.
aii
j=1, j6=i
I Finding pseudoinverse of a full-rank matrix is easy.
I Tikhonov regularization provides singularity-robust solutions.
Gauss-Seidel method: I Iterative methods may have an edge in certain situations!

i 1
X n
X
(k+1) 1 (k+1) (k)
xi = bi aij xj aij xj for i = 1, 2, 3, , n.
aii
j=1 j=i +1
Necessary Exercises: 1,2,3,4
The category of relaxation methods:

diagonal dominance and availability of good initial
approximations
Applied Mathematical Methods Eigenvalues and Eigenvectors 67, Applied Mathematical Methods Eigenvalues and Eigenvectors 68,
Outline Eigenvalue Problem

Generalized Eigenvalue Problem
Some Basic Theoretical Results
Eigenvalue Problem Eigenvalue Problem
Power Method Power Method
In mapping A : R n R n , special vectors of matrix A R nn
I mapped to scalar multiples, i.e. undergo pure scaling
Av = v
Eigenvalues and Eigenvectors
Eigenvalue Problem Eigenvector (v) and eigenvalue (): eigenpair (, v)
Generalized Eigenvalue Problem algebraic eigenvalue problem
Power Method (I A)v = 0
For non-trivial (non-zero) solution v,
det(I A) = 0
Characteristic equation: characteristic polynomial: n roots

I n eigenvalues for each, find eignevector(s)
Multiplicity of an eigenvalue: algebraic and geometric
Multiplicity mismatch: diagonalizable and defective matrices
Generalized Eigenvalue Problem Eigenvalue Problem

Some Basic Theoretical Results Eigenvalue Problem
1-dof mass-spring system: mx + kx = 0
q Eigenvalues of transpose
k
Natural frequency of vibration: n = m Eigenvalues of AT are the same as those of A.
Free vibration of n-dof system: Caution: Eigenvectors of A and AT need not be same.
Mx + Kx = 0, Diagonal and block diagonal matrices
Natural frequencies and corresponding modes? Eigenvalues of a diagonal matrix are its diagonal entries.
Assuming a vibration mode x = sin(t + ), Corresponding eigenvectors: natural basis members (e 1 , e2 etc).
( 2 M + K) sin(t + ) = 0 K = 2 M Eigenvalues of a block diagonal matrix: those of diagonal blocks.

Eigenvectors: coordinate extensions of individual eigenvectors.
Reduce as M1 K = 2 ? Why is it not a good idea? With (2 , v2 ) as eigenpair of block A2 ,

K symmetric, M symmetric and positive definite!! A1 0 0 0 0 0

A v2 = 0 A2 0 v2 = A2 v2 = 2 v2

With M = LLT , = LT and K = L1 KLT , 0 0 A3 0 0 0

K = 2

Triangular and block triangular matrices
Eigenvalues of a triangular matrix are its diagonal entries. Shift theorem
Eigenvalues of a block triangular matrix are the collection of Eigenvectors of A + I are the same as those of A.
eigenvalues of its diagonal blocks. Eigenvalues: shifted by .
Take
A B
H= , A R r r and C R ss Deflation
0 C
For a symmetric matrix A, with mutually orthogonal eigenvectors,
If Av = v, then having (j , vj ) as an eigenpair,

v A B v Av v v vj vjT
H = = = =
0 0 C 0 0 0 0 B = A j
vjT vj
If is an eigenvalue of C, then it is also an eigenvalue of and CT
has the same eigenstructure as A, except that the eigenvalue
T
0 A 0 0 0 corresponding to vj is zero.
CT w = w HT = =
w BT CT w w

Power Method Eigenvalue Problem
Consider matrix A with
Eigenspace
If v1 , v2 , , vk are eigenvectors of A corresponding to the same |1 | > |2 | |3 | |n1 | > |n |
eigenvalue , then
eigenspace: < v1 , v2 , , vk > and a full set of n eigenvectors v1 , v2 , , vn .
For vector x = 1 v1 + 2 v2 + + n vn ,
Similarity transformation p p p
2 3 n
B = S1 AS: same transformation expressed in new basis. Ap x = p1 1 v1 + 2 v2 + 3 v3 + + n vn
1 1 1
det(I A) = det S1 det(I A) det S = det(I B)
As p , Ap x p1 1 v1 , and
Same characteristic polynomial!
Eigenvalues are the property of a linear transformation, (Ap x)r
1 = lim , r = 1, 2, 3, , n.
not of the basis. p (Ap1 x)r
An eigenvector v of A transforms to S 1 v, as the corresponding At convergence, n ratios will be the same.

eigenvector of B.
Question: How to find the least magnitude eigenvalue?
Applied Mathematical Methods Eigenvalues and Eigenvectors 75, Applied Mathematical Methods Diagonalization and Similarity Transformations 76,
Points to note Eigenvalue Problem

Outline Diagonalizability
Canonical Forms
Symmetric Matrices
Power Method Similarity Transformations
I Meaning and context of the algebraic eigenvalue problem

I Fundamental deductions and vital relationships Diagonalization and Similarity Transformations
I Power method as an inexpensive procedure to determine Diagonalizability
extremal magnitude eigenvalues Canonical Forms
Symmetric Matrices
Similarity Transformations
Applied Mathematical Methods Diagonalization and Similarity Transformations 77, Applied Mathematical Methods Diagonalization and Similarity Transformations 78,
Diagonalizability Diagonalizability
Canonical Forms
Symmetric Matrices
Diagonalizability Diagonalizability
Canonical Forms
Symmetric Matrices
Similarity Transformations Similarity Transformations
Consider A R nn , having n eigenvectors v1 , v2 , , vn ;

with corresponding eigenvalues 1 , 2 , , n . Diagonalizability:
A matrix having a complete set of n linearly independent
eigenvectors is diagonalizable.
AS = A[v1 v2 vn ] = [1 v1 2 v2 n vn ]
Existence of a complete set of eigenvectors:
1 0 0
0 2 0 A diagonalizable matrix possesses a complete set of n

= [v1 v2 vn ] .. .. . . . = S linearly independent eigenvectors.
. . . ..
0 0 n I All distinct eigenvalues implies diagonalizability.
1 1
I But, diagonalizability does not imply distinct eigenvalues!
A = SS and S AS =
I However, a lack of diagonalizability certainly implies a
multiplicity mismatch.
Diagonalization: The process of changing the basis of a linear
transformation so that its new matrix representation is diagonal,
i.e. so that it is decoupled among its coordinates.
Canonical Forms Diagonalizability

Canonical Forms
Symmetric Matrices
Canonical Forms
Symmetric Matrices
Jordan canonical form (JCF): composed of Jordan blocks

1
J1 1
Jordan canonical form (JCF)
J2 .
J= .. , J r =
..

Diagonal (canonical) form . ..
. 1
Jk
Triangular (canonical) form
The key equation AS = SJ in extended form gives

..
.
Other convenient forms A[ Sr ] = [ Sr ] Jr ,

Tridiagonal form ..
.
Hessenberg form
where Jordan block Jr is associated with the subspace of
Sr = [v w2 w3 ]

Canonical Forms
Symmetric Matrices
Canonical Forms
Symmetric Matrices
Equating blocks as ASr = Sr Jr gives Similarity Transformations Similarity Transformations

1
1 Diagonal form

[Av Aw2 Aw3 ] = [v w2 w3 ]
.. Special case of Jordan form, with each Jordan block of 1 1
.

I
.. size
.
I Matrix is diagonalizable
Columnwise equality leads to I Similarity transformation matrix S is composed of n linearly
independent eigenvectors as columns
Av = v, Aw2 = v + w2 , Aw3 = w2 + w3 ,
I None of the eigenvectors admits any generalized eigenvector
Generalized eigenvectors w2 , w3 etc: I Equal geometric and algebraic multiplicities for every
eigenvalue
(A I)v = 0,
(A I)w2 = v and (A I)2 w2 = 0,
(A I)w3 = w2 and (A I)3 w3 = 0,

Canonical Forms
Symmetric Matrices
Canonical Forms
Symmetric Matrices
Triangular form Forms that can be obtained with pre-determined number of

Triangularization: Change of basis of a linear tranformation so as arithmetic operations (without iteration):
to get its matrix in the triangular form Tridiagonal form: non-zero entries only in the (leading) diagonal,
sub-diagonal and super-diagonal
I For real eigenvalues, always possible to accomplish with I useful for symmetric matrices
orthogonal similarity transformation Hessenberg form: A slight generalization of a triangular matrix
I Always possible to accomplish with unitary similarity
transformation, with complex arithmetic

I Determination of eigenvalues

Hu = .. .. . .
Note: The case of complex eigenvalues: 2 2 real diagonal block

. . .. ..
. . . . ..
. . .
+ i 0

0 i
Note: Tridiagonal and Hessenberg forms do not fall in the
category of canonical forms.
Symmetric Matrices Diagonalizability

Canonical Forms
Symmetric Matrices
Canonical Forms
Symmetric Matrices
A real symmetric matrix has all real eigenvalues and Proposition: Eigenvalues of a real symmetric matrix must be real.
is diagonalizable through an orthogonal similarity
transformation. Take A R nn such that A = AT , with eigenvalue = h + ik.
Since I A is singular, so is
Eigenvalues must be real.
B = (I A) (I A) = (hI A + ikI)(hI A ikI)
A complete set of eigenvectors exists.
Eigenvectors corresponding to distinct eigenvalues are = (hI A)2 + k 2 I
necessarily orthogonal.
Corresponding to repeated eigenvalues, orthogonal eigenvectors For some x 6= 0, Bx = 0, and
are available.
xT Bx = 0 xT (hI A)T (hI A)x + k 2 xT x = 0
In all cases of a symmetric matrix, we can form an
orthogonal matrix V, such that V T AV = is a real Thus, k(hI A)xk2 + kkxk2 = 0
diagonal matrix.
k = 0 and = h
Further, A = VVT .
Similar results for complex Hermitian matrices.

Canonical Forms
Symmetric Matrices
Canonical Forms
Symmetric Matrices
Proposition: A symmetric matrix possesses a complete set of Proposition: Eigenvectors of a symmetric matrix corresponding to
eigenvectors. distinct eigenvalues are necessarily orthogonal.
Consider a repeated real eigenvalue of A and examine its Jordan Take two eigenpairs (1 , v1 ) and (2 , v2 ), with 1 6= 2 .
block(s).
v1T Av2 = v1T (2 v2 ) = 2 v1T v2
Suppose Av = v.
v1T Av2 = v1T AT v2 = (Av1 )T v2 = (1 v1 )T v2 = 1 v1T v2
The first generalized eigenvector w satisfies (A I)w = v, giving
From the two expressions, (1 2 )v1T v2 = 0
vT (A I)w = vT v vT AT w vT w = vT v
v1T v2 = 0
(Av)T w vT w = kvk2
kvk2 = 0 Proposition: Corresponding to a repeated eigenvalue of a
symmetric matrix, an appropriate number of orthogonal
which is absurd.
eigenvectors can be selected.
An eigenvector will not admit a generalized eigenvector.
If 1 = 2 , then the entire subspace < v1 , v2 > is an eigenspace.
All Jordan blocks will be of 1 1 size. Select any two mutually orthogonal eigenvectors for the basis.

Canonical Forms
Symmetric Matrices
Similarity Transformations Diagonalizability
Canonical Forms
Symmetric Matrices
Facilities with the omnipresent symmetric matrices:
I Expression General Hessenberg
T
A = VV

Symmetric Tridiagonal Triangular
1 v1T
2 T
v2
= [v1 v2 vn ] .. .. Symmetric Tridiagonal
. .
n vnT Diagonal
X n
= 1 v1 v1T + 2 v2 v2T + + n vn vnT = i vi viT Figure: Eigenvalue problem: forms and steps
i =1
I Reconstruction from a sum of rank-one components How to find suitable similarity transformations?
I Efficient storage with only large eigenvalues and corresponding 1. rotation
eigenvectors 2. reflection
I Deflation technique
3. matrix decomposition or factorization
I Stable and effective methods: easier to solve the eigenvalue
problem 4. elementary transformation
Applied Mathematical Methods Diagonalization and Similarity Transformations 91, Applied Mathematical Methods Jacobi and Givens Rotation Methods 92,
Points to note Diagonalizability

Canonical Forms
Symmetric Matrices
Outline Plane Rotations
Jacobi Rotation Method
Givens Rotation Method
I Generally possible reduction: Jordan canonical form

I Condition of diagonalizability and the diagonal form
I Possible with orthogonal similarity transformations: triangular Jacobi and Givens Rotation Methods
form (for symmetric matrices)
I Useful non-canonical forms: tridiagonal and Hessenberg Plane Rotations
I Orthogonal diagonalization of symmetric matrices Givens Rotation Method
Caution: Each step in this context to be effected through
similarity transformations
Applied Mathematical Methods Jacobi and Givens Rotation Methods 93, Applied Mathematical Methods Jacobi and Givens Rotation Methods 94,
Plane Rotations Plane Rotations

Y Y/
Orthogonal change of basis:
P (x, y)
0
x cos sin x
r= = = <r0
y sin cos y0
y
y /
L x M
Mapping of position vectors with
X
O
x /
cos sin
K N
<1 = <T =
sin cos
X/
Figure: Rotation of axes and change of basis In three-dimensional (ambient) space,

cos sin 0 cos 0 sin

<xy = sin cos 0 , <xz = 0 1 0 etc.
x = OL + LM = OL + KN = x 0 cos + y 0 sin 0 0 1 sin 0 cos
y = PN MN = PN LK = y 0 cos x 0 sin

Jacobi Rotation Method Plane Rotations
Generalizing to n-dimensional Euclidean space (R n ),

1 0 0 0 0
apr = arp = carp sarq for p 6= r 6= q,
1 0 0 0 0
.. .. .. aqr = arq = carq + sarp for p 6= r 6= q,
. . .
0
app = c 2 app + s 2 aqq 2scapq ,
1 0 0

0 0 0 c 0 0 s 0 0
aqq = s 2 app + c 2 aqq + 2scapq , and

0 1 0 0 0
Ppq = apq = aqp = (c 2 s 2 )apq + sc(app aqq )
.. .. ..
. . .

0 1 0 In a Jacobi rotation,

0 0 0 s 0 0 c 0
c2 s2 aqq app
.. .. . . 0
apq =0 = = k (say).
. . . 2sc 2apq
0 0 1
Left side is cot 2: solve this equation for .
Matrix A is transformed as
Jacobi rotation transformations P 12 , P13 , , P1n ; P23 , , P2n ;
A0 = P1 T
pq APpq = Ppq APpq , ; Pn1,n complete a full sweep.
only the p-th and q-th rows and columns being affected. Note: The resulting matrix is far from diagonal!
Jacobi Rotation Method Plane Rotations

Givens Rotation Method Plane Rotations
Sum of squares of off-diagonal terms before the transformation 0 = 0: tan = rq a
While applying the rotation Ppq , demand arq arp
X X X r = p 1: Givens rotation
S = 2
|ars | = 2 2
arp + 2
arq
r 6=s r 6=p p6=r 6=q
I Once ap1,q is annihilated, it is never updated again!

X Sweep P23 , P24 , , P2n ; P34 , , P3n ; ; Pn1,n to
= 2 2
(arp 2
+ arq 2
) + apq annihilate a13 , a14 , , a1n ; a24 , , a2n ; ; an2,n .
p6=r 6=q
Symmetric tridiagonal matrix
and that afterwards

X
S0 = 2 02
(arp 02
+ arq 02
) + apq How do eigenvectors transform through Jacobi/Givens rotation
p6=r 6=q steps?
X T T
= 2 2
(arp 2
+ arq ) A = P(2) P(1) AP(1) P(2)
p6=r 6=q
Product matrix P(1) P(2) gives the basis.
differ by
To record it, initialize V by identity and keep multiplying new
S = S 0 S = 2apq
2
0; and S 0. rotation matrices on the right side.
Givens Rotation Method Plane Rotations

Points to note Plane Rotations
Contrast between Jacobi and Givens rotation methods Rotation transformation on symmetric matrices
I What happens to intermediate zeros? I Plane rotations provide orthogonal change of basis that can
I What do we get after a complete sweep? be used for diagonalization of matrices.
I How many sweeps are to be applied? I For small matrices (say 4 n 8), Jacobi rotation sweeps
I What is the intended final form of the matrix? are competitive enough for diagonalization upto a reasonable
tolerance.
I How is size of the matrix relevant in the choice of the method?
I For large matrices, one sweep of Givens rotations can be
applied to get a symmetric tridiagonal matrix, for efficient
Fast forward ... further processing.
I Housholder method accomplishes tridiagonalization more
efficiently than Givens rotation method.
I But, with a half-processed matrix, there come situations in
which Givens rotation method turns out to be more efficient!
Applied Mathematical Methods Householder Transformation and Tridiagonal Matrices 101, Applied Mathematical Methods Householder Transformation and Tridiagonal Matrices 102,
Outline Householder Reflection Transformation

Householder Method
Eigenvalues of Symmetric Tridiagonal Matrices
Householder Reflection Transformation
Householder Method
u uv
w
Plane of
O Reflection
Householder Transformation and Tridiagonal Matrices Figure: Vectors in Householder reflection

Householder Method uv
Eigenvalues of Symmetric Tridiagonal Matrices Consider u, v R k , kuk = kvk and w = kuvk .
Householder reflection matrix
Hk = Ik 2wwT
is symmetric and orthogonal.

For any vector x orthogonal to w,
Hk x = (Ik 2wwT )x = x and Hk w = (Ik 2wwT )w = w.
Hence, Hk y = Hk (yw + y ) = yw + y , Hk u = v and Hk v = u.

Householder Method Householder Reflection Transformation

Householder Method
Householder Method Householder Reflection Transformation
Householder Method
Next, with v2 = ku2 ke1 , we form
Consider n n symmetric matrix A.

Let u = [a21 a31 an1 ]T R n1 and v = kuke1 R n1 . I2 0
P2 =
1 0 0 Hn2
Construct P1 = and operate as
0 Hn1
and operate as A(2) = P2 A(1) P2 .
After j steps,
1 0 a11 uT 1 0
A(1) = P1 AP1 =
0 Hn1 u A1 0 Hn1 d1 e2

=
a11 vT
. e2 d2 . . .
v Hn1 A1 Hn1
A(j) =

..
.
..
. ej+1

ej+1 dj+1 uT j+1

Reorganizing and re-naming,
uj+1 Aj+1

d1 e2 0 By n 2 steps, with P = P1 P2 P3 Pn2 ,
A(1) = e2 d2 uT
2
.
0 u 2 A2 A(n2) = PT AP
is symmetric tridiagonal.
Eigenvalues of Symmetric TridiagonalHouseholder

Matrices
Householder
Reflection Transformation
Method
Matrices
Householder
Method
Characteristic polynomial of the leading k k sub-matrix: p k ()

d1 e2
..
e2 d2 .

T=
..
.
..
. en1

p0 () = 1,
p1 () = d1 ,
en1 dn1 en
en dn p2 () = ( d2 )( d1 ) e22 ,
,
Characteristic polynomial 2
pk+1 () = ( dk+1 )pk () ek+1 pk1 ().

d1 e2

..
e2 d2 . P() = {p0 (), p1 (), , pn ()}

p() = . .. . .. . I a Sturmian sequence if ej 6= 0 j
en1

en1 dn1 en
Question: What if ej = 0 for some j?!
en dn
Answer: That is good news. Split the matrix.

Matrices
Householder
Method
Matrices
Householder
Method
Sturmian sequence property of P() with e j 6= 0:

Next, we assume that the statement is true for k = i.
Interlacing property: Roots of pk+1 () interlace the Roots of pi (): 1 > 2 > > i
roots of pk (). That is, if the roots of pk+1 () are Roots of pi +1 (): 1 > 2 > > i > i +1
1 > 2 > > k+1 and those of pk () are Roots of pi +2 (): 1 > 2 > > i > i +1 > i +2
1 > 2 > > k ; then
Assumption: 1 > 1 > 2 > 2 > > i > i > i +1
1 > 1 > 2 > 2 > > k > k > k+1 . i+2 i i1 2 1 1

8
i+1 i 2 1
This property leads to a convenient procedure . (a) Roots of p ( ) and p

i i+1
()
Proof
j+1 j j1
p1 () has a single root, d1 . j
j+1
p2 (d1 ) = e22 < 0, ve ve
(b) Sign of pi pi+2
Since p2 () = > 0, roots t1 and t2 of p2 () are separated as

> t1 > d1 > t2 > . Figure: Interlacing of roots of characteristic polynomials
The statement is true for k = 1.

To show: 1 > 1 > 2 > 2 > > i +1 > i +1 > i +2

Matrices
Householder
Method
Matrices
Householder
Method
Since 1 > 1 , pi (1 ) is of the same sign as pi (), i.e. positive. Examine sequence P(w ) = {p0 (w ), p1 (w ), p2 (w ), , pn (w )}.
Therefore, pi +2 (1 ) = ei2+2 pi (1 ) is negative. If pk (w ) and pk+1 (w ) have opposite signs then pk+1 () has one
But, pi +2 () is clearly positive. root more than pk () in the interval (w , ).
Hence, 1 (1 , ). Number of roots of pn () above w = number of sign
Similarly, i +2 (, i +1 ). changes in the sequence P(w ).
Question: Where are the rest of the i roots of p i +2 ()?
Consequence: Number of roots of pn () in (a, b) = difference
pi +2 (j ) = (j di +2 )pi +1 (j ) ei2+2 pi (j ) = ei2+2 pi (j ) between numbers of sign changes in P(a) and P(b).
a+b
pi +2 (j+1 ) = ei2+2 pi (j+1 ) Bisection method: Examine the sequence at 2 .
Separate roots, bracket each of them and then squeeze
That is, pi and pi +2 are of opposite signs at each . the interval!
Refer figure.
Over [i +1 , 1 ], pi +2 () changes sign over each sub-interval Any way to start with an interval to include all eigenvalues?
[j+1 , j ], along with pi (), to maintain opposite signs at each .
Conclusion: pi +2 () has exactly one root in (j+1 , j ). |i | bnd = max {|ej | + |dj | + |ej+1 |}
1jn

Matrices
Householder
Method
Points to note Householder Reflection Transformation
Householder Method
I A Householder matrix is symmetric and orthogonal. It effects

Algorithm
a reflection transformation.
I Identify the interval [a, b] of interest. I A sequence of Householder transformations can be used to
I For a degenerate case (some ej = 0), split the given matrix. convert a symmetric matrix into a symmetric tridiagonal form.
I For each of the non-degenerate matrices, I Eigenvalues of the leading square sub-matrices of a symmetric
I by repeated use of bisection and study of the sequence P(), tridiagonal matrix exhibit a useful interlacing structure.
bracket individual eigenvalues within small sub-intervals, and
I by further use of the bisection method (or a substitute) within I This property can be used to separate and bracket eigenvalues.
each such sub-interval, determine the individual eigenvalues to I Method of bisection is useful in the separation as well as
the desired accuracy. subsequent determination of the eigenvalues.
Note: The algorithm is based on Sturmian sequence property . Necessary Exercises: 2,4,5
Applied Mathematical Methods QR Decomposition Method 113, Applied Mathematical Methods QR Decomposition Method 114,
Outline QR Decomposition
QR Iterations
Conceptual Basis of QR Method*
QR Decomposition QR Decomposition
QR Iterations
QR Algorithm with Shift* QR Algorithm with Shift*
Decomposition (or factorization) A = QR into two factors,
orthogonal Q and upper-triangular R:
(a) It always exists.
(b) Performing this decomposition is pretty straightforward.
QR Decomposition Method (c) It has a number of properties useful in the solution of the
QR Decomposition eigenvalue problem.
QR Iterations r11 r1n
Conceptual Basis of QR Method* .. ..
[a1 an ] = [q1 qn ] . .
QR Algorithm with Shift* rnn
A simple method based on Gram-Schmidt P orthogonalization:
Considering columnwise equality a j = ji =1 rij qi ,
for j = 1, 2, 3, , n;
j1
X
rij = qT
i aj i < j, a0j = aj rij qi , rjj = ka0j k;
i =1

a0j /rjj , if rjj 6= 0;
qj =
any vector satisfying qTi qj = ij for 1 i j, if rjj = 0.
QR Iterations
QR Iterations
Practical method: one-sided Householder transformations,
starting with
Alternative method useful for tridiagonal and Hessenberg
u0 v 0
u0 = a1 , v0 = ku0 ke1 R n and w0 = matrices: One-sided plane rotations
ku0 v0 k
I rotations P12 , P23 etc to annihilate a21 , a32 etc in that
and P0 = Hn = In 2w0 w0T . sequence
Givens rotation matrices!

ka1 k
Pn2 Pn3 P2 P1 P0 A = Pn2 Pn3 P2 P1
0 A0
Application in solution of a linear system: Q and R factors of
r11
a matrix A come handy in the solution of Ax = b
= Pn2 Pn3 P2 r22 = = R
A1 QRx = b Rx = QT b
With
needs only a sequence of back-substitutions.
Q = (Pn2 Pn3 P2 P1 P0 )T = P0 P1 P2 Pn3 Pn2 ,
we have QT A = R A = QR.
QR Iterations QR Decomposition
QR Iterations
QR Iterations QR Decomposition
QR Iterations
Multiplying Q and R factors in reverse, Quasi-upper-triangular form:

1 ??
0 T
A = RQ = Q AQ, 2 ??

..
. ??
an orthogonal similarity transformation.
r ??

1. If A is symmetric, then so is A0 . ,
2. If A is in upper Hessenberg form, then so is A 0 .

Bk

.. .. ..
. .
3. If A is symmetric tridiagonal, then so is A0. .

Complexity of QR iteration: O(n) for a symmetric tridiagonal
matrix, O(n 2 ) operation for an upper Hessenberg matrix and
O(n3 ) for the general case. with |1 | > |2 | > .
I Diagonal blocks Bk correspond to eigenspaces of equal/close
Algorithm: Set A1 = A and for k = 1, 2, 3, , (magnitude) eigenvalues.
I decompose Ak = Qk Rk , I 2 2 diagonal blocks often correspond to pairs of complex
I reassemble Ak+1 = Rk Qk . eigenvalues (for non-symmetric matrices).
I For symmetric matrices, the quasi-upper-triangular form
As k , Ak approaches the quasi-upper-triangular form.
reduces to quasi-diagonal form.
Conceptual Basis of QR Method* QR Iterations
QR Algorithm with Shift* QR Iterations
QR Algorithm with Shift* k
QR Algorithm with Shift*
i
QR decomposition algorithm operates on the basis of the relative For i < j , entry aij decays through iterations as j .
magnitudes of eigenvalues and segregates subspaces. With shift,
With k , Ak = Ak k I;
Ak = Qk Rk , Ak+1 = Rk Qk ;
Ak Range{e1 } = Range{q1 } Range{v1 }
Ak+1 = Ak+1 + k I.
and (a1 )k QT T
k Aq1 = 1 Qk q1 = 1 e1 .
Resulting transformation is
Further,
Ak+1 = Rk Qk + k I = QT
k Ak Qk + k I
Ak Range{e1 , e2 } = Range{q1 , q2 } Range{v1 , v2 }. = QT T
k (Ak k I)Qk + k I = Qk Ak Qk .

(1 2 )1 For the iteration,
T
and (a2 )k Qk Aq2 = 2 .
i k
0 convergence ratio = j k .
And, so on ...
Question: How to find a suitable value for k ?
Applied Mathematical Methods QR Decomposition Method 121, Applied Mathematical Methods Eigenvalue Problem of General Matrices 122,
Points to note QR Decomposition

QR Iterations
Outline Introductory Remarks
Reduction to Hessenberg Form*
QR Algorithm on Hessenberg Matrices*
QR Algorithm with Shift* Inverse Iteration
Recommendation
I QR decomposition can be effected on any square matrix.
I Practical methods of QR decomposition use Householder
transformations or Givens rotations.
I A QR iteration effects a similarity transformation on a matrix, Eigenvalue Problem of General Matrices
preserving symmetry, Hessenberg structure and also a Introductory Remarks
symmetric tridiagonal form. Reduction to Hessenberg Form*
I A sequence of QR iterations converge to an almost
Inverse Iteration
upper-triangular form.
Recommendation
I Operations on symmetric tridiagonal and Hessenberg forms
are computationally efficient.
I QR iterations tend to order subspaces according to the
relative magnitudes of eigenvalues.
I Eigenvalue shifting is useful as an expediting strategy.
Necessary Exercises: 1,3
Applied Mathematical Methods Eigenvalue Problem of General Matrices 123, Applied Mathematical Methods Eigenvalue Problem of General Matrices 124,
Introductory Remarks Introductory Remarks

Reduction to Hessenberg Form* Introductory Remarks
Inverse Iteration Inverse Iteration
Recommendation Recommendation
I A general (non-symmetric) matrix may not be diagonalizable. Methods to find appropriate similarity transformations
We attempt to triangularize it. 1. a full sweep of Givens rotations,
I With real arithmetic, 2 2 diagonal blocks are inevitable 2. a sequence of n 2 steps of Householder transformations, and
signifying complex pair of eigenvalues. 3. a cycle of coordinated Gaussian elimination.
I Higher computational complexity, slow convergence and lack
of numerical stability. Method based on Gaussian elimination or elementary
transformations:
A non-symmetric matrix is usually unbalanced and is prone to The pre-multiplying matrix corresponding to the
higher round-off errors. elementary row transformation and the post-multiplying
matrix corresponding to the matching column
Balancing as a pre-processing step: multiplication of a row and transformation must be inverses of each other.
division of the corresponding column with the same number,
ensuring similarity. Two kinds of steps
Note: A balanced matrix may get unbalanced again through I Pivoting
similarity transformations that are not orthogonal! I Elimination
Introductory Remarks Introductory Remarks
Reduction to Hessenberg Form* Reduction to Hessenberg Form*
Recommendation Recommendation
Pivoting step: A = Prs APrs = P1
rs APrs . QR iterations: O(n 2 ) operations for upper Hessenberg form.
I Permutation Prs : interchange of r -th and s-th columns. Whenever a sub-diagonal zero appears, the matrix is split
I P1
rs = Prs : interchange of r -th and s-th rows. into two smaller upper Hessenberg blocks, and they are
I Pivot locations: a21 , a32 , , an1,n2 . processed separately, thereby reducing the cost drastically.
Elimination step: A = G1
r AGr with elimination matrix
Particular cases:

Ir 0 0 Ir 0 0 I an,n1 0: Accept ann = n as an eigenvalue, continue with
Gr = 0 1 0 1
and Gr = 0 1 0 . the leading (n 1) (n 1) sub-matrix.
0 k Inr 1 0 k Inr 1
0: Separately find the eigenvalues n1 and n
an1,n2
I
an1,n1 an1,n
from , continue with the leading
I G1
r : Row (r + 1 + i) Row (r + 1 + i) k i Row (r + 1) an,n1 an,n
for i = 1, 2, 3, , n r 1 (n 2) (n 2) sub-matrix.
I 1) Column (r + 1)+
Gr : Column (r + P
nr 1 Shift strategy: Double QR steps.
i =1 [ki Column (r + 1 + i) ]
Inverse Iteration Introductory Remarks

Inverse Iteration Pn Pn Inverse Iteration
Recommendation
Assumption: Matrix A has a complete set of eigenvectors. With y0 = j=1 j vj and y = j=1 j vj , [A (i )0 I]y = y0 gives
Recommendation
n
X n
X
(i )0 : a good estimate of an eigenvalue i of A.
j [A (i )0 I]vj = j vj
Purpose: To find i precisely and also to find vi . j=1 j=1
j
Step: Select a random vector y0 (with ky0 k = 1) and solve j [j (i )0 ] = j j = .
j (i )0
[A (i )0 I]y = y0 . i is typically large and eigenvector v i dominates y.
Avi = i vi gives [A (i )0 I]vi = [i (i )0 ]vi . Hence,
Result: y is a good estimate of vi and
[i (i )0 ]y [A (i )0 I]y = y0 .
1
(i )1 = (i )0 +
y0T y Inner product with y0 gives
is an improvement in the estimate of the eigenvalue. 1

[i (i )0 ]y0T y 1 i (i )0 + .
y0T y
How to establish the result and work out an algorithm ?

Recommendation Introductory Remarks
Algorithm: Recommendation Recommendation
Start with estimate (i )0 , guess y0 (normalized). Table: Eigenvalue problem: summary of methods
For k = 0, 1, 2, Type Size Reduction Algorithm Post-processing
General Small Definition: Polynomial Solution of
I Solve [A (i )k I]y = yk . (up to 4) Characteristic root finding linear systems
polynomial (eigenvalues) (eigenvectors)
y
I Normalize yk+1 = kyk .
Symmetric Intermediate Jacobi sweeps Selective
(say, 412) Jacobi rotations
1 Tridiagonalization Sturm sequence Inverse iteration
I Improve (i )k+1 = (i )k + yT y
. (Givens rotation property: (eigenvalue
k or Householder Bracketing and improvement
method) bisection and eigenvectors)
I If kyk+1 yk k < , terminate. (rough eigenvalues)
Large Tridiagonalization QR decomposition
(usually iterations
Important issues Householder method)
Balancing, and then
I Update eigenvalue once in a while, not at every iteration. Non-
symmetric
Intermediate
Large
Reduction to
Hessenberg form
QR decomposition
iterations
Inverse iteration
(eigenvectors)
(Above methods or (eigenvalues)
I Use some acceptable small number as artificial pivot. Gaussian elimination)
General Very large Power method,
I The method may not converge for defective matrix or for one (selective shift and deflation
requirement)
having complex eigenvalues.
I Repeated eigenvalues may inhibit the process.
Applied Mathematical Methods Eigenvalue Problem of General Matrices 131, Applied Mathematical Methods Singular Value Decomposition 132,
Points to note Introductory Remarks

Outline SVD Theorem and Construction
Properties of SVD
Pseudoinverse and Solution of Linear Systems
Inverse Iteration Optimality of Pseudoinverse Solution
Recommendation SVD Algorithm
I Eigenvalue problem of a non-symmetric matrix is difficult!

I Balancing and reduction to Hessenberg form are desirable
pre-processing steps. Singular Value Decomposition
I QR decomposition algorithm is typically used for reduction to SVD Theorem and Construction
an upper-triangular form. Properties of SVD
I Use inverse iteration to polish eigenvalue and find Pseudoinverse and Solution of Linear Systems
eigenvectors. Optimality of Pseudoinverse Solution
SVD Algorithm
I In algebraic eigenvalue problems, different methods or
combinations are suitable for different cases; regarding matrix
size, symmetry and the requirements.

Applied Mathematical Methods Singular Value Decomposition 133, Applied Mathematical Methods Singular Value Decomposition 134,
SVD Theorem and Construction SVD Theorem and Construction

Properties of SVD
Properties of SVD
Eigenvalue problem: A = UV 1 where U = V Optimality of Pseudoinverse Solution
SVD Algorithm
Question: How to construct U, V and ? Optimality of Pseudoinverse Solution
SVD Algorithm
Do not ask for similarity. Focus on the form of the decomposition. For A R mn ,
Guaranteed decomposition with orthogonal U, V, and
non-negative diagonal entries in . AT A = (VT UT )(UVT ) = VT VT = VVT ,
where = T is an n n diagonal matrix.

A = UVT such that UT AV =

1 |
SVD Theorem For any real matrix A R mn , there 2 |
exist orthogonal matrices U R mm and V R nn such
..
that
=
. | 0

UT AV = R mn p |

+
is a diagonal matrix, with diagonal entries 1 , 2 , 0,
obtained by appending the square diagonal matrix
0 |
diag (1 , 2 , , p ) with (m p) zero rows or (n p) Determine V and . Work out and we have
zero columns, where p = min(m, n).
A = UVT AV = U
Singular values: 1 , 2 , , p .
Similar result for complex matrices This provides a proof as well!

Properties of SVD
Properties of SVD SVD Theorem and Construction
Properties of SVD
Optimality of Pseudoinverse Solution Optimality of Pseudoinverse Solution
SVD Algorithm For a given matrix, the SVD is unique up to SVD Algorithm
From AV = U, determine columns of U.
(a) the same permutations of columns of U, columns of V and
1. Column Avk = k uk , with k 6= 0: determine column uk .
diagonal elements of ;
Columns developed are bound to be mutually
(b) the same orthonormal linear combinations among columns of
orthonormal!
T U and columns of V, corresponding to equal singular values;
Verify ui uj = 1i Avi
T 1
j Avj = ij . and
2. Column Avk = k uk , with k = 0: uk is left indeterminate (c) arbitrary orthonormal linear combinations among columns of
(free). U or columns of V, corresponding to zero or non-existent
3. In the case of m < n, identically zero columns Av k = 0 for singular values.
k > m: no corresponding columns of U to determine. Ordering of the singular values:
4. In the case of m > n, there will be (m n) columns of U left
indeterminate. 1 2 r > 0, and r +1 = r +2 = = p = 0.
Extend columns of U to an orthonormal basis. Rank(A) = Rank() = r
All three factors in the decomposition are constructed, as desired. Rank of a matrix is the same as the number of its
non-zero singular values.

Properties of SVD
Properties of SVD
SVD Algorithm In basis V, v = c1 v1 + c2 v2 + + cn vn = Vc and the norm is
SVD Algorithm

1 y1 given by
.. kAvk2 vT AT Av

Ax = UVT x = Uy = [u1 ur ur +1 um ] . kAk2 = max = max
r yr kvk2v v vT v
P 2 2
0 = max
T T T
c V A AVc
= max
cT T c
= max Pk k ck
.
cT VT Vc cT c 2
= 1 y1 u1 + 2 y2 u2 + + r yr ur k ck
c c c
has non-zero components along only the first r columns of U. r P 2 2

k k c k
U gives an orthonormal basis for the co-domain such that kAk = maxc P 2 = max
k ck
Range(A) = < u1 , u2 , , ur > . For a non-singular square matrix,

1 1 1
With VT x = y, vkT x = yk , and A1 = (UVT )1 = V1 UT = V diag , , , UT .
1 2 n
x = y1 v1 + y2 v2 + + yr vr + yr +1 vr +1 + yn vn .
Then, kA1 k = 1
min and the condition number is
V gives an orthonormal basis for the domain such that max
(A) = kAk kA1 k = .
min
Null(A) = < vr +1 , vr +2 , , vn > .

Properties of SVD
Pseudoinverse and Solution of Linear SVD
Systems
Theorem and Construction
Properties of SVD
SVD Algorithm Generalized inverse: G is called a generalized inverse or g-inverse
SVD Algorithm
Revision of definition of norm and condition number: of A if, for b Range(A), Gb is a solution of Ax = b.
The norm of a matrix is the same as its largest singular
The Moore-Penrose inverse or the pseudoinverse:
value, while its condition number is given by the ratio of
the largest singular value to the least. A# = (UVT )# = (VT )# # U# = V# UT
1
Arranging singular values in decreasing order, with Rank(A) = r , r 0 r 0
With = , # = .
0 0 0 0
U = [Ur U] and V = [Vr V],
1 |
T 2 |
r 0 Vr
A = UVT = [Ur U] , ..
0 0 VT Or, # =
. | 0
,
p |
or,
r
X +
A = Ur r VrT = k uk vkT . 0 |
k=1
1
Efficient storage and reconstruction! k , for k 6= 0 or for |k | > ;
where k =
0, for k = 0 or for |k | .
Pseudoinverse and Solution of Linear Properties

Systems
SVD Theorem and Construction
of SVD
Optimality of Pseudoinverse Solution SVD Theorem and Construction
Properties of SVD
Inverse-like facets and beyond SVD Algorithm Pseudoinverse solution of Ax = b: SVD Algorithm
r r
I (A# )# = A. X X
x = V# UT b = k vk uT
k b = (uT
k b/k )vk
I If A is invertible, then A# = A1 .
k=1 k=1
I A# b gives the correct unique solution.
I If Ax = b is an under-determined consistent system, then Minimize
A# b selects the solution x with the minimum norm. 1 1 1
E (x) = (Ax b)T (Ax b) = xT AT Ax xT AT b + bT b
I If the system is inconsistent, then A # b minimizes the least 2 2 2
square error kAx bk.
Condition of vanishing gradient:
I If the minimizer of kAx bk is not unique, then it picks up
that minimizer which has the minimum norm kxk among such E
minimizers. = 0 AT Ax = AT b
x
Contrast with Tikhonov regularization: V(T )VT x = VT UT b
Pseudoinverse solution for precision and diagnosis. (T )VT x = T UT b
Tikhonovs solution for continuity of solution over k2 vkT x = k uT
k b
variable A and computational efficiency. vkT x = uT for k = 1, 2, 3, , r .
k b/k
Optimality of Pseudoinverse Solution Properties

SVD Theorem and Construction
of SVD
Optimality of Pseudoinverse Solution SVD Theorem and Construction
Properties of SVD
SVD Algorithm Anatomy of the optimization through SVD SVD Algorithm
With V = [vr +1 vr +2 vn ], then Using basis V for domain and U for co-domain, the variables are
transformed as
r
X
VT x = y and UT b = c.
x= (uT
k b/k )vk + Vy = x + Vy.
k=1 Then,
Ax = b UVT x = b VT x = UT b y = c.
How to minimize kxk2 subject to E (x) minimum?
A completely decoupled system!
Minimize E1 (y) = kx + Vyk2 . Usable components: yk = ck /k for k = 1, 2, 3, , r .
For k > r ,
I completely redundant information (c k = 0)
Since x and Vy are mutually orthogonal,
I purely unresolvable conflict (c k 6= 0)
E1 (y) = kx + Vyk2 = kx k2 + kVyk2
SVD extracts this pure redundancy/inconsistency.
is minimum when Vy = 0, i.e. y = 0. Setting k = 0 for k > r rejects it wholesale!
At the same time, kyk is minimized, and hence kxk too.
Applied Mathematical Methods Singular Value Decomposition 145, Applied Mathematical Methods Vector Spaces: Fundamental Concepts* 146,
Points to note SVD Theorem and Construction

Properties of SVD
Outline Group
Field
Vector Space
Optimality of Pseudoinverse Solution Linear Transformation
SVD Algorithm Isomorphism
Inner Product Space
Function Space
I SVD provides a complete orthogonal decomposition of the
domain and co-domain of a linear transformation, separating
Vector Spaces: Fundamental Concepts*
out functionally distinct subspaces.
Group
I If offers a complete diagnosis of the pathologies of systems of Field
linear equations. Vector Space
I Pseudoinverse solution of linear systems satisfy meaningful Linear Transformation
optimality requirements in several contexts. Isomorphism
I With the existence of SVD guaranteed, many important Inner Product Space
results can be established in a straightforward manner. Function Space
Applied Mathematical Methods Vector Spaces: Fundamental Concepts* 147, Applied Mathematical Methods Vector Spaces: Fundamental Concepts* 148,
Group Group
Field
Vector Space
Field Group
Field
Vector Space
Linear Transformation Linear Transformation
Isomorphism Isomorphism
Inner Product Space Inner Product Space
A set G and a binary operation, say +, fulfilling
Function Space A set F and two binary operations, say + and , satisfying
Function Space
Closure: a + b G a, b G Group property for addition: (F , +) is a commutative group.

Associativity: a + (b + c) = (a + b) + c, a, b, c G (Denote the identity element of this group as 0.)
Existence of identity: 0 G such that a G , a + 0 = a = 0 + a Group property for multiplication: (F {0}, ) is a commutative
Existence of inverse: a G , (a) G such that group. (Denote the identity element of this group as
a + (a) = 0 = (a) + a 1.)
Distributivity: a (b + c) = a b + a c, a, b, c F .
Examples: (Z , +), (Z , +), (Q {0}, ), 2 5 real matrices,
Rotations etc. Concept of field: abstraction of a number system
I Commutative group Examples: (Q, +, ), (R, +, ), (C , +, ) etc.
I Subgroup I Subfield
Vector Space Group

Field
Vector Space
Vector Space Group
Field
Vector Space
A vector space is defined by Inner Product Space Suppose V is a vector space. Inner Product Space
Function Space Function Space
I a field F of scalars, Take a vector 1 6= 0 in it.
I a commutative group V of vectors, and Then, vectors linearly dependent on 1 :
1 1 V 1 F .
I a binary operation between F and V, that may be called
scalar multiplication, such that , F , a, b V; the
Question: Are the elements of V exhausted?
following conditions hold.
Closure: a V. If not, then take 2 V: linearly independent from 1 .
Identity: 1a = a. Then, 1 1 + 2 2 V 1 , 2 F .
Associativity: ()a = (a).
Scalar distributivity: (a + b) = a + b. Question: Are the elements of V exhausted now?
Vector distributivity: ( + )a = a + a.
Examples: R n , C n , m n real matrices etc. Question: Will this process ever end?
Field Number system Suppose it does.

Vector space Space finite dimensional vector space
Vector Space Group

Field
Vector Space
Linear Transformation Group
Field
Vector Space
Finite dimensional vector space Inner Product Space A mapping T : V W satisfying Inner Product Space
Suppose the above process ends after n choices of linearly T(a + b) = T(a) + T(b) , F and a, b V
independent vectors.
where V and W are vector spaces over the field F .
= 1 1 + 2 2 + + n n
Question: How to describe the linear transformation T?
Then, I For V, basis 1 , 2 , , n

I n: dimension of the vector space I For W, basis 1 , 2 , , m
I ordered set 1 , 2 , , n : a basis 1 V gets mapped to T(1 ) W.
I 1 , 2 , , n F : coordinates of in that basis
T(1 ) = a11 1 + a21 2 + + am1 m
R n , R m etc: vector spaces over the field of real numbers P
Similarly, enumerate T(j ) = m i =1 aij i .
I Subspace
Matrix A = [a1 a2 an ] codes this description!

Field
Vector Space
Field
Vector Space
A general element of V can be expressed as Inner Product Space Understanding: Inner Product Space
= x 1 1 + x 2 2 + + x n n
I Vector is an actual object in the set V and the column
x R n is merely a list of its coordinates.
Coordinates in a column: x = [x1 x2 xn ]T I T : V W is the linear transformation and the matrix A
simply stores coefficients needed to describe it.
Mapping:
I By changing bases of V and W, the same vector and the
T() = x1 T(1 ) + x2 T(2 ) + + xn T(n ), same linear transformation are now expressed by different x
and A, respectively.
with coordinates Ax, as we know!
Summary: Matrix representation emerges as the natural description

of a linear transformation between two vector spaces.
I basis vectors of V get mapped to vectors in W whose
coordinates are listed in columns of A, and
I a vector of V, having its coordinates in x, gets mapped to a Exercise: Set of all T : V W form a vector space of their own!!
vector in W whose coordinates are obtained from Ax. Analyze and describe that vector space.
Isomorphism Group
Field
Vector Space
Isomorphism Group
Field
Vector Space
Consider T : V W that establishes a one-to-one correspondence.
Inner Product Space
Function Space
Consider vector spaces V and W over the same field F and of the
Inner Product Space
Function Space
I Linear transformation T defines a one-one onto mapping, same dimension n.
which is invertible.
I dim V = dim W Question: Can we define an isomorphism between them?
I Inverse linear transformation T1 : W V Answer: Of course. As many as we want!
I T defines (is) an isomorphism.
I Vector spaces V and W are isomorphic to each other. The underlying field and the dimension together
completely specify a vector space, up to an isomorphism.
I Isomorphism is an equivalence relation. V and W are
equivalent!
If we need to perform some operations on vectors in one vector I All n-dimensional vector spaces over the field F are
space, we may as well isomorphic to one another.
1. transform the vectors to another vector space through an I In particular, they are all isomorphic to F n .
isomorphism, I The representation (columns) can be considered as the
2. conduct the required operations there, and objects (vectors) themselves.
3. map the results back to the original space through the inverse.
Inner Product Space Group

Field
Vector Space
Inner Product Space Group
Field
Vector Space
Inner product (a, b) in a real or complex vector space: a scalar
Inner Product Space
Function Space
Inner Product Space
Inner products bring in ideas of angle and length in the geometry
Function Space
function p : V V F satisfying of vector spaces.
Closure: a, b V, (a, b) F
Associativity: (a, b) = (a, b) Orthogonality: (a, b) = 0
Distributivity: (a + b, c) = (a, c) + (b, c) p
Norm: k k : V R, such that kak = (a, a)
Conjugate commutativity: (b, a) = (a, b) Associativity: kak = || kak
Positive definiteness: (a, a) 0; and (a, a) = 0 iff a = 0 Positive definiteness: kak > 0 for a 6= 0 and k0k = 0
Note: Property of conjugate commutativity forces (a, a) to be real. Triangle inequality: ka + bk kak + kbk
Cauchy-Schwarz inequality: (a, b) kak kbk
Examples: aT b, aT Wb in R, a b in C etc.
Inner product space: a vector space possessing an inner product A distance function or metric: dV : V V R such that
I Euclidean space: over R dV (a, b) = ka bk
I Unitary space: over C
Function Space Group

Field
Vector Space
Field
Vector Space
Suppose we decide to represent a continuous function Inner Product Space
Function Space
Vector space of continuous functions Inner Product Space
Function Space
f : [a, b] R by the listing

First, (F, +) is a commutative group.
vf = [f (x1 ) f (x2 ) f (x3 ) f (xN )]T
Next, with , R, x [a, b],
with a = x1 < x2 < x3 < < xN = b. I if f (x) R, then f (x) R
I 1 f (x) = f (x)
Note: The true representation will require N to be infinite! I ()f (x) = [f (x)]
Here, vf is a real column vector. I [f1 (x) + f2 (x)] = f1 (x) + f2 (x)
Do such vectors form a vector space? I ( + )f (x) = f (x) + f (x)
Correspondingly, does the set F of continuous functions I Thus, F forms a vector space over R.
over [a, b] form a vector space? I Every function in this space is an (infinite dimensional) vector.
infinite dimensional vector space I Listing of values is just an obvious basis.

Field
Vector Space
Field
Vector Space
Isomorphism
Linear dependence of (non-zero) functions f and f 1 Product Space
Inner 2 Inner product: For functions f (x) and g (x) in F, the usual inner
Isomorphism
Inner Product Space
I f2 (x) = kf1 (x) for all x in the domain product between corresponding vectors:
I k1 f1 (x) + k2 f2 (x) = 0, x with k1 and k2 not both zero. (vf , vg ) = vfT vg = f (x1 )g (x1 ) + f (x2 )g (x2 ) + f (x3 )g (x3 ) +
Linear independence: k1 f1 (x) + k2 f2 (x) = 0 x k1 = k2 = 0 P
Weighted inner product: (vf , vg ) = vfT Wvg = i wi f (xi )g (xi )
In general,
For the functions,
I Functions f1 , f2 , f3 , , fn F are linearly dependent if
k1 , k2 , k3 , , kn , not all zero, such that Z b
k1 f1 (x) + k2 f2 (x) + k3 f3 (x) + + kn fn (x) = 0 x [a, b]. (f , g ) = w (x)f (x)g (x)dx
a
I k1 f1 (x) + k2 f2 (x) + k3 f3 (x) + + kn fn (x) = 0 x [a, b]
k1 , k2 , k3 , , kn = 0 means that functions f1 , f2 , f3 , , fn are Rb
linearly independent. I Orthogonality: (f , g ) = a w (x)f (x)g (x)dx = 0
qR
b 2
Example: functions 1, x, x 2 , x 3 , are a set of linearly I Norm: kf k = a w (x)[f (x)] dx
independent functions. I Orthonormal R b basis:
(fj , fk ) = a w (x)fj (x)fk (x)dx = jk j, k
Incidentally, this set is a commonly used basis.
Applied Mathematical Methods Vector Spaces: Fundamental Concepts* 163, Applied Mathematical Methods Topics in Multivariate Calculus 164,
Points to note Group

Field
Vector Space
Outline Derivatives in Multi-Dimensional Spaces
Taylors Series
Chain Rule and Change of Variables
Linear Transformation Numerical Differentiation
Isomorphism An Introduction to Tensors*
Inner Product Space
Function Space
I Matrix algebra provides a natural description for vector spaces
and linear transformations.
I Through isomorphisms, R n can represent all n-dimensional Topics in Multivariate Calculus
real vector spaces. Derivatives in Multi-Dimensional Spaces
Taylors Series
I Through the definition of an inner product, a vector space
incorporates key geometric features of physical space.
Numerical Differentiation
I Continuous functions over an interval constitute an infinite An Introduction to Tensors*
dimensional vector space, complete with the usual notions.
Applied Mathematical Methods Topics in Multivariate Calculus 165, Applied Mathematical Methods Topics in Multivariate Calculus 166,
Derivatives in Multi-Dimensional Spaces

Taylors Series
Taylors Series
Numerical Differentiation Numerical Differentiation
Gradient An Introduction to Tensors* An Introduction to Tensors*
T Hessian
f f f f
f (x) (x) = 2f 2f 2f

x x1 x2 xn
x1 2 x2 x1 xn x1
2f 2f 2f
Up to the first order, f [f (x)]T x 2f x1 x2 x2 2
xn x2
H(x) = 2
=
.. .. .. ..

Directional derivative x . . . .
f f (x + d) f (x) 2f 2f 2f
= lim x1 xn x2 xn xn 2
d 0
h i
2f
Relationships: Meaning: f (x + x) f (x) x 2
(x) x
f f f f
= , = dT f (x) and = kf (x)k For a vector function h(x), Jacobian
ej xj d g

Among all unit vectors, taken as directions, h h h h
J(x) = (x) =
I the rate of change of a function in a direction is the same as x x1 x2 xn
the component of its gradient along that direction, and
I the rate of change along the direction of the gradient is the Underlying notion: h [J(x)]x
greatest and is equal to the magnitude of the gradient.
Taylors Series Derivatives in Multi-Dimensional Spaces

Taylors Series
Chain Rule and Change of Variables Derivatives in Multi-Dimensional Spaces
Taylors Series
Taylors formula in the remainder form: An Introduction to Tensors* For f (x), the total differential: An Introduction to Tensors*
f (x + x) = f (x) + f 0 (x)x f f f
1 1 1 df = [f (x)]T dx = dx1 + dx2 + + dxn
+ f 00 (x)x 2 + + f (n1) (x)x n1 + f (n) (xc )x n x1 x2 xn
2! (n 1)! n!
where xc = x + tx with 0 t 1 Ordinary derivative or total derivative:
Mean value theorem: existence of xc df dx
Taylors series: = [f (x)]T
dt dt
1
f (x + x) = f (x) + f 0 (x)x + f 00 (x)x 2 + f T dx
2! For f (t, x(t)), total derivative: df dt = t + [f (x)] dt
For a multivariate function, For f (v, x(v)) = f (v1 , v2 , , vm , x1 (v), x2 (v), , xn (v)),
1 T
f (x + x) = f (x) + [xT ]f (x) + [xT ]2 f (x) + f f f x f x
2! (v, x(v)) = + (v, x) = +[x f (v, x)] T
1 1 vi vi x vi vi vi
+ [xT ]n1 f (x) + [xT ]n f (x + tx) x x
(n 1)! n!
2 T
1 f x
f (x + x) f (x) + [f (x)]T x + xT (x) x f (v, x(v)) = v f (v, x) + (v) x f (v, x)
2 x 2 v

Taylors Series
Taylors Series
For a multiple integral
Let x R m+n and h(x) R m . An Introduction to Tensors*
Z Z Z
An Introduction to Tensors*
Partition x R m+n into z Rn and w R m. I = f (x, y , z) dx dy dz,

A
System of equations h(x) = 0 means h(z, w) = 0.
change of variables x = x(u, v , w ), y = y (u, v , w ), z = z(u, v , w )
Question: Can we work out the function w = w(z)? gives
Solution of m equations in m unknowns? Z Z Z
I = f (x(u, v , w ), y (u, v , w ), z(u, v , w )) |J(u, v , w )| du dv dw ,
Question: If we have one valid pair (z, w), then is it possible to A
develop w = w(z) in the local neighbourhood?
(x,y ,z)
h
Answer: Yes, if Jacobian w is non-singular. where Jacobian determinant |J(u, v , w )| = (u,v ,w ) .
For the differential
Implicit function theorem
P1 (x)dx1 + P2 (x)dx2 + + Pn (x)dxn ,

h h w w h 1 h
+ =0 = we ask: does there exist a function f (x),
z w z z w z I of which this is the differential;
h i
w I or equivalently, the gradient of which is P(x)?
Upto first order, w1 = w + z (z1 z).
Perfect or exact differential: can be integrated to find f .

Taylors Series
Numerical Differentiation Derivatives in Multi-Dimensional Spaces
Taylors Series
Differentiation under the integral sign

Numerical Differentiation
Forward difference formula Numerical Differentiation
R v (x) f (x + x) f (x)
How To differentiate (x) = (x, u(x), v (x)) = u(x) f (x, t) dt? f 0 (x) = + O(x)
In the expression x
Central difference formulae
du dv
0 (x) = + + , f 0 (x) =
f (x + x) f (x x)
+ O(x 2 )
x u dx v dx 2x
R v f
we have
x = u x (x, t)dt.
f (x + x) 2f (x) + f (x x)
f 00 (x) = + O(x 2 )
Now, considering function F (x, t) such that f (x, t) = Ft(x,t)
, x 2
Z v For gradient f (x) and Hessian,
F f 1
(x) = (x, t)dt = F (x, v ) F (x, u) (x, u, v ). (x) = [f (x + ei ) f (x ei )],
u t xi 2

Using = f (x, v ) and = f (x, u),
v u 2f f (x + ei ) 2f (x) + f (x ei )
Z (x) = , and
v (x)
f dv du xi 2 2
0
(x) = (x, t)dt + f (x, v ) f (x, u) . f (x + ei + ej ) f (x + ei ej )
u(x) x dx dx
2f f (x ei + ej ) + f (x ei ej )
(x) =
Leibnitz rule xi xj 4 2
An Introduction to Tensors* Derivatives in Multi-Dimensional Spaces

Taylors Series
Points to note Derivatives in Multi-Dimensional Spaces
Taylors Series
An Introduction to Tensors* An Introduction to Tensors*
I Indicial notation and summation convention

I Kronecker delta and Levi-Civita symbol I Gradient, Hessian, Jacobian and the Taylors series
I Rotation of reference axes I Partial and total gradients
I Tensors of order zero, or scalars I Implicit functions
I Contravariant and covariant tensors of order one, or vectors I Leibnitz rule
I Cartesian tensors I Numerical derivatives
I Cartesian tensors of order two
I Higher order tensors
I Elementary tensor operations Necessary Exercises: 2,3,4,8
I Symmetric tensors
I Tensor fields
I
Applied Mathematical Methods Vector Analysis: Curves and Surfaces 175, Applied Mathematical Methods Vector Analysis: Curves and Surfaces 176,
Outline Recapitulation of Basic Notions

Curves in Space
Surfaces*
Recapitulation of Basic Notions Recapitulation of Basic Notions
Curves in Space
Surfaces*
Dot and cross products: their implications

Scalar and vector triple products
Differentiation rules
Vector Analysis: Curves and Surfaces Interface with matrix algebra:
Recapitulation of Basic Notions
Curves in Space a x = aT x,
Surfaces* (a x)b = (baT )x, and
T
a x, for 2-d vectors
ax =
ax, for 3-d vectors
where

0 az ay
ay
a = and a = az 0 ax
ax
ay ax 0
Curves in Space Recapitulation of Basic Notions

Curves in Space
Surfaces*
Curves in Space
Surfaces*
Explicit equation: y = y (x) and z = z(x)

Curve r(t) is regular if r 0 (t) 6= 0 t.
Implicit equation: F (x, y , z) = 0 = G (x, y , z)
I Reparametrization with respect to parameter t , some
Parametric equation: strictly increasing function of t
r(t) = x(t)i + y (t)j + z(t)k [x(t) y (t) z(t)] T
Observations
I Arc length s(t) is obviously a monotonically increasing
I Tangent vector: r0 (t) function.
I Speed: kr0 k I For a regular curve, ds
dt 6= 0.
r0
I Unit tangent: u(t) = kr0 k
I Then, s(t) has an inverse function.
Rb Rbp I Inverse t(s) reparametrizes the curve as r(t(s)).
I Length of the curve: l = a kdrk = a r0 r0 dt
Arc length function
For a unit speed curve r(s), kr 0 (s)k = 1 and the unit tangent is
Z tp
s(t) = r0 ( ) r0 ( ) d u(s) = r0 (s).
a
p
with ds = kdrk = dx 2 + dy 2 + dz 2 and ds
dt = kr0 k

Curves in Space
Surfaces*
Curves in Space
Surfaces*
Curvature: The rate at which the direction changes with arc Binormal: b = u p
length.
(s) = ku0 (s)k = kr00 (s)k Serret-Frenet frame: Right-handed triad {u, p, b}
Unit principal normal: I Osculating, rectifying and normal planes
1 0
p= u (s) Torsion: Twisting out of the osculating plane

With general parametrization, I rate of change of b with respect to arc length s
dkr0 k du dkr0 k b0 = u0 p + u p0 = (s)p p + u p0 = u p0
r00 (t) = u(t) + kr0 (t)k = u(t) + (t)kr0 k2 p(t)
dt dt dt What is p0 ?
r/ Taking p0 = u + b,
AC = = 1/
I Osculating plane u
b0 = u (u + b) = p.
C
p
A
z r//
I Centre of curvature
r
I Radius of curvature Torsion of the curve
O y
x (s) = p(s) b0 (s)

Figure: Tangent and normal to a curve

Curves in Space
Surfaces*
Surfaces* Recapitulation of Basic Notions
Curves in Space
Surfaces*
We have u0 and b0 . What is p0 ? Parametric surface equation:

From p = b u, r(u, v ) = x(u, v )i+y (u, v )j+z(u, v )k [x(u, v ) y (u, v ) z(u, v )] T
0 0 0
p = b u + b u = p u + b p = u + b. Tangent vectors ru and rv define a tangent plane T .
N = ru rv is normal to the surface and the unit normal is

Serret-Frenet formulae
N ru r v
u0 = p, n= = .
p0 = u + b, kNk kru rv k

b0 = p
Question: How does n vary over the surface?
Intrinsic representation of a curve is complete with (s) and (s). Information on local geometry: curvature tensor
The arc-length parametrization of a curve is completely
I Normal and principal curvatures
determined by its curvature (s) and torsion (s)
functions, except for a rigid body motion. I Local shape: convex, concave, saddle, cylindrical, planar
Applied Mathematical Methods Vector Analysis: Curves and Surfaces 183, Applied Mathematical Methods Scalar and Vector Fields 184,
Points to note Recapitulation of Basic Notions

Curves in Space
Surfaces*
Outline Differential Operations on Field Functions
Integral Operations on Field Functions
Integral Theorems
Closure
I Parametric equation is the general and most convenient

representation of curves and surfaces.
I Arc length is the natural parameter and the Serret-Frenet Scalar and Vector Fields
frame offers the natural frame of reference. Differential Operations on Field Functions
I Curvature and torsion are the only inherent properties of a Integral Operations on Field Functions
curve. Integral Theorems
Closure
I The local shape of a surface patch can be understood through
an analysis of its curvature tensor.
Applied Mathematical Methods Scalar and Vector Fields 185, Applied Mathematical Methods Scalar and Vector Fields 186,
Differential Operations on Field Functions

Integral Theorems
Integral Theorems
Closure Closure
Scalar point function or scalar field (x, y , z): R 3 R
Gradient
Vector point function or vector field V(x, y , z): R 3 R 3
The del or nabla () operator
grad = i+ j+ k
x y z
i +j +k
x y z is orthogonal to the level surfaces.
I is a vector, Flow fields: gives the velocity vector.

I it signifies a differentiation, and
Divergence
I it operates from the left side.
Laplacian operator: For V(x, y , z) Vx (x, y , z)i + Vy (x, y , z)j + Vz (x, y , z)k,
2 2 2 Vx Vy Vz
2 + + = ?? div V V = + +
x 2 y 2 z 2 x y z
Laplaces equation:
Divergence of V: flow rate of mass per unit volume out of the
2 2 2 control volume.
+ 2 + 2 =0
x 2 y z
Similar relation between field and flux in electromagnetics.
Solution of 2 = 0: harmonic function

Integral Theorems
Integral Theorems
Closure Closure
Composite operations
Curl
Operator is linear.

i j k ( + ) = + ,

curl V V = x y z

(V + W) = V + W, and
V V V
x y z
(V + W) = V + W.
Vz Vy Vx Vz Vy Vx
= i+ j+ k
y z z x x y Considering the products , V, V W, and V W;
If V = r represents the velocity field, then angular velocity () = +
(V) = V + V
1
= curl V. (V) = V + V
2 (V W) = (W )V + (V )W + W ( V) + V ( W)
Curl represents rotationality. (V W) = W ( V) V ( W)
(V W) = (W )V W( V) (V )W + V( W)
Connections between electric and magnetic fields!
Note: the expression V Vx x + Vy y + Vz z is an operator!

Integral Theorems
Integral Operations on Field FunctionsDifferential Operations on Field Functions
Integral Theorems
Closure Closure
Second order differential operators Line integral along curve C :
Z Z
I = V dr = (Vx dx + Vy dy + Vz dz)
div grad () C C
curl grad () For a parametrized curve r(t), t [a, b],

div curl V ( V) Z Z b
dr
curl curl V ( V) I = V dr = V dt.
C a dt
grad div V ( V)
For simple (non-intersecting) paths contained in a simply
Important identities: connected region, equivalent statements:
I Vx dx + Vy dy + Vz dz is an exact differential.
div grad () = 2
I V = for some (r).
curl grad () = 0 R
C V dr is Hindependent of path.
I
div curl V ( V) = 0
I Circulation V dr = 0 around any closed path.
curl curl V ( V)
I curl V = 0.
= ( V) 2 V = grad div V 2 V
I Field V is conservative.
Integral Operations on Field FunctionsDifferential Operations on Field Functions

Integral Theorems
Integral Theorems Differential Operations on Field Functions
Integral Theorems
Closure Closure
Greens theorem in the plane
Surface integral over an orientable surface S: R: closed bounded region in the xy -plane
Z Z Z Z C : boundary, a piecewise smooth closed curve
J= V dS = V ndS F1 (x, y ) and F2 (x, y ): first order continuous functions
S S I Z Z
F2 F1
For r(u, w ), dS = kru rw k du dw and (F1 dx + F2 dy ) = dx dy
C R x y
Z Z Z Z
J= V ndS = V (ru rw ) du dw . y
S R y
D
d
y2(x)
R
x1(y) B
Volume integrals of point functions over a region T : R
A x2(y)
Z Z Z Z Z Z
M= dv and F= Vdv c
C y1(x)
T T O a b x O x
(a) Simple domain (b) General domain
Figure: Regions for proof of Greens theorem in the plane


Integral Theorems
Integral Theorems
Closure Closure
Proof: Gausss divergence theorem
Z Z Z b Z y2 (x) T : a closed bounded region
F1 F1
dxdy = dydx S: boundary, a piecewise smooth closed orientable
R y a y1 (x) y surface
Z b
F(x, y , z): a first order continuous vector function
= [F1 {x, y2 (x)} F1 {x, y1 (x)}]dx
a Z Z Z Z Z
Z a Z b
div Fdv = F ndS
= F1 {x, y2 (x)}dx F1 {x, y1 (x)}dx T S
Ib a
= F1 (x, y )dx Interpretation of the definition extended to finite domains.

C
Z Z Z d Z x2 (y ) I Z Z Z Z Z
F2 F2 Fx Fy Fz
dxdy = dxdy = F2 (x, y )dy + + dx dy dz = (Fx nx +Fy ny +Fz nz )dS
x x y z
x1 (y ) x
T S
R c C
H
R R F2 F1 R R R Fz R R
To show: z dx dy dz = S Fz nz dS
Difference: C (F1 dx + F2 dy ) = R x y dx dy T
H R R First consider a region, the boundary of which is intersected at
In alternative form, C F dr = R curl F k dx dy . most twice by any line parallel to a coordinate axis.

Integral Theorems
Integral Theorems
Closure Closure
Lower and upper segments of S: z = z1 (x, y ) and z = z2 (x, y ). Greens identities (theorem)
Z Z Z Z Z Z z2
Fz Fz Region T and boundary S: as required in premises of
dx dy dz = dz dx dy Gausss theorem
T z R z1 z
Z Z (x, y , z) and (x, y , z): second order continuous scalar
= [Fz {x, y , z2 (x, y )} Fz {x, y , z1 (x, y )}]dx dy functions
R Z Z Z Z Z
R: projection of T on the xy -plane ndS = (2 + )dv
Z Z S Z ZT Z
Projection of area element of the upper segment: n z dS = dx dy ( ) ndS = (2 2 )dv
Projection of area element of the lower segment: n z dS = dx dy S T
R R R Fz R R
Thus, T z dx dy dz = S Fz nz dS. Direct consequences of Gausss theorem
Sum of three such components leads to the result. To establish, apply Gausss divergence theorem on , and then
on as well.
Extension to arbitrary regions by a suitable subdivision of domain!

Integral Theorems
Integral Theorems
Closure Closure
Stokess theorem Represent S as z = z(x, y ) f (x, y ).

S: a piecewise smooth surface
C : boundary, a piecewise smooth simple closed curve f
Unit normal n = [nx ny nz ]T is proportional to [ x f
1]T .
y
F(x, y , z): first order continuous vector function
I Z Z z
ny = nz
F dr = curl F ndS y
C S
n: unit normal given by the right hand clasp rule on C Z Z Z Z

Fx Fx Fx Fx z
ny nz dS = + nz dS
S z y S y z y
For F(x, y , z) = Fx (x, y , z)i,
I Z Z Z Z Over projection R of S on xy -plane, (x, y ) = F x (x, y , z(x, y )).
Fx Fx Fx Fx
Fx dx = j k ndS = ny nz dS. Z Z I I
C S z y S z y
LHS = dx dy = (x, y )dx = Fx dx
R y C 0 C
First, consider a surface S intersected at most once by any line Similar results for Fy (x, y , z)j and Fz (x, y , z)k.
parallel to a coordinate axis.
Applied Mathematical Methods Scalar and Vector Fields 199, Applied Mathematical Methods Polynomial Equations 200,
Points to note Differential Operations on Field Functions

Integral Theorems
Outline Basic Principles
Analytical Solution
General Polynomial Equations
Closure Two Simultaneous Equations
Elimination Methods*
Advanced Techniques*
I The del operator

I Gradient, divergence and curl Polynomial Equations
I Composite and second order operators Basic Principles
I Line, surface and volume intergals Analytical Solution
I Greens, Gausss and Stokess theorems Two Simultaneous Equations
I Applications in physics (and engineering) Elimination Methods*
Applied Mathematical Methods Polynomial Equations 201, Applied Mathematical Methods Polynomial Equations 202,
Basic Principles Basic Principles

Analytical Solution
Analytical Solution Basic Principles
Analytical Solution
Two Simultaneous Equations Two Simultaneous Equations
Fundamental theorem of algebra Elimination Methods* Quadratic equation Elimination Methods*
Advanced Techniques* Advanced Techniques*
n n1 n2

p(x) = a0 x + a1 x + a2 x + + an1 x + an b b 2 4ac
ax 2 + bx + c = 0 x =
2a
has exactly n roots x1 , x2 , , xn ; with
Method of completing the square:
p(x) = a0 (x x1 )(x x2 )(x x3 ) (x xn ). 2
b b b2 c b 2 b 2 4ac
x2 + x + = 2 x+ =
In general, roots are complex. a 2a 4a a 2a 4a2
Multiplicity: A root of p(x) with multiplicity k satisfies
Cubic equations (Cardano):
p(x) = p 0 (x) = p 00 (x) = = p (k1) (x) = 0.
x 3 + ax 2 + bx + c = 0
I Descartes rule of signs Completing the cube?
I Bracketing and separation Substituting y = x + k,
I Synthetic division and deflation
y 3 + (a 3k)y 2 + (b 2ak + 3k 2 )y + (c bk + ak 2 k 3 ) = 0.
p(x) = f (x)q(x) + r (x)
Choose the shift k = a/3.

Analytical Solution
Analytical Solution
y 3 + py + q = 0 Elimination Methods*
Quartic equations (Ferrari) Elimination Methods*
y3 u3 v3
Assuming y = u + v , we have = + + 3uv (u + v ). a 2 a2
x 4 +ax 3 +bx 2 +cx +d = 0 x2 + x = b x 2 cx d
uv = p/3 2 4
u 3 + v 3 = q For a perfect square,
4p 3 2
y 2 ay
3 3 2 2
and hence (u v ) = q + . a a2 y
27 x2 + x + = b+y x2 + c x + d
2 2 4 2 4
Solution:
r Under what condition, the new RHS will be a perfect square?
3 q3 q2 p3
u ,v = + = A, B (say). ay 2
2 4 27 a2 y2
c 4 b+y d =0
2 4 4
u = A1 , A1 , A1 2 , and v = B1 , B1 , B1 2
Resolvent of a quartic:
y1 = A1 + B1 , y2 = A1 + B1 2 and y3 = A1 2 + B1 .
y 3 by 2 + (ac 4d)y + (4bd a2 d c 2 ) = 0
At least one of the roots is real!!

Analytical Solution
General Polynomial Equations Basic Principles
Analytical Solution
Elimination Methods* Analytical solution of the general quintic equation?Elimination Methods*
Procedure Galois: group theory:
I Frame the cubic resolvent. A general quintic, or higher degree, equation is not
I Solve this cubic equation. solvable by radicals.
I Pick up one solution as y . General polynomial equations: iterative algorithms
I Insert this y to form I Methods for nonlinear equations
a y 2 I Methods specific to polynomial equations
x2 + x + = (ex + f )2 .
2 2 Solution through the companion matrix
I Split it into two quadratic equations as Roots of a polynomial equation are the same as the
eigenvalues of its companion matrix.
a y
x 2 + x + = (ex + f ). 0 0 0 an
2 2
1 0 0 an1

Solve each of the two quadratic equations to obtain a total of ..
Companion matrix: ... ... . . . ...
I
four solutions of the original quartic equation. .

0 0 0 a2
0 0 1 a1
General Polynomial Equations Basic Principles

Analytical Solution
Two Simultaneous Equations Basic Principles
Analytical Solution
Elimination Methods* Elimination Methods*
Bairstows method p1 x 2 + q1 xy + r1 y 2 + u1 x + v1 y + w1 = 0
to separate out factors of small degree.
p2 x 2 + q2 xy + r2 y 2 + u2 x + v2 y + w2 = 0
Attempt to separate real linear factors? Rearranging,
a1 x 2 + b 1 x + c 1 = 0
Real quadratic factors
a2 x 2 + b 2 x + c 2 = 0
Synthetic division with a guess factor x2 + q1 x + q 2 : Cramers rule:
remainder r1 x + r2 x2 x 1
= =
b1 c2 b 2 c1 a1 c2 a 2 c1 a1 b2 a 2 b1
r = [r1 r2 ]T is a vector function of q = [q1 q2 ]T .
b1 c2 b 2 c1 a1 c2 a 2 c1
x = =
a1 c2 a 2 c1 a1 b2 a 2 b1
Iterate over (q1 , q2 ) to make (r1 , r2 ) zero.
Consistency condition:
Newton-Raphson (Jacobian based) iteration: see exercise. (a1 b2 a2 b1 )(b1 c2 b2 c1 ) (a1 c2 a2 c1 )2 = 0
A 4th degree equation in y
Basic Principles Basic Principles
Elimination Methods* Analytical Solution
Advanced Techniques* Analytical Solution
Three or more independent equations in as many unknowns?
The method operates similarly even if the degrees of the original
equations in y are higher. I Cascaded elimination? Objections!
I Exploitation of special structures through clever heuristics
What about the degree of the eliminant equation? (mechanisms kinematics literature)
Two equations in x and y of degrees n1 and n2 :
x-eliminant is an equation of degree n 1 n2 in y
I Grobner basis representation
(algebraic geometry)
Maximum number of solutions:
Bezout number = n1 n2 I Continuation or homotopy method by Morgan
Note: Deficient systems may have less number of solutions. For solving the system f(x) = 0, identify another
structurally similar system g(x) = 0 with known
solutions and construct the parametrized system
Classical methods of elimination
I Sylvesters dialytic method h(x) = tf(x) + (1 t)g(x) = 0 for t [0, 1].
I Bezouts method
Track each solution from t = 0 to t = 1.
Applied Mathematical Methods Polynomial Equations 211, Applied Mathematical Methods Solution of Nonlinear Equations and Systems 212,
Points to note Basic Principles

Analytical Solution
Outline Methods for Nonlinear Equations
Systems of Nonlinear Equations
Closure
Two Simultaneous Equations
I Roots of cubic and quartic polynomials by the methods of

Cardano and Ferrari
I For higher degree polynomials, Solution of Nonlinear Equations and Systems
I Bairstows method: a clever implementation of Methods for Nonlinear Equations
Newton-Raphson method for polynomials
I Eigenvalue problem of a companion matrix
Closure
I Reduction of a system of polynomial equations in two
unknowns by elimination
Applied Mathematical Methods Solution of Nonlinear Equations and Systems 213, Applied Mathematical Methods Solution of Nonlinear Equations and Systems 214,
Methods for Nonlinear Equations Methods for Nonlinear Equations

Closure
Closure
w y=x
Fixed point iteration y y = g(x) u v
Algebraic and transcendental equations in the form Rearrange f (x) = 0 in
the form x = g (x). m l
f (x) = 0
Example:
Practical problem: to find one real root (zero) of f (x) For f (x) = tan x x 3 2, b a n
possible rearrangements:
g1 (x) = tan1 (x 3 + 2) d
Example of f (x): x 3 2x + 5, x 3 ln x sin x + 2, etc. c
g2 (x) = (tan x 2)1/3 f

g3 (x) = tanxx2
e
2 g
If f (x) is continuous, then
Iteration: xk+1 = g (xk ) O x p qx r x
Bracketing: f (x0 )f (x1 ) < 0 there must be a root of f (x)
between x0 and x1 . Figure: Fixed point iteration
Bisection: Check the sign of f ( x0 +x
2 ). Replace either x0 or x1
1
with x0 +x 1
.
2 If x is the unique solution in interval J and
|g 0 (x)| h < 1 in J, then any x0 J converges to x .

Closure
Closure
Newton-Raphson method Secant method and method of false position

First order Taylor series f(x) f(x)
f (x + x) f (x) + f 0 (x)x
a
In the Newton-Raphson formula, f(x0)
f (x )f (x )
From f (xk + x) = 0, f 0 (x) xkk xk1k1
x = f (xk )/f 0 (xk ) e
xk xk1
Iteration: xk+1 = xk f (xk )f (xk1 ) f (xk )
xk+1 = xk f (xk )/f 0 (xk )
Draw the chord or
Convergence criterion: secant to f (x) through O
x1 x2 x3 x*
x0 x
f
|f (x)f 00 (x)| < |f 0 (x)|2 O
b
g
x*
d x0 x
(xk1 , f (xk1 )) and (xk , f (xk )). f(x1)
Draw tangent to f (x). c Take its x-intercept.

Take its x-intercept. Figure: Method of false position
Figure: Newton-Raphson method
Special case: Maintain a bracket over the root at every iteration.
Merit: quadratic speed of convergence: |x k+1 x | = c|xk x |2 The method of false position or regula falsi
Demerit: If the starting point is not appropriate,
haphazard wandering, oscillations or outright divergence! Convergence is guaranteed!

Closure
Systems of Nonlinear Equations Methods for Nonlinear Equations
Closure
Quadratic interpolation method or Muller method

Evaluate f (x) at three points y
f1 (x1 , x2 , , xn ) = 0,
and model y = a + bx + cx 2 .
Set y = 0 and solve for x. f2 (x1 , x2 , , xn ) = 0,
(x0,y0 )

Quadratic
Interpolation
fn (x1 , x2 , , xn ) = 0.
Inverse quadratic interpolation Inverse
Evaluate f (x) at three points (x1 ,y1 )

Quadratic
Interpolation f(x) = 0
and model x = a + by + cy 2 . O
x3
x3 x Number of variables and number of equations?
I
Set y = 0 to get x = a. (x2 ,y2)
I No bracketing!
I Fixed point iteration schemes x = g(x)?
Figure: Interpolation schemes
Newtons method for systems of equations
Van Wijngaarden-Dekker Brent method
I maintains the bracket, f
f(x + x) = f(x) + (x) x + f(x) + J(x)x
x
I uses inverse quadratic interpolation, and
I accepts outcome if within bounds, else takes a bisection step. xk+1 = xk [J(xk )]1 f(xk )
Opportunistic manoeuvring between a fast method and a safe one! with the usual merits and demerits!
Closure Methods for Nonlinear Equations

Closure
Points to note Methods for Nonlinear Equations
Closure
Modified Newtons method
xk+1 = xk k [J(xk )]1 f(xk ) I Iteration schemes for solving f (x) = 0

I Newton (or Newton-Raphson) iteration for a system of
Broydens secant method equations
xk+1 = xk [J(xk )]1 f(xk )
Jacobian is not evaluated at every iteration, but gets
developed through updates. I Optimization formulation of a multi-dimensional root finding
problem
Optimization-based formulation
Global minimum of the function Necessary Exercises: 1,2,3

kf(x)k2 = f12 + f22 + + fn2
Levenberg-Marquardt method
Applied Mathematical Methods Optimization: Introduction 221, Applied Mathematical Methods Optimization: Introduction 222,
Outline The Methodology of Optimization

Single-Variable Optimization
Conceptual Background of Multivariate Optimization
The Methodology of Optimization The Methodology of Optimization
I Parameters and variables

I The statement of the optimization problem
Optimization: Introduction
Minimize f (x)
The Methodology of Optimization
subject to g(x) 0,
h(x) = 0.
I Optimization methods
I Sensitivity analysis
I Optimization problems: unconstrained and constrained
I Optimization problems: linear and nonlinear
I Single-variable and multi-variable problems
Single-Variable Optimization The Methodology of Optimization

For a function f (x), a point x is defined as a relative (local)

minimum if such that f (x) f (x ) x [x , x + ]. Higher order analysis: From Taylors series,
f( x) f = f (x + x) f (x )
1 1 1
= f 0 (x )x + f 00 (x )x 2 + f 000 (x )x 3 + f iv (x )x 4 +
2! 3! 4!
For an extremum to occur at point x , the lowest order

O a x1 x2 x3 x4 x5 x6 b x derivative with non-zero value should be of even order.
Figure: Schematic of optima of a univariate function
If f 0 (x ) = 0, then
Optimality criteria I x is a stationary point, a candidate for an extremum.
First order necessary condition: If x is a local minimum or I Evaluate higher order derivatives till one of them is found to
maximum point and if f 0 (x ) exists, then f 0 (x ) = 0. be non-zero.
Second order necessary condition: If x is a local minimum point I If its order is odd, then x is an inflection point.
I If its order is even, then x is a local minimum or maximum,
and f 00 (x ) exists, then f 00 (x ) 0.
as the derivative value is positive or negative, respectively.
Second order sufficient condition: If f 0 (x ) = 0 and f 00 (x ) > 0
then x is a local minimum point.

Iterative methods of line search Bracketing:

Methods based on gradient root finding x1 < x2 < x3 with f (x1 ) f (x2 ) f (x3 )
I Newtons method
Exhaustive search method or its variants
f 0 (xk )
xk+1 = xk Direct optimization algorithms
f 00 (xk )
I Fibonacci search uses a pre-defined number N, of function
I Secant method evaluations, and the Fibonacci sequence
xk xk1
xk+1 = xk f 0 (xk ) F0 = 1, F1 = 1, F2 = 2, , Fj = Fj2 + Fj1 ,
f 0 (xk ) f 0 (xk1 )
I Method of cubic estimation to tighten a bracket with economized number of function
point of vanishing gradient of the cubic fit with evaluations.
f (xk1 ), f (xk ), f 0 (xk1 ) and f 0 (xk ) I Golden section search uses a constant ratio

51
I Method of quadratic estimation = 0.618,
2
point of vanishing gradient of the quadratic fit
the golden section ratio, of interval reduction, that is
through three points
determined as the limiting case of N and the actual
Disadvantage: treating all stationary points alike! number of steps is decided by the accuracy desired.

Convexity
Unconstrained minimization problem Set S R n is a convex set if
x is called a local minimum of f (x) if such that x1 , x2 S and (0, 1), x1 + (1 )x2 S.
f (x) f (x ) for all x satisfying kx x k < .
Function f (x) over a convex set S: a convex function if
x1 , x2 S and (0, 1),
Optimality criteria f (x1 + (1 )x2 ) f (x1 ) + (1 )f (x2 ).
From Taylors series,
Chord approximation is an overestimate at intermediate points!
1
f (x) f (x ) = [g(x )]T x + xT [H(x )]x + .
2 x2 f(x)
For x to be a local minimum, f(x2)

X2
necessary condition: g(x ) = 0 and H(x ) is positive semi-definite, X1 f(x1)
sufficient condition: g(x ) = 0 and H(x ) is positive definite. O x1 O x1 x2 x
Indefinite Hessian matrix characterizes a saddle point. Figure: A convex domain Figure: A convex function

First order characterization of convexity Quadratic function
From f (x1 + (1 )x2 ) f (x1 ) + (1 )f (x2 ), 1

q(x) = xT Ax + bT x + c
2
f (x2 + (x1 x2 )) f (x2 )
f (x1 ) f (x2 ) . Gradient q(x) = Ax + b and Hessian = A is constant.

As 0, f (x1 ) f (x2 ) + [f (x2 )]T (x1 x2 ). I If A is positive definite, then the unique solution of Ax = b
is the only minimum point.
Tangent approximation is an underestimate at intermediate points!
I If A is positive semi-definite and b Range(A), then the
Second order characterization: Hessian is positive semi-definite. entire subspace of solutions of Ax = b are global minima.
I If A is positive semi-definite but b
/ Range(A), then the
Convex programming problem: convex function over convex set function is unbounded!
A local minimum is also a global minimum, and all
minima are connected in a convex set. Note: A quadratic problem (with positive definite Hessian) acts as
Note: Convexity is a stronger condition than unimodality! a benchmark for optimization algorithms.

Optimization Algorithms
Convergence of algorithms: notions of guarantee and speed
From the current point, move to another point, hopefully better.
Global convergence: the ability of an algorithm to approach and
converge to an optimal solution for an arbitrary
Which way to go? How far to go? Which decision is first?
problem, starting from an arbitrary point
I Practically, a sequence (or even subsequence) of
Strategies and versions of algorithms:
monotonically decreasing errors is enough.
Trust Region: Develop a local quadratic model
1 Local convergence: the rate/speed of approach, measured by p,
f (xk + x) = f (xk ) + [g(xk )]T x + xT Fk x, where
2 kxk+1 x k
and minimize it in a small trust region around x k . = lim <
k kxk x kp
(Define trust region with dummy boundaries.)
Line search: Identify a descent direction d k and minimize the I Linear, quadratic and superlinear rates of
function along it through the univariate function
convergence for p = 1, 2 and intermediate.
() = f (xk + dk ). I Comparison among algorithms with linear rates
I Exact or accurate line search of convergence is by the convergence ratio .
I Inexact or inaccurate line search
I Armijo, Goldstein and Wolfe conditions
Applied Mathematical Methods Optimization: Introduction 233, Applied Mathematical Methods Multivariate Optimization 234,
Points to note The Methodology of Optimization

Outline Direct Methods
Steepest Descent (Cauchy) Method
Newtons Method
Hybrid (Levenberg-Marquardt) Method
Least Square Problems
I Theory and methods of single-variable optimization

I Optimality criteria in multivariate optimization
Multivariate Optimization
I Convexity in optimization Direct Methods
I The quadratic function Steepest Descent (Cauchy) Method
I Trust region Newtons Method
I Line search Hybrid (Levenberg-Marquardt) Method
I Global and local convergence

Applied Mathematical Methods Multivariate Optimization 235, Applied Mathematical Methods Multivariate Optimization 236,
Direct Methods Direct Methods

Newtons Method
Newtons Method
Hybrid (Levenberg-Marquardt) Method Hybrid (Levenberg-Marquardt) Method
Least Square Problems Nelder and Meads simplex method Least Square Problems
Direct search methods using only function values Simplex in n-dimensional space: polytope formed by n + 1 vertices
I Cyclic coordinate search Nelder and Meads method iterates over simplices that are
I Rosenbrocks method non-degenerate (i.e. enclosing non-zero hypervolume).
I Hooke-Jeeves pattern search First, n + 1 suitable points are selected for the starting simplex.
I Boxs complex method
Among vertices of the current simplex, identify the worst point x w ,
I Nelder and Meads simplex search the best point xb and the second worst point xs .
I Powells conjugate directions method Need to replace xw with a good point.
Useful for functions, for which derivative either does not exist at all
points in the domain or is computationally costly to evaluate. Centre of gravity of the face not containing x w :
n+1
X
1
Note: When derivatives are easily available, gradient-based xc = xi
n
algorithms appear as mainstream methods. i =1,i 6=w
Reflect xw with respect to xc as xr = 2xc xw . Consider options.

Newtons Method
Steepest Descent (Cauchy) Method Direct Methods
Newtons Method
Default xnew = xr . Hybrid (Levenberg-Marquardt) Method
Revision possibilities: From a point xk , a move through units in direction d k :
f(xb) f(x s ) f(xw) f (xk + dk ) = f (xk ) + [g(xk )]T dk + O(2 )

Expansion Default Positive Negative
Contraction Contraction
x new Descent direction dk : For > 0, [g(xk )]T dk < 0
xw xr x new x r = xnew xw xr xw xr
xw
Direction of steepest descent: dk = gk [ or dk = gk /kgk k]
Figure: Nelder and Meads simplex method Minimize
() = f (xk + dk ).
1. For f (xr ) < f (xb ), expansion: Exact line search:
xnew = xc + (xc xw ), > 1.
2. For f (xr ) f (xw ), negative contraction: 0 (k ) = [g(xk + k dk )]T dk = 0
xnew = xc (xc xw ), 0 < < 1.
Search direction tangential to the contour surface at (x k + k dk ).
3. For f (xs ) < f (xr ) < f (xw ), positive contraction:
xnew = xc + (xc xw ), with 0 < < 1. Note: Next direction dk+1 = g(xk+1 ) orthogonal to dk
Replace xw with xnew . Continue with new simplex.
Steepest Descent (Cauchy) Method Steepest Descent (Cauchy) Method
Newtons Method
Steepest Descent (Cauchy) Method Steepest Descent (Cauchy) Method
Newtons Method
Least Square Problems Least Square Problems
Analysis on a quadratic function
Steepest descent algorithm
1. Select a starting point x0 , set k = 0 and several parameters: For minimizing q(x) = 21 xT Ax + bT x, the error function:
tolerance G on gradient, absolute tolerance A on reduction
1
in function value, relative tolerance R on reduction in E (x) = (x x )T A(x x )
function value and maximum number of iterations M. 2
2
2. If kgk k G , STOP. Else dk = gk /kgk k. Convergence ratio: EE(x(xk+1 )
(A)1
k) (A)+1
3. Line search: Obtain k by minimizing () = f (xk + dk ),
> 0. Update xk+1 = xk + k dk . Local convergence is poor.
4. If |f (xk+1 ) f (xk )| A + R |f (xk )|,STOP. Else k k + 1.
Importance of steepest descent method
5. If k > M, STOP. Else go to step 2.
I conceptual understanding
Very good global convergence.
I initial iterations in a completely new problem
I spacer steps in other sophisticated methods
But, why so many STOPS?
Re-scaling of the problem through change of variables?
Newtons Method Direct Methods

Newtons Method
Newtons Method Direct Methods
Newtons Method
Second order approximation of a function: Hybrid (Levenberg-Marquardt) Method
1 Modified Newtons method

f (x) f (xk ) + [g(xk )]T (x xk ) + (x xk )T H(xk )(x xk )
2 I Replace the Hessian by Fk = H(xk ) + I .
Vanishing of gradient I Replace full Newtons step by a line search.
g(x) g(xk ) + H(xk )(x xk )
gives the iteration formula Algorithm

1. Select x0 , tolerance and > 0. Set k = 0.
xk+1 = xk [H(xk )]1 g(xk ).
2. Evaluate gk = g(xk ) and H(xk ).
Excellent local convergence property! Choose , find Fk = H(xk ) + I , solve Fk dk = gk for dk .
kxk+1 x k 3. Line search: obtain k to minimize () = f (xk + dk ).
Update xk+1 = xk + k dk .
kxk x k2
4. Check convergence: If |f (xk+1 ) f (xk )| < , STOP.
Caution: Does not have global convergence. Else, k k + 1 and go to step 2.
If H(xk ) is positive definite then dk = [H(xk )]1 g(xk )
is a descent direction.
Direct Methods
Newtons Method
Least Square Problems Direct Methods
Newtons Method
Methods of deflected gradients
Linear least square problem:
xk+1 = xk k [Mk ]gk
y () = x1 1 () + x2 2 () + + xn n ()
I identity matrix in place of Mk : steepest descent step For measured values y (i ) = yi ,
I Mk = F1
k : step of modified Newtons method n
X
I Mk = [H(xk )]1 and k = 1: pure Newtons step ei = xk k (i ) yi = [(i )]T x yi .
k=1
In Mk = [H(xk ) + k I ]1 , tune parameter k over iterations.
I Initial value of : large enough to favour steepest descent Error vector: e = Ax y
trend
Last square fit:
I Improvement in an iteration: reduced by a factor P
1
I Increase in function value: step rejected and increased Minimize E = 2
2
i ei = 12 eT e
Opportunism systematized!
Note: Cost of evaluating the Hessian remains a bottleneck. Pseudoinverse solution and its variants
Useful for problems where Hessian estimates come cheap!

Newtons Method
Newtons Method
Nonlinear least square problem
For model function in the form Levenberg-Marquardt algorithm
1. Select x0 , evaluate E (x0 ). Select tolerance , initial and its
y () = f (, x) = f (, x1 , x2 , , xn ), update factor. Set k = 0.
square error function 2. Evaluate gk and Hk = JT J + diag(JT J).
Solve Hk x = gk . Evaluate E (xk + x).
1 T 1X 2 1X 3. If |E (xk + x) E (xk )| < , STOP.
E (x) = e e= ei = [f (i , x) yi ]2
2 2 2
i i 4. If E (xk + x) < E (xk ), then decrease ,
P update xk+1 = xk + x, k k + 1.
Gradient: g(x) = E (x) = i [f (i , x) yi ]f (i , x) = JT e Else increase .
2 P 2
Hessian: H(x) = x 2
E (x) = JT J + i ei x T
2 f (i , x) J J 5. Go to step 2.
Combining a modified form diag(JT J) x = g(x) of steepest Professional procedure for nonlinear least square problems and also
descent formula with Newtons formula, for solving systems of nonlinear equations in the form h(x) = 0.
Levenberg-Marquardt step: [JT J + diag(JT J)]x = g(x)
Applied Mathematical Methods Multivariate Optimization 247, Applied Mathematical Methods Methods of Nonlinear Optimization* 248,
Points to note Direct Methods

Newtons Method
Outline Conjugate Direction Methods
Quasi-Newton Methods
Closure
I Simplex method of Nelder and Mead

I Steepest descent method with its global convergence
Methods of Nonlinear Optimization*
I Newtons method for fast local convergence
Conjugate Direction Methods
I Levenberg-Marquardt method for equation solving and least Quasi-Newton Methods
squares Closure
Necessary Exercises: 1,2,3,4,5,6
Applied Mathematical Methods Methods of Nonlinear Optimization* 249, Applied Mathematical Methods Methods of Nonlinear Optimization* 250,
Conjugate Direction Methods Conjugate Direction Methods

Closure
Closure
Conjugacy of directions:
Question: How to find a set of n conjugate directions?
Two vectors d1 and d2 are mutually conjugate with
respect to a symmetric matrix A, if d T
1 Ad2 = 0. Gram-Schmidt procedure is a poor option!
Linear independence of conjugate directions: Conjugate gradient method
Conjugate directions with respect to a positive definite
Starting from d0 = g0 ,
matrix are linearly independent.
Expanding subspace property: In R n , with conjugate vectors dk+1 = gk+1 + k dk
{d0 , d1 , , dn1 } with respect to symmetric positive definite A,
Imposing the condition of conjugacy of d k+1 with dk ,
for any x0 R n , the sequence {x0 , x1 , x2 , , xn } generated as
T Ad T (g
gkT dk gk+1 k gk+1 k+1 gk )
xk+1 = xk + k dk , with k = , k = =
dT
k Adk dT
k Adk k dT
k Adk
where gk = Axk + b, has the property that
Resulting dk+1 conjugate to all the earlier directions, for
xk minimizes q(x) = 12 xT Ax + bT x on the line
a quadratic problem.
xk1 + dk1 , as well as on the linear variety x0 + Bk ,
where Bk is the span of d0 , d1 , , dk1 .

Closure
Closure
Using k in place of k + 1 in the formula for d k+1 , Extension to general (non-quadratic) functions
I Varying Hessian A: determine the step size by line search.
dk = gk + k1 dk1
I After n steps, minimum not attained.
gkT gk But, gkT dk = gkT gk implies guaranteed descent.
gkT dk = gkT gk and k =
dT
k Adk
Globally convergent, with superlinear rate of convergence.
I What to do after n steps? Restart or continue?
Polak-Ribiere formula:
T (g Algorithm
gk+1 k+1 gk )
k = 1. Select x0 and tolerances G , D . Evaluate g0 = f (x0 ).
gkT gk 2. Set k = 0 and dk = gk .
No need to know A! 3. Line search: find k ; update xk+1 = xk + k dk .
Further, 4. Evaluate gk+1 = f (xk+1 ). If kgk+1 k G , STOP.
T (g
gk+1 k+1 gk )
T T
dk = 0 gk+1 gk = k1 (gkT + k dT 5. Find k = (Polak-Ribiere)
gk+1 k A)dk1 = 0. gkT gk
T
gk+1 gk+1
Fletcher-Reeves formula: or k = gkT gk
(Fletcher-Reeves).
T g
gk+1 Obtain dk+1 = g k+1 + k dk .
k+1 dT d
k = 6. If 1 kdk kk kdk+1 < D , reset g0 = gk+1 and go to step 2.
gkT gk k+1 k
Else, k k + 1 and go to step 3.

Closure
Closure
Powells conjugate direction method

For q(x) = 12 xT Ax + bT x, suppose Algoithm
x1 = xA + 1 d such that dT g1 = 0 and 1. Select x0 , and a set of n linearly independent (preferably
x2 = xB + 2 d such that dT g2 = 0. normalized) directions d1 , d2 , , dn ; possibly di = ei .
Then, dT A(x2 x1 ) = dT (g2 g1 ) = 0. 2. Line search along dn and update x1 = x0 + dn ; set k = 1.
Parallel subspace property: In R n , consider two parallel P along d1 , d2 , , dn in sequence to obtain
3. Line searches
linear varieties S1 = v1 + Bk and S2 = v2 + Bk , with z = xk + nj=1 j dj .
Bk = {d1 , d2 , , dk }, k < n. 4. New conjugate direction d = z xk . If kdk < , STOP.
If x1 and x2 5. Reassign directions dj dj+1 for j = 1, 2, , (n 1) and
minimize q(x) = 21 xT Ax+bT x on S1 and S2 , respectively, dn = d/kdk.
then x2 x1 is conjugate to d1 , d2 , , dk . (Old d1 gets discarded at this step.)
6. Line search and update xk+1 = z + dn ; set k k + 1 and
Assumptions imply g1 , g2 Bk and hence go to step 3.
(g2 g1 ) Bk dT T
i A(x2 x1 ) = di (g2 g1 ) = 0 for i = 1, 2, , k.

Closure
Quasi-Newton Methods Conjugate Direction Methods
Closure
I x0 -x1 and b-z1 : x1 -z1 is conjugate to b-z1 . Variable metric methods

I b-z1 -x2 and c-d-z2 : c-d, d-z2 and x2 -z2 are mutually attempt to construct the inverse Hessian B k .
conjugate.
pk = xk+1 xk and qk = gk+1 gk qk Hpk
x3
x3
With n such steps, B = PQ1 : update and construct Bk H1 .
d z2 Rank one correction: Bk+1 = Bk + ak zk zT
k ?
Rank two correction:
x2
c Bk+1 = Bk + ak zk zT T
k + b k wk wk
x2
x1 z1
Davidon-Fletcher-Powell (DFP) method
x1 a b
Select x0 , tolerance and B0 = In . For k = 0, 1, 2, ,
x0
I dk = Bk gk .
I Line search for k ; update pk = k dk , xk+1 = xk + pk ,
Figure: Schematic of Powells conjugate direction method qk = gk+1 gk .
I If kpk k < or kqk k < , STOP.
Performance of Powells method approaches that of the pk pT B k qk qTk Bk
I Rank two correction: BDFP
k+1 = Bk + pT
k
qT
.
conjugate gradient method! k qk k B k qk
Quasi-Newton Methods Conjugate Direction Methods

Quasi-Newton Methods Closure Conjugate Direction Methods
Closure Closure 23. Methods of Nonlinear Optimization*
Properties of DFP iterations:
1. If Bk is symmetric and positive definite, then so is B k+1 .
2. For quadratic function with positive definite Hessian H, Table 23.1: Summary of performance of optimization methods
Cauchy Newton Levenberg-Marquardt DFP/BFGS FR/PR Powell

(Steepest (Hybrid) (Quasi-Newton) (Conjugate (Direction
pT
i Hpj = 0 for 0 i < j k, Descent) (Deflected Gradient) (Variable Metric) Gradient) Set)
For Quadratic
and Bk+1 Hpi = pi for 0 i k. Problems:
Convergence steps N 1 N n n n2
Indefinite Unknown
Implications: Evaluations Nf 2f Nf (n + 1)f (n + 1)f n2 f

Ng 2g Ng (n + 1)g (n + 1)g
1. Positive definiteness of inverse Hessian estimate is never lost. 1H NH
2. Successive search directions are conjugate directions. Equivalent function

evaluations N (2n + 1) 2n2 + 2n + 1 N (2n2 + 1) 2n2 + 3n + 1 2n2 + 3n + 1 n2
3. With B0 = I, the algorithm is a conjugate gradient method. Line searches N 0 N or 0 n n n2
4. For a quadratic problem, the inverse Hessian gets completely Storage Vector Matrix Matrix Matrix Vector Matrix
constructed after n steps. Performance in
general problems Slow Risky Costly Flexible Good Okay
Practically good for Unknown Good NL Eqn. systems Bad Large Small
Variants: Broyden-Fletcher-Goldfarb-Shanno (BFGS) start-up functions NL least squares functions problems problems
method and the Broyden family of methods

197
Applied Mathematical Methods Methods of Nonlinear Optimization* 259, Applied Mathematical Methods Constrained Optimization 260,
Points to note Conjugate Direction Methods

Closure
Outline Constraints
Optimality Criteria
Sensitivity
Duality*
Structure of Methods: An Overview*
I Conjugate directions and the expanding subspace property

I Conjugate gradient method Constrained Optimization
I Powell-Smith direction set method Constraints
Optimality Criteria
I The quasi-Newton concept in professional optimization
Sensitivity
Duality*
Applied Mathematical Methods Constrained Optimization 261, Applied Mathematical Methods Constrained Optimization 262,
Constraints Constraints
Optimality Criteria
Sensitivity
Optimality Criteria
Sensitivity
Constrained optimization problem: Duality*
Duality*
Constraint qualification
Minimize f (x)
subject to gi (x) 0 for i = 1, 2, , l, or g(x) 0; h1 (x), h2 (x) etc are linearly independent, i.e. h(x) is
and hj (x) = 0 for j = 1, 2, , m, or h(x) = 0. full-rank.
Conceptually, minimize f (x), x . If a feasible point x0 , with h(x0 ) = 0, satisfies the constraint
Equality constraints reduce the domain to a surface or a manifold, qualification condition, we call it a regular point.
possessing a tangent plane at every point.
Gradient of the vector function h(x): At a regular feasible point x0 , tangent plane
T M = {y : [h(x0 )]T y = 0}
h
x1
hT gives the collection of feasible directions.
x2
h(x) [h1 (x) h2 (x) hm (x)] .. ,
.
Equality constraints reduce the dimension of the problem.
hT
xn
Variable elimination?
h
related to the usual Jacobian as Jh (x) = x = [h(x)]T .
Optimality Criteria
Sensitivity
Optimality Criteria Constraints
Optimality Criteria
Sensitivity
Duality* Duality*
Active inequality constraints gi (x0 ) = 0: Structure of Methods: An Overview*
Suppose x is a regular point with Structure of Methods: An Overview*
included among hj (x0 ) I active inequality constraints: g (a) (x) 0

for the tangent plane.
I inactive constraints: g (i ) (x) 0
Columns of h(x ) and g(a) (x ): basis for orthogonal
Cone of feasible directions:
complement of the tangent plane
[h(x0 )]T d = 0 and [gi (x0 )]T d 0 for i I Basis of the tangent plane: D = [d1 d2 dk ]
Then, [D h(x ) g(a) (x )]: basis of R n
where I is the set of indices of active inequality constraints.
Now, f (x ) is a vector in R n .
Handling inequality constraints:

I Active set strategy maintains a list of active constraints, z
keeps checking at every step for a change of scenario and f (x ) = [D h(x ) g(a) (x )]
updates the list by inclusions and exclusions. (a)
I Slack variable strategy replaces all the inequality constraints
with unique z, and (a) for a given f (x ).
by equality constraints as gi (x) + xn+i = 0 with the inclusion
of non-negative slack variables (x n+i ). What can you say if x is a solution to the NLP problem?

Optimality Criteria
Sensitivity
Optimality Criteria
Sensitivity
Duality* Duality*
Components of f (x ) in the tangent plane must be zero.Structure of Methods: An Overview* Finally, what about the feasible directions in the cone? Structure of Methods: An Overview*
Answer: Negative gradient f (x ) can have no component

z=0 f (x ) = [h(x )] + [g(a) (x )](a) (a) (a)
towards decreasing gi (x), i.e. i 0, i.
For inactive constraints, insisting on (i ) = 0, Combining it with i = 0,
(i )
0.
First order necessary conditions or Karusch-Kuhn-Tucker
(a)
f (x ) = [h(x )] + [g(a) (x ) g(i ) (x )] , (KKT) conditions: If x is a regular point of the constraints and
(i )
a solution to the NLP problem, then there exist Lagrange
or multiplier vectors, and , such that
f (x ) + [h(x )] + [g(x )] = 0
(a) (a) Optimality: f (x ) + [h(x )] + [g(x )] = 0, 0;
g (x) Feasibility: h(x ) = 0, g(x ) 0;
where g(x) = and = .
g(i ) (x) (i ) Complementarity: T g(x ) = 0.
Notice: g(a) (x ) = 0 and (i ) = 0 i gi (x ) = 0 i, or
Convex programming problem: Convex objective function f (x)
T g(x ) = 0.
and convex domain (convex gi (x) and linear hj (x)):
Now, components in g(x) are free to appear in any order. KKT conditions are sufficient as well!

Optimality Criteria
Sensitivity
Optimality Criteria
Sensitivity
Duality* Duality*
Lagrangian function: Structure of Methods: An Overview* Structure of Methods: An Overview*
Including contributions from all active constraints,
L(x, , ) = f (x) + T h(x) + T g(x)

Necessary conditions for a stationary point of the Lagrangian: d2
f (z(t)) = z(0)T HL (x )z(0) + [x L(x , , )]T z(0) 0,
dt 2
t=0
x L = 0, L = 0
2L P P
Second order conditions where HL (x) = x 2
= H(x) + j j Hhj (x) + i i Hgi (x).
Consider curve z(t) in the tangent plane with z(0) = x . First order necessary condition makes the second term vanish!
Second order necessary condition:
d2 d
f (z(t)) = [f (z(t))T z(t)]
dt 2 dt The Hessian matrix of the Lagrangian function is positive
t=0 t=0
= z(0)T H(x )z(0) + [f (x )]T z(0) 0 semi-definite on the tangent plane M.
Sufficient condition: x L = 0 and HL (x) positive definite on M.
Similarly, from hj (z(t)) = 0,
z(0)T Hhj (x )z(0) + [hj (x )]T z(0) = 0. Restriction of the mapping HL (x ) : R n R n on subspace M?

Optimality Criteria
Sensitivity
Sensitivity Constraints
Optimality Criteria
Sensitivity
Duality* Duality*
Structure of Methods: An Overview* Structure of Methods: An Overview*
Take y M, operate HL (x ) on it, project the image back to M. Suppose original objective and constraint functions as
Restricted mapping LM : M M f (x, p), g(x, p) and h(x, p)
Question: Matrix representation for L M of size (n m) (n m)? By choosing parameters (p), we arrive at x . Call it x (p).
Select local orthonormal basis D R n(nm) for M. Question: How does f (x (p), p) depend on p?
For arbitrary z R nm , map y = Dz R n as HL y = HL Dz. Total gradients
Its component along di : dT p f (x (p), p) = p x (p)x f (x , p) + p f (x , p),

i HL Dz
p h(x (p), p) = p x (p)x h(x , p) + p h(x , p) = 0,

Hence, projection back on M:
and similarly for g(x (p), p).
LM z = DT HL Dz,
In view of x L = 0, from KKT conditions,
The (n m) (n m) matrix LM = DT HL D: the restriction!
p f (x (p), p) = p f (x , p) + [p h(x , p)] + [p g(x , p)]

Second order necessary/sufficient condition: L M p.s.d./p.d.
Sensitivity Constraints
Optimality Criteria
Sensitivity
Duality* Constraints
Optimality Criteria
Sensitivity
Duality* Duality*
Structure of Methods: An Overview* Dual problem: Structure of Methods: An Overview*
Sensitivity to constraints
Reformulation of a problem in terms of the Lagrange multipliers.
In particular, in a revised problem, with h(x) = c and g(x) d,
Suppose x as a local minimum for the problem
using p = c,
Minimize f (x) subject to h(x) = 0,
p f (x , p) = 0, p h(x , p) = I and p g(x , p) = 0.
with Lagrange multiplier (vector) .
c f (x (p), p) =

f (x ) + [h(x )] = 0
Similarly, using p = d, we get df
(x (p), p) = . If HL (x ) is positive definite (assumption of local duality), then x
Lagrange multipliers and signify costs of pulling the minimum is also a local minimum of
point in order to satisfy the constraints!
f(x) = f (x) + T h(x).
I Equality constraint: both sides infeasible, sign of j identifies
one side or the other of the hypersurface. If we vary around , the minimizer of
I Inequality constraint: one side is feasible, no cost of pulling
L(x, ) = f (x) + T h(x)
from that side, so i 0.
varies continuously with .
Duality* Optimality Criteria
Sensitivity
Sensitivity
Duality* Duality*
In the neighbourhood of , define the dual function Structure of Methods: An Overview*
Hessian of the dual function:
T
() = min L(x, ) = min[f (x) + h(x)]. H () = x()x h(x())
x x
Differentiating x L(x(), ) = 0, we have

For a pair {x, }, the dual solution is feasible if and only
if the primal solution is optimal. x()HL (x(), ) + [x h(x())]T = 0.
Define x() as the local minimizer of L(x, ). Solving for x() and substituting,
T
() = L(x(), ) = f (x()) + h(x()) H () = [x h(x())]T [HL (x(), )]1 x h(x()),
First derivative:
negative definite!
() = x()x L(x(), ) + h(x()) = h(x()) At , x( ) = x , ( ) = h(x ) = 0, H ( ) is negative
definite and the dual function is maximized.
For a pair {x, }, the dual solution is optimal if and only
if the primal solution is feasible. ( ) = L(x , ) = f (x )
Constraints
Sensitivity
Structure of Methods: An Overview* Constraints
Optimality Criteria
Sensitivity
Duality* Duality*
Consolidation (including all constraints) Structure of Methods: An Overview* For a problem of n variables, with m active Structure
constraints,
of Methods: An Overview*
I Assuming local convexity, the dual function: nature and dimension of working spaces
Penalty methods (R n ): Minimize the penalized function
(, ) = min L(x, , ) = min[f (x) + T h(x) + T g(x)].
x x
q(c, x) = f (x) + cP(x).
I Constraints on the dual: x L(x, , ) = 0, optimality of the
primal. Example: P(x) = 21 kh(x)k2 + 12 [max(0, g(x))]2 .
I Corresponding to inequality constraints of the primal problem, Primal methods (R nm ): Work only in feasible domain, restricting
non-negative variables in the dual problem. steps to the tangent plane.
I First order necessary conditons for the dual optimality: Example: Gradient projection method.
equivalent to the feasibility of the primal problem. Dual methods (R m ): Transform the problem to the space of
I The dual function is concave globally! Lagrange multipliers and maximize the dual.
Example: Augmented Lagrangian method.
I Under suitable conditions, ( ) = L(x , ) = f (x ).
Lagrange methods (R m+n ): Solve equations appearing in the KKT
I The Lagrangian L(x, , ) has a saddle point in the combined
conditions directly.
space of primal and dual variables: positive curvature along x
Example: Sequential quadratic programming.
directions and negative curvature along and directions.
Applied Mathematical Methods Constrained Optimization 277, Applied Mathematical Methods Linear and Quadratic Programming Problems* 278,
Points to note Constraints

Optimality Criteria
Sensitivity
Outline Linear Programming
Quadratic Programming
Duality*
I Constraint qualification
I KKT conditions
Linear and Quadratic Programming Problems*
I Second order conditions
Linear Programming
I Basic ideas for solution strategy Quadratic Programming
Applied Mathematical Methods Linear and Quadratic Programming Problems* 279, Applied Mathematical Methods Linear and Quadratic Programming Problems* 280,
Linear Programming Linear Programming

Quadratic Programming Linear Programming Linear Programming
Standard form of an LP problem: The simplex method

Minimize f (x) = cT x, Suppose x R N , b R M and A R MN full-rank, with M < N.
subject to Ax = b, x 0; with b 0.
IM xB + A0 xNB = b0
Preprocessing to cast a problem to the standard form
Basic and non-basic variables: xB R M and xNB R NM
I Maximization: Minimize the negative function.
Basic feasible solution: xB = b0 0 and xNB = 0
I Variables of unrestricted sign: Use two variables.
At every iteration,
I Inequality constraints: Use slack/surplus variables.
I selection of a non-basic variable to enter the basis
I Negative RHS: Multiply with 1.
I edge of travel selected based on maximum rate of descent
Geometry of an LP problem I no qualifier: current vertex is optimal
I Infinite domain: does a minimum exist? I selection of a basic variable to leave the basis
I Finite convex polytope: existence guaranteed I based on the first constraint becoming active along the edge
I Operating with vertices sufficient as a strategy
I no constraint ahead: function is unbounded
I Extension with slack/surplus variables: original solution space I elementary row operations: new basic feasible solution
a subspace in the extented space, x 0 marking the domain Two-phase method: Inclusion of a pre-processing phase with
I Essence of the non-negativity condition of variables artificial variables to develop a basic feasible solution
Linear Programming Linear Programming

Quadratic Programming Quadratic Programming Linear Programming
General perspective A quadratic objective function and linear constraints define

LP problem:
a QP problem.
Minimize f (x, y) = cT+
1x cT
2 y;
subject to A11 x + A12 y = b1 , A21 x + A22 y b2 , y 0. Equations from the KKT conditions: linear!
Lagrangian: Lagrange methods are the natural choice!
L(x, y, , , ) = cT
1 x + cT
2y With equality constraints only,
+ T (A11 x + A12 y b1 ) + T (A21 x + A22 y b2 ) T y
1 T
Optimality conditions: Minimize f (x) = x Qx + cT x, subject to Ax = b.
2
c1 + A T T
11 + A21 = 0 and = c 2 + AT T
12 + A22 0
First order necessary conditions:
Substituting back, optimal function value: f = T b1 T b2
f
Sensitivity to the constraints: f
b1 = and b2 =
Q AT x c
=
Dual problem: A 0 b
maximize (, ) = bT T
1 b2 ; Solution of this linear system yields the complete result!
subject to AT
11 + A T = c ,
21 1 AT T
12 + A22 c2 , 0.
Caution: This coefficient matrix is indefinite.
Notice the symmetry between the primal and dual problems.
Quadratic Programming Linear Programming

Quadratic Programming Quadratic Programming Linear Programming
Active set method Linear complementary problem (LCP)

Slack variable strategy with inequality constraints
Minimize f (x) = 12 xT Qx + cT x;
1 T
subject to A 1 x = b1 , Minimize 2 x Qx + cT x, subject to Ax b, x 0.
A2 x b 2 .
KKT conditions: With x, y, , 0,
Start the iterative process from a feasible point.
I Construct active set of constraints as Ax = b. Qx + c + AT = 0,
I From the current point xk , with x = xk + dk , Ax + y = b,
1 xT = T y = 0.
f (x) = (xk + dk )T Q(xk + dk ) + cT (xk + dk )
2
1 T Denoting
= d Qdk + (c + Qxk )T dk + f (xk ).
2 k
x c Q AT
I Since gk f (xk ) = c + Qxk , subsidiary quadratic program: z= ,w = ,q = and M = ,
y b A 0
minimize 12 dT T
k Qdk + gk dk subject to Adk = 0.
I Examining solution dk and Lagrange multipliers, decide to w Mz = q, wT z = 0.

terminate, proceed or revise the active set. Find mutually complementary non-negative w and z.
Quadratic Programming Linear Programming

Quadratic Programming Points to note Linear Programming
If q 0, then w = q, z = 0 is a solution!
Lemkes method: artificial variable z 0 with e = [1 1 1 1]T :
I Fundamental issues and general perspective of the linear
Iw Mz ez0 = q programming problem
I The simplex method
With z0 = max(qi ),
I Quadratic programming
w = q + ez0 0 and z = 0: basic feasible solution I The active set method
I Lemkes method via the linear complementary problem
I Evolution of the basis similar to the simplex method.
I Out of a pair of w and z variables, only one can be there in
any basis. Necessary Exercises: 1,2,3,4,5
I At every step, one variable is driven out of the basis and its
partner called in.
I The step driving out z0 flags termination.
Handling of equality constraints? Very clumsy!!
Applied Mathematical Methods Interpolation and Approximation 287, Applied Mathematical Methods Interpolation and Approximation 288,
Outline Polynomial Interpolation

Piecewise Polynomial Interpolation
Interpolation of Multivariate Functions
Polynomial Interpolation Polynomial Interpolation
A Note on Approximation of Functions A Note on Approximation of Functions
Modelling of Curves and Surfaces* Problem: To develop an analytical representation of a function
Modelling of Curves and Surfaces*
from information at discrete data points.

Purpose
I Evaluation at arbitrary points
Interpolation and Approximation
Polynomial Interpolation
I Differentiation and/or integration
Piecewise Polynomial Interpolation I Drawing conclusion regarding the trends or nature
Interpolation of Multivariate Functions Interpolation: one of the ways of function representation
A Note on Approximation of Functions I sampled data are exactly satisfied
Polynomial: a convenient class of basis functions
For yi = f (xi ) for i = 0, 1, 2, , n with x0 < x1 < x2 < < xn ,
p(x) = a0 + a1 x + a2 x 2 + + an x n .
Find the coefficients such that p(xi ) = f (xi ) for i = 0, 1, 2, , n.

Values of p(x) for x [x0 , xn ] interpolate n + 1 values
of f (x), an outside estimate is extrapolation.

Modelling of Curves and Surfaces* Lagrange interpolation Modelling of Curves and Surfaces*
To determine p(x), solve the linear system
Basis functions:
Qn
1 x0 x02 x0n a0 f (x0 ) j=0,j6=k (x xj )
1 x1 x 2 x n Lk (x) = Qn
1 1 a1 f (x1 ) j=0,j6=k (xk xj )
1 x2 x 2 x n
2 2 a2 =
f (x2 )
? (x x0 )(x x1 ) (x xk1 )(x xk+1 ) (x xn )
.. .. .. . . ..

=
. . . . . (xk x0 )(xk x1 ) (xk xk1 )(xk xk+1 ) (xk xn )
1 xn xn2 xnn an f (xn )
Interpolating polynomial:
Vandermonde matrix: invertible, but typically ill-conditioned! p(x) = 0 L0 (x) + 1 L1 (x) + 2 L2 (x) + + n Ln (x)
Invertibility means existence and uniqueness of polynomial p(x). At the data points, Lk (xi ) = ik .
Two polynomials p1 (x) and p2 (x) matching the function f (x) at Coefficient matrix identity and i = f (xi ).
x0 , x1 , x2 , , xn imply Lagrange interpolation formula:
n-th degree polynomial p(x) = p1 (x) p2 (x) with Xn
n + 1 roots! p(x) = f (xk )Lk (x) = L0 (x)f (x0 )+L1 (x)f (x1 )+ +Ln (x)f (xn )
k=0
p 0 p1 (x) = p2 (x): p(x) is unique.
Existence of p(x) is a trivial consequence!

Piecewise Polynomial Interpolation Polynomial Interpolation
Two interpolation formulae Modelling of Curves and Surfaces* Piecewise linear interpolation Modelling of Curves and Surfaces*
I one costly to determine, but easy to process f (xi ) f (xi 1 )

f (x) = f (xi 1 ) + (x xi 1 ) for x [xi 1 , xi ]
xi xi 1
I the other trivial to determine, costly to process
Handy for many uses with dense data. But, not differentiable.
Newton interpolation for an intermediate trade-off: Q
p(x) = c0 + c1 (x x0 ) + c2 (x x0 )(x x1 ) + + cn n1 Piecewise cubic interpolation
i =0 (x xi )
With function values and derivatives at (n + 1) points,
Hermite interpolation
n cubic Hermite segments
uses derivatives as well as function values.
Data: f (xi ), f 0 (xi ), , f (ni 1) (xi ) at x = xi , for i = 0, 1, , m: Data for the j-th segment:
Pm
I At (m + 1) points, a total of n + 1 =
i =0 ni conditions
f (xj1 ) = fj1 , f (xj ) = fj , f 0 (xj1 ) = fj1
0
and f 0 (xj ) = fj0
Limitations of single-polynomial interpolation Interpolating polynomial:
With large number of data points, polynomial degree is high. pj (x) = a0 + a1 x + a2 x 2 + a3 x 3

0 , f0
Coefficients a0 , a1 , a2 , a3 : linear combinations of fj1 , fj , fj1
I Computational cost and numerical imprecision j
I Lack of representative nature due to oscillations Composite function C 1 continuous at knot points.

General formulation through normalization of intervals Modelling of Curves and Surfaces* Modelling of Curves and Surfaces*
Spline interpolation
x = xj1 + t(xj xj1 ), t [0, 1]
Spline: a drafting tool to draw a smooth curve through key points.
With g (t) = f (x(t)), g 0 (t) = (xj xj1 )f 0 (x(t));
Data: fi = f (xi ), for x0 < x1 < x2 < < xn .
g0 = fj1 , g1 = fj , g00 = (xj xj1 )fj1
0
and g10 = (xj xj1 )fj0 .
If kj = f 0 (xj ), then
Cubic polynomial for the j-th segment:
pj (x) can be determined in terms of fj1 , fj , kj1 , kj
qj (t) = 0 + 1 t + 2 t 2 + 3 t 3 and pj+1 (x) in terms of fj , fj+1 , kj , kj+1 .
Modular expression: 00 (x ): a linear equation in k
Then, pj00 (xj ) = pj+1 j j1 , kj and kj+1

1 1 From n 1 interior knot points,
t
qj (t) = [0 1 2 3 ] = [g0 g1 g00 g10 ] W t = Gj WT n 1 linear equations in derivative values k 0 , k1 , , kn .
t2 t2
t3 t3 Prescribing k0 and kn , a diagonally dominant tridiagonal system!
Packaging data, interpolation type and variable terms separately!
A spline is a smooth interpolation, with C 2 continuity.
Question: How to supply derivatives? And, why?
Interpolation of Multivariate FunctionsPiecewise

Interpolation of Multivariate FunctionsPolynomial Interpolation
Piecewise bilinear interpolation Modelling of Curves and Surfaces* Alternative local formula through reparametrization
xx y y
With u = xi xii1 and v = yj yjj1 , denoting
Data: f (x, y ) over a dense rectangular grid 1 1
x = x0 , x1 , x2 , , xm and y = y0 , y1 , y2 , , yn fi 1,j1 = g0,0 , fi ,j1 = g1,0 , fi 1,j = g0,1 and fi ,j = g1,1 ;
Rectangular domain: {(x, y ) : x0 x xm , y0 y yn } bilinear interpolation:

For xi 1 x xi and yj1 y yj , 0,0 0,1 1
g (u, v ) = [1 u] for u, v [0, 1].
1,0 1,1 v
a0,0 a0,1 1
f (x, y ) = a0,0 + a1,0 x + a0,1 y + a1,1 xy = [1 x] Values at four corner points fix the coefficient matrix as
a1,0 a1,1 y

0,0 0,1 1 0 g0,0 g0,1 1 1
With data at four corner points, coefficient matrix determined from = .
1,0 1,1 1 1 g1,0 g1,1 0 1

1 xi 1 a0,0 a0,1 1 1 fi 1,j1 fi 1,j
= . Concisely, g (u, v ) = UT WT Gi ,j WV in which
1 xi a1,0 a1,1 yj1 yj fi ,j1 fi ,j

1 1 1 1 fi 1,j1 fi 1,j
U= ,V= ,W= , Gi ,j = .
Approximation only C0 continuous. u v 0 1 fi ,j1 fi ,j
Interpolation of Multivariate FunctionsPiecewise

A Note on Approximation of Functions
Piecewise bicubic interpolation Modelling of Curves and Surfaces* Modelling of Curves and Surfaces*
f f
Data: f , x 2f
, y and xy over grid points A common strategy of function approximation is to
With normalizing parameters u and v , I express a function as a linear combination of a set of basis
functions (which?), and
g f g f
u = (xi xi 1 ) x , v = (yj yj1 ) y , and I determine coefficients based on some criteria (what?).
2g 2f
uv = (xi xi 1 )(yj yj1 ) xy
Criteria:
In {(x, y ) : xi 1 x xi , yj1 y yj } or {(u, v ) : u, v [0, 1]}, Interpolatory approximation: Exact agreement with sampled data
Least square approximation: Minimization of a sum (or integral) of
g (u, v ) = UT WT Gi ,j WV, square errors over sampled data
with U = [1 u u 2 u 3 ]T , V = [1 v v 2 v 3 ]T , and Minimax approximation: Limiting the largest deviation

g (0, 0) g (0, 1) gv (0, 0) gv (0, 1) Basis functions:
g (1, 0) g (1, 1) gv (1, 0) gv (1, 1)

Gi ,j = . polynomials, sinusoids, orthogonal eigenfunctions or
gu (0, 0) gu (0, 1) guv (0, 0) guv (0, 1) field-specific heuristic choice
gu (1, 0) gu (1, 1) guv (1, 0) guv (1, 1)
Applied Mathematical Methods Interpolation and Approximation 299, Applied Mathematical Methods Basic Methods of Numerical Integration 300,
Points to note Polynomial Interpolation

Outline Newton-Cotes Integration Formulae
Richardson Extrapolation and Romberg Integration
Further Issues
A Note on Approximation of Functions
I Lagrange, Newton and Hermite interpolations

I Piecewise polynomial functions and splines Basic Methods of Numerical Integration
I Bilinear and bicubic interpolation of bivariate functions Newton-Cotes Integration Formulae
Direct extension to vector functions: curves and surfaces! Richardson Extrapolation and Romberg Integration
Further Issues

Applied Mathematical Methods Basic Methods of Numerical Integration 301, Applied Mathematical Methods Basic Methods of Numerical Integration 302,
Newton-Cotes Integration Formulae Newton-Cotes Integration Formulae

Further Issues
Further Issues
Z Mid-point rule
b x +x
J= f (x)dx Selecting xi as xi = i 12 i ,
a Z xi Z b n
X
Divide [a, b] into n sub-intervals with f (x)dx hf (xi ) and f (x)dx h f (xi ).
xi 1 a i =1
a = x0 < x1 < x2 < < xn1 < xn = b, Error analysis: From Taylors series of f (x) about x i ,
Z xi Z xi
where xi xi 1 = h = ba
n .
(x xi )2
f (x)dx = f (xi ) + f 0 (xi )(x xi ) + f 00 (xi ) + dx
xi 1 xi 1 2
n
X
J = hf (xi ) = h[f (x1 ) + f (x2 ) + + f (xn )] h3 00 h5 iv
= hf (xi ) + f (xi ) + f (xi ) + ,
i =1 24 1920
Taking xi [xi 1 , xi ] as xi 1 and xi , we get summations J1 and J2 . third order accurate!
Over the entire domain [a, b],
As n (i.e. h 0), if J1 and J2 approach the same
Z b Xn n n
limit, then function f (x) is integrable over interval [a, b]. h3 X 00 X h2
f (x)dx h f (xi )+ f (xi ) = h f (xi )+ (ba)f 00 (),
a 24 24
A rectangular rule or a one-point rule i =1 i =1 i =1
Question: Which point to take as xi ? for [a, b] (from mean value theorem): second order accurate.

Further Issues
Further Issues
Trapezoidal rule Error estimate of trapezoidal rule

Approximating function f (x) with a linear interpolation, Z xi
Z xi h h3 h5 iv
h f (x)dx = [f (xi 1 ) + f (xi )] f 00 (xi ) f (xi ) +
f (x)dx [f (xi 1 ) + f (xi )] xi 1 2 12 480
xi 1 2
and Over an extended domain,
Z " n1
#
b
1 X 1 Z b " n1
#
f (x)dx h f (x0 ) + f (xi ) + f (xn ) . 1 X h2
a 2 2 f (x)dx = h {f (x0 ) + f (xn )} + f (xi ) (ba)f 00 ()+ .
i =1
a 2 12
i =1
Taylor series expansions about the mid-point:
h h2 00 h3 000 h4 iv The same order of accuracy as the mid-point rule!
f (xi 1 ) = f (xi ) f 0 (xi ) + f (xi ) f (xi ) + f (xi )
2 8 48 384 Different sources of merit
h 0 h2 00 h3 000 h4 iv
f (xi ) = f (xi ) + f (xi ) + f (xi ) + f (xi ) + f (xi ) + I Mid-point rule: Use of mid-point leads to symmetric
2 8 48 384
error-cancellation.
h h3 h5
[f (xi 1 ) + f (xi )] = hf (xi ) + f 00 (xi ) + f iv (xi ) + I Trapezoidal rule: Use of end-points allows double utilization
2 8 384
R xi of boundary points in adjacent intervals.
h3 00 h5 iv
Recall xi 1 f (x)dx = hf (xi ) + 24 f (xi ) + 1920 f (xi ) + .
How to use both the merits?

Further Issues
Newton-Cotes Integration Formulae
Further Issues
Simpsons rules To determine quantity F

Divide [a, b] into an even number (n = 2m) of intervals. I using a step size h, estimate F (h)
Fit a quadratic polynomial over a panel of two intervals. I error terms: h p , h q , h r etc (p < q < r )
For this panel of length 2h, two estimates: I F = lim0 F ()?
I plot F (h), F (h), F (2 h) (with < 1) and extrapolate?
M(f ) = 2hf (xi ) and T (f ) = h[f (xi 1 ) + f (xi +1 )]
1 F (h) = F + chp + O(hq )
h3 h5
J = M(f ) + f 00 (xi ) + f iv (xi ) + 2 F (h) = F + c(h)p + O(hq )
3 60
2h3 00 h5 4 F (2 h) = F + c(2 h)p + O(hq )
J = T (f ) f (xi ) f iv (xi ) +
3 15 Eliminate c and determine (better estimates of) F :
Simpsons one-third rule (with error estimate): F (h) p F (h)
Z xi +1 3 F1 (h) = = F + c1 hq + O(hr )
h h5 1 p
f (x)dx = [f (xi 1 ) + 4f (xi ) + f (xi +1 )] f iv (xi ) F (2 h) p F (h)
xi 1 3 90 5 F1 (h) = = F + c1 (h)q + O(hr )
1 p
Fifth (not fourth) order accurate! F1 (h)q F1 (h)
Still better estimate: 6 F2 (h) = 1q = F + O(hr )
A four-point rule: Simpsons three-eighth rule
Still higher order rules NOT advisable! Richardson extrapolation

Newton-Cotes Integration Formulae
Richardson Extrapolation and Romberg Integration Further Issues Newton-Cotes Integration Formulae
Rb
Further Issues Further Issues
Trapezoidal rule for J = a f (x)dx: p = 2, q = 4, r = 6 etc

Featured functions: adaptive quadrature
T (f ) = J + ch 2 + dh4 + eh6 + (xi xi 1 )
I With prescribed tolerance , assign quota i = ba of
With = 21 , half the sum available for successive levels. error to every interval [xi 1 , xi ].
I For each interval, find two estimates of the integral and
Romberg integration estimate the error.
I Trapezoidal rule with h = H: find J 11 . I If error estimate is not within quota, then subdivide.
I With h = H/2, find J12 .
2 Function as tabulated data
J12 21 J11 4J12 J11
J22 = 2 = . Only trapezoidal rule applicable?
1 1 3 I
2
I Fit a spline over data points and integrate the segments?
I If |J22 J12 | is within tolerance, STOP. Accept J J 22 .
I With h = H/4, find J13 .
Improper integral: Newton-Cotes closed formulae not applicable!
4
4J13 J12 J23 12 J22 16J23 J22 I Open Newton-Cotes formulae
J23 = and J33 = 4 = .
3 1 1 15 I Gaussian quadrature
2
I If |J33 J23 | is within tolerance, STOP with J J 33 .
Applied Mathematical Methods Basic Methods of Numerical Integration 309, Applied Mathematical Methods Advanced Topics in Numerical Integration* 310,
Points to note Newton-Cotes Integration Formulae

Further Issues
Outline Gaussian Quadrature
Multiple Integrals
I Definition of an integral and integrability

I Closed Newton-Cotes formulae and their error estimates
I Richardson extrapolation as a general technique Advanced Topics in Numerical Integration*
Gaussian Quadrature
I Romberg integration
Multiple Integrals
I Adaptive quadrature
Applied Mathematical Methods Advanced Topics in Numerical Integration* 311, Applied Mathematical Methods Advanced Topics in Numerical Integration* 312,
Gaussian Quadrature Gaussian Quadrature

Multiple Integrals Gaussian Quadrature Gaussian Quadrature
Multiple Integrals
P
A typical quadrature formula: a weighted sum ni=0 wi fi Gauss-Legendre quadrature
I fi : function value at i-th sampled point Z 1
I wi : corresponding weight f (x)dx = w1 f (x1 ) + w2 f (x2 )
1
Newton-Cotes formulae:
Four variables: Insist that it is exact for 1, x, x 2 and x 3 .
I Abscissas (xi s) of sampling prescribed
I Coefficients or weight values determined to eliminate Z 1

dominant error terms w1 + w 2 = dx = 2,
1
Gaussian quadrature rules: Z 1
I no prescription of quadrature points w1 x1 + w 2 x2 = xdx = 0,
1
I only the number of quadrature points prescribed Z 1
2
I locations as well as weights contribute to the accuracy criteria w1 x12 + w2 x22 = x 2 dx =
1 3
I with n integration points, 2n degrees of freedom Z 1
I can be made exact for polynomials of degree up to 2n 1 and w1 x13 + w2 x23 = x 3 dx = 0.
1
I best locations: interior points
I open quadrature rules: can handle integrable singularities x1 = x2 , w1 = w2 w1 = w2 = 1, x1 = 13 , x2 = 1

3

Multiple Integrals
Two-point Gauss-Legendre quadrature formula General Framework for n-point formula

R1
1 1
1 f (x)dx = f ( 3 ) + f ( 3 ) f (x): a polynomial of degree 2n 1
Exact for any cubic polynomial: parallels Simpsons rule! p(x): Lagrange polynomial through the n quadrature points
Three-point quadrature rule along similar lines:
f (x) p(x): a (2n 1)-degree polynomial having n of its roots at
Z 1 r ! r ! the quadrature points
5 3 8 5 3
f (x)dx = f + f (0) + f
1 9 5 9 9 5 Then, with (x) = (x x1 )(x x2 ) (x xn ),
A large number of formulae: Consult mathematical handbooks. f (x) p(x) = (x)q(x).

For domain of integration [a, b], P
Quotient polynomial: q(x) = in1=0 i x
i
a+b ba ba
x= + t and dx = dt Direct integration:
2 2 2
With scaling and relocation, Z 1 Z Z "
1 1 n1
X
#
Z b Z f (x)dx = p(x)dx + (x) i x i dx

ba 1 1 1 1 i =0
f (x)dx = f [x(t)]dt
a 2 1
How to make the second term vanish?

Multiple Integrals
Choose quadrature points x1 , x2 , , xn so that (x) is orthogonal Weight functions in Gaussian quadrature
to all polynomials of degree less than n.
Legendre polynomial What is so great about exact integration of polynomials?
Demand something else: generalization
Gauss-Legendre quadrature
Exact integration of polynomials times function W (x)
1. Choose Pn (x), Legendre polynomial of degree n, as (x).
2. Take its roots x1 , x2 , , xn as the quadrature points. Given weight function W (x) and number (n) of quadrature points,
3. Fit Lagrange polynomial of f (x), using these n points. work out the locations (xj s) of the n points and the
corresponding weights (wj s), so that integral
p(x) = L1 (x)f (x1 ) + L2 (x)f (x2 ) + + Ln (x)f (xn )
Z b n
X
4. Z Z Z W (x)f (x)dx = wj f (xj )
1 1 n
X 1 a j=1
f (x)dx = p(x)dx = f (xj ) Lj (x)dx
1 1 1
j=1 is exact for an arbitrary polynomial f (x) of degree up to
R1 (2n 1).
Weight values: wj = 1 Lj (x)dx, for j = 1, 2, , n

Multiple Integrals Multiple Integrals Gaussian Quadrature
Multiple Integrals
Z b Z g2 (x)
A family of orthogonal polynomials with increasing degree:
S= f (x, y ) dy dx
quadrature points: roots of n-th member of the family. a g1 (x)
Z g2 (x) Z b
F (x) = f (x, y ) dy and S = F (x)dx
For different kinds of functions and different domains, g1 (x) a
I Gauss-Chebyshev quadrature with complete flexibility of individual quadrature methods.
I Gauss-Laguerre quadrature
Double integral on rectangular domain
I Gauss-Hermite quadrature
Two-dimensional version of Simpsons one-third rule:
I
Z 1Z 1
Several singular functions and infinite domains can be handled. f (x, y )dxdy
1 1
A very special case: = w0 f (0, 0) + w1 [f (1, 0) + f (1, 0) + f (0, 1) + f (0, 1)]
For W (x) = 1, Gauss-Legendre quadrature! + w2 [f (1, 1) + f (1, 1) + f (1, 1) + f (1, 1)]
Exact for bicubic functions: w0 = 16/9, w1 = 4/9 and w2 = 1/9.

Multiple Integrals Gaussian Quadrature

Multiple Integrals Points to note Gaussian Quadrature
Multiple Integrals
Monte Carlo integration

Z
I = f (x)dV
I Basic strategy of Gauss-Legendre quadrature
Requirements: I Formulation of a double integral from fundamental principle
I a simple volume V enclosing the domain I Monte Carlo integration
I a point classification scheme
Generating random points in V ,

f (x) if x , Necessary Exercises: 2,5,6
F (x) =
0 otherwise .
N
V X
I F (xi )
N
i =1
Estimate of I (usually) improves with increasing N.
Applied Mathematical Methods Numerical Solution of Ordinary Differential Equations 321, Applied Mathematical Methods Numerical Solution of Ordinary Differential Equations 322,
Outline Single-Step Methods

Practical Implementation of Single-Step Methods
Systems of ODEs
Single-Step Methods Single-Step Methods
Systems of ODEs
Multi-Step Methods* Multi-Step Methods*
Initial value problem (IVP) of a first order ODE:
dy
= f (x, y ), y (x0 ) = y0
Numerical Solution of Ordinary Differential Equations dx
Single-Step Methods To determine: y (x) for x [a, b] with x 0 = a.
Systems of ODEs Numerical solution: Start from the point (x 0 , y0 ).
Multi-Step Methods*
I y1 = y (x1 ) = y (x0 + h) =?
I Found (x1 , y1 ). Repeat up to x = b.
Information at how many points are used at every step?

I Single-step method: Only the current value
I Multi-step method: History of several recent steps

Systems of ODEs
Systems of ODEs
Eulers method Initial slope for the entire step: is it a good idea?
dy
I At (xn , yn ), evaluate slope dx = f (xn , yn ).
y y
I For a small step h, C C
C1
y3 Q* Q2
C2
y3
yn+1 = yn + hf (xn , yn ) C3
Q C1
Q1
y0 y0 P
Repitition of such steps constructs y (x).
P1
First order truncated Taylors series: O x0 x1 x2 x3 x O x0 x1 x
Expected error: O(h 2 )

Figure: Eulers method Figure: Improved Eulers method
Accumulation over steps
Total error: O(h)
Improved Eulers method or Heuns method
Eulers method is a first order method.
yn+1 = yn + hf (xn , yn )
Question: Total error = Sum of errors over the steps? yn+1 = yn + h2 [f (xn , yn ) + f (xn+1 , yn+1 )]
Answer: No, in general.
The order of Heuns method is two.

Systems of ODEs
Systems of ODEs
Runge-Kutta methods With continuous choice of w2 ,
Second order method: a family of second order Runge Kutta (RK2) formulae
k1 = hf (xn , yn ), k2 = hf (xn + h, yn + k1 )
Popular form of RK2: with choice w2 = 1,
k = w 1 k1 + w 2 k2 ,
and xn+1 = xn + h, yn+1 = yn + k k1 = hf (xn , yn ), k2 = hf (xn + h2 , yn + k1
2)
xn+1 = xn + h, yn+1 = yn + k2
Force agreement up to the second order.
Fourth order Runge-Kutta method (RK4):
yn+1
= yn + w1 hf (xn , yn ) + w2 h[f (xn , yn ) + hfx (xn , yn ) + k1 fy (xn , yn ) + ] k1 = hf (xn , yn )
= yn + (w1 + w2 )hf (xn , yn ) + h2 w2 [fx (xn , yn ) + f (xn , yn )fy (xn , yn )] + k2 = hf (xn + h2 , yn + k21 )
k3 = hf (xn + h2 , yn + k22 )
From Taylors series, using y 0 = f (x, y ) and y 00 = fx + ffy , k4 = hf (xn + h, yn + k3 )
h2
y (xn+1 ) = yn + hf (xn , yn ) + [fx (xn , yn ) + f (xn , yn )fy (xn , yn )] + k = 61 (k1 + 2k2 + 2k3 + k4 )
2
1 1 xn+1 = xn + h, yn+1 = yn + k
w1 + w2 = 1, w2 = w2 = 2 == 2w2 , w1 = 1 w 2

Single-Step Methods
Systems of ODEs
Single-Step Methods
Systems of ODEs
Question: How to decide whether the error is within tolerance? Evaluation of a step:
Additional estimates:
> : Step size is too large for accuracy.
I handle to monitor the error
Subdivide the interval.
I further efficient algorithms
<< : Step size is inefficient!
Runge-Kutta method with adaptive step size
In an interval [xn , xn + h], Start with a large step size.
(1) Keep subdividing intervals whenever > .
yn+1 = yn+1 + ch5 + higher order terms
Fast marching over smooth segments and small steps in
Over two steps of size h2 , zones featured with rapid changes in y (x).
5
(2) h
yn+1 = yn+1 + 2c + higher order terms Runge-Kutta-Fehlberg method
2
With six function values,
Difference of two estimates:
An RK4 formula embedded in an RK5 formula
(1) (2) 15 5
= yn+1 yn+1 ch I two independent estimates and an error estimate!
16
(2)
(2)
16yn+1 yn+1
(1) RKF45 in professional implementations
Best available value: yn+1 = yn+1 15 = 15
Systems of ODEs Single-Step Methods

Systems of ODEs
Systems of ODEs Single-Step Methods
Systems of ODEs
Methods for a single first order ODE State space formulation is directly applicable when
directly applicable to a first order vector ODE the highest order derivatives can be solved explicitly.
A typical IVP with an ODE system: The resulting form of the ODEs: normal system of ODEs
dy Example:
= f(x, y), y(x0 ) = y0
dx r 22
d 2x dy dxdx d y
An n-th order ODE: convert into a system of first order ODEs y 3 + 2x
+4 = 0
dt 2 dt dtdt dt 2
Defining state vector z(x) = [y (x) y 0 (x) y (n1) (x)]T , 2 3/2
d 3y d y
work out dz
to form the state space equation. e xy 3 y + 2x + 1 = e t
dx dt dt 2
h iT
Initial condition: z(x0 ) = [y (x0 ) y 0 (x0 ) y (n1) (x0 )]T State vector: z(t) = x dx y dy d2y
dt dt dt 2
A system of higher order ODEs with the highest order derivatives With three trivial derivatives z10 (t) = z2 , z30 (t) = z4 and z40 (t) = z5
of orders n1 , n2 , n3 , , nk and the other two obtained from the given ODEs,
I Cast into the state space form with the state vector of we get the state space equations as dz
= f(t, z).
dt
dimension n = n1 + n2 + n3 + + nk
Multi-Step Methods* Single-Step Methods

Systems of ODEs
Points to note Single-Step Methods
Systems of ODEs
Single-step methods: every step a brand new IVP!

Why not try to capture the trend?
A typical multi-step formula: I Eulers and Runge-Kutta methods
I Step size adaptation
yn+1 = yn + h[c0 f (xn+1 , yn+1 ) + c1 f (xn , yn ) I State space formulation of dynamic systems
+ c2 f (xn1 , yn1 ) + c3 f (xn2 , yn2 ) + ]
Determine coefficients by demanding the exactness for leading

polynomial terms.
Explicit methods: c0 = 0, evaluation easy, but involves
extrapolation.
Implicit methods: c0 6= 0, difficult to evaluate, but better stability.
Predictor-corrector methods
Example: Adams-Bashforth-Moulton method
Applied Mathematical Methods ODE Solutions: Advanced Issues 333, Applied Mathematical Methods ODE Solutions: Advanced Issues 334,
Outline Stability Analysis

Implicit Methods
Stiff Differential Equations
Stability Analysis Stability Analysis
Implicit Methods
Boundary Value Problems Boundary Value Problems
Adaptive RK4 is an extremely successful method.
But, its scope has a limitation.
Focus of explicit methods (such as RK) is accuracy and efficiency.
ODE Solutions: Advanced Issues The issue of stabilty is handled indirectly.
Stability Analysis
Implicit Methods Stabilty of explicit methods
Stiff Differential Equations For the ODE system y0 = f(x, y), Eulers method gives
Boundary Value Problems yn+1 = yn + f(xn , yn )h + O(h2 ).
Taylors series of the actual solution:
y(xn+1 ) = y(xn ) + f(xn , y(xn ))h + O(h2 )
Discrepancy or error:
n+1 = yn+1 y(xn+1 )
= [yn y(xn )] + [f(xn , yn ) f(xn , y(xn ))]h + O(h 2 )

f
= n + (xn , yn )n h + O(h2 ) (I + hJ)n
y

Implicit Methods
Implicit Methods
Eulers step magnifies the error by a factor (I + hJ). 3
Using J loosely as the representative Jacobian, 2

UNSTABLE
UNSTABLE
n
n+1 (I + hJ) 1 . 1
RK2
Euler
For stability, n+1 0 as n .

Im(h)
0
O
Eigenvalues of (I + hJ) must fall within the unit circle 1

RK4
|z| = 1. By shift theorem, eigenvalues of hJ must fall 2
inside the unit circle with the centre at z 0 = 1.

3
5 4 3 2 1 0 1 2 3
2Re () Re(h)
|1 + h| < 1 h <
||2 Figure: Stability regions of explicit methods
Note: Same result for single ODE w 0 = w , with complex .
For second order Runge-Kutta method, Question: What do these stability regions mean with reference to

h 2 2 the system eigenvalues?
n+1 = 1 + h + n Question: How does the step size adaptation of RK4 operate on a
2
system with eigenvalues on the left half of complex plane?
2
Region of stability in the plane of z = h: 1 + z + z2 < 1 Step size adaptation tackles instability by its symptom!
Implicit Methods Stability Analysis

Implicit Methods
Implicit Methods Stability Analysis
Implicit Methods
Backward Eulers method 2
1.5
yn+1 = yn + f(xn+1 , yn+1 )h

1 STABLE
STABLE
0.5 UNSTABLE
Solve it? Is it worth solving?
Im(h)
0
O
0.5
n+1 yn+1 y(xn+1 ) STABLE
1.5
= [yn y(xn )] + h[f(xn+1 , yn+1 ) f(xn+1 , y(xn+1 ))] 2

1.5 1 0.5 0 0.5 1 1.5 2 2.5 3 3.5
Re(h)
= n + hJ(xn+1 , yn+1 )n+1

Notice the flip in the form of this equation. Figure: Stability region of backward Eulers method
How to solve g(yn+1 ) = yn + hf(xn+1 , yn+1 ) yn+1 = 0 for yn+1 ?

n+1 (I hJ)1 n
Typical Newtons iteration:
Stability: eigenvalues of (I hJ) outside the unit circle |z| = 1 h i
(k+1) (k) (k) (k)
yn+1 = yn+1 + (I hJ)1 yn yn+1 + hf xn+1 , yn+1
2Re ()
|h 1| > 1 h > Semi-implicit Eulers method for local solution:
||2
Absolute stability for a stable ODE, i.e. one with Re () < 0 yn+1 = yn + h(I hJ)1 f(xn+1 , yn )
Stiff Differential Equations Stability Analysis

Implicit Methods
Stiff Differential Equations Stability Analysis
Implicit Methods
Boundary Value Problems e 2t e 300t Boundary Value Problems
Example: IVP of a mass-spring-damper system: (c) c = 302, k = 600: x = 298
x + c x + kx = 0, x(0) = 0, x(0) = 1 4
x 10
3
4
x 10
3
(a) c = 3, k = 2: x = e t e 2t 3 3
(b) c = 49, k = 600: x = e 24t e 25t 2 2
1 1
1 1
0 0
x
x
0.8 0.8
1 1
0.6 0.6
2 2
0.4 0.4
0.2 0.2 3 3
0 0
x
4 4
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
t t
0.2 0.2
0.4 0.4
(c) With RK4 (d) With implicit Euler
0.6 0.6
0.8 0.8
1 1
Figure: Solutions of a mass-spring-damper system: stiff situation

0 0.5 1 1.5 2 2.5 3 3.5 4 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
t t
(a) Case of c = 3, k = 2 (b) Case of c = 49, k = 600

To solve stiff ODE systems,
Figure: Solutions of a mass-spring-damper system: ordinary situations use implicit method, preferably with explicit Jacobian.
Boundary Value Problems Stability Analysis

Implicit Methods
Implicit Methods
A paradigm shift from the initial value problems Shooting method

I A ball is thrown with a particular velocity. What trajectory follows the strategy to adjust trials to hit a target.
does the ball follow?
Consider the 2-point BVP
I How to throw a ball such that it hits a particular window at a
neighbouring house after 15 seconds? y0 = f(x, y), g1 (y(a)) = 0, g2 (y(b)) = 0,
Two-point BVP in ODEs: where g1 R n1 , g2 R n2 and n1 + n2 = n.

boundary conditions at two values of the independent
variable I Parametrize initial state: y(a) = h(p) with p R n2 .
I Guess n2 values of p to define IVP
Methods of solution
y0 = f(x, y), y(a) = h(p).
I Shooting method
I Finite difference (relaxation) method I Solve this IVP for [a, b] and evaluate y(b).
I Finite element method I Define error vector E(p) = g2 (y(b)).

Implicit Methods
Implicit Methods
Finite difference (relaxation) method
Objective: To solve E(p) = 0 adopts a global perspective.
E
From current vector p, n2 perturbations as p + ei : Jacobian p 1. Discretize domain [a, b]: grid of points
Each Newtons step: solution of n2 + 1 initial value a = x0 < x1 < x2 < < xN1 < xN = b.
problems! Function values y(xi ): n(N + 1) unknowns
2. Replace the ODE over intervals by finite difference equations.
Considering mid-points, a typical (vector) FDE:
I Computational cost
xi + xi 1 yi + yi 1
I Convergence not guaranteed (initial guess important) yi yi 1 hf , = 0, for i = 1, 2, 3, , N
2 2
nN (scalar) equations
Merits of shooting method
3. Assemble additional n equations from boundary conditions.
I Very few parameters to start
4. Starting from a guess solution over the grid, solve this system.
I In many cases, it is found quite efficient.
(Sparse Jacobian is an advantage.)
Iterative schemes for solution of systems of linear equations.
Applied Mathematical Methods ODE Solutions: Advanced Issues 345, Applied Mathematical Methods Existence and Uniqueness Theory 346,
Points to note Stability Analysis

Implicit Methods
Outline Well-Posedness of Initial Value Problems
Uniqueness Theorems
Extension to ODE Systems
Boundary Value Problems Closure
I Numerical stability of ODE solution methods

I Computational cost versus better stability of implicit methods
I Multiscale responses leading to stiffness: failure of explicit Existence and Uniqueness Theory
methods Well-Posedness of Initial Value Problems
Uniqueness Theorems
I Implicit methods for stiff systems
I Shooting method for two-point boundary value problems Closure
I Relaxation method for boundary value problems
Applied Mathematical Methods Existence and Uniqueness Theory 347, Applied Mathematical Methods Existence and Uniqueness Theory 348,
Well-Posedness of Initial Value Problems

Uniqueness Theorems
Uniqueness Theorems
Pierre Simon de Laplace (1749-1827):Closure
Initial value problem Closure
We may regard the present state of the

y 0 = f (x, y ), y (x0 ) = y0
universe as the effect of its past and the
From (x, y ), the trajectory develops according to y 0 = f (x, y ).
cause of its future. An intellect which at a
The new point: (x + x, y + f (x, y )x)
certain moment would know all forces that The slope now: f (x + x, y + f (x, y )x)
set nature in motion, and all positions of all
Question: Was the old direction of approach valid?
items of which nature is composed, if this
With x 0, directions appropriate, if
intellect were also vast enough to submit
these data to analysis, it would embrace in a lim f (x, y ) = f (x, y (x)),
xx
single formula the movements of the greatest
i.e. if f (x, y ) is continuous.
bodies of the universe and those of the If f (x, y ) = , then y 0 = and trajectory is vertical.
tiniest atom; for such an intellect nothing For the same value of x, several values of y !
would be uncertain and the future just like
y (x) not a function, unless f (x, y ) 6= , i.e. f (x, y ) is bounded.
the past would be present before its eyes.

Uniqueness Theorems
Uniqueness Theorems
Peanos theorem: If f (x, y ) is continuous Closure
and bounded in a Example: Closure
rectangle R = {(x, y ) : |x x0 | < h, |y y0 | < k}, with y 1

0
y = , y (0) = 1
|f (x, y )| M < , then the IVP y 0 = f (x, y ), y (x0 ) = y0 has a x
solution y (x) defined in a neighbourhood of x 0 . Function f (x, y ) = y 1
undefined at (0, 1).
x

y y

(x0,y0)
y +k
(x 0 ,y0 ) Premises of existence theorem not satisfied.
y +k

0

k
0
k

Mh

Mh
But, premises here are sufficient, not necessary!

y y

0 0

k
k
Mh
Mh
Result inconclusive.

y k
y0k

0

The IVP has solutions: y (x) = 1 + cx for all values of c.

h h h h

The solution is not unique.

O x 0 h x0 x 0+h x O x 0h x0 x0+h x
(a) Mh <= k (b) Mh >= k Example: y 02 = |y |, y (0) = 0
Figure: Regions containing the trajectories

Existence theorem guarantees a solution.
But, there are two solutions:
Guaranteed neighbourhood:
y (x) = 0 and y (x) = sgn(x) x 2 /4.
k
[x0 , x0 + ], where = min(h, M) >0

Uniqueness Theorems
Uniqueness Theorems
Closure Closure
Physical system to mathematical model Continuous dependence on initial condition

I Mathematical solution
Suppose that for IVP y 0 = f (x, y ), y (x0 ) = y0 ,
I Interpretation about the physical system
I unique solution: y1 (x).
Meanings of non-uniqueness of a solution Applying a small perturbation to the initial condition, the new IVP:
y 0 = f (x, y ), y (x0 ) = y0 +
I Mathematical model admits of extraneous solution(s)?
I unique solution: y2 (x)
I Physical system itself can exhibit alternative behaviours?
Question: By how much y2 (x) differs from y1 (x) for x > x0 ?
Indeterminacy of the solution Large difference: solution sensitive to initial condition
I Mathematical model of the system is not complete. I Practically unreliable solution
The initial value problem is not well-posed.
Well-posed IVP:
After existence, next important question: An initial value problem is said to be well-posed if there
exists a solution to it, the solution is unique and it
Uniqueness of a solution depends continuously on the initial conditions.
Uniqueness Theorems Well-Posedness of Initial Value Problems

Uniqueness Theorems
Uniqueness Theorems
Closure Closure
Lipschitz condition:
Z
|f (x, y ) f (x, z)| L|y z| E 0 (x) x
E 0 (x)
2L dx 2L(x x0 )
L: finite positive constant (Lipschitz constant) E (x) x0 E (x)
Theorem: If f (x, y ) is a continuous function satisfying a Integrating, E (x) E (x0 )e 2L(xx0 ) .

Lipschitz condition on a strip
Hence,
S = {(x, y ) : a < x < b, < y < }, then for any
|y1 (x) y2 (x)| e L(xx0 ) |(y1 )0 (y2 )0 |.
point (x0 , y0 ) S, the initial value problem of
y 0 = f (x, y ), y (x0 ) = y0 is well-posed.
Since x [a, b], e L(xx0 ) is finite.
Assume y1 (x) and y2 (x): solutions of the ODE y 0 = f (x, y ) with
initial conditions y (x0 ) = (y1 )0 and y (x0 ) = (y2 )0 |(y1 )0 (y2 )0 | = |y1 (x) y2 (x)| e L(xx0 )
Consider E (x) = [y1 (x) y2 (x)]2 .
continuous dependence of the solution on initial condition
E 0 (x) = 2(y1 y2 )(y10 y20 ) = 2(y1 y2 )[f (x, y1 ) f (x, y2 )]
Applying Lipschitz condition, In particular, (y1 )0 = (y2 )0 = y0 y1 (x) = y2 (x) x [a, b].
0 2 The initial value problem is well-posed.
|E (x)| 2L(y1 y2 ) = 2LE (x).
Need to consider the case of E 0 (x) 0 only.

Uniqueness Theorems
Extension to ODE Systems Well-Posedness of Initial Value Problems
Uniqueness Theorems
Closure Closure
A weaker theorem (hypotheses are stronger):
f
For ODE System
Picards theorem: If f (x, y ) and y are continuous and
bounded on a rectangle dy
= f(x, y), y(x0 ) = y0
R = {(x, y ) : a < x < b, c < y < d}, then for every dx
(x0 , y0 ) R, the IVP y 0 = f (x, y ), y (x0 ) = y0 has a
unique solution in some neighbourhood |x x 0 | h.
I Lipschitz condition:
From the mean value theorem,
kf(x, y) f(x, z)k Lky zk
f
()(y1 y2 ).
f (x, y1 ) f (x, y2 ) =
y I Scalar function E (x) generalized as

f E (x) = ky1 (x) y2 (x)k2 = (y1 y2 )T (y1 y2 )
With Lipschitz constant L = sup y ,
f f
Lipschitz condition is satisfied lavishly ! I Partial derivative y replaced by the Jacobian A = y
Note: All these theorems give only sufficient conditions!

I Boundedness to be inferred from the boundedness of its norm
Hypotheses of Picards theorem Lipschitz condition With these generalizations, the formulations work as usual.
Well-posedness Existence and uniqueness
Extension to ODE Systems Well-Posedness of Initial Value Problems

Uniqueness Theorems
Closure Well-Posedness of Initial Value Problems
Uniqueness Theorems
Closure Closure
IVP of linear first order ODE system
A practical by-product of existence and uniqueness results:
y0 = A(x)y + g(x), y(x0 ) = y0 I important results concerning the solutions
Rate function: f(x, y) = A(x)y + g(x)

A sizeable segment of current research: ill-posed problems
Continuity and boundedness of the coefficient functions I Dynamics of some nonlinear systems
in A(x) and g(x) are sufficient for well-posedness. I Chaos: sensitive dependence on initial conditions
An n-th order linear ordinary differential equation
For boundary value problems,
y (n) +P1 (x)y (n1) +P2 (x)y (n2) + +Pn1 (x)y 0 +Pn (x)y = R(x) No general criteria for existence and uniqueness
State vector: z = [y y0 y 00
y (n1) ]T
With z10 = z2 , z20 = z3 , , zn1
0 = zn and zn0 from the ODE, Note: Taking clue from the shooting method, a BVP in ODEs
I state space equation in the form z0 = A(x)z + g(x) can be visualized as a complicated root-finding problem!
Continuity and boundedness of P1 (x), P2 (x), , Pn (x) Multiple solutions or non-existence of solution is no surprise.
and R(x) guarantees well-posedness.
Applied Mathematical Methods Existence and Uniqueness Theory 359, Applied Mathematical Methods First Order Ordinary Differential Equations 360,
Points to note Well-Posedness of Initial Value Problems

Uniqueness Theorems
Outline Formation of Differential Equations and Their Solutions
Separation of Variables
ODEs with Rational Slope Functions
Closure Some Special ODEs
Exact Differential Equations and Reduction to the Exact Form
First Order Linear (Leibnitz) ODE and Associated Forms
Orthogonal Trajectories
Modelling and Simulation
I For a solution of initial value problems, questions of existence,
uniqueness and continuous dependence on initial condition are First Order Ordinary Differential Equations
of crucial importance. Formation of Differential Equations and Their Solutions
I These issues pertain to aspects of practical relevance
regarding a physical system and its dynamic simulation
Some Special ODEs
I Lipschitz condition is the tightest (avaliable) criterion for Exact Differential Equations and Reduction to the Exact Form
deciding these questions regarding well-posedness First Order Linear (Leibnitz) ODE and Associated Forms
Applied Mathematical Methods First Order Ordinary Differential Equations 361, Applied Mathematical Methods First Order Ordinary Differential Equations 362,
Formation of Differential Equations and Their

Separation Solutions
Formation of Differential Equations and Their Solutions
of Variables
Separation of Variables Formation of Differential Equations and Their Solutions
Some Special ODEs Some Special ODEs
Exact Differential Equations and Reduction to the Exact Form ODE form with separable variables: Exact Differential Equations and Reduction to the Exact Form
A differential equation represents a class of Orthogonal
functions.
Trajectories
Modelling and Simulation Modelling and Simulation
Example: y (x) = cx k dy (x)
y 0 = f (x, y ) = or (y )dy = (x)dx
dx (y )
dy d2y
With dx = ckx k1 and dx 2
= ck(k 1)x k2 ,
Solution as quadrature:
2
d 2y dy dy Z Z
xy 2 = x y
dx dx dx (y )dy = (x)dx + c.
A compact intrinsic description.

Separation of variables through substitution
Important terms Example:
I Order and degree of differential equations y 0 = g (x + y + )
I Homogeneous and non-homogeneous ODEs Substitute v = x + y + to arrive at
Z
Solution of a differential equation dv dv
= + g (v ) x = +c
I general, particular and singular solutions dx + g (v )
ODEs with Rational Slope FunctionsFormation of Differential Equations and Their Solutions
Some Special ODEs Formation of Differential Equations and Their Solutions
Exact Differential Equations and Reduction to the Exact Form Exact Differential Equations and Reduction to the Exact Form
f1 (x, y ) First Order Linear (Leibnitz) ODE and Associated Forms Clairauts equation First Order Linear (Leibnitz) ODE and Associated Forms
0
y = Orthogonal Trajectories
Modelling and Simulation y = xy 0 + f (y 0 )
f2 (x, y )
If f1 and f2 are homogeneous functions of n-th degree, then Substitute p = y 0 and differentiate:
substitution y = ux separates variables x and u.
dp dp dp
dy 1 (y /x) du 1 (u) dx 2 (u) p =p+x + f 0 (p) [x + f 0 (p)] = 0
= u+x = = du dx dx dx
dx 2 (y /x) dx 2 (u) x 1 (u) u2 (u)
dp
For y 0 = a1 x+b1 y +c1 = 0 means y 0 = p = m (constant)
a2 x+b2 y +c2 , coordinate shift dx
dy dY
I family of straight lines y = mx + f (m) as general solution
x = X + h, y = Y + k y 0 = =
dx dX Singular solution:
produces
dY a1 X + b1 Y + (a1 h + b1 k + c1 ) x = f 0 (p) and y = f (p) pf 0 (p)
= .
dX a2 X + b2 Y + (a2 h + b2 k + c2 )
Choose h and k such that Singular solution is the envelope of the family of straight
a1 h + b 1 k + c 1 = 0 = a 2 h + b 2 k + c 2 . lines that constitute the general solution.
If the system is inconsistent, then substitute u = a 2 x + b2 y .
Some Special ODEs Formation of Differential Equations and Their Solutions

Second order ODEs with the function not appearing Exact Differential Equations and Reduction to the Exact Form Mdx + Ndy : an exact differential if Exact Differential Equations and Reduction to the Exact Form
First Order Linear (Leibnitz) ODE and Associated Forms First Order Linear (Leibnitz) ODE and Associated Forms
explicitly Orthogonal Trajectories
M N
Modelling and Simulation Modelling and Simulation
f (x, y 0 , y 00 ) = 0 M= and N = , or, =
x y y x
Substitute y 0 = p and solve f (x, p, p 0 ) = 0 for p(x). M N
M(x, y )dx + N(x, y )dy = 0 is an exact ODE if y = x
Second order ODEs with independent variable not appearing
explicitly With M(x, y ) = x and N(x, y ) = y ,
f (y , y 0 , y 00 ) = 0
dx + dy = 0 d = 0.
Use y 0 = p and x y
dp dp dy dp dp Solution: (x, y ) = c
y 00 = = =p f (y , p, p ) = 0.
dx dy dx dy dy Working rule:
Z Z
Solve for p(y ). 1 (x, y ) = M(x, y )dx+g1 (y ) and 2 (x, y ) = N(x, y )dy +g2 (x)
Resulting equation solved through a quadrature as
Z Determine g1 (y ) and g2 (x) from 1 (x, y ) = 2 (x, y ) = (x, y ).
dy dy
= p(y ) x = x0 + . If M N
y 6= x , but y (FM) = x (FN)?
dx p(y )
F : Integrating factor
First Order Linear (Leibnitz) ODE andFormation

Separation
of Differential Equations and Their Solutions
Associated
of Variables Forms
First Order Linear (Leibnitz) ODE andSeparation
Associated
of Variables Forms
General first order linear ODE: Exact Differential Equations and Reduction to the Exact Form Bernoullis equation Exact Differential Equations and Reduction to the Exact Form
First Order Linear (Leibnitz) ODE and Associated Forms First Order Linear (Leibnitz) ODE and Associated Forms
Orthogonal Trajectories Orthogonal Trajectories
dy Modelling and Simulation dy k
+ P(x)y = Q(x) + P(x)y = Q(x)y
dx dx
Leibnitz equation Substitution: z = y 1k , dz
dx = (1 k)y k dy
dx gives
For integrating factor F (x), dz
+ (1 k)P(x)z = (1 k)Q(x),
dy d dF dx
F (x) + F (x)P(x)y = [F (x)y ] = F (x)P(x). in the Leibnitz form.
dx dx dx
Riccati equation
Separating variables,
Z Z Z y 0 = a(x) + b(x)y + c(x)y 2
dF
= P(x)dx ln F = P(x)dx.
F If one solution y1 (x) is known, then propose y (x) = y1 (x) + z(x).
R
P(x)dx y10 (x) + z 0 (x) = a(x) + b(x)[y1 (x) + z(x)] + c(x)[y1 (x) + z(x)]2
Integrating factor: F (x) = e
Since y10 (x) = a(x) + b(x)y1 (x) + c(x)[y1 (x)]2 ,
Z
R
P(x)dx
R
P(x)dx z 0 (x) = [b(x) + 2c(x)y1 (x)]z(x) + c(x)[z(x)]2 ,
ye = Q(x)e dx + C
in the form of Bernoullis equation.
Orthogonal Trajectories Formation of Differential Equations and Their Solutions

Points to note Formation of Differential Equations and Their Solutions
In xy -plane, one-parameter equation (x, y , c) = 0: Exact Differential Equations and Reduction to the Exact Form
Orthogonal Trajectories Orthogonal Trajectories
a family of curves Modelling and Simulation Modelling and Simulation
I Meaning and solution of ODEs

Differential equation of the family of curves:
I Separating variables
dy I Exact ODEs and integrating factors
= f1 (x, y )
dx I Linear (Leibnitz) equations
Slope of curves orthogonal to (x, y , c) = 0: I Orthogonal families of curves
dy 1
=
dx f1 (x, y )
Solving this ODE, another family of curves (x, y , k) = 0.
Orthogonal trajectories
If (x, y , c) = 0 represents the potential lines (contours),

then (x, y , k) = 0 will represent the streamlines!
Applied Mathematical Methods Second Order Linear Homogeneous ODEs 371, Applied Mathematical Methods Second Order Linear Homogeneous ODEs 372,
Outline Introduction
Homogeneous Equations with Constant Coefficients
Euler-Cauchy Equation
Introduction Introduction
Theory of the Homogeneous Equations Theory of the Homogeneous Equations
Basis for Solutions Second order ODE: Basis for Solutions
f (x, y , y 0 , y 00 ) = 0
Special case of a linear (non-homogeneous) ODE:
Second Order Linear Homogeneous ODEs
y 00 + P(x)y 0 + Q(x)y = R(x)
Introduction
Homogeneous Equations with Constant Coefficients Non-homogeneous linear ODE with constant coefficients:
Theory of the Homogeneous Equations y 00 + ay 0 + by = R(x)
Basis for Solutions
For R(x) = 0, linear homogeneous differential equation
y 00 + P(x)y 0 + Q(x)y = 0
and linear homogeneous ODE with constant coefficients
y 00 + ay 0 + by = 0

Introduction
Introduction
Basis for Solutions Basis for Solutions
00 0
I Real and equal (a2 = 4b): 1 = 2 = = 2a
y + ay + by = 0
only solution in hand: y1 = e x
Assume
y = e x y 0 = e x and y 00 = 2 e x . Method to develop another solution?
I Verify that y2 = xe x is another solution.
Substitution: (2 + a + b)e x = 0 y (x) = c1 y1 (x) + c2 y2 (x) = (c1 + c2 x)e x
Auxiliary equation:
2 + a + b = 0
I Complex conjugate (a2 < 4b): 1,2 = 2a i
Solve for 1 and 2 :
a a
Solutions: e 1 x and e 2 x y (x) = c1 e ( 2 +i )x + c2 e ( 2 i )x
ax
= e 2 [c1 (cos x + i sin x) + c2 (cos x i sin x)]
Three cases ax
= e 2 [A cos x + B sin x],
I Real and distinct (a2 > 4b): 1 6= 2
with A = c1 + c2 , B = i(c1 c2 ).
y (x) = c1 y1 (x) + c2 y2 (x) = c1 e 1 x + c2 e 2 x I A third form: y (x) = Ce 2 cos(x )
ax
Euler-Cauchy Equation Introduction

Theory of the Homogeneous Equations
Introduction
Basis for Solutions Basis for Solutions
x 2 y 00 + axy 0 + by = 0
Substituting y = x k , auxiliary (or indicial) equation: y 00 + P(x)y 0 + Q(x)y = 0
2
k + (a 1)k + b = 0 Well-posedness of its IVP:
The initial value problem of the ODE, with arbitrary
1. Roots real and distinct [(a 1)2 > 4b]: k1 6= k2 . initial conditions y (x0 ) = Y0 , y 0 (x0 ) = Y1 , has a unique
solution, as long as P(x) and Q(x) are continuous in the
y (x) = c1 x k1 + c2 x k2 . interval under question.
2. Roots real and equal [(a 1)2 = 4b]: k1 = k2 = k = a1
2 .
y (x) = (c1 + c2 ln x)x k . At least two linearly independent solutions:

3. Roots complex conjugate [(a 1)2 < 4b]: k1,2 = a1 y1 (x): IVP with initial conditions y (x 0 ) = 1, y 0 (x0 ) = 0
2 i.
I
a1 a1 I y2 (x): IVP with initial conditions y (x 0 ) = 0, y 0 (x0 ) = 1

y (x) = x 2 [A cos( ln x)+B sin( ln x)] = Cx 2 cos( ln x).
c1 y1 (x) + c2 y2 (x) = 0 c1 = c2 = 0
Alternative approach: substitution
dx dt 1 At most two linearly independent solutions?
x = e t t = ln x, = e t = x and = , etc.
dt dx x

Introduction
Introduction
Wronskian of two solutions y1 (x) and y2 (x):Basis for Solutions If y1 (x) and y2 (x) are linearly dependent, then = ky1 .
y2Solutions
Basis for

y1 y2 W (y1 , y2 ) = y1 y20 y2 y10 = y1 (ky10 ) (ky1 )y10 = 0
W (y1 , y2 ) = 0 = y1 y20 y2 y10
y1 y20
In particular, W [y1 (x0 ), y2 (x0 )] = 0
Solutions y1 and y2 are linearly dependent, if and only if x 0 Conversely, if there is a value x0 , where
such that W [y1 (x0 ), y2 (x0 )] = 0.
y (x ) y2 (x0 )
I W [y1 (x0 ), y2 (x0 )] = 0 W [y1 (x), y2 (x)] = 0 x. W [y1 (x0 ), y2 (x0 )] = 10 0 = 0,

y1 (x0 ) y20 (x0 )
I W [y1 (x1 ), y2 (x1 )] 6= 0 W [y1 (x), y2 (x)] 6= 0 x, and y1 (x)
then for
and y2 (x) are linearly independent solutions. y1 (x0 ) y2 (x0 ) c1
= 0,
Complete solution: y10 (x0 ) y20 (x0 ) c2
If y1 (x) and y2 (x) are two linearly independent solutions, coefficient matrixis singular.

then the general solution is c1
Choose non-zero and frame y (x) = c1 y1 + c2 y2 , satisfying
c2
y (x) = c1 y1 (x) + c2 y2 (x).
IVP y 00 + Py 0 + Qy = 0, y (x0 ) = 0, y 0 (x0 ) = 0.
And, the general solution is the complete solution .
Therefore, y (x) = 0 y1 and y2 are linearly dependent.
No third linearly independent solution. No singular solution.

Introduction
Basis for Solutions Introduction
Basis for Solutions For completely describing the solutions, we need Basis for Solutions
Pick a candidate solution Y (x), choose a point x 0 , evaluate
functions y1 , y2 , Y and their derivatives at that point, frame two linearly independent solutions.
No guaranteed procedure to identify two basis members!
y1 (x0 ) y2 (x0 ) C1 Y (x0 )
0 0 = 0 If one solution y1 (x) is available, then to find another?
y1 (x0 ) y2 (x0 ) C2 Y (x0 )
Reduction of order

C1 Assume the second solution as
and ask for solution .
C2
y2 (x) = u(x)y1 (x)
Unique solution for C1 , C2 . Hence, particular solution
and determine u(x) such that y2 (x) satisfies the ODE.
y (x) = C1 y1 (x) + C2 y2 (x)
is the unique solution of the IVP u 00 y1 + 2u 0 y10 + uy100 + P(u 0 y1 + uy10 ) + Quy1 = 0
y 00 + Py 0 + Qy = 0, y (x0 ) = Y (x0 ), y 0 (x0 ) = Y 0 (x0 ). u 00 y1 + 2u 0 y10 + Pu 0 y1 + u(y100 + Py10 + Qy1 ) = 0.

But, that is the candidate function Y (x)! Hence, Y (x) = y (x). Since y100 + Py10 + Qy1 = 0, we have y1 u 00 + (2y10 + Py1 )u 0 = 0

y0 Theory of the Homogeneous Equations
Function space perspective: Theory of the Homogeneous Equations
Denoting u 0 = U, U 0 + (2 y11 + P)U = 0. Basis for Solutions Basis for Solutions
Operator D means differentiation, operates on an infinite
Rearrangement and integration of the reduced equation: dimensional function space as a linear transformation.
dU dy1 R I It maps all constant functions to zero.
+2 + Pdx = 0 Uy12 e Pdx = C = 1 (choose). It has a one-dimensional null space.
U y1 I
Second derivative or D 2 is an operator that has a two-dimensional

Then, null space, c1 + c2 x, with basis {1, x}.
1 R Pdx
u0 = U = e , Examples of composite operators
y12 I (D + a) has a null space ce ax .
Integrating, Z I (xD + a) has a null space cx a .
1 R Pdx
u(x) = e dx, A second order linear operator D 2 + P(x)D + Q(x) possesses a
y12
two-dimensional null space.
and Z
1 R Pdx I Solution of [D 2 + P(x)D + Q(x)]y = 0: description of the
y2 (x) = y1 (x) e dx.
y12 null space, or a basis for it..
I Analogous to solution of Ax = 0, i.e. development of a basis
Note: The factor u(x) is never constant! for Null(A).
Applied Mathematical Methods Second Order Linear Homogeneous ODEs 383, Applied Mathematical Methods Second Order Linear Non-Homogeneous ODEs 384,
Points to note Introduction

Outline Linear ODEs and Their Solutions
Method of Undetermined Coefficients
Method of Variation of Parameters
Theory of the Homogeneous Equations Closure
Basis for Solutions
I Second order linear homogeneous ODEs

I Wronskian and related results Second Order Linear Non-Homogeneous ODEs
I Solution basis Linear ODEs and Their Solutions
I Reduction of order Method of Undetermined Coefficients
I Null space of a differential operator Method of Variation of Parameters
Closure

Applied Mathematical Methods Second Order Linear Non-Homogeneous ODEs 385, Applied Mathematical Methods Second Order Linear Non-Homogeneous ODEs 386,
Linear ODEs and Their Solutions Linear ODEs and Their Solutions
Linear ODEs and Their Solutions Linear ODEs and Their Solutions
The Complete Analogy Closure
Procedure to solve y 00 + P(x)y 0 + Q(x)y = R(x) Closure
1. First, solve the corresponding homogeneous equation, obtain a

Table: Linear systems and mappings: algebraic and differential basis with two solutions and construct
In ordinary vector space In infinite-dimensional function space yh (x) = c1 y1 (x) + c2 y2 (x).

Ax = b y 00 + Py 0 + Qy = R 2. Next, find one particular solution y p (x) of the NHE and
The system is consistent. P(x), Q(x), R(x) are continuous. compose the complete solution
A solution x A solution yp (x)
Alternative solution: x Alternative solution: y (x) y (x) = yh (x) + yp (x) = c1 y1 (x) + c2 y2 (x) + yp (x).
x x satisfies Ax = 0, y (x) yp (x) satisfies y 00 + Py 0 + Qy = 0, 3. If some initial or boundary conditions are known, they can be
is in null space of A. is in null space of D 2 + P(x)D + Q(x). imposed now to determine c1 and c2 .
Complete solution:
P Complete solution:
P Caution: If y1 and y2 are two solutions of the NHE, then
x = x + i ci (x0 )i yp (x) + i ci yi (x) do not expect c1 y1 + c2 y2 to satisfy the equation.
Methodology: Methodology: Implication of linearity or superposition:
Find null space of A Find null space of D 2 + P(x)D + Q(x) With zero initial conditions, if y1 and y2 are responses
i.e. basis members (x0 )i . i.e. basis members yi (x). due to inputs R1 (x) and R2 (x), respectively, then the
Find x and compose. Find yp (x) and compose. response due to input c1 R1 + c2 R2 is c1 y1 + c2 y2 .
Method of Undetermined CoefficientsLinear ODEs and Their Solutions

Method of Undetermined CoefficientsLinear ODEs and Their Solutions
Closure Closure
Example:
y 00 + ay 0 + by = R(x) (a) y 00 6y 0 + 5y = e 3x
I What kind of function to propose as y p (x) if R(x) = x n ? (b) y 00 5y 0 + 6y = e 3x
I And what if R(x) = e x ? (c) y 00 6y 0 + 9y = e 3x
I If R(x) = x n + e x , i.e. in the form k1 R1 (x) + k2 R2 (x)? In each case, the first official proposal: y p = ke 3x
The principle of superposition (linearity) (a) y (x) = c1 e x + c2 e 5x e 3x /4
(b) y (x) = c1 e 2x + c2 e 3x + xe 3x
Table: Candidate solutions for linear non-homogeneous ODEs 1 2 3x
(c) y (x) = c1 e 3x + c2 xe 3x + 2x e
RHS function R(x) Candidate solution y p (x)
pn (x) qn (x) Modification rule
e x ke x I If the candidate function (ke x , k1 cos x + k2 sin x or
cos x or sin x k1 cos x + k2 sin x k1 e x cos x + k2 e x sin x) is a solution of the corresponding
e x cos x or e x sin x k1 e x cos x + k2 e x sin x HE; with , i or i (respectively) satisfying the
pn (x)e x qn (x)e x auxiliary equation; then modify it by multiplying with x.
pn (x) cos x or pn (x) sin x qn (x) cos x + rn (x) sin x I In the case of being a double root, i.e. both e x and xe x
pn (x)e x cos x or pn (x)e x sin x qn (x)e x cos x + rn (x)e x sin x being solutions of the HE, choose yp = kx 2 e x .
Method of Variation of Parameters Linear ODEs and Their Solutions

Closure
From yp = u1 y1 + u2 y2 , Closure
Solution of the HE: yp0 = u10 y1 + u1 y10 + u20 y2 + u2 y20 .

yh (x) = c1 y1 (x) + c2 y2 (x),
Condition u10 y1 + u20 y2 = 0 gives
in which c1 and c2 are constant parameters.
yp0 = u1 y10 + u2 y20 .
For solution of the NHE, Differentiating,

how about variable parameters ? yp00 = u10 y10 + u20 y20 + u1 y100 + u2 y200 .
Propose Substitution into the ODE:
yp (x) = u1 (x)y1 (x) + u2 (x)y2 (x)
u10 y10 +u20 y20 +u1 y100 +u2 y200 +P(x)(u1 y10 +u2 y20 )+Q(x)(u1 y1 +u2 y2 ) = R(x)
and force yp (x) to satisfy the ODE.
A single second order ODE in u1 (x) and u2 (x). Rearranging,
We need one more condition to fix them. u10 y10 +u20 y20 +u1 (y100 +P(x)y10 +Q(x)y1 )+u2 (y200 +P(x)y20 +Q(x)y2 ) = R(x).
As y1 and y2 satisfy the associated HE, u10 y10 + u20 y20 = R(x)

Points to note Linear ODEs and Their Solutions
Closure Closure

y1 y2 u10 0
=
y10 y20 u20 R
I Function space perspective of linear ODEs
Since Wronskian is non-zero, this system has unique solution I Method of undetermined coefficients
y2 R y1 R I Method of variation of parameters
u10 = and u20 = .
W W
Direct quadrature:
Z Z
y2 (x)R(x) y1 (x)R(x) Necessary Exercises: 1,3,5,6
u1 (x) = dx and u2 (x) = dx
W [y1 (x), y2 (x)] W [y1 (x), y2 (x)]
In contrast to the method of undetermined multipliers,

variation of parameters is general. It is applicable for all
continuous functions as P(x), Q(x) and R(x).
Applied Mathematical Methods Higher Order Linear ODEs 393, Applied Mathematical Methods Higher Order Linear ODEs 394,
Outline Theory of Linear ODEs

Non-Homogeneous Equations
Theory of Linear ODEs Theory of Linear ODEs
Euler-Cauchy Equation of Higher Order Euler-Cauchy Equation of Higher Order
y (n) +P1 (x)y (n1) +P2 (x)y (n2) + +Pn1 (x)y 0 +Pn (x)y = R(x)
General solution: y (x) = yh (x) + yp (x), where
I yp (x): a particular solution
Higher Order Linear ODEs
I yh (x): general solution of corresponding HE
Theory of Linear ODEs
Homogeneous Equations with Constant Coefficients y (n) +P1 (x)y (n1) +P2 (x)y (n2) + +Pn1 (x)y 0 +Pn (x)y = 0
For the HE, suppose we have n solutions y 1 (x), y2 (x), , yn (x).
Euler-Cauchy Equation of Higher Order
Assemble the state vectors in matrix

y1 y2 yn
y10 y2 0 0
yn

y100 y 00 y 00
Y(x) = 2 n .
.. .. .. ..
. . . .
(n1) (n1) (n1)
y1 y2 yn
Wronskian:
W (y1 , y2 , , yn ) = det[Y(x)]
Theory of Linear ODEs Theory of Linear ODEs

Theory of Linear ODEs
I If solutions y1 (x), y2 (x), , yn (x) of HE are linearly
dependent, then for a non-zero k R n , y (n)
+ a1 y (n1)
+ a2 y (n2)
+ + an1 y 0 + an y = 0
X n Xn
ki yi (x) = 0
(j)
ki yi (x) = 0 for j = 1, 2, 3, , (n 1) With trial solution y = e x , the auxiliary equation:
i =1 i =1
[Y(x)]k = 0 [Y(x)] is singular, n + a1 n1 + a2 n2 + + an1 + an = 0
W [y1 (x), y2 (x), , yn (x)] = 0.
Construction of the basis:
I If Wronskian is zero at x = x0 , then P Y(x0 ) is singular and a 1. For every simple real root = , e x is a solution.
non-zero k Null[Y(x0 )] gives ni=1 ki yi (x) = 0, implying
y1 (x), y2 (x), , yn (x) to be linearly dependent. 2. For every simple pair of complex roots = i,
I Zero Wronskian at some x = x0 implies zero Wronskian e x cos x and e x sin x are linearly independent solutions.
everywhere. Non-zero Wronskian at some x = x 1 ensures 3. For every real root = of multiplicity r ; e x , xe x , x 2 e x ,
non-zero Wronskian everywhere and the corrseponding , x r 1 e x are all linearly independent solutions.
solutions as linearly independent. 4. For every complex pair of roots = i of multiplicity r ;
With n linearly independent solutions y 1 (x), y2 (x), P , yn (x) e x cos x, e x sin x, xe x cos x, xe x sin x, ,
I
of the HE, we have its general solution y h (x) = ni=1 ci yi (x), x r 1 e x cos x, x r 1 e x sin x are the required solutions.
acting as the complementary function for the NHE.
Non-Homogeneous Equations Theory of Linear ODEs

Non-Homogeneous Equations Theory of Linear ODEs
Method of undetermined coefficients Since each yi (x) is a solution of the HE,
y (n) + a1 y (n1) + a2 y (n2) + + an1 y 0 + an y = R(x) n
X (n1)
ui0 (x)yi (x) = R(x).
Extension of the second order case i =1
Method of variation of parameters
n
Assembling all conditions on u0 (x) together,
X
yp (x) = ui (x)yi (x) [Y(x)]u0 (x) = en R(x).
i =1 adj Y
Since Y1 = det(Y) ,
Imposed condition
P Derivative P 1 R(x)
n
u 0 (x)yi (x) = 0 yp0 (x) = ni=1 ui (x)yi0 (x) u0 (x) = [adj Y(x)]en R(x) = [last column of adj Y(x)].
Pin=1 i0 P det[Y(x)] W (x)
0
i =1 ui (x)yi (x) = 0 yp (x) = ni=1 ui (x)yi00 (x)
00
Using cofactors of elements from last row only,

Pn (n2) (n1) P (n1) Wi (x)
0
i =1 ui (x)yi (x) = 0 yp (x) = ni=1 ui (x)yi (x) ui0 (x) = R(x),
P P W (x)
(n) (n1) (n)
Finally, yp (x) = ni=1 ui0 (x)yi (x) + ni=1 ui (x)yi (x) with Wi (x) = Wronskian evaluated with en in place of i-th column.
Xn n
X h i R (x)R(x)
ui0 (x)yi
(n1)
(x)+
(n)
ui (x) yi + P1 yi
(n1)
+ + Pn yi = R(x). ui (x) = WiW (x) dx
i =1 i =1
Applied Mathematical Methods Higher Order Linear ODEs 399, Applied Mathematical Methods Laplace Transforms 400,
Points to note Theory of Linear ODEs

Basic Properties and Results
Application to Differential Equations
Euler-Cauchy Equation of Higher Order Handling Discontinuities
Convolution
Advanced Issues
I Wronskian for a higher order ODE

Laplace Transforms
I General theory of linear ODEs Introduction
I Variation for parameters for n-th order ODE Basic Properties and Results
Handling Discontinuities
Convolution
Necessary Exercises: 1,3,4 Advanced Issues
Applied Mathematical Methods Laplace Transforms 401, Applied Mathematical Methods Laplace Transforms 402,
Handling Discontinuities Handling Discontinuities
Convolution Another question: What if R(x) is not continuous? Convolution
Classical perspective Advanced Issues Advanced Issues
I When power is switched on or off, what happens?
I Entire differential equation is known in advance.
I If there is a sudden voltage fluctuation, what happens to the
I Go for a complete solution first.
equipment connected to the power line?
I Afterwards, use the initial (or other) conditions.
Or, does anything happen in the immediate future?
A practical situation Something certainly happens. The IVP has a solution!
I You have a plant Laplace transforms provide a tool to find the solution, in
I intrinsic dynamic model as well as the starting conditions. spite of the discontinuity of R(x).
I You may drive the plant with different kinds of inputs on Integral transform:
different occasions.
Z b
Implication T [f (t)](s) = K (s, t)f (t)dt

a
I Left-hand side of the ODE and the initial conditions are
known a priori. s: frequency variable
I Right-hand side, R(x), changes from task to task. K (s, t): kernel of the transform
Note: T [f (t)] is a function of s, not t.
Basic Properties and Results Introduction
Convolution Linearity: Convolution
With kernel function K (s, t) = e st , and limits a = 0, b = , Advanced Issues Advanced Issues
Laplace transform L{af (t) + bg (t)} = aL{f (t)} + bL{g (t)}

Z Z b First shifting property or the frequency shifting rule:
F (s) = L{f (t)} = e st f (t)dt = lim e st f (t)dt L{e at f (t)} = F (s a)
0 b 0
When this integral exists, f (t) has its Laplace transform. Laplace transforms of some elementary functions:
Z st
e 1
Sufficient condition: L(1) = e st dt = = ,
0 s 0 s
I f (t) is piecewise continuous, and Z st Z
e 1 st 1
I it is of exponential order, i.e. |f (t)| < Me ct for some (finite) L(t) = e st tdt = t + e dt = 2 ,
0 s 0 s 0 s
M and c.
n!
L(t n ) = n+1 (for positive integer n),
s
Inverse Laplace transform:
(a + 1)
L(t a ) = (for a R + )
f (t) = L1 {F (s)} s a+1
1
and L(e at ) = .
s a
Basic Properties and Results Introduction

Application to Differential Equations Introduction
Convolution Example: Convolution
Advanced Issues Advanced Issues
s Initial value problem of a linear constant coefficient ODE
L(cos t) = , L(sin t) = 2 ;
s 2 + 2 s + 2 y 00 + ay 0 + by = r (t), y (0) = K0 , y 0 (0) = K1
s a
L(cosh at) = 2 , L(sinh at) = 2 ;
s a2 s a2 Laplace transforms of both sides of the ODE:
s
L(e t cos t) = , L(e t sin t) = . s 2 Y (s) sy (0) y 0 (0) + a[sY (s) y (0)] + bY (s) = R(s)
(s )2 + 2 (s )2 + 2
(s 2 + as + b)Y (s) = (s + a)K0 + K1 + R(s)
Laplace transform of derivative:
Z
A differential equation in y (t) has been converted to an
L{f 0 (t)} = e st f 0 (t)dt algebraic equation in Y (s).
0
Z

= e st f (t) 0 + s e st f (t)dt = sL{f (t)} f (0) Transfer function: ratio of Laplace transform of output function
0 y (t) to that of input function r (t), with zero initial conditions
Using this process recursively, Y (s) 1
Q(s) = = 2 (in this case)
L{f (n) (t)} = s n L{f (t)} s (n1) f (0) s (n2) f 0 (0) f (n1) (0). R(s) s + as + b
Rt
For integral g (t) = 0 f (t)dt, g (0) = 0, and Y (s) = [(s + a)K0 + K1 ]Q(s) + Q(s)R(s)
L{g (t)} = sL{g (t)}g (0) = sL{g (t)} L{g (t)} = 1s L{f (t)}.
0
Solution of the given IVP: y (t) = L1 {Y (s)}
Handling Discontinuities Introduction

Unit step function Convolution Define Convolution
Advanced Issues Advanced Issues
0 if t <a 1/k ifa t a+k
u(t a) = fk (t a) =
1 if t >a 0 otherwise
Its Laplace transform: 1 1
= u(t a) u(t a k)
Z Z Z k k
a
e as
L{u(t a)} = e st u(t a)dt = 0dt + e st dt =
0 0 a s u(ta)
1
1 u(ta)
k 1
f (ta)
k (ta)
k k
For input f (t) with a time delay, 1 1 1 1
o a t o a a+k t o a a+k t o a t
0 if t<a 1
f (t a)u(t a) = 1
k
f (t a) if t>a 1 u(tak)
k
has its Laplace transform as (a) Unit step function (b) Composition (c) Function f
k
(d) Diracs function
Z
L{f (t a)u(t a)} = e st f (t a)dt Figure: Step and impulse functions
Za and note that its integral
= e s(a+ ) f ( )d = e as L{f (t)}. Z Z a+k
1
0 Ik = fk (t a)dt = dt = 1.
0 a k
Second shifting property or the time shifting rule
does not depend on k.

Convolution Introduction
Handling Discontinuities
A generalized product of two functions Handling Discontinuities
In the limit, Convolution
Advanced Issues Z t
Convolution
Advanced Issues
h(t) = f (t) g (t) = f ( )g (t ) d

(t a) = lim fk (t a) 0
k0
Z Laplace transform of the convolution:
if t = a Z Z t Z Z
or, (t a) = and (t a)dt = 1.
0 otherwise 0 H(s) = e st f ( )g (t )d dt = f ( ) e st g (t )dt d
0 0 0
Unit impulse function or Diracs delta function
t= t=

1

L{(t a)} = [L{u(t a)} L{u(t a k)}]
lim
k
k0
e as e (a+k)s

= lim = e as
k0 ks o t o t
(a) Original order (b) Changed order

Through step and impulse functions, Laplace transform
method can handle IVPs with discontinuous inputs. Figure: Region of integration for L{h(t)}
Convolution Introduction
Convolution Convolution
Through substitution t 0 = t , Advanced Issues Advanced Issues
Z Z
0
H(s) = f ( ) e s(t + ) g (t 0 ) dt 0 d I A paradigm shift in solution of IVPs
0 0
Z Z I Handling discontinuous input functions
0
= f ( )e s e st g (t 0 ) dt 0 d I Extension to ODE systems
0 0
I The idea of integral transforms
H(s) = F (s)G (s)
Convolution theorem:
Laplace transform of the convolution integral of two
functions is given by the product of the Laplace Necessary Exercises: 1,2,4
transforms of the two functions.
Utilities:
I To invert Q(s)R(s), one can convolute y (t) = q(t) r (t).
I In solving some integral equation.
Applied Mathematical Methods ODE Systems 413, Applied Mathematical Methods ODE Systems 414,
Outline Fundamental Ideas

Linear Homogeneous Systems with Constant Coefficients
Linear Non-Homogeneous Systems
Fundamental Ideas Fundamental Ideas
Nonlinear Systems Nonlinear Systems
y0 = f(t, y)
Solution: a vector function y = h(t)
ODE Systems Autonomous system: y0 = f(y)
Fundamental Ideas I Points in y-space where f(y) = 0:
Linear Homogeneous Systems with Constant Coefficients equilibrium points or critical points
Nonlinear Systems System of linear ODEs:
y0 = A(t)y + g(t)
I autonomous systems if A and g are constant

I homogeneous systems if g(t) = 0
I homogeneous constant coefficient systems if A is constant
and g(t) = 0
Fundamental Ideas Fundamental Ideas

Fundamental Ideas
For a homogeneous system,

y0 = Ay
y0 = A(t)y Non-degenerate case: matrix A non-singular
I Origin (y = 0) is the unique equilibrium point.
Attempt y = xe t y0 = xe t .
I Wronskian: W (y1 , y2 , y3 , , yn ) = |y1 y2 y3 yn |
Substitution: Axe t = xe t Ax = x
If A is diagonalizable,
If Wronskian is non-zero, then
I n linearly independent solutions y i = xi e i t corresponding to n
I Fundamental matrix: Y(t) = [y1 y2 y3 yn ],
eigenpairs
giving a basis.
If A is not diagonalizable?
General solution: All xi e i t together will not complete the basis.
n
X Try y = xte t ? Substitution leads to
y(t) = ci yi (t) = [Y(t)] c
i =1 xe t + xte t = Axte t xe t = 0 x = 0.
Absurd!

Fundamental Ideas
Linear Non-Homogeneous Systems Fundamental Ideas
Try a linearly independent solution in the form
y0 = Ay + g(t)
y = xte t + ue t .
Complementary function:
Linear independence here has two implications: in n
X
function space AND in ordinary vector space! yh (t) = ci yi (t) = [Y(t)]c
i =1
Substitution:
Complete solution:
xe t + xte t + ue t = Axte t + Aue t (A I)u = x
y(t) = yh (t) + yp (t)
Solve for u, the generalized eigenvector of A.
We need to develop one particular solution y p .
For Jordan blocks of larger sizes,
1 Method of undetermined coefficients
y1 = xe t , y2 = xte t +u1 e t , y3 = xt 2 e t +u1 te t +u2 e t etc.
2 Based on g(t), select candidate function G k (t) and propose
X
Jordan canonical form (JCF) of A provides a set of basis yp = uk Gk (t),
functions to describe the complete solution of the ODE k
system. vector coefficients (uk ) to be determined by substitution.

Method of variation of parameters
Method of diagonalization
If we can supply a basis Y(t) of the complementary function y h (t),
If A is a diagonalizable constant matrix, with X 1 AX = D, then we propose
changing variables to z = X1 y, such that y = Xz, yp (t) = [Y(t)]u(t)
Substitution leads to
Xz0 = AXz+g(t) z0 = X1 AXz+X1 g(t) = Dz+h(t) (say).
Y 0 u + Yu0 = AYu + g.
Single decoupled Leibnitz equations Since Y 0 = AY,
zk0 = dk zk + hk (t), k = 1, 2, 3, , n; Yu0 = g, or, u0 = [Y]1 g.
leading to individual solutions Complete solution:

Z Z
zk (t) = ck e dk t + e dk t e dk t hk (t)dt. y(t) = yh + yp = [Y]c + [Y] [Y]1 gdt
After assembling z(t), we reconstruct y = Xz. This method is completely general.

Applied Mathematical Methods ODE Systems 421, Applied Mathematical Methods Stability of Dynamic Systems 422,
Points to note Fundamental Ideas

Outline Second Order Linear Systems
Nonlinear Dynamic Systems
Lyapunov Stability Analysis
Nonlinear Systems
I Theory of ODEs in terms of vector functions

I Methods to find Stability of Dynamic Systems
I complementary functions in the case of constant coefficients Second Order Linear Systems
I particular solutions for all cases
Necessary Exercises: 1
Applied Mathematical Methods Stability of Dynamic Systems 423, Applied Mathematical Methods Stability of Dynamic Systems 424,
Second Order Linear Systems Second Order Linear Systems

A system of two first order linear differential equations: Characteristic equation:

y10 = a11 y1 + a12 y2 2 p + q = 0,
y20 = a21 y1 + a22 y2
with p = (a11 + a22 ) = 1 + 2 and q = a11 a22 a12 a21 = 1 2
or, y0 = Ay
Phase: a pair of values of y1 and y2 Discriminant D = p 2 4q and
Phase plane: plane of y1 and y2 r
Trajectory: a curve showing the evolution of the system for a p p 2 p D
1,2 = q = .
particular initial value problem 2 2 2 2
Phase portrait: all trajectories together showing the complete Solution (for diagonalizable A):
picture of the behaviour of the dynamic system
y = c 1 x 1 e 1 t + c 2 x 2 e 2 t
Allowing only isolated equilibrium points, Solution for deficient A:
I matrix A is non-singular: origin is the only equilibrium point.
Eigenvalues of A: y = c1 x1 e t + c2 (tx1 + u)e t
2 (a11 + a22 ) + (a11 a22 a12 a21 ) = 0 y0 = c1 x1 e t + c2 (x1 + u)e t + tc2 x1 e t

Table: Critical points of linear systems
y2 y2 y2
Type Sub-type Eigenvalues Position in p-q chart Stability
Saddle pt real, opposite signs q<0 unstable
y1 y1 y1
Centre pure imaginary q > 0, p = 0 stable
o o o
Spiral complex, both q > 0, p 6= 0 stable
non-zero components D = p 2 4q < 0 if p < 0,
Node real, same sign q > 0, p 6= 0, D 0 unstable
(a) Saddle point (b) Centre (c) Spiral
y2 improper unequal in magnitude D>0 if p > 0
y2
y2 proper equal, diagonalizable D=0
degenerate equal, deficient D=0
q
y1 y1 spiral spiral
y1 o o c unstable
o stable
e
n
t
r
e 0
node 4q = node
p2
(d) Improper node (e) Proper node (f) Degenerate node o p
saddle point
Figure: Neighbourhood of critical points
unstable
Figure: Zones of critical points in p-q chart

Nonlinear Dynamic Systems Second Order Linear Systems

Lyapunov Stability Analysis Second Order Linear Systems
Important terms
Phase plane analysis
Stability: If y0 is a critical point of the dynamic system
I Determine all the critical points. y0 = f(y) and for every > 0, > 0 such that
I Linearize the ODE system around each of them as
ky(t0 ) y0 k < ky(t) y0 k < t > t0 ,
y0 = J(y0 )(y y0 ).
then y0 is a stable critical point. If, further,
I With z = y y0 , analyze each neighbourhood from z 0 = Jz. y(t) y0 as t , then y0 is said to be
I Assemble outcomes of local phase plane analyses. asymptotically stable.
Positive definite function: A function V (y), with V (0) = 0, is
Features of a dynamic system are typically captured by
called positive definite if
its critical points and their neighbourhoods.
V (y) > 0 y 6= 0.
Limit cycles
Lyapunov function: A positive definite function V (y), having
I isolated closed trajectories (only in nonlinear systems)
continuous V
yi , with a negative semi-definite rate of
change
Systems with arbitrary dimension of state space?
V 0 = [V (y)]T f(y).
Lyapunov Stability Analysis Second Order Linear Systems

Points to note Second Order Linear Systems
Lyapunovs stability criteria:

I Analysis of second order systems
Theorem: For a system y 0 = f(y) with the origin as a
critical point, if there exists a Lyapunov function V (y), I Classification of critical points
then the system is stable at the origin, i.e. the origin is a I Nonlinear systems and local linearization
stable critical point. I Phase plane analysis
Further, if V 0 (y) is negative definite, then it is Examples in physics, engineering, economics,
asymptotically stable. biological and social systems
A generalization of the notion of total energy: negativity of its rate
I Lyapunovs method of stability analysis
correspond to trajectories tending to decrease this energy.
Note: Lyapunovs method becomes particularly important when a

linearized model allows no analysis or when its results are suspect.
Caution: It is a one-way criterion only!
Applied Mathematical Methods Series Solutions and Special Functions 431, Applied Mathematical Methods Series Solutions and Special Functions 432,
Outline Power Series Method

Frobenius Method
Special Functions Defined as Integrals
Power Series Method Power Series Method
Frobenius Method
Special Functions Arising as Solutions of ODEs
Methods to solve an ODE in terms of elementary functions:
I restricted in scope
Theory allows study of the properties of solutions!
Series Solutions and Special Functions When elementary methods fail,

I gain knowledge about solutions through properties, and
Power Series Method
I for actual evaluation develop infinite series.
Frobenius Method
Special Functions Defined as Integrals Power series:

X
y (x) = an x n = a 0 + a 1 x + a 2 x 2 + a 3 x 3 + a 4 x 4 + a 5 x 5 +
n=0
or in powers of (x x0 ).
A simple exercise:
Try developing power series solutions in the above form
and study their properties for differential equations
y 00 + y = 0 and 4x 2 y 00 = y .
Power Series Method Power Series Method

Frobenius Method Power Series Method Power Series Method
Frobenius Method
Special Functions Defined as Integrals P n
Differentiation of y (x) = n=0 an x as Special Functions Arising as Solutions of ODEs
00 0
y + P(x)y + Q(x)y = 0
X
X
If P(x) and Q(x) are analytic at a point x = x 0 , y 0 (x) = (n + 1)an+1 x n and y 00 (x) = (n + 2)(n + 1)an+2 x n
n=0 n=0
i.e. if they possess convergent series expansions in powers
leads to
of (x x0 ) with some radius of convergence R,
"
# X
n
X X X
then the solution is analytic at x0 , and a power series solution P(x)y 0 = pn x n (n + 1)an+1 x n = pnk (k + 1)ak+1 x n
n=0 n=0 n=0 k=0
y (x) = a0 + a1 (x x0 ) + a2 (x x0 )2 + a3 (x x0 )3 +
" # X
n
X X X
is convergent at least for |x x0 | < R. Q(x)y = qn x n an x n = qnk ak x n
n=0 n=0 n=0 k=0
For x0 = 0 (without loss of generality), suppose " #

X n
X n
X

X
P(x) = n 2 3
pn x = p 0 + p 1 x + p 2 x + p 3 x + , (n + 2)(n + 1)an+2 + pnk (k + 1)ak+1 + qnk ak x n = 0
n=0 k=0 k=0
n=0
X Recursion formula:
Q(x) = qn x n = q 0 + q 1 x + q 2 x 2 + q 3 x 3 + , 1 X n
n=0 an+2 = [(k + 1)pnk ak+1 + qnk ak ]

P (n + 2)(n + 1)
n. k=0
and assume y (x) = n=0 an x
Frobenius Method Power Series Method

Frobenius Method
Frobenius Method Power Series Method
Frobenius Method
Special Functions Arising as Solutions of ODEs Special Functions Arising as Solutions of ODEs
For the ODE y 00 + P(x)y 0 + Q(x)y = 0, a point x = x0 is Working steps:

ordinary point if P(x) and Q(x) are analytic at x = x 0 : power P
1. Assume the solution in the form y (x) = x r n
n=0 an x .
series solution is analytic
2. Differentiate to get the series expansions for y (x) and y 00 (x).
0
singular point if any of the two is non-analytic (singular) at x = x 0
3. Substitute these series for y (x), y 0 (x) and y 00 (x) into the
I regular singularity: (x x 0 )P(x) and
given ODE and collect coefficients of x r , x r +1 , x r +2 etc.
(x x0 )2 Q(x) are analytic at the point
I irregular singularity
4. Equate the coefficient of x r to zero to obtain an equation in
the index r , called the indicial equation as
The case of regular singularity r (r 1) + b0 r + c0 = 0;
b(x) c(x) allowing a0 to become arbitrary.
For x0 = 0, with P(x) = x and Q(x) = x2
,
5. For each solution r , equate other coefficients to obtain a 1 , a2 ,
2 00 0
x y + xb(x)y + c(x)y = 0 a3 etc in terms of a0 .
in which b(x) and c(x) are analytic at the origin. Note: The need is to develop two solutions.
Special Functions Defined as IntegralsPower Series Method

Frobenius Method
Special Functions Arising as SolutionsPower
ofSeries
ODEs
Frobenius
Method
Method
In the study of some important problems in physics,
R
Gamma function: (n) = 0 e x x n1 dx, convergent for n > 0. some variable-coefficient ODEs appear recurrently,
Recurrence relation (1) = 1, (n + 1) = n(n)
defying analytical solution!
allows extension of the definition for the entire real
line except for zero and negative integers. Series solutions properties and connections
(n + 1) = n! for non-negative integers. further problems further solutions
(A generalization of the factorial function.)
R1 Table: Special functions of mathematical physics
Beta function: B(m, n) = 0 x m1 (1 x)n1 dx =
R /2 2m1 Name of the ODE Form of the ODE Resulting functions
2 0 sin cos2n1 d; m, n > 0. Legendres equation (1 x 2 )y 00 2xy 0 + k(k + 1)y = 0 Legendre functions
Legendre polynomials
B(m, n) = B(n, m); B(m, n) = (m)(n)
(m+n) Airys equation y 00 k 2 xy = 0 Airy functions
(1 x 2 )y 00 xy 0 + k 2 y = 0
Rx 2
Chebyshevs equation
y 00 2xy 0 + 2ky = 0
Chebyshev polynomials
Error function: erf (x) = 2 0 e t dt. Hermites equation Hermite functions

Hermite polynomials
Bessels equation x 2 y 00 + xy 0 + (x 2 k 2 )y = 0 Bessel functions
(Area under the normal or Gaussian distribution) Neumann functions
Rx Hankel functions
Sine integral function: Si (x) = 0 sint t dt. Gausss hypergeometric x (1 x )y 00 + [c (a + b + 1)x ]y 0 aby = 0 Hypergeometric function
equation
Laguerres equation xy 00 + (1 x )y 0 + ky = 0 Laguerre polynomials

ofSeries
ODEs
Frobenius
Method
Method
ofSeries
ODEs
Frobenius
Method
Method
Legendres equation Legendre functions
(1 x 2 )y 00 2xy 0 + k(k + 1)y = 0
k(k + 1) 2 k(k 2)(k + 1)(k + 3) 4
P(x) = 1x2x k(k+1) y1 (x) = 1 x + x
2 and Q(x) = 1x 2 are analytic at x = 0 with 2! 4!
radius of convergence R = 1. (k 1)(k + 2) 3 (k 1)(k 3)(k + 2)(k + 4) 5
y2 (x) = x x + x
x = 0 isP
an ordinary point and a power series solution 3! 5!
y (x) = n
n=0 an x is convergent at least for |x| < 1.
Special significance: non-negative integral values of k
Apply power series method: For each k = 0, 1, 2, 3, ,
k(k + 1) one of the series terminates at the term containing x k .
a2 = a0 ,
2!
(k + 2)(k 1) Polynomial solution: valid for the entire real line!
a3 = a1 Recurrence relation in reverse:
3!
(k n)(k + n + 1)
and an+2 = an for n 2. k(k 1)
(n + 2)(n + 1) ak2 = ak
2(2k 1)
Solution: y (x) = a0 y1 (x) + a1 y2 (x)

ofSeries
ODEs
Frobenius
Method
Method
ofSeries
ODEs
Frobenius
Method
Method
Legendre polynomial 1
Choosing ak = (2k1)(2k3)31
P0 (x)
k! , 0.8
P1(x)
(2k 1)(2k 3) 3 1 0.6
Pk (x) = 0.4
P3 (x)
k!
P 2(x)
k(k 1) k2 k(k 1)(k 2)(k 3) k4 0.2
xk x + x .
Pn (x)
2(2k 1) 2 4(2k 1)(2k 3) 0
0.2
This choice of ak ensures Pk (1) = 1 and implies Pk (1) = (1)k . 0.4
Initial Legendre polynomials: 0.6 P5 (x)

P 4(x)
P0 (x) = 1, 0.8
P1 (x) = x, 1
1 0.8 0.6 0.4 0.2 0 0.2 0.4 0.6 0.8 1
x
1
P2 (x) = (3x 2 1),
2 Figure: Legendre polynomials
1
P3 (x) = (5x 3 3x),
2 All roots of a Legendre polynomial are real and they lie in [1, 1].
1
P4 (x) = (35x 4 30x 2 + 3) etc.
8 Orthogonality?

ofSeries
ODEs
Frobenius
Method
Method
ofSeries
ODEs
Frobenius
Method
Method
Bessels equation
Bessel functions:
x 2 y 00 + xy 0 + (x 2 k 2 )y = 0 1
Selecting a0 = 2k (k+1) and using n = 2m,
x = 0 is a regular singular point. (1)m

Frobenius method: carrying out the early steps, am = .
2k+2m m!(k + m + 1)

X
(r 2 k 2 )a0 x r +[(r +1)2 k 2 ]a1 x r +1 + [an2 +{r 2 k 2 +n(n+2r )}an ]x r +n = 0 Bessel function of the first kind of order k:
n=2
k+2m
X x k+2m X (1)m x2
Indicial equation: r2 k2
= 0 r = k Jk (x) = (1)m =
2k+2m m!(k + m + 1) m!(k + m + 1)
With r = k, (r + 1)2 k 2 6= 0 a1 = 0 and m=0 m=0
an2
an = for n 2. When k is not an integer, Jk (x) completes the basis.
n(n + 2r )
Odd coefficients are zero and For integer k, Jk (x) = (1)k Jk (x), linearly dependent!
a0 a0 Reduction of order can be used to find another solution.
a2 = , a4 = , etc. Bessel function of the second kind or Neumann function
2(2k + 2) 2 4(2k + 2)(2k + 4)
Applied Mathematical Methods Series Solutions and Special Functions 445, Applied Mathematical Methods Sturm-Liouville Theory 446,
Points to note Power Series Method

Frobenius Method
Outline Preliminary Ideas
Sturm-Liouville Problems
Eigenfunction Expansions
I Solution in power series

I Ordinary points and singularities
Sturm-Liouville Theory
I Definition of special functions
Preliminary Ideas
I Legendre polynomials Sturm-Liouville Problems
I Bessel functions Eigenfunction Expansions
Applied Mathematical Methods Sturm-Liouville Theory 447, Applied Mathematical Methods Sturm-Liouville Theory 448,
Preliminary Ideas Preliminary Ideas

A simple boundary value problem: Boundary value problems as eigenvalue problems

Explore the possible solutions of the BVP
y 00 + 2y = 0, y (0) = 0, y () = 0 y 00 + ky = 0, y (0) = 0, y () = 0.
General solution of the ODE:
I With k 0, no hope for a non-trivial solution. Consider
y (x) = a sin(x 2) + b cos(x 2)
k = 2 > 0.
I Solutions: y = a sin(x), only for specific values of (or k):
Condition y (0) = 0 b = 0. Hence, y (x) = a sin(x 2).
= 0, 1, 2, 3, ; i.e. k = 0, 1, 4, 9, .
Then, y () = 0 a = 0. Only solution is y (x) = 0.
Question:
Now, consider the BVP
I For what values of k (eigenvalues), does the given BVP
y 00 + 4y = 0, y (0) = 0, y () = 0. possess non-trivial solutions, and

I what are the corresponding solutions (eigenfunctions), up to
The same steps give y (x) = a sin(2x), with arbitrary value of a. arbitrary scalar multiples?
Infinite number of non-trivial solutions! Analogous to the algebraic eigenvalue problem Av = v!
Analogy of a Hermitian matrix: self-adjoint differential operator.

Consider the ODE y 00 + P(x)y 0 + Q(x)y = 0. The adjoint ODE

Question: I The adjoint of the ODE y 00 + P(x)y 0 + Q(x)y = 0 is
Is it possible to find functions F (x) and G (x) such that
F 00 + P1 F 0 + Q1 F = 0,
F (x)y 00 + F (x)P(x)y 0 + F (x)Q(x)y
where P1 = P and Q1 = Q P 0 .
gets reduced to the derivative of F (x)y 0 + G (x)y ? I Then, the adjoint of F 00 + P1 F 0 + Q1 F = 0 is
Comparing with 00 + P2 0 + Q2 = 0,
d where P2 = P1 = P and
[F (x)y 0 + G (x)y ] = F (x)y 00 + [F 0 (x) + G (x)]y 0 + G 0 (x)y , Q2 = Q1 P10 = Q P 0 (P 0 ) = Q.
dx
The adjoint of the adjoint of a second order linear
F 0 (x) + G (x) = F (x)P(x) and G 0 (x) = F (x)Q(x).
homogeneous equation is the original equation itself.
Elimination of G (x):
I When is an ODE its own adjoint?
00 0 0
F (x) P(x)F (x) + [Q(x) P (x)]F (x) = 0
I y 00 + P(x)y 0 + Q(x)y = 0 is self-adjoint only in the trivial case
of P(x) = 0.
This is the adjoint of the original ODE.
I What about F (x)y 00 + F (x)P(x)y 0 + F (x)Q(x)y = 0?

Second order self-adjoint ODE Casting a given ODE into the self-adjoint form:
Question: What is the adjoint of Fy 00 + FPy 0 + FQy = 0? Equation y 00 + P(x)y 0 + Q(x)y = 0 is converted to the
Rephrased question: What is the ODE that (x) has to satisfy if self-adjoint
R form through the multiplication of
F (x) = e P(x)dx .
d
Fy 00 + FPy 0 + FQy = Fy 0 + (x)y ?
dx General form of self-adjoint equations:
Comparing terms, d
[F (x)y 0 ] + R(x)y = 0
dx
d
(F ) + (x) = FP and 0 (x) = FQ. Working rules:
dx
d2 d
I To determine whether a given ODE is in the self-adjoint form,
Eliminating (x), we have dx 2
(F ) + FQ = dx (FP). check whether the coefficient of y 0 is the derivative of the
coefficient of y 00 .
F 00 + 2F 0 0 + F 00 + FQ = FP0 + (FP)0 I To convert an ODE into the self-adjoint form, first obtain the

F 00 + (2F 0 FP)0 + F 00 (FP)0 + FQ = 0 equation in normal form by dividing with the coefficient of y 00 .
If the coefficient of y 0 now
R is P(x), then next multiply the
This is the same as the original ODE, when F 0 (x) = F (x)P(x) resulting equation with e Pdx .
Sturm-Liouville Problems Preliminary Ideas

Orthogonality of eigenfunctions
Sturm-Liouville equation
Theorem: If ym (x) and yn (x) are eigenfunctions
[r (x)y 0 ]0 + [q(x) + p(x)]y = 0, (solutions) of a Sturm-Liouville problem corresponding to
distinct eigenvalues m and n respectively, then
where p, q, r and r 0 are continuous on [a, b], with p(x) > 0 on
Z b
[a, b] and r (x) > 0 on (a, b).
(ym , yn ) p(x)ym (x)yn (x)dx = 0,
With different boundary conditions, a
Regular S-L problem: i.e. they are orthogonal with respect to the weight
a1 y (a) + a2 y 0 (a) = 0 and b1 y (b) + b2 y 0 (b) = 0, function p(x).
vectors [a1 a2 ]T and [b1 b2 ]T being non-zero.
From the hypothesis,
Periodic S-L problem: With r (a) = r (b),
0 0 0 0
y (a) = y (b) and y 0 (a) = y 0 (b). (rym ) + (q + m p)ym = 0 (q + m p)ym yn = (rym ) yn
Singular S-L problem: If r (a) = 0, no boundary condition is (ryn0 )0 + (q + n p)yn = 0 (q + n p)ym yn = (ryn0 )0 ym
needed at x = a. If r (b) = 0, no boundary condition Subtracting,
is needed at x = b.
(We just look for bounded solutions over [a, b].) (m n )pym yn = (ryn0 )0 ym + (ryn0 )ym0 0
(rym 0 0
)yn0 (rym ) yn
0 0
0
= r (ym yn yn ym ) .

Integrating both sides, Example: Legendre polynomials over [1, 1]
Z b Legendres equation
(m n ) p(x)ym (x)yn (x)dx d
a [(1 x 2 )y 0 ] + k(k + 1)y = 0
= r (b)[ym (b)yn0 (b) yn (b)ym
0
(b)] r (a)[ym (a)yn0 (a) yn (a)ym
0
(a)]. dx
is self-adjoint and defines a singular Sturm Liouville problem over
I In a regular S-L problem, from the boundary condition at [1, 1] with p(x) = 1, q(x) = 0, r (x) = 1 x 2 and = k(k + 1).
x = a, the homogeneous
system

ym (a) ym 0 (a) a1 0 Z 1
= has non-trivial solutions.
0
yn (a) yn (a) a2 0 (mn)(m+n+1) Pm (x)Pn (x)dx = [(1x 2 )(Pm Pn0 Pn Pm
0 1
)]1 = 0
Therefore, ym (a)yn0 (a) yn (a)ym0 (a) = 0. 1
Similarly, ym (b)yn0 (b) yn (b)ym

0 (b) = 0. From orthogonal decompositions 1 = P 0 (x), x = P1 (x),
I In a singular S-L problem, zero value of r (x) at a boundary 1 1 2 1
x2 = (3x 2 1) + = P2 (x) + P0 (x),
makes the corresponding term vanish even without a BC. 3 3 3 3
I In a periodic S-L problem, the two terms cancel out together. 3 1 3 3 2 3
x = (5x 3x) + x = P3 (x) + P1 (x),
Since m 6= n , in all cases, 5 5 5 5
Z b 8 4 1
x4 = P4 (x) + P2 (x) + P0 (x) etc;
p(x)ym (x)yn (x)dx = 0. 35 7 5
a Pk (x) is orthogonal to all polynomials of degree less than k.

Eigenfunction Expansions Preliminary Ideas
Real eigenvalues
Eigenfunctions of Sturm-Liouville problems:
Eigenvalues of a Sturm-Liouville problem are real.
convenient and powerful instruments to represent and
Let eigenvalue = + i and eigenfunction y (x) = u(x) + iv (x). manipulate fairly general classes of functions
Substitution leads to
[r (u 0 + iv 0 )]0 + [q + ( + i)p](u + iv ) = 0. {y0 , y1 , y2 , y3 , }: a family of continuous functions over [a, b],
mutually orthogonal with respect to p(x).
Separation of real and imaginary parts:
[ru 0 ]0 + (q + p)u pv = 0 pv 2 = [ru 0 ]0 v + (q + p)uv Representation of a function f (x) on [a, b]:
0 0 2 0 0
[rv ] + (q + p)v + pu = 0 pu = [rv ] u (q + p)uv X
f (x) = am ym (x) = a0 y0 (x) + a1 y1 (x) + a2 y2 (x) + a3 y3 (x) +
Adding together, m=0
0
p(u 2 + v 2 ) = [ru 0 ]0 v + [ru 0 ]v 0 [rv 0 ]u 0 [rv 0 ]0 u = r (uv 0 vu 0 )
Generalized Fourier series
Integration and application of boundary conditions leads to
Z b Analogous to the representation of a vector as a linear combination
of a set of mutually orthogonal vectors.
p(x)[u 2 (x) + v 2 (x)]dx = 0.
a
Question: How to determine the coefficients (a n )?
= 0 and =

Inner product: In terms of a finite number of members of the family { k (x)},

Z b N
X
(f , yn ) = p(x)f (x)yn (x)dx
a
N (x) = m m (x) = 0 0 (x)+1 1 (x)+2 2 (x)+ +N N (x).
Z
b X
X m=0
= [am p(x)ym (x)yn (x)]dx = am (ym , yn ) = an kyn k2 Error
a m=0 m=0
Z " N
#2
where
b X
s E = kf N k2 = p(x) f (x) m m (x) dx
p Z b
a m=0
kyn k = (yn , yn ) = p(x)yn2 (x)dx
a Error is minimized when
" #
Fourier coefficients: an = (fky,yn kn2) Z b N
X
E
Normalized eigenfunctions: = 2p(x) f (x) m m (x) [n (x)]dx = 0
n a m=0
ym (x)
m (x) = Z b Z b
kym (x)k
n p(x)2n (x)dx = p(x)f (x)n (x)dx.
Generalized Fourier series (in orthonormal basis): a a

X n = c n
f (x) = cm m (x) = c0 0 (x)+c1 1 (x)+c2 2 (x)+c3 3 (x)+ best approximation in the mean or least square approximation
m=0

Using the Fourier coefficients, error Question: Does it converge to f ?

N
X N
X N
X N
X Z b
E = (f , f )2 cn (f , n )+ cn2 (n , n ) = kf k2 2 cn2 + cn2 lim p(x)[sk (x) f (x)]2 dx = 0?
k a
n=0 n=0 n=0 n=0
N
X Answer: Depends on the basis used.
E = kf k2 cn2 0. Convergence in the mean or mean-square convergence:
n=0 An orthonormal set of functions {k (x)} on an interval
Bessels inequality: a x b is said to be complete in a class of functions,
N
X Z b or to form a basis for it, if the corresponding generalized
cn2 kf k2 = p(x)f 2 (x)dx Fourier series for a function converges in the mean to the
n=0 a function, for every function belonging to that class.
Partial sum P 2 2
k
Parsevals identity: n=0 cn = kf k
X
sk (x) = am m (x) Eigenfunction expansion: generalized Fourier series in terms of
m=0
eigenfunctions of a Sturm-Liouville problem
Question: Does the sequence of {sk } converge? I convergent for continuous functions with piecewise continuous
Answer: The bound in Bessels inequality ensures convergence. derivatives, i.e. they form a basis for this class.
Applied Mathematical Methods Sturm-Liouville Theory 463, Applied Mathematical Methods Fourier Series and Integrals 464,
Points to note Preliminary Ideas

Outline Basic Theory of Fourier Series
Extensions in Application
Fourier Integrals
I Eigenvalue problems in ODEs

I Self-adjoint differential operators
Fourier Series and Integrals
I Sturm-Liouville problems
Basic Theory of Fourier Series
I Orthogonal eigenfunctions Extensions in Application
I Eigenfunction expansions Fourier Integrals
Applied Mathematical Methods Fourier Series and Integrals 465, Applied Mathematical Methods Fourier Series and Integrals 466,
Basic Theory of Fourier Series Basic Theory of Fourier Series

Fourier Integrals
Fourier Integrals
With q(x) = 0 and p(x) = r (x) = 1, periodic S-L problem:

Dirichlets conditions:
y 00 + y = 0, y (L) = y (L), y 0 (L) = y 0 (L) If f (x) and its derivative are piecewise continuous on
[L, L] and are periodic with a period 2L, then the series
Eigenfunctions 1, cos x x 2x 2x
L , sin L , cos L , sin L , converges to the mean f (x+)+f (x)
of one-sided limits, at
2
constitute an orthogonal basis for representing functions. all points.
For a periodic function f (x) of period 2L, we propose
Fourier series

X nx nx Note: The interval of integration can be [x 0 , x0 + 2L] for any x0 .
f (x) = a0 + an cos + bn sin
L L
n=1 I It is valid to integrate the Fourier series term by term.
and determine the Fourier coefficients from Euler formulae I The Fourier series uniformly converges to f (x) over an
Z L interval on which f (x) is continuous. At a jump discontinuity,
1
a0 =
2L L
f (x)dx, convergence to f (x+)+f2
(x)
is not uniform. Mismatch peak
Z L Z shifts with inclusion of more terms (Gibbs phenomenon).
1 mx 1 L mx
am = f (x) cos dx and bm = f (x) sin dx. I Term-by-term differentiation of the Fourier series at a point
L L L L L L
requires f (x) to be smooth at that point.
Question: Does the series converge?

Fourier Integrals
Extensions in Application Basic Theory of Fourier Series
Fourier Integrals
Multiplying the Fourier series with f (x), Original spirit of Fouries series
h I representation of periodic functions over (, ).
X nx nx i
f 2 (x) = a0 f (x) + an f (x) cos + bn f (x) sin Question: What about a function f (x) defined only on [L, L]?
L L
n=1 Answer: Extend the function as
Parsevals identity:
Z F (x) = f (x) for L x L, and F (x + 2L) = F (x).

1X 2 1 L
a02 + (an + bn2 ) = 2
f (x)dx Fourier series of F (x) acts as the Fourier series representation of
2 2L L f (x) in its own domain.
n=1
The Fourier series representation is complete. In Euler formulae, notice that bm = 0 for an even function.
I A periodic function f (x) is composed of its mean value and The Fourier series of an even function is a Fourier
several sinusoidal components, or harmonics. cosine series
I Fourier coefficients are corresponding amplitudes.
X nx
I Parsevals identity is simply a statement on energy balance! f (x) = a0 + an cos ,
L
Bessels inequality n=1
N RL RL
1X 1 where a0 = 1
f (x)dx and an = 2
f (x) cos nx
L dx.
a02 + (an2 + bn2 ) kf (x)k2 L 0 L 0
2 2L
n=1
Similarly, for an odd function, Fourier sine series.

Fourier Integrals
Fourier Integrals
Over [0, L], sometimes we need a series of sine terms only, or Half-range expansions
cosine terms only! I For Fourier cosine series of a function f (x) over [0, L], even
f(x) fc(x) periodic extension:

f (x) for 0 x L,
fc (x) = and fc (x+2L) = fc (x)
f (x) for L x < 0,
O L x 3L 2L L O L 2L 3L x
I For Fourier sine series of a function f (x) over [0, L], odd
(a) Function over ( 0,L) (b) Even periodic extension periodic extension:

f (x) for 0 x L,
fs (x) = and fs (x+2L) = fs (x)
fs(x)
f (x) for L x < 0,
To develop the Fourier series of a function, which is available as a
set of tabulated values or a black-box library routine,
3L 2L L O L 2L 3L x
integrals in the Euler formulae are evaluated numerically.
Important: Fourier series representation is richer and more
(c) Odd periodic extension
powerful compared to interpolatory or least square approximation
Figure: Periodic extensions for cosine and sine series in many contexts.
Fourier Integrals Basic Theory of Fourier Series

Fourier Integrals
Fourier Integrals
Question: How to apply the idea of Fourier series to a In the limit (if it exists), as L , p 0,
non-periodic function over an infinite domain? Z Z Z
1
Answer: Magnify a single period to an infinite length. f (x) = cos px f (v ) cos pv dv + sin px f (v ) sin pv dv dp.
0
Fourier series of function fL (x) of period 2L:
Fourier integral of f (x):
X Z
fL (x) = a0 + (an cos pn x + bn sin pn x),
n=1
f (x) = [A(p) cos px + B(p) sin px]dp,
0
n
where pn = L is the frequency of the n-th harmonic. where amplitude functions
Z Z
Inserting the expressions for the Fourier coefficients, 1 1
A(p) = f (v ) cos pv dv and B(p) = f (v ) sin pv dv
Z L
1
fL (x) = fL (x)dx are defined for a continuous frequency variable p.
2L L
Z L Z L
1X In phase angle form,
+ cos pn x fL (v ) cos pn v dv + sin pn x fL (v ) sin pn v dv p, Z
Z
n=1 L L 1
f (x) = f (v ) cos p(x v )dv dp.

where p = pn+1 pn = L . 0

Fourier Integrals
Points to note Basic Theory of Fourier Series
Fourier Integrals
e i +e i
Using cos = 2 in the phase angle form,
Z Z
1
f (x) = f (v )[e ip(xv ) + e ip(xv ) ]dv dp.
2 0 I Fourier series arising out of a Sturm-Liouville problem
With substitution p = q, I A versatile tool for function representation
Z Z Z 0 Z
I Fourier integral as the limiting case of Fourier series
f (v )e ip(xv ) dv dp = f (v )e iq(xv ) dv dq.
0
Complex form of Fourier integral

Z Z Z Necessary Exercises: 1,3,6,8
1
f (x) = f (v )e ip(xv ) dv dp = C (p)e ipx dp,
2
in which the complex Fourier integral coefficient is

Z
1
C (p) = f (v )e ipv dv .
2
Applied Mathematical Methods Fourier Transforms 475, Applied Mathematical Methods Fourier Transforms 476,
Outline Definition and Fundamental Properties

Important Results on Fourier Transforms
Discrete Fourier Transform
Definition and Fundamental Properties
Complex form of the Fourier integral:

Z Z
1 1
f (t) = f (v )e iwv dv e iwt dw
2 2
Fourier Transforms Composition of an infinite number of functions in the

Definition and Fundamental Properties e iwt
form 2
, over a continuous distribution of frequency w .
Discrete Fourier Transform Fourier transform: Amplitude of a frequency component:
Z
1
F(f ) f(w ) = f (t)e iwt dt
2
Function of the frequency variable.
Inverse Fourier transform
Z
1
F 1 (f) f (t) = f(w )e iwt dw
2
recovers the original function.

Example: Fourier transform of f (t) = 1? Fourier transform of the derivative of a function:

Let us find out the inverse Fourier transform of f(w ) = k(w ).
Z If f (t) is continuous
R in every interval and f 0 (t) is piecewise
1 k continuous, |f (t)|dt converges and f (t) approaches zero as
f (t) = F 1 (f) = k(w )e iwt dw =
2 2 t , then
Z
F(1) = 2(w ) 1
F{f 0 (t)} = f 0 (t)e iwt dt
2
Linearity of Fourier transforms: Z
1 1
= f (t)e iwt (iw )f (t)e iwt dt
F{f1 (t) + f2 (t)} = f1 (w ) + f2 (w ) 2 2
Scaling: = iw f(w ).
1 w n w o
Alternatively, differentiating the inverse Fourier transform,
F{f (at)} = f and F 1 f = |a|f (at)
|a| a a Z
d d 1
Shifting rules: [f (t)] = f(w )e iwt dw
dt dt 2
Z h i
F{f (t t0 )} = e iwt0 F{f (t)} 1
= f (w )e iwt dw = F 1 {iw f(w )}.
F {f(w w0 )} = e iw0 t F 1 {f(w )}
1 2 t

Under appropriate premises, Convolution of two functions:

00 2 2
Z
F{f (t)} = (iw ) f (w ) = w f (w ).
h(t) = f (t) g (t) = f ( )g (t )d

In general, F{f (n) (t)} = (iw )n f(w ).
Fourier transform of an integral:
h(w ) = F{h(t)}
f (t) is piecewise continuous on every interval,
RIf Z Z
1
|f (t)|dt converges and f (0) = 0, then = f ( )g (t )e iwt d dt
2
Z t Z Z
1 1
F f ( )d = f (w ). = f ( )e iw g (t )e iw (t ) dt d
iw 2
Z Z
iw 1 0
= f ( )e g (t 0 )e iwt dt 0 d
Derivative of a Fourier transform (with respect to the frequency 2
variable):
dn Convolution theorem for Fourier transforms:
F{t n f (t)} = i n n f(w ),
dw
R h(w ) = 2 f(w )g (w )
if f (t) is piecewise continuous and |t n f (t)|dt converges.

Discrete Fourier Transform Definition and Fundamental Properties
Conjugate of the Fourier transform: Consider a signal f (t) from actual measurement or sampling.
Z
1 We want to analyze its amplitude spectrum (versus frequency).
f (w ) = f (t)e iwt dt
2 For the FT, how to evaluate the integral over (, )?
Inner product of f(w ) and g (w ): Windowing: Sample the signal f (t) over a finite interval.
Z Z Z
1 A window function:
f (w )g (w )dw = f (t)e iwt dt g (w )dw
2
Z Z 1 for a t b
1 g (t) =
= f (t) g (w )e iwt dw dt 0 otherwise
2
Z
= f (t)g (t)dt. Actual processing takes place on the windowed function f (t)g (t).

Next question: Do we need to evaluate the amplitude for all
Parsevals identity: For g (t) = f (t) in the above, w (, )?
Z Z
kf(w )k2 dw = kf (t)k2 dt, Most useful signals are particularly rich only in their own
characteristic frequency bands.
equating the total energy content of the frequency spectrum of a Decide on an expected frequency band, say [w c , wc ].
wave or a signal to the total energy flow over time.

Time step for sampling? With discrete data at tk = k for k = 0, 1, 2, 3, , N 1,
With N sampling over [a, b),
h ki
f(w) = mj f(t),
wc , 2
h i
data being collected at t = a, a + , a + 2, , a + (N 1), where mj = e iwj and mjk is an N N matrix.
with N = b a. A similar discrete version of inverse Fourier transform.
Nyquist critical frequency
Reconstruction: a trigonometric interpolation of sampled data.
Note the duality.
I Structure of Fourier and inverse Fourier transforms reduces the
I Decision of sampling rate determines the band of frequency
problem with a system of linear equations [O(N 3 ) operations]
content that can be accommodated.
to that of a matrix-vector multiplication [O(N 2 ) operations].
I Decision of the interval [a, b) dictates how finely the h i
frequency spectrum can be developed. I Structure of matrix mjk , with patterns of redundancies,
opens up a trick to reduce it further to O(N log N) operations.
Shannons sampling theorem
A band-limited signal can be reconstructed from a finite Cooley-Tuckey algorithm:
number of samples. fast Fourier transform (FFT)

Points to note Definition and Fundamental Properties
DFT representation reliable only if the incoming signal is really
band-limited in the interval [wc , wc ].
Frequencies beyond [wc , wc ] distort the spectrum near w = wc
by folding back. I Fourier transform as amplitude function in Fourier integral
Aliasing I Basic operational tools in Fourier and inverse Fourier
Detection: a posteriori transforms
Bandpass filtering: If we expect a signal having components only I Conceptual notions of discrete Fourier transform (DFT)
in certain frequency bands and want to get rid of unwanted noise
frequencies,
for every band [w1 , w2 ] of our interest, we define window
function (w ) with intervals [w2 , w1 ] and [w1 , w2 ]. Necessary Exercises: 1,3,6
Windowed Fourier transform (w )f(w ) filters out frequency
components outside this band.
For recovery,
convolve raw signal f (t) with IFT (t) of (w ).
Applied Mathematical Methods Minimax Approximation* 487, Applied Mathematical Methods Minimax Approximation* 488,
Outline Approximation with Chebyshev polynomials

Minimax Polynomial Approximation Approximation with Chebyshev polynomials
Approximation with Chebyshev polynomials
Minimax Polynomial Approximation
Chebyshev polynomials:
Polynomial solutions of the singular Sturm-Liouville problem
hp i0 n2
(1 x 2 )y 00 xy 0 + n2 y = 0 or 1 x2 y0 + y =0
1 x2
Minimax Approximation*
over 1 x 1, with Tn (1) = 1 for all n.
Closed-form expressions:
Tn (x) = cos(n cos1 x),
or,
T0 (x) = 1, T1 (x) = x, T2 (x) = 2x 2 1, T3 (x) = 4x 3 3x, ;
with the three-term recurrence relation
Tk+1 (x) = 2xTk (x) Tk1 (x).

Immediate observations
I Coefficients in a Chebyshev polynomial are integers. In 1
1
P 8 (x)
T8 (x)
0.8
particular, the leading coefficient of T n (x) is 2n1 . 0.8

0.6
0.6
I For even n, Tn (x) is an even function, while for odd n it is an 0.4 0.4
odd function. 0.2 0.2

T3 (x)
I Tn (1) = 1, Tn (1) = (1)n and |Tn (x)| 1 for 1 x 1.

0
y
0.2
0.2
I Zeros of a Chebyshev polynomial T n (x) are real and lie inside 0.4
0.4
0.6
the interval [1, 1] at locations x = cos (2k1)

2n for 0.8
extrema
zeroes
0.6
1 0.8
k = 1, 2, 3, , n. 1.5 1 0.5 0 0.5 1 1.5 1

1 0.8 0.6 0.4 0.2 0 0.2 0.4 0.6 0.8 1
These locations are also called Chebyshev accuracy points. x x
Further, zeros of Tn (x) are interlaced by those of Tn+1 (x). Figure: Extrema and zeros of T3 (x) Figure: Contrast: P8 (x) and T8 (x)
I Extrema of Tn (x) are of magnitude equal to unity, alternate in
sign and occur at x = cos k n for k = 0, 1, 2, 3, , n. Being cosines and polynomials at the same time, Chebyshev
I Orthogonality and norms:
polynomials possess a wide variety of interesting properties!
Z 1 0 if m 6= n,
Tm (x)Tn (x)
dx = 2 if m = n 6= 0, and Most striking property:
1 1 x2
if m = n = 0. equal-ripple oscillations, leading to minimax property

Minimax property
Chebyshev series
Theorem: Among all polynomials pn (x) of degree n > 0
with the leading coefficient equal to unity, 2 1n Tn (x) f (x) = a0 T0 (x) + a1 T1 (x) + a2 T2 (x) + a3 T3 (x) +
deviates least from zero in [1, 1]. That is,
with coefficients
max |pn (x)| max |21n Tn (x)| = 21n . Z Z
1x1 1x1 1 1 f (x)T0 (x) 2 1 f (x)Tn (x)
a0 = dx and an = dx for n = 1, 2, 3,
1 1 x2 1 1 x2
If there exists a monic polynomial p n (x) of degree n such that
max |pn (x)| < 21n , Pn
1x1 A truncated series k=0 ak Tk (x):
then at (n + 1) locations of alternating extrema of 2 1n T n (x), the Chebyshev economization

polynomial
Leading error term an+1 Tn+1 (x) deviates least from zero over
qn (x) = 21n Tn (x) pn (x)
[1, 1] and is qualitatively similar to the error function.
will have the same sign as 21n Tn (x).
With alternating signs at (n + 1) locations in sequence, q n (x) will Question: How to develop a Chebyshev series approximation?
have n intervening zeros, even though it is a polynomial of degree Find out so many Chebyshev polynomials and evaluate coefficients?
at most (n 1): CONTRADICTION!

Minimax Polynomial Approximation Minimax Polynomial Approximation Approximation with Chebyshev polynomials
For approximating f (t) over [a, b], scale the variable as Situations in which minimax approximation is desirable:
I Develop the approximation once and keep it for use in future.
t = a+b ba
2 + 2 x, with x [1, 1].
P Requirement: Uniform quality control over the entire domain
Remark: The economized series nk=0 ak Tk (x) gives minimax
deviation of the leading error term a n+1 Tn+1 (x). Minimax approximation:
Assuming an+1 Tn+1 (x) to be the error, at the zeros of Tn+1 (x), deviation limited by the constant amplitude of ripple
the error will be officially zero, i.e.
Chebyshevs minimax theorem
n
X Theorem: Of all polynomials of degree up to n, p(x) is
ak Tk (xj ) = f (t(xj )), the minimax polynomial approximation of f (x), i.e. it
k=0
minimizes
where x0 , x1 , x2 , , xn are the roots of Tn+1 (x). max |f (x) p(x)|,
Recall: Values of an n-th degree polynomial at n + 1 if and only if there are n + 2 points xi such that
points uniquely fix the entire polynomial.
a x1 < x2 < x3 < < xn+2 b,
Interpolation of these n + 1 values leads to the same polynomial!
Chebyshev-Lagrange approximation where the difference f (x) p(x) takes its extreme values
of the same magnitude and alternating signs.

Minimax Polynomial Approximation Points to note Approximation with Chebyshev polynomials
Utilize any gap to reduce the deviation at the other extrema with
values at the bound.
y
d I Unique features of Chebyshev polynomials
f(x) p(x) I The equal-ripple and minimax properties
p(x)
/2 I Chebyshev series and Chebyshev-Lagrange approximation
l b
a w n
O
/2
m x I Fundamental ideas of general minimax approximation

Figure: Schematic of an approximation that is not minimax
Construction of the minimax polynomial: Remez algorithm
Note: In the light of this theorem and algorithm, examine how

Tn+1 (x) is qualitatively similar to the complete error function!
Applied Mathematical Methods Partial Differential Equations 497, Applied Mathematical Methods Partial Differential Equations 498,
Hyperbolic Equations
Parabolic Equations
Parabolic Equations
Elliptic Equations
Two-Dimensional Wave Equation
Quasi-linear second order PDEs Elliptic Equations
2u 2u 2u
a + 2b + c 2 = F (x, y , u, ux , uy )
x 2 xy y
Partial Differential Equations hyperbolic if b 2 ac > 0, modelling phenomena which evolve in

Introduction time perpetually and do not approach a steady state
Hyperbolic Equations parabolic if b 2 ac = 0, modelling phenomena which evolve in
Parabolic Equations time in a transient manner, approaching steady state
Elliptic Equations elliptic if b 2 ac < 0, modelling steady-state configurations,
Two-Dimensional Wave Equation without evolution in time
If F (x, y , u, ux , uy ) = 0,
second order linear homogeneous differential equation
Principle of superposition: A linear combination of different
solutions is also a solution.
Solutions are often in the form of infinite series.
I Solution techniques in PDEs typically attack the boundary
value problem directly.
Parabolic Equations
Parabolic Equations
Elliptic Equations
Method of separation of variables Elliptic Equations
Initial and boundary conditions For u(x, y ), propose a solution in the form
Time and space variables are qualitatively different.
u(x, y ) = X (x)Y (y )
I Conditions in time: typically initial conditions.
For second order PDEs, u and ut over the entire space and substitute
domain: Cauchy conditions
I Time is a single variable and is decoupled from the space ux = X 0 Y , uy = XY 0 , uxx = X 00 Y , uxy = X 0 Y 0 , uyy = XY 00
variables.
to cast the equation into the form
I Conditions in space: typically boundary conditions.
For u(t, x, y ), boundary conditions over the entire curve in the (x, X , X 0 , X 00 ) = (y , Y , Y 0 , Y 00 ).
x-y plane that encloses the domain. For second order PDEs,
I Dirichlet condition: value of the function If the manoeuvre succeeds then, x and y being independent
I Neumann condition: derivative normal to the boundary variables, it implies
I Mixed (Robin) condition
(x, X , X 0 , X 00 ) = (y , Y , Y 0 , Y 00 ) = k.
Dirichlet, Neumann and Cauchy problems Nature of the separation constant k is decided based on the
context, resulting ODEs are solved in consistency with the
boundary conditions and assembled to construct u(x, y ).
Hyperbolic Equations Introduction

Parabolic Equations
Parabolic Equations
Elliptic Equations Elliptic Equations
Transverse vibrations of a string Two-Dimensional Wave Equation Two-Dimensional Wave Equation
Under the assumptions, denoting c 2 = T
,
Q T
u
+
" #
2u u u
P
x x = c2 .

T
t 2 x Q x P
Q
P In the limit, as x 0, PDE of transverse vibration:
2u 2u
O x L x
= c2 2
t 2 x
Figure: Transverse vibration of a stretched string
one-dimensional wave equation
Small deflection and slope: cos 1, sin tan Boundary conditions (in this case): u(0, t) = u(L, t) = 0
Horizontal (longitudinal) forces on PQ balance. Initial configuration and initial velocity:
From Newtons second law, vertical (transverse) deflection u(x, t):
u(x, 0) = f (x) and ut (x, 0) = g (x)
2u
T sin( + ) T sin = x 2 Cauchy problem: Determine u(x, t) for 0 x L, t 0.
t

Parabolic Equations
Parabolic Equations
Solution by separation of variables Two-Dimensional Wave Equation Two-Dimensional Wave Equation
Corresponding solution:
2 Tn (t) = An cos n t + Bn sin n t
utt = c uxx , u(0, t) = u(L, t) = 0, u(x, 0) = f (x), u t (x, 0) = g (x)
Assuming Then, for n = 1, 2, 3, ,
u(x, t) = X (x)T (t),
nx
and substituting utt = XT 00 and uxx = X 00 T , variables are un (x, t) = Xn (x)Tn (t) = (An cos n t + Bn sin n t) sin
L
separated as
T 00 X 00 satisfies the PDE and the boundary conditions.
2
= = p 2 .
c T X
Since the PDE and the BCs are homogeneous, by superposition,
The PDE splits into two ODEs
00 2 00 2 2
X
X +p X =0 and T + c p T = 0. nx
u(x, t) = [An cos n t + Bn sin n t] sin .
Eigenvalues of BVP X 00 + p2X = 0, X (0) = X (L) = 0 are p = n L
L n=1
and eigenfunctions
nx Question: How to determine coefficients A n and Bn ?
for n = 1, 2, 3, .
Xn (x) = sin px = sin
L
00
Answer: By imposing the initial conditions.
Second ODE: T + n T = 0, with n = cn
2
L

Parabolic Equations
Parabolic Equations
Initial conditions: Fourier sine series of f (x) and g (x)
Two-Dimensional Wave Equation Two-Dimensional Wave Equation
DAlemberts solution of the wave equation

X nx
u(x, 0) = f (x) = An sin
L Method of characteristics
n=1

X Canonical form
nx
ut (x, 0) = g (x) = n Bn sin
L By coordinate transformation from (x, y ) to (, ), with
n=1
Hence, coefficients: U(, ) = u[x(, ), y (, )],
Z Z L hyperbolic equation: U =
2 L nx 2 nx
An = f (x) sin dx and Bn = g (x) sin dx parabolic equation: U =
L 0 L cn 0 L
Related problems: elliptic equation: U + U =
I Different boundary conditions: other kinds of series in which (, , U, U , U ) is free from second derivatives.
I Long wire: infinite domain, continuous frequencies and
For a hyperbolic equation, entire domain becomes a network of -
solution from Fourier integrals
coordinate curves, known as characteristic curves,
Alternative: Reduce the problem using Fourier transforms.
I General wave equation in 3-d: u tt = c 2 2 u along which decoupled solutions can be tracked!
I Membrane equation: utt = c 2 (uxx + uyy )

Parabolic Equations
Parabolic Equations
For a hyperbolic equation in the form Two-Dimensional Wave Equation
Substitution of derivatives Two-Dimensional Wave Equation
2u 2u 2u
a + 2b + c 2 = F (x, y , u, ux , uy ), ux = U x + U x = U + U uxx = U + 2U + U
x 2 xy y
roots of am 2 + 2bm + c are ut = U t + U t = cU + cU utt = c 2 U 2c 2 U + c 2 U

b b 2 ac into the PDE utt = c 2 uxx gives
m1,2 = ,
a
real and distinct. c 2 (U 2U + U ) = c 2 (U + 2U + U ).
Coordinate transformation
Canonical form: U = 0
= y + m1 x, = y + m2 x
leads to U = (, , U, U , U ). Integration: Z
For the BVP U = U d + () = ()
utt = c 2 uxx , u(0, t) = u(L, t) = 0, u(x, 0) = f (x), u t (x, 0) = g (x), Z
U(, ) = ()d + f2 () = f1 () + f2 ()
canonical coordinate transformation:
1 1
= x ct, = x + ct, with x = ( + ), t = ( ). DAlemberts solution: u(x, t) = f1 (x ct) + f2 (x + ct)
2 2c

Parabolic Equations
Parabolic Equations Introduction
Parabolic Equations
Two-Dimensional Wave Equation Heat conduction equation or diffusion equation: Two-Dimensional Wave Equation
Physical insight from DAlemberts solution:
u
f1 (x ct): a progressive wave in forward direction with speed c = c 2 2 u
t
Reflection at boundary: One-dimensional heat (diffusion) equation:
in a manner depending upon the boundary condition ut = c 2 uxx
Reflected wave f2 (x + ct): another progressive wave, this one in Heat conduction in a finite bar: For a thin bar of length L with
backward direction with speed c end-points at zero temperature,
Superposition of two waves: complete solution (response) ut = c 2 uxx , u(0, t) = u(L, t) = 0, u(x, 0) = f (x).
cn
Note: Components of the earlier solution: with n = L , Assumption u(x, t) = X (x)T (t) leads to
nx 1 h n n i
T0 X 00
cos n t sin = sin (x ct) + sin (x + ct) XT 0 = c 2 X 00 T = = p 2 ,
L 2 L L c 2T X
nx 1h n n i
sin n t sin = cos (x ct) cos (x + ct) giving rise to two ODEs as
L 2 L L
X 00 + p 2 X = 0 and T 0 + c 2 p 2 T = 0.

Parabolic Equations
Parabolic Equations
BVP in the space coordinate X 00 + p 2 X = 0, X (0) = X (L) = 0
Two-Dimensional Wave Equation Non-homogeneous boundary conditions: Two-Dimensional Wave Equation
has solutions ut = c 2 uxx , u(0, t) = u1 , u(L, t) = u2 , u(x, 0) = f (x).

nx
Xn (x) = sin .
L For u1 6= u2 , with u(x, t) = X (x)T (t), BCs do not separate!
cn
With n = L , the ODE in T (t) has the corresponding solutions Assume
2 u(x, t) = U(x, t) + uss (x),
Tn (t) = An e n t .
where component uss (x), steady-state temperature (distribution),
By superposition, does not enter the differential equation.

X 00 u2 u 1
nx 2n t uss (x) = 0, uss (0) = u1 , uss (L) = u2 uss (x) = u1 + x
u(x, t) = An sin e , L
L
n=1 Substituting into the BVP,
coefficients being determined from initial condition as Ut = c 2 Uxx , U(0, t) = U(L, t) = 0, U(x, 0) = f (x) u ss (x).

X nx Final solution:
u(x, 0) = f (x) = An sin ,
L X nx 2n t
n=1 u(x, t) = Bn sin e + uss (x),
L
a Fourier sine series. n=1
As t , u(x, t) 0 (steady state) Bn being coefficients of Fourier sine series of f (x) u ss (x).

Parabolic Equations
Parabolic Equations
Heat conduction in an infinite wire Two-Dimensional Wave Equation Solution using Fourier transforms Two-Dimensional Wave Equation
ut = c 2 uxx , u(x, 0) = f (x) ut = c 2 uxx , u(x, 0) = f (x)

In place of n Using derivative formula of Fourier transforms,
L ,
now we have continuous frequency p.
Solution as superposition of all frequencies: u
F(ut ) = c 2 (iw )2 F(u) = c 2 w 2 u,
Z Z t
2 2
u(x, t) = up (x, t)dp = [A(p) cos px+B(p) sin px]e c p t dp since variables x and t are independent.
0 0 Initial value problem in u(w , t):
Initial condition u
= c 2 w 2 u, u(0) = f(w )
Z t
2 2
u(x, 0) = f (x) = [A(p) cos px + B(p) sin px]dp Solution: u(w , t) = f(w )e c w t
0
Inverse Fourier transform gives solution of the original problem as
Z
gives the Fourier integral of f (x) and amplitude functions 1 2 2
u(x, t) = F 1 {u(w , t)} = f(w )e c w t e iwx dw
Z Z 2
1 1 Z Z
A(p) = f (v ) cos pv dv and B(p) = f (v ) sin pv dv . 1 2 2
u(x, t) = f (v ) cos(wx wv )e c w t dw dv .
0
Elliptic Equations Introduction

Parabolic Equations
Parabolic Equations
Heat flow in a plate: two-dimensional heat equation Two-Dimensional Wave Equation From BVP X 00 + p 2 X = 0, X (0) = X (a) = 0, X (x) = sin nx
n Wave Equation
Two-Dimensional
a
2 Corresponding solution of Y 00 p 2 Y = 0:
u u 2u
= c2 +
t x 2 y 2 ny ny
Yn (y ) = An cosh + Bn sinh
Steady-state temperature distribution: a a
2u 2u Condition Y (0) = 0 An = 0, and
+ =0
x 2 y 2 nx ny
un (x, y ) = Bn sin sinh
Laplaces equation a a
Steady-state heat flow in a rectangular plate: The complete solution:
uxx + uyy = 0, u(0, y ) = u(a, y ) = u(x, 0) = 0, u(x, b) = f (x);
X nx ny
a Dirichlet problem over the domain 0 x a, 0 y b. u(x, y ) = Bn sin sinh
a a
n=1
Proposal u(x, y ) = X (x)Y (y ) leads to
X 00 Y 00 The last boundary condition u(x, b) = f (x) fixes the coefficients
X 00 Y + XY 00 = 0 = = p 2 . from the Fourier sine series of f (x).
X Y
Separated ODEs: Note: In the example, BCs on three sides were homogeneous.
X 00 + p 2 X = 0 and Y 00 p 2 Y = 0 How did it help? What if there are more non-homogeneous BCs?

Parabolic Equations
Two-Dimensional Wave Equation Introduction
Parabolic Equations
Steady-state heat flow with internal heat generation

Transverse vibration of a rectangular membrane: Two-Dimensional Wave Equation
2
2u u 2u
2 u = (x, y ) 2
= c2 2
+ 2
t x y
Poissons equation
A Cauchy problem of the membrane:
Separation of variables impossible!
utt = c 2 (uxx + uyy ); u(x, y , 0) = f (x, y ), ut (x, y , 0) = g (x, y );
Consider function u(x, y ) as
u(0, y , t) = u(a, y , t) = u(x, 0, t) = u(x, b, t) = 0.
u(x, y ) = uh (x, y ) + up (x, y )
Separate the time variable from the space variables:
Sequence of steps
Fxx + Fyy T 00
I one particular solution up (x, y ) that may or may not satisfy u(x, y , t) = F (x, y )T (t) = 2 = 2
F c T
some or all of the boundary conditions
I solution of the corresponding homogeneous equation, namely Helmholtz equation:
uxx + uyy = 0 for uh (x, y )
I such that u = uh + up satisfies all the boundary conditions Fxx + Fyy + 2 F = 0

Parabolic Equations
Parabolic Equations
Assuming F (x, y ) = X (x)Y (y ), Two-Dimensional Wave Equation Composing Xm (x), Yn (y ) and Tmn (t) and superposing, Two-Dimensional Wave Equation
X
X
X 00 Y 00 + 2 Y mx ny
= = 2 u(x, y , t) = [Amn cos cmn t+Bmn sin cmn t] sin sin ,
X Y a b
m=1 n=1
X 00 + 2 X = 0 and Y 00 + 2 Y = 0, coefficients being determined from the double Fourier series
p X
such that = 2 + 2 . X mx ny
f (x, y ) = Amn sin sin
With BCs X (0) = X (a) = 0 and Y (0) = Y (b) = 0, a b
m=1 n=1
X X
mx ny mx ny
Xm (x) = sin and Yn (y ) = sin . and g (x, y ) = cmn Bmn sin sin .
a b a b
m=1 n=1
Corresponding values of are
r
m 2 n 2 BVPs modelled in polar coordinates
mn = + For domains of circular symmetry, important in many practical
a b
systems, the BVP is conveniently modelled in polar coordinates,
with solutions of T 00 + c 2 2 T = 0 as
the separation of variables quite often producing
Tmn (t) = Amn cos cmn t + Bmn sin cmn t. I Bessels equation, in cylindrical coordinates, and
I Legendres equation, in spherical coordinates
Applied Mathematical Methods Partial Differential Equations 521, Applied Mathematical Methods Analytic Functions 522,

Parabolic Equations
Outline Analyticity of Complex Functions
Conformal Mapping
Potential Theory
Elliptic Equations
I PDEs in physically relevant contexts

I Initial and boundary conditions
I Separation of variables Analytic Functions
I Examples of boundary value problems with hyperbolic, Analyticity of Complex Functions
parabolic and elliptic equations Conformal Mapping
I Modelling, solution and interpretation Potential Theory
I Cascaded application of separation of variables for problems
with more than two independent variables

Applied Mathematical Methods Analytic Functions 523, Applied Mathematical Methods Analytic Functions 524,
Analyticity of Complex Functions Analyticity of Complex Functions

Conformal Mapping
Potential Theory
Conformal Mapping
Potential Theory
Function f of a complex variable z Derivative of a complex function:

gives a rule to associate a unique complex number
f (z) f (z0 ) f (z0 + z) f (z0 )
w = u + iv to every z = x + iy in a set. f 0 (z0 ) = lim = lim
zz0 z z0 z0 z
Limit: If f (z) is defined in a neighbourhood of z 0 (except possibly
When this limit exists, function f (z) is said to be differentiable.
at z0 itself) and l C such that > 0, > 0 such that
Extremely restrictive definition!
0 < |z z0 | < |f (z) l| < ,
Analytic function
then A function f (z) is called analytic in a domain D if it is
l = lim f (z). defined and differentiable at all points in D.
zz0
Crucial difference from real functions: z can approach z 0 in all Points to be settled later:
possible manners in the complex plane. I Derivative of an analytic function is also analytic.
Definition of the limit is more restrictive. I An analytic function possesses derivatives of all orders.
Continuity: limzz0 f (z) = f (z0 ) A great qualitative difference between functions of a real variable
Continuity in a domain D: continuity at every point in D and those of a complex variable!

Conformal Mapping
Potential Theory
Conformal Mapping
Potential Theory
Cauchy-Riemann conditions Cauchy-Riemann equations or conditions

If f (z) = u(x, y ) + iv (x, y ) is analytic then u v u
= v
x = y and y x
u + iv are necessary for analyticity.
f 0 (z) = lim
x,y 0 x + iy Question: Do the C-R conditions imply analyticity?
along all paths of approach for z = x + iy 0 or x, y 0. Consider u(x, y ) and v (x, y ) having continuous first order partial
derivatives that satisfy the Cauchy-Riemann conditions.
y
3
y
z = iy
By mean value theorem,
2
z0
u u
z0 z = x u = u(x + x, y + y ) u(x, y ) = x (x1 , y1 ) + y (x1 , y1 )
4 1
x y
O x
5 O x with x1 = x + x, y1 = y + y for some [0, 1]; and
v v
v = v (x + x, y + y ) v (x, y ) = x (x2 , y2 ) + y (x2 , y2 )
x y
Figure: Paths approaching z0 Figure: Paths in C-R equations with x2 = x + x, y2 = y + y for some [0, 1].
Two expressions for the derivative: Then,

u v v u u v v u
f 0 (z) = +i = i f = x (x1 , y1 ) + iy (x2 , y2 ) +i x (x2 , y2 ) iy (x1 , y1 ) .
x x y y x y x y

Conformal Mapping
Potential Theory
Conformal Mapping
Potential Theory
v u u
Using C-R conditions y = v
= x ,
x and y Harmonic function
Differentiating C-R equations v
= u
and u
= v
u u u y x y x ,
f = (x + iy ) (x1 , y1 ) + iy (x2 , y2 ) (x1 , y1 )
x x x
2u 2v 2u 2v 2u 2v 2u 2v
v v v = , = , = , = 2
+ i(x + iy ) (x1 , y1 ) + ix (x2 , y2 ) (x1 , y1 ) x 2 xy y 2 y x y x y 2 xy x
x x x
f u v 2u 2u 2v 2v
= (x1 , y1 ) + i (x1 , y1 ) + + 2 =0= + 2.
z x
x x 2 y x 2 y
x v v y u u
i (x2 , y2 ) (x1 , y1 ) + i (x2 , y2 ) (x1 , y1 ) .
z x x z x x Real and imaginary components of an analytic functions
x y are harmonic functions.

Since z , z 1, as z 0, the limit exists and
Conjugate harmonic function of u(x, y ): v (x, y )
0 u v u v Families of curves u(x, y ) = c and v (x, y ) = k are mutually
f (z) = +i = i + .
x x y y orthogonal, except possibly at points where f 0 (z) = 0.
Cauchy-Riemann conditions are necessary and sufficient Question: If u(x, y ) is given, then how to develop the complete
for function w = f (z) = u(x, y ) + iv (x, y ) to be analytic. analytic function w = f (z) = u(x, y ) + iv (x, y )?
Conformal Mapping Analyticity of Complex Functions

Conformal Mapping
Potential Theory
Conformal Mapping
Potential Theory
Function: mapping of elements in domain to their images in range Conformal mapping: a mapping that preserves the angle between
Depiction of a complex variable requires a plane with two axes. any two directions in magnitude and sense.
Mapping of a complex function w = f (z) is shown in two planes. Verify: w = e z defines a conformal mapping.
Example: mapping of a rectangle under transformation w = e z Through relative orientations of curves at the points of
3.5
intersection, local shape of a figure is preserved.
2
1.5
D C C Take curve z(t), z(0) = z0 and image w (t) = f [z(t)], w0 = f (z0 ).
For analytic f (z), w (0) = f 0 (z0 )z(0), implying
2.5
2
1
1.5 |w (0)| = |f 0 (z0 )| |z(0)| and arg w (0) = arg f 0 (z0 ) + arg z(0).
v
0.5
y
1 D
0.5
For several curves through z0 ,
0
O A B
0
O B
image curves pass through w0 and all of them turn by the
same angle arg f 0 (z0 ).
0.5 A
0.5
1 1
1 0.5 0 0.5 1 1.5 2 1 0.5 0 0.5 1 1.5 2 2.5 3 3.5
x u
Cautions
(a) The z-plane (b) The w -plane I f 0 (z) varies from point to point. Different scaling and turning
effects take place at different points. Global shape changes.
Figure: Mapping corresponding to function w = e z I For f 0 (z) = 0, argument is undefined and conformality is lost.

Conformal Mapping
Potential Theory
Potential Theory Analyticity of Complex Functions
Conformal Mapping
Potential Theory
An analytic function defines a conformal mapping except Riemann mapping theorem: Let D be a simply connected
at its critical points where its derivative vanishes. domain in the z-plane bounded by a closed curve C . Then there
exists a conformal mapping that gives a one-to-one correspondence
Except at critical points, an analytic function is invertible.
between D and the unit disc |w | < 1 as well as between C and the
We can establish an inverse of any conformal mapping. unit circle |w | = 1, bounding the unit disc.
Examples Application to boundary value problems
I Linear function w = az + b (for a 6= 0) I First, establish a conformal mapping between the given
I Linear fractional transformation domain and a domain of simple geometry.
az + b I Next, solve the BVP in this simple domain.
w= , ad bc 6= 0
cz + d I Finally, using the inverse of the conformal mapping, construct
I Other elementary functions like z n , e z etc the solution for the given domain.
Special significance of conformal mappings:
Example: Dirichlet problem with Poissons integral formula
A harmonic function (u, v ) in the w -plane is also a
Z 2
harmonic function, in the form (x, y ) in the z-plane, as 1 (R 2 r 2 )f (Re i )
long as the two planes are related through a conformal f (re i ) = d
2 0 R 2 2Rr cos( ) + r 2
mapping.
Potential Theory Analyticity of Complex Functions

Conformal Mapping
Potential Theory
Points to note Analyticity of Complex Functions
Conformal Mapping
Potential Theory
Two-dimensional potential flow

I Velocity potential (x, y ) gives velocity components V x = x
and Vy = I Analytic functions and Cauchy-Riemann conditions
y .
I A streamline is a curve in the flow field, the tangent to which I Conformality of analytic functions
at any point is along the local velocity vector. I Applications in solving BVPs and flow description
I Stream function (x, y ) remains constant along a streamline.
I (x, y ) is the conjugate harmonic function of (x, y ).
I Complex potential function (z) = (x, y ) + i(x, y ) defines
the flow.
If a flow field encounters a solid boundary of a complicated shape,

transform the boundary conformally to a simple boundary
to facilitate the study of the flow pattern.
Applied Mathematical Methods Integrals in the Complex Plane 535, Applied Mathematical Methods Integrals in the Complex Plane 536,
Outline Line Integral

Cauchys Integral Theorem
Cauchys Integral Formula
Line Integral Line Integral
For w = f (z) = u(x, y ) + iv (x, y ), over a smooth curve C ,

Z Z Z Z
f (z)dz = (u+iv )(dx+idy ) = (udxvdy )+i (vdx+udy ).
C C C C
Extension to piecewise smooth curves is obvious.
Integrals in the Complex Plane
Line Integral With parametrization, for z = z(t), a t b, with z(t) 6= 0,
Z Z b
f (z)dz = f [z(t)]z(t)dt.
Cauchys Integral Formula C a
H
Over a simpleH closed curve, contour integral: C f (z)dz
Example: C z n dz for integer n, around circle z = e i
I Z 2
0 for n 6= 1,
z n dz = in+1 e i (n+1) d =
C 0 2i for n = 1.
The M-L inequality: If C is a curve of finite length L and
|f (z)| < M on C , then
Z Z Z

f (z)dz |f (z)| |dz| < M |dz| = ML.

C C C
Cauchys Integral Theorem Line Integral

I C is a simple closed curve in a simply connected domain D. Principle of path independence

I Function f (z) = u + iv is analytic in D. Two points z1 and z2 on the close curve C
H I two open paths C1 and C2 from z1 to z2
Contour integral C f (z)dz =?
Cauchys theorem on C , comprising of C 1 in the forward direction
If f 0 (z) is continuous, then by Greens theorem in the plane,
and C2 in the reverse direction:
I Z Z Z Z Z Z Z z2 Z Z
v u u v
f (z)dz = dxdy +i dxdy , f (z)dz f (z)dz = 0 f (z)dz = f (z)dz = f (z)dz
C R x y R x y
C1 C2 z1 C1 C2
where R is the region enclosed by C .
H
From C-R conditions, C f (z)dz = 0. R z function f (z) in a simply connected
For an analytic
0 (z) domain D, z12 f (z)dz is independent of the path and
Proof by Goursat: without the hypothesis of continuity of f
depends only on the end-points, as long as the path is
Cauchy-Goursat theorem completely contained in D.
HIf f (z) is analytic in a simply connected domain D, then Consequence: Definition of the function
C f (z)dz = 0 for every simple closed curve C in D.
Z z
F (z) = f ()d
Importance of Goursats contribution: z0
I continuity of f 0 (z) appears as consequence! What does the formulation suggest?

Indefinite integral
Principle of deformation of paths
Question: Is F (z) analytic? Is F 0 (z) = f (z)?
Z z+z Z z C*
F (z + z) F (z) 1 f (z) analytic everywhere other z2

f (z) = f ()d f ()d f (z) s1
z z z0 z0 than isolated points s1 , s2 , s3 D
C1
Z z+z
1 C2
= [f () f (z)]d Z Z Z
z z z1 s3
f (z)dz = f (z)dz = f (z)dz C3 s2
f is continuous , such that | z| < |f () f (z)| < C1 C2 C3
Choosing z < ,
Z z+z Not so for path C .
F (z + z) F (z) Figure: Path deformation
f (z) < d = .
z z z
If f (z) is analytic in a simply connected domain D, then The line integral remains unaltered through a continuous
there exists an analytic function F (z) in D such that deformation of the path of integration with fixed
end-points, as long as the sweep of the deformation
Z z2
includes no point where the integrand is non-analytic.
F 0 (z) = f (z) and f (z)dz = F (z2 ) F (z1 ).
z1

Cauchys Integral Formula Line Integral
Cauchys theorem in multiply connected domain f (z): analytic function in a simply connected domain D
L1
C1
C
For z0 D and simple closed curve C in D,
C2 I
L2 f (z)
dz = 2if (z0 ).
C z z0
C3
L3
Consider C as a circle with centre at z 0 and radius ,
with no loss of generality (why?).
Figure: Contour for multiply connected domain
I I I I I I I
f (z)dz f (z)dz f (z)dz f (z)dz = 0. f (z) dz f (z) f (z0 )
C C1 C2 C3
dz = f (z0 ) + dz
C z z0 C z z0 C z z0
If f (z) is analytic in a region bounded by the contour C
From continuity of f (z), such that for any ,
as the outer boundary and non-overlapping contours C 1 ,

C2 , C3 , , Cn as inner boundaries, then f (z) f (z0 )
|z z0 | < |f (z) f (z0 )| < and < ,
I n I
X z z0
f (z)dz = f (z)dz.
C Ci with < . From M-L inequality, the second integral vanishes.
i =1

Direct applications Poissons integral formula

I Evaluation of contour integral: Taking z0 = re i and z = Re i (with r < R) in Cauchys formula,
If g (z) is analytic on the contour Z 2
H and in the enclosed region, f (Re i )
I
the Cauchys theorem implies C g (z)dz = 0. 2if (re i ) = (iRe i )d.

0 Re i re i
I If the contour encloses a singularity at z0 , then Cauchys
formula supplies a non-zero contribution to the integral, if How to get rid of imaginary quantities from the expression?
2
f (z) = g (z)(z z0 ) is analytic. Develop a complement. With Rr in place of r ,
Evaluation of function at a point: If finding the integral on Z 2 Z 2
I
f (Re )i f (Re i )
the left-hand-side is relatively simple, then we use it to 0= (iRe i )d = (ire i )d.
0 i
Re e R2 i
0 re Re i
i
evaluate f (z0 ). r
Subtracting,
Significant in the solution of boundary value Z
2
problems! Re i re i
2if (re i ) = i f (Re i ) + i i
d
0 Re i
re i Re re
Example: Poissons integral formula Z 2
Z 2 (R 2 r 2 )f (Re i )
1 (R 2 r 2 )u(R, ) = i d
u(r , ) = d 0 (Re i re i )(Re i re i )
2 0 R 2 2Rr cos( ) + r 2 Z 2 2 2 i
1 (R r )f (Re )
for the Dirichlet problem over a circular disc. f (re i ) = d.
2 0 R 2 2Rr cos( ) + r 2

Cauchys integral formula evaluates contour integral of g (z),

I
if the contour encloses a point z0 where g (z) is f (z0 + z) f (z0 ) 1 1 1
= f (z) dz
non-analytic but g (z)(z z0 ) is analytic. z 2iz C z z0 z z z0
I
1 f (z)dz
If g (z)(z z0 ) is also non-analytic, but g (z)(z z 0 )2 is analytic? =
2i C (z z0 z)(z z0 )
I I
I 1 f (z)dz 1 1 1
1 f (z) = + f (z) dz
f (z0 ) = dz, 2i C (z z0 )2 2i C (z z0 z)(z z0 ) (z z0 )2
2i C z z0 I I
I 1 f (z)dz 1 f (z)dz
1 f (z) = + z
f 0 (z0 ) = dz, 2i C (z z0 )2 2i C (z z0 z)(z z0 )
2
2i C (z z0 )2
I If |f (z)| < M on C , L is path length and d 0 = min |z z0 |,
2! f (z) I
f 00 (z0 ) = dz, f (z)dz ML|z|
2i C (z z0 )3 z <
= , 2 d 2 (d |z|) 0 as z 0.
I C (z z0 z)(z z0 ) 0 0
n! f (z)
f (n) (z0 ) = dz. An analytic function possesses derivatives of all orders at
2i C (z z0 )n+1
every point in its domain.
The formal expressions can be established through differentiation
under the integral sign. Analyticity implies much more than mere differentiability!
Applied Mathematical Methods Integrals in the Complex Plane 547, Applied Mathematical Methods Singularities of Complex Functions 548,
Points to note Line Integral

Outline Series Representations of Complex Functions
Zeros and Singularities
Residues
Evaluation of Real Integrals
I Concept of line integral in complex plane

I Cauchys integral theorem Singularities of Complex Functions
I Consequences of analyticity Series Representations of Complex Functions
I Cauchys integral formula Zeros and Singularities
I Derivatives of arbitrary order for analytic functions Residues
Applied Mathematical Methods Singularities of Complex Functions 549, Applied Mathematical Methods Singularities of Complex Functions 550,
Series Representations of Complex Functions

Residues
Residues
Evaluation of Real Integrals Evaluation of Real Integrals
Taylors series of function f (z), analytic in a neighbourhood of z 0 : Laurents series: If f (z) is analytic on circles C 1 (outer) and C2

X (inner) with centre at z0 , and in the annulus in between, then
f (z) = an (zz0 )n = a0 +a1 (zz0 )+a2 (zz0 )2 +a3 (zz0 )3 + ,
n=0
X X X cm
f (z) = an (z z0 )n = bm (z z0 )m + ;
with coefficients n=
(z z0 )m
m=0 m=1
I
1 (n) 1 f (w )dw
an = f (z0 ) = , with coefficients
n! 2i C (w z0 )n+1 I
1 f (w )dw
where C is a circle with centre at z0 . an = ;
2i C (w z0 )n+1
Form of the series and coefficients: similar to real functions I I
1 f (w )dw 1
The series representation is convergent within a disc or, bm = m+1
, cm = f (w )(w z0 )m1 dw ;
|z z0 | < R, where radius of convergence R is the 2i C (w z0 ) 2i C
distance of the nearest singularity from z 0 . the contour C lying in the annulus and enclosing C 2 .
Note: No valid power series representation around z 0 , i.e. in Validity of this series representation: in annular region obtained by
powers of (z z0 ), if f(z) is not analytic at z0 growing C1 and shrinking C2 till f (z) ceases to be analytic.
Question: In that case, what about a series representation that Observation: If f (z) is analytic inside C 2 as well, then cm = 0 and
includes negative powers of (z z0 ) as well? Laurents series reduces to Taylors series.

Residues
Residues
Proof of Laurents series Proof of Laurents series (contd)
Cauchys integral formula for any point z in the annulus, Using q = wzz
z0 ,
0
I I
1 f (w )dw 1 f (w )dw 1 1 z z0 (z z0 )n1 z z0 n 1
f (z) = . = + + + +
2i C1 w z 2i C2 w z w z w z0 (w z0 )2 (w z0 )n w z0 w z
I
1 f (w )dw n1
Organization of the series: = a0 + a1 (z z0 ) + + an1 (z z0 ) + Tn ,
z 2i C1 w z
w
1 1 z0
with coefficients as required and
= I
w z (w z0 )[1 (z z0 )/(w z0 )] C2 C1
1 z z0 n f (w )
Tn = dw .
1 1 2i C1 w z0 w z
=
w z (z z0 )[1 (w z0 )/(z z0 )] z0
Similarly, with q = wzz ,
Figure: The annulus 0
I
1 f (w )dw
Using the expression for the sum of a geometric series, = a1 (z z0 )1 + + an (z z0 )n + Tn ,
2i C2 w z
1 qn 1 qn with appropriate coefficients and the remainder term
1+q+q 2 + +q n1 = = 1+q+q 2 + +q n1 + .
1q 1q 1q I
1 w z0 n f (w )
We use q = zz0
for integral over C1 and q = w z0
over C2 . Tn = dw .
w z0 zz0 2i C2 z z0 z w

Residues
Zeros and Singularities Series Representations of Complex Functions
Residues
Convergence of Laurents series Zeros of an analytic function: points where the function vanishes
n1
If, at a point z0 ,
X
f (z) = ak (z z0 )k + Tn + Tn , a function f (z) vanishes along with first m 1 of its
k=n derivatives, but f (m) (z0 ) =
6 0;
I n
1 z z0 f (w ) then z0 is a zero of f (z) of order m, giving the Taylors series as
where Tn = dw
2i C1 w z0 w z
I n f (z) = (z z0 )m g (z).
1 w z0 f (w )
and Tn = dw .
2i C2 z z0 z w An isolated zero has a neighbourhood containing no other zero.
I
f (w ) is
bounded For an analytic function, not identically zero, every point
zz0 z0
I w z0 < 1 over C1 and wzz 0
< 1 over C2 has a neighbourhood free of zeros of the function, except
Use M-L inequality to show that possibly for that point itself. In particular, zeros of such
remainder terms Tn and Tn approach zero as n . an analytic function are always isolated.
Remark: For actually developing Taylors or Laurents series of a Implication: If f (z) has a zero in every neighbourhood around
function, algebraic manipulation of known facts are employed quite z0 then it cannot be analytic at z0 , unless it is the zero function
often, rather than evaluating so many contour integrals! [i.e. f (z) = 0 everywhere].

Residues
Residues
Entire function: A function which is analytic everywhere Zeros and poles: complementary to each other
Examples: z n (for positive integer n), e z , sin z etc. I Poles are necessarily isolated singularities.
The Taylors series of an entire function has an infinite 1
I A zero of f (z) of order m is a pole of f (z) of the same order
radius of convergence. and vice versa.
I If f (z) has a zero of order m at z0 where g (z) has a pole of
Singularities: points where a function ceases to be analytic
the same order, then f (z)g (z) is either analytic at z 0 or has a
Removable singularity: If f (z) is not defined at z 0 , but has a limit. removable singularity there.
z
Example: f (z) = e z1 at z = 0. I Argument theorem:
Pole: If f (z) has a Laurents series around z 0 , with a finite If f (z) is analytic inside and on a simple closed
number of terms with negative powers. If a n = 0 for curve C except for a finite number of poles inside
n < m, but am 6= 0, then z0 is a pole of order m, and f (z) 6= 0 on C , then
limzz0 (z z0 )m f (z) being a non-zero finite number.
I 0
A simple pole: a pole of order one. 1 f (z)
dz = N P,
Essential singularity: A singularity which is neither a removable 2i C f (z)
singularity nor a pole. If the function has a Laurents
where N and P are total numbers of zeros and poles
series, then it has infinite terms with negative
inside C respectively, counting multiplicities (orders).
powers. Example: f (z) = e 1/z at z = 0.
Residues Series Representations of Complex Functions

Zeros and Singularities Evaluation of Real Integrals Series Representations of Complex Functions
H
Residues Residues
Term by term integration of Laurents
H series: C f (z)dz = 2ia1
General strategy
1
Residue: Res z0 f (z) = a1 = 2i C f (z)dz I Identify the required integral as a contour integral of a
If f (z) has a pole (of order m) at z0 , then complex function, or a part thereof.

X I If the domain of integration is infinite, then extend the
(z z0 )m f (z) = an (z z0 )m+n contour infinitely, without enclosing new singularities.
n=m
is analytic at z0 , and Example: Z 2

X I = (cos , sin )d
d m1 (m + n)!
m1
[(z z0 )m f (z)] = an (z z0 )n+1 0
dz (n + 1)!
n=1 With z = ei
and dz = izd,
I I
1 d m1 1 1 1 1 dz
Resz0 f (z) = a1 = (m 1)! zz lim m1
[(z z0 )m f (z)]. I = z+ , z = f (z)dz,
0 dz C 2 z 2i z iz C
Residue theorem: If f (z) is analytic inside and on simple closed where C is the unit circle centred at the origin.
curve C , with singularities at z1 , z2 , z3 , , zk inside C ; then Denoting poles falling inside the unit circle C as p j ,
I Xk X
f (z)dz = 2i Resf (z). I = 2i Resf (z).
pj
C
z i j
i =1
Evaluation of Real Integrals Series Representations of Complex Functions

Residues
Evaluation of Real Integrals Series Representations of Complex Functions
Residues
Example: For real rational function f (x), Example: Fourier integral coefficients
Z Z Z
I = f (x)dx, A(s) = f (x) cos sx dx and B(s) = f (x) sin sx dx

denominator of f (x) being of degree two higher than numerator. Consider Z
Consider contour C enclosing semi-circular region |z| R, y 0, I = A(s) + iB(s) = f (x)e isx dx.

large enough to enclose all singularities above the x-axis. Similar to the previous case,
I Z Z I Z R Z
R y
f (z)dz = f (x)dx + f (z)dz iR

f (z)e isz dz = f (x)e isx dx + f (z)e isz dz.
C R S C R S
M
p R S
As |e isz | = |e isx | |e sy | = |e sy | 1 for y 0, we have
For finite M, |f (z)| < on C R2
p
Z
p

Z
R O R x
f (z)e isz dz < M R = M ,
R2 R
f (z)dz < M R = M . S
R2 R Figure: The contour which yields, as R ,
S
X
Z X I = 2i Res[f (z)e isz ].
pj
I = f (x)dx = 2i Resf (z)
pj as R . j
j
Applied Mathematical Methods Singularities of Complex Functions 561, Applied Mathematical Methods Variational Calculus* 562,
Points to note Series Representations of Complex Functions

Residues
Eulers Equation
Direct Methods
I Taylors series and Laurents series

I Zeros and poles of analytic functions
Variational Calculus*
I Residue theorem
Introduction
I Evaluation of real integrals through contour integration of Eulers Equation
suitable complex functions Direct Methods
Necessary Exercises: 1,2,3,5,8,9,10
Applied Mathematical Methods Variational Calculus* 563, Applied Mathematical Methods Variational Calculus* 564,
Eulers Equation
Direct Methods
Eulers Equation
Direct Methods
Functionals and their extremization

Consider a particle moving on a smooth surface z = (q 1 , q2 ).
Suppose that a candidate curve is represented as a sequence of
With position r = [q1 (t) q2 (t) (q1 (t), q2 (t))]T on the surface points qj = q(tj ) at time instants
and r = [q1 q2 ()T q]T in the tangent plane, length of the
path from qi = q(ti ) to qf = q(tf ) is ti = t0 < t1 < t2 < t3 < < tN1 < tN = tf .
Z Z tf Z tf h i1/2
l = krk = krkdt = q12 + q22 + ( T q)2 dt. Geodesic problem: a multivariate optimization problem with the
ti ti 2(N 1) variables in {qj , 1 j N 1}.
For shortest path or geodesic, minimize the path length l. With N , we obtain the actual function.
Question: What are the variables of the problem? First order necessary condition: Functional is stationary with
respect to arbitrary small variations in {q j }.
Answer: The entire curve or function q(t). [Equivalent to vanishing of the gradient]
Variational problem:
This gives equations for the stationary points.
Optimization of a function of functions, i.e. a functional.
Here, these equations are differential equations!
Eulers Equation
Direct Methods
Eulers Equation Introduction
Eulers Equation
Direct Methods
Examples of variational problems Find out a function y (x), that will make the functional
Rb
Geodesic path: Minimize l = a kr0 (t)kdt Z x2
Minimal surface ofRrevolution: Minimize I [y (x)] = f [x, y (x), y 0 (x)]dx
Rb p x1
S = 2yds = 2 a y 1 + y 02 dx
The brachistochrone problem: To find the curve along which the stationary, with boundary conditions y (x 1 ) = y1 and y (x2 ) = y2 .
descent is fastest. Consider variation y (x) with y (x 1 ) = y (x2 ) = 0 and consistent
R R b q 1+y 02
Minimize T = ds variation y 0 (x).
v = a 2gy dx
Fermats principle: Light takes the fastest path. Z x2

R u x 02 +y 02 +z 02 f f
I = y + 0 y 0 dx
Minimize T = u12 c(x,y ,z) du x1 y y
Isoperimetric problem: Largest area in the plane enclosed by a Integration of the second term by parts:
closed curve of given perimeter. By extension, Z x2 Z x2 x2 Z x2
extremize a functional under one or more equality f 0 f d f d f
0
y dx = 0
(y )dx = y y dx
constraints. x1 y x1 y dx y 0 x1 x1 dx y
0
Hamiltons principle of least action: Evolution of a dynamic With y (x1 ) = y (x2 ) = 0, the first term vanishes identically, and
system through the minimization of the action Z x2
Z t2 Z t2 f d f
I = 0
y dx.
s= Ldt = (K P)dt x1 y dx y
t1 t1

Eulers Equation
Direct Methods
Eulers Equation
Direct Methods
For I to vanish for arbitrary y (x), Functionals of a vector function

d f f Z t2
dx y 0 y = 0.
I [r(t)] = f (t, r, r)dt
Functions involving higher order derivatives t1
Z x2 In terms of partial gradients f f
r and r ,
I [y (x)] = f x, y , y 0 , y 00 , , y (n) dx Z t2 " T T #
x1 f f
I = r + r dt
with prescribed boundary values for y , y 0 , y 00 , , y (n1) t1 r r
Z t2 T " T # t2 Z t2
Z x2 f f d f T
f f f f = rdt + r rdt
I = y + 0 y 0 + 00 y 00 + + (n) y (n) dx t1 r r t1 dt r
x1 y y y y t1
Z t2
Working rule: Starting from the last term, integrate one term at f d f T
= rdt.
a time by parts, using consistency of variations and BCs. t1 r dt r
Eulers equation:
Eulers equation: a system of second order ODEs
f d f d 2 f d n f
+ + (1)n n (n) = 0, d f f d f f
y dx y 0 dx 2 y 00 dx y =0 or = 0 for each i.
dt r r dt ri ri
an ODE of order 2n, in general.

Eulers Equation
Direct Methods
Direct Methods Introduction
Eulers Equation
Direct Methods
Functionals of functions of several variables Finite difference method

With given boundary values y (a) and y (b),
Z Z
I [u(x, y )] = f (x, y , u, ux , uy )dx dy Z b
D I [y (x)] = f [x, y (x), y 0 (x)]dx
a
f f f
Eulers equation: x ux + y uy u =0
I Represent y (x) by its values over xi = a + ih with
Moving boundaries
i = 0, 1, 2, , N, where b a = Nh.
Revision of the basic case: allowing non-zero y (x 1 ), y (x2 )
f I Approximate the functional by
At an end-point, y 0 y has to vanish for arbitrary y (x).
N
X
f
y 0 vanishes at the boundary. I [y (x)] (y1 , y2 , y3 , , yN1 ) = f (xi , yi , yi0 )h,
i =1
Euler boundary condition or natural boundary condition
x +x y +y y y
where xi = i 2 i 1 , yi = i 2 i 1 and yi0 = i h i 1 .
Equality constraints and isoperimetric problems
Rx Rx I Minimize (y1 , y2 , y3 , , yN1 ) with respect to yi ;
Minimize I = x12 f (x, y , y 0 )dx subject to J = x12 g (x, y , y 0 )dx = J0 .
for example, by solving y = 0 for all i.
In another level of generalization, constraint (x, y , y 0 ) = 0. i

Operate with f (x, y , y 0 , ) = f (x, y , y 0 ) + (x)g (x, y , y 0 ). Exercise: Show that yi = 0 is equivalent to Eulers equation.

Eulers Equation
Direct Methods
Eulers Equation
Direct Methods
The inverse problem: From
Rayleigh-Ritz method Z b !
N
X N
X
In terms of a set of basis functions, express the solution as
I [y (x)] () = f x, i wi (x), i wi0 (x) dx,
a i =1 i =1
N
X
y (x) = i wi (x).
2 0 1 0 1 3
N N N N
f f
Z b
0 0 0
X X X X
= 4 @x , i wi , i wi A wi (x ) + @x , i w i , i wi A wi (x )5 dx .
i =1 i a y i=1 i=1
y 0 i=1 i=1
Integrating the second term by parts and using w i (a) = wi (b) = 0,

Represent functional I [y (x)] as a multivariate function ().
Z b "X N
#

Optimize () to determine i s. = R i wi wi (x)dx,
i a i =1
Note: As N , the numerical solution approaches exactitude. f d f
For a particular tolerance, one can truncate appropriately. where R[y ] y dx y 0 = 0 is the Eulers equation of the
variational problem.
Observation: With these direct methods, no need to reduce the Def.: R[z(x)]: residual of the differential equation R[y ] = 0
variational (optimization) problem to Eulers equation! operated over the function z(x)
Question: Is it possible to reformulate a BVP as a variational Residual of the Eulers equation of a variational problem
problem and then use a direct method? operated upon the solution obtained by Rayleigh-Ritz
method is orthogonal to basis functions w i (x).

Eulers Equation
Direct Methods
Eulers Equation
Direct Methods
Galerkin method
Question: What if we cannot find a corresponding variational Finite element methods
problem for the differential equation? I discretization of the domain into elements of simple geometry
Answer: Work with the residual directly and demand I basis functions of low order polynomials with local scope
Z b
R[z(x)]wi (x)dx = 0.
I design of basis functions so as to achieve enough order of
a continuity or smoothness across element boundaries
Freedom to choose two different families of functions as basis I piecewise continuous/smooth basis functions for entire
functions j (x) and trial functions wi (x): domain, with a built-in sparse structure

Z b X I some weighted residual method to frame the algebraic
R j j (x) wi (x)dx = 0 equations
a j I solution gives coefficients which are actually the nodal values
A singular case of the Galerkin method:
delta functions, at discrete points, as trial functions Suitability of finite element analysis in software environments
I effectiveness and efficiency
Satisfaction of the differential equation exactly at the chosen
points, known as collocation points: I neatness and modularity
Collocation method
Applied Mathematical Methods Variational Calculus* 575, Applied Mathematical Methods Epilogue 576,

Eulers Equation
Direct Methods
Outline
I Optimization with respect to a function

I Concept of a functional
I Eulers equation
Epilogue
I Rayleigh-Ritz and Galerkin methods
I Optimization and equation-solving in the infinite-dimensional
function space: practical methods and connections

Applied Mathematical Methods Epilogue 577, Applied Mathematical Methods Epilogue 578,
Epilogue Epilogue
Source for further information:
http://home.iitk.ac.in/ dasgupta/MathBook
Some specialized courses in immediate continuation
Destination for feedback: I Linear Algebra and Matrix Theory
dasgupta@iitk.ac.in I Approximation Theory
I Variational Calculus and Optimal Control
Some general courses in immediate continuation I Advanced Mathematical Physics
I Advanced Mathematical Methods I Geometric Modelling
I Scientific Computing I Computational Geometry
I Advanced Numerical Analysis I Computer Graphics
I Optimization I Signal Processing
I Advanced Differential Equations I Image Processing
I Partial Differential Equations
I Finite Element Methods
Applied Mathematical Methods Selected References 579, Applied Mathematical Methods Selected References 580,
Outline Selected References I
F. S. Acton.
Numerical Methods that usually Work.
The Mathematical Association of America (1990).
C. M. Bender and S. A. Orszag.
Selected References
Advanced Mathematical Methods for Scientists and Engineers.
Springer-Verlag (1999).
G. Birkhoff and G.-C. Rota.
Ordinary Differential Equations.
John Wiley and Sons (1989).
G. H. Golub and C. F. Van Loan.
Matrix Computations.
The John Hopkins University Press (1983).
Applied Mathematical Methods Selected References 581, Applied Mathematical Methods Selected References 582,
Selected References II Selected References III

M. T. Heath.
Scientific Computing . W. H. Press, S. A. Teukolsky, W. T. Vellerling and B. P.
Tata McGraw-Hill Co. Ltd (2000). Flannery.
Numerical Recipes.
E. Kreyszig.
Cambridge University Press (1998).
Advanced Engineering Mathematics.
John Wiley and Sons (2002). G. F. Simmons.
Differential Equations with Applications and Historical Notes.
E. V. Krishnamurthy and S. K. Sen.
Tata McGraw-Hill Co. Ltd (1991).
Numerical Algorithms.
Affiliated East-West Press Pvt Ltd (1986). J. Stoer and R. Bulirsch.
Introduction to Numerical Analysis.
D. G. Luenberger.
Springer-Verlag (1993).
Linear and Nonlinear Programming .
Addison-Wesley (1984). C. R. Wylie and L. C. Barrett.
P. V. ONeil.
Tata McGraw-Hill Co. Ltd (2003).
Thomson Books (2004).

Lmaster 6

Загружено:

Сведения о документе

Оригинальное название

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

Lmaster 6

Загружено:

Авторское право:

Доступные форматы

Applied Mathematical Methods 1, Applied Mathematical Methods 2,

Matrices and Linear Transformations

Special Systems and Special Methods

Numerical Aspects in Linear Systems

Applied Mathematical Methods 3, Applied Mathematical Methods 4,

Contents II Contents III

Diagonalization and Similarity Transformations Vector Analysis: Curves and Surfaces

Jacobi and Givens Rotation Methods Scalar and Vector Fields

Householder Transformation and Tridiagonal Matrices Polynomial Equations

QR Decomposition Method Solution of Nonlinear Equations and Systems

Eigenvalue Problem of General Matrices Optimization: Introduction

Singular Value Decomposition Multivariate Optimization

Vector Spaces: Fundamental Concepts* Methods of Nonlinear Optimization*

Applied Mathematical Methods 5, Applied Mathematical Methods 6,

Interpolation and Approximation Second Order Linear Non-Homogeneous ODEs

Basic Methods of Numerical Integration Higher Order Linear ODEs

Advanced Topics in Numerical Integration* Laplace Transforms

Numerical Solution of Ordinary Differential Equations ODE Systems

ODE Solutions: Advanced Issues Stability of Dynamic Systems

Existence and Uniqueness Theory Series Solutions and Special Functions

Contents VI Contents VII

Partial Differential Equations

Integrals in the Complex Plane

Singularities of Complex Functions

Outline Theme of the Course

Notable beneficiaries (at two ends)

Course Contents Theme of the Course

Logistic Strategy Theme of the Course

Expected Background Theme of the Course

I moderate background of undergraduate mathematics I Put in effort, keep pace.

Necessary Exercises: Prerequisite problem sets ??

Prerequisite Problem Sets*

Question: What is a matrix?

Geometry and Algebra Matrices

Operate A on a large number of points x i R 3 .

The linear transformation represented by A implies the totality of

Matrix Terminology Matrices

I I A matrix defines a linear transformation from one vector space

Outline Range and Null Space: Rank and Nullity

Operational Fundamentals of Linear Algebra

2. Image of x R n in R m is unique, but pre-image of y R m

Range and Null Space: Rank and Nullity

Basis Range and Null Space: Rank and Nullity

Natural basis: With C = [c1 c2 cn ],

Change of Basis Range and Null Space: Rank and Nullity

Question: And, how does basis change affect the representation of

Change the basis of the domain through P R nn and that of the

Elementary Transformations Range and Null Space: Rank and Nullity

Similarly, an elementary column transformation is equivalent to

Outline Nature of Solutions

Coefficient matrix: A, augmented matrix: [A | b].

Rank(A) = Rank([A | b]) = n

Infinite solutions: For Rank(A) = Rank([A|b]) = k < n, solution

x = x + xN , with Ax = b and xN Null(A)

Basic Idea of Solution Methodology Nature of Solutions

For inverting matrix A, treat AA1 = In similarly. unk

Pivoting Nature of Solutions

Points to note Nature of Solutions

I Solution(s) of Ax = b may be non-existent, unique or

Necessary Exercises: 1,2,4,5,7

Gauss-Jordan Elimination Gauss-Jordan Elimination

Task: Solve Ax = b1 , Ax = b2 and Ax = b3 ; find A1 and

Gaussian Elimination with Back-Substitution