CLASSICAL OPTIMIZATION

TECHNIQUES

Single Variable Optimization

A function of one variable f(x) is said to have a
relative or local minimum at x=x* if f(x*) f(x*+h)
for all sufficiently small positive and negative
values of h. Similarly, a point x* is called a
relative or local maximum if x* if f(x*) f(x*+h) for
all values of h sufficiently close to zero.
A function f(x) is said to have a global or
absolute minimum at x* if f(x*) f(x) for all values
of x and not just close to x* ,in the domain over
which f(x) is defined.

Single Variable Optimization

Relative and Global Minima
f(x)

f(x)

A2

A3

Relative Minimum
Is Global Minimum

A1
B2
B1
o

a
A1,A2, A3 are relative maxima
A2 is global maximum

Single Variable Optimization

Similarly a point x* will be a global maximum of f(x) if
f(x*)f(x) for all x in the domain.
A single variable optimization problem is one in which the
value of x=x* to be found out in the interval [a,b] such that
x* minimizes f(x). Following two theorem provide
necessary and sufficient condition for the relative
minimum of function of single variable.
Theorem I : Necessary Condition
If a function f(x) is defined in the interval a x b and has
a relative minimum at x= x* , where a<x*< b , and if the
derivative df ( x )
dx

f ( x )

exists as finite number at x=x* , then f(x*) =0

Single Variable Optimization

Proof: It is given that
f ( x * h) f ( x * )
f ( x ) h 0
h
exists as a definite number, which we want
lim

to prove to be zero. Since x * is a relative

minimum, we have f(x * ) f ( x * h)
for the value sufficient ly close to zero.
f ( x * h) f ( x * )
0 if h 0
h
f(x * h) f ( x * )
0 if h 0
h

Single Variable Optimization

If h 0 through positive values of h
f ( x * ) 0 ( a )
while it gives the limit h 0 through
negative values
f ( x * ) 0 ( b )
The only way to satisfy (a) and (b) is
to have f ( x * ) 0,
this proves the theorem .
Notes :
1.This theorem can be proved even if
x * is a relative maximum.

Single Variable Optimization

Notes (continued ) :
2. The theorem does not say what happens if a
minimum or maximum occurs at a point x *
where derivative fails to exist for example
for the function shown in the next slide.
*
*
f
(
x

h
)

f
(
x
)
h lim 0
m or m
h
depending on whether h approaches zero
through positive or negative values respectively.

Unless the number m and m are equal, the

derivative f ( x * ) does not exist, the theorem
is not applicable .

f(x)
Negative slope
Positive
slope

f(x*)
X

Single Variable Optimization

3. The theorem does not say what happens if a minimum or
maximum occurs at an end point of the interval of
definition of the function. In this case
h

f ( x * h) f ( x * )
0
h

lim

Exists for all positive values of h only or all negative values

of h only, and hence the derivative is not defined at the
end point.
4.The theorem does not say that the function necessarily will
be maximum or minimum at every point where the
derivative is zero. For example the derivative f(x)=0 at
x=0 for the function shown in the next slide. However this

Single Variable Optimization

f(x)

Stationary (inflection)
point f(x)=0
O

FIG. STATIONARY (INFLECTION) POINT

Single Variable Optimization

If the function f(x) posses continuous
derivatives of every order that come in
question, in the neighborhood of x=x1* the
following theorem provides the sufficient
condition for the minimum or maximum
value of the function.

Theorem 2 : Sufficient condition let

f ( x * ) f ( x * ) f

( n 1 )

( x * ) 0,

but f n ( x * ) 0. Then f ( x * ) is (a) a minimum

value of f(x) if f n ( x * ) 0 and n is even
(b) a maximum value of f(x) if f n ( x * ) 0 and n is
even (c) neither a maximum nor minimum if n is odd.
Applying Taylor' s theorem with remainder after
n terms, we have
2
h
f ( x * h ) f ( x * ) h f ( x * )
f ( x * )
2!
h n 1
hn n *
( n 1 )
*
f
(x )
f ( x h)
( n 1)!
n!
for 0 1

Since f ( x * ) f ( x * ) f

n 1

( x* ) 0

hn n *
f ( x h) f ( x )
f ( x h)
n!
As f n ( x * ) 0, there exists an interval arround x *
*

for every point x of which the n th derivative f n ( x * )

has the same sign, normally that of f n ( x * ).
Thus for every point ( x * h) of this interval
f n ( x * h) has the sign of f n ( x * ).
Thus x* will be relative minimum if f n ( x * ) is
positive and a relative maximum if if f n ( x * ) is negative,
hn
when n is odd,
changes sign with the chane in sign
n!
of h and hence point x * is neither a maximum nor
minimum. In this case point x * is called a point of inflection .

Example
Determine the maximum and minimum values
of the function
f ( x ) 12 x 5 45 x 4 40 x 3 5
Solution :
f ( x ) 60 x 4 180 x 3 120 x 2

60 x 2 x 2 3 x 2

60 x 2 x 2 x 1
f ( x ) 0 at x 0, x 1 and x 2
The second derivative
f ( x ) 240 x 3 540 x 2 240 x

60 4 x 3 9 x 2 4 x
At x 1 f ( x ) 60 and hence x 1 is
a relative maximum.
Therefore f max f ( x 1) 12.

Example (continued)
At x 2, f ( x ) 240 and hence x 2 is a relative minimum.
Therefore, f min f ( x 2) 11
At x 0, f ( x ) 0 and hence we must investigat e the next
derivative .
f ( x ) 60(12 x 2 18 x 4) 240 at x 0, f ( x ) 0 and
hence x 0 is neither a maximum nor a minimum and
it is an inflection point.

Example
In a two stage compressor, the working gas leaving the first stage
of compression is cooled ( by passing it through an intercooler)
before it enters the second stage of compression to increase the
efficiency. The total work input to a compressor (W) for an ideal gas
for isentropic compression is given by

p2

W C p .Ti
p1

p3

p2

where Cp. Is the specific heat of the gas at constant pressure, is

the ratio of specific heat at constant pressure to that of at constant
volume of the gas, and Ti is the temperature at which the gas enters
the compressor. Find the pressure p2 at which inter cooling should be
done to minimize work input to the compressor. Also determine the

Example - Solution

The necessary condition for minimizing

the work done

on the compressor is
dW
d 2W
0 and sufficient condition is
0 and has to be
dp2
dp 2
a positive number.

1
1
2 1
1
1
1

. p2
.
. p2 p3 .

p1

1
1
1
2 1
1

C p .Ti .
. p1 . p2 p3 . p2 0

dW
C p .Ti .

dp2

p1
p2

( 1 )

2( 1)

p2

. p2

p3

p1 . p3
p1 . p3

. p2

2 1

Differenti ating further

1
1
1
2 1
d 2W
1
1
2 1

C p .Ti .
. p1
. . p2
p3 .

2
. p2

dp2

1 1
1
1 2 1
3 1
1

Cp .Ti .
. p1 . . p2 p3 .
. p2

substituti ng p2

p1 . p3

1
1
1 2 1
3 1
d 2W
1 1

C p .Ti .
. . p1 . p1 . p3 p3 .
. p1 p3 2
2

dp2

on simplifica tion it comes to

3 1
1
d 2W
1

2.C p .Ti .
. p1 2 . p3 2
2

dp2

which is always greater than zero as 1.

Minimum work done can be determined as under :

substituti ng the value of p 2 in the equation for work done

Wmin

p2

C p .Ti .
p1

p3

Cp .Ti .
p1

1
2

p3

p2
p3

p1

1
2

p3

2Cp .Ti .
1
p1

This is the minimum work done.

1
2

Optimality Criteria
Suppose at any point x* first derivative
is zero and first non-zero higher order
derivative is denoted by n, then
If n is odd x* is an inflection point
If n is even x* is a local optimum
If fn(x*) is positive x* is a local minimum
If fn(x*) is negative x* is a local maximum

Example

f ( x) x 2 x 2 x 1
x4 x3 x2

f ( x ) 4 x 3 3 x 2 2 x
f ( x ) 12 x 2 6 x 2
at x 0; f ( 0) 0 and f (0) 2
it is giving minimum

value of the fonction.

It satisfies the optimality criteria i.e.

necessary condition f (0) 0 and sufficient
condition f (0) a finite value

f ( x) x 3 x 2 x 1

Example

x5 x4 x3
f ( x ) 5 x 4 4 x 3 3 x 2 and f ( 0) 0
f ( x ) 20 x 3 12 x 2 3 x here also f ( 0) 0
Therefore x 0 is neither a maximum nor minimum
f ( x ) 60 x 2 24 x 6
f (0) 6 0
As per our optimality criteria n 3 i.e. odd
and hence x 0 is an infection point.

Example
f ( x ) 60 x 6 48 x 5 15 x 4 200 x 3 120 x 2 480 x 100
f ( x ) 60 x 5 240 x 4 60 x 3 600 x 2 240 x 480

f ( x ) 60 5 x 4 16 x 3 3 x 2 20 x 4
At x -1 and x 2 ; f ( x ) 0

It satisfies the necessary condition ,

At x -1 , f ( x ) 0 and
at x 2, f ( x ) 0
Taking higher order derivative s

f ( x ) 120 10 x 3 24 x 2 3 x 10
at x -1 , f ( x ) 0;

This does not satisfy the sufficient condition

and therefore x -1 is inflection point.

At x 2 f ( x ) 0
This satisfy the necessary condition.

f ( x ) 120 30 x 2 48 x 3
at x 2, f ( x ) 3240
This satisfy the sufficient condition.
x 2 is a minima.
f ( x )min 396

Bibliography
1. Engineering Optimization Theory And
Problems
S.S. Rao
New Age International (P) Limited, Publishers
New Delhi
2. Introduction To Optimum Design
J.S. Arora
Academic Press
An Imprint of Elsevier

THE END