Optimization Techniques

Problem Formulation
Dr. Nasir M Mirza
Email: nasirmm@yahoo.com
1

Optimal Problem Formulation

it is almost impossible to apply a single formulation
procedure for all engineering design problems.

Since the objective in a design problem vary from

product to product, different techniques need to be
used in different problems.

The purpose of the formulation procedure is to create a

mathematical model of the optimal design problem,
which then can be solved.

Since an optimization algorithm accepts an

optimization problem in a particular format, every
optimal design problem must be formulated in that
format.
nasir m mirza

Hierarchy of Optimal Design

Process
the designer needs to

choose the important

design variables
associated with the
design problem.
The formulation of
optimal design problems
involves other
considerations, such as
constraints, objective
function, and variable
bounds.
As shown in the figure,
there is usually a
hierarchy in the optimal
design process;
We discuss all these
aspects in the following
subsections.

Need for Optimization

Choose design variables

Formulate constraints

Formulate objective function

Setup variable bounds
Choose an optimization algo
Obtain solutions

nasir m mirza

Design variables
The formulation of an optimization problem begins with

suggestion of design variables, which are varied during the

optimization process.
A design problem -usually involves many design parameters,
of which some are highly sensitive to the proper working of
the design. These parameters are called design variables in
the parlance of optimization procedures.
Other (not so important) design parameters usually remain
fixed or vary in relation to the design variables.
There is no rigid guideline to choose the parameters which
may be important in a problem, because one parameter may
be more important with respect to minimizing the overall
cost of the design, while it may be insignificant with respect
to maximizing the life of the product.
Thus, the choice of the important parameters largely
depends on the user.
However, the efficiency and speed of optimization algorithms
depend on the number of chosen design variables.

nasir m mirza

Constraints
Having chosen the design variables, the next task is to identify

the constraints associated with the optimization problem.

The constraints represent some functional relationships among
the design variables and other design parameters satisfying
certain physical phenomenon and certain resource limitations.
Some of these considerations require that the design remain in
static or dynamic equilibrium. In many mechanical and civil
engineering problems, the constraints are formulated to satisfy
stress and deflection limitations.
Often, a component needs to be designed in such a way that it
can be placed inside a fixed housing, thereby restricting the
size of the component.
There is, however, no unique way to formulate a constraint in
all problems.
The nature and number of constraints to be included in the
formulation depend on the user.

nasir m mirza

Constraints
For example, a mechanical engineering component design problem may
involve a constraint to restrain the maximum stress developed anywhere in
the component to the strength of the material.

In an irregular-shaped component, there may not exist an exact

mathematical expression for the maximum stress developed in the
component.

A finite element simulation software may be necessary

There are two types of constraints: an inequality type or of an equality type.
Inequality constraints state that the functional relationships among design
variables are either greater than, smaller than, or equal to, a resource value.

For example, the stress (a( x)) developed anywhere in a component must be
smaller than or equal to the allowable strength (Sallowable) of the material.
Mathematically,
a(x) <= Sallowable

nasir m mirza

Constraints
Equality constraints state that the functional relationships
should exactly match a resource value.

For example, a constraint may require that the deflection

((x)) of a point in the component must be exactly equal to
5 mm, or mathematically, (x) = 5.

Equality constraints are usually more difficult to handle

and, therefore, need to be avoided wherever possible.

If the functional relationships of equality constraints are

simpler, it may be possible to reduce the number of
design variables by using the equality constraints.

In such a case, the equality constraints reduce the

complexity of the problem, thereby making it easier for
the optimization algorithms to solve the problem.
nasir m mirza

Constraints
Fortunately, in many engineering design optimization
problems, it may be possible to relax an equality
constraint by including two inequality constraints.
The (x) = 5 deflection equality constraint can be
replaced by two constraints:
(x) > 4,
(x) < 6,

Although this formulation is inexact as far as the

original requirement is concerned, this flexibility

allows a smoother operation of the optimization
algorithms.
Thus, the second thumb rule in the formulation of
optimization problems is that the number of complex
equality constraints should be kept as low as possible.
nasir m mirza

Objective function
The third task in the formulation procedure is to find the

objective function in terms of the design variables and other

problem parameters.
The common engineering objectives involve minimization of
overall cost of manufacturing, or minimization of overall
weight of a component;
Most of the above objectives can be quantified (expressed in
a mathematical form).
There are some objectives that may not be quantified easily.
For example, the aesthetic aspect of a design, ride
characteristics of a car suspension design, and reliability of a
design are important objectives that one may be interested
in maximizing in a design,
But the exact mathematical formulation may not be possible.
In such a case, usually an approximating mathematical
expression is used.

nasir m mirza

Objective Function
The objective function can be of two types.
Either the objective function is to be maximized or it has to be

minimized.
Unfortunately, the optimization algorithms are usually written
either for minimization problems or for maximization
problems and not for both.
Although in some algorithms, some minor structural changes
would enable to perform either minimization or maximization,
this requires extensive knowledge of the algorithm.
Moreover, if an optimization software is used for the
simulation, the modified software needs to be compiled before
it can be used for the simulation.
Fortunately, the duality principle helps by allowing the same
algorithm to be used for minimization or maximization with a
minor change in the objective function instead of a change in
the entire algorithm.

nasir m mirza

10

Objective Function
If the algorithm is

developed for solving

a minimization
problem, it can also
be used to solve a
maximization
problem by simply
multiplying the
objective function by
-1 and vice versa.
For example, consider
the maximization of
the single-variable
function f( x) = x2 (1 x) shown by a solid
line in Figure 1.2.

nasir m mirza

11

Objective Function
For example, consider the
maximization of the singlevariable function f( x) = x2 (1 x) shown by a solid line in
Figure 1.2.

The maximum point happens

to be at x* = 0.667. The
duality principle suggests that
the above problem is
equivalent to minimizing the
function f(x) = -x2 (1- x),
which is shown by a dashed
line in Figure 1.2.
nasir m mirza

12

Objective Function
The figure shows that the
minimum point of the function
f(x) is also at x* = 0.667. Thus,
the optimum solution remains
the same.

But once we obtain the

optimum solution by
minimizing the function f(x),
we need to calculate the
optimal function value of the
original function f( x) by
multiplying f(x) by -1.
nasir m mirza

13

Objective Function
After the above four tasks are completed, the optimization problem
can be mathematically written in a special format, known as
nonlinear programming (NLP) format. Denoting the design variables
as a column vector: x = (x1, x2, ... , xN )T, the objective function as
a scalar quantity f( x), K inequality constraints as gj(x) > 0, and K
equality constraints as hk(x) = 0, we write the NLP problem:

nasir m mirza

14

Developing Optimization Problem

Formulation of an optimization problem involves
taking statements, defining general goals and
requirements of a given activity, and transcribing
them into a series of well-defined mathematical
statements.

More precisely, the formulation of an optimization

problem involves:
1. Selecting one or more optimization variables,
2. Choosing an objective function, and
3. Identifying a set of constraints.

The objective function and the constraints must all

be functions of one or more optimization variables.

The following examples illustrate the process.

nasir m mirza

15

Example: Building Design

To save energy costs for heating and cooling, an architect is

considering designing a partially buried rectangular building.

The total floor space needed is 20,000 m2.

Plot size limits the building plan dimension to 50 m.
It has already been decided that the ratio between the plan
dimensions must be equal to the golden ratio (1.618) and that
each story must be 3.5 m high.

The heating and cooling costs are estimated at $100 per m2 of

the exposed surface area of the building.

The owner has specified that the annual energy costs should not
exceed $225,000.

Formulate the problem of determining building dimensions to

minimize cost of excavation.
nasir m mirza

16

Example: Building Design

Optimization variables

From the given data and Figure 1.1, it is

easy to identify the following variables
associated with the problem:
n = Number of stories;
d = Depth of building below ground;
h = Height of building above ground;
l = Length of building in plan;
w = Width of building in plan.

nasir m mirza

17

Example: Building Design

Objective Function
The stated design objective is to
minimize excavation cost.
Assuming the cost of excavation to
be proportional to the volume of
excavation, the objective function
can be stated as follows:
Minimize: f = dlw

nasir m mirza

18

Example: Building Design

Constraints

All optimization variables are not

independent. Since the height of
each story is given, the number of
stories and the total height are
related to each other as follows:
(d + h)/n = 3.5

Also, the requirement that the ratio

between the plan dimensions must
be equal to the golden ratio makes
the two plan dimensions dependent
on each other as follows:
l =1.618w
nasir m mirza

19

Example: Building Design

Constraints

The total floor space is equal to the

area per floor multiplied by the
number of stories. Thus, the floor
space requirement can be expressed
as follows:
new > 20,000

The lot size places the following

limits on the plan dimensions:
l 50 ; w 50
nasir m mirza

20

Example: Building Design

Constraints:

The energy cost is proportional to the

exposed building area which includes
the areas of the exposed sides and the
roof.

Thus, the energy budget places the

following restriction on the design:
100(2hl + 2hw +lw) 225,000

Explicitly state that the design

variables cannot be negative.

l, w, h, d > 0 ; n > 1 must be an

integer.
nasir m mirza

21

Example: Building Design

The complete optimization problem

can be stated as follows:
Find (n, l, w, h, d) in order to
Minimize:
f = dlw
Subject to:
(d+h)/n = 3.5
l = 1.618w
nlw 2: 20,000
l 50
w 50
100(2hl +2hw +lw) 225,000
n1
l, W, h, d 0
nasir m mirza

22

Example: Plant Operation

A tire manufacturing plant has the ability to

produce both radial and bias-ply automobile tires:
During the upcoming summer months, they have
contracts to deliver tires as follows.
Date

Radial tires

Bias-ply tires

June 30

5000

3000

July 31

6000

3000

August 31

4000

5000

Total:

15000

11000

nasir m mirza

23

Example: Plant Operation

The plant has two types of machines, gold

machines and black machines, with appropriate

molds to produce these tires.
The following production hours are available
during the summer months:
month

Gold machine

Black machine

June

700

1500

July

300

400

August

1000

300

nasir m mirza

24

Example: Plant Operation

The production rates for each machine type and tire
combination, in terms of hours per tire, are as follows

Type

Gold machine

Black machine

Radial

0.15

0.16

.12

0.14

Bias-ply

nasir m mirza

25

Example: Plant Operation

The labor costs of producing tires are $10.00 per operating hour,

regardless of which machine type is being used or which tire is being

produced.

The material costs for radial tires are $5.25 per tire and those for biasply tires are $4.15 per tire.

Finishing, packing and shipping costs are $0.40 per tire.

The excess tires are carried over into the next month but are subjected
to an inventory carrying charge of $0.15 per tire.

Wholesale prices have been set at $20 per tire for radials and $15 per
tire for bias-ply.

How should the production be scheduled in order to meet the delivery

requirements while maximizing profit for the company during the
three-month period?

nasir m mirza

26

Example: Plant Operation

Optimization Variables
From the problem statement, it is clear that the only variables that the production
manager has control over are the number and type of tires produced on each
machine type during a given month. Thus the optimization variables are as
follows:
x1 = Number of radial tires produced in June on the gold machines
x2 = Number of radial tires produced in July on the gold machines
x3 = Number of radial tires produced in August on the gold machines
x4 = Number of bias-ply tires produced in June on the gold machines
x5 = Number of bias-ply tires produced in July on the gold machines
x6 = Number of bias-ply tires produced in August on the gold machines
x7 = Number of radial tires produced in June on the black machines
x8 = Number of radial tires produced in July on the black machines
x9 = Number of radial tires produced in August on the black machines
x10 = Number of bias-ply tires produced in June on the black machines
x11 = Number of bias-ply tires produced in July on the black machines
x12 = Number of bias-ply tires produced in August on the black machines
nasir m mirza

27

Example: Plant Operation

Objective Function:

The objective of the company is to maximize profit. The profit is equal to

the total revenue from sales minus all costs associated with the
production, storing, and shipping.

Revenue from sales:

R = $20(x1 +x2 +x3 +x7 +x8 +x9) +$15(x4 +x5 +x6 +x10 +x11 +x12)

Material costs:
M = $5.25(x1+x2 +x3 +x7 +x8 +x9) + $4.15(x4 +x5 +x6 +x10
+x11+x12)

Labor costs:
L = $10{ 0.15(x1 +x2 +x3) +0.16(x7 +x8 +x9) +0.12(x4 +x5 +x6)
+ 0.14(x10 +x11 +x12)}

Finishing, packing, and shipping costs:

F = $0.14(xI +x2 +x3 +x4 +x5 +x6 +x7 +x8 +x9 +x10 +x11+x12)

nasir m mirza

28

Example: Plant Operation

The inventory-carrying charges are a little difficult to

formulate. Assuming no inventory is carried into or out of the

three summer months, we can determine the excess tires
produced as follows:

Excess tires produced by June 30

Ex1= (X1+X7 - 5,000) + (X4 +X10 - 3,000)

Excess tires produced by July 31

Ex2= (Xl +X2 +X7 +XB -11,000) + (X4 +X5 +X10 +X11 6,000)

nasir m mirza

29

Example: Plant Operation

By assumption, there are no excess tires left by the end of August

31. At $0.15 per tire, the total inventory carrying charges are as
follows:

Inventory cost:
IC = $0.15((x1+x7 - 5,000) + (x4 +x10 - 3,000) + (x1 +x2 +x7 +x8
-11,000) + (x4 +x5 +x10 +x11 - 6,000))

By adding all costs and subtracting from the revenue/ the objective
function to be maximized can be written as follows:

Maximize: f = 3,750 +12.55x1 + 12.7x2 +12.85x3 +8.95x4 + 9.1x5

+ 9.25X6 + 12A5x7 + 12.6x8 +12.75X9 +8.75x10 + 8.9x11
+9.05x12

nasir m mirza

30

Example: Plant Operation

Constraints
In a given month, the company must meet the delivery
contracts. During July and August, the company can also use
the excess inventory to meet the demand. Thus, the delivery
contract constraints can be written as follows:
x1 + x7 5.000
x4 + x10 3,000
x1 + x2 + x7 + x8 11,000
x4 + x5 + x10 + x11 6,000
x1 + x2 + x3 + x7 + x8 + x9 = 15,000
x4 + x5 + x8 + x10 + x11 + x12 = 11,000
Note that the last two constraints are expressed as equalities to
stay consistent with the assumption that no inventory is carried
into September.
nasir m mirza

31

Example: Plant Operation

Constraints
The production hours for each machine are limited. Using :the
time that it takes to produce a given tire on a given machine type,
these limitations can be expressed as follows:
0.15x1 + 0.12x4 700
0.15x2 + 0.12x5 300
0.15x3 + 0.12x6 1,000
0.16x7 + 0.14x10 1,500
0.16x8 + 0.l4x11 400
0.16x9 + 0.14x12 300
The only other requirement is that all optimization variables
must be positive.
nasir m mirza

32

Example: Plant Operation

The complete optimization problem can be stated as follows:
Find (x1, x2, . . . , x12) in order to
Maximize: f = 3,750 + 12.55x1 + 12.7x2 + 12.85x3 + 8.95x4 + 9.1x5 + 9.25x6
+ 12.45x7 + 12.6x8 + 12.75,x9 + 8.75x10 + 8.9x11 + 9.05x12
Subject to constraints:
x1 + x7 5.000
x4 + x10 3,000
x1 + x2 + x7 + x8 11,000
x4 + x5 + x10 + x11 6,000
x1 + x2 + x3 + x7 + x8 + x9 = 15,000
x4 + x5 + x8 + x10 + x11 + x12 = 11,000
0.15x1 + 0.12x4 700
0.15x2 + 0.12x5 300
0.15x3 + 0.12x6 1,000
0.16x7 + 0.14x10 1,500
0.16x8 + 0.l4x11 400
0.16x9 + 0.14x12 300

nasir m mirza

33

Example : Data Fitting

Finding the best function
that fits a given set of
data can be formulated as
an optimization problem.

As an example, consider
fitting a surface to the
data given in the
following table:

nasir m mirza

Point

zobserved

1.26

0.25

2.19

0.5

0.76

0.75

1.26

1.00

1.86

1.25

1.43

1.50

1.29

1.75

0.65

2.00

1.60

34

Example : Data Fitting

The form of the function is first chosen based on prior
knowledge of the overall shape of the data surface.
For the example data, consider the following general form:
z(computed) = c1x2 + c2y2 + c3xy

The goal now is to determine the best values of coefficients c1 ,

c2 and c3 in order to minimize the sum of squares of error
between the computed z values and the observed ones.

nasir m mirza

35

Example : Data Fitting

Optimization Variables:
Values of coefficients c1, c2, and c3
Objective Function:
Minimize f = [ zobserved (xi , yi) - zcomputed (xi , yi) ]2
Using the given numerical data, the objective function can be
written as follows:
f = (1.26 - c2)2 + (2.19 - 0.0625c1 - c2 - 0.25c3)2 + . . .
+ (1.6 4c1 4c2 4c3)2

or

f = 18.7 - 32.8462c1 + 34.2656c12 - 65.58c2 + 96.75c1c2 + 84c22 43.425c3 +79.875c1c3 + 123.c2 c3 +48.375c32
nasir m mirza

36

Example : Data Fitting

The complete optimization problem can be stated as follows:
Find (c1, c2, and c3) in order to Minimize:
f = 18.7 - 32.8462c1 + 34.2656c12 - 65.58c2 + 96.75c1c2 + 84c22
- 43.425c3 +79.875c1c3 + 123c2 c3 +48.375c32

This example represents a simple application from a wide field

known as Regression Analysis. For more details refer to many
excellent books on the Subject

nasir m mirza

37

The Standard Form of an Optimization

Problem

A large class of situations involving optimization can be

expressed in the following form:

Find a vector of optimization variables, x = (x1, x2, . . . , xn)T in
order to Minimize or Maximize an objective function,
f(x) = f(.)
Subject to:
gi(X) < 0 ; i = 1, 2, ,m ; inequality constraints (LE)
hi(X) = 0 ; i = 1, 2, , P ; Equality constraints (EQ)
xL < xi < xU ; i = 1, 2, , n ; Bounds on variables
The bounds constraints are in fact inequality constraints. They
are sometimes kept separate from the others because of their
simple form.
nasir m mirza

38

Multiple Objective Functions

There can be more than one objective to be optimized.
For example, we may want to maximize profit from an car that

we are designing and at the same time minimize the possibility

of damage to the car during a collision.
These types of problems are difficult to handle because the
objective functions are often contradictory.
Then assign weights to each objective function depending on
their relative importance and then define a composite objective
function as a weighted sum of all these functions, as follows:
f (x) = w1f1(x) +w2 f2(x) + . . .
where w1 , w2 , . . . are suitable weighting factors.
The success of the method clearly depends on a clever choice
of these weighting factors.

nasir m mirza

39

Classification of Optimization Problems

The methods for solving the general form of the optimization
problem tend to be complex
It requires considerable numerical effort.
Special, more efficient methods are available for certain
specials forms of the general problem.
For this purpose/ the optimization problems are usually
classified into the following types:
Unconstrained Problems
Linear Programming (LP) Problems
Quadratic Programming (QP) Problems
Nonlinear Programming (NLP) Problems

nasir m mirza

40

Unconstrained Problems
These problems have an objective function but no constraints.
The data-fitting problem, presented in the first section, is an

example of an unconstrained optimization problem.

The objective function must be nonlinear (because the
minimum of an unconstrained linear objective function is
obviously infinity).
Problems with simple bounds on optimization variables can
often be solved first as unconstrained.
After examining different options, one can pick a solution
that satisfies the bounds on the variables.

nasir m mirza

41

Linear Programming (LP) Problems

If the objective function and all the constraints are linear

functions of optimization variables, the problem is called a
linear programming problem.

The tire plant management problem presented here is an

example of a linear optimization problem.

An efficient and robust algorithm, called the Simplex method,

is available for solving these problems.

nasir m mirza

42

Quadratic Programming (QP) Problems

If the objective function is a quadratic function and all

constraint functions are linear functions of optimization
variables the problem is called a quadratic programming
problem.
The portfolio management problem is an example of a
quadratic optimization problem.
It is possible to solve QP problems using extensions of the
methods for LP problems.

nasir m mirza

43

Nonlinear Programming (NLP) Problems

The general constrained optimization problems, in which one

or more functions are nonlinear, are called nonlinear
programming problems.
The building design problem, presented in here, is an
example of a general nonlinear optimization problem.

nasir m mirza

44

Types of optimization methods

Single-variable optimization algorithms

Multi-variable optimization algorithms
Constrained optimization algorithms
Specialized optimization algorithms
Nontraditional optimization algorithms

nasir m mirza

45

Types of optimization methods

Single-variable optimization algorithms

Because of their simplicity, single-variable optimization

techniques will be discussed first.

These algorithms provide a good understanding of the

properties of the minimum and maximum points in a function
and how optimization algorithms work iteratively to find the
optimum point in a problem.

The algorithms are classified into two categories

Direct methods and

gradient-based methods.

Direct methods do not use any derivative information of the

objective function; only objective function values are used to
guide the search process.
nasir m mirza

46

Types of optimization methods

Multi-variable optimization algorithms
A number of algorithms for unconstrained,
multivariable optimization problems will be
discussed.

These algorithms demonstrate how the search

for the optimum point progresses in multiple
dimensions. Depending on whether the
gradient information is used or not used, these
algorithms are also classified into direct and
gradient-based techniques.

nasir m mirza

47

Types of optimization methods

Constrained optimization algorithms

Constrained optimization algorithms use the single

variable and multivariable optimization algorithms
repeatedly and simultaneously maintain the search
effort inside the feasible search region.

Since these algorithms are mostly used in

engineering optimization problems, the discussion
of these algorithms will be done in detail.

nasir m mirza

48

Types of optimization methods

Specialized optimization algorithms

There exist a number of structured algorithms,

which are ideal for only a certain class of
optimization problems.

Two of these algorithms - integer programming

and geometric programming-are often used in
engineering design problems.

Integer programming methods can solve

optimization problems with integer design

variables. Geometric programming methods solve
optimization problems with objective functions
and constraints written in a special form.

nasir m mirza

49

Types of optimization methods

Nontraditional optimization
algorithms
There exist a number of other search and
optimization algorithms which are
comparatively new and are becoming popular
in engineering design optimization problems in
the recent past.

Two such algorithms: genetic algorithms and

simulated annealing- will be discussed.

nasir m mirza

50