Basic Concepts in Optimization

Professor: Edward I. Olmedo

 Extreme point of a function f(X)
o Defines either a maximum or a
minimum of the function f
Necessary and Sufficient Conditions for Extrema
 Assuming that the first and second partial
derivatives of f(X) are continuous at every X
 For X0 to be an extreme point of f(X), the
partial derivative of f(X0) should be zero
f(X0) = 0

 A sufficient condition for a stationary point X0

to be extremum is that the Hessian matrix H=
2f(X) evaluated at X0 is:
o Positive when X0 is a minimum point
o Negative when X0 is a maximum point
Using an Example
f(X1, X2, X3) = X1 – 2X12 + X2 – 4X22 + X3 – 8X32 + 2X1X3

 The necessary condition for the extrema is:

f(X0) = 0

f
 = 1 − 4𝑥1 + 2𝑥3 = 0
𝑥1
f
 = 1 − 8𝑥2 = 0
𝑥2
f
 = 1 − 16𝑥3 + 2𝑥1 = 0
𝑥3

 X1 = 0.3; X2 = 0.125; X3 = 0.1

 Named after Joseph Louis Lagrange, an Italian
born mathematician
 Method widely used to solve challenging
constrained optimization problems
o Finding the maximum/minimum of a function
subject to a set of constraints
 Its advantage is that it allows the optimization
to be solved without explicit parameterization
in terms of the constraints
L(x, ) = f(x) + [  * g(x) ]

Where:
  is the Lagrange multiplier
 f(x) is a function representing controllable variables
o Such as cost-function that depends on number of
controllable output
 g(x) is a function representing the constraints
o Such as the balance between supply and demand
Using Karush-Kuhn-Tucker Conditions
 Originally developed by Harold W. Kuhn and
Albert W. Tucker and later developed by
William Karush
 Are necessary conditions for the solution of
non-linear problems that is desired to be
optimal
o Provided that some regularity conditions are
satisfied
Using Karush-Kuhn-Tucker Conditions
KKT Optimality Conditions
L
1.
xi
 Partial derivative of Lagrange with respect
to xi should be equal to zero
2. gi(x0) = 0:
 Restatement of constraint conditions
3. hi(x0) < 0
 Restatement of constraint conditions
4. i0gi(x0) = 0 where i0 > 0
 Slack variables or excess is equal to zero in
optimal condition
Example: Generation Supply and Demand
 Consider two generating plants supplying a given power
demand of 50 MW. The fuel costs of each is related to
the output power as detailed below:
2
F1 =5P1 + 0.01P1
2
F2 =2P2 + 0.04P2

 The objective is to minimize the total cost of generation

while satisfying the demand
Example: Generation Supply and Demand
 To solve, we need to determine f(x) and g(x)
 As previously discussed f(x) is the function we want to
optimize that represents controllable variables, while
g(x) represents the constraint
 Based on the problem, we want to minimize the cost of
generation, which is directly proportional to fuel cost
o Hence, f(x) is equal to the sum of F1 and F2

 g(x) is a function representing the constraints

o As a background, we have two types of constraints:
(a) equality and (b) inequality
o As of now, we will only use the equality constraint of
supply being equal to the demand, or:
 P1 + P2 = Demand
Example: Generation Supply and Demand
 Again:
o f(x) = F1 + F2
o g(x) = 0 = Demand – (P1 + P2)

 Let us formulate using the Lagrange method

L(x, ) = f(x) + [  * g(x) ]
= (F1 + F2) + [  * (Demand – P1 – P2) ]
2 2
= (5P1 + 0.01P1 + 2P2 + 0.04P2 ) + [  * (Demand – P1 – P2) ]
Example: Generation Supply and Demand
 Using KKT, get the partial derivative with respect to each variable:

2 2
L(x, ) = (5P1 + 0.01P1 + 2P2 + 0.04P1 ) + [  * (Demand – P1 – P2) ]

L
=(5 + 0.02P1 + 0 + 0) + [  ∗ (0 – 1 – 0) ] = 0
P1

L
=(0 + 0 + 2 +0.08P2 ) + [  ∗ (0 – 0 – 1) ] = 0
P2

L
=(0 + 0 + 0 +0) + [ 1 ∗ (Demand – P1 – P2) ] = 0


Since Demand = 50, then the answers are:

P1 = 10 MW; P2 = 40 MW;  = P5.2/kWh
Example: Generation Supply and Demand
 At different levels of demand:

Case Demand P1 P2 
1 50 10 40 5.2
2 100 50 50 6.0
3 200 130 70 7.6

 Since problem is quadratic, it does not mean that twice the

demand will result to generator schedules of P1 and P2 to be
twice of what they were
 Also, you will notice from Case 2 to Case 3:
o Case 3 demand was twice that of Case 2
o The generator schedules P1 and P2 did not increase
proportionately from Case 2 to Case 3
o Case 1 showed P2 having higher MW schedule, while Case 3
showed P1 having the higher MW schedule
 Continuing from the previous lecture where we tried to
solve a problem using KKT on the Lagrange equation
o It is easy to solve when there are a few variables (or
generators) in the problem
o But in reality, there are lots of generators in the Power
System that we have to consider
o Solution by simple algebra is extremely difficult
o In this regard, we can use different numerical methods
(through iterative techniques) to come up with the
final solution, such as:
 Lambda-iteration method
 Gradient search method
 Newton Method
Simplifying the Lagrange Method
 The total cost can be represented as:
𝑛 𝑛

𝐶𝑡𝑜𝑡𝑎𝑙 = ෍ 𝐶𝑖 = ෍ 𝑖 + 𝑖 𝑃𝑖 + 𝑖 𝑃𝑖 2
𝑖=1 𝑖=1

Where 𝑖 , 𝑖 , and 𝑖 are coefficients

 We can insert Ctotal in the Lagrange equation as f(x)

L(x, ) = σ𝑛𝑖=1 𝑖 + 𝑖 𝑃𝑖 + 𝑖 𝑃𝑖 2 + [  * (𝑃𝐷𝑒𝑚𝑎𝑛𝑑 − σ𝑛𝑖=1 𝑃𝑖 ) ]

 We can also get the standard equation for all generators upon
getting the partial derivative (KKT condition 1) using the above
equation
 − 𝑖
𝑃𝑖 =
2𝑖
Method 1: Lambda-Iteration Method
 Let us not forget the following standard equation derived
previously
 − i
Equation (1) Pi =
2i

 We will be trying to get the nearest value for lambda using this
method. The following are the steps:
1. Pick any high and low value for lambda (H and L )
2. Get the mid-point of H and L . This shall be M
 +
3. M = H L
2
4. Get Pi using M in equation (1)
5. If σm
i=1 Pi M − PDemand > 0, then H = M for the next iteration
6. If σm
i=1 Pi M − PDemand < 0, then L = M for the next iteration
7. Continue steps 5 and 6 until the absolute difference between H
and L is < a tolerance 
Example for Method 1: Lambda-Iteration Method
Using the same problem before
 Consider two generating plants supplying a given power
demand of 50 MW. The fuel costs of each is related to
the output power as detailed below:
2
F1 =5P1 + 0.01P1
2
F2 =2P2 + 0.04P2

 We will solve this using the Lambda-iteration method

Example for Method 1: Lambda-Iteration Method
Using the same problem before (continuation…)
 Let us pick H to be 20, L shall be 0

 For generator 1:
 −5  −5
P1 = =
2 0.01 0.02
 For generator 2:
 −2  −2
P2 = =
2 0.04 0.08

 Let us assume that the tolerance  = 0.005

Example for Method 1: Lambda-Iteration Method
Using the same problem before (continuation…)
 Here’s the process and the final answer

 We get almost exactly the same answer as before

Lambda-Iteration Method with Gen Limits
 Using the same problem before, but this time, let us set
limits for generators
o 5 < P1 < 20
 Generator 1 has minimum loading of 5 MW and a
maximum capacity of 20 MW
o 0 < P2 < 35
 Generator 2 has minimum loading of 0 MW and a
maximum capacity of 35 MW

 Use the additional process below in the lambda-iteration

method when there are generator limits
o If Pi > Pi-max then set Pi = Pi-max
o If Pi < Pi-min then set Pi = Pi-min
Example for Method 1: Lambda with Limits
 Here’s the process and the final answer

 Due to the limitation of P2 (up to 35 MW), P1 was scheduled

at a higher MW value (15 MW)
 This resulted to a higher cost (as seen in the new lambda)
from P5.2/kWh to P5.3/kWh
Method 2: Gradient Search Method
 It is used for optimizing functions that are
twice continuously differentiable.
 It generates successive points, starting from a
given initial point, either for maximization or
for minimization of the function.
 The gradient of the function at a point is used
because it is indicative of the fastest rate of
maximization or minimization
Method 2: Gradient Search Method
 To find the minimum of a function f(x)

𝑥 𝑘+1 = 𝑥 𝑘 − 𝛻𝑓
Where:
f
𝛻𝑓 =
xi
 is a scalar value
Method 2: Gradient Search Method
Remembering the Lagrange function
L(x, ) = f(x) + [  * g(x) ]

 We can derive the next lambda using Taylor Series

𝑘+1 = 𝑘 + ∆𝑘
Where:
∆Pk
∆k =
σm
1
i=1 2
i

∆Pk = PDemand − σm P
i=1 i
𝑘
Example for Method 2: Gradient Search Method
Using the same problem before
 Consider two generating plants supplying a given power
demand of 50 MW. The fuel costs of each is related to
the output power as detailed below:
2
F1 =5P1 + 0.01P1
2
F2 =2P2 + 0.04P2

 We will solve this using the Gradient Search method

Example for Method 2: Gradient Search Method
 Let us pick the initial lambda, 0 = 10
 Remember the standard equation
o For generator 1:
 −5  −5
P1 = =
2 0.01 0.02
o For generator 2:
 −2  −2
P2 = =
2 0.04 0.08

 With 0 = 10, then P1 = 250; P2 = 100

 ∆Pk = PDemand − σ
m P 𝑘 = 50 – (250 + 100) = -300
i=1 i
∆Pk −300
 ∆k = = = - 4.8
σm
1 1 + 1
i=1 2 2 0.01 2 0.04
i
Example for Method 2: Gradient Search Method
 We then obtain the lambda for the next iteration using
the formula previously stated
o k+1 = k + ∆k

 Let us assume  = 1.0

o Hence, k+1 = k + ∆k

 For iteration 1, we will get the following lambda

o 1 = 0 + ∆k = 10 – 4.8 = 5.2

 Repeat the problem until ∆Pk < tolerance 

 With 1 being 5.2, we will see that ∆Pk =0, then 2 = 0

o Hence, final  is P5.2/kWh
o P1 and P2 shall be 10 MW and 40 MW, respectively
Gradient Search Method with Gen Limits
 Similarly, we can solve using Gradient Search for
problems that has generator limits
 Using the same sample before, where the generators’
limits are as follows
o 5 < P1 < 20
o 0 < P2 < 35

 Use the same process in lambda-iteration for the

gradient search method when there are generator limits
o If Pi > Pi-max then set Pi = Pi-max
o If Pi < Pi-min then set Pi = Pi-min

 We should achieve the same answer where P1 = 15 MW

and P2 = 35 MW, while  = P5.3/kWh
Method 3: Newton’s Method
 Let us make translate the partial derivatives of the
lagrange equation to a matrix form
 Remember the following derived functions for the same
problem before

2 2
L(x, ) = (5P1 + 0.01P1 + 2P2 + 0.04P1 ) + [  * (Demand – P1 – P2) ]

L
=(5 + 0.02P1 + 0 + 0) + [  ∗ (0 – 1 – 0) ] = 0
x1

L
=(0 + 0 + 2 +0.08P2 ) + [  ∗ (0 – 0 – 1) ] = 0
x2

L
=(0 + 0 + 0 +0) + [ 1 ∗ (Demand – P1 – P2) ] = 0

Method 3: Newton’s Method
 Re-arranging the previous equations will

L
: 0.02P1 + 0 −  = −5
x1

L
: 0 + 0.08P2 −  = −2
x2

L
: P1 + P2 + 0= Demand

Method 3: Newton’s Method
 In matrix form:

0.02 0 -1 P1 -5
0 0.08 -1 P2 = -2
1 1 0  Demand

 It is the same as:

o A x B = C
o Where A, B, and C are matrices

 To get the unknown matrix B, we can get transpose matrix A

o B = A-1 x C
o Where A-1 is the inverse matrix of A

 We can now obtain the values for P1, P2, and lambda
Method 3: Newton’s Method with Gen Limits
 We can further expand the Lagrange equation based on
Condition 4 of the KKT
o i0gi(x0) = 0 where i0 > 0
 Slack variables or excess is equal to zero in optimal
condition
 This expansion is to consider generator limits in the
equation. It is formulated as follows
Method 3: Newton’s Method with Gen Limits
 If we solve this matrix equation

0.02 0 -1 P1 -5
0 0.08 -1 P2 = -2
1 1 0  Demand

 We will get P1 = 10 MW, P2 = 40 MW, and  = 5.2

 But if the generators’ limits are as follows

o 5 < P1 < 20
o 0 < P2 < 35

 P2 is already beyond its capacity of 35 MW

 We now have to fix P2 to this Pmax of 35 MW

Method 3: Newton’s Method with Gen Limits

 We now have a new variable 𝑖 𝑚𝑎𝑥

 We now have to get the partial derivative of the Lagrange

equation considering P2, which reached its max
Method 3: Newton’s Method with Gen Limits

2 2
L(x, ) = (5P1 + 0.01P1 + 2P2 + 0.04P1 ) + [  * (Demand – P1 – P2) ] +
max
2 𝑚𝑎𝑥 P2 − P2

L
=(5 + 0.02P1 + 0 + 0) + [  ∗ (0 – 1 – 0) ] + 0 = 0
P1

L
=(0 + 0 + 2 +0.08P2 ) + [  ∗ (0 – 0 – 1) ] − 2 𝑚𝑎𝑥 = 0
P2

L
=(0 + 0 + 0 +0) + [ 1 ∗ (Demand – P1 – P2) ] + 0 = 0


L max
= 0 + 0 + P2 − P2 = 0
 2 𝑚𝑎𝑥
Method 3: Newton’s Method with Gen Limits
 Adding the last equation, we now have four equations, four
unknowns

0.02 0 -1 0 P1 -5
0 0.08 -1 -1 P2 -2
=
1 1 0 0  Demand
0 1 0 0 2
𝑚𝑎𝑥
P2max

 With Demand being 50 MW, and P2max being 35 MW, we will

get the following results
o P1 = 15 MW
o P2 = 35 MW
o  = 5.3
o 2 𝑚𝑎𝑥 = - 0.5
End of Presentation