MS-C2105 Introduction to Optimization Ehtamo

Solutions 9
Demo 1: KKT conditions with inequality constraints
Using the Karush-Kuhn-Tucker conditions, see if the points x = (x1 , x2 ) = (2, 4) or
x = (x1 , x2 ) = (6, 2) are the local optima of the problem. Draw a picture.

max x31 + 3x2

s.e. −x21 + x2 ≤0
x1 + 2x2 − 10 ≤ 0
x1 − 3x2 ≤0

Solution
The feasible set is illustrated in the following picture.

8
−x21 + x2 ≤ 0
7

6 x1 + 2x2 − 10 ≤ 0

5
x2

0 x1 − 3x2 ≤ 0

−1
−3 −2 −1 0 1 2 3 4 5 6 7
x1

The KKT conditions are:

∇L = 0
λi gi (x) = 0 ∀i
λi ≤ 0 ∀i
where the function L(x, λ) is the problem’s Lagrange function. The function is
defined as
L(x, λ) = f (x) + λ1 g1 (x) + λ2 g2 (x) + λ3 g3 (x)

Functions gi (x) are constraint functions and λi the respective Lagrange multipliers.
Constraints are always of the form gi (x) ≤ 0. Now the constraint functions gi (x)
are:
g1 (x) = −x21 + x2
g2 (x) = x1 + 2x2 − 10
g3 (x) = x1 − 3x2

1
Note. The conditions above are the KKT conditions for a maximization problem.
For a minimization problem the conditions are similar, except the inequality of the
last condition is reversed, i.e. λi ≥ 0 ∀i.

Let us first look at x = (2, 4). We determine which constraints are active:

g1 (2, 4) = −22 − 4 = 0

g2 (2, 4) = 2 + 2 · 4 − 10 = 0
g3 (2, 4) = 2 − 3 · 3 = −7

Constraints g1 and g2 are active in the current point, i.e. they have the value 0.
The third constraint on the other hand is not active. According to the third KKT
condition, the Lagrange multipliers of inactive constraints are zero, i.e. now λ3 = 0.

The gradients of the objective function and constraint functions are:

3x21
       
2x1 1 1
∇f (x) = ∇g1 (x) = ∇g2 (x) = ∇g3 (x) =
3 3 2 −3

The gradients at the point under examination:

∇L(2, 4, λ1 , λ2 , λ3 ) = ∇f (2, 4) + λ1 ∇g1 (2, 4) + λ2 ∇g2 (2, 4) + λ3 ∇g3 (2, 4)

3 · 22
       
2·2 1 1
= + λ1 + λ2 + λ3
3 3 2 −3
 
12 + 4λ1 + λ2 + λ3
=
3 + 3λ1 + 2λ2 − 4λ3

We remember that λ3 = 0. Now the first KKT condition (∇L = 0) yields:

   
12 + 4λ1 + λ2 0
∇L = =
3 + 3λ1 + 2λ2 0

From the lower equation we solve λ1 :

2λ2 + 3
λ1 = −
3
Replace this into the upper equation:
2λ2 − 3
12 + · 4 − λ2 = 0
3
Now we can solve
21
λ2 = ,
5
and from the lower equation
24
λ1 = − .
5
We have solved both Lagrange multipliers and the other one is positive. The KKT
conditions require that at the optimum all multipliers are negative i.e. the KKT
conditions are not satisfied and the point is not a local optimum.

Let us examine the other point: x = (6, 2). In this point the active constraints are:

g1 (6, 2) = −62 + 2 = −34

g2 (6, 2) = 62 · 2 + 10 = 0

2
 g3 (6, 2) = 6 − 3 · 2 = 0
The active constraints are constraints 2 and 3. The first constaint is not active, so
λ1 = 0. We replace the point x = (6, 2) into the Lagrange function’s gradient and
require it to be zero:
3 · 62 + λ2 + λ3
   
0
∇L(6, 2, λ1 , λ2 , λ3 ) = = .
3 + 2λ2 − 3λ3 0
Now we can solve the Lagrange multipliers, which are:

λ1 = −65.4 ja λ2 = −42.6.

The multipliers are negative and thus the KKT conditions are satisfied. The point
satisfies the KKT conditions and it is a optimum candidate.

Demo 2: KKT conditions with equality and inequality constraints

Solve the problem graphically and see if the point satisfies the KKT conditions.
min x21 + x22
s.e. (x1 − 3)2 + 1 ≤ x2
1
2 x1 − x2 = −1

Solution
The feasible area is illustrated in the following picture.

4.5 (x1 − 3)2 + 1 ≤ x2

3.5

3
x2

2.5

1.5

1
1 2 x1 − x2 = −1

0.5

0
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
x1

In the problem we minimize the distance to the origin. The optimum is thus the
nearest point of the feasible set, x = (2, 2).

The KKT conditions are:

L(x, λ, µ) = f (x) + λg(x) + µh(x)
∇L =0
λg(x) = 0 ∀i
λi ≥ 0 ∀i

3
Note. The equality constraints’ Lagrange multipliers do not have similar positivity
or negativity constraints as the inequality constraints. The equality constraints’
Lagrange multipliers are often denoted µi .

The constraint functions g(x) and h(x) are

g(x) = (x1 − 3)2 − x2 + 1

1
h(x) = x1 − x2 + 1
2

We determine if the inequality constraint is active.

g(2, 2) = (2 − 3)2 + 1 − 2 = 0

The constraint is active and thus the respective Lagrange multiplier can be non-zero.
We solve the point in which the gradient of the Lagrange function is zero. For this,
we first need to determine the gradients of the objective and constraint functions:
     1 
2x1 2(x1 − 3) 2
∇f (x) = ∇g(x) = ∇h(x) =
2x2 −x2 −1

The gradient of the Lagrange function is zero in the optimum:

∇L(2, 2, λ, µ) = ∇f (2, 2) + λ∇g(2, 2) + µ∇h(2, 2)

2 · 2 + 2(2 − 3)λ + 21 µ
 
=
2 · 2 − 2λ − µ
4 − 2λ + 12 µ
   
0
= =
4 − 2λ − µ 0
We can solve the equations e.g. by first solving λ from the first equation and then
replacing it into the second. This yields λ = 2 and µ = 0. The Lagrange multiplier
of the inequality is positive, so point (2,2) satisfies the necessary KKT conditions of
the minimization problem.

Problem 1: KKT conditions with inequality constraints

Using the Karush-Kuhn-Tucker conditions, see if the point x = (x1 , x2 ) = (2, −1) is
the optimum of the problem

max x21 + x22

s.e. x22 + 1 ≤ x1
x1 ≤ 2

4
Solution

2
x1 ≤ 2
1.5

0.5
x2

−0.5

−1 x22 + 1 ≤ x1

−1.5

−2
−1 −0.5 0 0.5 1 1.5 2 2.5 3 3.5 4
x1

The constraint functions are:

g1 (x) = x22 + 1 − x1
g2 (x) = x1 − 2

We determine the active constraints at the point:

g1 (2, −1) = (−1)2 + 1 − 2 = 0
g2 (2, −1) = 2 − 2 = 0
Both constraints are active. We determine if the point is the zero point of the
Lagrange function’s gradient. The gradients of the objective and constraint functions
are:      
2x1 −1 1
∇f (x) = ∇g1 (x) = ∇g2 (x) =
2x2 2x2 0
We write the gradient of the Lagrange function in the examination point (2,-1):
∇L(2, −1, λ1 , λ2 ) = ∇f (2, −1) + λ1 ∇g1 (2, −1) + λ2 ∇g2 (2, −1)
     
2 · 2 + λ1 · (−1) + λ2 · 1 4 − λ1 + λ2 0
= = =
2 · (−1) + λ1 · 2 · (−1) + λ2 · 0 −2 − 2λ1 0
Now λ1 can be solved from the lower equation: λ1 = −1. Replacing this into the
upper equation yields λ2 = −5. In a maximization problem the KKT conditions
require that the Lagrange multipliers are negative. Now the multipliers are negative
and so the point satisfies the KKT conditions. Thus, the point is a local optimum
candidate. According to the picture the point is the global optimum.

Problem 2: KKT conditions with equality and inequality constraints

Solve the problem and see if the solution satisfies the KKT conditions.
min x1
s.e. (x1 + 4)2 − 2 ≤ x2
x1 − x2 + 4 = 0
x1 ≥ −10

5
Solution
The feasible area:

2 x1 ≥ −10

1 (x1 + 4)2 − 2 ≤ x2
x2

−1

−2
x1 − x2 + 4 = 0

−3
−11 −10 −9 −8 −7 −6 −5 −4 −3 −2 −1
x1

The constraint functions are:

g1 (x) = (x1 + 4)2 − 2 − x2

g2 (x) = x1 − x2 + 4

From the picture we see that the optimum is at (-5,-1). Let’s determine if it satifies
the KKT conditions. The active constraints:

g1 (−5, −1) = (−5 + 4)2 − 2 − (−1) = 0

g2 (−5, −1) = −(−5) − 10 = −5

Constrain 1 is active, but the other inequality constraint is not. This means that
the Lagrange multiplier of the second inequality constraint is zero, λ2 = 0. Next
we see if the point is the zero point of the gradient of the Lagrange function. The
gradients of the objective and constraint functions are:
       
1 2(x1 + 4) −1 −1
∇f (x) = ∇g1 (x) = ∇h(x) = ∇g2 (x) =
0 −1 1 0

The gradient of the Lagrange function is zero:

∇L(−5, −1, λ1 , µ, λ2 ) = ∇f (−5, −1)+λ1 ∇g1 (−5, −1)+µ∇h(−5, −1)+λ2 ∇g2 (−5, −1)
     
1 + λ1 · 2(−5 + 4) + µ · (−1) 1 − 2λ1 − µ 0
= = =
0 + λ1 · (−1) + µ · 1 −λ1 + µ 0
From the lower equation we get λ1 = µ, which can be replaced in the upper equation.
This yields 1 − 2µ − µ = 0, i.e. µ = 31 . Furthermore, λ1 = µ = 13 . The problem is a
minimization problem and the KKT conditions require that the Lagrange multipliers
for inequality constraints are positive. This is now the case and the point satisfies
the KKT conditions.

6
Problem 3: Linear Programming Problem
a) Solve the problem graphically and determine if the solution satisfies the KKT
conditions.

b) Find the dual of the optimization problem and solve it.

c) Compare the solution of the dual and the Lagrange multipliers of the primal
problem.

max x1 + x2
s.e. x1 + 3x2 ≤ 9
2x1 + x2 ≤ 8
x1 ≥ 0, x2 ≥0

Solution
a) The feasible set is illustrated in the following picture:

4 x1 ≥ 0 2x1 + x2 ≤ 8

x1 + 3x2 ≤ 9
x2

0
x2 ≥ 0

−1
−1 0 1 2 3 4 5
x1

THe picture shows that the optimum is at the point (3,2). The constraint
functions are:
g1 (x) = x1 + 3x2 − 9
g2 (x) = 2x1 + x2 − 8
g3 (x) = −x1
g4 (x) = −x2

Determine the active constraints:

g1 (3, 2) = 3 + 3 · 2 − 9 = 0

g2 (3, 2) = 2 · 3 + 2 − 8 = 0
g3 (3, 2) = −3
g4 (3, 2) = −2

7
 Constraints 1 and 2 are active, constraints 3 and 4 are not. Thus, λ3 = λ4 = 0
and we can leave the respective terms out of the Lagrange function. The
gradient of the Lagrange function is zero:
∇L(3, 2, λ1 , λ2 ) = ∇f (3, 2) + λ1 ∇g1 (3, 2) + λ2 ∇g2 (3, 2)
   
1 + λ1 · 1 + λ2 · 2 0
= =
1 + λ1 · 3 + λ2 · 1 0
The solution of the equations is λ1 = − 15 and λ2 = − 52 . The multipliers are
negative and the the problem is a maximization problem. Thus, the KKT
conditions are satisfied.
b) The dual of the problem is:
min 9y1 + 8y2
s.e. y1 + 2y2 ≥ 1
3y1 + y2 ≥ 1
y1 ≥ 0, y2 ≥ 0

2
y1 ≥ 0

1.5

1
y2

y1 + 2y2 ≥ 1

0.5

0
y2 ≥ 0
3y1 + y2 ≥ 1

−0.5
−0.5 0 0.5 1 1.5 2 2.5 3
y1

The dual can be solved graphically or using Excel. The solution is y = ( 15 , 25 ).

c) We see that the solution of the dual of a linear problem is the same as the
Lagrange multipliers of the primal. Now the Lagrange multipliers are λ1 = − 15
and λ2 = − 25 , as the solution of the dual is y1 = 15 and y2 = 52 . Duality
and Lagrange multipliers are linked very closely to each other. More on the
connection between them will be introduced on lecture 12.
Note. Here the Lagrange multipliers are the negative of the dual variables.
Some times the Lagrange function is defined L(x, λ, µ) = f (x)−λg(x)−µh(x),
and then the dual variables and Lagrange multipliers have the same sign.

Problem 4: Something’s fishy

Solve the problem graphically and see if it satisfies the KKT conditions. If not, what
might be the reason?
max x1
s.e. x2 ≤ −(x1 − 4)3
x2 ≥ 0

8
Solution
The feasible set is illustrated in the following picture:

4 x2 ≤ −(x1 − 4)3

3
x2

0
x2 ≥ 0

−1

−2
0 1 2 3 4 5 6
x1

The optimum is at (x1 , x2 ) = (4, 0). Let us determine if the point satisfies the KKT
conditions. The constraint functions are

g1 (x) = x2 + (x1 − 4)3

g2 (x) = −x2
Both constraints are active at the optimum so the respective Lagrange multipliers
can be non-zero. The gradients of the objective and constraint functions are:

3(x1 − 4)2
     
1 0
∇f (x) = ∇g1 (x) = ∇g2 (x) =
0 1 −1

The gradient of the Lagrange function is zero:

∇L(4, 0, λ1 , λ2 ) = ∇f (4, 0) + λ1 ∇g1 (4, 0) + λ2 ∇g2 (4, 0)

1 + λ1 · 3(4 − 4)2 + λ2 · 0
     
1 0
= = =
0 + λ1 · 1 + λ2 · (−1) λ1 − λ2 0
The first equation cannot be satisfied with any values of the multipliers λ1 and λ2 .
The point is clearly the optimum, so what went wrong?

The KKT conditions are the necessary conditions, assuming the gradients of the
constraint functions are linearly independent. Now the gradients at the optimum
are    
0 0
∇g1 (4, 0) = ∇g2 (4, 0) =
1 −1
i.e. g1 (4, 0) = −g2 (4, 0). The gradients are linearly dependent and thus the optimum
does not need to satisfy the KKT conditions. Note that this point is still the
optimum.

Note. Usually one does not need to check the linear dependence of the constraints.

9
Problem 5: The legendary Enchanted forest
Sepeteus is trying to catch the legendary forest troll. The Enchanted forest limits
the whereabouts of the troll to a certain area. The area is defined the constraints :
x2 ≤ 2e−x1
x1 ≤2
x2 ≥ 41 x21
x2 ≤ 2x1 + 2
Sepeteus has at his disposal a trap. He has also build a noise making contraption
onto point (1,-1). He knows the troll is afraid of the contraption and runs as far away
from the contraption as possible. Although, the troll will stay inside the Enchanted
forest.

a) Solve the point where the troll will run, i.e. the Sepeteus should place the
trap. You can solve the point graphically.
b) Determine if the point satisfies the KKT conditions.

Solution
Let’s draw a map of the Enchanted forest:

x2 ≤ 2e−x1
2
x2 ≥ 41 x21

1
x2

−1

−2 x1 ≤ 2
x2 ≤ 2x1 + 2

−3
−3 −2 −1 0 1 2 3
x1

The noise making contraption is marked with a cross. The troll wants to maximize
the distance from the contraption so we can write the troll’s optimization problem:
max (x1 − 1)2 + (x2 + 1)2
s.e. x2 ≤ 2e−x1
x1 ≤ 2
x2 ≥ 14 x21
x2 ≤ 2x1 + 2
From the map we see that the troll should run to the point (0,2). Let’s determine
whether the point satisfies the KKT conditions. The constaint functions are:
g1 (x) = x2 − 2e−x1

10
 g2 (x) = x1 − 2
1
g3 (x) = x21 − x2
4
g4 (x) = −2x1 + x2 − 2

The active constraints:

g1 (0, 2) = 2 − 2e−0 = 0
g2 (0, 2) = 0 − 2 = −2
1
g3 (0, 2) = 02 − 2 = −2
4
g4 (0, 2) = −2 · 0 + 2 − 2 = 0
Constraints 2 and 3 are not active, so the respective multipliers are zero: λ2 = λ3 =
0. The gradients of the objective and constraint functions:
   −x   
2(x1 − 1) 2e 1 1
∇f (x) = ∇g1 (x) = ∇g2 (x) =
2(x2 + 4) 1 0
1
   
2 x1 −2
∇g3 (x) = ∇g4 (x) =
−1 1
The gradient of the Lagrange function is zero:

∇L(0, 2, λ1 , λ2 , λ3 , λ4 ) = ∇f (0, 2)+λ1 ∇g1 (0, 2)+λ2 ∇g2 (0, 2)+λ3 ∇g3 (0, 2)+λ4 ∇g4 (0, 2)

2(0 − 1) + λ1 · 2e−0 + 0 · 1 + 0 · 21 · 0 + λ4 · (−2)

 
=
2(2 + 1) + λ1 · 1 + 0 · 0 + 0 · (−1) + λ4 · 1
   
−2 + 2λ1 − 2λ4 0
= =
6 + λ1 + λ4 0
The solution to this is λ1 = − 52 and λ4 = − 72 . Both Lagrange multipliers are
negative, so the KKT conditions are satisfied.

11