Академический Документы
Профессиональный Документы
Культура Документы
Optimization
Written Examination 2012-12-20
Time: 14:00 - 19:00
Allowed Tools: Pocket Calculator, one A4 paper with notes (machine written, font
size minimum 10 pt)
Maximum number of points: 36 (18 points to pass)
All answers must be motivated the get full points
Solution
b) Solve the problem by using the simplex method. Use the slack variables as
initial basic variables and employ Bland’s rule to determine the entering and
leaving variables. Give the solution for the optimizer x∗ and the value of the
objective function of the optimizer f (x∗ ). [4 pt]
Note: State clearly what your basic and non-basic variables are in every step.
Solution
1
cB = (0, 0)T , cN = (−9, −7)
b̂ = B −1 b = (6, 21)T
ŷ T = cTB B −1 = (0, 0)
ĉr = cTN − y T N = (−9, −7)T
The basis is not optimal. The entering variable is x1 (smallest index i in
ĉr,i < 0). On the other hand,
−1 1 0 −1
Â1 = B A1 = ,
0 1 3
xB = (x3 , x1 )T , xN = (x4 , x2 )T
1 −1 0 3 −1 1 3 1
B= , N= , B =
0 3 1 2 3 0 1
cB = (0, −9)T , cN = (0, −7)T
b̂ = B −1 b = (13, 7)T
ŷ T = cTB B −1 = (0, −3)
ĉr = cTN − y T N = (3, −1)T
The basis is not optimal since not all components of ĉr are positive. The
negative component corresponds to x2 which is the entering variable. On the
other hand,
−1 1 3 1 3 1 11
Â2 = B A2 = =
3 0 1 2 3 2
and
b̂i 39 21
min , A2,i ≥ 0 = min , ,
i Â2,i i 11 2
and, thus, the leaving variable is x3 .
3.) Third iteration
xB = (x1 , x2 )T , xN = (x4 , x3 )T
−1 3 1 0 −1 1 −2 3
B= , N= , B =
3 2 0 1 11 3 1
cB = (−9, 7)T , cN = (0, 0)T
1
ŷ T = cTB B −1 = (−3, −34)
11
2
1
ĉr = cTN − y T N =
(3, 34)T
11
The basis is now optimal since ĉr > 0. Stop here. The optimizer is
1
x∗ = b̂ = B −1 b = (51, 39)T ,
11
and the value of the objective function (with the primal problem written in
standard form) is −732/11.
c) Write down the dual of problem in (1) and give the solution for the dual
problem (the optimizer y ∗ and also value of the dual objective). [1.5 pt]
Solution
The explicit expression of the dual problem is not unique (in the same way
that the primal problem can be also expressed in different forms). For exam-
ple, if we directly calculate the dual from the primal expressed as in (1) we
obtain (see p.14 on the slides for Chapter 6. Duality and Sensitivity for an
identical example)
If, on the other hand, we decide to write first problem (1) in canonical form
we have that the primal and the dual problems can be equivalently expressed
as (see p.12 on the slides for Chapter 6. Duality and Sensitivity)
min g(x) ≡ −f (x) = −9x1 − 7x2 max v(y) ≡ −w(y) = −6y1 − 21y2
s.t. x1 − 3x2 ≥ −6 s.t. y1 − 3y2 ≤ −9
⇐⇒ (4)
− 3x1 − 2x2 ≥ −21 − 3y1 − 2y2 ≤ −7
x1 , x2 ≥ 0, y1 , y2 ≥ 0.
3
and y2 ≤ 0 (coming for the positivity of the primal—slack—variables x3 and
x4 ) are more restrictive. Note also that the dual in (5) is, of course, equivalent
to the dual in (4) simply by doing the change of variables y1 = −y1 and
y2 = −y2 .
From all the previous equivalent formulations, we have to use the last one
(with the primal expressed in standard form), as the theoretical results of
the weak and strong duality theory are based on the primal problem being
written in such a way (see p.21 on the slides for Chapter 6. Duality and
Sensitivity and pp. 179–180 on the book by Griva, Nash and Sofer (2009)).
Then, the optimizer is just the simplex multiplier from the last iteration i.e.
ŷ = y ∗ = (−3, −34)T /11 and the objective value of the dual function, written
in standard form as in (5), is the same −732/11 as for the primal problem,
by the strong duality theorem.
Solution
where x = (W, s)T and F (x) = (f1 (x), f2 (x), f3 (x))T . The residuals fi (x) are
defined as:
fi (x) = e−W ti (1 − s) + s − ni .
4
The partial derivatives with respect to the parameters are:
∂fi
= −ti e−W ti (1 − s),
∂W
∂fi
= −e−W ti + 1.
∂s
By using the Jacobian
∂f1 ∂f2 ∂f3
∂W ∂W ∂W
∇F (x) = ∂f ∂f2 ∂f3 ,
1
∂s ∂s ∂s
the derivative of the objective function can be expressed in a more compact
way as:
∇f (x) = ∇F (x)F (x)
The resulting Hessian matrix for the problem is, then:
3
X
∇2 f (x) = ∇F (x)∇F (x)T + fi (x)∇2 fi (x).
i=1
b) Perform one iteration with the Gauss-Newton method with a step length
of one. Start with the initial guess W = 1.46, s = 0.09 (use 4 digits in the
calculation). Check if the Armijo condition for a relaxation parameter µ = 0.1
is fulfilled. Give the value of the objective function. [5 pt]
Solution
We calculate all the components first. The vector of the residuals is then:
5
The approximated Hessian, according to the Gauss-Newton algorithm, is
T 0.05771 −0.2073
∇2 f x(0) ≈ H x(0) = ∇F x(0) ∇F x(0) =
.
−0.2073 2.163
Now, having all necessary components, we can calculate a quasi-Newton
search direction:
−1
p = −H x(0) ∇f x(0)
h T i−1
= − ∇F x(0) ∇F x(0) ∇F x(0) F x(0) = (−0.2193, 0.01173)T ,
Using the Armijo condition means to make relaxed linear prediction for the
change of the function value:
And we see the condition is fulfilled, and we have a sufficient decrease in the
step.
a) Write the algorithm for the steepest-descent method without line search. [0.5 pt]
b) Solve the problem using the algorithm in (3a).
i. Perform the first five iterations of the method starting from x0 = (1, 1)T .
Give a 6-digit approximation to the value of the norm of the gradient at
each iteration. Write the results in a table like the one below. [1.5 pt]
k x p k∇f (x) k
0
1
..
.
ii. What can you say about the convergence rate of the iterates? (do not
make numbers here, just an intuitive interpretation). [1 pt]
iii. Plot the iterates on the x1 x2 plane, together with a sketch of some contour
lines of f (x). [1 pt]
c) Perform again the first iterate as in (3b), but using an exact line search.
Discuss the optimum value obtained for α. [1 pt]
6
Solution
a) The algorithm for the steepest descent method without line search is simply
Input Select initial value for the variable x0 ; k = 0
repeat
Step 1 If xk is optimal, stop
Step 2 Determine a search direction
pk = −∇f (xk )
7
c) We perform now the first iterate but using a exact line search. To obtain the
value of the optimum step length αopt , we use the exact formula
∇f (x0 )T p0
α=− , (6)
pT0 Qp
1 T
where Q is the Hessian matrix of the quadratic function 2 x Qx − cT x. In
this case,
1/2 0
2
Q ≡ ∇ f (x) = ,
0 1/4
so the computation is straightforward and α = 20 9 . Should we take this
1 4
whole step, the new point would be at − 9 , 9 (red line on the plot), with
a gradient norm equal to 0.1242, so we will be improving the performance
of the algorithm. However, should we have used a backtracking strategy, we
would have started with an initial value α = 1, which corresponds to the first
step already calculated in (3b).
Note: In general, when the objective function is not quadratic, (6) cannot
be used, and we should solve the following unidimensional unconstrained op-
timization problem to obtain α
8
4. Consider an inequality-constrained problem with two constraints
min f (x)
s.t. g1 (x) = x21 + x22 + x23 − 3 ≥ 0
g2 (x) = 2x1 − 4x2 + x23 + 1 ≥ 0.
a) State the necessary optimality conditions for this kind of problems, without
using the explicit expressions for g1 and g2 . [0.5 pt]
T T
Consider now the points xa = (1, 1, 1) and xb = (3, 2, −1) .
b) Which of them (if any) is feasible? [0.5 pt]
c) Which of them (if any) is regular? [1 pt]
d) Compute a null-space matrix Z (x) (as stated in 4a) for each point. [1.5 pt]
e) What is the range of admissible values for the Lagrange multipliers if we want
the necessary conditions to be fulfilled at xa ? And at xb ? Can degeneracy
happen in any of these cases? [1.5 pt]
Solution
c) To check for regularity, we need the Jacobian matrix only of the active con-
straints. The general expression of the Jacobian is
T 2x1 2x2 2x3
∇g (x) = .
2 −4 2x3
9
At xa both constraints are active, so the Jacobian evaluated at xa is
T 2 2 2
∇g (xa ) = .
2 −4 2
The columns of ∇g (xa ) (or the rows of ∇g (xa )T ) are linearly independent so
xa is a regular point.
Regarding xb , only the second constraint is active, so we need only check the
corresponding column of the Jacobian, i.e.
∇g (xb )T = 2 −4 −2 ,
10
Note: Solving of the problem by simply expressing one variable in terms
of the other from the constraint, plugging it into the objective function and
minimizing this (now unidimensional) function will NOT be evaluated.
b) Either with a logarithmic barrier function or with a quadratic penalty func-
tion. Motivate your choice.
i. What is x (µ) (or x (ρ), depending on your choice)? [0.5 pt]
ii. What is λ (µ) (or λ (ρ), depending on your choice)? [0.5 pt]
iii. What is x∗ ? [0.5 pt]
iv. What is λ∗ ? [0.5 pt]
c) Compute the Hessian matrix B of the logarithmic barrier function for µ =
10−4 (or of the quadratic penalty function for ρ = 104 , depending on your
choice). What is the condition number of B? What is B −1 ? [2 pt]
Help: For a 2 × 2 nonsingular symmetric matrix Q the condition number is
λmax
cond (Q) = ,
λmin
where λmin , λmax are, respectively, the smallest and largest (in moduli) eigenvalues
of Q.
Solution
This is a linear equality constrained problem, with one single constraint. The
gradient and the Hessian of the objective function are
2x1 + x2 2 2 1
∇f (x) = , ∇ f (x) = .
x1 + 2x2 − 2 1 2
11
i. x∗ = (0, 2)T .
ii. λ∗ = 2.
iii. Yes, x∗ is a strict local minimizer, since the reduced Hessian at the solu-
tion is
T 2 ∗
2 1 −1
Z ∇ f (x ) Z = −1 1 = 2 > 0,
1 2 1
satisfying therefore the second order sufficient condition.
iv. x∗ is also a global minimizer since the function is convex and the feasible
set is convex.
b) We solve now the problem using a quadratic penalty function, because the
problem has only equality constraints.
m
gi (x)2 ,
X
1
ψ (x) = 2
i=1
∂πρ (x)
≡ 2x1 + x2 + ρ (x1 + x2 − 2) = 0
∂x1
∂πρ (x)
≡ x1 + 2x2 − 2 + ρ (x1 + x2 − 2) = 0.
∂x2
Subtracting both equations, we get that x1 = x2 − 2, and plugging it back
into any of the constraints we get that
i. The solution of the unconstrained problem, expressed in terms of ρ is
−2 4ρ + 4
x (ρ) = (x1 (ρ) , x2 (ρ)) = , .
2ρ + 3 2ρ + 3
12
c) We need to compute now the Hessian matrix of the quadratic penalty function
πρ (x), obtaining
2 2+ρ 1+ρ 10002 10001
B ≡ ∇x πρ (x) = −→ .
1 + ρ 2 + ρ ρ=104 10001 10002
The eigenvalues of this matrix are σ = {1, 2ρ + 3} = {1, 20003} , so the con-
dition number is
cond (B) = 2ρ + 3 = 20003.
Finally, the inverse of B is
!
ρ+2 ρ+1
− 2ρ+3
−1 2ρ+3 0.500025 −0.499975
B = ρ+1 ρ+2 −→ ,
− 2ρ+3 2ρ+3 ρ=104 −0.499975 0.500025
13
Top
Front
x
3
x
2
x
1
Bottom
a) Write the optimization problem. State clearly all the constraints (if any). [0.5 pt]
b) By geometric reasoning, the problem is guaranteed to have a solution x∗ =
(x∗1 , x∗2 , x∗3 ) such that x∗1 , x∗2 , x∗3 are strictly positive. Does this fact change
your formulation of the optimization problem? Motivate. [0.5 pt]
c) Through a suitable change of variables, it is possible to transform the problem
in (6b) into a linear equality constrained problem. Formulate one such change
of variables and the corresponding optimization problem. State again clearly
all the constraints p(if any), making use of the assumption in (6b).
Hint: minimizing f (x) for f (x) > 0 is equivalent to minimizing f (x). [1 pt]
d) State the first-order necessary conditions for the problem in (6c). [1 pt]
e) Find x∗ = (x∗1 , x∗2 , x∗3 ). [3 pt]
f) Verify the second-order sufficient condition for x∗ . [1 pt]
Solution
max f (x) = x1 x2 x3
s.t. 4x1 x2 + 3x1 x3 + 2x2 x3 = 72 (7)
x1 , x2 , x3 ≥ 0,
because we need to have double weight on the “Front” face (which has an
area equal to x1 x3 ) and on the “Top” and “Bottom” faces, which have an
area of x1 x2 .
b) As we know that the constraints x1 , x2 , x3 ≥ 0 will not be active at the
solution, we can remove, and we can thus express problem (7) as
max f (x) = x1 x2 x3
s.t. 4x1 x2 + 3x1 x3 + 2x2 x3 = 72. (8)
14
Problem (8) can be equivalently expressed as a minimization problem
is
4x2 + 3x3
∇g (x) = 4x1 + 2x3 . (12)
3x1 + 2x2
However, we can write problem (9) in a different (and simpler) way, making
use of the following changes of variables
y 1 = x1 x2 , y 2 = x1 x3 , y 3 = x2 x3 . (13)
Of course, the point y ∗ = (y1∗ , y2∗ , y3∗ ) corresponding to the original solution
x∗ = (x∗1 , x∗2 , x∗3 ) will be also guaranteed to be strictly positive at the solution.
Hence, problem (9) now reads
√
min f (y) = − y1 y2 y3
s.t. 4y1 + 3y2 + 2y3 − 72 = 0.
The gradient and the Hessian of the objective function in problem (14) are
y2 y3 0 y3 y2
∇f (y) = − y1 y3 , ∇2 f (y) = − y3 0 y1 .
y1 y2 y2 y1 0
15
so the constraint matrix is
A= 4 3 2 ,
and a choice for Z ′ (computed through the variable reduction method) would
be
−3/4 −1/2
Z′ = 1 0 .
0 1
which has to be solved, together with the feasibility requirement (15) in order
to obtain the stationary points of problem (14).
e) If we solve each equation on the gradient condition (16) for λ′ we obtain
y2 y3 ′ y1 y3 ′ y1 y2
λ′ = − , λ =− , λ =− . (17)
4 3 2
If we compare the right-hand sides of the first and second relationships in
(17), this yields
y2 y3 y1 y3
= ⇒ y2 = 43 y1 (18)
4 3
(here we can divide by y3 as the y1 , y2 , y3 are guaranteed to be strictly pos-
itive). Similarly, if we compare the right-hand sides of the first and third
relationships in (17), we get
y2 y3 y1 y2
= ⇒ y3 = 2y1 . (19)
4 2
We can plug now (18) and (19) into the constraint (15), and solve for y1
Replacing y1 into (17), (18) and (19) this yields y2 = 8 and y3 = 12, obtaining
the solution
y ∗ = (6, 8, 12) , λ′∗ = −24.
The solution in terms of the original variables is obtained from the original
relationships (13)
x1 x2 = 6, x1 x3 = 8, x2 x3 = 12,
dividing the second and the third equations (to solve for x2 in terms of x1 →
x2 = 32 x1 ), and the first and the third equations (to solve for x3 in terms of
x1 → x3 = 2x1 ), and plugging the result into the original constraint (11)
√
4x1 32 x1 + 3x1 (2x1 ) + 2 32 x1 (2x1 ) − 72 = 0 ⇒ x1 = 4 = 2,
16
and, from here, x2 = 3 and x3 = 4, so the solution is
x∗ = (2, 3, 4) .
We have to check now that this point is actually a stationary point of the
original problem (9). To do so, we check the first order optimality condition
for problem (9), and we observe that the gradient ∇f (x∗ ) and the Jacobian
∇g (x∗ ) are parallel at the solution
12 24
∇f (x∗ ) = ∇g (x∗ ) λ ⇒ − 8 = 16 λ,
6 12
Z T ∇2 f (x∗ ) Z
Good Luck!
Javier & Markus
17