ACM113 HW1 SOLUTION

XINWEI YU

Note.The numbering orresponds to the problem number in [1.

Problem 0.1. (2.1). Compute the gradient rf (x) and Hessian r2 f (x) of the
Rosenbro k fun tion
f (x) = 100(x2 x21 )2 + (1 x1 )2 :
Show that x = (1; 1)T is the only lo al minimizer of this fun tion, and that the
Hessian matrix at that point is positive denite.
Solution. We have
 2 
rf (x) = 400( x2 x1 )x1 2(1 x1 )
200(x2 x21 )
and  
r2 f (x) = x2 x21 ) + 800x21 + 2 400x1
400(
400x1 200
Solving rf (x) = 0, we get x = (1; 1) as the only solution. So f (x) has only
T
one irti al points. Sin e f (1; 1) = 0 and obviously f  0, we know that it's a lo al
minimizer ( global also ).
At (1; 1)T , we have  
r2 f =
802
400
400
200
It's obvious that both eigenvalues are positive, thus the Hessian SPD.
Problem 0.2. (2.3). Let  be a given n-ve tor, and A be a given n  n symmetri
matrix. Compute the gradient and Hessian of f1 (x) = aT x and f2 (x) = xT Ax.
Solution. We have

rf1 (x) a; r2 f1 (x) = 0

=
rf2 (x) 2
= 2Ax; r f2 (x) = 2A

Problem 0.3. ( 2.7) Suppose that f (x) is a onvex fun tion. Show that the set of
global minimizers of f is a onvex set.
Solution. Let S denote the set of the global minimizers. Obviously we have

f (x) = min f
for any x 2 S .
Take any x; y 2 S , we just need to show that x + (1 )y 2 S for any  2 (0; 1).
Now by onvexity,
min f  f (x + (1 )y)  f (x) + (1 )f (y) = min f
we know that equality must hold, whi h means what we want.
Date : 11th April 2003.

1
2 XINWEI YU

Problem 0.4. (2.10) Show that the symmetri rank-one update (2.17) and the
BFGS update (2.18) are s ale-invariant if the initial Hessian approximations B0
are hosen appropriately. That is, using the notation of the previous exer ise, show
that if these methods are applied to f (x) starting from x0 = Sz0 + s with initial
Hessian B0 , and to f~(z ) starting from z0 with initial Hessian S T B0 S , then all
iterates are related by xk = Szk + s. ( Assume for simpli ity that the methods take
unit step lengths. )
Solution. We denote all quantities with respe t to f~ with a~. Note that we have
rf ~= ST rf
SR1: We use indu tion. By onstru tion we have B~0 = S T B0 S . Thus by
p0 = B0 1 rf and p~0 = B~0 1 rf~
we easily have
p0 = S p~0
thus
s0 = S s~0
Now suppose the above relations hold for k . Then by simply omputing B~k+1
we would have
B~k+1 = S T Bk+1 S
whi h leads to
pk+1 = S p~k+1
whi h is just
sk+1 = S s~k+1
sin e unit step length is taken. Now sin e
X
xk = x0 + si =Sz + s
BFGS: similar.
Problem 0.5. (2.15) Consider the sequen e fxk g dened by
 1
( 4 )2
k
xk = ; k even:
xk 1 )=k; k odd:
(

Is this sequen e Q-superlinearly onvergent? Q-quadrati ally onvergent? R-

quadrati ally onvergent?
Solution. First we see that
xk ! 0
Superlinear? Yes.
kxk+1 0k  max( 1 ; k
( 1 )2k+2 2k ) ! 0
kxk 0k k 4
Quadrati ally? No.
kxk+1 0k = k
42k for k even
kxk 0k2
and the RHS is obviously unbounded.
R-Quadrati ally? Yes. Take

k = ( )2
1 k
2
 ACM113 HW1 SOLUTION 3

then k onverges to 0 quadrati ally. Now we have

k+1 = xk > xk+1 for all k even

and
p
k = xk > xk

So k is a dominating sequen e. So xk onverges R-quadrati ally.

Problem 0.6. ( MATLAB programming ). For the fun tion

f (x) = hx; Qxi + hx; i

program a steepest des ent algorithm using an exa t line sear h for x 2 R2 , with
Q SPD. Constru t three examples where the ondition number of Q is

; ;
1 10 100

respe tively. The ondition number (Q) is the ratio of the largest and the smallest
eignevalues of Q for Q SPD. Estimate the onvergen e onstant for ea h ase. Illus-
trate the onvergen e path by superimposing the steps on the obje tive ontours,
i.e., the ontours of the obe tive fun tion.

Solution. The main part of the algorithm is minimizing

f (xk trf (xk ))

where rf (xk ) = 2Qxk + . This an be rewritten as

t2 hrf; Qrf ) thrf; rf i + f (xk )

whi h yields

t=
hrf; rf i
2 hrf; Qrf i
In numeri al omputation, we take
 
1 0
2 Q= 0 k

with k = 1; 10; 100, and denote the Q's with subs ript 1; 2; 3 respe tively.
Following are the plots of the onvergen e path and rate:
4 XINWEI YU

k=1
5

5
5 4 3 2 1 0 1 2 3 4 5

k=10
5
0.4767

3 0.4767

0.4767
1

0
0.4767

0.4767
2

3
0.4767

5 0.4767
5 4 3 2 1 0 1 2 3 4 5 0 5 10 15 20 25 30
 ACM113 HW1 SOLUTION 5

k=100

0.9587

0.9587

0.9587

0.9587

1
0.9587

2
0.9587

3
0.9587
2 1 0 1 2 3 4 0 50 100 150 200 250 300 350 400 450 500

Now it's pretty lear how ondition number ae ts onvergen e rate.
Referen es
[1 Jorge No edal, Stephen J. Wright, Numeri al Optimization , Springer, 1999.