Trust Region Methods

Part I

Andrew R. Conn
arconn@us.ibm.com

Mathematical Sciences
IBM T.J. Watson Research Center

February 2007, Montreal

Overview

1 Trust-Region/Modelling Methods
 Trust-Region/Modelling Methods

What is a trust-region method?

-1

-2

-3

-3 -2 -1 0 1 2 3

Problem: Minimize
 
−10x12 + 10x22 + 4 sin x1 x2 − 2x1 + x14
 Trust-Region/Modelling Methods

What is a trust-region method? (continued)

-1

-2

-3

-3 -2 -1 0 1 2 3

The model and trust region around x0

 Trust-Region/Modelling Methods

What is a trust-region method? (continued)

-1

-2

-3

-3 -2 -1 0 1 2 3

The model and trust region around x1

 Trust-Region/Modelling Methods

What is a trust-region method?

-1

-2

-3

-3 -2 -1 0 1 2 3

The model and trust region around x2

 Trust-Region/Modelling Methods

What is a trust-region method? (continued)

-1

-2

-3

-3 -2 -1 0 1 2 3

The model and trust region around x3 = x2

 Trust-Region/Modelling Methods

What is a trust-region method? (continued)

-1

-2

-3

-3 -2 -1 0 1 2 3

The model and trust region around x4

 Trust-Region/Modelling Methods

What is a trust-region method? (continued)

-1

-2

-3

-3 -2 -1 0 1 2 3

The model and trust region around x5

 Trust-Region/Modelling Methods

What is a trust-region method? (continued)

-1

-2

-3

-3 -2 -1 0 1 2 3

The model and trust region around x6 (x7 = x ∗ )

 Trust-Region/Modelling Methods

What is a trust-region method? (continued)

3 3 3

2 2 2

1 1 1

0 0 0

-1 -1 -1

-2 -2 -2

-3 -3 -3

-3 -2 -1 0 1 2 3 -3 -2 -1 0 1 2 3 -3 -2 -1 0 1 2 3

3 3 3

2 2 2

1 1 1

0 0 0

-1 -1 -1

-2 -2 -2

-3 -3 -3

-3 -2 -1 0 1 2 3 -3 -2 -1 0 1 2 3 -3 -2 -1 0 1 2 3

Six iterations —different initial point

 Trust-Region/Modelling Methods

Basic Algorithm

Initialize: x0 , ∆
Compute Model: mk ( )
Compute Step: Compute sk from

min mk (xk + s)
ksk≤∆

f (xk )−f (xk +sk )

Trust-region Update: ρ = mk (xk )−mk (xk +sk )

If ρ > 0.75 ∆ ← 2.0∆ Accept xk + sk

If 0.25 < ρ < 0.75 ∆←∆ Accept xk + sk

If ρ < 0.25 ∆ ← 0.5∆ Reject xk + sk

 Trust-Region/Modelling Methods

Assumptions to prove convergence

On the problem
Smooth f ∈ C2
Bounded Below f bounded below
Bounded Hessian k∇xx f k bounded above
 Trust-Region/Modelling Methods

Assumptions to prove convergence

On the model (∀k)

Smooth mk ∈ C 2
Interpolates mk (xk ) = f (xk )
Interpolates Gradient ∇x mk (xk ) = ∇x f (xk )
Bounded Hessian
maxx∈Bk k∇xx mk (x)k bounded above where,
Bk = {x ∈ <n | kx − xk kk ≤ ∆k }
 Trust-Region/Modelling Methods

Standard Model in the differentiable case

Typical trust region or line search method builds linear or
quadratic model of the objective function f .
The model has to satisfy Taylor-like error bounds.
First Order
|f (x) − m(x)| ≤ O(∆2 )
|∇f (x) − ∇m(x)| ≤ O(∆)
Second Order
|f (x) − m(x)| ≤ O(∆3 )
|∇f (x) − ∇m(x)| ≤ O(∆2 )
|∇2 f (x) − ∇2 m(x)| ≤ O(∆)

In fact it typically is a first (or second) order Taylor series

approximation.
In derivative based methods constants in O depend only on f
(and its derivatives).
By reducing the trust region or step size one guarantees better
 Trust-Region/Modelling Methods

Standard Model in the differentiable case

Typical trust region or line search method builds linear or

quadratic model of the objective function f .
The model has to satisfy Taylor-like error bounds.
Second Order
|f (x) − m(x)| ≤ O(∆3 )
|∇f (x) − ∇m(x)| ≤ O(∆2 )
|∇2 f (x) − ∇2 m(x)| ≤ O(∆)

In fact it typically is a first (or second) order Taylor series

approximation.
In derivative based methods constants in O depend only on f
(and its derivatives).
By reducing the trust region or step size one guarantees better
accuracy.
14
 Trust-Region/Modelling Methods

Standard Model in the differentiable case

Typical trust region or line search method builds linear or

quadratic model of the objective function f .
The model has to satisfy Taylor-like error bounds.
Second Order
|f (x) − m(x)| ≤ O(∆3 )
|∇f (x) − ∇m(x)| ≤ O(∆2 )
|∇2 f (x) − ∇2 m(x)| ≤ O(∆)

In fact it typically is a first (or second) order Taylor series

approximation.
In derivative based methods constants in O depend only on f
(and its derivatives).
By reducing the trust region or step size one guarantees better
accuracy.
14
 Trust-Region/Modelling Methods

Standard Model in the differentiable case

Typical trust region or line search method builds linear or

quadratic model of the objective function f .
The model has to satisfy Taylor-like error bounds.
Second Order
|f (x) − m(x)| ≤ O(∆3 )
|∇f (x) − ∇m(x)| ≤ O(∆2 )
|∇2 f (x) − ∇2 m(x)| ≤ O(∆)

In fact it typically is a first (or second) order Taylor series

approximation.
In derivative based methods constants in O depend only on f
(and its derivatives).
By reducing the trust region or step size one guarantees better
accuracy.
14