2 views

Uploaded by Ryan Michael

optimization

- newasdsa
- Gradient Descent Tutorial
- 311_spring_2017_final_exam_v1%281%29 (1)
- me syllabus 2013 regulation
- Gander
- Algorithmic Approach to Pushback Design Based on Stochastic Programming Method Application and Comparisons
- Quiz Feedback _ Coursera
- An Emergency Service Facility Location Problem With Fuzzy Objective and Constraint
- Least Squares Optimization With L1-Norm Regularization
- 171901(12.12.15)
- Midterm 1 Review Solution
- Inventry Tech
- IE 416 Project1.1
- A further review of ESO type methods for topology optimization.pdf
- Comparison of Cost Estimation Methods Using Hybrid Artificial Intelligence on Schematic Design Stage, RANFIS and CBR-GA
- Paper - Ajuste de Una Onda Sinusoidal
- dgms.pdf
- Planning and Scheduling
- 5.4 Security Constrained Optimal Power Flow
- Linear Programming

You are on page 1of 66

Optimization models

Problem specification

Suppose we have a cost function (or objective function)

minimize this function

equality:

nonequality:

a minimum of f(x)

Types of minima

f(x)

strong

local

minimum

weak

local

minimum

strong

global

minimum

feasible region

strong

local

minimum

point

such minima often occur in real applications

Assume we can start close to the global minimum

Approximation methods

1. Polynomial interpolation

2. Newton method

Inexact Line Search (Fletcher)

1D function

As an example consider the function

(assume we do not know the actual function expression from now on)

Search methods

Start with the interval (bracket) [xL, xU] such that the

minimum x* lies inside.

Evaluate f(x) at two point inside the bracket.

Reduce the bracket.

Repeat the process.

essential.

Search methods

xU

xL

xL

xL

xU

xU

Dichotomous

Fibonacci:

1 1 2 3 5 8

Ik+5 Ik+4 Ik+3 Ik+2 Ik+1 Ik

1 2 3

xL

xL

8

xU

xU Golden-Section Search

divides intervals by

K = 1.6180

Polynomial interpolation

Bracket the minimum.

Fit a quadratic or cubic polynomial which

interpolates f(x) at some points in the interval.

Jump to the (easily obtained) minimum of the

polynomial.

Throw away the worst point and repeat the

process.

Polynomial interpolation

Other methods to interpolate?

2 points and one gradient

Cubic interpolation

Newton method

Fit a quadratic approximation to f(x) using both gradient and

curvature information at x.

approximation.

Update x.

Newton method

quadratic convergence (decimal accuracy doubles

at every iteration)

Newton method

Global convergence of Newtons method is poor.

Often fails if the starting point is too far from the minimum.

which reduces the step length until function decrease is

assured

How big N can be?

problem sizes can vary from a handful of parameters to

many thousands

function surfaces can be visualized.

An Optimization Algorithm

Start at x0, k = 0.

2. Compute a step length k, such that f(xk + k pk ) < f(xk)

k = k+1

3. Update xk+1 = xk + k pk

4. Check for convergence (stopping criteria)

e.g. df/dx = 0 or

Rates of Convergence

x* minimum

p order of convergence

convergence ratio

Linear conv.:

Superlinear conv.:

Quadratic conv.:

p=1, <1

p=1, =0 or p=>2

p=2

Taylor expansion

A function may be approximated locally by its Taylor series

expansion about a point x*

is the vector

Quadratic functions

Second order approximation of any function by the Taylor

expansion is a quadratic function.

eigenvectors

or is unchanging according to whether i is

positive, negative or zero

Case 1: both eigenvalues positive

with

minimum

positive

definite

Case 2: eigenvalues have different sign

with

saddle point

indefinite

Case 3: one eigenvalues is zero

with

parabolic cylinder

positive

semidefinite

Assume that H is positive definite

Steepest descent

by a series of 1D line-minimizations:

the gradient

For quadratic forms there is a closed form solution:

Prove it!

Steepest descent

lines.

After each line minimization the new gradient is always

orthogonal to the previous step direction (true of any line

minimization).

Consequently, the iterates tend to zig-zag down the

valley in a very inefficient manner

Conjugate gradient

directions with respect to the Hessian H:

independent.

Prove it!

pk-1,

, and

Conjugate gradient

most N conjugate descent steps.

Minimum is reached in exactly 2 steps.

Powells Algorithm

derivatives

Conjugate directions are generated through a series of

line searches

searches

Rosenbrocks function

Minimum at [1, 1]

Steepest descent

The 1D line minimization must be performed using one

of the earlier methods (usually cubic polynomial

interpolation)

The algorithm crawls down the valley

Conjugate gradient

Again, an explicit line minimization must be used at every

step

Far superior to steepest descent

Newton method

Expand f(x) by its Taylor series about the point xk

Newton method

For a minimum we require that

with solution

, and so

The method has quadratic convergence (as in the 1D case).

The solution

is guaranteed to be a downhill direction.

Rather than jump straight to the minimum, it is better to perform a line

minimization which ensures global convergence

to the 98 for conjugate gradients.

However, the method requires computing the Hessian

matrix at each iteration this is not always feasible

Quasi-Newton methods

dense then it may be infeasible/inconvenient to compute

it directly.

Quasi-Newton methods avoid this problem by keeping a

rolling estimate of H(x), updated at each iteration using

new gradient information.

Common schemes are due to Broyden, Goldfarb,

Fletcher and Shanno (BFGS), and also Davidson,

Fletcher and Powell (DFP).

The idea is based on the fact that for quadratic functions

holds

and by accumulating gks and xks we can calculate H.

BFGS example

18 for the full-Newton method

function f(x) to be the sum of a large number of

squared residuals

parameters x then the minimization of f(x) is a

non-linear least squares problem.

as

Consider

Hence

Gauss-Newton

approximation

residuals ri.

mean = 0.

Hence, explicit computation of the full Hessian can again be avoided.

Gauss-Newton example

residual vector

Gauss-Newton example

line search takes only 11 iterations

Levenberg-Marquardt Algorithm

Combines Gauss-Newton with Steepest Descent

Fast convergence even for very flat functions

Descend direction :

Newton

- Steepest Descent

Gauss-Newton:

Comparison

CG

Quasi-Newton

Newton

Gauss-Newton

Derivative-free optimization

Downhill

simplex

method

Downhill Simplex

Comparison

CG

Quasi-Newton

Newton

Downhill Simplex

Constrained Optimization

Subject to:

Equality constraints:

Nonequality constraints:

Constraints define a feasible region, which is nonempty.

Equality constraints

Minimize f(x) subject to:

for

linear combination of the gradients of ai(x) with

Lagrange multipliers as the coefficients.

f3 > f2 > f1

f 3 > f 2 > f1

x* is a minimizer, *<0

is not a minimizer

x* is a minimizer, *>0

f3 > f2 > f1

x* is not a minimizer

3D Example

3D Example

f(x) = 3

Gradients of constraints and objective function are linearly independent.

3D Example

f(x) = 1

Gradients of constraints and objective function are linearly dependent.

Inequality constraints

Minimize f(x) subject to:

for

linear combination of the gradients of cj(x), which are

active ( cj(x) = 0 )

and Lagrange multipliers must be positive,

f3 > f2 > f1

f3 > f 2 > f1

x* is not a minimizer, <0

No active constraints at

x*,

f3 > f2 > f 1

x* is a minimizer, >0

Lagrangien

We can introduce the function (Lagrangien)

satisfied).

These are Karush-Kuhn-Tucker conditions

Like in the unconstrained case, it is important to study

quadratic functions. Why?

Because general nonlinear problems are solved as a

sequence of minimizations of their quadratic

approximations.

QP with constraints

Minimize

subject to linear constraints.

Minimize

Subject to:

Ass.: A is p N and has full row rank (p<N)

Convert to unconstrained problem by variable

elimination:

Z is the null space of A

A+ is the pseudo-inverse.

Minimize

This quadratic unconstrained problem can be solved, e.g.,

by Newton method.

Minimize

Subject to:

First we check if the unconstrained minimizer

is feasible.

If yes we are done.

If not we know that the minimizer must be on the

boundary and we proceed with an active-set method.

xk is the current feasible point

Next iterate is given by

Active-set method

To remain active

thus

The objective function at xk+d becomes

where

subject to:

or

Active-set method

and

If YES we are done.

If NO we remove the constraint from the active set

with the most

negative

and solve the QP sub-problem again but this time with

less active constraints.

We can move to

but some inactive constraints

may be violated on the way.

In this case, we move by

till the first inactive constraint

becomes active, update

, and solve the QP sub-problem again

but this time with more active constraints.

Minimize

subject to:

f(x)

nonlinear.

1.

2.

3.

4.

5.

For a given

approximate Lagrangien by

Taylor series QP problem

Solve QP descent direction

Perform line search in the direction

Repeat from Step 1.

Lagrangien

At the kth iterate:

and we want to compute a set of increments:

First order approximation of

and constraints:

SQP example

Minimize

subject to:

with constraints.

Two forms:

1.

Minimize

subject to:

2.

Minimize

subject to:

A is p N and has

full row rank (p<N)

Prove it!

If the LP minimizer exists it must be one of the vertices of

the feasible region.

A fast method that considers vertices is the Simplex

method.

- newasdsaUploaded bynikos
- Gradient Descent TutorialUploaded byAntonio
- 311_spring_2017_final_exam_v1%281%29 (1)Uploaded byEric Su
- me syllabus 2013 regulationUploaded byapi-282104564
- GanderUploaded byMuhammad Saepudin
- Algorithmic Approach to Pushback Design Based on Stochastic Programming Method Application and ComparisonsUploaded byGaluizu001
- Quiz Feedback _ CourseraUploaded byanon_74471635
- An Emergency Service Facility Location Problem With Fuzzy Objective and ConstraintUploaded byHasan Safadi
- Least Squares Optimization With L1-Norm RegularizationUploaded byqsmeds
- 171901(12.12.15)Uploaded byVIPUL
- Midterm 1 Review SolutionUploaded bytehkronos
- Inventry TechUploaded byDinesh Tomar
- IE 416 Project1.1Uploaded byPeyman Panahi
- A further review of ESO type methods for topology optimization.pdfUploaded byAlexandra Rampinelli
- Comparison of Cost Estimation Methods Using Hybrid Artificial Intelligence on Schematic Design Stage, RANFIS and CBR-GAUploaded bybidangciptakarya
- Paper - Ajuste de Una Onda SinusoidalUploaded byVicente Estopier
- dgms.pdfUploaded byEthel Mendoza Candaganan
- Planning and SchedulingUploaded byjignesh_pandav
- 5.4 Security Constrained Optimal Power FlowUploaded byMohammedSaadaniHassani
- Linear ProgrammingUploaded bysalim
- Notes LagrangeUploaded byLennard Pang
- Modelling Simulation and OptimizationUploaded byCrucifiedBishwarup
- eka1Uploaded byEka Handayani
- LAO-CR-QPUploaded byRohit Kumar
- 01. Optimal Portfolio Under VaR ConstraintUploaded bycon_lua92
- C2_S2_BSSUploaded byRoxana Vlad
- FunctionsUploaded bybatooluzma
- WATER DISTRIBUTION NETWORK DESIGN BY PARTIAL ENUMERATIONUploaded byjoreli
- Lecture 1Uploaded bySanwar Tagra
- Energy Efficient and Distributed Resource Allocation for Wireless Powered OFDMA Multi-cell NetworksUploaded byhendra

- Service sector and women labour participation, HNLUUploaded byRyan Michael
- General IntelligenceUploaded byRabindraMaharana
- ChemistryUploaded byRyan Michael
- Annual Calender 2017 IESUploaded byAdam Jensen
- s5stoichiometryUploaded byRyan Michael
- Mata Vaishnodevi Darshan, Shri Shakti Express, IRCTC Tour PackageUploaded byRyan Michael
- Application of low-cost adsorbents for dye removal – A review.pdfUploaded byVinh_LX_88
- Removal of Heavy Metals From Water (Cu and Pb) Using Household Waste as an Adsorbent _ Open Access JournalsUploaded byRyan Michael
- Weight Gaining FoodsUploaded byRyan Michael
- Weight Gaining FoodsUploaded byRyan Michael
- ChemistryUploaded byRyan Michael
- A Review of Biochar as a Low Cost Adsorbent for Aqueous Heavy Metal RemovalUploaded byRyan Michael
- Geometric Progression - Series and Sums - An IntroductionUploaded byRyan Michael
- Chapter 15Uploaded byanil.gelra5140
- 4 Semester Btech Revised Aug2004Uploaded byRyan Michael
- KK PPTUploaded byRyan Michael
- MANTHAN Current Affairs Supplement – March, 2017Uploaded byRyan Michael
- Test 1 PolityUploaded byRyan Michael
- Determination of Conductivity Exp3_pdfUploaded bySusheel Talreja
- TEST 1 CSAT QUESTION.pdfUploaded byRyan Michael
- 05011120124Uploaded byRyan Michael
- Matlab File CompleteUploaded byRyan Michael
- Branch BoundUploaded byRyan Michael
- What Are BiosensorsUploaded byRyan Michael
- Undergraduate Heat Transfer Experiment Measurement of Thermal Conductivity of Liquids and GasesUploaded byRyan Michael
- 5_6332091823753789455Uploaded byRyan Michael
- Write Short Notes Following Agricultural Schemes Initiated by PM Modi's GovernmentUploaded byRyan Michael
- Matlab ProjectUploaded byRyan Michael
- June 2017 CA EnglishUploaded byUmesh Tiwari
- Budget HighLights 2017-18Uploaded bywww.TdsTaxIndia.com

- OR_PS3_Duality and Sensitivity AnalysisUploaded bySai Sujan
- Sub GradientsUploaded byAustin Zhang
- Ant Colony OptimizationUploaded byKaveri Somasundaram
- MathUploaded byPranay Goswami
- algo1Uploaded byJesus Alberto Dominguez Coronel
- Flow NetworksUploaded bySören Wellhöfer
- Article - On Multi-Objective Optimization Heuristics for NSPUploaded byDavid
- Global Optimization Toolbox R2013a (1)Uploaded bymohsindalvi87
- 317586288-Cse-Vi-Operations-Research-10cs661-Assignment.pdfUploaded byshivu_bn5341
- Lecture 3Uploaded bysolitary_monk
- algoritma genetika untuk penjadwalan matakuliahUploaded byChoiryaldi Setya Pratama
- MS12 08Uploaded byKirul Nizam Aziz
- IndexUploaded bycassindrome
- 2.2. ORUploaded byDeniz Demirci
- Duality (Optimization)Uploaded byAndrea Spencer
- HW01Uploaded byErkan Ak
- Multivariate Optimization LPPUploaded bymithun_sarkar8784
- NR 310206 Optimization TechniquesUploaded bySrinivasa Rao G
- Capitulos 1-5 Del Libro Introduction to Linear OptimizationUploaded byJavier Silva Sandoval
- Application of Particle Swarm Optimization Algorithm for Solving Power Economic Dispatch with Prohibited operating zones and Ramp-rate limit ConstraintsUploaded byijeteeditor
- Solving fuzzy transportation problems based on extension principleUploaded byrisetbacaan
- Optimization Techniques Course Study Outline Tri-2 _2018-2019.pdfUploaded byMarvin Liwanag
- Free-Flight Trajectory Optimization by Mixed Integer programming.pdfUploaded byTudorel Afilipoae
- LPP Duality Home AssignmentUploaded bytanmoyt
- Gams user's guide by McCarlUploaded byBianca Marques
- Assignment ProblemsUploaded byrinkirola7576
- Algorithms VLSI CAD Final f07Uploaded bychilledkarthik
- Two.Phase.Simplex.pdfUploaded byamo1123
- CH1Uploaded byAhmad Zikri
- Lec 13 Newton Raphson MethodUploaded byMuhammad Bilal Junaid