Академический Документы
Профессиональный Документы
Культура Документы
xk+1 = xk k F k
xk , xk+1 F(x) F
= values of the variables in the k and k+1 iteration. = objective function to be minimized (or maximized) = gradients of the objec tive function, constituting the direction of travel = the size of the step in the direction of travel
Refer to Section 3.5 for Algorithm and Stopping Criteria Advantage: Simple Disadvantage: Seldom converges reliably.
Georgia Institute of Technology Systems Realization Laboratory 79
Start again with Taylor expansion: y(x) = y(xk) + y(xk)(x-xk) + 0.5 ( -xk)H(xk) (x-xk) x
T
See Note that H is the Hess ian containing the second order derivatives . Sec. 1.4.
xk+1 = xk -
y(xk) H(xk)
New tons method for finding an extreme point is xk+1 = xk - H-1(xk) y(xk)
Dont confuse H-1 with .
Optimization in Engineering Design
Like the Steepest Descent Method, Newtons searches in the negative gradient direction.
Georgia Institute of Technology Systems Realization Laboratory 80
Good properties (fast convergence) if started near solution. However, needs modifications if started far away from solution. Also, (inverse) Hessian is expensive to calculate. To overcome this, several modifications are often made. One of them is to add a search parameter in from of the Hessian. (similar to steepest descent). This is often referred to as the modified Newton's method.
Other modification focus on enhancing the properties of the second and first order gradient combination.
Quasi-Newton methods build up curvature information by observing the behavior of the objective functions and its first order gradient. This info is used to generate an approximation of the Hessian.
Georgia Institute of Technology Systems Realization Laboratory 81
Basic Principle
Definition: Given a symmetric matrix Q, two vectors d1 and d2 are said to be Q orthogonal or Q conjugate (with respect to Q) if d1TQd2 = 0. Note that orthogonal vectors (d1Td2 = 0)are a special case of conjugate vectors So, since the vectors di are independent, the solution to the nxn quadratic problem can be rewritten as x* = 0d0 + ... + n-1 dn-1 Multiplying by Q and by taking the scalar product with di, you can express in terms of d, Q, and either x* or b Note that A is used instead of Q in your textbook
Optimization in Engineering Design
d k g k id i
i 0
k 1
or
d k 1 g k 1 k d k
Three advantages:
1) Gradient is always nonzero and linearly independent of all previous direction vectors. 2) Simple formula to determine the new direction. Only slightly more complicated than steepest descent. 3) Process makes good progress because it is based on gradients.
Optimization in Engineering Design
2 - Calculate the new conjugate gradient direction dk+1 , according to: dk+1 = - gk+1 + k dk where gk+1 TQd k k= d kTQd k
This is slightly different than your current textbook
Difference in methods:
find k through line search different formulas for calculating k than the pure Conjugate Gradient algorithm
Georgia Institute of Technology Systems Realization Laboratory 86
( g k 1 )T ( g k 1 ) (g k ) (g k )
T
( g k 1 )T ( g k 1 g k ) ( g k )T ( g k )
( g k 1 ) ( g k 1 ) ( g k )T ( g k )
Attractive are the simple formulae for updating the direction vector. Method is slightly more complicated than steepest descent, but converges faster.
See em in action!
For animations of each of ALL preceding search techniques, check out: http://www.esm.vt.edu/~zgurdal/COURSES/4084/4084-Docs/Animation.html
Georgia Institute of Technology Systems Realization Laboratory 89