AA222: MDO 53 Friday 6th April, 2012 at 12:06

Chapter 3

Gradient-Based Optimization

3.1 Introduction
In Chapter 2 we described methods to minimize (or at least decrease) a function of one variable.
While problems with one variable do exist in MDO, most problems of interest involve multiple
design variables. In this chapter we consider methods to solve such problems, restricting ourselves
to smooth unconstrained problems. Later, we extend this to constrained problems in Chapter 5,
and remove the smoothness requirement in Chapter 6.
The unconstrained optimization problem can be stated as,
minimize f (x)
(3.1)
with respect to x ∈ Rn

where x is the n-vector x = [x1 , x2 , . . . , xn ]T . The objective function f can be nonlinear, but one
important assumption in this chapter is that f is a sufficiently smooth function of x: in some cases
we will demand that f has continuous first (partial) derviatives, and in others we require f to also
have continuous second (partial) derivatives.
For large numbers of variables n, gradient-based methods are usually the most efficient algo-
rithms. This class of methods uses the gradient of the objective function to determine the most
promising directions along which we should search. And here is where the line search comes in: it
provides a way to search for a better point along a line in n-dimensional space.
All algorithms for unconstrained gradient-based optimization can be described as shown in
Algorithm 13. The outer loop represents the major iterations. The design variables are updated
at each major iteration k using
xk+1 = xk + αk pk (3.2)
| {z }
∆xk
where pk is the search direction for major iteration k, and αk is the accepted step length from
the line search. The line search usually involves multiple iterations that do not count towards the
major iterations. This is an important distinction that needs to be considered when comparing the
computational cost of the various approaches.
The two iterative loops represent the two subproblems in this type of algorithm: 1) The com-
putation a search direction pk , and; 2) The search for an acceptable step size (controlled by αk ).
These two subproblems are illustrated in Fig. 3.1. The various types of gradient-based algorithms
are classified based on the method that is used for computing the search direction (the first sub-
problem). Any line search that satisfies sufficient decrease can be used with a given gradient-based
method.
AA222: MDO 54 Friday 6th April, 2012 at 12:06

Algorithm 4 General algorithm for smooth functions

Input: Initial guess, x0
Output: Optimum, x∗
k←0
while Not converged do
Compute a search direction pk
Find a step length αk , such that f (xk + αk pk ) < f (xk ) (the curvature condition may also be
included)
Update the design variables: xk+1 ← xk + αk pk
k ←k+1
end while

x0

Optimizer

Search
Analysis
direction

x Sensitivity
Line search Analysis

Converged?

x*

Figure 3.1: General gradient-based method and its relation to sensitivity analysis. “Analysis” is
the evaluation of the objective function f , and “Sensitivity Analysis” the evaluation of ∇f

3.2 Gradients and Hessians

Consider a function f (x). The gradient of this function is given by the vector of partial derivatives
with respect to each of the independent variables,

∂f
 
 ∂x1 
 ∂f 
 
∂x2 
 
∇f (x) ≡ g(x) ≡ 
 .  (3.3)
 .. 
 
 ∂f 
∂xn
AA222: MDO 55 Friday 6th April, 2012 at 12:06

In the multivariate case, the gradient vector is perpendicular to the the hyperplane tangent to
the contour surfaces of constant f .
Higher derivatives of multi-variable functions are defined as in the single-variable case, but note
that the number of gradient components increase by a factor of n for each differentiation.
While the gradient of a function of n variables is an n-vector, the “second derivative” of an n-
variable function is defined by n2 partial derivatives (the derivatives of the n first partial derivatives
with respect to the n variables):
∂2f ∂2f
, i 6= j and , i = j. (3.4)
∂xi ∂xj ∂x2i

If the partial derivatives ∂f /∂xi , ∂f /∂xj and ∂ 2 f /∂xi ∂xj are continuous and f is single valued,
∂ 2 f /∂xi ∂xj = ∂ 2 f /∂xj ∂xi . Therefore the second-order partial derivatives can be represented by a
square symmetric matrix called the Hessian matrix,
∂2f ∂2f
 
···
 ∂ 2 x1 ∂x1 ∂xn 
2
∇ f (x) ≡ H(x) ≡ 
 .
.. .. 
(3.5)
.
 

 ∂2f ∂ f2 
··· ,
∂xn ∂x1 ∂ 2 xn
which contains n(n + 1)/2 independent elements.
If f is quadratic, the Hessian of f is constant, and the function can be expressed as
1
f (x) = xT Hx + g T x + α. (3.6)
2

3.3 Optimality Conditions

As in the single-variable case, the optimality conditions can be derived from the Taylor-series
expansion of f about x∗ :
1
f (x∗ + εp) = f (x∗ ) + εpT g(x∗ ) + ε2 pT H(x∗ + εθp)p, (3.7)
2
where θ ∈ (0, 1), ε is a scalar, and p is an n-vector.
For x∗ to be a local minimum, then for any vector p there must be a finite ε such that f (x∗ +εp) ≥
f (x∗ ), i.e. there is a neighborhood in which this condition holds. If this condition is satisfied, then
f (x∗ + εp) − f (x∗ ) ≥ 0 and the first and second order terms in the Taylor-series expansion must be
greater than or equal to zero.
As in the single variable case, and for the same reason, we start by considering the first order
terms. Since p is an arbitrary vector and ε can be positive or negative, every component of the
gradient vector g(x∗ ) must be zero.
Now we have to consider the second order term, ε2 pT H(x∗ + εθp)p. For this term to be
non-negative, H(x∗ + εθp) has to be positive semi-definite, and by continuity, the Hessian at the
optimum, H(x∗ ) must also be positive semi-definite.

• The matrix H ∈ Rn×n is positive definite if pT Hp > 0 for all nonzero vectors p ∈ Rn .
If H = H T then all the eigenvalues of H are strictly positive.
• The matrix H ∈ Rn×n is positive semi-definite if pT Hp ≥ 0 for all vectors p ∈ Rn .
If H = H T then the eigenvalues of H are positive or zero.
AA222: MDO 56 Friday 6th April, 2012 at 12:06

(a) Positive definite (b) Positive semi-definite

(c) Indefinite (d) Negative definite

Figure 3.2: Various possibilities for a quadratic function

AA222: MDO 57 Friday 6th April, 2012 at 12:06

Figure 3.3: Critical points of f (x) = 1.5x21 + x22 − 2x1 x2 + 2x31 + 0.5x41

• The matrix H ∈ Rn×n is indefinite if there exists p, q ∈ Rn such that pT Hp > 0 and q T Hq < 0.
If H = H T then H has eigenvalues of mixed sign.

• The matrix H ∈ Rn×n is negative definite if pT Hp < 0 for all nonzero vectors p ∈ Rn .
If H = H T then all the eigenvalues of H are strictly negative

Necessary conditions (for a local minimum):

kg(x∗ )k = 0 and H(x∗ ) is positive semi-definite. (3.8)

Sufficient conditions (for a strong local minimum):

kg(x∗ )k = 0 and H(x∗ ) is positive definite. (3.9)

Example 3.4. Critical Points of a Function Consider the function:

f (x) = 1.5x21 + x22 − 2x1 x2 + 2x31 + 0.5x41

Find all stationary points of f and classify them.

Solve ∇f (x) = 0, get three solutions:

(0, 0) local minimum

√ √
1/2(−3 − 7, −3 − 7) global minimum
√ √
1/2(−3 + 7, −3 + 7) saddle point

To establish the type of point, we have to determine if the Hessian is positive definite and compare
the values of the function at the points.
AA222: MDO 58 Friday 6th April, 2012 at 12:06

3.4 Steepest Descent Method

The steepest descent method uses the gradient vector at xk as the search direction for the major
iteration k. As mentioned previously, the gradient vector is orthogonal to the plane tangent to the
isosurfaces of the function. The gradient vector, g(xk ), is also the direction of maximum rate of
change (maximum increase) of the function at that point. This rate of change is given by the norm,
kg(xk )k.

Algorithm 5 Steepest descent

Input: Initial guess, x0 , convergence tolerances, εg , εa and εr .
Output: Optimum, x∗
k←0
repeat
Compute the gradient of the objective function, g(xk ) ≡ ∇f (xk )
Compute the normalized search direction, pk ← −g(xk )/kg(xk )k
Perform line search to find step length αk
Update the current point, xk+1 ← xk + αk pk
k ←k+1
until |f (xk ) − f (xk−1 )| ≤ εa + εr |f (xk−1 )| and kg(xk−1 )k ≤ εg

Here, |f (xk+1 ) − f (xk )| ≤ εa + εr |f (xk )| is a check for the successive reductions of f . εa is the
absolute tolerance on the change in function value (usually small ≈ 10−6 ) and εr is the relative
tolerance (usually set to 0.01).
If we use an exact line search, the steepest descent direction at each iteration is orthogonal to
the previous one, i.e.,

df (xk+1 )
=0
dα
∂f (xk+1 ) ∂ (xk + αpk )
⇒ =0
∂xk+1 ∂α
⇒ ∇T f (xk+1 )pk = 0
⇒ −g T (xk+1 )g(xk ) = 0

Because of this property of the search directions, the steepest descent method “zigzags” in the
design space and is inefficient, in general. Although a substantial decrease may be observed in the
first few iterations, the method is usually very slow to converge to a local optimum. In particular,
while the algorithm is guaranteed to converge, it may take an infinite number of iterations. The
rate of convergence is linear.
Consider, for example the quadratic function of two variables,

f (x) = (1/2)(x21 + 10x22 )

.
The following figure shows the iterations given by steepest descent when started at x0 = (10, 1).
AA222: MDO 59 Friday 6th April, 2012 at 12:06

For steepest descent and other gradient methods that do not produce well-scaled search direc-
tions, we need to use other information to guess a step length.
One strategy is to assume that the first-order change in xk will be the same as the one obtained
T p
in the previous step. i.e, that ᾱgkT pk = αk−1 gk−1 k−1 and therefore:

T p
gk−1 k−1
ᾱ = αk−1 . (3.10)
gkT pk

Example 3.5. Steepest Descent:

Consider the following function.
2 2
f (x1 , x2 ) = 1 − e−(10x1 +x2 ) . (3.11)

The function f is not quadratic, but, as |x1 | and |x2 | → 0, we see that

f (x1 , x2 = 10x21 + x22 + O(x41 ) + O(x42 )

Thus, this function is essentially a quadratic near the minimum (0, 0)T . Figure 3.4 shows the path
taken by steepest descent starting from the initial point, (−0.1, 0.6)T .

3.5 Conjugate Gradient Method

A small modification to the steepest descent method takes into account the history of the gradients
to move more directly towards the optimum.
Suppose we want to minimize a convex quadratic function
1
φ(x) = xT Ax − bT x (3.12)
2
where A is an n × n matrix that is symmetric and positive definite. Differentiating this with respect
to x we obtain,
∇φ(x) = Ax − b ≡ r(x). (3.13)
Minimizing the quadratic is thus equivalent to solving the linear system,

∇φ = 0 ⇒ Ax = b. (3.14)

The conjugate gradient method is an iterative method for solving linear systems of equations such
as this one.
AA222: MDO 60 Friday 6th April, 2012 at 12:06

Figure 3.4: Solution path of the steepest descent method

A set of nonzero vectors {p0 , p1 , . . . , pn−1 } is conjugate with respect to A if

pTi Apj = 0, for all i 6= j. (3.15)

Suppose that we start from a point x0 and a set of conjugate directions {p0 , p1 , . . . , pn−1 } to
generate a sequence {xk } where
xk+1 = xk + αk pk (3.16)

where αk is the minimizer of φ along xk + αpk , given by

 
dφ(xk + αpk ) d 1 T T
= (xk + αpk ) A(xk + αpk ) − b (xk + αpk ) = 0
dα dα 2
⇒ pTk A(xk + αpk ) − pTk b = 0
⇒ pTk (Axk − b) +αpTk Apk = 0
| {z }
rk

Rearranging the last equation we find

rkT pk
αk ≡ − (3.17)
pTk Apk

We will see that for any x0 the sequence {xk } generated by the conjugate direction algorithm
converges to the solution of the linear system in at most n steps.
AA222: MDO 61 Friday 6th April, 2012 at 12:06

The conjugate directions are linearly independent, so they span n-space. Therefore, we can
expand the vector x∗ − x0 using the pk as a basis (recall that x∗ is the solution).

x∗ − x0 = σ0 p0 + · · · + σn−1 pn−1 (3.18)

Premultiplying by pTk A and using the conjugacy property we obtain

pTk A(x∗ − x0 )
σk = (3.19)
pTk Apk

Now we will show that the σ’s are really the α’s defined in (3.17). A given iteration point can
be written as a function of the search directions and step sizes so far,

xk = x0 + α0 p0 + · · · + αk−1 pk−1 . (3.20)

Premultiplying by pTk A and using the conjugacy property we obtain

pTk A(xk − x0 ) = 0. (3.21)

Therefore

pTk A(x∗ − x0 ) = pTk A(x∗ − xk + xk − x0 )

= pTk A(x∗ − xk ) + pTk A(xk − x0 )
| {z }
=0
∗
= pTk (Ax
|{z} −Axk )
=b
= pTk (b − Ax )
| {z k}
=−rk
T
= −pk rk

Dividing the leftmost term and the rightmost term by pTk Apk we get the equality,

pTk A(x∗ − x0 ) pTk rk

= − ⇒ σk = αk . (3.22)
pTk Apk pTk Apk

In light of this relationship and (3.18), if we step along the conjugate directions using αk , we will
solve the linear system Ax = b in at most n iterations.
There is a simple interpretation of the conjugate directions. If A is diagonal, the isosurfaces are
ellipsoids with axes aligned with coordinate directions. We could then find minimum by performing
univariate minimization along each coordinate direction in turn and this would result in convergence
to the minimum in n iterations.
When A a positive-definite matrix that is not diagonal, its contours are still elliptical, but they
are not aligned with the coordinate axes. Minimization along coordinate directions no longer leads
to solution in n iterations (or even a finite n). Thus, we need a different set of search directions to
recover convergence in n iterations: this is the role of the conjugate directions.
The original variables x are related to the new ones by x = S x̂. Inverting this transformation,

x̂ = S −1 x, (3.23)
AA222: MDO 62 Friday 6th April, 2012 at 12:06

 
where S = p0 p1 · · · pn−1 , a matrix whose columns are the set of conjugate directions with
respect to A. The quadratic now becomes

1
T
φ̂(x̂) = x̂T S T AS x̂ − S T b x̂


(3.24)
2

By conjugacy, S T AS is diagonal so we can do a sequence of n line minimizations along the

coordinate directions of x̂. Each univariate minimization determines a component of x∗ correctly.

When the conjugate-gradient method is adapted to general nonlinear problems, we obtain the
Fletcher–Reeves method.

Algorithm 6 Nonlinear conjugate gradient method

Input: Initial guess, x0 , convergence tolerances, εg , εa and εr .
Output: Optimum, x∗
k←0
repeat
Compute the gradient of the objective function, g(xk )
if k=0 then
Compute the normalized steepest descent direction, pk ← −g(xk )/kg(xk )k
else
gT g
Compute β ← gT k g k
k−1 k−1
Compute the conjugate gradient direction pk = −gk /kg(xk )k + βk pk−1
end if
Perform line search to find step length αk
Update the current point, xk+1 ← xk + αk pk
k ←k+1
until |f (xk ) − f (xk−1 )| ≤ εa + εr |f (xk−1 )| and kg(xk−1 )k ≤ εg

Usually, a restart is performed every n iterations for computational stability, i.e. we start with
a steepest descent direction.
The conjugate gradient method does not produce well-scaled search directions, so we can use
same strategy to choose the initial step size as for steepest descent.
Several variants of the Fletcher–Reeves CG method have been proposed. Most of these variants
differ in their definition of βk . For example, Dai and Yuan [2] propose

kgk k2
βk = . (3.25)
(gk − gk−1 )T pk−1

Another variant is the Polak–Ribière formula

gkT (gk − gk−1 )

βk = T g
. (3.26)
gk−1 k−1

The only difference relative to the steepest descent is that the each descent direction is modified
by adding a contribution from the previous direction.
The convergence rate of the nonlinear conjugate gradient is linear, but can be superlinear,
converging in n to 5n iterations.
AA222: MDO 63 Friday 6th April, 2012 at 12:06

Figure 3.5: Solution path of the nonlinear conjugate gradient method.

Since this method is just a minor modification away from steepest descent and performs much
better, there is no excuse for steepest descent!
Example 3.6. Conjugate Gradient Method:
Figure 3.5 plots the solution path of the nonlinear conjugate gradient method applied to the ob-
jective function f given by (3.11). The qualitative improvement over the steepest-descent path
(Figure 3.4) is evident.

3.6 Newton’s Method

The steepest descent and conjugate gradient methods only use first order information (the first
derivative term in the Taylor series) to obtain a local model of the function.
Newton methods use a second-order Taylor series expansion of the function about the current
design point, i.e. a quadratic model

1
f (xk + sk ) ≈ fk + gkT sk + sTk Hk sk , (3.27)
2
where sk is the step to the minimum. Differentiating this with respect to sk and setting it to zero,
we can obtain the step for that minimizes this quadratic,

Hk sk = −gk . (3.28)

This is a linear system which yields the Newton step, sk , as a solution.

AA222: MDO 64 Friday 6th April, 2012 at 12:06

If f is a quadratic function and Hk is positive definite, Newton’s method requires only one
iteration to converge from any starting point. For a general nonlinear function, Newton’s method
converges quadratically if x0 is sufficiently close to x∗ and the Hessian is positive definite at x∗ .
As in the single variable case, difficulties and even failure may occur when the quadratic model
is a poor approximation of f far from the current point. If Hk is not positive definite, the quadratic
model might not have a minimum or even a stationary point. For some nonlinear functions, the
Newton step might be such that f (xk + sk ) > f (xk ) and the method is not guaranteed to converge.
Another disadvantage of Newton’s method is the need to compute not only the gradient, but
also the Hessian, which contains n(n + 1)/2 second order derivatives.
A small modification to Newton’s method is to perform a line search along the Newton direction,
rather than accepting the step size that would minimize the quadratic model.

Algorithm 7 Modified Newton’s method

Input: Initial guess, x0 , convergence tolerances, εg , εa and εr .
Output: Optimum, x∗
k←0
repeat
Compute the gradient of the objective function, g(xk )
Compute the Hessian of the objective function, H(xk )
Compute the search direction, pk = −H −1 gk
Perform line search to find step length αk , starting with α = 1
Update the current point, xk+1 ← xk + αk pk
k ←k+1
until |f (xk ) − f (xk−1 )| ≤ εa + εr |f (xk−1 )| and kg(xk−1 )k ≤ εg

Although this modification increases the probability that f (xk +pk ) < f (xk ), it is still vulnerable
to the problem of having an Hessian that is not positive definite and has all the other disadvantages
of the pure Newton’s method.
We could also introduce a modification to use a symmetric positive definite matrix instead of
the real Hessian to ensure descent:
Bk = Hk + γI,

where γ is chosen such that all the eigenvalues of Bk are greater than a tolerance δ > 0 (if δ is too
close to zero, Bk might be ill-conditioned, which can lead to round of errors in the solution of the
linear system).
When using Newton’s method, or quasi-Newton methods described next, the starting step
length ᾱ is usually set to 1, since Newton’s method already provides a good guess for the step size.
The step size reduction ratio (ρ in the backtracking line search) sometimes varies during the
optimization process and is such that 0 < ρ < 1. In practice ρ is not set to be too close to 0 or 1.

Example 3.7. Modified Newton’s Method:

In this example, we apply Algorithm 7 to the function f given by (3.11) starting from (−0.1, 0.6)T .
The resulting optimization path is shown in Fig. 3.6, where we can see the rapid convergence of
the method: indeed, the first step is indistinguishable from subsequent steps, since f is almost a
quadratic. However, moving the initial guess slightly away from the origin leads to divergence,
because the Newton search direction ceases to be a descent direction.
AA222: MDO 65 Friday 6th April, 2012 at 12:06

Figure 3.6: Solution path of the modified Newton’s method.

AA222: MDO 66 Friday 6th April, 2012 at 12:06

3.7 Quasi-Newton Methods

This class of methods uses first order information only, but build second order information —
an approximate Hessian — based on the sequence of function values and gradients from previous
iterations. The various quasi-Newton methods differ in how they update the approximate Hessian.
Most of these methods also force the Hessian to be symmetric and positive definite, which can
greatly improve their convergence properties.

3.7.1 Davidon–Fletcher–Powell (DFP) Method

One of the first quasi-Newton methods was devised by Davidon in 1959, who a physicist at Argonne
National Laboratories. He was using a coordinate descent method, and had limited computer
resources, so he invented a more efficient method that resulted in the first quasi-Newton method.
This was one of the most revolutionary ideas in nonlinear optimization. Davidon’s paper was
not accepted for publication! It remained a technical report until 1991, when it was published as
the first paper in the inaugural issue of the SIAM Journal on Optimization [3].
Fletcher and Powell later demonstrated modified the method and showed that it was much
faster than current ones [5], and hence it became known as the Davidon–Fletcher–Powell (DFP)
method.
Suppose we model the objective function as a quadratic at the current iterate xk :
1
φk (p) = fk + gkT p + pT Bk p, (3.29)
2
where Bk is an n × n symmetric positive definite matrix that is updated every iteration.
The step pk that minimizes this convex quadratic model can be written as,

pk = −Bk−1 gk . (3.30)

This solution is used to compute the search direction to obtain the new iterate

xk+1 = xk + αk pk (3.31)

where αk is obtained using a line search.

This is the same procedure as the Newton method, except that we use an approximate Hessian
Bk instead of the true Hessian.
Instead of computing Bk “from scratch” at every iteration, a quasi-Newton method updates it
in a way that accounts for the curvature measured during the most recent step. We want to build
an updated quadratic model,

T 1
φk+1 (p) = fk+1 + gk+1 p + pT Bk+1 p. (3.32)
2
What requirements should we impose on Bk+1 , based on the new information from the last step?
Using the secant method we can find the univariate quadratic function along the previous
direction pk based on the two last two gradients gk+1 and gk , and the last function value fk+1 . The
slope of the univariate function is the gradient of the function projected onto the p direction, i.e.,
f 0 = g T p. The univariate quadratic is given by

0 θ2 ˜00
φk+1 (θ) = fk+1 + θfk+1 + f (3.33)
2 k+1
AA222: MDO 67 Friday 6th April, 2012 at 12:06

where s = αkpk and f˜k+100 is the approximation to the curvature of f , which is given by a forward
finite difference on the slopes, i.e.,
f 0 − fk0
f˜k+1
00
= k+1 (3.34)
αk kpk k
These slopes are easily obtained by projecting the respective gradients onto the last direction pk .
The result is a quadratic that matches f at the current point, since φ(0) = fk+1 . The slope of this
quadratic is given by
φ0k+1 (θ) = fk+1
0
+ θf˜k+1
00
(3.35)
At θ = 0 this slope also matches that of f , since φ0k+1 (0) = fk+1
0 .
The previous point xk corresponds to θ = −αk kpk k. Substituting the curvature approxima-
tion (3.34) into the slope of the quadratic (3.35), and evaluating it at xk we obtain

φ0k+1 (−αk kpk k) = fk0 (3.36)

Thus, the quadratic based on the secant method matches the slope and value at the current point,
and the slope of the previous point. This is illustrated in Fig. 3.7.

fk0

0
fk+1 φ

xk xk+1

Figure 3.7: Projection of the quadratic model onto the last search direction, illustrating the secant
condition

Going back to n-dimensional space, the gradient of the quadratic model (3.32) is

∇φk+1 (p) = gk+1 + Bk+1 p. (3.37)

For this gradient to match that of the actual function at the previous point xk ,

∇φk+1 (−αk pk ) = gk+1 − αk Bk+1 pk = gk . (3.38)

AA222: MDO 68 Friday 6th April, 2012 at 12:06

xk+1
pk+1
pk
xk gk+1
gk

Figure 3.8: The difference in the gradients at two subsequent points gk and gk+1 is projected onto
the direction pk to enforce the secant condition show in Fig. 3.7.

Rearranging we obtain,
Bk+1 αk pk = gk+1 − gk . (3.39)
which is called the secant condition.
For convenience, we will set the difference of the gradients to yk = gk+1 − gk , and sk = xk+1 − xk
so the secant condition is then written as

Bk+1 sk = yk . (3.40)

We have n(n + 1)/2 unknowns and only n equations, so this system has an infinite number of
solutions for Bk+1 . To determine the solution uniquely, we impose a condition that among all the
matrices that satisfy the secant condition, selects the Bk+1 that is “closest” to the previous Hessian
approximation Bk , i.e. we solve the following optimization problem,

minimize kB − Bk k
with respect to B (3.41)
T
subject to B = B , Bsk = yk .

Using different matrix norms result in different quasi-Newton methods. One norm that makes
it easy to solve this problem and possesses good numerical properties is the weighted Frobenius
norm
kAkW = kW 1/2 AW 1/2 kF , (3.42)
Pn Pn
where the norm is defined as kCkF = i=1 j=1 c2ij . The weights W are chosen to satisfy certain
favorable conditions. The norm is adimensional (i.e., does not depend on the units of the problem)
if the weights are chosen appropriately.
For further details on the derivation of the quasi-Newton methods, see Dennis and Moré’s
review [4].
Using this norm and weights, the unique solution of the norm minimization problem (3.41) is,

yk s T sk y T yk y T
   
Bk+1 = I − T k Bk I − T k + T k , (3.43)
yk sk yk sk yk s k

which is the DFP updating formula originally proposed by Davidon.

Using the inverse of Bk is usually more useful, since the search direction can then be obtained
by matrix multiplication. Defining,
Vk = Bk−1 . (3.44)
AA222: MDO 69 Friday 6th April, 2012 at 12:06

The DFP update for the inverse of the Hessian approximation can be shown to be

Vk yk ykT Vk sk sTk
Vk+1 = Vk − + (3.45)
ykT Vk yk ykT sk

Note that this is a rank 2 update. A rank one update is Bk+1 = Bk + uv T , which adds a matrix
whose columns are the same vector u multiplied by the scalars in v. this means that the columns
are all linearly dependent and the matrix spans 1-space and is rank 1. A rank two update looks
like Bk+1 = Bk + u1 v1T + u2 v2T and the update spans 2-space.

Algorithm 8 Quasi-Newton method with DFP update

Input: Initial guess, x0 , convergence tolerances, εg , εa and εr .
Output: Optimum, x∗
k←0
V0 ← I
repeat
Compute the gradient of the objective function, g(xk )
Compute the search direction, pk ← −Vk gk
Perform line search to find step length αk , starting with α ← 1
Update the current point, xk+1 ← xk + αk pk
Set the step length, sk ← αk pk
Compute the change in the gradient, yk ← gk+1 − gk
Vk yk ykT Vk
Ak ← ykT Vk yk
sk sT
Bk ← sT
k
k yk
Compute the updated approximation to the inverse of the Hessian, Vk+1 ← Vk − Ak + Bk
until |f (xk ) − f (xk−1 )| ≤ εa + εr |f (xk−1 )| and kg(xk−1 )k ≤ εg

3.7.2 Broyden–Fletcher–Goldfarb–Shanno (BFGS) Method

The DFP update was soon superseded by the BFGS formula, which is generally considered to be
the most effective quasi-Newton update. Instead of solving the norm minimization problem (3.41)
of B we now solve the same problem for its inverse V using equivalent conditions and and the
solution is given by,
sk ykT yk sTk sk sT
   
Vk+1 = I − T Vk I − T + T k. (3.46)
s k yk s k yk s k yk
The relative performance between the DFP and BFGS methods is problem dependent.
Example 3.8. BFGS:
As for the previous methods, we apply BFGS to the function f given by (3.11). The solution path,
starting at (−0.1, 0.6)T , is shown in 3.10. Compare the path of BFGS with that from modified
Newton’s method (Figure 3.6) and the conjugate gradient method (Figure 3.5).
Example 3.9. Minimization of the Rosenbrock Function
Minimize the function,

2
(x) = 100 x2 − x21 + (1 − x1 )2 , (3.47)
starting from x0 = (−1.2, 1.0)T .
AA222: MDO 70 Friday 6th April, 2012 at 12:06

Figure 3.9: Broyden, Fletcher, Goldfarb and Shanno at the NATO Optimization Meeting (Cam-
bridge, UK, 1983), which was a seminal meeting for continuous optimization (courtesy of Prof.
John Dennis)
AA222: MDO 71 Friday 6th April, 2012 at 12:06

Figure 3.10: Solution path of the BFGS method.

15

2
0.5
15

0
15

20
10

1.5
20

0
20

0.5
5
1 0.2.2
10

20
1 5
2
5

10

0
10
1
1
20
10
5
10

5
15

15

0.5

20
2
15

0.5
5

20 0
x2

20

0.

10
200

1 00

5
5

100

2
20
15
40 0

10

1
300

300

0 5
5 00

10 2

15
600

20
-0.5
0

4 00
20

7 00
20

10

0
0

10
400
500
60 0

30

-1
30
0

-1.5 -1 -0.5 0 0.5 1 1.5

x1

Figure 3.11: Solution path of the steepest descent method

AA222: MDO 72 Friday 6th April, 2012 at 12:06

10
0
20

20
1.5

10

2
0

15
10

0.2 0.22 1
5

0.5
0

5
10 0
10

15
1

10 0

10
2

0.5
15

5
20
5

20
1
10
15
20 0
20

0.5 15

10 0
10

2
x2

1
2
300

5
5
4 00

5
0.
100

200
15
0

10
2

10
1
5

20

50 0
15

300
20
200

-0.5

0
10
60 0

700 00
40 0
3 00

6
5 00

400

10
0

0
70 0

20
-1
-1.5 -1 -0.5 0 0.5 1 1.5
x1

Figure 3.12: Solution path of the nonlinear conjugate gradient method

0.2

1.5
20
10
0
5

2
15

200
15
2
1
2105

5
20

0.5

0
10
10

10
0
15 2

1
100

10

.2
00.5
10

1
2 00

5
0.2

1 00
5

0.5
20

20
2
x2

15
5
15

15

5
2
2
0.
1

10
300

20

5
0.
10

2
0
1

200

500
10

10
0

3 00
4 00

5
2 00
400

15
6 00

20
0
10

-0.5
7 00
30
0
500
6 00

1 00
-1
-1.5 -1 -0.5 0 0.5 1 1.5
x1

Figure 3.13: Solution path of the modified Newton method

AA222: MDO 73 Friday 6th April, 2012 at 12:06

300

20
20
0

5
20

0.2
2
10
1.5

5
5

15
10 0

2
10

0.5
1
10
0
10
1

20

0.2
1
10

15
15

0.5
15

20

20 10

200
20

5
0.2
5

2
300

0.5 15

5
x2

1 00
10 2
100

15
10 0.
2 00

40 0
1
0
20

15
15

5 10

20

5 00
20 0

300
40 0

-0.5
30 0
700

500
60 0

600
0
10
10
0
20
0

0
-1

40
-1.5 -1 -0.5 0 0.5 1 1.5
x1

Figure 3.14: Solution path of the BFGS method

Figure 3.15: Comparison of convergence rates for the Rosenbrock function

AA222: MDO 74 Friday 6th April, 2012 at 12:06

3.7.3 Symmetric Rank-1 Update Method (SR1)

If we drop the requirement that the approximate Hessian (or its inverse) be positive definite, we
can derive a simple rank-1 update formula for Bk that maintains the symmetry of the matrix and
satisfies the secant equation. Such a formula is given by the symmetric rank-1 update (SR1) method
(we use the Hessian update here, and not the inverse Vk ):

(yk − Bk sk )(yk − Bk sk )T
Bk+1 = Bk + . (3.48)
(yk − Bk sk )T sk

With this formula, we must consider the cases when the denominator vanishes, and add the neces-
sary safe-guards:

1. if yk = Bk sk then the denominator is zero, and the only update that satisfies the secant
equation is Bk+1 = Bk (i.e., do not change the matrix).

2. if yk 6= Bk sk and (yk − Bk sk )T sk = 0 then there is no symmetric rank-1 update that satisfies

the secant equation.

To avoid the second case, we update the matrix only if the following condition is met:

|ykT (sk − Bk yk )| ≥ rksk kkyk − Bk sk k,

where r ∈ (0, 1) is a small number (e.g., r = 10−8 ). Hence, if this condition is not met, we use
Bk+1 = Bk .
Why would we be interested in a Hessian approximation that is potentially indefinite? In
practice, the matrices produced by SR1 have been found to approximate the true Hessian matrix
well (often better than BFGS). This may be useful in trust-region methods (see next section)
or constrained optimization problems; in the latter case the Hessian of the Lagrangian is often
indefinite, even at the minimizer.
To use the SR1 method in practice, it may be necessary to add a diagonal matrix γI to Bk
when calculating the search direction, as was done in modified Newton’s method. In addition, a
simple back-tracking line search can be used, since the Wolfe conditions are not required as part of
the update (unlike BFGS).

3.8 Trust Region Methods

Trust region, or “restricted-step” methods are a different approach to resolving the weaknesses of
the pure form of Newton’s method, arising from an Hessian that is not positive definite or a highly
nonlinear function.
One way to interpret these problems is to say that they arise from the fact that we are stepping
outside a the region for which the quadratic approximation is reasonable. Thus we can overcome
this difficulties by minimizing the quadratic function within a region around xk within which we
trust the quadratic model.
The reliability index, rk , is the ratio of the actual reduction to the predicted reduction; the
closer it is to unity, the better the agreement. If fk+1 > fk (new point is worse), rk is negative.
AA222: MDO 75 Friday 6th April, 2012 at 12:06

Algorithm 9 Trust region algorithm

Input: Initial guess x0 , convergence tolerances, εg , εa and εr , initial size of the trust region, h0
Output: Optimum, x∗
k←0
repeat
Compute the Hessian of the objective function H(xk ), and solve the quadratic subproblem:
1
minimize q(sk ) = f (xk ) + g(xk )T sk + sTk H(xk )sk
2
w.r.t. sk
s.t. − hk ≤ sk ≤ hk , i = 1, . . . , n

Evaluate f (xk + sk ) and compute the ratio that measures the accuracy of the quadratic model,

f (xk ) − f (xk + sk ) ∆f
rk ← =
f (xk ) − q(sk ) ∆q

if rk < 0.25 then

hk+1 ← ks4k k {Model is not good; shrink the trust region}
else if rk > 0.75 and hk = ksk k then
hk+1 ← 2hk {Model is good and new point on edge; expand trust region}
else
hk+1 ← hk {New point with trust region and the model is reasonable; keep trust region the
same size}
end if
if rk ≤ 0 then
xk+1 ← xk {Keep trust region centered about the same point}
else
xk+1 ← xk + sk {Move center of trust region to new point}
end if
k ←k+1
until |f (xk ) − f (xk−1 )| ≤ εa + εr |f (xk−1 )| and kg(xk−1 )k ≤ εg

The initial value of h is usually 1. The same stopping criteria used in other gradient-based
methods are applicable.

Bibliography
[1] Ashok D. Belegundu and Tirupathi R. Chandrupatla. Optimization Concepts and Applications
in Engineering, chapter 3. Prentice Hall, 1999.

[2] Y.H. Dai and Y. Yuan. A nonlinear conjugate gradient with a strong global convergence prop-
erty. SIAM Journal on Optimization, 10(1):177–182, 1999.

[3] William C. Davidon. Variable metric method for minimization. SIAM Journal on Optimization,
1(1):1–17, February 1991.
AA222: MDO 76 Friday 6th April, 2012 at 12:06

[4] J.E. Dennis and J. J. Moreé. Quasi-Newton methods, motivation and theory. SIAM Review,
19(1):46–89, 1977.

[5] R. Fletcher and M. J. D. Powell. A rapidly convergent descent method for minimization. The
Computer Journal, 6(2):163–168, 1963. doi: 10.1093/comjnl/6.2.163.

[6] Chinyere Onwubiko. Introduction to Engineering Design Optimization, chapter 4. Prentice Hall,
2000.