35241 OQM

Lecture Note – Part 8

Unconstrained Nonlinear Optimisation

1 Introduction

Nonlinear programming (NLP) is somewhat more complicated

than linear programming (LP). We start from discussing a simpli-
fied version – unconstrained NLP, i.e. NLP with no constraints.
Assume that f (x) is a function of vector x = (x1 , x2 , . . . , xn )T
defined over the domain D ⊆ Rn . Consider the optimisation
problem
min (or max)f (x).
x∈D x∈D
n
If D ≡ R , then we have an unconstrained optimisation problem

min (or max)f (x),

where no constraints are placed on the decision variables x.

Although much of the content in this chapter is largely related to
calculus you may have learnt, we address more closely the question
of how to actually find an optimal solution instead of how to
recognise one.
To carefully specify what sort of NLP problems we will consider,
let’s start by discussing convexity for minimisation NLP (or con-
cavity for maximisation NLP).

1
2 Convexity
In this section, we briefly consider what kind of situations could
make an optimisation problem particularly hard to solve. We will
need the following important definitions.

2.1 Global minimum

A point x∗ is called a global minimiser or a global minimum point

of a function f (x) if
x∗ ∈ D, and
f (x∗ ) ≤ f (x) for any x ∈ D.
The value f (x∗ ) is called a global minimum value of f (x).
Similarly defined is a strict global minimiser or a strict global
minimum point, where

f (x∗ ) < f (x) for any x ∈ D \ {x∗ }.

2.2 Local minimum

A point x∗ is called a local minimiser or a local minimum point

of a function f (x) if
x∗ ∈ D,
and there exists an ε > 0 such that

f (x∗ ) ≤ f (x) for any x ∈ D satisfying

0 < kx − x∗ k < ε.
The value f (x∗ ) is called a local minimum value of f (x).

2
Similarly defined is a strict local minimiser or a strict local mini-
mum point, where

f (x∗ ) < f (x) for any x ∈ D \ {x∗ } satisfying

0 < kx − x∗ k < ε.
It is possible for a function to have

• both global and local minimisers;

• neither global nor local minimisers;

• a local minimiser and yet no global minimiser;

• multiple global minimisers;

• various other combinations.

The basic notion of the solution techniques for the minimisation

NLP we will introduce later is to move from some solution “down-
hill” to a better solution. These methods are guaranteed to find
a local minimum, but in some cases there could be many different
local minima such that the same algorithm with different starting
points will end up with different local minima. And in most cases
we are not able to ensure if the obtained local minimum is the
global minimum.
Opposite to the convexity, global minimum and local minimum
for the minimisation NLP, the concavity, global maximum and lo-
cal maximum for the maximisation NLP can be similarly defined.

3
Now we limit ourselves to a specific type of NLP problems – min-
imising a convex function (or maximising a concave function) over
a convex set.

2.3 Convex function

A function f (x) is called convex if for any two points (or vectors)
x1 ∈ D and x2 ∈ D and for any α ∈ [0, 1] we have


f αx1 + (1 − α)x2 ≤ αf (x1 ) + (1 − α)f (x2 ).

Function f (x) is called strictly convex if for any x1 6= x2 ∈ D and

for any α ∈ (0, 1) we have


f αx1 + (1 − α)x2 < αf (x1 ) + (1 − α)f (x2 ).

Assume that f (x) has continuous second-order partial deriva-

tives. At each point x = (x1 , x2, . . . , xn ), we denote by
 T
∂f (x) ∂f (x) ∂f (x)
∇f (x) = , , ··· ,
∂x1 ∂x2 ∂xn
the gradient of f (x) at point x, and by
 
∂ 2 f (x) ∂ 2 f (x) ∂ 2 f (x)
···
 2∂x21 ∂x1 ∂x2 ∂x1 ∂xn 
 ∂ f (x) ∂ 2 f (x) ∂ 2 f (x) 
···
∇2 f (x) =  ∂x22
 ∂x2 ∂x1 ∂x2 ∂xn 
.. .. .. .. 

 . . . . 

∂ 2 f (x) ∂ 2 f (x) ∂ 2 f (x)
∂xn ∂x1 ∂xn ∂x2 ··· ∂x2n

the Hessian matrix for f (x) at point x. Note that the Hessian is a
symmetric matrix since if f (x) has second-order partial derivatives

4
at point x we have for 1 ≤ i, j ≤ n
∂ 2 f (x) ∂ 2 f (x)
= .
∂xi ∂xj ∂xj ∂xi

Definition. For each i = 1, . . . , n, the ith principal minor(s) of

an n × n matrix is the determinant of any i × i matrix obtained
by deleting n − i rows and the corresponding n − i columns of the
matrix.
Example 1. For the matrix
!
−2 −1
A= .
−1 4

The 1st principal minors are −2 and 4.

The 2nd principal minor is |A| = (−2)(4) − (−1)(−1) = −9.
For any matrix, the 1st principal minors are just the diagonal
entries of the matrix.
By applying the theorem stated below, the Hessian matrix can
be used to determine whether a function f (x) is convex.
Theorem 1. Suppose that f (x) has continuous second-order par-
tial derivatives at each point x = (x1 , x2, . . . , xn ) ∈ D. Then f (x)
is a convex function on D if and only if for each x ∈ D all principal
minors of its Hessian are nonnegative.
Example 2. The Hessian matrix of the function f (x) = x21 +
2x1x2 + x22 at any point x = (x1 , x2 ) ∈ R2 is
!
2 2
∇2 f (x) =
2 2

5
The 1st principal minors are 2 > 0 and 2 > 0. The 2nd principal
minor is 2 × 2 − 2 × 2 = 0. So, Theorem 1 shows that f (x) is a
convex function on R2.

2.4 Concave function

A function f (x) is called concave if for any two points (or vectors)
x1 ∈ D and x2 ∈ D and for any α ∈ [0, 1] we have


f αx1 + (1 − α)x2 ≥ αf (x1 ) + (1 − α)f (x2 ).

Function f (x) is called strictly convex if for any x1 6= x2 ∈ D and

for any α ∈ (0, 1) we have


f αx1 + (1 − α)x2 > αf (x1 ) + (1 − α)f (x2 ).

Note:

• It is apparent that a function f (x) is (strictly) concave if and

only if the function −f (x) is (strictly) convex.

• If f (x) is a linear function, i.e. f (x) = cT x, then

f αx1 + (1 − α)x2 = αf (x1 ) + (1 − α)f (x2 ).

Thus, linear functions are both convex and concave. But

they are neither strictly convex nor strictly concave.

Theorem 2. Suppose that f (x) has continuous second-order par-

tial derivatives at each point x = (x1 , x2, . . . , xn ) ∈ D. Then f (x)
is a concave function on D if and only if for each x ∈ D and each

6
k = 1, . . . , n all nonzero k th principal minors of its Hessian matrix
have the same sign as (−1)k .
Example 3. The Hessian of the function f (x) = −3x21 + 4x1x2 −
2x22 at any point x = (x1, x2 ) ∈ R2 is
!
−6 4
∇2 (x) =
4 −4

The 1st principal minors are −6 < 0 and −4 < 0. The 2nd princi-
pal minor is (−6) × (−4) − (4) × (4) = 8 > 0. Theorem 2 shows
that f (x) is a concave function on R2.
Now we discuss the domain, i.e. feasible region, of the considered
objective function.

2.5 Convex set

Recall that the set S is called convex if for any two x1 , x2 ∈ S and
any α ∈ (0, 1) we have αx1 + (1 − α)x2 ∈ S.1
Note:

• The feasible set we consider in LP

Ax ≤ b
x≥0

is convex.2

• If g(x) is a convex function, then the set S = {x : g(x) ≤ c}

for any constant c (if existing) is convex.
1
We had introduced it in Chapter 2.
2
However, the set of integers satisfying the above conditions is not a convex set. So IP
requires special techniques.

7
 • If g(x) is a convex function, then the set

S = {u = (x|y) = (x1, x2, · · · , xn , y) : y ≥ g(x)}

is a convex set of Rn+1 . If you “colour in” above the graph

of a convex function, then you get a convex set.

Theorem 3. If f (x) is a convex function, and S is a convex set,

then any local minimum of the minimisation NLP

min f (x)
s.t. x ∈ S

is also a global minimum. If f (x) is a strictly convex function,

then the global minimum will be unique.3

3 Types of Optimality Conditions

3.1 One-dimensional case

Assume that a single-variable function f (x) is defined and has

continuous second-order derivatives on R. Then for any point x∗ ,
the Taylor’s theorem states that for “small” value δ we have
1
f (x∗ + δ) = f (x∗ ) + f ′ (x∗ )δ + f ′′ (x∗ )δ 2 + o(δ 2 ),
2
where o(δ 2 ) indicates a term that goes to zero faster than δ 2 does
as δ → 0. In other words,
o(δ 2 )
lim = 0.
δ→0 δ 2
3
The similar result applies to the concave function and maximisation NLP.

8
This is a formal way of denoting that this remainder term gets
very small if δ is close to zero, and is “dominated” by the other
terms.
The Taylor’s formula leads to the following necessary condition
and sufficient condition for x∗ to be a local minimum of f (x).
First-order condition: If x∗ is a local minimum of f (x), then
f ′ (x∗ ) = 0.
This condition is also referred to as a necessary condition for a
local minimum, since it must happen in order for x∗ being a local
minimum. But if f ′ (x∗ ) = 0, we don’t know for sure that we have
a local minimum. Thus, it is not a sufficient condition, since it
does not guarantee that x∗ will be a local minimum.
Second-order condition: If f ′ (x∗ ) = 0 and f ′′ (x∗ ) > 0, then x∗
is a local minimum of f (x).
This condition is also referred to as a sufficient condition for a
local minimum.

3.2 Multi-dimensional case

Assume that an n-variable function f (x) is defined and has con-

tinuous second-order partial derivatives on Rn . Then for any
point x∗ , the Taylor’s theorem states that for “small” deviation
d = (d1 , d2 , · · · , dn )T we have the formula
n n n
X ∂f (x∗ ) 1 X X ∂ 2 f (x∗ )
∗
f (x +d) = f (x )+∗
di + di dj +o(kdk2)
i=1
∂xi 2 i=1 j=1 ∂xi∂xj

1
= f (x∗ ) + (∇f (x∗ ))T d + dT ∇2 f (x∗ )d + o(kdk2 ).
2
9
The optimality conditions for the n-dimensional case is shown as
follows.

3.2.1 First order optimality condition

If x∗ is a local minimum of f (x), then ∇f (x∗ ) = 0.

This condition is also referred to as a necessary condition for a

local minimum since it must happen in order for x∗ being a local
minimum. But if ∇f (x∗ ) = 0, we have no idea about whether we
have a local minimum. It is not a sufficient condition since it does
not guarantee that x∗ will be a local minimum.
We call a point x∗ , where ∇f (x∗ ) = 0, a stationary point of f (x).
For unconstrained NLP problems, all local minima are stationary
points.
Example 4. Consider the function f (x) = f (x1 , x2) = x21 −
x1x2 + x22 − 3x2. Then we have
 T
∂f (x) ∂f (x)
∇f (x) = , = (2x1 − x2 , −x1 + 2x2 − 3).
∂x1 ∂x2

Solving the system of equations ∇f (x) = 0 gives x∗ = (1, 2)T ,

which is a stationary point of f (x).

3.2.2 Second Order Optimality Condition

Recall that in one-dimensional case the sufficient condition for a

local minimum is “f ′ (x∗ ) = 0 and f ′′ (x∗ ) > 0”. If f ′′ (x∗ ) = 0,
then further investigations are necessary.

10
In the n-dimensional case, the second-order derivative is gener-
alised to the Hessian matrix. If x∗ is a stationary point, then the
Taylor’s formula at x∗ gives an approximation
1
f (x∗ + d) ∼
= f (x∗ ) + dT ∇2 f (x∗ )d.
2
If f (x∗ + d) ≥ f (x∗ ), i.e. x∗ is a local minimum, then we have
dT ∇2 f (x∗ )d ≥ 0 for any d.
Definition: An n × n symmetric matrix A is called

• positive definite if xT Ax > 0 for any n-dimensional vector

x 6= 0,

• positive semidefinite if xT Ax ≥ 0 for any n-dimensional vec-

tor x,

• negative definite if xT Ax < 0 for any n-dimensional vector

x 6= 0,

• negative semidefinite if xT Ax ≤ 0 for any n-dimensional vec-

tor x,

• indefinite otherwise, i.e. xT Ax has positive and negative

values for different x.

Example 5. Let I be the n × n identity matrix. Then for any

nonzero n-dimensional vector x, we have

xT Ix = xT x = x21 + x22 + · · · + x2n > 0.

Hence, I is positive definite.

11
 !
1 −1
Example 6. The matrix A = gives
−1 1
! !
1 −1 x1
= (x1 − x2 )2 ≥ 0.


x1 x2
−1 1 x2
for all x ∈ R2. So A is positive semidefinite. Note that for any
vector x = (x1 , x1 )T we have xT Ax = 0. Hence, A is not positive
definite.
Definition: Assume that A is an n × n matrix. A nonzero n-
dimensional vector x is called an eigenvector of A if it satisfies
the equality Ax = λx for some scalar λ. The scalar λ is called an
eigenvalue of A.
The eigenvalues of A can be found by solving the characteristic
equation
det(A − λI) = 0,
where “det” indicates the determinant.
Theorem 4. A symmetric matrix A is positive definite if and
only if all its eigenvalues are positive.
Now we turn back to the optimality condition.
Theorem 5. (Second-order necessary condition)
If x∗ is a local minimum for an unconstrained NLP problem min f (x),
then
(i) ∇f (x∗ ) = 0, and

(ii) ∇2f (x∗ ) is positive semidefinite.

12
Theorem 6. (Second-order sufficient condition)
If
(i) ∇f (x∗ ) = 0, and

(ii) ∇2f (x∗ ) is positive definite,

then x∗ is a local minimum for the unconstrained NLP problem

min f (x).

Example 7. Consider the function f (x) = f (x1 , x2) = x21 −

x1x2 + x22 − 3x2 in Example 4. The function has a stationary point
x∗ = (1, 2)T . Its Hessian matrix is
 2 
∂ f (x) ∂ 2 f (x)
!
2 ∂x1 ∂x2  2 −1
∇2 f (x) = ∇2 f (x1 , x2 ) =  ∂ 2∂x 1
f (x) ∂ 2 f (x)
= .
∂x ∂x ∂x 2 −1 2
2 1 2

!
2 −1
Thus, we have ∇2f (x∗ ) = ∇2f (1, 2) = .
−1 2
The eigenvalues of this matrix are found by solving the equation
 
2− λ −1 
2
det(∇ f (1, 2) − λI) = 


 −1 2 − λ

= (2 − λ)2 − 1 = (3 − λ)(1 − λ) = 0.
⇒ λ1 = 3, λ2 = 1.
Since both eigenvalues are positive, the Hessian matrix ∇2 f (1, 2)
is positive definite. Hence, by the second-order sufficient condition
the point x∗ = (1, 2)T is a local minimum of f (x).

13
4 Convexity Revisited
Recall that a function f (x) is called convex if for any two points
x1 and x2 in its domain and for any α ∈ [0, 1] we have


f αx1 + (1 − α)x2 ≤ αf (x1 ) + (1 − α)f (x2 ).

And a function f (x) is called strictly convex if for any x1 6= x2 in

its domain and for any α ∈ (0, 1) we have


f αx1 + (1 − α)x2 < αf (x1 ) + (1 − α)f (x2 ).

Theorem 7. Consider a function f (x) defined in a convex do-

main. Then

(i) (Necessary condition for convexity) If f (x) is convex, then

∇2f (x) is positive semidefinite everywhere in its domain.

(ii) (Sufficient condition for strict convexity) Function f (x) is

strictly convex if its Hessian matrix ∇2 f (x) is positive defi-
nite for all x in its domain.

In Example 7, the Hessian matrix of function f (x) = x21 − x1 x2 +

x22 − 3x2 has positive eigenvalues, which do not depend on x, so
it is positive definite everywhere. Hence, f (x) is strictly convex.
Furthermore, Theorem 3 implies that the local minimum x∗ =
(1, 2)T is a global minimum and a unique global minimum of f (x).

14
5 Gradient Methods
In this section, we will study an important class of solution pro-
cedures for the unconstrained NLP.4 Once again, we consider the
unconstrained minimisation NLP problem

min f (x) (1)

Let x0 be an initial approximation to the solution of (1). Assume

that x0 is not a stationary point of f (x). Then we choose an initial
direction d0 , and an arbitrary scalar value α. For a sufficient small
value of α, Taylor’s theorem gives

f (x0 + αd0 ) = f (x0 ) + (∇f (x0 ))T (αd0 ) + o(αkd0 k)

= f (x0 ) + α(∇f (x0 ))T d0 + o(αkd0 k).

If the direction d0 has been chosen in such a way that (∇f (x0 ))T d0 <
0, then for a sufficient small positive α we have

f (x0 + αd0 ) < f (x0 ).

Finding the value α0 such that

f (x0 + α0 d0 ) = min f (x0 + αd0 )

α>0

gives the point x1 = x0 + α0 d0 to be a better approximation

solution for (1). This procedure can be repeated until we find a
“good enough” approximation solution for (1).
The basic idea of the gradient methods is that we choose a starting
point x0 and then at each iteration k ≥ 1
4
The applicability of these methods can be extended to the constrained NLP.

15
 1. choose a direction dk such that (∇f (xk ))T dk < 0, and

2. find the value αk satisfying

f (xk + αk dk ) = min f (xk + αdk ), and

α>0

3. take xk+1 = xk + αk dk as a better approximation solution

for (1).

The value αk is then called the step size of iteration k.

In many algorithms of this category, the direction dk is chosen
by taking a symmetric and positive definite matrix Dk and then
calculating
dk = −Dk ∇f (xk ).
This direction dk satisfies the required condition since

(∇f (xk ))T dk = (∇f (xk ))T (−Dk ∇f (xk )) = −(∇f (xk ))T Dk ∇f (xk ) < 0.

Now we introduce two algorithms which fall into the category

gradient methods.

5.1 Steepest descent method

At each iteration k ≥ 1, the steepest descent method chooses

Dk = I, the n × n identity matrix, which is positive definite. So
the direction chosen is

dk = −I · ∇f (xk ) = −∇f (xk ),

and this is a descent direction. Actually it is the direction in which

the function f (x) decreases most rapidly when moving from xk .

16
Algorithm for Steepest Descent Method

Step 0. Choose a starting point x0 , and a small positive scalar

ǫ. Set k = 0.
Step 1. If k∇f (xk )k < ǫ, then STOP: xk is a satisfactory ap-
proximation minimum of f (x). Otherwise, set

dk = −∇f (xk ).

Step 2. Choose the step size αk by solving the one-dimensional

problem
min g(α) = min f (xk + αdk ).
α>0 α>0

Let αk = arg min g(α), and then xk+1 = xk + αk dk .

α>0
Set the k = k + 1 and GO TO Step 1.

Example 8. Consider the unconstrained NLP problem

min f (x1 , x2 ) = x21 − x1 x2 + x22 − 3x2 .

Use the steepest descent method to find an approximation solution

with ǫ = 0.5 and a starting point at the origin.
Solution. We have

∇f (x1 , x2) = (2x1 − x2, −x1 + 2x2 − 3)T .

Iteration 0.
Step 0. Let x0 = (0, 0)T .
p
Step 1. ∇f (x0 ) = (0, −3)T , and k∇f (x0 )k = 02 + (−3)2 =
3 > 0.5 = ǫ.
Let d0 = −∇f (x0 ) = (0, 3)T . GO TO Step 2.

17
Step 2. To find the the step size value α0 , we need to solve the
problem
min g(α) = min f (x0 + αd0).
α>0 α>0
Then we have

x0 + αd0 = (0, 0)T + α(0, 3)T = (0, 3α)T

⇒ g(α) = f (x0 + αd0 ) = f (0, 3α) = 9α2 − 9α

dg(α) 1
= 9(2α − 1) = 0 ⇒ α0 = > 0
dα 2
Hence,
1 TT
 3 T
x1 = x0 + α0 d0 = (0, 0) + (0, 3) = 0, .
2 2

Iteration 1. q
Step 1. ∇f (x1 ) = (− 32 , T
0) , and k∇f (x1 )k = (− 23 )2 + 02 =
3
2
> ǫ = 0.5.
3

T
So let d1 = −∇f (x1 ) = 2, 0 and GO TO Step 2.

Step 2. To find the the step size value α1 , we need to solve the
problem
min g(α) = min f (x1 + αd1).
α>0 α>0
Then we have
 3 T  3 T  3α 3 T
x1 + αd1 = 0, +α , 0 = ,
2 2 2 2
 3α 3  9
⇒ g(α) = f (x1 + αd1 ) = f , = (α2 − α − 1)
2 2 4

18
 dg(α) 9 1
= (2α − 1) = 0 ⇒ α1 = > 0
dα 4 2
Hence,
 3 T 1  3 T  3 3 T
x2 = x1 + α1 d1 = 0, + ,0 = , .
2 2 2 4 2

Iteration 2.

T
Step 1. ∇f (x2 ) = 0, − 34 , and k∇f (x2 )k = 43 > ǫ = 0.5.

T
So let d2 = −∇f (x2 ) = 0, 34 and GO TO Step 2.

Step 2. To find the the step size value α2 , we need to solve the
problem
min g(α) = min f (x2 + αd2).
α>0 α>0
Then we have
 3 3 T  T
3
T 3 3 3α
x2 + αd2 = , + α 0, = , +
4 2 4 4 2 4
 3 3 3α  9
⇒ g(α) = f (x2 + αd2 ) = f , + = (α2 − α − 5)
4 2 4 16
dg(α) 9 1
= (2α − 1) = 0 ⇒ α1 = > 0
dα 16 2
Hence,
 3 3 T 1  3 T  3 15 
x3 = x2 + α2 d2 = , + 0, = , .
4 2 2 4 4 8

Iteration 3.

T
Step 1. ∇f (x3 ) = − 38 , 0
, and k∇f
 (x3 )k
3
 = 8 = 0.375 <
ǫ = 0.5. Stop and declare that x3 = 34 , 15
8 is a satisfactory
approximation solution.

19
Recall that the optimum to this minimisation NLP occurs at x =
(1, 2)T .
Steepest descent method was invented in the nineteenth century
by Cauchy. The advantages of this method is that it does not
require

• the computation of second-order derivatives,

• solving a system of equations to compute the search direction,

and

• storing matrices.

Its disadvantage is the slow rate of convergence. As a result, even

though the cost per iteration is low, the overall cost of generating
an approximation solution is high.

5.2 Newton’s method

At each iteration k, Newton’s method takes

Dk = (∇2 f (xk ))−1 .

This choice works as long as ∇2f (xk ) is positive definite, and it

will be only if f (x) is strictly convex.
The idea of the method is that at each iteration instead of finding
a minimum of f (x) we find a minimum of the quadratic approxi-
mation of f (x) around the current point xk
1
f (x) ≈ f (xk )+(∇f (xk ))T (x−xk )+ (x−xk )T ∇2 f (xk )(x−xk ) = g(x).
2

20
The minimum of g(x) occurs when the gradient of this quadratic
is zero, i.e. the next approximation can be taken as the solution
of the vector equation

∇g(x) = ∇f (xk ) + ∇2 f (xk )(x − xk ) = 0

Solving this vector equation and setting the solution x to xk+1

gives
xk+1 = xk − (∇2 f (xk ))−1 ∇f (xk ),
which is equivalent to setting

αk = 1 and Dk = (∇2 f (xk ))−1 .

Algorithm for Newton’s Method

Step 0. Choose a starting point x0 and a small positive scalar ǫ.

Let k = 0.
Step 1. If k∇f (xk )k < ǫ, then STOP: xk is a satisfactory ap-
proximation minimum of f (x). Otherwise, let

xk+1 = xk − (∇2 f (xk ))−1 ∇f (xk ).

Step 2. Set the k = k + 1 and GO TO Step 1.

Note: Newton’s method selects a step size αk = 1 at each itera-

tion k.

Example 9. Consider the unconstrained NLP problem

min f (x1 , x2 ) = x21 − x1 x2 + x22 − 3x2

21
Apply Newton’s method to find an approximation solution with
ǫ = 0.5 and a starting point at the origin.
Solution. We have

∇f (x1 , x2) = (2x1 − x2, −x1 + 2x2 − 3)T .

!
2 −1
∇2 f (x1 , x2) = .
−1 2
Iteration 0.
Step 0. Let x0 = (0, 0)T and k = 0.
Step 1.
p
∇f (x0 ) = (0, −3)T ⇒ k∇f (x0 )k = 02 + (−3)2 = 3 > ǫ = 0.5
! !
2 1
2 −1
∇2 f (x0 ) = ⇒ (∇2 f (x0 ))−1 = 31 32
−1 2 3 3
! ! !
2 1
2 −1 0 −1
(∇ f (x0 )) ∇f (x0 ) = 31 32 =
3 3 −3 −2
Thus,
! ! !
0 −1 1
x1 = x0 − (∇2 f (x0 ))−1 ∇f (x0 ) = − = .
0 −2 2

Step 2. Set k = 0 + 1 = 1.

Iteration 1.
Step 1.
!
0
∇f (x1 ) = ⇒ k∇f (x1 )k = 0 < ǫ = 0.5
0

22
Stop and declare that a satisfactory approximation solution is
found.
The point x1 = (1, 2)T is not only a satisfactory approximation
minimum but also a stationary point. Since ∇2 f (x) is positive
definite for any x ∈ R2 , it is actually a global minimum for f (x).

Newton’s method usually converges much more rapidly than the

steepest descent method, but sometimes it could fail. Besides,
the calculation of the matrix inverse (∇2 f (x))−1 is hard work.
Hence, there have been several modified methods, which will be
introduced in the advanced subject “Nonlinear Methods in Quan-
titative Management”.

Further reading: Section 11.1–11.7 in the reference book “Operations Research: Ap-
plications and Algorithms” (Winston, 2004)

23