OPTIMIZATION METHODS

01202N

Lectures Notes

Lecture 8

Fall 2024

1
Example 4: Find all local minimum, local maximum, and saddle points of

𝑓(𝑥, 𝑦) = 𝑥 2 + 𝑦 2 − 2𝑥 − 6𝑦 + 14

Solution:

𝑓𝑥 = 0 → 2𝑥 − 2 = 0 → 𝑥 = 1

𝑓𝑦 = 0 → 2𝑦 − 6 = 0 → 𝑦 = 3

We have one critical point (1,3)

𝑓𝑥𝑥 = 2, 𝑓𝑦𝑦 = 2 and 𝑓𝑥𝑦 = 0

2
𝐷(𝑥, 𝑦) = 𝑓𝑥𝑥 𝑓𝑦𝑦 − 𝑓𝑥𝑦 = 4 − 0 = 4 > 0 and 𝑓𝑥𝑥 = 2 > 0

→ The point (1,3) is a local min

Example 5: Find all local minimum, local maximum, and saddle points of

𝑓(𝑥, 𝑦) = 𝑥 4 + 𝑦 4 − 4𝑥𝑦 + 1

Solution:

𝑓𝑥 = 0 → 4𝑥 3 − 4𝑦 = 0 → 𝑥 3 = 𝑦 (1)

𝑓𝑦 = 0 → 4𝑦 3 − 4𝑥 = 0 → 𝑦 3 = 𝑥 (2)

From (1) 𝑥 9 = 𝑦 3 , use (2) 𝑥9 = 𝑥

→ 𝑥 9 − 𝑥 = 0 → 𝑥(𝑥 8 − 1) = 0

→ 𝑥(𝑥 4 − 1)(𝑥 4 + 1) = 0

→ 𝑥(𝑥 2 − 1)(𝑥 2 + 1)(𝑥 4 + 1) = 0

2
→ 𝑥(𝑥 − 1)(𝑥 + 1)(𝑥 2 + 1)(𝑥 4 + 1) = 0

→ 𝑥 = 0, 𝑥 = 1, 𝑥 = −1

Substitute in (2), the critical point are (0,0), (1,1), and (−1, −1)
2
Now, 𝐷(𝑥, 𝑦) = 𝑓𝑥𝑥 𝑓𝑦𝑦 − 𝑓𝑥𝑦

= (12𝑥 2 )(12𝑦 2 ) − (−4)2

• For the critical point (1,1):

2
𝐷(1,1) = 𝑓𝑥𝑥 (1,1)𝑓𝑦𝑦 (1,1) − 𝑓𝑥𝑦 (1,1)

= 12 ∗ 12 − 16 = 144 − 16 > 0 and 𝑓𝑥𝑥 (1,1) = 12 > 0

→ (1,1) is a local min.

• For the critical point (−1, −1)

2
𝐷(−1, −1) = 𝑓𝑥𝑥 (−1, −1)𝑓𝑦𝑦 (−1, −1) − 𝑓𝑥𝑦 (−1, −1)

= 12 ∗ 12 − 16 > 0 and 𝑓𝑥𝑥 (−1, −1) = 12 > 0

→ (−1, −1) is a local min

• For the critical point (0,0)

2
𝐷(0,0) = 𝑓𝑥𝑥 (0,0)𝑓𝑦𝑦 (0,0) − 𝑓𝑥𝑦 (0,0)

= 0 − (−4)2 = −16 < 0, So (1,1) is a saddle point

3
 3) Function of n variables:

Consider the function of 𝑛 variables,

𝑓: 𝑅𝑛 → 𝑅

The gradient vector: If 𝑓 ∈ 𝐶 1 is a function of n variables, then the gradient of 𝑓 is the

vector function ∇𝑓 defined by
𝜕𝑓
𝜕𝑥1
𝜕𝑓
∇𝑓 = 𝜕𝑥2
⋮
𝜕𝑓
(𝜕𝑥𝑛 )

Note: At every point (𝑥0 , 𝑦0 ) in the domain of a differentiable function 𝑓(𝑥, 𝑦), the
gradient of 𝑓 is normal to the level curve through (𝑥0 , 𝑦0 ) (see the following)

The Hessian matrix: 𝐿𝑒𝑡 𝑓 ∈ 𝐶 2 . The matrix

𝑓𝑥1𝑥1 𝑓𝑥1𝑥2 … 𝑓𝑥1𝑥𝑛

𝑓 𝑓𝑥2𝑥2 … 𝑓𝑥2𝑥𝑛
∇2 𝑓 = 𝑥2𝑥1
⋮ ⋮ ⋮
⋯
𝑓 𝑓𝑥𝑛 𝑥2 𝑓𝑥𝑛 𝑥𝑛
( 𝑥𝑛 𝑥1 )
is called the Hessian matrix

Note: The Hessian matrix is symmetric.

4
Definition:

𝑙𝑒𝑡 𝐴 ∈ 𝑅𝑛×𝑛 be a symmetric matrix.

1) 𝐴 is positive definite, if all its eigen values are positive.

2) 𝐴 is positive semi-definite, if all its eigen values are non-negative. i.e, some of its eigen
values are zero and the rest are positive.

3) 𝐴 is negative definite, if all its eigen values are negative.

4) 𝐴 is negative semi-definite, if all its eigen values are non-positive. i.e., some of its
eigen values are zero and the rest are negative.

5) 𝐴 is indefinite matrix, if some of its eigen values are positive and some are negative.

𝐄𝐱𝐚𝐦𝐩𝐥𝐞 𝟏:

Consider the matrix

2 0
𝐴=( ), 𝐴 is symmetric and its eigen values are 2 and 3, then A is positive
0 3
definite.

Example 2: Use Matlab function eig(A) to determine which of the following matrices
are positive definite.

2 1 0 −2
1 −1 1 3
𝐴=[ ]
0 1 2 −1
−2 3 −1 0
2 −1 0 0
−1 2 −1 0
𝐴=[ ]
0 −1 2 −1
0 0 −1 2
3 2 1 0
2 4 1 1
𝐴=[ ]
1 1 5 2
0 1 2 6

5
Definition:

Let 𝐴 be an 𝑛 × 𝑛 symmetric matrix and let 𝐷𝑘 for 𝑘 = 1,2, … , 𝑛 be its leading principal
minors, then

1) 𝐴 is positive definite, if and only if 𝐷𝑘 > 0 for 𝑘 = 1,2, … , 𝑛.

2) 𝐴 is negative definite if and only if (−1)𝑘 𝐷𝑘 > 0 for 𝑘 = 1,2, … , 𝑛.

Example 3: Consider the matrix

−3 2 0
𝐴 = [ 2 −3 0 ]
0 0 −5
𝐷1 = 𝑑𝑒𝑡([−3]) = −3 < 0
−3 2 −3 2
𝐷2 = 𝑑𝑒𝑡 ([ ]) = | |=9−4=5>0
2 −3 2 −3
−3 2 0
𝐷3 = 𝑑𝑒𝑡(𝐴) = | 2 −3 0 | = −25 < 0
0 0 −5
∴ The matrix 𝐴 is negative definite

Example 4: Consider the matrix

2 0 0
𝐴 = (0 5 3)
0 3 7
is positive definite, since

𝐷1 = 𝑑𝑒𝑡([2]) = 2 > 0
2 0 2 0
𝐷2 = 𝑑𝑒𝑡 ([ ]) = | | = 10 > 0
0 5 0 5
2 0 0
𝐷3 = 𝑑𝑒𝑡(𝐴) = |0 5 3| = 2(35 − 9) > 0
0 3 7
∴ The matrix 𝐴 is positive definite

6
Unconstrained optimization:
Consider

𝐦𝐢𝐧𝒏 𝒇(𝒙)
𝒙∈𝑹

1) A point 𝑥 ∗ ∈ 𝑅𝑛 is a critical point if ∇𝑓(𝑥 ∗ ) = 0

2) A point 𝑥 ∗ ∈ 𝑅𝑛 is a minimizer of 𝑓 if ∇𝑓(𝑥 ∗ ) = 0 and ∇2 𝑓(𝑥 ∗ ) is positive definite.

3) A point 𝑥 ∗ ∈ 𝑅𝑛 is a maximizer of 𝑓 if ∇𝑓(𝑥 ∗ ) = 0 and ∇2 𝑓(𝑥 ∗ ) is negative definite.

Example 5:

Find and classify all the critical points of the function:

𝑓(𝑥, 𝑦, 𝑧) = 𝑥 2 + 2𝑦 2 + 6𝑧 2 − 2𝑥𝑦 + 2𝑦𝑧 + 2𝑧𝑥 − 2𝑥 − 2𝑧

Solution:

𝑓𝑥 2𝑥 − 2𝑦 + 2𝑧 − 2
∇𝑓 = (𝑓𝑦 ) = ( −2𝑥 + 4𝑦 + 2𝑧 )
𝑓𝑧 2𝑥 + 2𝑦 + 12𝑧 − 2

The critical point satisfies ∇𝑓 = 0

This implies

𝑥−𝑦+𝑧 =1 (1)

−𝑥 + 2𝑦 + 𝑧 = 0 (2)

𝑥 + 𝑦 + 6𝑧 = 1 (3)

(1) + (2) → 𝑦 + 2𝑧 = 1 (4)

5
(1) − (3) → −2𝑦 − 5𝑧 = 0 → 𝑦 = − 𝑧 (5)
2

5
Substitute in (4) − 𝑧 + 2𝑧 = 1 → 𝑧 = −2
2

From (5) 𝑦=5

Substitute in (1) 𝑥−5−2=1→𝑥 =8

The critical point is (8,5, −2)

7
Now,

𝑓𝑥𝑥 𝑓𝑥𝑦 𝑓𝑥𝑧 2 −2 2

∇2 𝑓 = (𝑓𝑦𝑥 𝑓𝑦𝑦 𝑓𝑦𝑧 ) = (−2 4 2)
𝑓𝑧𝑥 𝑓𝑧𝑦 𝑓𝑧𝑧 2 2 12

𝐷1 = 𝑑𝑒𝑡([2]) = 2 > 0
2 −2
𝐷2 = 𝑑𝑒𝑡 ([ ]) = 8 − 4 = 5 > 0
−2 4
𝐷3 = 𝑑𝑒𝑡(𝐴) = 2(48 − 4) + 2(−24 − 4) + 2(−4 − 8) = 8 > 0

→ ∇2 𝑓 is positive definite

→ the critical point (8,5, −2) is a minimum pt.

Example 6:

Consider the problem

min 2 𝑓(𝑥, 𝑦) = 𝑥 2 − 4𝑥 − 𝑥𝑦 + 𝑦 2 − 𝑦
(𝑥,𝑦)∈𝑅

Prove that the point (3,2) in the only minimizer of the above function.

Solution:

𝜕𝑓
= 2𝑥 − 4 − 𝑦
𝜕𝑥
𝜕𝑓
= −𝑥 + 2𝑦 − 1
𝜕𝑦
𝜕𝑓
= 0 → 2𝑥 − 𝑦 = 4 (1)
𝜕𝑥
𝜕𝑓
= 0 → −𝑥 + 2𝑦 = 1 (2)
𝜕𝑦

Solving (1) & (2) implies that (3,2) is the only critical point.

The Hessian matrix

𝑓𝑥𝑥 𝑓𝑥𝑦 2 −1
∇2 𝑓 = [ ]=[ ]
𝑓𝑦𝑥 𝑓𝑦𝑦 −1 2

8
𝐷1 = 2 > 0
2 −1
𝐷2 = | |=3>0
−1 2
The matrix ∇2 𝑓 is positive definite

→ The point (3,2) is local minimizer.

Example 7:

Consider the function

8
𝑓(𝑥) = 𝑥 4 − 𝑥 3 − 18𝑥 2 + 72𝑥
3

a) Find all critical points.

b) Classify all these points.

Solution:

𝑓 ′ (𝑥) = 4𝑥 3 − 8𝑥 2 − 36𝑥 − 27 = 0

The critical points are 𝑥 = 2, 𝑥 = 3, 𝑥 = −3

Now, using the second derivative test for local extrema, since
𝑓 ′′ (𝑥) = 12𝑥 2 − 16𝑥 − 36, then

𝑓 ′′ (2) = −20 < 0 → 𝑥 = 2 is a max. point

𝑓 ′′ (3) = 24 > 0 → 𝑥 = 3 is a min. point

𝑓 ′′ (−3) = 120 > 0 → 𝑥 = −3 is a min. point