“Optimality Conditions”

College of Engineering & Technology

Mechanical Engineering Department

Prof. Yasser El-Shaer

Eng. Mohamed Helmy
ME 554: Optimum Design
Smart Village – March 2023
 Optimization Methods

Optimality criteria
Search Methods
methods
(direct methods)
(indirect methods)

Constrained Unconstrained Constrained Unconstrained

problem problem problem problem

Optimality Criteria Methods—Optimality criteria are the conditions a function must satisfy at its
minimum point. Optimization methods seeking solutions (perhaps using numerical methods) to
the optimality conditions are often called optimality criteria or indirect methods.

Search Methods—Search (direct) methods are based on a different philosophy. Here we start
with an estimate of the optimum design for the problem. Usually, the starting design will not
satisfy the optimality criteria; therefore, it is improved iteratively until they are satisfied. Thus, in
the direct approach we search the design space for optimum points.
2
DEFINITIONS OF GLOBAL AND LOCAL MINIMA

• A global minimum point is the one where there are no other feasible points with better cost
function values.

• A local minimum point is the one where there are no other feasible points “in the vicinity”
with better cost function values.

3
REVIEW OF SOME BASIC CALCULUS CONCEPTS

Gradient Vector: Partial Derivatives of a Function

• Consider a function f(x) of n variables x , x , . . ., x . The partial derivative of the function. The
1 2 n

partial derivative of the function with respect to x at a given point x* is defined as ∂f(x*)/∂x
1 1

with respect to x as ∂f(x*)/∂x and so on.

2 2

4
Geometrically, the gradient vector is normal to the tangent plane at the point x*, as
shown in Figure for a function of three variables. Also, it points in the direction of
maximum increase in the function.

5
REVIEW OF SOME BASIC CALCULUS CONCEPTS

Differentiating the gradient vector once again, we obtain a matrix of second partial derivatives
for the function f(x) called the Hessian matrix.

6
REVIEW OF SOME BASIC CALCULUS CONCEPTS

Differentiating the gradient vector once again, we obtain a matrix of second partial derivatives
for the function f(x) called the Hessian matrix.

7
Stationary Points
• A stationary point of a function f(x) is a point where the derivative of f(x) is equal to 0.
These points are called “stationary” because at these points the function is neither
increasing nor decreasing. Graphically, this corresponds to points on the graph of f(x)
where the tangent to the curve is a horizontal line.

8
9
Stationary Points
• A stationary point is called a turning point if the derivative changes sign (from positive to
negative, or vice versa) at that point. There are two types of turning point:
❑ A local maximum, the largest value of the function in the local region.
❑ A local minimum, the smallest value of the function in the local region.

Note: all turning points are stationary points, but not all stationary points are turning points.

• A point where the derivative of the function is zero but the derivative does not change
sign is known as a point of inflection, or saddle point. These are sometimes referred to
as rising or falling points of inflection, depending on whether the derivative of the
function is positive or negative on either side of the stationary point.

• More about saddle point: https://www.khanacademy.org/math/multivariable-

calculus/applications-of-multivariable-derivatives/optimizing-multivariable-functions-
videos/v/saddle-points
10
1- A function of two variables is defined as:
𝑓 𝑥1 , 𝑥2 = 𝑥12 + 𝑥1 𝑥2 + 𝑥22 + 3𝑥1 − 3𝑥2 + 4
Calculate the gradient vector of the function and determine the stationary
points, if it exists. Calculate a Hessian matrix of the function, and determine if
the stationary point found gives a minimum value of the function f.

∂𝑓 𝑥1 ,𝑥2
∂𝑓𝑥1 2𝑥1 + 𝑥2 + 3
• 𝑐 = ∇𝑓 𝑥1 , 𝑥2 = =
∂𝑓 𝑥1 ,𝑥2 𝑥1 + 2𝑥2 − 3
∂𝑓𝑥2

To find Stationary Points:

• 2𝑥1 + 𝑥2 + 3 = 0
• 𝑥1 + 2𝑥2 − 3 = 0
Solve the equations simultaneously
• 𝑥1 = −3
• 𝑥2 = 3
• Stationary point: (−3,3)
11
Hessian matrix

∂2𝑓 ∂2𝑓
∂𝑥12 ∂𝑥1 ∂𝑥2 2 1
𝐻= =
∂2𝑓 ∂2𝑓 1 2
∂𝑥2 ∂𝑥1 ∂𝑥22
𝟐 × 𝟐 𝑴𝒂𝒕𝒓𝒊𝒙 ∂2𝑓 𝑷𝒐𝒔𝒊𝒕𝒊𝒗𝒆 𝒅𝒆𝒇𝒊𝒏𝒊𝒕𝒆
• Determinant of H = 𝑯 = 𝟐 × 𝟐 − 𝟏 × 𝟏 = 𝟑 >0
∂𝑥12
𝑯 >0
• 𝑳𝒆𝒂𝒅𝒊𝒏𝒈 𝒑𝒓𝒊𝒄𝒊𝒑𝒂𝒍 𝒎𝒊𝒏𝒐𝒓𝒔:
∂2𝑓
2 >0 ∂2𝑓 𝑵𝒆𝒈𝒂𝒕𝒊𝒗𝒆 𝒅𝒆𝒇𝒊𝒏𝒊𝒕𝒆
∂𝑥1 2 <0
∂𝑥1
𝑯 >0 𝑯 >0

𝑯 <0 𝒊𝒏𝒅𝒆𝒇𝒊𝒏𝒊𝒕𝒆

12
Hessian matrix

∂2𝑓 ∂2𝑓
∂𝑥12 ∂𝑥1 ∂𝑥2 2 1
𝐻= =
∂2𝑓 2
∂ 𝑓 1 2
∂𝑥2 ∂𝑥1 ∂𝑥22
𝟐 × 𝟐 𝑴𝒂𝒕𝒓𝒊𝒙 ∂2𝑓 𝑷𝒐𝒔𝒊𝒕𝒊𝒗𝒆 𝒔𝒆𝒎𝒊𝒅𝒆𝒇𝒊𝒏𝒊𝒕𝒆
• Determinant of H = 𝑯 = 𝟐 × 𝟐 − 𝟏 × 𝟏 = 𝟑 ∆𝟏 = 2 ≥ 0
∂𝑥1
∂2𝑓
∆𝟏 = 2 ≥ 0
∂𝑥2
• 𝒑𝒓𝒊𝒄𝒊𝒑𝒂𝒍 𝒎𝒊𝒏𝒐𝒓𝒔:
∂2𝑓 ∆𝟐 = 𝑯 ≥ 0
∆𝟏 = 2 ≥ 0
∂𝑥1 𝑷𝒐𝒔𝒊𝒕𝒊𝒗𝒆 𝒔𝒆𝒎𝒊𝒅𝒆𝒇𝒊𝒏𝒊𝒕𝒆 ∂2𝑓 𝑵𝒆𝒈𝒂𝒕𝒊𝒗𝒆 𝒔𝒆𝒎𝒊𝒅𝒆𝒇𝒊𝒏𝒊𝒕𝒆
2
∂ 𝑓 ∆𝟏 = 2 ≤ 0
∂𝑥1
∆𝟏 = 2 ≥ 0 ∂2𝑓
∂𝑥2 ∆𝟏 = 2 ≤ 0
∂𝑥2

∆𝟐 = 𝑯 ≥ 0
∆𝟐 = 𝑯 ≥ 0

13
3- A function of two variables is defined as:
𝑓 𝑥1 , 𝑥2 = 𝑥13 + 3𝑥1 𝑥2 + 𝑥23
Calculate the gradient vector of the function and determine the stationary
points, if it exists. Calculate a Hessian matrix of the function, and determine if
the stationary point found gives a minimum value of the function f.

∂𝑓 𝑥1 ,𝑥2
∂𝑓𝑥1 3𝑥12 + 3𝑥2
• 𝑐 = ∇𝑓 𝑥1 , 𝑥2 = =
∂𝑓 𝑥1 ,𝑥2 3𝑥1 + 3𝑥22
∂𝑓𝑥2
To find Stationary Points:
• 3𝑥12 + 3𝑥2 = 0 𝑥12 = −𝑥2 • 𝑥12 = −𝑥2
• 3𝑥1 + 3𝑥22 = 0 𝑥22 = −𝑥1 • 𝑥22 = −𝑥1
• 𝑥2 = −𝑥12
• 𝑥14 = −𝑥1 𝑥14 + 𝑥1 = 0 • 𝑥1 = 0 𝑥2 = 0
• 𝑥1 (𝑥13 + 1) = 0
• 𝑥1 = 0 𝑥1 = −1 • 𝑥1 = −1 𝑥22 = 1 𝑥2 = ±1, 𝑏𝑢𝑡 𝑥2 =
− 1, because 𝑥12 can not be negative.
14
• Stationary points: 0,0 𝑎𝑛𝑑 −1, −1
Hessian matrix

∂2𝑓 ∂2𝑓
∂𝑥12 ∂𝑥1 ∂𝑥2 6𝑥1 3
𝐻= =
∂2𝑓 2
∂ 𝑓 3 6𝑥2
∂𝑥2 ∂𝑥1 ∂𝑥22
• 0,0 : 𝟐 × 𝟐 𝑴𝒂𝒕𝒓𝒊𝒙 ∂2𝑓 𝑷𝒐𝒔𝒊𝒕𝒊𝒗𝒆 𝒅𝒆𝒇𝒊𝒏𝒊𝒕𝒆
0 3 >0
• 𝐻= ∂𝑥12
3 0 𝑯 >0
• Determinant of H = 𝑯 = −9
𝑺𝒂𝒅𝒅𝒍𝒆 𝑷𝒐𝒊𝒏𝒕 (𝒊𝒏𝒅𝒆𝒇𝒊𝒏𝒊𝒕𝒆)
∂2𝑓 𝑵𝒆𝒈𝒂𝒕𝒊𝒗𝒆 𝒅𝒆𝒇𝒊𝒏𝒊𝒕𝒆
2 <0
∂𝑥1
𝑯 >0
𝑯 <0
𝑯 <0 𝑺𝒂𝒅𝒅𝒍𝒆 𝑷𝒐𝒊𝒏𝒕

15
• Stationary points: 0,0 𝑎𝑛𝑑 −1, −1
Hessian matrix

∂2𝑓 ∂2𝑓
∂𝑥12 ∂𝑥1 ∂𝑥2 −6 3
𝐻= =
∂2𝑓 ∂2𝑓 3 −6
∂𝑥2 ∂𝑥1 ∂𝑥22
• −1, −1 : 𝟐 × 𝟐 𝑴𝒂𝒕𝒓𝒊𝒙 ∂2𝑓 𝑷𝒐𝒔𝒊𝒕𝒊𝒗𝒆 𝒅𝒆𝒇𝒊𝒏𝒊𝒕𝒆
−6 3 >0
• 𝐻= ∂𝑥12
3 −6 𝑯 >0
• Determinant of H = 𝑯 = 27

∂2𝑓 ∂2𝑓 𝑵𝒆𝒈𝒂𝒕𝒊𝒗𝒆 𝒅𝒆𝒇𝒊𝒏𝒊𝒕𝒆

<0 2 <0
∂𝑥1
∂𝑥12 𝑯 >0
𝑯 >0
𝑯 <0 𝑺𝒂𝒅𝒅𝒍𝒆 𝑷𝒐𝒊𝒏𝒕

16
• Stationary points: 0,0 𝑎𝑛𝑑 −1, −1
Hessian matrix

∂2𝑓 ∂2𝑓
∂𝑥12 ∂𝑥1 ∂𝑥2 −6 3
𝐻= =
∂2𝑓 ∂2𝑓 3 −6
∂𝑥2 ∂𝑥1 ∂𝑥22
∂2𝑓 𝟐 × 𝟐 𝑴𝒂𝒕𝒓𝒊𝒙 ∂2𝑓
∆𝟏 = 2 ≥ 0
𝑷𝒐𝒔𝒊𝒕𝒊𝒗𝒆 𝒔𝒆𝒎𝒊𝒅𝒆𝒇𝒊𝒏𝒊𝒕𝒆
∆𝟏 = 2 < 0 ∂𝑥1
∂𝑥1 ∂2𝑓
∆𝟏 = 2 ≥ 0
∂2𝑓 ∂𝑥2
∆𝟏 = 2 < 0
∂𝑥2 ∆𝟐 = 𝑯 ≥ 0

∂2𝑓 𝑵𝒆𝒈𝒂𝒕𝒊𝒗𝒆 𝒔𝒆𝒎𝒊𝒅𝒆𝒇𝒊𝒏𝒊𝒕𝒆

𝑵𝒆𝒈𝒂𝒕𝒊𝒗𝒆 𝑺𝒆𝒎𝒊𝒅𝒆𝒇𝒊𝒏𝒊𝒕𝒆 ∆𝟏 = 2 ≤ 0
∂𝑥1
∂2𝑓
∆𝟏 = 2 ≤ 0
∂𝑥2

∆𝟐 = 𝑯 ≥ 0

17
2- A function of three variables is defined as:
𝑓 𝑥1 , 𝑥2 , 𝑥3 = 𝑥12 + 2𝑥1 𝑥2 + 2𝑥22 + 𝑥32 − 2𝑥1 + 𝑥2 + 8
Calculate the gradient vector of the function and determine the stationary
points, if it exists. Calculate a Hessian matrix of the function, and determine if
the stationary point found gives a minimum value of the function f.

∂𝑓 𝑥1 ,𝑥2 ,𝑥3
∂𝑓𝑥1 2𝑥1 + 2𝑥2 − 2
∂𝑓 𝑥1 ,𝑥2 ,𝑥3
• 𝑐 = ∇𝑓 𝑥1 , 𝑥2 , 𝑥3 = = 2𝑥1 + 4𝑥2 + 1
∂𝑓𝑥2
2𝑥3
∂𝑓 𝑥1 ,𝑥2 ,𝑥3
∂𝑓𝑥3

18
To find Stationary Points:
• 2𝑥1 + 2𝑥2 − 2 = 0
• 2𝑥1 + 4𝑥2 + 1 = 0
• 2𝑥3 = 0
Solve the equations simultaneously
• 𝑥1 = 2.5
• 𝑥2 = −1.5
• 𝑥3 = 0
• Stationary point: (2.5, −1.5,0)

19
Hessian matrix

∂2𝑓 ∂2𝑓 ∂2𝑓

∂𝑥12 ∂𝑥1 ∂𝑥2 ∂𝑥1 ∂𝑥3
∂2𝑓 ∂2𝑓 ∂2𝑓 2 2 0
𝐻= = 2 4 0
∂𝑥2 ∂𝑥1 ∂𝑥22 ∂𝑥2 ∂𝑥3
0 0 2
∂2𝑓 ∂2𝑓 2
∂ 𝑓
∂𝑥3 ∂𝑥1 ∂𝑥3 ∂𝑥2 ∂𝑥32

20
 3× 𝟑 𝑴𝒂𝒕𝒓𝒊𝒙 ∂2𝑓 𝑳𝒐𝒄𝒂𝒍 𝑴𝒊𝒏𝒊𝒎𝒖𝒎
>0
∂𝑥12
∂2𝑓 ∂2𝑓 ∂2𝑓
∂𝑥12 ∂𝑥1 ∂𝑥2 ∂𝑥1 ∂𝑥3 𝑫 >0
∂2𝑓 ∂2𝑓 ∂2𝑓
𝐻=
∂𝑥2 ∂𝑥1 ∂𝑥22 ∂𝑥2 ∂𝑥3
∂2𝑓 ∂2𝑓 ∂2𝑓 𝑯 >0
∂𝑥3 ∂𝑥1 ∂𝑥3 ∂𝑥2 ∂𝑥32

∂2𝑓 ∂2𝑓 ∂2𝑓 𝑳𝒐𝒄𝒂𝒍 𝑴𝒂𝒙𝒊𝒎𝒖𝒎

2 <0
∂𝑥12 ∂𝑥1 ∂𝑥2 ∂𝑥1
𝐷=
∂2𝑓 ∂2𝑓
𝑫 >0
∂𝑥2 ∂𝑥1 ∂𝑥22
𝑯 <0

𝑫 <0 𝑺𝒂𝒅𝒅𝒍𝒆 𝑷𝒐𝒊𝒏𝒕

21
• Determinant of D = 𝑫 = 𝟐 × 𝟒 − 𝟐 × 𝟐 = 𝟒
• Determinant of H = 𝑯 = 𝟖

• 𝑳𝒆𝒂𝒅𝒊𝒏𝒈 𝒑𝒓𝒊𝒄𝒊𝒑𝒂𝒍 𝒎𝒊𝒏𝒐𝒓𝒔:

∂2𝑓
2 >0
∂𝑥1

𝑫 >0

𝑯 >0

22
 3× 𝟑 𝑴𝒂𝒕𝒓𝒊𝒙 ∂2𝑓 𝑷𝒐𝒔𝒊𝒕𝒊𝒗𝒆 𝒔𝒆𝒎𝒊𝒅𝒆𝒇𝒊𝒏𝒊𝒕𝒆
∆𝟏 = 2 ≥ 0
∂𝑥1
∂2𝑓 ∂2𝑓 ∂2𝑓 ∂2𝑓
∆𝟏 = 2 ≥ 0
∂𝑥12 ∂𝑥1 ∂𝑥2 ∂𝑥1 ∂𝑥3 ∂𝑥2
∂2𝑓 ∂2𝑓 ∂2𝑓 ∂2𝑓
𝐻= ∆𝟏 = 2 ≥ 0
∂𝑥2 ∂𝑥1 ∂𝑥22 ∂𝑥2 ∂𝑥3 ∂𝑥3
∂2𝑓 ∂2𝑓 ∂2𝑓
∆𝟐 = 𝑫 ≥ 0
∂𝑥3 ∂𝑥1 ∂𝑥3 ∂𝑥2 ∂𝑥32 ∆𝟐 = 𝑬 ≥ 0
∆𝟐 = 𝑭 ≥ 0
∂2𝑓 ∂2𝑓
∂𝑥12 ∂𝑥1 ∂𝑥2 ∆𝟑 = 𝑯 ≥ 0
𝐷=
∂2𝑓 ∂2𝑓
∂2𝑓 𝑵𝒆𝒈𝒂𝒕𝒊𝒗𝒆 𝒔𝒆𝒎𝒊𝒅𝒆𝒇𝒊𝒏𝒊𝒕𝒆
∂𝑥2 ∂𝑥1 ∂𝑥22 ∆𝟏 = 2 ≤ 0
∂𝑥1
∂2𝑓
∂2𝑓 ∂ 2𝑓 ∆𝟏 = 2 ≤ 0
∂𝑥2
∂𝑥12 ∂𝑥1 ∂𝑥3
𝐸= ∂2𝑓
∂2𝑓 ∂ 2𝑓 ∆𝟏 = 2 ≤ 0
∂𝑥3
∂𝑥3 ∂𝑥1 ∂𝑥32
∆𝟐 = 𝑫 ≥ 0
∂2𝑓 ∂ 2𝑓 ∆𝟐 = 𝑬 ≥ 0
∂𝑥22 ∂𝑥2 ∂𝑥3 ∆𝟐 = 𝑭 ≥ 0
𝐹=
∂2𝑓 ∂ 2𝑓
∂𝑥3 ∂𝑥2 ∂𝑥32 ∆𝟑 = 𝑯 ≤ 0

23
• Determinant of H = 𝑯 = 𝟖

• Determinant of D = 𝑫 = 𝟐 × 𝟒 − 𝟐 × 𝟐 = 𝟒
• Determinant of E = 𝑬 = 𝟒
• Determinant of F = 𝐅 = 𝟖

∂2𝑓
• ∂𝑥 2 = 𝟐
1
∂𝑓
2
• ∂𝑥 2 = 𝟒
2
∂2𝑓
• ∂𝑥 2 = 𝟐 𝑷𝒐𝒔𝒊𝒕𝒊𝒗𝒆 𝒔𝒆𝒎𝒊𝒅𝒆𝒇𝒊𝒏𝒊𝒕𝒆
3

24
Quadratic Form
• The quadratic form is a special nonlinear function having only second-order terms (either
the square of a variable or the product of two variables).

The Matrix of the Quadratic Form

𝐹 𝑥 = 𝑥 𝑇 𝑃𝑥

There are many matrices associated with the given quadratic form; in fact, there are infinite
such matrices. All of the matrices are asymmetric except one.

𝐹 𝑥 = 𝑥 𝑇 𝐴𝑥

25
 𝑓 𝑥1 , 𝑥2 = 2𝑥12 + 2𝑥1 𝑥2 + 4𝑥1 𝑥3 − 6𝑥22 − 4𝑥2 𝑥3 + 5𝑥32

• 𝐹 𝑥 = 𝑥 𝑇 𝑃𝑥

2 2 4
𝑃 = 0 −6 −4
0 0 5

2 0.5 1
𝑃 = 1.5 −6 −6
3 2 5

2 4 5
𝑃 = −2 −6 4
−1 −8 5
26
 𝑓 𝑥1 , 𝑥2 = 2𝑥12 + 2𝑥1 𝑥2 + 4𝑥1 𝑥3 − 6𝑥22 − 4𝑥2 𝑥3 + 5𝑥32

𝐹 𝑥 = 𝑥 𝑇 𝐴𝑥

2 1 2
A= 1 −6 −2
2 −2 5

27
Determine the form of the following matrix

2 0 0
A= 0 4 0
0 0 3

• 𝐹 𝑥 = 𝑥 𝑇 𝐴𝑥 = 2𝑥12 + 4𝑥22 + 3𝑥32 > 0

Positive Definite

28
Determine the form of the following matrix

−1 1 0
A = 1 −1 0
0 0 −1

• 𝐹 𝑥 = 𝑥 𝑇 𝐴𝑥 = −𝑥12 − 𝑥22 + 2𝑥1 𝑥2 − 𝑥32 > 0

• −𝑥32 − (𝑥1 − 𝑥2 ) 2 ≤ 0

• For all 𝑥 𝑎𝑛𝑑 𝑥 𝑇 𝐴𝑥 = 0 when 𝑥3 = 0, and 𝑥1 = 𝑥2 . The quadratic form is

not negative definite but is semidefinite since it can have a zero value for
nonzero x.

29
Determine the form of the following matrix

1 1 0
A= 1 2 0
0 0 1

• 𝐹 𝑥 = 𝑥 𝑇 𝐴𝑥 = 𝑥12 + 2𝑥22 + 𝑥32 + 2𝑥1 𝑥2 > 0

• 𝑥32 + (𝑥1 + 𝑥2 ) 2 > 0

Semi-Positive Definite

30
Thank you