4.

6 General Constrained Problems

Problems with inequality constraints:
𝑔𝑖 𝒙 ≤ 0; 𝑖 = 1 𝑡𝑜 𝑚

Solution via Lagrange multipliers requires active constraints:

ℎ𝑖 𝒙 = 0

If we don’t know before solving a problem which constraints are

active at the optimum point, how do we solve?

Answer: Transform an inequality constraint into an equality

constraint by adding a slack variable (s).

1
Karush-Kuhn-Tucker (KKT) Optimality Conditions
Let x* be a regular point of the feasible set that is a local
minimum for f(x), subject to
ℎ𝑖 𝒙 = 0; 𝑖 = 1 𝑡𝑜 𝑝
𝑔𝑗 𝒙 ≤ 0; 𝑗 = 1 𝑡𝑜 𝑚
Then there exist Lagrange multipliers v* (p-vector) and u* (m-
vector) such that the Lagrangian function is stationary with
respect to each element of x, v, u, and s at the point x*.

1. Lagrangian Function in the Standard Form

𝑝 𝑚

𝐿 𝒙, 𝒗, 𝒖, 𝒔 = 𝑓 𝒙 + ෍ 𝑣𝑖 ℎ𝑖 𝒙 + ෍ 𝑢𝑗 𝑔𝑗 𝒙 + 𝑠𝑗2
𝑖=1 𝑗=1
= 𝑓 𝒙 + 𝑣 ℎ 𝒙 + 𝑢𝑇 𝑔 𝒙 + 𝑠 2
𝑇

(continued…) 2
Karush-Kuhn-Tucker (KKT) Optimality Conditions
2. Gradient Conditions
𝑝 𝑚
𝜕𝐿 𝜕𝑓 ∗
𝜕ℎ𝑖 ∗
𝜕𝑔𝑖
= + ෍ 𝑣𝑖 + ෍ 𝑢𝑗 = 0; 𝑘 = 1 𝑡𝑜 𝑛
𝜕𝑥𝑘 𝜕𝑥𝑘 𝜕𝑥𝑘 𝜕𝑥𝑘
𝑖=1 𝑗=1
𝜕𝐿
=0 → ℎ𝑖 𝒙∗ = 0; 𝑖 = 1 𝑡𝑜 𝑝
𝜕𝑣𝑖
𝜕𝐿
=0 → 𝑔𝑗 𝒙∗ + 𝑠𝑗2 = 0; 𝑗 = 1 𝑡𝑜 𝑚
𝜕𝑢𝑗

3. Feasibility of Inequalities
𝑠𝑗2 ≥ 0 ↔ 𝑔𝑗 ≤ 0; 𝑗 = 1 𝑡𝑜 𝑚
4. Switching Conditions
𝜕𝐿
=0 ↔ 2𝑢𝑗 ∗ 𝑠𝑗 = 0; 𝑗 = 1 𝑡𝑜 𝑚
𝜕𝑠𝑗

(continued…) 3
Karush-Kuhn-Tucker (KKT) Optimality Conditions
5. Non-Negativity of Lagrange Multipliers for Inequalities
𝑢𝑗 ≥ 0; 𝑗 = 1 𝑡𝑜 𝑚
6. Regularity
Gradients of all active constraints must be linearly independent and the
Lagrange multipliers of all constraints must be unique.

4
 Items to Note about KKT Conditions
• KKT conditions are only applicable at regular points
– Recall: a point x* is a regular point of the feasible set S if f(x*) is
differentiable and gradient vectors of all active constraints at x* are
linearly independent (no gradient can be expressed as a linear
combination of the others)
• Points that satisfy the KKT conditions are called KKT points
• Any non-KKT point cannot be a local minimum point (unless it is an
irregular point and the KKT conditions are not applicable)
• KKT points can be unconstrained or constrained
– Unconstrained if there are no equality constraints and all inequality
constraints are inactive
– Constrained otherwise
• If a KKT point is unconstrained, the 2nd order necessary/sufficient
conditions can determine whether it is a minimum, maximum, or
inflection point

(continued…) 5
 Items to Note about KKT Conditions
At a constrained KKT point
• If there are only equality constraints and no active
inequality constraints (u = 0), then the KKT point is
stationary.
– It can be a minimum, maximum, or inflection point.
• If there are some active inequality constraints and
their Lagrange multipliers are strictly positive, then
the KKT point cannot be a local maximum.
– It can be a minimum or inflection point.

(continued…) 6
 Items to Note about KKT Conditions
• The Lagrange multiplier for each constraint depends on the form of the
constraint

For example, the constraints

2𝑥1 + 3𝑥2 ≤ 0
2 𝑥1
+1≤0
3 𝑥2
0.2𝑥1 + 0.3𝑥2 ≤ 0
will have different Lagrange multipliers, but the optimum solution (𝑥1∗ , 𝑥2∗ ) will
be the same.

• The switching conditions have two cases (si = 0 and ui = 0), each of which
can have multiple solutions.
– Each inequality in the problem formulation doubles the number of KKT
solution cases (1 inequality = 2 KKT cases, 2 inequalities = 4 KKT cases, 3
inequalities = 8 cases, etc.)

7
Minimize
𝑓 𝒙 = 𝑥1 2 + 𝑥2 2 − 3𝑥1 𝑥2
Subject to
𝑔 = 𝑥1 2 + 𝑥2 2 − 6 ≤ 0
1. Write equations for the
KKT conditions
2. Solve for candidate
points for each case of
switching conditions

Figure 4.17 Graphical solution for Example 4.31. Local minimum points, A and B (the vectors are not to scale).

8
Introduction to Optimum Design 3e. Copyright © 2012 Elsevier, Inc. All rights reserved.
 5.3: Second Order Necessary Conditions
for General Constrained Problems
Recall: Unconstrained Problems

1st order NC:

𝜕𝑓 𝒙∗
= 0 for 𝑖 = 1 𝑡𝑜 𝑛
𝜕𝑥𝑖

2nd order NC:

𝐻 𝒙∗ positive semidefinite (NC)
or positive definite (SC)

Now: Constrained Problems

Consider feasible changes (d) about active constraints at 𝒙∗

9
 Constraint Tangent Hyperplane

Any feasible moves away from 𝒙∗ must be along 𝒅

and 𝛻𝑔𝑖 𝒙∗ ∙ 𝒅 = 0

So the feasible region around 𝒙∗ is defined by

𝛻ℎ𝑖 𝑇 𝒅 = 0
𝛻𝑔𝑖 𝑇 𝒅 = 0 for all active constraints at 𝒙∗

10
 2nd Order NC for General Constrained Problems
Let x* be a candidate point satisfying the KKT conditions.
Define the Hessian of the Lagrange Function
𝑝 𝑚
∗ ∗
𝛻 2𝐿 = 𝛻2𝑓 + ෍ 𝑣𝑖 𝛻 2ℎ 𝑖 + ෍ 𝑢𝑖 𝛻 2𝑔
𝑖
𝑗=1 𝑗=1
Let there be nonzero feasible directions 𝒅 ≠ 0 satisfying
𝛻ℎ𝑖 𝑇 𝒙∗ 𝒅 = 0 for 𝑖 = 1 𝑡𝑜 p
𝛻𝑔𝑗 𝑇 𝒙∗ 𝒅 = 0 for all active inequalities (where 𝑔𝑗 𝑇 (𝒙∗ ) = 0)
Then for 𝒙∗ to be a local minimum, it must be true that
𝒅𝑇 𝛻 2 𝐿 𝒙 ∗ 𝒅 ≥ 0

𝑄≥0
Note: This is just the Necessary Condition. Does not guarantee
optimality.

11
 Sufficient Conditions for General Constrained Problems
Let x* be a candidate point satisfying the KKT conditions.
Define the Hessian of the Lagrange Function
𝑝 𝑚
∗ ∗
𝛻2𝐿 = 𝛻2𝑓 + ෍ 𝑣𝑖 𝛻2ℎ 𝑖 + ෍ 𝑢𝑖 𝛻 2 𝑔𝑖
𝑗=1 𝑗=1
Let there be nonzero feasible directions 𝒅 ≠ 0 satisfying
𝛻ℎ𝑖 𝑇 𝒙∗ 𝒅 = 0 for 𝑖 = 1 𝑡𝑜 p
𝛻𝑔𝑗 𝑇 𝒙∗ 𝒅 = 0 for all active inequalities with 𝑢𝑖 ∗ > 0
and let
𝛻𝑔𝑗 𝑇 𝒙∗ 𝒅 ≤ 0 for all active inequalities with 𝑢𝑖 ∗ = 0

If Q > 0 then 𝒙∗ is a strict (isolated) local minimum 12

 Notes on NCs and SCs for General Constrained Problems

• If 𝛻 2 𝐿 𝒙∗ is negative definite or negative

semidefinite, the 2nd order NC is violated
– 𝒙∗ cannot be a local minimum
• If 𝛻 2 𝐿 𝒙∗ is positive definite for any 𝒅 ≠ 0,
the SC is satisfied
– No further checks are needed
• If 𝛻 2 𝐿 𝒙∗ is positive semidefinite or
indefinite, we must calculate 𝒅 and check the
SCs

13
Minimize
𝑓 𝒙 = 𝑥1 2 + 𝑥2 2 − 3𝑥1 𝑥2
Subject to
𝑔 = 𝑥1 2 + 𝑥2 2 − 6 ≤ 0

Candidate Points
Point 𝑥1 ∗ 𝑥2 ∗ 𝑢∗

O 0 0 0
1
A 3 3
2 Figure 4.17 Graphical solution for Example 4.31. Local minimum
points, A and B (the vectors are not to scale).
1
B − 3 − 3
2

14
Introduction to Optimum Design 3e. Copyright © 2012 Elsevier, Inc. All rights reserved.