Karush-Kuhn-Tucker Conditions

Richard Lusby

Department of Management Engineering

Technical University of Denmark
Today’s Topics
(jg

Unconstrained Optimization
Equality Constrained Optimization
Equality/Inequality Constrained Optimization

R Lusby (42111) KKT Conditions 2/40

 Unconstrained Optimization

R Lusby (42111) KKT Conditions 3/40

Unconstrained Optimization
(jg

Problem
minimize f (x)
subject to: x ∈ Rn

First Order Necessary Conditions

If x∗ is a local minimizer of f (x) and f (x) is continuously differentiable in
an open neighbourhood of x∗ , then

∇f (x∗ ) = 0

That is, f (x) is stationary at x∗

R Lusby (42111) KKT Conditions 4/40

Unconstrained Optimization
(jg
Second Order Necessary Conditions
If x∗ is a local minimizer of f (x) and ∇2 f (x) is continuously differentiable
in an open neighbourhood of x∗ , then

∇f (x∗ ) = 0

∇2 f (x∗ ) is positive semi definite

Second Order Sufficient Conditions

Suppose that ∇2 f (x) is continuously differentiable in an open
neighbourhood of x∗ . If the following two conditions are satisfied, then x∗
is a local minimum of f (x).

∇f (x∗ ) = 0

∇2 f (x∗ ) is positive definite

R Lusby (42111) KKT Conditions 5/40
 Equality Constrained Optimization

R Lusby (42111) KKT Conditions 6/40

Equality Constrained Optimization
(jg

Problem
minimize f (x)
subject to: hi (x) = 0 ∀i = 1, 2, . . . m
x ∈ Rn

R Lusby (42111) KKT Conditions 7/40

Equality Constrained Optimization
Consider the following example(jg

Example
minimize 2x12 + x22
subject to: x1 + x2 = 1

Let us first consider the unconstrained case

Differentiate with respect to x1 and x2

∂f (x1 , x2 )
= 4x1
∂x1
∂f (x1 , x2 )
= 2x2
∂x2
These yield the solution x1 = x2 = 0
Does not satisfy the constraint
R Lusby (42111) KKT Conditions 8/40
Equality Constrained Optimization
Example Continued(jg

Let us penalize ourselves for not satisfying the constraint

This gives

L(x1 , x2 , λ1 ) = 2x12 + x22 + λ1 (1 − x1 − x2 )

This is known as the Lagrangian of the problem

Try to adjust the value λ1 so we use just the right amount of resource
λ1 = 0 → get solution x1 = x2 = 0, 1 − x1 − x2 = 1
λ1 = 1 → get solution x1 = 14 , x2 = 21 , 1 − x1 − x2 = 1
4
λ1 = 2 → get solution x1 = 12 , x2 = 1, 1 − x1 − x2 = − 12
4
λ1 = 3 → get solution x1 = 31 , x2 = 23 , 1 − x1 − x2 = 0

R Lusby (42111) KKT Conditions 9/40

Equality Constrained Optimization
Generally Speaking(jg

Given the following Non-Linear Program

Problem
minimize f (x)
subject to: hi (x) = 0 ∀i = 1, 2, . . . m
x ∈ Rn

A solution can be found using the Lagrangian

m
X
L(x, λ) = f (x) + λi (0 − hi (x))
i=1

R Lusby (42111) KKT Conditions 10/40

Equality Constrained Optimization
Why is L(x, λ) interesting?(jg

Assume x∗ minimizes the following

minimize f (x)
subject to: hi (x) = 0 ∀i = 1, 2, . . . m
x ∈ Rn

The following two cases are possible:

1 The vectors ∇h1 (x∗ ), ∇h2 (x∗ ), . . . , ∇hm (x∗ ) are linearly dependent
2 There exists a vector λ∗ such that
∂L(x∗ , λ∗ ) ∂L(x∗ , λ∗ ) ∂L(x∗ , λ∗ ) ∂L(x∗ , λ∗ )
= = =, . . . , = =0
∂x1 ∂x2 ∂x3 ∂xn
∂L(x∗ , λ∗ ) ∂L(x∗ , λ∗ ) ∂L(x∗ , λ∗ ) ∂L(x∗ , λ∗ )
= = =, . . . , = =0
∂λ1 ∂λ2 ∂λ3 ∂λm

R Lusby (42111) KKT Conditions 11/40

Case 1: Example
(jg

Example
minimize x1 + x2 + x32
subject to: x1 = 1
x1 + x22 = 1
2

The minimum is achieved at x1 = 1, x2 = 0, x3 = 0

The Lagrangian is:

L(x1 , x2 , x3 , λ1 , λ2 ) = x1 + x2 + x32 + λ1 (1 − x1 ) + λ2 (1 − x12 − x22 )

Observe that:
∂L(1, 0, 0, λ1 , λ2 )
= 1 ∀λ1 , λ2
∂x2
h i h i
Observe ∇h1 (1, 0, 0) = 1 0 0 and ∇h2 (1, 0, 0) = 2 0 0
R Lusby (42111) KKT Conditions 12/40
Case 2: Example
(jg

Example
minimize 2x12 + x22
subject to: x1 + x2 = 1

The Lagrangian is:

L(x1 , x2 , λ1 ) = 2x12 + x22 + λ1 (1 − x1 − x2 )

Solve for the following:

∂L(x1∗ , x2∗ , λ∗1
) = 4x1∗ − λ∗1 = 0
∂x1
∂L(x1∗ , x2∗ , λ∗1
) = 2x2∗ − λ∗1 = 0
∂x2
∂L(x1∗ , x2∗ , λ∗1 )
= 1 − x1∗ − x2∗ = 0
∂λ
R Lusby (42111) KKT Conditions 13/40
Case 2: Example continued
(jg

Solving this system of equations yields x1∗ = 31 , x2∗ = 32 , λ∗1 = 4

3
Is this a minimum or a maximum?

R Lusby (42111) KKT Conditions 14/40

Graphically
(jg

x2

x1
1

x1 + x2 = 1

R Lusby (42111) KKT Conditions 15/40

Graphically
(jg

x2

1
2 ∇f (x∗ ) = λ∗ ∇h(x∗ )
x2∗ = 3

x1
1
x1∗ = 3 1

x1 + x2 = 1

R Lusby (42111) KKT Conditions 15/40

Geometric Interpretation
(jg

Consider the gradients of f and h at the optimal point

They must point in the same direction, though they may have
different lengths
∇f (x∗ ) = λ∗ ∇h(x∗ )

Along with feasibility of x∗ , is the condition ∇L(x∗ , λ∗ ) = 0

From the example, at x1∗ = 31 , x2∗ = 23 , λ∗1 = 34
h i h i
∇f (x1∗ , x2∗ ) = 4x1∗ 2x2∗ = 4
3
4
3
h i
∇h1 (x1∗ , x2∗ ) = 1 1

R Lusby (42111) KKT Conditions 16/40

Geometric Interpretation
(jg

∇f (x) points in the direction of steepest ascent

−∇f (x) points in the direction of steepest descent
In two dimensions:
I ∇f (xo ) is perpendicular to a level curve of f
I ∇hi (xo ) is perpendicular to the level curve hi (xo ) = 0

R Lusby (42111) KKT Conditions 17/40

 Equality, Inequality Constrained Optimization

R Lusby (42111) KKT Conditions 18/40

Inequality Constraints
What happens if we now include inequality constraints?(jg

General Problem
maximize f (x)
subject to: gi (x) ≤ 0 (µi ) ∀i ∈ I
hj (x) = 0 (λj ) ∀i ∈ J

Given a feasible solution xo , the set of binding constraints is:

I = {i : gi (xo ) = 0}

R Lusby (42111) KKT Conditions 19/40

The Lagrangian
(jg

m
X k
X
L(x, λ, µ) = f (x) + µi (0 − gi (x)) + λj (0 − hj (x))
i=1 j=1

R Lusby (42111) KKT Conditions 20/40

Inequality Constrained Optimization
(jg
Assume x∗ maximizes the following

maximize f (x)
subject to: gi (x) ≤ 0 (µi ) ∀i ∈ I
hj (x) = 0 (λj ) ∀i ∈ J

The following two cases are possible:

1 ∇h1 (x∗ ), . . . , ∇hk (x∗ ), ∇g1 (x∗ ), . . . , ∇gm (x∗ ) are linearly dependent
2 There exist vectors λ∗ and µ∗ such that
k
X m
X
∇f (x∗ ) − λj ∇hj (x∗ ) − µi ∇gi (x∗ ) = 0
j=1 i=1

µ∗i gi (x∗ ) = 0
µ∗ ≥ 0
R Lusby (42111) KKT Conditions 21/40
Inequality Constrained Optimization
(jg

These conditions are known as the Karush-Kuhn-Tucker Conditions

We look for candidate solutions x∗ for which we can find λ∗ and µ∗
Solve these equations using complementary slackness
At optimality some constraints will be binding and some will be slack
Slack constraints will have a corresponding µi of zero
Binding constraints can be treated using the Lagrangian

R Lusby (42111) KKT Conditions 22/40

Constraint qualifications
(jg

KKT constraint qualification

∇gi (xo ) for i ∈ I are linearly independent

Slater constraint qualification

gi (x) for i ∈ I are convex functions
A non boundary point exists: gi (x) < 0 for i ∈ I

R Lusby (42111) KKT Conditions 23/40

Case 1 Example
(jg

The Problem
maximize x
subject to: y ≤ (1 − x)3
y ≥0

Consider the global max: (x, y ) = (1, 0)

After reformulation, the gradients are

∇f (x, y ) = (1, 0)
∇g1 = (3(x − 1)2 , 1)
∇g2 = (0, −1)
P2
Consider ∇f (x, y ) − i=1 µi ∇gi (x, y )

R Lusby (42111) KKT Conditions 24/40

Graphically
(jg

y
1

x
1

y = (1 − x)3

R Lusby (42111) KKT Conditions 25/40

Case 1 Example
(jg

We get: " # " # " #

1 0 0
− µ1 − µ2
0 1 −1

No µ1 and µ2 exist such that:

2
X
∇f (x, y ) − µi ∇gi (x, y ) = 0
i=1

R Lusby (42111) KKT Conditions 26/40

Case 2 Example
(jg

The Problem
maximize −(x − 2)2 − 2(y − 1)2
subject to: x + 4y ≤ 3
x ≥y

The Problem (Rearranged)

maximize −(x − 2)2 − 2(y − 1)2
subject to: x + 4y ≤ 3
−x + y ≤ 0

R Lusby (42111) KKT Conditions 27/40

Case 2 Example
(jg

The Lagrangian is:

L(x1 , y , µ1 , µ2 ) = −(x −2)2 −2(y −1)2 +µ1 (3−x −4y )+µ2 (0+x −y )

This gives the following KKT conditions

∂L
= −2(x − 2) − µ1 + µ2 = 0
∂x
∂L
= −4(y − 1) − 4µ1 − µ2 = 0
∂y
µ1 (3 − x − 4y ) = 0
µ2 (x − y ) = 0
µ1 , µ2 ≥ 0

R Lusby (42111) KKT Conditions 28/40

Case 2 Example
Continued(jg

We have two complementarity conditions → check 4 cases

1 µ1 = µ2 = 0 → x = 2, y = 1
2 µ1 = 0, x − y = 0 → x = 43 , µ2 = − 43
3 3 − x − 4y = 0, µ2 = 0 → x = 53 , y = 13 , µ1 = 2
3
4 3 − x − 4y = 0, x − y = 0 → x = 53 , y = 35 , µ1 = 22
25 , µ2 = − 48
25

Optimal solution is therefore x ∗ = 35 , y ∗ = 13 , f (x ∗ , y ∗ ) = − 49

R Lusby (42111) KKT Conditions 29/40

Case 2 Example
Continued(jg

We have two complementarity conditions → check 4 cases

1 µ1 = µ2 = 0 → x = 2, y = 1
2 µ1 = 0, x − y = 0 → x = 43 , µ2 = − 43
3 3 − x − 4y = 0, µ2 = 0 → x = 53 , y = 13 , µ1 = 2
3
4 3 − x − 4y = 0, x − y = 0 → x = 53 , y = 35 , µ1 = 22
25 , µ2 = − 48
25

Optimal solution is therefore x ∗ = 35 , y ∗ = 13 , f (x ∗ , y ∗ ) = − 49

R Lusby (42111) KKT Conditions 29/40

Continued
(jg

The Problem
minimize (x − 3)2 + (y − 2)2
subject to: x2 + y2 ≤ 5
x + 2y ≤ 4
x, y ≥ 0

The Problem (Rearranged)

maximize −(x − 3)2 − (y − 2)2
subject to: x2 + y2 ≤ 5
x + 2y ≤ 4
−x, −y ≤ 0

R Lusby (42111) KKT Conditions 30/40

Inequality Example
(jg

The gradients are:

∇f (x, y ) = (6 − 2x, 4 − 2y )
∇g1 (x, y ) = (2x, 2y )
∇g2 (x, y ) = (1, 2)
∇g3 (x, y ) = (−1, 0)
∇g4 (x, y ) = (0, −1)

R Lusby (42111) KKT Conditions 31/40

Inequality Example
Continued(jg

Consider the point (x, y ) = (2, 1)

It is feasible I = {1, 2}

This gives " # " # " # " #

2 4 1 0
− µ1 − µ2 =
2 2 2 0

µ1 = 13 , µ2 = 2
3 satisfy this

R Lusby (42111) KKT Conditions 32/40

Sufficient condition
(jg

General Problem
maximize f (x)
subject to: gi (x) ≤ 0 ∀i ∈ I

Theorem
If f (x) is concave and gi (x) for i ∈ I are convex functions then a feasible
KKT point is optimal

An equality constraint is equivalent to two inequality constraints:

hj (x) = 0 ⇔ hj (x) ≤ 0 and − hj (x) ≤ 0

The corresponding two nonnegative multipliers may be combined to

one free one
λj+ ∇h(x) + λj− (−∇h(x)) = λj ∇h(x)
R Lusby (42111) KKT Conditions 33/40
Equality constraints
(jg

General Problem
maximize f (x)
subject to: gi (x) ≤ 0 ∀i ∈ I
hj (x) = 0 ∀j ∈ J

Let xo be a feasible solution

As before, I = {i : gi (xo ) = 0}
Assume constraint qualification holds

R Lusby (42111) KKT Conditions 34/40

Equality constraints
Continued(jg

KKT Necessary Optimality Conditions

If xo is a local maximum, there exist multipliers µi ≥ 0 ∀i ∈ I and λj
∀j ∈ J such that
X X
∇f (xo ) − µi ∇gi (xo ) − λj ∇hj (xo ) = 0
i∈I j

KKT Sufficient Optimality Conditions

If f (x) is concave, gi (x) ∀ i ∈ I are convex functions and hj ∀j ∈ J are
affine (linear) then a feasible KKT point is optimal

R Lusby (42111) KKT Conditions 35/40

KKT Conditions - Summary
(jg

General Problem
maximize f (x)
subject to: gi (x) ≤ 0 ∀i ∈ I
hj (x) = 0 ∀j ∈ J

KKT conditions
∇f (xo ) − µi ∇gi (xo ) − λj ∇hj (xo ) = 0
P P
i j
µi gi (xo ) = 0 ∀i ∈ I
µi ≥ 0 ∀i ∈ I
o
x feasible

R Lusby (42111) KKT Conditions 36/40

Alternative Formulation
Vector Function Form(jg

General Problem
maximize f (x)
subject to: g(x) ≤ 0
h(x) = 0

KKT Conditions
∇f (xo ) − µ∇g(xo ) − λ∇h(xo ) = 0
µg(xo ) = 0
µ ≥0
o
x feasible

R Lusby (42111) KKT Conditions 37/40

Class Exercise 1
(jg

The Problem
maximize ln(x + 1) + y
subject to: 2x + y ≤3
x, y ≥0

R Lusby (42111) KKT Conditions 38/40

Class Exercise 2
(jg

The problem
minimize x 2 + y 2
subject to: x 2 + y 2 ≤ 5
x + 2y = 4
x, y ≥ 0

R Lusby (42111) KKT Conditions 39/40

Class Exercise 3
(jg

Write the KKT conditions for

maximize cT x
subject to: Ax ≤ b
x ≥0

R Lusby (42111) KKT Conditions 40/40