Notes2 PDF

2 The Method of Lagrange Multipliers
A well-known method for solving constrained optimization problems is the method

of Lagrange multipliers. The idea behind this method is to reduce constrained opti-
mization to unconstrained optimization, and to take the (functional) constraints into
account by augmenting the objective function with a weighted sum of them. To this
end, define the Lagrangian associated with (1.1) as
L(x, λ) = f(x) − λT (h(x) − b), (2.1)
where λ ∈ Rm is a vector of Lagrange multipliers.
2.1 Lagrangian Sufficiency

The following result provides a condition under which minimizing the Lagrangian,
subject only to the regional constraints, yields a solution to the original constrained
problem. The result is easy to prove, yet extremely useful in practice.
Theorem 2.1 (Lagrangian Sufficiency Theorem). Let x ∈ X and λ ∈ Rm such that

L(x, λ) = inf x ′ ∈X L(x ′ , λ) and h(x) = b. Then x is an optimal solution of (1.1).
Proof. We have that
min f(x ′ ) = ′min[f(x ′ ) − λT (h(x ′ ) − b)]

x ′ ∈X(b) x ∈X(b)
> min
′
[f(x ′ ) − λT (h(x ′ ) − b)]
x ∈X
= f(x) − λT (h(x) − b) = f(x).
Equality in the first line holds because h(x ′ ) − b = 0 when x ′ ∈ X(b). The inequality
on the second line holds because the minimum is taken over a larger set. In the third
line we finally use that x minimizes L and that h(x) = b.
Two remarks are in order. First, a vector λ of Lagrange multipliers satisfying the
conditions of the theorem is not guaranteed to exist in general, but it does exist for a
large class of problems. Second, the theorem appears to be useful mainly for showing
that a given solution x is optimal. In certain cases, however, it can also be used to find
an optimal solution. Our general strategy in these cases will be to minimize L(x, λ) for
all values of λ, in order to obtain a minimizer x∗ (λ) that depends on λ, and then find
λ∗ such that x∗ (λ∗ ) satisfies the constraints.
5
6 2 · The Method of Lagrange Multipliers
2.2 Using Lagrangian Sufficiency

We begin by applying Theorem 2.1 to a concrete example.
Example 2.2. Assume that we want to
minimize x1 − x2 − 2x3
subject to x1 + x2 + x3 = 5
x21 + x22 = 4.
The Lagrangian of this problem is
L(x, λ) = x1 − x2 − 2x3 − λ1 (x1 + x2 + x3 − 5) − λ2 (x21 + x22 − 4)

= (1 − λ1 )x1 − λ2 x21 + (−1 − λ1 )x2 − λ2 x22 + (−2 − λ1 )x3 + 5λ1 + 4λ2 .
For a given value of λ, we can minimize L(x, λ) by independently minimizing the

terms in x1 , x2 , and x3 , and we will only be interested in values of λ for which the
minimum is finite.
The term (−2 − λ1 )x3 does not have a finite minimum unless λ1 = −2. The terms
in x1 and x2 then have a finite minimum only if λ2 < 0, in which case an optimum
occurs when
∂L
= 1 − λ1 − 2λ2 x1 = 3 − 2λ2 x1 = 0 and
∂x1
∂L
= −1 − λ1 − 2λ2 x2 = 1 − 2λ2 x2 = 0,
∂x2
i.e., when x1 = 3/(2λ2 ) and x2 = 1/(2λ2 ). The optimum is indeed a minimum, because
   
∂2 L ∂2 L
∂x ∂x ∂x1 ∂x2 
−2λ 2 0
HL =  12 1 2
= ,
∂ L ∂ L
∂x2 ∂x1 ∂x2 ∂x2
0 −2λ2
is positive semidefinite when λ2 < 0.

Let Y be the set of values of λ such that L(x, λ) has a finite minimum, i.e.,
Y = {λ ∈ R2 : λ1 = −2, λ2 < 0}.
For every λ ∈ Y, the unique optimum of L(x, λ) occurs at x∗ (λ) = (3/(2λ2 ), 1/(2λ2 ), x3 )T ,
and we need to find λ ∈ Y such that x∗ (λ) is feasible to be able to apply Theorem 2.1.
Therefore,
9 1
x21 + x22 = 2 + 2 = 4
4λ2 4λ2
p
and thus λ2 = − 5/8. We can now use Theorem p 2.1 to conclude
p that the minimization
p has an optimal solution at x1 = −3 2/5, x2 = − 2/5, and x3 = 5 − x1 − x2 =
problem
5 + 4 2/5.
2.3 · Complementary Slackness 7
Let us formalize the strategy we have used to find x and λ satisfying the conditions
of Theorem 2.1 for a more general problem. To
minimize f(x) subject to h(x) 6 b, x ∈ X (2.2)
we proceed as follows:
1. Introduce a vector z of slack variables to obtain the equivalent problem
minimize f(x) subject to h(x) + z = b, x ∈ X, z > 0.
2. Compute the Lagrangian L(x, z, λ) = f(x) − λT (h(x) + z − b).

3. Define the set
Y = {λ ∈ Rm : inf x∈X,z>0 L(x, z, λ) > −∞}.
4. For each λ ∈ Y, minimize L(x, z, λ) subject only to the regional constraints, i.e.,
find x∗ (λ), z∗ (λ) satisfying
L(x∗ (λ), z∗ (λ), λ) = inf x∈X,z>0 L(x, z, λ). (2.3)
5. Find λ∗ ∈ Y such that (x∗ (λ∗ ), z∗ (λ∗ )) is feasible, i.e., such that x∗ (λ∗ ) ∈ X,
z∗ (λ∗ ) > 0, and h(x∗ (λ∗ )) + z∗ (λ∗ ) = b. By Theorem 2.1, x∗ (λ∗ ) is optimal
for (2.2).
2.3 Complementary Slackness

It is worth pointing out a property known as complementary slackness, which follows
directly from (2.3): for every λ ∈ Y and i = 1, . . . , m,
(z∗ (λ))i 6= 0 implies λi = 0 and

λi 6= 0 implies (z∗ (λ))i = 0.
Indeed, if the conditions were violated for some i, then the value of the Lagrangian
could be reduced by reducing (z∗ (λ))i , while maintaining that (z∗ (λ))i > 0. This
would contradict (2.3). Further note that λ ∈ Y requires for each i = 1, . . . , m either
that λi 6 0 or that λi > 0, depending on the sign of bi . In the case where where λi 6 0,
we for example get that
(h(x∗ (λ∗ )))i < bi implies λ∗i = 0 and

λ∗i < 0 implies (h(x∗ (λ∗ )))i = bi .
Slack in the corresponding inequalities (h(x∗ (λ∗ )))i 6 bi and λ∗i 6 0 has to be comple-
mentary, in the sense that it cannot occur simultaneously in both of them.
8 2 · The Method of Lagrange Multipliers
Example 2.3. Consider the problem to
minimize x1 − 3x2
subject to x21 + x22 6 4
x1 + x2 6 2.
By adding slack variables, we obtain the following equivalent problem:
minimize x1 − 3x2
subject to x21 + x22 + z1 = 4
x1 + x2 + z2 = 2
z1 > 0, z2 > 0.
The Lagrangian of this problem is
L(x, z, λ) = x1 − 3x2 − λ1 (x21 + x22 + z1 − 4) − λ2 (x1 + x2 + z2 − 2)

= (1 − λ2 )x1 − λ1 x21 + (−3 − λ2 )x2 − λ1 x22 − λ1 z1 − λ2 z2 + 4λ1 + 2λ2 .
Since z1 > 0 and z2 > 0, the terms −λ1 z1 and −λ2 z2 have a finite minimum only if
λ1 6 0 and λ2 6 0. In addition, the complementary slackness conditions λ1 z1 = 0 and
λ2 z2 = 0 must hold at the optimum.
Minimizing L(x, z, λ) in x1 and x2 yields
∂L
= 1 − λ2 − 2λ1 x1 = 0 and
∂x1
∂L
= −3 − λ2 − 2λ1 x2 = 0,
∂x2
and we indeed obtain a minimum, because
   
∂2 L ∂2 L
∂x1 ∂x1 ∂x1 ∂x2
−2λ1 0
HL =  = 
∂2 L ∂2 L
∂x2 ∂x1 ∂x2 ∂x2
0 −2λ1
is positive semidefinite when λ1 6 0.

Setting λ1 = 0 leads to inconsistent values for λ2 , so we must have λ1 < 0, and,
by complementary slackness, z1 = 0. Also by complementary slackness, there are now
two more cases to consider: the one where λ2 < 0 and z2 = 0, and the one where
λ2 = 0.p The former case pleads to a contradiction, the latter to the unique minimum at
x1 = − 2/5 and x2 = 3 2/5.

Notes2 PDF

Uploaded by

Copyright:

Available Formats

Notes2 PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Notes2 PDF

Uploaded by

Copyright:

Available Formats

2 The Method of Lagrange Multipliers

A well-known method for solving constrained optimization problems is the method

L(x, λ) = f(x) − λT (h(x) − b), (2.1)

where λ ∈ Rm is a vector of Lagrange multipliers.

2.1 Lagrangian Sufficiency

Theorem 2.1 (Lagrangian Sufficiency Theorem). Let x ∈ X and λ ∈ Rm such that

Proof. We have that

min f(x ′ ) = ′min[f(x ′ ) − λT (h(x ′ ) − b)]

= f(x) − λT (h(x) − b) = f(x).

2.2 Using Lagrangian Sufficiency

Example 2.2. Assume that we want to

The Lagrangian of this problem is

L(x, λ) = x1 − x2 − 2x3 − λ1 (x1 + x2 + x3 − 5) − λ2 (x21 + x22 − 4)

For a given value of λ, we can minimize L(x, λ) by independently minimizing the

is positive semidefinite when λ2 < 0.

Y = {λ ∈ R2 : λ1 = −2, λ2 < 0}.

minimize f(x) subject to h(x) 6 b, x ∈ X (2.2)

minimize f(x) subject to h(x) + z = b, x ∈ X, z > 0.

2. Compute the Lagrangian L(x, z, λ) = f(x) − λT (h(x) + z − b).

L(x∗ (λ), z∗ (λ), λ) = inf x∈X,z>0 L(x, z, λ). (2.3)

2.3 Complementary Slackness

(z∗ (λ))i 6= 0 implies λi = 0 and

(h(x∗ (λ∗ )))i < bi implies λ∗i = 0 and

Example 2.3. Consider the problem to

By adding slack variables, we obtain the following equivalent problem:

The Lagrangian of this problem is

L(x, z, λ) = x1 − 3x2 − λ1 (x21 + x22 + z1 − 4) − λ2 (x1 + x2 + z2 − 2)

is positive semidefinite when λ1 6 0.

You might also like