Optimization Methods

Rafał Zdunek
Linear Programming (4h.)
 Introduction
• Transportation problem (example)
• Linear programming
• Simplex method
• Duality
 Bibliography
• J. Nocedal, S. J. Wright, Numerical Optimization,
Springer, 1999, pp. 360 – 415,
• D. G. Luenberger, Y. Ye, Linear and Nonlinear
Programming, Springer, 2008 (3rd Edition).
pp. 11 – 140,
• A. Stachurski, Wprowadzenie do optymalizacji,
Oficyna Wydawnicza Politechniki Warszawskiej,
2009, pp. 9 – 110.
 Example: transportation problem
Transportation problem: Quantities a1 , a2 ,… , am , respectively, of a certain
product are to be shipped from each of m locations and received in amounts
b1 , b2 ,… , bn , respectively, at each of n destinations. Associated with the shipping
of a unit of product from origin i to destination j is a unit shipping cost cij .
Determine the amounts xij to be shipped between each origin–destination pair
i = 1, 2,… , m ; j = 1, 2,… , n ; so as to satisfy the shipping requirements and minimize
the total cost of transportation.
Example: transportation problem

Schematics of transportation problem

 Example: transportation problem
m n
min ∑∑ cij xij ,
x
i =1 j =1
⎧ m

∑
subject to
⎪ xij = b j , j = 1,… , n
⎪ i =1

⎪ n

⎨ ∑ xij = ai , i = 1,… , m
Transportation balance ⎪ j =1

m n ⎪ x ≥ 0, j = 1,… , n, i = 1,… , m
∑ a = ∑b ⎪ ij
i =1
i
j =1
j
⎩
 Linear programming: definition

Definition: Linear Programming is a type of optimization,

in which the minimized (or maximized) objective function
and the constraints are all in linear form:
n

∑a x
j =1
j j =b
 Linear programming (LP):
model

x
{ }
min c x T
(or max)

where x ∈ℜ , c ∈ℜ
n n

subject to (s.t.) linear (equality or inequality) constraints.

 LP: constraints
There are three kinds of constraints in general:

µ Type 1: ≤ type constraint

a x + a x +…+ a x ≤ b
i1 1 i2 2 in n i

µ Type 2: = type constraint

a x +a x
j1 1 j2 2
+ … + a jn xn = b j

µ Type 3: ≥ type constraint

a x +a x
k1 1 k2 2
+ … + a kn xn ≥ bk
 LP: Standard form
min {c1 x1 + c2 x2 +… cn xn }
x

s.t. ⎧ a11 x1 + a12 x2 + … + a1n xn = b1

⎪ a x + a x +… + a x = b
⎪⎪ 21 1 22 2 2n n 2

⎨
⎪a x + a x + … + a x = b
⎪ m1 1 m 2 2 mn n m

⎪⎩ x1 ≥ 0, x2 ≥ 0, … , xn ≥ 0.
 LP: Standard form
in a compact matrix notation

x
{ }
min c x
T

s.t. Ax = b and x ≥ 0
m×n
where x ∈ℜ , c ∈ℜ , A ∈ℜ
n n

Usually rank ( A) = m < n

 LP: Inequality constraints (I)
min {c1 x1 + c2 x2 +… cn xn }
x

s.t. ⎧ a11 x1 + a12 x2 + … + a1n xn ≤ b1

⎪ a x + a x +… + a x ≤ b
⎪⎪ 21 1 22 2 2n n 2

⎨
⎪a x + a x + … + a x ≤ b
⎪ m1 1 m 2 2 mn n m

⎪⎩ x1 ≥ 0, x2 ≥ 0, … , xn ≥ 0.
 LP: Slack variables
min {c1 x1 + c2 x2 +… cn xn }
x

s.t. ⎧ a11 x1 + a12 x2 + … + a1n xn + y1 = b1

⎪ a x + a x +… + a x + y = b
⎪⎪ 21 1 22 2 2 n n 2 2

⎨
⎪ a x + a x +… + a x + ym = bm
⎪ m1 1 m2 2 mn n

⎪⎩ x1 ≥ 0, x2 ≥ 0, … , xn ≥ 0, y1 ≥ 0,… , y m ≥ 0,
 LP: Inequality constraints (II)
min {c1 x1 + c2 x2 +… cn xn }
x

s.t. ⎧ a11 x1 + a12 x2 + … + a1n xn ≥ b1

⎪ a x + a x +… + a x ≥ b
⎪⎪ 21 1 22 2 2n n 2

⎨
⎪a x + a x + … + a x ≥ b
⎪ m1 1 m 2 2 mn n m

⎪⎩ x1 ≥ 0, x2 ≥ 0, … , xn ≥ 0.
 LP: Surplus variables
min {c1 x1 + c2 x2 +… cn xn }
x

s.t. ⎧ a11 x1 + a12 x2 + … + a1n xn − y1 = b1

⎪ a x + a x +… + a x − y = b
⎪⎪ 21 1 22 2 2 n n 2 2

⎨
⎪ a x + a x +… + a x − ym = bm
⎪ m1 1 m2 2 mn n

⎪⎩ x1 ≥ 0, x2 ≥ 0, … , xn ≥ 0, y1 ≥ 0,… , y m ≥ 0,
 LP: Free variables
If a linear program is given in the standard form except one
or more of unknown variables it is not required to be
nonnegative; the problem can be transformed to a standard
form using substitution or elimination techniques:

• Substitution: If x j is unsigned, then x j := u j − v j , u j , v j ≥ 0,

• Elimination: If x j is unsigned, then eliminate this variable

from the constraints and the objective function.
 LP: Feasible solution
• For the standard form:
x
{ }
min cT x
the solution satisfying the constraints: Ax = b, x ≥ 0
is called a feasible solution.
 LP: Basic variables
Given Ax = b , where A ∈ℜm×n , let B be any nonsingular
m×m submatrix made up of columns of A. Then, if all n − m
components of x not associated with columns of B are set
equal to zero, the solution to the resulting set of equations is
said to be a basic solution to Ax = b with respect to the basis
B. The components of x associated with columns of B are
called basic variables. The others are called non-basic
variables.
The system Bx B = b has the unique solution x B = B −1b
 LP: Basic variables
• Suppose rank ( A) = m , and the first m columns of A are
linearly independent, i.e. the basis matrix B:
B = [a1 ,… , a m ]
is nonsingular.
The basic solution to Ax = b with respect to B is given
by
x = ( x B , 0 ) ∈ℜ
−1
where x B = B b ∈ℜ
n m
 LP: Degenerate variables
• Definition. If one or more of the basic variables in a
basic solution xB equals zero, that solution is said to
be a degenerate basic solution.
• If a basic solution is also a feasible solution, it is
called a basic feasible solution.
• The number of basic feasible solutions is at most:
n!
C =
m

m !( n − m ) !
n
 LP: Example
min {−2 x1 − x2 } ,
x

subject to
⎧ 8
⎪ x1 + 3 x2 ≤ 4,
⎪⎪
⎨ x1 + x2 ≤ 2,
⎪ 2 x1 ≤ 3,
⎪
⎪⎩ x1 ≥ 0, x2 ≥ 0.
LP: Feasible set

Constraint set in ℜ2
⎧ 8
⎪ x1 + 3 x2 ≤ 4,
⎪⎪
⎨ x1 + x2 ≤ 2,
⎪ 2 x1 ≤ 3,
⎪
⎪⎩ x1 ≥ 0, x2 ≥ 0.
LP: Extreme points
Objective function: z = −2 x1 − x2 ,
Slack variables extend the constraint set to ℜ5
⎧ 8
⎪ x1 + x2 + x3 = 4,
3
⎪⎪
⎨ x1 + x2 + x4 = 2,
⎪ 2 x1 + x5 = 3,
⎪
⎪⎩ x1 ≥ 0, x2 ≥ 0, x3 ≥ 0, x4 ≥ 0, x5 ≥ 0.
⎛3 1⎞
Optimal point occurs at the extreme point: ⎜ , ⎟
⎝2 2⎠
 LP: Extreme points
Remark: A basic solution for this system is obtained by
setting any two variables to zero and solving for
the remaining three. Each edge of polytope corresponds
to one variable being zero, and the extreme points are
the points where two variables are zero.
 LP: Basic solutions
• Choosing, e.g. B = [a3 , a 4 , a5 ] = I 3×3 , then B is the
basis, and x = [ 0, 0, 4, 2,3] ∈ ℜ5
T

is a non-degenerate basic solution. It satisfies

the constraints, thus it is a basic feasible
solution. The variables: x3 , x4 , x5 are basic
variables. However, this solution is not an
optimal solution to the LP problem.
LP: Convexity
LP: Unbounded feasible polytope
LP: 3D polytope
Simplex: path
 LP: Simplex method
¾ Simplex method was first propsed by G.B. Dantzig in 1947.

¾ Simple search for all of the basic solutions is not applicable,

m
because the whole number is C n .

¾ Basic idea of simplex: Give a rule to transfer from one extreme point
to another, such that the objective function is decreased. This rule must be
easy to implement.
 Simplex: constraints
a11 x1 + a12 x2 + … + a1n xn = b1
a21 x1 + a22 x2 + … + a2 n xn = b2

am1 x1 + am 2 x2 + … + amn xn = bm ,
where m≤n
 Simplex: canonical form

x1 + y1,m +1 xm +1 + y1,m + 2 xm + 2 + … + y1,n xn = y10

x2 + y2,m +1 xm +1 + y2,m + 2 xm + 2 + … + y2,n xn = y20

xm + ym ,m +1 xm +1 + ym ,m + 2 xm + 2 + … + ym ,n xn = ym 0 .
 Simplex: tableaux
1 0 … 0 y1,m +1 y1,m + 2 … y1n y10
0 1 … 0 y2,m +1 y2,m + 2 … y2 n y20
0 0 … 0 ⋅ ⋅ ⋅ ⋅ ⋅
⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅
⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅
⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅
0 0 … 1 ym ,m +1 ym ,m + 2 … ymn ym 0
 Simplex: pivot equations
⎧ y pj
⎪ yij ← yij − yip , i≠ p
⎪ y pq
⎨ y pj
⎪ y pj ← .
⎪⎩ y pq

y pq - pivot element
 Simplex: example
x1 − x4 + 2 x6 = 5
x2 + x4 + x5 − 2 x6 = −3
x3 + 3 x4 + x5 + 3 x6 = 14
Find the basic variables: x4 , x5 , x6
x1 x2 x3 x4 x5 x6
1 0 0 − 1 0 2 5
0 1 0 1 1 − 2 − 3
0 0 1 3 1 3 14
 Simplex: example
x1 x 2 x3 x4 x5 x6
− 1 0 0 1 0 − 2 − 5
0 1 0 1 1 − 2 − 3
0 0 1 3 1 3 14

x1 x 2 x3 x4 x5 x6
− 1 0 0 1 0 − 2 − 5
1 1 0 0 1 0 2
3 0 1 0 1 9 29
 Simplex: example
x1 x 2 x3 x 4 x5 x6
− 1 0 0 1 0 − 2 − 5
1 1 0 0 1 0 2
2 − 1 1 0 0 9 27

x1 x2 x3 x4 x5 x6
− 5 / 9 − 2 / 9 2 / 9 1 0 0 1
1 1 0 0 1 0 2
2 / 9 − 1 / 9 1 / 9 0 0 1 3

Solution: x 4 = 1, x 5 = 2 , x 6 = 3
 Simplex: tableaux
a1 a 2 am a m +1 a m+2 aj an b
1 0 0 y1,m +1 y1,m + 2 y1 j y1n y10
0 1 ⋅ ⋅ ⋅ ⋅ ⋅ ⋅
⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅
⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅
0 0 ⋅ yi ,m +1 yi , m + 2 yij yin yi 0
⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅
⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅
0 0 1 ym ,m +1 ym ,m + 2 ymj ymn ym 0
0 0 0 rm +1 rm + 2 rj rn − z0
 Simplex: algorithm
Step 0: Form a tableau corresponding to a basic feasible solution. The relative
cost coefficients can be found by row reduction.

Step 1: If each rj ≥ 0, stop – the current basic feasible solution is optimal.

Step 2: Select q such that rq < 0 to determine which nonbasic variable is to

become basic.

Step 3: Calculate the ratios yi0/yiq for yiq> 0, i=1,2,...,m. If no yiq > 0, stop
– the problem is unbounded. Otherwise, select p as the index i corresponding to
the minimum ratio.

Step 4: Pivot on the pq-th element, updating all rows including the last.
Return to Step 1.
 Simplex: example
Maximize 16 x1 + 10 x2 subject to

2 x1 − 2 x2 ≤ 8
4 x1 + 6 x2 ≤ 36
x1 ≥ 0, x2 ≥ 0.
 Simplex: example
a1 a2 a3 a 4 b
2 −2 1 0 8
4 6 0 1 36
rT − 16 − 10 0 0 0 First tableau

1 −1 1/ 2 0 4
1 3 / 2 0 1/ 4 9
1 5/8 0 0 0
 Simplex: example
1 −1 1/ 2 0 4
0 5 / 2 −1/ 2 1/ 4 5
Second tableau 0 13 / 8 − 1 / 2 0 − 4
1 −1 1/ 2 0 4
0 1 −1/ 5 1/10 2
0 1 − 4 /13 0 − 32/13
 Simplex: example
1 0 3 / 10 1 / 10 6
0 1 −1/ 5 1 / 10 2
Final tableau 0 0 − 7 / 65 − 1 / 10 − 58 / 13
1 0 3 / 10 1 / 10 6
−1/5
Negative objective
0 1 1 / 10 2 function

0 0 7/5 13 / 10 58
x1 = 6 , x 2 = 2 , x 3 = 0 , x 4 = 0
Solution:
Simplex algorithm: structure
diagram

TRUE FALSE

STOP
 Simplex: artificial variables
A basic feasible solution to Ax ≤ b, s.t. x ≥ 0 with b ≥ 0
can be found solving the (artificial) minimization problem:

min e yT
x
s.t. Ax + y = b, x ≥ 0, y ≥ 0,

e = [1,… ,1] ∈ℜ
T m
 Simplex: artificial vars. (example)
Find a basic feasible solution to:

2 x1 + x2 + 2 x3 = 4
x1 + x2 + 4 x3 = 5 x1 , x2 , x3 ≥ 0.
x1 x2 x3 x4 x5 b
Initial tableau
2 1 2 1 0 4
1 1 4 0 1 5
cT 0 0 0 1 1 0
 Simplex: artificial vars. (example)
2 1 2 1 0 4
1 1 4 0 1 5
First tableau r T
−3 −2 −6 0 0 −9

3/ 4 1/ 4 0 1/ 2 −1/ 4 3/ 4
1/ 4 1/ 4 1 0 1/ 4 5/ 4
− 1 / 4 − 1 / 12 0 0 1/ 4 −1/ 4
 Simplex: artificial vars. (example)
3 1 0 2 −1 3
1 1 4 0 1 5
Second tableau −1 −1/ 3 0 0 1 −1
1 1/ 3 0 2 / 3 −1/ 3 1
1 1 4 0 1 5
−1 −1/ 3 0 0 1 −1
Simplex: artificial vars. (example)
1 1/ 3 0 2 / 3 −1/ 3 1
0 1/ 6 1 −1/ 6 1/ 3 1
Final tableau 0 0 0 1 1 0

Solution: x 1 = 1, x 2 = 0 , x 3 = 1 .
Simplex: two-phase method
(example 1)
minimize 4 x1 + x2 + x3
subject to 2 x1 + x2 + 2 x3 = 4
x1 , x2 , x3 ≥ 0.
3 x1 + 3 x2 + x3 = 3
x1 x2 x3 b Initial tableau

0 −3/ 4 1 3/ 2
1 5/ 4 0 1/ 2
cT 4 1 1 0
 Simplex: two-phase method
0 −3/ 4 1 3/ 2
1 5/ 4 0 1/ 2
First tableau 0 − 13 / 4 0 − 7 / 2
3/ 5 0 1 9/5
4/5 1 0 2/5
13 / 5 0 0 − 11 / 5 Second tableau

The optimal solution: x1 = 0 , x 2 = 2 / 5 , x 3 = 9 / 5 .

Simplex: two-phase method
(example 2)
minimize − 2 x1 + 4 x2 + 7 x3 + x4 + 5 x5

subject to − x1 + x2 + 2 x3 + x4 + 2 x5 = 7
− x1 + 2 x2 + 3 x3 + x4 + x5 = 6
− x1 + x2 + x3 + 2 x4 + x5 = 4
x1 unsigned, x 2 , x3 , x4 , x5 ≥ 0.
 Simplex: two-phase method
(example 2)
Initial tableau Equivalent problem

x1 x2 x3 x4 x5 b 1 −1 − 2 −1 − 2 − 7
−1 1 2 1 2 7
0 1 1 0 −1 −1
−1 2 3 1 1 6
−1 1 1 2 1 4 0 0 −1 1 −1 − 3
cT −2 4 7 1 5 0 0 2 3 − 1 1 − 14
 Simplex: two-phase method
(example 2)
x 2 x3 x 4 x5 x6 x7 b
−1 −1 0 1 1 0 1
0 1 −1 1 0 1 3
cT 0 0 0 0 1 1 0 Initial tableau for phase I

x 2 x3 x4 x5 x6 x7 b First tableau for phase I

−1 −1 0 1 1 0 1
0 1 −1 1 0 1 3
rT 1 0 1 −2 0 0 −4
 Simplex: two-phase method
(example 2)
x2 x3 x4 x5 x6 x7 b
−1 −1 0 1 1 0 1
1 2 −1 0 −1 1 2
Second tableau for phase I −1 − 2 1 0 2 0 −2

0 1 −1 1 0 1 3
1 2 −1 0 −1 1 2
0 0 0 0 1 1 0 Final tableau for phase I
Simplex: two-phase method
(example 2)
x2 x3 x4 x5 b
0 1 −1 1 3
1 2 −1 0 2
cT 2 3 −1 1 − 14

Initial tableau for phase II

 Simplex: two-phase method
(example 2)
0 1 −1 1 3 First tableau for phase II

1 2 −1 0 2
0 − 2 2 0 − 21
−1/ 2 0 −1/ 2 1 2
1/ 2 1 −1/ 2 0 1
Final tableau for phase II 1 0 1 0 − 19
 Simplex: two-phase method
(example 2)
The solution x3 = 1, x5 = 2 can be insert in the expression for x1 giving:

x1 = −7 + 2 ⋅1 + 2 ⋅ 2 = −1;

thus the final solution is:

x1 = −1, x2 = 0, x3 = 1, x4 = 0, x1 = −2.
 Simplex: matrix form
• Let A = [ B, D] ∈ℜm×n, x = ( x B , x D ) ∈ℜn
(
c = c ,c
T T
B
T
D ) ∈ℜ n
, B ∈ℜm×m - columns of A corresponding
to basic variables

minimize cTB x B + cTD x D

subject to BxB + DxD = b

If x D = 0 : x B = B −1b, hence x = ( x B , 0 )
 Simplex: matrix form
If x D ≠ 0 : x B = B −1b − B −1DxD , (basic variables)

z = cTB (B −1b − B −1DxD ) + cTD x D = cTB B −1b + (cTD − cTB B −1D)x D ,

(objective function)

r DT = c TD − c TB B − 1 D (relative cost vector)

 Simplex: matrix form
⎡A b⎤ ⎡ B D b⎤
Initial tableau:
⎢cT ⎥ =⎢ T ⎥ ,
⎣ 0 ⎦ ⎣c B c T
D 0⎦
Final tableau:
⎡I BD-1
Bb ⎤
-1
T= ⎢ T -1 ⎥
.
⎣0 cD − cBB D − cBB b⎦
T T -1
 Simplex: Revised simplex
The revised form of the simplex method is as follows: Given the inverse B-1 of a current
basis, and the current solution x B = y 0 = B −1b,
Step 1: Calculate the current relative cost coefficients rDT = cTD - cTB B −1D. This can best be
done by first calculating λ T = cTB B −1 and then the relative cost vector rDT = cTD - λ T D.
If rD ≥ 0 stop; the current solution is optional.
Step 2: Determine which vector aq is to enter the basis by selecting the most negative
cost coefficient and calculate y q = B −1a q , which gives the vector aq expressed in terms
of the current basis.

Step 3: If no yiq>0 stop; the problem is unbounded. Otherwise, calculate the ratios yi0/ yiq
for yiq>0 to determine which vector leaves the basis.

Step 4: Update B-1 and the current solution B-1b. Return to Step1.
 Simplex: Revised simplex
(example)
maximize 3 x1 + x2 + 3x3 subject to 2 x1 + x2 + x3 ≤ 2
x1 + 2 x2 + 3 x3 ≤ 5
x1 , x2 , x3 ≥ 0. 2 x1 + 2 x2 + x3 ≤ 6

a1 a 2 a3 a 4 a5 a6 b
2 1 1 1 0 0 2
1 2 3 0 1 0 5 cT = [− 3, − 1,−3,0,0,0].
2 2 1 0 0 1 6
 Simplex: Revised simplex
(example)
Variable B −1 xB

4 1 0 0 2
5 0 1 0 5
6 0 0 1 6
Step 1: λ T = [0,0,0]B −1 = [0,0,0]
rDT = cTD − λ T D = [−3,−1,−3].
 Simplex: Revised simplex
(example)
Variable B −1 xB y2
Steps 2 and 3:
(if a2 is selected 4 1 0 0 2 1
to enter the basis)
5 0 1 0 5 2
6 0 0 1 6 2
Step 4: Variable B −1 xB
λ T = [ − 1,0,0 ]B −1 = [ − 1,0,0 ]
2 1 0 0 2
r1 = − 1, r3 = − 2, r4 = 1 .
5 −2 1 0 1
The a3 is decided to enter the basis.
6 −2 0 1 2
 Simplex: Revised simplex
(example)
Variable B −1 xB y 3
2 1 0 0 2 1
5 −2 1 0 1 1
6 − 2 0 1 2 −1
Variable B −1 xB λ T = [ − 1, − 3,0 ]B −1 = [3, − 2,0 ],
2 3 −1 0 1 r1 = − 7 , r3 = − 3, r4 = 2 .
3 −2 1 0 1
The a1 is decided to enter the basis.
6 −4 1 1 3
 Simplex: Revised simplex
(example)
Variable B −1 xB y1
2 3 −1 0 1 5
3 −2 1 0 1 −3
6 −4 1 1 3 −5
B −1
λ T = [ − 3, − 3,0 ]B −1 = [ − 6 / 5, − 3 / 5,0 ],
Variable xB
1 3 / 5 −1/ 5 0 1/ 5 r2 = 7 / 5, r4 = 6 / 5, r5 = 3 / 5 .
The optimal solution is found:
3 −1/ 5 2 / 5 0 8 / 5 ⎡1 8 ⎤
x=⎢ 0 0 0 4⎥
6 −1 0 1 4 ⎣5 5 ⎦
 Simplex: one-step of simplex
Given β,ℵ, xB=B-1b≥ 0, xN=0;
Solve BTπ = cB for π,
Compute sN= cN - NT π ;
if sN ≥ 0
STOP; (// optimal point found)
Select q ∈ ℵ with sq< 0 as the entering index;
Solve Bt = Aq for t;
if t ≤ 0
STOP; (// problem is unbounde)
Calculate x + = min ( xB )i and use p to denote the index of the basic
B
i|ti > 0 ti
variable for which this minimum is achieved;
Update x B+ = x B − tx q+ , x N+ = ( 0 , … , 0 , x q+ , 0 , … , 0 ) T ;
Change β by adding q and removing p.
 Simplex: LU decomposition
Step 1: Calculate the current solution xB = y0 satisfying By0 = b.

Step 2: Solve λTB = cTB, and set rDT = cTD − λT D. If rD ≥ 0, stop; the current
solution is optimal.

Step 3: Determine which vector aq is to enter the basis by selecting the most
negative relative cost coefficient, and solve Byq = aq.

Step 4: If no yiq> 0, stop; the problem is unbounded. Otherwise, calculate the ratios
yi0/ yiq for yiq> 0 and select the smallest nonnegative one to determine which
vector is to leave the basis.

Step 5: Update B. Return to Step 1.

 Duality: problems
Primal Dual
minimize cT x maximize λTb
subject to Ax ≥ b subject to λ T A ≤ cT
x≥0 λ ≥0

Primal Dual
minimize cT x maximize λTb
subject to Ax = b subject to λ T A ≤ cT
x≥0
 Duality: theorems

Theorem (Weak Duality): If x and λ are feasible for the equality

constrained LP problem, then cT x ≥ λ T b

Duality Theorem for LP: If either of the primal or dual problem has a
finite optimal solution, so does the other, and the corresponding values of
the objective functions are equal. If either problem has an unbounded
objective, the other (problem) has no feasible solution.
 Duality: theorems
• If the equality constrained primal LP
program has an optimal basic feasible
solution corresponding to the basis B, then
the vector λ satisfying λ T = cTB B −1 is an
optimal solution to the respective dual
program. The optimal values of both
problems are equal: λ T b = cTB B −1b = cTB x B
 Duality: example
Primal program:
minimize − x1 − 4 x2 − 3 x3
subject to 2 x1 + 2 x2 + x3 ≤ 4
x1 + 2 x2 + 2 x3 ≤ 3

x1 , x2 , x3 ≥ 0.
 Duality: example
2 2 1 1 0 4
Tableau:
1 2 2 0 1 6
−1 − 4 −3 0 0 0

1 1 1/ 2 1/ 2 0 2
−1 0 1 −1 1 2
3/2 1 0 1 −1/ 2 1 3 0 −1 2 0 8
−1 0 1 −1 1 2
2 0 0 1 1 10

Optimal solution: x 1 = 0 , x 2 = 1, x 3 = 2 .
 Duality: example
minimize 4λ 1 + 6λ 2
Dual program:
subject to 2λ 1 + λ 2 ≤ −1
2λ 1 + 2λ 2 ≤ −4
λ 1 + 2λ 2 ≤ −3
λ 1 , λ 2 ≤ 0.
Optimal solution: λ 1 = −1, λ 2 = −1. (the last raw of the final tableau)

Primal objective function: Dual objective function:

cT x = −1 ⋅ 0 − 4 ⋅1 − 3 ⋅ 2 = −10 λ T b = −1 ⋅ 4 − 1 ⋅ 6 = −10
 Duality: geometrical interpretation
minimize 18 x1 + 12 x2 + 2 x3 + 6 x4
Primal:
subject to 3x1 + x2 − 2 x3 + x4 = 2
x1 + 3x2 − x4 = 2
x1 , x2 , x3 , x4 ≥ 0.
Dual: minimize 2λ 1 + 2λ 2
subject to 3λ 1 + λ 2 ≤ 18
λ 1 + 3λ 2 ≤ 12
- 2λ 1 ≤ 2
λ 1 − λ 2 ≤ 6.
Duality: geometrical interpretation

The primal requirements space The dual in activity space

 Duality: problems
Primal Dual
minimize cT x maximize λTb
Pair (1) subject to Ax ≥ b subject to λ T A ≤ cT
x≥0 λ ≥0

Primal Dual
Pair (2) minimize cT x maximize λTb
subject to Ax = b subject to λ T A ≤ cT
x≥0
 Duality: Complementary slackness

Theorem 1:
(Complementary slackness – asymmetric form). Let x and λ be feasible solutions for
primal and dual programs, respectively, in the pair (2). A necessary and sufficient
condition that they both be optimal solutions is for all i.

xi >0 ⇒ λTai = ci
xi = 0 ⇐ λTai < ci.
 Duality: Complementary slackness
Theorem 2:
(Complementary slackness – symmetric form). Let x and λ be feasible
solutions for the primal and dual programs, respectively, in the pair (1).
A necessary and sufficient condition that they both be optimal solutions
is for all i and j.

xi >0 ⇒ λTai = ci
xi = 0 ⇐ λTai < ci
λi >0 ⇒ ajx = bj
λi = 0 ⇒ ajx > bj
(where aj is the jth row of A).
 Duality: Dual Simplex
Step 1: Given a dual feasible basic solution xB, if xB ≥ 0 the solution is optimal. If xB is
not nonnegative, select an index i such that the ith component of xB, xBi <0.

x B = B −1b,
Step 2: If all yij ≥ 0, j = 1,2,...,n, then the dual has no maximum (λ is feasible for all ε >0).
If yij < 0 for some j, then let

z k − ck ⎧⎪ z j − c j ⎫⎪
ε0 = = min ⎨ : yij < 0⎬ z j = λT a j , λ T = cTB B −1 ,
yik j ⎪⎩ yij ⎪⎭
Step 3: Form a new basis B by replacing ai by ak. Using this basis determine the
corresponding basic dual feasible solution xB and return to Step 1.
 Duality: Dual Simplex (example)
minimize 3 x1 + 4 x2 + 5 x3
subject to x1 + 2 x2 + 3 x3 ≥ 5
2 x1 + 2 x2 + x3 ≥ 6
−1 −2 −3 1 0 −5 x1 , x2 , x3 ≥ 0.
−2 −2 −1 0 1 −6
3 4 5 0 0 0 Initial tableau

∀j : c j − z j ≥ 0, Choosing any xBi < 0, say x5 = −6, ⎧⎪ z j − c j ⎫⎪

min ⎨ : y2 j < 0 ⎬
the pivot element appears in the second row according to j
⎪⎩ y2 j ⎪⎭
 Duality: Dual Simplex (example)
0 −1 −5/2 1 −1/ 2 −2
1 1 1/ 2 0 −1/ 2 3
0 1 7/2 0 3/2 9 Second tableau

0 1 5 / 2 −1 1/ 2 2
1 0 − 2 1 −1 1
Final tableau 0 0 1 1 1 11

The third tableau yields a feasible solution to the primal which must be optimal.
Thus the solution is x1 = 1, x2 = 2, x3 = 0.
 Duality: Primal-Dual algorithm
• Step 1. Given a feasible solution λ 0 to the dual program:
maximize λTb
subject to λ T A ≤ cT .
determine the associated restricted primal according to
the associated restricted primal problem:
minimize 1T y 1T = [1,… ,1] ∈ℜm
subject to Ax + y = b
x, y ≥ 0. xi = 0 for i ∉ P
 Duality: Primal-Dual algorithm
• Given a feasible solution λ to the dual, the subset P is
defined as follows:

{ }
P = i : λ T ai = ci , i = 1,… , n, A = [a1 ,… , a n ]

• If i ∉ P : λ T ai < ci
 Duality: Primal-Dual algorithm
• Step 2. Optimize the associated restricted primal. If the
minimal value of this problem is zero, the corresponding
solution is optimal for the original primal by the Primal–Dual
Optimality Theorem.

Primal- Dual Optimality Theorem. Suppose that λ is feasible for the

dual and that x and y = 0 is feasible and of course optimal) for the
associated restricted primal. Then x and λ are optimal for the original
primal and dual programs, respectively.
 Duality: Primal-Dual algorithm
• Step 3. If the minimal value of the associated restricted primal is
strictly positive, obtain from the final simplex tableau of the
restricted primal, the solution u0 of the associated restricted dual:

maximize uTb
subject to u T a i ≤ 0,
u ≤ 1. i∉ P
 Duality: Primal-Dual algorithm
• If there is no j for which uT0 a j > 0, conclude that the primal
has no feasible solutions. If, on the other hand, for at least one
j: uT0 a j > 0 , define the new dual feasible vector

λ = λ 0 + ε 0u 0 ,
where ck − λ T0 a k ⎧⎪ c j − λ T0 a j T ⎫⎪
ε0 = = min ⎨ T : u 0 a j > 0⎬.
u0 ak ⎪⎩ u 0 a j ⎪⎭
T j

Now go back to Step 1 using this λ.

Duality: Primal-Dual algorithm
(example)
minimize 2 x1 + x2 + 4 x3 Since
subject to x1 + x2 + 2 x3 = 3 ∀i : ci ≥ 0,
2 x1 + x2 + 3 x3 = 5 λ = ( 0, 0 )
x1 , x2 , x3 ≥ 0.
a1 a2 a3 ⋅ ⋅ b
1 1 2 1 0 3
First tableau 2 1 3 0 1 5
−3 − 2 −5 0 0 −8
ci − λ T ai → 2 1 4 ⋅ ⋅ ⋅
 Duality: Primal-Dual algorithm
(example)
Since there are no zeros in the last row, no progress can be made in the associated
restricted primal and hence the original solution x1= x2= x3=0, y1= 3, y2= 5 is optimal
for this λ. The solution u0 to the associated restricted dual is u0 =(1,1), and the
numbers u0Tai, i = 1,2,3 are equal to the first three elements in the third row.
Thus, we compute the three ratios 2 , 1 , 4 from which we find ε 0 = 1 . The new values
3 2 5 2
for the fourth row are now found by adding ε0 times the (first three) elements of the
third row to the fourth row.
a1 a2 a3 ⋅ ⋅ b
Second tableau: 1 1 2 1 0 3
2 1 3 0 1 5
−3 −2 −5 0 0 −8
1/ 2 0 3/ 2 ⋅ ⋅ ⋅
 Duality: Primal-Dual algorithm
(example)
a1 a2 a3 ⋅ ⋅ b
Now we calculate again the ratios
1 1 2 1 0 3 1/2, 3/2, obtaining ε 0 = 0.5 ,
1 0 1 −1 1 2 and adding this multiple of the third
−1 0 −1 2 0 −2 row to the fourth row, we have
the next tableau:
−1/ 2 0 3/2 ⋅ ⋅ ⋅
a1 a2 a3 ⋅ ⋅ b
1 1 2 1 0 3
1 0 1 −1 1 2
−1 0 −1 2 0 − 2
0 0 1 ⋅ ⋅ ⋅
 Duality: Primal-Dual algorithm
(example)
a1 a2 a3 ⋅ ⋅ b
0 1 1 2 −1 1
Final tableau 1 0 1 −1 1 2
0 0 0 1 1 0
0 0 1 ⋅ ⋅ ⋅

Optimal feasible solution: x 1 = 2 , x 2 = 1, x 3 = 0 .

 Duality: redundant equations
The system of linear equalities: Ax = b, x ≥ 0, has
redundant equations if there is a nonzero λ ∈ℜm , satisfying:
λ T A = 0,
λ T b = 0,