Numerical Optimization

Linear Programming
Shirish Shevade
Computer Science and Automation
Indian Institute of Science
Bangalore 560 012, India.

NPTEL Course on Numerical Optimization

Shirish Shevade

Numerical Optimization

Transportation Problem

min

ij cij xij

s.t.

P3

j=1 xij

P2

ai , i = 1, 2

bj , j = 1, 2, 3
xij 0 i, j

i=1 xij

Shirish Shevade

ai : Capacity of the plant Fi

bj : Demand of the outlet Rj
cij : Cost of shipping one unit
of product from Fi to Rj
xij : Number of units of the
product shipped from Fi to Rj
(variables)
The
P objective is to minimize
ij cij xij
P3
j=1 xij ai , i = 1, 2
(constraints)
P2
i=1 xij bj , j = 1, 2, 3
(constraints)
xij 0 i, j (constraints)
Numerical Optimization

The Diet Problem: Find the most economical diet that satisfies
minimum nutritional requirements.
Number of food items: n
Number of nutritional ingredient: m
Each person must consume at least bj units of nutrient j per
day
Unit cost of food item i: ci
Each unit of food item i contains aji units of the nutrient j
Number of units of food item i consumed: xi
Constraint corresponding to the nutrient j:
aj1 x1 + aj2 x2 + . . . + ajn xn bj , xi 0 i
Cost:
c1 x1 + c2 x2 + . . . + cn xn
Shirish Shevade

Numerical Optimization

Problem:
min
c1 x1 + c2 x2 + . . . + cn xn
s.t. aj1 x1 + aj2 x2 + . . . + ajn xn bj j
xi 0 i
Given: c = (c1 , . . . , cn )T , A = (a1 | . . . |an ), b = (b1 , . . . , bm )T .
Linear Programming Problem (LP):
min
cT x
s.t. Ax b
x0
where A Rmn , c Rn and b Rm .
Assumption: m n, rank(A) = m
Linear Constraints can be of the form Ax = b or Ax b
Shirish Shevade

Numerical Optimization

Constraint (Feasible) Set:

Inequality constraint of the type {x : aT x b} or
{x : aT x b} denotes a half space
Equality constraint, {x : aT x = b}, represents an affine
space
Non-negativity constraint, x 0
Constraint set of an LP is a convex set
Polyhedral Set
X = {x : Ax b, x 0}
Polytope: A bounded polyhedral set

Shirish Shevade

Numerical Optimization

Consider the constraint set in R2 :

{(x1 , x2 ) : x1 + x2 2, x1 1, x1 0, x2 0}

Shirish Shevade

Numerical Optimization

Consider the constraint set in R2 :

{(x1 , x2 ) : x1 1, x2 x1 }

Shirish Shevade

Numerical Optimization

Feasible set can be a singleton set

Feasible Set = {(x1 , x2 ) : x1 + x2 = 2, x1 + x2 = 1} = {A}

Shirish Shevade

Numerical Optimization

Feasible set can be empty!

Feasible Set = {(x1 , x2 ) : x1 + x2 2, x1 + x2 1} =

Shirish Shevade

Numerical Optimization

Definition
Let X be a convex set. A point x X is said to be an extreme
point (corner point or vertex) of X if x cannot be represented
as a strict convex combination of two distinct points in X.

Extreme Points: A, B, C and D.

E is not an extreme point.
Shirish Shevade

Numerical Optimization

Extreme Point: A
Shirish Shevade

Numerical Optimization

Constraint Set:
X = {(x1 , x2 ) : x1 + x2 2, x1 1, x1 0, x2 0}
4 constraints in R2

Two constraints are binding (active) at every extreme point

Fewer than two constraints are binding at other points
Shirish Shevade

Numerical Optimization

Consider the constraint set: X = {x : Ax b, x 0} where

A Rmn and rank(A) = m.
m + n hyperplanes associated with m + n halfspaces
m + n halfspaces define X
An extreme point lies on n linearly independent defining
hyperplanes of X
If X is nonempty, the set of extreme points of X is not
empty and has a finite number of points.
An edge of X is formed by intersection of n 1 linearly
independent hyperplanes
Two extreme points of X are said to be adjacent if the line
segment joining them is an edge of X

Shirish Shevade

Numerical Optimization

For example, B and C are adjacent points

Adjacent extreme points have n 1 common binding
linearly independent hyperplanes
Shirish Shevade

Numerical Optimization

Remarks:
Consider the constraint set: X = {x : Ax = b, x 0} where
A Rmn and rank(A) = m.
Let x X be an extreme point of X
m equality constraints are active at x
Therefore, n m additional hyperplanes (from the
non-negativity constraints) are active at x

Shirish Shevade

Numerical Optimization

Geometric Solution of a LP:

min
cT x
s.t. Ax b
x0
where A Rmn , c Rn and b Rm .

Shirish Shevade

Numerical Optimization

Shirish Shevade

Numerical Optimization

Shirish Shevade

Numerical Optimization

Shirish Shevade

Numerical Optimization

Shirish Shevade

Numerical Optimization

Shirish Shevade

Numerical Optimization

Shirish Shevade

Numerical Optimization

Example:
min
s.t.

2x1 x2
x1 + x2 5
x1 + 2x2 6
x1 , x2 0

Shirish Shevade

Numerical Optimization

Example:
min
s.t.

2x1 x2
x1 + x2 5
x1 + 2x2 6
x1 , x2 0

Shirish Shevade

Numerical Optimization

Example:
min
s.t.

2x1 x2
x1 + x2 5
x1 + 2x2 6
x1 , x2 0

Shirish Shevade

Numerical Optimization

Example:
min
s.t.

2x1 x2
x1 + x2 5
x1 + 2x2 6
x1 , x2 0

Shirish Shevade

Numerical Optimization

Example:
min cT x
s.t. x X

Shirish Shevade

Numerical Optimization

Example:
min cT x
s.t. x X

Shirish Shevade

Numerical Optimization

Example:
min cT x
s.t. x X

Shirish Shevade

Numerical Optimization

Example:
min cT x
s.t. x X

Shirish Shevade

Numerical Optimization

Consider a linear programming problem LP:

min
cT x
s.t. aTi x (, =, ) bi , i = 1, . . . , m
x0
Let X = {x : aTi x (, =, ) bi , i = 1, . . . , m, x 0}.
Remarks:
X is a closed convex set
The set of optimal solutions is a convex set.
The linear program may have no solution or a unique
solution or infinitely many solutions.
If x is an optimal solution to LP, then x must be a
boundary point of X. If z = cT x , then {x : cT x = z} is a
supporting hyperplane to X.
If X is compact and if there is an optimal solution to LP,
then at least one extreme point of X is an optimal solution
to the linear programming problem.
Shirish Shevade

Numerical Optimization

LP in Standard Form:
min
cT x
s.t. Ax = b
x0
where A Rmn and rank(A) = m.
Assumption: Feasible set is non-empty

Shirish Shevade

Numerical Optimization

Any linear program can be converted to the Standard Form.

(a) max cT x = min cT x
(b) Constraint of the type
aT x b, x 0
can be written as
aT x + y = b
x 0
y 0

Shirish Shevade

Numerical Optimization

(c) Constraint of the type

aT x b, x 0
can be written as
aT x z = b
x 0
z 0
(d) Free variables (xi R) can be defined as
xi = xi+ xi , xi+ 0, xi 0

Shirish Shevade

Numerical Optimization

Example:
x1 2x2 3x3
x1 + 2x2 + x3 14
x1 + 2x2 + 4x3 12
x1 x2 + x3 = 2
x1 , x2 unrestricted
x3 3constraints
Write the constraints as equality
min
s.t.

x1 + 2x2 + x3 + x4 = 14, x4 0
x1 + 2x2 + 4x3 x5 = 12, x5 0

Define new variables x1+ , x1 , x2+ , x2 and x30 such that

x1 = x1+ x1 , where x1+ 0, x1 0
x2 = x2+ x2 , where x2+ 0, x2 0
x30 = 3 x3 so that x30 0

Shirish Shevade

Numerical Optimization

Therefore, the program

min
x1 2x2 3x3
s.t. x1 + 2x2 + x3 14
x1 + 2x2 + 4x3 12
x1 x2 + x3 = 2
x1 , x2 unrestricted
x3 3
can be converted to the standard form:
min
s.t.

x1+ x1 2(x2+ x2 ) + 3(3 + x30 )

x1+ x1 + 2(x2+ x2 ) x30 + x4 = 17
x1+ x1 + 2(x2+ x2 ) 4x30 x5 = 24
x1+ x1 x2+ + x2 x30 = 5
x1+ , x1 , x2+ , x2 , x30 , x4 , x5 0
Shirish Shevade

Numerical Optimization

Consider the linear program in standard form (SLP):

min
cT x
s.t. Ax = b
x0
where A Rmn , rank(A) = rank(A|b) = m.
Let B Rmm be formed using m linearly independent
columns of A.
Therefore, the system of equations, Ax = b can be written as,


xB
(B N)
= b.
xN
Letting xN = 0, we get
BxB = b xB = B1 b.

(xB : Basic Variables)

(xB 0)T : Basic solution w.r.t. the basis matrix B

Shirish Shevade

Numerical Optimization

Basic Feasible Solution

If xB 0, then (xB 0)T is called a basic feasible solution of
Ax = b
x0
w.r.t. the basis matrix B.
Theorem
Let X = {x : Ax = b, x 0}. x is an extreme point of X if
and only if x is a basic feasible solution of
Ax = b
x 0.

Shirish Shevade

Numerical Optimization

Proof.
(a) Let x be a basic feasible solution of Ax = b, x 0.
Therefore, x = (x1 , . . . , xm , 0, . . . , 0). Let B = (a1 |a2 | . . . |am )
| {z } | {z }
0

nm

where a1 , . . . , am are linearly independent. So,

x1 a1 + . . . + xm am = b.
Suppose x is not an extreme point of X.
Let y, z X, y 6= z and x = y + (1 )z, 0 < < 1.
Since y, z 0, we have

ym+1 = . . . = yn = 0
y a + . . . + ym am = b
and 1 1
zm+1 = . . . = zn = 0
z1 a1 + . . . + zm am = b
Since a1 , . . . , am are linearly independent, x = y = z, a
contradiction. So, x is an extreme point of X.
Shirish Shevade

Numerical Optimization

Proof.(continued)
(b) Let x be an extreme point of X.
x X Ax = b, x 0.
There exist n linearly independent constraints active at x.
m active constraints associated with Ax = b.
n m active constraints associated with n m
non-negativity constraints
x is the unique solution of Ax = b, xN = 0.
Ax = b BxB + NxN = b xB = B1 b 0
Therefore, x = (xB xN )T is a basic feasible solution.

 
n
Number of basic solutions
m
Enough to search the finite set of vertices of X to get an optimal
Shirish Shevade
Numerical Optimization
solution

Theorem
Let X be non-empty and compact constraint set of a linear
program. Then, an optimal solution to the linear program exists
and it is attained at a vertex of X.
Proof.
Objective function, cT x, of the linear program is continuous
and the constraint set is compact. Therefore, by Weierstrass
Theorem, optimal solution exists.
The set of vertices, {x1 , . . . , xk }, of X is finite.
Therefore, X is the convex hull
P of x1 , . . . , xk .
Hence, for every x X, x = ki=1 i xi where
P
i 0, ki=1 i = 1.
Let z = min
cT xi . Therefore,
x X,
P1ik
Pk for any
k
T
T

z = c x = i=1 i c xi z
i=1 i = z . So, the minimum
value of cT x is attained at a vertex of X.
Shirish Shevade

Numerical Optimization

Consider the constraints:

x1 + x2 2
x1 1
x1 , x2 0

Shirish Shevade

Numerical Optimization

The given constraints

x1 + x2 2
x1 1
x1 , x2 0
can be written in the form, Ax = b, x 0:
=2
x1 +x2 + x3
x1
+x4 = 1
x1 , x2 , x3 ,
x4
0


 
1 1 1 0
2
Let A =
= (a1 |a2 |a3 |a4 ) and b =
.
1 0 0 1
1

Shirish Shevade

Numerical Optimization

1 1 1 0
1 0 0 1

2
1

= (a1 |a2 |a3 |a4 ) and b =

.


1 1
(1) B = (a1 |a2 ) =
1 0
1
T
xB = (x1 x2 ) = B b = (1 1)T and xN = (x3 x4 )T = (0 0)T .
x = (xB xN )T corresponds to the vertex C.
A=

Shirish Shevade

Numerical Optimization

1 1 1 0
1 0 0 1

2
1

= (a1 |a2 |a3 |a4 ) and b =

.


1 1
(2) B = (a1 |a3 ) =
1 0
1
T
xB = (x1 x3 ) = B b = (1 1)T and xN = (x2 x4 )T = (0 0)T .
x = (xB xN )T corresponds to the vertex B.
A=

Shirish Shevade

Numerical Optimization


A=

1 1 1 0
1 0 0 1


= (a1 |a2 |a3 |a4 ) and b =


(3) B = (a1 |a4 ) =

1 0
1 1

2
1


.

xB = (x1 x4 )T = B1 b = (2 1)T and xN = (x2 x3 )T = (0 0)T .

x = (xB xN )T is not a basic feasible point

Shirish Shevade

Numerical Optimization

1 1 1 0
1 0 0 1

2
1

= (a1 |a2 |a3 |a4 ) and b =

.


1 0
(4) B = (a2 |a4 ) =
0 1
1
T
xB = (x2 x4 ) = B b = (2 1)T and xN = (x1 x3 )T = (0 0)T .
x = (xB xN )T corresponds to the vertex D.
A=

Shirish Shevade

Numerical Optimization

1 1 1 0
1 0 0 1

2
1

= (a1 |a2 |a3 |a4 ) and b =

.


1 0
(5) B = (a3 |a4 ) =
0 1
1
T
xB = (x3 x4 ) = B b = (2 1)T and xN = (x1 x2 )T = (0 0)T .
x = (xB xN )T corresponds to the vertex A.
A=

Shirish Shevade

Numerical Optimization

Example:
min 3x1 x2
s.t. x1 + x2 2
x1 1
x1 , x2 0

Shirish Shevade

Numerical Optimization