http://www.amaderforum.

com

CHAPTER 10. PROBABILITY AND STATISTICS

Linear Programming
The LP formulation and the underlying assumptions

A prototype LP problem:

The general LP formulation

Graphical solution of 2-var LP's

Feasible Regions of Two-Var LP's

The solution space of a single equality constraint

The solution space of a single inequality constraint

Representing the Objective Function in the LP solution space

Graphical solution of the prototype example: a 2-var LP with a unique optimal

solution

2-var LP's with many optimal solutions

Infeasible 2-var LP's

Unbounded 2-var LP's

Generalization to the n-var case: the ``geometry'' of the LP feasible region and the
Fundamental Theorem of Linear Programming

Generalization to the n-var case

Polytope Convexity and Extreme Points

The Fundamental Theorem of Linear Programming

An algebraic characterization of the solution search space: Basic Feasible Solutions

Example:

http://www.amaderforum.com
 http://www.amaderforum.com

LP's in ``standard form''

Basic Feasible Solutions: An algebraic characterization of extreme points for LP's

in ``standard form''

Example:

The Simplex Algorithm

The basic Simplex iteration through an example:

http://www.amaderforum.com
 http://www.amaderforum.com

The LP formulation and the underlying assumptions

A Linear Programming problem is a special case of a Mathematical Programming
problem. From an analytical perspective, a mathematical program tries to identify an
extreme (i.e., minimum or maximum) point of a function , which
furthermore satisfies a set of constraints, e.g., . Linear
programming is the specialization of mathematical programming to the case where both,
function f - to be called the objective function - and the problem constraints are linear.

From an applications perspective, mathematical (and therefore, linear) programming is an

optimization tool, which allows the rationalization of many managerial and/or
technological decisions required by contemporary techno-socio-economic applications.
An important factor for the applicability of the mathematical programming methodology
in various application contexts, is the computational tractability of the resulting analytical
models. Under the advent of modern computing technology, this tractability requirement
translates to the existence of effective and efficient algorithmic procedures able to
provide a systematic and fast solution to these models. For Linear Programming
problems, the Simplex algorithm, discussed later in the text, provides a powerful
computational tool, able to provide fast solutions to very large-scale applications,
sometimes including hundreds of thousands of variables (i.e., decision factors). In fact,
the Simplex algorithm was one of the first Mathematical Programming algorithms to be
developed (George Dantzig, 1947), and its subsequent successful implementation in a
series of applications significantly contributed to the acceptance of the broader field of
Operations Research as a scientific approach to decision making.

As it happens, however, with every modeling effort, the effective application of Linear
Programming requires good understanding of the underlying modeling assumptions, and
a pertinent interpretation of the obtained analytical solutions

http://www.amaderforum.com
 http://www.amaderforum.com

A prototype LP problem:
Consider a company which produces two types of products and . Production of these
products is supported by two workstations

and , with each station visited by both product types. If workstation is dedicated
completely to the production of product type , it can process 40 units per day, while if
it is dedicated to the production of product , it can process 60 units per day. Similarly,
workstation can produce daily 50 units of product and 50 units of product ,
assuming that it is dedicated completely to the production of the corresponding product.
If the company's profit by disposing one unit of product is $200 and that of disposing
one unit of is $400, and assumning that the company can dispose its entire production,
how many units of each product should the company produce on a daily basis to
maximize its profit?

Solution:

First notice that this problem is an optimization problem. Our objective is to maximize the
company's profit, which under the problem assumptions, is equivalent to maximizing the
company's daily profit. Furthermore, we are going to maximize the company profit by
adjusting the levels of the daily production for the two items and . Therefore, these
daily production levels are the control/decision factors, the values of which we are called
to determine. In the analytical formulation of the problem, the role of these factors is
captured by modeling them as the problem decision variables:

• := number of units of product to be produced daily

• := number of units of product to be produced daily

In the light of the above discussion, the problem objective can be expressed analytically
as:

Equation1 will be called the objective function of the problem, and the coefficients 200
and 400 which multiply the decision variables in it, will be called the objective function
coefficients.

Furthermore, any decision regarding the daily production levels for items and in
order to be realizable in the company's operation context must observe the production
capacity of the two worksations and . Hence, our next step in the problem

http://www.amaderforum.com
 http://www.amaderforum.com

formulation seeks to introduce these technological constraints in it. Let's focus first on
the constraint which expresses the finite production capacity of workstation .
Regarding this constraint, we know that one day's work dedicated to the production of
item can result in 40 units of that item, while the same period dedicated to the
production of item will provide 60 units of it. Assuming that production of one unit of
product type , requires a constant amount of processing time at
workstation , it follows that: and . Under the further assumption that
the combined production of both items has no side-effects, i.e., does not impose any
additional requirements for production capacity of workstation (e.g., zero set-up
times), the total capacity (in terms of time length) required for producing units of
product and units of product is equal to . Hence, the technological
constraint imposing the condition that our total daily processing requirements for
workstation should not exceed its production capacity, is analytically expressed by:

Notice that in Equation2 time is measured in days.

Following the same line of reasoning (and under similar assumptions), the constraint
expressing the finite processing capacity of workstation is given by:

Constraints2 and3 are known as the technological constraints of the problem. In

particular, the coefficients of the variables in them, , are

known as the technological coefficients of the problem formulation, while the values on
the right-hand-side of the two inequalities define the right-hand side (rhs) vector of the
constraints.

Finally, to the above constraints we must add the requirement that any permissible value
for variables must be nonnegative, i.e.,

since these values express production levels. These constraints are known as the variable
sign restrictions.

Combining Equations1 to4, the analytical formulation of our problem is as follows:

http://www.amaderforum.com
 http://www.amaderforum.com

s.t.

http://www.amaderforum.com
 http://www.amaderforum.com

The general LP formulation

Generalizing formulation5, the general form for a Linear Programming problem is as
follows:

Objective Function:

s.t.

Technological Constraints:

Sign Restrictions:

where ``urs'' implies unrestricted in sign.

The formulation of Equations6 to8 has the general structure of a mathematical

programming problem, presented in the introduction of this section, but it is further
characterized by the fact that the functions involved in the problem objective and the left-
hand-side of the technological constraints are linear. It is the assumptions implied by
linearity that to a large extent determine the applicability of the above model in real-
world applications.

To provide a better feeling of the linearity concept, let us assume that the different
decision variables correspond to various activities from which any solution
will be eventually synthesized, and the values assigned to the variables by any given
solution indicate the activity level in the considered plan(s). For instance, in the above
example, the two activities are the production of items and , while the activity levels
correspond to the daily production volume. Furthermore, let us assume that each
technological constraint of Equation7 imposes some restriction on the consumption of a
particular resource. Referring back to the prototype example, the two problem resources
are the daily production capacity of the two workstations and . Under this
interpretation, the linearity property implies that:

http://www.amaderforum.com
 http://www.amaderforum.com

Additivity assumption:
the total consumption of each resource, as well as the overall objective value are
the aggregates of the resource consumptions and the contributions to the problem
objective, resulting by carrying out each activity independently, and

Proportionality assumption:
these consumptions and contributions for each activity are proportional to the
actual activity level.
It is interesting to notice how the above statement reflects to the logic that was applied
when we derived the technological constraints of the prototype example: (i) Our
assumption that the processing of each unit of product at every station requires a constant
amount of time establishes the proportionality property for our model. (ii) The
assumption that the total processing time required at every station to meet the production
levels of both products is the aggregate of the processing times required for each product
if the corresponding activity took place independently, implies that our system has an
additive behavior. It is also interesting to see how the linearity assumption restricts the
modeling capabilities of the LP framework: As an example, in the LP paradigm, we
cannot immediately model effects like economies of scale in the problem cost structure,
and/or situations in which resource consumption by one activity depends on the
corresponding consumption by another complementary activity. In some cases, one can
approach these more complicated problems by applying some linearization scheme. The
resulting approximations for many of these cases have been reported to be quite
satisfactory.

Another approximating element in many real-life LP applications results from the so

called divisibility assumption. This assumption refers to the fact that for LP theory and
algortihms to work, the problem variables must be real. However, in many LP
formulations, meaningful values for the levels of the activities involved can be only
integer. This is, for instance, the case with the production of items and in our
prototype example. Introducing integrality requirements for some of the variables in an
LP formulation turns the problem to one belonging in the class of (Mixed) Integer
Programming (MIP). The complexity of a MIP problem is much higher than that of LP's.
Actually, the general IP formulation has be shown to belong to the notorious class of NP-
complete problems. (This is a class of problems that have been ``formally'' shown to be
extremely ``hard'' computationally). Given the increased difficulty of solving IP
problems, sometimes in practice, near optimal solutions are obtained by solving the LP
formulation resulting by relaxing the integrality requirements - known as the LP
relaxation of the corresponding IP - and (judiciously) rounding off the fractional values
for the integral variables in the optimal solution. Such an approach can be more easily
justified in cases where the typical values for the integral variables are in the order of tens
or above, since the errors introduced by the rounding-off are rather small, in a relative
sense.

We conclude our discussion on the general LP formulation, by formally defining the

solution search space and optimality. Specifically, we shall define as the feasible region

http://www.amaderforum.com
 http://www.amaderforum.com

of the LP of Equations6 to8, the entire set of vectors that satisfy

the technological constraints of Eq.7 and the sign restrictions of Eq.8. An optimal
solution to the problem is any feasible vector that further satisfies the optimality
requirement expressed by Eq.6. In the next section, we provide a geometric
characterization of the feasible region and the optimality condition, for the special case of
LP's having only two decision variables.

http://www.amaderforum.com
 http://www.amaderforum.com

Graphical solution of 2-var LP's

In this section, we develop a solution approach for LP problems, which is based on a
geometrical representation of the feasible region and the objective function. In particular,
the space to be considered is the n-dimensional space with each dimension defined by
one of the LP variables

The objective function will be described in this n-dim space by its contour plots, i.e., the
sets of points that correspond to the same objective value. To the extent that the proposed
approach requires the visualization of the underlying geometry, it is applicable only for
LP's with upto three variables. Actually, to facilitate the visualization of the concepts
involved, in this section we shall restrict ourselves to the two-dimensional case, i.e., to
LP's with two decision variables. In the next section, we shall generalize the geometry
introduced here for the 2-var case, to the case of LP's with n decision variables, providing
more analytic (algebraic) characterizations of these concepts and properties.

http://www.amaderforum.com
 http://www.amaderforum.com

Feasible Regions of Two-Var LP's

The primary idea behind the geometrical representation adopted in the subsequent
analysis, is to correspond every vector denoting the variables of a 2-var LP,
to the point with co-ordinates

in a 2-dim (planar) Cartesian system. Under this correspondence, the feasible

region of a 2-var LP is depicted by the set of points the coordinates of which satisfy the
LP constraints and the sign restrictions. Since all these constraints are expressed by
linear inequalities, to geometrically characterize the feasible region, we must first
characterize the set of points that constitute the solution space of a linear inequality.
Then, the LP feaslible region will result from the intersection of the solution spaces
corresponding to each technological constraint and/or sign restriction.

http://www.amaderforum.com
 http://www.amaderforum.com

The solution space of a single equality constraint

We start our investigation regarding the geometrical representation of 2-var linear
constraints by considering first constraints of the equality type, i.e.,

It is a well-known result that, assuming , this equation corresponds to a straight

line with slope and intercept

. In the special case where , the solution space (locus) of Equation9 is still a
straight line perpendicular to the -axis, intersecting it at the point . Notice that
the presence of an equality constraint restricts the dimensionality of the feasible solution
space by one degree of freedom, i.e., it turns it from a planar area to a line segment.

http://www.amaderforum.com
 http://www.amaderforum.com

The solution space of a single inequality constraint

Consider the constraint:

The solution space of this constraint is one of the closed half-planes defined by the
equation: . To show this, let us consider a point which
satsifies Equation10 as equality, and another point for which Equation10 is also
valid. For any such pair of points, it holds that:

Interpreting the left side of Eq.11 as the inner (dot) product of the two vectors
and , and recognizing that
, it follows that line , itself, can be
defined by point and the set of points such that vector is at right
angles with vector . Furthermore, the set of points that satisfy the > (<) part of
Equation11 have the vector forming an acute (obtuse) angle with vector , and
therefore, they are ``above'' (``below'') the line. Hence, the set of points satisfying each of
the two inequalities implied by Equation10 is given by one of the two half-planes the
boundary of which is defined by the corresponding equality constraint. Figure1
summarizes the above discussion.

http://www.amaderforum.com
 http://www.amaderforum.com

Figure 1: Half-planes: the feasible region of a linear inequality

An easy way to determine the half-plane depicting the solution space of a linear
inequality, is to draw the line depicting the solution space of the corresponding equality
constraint, and then test whether the point (0,0) satisfies the inequality. In case of a
positive answer, the solution space is the half-space containing the origin, otherwise, it is
the other one.

From the above discussion, it follows that the feasible region for the prototype LP of
Equation5 is the shaded area in the following figure:

Figure 2: The feasible region of the prototype example LP

http://www.amaderforum.com
 http://www.amaderforum.com

Representing the Objective Function in the LP solution

space
The most typical way to represent a two-variable function is to perceive it as a
surface in an (orthogonal) three-dimensional space, where two of the dimensions
correspond to the independent variables and , while the third dimension provides
the function value for any pair . However, in the context of our discussion, we
are interested in expressing the information contained in the two-var LP objective
function

in the Cartesian plane defined by the two independent variables and . For this
purpose, we shall use the concept of contour plots. Contour plots depict a function by
identifying the set of points that correspond to a constant value of the function
, for any given range of 's. The plot obtained for any fixed value of is a
contour of the function. Studying the structure of a contour is expected to identify some
patterns that essentially depict some useful properties of the function.

In the case of LP's, the linearity of the objective function implies that any contour of it
will be of the type:

i.e., a straight line. For a maximization (minimization) problem, this line will be called an
isoprofit (isocost) line. Assuming that (o.w., work with ), Equation12 can be
rewritten as:

which implies that by changing the value of , the resulting isoprofit/isocost lines have
constant slope and varying intercept, i.e, they are parallel to each other (which makes
sense, since by the definition of this cocnept, isoprofit/isocost lines cannot intersect).
Hence, if we continously increase from some initial value , the corresponding
isoprofit lines can be obtained by ``sliding'' the isprofit line corresponding to
parallel to itself, in the direction of increasing or decreasing intercepts,
depending on whether is positive or negative.

http://www.amaderforum.com
 http://www.amaderforum.com

Graphical solution of the prototype example: a 2-var LP

with a unique optimal solution
The `` sliding motion'' described above suggests a way for identifying the optimal values
for, let's say, a max LP problem. The underlying idea is to keep ``sliding'' the isoprofit
line in the direction of increasing 's, until we cross the boundary of
the LP feasible region. The implementation of this idea on the prototype LP of Equation5
is depicted in Figure3.

Figure 3: Graphical solution of the prototype example LP

From this figure, it follows that the optimal daily production levels for the protoype LP
are given by the coordinates of the point corresponding to the intersection of line

with the -axis, i.e., . The maximal daily profit

is . Notice that the optimal point is one

of the ``corner'' points of the feasible region depicted in Figure3. Can you argue that for
the geometry of the feasible region for 2-var LP's described above, if there is a bounded
optimal solution, then there will be one which corresponds to one of the corner points?
(This argument is developed for the broader context of n-var LP's in the next section.)

http://www.amaderforum.com
 http://www.amaderforum.com

2-var LP's with many optimal solutions

Consider our prototype example with the unit profit of item being $600 instead of
$200. Under this modification, the problem isoprofit lines become:

and they are parallel to the line corresponding to the first problem constraint:

Therefore, if we try to apply the optimizing technique of the previous paragraph in this
case, we get the situation depicted below (Figure4), i.e., every point in the line segment
CD is an optimal point, providing the optimal objective value of $24,000.

Figure 4: An LP with many optimal solutions

It is worth-noticing that even in this case of many optimal solutions, we have two of them
corresponding to ``corner'' points of the feasible region, namely points C and D.

http://www.amaderforum.com
 http://www.amaderforum.com

Infeasible 2-var LP's

Consider again the original prototype example, modified by the additional requirements
(imposed by the company's marketing department) that the daily production of product
must be at least 30 units, and that of product should exceed 20 units. These
requirements introduce two new constraints into the problem formulation, i.e.,

Attempting to plot the feasible region for this new problem, we get Figure5, which
indicates that there are no points on the -plane that satisfy all constraints, and
therefore our problem is infeasible (over-constrained).

Figure 5: An infeasible LP

http://www.amaderforum.com
 http://www.amaderforum.com

Unbounded 2-var LP's

In the LP's considered above, the feasible region (if not empty) was a bounded area of the
-plane. For this kind of problems it is obvious that all values of the LP objective
function (and therefore the optimal) are bounded. Consider however the following LP:

s.t.

The feasible region and the direction of improvement for the isoprofit lines for this
problem are given in Figure6

Figure 6: An unbounded LP

http://www.amaderforum.com
 http://www.amaderforum.com

It is easy to see that the feasible region of this problem is unbounded, and furthermore,
the orientation of the isoprofit lines is such that no matter how far we ``slide'' these lines
in the direction of increasing the objective function, they will always share some points
with the feasible region. Therefore, this is an example of a (2-var) LP whose objective
function can take arbitrarily large values. Such an LP is characterized as unbounded.
Notice, however, that even though an unbounded feasible region is a necessary condition
for an LP to be unbounded, it is not sufficient; to convince yourself, try to graphically
identify the optimal solution for the above LP in the case that the objective function is
changed to: .

Summarizing the above discussion, we have shown that a 2-var LP can either

• have a unique optimal solution which corresponds to a ``corner'' point of the

feasible region, or
• have many optimal solutions that correspond to an entire ``edge'' of the feasible
region, or
• be unbounded, or
• be infeasible.

In the next section, we generalize this geometrical description of the LP solution space
for the n-var LP case, and we provide a brief (informal) derivation of the Fundamental
Theorem of Linear Programming. The latter states that if an LP has a bounded optimal
solution, then it must have one which is an extreme point of its feasible region. The
Simplex algorithm to be discussed in the last section of this set of notes essentially
exploits this fundamental result to reduce the space to be searched for an optimal
solution.

Finally, you can actively try the graphical approach to the solution of 2-var LP's
discussed above, by using the code developed by the group of Professors Ken Goldberg
and Ilan Adler, at the University of Berkeley, Dept. of Industrial Engineering and
Operations Research. Notice that the problem formulation uses a "minimization"
objective, and all variables are considered to be "urs". Therefore, (i) any maximization
problem must be turned into a minimization one (can you think of an easy way?), and (ii)
all sign restrictions must be introduced in the formulation explicitly. Other than that, the
software is self-explanatory.

http://www.amaderforum.com
 http://www.amaderforum.com

Generalization to the n-var case: the ``geometry'' of the

LP feasible region and the Fundamental Theorem of
Linear Programming
In this section we generalize the geometry of the 2-var LP's, presented in the previous
section, to LP's with n decision variables. The study of the ``geometric'' properties of the
LP feasible region for this general case, will eventually lead to the Fundamental Theorem
of Linear Programming, which is at the basis of the Simplex algorithm.

http://www.amaderforum.com
 http://www.amaderforum.com

The geometry of n-var LP's

n-var LP's require an n-dimensional space to ``geometrically'' represent a vector

corresponding to a pricing of their decision variables. However, the concepts and
techniques that allowed the geometric representation of the 2-var LP's in the previous
section, generalize quite straightforwardly in this more complicated case.

Hence, given a linear constraint:

and a point satisfying Equation14 as equality, we can

perceive the solution space of

• Equation
as the set of points for which the vector
is at right angles with vector
,
• Inequality as the set of points for which the
vector
forms an acute angle with vector
, and
• Inequality
as the set of points for which the vector
forms an obtuse angle with vector
.

From the above description, it follows that the solution space of the equation

, i.e., the 3-dim case, is a plane perpendicular to vector , which is

characterized as the plane normal. Since this normality concept carries over to the more
general n-dim case, we characterize the solution space of an n-var linear equation as a
hyperplane. Furthermore, similar to the 2-var LP case, a hyperplane defined by the
equation
, divides the n-dim space into two half-spaces: one of them is the solution space of the
inequality

http://www.amaderforum.com
 http://www.amaderforum.com

, and the other one is the solution space of the inequality

Hence, the solution space (feasible region) of an n-var LP is geometrically defined by the
intersection of a number of half-spaces and/or hyperplanes equal to the LP constraints,
including the sign restrictions. Such a set is characterized as a polytope. In particular, a
bounded polytope is called a polyhedron.

Two other concepts of interest in the following discussion are those of the straight line
and the line segment. Given two points
and
, the straight line passing through them is algebraicaly defined by the set of points

As it is seen in the figure above, the line segment between points and is analytically
defined by:

Notice that Equation16 can be rewritten as:

or

We say that Equations17 and18 define a convex combination of points and .

http://www.amaderforum.com
 http://www.amaderforum.com

Polytope Convexity and Extreme Points

In this paragraph we show that polytopes are convex sets. This property is important for
eventually proving the Fundamental Theorem of LP. A set of points S in the n-dim space
is convex, if the line segment connecting any two points , belongs
completely in S. According to the previous mathematical definition of line segments, this
definition of convexity is mathematically expressed as:

Figure8 depicts the concept.

Figure 8: Convex sets

To show that a polytope is a convex set, we first establish that the solution space of any
linear constraint (i.e., hyperplanes and half-spaces) is a convex set. Since a polytope is
the intersection of a number of hyperplanes and half-spaces, the convexity of the latter
directly implies the convexity of the polytope (i.e., a line segment belonging to each
defining hyperplane and/or half-space will also belong to the polytope).

To establish the convexity of the feasible region of a linear constraint, let's consider the
constraint:

http://www.amaderforum.com
 http://www.amaderforum.com

and two points satisfying it. Then,

for any , we have:

Adding Equations21 and22, we get:

i.e., point belongs in the solution space of the constraint.

A last concept that we must define before the statement of the Fundamental Theorem of
LP, is that of the extreme point of a convex set. Given a convex set S, point is an
extreme point, if each line segment that lies completely in S and contains point , has
as an end point of the line segment. Mathematically,

http://www.amaderforum.com
 http://www.amaderforum.com

The Fundamental Theorem of Linear Programming

Having established all the necessary concepts and properties of the solution space of n-
var LP's, we are now ready to discuss the Fundamental Theorem of Linear Programming.
This theorem can be stated as follows:

Proof (Sketch):
We establish the validity of Theorem1, through a series of observations:

1. First notice that according to the previous discussion, the feasible region of an LP
is a polytope, and thus, convex.
2. Furthermore, since we assume that the LP has an optimal solution, let denote
such an optimal point. The optimal objective value will be denoted by
.
3. Then notice that point cannot be interior to a line segment that is not
perpendicular to the direction of improvement to the ``isoprofit'' hyperplanes -
w.l.o.g., let's assume a maximization LP for our discussion - defined by vector .
Otherwise, by moving on this line segment in the direction of improvement of the
``isoprofit'' hyperplanes,we would be able to obtain another point of the
feasible region, such that . But this contradicts the assumption that
is an optimal point. Figure9 depicts this argument.

http://www.amaderforum.com
 http://www.amaderforum.com

Figure 9: Why an optimal solution to an LP cannot be interior to a line segment

not perpendicular to the direction of improvement of the "isoprofit" hyperplanes

4. However, point can be interior to a line segment of the feasible solution space
which is perpendicular to the direction of improvement of the optimal ``isoprofit''
hyperplane. This, in fact, corresponds to a situation of many optimal solutions. In
this case, notice that this line segment must have at least one end defined by
the fact that one (or more) additional constraint is binding at this point. Otherwise,
the problem is ill-posed, since we can vary some variable(s) at will over
with this variation affecting neither the constraints nor the objective.
5. Hence, , being on the optimal ``isoprofit'' hyperplane, is another optimal point
at which an additional constraint is binding. Then, there are two possibilities: (i)
is an extreme point of the feasible region, in which case we are done, or (ii)
is interior point to another line segment lying in the optimal ``isoprofit''
hyperplane , which binds, however, an additional constraint,
compared to point . In this case, repeating the argument above, we establish
the existence of another end point , determined by the binding of at least one
more constraint. Then, we repeat the entire argument for , and so on. Figure10
depicts this part of the proof.

http://www.amaderforum.com
 http://www.amaderforum.com

Figure 10: Identifying an optimal extreme point on the optimal isoprofit

hyperplane

6. Finally, notice that every time we bind an additional constraint, we restrict the
(sub-)space of optimal solutions considered by one ``degree of freedom''. Since an
n-dim space has n ``degrees of freedom'', the number of end points visited in the
argument above before we find one that it is an extreme point is finite. Thus, this
last observation establishes the existence of an optimal extreme point for the case
of many optimal solutions, and the proof is complete.

The discourse of the previous proof has also revealed a very important property of
extreme points: At these points, the number of binding constraints is such that it allows
zero ``degrees of freedom'', or, in other words, these constraints define the point uniquely.
Starting from this observation, in the next section we provide a series of algebraic
characterizations of the extreme points, which will eventually allow us to analytically
manipulate the set of extreme points of an LP, in the context of the Simplex algorithm.
As it has been previously mentioned, this algorithm exploits the result stated in the
Fundamental Theorem above, by limiting the search for an optimal solution over the set
of extreme points of the polytope defining the LP feasible region. In the next section, we
shall show that this set is finite and discrete, so it can even be exhaustively enumerated.
Simplex algorithm provides an efficient way to search this set.

http://www.amaderforum.com
 http://www.amaderforum.com

An algebraic characterization of the solution search

space: Basic Feasible Solutions
In the previous section we showed that if an LP has a (bounded) optimal solution, then it
has (at least) one which corresponds to an extreme point of its feasible region. To exploit
this result algorithmically, we need an algebraic characterization of the extreme point
concept. This is the topic of this section.

The starting point of this discussion is the observation made at the end of the previous
section, that at an extreme point, the set of binding constraints is such that it characterizes
the point uniquely. Let's try to investigate what is the algebraic structure implied by this
statement. Also, staying close to the general spirit of our discussion, let's examine this
issue in an inductive manner.

• We know that in the 1-dim space, i.e., on the line of real numbers, a point can be
identified uniquely by an equation a X = b, where .
• In the 2-dim space, a linear equation defines a line, i.e., a
subspace with 1 ``degree of freedom'', while the definition of a unique point
requires a system of two linear equations:

with a unique solution, i.e., with

Such a system of equations is characterized as linearly independent, and

geometrically, it corresponds to two intersecting straight lines.

• In the 3-dim space, a linear equation corresponds to a

plane perpendicular to the vector . A system of two linear
equations:

for which:

http://www.amaderforum.com
 http://www.amaderforum.com

corresponds to the intersection of two planes, i.e., a straight line.

Defining a unique point in the 3-dim space requires three linearly independent
equations, i.e.,

with

• In a similar fashion, in the n-dim space, a point is uniquely defined by n linear

equations which are linearly independent, i.e.,

with

http://www.amaderforum.com
 http://www.amaderforum.com

Example:
We demonstrate the findings of the above discussion on the feasible region of the
prototype example, which for convenience is reproduced in Figure11.

Figure 11: The feasible region of the prototype example LP

As we can see in the figure, each of the extreme points of this region corresponds to the
binding of a pair of the LP constraints, i.e.,

• point A corresponds to the binding of the two sign restriction constraints,

• point B corresponds to the binding of the second technological constraint and the
sign restriction of variable ,
• point C corresponds to the binding of both technological constraints, and
• point D corresponds to the binding of the first technological constraint and the
sign restriction of variable .

Notice, however, that points E and F, even though they are defined by the binding of the
constraint pairs (tech. con. 1, sign res. of ) and (tech. con. 2, sign res. of ),
respectively, are not extreme points of the feasible region just because they are infeasible
(i.e., some other LP constraints are violated). Hence, having n linearly independent
constraints binding at certain point is a necessary condition for it to be an extreme point
of the feasible region, but not sufficient.

http://www.amaderforum.com
 http://www.amaderforum.com

Finally, it should be easy to see that for an n-var LP, n is the minimum number of
constraints binding at an extreme point. If more than this minimum number of constraints
are binding to an extreme point, the point (and the corresponding solution) are
characterized as degenerate. Degeneracy can complicate the search for the optimal
solution carried out by the Simplex method.

http://www.amaderforum.com
 http://www.amaderforum.com

LP's in ``standard form''

To further exploit the previous characterization of extreme points as the solution to n
binding linearly independent constraints, we must define the concept of LP's in
``standard form''. An LP is said to be in ``standard form'', if: (i) all technological
constraints are equality constraints, and (ii) all the variables have a nonnegativity sign
restriction.

Every LP can be brought into ``standard form'' through the following transformations:

• an inequality constraint:

can be converted into an equality one, through the introduction of a slack (excess)
variable :

• A variable with sign restriction can substituted by with

• Finally, a urs variable can be substituted by with .

You can get a clearer idea about how the above transformation works, by having it
applied to your own examples. To interpret the obtained results, notice that the final
outcome is presented in a"tableau" format.
Submit your examples

http://www.amaderforum.com
 http://www.amaderforum.com

Basic Feasible Solutions: An algebraic characterization

of extreme points for LP's in ``standard form''
For LP's in ``standard form'' the previous characterization of extreme points as the
solution of n linearly independent binding constraints, which is, furthermore, a feasible
point, can become even more concise. Consider, for instance, the LP

with m technological constraints and n variables. In ``standard form'', it becomes:

where I is the identity matrix, and is the vector of slack

variables. It is interesting to notice that for every binding constraint from the m+n
constraints of the original formulation (Eq.25), one of the m+n variables of the ``standard
form'' formulation must be equal to 0. Specifically, if the i-th technological constraint is
binding, the corresponding slack variable . Similarly, if one of the sign restriction
constraints is binding, then the corresponding varialbe . Since an extreme point of
the feasible region of formulation25 requires n binding constraints for its definition, it
follows that n from the n+m variables in the corresponding solution of the ``standard
form'' formulation (Eq.26) must be equal to zero. Furthermore, since this point is
uniquely defined, the system of equations defined by the m technological constraints in
the ``standard form'' formulation and the m remaining variables, must have a unique
solution. In other words, the columns of the ``standard form'' formulation corresponding
to these m variables must be linearly independent (and their determinant must have a
nonzero value). Finally, since the extreme point considered belongs in the feasible region
of the problem, it follows that the unique solution of the aforementioned system of m
equations in the m nonzero variables must be nonnegative (to meet the sign restrictions
required by ``standard form'').

The structure of the ``standard form'' solutions corresponding to extreme points of the
original feasible region (i.e., that defined with respect to the primary LP variables ),
described for the example above, actually applies to any other LP in ``standard form''. We
formally characterize this structure through the definition of basic feasible solutions for
LP's in ``standard form'' (taken from Winston, ``Introduction to Mathematical
Programming'').

http://www.amaderforum.com
 http://www.amaderforum.com

http://www.amaderforum.com
 http://www.amaderforum.com

Example:
Returning to the prototype example, it is easy to see that its ``standard form'' formulation
is as follows:

s.t.

This formulation involves four variables and two technological constraints. Therefore,
any basic solution will be defined by selecting a basis of two variables, with the
remaining two being set equal to zero. For example, selecting as basis the variable set
implies that (since they are the remaining non-basic variables).
This further implies that the considered basic solution corresponds to the extreme point of
the LP feasible region defined by the binding of the two technological constraints.
Finally, the values of the two basic variables and are obtained by solving the system
of equations:

which results from the technological constraints in ``standard form'', by eliminating the
non-basic variables.

On the other hand, consider the extreme point B of the feasible region, which is the
optimal solution of this example LP. It has been already shown (cf. previous example)
that this point is defined by the binding of the second technological constraint and the
sign restriction imposed on variable . Hence, at this point, , and the
corresponding basis consists of variables and . To compute the values for these
variables, we solve the system of equations:

http://www.amaderforum.com
 http://www.amaderforum.com

Finally, the basic solution defined by the basis implies that , and
therefore, the corresponding point on the -plane, F, is defined by the binding of
second technological constraint and the sign restriction of variable . Solving the
system of equations:

we obtain: . This basic solution is not feasible, which is also

reflected in Figure2 by the fact that point F is not an extreme point of the feasible region.
.

In the example above, extreme points B and C are adjacent, in the sense that they are
linked by one edge of the feasible region. This reflects in the structure of the
corresponding bases by the fact that they differ in only one binding constraint. This
observation generalizes to the n-dimensional case: extreme points connected by ``edges''
of the feasible region have n-1 common binding constraints, and therefore, their
corresponding bases will differ in one variable only. Hence, we have the following
definition:

The characterization of the extreme points of the feasible region of an LP as basic

feasible solutions for its ``standard form'' representation provides the analytical means for
organizing the search for an optimal extreme point performed by the Simplex algorithm.
The details of this algorithm is the topic of the next section.

http://www.amaderforum.com
 http://www.amaderforum.com

The Simplex Algorithm

In the previous sections we have established the following two important results:

1. If an LP has a bounded optimal solution, then there exists an extreme point of the
feasible region which is optimal.
2. Extreme points of the feasible region of an LP correspond to basic feasible
solutions of its ``standard form'' representation.

The first of these results implies that in order to obtain an optimal solution of an LP, we
can constrain our search on the set of the extreme points of its feasible region. The
second result provides an algebraic characterization of this set: each of these points is
determined by selecting a set of basic variables, with cardinality equal to the number of
the technological constraints of the LP, and the additional requirement that the (uniquely
determined) values of these variables are nonnegative (cf. discussion on basic feasible
solutions). This further implies that the set of extreme points for an LP with m
technological constraints and N variables in its ``standard form'' representation can have

only a finite number of extreme points; specifically, is an upper

bound for the cardinality of this set.

The last observation would make one think that a (naive) approach to the problem would
be to enumerate the entire set of extreme points, compare their corresponding objective
values, and eventually select one which minimizes the objective function over this set.
Such an approach would actually work for rather small formulations. But for reasonably
sized LP's, the set of extreme points, even though finite, can become extremely large. For
example, a small LP with 10 variables (in ``standard form'') and 3 technological
constraints can have upto 120 extreme points, while an LP with 100 variables and 20
constraints can have upto extreme points. And yet, this is a rather small LP!

Hence, we need a more systematic approach to organize the search so that we manage the
complexity resulting from the size of the search space. Such a systematic approach is
provided by the Simplex algorithm. The basic logic of the algorithm is depicted in
Figure12.

http://www.amaderforum.com
 http://www.amaderforum.com

Figure 12: The basic Simplex logic

The algorithm starts with an initial basic feasible solution (bfs) and tests its optimality. If
some optimality condition is verified, then the algorithm terminates. Otherwise, the
algorithm identifies an adjacent bfs, with a better objective value. The optimality of this
new solution is tested again, and the entire scheme is repeated, until an optimal bfs is
found. Since every time a new bfs is identified the objective value is improved (except
from a certain pathological case that we shall see later), and the set of bfs's is finite, it
follows that the algorithm will terminate in a finite number of steps (iterations).

It is also interesting to examine the geometrical interpretation of the behavior of Simplex

algorithm. Given the above description of the algorithm and the correspondence of bfs's
to extreme points, it follows that Simplex essentially starts from some initial extreme
point, and follows a path along the edges of the feasible region towards an optimal
extreme point, such that all the intermediate extreme points visited are improving (more
accurately, not worsening) the objective function.

In the following, we explain how the Simplex algorithm implements this logic in its
computations by appplying it on our prototype LP.

http://www.amaderforum.com
 http://www.amaderforum.com

The basic Simplex iteration through an example:

Consider our prototype LP in standard form, repeated below for convenience:

s.t.

Finding an initial bfs

To start the Simplex algorithm on this problem, we need to identify an initial bfs. For this
particular problem, a bfs will have two basic variables, since we have two technological
constraints. Taking a closer look to the structure of these constraints in Equation28, it can
be seen that a convenient selection is

, where denotes the set of basic variables (basis). Indeed, setting

, we readily obtain, .

The above easy computation of the values of the basic variables resulted from the fact
that each of these variables could be associated with one and only one constraint. More
specifically, (i) each of these variables appeared in only one constraint, (ii) the coefficient
of the variable in that constraint was equal to 1.0, and (iii) any pair of basic variables
showed up in different constraints. An LP the constraints of which satisfy these three
properties with respect to a certain basis, is said to be in canonical form with respect to
that basis.

On the other hand, thefeasiblity of the above basis, , was established by

the fact that the right-hand-side coefficients of the constraints in their canonical form
with respect to basis are non-negative. We shall address the problem of how to
compute an initial bfs in the more general case where one is not readily available by
inspection, in a later section.

http://www.amaderforum.com
 http://www.amaderforum.com

Testing bfs optimality: Checking the sign of the objective function coefficients for
nonbasic variables Notice that the LP objective value corresponding to basis is
determined by the fact that , and therefore, z= 0. So, we pose the question:
Is it possible that there exists another bfs with a better objective value? Obviously, any
bfs for which and/or is a basic variable has a considerable chance of having a better
(i.e., strictly positive) objective value, since the objective function coefficients for these
variables are positive numbers. Hence, the answer to the previous question is that the
nonnegativity of the objective function coefficients of the nonbasic variables
implies that there is potential for improvement of the objective value, z, and both
variables are good candidates for entering the basis.

Selecting the entering variable

Given that at every objective-improving iteration the Simplex algorithm considers
adjacent bfs's, it follows that only one of the candidate nonbasic variables will eventually
enter the basis. Typically, the variable selected to enter the basis is the one that will incur
the maximum improvement to the objecive value per unit of increase of the variable. In
our case, this translates to selecting the (nonbasic) variable with the most positive
coefficient in the objective function, i.e., variable .

Selecting the variable to leave the basis: the ratio test Once we have selected the
variable to enter the basis, we are faced with the question of which of the current basic
variables will be dropped out of it, in order to obtain the improving adjacent bfs. The
logic behind this step is as follows: Since increasing from its current (zero) value
improves the objective value, we would like to increase it as much as we can. What
constrains us in this increase, is the requirement to meet the technological constraints:

as well as the sign (nonnegativity) restrictions imposed on the LP variables. In particular,

since variable will remain nonbasic, its value will remain equal to zero, and therefore
it vanishes from the above set of equations. Hence, we have:

http://www.amaderforum.com
 http://www.amaderforum.com

Notice that as increases, both and are decreased. Obviously, cannot increase
beyond a value that makes any of and negative. So, the maximal allowable increase
for is obtained by solving the system of inequalities:

Hence,

i.e., . Equation30 is known as the ratio test in the theory of Simplex algorithm,
and for an entering variable , its general form is:

The variable leaving the basis is anyone of those corresponding to a technological

constraint with index i minimizing the ratio of Equation31; indeed, setting to the min-
ratio value drives these variables to zero. Hence, in our case, the variable to leave the
basis is , and the new bfs is . The new (improved) objective value is
.

Obtaining the canonical form with respect to the new bfs: Pivoting the entering
variable At this point, we must reset to ourselves the question regarding the optimality of
basis . Notice, however, that the way that we addressed this question - as well as all the
other questions - with respect to basis , was facilitated by the fact that the original set
of technological constraints in Equation28 were in canonical form with respect to that
basis. To be able to address the same set of questions regarding basis , we must
transform the original set of constraints into canonical form with respect to this new
basis.

To perform this transformation, first, we rewrite the original LP equations in the form:

http://www.amaderforum.com
 http://www.amaderforum.com

This representation of the LP technological constraints and the objective-function

equation is known as the LP tableau. In particular, the row corresponding to the
objective-function equation is known as the Row-0 of the tableau, and the coefficients of
the (nonbasic) variables in that row are known as the Row-0 coefficients. Notice also that
the right-hand-side entry of Row-0 provides the objective value of the current bfs.

Under the above tableau representation, the columns corresponding to the basic variables
and are essentially the elementary (unit) vectors: and ,
respectively, while the third unit vector is the column of the objective
variable z. This is another way to characterize the fact that the above tableau is in
canonical form with respect to variables . Hence, to obtain the tableau
corresponding to basis , we must convert the column of in the above
tableau to the unit vector , making sure that while we are doing so, we do
not alter the content of these three equations. This can be done by exploiting the
following two properties of a system of linear equations (generally, known as elementary
row operations):

• If we multiply any of the system equations with a nonzero constant, we obtain an

equivalent system of equations (i.e., one with the same solution set).
• if we multiply one of the system equations with a constant and add it to a second
equation, we obtain an equivalent system of equations.

Applying the first of these properties to the third of equations32, with a coefficient of 50,
we get the equivalent system of equations:

Multiplying the third equation above with -1/60 and adding it to the second equation, we
get:

Finally, multiplying the third equation with 400 and adding it to the first equation, we
get:

http://www.amaderforum.com
 http://www.amaderforum.com

This the new tableau is in canonical form with respect to basis . As it was expected, the
transformation provided also (automatically) the values of the new basic variables, and
the objective function value corresponding to the new basis (i.e., the right-hand-side of
Row-0).

Considering the signs of the Row-0 coefficients in this new tableau, we can see that
increasing any of the non-basic variables from their zero value will have a descreasing
effect on the objective value. Hence, we can conclude that is an optimal basis for our
example LP. The optimal values for the basic variables are: and , and
the optimal objective value is .

You can extend your understanding of this basic Simplex logic, by seeing how it applies
to your own examples. Remember that for this basic Simplex version to work, all the
constraints must be of the "<=" type, and all the decision variables as well as the rhs-
coefficients must be nonnegative. Submit your examples

Obtaining an initial bfs in the general case As we saw in the previous example, if all
the constraints in the original LP formulation are of the ` '-type, we can readily obtain
an initial bfs for Simplex, consisting of all the slack variables in the corresponding
``standard form'' formulation. In this section we consider the more general case, where
the original LP formulation might contain also ` '-type inequality as well as equality
constraints. To facilitate the subsequent discussion, let us consider the following LP:

s.t.

which in ``standard form'' becomes:

s.t.

http://www.amaderforum.com
 http://www.amaderforum.com

For this LP, can constitute an initial basic variable associated with the first constraint.
However, the excess variable cannot be a basic variable for constraint 2, even though it
appears only in this constraint, since this would imply a negative value for it (i.e.,
), and the resulting basic solution would not be feasible. The last constraint
(#3), does not even involve an auxiliary variable.

To overcome this problem, we ``synthesize'' an initial bfs by introducing additional

(artificial) variables for the two ``problematic'' constraints, with .
Hence, our initial bfs is , with . Notice,
however, that even though we obtained a bfs for our set of constraints, we have altered
the structure - and therefore, the content - of the original formulation. In fact, it is easy to
check that this bfs (together with the implied zero values for the nonbasic variables) is
not even feasible for the original LP! On the other hand, if we were able to obtain another
bfs for the modified problem in which the introduced artificial variables are nonbasic,
then this bfs would be also feasible for the original LP: the artificial variables, being
nonbasic, would be equal to zero, and therefore, they would vanish (i.e., have no effective
contribution) in the corresponding ``canonical form'' representation. To obtain the effect
just described, we try to drive the artificial variables to zero, by initially trying to achieve
the following objective:

s.t.

Synthesizing and solving the above LP is known as the Phase I - step of the Simplex
algorithm. If the original LP (Eq.36) has a feasible solution, then the nonnegativity of
implies that the optimal value of this new LP will be zero, and by the end of its
solution we shall also have a feasible bfs for the original formulation. On the other hand,
having a strictly positive optimal objective value for the LP of Equation37 implies that
the artificial variables are absolutely necessary to obtain a feasible solution for this set of
constraints, and therefore, our original LP is infeasible. Hence, infeasibility is tested
during the Phase-I step of the Simplex algorithm.

http://www.amaderforum.com
 http://www.amaderforum.com

Applying the previously described Simplex algorithm on the Phase-I LP of Equation37,

we obtain the optimal tableau:

Therefore, a feasible basis for the original LP is . The canonical form

of the original tableau with respect to basis is obtained by:

1. dropping the columns corresponding to the artificial variables from the

tableau of Equation38:

2. re-introducing in Row-0 of the resulting tableau the original LP objective:

3. and, finally, bringing the tableau of Equation40 into canonical form by

performing the appropriate elementary row operations (the Row-0 coefficients of
the basic variables and must be zero in the corresponding canonical-form
formulation):

Then, we are ready to restart Simplex, but this time on the original LP formulation. In
this particular case, it is easy to see that, since we have a minimization problem, the
current basis is already optimal (i.e., increasing from its zero value will only
increase the objective function).

http://www.amaderforum.com
 http://www.amaderforum.com

Unbounded LP's and LP's with many optimal solutions In the previous section, we
saw that Simplex detects infeasibility while trying to solve the Phase-I LP. It is
instructive to consider how the algorithm behaves on unbounded LP's as well as on LP's
with many optimal solutions. To understand these aspects of the algorithm, try to
implement it on the corresponding examples provided in Section 2. What is happening?

In your...explorations, and for further experimentation with the (2-phase) Simplex

algorithm, you can use the software developed by Dr. Timothy Wisniewski, in a
collaboration of Argonne National Laboratory and Northwestern University. In
interpreting the program calculations, it should be noticed, however, that in that code,
the reduced costs of the nonbasic variables are defined as the opposites of the reduced
costs used in this document. Therefore, in the logic of the optimality test, the
interpretation of the reduced cost signs must be inverted. As an example, optimality of a
basis for a minimization problem is implied by the reduced costs of all nonbasic
variables being nonnegative. Additional instructions about the software can be found in
its introductory homepage.

http://www.amaderforum.com