Math 407 Definitions : Sections 1–3

Section 1
• Mathematical Optimization: A mathematical optimization problem is one in which some real-valued
function is either maximized or minimized relative to a given set of feasible alternatives. In this
course we only consider optimization problems over Rn . That is, we only consider optimization
problems having the mathematical representation
P maximize f (x)
,
subject to x ∈ X
where f : Rn 7→ R and X is a closed subset of Rn .

• Objective Function: The objective function in a mathematical optimization problem is the real-valued
function whose value is to be either minimized or maximized over the set of feasible alternatives. In
problem P above, the function f is the objective function.

• Decision Variable: The decision variables in an optimization problem are those variables whose
values can vary over the feasible set of alternatives in order to either increase or decrease the value
of the objective function. In problem P above, the vector x is the vector of decision variables.

• Feasible Region: The feasible region for an optimization problem is the full set of alternatives for the
decision variables over which the objective function is to be optimized. In problem P above, the set
X is the feasible region. The feasible region is often also referred to as the constraint region.

• Optimal Solution: The optimal solution to an optimization problem is given by the values of the
decision variables that attain the maximum (or minimum) value of the objective function over the
feasible region. In problem P above, the point x∗ is an optimal solution to P if x∗ ∈ X and
f (x∗ ) ≥ f (x) for all x ∈ X. It is possible that there may be more than one optimal solution, indeed,
there may be infinitely many.

• Optimal Value: In an optimization problem were the objective function is to be maximized the
optimal value is the least upper bound of the objective function values over the entire feasible
region. If there is no upper bound, then we say that the optimal value is +∞, while if the feasible
region is the empty set, we define the optimal value of a maximization problem to be −∞.
Conversely, in an optimization problem were the objective function is to be minimized the optimal
value is the greatest lower bound of the objective function values over the entire feasible region. If
there is no lower bound, then we say that the optimal value is −∞, while if the feasible region is the
empty set, we define the optimal value of a minimization problem to be +∞.
Therefore, every optimization problem has a well-defined optimal value. But not every optimiza-
tion problem has an optimal solution. For example, consider the optimization problem min {ex : x ∈ R}.
this problem has an optimal value of zero, but there is no optimal solution.

• Linear Function: A linear function on Rn is any function of the form

f (x) = aT x = a1 x1 + a2 x2 + · · · + an xn ,
where a = [a, . . . , an ]T ∈ Rn .

• Linear Inequality: A linear inequality is an inequality that can be written in one of the following two
forms:
aT x ≤ b or aT x ≥ b,
where a ∈ Rn and b ∈ R.

1
• The solution set of a system of linear inequalities: Given A ∈ Rm×n and b ∈ Rm , we write Ax ≤ b
to denote the system of linear inequalities
n
X
aij xj ≤ bi i = 1, . . . , m.
j=1

The set of solutions of this system is

{x ∈ Rn : Ax ≤ b} .

• A Linear Program: A linear program is an optimization problem in finitely many variables having
a linear objective function and a constraint region determined by a finite number of linear equality
and/or inequality constraints.

• Linear Programming: Linear programming is the study of linear programs: modeling, formulation,
algorithms, and analysis.

• Explicit and Implicit Linear Constraints: The explicit linear constraints are those that are explicitly
stated in a given problem. The implicit constraints are those constraints that are part of the natural
description of the phenomenon under study. These are typically bound constraints on the decision
variables. For example, the width of an object is necessarily non-negative.

• Standard Form: Any LP having the representation

maximize cT x
subject to Ax ≤ b, 0 ≤ x
is said to be in standard form.

• Feasible Solution: An feasible solution to an LP is any point that is feasible for the LP, i.e. any
point in the feasible region.

• Infeasible LP: An infeasible LP is an LP whose constraint region, or, equivalently, feasible set, is
empty.

• Unbounded LP: An unbounded LP is one for which there is a sequence of feasible points whose
objective value diverges to +∞ in the case of maximization, and diverges to −∞ in the case of
minimization.

• Sensitivity Analysis: This is the study of the behavior the optimal value and the optimal solution to
an optimization problem subject to changes in the problem specification.

• Marginal Values: The maginal value, or shadow price of a resource in an linear program is the change
in the value of this resourse due to the optimization process.

• The dual to an LP in standard form: The dual to the primal LP

P maximize cT x
subject to Ax ≤ b, 0 ≤ x
is the dual LP
D minimize bT y
subject to AT y ≥ c, 0 ≤ y .

• The Weak Duality Theorem:

If x ∈ Rn is feasible for P and y ∈ Rm is feasible for D, then
cT x ≤ y T Ax ≤ bT y.
 Thus, if P is unbounded, then D is necessarily infeasible, and if D is unbounded, then P is necessarily
infeasible. Moreover, if cT x̄ = bT ȳ with x̄ feasible for P and ȳ feasible for D, then x̄ must solve P
and ȳ must solve D.

Section 2
• Slack Variables: Slack variables are introduced into an LP formulation in order to turn linear in-
equalities into linear equalities. For example, the constraint region for an LP in standard form is
{x ∈ Rn : Ax ≤ b, 0 ≤ x} where A ∈ Rm×n and b ∈ Rm . The system of inequalities is converted
into system of equations involving n + m variables by setting
n
X
xn+i = bi − aij xj , i = 1, 2, . . . , m ,
j=1

with xn+i ≥ 0 i = 1, . . . , m. The new variables xn+i are called slack variables.

• Objective variable for an LP in Standard Form: The variable z defined by the equation z = c1 x1 +
c2 x2 + · · · + cn xn = cT x is called the objective variable.

• Dictionary for an LP in Standard Form: Given the LP in standard form,

maximize cT x
,
subject to Ax ≤ b, 0 ≤ x
we define the initial dictionary for this LP to be the system
Pn
xn+i = bi − aij xj , i = 1, 2, . . . , m ,
DI Pj=1
n .
z= j=1 cj xj

In general, a dictionary for this LP is any system of the form

Pn
xi = b̂i − âij xj , i ∈ B ,
DB Pj∈N
n
z = ẑ + j∈N ĉj xj

having the same set of solutions as the system DI and where the index sets B and N satisfy (a)
B ∪ N = {1, 2, . . . , n + m}, (b) B ∩ N = ∅, and (c) B contains precisely m elements.

• Feasible dictionary for an LP in Standard Form: The dictionary DB given above is said to be feasible
if b̂i ≥ 0 for all i ∈ B.

• The initial tableau for an LP in standard form: The initial tableau for an LP in standard form is
the augmented matrix associated with the the initial dictionary and has the following block matrix
structure:  
0 A I b
.
−1 cT 0 0
The first column of this augmented matrix is often omitted in practise since it remains unaltered
throughout the pivoting process.

• A simplex tableau for an LP in Standard Form: A simplex tableau for an LP in Standard Form is
the augmented matrix associated with any dictionary for an LP in standard form.

• A feasible simplex tableau for an LP in Standard Form: A simplex tableau is said to be feasible if
its associated dictionary is feasible.

• Feasible Solution of an LP: A feasible solution of an LP is any point in the feasible region for the
LP.

• Optimal Solution for an LP: An optimal solution for an LP is any feasible solution whose objective
value equals the optimal value of the LP.
 • Basic variables: The basic variables for a dictionary are those variables that are being defined in
terms of the remaining varibles (or, non-basic varibles) in the dictionary.

• A basic solution: A basic solution to an LP in standard form is any solution to the linear equations
for any dictionary for the LP obtained by setting the non-basic variables equal to zero.

• A basic feasible solution: A basic feasible solution to an LP in standard form is any basic solution
that is also feasible for the LP.

• Non-Basic variables: The non-basic variables for a dictionary are those variables whose values de-
termine the value of the basic variables. We think of the non-basic variables as taking the value
zero.

• A basis for an LP: A subset of the variables {xj : j = 1, 2, . . . , n + m} is said to form a basis for the
LP if there is a dictionary for the LP in which these variables are basic.

• Pivot column: The pivot column is the column of the tableau corresponding the non-basic variable
selected to enter the basis in a simplex pivot. If DB is the dictionary associated with this tableau,
then it must be the case that DB is feasible and if j ∈ N is the index associated with the pivot
column, then we must have ĉj > 0.

• Pivot row: The pivot row is chosen after a pivot column has been specified. Consider the simplex
tableau whose dictionary is given by DB above. Assume that DB is feasible and that j0 ∈ N is the
index for the specified pivot column. Then the pivot row is any row i0 ∈ B for which ai0 j0 > 0 and
(  )
b̂i0 b̂i 
= min  i ∈ B, aij0 > 0 .
ai0 j0 aij0 

• Pivoting: Pivoting in a tableau corresponds to Gauss-Jordan elimination on the pivot column with
the pivot being the matrix entry occuring in both the pivot row and column.

• The pivoting rule for the entering variable: Any variable whose objective row coefficient in the
current dictionary is positive.

• The pivoting rule for the leaving variable: Any basic variable whose non-negativity places the greatest
restriction on increasing the value of the entering variable.

• Optimal dictionary: An optimal dictionary is any dictionary associated with an optimal basic feasible
solution. In particular, the dictionary
Pn
DB xi = b̂i − âij xj , i ∈ B ,
Pj∈N
n
z = ẑ + j∈N j xj
c

is optimal if and only if it is feasible (b̂i ≥ 0, i ∈ B) and cj ≤ 0, j ∈ N .

• Optimal tableau: An optimal tableau is any simplex tableau associated with an optimal basic feasible
solution. In particular, the tableau
 
0 RA R Rb
.
−1 (c − AT y)T −y T −y T b
is optimal if and only if it is feasible (Rb ≥ 0) and the vector y is dual feasible (0 ≤ y and AT y ≥ c).

Section 3
 • An LP with Feasible Origin: An LP with feasible origin is any LP whose feasible region contains the
origin. For an LP in standard form, this implies that bi ≥ 0, i = 1, . . . , m.

• Basic Rule for Choosing the Entering Variable: Given a feasible dictionary, or, equivalently, a feasible
tableau for an LP in standard form, the basic rule for choosing the variable to enter the basis is to
choose any one of the currently non-basic variables whose objective row coefficient is positive.

• Basic Rule for Choosing the Leaving Variable: Given a feasible dictionary, or, equivalently, a feasible
tableau for an LP in standard form, the basic rule for choosing the variable to leave the basis is to
choose any one of the currently basic variables whose non-negativity places the greatest restriction
on increasing the value of the entering variable.

• Degeneracy: Degeneracy occurs in the simplex algorithm when a simplex pivot does not change
either the current value of the objective or the point identified by the dictionary. A dictionary (or
tableau) for an LP in standard for is said to be degenerate if one of the currently basic variables is
assigned the value zero by the dictionary (or tableau).

• Degenerate Basic Solution: A basic feasible solution to an LP in standard form is any feasible solution
identified by a feasible dictionary for the LP. Such a solution is said to be degenerate if one of the
currently basic variables in the basic feasible solution has the value zero.

• Degenerate Simplex Iteration: A degenerate simplex iteration is any simplex iteration that does not
change the value of the objective.

• Cycling: Cycling occurs in the simplex algorithm when a string of degenerate dictionaries is repeated
over and over again infinitely often.

• The Smallest Subscript Rule: The smallest subscript rule is an anti-cycling rule for choosing the
entering and leaving variables in the simplex algorithm. The rule states that the entering and
leaving variables are chosen to have the smallest subscript from among all viable candidates for
entering and leaving the basis.

• Auxiliary Problem: The auxiliary problem to an LP in standard form is an LP whose purpose is to

determine the feasibility of the LP, and if feasible, to locate an initial basic feasible solution and its
associated dictionary or tableau. If
maximize cT x
subject to Ax ≤ b, 0 ≤ x
is the LP in standard form, then one possible auxiliary problem is the LP
maximize −x0
,
subject to Ax − x0 e ≤ b, 0 ≤ x, x0
where e ∈ Rm is the vector of all ones.

• Two Phase Simplex Method: The two phase simplex algorithm is applied to LPs in standard form
that do not have feasible origin.
Phase I: Apply the simplex algorithm to the auxiliary problem. Two outcomes are possible.
(i) The optimal value is positive which implies that the LP is infeasible.
(ii) The optimal value is zero and an initial basic feasible solution is determined.
Phase II: Apply the simplex algorithm initiated at the basic feasible solution provided in Phase I. Again,
two outcomes are possible.
(i) The LP is determined to be unbounded.
(ii) An optimal basic feasible solution is determined.
• The Fundamental Theorem of Linear Programming: Every LP has the following three properties:
(i) If it has no optimal solution, then it is either infeasible or unbounded.
(ii) If it has a feasible solution, then it has a basic feasible solution.
(iii) If it is bounded, then it has an optimal basic feasible solution.

• What is the structure of both the initial and the optimal tableaus, and why does the optimal tableau
have this structure? The initial tableau for an LP in standard for has the structure
 
0 A I b
.
−1 cT 0 0
A simplex pivot on a tableau corresponds to left matrix multiplication on the tableau by a Gauss-
Jordan pivot matrix. By successively multiplying these matrices together we see that every tableau
can be obtained from the initial tableau by a single matrix multiply. Therefore, if
 
0 A b R bb
Tk =
−1 b cT −y T zb
is some tableau for the LP, then there is a matrix
 
M u
G= ,
vT β
the product of Gauss-Jordan pivoting matrices, such that
 
0 A b R bb
= Tk
−1 b cT −y T zb
= GT0
  
M u 0 A I b
=
vT β −1 cT 0 0
−u M A + ucT M M b
 
= .
−β v T A + βcT v T v T b
By equating the blocks on the left and right sides of this equation, we find that
u = 0, β = 1, M = R, and v = −y.
Putting all of this together gives the following representation of the k th tableau Tk :
    
R 0 0 A I b 0 RA R Rb
Tk = = ,
−y T 1 −1 cT 0 0 −1 cT − y T A −y T −y T b
where the matrix R is necessarily invertible since the matrix
 
R 0
G=
−y T 1
is invertible with
R−1
 
−1 0
G = T −1 .
y R 1
If Tk is the optimal tableau, then we must have
Rb ≥ 0, AT y ≥ c, and 0≤y ,
T
and the optimal value of the LP is b y. Note that this implies that the vector y is dual feasible with
bT y equal to the optimal value in the primal LP. Therefore, by the Weak Duality Theorem for Linear
Programming, the vector y must be the solution to the dual LP.

• How do you read off the optimal solutions for both the primal and dual problems from the optimal
tableau? The primal solution is obtained by setting the non-basic variables in the optimal tableau
equal to zero and setting the basic variables equal to the corresponding element of the right-hand
side vector Rb. The dual solution is the vector y whose components appear as the negative of the
slack variable coefficients in the objective row of the optimal tableau.