Introduction to OR and Linear Algebra

Part-I

Dr. Prasun Das

Indian Statistical Institute
SQC & OR Unit
Kolkata, India
dasp asu @ ediff ail.co
dasprasun@rediffmail.com

16-Jul-18 1
 OR-The Historical Development

• The beginning
g g of operations
p research has ggenerally
y been attributed
to the military services during World War II.

• There was an urgent need to allocate scarce resources to the various

military operations in an effective manner.

• The military commands of U.K.U K and U.S.A

U S A engaged several inter-
disciplinary teams of scientists to undertake scientific research into
strategic and tactical military operations related to air and land
defense of the country (the efficient ocean transport,
transport effective
bombing etc.).

• H
Hence OR can be b associated
i d with
i h “an
“ art off winning
i i the
h war
without actually fighting it”.

16-Jul-18 2
 OR-The Historical Development

• As the name implies,

p ‘Operations
p Research’ was apparently
pp y invented
because the team was dealing with research on (military) operations.

• Following the end of war, the success of military teams attracted the
attention of industry people. In this way, operations research began to
creep into industry, business and civil government.

• American Mathematician, George B. Dantzig, developed the first

mathematical technique in this field in 1947. By 1951, it had already
taken hold in Great Britain and was in the process of doing so in the
United States for the invention of new flight patterns, planning sea
mining and effective utilization of electronic equipment.

16-Jul-18 3
 OR-The Historical Development

• Today
Today, the impact of OR can be felt in many areas like transportation
system, libraries, hospitals and city planning, financial institutions
apart from military and business applications. In fact, linear
programming can be viewed as part of a great revolutionary
development to lay out a path of detailed decisions to take in order to
achieve
hi it goals
its l when
h f
faced
d with
ith practical
ti l situations
it ti off greatt
complexity.

• Some of the well-known names for the contribution during the

beginning of operations research are von Neumann, Kantorovich,
Leontief, Koopmans and Dantzig.

16-Jul-18 4
 What is OR ?

Techniques for Optimizing the Performance of a System.

System: a Process, Product, Institute, Organization etc.

Performance Measure: Any criterion/property measuring the way system is

working.
wo g. So
Somee important
po ta t pe
performance
o a ce measures
easu es aaree p
profit,
ofit, cost, p
profitability,
ofitability,

market share etc. Performance measures are of two types: PROFIT measure

(higher is desirable) and COST measure (lower is desirable).

Optimi ing
Optimizing a Performance: either maximizing
i ii or minimizing
i i ii a

criterion/property.

16-Jul-18 5
 OR Techniques

Linear Programming: To optimize a single linear objective function subject to

linear constraints. It is a technique with many applications.
g Techniques
Nonlinear Programming:
g q for solving
g mathematical models
involving nonlinear functions.
Integer and Combinatorial Optimization: Techniques for solving mathematical
models involving discrete decision variables.
Network Programming: Optimization problems that can be described over a
graph structure. Many applications in transportation, distribution, etc.
D
Dynamic
i Programming:
P i T h i
Technique f solving
for l i multi-period
lti i d or multi-stage
lti t
decision problems.

16-Jul-18 6
 OR Techniques
Multi-criteria Decision-making: To choose the best among a small number of
alternatives, when there are many important characteristics to consider.
Multi-objective Programming: Techniques for obtaining good solutions to
optimization models involving two or more objective functions that need to be
optimized simultaneously. Goal Programming is one technique for solving multi-
objective programs.
Project Scheduling (PERT, CPM): Techniques for controlling, scheduling
various jobs in a project, which have to obey a precedence relationship.
Inventoryy Management:
g Techniques
q for maintaining
g inventories to meet
demands for goods, while at same time minimizing inventory holding costs.
Queuing Problems: To optimize performance of systems involving waiting lines.
lines
16-Jul-18 7
 Basic Concepts
p of Linear Algebra
g

• Vectors,
V t M ti
Matrices, Li
Linear I d
Independence/Dependence
d /D d off vectors,
t R k Basis,
Rank, B i
System of linear eqns, Hyperplane, Convex set, Convex polyhedron, Extreme
points, Basic feasible solution etc.
• Graphical solution method (for 2-variable problems)

16-Jul-18 8
 Definitions – Euclidean Space
p
n-dimensional real Euclidean space (ƒn): Cartesian product of n sets, each
t ƒ1:
equall to

ƒn = ƒ1 x ƒ1 … ƒ1 x ƒ1 n times
So ƒn consists of n-tuples of real numbers as ƒ1xn = {(x1, x2,…,xn): xi ƒ1} or
ƒnx1 = {(x1, x2,…,xn)t: xi ƒ1}

Vector addition and Scalar multiplication: The set of all n-component vectors
with addition [ x + y = (x1+y1, x2 +y2,…, xn +yn)t ] and multiplication by scalar
[ O.x = ((Ox1, Ox2,,…,, Oxn)t ] and the laws of matrix algebra
g being
g used to carry
y on
n 2

the algebra in the set, together with distance | x  y | ¦ x

i 1
i  yi  gives the n-
dimensional Euclidean Space denoted by ƒn or En.
16-Jul-18 9
 Definitions – Vector Space
p
A non-empty set S Ž ƒn is called a Vector Space in ƒn (or, a subspace in ƒn),

if both of the following conditions hold:

a) S is closed under vector addition (xS, yS Ÿ x+yS)

b)) S is closed under scalar multiplication

p ((xS Ÿ DxS for all D ƒ1)

In other words, a non-empty set S Ž ƒn is a subspace in ƒn if x+DyS for every

x, yS and every D ƒ1.

Trivial subspace: The set {0}  ƒn is itself a subspace of ƒn

16-Jul-18 10
 Definitions – L.C. and Span
p
Linear Combination: Let A = {a1, a2,…,an} be a set of n vectors with ai ƒm,
i=1,…,n and x1, x2,…,xn are real numbers.
n
Then, the vector b = ¦ xi ai = x1a1+ x2a2+…+xnan is called a linear combination of
i 1
a1, a2,…,an.
Span of A: The set of all possible linear combinations of the ai’s is called the
span of A, written as Sp(A) = {D1a1+ D2a2+…+Dnan: Diƒ1, i = 1,2,…,n}

Result:
R lt Let
L t A be
b a non-empty
t sett off vectors
t i ƒm. The
in Th span off A is
i a subspace
b
of ƒm.

Result: Let A = {a1, a2,…,an} be a set of vectors from some subspace S Ž ƒm.
Let A = [a1, a2,…,an] be the m x n matrix with ai as the i-th column. Then A
spans S iff Ax = b is a consistent system for every bS.

16-Jul-18 11
 Definitions – Linear Independence
p ((l.i.))

Let A = {a1, a2,…,an} be a finite set of vectors with each aiƒm.

Linear independence (l.i.):

(l i ): A is said to be l.i.
l i if whenever there exist
real numbers x1, x2,…,xn, s.t.
x1a1+ x2a2 +…+ xnan = 0, Ÿ x1= x2=…= xn = 0.

On the other hand,

Linear dependence (l.d.): A is said to be l.i. if whenever there exist real
numbers x1, x2,…,xn, not all zero, such that x1a1+ x2a2 +…+ xnan = 0.

16-Jul-18 12
 Definitions – Basis

Let, Bmxm be a submatrix of Amxn comprising of m(<n) l.i columns of A.

Then, Bmxm = {a1,a2,…,am} is called a basis set for Em if,
‰ B is a l.i.
l i set of vectors;
‰ B spans Em i.e., any member of Em can be expressed as a linear
combination of others.

Remark: {e1,e2,…,en} is called a basis for En since

¾ ¦Oiei = 0 Ÿ Oi = 0 for all i = 1,2,…,n
¾ Let, xEn, i.e. x = (x1, x2,…, xn). Then, x = ¦xiei

16-Jul-18 13
 Definitions – System
y of linear equations
q

Ax = b, where A = (aijj)m×n; x = (x1, x2,…, xn)tnx1; b = (b1, b2,…, bm)tmx1

If r(A) = r(A b) = k (k dm, kdn) Ÿ system is consistent (at least one soln.)
(A) z r(A
If r(A) (A b) Ÿ system
t isi inconsistent
i i t t (no
( soln.
l exists)
it)

Case-1. k < m Ÿ system

y is redundant ((at least one eqn.
q may
y be dropped)
pp )
Case-2. k = m Ÿ system is not redundant
3. k = n Ÿ system is having unique solution.
Case-3.
Case
Case-4. k < n Ÿ system is having infinitely many solutions.

Rank of A: (i) maximum number of l.i. rows/columns of A; or

((ii)) order of the largest
g non-vanishing
g minor of A.

16-Jul-18 14
 Definitions – Basic Variables & Basis Matrix
If rank (A) = m (no-redundancy) with m d n (unique/infinite solutions) for a
consistent system Ax = b, then the matrix A has at least one set of m l.i. columns.

Variables, xj, j = 1,…,m, associated with the jth column of the matrix A are called
basic variables. The remaining (n
(n-m)
m) variables are called non
non-basic
basic variables.

The original matrix A can be partitioned into two matrices B and N respectively
such that A = [B N] where, B is called the basis matrix associated with m basic
variables
i bl and
d N is
i called
ll d non-basis
b i matrix
t i associated
i t d with
ith (n-m)
( ) non-basic
b i
variables.

16-Jul-18 15
 Definitions – Basic Solution & B.F.S

• Linear System;
y ( ) = m d n.
; Ax = b. Assume r(A)
• Partition columns as A = [B:N], where B is m x m basis matrix and N is
m x (n
(n-m)
m) non
non-basis
basis matrix.
matrix
• The linear system becomes: BxB + NxN = b, where xB is vector of basic
variables and xN is vector of non-basic
non basic variables.
variables
• Since B is non-singular, it is invertible. Hence, xB = B-1b – B-1NxN.
• Basic
B i Solution
S l ti for
f a linear
li system: Set
S xN = 0, h xB = B-11b.
0 then, b
• If xB t 0 (i.e. B–1b t 0), then the solution x = (B-1b, 0)t is called a basic
feasible solution (b.f.s).

16-Jul-18 16
16-Jul-18 17
 Definitions - Hyperplane and Half Space

• Hyperplane H: {x: ctx = J}, where cRn and JR1

(Ex: Line c1x1 + c2x2 = J in R2, Plane c1x1+c2x2+c3x3 = J in R3)
• Closed halfspaces
p H+: {{x: ctx t J} and H-: {{x: ctx d J}
• Open halfspaces H+: {x: ctx > J} and H-: {x: ctx < J}
• A polyhedral set is the intersection of halfspaces.

16-Jul-18 18
 Definitions - Convexity and Convex polyhedron
Point Set: Any subset XŽEn will be called a point set of En.
Line Segment in En: line joining distinct points x1, x2 in En is defined as S = {x:
x = (1-O)x1 + Ox2, 0dOd1}
Convex Set: Let XŽEn. Then X is called a convex set if for x1, x2En, (1-O)x1 +
Ox2 X where, 0dOd1.

16-Jul-18 19
 Definitions - Convexity and Convex polyhedron

Convex Polyhedran:
Let x1, x2,…, xp  En. If x = ¦Oixi, Oi t 0, ¦Oi = 1, then x is called a convex
combination of x1, x2,…, xp. The set of all convex combinations of x1, x2,…, xp
is called Convex Polyhedran generated (or, spanned) by the points x1, x2,…, xp.

• A Convex Polyhedron is an intersection of half spaces.

• It is Not necessarily bounded.
bounded
• A bounded convex polyhedron is a polytope.
• A polytope
l t i R2 is
in i a convex polygon.
l

16-Jul-18 20
 Geometry: Convex Polyhedron

16-Jul-18 21
 Definition: Extreme Points

A corner
corner--point (extreme point) feasible solution is a feasible solution
that cannot be written as a convex combination of other two feasible
solutions. ((does not lie on any
y line segment
g connecting
g 2 other
feasible solutions.

16-Jul-18 22
 Definition: Extreme Points

Extreme Points of Convex Set:

Let X be a convex set in En and  X. Then, is called an extreme point of X if
there do not exist x1, x2En, x1z x2 s.t. ) 1 + Ox2 X where, 0<O<1. ie.,
= ((1-O)x
should not occur strictly in between any line segment in X.
Remark: A convex set with no extreme point is a hyperplane
Question: If X is defined as the interior of the circle as well as points on the
circumference then what about existence of extreme points of X?
circumference,

16-Jul-18 23
 Definition: Extreme Points in case of a LPP

16-Jul-18 24
 Definition: Extreme Points in case of a LPP

Feasible
F ibl region
i i a convex
is
polyhedron; Optimal solution is
a corner.

16-Jul-18 25
 References:
1. Linear Algebra – C Y Hsiung and G Y Mao
2. Applied Linear Algebra and Matrix Analysis – Thomas S. Shores
3. Linear Algebra with Geometric Applications – Mansfield, L.E.
4. Linear Algebra and Matrix Analysis for Statistics – Sudipto
j and Anindya
Banerjee y Royy

16-Jul-18 26