Dynamic Programming Theory and Applications

Jaroslav Sklenar
Department of Statistics and Operations Research
Faculty of Science
University of Malta

2017

CONTENTS

1. Introduction
2. Theoretical foundations
2.1 Separability
2.2 Principle of optimality
2.3 Decomposable functions
2.4 Concept of state
2.5 Multistage decision model
2.6 Backward procedure algorithm
2.7 Progressive problems
3. Stochastic dynamic programming
3.1 Random return
3.2 Random transition
4. Selected applications
4.1 Allocation problems
4.2 Work force size
4.3 Equipment replacement
4.4 Periodic review inventory
4.5 Critical path analysis
4.6 Traveling salesman
4.7 Scheduling
5. Performance of the DP algorithm
References
Appendix: Critical path analysis – activity on arcs approach

1
1. Introduction

Dynamic Programming (DP) is an optimization technique based on decomposition of a

complex optimization problem into a sequence of simpler problems in such a way that the
total time needed to solve them is smaller than the time needed to solve the original problem.
It is rather a general principle than a particular optimization method – [1]. Individual
subproblems are often called decision stages because originally the technique has been used to
optimize discrete time dynamic processes that can be represented as a sequence of decisions in
time. That is the origin of the word "dynamic". The second half "programming" comes from
the fact that DP is a special case of mathematical programming methods. Today the scope of
problems solved by DP is very wide. Sometimes the decision stages are defined even for
problems that in fact do not involve a sequence of decisions in order to decompose the
problem. This text has to be taken as a summary of basic facts, examples and explanatory
drawings will be given during lectures.

Multistage decision model - informally

Dynamic programming is based on decomposition of an n-dimensional optimization problem

opt{f(x)|x  S  n } into a sequence of simpler, mostly univariate problems solved at the so-
called decision stages. For each stage we define possible decisions and possible states that
represent conditions (circumstances) under which the decision is taken. In order to select a
decision we need a criterion that evaluates numerically how good a decision is. This criterion
is typically called return. Return is thus a scalar function of three arguments: state, decision,
and the stage number. It represents the immediate contribution brought by a particular
decision at a particular state at a given stage. It is obvious that stages are not independent.
Their links are represented by the so-called transition function that represents the effect of
taking a certain decision at a certain state at stage t upon the stage t+1. So its value is the state
at the stage t+1. This is sometimes not needed for the last stage and also the initial state, or set
of initial states, must be given. It is also clear that in order to find the optimum of the original
problem the stages can not be optimized separately. This fact is represented by the so-called
functional equation of dynamic programming. To explain its purpose we have to introduce
informally the so-called backward procedure of dynamic programming:

First optimize the last stage n by selecting for each state the decision that brings the
optimal return. Then recursively for each stage t = n-1, n-2, ... ,1 we assume that we
know the optimal decision and the associated optimal objective value for each generated
state at the stage t+1. So for each state at stage t we select the best decision that together
with the optimal decision at t+1 creates an optimal sequence starting at stage t for the
given state. Optimal sequence starting at stage 1 is then the optimizer for the original
problem. Functional equation defines the way how the return at stage t and the optimal
objective at stage t+1 are combined. Most practical applications of dynamic
programming are based on additive objective.

So we have these 6 elements of dynamic programming models together with symbols that will
be used in this text:

1. Stages t  {1 … T } where T is the number of stages

2. Set of states for each stage yt  Yt
3. Set of acceptable decisions for each stage-state pair (t, yt) xt  Xt(yt)

2
 4. Transition function yt+1 = gt(xt ,yt )
5. Initial state y1
6. Functional equation giving returns and the way how they are combined.

Note that feasibility x  S of the original problem is expressed implicitly by the sets of
possible states and possible decisions at the stages. To specify the functional equation we
introduce for each stage the stage objective function ft whose arguments are the state yt and the
decision xt. The function value is the partial objective value defined for the stage t and all
subsequent stages. The optimal value for an optimal decision at stage t will be called Ft. Note
that Ft has only one argument yt. The optimal objective value of the original problem is then
equal to F1(y1). We know that the triple (t, xt , yt) defines the state of the next stage yt+1
through the transition function gt. This next state yt+1 then generates the optimal value of the
next stage Ft+1(yt+1). In order to make direct use of this optimal value, we shall add it as the
3rd argument of the function ft. The functional equation for the backward procedure is then:

Ft ( yt )  opt { f t ( xt , yt , Ft 1 ( yt 1 ))}  opt { f t ( xt , yt , Ft 1 ( g t ( xt , yt )))} (1)

xt  X t ( yt ) xt  X t ( yt )

To make use of the stage return we define another form of the stage objective function:

ft ( xt , yt , Ft 1 ( yt 1 ))  ft (rt ( xt , yt ), Ft 1 ( yt 1 )) (2)

where rt( xt , yt ) is the immediate return at stage t, state yt and decision xt. For simplicity we
shall use the same name ft for both forms of the stage objective function. Note that both forms
of ft assume separability of the original problem. Considering the return rt we get an
alternative form of the functional equation:

Ft ( yt )  opt { f t (rt ( xt , yt ), Ft 1 ( yt 1 ))}  opt { f t (rt ( xt , yt ), Ft 1 ( g t ( xt , yt )))} (3)

xt  X t ( yt ) xt  X t ( yt )

Mostly additive objective functions are used:

ft (rt ( xt , yt ), Ft 1 ( yt 1 ))  rt ( xt , yt )  Ft 1 ( yt 1 ) (4)

(4) inserted into (3) gives the typical version of DP functional equation:

Ft ( yt )  opt {rt ( xt , yt )  Ft 1 ( yt 1 )}  opt {rt ( xt , yt )  Ft 1 ( g t ( xt , yt ))} (5)

xt  X t ( yt ) xt  X t ( yt )

It is also assumed that for finite number of stages the function FT+1(yT+1) is given. It is
necessary for the evaluation of the last stage T that is the first stage solved. Often it is zero,
but generally it represents the value of the given terminal state after performing the sequence
of T decisions. Note that the functional equation does not allow evaluation of the original
objective function f for all possible sequences of decisions because at stage t only optimal
decisions at stage t+1 are considered. Basic relationships at stage t can be expressed
graphically as follows:

3
 xt  Xt(yt) decision taken at stage t

state at stage t yt t yt+1 = gt(xt ,yt) state at t+1

rt(xt ,yt) return at stage t

Ft(yt) optimal objective for the pair (t, yt)

2. Theoretical foundations

Not all problems can be decomposed into a sequence of interrelated stages and solved by the
functional equation (1). In this chapter we study the related requirements and conditions.

2.1 Separability

It is clear (intuitively) that for additive returns the functional equation (5) is correct. Generally
two conditions have to be satisfied: separability and optimality. In this chapter we deal with
separability. We assume that stage returns rt(xt ,yt) are defined for each pair (t, yt) and each
decision xt, t  {1 … T }. Then we assume that the objective of the original problem can be
expressed as a certain function 1 with stage returns as arguments:

F1 ( y1 )  opt 1 (r1 ( x1 , y1 ), , rt ( xt , yt ), , rT ( xT , yT ))

X 1W1 ( y1 )

Where: Xt is a plan (sequence) made of T-t+1 decisions (xt , … ,xT)

Wt(yt) is the set of all possible plans that start at (t, yt)

The separability condition expressed by the existence of the function ft can be formulated
verbally as follows:
"For every plan the objective value of every stage must be expressible as a function of the
immediate return and the objective value of the succeeding stage". Formally:

t ( rt ( xt , yt ), , rT ( xT , yT ))   t ( yt , X t )  f t ( rt ( xt , yt ), t 1 ( yt 1 , X t 1 )), t  1 T  1
(6)
T ( rT ( xT , yT ))   T ( yT , xT )  rT ( xT , yT )

Where:  t 1 ( yt 1 , X t 1 )  t 1 (rt 1 ( xt 1 , yt 1 ), , rT ( xT , yT ))
and fi, i and i are appropriate functions.

Some functions for which the separability condition is satisfied:

Additive returns: f(x,y) = x + y

4
 T
 t ( yt , X t )  rt ( xt , yt )   rT ( xT , yT )   ri ( xi , yi ) 
i t
T
rt ( xt , yt )   r ( x , y )  r ( x , y ) 
i  t 1
i i i t t t t 1 ( yt 1 , X t 1 )

Additive discounted returns: f(x,y) = x + by where b=1/(1+R), R = rate of return

T
 t ( yt , X t )  rt ( xt , yt )  brt 1 ( xt 1 , yt 1 )   bT t rT ( xT , yT )   b i t ri ( xi , yi ) 
i t
T
rt ( xt , yt )  b  b i  ( t 1) ri ( xi , yi )  rt ( xt , yt )  b t 1 ( yt 1 , X t 1 )
i  t 1

Multiplicative returns: f(x,y) = xy

T
 t ( yt , X t )  rt ( xt , yt )   rT ( xT , yT )   ri ( xi , yi ) 
i t
T
 rt ( xt , yt )   ri ( xi , yi )  rt ( xt , yt )  t 1 ( yt 1 , X t 1 )
i  t 1

Counter examples:

1)  1 ( y1 , X1 )  r3 ( x3 , y3 )  r2 ( x2 , y2 )  r3 ( x3 , y3 )  r1 ( x1 , y1 )  r1 ( x1 , y1 )  r2 ( x2 , y2 ) (for T=3)
T
2) F1 ( y1 )  opt {( ri ( xi , yi )) 2 }
X 1W1 ( y1 ) i 1

So we have the first necessary condition of solving a problem by dynamic programming: it

must be possible to identify separate decision stages with decisions being quantified by real
valued functions that take as an argument the objective of the next stage. Preferably we want
to identify returns of individual stages. Then the condition (6) has to be satisfied.

2.2 Principle of optimality

The above separability condition is not sufficient. Another requirement is the correct
application of the functional equation (1) or (3) that are based on repeated optimization of
inner problems. For equation (3) this can be expressed as follows:

Ft ( yt )  opt { f t (rt ( xt , yt ), Ft 1 ( yt 1 ))} 

xt  X t ( yt )
(7)
opt { f t (rt ( xt , yt ), opt { f t 1 (rt 1 ( xt 1 , yt 1 ), Ft  2 ( yt  2 ))}
xt  X t ( yt ) xt 1 X t 1 ( yt 1 )

Sufficient condition called principal of optimality was formulated for the first time by Richard
Bellman in 1957. It represents the basic idea of DP that in order to solve the stage t, we can
consider only optimal decisions at the stage t+1. The condition reads as follows:

5
 "For every stage and every decision that starts an optimal plan from this stage the
optimal plan consists of the given decision followed by the plan which is optimal
with respect to the succeeding stage".

Alternative formulations:

"Every optimal plan must have the property that regardless of the route taken to enter
a particular stage, the remaining decisions must constitute an optimal plan for leaving
that stage".
"Every optimal solution can only be formed by partial optimal solutions"

Now let's express the optimality condition algebraically. A plan Xt consists of a decision xt
followed by a plan Xt+1 that starts at stage t+1:

Xt = xt  Xt+1  t ( yt , X t )   t ( yt , xt  X t 1 )

In the above formulae the operator "" is concatenation (joining) of decisions. Let Pt+1 be an
optimal plan starting from the stage-state pair (t+1, yt+1). In other words it is an optimal plan
starting at the successor of the stage-state pair (t, yt) given decision xt. Then (assuming
minimization) the optimality condition requires that for every stage-state pair (t, yt), decision
xt and plan Dt+1:

 t ( yt , xt  Pt 1 )   t ( yt , xt  Dt 1 ) (8)

Some functions for which the optimality condition is satisfied:

Additive returns: f(x,y) = x + y

For additive returns under any plan:

 t ( yt , xt  Dt 1 )  rt ( xt , yt )  t 1 ( yt 1 , Dt 1 )

So the optimality condition requires that:

rt ( xt , yt )  t 1 ( yt 1 , Pt 1 )  rt ( xt , yt )  t 1 ( yt 1 , Dt 1 )

By subtracting the return from both sides we get

 t 1 ( yt 1 , Pt 1 )   t 1 ( yt 1 , Dt 1 )

which is true by definition of the plan Pt+1. So the optimality condition is satisfied.

Additive discounted returns: f(x,y) = x + by where b=1/(1+R), R = rate of return

For discounted returns under any plan:

 t ( yt , xt  Dt 1 )  rt ( xt , yt )  b t 1 ( yt 1 , Dt 1 )

6
So the optimality condition requires that:

rt ( xt , yt )  b t 1 ( yt 1 , Pt 1 )  rt ( xt , yt )  b t 1 ( yt 1 , Dt 1 )

b t 1 ( yt 1 , Pt 1 )  b t 1 ( yt 1 , Dt 1 )

This is true for non-negative b by definition of the plan Pt+1. So the optimality condition is
satisfied for rates of return higher than -100%, that means for all practical situations.

Multiplicative returns: f(x,y) = xy

For multiplicative returns under any plan:

 t ( yt , xt  Dt 1 )  rt ( xt , yt )  t 1 ( yt 1 , Dt 1 )

So the optimality condition requires that:

rt ( xt , yt )  t 1 ( yt 1 , Pt 1 )  rt ( xt , yt )  t 1 ( yt 1 , Dt 1 )

For positive returns we get:

 t 1 ( yt 1 , Pt 1 )   t 1 ( yt 1 , Dt 1 )

which is true by definition of the plan Pt+1. So the optimality condition is for positive returns
satisfied.

Counter examples:

1) Let’s consider this objective function to be maximized:

 t ( yt , Dt )  rt ( xt , yt )  t 1 ( yt 1 , Dt 1 )  max{0, t 1 ( yt 1 , Dt 1 )  4}2 , t  1 T  1

This may represent a situation where the immediate return is a certain profit, but there is a
progressive tax (the square of the difference between the profit and some fixed amount). Now
let’s assume, that there are three possible plans Dt+1 with the objective values 3, 4 and 6
respectively. Obviously the third one with the value 6 is optimal at (t+1,yt+1). But the two
right terms in the above objective function

 t 1 ( yt 1 , Dt 1 )  max{0, t 1 ( yt 1 , Dt 1 )  4}2

give for these three plans the values 3, 4 and 2 respectively. So for the stage t the second plan
is the best one, which violates the principle of optimality. In practice, this means that under a
progressive tax the highest profit at stage t+1 is not the best one to maximize the total profit at
stage t.

2) Let’s consider minimization of the maximal return:

7
 F1 ( y1 )  min {max(r1 ( x1 , y1 ), , rT ( xT , yT ))}
X1W1 ( y1 )

This problem is solved for example in job scheduling if maximum lateness of a collection of
jobs has to be minimized. Returns are then lateness' of jobs scheduled at the particular stage =
order. Now assume that the first return is always the greatest one – for example there is a job
with long processing and early due date that always generates the maximum lateness even if
this job is scheduled as the first one. Then all schedules with this particular job at the first
position are optimal irrespective of the order of remaining jobs. So for this problem the
optimality condition (8) holds, but an optimal plan can continue as a plan that is not optimal
with respect to the next stage.
From the second counterexample it follows that the condition (8) is not equivalent with the
principle of optimality. Anyway it is the necessary and the sufficient condition of using the
functional equation of dynamic programming as stated by the following fundamental theorem:

Theorem 1: The optimum

Ft ( yt )  opt t ( rt ( xt , yt ), , rT ( xT , yT ))

X t Wt ( yt )

can be found by the functional equation

Ft ( yt )  opt { f t (rt ( xt , yt ), Ft 1 ( yt 1 ))}

xt  X ( t , yt )

carried out over all relevant states and decisions from given terminal values if and only
if the separability condition (6) and the optimality condition (8) are satisfied.

Proof: The separability condition is clearly necessary for the existence of the function ft. Now
Ft(yt) is by definition the optimal objective at stage t for state yt. So using (6) it takes this
form:

Ft ( yt )  opt { t ( yt , X t )}  opt { f t ( rt ( xt , yt ), t 1 ( yt 1 , X t 1 ))}

X t Wt ( yt ) X t Wt ( yt )

By inserting (6) into (8) we get for minimization:

 t ( yt , xt  Pt 1 )   t ( yt , xt  X t 1 )
ft (rt ( xt , yt ), t 1 ( yt 1 , Pt 1 ))  ft (rt ( xt , yt ), t 1 ( yt 1 , X t 1 ))

By definition of the optimal plan Pt+1 the second argument on the left side is Ft+1(yt+1):

ft (rt ( xt , yt ), Ft 1 ( yt 1 ))  ft (rt ( xt , yt ), t 1 ( yt 1 , X t 1 ))

Now let’s minimize both sides through all plans Xt that can also be expressed as minimization
through all decisions xt and then through all plans Xt+1:

min{min{ ft (rt ( xt , yt ), Ft 1 ( yt 1 ))}}  min{ ft (rt ( xt , yt ), t 1 ( yt 1 , X t 1 ))}

xt X t 1 Xt

8
As the left side does not depend on Xt+1 we get:

min{ ft (rt ( xt , yt ), Ft 1 ( yt 1 ))}  min{ ft (rt ( xt , yt ), t 1 ( yt 1, X t 1 ))}

xt Xt

The right side is by definition the optimum for all plans starting at stage t, so the above
inequality reduces to equality and the condition (8) is sufficient. To show that it is necessary,
let’s assume that (8) is not true. Then there must exist at least one plan Xt+1 that together with
a certain decision xt generates a better objective than xt together with the optimal plan Pt+1:

 t ( yt , xt  Pt 1 )   t ( yt , xt  X t 1 )

Now let’s assume that the decision xt is the optimal one at (t, yt). By applying the functional
equation (1) that takes the objective for Pt+1 we get a wrong suboptimal result, so generally the
condition (8) is necessary. For maximization the proof is analogous.

2.3 Decomposable functions

We have proved separability (6) and validity of the optimality condition (8) for some selected
functions. Next we study the properties of the so-called decomposable functions because
decomposability is a necessary and sufficient condition of optimality. To improve readability
we also simplify the notation. Let the original problem be an unconstrained one with the scalar
objective function of k+1 real arguments:

f ( x, y1 , , yk )  f ( x, y)

The arguments can be in the context of DP interpreted as k+1 stage decisions. Also let's
assume that the function f is separable in the following way:

f ( x, y1 , , yk )  f ( x, y)  f1 ( x, f 2 ( y1, , yk ))  f1 ( x, f 2 ( y))

Using this simplified notation the equation (7) can be reformulated as the equality of two
optimization problems:

opt{ f ( x, y )}  opt{ f1 ( x, opt{ f 2 ( y )})} (9)

x ,y x y

Note that the above equality decomposes a problem in k+1 variables into two smaller
problems with 1 and k variables respectively. The inner problem can be further reduced into 2
problems with 1 and k-1 variables, etc. Finally we obtain k+1 problems of 1 variable each.
Their solution is typically much faster than the solution of the original problem in k+1
variables, especially for finite not very large sets of acceptable decisions. Equation (9) does
not hold generally, but it holds for the so-called decomposable functions:

Definition 1: We say that a real valued multivariate function f is decomposable into f1 and f2 if
f is separable: f(x,y) = f1(x,f2(y)) and if moreover the function f1 is monotone non-
decreasing relative to its second argument.

9
Exercise: Consider the following functions, where y is for simplicity a scalar:

f(x,y) = 3x + 5y f(x,y) = x/(y2 – y) f(x,y) = 3x + 5y – 7xy

Check whether these functions are separable (find the functions f1 and f2 if they
exist) and check whether they are also decomposable.

Theorem 2 called the optimality theorem shows the importance of decomposable functions for
dynamic programming:

Theorem 2: Let f be a real valued multivariate function decomposable into f1 and f2 with
f(x,y) = f1(x,f2(y)). Then the following equality is satisfied

opt{ f ( x, y )}  opt{ f1 ( x, opt{ f 2 ( y )})}

x ,y x y

Proof: To prove an equality a=b we show that both ab and ab is true. We take opt = min.
For f(x,y) = f1(x,f2(y)) by the definition of minimum we have

min{ f1 ( x, f 2 (y))}  f1 ( x0 , f 2 (y 0 )) x0 , y 0

x,y

This inequality holds for any y0, so it holds also for y' that minimizes the inner function f2:

y '  arg min{ f 2 (y )} or f 2 (y ')  min{ f 2 (y)}

y y

where argmin is the minimizer. By inserting into the above inequality we get

min{ f1 ( x, f 2 (y))}  f1 ( x0 , f 2 (y '))  f1 ( x0 , min{ f 2 (y)}) x0

x ,y y

This inequality holds for any x0, so it holds also for x0 that minimizes the right-hand side:

min{ f1 ( x, f 2 (y ))}  min{ f1 ( x, min{ f 2 (y)})}

x,y x y

Now we have to show, that the inequality holds also in opposite direction. By the assumption
of the theorem the function f is decomposable which means that the function f1 is monotone
non-decreasing relative to its second argument. Hence we can write:

f1 ( x0 , min{ f 2 (y )})  f1 ( x0 , f 2 (y 0 )) x0 , y 0

y

This inequality holds for any y0, so it holds also for y0 that minimizes the right-hand side:

f1 ( x0 , min{ f 2 (y)})  min{ f1 ( x0 , f 2 (y))} x0

y y

This inequality holds for any x0, so it holds also for x' that minimizes the right-hand side:

x '  arg min{min{ f1 ( x, f 2 (y))}}

x y

10
 f1 ( x ', min{ f 2 (y)})  min{min{ f1 ( x, f 2 (y))}}  min{ f1 ( x, f 2 ( y))}
y x y x ,y

The left-hand side is the value for a fixed x', so it can not be smaller than its minimum over x:

min{ f1 ( x, min{ f 2 (y )})}  f1 ( x ', min{ f 2 (y)})

x y y

By combining the two inequalities we finally get:

min{ f1 ( x, f 2 (y ))}  min{ f1 ( x, min{ f 2 (y)})}

x,y x y

This completes the proof. For opt = max the proof is analogous. Theorem 2 can easily be
extended to constrained problems:

opt{ f ( x, y ) | ( x, y )  S }  opt { f ( x, y )}
x ,y ( x , y )S

where S is the set of feasible solutions. Let Sx and Sy be the sets of feasible values of x and y:

S x  {x | x  , y  k
: ( x, y )  S} S y  {y | y  k
, x  : ( x, y )  S }

and let Sy(x) be the set of feasible vectors y for a particular x:

S y ( x)  {y | y  k
: ( x, y )  S }

Theorem 3: Let f be a real valued multivariate function decomposable into f1 and f2 with
f(x,y) = f1(x,f2(y)). Then the following equality is satisfied

opt { f ( x, y )}  opt{ f1 ( x , opt { f 2 ( y )})}

( x , y )S xS x yS y ( x )

Proof: Analogous to the proof of Theorem 2. Note: in [1] the outer problem is considered as
an unconstrained one where for an empty set Sy(x) the result of the inner optimization is
defined to be + for minimization and - for maximization respectively.

Examples of decomposable functions:

1) Additive that can be generalized to positive linear combination (discounted)

2) Multiplicative for nonnegative arguments
3) Optimum (minimum or maximum) of real valued functions:

f1 ( x, f 2 (y))  opt{x, f 2 (y)}

In chapter 2.2 the second counterexample shows that for the above function the principle of
optimality is not always satisfied, so the conclusion is that decomposability as defined by
Definition 1 is not a sufficient condition for the validity of the principal of optimality. In order

11
to ensure its validity we have to replace the monotonicity assumption of Definition 1 by strict
monotonicity.

Definition 2: We say that a real valued multivariate function f is decomposable in the strict
sense if f is separable: f(x,y) = f1(x,f2(y)) and if moreover the function f1 is strictly
increasing relative to its second argument:

z1 < z2  f1(x,z1) < f1(x,z2), x

z1 = z2  f1(x,z1) = f1(x,z2), x

It can be shown that strict decomposability is a sufficient condition for the principal of
optimality to be satisfied. Note that the function f1 ( x, f 2 (y))  opt{x, f 2 (y)} is not
decomposable in the strict sense. Also multiplicative function is strictly decomposable for
positive arguments only, which agrees with the conclusion in chapter 2.2 that the optimality
principle is satisfied for multiplicative positive returns.

2.4 Concept of state

In this chapter we shall concentrate on the computational aspect of the evaluation of the DP
functional equation that will justify the existence of stages and states that were intuitively
introduced in chapter 1. For motivation let's consider the equation from Theorem 3 with scalar
y for simplicity:
opt { f ( x, y )}  opt{ f1 ( x , opt { f 2 ( y )})}
( x , y )S xS x yS y ( x )

To solve the decomposed problem, we have to solve the inner problem for each value of x and
then to optimize the outer problem with respect to x. Regarding the computation time such
decomposition brings hardly any improvement. Let's consider the discrete case with S in the
form of a Cartesian product. If x takes M possible values and y takes N values, the full
enumeration optimization of f involves evaluation and comparison of f at MxN points. After
decomposition we solve M times the inner problem involving evaluation and comparison of f2
at N points. Solution to the outer problem then involves evaluation and comparison of f1 at M
points. It is clear that the difference in computation time will not be significant because there
are again MxN + M computational steps (with the inner ones being simpler).
To speed up computation we may utilize the fact that often the inner problem opt{ f 2 ( y)} is in
fact the same for groups of values of x. In other words the feasible values of x generate a
certain number of situations that we shall call states for which we have to solve the inner
problem. If the number of states is K, we solve the inner problem K times which results in the
total number of computational steps KxN + M. Obviously we try to find a state in such a way
that K < M. Introduction of state creates a link between the univariate problem in x and the
multivariate problem in y that we shall soon specify formally. In fact the state is a sort of
output from the problem in x and the input to the problem in y. So far we understand that the
inner problem has to be solved for each state value. Then when solving the outer problem we
find for each value of x the generated state and the corresponding optimum of the inner
problem that we need in order to evaluate the function f1. As the problem in y can be
recursively decomposed in the same way, we eventually obtain the sequence of univariate
problems - stages, each of them generating state values to be considered in the succeeding

12
stage. Now we shall formalize these ideas. Generally we consider a constrained optimization
problem in n variables:
opt{ f (x) | x  S}  opt{ f (x) | g (x)  Yt } (10)

where x  n , S  n is the set of feasible solutions, f : n   is the scalar objective

function, and Yt   is the set of acceptable terminal states: S = {x | g(x)  Yt }. The so-called
transition function g : n   assigns a state to each vector x. So we are searching for such x
that optimizes f provided an acceptable terminal state is reached. Note that for simplicity we
work with scalar states. This can be easily generalized to vector states from m, with only
formal modifications in the formulae. We assume that f is decomposable:

f(x) = f1(x1, f2(x2))

where x = x1 = (x1, x2), x2 = (x2, x3, ... xn). Then we assume that g is separable as follows:

g(x) = g2(x2, g1(x1))

where g1(x1) :    generates a state for each value of x1 and g2(x2, y) : n   generates a
terminal state for each vector x2 and each state y generated by g1. Note that from now we shall
reserve the symbol "x" for decision variables and the symbol "y" for states because most
functions will take these two arguments – see the informal introduction in chapter 1.
Functions g1 and g2 link the feasible set S and the set Y of all possible states involved in the
decomposition:

x1  S x1  {x1 | x 2 : ( x1 , x 2 )  S }  g1 ( x1 )  Y
y  Y , x 2  S x2  {x 2 | x1 : ( x1 , x 2 )  S }  g 2 ( x 2 , y )  Y

Note that S x1 is the set of feasible values of x1 and Sx2 is the set of feasible vectors x2. We can
also define the set Y2 of states generated by g1. Note that we use the subscript 2 because these
states are inputs to the second problem in x2.

Y2  { y | x  S x1 : g1 ( x)  y}

Considering the decomposable f and separable g, we can express the constrained problem (10)
as follows:
opt{ f ( x) | g ( x)  Yt }  opt{ f1 ( x1 , f 2 ( x 2 )) | g 2 ( x 2 , g1 ( x1 ))  Yt }
x x1 , x 2

By applying the Theorem 3 we get this form of the problem:

opt{ f ( x) | g ( x)  Yt }  opt{ f1 ( x1 , opt{ f 2 ( x 2 ) | g 2 ( x 2 , g1 ( x1 ))  Yt }) | x1  S x1 } 

x x1 x2

opt{ f1 ( x1 , opt{ f 2 ( x 2 ) | g 2 ( x 2 , y2 )  Yt }) | x1  S x1 }
x1 x2

where y2 = g1(x1) is the state that parameterizes the inner problem. So we have a family of
parameterized optimization problems in n-1 variables:

13
 F2 ( y2 )  opt{ f 2 ( x 2 ) | g 2 ( x 2 , y2 )  Yt } (11)
x2

If the inner problem is solved for each y2  Y2, or in other words if we know the function F2,
the outer univariate problem takes this form:

opt{ f ( x) | g ( x)  Yt }  opt{ f1 ( x1 , F2 ( g1 ( x1 ))) | x1  S x1 }  opt{ f1 ( x1 , F2 ( y 2 )) | x1  S x1 } (12)

x x1 x1

To evaluate f1 for given x1, we first evaluate y2 = g1(x1), then f2opt(x1) = F2(y2) and finally
f1(x1,f2opt(x1)). All this brings a benefit if for a discrete case K = |Y2| < M = | S x1 |.
Now we need to decompose the problem (11) recursively until we obtain a sequence of
univariate problems. Note that there is a difference between the problems (10) and (11)
because the original problem (10) is not parameterized. In order to achieve a unified structure
of all stages we shall introduce a state also for the first stage. This state y1  Y1, where Y1 is a
given set of parameters (initial states), will parameterize the whole problem to be solved:
F ( y1 )  opt{ f (x) | g (x, y1 )  Yt } . Let's also use the symbol f for multivariate functions. As in
our notation x = x1, we shall express the original objective function f in a way unified for all
stages. So considering the decomposition and applying the same notation for f2, we get:

f (x)  f1 (x1 )  f1 ( x1 , f 2 ( x 2 ))

Similarly for separable parameterized function g:

g (x, y1 )  g1 (x1 , y1 )  g2 (x2 , g1 ( x1 , y1 ))  g2 (x2 , y2 )

Using this notation we can summarize the decomposition based on the concept of state as
follows. We solve a parameterized constrained problem in n variables:

F1 ( y1 )  opt{ f1 (x1 ) | g1 (x1 , y1 )  Yt } (13)

x1

This problem is by assuming decomposability of f1 and separability of g1 decomposed into

two problems: the inner one in n-1 variables and the outer univariate problem:

F2 ( y2 )  opt{ f 2 (x 2 ) | g 2 ( x 2 , y2 )  Yt } (14)
x2

F1 ( y1 )  opt{ f1 ( x1 , F2 ( g1 ( x1 , y1 ))) | x1  S x1 }  opt{ f1 ( x1 , F2 ( y 2 )) | x1  S x1 } (15)

x1 x1

The univariate problem can be expressed as an unconstrained one:

F1 ( y1 )  opt{ f1 ( x1 , F2 ( g1 ( x1 , y1 )))}  opt{ f1 ( x1 , F2 ( y2 ))} (16)

x1 x1

Then if the inner problem is infeasible we set F2(y2) = + for minimization and F2(y2) = -
for maximization respectively. Note that the problems (13) and (14) have the same structure,

14
so we can continue in the decomposition until we obtain a sequence of n univariate problems
with the last one being:

Fn ( yn )  opt{ f n (x n ) | g n (xn , yn )  Yt }  opt{ f n ( xn ) | g n ( xn , yn )  Yt } (17)

xn xn

Note that the above decomposition indeed unifies the structure of all stages. All have the input
and the output state. The input to the first stage parameterizes the problem to be solved, the
output from the last stage is a terminal state. Many practical problems solved by DP are
compatible with these ideas, like for example: optimize inventory policy during a certain
number of periods for given initial and terminal inventory levels; optimize work force strategy
for given initial and terminal numbers of workers, etc. With these assumptions we do not
loose generality. If the whole problem is not parameterized, we can always consider one fixed
(redundant) input state to the first stage. Also the output state from the last stage can be
considered as irrelevant for unconstrained problems. Specification of the problem solved at
each stage is called the functional equation of dynamic programming:

Ft ( yt )  opt{ f t ( xt , Ft 1 ( g t ( xt , yt )))}  opt{ f t ( xt , Ft 1 ( yt 1 ))}, t  1 n 1

xt xt

Fn ( yn )  opt{ f n ( xn ) | g n ( xn , yn )  Yt } (18)
xn

In chapter 1 we considered the same form of the functional equation for all stages. In order to
evaluate the last stage we have defined the function value, here Fn+1, for each terminal state
yn+1. By accepting this assumption we get the following equation for the last stage:

Fn ( yn )  opt{ f n ( xn , Fn 1 ( yn 1 )) | g n ( xn , yn )  Yt }
xn

In the following chapter we introduce the set of feasible states and the set of feasible decisions
which will unify all stages completely. Note the difference between the functional equations
(1) and (18). In (1) we parameterize for generality also the function ft, in equation (3) we
parameterize the return. Exact form of this parameterization is application dependent.

2.5 Multistage decision model

Functional equation (18) considers the unconstrained case except for the last stage constrained
with respect to acceptable terminal states. So without further restrictions the last stage has to
be solved for all states from Y and for all feasible values of xn. For efficient computation we
want to limit the number of states and the number of decisions as much as possible. To cope
with this requirement in our formal model we introduce for each stage the set of possible
states for which the stage has to be solved and for each stage and each state the set of possible
decisions to be considered in optimization. These sets are of course application dependent and
the art of dynamic programming consists in proper definition of the state in order to limit
these two sets as much as possible. Various application dependent restrictions expressed
typically as acceptable ranges of states and/or decisions help a lot. We shall use the DP model
based on the following definition:

15
Definition 3: The so-called Multistage Decision (MSD) model is a collection (T,Y,X,g,Y1,f)
where:

1) T is the number of stages. t  = {1 … T } is the stage number.

2) Y is the set of states expressed as a union of state sets of individual stages:
T 1
Y Yt where YT+1 is the set of terminal states.
t 1

3) X is a mapping X:  Y  2D that assigns a set of acceptable decisions Xt(yt) to each

stage-state pair (t, yt). D   is the decision space.
4) g is the transition function g:  YD  Y that assigns the state yt+1 = gt(yt ,xt) at the
stage t+1 to each stage-state pair (t, yt) and decision xt. yT+1 is the terminal state.
5) Y1   is the given set of initial states. Often only one is given: Y1 = {y1}.
6) f: Y1DT  is the objective function for which the separability and the optimality
conditions are satisfied. To evaluate it we use the functional equation (18).

Note that Definition 3 holds also for models with vector states without any change.

2.6 Backward procedure algorithm

Definition 3 together with the functional equation (18) give the algorithm of the backward
procedure of dynamic programming:

1. Evaluate the function FT+1(yT+1) for all given terminal states yT+1  YT+1.
2. For t = T down to 1 evaluate the stage t by using the functional equation

Ft ( yt )  opt { f t ( xt , Ft 1 ( g t ( xt , yt )))}  opt { f t ( xt , Ft 1 ( yt 1 ))} yt  Yt

xt  X t (yt ) xt  X t (yt )

for all relevant states yt and decisions xt. For each decision xt perform these steps:
- evaluate gt to obtain yt+1
- retrieve Ft+1(yt+1)
- evaluate ft.
For each state yt store the optimal value Ft(yt) and the associated set of optimal
decisions Xtopt(yt).

3. The value F1(y1) is the optimal objective value.

4. Retrieve recursively the set of optimal solutions Xopt(y1) = {xopt} = {(x1, x2, ... xT)}
such that x1  X1opt(y1) and xt  Xtopt(yt), yt = gt-1(yt-1, xt-1) for t = 2 ... T.

Note that in the steps 2 and 4 we cope with the possibility that for a given stage-state pair
there can be more than one optimal decision. Optimal decisions xopt are all paths from y1 to
some terminal state that pass through optimal decisions. Note that the algorithm applies
also to models with vector states without any change.

16
2.7 Progressive problems

So far our assumption was that all T decisions are always performed. Problems based on this
assumption are called serial problems. Passing always through all stages is for some problems
a limiting assumption. There are applications where some stages can be under certain
conditions skipped. We call such problems progressive problems. We assume that in
progressive problems after taking a decision at stage t we can move to any stage u  {t+1,…,
T}. Note that return back before the stage t is not allowed. To cope with progressive problems
we have to generalize the transition function g:  YD   Y that assigns the stage-state
pair (u, yu) = gt(yt ,xt) to each stage-state pair (t, yt) and decision xt.. Note that in this context the
serial problems are a special case of progressive problems with u = t+1. The functional
equations (1), (3) and (18) then take the following forms:

Ft ( yt )  opt { f t ( xt , yt , Fu ( yu )) | g t ( xt , yt )  (u , yu )} (1’)
xt  X t ( yt )

Ft ( yt )  opt { f t ( rt ( xt , yt ), Fu ( yu )) | g t ( xt , yt )  (u , yu )} (3’)
xt  X t ( yt )

Ft ( yt )  opt{ f t ( xt , Fu ( yu )) | g t ( xt , yt )  (u , yu )}, t  1 n 1
xt

Fn ( yn )  opt{ f n ( xn ) | yn 1  Yt , g n ( xn , yn )  (n  1, yn 1 )} (18’)
xn

The algorithm of the backward procedure of dynamic programming applies with obvious
minor modification based on the above functional equations. For each stage, state and
decision we first evaluate the transition function that gives the next stage number u and state
yu. These define the optimal objective Fu(yu) needed for evaluation of ft. Examples of
progressive problems are situations where certain decisions terminate the whole process,
various set partitioning problems or graph related problems with shortcuts. All these result in
varying number of stages taken.

3. Stochastic dynamic programming

All models presented so far are deterministic based on the assumption that parameters of all
functions involved and other figures are known exactly. We know that in practice this is rarely
the case. One solution to the problem of unknown figures is introduction of randomness into
the model. Stochastic programming defines the so-called underlying stochastic program in
this general form with random objective and random feasible set:

opt{f(x,)|x  S()} (19)

where  is a random vector with known distribution. The problem with the stochastic program
(19) is that though syntactically correct it does not have an interpretation (semantics). That's
why there are various deterministic reformulations of this program. We also assume that
decisions are taken before a realization of  is known, so in stochastic programming
terminology we solve the so-called "here and now" problem. Out of various deterministic
reformulations we shall consider the most frequent Expected Objective (EO) reformulation

17
that optimizes the expectation of the objective function. With respect to the Definition 3
randomness can affect various functions and sets. Out of various combinations we shall
consider only few typical cases.

3.1 Random return

We assume that the deterministic return rt(xt, yt) is replaced with the random one rt(xt, yt, ).
The functional equation (3) then takes for EO reformulation the following form:

Ft ( yt )  opt {E ξ [ f t (rt ( xt , yt , ξ ), Ft 1 ( yt 1 ))]} (20)

xt  X t ( yt )

For additive (or additive discounted) ft we use the expectation of the return:

Ft ( yt )  opt {E ξ [rt ( xt , yt , ξ )]  Ft 1 ( yt 1 )} (21)

xt  X t ( yt )

Note that the expectation has to be evaluated for each state and each decision, so practical
requirement is a discrete, preferably scalar  with not many possible values. If s are the
realizations of  with corresponding probabilities ps and S is the total number of realizations,
we solve the following problem:

S
Ft ( yt )  opt { ps rt ( xt , yt ,  s )  Ft 1 ( yt 1 )} (22)
xt  X t ( yt ) s 1

Note that in all above functional equations the next state yt+1 = gt(xt ,yt) is deterministic.
Random return has to be considered in various allocation or investment problems where the
outcome of taking a certain decision, like for example investing a certain amount into a
certain fund, is a priory unknown, but we are able to find, at least approximately, the
distribution of the outcome.

3.2 Random transition

We assume that the deterministic transition function gt(xt ,yt) is replaced with the random one
gt(xt ,yt, ). The functional equation (3) then takes for EO reformulation the following form:

Ft ( yt )  opt {E ξ [ f t ( rt ( xt , yt ), Ft 1 ( g t ( xt , yt , ξ ))]} (23)

xt  X t ( yt )

For additive (or additive discounted) ft we use the expectation of the next optimal objective:

Ft ( yt )  opt {rt ( xt , yt )  E ξ [ Ft 1 ( g t ( xt , yt , ξ ))]} (24)

xt  X t ( yt )

Note that again the expectation has to be evaluated for each state and each decision, so
practical requirement is a discrete, preferably scalar  with not many possible values. If s are
the realizations of  with corresponding probabilities ps and S is the total number of
realizations, we solve the following problem:

18
 S
Ft ( yt )  opt {rt ( xt , yt )   ps Ft 1 ( gt ( xt , yt ,  s ))} (25)
xt X t ( yt ) s 1

Note that in both above functional equations the return rt(xt, yt) is deterministic. It is left to the
reader to combine random return and random transition. Both types of randomness can be also
introduced into progressive models. Note that for random transition both next stage number
and state can be random.

4. Selected applications

To keep unified notation we shall use the symbols from Definition 3 for all models. For each
application we state the assumptions, the MSD model with application dependent details and
some examples. Unless specified otherwise we assume that the functional equation is solved
by the algorithm given in chapter 2.6.

4.1 Allocation problems

Assumptions:
 There are T activities that can be carried out at various levels (numbers of products to be
produced, amounts invested into various opportunities, etc.). xt is the level of activity t. If
possible we try to identify bounds for each activity: lt  xt  ut. Mostly, not necessarily, we
assume at least nonnegativity xt  0.
 There are m resources required to carry out the activities. The m vector b contains the
available amounts of the resources.
 For each activity we know the resource requirements. These can be arranged in an mxT
matrix A where aij is the amount of resource i needed by activity j at level 1. For example
number of machine hours (resource i) needed to produce one unit of product j.
 Contribution from carrying an activity at a certain level is known:

ct(xt) = contribution from activity t at level xt. Ex: ct ( xt )  kt xt

 We want to find such levels of activities that maximize the total contribution provided the
use of resources does not exceed their availability.

MSD model: (T,Y,X,g,Y1,f)

1. Stages: stage = activity t  {1 … T }

2. States: state vector yt = m vector that contains amounts of resources left after allocating
resources to the preceding activity:

T t 1

 liai  y t  b   liai , t  2
i t i 1
T

where ai is the i-th column of the matrix A.

19
3. Decisions: decision xt = level of activity t.

X t ( y t )  { xt | lt  xt  min(ut , wt )}
T
wt  max{ x | xa t  y t   l a },
i  t 1
i i t  1 T 1

wt  max{ x | xa t  y t }, t  T

4. Transition function: gt(xt, yt) = yt - xtat

5. Initial state: y1 = b

6. Functional equation:

Ft ( y t )  max {ct ( xt )  Ft 1 ( y t  xt a t )}, t  1 T

xt  X t ( y t )

FT 1 ( y T 1 )  0
Notes:
1) Bounds on states are given by the following: there must be enough resources for minimum
levels of activity t and all succeeding activities. Upper bound is b less resources needed by
preceding activities at minimum levels.
2) For simplicity the bounds are valid only for nonnegative levels. Negative levels, like for
example short sales in finance, in fact increase the resources available.
3) For decisions the level wt is the maximum level such that enough resources are left for
succeeding activities at minimum levels.
4) For linear contribution the problem can be better solved by linear programming. Use of
dynamic programming is justified for nonlinear contribution and discrete decisions. Note
that also the use of resources can be nonlinear, given for example by a table.
5) The function FT+1 may represent the market value of unused resources.

Examples:

1) A truck can carry a total of 10 tons. Three types of products are available for shipment in
any numbers with following weights and values:

Type Value Weight

A 20 1
B 50 2
C 60 2

Find a loading with maximum possible value provided at least one product of each type is
loaded.

2) Suppose that 4 million is available for investment in three projects. The probability
distribution of the Net Present Value (NPV) earned from each project depends on how
much is invested in each project. The following table gives the distribution for each project
and each investment. We assume that only 1, 2, 3 or 4 million can be invested. If nothing is
invested, the NPV is zero. Example: if one million is invested to the Project 1, there are
three possible outcomes: 2, 4, and 5 million with probabilities 0.6, 0.3 and 0.1 respectively.

20
 Investment Distribution
1 2 / 0.6 4 / 0.3 5 / 0.1
Project 1 2 4 / 0.5 6 / 0.3 8 / 0.2
3 6 / 0.4 7 / 0.5 10 / 0.1
4 7 / 0.2 9 / 0.4 10 / 0.4
1 1 / 0.5 2 / 0.4 4 / 0.1
Project 2 2 3 / 0.4 5 / 0.4 6 / 0.2
3 4 / 0.3 6 / 0.3 8 / 0.4
4 3 / 0.4 8 / 0.3 9 / 0.3
1 0 / 0.2 4 / 0.6 5 / 0.2
Project 3 2 4 / 0.4 6 / 0.4 7 / 0.2
3 5 / 0.3 7 / 0.4 8 / 0.3
4 6 / 0.1 8 / 0.5 9 / 0.4

a) Use stochastic dynamic programming to determine an investment allocation that

maximizes the expected NPV obtained from the three investments.

b) Assume that 5 million is available for investment and at least one million has to be
invested in each project. Find an optimum allocation in this case.

4.2 Work force size

Assumptions:
 The project span is T periods (e.g. weeks).
 During the period t the labor force required is bt. It is possible to have more, so xt  bt is
the actual force kept.
 Salaries of the required workers are ignored in the model (fixed amount).
 Costs involved are the same for all periods. These cost functions are considered:
k x x  0
c1(x) = cost of maintaining the excess x workers. Ex: c1 ( x )   1
 0 x0

k  k x x0
c2(x) = cost of hiring additional x workers. Ex: c2 ( x )   21 22
 0 x0

k x x  0
c3(x) = cost of firing x workers (severance). Ex: c3 ( x )   3
 0 x0
 During the last period there should be bT workers. Their severance (if any) is ignored.
 We want the strategy that minimizes total costs.

MSD model: (T,Y,X,g,Y1,f)

1. Stages: stage = period t  {1 … T }

2. States: state yt = number of workers from the previous period.

Yt = { bt-1, … , max(bt-1, bt, bt+1, … , bT) }, t=2…T

21
3. Decisions: decision xt = number of workers kept at period t.

Xt(yt) = { bt, … , max(bt, bt+1, … , bT) }, t=1…T

4. Transition function: gt(xt, yt) = xt

5. Initial state: y1 = number of workers initially available.

6. Functional equation:

Ft ( yt )  min {c1 ( xt  bt )  c2 ( xt  yt )  c3 ( yt  xt )  Ft 1 ( xt )}, t  1 T

xt  X t ( yt )

FT 1 ( yT 1 )  0

Notes:
1) Cost functions can be more complicated like: “for hiring new workers pay the agency a
fixed amount 100, then 20 per each worker till 10 workers, then 15 for each additional
worker”.
2) Fixed cost functions can easily be relaxed (cost functions constants can change with t).
3) The limits on states and decisions are given by the fact that at any stage the number of
kept workers should not exceed the future required maximum.

Example:
A construction contractor estimates the following work force size over the next 5 weeks: 5,
7, 8, 4 and 6 workers. Excess labor costs 300/worker/week. Hiring costs are 400 fixed cost
plus 200/worker, all in appropriate currency units. There is no severance and the initial
number of workers available is 0. Find the cheapest policy that would satisfy the
requirements.

4.3 Equipment replacement

Assumptions:
 The span is T periods (mostly years).
 After buying a new equipment during the first year of use the age of the equipment is 0,
during the second year it is 1, etc.
 At the beginning of each year there are generally two possible decisions: keep the
equipment for one more year or replace by a new one. There may be a maximum
acceptable age after which the equipment has to be replaced.
 Costs involved:
r(t) = yearly revenue of a t-years-old equipment.

c(t) = yearly operating costs of a t-years-old equipment.

s(t) = salvage (market) value of a t-years-old equipment.

C = cost of a new equipment.

 At the beginning of the first period the age of the equipment is known (0 for a new one).
 Initial investment (if any) is not considered in the model.
 It is assumed that after the last period the equipment is salvaged (sold at market).

22
MSD model: (T,Y,X,g,Y1,f)

1. Stages: stage = period (year) t  {1 … T }

2. States: state yt = age of the equipment at the beginning of the year t. yt  0.

3. Decisions: replace (R) or keep (K). xt  {R, K}. After reaching the maximum age (if any)
the equipment has to be replaced.

 1 , xt  R
4. Transition function: gt ( xt , yt )  
 yt  1 , xt  K

5. Initial state: y1 = age of the equipment at start time.

6. Functional equation:

Ft ( yt )  max ( r ( yt )  c ( yt )  Ft 1 ( yt  1), r (0)  c(0)  s ( yt )  C  Ft 1 (1)), t  1 T

FT 1 ( yT 1 )  s ( yT 1 )

Notes:
1) The first term in max represents keeping the equipment, the second one is replacement. If
replacement is necessary, Ft(yt) is equal to the second term.
2) All costs can also depend on the period, for example the last salvage value can be zero.
3) Drawing a network of possible states may decrease the number of operations.
4) In some models some cost terms may not be used. Alternatively we can minimize total
costs by changing signs of cost terms.

Example:
A company needs to determine the optimal replacement policy for a current 3-years-old
machine over the next 4 years (until the beginning of the year 5). Due to security reasons a
6-years-old machine has to be replaced. New machine costs Lm 100,000. The following
table gives age dependent yearly revenue, operating costs, and salvage value:

Age r(t) c(t) s(t)

0 20,000 200 -
1 19,000 600 80,000
2 18,500 1,200 60,000
3 17,200 1,500 50,000
4 15,500 1,700 30,000
5 14,000 1,800 10,000
6 - - 5,000

Help: the following network gives possible states (ages) that can be reached at various
stages. Note that a computer implementation based on the backward procedure would
probably evaluate all, even not reachable states.

23
4.4 Periodic review inventory

Assumptions:
 Inventory level of a certain commodity is reviewed periodically over a finite number T of
periods (typically months).
 For each period t there is: Period t:

yt = inventory level at the start of the period. xt

dt = demand during the period that has to be satisfied.
xt = amount ordered (produced) during this period. yt dt yt+1

 Costs involved:
Inventory holding cost. For simplicity we assume that inventory (holding) is paid at the
end of the period. Cost paid = ht(yt + xt – dt) = htyt+1

k  k2t xt xt  0
ct(xt) = production cost during the period t. Ex: ct ( xt )   1t
 0 xt  0
where k1t = setup cost in period t, k2t = unit production cost in period t.

24
 The inventory level at the beginning of the first period is given.
 We assume that the required closing inventory is added to the demand of the last period.

MSD model: (T,Y,X,g,Y1,f)

1. Stages: stage = period t  {1 … T }

2. States: state yt = inventory level at the beginning of the period t.

Yt = { 0, … , dt + dt+1 + dt+2 + … + dT) }, t=2…T

3. Decisions: xt = production/order during the period t.

Xt(yt) = { xtmin, … , xtmax }, t=1…T

xtmin = max{0, dt – yt} xtmax = (dt + dt+1 + dt+2 + … + dT) – yt

4. Transition function: gt ( xt , yt )  yt  xt  dt

5. Initial state: y1 = initial inventory level.

6. Functional equation:

Ft ( yt )  min {ct ( xt )  ht yt 1  Ft 1 ( yt 1 )}, t  1 T

xt  X t ( yt )

FT 1 ( yT 1 )  0

Notes:
1) The production cost function can be more complicated, for example a piecewise linear
function with decreasing slope.
2) Note that the limits on state and production/order represent the fact, that it is always
possible to order such quantity that will satisfy the next and all the future demands.
3) The above functional equation will generate empty inventory at the end of the last period.
Note that the closing inventory was added to the demand in the last period.

Example:
Consider the following three-period inventory situation where the costs are represented in
Lm (setup) and Lm/unit (holding):

Period Demand Production Holding

setup cost cost
1 3 3 1
2 2 7 3
3 4 6 2

The starting inventory level is 1 unit; at the end of the third period the level should be zero.
The production cost is the setup plus Lm10/unit for the first three units and then Lm20 for
each additional unit. Find the production strategy that would minimize total costs.

25
  0 xt  0

Help: this is the production cost: ct ( xt )   k1t  10 xt 0  xt  3
 k  30  20( x  3) x  3
 1t t t

4.5 Critical path analysis

Introduction:
Network analysis = group of techniques used to aid management in the planning and control
of projects. Techniques show the inter-relationship of various jobs and clearly identify the
critical parts of the project. They can provide information on time, cost and resource
aspects of a project. Useful for large complex projects with restrictions.

Origin: USA 1958 PERT (Programme Evaluation and Review Technique).

Also called: CPP Critical Path Planning CPS Critical Path Scheduling
CPA Critical Path Analysis CPM Critical Path Method

Basic Terminology:
We generalize the term project to represent any complex activity that can be divided into
simpler sub-tasks that are called activities in Project Management. A project is thus defined by
a set of activities together with their requirements and relationships. In particular:

Activity = task taking time and resources. Using the Graph Theory terminology an
activity is a vertex (node).

Arc = precedence relationship between 2 activities. Precisely an arc (i, j) represents the
fact that the activity i must be completed before the activity j can start.

Network = combination of activities and arcs. Using the Graph Theory terminology a
network is a directed acyclic graph.

The Dynamic Programming algorithm presented in this chapter requires such numbering of
activities that for each arc (i, j), i < j. Since a network is an acyclic graph, it is possible to
assign ranks to vertices (activities). The following is the algorithm that assigns ranks to
vertices:
x=0
while (there are vertices without rank) do
Vertices without predecessors are given the rank x
Remove these vertices together with all incident arcs
x=x+1
endwhile

Then the vertices can be numbered taking one rank at a time and assigning increasing numbers
to vertices within a rank in any order. This will guarantee that for each arc (i, j), i < j.
For the Dynamic Programming algorithm we also define two fictitious activities START and
FINISH with duration 0 that start and finish the whole project. The activity START is the
predecessor of all activities that have no other predecessors; the activity FINISH is the
successor of all activities that have no other successors. After applying the above algorithm
the activity START will have the rank 0, the activity FINISH will have the maximum rank.

26
The above algorithm can be performed directly in the table of activities that defines the project
and at the same time it can be used to draw the network. This is then the algorithm:

x = 0; n = 0; Prepare drawing on the left side

while (there are activities without rank) do
Assign rank x to activities without predecessors
Move to right, draw the activities with rank x in a column
for (each activity a with rank x)
Assign the number n to activity a; n = n+1
Draw arcs from all predecessors of a to a
endfor
Remove the activities with rank x
x=x+1
endwhile

Suggested format for drawing activities: i D START FINISH

EST EFT 0 x
(x = project duration) LST LFT 0 x

Exercise: Draw a network defined by the following table. Hint: don’t forget to add the
activities START and FINISH.

Activity a b c d e f g h i j k l
Preceding activity - a a a b c e f c g, h, i d j, k
Rank
Number

Note: there is a dual, the so-called activity on arcs notation, with activities as arcs and nodes
(events) representing points in time when activities end and start. This notation is probably
more intuitive, but the so-called dummy activities are sometimes needed to keep the
precedence rules. The notation is described in the appendix.

27
Time estimates:
Mostly single time estimates for each activity are considered. Obviously the exact figures are
often not known. We can work with expected values that are often computed from optimistic
(O), most likely (ML), and pessimistic (P) estimates by the following formula:

Expected_duration = (O + 4ML + P)/6

DP assumptions:
 Network with activities START and FINISH exists where for each arc (i, j), i < j.
 Pi is the set of predecessors of the activity i, Si be the set of successors of the activity i.
 D(i) is the duration of the activity i.
 Dynamic programming will be used to find for each activity:

EST (Earliest Start Time) = earliest time when the activity can start.

EFT (Earliest Finishing Time) = earliest time when the activity can finish.

EFT(i) = EST(i) + D(i)

LFT (Latest Finishing Time) = latest time when the activity has to finish in order not
to delay the whole project.

LST (Latest Start Time) = latest time when the activity has to start in order not to delay
the whole project.

LST(i) = LFT(i) - D(i)

 These times are then used to find critical path(s), floats of activities, and they are also
used by crashing and leveling algorithms (not dealt with in this text).

MSD model: (T,Y,X,g,Y1,f)

1. Stages: stage = activity t  {0, 1, … T, T+1 }. Note that 0 and T+1 are the two
fictitious activities with duration 0 that start and end the project.

2. States: Yt = { t }. Degenerate case - only one state for each stage.

3. Decisions: Xt(yt) = Pt (In forward pass that computes EST)

Xt(yt) = St (In backward pass that computes LFT)

4. Transition functions: gt ( xt , yt )  t  1 Forward pass

gt ( xt , yt )  t  1 Backward pass

Note that the transition function is in fact irrelevant because there is only one state per
stage and the problem is a serial one.

28
5. Initial states: yT+1 = T+1 for forward pass, y0 = 0 for backward pass. The initial states are
again irrelevant because there is only one state per stage and the problem is a serial one.

6. Functional equations:

Forward pass - determination of EST and EFT: Ft(t) = EST(t)

EST (0)  EFT (0)  0

EST (t )  max{EST ( xt )  D ( xt )}  max{EFT ( xt )}, t  1 T 1
xt Pt xt Pt

EFT (t )  EST (t )  D (t )

Note that here we apply the so-called forward procedure of dynamic programming that
processes the stages in increasing order.

Backward pass - determination of LST and LFT: Ft(t) = LFT(t)

LFT (T  1)  LST (T  1)  EST (T  1)

LFT (t )  min{LFT ( xt )  D ( xt )}  min{LST ( xt )}, t  T , T  1 0
xt St xt St

LST (t )  LFT (t )  D (t )

After performing the time analysis by the forward and the backward pass, we find additional
useful information about the project:

- Project duration is the EST = EFT of the activity FINISH

- Critical activities are those that cannot be delayed without increasing the project
duration. Critical activities are those where EST = LST (alternatively EFT = LFT).
- Critical path is a sequence of critical activities. Note that there can be more critical
paths in a project, they can split and/or join.
- Floats are spare times of non-critical activities. We define the following floats:

Total float is the time an activity can be delayed without affecting the overall
project duration. This float is sometimes called slack.

Total Float (t )  LFT (t )  EST (t )  D(t )

Free float is the time an activity can be delayed without affecting subsequent
activities.

Free Float (t )  min{EST ( xt )}  EST (t )  D (t )

xt St

Independent Float is the time an activity can be delayed without affecting any
other activities.

Independent Float (t )  min{EST ( xt )}  max{LFT ( xt )}  D (t )

xt St xt Pt

29
Example: Find the Total Float, Free Float and Independent Float of Activity C.

A 3 E 4

8 11 28 32

16 19 28 32
C 9
12 21
19 28
B 3 F 4
9 12 26 30
16 19 31 35

Results: Total Float = 28 – 12 – 9 = 7,

Free Float = 26 – 12 – 9 = 5
Independent Float = 26 – 19 – 9 = -2 (interpreted as 0)

Examples:
Perform time analysis of the following projects:

Activity Duration Preceding activities

1 2 -
2 4 -
3 3 -
4 1 3
5 6 2, 4
6 4 3
7 6 1
8 9 1
9 3 5, 7

Activity Preceding Activity Description Duration (Days)

A - Design Hull 9
B - Prepare Boat Shed 3
C A Design Mast and Mast Mount 8
D A Obtain Hull 2
E A Design Sails 3
F C Obtain Mast Mount 2
G C Obtain Mast 6
H C Design Rigging 1
J B,D Prepare Hull 4
K F,J Fit Mast Mount to Hull 1
L E,H,G,K Step Mast 2
M E,H Obtain Sails and Rigging 3
N L,M Fit Sails and Rigging 4

30
4.6 Traveling salesman

Introduction:
Traveling Salesman (TS) problem can be expressed in various ways, but here we shall assume
the following formulation: Let G = (V, A) be a complete directed graph where every arc aA
connecting the vertices i, j  V (in this order) is assigned a nonnegative real number d(i,j)
called length or distance of the arc. Then the problem is to find a minimum length
Hamiltonian cycle. A Hamiltonian (spanning) cycle visits each vertex exactly once and returns
back to the starting vertex.
Distance of an arc can also represent time or cost. For example in job sequencing problems a
machine has to be adjusted after each operation and the adjustment time/cost depends on the
previous and the next operations. The problem then is to find such sequence of jobs that
would minimize the total time/cost to complete all jobs.
Unfortunately a polynomial algorithm to solve the TS problem is unknown and it is almost
sure that it does not exist. So there are two ways how to solve TS problems: enumeration and
heuristics. For example the so-called nearest neighbor heuristics is based on the following
algorithm: start from any vertex and always travel to the nearest so far not visited vertex. Note
that at stage i there are n-i options.
Complete enumeration has to generate and compare all possible (n-1)! routes that obviously is
not acceptable for bigger n (number of vertices). Note that 100!  93x10157 (10100  70! is
often considered as an upper bound of the number of atoms in visible universe).
Dynamic programming is also an enumeration method, but compared with full enumeration, it
can increase the size of soluble problems. Its performance is O(2n) compared with the
performance O(n!) of complete enumeration. This text is based on TS problem solution that
can be found at http://www.tutor.ms.unimelb.edu.au/tsp/tsp.html where you can find how
many years would take the full enumeration solution and you can solve small problems there.

Assumptions:
 To keep compatibility with our DP notation let’s assume without loss of generality that the
starting (home) vertex has number 0 and that there are another T vertices to be visited (n =
T+1).
 The graph is complete, so there is a possible move between any two vertices in both
directions. Non-existing possibilities can be modeled by very large prohibitive distances.

MSD model: (T,Y,X,g,Y1,f)

1. Stages: stage = number of already visited vertices. t  {1 … T }. t = 1 means starting at

home and selecting the first move. t = T means return home after the last visit. Each stage
adds one arc (vertex) to the cycle.

2. States: state yt = (vt, Vt) where vt is the vertex where we have arrived at stage t and Vt is
the set of vertices to be still visited. The home vertex 0 is not in Vt because it has already
been visited. Clearly y1 = (0, V), yT = (vT, )

3. Decisions: xt = vertex selected for the next move in the stage t.

Xt(yt) = Vt, t = 1…T-1 ; xT = 0 (return home)

31
4. Transition function: gt ( xt ,(t ,Vt ))  ( xt ,Vt \{xt }), t  1 T 1

5. Initial state: y1= (0, V) (we are at starting point, all vertices to be visited)

6. Functional equation:

Ft ((vt , Vt ))  min{d (vt , xt )  Ft 1 (( xt , Vt \ { xt }))}, t  1 T 1

xt Vt

FT ((vT ,  ))  d (vT , 0)

Notes:
1) The function Ft((vt,Vt)) is the shortest path from the vertex vt home (to 0) that goes through
all the vertices from Vt.
2) The algorithm is NP hard because the number of states grows exponentially with T, at
least in “middle” stages. The 1st stage has one state (0, V). The 2nd stage has T states
according to the selection taken in the 1st stage. The 3rd stage has T*(T-1) states, etc.
3) For undirected graphs each optimal cycle will be included twice for both directions.
4) At http://staff.um.edu.mt/jskl1/tsp2.pas you will find a Pascal program that implements
the above DP algorithm. The table of distances can be inserted as a typed constant or it can
be generated randomly. The maximum feasible size is 13 locations except the home.

Example:
Solve the TS problem for the 4 locations A,B,C and D whose distances are given by the
following table. Even if any location can be selected as the home one, you may select A as
the home location.
i/j A B C D
A - 1 2 3
B 1 - 3 5
C 2 3 - 4
D 3 5 4 -

4.7 Scheduling

Introduction:
Job scheduling represents a group of problems related to allocation of jobs to machines with
respect to certain criteria. We shall use dynamic programming to solve one particular
scheduling problem that would be expressed as n/1//T problem. Scheduling problems are
classified by 4 parameters: n/m/A/B where
n is the number of jobs (here any positive integer).
m in the number of machines (here we assume only one machine: m = 1).
A describes the rules (discipline) in the workshop. A is irrelevant for m = 1.
B describes the performance measure by which the schedule is evaluated. Here we minimize
the so-called total tardiness T. To define the tardiness we need parameters that describe
particular jobs. Assuming a job Ji, i = 1 … n, we define these parameters:

ri is the ready time of the job Ji (the time when Ji can start).
di is the due date (time) of the job Ji (the time when Ji is supposed to be completed).

32
ai is the allowance of the job Ji (the period allowed for processing Ji ). ai = di - ri.
pi is the processing time (duration) of the job Ji.
ci is the completion time of the job Ji (the time when processing of Ji finishes).
fi is the flow time of the job Ji (the time that Ji spends in the workshop). fi = ci - ri.
li is the lateness of the job Ji. li = ci - di. Note, that for early jobs it is negative.
ti is the tardiness of the job Ji. ti = max(li , 0). Some authors call this lateness.
ei is the earliness of the job Ji. ei = max(-li , 0).
n
Then T   ti (to get mean tardiness we divide T by n).
i 1

The DP algorithm will be based on the following result related to the n/1//T problem:

Theorem: An optimal sequence of jobs (J1, J2, … Jn) can be decomposed into two parts
A=(J1, J2, … Jk) and B=(Jk+1, Jk+2, … Jn) such that A is optimal for the first k jobs and
k
B is optimal for the last n-k jobs provided they all can start at time: Pk   pi .
i 1

Also the total tardiness T = TA + TB where TA is the total tardiness of the first k jobs
and TB is the total tardiness of the last n-k jobs when started at time Pk.

The proof is obvious. If any of the two sequences A and B were not optimal, the original
sequence would not be optimal either. The second part of the theorem comes directly from the
definition of the total tardiness.

Other assumptions:
 Jobs cannot be canceled; all are processed to completion.
 Jobs cannot be interrupted, once started a job is always completed.
 Processing time is independent of the schedule.
 Machine never breaks down and it is available for the whole scheduling period.
 There is no randomness - all parameters are fixed and known in advance.
 All ready times ri are equal to zero (all jobs can start immediately).

MSD model: (n,Y,X,g,Y1,f)

1. Stages: stage = processing order. t  {1 … n }. Decision at t = 1 means selecting the

first job to be processed, etc. Note that T has been replaced by n because in scheduling T
is the total tardiness.

2. States: state yt = (st, St) where St is the set of jobs that remain to be scheduled and st is the
starting time of the job selected at stage t (the sum of processing times of already
completed jobs):
st   p j
jSt

3. Decisions: xt = job selected for processing at stage t. Xt( yt) = St

4. Transition function: gt ( xt , ( st , St ))  ( st  pxt , St \{xt })

5. Initial state: y1= (0, {1,2, … , n}) (time 0, all jobs to be scheduled)

33
6. Functional equation:

Ft (( st , S t ))  min{max ( st  p xt  d xt , 0)  Ft 1 ( st  p xt , S t \ { xt })}, t  1 T
xt St

FT 1 (( sT 1 ,  ))  0

Notes:
1) The function Ft((st, St)) is the minimum total tardiness of all jobs from St provided their
processing starts at st.
2) Note that in the functional equation the first term is the tardiness of the selected job and
the second term is the total tardiness of the remaining jobs.
3) The algorithm is NP hard because the number of states grows exponentially with n, at least
in “middle” stages. The 1st stage has one state (0, {1,2, … , n}). The 2nd stage has n states
according to the selection taken in the 1st stage. The 3rd stage has n*(n-1) states, etc.
4) Note that sT+1 is the makespan, the time when the last job is completed.

Example:
Solve the 4/1//T problem given by the following table:

Job J1 J2 J3 J4
Processing (pi) 8 6 10 7
Due time (di) 14 9 16 19

All jobs can start at time 0.

5. Performance of the DP algorithm

To assess the speed and memory requirements of the backward procedure algorithm, let's take
a serial problem with n stages, the maximum number of states per stage is m and k is the
maximum number of decisions per stage-state pair. Then the time needed to complete the
algorithm will grow linearly with these 3 figures. Using the "big O" notation we have an
O(nmk) polynomial algorithm. There are applications where one or two of these parameters
are clearly limited. For example in the work force problem both states and decisions are
limited by the maximum work force required. In this case we have an O(n) algorithm with
time growing linearly with the number of stages. Unfortunately the numbers are not always
limited. For example the numbers of states in "middle" stages of TS and scheduling problems
grow exponentially with the number of stages. So there are two types of successful
applications of dynamic programming. They either provide a polynomial algorithm or the
algorithm is still NP hard, but the size of soluble problems is increased compared with other
methods like full enumeration.
Regarding memory requirements we have to store for each stage-state pair the optimal
decision and the associated optimal objective, so we need a total of 2nm memory locations.
Considering the memory capacity of modern computers, this should not be a limitation, unless
we solve a large NP hard problem.

34
References

[1] Minoux, M. (1986) Mathematical Programming - Theory and Algorithms, Wiley.

[2] Sniedovich, M. (1992) Dynamic Programming, Marcel Dekker.
[3] Bertsekas, D.P. (1987) Dynamic Programming: Deterministic and Stochastic Models,
Prentice Hall.
[4] Hastings, N.A.J. (1973) Dynamic Programming with Management Applications, Crane-
Russak.
[5] Bather, J. (2000) Decision Theory. An Introduction to Dynamic Programming and
Sequential Decisions, Wiley.
[6] Denardo, E.V. (1982) Dynamic Programming Models and Applications, Prentice Hall.
[7] Dreyfus, S.E., Law, A.M. (1977) The Art and Theory of Dynamic Programming,
Academic Press.
[8] Norman, J.M. (1975) Elementary Dynamic Programming, Edward Arnold.
[9] Bronson, R (1983) Schaum's Outline Series: Theory and Problems of Operations
Research, McGraw-Hill.
[10] Hillier, F.S., Lieberman, G.J. (1995) Introduction to Operations Research, McGraw-Hill.
[11] Render, B., Stairs, R.M. (1997) Quantitative Tools for Management, Prentice Hall.
[12] Taha, H.A. (2003) Operations Research An Introduction, Prentice Hall.
[13] Winston, W.L. (1994) Operations Research Applications and Algorithms, Duxbury Press.
[14] Pinedo, M. (2002) Scheduling Theory, Algorithms, and Systems, Prentice Hall.

Appendix: Critical path analysis – activity on arcs approach

Here we take a dual view when compared with the activity on nodes approach presented in
chapter 4.5. In addition to activities we also need events and dummy activities:

Activity is a directed arc:

Beginning End

Event is point in time indicating start/finish of activity(ies).

Dummy activity = activity taking no time and no resources. It is used to keep network
properties.

Network = combination of activities, dummy activities and events in logical sequence

drawn according to rules for drawing networks.

Rules for drawing networks:

tail head
- Each activity has one preceding tail
event and one succeeding head event.

- There is exactly one start event (tail event only) and one finish event (head event only):

35
 start finish

- An activity must not share the same tail event and the same head event with another activity.

Solution:

- Closed paths are not allowed.

Network interpretation:
 No activity can start until its tail event is completed (reached).
 An event is not completed until all activities leading to it are finished.

Conventions for drawing networks:

 Network proceeds from left to right.
 Arrows are not drawn to time scale, they need not be necessarily horizontal, but if
possible they should proceed from left to right.
 Events should be numbered progressively from left to right (e.g. 5, 10, 15, 20, ... to
allow later insertions).
A C
Using dummy activities (another example):

Activity Preceding Activity

B D
A, B Irrelevant here
C A A C
D A, B
D
B
Exercise: Draw a network defined by the following table:

Activity 1 2,3,4 5 6 7 8 9 10 11 12
Preceding activity - 1 2 3 5 6 3 7,8,9 4 10,11

DP assumptions:
 Network exists where for each arc (i, j), i < j. To number events, use the algorithm
presented in chapter 4.5.
 Pi is the set of predecessors of the vertex i, Si be the set of successors of the vertex i.
 d(i,j) is the duration of the arc = activity (i, j).
 Dynamic programming will be used to find for each activity:

36
 EST (Earliest Start Time) = earliest time when all activities ending in this event can be
finished = earliest time when the activities starting at this event can start.

LFT (Latest Finishing Time) = time when all activities ending in this event have to be
finished = time when some of the activities starting at this event must start in
order not to delay the whole project.
Drawing:
 Then the EST and the LFT will be used to find: Event
 Critical path(s). EST LFT
 Slacks in events.
 Floats of activities.
 They are also used by crashing and leveling algorithms.

MSD model: (T,Y,X,g,Y1,f)

7. Stages: stage = event t  {1 … T }

8. States: Yt = { t }. Degenerate case - only one state for each stage.

9. Decisions: Xt(yt) = Pt (In forward pass that computes EST)

Xt(yt) = St (In backward pass that computes LFT)

10. Transition function: gt ( xt , yt )  t  1 (Irrelevant - one state only.)

11. Initial state: y1 = 1 (Irrelevant - one state only.)

12. Functional equations:

Forward pass - determination of EST: Ft(t) = EST(t)

Ft (t )  max{d ( xt , t )  Fxt ( xt )}, t  2 T

xt Pt

F1 (1)  0

Note that here we apply the so-called forward procedure of dynamic programming that
processes the stages starting from t=1 in increasing order.

Backward pass - determination of LFT: Ft(t) = LFT(t)

Ft (t )  min{Fxt ( xt )  d (t , xt )}, t  T  1 , T  2 1
xt St

FT (T )  EST (T )

Formulae for floats: Total Float = LFT(Head) – EST(Tail) – D

Free Float = EST(Head) – EST(Tail) – D
Independent Float = EST(Head) – LFT(Tail) – D

Slack in an event = LFT - EST

37