01-Feb-16

OPERATION RESEARCH-2
Dynamic Programming

Prof.Dr.H.M.Yani Syafei,MT

Prof.Dr.Ir.H.M.Yani Syafei,MT

What is The Dynamic ProgrammingLOGO

 Dynamic Programming is a useful mathematical
technique for making a sequence of interrelated
decisions.
 It provides a systematic procedure for determining the
optimal combination of decisions.
 Dynamic programming is a widely-used mathematical
technique for solving problems that can be divided into
stages and where decisions are required in each stage.
 The goal of dynamic programming is to find a
combination of decisions that optimizes a certain
amount associated with a system.
 A Dynamic Programming Model is basically Recursive
Equation linking the different stages of the problem in a
manner that guarantees that each stage`s optimal
feasible solution is also optimal and feasible for the
entire problem.
Prof.Dr.Ir.H.M.Yani Syafei,MT

1
 01-Feb-16

Recursive Nature of Computations LOGO

in DP
 Computations in DP are done recursively, so that the
optimum solution of one subproblem is used as an input
to the next subproblem.
 By the time the last subproblem is solved, the optimum
solution for the entire problem is at hand.
 In contrast to linear programming, there does not exist a
standard mathematical formulation of “the” dynamic
programming problem.
 Rather, dynamic programming is a general type of
approach to problem solving, and the particular
equations used must be developed to fit each situation.
 Therefore, a certain degree of ingenuity and insight into
the general structure of dynamic programming problems
is required to recognize when and how a problem can be
solved by dynamic programming procedures.

Prof.Dr.Ir.H.M.Yani Syafei,MT

CHARACTERISTICS OF DYNAMIC LOGO

PROGRAMMING PROBLEMS
1. The problem can be divided into stages, with a policy
decision required at each stage.
2. Each stage has a number of states associated with the
beginning of that stage.
3. The effect of the policy decision at each stage is to
transform the current state to a state associated with the
beginning of the next stage (possibly according to a
probability distribution).
4. The solution procedure is designed to find an optimal policy
for the overall problem, i.e., a prescription of the optimal
policy decision at each stage for eachof the possible states.
5. Given the current state, an optimal policy for the remaining
stages is independent of the policy decisions adopted in
previous stages.Therefore, the optimal immediate decision
depends on only the current state and not on how you got
there. This is the principle of optimality for dynamic
programming.
Prof.Dr.Ir.H.M.Yani Syafei,MT

2
 01-Feb-16

CHARACTERISTICS OF DYNAMIC LOGO

PROGRAMMING PROBLEMS
6. The solution procedure begins by finding the optimal
policy for the last stage.
 The optimal policy for the last stage prescribes the
optimal policy decision for eachof the possible states
at that stage.
7. A recursive relationship that identifies the optimal
policy for stage n, given the optimal policy for stage n
+1 , is available.
 Therefore, finding the optimal policy decision when you
start in state s at stage n requires finding the
minimizing value of . For this particular problem, the
corresponding minimum cost is achieved by using this
value of and then following the optimal policy when
you start in state at stage n + 1.

Prof.Dr.Ir.H.M.Yani Syafei,MT

CHARACTERISTICS OF DYNAMIC LOGO

PROGRAMMING PROBLEMS
 Notation :
 N = number of stages.
 n = label for current stage (n1, 2, . . . ,N).
 =current statefor stage n.
 =decision variable for stage n.
 ∗ = optimal value of (given ).
 ( , ) = contribution of stages n, n + 1,….. ,N to
objective function if system starts in state at stage n,
immediate decision is , and optimal decisions are
made thereafter.
 ∗ ( ) = ( , ∗ ).

Prof.Dr.Ir.H.M.Yani Syafei,MT

3
 01-Feb-16

CHARACTERISTICS OF DYNAMIC LOGO

PROGRAMMING PROBLEMS
 The recursive relationship will always be of the form :
∗ ( ) = max { ( , ) } or
∗ ( ) = min { ( , )}
 where ( , ) would be written in terms of ,
∗
( ),
 and probably some measure of the immediate
contribution of to the objective function.
 It is the inclusion of ∗ ( ) on the right-hand side, so
that ∗ ( ) is defined in terms ∗ ( ) , that makes
the expression for ∗ ( ) a recursive relationship.

Prof.Dr.Ir.H.M.Yani Syafei,MT

CHARACTERISTICS OF DYNAMIC LOGO

PROGRAMMING PROBLEMS
8. When we use this recursive relationship, the solution
procedure starts at the end and moves backward stage
by stage—each time finding the optimal policy for that
stage— until it finds the optimal policy starting at the
initial stage.
 This optimal policy immediately yields an optimal
solution for the entire problem, namely, ∗ for the initial
state , then ∗ for the resulting state , then ∗ for the
resulting state , and so forth to ∗ for the resulting
stage .

Prof.Dr.Ir.H.M.Yani Syafei,MT

4
 01-Feb-16

LOGO
Case of Shortest-Route Problem

Prof.Dr.Ir.H.M.Yani Syafei,MT

Decomposition of shortest-route problem intoLOGO

stages

Prof.Dr.Ir.H.M.Yani Syafei,MT

5
 01-Feb-16

Solution of Shortest-Route Problem1 LOGO

(Forward Recursion)

Prof.Dr.Ir.H.M.Yani Syafei,MT

LOGO
Solution of Shortest-Route Problem

Prof.Dr.Ir.H.M.Yani Syafei,MT

6
 01-Feb-16

LOGO
Solution of Shortest-Route Problem1

Prof.Dr.Ir.H.M.Yani Syafei,MT

LOGO
Solution of Shortest-Route Problem2

Prof.Dr.Ir.H.M.Yani Syafei,MT

7
 01-Feb-16

Solution of Shortest-Route Problem3 LOGO

(Forward Recursion_

Prof.Dr.Ir.H.M.Yani Syafei,MT

LOGO
Backward Recursion

Prof.Dr.Ir.H.M.Yani Syafei,MT

8
 01-Feb-16

LOGO
Backward Recursion

Prof.Dr.Ir.H.M.Yani Syafei,MT

LOGO
Backward Recursion

Prof.Dr.Ir.H.M.Yani Syafei,MT

9
 01-Feb-16

A typical example: Shortest Path LOGO

• Ben plans to drive from NY to LA

• Has friends in several cities
• After 1 day’s driving can reach Columbus, Nashville, or
Louisville
• After 2 days of driving can reach Kansas City, Omaha, or
Dallas
• After 3 days of driving can reach Denver or San Antonio
• After 4 days of driving can reach Los Angeles
• The actual mileages between cities are given in the figure
(next slide)
• Where should Ben spend each night of the trip to minimize
the number of miles traveled?
19

Shortest Path: network figure LOGO

Columbu 680 Kansas

s City
2 5 610
790
790
1050
550 Denver
8 1030

580 540
New Nashvil Los
900 760 Omaha Angele
York le
6 s
1 3
10
Stage 660 940
1 1390 Stage
5
San
Antonio
770 9
510 790
700
Stage
4
Louisvil 270
830 Dallas
le
7
4

Stage Stage
2 3

20

10
 01-Feb-16

LOGO
Shortest Path problem: Solution
• The problem is solved recursively by working
backward in the network
• Let cij be the mileage between cities i and j
• Let ft(i) be the length of the shortest path from city i
to LA (city i is in stage t)

Stage 4 computations are obvious:

• f4(8) = 1030
• f4(9) = 1390

21

Stage 3 computations LOGO

 Work backward one stage (to stage 3 cities) and find the shortest path to
LA from each stage 3 city.
 To determine f3(5), note that the shortest path from city 5 to LA must be
one of the following:
• Path 1: Go from city 5 to city 8 and then take the shortest path from city
8 to city 10.
• Path 2: Go from city 5 to city 9 and then take the shortest path from city
9 to city 10.
c 58  f 4 (8)  610 1030  1640 *
f 3 (5)  min
c 59  f 4 (9)  790 1390  2180
c 68  f 4 (8)  540 1030  1570 *
f (6)  min
Similarly, c  f (9)  940 1390  2330
3
69 4
c 78  f 4 (8)  790 1030  1820
f 3 (7)  min
c 79  f 4 (9)  270 1390  1660 *
22

11
 01-Feb-16

Stage 2 computations LOGO

 Work backward one stage (to stage 2 cities) and find the
shortest path to LA from each stage 2 city.
c 25  f 3 (5)  680  1640  2320 *

f 2 (2)  min c 26  f 3 (6)  790  1570  2360

c 27  f 3 (7)  1050  1660  2710
c 35  f 3 (5)  580  1640  2220 *

f 2 (3)  min  c 36  f 3 (6)  760  1570  2330

 c 37  f 3 ( 7)  660  1660  2320
c 45  f 3 (5)  510  1640  2150 *

f 2 ( 4 )  min  c 46  f 3 (6)  700  1570  2270

 c 47  f 3 ( 7)  830  1660  2490
23

LOGO
Stage 1 computations

Now we can find f1(1), and the shortest path from NY to LA.
c12  f 2 (2)  550  2320  2870 *

f1 (1)  minc13  f 2 (3)  900  2220  3120

c14  f 2 (4)  770  2150  2920

Checking back our calculations, the shortest path is

1 – 2 – 5 – 8 – 10
that is,
NY – Columbus – Kansas City – Denver – LA
with total mileage 2870.
24

12
 01-Feb-16

LOGO
The Stagecoach Problem
 The STAGECOACH PROBLEM is a problem specially
constructed to illustrate the features and to introduce
the terminology of dynamic programming.
 It concerns a mythical fortune seeker in Missouri who
decided to go west to join the gold rush in California
during the mid-19th century. The journey would require
traveling by stagecoach through unsettled country
where there was serious danger of attack by marauders.
 Although his starting point and destination were fixed,
he had considerable choice as to which states (or
territories that subsequently became states) to travel
through en route. The possible routes are shown in Fig.
11.1, where each state is represented by a circled letter
and the direction of travel is always from left to right in
the diagram.

Prof.Dr.Ir.H.M.Yani Syafei,MT

LOGO
The Stagecoach Problem
 Thus, four stages (stagecoach runs) were required to
travel from his point of embarkation in state A(Missouri)
to his destination in state J(California).
 This fortune seeker was a prudent man who was quite
concerned about his safety. After some thought, he
came up with a rather clever way of determining the
safest route.

Prof.Dr.Ir.H.M.Yani Syafei,MT

13
 01-Feb-16

LOGO
The Stagecoach Problem
 The cost for the standard policy on the stagecoach run
from state i to state j, which will be denoted by , is :

Prof.Dr.Ir.H.M.Yani Syafei,MT

Solution : The Stagecoach ProblemLOGO

Prof.Dr.Ir.H.M.Yani Syafei,MT

14
 01-Feb-16

Solution : The Stagecoach ProblemLOGO

Prof.Dr.Ir.H.M.Yani Syafei,MT

Solution : The Stagecoach ProblemLOGO

Prof.Dr.Ir.H.M.Yani Syafei,MT

15
 01-Feb-16

Solution : The Stagecoach ProblemLOGO

Prof.Dr.Ir.H.M.Yani Syafei,MT

Solution : The Stagecoach ProblemLOGO

Prof.Dr.Ir.H.M.Yani Syafei,MT

16
 01-Feb-16

Solution : The Stagecoach ProblemLOGO

Prof.Dr.Ir.H.M.Yani Syafei,MT

Distributing Medical Teams to LOGO

Countries
 The WORLD HEALTH COUNCIL is devoted to improving health
care in the underdeveloped countries of the world. It now has
five medical teams available to allocate among three such
countries to improve their medical care, health education, and
training programs. Therefore, the council needs to determine
how many teams (if any) to allocate to each of these countries
to maximize the total effectiveness of the five teams.
 The teams must be kept intact, so the number allocated to
each country must be an integer. The measure of performance
being used is additional person-years of life.(For a particular
country, this measure equals the increased life expectancy in
years times the country’s population.) Table 11.1 gives the
estimated additional person-years of life (in multiples of
1,000) for each country for each possible allocation of medical
teams.
 Which allocation maximizes the measure of performance?

Prof.Dr.Ir.H.M.Yani Syafei,MT

17
 01-Feb-16

Distributing Medical Teams to LOGO

Countries

Prof.Dr.Ir.H.M.Yani Syafei,MT

LOGO
Formulation
 This problem requires making three
interrelated decisions, namely, how many
medical teams to allocate to each of the three
countries.
 The decision variables xn (n = 1, 2, 3) are the
number of teams to allocate to stage (country)
n.
 sn = number of medical teams still available
for allocation to remaining countries(n, . . . ,
3).
 To state the overall problem mathematically,
let pi(xi) be the measure of performance from
allocating xi medical teams to country i, as
given table

36

18
 01-Feb-16

0 0
0 0 0 LOGO
20 150 50

1 1 70
0
45 20
105 110

2 2
0
45 20 75
90
75 80
120 45
3 3
0
75 20 45
70
110 100

4 4
0
45 20
130

5 5 5
0 0
37

LOGO
Basic Structure

stage n stage n+1

xn
stage sn sn-xn
pn(xn)

value f n (s n , xn ) f n*1 ( s n  x n )
 f n*1 ( s n  xn )

38

19
 01-Feb-16

LOGO
Formulation
3
Maximize 
i1
pi(xi)
3
subject to 
i1
x i  5
and xi are nonnegative integers. Using the notation
presented in Sec. 11.2, we see that
fn(sn, xn) is
3
f n ( s n , x n )  p n ( x n )  max 
i n 1
pi (xi )
where 3the maximum is taken over xn+1,…,x3 such
that  x i  s n . And the xi are nonnegative integers. In
i n
addition,

f n* ( s n , x n )  max f n (sn , xn )
x n  0 ,1 ,..., s n

39

LOGO
Formulation
Therefore,
f n ( sn , xn )  pn ( xn )  f n*1 ( sn  xn )
Consequently, the recursive relationship relating
functions f * , f * , and f * for this problem is
1 2 3

f n* ( sn )  max
x n  0 ,1,...., s n
 p (x )  f
n n
*
n 1 ( s n  xn )

For the last stage (n=3)

f 3* ( s3 )  max p3 ( x3 )
xn  0 ,1,...., s3

40

20
 01-Feb-16

LOGO
Solution Procedure
0
Stage n = 3 0 0

50

1 70

80

3
100

130

41

LOGO
Solution Procedure

 Stage n = 2
Formula
f 2 (2, x2 )  p2 ( x2 )  f 3* ( 2  x2 )
x2 = 0:
f2 (2,0)  p2 (0)  f3*(2) 
x2 = 1: f2 (2,1)  p2 (1)  f3* (1) 
x2 = 2: f2 (2,2)  p2 (2)  f3*(0) 

Because the objective is maximization,

……… or ……… with ……..

42

21
 01-Feb-16

LOGO
Solution Procedure

43

LOGO
Solution Procedure
Stage n = 2
x2 f 2 (s2 , x2 )  p2 ( x2 )  f 3* (s2  x2 )
s2 f 2* ( s 2 ) x 2*
0 1 2 3 4 5
0
1

2 70 70 45 70 0 or 1

44

22
 01-Feb-16

LOGO
Solution Procedure (n=2)

45

LOGO
Solution Procedure
Stage n = 1
Formula f1 (5, x1 )  p1 ( x1 )  f 2* (5  x1 )
x1  0 : f 1 ( 5 , 0 )  p 1 ( 0 )  f 2* ( 5 ) 
x1  1 : f 1 ( 5 ,1 )  p 1 (1 )  f 2* ( 4 ) 

x1  5 : f 1 ( 5 , 5 )  p 1 ( 5 )  f 2* ( 0 ) 

The similar calculations for x1 =2,

3, 4 (try it) verify that ……..
with …… , as shown in
the following table.

46

23
 01-Feb-16

LOGO
Solution Procedure
Stage n = 1
Formula f1 (5, x1 )  p1 ( x1 )  f 2* (5  x1 )

47

LOGO
Solution Procedure

Stage n = 1

48

24
 01-Feb-16

LOGO
Solution Procedure
*
 Thus, the optimal solution has x1  1 ,
*
which makes s2 =5 –1=4, so x 2  3, which *
makes s3 = 4 – 3, so x 3*  1 . Since, f 1 (5)  170
this (1, 3, 1) allocation of medical teams to
the three countries will yield an estimated
total of 170,000 additional person-years of
life, which is at least 5,000 more than for
any other allocation.
 These results of the dynamic programming
analysis also are summarized in Figure in
the next slide

49

0 0 LOGO
0 0 0
50
( x1*  1 )
50 50
0
1 1 70

20
70

2 2
0
70
45 80

3 3

95 75 80
( x 2*  3 ) 100

4 4
( x 1*  1) 125 100
110
45 130

stage: 5 5 5

170 160 130

50

25
 01-Feb-16

0 0
0 0 0 LOGO
20 150 50

1 1 70
0
45 20
105 110

2 2
0
45 20 75
90
75 80
120 45
3 3
0
75 20 45
70
110 100

4 4
0
45 20
130

5 5 5
0 0
51

Case : Distributing Scientists to LOGO

Research Teams

Prof.Dr.Ir.H.M.Yani Syafei,MT

26
 01-Feb-16

Case : Distributing Scientists to LOGO

Research Teams

Prof.Dr.Ir.H.M.Yani Syafei,MT

LOGO
Formulation

Prof.Dr.Ir.H.M.Yani Syafei,MT

27
 01-Feb-16

LOGO
Formulation

Prof.Dr.Ir.H.M.Yani Syafei,MT

LOGO
Formulation

Prof.Dr.Ir.H.M.Yani Syafei,MT

28
 01-Feb-16

LOGO
Basic Structure

Prof.Dr.Ir.H.M.Yani Syafei,MT

LOGO
Solution
 The resulting dynamic programming calculations are
as follows :
 Stage 3 :

 Stage 2 :

Prof.Dr.Ir.H.M.Yani Syafei,MT

29
 01-Feb-16

LOGO
Solution
 Stage 1 :

Prof.Dr.Ir.H.M.Yani Syafei,MT

LOGO
Optimal Solution

Prof.Dr.Ir.H.M.Yani Syafei,MT

30
 01-Feb-16

LOGO
Problem of Linier Programming

Prof.Dr.Ir.H.M.Yani Syafei,MT

LOGO
Formulation to DP

Prof.Dr.Ir.H.M.Yani Syafei,MT

31
 01-Feb-16

LOGO
Solving

Prof.Dr.Ir.H.M.Yani Syafei,MT

LOGO
Solving

Prof.Dr.Ir.H.M.Yani Syafei,MT

32
 01-Feb-16

LOGO
Solving

Prof.Dr.Ir.H.M.Yani Syafei,MT

LOGO
Optimal Solution

Prof.Dr.Ir.H.M.Yani Syafei,MT

33
 01-Feb-16

LOGO

34