DYNAMIC PROGRAMMING

SUBMITTED BY:
DR. DEBANJANA GHOSH DR. PRABHPREET KAUR DR. PRASHANTH KULALA DR. SANDIP KR NEOGI SHWETNISHA BOSE

DYNAMIC PROGRAMMING
Technique that divides the problem to be solved into a number of sub problems and then solves each sub-problem in such a way that the overall solution is optimal to the original problem. Invented by American mathematician Richard Bellman in the 1950s to solve optimization problems Programming here means planning Dynamic programming is a stage-wise search method suitable for optimization problems whose solutions may be viewed as the result of a sequence of decisions.

MAIN IDEA
 set up a recurrence relating a solution to a larger instance to solutions of some smaller instances  solve smaller instances once  record solutions in a table  extract solution to the initial instance from that table

Example: Fibonacci numbers using recursion

Fibonacci numbers:
F(n) = F(n-1) + F(n-2) F(0) = 0 F(1) = 1

Computing the nth Fibonacci number recursively (top-down): F(n) F(n-1) F(n-2) + F(n-3) + F(n-2) F(n-3) + F(n-4)
Bad algorithm as numerous repetitive calculations are there

...

Example Fibonacci Numbers using Dynamic programming f(5) f(4) f(3) f(2) f(1) f(2) f(2) f(3) f(1)

Calculation from bottom to top and Values are stored for later use This reduces repetitive calculation

CHARACTERISTICS OF DYNAMIC PROGRAMMING

The problem can be divided into stages with a decision required at each stage. In the capital budgeting problem the stages were the allocations to a single plant. The decision was how much to spend. In the shortest path problem, they were defined by the structure of the graph. The decision was were to go next. Each stage has a number of states associated with it. The states for the capital budgeting problem corresponded to the amount spent at that point in time. The states for the shortest path problem was the node reached. The decision at one stage transforms one state into a state in the next stage. The decision of how much to spend gave a total amount spent for the next stage. The decision of where to go next defined where you arrived in the next stage.

 Given the current state, the optimal decision for each of the remaining states does not depend on the previous states or decisions. In the budgeting problem, it is not necessary to know how the money was spent in previous stages, only how much was spent. In the path problem, it was not necessary to know how you got to a node, only that you did. There exists a recursive relationship that identifies the optimal decision for stage j, given that stage j+1 has already been solved. The final stage must be solvable by itself. The last two properties are tied up in the recursive relationships given above.

STEPS OF DYNAMIC PROGRAMMING

IDENTIFY DECISION VARIABLES

SPECIFY OBJECTIVE FUNCTION

DECOMPOSE INTO SMALLER PROBLEMS ( STAGE )

IDENTIFY STATE VARIABLES

STATE THE GENERAL RECURSIVE RELATIONS. ALSO DECIDE FORWARD/BACKWARD

CALCULATE REQUIRED VALUES OF RETURN FUNCTION BY APPROPRIATE FORMULAE AT EACH STAGE

REPEAT AND DETERMINE THE OVERALL OPTIMAL POLICY

THE STAGECOACH PROBLEM

A MINIMUM PATH PROBLEM

Given a series of paths from point A to point B A and B are not directly connected Each path has a value linked to it Find the best route from A to B

The idea for this problem is that a salesman is traveling from one town to another town, in the old west. His means of travel is a stagecoach. Each leg of his trip cost a certain amount and he wants to find the minimum cost of his trip, given multiple paths.

A SAMPLE STAGECOACH PROBLEM

We begin by dividing the problem into stages

 Suppose we are at node i, we want to find the lowest cost route from i to 10 Start at node 10, and work backwards through the network. Define variables such that: cost of travel from node i to node j node chosen for stage n = 1; 2; 3; 4 state

START AT STAGE 1 (THE LAST STAGE), THEN

STATE 8 9

DECISION

10

BEST DECISION

BEST DISTANCE

2+0 4+0

10 10

2 4

AT STAGE 2
STATE

DECISION 8

DECISION 9

BEST DECISION

BEST DISTANCE

5 6 7

1+2=3 6+2=8 3+2=5

4+4=8 3+4=7 3+4=7

8 9 8

3 7 5

APPLICATIONS

This problem can be used in Computer Networks Plane travel

BELLMANS PRINCIPLE OF OPTIMALITY An optimal policy has the property that whatever the initial state and initial decisions are, the remaining decisions must constitute an optimal policy with regard to the state resulting from the first decision

OPTIMALITY PRINCIPLE
Assume that optimal path is the shortest one OP indicates that any portion of any optimal path is also optimal
A
2 2 1 4

B
3

C
2 1

Optimal path between A and F is A-B-C-E-F

F

Then, the optimal path between B and F is B-C-E-F

Set of optimal paths from all sources to a given destination forms a tree that is routed at the destination

PRINCIPLE OF OPTIMALITY
Jab Jbe b Jbce

c Assertion: If abe is the optimal path from a to e, then be is the optimal path from b to e. Proof: Suppose it is not. Then there is another path (note that existence is assumed here) bce which is optimal from b to e, i.e. Jbce > Jbe but then Jabe = Jab+Jbe< Jab+ Jbce= Jabce This contradicts the hypothesis that abe is the optimal path from a to e.

PRINCIPLE OF OPTIMALITY
4 2 5 9 1 3 6 8 7

Bellmans Principle of Optimality: At any intermediate state xi in an optimal path from x0 to xf, the policy from xi to goal xf must itself constitute optimal policy

BELLMANS PROGRAMMING
Transforms a complex optimization problem into a sequence of simpler ones. Usually begins at the end and works backwards Can handle a wide range of problems Relies on recursion, and on the principle of optimality Developed by Richard Bellman

MARKOVIAN REQUIREMENT
An optimal policy starting in a given state depends only on the state of the process at that stage and not on the state at preceding stages. The path is of no consequence, only the present stage and state The state variables fully describe the state of the system at each stage and capture all past decisions

Optimal Capacity Expansion: What is the least cost way of building plants?

Cost of $15 million in any year in which a plant is built. At most 3 plants a year can be built
24

Forwards recursion
Start with stage 1 and determine values for larger stages using recursion. In forward recursion f(n) is determined from f(k) where k < n. Determine the value of the other stages using the value of the first stage with the help of forward recursion.

25

Backwards Recursion
Boundary conditions: optimum values for states at the last stage. In backward recursion value of f(k) is determined from f(n) where k<n, i.e. k is the first stage and n is the last stage. Determine optimum values for other stages using backwards recursion

26

Determining state and stage

Stage: year Y State: number of plants f(j, Y) = optimum cost of capacity expansion starting with j plants at the end of year Y. We want to compute f(0, 2002). We will first compute what is f(j, 2007) for different j.
27

Determining state and stage

From f(j,2007) for each j we will compute f(j, 2006) for each j. After that we will complete f(j, 2005) for each j. Then compute f(j, 2004) for each j using f(j,2005). From f(j,2004) f(j,2003) for each j will be computed. Lastly from f(j,2003) f(j,2002)can be calculated.
28

TRAVELING SALESMAN

2 Start 1 East 4 3

5 6 7 8 9 End 10 West

INSURANCE POLICY COST

T O 2 F 1 2 R O M 3 4 4 3 5 6 7 2 7 4 6 5 3 3 2 4 6 4 4 1 5 7 8 9 10

1 4 8 3 6 3 9 5 3 3

Minimize the cost for each run

1 2 6 9 10 Total cost = 14 However, 1 4 6 9 10 Total cost = 12 !

CONDITIONS
Those that can satisfy Bellmans principle of optimality:
The globally optimum solution includes no suboptimal local

decision. or in Bellmans original words from 1957:

 An optimal policy has the property that, regardless of the

decisions taken to enter a particular state, the remaining decisions made for leaving that stage must constitute an optimal policy.

APPLICATIONS
In numerous multistage decision problems of optimum fit such as:
 Error correction coding (Viterbi algorithm).  Automatic labelling of speech segments (dynamic time warping).  Video coding.  Digital watermark detection.  Cell library binding (as part of logic optimization).  Flight trajectory planning.  Genome sequencing (Smith-Waterman and Needleman-Wunsch algorithms).  Knapsack problems.  Stereo vision.

Dynamic Programming v/s Linear Programming

Dynamic Programming is different from Linear Programming on two counts. 1). There does not exist a standard mathematical formulation of Dynamic Programming problem. There is no algorithm like Simplex method , that may be programmed to solve all the problems. Instead, Dynamic Programming is a technique which dissects difficult problems into a sequence of sub- problems, which are then evaluated by stages. 2). While Linear Programming is a method which gives single stage, i.e. one time period solutions. But Dynamic Programming ahs the power to determine the optimal solution e.g. one year time horizon divided into 12 months time horizon. Thus , it is a multistage approach to solve a problem.

Solving linear programming by dynamic programming

Eg: Maximize Z= c1x1+c2x2+ Subject to the constraints aixi+aix+ ..+ax<=bi +cnxn

Where i= 1,2, .m And x>=0 where j=1,2 n

Consider this LPP as multistage problem with each activity j (j = 1,2, ..n) as individual stage so this is an n- stage problem

SUMMARY
The Dynamic Programming technique is useful for making a sequence of inter-related decisions where the objective is to optimize the overall outcome of the entire sequence of decisions over a period of time. This technique decomposes the original problem into a sequence of stages, which can then be handled more efficiently from the computational point of view. In contrast, most linear and nonlinear programming approaches attempt to solve such problems by considering all the constraints simultaneously. The dynamic programming technique is used in various fields such as inventory control, allocation, Bidding problems etc.