Dynamic Programming

Introduction:

Dynamic Programming (D.P) is a mathematical technique which deals

with the optimization of multistage decision-making problems. A
multistage decision- making Problem can be separated into a number of
sequential steps and stages, which many be accomplished in one or more
ways. The solution of each stage is a decision and the sequence of
decisions for all the stages constitutes a decision policy Mathematically a
D. P. Problem is a decision making problem in n variables, the problem
being sub deviled into n sub-problems (segments), each sub problem
being a decision making problem in one variable only. The solution to a
D. P problem is achieved sequentially, starting from one (initial) stage to
the next, till the final stage is reached.

Stage-1: A stage rigidifies a portion of the total problem for which a

decision can be taken. At each stage there are a number of alternatives,
and the best out of those is called the stage decision. which may not be
optimal for the stage, but contributes to obtain the optimal decision
policy.

State-2: The condition of the decision process at a stage is called its state.
The variables which specify the condition of the decision process, i.e.,
describe the status of the system at a particular stage are called state
variables. The number of state variables should be as small as possible,
since larger the number of state variables, more complicated is the
decision process.

Principle of optimality: Bellman's principle of optimality states. "An

optimal policy (a sequence of decisions) has the property that whatever
the initial state and decisions are, the remaining decisions must constitute

1
an optimal policy with regard to the state resulting from the first
decision." This principle implies that a wrong decision taken at a stage
does not prevent form taking optimal decisions for the remaining stages.

Decision Tree: A multistage decision system in which each decision and

state variable can take only finite number of values can be represented
graphically by a decision tree.

Stage -1

Stage -2

Stage -3

Circles represent nodes corresponding to stages and lines between circles

denotes arcs corresponding to decisions. The DP technique deals with
such situation by dividing the given problem into sub problems or stages.

Dynamic programming is a mathematical technique dealing with

multistage decision process.

Dynamic programming significant can be given a more name as recersive

optimization.

This technique convents one problem of variables into sub problems

(stages), each in on variable. The optimums solution is obtained in an
stage orderly starting from one manner to the next and is completed till
the final stage is reached.

Stage: The points at which decisions called. as stages. Each stage for are
referred to as stages. Each stage can be thought of having beginning and

2
an end. The different stages come in a sequence, with the ending of a
stage marking the beginning of the immediately succeeding stage.

State: The variable that tit links up two stages is called a state variable. -
any stage, the status of the problem can be described by the values the
state variable can take. These values. are reffered to as states.

While, solving and state the problem we use the concepts of stage and
state. Moreover, the problem is solved stage by stage and to ensure that
suboptimal solution does not result, we cumulated the objective function
value in a particular way.

Working backwards, for every stage, we found the decisions in that stage
that will allow us to reach the final destination optimally, starting from
each of the states of the stage.

These decisions could be taken optimally, without the knowledge of how

we actually reach the different states. This has been stated as the Principle
of optimally in Dynamic literature.

Consider an optimal sub-division problem where a positive quantity b is

to be divided into n parts. The objective is to determine the optimum
subdivision of b in order to maximize the product of n pants, A This
problem can be solved by using the method of Lagrangian multipliers, but
in more complicated example, simultaneous equations. resulting from the
classical calculus approach may be extremely difficult to solve.

Also the calculus approach can not be applicable if non differentiable

fun's are. Involved.

If it is possible to reformulate the n variable as a series of n problems

each in one variable, then computational procedure is expected to be
reduced to some extent.

3
The dynamic programming approach removes these difficulties by:

* First breaking each problems and the problem into st small en sub sub
problem is referred to a stage.

*A stage signifies a portion of the decision problem for which a 'separate'

decision can be taken.

* The resulting decision will be meaningful if it is optimal for the stage it

represents and can directly as a be used directly solution to the problem.

Characteristic of D. P. Problem:

1. The problem can be divided into stages with a policy decision is

required at each stages.

2. Every stages, consists of number of states associated with it. The stages
are the different associated with it possible conditions in which the
system may find itself at the stage of the problem.

3. Decision at each stage converts the current stage into state associated
with the next stage.

4. The state of the system at a stage is described by state variables called

state variables.

5. When the current state is known an optimal policy for the remaining
stages is independent of the policy of the previous ones.

6. The solution procedure begins finding the optimal policy for each state
to the last stage.

7. A recursive relationship which identifies the optimal policy for each

state with n stages remaining given the optimal policy each state, with (n-
1) stages left.

4
8. Using recursive equation approach each time the solution procedure
moves backward stage for the particular stage fill it attains the optimum
policy beginning at the initial stage.

Dynamic Programming Algorithm:

Step1: Identify the decision variables and specify the objective function
to be optimized under certain limitations if any.

Step2: Decompose the given problem into a number of smaller sub

problems. Identify the state variable at each stage.

Step3: Write down the general recursive relationship for computing the
optimal policy. Decide whether forward or backward method is to be
followed to solve the problem.

Step4: Construct appropriate stage to show the required valves of the

return function at each stages.

Step5: Determine the overall optimal policy on decisions and its value at
each. "There may be more than such optimal policy".

Home Work:

1.

Minimize Z= y 21 + y 22+ y 23

Subject to y 1 + y 2 + y 3=10

When (a) y 1 , y 2 , y 3 are non-negative.

(b) y 1 , y 2 , y 3 are non-negative integers.

Solution: (a) When y 1 , y 2 , y 3 are continuous non-negative integers, the

problem can conveniently be solved by the tabular or enumeration
method, treating it as a three stage problem. At stage-1, the state variable
x 1 take any integer value from 0 to 10, with return f 1 ( x 1 )=min ( y 21 )= y12.

5
 0 ≤ y1 ≤ 1 0

At stage-2.

f 2 ( x 2 )=min ( y 22 +f ¿1 ( x 2− y 2 )2 )

0 ≤ y 2 ≤ 10

The computations for 2nd stage are shown below, where the minimum
f 2 ( x 2 ) values are identified by asterisks.

y1 y1
2
0 1 2 3 4 5 6 7 8 9 10

y2 y2
2
0 1 2 3 4 5 6 7 8 9 10
¿ ¿
0 0 0 1 4 9 16 25 36 49 64 81 100
¿ ¿ ¿
1 1 1 2 5 10 17 26 37 50 65 82

¿ ¿
2 4 4 5 8 13 20 29 40 53 68

¿ ¿ ¿
3 9 9 10 13 18 25 34 45 58

¿ ¿ ¿
4 16 16 17 20 25 32 41 52

¿
5 25 25 26 29 34 41 50

6 36 36 37 40 45 52

7 49 49 50 53 58

8 64 64 65 68

9 81 81 82

1 100 10
0 0

The optimal values of f 2 (x 2) are

6
 x2 0 1 2 3 4 5 6 7 8 9 10
¿
f 2 ( x2 ) 0 1 2 5 8 13 1 25 3 41 50
8 2

¿
At stage-3, f 3 ( x 3 )=min¿

=min ( y 3∨f 2 ( x 3− y 3 ) )
2 ¿ 2

x3 0 1 2 3 4 5 6 7 8 9 10

x 3− y 3 10 9 8 7 6 5 4 3 2 1 0

2
y3 0 1 4 9 16 25 3 49 64 8 100
6 1
¿
f 2 ( x 3− y3 ) 50 41 3 25 18 13 8 5 2 1 0
2
¿ ¿
f 3 ( x 3) 50 42 3 34 34 38 4 54 66 8 100
6 4 2

¿
f 3 ( x 3 ) =3 4 for y ¿3=3∨4

¿ ¿
For y 3=3 , f 2 ( x 2 )=25, for which y 1=3 and y 2=4

Or, y 1=4 and y 2=3

¿ ¿
For y 3=4 , f 2 ( x 2 )=18 , for which y 1=3 and y 2=3 .

Thus the minimum value of 34 corresponds to

{ y 1 , y 2 , y 3 }= (3 ,3 , 4 ) , ( 3 , 4 , 3 ) ,(4 ,3 , 3).

2.

7
Use dynamic programming to solve

Maximize Z= x 21+ 2 x 22 +4 x3

Subject to x 1+ 2 x 2 + x3 ≤ 8

And x 1 , x 2 , x 3 ≥ 0 .

Solution: This dynamic programming problems can be treated as a three

stage problem, with state variables y 1 and return f i ( y i ) such that

At stage-3, y 3=x 1 +2 x2 + x 3=8

At stage-2, y 2= y 3 −x3 =x1 +2 x 2

At stage-1, y 1= y 2−2 x 2=x 1

The recursive equations are

f 3 ( y3 ) =m ax x ⁡{4 x3 + f 2 ( y 2 ) }
3

f 2 ( y 2 ) =max x {2 x 22 +f 1 ( y 1 ) }
2

2 2
And f 1 ( y 1 ) =max x {x 1 }=x 1
1

¿ 2
∴ f 1 ( y 1 )=x 1

Since, y 1= y 2−2 x 2
2
And f 1 ( y 1 ) =x1

∴ f 2 ( y 2 )=max x {2 x 22+ x21 }¿ max x {2 x 2 + ( y 2−2 x 2 ) }

2 2
2 2

2 2
Differentiating 2 x 2+ ( y 2−2 x 2 ) w.r.t. x 2 and equating with zero.

y3
4 x 2−4 ( y 2−2 x 2 ) =0∴ x 2=
3

¿ 2 2 1 2
So, f 2 ( y 2 ) =2 x 2 + ( y 2−2 x 2 ) = 3 y 2

516 page and 517 page…. Kindly ektu dheken… ami khubi osustho.

Book OR (Ganesh ) titas

8
R lecture er math…. Ja amr kce nai…