Design & Analysis of Algorithms: Bits, Pilani - K. K. Birla Goa Campus

BITS, PILANI – K. K.
BIRLA GOA CAMPUS
Design & Analysis of Algorithms

(CS F364)
Lecture No. 10
1
Optimal Substructure Property
Optimal Substructure Property/ Principle of
Optimality
• The optimal solution to the original problem

incorporates optimal solutions to the subproblems.
This is a hallmark of problems amenable to dynamic

programming.
• Not all problems have this property.
How to prove Optimal Substructure Property?
Generally by Cut-Paste Argument or By
Contradiction
Note
Optimal Substructure Property looks obvious
But it does not apply to every problem.
Exercise:
Give an problem which does not exhibit Optimal
Substructure Property.
Dynamic Programming Algorithm
The dynamic-programming algorithm can be broken
into a sequence of four steps.
1. Characterize the structure of an optimal solution.
2. Recursively define the value of an optimal solution.
3. Compute the value of an optimal solution in a
bottom-up fashion.
Overlapping subproblems
4. Construct an optimal solution from computed
information.
(not always necessary)
Matrix Chain Multiplication
Recalling Matrix Multiplication
If A is p x q matrix and B is q x r matrix
then the product C = AB is a p x r matrix given by
𝒒
𝒄 𝒊, 𝒋 = ෍ 𝒂 𝒊, 𝒌 𝒃[𝒌, 𝒋]
𝒌=𝟏
where 1≤ i ≤ p and 1≤ j ≤ r
OR
Dot product of ith row of A with jth column of B
Properties
• Matrix multiplication is associative
i.e., A1(A2A3) = (A1A2 )A3
So parenthesization does not change result
• It may appear that the amount of work done
won’t change if you change the
parenthesization of the expression
• But that is not the case!
Example
• Let us use the following example:
– Let A be a 2x10 matrix
– Let B be a 10x50 matrix
– Let C be a 50x20 matrix
• Consider computing A(BC):
Total multiplications = 10000 + 400 = 10400
• Consider computing (AB)C:
Total multiplications = 1000 + 2000 = 3000
Substantial difference in the cost for computing
• Thus, our goal today is:
• Given a chain of matrices to multiply,
determine the fewest number of multiplications
necessary to compute the product.
• Let dixdi+1 denote the dimensions of matrix Ai.
• Let A = A0 A1 ... An-1
• Let Ni,j denote the minimal number of
multiplications necessary to find the product: Ai Ai+1
... Aj.
• To determine the minimal number of multiplications
necessary N0,n-1 to find A,
• That is, determine how to parenthisize the
multiplications
1st Approach –
Brute Force
• Given the matrices A1,A2,A3,A4 Assume the
dimensions of A1=d0×d1 etc
• Five possible parenthesizations of these arrays, along
with the number of multiplications:
(A1A2)(A3A4):d0d1d2+d2d3d4+d0d2d4
((A1A2)A3)A4:d0d1d2+d0d2d3+d0d3d4
(A1(A2A3))A4:d1d2d3+d0d1d3+d0d3d4
A1((A2A3)A4):d1d2d3+d1d3d4+d0d1d4
A1(A2(A3A4)):d2d3d4+d1d2d4+d0d1d4
Questions?
• How many possible parenthesization?
• At least lower bound?
The number of parenthesizations is atleast Ω(2n)
Exercise: Prove
The exact number is given by the recurrence relation
𝑛−1
𝑇 𝑛 = ෍ 𝑇 𝑘 𝑇(𝑛 − 𝑘)
𝑘=1
Because, the original product can be split into two parts
In (n-1) places.
Each split is to be parenthesized optimally
Solution to the recurrence is the famous Catalan
Numbers
T(n) = Ω(4n/3n/2)
Question : Any better approach?
Yes
Dynamic Programming
Step1:
If a particular parenthesization of the whole product is
optimal,
then any sub-parenthesization in that product is optimal as
well.
What does it mean?
– If (A (B ((CD) (EF)) ) ) is optimal
– Then (B ((CD) (EF)) ) is optimal as well
How to Prove?
• Cut - Paste Argument
– Because if it wasn't,
and say ( ((BC) (DE)) F) was better,
then it would also follow that
(A ( ((BC) (DE)) F) ) was better than
(A (B ((CD) (EF)) ) ),
– contradicting its optimality!
Step 2:
Recursive Formulation
Let M[i,j] represent the minimum number of
multiplications required for matrix product Ai ×⋯×
Aj, For 1≤i≤j<n
High-Level Parenthesization for Ai..j
Notation: Ai..j = Ai x ….x Aj
For any optimal multiplication sequence,
at the last step we are multiplying two matrices
Ai..k and Ak+1..j for some k, i.e.,
Ai..j = (Ai x ….x Ak) (Ak+1 x ….x Aj) = Ai..k Ak+1..j
Thus,
M[i,j]=M[i,k]+M[k+1,j]+ di-1dkdj
Thus the problem of determining the optimal
sequence of multiplications is broken down to the
following question?
How do we decide where to split the chain?
OR (what is k)?
Answer:
Search all possible values of k & take the minimum
of it.
Therefore,
0, 𝑖𝑓 𝑖 = 𝑗
𝑀 𝑖, 𝑗 = ቐ min {𝑀 𝑖, 𝑘 + 𝑀 𝑘 + 1, 𝑗 +𝑑 𝑑 𝑑 }, 𝑖𝑓 𝑖 < 𝑗
𝑖−1 𝑘 𝑗
𝑖≤𝑘<𝑗
Step3:
Compute the value of an optimal solution in a
bottom-up fashion
Overlapping Subproblem
Which sub-problems are necessary to solve first?
By Definition M[i,i] = 0
Clearly it's necessary to solve the smaller problems
before the larger ones.
• In particular, we need to know M[i, i+1], the number of
multiplications to multiply any adjacent pair of
matrices before we move onto larger tasks.
Chains of length 1
• The next task we want to solve is finding all the values
of the form M[i, i+2], then M[i, i+3], etc.
Chains of length 2 & then chains of length 3 & so on
That is, we calculate in the order
• This tells us the order in which to build the
table:
By diagonals
Diagonal indices:
• On diagonal 0, j=i
• On diagonal 1, j=i+1
• On diagonal q, j=i+q
• On diagonal n−1, j=i+n−1
Example
• Array dimensions:
• A1: 2 x 3 , A2: 3 x 5 , A3: 5 x 2
• A4: 2 x 4 , A5: 4 x 3
𝑀 2,2 + 𝑀 3,5 +𝑑1 𝑑2 𝑑5
𝑀 2, 5 = 𝑚𝑖𝑛 ൞𝑀 2,3 + 𝑀 4,5 +𝑑1 𝑑3 𝑑5
𝑀 2,4 + 𝑀 5,5 +𝑑1 𝑑4 𝑑5
Table for M[i, j]
𝑴 𝟐, 𝟐 + 𝑴 𝟑, 𝟓 +𝒅𝟏 𝒅𝟐 𝒅𝟓
𝑴 𝟐, 𝟓 = 𝒎𝒊𝒏 ൞𝑴 𝟐, 𝟑 + 𝑴 𝟒, 𝟓 +𝒅𝟏 𝒅𝟑 𝒅𝟓
𝑴 𝟐, 𝟒 + 𝑴 𝟓, 𝟓 +𝒅𝟏 𝒅𝟒 𝒅𝟓
Optimal locations for parentheses:
Idea: Maintain an array s[1….n,1….n] where s[i, j] denotes k
for the optimal splitting in computing Ai..j = Ai..k Ak+1..j
Array s[1….n,1….n] can be used recursively to compute
the multiplication sequence
Table for s[i, j]
The multiplication sequence is recovered
as follows.
s[1, 5] = 3 (A1 A2 A3 ) ( A4 A5)

s[1, 3] = 1 (A1 (A2 A3 ) )
Hence the final multiplication sequence is
(A1 (A2 A3 ) ) ( A4 A5)

Pseudo code
Observations
Question 1
How many subproblems are used in an optimal solution
for the original problem?
• Coin Change Problem
Two Which?
• Matrix Chain Multiplication
Two Which?
Question 2
How many choices we have in determining which
subproblems to use in an optimal solution?
• Coin Change Problem
Two Why?
• Matrix Chain Multiplication
j - i choices for k (splitting the product)
DP – Time Complexity
Intuitively, the running time of a dynamic
programming algorithm depends on two factors:
1. Number of subproblems overall
2. How many choices we have for each subproblem
Matrix multiplication:
– O(n2) subproblems
– At most n-1 choices
Therefore, Time complexity is O(n3) overall

Design & Analysis of Algorithms: Bits, Pilani - K. K. Birla Goa Campus

Uploaded by

Copyright:

Available Formats

Design & Analysis of Algorithms: Bits, Pilani - K. K. Birla Goa Campus

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Design & Analysis of Algorithms: Bits, Pilani - K. K. Birla Goa Campus

Uploaded by

Copyright:

Available Formats

BITS, PILANI – K. K.

BIRLA GOA CAMPUS

Design & Analysis of Algorithms

• The optimal solution to the original problem

This is a hallmark of problems amenable to dynamic

Question : Any better approach?

s[1, 5] = 3 (A1 A2 A3 ) ( A4 A5)

Hence the final multiplication sequence is

(A1 (A2 A3 ) ) ( A4 A5)

You might also like