CS 332 - Design and Analysis of Algorithms

Substitution Method
Master Theorem

2
How to determine complexity of code structures
If Statement: Take the complexity of the most expensive case :
char key;
int[][] A = new int[5][5];
int[][] B = new int[5][5];
int[][] C = new int[5][5];
........
if(key == '+') {
for(int i = 0; i < n; i++)
for(int j = 0; j < n; j++) O(n2)
C[i][j] = A[i][j] + B[i][j];
} // End of if block
Overall
complexity
else if(key == 'x')
O(n3)
C = matrixMult(A, B);
O(n3)
else
System.out.println("Error! Enter '+' or 'x'!");
O(1)
 1 n=0
T(n) =
Solving the Recurrence Relation T(n) = T(n-1) + 1 n>0

T(n) = T(n-1) + 1 n>0

Substitution Method
Put n=n-1 both Sides
T(n) = T(n-1) + 1 n>0 T(n-1) = T(n-1-1) + 1

T(n-1) = T(n-2) + 1
T(n) = {T(n-2) + 1} + 1
Put n=n-1 both Sides
T(n) = T(n-2) + 2
T(n-1-1) = T(n-1-2) + 1
T(n) = [T(n-3) + 1 ] + 2
T(n-2) = T(n-3) + 1
T(n) = T(n-3) + 3
 1 n=0
T(n) =
Solving the Recurrence Relation T(n) = T(n-1) + 1 n>0

Substitution Method

T(n) = T(n-1) + 1 n>0

T(n) = T(n-2) + 2

T(n) = T(n-3) + 3

T(n) = T(n-4) + 4

T(n) = T(n-5) + 5

T(n) = T(n-k) + k
Solving the Recurrence Relation
A time will come k=n
Substitution Method
So we can put k=n
T(n) = T(n-1) + 1 n>0
T(n) = T(n-n) + n
T(n) = T(n-2) + 2
T(n) = T(0) + n
T(n) = T(n-3) + 3

T(n) = T(n-4) + 4

T(n) = T(n-5) + 5

T(n) = 1 + n
T(n) = T(n-k) + k
T(n) = theta(n)
Example
Recurrence Relation Solution
– Recursion Tree T(n)

1 n=0 n T (n-1)
T(n) =
T(n) = T(n-1) + n n>0
n-1 T (n-2)

n-2 T (n-3)

n-k T (n-k-1)

1 T (0)
Recurrence Relation Solution – Recursion Tree 1 n=0

T(n) n T(n) =
T(n) = T(n-1) + n n>0

n T (n-1) n-1

n-1 T (n-2) n-2

n-2 T (n-3) n-3

n-k T (n-k-1) n-k

1 T (0) 0
Recurrence Relation Solution – Recursion Tree 1 n=0

T(n) n T(n) =
T(n) = T(n-1) + n n>0

n T (n-1) n-1

n-1 T (n-2) n-2

n-2 T (n-3) n-3

T(n)= 1+2+3+4 . . .+ n-1+ n

n-k T (n-k-1) n-k
T(n)= n(n+1)/2
1
T(n)= Theta (n^2)
1 T (0) 0
 The M a s te r Theorem

The M a s t e r Theorem is a theorem

for asymptotically bounding recurrences of the
type we've seen so far.
Intuitively, categorizes recurrences into one of
the three groups just mentioned, then
determines the runtime based on that
category.
 The M a s te r Theorem
T h e o r e m : Let T(n) be defined as follows:
T(1) ≤ Θ(1)
T(n) ≤ aT(⌈n / b⌉) + O (n d )
Then

O (n d ) if log b a < d
T(n) = O (n d log n) if log b a =
log b a
O (n )
d if log b a
> d
 The M a s te r Theorem

• Let a>=1 and b > 1 be constants, let f (n) be a function, and let T (n) be
defined on the nonnegative integers by the recurrence

T h e o r e m : Let T(n) be defined as follows:

T(1) ≤ Θ(1)
T(n) ≤ aT(⌈n / b⌉) + O (n d )

12
 Solving Existing Recurrences
● Consider the mergesort recurrence

T(0) = Θ(1)
T(1) = Θ(1)
T(n) = T(⌈n / 2⌉) + T(⌊n / 2⌋) + Θ (n)
 Solving Existing Recurrences
● Consider the mergesort recurrence

T(0) = Θ(1)
T(1) = Θ(1)
T(n) ≤ 2T(⌈n / 2⌉) + Θ (n)

● What are a, b, and d? a = 2 , b = 2 , d = 1 .

● What is log b a? 1
●
By the M a s te r Theorem, T(n) = O ( n l o g
n).
 Solving Existing Recurrences
● Consider the mergesort recurrence

T(0) = Θ(1)
T(1) = Θ(1)
T(n) ≤ 2T(⌈n / 2⌉) + Θ (n)

● What are a, b, and d? a = 2 , b = 2 , d = 1 .

● What is log b a? 1
●
By the M a s te r Theorem, T(n) = O ( n l o g
n).
 Solving Existing Recurrences
● Consider the mergesort recurrence

T(0) = Θ(1)
T(1) = Θ(1)
T(n) ≤ 2T(⌈n / 2⌉) + Θ (n)

● What are a, b, and d? a = 2 , b = 2 , d = 1 .

● What is log b a? 1
●
By the M a s te r Theorem, T(n) = O ( n l o g
n).
 Solving Existing Recurrences
● Consider the mergesort recurrence

T(0) = Θ(1)
T(1) = Θ(1)
T(n) ≤ 2T(⌈n / 2⌉) + Θ (n)

● What are a, b, and d? a = 2 , b = 2 , d = 1 .

● What is log b a? 1
●
By the M a s te r Theorem, T(n) = O ( n l o g
n).
 Solving Existing Recurrences
● Consider the mergesort recurrence

T(0) = Θ(1)
T(1) = Θ(1)
T(n) ≤ 2T(⌈n / 2⌉) + Θ (n)

● What are a, b, and d? a = 2 , b = 2 , d = 1 .

● What is log b a? 1
●
By the M a s te r Theorem, T(n) = O ( n l o g
n).
 Solving Existing Recurrences
● Consider the mergesort recurrence

T(0) = Θ(1)
T(1) = Θ(1)
T(n) ≤ 2T(⌈n / 2⌉) + Θ (n)

● What are a, b, and d? a = 2 , b = 2 , d = 1 .

● What is log b a? 1
●
By the M a s te r Theorem, T(n) = O ( n l o g
n).
 Solving Existing Recurrences
●
Consider the weakly unimodal maximum
recurrence:
T(1) ≤ c
T(n) ≤ 2T(n / 2) + c
● What are a, b, d? a = 2 , b = 2 , d = 0
● What is log b a? 1
●
By the M a s te r Theorem, T(n) = O ( n )
 Solving Existing Recurrences
●
Consider the weakly unimodal maximum
recurrence:
T(1) ≤ c
T(n) ≤ 2T(⌈n / 2⌉) + c
● What are a, b, d? a = 2 , b = 2 , d = 0
● What is log b a? 1
●
By the M a s te r Theorem, T(n) = O ( n )
 Solving Existing Recurrences
●
Consider the weakly unimodal maximum
recurrence:
T(1) ≤ c
T(n) ≤ 2T(⌈n / 2⌉) + c
● What are a, b, d? a = 2 , b = 2 , d = 0
● What is log b a? 1
●
By the M a s te r Theorem, T(n) = O ( n )
 Solving Existing Recurrences
●
Consider the weakly unimodal maximum
recurrence:
T(1) ≤ c
T(n) ≤ 2T(⌈n / 2⌉) + c
● What are a, b, d? a = 2 , b = 2 , d = 0
● What is log b a? 1
●
By the M a s te r Theorem, T(n) = O ( n )
 Solving Existing Recurrences
●
Consider the weakly unimodal maximum
recurrence:
T(1) ≤ c
T(n) ≤ 2T(⌈n / 2⌉) + c
● What are a, b, d? a = 2 , b = 2 , d = 0
● What is log b a? 1
●
By the M a s te r Theorem, T(n) = O ( n )
 Solving Existing Recurrences
●
Consider the weakly unimodal maximum
recurrence:
T(1) ≤ c
T(n) ≤ 2T(⌈n / 2⌉) + c
● What are a, b, d? a = 2 , b = 2 , d = 0
● What is log b a? 1
●
By the M a s te r Theorem, T(n) = O ( n )
 Solving Existing Recurrences
●
Consider the weakly unimodal maximum
recurrence:
T(1) ≤ c
T(n) ≤ 2T(⌈n / 2⌉) + c
● What are a, b, d? a = 2 , b = 2 , d = 0
● What is log b a? 1
●
By the M a s te r Theorem, T(n) = O ( n )
 Solving Existing Recurrences
●
Consider the recurrence for the code to
find the maximum value in an array:
T(1) ≤ c
T(n) ≤ T(⌈n / 2⌉) + c n
● What are a, b, d? a = 1 , b = 2 , d = 1
● What is log b a? 0
●
By the M a s te r Theorem, T(n) = O ( n )
 Solving Existing Recurrences
●
Consider the recurrence for the code to
find the maximum value in an array:
T(1) ≤ c
T(n) ≤ T(⌈n / 2⌉) + c n
● What are a, b, d? a = 1 , b = 2 , d = 1
● What is log b a? 0
●
By the M a s te r Theorem, T(n) = O ( n )
 Solving Existing Recurrences
●
Consider the recurrence for the code to
find the maximum value in an array:
T(1) ≤ c
T(n) ≤ T(⌈n / 2⌉) + c n
● What are a, b, d? a = 1 , b = 2 , d = 1
● What is log b a? 0
●
By the M a s te r Theorem, T(n) = O ( n )
 Solving Existing Recurrences
●
Consider the recurrence for the code to
find the maximum value in an array:
T(1) ≤ c
T(n) ≤ T(⌈n / 2⌉) + c n
● What are a, b, d? a = 1 , b = 2 , d = 1
● What is log b a? 0
●
By the M a s te r Theorem, T(n) = O ( n )
 Solving Existing Recurrences
●
Consider the recurrence for the code to
find the maximum value in an array:
T(1) ≤ c
T(n) ≤ T(⌈n / 2⌉) + c n
● What are a, b, d? a = 1 , b = 2 , d = 1
● What is log b a? 0
●
By the M a s te r Theorem, T(n) = O ( n )
 Solving Existing Recurrences
●
Consider the recurrence for the code to
find the maximum value in an array:
T(1) ≤ c
T(n) ≤ T(⌈n / 2⌉) + c n
● What are a, b, d? a = 1 , b = 2 , d = 1
● What is log b a? 0
●
By the M a s te r Theorem, T(n) = O ( n )
 Proving the M a s te r Theorem
●
We c a n prove the M a s te r Theorem
by writing out a generic proof using
a recursion tree.
● Draw out the tree.
● Determine the work per level.
● S u m across all levels.
●
The three cases of the M a s te r Theorem
correspond to whether the recurrence is
topheavy, balanced, or bottomheavy.
 Simplifying the Recurrenc e
●
The recurrence given by the M a s te r
Theorem is shown here:
T(1) ≤ Θ(1)
T(n) ≤ aT(⌈n / b⌉) + O (n d )
● We will apply our standard simplifications
to this recurrence:
● Assume inputs are powers of b.
● Replace Θ and O with constant multiples.
T(1) ≤ c
T(n) ≤ aT(n / b) + c n d
 Simplifying the Recurrenc e
●
The recurrence given by the M a s te r
Theorem is shown here:
T(1) ≤ Θ(1)
T(n) ≤ aT(⌈n / b⌉) + O (n d )
●
We will apply our standard simplifications
to this recurrence:
● Assume inputs are powers of b.
● Replace Θ and O with constant multiples.
T(1) ≤ c
T(n) ≤ aT(n / b) + c n d
T(1) ≤ c
T(n) ≤ aT(n / b) + c n d

cn d cn d

c (n / b)d c (n / b)d ... c (n / b)d ac (n / b)d

c (n / b 2 )d c (n / b 2 )d c (n / b 2 )d a2 c (n / b 2 )d

c (n / b 3 )d c (n / b 3 )d c (n / b 3 )d a3 c (n / b 3 )d
...

log b n
c c ... c ca