Algorithms and Complexity

Speed up recursive algorithms

Lhouari Nourine

Master ICS

Algorithms and Complexity Speed up recursive algorithmsMaster ICS 1 / 29

Recursive algorithms

"Thinking recursively ......is to imagine a tree"

Recursion is a method where the solution to a problem depends on
solutions to smaller instances of the same problem
or, in other words, a programming technique in which a method can
call itself to solve a problem.

Algorithms and Complexity Speed up recursive algorithmsMaster ICS 2 / 29

Thinking recursively for sorting
Algorithm 1: MERGE − SORT (A, l, r )
begin
if l < r then
mid = (l + (r − l)/2);
MERGE − SORT (A, l, mid);
MERGE − SORT (A, mid + 1, r );
Merge(A, I , mid, r );

Algorithms and Complexity Speed up recursive algorithmsMaster ICS 3 / 29

The recursion tree of Merge-Sort

Algorithms and Complexity Speed up recursive algorithmsMaster ICS 4 / 29

The recursion tree of Fibonacci sequences
Algorithm 2: Fib(n)
begin
if n ≤ 1 then
return(n);
else
return(Fib(n − 1) + Fib(n − 2));

Algorithms and Complexity Speed up recursive algorithmsMaster ICS 5 / 29

Time complexity of recursive algorithms

Using the recursion tree:

⇒ Number of nodes = Numbers of calls.

⇒ The cost of a call is the time complexity of the call outside the recursive
steps.

1 T (n) = #nodes × worst cost of a call.

2 T (n) = #levels × worst cost of a level.
3 #levels = the longest path from the root to a leaf.

Algorithms and Complexity Speed up recursive algorithmsMaster ICS 6 / 29

Time complexity of recursive algorithms: Merge-Sort

Algorithms and Complexity Speed up recursive algorithmsMaster ICS 7 / 29

Time complexity of recursive algorithms: Fib

Algorithms and Complexity Speed up recursive algorithmsMaster ICS 8 / 29

Time complexity of recursive algorithms

Using the recurrence equation:

1 Associate the recurrence equation corresponding to the algorithm.
2 Solve or bound the solution of the equation, i.e. T (n) ∈ O(g (n)).
3 Use the master theorem for the equations of the following form:
T (n) = aT ( bn ) + f (n), where a ≥ 1 and b > 1.
I a is the number of subproblems in each recursive step, i.e. the number
of children of a node.
I b is the amount we are reducing the subproblems by. bn is the size of
each subproblem.
I f (n) is the time complexity of a the call with n outside the recursive
steps.

Algorithms and Complexity Speed up recursive algorithmsMaster ICS 9 / 29

Time complexity of recursive algorithms: Merge-Sort

Algorithm 3: MERGE − SORT (A, l, r )

begin
if l < r then
mid = (l + (r − l)/2);
MERGE − SORT (A, l, mid);
MERGE − SORT (A, mid + 1, r );
Merge(A, I , mid, r );

Recurrence equation:
T (1) = 1
T (n) = 2T ( n2 ) + n, n ≥ 2

Algorithms and Complexity Speed up recursive algorithms

Master ICS 10 / 29
Time complexity of recursive algorithms: Fib

Algorithm 4: Fib(n)
begin
if n ≤ 1 then
return(n);
else
return(Fib(n − 1) + Fib(n − 2));

Recurrence equation:
T (0) = T (1) = 1
T (n) = T (n − 1) + T (n − 2), n ≥ 2

Algorithms and Complexity Speed up recursive algorithms

Master ICS 11 / 29
Time complexity of recursive algorithms: Master Theorem

The Master Theorem applies to recurrences of the following form:

T (n) = aT ( bn ) + f (n), where a ≥ 1 and b > 1. are constants and f (n) is
an asymptotically positive function. There are 3 cases:
1 If f (n) = O(nlogb a−) for some constant  > 0, then T (n) = θ(nlogb a ).
2 If f (n) = θ(nlogb a ) then T (n) = θ(nlogb a logn).
3 If f (n) = Ω(nlogb a+ ) with  > 0 then T (n) = θ(f (n)).

Algorithms and Complexity Speed up recursive algorithms

Master ICS 12 / 29
Time complexity of recursive algorithms

From a recurrence equation of the form T (n) = aT ( bn ) + f (n), where

a ≥ 1 and b > 1 to a recursive algorithm:
Given a recurrence equation T (n) we can associate a recursive algorithm
with time complexity O(T (n)).

Example: T (n) = 2T ( n3 ) + T ( n2 ) + n2

Algorithms and Complexity Speed up recursive algorithms

Master ICS 13 / 29
Examples of recursive algorithms

1 Shortest path between two vertices in a weighted directed graphs.

2 Counting Inversions: number of inversions between two rankings.
My rank: 1, 2, . . . , n.
Your rank: a1 , a2 , . . . , an .
i and j inverted if i < j, but ai > aj .
3 Closest Pair of Points: Given n points in the plane, find a pair with
smallest Euclidean distance between them. No two points have same
x coordinate.
4 Matrix multiplication. Given two n-by-n matrices A and B, compute
C = AB.

Algorithms and Complexity Speed up recursive algorithms

Master ICS 14 / 29
Speed up recursive algorithms

Idea:
1 Reuse already computed functions.
2 Cut the unnecessary subtrees.

Difficulty: How to recognize easily that a function with given parameters

has been already computed?

Algorithms and Complexity Speed up recursive algorithms

Master ICS 15 / 29
Speed up recursive algorithms

The difficulty depends on types of parameters:

1 The function has only one integer parameter n:
Use an array of size n.
2 The function has two integers parameters n and m:
Use an array of size n × m.
3 The function has k integers parameters n1 , n2 , .., nk :
Use an array of size n1 × ... × nk .
4 The function has a set parameter:
Use the characteristic function or bitmap.
5 The function has a graph parameter:
think of isomorphism not known if it is in P

Algorithms and Complexity Speed up recursive algorithms

Master ICS 16 / 29
Space complexity of recursive algorithms

The space complexity depends on the depth of the recursion tree; think of
the stack overflow!!!

Algorithms and Complexity Speed up recursive algorithms

Master ICS 17 / 29
Recursive algorithms vs Dynamic programming

"Solving a problem recursively ......is decomposition into subproblems"

1 Find a decomposition into simpler subproblems
2 Write it as a recurrence equation including base cases.
3 Speed up algorithms using memoisation.

For which problems the memoisation technique can improve the time
complexity?

Algorithms and Complexity Speed up recursive algorithms

Master ICS 18 / 29
Dynamic programming : Writing as sum
Given a positive integer n. Find the number of different ways to write n as
the sum of 1, 3, 5

Let Sn be the number of ways to write n as the sum of 1,3,5.

For example: 6 can be written 3 + 1 + 1 + 1 or 1 + 1 + 1 + 3.

Let n = x1 + x2 + ... + xm . There are three cases : xm = 1, xm = 3 or

xm = 5.

Recurrence equation:
Base cases : S1 = 1 and Sk = 0 for all negative integers.
Sn = Sn−1 + Sn−3 + Sn−5

Write the recurrence when 3 + 1 + 1 + 1 is the same as 1 + 1 + 1 + 3

(without repetitions)

Algorithms and Complexity Speed up recursive algorithms

Master ICS 19 / 29
Dynamic programming : Writing as sum
Recurrence equation:
Base cases : S1 = 1 and Sk = 0 for all negative integers.
Sn = Sn−1 + Sn−3 + Sn−5
Complexity:

Algorithms and Complexity Speed up recursive algorithms

Master ICS 20 / 29
Dynamic programming : Writing as sum without repetitions
Let A(n) be the number of ways to write n as the sum of 1,3,5,
B(n) be the number of ways to write n as the sum of 1,3,
C (n) be the number of ways to write n as the sum of 1.
Recurrence equation:
Base cases : C (m) = 1 for all positive integers, and
A(k) = B(k) = C (k) = 0 for all negative integers,
A(n) = A(n − 5) + B(n − 3) + C (n − 1)
B(n) = B(n − 3) + C (n − 1)
Complexity:

Algorithms and Complexity Speed up recursive algorithms

Master ICS 21 / 29
Dynamic programming : Quadratic

Knapsack Problem: A set of items S = {1, ..., n}, where item i has weight
wi and Find a subset S 0 ⊆ S such
P utility vi , and a knapsack capacity B. P
that i∈S 0 wi ≤ B and maximizes the utility i∈S 0 .

Let S(i, b) be the value of the optimal solution to the knapsack instance
defined by on i items and a capacity b.

Recurrence equation:
Base cases :
S(0, b) = 0 for 0 ≤ b ≤ B, no item
S(i, b) = −∞ for b < 0, illegal
S(i, b) = max(vi + S(i − 1, b − wi ), S(i − 1, b)) take i or not
where 1 ≤ i ≤ n, 0 ≤ b ≤ w .

Algorithms and Complexity Speed up recursive algorithms

Master ICS 22 / 29
Dynamic programming : Quadratic
Recurrence equation:
Base cases :
S(0, b) = 0 for 0 ≤ b ≤ B, no item
S(i, b) = −∞ for b < 0, illegal
S(i, b) = max(vi + S(i − 1, b − wi ), S(i − 1, b)) take i or not
where 1 ≤ i ≤ n, 0 ≤ b ≤ w .
Complexity:

Algorithms and Complexity Speed up recursive algorithms

Master ICS 23 / 29
Dynamic programming : subset

Problem: Given a weighted graph G = (V , E ) with n nodes, find the

shortest path that visits every node exactly once (Traveling Salesman
Problem).

Brute force algorithm takes O(n!) time.

Define subproblems:
D(S, v ): the length of the optimal path that visits every node in the
set S exactly once and ends at v .
There are approximately n2n subproblems
Answer is minv ∈V D(V , v ), where V is the given set of nodes

Algorithms and Complexity Speed up recursive algorithms

Master ICS 24 / 29
Dynamic programming : subset

Base cases: For each node v ∈ V , D({v }, v ) = 0.

Recurrence equation
Consider a path that visits all nodes in S exactly once and ends at v .
Right before arriving v , the path comes from some u in S − {v }.
And that subpath has to be the optimal one that covers S − {v },
ending at u.
We just try all possible candidates for u

D(S, v ) = min(u,v )∈E D(S − {v }, u) + cost(u, v )

Algorithms and Complexity Speed up recursive algorithms

Master ICS 25 / 29
Dynamic programming : subset
Recurrence equation:
Base cases : For each node v ∈ V , D({v }, v ) = 0.
for u, D(S, v ) = min(u,v )∈E D(S − {v }, u) + cost(u, v )
Complexity:

Algorithms and Complexity Speed up recursive algorithms

Master ICS 26 / 29
Dynamic programming

Which NP-complete problems have an efficient DP?

The time complexity of an algorithm is pseudo-polynomial if its running

time is polynomial in the numerical values of the input, but not necessary
the size of the numerical values in binary.

A decision problem is pseudo-polynomial or weakly NP-complete if it can

be solved by a pseudo-poly time algorithm.

Example: Primality testing, Subset sum, KNAPSACK.

Algorithms and Complexity Speed up recursive algorithms

Master ICS 27 / 29
Dynamic programming

Problem
Questions to Answer (QA)
Instance : A set of questions {1, 2, 3, ..., n}, a set {p1 , p2 , p3 , ..., pn }
where pi corresponds to pints to receive for answering the question i,
and a set {f1 , f2 , f3 , ..., fn } where fi corresponds to the number of
questions following i that your are unable to solve when you solve the
question i.
Question : Compute a subset Q ⊆ {1, 2, 3, ..., n} that maximizes
P
i∈Q pi , and for each i ∈ Q, Q∩]i, i + fi ] = ∅.

Algorithms and Complexity Speed up recursive algorithms

Master ICS 28 / 29