Computational Tractability Asymptotic Order of Growth Common Running Times

Analysis of Algorithms
T. M. Murali

January 16, 2008

T. M. Murali January 16, 2008 Analysis of Algorithms

Computational Tractability Asymptotic Order of Growth Common Running Times

What is algorithm analysis?

I Measure resource requirements: how does the amount of time

and space an algorithm uses scale with increasing input size?
I How do we put this notion on a concrete footing?
I What does it mean for one function to grow faster or slower
than another?

T. M. Murali January 16, 2008 Analysis of Algorithms

Computational Tractability Asymptotic Order of Growth Common Running Times

What is algorithm analysis?

I Measure resource requirements: how does the amount of time

and space an algorithm uses scale with increasing input size?
I How do we put this notion on a concrete footing?
I What does it mean for one function to grow faster or slower
than another?
I Goal: Develop algorithms that provably run quickly and use low
amounts of space.

T. M. Murali January 16, 2008 Analysis of Algorithms

Computational Tractability Asymptotic Order of Growth Common Running Times

Worst-case running time

I We will measure worst-case running time of an algorithm.

I Bound the largest possible running time the algorithm over all
inputs of size n, as a function of n.

T. M. Murali January 16, 2008 Analysis of Algorithms

Computational Tractability Asymptotic Order of Growth Common Running Times

Worst-case running time

I We will measure worst-case running time of an algorithm.

I Bound the largest possible running time the algorithm over all
inputs of size n, as a function of n.
I Why worst-case? Why not average-case or on random inputs?

T. M. Murali January 16, 2008 Analysis of Algorithms

Computational Tractability Asymptotic Order of Growth Common Running Times

Worst-case running time

I We will measure worst-case running time of an algorithm.

I Bound the largest possible running time the algorithm over all
inputs of size n, as a function of n.
I Why worst-case? Why not average-case or on random inputs?
I Input size = number of elements in the input.

T. M. Murali January 16, 2008 Analysis of Algorithms

Computational Tractability Asymptotic Order of Growth Common Running Times

Worst-case running time

I We will measure worst-case running time of an algorithm.

I Bound the largest possible running time the algorithm over all
inputs of size n, as a function of n.
I Why worst-case? Why not average-case or on random inputs?
I Input size = number of elements in the input. Values in the
input do not matter.
I Assume all elementary operations take unit time: assignment,
arithmetic on a xed-size number, comparisons, array lookup,
following a pointer, etc.

T. M. Murali January 16, 2008 Analysis of Algorithms

Computational Tractability Asymptotic Order of Growth Common Running Times

Polynomial time
I Brute force algorithm: Check every possible solution.

T. M. Murali January 16, 2008 Analysis of Algorithms

Computational Tractability Asymptotic Order of Growth Common Running Times

Polynomial time
I Brute force algorithm: Check every possible solution.
I What is a brute force algorithm for sorting: given n numbers,
permute them so that they appear in increasing order?

T. M. Murali January 16, 2008 Analysis of Algorithms

Computational Tractability Asymptotic Order of Growth Common Running Times

Polynomial time
I Brute force algorithm: Check every possible solution.
I What is a brute force algorithm for sorting: given n numbers,
permute them so that they appear in increasing order?
I Try all possible n! permutations of the numbers.
I For each permutation, check if it is sorted.

T. M. Murali January 16, 2008 Analysis of Algorithms

Computational Tractability Asymptotic Order of Growth Common Running Times

Polynomial time
I Brute force algorithm: Check every possible solution.
I What is a brute force algorithm for sorting: given n numbers,
permute them so that they appear in increasing order?
I Try all possible n! permutations of the numbers.
I For each permutation, check if it is sorted.

I Running time is nn!. Unacceptable in practice!

T. M. Murali January 16, 2008 Analysis of Algorithms

Computational Tractability Asymptotic Order of Growth Common Running Times

Polynomial time
I Brute force algorithm: Check every possible solution.
I What is a brute force algorithm for sorting: given n numbers,
permute them so that they appear in increasing order?
I Try all possible n! permutations of the numbers.
I For each permutation, check if it is sorted.

I Running time is nn!. Unacceptable in practice!

I Desirable scaling property: when the input size doubles, the

algorithm should only slow down by some constant factor c .

T. M. Murali January 16, 2008 Analysis of Algorithms

Computational Tractability Asymptotic Order of Growth Common Running Times

Polynomial time
I Brute force algorithm: Check every possible solution.
I What is a brute force algorithm for sorting: given n numbers,
permute them so that they appear in increasing order?
I Try all possible n! permutations of the numbers.
I For each permutation, check if it is sorted.

I Running time is nn!. Unacceptable in practice!

I Desirable scaling property: when the input size doubles, the

algorithm should only slow down by some constant factor c .
I An algorithm has a polynomial running time if there exist
constants c > 0 and d > 0 such that on every input of size n,
the running time of the algorithm is bounded by cnd steps.

T. M. Murali January 16, 2008 Analysis of Algorithms

Computational Tractability Asymptotic Order of Growth Common Running Times

Polynomial time
I Brute force algorithm: Check every possible solution.
I What is a brute force algorithm for sorting: given n numbers,
permute them so that they appear in increasing order?
I Try all possible n! permutations of the numbers.
I For each permutation, check if it is sorted.

I Running time is nn!. Unacceptable in practice!

I Desirable scaling property: when the input size doubles, the

algorithm should only slow down by some constant factor c .
I An algorithm has a polynomial running time if there exist
constants c > 0 and d > 0 such that on every input of size n,
the running time of the algorithm is bounded by cnd steps.

Denition
An algorithm is ecient if it has a polynomial running time.
T. M. Murali January 16, 2008 Analysis of Algorithms
Computational Tractability Asymptotic Order of Growth Common Running Times

Upper and lower bounds

Denition
Asymptotic upper bound: A function f (n) is O (g (n)) if there exist
constants c > 0 and n0 ≥ 0 such that for all n ≥ n0 , we have
f (n) ≤ cg (n).
Denition
Asymptotic lower bound: A function f (n) is Ω(g (n)) if there exist
constants c > 0 and n0 ≥ 0 such that for all n ≥ n0 , we have
f (n) ≥ cg (n).
Denition
Asymptotic tight bound: A function f (n) is Θ(g (n)) if f (n) is
O (g (n)) and f (n) is Ω(g (n)).

T. M. Murali January 16, 2008 Analysis of Algorithms

Computational Tractability Asymptotic Order of Growth Common Running Times

Upper and lower bounds

Denition
Asymptotic upper bound: A function f (n) is O (g (n)) if there exist
constants c > 0 and n0 ≥ 0 such that for all n ≥ n0 , we have
f (n) ≤ cg (n).
Denition
Asymptotic lower bound: A function f (n) is Ω(g (n)) if there exist
constants c > 0 and n0 ≥ 0 such that for all n ≥ n0 , we have
f (n) ≥ cg (n).
Denition
Asymptotic tight bound: A function f (n) is Θ(g (n)) if f (n) is
O (g (n)) and f (n) is Ω(g (n)).
I In these denitions, c is a constant independent of n .
I Abuse of notation: say g (n ) = O (f (n )), g (n ) = Ω(f (n )),
g (n) = Θ(f (n)).
T. M. Murali January 16, 2008 Analysis of Algorithms
Computational Tractability Asymptotic Order of Growth Common Running Times

Properties of Asymptotic Growth Rates

Transitivity
I If f = O (g ) and g = O (h), then f = O (h).
I If f = Ω(g ) and g = Ω(h), then f = Ω(h).
I If f = Θ(g ) and g = Θ(h), then f = Θ(h).

T. M. Murali January 16, 2008 Analysis of Algorithms

Computational Tractability Asymptotic Order of Growth Common Running Times

Properties of Asymptotic Growth Rates

Transitivity
I If f = O (g ) and g = O (h), then f = O (h).
I If f = Ω(g ) and g = Ω(h), then f = Ω(h).
I If f = Θ(g ) and g = Θ(h), then f = Θ(h).
Additivity
I If f = O (h) and g = O (h), then f + g = O (h).
I Similar statements hold for lower and tight bounds.

T. M. Murali January 16, 2008 Analysis of Algorithms

Computational Tractability Asymptotic Order of Growth Common Running Times

Properties of Asymptotic Growth Rates

Transitivity
I If f = O (g ) and g = O (h), then f = O (h).
I If f = Ω(g ) and g = Ω(h), then f = Ω(h).
I If f = Θ(g ) and g = Θ(h), then f = Θ(h).
Additivity
I If f = O (h) and g = O (h), then f + g = O (h).
I Similar statements hold for lower and tight bounds.
I If k is a constant and there are k functions
fi = O (h), 1 ≤ i ≤ k ,

T. M. Murali January 16, 2008 Analysis of Algorithms

Computational Tractability Asymptotic Order of Growth Common Running Times

Properties of Asymptotic Growth Rates

Transitivity
I If f = O (g ) and g = O (h), then f = O (h).
I If f = Ω(g ) and g = Ω(h), then f = Ω(h).
I If f = Θ(g ) and g = Θ(h), then f = Θ(h).
Additivity
I If f = O (h) and g = O (h), then f + g = O (h).
I Similar statements hold for lower and tight bounds.
I If k is a constant and there are k functions
fi = O (h), 1 ≤ i ≤ k , then f1 + f2 + . . . + fk = O (h).

T. M. Murali January 16, 2008 Analysis of Algorithms

Computational Tractability Asymptotic Order of Growth Common Running Times

Properties of Asymptotic Growth Rates

Transitivity
I If f = O (g ) and g = O (h), then f = O (h).
I If f = Ω(g ) and g = Ω(h), then f = Ω(h).
I If f = Θ(g ) and g = Θ(h), then f = Θ(h).
Additivity
I If f = O (h) and g = O (h), then f + g = O (h).
I Similar statements hold for lower and tight bounds.
I If k is a constant and there are k functions
fi = O (h), 1 ≤ i ≤ k , then f1 + f2 + . . . + fk = O (h).
I If f = O (g ), then f + g =

T. M. Murali January 16, 2008 Analysis of Algorithms

Computational Tractability Asymptotic Order of Growth Common Running Times

Properties of Asymptotic Growth Rates

Transitivity
I If f = O (g ) and g = O (h), then f = O (h).
I If f = Ω(g ) and g = Ω(h), then f = Ω(h).
I If f = Θ(g ) and g = Θ(h), then f = Θ(h).
Additivity
I If f = O (h) and g = O (h), then f + g = O (h).
I Similar statements hold for lower and tight bounds.
I If k is a constant and there are k functions
fi = O (h), 1 ≤ i ≤ k , then f1 + f2 + . . . + fk = O (h).
I If f = O (g ), then f + g = Θ(g ).

T. M. Murali January 16, 2008 Analysis of Algorithms

Computational Tractability Asymptotic Order of Growth Common Running Times

Examples

I f (n) = pn2 + qn + r is

T. M. Murali January 16, 2008 Analysis of Algorithms

Computational Tractability Asymptotic Order of Growth Common Running Times

Examples

I f (n) = pn2 + qn + r is θ(n2). Can ignore lower order terms.

T. M. Murali January 16, 2008 Analysis of Algorithms

Computational Tractability Asymptotic Order of Growth Common Running Times

Examples

I f (n) = pn2 + qn + r is θ(n2). Can ignore lower order terms.

I Is f (n) = pn2 + qn + r = O (n3 )?

T. M. Murali January 16, 2008 Analysis of Algorithms

Computational Tractability Asymptotic Order of Growth Common Running Times

Examples

I f (n) = pn2 + qn + r is θ(n2). Can ignore lower order terms.

I Is f (n) = pn2 + qn + r = O (n3 )?
I f (n) = P0≤i ≤d ai ni =

T. M. Murali January 16, 2008 Analysis of Algorithms

Computational Tractability Asymptotic Order of Growth Common Running Times

Examples

I f (n) = pn2 + qn + r is θ(n2). Can ignore lower order terms.

I Is f (n) = pn2 + qn + r = O (n3 )?
I f (n) = P0≤i ≤d ai ni = O (nd ), if d > 0 is an integer constant
and ad > 0. Denition of polynomial time

T. M. Murali January 16, 2008 Analysis of Algorithms

Computational Tractability Asymptotic Order of Growth Common Running Times

Examples

I f (n) = pn2 + qn + r is θ(n2). Can ignore lower order terms.

I Is f (n) = pn2 + qn + r = O (n3 )?
I f (n) = P0≤i ≤d ai ni = O (nd ), if d > 0 is an integer constant
and ad > 0. Denition of polynomial time
I Is an algorithm with running time O (n1.59 ) a polynomial-time
algorithm?

T. M. Murali January 16, 2008 Analysis of Algorithms

Computational Tractability Asymptotic Order of Growth Common Running Times

Examples

I f (n) = pn2 + qn + r is θ(n2). Can ignore lower order terms.

I Is f (n) = pn2 + qn + r = O (n3 )?
I f (n) = P0≤i ≤d ai ni = O (nd ), if d > 0 is an integer constant
and ad > 0. Denition of polynomial time
I Is an algorithm with running time O (n1.59 ) a polynomial-time
algorithm?
I O (loga n) = O (logb n) for any pair of constants a, b > 1.
I For every x > 0, log n = O (nx ).

T. M. Murali January 16, 2008 Analysis of Algorithms

Computational Tractability Asymptotic Order of Growth Common Running Times

Examples

I f (n) = pn2 + qn + r is θ(n2). Can ignore lower order terms.

I Is f (n) = pn2 + qn + r = O (n3 )?
I f (n) = P0≤i ≤d ai ni = O (nd ), if d > 0 is an integer constant
and ad > 0. Denition of polynomial time
I Is an algorithm with running time O (n1.59 ) a polynomial-time
algorithm?
I O (loga n) = O (logb n) for any pair of constants a, b > 1.
I For every x > 0, log n = O (nx ).
I For every r > 1 and every d > 0, nd = O (r n ).

T. M. Murali January 16, 2008 Analysis of Algorithms

Computational Tractability Asymptotic Order of Growth Common Running Times

Linear time

I Running time is at most a constant factor times the size of the

input.

T. M. Murali January 16, 2008 Analysis of Algorithms

Computational Tractability Asymptotic Order of Growth Common Running Times

Linear time

I Running time is at most a constant factor times the size of the

input.
I Finding the minimum, merging two sorted lists.

T. M. Murali January 16, 2008 Analysis of Algorithms

Computational Tractability Asymptotic Order of Growth Common Running Times

Linear time

I Running time is at most a constant factor times the size of the

input.
I Finding the minimum, merging two sorted lists.
I Sub-linear time.

T. M. Murali January 16, 2008 Analysis of Algorithms

Computational Tractability Asymptotic Order of Growth Common Running Times

Linear time

I Running time is at most a constant factor times the size of the

input.
I Finding the minimum, merging two sorted lists.
I Sub-linear time. Binary search in a sorted array of n numbers
takes O (log n) time.

T. M. Murali January 16, 2008 Analysis of Algorithms

Computational Tractability Asymptotic Order of Growth Common Running Times

O (n log n) time

I Any algorithm where the costliest step is sorting.

T. M. Murali January 16, 2008 Analysis of Algorithms

Computational Tractability Asymptotic Order of Growth Common Running Times

Quadratic time

I Enumerate all pairs of elements.

T. M. Murali January 16, 2008 Analysis of Algorithms

Computational Tractability Asymptotic Order of Growth Common Running Times

Quadratic time

I Enumerate all pairs of elements.

I Given a set of n points in the plane, nd the pair that are the
closest.

T. M. Murali January 16, 2008 Analysis of Algorithms

Computational Tractability Asymptotic Order of Growth Common Running Times

Quadratic time

I Enumerate all pairs of elements.

I Given a set of n points in the plane, nd the pair that are the
closest. Surprising fact: can solve this problem in O (n log n)
time later in the semester.

T. M. Murali January 16, 2008 Analysis of Algorithms

Computational Tractability Asymptotic Order of Growth Common Running Times

O (nk ) time

I Does a graph have an independent set of size k , where k is a

constant, i.e. there are k nodes such that no two are joined by
an edge?

T. M. Murali January 16, 2008 Analysis of Algorithms

Computational Tractability Asymptotic Order of Growth Common Running Times

O (nk ) time

I Does a graph have an independent set of size k , where k is a

constant, i.e. there are k nodes such that no two are joined by
an edge?
I Algorithm: For each subset S of k nodes, check if S is an
independent set. If the answer is yes, report it.

T. M. Murali January 16, 2008 Analysis of Algorithms

Computational Tractability Asymptotic Order of Growth Common Running Times

O (nk ) time

I Does a graph have an independent set of size k , where k is a

constant, i.e. there are k nodes such that no two are joined by
an edge?
I Algorithm: For each subset S of k nodes, check if S is an
independent set. If the answer is yes, report it.
I Running time is

T. M. Murali January 16, 2008 Analysis of Algorithms

Computational Tractability Asymptotic Order of Growth Common Running Times

O (nk ) time

I Does a graph have an independent set of size k , where k is a

constant, i.e. there are k nodes such that no two are joined by
an edge?
I Algorithm: For each subset S of k nodes, check if S is an
independent set. If the answer is yes, report it.
Running time is O (k 2 kn ) = O (nk ).


I

T. M. Murali January 16, 2008 Analysis of Algorithms

Computational Tractability Asymptotic Order of Growth Common Running Times

Beyond polynomial time

I What is the largest size of an independent set in a graph with n

nodes?

T. M. Murali January 16, 2008 Analysis of Algorithms

Computational Tractability Asymptotic Order of Growth Common Running Times

Beyond polynomial time

I What is the largest size of an independent set in a graph with n

nodes?
I Algorithm: For each 1 ≤ i ≤ n, check if the graph has an
independent size of size i . Output largest independent set found.

T. M. Murali January 16, 2008 Analysis of Algorithms

Computational Tractability Asymptotic Order of Growth Common Running Times

Beyond polynomial time

I What is the largest size of an independent set in a graph with n

nodes?
I Algorithm: For each 1 ≤ i ≤ n, check if the graph has an
independent size of size i . Output largest independent set found.
I What is the running time?

T. M. Murali January 16, 2008 Analysis of Algorithms

Computational Tractability Asymptotic Order of Growth Common Running Times

Beyond polynomial time

I What is the largest size of an independent set in a graph with n

nodes?
I Algorithm: For each 1 ≤ i ≤ n, check if the graph has an
independent size of size i . Output largest independent set found.
I What is the running time? O (n2 2n ).

T. M. Murali January 16, 2008 Analysis of Algorithms