Algorithm Analysis

Why we study algorithm?

• Suppose computers are infinitely fast and computer memory
was free. Would you have any reason to study algorithm?
– Yes, because we want to demonstrate that the algorithm is
correct; it terminates with the intended solution for all
inputs given.
• However, the reality shows that :
– Computers may be fast, but they are not infinitely fast
– Memory may be cheap, but it is not free
– Computing time and resources are therefore a bounded
resources
Algorithm Design & Analysis Process
Understand the problem

Decide
Decideonon:algorithm
algorithm
design
design
techniques
techniquesetc.

Design an algorithm
Correctness

Prove correctness
Efficiency
Analyze efficiency

Code the algorithm

Analysis of Algorithm
• As we may have noted, there are often multiple algorithms one can
use to solve the same problem.
–In searching from a sequence of list, one can use linear search,
binary search, …..
–For sorting a sequence of list, we may consider bubble sort,
selection sort, quick sort, merge sort, heap sort,…
–We can come up with our own variants.
• Why all these algorithms? Do we need them?
• How do we choose which algorithm is the best?
–The fastest/most efficient algorithm.
–The one that uses the fewest resources.
–The clearest.
–The shortest, ...
Cont..
• Analysis of algorithm is the analysis of resource usage of a given
algorithm
–It means predicting the resources that the algorithm requires
• The main resources are running time and memory usage
–An algorithm that solves a problem but requires a year and GBs of
main memory is hardly of any use.
• The objective of algorithm analysis is
–to measure resources (e.g., time, space) requirements of an
algorithm so as to determine :-
•how quickly (with less memory) an algorithm executes in
practice
• An algorithm should make efficient use of computer resources
–Most frequently, we look at efficiency, i.e.
•how long does the algorithm take to run
•What is the best way to represent the running time of an
algorithm?
Efficiency
• An algorithm must solve a problem with the least amount of
computational resources such as time and space.
– an algorithm should run as fast as possible using as little
memory as possible
• Two types of algorithmic efficiency evaluation
– Time efficiency - indicates how fast the algorithm runs
– Space efficiency - indicates how much memory the algorithm
needs
• What to analyze?
– To keep things simple, we will concentrate on the running time
of algorithms and will not look at the space (the amount of
memory) needed or required.
– So, efficiency considerations of algorithm usually focus on the
amount of time elapsed (called running time of an algorithm)
when processing data.
Analysis of Insertion Sort
algorithm INSERTION-SORT(A) cost times
1. for j  2 to length[A] do c1 n
2. key  A[j] c2 n -1
3. i  j-1 c3 n -1
n
4. while i >0 and A[i]>key do c4 t
j =2
j

n
5. A[i+1]  A[i] c5 t
j =2
j −1

6. i  i-1 c6 n

t
j =2
j −1
7. A[i+1]  key c7
n -1
(tj is the number of times the while loop in line 4 is executed for that value of j)
• The running time, T(n), of the insertion algorithm is the sum of
running times for each statement executed, i.e.:
=c1n+ c2(n-1)+ c3(n-1)+ c4nj=2 tj+ c5nj=2 (tj-1)+ c6nj=2(tj-1)+ c7(n-1)
Best Case Analysis of Insertion Sort
• Occurs if the array contains an already sorted values
–For each j = 2, 3, 4,…, n, we then find that A[i] ≤ key in line 4
when i has its initial value of j – 1.
–Thus tj=1 for j = 2, 3,…, n, and line 5 and 6 will be executed 0
times
• The best case running time is
T(n) = c1n + c2(n-1) + c3(n-1) + c4(n-1) + c7(n-1)
= (c1 + c2 + c3 + c4 + c7)n – (c2 + c3 + c4 + c7)
–This running time can be expressed as an + b for constants a and
b that depends on the statement cost ci;
• it is thus a linear function of n
Worst Case Analysis of Insertion Sort
• Occurs if the array contains values that are in reverse sorted
order, that is in decreasing order
• We must compare each element A[j] with each element in
the entire sorted subarray A[1..j-1]. So, tj = j for j = 2,3,…,n.
n n
n(n + 1) n n
n(n − 1)
t = 
j =2
j
j =2
j=
2
− 1   (t j − 1) =  ( j − 1) =
j =2 j =2 2
Therefore the worst case running time of INSERTION-SORT is T(n)
n(n + 1) n(n − 1) n(n − 1)
= c1n + c2 (n − 1) + c3 (n − 1) + c4 ( − 1) + c5 ( ) + c6 ( ) + c7 (n − 1)
2 2 2
c4 + c5 + c6 2 c4 c5 c6
=( )n + (c1 + c2 + c3 + + + + c7 )n + (c2 + c3 + c4 + c7 )
2 2 2 2
–This worst case running time can be expressed as an2 + bn + c for
constants a, b, c, it is thus a quadratic function on n
 Average Case Analysis of Insertion Sort
• Suppose that we randomly choose n numbers and apply
insertion sort
• How long does it take to determine where in subarray
A[1..j-1] to insert the element A[j]?
–On average, half the elements in A[1..j-1] are less than A[j], and
half the elements are greater
–On average, therefore, we check half the subarray A[1..j-1], so tj =
j/2 and T(n) will still be in the order of n2,
• This average case running time can then be expressed as
quadratic function, an2 + bn + c for constants a, b, c,
which is the same as worst case
• In summary, the running time of insertion sort for
–Best case: an – b
–Worst case: an2 + bn - c
–Average case: an2 + bn - c
Asymptotic Analysis
• When analyzing the running time or space usage of programs, we usually
try to estimate the time or space as a function of the input size.
– For example, when analyzing the worst case running time of an
insertion algorithm, we say the running time of insertion sort is, T(n) =
an2 + bn - c, for some constants a, b & c.
• The asymptotic behavior of a function f(n) (such as f(n)=c*n or f(n)=c*n 2,
etc.) refers to the growth of f(n) as n gets large. We typically ignore small
values of n, since we are usually interested in estimating how slow the
program will be on large inputs.
– A good rule of thumb is: the slower the asymptotic growth rate, the
better the algorithm.
• By this measure, a linear algorithm; f(n)=d*n+k, is always asymptotically
better than a quadratic one; f(n)=c*n2+q. That is because for any given
(positive) c, k, d & q there is always some n at which the magnitude of
c*n2+q overtakes d*n+k.
– For moderate values of n, the quadratic algorithm could very well take
less time than the linear one. However, the linear algorithm will
always be better for sufficiently large inputs.
Asymptotic analysis
• There are five notations used to describe a running time function:
Big-Oh, Big-Omega, Theta, Little-o, little-omega
• Demonstrating that a function T(n) is in big-O (or others) of a
function f(n) requires that we find specific constants C and no for
which the inequality holds
• The following points are facts that can be used for efficiency
comparison

1 ≤ n for all n ≥ 1 2n ≤ n! for all n ≥ 4

n ≤ n2 for all n ≥ 1 log2n ≤ n for all n ≥ 2

n ≤ nlog2n for all n ≥ 2

Asymptotic notations: Big-Oh (O)
• Definition
– Let f(n) and g(n) be functions mapping nonnegative integers to
real numbers.
– A function f(n) = O(g(n)), if there is some positive constant c > 0
and a non-negative integer no ≥ 1 such that
f(n) ≤ c.g(n) for all n ≥ no
• Big-O expresses an upper bound on the growth rate of a function,
for sufficiently large values of n
– An upper bound is the best algorithmic solution that has been
found for a problem (“what is the best thing that we know we
can do?”)
• In simple words, f(n) = O(g(n)) means that the growth rate of f(n)
is less than or equal to g(n).
– The statement f(n) = O(g(n)) states only that c.g(n) is un upper
bound on the value of f(n) for all n, n ≥ n0
Big-Oh theorems
• Theorem 1: If k is a constant, then k is O(1)
– Example: f(n) = 2100 = O(1)
• Theorem 2: If f(n) is a polynomial of degree k, then
f(n) = O(nk)
– If f(n) = a0+ a1n + a2n2 + … + aknk, where ai and k are constants,
then f(n) is in O(nk)
– Polynomial’s growth rate is determined by the leading term
Example: f(n) = 7n4 + 3n2 + 5n + 1000 is O(n4)
• Theorem 3: Constant factors may be ignored
If g(n) is in O(f(n)), then k * g(n) is O(f(n)), k >0
Example:
• T(n) = 7n4 +3n2 + 5n +1000 is O(n4)
• T(n) = 28n4 + 12n2 + 20n + 4000 is O(n4)
Big-Oh theorems
• Theorem 4 (Transitivity)
– If T(n) is O(f(n))and f(n) is O(g(n)), then T(n) is O(g(n)).
• Theorem 5
If T(n) is in O(f(n)) and g(n) is in O(h(n)), then T(n) + g(n) is in
O(T(n) + g(n))
• Theorem 6
If T(n) is in O(f(n)) and g(n) is in O(h(n)), then T(n) . g(n) is in
O(T(n) . g(n))
product of upper bounds is upper bound for the product
• Theorem 7
If f1(n) = O(g1(n)) and f2(n) = O(g2(n)), then f1(n) + f2(n) =
O(max(g1(n), g2(n))
 Order of growth of functions
• Typical orders: Here is a table of some typical cases. It
shows that the typical orders is:
O(1) < O(log n) < O(n) < O(nlog n) < O(n2) < O(n3) < O(2n)
Example: Big-Oh (O)
Find O(f(n) for the given functions:

• f(n) = 2n + 6

• f(n) = 13n3 + 42n2 + 2n log n

• If f(n) = 3n2 + 4n + 1 then show that f(n) = O(n2)

• If f(n) = 10n + 5 and g(n) = n, then show that f(n) is

O(g(n))
Asymptotic notations: Big-Omega ()
• Definition
–Let f(n) and g(n) be functions mapping nonnegative integers to
real numbers.
–A function f(n) = (g(n)), if there is some positive constant c >
0 and a negative integer no ≥ 1 such that
f(n) ≥ c.g(n) for all n ≥ no
• The statement f(n) = (g(n)) states only that c.g(n) is a lower
bound on the value of f(n) for all n, n ≥ n0
–In simple terms, f(n) = (g(n)) means that the growth rate of
f(n) is greater than or equal to g(n)
Big-Omega- Example
• Show that the function T(n) = 5n2 – 64n + 256 = Ω(n2)
– We need to show that for non-negative integer n0 and a constant
c > 0, T(n) ≥ c.n2 for all integers n ≥ n0
– we have that for c=1 and n0 = 0, T(n) ≥ cn2 for all integers n ≥ n0
–What if c = 2 and n0 = 16 ?

• Show if f(n) = 10n2 + 4n + 2 and g(n) = n2 , then f(n) = Ω(n2)

• Show that 3n2 + 5 ≠ Ω(n3)

Asymptotic notations: Theta ()
• Definition
–Let f(n) and g(n) be functions mapping nonnegative integers to real
numbers.
–A function f(n) = (g(n)), if there exist some positive constant c1
and c2 and a negative integer constant no ≥ 1 such that c1.g(n) ≤
f(n) ≤ c2.g(n) for all n ≥ no
• The Theta notation is used when the function f can be bounded both
from above and below by the same function
–When we write f(n) = (g(n)), we mean that f lies between c1
times the function g and c2 times the function g except possibly
when n is smaller than n0
• Another way to view the θ-notation is that the function
–f(n) = θ(n) if and only if
f(n) = Ο(g(n)) and f(n) = Ω(g(n))
Asymptotic Tightness
• The theta notation is more precise than both the Big-O and Big-
notations.
–The function f(n) = (g(n)) iff g(n) is both an upper and lower
bounds on f(n).
• Big-Oh does not have to be asymptotically tight
–f(n) = ½n is O(n) with c=1, n0=1, but is also in O(n100)…
• Big- isn’t tight either
–f(n) = n5 is (n) with c=1, n0 = 1…
• Theta () is tight…
–f(n) must be in same growth classes to meet definition.
–Can you prove this assertion? Prove that f(n)=3n3+2n2+1 is
(n3)
• Show that f(n) is O(n3), and also, f(n) is (n3)
Exercise: Algorithmic efficiency
• What is the best, worst and average case time complexity of
Insertion sort algorithms?
• Write (i) an algorithm for linear search and (ii) analyze its time
complexity, which scans through the given sequence of inputs, A =
<a1, a2, …, an>, looking for Key.
The output is either
– One or more index i (the position of all values if key = A[i]) or
– the special message “Not Found” if Key does not appear in the
list.
• Write (i) an algorithm and (ii) a code that scans through the given
sequence of inputs, A = <a1, a2, …, an>, finding for one or more
minimum value(s).