Algorithms:

analysis, complexity
Click to add Text

Based on the lecture note Algorithms: analysis, complexity, MIT

Plan

 Introduction
 Analysis of algorithms
 Recursive algorithm analysis

2
Algorithms

 Finite set of instructions that solves a given

problem
(a)
 Characteristics:
 Input. Zero or more quantities are supplied
 Output. At least one quantity is computed
 Definiteness. Each instruction is computable
 Finiteness. The algorithm terminates with the
answer or by telling us no answer exists

3
Algorithms: forms of analysis
 How to devise an algorithm
 How to validate the algorithm is correct -
Correctness proofs
 How to analyze running time and space of algorithm
Complexity analysis: asymptotic, empirical, others
 How to choose or modify an algorithm to solve a
problem
 How to implement and test an algorithm in a
program
Keep program code short and correspond closely
to algorithm steps
4
Plan

 Introduction
 Analysis of algorithms
 Recursive algorithm analysis

5
Analysis of algorithms
 Time complexity of a given algorithm
 How does time depend on problem size?
 Does time depend on problem instance or details?
 Is this the fastest algorithm?
 How much does speed matter for this problem? •
 Space complexity
 How much memory is required for a given problem size?
 Assumptions on computer word size, processor
 Fixed word/register size
 Single or multi (grid, hypercube) processor
 Solution quality
 Exact or approximate/bounded
 Guaranteed optimal or heuristic
6
Methods of complexity analysis

 Asymptotic analysis
 Create recurrence relation and solve

 This relates problem size of original problem to number and size of sub-
problems solved
 Different performance measures are of interest
 Worst case (often easiest to analyze; need one ‘bad’ example)
 Best case (often easy for same reason)
 Data-specific case (usually difficult, but most useful)
 Write implementation of algorithm (on paper)
 Create table (on paper) of frequency and cost of steps

 Sum up the steps; relate them to problem size

 Implement algorithm
 Count steps executed with counter variables, or use timer

 Vary problem size and analyze the performance

7
Asymptotic notation: upper bound O(..)
 f(n)= O(g(n)) if and only if
 f(n) ≤ c * g(n)
 where c > 0
 for all n > n0
 Example
 f(n)= 6n + 4√n
 g(n)= n
 c= 10 (not unique)
 f(n)= c * g(n) when n= 1
 f(n) < g(n) when n > 1
 Thus, f(n)= O(n)
 O(..) is worst case (upper bound) notation for an algorithm’s
complexity (running time)
8
Asymptotic notation: lower bound Ω(..)
 f(n)= Ω(g(n)) if and only if
 f(n) ≥ c * g(n)
 where c > 0
 for all n > n0
 Example
 f(n)= 6n + 4√n
 g(n)= n
 c= 6 (again, not unique)
 f(n)= c * g(n) when n=0
 f(n) > g(n) when n > 0
 Thus, f(n)= Ω (n)
 Ω(..) is best case (lower bound) notation for an algorithm’s
complexity (running time)
9
Asymptotic notation

 Worst case or upper bound: O(..)

 f(n)= O(g(n)) if f(n) ≤ c* g(n)
 Best case or lower bound: Ω(..)
 f(n)= Ω(g(n)) if f(n) ≥ c* g(n)

 Composite bound: Θ(..)

 f(n)= Θ(g(n)) if c1* g(n) ≤f(n) ≤ c2* g(n)

 Average or typical case notation is less

formal
 We generally say “average case is O(n)”
10
Example performance of some common
algorithms

11
Analysis of Algorithms

 Analysis of algorithms is used to describe

approaches to the study of the performance of
algorithms.
 the worst-case runtime complexity of the algorithm is the function
defined by the maximum number of steps taken on any instance
of size a.
 the best-case runtime complexity of the algorithm is the function
defined by the minimum number of steps taken on any instance
of size a.
 the average case runtime complexity of the algorithm is the
function defined by an average number of steps taken on any
instance of size a.
 the amortized runtime complexity of the algorithm is the function
defined by a sequence of operations applied to the input of size a
and averaged over time.

12
Analysis of Algorithms - Example

 Sequential searching in an array of size n

i=0;
Found=0;
while(i<=n-1 && !Found)
if (A[i] == x) Found=1;
else i=i+1;

 Its worst-case runtime complexity is O(n)

 Its best-case runtime complexity is Ω (1)
 Its average case runtime complexity is
O(n/2)=O(n)

13
Analysis of time complexity

 Time complexity estimates the time to run an algorithm. It's

calculated by counting elementary operations.
 Execution time depends only on the algorithm and its input.
 This can be achieved by choosing an elementary operation,
which the algorithm performs repeatedly
 The time complexity T(n) as the number of such operations the
algorithm performs given an array of length n.

 The worst case time complexity:

 Let T1(n), T2(n), … be the execution times for all possible inputs
of size n.
 The worst-case time complexity W(n) is then defined as
W(n) = max(T1(n), T2(n), …).

14
Examples
 Example 1
/*1*/ Sum=0;
/*2*/ for(i=1;i<=n;i++) {
/*3*/ scanf(“%d”,&x);
/*4*/ Sum=Sum+x;
}

 Example 2
/*1*/ Sum1=0;
/*2*/ k=1;
/*3*/ while (k<=n) {
/*4*/ for(j=1;j<=n;j++)
/*5*/ Sum1=Sum1+1;
/*6*/ k=k*2;
}
15
Plan

 Introduction
 Analysis of algorithms
 Recursive algorithm analysis

16
Recursive algorithm analysis

 Create and solve recurrence equation

 Recurrence equation
 C(n)
T(n)  
 F(T(k))  d(n)

C(n): time complexity of the trivial case

F(T(k)):a polynomial function of T(k)
d(n): time complexity of dividing, synthesizing sub
problems

17
Example: Counting number of bits

 Input: A positive decimal integer n.

 Output: The number of binary digits in n’s binary
representation.

18
Example: Counting number of bits

 Input: A positive decimal integer n.

 Output: The number of binary digits in n’s binary
representation.

NON-RECURSIVECOUNT(n)
count= 1
while n >1
count = count+ 1
n= n/2
return count

RECURSIVECOUNT(n)
if n= 1 return 1
else return RecursiveCount(n/2) + 1

19
Solving recurrence equation
Iterative method
 Expand (iterate) the recurrence and express it as a
summation of terms depending only on n and the
initial conditions.
 Example:
T(n)= 2 if n= 0
T(n)= 2 + T(n-1) if n> 0
To solve for T(n)
T(n) = 2 + T(n-1)
= 2 + 2 + T(n-2)
= 2*2 + T(n-2)
…
= n*2 + T(0)
= 2n + 2
Thus, T(n) = O(n)

20
Solving recurrence equation
Substitution method
 Idea: Make a guess for the form of the
solution and prove by induction
 No “recipe” for the guess
 Try iteratively O(n3), O(n2), and so on

 Can be used to prove both upper bounds

O() and lower bounds Ω()

 Example:
T(n)= 1 if n= 1
T(n)= 2T(n/2) + n if n>1

21
Solving recurrence equation
Substitution method - Example
Guess T(n)≤ cnlogn for some constant c
(that is, T(n) = O(nlogn))
Proof
Base case: we need to show that our guess holds for some base
case . Ok, since function constant for small constant n.
n
Assume holds for :
2
T( n2 ) ≤ c n2 log n2 (Question: Why not n-1?)
Prove that holds for n: T(n)≤ cnlogn
n
T(n) = 2T( 2 ) +n
≤ 2(c n log n ) +n= cnlog n + n
2 2 2
= cnlogn−cnlog2 +n = nlogn-cn+n
So ok if c≥1
22
Solving recurrence equation
Master theorem

Solving recurrences of this form

c, n  1

T(n)   n
 aT( )  (nd ), n  1
b
- n=bk, for positive integer k
- a≥1, b≥2, c>0, d≥0 are constants

 O(n d ), a  b d

T(n)   O(n d log b n), a  b d
 log b a
 O(n ), a  b d

23
Solving recurrence equation
Master theorem - Examples

Solving
equations, T(1)=1:
– T(n) = 4T(n/2) + n
– T(n) = 4T(n/2) + n2
– T(n) = 4T(n/2) + n3

24
Thanks for your attention!

25