Analysis of Algorithms

CS 302 - Data Structures

Section 2.6
 What is Algorithm Analysis?
• Algorithm: A clearly specified finite set of
instructions a computer follows to solve a
problem.
• Algorithm analysis:
– Predicting the resources that an algorithm requires.
i.e computational time and memory.
– a process of determining the amount of time,
resource, etc. required when executing an algorithm.
 Model of Computation (Assumptions)

•Analysis independent of the variations in

– Machine
– Operating system
– Programming languages
– Compiler etc.
•Low-level details will not be considered
•Our model will be an abstraction of a standard generic
single-processor machine, called a random access machine
or RAM.
 Model of Computation (Assumptions)
• A RAM is assumed to be an idealized machine
– Infinitely large random-access memory
– Instructions execute sequentially
• Every instruction is in fact a basic operation on two values in
the machines memory which takes unit time.
• These might be characters or integers.
• Example of basic operations include
– Assigning a value to a variable
– Arithmetic operation (+, - , × , /) on integers
– Performing any comparison e.g. a < b
– Boolean operations
– Accessing an element of an array.
 Model of Computation (Assumptions)
• In theoretical analysis, computational complexity
– Estimated in asymptotic sense, (in computer science in
the analysis of algorithms, considering the performance
of algorithms when applied to very large input
datasets).i.e. Estimating for large inputs
• Big O, Omega, Theta etc. notations are used to compute the
complexity
• Asymptotic notations are used because different
implementations of algorithm may differ in efficiency
• Efficiencies of two given algorithm are related
– By a constant multiplicative factor
– Called hidden constant.
Algorithms Analysis and Design
 Drawbacks in Model of Computation
First poor assumption
• We assumed that each basic operation takes constant
time, i.e. model allows
– Adding
– Multiplying
– Comparing etc.
two numbers of any length in constant time
• Addition of two numbers takes a unit time!
– Not good because numbers may be arbitrarily
• Addition and multiplication both take unit time!
– Again very bad assumption
 Model of Computation not so Bad
Finally what about Our Model?
• But with all these weaknesses, our model is not so bad
because we have to give the
– Comparison not the absolute analysis of any algorithm.
– We have to deal with large inputs not with the small size
• Model seems to work well describing computational
power of modern nonparallel machines

Can we do Exact Measure of Efficiency ?

• Exact, not asymptotic, measure of efficiency can be
sometimes computed but it usually requires certain
assumptions concerning implementation
Algorithms Analysis and Design
 A Simple Example
// Input: int A[N], array of N
integers
// Output: Sum of all numbers in
array A

int Sum(int A[], int N) {

int s=0;
for (int i=0; i< N; i++)
s = s + A[i];
return s;
}
How should we analyse this?

Applied Algo Fall-03 8/31

A Simple Example
Analysis of Sum
1.) Describe the size of the input in terms of one ore
more
parameters:
- Input to Sum is an array of N ints, so size is N.

2.) Then, count how many steps are used for an input
of that size:
- A step is an elementary operation such as
+, <, =, A[i]

Applied Algo Fall-03 9/31

A Simple Example

Analysis of Sum (2)

// Input: int A[N], array of N integers
// Output: Sum of all numbers in array A

int Sum(int A[], int N {

int s=0; 1

for (int i=0; i< N; i++)

2 3 4
s = s + A[i];
5
6 7 1,2,8: Once
return s; 3,4,5,6,7: Once per each iteration
}
8 of for loop, N iteration
Total: 5N + 3
The complexity function of the
algorithm is : f(N) = 5N +3

Applied Algo Fall-03 10/31

Analysis: A Simple Example

How 5N+3 Grows

Estimated running time for different values of N:

N = 10 => 53 steps
N = 100 => 503 steps
N = 1,000 => 5003 steps
N = 1,000,000 => 5,000,003 steps

As N grows, the number of steps grow in linear proportion to

N for this Sum function.

Applied Algo Fall-03 11/31

 Summary : Computational Model
• Analysis will be performed with respect to this
computational model for comparison of algorithms
• We will give asymptotic analysis not detailed
comparison i.e. for large inputs
• We will use generic uniprocessor random-access
machine (RAM) in analysis
– All memory equally expensive to access
– No concurrent operations
– All reasonable instructions take unit time, except, of
course, function calls
 Analysis of Algorithms
What is the goal?
• Analyze time requirements - predict how
running time increases as the size of the
problem increases:

time = f(size)
Why is it useful?
• To compare different algorithms.
 Defining “problem size”
• Typically, it is straightforward to identify the size of
a problem, e.g.:
– size of array

– size of stack, queue, list etc.

– vertices and edges in a graph

• But not always …

 Time Analysis
• Provides upper and lower bounds of running time.

Lower Bound  Running Time Upper Bound

• Different types of analysis:

- Worst case
- Best case
- Average case
 Worst Case

• Provides an upper bound on running time.

• An absolute guarantee that the algorithm would not
run longer, no matter what the inputs are.

Lower Bound  Running Time  Upper Bound

 Best Case

• Provides a lower bound on running time.

• Input is the one for which the algorithm runs the
fastest.

Lower Bound  Running Time  Upper Bound

 Average Case

• Provides an estimate of “average” running time.

• Assumes that the input is random.
• Useful when best/worst cases do not happen very
often (i.e., few input cases lead to best/worst cases).

Lower Bound  Running Time Upper Bound

 Example: Searching
• Problem of searching an ordered list.
– Given a list L of n elements that are sorted into
a definite order (e.g., numeric, alphabetical),
– And given a particular element x,
– Determine whether x appears in the list, and if
so, return its index (i.e., position) in the list.
 Linear Search
procedure linear search
(x: integer, a1, a2, …, an: distinct integers)
i := 1
while (i  n  x  ai) NOT EFFICIENT!
i := i + 1
if i  n then location := i
else location := 0
return location
 How do we analyze an algorithm?

• Need to define objective measures.

(1) Compare execution times?

Not good: times are specific to a particular machine.

(2) Count the number of statements?

Not good: number of statements varies with
programming language and programming style.
 Example
Algorithm 1 Algorithm 2
 
arr[0] = 0; for(i=0; i<N; i++)
arr[1] = 0; arr[i] = 0;
arr[2] = 0;
...
arr[N-1] = 0;
 How do we analyze an algorithm?
(cont.)
(3) Express running time t as a function of
problem size n (i.e., t=f(n) ).

- Given two algorithms having running times f(n)

and g(n), find which functions grows faster.

- Such an analysis is independent of machine

time, programming style, etc.
 How do we find f(n)?
(1) Associate a "cost" with each statement.
(2) Find total number of times each statement is executed.
(3) Add up the costs.

Algorithm 1 Algorithm 2
Cost Cost
arr[0] = 0; c1 for(i=0; i<N; i++) c2
arr[1] = 0; c1 arr[i] = 0; c1
arr[2] = 0; c1
...
arr[N-1] = 0; c1 
----------- -------------
c1+c1+...+c1 = c1 x N (N+1) x c2 + N x c1 =
(c2 + c1) x N + c2
 How do we find f(n)? (cont.)

Cost
  sum = 0; c1
for(i=0; i<N; i++) c2
for(j=0; j<N; j++) c2
sum += arr[i][j]; c3
------------
c1 + c2 x (N+1) + c2 x N x (N+1) + c3 x N x N
 Comparing algorithms
• Given two algorithms having running times
f(n) and g(n), how do we decide which one
is faster?

•Compare “rates of growth” of f(n) and g(n)

 Understanding Rate of Growth

• Consider the example of buying elephants and

goldfish:

Cost: (cost_of_elephants) + (cost_of_goldfish)

Approximation:

Cost ~ cost_of_elephants
 Understanding Rate of Growth
(cont’d)

• The low order terms of a function are relatively

insignificant for large n

n4 + 100n2 + 10n + 50

Approximation:

n4

• Highest order term determines rate of growth!

 Example
• Suppose you are designing a website to process
user data (e.g., financial records).
• Suppose program A takes fA(n)=30n+8
microseconds to process any n records, while
program B takes fB(n)=n2+1 microseconds to
process the n records.
• Which program would you choose, knowing
you’ll want to support millions of users?

Compare rates of growth:

30n+8 ~ n and n2+1 ~ n2
 Visualizing Orders of Growth
• On a graph, as
you go to the

Value of function 
fA(n)=30n+8
right, a faster
growing
function
fB(n)=n2+1
eventually
becomes
larger... Increasing n 
 Rate of Growth ≡Asymptotic Analysis

• Using rate of growth as a measure to compare

different functions implies comparing them
asymptotically (i.e., as n   )

• If f(x) is faster growing than g(x), then f(x)

always eventually becomes larger than g(x) in
the limit (i.e., for large enough values of x).
 Asymptotic Notation

• O notation: asymptotic “less than”:

f(n)=O(g(n)) implies: f(n) “≤” c g(n) in the limit*

c is a constant

(used in worst-case analysis)

*
formal definition in CS477/677
 Asymptotic Notation

  notation: asymptotic “greater than”:

f(n)=  (g(n)) implies: f(n) “≥” c g(n) in the limit*
c is a constant

(used in best-case analysis)

*
formal definition in CS477/677
 Asymptotic Notation

  notation: asymptotic “equality”:

f(n)=  (g(n)) implies: f(n) “=” c g(n) in the limit*
c is a constant

(provides a tight bound of running time)

(best and worst cases are same)

*
formal definition in CS477/677
Big-O Notation - Examples

fA(n)=30n+8 is O(n)

fB(n)=n2+1 is O(n2)

10n3 + 2n2 is O(n3)

n3 - n2 is O(n3)

1273 is O(1)
 More on big-O
O(g(n)) is a set of functions f(n)
f(n) ϵ O(g(n)) if “f(n)≤cg(n)”
 Big-O Notation - Examples

fA(n)=30n+8 is O(n) or O(n2)

fB(n)=n2+1 is O(n2) or O(n4)

But it is important to
10n3 + 2n2 is O(n3) or O(n4) use as “tight” bounds
as possible!

n3 - n2 is O(n3) or O(n5)

1273 is O(1) or O(n)

Common orders of magnitude
Algorithm speed vs function growth
• An O(n ) algorithm will be slower than an O(n)
2

algorithm (for large n).

• But an O(n2) function will grow faster than an
O(n) function. Value of function 

fA(n)=30n+8

fB(n)=n2+1

Increasing n 
 Estimating running time
Algorithm 1 Algorithm 2
Cost Cost
arr[0] = 0; c1 for(i=0; i<N; i++) c2
arr[1] = 0; c1 arr[i] = 0; c1
arr[2] = 0; c1
...
arr[N-1] = 0; c1 
----------- -------------
c1+c1+...+c1 = c1 x N (N+1) x c2 + N x c1 =
(c2 + c1) x N + c2

O(N)
 Estimate running time (cont.)
Cost
  sum = 0; c1
for(i=0; i<N; i++) c2
for(j=0; j<N; j++) c2
sum += arr[i][j]; c3

------------
c1 + c2 x (N+1) + c2
O(Nx N2)x (N+1) + c3 x N x N
Running time of various statements
while-loop for-loop
 Examples
i = 0;
while (i<N) {
X=X+Y; // O(1)
result = mystery(X); // O(N), just an example...
i++; // O(1)
}
• The body of the while loop: O(N)
• Loop is executed: N times
N x O(N) = O(N2)
 Examples (cont.’d)
if (i<j)
O(N)
for ( i=0; i<N; i++ )
X = X+i;
else O(1)
X=0;

Max ( O(N), O(1) ) = O (N)

Examples (cont.’d)
Examples (cont.’d)
 Examples (cont.’d)
• Analyze the complexity of the following
code segments: