Algorithm Analysis and Recursion

CNG 213 - Data Structures

Lecture - 2
Dr. Meryem Erbilek

CNG 213 - Lecture 2 1/59

 Data vs. Information

•What is the difference between data, information and

knowledge?

•Why do we need data structures?

CNG 213 - Lecture 2 2/59

 Data vs. Information
Data Information
Raw facts Data with context
No context Processed data
Just numbers and text Interpreted data
Value-added to data
summarized
organized
analyzed

CNG 213 - Lecture 2 3/59

 Data vs. Information vs. Knowledge
Example
• Data:
– 100

• Information:
– 100 Miles

• Knowledge :
– 100 Miles is a quite far distance

CNG 213 - Lecture 2 4/59

 Converting Data to Information

Data Interpretation/ Information

understanding/Analyzing

Data Information New Information

CNG 213 - Lecture 2 5/59

Data  Information  Knowledge
Data

Summarizing the data

Averaging the data
Selecting part of the data
Graphing the data
Adding context
Adding value

Information
CNG 213 - Lecture 2 6/55
Data  Information  Knowledge
Information

How is the info tied to outcomes?

Are there any patterns in the info?
What info is relevant to the problem?
How does this info effect the system?
What is the best way to use the info?
How can we add more value to the info?

Knowledge
CNG 213 - Lecture 2 7/55
 What is Data Structure?
• A data structure is a way of storing data in a
computer so that it can be used efficiently.
– It is an organisation of mathematical and logical
concepts of data.

• Different kinds of data structures are suited to

different kinds of applications, and some are highly
specialized to certain tasks.
– For example, B-trees are particularly well-suited for
implementation of databases.

CNG 213 - Lecture 2 8/59

 Why Data Structure?
• A well-designed data structure allows a variety of
critical operations to be performed at minimum cost.
– As less resources as possible such as memory space.
– As less execution time as possible.

• In other words, data structures are needed in order to

be able to design efficient algorithms.

• In some applications domains, designing a program

is not feasible without proper data structure design.

CNG 213 - Lecture 2 9/59

 Program = Data Structure + Algorithm
Problem Analysis
After the data structures are chosen,
the algorithms to be used often
Solution Design become relatively obvious.

Data Structure Design Sometimes things work in the opposite

direction. Data structures are chosen
Algorithms Design
because certain key tasks have
algorithms that work best with
User Interface Design
particular data structures.

Coding\testing\deploying In either case, the choice of

appropriate data structures is crucial.

CNG 213 - Lecture 2 10/59

 Algorithm
• An algorithm is a set of instructions to be followed to solve
a problem.
– There can be more than one solution (more than one
algorithm) to solve a given problem.
– An algorithm can be implemented using different
programming languages on different platforms.
• An algorithm must be correct and it should correctly solve
the problem.
• Once we have a correct algorithm for a problem, we have
to determine the efficiency (resources & time) of that
algorithm.

CNG 213 - Lecture 2 11/59

 Why Algorithm Analysis?
• Generally, we use a computer because we need to
process a large amount of data.

• When we run a program on large amounts of input,

besides to make sure the program is correct, we must be
certain that the program terminates within a reasonable
amount of time.

• Algorithm anaylsis: process of determining the amount of

time, resources, etc. required when executing an
algorithm. It is only performed on correct algorithms.

CNG 213 - Lecture 2 12/59

 Algorithmic Performance
• There are two aspects of algorithmic
performance:
– Time complexity:
– Instructions take time…
– How fast does the algorithm perform?
– What affects its runtime?

– Space complexity:
– Data structures take space…
– What kind of data structures can be used?
– How does the choice of data structure affect the runtime?

• We will focus on time complexity:

– How to estimate the time required for an algorithm?
– How to reduce the time required?
CNG 213 - Lecture 2 13/59
 Analysis of Algorithms
• Analysis of Algorithms is the area of computer science
that provides tools to analyze the efficiency of different
methods of solutions.

• How do we compare the time efficiency of two algorithms

that solve the same problem?
(Algorithmic performance - time complexity)

– Naïve Approach - Experimental Approach

– Theoretical Approach

CNG 213 - Lecture 2 14/59

Naïve Approach - Experimental Approach
• Write a program that implements the algorithm (e.g., C or C++);
• Run the program with data sets of varying size.
• Determine the actual running time using a system call to measure
time (e.g. system (date) );
• Problems?
– How are the algorithms coded?
• Comparing running times means comparing the implementations.
• We should not compare implementations, because they are
sensitive to programming style that may cloud the issue of which
algorithm is inherently more efficient.
– What computer should we use?
• We should compare the efficiency of the algorithms independent of
a particular computer.
– What data should the program use?
• An analysis must be independent of specific data.

CNG 213 - Lecture 2 15/59

 Theoretical Approach
• When we analyze algorithms, we should employ mathematical
techniques independent of specific implementations, computers, or
data.
– Based on a high-level description of the algorithms, rather than language-
dependent implementations.
– Makes possible an evaluation of the algorithms that is independent of the
hardware and software environments.

• To analyze algorithms:
– First, we start to count the number of significant operations in a particular
solution to assess its efficiency.
– Then, we will express the efficiency of algorithms using growth functions.

Generality!!!

CNG 213 - Lecture 2 16/59

 What to Analyze?
• An algorithm can require different times to solve different problems of the
same size.
• Worst-Case Analysis:The maximum amount of time that an algorithm requires to
solve a problem of size n.
– This gives an upper bound for the time complexity of an algorithm.

• Best-Case Analysis:The minimum amount of time that an algorithm requires to solve

a problem of size n.
– The best-case behavior of an algorithm is NOT so useful.

• Average-Case Analysis:The average amount of time that an algorithm requires to

solve a problem of size n.
– Sometimes, it is difficult to find the average-case behavior of an algorithm.
– We have to look at all possible data organizations of a given size n, and their
distribution probabilities of these organizations.
• Worst-case analysis is more common than average-case analysis.

CNG 213 - Lecture 2 17/59

 What to Analyze?
• For example; Searching an item in a list of n elements
using sequential (linear) search.

• What will be best, worst and average case analysis?

CNG 213 - Lecture 2 18/59

 What is Important?
• We have to weigh the trade-offs between an algorithm’s
time requirement and memory requirements.

• We have to compare algorithms for both style and

efficiency.
– The analysis should focus on gross differences in efficiency and
not reward coding tricks that save a small amount of time.
– That is, there is no need for coding tricks if the gain is not too
much.
– Easily understandable program is also important.

• Order-of-magnitude analysis focuses on large problems.

CNG 213 - Lecture 2 19/59

 Algorithm Description
• How to describe algorithms independent of a programming
language?
– Pseudo-Code: a description of an algorithm that is more structured than usual
prose but less formal than a programming language.
– Diagrams (Flowcharts)

• Example: find the maximum element of an array.

Algorithm arrayMax(A, n):
Input: An array A storing n integers.
Output: The maximum element in A.
currentMax  A[0]
for i 1 to n -1 do
if currentMax < A[i] then currentMax  A[i]
return currentMax

CNG 213 - Lecture 2 20/59

 Pseudo Code
• Expressions: use standard mathematical symbols
– use  for assignment ( ? in C/C++)
– use = for the equality relationship (? in C/C++)
• Method Declarations:
– Algorithm name(param1, param2)
• Programming Constructs:
– decision structures: if ... then ... else ...
– while-loops: while ... do
– repeat-loops: repeat ... until ...
– for-loop: for ... to ... do
– array indexing: A[i]
• Methods
– calls: object method (args)
– returns: return value
• Use comments
• Instructions have to be basic enough and feasible!
21/59
 Low Level Algorithm Analysis
• Assess efficiency : Inspect the pseudo-code and count the
number of primitive operations executed by the algorithm.

• Primitive operations (low-level computations independent from

the programming language)

• E.g.:
– Make an addition = 1 operation
– Calling a method or returning from a method = 1 operation
– Index in an array = 1 operation
– Comparison = 1 operation etc.

CNG 213 - Lecture 2 22/59

 Formulating the Execution Time of Algorithms
• Each operation in an algorithm (or a program) has a cost (a
certain amount of time).
• Example:
count = count + 1;
 takes a certain amount of time, but it is constant.

• Example: A sequence of operations;

count = count + 1 Cost: c1  Time:1

sum = sum + count; Cost: c2  Time:1 Total Cost = c1 + c2

CNG 213 - Lecture 2 23/59

 The Execution Time of Algorithms (cont.)

• Example: Simple If-Statement

Cost Times
if (n < 0) c1 1
absval = -n c2 1
else
absval = n; c3 1

Total Cost <= c1 + max(c2,c3)

CNG 213 - Lecture 2 24/59

 The Execution Time of Algorithms (cont.)

• Example: Simple Loop

Cost Times
i = 1; c1 1
sum = 0; c2 1
while (i <= n){ c3 n+1
i = i + 1; c4 n
sum = sum + i;} c5 n

Total Cost = c1 + c2 + (n+1)*c3 + n*c4 + n*c5

The time required for this algorithm is proportional to n

CNG 213 - Lecture 2 25/59

 The Execution Time of Algorithms (cont.)

• Example: Nested Loop

Cost Times
i=0; c1 1
sum = 0; c2 1
while (i < n){ c3 n+1
j=1; c4 n
while (j <= n){ c5 n*(n+1)
sum = sum + i; c6 n*n
j = j + 1;} c7 n*n
i = i +1; } c8 n

Total Cost = c1 + c2 + (n+1)*c3 + n*c4 + n*(n+1)*c5+n*n*c6+n*n*c7+n*c8

The time required for this algorithm is proportional to n2

CNG 213 - Lecture 2 26/59

 General Rules for Estimation of Execution Time
• Loops: The running time of a loop is at most the running
time of the statements inside of that loop times the number
of iterations.
• Nested Loops: Running time of a nested loop containing a
statement in the inner most loop is the running time of
statement multiplied by the product of the size of all outer
loops.
• Consecutive Statements: Just add the running times of
those consecutive statements.
• If/Else: Never more than the running time of the test plus
the larger of running times of S1 and S2.

CNG 213 - Lecture 2 27/59

 Algorithm Growth Rates
• An algorithm’s formulated proportional time requirement is known as
growth rate. We can compare the efficiency of the two algorithms
by comparing their growth rates.
• We measure an algorithm’s time requirement as a function of the
problem size.
– Problem size depends on the application: e.g. number of elements in a list for a
sorting algorithm.

• So, for instance, we say that (if the problem size is n)

– Algorithm A requires n2/5 time units to solve a problem of size n.
– Algorithm B requires 5*n time units to solve a problem of size n.

• The most important thing to learn is how quickly the algorithm’s time
requirement grows as a function of the problem size.

CNG 213 - Lecture 2 28/59

 Algorithm Growth Rates (cont.)
Example: Time requirements as a function of the problem size n

CNG 213 - Lecture 2 29/55

Common Growth Rates
Function Growth Rate Name
c Constant
log N Logarithmic
log2N Log-squared
N Linear
N log N Linear x log
N2 Quadratic
N3 Cubic
2N or eN Exponential

30/55
CNG 213 - Lecture 2
 Running times for small inputs
Running time

Input size (N)

CNG 213 - Lecture 2 31/59
Upper and Lower Bounds of a Function

Upper bound

lower bound

CNG 213 - Lecture 2 32/59

 Running-Time of Algorithms
• Bounds are for the algorithms, rather than programs
– Programs are just implementations of an algorithm.

– Almost always the details of the program do not affect

the bounds.

– A problem can be solved with several algorithms, some

are more efficient than others.

CNG 213 - Lecture 2 33/59

 Big-Oh - Definition
• A theoretical measure of the execution of an
algorithm, usually the time, given the problem size n,
which is usually the number of items.

• Notation example: f(n) = O(g(n))

– Informally, saying some equation f(n) = O(g(n)) means
it is less than some constant multiple of g(n).

• The notation is read, "f of n is big oh of g of n".

CNG 213 - Lecture 2 34/59

 Big-Oh - Formal Definition

• f(N) = O(g(N))

• if and only if there are positive

constants c and n0 such that
f(N)  c g(N) when N  n0

• The growth rate of f(N) is less than

or equal to the growth rate of g(N)

• g(N) is an upper bound on f(N)

Also called asymptotic upper bound.

CNG 213 - Lecture 2 35/59
 Order-of-Magnitude Analysis
and Big Oh Notation
• If Algorithm A requires time proportional to f(n);
– Algorithm A is said to be order f(n), and it is denoted as O(f(n)).

• The function f(n) is called the algorithm’s growth-rate

function.

• Since the capital O is used in the notation, this notation is called

the Big O notation.

• If Algorithm A requires time proportional to n2, it is O(n2).

• If Algorithm A requires time proportional to n, it is O(n).

CNG 213 - Lecture 2 36/59

 Order of an Algorithm
• If an algorithm requires f(n) = n2–3*n+10 seconds to solve a
problem size n.

• If constants c and n0 exist such that f(N)  c g(N) when N  n0

n2–3*n+10  c*n2 for all n  n0

the algorithm is order n2 (In fact, c is 3 and n0 is 2)

n2–3*n+10  c*n2 for all n  2 .

• Thus, the algorithm requires no more than c*n2 time units for
n  n0 , so it is O(n2) 37
CNG 213 - Lecture 2 37/59
 Big-Oh: Examples
• “n² + 3n + 4” is O(n²) f(N) = O(g(N))
if and only if there are
positive constants c and n0
– since n² + 3n + 4 < 2n² for all n > 10
such that
f(N)  c g(N) when N  n0

• “n3 + 3n + 4” is O(n3)

– since n3 + 3n + 4 < 3 n3 for all n > 5

CNG 213 - Lecture 2 38/59

 Big-Oh: Examples
• (N2 / 2) – 3N = O(N2)

• 1 + 4N = O(N)

• 7N2 + 10N + 3 = O(N2) = O(N3)

• log10 N = log2 N / log2 10 = O(log2 N) = O(log N)

• sin N = O(1)

• 10 = O(1)

• 1010 = O(1)

• log N + N = O(N)

CNG 213 - Lecture 2 39/59

A Comparison of Growth-Rate Functions

CNG 213 - Lecture 2 40/55

 General Rules
• Loops
– At most the running time of the statements inside the loop
(including tests) times the number of iterations.

O(N)

• Nested loops
– The running time of the statement multiplied by the product of
the sizes of all the loops.

O(N2)
CNG 213 - Lecture 2 41/59
 General rules (cont’d)
• Consecutive statements
– These are just adds.

O(N) + O(N2) = O(N2)

• If Statement
If S1 Else S2

larger of the running times of S1 and S2.

CNG 213 - Lecture 2 42/59
 Properties of Growth-Rate Functions
1. We can ignore low-order terms in an algorithm’s growth-rate function.
– If an algorithm is O(n3+4n2+3n), it is also O(n3).
– We only use the higher-order term as the algorithm’s growth-rate
function.

2. We can ignore a multiplicative constant in the higher-order term of an

algorithm’s growth-rate function.
– If an algorithm is O(5n3), it is also O(n3).

3. We can combine growth-rate functions: O(f(n)) + O(g(n)) = O(f(n)+g(n))

– If an algorithm is O(n3) + O(4n), it is also O(n3 +4n)  So, it is O(n3).
– Similar rules hold for multiplication.

CNG 213 - Lecture 2 43/59

 Growth-Rate Functions
• If an algorithm takes 1 second to run with problem size 8, what is
the time requirement (approximately) for that algorithm with
problem size 16?

• If its order is:

O(1)  T(n) = 1 second
O(log2n)  T(n) = (1*log216) / log28 = 4/3 seconds
O(n)  T(n) = (1*16) / 8 = 2 seconds
O(n*log2n)  T(n) = (1*16*log216) / 8*log28 = 8/3 seconds
O(n2)  T(n) = (1*162) / 82 = 4 seconds
O(n3)  T(n) = (1*163) / 83 = 8 seconds
O(2n)  T(n) = (1*216) / 28 = 28 seconds = 256 seconds

CNG 213 - Lecture 2 44/59

A Comparison of Growth-Rate Functions

CNG 213 - Lecture 2 45/55

 Growth-Rate Functions – Example1
Cost Times
i = 1; c1 1
sum = 0; c2 1
while (i <= n) { c3 n+1
i = i + 1; c4 n
sum = sum + i; } c5 n

T(n) = c1 + c2 + (n+1)*c3 + n*c4 + n*c5

= (c3+c4+c5)*n + (c1+c2+c3)
= a*n + b

 So, the growth-rate function for this algorithm is O(n)

CNG 213 - Lecture 2 46/59

 Growth-Rate Functions – Example2
Cost Times
i=1; c1 1
sum = 0; c2 1
while (i <= n) { c3 n+1
j=1; c4 n
while (j <= n) { c5 n*(n+1)
sum = sum + i; c6 n*n
j = j + 1; } c7 n*n
i = i +1; } c8 n

T(n) = c1 + c2 + (n+1)*c3 + n*c4 + n*(n+1)*c5+n*n*c6+n*n*c7+n*c8

= (c5+c6+c7)*n2 + (c3+c4+c5+c8)*n + (c1+c2+c3)
= a*n2 + b*n + c

 So, the growth-rate function for this algorithm is O(n2)

CNG 213 - Lecture 2 47/59

 Growth-Rate Functions
Recursive Algorithms
A Recursive Algorithm Example:
Towers of Hanoi Problem:
– Source peg, Destination peg, Auxilliary peg
– k disks on the source peg, a bigger disk can never be on
top of a smaller disk
– Need to move all k disks to the destination peg using the
auxilliary peg, without ever keeping a bigger disk on the
smaller disk.

CNG 213 - Lecture 2 48/59

 Growth-Rate Functions - Hanoi Towers

void hanoi(int n, char source, char dest, char spare) { Cost

if (n > 0) { c1
hanoi(n-1, source, spare, dest); c2
printf("Move top disk from %c pole to pole %c", c3
source, dest);
hanoi(n-1, spare, dest, source); c4
} }

CNG 213 - Lecture 2 49/59

 Growth-Rate Functions - Hanoi Towers

• The time-complexity function T(n) of a recursive

algorithm is defined in terms of itself, and this is
known as recurrence equation for T(n).

• To find the growth-rate function for that recursive

algorithm, we have to solve that recurrence relation.

CNG 213 - Lecture 2 50/59

 Growth-Rate Functions – Hanoi Towers
• What is the cost of hanoi(n,’A’,’B’,’C’)?
void hanoi(int n, char source, char dest, char spare) { Cost
if (n > 0) { c1
hanoi(n-1, source, spare, dest); c2
printf("Move top disk from %c pole to pole %c", c3
source, dest);
hanoi(n-1, spare, dest, source); c4
} }
Base case: when n=0  T(0) = c1
Recursive case: when n>0
T(n) = c1 + c2 + T(n-1) + c3 + c4 + T(n-1)
= 2*T(n-1) + (c1+c2+c3+c4)
= 2*T(n-1) + c  recurrence equation for the growth-rate function of hanoi-towers algorithm

• Now, we have to solve this recurrence equation to find the growth-rate function of
hanoi-towers algorithm.

CNG 213 - Lecture 2 51/59

 Growth-Rate Functions – Hanoi Towers
• There are many methods to solve recurrence equations, but we will use a
simple method known as repeated substitutions.

T(n) = 2*T(n-1) + c
= 2 * (2*T(n-2)+c) + c
= 2 * (2* (2*T(n-3)+c) + c) + c
= 23 * T(n-3) + (22+21+20)*c (assuming n>2)
when substitution repeated i-1th times
= 2i * T(n-i) + (2i-1+ ... +21+20)*c
when i=n
= 2n * T(0) + (2n-1+ ... +21+20)*c
= 2n * c1 + ( )*c
= 2n * c1 + ( 2n-1 )*c = 2n*(c1+c) – c  So, the growth rate function is
O(2n)

CNG 213 - Lecture 2 52/59

 More Recursive Algorithms
What is the cost of;

int fact(int n)
{
if (n ==0)
return (1);
else
return (n * fact(n-1));
}

CNG 213 - Lecture 2 53/59

 More Recursive Algorithms
What is the cost of;

bool isPalindrome (string s)

{
if (s.length() <= 1)
return true;

return (s[0]==s[s.length()-1]) &&

isPalindrome(s.substr(1,s.length()-2));
}

CNG 213 - Lecture 2 54/59

 Big-Omega
• f(N) = (g(N)) if there are positive
constants c and n0 such that
f(N)  c g(N) when N  n0

• The growth rate of f(N) is greater

than or equal to the growth rate of
g(N).

CNG 213 - Lecture 2 55/59

 Big-Omega: example

• Let f(N) = 2N2. Then f(N) = (g(N)) if there are

positive constants c and n0 such
that f(N)  c g(N) when N  n0
– 2N2  cN, for c=1 and N  1
f(N) = (N)

– f(N) = (N2) (best answer)

CNG 213 - Lecture 2 56/59

 Big-Theta

• f(N) = (g(N)) iff

f(N) = O(g(N)) and f(N) =
(g(N))

• The growth rate of f(N) equals

the growth rate of g(N)

CNG 213 - Lecture 2 57/59

 Big-Theta- Example

• Let f(N)=2N2 , g(N)=N2

– Since f(N) = O(g(N)) for c2 >2 and
– f(N) = (g(N)) for c1 < 2,
thus f(N) = (g(N)).

• Big-Theta means the bound is the tightest possible.

CNG 213 - Lecture 2 58/59

 Summary
• What is data? What is information? Differences?
• What is data structure? Why?
• What is an algorithm?
• Algorithm analysis, why? how?
• Algorithmic performance
• Algorithm description, Pseudo Code
• Execution time of algorithms, growth rates
• Big-Oh, what is it? how to compute?
• Growth rate, examples, recursive functions, Hanoi Tower
example
• Big-Omega
• Big-Theta

CNG 213 - Lecture 2 59/59