Fundamentals of the

Analysis of Algorithm
Efficiency
0750323 ALGORITHMS
DR. RANEEM QADDOURA

0750323 Algorithms Philadelphia University Dr. Raneem Qaddoura

Introduction to Design and Analysis of Algorithms – Anany Levitin
The Analysis Framework

0750323 Algorithms Philadelphia University Dr. Raneem Qaddoura

Introduction to Design and Analysis of Algorithms – Anany Levitin
Analysis of algorithms
Dimensions:
◦ Generality
◦ Simplicity
◦ Time efficiency
◦ Space efficiency
The term “analysis of algorithms” is usually used in a narrower, technical sense to mean an
investigation of an algorithm’s efficiency with respect to two resources: running time and memory
space

Approaches:
◦ Theoretical analysis
◦ Empirical analysis

0750323 Algorithms Philadelphia University Dr. Raneem Qaddoura

Introduction to Design and Analysis of Algorithms – Anany Levitin
Analysis of algorithms
•Time efficiency, also called time complexity, indicates how fast an algorithm
runs.
•Space efficiency, also called space complexity, refers to the amount of memory
units required by the algorithm in addition to the space needed for its input and
output.

0750323 Algorithms Philadelphia University Dr. Raneem Qaddoura

Introduction to Design and Analysis of Algorithms – Anany Levitin
Theoretical analysis of time efficiency
Time efficiency is analyzed by determining the number of repetitions of the basic operation as a
function of input size
Basic operation: the operation that contributes the most towards the running time of the algorithm
input size

T(n) ≈ copC(n)
running time execution time Number of times
for basic operation basic operation is
or cost executed
Note: Different basic operations may cost differently!

0750323 Algorithms Philadelphia University Dr. Raneem Qaddoura

Introduction to Design and Analysis of Algorithms – Anany Levitin
Measuring the Input’s size
•Almost all algorithms run longer on larger inputs.
•For example, it takes longer to sort larger arrays, multiply larger matrices, and so on.
•Therefore, it is logical to investigate an algorithm’s efficiency as a function of some parameter n
indicating the algorithm’s input size

0750323 Algorithms Philadelphia University Dr. Raneem Qaddoura

Introduction to Design and Analysis of Algorithms – Anany Levitin
Input size and basic operation examples
Problem Input size measure Basic operation
Searching for key in a list of n
Number of list’s items, i.e. n Key comparison
items
Matrix dimensions or total
Multiplication of two matrices Multiplication of two numbers
number of elements
Checking primality of a given n’size = number of digits (in
Division
integer n binary representation)
Visiting a vertex or traversing an
Typical graph problem #vertices and/or edges
edge

0750323 Algorithms Philadelphia University Dr. Raneem Qaddoura

Introduction to Design and Analysis of Algorithms – Anany Levitin
Types of formulas for basic operation’s
count
Exact formula
e.g., C(n) = n(n-1)/2

Formula indicating order of growth with specific multiplicative constant

e.g., C(n) ≈ 0.5 n2

Formula indicating order of growth with unknown multiplicative constant

e.g., C(n) ≈ cn2

0750323 Algorithms Philadelphia University Dr. Raneem Qaddoura

Introduction to Design and Analysis of Algorithms – Anany Levitin
Order of growth
Most important: Order of growth within a constant multiple as n→∞

Example:
◦ How much faster will algorithm run on computer that is twice as fast?

◦ How much longer does it take to solve problem of double input size?

0750323 Algorithms Philadelphia University Dr. Raneem Qaddoura

Introduction to Design and Analysis of Algorithms – Anany Levitin
 Units for measuring running time
T(n) ≈ copC(n)
• “How much faster would this algorithm run on a machine that is 10 times faster than the one we have?” The answer is,
obviously, 10 times.
1
• Assuming that 𝐶 𝑛 = 𝑛 𝑛 − 1 , how much longer will the algorithm run if we double its input size? The answer is
2
about four times longer.
1 1 1 1
•𝐶 𝑛 = 𝑛 𝑛 − 1 = 𝑛2 − 𝑛 ≈ 𝑛2
2 2 2 2
2 1
𝑇(2𝑛) copC(2n) 2(2𝑛)
• ≈ ≈ 1 2 =4
𝑇(𝑛) copC(n) 𝑛
2
• cop was neatly cancelled out in the ratio
1
• Multiplicative constant in the formula for the count C(n) which is , was also cancelled out
2
• The efficiency analysis framework ignores multiplicative constants and concentrates on the count’s order of growth to
within a constant multiple for large-size inputs.

0750323 Algorithms Philadelphia University Dr. Raneem Qaddoura

Introduction to Design and Analysis of Algorithms – Anany Levitin
Values of some important functions
as n → 

0750323 Algorithms Philadelphia University Dr. Raneem Qaddoura

Introduction to Design and Analysis of Algorithms – Anany Levitin
Example:
Sequential search
• Worst case: n key
comparisons
• when there are no matching
elements or the first matching
element happens to be the last
one on the list

• Best case: 1 comparisons

• When the list of size n with its
first element equals to a search
key

• Average case:
• K is in A => (n+1)/2 key comparisons
• K is not in A => n key comparisons

0750323 Algorithms Philadelphia University Dr. Raneem Qaddoura

Introduction to Design and Analysis of Algorithms – Anany Levitin
Best-case, average-case, worst-case
For some algorithms, efficiency depends on form of input:
•Worst case: Cworst(n) – maximum over inputs of size n
•The worst-case efficiency of an algorithm is its efficiency for the worst-case input of size n, which is
an input (or inputs) of size n for which the algorithm runs the longest among all possible inputs of
that size.
•Best case: Cbest(n) – minimum over inputs of size n
•The best-case efficiency of an algorithm is its efficiency for the best-case input of size n, which is an
input (or inputs) of size n for which the algorithm runs the fastest among all possible inputs of that
size.
•Average case: Cavg(n) – “average” over inputs of size n
◦ Number of times the basic operation will be executed on typical input
◦ NOT the average of worst and best case

0750323 Algorithms Philadelphia University Dr. Raneem Qaddoura

Introduction to Design and Analysis of Algorithms – Anany Levitin
Asymptotic Notations
and Basic Efficiency
Classes

0750323 Algorithms Philadelphia University Dr. Raneem Qaddoura

Introduction to Design and Analysis of Algorithms – Anany Levitin
Asymptotic order of growth
A way of comparing functions that ignores constant factors and small input sizes (because?)
• O(g(n)): class of functions f(n) that • Ω(g(n)): class of functions f(n) • Θ(g(n)): class of functions f(n)
grow no faster than g(n). that grow at least as fast as g(n) that grow at same rate as g(n)
• The set of all functions with a lower • The set of all functions with a • The set of all functions that
or same order of growth as g(n) higher or same order of growth have the same order of growth
as g(n) as g(n)

0750323 Algorithms Philadelphia University Dr. Raneem Qaddoura

Introduction to Design and Analysis of Algorithms – Anany Levitin
Big-oh vs Big-omega vs Big-theta
Big-oh Big-omega Big-theta

0750323 Algorithms Philadelphia University Dr. Raneem Qaddoura

Introduction to Design and Analysis of Algorithms – Anany Levitin
Formal definition
◦ O-notation: A function t(n) is said to be in O(g(n)), denoted t(n)  O(g(n)), if t(n) is
bounded above by some constant multiple of g(n) for all large n, i.e., if there exist
some positive constant c and some nonnegative integer n0 such that
t(n)  cg(n) for all n  n0
◦ -notation: A function t(n) is said to be in (g(n)), denoted t(n)  (g(n)), if t(n) is
bounded below by some constant multiple of g(n) for all large n, i.e., if there exist
some positive constant c and some nonnegative integer n0 such that
t(n)  cg(n) for all n  n0
◦ -notation: A function t(n) is said to be in (g(n)), denoted t(n)  (g(n)), if t(n) is
bounded both above and below by some positive constant multiples of g(n) for all
large n, i.e., if there exist some positive constant c1 and c2 and some nonnegative
integer n0 such that
c2 g(n)  t(n)  c1 g(n) for all n  n0

0750323 Algorithms Philadelphia University Dr. Raneem Qaddoura

Introduction to Design and Analysis of Algorithms – Anany Levitin
Some properties of asymptotic order of
growth
•f(n)  O(g(n)) iff g(n) (f(n))
• For example, 𝑛2  O(𝑛3 ) and 𝑛3  (𝑛2 )

•If f (n)  O(g (n)) and g(n)  O(h(n)) , then f(n)  O(h(n))
• For example, 𝑛  O(𝑛2 ) and 𝑛2  O(𝑛3 ) and 𝑛  O(𝑛3 )

•If f1(n)  O(g1(n)) and f2(n)  O(g2(n)) , then f1(n) + f2(n)  O(max{g1(n), g2(n)})
◦ True for the -notation and -notation.
◦ Implication: The algorithm’s overall efficiency will be determined by the part with a larger order of
growth, i.e., its least efficient part.
◦ For example, 5𝑛2 + 3𝑛  O(𝑛2 ) why?
◦ 5𝑛2  O(𝑛2 ) and 3𝑛  O(𝑛) then 5𝑛2 + 3𝑛  O{max {𝑛, 𝑛2 }} which is 5𝑛2 + 3𝑛  O(𝑛2 )

0750323 Algorithms Philadelphia University Dr. Raneem Qaddoura

Introduction to Design and Analysis of Algorithms – Anany Levitin
Mathematical Analysis
of Non-recursive
Algorithms

0750323 Algorithms Philadelphia University Dr. Raneem Qaddoura

Introduction to Design and Analysis of Algorithms – Anany Levitin
EXAMPLE 1: Max
value of an array
Input’s size is the number of
elements in the array, i.e., n.
The operations that are going to
be executed most often are in
the algorithm’s for loop
There are two operations in the
loop’s body: the comparison
(A[i] > maxval) and the
assignment (maxval ← A[i]).
Since the comparison is
executed on each repetition of
the loop and the assignment is
not, we should consider the
comparison to be the
algorithm’s basic operation.
0750323 Algorithms Philadelphia University Dr. Raneem Qaddoura
Introduction to Design and Analysis of Algorithms – Anany Levitin
EXAMPLE 1: Max
value of an array
The number of comparisons will
be the same for all arrays of size
n; therefore, in terms of this
metric, there is no need to
distinguish among the worst,
average, and best cases here.
C(n) the number of times this
comparison is executed and try
to find a formula expressing it
as a function of size n.
The algorithm makes one
comparison on each execution
of the loop, which is repeated
for each value of the loop’s
variable i within the bounds 1
and n − 1,

0750323 Algorithms Philadelphia University Dr. Raneem Qaddoura

Introduction to Design and Analysis of Algorithms – Anany Levitin
General Plan for Analyzing the Time
Efficiency of Nonrecursive Algorithms
1. Decide on a parameter (or parameters) indicating an input’s size.
2. Identify the algorithm’s basic operation. (As a rule, it is located in the innermost loop.)
3. Check whether the number of times the basic operation is executed depends only on the size
of an input. If it also depends on some additional property, the worst-case, average-case,
and, if necessary, best-case efficiencies have to be investigated separately.
4. Set up a sum expressing the number of times the algorithm’s basic operation is executed.
5. Using standard formulas and rules of sum manipulation, either find a closed form formula for
the count or, at the very least, establish its order of growth.

0750323 Algorithms Philadelphia University Dr. Raneem Qaddoura

Introduction to Design and Analysis of Algorithms – Anany Levitin
Summation formulas and rules useful in
analysis of algorithms

0750323 Algorithms Philadelphia University Dr. Raneem Qaddoura

Introduction to Design and Analysis of Algorithms – Anany Levitin
Example 2: Element
uniqueness problem
The natural measure of the
input’s size is n, the number of
elements in the array.
Since the innermost loop
contains a single operation (the
comparison of two elements),
we should consider it as the
algorithm’s basic operation.
The number of element
comparisons depends not only
on n but also on whether there
are equal elements in the array
and, if there are, which array
positions they occupy.
We will limit our investigation to
the worst case only.

0750323 Algorithms Philadelphia University Dr. Raneem Qaddoura

Introduction to Design and Analysis of Algorithms – Anany Levitin
Example 2: Element
uniqueness problem
By definition, the worst case input is an
array for which the number of element
comparisons 𝐶𝑤𝑜𝑟𝑠𝑡 (n) is the largest
among all arrays of size n.
An inspection of the innermost loop
reveals that there are two kinds of worst-
case inputs: Inputs for which the
algorithm does not exit the loop
prematurely:
• Arrays with no equal elements
• Arrays in which the last two elements
are the only pair of equal elements
For such inputs, one comparison is made
for each repetition of the innermost loop
i.e., for each value of the loop variable j
between its limits i + 1 and n − 1; this is
repeated for each value of the outer
loop, i.e., for each value of the loop
variable i between its limits 0 and n − 2.

0750323 Algorithms Philadelphia University Dr. Raneem Qaddoura

Introduction to Design and Analysis of Algorithms – Anany Levitin
Example 3: Matrix
Multiplication
We measure an input’s size by matrix order n.
There are two arithmetical operations in the
innermost loop here—multiplication and
addition—that, in principle, can compete for
designation as the algorithm’s basic
operation.
Actually, we do not have to choose between
them, because on each repetition of the
innermost loop each of the two is executed
exactly once.
By counting one we automatically count the
other.
Following a well-established tradition, we
consider multiplication as the basic operation.
Set up a sum for the total number of
multiplications M(n) executed by the
algorithm.
Since this count depends only on the size of
the input matrices, we do not have to
investigate the worst-case, average-case, and
best-case efficiencies separately.

0750323 Algorithms Philadelphia University Dr. Raneem Qaddoura

Introduction to Design and Analysis of Algorithms – Anany Levitin
Example 3: Matrix
Multiplication
There is just one multiplication
executed on each repetition of
the algorithm’s innermost loop,
which is governed by the
variable k ranging from the
lower bound 0 to the upper
bound n − 1.
The number of multiplications
made for every pair of specific
values of variables i and j is:
σ𝑛−1
𝑘=0 1

Total number of multiplications

M(n) is expressed by the
following triple sum:

0750323 Algorithms Philadelphia University Dr. Raneem Qaddoura

Introduction to Design and Analysis of Algorithms – Anany Levitin
Example 3: Matrix
Multiplication
Running time of the algorithm
on a particular machine

where 𝑐𝑚 is the time of one

multiplication on the machine in
question.
A more accurate estimate if we
took into account the time
spent on the additions

where 𝑐𝑎 is the time of one

addition.

0750323 Algorithms Philadelphia University Dr. Raneem Qaddoura

Introduction to Design and Analysis of Algorithms – Anany Levitin
Mathematical Analysis
of Recursive Algorithms

0750323 Algorithms Philadelphia University Dr. Raneem Qaddoura

Introduction to Design and Analysis of Algorithms – Anany Levitin
General Plan for Analyzing the Time
Efficiency of Recursive Algorithms
1. Decide on a parameter (or parameters) indicating an input’s size.
2. Identify the algorithm’s basic operation.
3. Check whether the number of times the basic operation is executed can vary on different
inputs of the same size; if it can, the worst-case, average-case, and best-case efficiencies
must be investigated separately.
4. Set up a recurrence relation, with an appropriate initial condition, for the number of times
the basic operation is executed.
5. Solve the recurrence or, at least, ascertain the order of growth of its solution

0750323 Algorithms Philadelphia University Dr. Raneem Qaddoura

Introduction to Design and Analysis of Algorithms – Anany Levitin
Example 1:
Factorial
For simplicity, we consider n itself
as an indicator of this algorithm’s
input size (rather than the
number of bits in its binary
expansion)
The basic operation of the
algorithm is multiplication, whose
number of executions we denote
M(n).
we use what can be called the
method of backward substitutions
Take advantage of the initial
condition given. Since it is
specified for n = 0, we have to
substitute i = n in the pattern’s
formula to get the ultimate result
of our backward substitutions

0750323 Algorithms Philadelphia University Dr. Raneem Qaddoura

Introduction to Design and Analysis of Algorithms – Anany Levitin
Example 2: Number
of binary digits
Let us set up a recurrence and an
initial condition for the number of
additions A(n) made by the
algorithm. The number of
additions made in computing
BinRec(n/2) is A(n/2), plus one
more addition is made by the
algorithm to increase the
returned value by 1.
The presence of n/2 in the
function’s argument makes the
method of backward substitutions
stumble on values of n that are
not powers of 2.
Therefore, the standard approach
to solving such a recurrence is to
solve it only for n = 2𝑘 and then
take advantage of the theorem
called the smoothness rule

0750323 Algorithms Philadelphia University Dr. Raneem Qaddoura

Introduction to Design and Analysis of Algorithms – Anany Levitin
Example 3:
Tower of Hanoi
In this puzzle, we have n disks of different
sizes that can slide onto any of three
pegs.
Initially, all the disks are on the first peg
in order of size, the largest on the bottom
and the smallest on top.
The goal is to move all the disks to the
third peg, using the second one as an
auxiliary, if necessary.
We can move only one disk at a time, and
it is forbidden to place a larger disk on
top of a smaller one.
To move n > 1 disks from peg 1 to peg 3
(with peg 2 as auxiliary), we first move
recursively n − 1 disks from peg 1 to peg 2
(with peg 3 as auxiliary), then move the
largest disk directly from peg 1 to peg 3,
and, finally, move recursively n − 1 disks
from peg 2 to peg 3 (using peg 1 as
auxiliary). Of course, if n = 1, we simply
move the single disk directly from the
source peg to the destination peg.

0750323 Algorithms Philadelphia University Dr. Raneem Qaddoura

Introduction to Design and Analysis of Algorithms – Anany Levitin
Example 3:
Tower of Hanoi
The number of disks n is the
obvious choice for the input’s
size indicator, and so is moving
one disk as the algorithm’s basic
operation.
we have an exponential
algorithm, which will run for an
unimaginably long time even for
moderate values of n. This is not
due to the fact that this
particular algorithm is poor; in
fact, it is not difficult to prove
that this is the most efficient
algorithm possible for this
problem. It is the problem’s
intrinsic difficulty that makes it
so computationally hard.

0750323 Algorithms Philadelphia University Dr. Raneem Qaddoura

Introduction to Design and Analysis of Algorithms – Anany Levitin
Example 3:
Tower of Hanoi
When a recursive algorithm
makes more than a single call to
itself, it can be useful for
analysis purposes to construct a
tree of its recursive calls.
In this tree, nodes correspond
to recursive calls, and we can
label them with the value of the
parameter (or, more generally,
parameters) of the calls.
By counting the number of
nodes in the tree, we can get
the total number of calls made
by the Tower of Hanoi algorithm

0750323 Algorithms Philadelphia University Dr. Raneem Qaddoura

Introduction to Design and Analysis of Algorithms – Anany Levitin
Empirical analysis of
time efficiency

0750323 Algorithms Philadelphia University Dr. Raneem Qaddoura

Introduction to Design and Analysis of Algorithms – Anany Levitin
Empirical analysis of time efficiency
•Select a specific (typical) sample of inputs

•Use physical unit of time (e.g., milliseconds)

or
Count actual number of basic operation’s executions

•Analyze the empirical data

0750323 Algorithms Philadelphia University Dr. Raneem Qaddoura

Introduction to Design and Analysis of Algorithms – Anany Levitin
References
Levitin, A. (2011). Introduction to the design & analysis of algorithms. Boston: Pearson.

0750323 Algorithms Philadelphia University Dr. Raneem Qaddoura

Introduction to Design and Analysis of Algorithms – Anany Levitin