CS211 - Data Structure and Algorithms

Algorithm Analysis
Lecture # 4

1
Nasir Uddin
 CS211 - Data Structure and Algorithms

Introduction
• Algorithms are designed to solve problems, but a given problem can
have many different solutions.
• How then are we to determine which solution is the most efficient for
a given problem?
• One approach is to measure the execution time, but the execution
time is dependent on several factors.
• Input data
• Hardware
• programming language and compiler

2
Nasir Uddin
 CS211 - Data Structure and Algorithms

Complexity Analysis
• To determine the efficiency of an algorithm,
• count the number of logical comparisons (>, <, ≤, ≥, ! =, =, …)
• data interchanges, or arithmetic operations. (+, −,×,÷, …)

• Version 1 complexity  2𝑛2

• Version 2 complexity  𝑛2 + 𝑛

3
Nasir Uddin
 CS211 - Data Structure and Algorithms

Complexity Analysis
• Version 1 complexity  2𝑛2
• Version 2 complexity  𝑛2 + 𝑛

4
Nasir Uddin
 CS211 - Data Structure and Algorithms

Complexity Analysis – Big ‘O’ notation

• Assume we have a function 𝑇 𝑛 that represents the approximate
number of steps required by an algorithm for an input of size n. For
the second version of our algorithm in the previous section, this
would be written as,
𝑇 𝑛 = 𝑛2 + 𝑛
• Now, suppose there exists a function f(n) defined for the integers 𝑛 ≥
0, such that for some constant 𝑐, and some constant 𝑚,
𝑇 𝑛 ≤ 𝑐𝑓(𝑛)
for all sufficiently large values of 𝑛 ≥ 𝑚.
• Then, such an algorithm is said to have a time-complexity of, or
executes on the order of, 𝑓(𝑛) relative to the number of operations it
requires. 5
Nasir Uddin
 CS211 - Data Structure and Algorithms

Complexity Analysis – Big ‘O’ notation

• The function 𝑓(𝑛) indicates the rate of growth at which the run time
of an algorithm increases as the input size, 𝑛, increases.
• To specify the time-complexity of an algorithm, which runs on the
order of 𝑓(𝑛), we use the notation, 𝒪(𝑓 𝑛 )
• Consider the two versions of our algorithm from earlier. For version
one, the
• time was computed to be 𝑇1 𝑛 = 2𝑛2 . If we let 𝑐 =, 2, then 2𝑛2 ≤ 2𝑛2
for a result of 𝒪(𝑛2 ).

6
Nasir Uddin
 CS211 - Data Structure and Algorithms

Complexity Analysis – Big ‘O’ notation

• For version two, we computed a time of 𝑇2 𝑛 = 𝑛2 + 𝑛. Again, if we let 𝑐 = 2,
then,
𝑛2 + 𝑛 ≤ 2𝑛2
for a result of 𝒪(𝑛2 ). In this case, the choice of 𝑐 comes from the observation that
when 𝑛 ≥ 1, we have 𝑛 ≤ 𝑛2 and 𝑛2 + 𝑛 ≤ 𝑛2 + 𝑛2 , which satisfies the equation
in the definition of big-O.
• The function 𝑓 𝑛 = 𝑛2 is not the only choice for satisfying the condition 𝑇 𝑛 ≤
𝑐𝑓(𝑛). We could have said the algorithms had a run time of 𝒪(𝑛3 ) or 𝒪 𝑛4 .
• Since 2𝑛2 ≤ 𝑛3 and 2𝑛2 ≤ 𝑛4 when 𝑛 > 1.
• The objective, however, is to find a function 𝑓(. ) that provides the tightest
(lowest) upper bound or limit for the run time of an algorithm.
• The big-O notation is intended to indicate an algorithm’s efficiency for large
values of 𝑛. There is usually little difference in the execution times of algorithms
when n is small. 7
Nasir Uddin
 CS211 - Data Structure and Algorithms

Complexity Analysis
• Constant of Proportionality
• The constant of proportionality "𝑐“ is only
crucial when two algorithms have the
same 𝑓(𝑛).
• Constructing 𝑇(𝑛)
• Assume that each basic operation or
statement, at the abstract level, takes the
same amount of time and, thus, each is
assumed to cost 𝑐𝑜𝑛𝑠𝑡𝑎𝑛𝑡 𝑡𝑖𝑚𝑒.
• The total number of operations required
by an algorithm can be computed as a
sum of the times required to perform
each step:

8
Nasir Uddin
 CS211 - Data Structure and Algorithms

Complexity Analysis
• Choosing the Function
• The function f(n) used to categorize a particular algorithm is chosen to be the
dominant term within 𝑇(𝑛).
• That is, the term that is so large for big values of n, that we can ignore the
other terms when computing a big-O value. For example, in the expression
𝑛2 + log 2 𝑛 + 3𝑛
the term n2 dominates the other terms since for 𝑛 ≥ 3, we have
𝑛2 + log 2 𝑛 + 3𝑛 ≤ 𝑛2 + 𝑛2 + 𝑛2
𝑛2 + log 2 𝑛 + 3𝑛 ≤ 3𝑛2

Which leads to a time-complexity of 𝒪 𝑛2 .

9
Nasir Uddin
 CS211 - Data Structure and Algorithms

Complexity Analysis
• Classes of Algorithms

10
Nasir Uddin
 CS211 - Data Structure and Algorithms

Complexity Analysis
• Polynomial Algorithms
• an efficiency expressed as a polynomial of the form are characterized by a
time-complexity of 𝒪(𝑛𝑚 ) since the dominant term is the highest power of 𝑛.

𝑎𝑚 𝑛𝑚 + 𝑎𝑚−1 𝑛𝑚−1 + 𝑎𝑚−2 𝑛𝑚−2 + ⋯ + 𝑎2 𝑛2 + 𝑎1 𝑛 + 𝑎0

• An algorithm whose efficiency is characterized by a dominant term in the

form 𝑎𝑛 is called exponential.
• Exponential algorithms are among the worst algorithms in terms of time-
complexity.

11
Nasir Uddin
 CS211 - Data Structure and Algorithms

Evaluating Python Code

• Constant Time
• 𝒪(1)
• Linear Time
• 𝑇 𝑛 = 𝑛, 𝒪 𝑛
• 𝑇 𝑛 = 𝑛 + 𝑛 = 2𝑛, 𝒪(𝑛)

12
Nasir Uddin
 CS211 - Data Structure and Algorithms

Evaluating Python Code

• Quadratic Time Examples
• 𝑇 𝑛 = 𝑛 ∗ 𝑛 = 𝑛2 , 𝒪(𝑛2 )
• 𝑇 𝑛 = 𝑛, 𝒪(𝑛)
𝑛(𝑛+1) 𝑛2 +𝑛
•𝑇 𝑛 = = , 𝒪(𝑛2 )
2 2

13
Nasir Uddin
 CS211 - Data Structure and Algorithms

Evaluating Python Code

• Logarithmic Time Examples def ex6( n ):
• 𝑇 𝑛 = log 2 𝑛 + 1, 𝒪(log 2 𝑛) count = 0
i=n
• 𝒪(𝑛 log 2 𝑛) while i >= 1 :
count += 1
i = i // 2
return count

14
Nasir Uddin
 CS211 - Data Structure and Algorithms

Complexity Analysis – Different cases

• Some algorithms can have run times that are different orders of
magnitude for different sets of inputs of the same size.
• These algorithms can be evaluated for their best, worst, and average
cases.
• Algorithms that have different cases can typically be identified by the
inclusion of an event-controlled loop or a conditional statement.

15
Nasir Uddin
 CS211 - Data Structure and Algorithms

Complexity Analysis – Different cases

• In 𝑊𝑜𝑟𝑠𝑡 𝐶𝑎𝑠𝑒, the function must examine all steps.
• In 𝐵𝑒𝑠𝑡 𝐶𝑎𝑠𝑒, the function only examine the
minimum number of steps.
• In 𝐴𝑣𝑒𝑟𝑎𝑔𝑒 𝐶𝑎𝑠𝑒, the function examines the
average number of steps.

16
Nasir Uddin
 CS211 - Data Structure and Algorithms

Evaluating the Python List

• List Traversal
• 𝒪(𝑛)
• List Allocation
•𝒪 1
• 𝒪(𝑛)
• Appending to a List
• 𝐵𝑒𝑠𝑡 𝐶𝑎𝑠𝑒, 𝒪(1), when element is placed in index 0th position.
• 𝑊𝑜𝑟𝑠𝑡 𝐶𝑎𝑠𝑒, 𝑇 𝑛 = 1 + 1 + 𝑛, 𝒪(𝑛), when array is full and need to extend
the array, place all previous 𝑛 items, and place new item in the list.

17
Nasir Uddin
 CS211 - Data Structure and Algorithms

Complexity Analysis - Amortized Cost

• 𝐴𝑚𝑜𝑟𝑡𝑖𝑧𝑒𝑑 analysis is the process of computing the time-complexity
for a sequence of operations by computing the average cost over the
entire sequence.
• The cost per operation must be known and
• It must vary in which many of the operations in the sequence
contribute little cost and
• Only a few operations contribute a high cost to the overall time.
• Example, list append() operation.

18
Nasir Uddin
 CS211 - Data Structure and Algorithms

Complexity Analysis - Amortized Cost

• aggregate method
• it computes the total from the individual
operations.
• total storage cost 𝑠𝑖
• 16
• total expansion cost (𝑒𝑖 )
• 15
• the sum of the storage and expansion
costs, 𝑠𝑖 + 𝑒𝑖 , will never be more than
𝑇 𝑛 = 2𝑛.
2𝑛
• amortized cost 𝑇 𝑛 = , 𝒪(1)
𝑛
19
Nasir Uddin
 CS211 - Data Structure and Algorithms

Evaluating the Set ADT

20
Nasir Uddin
 CS211 - Data Structure and Algorithms

Evaluating the Set ADT

21
Nasir Uddin
 CS211 - Data Structure and Algorithms

Evaluating the Set ADT

22
Nasir Uddin
 CS211 - Data Structure and Algorithms

Application: The Sparse Matrix

• A matrix containing a large number of zero
elements is called a 𝑆𝑝𝑎𝑟𝑠𝑒 𝑀𝑎𝑡𝑟𝑖𝑥.
• A sparse matrix is formally defined to be an 𝑚 × 𝑛
matrix that contains 𝑘 non-zero elements such
that 𝑘 ≪ 𝑚 × 𝑛.
• The 2-D array data structure used to implement
the Matrix ADT.
• But when used to store huge sparse matrices,
large amounts of memory can be wasted, and the
operations can be inefficient since the zero
elements are also stored in the 2-D array.
23
Nasir Uddin
 CS211 - Data Structure and Algorithms

Application: The Sparse Matrix

• Is there a different structure or organization we
can use to store the elements of a sparse matrix
that does not waste space?
• One approach is to organize and store the non-
zero elements of the matrix within a single list
instead of a 2-D array.

24
Nasir Uddin
 CS211 - Data Structure and Algorithms

The Sparse Matrix - List-Based Implementation

25
Nasir Uddin
 CS211 - Data Structure and Algorithms

The Sparse Matrix - List-Based Implementation

26
Nasir Uddin
 CS211 - Data Structure and Algorithms

The Sparse Matrix - List-Based Implementation

27
Nasir Uddin
 CS211 - Data Structure and Algorithms

The Sparse Matrix - List-Based Implementation

28
Nasir Uddin
 CS211 - Data Structure and Algorithms

The Sparse Matrix - Efficiency Analysis

29
Nasir Uddin