RMIT Classification: Trusted

Algorithms Analysis and

Design
 RMIT Classification: Trusted

Learning objectives
1. Understand why it is important to be able to compare the
complexity of algorithms
2. Measure the complexity of algorithms
3. Analyse the performance of algorithms
4. Be able to perform empirical analyses of algorithms
5. Develop new algorithms

2
 RMIT Classification: Trusted

Example: Which is Faster?

• Scenario: You work for a large medical company. Your
boss comes and asks your opinion on which of two
approaches is faster for employees to search over an
unordered set of medicines.
• Question: Which of the following two approaches will you
tell your boss is faster?
1. Search one by one
2. Sort then search

3
 RMIT Classification: Trusted

Array Search
Items Average number of tries
• 10 5
• 100 50
• 1000 500
• 10,000 5000
• 100,000 50,000
• 1,000,000 500,000
• 10,000,000 5,000,000 For a non-sorted array of size N, the number of
tries to find a value is correlated to N

4
 RMIT Classification: Trusted

Binary Search
Items Tries
• 10 4
• 100 7
• 1000 10
• 10,000 14
• 100,000 17
• 1,000,000 20
• 10,000,000 24 For a sorted array of size N, the
number of tries to find a value is
correlated to log2N
5
 RMIT Classification: Trusted

Sorting
• This is a very interesting topic that we will cover extensively
in class as many problems require using or generating
sorted data
• But to answer the question on the medical company we
need to be able to determine how fast sorting can be done

6
 RMIT Classification: Trusted

Which is Faster?
• Assume that sorting takes 10 n*log(n) msec, linear search takes 2*n
msec and binary search takes 2*log(n) msec
• If you do the search 10 times on 106 which approach is best?
1. 10*2*106 = 20*106 msec = 20,000 sec
2. Sorting: 10 * 10 6 * log 106 ≈ 10*106 *13.8 msec ≈ 138,000 sec +
Searching: 10*2*log 10 6 = 0.276 sec. Total ≈ 138,000 sec

• If you do the search 1000 times and the data size is 106 then
1. 1000*2*106 = 2000*106 msec = 2,000,000 sec
2. Sorting: 10 * 10 6 * log 106 ≈ 10*106 *13.8 msec ≈ 138,000 secs +
Searching: 1000*2*log 10 6 = 27.6 seconds. Total ≈ 138,027 seconds

7
 RMIT Classification: Trusted

Program Performance
• Assume that we can find a formula that, for a
specific machine, determines how long a
program takes to run on various input (N) sizes
• When do programs A and B become impractical?
o If n = 10 then A takes 20 and B takes 50 secs

o If n = 25 they both take 125 secs.

o If n = 103 then A takes 200,000 secs and B takes 5,000 secs.
o If n = 106 then A takes 2x10 11 seconds and B take 5x106 seconds

• Algorithmic performance is critical when analyzing large data sets

8
 RMIT Classification: Trusted

Finding Primes (*)

How do you check if a number, n, is prime
• Checking if each number from 2 to n-1 divides n
• Running the loop up to n/2
• Use the fact that for p > 3, p can be written as 6n ± 1
o So if the number is not of this form, it cannot be prime
• Running the loop up to the square root of n
• CAN WE DO BETTER?

9
 RMIT Classification: Trusted

Improve Primality Test even more (*)

• Use the fact that for p > 3 , p2 - 1 is a multiple of 24
o p2 - 1 = (p - 1) x (p + 1)
o As p must be odd, both p - 1 and p + 1 are even. Moreover, since p - 1 and
p + 1 are 2 consecutive even numbers - one of them must be a multiple of
4. Therefore p is divisible by 8
o As p - 1, p, p + 1 form three consecutive numbers - one of them is a
multiple of 3. As p is prime, then either p - 1 or p + 1 is a multiple of 3 and
therefore is divisible by 3
o Hence p 2 - 1 is divisible by 24 (8*3)
• Can Wilson’s theorem help – (n-1)! + 1 is divisible by n iff n is prime?
10
 RMIT Classification: Trusted

Comparing Performance
• Different algorithms that solve the same problem can have very
different performance
o How do we compare the performance of two programs that solve the same
problem?
o Should we just run them and compare their time requirements?
• Need address issues such as - what computer to use, what data to use
and how much space is needed?
• An inefficient algorithm on a small data set (i.e., one that grows
quadratically) would be unfeasible to run on a large data set

11
 RMIT Classification: Trusted

What to measure?
• In this lecture, we look at the ways of estimating the
running time of a program and how to compare the running
times of two programs without ever implementing them
• It is vital to analyse the resource use of an algorithm, well
before it is implemented and deployed
• Space is also important but we focus on time in this course

12
 RMIT Classification: Trusted

Time Efficiency
 RMIT Classification: Trusted

How to Estimate Running Time?

• We employ mathematical techniques that analyze
algorithms independently of specific implementations,
computers or data
• Approach:
o First, we start to count the number of significant operations in a
particular solution to assess its efficiency
o Then, we will express the efficiency of algorithms using growth
functions

14
 RMIT Classification: Trusted

Theoretical Analysis of Time Efficiency

• Idea: An algorithm consists of some operations executed a
number of times.
• Hence, an estimate of the running time/time efficiency of an
algorithm can be obtained from
o Determining these operations
o How long to execute them, and
o The number of times they are executed.
• These operations are called basic operations and the number
of times is based on the input size of the problem

15
 RMIT Classification: Trusted

Basic operation
Operation(s) that contribute most towards the total running
time
• Examples:
o Compare ( i != j )
o Add ( i + j )
o Multiply ( i * j )
o Divide ( i / j )
o Assignment ( i = j )
16
 RMIT Classification: Trusted

Execution Time of Algorithms - 1

• Each operation in an algorithm (or a program) has a cost
o count = count + 1; takes a certain amount of time, but it is
constant
• Therefore for a sequence of operations:
count = count + 1 Cost: c1
sum = sum + count Cost: c2
Total Cost = c1 + c2

17
 RMIT Classification: Trusted

Execution Time of Algorithms - 2

Example: Simple If-Statement
Cost Occurrences
if (n < 0): c1 1
absval = -n c2 1
else:
absval = n c3 1

Total Cost <= c1 + max(c2,c3)

18
 RMIT Classification: Trusted

Execution Time of Algorithms - 3

Example: Simple Loop
Cost Occurrences
i=1 c1 1
sum = 0 c2 1
while (i <= n): c3 n+1
i=i+1 c4 n
sum = sum + i c5 n
Total Cost = c1 + c2 + (n+1)*c3 + n*c4 + n*c5

19
 RMIT Classification: Trusted

Execution Time of Algorithms - 4

Example: Nested Loop
Cost Occurrences
i=1 c1 1
sum = 0 c2 1
while (i <= n): c3 n+1
j=1 c4 n
while (j <= n): c5 n*(n+1)
sum = sum + i c6 n*n
j=j+1 c7 n*n
i = i +1 c8 n
Total Cost = c1 + c2 + (n+1)*c3 + n*c4 + n*(n+1)*c5+n*n*c6+n*n*c7+n*c8

20
 RMIT Classification: Trusted

General Rules for Estimation

• Consecutive Statements: Just add the running times of those consecutive
statements.
• If/Else: Never more than the running time of the test plus the larger of running
times of S1 and S2.
• Loops: The running time of a loop is at most the running time of the statements
inside of that loop times the number of iterations.
• Nested Loops: Running time of a nested loop containing a statement in the
inner most loop is the running time of statement multiplied by the product of the
sized of all loops.

21
 RMIT Classification: Trusted

Runtime Complexity
• Worst Case - Given an input of n items, what is the
maximum running time for any possible input?
• Best Case - Given an input of n items, what is the
minimum running time for any possible input?
• Average Case - Given an input of n items, what is the
average running time across all possible inputs?

NOTE: Average Case is not the average of the worst and best case.
Rather, it is the average performance across all possible inputs.
22
 RMIT Classification: Trusted

Runtime Complexity
• Sequential Search: search for the key by traversing
values in the array one-by-one
• Best-case:
o The best case input is when the item being searched for
is the first item in the list, so Cbest (n) = 1.
• Worst-case:
o The worst case input is when the item being searched
for is not present in the list, so Cworst (n) = n.
23
 RMIT Classification: Trusted

Runtime Complexity
• Average Case: What does average case mean?
o Recall: average across all possible inputs – how to
analyse this?
o Typically not straight forward

24
 RMIT Classification: Trusted

Runtime Complexity
Average Case Analysis: p is the probability of a successful
search.
If search is successful
Cavg(n) = (1 + 2 + … + n)/n = (n + 1)/2
(Assume the probability of finding the search value at each
element is the same)
If search is unsuccessful
Cavg(n) = n
25
 RMIT Classification: Trusted

Summary
• Input size, basic operation
• Time complexity estimate using input size and
basic operation
• Best, worst, and average cases

26
 RMIT Classification: Trusted

Asymptotic Complexity
 RMIT Classification: Trusted

Asymptotic Complexity
• Problem:
o We now have a way to analyse the running time (a.k.a.
time complexity) of an algorithm, but every algorithm
has their own time complexity.
• 𝑇1 = 𝑐1 ∙ 𝐶 𝑛1 , 𝑇2 = 𝑐2 ∙ 𝐶 𝑛2 , 𝑇3 = 𝑐3 ∙ 𝐶 𝑛3 …
o How to compare in a meaningful way?

28
 RMIT Classification: Trusted

Asymptotic Complexity
• Solution:
o Group them into equivalence classes (for
easier comparison and understanding), with
respect to the input size
o Focus of this part: asymptotic complexity and
equivalence classes

29
 RMIT Classification: Trusted

Asymptotic Complexity
Consider the running time
estimates of two algorithms:
• Algo 1: 𝑇1 𝑛 = 5.1𝑛
• Algo 2: 𝑇2 𝑛 = 5.2𝑛

Do they have similar timing

profiles as n grows?

30
 RMIT Classification: Trusted

Asymptotic Complexity
What about the
followings:
• Algo 3: 𝑇3 𝑛 = 5.1𝑛2
• Algo 4: 𝑇4 𝑛 = 5.2𝑛2

31
 RMIT Classification: Trusted

Asymptotic Complexity - Bounds

• Idea: Use bounds and asymptotic complexity

• In other words:

o As n becomes large, what is the dominant term

contributing to the running time 𝑡(𝑛)?

32
 RMIT Classification: Trusted

Asymptotic Complexity - upper bounds

Definition:
• Given a function 𝑡(𝑛) (i.e.,
the running time of an
algorithm):
• Let 𝒄 ∙ 𝒈(𝒏) be a function
that is an upper bound on
𝑡(𝑛) for some 𝑐 > 0 and for
“large” 𝑛.

33
 RMIT Classification: Trusted

Example: Upper Bounds

34
 RMIT Classification: Trusted

Asymptotic Complexity – O(n)

Rather than talking about individual upper bounds for each running
time function, we seek to group them into equivalence classes that
describe their order of growth.
Big-O notation: O(n)
Given a function 𝑡(𝑛),
• Formally: 𝑡(𝑛) ∈ O(𝑔(𝑛)), if 𝑔(𝑛) is a function and 𝑐 ∙ 𝑔(𝑛) is an
upper bound on 𝑡(𝑛) for some 𝑐 > 0 and for “large” 𝑛.
• Informally: 𝑡(𝑛) ∈ O(𝑔(𝑛)) means 𝑔(𝑛) is a function that, as 𝑛
increases, provides an upper bound for 𝑡(𝑛).

35
 RMIT Classification: Trusted

Asymptotic Complexity – O(n)

If 𝒕(𝒏) = 𝟓. 𝟏𝒏
• 𝑔(𝑛) = 𝑛
• 𝑔(𝑛) = 0.001𝑛 − 6
• 𝑔(𝑛) = 𝑛2

What 𝒈(𝒏) to use if 𝒕(𝒏) = 𝟓. 𝟐𝒏?

• Any of the above 𝑔(𝑛) functions are possible!

36
 RMIT Classification: Trusted

Asymptotic Complexity
If an algorithm requires n2–3*n+10 seconds to solve a problem size n. If
constants k and n0 exist such that
k*n2 > n2–3*n+10 for all n  n0
the algorithm is order n2 (In fact, k is 3 and n0 is 2)
3*n2 > n2–3*n+10 for all n  2
Thus, the algorithm requires no more than
k*n2 time units for n  n0 where k = 3 and no = 2
So, it is O(n2)

37
 RMIT Classification: Trusted

Common Equivalence Classes

O(1) Constant, (access array element)
O(log2n) Logarithmic (binary search)
O(n) Linear (finding maximum in an array)
O(nlog2n) Linearithmic (best sorting algorithms such as merge sort)
O(n2) Quadratic (simple sorting algorithms)
O(n3) Cubic (matrix multiplication)
NP Non-deterministic polynomial (travelling salesman, zero-sum subset)
O(2n) Exponential (generating all subsets, solve the tower of Hanoi
best known algorithms for travelling salesman and integer factorization)
O(N!) Factorial – generating all permutations of N elements

List sorted from best to worst. Many other options e.g. sqrt(n), log(log(n)), etc.

38
 RMIT Classification: Trusted

Bike Tour (*)

• Suppose you decide to ride a
bicycle around Ireland
o You will start in Dublin and your
goal is to visit Cork, Galway,
Limerick, and Belfast before
returning to Dublin
• What is the best itinerary?
o How can you minimize the
number of kilometers yet make
sure you visit all the cities?

39
 RMIT Classification: Trusted

Optimal Tour (*)

• If there are only 5 cities it’s not too hard to figure out the
optimal tour
o The shortest path is most likely a “loop”
o Any path that crosses over itself will be longer than a path that
travels in a big circle

40
 RMIT Classification: Trusted

Increasing the Number of Cities (*)

• As we add cities to our tour, however, it is much harder to
figure out the optimal tour
o A simple strategy of going to the
closest city does not always lead
to the shortest tour
• This example has 25 cities
o After visiting 4 cities, where
would you go next?
Going to this city does not
lead to the shortest tour...

41
 RMIT Classification: Trusted

Is Exhaustive Search Possible? (*)

• One option is to do an exhaustive search, but
o Number of possible tours of a map with n cities is (n − 1)! / 2
o n! is the product n × (n − 1) × (n − 2) ... × 2 × 1

• The number of tours grows incredibly quickly as we add cities map

#Cities #Tours
o The number of tours for 25 cities is 5 12

6 60
310,224,200,866,619,719,680,000
7 360

8 2,520

9 20,160

10 181,440

42
 RMIT Classification: Trusted

Traveling Salesman Problem (*)

• The traveling salesman problem is finding the shortest path to visit n points
o First posed by Irish mathematician W. R. Hamilton in the 19th century.
o Intensely studied in operations research and other areas since 1930
o Other variants include is there a tour that is less than X, what is the shortest tour

• One example of the famous P versus NP problems which are problems where
the best known solutions are really hard (exponential) to solve but are relative
easy (polynomial) to check
o Examples - subset sum, jigsaw solving
1 -3 7 2 11 -9 -6
14 5 11 -10 -1 22 -11

43
 RMIT Classification: Trusted

Real-Life Applications (*)

• It’s not likely anyone would want to plan a bike trip to 25 cities
• But the solution of many important real-world problems is the same as
finding a tour of a large number (n>1000) of cities
o Transportation: school bus routes, service calls
o Manufacturing: industrial robot drilling holes in a
printed circuit, VLSI (microchip) layout
o Planning a telecom network
• https://www.quantamagazine.org/computer-scientists-break-traveling-
salesperson-record-20201008/

44
 RMIT Classification: Trusted

Is P = NP? (*)
• Open Question: if one can check an answer in polynomial time, is there
an algorithm to find the answer in polynomial time
• Examples : sudoku, Jigsaws, subset-sum, graph colouring, map
colouring, travelling salesman, Hamiltonian path

45
 RMIT Classification: Trusted

Common Complexity Bounds

46
 RMIT Classification: Trusted

Common Complexity Bounds

Recall: We want to find equivalence function class that
upper bounds different 𝑡(𝑛).
But we might be given an upper bound 𝑔(𝑛) that isn’t quite in
the form of the equivalence classes.
• In the previous slides, we mentioned multiple upper bounds, e.g., 𝑛,
0.001𝑛 − 6, 𝑛2
How to get the equivalence classes?

47
 RMIT Classification: Trusted

Growth-Rate Functions
• Ignore low-order terms in the growth-rate function
o If an algorithm is O(n3+4n2+3n+5), it is also O(n 3)
• Ignore a multiplicative constant in the higher-order term
o If an algorithm is O(5n 3), it is also O(n3)

• Combine two modules with growth rates O(f(n)) and O(g(n)) is

O(f(n)+g(n))
o If an algorithm is O(n 3) + O(4n), it is also O(n 3 +4n), so, it is O(n3)
• We can also multiply two growth rates e.g. there is a loop inside a loop
o So a O(n2) loop inside O(n) loop is O(n 3)
48
 RMIT Classification: Trusted

Common Complexity Bounds

Let 𝑔(𝑛) = 2𝑛4 + 43𝑛3 − 𝑛 + 50
be an upper bound for the running time 𝑡(𝑛) of an algorithm.

What equivalence class should you use to describe the

order of growth of the algorithm?

Answer: O(n4)

49
 RMIT Classification: Trusted

Analysis of Algorithms
 RMIT Classification: Trusted

Time Efficiency of Algorithms

Typically, we are given the pseudo code of an
algorithm, not a nice 𝑡(𝑛) function.

So how do we determine bounds on

the order of growth of an algorithm?

51
 RMIT Classification: Trusted

Example: Sequential Search

Basic operations comparison, addition, assignment
Input size n

52
 RMIT Classification: Trusted

Example: an

Input size: n

Basic operation: multiplication

C(n):

=n

53
 RMIT Classification: Trusted

Example: Adding Matrices

Given two (square) matrices A and B, both of dimensions n by n, the
following algorithm computes C = A + B.

for (int i = 0; i <= n−1; i++) { Input size: n

for (int j = 0; j <= n−1; j++) {
C[i, j] = A[i, j] + B[i, j]; Basic op.: addition
}
} C(n):

= n2
54
 RMIT Classification: Trusted

Growth-Rate Functions – Ex 1
Cost Occurrences
i=1 c1 1
sum = 0 c2 1
while (i <= n): c3 n+1
i=i+1 c4 n
sum = sum + i c5 n

T(n) = c1 + c2 + (n+1)*c3 + n*c4 + n*c5 = (c3+c4+c5)*n + (c1+c2+c3) = a*n + b

➔ So, the growth-rate function for this algorithm is O(n)

55
 RMIT Classification: Trusted

Growth-Rate Functions – Ex 2
Cost Occurrences
i=1 c1 1
sum = 0 c2 1
while (i <= n): c3 n+1
j=1 c4 n
while (j <= n): c5 n*(n+1)
sum = sum + i c6 n*n
j=j+1 c7 n*n
i = i +1 c8 n
T(n) = c1 + c2 + (n+1)*c3 + n*c4 + n*(n+1)*c5+n*n*c6+n*n*c7+n*c8
= (c5+c6+c7)*n 2 + (c3+c4+c5+c8)*n + (c1+c2+c3) = a*n2 + b*n + c
➔ So, the growth-rate function for this algorithm is O(n2)
56
 RMIT Classification: Trusted

Recursion
• Recursion is fundamental tool in computer science.
o A recursive program (or function) is one that calls itself.
o It must have a termination condition defined.
• Many interesting algorithms are simply expressed with a
recursive approach

57
 RMIT Classification: Trusted

Recursive Example: Factorial

Input size: n

Basic operation: multiplication

58
 RMIT Classification: Trusted

Recursive Example: Factorial

Recurrence relation and conditions:
The number of multiplications is
𝐶(𝑛) = 𝐶(𝑛 − 1) + 1 for 𝑛 > 1, and 𝐶(1) = 0
• +1 is the number of multiplication operations at
each recursive step.
• When 𝑛 = 1, we have our termination/base
case, where we stop the recursion. When we
reach this base case, the number of
multiplications is 0, hence 𝐶(1) = 0.

59
 RMIT Classification: Trusted

Backward Substitution
Recurrence: 𝐶(𝑛) = 𝐶(𝑛 − 1) + 1 for 𝑛 > 1, and 𝐶(1) = 0

Aim of simplification and backward substitution:

Convert 𝐶(𝑛) = 𝐶(𝑛 − 1) + 1 to 𝐶(𝑛) = 𝑓𝑢𝑛𝑐𝑡𝑖𝑜𝑛(𝑛)
For example, 𝐶(𝑛) = 𝑛 + 1

60
 RMIT Classification: Trusted

Backward Substitution
Aim of simplification and backward substitution: Convert
𝐶(𝑛) = 𝐶(𝑛 − 1) + 1 to 𝐶(𝑛) = 𝑓𝑢𝑛𝑐𝑡𝑖𝑜𝑛(𝑛), e.g., 𝐶(𝑛) = 𝑛 + 1

1. Start with the recurrence relation: 𝐶(𝑛) = 𝐶(𝑛 − 1)+.. .

2. Substitute 𝐶(𝑛 − 1) with its RHS (𝐶(𝑛 − 1) = 𝐶(𝑛 − 2)+. . . ) in original
equation 𝐶(𝑛) = 𝐶(𝑛 − 2)+ .. .
3. Repeat Step 2
4. Spot the pattern of 𝐶(𝑛) and introduce a variable to express this pattern.
5. Determine when the value of this variable that relates 𝐶(𝑛) = 𝐶(1)+. . .
6. Substitute the value of 𝐶(1) and get 𝐶(𝑛) in terms of some expression of 𝑛

61
 RMIT Classification: Trusted

Backward Substitution
Recurrence: 𝑪(𝒏) = 𝑪(𝒏 − 𝟏) + 𝟏 for 𝒏 > 𝟏, and 𝑪(𝟏) = 𝟎
1. 𝐶(𝑛) = 𝐶(𝑛 − 1) + 1
2. Substitute 𝐶(𝑛 − 1) = 𝐶(𝑛 − 2) + 1 into original equation
3. 𝐶(𝑛) = [𝐶(𝑛 − 2) + 1] + 1 = 𝐶(𝑛 − 2) + 2
4. Substitute C(n − 2) = C(n − 3) + 1 into original equation
5. 𝐶(𝑛) = [𝐶(𝑛 − 3) + 1] + 2 = 𝐶(𝑛 − 3) + 3
6. We see the pattern 𝑪(𝒏) = 𝑪(𝒏 − 𝒊) + 𝒊 emerge, where 1 ≤ 𝑖 ≤ 𝑛
7. Now, we know 𝐶(1) = 0 and want to determine when 𝐶(𝑛 − 𝑖) = 𝐶(1), or when
𝑛 − 𝑖 = 1. This value is 𝑖 = 𝑛 − 1
𝐶(𝑛) = 𝐶(𝑛 − 𝑖) + 𝑖 = 𝐶(𝑛 − (𝑛 − 1)) + 𝑛 − 1 = 𝐶(1) + 𝑛 − 1 = 0 + 𝑛 − 1 = 𝑛 − 1
Hence 𝒕(𝒏) = 𝒄𝒐𝒑 ∙ 𝒏 ∈ 𝑶(𝒏)
62
 RMIT Classification: Trusted

Empirical Analysis
• Theoretical analysis of the complexity of an algorithm gives
an estimate of the running time and growth rate, but not the
real time
• Measuring the actual time of an implementation takes is in
the real world is very important, especially when comparing
two algorithms with the same time complexity

63
 RMIT Classification: Trusted

Estimating Execution Time

• With the basic operation and input size, to estimate the
actual running time of an algorithm we apply the following:

64
 RMIT Classification: Trusted

Estimating Execution Time

Example 1: Given sample N and Time values, we can generate a formula for
calculating the performance as follow:
o N = 10, T = 63.9 (≈ 64)
o N = 15, T = 88
o N = 20, T = 113
As the growth appears to be linear, we can write the equations:
64 = 10 * x + c (1) and 88 = 15 * x + c (2)
By subtracting the above equations, we get, 5x = 24 => x = 4.8
By substituting x in (1), we get 48 + c = 64 => c = 16
So, the approximations formula is T = 4.8 * N + 16
We can check this by substituting N = 20 => 4.8 * 20 + 16 = 112 (≈113)

65
 RMIT Classification: Trusted

Estimating Execution Time - 2

Example 2
• N = 5, T = 46.9 (≈ 47)
• N = 10, T = 143.1 (≈ 143)
• N = 20, T = 483
As the growth appears to be quadratic, we use ax2 + bx + c
For N = 5, T = 47, the equation is 25a + 5b + c = 46 (1)
For N = 10, T = 143, the equation is 100a + 10b + c = 141 (2)
For N = 20, T = 483, the equation is 400a + 20b + c = 481 (3)
To solve, multiply (2) by 4 and subtract (3) from it
400a + 40b + 4c = 572 and 400a + 20b + c = 483

66
 RMIT Classification: Trusted

Estimating Execution Time - 3

This leads to 20b + 3c = 89 (4)
The multiply (1) by 4 and subtract (2) from it
100a + 20b + 4c = 188
100a + 10b + c = 143
This leads to 10b + 3c = 45 (5)
Now subtracting (5) from (4) leads to 10b = 44 => b = 4.4
Substituting the value of b in (5) => 44 + 3c = 45 => c = 0.33
Substituting value of b and c in (1) => 100a + 44 + 0.33 = 143
=> a = 0.99 ≈ 1
So the approximation formula is T = n2 + 4.4 n + 0.33

67
 RMIT Classification: Trusted

Growth Rate Functions

Remember
If an algorithm takes 1 second to run with the
problem size 8, approximately how long would it take if doubling the input size
for that algorithm with the problem size 16? takes
• 2 times longer, its O(n)
• O(1) T(n) = 1 second
• 4 times longer, its O(n2)
• O(log2n) T(n) = 1*log 216 / log 28 = 4/3 seconds
• 8 times longer, is O(n3)
• O(n) T(n) = 1*16 / 8 = 2 seconds
• O(n*log2n) T(n) = 1*16*log 216 / 8*log 28 = 8/3 secs if tripling the input size
• O(n2) T(n) = 1*162 / 82 = 4 seconds takes
• O(n3) T(n) = 1*163 / 83 = 8 seconds • 3 times longer, its O(n)
• O(2n) T(n) = 1*216 / 28 = 28 seconds = 256 secs • 9 times longer, its O(n2)
• 27 times longer, is O(n3)
68
 RMIT Classification: Trusted

Estimating Performance
• We can estimate the growth rate if we collect performance data on a number of
inputs of different sizes
• We can then apply it to other input sizes

Input 1 Time 1 Input 2 Time 2 Big O Input 3 Time 3

100 5 200 10 O(n) 1000 50
10 5 20 20 O(n2) 1000 50,000
10 10 30 90 O(n2) 10000 107
50 6 100 48 O(n3) 5000 6*106
50 5 100 5 O(1) 10000 5
100 10 300 270 O(n3) 1000000 1013

69
 RMIT Classification: Trusted

Benchmarking Algorithms
Theoretical: Merge-sort and Quick-sort have O(n log(n))
complexity, while Selection-sort is O(n2).
Empirical: Running Times (in seconds) for different sorting
algorithms on a randomised list:
Input (list size) 500 2,500 10,000
Merge sort 0.8 8.1 39.8
Quick sort 0.3 1.3 5.3
Selection sort 1.5 35.0 534.7

70
 RMIT Classification: Trusted

Benchmarking Algorithms - 2
Theoretical: Merge-sort and Quick-sort have O(n log(n))
complexity, while Selection-sort is O(n2).
Empirical: Running Times (in seconds) for different sorting
algorithms on an ordered list:

N = 2,500 Random In Order In Reverse Order

Merge sort 8.1 7.8 7.9
Quick sort 1.3 35.1 37.1
Selection sort 35.0 34.4 35.3

71
 RMIT Classification: Trusted

Question
One of your data sources to predict the weather are images
sent from satellites. When you receive this data, you find that
for 5, 10 and 20 images your code took 24 msec, 99 msec
and 401 msec respectively.
o What is order of complexity of your algorithm and justify your
answer
o Your final input file has 1,000,000 images. Without using a
calculator, can you determine approximately how many days it will
take your code to run?

72
 RMIT Classification: Trusted

Best Practices
• If the problem size is always small, we can probably ignore the
algorithm’s efficiency and should choose the simplest algorithm
• Use adaptive algorithms that work differently depending on
o Input size
o Input characteristics (e.g. random or partially sorted, range)
• We should compare both the style and efficiency of each algorithm
o No need for coding tricks if the gain is small
o Should prefer easily understandable and maintainable program
• Implementation complexity: easy or complex to implement an algorithm

73
 RMIT Classification: Trusted

Theory vs. Practice

• Formal Analysis
o Pros: Independent of implementation / hardware details
o Cons: Hides constant factors. Only feasible on simple examples
• Empirical Analysis:
o Pro: Measure and model the performance on working code. Can be
used on complex examples
o Con: Implementation specific. May be running on the “wrong”
inputs.

74
 RMIT Classification: Trusted

Ways to Improve Performance

• Reduce the Order of Magnitude of the algorithms
• Reduce the coefficients
• Change data structure
o Array-based retrieve is O(1) vs. a linked-list retrieve operation is O(n). However,
insert and delete operations are much faster in a linked-list than in an array

• Weigh trade-offs between an

o Time versus memory

o Time versus accuracy

75
 RMIT Classification: Trusted

Other Approaches
• Heuristic algorithms
o Do not guarantee that the best will be found but usually will quickly find a
solution close to the best one.
o Sometimes these algorithms actually find the best solution, but the
algorithm is still called heuristic until this solution is proven to be the best.
• In general if X is the found solution and Y is the optimal solution, then
o Deterministic algorithm: X = Y
o Approximation Algorithm : X is close to Y
o Probabilistic algorithm - X is "usually" Y
o Heuristic algorithm – X is usually "close" to Y

76
 RMIT Classification: Trusted

Other Topics (*)

 RMIT Classification: Trusted

Other Types of Bounds (*)

• So we talked about Big O notation for upper
bounds…
• What about other types of bounds?

78
 RMIT Classification: Trusted

Lower Bound – Ω(n) (*)

Definition: given a function 𝑡(𝑛),

• Formally: 𝑡(𝑛) ∈ Ω(𝑔(𝑛)), if 𝑔(𝑛) is a function and 𝑐 ∙

𝑔(𝑛) is a lower bound on 𝑡(𝑛) for some 𝑐 > 0 and for
“large” 𝑛.

• Informally: 𝑡(𝑛) ∈ Ω(𝑔(𝑛)) means 𝑔(𝑛) is a function

that, as 𝑛 increases, is a lower bound of 𝑡(𝑛)

79
 RMIT Classification: Trusted

Lower Bound – Ω(n) (*)

80
 RMIT Classification: Trusted

Exact Bounds – Θ(n) (*)

Exact Bound Definition: Given a function 𝑡(𝑛),

• Formally: 𝑡(𝑛) ∈ Θ(𝑔(𝑛)), if 𝑔(𝑛) is a function and 𝑐1 ∙ 𝑔(𝑛)

is an upper bound on 𝑡(𝑛) and 𝑐2 ∙ 𝑔(𝑛) is an lower bound
on 𝑡(𝑛), for some 𝑐1 > 0 and 𝑐2 > 0 and for “large” 𝑛

• Informally: 𝑡(𝑛) ∈ Θ(𝑔(𝑛)) means 𝑔(𝑛) is a function that,

as 𝑛 increases, is both an upper and a lower bound of 𝑡(𝑛)

81
 RMIT Classification: Trusted

Exact Bounds – Θ(n) (*)

82
 RMIT Classification: Trusted

Examples (*)
Upper bound Lower bound
t (n) O(n) O(n2) O(n3) Ω(n) Ω(n2) Ω(n3)
log2 n T T T F F F
10n + 5 T T T T F F
n(n − 1)/2 F T T T T F
(n + 1)3 F F T T T T
2n F F F T T T

83
 RMIT Classification: Trusted

Some Clarifications…
• Generally O(n) is most commonly used…
• But exact bounds Θ(n) tell us the bounds are tight and the
algorithm doesn’t have anything outside what we expect
• Lower bounds Ω(n) are useful to describe the (theoretical)
limits of whole classes of algorithms, and also sometimes
useful to state how fast can the best case reach.

84
 RMIT Classification: Trusted

Some clarifications…
• O(n) is not the same thing as “Worst Case Efficiency”
• Ω(n) is not the same thing as “Best Case Efficiency”
• Θ(n) is not the same thing as “Average Case Efficiency”

However, it is perfectly reasonable to want the Ω(n) and O(n)

worst case efficiency bounds for a class of algorithms.

85
 RMIT Classification: Trusted

Copyright © 2017 RMIT University Vietnam