Asymptotics &

Asymptotic Analysis -
Iterative

Md. Asif Bin Khaled

Ram Model

Machine-Independent algorithm design upon a hypothetical computer called the Random Access Machine(RAM).
Under this model of computation we are confronted with a computer where:
1. Each simple operation(+,*,-,=,if,call) takes exactly one step.
2. Loops and subroutines are not considered simple operations. Instead, they are the composition of many
single-step operations.
3. The data types in the model are integer and floating point.

2
RAM Model Continued.

4. Each memory access takes exactly one step. However, we have as much as memory as we need. The RAM
model takes no notice of whether an item is in cache or on the disk.
5. Runtime is measured by counting up the number of steps an algorithm takes on a given problem instance. For
example, multiplying or adding two numbers take only one step. However in reality multiplying is more
costly.
Despite being an impractical model, the RAM model helps us to analyze algorithms in a machine-independent way.

3
Best, Worst & Average Case Complexity

Using RAM model of computation, we can count how many steps our algorithm takes on any given input instance
by executing it however, to understand how good or bad an algorithm is in general, we must know how it works
over all instance. For example, in the problem of sorting, the set of possible input instances consists of all possible
arrangements of n keys, over all possible values of n.
The running time of an algorithm on a particular input is the number of primitive operations or steps executed.

4
Best, Worst & Average Case Complexity Continued.

We can represent each input instance as a point on a graph (for example the problem of sorting) where the x-axis
represents the size of the input problem and the y-axis denotes the number of steps taken by the algorithm in this
instance.

5
Best, Worst & Average Case Complexity Continued.

We can define three functions from all these points,

1. Worst Case Complexity: The worst case complexity of the algorithm is the function defined by the
maximum number of steps taken in any instance of size n.
2. Best Case Complexity: The best case complexity of the algorithm is the function defined by the minimum
number of steps taken in any instance of size n.
3. Average Case Complexity: The average case complexity of the algorithm, which is the function defined by
the average number of steps over all instances of size n.

6
Best, Worst & Average Case Complexity Continued.

The important thing is to realize that each of these time complexities define numerical function, representing time
vs problem size. These functions are as well defined as any other numerical function, be it y = x3 + 3 or y = 6541n3
+ 3213n2 + 56n + 231

However time complexities are such complicated function that we must simplify them to work with them. To
resolve this we will be introducing notations.

7
Best, Worst & Average Case Complexity Continued.

In the average case, we do not bound the worst case running time, but try to compute the average time spent over all
possible inputs of the argument size. This kind of analysis is generally harder, since it involves statistical arguments
and often requires properties about the distribution of inputs that may be difficult to justify.

The average case of an algorithm can be more useful because sometimes the worst case behavior of an algorithm is
misleadingly bad.

8
Loop Invariant

A loop invariant is a property of a program loop that is true before and after each iteration.
Max(A,n) {
// return the maximum value in a[0...n-1]
m = A[0];
for i=1 to n-1:
// m equals the maximum value in A[0...i-1]
if (m < A[i])
m = A[i];
// m equals the maximum value in A[0...i]
// m equals the maximum value in A[0...n-1]
return m;
}

9
Loop Invariant Continued

The loop invariant must be true

1. Before the loop starts.
2. Before each iteration of the loop.
3. After the loop terminates.

10
The Problem of Sorting

If we want to solve a problem then we would like to have the properly define the problem otherwise there may arise
a lot of ambiguity.
Input: Sequence <a0, a1, …, an-1> of numbers.

Output: Permutation <a'0, a'1, …, a'n-1> such that a'0 ≤ a'1 ≤ … ≤ a'n-1
For example,
Input: 12, 7, 88, 5, 46
Output: 5, 7, 12, 46, 88

11
Insertion Sort Pseudocode

Insertion_Sort(A):
for j=1 to A.length - 1:
key=A[j]
//insert A[j] into the sorted sequence A[0...j-1]
i=j-1
while i≥0 and A[i]>key:
A[i+1]=A[i]
i=i-1
A[i+1]=key

12
Simulation of Insertion Sort

A single element is always sorted.

12 7 88 5 46

13
Simulation of Insertion Sort

j=1
i=j-1
key=7

12 7 88 5 46

14
Simulation of Insertion Sort

j=1
i=0
key=7

12 7 88 5 46

15
Simulation of Insertion Sort

Comparing A[i]>key and if true shifting, A[i+1]=A[i]

j=1
i=0
key=7

12 12 88 5 46

16
Simulation of Insertion Sort

End of the inner loop

j=1
i=-1
key=7

12 12 88 5 46

17
Simulation of Insertion Sort

Assigning A[i+1]=key
j=1
i=-1
key=7

7 12 88 5 46

18
Simulation of Insertion Sort

j=2
i=j-1
key=88

7 12 88 5 46

19
Simulation of Insertion Sort

j=2
i=1
key=88

7 12 88 5 46

20
Simulation of Insertion Sort

Comparing A[i]>key and if true shifting, A[i+1]=A[i]

j=2
i=1
key=88

7 12 88 5 46

21
Simulation of Insertion Sort

End of the inner loop

j=2
i=1
key=88

7 12 88 5 46

22
Simulation of Insertion Sort

Assigning A[i+1]=key
j=2
i=1
key=88

7 12 88 5 46

23
Simulation of Insertion Sort

j=3
i=j-1
key=5

7 12 88 5 46

24
Simulation of Insertion Sort

j=3
i=2
key=5

7 12 88 5 46

25
Simulation of Insertion Sort

Comparing A[i]>key and if true shifting, A[i+1]=A[i]

j=3
i=2
key=5

7 12 88 88 46

26
Simulation of Insertion Sort

Comparing A[i]>key and if true shifting, A[i+1]=A[i]

j=3
i=1
key=5

7 12 12 88 46

27
Simulation of Insertion Sort

Comparing A[i]>key and if true shifting, A[i+1]=A[i]

j=3
i=0
key=5

7 7 12 88 46

28
Simulation of Insertion Sort

End of the inner loop

j=3
i=-1
key=5

7 7 12 88 46

29
Simulation of Insertion Sort

Assigning A[i+1]=key
j=3
i=-1
key=5

5 7 12 88 46

30
Simulation of Insertion Sort

j=4
i=j-1
key=46

5 7 12 88 46

31
Simulation of Insertion Sort

j=4
i=3
key=46

5 7 12 88 46

32
Simulation of Insertion Sort

Comparing A[i]>key and if true shifting, A[i+1]=A[i]

j=4
i=3
key=46

5 7 12 88 88

33
Simulation of Insertion Sort

End of the inner loop

j=4
i=2
key=46

5 7 12 88 88

34
Simulation of Insertion Sort

Assigning A[i+1]=key
j=4
i=2
key=46

5 7 12 46 88

35
Analysis of Insertion Sort

1. The running time of insertion sort depends on the input. An already sorted sequence is easier to sort.
2. We can parameterize the running time by the size of the input, since short sequences are easier to sort than
long ones.
3. Generally, we seek upper bounds on the running time, because everybody likes a guarantee.

36
Analysis of Insertion Sort Continued.

Let us assume for now that a constant amount of time is required to execute each line of our pseudocode. One line
may take a different amount of time than another, but we shall assume that each execution of the i th line takes ci,

where ci is a constant. This viewpoint is in keeping with the RAM model.

37
Analysis of Insertion Sort Continued.

38
Analysis of Insertion Sort (Best Case) Continued.

Best case occurs if the array is already sorted. We can express this running time as an + b for constants a and b that
depend on the statement costs c i. It is thus a linear function of n. [Θ(n), yet to be discussed.]

39
Analysis of Insertion Sort (Worst Case) Continued.

When the array is in reverse sorted order which means that the array is in decreasing order, is the worst case for this
algorithm. We can express this worst case running time as an2 + bn + c for constants a, b and c that again depend on
the statement costs ci. It is thus a quadratic function of n. [Θ(n2), yet to be discussed.]

40
Analysis of Insertion Sort (Worst Case) Continued.

41
Analysis of Insertion Sort (Average Case) Continued.

So, how can we express the following?

42
Machine-Independent Time

1. Relative speed (on the same machine): Whether an algorithm is better when comparing by running on the
same machine.
2. Absolute speed (on different machines): Whether an algorithm is better no matter what machine it is run on.
The main idea of the asymptotic analysis is to ignore the machine-dependent constants and look at growth of T(n)
as n → ∞ (n approaches infinity).
It is convenient to define the notion of steps with running time so that it is as machine-independent as possible.

43
Asymptotic Analysis

When analyzing the running time or space usage of programs, we usually try to estimate the time or space as
function of the input size. For example, when analyzing the worst case running time of a function that sorts a list of
numbers, we will be concerned with how long it takes as a function of the length of the input list.

In mathematical analysis, asymptotic analysis, also known as asymptotics, is a method of describing limiting
behavior.

For large inputs, the slower the asymptotic growth rate, the better the algorithm is.

44
Asymptotic Analysis Continued

The asymptotic behavior of a function f(n) (such as f(n)=c*n or f(n)=c*n 2, etc.) refers to the growth of f(n) as n gets
large.

Suppose that we are interested in the properties of a function f(n) as n becomes very large. If f(n) = n 2 + 3n, then as
n becomes very large, the term 3n becomes insignificant compared to n 2. The function f(n) is said to be
"asymptotically equivalent to n 2, as n → ∞". This is often written symbolically as f(n) ~ n 2, which is read as "f(n) is
asymptotic to n2".

45
Asymptotic Function Order

Function Comment

n0 or 1 Constant

lgn logarithmic

√n
Increasing
n linear

nlgn

nk Polynomial, k>1

an Exponential,

46
Asymptotic Notation

The notation we use to describe the asymptotic running time of an algorithm are defined in terms of functions
whose domains are the set of natural numbers N={0,1,2,....}. Such notations are convenient for describing the worst
case running time function T(n), which usually is defined only on integer input sizes.

47
Big O (Oh) Notation (Upper Bounds)

We write f(n) ∊ O(g(n)) if there exist constants c > 0, n0 > 0

such that,

0 ≤ f(n) ≤ c*g(n)
for all n ≥ n0.

Set Definition,
O(g(n)) = {f(n): there exist constants c > 0, n0 > 0 such that

0 ≤ f(n) ≤ c*g(n) for all n ≥ n0}

48
Big O (Oh) Notation (Upper Bounds) Continued

49
Big O (Oh) Notation (Upper Bounds) Continued

When we say that an algorithm runs in time T(n), we mean that T(n) is an upper bound on the running time that
holds for all inputs of size n. This is called worst-case analysis. The algorithm may very well take less time on some
inputs of size n, but it doesn't matter.

If an algorithm takes T(n)=c*n 2+k steps on only a single input of each size n and only n steps on the rest, we still
say that it is a quadratic algorithm.

50
Big O (Oh) Notation (Upper Bounds) Continued.

We use O-notation to give an upper bound on a function, within a constant factor. We write f(n) ∊ O(g(n)) to
indicate that a function f(n) is a member of the set O(g(n)). When we are writing f(n) ∊ O(g(n)), we are claiming
that some constant multiple of g(n) is an asymptotic upper bound on f(n), with no claim about how tight an upper
bound it is.

For example, we have 2n2 ∊ O(n2), but we also have 2n2 ∈ O(n3).

51
Big O (Oh) Notation (Upper Bounds) Proof

1. 2n2 ∊ O(n3) 2. 10n3 ∊ O(n3)

here, here,
f(n) = n2 , g(n) = n3 f(n) = 10n3 , g(n) = n3
If n ≥ 1, then hence,
2n2 ≤ 2n2(n) = 2n3 10n3 ≤ 10n3
So we can choose So we can choose
c = 2 & n0 = 1 c = 10 & n0= 1

52
Big O (Oh) Notation (Upper Bounds) Proof
Continued.

3. 3n2 + 2n - 4 ∈ O(n2) 4. 10n4 ∈ O(n3)

here, here,
f(n) = 3n2 + 2n - 4 , g(n) = n2 f(n)=10n4 , g(n)=n3
If n ≥ 1, then hence,
3n2 + 2n - 4 ≤ 3n2 + 2n(n) = 5n2 10n4 ≤ c*n3
So we can choose 10n4/n3 ≤ c*n3/n3
c = 5 & n0 = 1 10n ≤ c
As c can not ever outrun n hence, the
inequality does not stand.

53
Big Ω (Omega) Notation (Lower Bounds)

We write f(n) ∈ Ω(g(n)) if there exist constants c > 0, n0 > 0

such that,

0 ≤ c*g(n) ≤ f(n)
for all n ≥ n0.

Set Definition,
Ω(g(n)) = {f(n): there exist constants c > 0, n0 > 0 such that

0 ≤ c*g(n) ≤ f(n) for all n ≥ n0}

54
Big Ω (Omega) Notation (Lower Bounds) Continued

55
Big Ω (Omega) Notation (Lower Bounds) Proof

1. 2n2 + 7 ∈ Ω(n) 2. 4n2 – 5n ∈ Ω(n2)

here, here,
f(n) = 2n2 + 7, g(n) = n f(n) = 4n2 – 5n , g(n) = n2
If n ≥ 1, then If n ≥ 5, then
2n2 + 7 ≥ 2n2 ≥ 2n2/n ≥ 2n 4n2 – 5n ≥ 4n2 – 5n(n/5)
So we can choose ≥ 4n2 - n2 = 3n2
c = 2 & n 0= 1 So we can choose
c = 3 & n 0= 5

56
Big Θ (theta) Notation (Tight Bounds)

We write f(n) ∈ Θ(g(n)) if there exist constants c1, > 0, c2, > 0, n0 > 0
such that,
0 ≤ c1*g(n) ≤ f(n) ≤ c2*g(n)

for all n ≥ n0.

Set Definition,
0(g(n)) = {f(n): there exist constants c1 > 0, c2 > 0, n0 > 0 such that

0 ≤ c1*g(n) ≤ f(n) ≤ c2*g(n) for all n ≥ n0}

57
Big Θ (theta) Notation (Tight Bounds) Continued

58
Big Θ (theta) Notation (Tight Bounds) Continued

c1*g(n) ≤ f(n) is denoted by Ω(g(n))

f(n) ≤ c2*g(n) is denoted by O(g(n))

Whereas
c1*g(n) ≤ f(n) ≤ c2*g(n) is denoted by Θ(g(n))
Therefore,
Θ(g(n)) = O(g(n)) ∩ Ω(g(n))

59
Asymptotic Notation Summary

1. O(f(n)): The set of functions that grows no faster than f(n), asymptotic upper-bound on growth rate
2. Ω(f(n)): The set of functions that grows no slower than f(n), asymptotic lower-bound on growth rate
3. Θ(f(n)): The set of functions that grows no faster and no slower than f(n), asymptotic tight-bound on
growth rate

60
Complexity Examples

1. int a,b,c; 2. int a,b,c;

for(int i=0; i<n; i++){ for(int i=0; i<n; i+=2){
a=b; a=b;
b=c; b=c;
} }

Complexity: O(n) Complexity: O(n)

61
Complexity Examples Continued

3. int a,b,c; 4. int a,b,c;

for(int i=1; i<n; i*=2){ for(int i=1; i<n; i*=3){
a=b; a=b;
b=c; b=c;
} }

Complexity: O(log2n) Complexity: O(log3n)

62
Complexity Examples Continued

5. int a,b,c; 6. int a,b,c;

for(int i=n; i>=0; i/=2){ for(int i=n; i>=0; i/=3){
a=b; a=b;
b=c; b=c;
} }

Complexity: O(log2n) Complexity: O(log3n)

63
Complexity Examples Continued

7. int i=0,s=0; 8. int a,b,c;

while(s<=n){ for(int i=n; i>=0; i/=3){
i++; a=b;
s=s+i; b=c;
} }
for(int i=n; i>=0; i/=2){
a=b;
b=c;
}
Complexity: O(√n) Complexity: O(log3n + log2n) -> O(log2n)

64
Complexity Examples Continued

9. int a,b,c; 10. int a,b,c;

for(int i=0; i<m; i++){ for(int i=0; i<p; i++){
for(int j=0; i<n; j++){ for(int j=0; j<q; j++){
a=b; a=b;
b=c; }
} for(int j=0; j<r; j++){
} a=b;
}
}
Complexity: O(mn) Complexity: O(p(q+r))

65
Complexity Examples Continued

11. int a = 0; 12. int a = 0;

for (i = 0; i < n; i++) { while (n > 0) {
for (j = n; j > i; j--) { a += n;
a = a + i + j; n /= 2;
} }
}

Complexity: O(n2) Complexity: O(log2n)

66
Shortcomings of Asymptotic Analysis

In practice, other considerations beside asymptotic analysis are important when choosing between algorithms. Let
algorithm A be asymptotically better than algorithm B,

1. Implementation Complexity: Algorithms with better complexity are often (much) more complicated. This
can increase coding time and the constants.
2. Small Input Sizes Asymptotic analysis ignores small input sizes. At small input sizes, constant factors or low
order terms could dominate running time, causing B to outperform A.

67
Shortcomings of Asymptotic Analysis Continued

3. Worst Case versus Average Case Performance: If A has better worst case performance than B, but the
average performance of B given the expected input is better, then B could be a better choice than A.
Conversely, if the worst case performance of B is unacceptable (say for life-threatening or mission-critical
reasons), A must still be used.

68
Reference

1. https://www.cs.cornell.edu/courses/cs312/2004fa/lectures/lecture16.htm
2. https://www.khanacademy.org/computing/computer-science/algorithms/asymptotic-notation/a/asymptotic-not
ation

69