Nhân bản – Phụng sự – Khai phóng

Algorithm Analysis
Data Structures & Algorithms
 CONTENTS

• Introduction to algorithm

• Algorithm analysis

• Estimating running time

• Algorithm growth rates (Big O, Omega, Theta)

• Worst-Case, Best-Case, Average-Case

Data Structures & Algorithms 2

 CONTENTS

• Introduction to algorithm
• Algorithm analysis

• Estimating running time

• Algorithm growth rates (Big O, Omega, Theta)

• Worst-Case, Best-Case, Average-Case

Data Structures & Algorithms 3

 Introduction

• An algorithm is a sequence of instructions to be followed to solve a

problem
• There are often many solutions/algorithms to solve a given problem
• An algorithm can be implemented using different programming
languages on different platforms

• An algorithm must be correct. It should correctly solve the problem

• Once we have a correct algorithm for a problem, we have to determine

the efficiency of that algorithm

Data Structures & Algorithms 4

 …Introduction

• Program = Data structures + Algorithms

Program
Input = Output
Data structures + Algorithms

• Correctness
• An algorithm is said to be correct if for every input instance, it halts
with the correct output.
• Efficiency
• Computing time and memory space are two important resources.

Data Structures & Algorithms 5

 …Introduction

• Time
• Instructions take time
• How fast does the algorithm perform?
• What affects its running time?
• Space
• Data structures take space
• What kind of data structures can be used?
• How does choice of data structure affect the running time?
ð Focusing on running time
• How to estimate the time required for an algorithm?
• How to reduce the required time?

Data Structures & Algorithms 6

 CONTENTS

• Introduction to algorithm

• Algorithm analysis
• Estimating running time

• Algorithm growth rates (Big O, Omega, Theta)

• Worst-Case, Best-Case, Average-Case

Data Structures & Algorithms 7

 Algorithm analysis

• Why do we need algorithm analysis?

• Showing the algorithm is correct

• Writing a working program is not good enough

• The program may be inefficient

• If the program is run on a large data set, then the running time
becomes an issue

Data Structures & Algorithms 8

 …Algorithm analysis

• Example: Selection Problem (1/3)

• Given a list of N numbers, determine the kth largest, where k £ N.

• Algorithm 1
(1) Read N numbers into an array
(2) Sort the array in decreasing order by some simple algorithm
(3) Return the element in position k

Data Structures & Algorithms 9

 …Algorithm analysis

• Example: Selection Problem (2/3)

• Algorithm 2
(1) Read the first k elements into an array and sort them in
decreasing order
(2) Each remaining element is read one by one
–If smaller than the kth element, then it is ignored
–Otherwise, it is placed in its correct position in the array,
getting one element out of the array
(3) The element in the kth position is returned as the answer

Data Structures & Algorithms 10

 …Algorithm analysis

• Example: Selection Problem (3/3)

• Which algorithm is better when

1. N = 100 and k = 100?

2. N = 100 and k = 1?

Data Structures & Algorithms 11

 …Algorithm Analysis
• Factors affecting the running time
• computer
• compiler
• algorithm
• input to the algorithm
• The content of the input affects the running time
• Typically, the input size (number of items in the input) is
the main consideration
– E.g. sorting problem Þ the number of items to be sorted
– E.g. multiply 2 matrices together Þ the total number of elements
in the 2 matrices

Data Structures & Algorithms 12

 …Algorithm analysis

• Analyzing algorithms
• Employing mathematical techniques that analyze algorithms
independently of specific compilers, computers.

• To analyze algorithms
1. Starting to count the number of significant operations in a
particular solution to assess its efficiency
2. Expressing the efficiency of algorithms using growth functions

Data Structures & Algorithms 13

 CONTENTS

• Introduction to algorithm

• Algorithm analysis

• Estimating running time

• Algorithm growth rates (Big O, Omega, Theta)

• Worst-Case, Best-Case, Average-Case

Data Structures & Algorithms 14

 Estimating running time
• Each operation in algorithm/program has a cost
ð Each operation takes a certain of running time

Ex: count = count + 1;

ð takes a certain amount of time, but it is constant

A sequence of operations:
count = count + 1; //cost: c1
sum = sum + count; //cost: c2

ð Total Cost = c1 + c2

Data Structures & Algorithms 15

 …Estimating running time

• Ex: Simple If-Statement

Cost Times
if (n < 0) c1 1
absval = -n c2 1
else
absval = n; c3 1

ð Total Cost <= c1 + max(c2,c3)

Data Structures & Algorithms 16

 …Estimating running time

• Ex: Simple Loop

Cost Times
i = 1; c1 1
sum = 0; c2 1
while (i <= n) { c3 n+1
i = i + 1; c4 n
sum = sum + i; c5 n
}
ð Total Cost = c1 + c2 + (n+1)*c3 + n*c4 + n*c5
ð The time required for this algorithm is proportional to n
– When n tends to infinity

Data Structures & Algorithms 17

 …Estimating running time
• Ex: Nested Loop
Cost Times
i=1; c1 1
sum = 0; c2 1
while (i <= n) { c3 n+1
j=1; c4 n
while (j <= n) { c5 n*(n+1)
sum = sum + i; c6 n*n
j = j + 1; c7 n*n
}
i = i + 1; c8 n
}
ð Total Cost = c1 + c2 + (n+1)*c3 + n*c4 + n*(n+1)*c5+n*n*c6+n*n*c7+n*c8
ð The time required for this algorithm is proportional to n2
– When n tends to infinity
Data Structures & Algorithms 18
 …Estimating running time

• General rules for running time estimation

• Consecutive Statements: Just add the running times of those
consecutive statements

• Conditional Statements (If/Else): Never more than the running time

of the test plus the larger of running times of two branches

• Loops: The running time of a loop is at most the running time of the
statements inside of that loop times the number of iterations

• Recursion: Determine and solve the recurrence relation

(we don’t focus on this case in this course)

Data Structures & Algorithms 19

 …Estimating running time

• Function calls
• Non recursive calls
• A function call is considered as a statement
ð The running time of a function call is considered as the running
time of a statement (that function)

• Recursive calls
• Set up the recurrence relation
• Solve the recurrence
–May be very complicated

Data Structures & Algorithms 20

 …Estimating running time

• Example
T(n) = n
sum = 0
for (j=0; j<n; j++)
sum++;

• Example
T(n) = n2
sum = 0
for (j=0; j<n; j++)
for (k=0; k<n; k++)
sum++;

Data Structures & Algorithms 21

 …Estimating running time

• Example
T(n) = n3
sum = 0
for (j=0; j<n; j++)
for (k=0; k<n*n; k++)
sum++; • Example
T(n) = n 4
sum = 0;
for (j=0; j<n; j++)
for (k=0; k<j*j; k++)
if (k%j == 0)
for (m=0; m<k; m++)
sum++;
Data Structures & Algorithms 22
 …Estimating running time

• Example

int fact(int n){ Recurrence relation

if (n==0) T(n) = T(n-1) + 1, T(0) = 0
return 1;
else T(n) = n
return (n * fact(n-1));
}

Data Structures & Algorithms 23

 CONTENTS

• Introduction to algorithm

• Algorithm analysis

• Estimating running time

• Algorithm growth rates (Big O, Omega, Theta)

• Worst-Case, Best-Case, Average-Case

Data Structures & Algorithms 24

 Algorithm growth rates
• Measuring an algorithm’s time requirement as a function of the input size
• Input size depends on the problem
Ex: number of elements in a list for a sorting algorithm
If the input size is n:
• Algorithm A requires 5*n2 time units to solve a problem of input size n
• Algorithm B requires 7*n time units to solve a problem of input size n
• The algorithm’s time requirement grows as a function of the input size
• Algorithm A requires time proportional to n2
• Algorithm B requires time proportional to n
• An algorithm’s proportional time requirement is known as growth rate
• Comparing the efficiency of two algorithms by comparing their growth rates
Data Structures & Algorithms 25
 …Algorithm growth rates

Time requirements as
a function of the input size n

Data Structures & Algorithms 26

 …Algorithm growth rates

• Common Growth Rates

Function Growth Rate Name

c Constant
log N Logarithmic
log2 N Log-squared
N Linear
N log N Log-linear
N2 Quadratic
N3 Cubic
2N Exponential
Data Structures & Algorithms 27
 …Algorithm growth rates

Running Times for Small Inputs

Running time

Input size (x = n)
Data Structures & Algorithms 28
 …Algorithm growth rates

Running Times for Large Inputs

Running time

Input size (x = n)
Data Structures & Algorithms 29
 …Algorithm growth rates

• Asymptotic notations / growth-rate functions

• Upper bound O(g(n)

• Lower bound W(g(n))

• Tight bound Q(g(n))

Data Structures & Algorithms 30

 …Algorithm growth rates

• Big O
• f(n) = O(g(n))
• There are positive constants c and n0 such that
f(n) £ c g(n) when n ³ n0

• growth rate of f(n) £ growth rate of g(n)

• g(n) is an upper bound on f(n)

Data Structures & Algorithms 31

 …Algorithm growth rates

• Big O
• If Algorithm A requires time proportional to g(n), Algorithm A is
said to be order g(n), and it is denoted as O(g(n)).

• The function g(n) is called the algorithm’s growth-rate function.

• The capital O is used in the notation

ð called the Big O notation.

• If Algorithm A requires time proportional to n2, it is O(n2).

• If Algorithm A requires time proportional to n, it is O(n).

Data Structures & Algorithms 32

 …Algorithm growth rates

• Big O – Example
• Let f(n) = 2n2. Then
• f(n) = O(n4)
• f(n) = O(n3)
• f(n) = O(n2) (best answer, asymptotically tight)

• O(n2): reads “order n-squared” or “Big-O n-squared”

Data Structures & Algorithms 33

 …Algorithm growth rates

• Big O – Some rules

• Ignore the lower order terms

• Ignore the coefficients of the highest-order term

• If T(n) is an asymptotically positive polynomial of degree k,

then T(n) = O(nk)

Ex: 7n2 + 10n + 3 = O(n2)

Data Structures & Algorithms 34

 …Algorithm growth rates

• Big O – Some rules

• No need to specify the base of logarithm

• Changing the base from one constant to another changes
the value of the logarithm by only a constant factor
• For logarithmic functions,
T(logm n) = O(log n), (use: T(logm n) = T((log2 n) / (log2 m)))

• If T1(n) = O(f(n) and T2(n) = O(g(n)),

• T1(n) + T2(n) = max( O(f(n)), O(g(n)) )
• T1(n) * T2(n) = O( f(n) * g(n) )

Data Structures & Algorithms 35

 …Algorithm growth rates

• Big O – more example

• n2 / 2 – 3n = O(n2)
• 1 + 4n = O(n)
• 7n2 + 10n + 3 = O(n2) = O(n3)
• log10 n = log2 n / log2 10 = O(log2 n) = O(log n)
• 10 = O(1), 1010 = O(1)
• log n + n = O(n)

N N
å i =1
2
i £ N × N = O( N ) åi =1 i 2
£ N × N 2
= O ( N 3
)

Data Structures & Algorithms 36

 …Algorithm growth rates

• Big Omega
• f(n) = W(g(n))
• There are positive constants c and n0 such that
f(n) ³ c g(n) when n ³ n0

• growth rate of f(n) ³ growth rate of g(n)

• g(n) is a lower bound on f(n)

• f(n) grows no slower than g(n) for “large” n

Data Structures & Algorithms 37

 …Algorithm growth rates

• Big Omega – Examples

• Let f(n) = 2n2

• f(n) = W(n)
• f(n) = W(n2) (best answer)

Data Structures & Algorithms 38

 …Algorithm growth rates

• Big Theta
• f(n) = Q(g(n)) iff f(n) = O(g(n)) and f(n) = W(g(n))

• growth rate of f(n) = growth rate of g(n)

• Big-Theta means the bound is the tightest possible

• Example: Let f(n)=2n2 , g(n)=n2

• since f(n) = O(g(n)) and f(n) = W(g(n)),
thus f(n) = Q(g(n))

Data Structures & Algorithms 39

 …Algorithm growth rates

• Big Theta – Some rules

• If T(n) is a asymptotically positive polynomial of degree k,

then T(n) = Q(nk)

• For logarithmic functions,

T(logm n) = Q(log n), (use: T(logm n) = T((log2 n) / (log2 m)))

Data Structures & Algorithms 40

 …Algorithm growth rates

A Comparison of Growth-Rate Functions

Data Structures & Algorithms 41

 …Algorithm growth rates

• Asymptotic notations
• When n goes to infinity
• Upper bound O(g(n)
–the most popular

• Lower bound W(g(n))

• Tight bound Q(g(n))

Data Structures & Algorithms 42

 …Algorithm growth rates

Growth-Rate Functions

O(1) Time requirement is constant, and it is independent of the input size.

O(log2n) Time requirement for a logarithmic algorithm increases increases slowly
as the input size increases.
O(n) Time requirement for a linear algorithm increases directly with the input
size.
O(n*log2n) Time requirement for a n*log2n algorithm increases more rapidly than
a linear algorithm.
O(n2) Time requirement for a quadratic algorithm increases rapidly with the
input size.
O(n3) Time requirement for a cubic algorithm increases more rapidly with the
input size than the time requirement for a quadratic algorithm.
O(2n) As the input size of the problem increases, the time requirement for an
exponential algorithm increases too rapidly to be practical.
Data Structures & Algorithms 43
 …Algorithm growth rates

• Reminder of Properties of Growth-Rate Functions

• We can ignore low-order terms in an algorithm’s growth-rate
function.
• If an algorithm is O(n3+4n2+3n), it is also O(n3).
• We only use the higher-order term as algorithm’s growth-rate function.
• We can ignore a multiplicative constant in the higher-order term
of an algorithm’s growth-rate function.
• If an algorithm is O(5n3), it is also O(n3).
• O(f(n)) + O(g(n)) = O(f(n)+g(n))
• If an algorithm is O(n3) + O(4n), it is also O(n3 +4n) ð it is O(n3)
• O(f(n)) * O(g(n)) = O(f(n) * g(n))
Data Structures & Algorithms 44
 …Algorithm growth rates

• Example 1
Cost Times
i = 1; c1 1
sum = 0; c2 1
while (i <= n) { c3 n+1
i = i + 1; c4 n
sum = sum + i; c5 n
}

T(n) = c1 + c2 + (n+1)*c3 + n*c4 + n*c5

= (c3+c4+c5)*n + (c1+c2+c3)
= a*n + b
ð the growth-rate function for this algorithm is O(n)
Data Structures & Algorithms 45
 …Algorithm growth rates

• Example 2

for (i=1; i<=n; i++)

for (j=1; j<=i; j++)
for (k=1; k<=j; k++)
x=x+1;

n i j
T(n) = å å å1
i=1 j=1 k=1

ð the growth-rate function for this algorithm is O(n3)

Data Structures & Algorithms 46

 …Algorithm growth rates

• Example 3 Cost Times

i=1; c1 1
sum = 0; c2 1
while (i <= n) { c3 n+1
j=1; c4 n
while (j <= n) { c5 n*(n+1)
sum = sum + i; c6 n*n
j = j + 1; c7 n*n
}
i = i +1; c8 n
}
T(n) = c1 + c2 + (n+1)*c3 + n*c4 + n*(n+1)*c5+n*n*c6+n*n*c7+n*c8
= (c5+c6+c7)*n2 + (c3+c4+c5+c8)*n + (c1+c2+c3)
= a*n2 + b*n + c
ð the growth-rate function for this algorithm is O(n2)
Data Structures & Algorithms 47
 CONTENTS

• Introduction to algorithm

• Algorithm analysis

• Estimating running time

• Algorithm growth rates (Big O, Omega, Theta)

• Worst-Case, Best-Case, Average-Case

Data Structures & Algorithms 48

 Worst-Case, Best-Case, Average-Case
• Running time of algorithm depends on not only the input size but also the input content
• Worst-Case
• The maximum amount of time that an algorithm require to solve a problem of size n.
• This gives an upper bound for the time complexity of an algorithm.
• Normally, we try to find worst-case behavior of an algorithm.
• Best-Case
• The minimum amount of time that an algorithm require to solve a problem of size n.
• The best case behavior of an algorithm is NOT so useful.
• Average-Case
• The average amount of time that an algorithm require to solve a problem of size n.
• Sometimes, it is difficult to find the average-case behavior of an algorithm.
• We have to look at all possible data organizations of a given size n, and their
distribution probabilities of these organizations.
• Worst-case analysis is more common than average-case analysis.

Data Structures & Algorithms 49

 …Worst-Case, Best-Case, Average-Case

• Sequential Search – Analysis

. int sequentialSearch(const int a[], int item, int n){
for (int i = 0; i < n && a[i]!= item; i++);
if (i == n) return –1;
return i;
}
• Unsuccessful Search: O(n)
• Successful Search:
Best-Case: item is in the first location of the array ð O(1)
Worst-Case: item is in the last location of the array ð O(n)
Average-Case: The number of key comparisons 1, 2, ..., n
n
∑i ( n 2 +n)/ 2
i=1
=
ð O(n)
n n
Data Structures & Algorithms 50
 …Worst-Case, Best-Case, Average-Case

• Binary Search – Analysis

int binarySearch(int a[], int size, int x) {
int low =0;
int high = size –1;
int mid; // mid will be the index of target when it’s found.
while (low <= high) {
mid = (low + high)/2;
if (a[mid] < x) low = mid + 1;
else if (a[mid] > x) high = mid – 1;
else return mid;
}
return –1;
We can do binary search if the array is sorted
}
Data Structures & Algorithms 51
 …Worst-Case, Best-Case, Average-Case

• Binary Search – Analysis

. • Unsuccessful Search
• The size of the list for each iteration: n/2, n/22, n/23, …, n/2k
• Loop stops when n/2k = 1, where k is the number of iterations
• Then, the number of iterations k in the loop is log2n ð O(log2n)

• Successful Search
Best-Case: The number of iterations is 1 ð O(1)
Worst-Case: The number of iterations is log2n ð O(log2n)
Average-Case: The avg. number of iterations < log2n ð O(log2n)

Data Structures & Algorithms 52

 …Worst-Case, Best-Case, Average-Case

• How much better is O(log2n)?

. n O(log2n)
16 4
64 6
256 8
1024 10
16,384 14
131,072 17
262,144 18
524,288 19
1,048,576 20
1,073,741,824 30
Data Structures & Algorithms 53
 Conclusions
• Running time depends on
• Input size
• A function of n (input size)
• Running time is significant, when n goes to infinity
• Input content
• Best-case, Average-case, Worst-case
• Steps for estimating running time (compelxity)
1. Counting the number of significant operations/statements
• We often consider the worst-case
2. Applying the growth rate functions (O, W, Q)
• When n goes to infinity
• O is the most popularly used
3. Classifying the algorithm running time/complexity
• The algorithm is impractical, if its running time is more than O(n3)
Data Structures & Algorithms 54
 SUMMARY

• Introduction to algorithm

• Algorithm analysis

• Estimating running time

• Algorithm growth rates

• Worst-Case, Best-Case, Average-Case

Data Structures & Algorithms 55

 Nhân bản – Phụng sự – Khai phóng

Enjoy the Course…!

Data Structures & Algorithms 56