Algorithm Analysis

with

Big Oh

Data Structures and Design with Java and JUnit

Chapter 12
©Rick Mercer
 Algorithm Analysis
w Objectives
⎯  Analyze the efficiency of algorithms
⎯  Analyze a few classic algorithms
•  Linear Search, Binary Search, Selection Sort
⎯  Know the differences between O(1), O(n),
O(log n), and O(n2)
⎯  Visualize runtime differences with
experiments
 Algorithms continued

w Computer Scientists focus on problems such as

⎯  How fast do algorithms run
⎯  How much memory does the process require
w Example Applications
⎯  Make the Internet run faster
•  Pink-Degemark's routing algorithms
•  Gene Meyers determined the sequences of the Human
genome using his whole genome shotgun algorithm
 Analysis of Algorithms

w We have ways to compare algorithms

⎯  Generally, the larger the problem, the longer it
takes the algorithm to complete
⎯  Sorting 100,000 elements can take much more
time than sorting 1,000 elements
•  and more than 10 times longer
⎯  the variable n suggests the "number of things"
⎯  If an algorithm requires 0.025n2 + 0.012n +
0.0005 seconds, just plug in a value for n
 A Computational Model

w To summarize algorithm runtimes, we can

use a computer independent model
⎯  instructions are executed sequentially
⎯  count all assignments, comparisons, and
increments there is infinite memory
⎯  every simple instruction takes one unit of time
 Simple Instructions

w Count the simple instructions

⎯  assignments have cost of 1
⎯  comparisons have a cost of 1
⎯  let's count all parts of the loop
for (int j = 0; j < n; j++)
•  j=0 has a cost of 1, j<n executes n+1 times,and j++
executes n times for a total cost of 2n+2
⎯  each statement in the repeated part of a loop have
have a cost equal to number of iterations
 Examples
Cost
sum = 0; -> 1
sum = sum + next; -> 1 Total Cost: 2

Cost
for (int i = 1; i <= n; i++) -> 1 + n+1 + n = 2n+2
sum = sum++; -> n Total Cost: 3n + 2

Cost
k = 0 -> 1
for (int i = 0; i < n; i++) -> 2n+2
for (int j = 0; j < n; j++) -> n(2n+2) = 2n2 +2n
k++; -> n2 Total Cost: 3n2 + 4n + 3`
 Total Cost of Sequential Search
Cost

for (int index = 0; index < n; index++) -> 2n + 2

if(searchID.equals(names[index]) -> n
return index; -> 0 or 1

return -1 // if not found -> 0 or 1

Total cost = 3n+3

 Different Cases
w The total cost of sequential search is 3n + 3
⎯  But is it always exactly 3n + 3 instructions?
⎯  The last assignment does not always execute
•  But does one assignment really matter?
⎯  How many times will the loop actually execute?
•  that depends
⎯  If searchID is found at index 0: _____ iterations
•  best case
⎯  If searchID is found at index n-1:_____ iterations
•  worst case
 Typical Case of sequential (linear)
w The average describes the more typical case
w First, let the the entire cost be simplified to n
⎯  Assume the target has a 50/50 chance of being in the array
•  n comparisons are made: worst-case occurs 1/2 the time
⎯  Assume if it's in a, it's as likely to be in one index as another
1 1 n n n 3
n + × = + = n
2 2 2 2 4 4

Half the time it is n comparisons, the other half it is n/2 comparisons

⎯  So the typical case is 3/4 n comparisons
 The Essence of Linear Search
Plot the function this is why sequential search is also
called linear search. As n increases, runtime forms a line
f(n)
75

45

n
60 100
 Linear Search Continued

u  This equation is a polynomial: 3n + 3

u  The fastest growing term is the high-order term
u  The high order term (which could be n2 or n3),
represents the most important part of the
analysis function
u  We refer to the rate of growth as the order of
magnitude, which measure the rate of growth
Rate of Growth

w Imagine two functions:

f(n) = 100n g(n) = n2 + n
⎯  When n is small, which is the bigger function?
⎯  When n is big, which is the bigger function?
⎯  We can say: g(n) grows faster than f(n)
Rate of Growth, another view
Function growth and weight of terms as a percentage of
all terms as n increases for
f(n) = n2 + 80n + 500

Conclusion: consider highest order term with the coefficient

dropped, also drop all lower order terms
 Definition
w The asymptotic growth of an algorithm
⎯  describes the relative growth of an algorithm as n gets
very large
⎯  With speed and memory increases doubling every two
years, the asymptotic efficiency where n is very large is
the thing to consider
⎯  There are many sorting algorithm that are "on the order
of" n2 (there are roughly n×n instructions executed)
⎯  Other algorithms are "on the order of" n×log2n
•  and this is a huge difference when n is very large
 Constant Function
w Some functions don't grow with n
⎯  If the sorting program initializes a few variables
first, the time required does not change when n
increases
⎯  These statements run in constant time
•  e.g. construct an empty List with capacity 20
⎯  The amount of time can be described as a
constant function f(n) = k, where k is a constant
⎯  it takes ~0.0003 seconds no matter the size of n
 Big O
w Linear search is "on the order of n", which can be
written as O(n) to describe the upper bound on the
number of operations
w This is called big O notation
w Orders of magnitude:
O(1) constant (the size of n has no effect)
O(n) linear
O(log n) logarithmic
O(n log n) no other way to say it, John K’s License plate
O(n2) quadratic
O(n3) cubic
O(2n) exponential
 Binary Search
w We'll see that binary search can be a more
efficient algorithm for searching
If the element in the middle is the target
report target was found and the search is done
if the key is smaller
search the array to the left
Otherwise
search the array to the right
w This process repeats until the target is found
or there is nothing left to search
w Each comparison narrows search by half
Binary Search Harry
Data reference loop 1 loop 2
Bob a[0] left
Carl a[1]
Debbie a[2]
Evan a[3]
Froggie a[4] mid
Gene a[5] left
Harry a[6] mid
Igor a[7]
Jose a[8] right
right
How fast is Binary Search?
w Best case: 1
w Worst case: when target is not in the array
w At each pass, the "live" portion of the array is
narrowed to half the previous size.
w The series proceeds like this:
⎯  n , n/2, n/4, n/8, ...
w Each term in the series represents one
comparison How long does it take to get to 1?
⎯  This will be the number of comparisons
 Binary Search (con.)
w Could start at 1 and double until we get to n
1, 2, 4, 8, 16, ... , k >= n or
20, 21, 22, 23, 24, ... , 2c >= n
w The length of this series is c+1
w The question is
2 to what power c is greater than or equal to n?
•  if n is 8, c is 3
•  if n is 1024, c is 10
•  if n is 16,777,216, c is 24
w Binary search runs O(log n) logarithmic
 Comparing O(n) to O(log n)

Rates of growth and logarithmic functions

Power of 2 n log2n
24 16 4
28 128 8
212 4,096 12
224 16,777,216 24
 Graph Illustrating Relative
Growth n, log n, n2
f(n)
n2
n

log n

n
 Other logarithm examples
w The guessing game:
⎯  Guess a number from 1 to 100
•  try the middle, you could be right
•  if it is too high
–  check near middle of 1..49
•  if it is too low
–  check near middle of 51..100
⎯  Should find the answer in a maximum of 7 tries
•  If 1..250, a maximum of 2c >= 250, c == 8
•  If 1..500, a maximum of 2c >= 500, c == 9
•  If 1..1000, a maximum of 2c >= 1000, c == 10
 Logarithmic Explosion

w Assuming an infinitely large piece of paper

that can be cut in half, layered, and cut in half
again as often as you wish.
⎯  How many times do you need to cut and layer
until paper thickness reaches the moon?
⎯  Assumptions
•  paper is 0.002 inches thick
•  distance to moon is 240,000 miles
–  240,000 * 5,280 feet per mile * 12 inches per foot =
152,060,000,000 inches to the moon
Examples of Logarithmic Explosion

w The number of bits required to store a binary

number is logarithmic add 1 bit to get much larger ints
⎯  8 bits stored 256 values log2256 = 8
⎯  log 2,147,483,648 = 31
w The inventor of chess asked the Emperor to be
paid like this:
⎯  1 grain of rice on the first square, 2 on the next,
double grains on each successive square 263
 Compare Sequential and
Binary Search
w Output from CompareSearches.java (1995)
Search for 20000 objects
Seconds  2013
Binary Search 1200

#Comparisons: 267248 1000

Average: 13 800
Run time: 20ms
600

Sequential Search 400

#Comparisons: 200010000 200

Average: 10000
0
Run time: 9930ms 0 200 400 600 800 1000

Difference in comparisons : 199742752

Difference in milliseconds: 9910
O(n2) quadratic

w O(n2) reads on the order of n squared or quadratic

w When n is small, rates of growth don’t matter
w Quadratic algorithms are greatly affected by
increases in n
⎯  Consider the selection sort algorithm
•  Find the largest, n-1 times
Actual observed data for O(n2) sort
Time required to sort an array of size n
400
350
300
250
200 Seconds
150
100
50
0
1 10 20 30 40
n in thousands
Two O(n2) algorithms
w Many known sorting algorithms are O(n2)
w Given n points, find the pair that are closest
Compare p1 with p2, p3, p4, p5 (4 comparisons)
Compare p2 with p3, p4, p5 (3 comparisons)
Compare p3 with p4, p5 (2 comparisons)
Compare p4 with p5 (1 comparisons)
⎯  When n is 5, make 10 comparisons
⎯  In general, #comparisons is
n(n-1) / 2 == n2/2 - n/2
⎯  highest order term is n2, drop ½ and runtime
is O(n2)
O(n3) algorithms
w Matrix Multiplication (naïve):
for(int i = 0; i < m.length; i++) {
for(int j = 0; j < m2.length - 1; j++) {
for(int k = 0; k < m2.length; k++){
m[i][j] += m[i][k] * m2[k][j];
}
}
}
Big O and Style Guidelines

w Big O is similar to saying the runtime is

less than or equal to Big O notation.
⎯  O(f) is an upper bound
w Don't use constants or lower-order terms
⎯  These are no-nos for now (you will use
coefficients in C Sc 345)
O(n2 + n) should be written O(n2)
O(5500n) should be written O(n)
O(2.5n) should be written O(n)
 Properties of Big-O
Summarizing two main properties
If f(n) is a sum of several terms, the one with the largest growth
rate is kept, and all others omitted

If f(n) is a product of several factors, any constants (terms in the

product that do not depend on n) are omitted – which means you
can drop coefficients
 Properties of Big-O
We can drop coefficient

Example:

f(n) = 100*n
then f(n) is O(n)
Summation of same Orders
The property is useful when an algorithm contains
several loops of the same order

Example:

f(n) is O(n)
f2(n) is O(n)
then f(n) + f2(n) is O(n) + O(n), which is O(n)
Summation of different Orders
This property works because we are only concerned
with the term of highest growth rate

Example:

f1(n) is O(n2)
f2(n) is O(n)
so f1(n) + f2(n) = n2 + n is O(n2)
Product
This property is useful for analyzing segments of an
algorithm with nested loops

Example:

f1(n) is O(n2)
f2(n) is O(n)
then f1(n) x f2(n) is O(n2) x O(n), which is O(n3)
Limitations of Big-Oh Analysis
w Constants sometimes make a difference
⎯  n log n may be faster than 10000n
⎯  Doesn't differentiate between data cache
memory, main memory, and data on a disk--there
is a huge time difference to access disk data
•  thousands of times slower
⎯  Worst case doesn't happen often
•  it's an overestimate
Quick Analysis

w Can be less detailed

w Running time of nested loops is
⎯  the product of each loop's number of iterations
w Several consecutive loops
⎯  the longest running loop
⎯ 3n is O(n) after all
Runtimes with for loops
int n = 1000;
int[] x = new int[n];

w  O(n)
for(int j = 0; j < n; j++)
x[j] = 0;

w O(n2)
int sum = 0;
for (int j = 0; j < n; j++)
for (int k = 0; k < n; k++)
sum += j * k;
Run times with for loops
w O(n3)
for (int j = 0; j < n; j++)
for (int k = 0; k < n; k++)
for (int l = 0; l < n; l++)
sum += j * k * l;
w O(n)
for (int j = 0; j < n; j++)
sum++;
for (int j = 0; j < n; j++)
sum--;

w O(log n)
for (int j = 1; j < n; j = 2 * j)
sum += j;
Analyze this

public void swap(int[] a, int left, int right) {

int temp = array[left];
array[left] = array[right];
array [right] = temp;
}
Analyze that

for (int j = 0; j < n; j++)

sum += l;

for (int k = 0; k < n; k++)

sum += l;

for (int l = 0; l < n; l++)

sum += l;
Analyze that

for (int j = 0; j < n; j++)

for (int k = 0; k < n; k++)
sum += k + l;

for (int l = 0; l < n; l++)

sum += l;
Analyze this

for (int top = 0; top < n - 1; top++) {

int smallestIndex = top;
for (int index = top; index < n; index++) {
if(a[index] < a[smallestIndex])
smallestIndex = index;
}
// Swap smallest to the top index
swap(a, top, smallestIndex);
}