Lecture 2

Divide-and-conquer, MergeSort, and Big-O notation

Announcements
• Homework!
• HW1 will be released Friday.
• It is due the following Friday.

• See the website for guidelines on homework, including:

• Collaboration policy
• Best practices/style guide
• Will be posted by Friday!
 Cast
Last time
Philosophy
• Algorithms are awesome and
powerful!
Plucky the Lucky the
• Algorithm designer’s question: pedantic lackadaisical
Can I do better? penguin lemur

Technical content
• Karatsuba integer multiplication
• Example of “Divide and Conquer”
• Not-so-rigorous analysis Ollie the Siggi the
over-achieving studious
ostrich stork
Today
• Things we want to know about algorithms:
• Does it work?
• Is it efficient?

• We’ll start to see how to answer these by looking at

some examples of sorting algorithms.
• InsertionSort
• MergeSort

SortingHatSort not discussed

 The plan
• Part I: Sorting Algorithms
• InsertionSort: does it work and is it fast?
• MergeSort: does it work and is it fast?
• Skills:
• Analyzing correctness of iterative and recursive algorithms.
• Analyzing running time of recursive algorithms (part 1…more next time!)

• Part II: How do we measure the runtime of an algorithm?

• Worst-case analysis
• Asymptotic Analysis
Sorting
• Important primitive
• For today, we’ll pretend all elements are distinct.

6 4 3 8 1 5 2 7

1 2 3 4 5 6 7 8
I hope everyone did the
pre-lecture exercise!
def mysteryAlgorithmOne(A):
B = [None for i in range(len(A))]
for i in range(len(B)):
What was the if B[i] == None or B[i] > x:
mystery sort j = len(B)-1
while j > i:
algorithm? B[j] = B[j-1]
j -= 1
B[i] = x
break
1. MergeSort return B

2. QuickSort def MysteryAlgorithmTwo(A):

for i in range(1,len(A)):
3. InsertionSort current = A[i]
j = i-1
4. BogoSort while j >= 0 and A[j] > current:
A[j+1] = A[j]
j -= 1
A[j+1] = current
 We’re going to go
Benchmark: insertion sort through this in some
detail – it’s good
practice!

• Say we want to sort: 6 4 3 8 5

• Insert items one at a time.

• How would we actually implement this?

In your pre-lecture exercise…

def InsertionSort(A):
for i in range(1,len(A)):
current = A[i]
j = i-1
while j >= 0 and A[j] > current:
A[j+1] = A[j]
j -= 1
A[j+1] = current
InsertionSort 6 4 3 8 5
example
Start by moving A[1] toward
the beginning of the list until
you find something smaller
(or can’t go any further): Then move A[3]:

6 4 3 8 5 3 4 6 8 5
4 6 3 8 5 3 4 6 8 5
Then move A[2]: Then move A[4]:
4 6 3 8 5 3 4 6 8 5
3 4 6 8 5 3 4 5 6 8
Then we are done!
Insertion Sort
1. Does it work?
2. Is it fast?
Empirical answers…
• Does it work?
• You saw it worked on the pre-Lecture exercise.
• Is it fast?
• IPython notebook lecture2_sorting.ipynb says:
Insertion Sort
1. Does it work?
2. Is it fast?

• The “same” algorithm can

be faster or slower
depending on the
implementation…

• We are interested in how

fast the running time
scales with n, the size of
the input.
 Insertion Sort: running time
def InsertionSort(A):
for i in range(1,len(A)):
current = A[i]
j = i-1
n-1 iterations
while j >= 0 and A[j] > current:
of the outer
A[j+1] = A[j]
loop
j -= 1
A[j+1] = current

In the worst case,

about n iterations
of this inner loop Seems
plausible
Running time scales like n2
Insertion Sort
1. Does it work?
2. Is it fast?

• Okay, so it’s pretty obvious that it works.

• HOWEVER! In the future it won’t be so

obvious, so let’s take some time now to
see how we would prove this rigorously.
Why does this work?

• Say you have a sorted list, 3 4 6 8 , and

another element 5 .

• Insert 5 right after the largest thing that’s still

smaller than 5 . (Aka, right after 4 ).

• Then you get a sorted list: 3 4 5 6 8

So just use this logic at every step.
6 4 3 8 5 The first element, [6], makes up a sorted list.

So correctly inserting 4 into the list [6] means

4 6 3 8 5 that [4,6] becomes a sorted list.

The first two elements, [4,6], make up a

4 6 3 8 5 sorted list.
So correctly inserting 3 into the list [4,6] means
3 4 6 8 5 that [3,4,6] becomes a sorted list.
The first three elements, [3,4,6], make up a
3 4 6 8 5 sorted list.
So correctly inserting 8 into the list [3,4,6] means
3 4 6 8 5 that [3,4,6,8] becomes a sorted list.
The first four elements, [3,4,6,8], make up a
3 4 6 8 5 sorted list.
So correctly inserting 5 into the list [3,4,6,8]
3 4 5 6 8 means that [3,4,5,6,8] becomes a sorted list.
YAY WE ARE DONE!
 This slide skipped in class;
for reference only.
Recall: proof by induction
A loop invariant is something that
• Maintain a loop invariant. should be true at every iteration.

• Proceed by induction.

• Four steps in the proof by induction:

• Inductive Hypothesis: The loop invariant holds after the
ith iteration.
• Base case: the loop invariant holds before the 1st
iteration.
• Inductive step: If the loop invariant holds after the ith
iteration, then it holds after the (i+1)st iteration
• Conclusion: If the loop invariant holds after the last
iteration, then we win.
Formally: induction A “loop invariant” is
something that we maintain
at every iteration of the
• Loop invariant(i): A[:i+1] is sorted. algorithm.

• Inductive Hypothesis:
• The loop invariant(i) holds at the end of the ith iteration (of the outer
loop).
• Base case (i=0):
• Before the algorithm starts, A[:1] is sorted. ✓
• Inductive step:
• Conclusion:
• At the end of the n-1’st iteration (aka, at the end of the algorithm),
A[:n] = A is sorted.
• That’s what we wanted!✓

The first two elements, [4,6], make up a This was

4 6 3 8 5 sorted list. iteration i=2.

So correctly inserting 3 into the list [4,6] means

3 4 6 8 5 that [3,4,6] becomes a sorted list.
 Aside: proofs by induction
• We’re gonna see/do/skip over a lot of them.
• I’m assuming you’re comfortable with them from CS103.
• When you assume…
• If that went by too fast and was confusing:
• Slides [there’s a hidden one with more info]
• Lecture notes
Make sure you really understand the
• Book argument on the previous slide!
• Office Hours

Siggi the Studious Stork

To summarize

InsertionSort is an algorithm that

correctly sorts an arbitrary n-element
array in time that scales like n2.

Can we do better?
 The plan
• Part I: Sorting Algorithms
• InsertionSort: does it work and is it fast?
• MergeSort: does it work and is it fast?

• Skills:
• Analyzing correctness of iterative and recursive algorithms.
• Analyzing running time of recursive algorithms (part A)

• Part II: How do we measure the runtime of an algorithm?

• Worst-case analysis
• Asymptotic Analysis
 Can we do better?
• MergeSort: a divide-and-conquer approach
• Recall from last time:

Divide and
Conquer: Big problem

Smaller Smaller
problem problem
Recurse! Recurse!

Yet smaller Yet smaller Yet smaller Yet smaller

problem problem problem problem
 MergeSort
6 4 3 8 1 5 2 7

6 4 3 8 1 5 2 7
Recursive magic! Recursive magic!

3 4 6 8 1 2 5 7

How would
you do this
MERGE! 1 2 3 4 5 6 7 8 in-place?

Code for the MERGE step is given in the

Lecture2 notebook or the Lecture Notes Ollie the over-achieving Ostrich
 See Lecture 2 IPython notebook for MergeSort Python Code.

MergeSort Pseudocode
MERGESORT(A):
• n = length(A)
• if n ≤ 1: If A has length 1,
It is already sorted!
• return A
Sort the left half
• L = MERGESORT(A[ 0 : n/2])
Sort the right half
• R = MERGESORT(A[n/2 : n ])
• return MERGE(L,R) Merge the two halves
What actually happens?
First, recursively break up the array all the way down to the
base cases

6 4 3 8 1 5 2 7

6 4 3 8 1 5 2 7

6 4 3 8 1 5 2 7

6 4 3 8 1 5 2 7
This array of
length 1 is
sorted!
Then, merge them all back up!
Sorted sequence!

1 2 3 4 5 6 7 8
Merge!

3 4 6 8 1 2 5 7
Merge! Merge!

4 6 3 8 1 5 2 7
Merge! Merge! Merge! Merge!

6 4 3 8 1 5 2 7
A bunch of sorted lists of length 1 (in the order of the original sequence).
Two questions

1. Does this work?

IPython notebook says…
2. Is it fast?

Empirically:
1. Seems to.
2. Maybe?
 Again we’ll use induction.
It works Let’s assume n = 2t This time with an invariant
that will remain true after
every recursive call.
• Inductive hypothesis:
“In every recursive call,
MERGESORT returns a sorted array.”

• Base case (n=1): a 1-element

array is always sorted.
• Inductive step: Suppose that L • n = length(A)
and R are sorted. Then • if n ≤ 1:
MERGE(L,R) is sorted. • return A
• Conclusion: “In the top recursive • L = MERGESORT(A[1 : n/2])
call, MERGESORT returns a • R = MERGESORT(A[n/2+1 : n ])
sorted array.” • return MERGE(L,R)
Fill in the inductive step! (Either do it
yourself or read it in CLRS!)
 It’s fast Let’s keep assuming n = 2t
CLAIM:

MERGESORT requires at most 11n (log(n) + 1)

operations to sort n numbers.

What exactly is an “operation” here?

We’re leaving that vague on purpose.
Also I made up the number 11.
How does this compare to
InsertionSort?

Scaling like n2 vs scaling like nlog(n)?

 [See Lecture 2 Notebook for code]

Empirically

This
grows
like n2

This supposedly
grows like nlog(n)
The constant doesn’t matter:
eventually, 𝑛2 > 111111 ⋅ 𝑛 log(𝑛)
 All logarithms in this
Quick log refresher course are base 2

• log(n) : how many times do you need to divide n by

2 in order to get down to 1?

32 64 log(128) = 7
16 32 log(256) = 8
16 log(512) = 9
8 .
8
4 4 .
2 .
2 log(number of particles in
1 1 the universe) < 280
log(32) = 5 log(64) = 6
It’s fast!
CLAIM:

MERGESORT requires at most 11n (log(n) + 1)

operations to sort n numbers.

Much faster than InsertionSort for large n!

(No matter how the algorithms are implemented).
(And no matter what that constant “11” is).
Let’s prove the claim
• Later we’ll see more principled Size n Level 0
ways of analyzing divide-and-
conquer algs. Level 1
n/2 n/2
• But for today let’s just wing it.

n/4 n/4 n/4 n/4

Focus on just one of …

these sub-problems Level t

n/2t n/2t n/2t n/2t n/2t n/2t

(Size 1)
How much work in this sub-problem?

Time spent MERGE-ing

n/2t the two subproblems

+
Time spent within the
n/2t+1 n/2t+1 two sub-problems
How much work in this sub-problem?
Let k=n/2t…

Time spent MERGE-ing

k the two subproblems

+
Time spent within the
k/2 k/2 two sub-problems
 Code for the MERGE
How long does it k step is given in the
Lecture2 notebook.
take to MERGE? k/2 k/2

k/2 k/2

3 4 6 8 1 2 5 7

MERGE! 1 2 3 4 5 6 7 8

k
 Code for the MERGE
How long does it k step is given in the
Lecture2 notebook.
take to MERGE? k/2 k/2

• Time to initialize an
array of size k
• Plus the time to
initialize three counters
• Plus the time to
increment two of those Let’s say no more
counters k/2 times
each than 11k operations.
• Plus the time to
compare two values at
There’s some
least k times
justification for this
• Plus the time to copy k
number “11” in the
values from the
lecture notes, but it’s
existing array to the big
really pretty arbitrary.
array.
• Plus…
Plucky the Lucky the
Pedantic Penguin lackadaisical lemur
 Recursion tree #
Size of
each
Amount of work
Level problems at this level
problem

Size n 11n
0 1 n
n/2 n/2
1 2 n/2 11n
n/4 n/4 n/4 n/4 2 4 n/4 11n
… …
n/2t n/2t n/2t n/2t n/2t
n/2t t 2t n/2t 11n
… …
log(n) n 1 11n
(Size 1)
Total runtime…

• 11n steps per level, at every level

• log(n) + 1 levels

• 11n (log(n) + 1) steps total

That was the claim!

A few reasons to be grumpy
• Sorting

1 2 3 4 5 6 7 8
should take zero steps…

• What’s with this 11k bound?

• You (Mary) made that number “11” up.
• Different operations don’t take the same
amount of time.
How we will deal
with grumpiness

• Take a deep breath…

• Worst case analysis
• Asymptotic notation
 The plan
• Part I: Sorting Algorithms
• InsertionSort: does it work and is it fast?
• MergeSort: does it work and is it fast?
• Skills:
• Analyzing correctness of iterative and recursive algorithms.
• Analyzing running time of recursive algorithms (part A)

• Part II: How do we measure the runtime of an algorithm?

• Worst-case analysis
• Asymptotic Analysis
 Sorting a sorted list
Worst-case analysis should be fast!!

12345678
• In this class, we will focus on worst-case analysis
Here is my algorithm!
Here is an
Algorithm: input!
Do the thing
Do the stuff
Return the answer

Algorithm
designer

• Pros: very strong guarantee

• Cons: very strong guarantee
 How long does an
operation take? Why are
Big-O notation we being so sloppy about
that “11”?

• What do we mean when we measure runtime?

• We probably care about wall time: how long does it take
to solve the problem, in seconds or minutes or hours?

• This is heavily dependent on the programming

language, architecture, etc.
• These things are very important, but are not the
point of this class.
• We want a way to talk about the running time of an
algorithm, independent of these considerations.
Main idea:
Focus on how the runtime scales with n (the input size).
Asymptotic Analysis
How does the running time scale as n gets large?

One algorithm is “faster” than another if its

runtime scales better with the size of the input.

Pros: Cons:
• Abstracts away from • Only makes sense if n is
hardware- and language- large (compared to the
specific issues. constant factors).
• Makes algorithm analysis 2100000000000000 n
much more tractable. is “better” than n2 ?!?!
 pronounced “big-oh of …” or sometimes “oh of …”

O(…) means an upper bound

• Let T(n), g(n) be functions of positive integers.
• Think of T(n) as being a runtime: positive and increasing in n.

• We say “T(n) is O(g(n))” if g(n) grows at least as fast as

T(n) as n gets large.

• Formally,
𝑇 𝑛 =𝑂 𝑔 𝑛
⟺
∃𝑐, 𝑛5 > 0 𝑠. 𝑡. ∀𝑛 ≥ 𝑛5 ,
0 ≤ 𝑇 𝑛 ≤ 𝑐 ⋅ 𝑔(𝑛)
 𝑇 𝑛 =𝑂 𝑔 𝑛
Example ⟺
∃𝑐, 𝑛5 > 0 𝑠. 𝑡. ∀𝑛 ≥ 𝑛5 ,
2𝑛< + 10 = 𝑂 𝑛< 0 ≤ 𝑇 𝑛 ≤ 𝑐 ⋅ 𝑔(𝑛)

3n2

2n2 + 10

n2
 𝑇 𝑛 =𝑂 𝑔 𝑛
Example ⟺
∃𝑐, 𝑛5 > 0 𝑠. 𝑡. ∀𝑛 ≥ 𝑛5 ,
2𝑛< + 10 = 𝑂 𝑛< 0 ≤ 𝑇 𝑛 ≤ 𝑐 ⋅ 𝑔(𝑛)

Formally:
3n2 • Choose c = 3
• Choose n0 = 4
• Then:
∀𝑛 ≥ 4,
0 ≤ 2𝑛< + 10 ≤ 3 ⋅ 𝑛<
n2
 𝑇 𝑛 =𝑂 𝑔 𝑛
same Example ⟺
∃𝑐, 𝑛5 > 0 𝑠. 𝑡. ∀𝑛 ≥ 𝑛5 ,
2𝑛< + 10 = 𝑂 𝑛< 0 ≤ 𝑇 𝑛 ≤ 𝑐 ⋅ 𝑔(𝑛)

Formally:
7n2 • Choose c = 7
• Choose n0 = 2
• Then:
∀𝑛 ≥ 2,
0 ≤ 2𝑛< + 10 ≤ 7 ⋅ 𝑛<
n2
 𝑇 𝑛 =𝑂 𝑔 𝑛

Another example: ⟺
∃𝑐, 𝑛5 > 0 𝑠. 𝑡. ∀𝑛 ≥ 𝑛5 ,

𝑛 = 𝑂(𝑛 )2 0 ≤ 𝑇 𝑛 ≤ 𝑐 ⋅ 𝑔(𝑛)

g(n) = n2 • Choose c = 1
• Choose n0 = 1
• Then

∀𝑛 ≥ 1,
T(n) = n 0 ≤ 𝑛 ≤ 𝑛<
Ω(…) means a lower bound
• We say “T(n) is Ω(g(n))” if g(n) grows at most as fast
as T(n) as n gets large.

• Formally,
𝑇 𝑛 =Ω 𝑔 𝑛
⟺
∃𝑐, 𝑛5 > 0 𝑠. 𝑡. ∀𝑛 ≥ 𝑛5 ,
0≤𝑐⋅𝑔 𝑛 ≤𝑇 𝑛
Switched these!!
 𝑇 𝑛 =Ω 𝑔 𝑛

Example ⟺
∃𝑐, 𝑛5 > 0 𝑠. 𝑡. ∀𝑛 ≥ 𝑛5 ,

𝑛 log < 𝑛 = Ω 3𝑛 0≤𝑐⋅𝑔 𝑛 ≤𝑇 𝑛

• Choose c = 1/3
• Choose n0 = 3
• Then
∀𝑛 ≥ 3,
3𝑛
0≤ ≤ 𝑛 log < 𝑛
3
Θ(…) means both!
• We say “T(n) is Θ(g(n))” if:

T(n) = O(g(n))
-AND-
T(n) = Ω(g(n))
Some more examples
• All degree k polynomials* are O(nk)
• For any k ≥ 1, nk is not O(nk-1)

*Need some caveat here…what is it?

(On the board if we have time…

if not see the lecture notes!)
Take-away from examples
• To prove T(n) = O(g(n)), you have to come up with c
and n0 so that the definition is satisfied.

• To prove T(n) is NOT O(g(n)), one way is proof by

contradiction:
• Suppose (to get a contradiction) that someone gives you
a c and an n0 so that the definition is satisfied.
• Show that this someone must by lying to you by deriving
a contradiction.
 Yet more examples
• n3 + 3n = O(n3 – n2)
Work through any of
• n3 + 3n = Ω(n3 – n2) these that we don’t
• n3 + 3n = Θ(n3 – n2) have time to go
through in class!

• 3n is NOT O(2n)
• log(n) = Ω(ln(n))
• log(n) = Θ( 2loglog(n) )

Siggi the Studious Stork

remember that log = log2 in this class.
Some brainteasers
• Are there functions f, g so that NEITHER f = O(g)
nor f = Ω(g)?
• Are there non-decreasing functions f, g so that
the above is true?
• Define the n’th fibonacci number by F(0) = 1,
F(1) = 1, F(n) = F(n-1) + F(n-2) for n > 2.
• 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, …
True or false:
• F(n) = O(2n)
• F(n) = Ω(2n)

Ollie the Over-achieving Ostrich

 This is my

What have we learned? happy face!

Asymptotic Notation
• This makes both Plucky and Lucky happy.
• Plucky the Pedantic Penguin is happy because
there is a precise definition.
• Lucky the Lackadaisical Lemur is happy because we
don’t have to pay close attention to all those pesky
constant factors like “11”.

• But we should always be careful not to abuse it.

• In the course, (almost) every algorithm we see

will be actually practical, without needing to
take 𝑛 ≥ 𝑛5 = 2B5555555 .
 The plan
• Part I: Sorting Algorithms
• InsertionSort: does it work and is it fast?
• MergeSort: does it work and is it fast?
• Skills:
• Analyzing correctness of iterative and recursive algorithms.
• Analyzing running time of recursive algorithms (part A)

• Part II: How do we measure the runtime of an algorithm?

• Worst-case analysis
• Asymptotic Analysis

Wrap-Up
Recap
• InsertionSort runs in time O(n2)
• MergeSort is a divide-and-conquer algorithm that
runs in time O(n log(n))

• How do we show an algorithm is correct?

• Today, we did it by induction
• How do we measure the runtime of an algorithm?
• Worst-case analysis
• Asymptotic analysis
Next time
• A more systematic approach to analyzing the
runtime of recursive algorithms.

Before next time

• Pre-Lecture Exercise:
• A few recurrence relations (see website)