Sorting

g Review
z Insertion Sort
Introduction to Algorithms z T(n) = Θ(n2)
In-place
Quicksort z
z
Merge
g Sort
z T(n) = Θ(n lg(n))
z Not in-place
CSE 680 z Selection Sort (from homework)
z T(n) = Θ(n2)
Prof. Roger Crawfis z In-place Seems pretty good.
z Heap Sort Can we do better?
z T(n) = Θ(n lg(n))
z In-place

Sorting
g Comparison
p Sorting
g

z Assumptions z Given a set of n values,

values there can be n!
1. No knowledge of the keys or numbers we permutations of these values.
are sorting on
on. z So if we look at the behavior of the
2. Each key supports a comparison interface sorting algorithm over all possible n!
or operator. inputs we can determine the worst-case
worst case
3. Sorting entire records, as opposed to complexity of the algorithm.
numbers,, is an implementation
p detail.
4. Each key is unique (just for convenience).
Comparison Sorting
Decision Tree Decision Tree Model
z Decision tree model ≤ 1:2 >
z Full binary tree
2:3 1:3 >
z A full binary tree (sometimes proper binary tree or 2- ≤ > ≤
tree) is a tree in which every node other than the leaves
h ttwo children
has hild <1,2,3> 1:3 <2,1,3> 2:3
≤ > ≤ >
z Internal node represents a comparison.
z Ignore control, movement, and all other operations, just <1,3,2> <3,1,2> <2,3,1> <3,2,1>
see comparison
z Each leaf represents one possible result (a
permutation of the elements in sorted order). Internal node i:j indicates comparison between ai and aj.
z The height of the tree (i.e.,
(i e longest path) is the suppose three elements < a1, a1 a2,
a2 a3> with instance <6
<6,8,5>
8 5>
lower bound. Leaf node <π(1), π(2), π(3)> indicates ordering aπ(1)≤ aπ(2)≤ aπ(3).
Path of bold lines indicates sorting path for <6,8,5>.
There are total 3!=6 possible permutations (paths).

Decision Tree Model QuickSort Design

g
z The longest path is the worst case number of z Follows the divide-and-conquer paradigm.
comparisons. The length of the longest path is the z Divide: Partition (separate) the array A[p..r] into two
height of the decision tree. (possibly empty) subarrays A[p..q–1] and A[q+1..r].
z Theorem 8.1: Any y comparison
p sort algorithm
g z Each element in A[p..q–1] < A[q].
requires Ω(nlg n) comparisons in the worst case. z A[q] < each element in A[q+1..r].
z Index q is computed as part of the partitioning procedure.
z Proof:
z Conquer: Sort the two subarrays by recursive calls to
z Suppose height of a decision tree is h,
h and number of quicksort.
i k t
paths (i,e,, permutations) is n!.
z Since a binary tree of height h has at most 2h leaves, z Combine: The subarrays are sorted in place – no
z n! ≤ 2h , so h ≥ lg
g ((n!)) ≥ Ω(nlg
( g n)) ((By
y equation
q 3.18).
) work is needed to combine them.
z That is to say: any comparison sort in the worst z How do the divide and combine steps of quicksort
case needs at least nlg n comparisons. compare with those of merge sort?
Pseudocode Example
p
p r
Quicksort(A, p, r)
initially: 2 5 8 3 9 4 1 7 10 6 note: pivot (x) = 6
if p < r then Partition(A,
P titi (A p, r)) i j
q := Partition(A, p, r); x, i := A[r], p – 1;
Quicksort(A, p, q – 1); for j := p to r – 1 do next iteration: 2 5 8 3 9 4 1 7 10 6
Quicksort(A, q + 1, r) if A[j] ≤ x then i j Partition(A, p
Partition(A p, r)
i := i + 1; x, i := A[r], p – 1;
A[p..r] A[i] ↔ A[j] next iteration: 2 5 8 3 9 4 1 7 10 6 for j := p to r – 1 do
A[i + 1] ↔ A[r]; i j if A[j] ≤ x then
5 return i + 1 i := i + 1;
next iteration: 2 5 8 3 9 4 1 7 10 6 A[i] ↔ A[j]
A[p..q – 1] A[q+1..r] i j A[i[ + 1]] ↔ A[r];
[ ];
Partition return i + 1
5 next iteration: 2 5 3 8 9 4 1 7 10 6
i j
≤5 ≥5

Example
p ((Continued)) Partitioning
g
next iteration: 2 5 3 8 9 4 1 7 10 6 z Select the last element A[r] in the subarray
i j A[p..r] as the pivot – the element around which
next iteration: 2 5 3 8 9 4 1 7 10 6 to partition.
i j
next iteration: 2 5 3 4 9 8 1 7 10 6 Partition(A, p
Partition(A p, r) z As the pprocedure executes,, the arrayy is
i j x, i := A[r], p – 1; partitioned into four (possibly empty) regions.
next iteration: 2 5 3 4 1 8 9 7 10 6 for j := p to r – 1 do 1. A[p..i ] — All entries in this region are < pivot.
i j if A[j] ≤ x then 2. A[i+1..j
A[i 1..j – 1] — All entries in this region are > pivot.
next iteration: 2 5 3 4 1 8 9 7 10 6 i := i + 1; 3. A[r] = pivot.
i j A[i] ↔ A[j] 4. A[j..r – 1] — Not known how they compare to pivot.
next iteration: 2 5 3 4 1 8 9 7 10 6 A[i[ + 1]] ↔ A[r];
[] z The above hold before each iteration of the for
i j return i + 1
loop, and constitute a loop invariant. (4 is not part
after final swap: 2 5 3 4 1 6 9 7 10 8 of the loopi.)
i j
Correctness of Partition Correctness of Partition
z Use loop invariant. Case 1:
z I iti li ti
Initialization:
p i j r
z Before first iteration
>x x
A[p..i]
A[p i] and A[i+1..j
z A[i+1 j – 1] are empty – Conds.
Conds 1 and 2 are satisfied
(trivially).
z r is the index of the pivot Partition(A, p, r) ≤x >x
z Cond.
Cond 3 is satisfied
satisfied. x i ::= A[r],
x, A[r] p – 1; p i j r
for j := p to r – 1 do
z Maintenance: if A[j] ≤ x then
x

z Case 1: A[j] > x i := i + 1;;

A[i] ↔ A[j] ≤x >x
z Increment j only.
A[i + 1] ↔ A[r];
z Loop Invariant is maintained.
return i + 1

Correctness of Partition Correctness of Partition

z [j] ≤ x
Case 2: A[j] z Increment j z Termination:
z Increment i z Condition 2 is z When the loop terminates, j = r, so all elements
maintained.
z Swap A[i] and A[j] in A are partitioned into one of the three cases:
z A[r] is unaltered.
z Condition 1 is z i] ≤ pivot
A[p..i]
A[p
maintained. z Condition 3 is
maintained.
z A[i+1..j – 1] > pivot
p i j r z A[r] = pivot
≤x x z The last two lines swap A[i+1] and A[r].
A[r]
z Pivot moves from the end of the array to
≤x >x between the two subarrays.
p i j r z Thus, procedure partition correctly performs
x the divide step.

≤x >x
Complexity
p y of Partition Quicksort Overview

z PartitionTime(n) is given by the number z T sortt a[left...right]:

To [l ft i ht]
of iterations in the for loop. 1. if left < right:
z Θ(n) : n = r – p + 11. 1 1 Partition
1.1. P titi a[left...right]
[l ft i ht] such
h that:
th t
Partition(A, p, r)
x, i := A[r], p – 1; all a[left...p-1] are less than a[p], and
for j :=
: p to r – 1 do
if A[j] ≤ x then all a[p+1...right] are >= a[p]
i := i + 1;
[ ] ↔ A[j]
A[i] [j]
1.2. Quicksort a[left...p-1]
A[i + 1] ↔ A[r]; 1.3. Quicksort a[p+1...right]
return i + 1
2. Terminate

Partitioning
g in Quicksort Alternative Partitioning
g

z A key step in the Quicksort algorithm is z Choose an array y value ((say,

y, the first)) to use
partitioning the array as the pivot
z Starting from the left end, find the first
z We choose some ((any) y) number p in the
array to use as a pivot
element that is greater than or equal to the
pivot
z We partition the array into three parts:
z Searching backward from the right end, find
the first element that is less than the pivot
p z Interchange (swap) these two elements
z Repeat, searching from where we left off,
p
numbers less numbers greater than or until done
than p equal to p
Alternative Partitioning
g Example
p of p
partitioning
g
z choose pivot: 436924312189356
z T partition
To titi a[left...right]:
[l ft i ht] z search: 436924312189356
1. Set pivot = a[left], l = left + 1, r = right; z swap: 433924312189656
2 while l < r,
2. r do
z search: 433924312189656
2.1. while l < right & a[l] < pivot , set l = l + 1 z swap: 433124312989656
2 2 while
2.2. hil r > lleft
ft & a[r]
[ ] >= pivot
i t , sett r = r - 1 z search: 433124312989656
2.3. if l < r, swap a[l] and a[r] z swap: 433122314989656
3. Set
3 S t a[left]
[l ft] = a[r],
[ ] a[r]
[ ] = pivot
i t
z search: 433122314989656
4. Terminate
z swap with pivot: 133122344989656

Partition Implementation (Java) Quicksort Implementation (Java)

static int Partition(int[] a, int left, int right) { static void QQuicksort(int[]
( [] array,
y, int left,, int right)
g )
i t p = a[left],
int [l ft] l = lleft
ft + 1
1, r = right;
i ht
while (l < r) { {
while (l < right && a[l] < p) l++; if (left < right) {
while (r > left && a[r] >= p) r--; r ; int p = Partition(array,
Partition(array left,
left right);
if (l < r) { Quicksort(array, left, p - 1);
int temp = a[l]; a[l] = a[r]; a[r] = temp; Quicksort(array, p + 1, right);
}
} }
a[left] = a[r]; }
[ ] = p;
a[r]
return r;
}
Analysis of quicksort—best case Partitioning
g at various levels

z Suppose each partition operation

divides the array almost exactly in half
z Then the depth of the recursion in log2n
z Because that’s how many times we can
halve n
z We note that
z Each partition is linear over its subarray
z All the partitions at one level cover the
array

Best Case Analysis

y Worst case

z We cut the array size in half each time z In the worst case, partitioning always
z So the depth of the recursion in log2n divides the size n array into these three
z At each level of the recursion
recursion, all the parts:
partitions at that level do work that is linear z A length one part, containing the pivot itself
in n z A length zero part, and
z th n-1
A length
l 1 part,
t containing
t i i everything
thi elsel
z O(log2n) * O(n) = O(n log2n)
z We don’t recur on the zero-length part
z Hence in the best case, quicksort has time
complexity O(n log2n) z R
Recurring
i on th th n-1
the llength n 1 partt requires
i
(in the worst case) recurring to depth n-1
z What about the worst case?
Worst case partitioning
p g Worst case for quicksort
q
z In the worst case, recursion may be n levels deep
i n))
(f an array off size
(for
z But the partitioning work done at each level is still n
z O(n) * O(n) = O(n2)
z So worst case for Quicksort is O(n2)
z When does this happen?
z There are many arrangements that could make this
happen
z Here are two common cases:
z When the array is already sorted
z When the array is inversely sorted (sorted in the opposite
order)

Typical
yp case for q
quicksort Picking
g a better p
pivot
z If the array is sorted to begin with, z Before, we ppicked the first element of the
Quicksort is terrible: O(n2) subarray
b t use as a pivot
to i t
z If the array is already sorted, this results in
z It is p
possible to construct other bad cases O(n2) behavior
z It’s no better if we pick the last element
z However, Quicksort is usually O(n log2n) z We could do an optimal quicksort
z The constants are so good that Quicksort is (guaranteed O(n log n)) if we always picked
generally the faster algorithm. a pivot value that exactly cuts the array in
half
z Most real-world
real world sorting is done by z Such a value is called a median: half of the
values in the array are larger, half are smaller
Quicksort z The easiest way to find the median is to sort
the array
y and ppick the value in the middle ((!))
Median of three Quicksort for Small Arrays
y

z Obviously, it doesn’t
doesn t make sense to sort the z For very small arrays (N<= 20)
20), quicksort
array in order to find the median to use as a does not perform as well as insertion sort
p
pivot.
z A good cutoff range is N=10
z Instead, compare just three elements of our
z Switching to insertion sort for small
((sub)array—the
) y first,, the last,, and the middle
z Take the median (middle value) of these three as
arrays can save about
b t 15% iin th the
the pivot running time
z It’s possible (but not easy) to construct cases which
will make this technique O(n2)

Mergesort
g vs Quicksort Mergesort
g vs Quicksort

z Both run in O(n lgn) In C++

C++, copying objects can be expensive
z Compared with Quicksort, Mergesort has while comparing objects often is
less number of comparisons but larger relatively cheap
cheap. Therefore
Therefore, quicksort is
number of moving elements the sorting routine commonly used in
z In Java, an element comparison is C++ libraries
expensive but moving elements is
cheap.p Therefore, Mergesort
g is used in
the standard Java library for generic
sorting