Central Algorithmic Techniques

Iterative Algorithms
COSC 3101, PROF. J. ELDER
2
Code
Representation of an Algorithm
class InsertionSortAlgorithm extends SortAlgorithm {
void sort(int a[]) throws Exception {
for (int i = 1; i < a.length; i++) {
int j = i;
int B = a[i];
while ((j > 0) && (a[j-1] > B)) {
a[j] = a[j-1];
j--; }
a[j] = B;
}}
Pros and Cons?
COSC 3101, PROF. J. ELDER
3
Code
Representation of an Algorithm
Runs on computers
Precise and succinct
I am not a computer
I need a higher level of
intuition.
Prone to bugs
Language dependent
Pros: Cons:
COSC 3101, PROF. J. ELDER
4
Two Key Types of Algorithms
Iterative Algorithms
Recursive Algorithms
COSC 3101, PROF. J. ELDER
5
Iterative Algorithms
Take one step at a time
towards the final destination
loop (done)
take step
end loop
COSC 3101, PROF. J. ELDER
6
Loop Invariants
A good way to structure many programs:
Store the key information you currently know in some
data representation.
In the main loop,
take a step forward towards destination
by making a simple change to this data.
COSC 3101, PROF. J. ELDER
7
The Getting to School Problem
COSC 3101, PROF. J. ELDER
8
Problem Specification
Pre condition: location of home and school
Post condition: Traveled from home to school
COSC 3101, PROF. J. ELDER
9
General Principle
Do not worry about the entire computation.
Take one step at a time!
COSC 3101, PROF. J. ELDER
10
A Measure of Progress
79 km
to school
75 km
to school
COSC 3101, PROF. J. ELDER
11
Algorithm specifies from which locations
it knows how to step.
Safe Locations
COSC 3101, PROF. J. ELDER
12
The computation is presently in a safe location.
May or may not be true.
Loop Invariant
COSC 3101, PROF. J. ELDER
13
Defining Algorithm
From every safe location,
define one step towards school.
COSC 3101, PROF. J. ELDER
14
Take a step
What is required of this step?
COSC 3101, PROF. J. ELDER
15
Can we be assured that the computation
will always be in a safe location?
If the computation is in a safe location,
it does not step into an unsafe one.
Maintain Loop Invariant
No. What if it is not initially true?
COSC 3101, PROF. J. ELDER
16
From the Pre-Conditions on the input instance
we must establish the loop invariant.
Establishing Loop Invariant
COSC 3101, PROF. J. ELDER
17
Maintain Loop Invariant
Can we be assured that the
computation will always be
in a safe location?
By what principle?
COSC 3101, PROF. J. ELDER
18
Maintain Loop Invariant
By Induction the computation will
always be in a safe location.
(0)
, ( )
, ( ) ( 1)
S
i S i
i S i S i

+
)
COSC 3101, PROF. J. ELDER
19
Ending The Algorithm
Define Exit Condition

Termination: With sufficient progress,
the exit condition will be met.

When we exit, we know
exit condition is true
loop invariant is true
from these we must establish
the post conditions.
Exit
Exit
0 km Exit
COSC 3101, PROF. J. ELDER
20
Lets Recap
COSC 3101, PROF. J. ELDER
21
Designing an Algorithm
Define Problem Define Loop Invariants
Define Measure of
Progress
Define Step Define Exit Condition Maintain Loop Inv
Make Progress Initial Conditions Ending
km
79 km
to school
Exit
Exit
79 km 75 km
Exit
Exit
0 km Exit
Simple Example
Insertion Sort Algorithm
COSC 3101, PROF. J. ELDER
23
Code
Representation of an Algorithm
class InsertionSortAlgorithm extends SortAlgorithm {
void sort(int a[]) throws Exception {
for (int i = 1; i < a.length; i++) {
int j = i;
int B = a[i];
while ((j > 0) && (a[j-1] > B)) {
a[j] = a[j-1];
j--; }
a[j] = B;
}}
COSC 3101, PROF. J. ELDER
24
Higher Level Abstract View
Representation of an Algorithm
9 km
5 km
COSC 3101, PROF. J. ELDER
25
Designing an Algorithm
Define Problem Define Loop Invariants
Define Measure of
Progress
Define Step Define Exit Condition Maintain Loop Inv
Make Progress Initial Conditions Ending
km
79 km
to school
Exit
Exit
79 km 75 km
Exit
Exit
0 km Exit
COSC 3101, PROF. J. ELDER
26
Problem Specification
Precondition: The input is a list of n values
with the same value possibly repeated.
Post condition: The output is a list consisting
of the same n values in non-decreasing order.
88
14
98
25
62
52
79
30
23
31
14,23,25,30,31,52,62,79,88,98
COSC 3101, PROF. J. ELDER
27
88
14
98
25
62
52
79
30
23
31
14,23,25,30,31,52,62,79,88,98
14
98
25
62
79
30
23,31,52,88
Some subset of the elements are sorted
The remaining elements are off to the side.
Define Loop Invariant
COSC 3101, PROF. J. ELDER
28
Defining Measure of Progress
14
98
25
62
79
30
23,31,52,88
6 elements
to school
COSC 3101, PROF. J. ELDER
29
Define Step
Select arbitrary element from side.
Insert it where it belongs.
98
25
62
79
23,31,52,88
14
98
25
79
30
23,31,52,62,88
14 30
COSC 3101, PROF. J. ELDER
30
Making progress while Maintaining the
loop invariant
98
25
62
79
23,31,52,88
14
98
25
79
30
23,31,52,62,88
14 30
79 km 75 km
5 elements
to school
6 elements
to school
Exit
Sorted sublist
COSC 3101, PROF. J. ELDER
31
88
14
98
25
62
52
79
30
23
31
88
14
98
25
62
52
79
30
23
31
n elements
to school
14,23,25,30,31,52,62,79,88,98
14,23,25,30,31,52,62,79,88,98
0 elements
to school
Beginning &
Ending
km
Exit
0 km Exit
COSC 3101, PROF. J. ELDER
32
Running Time
Inserting an element into a list of size i
takes u(i) time.
Total = 1+2+3++n = u(n
2
)
98
25
62
79
23,31,52,88
14
98
25
79
30
23,31,52,62,88
14 30
COSC 3101, PROF. J. ELDER
33
Ok
I know you knew Insertion Sort
But hopefully you are beginning to appreciate
Loop Invariants
for describing algorithms

Assertions
in Algorithms
COSC 3101, PROF. J. ELDER
35
Purpose of Assertions
Useful for
thinking about algorithms
developing
describing
proving correctness

COSC 3101, PROF. J. ELDER
36
Definition of Assertions
An assertion is a statement about the
current state of the data structure that is
either true or false.

eg. the amount in your bank account is not
negative.
COSC 3101, PROF. J. ELDER
37
Definition of Assertions
It is made at some particular point during the
execution of an algorithm.
If it is false, then something has gone wrong in the
logic of the algorithm.
COSC 3101, PROF. J. ELDER
38
Definition of Assertions
An assertion is not a task for the algorithm to
perform.
It is only a comment that is added for the
benefit of the reader.

COSC 3101, PROF. J. ELDER
39
Specifying a Computational Problem
Example of Assertions
Preconditions: Any assumptions that must be
true about the input instance.
Postconditions: The statement of what must
be true when the algorithm/program returns..

COSC 3101, PROF. J. ELDER
40
Definition of Correctness
<PreCond> & <code> <PostCond>

If the input meets the preconditions,
then the output must meet the postconditions.

If the input does not meet the preconditions, then
nothing is required.

COSC 3101, PROF. J. ELDER
41
An Example:
A Sequence of Assertions
<assertion
0
>
if( <condition
1
> ) then
code
<1,true>
else
code
<1,false>
end if
<assertion
1
>

<assertion
r-1
>
if( <condition
r
> ) then
code
<r,true>
else
code
<r,false>
end if
<assertion
r
>

<assertion
0
>
any <conditions>
code

<assertion
r
>
How is this proved?
Definition of Correctness
Must check 2
r
different
settings of <conditions>
paths through the code.
Is there a faster way?
COSC 3101, PROF. J. ELDER
42
An Example:
A Sequence of Assertions
<assertion
0
>
if( <condition
1
> ) then
code
<1,true>
else
code
<1,false>
end if
<assertion
1
>

<assertion
r-1
>
if( <condition
r
> ) then
code
<r,true>
else
code
<r,false>
end if
<assertion
r
>

Step 1
<assertion
0
>
<condition
1
>
code
<1,true>
<assertion
1
>
<assertion
0
>
<condition
1
>
code
<1,false>
<assertion
1
>
COSC 3101, PROF. J. ELDER
43
An Example:
A Sequence of Assertions
<assertion
0
>
if( <condition
1
> ) then
code
<1,true>
else
code
<1,false>
end if
<assertion
1
>

<assertion
r-1
>
if( <condition
r
> ) then
code
<r,true>
else
code
<r,false>
end if
<assertion
r
>

Step 2
<assertion
1
>
<condition
2
>
code
<2,true>
<assertion
2
>
<assertion
1
>
<condition
2
>
code
<2,false>
<assertion
2
>
COSC 3101, PROF. J. ELDER
44
An Example:
A Sequence of Assertions
<assertion
0
>
if( <condition
1
> ) then
code
<1,true>
else
code
<1,false>
end if
<assertion
1
>

<assertion
r-1
>
if( <condition
r
> ) then
code
<r,true>
else
code
<r,false>
end if
<assertion
r
>

Step r
<assertion
r-1
>
<condition
r
>
code
<r,true>
<assertion
r
>
<assertion
r-1
>
<condition
r
>
code
<r,false>
<assertion
r
>
COSC 3101, PROF. J. ELDER
45
A Sequence of Assertions
<assertion
0
>
if( <condition
1
> ) then
code
<1,true>
else
code
<1,false>
end if
<assertion
1
>

<assertion
r-1
>
if( <condition
r
> ) then
code
<r,true>
else
code
<r,false>
end if
<assertion
r
>


Step r
<assertion
r-1
>
<condition
r
>
code
<r,true>
<assertion
r
>
<assertion
r-1
>
<condition
r
>
code
<r,false>
<assertion
r
>

COSC 3101, PROF. J. ELDER
46
Another Example:
A Loop
<preCond>
codeA
loop
<loop-invariant>
exit when <exit Cond>
codeB
endloop
codeC
<postCond>

Type of Algorithm:
Iterative
Type of Assertion:
Loop Invariants
COSC 3101, PROF. J. ELDER
47
<preCond>
codeA
loop
<loop-invariant>
exit when <exit Cond>
codeB
endloop
codeC
<postCond>

Definition of Correctness?
Iterative Algorithms
Loop Invariants
COSC 3101, PROF. J. ELDER
48
<preCond>
codeA
loop
<loop-invariant>
exit when <exit Cond>
codeB
endloop
codeC
<postCond>

Definition of Correctness
<preCond>
any <conditions>
code
<postCond>
How is this proved?
Iterative Algorithms
Loop Invariants
COSC 3101, PROF. J. ELDER
49
<preCond>
codeA
loop
<loop-invariant>
exit when <exit Cond>
codeB
endloop
codeC
<postCond>

The computation may go around
the loop an arbitrary number of times.
<preCond>
any <conditions>
code
<postCond>
Is there a faster way?
Iterative Algorithms
Loop Invariants
Definition of Correctness
COSC 3101, PROF. J. ELDER
50
<preCond>
codeA
loop
<loop-invariant>
exit when <exit Cond>
codeB
endloop
codeC
<postCond>

Step 0
<preCond>
codeA

<loop-invariant>
Iterative Algorithms
Loop Invariants
COSC 3101, PROF. J. ELDER
51
<preCond>
codeA
loop
<loop-invariant>
exit when <exit Cond>
codeB
endloop
codeC
<postCond>

Step 1
<loop-invariant>
<exit Cond>
codeB
<loop-invariant>

Iterative Algorithms
Loop Invariants
COSC 3101, PROF. J. ELDER
52
<preCond>
codeA
loop
<loop-invariant>
exit when <exit Cond>
codeB
endloop
codeC
<postCond>

Step 2
<loop-invariant>
<exit Cond>
codeB
<loop-invariant>

Iterative Algorithms
Loop Invariants

COSC 3101, PROF. J. ELDER
53
<preCond>
codeA
loop
<loop-invariant>
exit when <exit Cond>
codeB
endloop
codeC
<postCond>

Step 3
<loop-invariant>
<exit Cond>
codeB
<loop-invariant>

Iterative Algorithms
Loop Invariants
COSC 3101, PROF. J. ELDER
54
<preCond>
codeA
loop
<loop-invariant>
exit when <exit Cond>
codeB
endloop
codeC
<postCond>

Step i
<loop-invariant>
<exit Cond>
codeB
<loop-invariant>

Iterative Algorithms
Loop Invariants
All these steps are the same
and therefore only need be done once!
COSC 3101, PROF. J. ELDER
55
<preCond>
codeA
loop
<loop-invariant>
exit when <exit Cond>
codeB
endloop
codeC
<postCond>

Last Step
<loop-invariant>
<exit Cond>
codeC
<postCond>

Iterative Algorithms
Loop Invariants
COSC 3101, PROF. J. ELDER
56
Partial Correctness
Proves that IF the program terminates then it works
<PreCond> & <code> <PostCond>
<loop-invariant>
<exit Cond>
codeC
<postCond>

Clean up loose ends

<loop-invariant>
<exit Cond>
codeB
<loop-invariant>

Maintaining Loop Invariant
<preCond>
codeA

<loop-invariant>
Establishing Loop Invariant

Exit
Exit
COSC 3101, PROF. J. ELDER
57
Algorithm Termination
km
79 km 75 km
Measure of progress
0 km
Exit
COSC 3101, PROF. J. ELDER
58
Algorithm Correctness
Partial Correctness
+ Termination
Correctness
COSC 3101, PROF. J. ELDER
59
Designing Loop Invariants
Coming up with the loop invariant is the hardest part of
designing an algorithm.
It requires practice, perseverance, and insight.
Yet from it
the rest of the algorithm
follows easily
COSC 3101, PROF. J. ELDER
60
Dont start coding
You must design a working algorithm first.
COSC 3101, PROF. J. ELDER
61
Exemplification:
Try solving the problem
on small input examples.
COSC 3101, PROF. J. ELDER
62
Start with Small Steps
What basic steps might you follow to make some kind of
progress towards the answer?
Describe or draw a picture of what the data structure might
look like after a number of these steps.

COSC 3101, PROF. J. ELDER
63
Picture from the Middle
Leap into the middle of the algorithm.

What would you like your data structure to look like
when you are half done?

COSC 3101, PROF. J. ELDER
64
Ask for 100%
Pretend that a genie has granted your wish.
You are now in the middle of your computation and your
dream loop invariant is true.

COSC 3101, PROF. J. ELDER
65
Ask for 100%
Maintain the Loop Invariant:
From here, are you able to take some computational steps that
will make progress while maintaining the loop invariant?


COSC 3101, PROF. J. ELDER
66
Ask for 100%
If you can maintain the loop invariant, great.
If not,
Too Weak: If your loop invariant is too weak, then the genie has
not provided you with everything you need to move on.
Too Strong: If your loop invariant is too strong, then you will not
be able to establish it initially or maintain it.

COSC 3101, PROF. J. ELDER
67
Differentiating between Iterations
x=x+2
Meaningful as code
False as a mathematical statement

x = x
i
= value at the beginning of the iteration
x = x
i+1
= new value after going around the loop one more time.
x'' = x'+2
Meaningful as a mathematical statement

Loop Invariants
for
Iterative Algorithms
Three
Search Examples
COSC 3101, PROF. J. ELDER
69
Define Problem: Binary Search
PreConditions
Key 25
Sorted List


PostConditions
Find key in list (if there).


3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
COSC 3101, PROF. J. ELDER
70
Define Loop Invariant
Maintain a sublist.
If the key is contained in the original list, then the key is
contained in the sublist.









key 25
3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
COSC 3101, PROF. J. ELDER
71
Define Step
Make Progress
Maintain Loop Invariant
key 25
3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
COSC 3101, PROF. J. ELDER
72
Define Step
Cut sublist in half.
Determine which half the key would be in.
Keep that half.
key 25
3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
If key mid,
then key is in
left half.
If key > mid,
then key is in
right half.
mid
COSC 3101, PROF. J. ELDER
73
Define Step
It is faster not to check if the middle element is the key.
Simply continue.
key 43
3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
If key mid,
then key is in
left half.
If key > mid,
then key is in
right half.
COSC 3101, PROF. J. ELDER
74
Make Progress
The size of the list becomes smaller.








3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
79 km
75 km
79 km 75 km
Exit
COSC 3101, PROF. J. ELDER
75
Initial Conditions
km
key 25
3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
If the key is contained in the
original list,
then the key is contained in the
sublist.
The sublist is the
entire original list.
n km
COSC 3101, PROF. J. ELDER
76
Ending Algorithm
If the key is contained in the
original list,
then the key is contained in the
sublist.
Sublist contains one element.

Exit
Exit
3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
0 km
If the key is
contained in the
original list,
then the key is at
this location.

key 25
COSC 3101, PROF. J. ELDER
77
If key not in original list
If the key is contained in the
original list,
then the key is contained in the
sublist.

Loop invariant true,
even if the key is not
in the list.

If the key is contained in
the original list, then the
key is at this location.

3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
key 24
Conclusion still solves the
problem.
Simply check this one location
for the key.

COSC 3101, PROF. J. ELDER
78
Running Time
The sublist is of size n,
n
/
2
,
n
/
4
,
n
/
8
,,1
Each step u(1) time.
Total = u(log n)
key 25
3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
If key mid,
then key is in
left half.
If key > mid,
then key is in
right half.
COSC 3101, PROF. J. ELDER
79
<precondition>: A[1..n] is sorted in non-decreasing order
<postcondition>: If is in A[1..n], algorithm returns
1,
its location
loop-invariant>: If is
BinarySea
in
rch(A[1..n],
whil
)
e
p q
key
key
q p
e
n
k y
<
>
= =
if [ ]
els
2
1
return( )
return("Key n
A[1..n], then
e
end
end
if [ ]
end
is in A[p..
ot in list")
q]
p q
mid
q mid
p mi
key A m
key
id
key A p
e
d
p
lse
s
+
(
=
(

=
= +
=
COSC 3101, PROF. J. ELDER
80
Algorithm Definition Completed
Define Problem Define Loop Invariants
Define Measure of
Progress
Define Step Define Exit Condition Maintain Loop Inv
Make Progress Initial Conditions Ending
km
79 km
to school
Exit
Exit
79 km 75 km
Exit
Exit
0 km Exit
COSC 3101, PROF. J. ELDER
81
<precondition>: A[1..n] is sorted in non-decreasing order
<postcondition>: If is in A[1..n], algorithm returns
1,
its location
loop-invariant>: If is
BinarySea
in
rch(A[1..n],
whil
)
e
p q
key
key
q p
e
n
k y
<
>
= =
if [ ]
els
2
1
return( )
return("Key n
A[1..n], then
e
end
end
if [ ]
end
is in A[p..
ot in list")
q]
p q
mid
q mid
p mi
key A m
key
id
key A p
e
d
p
lse
s
+
(
=
(

=
= +
=
COSC 3101, PROF. J. ELDER
82
Simple, right?
Although the concept is simple, binary search is
notoriously easy to get wrong.
Why is this?
COSC 3101, PROF. J. ELDER
83
The Devil in the Details
The basic idea behind binary search is easy to grasp.
It is then easy to write pseudocode that works for a
typical case.
Unfortunately, it is equally easy to write pseudocode that
fails on the boundary conditions.
COSC 3101, PROF. J. ELDER
84
1
if [ ]
else
end
q mid
p
key A mid
mid
=
=
s
+
The Devil in the Details
1
if [ ]
else
end
q mid
p
key A mid
mid
=
=
<
+
or
What condition will break the loop invariant?
COSC 3101, PROF. J. ELDER
85
The Devil in the Details
key 36
3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
mid
s C e od le k c ey t A[m rig e h id] t lf : ha >
Bug!!
COSC 3101, PROF. J. ELDER
86
1
if [ ]
else
end
q mid
p
key A mid
mid
=
=
s
+
The Devil in the Details
1
if [ ]
else
end
q mid
p
key A mid
mid
=
=
<
+
if < [ ]
else
end
1 q mid
p
key A mid
mid
=
=
OK OK Not OK!!
COSC 3101, PROF. J. ELDER
87
key 25
3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
The Devil in the Details
mid
2
+
(
=
(

p q
mid
2
+
(
=
(
(
p q
or
Shouldnt matter, right? Select mid
2
p q +
(
=
(
(
COSC 3101, PROF. J. ELDER
88
6 74
The Devil in the Details
key 25
95 91 88 83 72 60 53 51 49 43 36 25 21 21 18 13 5 3
If key mid,
then key is in
left half.
If key > mid,
then key is in
right half.
mid
COSC 3101, PROF. J. ELDER
89
25 18 74
The Devil in the Details
key 25
95 91 88 83 72 60 53 51 49 43 36 21 21 13 6 5 3
If key mid,
then key is in
left half.
If key > mid,
then key is in
right half.
mid
COSC 3101, PROF. J. ELDER
90
25 13 74
The Devil in the Details
key 25
95 91 88 83 72 60 53 51 49 43 36 21 21 18 6 5 3
If key mid,
then key is in
left half.
If key > mid,
then key is in
right half.
Another bug!
No progress
toward goal:
Loops Forever!
mid
79 km 75 km
Exit
COSC 3101, PROF. J. ELDER
91
if [
mid
]
2
1
else
end
key A mi
p q
q mid
p mid
d
+
(
=
(

= +
s
=
The Devil in the Details
if [
mid
]
2
1
else
end
key A mi
p q
q mid
p mid
d
+
(
=
(
(
= +
s
=
if < [
mid
2
1
]
else
end
key A mid
p q
q mid
p mid
+
(
=
(
(
=
=
OK OK Not OK!!
COSC 3101, PROF. J. ELDER
92
if [
mid
]
2
1
else
end
key A mi
p q
q mid
p mid
d
+
(
=
(

= +
s
=
How Many Possible Algorithms?
mid r
2
o ?
p q +
(
=
(
(
if < [ or ? ] key A mid
else
o
end
1 r q mid
p mid
=
=
COSC 3101, PROF. J. ELDER
93
<precondition>: A[1..n] is sorted in non-decreasing order
<postcondition>: If is in A[1..n], algorithm returns
1,
its location
loop-invariant>: If is
BinarySea
in
rch(A[1..n],
whil
)
e
p q
key
key
q p
e
n
k y
<
>
= =
2
retur
A[1..n
n(mid)
], then
if = [ ]
elseif <
is
[
in A[p
]
1
1
return( )
return("Key
e
not i
lse
n
..q
end
end
if
list"
[ ]
end
)
]
key A mid
key A
p q
mid
q mid
mid
key A p
el
key
se
p mid
p
+
(
=
(

=
= +
=
Alternative Algorithm: Less Efficient but More Clear
COSC 3101, PROF. J. ELDER
94
Moral
Use the loop invariant method to think about algorithms.
Be careful with your definitions.
Be sure that the loop invariant is always maintained.
Be sure progress is always made.
Having checked the typical cases, pay particular
attention to boundary conditions and the end game.
Loop Invariants
for
Iterative Algorithms
A Second
Search Example:
The Binary Search Tree
COSC 3101, PROF. J. ELDER
96
38
25
17
4 21
31
28 35
51
42
40 49
63
55 71
Define Problem: Binary Search Tree
PreConditions
Key 25
A binary search tree.





PostConditions
Find key in BST (if there).


COSC 3101, PROF. J. ELDER
97
38
25
17
4 21
31
28 35
51
42
40 49
63
55 71
Binary Search Tree
All nodes in left subtree Any node All nodes in right subtree





COSC 3101, PROF. J. ELDER
98
Define Loop Invariant
Maintain a sub-tree.
If the key is contained in the original tree, then the key
is contained in the sub-tree.









key 17
38
25
17
4 21
31
28 35
51
42
40 49
63
55 71
COSC 3101, PROF. J. ELDER
99
Define Step
Cut sub-tree in half.
Determine which half the key would be in.
Keep that half.









key 17
38
25
17
4 21
31
28 35
51
42
40 49
63
55 71
If key < root,
then key is
in left half.
If key > root,
then key is
in right half.
If key = root,
then key is
found
COSC 3101, PROF. J. ELDER
100
Algorithm Definition Completed
Define Problem Define Loop Invariants
Define Measure of
Progress
Define Step Define Exit Condition Maintain Loop Inv
Make Progress Initial Conditions Ending
km
79 km
to school
Exit
Exit
79 km 75 km
Exit
Exit
0 km Exit
End of Lecture 4
Sept 20, 2007
COSC 3101, PROF. J. ELDER
102
A volunteer, please.
Card Trick
Loop Invariants
for
Iterative Algorithms
A Third
Search Example:
A Card Trick
COSC 3101, PROF. J. ELDER
104

Pick a Card
Done
COSC 3101, PROF. J. ELDER
105
Loop Invariant:
The selected card is one
of these.
COSC 3101, PROF. J. ELDER
106

Which
column?
left
COSC 3101, PROF. J. ELDER
107
Loop Invariant:
The selected card is one
of these.
COSC 3101, PROF. J. ELDER
108
Selected column is placed
in the middle
COSC 3101, PROF. J. ELDER
109
I will rearrange the cards
COSC 3101, PROF. J. ELDER
110
Relax Loop Invariant:
I will remember the same
about each column.
COSC 3101, PROF. J. ELDER
111

Which
column?
right
COSC 3101, PROF. J. ELDER
112
Loop Invariant:
The selected card is one
of these.
COSC 3101, PROF. J. ELDER
113
Selected column is placed
in the middle
COSC 3101, PROF. J. ELDER
114
I will rearrange the cards
COSC 3101, PROF. J. ELDER
115

Which
column?
left

COSC 3101, PROF. J. ELDER
116
Loop Invariant:
The selected card is one
of these.
COSC 3101, PROF. J. ELDER
117
Selected column is placed
in the middle
COSC 3101, PROF. J. ELDER
118
Here is your
card.
Wow!

COSC 3101, PROF. J. ELDER
119
Algorithm Definition Completed
Define Problem Define Loop Invariants
Define Measure of
Progress
Define Step Define Exit Condition Maintain Loop Inv
Make Progress Initial Conditions Ending
km
79 km
to school
Exit
Exit
79 km 75 km
Exit
Exit
0 km Exit
COSC 3101, PROF. J. ELDER
120
Ternary Search
How many iterations are required to guarantee
success?
Loop Invariant: selected card in central subset of
cards
1
Size of subset = / 3
where
total number of cards
iteration index
i
n
n
i

(
(
=
=
Loop Invariants
for
Iterative Algorithms
A Fourth Example:
Partitioning
(Not a search problem:
can be used for sorting, e.g., Quicksort)
COSC 3101, PROF. J. ELDER
122
The Partitioning Problem
88
14
98
25
62
52
79
30
23
31
Input:
14
25
30
23
31
88
98
62
79
52
Output:
x=52
Problem: Partition a list into a set of small values and a set of large values.
COSC 3101, PROF. J. ELDER
123
Precise Specification
[ ... ] is an arbitrary list of values. [ ] is the pivo Precondit . i t on: A p r x A r =
p
r
is rearranged such that [ ... 1] [ ] [ 1... ]
for some q.
Postcondition: A A p q A q x A q r s = s +
p
r
q
COSC 3101, PROF. J. ELDER
124
3 subsets are maintained
One containing values less
than or equal to the pivot
One containing values
greater than the pivot
One containing values yet
to be processed
Loop Invariant
COSC 3101, PROF. J. ELDER
125
Maintaining Loop Invariant
Consider element at location j
If greater than pivot, incorporate into
> set by incrementing j.
If less than or equal to pivot,
incorporate into set by swapping
with element at location i+1 and
incrementing both i and j.

Measure of progress: size of unprocessed set.
COSC 3101, PROF. J. ELDER
126
Maintaining Loop Invariant
COSC 3101, PROF. J. ELDER
127
Establishing Loop Invariant
COSC 3101, PROF. J. ELDER
128
Establishing Postcondition
on exit j =
Exhaustive on exit
COSC 3101, PROF. J. ELDER
129
Establishing Postcondition
COSC 3101, PROF. J. ELDER
130
An Example
COSC 3101, PROF. J. ELDER
131
Running Time
Each iteration takes u(1) time Total = u(n)
or
COSC 3101, PROF. J. ELDER
132
Algorithm Definition Completed
Define Problem Define Loop Invariants
Define Measure of
Progress
Define Step Define Exit Condition Maintain Loop Inv
Make Progress Initial Conditions Ending
km
79 km
to school
Exit
Exit
79 km 75 km
Exit
Exit
0 km Exit
More Examples of Iterative Algorithms
Using Constraints on Input to Achieve Linear-
Time Sorting
COSC 3101, PROF. J. ELDER
134
Recall: InsertionSort
2
2
( 1)
Worst case (reverse order): : 1 ( ) ( )
2
n
j
j
n n
t j j T n n u
=
+
= = e

COSC 3101, PROF. J. ELDER

135
Recall: MergeSort
( ) ( log ) T n n n u e
COSC 3101, PROF. J. ELDER
136
Comparison Sorts
InsertionSort and MergeSort are examples of (stable)
Comparison Sort algorithms.
QuickSort is another example we will study shortly.
Comparison Sort algorithms sort the input by successive
comparison of pairs of input elements.
Comparison Sort algorithms are very general: they
make no assumptions about the values of the input
elements.
COSC 3101, PROF. J. ELDER
137
Comparison Sorts
2
InsertionSort is ( ). n u
MergeSort is ( log ). n n u
Can we do better?
COSC 3101, PROF. J. ELDER
138
Comparison Sort: Decision Trees
Example: Sorting a 3-element array A[1..3]
COSC 3101, PROF. J. ELDER
139
Comparison Sort
Worst-case time is equal to the height of the binary
decision tree.
The height of the tree is the log of the number of leaves.
The leaves of the tree represent all possible
permutations of the input. How many are there?

( )
log ! ( log ) n n n eO
Thus MergeSort is asymptotically optimal.
COSC 3101, PROF. J. ELDER
140
Linear Sorts?
Comparison sorts are very general, but are ( log ) n n O
Faster sorting may be possible if we can constrain the nature of the input.
COSC 3101, PROF. J. ELDER
141
Example 1. Counting Sort
Counting Sort applies when the elements to be sorted
come from a finite (and preferably small) set.
For example, the elements to be sorted are integers in
the range [0k-1], for some fixed integer k.
We can then create an array V[0k-1] and use it to
count the number of elements with each value [0k-1].
Then each input element can be placed in exactly the
right place in the output array in constant time.
COSC 3101, PROF. J. ELDER
142
Counting Sort
Input: N records with integer keys between [0k-1].
Output: Stable sorted keys.
Algorithm:
Count frequency of each key value to determine transition
locations
Go through the records in order putting them where they go.
Input:
Output: 0 0 0 0 0 1 1 1 1 1 1 1 1 1 2 2 3 3 3
1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
COSC 3101, PROF. J. ELDER
143
CountingSort
Stable sort: If two keys are the same, their order does not change.
Input:
Output:
1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
Thus the 4
th
record in input with digit 1 must be
the 4
th
record in output with digit 1.
It belongs at output index 8, because 8 records go before it
ie, 5 records with a smaller digit & 3 records with the same digit
0 0 0 0 0 1 1 1 1 1 1 1 1 1 2 2 2 3 3
Count These!
Index:
11 10 9 8 7 6 5 4 3 2 1 0 12 13 14 15 16 17 18
COSC 3101, PROF. J. ELDER
144
CountingSort
Input:
Output:
Index:
11 10 9 8 7 6 5 4 3 2 1 0 12 13 14 15 16 17 18
Value v:
# of records with digit v:
1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
3 2 1 0
2 3 9 5
N records. Time to count?
O(N)
COSC 3101, PROF. J. ELDER
145
CountingSort
Input:
Output:
Index:
11 10 9 8 7 6 5 4 3 2 1 0 12 13 14 15 16 17 18
Value v:
# of records with digit v:
# of records with digit < v:
1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
3 2 1 0
3 3 9 5
17 14 5 0
N records, k different values. Time to count? O(k)
COSC 3101, PROF. J. ELDER
146
CountingSort
Input:
Output:
Index:
11 10 9 8 7 6 5 4 3 2 1 0 12 13 14 15 16 17 18
Value v:
# of records with digit < v:
1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
3 2 1 0
17 14 5 0
= location of first record with digit v.
0 0 0 0 0 1 1 1 1 1 1 1 1 1 2 2 2 3 3
COSC 3101, PROF. J. ELDER
147
CountingSort
Input:
Output:
Index:
11 10 9 8 7 6 5 4 3 2 1 0 12 13 14 15 16 17 18
Value v:
1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
3 2 1 0
17 14 5 0 Location of first record
with digit v.
Algorithm: Go through the records in order
putting them where they go.
1
0 ?
COSC 3101, PROF. J. ELDER
148
Loop Invariant
The first i-1 keys have been placed in the correct
locations in the output array
The auxiliary data structure v indicates the location at
which to place the i
th
key for each possible key value
from [1..k-1].
COSC 3101, PROF. J. ELDER
149
CountingSort
Input:
Output:
Index:
11 10 9 8 7 6 5 4 3 2 1 0 12 13 14 15 16 17 18
Value v:
1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
3 2 1 0
17 14 5 0 Location of next record
with digit v.
1
Algorithm: Go through the records in order
putting them where they go.
COSC 3101, PROF. J. ELDER
150
CountingSort
Input:
Output:
Index:
11 10 9 8 7 6 5 4 3 2 1 0 12 13 14 15 16 17 18
Value v:
1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
3 2 1 0
17 14 6 0 Location of next record
with digit v.
0
Algorithm: Go through the records in order
putting them where they go.
1
COSC 3101, PROF. J. ELDER
151
CountingSort
Input:
Output:
Index:
11 10 9 8 7 6 5 4 3 2 1 0 12 13 14 15 16 17 18
Value v:
1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
3 2 1 0
17 14 6 1 Location of next record
with digit v.
0 0
Algorithm: Go through the records in order
putting them where they go.
1
COSC 3101, PROF. J. ELDER
152
CountingSort
Input:
Output:
Index:
11 10 9 8 7 6 5 4 3 2 1 0 12 13 14 15 16 17 18
Value v:
1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
3 2 1 0
17 14 6 2 Location of next record
with digit v.
0 1
Algorithm: Go through the records in order
putting them where they go.
1 0
COSC 3101, PROF. J. ELDER
153
CountingSort
Input:
Output:
Index:
11 10 9 8 7 6 5 4 3 2 1 0 12 13 14 15 16 17 18
Value v:
1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
3 2 1 0
17 14 7 2 Location of next record
with digit v.
0 1
Algorithm: Go through the records in order
putting them where they go.
1 0
3
COSC 3101, PROF. J. ELDER
154
CountingSort
Input:
Output:
Index:
11 10 9 8 7 6 5 4 3 2 1 0 12 13 14 15 16 17 18
Value v:
1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
3 2 1 0
18 14 7 2 Location of next record
with digit v.
0 1
Algorithm: Go through the records in order
putting them where they go.
1 0
1 3
COSC 3101, PROF. J. ELDER
155
CountingSort
Input:
Output:
Index:
11 10 9 8 7 6 5 4 3 2 1 0 12 13 14 15 16 17 18
Value v:
1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
3 2 1 0
18 14 8 2 Location of next record
with digit v.
0 1
Algorithm: Go through the records in order
putting them where they go.
1 0
1 3 1
COSC 3101, PROF. J. ELDER
156
CountingSort
Input:
Output:
Index:
11 10 9 8 7 6 5 4 3 2 1 0 12 13 14 15 16 17 18
Value v:
1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
3 2 1 0
18 14 9 2 Location of next record
with digit v.
0 1
Algorithm: Go through the records in order
putting them where they go.
1 0
3 3 1
1
COSC 3101, PROF. J. ELDER
157
CountingSort
Input:
Output:
Index:
11 10 9 8 7 6 5 4 3 2 1 0 12 13 14 15 16 17 18
Value v:
1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
3 2 1 0
19 14 9 2 Location of next record
with digit v.
0 1
Algorithm: Go through the records in order
putting them where they go.
1 0
1 3 1
1
3
COSC 3101, PROF. J. ELDER
158
CountingSort
Input:
Output:
Index:
11 10 9 8 7 6 5 4 3 2 1 0 12 13 14 15 16 17 18
Value v:
1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
3 2 1 0
19 14 10 2 Location of next record
with digit v.
0 1
Algorithm: Go through the records in order
putting them where they go.
1 0
0 1
1 1 3 3
COSC 3101, PROF. J. ELDER
159
CountingSort
Input:
Output:
Index:
11 10 9 8 7 6 5 4 3 2 1 0 12 13 14 15 16 17 18
Value v:
1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
3 2 1 0
19 14 10 3 Location of next record
with digit v.
0 1
Algorithm: Go through the records in order
putting them where they go.
1 0
2 1
1 1 3 3 0
COSC 3101, PROF. J. ELDER
160
CountingSort
Input:
Output:
Index:
11 10 9 8 7 6 5 4 3 2 1 0 12 13 14 15 16 17 18
Value v:
1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
3 2 1 0
19 15 10 3 Location of next record
with digit v.
0 1
Algorithm: Go through the records in order
putting them where they go.
1 0
1 1
1 1 3 3 0 2
COSC 3101, PROF. J. ELDER
161
CountingSort
Input:
Output:
Index:
11 10 9 8 7 6 5 4 3 2 1 0 12 13 14 15 16 17 18
Value v:
1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
3 2 1 0
19 15 10 3 Location of next record
with digit v.
0 1
Algorithm: Go through the records in order
putting them where they go.
1 0
1 1
1 1 3 3 0 2
COSC 3101, PROF. J. ELDER
162
CountingSort
Input:
Output:
Index:
11 10 9 8 7 6 5 4 3 2 1 0 12 13 14 15 16 17 18
Value v:
1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
3 2 1 0
19 17 14 5 Location of next record
with digit v.
0 1 1 0
1 1
1 1 3 3 0 2 0 0
1 1
1 2 2
O(N) Time =
O(N+k) Total =
End of Lecture 5
Sept 25, 2007
COSC 3101, PROF. J. ELDER
164

Input:
A of stack of N punch cards.
Each card contains d digits.
Each digit between [0k-1]
Output:
Sorted cards.
Example 2. RadixSort
344
125
333
134
224
334
143
225
325
243
Digit Sort:
Select one digit
Separate cards into k piles
based on selected digit (e.g., Counting Sort).
125
224
225
325
333
134
334
344
143
243
Stable sort: If two cards are the same for that digit,
their order does not change.
COSC 3101, PROF. J. ELDER
165
RadixSort
344
125
333
134
224
334
143
225
325
243
Sort wrt which
digit first?
The most
significant.
125
134
143
224
225
243
344
333
334
325
Sort wrt which
digit Second?
The next most
significant.
125
224
225
325
134
333
334
143
243
344
All meaning in first sort lost.
COSC 3101, PROF. J. ELDER
166
RadixSort
344
125
333
134
224
334
143
225
325
243
Sort wrt which
digit first?
Sort wrt which
digit Second?
The least
significant.
333
143
243
344
134
224
334
125
225
325
The next least
significant.
224
125
225
325
333
134
334
143
243
344


COSC 3101, PROF. J. ELDER
167
RadixSort
344
125
333
134
224
334
143
225
325
243
Sort wrt which
digit first?
Sort wrt which
digit Second?
The least
significant.
333
143
243
344
134
224
334
125
225
325
The next least
significant.
2 24
1 25
2 25
3 25
3 33
1 34
3 34
1 43
2 43
3 44


Is sorted wrt least sig. 2 digits.
COSC 3101, PROF. J. ELDER
168
Sort wrt i+1st
digit.
2 24
1 25
2 25
3 25
3 33
1 34
3 34
1 43
2 43
3 44
Is sorted wrt
first i digits.
1 25
1 34
1 43
2 24
2 25
2 43
3 25
3 33
3 34
3 44
Is sorted wrt
first i+1 digits.
i+1
These are in the
correct order
because sorted
wrt high order digit
RadixSort
COSC 3101, PROF. J. ELDER
169
Sort wrt i+1st
digit.
2 24
1 25
2 25
3 25
3 33
1 34
3 34
1 43
2 43
3 44
Is sorted wrt
first i digits.
1 25
1 34
1 43
2 24
2 25
2 43
3 25
3 33
3 34
3 44
i+1
These are in the
correct order
because was sorted &
stable sort left sorted
Is sorted wrt
first i+1 digits.
RadixSort
COSC 3101, PROF. J. ELDER
170
Loop Invariant
The keys have been correctly stable-sorted with respect
to the i-1 least-significant digits.
COSC 3101, PROF. J. ELDER
171
Running Time
Running time is ( ( ))
Where
# of digits in each number
# of elements to be sorted
# of possible values for each digit
d n k
d
n
k
O +
=
=
=
COSC 3101, PROF. J. ELDER
172
Example 3. Bucket Sort
Applicable if input is constrained to finite interval, e.g.,
[01).
If input is random and uniformly distributed, expected
run time is (n).
COSC 3101, PROF. J. ELDER
173
Bucket Sort
COSC 3101, PROF. J. ELDER
174
Loop Invariants
Loop 1
The first i-1 keys have been correctly placed into buckets of
width 1/n.
Loop 2
The keys within each of the first i-1 buckets have been correctly
stable-sorted.
COSC 3101, PROF. J. ELDER
175
PseudoCode
(1) O
(1) O
( ) n O
n
( ) n O
COSC 3101, PROF. J. ELDER
176
Examples of Iterative Algorithms
Binary Search
Partitioning
Insertion Sort
Counting Sort
Radix Sort
Bucket Sort
Which can be made stable?
Which sort in place?
How about MergeSort?
End of Iterative Algorithms