Lecture 1:

Introduction to Algorithms (1997)

Steven Skiena
Department of Computer Science
State University of New York
Stony Brook, NY 117944400
http://www.cs.sunysb.edu/skiena
What Is An Algorithm?
Algorithms are the ideas behind computer programs.
An algorithm is the thing which stays the same whether the
program is in Pascal running on a Cray in New York or is in
BASIC running on a Macintosh in Kathmandu!
To be interesting, an algorithm has to solve a general,
specied problem. An algorithmic problem is specied by
describing the set of instances it must work on and what
desired properties the output must have.
Example: Sorting
Input: A sequence of N numbers a
1
...a
n
Output: the permutation (reordering) of the input sequence
such as a
1
a
2
. . . a
n
.
We seek algorithms which are correct and efcient.
Correctness
For any algorithm, we must prove that it always returns the
desired output for all legal instances of the problem.
For sorting, this means even if (1) the input is already sorted,
or (2) it contains repeated elements.
Correctness is Not Obvious!
The following problem arises often in manufacturing and
transportation testing applications.
Suppose you have a robot arm equipped with a tool, say a
soldering iron. To enable the robot arm to do a soldering job,
we must construct an ordering of the contact points, so the
robot visits (and solders) the rst contact point, then visits
the second point, third, and so forth until the job is done.
Since robots are expensive, we need to nd the order which
minimizes the time (ie. travel distance) it takes to assemble
the circuit board.
You are given the job to program the robot arm. Give me an
algorithm to nd the best tour!
Nearest Neighbor Tour
A very popular solution starts at some point p
0
and then walks
to its nearest neighbor p
1
rst, then repeats from p
1
, etc. until
done.
Pick and visit an initial point p
0
p = p
0
i = 0
While there are still unvisited points
i = i + 1
Let p
i
be the closest unvisited point to p
i1
Visit p
i
Return to p
0
from p
i
This algorithm is simple to understand and implement and
very efcient. However, it is not correct!
-1 0 1 3 11
-21
-5
-1 0 1 3 11
-21
-5
Always starting from the leftmost point or any other point
will not x the problem.
Closest Pair Tour
Always walking to the closest point is too restrictive, since
that point might trap us into making moves we dont want.
Another idea would be to repeatedly connect the closest pair
of points whose connection will not cause a cycle or a three-
way branch to be formed, until we have a single chain with
all the points in it.
Let n be the number of points in the set
d =
For i = 1 to n 1 do
For each pair of endpoints (x, y) of partial paths
If dist(x, y) d then
x
m
= x, y
m
= y, d = dist(x, y)
Connect (x
m
, y
m
) by an edge
Connect the two endpoints by an edge.
Although it works correctly on the previous example, other
data causes trouble:
This algorithm is not correct!
A Correct Algorithm
We could try all possible orderings of the points, then select
the ordering which minimizes the total length:
d =
For each of the n! permutations
i
of the n points
If (cost(
i
) d) then
d = cost(
i
) and P
min
=
i
Return P
min
Since all possible orderings are considered, we are guaranteed
to end up with the shortest possible tour.
Because it trys all n! permutations, it is extremely slow, much
too slow to use when there are more than 10-20 points.
No efcient, correct algorithm exists for the traveling
salesman problem, as we will see later.
Efciency
Why not just use a supercomputer?
Supercomputers are for people too rich and too stupid to
design efcient algorithms!
A faster algorithm running on a slower computer will always
win for sufciently large instances, as we shall see.
Usually, problems dont have to get that large before the faster
algorithm wins.
Expressing Algorithms
We need some way to express the sequence of steps
comprising an algorithm.
In order of increasing precision, we have English, pseu-
docode, and real programming languages. Unfortunately,
ease of expression moves in the reverse order.
I prefer to describe the ideas of an algorithm in English,
moving to pseudocode to clarify sufciently tricky details of
the algorithm.
The RAM Model
Algorithms are the only important, durable, and original part
of computer science because they can be studied in a machine
and language independent way.
The reason is that we will do all our design and analysis for
the RAM model of computation:
Each simple operation (+, -, =, if, call) takes exactly 1
step.
Loops and subroutine calls are not simple operations, but
depend upon the size of the data and the contents of a
subroutine. We do not want sort to be a single step
operation.
Each memory access takes exactly 1 step.
We measure the run time of an algorithm by counting the
number of steps.
This model is useful and accurate in the same sense as the
at-earth model (which is useful)!
Best, Worst, and Average-Case
The worst case complexity of the algorithm is the function
dened by the maximum number of steps taken on any
instance of size n.
1 2 3 4 . . . . . .
N
# of
Steps
Worst Case
Complexity
Average Case
Complexity
Best Case
Complexity
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
The best case complexity of the algorithm is the function
dened by the minimum number of steps taken on any
instance of size n.
The average-case complexity of the algorithm is the function
dened by an average number of steps taken on any instance
of size n.
Each of these complexities denes a numerical function
time vs. size!
Insertion Sort
One way to sort an array of n elements is to start with
a
n
empty list, then successively insert new elements in the
proper position:
a
1
a
2
. . . a
k
| a
k+1
. . . a
n
At each stage, the inserted element leaves a sorted list, and
after n insertions contains exactly the right elements. Thus
the algorithm must be correct.
But how efcient is it?
Note that the run time changes with the permutation instance!
(even for a xed size problem)
How does insertion sort do on sorted permutations?
How about unsorted permutations?
Exact Analysis of Insertion Sort
Count the number of times each line of pseudocode will be
executed.
Line InsertionSort(A) #Inst. #Exec.
1 for j:=2 to len. of A do c1 n
2 key:=A[j] c2 n-1
3 /* put A[j] into A[1..j-1] */ c3=0 /
4 i:=j-1 c4 n-1
5 while i > 0&A[1] > key do c5 tj
6 A[i+1]:= A[i] c6
7 i := i-1 c7
8 A[i+1]:=key c8 n-1
The for statement is executed (n 1) + 1 times (why?)
Within the for statement, key:=A[j] is executed n-1 times.
Steps 5, 6, 7 are harder to count.
Let t
j
= 1+ the number of elements that have to be slide right
to insert the jth item.
Step 5 is executed t
2
+ t
3
+ ... + t
n
times.
Step 6 is t
21
+ t
31
+ ... + t
n1
.
Add up the executed instructions for all pseudocode lines to
get the run-time of the algorithm:
c
1
n + c
2
(n 1) + c
4
(n 1)+ c
5

n
j=2
t
j
+ c
6

n
j=2
(t
j
1)
+c
7

n
j=2
(t
j
1) + c
8
What are the t

j
s? They depend on the particular input.
Best Case
If its already sorted, all t
j
s are 1.
Hence, the best case time is
c
1
n + (c
2
+ c
4
+ c
5
+ c
8
)(n 1) = Cn + D
where C and D are constants.
Worst Case
If the input is sorted in descending order, we will have to
slide all of the already-sorted elements, so t
j
= j, and step 5
is executed
n

j=2
j = (n
2
+ n)/2 1