Theory of Algorithms: Introduction

Michelle Kuttel & Sonia Berman

(mkuttel | sonia@cs.uct.ac.za)
17 September - 8 October (mkuttel) The most valuable acquisitions in a scientific or technical education are the general-purpose mental tools which remain serviceable for a life-time. George Forsythe, What to do till the computer scientist comes. (1968)

Prescribed Book
Anany Levitin, Introduction to the Design and Analysis of Algorithms, International Edition, Addison-Wesley, 2003

Objectives of this lecture

To define an algorithm To introduce:
Problem types The process of algorithm design Solution strategies Ways of analysing algorithms

To cover the structure of the course and practicals

Definitions
An algorithm is a sequence of unambiguous instructions for solving a problem
For obtaining a required output for any legitimate input in a finite amount of time Does not require implementation in software Not an answer but a method for deriving an answer

Background
Named after Muhammad ibn Musa al-Khwarizmi 9th century Persian mathematician Algorithmics is more than a branch of computer science. It is the core of computer science and, in all fairness, can be said to be relevant to most of science, business and technology.
David Harel, Algorithmics: the Spirit of Computing

Donald Knuth
Professor Emeritus of The Art of Computer Programming at Stanford University

Notion of an Algorithm
problem algorithm input computer output

Each step of the algorithm must be unambiguous The range of inputs must be specified carefully The same algorithm can be represented in different ways Several algorithms for solving the same problem may exist with different properties

What is an Algorithm?
Recipe, process, method, technique, procedure, routine, with the following requirements:
Finiteness
Terminates after a finite number of steps

Definiteness
Rigorously and unambiguously specified

Input
Valid inputs are clearly specified

Output
Can be proved to produce the correct output given a valid input

Effectiveness
Steps are sufficiently simple and basic

Example: Sorting
Statement of problem:
Input: A sequence of n numbers <a1, a2, , an > Output: A reordering of the input sequence <a1, a2, , an> s.t. ai aj whenever i < j

Instance: The sequence <5, 3, 2, 8, 3> Algorithms:

Selection sort Insertion sort Merge sort (many others)

Selection Sort
Input: array a[1],..,a[n] Output: array a sorted in non-decreasing order Algorithm:
for i=1 to n swap a[i] with smallest of a[i],,a[n] for i1 to n do min i for j i+1 to n do if a[j] < a[min] min j swap a[i] and a[min]

Exercise: Bridge Puzzle

Problem:
4 People want to cross a bridge. You have 17 minutes to get them across

Constraints:
It is night and you have 1 flashlight. Max of 2 on the bridge at one time. All start on the same side Those crossing must have the flashlight with them. The flashlight must be walked back and forth (no throwing) People walk at different speeds: person A = 1 minute to cross, person B = 2 minutes, person C = 5 minutes, person D = 10 minutes A pair walks at the speed of the slower persons pace

Rumour: this problem is given to Microsoft interviewees

Solution: Bridge Puzzle

Start (0 min): AB Across (2 min): AB A Back (+1 min): B CD Across (+10 min): B C D B Back (+2 min): CD AB Across (+2 min): A B C D ABCD CD ACD A AB

Total Time = 17 minutes

Extension to Exercise
This is an instance of a problem. How would you generalise it? Can you derive an algorithm to solve this generalised problem?
Must output the minimum time required for crossing Are there any special cases to watch out for? Are there any constraints on the input?

Extension Solution
Input: a list <a> of crossing times for n people, numbered 1, , n Output: total time to cross Strategy: use 1 & 2 as shuttles and send the others across in pairs
for i 2 to n/2 do t a[2] t t + a[1] t t + a[i*2] t t + a[2] t t + a[2] return t // 1 & 2 across // 1 back // i*2 & (i*2)-1 across // 2 back // 1 & 2 across

Extension Problems
This is an inadequate solution that falsely assumes certain inputs List may not be sorted in ascending order
Sort <a>

n may not be even numbered

Add an extra clause after the loop

n > 3 not guaranteed

Special case for n = 1, 2, 3

Is not optimal for all inputs, e.g. {1, 20, 21, 22}
Can you quantify the nature of these inputs? Suggest an alternative

Final solution is left as an exercise. Attempt to make your solution elegant

Why Study Algorithms?

Theoretical importance The core of computer science Practical importance A practitioners toolkit of known algorithms Framework for designing and analyzing algorithms for new problems Useful mindset

Solving Algorithmic Problems

Understand the Problem

Decide on computation type, approximation, data structures and algorithm design technique Design an Algorithm

Prove Correctness

Analyze the Algorithm

Code the Algorithm

Understanding the Problem

Fundamentals of Algorithmic Problem Solving

Make sure you are solving the correct problem and for all legitimate inputs

Ascertaining the Capabilities of a Computational Device

Sequential vs. Parallel. What are the speed and memory limits?

Choosing between exact and approximate Problem Solving

Is absolute precision required? Sometimes this may not be possible

Deciding on Appropriate Data Structures

Algorithms often rely on carefully structuring the data Fundamental Data Structures: array, linked list, stacks, queues, heaps, graphs, trees, sets

Fundamentals of Algorithm Design

Applying an Algorithm Design Technique
Using a general approach to problem solving that is applicable to a variety of problems

Specifying the Algorithm

Pseudocode is a mixture of natural language and programming constructs that has replaced flowcharts

Proving an Algorithms Correctness

Prove that an algorithm yields a required result for legitimate inputs in finite time

Analyzing an Algorithm
Consider time efficiency, space efficiency, simplicity, generality, optimality Analysis can be empirical or theoretical

Coding an Algorithm

Well known Computational Problems

Sorting Searching String Processing
String Matching

Graph Problems
Graph Traversal, Shortest Path, Graph Colouring

Combinatorial Problems
Find a combinatorial object - permutation, combination, subset subject to constraints

Geometric Problems
Closest-Pair, Convex-Hull

Numerical Problems
Solving systems of equations, computing definite integrals, evaluating functions, etc.

Algorithm Design Strategies

Brute force
A straightforward approach to solving a problem, usually directly based on the problems statement

Divide and conquer

Divide a problem into smaller instances, solve smaller instances (perhaps recursively), combine

Decrease and conquer

Exploit relationship between the problem and a smaller instance reduced by some factor (often 1)

Transform and conquer

Transform the problem to a simpler instance, another representation or an instance with a known solution

More Algorithm Design Strategies

Greedy approach
Make locally optimal steps which (hopefully) lead to a globally optimal solution for an optimization problem

Dynamic programming
Technique for solving problems with overlapping subdomains

Backtracking and Branch and bound

A way of tackling difficult optimization and combinatorial problems without exploring all state-space

Space and time tradeoffs

Preprocess the input and store additional information to accelerate solving the problem

Course Structure
Introduction (Ch. 1) - today Fundamentals of the Analysis of Algorithms (Ch. 2)
Asymptotic notations, analysis of recursive and non-recursive algorithms, empirical analysis

Algorithmic Strategies (Ch. 3-9)

Brute force, Divide-and-Conquer, Decrease-and-Conquer, Transform-and-Conquer, Space and Time Tradeoffs, Greedy Techniques, Dynamic Programming

Limitations of Algorithms (Ch. 10 + handouts)

Turing Machines, Computability, Problem Classification

Coping with Limitations on Algorithms (Ch. 11)

Backtracking and Branch and Bound

Practicals
Weekly 3-hour mini prac-exams Open Book Given a problem specification that is solvable using the algorithm design strategies presented in the course
Design Algorithm Code it in C++ or Java Submit it for automatic marking

No internet, e-mail or communicating with class-mates But can get limited help from tutors 1st one is a practice one