Design and Analysis of Algorithms 214.

Introduction to Algorithms.

U.U.Samantha Rajapaksha
BSc. Eng. (Moratuwa), MSc in IT
Senior Lecturer
Sri Lanka Institute of Information Technology
New Kandy Road,
Malabe, Sri Lanka
Tel:0112-301904
email: samantha.r@sliit.lk
Web: www.sliit.lk

DAA 214 Sri Lanka Institute of Information Technology. 1

Course Information.
Assignment (20%).
Mid term test (20%).
Final Examination (60%).
Lab classes (Based on C).

DAA 214 Sri Lanka Institute of Information Technology. 2

Contents.
1. Big O, recurrences and Data structures
2. Divide and Conquer
3. Priority Queues and heaps
4. Graphs (Reading Assignment)
5. Greedy algorithms
6. Data compression
7. String searching
8. Dynamic Programming
9. Matrix multiplication
10. Backtracking (Reading Assignment)

DAA 214 Sri Lanka Institute of Information Technology. 3

Text Books
Introduction to Algorithms
by Cormen, Leiserson, Rivest and
Stein (second edition)

Data Structures, Algorithms, and

Applications in C++
by Sartaj Sahni

Algorithms in C,
Parts 1-5, Third Edition by Robert
Sedgewick.

Data Structures
and Algorithms
Using C#
by Michael
McMillan
DAA 214 Sri Lanka Institute of Information Technology. 4
ALGORITHMS
Algorithm is any well defined computational
procedure that takes some value or set of values as
input and produce some value or set of values as
output.

3,1,7,2,9,8,5,4,6 1,2,3,4,5,6,7,8,9
ALGORITHM
INPUT OUTPUT

DAA 214 Sri Lanka Institute of Information Technology. 5

ALGORITHM (Contd.)
1.Get the smallest value from the input.
2.Remove it and output.
3.Repeat above 1,2 for remaining input until there is no
item in the input.

DAA 214 Sri Lanka Institute of Information Technology. 6

Properties of an Algorithm.
Be correct.
Be unambiguous.
Give the correct solution for all cases.
Be simple.
It must terminate.

Necker_cube_and_impossible_cube
Source:http://en.wikipedia.org/wiki/Ambiguity#Mathematical_i
nterpretation_of_ambiguity

DAA 214 Sri Lanka Institute of Information Technology. 7

 Applications of Algorithms are ubiquitous
Data retrieval
Network routing
Sorting
Searching
Shortest paths in a graph
Minimum spanning tree
Traveling salesman problem
Knapsack problem
Games
Internet algorithms
Electronic commerce

DAA 214 Sri Lanka Institute of Information Technology. 8

Designing of an Algorithm.
Divide and Conquer.
The Greedy Method.
Dynamic Programming.
Backtracking.

DAA 214 Sri Lanka Institute of Information Technology. 9

Advantages of above methods
Provide a template.
Translation to data structures is easy.
The temporal and spatial requirements can be precisely
analyzed.

DAA 214 Sri Lanka Institute of Information Technology. 10

Pseudocode
Method of writing down a algorithm.
Easy to read and understand.
Just like other programming language.

More expressive method.

Does not concern with the issue of software engineering.

DAA 214 Sri Lanka Institute of Information Technology. 11

Pseudocode Conventions.
English.
Indentation.
Separate line for each instruction.
Looping constructs and conditional constructs.
indicate a comment line.
indicate the assignment.

DAA 214 Sri Lanka Institute of Information Technology. 12

Pseudocode (Contd.)
Array elements are accessed by specifying the array name
followed by the index in the square bracket.

The notation .. is used to indicate a range of values within the

array.A[1..i] indicates the sub array of A consisting of elements
A[1],A[2],..,A[i].

DAA 214 Sri Lanka Institute of Information Technology. 13

Analysis of Algorithms
Idea is to predict the resource usage.
Memory
Logic Gates
Computational Time(***)

Why do we need a analysis?

To compare
Predict the growth of run time

DAA 214 Sri Lanka Institute of Information Technology. 14

Analysis of Algorithms
Efficiency measure:
- Speed: How long an algorithm takes to produce
results usual measure.
- Space/memory requirement.
Use the same computation model for the analyzed
algorithms

DAA 214 Sri Lanka Institute of Information Technology. 15

Analysis of Algorithms
In general, the time taken by an algorithm grows with
the size of input
Input size: depends of problems being studied (e.g.
#elements, #nodes, #links).
Running time on a particular input: #of primitive
operations or steps executed.

DAA 214 Sri Lanka Institute of Information Technology. 16

Worst,Best and Average case.
Running time will depend on the chosen instance
characteristics.
Best case:minimum number of steps taken on any instance of
size n.
Worst case:maximum number of steps taken on any instance of
size n.
Average case:An average number of steps taken on any
instance of size n.

DAA 214 Sri Lanka Institute of Information Technology. 17

 Worst,Best and Average
case(Contd.)
worst case
# of steps

Average case

Best case

Input Size
DAA 214 Sri Lanka Institute of Information Technology. 18
 Operation count
Select one or more operations such as add,multiply and
compare.

Methods for time complexity analysis.

Operation count:Omits accounting for the time spent on all but

the chosen operations.

DAA 214 Sri Lanka Institute of Information Technology. 19

 Step Count (RAM Model)
Assume a generic one processor.
Instructions are executed one after another,with no concurrent
operations.
+,-,=,it takes exactly one step.
Each memory access takes exactly 1 step.
Running Time= Sum of the steps.

DAA 214 Sri Lanka Institute of Information Technology. 20

 RAM Model Analysis.
n 100 1step
n n + 100 2steps
Print n 1step
4steps

sum 0 1 assignment
n+2 assignments
for i 0 to n n+2 comparisons
n+1 additions
sum sum A[i ] n+1 assignments
n+1 additions

Steps 6n+9 n+1 memory accesses

DAA 214 Sri Lanka Institute of Information Technology. 21

Problems with RAM Model
Differ number of steps with different architecture.
eg: sum sum + A[i] is a one step in the CISC processor.
It is difficult to count the exact number of steps in the algorithm.
eg: See the insertion sort

DAA 214 Sri Lanka Institute of Information Technology. 22

 Analysis of Insertion sort.
This is an efficient algorithm for sorting small number of
elements.
eg: Sorting a hand of cards using insertion sort.

DAA 214 Sri Lanka Institute of Information Technology. 23

Pseudocode for insertion sort.
INSERTION-SORT(A)
1 for j 2 to length[A]
2 do key A[j]
3 Insert A[j] into the sorted sequence A[1..j-1]
4 ij-1
5 While i > 0 and A[i] > key
6 do A[i+1] A[i]
7 i i-1
8 A[i+1] key

DAA 214 Sri Lanka Institute of Information Technology. 24

Example

The operation of INSERTION-SORT on the array A = 5, 2, 4, 6, 1, 3.

Array indices appear above the rectangles, and values stored in the array
positions appear within the rectangles. (a)-(e) The iterations of the for
loop of lines 1-8. In each iteration, the black rectangle holds the key
taken from A[j], which is compared with the values in shaded rectangles
to its left in the test of line 5. Shaded arrows show array values moved
one position to the right in line 6, and black arrows indicate where the
key is moved to in line 8. (f) The final sorted array.

DAA 214 Sri Lanka Institute of Information Technology. 25

Exact analysis of Insertion sort
Time taken for the algorithm will depend on the input
size(number of elements of the array)

Running Time (Time complexity):

This is the number of primitive operations or steps executed of an
algorithm on a particular input.

DAA 214 Sri Lanka Institute of Information Technology. 26

Running Time : T(n)
Each execution of the ith line takes time ci where ci is a
constant.
For each j=2,3,,n , we let tj be the number of times the
while loop test in line 5 is executed for that value of j.

DAA 214 Sri Lanka Institute of Information Technology. 27

 Running Time : T(n)
INSRTION-SORT(A) cost Times
1 for j 2 to length[A] c1 n
2 do key A[j] c2 n-1
3 Insert A[j] into the sorted
sequence A[1..j-1] 0 n-1
4 ij1 c4 n-1
5 While i > 0 and A[i] > key c5 n j=2 tj
6 do A[i+1] A[i] c6 n j=2 ( tj - 1)
7 i i-1 c7 n j=2 ( tj - 1)
8 A[i+1] key c8 n-1

DAA 214 Sri Lanka Institute of Information Technology. 28

 Running Time(contd.)
T(n) = c1 n + c2 (n-1) + c4 (n-1) + c5 n j=2 tj

+ c6 n j=2 ( tj - 1) + c7 n j=2 ( tj - 1) +

c8(n-1)
Best Case T(n) an+b
Worst Case T(n) cn2 + dn + e

Analysis is tedious one.

In the RAM model also we had problem so we move to asymptotic
notation.

DAA 214 Sri Lanka Institute of Information Technology. 29

Worst Case T(n) cn2 + dn + e

DAA 214 Sri Lanka Institute of Information Technology. 30

Worst Case T(n) cn2 + dn + e

DAA 214 Sri Lanka Institute of Information Technology. 31

 Asymptotic Notations
RAM Model have some problems.
Exact analysis is very complicated.

Here we focus on determining the biggest term in the

complexity function.
Sufficiently large size of n.

DAA 214 Sri Lanka Institute of Information Technology. 32

 Asymptotic Notations(Contd.)
There are three notations.

O - Notation
- Notation
- Notation

DAA 214 Sri Lanka Institute of Information Technology. 33

Asymptotic Notations (Contd.)
A way to describe behavior of functions in the limit. Were
studying asymptotic efficiency.
Describe growth of functions.
Focus on whats important by abstracting away low-order terms
and constant factors.
How we indicate running times of algorithms.
A way to compare sizes of functions:
O

=
o<

DAA 214 Sri Lanka Institute of Information Technology. 34

 Big O - Notation
Introduced by Paul Bechman in 1892.
We use Big O-notation to give an upper bound on a function.
Definition:
O(g(n)) = { f(n) : there exist positive constants c and no such
that 0 f(n) cg(n) for all n no}.

Eg:What is the big O value of f(n)=2n + 6 ?

c=4
no = 3
g(n)=n therefore f(n)=O(n)

DAA 214 Sri Lanka Institute of Information Technology. 35

DAA 214 Sri Lanka Institute of Information Technology. 36
 graphs of functions

DAA 214 Sri Lanka Institute of Information Technology. 37

DAA 214 Sri Lanka Institute of Information Technology. 38
 Big O - Notation

In general, we have the following theorem:

an xn + ... + a1 x + a0 is O( xn ) for any real
numbers an , ..., a0 and any nonnegative number n .

DAA 214 Sri Lanka Institute of Information Technology. 39

 Big O Notation(Contd.)
Find the Big Oh value for following fragment of code.

for i 1 to n
for j 1 to i O(n2)
Print j

DAA 214 Sri Lanka Institute of Information Technology. 40

 Big O Notation(Contd.)

Assignment (s 1)
Addition (s+1) O(1)
Multiplication (s*2)
Comparison (S<10)

DAA 214 Sri Lanka Institute of Information Technology. 41

 Big O Notation(Contd.)
Find the Bog O value for the following functions.
(i) T(n)= 3 +5n + 3n2
(ii) f(n)= 2n + n2 +8n +7
(iii) T(n)= n + logn +6
Answers:
(i) O(n2)
(ii) O(2n)
(iii) O(n)

DAA 214 Sri Lanka Institute of Information Technology. 42

Back to the example
Alternative calculation:
cost times

sum 0 c1 1
for i 1 to n c2 n+1
sum sum A[i ] c3 n
T(n) = c1 + c2 (n+1) + c3 n = (c1 + c2) + (c2 + c3) n = c4 + c5 n
O (n)

Proof: c4 + c5 n c n TRUE for n1 and c c4 + c5

DAA 214 Sri Lanka Institute of Information Technology. 43

Simple Big-O

s 0
s 1 All O(1)
s 1
s 1

for i 0 to n n

some O(1) process O(1) O(n)

i 0

DAA 214 Sri Lanka Institute of Information Technology. 44

Want least upper bound

All useful algorithms are O(2n), so saying an algorithm is

O(2n) is not very useful
Try and find the least O( f (n) ) expression
If f(n) is the smallest possible, then it is said to be a tight
upper bound; if f(n) is O(n2), it is also O(n3), however, the
first expression is better.
Not to be confused with tight asymptotic bound big theta
()

DAA 214 Sri Lanka Institute of Information Technology. 45

 - Notation
Which will provide the lower bound of the function.
Definition:
(g(n)) = { f(n) : there exist positive constants c and n0 such that
0 cg(n) f(n) for all n no}

DAA 214 Sri Lanka Institute of Information Technology. 46

DAA 214 Sri Lanka Institute of Information Technology. 47
 - Notation
In general, we have the following theorem:
an xn + ... + a1 x + a0 is ( xn ) for any real
numbers an , ..., a0 and any nonnegative number n .

DAA 214 Sri Lanka Institute of Information Technology. 48

 - Notation
This is used when the function f can be bounded both
from above and below by the same function g.
Definition:
(g(n)) ={ f(n): there exist positive constant c1, c2, and
n0 such that 0 c1 g(n) f(n) c2g(n) for all n n0 }

DAA 214 Sri Lanka Institute of Information Technology. 49

DAA 214 Sri Lanka Institute of Information Technology. 50
 - Notation(Contd.)

In general, we have the following theorem:

an xn + ... + a1 x + a0 is ( xn ) for any real
numbers an , ..., a0 and any nonnegative number n .

DAA 214 Sri Lanka Institute of Information Technology. 51

 Summary
What is an algorithm?
Properties.
Design methods.
Pseudocode.
Analysis(Operation count & Step count).
RAM model.
Insertion Sort.

DAA 214 Sri Lanka Institute of Information Technology. 52

 Summary (Contd.)
Asymptotic Notations.
O - Notation
- Notation
- Notation

DAA 214 Sri Lanka Institute of Information Technology. 53

 Thank You

DAA 214 Sri Lanka Institute of Information Technology. 54