Chapter 1

Introduction

Design and Analysis of Algorithms Unit I

1-0

Algorithm
An Algorithm is a sequence of unambiguous instructions for solving a problem, i.e., for obtaining a required output for any legitimate input in a finite amount of time.

Design and Analysis of Algorithms Unit I

1-1

Notion of algorithm
problem

algorithm

input

computer

output

Algorithmic solution
Design and Analysis of Algorithms Unit I

1-2

PSEUDOCODE
Pseudocode (pronounced SOO-doh-kohd) is a detailed yet readable description of what a computer program or algorithm must do, expressed in a formally-styled natural language rather than in a programming language. It is sometimes used as a detailed step in the process of developing a program. It allows programmers to express the design in great detail and provides programmers a detailed template for the next step of writing code in a specific programming language.
Design and Analysis of Algorithms Unit I
1-3

Formatting and Conventions in Pseudocoding

INDENTATION in pseudocode should be identical to its implementation in a programming language. Try to indent at least four spaces. The pseudocode entries are to be cryptic, AND SHOULD NOT BE PROSE. NO SENTENCES. No flower boxes in pseudocode. Do not include data declarations in pseudocode.

Design and Analysis of Algorithms Unit I

1-4

Some Keywords That Should be Used

For looping and selection, Do While...EndDo; Do Until...Enddo; Case...EndCase; If...Endif; Call ... with (parameters); Call; Return ....; Return; When; Always use scope terminators for loops and iteration.

Design and Analysis of Algorithms Unit I

1-5

Some Keywords
As verbs, use the words generate, Compute, Process, Set, reset, increment, calculate, add, sum, multiply, ... print, display, input, output, edit, test , etc.

Design and Analysis of Algorithms Unit I

1-6

Methods of finding GCD

M-1 M-2

M-3

Design and Analysis of Algorithms Unit I

1-7

Euclids Algorithm
Problem: Find gcd(m,n), the greatest common divisor of two nonnegative, not both zero integers m and n Examples: gcd(60,24) = 12, gcd(60,0) = 60, gcd(0,0) = ? Euclids algorithm is based on repeated application of equality gcd(m,n) = gcd(n, m mod n) until the second number becomes 0, which makes the problem trivial. Example: gcd(60,24) = gcd(24,12) = gcd(12,0) = 12

Design and Analysis of Algorithms Unit I

1-8

Two descriptions of Euclids algorithm

Step 1 If n = 0, return m and stop; otherwise go to Step 2 Step 2 Divide m by n and assign the value fo the remainder to r Step 3 Assign the value of n to m and the value of r to n. Go to Step 1.

while n 0 do r m mod n m n n r return m

Design and Analysis of Algorithms Unit I
1-9

Other methods for computing gcd(m,n)

Consecutive integer checking algorithm
Step 1 Assign the value of min{m,n} to t Step 2 Divide m by t. If the remainder is 0, go to Step 3; otherwise, go to Step 4 Step 3 Divide n by t. If the remainder is 0, return t and stop; otherwise, go to Step 4 Step 4 Decrease t by 1 and go to Step 2

Design and Analysis of Algorithms Unit I

1-10

Other methods for gcd(m,n) [cont.]

Middle-school procedure
Step 1 Step 2 Step 3 Step 4 Find the prime factorization of m Find the prime factorization of n Find all the common prime factors Compute the product of all the common prime factors and return it as gcd(m,n)

Is this an algorithm?

Design and Analysis of Algorithms Unit I

1-11

Sieve of Eratosthenes
Input: Integer n 2 Output: List of primes less than or equal to n for p 2 to n do A[p] p for p 2 to n do if A[p] { 0 //p hasnt been previously eliminated from the list j p* p while j n do A[j] j

0 //mark element as eliminated j+p

Example: 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Design and Analysis of Algorithms Unit I
1-12

Termination of Euclids Algorithm

The second number of the pair gets smaller with each iteration and cannot become negative: Indeed, the new value of n is r = m mod n, which is always smaller than n. Eventually, r becomes zero, and the algorithms stops.

Design and Analysis of Algorithms Unit I

1-13

Compute the GCD of 120 and 23.

Design and Analysis of Algorithms Unit I

1-14

Fundamentals of Algorithmic Problem Solving

Design and Analysis of Algorithms Unit I

1-15

Fundamentals of Algorithmic Problem Solving

Algorithm = Procedural Solutions to Problem NOT an answer, BUT rather specific instructions of getting answers. Therefore, requires steps in designing and analyzing an algorithm

Design and Analysis of Algorithms Unit I

1-16

Algorithm Design & Analysis Process

Design and Analysis of Algorithms Unit I

1-17

Step 1: Understand the Problem

Before designing an algorithm - understand completely the problem given. Read the problems description carefully and ask questions if you have any doubts about the problem, Do a few small examples by hand, think about special cases, and ask questions again if needed.

Design and Analysis of Algorithms Unit I

1-18

Step 1: Understand the Problem

An input to an algorithm specifies an instance of the problem the algorithm solves. It is very important to specify exactly the range of instances the algorithm needs to handle. Failing which the algorithm works correctly for some inputs , but crashes on some boundary values. Remember that a correct algorithm is not one that works most of the time but one that works correctly for all legitimate inputs.

Design and Analysis of Algorithms Unit I

1-19

Step 2: Ascertaining the capabilities of a computational device

Algorithms designed to be executed on machines that executes intstructions one after another are called sequential algorithms. Algorithms that take advantage of computers that can execute operations concurrently are called parallel algorithms.

Design and Analysis of Algorithms Unit I

1-20

Step 3: Choosing between Exact & Approximate Problem Solving

Solving the problem exactly - Exact algorithms Solving the problem approximately - Approximation algorithms Why approximation algorithms? 1. Problems cannot be solved exactly. Eg. Extracting square roots, solving non-linear equations 2. Available exact algorithms are unacceptably slow because of problems complexity Eg. Traveling Salesman Problem 3. Approx. Algs can be a part of algorithms that solve the problem exactly.
Design and Analysis of Algorithms Unit I
1-21

Step 4: Deciding on Appropriate Data Structures

In the new world of object-oriented programming, data structures remain important for both design and analysis of algorithms. However, we will assume a very basic data structure for now and concentrate on the algorithm side.

Design and Analysis of Algorithms Unit I

1-22

Step 5: Algorithm Design Techniques

An algorithm design technique (or strategy or paradigm) is a general approach to solving problems algorithmically that is applicable to a variety of problems from different areas of computing. Eg. Brute force, Divide-and-Conquer, Transform-andConquer Importance: 1. Provide guidance for designing algorithms for new problems. 2. To classify algorithms according to an underlying design idea.
Design and Analysis of Algorithms Unit I
1-23

Step 6: Methods of Specifying an Algorithm

Pseudocode, a mixture of a natural language and programming language-like constructs. languageflowchart, flowchart, a method of expressing an algorithm by a collection of connected geometric shapes containing descriptions of the algorithms steps.

Design and Analysis of Algorithms Unit I

1-24

Step 7: Proving an Algorithms Correctness

Prove algorithms correctness = prove that the algorithm yields a required result for every legitimate input in a finite amount of time. For an approximation algorithm, correctness means to be able to show that the error produced by the algorithm does not exceed a predefined limit.

Design and Analysis of Algorithms Unit I

1-25

Step 8: Analyzing an Algorithm

1. Efficiency Time efficiency indicates how fast the algorithm runs. space efficiency indicates how much extra memory the algorithm needs. 2. Simplicity 3. Generality Design an algorithm for a problem posed in more general terms. Design an algorithm that can handle a range of inputs that is natural for the problem at hand.

Design and Analysis of Algorithms Unit I

1-26

Step 9: Coding the algorithm

More than implementation Peril of incorrect & inefficient implementation Require testing & debugging Require code optimizing

Design and Analysis of Algorithms Unit I

1-27

Important Problem Types

Design and Analysis of Algorithms Unit I

1-28

Important Problem Types

Sorting Searching String processing Graph problems Combinatorial problems Geometric problems Numerical problems

Design and Analysis of Algorithms Unit I

1-29

Sorting
The sorting problem asks us to rearrange the items of a given list in ascending order. we usually need to sort lists of numbers, characters from an alphabet, character strings, records similar to those maintained by schools about their students, libraries about their holdings, companies about their employees.
Design and Analysis of Algorithms Unit I
1-30

Searching
The searching problem deals with finding a given value, called a search key, in a given set (or a multiset, which key, permits several elements to have the same value).

Design and Analysis of Algorithms Unit I

1-31

String Processing
A string is a sequence of characters from an alphabet. String of particular interest: 1. Text string comprises letters, numbers, and special characters 2. Bit string comprises zeros and ones 3. Gene sequence Mainly string matching problem: searching for a given problem: word in a text

Design and Analysis of Algorithms Unit I

1-32

Graph Problems
A graph can be thought of as a collection of points called vertices, some of which are connected by line segments called edges. Used for modeling a wide variety of real-life applications. realBasic graph algorithms include: 1. Graph traversal algorithms - How can one visit all the points in a network? 2. Shortest-path algorithms - What is the best Introduction Shortestroute between two cities? 3. Topological sorting for graphs with directed edges

Design and Analysis of Algorithms Unit I

1-33

Combinatorial Problems
combinatorial problems: problems that ask (explicitly or implicitly) to find a combinatorial objectsuch as a object permutation, a combination, or a subsetthat satisfies subset certain constraints and has some desired property (e.g., maximizes a value or minimizes a cost). 1. Combinatorial grows extremely fast with problem size 2. No known algorithm solving most such problems exactly in an acceptable amount of time.

Design and Analysis of Algorithms Unit I

1-34

Geometric Problems
Geometric algorithms deal with geometric objects such as points, lines, and polygons. 2 class problems: The closest pair problem: given n points in the plane, find problem: the closest pair among them. The convex hull problem asks to find the smallest convex polygon that would include all the points of a given set. If

Design and Analysis of Algorithms Unit I

1-35

Convex hull problem

Design and Analysis of Algorithms Unit I

1-36

Numerical Problems
Numerical problems, another large special area of problems, applications, are problems that involve mathematical objects of continuous nature: solving equations and systems of equations, computing definite integrals, evaluating functions, and so on.

Design and Analysis of Algorithms Unit I

1-37

Fundamentals of Analysis of algorithm efficiency

Design and Analysis of Algorithms Unit I

1-38

Analysis of algorithms
Issues: correctness time efficiency space efficiency optimality Approaches: theoretical analysis empirical analysis
Design and Analysis of Algorithms Unit I
1-39

Theoretical analysis of time efficiency

Time efficiency is analyzed by determining the number of asic repetitions of the basic operation as a function of input size Basic operation: the operation that contributes the most operation: towards the running time of the algorithm
input size

T(n) copC(n)
running time execution time for basic operation or cost Number of times basic operation is executed

Note: Different basic operations may cost differently!

Design and Analysis of Algorithms Unit I
1-40

Input size and basic operation examples

Problem Searching for key in a list of n items Multiplication of two matrices Checking primality of a given integer n Input size measure Number of lists items, i.e. n Basic operation Key comparison

Matrix dimensions or Multiplication of total number of two numbers elements nsize = number of digits (in binary representation) Division Visiting a vertex or traversing an edge 1-41

Typical graph #vertices and/or problem edges Design and Analysis of Algorithms Unit I

Empirical analysis of time efficiency

Select a specific (typical) sample of inputs Use physical unit of time (e.g., milliseconds) or Count actual number of basic operations executions Analyze the empirical data

Design and Analysis of Algorithms Unit I

1-42

Efficiencies
Worst Case Efficiency: Is its efficiency for the worst case input of size n, which is an input of size n for which the algorithm runs the longest among all possible inputs of that size Cworst(n) BestBest-case efficiency: Is its efficiency for the worst case input of size n, which is an input of size n for which the algorithm runs the fastest among all possible inputs of that size Cbest(n)

Design and Analysis of Algorithms Unit I

1-43

Amortized efficiency
It applies not to a single run of an algorithm, but rather to a sequence of operations performed on the same data structure

Design and Analysis of Algorithms Unit I

1-44

BestBest-case, average-case, worst-case averageworstFor some algorithms, efficiency depends on form of input: Worst case: Cworst(n) maximum over inputs of size n Best case: Cbest(n) minimum over inputs of size n Average case: Cavg(n) average over inputs of size n Number of times the basic operation will be executed on typical input NOT the average of worst and best case Expected number of basic operations considered as a random variable under some assumption about the probability distribution of all possible inputs. So, avg = expected under uniform distribution.
Design and Analysis of Algorithms Unit I
1-45

Example: Sequential search

Worst case Best case Average case

n key comparisons 1 comparisons (n+1)/2, assuming K is in A

1-46

Design and Analysis of Algorithms Unit I

Types of formulas for basic operations count

Exact formula e.g., C(n) = n(n-1)/2 C(n Formula indicating order of growth with specific multiplicative constant e.g., C(n) 0.5 n2 C(n Formula indicating order of growth with unknown multiplicative constant e.g., C(n) cn2 C(n

Design and Analysis of Algorithms Unit I

1-47

Order of growth
Most important: Order of growth within a constant multiple as n Example: How much faster will algorithm run on computer that is twice as fast? How much longer does it take to solve problem of double input size?

Design and Analysis of Algorithms Unit I

1-48

Values of some important functions as n p g

Design and Analysis of Algorithms Unit I

1-49

Asymptotic Notations
O (Big-Oh)-notation (Big-Omega) -notation (Big-Theta) -notation

Design and Analysis of Algorithms Unit I

1-50

Asymptotic order of growth

A way of comparing functions that ignores constant factors and small input sizes (because?) O(g O(g(n)): class of functions f(n) that grow no faster than g(n) (g(n)): class of functions f(n) that grow at same rate as g(n) (g(n)): class of functions f(n) that grow at least as fast as g(n)

Design and Analysis of Algorithms Unit I

1-51

O-notation
Definition: A function t(n) is said to be in O(g(n)), denoted t(n) t( O(g t(n) O(g(n)) is bounded above by some constant multiple of g(n) for all large n, i.e., there exist positive constant c and nonnonnegative integer n0 such that f(n) c g(n) for every n n0 g(

Design and Analysis of Algorithms Unit I

1-52

BigBig-oh

Design and Analysis of Algorithms Unit I

1-53

;-notation
Formal definition A function t(n) is said to be in ;(g(n)), denoted t(n) ;(g(n)), if t(n) is bounded below by some constant multiple of g(n) for all large n, i.e., if there exist some positive constant c and some nonnegative integer n0 such that t(n) u cg(n) for all n u n0

Design and Analysis of Algorithms Unit I

1-54

BigBig-omega

Design and Analysis of Algorithms Unit I

1-55

5-notation
Formal definition A function t(n) is said to be in 5(g(n)), denoted t(n) 5(g(n)), if t(n) is bounded both above and below by some positive constant multiples of g(n) for all large n, i.e., if there exist some positive constant c1 and c2 and some nonnegative integer n0 such that c2 g(n) e t(n) e c1 g(n) for all n u n0

Design and Analysis of Algorithms Unit I

1-56

BigBig-theta

Design and Analysis of Algorithms Unit I

1-57

Theorem
If t1(n) O(g1(n)) and t2(n) O(g2(n)), then t1(n) + t2(n) O(max{g1(n), g2(n)}). The analogous assertions are true for the ;-notation and 5-notation. Proof. There exist constants c1, c2, n1, n2 such that t1(n) e c1*g1(n), for all n u n1 t2(n) e c2*g2(n), for all n u n2 Define c3 = c1 + c2 and n3 = max{n1,n2}. Then t1(n) + t2(n) e c3*max{g1(n), g2(n)}, for all n u n3

Design and Analysis of Algorithms Unit I

1-58

Some properties of asymptotic order of growth

f(n) O(f(n)) O(f f(n) O(g(n)) iff g(n) ;(f(n)) O(g ;( If f (n) O(g (n)) and g(n) O(h(n)) , then f(n) O(h(n)) O(g O(h O(h Note similarity with a b If f1(n) O(g1(n)) and f2(n) O(g2(n)) , then O(g O(g f1(n) + f2(n) O(max{g1(n), g2(n)}) O(max{g Also, 71eien 5(f(i)) = 5 (71eien f(i))
Design and Analysis of Algorithms Unit I
1-59

Establishing order of growth using limits

0 order of growth of T(n) < order of growth of g(n)

lim T(n)/g(n) = )/g n

c > 0 order of growth of T(n) = order of growth of g(n) order of growth of T(n) > order of growth of g(n)

Design and Analysis of Algorithms Unit I

1-60

LHpitals rule and Stirlings formula

LHpitals rule: If limnpg f(n) = limnpg g(n) = g and g(n) the derivatives f, g exist, then lim
npg

f(n) g(n)

lim
npg

f (n) g (n)

Example: log n vs. n

Stirlings formula: n! } (2Tn)1/2 (n/e)n (2T Example: 2n vs. n!

Design and Analysis of Algorithms Unit I

1-61

Orders of growth of some important functions

All logarithmic functions loga n belong to the same class 5(log n) no matter what the logarithms base a > 1 is because

log a n ! log b n / log b a

All polynomials of the same degree k belong to the same class: aknk + ak-1nk-1 + + a0 5(nk) Exponential functions an have different orders of growth for different as order log n < order nE (E>0) < order an < order n! < order nn
Design and Analysis of Algorithms Unit I
1-62

Basic asymptotic efficiency classes

1 log n n n log n n2 n3 2n n!
Design and Analysis of Algorithms Unit I

constant logarithmic linear n-log-n logquadratic cubic exponential factorial

1-63

Plan for analyzing nonrecursive algorithms

General Plan for Analysis Decide on parameter n indicating input size Identify algorithms basiyc operation Determine worst, average, and best cases for input of size n worst, average, Set up a sum for the number of times the basic operation is executed Simplify the sum using standard formulas and rules (see Appendix A)
Design and Analysis of Algorithms Unit I

1-64

Useful summation formulas and rules

7leien1 = 1+1++1 = n - l + 1 In particular, 7leien1 = n - 1 + 1 = n 5(n) 71eien i = 1+2++n = n(n+1)/2 } n2/2 5(n2) 1+2++n 71eien i2 = 12+22++n2 = n(n+1)(2n+1)/6 } n3/3 5(n3) ++n +1)(2n 70eien ai = 1 + a ++ an = (an+1 - 1)/(a - 1) for any a { 1 (a 1)/(a In particular, 70eien 2i = 20 + 21 ++ 2n = 2n+1 - 1 5(2n ) 7(ai bi ) = 7ai 7bi 7cai = c7ai 7leieuai = 7leiemai + 7m+1eieuai +1e
1-65

Design and Analysis of Algorithms Unit I

Example 1: Maximum element

T(n) = 71eien-1 1 = n-1 = 5(n) comparisons n-

Design and Analysis of Algorithms Unit I

1-66

Example 2: Element uniqueness problem

T(n) = 70eien-2 (7i+1ejen-1 1) i+1e = 70eien-2 n-i-1 = (n-1+1)(n-1)/2 = 5( n 2 ) comparisons

Design and Analysis of Algorithms Unit I
1-67

Example 3: Matrix multiplication

T(n) = 70eien-1 70eien-1 n = 70eien-1 5( n 2 ) = 5( n 3 ) multiplications

Design and Analysis of Algorithms Unit I
1-68

Example 4: Gaussian elimination

Algorithm GaussianElimination(A[0..n-1,0..n]) GaussianElimination( [0..n 1,0..n //Implements Gaussian elimination on an n-by-(n+1) matrix A byfor i n 0 to n - 2 do for j n i + 1 to n - 1 do for k n i to n do A[j,k] n A[j,k] - A[i,k]  A[j,i] / A[i,i] Find the efficiency class and a constant factor improvement.
for i n 0 to n - 2 do for j n i + 1 to n - 1 do B n A[j,i] / A[i,i] j,i] for k n i to n do A[j,k] n A[j,k] A[i,k] * B i,k]
Design and Analysis of Algorithms Unit I
1-69

Example 5: Counting binary digits

Design and Analysis of Algorithms Unit I

1-70

Plan for Analysis of Recursive Algorithms

Decide on a parameter indicating an inputs size. Identify the algorithms basic operation. Check whether the number of times the basic op. is executed may vary on different inputs of the same size. (If it may, the worst, average, and best cases must be investigated separately.) Set up a recurrence relation with an appropriate initial condition expressing the number of times the basic op. is executed. Solve the recurrence (or, at the very least, establish its solutions order of growth) by backward substitutions or another method.
Design and Analysis of Algorithms Unit I
1-71

Example 1: Recursive evaluation of n!

Definition: n ! = 1  2  (n-1)  n for n 1 and 0! = 1 Recursive definition of n!: F(n) = F(n-1)  n for n 1 and F(0) = 1

Size: Basic operation: Recurrence relation:

Design and Analysis of Algorithms Unit I

n multiplication M(n) = M(n-1) + 1 M(0) = 0

1-72

Solving the recurrence for M(n) M(n

M(n M(n M(n) = M(n-1) + 1, M(0) = 0
M(n) = M(n-1) + 1 = (M(n-2) + 1) + 1 = M(n-2) + 2 = (M(n-3) + 1) + 2 = M(n-3) + 3 = M(n-i) + i = M(0) + n =n The method is called backward substitution.
Design and Analysis of Algorithms Unit I
1-73

Example 2: The Tower of Hanoi Puzzle

Recurrence for number of moves:

Design and Analysis of Algorithms Unit I

M(n) = 2M(n-1) + 1
1-74

Solving recurrence for number of moves

M(n 2M(n M(n) = 2M(n-1) + 1, M(1) = 1
M(n) = 2M(n-1) + 1 = 2(2M(n-2) + 1) + 1 = 2^2*M(n-2) + 2^1 + 2^0 = 2^2*(2M(n-3) + 1) + 2^1 + 2^0 = 2^3*M(n-3) + 2^2 + 2^1 + 2^0 = = 2^(n-1)*M(1) + 2^(n-2) + + 2^1 + 2^0 = 2^(n-1) + 2^(n-2) + + 2^1 + 2^0 = 2^n - 1
Design and Analysis of Algorithms Unit I
1-75

Tree of calls for the Tower of Hanoi Puzzle

n n-1 n-2 2 1 n-2 2 1 1 n-2 n-1

n-2 2 1 1

...
1

...

...

2 1

Design and Analysis of Algorithms Unit I

1-76

Example 3: Counting #bits

A(n) = A( n / 2 ) + 1, A(1) = 0 A(2 k ) = A( 2 k 1 + 1, A( 2 0 = 1 (using the Smoothness Rule) ) ) = (A( 2 k 2) + 1) + 1 = A( 2 k 2) + 2 = A(2 k i) + i = A( 2 k k) + k = k + 0 = log 2 n
Design and Analysis of Algorithms Unit I
1-77