ANALYSIS AND

DESIGN OF
ALGORITHMS
(BCS401)

2025
_______________________________________________________________________________________________
 TITLE :
MODULE 1 :
MODULE 2 :
MODULE 3 :
MODULE 4 :
MODULE 5 :
QUESTION PAPERS :
Q1 :
Q2 :
Q3 :
Q4 :
Q5 :
Q6 :
Q7 :
_________________________________________
 Lecture Notes on
Analysis and Design of Algorithms
BCS401

Module-1: Introduction to Algorithm

Contents
1. Introduction
1.1. What is an Algorithm?
1.2. Algorithm Specification
1.3. Analysis Framework
2. Performance Analysis
2.1. Space complexity
2.2. Time complexity
3. Asymptotic Notations
3.1. Big-Oh notation
3.2. Omega notation
3.3. Theta notation
3.4. Little-oh notation
3.5. Basic symptotic notations
3.6. Mathematical analysis of Recursive and non-recursive algorithm
4. Brute force design technique:
4.1. Selection sort
4.2. Sequential search
4.3. String matching algorithm with complexity Analysis.

1
ADA BCS401

Module-1: Introduction
1.1 Introduction
1.1.1 What is an Algorithm?
Algorithm: An algorithm is a finite sequence of unambiguous instructions to solve a particular
problem.

a. Input. Zero or more quantities are externally supplied.

b. Output. At least one quantity is produced.
c. Definiteness. Each instruction is clear and unambiguous. It must be perfectly clear what
should be done.
d. Finiteness. If we trace out the instruction of an algorithm, then for all cases, the algorithm
terminates after a finite number of steps.
e. Effectiveness. Every instruction must be very basic so that it can be carried out, in principle,
by a person using only pencil and paper. It is not enough that each operation be definite as in
criterion c; it also must be feasible.
1.1.2. Algorithm Specification
An algorithm can be specified in
1) Simple English
2) Graphical representation like flow chart
3) Programming language like C++/java
4) Combination of above methods.
Example: Combination of simple English and C++, the algorithm for selection sort is specified as
follows.

Example: In C++ the same algorithm can be specified as follows. Here Type is a basic or user
defined data type.

CSE, SVIT 1

2
ADA BCS401

1.1.3. Analysis Framework

Measuring an Input’s Size
It is observed that almost all algorithms run longer on larger inputs. For example, it takes longer to
sort larger arrays, multiply larger matrices, and so on. Therefore, it is logical to investigate an
algorithm's efficiency as a function of some parameter n indicating the algorithm's input size.
There are situations, where the choice of a parameter indicating an input size does matter. The
choice of an appropriate size metric can be influenced by operations of the algorithm in question. For
example, how should we measure an input's size for a spell-checking algorithm? If the algorithm
examines individual characters of its input, then we should measure the size by the number of
characters; if it works by processing words, we should count their number in the input.
We should make a special note about measuring the size of inputs for algorithms involving properties
of numbers (e.g., checking whether a given integer n is prime). For such algorithms, computer
scientists prefer measuring size by the number b of bits in the n's binary representation:= log2 n ] + 1.
This metric usually gives a better idea about the efficiency of algorithms in question.

Units for Measuring Running lime

To measure an algorithm's efficiency, we would like to have a metric that does not depend on these
extraneous factors. One possible approach is to count the number of times each of the algorithm's
operations is xecuted. This approach is both excessively difficult and, as we shall see, usually
unnecessary. The thing to do is to identify the most important operation of the algorithm, called the
basic operation, the operation contributing the most to the total running time, and compute the number
of times the basic operation is executed.
For example, most sorting algorithms work by comparing elements (keys) of a list being sorted with
each other; for such algorithms, the basic operation is a key comparison.
As another example, algorithms for matrix multiplication and polynomial evaluation require two
arithmetic operations: multiplication and addition.
Let cop be the execution time of an algorithm's basic operation on a particular computer, and let C(n) be
the number of times this operation needs to be executed for this algorithm. Then we can estimate the
running time T(n) of a program implementing this algorithm on that computer by the formula:
𝑇(𝑛) ≈ 𝑐𝑜𝑝𝐶(𝑛)
Unless n is extremely large or very small, the formula can give a reasonable estimate of the algorithm's
running time.
It is for these reasons that the efficiency analysis framework ignores multiplicative constants and
concentrates on the count's order of growth to within a constant multiple for large-size inputs.

CSE, SVIT 2

3
ADA BCS401

Orders of Growth
Why this emphasis on the count's order of growth for large input sizes? Because for large values of n, it
is the function's order of growth that counts: just look at table which contains values of a few functions
particularly important for analysis of algorithms.
Table: Values of several functions important for analysis of algorithms

Algorithms that require an exponential number of operations are practical for solving only problems of
very small sizes.

1.2. Performance Analysis

There are two kinds of efficiency: time efficiency and space efficiency.
● Time efficiency indicates how fast an algorithm in question runs;
● Space efficiency deals with the extra space the algorithm requires.
In the early days of electronic computing, both resources time and space were at a premium. The
research experience has shown that for most problems, we can achieve much more spectacular progress
in speed than inspace. Therefore, we primarily concentrate on time efficiency.
1.2.1 Space complexity
Total amount of computer memory required by an algorithm to complete its execution is called as space
complexity of that algorithm. The Space required by an algorithm is the sum of following components
● A fixed part that is independent of the input and output. This includes memory space for codes,
variables, constants and so on.
● A variable part that depends on the input, output and recursion stack. ( We call these
parameters as instance characteristics)
Space requirement S(P) of an algorithm P, S(P) = c + Sp where c is a constant depends on the fixed
part, Sp is the instance characteristics\
Example-1: Consider following algorithm abc()

Here fixed component depends on the size of a, b and c. Also instance characteristics Sp=0

CSE, SVIT 3

4
ADA BCS401

Example-2: Let us consider the algorithm to find sum of array. For the algorithm given here the
problem instances are characterized by n, the number of elements to be summed. The space needed by
a[ ]depends on n.So the space complexity can be written as;Ssum(n) ≥ (n+3); n for a[ ], One each for
n, i and s.

1.2.2 Time complexity

Usually, the execution time or run-time of the program is refereed as its time complexity denoted by
tp(instance characteristics). This is the sum of the time taken to execute all instructions in the program.
Exact estimation runtime is a complex task, as the number of instructions executed is dependent on the
input data. Also different instructions will take different time to execute. So for the estimation of the
time complexity we count only the number of program steps. We can determine the steps needed
by a program to solve a particular problem instance in two ways.
Method-1: We introduce a new variable count to the program which is initialized to zero. We also
introduce statements to increment count by an appropriate amount into the program. So when each
time original program executes, the count also incremented by the step count.
Example: Consider the algori hm sum(). After the introduction of the count the program will be as
follows. We can estimate that invocation of sum() executes total number of 2n+3 steps.

Method-2: Determine the step count of an algorithm by building a table in which we list the total
number of steps contributed by each statement. An example is shown below. The code will find the sum
of n numbers.
Example: Matrix addition

CSE, SVIT 4

5
ADA BCS401

The above method is both excessively difficult and, usually unnecessary. The thing to do is to identify the most
important operation contributing the m operation of the algorithm, called the basic operation, the st to the total
running time, and compute the number of times the basic operation is executed.

Trade-off
There is often a time-space-tradeoff involved in a problem, that is, it cannot be solved with few
computing time and low memory consumption. One has to make a compromise and to exchange
computing time for memory consumption or vice versa, depending on which algorithm one chooses
and how one parameterizes it.

1.3. Asymptotic Notations

The efficiency analysis framework concentrates on the order of growth of an algorithm’s basic operation count
as the principal indicator of the algorithm’s efficiency. To compare and rank such orders of growth, computer
scientists use three notations: O(big oh), Ω(big omega), Θ (big theta) and o(little oh)

1.3.1. Big-Oh notation

Definition: A function t(n) is said to be in O(g(n)),
denoted t(n)∈O(g(n)), if t (n) is bounded above by
some constant multiple of g(n) for all large n, i.e., if
there exist some positive constant c and some
nonnegative integer n0 such that
t(n) ≤ cg(n) for all n ≥ n0.

Informally, O(g(n)) is the set of all functions with a lower or same order of growth as g(n). Note
that the definition gives us a lot of freedom in choosing specific valuesfor constants c and n0.

Examples: 𝑛 є (𝑛2), 100𝑛 + 5 є (𝑛2), 1 (𝑛 − 1)є𝑂(𝑛2)

2

𝑛3 ∉ (𝑛2), 0.00001𝑛3 ∉ (𝑛2), 𝑛4 + 𝑛 + 1 ∉ (𝑛2)

CSE, SVIT 5

6
ADA BCS401

Strategies to prove Big-O: Sometimes the easiest way to prove that f (n) = O(g(n)) is to take c to
be the sum of the positive coefficients off(n). We can usually ignore the negative coefficients.

Example: To prove 100n + 5 ∈ O(n2)

100n + 5 ≤ 105n2. (c=105, n0=1)
Example: To prove n2 + n = O(n3)
Take c = 1+1=2, if n ≥n0=1, then n2 + n
= O(n3)

i) Prove 3n+2=O(n) ii) Prove 1000n2+100n-6 = O(n2)

1.3.2. Omega notation
Definition: A function t(n) is said to be in Ω(g(n)),
denoted t(n)∈Ω(g(n)), if t(n) is bounded below by
some positive constant multiple of g(n) for all large
n,i.e., if there exist some positive constant c and some
nonnegative integer n0 suchthatt(n) ≥ c g(n) for
all n ≥ n0.

Here is an example of the formal proof that n3 ∈Ω(n2):n3 ≥ n2 for all n ≥ 0, i.e., we
can select c = 1 and n0 = 0.

Example:

Example: To prove n3 + 4n2 = Ω(n2)

We see that, if n≥0, n3+4n2≥ n3≥ n2; Therefore n3+4n2 ≥ 1n2for
alln≥0 Thus, we have shown that n3+4n2 =Ω(n2) where c = 1 &
n0=0

1.3.3. Theta notation

A function t(n) is said to be in Θ(g(n)), denoted t(n)
∈ Θ(g(n)),if t (n) is bounded both above and below
by some positive constant multiples ofg(n)
for all large n, i.e., if there exist some positive
constants c1 and c2 and somenonnegative integer n0
such that

c2g(n) ≤ t(n) ≤c1g(n) for all n ≥ n0.

CSE, SVIT 6

7
ADA BCS401

CSE, SVIT 7

8
ADA BCS401

Example: n2 + 5n + 7 = Θ(n2)

Strategies for Ω and Θ

● Proving that a f(n) = Ω(g(n)) often requires more thought.
– Quite often, we have to pick c < 1.
– A good strategy is to pick a value of c which you think will work, and determine
which value of n0 is needed.
– Being able to do a little algebra helps.
– We can sometimes simplify by with the positive
ignoring terms of f(n) coefficients.
● The following theorem shows us that proving f(n) = Θ(g(n)) is nothing new:
Theorem: f(n) = Θ(g(n)) if and only iff(n) = O(g(n)) and f(n) = Ω(g(n)).
Thus, we just apply the previous two strategies.

CSE, SVIT 8

9
ADA BCS401

Theorem: If t1(n) ∈O(g1(n)) and t2(n) ∈O(g2(n)), then t1(n) + t2(n) ∈O(max{g1(n), g2(n)}). (The
analogous assertions are true for the Ω and Ө notations as well.)
Proof: The proof extends to orders of growth the following simple fact aboutfour arbitrary real numbers
a1, b1, a2, b2: if a1 ≤ b1 and a2 ≤ b2, then a1 + a2 ≤ 2 max{b1, b2}.
Since t1(n) ∈O(g1(n)), there exist some positive constant c1 and some nonnegative integer n1 such that
t1(n) ≤ c1g1(n) for all n ≥ n1.
Similarly, since t2(n) ∈O(g2(n)), t2(n) ≤ c2g2(n) for all n ≥ n2.
Let us denote c3 = max{c1, c2} and consider n>=max{n1,n2} so that
we can use both inequalities. Adding them yields the following: t1(n) + t2(n) ≤ c1g1(n) + c2g2(n)
≤ c3 g1(n) + c3g2(n) = c3[g1(n) + g2(n)]

≤ c32 max{g1(n), g2(n)}.

Hence, t1(n) + t2(n) ∈ O(max{g1(n), g2(n)}), with the constants c and n0 required by the O
definition being 2c3 = 2 max{c1, c2} and max{n1, n2}, respectively.

3.4. Little Oh The function f(n)= o(g(n)) [ i.e f of n is a little oh of g of n ] if and only if

lim ƒ(𝑛) = 0

𝑛→∞ 𝑔(𝑛)
Example:

For comparing the order of growth limit is used

If the case-1 holds good in the above limit, we represent it by little-oh.

CSE, SVIT 9

10
ADA BCS401

1.3.5. Basic asymptotic efficiency Classes

Class Name Comments

CSE, SVIT 10

11
ADA BCS401

1.3.6. Mathematical Analysis of Non-recursive & Recursive Algorithms

Analysis of Non-recursive Algorithms
General Plan for Analyzing the Time Efficiency of Nonrecursive Algorithms
1. Decide on a parameter (or parameters) indicating an input’s size.
2. Identify the algorithm’s basic operation. (As a rule, it is located in innermost loop.)
3. Check whether the number of times the basic operation is execut d depends only on the size of
an input. If it also depends on some additional property, the worst-case, average-case, and, if
separately.
4. Set up a sum expressing the number of times the algorithm’s executed.
5. Using standard formulas and rules of sum manipulation, either
Example-1: To find maximum element in the given array

Here comparison is the basic operation. Note that number of comparisions will be same for all
arrays of size n. Therefore, no need to distinguish worst, best and average cases. Total number of
basic operations are,

Example-2: To check whether all the elements in the given array are distinct

CSE, SVIT 11

12
ADA BCS401

Here basic operation is comparison. The maximum no. of comparisons happens in the worst case. i.e.
all the elements in the array are distinct and algorithms return true).
Total number of basic operations (comparison) in the worst case are,

Other than the worst case, the total comparisons areless than 1 𝑛2. For example if the first two elements
2
of the array are equal, only one comparison is computed. So in general C(n) =O(n2)

Example-3: To perform matrix multiplication

Number of basic operations (multiplications) is

Total running time:

Suppose if we take into account of addition; Algoritham also have same number of additions
A(n) = n3

Total running time:

CSE, SVIT 12

13
ADA BCS401

Example-4: To count the bits in the binary representation

The basic operation is count=count + 1 repeats no. of times

Analysis of Recursive Algorithms
General plan for analyzing the time efficiency of recursive algorithms

1. Decide on a parameter (or parameters) indicating an input’s size.

2. Identify the algorithm’s basic operation.
3. Check whether the number of times the basic operation is executed can varyon different inputs of the same
size; if it can, the worst-case, average-case, and best-case efficiencies must be investigated separately. Set
up a recurrence relation, with an appropriate initial condition, for the number of times the basic operation
is executed.
4. Solve the recurrence or, at least, ascertain the order of growth of its solution.

Example-1

Since the function F(n) is computed according to the formula

The number of multiplicationsM(n) needed to compute it must satisfy the equality

Such equations are called recurrence relations

CSE, SVIT 13

14
ADA BCS401

Condition that makes the algorithm stopif n = 0 return 1. Thus recurrence relation and initial
conditionfor the algorithm’s number of multiplications M(n) can be stated as

We can use backward substitutions method to solve this

….

Example-2: Tower of Hanoi puzzle. In this puzzle, There are n disks of different sizes that canslide
onto any of three pegs. Initially, all the disks are on the first peg in order ofsize, the largest on the bottom
and the smallest on top. The goal is to move all thedisks to the third peg, using the second one as an
auxiliary, if necessary. We canmove only one disk at a time, and it is forbidden to place a larger disk on
top of asmaller one.The problem has an elegant recursive solution, which is illustrated in Figure.

1. If n = 1, we move the single disk directly from the source peg to the destination peg.
2. To move n>1 disks from peg 1 to peg 3 (with peg 2 as auxiliary),
o we first move recursively n-1 disks from peg 1 to peg 2 (with peg 3 as auxiliary),
o then move the largest disk directly from peg 1 to peg 3, and,
o finally, move recursively n-1 disks from peg 2 to peg 3 (using peg 1 as auxiliary).

Figure: Recursive solution to the Tower of Hanoi puzzle

Algorithm: TowerOfHanoi(n, source, dest, aux)

If n == 1, THEN
move disk from
source to dest else
TowerOfHanoi (n - 1, source, aux, dest)
move disk from source to dest
TowerOfHanoi (n - 1, aux, dest, source)
End if

CSE, SVIT 14

15
ADA BCS401

Computation of Number of Moves

The number of moves M(n) depends only on n. The recurrence equation is

We have the following recurrence relation for the number of moves M(n):

We solve this recurrence by t e same method of backward substitutions:

The pattern of the first three sums on the left suggests that the next one will be

24M(n − 4) + 23 + 22 + 2 + 1, and generally, after i substitutions, we get

Since the initial condition is specified for n = 1, which is achieved for i = n-1, we get the
following formula for the solution to recurrence,

Example-3: To count bits of a decimal number in its binary representation

The recurrence relation can be written as

.
Also note that A(1) = 0.

The standard approach to solving such a recurrence is to solve it only for n = 2k and then take
advantage of the theorem called the smoothness rule which claims that under very broad

CSE, SVIT 15

16
ADA BCS401

assumptions the order of growth observed for n = 2k gives a correct answer about the order of
growth for all values of n.

CSE, SVIT 16

17
ADA BCS401

1.4. Brute force design technique:

Brute force is straight forward approach to solving a problem, usually directly based on the
problem statement and definitions of the concepts involved.

1.4.1 Selection sort

We start selection sort by scanning the entire given list to find its smallest element and exchange it with
the first element, putting the smallest element in its final position in the sorted list. Then we scan the list, starting
with the second element, putting the second smallest element in its final position. Generally, on the ith pass
through the list, which we number from 0 to n-2, the algorithm searches for the last n-I elements and swaps it with
Ai:

A0 ≤ A1 ≤ …… ≤Ai-1 | Ai…….Amin ….. An-1

in their final positions the last n-i elements

After n-1 passes, the list is sorted.

The number of times the algorithm executed depends only on the array’s size and is given by

After solving using summation formulas

Thus selection sort has a Θ(n2) time complexity.

1.4.2 Sequential search

This is also called as Linear search. Here we start from the initial element of the array and compare it with the
search key. We repeat the same with all the elements of the array till we encounter the search key or till we reach
end of the array.

CSE, SVIT 17

18
ADA BCS401

The time efficiency in worst case is O(n), where n is the number of elements of the array. In best case it is O(1), it
means the very first element is the search key.

1.4.3 String matching algorithm with complexity Analysis

Another example of Brute force approach is string matching, where string of n characters called text and a string
of m characters (m<=n) called the pattern is given. Here job is to find whether the pattern is present in text or not.
If we want to find i-the index of the leftmost character of the first matching substring in the

We start matching with the very first character, if a match then only j is incremented and again compared with next
character of both the strings. If not then I is incremented and j starts from beginning of pattern string. If pattern
found we return the position from where the pattern began. Pattern is tried to match till n-m elements, later we
need not try to match as the elements will be lesser than pattern. If it doesn’t match by n-m elements then pattern is
not matched.

The worst case is Θ(nm). Best case is Θ(m).

CSE, SVIT 18

19
 Lecture Notes on
Analysis and Design of Algorithms
BCS401

Module-2 :Divide and Conquer

Contents
1. General method
2. Recurrence equation
3. Algorithm: Binary search
4. Algorithm: Finding the maximum and minimum
5. Algorithm: Merge sort
6. Algorithm: Quick sort
7. Advantages and Disadvantages
8. Decrease and Conquer Approach
9. Algorithm: Topological Sort

20
ADA BCS401

1. General method
Divide and Conquer is one of the best-known general algorithm design echnique. It works
according to the following general plan:
 Given a function to compute on ‘n’ inputs the divide-and-conquer strategy suggests
splitting the inputs into ‘k’ distinct subsets, 1<k<=n, yielding ‘k’ sub problems.
 These sub problems must be solved, and then a method must be found to combine sub
solutions into a solution of the whole.
 If the sub problems are still relatively large, then the divide-and-conquer strategy can
possibly be reapplied.
 Often the sub problems resulting from a divide-and-conquer design are of the same
type as the original problem.For those cases the reapplication of the divide-and-
conquer principle is naturally expressed by a recursive algorithm.

A typical case with k=2is diagrammatically shown below.

Problem of size n

Sub Problem of size n/2 Sub Problem of size n/2

Solution to sub problem 1 Solution to sub problem 2

Solution to the original problem

Control Abstraction for divide and conquer:

In the above specification,

 Initially DAndC(P) is invoked, where ‘P’ is the problem to be solved.
 Small (P) is a Boolean-valued function that determines whether the input size is small
enough that the answer can be computed without splitting. If this so, the function ‘S’
is invoked.Otherwise, the problem P is divided into smaller sub problems.These sub
problems P1, P2 …Pk are solved by recursive application of DAndC.
 Combine is a function that determines the solution to P using the solutions to the ‘k’
sub problems.

CSE, SVIT 1

21
ADA BCS401

2. Recurrence equation for Divide and Conquer

If the size of problem ‘p’ is n and the sizes of the ‘k’ sub problems are n1, n2….nk,
respectively, then the computing time of divide and conquer is described by the recurrence
relation

Where,
 T(n) is the time for divide and conquer method on any input of size n and
 g(n) is the time to compute answer directly for small inputs.
 The function f(n) is the time for dividing the problem ‘p’ and combining the solutions
to sub problems.
For divide and conquer based algorithms that produce sub problems of the same type as the
original problem, it is very natural to first describe them by using recursion.
More generally, an instance of size n can be divided into b instances of size n/b, with a of
them needing to be solved. (Here, a and b are constants; a>=1 and b > 1.). Assuming that
size n is a power of b(i.e. n=bk), to simplify our analysis, we get the following recurrence for
the running time T(n):
..... (1)
where f(n) is a function that accounts for the time spent on dividing the problem into smaller
ones and on combining their solutions.
The recurrence relation can be solved by i) substitution method or by using ii) master
theorem.
1. Substitution Method - This method repeatedly makes substitution for each
occurrence of the function T in the right-hand side until all such occurrences
disappear.
2. Master Theorem - The efficiency analysis of many divide-and-conquer algorithms is
greatly simplified by the master theorem. It states that, in recurrence equation T(n) =
aT(n/b) + f (n), If f (n)∈ Θ (nd) where d ≥ 0 then

Analogous results hold for the Ο and Ω notations, too.

For example, the recurrence for the number of additions A(n) made by thedivide-and-
conquer sum-computation algorithm (see above) on inputs of sizen = 2k is

Thus, for this example, a = 2, b = 2, and d = 0; hence, since a >bd,

CSE, SVIT 2

22
ADA BCS401

Problems on Substitution method & Master theorem to solve the recurrence relation

CSE, SVIT 3

23
ADA BCS401

CSE, SVIT 4

24
ADA BCS401

CSE, SVIT 5

25
ADA BCS401

CSE, SVIT 6

26
ADA BCS401

3. Binary Search
Problem definition:Let ai, 1 ≤ i ≤ n be a list of elements that are sorted in non-decreasing
order. The problem is tofind whether a given element x is present in the list or not. If x is
present we have to determine a value j (element’s position) such that aj=x. If x is not in the
list, then j is set to zero.
Solution:Let P = (n, ai…al , x) denote an arbitrary instance of search problem where n is the
number of elements in the list, ai…alis the list of elements andx is the key element to be
searched for in the given list.Binary search on the list is done as follows:
Step1: Pick an index q in the middle range [i, l] i.e. q= (𝑛 + 1)/2] and compare x with aq.
Step 2: if x = aqi.e key element is equal to mid element, the problem is immediately solved.
Step 3: if x <aqin this case x has to be searched for only in the sub-list ai, ai+1, ……, aq-
Therefore, problem reduces to (q-i, ai…aq-1, x).
Step 4: if x >aq,x has to be searched for only in the sub-list aq+1, ...,., al . Therefore problem
reduces to (l-i, aq+1…al, x).
For the above solution procedure, the Algorithm can be implemented as recursive or non-
recursive algorithm.

CSE, SVIT 7

27
ADA BCS401

Recursive binary search algorithm

Iterative binary search:

CSE, SVIT 8

28
ADA BCS401

Analysis
In binary search the basic operation is key comparison. Binary Search can be analyzed with
the best, worst, and average case number of comparisons.The numbers of comparisons for the
recursive and iterative versions of Binary Search are the same, if comparison counting is
relaxed slightly. For Recursive Binary Search, count each pass through the if-then-else block
as one comparison. For Iterative Binary Search, count each pass through the while block as
one comparison.Let us find out how many such key comparison does the algorithm make on
an array of n elements.
Best case –Θ(1) In the best case, the key is the middle in the array. A constant number of
comparisons (actually just 1) are required.
Worst case - Θ(log2 n) In the worst case, the key does not exist in the array at all. Through
each recursion or iteration of Binary Search, the size of the admissible range is halved. This
halving can be done ceiling(log2n ) times. Thus, ] log2 n ] comparisons are required.
Sometimes, in case of the successful search, it may take maximum number of comparisons.
] log2 n ]. So worst case complexity of successful binary search is Θ (log2 n).
Average case - Θ (log2n) To find the average case, take the sum of the product of number of
comparisons required to find each element and the probability of searching for that element.
To simplify the analysis, assume that no item which is not in array will be searched for, and
that the probabilities of searching for each element are uniform.

Space Complexity - The space requirements for the recursive and iterative versions of binary
search are different. Iterative Binary Search requires only a constant amount of space, while
Recursive Binary Search requires space proportional to the number of comparisons to
maintain the recursion stack.
Advantages: Efficient on very big list, Can be implemented iteratively/recursively.
Limitations:
 Interacts poorly with the memory hierarchy
 Requires sorted list as an input
 Due to random access of list element, needs arrays instead of linked list.

CSE, SVIT 9

29
ADA BCS401

4. Finding the maximum and minimum

Problem statement:Given a list of n elements, the problem is to find the maximum and
minimum items.
StraightMaxMin: A simple and straight forward algorithm to achieve this is given below.

Explanation:
 StraightMaxMin requires 2(n-1) comparisons in the best, average & worst cases.
 By realizing the comparison of a[i]>max is false, improvement in a algorithm can be
done. Hence we can replace the contents of the for loop by,
If(a[i]>Max) then Max = a[i]; Else if (a[i]<min) min=a[i]
 On the average a[i] is > max half the time.So, the avg. no. of comparison is 3n/2-1.

Algorithm based on Divide and Conquer strategy

Let P = (n, a [i],……,a [j]) denote an arbitrary instance of the problem. Here ‘n’ is the no. of
elements in the list (a[i],….,a[j]) and we are interested in finding the maximum and minimum
of the list.If the list has more than 2 elements, P has to be divided into smaller instances.
For example, we might divide ‘P’ into the 2 instances,
P1=( [n/2],a[1], a[n/2])
P2= ( n-[n/2], a[[n/2]+1], , a[n])
After having divided ‘P’ into 2 smaller sub problems, we can solve them
byrecursivelyinvoking the same divide-and-conquer algorithm.
Algorithm:

CSE, SVIT 10

30
ADA BCS401

Example:

of recursive calls of MaxMin is as follows

Analysis - Time Complexity

CSE, SVIT 11

31
ADA BCS401

Compared with the straight forward method (2n-2) this method saves 25% in comparisons.

Space Complexity
Compared to the straight forward method, the MaxMin method requires extra stack space for
i, j, max, min, max1 and min1. Given n elements there will be 𝑙o𝑔2𝑛] + 1 levels of
recursion and we need to save seven values for each recursive call. (6 + 1 for return address).

5. Merge Sort
Merge sort is a perfect example of a successful application of the divide-and conquer
technique. It sorts a given array A [O ... n - 1] by dividing it into two halves A [0 .. 𝑛/2]-1]
and A [ /2] .. n-1], sorting each of them recursively, and then merging the two smaller sorted
arrays into a single sorted one.

The merging of two sorted arrays can be done as follows.

 Two pointers (array indices) are initialized to point to the first elements of the arrays
being merged.
 The elements pointed to are compared, and the smaller of them is added to a new
array being constructed

CSE, SVIT 12

32
ADA BCS401

 After that, the index of the smaller element is incremented to point to its immediate
successor in the array it was copied from. This operation is repeated until one of the
two given arrays is exhausted, and then the remaining elements of the other array are
copied to the end of the new array.

Example:
The operation of the algorithm on the
list 8, 3, 2, 9, 7, 1, 5, 4 is illustrated in
the figure

Analysis

Here the basic operationis key comparison. As merge sort execution does not depend on the
order of the data, best case and average case runtime are the same as worst case runtime.

Worst case:During key comparison, neither of the two arrays becomes empty before the
other one contains just one element leads to the worst case of merge sort. Assuming for

CSE, SVIT 13

33
ADA BCS401

simplicity that total number of elementsn is a power of 2, therecurrence relation for the
number of key comparisons C(n) is

where, Cmerge(n) is the number of key comparisons made during the merging stage.
Let us analyze Cmerge(n), the number of key comparisons performed during themerging stage.
At each step, exactly one comparison is made, after which the totalnumber of elements in the
two arrays still needing to be processed is reducedby 1. In the worst case, neither of the two
arrays becomes empty before theother one contains just one element (e.g., smaller elements
may come from thealternating arrays).Therefore, for the worst case,Cmerge(n) = n –1.
Now,

Solving the recurrence equation using master theorem:

Here a = 2, b = 2, f (n) = n, d = 1. Therefore 2 = 21, case 2 holds in the master theorem
Cworst(n) = Θ (nd log n) = Θ (n1 log n) = Θ (n log n)Therefore Cworst(n) = Θ (n log n)

Advantages:
 Number of comparisons performed is nearly optimal.
 For large n, the num er of comparisons made by this algorithm in the average case
turns out to be about 0.25n less and hence is also in Θ(n log n).
 Mergesort will never degrade to O (n2)
 Anotheradvantage of mergesort over quicksort and heapsort is its stability. (A sorting
algorithm is said to be stable if two objects with equal keys appear in the same order
in sorted output as they appear in the input array to be sorted. )
Limitations:
 The principal shortcoming of mergesort is the linear amount [O(n) ] of extra storage
the algorithm requires. Though merging can be done in-place, the resulting algorithm
is quite complicated and of theoretical interest only.

Variations of merge sort

1. Thealgorithm can be implemented bottom up by merging pairs of the array’s
elements,then merging the sorted pairs, and so on. (If n is not a power of 2, only
slightbookkeeping complications arise.) This avoids the time and space overhead
ofusing a stack to handle recursive calls.
2. We can divide a list to be sortedin more than two parts, sort each recursively, and
then merge them tog ther. Thisscheme, which is particularly useful for sorting files
residing on secondary memorydevices, is called multiway mergesort.

CSE, SVIT 14

34
ADA BCS401

6. Quick sort
Quicksort is the other important sorting algorithm that is based on the divide-and-conquer
approach. Unlike mergesort, which divides its input elements according to their position in
the array, quicksort divides ( or partitions) them according to their value.
A partition is an arrangement of the array’s elements so that all the elements to the left of
some element A[s] are less than or equal to A[s], and all the elements to the right of A[s] are
greater than or equal to it:

Obviously, after a partition is achieved, A[s] will be in its final position in thesorted array,
and we can continue sorting the two subarrays to the left and to theright of A[s]
independently (e.g., by the same method).
In quick sort, the entire workhappens in the division stage, with no work required to combine
the solutions tothe subproblems.

Partitioning
We start by selecting a pivot—an element with respect to whose valuewe are going to divide
the subarray.There are several different strategies forselecting a pivot. We use
thesophisticated method suggested byC.A.R. Hoare, the prominent British computer scientist
who invented quicksort.
Select the subarray’s firstelement: p = A[l].Now scan the subarray from both ends,comparing
the subarray’s elements to the pivot.
 The left-to-right scan, denoted below by index pointer i, starts with the second
element. Since we want elements smaller than the pivot to be in the left part of the
subarray, this scan skips over elements that are smaller than the pivot and stops upon
encountering the first element greater than or equal to the pivot.
 The right-to-left scan, denoted below by index pointer j, starts with the last element of
the subarray. Since we want elements larger than the pivot to be in the right part of the

CSE, SVIT 15

35
ADA BCS401

subarray, this scan skips over elements that are larger than the pivot and stops on
encountering the first element smaller than or equal to the pivot.
After both scans stop, three situations may arise, depending on whether or notthe scanning
indices have crossed.
1. If scanning indices i and j have not crossed, i.e., i< j, we simply exchange A[i] and
A[j ] and resume the scans by incrementing I and decrementing j, respectively:

2. If the scanning indices have crossed over, i.e., i> j, we will have partitioned the
subarray after exchanging the pivot with A[j]:

3. If the scanning indices stop while pointing to the same element, i.e., i = j, the value
they are pointing to must be equal to p. Thus, we have the subarray partitioned, with
the split position s = i = j :

We can combine this with the case-2 byexchanging the pivot with A[j] whenever i≥j

ALGORITHM HoarePartition(A[l..r])
//Partitions a subarray by Hoare’s algorithm, using the first element as a pivot
//Input: Subarray of array A[0..n − 1], defined by its left and right indices l and r (l<r)
//Output: Partition of A[l..r], with the split position returned as this function’s value

Note that index i can go out of the subarray’s bounds in this pseudocode.

CSE, SVIT 16

36
ADA BCS401

Example: Example of quicksort operation. (a) Array’s transformations with pivotsshown in

bold. (b) Tree of recursive calls to Quicksort with input values land r of subarray bounds and
split position s of a partition obtained.

Analysis
Best Case -Here the basic operation is key comparison. Numberof key comparisons made
before a partition is achieved is n + 1 if the scanningindices cross over and n if they coincide.
If all the splits happen in themiddle of corresponding subarrays, we will have the best case.
The number of keycomparisons in the best case satisfies the recurrence,

According to the Master Theorem, Cbest(n) ∈Θ(n log2 n); solving it exactly forn = 2k yields
Cbest(n) = n log2 n.
Worst Case – In the worst case, all the splits will be skewed to the extreme: one of thetwo
subarrays will be empty, and the size of the other will be just 1 less than thesize of the
subarray being partitioned. This unfortunate situation will happen, inparticular, for increasing
arrays. Indeed, if A[0..n − 1] is a strictly increasing array and we use A[0] as thepivot, the

CSE, SVIT 17

37
ADA BCS401

left-to-right scan will stop on A[1] while the right-to-left scan will go allthe way to reach
A[0], indicating the split at position 0:So, after making n + 1 comparisons to get to this
partition and exchanging thepivot A[0] with itself, the algorithm will be left with the strictly
increasing arrayA[1..n − 1] to sort. This sorting of strictly increasing arrays of diminishing
sizes willcontinue until the last one A[n−2.. n−1] has been processed. The total numberof key
comparisons made will be equal to

Average Case - Let Cavg(n) be the average number of key comparisons made byquicksort on
a randomly ordered array of size n. A partition can happen in anyposition s (0 ≤ s ≤ n−1) after
n+1comparisons are made to achieve the partition.After the partition, the left and right
subarrays will have s and n − 1− s elements,respectively. Assuming that the partition split can
happen in each position s withthe same probability 1/n, we get the following recurrence
relation:

Its solution, which is much trickier than the worst- and best-case analyses, turnsout to be

Thus, on the average, quicksort makes only 39% more comparisons than in thebest case.
Moreover, its innermost loop is so efficient that it usually runs faster thanmergesort on
randomly ordered arrays of nontrivial sizes. This certainly justifies the namegiven to the
algorithm by its inventor.
Variations: Because of quicksort’s importance, there have been persistent efforts over
theyears to refine the basic algorithm. Among several improvements discovered
byresearchers are:
 Better pivot selection methods such as randomized quicksort that uses a random
element or the median-of-three method that uses the median of the leftmost,
rightmost, and the middle element of the array
 Switching to insertion sort on very small subarrays (between 5 and 15 elements for
most computer systems) or not sorting small subarrays at all and finishing the
algorithm with insertion sort applied to the entire nearly sorted array
 Modifications of the partitioning algorithm such as the three-way partition into
segments smaller than, equal to, and larger than the pivot
Limitations: 1. It is not stable. 2. It requires a stack to store parameters of subarrays that are
yet to be sorted. 3. WhilePerformance on randomly ordered arrays is known to be sensitive
not only to the implementation details of the algorithm but also to both computer architecture
and data type.

CSE, SVIT 18

38
ADA BCS401

7. Advantages and Disadvantages of Divide & Conquer

Advantages
1. Parallelism: Divide and conquer algorithms tend to have a lot of inherent parallelism.
Once the division phase is complete, the sub-problems are usually independent and
can therefore be solved in parallel. This approach typically generates more enough
concurrency to keep the machine busy and can be adapted for execution in multi-
processor machines.
2. Cache Performance: divide and conquer algorithms also tend to have good cache
performance. Once a sub-problem fits in the cache, the standard recursive solution
reuses the cached data until the sub-problem has been completely solved.
3. It allows solving difficult and often impossible looking problems like the Tower of
Hanoi. It reduces the degree of difficulty since it divides the problem into sub
problems that are easily solvable, and usually runs faster than other algorithms would.
4. Another advantage to this paradigm is that it often plays a part in finding other
efficient algorithms, and in fact it was the central role in finding the quick sort and
merge sort algorithms.
Disadvantages
5. One of the most common issues with this sort of algorithm is the fact that the
recursion is slow, which in some cases outweighs any advantages of this divide and
conquer process.
6. Another concern with it is the fact that sometimes it can become more complicated
than a basic iterative approach, especially in cases with a large n. In other words, if
someone wanted to add a large amount of numbers together, if they just create a
simple loop to add them together, it would turn out to be a much simpler approach
than it would be to divide the numbers up into two groups, add these groups
recursively, and then add the sums of the two groups together.
7. Another downfall is that sometimes once the problem is broken down into sub
problems, the same sub problem can occur many times. It is solved again. In cases
like these, it can often be easier to identify and save the solution to the repeated sub
problem, which is commonly referred to as memorization.

8. Decrease and Conquer Approach

Decrease-and-conquer is a general algorithm design technique, based on exploiting a
relationship between a solution to a given instance of a problem and a solution to a smaller
instance of the same problem. Once such a relationship is established, it can be exploited
either top down (usually recursively) or bottom up.There are three major variations of
decrease-and-conquer:
1. decrease-by-a-constant, most often by one (e.g., insertion sort)
2. decrease-by-a-constant-factor, most often by the factor of two (e.g., binary search)
3. variable-size-decrease (e.g., Euclid’s algorithm)

CSE, SVIT 19

39
ADA BCS401

In the decrease-by-a-constant variation, the size of an instance is reducedby the same

constant on each iteration of the algorithm. Typically, this constant is equal to one although
other constant size reductions do happenoccasionally.

Figure: Decrease-(by one)-and-conquer

technique
Example: an = an-1×a

The decrease-by-a-constant-factor technique suggests reducing a probleminstance by the

same constant factor on each teration of the algorithm. In mostapplications, this constant
factor is equal to two.

Figure: Decrease-(byhalf)-and-conquer
technique.

Example:

Finally, in the variable-size-decrease variety of decrease-and-conquer, thesize-reduction

pattern varies from one iteration of an algorithm to another.
Example: Euclid’salgorithm for computing the greatest common divisor. It is based on the
formula. gcd(m, n) = gcd(n, m mod n).
Though the value of the second argument is always smaller on the right-hand sidethan on the
left-hand side, it decreases neither by a constant nor by a constantfactor.

CSE, SVIT 20

40
Insertion sort

It is an application for decease-by-one technique to sort an array A[0…n-1]. Everytime the element is
inserted, it is stored in its position. So after every insertion the array remains sorted.

The number of key comparisons for such an input is

is the worst case.

For sorted array, the number of key comparisons is

And average case is Cavg(n) ᴝn2\4 € Θ(n2)

41
9. Topological Sort
Background: A directed graph, or digraph for short, is a graph with directions specified for
all its edges. The adjacency matrix and adjacency lists are the two principal means of
representing a digraph.Ther are only two notable differences between undirected and
directed graphs in representing them: (1) the adjacency matrix of a directed graph does not
have to be symmetric; (2) an edge in a directed graph has just one (not wo) corresponding
nodes in the digraph’s adjacency lists.
Depth-first search and breadth-first search are principal traversal algorithms for traversing
digraphs as well, but the structure of corresponding forests can be more complex than for
undirected graphs. Thus, even for the simple example of Figure, the depth-first search forest
(Figure b) exhibits all four types of edges possible in a DFS forest of a directed graph:
 tree edges (ab, bc, de),
 back edges (ba) from vertices to their ancestors,
 forward edges (ac) from vertices to their descendants in the tree other than their
children, and
 cross edges (dc), whichare none of the aforementioned types.

Note that a back edge in a DFS forest of a directed graph can connect a vertexto its parent.
Whether or not it is the case, the presence of a back edge indicatesthat the digraph has a
directed cycle. A directed cycle in a digraph is a sequenceof three or more of its vertices that
starts and ends with the same vertex and inwhich every vertex is connected to its immediate
predecessor by an edge directedfrom the predecessor to the successor. For example, a, b, a is
a directed cycle in the digraph in Figure given above. Conversely, if a DFS forest of a
digraph has no backedges, the digraph is a dag, an acronym for directed acyclic graph.

Motivation for topological sorting: Consider a set of five required

courses {C1, C2, C3, C4,C5} a part-time student has to take in some
degree program. The courses can betaken in any order as long as the
following course prerequisites are met: C1 andC2 have no prerequisites,
C3 requires C1 and C2, C4 requires C3, and C5 requiresC3 and C4. The
student can take only one course per term. In which order shouldthe student take the

42
ADA BCS401

courses?The situation can be modeled by a digraph in which vertices represent courses and
directed edges indicate prerequisite requirements.
In terms of this digraph, the uestion is whether we can list its vertices in such an order that
for every edge in the graph, the vertex where the edge starts is listed before the vertex where
the edge ends. In other words, can you find such an ordering of this digraph’s vertices? This
problem is called topological sorting.
Topological Sort: For topological sorting to be possible, a digraph in question must be a
DAG. i.e., if a digraph has nodirected cycles, the topological sorting problem for it has a
solution.
There are two efficient algorithms that both verify whether a digraph is a dag and, if it is,
produce an ordering of vertices that solves the topological sortingproblem.The first one is
based on depth-first search; the second is based on a direct application of the decrease-by-one
technique.
Topological Sorting based on DFS
Method
1. Perform a DFS traversal and note the order in which vertices become dead-ends
2. Reversing this order yields a solution to the topological sorting problem, provided, of
course, no back edge has been encountered during the traversal. If a back edge has
been encountered, the digraph is not a DAG, and topological sorting of its vertices is
impossible.
Illustration
a) Digraph for which the topological sorting problem needs to be solved.
b) DFS traversal stack w th the subscript numbers indicating the popping off order.
c) Solution to the problem. Here we have drawn the edges of the digraph, and they all
point from left to right as the problem’s statement requires. It is a convenient way to
check visually the correctness of a solution to an instance of the topological sorting
problem.

Topological Sorting using decrease-and-conquer technique

Method: The algorithm is based on a direct implementation of the decrease-(byone)-and-
conquer technique:

CSE, SVIT 23

43
ADA BCS401

1. Repeatedly, identify in a remaining digraph a source, which is a vertex with no

incoming edges, and delete it along with all the edges outgoing from it. (If there are
several sources, break the tie arbitrarily. If there are none, stop because the problem
cannot be solved.)
2. The order in which the vertices are deleted yields a solution to the topological sorting
problem.
Illustration - Illustration of the source-removal algorithm for the topological sorting problem
is given here. On each iteration, a vertex with no incoming edges is deleted from the digraph.

Note: The solution obtained by the source-removal algorithm is different from the one
obtained by the DFS-based algorithm. Both of them are correct, of course; the topological
sorting problem may have several alternative solutions.
Depth first search:

CSE, SVIT 24

44
Breadth first search:

45
46
Applications of Topological Sorting
 Instruction scheduling in program compilation
 Cell evaluation ordering in spreadsheet formulas,
 Resolving symbol dependencies in linkers.

***

47
 Lecture Notes on
Analysis and Design of Algorithms
BCS401

Module-3: Greedy Method

Contents

1. Introduction to Greedy method 3. Single source shortest paths

1.1. General method, 3.1. Dijkstra's Algorithm
1.2. Coin Change Problem 4. Optimal Tree problem:
1.3. Knapsack Problem 4.1. Huffman Trees and Codes
1.4. Job sequencing with deadlines 5. Transform and Conquer Approach:
2. Minimum cost spanning trees: 5.1. Heaps
2.1. Prim’s Algorithm, 5.2. Heap Sort
2.2. Kruskal’s Algorithm

48
ADA BCS401

1. Introduction to Greedy method

1.1 General method
The greedy method is the straight forward design technique applicable to variety of
applications.
The greedy approach suggests constructing a solution through a sequence of steps, each
expanding a partially constructed solution obtained so far, until a complete solution to the
problem is reached. On each step the choice made must be:
 feasible, i.e., it has to satisfy the problem’s constraints
 locally optimal, i.e., it has to be the best local choice among all feasible choices
available on that step
 irrevocable, i.e., once made, it cannot be changed on subsequent steps of the algorithm
As a rule, greedy algorithms are both intuitively appealing and simple. Given an optimization
problem, it is usually easy to figure out how to proceed in a greedy manner, possibly after
considering a few small instances of the problem. What is usually more difficult is to prove
that a greedy algorithm yields an optimal solution (when it does).

1.2. Coin Change Problem

Problem Statement: Given coins of several denominations find out a way to give a customer
an amount with fewest number of coins.
Example: if denominations are 1,5,10, 25 and 100 and the change required is 30, the solutions
are,
Amount : 30
Solutions : 3 x 10 ( 3 coins ), 6 x 5 ( 6 coins )
1 x 25 + 5 x 1 ( 6 coins ) 1 x 25 + 1 x 5 ( 2 coins )
The last solution is the optimal one as it gives us change only with 2 coins.

CSE, SVIT 1

49
ADA BCS401

Solution for coin change problem using greedy algorithm is very intuitive and called as
cashier’s algorithm. Basic principle is: At every iteration for search of a coin, take the
largest coin which can fit into remain amount to be changed at that particular time. At
the end you will have optimal solution.

1.3. Knapsack Problem (Fractional knapsack problem)

Consider the following instance of the knapsack problem:

n=3, m=20, (p1, p2, p3) =(25, 24, 15), (w1, w2, w3) =(18, 15, 10)

There are several greedy methods to obtain the feasible solutions. Three are discussed here
a) At each step fill the knapsack with the object with largest profit - If the object under
consideration does not fit, then the fraction of it is included to fill the knapsack. This method
does not result optimal solution. As per this method the solution to the above problem is as
follows;
Select Item-1 with profit p1=25, here w1=18, x1=1. Remaining capacity = 20-18 = 2
Select Item-2 with profit p1=24, here w2=15, x1=2/15. Remaining capacity = 0
Total profit earned = 28.2.
Therefore optimal solution is (x1, x2, x3) = (1, 2/15, 0) with profit = 28.2
b) At each step fill the object with smallest weight
Select Item-3 with profit p1=15, here w1=10, x3=1. Remaining capacity = 20-10 = 10
Select Item-2 with profit p1=24, here w2=15, x1=10/15. Remaining capacity = 0
Total profit earned = 31.
Optimal solution using this method is (x1, x2, x3) = (0, 2/3, 1) with profit = 31
Note: Optimal solution is not guaranteed using method a and b

CSE, SVIT 2

50
ADA BCS401

c) At each step include the object with maximum profit/weight ratio

Select Item-2 with profit p1=24, here w2=15, x1=1. Remaining capacity = 20-15=5
Select Item-3 with profit p1=15, here w1=10, x1=5/10. Remaining capacity = 0
Total profit earned = 31.5
Therefore, optimal solution is (x1, x2, x3) = (0, 1, 1/2) with profit = 31.5
This greedy approach always results optimal solution.
Algorithm: The algorithm given below assumes that the objects are sorted in non-increasing
order of profit/weight ratio

Analysis:
Disregarding the time to initially sort the object, each of the above strategies use O(n) time,

0/1 Knapsack problem

Note: The greedy approach to solve 0/1 knapsack problem does not necessarily yield an optimal
solution

CSE, SVIT 3

51
ADA BCS401

1.4. Job sequencing with deadlines

The greedy strategy to solve job sequencing problem is, “At each time select the job that that
satisfies the constraints and gives maximum profit. i.e consider the jobs in the non-increasing
order of the pi’s”
By following this procedure, we get the 3rd solution in the example 4.3. It can be proved that,
this greedy strategy always results optimal solution

High level description of job sequencing algorithm

CSE, SVIT 4

52
ADA BCS401

Algorithm/Program 4.6: Greedy algorithm for sequencing unit time jobs with deadlines and
profits

Analysis:

Fast Job Scheduling Algorithm

CSE, SVIT 5

53
ADA BCS401

Algorithm: Fast Job Scheduling is shown in next page

Analysis

Algorithm: Fast Job Scheduling

CSE, SVIT 6

54
ADA BCS401

Problem: Find solution generated by job sequencing problem with deadlines for 7 jobs given
profits 3, 5, 20, 18, 1, 6, 30 and deadlines 1, 3, 4, 3, 2, 1, 2 respectively.
Solution: Given
J1 J2 J3 J4 J5 J6 J7
Profit 3 5 20 18 1 6 30
Deadline 1 3 4 3 2 1 2
Sort the jobs as per the decreasing order of profit
J7 J3 J4 J6 J2 J1 J5
Profit 30 20 18 6 5 3 1
Deadline 2 4 3 1 3 1 2
Maximum deadline is 4. Therefore create 4 slots. Now allocate jobs to highest slot, starting
from the job of highest profit
Select Job 7 – Allocate to slot-2 Slot 1 2 3 4
Select Job 3 – Allocate to slot-4 Job J6 J7 J 4 J3
Select Job 4 – Allocate to slot-3
Select Job 6 – Allocate to slot-1 Total profit earned is = 30+20+18+6=74

Problem: What is the solution generated by job sequencing when n = 5, (P1, P2, P3, P4, P5)
= (20, 15, 10, 5, 1), (d1, d2, d3, d4, d5) = (2, 2, 1, 3, 3)
Solution
The Jobs are already sorted according to decreasing order of profit.
Maximum deadline is 3. Therefore create 4 slots. Allocate jobs to highest slot, starting from
the job of highest profit
Select Job 1 – Allocate to slot-2
Select Job 2 – Allocate to slot-1 as 2 is already filled Slot 1 2 3
Job J2 J1 J4
Select Job 3 –Slot-2 &1 are already filled. Cannot be allocated.
Select Job 4 – Allocate to slot-3
Total profit earned is = 20+15+5=40

CSE, SVIT 7

55
ADA BCS401

2. Minimum cost spanning trees

Definition: A spanning tree of a connected graph is its connected acyclic subgraph (i.e., a tree)
that contains all the vertices of the graph. A minimum spanning tree of a weighted connected
graph is its spanning tree of the smallest weight, where the weight of a tree is defined as the
sum of the weights on all its edges. The minimum spanning tree problem is the problem of
finding a minimum spanning tree for a given weighted connected graph.

2.1. Prim’s Algorithm

Prim's algorithm constructs a minimum spanning tree through a sequence of expanding sub-
trees. The initial subtree in such a sequence consists of a single vertex selected arbitrarily from
the set V of the graph's vertices. On each iteration it expands the current tree in the greedy
manner by simply attaching to it the nearest vertex not in that tree. The algorithm stops after
all the graph's vertices have been included in the tree being constructed. Since the algorithm
expands a tree by exactly one vertex on each of its iterations, the total number of such iterations
is n - 1, where n is the number of vertices in the graph. The tree generated by the algorithm is
obtained as the set of edges.

Correctness: Prim’s algorithm always yields a minimum spanning tree.

CSE, SVIT 8

56
ADA BCS401

Example: An example of prim’s algorithm is shown below.

The parenthesized labels of a vertex in the middle column
indicate the nearest tree vertex and edge weight; selected
vertices and edges are shown in bold.

Tree vertices Remaining vertices Illustration

Analysis of Efficiency
The efficiency of Prim’s algorithm depends on the data structures chosen for the graph itself
and for the priority queue of the set V − VT whose vertex priorities are the distances to the
nearest tree vertices.
1. If a graph is represented by its weight matrix and the priority queue is implemented as
an unordered array, the algorithm’s running time will be in Θ(|V|2). Indeed, on each

CSE, SVIT 9

57
ADA BCS401

of the |V| − 1iterations, the array implementing the priority queue is traversed to find
and delete the minimum and then to update, if necessary, the priorities of the remaining
vertices.
We can implement the priority queue as a min-heap. (A min-heap is a complete binary tree in
which every element is less than or equal to its children.) Deletion of the smallest element from
and insertion of a new element into a min-heap of size n are O(log n) operations.
2. If a graph is represented by its adjacency lists and the priority queue is implemented
as a min-heap, the running time of the algorithm is in O(|E| log |V |).
This is because the algorithm performs |V| − 1 deletions of the smallest element and makes |E|
verifications and, possibly, changes of an element’s priority in a min-heap of size not exceeding
|V|. Each of these operations, as noted earlier, is a O(log |V|) operation. Hence, the running
time of this implementation of Prim’s algorithm is in
(|V| − 1+ |E|) O (log |V |) = O(|E| log |V |) because, in a connected graph, |V| − 1≤ |E|.

2.2. Kruskal’s Algorithm

Background: Kruskal's algorithm is another greedy algorithm for the minimum spanning tree
problem that also always yields an optimal solution. It is named Kruskal's algorithm, after
Joseph Kruskal. Kruskal's algorithm looks at a minimum spanning tree for a weighted
connected graph G = (V, E) as an acyclic sub graph with |V | - 1 edges for which the sum of
the edge weights is the smallest. Consequently, the algorithm constructs a minimum spanning
tree as an expanding sequence of sub graphs, which are always acyclic but are not necessarily
connected on the intermediate stages of the algorithm.
Working: The algorithm begins by sorting the graph's edges in non-decreasing order of their
weights. Then, starting with the empty subgraph, it scans this sorted list adding the next edge
on the list to the current sub graph if such an inclusion does not create a cycle and simply
skipping the edge otherwise.

CSE, SVIT 10

58
ADA BCS401

The fact that ET ,the set of edges composing a minimum spanning tree of graph G actually a
tree in Prim's algorithm but generally just an acyclic sub graph in Kruskal's algorithm.

Kruskal’s algorithm is not simpler because it has to check whether the addition of the next
edge to the edges already selected would create a cycle.

We can consider the algorithm's operations as a progression through a series of forests

containing all the vertices of a given graph and some of its edges. The initial forest consists of
|V| trivial trees, each comprising a single vertex of the graph. The final forest consists of a
single tree, which is a minimum spanning tree of the graph. On each iteration, the algorithm
takes the next edge (u, v) from the sorted list of the graph's edges, finds the trees containing the
vertices u and v, and, if these trees are not the same, unites them in a larger tree by adding the
edge (u, v).

Analysis of Efficiency
The crucial check whether two vertices belong to the same tree can be found out using union-
find algorithms.
Efficiency of Kruskal’s algorithm is based on the time needed for sorting the edge weights of
a given graph. Hence, with an efficient sorting algorithm, the time efficiency of Kruskal's
algorithm will be in O (|E| log |E|).

Illustration
An example of Kruskal’s algorithm is shown below. The
selected edges are shown in bold.

CSE, SVIT 11

59
ADA BCS401

3. Single source shortest paths

Single-source shortest-paths problem is defined as follows. For a given vertex called the
source in a weighted connected graph, the problem is to find shortest paths to all its other
vertices. The single-source shortest-paths problem asks for a family of paths, each leading from
the source to a different vertex in the graph, though some paths may, of course, have edges in
common.
3.1. Dijkstra's Algorithm
Dijkstra's Algorithm is the best-known algorithm for the single-source shortest-paths problem.
This algorithm is applicable to undirected and directed graphs with nonnegative weights only.
Working - Dijkstra's algorithm finds the shortest paths to a graph's vertices in order of their
distance from a given source.
 First, it finds the shortest path from the source to a vertex nearest to it, then to a second
nearest, and so on.
 In general, before its ith iteration commences, the algorithm
has already identified the shortest paths to i-1 other vertices
nearest to the source. These vertices, the source, and the
edges of the shortest paths leading to them from the source
form a subtree Ti of the given graph shown in the figure.
 Since all the edge weights are nonnegative, the next vertex
nearest to the source can be found among the vertices adjacent to the vertices of Ti. The

CSE, SVIT 12

60
ADA BCS401

set of vertices adjacent to the vertices in Ti can be referred to as "fringe vertices"; they
are the candidates from which Dijkstra's algorithm selects the next vertex nearest to the
source.
 To identify the ith nearest vertex, the algorithm computes, for every fringe vertex u, the
sum of the distance to the nearest tree vertex v (given by the weight of the edge (v, u))
and the length d., of the shortest path from the source to v (previously determined by
the algorithm) and then selects the vertex with the smallest such sum. The fact that it
suffices to compare the lengths of such special paths is the central insight of Dijkstra's
algorithm.
 To facilitate the algorithm's operations, we label each vertex with two labels.
o The numeric label d indicates the length of the shortest path from the source to this
vertex found by the algorithm so far; when a vertex is added to the tree, d indicates
the length of the shortest path from the source to that vertex.
o The other label indicates the name of the next-to-last vertex on such a path, i.e.,
the parent of the vertex in the tree being constructed. (It can be left unspecified for
the sources and vertices that are adjacent to none of the current tree vertices.)
With such labeling, finding the next nearest vertex u* becomes a simple task of finding
a fringe vertex with the smallest d value. Ties can be broken arbitrarily.
 After we have identified a vertex u* to be added to the tree, we need to perform two
operations:
o Move u* from the fringe to the set of tree vertices.
o For each remaining fringe vertex u that is connected to u* by an edge of weight
w(u*, u) such that du*+ w(u*, u) <du, update the labels of u by u* and du*+ w(u*,
u), respectively.

Illustration: An example of Dijkstra's algorithm is

shown below. The next closest vertex is shown in
bold. (see the figure in next page)

The shortest paths (identified by following nonnumeric labels backward from a destination
vertex in the left column to the source) and their lengths (given by numeric labels of the tree
vertices) are as follows:

The pseudocode of Dijkstra’s algorithm is given below. Note that in the following pseudocode,
VT contains a given source vertex and the fringe contains the vertices adjacent to it after
iteration 0 is completed.

CSE, SVIT 13

61
ADA BCS401

Analysis:
The time efficiency of Dijkstra’s algorithm depends on the data structures used for
implementing the priority queue and for representing an input graph itself.

CSE, SVIT 14

62
ADA BCS401

Efficiency is Θ(|V|2) for graphs represented by their weight matrix and the priority queue
implemented as an unordered array.
For graphs represented by their adjacency lists and the priority queue implemented as a min-
heap, it is in O (|E| log |V| )
Applications
 Transportation planning and packet routing in communication networks, including the
Internet
 Finding shortest paths in social networks, speech recognition, document formatting,
robotics, compilers, and airline crew scheduling.

4. Optimal Tree problem

Background:
Suppose we have to encode a text that comprises characters from some n-character alphabet
by assigning to each of the text's characters some sequence of bits called the codeword. There
are two types of encoding: Fixed-length encoding, Variable-length encoding
Fixed-length encoding: This method assigns to each character a bit string of the same length
m (m >= log2n). This is exactly what the standard ASCII code does.
One way of getting a coding scheme that yields a shorter bit string on the average is based on
the old idea of assigning shorter code-words to more frequent characters and longer code-words
to less frequent characters.
Variable-length encoding: This method assigns code-words of different lengths to different
characters, introduces a problem that fixed-length encoding does not have. Namely, how can
we tell how many bits of an encoded text represent the first character? (or, more generally, the
ith) To avoid this complication, we can limit ourselves to prefix-free (or simply prefix) codes.
In a prefix ode, no code word is a prefix of a codeword of another character. Hence, with such
an encoding, we can simply scan a bit string until we get the first group of bits that is a
codeword for some character, replace these bits by this character, and repeat this operation
until the bit string's end is reached.
If we want to create a binary prefix code for some alphabet, it is natural to associate the
alphabet's characters with leaves of a binary tree in which all the left edges are labelled by 0
and all the right edges are labelled by 1 (or vice versa). The codeword of a character can then
be obtained by recording the labels on the simple path from the root to the character's leaf.
Since there is no simple path to a leaf that continues to another leaf, no codeword can be a
prefix of another codeword; hence, any such tree yields a prefix code.
Among the many trees that can be constructed in this manner for a given alphabet with known
frequencies of the character occurrences, construction of such a tree that would assign shorter
bit strings to high-frequency characters and longer ones to low-frequency characters can be
done by the following greedy algorithm, invented by David Huffman.

CSE, SVIT 15

63
ADA BCS401

4.1 Huffman Trees and Codes

Huffman's Algorithm
Step 1: Initialize n one-node trees and label them with the characters of the alphabet. Record
the frequency of each character in its tree's root to indicate the tree's weight. (More generally,
the weight of a tree will be equal to the sum of the frequencies in the tree's leaves.)
Step 2: Repeat the following operation until a single tree is obtained. Find two trees with the
smallest weight. Make them the left and right subtree of a new tree and record the sum of their
weights in the root of the new tree as its weight.
A tree constructed by the above algorithm is called a Huffmantree. It defines-in the manner
described-a Huffman code.
Example: Consider the five-symbol alphabet {A, B, C, D, _} with the following occurrence
frequencies in a text made up of
these symbols:
The Huffman tree construction
for the above problem is shown below:

The resulting codewords are as follows:

CSE, SVIT 16

64
ADA BCS401

Hence, DAD is encoded as 011101, and 10011011011101 is decoded as BAD_AD.

With the occurrence frequencies given and the code word lengths obtained, the average
number of bits per symbol in this code is
2 *0.35 + 3 *0.1+ 2 *0.2 + 2 *0.2 + 3 *0.15 = 2.25.
Had we used a fixed-length encoding for the same alphabet, we would have to use at least 3
bits per each symbol. Thus, for this example, Huffman’s code achieves the compression ratio
(a standard measure of a compression algorithm’s effectiveness) of (3−2.25)/3*100%= 25%.
In other words, Huffman’s encoding of the above text will use 25% less memory than its fixed-
length encoding.

5. Transform and Conquer Approach

We call this general technique transform-and-conquer because these methods work as two-
stage procedures. First, in the transformation stage, the problem’s instance is modified to be,
for one reason or another, more amenable to solution. Then, in the second or conquering stage,
it is solved.
There are three major variations of this idea that differ by what we transform a given instance
to (Figure 6.1):
 Transformation to a simpler or more convenient instance of the same problem—we call
it instance simplification.
 Transformation to a different representation of the same instance—we call it
representation change.
 Transformation to an instance of a different problem for which an algorithm is already
available—we call it problem reduction.

5.1. Heaps
Heap is a partially ordered data structure that is especially suitable for implementing priority
queues. Priority queue is a multiset of items with an orderable characteristic called an item’s
priority, with the following operations:

CSE, SVIT 17

65
ADA BCS401

 finding an item with the highest (i.e., largest) priority

 deleting an item with the highest priority
 adding a new item to the multiset
Notion of the Heap
Definition: A heap can be defined as a binary tree with keys assigned to its nodes, one key per
node, provided the following two conditions are met:
1. The shape property—the binary tree is essentially complete (or simply complete),
i.e., all its levels are full except possibly the last level, where only some rightmost leaves
may be missing.
2. The parental dominance or heap property—the key in each node is greater than or
equal to the keys in its children.
Illustration: The illustration of the definition of heap is shown bellow: only the left most tree
is heap. The second one is not a heap, because the tree’s shape property is violated. The left
child of last subtree cannot be empty. And the third one is not a heap, because the parental
dominance fails for the node with key 5.

Properties of Heap
1. There exists exactly one essentially complete binary tree with n nodes. Its height is
equal to ⌊𝑙𝑜𝑔2𝑛⌋
2. The root of a heap always contains its largest element.
3. A node of a heap considered with all its descendants is also a heap.
4. A heap can be implemented as an array by recording its elements in the top down, left-
to-right fashion. It is convenient to store the heap’s elements in positions 1 through n
of such an array, leaving H[0] either unused or putting there a sentinel whose value is
greater than every element in the heap. In such a representation,
a. the parental node keys will be in the first ⌊n/2⌋. positions of the array, while the
leaf keys will occupy the last ⌊n/2⌋ positions;
b. the children of a key in the array’s parental position i (1≤ i ≤⌊𝑛/2⌋) will be in
positions 2i and 2i + 1, and, correspondingly, the parent of a key in position i (2
≤ i≤ n) will be in position ⌊𝑛/2⌋.

CSE, SVIT 18

66
ADA BCS401

Heap and its array representation

Thus, we could also define a heap as an array H[1..n] in which every element in position i in
the first half of the array is greater than or equal to the elements in positions 2i and 2i + 1, i.e.,
H[i]≥max {H [2i], H [2i + 1]} for i= 1. . .⌊𝑛/2⌋

Constructions of Heap - There are two principal alternatives for constructing Heap.
1) Bottom-up heap construction 2) Top-down heap construction

Bottom-up heap construction:

The bottom-up heap construction algorithm is illustrated bellow. It initializes the essentially
complete binary tree with n nodes by placing keys in the order given and then “heapifies” the
tree as follows.
 Starting with the last parental node, the algorithm checks whether the parental
dominance holds for the key in this node. If it does not, the algorithm exchanges the
node’s key K with the larger key of its children and checks whether the parental
dominance holds for K in its new position. This process continues until the parental
dominance for K is satisfied. (Eventually, it has to because it holds automatically for
any key in a leaf.)
 After completing the “heapification” of the subtree rooted at the current parental node,
the algorithm proceeds to do the same for the node’s immediate predecessor.
 The algorithm stops after this is done for the root of the tree.

CSE, SVIT 19

67
ADA BCS401

Illustration
Bottom-up construction of a heap for the list 2, 9, 7, 6, 5, 8. The double headed arrows show
key comparisons verifying the parental dominance.

Analysis of efficiency - bottom up heap construction algorithm:

Assume, for simplicity, that n = 2k− 1 so that a heap’s tree is full, i.e., the largest possible
number of nodes occurs on each level. Let h be the height of the tree.
According to the first property of heaps in the list at the beginning of the section, h=⌊𝑙𝑜𝑔2𝑛⌋
or just ⌊𝑙𝑜𝑔2(𝑛 + 1)⌋= k − 1 for the specific values of n we are considering.
Each key on level I of the tree will travel to the leaf level h in the worst case of the heap
construction algorithm. Since moving to the next level down requires two comparisons—one

CSE, SVIT 20

68
ADA BCS401

to find the larger child and the other to determine whether the exchange is required—the total
number of key comparisons involving a key on level I will be 2(h − i).
Therefore, the total number of key comparisons in the worst case will be

where the validity of the last equality can be proved either by using the closed-form formula

for the sum or by mathematical induction on h.

Thus, with this bottom-up algorithm, a heap of size n can be constructed with fewer than 2n
comparisons.
Top-down heap construction algorithm:
It constructs a heap by successive insertions of a new key into a previously constructed heap.
1. First, attach a new node with key K in it after the last leaf of the existing heap.
2. Then shift K up to its appropriate place in the new heap as follows.
a. Compare K with its parent’s key: if the latter is greater than or equal to K, stop (the
structure is a heap); otherwise, swap these two keys and compare K with its new
parent.
b. This swapping continues until K is not greater than its last parent or it reaches root.
Obviously, this insertion operation cannot require more key comparisons than the heap’s
height. Since the height of a heap with n nodes is about log2n, the time efficiency of insertion
is in O(log n).
Illustration of inserting a new key: Inserting a new key (10) into the
heap is constructed bellow. The new key is shifted up via a swap with its
parents until it is not larger than its parents (or is in the root).

Delete an item from a heap: Deleting the root’s key from a heap can be done with the
following algorithm:
Maximum Key Deletion from a heap
1. Exchange the root’s key with the last key K of the heap.
2. Decrease the heap’s size by 1.
3. “Heapify” the smaller tree by sifting K down the tree exactly in the same way we did it
in the bottom-up heap construction algorithm. That is, verify the parental dominance

CSE, SVIT 21

69
ADA BCS401

for K: if it holds, we are done; if not, swap K with the larger of its children and repeat
this operation until the parental dominance condition holds for K in its new position.

Illustration

The efficiency of deletion is determined by the number of key comparisons needed to

“heapify” the tree after the swap has been made and the size of the tree is decreased by 1.Since
this cannot require more key comparisons than twice the heap’s height, the time efficiency of
deletion is in O (log n) as well.

5.2. Heap Sort

Heapsort - an interesting sorting algorithm is discovered byJ. W. J. Williams. This is a two-
stage algorithm that works as follows.
Stage 1 (heap construction): Construct a heap for a given array.
Stage 2 (maximum deletions): Apply the root-deletion operation n−1 times to the
remaining heap.
As a result, the array elements are eliminated in decreasing order. But since under the array
implementation of heaps an element being deleted is placed last, the resulting array will be
exactly the original array sorted in increasing order.
Heap sort is traced on a specific input is shown below:

CSE, SVIT 22

70
ADA BCS401

Analysis of efficiency: Since we already know that the heap construction stage of the algorithm
is in O(n), we have to investigate just the time efficiency of the second stage. For the number
of key comparisons, C(n), needed for eliminating the root keys from the heaps of diminishing
sizes from n to 2, we get the following inequality:

This means that C(n) ∈ O(n log n) for the second stage of heapsort. For both stages, we get
O(n) + O(n log n) = O(n log n). A more detailed analysis shows that the time efficiency of
heapsort is, in fact, in Θ(n log n) in both the worst and average cases. Thus, heapsort’s time
efficiency falls in the same class as that of mergesort.
Heapsort is in-place, i.e., it does not require any extra storage. Timing experiments on random
files show that heapsort runs more slowly than quicksort but can be competitive with mergesort.

*****

CSE, SVIT 23

71
 Lecture Notes on
Analysis and Design of Algorithms
BCS401

Module-4 : Dynamic Programming

Contents

1. Introduction to Dynamic Programming

1.1. General method with Examples
1.2. Multistage Graphs
2. Transitive Closure:
2.1. Warshall’s Algorithm
3. All Pairs Shortest Paths:
3.1. Floyd's Algorithm
4. Knapsack problem
5. Bellman-Ford Algorithm
6. Travelling Sales Person problem

72
ADA BCS401

1. Introduction to Dynamic Programming

Dynamic programming is a technique for solving problems with overlapping subproblems.
Typically, these subproblems arise from a recurrence relating a given problem’s solution to
solutions of its smaller subproblems. Rather than solving overlapping subproblems again and
again, dynamic programming suggests solving each of the smaller subproblems only once
and recording the results in a table from which a solution to the original problem can then be
obtained. [From T1]
The Dynamic programming can be used when the solution to a problem can be viewed as the
result of sequence of decisions. [ From T2]. Here are some examples.

Example 1

Example 2

Example 3

Example 4

CSE, SVIT 1

73
ADA BCS401

CSE, SVIT 2

74
ADA BCS401

1.2 Multistage Graphs

CSE, SVIT 3

75
ADA BCS401

Figure: Five stage graph

CSE, SVIT 4

76
ADA BCS401

CSE, SVIT 5

77
ADA BCS401

Backward Approach

CSE, SVIT 6

78
ADA BCS401

2. Transitive Closure using Warshall’s Algorithm,

Definition: The transitive closure of a directed graph with n vertices can be defined as the n
× n boolean matrix T = {tij}, in which the element in the ith row and the jth column is 1 if there
exists a nontrivial path (i.e., directed path of a positive length) from the ith vertex to the jth
vertex; otherwise, tij is 0.
Example: An example of a digraph, its adjacency matrix, and its transitive closure is given
below.

(a) Digraph. (b) Its adjacency matrix. (c) Its transitive closure.

We can generate the transitive closure of a digraph with the help of depth first search or
breadth-first search. Performing either traversal starting at the ith vertex gives the information
about the vertices reachable from it and hence the columns that contain 1’s in the ith row of
the transitive closure. Thus, doing such a traversal for every vertex as a starting point yields
the transitive closure in its entirety.
Since this method traverses the same digraph several times, we can use a better algorithm
called Warshall’s algorithm. Warshall’s algorithm constructs the transitive closure through
a series of n × n boolean matrices:

Each of these matrices provides certain information about directed paths in the digraph.
Specifically, the element𝑟𝑖𝑗(𝑘) in the ith row and jth column of matrix R(k) (i, j = 1, 2, . . . , n, k =
0, 1, . . . , n) is equal to 1 if and only if there exists a directed path of a positive length from
the ith vertex to the jth vertex with each intermediate vertex, if any, numbered not higher than
k.
Thus, the series starts with R(0), which does not allow any intermediate vertices in its paths;
hence, R(0) is nothing other than the adjacency matrix of the digraph. R(1) contains the
information about paths that can use the first vertex as intermediate. The last matrix in the
series, R(n),reflects paths that can use all n vertices of the digraph as intermediate and hence is
nothing other than the digraph’s transitive closure.
This means that there exists a path from the ith vertex vi to the jth vertex vj with each
intermediate vertex numbered not higher than k:
vi, a list of intermediate vertices each numbered not higher than k, vj . --- (*)
Two situations regarding this path are possible.

CSE, SVIT 7

79
ADA BCS401

1. In the first, the list of its intermediate vertices does not contain the kth vertex. Then this
path from vi to vj has intermediate vertices numbered not higher than k−1. i.e. rij(k−1) = 1
th
2. The second possibility is that path (*) does contain the k vertex vk among the
intermediate vertices. Then path (*) can be rewritten as;
vi, vertices numbered ≤ k − 1, vk, vertices numbered ≤ k − 1, vj .
i.e r(k−1) = 1 and r(k−1) = 1
ik kj

Thus, we have the following formula for generating the elements of matrix R(k) from the
elements of matrix R(k−1)

The Warshall’s algorithm works based on the above formula.

As an example, the application of Warshall’s algorithm to the digraph is shown below. New
1’s are in bold.

CSE, SVIT 8

80
ADA BCS401

Analysis
Its time efficiency is Θ(n3). We can make the algorithm to run faster by treating matrix rows
as bit strings and employ the bitwise or operation available in most modern computer
languages.
Space efficiency: Although separate matrices for recording intermediate results of the
algorithm are used, that can be avoided.

3. All Pairs Shortest Paths using Floyd's Algorithm,

Problem definition: Given a weighted connected graph (undirected or directed), the all-pairs
shortest paths problem asks to find the distances—i.e., the lengths of the shortest paths - from
each vertex to all other vertices.
Applications: Solution to this problem finds applications in communications, transportation
networks, and operations research. Among recent applications of the all-pairs shortest-path
problem is pre-computing distances for motion planning in computer games.
We store the lengths of shortest paths in an n x n matrix D called the distance matrix: the
element dij in the ith row and the jth column of this matrix indicates the length of the shortest
path from the ith vertex to the jth vertex.

(a) Digraph. (b) Its weight matrix. (c) Its distance matrix
We can generate the distance matrix with an algorithm that is very similar to Warshall’s
algorithm. It is called Floyd’s algorithm.
Floyd’s algorithm computes the distance matrix of a weighted graph with n vertices through a
series of n × n matrices:

CSE, SVIT 9

81
ADA BCS401

The element (𝑘)𝑖𝑗in the ith row and the jth column of matrix D(k) (i, j = 1, 2, . . . , n, k = 0, 1, .
. . , n) is equal to the length of the shortest path among all paths from the ith vertex to the jth
vertex with each intermediate vertex, if any, numbered not higher than k.
As in Warshall’s algorithm, we can compute all the elements of each matrix D(k) from its
immediate predecessor D(k−1)

If (𝑘)𝑖𝑗 = 1, then it means that there is a path;

vi, a list of intermediate vertices each numbered not higher than k, vj .

We can partition all such paths into two disjoint subsets: those that do not use thekth vertex vk
as intermediate and those that do.
i. Since the paths of the first subset have their intermediate vertices numbered not higher
than k − 1, the shortest of them is, by definition of our matrices, of length 𝑑𝑖𝑗
(𝑘−1)

ii. In the second subset the paths are of the form

vi, vertices numbered ≤ k − 1, vk, vertices numbered ≤ k − 1, vj .

The situation is depicted symbolically in Figure, which shows

the underlying idea of Floyd’s algorithm.

Taking into account the lengths of the shortest paths in both subsets leads to the following
recurrence:

Analysis: Its time efficiency is Θ(n3), similar to the warshall’s algorithm.

CSE, SVIT 10

82
ADA BCS401

Application of Floyd’s algorithm to the digraph is shown below. Updated elements are shown
in bold.

4. Knapsack problem
We start this section with designing a dynamic programming algorithm for the knapsack
problem: given n items of known weights w1, . . . ,wn and valuesv1, . . . , vn and a knapsack of
capacity W, find the most valuable subset of the items that fit into the knapsack. To design a
dynamic programming algorithm, we need to derive a recurrence relation that expresses a
solution to an instance of the knapsack problem in terms of solutions to its smaller sub
instances. Let us consider an instance defined by the first i items, 1≤ i ≤ n, with weights w1, . .
. ,wi, values v1, . . . , vi , and knapsack capacity j, 1 ≤ j ≤ W. Let F(i, j) be the value of an
optimal solution to this instance. We can divide all the subsets of the first i items that fit the
knapsack of capacity j into two categories: those that do not include the ith item and those that
do. Note the following:

CSE, SVIT 11

83
ADA BCS401

i. Among the subsets that do not include the ith item, the value of an optimal subset is,
by definition, i.e F(i , j) = F(i − 1, j).
ii. Among the subsets that do include the ith item (hence, j − wi≥ 0), an optimal subset is
made up of this item and an optimal subset of the first i−1 items that fits into the
knapsack of capacity j − wi. The value of such an optimal subset is vi+ F(i − 1, j − wi).
Thus, the value of an optimal solution among all feasible subsets of the first I items is the
maximum of these two values.

It is convenient to define the initial conditions as follows:

F(0, j) = 0 for j ≥ 0 and F(i, 0) = 0 for i ≥ 0.
Our goal is to find F(n, W), the maximal value of a subset of the n given items that fit into
the knapsack of capacity W, and an optimal subset itself.

The algorithm for the knapsack problem can be stated as follows

Input: n – total items, W – capacity of the knapsack
wi– weight of the ith item, vi– value of the ith item,
Output: F(i, j) be the value of an optimal solution to this instance considering first i items
with capacity j. F(n,W) is the optimal solution
Method:

CSE, SVIT 12

84
ADA BCS401

Example-1:Let us consider the instance given by the following data:

The dynamic programming table, filled by applying formulas is given below

Thus, the maximal value is F(4, 5) = $37.

We can find the composition of an optimal subset by back tracing the computations of this
entry in the table. Since F(4, 5) > F(3, 5), item 4 has to be included in an optimal solution
along with an optimal subset for filling 5 − 2 = 3 remaining units of the knapsack capacity.
The value of the latter is F(3, 3). Since F(3, 3) = F(2, 3), item 3 need not be in an optimal
subset. Since F(2, 3) > F(1, 3), item 2 is a part of an optimal selection, which leaves element
F(1, 3 − 1) to specify its remaining composition. Similarly, since F(1, 2) > F(0, 2), item 1 is
the final part of the optimal solution {item 1, item 2, item 4}.

Analysis
The time efficiency and space efficiency of this algorithm are both in Θ(nW). The time
needed to find the composition of an optimal solution is in O(n).

Memory Functions
The direct top-down approach to finding a solution to such a recurrence leads to an algorithm
that solves common subproblems more than once and hence is very inefficient.
The classic dynamic programming approach, on the other hand, works bottom up: it fills a
table with solutions to all smaller subproblems, but each of them is solved only once. An
unsatisfying aspect of this approach is that solutions to some of these smaller subproblems
are often not necessary for getting a solution to the problem given. Since this drawback is not
present in the top-down approach, it is natural to try to combine the strengths of the top-down
and bottom-up approaches. The goal is to get a method that solves only subproblems that are
necessary and does so only once. Such a method exists; it is based on using memory
functions.
This method solves a given problem in the top-down manner but, in addition, maintains a
table of the kind that would have been used by a bottom-up dynamic programming algorithm.

CSE, SVIT 13

85
ADA BCS401

Initially, all the table’s entries are initialized with a special “null” symbol to indicate that they
have not yet been calculated. Thereafter, whenever a new value needs to be calculated, the
method checks the corresponding entry in the table first: if this entry is not “null,” it is simply
retrieved from the table; otherwise, it is computed by the recursive call whose result is then
recorded in the table.
The following algorithm implements this idea for the knapsack problem. After initializing the
table, the recursive function needs to be called with i = n (the number of items) and j = W
(the knapsack capacity).
AlgorithmMFKnapsack(i, j )
//Implements the memory function method for the knapsack problem
//Input: A nonnegative integer i indicating the number of the first items being
considered and a nonnegative integer j indicating the knapsack capacity
//Output: The value of an optimal feasible subset of the first i items
//Note: Uses as global variables input arrays Weights[1..n], Values[1..n],and
table F[0..n, 0..W ] whose entries are initialized with −1’s except for
row 0 and column 0 initialized with 0’s

Example-2 Let us apply the memory function method to the instance considered in Example
1. The table in Figure given below gives the results. Only 11 out of 20nontrivial values (i.e.,
not those in row 0 or in column 0) have been computed. Just one nontrivial entry, V (1, 2), is
retrieved rather than being recomputed. For larger instances, the proportion of such entries
can be significantly larger.

Figure: Example of solving an instance of the knapsack problem by the memory function algorithm
In general, we cannot expect more than a constant-factor gain in using the memory function
method for the knapsack problem, because its time efficiency class is the same as that of the
bottom-up algorithm

CSE, SVIT 14

86
ADA BCS401

5. Bellman-Ford Algorithm (Single source shortest path with –ve weights)

Problem definition: Given a graph and a source vertex s in graph, find shortest paths
from s to all vertices in the given graph. The graph may contain negative weight edges.
Note that we have discussed Dijkstra’s algorithm for single source shortest path problem.
Dijksra’s algorithm is a Greedy algorithm and time complexity is O(VlogV). But Dijkstra
doesn’t work for graphs with negative weight edges.
Bellman-Ford works for such graphs. Bellman-Ford is also simpler than Dijkstra and suites
well for distributed systems. But time complexity of Bellman-Ford is O(VE), which is more
than Dijkstra.
How it works? - Like other Dynamic Programming Problems, the algorithm calculates
shortest paths in bottom-up manner. It first calculates the shortest distances for the shortest
paths which have at-most one edge in the path. Then, it calculates shortest paths with at-most
2 edges, and so on.
Iteration i finds all shortest paths that use i edges. There can be maximum |V| – 1 edges in
any simple path, that is why the outer loop runs |v| – 1 times. The idea is, assuming that there
is no negative weight cycle, if we have calculated shortest paths with at most i edges, then an
iteration over all edges guarantees to give shortest path with at-most (i+1) edges

CSE, SVIT 15

87
ADA BCS401

Bellman-Ford algorithm to compute shortest path

CSE, SVIT 16

88
ADA BCS401

Another example

Figure: Steps of the Bellman Ford algorithm. The numbers with red squares indicate what changed on
each step.

6. Travelling Sales Person problem (T2:5.9),

CSE, SVIT 17

89
ADA BCS401

CSE, SVIT 18

90
ADA BCS401

CSE, SVIT 19

91
ADA BCS401

***
7. Space-Time Tradeoffs

Introduction

The main idea is to preprocess the problem’s input, in whole or in part, and store the additional
information obtained to accelerate solving the problem afterward. We call this approach as input
enhancement. The algorithms based on this approach are:

 Counting methods for sorting

 Boyer-Moore algorithm for string matching and its simplified version suggested by
Horspool

Sorting by Counting

The main idea here is to use the count variable. This variable shall give us the position of
the elements in its sorted order. Keeping the count variable as reference we can copy the elements
to a new list, so that we have a sorted list with us. The algorithm to do his is called the comparison
counting sort.

CSE, SVIT 20

92
93
94
Input Enhancement in String Matching- Harspool’s algorithm.

95
96
97
98
 Lecture Notes on
Analysis and Design of Algorithms
BCS401

Module-5 : Backtracking

Contents

1. Backtracking: 3. 0/1Knapsack problem

1.1. General method 3.1. LC Branch and Bound solution
1.2. N-Queens problem 3.2. FIFO Branch and Bound solution
1.3. Sum of subsets problem 4. NP-Complete and NP-Hard problems
1.4. Graph coloring 4.1. Basic concepts
1.5. Hamiltonian cycles 4.2. Non-deterministic algorithms
2. Branch and Bound: 4.3. P, NP, NP-Complete, and NP-Hard
2.1. Assignment Problem, classes
2.2. Travelling Sales Person problem

99
ADA BCS401

1. Backtracking
Some problems can be solved, by exhaustive search. The exhaustive-search technique
suggests generating all candidate solutions and then identifying the one (or the ones) with a
desired property.
Backtracking is a more intelligent variation of this approach. The principal idea is to
construct solutions one component at a time and evaluate such partially constructed
candidates as follows. If a partially constructed solution can be developed further without
violating the problem’s constraints, it is done by taking the first remaining legitimate option
for the next component. If there is no legitimate option for the next component, no
alternatives for any remaining component need to be considered. In this case, the algorithm
backtracks to replace the last component of the partially constructed solution with its next
option.
It is convenient to implement this kind of processing by constructing a tree of choices being
made, called the state-space tree. Its root represents an initial state before the search for a
solution begins. The nodes of the first level in the tree represent the choices made for the first
component of a solution; the nodes of the second level represent the choices for the second
component, and soon. A node in a state-space tree is said to be promising if it corresponds to
a partially constructed solution that may still lead to a complete solution; otherwise, it is
called non-promising. Leaves represent either non-promising dead ends or complete
solutions found by the algorithm.
In the majority of cases, a statespace tree for a backtracking algorithm is constructed in the
manner of depth-first search. If the current node is promising, its child is generated by adding
the first remaining legitimate option for the next component of a solution, and the processing
moves to this child. If the current node turns out to be non-promising, the algorithm
backtracks to the node’s parent to consider the next possible option for its last component; if
there is no such option, it backtracks one more level up the tree, and so on. Finally, if the
algorithm reaches a complete solution to the problem, it either stops (if just one solution is
required) or continues searching for other possible solutions.

1.1 General method (Textbook T2:7.1)

CSE, SVIT 1

100
ADA BCS401

General Algorithm (Recursive)

CSE, SVIT 2

101
ADA BCS401

General Algorithm (Iterative)

General Algorithm for backtracking (From textbook T1)

1.2 N-Queens problem

The problem is to place n queens on an n × n chessboard so that no two queens attack each
other by being in the same row or in the same column or on the same diagonal.
So let us consider the four-queens problem and solve it by the backtracking technique.
Since each of the four queens has to be placed in its own row, all we need to do is to assign a
column for each queen on the board presented in figure.

CSE, SVIT 3

102
ADA BCS401

We start with the empty board and then place queen 1 in the first possible position of its row,
which is in column 1 of row 1. Then we place queen 2, after trying unsuccessfully columns 1
and 2, in the first acceptable position for it, which is square (2, 3), the square in row 2 and
column 3. This proves to be a dead end because there is no acceptable position for queen 3.
So, the algorithm backtracks and puts queen 2 in the next possible position at (2, 4). Then
queen 3 is placed at (3, 2), which proves to be another dead end. The algorithm then
backtracks all the way to queen 1 and moves it to (1, 2). Queen 2 then goes to (2, 4), queen 3
to(3, 1), and queen 4 to (4, 3), which is a solution to the problem. The state-space tree of this
search is shown in figure.

Figure: State-space tree of solving the four-queens problem by backtracking. ×

denotes an unsuccessful attempt to place a queen in the indicated column. The
numbers above the nodes indicate the order in which the nodes are generated.

If other solutions need to be found, the algorithm can simply resume its operations at the leaf
at which it stopped. Alternatively, we can use the board’s symmetry for this purpose.
Finally, it should be pointed out that a single solution to the n-queens problem for any n ≥ 4
can be found in linear time.

Note: The algorithm NQueens() is not in the syllabus. It is given here for interested learners.
The algorithm is referred from textbook T2.

CSE, SVIT 4

103
ADA BCS401

1.3 Sum of subsets problem

Problem definition: Find a subset of a given set A = {a1, . . . , an} of n positive integers
whose sum is equal to a given positive integer d.
For example, for A = {1, 2, 5, 6, 8} and d = 9, there are two solutions: {1, 2, 6} and {1, 8}.
Of course, some instances of this problem may have no solutions.
It is convenient to sort the set’s elements in increasing order. So, we will assume that
a1< a2< . . . < an.
The state-space tree can be constructed as a binary tree like that in Figure shown below for
the instance A = {3, 5, 6, 7} and d = 15.
The number inside a node is the sum of the elements already included in the subsets
represented by the node. The inequality below a leaf indicates the reason for its termination.

CSE, SVIT 5

104
ADA BCS401

The root of the tree represents the starting point, with no decisions about the given elements
made as yet. Its left and right children represent, respectively, inclusion and exclusion of a1 in
a set being sought.
Similarly, going to the left from a node of the first level corresponds to inclusion of a2 while
going to the right corresponds to its exclusion, and so on. Thus, a path from the root to a node
on the ith level of the tree indicates which of the first in numbers have been included in the
subsets represented by that node.
We record the value of s, the sum of these numbers, in the node. If s is equal to d, we have a
solution to the problem. We can either report this result and stop or, if all the solutions need
to be found, continue by backtracking to the node’s parent. If s is not equal to d, we can
terminate the node as non-promising if either of the following two inequalities holds:

Example: Apply backtracking to solve the following instance of the subset sum problem: A
= {1, 3, 4, 5} and d = 11.

1.4 Graph coloring

CSE, SVIT 6

105
ADA BCS401

CSE, SVIT 7

106
ADA BCS401

CSE, SVIT 8

107
ADA BCS401

Analysis

CSE, SVIT 9

108
ADA BCS401

1.5 Hamiltonian cycles

CSE, SVIT 10

109
ADA BCS401

CSE, SVIT 11

110
ADA BCS401

2. Branch and Bound

Recall that the central idea of backtracking, discussed in the previous section, is tocut off a
branch of the problem’s state-space tree as soon as we can deduce that it cannot lead to a
solution. This idea can be strengthened further if we deal with an optimization problem.
An optimization problem seeks to minimize or maximize some objective function (a tour
length, the value of items selected, the cost of an assignment, and the like), usually subject to
some constraints. An optimal solution is a feasible solution with the best value of the
objective function (e.g., the shortest Hamiltonian circuit or the most valuable subset of items
that fit the knapsack).
Compared to backtracking, branch-and-bound requires two additional items:
1. a way to provide, for every node of a state-space tree, a bound on the best value of
the objective function on any solution that can be obtained by adding further
components to the partially constructed solution represented by the node
2. the value of the best solution seen so far
In general, we terminate a search path at the current node in a state-space tree of a branch-
and-bound algorithm for any one of the following three reasons:
1. The value of the node’s bound is not better than the value of the best solution seen so
far.
2. The node represents no feasible solutions because the constraints of the problem are
already violated.
3. The subset of feasible solutions represented by the node consists of a single point (and
hence no further choices can be made)—in this case, we compare the value of the
objective function for this feasible solution with that of the best solution seen so far
and update the latter with the former if the new solution is better.

2.1 Assignment Problem

Let us illustrate the branch-and-bound approach by applying it to the problem of assigning n
people to n jobs so that the total cost of the assignment is as small as possible.
An instance of the assignment problem is specified by an n × n cost matrix C so that we can
state the problem as follows: select one element in each row of the matrix so that no two
selected elements are in the same column and their sum is the smallest possible. We will
demonstrate how this problem can be solved using the branch-and-bound technique by
considering the small instance of the problem. Consider the data given below.

CSE, SVIT 12

111
ADA BCS401

How can we find a lower bound on the cost of an optimal selection without actually solving
the problem?
We can do this by several methods. For example, it is clear that the cost of any solution,
including an optimal one, cannot be smaller than the sum of the smallest elements in each
of the matrix’s rows. For the instance here, this sum is 2 + 3+ 1+ 4 = 10.We can and will
apply the same thinking to partially constructed solutions. For example, for any legitimate
selection that selects 9 from the first row, the lower bound will be 9 + 3 + 1+ 4 = 17.
Rather than generating a single child of the last promising node as we did in backtracking, we
will generate all the children of the most promising node among non-terminated leaves in the
current tree. (Non terminated, i.e., still promising, leaves are also called live.) How can we
tell which of the nodes is most promising? We can do this by comparing the lower bounds of
the live nodes. It is sensible to consider a node with the best bound as most promising,
although this does not, of course, preclude the possibility that an optimal solution will
ultimately belong to a different branch of the state-space tree. This variation of the strategy is
called the best-first branch-and-bound.
We start with the root that corresponds to no elements selected from the cost matrix. The
lower-bound value for the root, denoted lb, is 10. The nodes on the first level of the tree
correspond to selections of an element in the first row of the matrix, i.e., a job for person a.
See the figure given below.

Figure: Levels 0 and 1 of the state-space tree for the instance of the assignment
problem being solved with the best-first branch-and-bound algorithm. The number
above a node shows the order in which the node was generated. A node’s fields
indicate the job number assigned to person a and the lower bound value, lb, for this
node.

So we have four live leaves—nodes 1 through 4—that may contain an optimal solution. The
most promising of them is node 2 because it has the smallest lower bound value. Following
our best-first search strategy, we branch out from that node first by considering the three
different ways of selecting an element from the second row and not in the second column -
the three different jobs that can be assigned to person b. See the figure given below (Fig
12.7).

CSE, SVIT 13

112
ADA BCS401

Of the six live leaves—nodes 1, 3, 4, 5, 6, and 7—that may contain an optimal solution, we
again choose the one with the smallest lower bound, node 5. First, we consider selecting the
third column’s element from c’s row (i.e., assigning person c to job 3); this leaves us with no
choice but to select the element from the fourth column of d’s row (assigning person d to job
4). This yields leaf 8 (Figure 12.7), which corresponds to the feasible solution {a→2, b→1,
c→3, d →4} with the total cost of 13. Its sibling, node 9, corresponds to the feasible solution
{a→2,b→1, c→4, d →3} with the total cost of 25. Since its cost is larger than the cost of the
solution represented by leaf 8, node 9 is simply terminated. (Of course, if its cost were
smaller than 13, we would have to replace the information about the best solution seen so far
with the data provided by this node.)

Now, as we inspect each of the live leaves of the last state-space tree—nodes1, 3, 4, 6, and 7
in Figure 12.7—we discover that their lower-bound values are not smaller than 13, the value
of the best selection seen so far (leaf 8). Hence, we terminate all of them and recognize the
solution represented by leaf 8 as the optimal solution to the problem.

2.2 Travelling Sales Person problem

We will be able to apply the branch-and-bound technique to instances of the traveling
salesman problem if we come up with a reasonable lower bound on tour lengths. One very
simple lower bound can be obtained by finding the smallest element in the intercity distance
matrix D and multiplying it by the number of cities n.

CSE, SVIT 14

113
ADA BCS401

But there is a less obvious and more informative lower bound for instances with symmetric
matrix D, which does not require a lot of work to compute. We can compute a lower bound
on the length l of any tour as follows. For each city i, 1≤ i ≤ n, find the sum si of the distances
from city i to the two nearest cities; compute the sums of these n numbers, divide the result
by 2, and, if all the distances are integers, round up the result to the nearest integer:
lb = ⌈s/2⌉... (1)
For example, for the instance in Figure 2.2a, formula (1) yields

Moreover, for any subset of tours that must include particular edges of a given graph, we can
modify lower bound (formula 1) accordingly. For example, for all the Hamiltonian circuits of
the graph in Figure 2.2a that must include edge (a, d), we get the following lower bound by
summing up the lengths of the two shortest edges incident with each of the vertices, with the
required inclusion of edges (a, d)and (d, a):

We now apply the branch-and-bound algorithm, with the bounding function given by
formula-1, to find the shortest Hamiltonian circuit for the graph in Figure 2.2a.
To reduce the amount of potential work, we take advantage of two observations.
1. First, without loss of generality, we can consider only tours that start at a.
2. Second, because our graph is undirected, we can generate only tours in which b is
visited before c. (Refer Note at the end of section 2.2 for more details)
In addition, after visiting n−1= 4 cities, a tour has no choice but to visit the remaining
unvisited city and return to the starting one. The state-space tree tracing the algorithm’s
application is given in Figure 2.2b.

Note: An inspection of graph with 4 nodes (figure given below) reveals three pairs of tours
that differ only by their direction. Hence, we could cut the number of vertex permutations by
half. We could, for example, choose any two intermediate vertices, say, b and c, and then
consider only permutations in which b precedes c. (This trick implicitly defines a tour’s
direction.)

Figure: Solution to a small instance of the traveling salesman problem by exhaustive search.

CSE, SVIT 15

114
ADA BCS401

Figure 2.2(a)Weighted graph. (b) State-space tree of the branch-and-bound algorithm to find
a shortest Hamiltonian circuit in this graph. The list of vertices in a node specifies a
beginning part of the Hamiltonian circuits represented by the node.

Discussion
The strengths and weaknesses of backtracking are applicable to branch-and-bound as well.
The state-space tree technique enables us to solve many large instances of difficult
combinatorial problems. As a rule, however, it is virtually impossible to predict which
instances will be solvable in a realistic amount of time and which will not.
In contrast to backtracking, solving a problem by branch-and-bound has both the challenge
and opportunity of choosing the order of node generation and finding a good bounding
function. Though the best-first rule we used above is a sensible approach, it may or may not
lead to a solution faster than other strategies. (Artificial intelligence researchers are
particularly interested in different strategies for developing state-space trees.)
Finding a good bounding function is usually not a simple task. On the one hand, we want this
function to be easy to compute. On the other hand, it cannot be too simplistic - otherwise, it
would fail in its principal task to prune as many branches of a state-space tree as soon as
possible. Striking a proper balance between these two competing requirements may require
intensive experimentation with a wide variety of instances of the problem in question.

CSE, SVIT 16

115
ADA BCS401

3. 0/1 Knapsack problem

Note: For this topic as per the syllabus both textbooks T1 & T2 are suggested.
Here we discuss the concepts from T1 first and then that of from T2.
Topic form T1 (Levitin)
Let us now discuss how we can apply the branch-and-bound technique to solving the
knapsack problem. Given n items of known weights wi and values vi, i = 1, 2, . . . , n, and a
knapsack of capacity W, find the most valuable subset of the items that fit in the knapsack.
∑ 𝑤𝑖𝑥𝑖 ≤ 𝑊 𝑎𝑛𝑑 ∑ 𝑖𝑥𝑖 𝑖𝑠 𝑚𝑎𝑥𝑖𝑚𝑖𝑧𝑒𝑑, 𝑤ℎ𝑒𝑟𝑒 𝑥𝑖 = 0 𝑜𝑟 1
1≤𝑖≤𝑛 1≤𝑖≤𝑛
It is convenient to order the items of a given instance in descending order by their value-to-
weight ratios.

Each node on the ith level of state space tree, 0 ≤ i ≤ n, represents all the subsets of n items
that include a particular selection made from the first i ordered items. This particular
selection is uniquely determined by the path from the root to the node: a branch going to the
left indicates the inclusion of the next item, and a branch going to the right indicates its
exclusion.
We record the total weight w and the total value v of this selection in the node, along with
some upper bound ub on the value of any subset that can be obtained by adding zero or more
items to this selection. A simple way to compute the upper bound ub is to add to v, the total
value of the items already selected, the product of the remaining capacity of the knapsack W
– w and the best per unit payoff among the remaining items, which is vi+1/wi+1:
ub = v + (W − w)(vi+1/wi+1).
Example: Consider the following problem. The items are already ordered in descending order
of their value-to-weight ratios.

Let us apply the branch-and-bound algorithm. At the root of the state-space tree (see Figure
12.8), no items have been selected as yet. Hence, both the total weight of the items already
selected w and their total value v are equal to 0. The value of the upper bound is 100.
Node 1, the left child of the root, represents the subsets that include item 1. The total weight
and value of the items already included are 4 and 40, respectively; the value of the upper
bound is 40 + (10 − 4) * 6 = 76.

CSE, SVIT 17

116
ADA BCS401

Node 2 represents the subsets that do not include item 1. Accordingly, w = 0, v = 0, and ub =
0 + (10 − 0) * 6 = 60. Since node 1 has a larger upper bound than the upper bound of node 2,
it is more promising for this maximization problem, and we branch from node 1 first. Its
children—nodes 3 and 4—represent subsets with item 1 and with and without item 2,
respectively. Since the total weight w of every subset represented by node 3 exceeds the
knapsack’s capacity, node 3 can be terminated immediately.
Node 4 has the same values of w and v as its parent; the upper bound ub is equal to 40 + (10
− 4) * 5 = 70. Selecting node 4 over node 2 for the next branching (Due to better ub), we get
nodes 5 and 6 by respectively including and excluding item 3. The total weights and values as
well as the upper bounds for these nodes are computed in the same way as for the preceding
nodes.
Branching from node 5 yields node 7, which represents no feasible solutions, and node 8,
which represents just a single subset {1, 3} of value 65. The remaining live nodes 2 and 6
have smaller upper-bound values than the value of the solution represented by node 8. Hence,
both can be terminated making the subset {1, 3} of node 8 the optimal solution to the
problem.

Solving the knapsack problem by a branch-and-bound algorithm has a rather unusual

characteristic. Typically, internal nodes of a state-space tree do not define a point of the
problem’s search space, because some of the solution’s components remain undefined. (See,
for example, the branch-and-bound tree for the assignment problem discussed in the

CSE, SVIT 18

117
ADA BCS401

preceding subsection.) For the knapsack problem, however, every node of the tree represents
a subset of the items given. We can use this fact to update the information about the best
subset seen so far after generating each new node in the tree. If we had done this for the
instance investigated above, we could have terminated nodes 2 and 6 before node 8 was
generated because they both are inferior to the subset of value 65 of node 5.

Concepts form textbook T2 (Horowitz)

Let us understand some of the terminologies used in backtracking &branch and bound.
 Live node - a node which has been generated and all of whose children are not yet been
generated.
 E-node - is a live node whose children are currently being explored. In other words, an E-
node is a node currently being expanded.
 Dead node - a node that is either not to be expanded further, or for which all of its
children have been generated
 Bounding Function - will be used to kill live nodes without generating all their children.
 Backtracking - is depth first node generation with bounding functions.
 Branch-And-Bound is a method in which E-node remains E-node until it is dead.
 Breadth-First-Search: Branch-and Bound with each new node placed in a queue. The
front of the queen becomes the new E-node.
 Depth-Search (D-Search): New nodes are placed in to a stack. The last node added is the
first to be explored.

CSE, SVIT 19

118
ADA BCS401

0/1 Knapsack problem - Branch and Bound based solution

As the technique discussed here is applicable for minimization problems, let us convert the
knapsack problem (maximizing the profit) into minimization problem by negating the
objective function

CSE, SVIT 20

119
ADA BCS401

3.1 LC (Least Cost) Branch and Bound solution

CSE, SVIT 21

120
ADA BCS401

CSE, SVIT 22

121
ADA BCS401

3.2 FIFO Branch and Bound solution

CSE, SVIT 23

122
ADA BCS401

Conclusion

CSE, SVIT 24

123
ADA BCS401

4. NP-Complete and NP-Hard problems

4.1 Basic concepts
For many of the problems we know and study, the best algorithms for their solution have
computing times can be clustered into two groups;
1. Solutions are bounded by the polynomial- Examples include Binary search O(log n),
Linear search O(n), sorting algorithms like merge sort O(n log n), Bubble sort O(n2)
&matrix multiplication O(n3) or in general O(nk) where k is a constant.
2. Solutions are bounded by a non-polynomial-Examples include travelling salesman
problem O(n22n) & knapsack problem O(2n/2). As the time increases exponentially,
even moderate size problems cannot be solved.
So far, no one has been able to device an algorithm which is bounded by the polynomial for
the problems belonging to the non-polynomial. However impossibility of such an algorithm
is not proved.

4.2 Non deterministic algorithms

We also need the idea of two models of computer (Turing machine): deterministic and non-
deterministic. A deterministic computer is the regular computer we always thinking of; a non-
deterministic computer is one that is just like we’re used to except that is has unlimited
parallelism, so that any time you come to a branch, you spawn a new “process” and examine
both sides.
When the result of every operation is uniquely defined then it is called deterministic
algorithm.
When the outcome is not uniquely defined but is limited to a specific set of possibilities, we
call it non deterministic algorithm.
We use new statements to specify such non deterministic algorithms.
 choice(S) - arbitrarily choose one of the elements of set S
 failure - signals an unsuccessful completion
 success - signals a successful completion
The assignment X = choice(1:n) could result in X being assigned any value from the integer
range[1..n]. There is no rule specifying how this value is chosen.

“The nondeterministic algorithms terminates unsuccessfully iff there is no set of choices

which leads to the successful signal”.

Example-1: Searching an element x in a given set of elements A(1:n). We are required to

determine an index j such that A(j) = x or j = 0 if x is not present.
j := choice(1:n)
if A(j) = x then print(j); success endif

CSE, SVIT 25

124
ADA BCS401

print(‘0’); failure

Example-2: Checking whether n integers are sorted or not

procedure NSORT(A,n);
//sort n positive integers//
var integer A(n), B(n), n, i, j;
begin
B := 0; //B is initialized to zero//
for i := 1 to n do
begin
j := choice(1:n);
if B(j) <> 0 then failure;
B(j) := A(j);
end;

for i := 1 to n-1 do //verify order//

if B(i) > B(i+1) then failure;
print(B);
success;
end.
“A nondeterministic machine does not make any copies of an algorithm every time a choice
is to be made. Instead it has the ability to correctly choose an element from the given set”.
A deterministic interpretation of the nondeterministic algorithm can be done by making
unbounded parallelism in the computation. Each time a choice is to be made, the algorithm
makes several copies of itself, one copy is made for each of the possible choices.

Decision vs Optimization algorithms

An optimization problem tries to find an optimal solution.
A decision problem tries to answer a yes/no question. Most of the problems can be specified
in decision and optimization versions.
For example, Traveling salesman problem can be stated as two ways
 Optimization - find hamiltonian cycle of minimum weight,
 Decision - is there a hamiltonian cycle of weight  k?
For graph coloring problem,
 Optimization – find the minimum number of colors needed to color the vertices of a
graph so that no two adjacent vertices are colored the same color
 Decision - whether there exists such a coloring of the graph’s vertices with no more
than m colors?
Many optimization problems can be recast in to decision problems with the property that the
decision algorithm can be solved in polynomial time if and only if the corresponding
optimization problem.

CSE, SVIT 26

125
ADA BCS401

4.3 P, NP, NP-Complete and NP-Hard classes

NP stands for Non-deterministic Polynomial time.
Definition: P is a set of all decision problems solvable by a deterministic algorithm in
polynomial time.
Definition: NP is the set of all decision problems solvable by a nondeterministic algorithm in
polynomial time. This also implies P ⊆ NP
Problems known to be in P are trivially in NP — the nondeterministic machine just never
troubles itself to fork another process, and acts just like a deterministic one. One example of a
problem not in P but in NP is Integer Factorization.

But there are some problems which are known to be in NP but don’t know if they’re in P. The
traditional example is the decision-problem version of the Travelling Salesman Problem
(decision-TSP). It’s not known whether decision-TSP is in P: there’s no known poly-time
solution, but there’s no proof such a solution doesn’t exist.
There are problems that are known to be neither in P nor NP; a simple example is to
enumerate all the bit vectors of length n. No matter what, that takes 2n steps.
Now, one more concept: given decision problems P and Q, if an algorithm can transform a
solution for P into a solution for Q in polynomial time, it’s said that Q is poly-time
reducible (or just reducible) to P.
The most famous unsolved problem in computer science is “whether P=NP or P≠NP? ”

Figure: Commonly believed

relationship between P and NP

Figure: Commonly believed relationship between P, NP, NP-

Complete and NP-hard problems

Definition: A decision problem D is said to be NP-complete if:

1. it belongs to class NP
2. every problem in NP is polynomially reducible to D
The fact that closely related decision problems are polynomially reducible to each other is not
very surprising. For example, Hamiltonian circuit problem is polynomially reducible to the
decision version of the traveling salesman problem.

CSE, SVIT 27

126
ADA BCS401

NP-Complete problems have the property that it can be solved in polynomial time if all other
NP-Complete problems can be solved in polynomial time. i.e if anyone ever finds a poly-time
solution to one NP-complete problem, they’ve automatically got one for all the NP-complete
problems; that will also mean that P=NP.
Example for NP-complete is CNF-satisfiability problem. The CNF-satisfiability problem
deals with boolean expressions. This is given by Cook in 1971. The CNF-satisfiability
problem asks whether or not one can assign values true and false to variables of a given
boolean expression in its CNF form to make the entire expression true.
Over the years many problems in NP have been proved to be in P (like Primality Testing).
Still, there are many problems in NP not proved to be in P. i.e. the question still remains
whether P=NP? NP Complete Problems helps in solving this question. They are a subset
of NP problems with the property that all other NP problems can be reduced to any of them in
polynomial time. So, they are the hardest problems in NP, in terms of running time. If it can
be showed that any NP-Complete problem is in P, then all problems in NP will be in P
(because of NP-Complete definition), and hence P=NP=NPC.
NP Hard Problems - These problems need not have any bound on their running time. If
any NP-Complete Problem is polynomial time reducible to a problem X, that problem X
belongs to NP-Hard class. Hence, all NP-Complete problems are also NP-Hard. In other
words if a NP-Hard problem is non-deterministic polynomial time solvable, it is a NP-
Complete problem. Example of a NP problem that is not NPC is Halting Problem.
If a NP-Hard problem can be solved in polynomial time then all NP-Complete can be solved
in polynomial time.
“All NP-Complete problems are NP-Hard but not all NP-Hard problems are not NP-
Complete.”NP-Complete problems are subclass of NP-Hard
The more conventional optimization version of Traveling Salesman Problem for finding the
shortest route is NP-hard, not strictly NP-complete.

*****

CSE, SVIT 28

127
 BCS401
Model Question Paper with effect from 2023-24 (CBCS Scheme)

USN

Fourth Semester B.E. Degree Examination

Analysis and Designs of Algorithms

TIME: 03 Hours Max. Marks: 100

Note: 01. Answer any FIVE full questions, choosing at least ONE question from each MODULE.

Module -1 BL Marks
Q.01 a Define algorithm. Explain asymptotic notations Big Oh, Big Omega and Big Theta L2 08
notations
b Explain the general plan for analyzing the efficiency of a recursive algorithm. Suggest L2 08
a recursive algorithm to find factorial of number. Derive its efficiency
c If t1(n) ∈ O(g1(n)) and t2(n) ∈ O(g2(n)), then show that t1(n) + t2(n) ∈ O(max{g1(n), L2 04
g2(n)}).
OR
Q.02 a With neat diagram explain different steps in designing and analyzing an algorithm L2 08
b Explain the general plan for analyzing the efficiency of a non-recursive algorithm. L2 08
Suggest a non-recursive algorithm to find maximum element in the list of n numbers.
Derive its efficiency
c With the algorithm derive the worst case efficiency for Bubble sort L2 04
Module-2
Q. 03 a Explain the concept of divide and conquer. Design an algorithm for merge sort and L2 10
derive its time complexity
b Design an insertion sort algorithm and obtain its time complexity. Apply insertion sort L3 10
on these elements. 25,75,40,10,20,
OR
Q.04 a Explain Strassen’s matrix multiplication and derive its time complexity L2 10
b Design an algorithm for quick sort algorithm. Apply quick sort on these elements. L3 10
25,75,40,10,20,05,15
Module-3
Q. 05 a Define AVL Trees. Explain its four rotation types L2 10
b Construct bottom up heap for the list 2,9,7,6,5,8. Obtain its time complexity L3 10
OR
Q. 06 a Define heap. Explain the properties of heap along with its representation. L2 10
b Design Horspools algorithm for string matching. Apply Horspools algorithm to find L3 10
the pattern BARBER in the text: JIM_SAW_ME_IN_A_BARBERSHOP
Module-4
Q. 07 a Construct minimum cost spanning tree using Kruskals algorithm for the following L3 10
graph.

b What are Huffman Trees? Construct the Huffman tree for the following data. L3 10
Character A B C D E -
Probability 0.5 0.35 0.5 0.1 0.4 0.2
Encode DAD-CBE using Huffman Encoding.

Page 01 of 02
128
 BCS401
OR
Q. 08 a Apply Dijkstra’s algorithm to find single source shortest path for the given graph by L3 10
considering S as the source vertex.

b Define transitive closure of a graph. Apply Warshalls algorithm to compute transitive L3 10

closure of a directed graph

Module-5
Q. 09 a Explain the following with examples L2 10
i) P problem
ii) NP Problem
iii) NP- Complete problem
iv) NP – Hard Problems
b What is backtracking? Apply backtracking to solve the below instance of sum of subset L3 10
problem S={5,10,12,13,15,18} d=30
OR
Q. 10 a Illustrate N queen’s problem using backtracking to solve 4-Queens problem L2 10
b Using Branch and Bound technique solve the below instance of knapsack problem. L3 10
Item Weight Value
1 2 12
2 1 10
3 3 20
4 2 5

Capacity 5

Page 02 of 02
129