Module 1: INTRODUCTION TO ALGORITHMS

Algorithm, Algorithm pseudocode conventions, Performance analysis - Space complexity, Time complexity,
Asymptotic notations; Amortized analysis – Aggregate analysis, Accounting method, Potential method;
Recurrences - Substitution method, Recursion-tree method, Master method.

Algorithm
The word Algorithm means ” A set of finite rules or instructions to be followed in calculations or other problem-
solving operations ” Or ” A procedure for solving a mathematical problem in a finite number of steps that frequently
involves recursive operations”. Therefore Algorithm refers to a sequence of finite steps to solve a particular problem.

Use of the Algorithms:-

Algorithms play a crucial role in various fields and have many applications. Some of the key areas where algorithms
are used include:
Computer Science: Algorithms form the basis of computer programming and are used to solve problems ranging
from simple sorting and searching to complex tasks such as artificial intelligence and machine learning.
Mathematics: Algorithms are used to solve mathematical problems, such as finding the optimal solution to a system
of linear equations or finding the shortest path in a graph.
Operations Research: Algorithms are used to optimize and make decisions in fields such as transportation, logistics,
and resource allocation.
Artificial Intelligence: Algorithms are the foundation of artificial intelligence and machine learning, and are used
to develop intelligent systems that can perform tasks such as image recognition, natural language processing, and
decision-making.
Data Science: Algorithms are used to analyze, process, and extract insights from large amounts of data in fields
such as marketing, finance, and healthcare.
These are just a few examples of the many applications of algorithms. The use of algorithms is continually expanding
as new technologies and fields emerge, making it a vital component of modern society.
What is the need for algorithms:
1.Algorithms are necessary for solving complex problems efficiently and effectively.
2.They help to automate processes and make them more reliable, faster, and easier to perform.
3.Algorithms also enable computers to perform tasks that would be difficult or impossible for humans to do
manually.
4.They are used in various fields such as mathematics, computer science, engineering, finance, and many others to
optimize processes, analyze data, make predictions, and provide solutions to problems.

What are the Characteristics of an Algorithm?

An algorithm is a finite set of instructions that, if followed, accomplishes a particular task. In addition, all algorithms
must satisfy the following criteria:
1. Input. Zero or more quantities are externally supplied.
2. Output. At least one quantity is produced.
3. Definiteness. Each instruction is clear and unambiguous.
4. Finiteness. If we trace out the instructions of an algorithm, then for all cases, the algorithm terminates after a
finite number of steps.
5. Effectiveness. Every instruction must be very basic so that it can be carried out, in principle, by a person using
only pencil and paper. It is not enough that each operation be definite as in criterion3; it also must be feasible.

 According to criterion 3, each operation must be definite, meaning that it must be perfectly clear what should be
done. Directions such as add 9 or 26 to z or compute 9/0 are not permitted because it is not clear which of the
two possibilities should be done or what the result is.
 Effectiveness example – Performing arithmetic on integers is an example of an effective operation, but arithmetic
with real numbers is not, since some values may be expressible only by infinitely long decimal expansion.
Adding two such numbers would violate the effectiveness property.
Algorithms that are definite and effective are also called computational procedures.
The study of algorithms includes 4 distinct areas:
1. How to devise algorithm – By mastering in algorithms design strategies, it will become easier for programmers
to devise new and useful algorithms.
2. How to validate algorithm – Once an algorithm is devised, it is necessary to show that it computes the correct
answer for all possible legal inputs. We refer to this process as algorithm validation. After completion of
algorithm validation, a program can be written and a second phase begins. This phase is referred to as program
proving or sometimes as program verification.

3. How to analyse algorithms – This field of study is called analysis of algorithms. As an algorithm is executed, it
uses the computer's central processing unit (CPU) to perform operations and its memory (both immediate and
auxiliary) to hold the program and data. Analysis of algorithms or performance analysis refers to the task of
determining how much computing time and storage an algorithm requires. This is a challenging area which
sometimes requires great mathematical skill. An important result of this study is that two algorithms can be
compared quantitatively.

How to test a program – Testing a program consists of two phases: debugging and profiling (or performance
measurement). Debugging is the process of executing programs on sample data sets to determine whether faulty
results occur and, if so, to correct them. However, as E.Dijkstra has pointed out, "debugging can only point to the
presence of errors, but not to their absence. "In cases in which we cannot verify the correctness of output on
sample data, the following strategy can be employed: let more than one programmer develop programs for the
same problem, and compare the outputs produced by these programs. If the outputs match, then there is a good
chance that they are correct. Profiling or performance measurement is the process of executing a correct program
on datasets and measuring the time and space it takes to compute the results.
Types of Algorithms:
There are several types of algorithms available. Some important algorithms are:
1. Brute Force Algorithm: It is the simplest approach for a problem. A brute force algorithm is the first approach
that comes to finding when we see a problem.
2. Recursive Algorithm: A recursive algorithm is based on recursion. In this case, a problem is broken into
several sub-parts and called the same function again and again.
3. Backtracking Algorithm: The backtracking algorithm basically builds the solution by searching among all
possible solutions. Using this algorithm, we keep on building the solution following criteria. Whenever a solution
fails we trace back to the failure point and build on the next solution and continue this process till we find the
solution or all possible solutions are looked after.
4. Searching Algorithm: Searching algorithms are the ones that are used for searching elements or groups of
elements from a particular data structure. They can be of different types based on their approach or the data
structure in which the element should be found.
5. Sorting Algorithm: Sorting is arranging a group of data in a particular manner according to the requirement.
The algorithms which help in performing this function are called sorting algorithms. Generally sorting algorithms
are used to sort groups of data in an increasing or decreasing manner.
6. Hashing Algorithm: Hashing algorithms work similarly to the searching algorithm. But they contain an index
with a key ID. In hashing, a key is assigned to specific data.
7. Divide and Conquer Algorithm: This algorithm breaks a problem into sub-problems, solves a single sub-
problem and merges the solutions together to get the final solution. It consists of the following three steps:
 Divide
 Solve
 Combine
8. Greedy Algorithm: In this type of algorithm the solution is built part by part. The solution of the next part is
built based on the immediate benefit of the next part. The one solution giving the most benefit will be chosen as
the solution for the next part.
9. Dynamic Programming Algorithm: This algorithm uses the concept of using the already found solution to
avoid repetitive calculation of the same part of the problem. It divides the problem into smaller overlapping
subproblems and solves them.
10. Randomized Algorithm: In the randomized algorithm we use a random number so it gives immediate benefit.
The random number helps in deciding the expected outcome.
Advantages of Algorithms:
 It is easy to understand.
 An algorithm is a step-wise representation of a solution to a given problem.
 In Algorithm the problem is broken down into smaller pieces or steps hence, it is easier for the programmer
to convert it into an actual program.

Disadvantages of Algorithms:
 Writing an algorithm takes a long time so it is time-consuming.
 Understanding complex logic through algorithms can be very difficult.
 Branching and Looping statements are difficult to show in Algorithms(imp).

How to Design an Algorithm?

In order to write an algorithm, the following things are needed as a pre-requisite:

1. The problem that is to be solved by this algorithm i.e. clear problem definition.
2. The constraints of the problem must be considered while solving the problem.
3. The input to be taken to solve the problem.
4. The output to be expected when the problem is solved.
5. The solution to this problem, is within the given constraints.

Example: Consider the example to add three numbers and print the sum.
Algorithm to add 3 numbers and print their sum:
1. START
2. Declare 3 integer variables num1, num2 and num3.
3. Take the three numbers, to be added, as inputs in variables num1, num2, and num3 respectively.
4. Declare an integer variable sum to store the resultant sum of the 3 numbers.
5. Add the 3 numbers and store the result in the variable sum.
6. Print the value of the variable sum
7. END

ALGORITHM SPECIFICATION
Pseudocode Convention

Pseudocode for expressing algorithms

What is a pseudocode?
Pseudocode, as the name suggests, is a false code or is a high-level description of an algorithm. In other words, we
can say that pseudocode is a cooked-up representation of an algorithm. Pseudocode can be easily understood by
someone who has basic knowledge of programming.
Pseudocode does not have a specific syntax unlike a program that is written using syntaxes of a particular language.
Hence, pseudocode cannot be executed on a computer, rather it eases the work of a programmer as it can be easily
understood.
While writing pseudocode one can use control structures that include for, while, if-then-else, repeat until. These
control structures are common in almost all programming languages and hence can be used when required. Once we
are done with the pseudocode we can easily modify it into a program in our preferred language.
Another advantage of pseudocode is that the running time of the program can be estimated in a general manner using
it. A pseudocode gives us a count of fundamental operations. Also, pseudocode avoids any ambiguity that might be
present in plain text.

Difference between Algorithm and Pseudocode:

An algorithm is a well defined sequence of instructions that provide a solution to the given problem.
A pseudocode is a method which is used to represent an algorithm.
An algorithm has some specific characteristics that describe the process.
A pseudocode on the other hand is not restricted to something. It’s only objective is to represent an algorithm in a
realistic manner.
An algorithm is written in plain English or general language.
A pseudocode is written with a hint of programming concepts such as control structures.
In computational theory, we distinguish between an algorithm and a program. The latter does not have to satisfy the
finiteness condition. For example, we can think of an operating system that continues in a loop until more jobs are
entered. Such a program does not terminate unless the system crashes.
We can describe an algorithm in many ways. We can use a natural language like English, although if we select this
option, we must make sure that the resulting instructions are definite. Graphic representations called flowcharts are
another possibility, but they work well only if the algorithm is small and simple.
 Algorithms are presented using a pseudocode consists of the steps as

1. Comments begin with // and continue until the end of line.

2. Blocks are indicated with matching braces: { and }. A compound statement (i.e., a collection of simple
statements) can be represented as a block. The body of a procedure also forms a block. Statements are delimited
by ;.

3. An identifier begins with a letter. The data types of variables are not explicitly declared. The types
will be clear from the context. Whether a variable is global or local to a procedure will also be evident from the
context. We assume simple datatypes such as integer, float, char, boolean, and soon. Compound data types can
be formed with records. Here is an example:
node = record
{ datatype_1 data_1;
.
.
.
datatype_n data_n;
node *link;
 }
In this example, link is a pointer to the record type node. Individual data items of a record can be
accessed with → and period. For instance, if p points to a record of type node, p→data_1 stands for the value
of the first field in the record. On the other hand, if q is a record of type node, q.data_1 will denote its first
field.

4. Assignment of values to variables is done using the assignment statement

〈variable〉 := 〈expression〉;
5. There are two Boolean values true and false. In order to produce these values, the logical operators and, or, and
not and the relational operators <, ≤, =, ≠, ≥ and > are provided.

6. Elements of multidimensional arrays are accessed using [ and ]. For example, if A is a two dimensional array,
the (i, j)th element of the array is denoted as - A[i, j]. Array indices start at zero.

7. The following looping statements are employed: for, while, and repeat-until. The while loop takes the following
form
while 〈condition〉 do
{ 〈statement 1〉
.
.
.
〈statement n〉
}
As long as 〈condition〉 is true, the statements get executed. When (condition) becomes false, the loop is exited.
The value of 〈condition〉 is evaluated at the top of the loop. The general form of a for loop is
for variable := valuel to value2 step step do
{ 〈statement1〉
.
.
.
〈statementn〉
}
Here valuel, value2, and step are arithmetic expressions. A variable of type integer or real or a numerical constant
is a simple form of an arithmetic expression. The clause step step is optional and taken as +1 if it does not occur,
step could either be positive or negative. Variable is tested for termination at the start of each iteration. The for
loop can be implemented as a while loop as follows:

variable := valuel;
fin := value2;
incr := step;
while ((variable - fin) * step ≤ 0) do
{
〈statement 1〉
.
.
.
 〈statement n〉
variable := variable + incr;
}
A repeat-until statement is constructed as follows:
repeat
〈statement 1〉
.
.
.
〈statement n〉

until 〈condition〉
Statements are executed as long as the condition is false. Also, we can use break and return statements.
8. A conditional statement has the following forms:
if 〈condition〉 then 〈statement〉
if 〈condition) then〈statement1〉 else 〈statement2〉

9. case
{
:〈condition 1〉: 〈sttement 1〉
.
.
.
:〈condition n〉: 〈sttement n〉
:else: 〈sttement n + 1〉
}

10. Input and output are done using the instructions read and write. No format is used to specify the size of input or
output quantities.

11. There is only one type of procedure: Algorithm. An algorithm consists of a heading and a body. The heading
takes the form Algorithm Name(〈parameter list〉) where Name is the name of the procedure and (〈parameter list〉)
is a listing of the procedure parameters. The body has one or more (simple or compound) statements enclosed
within braces { and }. An algorithm may or may not return any values. Simple variables to procedures are passed
by value. Arrays and records are passed by reference. An array name or a record name is treated as a pointer to
the respective datatype.

Consider an example of calculating the area of a circle.

Algorithm:
1. Start.
2. Read r: radius value as the input given by the user.
3. Calculate the Area: 3.14 * r * r.
4. Display the Area.
5. End.
 Pseudocode:
AreaofCircle()
{
BEGIN
Read: Number radius, Area;
Input r;
Area = 3.14 * r * r;
Output Area;
END
}
Advantages of Pseudocode

The following algorithm finds and then returns maximum value form a given set of n values.
1 Algorithm Max(A, n)
2 // A is array of size n
3 {
4 Result := A[0];
5 for i := 2 to n do
6 if A[i] > Result then
7 Result := A[i];
8 return Result;
9 }
Name of algorithm is Max, A and n are procedure parameters. Result and i are local variables.

Selection Sort Algorithm Pseudocode

1 Algorithm SelectionSort(A, n)
2 sort the array A[1: n] into nondecreasing order
3 {
4 for i := 1 to n do
5 {
6 j := i;
7 for k := i + 1 to n do
8 if (A[k] < A[j] ) then
9 j := k;
10 t := A[i];
11 A[i] := A[j];
12 A[j] := t;
13 }
14 }

Performance Analysis

What is Performance Analysis of an algorithm?

If we want to go from city "A" to city "B", there can be many ways of doing this. We can go by flight, by bus, by
train and also by bicycle. Depending on the availability and convenience, we choose the one which suits us.
Similarly, in computer science, there are multiple algorithms to solve a problem. When we have more than one
algorithm to solve a problem, we need to select the best one. Performance analysis helps us to select the best
algorithm from multiple algorithms to solve a problem.
Performance of an algorithm is a process of making evaluative judgment about algorithms.
Performance of an algorithm means predicting the resources which are required to an algorithm to perform
its task.
Generally, the performance of an algorithm depends on the following elements...
1. Whether that algorithm is providing the exact solution for the problem?
2. Whether it is easy to understand?
3. Whether it is easy to implement?
4. How much space (memory) it requires to solve the problem?
5. How much time it takes to solve the problem? Etc.

Performance analysis of an algorithm is performed by using the following measures...

1. Space required to complete the task of that algorithm (Space Complexity). It includes program space and data
space
2. Time required to complete the task of that algorithm (Time Complexity)

Space Complexity
What is Space complexity?

When we design an algorithm to solve a problem, it needs some computer memory to complete its execution. For
any algorithm, memory is required for the following purposes...
1. To store program instructions.
2. To store constant values.
3. To store variable values.
4. And for few other things like function calls, jumping statements etc,.

space complexity
The space complexity of an algorithm is the amount of memory it needs to run to completion. The space needed by
an algorithm is the sum of the following components:

1) Fixed space which is independent of the characteristics (e.g., number and size) of the inputs and outputs) this
part typically includes the instruction space (space for the code), space for the simple variables and fixed size
component variables, space for constants, and so on.
2) A variable part that consists of the space needed by component variables whose size is dependent on the
particular problem instance being solved, the space needed by referenced variables (to the extent that this
depends on instance characteristics), and the recursion stack space.

The space requirement S(P) of any algorithm P may therefore be written as S(P) = c + SP(instance characteristics)
where c is a constant.
Instance characteristics are problem specific. Number of input and output variables and the size of each input and
output variable are the important factors for finding space complexity of an algorithm.
 Calculate the time complexity of the following algorithm:
-----------------------------------------------------------------------------------------------------
1 Algorithm abc(a, b, c)
2 {
3 return a + b + b*c + (a + b – c)/(a + b) + 9.0;
4 }
------------------------------------------------------------------------------------------------------
Algorithm_1 computes return a + b + b*c + (a + b – c)/(a + b) + 9.0;
Time complexity of Algorithm_1 = SP(instance characteristics) = 0. Because there are component variables,
referenced variables and recursion stack space.

Calculate the time complexity of the following algorithm:

-----------------------------------------------------------------------------------------------------
1 Algorithm sum(a, n)
2 {
3 s := 0.0;
4 for i := 1 to n do
5 s := s + a[i];
6 return s;
7 }
------------------------------------------------------------------------------------------------------
Algorithm_2 iterative function for finding sum of all the elements in the array, a.
Space needed for the integer variable n = 1 word
Space needed for the integer variable i = 1 word
Space needed for the float variable s = ≥1 word
Space needed for the array variable a = ≥ n words
Therefore space complexity of an algorithm = SSum(n) ≥ (n + 3)

Calculate the time complexity of the following algorithm:

--------------------------------------------------------------------------------------------------------------------
1 Algorithm RSum(a, n)
2 {
3 if(n ≤ 0) then
4 return 0.0;
5 else
6 return RSum(a, n – 1) + a[n];
7 }
--------------------------------------------------------------------------------------------------------------------
Algorithm_3 recursive function for finding sum of all the elements in the array, a.
The recursion stack space includes space for the formal parameters, the local variables, and the return address.
Assume that the return address requires only one word of memory. Each call to RSum requires at least three words
(including space for the values of n, the return address, and a pointer to a[]). Since the depth of recursion is n + 1,
the recursion stack space needed is ≥ 3(n + 1).
That is RSum(n) ≥ 3(n + 1).
Time Complexity
What is Time complexity?

Every algorithm requires some amount of computer time to execute its instruction to perform the task. This computer
time required is called time complexity. The time complexity of an algorithm can be defined as follows...
The time complexity of an algorithm is the total amount of time required by an algorithm to complete its
execution.
Generally, the running time of an algorithm depends upon the following...
1. Whether it is running on Single processor machine or Multi processor machine.
2. Whether it is a 32 bit machine or 64 bit machine.
3. Read and Write speed of the machine.
4. The amount of time required by an algorithm to perform Arithmetic operations, logical operations, return value
and assignment operations etc.,
5. Input data

To calculate the time complexity of an algorithm, we need to define a model machine. Let us assume a machine with
following configuration...
1. It is a Single processor machine
2. It is a 32 bit Operating System machine
3. It performs sequential execution
4. It requires 1 unit of time for Arithmetic and Logical operations
5. It requires 1 unit of time for Assignment and Return value
6. It requires 1 unit of time for Read and Write operations
Algorithm time complexity
The time complexity of an algorithm is the amount of computer time it needs to run to completion.
Performance evaluation of an algorithm is divided into 2 steps:
1) a priori estimates and
2) a posteriori testing
These two steps are called performance analysis and performance measurement respectively of the algorithm.

Time complexity of a program P is denoted by T(P) is the sum of the compile time and the run time. The compile
time does not depend on the instance characteristics and only run time is important. This run time is denoted by
tP(instance characteristics)
There are two ways for finding the number of steps needed by a program to solve a particular problem instance. In
the first method a global variable with initial value 0 is used. Each time a statement in the original program is
executed, count is incremented.

Calculate the time complexity of the following algorithm using step count first method
-----------------------------------------------------------------------------------------------------
1 Algorithm sum(a, n)
2 {
3 s := 0.0;
4 count := count + 1; //count is a global variable it is initialized to zero
5 for i := 1 to n do //
6 {
 7 count := count + 1; // For for
8 s := s + a[i]; count := count + 1; // For assignment
9 }
10 count := count + 1; // For the last time of for
11 count := count + 1; // For the return
12 return s;
13 }
------------------------------------------------------------------------------------------------------
Step table method is the second way of finding step count of the program. The number of steps per execution (s/e)
of the statement and the total number of times (i.e., frequency) each statement is executed. The s/e of a statement is
the amount by which the count changes as a result of the execution of that statement. By combining these two
quantities, the total contribution of each statement is obtained. By adding the contributions of all statements, the step
count for the entire algorithm is obtained.

Statement s/e frequency Total steps

1 Algorithm sum(a, n) 0 – 0
2 { 0 – 0
3 s := 0.0; 1 1 1
4 for i := 1 to n do 1 n+1 n+1
5 s := s + a[i]; 1 n n
6 return s; 1 1 1
7 } 0 – 0
Total 2n + 3
Table-1 Step table for algorithm_2

Statement s/e frequency Total steps

n=0 n>0 n=0 n>0
1 Algorithm RSum(a, n) 0 – – 0 0
2 {
3 if(n ≤ 0) then 1 1 1 1 1
4 return 0.0; 1 1 0 1 0
5 else
6 return RSum(a, n – 1) + a[n]; 1+x 0 1 0 1+x
7 } 0 – – 0 0
Total 2 2+x
x = tRSum(n – 1)
Table-2 Step table for Algorithm_3 recursive sum

Statement s/e frequency Total steps

1 Algorithm Add(a, b, c, m, n) 0 – 0
2 { 0 – 0
3 for i := 1 to m do 1 m+1 m +1
4 for j := 1 to n do 1 m(n + 1) mn + m
5 c[i, j] := a[i, j] + b[i, j]; 1 mn mn
6 } 0 – 0
Total 2mn + 2m + 1
Table_3 Step table for Matrix addition
 Asymptotic notations
There are three types of notations in which the complexity can be represented:
1) Big Oh notation(O):It is used to indicate the upper bound or the worst case time complexity of an algorithm.
2) Big Theta notation(θ): It is used to indicate the complexity for an algorithm averaged over all possible inputs.
3) Big Omega notation(Ω): It is used to indicate the lower bound or the best case time complexity of an algorithm.
4) O-Notation (Upper Bound)
a. Let f(n) and g(n) are two non-negative functions
b. The function f(n) = O(g(n)) if and only if there exists positive constants c and n0 such that f(n)≤c*g(n)
for all n , n ≥ n0.

O-Notation Examples
 If f(n)=3n+2 then prove that f(n) = O(n)
Let f(n) =3n+2, c=4, g(n) =n
if n=1 3n+2 ≤ 4n
3(1)+2 ≤ 4(1)
3+2 ≤ 4
5 ≤ 4 (F)
if n=2 3n+2≤4n
3(2)+2 ≤ 4(2)
8 ≤ 8 (T)
3n+2 ≤ 4n for all n ≥ 2
This is in the form of f(n) ≤ c*g(n) for all n ≥ n0, where c=4, n0 =2
Therefore, f(n) = O(n),
O-Notation Examples
 f(n)=10n2 + 4n + 2 then prove that f(n)=O(n2 )

Let f(n) = 10n2 + 4n + 2, c=11, g(n) =n2

if n=1 10n2 + 4n + 2 ≤ 11n2
16≤11 (F)
if n=2 10n2 + 4n + 2 ≤ 11n2
52 ≤ 44 (F)
if n=5 272 ≤ 275 (T)
10n2 + 4n + 2 ≤ 11n2 for all n ≥ 5
This is in the form of f(n) ≤ c*g(n) for all n ≥ n0, where c=11, n0 =5
Therefore, f(n) = O(n2),
Ω-Notation (Lower Bound)
 Let f(n) and g(n) are two non-negative functions
 The function f(n) = Ω(g(n)) if and only if there exists positive constants c and n0 such that f(n) ≥
c*g(n) for all n , n ≥ n0.

Ω-Notation Examples
 f(n)=3n+2 then prove that f(n) = Ω(g(n))
 Let f(n) =3n+2, c=3, g(n) =n
 if n=1 3n+2 ≥ 3n
 3(1)+2 ≥ 3(1)
 5 ≥ 3 (T)
 3n+2 ≥ 4n for all n ≥ 1
 This is in the form of f(n) ≥ c*g(n) for all n ≥ n0, where c=3, n0 =1
 Therefore, f(n) = Ω(n).

 f(n)= 6n3+4n+2 then Prove that f(n)=Ω(n3)

 f(n)=50n2+10n-5 then Prove that f(n)=Ω(n2)
 θ -Notation
 Let f(n) and g(n) be two non-negative functions.
 The function f(n) = θ(g(n)) if and only if there exists positive constants c1 , c2 and n0 such that
c1*g(n) ≤ f(n)≤c2* g(n) for all n, n≥n0

θ -Notation Examples
 f(n)=3n+2 then Prove that f(n) = θ(g(n))
Lower bound = 3n+2 ≥ 3n for all n ≥ 1
c1=3,g(n)=n,n0=1
Upper Bound = 3n+2 ≤ 4n for all n ≥ 2
c2=4, g(n)=n , n0=2
3(n) ≤ 3n+2 ≤ 4(n) for all n, n ≥ 2
This is in the form of c1*g(n) ≤ f(n) ≤ c2* g(n) for all n≥n0
Where c1=3, c2=4, g(n)=n, n0=2
Therefore f(n)= θ(n)

 f(n)=10n2+4n+2 then prove that f(n)= θ(n2)

Worst, Average and Best Case Analysis of Algorithms

Measurement of Complexity of an Algorithm

Based on the above three notations of Time Complexity there are three cases to analyze an algorithm:

1. Worst Case Analysis (Mostly used)

In the worst-case analysis, we calculate the upper bound on the running time of an algorithm. We must know the
case that causes a maximum number of operations to be executed. For Linear Search, the worst case happens when
the element to be searched (x) is not present in the array. When x is not present, the search() function compares it
with all the elements of arr[] one by one. Therefore, the worst-case time complexity of the linear search would be
O(n).
2. Best Case Analysis (Very Rarely used)
In the best-case analysis, we calculate the lower bound on the running time of an algorithm. We must know the
case that causes a minimum number of operations to be executed. In the linear search problem, the best case occurs
when x is present at the first location. The number of operations in the best case is constant (not dependent on n).
So time complexity in the best case would be Ω(1)

3. Average Case Analysis (Rarely used)

In average case analysis, we take all possible inputs and calculate the computing time for all of the inputs. Sum all
the calculated values and divide the sum by the total number of inputs. We must know (or predict) the distribution
of cases. For the linear search problem, let us assume that all cases are uniformly distributed (including the case of
x not being present in the array). So we sum all the cases and divide the sum by (n+1). Following is the value of
average-case time complexity.

Examples with their complexity analysis:

Linear search algorithm:
// C implementation of the approach
#include <stdio.h>
// Linearly search x in arr[].
// If x is present then return the index,
// otherwise return -1
int search(int arr[], int n, int x)
{
int i;
for (i = 0; i < n; i++) {

if (arr[i] == x)
return i;
}
return -1;
}

/* Driver's code*/
int main()
{
int arr[] = { 1, 10, 30, 15 };
int x = 30;
int n = sizeof(arr) / sizeof(arr[0]);
// Function call
printf("%d is present at index %d", x,
search(arr, n, x));
getchar();
return 0;
}
Output
30 is present at index 2
Time Complexity Analysis: (In Big-O notation)
Best Case: O(1), This will take place if the element to be searched is on the first index of the given list. So, the
number of comparisons, in this case, is 1.
 Average Case: O(n), This will take place if the element to be searched is on the middle index of the given
list.
 Worst Case: O(n), This will take place if:
 The element to be searched is on the last index
 The element to be searched is not present on the list

Advantages:
1. This technique allows developers to understand the performance of algorithms under different scenarios, which
can help in making informed decisions about which algorithm to use for a specific task.
2. Worst case analysis provides a guarantee on the upper bound of the running time of an algorithm, which can help
in designing reliable and efficient algorithms.
3. Average case analysis provides a more realistic estimate of the running time of an algorithm, which can be useful
in real-world scenarios.

Disadvantages:
1. This technique can be time-consuming and requires a good understanding of the algorithm being analyzed.
2. Worst case analysis does not provide any information about the typical running time of an algorithm, which can
be a disadvantage in real-world scenarios.
3. Average case analysis requires knowledge of the probability distribution of input data, which may not always be
available.

Amortize Analysis

• Amortized analysis is used for algorithms that have expensive operations that happen only rarely.
• Amortized complexity analysis is most commonly used with data structures that have state that persists
between operations.
• Amortized complexity is a concept in algorithm analysis that deals with the average time or space complexity
per operation in a sequence of operations, rather than focusing on the worst-case scenario for each individual
operation. It provides a more holistic view of the performance of an algorithm over a series of operations.
• The basic idea is that an expensive operation can alter the state so that the worst case cannot occur again for
a long time, thus amortizing its cost.
• This analysis is used when the occasional operation is very slow, but most of the operations which are
executing very frequently are faster. Data structures we need amortized analysis for Hash Tables, Disjoint
Sets etc.
• Amortized analysis is useful for designing efficient algorithms for data structures such as dynamic arrays,
priority queues, and disjoint-set data structures. It provides a guarantee that the average-case time complexity
of an operation is constant, even if some operations may be expensive.
• In the Hash-table, the most of the time the searching time complexity is O(1), but sometimes it executes O(n)
operations. When we want to search or insert an element in a hash table for most of the cases it is constant
time taking the task, but when a collision occurs, it needs O(n) times operations for collision resolution.

For example, suppose you have an algorithm that performs a series of insertions into a dynamic array. Each
insertion is fast, but if the array becomes full, the algorithm must perform a slower operation to resize the
 array and make room for the new insertion. The amortized cost of each insertion is the average cost per
insertion, taking into account the cost of the occasional slower operation to resize the array.
Insertion 1: Cost = 1 Credit = 1
Insertion 2: Cost = 1 Credit = 2
Insertion 3: Array is full, must resize Cost = 2 (insertion + resize) Credit = 0
Insertion 4: Cost = 1 Credit = 2
Insertion 5: Cost = 1 Credit = 3
…
Advantages of amortized analysis:
1. More accurate predictions: Amortized analysis provides a more accurate prediction of the average-case
complexity of an algorithm over a sequence of operations, rather than just the worst-case complexity of
individual operations.
2. Provides insight into algorithm behavior: By analyzing the amortized cost of an algorithm, we can gain
insight into how it behaves over a longer period of time and how it handles different types of inputs.

Methods in Amortized analysis

1.Aggregate Method
2.Accounting method
3.Potential method

Dynamic Array-Based Stack:

Assume we have a stack implemented using a dynamic array. The array has an initial capacity, and when the
number of elements reaches the capacity, a resizing operation is triggered (e.g., doubling the capacity).

Basic Stack Operations:

Push Operation: Adds an element to the top of the stack.

Pop Operation: Removes the element from the top of the stack.

• Accounting Method Example:

• Let's perform a sequence of push and pop operations on the stack:
• Push(1): The array becomes [1], and no resizing is needed.
• Push(2): The array becomes [1, 2], and no resizing is needed.
• Pop(): The array becomes [1], and no resizing is needed.
• Push(3): The array becomes [1, 3], and no resizing is needed.
• Push(4): Resizing occurs, and the array becomes [1, 3, 4, 0] (assuming doubling the capacity)
• Push(5): The array becomes [1, 3, 4, 0, 5], and no resizing is needed.
• Pop(): The array becomes [1, 3, 4, 0], and no resizing is needed.
• Pop(): The array becomes [1, 3, 4], and no resizing is needed.
• Pop(): Resizing occurs, and the array becomes [1, 3] (assuming halving the capacity).
• Amortized Analysis:
• Let's use the accounting method to analyze the amortized cost per operation.
• Assign a charge of 2 for each push operation.
• Assign a charge of 0 for each pop operation.
• Actual cost of push is 1 (copying to the new array during resizing).
 • Store 1 unit of credit for each push operation to be used in the future i.e resize
• Push(1): Charge = 2, Credit = 1 (remaining).
• Push(2): Charge = 2, Credit = 0 (remaining).
• Pop(): Charge = 0 (no charge for pop), Credit = 1 (remaining).
• Push(3): Charge = 2, Credit = 0 (remaining).
• Push(4): Charge = 2, Credit = 0 (remaining). Resizing occurs, but it's covered by the accumulated
credit.
• Push(5): Charge = 2, Credit = 1 (remaining).
• Pop(): Charge = 0, Credit = 1 (remaining).
• Pop(): Charge = 0, Credit = 2 (remaining).
• Pop(): Charge = 0, Credit = 3 (remaining). Resizing occurs, but it's covered by the accumulated credit.
• Amortized Cost Calculation:
• The amortized cost per operation is the actual cost plus the change in credit.
• Amortized Cost = Actual Cost + (Credit Change)
• Amortized Cost = 1 + (Credit Change) for push operations.
• Amortized Cost = 0 + (Credit Change) for pop operations.
In this example, the amortized cost per operation is O(1), ensuring a constant average cost for both push and pop
operations despite occasional resizing

Aggregate Method

• Let's assume that resizing (copying elements to a new array) takes O(n) time, where n is the number of
elements in the stack.
• Sequence of Operations:
• Push(1)
• Push(2)
• Pop()
• Push(3)
• Push(4)
• Push(5)
• Pop()
• Pop()
• Pop()

• Aggregate Analysis:
• Now, let's analyze the aggregate cost of the sequence:
• Push(1): No resizing needed. Cost = O(1).
• Push(2): No resizing needed. Cost = O(1).
• Pop(): No resizing needed. Cost = O(1).
• Push(3): No resizing needed. Cost = O(1).
• Push(4): Resizing occurs (O(n)). Cost = O(n).
• Push(5): No resizing needed. Cost = O(1).
• Pop(): No resizing needed. Cost = O(1).
• Pop(): No resizing needed. Cost = O(1).
• Pop(): Resizing occurs (O(n)). Cost = O(n).
• The total cost is O(1 + 1 + 1 + 1 + n + 1 + 1 + 1 + n) = O(4 + 2n). The average cost per operation is O((4 +
2n) / 9), which simplifies to O(n).
In the aggregate method, we calculate the total cost of a sequence of operations and then find the average cost per
operation. In this example, the average cost per operation is O(n), indicating that, on average, each operation has a
linear cost.

• The potential method is another approach used in amortized analysis to analyze the performance of
algorithms over a sequence of operations. It involves assigning a "potential" to the data structure after each
operation, and the amortized cost is the actual cost plus the change in potential. Let's illustrate the potential
method with a simple example using a dynamic array-based stack.
Potential Function:
Define the potential function Φ to be the difference between the current capacity of the array and the number
of elements in it: Φ = Capacity - Size.

Sequence of Operations:

Push(1)
Push(2)
Pop()
Push(3)
Push(4)
Push(5)
Pop()
Pop()
Pop()

Potential Method Analysis:

Now, let's analyze the amortized cost using the potential method:
Push(1):
• Actual cost: O(1)
• Change in potential (ΔΦ): 1 (capacity increases from 1 to 2)
• Amortized cost = O(1 + 1) = O(2)
Push(2):
• Actual cost: O(1)
• ΔΦ: 1 (capacity increases from 2 to 4)
• Amortized cost = O(1 + 1) = O(2)
Pop():
• Actual cost: O(1)
• ΔΦ: 1 (capacity decreases from 4 to 2)
• Amortized cost = O(1 + 1) = O(2)
Push(3):
• Actual cost: O(1)
• ΔΦ: 1 (capacity increases from 2 to 4)
• Amortized cost = O(1 + 1) = O(2)
Push(4):
• Actual cost: O(1)
• ΔΦ: 4 (capacity increases from 4 to 8)
• Amortized cost = O(1 + 4) = O(5)
Push(5):
 • Actual cost: O(1)
• ΔΦ: 4 (capacity increases from 8 to 16)
• Amortized cost = O(1 + 4) = O(5)
Pop():
• Actual cost: O(1)
• ΔΦ: 8 (capacity decreases from 16 to 8)
• Amortized cost = O(1 + 8) = O(9)
Pop():
• Actual cost: O(1)
• ΔΦ: 4 (capacity decreases from 8 to 4)
• Amortized cost = O(1 + 4) = O(5)
Pop():
• Actual cost: O(1)
• ΔΦ: 2 (capacity decreases from 4 to 2)
Amortized cost = O(1 + 2) = O(3)

• In the potential method, the amortized cost per operation is the actual cost plus the change in potential. The
potential function helps in understanding how the resizing operations are accounted for in the overall
amortized cost. In this example, the amortized cost per operation is O(2), O(5), O(9), O(5), O(3) for the
respective operations.

SOLVING RECURRENCES
 3 Methods
 using substitution
 using recurrence trees
 using master theorem
 Method of substitution
 Method expanding recurrence to see a pattern.
 Expand (iterate) the recurrence and express it as a summation of terms dependent only on n and the
initial conditions
 Key focus on 2 parameters
 the number of times the recurrence needs to be iterated to reach the boundary condition.
 the sum of terms arising from each level of the iteration process
 Techniques for evaluating summations can then be used to provide bounds on solution
Example 1 : Solve the recurrence relation

1 n=0

T(n) = T(n-1)+1 n>0

T(n) = T(n-1)+1
Substitute T(n-1) T(n) = T(n-1) + 1
T(n) = [T(n-2) + 1] + 1 T(n - 1) = T(n-2) + 1
T(n) = T(n-2)+2
T(n) = [T(n-3) + 1] + 2 T(n - 2) = T(n-3) + 1
T(n) = T(n-3) + 3
Continue for K times
T(n) = T(n-k) + k
Assume n-k = 0
n=k
T(n) = T(n-n) + n
= T(0) + n
= 1+n
Therefore Time complexity = ɵ(n)

Example 2 : Solve the Recurrence relation

1 n=0

T(n-1) + n n>0

T(n) = T(n-1) + n …………. (1)

T(n) = [ T(n-2) + n-1 ] + n T(n) = T(n-1) + 1
T(n) = T(n-2) + (n-1) + n…….(2) T(n-1) = T(n-2) + n - 1
T(n) = [ T(n-3) + n-2] + (n-1) + n T(n-2) = T(n-3) + n - 1
T(n) = T(n - 3) + (n – 2) + (n – 1) + n………..(3)
T(n) = T(n - k) + (n - (k - 1)) + (n - (k - 2)) + …………… + (n - 1) + n…………(4)
Continue for k times
Assume n - k = 0
n=k
T(n) = T(n - n) + (n - n + 1) + (n - n + 2) + ………. +(n-1) + n
T(n) = T(0) + 1 + 2 + 3 + ………..+ (n-1) + n
T(n) = 1 + n (n + 1)
2
Therefore Time Complexity = ɵ(n2)
Example 3: Solve the Recurrence relation

1 n=0

T(n – 1) + log n n>0

T(n) = T(n - 1) + log n……………………(1)

T(n) = [T(n - 2) + log(n - 1)] + log n
T(n) = T(n - 2) + log(n - 1) + log n ………….(2)
T(n) = [T(n - 3) + log(n - 2)] + log(n - 1) + log n
T(n) = T(n - 3) + log(n - 2) + log(n -1) + log n………….(3)
T(n) = T(n - k) + log 1 + log 2 + log 3 + ………….+ log(n - 1) + log n
Assume n - k = 0 , n = k
T(n) = T(0) + log n!
= 1 + log n! = O(n log n)

Example 4: Solve the Recurrence relation

1 n=0
2T(n – 1) + 1 n>0

T(n) = 2T(n - 1) + 1……………………(1)

T(n) = 2[2T(n - 2) + 1)] + 1
T(n) = 22T(n - 2) + 2 + 1 ………….(2)
T(n) = 22[2T(n - 3) + 1] + 2 + 1
T(n) = 23T(n - 3) + 22 + 2 + 1………….(3)
T(n) = 2kT(n - k) + 2k - 1 + 2k - 2 + …………….22 + 2 + 1 …………….(4)
Assume n - k = 0 , n = k
T(n) = 2n T(0) + 1 + 2 + 22 + …………..+ 2n - 1 = 2n * 1 + 2n - 1
= 2n + 2n - 1 = 2n+1 - 1 = O(2n)

RECURRENCE TREE
 A convenient way to visualize what happens when a recursion is iterated
 It is good for generating guesses for the substitution method.
 We may describe the execution of a recursive program on a given input by a rooted tree, which is known as
recurrence tree
 The steps involved in building the Recurrence Tree
 Determine height of tree and the size(sum) of each level as a function of input size
 Add the sizes of each level and Multiply by the height of the tree
  Recurrence tree Example

MASTER THEOREM

T(1)= d
T(n)=aT(n/b)+cn
Has solution
T(n) = O(n) if a<b
T(n) = O(nlogn) If a=b
T(n) = O(n logba ) if a>b

Example

T(1)= 0
T(n)=4T(n/2)+cn for n=2
Has solution
a = 4; b = 2; a>b
T(n) = O(n logba ) if a>b
T(n) = O(n log24 ) = O(n2)