Chapter 1: Introduction to analysis of algorithm

Table of Contents

 Chapter Learning Outcomes

 Introduction
 Characteristics of algorithm
 Asymptotic analysis of complexity bounds
 Performance measurements of algorithm
 Time and space trade-offs
 Substitution method
 Recursion tree methods
 Master’s theorem
 Summary

Introduction to analysis of algorithm 1

  Chapter Learning Outcomes
At the end of this module, you are expected to:

 Define efficiency of algorithms.

 Differentiate space and time complexity.

 Explain Algorithm Design.

Introduction to analysis of algorithm 2

  Introduction
The Introduction to Analysis of Algorithms is a fundamental topic in computer
science and software engineering. It involves studying the performance and
efficiency of algorithms to solve computational problems. Below are the key aspects
typically covered in an introductory explanation:

 What is an Algorithm?

An algorithm is a step-by-step procedure or set of rules to solve a specific problem or

perform a computation. Algorithms can be expressed in natural language,
pseudocode, or programming languages.

 Why Analyze Algorithms?

Analysis of algorithms is essential to:

 Measure efficiency: Understand how fast an algorithm runs and how much
memory it uses.
 Compare algorithms: Decide which algorithm is better suited for a problem.
 Optimize performance: Improve existing algorithms or design new ones.
 Predict scalability: Assess how an algorithm behaves as the input size
grows.

 Types of Analysis

Algorithms are analyzed for both time complexity and space complexity:

 Time Complexity: The amount of time an algorithm takes to run, often

expressed as a function of input size (n).
 Space Complexity: The amount of memory an algorithm uses during its
execution.

Introduction to analysis of algorithm 3

  Techniques for Analysis

1. Worst-case analysis: Measures the maximum time/space required for any

input of size n.
2. Best-case analysis: Measures the minimum time/space required for any
input of size n.
3. Average-case analysis: Measures the expected time/space required on
average.
4. Amortized analysis: Measures the average performance over a sequence of
operations.

 Asymptotic Notations

Asymptotic notations are used to express the growth of an algorithm's time or space
requirements:

 Big-O Notation (O): Upper bound on the running time (worst-case

performance).
 Omega Notation (Ω): Lower bound on the running time (best-case
performance).
 Theta Notation (Θ): Tight bound that describes both upper and lower bounds.

Example: Analyzing a Simple Algorithm

For instance, consider a function that calculates the sum of the first n integers:

 Time Complexity: The for loop runs n times -> O(n).

 Space Complexity: The space used is constant, independent of n -> O(1).
 Applications

Introduction to analysis of algorithm 4

  Finding the most efficient algorithm for tasks like sorting, searching, and data
manipulation.
 Designing scalable systems in areas like databases, artificial intelligence, and
web development.

 Characteristics of algorithm

 The characteristics of an algorithm define what makes a process qualify as

an algorithm and ensure that it is effective, reliable, and efficient in solving
problems. These characteristics are crucial for ensuring that an algorithm
works as intended, irrespective of the type of problem it addresses.
 Here is a detailed explanation of each characteristic:

1. Input

 Definition: An algorithm must have well-defined inputs. These are the values
or data that are provided to the algorithm before it starts its execution.
 Requirements:
o The number and type of inputs should be specified.
o Inputs can be zero, one, or more depending on the problem.
 Example:
o A sorting algorithm requires an array or list of numbers as input.
o A factorial algorithm may take a single integer as input.

2. Output

 Definition: An algorithm must produce at least one output, which is the result
of processing the input through its steps.
 Requirements:
o The output should be well-defined and related to the input.
o The output must meet the problem's requirements.
 Example:
o A sorting algorithm outputs the same numbers in a sorted order.

Introduction to analysis of algorithm 5

 o A search algorithm returns the position of an element in a list or
indicates that the element is not present.

3. Definiteness (Unambiguous Instructions)

 Definition: Every step of the algorithm must be clearly and precisely defined,
with no ambiguity in what action is to be performed.
 Requirements:
o Each step must have only one interpretation.
o The instructions must be executable and understandable.
 Importance: Ambiguous instructions can lead to errors or incorrect outputs.
 Example:
o "Add 2 to the current value" is definite.
o "Do something to get a result" is ambiguous and not acceptable.

4. Finiteness

 Definition: An algorithm must always terminate after a finite number of steps.

It cannot run indefinitely.
 Requirements:
o The algorithm must have a stopping condition.
o Infinite loops or unending processes are not allowed.
 Importance: If an algorithm does not terminate, it cannot provide a useful
result.
 Example:
o A loop that processes each element in a list will eventually terminate
after reaching the last element.
o An algorithm that checks divisors of a number and stops after finding
all divisors is finite.

5. Effectiveness

 Definition: All operations in the algorithm must be simple enough to be

performed manually with paper and pencil or implemented programmatically.
 Requirements:

Introduction to analysis of algorithm 6

 o The steps must be basic and executable within a finite time.
o The operations should not rely on undefined or impractical processes.
 Importance: Ensures that the algorithm is implementable in the real world.
 Example:
o Calculating the sum of two numbers is effective.
o Performing an operation that requires infinite precision or undefined
rules is not effective.

6. Correctness

 Definition: An algorithm must correctly solve the problem it is designed for

and produce the expected output for all valid inputs.
 Requirements:
o It should handle edge cases and produce accurate results for all inputs.
o The correctness can be verified through testing or mathematical proofs.
 Importance: Ensures reliability and usability.
 Example:
o An algorithm for addition should always return the correct sum of two
numbers, regardless of their magnitude.

7. Generality

 Definition: An algorithm should be general enough to solve all problems of a

specific type, not just a single instance.
 Requirements:
o It should work for a broad range of inputs.
o Special cases should be included but not limit the algorithm's scope.
 Importance: Ensures reusability and adaptability.
 Example:
o A sorting algorithm should work for lists of any size or order, not just for
a fixed number of elements.

Introduction to analysis of algorithm 7

8. Feasibility

 Definition: An algorithm should be practical to implement and execute with

available resources, such as time, memory, and computational power.
 Requirements:
o The time and space complexity of the algorithm should be reasonable
for the problem size.
o Algorithms that require excessive resources may be impractical.
 Importance: Ensures that the algorithm can be executed in real-world
scenarios.
 Example:
o Algorithms with linear time complexity (O(n) are generally more
feasible than those with exponential complexity (O(2n) for large inputs.

9. Independence

 Definition: An algorithm should be independent of any specific programming

language, operating system, or hardware. It should focus on the logic of the
problem rather than the implementation details.
 Requirements:
o It can be expressed in pseudocode, flowcharts, or any other abstract
representation.
o It should be easily translatable into different programming languages.
 Importance: Ensures portability and flexibility.
 Example:
o A pseudocode representation of a sorting algorithm can be
implemented in Python, Java, or C++ without altering its logic.

10. Step-by-Step Execution

 Definition: An algorithm progresses through a series of steps, each building

upon the results of the previous one.
 Requirements:
o Each step should logically lead to the next.
o The sequence must be structured and follow a clear flow.

Introduction to analysis of algorithm 8

  Importance: Prevents confusion and makes the process traceable.
 Example:
o In a binary search algorithm, each step narrows the search range by
dividing it into halves until the target is found.

Example: Algorithm Characteristics in Practice

Let’s take an example of an algorithm to find the factorial of a number n.

Pseudocode:

1. Input n.
2. Initialize result = 1.
3. For i = 1 to n:
o Multiply result by i.
4. Output result.

Analysis:

 Input: n, an integer.
 Output: Factorial of n.
 Definiteness: Each step is precise (e.g., initialize, multiply).
 Finiteness: The loop runs n times and stops.
 Effectiveness: The operations (multiplication, initialization) are simple.
 Correctness: The algorithm correctly computes the factorial.
 Generality: It works for any non-negative integer n.

 Asymptotic analysis of complexity bounds

Asymptotic Analysis of Complexity Bounds is a mathematical approach to

analyze the performance of algorithms as the input size grows very large. This helps
in evaluating the efficiency of an algorithm in terms of time complexity (execution
time) and space complexity (memory usage).

Why Asymptotic Analysis?

Introduction to analysis of algorithm 9

  Scalability: Determines how an algorithm performs as the size of the input (n)
increases.
 Independence: Focuses on growth trends rather than hardware or
implementation specifics (e.g., programming language).
 Comparison: Allows comparison of algorithms by their efficiency in handling
large inputs.

Key Concepts in Asymptotic Analysis

1. Input Size (n):

o Represents the size of the input data. For example:
 Number of elements in a list for sorting algorithms.
 Size of a graph for graph traversal algorithms.

2. Growth Rate:
o Describes how the resource consumption (time or space) of an
algorithm increases as n grows.

3. Asymptotic Notations:
o These are mathematical tools to describe the upper, lower, and tight
bounds of an algorithm's complexity.

Asymptotic Notations

Asymptotic notations describe how an algorithm's resource usage grows with input
size, abstracting away constants and lower-order terms.

1. Big-O Notation (O)

 Definition: Represents the upper bound of an algorithm's growth rate. It

describes the worst-case performance.
 Mathematical Representation:

f(n)∈O(g(n)) if there exist constants c>0 and n0 such that f(n) ≤c⋅g(n), ∀n≥n0.

Interpretation:

Introduction to analysis of algorithm 10

 o g(n) dominates f(n) for large n.
o Used to describe the maximum time or space required.
 Example:
o If f(n)=3n2+5n+2, then f(n)∈O(n2) as n2 dominates for large n.

2. Omega Notation (Ω)

 Definition: Represents the lower bound of an algorithm's growth rate. It

describes the best-case performance.
 Mathematical Representation: f(n) ∈ Ω(g(n)) if there exist constants c >
0 and n0 such that f(n) ≥ c⋅g(n), ∀ n ≥ n0.
 Interpretation:
o g(n) provides a minimum bound for f(n) as n grows.
o Ensures that the algorithm performs at least as well as g(n) in the best-
case scenario.
 Example:
o If f(n)=3n2+5n+2, then f(n) ∈ Ω(n2).

3. Theta Notation (Θ)

 Definition: Represents the tight bound of an algorithm's growth rate. It

describes the performance in both the best-case and worst-case scenarios.
 Mathematical Representation: f(n) ∈ Θ(g(n))
if there exist constants c1,c2>0 and n0 such that c1⋅g(n) ≤f(n) ≤c2⋅g(n), ∀ n ≥
n0.
 Interpretation:
o g(n) tightly bounds f(n) from above and below.
o Ensures that g(n)g(n)g(n) represents the algorithm's actual growth rate
for large n.
 Example:
o If f(n)=3n2+5n+2, then f(n) ∈ Θ(n2).

Introduction to analysis of algorithm 11

  Performance measurements of algorithm

Performance measurement of algorithms involves evaluating how well an

algorithm performs in terms of resource usage, such as time and memory, for a
given problem. Understanding these metrics helps in choosing the most efficient
algorithm for a particular task. Performance is typically measured in two primary
ways:

1. Time Complexity: The time taken by an algorithm to complete its task.

2. Space Complexity: The memory or storage required by the algorithm.

1. Time Complexity

Time complexity measures the amount of time an algorithm takes to complete its
task as a function of the input size (n). It evaluates the efficiency of the algorithm with
respect to execution time.

Key Concepts in Time Complexity

 Execution Time: The actual time taken to execute an algorithm, which

depends on hardware, compiler, and system load.
 Growth Rate: Focuses on how the running time of an algorithm increases
with the size of the input.

Methods to Measure Time Complexity

1. Counting Basic Operations:

o Basic operations include assignments, comparisons, and arithmetic
operations.
o The time complexity is analyzed by counting the number of these
operations.
o Example:
 For a loop that runs n times, the number of operations is
proportional to n, so the time complexity is O(n).

Introduction to analysis of algorithm 12

 2. Experimental Measurement:
o Measure execution time using a timer in code.
o Example (in Python):

Mathematical Analysis:

 Analyzes the algorithm theoretically to derive a formula for time complexity.

 Example:
o For a nested loop:

2. Space Complexity

Space complexity measures the amount of memory required by an algorithm to

complete its task. This includes:

 Fixed Part: Memory required for constants and variables.

 Variable Part: Memory required for dynamic data structures, recursion, and
intermediate computations.

Key Components of Space Complexity

1. Input Space:

Introduction to analysis of algorithm 13

 o Memory required to store input data.
2. Auxiliary Space:
o Additional memory used for temporary variables, data structures, or
recursion stacks.
3. Program Code Space:
o Memory used by the compiled algorithm itself.

Example of Space Complexity

 Iterative algorithms often have a lower space complexity than recursive ones
because they don’t require a call stack.
o Iterative factorial calculation: O(1) auxiliary space.
o Recursive factorial calculation: O(n) auxiliary space (for the recursion
stack).

Performance Metrics

Performance measurements of algorithms can be evaluated using the following

metrics:

1. Execution Time

 Measures the time taken for the algorithm to execute.

 Depends on:
o Input size.
o Hardware configuration.
o Implementation details.

2. Memory Usage

 Evaluates the amount of memory consumed during execution.

 Includes both fixed and dynamic memory usage.

3. Input Size

 The size of the input significantly affects both time and space complexity.
 Algorithms often perform differently on small vs. large inputs.

Introduction to analysis of algorithm 14

4. Scalability

 How well the algorithm adapts to increasing input size.

 An algorithm with lower time and space complexity generally scales better.

5. Number of Basic Operations

 The total number of elementary operations performed.

 Used in theoretical analysis to determine complexity.

Practical Example: Performance Analysis

Problem: Compare bubble sort and merge sort for sorting an array.

1. Bubble Sort:
o Time Complexity:
 Worst-case: O(n2).
 Best-case: O(n) (if already sorted).
o Space Complexity: O(1).

2. Merge Sort:
o Time Complexity:
 Worst-case: O(nlogn).
 Best-case: O(nlogn).
o Space Complexity: O(n) (due to auxiliary arrays).

Observation:

 For large inputs, merge sort performs significantly better due to its lower time
complexity.
 Bubble sort is simpler but impractical for large datasets.

 Time and space trade-offs

Time and Space Trade-offs refer to the balance between the time complexity
(execution speed) and space complexity (memory usage) of an algorithm. Improving

Introduction to analysis of algorithm 15

one often comes at the expense of the other, and this trade-off helps in designing
algorithms that are efficient for specific requirements or constraints.

Understanding the Trade-offs

1. Time-Optimized Algorithms:
o Focus on reducing execution time.
o Often require additional memory to store intermediate results or
precomputed data.
o Example: Using a hash table for fast lookups increases space usage
but reduces the time complexity from O(n) (linear search) to O(1)
(constant time).

2. Space-Optimized Algorithms:
o Focus on minimizing memory usage.
o Often require additional computations, increasing execution time.
o Example: Calculating Fibonacci numbers iteratively requires O(1)
space, but recursive methods with memoization require more memory
but reduce redundant calculations.

Examples of Time and Space Trade-offs

1. Dynamic Programming vs. Recursion

 Dynamic Programming:
o Uses additional memory to store results of subproblems (memoization).
o Reduces time complexity by avoiding redundant calculations.
o Example: Fibonacci sequence with memoization has O(n) time
complexity but O(n) space complexity.
 Recursion:
o Minimizes space by not storing subproblem results.
o Increases time complexity due to redundant calculations.
o Example: Plain recursive Fibonacci sequence has O(2n) time
complexity and O(n) space complexity (for the recursion stack).

Introduction to analysis of algorithm 16

2. Sorting Algorithms

 Merge Sort:
o Time Complexity: O(nlogn).
o Space Complexity: O(n) (requires auxiliary arrays).
 In-Place Sorting (e.g., Quick Sort):
o Time Complexity: O(nlogn) on average.
o Space Complexity: O(logn) (due to recursion stack).

3. Hashing vs. Brute Force

 Hashing:
o Uses extra memory for hash tables.
o Achieves constant-time lookups O(1).
 Brute Force:
o Requires no additional memory.
o Takes O(n) time for lookups in unsorted data.

4. Data Compression

 Compressed data uses less space but requires more time to decompress
when accessed.
 Example: Compressed image files save storage but increase the time
required for decompression during viewing.

 Substitution method

The Substitution Method is a mathematical approach used to solve

recurrence relations and analyze the time complexity of recursive algorithms.
It involves hypothesizing a solution to a recurrence relation and then proving
that the hypothesis is correct using mathematical induction.

Steps in the Substitution Method

1. Hypothesize a Solution:

Introduction to analysis of algorithm 17

 o Guess a function T(n) that satisfies the recurrence relation.

2. Prove by Induction:
o Use mathematical induction to prove that the hypothesis is valid.
o Show that the solution holds for a base case (small values of nnn).
o Assume that the solution holds for some k (T(k)).
o Prove that it holds for k+1 (T(k+1)).

3. Validate Constants:
o Adjust constants or coefficients in the hypothesis, if necessary, to make
the proof valid.

Advantages of the Substitution Method

1. Intuitive: Provides a straightforward way to solve recurrence relations.

2. General: Can be applied to a wide variety of recurrence relations.
3. Precise: Helps derive tight bounds for time complexity.

Challenges of the Substitution Method

1. Guessing the Solution:

o Requires intuition or prior knowledge to hypothesize the correct
solution.
o If the guess is incorrect, the proof fails.

2. Complex Recurrences:
o For highly complex recurrence relations, guessing the solution can be
difficult.

3. Manual Adjustments:
o May require manual fine-tuning of constants to validate the proof.

 Recursion tree methods

Introduction to analysis of algorithm 18

 The Recursion Tree Method is a technique used to solve recurrence
relations that arise in the analysis of recursive algorithms. It provides a visual
and intuitive way to represent the recursive calls of an algorithm and helps in
determining its time complexity.

What is a Recursion Tree?

A recursion tree is a diagrammatic representation of the process of breaking down a

problem into smaller subproblems in recursive algorithms. Each node in the tree
represents a subproblem, and the edges represent the recursive calls.

The goal is to compute the total work done at all levels of the recursion and derive
the algorithm's overall complexity.

Steps for Solving Recurrences Using the Recursion Tree Method

1. Write the Recurrence Relation: Start with the recurrence relation that
represents the algorithm's time complexity.
2. Expand the Recurrence: Break down the recurrence relation level by level
into its subproblems until reaching the base case.
3. Calculate Work per Level: Sum the work done at each level of the tree
(combine all nodes at the same depth).
4. Determine the Tree's Depth: Find the number of levels in the recursion tree,
which corresponds to the height of the tree.
5. Sum the Work Across All Levels: Add up the work done at all levels to
compute the total time complexity.
6. Identify Dominant Term: Use asymptotic analysis to determine the dominant
term in the final expression, which represents the overall time complexity.

Advantages of the Recursion Tree Method

1. Intuitive:
o Provides a clear, visual representation of recursive calls and their
associated costs.

2. Applicable to Complex Recurrences:

Introduction to analysis of algorithm 19

 o Can handle recurrences with varying work at different levels.

3. Easier to Interpret:
o The level-wise breakdown helps in understanding the contributions of
different parts of the recursion.

 Master’s theorem
Master’s Theorem is a powerful tool for analyzing the time complexity of divide-
and-conquer algorithms, particularly those that can be expressed using a specific
form of recurrence relation. It provides a straightforward way to derive asymptotic
bounds for such recurrences without requiring detailed expansions or
mathematical induction.

Intuition Behind Master’s Theorem

1. Dividing the Problem:

o The problem of size n is divided into a smaller subproblems, each of
size n/b.
2. Combining the Results:
o Combining the results of the subproblems incurs an additional cost,
given by f(n).
3. Balancing Work Across Levels:
o Master’s Theorem compares the cost of f(n) with the work done in the
recursive calls to determine the dominant contributor to the total
complexity.

Conditions for Master’s Theorem

1. The recurrence must be in the form T(n)=aT(n/b)+f(n).

2. a≥ 1 b>1, and f(n) must be asymptotically positive.
3. f(n) should be polynomially bounded to npp+ϵ, where p=logba.

If any of these conditions are not met, Master’s Theorem does not apply.

Advantages of Master’s Theorem

Introduction to analysis of algorithm 20

 1. Simplifies Analysis:
o Provides a quick and efficient way to determine time complexity.
2. Wide Applicability:
o Applies to many common divide-and-conquer algorithms.
3. Eliminates Detailed Expansion:
o No need to manually expand and sum terms.

Limitations of Master’s Theorem

1. Restricted Form:
o Only applies to recurrences of the specific form T(n)=aT(n/b)+f(n).
2. Non-Polynomial Functions:
o Does not handle f(n) with logarithmic or exponential growth effectively.
3. Regularity Conditions:
o Case 3 requires additional conditions for f(n) to dominate.

 Summary
 The study of algorithms is called algorithmics.
 Algorithmics is more than a branch of computer science. It is the core of
computer science, and, in all fairness, can be said to be relevant to most of
science, business, and technology.
 An Algorithm is a step-by-step process of solving a problem in a finite number
of steps.
 Algorithms must have a unique name
 Algorithms should have explicitly defined a set of inputs and outputs
 Algorithms are well-ordered with unambiguous operations
 Algorithms halt in a finite amount of time. Algorithms should not run for infinity,
i.e., an algorithm must end at some point
 An algorithm design technique (or “strategy” or “paradigm”) is a general
approach to solving problems algorithmically that is applicable to a variety of
problems from different areas of computing.
 There are two kinds of efficiency: Time efficiency and space efficiency.

Introduction to analysis of algorithm 21

  Time efficiency, also called time complexity, indicates how fast an algorithm in
question runs.
 Space efficiency, also called space complexity, refers to the number of
memory units required by the algorithm in addition to the space needed for its
input and output.
 An asymptotic notation is used to compute the complexity of an algorithm in
terms of time and
 space. It is mentioned using the following terms:
 O(g(n)): Set of functions that grow no faster than g(n)
 Ω(g(n)): Set of functions that grow at least as fast as g(n)
 Θ(g(n)): Set of functions that grow at the same rate as g(n)

Introduction to analysis of algorithm 22