Introduction

Algorithms & Complexity - Introduction

Nicolas Stroppa - nstroppa@computing.dcu.ie

CA313@Dublin City University. 2006-2007.

November 6, 2006
Introduction

Outline of the lectures

I Introduction to Complexity Theory

I How to compute the formal complexity of a program?
I The class P. Polynomial-time reducibility
I The class NP. Cook’s theorem
I Time and Space complexity classes
I Relations between time and space
I Heuristics for NP complete problems:
Genetic algorithms, Tabu search, Beam search
I Heuristics for NP complete problems:
Ant algorithms, Simulated annealing
Introduction

Time complexity: an informal definition

Definition (Time complexity)

The (time) complexity of a program (for a given input) is the
number of elementary instructions that this program executes.
This number is computed with respect to the size n of the input
data.

Note
We thus make the assumption that each elementary instruction
takes the same amount of time (because it is probably one
processor instruction, which is true for example for addition and
multiplication between integers).
Introduction

Space complexity: an informal definition

Definition (Space complexity)

The (space) complexity of a program (for a given input) is the
number of elementary objects that this program needs to store
during its execution.
This number is computed with respect to the size n of the input
data.

Note
We thus make the assumption that each elementary object needs
the same amount of space (which is almost true for the basic
types: one machine word).
Introduction

Complexity: why bother?

Estimation/Prediction
When you write/run a program, you need to be able to predict its
needs, its requirements.

Usual requirements
I execution time
I memory space

Quantities to estimate
I execution time ⇒ time complexity
I memory space ⇒ space complexity
Introduction

Complexity: why bother?

It is pointless to run a program that requires:

I 6TeraB of RAM on a desktop machine;
I 10,000 years to run. . .

You do not want to wait for an hour:

I for the result of your query on Google;
I when you are checking your bank account online;
I when you are opening a picture file on Photoshop;
I etc.

⇒ It is important to write efficient algorithms!

Introduction

Complexity: why bother?

It is very important that you know what you can and cannot
do to improve (i.e. optimize) your algorithms:
I It is not wise to run an algorithm with O(n2 ) complexity if
there is an algorithm with O(ln(n)) doing the same job. . .
I Do not try to optimize an algorithm for a problem if we
formally know there is no efficient (i.e. polynomial)
algorithm. . .
I ⇒ You will need to know how to compute and estimate the
complexity of algorithms if you want to be able to say that.
Introduction

“Speed” of Programming Languages

You will read things like “C is faster than Java”, “Java is faster
than Python”, etc.

More important than the programming language are:

I (i) the formal complexity of the algorithm;
I (ii) the programmer (i.e. simplicity, quality of the program,
etc.)

Note
An algorithm with complexity O(n × ln(n)) in Python will probably
be faster than an algorithm with complexity O(n2 ) in C. . .
Introduction

A formal definition
Definition (Time complexity)
We will note T (A, n) the number of elementary instructions that
an algorithm A executes when the size of the input data is n, or
T (A) when unambiguous.
Example
// note: x is an unsorted array
int findMin(int[] x) {
int k = 0; int n = x.length;
for (int i = 1; i < n; i++) {
if (x[i] < x[k]) {
k = i;
}
}
return k;
}
Introduction

Computing the complexity

Sequence of instructions
The number of elementary instructions executed in a sequence of
instructions is the sum of the number of elementary instructions
executed in each instruction:

T (A1 ; A2 ; A3 ) = T (A1 ) + T (A2 ) + T (A3 )

Loop of instructions
The number of elementary instructions executed in a loop is (at
most) the number of iterations multiplied by the number of
elementary instructions executed in the body of the loop:

T (for (int i = 1; i < n; i + +){B}) = n × T (B)

Introduction

Example
// note: x is an unsorted array
int findMin(int[] x) {
int k = 0; int n = x.length;
for (int i = 1; i < n; i++) {
if (x[i] < x[k]) {
k = i;
}
}
return k;
}

T (findMin, n) = 2 + T (findMinLoop) × n + 1
T (findMinLoop) = 2
T (findMin, n) = 3 + 2 × n
Introduction

The O

In order to speak more easily about the complexities algorithms,

they are usually grouped into “families” (logarithmic, polynomial,
etc.).
We will write f (n) = O(g (n)), where f is usually a combination of
polynomial, exponential and logarithmic functions.
We write:
f = O(g )
if and only if:

∃N, ∃C , ∀n > N, 0 ≤ f (n) ≤ C × g (n)

Introduction

The O - Some simple rules

∀f , g , if f = O(g ) and f 0 = O(g 0 ), then:

f + f 0 = O(g + g 0 )
f × f 0 = O(g × g 0 )
Introduction

The O - Examples

f (n) = 2n + 3; f (n) = O(n)

f (n) = 6n2 + 235; f (n) = O(n2 )
f (n) = 6n + 567ln(n); f (n) = O(n)
f (n) = 6n × ln(n); f (n) = O(n)
f (n) = 2exp(n) + nln(n) + n4 56; f (n) = O(exp(n))
Introduction

Average vs. worst-case complexity

Definition (Worst-case complexity)

The worst-case complexity is the complexity of an algorithm when
the input is the worst possible with respect to complexity.

Definition (Average complexity)

The average complexity is the complexity of an algorithm that is
averaged over all the possible inputs (assuming a uniform
distribution over the inputs).
Introduction

Average vs. worst-case complexity

We assume that the complexity of the algorithm is T (i) for an

input i. The set of possible inputs of size n is denoted In .

Worst-case complexity

Twc (n) = maxi∈In T (i).

Average complexity

1 X
Tav (n) = × T (i),
| In |
i∈In

where | In | denotes the size of In , i.e. the number of possible inputs

of size n.
Introduction

A hierarchy of common formal complexities

I constant: O(1)
I logarithmic: O(ln(n))
I linear: O(n)
I quadratic: O(n2 )
I cubic: O(n3 )
I polynomial: O(np ), p ∈ N
I exponential: O(exp(n))
Introduction

A hierarchy of common formal complexities:

some examples

I constant: O(1). Print the first element in a list.

I logarithmic: O(ln(n)). Search a number in a sorted list of size
n.
I linearithmic: O(nln(n)). Sorting a list of size n.
I linear: O(n). Search a number in an unsorted list of size n.
I quadratic: O(n2 ). Multiplication of matrices.
I cubic: O(n3 )
I polynomial: O(np ), p ∈ N
I exponential: O(exp(n)).