Time Complexity

Step Count & Frequency count Method in Algorithm Analysis

The step count method is one of the method to analyze the algorithm. In this
method, we count number of times one instruction is executing. From that we
will try to find the complexity of the algorithm.
Suppose we have one algorithm to perform sequential search. Suppose each
instruction will take c1, c2, …. amount of time to execute, then we will try to
find out the time complexity of this algorithm
Algorithm Numbe Step Frequency Cost
r of Count count
times FC
(SC *
FC)

seqSearch(arr, n, key) 0 0 0 c1
i := 0 1 1 1 c2
while i < n, do n+1 1 n+1 c3
if arr[i] = key, then n 1 n c4
break
end if
i=i+1 n+1 1 n+1 c5
done 0 1 0 c6
return i
Total Cost = c2 + c3(n+1) + c4 (n) + c5 (n+1)
= c2 + 8 c3 + 7 c4 +8 c5 = value Time Complexity

Code SC FC Cost
float sum( float list[], int n) 0 0 1
{
float sum = 0; 1 1 1
int i;
for (i=0; i<n; i++) 1 n+1 1
sum = sum + list[i]; 1 n 1
return sum; 1 1 1
}

Total Cost = 0 + 1+ (n+1) + n + 1 = 2n + 3 = O(n)

list n=3
40 23 67
 0 1 2 3
I =3 sum = 130.0

Now if we add the cost by multiplying the number of times it is executed,

(considering the worst case situation), we will get
Cost=c1+(n+1)c 2+nc3+c 4+c 5
Cost=c1+nc 2+c2+nc 3+c 4+c5
Cost=n(c 2+c3)+c 1+c 4+c5
Cost=n(c 2+c3)+C
Considering the c1 + c4 + c5 is C, so the final equation is like straight line y =
mx + b. So we can say that the function is linear. The complexity will be O(n).
 First define steps:
1. int mean(int a[], size_t n)
2. {
3. int sum = 0; // 1 step, frequency: 1
4. for (int i = 0; i < n; i++) // 1 step, frequency: n+1
5. sum += a[i]; // 1 step, frequency : n
6. return sum; // 1 step, frequency: 1
7. }
Total = 2n+3 means O(n)

 Next determine the frequency of the steps based on N:

1. int mean(int a[], size_t n)
2. {
3. int sum = 0; // 1 step * 1
4. for (int i = 0; i < n; i++) // 1 step * (N+1)
5. sum += a[i]; // 1 step * N
6. return sum; // 1 step * 1
7. }
 Add up the steps: 1 + (N+1) + N + 1
 Reduce: 2N + 3
 Throw away factors that don't grow with N and you're done: O(N)

FREQUENCY COUNT:
Number of times each statement is executed
For a non-executable statement frequency count is 0
SC * FC = Total steps for each statement

Example (iterative algorithm)

What’s the running time of the following algorithm?

// Compute the maximum element in the array a.

Algorithm max(a):
max ← a[0]
for i = 1 to len(a)-1
if a[i] > max
max ← a[i]
return max

The answer depends on factors such as input, programming

language and runtime, coding skill, compiler, operating system,
and hardware.
We often want to reason about execution time in a way that
depends only on the algorithm and its input. This can be achieved
by choosing an elementary operation, which the algorithm
performs repeatedly, and define the time complexity T(n) as the
number of such operations the algorithm performs given an array
of length n.
For the algorithm above we can choose the comparison a[i] >
max as an elementary operation.
1. This captures the running time of the algorithm well, since
comparisons dominate all other operations in this particular
algorithm.
2. Also, the time to perform a comparison is constant: it doesn’t
depend on the size of a.
The time complexity, measured in the number of comparisons, then
becomes T(n) = n - 1.
In general, an elementary operation must have two properties:
1. There can’t be any other operations that are performed more
frequently as the size of the input grows.
2. The time to execute an elementary operation must be constant: it
mustn’t increase as the size of the input grows. This is known
as unit cost.
Worst-case time complexity
Consider this algorithm.

// Tell whether the array a contains x.

Algorithm contains(a, x):
for i = 0 to len(a)-1
if x == a[i]
return true
return false

The comparison x == a[i] can be used as an elementary operation

in this case. However, for this algorithm the number of
comparisons depends not only on the number of elements, n, in the
array but also on the value of x and the values in a:
 If x isn’t found in a the algorithm makes n comparisons,
 but if x equals a[0] there is only one comparison.
Because of this, we often choose to study worst-case time
complexity:
 Let T1(n), T2(n), … be the execution times for all possible inputs
of size n.
 The worst-case time complexity W(n) is then defined as
W(n) = max(T1(n), T2(n), …).
The worst-case time complexity for the contains algorithm thus
becomes W(n) = n.
Worst-case time complexity gives an upper bound on time
requirements and is often easy to compute. The drawback is that
it’s often overly pessimistic.
See Time complexity of array/list operations for a detailed look at the
performance of basic array operations.
Average-case time complexity
Average-case time complexity is a less common measure:
 Let T1(n), T2(n), … be the execution times for all possible inputs
of size n,
and let P1(n), P2(n), … be the probabilities of these inputs.
 The average-case time complexity is then defined as P 1(n)T1(n) +
P2(n)T2(n) + …
Average-case time is often harder to compute, and it also requires
knowledge of how the input is distributed.
Quadratic time complexity
Finally, we’ll look at an algorithm with poor time complexity.

// Reverse the order of the elements in the array a.

Algorithm reverse(a):
for i = 1 to len(a)-1 step 1 Frequency = n
x ← a[i] step 2 Frequency = n
for j = i downto 1 step 3 Frequency = n(n+1) / 2
a[j] ← a[j-1] step 4 Frequency = n(n+1) / 2
a[0] ← x step 5 Frequency : n O(n2)

We choose the assignment a[j] ← a[j-1] as elementary operation.

Updating an element in an array is a constant-time operation, and
the assignment dominates the cost of the algorithm.
The number of elementary operations is fully determined by the
input size n. In fact, the outer for loop is executed n - 1 times. The
time complexity therefore becomes
W(n) = 1 + 2 + … + (n - 1) = n(n - 1)/2 = n2/2 - n/2.
The quadratic term dominates for large n, and we therefore say that
this algorithm has quadratic time complexity. This means that the
algorithm scales poorly and can be used only for small input: to
reverse the elements of an array with 10,000 elements, the
algorithm will perform about 50,000,000 assignments.
In this case it’s easy to find an algorithm with linear time
complexity.

Algorithm reverse(a):
for i = 0 to n/2
swap a[i] and a[n-i-1]

This is a huge improvement over the previous algorithm: an array

with 10,000 elements can now be reversed with only 5,000 swaps,
i.e. 10,000 assignments. That’s roughly a 5,000-fold speed
improvement, and the improvement keeps growing as the the input
gets larger.
It’s common to use Big O notation when talking about time
complexity. We could then say that the time complexity of the first
algorithm is Θ(n2), and that the improved algorithm has Θ(n) time
complexity.
For any defined problem, there can be N number of solution. This is true in
general. If I have a problem and I discuss about the problem with all of my
friends, they will all suggest me different solutions. And I am the one who has
to decide which solution is the best based on the circumstances.
Similarly for any problem which must be solved using a program, there can be
infinite number of solutions. Let's take a simple example to understand this.
Below we have two different algorithms to find square of a number(for some
time, forget that square of any number n is n*n):
One solution to this problem can be, running a loop for n times, starting with the
number n and adding n to it, every time.
/*
we have to calculate the square of n
*/
for i=1 to n
do n = n + n
// when the loop ends n will hold its square
return n
Or, we can simply use a mathematical operator * to find the square.
/*
we have to calculate the square of n
*/
return n*n

In the above two simple algorithms, you saw how a single problem can have
many solutions. While the first solution required a loop which will execute
for n number of times, the second solution used a mathematical operator * to
return the result in one line. So which one is the better approach, of course the
second one.

What is Time Complexity?

Time complexity of an algorithm signifies the total time required by the
program to run till its completion.
The time complexity of algorithms is most commonly expressed using the big
O notation. It's an asymptotic notation to represent the time complexity. We
will study about it in detail in the next tutorial.
Time Complexity is most commonly estimated by counting the number of
elementary steps performed by any algorithm to finish execution. Like in the
example above, for the first code the loop will run n number of times, so the
time complexity will be n atleast and as the value of n will increase the time
taken will also increase. While for the second code, time complexity is constant,
because it will never be dependent on the value of n, it will always give the
result in 1 step.
And since the algorithm's performance may vary with different types of input
data, hence for an algorithm we usually use the worst-case Time complexity of
an algorithm because that is the maximum time taken for any input size.
Calculating Time Complexity
Now lets tap onto the next big topic related to Time complexity, which is How
to Calculate Time Complexity. It becomes very confusing some times, but we
will try to explain it in the simplest way.
Now the most common metric for calculating time complexity is Big O
notation. This removes all constant factors so that the running time can be
estimated in relation to N, as N approaches infinity. In general you can think of
it like this :
statement;
Above we have a single statement. Its Time Complexity will be Constant. The
running time of the statement will not change in relation to N.

for(i=0; i < N; i++)

{
statement;
}
The time complexity for the above algorithm will be Linear. The running time
of the loop is directly proportional to N. When N doubles, so does the running
time.

for(i=0; i < N; i++)

{
for(j=0; j < N;j++)
{
statement;
}
}
This time, the time complexity for the above code will be Quadratic. The
running time of the two loops is proportional to the square of N. When N
doubles, the running time increases by N * N.

while(low <= high)

{
mid = (low + high) / 2;
if (target < list[mid])
high = mid - 1;
else if (target > list[mid])
low = mid + 1;
else break;
}
This is an algorithm to break a set of numbers into halves, to search a particular
field(we will study this in detail later). Now, this algorithm will have
a Logarithmic Time Complexity. The running time of the algorithm is
proportional to the number of times N can be divided by 2(N is high-low here).
This is because the algorithm divides the working area in half with each
iteration.

void quicksort(int list[], int left, int right)

{
int pivot = partition(list, left, right);
quicksort(list, left, pivot - 1);
quicksort(list, pivot + 1, right);
}
Taking the previous algorithm forward, above we have a small logic of Quick
Sort(we will study this in detail later). Now in Quick Sort, we divide the list into
halves every time, but we repeat the iteration N times(where N is the size of
list). Hence time complexity will be N*log( N ). The running time consists of N
loops (iterative or recursive) that are logarithmic, thus the algorithm is a
combination of linear and logarithmic.
NOTE: In general, doing something with every item in one dimension is linear,
doing something with every item in two dimensions is quadratic, and dividing
the working area in half is logarithmic.

Types of Notations for Time Complexity

Now we will discuss and understand the various notations used for Time
Complexity.
1. Big Oh denotes "fewer than or the same as" <expression> iterations.
2. Big Omega denotes "more than or the same as" <expression> iterations.
3. Big Theta denotes "the same as" <expression> iterations.
4. Little Oh denotes "fewer than" <expression> iterations.
5. Little Omega denotes "more than" <expression> iterations.

Understanding Notations of Time Complexity with Example

O(expression) is the set of functions that grow slower than or at the same rate
as expression. It indicates the maximum required by an algorithm for all input
values. It represents the worst case of an algorithm's time complexity.
Omega(expression) is the set of functions that grow faster than or at the same
rate as expression. It indicates the minimum time required by an algorithm for
all input values. It represents the best case of an algorithm's time complexity.
Theta(expression) consist of all the functions that lie in both O(expression) and
Omega(expression). It indicates the average bound of an algorithm. It represents
the average case of an algorithm's time complexity.
Suppose you've calculated that an algorithm takes f(n) operations, where,
f(n) = 3*n^2 + 2*n + 4. // n^2 means square of n
Since this polynomial grows at the same rate as n2, then you could say that the
function f lies in the set Theta(n2). (It also lies in the
sets O(n2) and Omega(n2) for the same reason.)
The simplest explanation is, because Theta denotes the same as the expression.
Hence, as f(n) grows by a factor of n2, the time complexity can be best
represented as Theta(n2).
Time and Space Complexity
Sometimes, there are more than one way to solve a problem. We need to learn
how to compare the performance different algorithms and choose the best one to
solve a particular problem. While analyzing an algorithm, we mostly consider
time complexity and space complexity. Time complexity of an algorithm
quantifies the amount of time taken by an algorithm to run as a function of the
length of the input. Similarly, Space complexity of an algorithm quantifies the
amount of space or memory taken by an algorithm to run as a function of the
length of the input.
Time and space complexity depends on lots of things like hardware, operating
system, processors, etc. However, we don't consider any of these factors while
analyzing the algorithm. We will only consider the execution time of an
algorithm.
Lets start with a simple example. Suppose you are given an array A and an
integer x and you have to find if x exists in array A.
Simple solution to this problem is traverse the whole array A and check if the
any element is equal to x.
for i : 1 to length of A
if A[i] is equal to x
return TRUE
return FALSE
Each of the operation in computer take approximately constant time. Let each
operation takes c time. The number of lines of code executed is actually
depends on the value of x. During analyses of algorithm, mostly we will
consider worst case scenario, i.e., when x is not present in the array A. In the
worst case, the if condition will run N times where N is the length of the
array A. So in the worst case, total execution time will be (N∗c+c). N∗c for
the if condition and c for the return statement ( ignoring some operations like
assignment of i ).
As we can see that the total time depends on the length of the array A. If the
length of the array will increase the time of execution will also increase.
Order of growth is how the time of execution depends on the length of the
input. In the above example, we can clearly see that the time of execution is
linearly depends on the length of the array. Order of growth will help us to
compute the running time with ease. We will ignore the lower order terms, since
the lower order terms are relatively insignificant for large input. We use
different notation to describe limiting behavior of a function.
O-notation:
To denote asymptotic upper bound, we use O-notation. For a given
function g(n), we denote by O(g(n)) (pronounced “big-oh of g of n”) the set of
functions:
O(g(n))= { f(n) : there exist positive constants c and n0 such
that 0≤f(n)≤c∗g(n) for all n≥n0 }
Ω-notation:
To denote asymptotic lower bound, we use Ω-notation. For a given
function g(n), we denote by Ω(g(n)) (pronounced “big-omega of g of n”) the set
of functions:
Ω(g(n))= { f(n) : there exist positive constants c and n0 such
that 0≤c∗g(n)≤f(n) for all n≥n0 }
Θ-notation:
To denote asymptotic tight bound, we use Θ-notation. For a given function g(n),
we denote by Θ(g(n)) (pronounced “big-theta of g of n”) the set of functions:
Θ(g(n))= { f(n) : there exist positive constants c1,c2 and n0 such
that 0≤c1∗g(n)≤f(n)≤c2∗g(n) for all n>n0 }

Time complexity notations

While analysing an algorithm, we mostly consider O-notation because it will
give us an upper limit of the execution time i.e. the execution time in the worst
case.
To compute O-notation we will ignore the lower order terms, since the lower
order terms are relatively insignificant for large input.
Let f(N)=2∗N2+3∗N+5
O(f(N))=O(2∗N2+3∗N+5)=O(N2)
Lets consider some example:
1.
int count = 0;
for (int i = 0; i < N; i++)
for (int j = 0; j < i; j++)
count++;
Lets see how many times count++ will run.
When i=0, it will run 0 times.
When i=1, it will run 1 times.
When i=2, it will run 2 times and so on.
Total number of times count++ will run is 0+1+2+...+(N−1)=N∗(N−1)2. So the
time complexity will be O(N2).
2.
int count = 0;
for (int i = N; i > 0; i /= 2)
for (int j = 0; j < i; j++)
count++;
This is a tricky case. In the first look, it seems like the complexity
is O(N∗logN). N for the j′s loop and logN for i′s loop. But its wrong. Lets see
why.
Think about how many times count++ will run.
When i=N, it will run N times.
When i=N/2, it will run N/2 times.
When i=N/4, it will run N/4 times and so on.
Total number of times count++ will run is N+N/2+N/4+...+1=2∗N. So the time
complexity will be O(N).
The table below is to help you understand the growth of several common time
complexities, and thus help you judge if your algorithm is fast enough to get an
Accepted ( assuming the algorithm is correct ).
Length of Input (N) Worst Accepted Algorithm
≤[10..11] O(N!),O(N6)
≤[15..18] O(2N∗N2)
≤[18..22] O(2N∗N)
≤100 O(N4)
≤400 O(N3)
≤2K O(N2∗logN)
≤10K O(N2)
≤1M O(N∗logN)
≤100M O(N),O(logN),O(1)

Frequency counts for all statements, Data Structure & Algorithms

Assignment Help:
Evaluate the frequency counts for all statements in the following given program
segment.
for (i=1; i ≤ n; i ++)
for (j = 1; j ≤ i; j++)
for (k =1; k ≤ j; k++)
y ++;
Ans.
S1: for (i=1; i<= n; i++) S2: for (j=1; j,= i; j++)
S3 : for (k=1; k<= j; k++) S4 : y++;
Frequency counts of
S1=n
S2 = 1+ (1+2)+ (1+2+3) +....... (1+2+....n)
S3 = 1+ [ 1+ (1+2)] + .....[ 1+ (1+2) +....
(1+2+3+...n)]
S4= same as S3.

for(i=1;i<=n;i++) // Executes n times

for(j=1;j<=i;j++) // Executes i times for every i -> (1 + 2 + 3 + 4....n)
for(k=1;k<=j;k++) // Executes j times for every i,j --->
(1+(1+2)+(1+2+3).....(1+2+3...n))
x=x+1; // Executes every time for every i,j,k ---> (1+(1+2)+.....
(1+2+3...n)

So, you can figure out from this that :

n + n*(n+1)/2 + (n+(n-1)2+(n-2)3.....(1)n)*2 ... = n + n(n+1)/2+ ((n)(n+1)
(n+2)/6)*2 .. This is your answer.

The first for loop executes n times.

The second for loop executes: 1 + 2 + 3 +..... +n = n(n+1)/2
The third for loop executes: ∑ j(j+1)/2 for j = 1 to n, i.e. 1 + 3 + 6 + 10 + .. +
n(n+1)/2. This is called the triangular sequence and the sum is n(n+1)(n+2)/6.

Start by looking at just

for(i=1;i<=n;i++)
for(j=1;j<=i;j++)
expression;
It will go like:
for(i=1;i<=n;i++)
// i=1, 2, ..., n Total = n
for(j=1;j<=i;j++)
expression;
// i=1 -> j=1 Total = 1
// i=2 -> j=1, 2 Total = 2
// ...
// i=n -> j=1, 2, ..., n Total = n
So the expression is executed 1+2+...+n times which is (n+1)*n/2
Now you can calculate the frequency of the individual expressions.
i=1; // 1
i<=n; // n+1
i++; // n
j=1; // n
j<i; // ((n+2)*(n+1)/2) - 1 (2+3+...+(n+1))
j++; // (n+1)*n/2
Using the same method you can add the for(k=1;k<=j;k++) and recalculate
the frequency