Com c3&4
Com c3&4
Com c3&4
1. Computability
After the completion of this chapter students will be able to:
Define what recursive is and recursively enumerable languages are.
Differentiate recursively enumerable and non-enumerable languages
How to represent machines as a string.
Know what language enumerators are.
What is Recursive?
Definition: A language L is recursive if there exists a TM M such that L=L(M) and M halts on
every w2 Σ+.
1.1. Recursive and Recursively Enumerable Languages
Remember that there are three possible outcomes of executing a Turing machine over a
given input. The Turing machine may
Never halt.
A language is recursive if there exists a Turing machine that accepts every string of the
language and rejects every string (over the same alphabet) that is not in the language.
Note that, if a language L is recursive, then its complement must also be recursive.
A language is recursively enumerable if there exists a Turing machine that accepts every
string of the language, and does not accept strings that are not in the language. (Strings
that are not in the language may be rejected or may cause the Turing machine to go into an
infinite loop.)
A language, L is recursively enumerable if it is the language accepted by some Turing
machine M. That is, w € L if and only if M halts on input w in an accepting state. Note that if
we process some word v not element of L with M then M might halt in a non-accepting
state, might crash, or might simply run forever. This situation is rather unsatisfactory – we
know that if a ‘yes’ answer is coming it will arrive eventually, but if the answer is ‘no’ we
may never be sure.
That makes it natural to define a (possibly) stronger notion: a language, L, is recursive if
it is the language accepted by some Turing machine M that halts on all inputs. Now we are
guaranteed a yes or no answer.
Proof. Let Min accept L and Mout accept Σ∗\L. We use these two machines to put together
a multi tape machine. Given an input w we first copy w onto another tape. Then we
run Min on one tape in parallel with Mout on the other (the states of this new machine
correspond to pairs of states of the two given machines, so there are still only finitely
many of them). As soon as one of the two simulations halts, we can halt and announce
the status of w. Since one of the two simulations is guaranteed to halt (Min if w € L and
Mout otherwise) this is sufficient.
1.2. Language-enumerators
There are certainly circumstances in which we might be more interested in making a list of
the elements of a language L rather than recognizing membership in L (for instance, a list
of primes, a list of possible solutions to some optimization problem, . . . ). A machine M is
said to enumerate L if it produces on some output tape (which always moves to the right,
though not necessarily at every computation step) a sequence:
where # is an extra symbol (not from the alphabet of L) and w € L if and only if w = ui for
some i. In other words, the machine just lists the elements of L in some order, and possibly
with repetitions, but every element gets listed eventually.
Proposition 3.1. If there is a TM that enumerates L, then there is a TM that enumerates L
without repetitions.
Proof. Take a TM enumerating L and augment it with another output tape – the ‘real’
output tape. Each time M writes a word uk on its output tape, pause M and use a separate
control module to check whether uk = uj for some j < k. If so, just restart M. If not, write uk
to the ‘real’ output tape (followed by a #) and then restart M.
Of course the main result (and the reason for the name ‘recursively enumerable’) is:
Theorem 3.2. A language L is enumerated by some TM if and only if it is recursively
enumerable.
Proof. Suppose that L is enumerated by M. To build a machine T that accepts M simply add
an extra input tape on which an input word w is written. Then start M. Each time M
produces a new word ui check whether ui = w. If so, halt and accept. If not, carry on.
Thus, if w 2 L we halt and accept, and otherwise we do not.
Now suppose that L is recursively enumerable, accepted by some machine T. To build an
enumerator M we use the ‘simulate for a fixed number of steps on each possible input of a
fixed length’ idea. That is, we build a machine M that successively runs T for k steps on all
input sequences of length at most k for k from 1 to · · · . Each time we spot an acceptance
we add it to the output tape. Thus, we eventually add all the words that T accepts to the
output tape and no others.
The problem with language enumerators (as with recursively enumerable languages) is
that if we haven’t seen a word yet we have no indication whether we ever will. If we could
enumerate the strings of a language in order of their length, then that would be a different
story – once we saw a string longer than the one we were interested in we would know
that waiting any longer would be a forlorn hope.
Say that L can be enumerated by length if there is an enumerator for L which first lists all
the strings of length 0 in L, then all the strings of length 1, then of length 2, and so on. Not
surprisingly
Theorem 3.3. A language L can be enumerated by length if and only if it is recursive.
Proof. If L is recursive, take a machine M that accepts it and halts on all inputs. Run it on
each string of length 0, then each string of length 1 and so on. Each time a string is
accepted, add it to an output tape. The resulting machine enumerates L by length.
Conversely, suppose that L can be enumerated by length. If L is finite then it is regular and
so certainly recursive, and there is nothing to prove. Otherwise, take the enumerator by
length and a desired input word w. Wait until either w or some longer word appears, and
accept or reject accordingly.
1.3. Representing machines as strings
In several of the preceding arguments we have been piecing together new machines from
old in an ad hoc fashion as required. There’s nothing wrong with that, but effectively we’re
treating machines as modules here – and it would seem sensible to perhaps represent
them in some uniform fashion (then we might feel more comfortable about adding certain
operations ‘erase the working tape and write the first string of length 6 on it, then run M
for 6 steps’).
A first step in that process is to come up with a representation of (basic) TMs as strings
– and there’s nothing particularly special about the representation we might choose – in
practice, anything vaguely sensible will do. Since we’re worried more about theoretical
rather than practical arguments we’ll just take a very simple representation.
We’ll use a set alphabet of f0; 1g. The 0 symbol will be used only as punctuation, so we’ll
effectively encode everything else in unary. What do we need to encode?
• An actual alphabet Γ. Call the elements of Γ, a1 through an and encode ai as 1i (a
string of i 1’s).
• A set of states. Call the states q1 through qm and encode qj as 1j
• The directions. Call L, 1 and R, 11.
• The transitions. Transitions can be thought of as 5-tuples (qstart; aread; awrite;
dmove; q finish). So, simply encode these as the encodings of the individual parts
separated from one another by single 0’s.
• The complete set of transitions. Take each transition, encode it, and separate it
from
the next by 00.
• The start/finish of the encoding. Add a 000 at the beginning and at the end.
For convenience, having done all this to a Turing machine M we’ll call the resulting string
the representation of M and denote it by R(M).
1.4. Recursively Enumerable Languages
Definition: A language L is recursively enumerable if there exists a TM M such
that L=L(M).
Now let's also suppose that the complement of L, -L = {w: w L}, is recursively enumerable.
That means there is some other Turing machine T2 that, given any string of -L, halts and
accepts that string.
Clearly, any string (over the appropriate alphabet) belongs to either L or -L. Hence, any
string will cause either T1 or T2 (or both) to halt. We construct a new Turing machine that
emulates both T1 and T2, alternating moves between them. When either one stops, we can
tell (by whether it accepted or rejected the string) to which language the string belongs.
Thus, we have constructed a Turing machine that, for each input, halts with an answer
whether or not the string belongs to L. Therefore L and -L are recursive languages.
Recursively Enumerable But Not Recursive
We know that the recursive languages are a subset of the recursively enumerable
languages, We will now show that they are a proper subset.
We have shown how to enumerate strings for a given alphabet, w1, w2, w3, .... We have
also shown how to enumerate Turing machines, T1, T2, T3, .... (Recall that each Turing
machine defines a recursively enumerable language.) Consider the language
L = {wi: wi L(Ti)}
A little thought will show that L is itself recursively enumerable. But now consider its
complement:
-L = {wi: wi L(Ti)}
If -L is recursively enumerable, then there must exist a Turing machine that recognizes it.
This Turing machine must be in the enumeration somewhere -- call it Tk.
Does wk belong to L?
If wk belongs to L then (by the way we have defined L) Tk accepts this string. But Tk
accepts only strings that do not belong to L, so we have a contradiction.
If wk does not belong to L, then it belongs to -L and is accepted by Tk. But since Tk
accepts wk, wk must belong to L. Again, a contradiction.
We have now defined a recursively enumerable language L and shown by contradiction
that - L is not recursively enumerable.
or the same. Since the limits to human cognition are finite, it can be argued that the set of
all possible questions consists of finite, enumerable elements. The same holds for the
answers.
Furthermore, it can be shown that any infinite set of enumerable elements is equal to the
set of natural numbers (in a sense that there is a bijection between the set of enumerable
elements and the set of natural numbers). Thus, a computation can be viewed as a
transformation from a natural number q (the question) to a natural number a (the answer).
Computation always requires a computer", i.e., a system that performs the computation.
A configuration of a computer is a complete description of the state of such a system at
some time point. The initial configuration of the computer is the question plus the initial
part of its state that is independent of the question (this usually consists of the
computational rules or algorithms the computer will apply in order to answer the
question). The final configuration of the computer contains the answer. Note that a
computer may not necessarily always reach a final configuration (that is, there may be
questions for which it runs forever). Mathematically, a computer can be viewed as a
(partial) mapping that maps the set of questions to the set of answers, i.e. a function of the
form (or, using our arguments above, simply
A fundamental problem has been: is there a “universal" model for computation? Or in
other words, is there a model of a computer that ensures that for every function f that is
“computable" there is a computer with an initial configuration (independent of the
question) so that for every q 2 Q it outputs f(q) in finite time (if f(q) exists)? The main di–
culty in answering this problem is that we simply do not know how powerful computation
can be.
Obviously, any realizable computer must be a finite physical system. The operation of any
such
system is limited by the physical laws. However, as long as we do not have a complete
picture of
the physical laws, we do not know what kind of computations, or changes in state, are
possible.
All computational models developed so far (including Quantum computing models) have
been shown to be computationally equal to a very simple model, the so-called Turing
machine.
This lead Church and Turing in 1936 to the conjecture that the limitations on what can be
computed do not depend on the understanding of physics, but are universal. This is called
the
Church-Turing hypothesis:
Every computable function can be computed by a Turing machine.
One can formulate a more restrictive version of this hypothesis:
Every e–ciently computable function can be e–ciently computed by a Turing machine.
Whereas this was shown to be true for all classical computational models, it is most likely
not true for the Quantum computational models, since there is strong evidence that
Quantum
effects cannot be simulated e–ciently by a (classical) Turing machine. However, since we
are still far away from building a Quantum computer, the hypothesis still holds for all
existing systems.
Definition 1 Big-O Notation
Introduction
All are important but we will mostly talk about time complexity (CPU usage).
What happens as the size of the problem being solved gets larger?
When we are trying to find the complexity of the function/ procedure/ algorithm/
program, we are not interested in the exact number of operations that are being
performed. Instead, we are interested in the relation of the number of operations to the
problem size.
Typically, we are usually interested in the worst case: what is the maximum number of
operations that might be performed for a given problem size. For example, inserting an
element into an array, we have to move the current element and all of the elements that
come after it one place to the right in the array. In the worst case, inserting at the beginning
of the array, all of the elements in the array must be moved. Therefore, in the worst case,
the time for insertion is proportional to the number of elements in the array, and we say
that the worst- case time for the insertion operation is linear in the number of elements in
the array. For a linear-time algorithm, if the problem size doubles, the number of
operations also doubles.
Note that the big-O expressions do not have constants or low-order terms. This is because,
when N gets large enough, constants and low-order terms don't matter (a constant-time
algorithm will be faster than a linear-time algorithm, which will be faster than a quadratic-
time algorithm).
A function T(N) is O(F(N)) if for some constant c and for values of N greater than some
value n0:
The idea is that T(N) is exact complexity of the procedure/function algorithm as a function
of the problem size N, and that F(N) is an upper bound on that complexity(i.e. the actual
time/space or whatever for a problem of size N will be no worse than F(N)).
In practice, we want the smallest F(N) -- the least upper bound on the actual complexity.
In general, how can you determine the running time of a piece of code? The answer is that
it depends on what kinds of statements are used.
Sequence of statements
statement 1;
statement 2;
...
statement k;
The total time is found by adding the times for all statements:
Total time = time (statement 1) + time (statement 2) + ... + time (statement k)
If each statement is "simple" (only involves basic operations) then the time
for each statement is constant and the total time is also constant: O(1).
If-Then-Else
if (condition)
{
----------
----------
}
else if (condition)
{
----------
----------
}
else if (condition)
{
----------
----------
}
else
{
----------
----------
}
Example 1
#include<iostream.h>
void main()
{
int num;
if (num>0)
cout << num <<" is Positive.";
else
cout << num <<" is Negative.";
}
Output :
Example 2
#include<iostream.h>
void main()
{
int amt,discount;
cout << "Enter amount : ";
cin >> amt;
if (amt>20000)
discount=15;
else if (amt>15000)
discount=10;
else if (amt>10000)
discount=5;
else
discount=0;
cout << "You will get " << discount << "% discount.";
}
Output :
Here, either block 1 will execute, or block 2 will execute. Therefore, the worst-case time is
the slower of the two possibilities:
If block 1 takes O(1) and block 2 takes O(N), the if-then-else statement would be O(N).
Loops
initialization;
while (condition)
{
----------
----------
inc/dec;
}
Example 1
#include<iostream.h>
void main()
{
int a=1,num;
while (a<=num)
{
cout << "\nHello...!!";
a++;
}
}
Output :
Enter any number : 5
Hello...!!
Hello...!!
Hello...!!
Hello...!!
Hello...!!
Initialization;
do
{
----------
----------
inc/dec;
} while (condition);
Example 2
#include<iostream.h>
void main()
{
int a=1,num;
do
{
cout << "\nHello...!!";
a++;
} while (a<=num);
}
Output :
The loop executes N times, so the sequence of statements also executes N times. If we
assume the statements are O(1), the total time for the for loop is N * O(1), which is O(N)
overall.
for (initialization;condition;inc/dec)
{
----------
----------
}
void main()
{
int a,num;
for (a=1;a<=num;a++)
cout << "\nHello...!!";
}
Output :
Nested loops
…………………………………………………………………..
…………………………………………………………………..
Definition - 3 Big-Oh notation
If f, g are two functions from N to N, then we (1) say that f = O(g) if there exists a constant c
such that f(n) ≤ c · g(n) for every sufficiently large n, (2) say that f = Ω(g) if g = O(f), (3) say
that f = Θ(g) is f = O(g) and g = O(f), (4) say that f = o(g) if for every > 0, f(n) ≤ · g(n) for
every sufficiently large n, and (5) say that f = ω(g) if g = o(f).
To emphasize the input parameter, we often write f(n) = O(g(n)) instead of f = O(g), and use
similar notation for o, Ω, ω, Θ.
Example 1.3
Here are some examples for use of big-Oh notation:
1. If f(n) = 100n log n and g(n) = n2 then we have the relations f = O(g), g = Ω(f), f = o(g), g =
ω(f).
2. If f(n) = 100n2 + 24n + 2logn and g(n) = n2 then f = O(g). We will often write this relation
as f(n) = O(n2). Note that we also have the relation g = O(f) and hence f = Θ(g) and
g = Θ(f).
3. If f(n) = min{n, 106} and g(n) = 1 for every n then f = O(g). We use the notation f = O(1)
to denote this. Similarly, if h is a function that tends to infinity with n (i.e., for every c it
holds that h(n) > c for n sufficiently large) then we write h = ω(1).
4. If f(n) = 2n then for every number c ∈ N, if g(n) = nc then g = o(f). We sometimes write
this as 2n = nω(1). Similarly, we also write h(n) = nO(1) to denote the fact that h is bounded
from above by some polynomial. That is, there exist a number c > 0 such that for sufficiently
large n, h(n) ≤ nc.
Definition-4-Big-O-notation
In complexity theory, one is mostly interested in the “bigger picture”. Therefore, constant
factors
are usually ignored when considering the resource requirements of computations,
although they
are certainly important in practice. For this, the O-notation has been introduced. For any
function
, the set
1) O(f(n)) denotes the set of all functions with the property that
there are
that and
4) o(f(n)) denotes the set of all functions with the property that
and
the correct relation, but it has been used so often that people just got stuck with it.
2.3. NP and mathematical proofs
By definition, NP is the set of languages where membership has a short certificate. This is
reminiscent of another familiar notion, that of a mathematical proof. As noticed in the past
century, in principle all of mathematics can be axiomatized, so that proofs are merely
formal manipulations of axioms. Thus the correctness of a proof is rather easy to verify —
just check that each line follows from the previous lines by applying the axioms. In fact, for
most known axiomatic systems (e.g., Peano arithmetic or Zermelo-Fraenkel Set Theory)
this verification runs in time polynomial in the length of the proof. Thus the following
problem is in NP for any of the usual axiomatic systems A:
THEOREMS = {(ϕ, 1n) : ϕ has a formal proof of length ≤ n in system A}
In fact, the exercises ask you to prove that this problem is NP-complete. Hence the P versus
NP question is a rephrasing of G¨odel’s question (see quote at the beginning of the chapter),
which asks whether or not there is a algorithm that finds mathematical proofs in time
polynomial in the length of the proof.
Of course, all our students know in their guts that finding correct proofs is far harder than
verifying their correctness. So presumably, they believe at an intuitive level that P is not
equal NP.
Intuitively, the complexity classes P and NP refer to decision problems that can be solved
efficiently and those for which yes-instances can be verified efficiently, respectively. If we 3
insisted on formal correctness here, we would define these classes in terms of a specific
computer model called Turing machines. However, this is beyond the scope of this course
and we therefore take the freedom to introduce these classes using a more high-level (but
essentially equivalent) point of view.
We define the complexity class P (which stands for polynomial-time).
Definition 5.3. A decision problem Π belongs to the complexity class P if there exists an
algorithm that for every instance I ∈ I determines in polynomial time whether I is a yes
instance or a no-instance.
All problems that we have treated so far in this course belong to this class. But also the
LINEAR PROGRAMMING PROBLEM (LP) belongs to this class, even though the simplex
algorithm is not a polynomial-time algorithm for LP.
The simplex algorithm works almost always very fast in practice for any LP of whatever
size, but as mentioned before the running time of an algorithm is determined by its worst-
case running time. For most pivoting rules devised for the simplex algorithm, there have
been constructed instances on which the algorithm has to visit an exponential number of
basic feasible solutions in order to arrive at an optimal one. A polynomial-time algorithm
for LP is the ellipsoid method. This algorithm is an example where the time bound is
polynomial in the logarithm of the largest coefficient in the instance next to the number of
variables and number of restrictions. One of the most interesting open research questions
in Operations Research is whether there exists an algorithm for LP whose running time is a
polynomial in the number of variables and the number of restrictions only.
Next, we define the complexity class NP. NP does not stand for “non-polynomial-time” as
one might guess, but for “non-deterministic polynomial-time” because this class is formally
defined in terms of non-deterministic Turing machines.
Given a yes-instance I ∈ I of a decision problem Π, we say that S is a certificate for I if S ∈ F
and c(S) ≤ K. Note that every yes-instance I must have a certificate. The specialty of a
problem in NP is that yes-instances admit certificates that can be verified in polynomial
time.
Definition 5.4. A decision problem Π belongs to the complexity class NP if every yes
instance I ∈ I admits a certificate whose validity can be verified in polynomial time.
Note that the polynomial-time verifiability of S implies that the size of S must be
polynomially bounded in |I| (because we need to look at S to verify its validity). That is, the
definition above also states that yes-instances of problems in NP have short, i.e.,
polynomialy bounded, certificates.
We consider some examples:
Example 5.3. HAMILTONIAN CYCLE is in NP: A certificate for a yes-instance corresponds to
a set of edges S ⊆ E. One can verify in O(n) time whether S constitutes a cycle in G that visits
all vertices exactly once.
Example 5.4. Consider the decision variant of LINEAR PROGRAMMING:
After thinking for a little while, we conclude that P ⊆ NP. Several decades of intensive
research seem to suggest that there are problems in NP that are intrinsically more difficult
than the ones in P and thus P = NP: Despite the many research efforts, no polynomial time
algorithms have been found for problems in NP such as TSP, HAMILTONIAN CYCLE,
STEINER TREE, etc.
On the other hand, all attempts to show that these problems are in fact harder than the
ones in P failed as well. The question whether P = NP is one of the biggest mysteries in
mathematics to date and constitutes one of the seven millennium-prize
problems; see http://www.claymath.org/millennium for more information.
Complexity theory attempts to give theoretical evidence to the conjecture that P = NP. It
defines within the complexity class NP a subclass of most difficult problems, the so-called
NP-complete problems. This subclass is defined in such a way that if for any of the NP
complete problems there will ever be found a polynomial-time algorithm then this implies
that for every problem in NP there exists a polynomial-time algorithm, and thus P = NP.
The definition of this class crucially relies on the notion of polynomial-time reductions:
Definition 5.5. A polynomial-time reduction from a decision problem Π1 to a decision
problem Π2 is a function ϕ : I1 → I2 that maps every instance I1 ∈ I1 of Π1 to an instance
I2 = ϕ(I1) ∈ I2 of Π2 such that:
1. the mapping can be done in time that is polynomially bounded in the size of I 1;
2. I1 is a yes-instance of Π1 if and only if I2 is a yes-instance of Π2.
If there exist such a polynomial-time reduction from Π1 to Π2 then we say that Π1 can be
reduced to Π2, and we will write Π1 Π2.
Let us think about some consequences of the above definition in terms of polynomial-time
computability.
Suppose Π1 Π2. Then Π2 is more difficult to solve than Π1 (which also justifies the use of
the symbol).
To see this, note that every polynomial-time algorithm ALG2 for Π2 can be used to derive a
polynomial-time algorithm ALG1 for Π1 as follows:
1. Transform the instance I1 of Π1 to a corresponding instance I2 = ϕ(I1) of Π2.
2. Run ALG2 on I2 and report that I1 is a yes-instance if and only if ALG2 concluded that I2
is a yes-instance.
By the first property of Definition 5.5, the transformation in Step 1 above takes time
polynomial in the size n1 = |I1| of I1. As a consequence, the size n2 = |I2| of I2 is
polynomially bounded in n1. (Think about it!) In Step 2, ALG2 solves I2 in time polynomial
in the size n2 of I2, which is polynomial in the size n1.2 The overall time needed by ALG1 to
output a solution for I1 is thus bounded by a polynomial in n1. Note that the second
property of
Definition 5.5 ensures that ALG1 correctly identifies whether I1 is a yes-instance or not.
Observe the existence of a polynomial-time algorithm for Π1 has in general no implications
for the existence of a polynomial-time algorithm for Π2, even if we assume that we can
compute the inverse of ϕ efficiently. The reason for that is that ϕ is not necessarily a one-
to-one mapping and may thus map the instances of Π1 to a subset of the instances of Π2
which correspond to easy instances of Π2. Thus, being able to efficiently solve every
instance of Π1 reveals nothing about the problem of solving Π2.
It is not hard to show that polynomial-time reductions are transitive:
Lemma 5.1. If Π1 Π2 and Π2 Π3 then Π1 Π3.
We can now define the class of NP-complete problems.
Definition 5.6. A decision problem Π is NP-complete if
1. Π belongs to NP;
2. every problem in NP is polynomial-time reducible to Π.
Intuitively, the above definition states that an NP-complete problem is as difficult as any
other problem in NP. The above definition may not seem very helpful at first sight: How do
we prove that every problem in NP is polynomial-time reducible to the problem Π we are
interested in? Let us assume for the time being that there are some problems that are
known to be NP-complete. In order to prove NP-completeness of Π it is then sufficient to
show that Π is in NP and that some NP-complete problem is polynomial-time reducible to
Π. (Think about it!) That is, showing NP-completeness of a problem becomes much easier
now because we “just” need to find an appropriate NP-complete problem that can be
reduced to it. Nevertheless, we remark that the reductions of many NP-completeness
proofs are highly non-trivial and often require a deep understanding of the structural
properties of the problem.
The class NP has a very precise definition in terms of executions of non-deterministic
Turning machines (which we skipped and persist in skipping), which enabled Steven Cook
in 1974 to prove that any such execution can be reduced to an instance of a famous
problem in Boolean logic called the SATISFIABILITY problem (stated below). Thus, Cook
provided us with a problem that is NP-complete. Starting from this, many other problems
were proven to be NP-complete.
We list some more NP-complete problems (without proof) that are often used in
reductions.
2-PARTITION:
Instance: Integers s1,...,sn.
Goal: Decide whether there is a set S ⊆ {1,...,n} such that∑i∈S si = 1 2 ∑n i=1 si.
3-PARTITION:
Instance: Rational numbers s1,...,s3n with 1 4 < si < 1 2 for every i = 1,...,3n.
Goal: Determine whether the set {1,...,3n} can be partitioned into n triplets S1,...,Sn such
that ∑i∈Sk ai = 1 for every k = 1,...,n.
SET COVER.:
Instance: A universe U = {1,... ,n} of n elements, a family of m subsets S1,...,Sm ⊆ U and an
integer K.
Goal: Determine whether there is a selection of at most K subsets such that their union is U.
For example NP-completeness of 2-PARTITION allows an easy proof of the following basic
scheduling problem.
MAKESPAN SCHEDULING:
Instance: m machines and n jobs; each job j has a processing time pj, j = 1,...,n, a constant K.
Goal: Determine whether there is a schedule of the jobs on the machines, in which each
machine cannot process two jobs simultaneously, each job is processed uninterruptedly by
exactly one machine, and all jobs complete processing before time K.
The MAKESPAN SCHEDULING can be interpreted as a load balancing problem: divide the
total work as equally as possible over the machines, such that the load of the machine with
heaviest load is minimized.
Exercise . Show that MAKESPAN SCHEDULING is NP-complete by a reduction from 2-
PARTITION.
2.7. NP-hard Problems
Sometimes we may be unable to prove that a problem Π is in NP but nevertheless can show
that all problems in NP are reducible to Π. According to previous Definition, Π does not
qualify to be an NP-complete problem because it is not in NP. Yet, Π is as hard as any other
problem in NP and thus probably a difficult problem. We call such problem NP-hard. An
example of such a problem is the KTH HEAVIEST SUBSET problem:
KTH HEAVIEST SUBSET:
Instance: Integers w1,...,wn,t and a parameter K.
Goal: Determine whether the weight of the tth heaviest subset of {1,... ,n}
is at least K. (Formally, determine whether there are t distinct subsets
S1,...,St ⊆ {1,...,n} such that w(Si) = ∑j∈Si wj ≥ K for every i = 1,...,t.)
It can be proven that all problems in NP are polynomial-time reducible to the KTH
HEAVIEST SUBSET problem (see [PS82, Theorem 16.8]). However, a proof that short
certificates exist for yes-instance is non-existent. How else could we provide a certificate
for a yes instance other than explicitly listing t subsets that are heavier than K? (Note that
this is not a short certificate because t can be exponential in n.)
We use NP-hardness also for the complexity of optimisation problems whose recognition
version is NP-complete. It is clear that these problems are at least as hard as their decision
versions. Formally they may be excluded from NP. because we cannot speak of certificates
of a yes-instance. But even if polynomial verifiability of a certificate for optimality would
permit membership to the class NP, it is not so clear that they are in NP.
A natural certificate for an optimal solution would be the solution itself, but how would one
verify that this is indeed the optimal solution. In LP it is easy, we could also give (or
compute) the corresponding dual solution, and verification of feasibility of both solutions
and equality of their objective values can be done in polynomial time. In Max Flow, an
optimal flow and a corresponding minimum cut have the same easy verifiability. But for
TSP of VERTEX COVER we do not have such characterizations of optimal solutions and
Verifications of claimed optimality of a given solution is much less obvious.
Usually we do not bother about this and any problem that is not in NP, but for which
polynomial solvability would imply that all problems in the class NP can be solved in
polynomial time is called an NP-Hard problem. In the same way the term NP-Easy is
sometimes used for problems that can be solved in polynomial time using the solution of a
problem in NP.
Officially, NP-hard problems may be in a higher complexity class.
Another complexity class that is related to NP is the class co-NP (which stands for
complement of NP). Here one considers complements of decision problems. As an example,
the complement of HAMILTONIAN CYCLE reads as follows:
HAMILTON CYCLE COMPLEMENT:
Instance: An undirected graph G = (V,E).
Goal: Determine whether G does not contain a Hamiltonian cycle.
There are no short certificates known for yes-instances of this problem.
Definition 5.7. A decision problem Π belongs to the class co-NP if and only if its complement
belongs to the class NP. Said differently, a decision problem belongs to co-NP if every no-
instance I ∈ I admits a certificate whose validity can be verified in polynomial time.
It is not hard to see that every problem in P also belongs to co-NP. Thus, P ⊆ NP∩ co-NP.
Similar to the P = NP conjecture, it is widely believed that NP = co-NP.
Theorem 5.8. If the complement of an NP-complete problem is in NP, then NP = co-NP.
Proof. Assume that the complement Π ¯ 2 of an NP-complete problem Π2 is is in NP. We will
show that the complement Π ¯ 1 of an arbitrary problem Π1 ∈ NP is also in NP thus showing
that NP = co-NP.
Because Π2 is NP-complete, we know that Π1 is polynomial-time reducible to Π2. Note that
the reduction ϕ from Π1 to Π2 is also a polynomial-time reduction from Π ¯ 1 to Π ¯ 2. We
can therefore exhibit a short certificate for every yes-instanceI¯1 of Π ¯ 1 as follows: We first
transform I¯1 to I¯2 = ϕ(I¯1) and then use the short certificate for the yes-instance I¯2
(which must exist because Π ¯ 2 ∈ NP). We conclude that Π ¯ 1 is in NP which finishes the
proof.
Note that the above theorem also implies that if the complement of a problem in NP is also
in NP then (unless NP = co-NP) this problem is not NP-complete. Said differently, a problem
that belongs to NP ∩ co-NP is unlikely to be NP-complete. As an example, consider the
linear programming problem LP. Using duality theory, it is not hard to see that LP ∈ NP∩
co-NP. Before LP was known to be polynomial-time solvable, it was in fact the above
observation that gave strong evidence to the conjecture that LP ∈ P.
Exercise 4.5. Show that LP ∈ NP∩ co-NP.
2.9. Pseudo-polynomiality and Strong NP-completeness
Sometimes the running time of an algorithm is polynomial in the size of the instance and
the largest number in the input. As an example, consider the INTEGER KNAPSACK problem.
INTEGER KNAPSACK:
Instance: Integers c1,...,cn and a parameter K.
Goal: Determine whether there exist integers x1,...,xn such that ∑n k=1 ckxk = K.
This problem can be solved as follows: Create a directed graph G = (V,A) with K + 1
vertices V = {0,1,... ,K} and O(nK) arcs:
A = {(i, j) : 0 ≤ i < j ≤ K and j = i+ ck for some k}.
It is not hard to prove that an instance I of the INTEGER KNAPSACK problem is a yes
instance if and only if there exists a path from 0 to K in G. The latter problem can be solved
in time O(n+m) = O(nK). This running time is not polynomial in the size of I. To see this,
recall that we defined the size |I| of I to be the number of bits that are needed to represent I
in binary. Making the (reasonable) assumption that c1,...,cn < K, the size of I is therefore at
most O(nlogK). The running time of the algorithm is therefore not polynomially bounded in
general. However, if K is bounded by a polynomial function of n then the algorithm would
be a polynomial-time algorithm. That is, depending on the application the above algorithm
might indeed be considered to be reasonably efficient, despite the fact that the problem is
NP-complete (which it is).
The above observation gives rise to the following definition. Given an instance I, let num(I)
refer to the largest integer appearing in I.
Definition 5.8. An algorithm ALG for a problem Π is pseudo-polynomial if it solves every
instance I ∈ I of Π in time bounded by a polynomial function in |I| and num(I).
Problems that remain NP-complete even if the largest integer appearing in its description is
bounded polynomially in the size of the instance is called strongly NP-complete.
Definition 5.9. A problem Π is strongly NP-complete if the restriction of Π to instances I ∈ I
satisfying that num(I) is polynomially bounded in |I| is NP-complete.
Note that many problems that we showed to be NP-complete do not involve any numerical
data that is larger than the input size itself. For example, all graph problems such as
HAMILTONIAN CYCLE, CLIQUE, INDEPENDENT SET, VERTEX COVER, etc. satisfy num(I) =
O(n) and are therefore even strongly NP-complete by definition. In fact, also TSP is strongly
NP-complete because we established NP-completeness even for instances with distances 1
or 2.
As the theorem below shows, we cannot expect to find a pseudo-polynomial algorithm for a
strongly NP-complete problem (unless P = NP).
Theorem 5.9. There does not exist a pseudo-polynomial algorithm for a strongly
NPcomplete problem, unless P = NP.
15Proof. Let Π be a strongly NP-complete problem and suppose that ALG is a
pseudopolynomial algorithm for Π. Consider the restriction Π ¯ of Π to instances I ∈ I that
satisfy that num(I) is polynomially bounded in |I|. By Definition 5.9, Π ¯ is NP-complete. But
ALG can solve every instance I¯ of Π ¯ in time polynomial in |I¯| and num(I¯), which is
polynomial in |I¯|. This is impossible unless P = NP.
2.10. Complexity Class PSPACE
The common criterion that we used to define the complexity classes P and NP was time: P
refers to the set of problems that are solvable in polynomial time; NP contains all problems
for which yes-instances can be verified in polynomial time. There are other complexity
classes that focus on the criterion space instead: The complexity class PSPACE refers to the
set of problems for which algorithms exist that only require a polynomial amount of space
(in the size of the input).
Definition 5.10. A decision problem Π belongs to the complexity class PSPACE if there exists
an algorithm that for every instance I ∈ I determines whether I is a yes-instance or a no-
instance using space that is polynomially bounded in the size of I.
Clearly, every polynomial-time algorithm cannot consume more than polynomial space and
thus P ⊆ PSPACE. However, even exponential-time algorithms are feasible as long as they
only require polynomial space. We can use this observation to see that NP ⊆ PSPACE:
Consider an arbitrary problem Π in NP. We know that every yes-instances of Π admits a
short certificate. We can therefore generate all potential short certificates one after another
and verify the validity of each one. If we encounter a valid certificate throughout this
procedure then we report that the instance is a yes-instance; otherwise, we report that it is
a no-instance. The algorithm may take exponential time because the number of certificates
to be checked might be exponential. However, it can be implemented to use only
polynomial space by deleting the previously generated certificate each time.
What if P = NP?
If P = NP —specifically, if an NP-complete problem like 3SAT had a very efficient algorithm
running in say O(n2) time— then the world would be mostly a Utopia. Mathematicians
could be replaced by efficient theorem-discovering programs (a fact pointed out in Kurt
G¨odel’s 1956 letter and discovered three decades later). In general for every search
problem whose answer can be efficiently verified (or has a short certificate of correctness),
we will be able to find the correct answer or the short certificate in polynomial time. AI
software would be perfect since we could easily do exhaustive searches in a large tree of
possibilities. Inventors and engineers would be greatly aided by software packages that can
design the perfect part or gizmo for the job at hand.
VLSI designers will be able to whip up optimum circuits, with minimum power
requirements. Whenever a scientist has some experimental data, she would be able to
automatically obtain the simplest theory (under any reasonable measure of simplicity we
choose) that best explains these measurements; by the principle of Occam’s Razor the
simplest explanation is likely to be the right one.
Of course, in some cases it took scientists centuries to come up with the simplest theories
explaining the known data. This approach can be used to solve also non-scientific
problems: one could find the simplest theory that explains, say, the list of books from the
New York Times’ bestseller list.
Somewhat intriguingly, this Utopia would have no need for randomness. If P = NP then
randomized algorithms would buy essentially no efficiency gains over deterministic
algorithms; (Philosophers should ponder this one.)
This Utopia would also come at one price: there would be no privacy in the digital domain.
Any encryption scheme would have a trivial decoding algorithm. There would be no digital
cash,
no SSL, RSA or PGP. We would just have to learn to get along better without
these, folks.
This utopian world may seem ridiculous, but the fact that we can’t rule it out shows how
little
we know about computation. Taking the half-full cup point of view, it shows how many
wonderful
things are still waiting to be discovered.
What if NP = coNP?
If NP = coNP, the consequences still seem dramatic. Mostly, they have to do with existence of
short certificates for statements that do not seem to have any. To give an example, remember the
NP-complete problem of finding whether or not a set of multivariate polynomials has a common
root,
In other words, deciding whether a system of equations of the following type has a solution:
f1(x1, . . . , xn) = 0
f2(x1, . . . , xn) = 0
...
fm (x1, . . . , xn) = 0
Where each fi is a quadratic polynomial.
If a solution exists, then that solution serves as a certificate to this effect (of course, we have
to
also show that the solution can be described using a polynomial number of bits, which we omit).
The problem of deciding that the system does not have a solution is of course in coNP. Can we give
a certificate to the effect that the system does not have a solution? Hilbert’s Nullstellensatz
Theorem seems to do that: it says that the system is infeasible iff there is a sequence of polynomials
g1, g2, . . . , gm such that Pi figi = 1, where 1 on the right hand side denotes the constant polynomial 1.
What is happening?
Does the Nullstellensatz prove coNP = NP? No, because the degrees of the gi’s —and hence the
number of bits used to represent them— could be exponential in n, m.
(And it is simple to construct fi’s for which this is necessary.)
However, if NP = coNP then there would be some other notion of a short certificate to the effect that
the system is infeasible. The effect of such a result on mathematics would probably be even greater
than the effect of Hilbert’s Nullstellensatz. Of course, one can replace Nullstellensatz with any other
coNP problem in the above discussion.