Computational Complexity

. Introduction
Computational Complexity studies:

the efficiency of algorithms

the inherent "difficulty" of problems of practical and/or theoretical

importance

A major discovery in the area was that computational problems can vary
tremendously in the effort required to solve them precisely. The technical
term for a hard problem is "NP-complete" which essentially means: "abandon
all hope of finding an efficient algorithm for the exact solution of this
problem". Proving or knowing that a problem is NP-complete is not at all that
negative of course. Knowing such limitations, experts do not waste time in
impossible projects and instead turn to less ambitious approaches, for
example to find approximate solutions, to solve special cases or to alter the
problem a little so that it becomes tractable (even at the loss of some fit to
the real-life situation). The goal of this theory is therefore to assist algorithm
designers in directing their efforts towards promising areas and avoid
impossible tasks.

Measuring the efficiency of algorithms

Regarding the efficiency of algorithms, the rapid advances in computer
technology make physical measures (run-time, memory requirements)
irrelevant. A more standardized measure is the number of elementary
computer operations it takes to solve the problem in the worst case. Average
performance is not safe. There may be particular cases (technically called
instances of a problem) that behave much worse than the average and
nobody can afford to rely on his/her luck.
The number of elementary operations depends of course on the "size" of the
problem data. Sorting 1 billion numbers is quite different from sorting 10
ones, so we express the number of elementary operations (expected in the
worst-case scenario) as a function of some characteristic number(s) that
suffice to capture the volume of the work to be done. For a sorting algorithm,
this number is simply the number n of numbers to be sorted.
Now, suppose an algorithm solves a problem of size n in at most 7n 3 + 5n2 +
27 operations.

For such functions, we are primarily interested in their rate of growth as n

increases. We want to distinguish between mild and "exploding" growth
rates, therefore differences as that between 7n3 and n3 are not really
important (besides, large differences in the constants do not arise in
practice). We can also discard the lower order terms, because at large sizes it
is the highest degree that determines the rate of growth.
The end result is that the complexity of this algorithm can be sufficiently
described by the function g(n) = n3. Formally, we say that this algorithm is
"of order O(n3)". It is also usual to say that this algorithm "takes time O(n3)".
This symbolism is a reminder that this function expresses the worst-case
behavior at sufficiently large sizes.
Note that for sufficiently large n: logn < n < nlogn < n2 < n3 < 2n
logn is interpreted as base-2 logarithm (it does not really matter, since log2n
= log10n / log102 and constants are ignored as already mentioned).
This is sometimes stated as: O(logn) < O(n ) < O(nlogn) < O(n 2) < O(n3) <
O(2n)
Examples:
1. The Traveling Salesman problem is a well-known problem in Combinatorial
Optimization, and a notoriously difficult one to solve exactly. The formulation
is very simple though: "Given the coordinates of n cities, find the shortest
closed tour which visits each city exactly once". This problem has served as a
benchmark for almost every new algorithmic idea and was one of the first
optimization problems conjectured and then shown to be NP-complete.

2 different tours on the same set of cities

To see an algorithm for this famous problem in action, visit Onno Waalewijn's
site and try his TSP Java applet.
A "brute force" approach problem would be a "complete enumeration"
scheme:

begin
input n (number of cities), C(n x n) (non negative integer distance matrix)
min = sufficiently large number (say, n times the maximum element of C)
for all possible tours, i.e. cyclic permutations of {1,2,...,n}
find length of tour
if length < min then
min=length, store tour
endif
next tour
end
Since there are (n-1)! possible tours, the for ... next loop is executed (n-1)!
times, each one requiring time O(n) (the time it takes to find the length of
the tour, that is add n integers). Therefore, this algorithm is of order (n-1)!
O(n) = O(n!). This is certainly prohibitive. If for example a solution to a 10city problem can be found in 1 second, increasing the number of cities to 20
would result to time: 20!/10! or about 20,000 years.
Compare this result to the performance of a good approximate algorithm for
the same problem, namely Lin's 3-optimal heuristic published in [2]. At every
iteration of this algorithm, a tour consists of links between cities. The main
idea is to locate at each step a triplet of cities which can be connected in a
better (length reducing) way. The algorithm stops when no such triplet can
be found. The solution is not guaranteed to be optimal but the time required
is O(n3). Now, if a 10-city problem takes 1 second, a 20-city one would
require only 7 more seconds.
2. An O(nlogn) sorting algorithm
The following algorithm is based on the observation that a list can be easily
sorted if it consists of two individually sorted lists. To sort the n elements of a
vector A(1),...,A(n) one would call the following recursive procedure with
parameters low=1, high=n:
Procedure mergesort(A,low,high)
input: A(low),A(low+1),...,A(high)
output: same numbers in non-decreasing order
begin
if low <high then
mid = [(low+high)/2] ! comment: [] stands for "greater integer smaller
than"
call mergesort(low,mid)
call mergesort(mid+1,high)
merge(low,mid,high) ! comment: auxiliary routine to combine the results,

takes time O(n)

endif
end
If the computing time for mergesort is T(n) and that for merge is cn, then:
T(1)=a=constant (the time it takes to call the routine with a single number,
do nothing and return), and
T(n)=2T(n/2)+cn, for n>1 (two calls to mergesort and one call to merge)
If n is a power of 2, say n = 2k:
T(n) = 2T(n/2)+cn = 2(2T(n/4)+cn/2)+cn = 4T(n/4)+2cn =
4(2T(n/8)+cn/4)+2cn = 8T(n/8)+3cn = ... = 2kT(1)+kcn =an+cnlogn
If n is not a power of 2, it is less than some 2k, so T(n) is bounded by the
above. Therefore T(n)=O(nlogn) for all n.
(For a more detailed description, including an efficient implementation of the
auxiliary procedure merge.

A3. Polynomial and Exponential time algorithms

Example #1 illustrated a very important point: Algorithms which have a
polynomial or sub-polynomial time complexity (that is, they take time O(g(n))
where g(n) is either a polynomial or a function bounded by a polynomial), are
practical.
Such algorithms with running times of orders O(logn), O(n ), O(nl2ogn), O(n2),
O(n3) etc. are called polynomial-time algorithms.
On the other hand, algorithms with complexities which cannot be bounded
by polynomial functions are called exponential-time algorithms. These
include "exploding-growth" orders which do not contain exponential factors,
like n!.
There are several arguments to support the thesis that polynomial is a
synonym to practical:
First, polynomial-time algorithms are in a much better position to exploit
technological advancements which result to increases in computers speed.
This can be better shown in the following table:
Time

Size of Largest Problem Instance solvable in 1 Hour

complexity
function

With present
computer

With computer
100 times
faster

With computer
1000 times
faster

N1

100N1

1000N1

n2

N2

10 N2

31.6 N2

n3

N3

4.64 N3

10 N3

2n

N4

N4+6.64

N4+9.97

3n

N5

N5+4.19

N5+6.29

For example, in the O(2n) case:

present computer runs for a2n secs, 1hr = a2N4 ,
100 times faster computer runs for a2n/100 secs, 1hr = a2Nx /100, therefore
100 2N4 = 2Nx, Nx = N4+(log100/log2) = N4+6.64
That is, the effect of improved technology is multiplicative in polynomial-time
algorithms and only additive in exponential-time algorithms. The situation is
much worse than that shown in the table if complexities involve factorials. In
example #1, if an algorithm of order O(n!) solves a 300-city problem in the
maximum time allowed, increasing the computation speed by 1000 will not
even enable solution of problems with 302 cities in the same time.
A second argument is that, polynomial-time algorithms, once they are
discovered, undergo a series of improvements, that is, reductions in the
constants in their complexity functions and the degree itself. This is an
empirical result and partly explains why complexities like O(n80) or O(10100n)
do not appear in practice. Polynomial-time algorithms have typically a
degree about 2 or 3, and do not involve extremely large coefficients. The real
breakthrough for the solution of a problem is in finding the first polynomialtime algorithm. For combinatorial problems, the "jump" to the polynomial
class usually requires a deep insight into the nature of the problem (if this
jump can be made at all).
There are, of course, exceptions to these rules. The most famous case is the
excellent performance of the Simplex method for Linear Programming which
has always worked very well in practice, although it is provably exponential
(there are artificial "pathological" cases where Simplex shows its hidden
exponential behavior). It might be that the method captures some significant
property of the problem which has yet to be discovered.

The general conclusion however is that a problem can be considered

"efficiently solved" when a polynomial-time algorithm has been found for it.
Some examples of optimization problems which are considered "well solved"
in that sense:

Linear Programming (polynomial algorithms do exist)

some Shortest Path problems

some Longest Path problems (for example simple PERT/CPM

calculations)

the Shortest Spanning Tree problem

some network flow problems

Matching problems

Transportation, Assignment and Transshipment problems

several special cases of "hard" problems

Classes P and NP
The class P consists of all those recognition problems for which a
polynomial-time algorithm exists.
For the class NP, we simply require that any "yes" answer is "easily"
verifiable. That is, both the encoding of the answer and the time it
takes to check its validity must be "short", i.e. polynomial or
polynomially-bounded. Formally we say that any "yes" instance of the
problem has the "succinct certificate" property.

NP stands for "Non-deterministic Polynomial", because of an alternative (and

equivalent) definition based on the notion of non-deterministic algorithms.
This will be explained later. A remark on the definition given above: in order
to show that a problem belongs to class NP, it is not necessary to show how
this easily verifiable answer can be produced (in other words, solve the
problem). Suffices to to show that such a certificate exists.

To clarify, for the TSP-rec problem (see B1), just imagine we have a "yes"
answer, that is an ordering of the cities, say c1, c2, c3, ..., cn, such that the
sum of the links (c1,c2), (c2,c3), ... , (cn,c1) is less than or equal to L. How we
obtain this solution is irrelevant. It is clear that we can easily check the
length of the tour, so we know that TSP-rec is in NP. On the other hand, we
do not know if this problem is also in P, because no polynomial algorithm has
yet been found for it.
Class NP consists therefore of all "reasonable" problems of practical and/or
theoretical importance. For problems not in NP, even verifying that a solution
is valid can be extremely difficult.
As expected, the "easy" problems in class P are also "reasonable", i.e. they
also belong to NP. That is:
P is a subset of NP

To show this:
Suppose that there is a polynomial-time algorithm A for problem PA. Given
any "yes" instance x of PA, A will operate on x for a polynomial number of
steps and answer "yes". Now the record of this operation of A on x is a valid
certificate (polynomial in length, by definition). Indeed, one can check the
certificate simply by checking that it is a valid execution of A.
Hence, P NP. The important question now is whether P is a proper subset of
NP, i.e. P NP. This is a major conjecture in Complexity Theory, in words: "not
all reasonable problems can be solved efficiently". Although no formal
proof exists yet, it will be shown later that there is strong evidence for the
validity of this conjecture.
Another example of a recognition problem which, like TSP-rec, belongs to NP
but we do not know if it also belongs to P, is the recognition version of the
Maximum Clique problem. This problem is in NP, since a "yes" answer can
simply be verified by examining the list of nodes that are supposed to form a
clique and see that they are all connected to each other:
begin
(nodes v1,v2 ,..., vk are supposed to form a clique)
for i=1,...,k-1
for j=i+1,...,k ! comment: this double loop examines every pair of nodes in
v1,v2 ,..., vk
if there is no link between vi and vj exit and report "false"

next j
next i
report "true" and exit ! comment: no missing link found
end

Polynomial reductions and transformations

The basic tools for relating the complexities of various problems are
polynomial reductions and transformations.
We say that a problem A reduces in polynomial-time to
another problem B, if and only if:
1. there is an algorithm for A which uses a subroutine for
B, and
2. each call to the subroutine for B counts as a single step,
and
3. the algorithm for A runs in polynomial-time.

(the latter implies that the subroutine for B can be called at most a
polynomially bounded number of times).

The practical implication comes from the following proposition:

If A polynomially reduces to B and there is a polynomial-time algorithm
for B, then there is a polynomial-time algorithm for A

The proof is based on the fact that the composition of two polynomials is
polynomial.
The three basic cases are as follows:

A reduces to B) and (B is "easy")

applications)

A is easy (see [1] for many such

(A reduces to B) and (A is "hard")

would be hard - contradiction)

B is hard (because if B were easy A

(A reduces to B) and (B is "hard")

common case)

(no conclusion for A - a much

Now, suppose we represent the "hardness" of a problem with a binary

variable d, that is, d=1 if a problem is "hard" and d=0 if a problem is "easy".
Then, "A polynomially reduces to B" can be expressed with dB

(dB

dA) and (dB=0)

dA=0

(dB

dA) and (dA=1)

dB=1

(dB

dA) and (dB=1)

dA is free to take any value

dA. Indeed:

Hence, we are justified in saying that "B is at least as hard as A":

If A polynomially reduces to B then "B is at least as hard
as A"

An example has already been presented in section B1: TSP reduces to TSPrec so TSP-rec is at least as hard as TSP.
A special case of polynomial reductions is of the following form:
The algorithm for A first constructs the input for the hypothetical subroutine
for B, then calls the latter exactly once, and finally returns the answer as its
own.
Reductions of this kind are called polynomial transformations (we then say
that problem A transforms in polynomial-time to problem B, and use the
notation A
B), and are especially useful because the polynomial
transformability relationship is transitive as it can easily be shown using
again the fact that the composition of two polynomials is a polynomial.
As an example, we will use the recognition version of the Integer
Programming Problem (IP) and show that SAT

IP-rec.

Recall the example of a Boolean formula in CNF, examined in section B1.

From this, an equivalent IP can be constructed:

x1 + x2 + x3 1
x1 + (1-x2) 1
x2 + (1-x3) 1
x3 + (1-x1) 1
(1-x1) + (1-x2) + (1-x3)

that is, we assign 1 to "true", 0 to "false" and we represent xi as (1-xi). To

see why this works, take for example the second clause, (x1 OR x2). This
clause is true if either x1 is true or x2 is false or both. This is equivalent to
requiring x1=1 or (1-x2)=1 or x1 + (1-x2) = 2 which is exactly what is imposed
by the corresponding constraint.
To complete the IP construction, we choose one constraint, say the first one,
x1 + x2 + x3 1, in the above example, and turn it into an objective function
to maximize:
Maximize (x1 + x2 + x3)
Subject to:
x1 + (1-x2) 1
x2 + (1-x3) 1
x3 + (1-x1) 1
(1-x1) + (1-x2) + (1-x3) 1
If in the optimal solution, x1 + x2 + x3 turns out to be 1, then the original
Boolean formula is satisfiable. Therefore, an algorithm for the SAT:

constructs an equivalent IP-rec in polynomial time,

calls once the algorithm for IP-rec and obtains a a "yes"or "no" answer,

returns the answer above as the answer to the instance of SAT

This means that SAT

IP-rec.

For a very detailed discussion on complexity issues concerning TSP-related

problems (and not only), see [2].

NP-complete problems
We say that a problem is NP-complete, if:
i. it belongs to class NP
ii. all the other problems in NP polynomially transform to it

An NP-complete problem has the following most important property:

If there is an efficient (i.e. polynomial) algorithm for some NP-complete
problem, then there is an efficient algorithm for every problem in NP

Because, if say P* is this problem:

all problems in NP polynomially transform to P* and since polynomial

transformability is a special case of polynomial reducibility, applying
case 1:

(all problems in NP reduce to P*) and (P* is "easy")

are easy

all problems in NP

Hence, NP-complete problems are "the main targets" in the search for
algorithmic efficiency.
Now, to show that a "reasonable" problem (that is a problem in NP) is NPcomplete, it is sufficient to show that some other problem already known to
be NP-complete polynomially transforms to it, as the figure below suggests
(recall that polynomial transformability is transitive):

This would imply that all other problems in NP polynomially transform also to
A, therefore A is also NP-complete.
To start with, we still need to use the definition explicitly in order to find
some "first" NP-complete problem. This is discussed next.