Intractability

Analysis of Algorithms
Problem-Solving Models
We are now shifting gears from solving
specific problems to problem-solving models
We are also moving away from
linear/quadratic-scale problems to
polynomial/exponential-scale problems
This changes the emphasis from discussing the
details of implementation to discussing a
conceptual framework
Problem Classifications
Goal: Categorize computational problems
according to how difficult they are to solve
Difficult to solve = lots of computing resources
Note that there are a huge (infinite?) number
of computational problems that defy
categorization

Problem Classifications
http://cmapspublic3.ihmc.us/rid=1194324366996_1367824329_12460/ComplexityTheory.cmap
Computers and Intractability
From Computers and Intractability: A Guide to the Theory of NP-Completeness by Garey & J ohnson
From Computers and Intractability: A Guide to the Theory of NP-Completeness by Garey & J ohnson
Computers and Intractability
From Computers and Intractability: A Guide to the Theory of NP-Completeness by Garey & J ohnson
Computers and Intractability
Complexity Theory
The field of complexity theory deals with the
questions of just how difficult a given problem
is, and it supplies us with the most famous
and important unsolved problem in
theoretical computer science:
is P = NP?
Complexity Theory
The aim of complexity theory is to classify
problems according to their intrinsic difficulty
For which problems can we find efficient
algorithms?
This, of course, depends on the definition of
efficient, which is a subjective matter
Complexity Theory
In the 1960s Jack Edmonds proposed (in a
paper called Paths, Trees, and Flowers) that
an algorithm should be called efficient if its
running time was a polynomial function of the
size of the input, and inefficient otherwise
Edmunds conjectured that there is no
polynomial-time algorithm that solves the
Traveling Salesman problem
Complexity Theory
This suggestion was adopted, and since then,
a problem is considered tractable if it has a
polynomial time solution and intractable if it
does not
It is a measure of our inadequacy that for
most problems we still do not know whether
they are tractable or not
Complexity Theory
To formally define what we mean by tractable
problems, we have to be specific about what
we mean by the size of the input and be
completely specific about exactly what an
algorithm is
The Class P
Most of the algorithms that we have
considered so far are considered polynomial
time algorithms because they all have run
times or costs that are polynomial functions
Together these algorithms are said to be in the
class P
The Class P
The class P consists of those decision problems
for which there is a deterministic Turing
machine that when presented with a string x
describing an instance of those problems will
correctly respond yes or no in time bounded
above by a polynomial function of |x|
The Class P
The class P is robust in that the problems it
contains are the same even if we change the
model of computation in various reasonable
ways
At first sight to show that a problem is in P we
would have to actually find a Turing machine
to solve that problem in polynomial time
The Class P
Fortunately we can stick with our normal
complexity analysis in terms of such
elementary operations such as array accesses,
additions, etc., because it can be shown that a
Turing machine can implement all of those in
polynomial time
Therefore, such problems such as Minimum
Spanning Tree are clearly in P
The Class P
For many other problems, nobody knows
whether or not they are in P
Sometimes there is an obvious nave
algorithm which consists of trying all the
possibilities
For example, one could solve the Traveling
Salesman problem by trying all N! possible
tours
The Class P
There are no known algorithms for these
problems that are substantially better than
trying all the possibilities
There are two possible reasons why this is so:
We have not yet been clever enough to find a
polynomial time algorithm to solve the problem
A polynomial time algorithm does not exist
The Class NP
The class NP contains a collection of problems
that might have polynomial time solutions
The idea is to exclude problems such as the
Halting problem which can never have
polynomial time solutions
The Class NP
The class NP consists of those decision
problems for which there is a non-
deterministic Turing machine that when
presented with a string x describing an
instance of those problems will correctly
respond yes or no in time bounded above by a
polynomial function of |x|
The Class NP
Always remember:

P = Polynomial
NP = Non-deterministic Polynomial
The Class NP
There are two views that may help with
understanding the class NP:
NP is the class of problems that can be solved in
polynomial time by an infinite number of Turing
machines running in parallel
NP is the class of problems for which a correct
answer can be verified in polynomial time,
assuming that an answer is available
The Class NP
The class P is a subset of the class NP, because
we could construct an algorithm that verifies
our class P problems with the same process
that is used to verify class NP problems
The difference is that for class P, we know we
can find the solution in polynomial time, but
for class NP, we arent sure if we can find the
solution in polynomial time
Thus, it is easy to see that class P is contained
within NP, so we have the following relationship:
P
NP
The Class NP
Despite the fact that the definition of NP is so
much broader than that of P, nobody has yet
managed to find one single problem that can
be shown to be in NP, but not in P
After some 25-30 years of sustained effort
there still remains the question:
Is P = NP?
Is P = NP?
In order to solve this question a vast body of
theory has been built up
We shall merely scratch the surface of this by
considering those problems that are known to
be NP-complete
Is P = NP?
Is P = NP?
NP
P NPC
If NP-Complete problems are no more difficult than P problems
(the right diagram), then P must be equal to NP.
P
NPC
NP
P NP
P =NP
Problems and Languages
A problem to be solved can be thought of as a
language
An instance of the problem can be represented as
an encoding of a string
If the language accepts the string, then the string
represents a solution to the problem, otherwise it
is not a solution
Reducing problem A to problem B is the same
thing as reducing language L
1
to language L
2

Problems and Languages
Definition: If L
1
, L
2
{0,1}* are languages such
that L
1

p
L
2
, then L
2
P implies L
1
P
L
1
is considered to be easier than L
2
, or at least no
harder than L
2

When trying to prove that a problem is NP-
complete, we are interested in going in the
opposite direction, i.e., we already know about L
1
,
and we are trying to show that L
2
is harder than,
or at least as hard as L
1


NP-Complete Problems
A language L {0,1}* is NP-complete if:

1. L NP
2. L
p
L for every L NP

If L satisfies 2 but not necessarily 1 then L is
called NP-hard
Suppose we have a problem, A, that we would
like to solve efficiently
Also suppose we have an algorithm that can
solve a different problem, B, efficiently
We dont have to understand the algorithm for
problem B, just understand the problem that it
solves
Problem A seems a lot like problem B
Could we use the algorithm for B to solve A?
Problem Reductions
Problem Reductions
If we could somehow transform problem A
into problem B, and then run it through the
algorithm for B, we could then un-transform
the output and get our solution for A
Essentially, we know how to solve B, but we dont
know how to solve A
Transform (reduce) A to B, and use the solution for
B as a solution for A
Problem A reduces to problem B if we can use
an algorithm that solves B to solve A
Cost of solving A =cost of solving B +cost of reducing A to B.
Solution for A
Algorithm for B
Instance of A
Algorithm for A
A B
B A
Problem Reductions
Problem Reductions
If we can do the transformation from A to B in
polynomial time, and we can solve B in
polynomial time, we know that A is also
solvable in polynomial time
This technique is used to prove that certain
problems are NP-complete, with one
difference
Problem Reductions
A polynomial-time reduction algorithm uses the
easiness of B to prove the easiness of A
An NP-complete proof is about showing how
hard a problem is, not how easy it is, therefore
we use the polynomial-time reduction in the
opposite direction
Problem Reductions
Proof-by-contradiction: Assume the algorithm
for problem B is efficient and A
p
B
If this assumption is true, then we could use the algorithm
for B to efficiently solve problem A
If were pretty sure that we cant solve A efficiently, what
does that tell us about our assumption regarding the
algorithm for B?
Solution for
A
Algorithm for B
Instance of A
Algorithm for A
A B
B A
Problem Reductions
Solution for
Hard Problem, A ?
Hard Problem, A
Algorithm for A
A B
B A
Lets say we have an algorithm that solves problem B, but we dont
know its efficiency. All we know is that problem B is at least as hard to
solve as problem A (probably harder).

So, if A is a really hard problem and A can be polynomially-time
reducible to B, then the algorithm for B must take at least as much time
as the algorithm for A, probably more time (since the algorithm for B
can solve even harder problems).
Crossword Puzzles
Consider solving a crossword puzzle
If you were able to correctly guess which letter goes in
each square, then the puzzle is solvable in O(N) where
N is the number of words
It is the guessing part of the algorithm that makes it
non-deterministic (non-determinism = guessing
correctly = NP)
A sequence of guesses that leads to the solution is
called the certificate (also called a witness)
Using the certificate, lets prove that solving
crossword puzzles (C-PUZZLE) is NP-complete
Assume that we already know that GRAPH-
COLOR is NP-complete (it is), show:
GRAPH-COLOR
p
C-PUZZLE
To do this, show that a correct solution to C-
PUZZLE will allow us to solve GRAPH-COLOR
Since there is no known efficient solution for
GRAPH-COLOR, then we wont be able to solve C-
PUZZLE efficiently either
Crossword Puzzles
Take a specific instance of graph coloring, and explain how you
would reduce it to a specific instance of a crossword puzzle:
Possible words:
RG, RB, GR, BR, BG,
RRR, GGG, BBB,
RRRRR, GGGGG,
BBBBB
The vertices in the graph will become 3-letter words in the corners of a puzzle.
The edges in the graph will become 5-letter words connecting the corners of
the puzzle. The restriction that no two adjacent vertices in the graph can be
connected by an edge corresponds to 2-letter words in the puzzle. Note that all
3- and 5-letter possible words will contain only the same letters, whereas the
2-letter words will always contain two different letters, to make it fit the graph.
Crossword Puzzles
If we had a program that could solve crossword puzzles,
we could create our special crossword puzzle that
corresponds to our graph coloring problem:
Crossword Puzzles
R
R
R
R
R R
R
R R R R
R
R
R R
R
R
G
G
G
G
G G G
G
G
G G G G
The top part of the puzzle corresponds to the solution
for the graph where the upper-left vertex is colored red
and the upper-right vertex is colored green. The 2-letter
words guarantee that adjacent vertices have different colors.

Since our program solves crossword puzzles, we could use it to solve graph
coloring. But since we know that graph coloring cannot be solved efficiently
(unless P = NP), then we also know that solving crossword puzzles must
also be inefficient. Therefore, C-PUZZLE is NP-complete.
Traveling Salesman
In the traveling salesman problem (TSP), a salesman
must visit N cities, visiting each city exactly once and
finishing at the city from which he started
The salesman incurs a non-negative cost to travel from city
to city, and the salesman wishes to make the tour whose
total cost is minimum
The total cost is the sum of the individual costs along the
edges of the tour
Minimum cost tour
{u, w, v, x, u} =7
We can model the traveling salesman problem using a
complete graph with N vertices
A Hamilton Cycle (HAM-CYCLE) is a path through a graph
that visits each of the nodes exactly once and returns to
the starting node
Assuming that we already know that HAM-CYCLE is NP-
Complete, we can use HAM-CYCLE to prove that TSP is
NP-Complete
We can reduce HAM-CYCLE to TSP because HAM-CYCLE is a
TSP, with each edge having a weight of 1
Thus, TSP is a reduction from HAM-CYCLE
Traveling Salesman
Traveling Salesman
How to reduce HAM-CYCLE to TSP (in p-time):
Each node in the Hamilton cycle becomes a city
Give every edge in the Hamilton cycle a cost of 1
Add more edges to the Hamilton cycle, each with a cost of 2, to fully
connect all cities: now it is a TSP problem
If the TSP tour has cost of N, then there is a Hamilton
cycle of length N
If the TSP tour is greater than N, then there is no
Hamiltonian cycle because the tour must travel
between cities where there is no corresponding
Hamilton edge (the edges with cost 2)

NP-Complete Proofs
If any NP-complete problem can be solved in
polynomial-time, then every problem in NP has
a polynomial-time solution, i.e., P = NP
Therefore, it is very important to identify those
problems that are indeed NP-complete
But must we consider all of the problems in NP to
make the proof?
Part 2 from of definition of NP-complete seems to
imply yes!
1. L NP
2. L
p
L for every L NP
Coping with NP-Completeness
1. Brute force - you will need additional resources (parallel
machines) and probably will still not be able to solve
2. Efficient non-deterministic machines (quantum computers?)
3. Small inputs
4. Special cases
5. Restrict the problem (parameterize the complexity)
For example, in chess there are 448 possible moves for each play. Therefore,
for n plays, there are 448
n
possible moves. If you create a game tree with each
branch labeled with a possible move, you could win the game by looking ahead
and following the branches that lead to the winning game configuration. This
would require a tree with 448
n
branches. Too big. Prune the tree by using a
heuristic to cut off unpromising branches, or look ahead a few branches at a time.

Coping with NP-Completeness
6. Use a heuristic (a rule-of-thumb, usually arising from trial-
and-error or experimentation)
- Neural networks, genetic algorithms, fuzzy logic, simulated annealing,
tabu search, hill climbing (problem of local optimum, may not be globally
optimal)
- For example, to navigate through a maze, you could use a heuristic
such as the right-hand-rule (place your right hand on the wall and always
follow the wall that your hand is touching - does not always work). Or you could
drop pebbles at intersections to mark corridors that have already been investigated.
This will always work, but you may need LOTS of pebbles

7. Randomization (Monte-Carlo techniques)
- Throw a die. Trade memory for randomness (random walk).
8. Use an approximation or greedy algorithm
- You will have to settle for a near-optimal solution.