COMP3310 Theory of Computation Notes - Regular Languages, Pumping Lemma

COMP3310: Theory of Computation
Tasos Viglas
tasos@it.usyd.edu.au
c Tasos Viglas, 2005
COMP3310 Theory of Computation p.1/31
COMP3310- Week 2

School of Information Technologies
Tasos Viglas, Madsen G87
Week 02: Thursday August 4, 2005
- Non-deterministic automata
- Regular expressions
Non-determinism
Computation: sequence of steps
Given the current state and input symbol, the next

state is completely determined
Non-deterministic computation: many possible next

states
example: Non-deterministic FA
empty transitions
many outgoing arrows for the same symbol
multiple ways to read through an input
try all possibilities in parallel. accept if any

computation path exists
Examples of NFAs
alphabet = {a, b}
L1 = {w|w contains aba as a substring}
L2 = {w|length of w is a multiple of 3}
L3 = L1 L2
Definition of an NFA
a non-deterministic FA is a 5-tuple
(Q, , , q0 , F ):
Q is a finite set of states
is a finite alphabet (set of symbols)
: Q ( {}) P(Q) is the transition function
q0 Q is the starting state
F Q are the final (accepting) states
(Def 1.37p53)
Two machines are called equivalent if they accept

the same language
(Th 1.39p55)
Every NFA has an equivalent FA
Proof: by construction
Proof of NFA and FA equivalence
Given an NFA N = (Q, , , q0 , F ), construct an

equivalent FA M = (Q , , , q0 , F )
First, assume there are no arrows
If the number of states in Q is k the all possible

subsets of states is 2k
Q = P(Q)
for R Q , a define
(R, a) = {q Q|q (r, a), r R}
or define (R, a) = rR (r, a)
q0 = {q0 }, and
F = {R Q |R contains a final state of N }
Proof of NFA and FA equivalence: arrows
define E(R) =
{all states reachable from R by moves only}
modify definition
(R, a) = {q Q|q E((r, a)), r R}
start state q0 = {E(q0 )}
(end of proof)
A language is regular if and only if
some NFA accepts it
(Corollary 1.40p56)
Closure under regular operations

Regular languages are closed
under union, concatenation and star
(Theorems 1.45-49p59)
Proof: by construction
Regular expressions
Assume an alphabet
(Definition 1.52p64)
a regular expression R is either:
a
(the empty string)
the empty regular ecpression
(R1 R2 ) , (R1 R2 ) , (R), where R, R1 , R2 are
regular expressions
examples
0 10 = {w|w contains exactly one 1}
1
(0 1) 1(0 1)
FAs and regular expressions

a language is regular if and only if
some regular expression describes it
(Theorem 1.54p66)
proof: by construction, two directions

Any regular expression can be
converted into an NFA
(Lemma 1.55p67)
Any DFA can be converted in to a

regular expression
(Lemma 1.60p69)
proof of the first lemma is straightforward: go

through inductive definition of a regular expression
and show how to build the NFA for every case
second lemma requires some more work
GNFA generalized NFA: arrow labels are regular
DFA regular expression
start state has arrows to every other state and no

incoming arrows
single accepting state
only one arrow between two states, no multiple

labels
DFAGNFA with k states k 1 states . . . 2

states
Remove any state and repair the machine (update

labels)
complete proof by induction (proving that the

conversion step is correct)
Non-regular languages
power of FAs, regular languages
B = {0n 1n |n 0}
C = {w|w has the same number of 0s and 1s}
D = {w|w has the same number of occurrences of

01 and 10 as substrings}
FAs cannot count
proving that languages are not regular: the pumping

lemma
Pumping lemma
If A is regular then i
there exists p (pumping length) such that,
if s is any string in A of length at least p
then s may be divided in 3 parts, s = xyz satisfying:
1. for each i 0, xy i z A
2. |y| > 0 and
3. |xy| p
(Theorem 1.70p78)
COMP3310- Week 3

Week 03: Monday August 8, 2005
- Non-deterministic automata, regular
expressions
- Non-regular languages, the pumping lemma
Regular expressions
Assume an alphabet
(Definition 1.52p64)
a regular expression R is either:
a
the empty string
the empty regular expression
(R1 R2 ) , (R1 R2 ) , (R), where R, R1 , R2 are
regular expressions
examples
0 10 = {w|w contains exactly one 1}
1
(0 1) 1(0 1)
FAs and regular expressions

a language is regular if and only if
some regular expression describes it
(Theorem 1.54p66)
proof: by construction, two directions

Any regular expression can be
converted into an NFA
(Lemma 1.55p67)
Any DFA can be converted in to a

regular expression
(Lemma 1.60p69)
proof of the first lemma is straightforward: use

inductive definition of a regular expressions
second lemma requires some more work
GNFA generalized NFA: arrow labels are regular

expressions
Generalized NFAs
a GNFA is a 5-tuple (Q, , , qstart , qaccept ), where

Q, finite set of states, and alphabet
: (Q {qaccept }) (Q {qstart }) R
qstart is the start state
qaccept is the accept state
convert DFA to a GNFA such that:

start state has arrows to every other state and no
incoming arrows
single accepting state
only one arrow between two states, no multiple
labels
DFAGNFA with k states k 1 states . . . 2

states
Remove any state and repair the machine (update

labels)
complete proof by induction (proving that the

conversion step is correct)
main step: removing states from the GNFA

q
i
R1
R4
q
rip
q
j
q
i
(R1)(R2)*(R3)+(R4)
q
R
3
R2
remove any internal state and update transition

labels
power of FAs, regular languages
B = {0n 1n |n 0}
C = {w|w has the same number of 0s and 1s}
D = {w|w has the same number of occurrences of

01 and 10 as substrings}
B and C are not regular

but D is regular (why ?)
limitation: fixed number of states

fixed amount of memory
proving that languages are not regular: the pumping

lemma
Long paths in DFAs
long strings correspond to long (computation) paths
long paths must repeat states (visit a state twice)
therefore there is a loop in the path
consider a DFA with |Q| = k that accepts a string w

of size p k
on input w the DFA will visit p + 1 states
q1 , . . . , qi , . . . , qi , . . . , qk
repeat qi . . . , qi part. same final state
The Pumping lemma

Pumping lemma
If A is regular then
there exists p (pumping length) such that,
if s is any string in A of length at least p
then s may be divided in 3 parts, s = xyz satisfying:
1. for each i 0, xy i z A
2. |y| > 0 and
3. |xy| p
(Theorem 1.70p78)
The Pumping lemma- proof
M = (Q, , , q1 , F ), Q = {q1 , . . . , qp }
let s = s1 . . . sn L(M ) with n p
computational path of M on s is
r1 , . . . , rn+1 , r1 = q1 , and rn+1 F ,
rt+1 = (rt , st ) for 1 t n
n + 1 p + 1 implies that a state repeatsj < k : rj = rk

x
! "# $! "# $ ! "# $

s1 . . . sj1 sj . . . sk1 sk . . . sn
r1 . . . rj1 rj . . . rk1 rk . . . rn rn+1
xy i z takes M from q1 to Rn+1 F for any i 0

note that |y| > 0 and |xy| p
Pumping examples -1
B = {0n 1n |n 0} (ex. 1.73p80)

assume B is regular and p is the pumping length,
and s = 0p 1p
then x, y, z : s = xyz = 0p 1p and i 0 : xy i z B
three cases for i:
1. if y = 0k then xyyz = 0p+k 1p B - contradiction
2. if y = 1k then xyyz = 0p 1p+k B - contradiction
3. if y = 0k 1l then xyyz = 0p+k 1l 0k 1p+l B contradiction
Pumping examples -2
F = {ww|w } (ex. 1.75p81)

Let p be the pumping length and consider s = 0p 10p 1
s = xyz and |xy| p. One case: y = 0k and
xyyz = 0p+k 10p 1 F
Pumping examples -3
E = {O i 1j |i j} (ex. 1.75p81)
consider s = 0p 1p . cases as first example
cases 2 and 3 still give contradictions. but pumping
up the zeroes will not give a contradiction xyyz E
Pump down s = xz = 0pk 1p
note: pumping lemma implies that all FAs accept

short strings
COMP3310- Week 3

- Algorithms for automata
Properties of regular languages
deciding properties of sets of languages can be a

very difficult problem
automata are simple enough to allow simple

algorithms
properties of regular languages: halting, pumping

lemma
Problems on FAs
acceptance (or membership): does M accept string

w?
AF A = {< M, w > |M is a FA that accepts w}
membership for regular expressions
emptiness: given A, is L(A) = ?

EF A = {< A > |L(A) = }
equivalence: do A and B accept the same

language? EQF A = {< A, B > |L(A) = L(B)}
Descriptions
< M > denotes a description of the machine M
checking if a given string is a valid DFA
encode everything as a string
description of a machine and enumeration
self-reference
Membership
theorem 4.1p166
acceptance (or membership): does M accept string

w?
AF A = {< M, w > |M is a FA that accepts w}
note: this problem deals with all possible DFAs, not

a specific instance
check the description of M
simulate M on w
Emptiness
theorem 4.4p168
emptiness: given A, is L(A) = ?

EF A = {< A > |L(A) = }
let M = (Q, , , q0 , F ) and |Q| = k
check if any final state is reachable from q0 within k

steps
note: we are using the pumping lemma property

if M accepts any string, then it must accept a short
string
Equivalence
theorem 4.5p169
equivalence: do A and B accept the same

language? EQF A = {< A, B > |L(A) = L(B)}
construct the symmetric difference
L = (L(A) L(B))
(L(A)
L(B))
note: the symmetric difference is empty iff

L(A) = L(B)
check L for emptiness
Equivalence of states
given a DFA M , two states p, q are called equivalent

if for all strings w ,
starting from state p on input w we end up to an
accept iff starting from state q on input w we end up
to an accept
if two states are not equivalent, they are called

distinguishable
testing equivalence: table filling method
Equivalence of states- 1
construct a table as follows
q1
q2
qk
q0 q1 . . . qk1
for states p, q if there exists an input symbol a such

that (p, a) = r = s = (q, a) then p, q are
disitinguishable
if not, mark the apropriate entry of the table
Equivalence of states- example

b
a,b
A
a,b
D
a
B X
C X X
D X
X
A B C
states B, D are equivalent
Equivalence of states- example

b
a,b
A
BD
a,b
Equivalence of states- application
apply previous algorithm for deciding equivalence of

two automata
are the two starting states equivalent ?
use the previous algorithm and treat both automata

as one
if their starting states are found to be equivalent,

then the automata are equivalent
Minimizing automata
(see also problem 7.40p299)
given a DFA M construct a DFA M with minimal

number of states such that L(M ) = L(M )
input < M > where M = (Q, , , q0 , A)

output < M > such that L(M ) = L(M )
remove useless states (unreachable from q0 )
find all pairs of equivalent states
partition into subsets of equivalent states
merge equivalent states into one new state
Minimizing automata- 1
proof that the output of the previous algorithm is

optimal?
minimality
Minimized DFAs are ... minimal
assume not: M is the minimized automaton from the

previous algorithm, and there exists an even smaller
one N
start states of M and N are equivalent since

L(M ) = L(N )
if p, q are equivalent then their successors on any

one input symbol are equivalent
minimality- 1
all states of M are distinguishable (same for N )
all states are reachable in both M, N
every state p of M is equivalent to some state q in N

s
s
for s such that M : p0 p then N : q0 q
N is smaller, two states of M are equivalent to the

same N state
therefore they are equivalent - contradiction
COMP3310- Week 3

- context free languages - ambiguity
Context free languages
context free grammars were proposed by N

Chomsky as a model for natural languages
CF can describe non-regular languages

context free grammars
CFL strictly contains RL
there are however non-CF languages

{0n 1n |n 0} is CF
{0n 1n 0n |n 0} is not CF
Context free- summary
context free grammars
context free languages contain all regular languages
derivations, parsing, ambiguity
Chomsky normal form
push down automata (stack automata) and CFL
non-context-free languages and the pumping lemma
Context free grammars
the language {0n 1n |n 0} can be described as

follows
start from S and apply the following rules:

1. S 0S1
2. S
we can derive any string 0n 1n as follows

S 0S1 00S11 . . . 0n S1n 0n 1n
CF grammars definition
A context free grammar is a 4-tuple G = (V, , R, S)

V : a finite set of variables
: finite set of terminals (with V = )
R: finite set of subsitution rules V (V )
S : start symbol, S V
L(G) = the language accepted by grammar G
Derivations
a single step derivation is denoted by

substitution of a variable by a string according to a
rule
a sequence of any number of derivations (including
none) is denoted by
the language L(G) contains only strings of terminals
L(G) = {w |S w}
notation: A B|01|AA
the first rule usualy refers to the start symbols
(unless stated otherwise)
CF grammars- example
S 0S1|0Z1
Z 0Z|
{0i 1j |i j}
V = {S, Z}, = {0, 1, (, ), , , }

S 0|1|(S)|(S) (S)|(S) (S)
propositional (boolean) formulas
natural languages
Parse trees
derivation of a string can be represented as a parse

tree
leftmost derivations
Ambiguity
we say that a string is derived ambiguously from a

grammar G if it has more than one leftmost
derivations
ambiguity and parse trees

example: 0 1 + 1
two different derivations do not imply an ambiguous

string
both derivations may have the same parse tree)
a CF language that can be generated only by

ambiguous grammars is called inherently ambiguous
RL and CFL
context free languages: all languages that can be

generated by a context free grammar
all regular languages are context free
{0n 1n 0n |n 0} is not context free
RL CF L
theorem: every regular language is context free

RL CF L
proof idea: given a DFA M = (Q, , , q0 , F )

construct the grammar GM = (V, , R, S)
V = Q, S = q0 and R is as follows:
qi x(qi , x), for all qi V and all x
qi , for all qi F
Chomsky Normal Form
(definition 2.8p107) a CFG is in Chomsky normal

form if all the rules have the form
A BC or
Ax
for the start variable we also allow the rule S
grammars in this form are easy to analyse
Chomsky normal form- 1
theorem 2.9p107: any CFL is generated by a CFG in

Chomsky normal form
proof: assume G = (V, , R, S)

rewrite all rules to the Chomsky format
add new start symbol and S0 S
before
after
B xAy and A
B xAy|xy
A B and B xCy
A xCy , B xCy
A B1 B2 . . . Bk
A B1 A1 , . . . , Ak2 Bk1 Bk
ill-placed terminals a
replace by Ta and add Ta a
Chomsky example
CFG S aSb|
in Chomsky normal form:

S0 |Ta Tb |Ta X
X STb
S Ta Tb |Ta X
Ta a
Tb b
COMP3310- Week 5

- push down automata
- the pumping lemma for context-free
outline
Stack automata
Equivalence with context free grammars
RL CF L
Non context free languages
The pumping lemma for context free
Some closure properties for CFL
Stack automaton
context free grammars and CF languages
Push down automata and context free

Generalization of a NFA
Allows limited extra memory
Stack automata are equivalent to CF grammars
Push down automaton
Non-deterministic automaton plus a stack
Input is still read-once, and there is no output
Two ways of accepting

final/accepting state
empty stack
both ways are equivalent
Example of a PDA
non-regular, context free language

L = {0n 1n |n 0}
Very easy to design a pda
Definition of a PDA
a push down automaton is a 6-tuple
(Q, , , , q0 , F ):
Q is the finite set of states
is the finite input alphabet
is the finite stack alphabet
: Q P(Q ) is the transition
function
q0 Q is the starting state
F Q are the final (accepting) states
(Def 2.13p111)
Example of a PDA
context free language
L = {a b c |i = j or i = k}
(Example 2.16p113)
i j k
non-determinism is needed for this language
PDAs and context free languages

A language is context free if and
only if some pushdown automaton recognizes
(Theorem 2.20p115)
two directions to prove
(Lemma 2.21p115)
If L is CF then a PDA recognizes it
If a PDA recognizes a language

then it is context free
(Lemma 2.27p119)
CF grammar to PDA
convert a CFG to an automaton
start from start symbol and guess series of

derivations
keep intermediate results on the stack
match terminals from the input
CF grammar to PDA: algorithm

1. Push tha start symbol on the stack
2. Repeat:
(a) if the top is a variable, non-det. choose a rule for it
and replace A on the stack using that rule. match
any terminals in the beginning of the rules right
part from the input.
(b) if the top is a terminal read the next input symbol
and repeat if they match. if they dont, stop
("reject")
(c) if the stack is empty accept
PDA to CF grammar
Write a grammar for a given PDA P
for every pair of states of P use a variable Apq
Grammar will generate all possible strings that take

P from p to q with empty stack
two cases: stack empties only when reaching q or it

empties in an intermediate step as well
add the rules Apq aArs b and Apq Apr Arq
RL and CFL
(Corollary 2.32p122)
All regular languages are context free
The pumping lemma for CFL

If A is a CFL then
there exists a p (pumping length) such that,
any s L with |s| p can be divided in s = uvxyz
for all i 0, uv i xy i z A
|vy| > 0
|vxy| p
(Theorem 2.34p115)
v x
u v R
R
x
y
v x y
Pumping lemma proof idea
pumping length p = b|V |+2

where b be the max number of symbols on the right
of a rule
a parse tree with height h generates a string at least

bh + 1 long or
a string of length bh + 1 must have derivation tree
height at least h + 1
length p implies height (or a path of length) |V | + 1
that path must repeat a non-terminal symbol
pumping example
(Example 2.36p126)
B = {an bn cn |n 0} is not context
free
s = ap bp cp = uvxyz
v, y contain each one type of symbol. then uv 2 xy 2 z

must be imbalanced
otherwise repeating v, y would repeat symbols out of

order
Closure properties
CFLs are closed under union and concatenation

For union: the new grammar is the union of the
previous two plus an extra start symbol and the rule
S S1 |S2
CFLs are not closed under intersection and

complement
CFLs are closed under intersection with a regular

language
run the PDA and the NFA in parallel and accept if
they both accept
COMP3310- Week 5

- Turing Machines
-
outline
Part 2: Computability theory

decidability
the halting problem
Church-Turing thesis
reducibility
This lecture:
Turing machines
multi-tape Turing machines
non-deterministic Turing Machines
Computability
regular, context-free and beyond
L = {1n | n and n + 2 are prime}
Alan Turing
Alan Turing
1912-1954, UK
Cambridge University, Princeton University
1936, universal machines and the halting problem
Alonzo Church
1903-1995, USA
Princeton University
1936, lambda calculus and undecidable problems
Turing machines
The universal machine, a model for computation
Read, write, move to a new cell, change state
Initially all the tape cells are blank
Two halting states, Accept and Reject

$ 0 1 0 1 0 1
oneway infinite tape
finite state
control
Turing machine- definition

(definition 3.3p140)
A Turing machine is a 7-tuple

(Q, , , , q0 , qaccept , qreject )
Q is the finite set of states
is the finite input alphabet, not containing 4, the

blank symbol
is the finite tape alphabet, 4 and
: Q Q {L, R} is the transition function
q0 Q the starting state
qaccept is the accept state
qreject is the reject state and qaccept = qreject
Turing machines
start with the input string on the tape

start computing...
until any of the two halting states is reached
A configuration of a TM consists of
current state, tape contents, head position
the set of strings accepted by a TM M is the

language of M or the language recognized by M ,
L(M )
Example: TM for {w#w|w {0, 1} }
Decidable problems
A language is called Turing
recognizable if some Turing machine recognizes it
A recognizer accepts all yes-inputs. may loop for

ever on no-inputs
Also called recursively enumerable

decidable if some Turing machine decides it
always gives an answer
Also called recursive
Example
(example 3.7p143)
2n
Example of a Turing machine for {0 |n 0}
from left to right, cross off every other 0
Variants of Turing machines
Multitape: a Turing machine with several tapes
Non-deterministic machines
Enumerators
Multitape Turing machines
k -tape Turing machine

: Q k Q k {L, R, S}k
(theorem 3.13p149)
A k -tape Turing machine is equivalent to a single

tape Turing machine
Proof: Simulate a multitape machine by a single

tape one
Non-deterministic Turing machines
non-determinism allows several choices for each

step of the computation
: Q P(Q {L, R})
many possible next steps
non-deterministic computation represented as a tree
Several computation paths
Computation is accepting if an accept leaf exists
A non-deterministic branch may be infinite

(theorem 3.16p150)
Every non-deterministic Turing machine has an

equivalent deterministic Turing machine
Simulate the machine deterministically
Do all possible computation paths, accept if any one

of them accepts
problem: Non-deterministic computation may

contain infinite paths
simulate in parallel (breadth first traversal of the

computation tree)
Dove-tailing
Given a non-deterministic machine N simulate it

with a 3 tape deterministic machine
tape 3 is used as an index tape for the

non-deterministic choices, pointing to a path into the
computation tree
a number 235 means take the first choice at the

root, then the third etc

1. tape 1 has the input, tapes 2 and 3 are empty
2. copy the input to tape 2
3. use tape 2 to simulate the machine N . Use the
symbols on tape 3 for the choices
if the symbol describes an unavailable choice
goto 4
if no more symbols on tape 3, goto 4
if rejecting configuration is encountered, goto 4
if accept is encountered then {accept}
4. replace tape 3 with the next (lexicographically) string
goto 2
Enumerators
An enumerator is a Turing machine that prints

(enumerates) all strings in a language
a language is Turing recognizable
if and only if it has an enumerator
(theorem 3.21p153)
Hilberts tenth problem (1900)

Find an algorithm that decides whether a given
polynomial has an integral root
.."a process by which it can be determined in a finite

number of steps"..
Matijasevic 1970: Hilberts tenth is unsolvable

Martin Davis, Hilary Putnam, Julia Robinson
Church-Turing thesis: Computable

means computable by a Turing
machine
COMP3310- Week 6

Week 06: Mon Aug 29 and Thu Sep 1, 2005
- Decidability
- Church Turing thesis
- Diagonalization
- The halting problem
Decidable problems
recognizable if some Turing machine recognizes it
A recognizer accepts all yes-inputs. may loop for
ever on no-inputs
Also called recursively enumerable

decidable if some Turing machine decides it
always gives an answer
Also called recursive
non-determinism allows several choices for each

step of the computation
: Q P(Q {L, R})
many possible next steps
non-deterministic computation represented as a tree
Several computation paths
Computation is accepting if an accept leaf exists
A non-deterministic branch may be infinite

(theorem 3.16p150)
Every non-deterministic Turing machine has an

equivalent deterministic Turing machine
Simulate the machine deterministically
Do all possible computation paths, accept if any one

of them accepts
problem: Non-deterministic computation may

contain infinite paths
simulate in parallel (breadth first traversal of the

computation tree)
Dove-tailing
Given a non-deterministic machine N simulate it

with a 3 tape deterministic machine
tape 3 is used as an index tape for the

non-deterministic choices, pointing to a path into the
computation tree
a number 135 means take the first choice at the

root, then the third etc

1. tape 1 has the input, tapes 2 and 3 are empty
2. copy the input to tape 2
3. use tape 2 to simulate the machine N . Use the
symbols on tape 3 for the choices
if the symbol describes an unavailable choice
goto 4
if no more symbols on tape 3, goto 4
if rejecting configuration is encountered, goto 4
if accept is encountered then accept
4. replace tape 3 with the next (lexicographically) string
goto 2
Enumerators
An enumerator is a Turing machine that prints

(enumerates) all strings in a language
a language is Turing recognizable
if and only if it has an enumerator
(theorem 3.21p153)
Defining Computation
Hilberts tenth problem (1900)

Find an algorithm that decides whether a given
polynomial has an integral root
.."a process by which it can be determined in a finite

number of steps"..
Matijasevic 1970: Hilberts tenth is unsolvable

Martin Davis, Hilary Putnam, Julia Robinson
Church-Turing thesis:
computable means
computable by a Turing machine
all proposed computation models are equivalent,
including
Turing machines
-calculus
Recursive functions
Quantum Turing machines
Decidable problems for regular languages

(theorems 4.1-4.5p166-169)
The following problems on regular laguages are

decidable
Does a given DFA M accept input w ?
Same for non-deterministic automata
Given a regular expression R, does it generate a
given string w ?
Given a DFA, does it accept any string?
Given two DFAs are they equivalent ?
Decidable problems for context free

(theorems 4.7-4.9p170-172)
the following problems on context free languages

are decidable
Given a CFG G and a string w , does G generate
w?
Given a CFG G is L(G) = ?
Every context free language is decidable
Equivalence is not decidable
COMP3310- Week 6

Week 06: Thu Sep 1, 2005
- Diagonalization
- The halting problem
Announce
Afternoon tea with the staff of SIT

thursday 1st september,4pm Madsen G93
Honours program in the School of IT

Information session
Tuesday 6th September, 12-3 pm
labs, demos, talk to current honours students and
staff
Undecidable problems
A specific problem that is algorithmically unsolvable
Fundamental limitation of computing
Example of an unsolvable problem:

software verification- Given a program, does it
conform to its specification?
A more simple unsolvable problem: Given a TM M

and a string w , does M halt on input w ?
In fact, (almost) "all" languages are undecidable
... and they are not even recognizable
Existence of undecidable problems
Counting infinities
How many languages L are there?

k
2k
For L there are 2 , a finite number
For L the number of langauges is infinite...
Small infinite sets and big infinite sets
Georg Cantor, 1873
Two sets have the same size if the elements of one

set can be paired with the elements of the other
Mappings and functions
A function f : A B is a mapping of the elements

of A to the elements of B
f is ...
if f (a) = f (b) whenever a = b
onto if f (A) = B
correspondence if it is both 1-to-1 and onto
one-to-one
Two sets A and B have the same size if there is a

correspondence f : A B between them.
Countable and uncountable
The set of natural numbers is N = {0, 1, 2, . . .}
A set is called countable if it is finite or it has the

same size with N
Example: Q = { m
n |m, n N }, the set of rational
numbers is countable
A set is called uncountable if it is not countable
The set of real numbers R is uncountable
Proofs: by diagonalization
Infinite binary strings
Let B = {all infinite binary strings}
B is uncountable
0 0 0 0 1 0
1 0 0 1 0 0
2 1 1 0 1 0
3 0 1 1 1 0
4 ... ... ... ... ...
...
...
...
...
...
Counting languages and TMs

(theorem 4.18p178)
Some languages are not Turing-recognizable
proof by diagonalization
the set of all TMs is countable

a TM can be described by finite a string over a
finite alphabet
arrange the strings alphabetically
Counting languages and TMs
the set of all languages over an alphabet is

uncountable
P( )
The set of all infinite binary sequences B is
uncountable
is countable (arrange alphabetically)
Every language corresponds to a characteristic
sequence from B
More on counting
Examples of countable sets: correspondence

between N and Z , N 2 , {a} , {a, b}
0
1
2
3
4
5
6
0
+1
1
+2
2
+3
3
(0, 0) (0, 1) (1, 0) (0, 2) (1, 1) (2, 0) (0, 3)
a
aa
aaa aaaa aaaaa
...
a
b
aa
ab
ba
bb
Counting N
Consider N , the set of finite sequences of numbers
N is countable,
there is a bijection between N and N
Use the unique prime factorization theorem

Every number can be written as a product of prime
factors
primes p1 = 2, p2 = 3, p3 = 5, p4 = 7, . . .
x = pe11 pe22
525 = p2 p3 p3 p4
Counting N
Consider (n1 , n2 , . . . , nk )
Map this to number
m = pn1 1 +1 pn2 2 +1 pnk k +1
+1 is used to encode zero

(1, 2, 0) versus (1, 2)
Correspondence works since there are infinitely

many primes and the unique factorization theorem
holds
Universal Turing Machine
Universal Turing Machine U is given a description of

any TM M and and input w and simulates M on w
Use 2 tapes. The first tape contains the description

of M and the input w
Write down starting < q0 w > configuration on the

2nd tape
repeat until halting configuration is reached:

replace the configuration on tape 2, according to
the transition function of M
Accept if qaccept is reached, reject if qreject is reached
The halting problem
The halting problem for TMs is undecidable

HT M = {< M, w > |M halts on input w}
proof by contradiction
Assume that HT M is decidable and Halts(M, w) is

the TM that decides it
Define the following TM:
D(M ) := if Halts(M, M ) then loop for ever

otherwise halt (accept)
What is the output of D(D)? Does it halt?
The halting problem

D(D) := if D(D) halts then loop for ever
otherwise halt (accept)
More on the halting problem

(theorem 4.11p174,179)
The acceptance problem for TMs,

AT M = {< M, w > |M accepts input w} is
undecidable
Proof by contradiction
Assume AT M is decidable and H(M, w) is the TM

that decides it
%
accept if M accepts input w
H(M, w) =
reject otherwise
Define a new machine as follows

%
accept if M does not accept input M
D(M ) =
reject if M accepts input M
What is the output of D(D) ?

M1
M2
M3
M4
... D ...
M1 accept accept reject accept . . .
M2 reject accept accept reject . . .
M3 reject reject reject reject . . .
M4 reject accept accept reject . . .
...
...
...
...
...
... ...
D reject reject accept reject
...
Un-recognizable languages
AT M is Turing recognizable
(theorem 4.22p181)
A language L is decidable if and only if L and its

complement L are both Turing recognizable
L is called co-Turing-recognizable if its
complement L is Turing-recognizable
(corollary 4.23p182)
H T M and AT M , the complements of the halting

problem and the acceptance problem, are not Turing
recognizable
Un-recognizable languages
AT M is not Turing-recognizable
but it is co-Turing-recognizable
ET M = {G|G is a TM and L(G) = } is not Turing

recognizable
EQT M = {< G, H > |TMs G, H and L(G) = L(H)}

is not even co-TM-recognizable
Classifying problems
Decidable (Recursive)
Context free
Regular
coRecognizable (coRE)
Recognizable (RE)
EQ
TM
ATM
nnn
abc
Context free
Regular
(0+1)*1
Recognizable (RE)
__
ATM
ww R
Multistack machines
theorem: a PDA with two stacks is as powerful as a

TM
proof: simulate a TM by a two stack machine
one stack holds the TM tape contents to the left of

the head and the second the contents to the right of
the head
Counter machines
a k -counter machine has access to k counters
the machine can add/subtract one from each

counter or test for zero
transition depends on state, input symbol, and which

counters are zero
equivalent definition: k -stack machine that can push

a single symbol on its stacks
what is the power of a counter machine?
Counter machines
(theorem) a three counter machine can simulate a

TM
simulate a 2 stack PDA with two counters
each counter has a numeric representation of the
stack contents
the third is used for adjusting the other two
counters
(theorem) a two counter machine can simulate a TM
Counter machines
(theorem) a two counter machine can simulate a TM

simulate 3 counters with 2
use one counter to store an encoding of the three
counters/numbers (i, j, k): store 2i 3j 5k
use the second counter for adjusting the first
incrementing i is multiplying the counter by 2 etc
COMP3310- Week 7

Week 07: Mon Sep 5, 2005
- Reductions
- More undecidability
- Rices theorem
Announce
Honours program in the School of IT

Information session
Tuesday 6th September, 12-3 pm
labs, demos,
talk to current honours students and staff
Context free
Regular
Recognizable (RE)
EQ
TM
ATM
nnn
abc
Context free
Regular
(0+1)*1
Recognizable (RE)
__
ATM
ww R
The halting problem again
Reductions between two problems

"A reduces to B"
A B , if we can solve B then we can solve A
(theorem 5.1p188)
The halting problem is undecidable
HALTT M = {< M, w > |M halts on input w}
prove by reducing the acceptance problem AT M to

HALTT M
Assume R solves HALTT M
The halting problem again
Construct S as follows:
S : On input < M, w >
1. Run R(< M, w >)
2. if R rejects, reject
3. if R accepts, simulate M on w
S solves AT M if R exists
Reductions
Proving undecidability using reductions
To prove L is undecidable, prove the following:

"if L is decidable then AT M is decidable"
Proofs by contradiction based on a construction
Given a TM for L show how you can use it to solve

AT M or any other undecidable problem
Emptiness
(theorem 5.2p189)
The emptiness problem for TMs is
undecidable
ET M = {< M > |L(M ) = }
Proof: reduce from AT M

given a TM R for ET M build one for AT M
Modify M to accept only w if it is non-empty
M1 : on input < x >

1. if x = w , reject
2. if x = w simulate M on w and accept if M (w)
accepts
Emptiness
S : on input < M, w >

1. From M construct M1
2. Simulate R on M1
S constructs M1 by adding some new states to it that

simply check for x = w
L(M1 ) = M does not accept w
L(M1 ) = {w} M accepts w
Regularity
(theorem 5.3p191)
REGU LART M is undecidable
REGU LART M = {< M > |L(M ) is regular}
proof: reduce from AT M
assume REG(M ) decides regularity.

use it to solve AT M (M, w)
S : on input < M, w >

1. Construct M2 : on input x
(a) if x is 0n 1n accept
(b) otherwise run M (w) and accept if it accepts
2. Run REG on M2
3. accept if REG(M2 ) accepts
Regularity
Construct M2 : on input x
1. if x is 0n 1n accept
2. otherwise run M (w) and accept if it accepts
M2 accepts 0n 1n if M does not accept w

M2 accepts if M accepts w
Deciding equality
(theorem 5.4p192)
EQT M is undecidable
EQT M = {< M1 , M2 > |L(M1 ) = L(M2 )}
Reduce ET M to equality
S : on input M
1. run EQT M on < M, T > where T rejects all
inputs
Other undecidable properties
is L(M )
1. context free
2. finite
3.
Rices theorem: all non-trivial TM properties are

undecidable
Rices Theorem
Let P be a non-trivial property of
the language of a TM. Prove that determining if the
language of a given TM has this property is
undecidable
(problem 5.28p213)
P (M ) = {< M > |L(M ) has property P}
non-trivial property: it contains some but not all TM

languages
P is a property of the TM language

whenever L(M1 ) = L(M2 ) then M1 P iff M2 P
Proof: assume P () decides the property P

reduce AT M to P () by constructing S
Rices theorem
let T be a TM that always rejects L(T ) =
wlog assume T P (otherwise proceed with P )
P non-trivial implies T P
use the ability of P () to distinguish between T and T
Rices theorem
define S : on input < M, w >

1. Based on M and w construct Mw (x)
(a) Simulate M (w) if it halts and rejects, reject
(b) if it accepts, simulate T on x. if it accepts,
accept
2. Use P () to determine whether Mw has property
P . if it does, accept, otherwise reject
Mw simulates T if M (w) accepts

L(Mw ) = L(T ) if M accepts w
L(Mw ) = otherwise
Other undecidable properties
is L(M )
1. context free
2. finite
3.
Rices theorem: all non-trivial TM properties are

undecidable
Reductions
Reductions so far have been straightforward
properties that involve TM
for more general questions, use different reductions
Computation histories
An accepting computation history
of M on w is a sequence of configurations C1 , . . . Ck
C1 is the start configuration of M on w
Ck is an accepting configuration
Ci M Ci+1 , a valid computation stem of M
if Ck is a rejecting configuration, then this sequence

is a rejecting computation history
computation histories are finite sequences
if M does not halt on w no computation sequence

exists
Computation histories
< M, w > AT M is equivalent to "all computation

histories of M on w are non-accepting"
< M, w > AT M is equivalent to "there exists a

computation history of M on w that accepts"
Deciding CF properties
(theorem 5.13p197)
ALLCF L is undecidable
ALLCF L = {< G >

|G is a CFG and generates all }
proof: use computation histories to reduce AT M to

this problem
AT M : does M accept w ?
AT M : does there exists an accepting computation

history for M on w ?
CFL
< M, w > AT M implies all histories x are

non-accepting
Define LCF L a CFL that contains all accepting

histories x of M on w
< M, w > AT M translates to x LCF L ?
a CFG G can describe all accepting histories
x = (C1 , . . . Ck ) G iff
1. C1 is not proper start configuration, or
2. there is an invalid Ci Ci+1
3. last configuration in x is not accepting
CFL
ALLCF L = {< G >

|G is a CFG and generates all } is undecidable
LCF L describes all strings that fail to be accepting

histories
if M does not accept w then all strings fail to be

accepting histories and therefore LCF L =
CFL equality
theorem:
EQCF L = {< G1 , G2 >
|G1 , G2 are CFG and L((G1 ) = L(G2 )}
proof: let G = . decide < G > is in ALLCF L by

EQ(G, G )
COMP3310- Week 7

- Post correspondence problem (PCP)
- ambiguity in CFGs
Post correspondence problem
An undecidable problem concerning manipulations

of strings
a domino contains two strings (top and bottom)

&a'
bc
a collection of dominos
() * & ' & ' )
*+
b
a
ca
abc
,
,
,
ca
ab
a
c
PCP: given a collection of dominos find a match, a

sequence (repetitions allowed) such that the top
string is the same as the bottom
Post correspondence problem
a collection of dominos
() * & ' & ' )
*+
b
a
ca
abc
,
,
,
ca
ab
a
c
PCP: given a collection of dominos find a match, a

sequence (repetitions allowed) such that the top
string is the same as the bottom
& a ' ) b * & ca ' & a ' ) abc *
ab ca
a ab
c
PCP
for some collections, a match does not exist
Post correspondence problem: determine whether a

given collection of dominos has a match
,& ' & '
& 't1
t2
tk
collection: P =
,
,
.
.
.
,
b1
b2
bk
a match is a sequence i1 , i2 , . . . , in such that

ti1 ti2 tin = bi1 bi2 bin
the PCP language is

P CP = { |a PCP problem that has a match}
PCP is undecidable
(theorem 5.15p200)
PCP is undecidable
proof: reduce from AT M using computation histories
given M and w , construct the dominos to be parts of

the computation history of M on w
these dominos will form a match only of an

accepting computation history exists
start with M P CP , the modified &PCP:

' the match
must start with the first domino
t1
b1
PCP is undecidable
#q0 w1 w2 wn # is the starting configuration of M

on w
&
'
#
put #q0 w1 w2 wn # as the first domino
need to start from this domino- need to match the
bottom part at the top
this forces the next domino to have a certain kind of

top part
that domino will have a bottom part that is the next

configuration of M (w)
PCP is undecidable
. qa /
(q, a) = (r, b, R)
br /
. cqa
(q, a) = (r, b, L)
.rcb
/
a
a
& ' a& '
#
#
copy #, add a blank
,
#
&
' &4#
'
aqaccept
qaccept a
a
,
qaccept
qaccept
PCP is undecidable
last step: build in the problem the requirement to

start with the first domino
for u = u1 u2 un define
u = u1 u2 un
u = u1 u2 un
u = u1 u2 un
,& ' & '
& 't1
t2
tk
convert
,
,
.
.
.
,
b1
b2
bk
,& ' & ' & '
& ' & 't1
t1
t2
tk
to
,
,
,
.
.
.
,
,
b1
b1
b2
bk
Mapping reducibility
Many different kinds of reductions
Mapping or many-one reducibility

a function f : is a
computable function if some TM on input w halts with
f (w) on its tape
A is mapping reducible to B
written A m B if there is a computable function
f : such that for all w
w A f (w) B
The function f is called the reduction from A to B
Reducibility
if A m B and B is decidable then A is decidable

A m decidable
if A m B and A is undecidable then B is

undecidable
undecidable m B
if A m B and B is Turing recognizable then A is

T-recognizable
if A m B and A is not T-recognizable then B is not

T-recognizable
A m B is the same as A m B
Equality of TM languages
EQT M is neither T-recognizable
nor co-T-recognizable
(theorem 5.30p210)
prove two things:

EQT M not T-recognizable AT M m EQT M
EQT M not co-T-recognizable
EQT M not T-recognizable AT M m EQT M
AT M m EQT M
AT M reduces to EQT M
F : on input M, w
1. construct two machines
M1 = reject all inputs
M2 = run M (w) if it accepts, accept
2. output < M1 , M2 >
M (w) accepts
M1 nothing, M2 everything
M (w) does not accept M1 nothing, M2 nothing
AT M m EQT M
AT M reduces to EQT M
G: on input M, w
1. construct two machines
M1 = accept all inputs
M2 = run M (w) if it accepts, accept
2. output < M1 , M2 >
M (w) does not accept M1 nothing, M2 everything

M (w) accepts
M1 nothing, M2 nothing
Ambiguity for CFGs
Determining whether a CFG is ambiguous is

undecidable
proof: reduce from PCP
two strings of integers/indices (viewed as strings)
strings correspond to PCP instances, one list for the

top part one for the bottom part of the dominos
each string can be generated by a grammar
if the PCP instance has a solution, then there are

two derivations of the same string (ambiguity)
Ambiguity for CFGs
& ' & '

& '
a PCP instance bt11 , bt22 , . . . , btkk
Construct a grammar GA that generates all possible

top strings
A t1 Aa1 |t2 Aa2 | |tk Aak
A t1 a1 |t2 a2 | |tk ak
same for the bottom strings, GB

B b1 Ba1 |b2 Ba2 | |bk Bak
B b1 a1 |b2 a2 | |bk ak
a string in GA is of the form ti1 tin ti1 ui1 ui2 uin
Ambiguity for CFGs
a string in GA is of the form ti1 ti2 ti1 ai1 ai2 ain
first part of derived strings: PCP strings
tails: indices of used dominos
Both grammars: derivations are unique and different
construct the union of the two grammars GAB by

adding S A|B
Ambiguity for CFGs
theorem: GAB is ambiguous if and only if the

corresponding PCP problem has a solution
PCP has a solution, there is a match

ti1 ti2 ti1 ai1 ai2 ain = bi1 bi2 bi1 ai1 ai2 ain
indices (tails) indicate two derivations of the same

string (through GA and GB )
if GAB is ambiguous the is a string that has two

derivations
two derivations must go through GA and GB
strings correspond to a match of the PCP problem
COMP3310- Week 8

Week 08: Sep 12-16, 2005
- Recursion theorem
- Self reference
Recursion theory
mathematical logic and computability
proving undecidability:
self reference leading to a paradox
Self-reference
Is it possible for a (finite) object to contain a

description of itself ?
Can you have a picture that contains itself in it ?
A finite to way to describe infinite detail
M printing < M >
design a TM SELF that prints its own description
or, write a program in Java that prints out itself
a straightforward approach would cause an infinite

recursion
define Pw the TM that ignores its input and simply

prints the string w and then halts.
denote the description of Pw by q(w) =< Pw >
M printing < M >
design the machine SELF to have two parts, A and

B
such that < SELF >=< AB >
think of A printing out B and B printing out A
define A to be P
A prints out 
we cannot define B as P<A>

this would give a circular definition and infinite
recursion
M printing < M >
SELF : run A first, then B
A is P . writes on the tape
B prints out < A >, the description of a machine

that prints .
what is ??
 is written on the tape after A finishes
therefore is: read w from the tape, which is a

TM computation description, and write down the
description of the TM that prints w concatenated
with w
this completely defines B and therefore A
M printing < M >
A = P
B = on input < M >

1. compute q(< M >), the description of P<M >
2. combine it with < M > to get a TM and print its
description
The program SELF
A = P = P<1.compute q(<M >)>
B = on input < M >

1. compute q(< M >), the description of P<M >
2. combine it with < M > to get a TM and print its
description
A writes =< 1.compute q(< M >) > on

the tape
B reads its own description, and writes

< P > or
P<1.compute q(<M >)> < 1.compute q(< M >) >
self-reference "without" self-reference
self-reference:
print out this sentence
"this" is a self-reference
no self-reference:
print out two copies of the following, the second in quotes:
"print out two copies of the following, the second in

quotes:"
self-reference is built-in the program
mixing description and meaning
Recursion theorem
let t : be a
computable function. There is a TM R that computes
r : such that
(theorem 6.3p220)
r(w) = t(< R >, w)
SELF was an application of the recursion theorem

t(x, y) = x
proof: similar to SELF construction
R has three parts A, B, T

A = P<BT > , on input w , print out w < BT >
B obtains a description of < ABT >=< R >
writes < R, w > on the tape and runs T
Using the recursion theorem
recursion theorem implies that you can use

"obtain own description"
in your algorithms or TM design
SELF = on any input

1. obtain own description < SELF >
2. print out < SELF >
"obtain own description" is implemented by the

recursion theorem
for SELF program, use T : on input < M, w > print

<M >
recursion theorem
to make a TM that obtains its own description

design a machine T that takes this description as
extra input
then produce a new machine R
R operates exactly like T
and has Rs description filled in automatically
Any computation t(, ) has a corresponding TM R
r(w) = t(< R >, w)
More on the recursion theorem
Effective enumeration of computable functions
1 , 2 , . . . , m , . . .
S-m-n theorem: for every computable function

= m (n, x) there exists a computable function S
such that Sm,n (x) = m (n, x)
theorem: for each computable function f there exists

i N such that
(n)
(n)
i = f (i)
using the recursion theorem

(theorem 6.5p222)
AT M is undecidable
proof: by the recursion theorem
define B = on input w
obtain own description 
use AT M on < B, w > and
if B(w) halts, loop for ever
otherwise, halt
Minimal TMs
A TM M is called minimal if there is no TM

equivalent to M with shorter description < M >
(theorem 6.5p222)
M INT M is not TM-recognizable
proof: use the recursion theorem
assume E enumerates M INT M
define C = on input w
1. obtain own description < C >
2. run the enumerator E until a machine D appears
with longer description than C
3. simulate D on w
Minimal TMs
M INT M is infinite, must contain a longer TM than C
C always simulates D so it is equivalent to it
C is shorter that D so D is not minimal
contradiction: D shouldnt be on the enumerators list
A fixed point theorem
x is a fixed point of f if f (x) = x
TM encodings: assume that any string that is not a

proper TM encoding, corresponds to a TM that
rejects everything
let t : be a computable
function. Then there is a TM F such that
t(< F >) =< G > and G is equivalent to F .
(theorem 6.8p223)
A fixed point theorem

let t : be a computable
function. Then there is a TM F such that
t(< F >) =< G > and G is equivalent to F .
(theorem 6.8p223)
define F = on input w
1. obtain own description < F >
2. compute t(< F >) =< G >
3. simulate G on w
clearly < G > and < F > are equivalent
Rices theorem
All non-trivial properties P of languages of TMs are

undecidable
P is non-trivial: exist < A > P and P
proof using the recursion theorem
Assume X decides P
define R = on input w :
obtain own description < R >
using the recursion theorem
Run X on < R >
if X accepts, simulate B , else simulate A
COMP3310- Week 8

- Logical theories
- Models and proofs
Logical theories
is a mathematical statement true or false?
is this problem decidable?
is a mathematical statement provable?
truth versus proof
decidability of a statement depends on the

mathematical domain
Formulas and propositions
define a precise language to use for mathematical

statements
symbols, syntax
define a model, a way to interpret the language

symbols and
meaning of the symbol sequence
formulas
Examples
qpx, y[p > q (x, y > 1 xy = p)]
a, b, c, n[(a, b, c > 0 n > 2) an + bn = cn ]
qpx, y[p > q (x, y > 1 (xy = p xy = p + 2))]
Language definition
alphabet
{, , , (, ), , }
Variables x, y, z, w, . . . , x1 , x2 , . . .
Relation symbols R1 , R2 , . . .
Constants and functions can be represented as
relations
boolean operations, connectives , ,
Quantifiers ,
Formulas
atomic formula
Ri (xi , . . . , xj )
A well formed formula or just formula

is an atomic formula
has the form 1 2 or 1 2 or 1 where i
are formulas
has the form x[1 ] or x[1 ]
Formulas
scope of the quantifier x[1 ] is 1
Prenex normal form: all quantifiers at the front
free variable: a variable that is not within the scope

of any quantifier
sentence: formula with no free variables
Formulas and meaning
So far, formulas are well formed sequences of

symbols
To add meaning to them, need to define an

interpretation
a model defines the meaning of the formula by

providing
universe: a set of values for the variables
relations: assign a meaning to the relation
symbols
M : {U, P1 , P2 , . . . Pk }
a model is also called a structure or an interpretation
Examples of models
consider = xy[R1 (x, y) R1 (y, x)]
let M = (N , )
a b or b a for all natural numbers
this model makes the formula true
the model M = (N , <) would make the formula

false
the definition of truth is given by induction
Definition of truth
Simple case: no quantifiers, no predicates

these formulas are called propositional formulas
truth assignment : X {T RU E, F ALSE}

assigns values to all variables
extend to all formulas inductively

for example () = T RU E iff () = F ALSE
If we have quantifiers, then a model must be defined
Quantifiers over variables, first order logic
a formula is true in model M

the interpretation of using M is true
Theory of M
definition: the theory of M, denote T h(M) is the

collection of all true sentences in the language of
that model
is a given theory decidable?
decidable theory:
the set T h(M) must be decidable
T h(M) is a set of formulas
given a formula, is it a member of T h(M)?
in other words, given a formula, is there a way to
decide if it is true?
decision procedure for proving theorems in a

theory/model
Decidable theories
Only relatively weak theories are decidable
(theorem 6.12p227)
T h(N , +) is decidable
proof:
FAs can (kind of) add numbers
(see problem 1.32p88)
many FA properties are decidable
given a sentence = Q1 x1 Qk xk []
define i = Qi+1 xi+1 Qk xk []
0 = and k =
i has i free variables
Decidable theories
for i = k, . . . , 0 construct Ai , the FA that recognizes

the strings representing i-tuples of numbers that
make i true
Ak can be constructed directly. use Ai to construct

Ai1
once we get A0 , test if it accepts the empty string

which means that was true
Decidable theories
Constructing the automata

for Ai : assume all free variables have been fixed to some
values from the universe set (x1 , . . . , xi ) = (ai , . . . , ai )

Ak corresponds to an atomic formula (single addition or
some boolean operation)

constructing Ai from Ai+1 : i = xi i+1
non-deterministically guess the correct value for xi = ai
and use Ai+1

i = xi i+1 = xi i+1
A0 accepts any input if is true
if is false, no input will be accepted by the

automata
An undecidable theory
(theorem 6.13p229)
T h(N , +, ) is undecidable
proof: reduce from AT M
encode the TM computation as a formula using plus

and times to ensure correctness of the computation
Given a TM M and a string w we
can construct M,w in the language of T h(N , +, )
with a single free variable x, and
(lemma 6.14p229)
xM,w is true, iff M accepts w
An undecidable theory
Given a TM M and a string w we
can construct M,w in the language of T h(N , +, )
with a single free variable x, and
(lemma 6.14p229)
xM,w is true, iff M accepts w
M,w describes that x is a valid accepting

computation history of M on w
use computation histories in a form that can be

checked by only +,
Gdels incompleteness theorem
truth versus proof
what is a proof?
what was this (N , +, ) business?

what is + ?
addition and multiplication is given by a set of axioms
axioms for natural numbers
The natural numbers can be defined by the Peano

axioms
natural numbers with the successor, (N , 0, S)

1. Every natural number a has a successor, denoted
by Sa
2. there is no natural number whose successor is 0
3. distinct natural numbers have distinct successors:
if a = b, then Sa = Sb
4. If a property P is true for zero, P (0) and also
whenever P (x) then also P (Sx) then P (x) for all
numbers x
axioms for natural numbers
in a different language
x(0 = Sx)
xy((x = y) (x = y))
(P (0) x(P (x) P (Sx))) xP (x)
axioms and proofs for arithmetic
natural number axioms, axioms for addition and

multiplication (commutativity, associativity etc)
inference rules: given two valid (true) formulas, a

new formula must be valid
modus ponens: A and A B then we can infer B
a proof of is a sequence of formulas

s1 , s2 , . . . , sk = such that every formula is either an
axiom, or it follows from previous formulas by
applying an inference rule of the system
proof is a syntactic notion
Proving all truths
Given an axiomatic system (model, axioms,

inference rules)
Completeness: prove all true sentences
Soundness: prove only true sentences
does there exist a sound and complete axiomatic

system for arithmetic?
Gdels incompleteness theorem: No.
Gdels incompleteness theorem
There does not exist any axiomatic theory that

proves exactly the true statements of arithmetic
arithmetic: (N , +, )
Any axiomatizable theory of arithmetic that is sound

is necessarily incomplete
about arithmetic
Provable statements in
T h(N , +, ) is Turing-recognizable
(theorem 6.15p230)
proof: algorithm P accepts input if it is provable

test every string as a potential proof of
Some true statement in
T h(N , +, ) is not provable
(theorem 6.16p231)
proof: by contradiction. construct an algorithm that

decides if any proposition is true (contradiction with
theorem 6.13)
about arithmetic
Some true statement in
T h(N , +, ) is not provable
(theorem 6.16p231)
on input
run P the theorem recognizer of 6.15 in parallel
for both and .
at least one of them will finish
soundness of the system implies that we can
decide the validity of
about arithmetic
There is a sentence u in
T h(N , +, ) that is unprovable
(theorem 6.17p231)
proof: construct a sentence that says "this sentence

is unprovable"
use the recursion theorem for the self-reference
define S = on any input
1. obtain own description < S >
2. construct u = c[S,0 ]
using lemma 6.14
3. run theorem recognizer P on
4. if it accepts, accept. If it halts and rejects, reject
about arithmetic
u is true iff S does not accept 0
if S finds a proof of then S accepts 0, giving a

contradiction since then would be false and
provable (impossible since the system is sound)
so S fails to find a proof of
S does not accept 0, and therefore is true
Turing reducibility
mapping reducibility: is it possible to define more

general (strong) reducibilities?
yes, Turing reducibility
Reducibility: given the algorithm for one problem

design a new algorithm for a different problem
algorithm for a problem is used as a sub-procedure
Turing reductions
An oracle for language B is an external capability to

answer questions x B ?
An oracle Turing machine is a modified TM that has

access to an oracle
Denote a TM M with access to an oracle B as M B
the oracle answers any query about B
A is Turing reducible to B written A T B if A is

decidable relative to B
Example: ET M T AT M
(example 6.19p232) ET M T AT M
ET M is decidable relative to AT M
given an oracle for AT M we can solve ET M
T AT M = on input M
1. construct N = on any input
(a) run M in parallel on all strings
(b) if M accepts any string accept
2. Query the oracle to determine whether N, 0 is in
AT M
3. if NO accept, if YES reject
Turing reducibility
(theorem 6.21p233)
if A T B and B is decidable then
A is decidable
If A m B then A T B
Information
some strings seem to be more interesting than

others
0101010101010101010101010101. . .
0100110101000111010010111101. . .
why is the string more boring ?
carries less information
some strings can be defined by a short description
some strings look random, have no patterns
Descriptions
what is a description?
we consider a description of a string x

a precise algorithm to produce the string
a TM with some seed input that produces the
string
a string may have several descriptions
the shortest possible description is a good

characterization of the string
Encodings
the description of a string x will be < M, w > for

some TM M , which on input w outputs x
encodings of TM
1. many ways to encode a TM
2. interested in efficient descriptions, so good
encodings will make a difference
3. use this encoding < M > w
Minimal descrptions
the minimal description of x
denoted d(x) is the shortest string < M, w > such
that M (w) outputs x. Define the descriptive or
Kolmogorov complexity of x as
K(x) = |d(x)|
(theorem 6.24p236)
cx[K(x) |x| + c]
proof: choose < M, x > where M does nothing
Descriptive complexity
(theorem 6.25p236)
cx[K(xx) K(x) + c]
proof: minimal description < M > d(x) where

M on input < N, w > runs N (w) to get an output
s
and prints out ss
cx, y[K(xy) 2K(x) + K(y) + c]
proof: < M > d(x)d(y)
better encoding of machine descriptions:

instead of < M with bits doubled> 01w use
(length of M s description)< M > w
K(xy) 2 log(K(x)) + K(x) + K(y) + c
Optimality of the definition
maybe a different definition of descriptive complexity

would be better?
consider any p : and dp (x) the

corresponding minimal description of x using p
define Kp (x) = |dp (x)|
for example we could choose as p Java programs

dJ (x) is the shortest java program (in binary) that
outputs x
Optimality of the definition

(theorem 6.27p238)
for any description p
cx[K(x) Kp (x) + c]
and c depends only on the choice of p
proof: choose < M, w > where M outputs p(w)
Incompressible strings
x is c-compressible if
K(x) |x| + c
if x is incompressible by 1 then it is incompressible

(theorem 6.29p239)
incompressible strings of every
length exist
proof: count
binary strings of length n: 2n
number of descriptions of length up to n 1
0
2i = 2n 1
0in1
at least 2n 2nc+ + 1 strings of
length n are incompressible by c
proof:
strings of length n: 2n
descriptions of length up to n c: 2nc+1
incompressible strings have similar properties to

random strings
we say that a property holds for almost all strings if

the fraction of strings of length n that do not have
this property goes to zero for large n
next theorem: every computable property that holds

for almost all strings, also holds for all long enough
incompressible strings
if a computable property f holds
for almost all strings then for any b > 0 only a finite
number of b-incompressible strings do not have this
property
(theorem 6.31p240)
proof idea:
define machine M that on input i finds the i-th
string s that fails f
x can be described as < M, ix > where i is the
index on the above fail-list
the description is short because almost all strings
have the property f and therefore the fail-list is
short
finding incompressible strings is an undecidable
determining whether a given string is incompressible

or not is undecidable
it is however possible to prove the existence of

incompressible strings
Density of primes
how many numbers less than n are primes ?
let p1 , p2 , . . . , pm be all primes n
we know that
n = pe11 pe22 pemm

and the prime factorization is unique
< e1 , e2 , . . . , em > is a unique description of n
also all ei log n
the description is m log log n bits
incompressibility theorem: there are n with

K(n) log n
therefore m
log n
log log n
COMP3310- Week 9

Week 9: Sep 19 and 22, 2005
- Introduction to Complexity theory
- Polynomial time
Complexity theory
main question:
computability: what is computable ?
complexity: what is computable ?
efficient computation
classify problems according to their computational

requirements
time, space, randomness
complexity of the problem, and not the complexity of

a specific solution/algorithm
Time and space
Time and space complexity
how many steps does it take to solve a problem?
how much memory is required to solve a problem?
use the Turing machine model to count resource

requirements
Context free
Regular
Recognizable (RE)
Complexity
many Turing machine models: single tape,

multi-tape, non-deterministic etc.
a single tape TM simulates all of them, but with an

overhead
develop a model-independent way to measure

resource requirements
and therefore characterize the complexity or difficulty

of the problem and not
the power of a particular machine
efficiency of a specific algorithm
Compare different problems: easy problems, hard

problems
Abstract from the input size
time complexity: how many steps?
Example: is x N prime?
we would like to characterize the difficulty of determining
primality
independent of size of the number x
abstract the complexity of determining primality from the fact
that you are looking at a big numbers primality property

a large size input makes all problems more difficult anyway
difficulty "per unit input length"
time complexity as a function of input length
Asymptotics
asymptotics (big-Oh notation) play a crucial role
abstract complexity becomes robust and model

independent for asymptotic complexity
sorting n numbers might require

5n log n + 1270 steps on a deterministic two tape
TM
120n log n + 110 on a single tape deterministic TM
but it takes O(n log n) on (almost) any TM
Running time
Let M be a TM that halts on all inputs

the running time of M is the
function f : N N where
f (n) = max|x|=n {no. of steps of M on x}
we say M is a f (n)-time TM or
the running time of M is f (n)
Big Oh
let f, g : N R+ . f (n) = O(g(n))
if positive constants c, n0 exist such that for every
integer n n0
f (n) cg(n)
g(n) in an asymptotic upper bound for f (n)

asymptotic: modulo some initial ns and the constant
c
remember, we are interested in the (growth of the)

function, and not its absolute value
some examples
O(1), nO(1) , O(n + m), . . .
small-oh
let f, g : N R+ . f (n) = o(g(n)) if
f (n)
lim
=0
n g(n)
f (n) = o(g(n)) if for any c > 0 there exists n0 such

that for every integer n n0
f (n) < cg(n)
O(g) = { all functions asymptotically less or equal to g}
o(g) = { all functions asymptotically strictly less than g}
both notations used: f O(g) and f = O(g)
Omegas, ,
Ohs were defined using less than
and therefore are used for denoting upper bounds

(for example worst case running times)
corresponding notation is used for lower bounds
(g), (g)
Analyzing running times
example: consider a single tape TM accepting the

language {0k 1k }
we can do this in O(n2 )
is this the real complexity of the problem

or just a not-so-clever solution ?
how about O(n log n) ?
Analyzing running times
checking {0k 1k }
repeat
1. check odd-even parity of all 0s and 1s.
if odd reject
2. cross-off every other 0 and every other 1
until no 0s or no 1s remain
if no 0s and no 1s remain accept. otherwise reject
algorithm checks if all parities agree for 0s and 1s
T IM E(t(n))
for t : N R+ , define T IM E(t(n))
to be the collection of all languages that are
decidable by an O(t(n))-time TM
for example the problem of checking if an input has

the form L = {0k 1k } is in T IM E(n log n)
a two tape machine can solve L in O(n)
but for a single tape TM L T IM E(n)
Model dependencies
the running times and the classes of languages

T IM E(t) are model dependent
T IM E(n) contains different languages for single

tape TMs and different for 2 tapes
define model independent classes of languages
"robust" complexity classes
Complexity across models
L = {0k 1k } is in
T IM E(n log n) for single tape deterministic TMs
T IM E(n) for two tape deterministic TMs
for t(n) n any t(n) time multi-tape
TM can be simulated by a single tape t2 (n) machine
(theorem 7.8p254)
proof: use the simulation of multi-tape machines by

a single tape as mentioned in theorem 3.13p149
every step of the multi-tape machine requires at
most O(t(n)) steps in the single tape simulation
total running time t2
non-deterministic computation
recall the notion of a computation tree for

non-deterministic computations
Every t(n) step non-deterministic
single tape computation has an equivalent 2O(t(n))
deterministic single tape computation
(theorem 7.9p256)
proof: use simulation from theorem 3.16p150

computation tree of height t, fan-out b
explore (simulate) all tree paths breadth first
every node can have at most b children (b is
determined by the machines transition function)
number of nodes is less than twice the number of

leaves, so O(bt(n))
to visit a new node, start from the root down the

non-deterministic path- at most t(n) steps
total time O(tbt ) = 2O(t)
Polynomial time
defining robust model independent complexity

classes
P is the class of languages that

decidable in polynomial time on a single tape
deterministic TM
1
P =
T IM E(nk )
Robustness: P is invariant for all polynomially

equivalent models of computation
P is a reasonable characterization of problems that

we can solve efficiently
Strong Church Turing thesis
All reasonable computation models are polynomial

time/space equivalent
It is possible to simulate a model with a machine

from another model in polynomial time/space
Unreasonable models
Unreasonable models may have unrealistic

requirements (for our current knowledge)
analog computing and infinite precision, unbounded

parallel computing
what about non-determinism ?
what about quantum computing ?
Problems in P
given a graph, is there a path connecting a given

pair of its vertices?
P AT H P
assume graph is given by its adjacency matrix
on input G, s, t do the following

1. mark node s
2. repeat the following until no new nodes are
marked
3. scan all edges of G. if there is an edge (a, b)
where only a is marked then mark b as well
4. if t is marked accept otherwise reject
Problems in P
(theorem 7.16p262)
Every context free language is in P
proof: use dynamic programming

recall decidability of CFL, where we used
Chosmky normal form
that yields an exponential algorithm for CFLs
use a more clever dynamic programming
algorithm that runs in polynomial time
CFLs in P
CFG in Chomsky normal form, and a string w

(length n)
construct an n n matrix A where
the entry aij contains all grammar variables that can

produce the substring wi wi+1 wj
start from all 1 symbol sub-strings of w and fill in the

table
for a k + 1 substring of w , consider all ways to split it

into two substrings bc (k ways)
is there a derivation A BC such that B can

produce b and C can produce c ? then add A in the
table
COMP3310- Week 9

- P , polynomial time
- N P non-deterministic polynomial time
- Tutorial: Cook-Levin theorem proof
Introduction to Complexity theory
complexity theory: classifying problems
resource requirements (time, space)
model independent characterizations, when possible
classes of difficult problems, easy problems etc
Polynomial time
defining robust model independent complexity

classes
P is the class of languages that

decidable in polynomial time on a single tape
deterministic TM
1
P =
T IM E(nk )
Robustness: P is invariant for all polynomially

equivalent models of computation
P is a reasonable characterization of problems that

we can solve efficiently
Problems in P
given a graph, is there a path connecting a given

pair of its vertices?
P AT H P
assume graph is given by its adjacency matrix
on input G, s, t do the following

1. mark node s
2. repeat the following until no new nodes are
marked
3. scan all edges of G. if there is an edge (a, b)
where only a is marked then mark b as well
4. if t is marked accept otherwise reject
Relatively prime numbers
Checking whether two numbers are relatively prime
two numbers are called relatively prime if they do not

have any common divisors (except 1)
RELP RIM E = {< x, y > |x, y are relatively prime}
(theorem 7.15p261)
RELP RIM E P
proof: use the Euclidean algorithm for the greatest

common divisor
note that a straightforward approach does not yield a

polynomial algorithm
brute-force is exponential: try all smaller numbers to

see if there is any non-trivial divisor
The Euclidean algorithm
The Euclidean algorithm finds the greatest common

divisor of two natural numbers
on input (x, y)
1. repeat until y = 0
(a) x = x mod y = x xy y
(b) swap x and y
2. Output x
every loop cuts a number by half (at least)
at the beginning of each iteration x > y
if x/2 y then x mod y < y x/2
if x/2 < y then x mod y = x y < x/2
Problems in P
(theorem 7.16p262)
Every context free language is in P
proof: use dynamic programming

recall decidability of CFL, where we used
Chosmky normal form
that yields an exponential algorithm for CFLs
use a more clever dynamic programming
algorithm that runs in polynomial time
CFLs in P
CFG in Chomsky normal form, and a string w

(length n)
construct an n n matrix A where
the entry aij contains all grammar variables that can

produce the substring wi wi+1 wj
start from all 1 symbol sub-strings of w and fill in the

table
for a k + 1 substring of w , consider all ways to split it

into two substrings bc (k ways)
is there a derivation A BC such that B can

produce b and C can produce c ? then add A in the
table
The class N P
Brute-force search sometimes lead to inefficient

algorithms
Can we always avoid brute-force search ?
N P , a class of problems
many practical problems of similar complexity do not

seem to allow any efficient solution
Polynomial solutions are not known, but it is also not

known whether those problems actually require
more than polynomial time
Hamiltonian paths
given a directed graph G and two of its vertices s, t,

is there a path connecting s with t that visits all
vertices exactly once?
HAM P AT H = {(G, s, t)|G is a directed graph, with

a Hamiltonian path from s to t}
brute-force algorithm will work. check all possible

path from s to t
it is an open problem whether a polynomial solution

exists
On N P
many other problems seem to resist polynomial time

solutions
many of these problems are strongly related: if you

solve one you have a solution for all of them
HAM P AT H : two interesting properties

1. no poly-time algorithm known
2. a correct solution can be verified in poly-time
in other words, the problem seems hard to solve but

easy to verify
Solving versus verifying
is it easier to solve a problem or just verify the

solution to problem?
this question can be formalized in complexity

theoretic terms
verifiers and poly-time verifiable problems
HAM P AT H , COM P OSIT ES
what about primality ? is it poly time verifiable?
primality is now known to be in P
Verifiers and certificates
consider a problem A and the associated decision

problem x A?
sometimes it is possible to present proof of

membership or a certificate that proves that a given
xA
A verifier for a language A is an
algorithm V where
A = {w|V accepts(w, c) for some string c}
a polynomial verifier for A runs in time poly. in w
if A has a polynomial verifier, then it is called

polynomially verifiable
Verifying certificates
the verifier uses the information of the certificate c
note that for polynomial verifiers, the certificate must

be polynomial in length
examples of certificates
HAM P AT H : a directed path
COM P OSIT ES : a divisor
what about primality?
Definitions of N P
N P is the class of languages that
have polynomial time verifiers
remember: N P is a polynomial time class
A language is in N P iff it is decided

by some non-deterministic poly-time Turing machine
(theorem 7.20p266)
N P stands for non-deterministic polynomial
P : languages with poly-time deciders
N P : languages with poly-time verifiers
N P and non-determinism

(theorem 7.20p266)
proof: show how to convert a certificate to a NTM

and vice versa
: assume the verifier V runs in time nk
to decide w A, on input w (of length n

1. non-deterministically select the certificate c of
length at most nk
2. run V on (w, c)
3. if V accepts, accept, o.w. reject
N P and non-determinism

(theorem 7.20p266)
proof: show how to convert a certificate to a NTM

and vice versa
: assume N is the NTM that decides A
construct a verifier as follows: on input (w, c)

1. simulate N on (w, c). use every symbol of c to
pick which non-deterministic choice of N we
should follow each time (remove
non-determinism)
2. if c leads down a path to an accept, accept; ow.
reject
non-deterministic time
N T IM E(t(n)) =
{L|L is a language decided by a O(t(n)) NTM }
NP =
N T IM E(nk )
non-determinism
views of non-determinism:
1. several choices, computation tree
2. ability to guess: guess (a poly-length string that
defines a path down the tree to an accept) and
verify that your guess was correct
3. for input w , c : P olyCheck(c, w)
P does not need the quantifier. for input w,
P olyCheck(x)
Examples of problems in N P
CLIQU E : given a graph, does it contain a k -clique?
HAM P AT H : given a graph does it contain a

hamiltonian path ?
SU BSET SU M : given a collection of numbers

and a target t, does there exist a subset with sum
equal to t ?
SU BSET SU M = {(S, t)|S =
2
{s1 , . . . , sk } and for some {y1 , . . . , yl } S
yi = t}
what about the complements of these sets ?
what kind of certificate can you give for something

that does not exist?
The P versus N P question
is P = N P ?
the greatest unsolved problem in theoretical

computer science
acknowledged as one of the greatest open problems

in contemporary mathematics
it is most widely believed that P = N P
is it easier to verify a solution to a problem than it is

to find that solution?
Boolean Satisfiability
consider a boolean formula over boolean variables

and the usual boolean operations, and, or, not.
for example = (x y) (x z)
literal x, x, clause (x y)
conjunction: (x y), disjunction: (x y)
conjunctive normal form or cnf: an AND of ORs

conjunction of disjunctions
example of cnf: = (x y) (x z)
given a boolean formula does there exist an

assignment of 0,1 to the variables, that makes the
entire formula true?
if a satisfying truth assignment exists for then is

called satisfiable
SAT = {| is a satisfiable boolean formula}
SAT N P
Reducibility
recall the notion of mapping-reducibility, A m B
A is mapping reducible to B
written A m B if there is a computable function
f : such that for all w
w A f (w) B
The function f is called the reduction from A to B
extend the definition for poly-time mapping

reductions
Poly-time reducibility
a function f : is a
poly-time computable function if some polynomial time
TM on input w halts with f (w) on its tape
A is poly-time mapping reducible

to B written A P B if there is a poly-time
computable function f : such that for all w
w A f (w) B
The function f is called the polynomial time
reduction from A to B
this is also called polynomial time many-one

reducibility
Comparing problems and their complexity
is SAT more difficult than CLIQU E ?
(theorem 7.32p274)
3SAT is poly-time reducible to
CLIQU E
if we can solve CLIQU E then we can solve 3SAT
proof: reduce 3SAT to CLIQU E
N P -completeness
A language B is N P -complete if it
satisfies two conditions:
1. B N P
2. all A in N P polynomially reduce to B :
A N P : A P B
N P -complete is a class of languages so we can also

say or write B N P -complete
Proving N P -completeness
(theorem 7.35p276)
if B is N P -complete and B P
then P = N P
(theorem 7.36p276)
if B is N P -complete and
1. B P C and
2. C N P
then C is N P -complete
proving N P -completeness: reduce from a known

N P -complete problem
How do we prove the first N P -complete problem?
N P -completeness
N P -completeness was introduced independently by

Cook and Levin (70s)
the first N P -complete problem, SAT
the Cook-Levin theorem: SAT is N P -complete

(theorem 7.27p267)
proof: use computation histories: construct a

formula such that an accepting computation exists iff
the formula is satisfiable
show how to reduce any A N P to a SAT question
all we know is that A N P and therefore the only

thing we can use is that a NTM computation exists
3SAT
3SAT is N P -complete
proof: any SAT formula can be converted to a

3SAT that is up to polynomially longer
use the distributive laws of boolean algebra to write

the Cook-Levin reduction formula as a cnf
big clauses (more than 3 literals) can be split into

smaller ones by introducing new variables
COMP3310- Week 10

Week 10: Thu Oct 6, 2005
- N P -completeness
- Tutorial: N P -completeness proofs
The class N P
Brute-force search sometimes lead to inefficient

algorithms
Can we always avoid brute-force search ?
N P , a class of problems
many practical problems of similar complexity do not

seem to allow any efficient solution
Polynomial solutions are not known, but it is also not

known whether those problems actually require
more than polynomial time
Hamiltonian paths


brute-force algorithm will work. check all possible

path from s to t
it is an open problem whether a polynomial solution

exists
On N P
many other problems seem to resist polynomial time

solutions
many of these problems are strongly related: if you

solve one you have a solution for all of them
HAM P AT H : two interesting properties

1. no poly-time algorithm known
2. a correct solution can be verified in poly-time
in other words, the problem seems hard to solve but

easy to verify
Definitions of N P
N P is the class of languages that
have polynomial time verifiers
remember: N P is a polynomial time class

(theorem 7.20p266)
N P stands for non-deterministic polynomial
P : languages with poly-time deciders
N P : languages with poly-time verifiers
Verifiers and certificates
consider a problem A and the associated decision

problem x A?
sometimes it is possible to present proof of

membership or a certificate that proves that a given
xA
A verifier for a language A is an
algorithm V where
A = {w|V accepts(w, c) for some string c}
a polynomial verifier for A runs in time poly. in w
if A has a polynomial verifier, then it is called

polynomially verifiable
non-deterministic time
N T IM E(t(n)) =
{L|L is a language decided by a O(t(n)) NTM }
NP =
N T IM E(nk )
consider a boolean formula over boolean variables

and the usual boolean operations, and, or, not.
for example = (x y) (x z)
literal x, x, clause (x y)
conjunction: (x y), disjunction: (x y)
conjunctive normal form or cnf: an AND of ORs

conjunction of disjunctions
example of cnf: = (x y) (x z)
given a boolean formula does there exist an

assignment of 0,1 to the variables, that makes the
entire formula true?
if a satisfying truth assignment exists for then is

called satisfiable
SAT = {| is a satisfiable boolean formula}
SAT N P
Poly-time reducibility
a function f : is a
poly-time computable function if some polynomial time
TM on input w halts with f (w) on its tape
A is poly-time mapping reducible

to B written A P B if there is a poly-time
computable function f : such that for all w
w A f (w) B
The function f is called the polynomial time
reduction from A to B
this is also called polynomial time many-one

reducibility
Comparing problems and their complexity
is SAT more difficult than CLIQU E ?
(theorem 7.32p274)
3SAT is poly-time reducible to
CLIQU E
if we can solve CLIQU E then we can solve 3SAT
proof: reduce 3SAT to CLIQU E
N P -completeness
A language B is N P -complete if it
satisfies two conditions:
1. B N P
2. all A in N P polynomially reduce to B :
A N P : A P B
N P -complete is a class of languages so we can also

say or write B N P -complete
Proving N P -completeness
(theorem 7.35p276)
if B is N P -complete and B P
then P = N P
(theorem 7.36p276)
if B is N P -complete and
1. B P C and
2. C N P
then C is N P -complete
proving N P -completeness: reduce from a known

N P -complete problem
How do we prove the first N P -complete problem?
N P -completeness
N P -completeness was introduced independently by

Cook and Levin (70s)
the first N P -complete problem, SAT
the Cook-Levin theorem: SAT is N P -complete

(theorem 7.27p267)
proof: use computation histories: construct a

formula such that an accepting computation exists iff
the formula is satisfiable
show how to reduce any A N P to a SAT question
all we know is that A N P and therefore the only

thing we can use is that a NTM computation exists
3SAT
3SAT is N P -complete
proof: any SAT formula can be converted to a

3SAT that is up to polynomially longer
use the distributive laws of boolean algebra to write

the Cook-Levin reduction formula as a cnf
cell start move accept
big clauses (more than 3 literals) can be split into

smaller ones by introducing new variables
Cliques
a clique is a complete graph
The clique problem: given a graph G and a number

k , does G contain a sub-graph of k nodes that form
a clique?
CLIQU E is N P -complete
CLIQU E is in N P and
3SAT reduces to CLIQU E ,
3SAT P CLIQU E
Vertex cover
given a graph G, a vertex cover is a subset of

vertices such that every edge in G connects to a
vertex in the vertex cover
decision version: does G contain a vertex cover of

size k ?
(theorem 7.44p284)
V ERT EXCOV ER is
N P -complete
reduce from 3SAT
Hamiltonian paths


(theorem 7.46p286)
HAM P AT H is N P -complete
reduce from 3SAT
Undirected Hamiltonian path

(theorem 7.55p291)
U HAM P AT H is N P -complete
reduce the directed version to the undirected
Sub-set sums
SU BSET SU M : given a collection of numbers

and a target t, does there exist a subset with sum
equal to t ?
SU BSET SU M = {(S, t)|S =
2
{s1 , . . . , sk } and for some {y1 , . . . , yl } S
yi = t}
(theorem 7.55p291)
SU BSET SU M is N P -complete
reduce from 3SAT
2SAT
2SAT P
COMP3310- Week 11

Week 11: Mon Oct 10, 2005
- Space complexity
- Savitchs theorem
Space complexity
characterization of problems in terms of

space/memory requirements
measuring space: use the Turing machine model
space behaves "better" that time
space can be re-used
Space complexity
The space complexity of a
deterministic TM M is the function f : N N where
f (n) is the maximum number of tape cells that M
uses on any input of length n
the definition also requires that M halts on all inputs
Space complexity
the space complexity classes
SP ACE(f (n)) and N SP ACE(f (n)) are defined as
follows:
SP ACE(f (n)) = {L| there exists a deterministic
TM that decides L in space f (n)}
N SP ACE(f (n)) = {L| there exists a
non-deterministic TM that decides L in space
f (n)}
Example
solving SAT brute-force
given a formula on the variables x1 , x2 , . . . , xn , try

all assignments to the variables and evaluate on
each assignment
what is the space complexity of this algorithm ?
Example
solving SAT brute-force
given a formula on the variables x1 , x2 , . . . , xn , try

all assignments to the variables and evaluate on
each assignments
what is the space complexity of this algorithm ?
the space complexity is linear: re-use space
Space and time
f -space computation may run for f 2O(f ) time steps

at most
cannot run for more than that because it would

repeat a configuration and therefore lead to an
infinite loop
Savitchs theorem
comparing the power of deterministic and

non-deterministic space
for any f (n) n,
N SP ACE(f (n)) SP ACE(f 2 (n))
(theorem 8.5p306)
non-determinism gives little extra power in terms of

space complexity
the equivalent problem for time is the P versus N P

question
proof: simulation of a NTM deterministically.

main idea: re-use space
Savitchs theorem
straightforward simulation does not work
f -space computation may go for 2f time steps
we can simulate all possible non-deterministic

branches but that requires remembering all
non-deterministic choices
space requirements would be 2f worst case.
Proof of Savitchs theorem
consider the yieldability (or reachability) problem:

given a NTM N , input w , configurations c1 , c2 and a
number t, can c1 yield c2 in t steps?
if we have a way of solving this problem in limited

space, then we can simulate a NTM N :
given N and input x, is it possible for the start

configuration c1 to yield the accept in the max
possible number of steps ?
CAN Y IELD(c1 , c2 , t) :
1. t = 1, test c1 = c2 or c1 leads to c2 directly (check
N s transition function)
2. for each configuration cm of N on input w
3. run CAN Y IELD(c1 , cm , t/2)
4. run CAN Y IELD(cm , c2 , t/2)
5. if such a mid-point is found, accept
6. otherwise reject
analysis of space requirements of the simulation
we need space for the recursion (stack)
t starts as 2f (n) and is halved at every recursive call
so the depth of the recursion is O(log t) or O(f (n))
each level of the recursion needs to store c1 , c2 , t on

the stack. that requires O(f (n) space
total space O(f 2 (n))
technical problem: we do not know f (n) beforehand.

try all possible values, reusing space
P SP ACE
P SP ACE is the class of
languages that are decidable in polynomial space on
a deterministic TM
P SP ACE = k SP ACE(nk )
define N P SP ACE , the non-deterministic version of

P SP ACE
P SP ACE = N P SP ACE by Savitchs theorem
define EXP T IM E as deterministic exponential

time
P N P P SP ACE = N P SP ACE EXP T IM E

P SP ACE -completeness
a language B is called
P SP ACE -complete if it satisfies two conditions:
1. B P SP ACE
2. for every A P SP ACE , A P B
if B satisfies only (2) then it is called P SP ACE -hard
more complexity theory
week 11: space complexity,

P SP ACE -completeness, games, L and N L
week 12: hierarchy theorems and diagonilization,

oracles and the polynomial time hierarchy
week 13: review (no tutorial time)
COMP3310- Week 11

- P SP ACE completeness
- L, N L, Immerman-Szelepscnyi theorem
On N P
graph theoretic properties and N P
first order logic

how do you express reachability in first-order logic?
existential second order logic P where is

first-order
Fagins theorem
the class of all graph-theoretic properties expressible

in existential second order logic is precisely N P
P SP ACE
P SP ACE is the class of
languages that are decidable in polynomial space on
a deterministic TM
1
P SP ACE =
SP ACE(nk )
define N P SP ACE , the non-deterministic version of

P SP ACE
P SP ACE = N P SP ACE by Savitchs theorem
let EXP T IM E denote det. exponential time
P N P P SP ACE = N P SP ACE EXP T IM E

P SP ACE -completeness
a language B is called
P SP ACE -complete if it satisfies two conditions:
1. B P SP ACE
2. for every A P SP ACE , A P B
if B satisfies only (2) then it is called P SP ACE -hard
Quantified formulas
SAT is an N P -complete problem
involves boolean formulas, but no quantifiers
Quantified boolean formulas: boolean formulas with

existential or universal quantifiers
examples:
x(x + 1 > x)
xy[(x y) (x y)]
all variables quantified: fully quantified formulas or

sentences
fully quantified formulas are either true or false
T QBF
True quantified boolean formula T QBF
given a fully quantified boolean formula, decide

whether it is true or false
T QBF = {| is a true fully quantified boolean

formula}
T QBF is P SP ACE -complete

(theorem 8.9p311)
proof:
T QBF P SP ACE : try out all possible
assignments, reusing space
all of P SP ACE reduces to T QBF : encode the
simulation of any P SP ACE computation by a
formula
T QBF P SP ACE
here is a polynomial space algorithm T ()for T QBF

on input , T ():
1. if has no quantifiers, just evaluate it
2. if is x , call T () once with x = 0 and once
with x = 1 accept if any of the two accepts
otherwise reject
3. if is x , call T () once with x = 0 and once
with x = 1 accept if both of them accept,
otherwise reject
space required: depth of the recursion is equal to

the number of variables, and constant space for
each recursive call
T QBF is P SP ACE -hard
let A be in SP ACE(nk ), decided by a TM M .

reduce it to T QBF as follows
the reduction will map any string w to a formula

that is true iff M accepts w
construction similar to the Cook-Levin theorem

formula
start cell move accept
formula cannot be used in the same way as in

Cook-Levin theorem: the running time may be
exponential (the computation tableau is too big)
solution: break up formula into parts and represent

each part with the same subformula plus quantifiers
T QBF is P SP ACE -hard
construct a formula c1 ,c2 ,t
c1 ,c2,t = m1 [c1 ,cm , 2t cm ,c2, 2t ]
reduce formula size

c1 ,c2,t = m1 (c3 , c4 ) {(c1 , m1 ), (m1 , c2 )}[c3 ,c4, 2t ]
use x[(x = y x = z) . . .] instead of

x {y, z}[. . .]
Other P SP ACE -hard problems
other P SP ACE -complete problems include games

and finding winning strategies in games
example: game of GO in an n n board

variant of GO (bounded moves and some simplified
rules)
Logspace
sublinear space bounds, log n
a machine can read the entire input, but does not

have enough space to store it
modify the Turing machine model to allow a

read-only input tape, plus a working tape
the space bound applies on the working tape only
log-space is an class that contains interesting

problems and has robustness properties under
model and input encoding variations
L and N L
L is the class of languages decidable in

deterministic logarithmic space
L = SP ACE(log n)
N L is the class of languages decidable in

non-deterministic logarithmic space
N L = N SP ACE(log n)
Is there a path ?
the reachability problem on directed graphs is in N L
for TM with a read only input tape, define the

configuration the TM on an input w to include the
contents of the work tape, the state, and the
positions of all head pointers (including the input
tape pointer)
an f (n)-space machine may have at most n2O(f (n))

configurations
with this definition, Savitchs theorem works for any

space bound f (n) log n
Completeness and reductions
characterizing L and N L, using completeness
(open) question L = N L?
reducibility: polynomial time reducibility is not useful,

since all N L problems are reducible to one another
poly-time reductions are too powerful to reveal

interesting properties within N L
use log-space reducibility
log-space reducibility
a log -space transducer is a TM
with a read only input tape and a write-only output
tape that works in O(log n) space. the transducer
computes a function f :
f is called a log-space computable function
a language A is called log space reducible to B

written A L B if A is mapping-reducible to B by a
log space computable function
N L-completeness
a language B is N L-complete if
1. B N L
2. every A N L is log-space reducible to B
if only the second property holds, then B is called

N L-hard
if any N L-complete problem is in L then N L = L
Path is N L-complete
(theorem 8.25p325)
PATH is N L-complete
proof: construct a directed graph that represents the

computation of a log-space computation
NL P
N L versus co-N L
the following is considered a surprising result
(theorem 8.27p327)
N L = co-N L
proof: show that P AT H is in N L
Immerman-Szelepscnyi theorem: For reasonable

s(n) log n, N SP ACE(s(n)) = co-N SP ACE(s(n))
Complexity classes
the known relationships among some classes
L N L = co-N L P P SP ACE
we know that N L = P SP ACE
we believe that all these containments are proper
COMP3310- Week 12

Week 12: Mon Oct 17, 2005
- hierarchy theorems
- probabilistic and approximation algorithms
Complexity classes
P , efficiently solvable problems
N P , problems with short yes-certificates
complexity within P ? or N P ?
Hierarchies
easy problems, hard problems, very hard problems,

etc
L P N P P SP ACE
P versus N P problem: is there a language in N P

that is not in P ?
Time hierarchies
deterministic time hierarchy
recall the definition of DT IM E(t(n))
consider DT IM E(n) and DT IM E(n2 )
are these two classes equal ? is there a problem

that can be solved in time n2 but no n-time algorithm
exists for it?
two ways to prove such a result:

1. present a specific problem solvable in time n2 but
not in n
2. prove the existence of such a problem
Constructibility
a function t : N N, where
t(n) = (n log n) is called time-constructible if it is
computable in time O(t(n))
in these definitions computable means that we can

compute the value t(n) from input 1n
a function f : N N, where
t(n) = (log n) is called space-constructible if it is
computable in space O(f (n))
Time hierarchy theorem

for any time constructible function
t : N N a language A exists that is decidable in
time O(t(n)) but not in time o(t(n)/ log t(n))
(theorem 9.10p341)
proof by diagonalization: construct a language that

differs with all languages decidable in time
t(n)/ log t(n) but is computable in time t(n)
construct a machine D that takes input w of the form

< M > and simulates M on w for t(n) steps.
if M finishes in that time, then do the opposite
D must operate efficiently: it must always operate in

time t(n) and must avoid all languages decidable in
time less than t/ log t
the simulation introduces a log-factor overhead (for

time)
(space hierarchy theorem only introduces a constant

overhead)
D on input w:
1. let n be the length of w
2. compute t(n) and store the value t/ log t in a
counter. decrement this counter before every TM
step of 3,4,5
3. if w is not < M > 10 reject
4. simulate M on w
5. if M accepts then reject. if M rejects, accept
time hierarchy
for any two real numbers
0 1 2
DT IM E(n1 ) DT IM E(n2 )
P EXP T IM E
Space hierarchy
for any space constructible function
f a language A exists that is decidable in space
O(f (n)) but not in o(f (n))
(theorem 9.3p337)
space hierarchy is more tight (recall the extra

log-factor of the time hierarchy theorem)
proof by a diagonalization construction, similar to the

time hierarchy theorem
simulation introduces only a constant overhead for

space
space hierarchy
(corollary 9.4p339)
for any two real numbers
0 1 2
SP ACE(n1 ) SP ACE(n2 )
(corollary 9.4p339)
N L P SP ACE
therefore T QBF is not in N L
non-deteministic hierarchies
similar time and space hierarchies exist for

non-deterministic complexity classes
proofs are more complicated
Dealing with hardness
N P contains practical but hard problems
efficient solutions are not known. how do we cope?
probabilistic and approximation algorithms
Probabilistic algorithms
randomness is a tool that may add power to a

computation
yet another open question: compare the two classes

polynomial time, P
polynomial time plus random choices, RP
a poly-time probabilistic algorithm, runs in

polynomial time, and gives an answer. but it may
make a mistake with small probability
trading time efficiency for a probability of a mistake
example: polynomial identity
Approximation algorithms
think of optimization problems, or in general,

problems whose solution is more than a yes-no
answer
example: find the smallest vertex cover or the

biggest clique in a graph
an approximation algorithm will find a solution, but

possibly not the best one
an approximation algorithm always comes with a

guarantee of the quality of the solution
example: minimum cuts
More complexity
randomization, pseudo-randomness
interactive proof systems
one-way functions, cryptography
parallel complexity, circuit complexity
lower bounds
COMP3310- Week 12

- P versus N P
- Average case complexity
Problems in N P
networks and systems: min-cost routing, network

optimization, scheduling, load balancing
programming: program verification, optimization,

compilers
mathematical programming (general optimization)
Dealing with hardness
N P contains practical but hard problems
efficient solutions are not known
how do we deal with N P -completeness?
probabilistic and approximation algorithms
Probabilistic algorithms
randomness is a tool that may add power to a

computation
yet another open question: compare the two classes

solvable in poly time
solvable in poly time, and the solution is usually
correct
a poly-time probabilistic algorithm, runs in

polynomial time, and gives an answer. but it may
make a mistake with small probability
trading time efficiency for a probability of a mistake
example: polynomial identity
Approximation algorithms
think of optimization problems, or in general,

problems whose solution is more than a yes-no
answer
example: find the smallest vertex cover or the

biggest clique in a graph
an approximation algorithm will find a solution, but

possibly not the best one
an approximation algorithm always comes with a

guarantee of the quality of the solution
example: minimum cuts
Worst case hardness
SAT is N P -complete, and therefore seems to be

hard in the worst case
assume that P = N P . how often is SAT hard?
hard instances versus easy instances
average case complexity
P versus N P
P versus N P is a very general problem
four worlds, depending on P = N P

Russell Impagliazzo, FOCS 1995
1. P = N P , most things easy to solve
2. most things easy on the average
3. there are hard on the average problems, but no
one-way functions
4. one-way functions exist
Algorithmica
P = N P (or N P BP P )
we can automatically produce optimal solutions from

the description of a problem
practical stuff
VLSI design, layout optimization
programming: no need to describe how to do
things, just describe what the output should be
machine learning, AI, natural languages
proofs for any theorem
Algorithmica
Cryptography is impossible in this world
any code can be broken, easy to learn any crypto

algorithm and decipher
cannot keep an algorithm secret, a machine can

learn the algorithm given enough plain-crypto text
examples
identification is possible only by physical

measurements
any remotely accessible info is essentially public
to show that we are in algorithmica, give an efficient

algorithm for an N P -complete problem
Heuristica
N P contains hard in the worst case problems, but

easy on average
somewhat paradoxical world
hard instances exist, but finding them is an

intractable problem
this world is in some sense indistinguishable from

algorithmica
but still there are significant differences
Heuristica
example: VLSI, circuit minimization

heuristica guarantees that your algorithms will work
well for most circuits. but we want an algorithm that
works well on most minimal circuit descriptions.
what is that distribution like?
note that the circuit minimization problem is above

N P , in the poly-time hierarchy p2
Heuristica
to show we are in heuristica, give an efficient

algorithm for an average-case complete problem on
the uniform distribution
Pessiland
problems are hard on the average, but one-way

functions do not exist
easy to generate hard instances, but hard to

generate hard solved instances
generic methods of problem solving would fail on

almost all domains
how do you use hard problems for cryptography?

not clear
to prove we are in pessiland, give an average case

lower bound for some problem in N P
crypto
one-way functions exist, and public-key cryptography

is possible
(it is not easy to formalize secret-key agreement

protocols and public-key protocols)
any crypto-task can be done easily
privacy, anonymity
closest to the real world, as far as we know
to show we are in crypto, we need to prove that a

particular secret key exchange protocol is secure

COMP3310 Theory of Computation Notes - Regular Languages, Pumping Lemma

Загружено:

Сведения о документе

Оригинальное название

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

COMP3310 Theory of Computation Notes - Regular Languages, Pumping Lemma

Загружено:

Авторское право:

Доступные форматы

COMP3310: Theory of Computation

c Tasos Viglas, 2005

COMP3310 Theory of Computation p.1/31