Polynomial-Time Algorithms for Permutation Groups

Merrick Furst l
2
John Hopcroft
3
Eugene Luks

shown to include the other? We provide posit1ve

answers to each of these questions in the form of
polynomial-time algorithms.
Classical algorithmS
to solve these problems have been known to
computational group theorists' for some time. but
without accurate analyses of running times [3.7].

Abstract

A permutation group on n letters may always be

represented by a small set of generators. even
though its S1ze may be exponential in n. We show
that it is practical to use such a representation
since many prOblems such as membership testing.
equal1ty testing. and inclusion testing are
decidable in polynomial t~e.
In addition. we
demonstrate that the normal closure of a subgroup
can be computed in polynomial t~e. and that this
proceaure can be used to test a group for
solvability.
We also describe an approach to
comput1ng the intersection of two groups.
The
procedures and techniques have wide applicability
and have recently been used to improve many graph
isomorphism algorithms.

Permuta tion groups. depending upon how they

are represented.
either have polynomial-time
membership tests. or they don't. Those that do we
call polynomia1-.t.i.Dw. recoinizable.
Any group
represented by generators
is polynomial-time
recognizable. The automorphism group of a graph is
also
polynomial-time
recognizable.
even
if
generators for it are unknown, since testing
whether a permutation is an automorphism is easy.
As an important corollary we will prove that given
generators for a group G, generators for any
polynom.ial-t~e
recognizable subgroup with small
index can be found in polynomial time.

Introduction

Using these techniques, in [4] L. Babai's

probabilistic polynomial-time isomorphism test for
graphs of bounded color multiplicity [2] was
improved
to
deterministic
~olynomial
time.
O
C H0 ff mann ' s pro ba bOlO
1 1st1C O( n clogn) 1somorph 18m
test for cone graphs was improved todet~rministic

Let gI gk be pe~utations of the set

{l n}.
The
collection
of
pe~utations
expressible as finite compositions of these gi
forms a group: a group whose size may easily be
exponent1al in nand k. For example. take the set
Sn of all nJ pe~utations. It can be generated by
just two permutations. a cyclic shift of the n
letters. and a transposition of the letters I and
2.
In fact. every subgroup G of Sn can be
represented succinctly by o(logIGI> generators.

IDepartment of Computer Science

Carnegie-Mellon University
Pittsburgh, Pennsylvan1a
2

It is natural to ask, from a computational

perspecc1ve,
whether
using
such
a
short
representation of such a large collection is
practical. More to the point. is it possible to
answer bas1c questions about a group G that is
defined by a list of generators? For ex.ple, can
membership be tested? can the size be deterained?
can two groups be tested for equality? can one be

Department of Computer Science

Cornell University
Ithaca, New York

3Department of Mathematics
Bucknell University
Lewisburg. Pennsylvania
This research vas supported in part by ONR
contractN00014-76-C-0018 , and NSF grant
SPI-7914927.

36

CH1498-5/80/0000-00J6$OO.75 C 1980 IEEE

O(nc1ogn) time. and a subexponentia1 algoritnm for

trivalent graph isomorphism was described.
In
these proceedings E. Luks [6] shows that the last
two problems are decidable in deterministic
polynomial
time.
Many
other
problems
of
permucat~on groups
seem to have polynomial-time
solutions. We give two examples. an algorithm for
comput~llg the norma! closure of a subgroup. and an
algorithm to test a group for solvability.
In
addition. we describe an approach to the problem of
computing group intersections.

and moves 3 to the correct place. etc.

The
collection of elements in G /G + i=O n-l, are
i i 1
called JI..t.I.Q.Di. generators for G. Our first theorem
yields an algorithm. like the one proposed by Sims
[7]. that computes coset representatives for each
of the quotients in polynomial time.
Theorem 1: Let G be a group generated by gl gk.
Let G be the subgroup of G fixing letters 1 i.
i
Coset representatives for G /G + i=0 n-1 can
i i 1
be determined in polynomial time.
~:

The maximum value that IG./G. l' can have is

.~ .. ~+
n since cosets only differ by where they map the
letter i+l. We will construct a table T with n
rows. labelled 0 to n-l. and n columns. labelled 1
to n. whose ith row is a set of right coset
representatives for Go./G. 1.
The table will be
1
1+
organized in such a way that the permutation in the
i.jth posit10n fixes letters 1 i-l and maps
letter i to position j. Thus the entries in Twill
only l1e on or above the diagonal.

Preliminaries
permutation ~ on {I n} is a
ot 1-1 maps from {I h} onto itself
that forms a group under composition. Let G be a
group.
The ~ of G. IG I. is the number of
permutations in G.
Let H be a subgroup of G.
written G ~ H. The quotient symbol G/H stands for
the collection of cosets of H in G. i.e.. the
co!lect~on or equivalence classes of elements of G
1
in Which x
y if and only if x- y is in H. Fr~
Lagrange's theorem we know that every coset of G/H
has the same size and. therefore. IG I = IG/H I IH 1.
The s~ze ot G/H is called the ~ of H in G.
A

collect~on

When we are f in~shed. the table should have

the property that g is in G if and only if g can be
expressed as a a a - where a is a member of
i
r 1
O1
the ith row.
This we will call the canonical
representation of g.

To start. initialize T with the diagonal

elements equal. to the identity and all others
empty.
The procedure sift(x). derined below.
modifies the table by inserting at most one new
coset representative in such a way that x can be
written in canonical form.

If gl gk are permutations. then the group

<gl gk>. generated by the gi' is the group of
all permucations formed by products of the gi. We
use the symbol I
to mean the unique group
generated by the identity permutation.

Testini Membership &nd Determinini

~ ~ gf

sift(x):

i ... 0
~

(i ~ n-l and
there is a y in row i such that
y and x map i+l to the same letter)

Let G be a group of permutations on {1 n}

generated by gl, ,gk.
There is a descending
chain of subgroups. G = GO ~ G ~ ~ G = I,
1
n
from G to the identity group. in which G. is the
.~
subgroup of G fixing 1 i.
Cons ider the
quotients Gi/G + for i=O n-l.
The group G
i 1
can be expressed as
G = GO = (GO/G1)G 1
or. G = (GO/Gl)(G]/G2) (Gn_l/Gn).
Therefore. any element g in G can ce written in the
form g = a a a - where a is an element of
n 1
Ol
i
G /G +
Intuitively
this
says
that
any
i i 1
permutation in G can be realized as a permutation
that moves 1 to the correct place. followed by a
permutat~on that fixes 1 and moves 2 to the correct
place. followed by a permutation that fixes 1 and 2

i ... i + 1
-1

x'" y x
i f x is not a member if row i
~ insert x in row i

As an example of how sift works. suppose the

table for the tower GO ~ G ~ G ~ G = I at some
2
3
1
point looks like Figure 1.

37

(G =)

~1

I-L-I ~1

1_ _ 1

~1

(G =)
.1

~2

1_ _ 1

~2

1_ _ 1

~2

(G =)

~3

1_ _ 1

~3

1_ _ 1

~3

.0

.2

in canonical form.
It is important to note here
that writing xy in canonical form does not require
the introduction of any elements from lower
numbered rows than the one y comes from.
The whole algorithm can be described as
Step 1. Sift all the generators.
Step 2. Close the table such that the
product xy. for every pair (x.y)
in the table. can be written in
canonical form.

Figure 1

Consider the call sift(b) for some b in GO b~t

not in G If a-lb is in G and a -lb is riot in
1
1
G then atter the call sift(b) the table would
2
iook like Figure 2. In this table b is expressible
1
in canonical form as a(a- b)l.

To perform Step 2 simply run through all pa1rs

(x.y) from the table and sift their product. The
number of coset representatives in the table is at
most n 2 The number of calls to sift is at most
(n 2 )(n 2 ) and each call to sift takes roughly n 2
6
time.
Therefore. the running time is 0(n ). a
polynomial in n.

o
Once the coset representatives have been
found. testing membership and comput1ng the size of
G is not hard. To determine if x is an element of
G. run sift with argument x. If x can be written
in canonical form without the introduction of new
elements into the table. then x has can be written
as a product of generators. If x cannot be written
in canonical form. then x is not in G. The order
of G is the product of the sizes of Gi/G +
i I

At this point we make a key observation: all

of the coset representatives have been found if and
only if
(1) each generator can be written in
canonical form. and
(2) each product of a pair of
representatives in the table
can be written in canonical form.

The
group
G = <gl gk>
contains
H = <hI h s > if and only if each hi is a member
of G. Two groups are equal if and only if each
contains the other. Therefore. the polynomial-time
membership test gives polynomial-time inclusion and
equality tests for permutation groups.

Since we will only sift elements from G we

need only verify that when (1) and (2) are
satisfied any g in G can be written in the
canonical form. Let g be an element at G. Write g
as a proauct of generators and write each generator
in canonical form.
If this product is not in
canonical form. use (2) to rewrite it.

The following important corollary to Theorem 1

is used by E. Luks [6] in his polynomial-time.
bounded-valence graph isomorphism test.
Corollary 1: Let G be a permutation group on
{I n} generated by gI gk.
Let H be a
polynomial-time recognizable subgroup of G. whose
index in G is at most a polynomial in n.
Generators for H can be found in polynomial time.

By using (2) we can take an adjacent pair of

representatives x.y in the string representing g
and. it x comes from a higher numbered row than y.
rewrite xy in canonical form. This has the effect
of moving an element from a lower numbered row past
an element of a higher numbered row to the left in
the string representing g. Moving all the row 0
representatives to the left. then all the row 1
representatives. and so on. we can put the string

The
sequence
of
groups
G ~ H ~ HI ~ ~ H = I. where Hi is the subgroup
n
of H that fixes the letters 1 i. can be used
just as the sequence G ~ G ~ ~ G was used in
n
I
~heorem 1. with two changes.
The first is that row
o may have a polynomial number of entries instead

~:

38

of at most n. The second is that to test whether

two elements x and y of G are in different cosets
-1
of G/R. the permutation K y has to be tested for
membership in R.

o
Straight-Line Programs
By a strai~ht line prQ~ram we mean a sequence
of instructions h h in which
m
1

hj~

~-Iheoretic

Applications

Let H be a subgroup of a permutation group G

on {l n}. H is .l1QlJll.Q.l in G if g-IHg = H for
any g in G. The.l1QlJll.Q.l closure of H in G is the
smallest normal subgroup. K. of G that contains H.
To il16strate the usefulness of the sift and close
operations we present a polynomial-time algorithm
to compute generators for the normal closure of H
in G.

with j.k < i. or

Theorem 3: Let H = <hI h > be a subgroup of

r
G = <gl gk>
Geueratbrs for K. the normal
closure of H in G. can be computed" in polynomial
time.

where g. is a generator.

~:

h.

{ g .
J

Not every permutation x in G can necessarily be

expressed by a polynomial-length word in the gi'
however. our next theorem states that every x can
be expressed as the result of a short straight-line
program.
Theorem 2: Let G = <gl gk> be a group of
permutations on {l n}. There is a polynomial p
such that each permutation x in G can be computed
by a straight-line program h h = x. where
m
1
m ~ p(n).
~:

Using the polynomial-time procedure of

Theorem 1. compute a table of strong generators for
G. Bef ore the algori thm begins. form a d~rected
acycl~c graph w~th k leaves. one for each of the
generators. and no edges. As the table is built.
new products are formed. Each time a product is
formed J add a new node to the dag in such a way
that the sons of this node correspond to the
factors or the product.

Form a table T of strong generators for H

using the algorithm ot Theorem 1. In order to get
generators for K we will modify T until it is a
table of strong generators for K.

Since H is a subset of K. and K is normal in

-1
product of the form gi hg where gi is a
i
g~nerator of G and h is a coset representative from
T. should be in K. In order to achieve this we
will take every such product and sift it into T.
The following program takes T and augments it using
the sift and close steps until T has the property
that for all h in T. and for all generators g. of
G. the product g ~lhg. is expressible in canonical
~
~
:form using representatives of T.
G.

ea~h

there is an x in T
not processed by this loop

fQr ~

close T

When the procedure terminates. a polynomialsize dag will have been formed and every
permutation represented by a node in the graph will
have an obvious polynomial-length straight-line
program to compute it. Since each permutation in G
is the product ot exactly n strong generators. each
can be computed by a polynomial-length straightl~ne program.

39

-1
gi: sift(gi xg i )

The main loop is executed at most a polynomial

number of times.
Since sifting and closing are
polynomial-time operations. the whole program runs
in polynomial thne.
Let Group(T) stand for the group generated by
the permutations in the table T.
Certainly
Group(T) contains H.
A simple induction proves
that the T produced by the above algorithm contains
only permutations that are generated from products
-1
of the form x or gi xg where x is an element of
i
the normal" closure of H in G.
Therefore. if
Group(T) is normal in b. then :i:t is the smallest
normal dubgroup of G that contains H.

TO see that Group(T) is normal in G. observe

that for each gene~ator g. of G. and each generator
1
-1
x of Group(T). the product g. xg. is in Group(T).
. 1 1
Let xl x
be some of the generators, of
m
Group(T). For each generator g. of G. define x. to
-1
1.
J
be
g. x.g..
an
element
of
Group(T).
If
1
J 1
Y = x1 x2 xm' then

polynomial time.

Intersection Problem

In [4]. [5]. and [6J a relationship has been

established between certain graph isomorphism
problems and the prOblem of computing generators
for the intersection of two groups.
Given an
arbitrary pair of groups. G=<gl. .gk>' and
H=<h h > we do not know whether it is possible
1
r
to compute generators for their intersection
quickly. There are. however. certain situations in
which the intersection can be found in polynomial
time.

x .x ' an element ot Group(T).

1
m
It is not hard to see from this that if g is any
element OL G and y is any element of Group(T). then
-1
g yg is an element of Group(T).
Therefore.
Group(T) is the normal closure of H in G.

Polynomially Accessible Towers

o
The theorem proved in section 1 that computes
coset representatives for the quotient groups
G /G +
relies on four properties of the tower
i i 1
GO
~ J G = I for its polynomial running time.
_
r
They are
(i) the number of groups is polynomial in

The derived subiroup G' of a group G is

defined to be the group generated by all products
of the form a- 1 b- 1 ab. where a.b are elements of G.
The group G is called solVable if the sequence
G ~ G' ~ ~ G(i)~... terminates at I.
Solvable
groups piay an important role in the study of field
extensions and ultimately relate to the conditions
under which a polynomial equation has a solution in
radicals. Using the normal closure algorithm we
can get a polynomial-time test for solvability.

(ii) generators for G are known.

(iii) the size of G.YG. 1 is bounded by a
1
1+
polynomial in n. and
(iv) there is a polynomial-time test to
determine if a and b
from G. are from the same coset of

Theorem 4: Let G = <g1 gk> be a group of

permutations on {I n}.
In polynomial time G
can be tested for solvability.

G./G+
.1.
1. 1
Any tower that satisfies these four conditions we
call
polynomially
acsessible.
Using
this
definition we can restate the first theorem.

~:

It is a fact. which we don't prove here.

that the derived subgroup of G is equal to the
normal closure in G of the sUbfroup generated by
-1 all products of the form g. g. g. g. By forming
1
J 1 J
all such products. and computing the normal
closure. the derived subgroup of G can be computed
in polynomial time.

Theorem 5: Let GO :::). :::) G = I be a polynomially

r
accessible tower of groups. Coset representatives
for the quotients G /G + for i=O r-l. can be
i i 1
determined in polynomial time.

This allows us to prove the main theorem of

this section.

If the sequence G ~ G' ~... converges to I.

then it does sC' wif~in a polynomial number of
steps. since IG 1) IG 1+1) I ~. 2. If the sequence
doesn't converge to I. then for some polynomially
. d groups can
G( i} = G( i+1) The d er1ve
boundeQ 1.
be computed. and group equality can be tested in
polynomial t~e. Therefore. it can be determined.
in polynomial time. whether the sequence converges
to I. Bence. G can be tested for solvability in

Theorem 6: Let G and B be any two polynomial-time

recognizable groups. Let S be a group for which
generators are known. If S contains both G and H.
and there are two polynomially accessible towers.
one from S through G to- I. and the other from S
through B to I. then generators for GnB can be
found in polynomial time.

40

~:

Let
S = H ~ ~ H = H ~ H + ~
~ H =
I.
and
s l
O
r
r l
~ G 1 =I
be
S = GO ~ ~ G = G ~ G 1 ~
the two towers. p Constrl:t an s x qq-table whose
i.jth entry is the group G.nH.. Each entry is a
recogn~zable group since it~isJthe intersection of
recognizable groups.

References

[1] L. Babai, Isomorphism Testing and

Symmetry of Graphs, unpublished
manuscript.
[2] L. Babai, Monte-Carlo AlgorithmS

in Graph Isomorphism Testing,

Submitted to Siam J. of Comput~ng,
(1979)

Both

IHi/H j+l and IG i /G i +l ' are bounded by

some polynom~al p(n).
Consider the two groups
'
G.nH. and G.nH. 1. Let a and b be distinct coset
~
J
. ~ J+
r epres entat i v es of X
( G. nH . ) / ( G. nH. 1).
The
~
J
. ~ J+
elements a and b are both from the group H.
-1
J
Furthermore. it a b were an element of H. l' ~t
J+
would also be an element of G.nH.+ Since a and b
l
are from different cosets of
i~ follows that a- l b
is not in H. Therefore. a and b are distinct
J+ 1
coset
representatives
ot
H./H. 1.
Hence
J J+
I{G.nH.)/{G.nH. 1) I is less than or equal to
~
J
~
J+
IH ./H. 1 I
~
p( n)
Similarly.
1{~.nd~)/{G.+lnH.)1 ~ p(n).

[3] Curran, M.P.J., ~

.Qn ~

and Computation.
Academic Press (1977).

[4] M. Furst, J. Hopcrott, E. Luks

A sUbexponential Algorithm for

Trivalent Graph Isomorphism. Proc.
Eleventh Southeastern Conf. on
Graph Theory and Comput~ng. to appear.

[5] C. Hoffmann. Testing Isomorphism of

Let P be any path in the table beginning at S.

moving only one row down or one column across at a
t~met passing through GnH. and ending at I.
This
path P describes a polynomially accessible tower of
groups from S through GnH to I.

Cone Graphs, Proc. Twelfth Annual

ACM Symposium on the Theory
of Computing (1980).
[6J E. Luks. Isomorphism of Graphs of Bounded
Valence Can Be Tested in Polynomial Time.
Proc. 21st FOCS (1980).

For
example.
the
tower
S ~ HI ~ HZ ~ ~ H ~ G nH ~
l
~ G nH ~ GnH ~ G
nH ~ ~ I can be used to
l
PH+ .
.p
f or Gn
1
. 1 t~me.
.
get generators
~n po ynom~a

[7J C. Sims. Computational Problgms

in

Abstract Al&ebra

Ed. John Leech. Pergamon Press

(1970). pp.176-177.

A natural conclusion that we can draw from

this theorem is that many group intersection
problems lie in NPncoNP. For example. this will be
the case wnenever we know that G and H
(I) lie under a common group S, and
(Z) there exists a chain ot groups
between Sand G. and a chain
between Sand H whose indices
are'polynomially bounded.
The reason computing GnH, for such G and H, is in
NPncoNP is that one can use nondeterminism to guess
generators for S and for each ot the groups in the
chains. Towers from G to I, and from H to I can be
obtained by fixing letters. and then the algorithm
in Theoreml 5 can be used to verify the guesses.

41