SPECIALARTICLE

THE CATEGORY-PARTITION METHOD FOR

SPECIFYING AND GENERATING
FUNCTIONAL TESTS

A method for creating functional test suites has been developed in zohich a
test engineer analyzes the system specification, writes a series of formal test
specifications, and then uses a generator tool to produce test descriptions
from which test scripts are written. The advantages of this method are that
the tester can easily modify the test specification when necessary, and can
control the complexity and number of the tests by annotating the tests
specification with constraints.

THOMAS J. OSTRAND and MARC J. BALCER

The goal of functional testing of a software system is to
find discrepancies between the actual behavior of the
implemented system’s functions and the desired behav-
ior as described in the system’s functional specification.
To achieve this goal requires first, that tests be exe-
cuted for all of the system’s functions, and second, that
the tests be designed to maximize the chances of find-
ing errors in the software. Although a particular
method or testing group may emphasize one or the
other. these two aspects of testing are mutually comple-
mentary, and both are necessary for maximally produc-
tive testing. It is not enough merely to “cover all the
functionality”: the tests must be aimed at the most vul-
nerable parts of the implementation. For functional
testing, this implies testing boundary conditions, special
cases, error handlers, and cases where inputs and the
system environment interact in potentially dangerous
ways. Conversely, it is not enough to write excellent,
error-exposing tests for only some of a system’s func-
tions, or to write tests aimed only at certain types of
errors or certain characteristics of a specification or im-
plementation.
A preliminary version of this paper appeared as “Machine-Aided
of Software Tests,” in the Proceedings of the Fifth Annual Pacific Northwest
Software Quality Conference, Portland, Oregon, October 19-20, 1987.
0 1988 ACM OOOI-0782/88/0600-0676 $1.50
Functional tests can be derived from the software’s
specifications, from design information, or from the
code itself. All three test sources provide useful infor-
mation, and none of them should be ignored. Code-
based tests relate to the modular structure, logic, con-
trol flow, and data flow of the software. They have the
particular advantage that a program is a formal object,
and it is therefore easy to make precise statements
about the adequacy or thoroughness of code-based
tests. Design-based tests relate to the progra:mming ab-
stractions, data structures, and algorithms used to con-
struct the software. Specification-based tests relate di-
rectly to what the software is supposed to do, and
therefore are probably the most intuitively appealing
type of functional tests.
A standard approach to generating specification-
based functional tests is first to partition the input do-
main of a function being tested, and then to select test
data from each class of the partition. The idea is that all
elements within an equivalence class are essentially
the same for the purposes of testing. If the testing’s
main emphasis is to attempt to show the presence of
errors, then the assumption is that any element of a
class will expose the error as well as any otber one. If
Production the testing’s main emphasis is to attempt to give confi-
dence in the software’s correctness, then the assump-
tion is that correct results for a single element in a class
will provide confidence that all elements in the class
would be processed correctly.

676 Communications of the ACM June 1988 Volume 31 Number 6


Various methods of creating a test partition are dis-
cussed in the literature [l, 5, 6, 9, lo]. Despite the
general agreement on this key technique of functional
testing, and its use since the earliest days of computing,
the partitioning process lacks a systematic approach.
According to Howden [i’],
. . . there have traditionally been no firm guidelines
for the application of the method [of functional
testing] . . . Dissatisfaction with the fuzziness of
functional testing was partly responsible for the de-
velopment of systematic approaches to testing
which may be inadequate in terms of the errors
they are capable of discovering, but which have
the advantage that they are well defined.
In this article we describe a systematic, specification-
based method that uses partitioning to generate func-
tional tests for complex software systems. Our ap-
proach, called the category-partition method, includes the
use of formal test specifications and is supported by a
generator tool that produces test case descriptions from
test specifications. The method goes through a series of
decompositions, starting with the original functional
specification, and continuing through the individual
details of each subprogram being tested.
The main characteristics of the category-partition
method include the following:
A. The test specification is a concise and uniform
representation of the test information for a func-
tion.
B. The test specification can be easily modified, if
this is necessitated by
l changes in the functional specification of a
command;
l mistakes in an original test specification;
l a desire for more or fewer test cases.
C. The test specification gives the tester a logical
way to control the volume of tests.
D. The generator tool provides an automated way to
produce thorough tests for each function, and to
avoid impossible or undesirable combinations of
parameters and environments.
E. The method emphasizes both the specification
coverage and the error detection aspects of
testing.
A STRATEGY FOR TEST CASE GENERATION
Although formal requirements and specification lan-
guages [2, 31 are becoming more widely used, many
software specifications are still written in natural lan-
guage. These documents are frequently wordy and un-
structured, making them difficult to use directly as a
basis for functional testing. The tester must transform
such a specification into some more concise and struc-
tured intermediate representation, from which test
cases and test scripts can then be generated.
Not only do natural language specifications lack
structure, but they are frequently incomplete, ambigu-
ous, or self-contradictory. The process of transforming
]une 1988 Volume 31 Number 6
Special Section
the specification into an intermediate representation
can be useful for revealing such problems. Frequently,
the tester has to ask the specification writer to clarify
the intention of a particular section or sentence. These
questions are themselves a form of testing, as they may
expose errors in the specification before any code is
written or in the implementation before any code is
executed.
The first steps of the category-partition method help
to accomplish specification coverage and analysis. The
later steps provide information used for generating
error-detecting tests. The method begins with the de-
composition of the functional specification into func-
tional units that can be tested independently. A func-
tional unit can be either a top-level user command or a
function described in the specifications that is called by
other functions of the system. For a very large or com-
plex specification, this decomposition may require sev-
eral stages, beginning with the identification of major
subcomponents that can be executed independently,
and leading eventually to the individual functional
units. The process is very similar to the high-level de-
composition done by software designers. In fact, a care-
fully done software design decomposition could serve
as the basis for the test decomposition, and could save
considerable duplication of effort. An advantage of an
independent test decomposition, however, is that it pro-
vides a distinct and fresh point of view of the specifica-
tion. Independent decomposition is highly recom-
mended whenever error detection is a high priority
goal, since it greatly increases the chances of finding
problems in the specification, and also increases the
chances that the resulting decomposition will lead to
more effective tests for the software.
After the functional units of a subcomponent have
been identified, the next decomposition step is to iden-
tify the parameters and environment conditions that affect
the function’s execution behavior. Parameters are the
explicit inputs to a functional unit, supplied by either
the user or another program. Environment conditions
are characteristics of the system’s state at the time of
executing a functional unit.
The actual test cases for a functional unit are com-
posed of specific values of the parameters and environ-
ment conditions, chosen to maximize the chances of
finding errors in the unit’s implementation. To select
these values, the next step in the decomposition pro-
cess finds categories of information that characterize
each parameter and environment condition. A category
is a major property or characteristic of a parameter or
environment condition. Selecting the categories is done
by careful reading of the specification. For each param-
eter or environment condition, the tester marks phrases
in the specification that describe how the functional
unit behaves with respect to some characteristic of the
parameter or environment condition. Each characteris-
tic that can be identified in this way is recorded as a
category. Ambiguous, contradictory, or missing descrip-
tions of a function’s behavior are frequently discovered
during this process.
Communications of the ACM 677
Special Section
Deriving the categories is the final step of analyzing
the specification for coverage purposes. The next de-
composition step is to partition each category into dis-
tinct choices that include all the different kinds of val-
ues that are possible for the category. Each choice
within a category is a set of similar values that can be
assumed by the type of information in the category.
The choices are the partition classes from which repre-
sentative elements will be taken to build test cases.
The concepts of category and choice can be illus-
trated with an example of a sorting program specifica-
tion. The specification describes the expected input as a
variab1.e length array of values of arbitrary type, and
the expected output as a permutation of the input, with
the values sorted according to some total ordering crite-
rion. The specification also requires that the program
produc:e separate outputs containing the minimum and
maximum values found in the input array. Based on
this specification, the tester identifies the following cat-
egories: the array’s size, the type of the elements, the
maximum element value, the minimum element
value, and the positions in the array of the maximum
and minimum values.
While the categories are derived entirely from infor-
mation in the specification, the choices can be based on
the specification, the tester’s past experience in select-
ing effective test cases, and knowledge of likely situa-
tions for errors to occur. If the code is available for
analysis, the tester can also base the choices on the
program’s internal structure. In the sorting program ex-
ample, one possible way to partition the category array
size int.o choices is size = 0, size = 1, 2 + size + 100,
and size > 100. These choices are based primarily on
experience with likely errors. If the tester happens to
know that memory for variable size arrays is allocated
in blocks of size 256, then it might be wise to include
choices of size C 256, size = 256, and size > 256.
Once the categories and choices have been estab-
lished, they are written into a formal test specification
for eac:h functional unit. The test specification consists
of a list of the categories, and lists of the choices within
each category. The information in a specification is
used to produce a set of test frames that are the basis for
constructing the actual test cases. A test frame consists
of a set of choices from the specification, with each
category contributing either zero or one choice. An ac-
tual test case is built from a test frame by specifying a
single element from each of the choices in the frame. In
the sorting program example, the array size category
could contribute any of four choices to a test frame. For
each of the first two choices, there is only a single
element to specify for the test case. The third choice
allows any size between 2 and 100 inclusive to be spec-
ified, while the fourth choice allows any array size over
100 to be specified. The tester placed 0 and 1 into sepa-
rate partition classes of the array size category because
he wanted to assure that arrays of those specific sizes
became part of some test cases. Placing 2-100 into a
single partition class indicates the belief that a test case
676 Communications of the ACM
with any of these sizes is essentially as good as one
with any other size in that range.
A test specification as described so far is art unre-
stricted specification, since there are no relations indi-
cated among any of its choices. Each test frame gener-
ated from an unrestricted specification conta:ins exactly
one choice from each category. The total number of
frames generated from an unrestricted specification is
equal to the product of the number of choices in each
category.
Since the choices in different categories frequently
interact with each other in ways that affect the result-
ing test cases, the choices in a test specification can be
annotated with constraints that indicate these relations.
A typical constraint says that a choice from one cate-
gory cannot occur together in a test frame with certain
choices from other categories. Constraints are used to
refine the unrestricted set of test frames to a set consid-
ered both technically effective and economically feasi-
ble. The final determination of an appropriate set of test
cases is a pragmatic decision that depends or. the criti-
cality of the software, the project’s budget, and perhaps,
on the skill levels of the programmers and the testers.
After a test specification has been annotated with
constraints, it is processed by a generator tool that pro-
duces the test frames. Since this detailed combining of
information is done by an automated tool, it is rela-
tively easy for the user to make changes to the test
specification, and then rerun the tool to prod.uce a re-
vised set of test frames.
The final steps in the test production process are to
transform the generated test frames into test cases, and
then to combine the cases into test scripts. Information
from the parameter categories of a test frame deter-
mines the choice of specific inputs for a test case. The
environment categories guide the establishment of the
system state when the test case is run.
A test script is a sequence of related test cases for one
or more functional units. The tester must de’cide
whether test cases in a script should be independent, or
share common environment settings. Since each case
requires that its environment be set up before execut-
ing the command, an efficient approach groups test
cases that require the same environment into a single
script. This technique is most advantageous for test
cases that do not modify their environment. Another
approach is to use the result of case K in the script to
establish the environment for case K + 1. With both of
these methods of test script construction, if a test case
does not perform as expected, it is possible tlhat the
remainder of the script is useless. A safer method is to
set up each test case’s environment individually, just
before executing the test.
We now summarize this discussion in a series of
steps that constitute the category-partition method. The
section entitled A Test Specification LanguaLge and
Tool describes these steps in more detail, and the
method is illustrated with an example.
The category-partition method steps are as follows:
June 1988 Volume ::I Number 6
 Special Section

A. Analyze the specification. The tester identifies rences of a pattern in a file. The command’s natural
individual functional units that can be sepa- language specification is shown in Figure 1.
rately tested. For each unit, the tester identifies:
l parameters of the functional unit; Step A: Analyze the specification.
l characteristics of each parameter; The find command is an individual function that can
l objects in the environment whose state could be separately tested. Its parameters are pattern and file.
affect the functional unit’s operation; The specification mentions four characteristics of the
l characteristics of each environment object. pattern parameter:
The tester then classifies these items into cate-
gories that have an effect on the behavior of the l the length of the pattern;
functional unit. l whether the pattern is enclosed in quotes;
l whether the pattern contains embedded blanks;
If. Partition the categories into choices. The l whether the pattern contains embedded quotes.
tester determines the different significant cases
that can occur within each parameter/environ- There are several additional characteristics of the pat-
ment category. tern that are not explicitly mentioned in the specifica-
C. Determine constraints among the choices. The tion, but that a careful tester might wish to include as
tester decides how the choices interact, how the part of the information from which to construct tests.
occurrence of one choice can affect the existence These include:
of another, and what special restrictions might
l whether a quoted search pattern always has to in-
affect any choice.
clude a blank character;
(Step C and the following two steps frequently occur l whether several successive quotes can be included
repeatedly as the tester refines the test specification to in a pattern;
achieve the desired level of functional tests.) l the possibly ambiguous meaning of a leading quote
in the pattern: is it used to quote the entire pattern,
D. Write and process test specification. The cate-
or is it used to insert a literal quote character?
gory, choice, and constraint information is writ-
ten in a formal Test Specification. The written The file can be considered both as a parameter of the
specification is then processed by a generator find operation and as an object in the environment. We
that produces a set of test frames for the func- take the view that properties relating to the name of the
tional unit. file are parameter characteristics, while properties that
E. Evaluate generator output. The tester exam- have to do with the contents of the file are environment
ines the test frames produced by the generator, characteristics. In terms of the test cases produced by
and determines if any changes to the test specifi- the category-partition method, this distinction is not
cation are necessary. Reasons for changing the important, since the generator tool treats both types of
test specification include the absence of some
obviously necessary test situation, the appear-
ance of impossible test combinations, or a judg-
ment that too many test cases have been pro-
syntax:
duced. If the specification must be changed, find -qnttern> <file>
Step D is repeated.
FuncIion:
F. Transform into test scripts. When the test The find command is used to locate we w more instnnces of
a given pmern in a text file. All lines in the file that contitin
specification is stable, the tester converts the test the pattern we written to stsndard output. A line containing
frames produced by the tool into test cases, and the pattern Is written only once. regardless of the number of
limes the pattern occurs in it.
organizes the test cases into test scripts.
The pclttern is any sequence of characters wbosc length does
not exceed the maximom length of a linc in the ftle. To
A TEST SPECIFICATION LANGUAGE AND TOOL include a blank in the pattern, the cntirc pattern nwt be
enclosed in quotes (‘9. To include J quotation mark in the
The Test Specification Language (TSL) is the means of pattern, two quows in a row (“‘) must be used.
implementing the category-partition strategy described
in the previous section. The tester specifies categories of Examples:
find john myfilc
environments and input values, partitions each cate- displays lines in the file myfile which comaio john
gory into a set of mutually exclusive choices, and de- finch "john smictt" myfile
scribes constraints that govern the interactions between displays lines in the file myfile which cont;lb~ John smith
occurrences of choices from different categories. This firld "john"" bntith" mylilc:
information is written in a formal test specification that displays lines in the file myfile which contain john” smith
can be processed by a test generator tool.
We illustrate the use of TSL and the generator tool
with a simple find command that searches to occur- FIGURE1. Natural Language Specification for FINDCommand

/me 1988 Volume 31 Number 6 Communications of the ACM 619

Special Section

characteristics the same way. The main reason to em- characteristics number of occurrences of patterv in file and
phasiz’e both the parameter and environment charac- pattern occurrences on target line. These charac:teristics
teristics is to remind the tester to think of both when are the categories of the test specification that are parti-
analyzing the specification. tioned in the next step of the method.
When file is considered as a parameter, its significant
characteristic is whether or not it names an existing
Step B: Partition the categories.
file. When file is considered as an object in the environ-
In partitioning the categories, it is important to include
ment, its characteristics are:
any situation that might reasonably occur, plus at least
l number of occurrences of the pattern in the file; some that might not be expected. The latter type of
l number of occurrences of the pattern in a line that situation will become the basis for error tests for the
contains it (in the test specification, we call such a function. For the find function, the categories and
line a target line.); choices are shown in Figure 2. It should be noted that
l maximum line length in the file. the categories are divided into the two broad groups of
Parameters and Environments. Lines beginning with #
These characteristics come directly from the specifi-
are comments.
cation Again, it is possible to consider additional un-
The information in Figure 2 could serve as an unre-
specified characteristics of the file that could affect stricted test specification. However, it will generate
the belhavior of find. Examples are the file type (text,
1,944 test frames, many of which express situations
binary, executable, etc.), whether the pattern over- with contradictory requirements. The frame in Figure 3
laps itself in a line of the file, and whether the pattern is an example. According to this frame, the pattern
extends across more than one line.
parameter is supposed to be an empty string that con-
tains several embedded blanks. In addition, i.here are
# Unrestricted Test Specification for FIND command supposed to be no occurrences of the pattern in the file,
but one occurrence on the target line.
Parameters:
Pattern size:
empty
single character Patternsize : empty
many character Quoting : pattern is quoted
longer than any line in the file
Embeddedblanks : several embedded blanks
Quoting: Embedded quotes : no embedded quotes
pattern is quoted
pattern is not quoted File name : good file name
pattern is improperly quoted Number of occurrences of pattern in file : none
Embeddedblanks: Pattern occurrences ontargetline : one
no embedded blank
one embedded blank
several embedded blanks FIGURE3. Test Frame with Contradictory Requirements
Embeddedquotes:
no embedded quotes
one embedded quote Step C: Determine constraints among the choices.
several embedded quotes Contradictory test frames are produced because the
Filename: generator combines choices without any regard to their
good file name meaning and how they might interact. To eliminate the
no file with this name contradictory frames, constraints on the use of choices
omitted
are added to an unrestricted specification. Constraints
Environments: are indicated with properties that can be assigned to
Numberof occurrences ofpattern in file: certain choices, and tested for by other choices. Each
none individual choice in a test specification can be anno-
exactly one
more than one tated with a property list and a selector expres.sion. The
Patternoccuxrences ontargetline:
properties assigned to a choice are associated with
# assumes line contains the paiztern every test frame that includes the annotated choice.
one The form of a property list is
moxe than one
[property A, B, . . .]
FIGURE2. Category Partitions for FIND where A, B, are individual property names.
A selector expression is a conjunction of properties
To keep the size of the example reasonable, we use that were previously assigned to other choices. An
only the characteristics that are directly mentioned in expression tells the test frame generator not to include
the specification. We also omit the third file character- the annotated choice in a test frame unless the proper-
istic, maximum line length. We use the five parameter ties in the expression are associated with choices that
characteristics pattern size, quoting, embedded blanks, em- are already in the frame. A selector expression is as-
bedded quotes, and file name, and the two environment signed to a choice by appending a suffix of the form

660 Communications of the ACM June 1988 Volume 31 Number 6

 Special Section

[if A] the test frames that will include the choice, and the
properties are assigned to any partial frames containing
Or
the choice.
c-if A and B and . ..I The restricted specification of Figure 4 produces
678 test frames, a reduction of two-thirds from the orig-
to the choice.
inal 1,944. However, 678 is still too many test cases to
Figure 4 shows the find command test specification
be economically and practically feasible for a fairly
annotated with property lists and selector expressions.
simple operation like find.
Since we want to distinguish empty patterns from non-
empty ones, the two properties Empty and NonEmpty
are assigned to the appropriate choices of the category Steps D and E: Write test specification and evaluate
Pattern size. generator output.
When the generator encounters a choice with a selec- If we examine the test frames produced from the re-
tor expression, it omits that choice from any partial test stricted specification, we find that many are redundant;
frame that does not already have the properties speci- they test the same essential situation. They differ only
fied in the selector expression. For example, all of in varying parameters that have no effect on the com-
the choices in the categories Embedded blanks mand’s outcome. This occurs frequently when some
and Embedded quotes, as well as the last two in parameter or environment condition causes an error.
Quoting would never be combined with the choice For instance, every find command for which the
Pattern size: empty. In addition, the second named file does not exist will result in the same error,
and third choices in Embedded blanks would only regardless of the form of the pattern.
be combined with a tuple that contains the choice TSL allows the special annotation [error I to be
Quoting: pattern is quoted. attached to a category choice. [error ] tests are de-
A choice may be annotated with both a selector signed to test a particular feature which will cause an
expression and a property list: for example, the choices exception or other error state. It is assumed that if the
exactly one and more than one in the Environ- annotated parameter or environment has this particu-
ments section. In such a case, both are applied inde- lar value, any call of the function using that choice
pendently; the selector expression is applied to restrict will result in the same error. A choice marked with

#I Test Specification for find command

# Modified: property lists and selector expressions added
Parameters:
Pattern size:
empty Lp=qJerty hlptyl
single character [property NonEmpty]
many character [property NonEmpty]
longer than any line in the file [property NonEmpty]
Quoting:
pattern is quoted [property Quoted1
pattern is not quoted [if NonEmpty]
pattern is improperly quoted [if NonEmpty]
Embeddedblanks:
no embedded blank [if NonEmpty]
one embedded blank [if NonEmpty and Quoted1
several embedded blanks [if NonEmpty and Quoted]
Embeddedquotes:
no embedded quotes [if NonEmpty]
one embedded quote [if NonEmpty]
several embedded quotes [if NonEmpty]
Filename:
good file name
no file with this name
omitted

Environments:
Number of occurrences ofpatterninfile:
none [if NonEmpty]
exactly one [if NonEmpty] [property Match]
more than one [if NonEmpty] [property Match]
Pattern occurrencesontargetline:
# assumes line contains the pattern
one [if Match1
more than one [if Match1

FIGURE4. Specification with Property Lists and Selector Expressions

June 1988 Volume 31 Number 6 Communications of the ACM 661

 Special Section

[error ] is not combined with choices in the other combinations against the pragmatic limits of time and
categories to create test frames. The generator creates a personnel that usually prevail in a software (develop-
test frame that contains only the [error] choice. The ment project.
test case defined from such a frame must fulfill the Step F: Transform into test scripts.
statement of the marked (error ] choice, but the Figure 6 shows one test frame generated from the final
tester can set the test’s other parameters and environ- specification for find, together with the operating sys-
ment (conditions at will. Adding four [error ] mark- tem command to establish the file environment, the
ings to the restricted specification reduces the number instance of find that performs the test, and a descrip-
of test frames to 125. tion of the test’s expected output.

# Test SpeciEication for find command

# Modified: property lists and selector expressions added
# Modified: error and single annotations added
Parameters:
Pattern size:
empty [property Empty1
single character [property NonEmpty]
many character [property NonEmpty]
longer than any line in the file [fSX0rl
Quoting:
pattern is quoted [property Quoted]
pattern is not quoted
pattern is improperly quoted
Embeddedblanks:
no embedded blank [if NonEmpty]
one embeddeil blatik [if NonEmpty and Quoted]
several embedded blLnks [if NonEmpty and Quoted]
Embeddedquotes: _)__
no embedded quotes [if NonEmpty
one embedded -quote [if NonEmpty]
several embedded quotes [if NonEmpty] [single]
File name:
good file name
' no file with this name
omitted

Environments:
Number of occurreficesofpatternin file:
none [if 14onEmptyJ [single1
exactly one (if NonEmptYj (property Match1
more than one [if NonEmpty] [property Match1
Patternoccurrences ontargetline:
# assumes line contains the pattern
one [if Match]
more than one [if Match) [single]

FIGURE5. Final Test Specification for FIND

The number may be further reduced by using The test frame contains two numbers. The “Test
[single I markings. This annotation is intended to Case” number sequentially identifies the test. The
describe special, unusual, or redundant conditions that “Key” number refers to the choices made from each of
do not have to be combined with all possible choices. the frame’s categories (choice 3 from the first category,
As with (error I choices, the generator does not com- choice 1 from the second, and so forth). Once the cate-
bine a [single] choice with any choices from other gories and choices in a test specification are fixed, the
categories. A single frame is produced that contains key number for a given test frame stays the same even
only the marked [ single ] choice. This reduces the if the property lists and selector expressions change.
total number of frames, but assures that one frame will In the example shown in Figure 6, the file case-28
be created with the marked choice. By marking three has been created earlier and stored in another direc-
choices of the find specification as ( single ] , the total tory. It is copied into a testing directory to set up the
number of test frames drops to 40. The final, com- environment for the test. The output of find is either a
pletely marked specification is shown in Figure 5. display of matching lines or a message that no matches
The decision to use a [single ] marking is a judg- were found. Test result checking could be done either
ment by the tester that the marked choice can be ade- manually or by directing the command’s output to a log
quately tested with only one test case. It represents an file which is later compared to a prepared output file.
attempt to balance the desire for complete testing of all In cases where the tested function modifies the system

662 Communications of the ACM June 1988 Volume 31 Number 6

 Specinl Se&ion

state, additional commands may be necessary to verify
that it performed correctly.
Some inconsistencies, redundancies, and ambiguities
in the test specifications may only become apparent
when the actual test scripts are created. A combination
of parameter values and environment conditions that
looked right in the specification may actually be impos-
sible to test. One test case may be identical to another.
Such situations require that the tester modify the origi-
nal test specification, usually by adding property lists
and selector expressions, to correct the problem.
EXPERIENCE WITH TSL
The TSL is now being used for the design of functional
tests for the version and configuration management
(VCM) component of an interactive programming sup-
port environment. The other components of this system
are a text editor and a source-language debugger. The
entire system has been designed using object-oriented
techniques and is implemented in Ada.
The TSL has been used to create test scripts for the
second and third releases of the VCM component. The
second release consisted of approximately 35,000 Ada
source statements (counted by semicolons), making up
over 2,200 Ada procedures in 131 packages. Function
testing for this release consisted of testing 91 high-level
procedures, each roughly equivalent to one user-level
command.
The VCM design and implementation team consists
of four members. When functional testing was started,
one team member became primarily responsible for
testing. This test engineer used the written specification
documents, Ada package specifications, and his own
experiential knowledge of the system to write the test
specifications. One test specification was written for
each of the 91 high-level procedures. All of the test
specifications were written in approximately three
weeks.
The four other team members were then given a one-
page overview of the TSL format. These team members
individually reviewed the test specifications, noting er-
rors and inconsistencies and suggesting changes. Judg-
ing by the quality of the reviews, they found the TSL
format to be relatively easy to understand.
Surprisingly, most of the review comments simplified
the test specifications. Many environment conditions
were eliminated as unnecessary and others were
marked [ sing 1 e 1. In the few instances where func-
tionality had changed since the last revisions of the
program specifications, the entire test specification had
to be rewritten (and the program specification was
quickly revised).
The TSL generator was run on each of the test speci-
fications to produce a set of test frames for each proce-
dure. The entire system required 1,022 test cases.
Writing the actual test scripts was the most time-
consuming part of the process. Each procedure’s script
consisted of some setup commands, the test cases them-
selves, and cleanup commands. Common data struc-
tures were used as appropriate. Although it would have
cut down on the total number of commands, it was
decided not to use one test case to set the environment
for a following test case (e.g., test case A creates a file,
test case B reads the file). Wherever possible, the test
cases included commands to check the results of the
test.

Test Frame:
Test Case 28: (Key = 3.1.3.2.1.2.1.)
Pattern size : many character
Quoting : pattern is quoted
Embeddedblanks : several embedded blanks
Embeddedquotes : one embedded quote
Filename : good file name
Number of occurxences ofpatternin file : exactly one
Pattern occurrences ontargetline : one

Commands to set up the test:

copy /testing/sources/case_28 testfile

find command to perform the test:

find "has Nn one quote" testfile

Instructions for checking the test:

The following line should be displayed:
This line has " one quote on it

FIGURE6. A Test Frame and Test Case

June 1988 Volume 31 Number 6 Communications of the ACM 663

 Specinl Section
Once the test cases were written, it took approxi-
mately two weeks to completely run all of the tests.
Many times there were bugs in either the test scripts or
the original test specifications themselves, despite the
fact that the test specifications had been reviewed prior
to the time the test cases were generated and written.
This required modifying the scripts, and in a few cases,
the test specifications themselves. Thirty-nine bona fide
bugs in the software were found and corrected using
the tests produced with the category-partition method.
Following the development group’s testing with the
category-partition tests, the system was sent to an inde-
pendent system testing group for additional testing.
This group found 61 additional problems with the VCM
software, of which 19 were classified as implementation
faults, and the remaining 42 as “documentation errors,”
“unclear messages,” “ operating system errors,” and
“suggestions for improvement.”
The complete suite of test scripts has been saved as a
regression test base for future releases. All of the test
specifications and scripts have been version controlled.
To facilitate checking the regression tests, a transcript
of the last complete run of the test scripts was saved so
that it can be compared mechanically against a tran-
script of a run of the same scripts against a future re-
lease.
RELATION TO OTHER METHODS
The Condition Table Method
Goodenough and Gerhart’s [5] test case definition tech-
nique is called the condition table method. They con-
struct a condition table in which each column repre-
sents a combination of conditions that can occur during
execution of the program. By examining the program’s
specifications, they try to identify conditions that have
a significant impact on the execution behavior of the
program. In the cited article, their main concern was to
provide a method that could be used in conjunction
with their theory of testing, and they do not discuss any
automatization of condition tables.
Some aspects of the Goodenough and Gerhart method
are quite similar to the category-partition method. The
conditions of Goodenough and Gerhart are what we call
categories. They define the value set of a condition as the
possible values for the condition; the value set corre-
sponds to what we call choices for each category. Good-
enough and Gerhart also identify constrnints between
conditions, but use them in a more restricted way than
we do. The Goodenough and Gerhart constraints are
only used to express interactions between conditions
that are consequences of the conditions’ definitions. For
example, a constraint may say that if condition 1 holds
then neither condition 2 nor condition 3 can hold. This
type of statement limits the number of combinations
that a.re possible, and hence limits the number of test
predicates that must be considered. In the category-
partition method, constraints are used both to express
defin:ition-based interactions of the above type, and also
as a means for the tester to introduce additional restric-
684 Communications of the ACM
tions to limit the type and number of test ca.ses that are
generated.
Cause-Effect Graphing
Another strategy for transforming a specification is
cause-effect graphing, originally defined by Elmendorf [4]
and illustrated by Myers [8]. This strategy begins with
the identification of each individual function or com-
mand of the system to be tested. Then for each func-
tion, the tester identifies all significant causes that in-
fluence the function’s behavior, and all effects of the
function. The next step is to construct a graph that
relates combinations of the causes to the effects they
produce. Test cases are defined for each effect by con-
sidering all combinations of causes that proc!uce that
effect.
Although careful use of the cause-effect method can
produce effective tests, the method is difficult to apply
in practice. In particular, the cause-effect graph can
become very complex when a function has a large
number of causes. To keep the complexity under con-
trol, the tester must add intermediate nodes to repre-
sent logical combinations of several causes. .An appro-
priate choice of intermediate nodes is frequently not
obvious. Other problems with the cause-effect graph
are the difficulty of verifying its correctness. or of up-
dating it when the specification changes or when the
tester realizes that some information has been over-
looked. If new nodes are added to the input, or cause,
side of the graph, much of its internal structure may
have to be redesigned.
By transforming a written specification into a set of
cause-effect graphs, the tester is replacing one complex
representation with another. Any changes that occur in
the specification must be translated into corresponding
changes in the graph. Naturally, changes in the specifi-
cation are a problem for any method, but a simpler
intermediate representation can ease the difficulty.
Revealing Subdomains
Weyuker and Ostrand [lo] proposed a combined black-
box and white-box method to derive a partition of a
function’s input domain into revealing subdomains. A re-
vealing subdomain contains elements that are either all
processed correctly or all processed incorrectly. Once
such a subdomain has been identified, executing the
program on a single one of its elements is su.fficient to
test the entire subdomain.
The authors note that, theoretically, the existence of
a correctly processed input from a subdomain, together
with showing that the subdomain is revealing, are
equivalent to proving the program’s correctness for all
inputs in the subdomain. Given the impossibility in
general of proving program correctness, one should not
expect to be able to produce revealing subdomains at
will. To render the theory somewhat more practically
applicable, the notion of reuealing subdomain for a spe-
cific error E is introduced. With such a subdomain, if the
error E is present and affects some element of the sub-
domain, then every element of the subdomain is pro-
June 1988 Volume .31 Number 6

cessed incorrectlv. This definition makes it easier to
find revealing s&domains, and guides the tester to con-
centrate on probable errors and the sets of inputs that
they might affect.
The article presents a technique that attempts to con-
struct revealing subdomains by identifying the most
likely places for errors to occur. The technique uses
information from both the specification and the code.
The tester first creates a problem partition from the spec-
ification, by looking for classes of inputs that should be
treated the same way by the program. The next step is
to create a path partition, whose equivalence classes
contain inputs that actually are treated the same way
by the program. The partition used for functional test-
ing is then created by intersecting the problem partition
and the path partition, creating a set of equivalence
classes whose elements both should be and are treated
the same way by the program. A test set is built by
choosing one element from each of the testing parti-
tion’s classes.
The revealing subdomain concept emphasizes the
importance of choosing test data from both program-
dependent and program-independent sources. The
main difficulty in its implementation is that there are
no formal or systematic guidelines for creating the
problem partition; [lo] states only that it should be
formed “on the basis of common properties implied by
the specifications, algorithm, and data structures.”
Since the category-partition method provides a sys-
tematic approach to creating test sets on the basis of the
specification, it could conceivably be used to help cre-
ate the problem partition.
Equivalence Partitioning
Richardson and Clarke’s [9] equivalence partitioning
method is very similar to the revealing subdomain
approach. They partition a function’s input domain
into a procedure partition, which is the intersection of a
program-based path domain and a specification-based
specification domain. The path domain is constructed by
standard symbolic execution techniques applied to the
program. To construct the specification domain, the au-
thors assume that the specification is presented in a
formal specification language, to which symbolic exe-
cution techniques can be applied.
Test data are selected in two ways, depending on the
types of errors the tester is looking for. To find compu-
tation errors (incorrect computations), the tester should
study the path and specification domains, and use
“computationally sensitive data,” such as extreme val-
ues and special values within those domains. To find
domain errors (input data following the wrong path in
the program), the tester should use test cases that are
boundary points of the procedure domain.
Equivalence partitioning depends crucially on a for-
mal specification to allow the symbolic analysis that
creates the specification domain. Assuming such a for-
mal specification exists, the method probably will pro-
duce test data that are similar to those produced by the
revealing subdomain method. For a system that is spec-
June 1988 Volume 31 Number 6
Special Section
ified in natural language, the category-partition tech-
nique could be used to determine an appropriate set of
specification domains. The informal representations of
these domains could be converted into a format similar
to that produced by symbolic execution of a formal
specification, and the result used to form the procedure
partition required by equivalence partitioning.
PLANNED EXTENSIONS
The TSL was developed during the implementation of a
major software project, to meet an immediate need for
a systematic test development strategy and tool. The
criteria were to produce an easily usable and easily
understood system as soon as possible. While the pres-
ent system greatly facilitated testing the VCM software,
we are currently considering the following enhance-
ments.
A. More general ways of specifying and checking
properties. At present, the selector expression
on a choice can only be the conjunction of previ-
ously assigned properties. A simple change is to
allow any Boolean combination of properties. Al-
though in practice the inability to specify any
logical combination has not been a problem, this
generalization would provide more flexibility in
specifying combinations.
B. Generalize error and single tests. At present,
error and single tests are generated from a single
parameter or environment choice, forcing the
tester to make an all-or-nothing decision. The
ability to use Boolean combinations of choices to
specify them would give the tester finer-grained
control in designing error or single situations.
C. Specifying test results. The present version of
TSL provides no means for specifying results of
test cases. It is up to the tester to define the
expected results for each case when the test
scripts are created from test frames. The next
version of the tool will allow the tester to in-
clude a Results section for each function. The
user will decompose the possible results for a
function into choices that are similar to the
choices in the decomposition of parameter and
environment influences. A Boolean expression
attached to each result choice will describe com-
binations of the parameter and environment
choices that cause that result to occur. The infor-
mation contained in such a result-augmented
test specification will be very similar to the in-
formation in a cause-effect graph.
The test generator will then be able to produce
the following types of output:
l Test frames that include a list of expected re-
sults.
l A result frame that, for a given result, lists all
combinations of parameters and environments
that produce that result.
Communications of the ACM 665
Special Section
l A list of results that are not produced by any
allowable combination of choices.
l A list of choice combinations for which a test
frame is produced, but which have no associ-
ated result.
D. Automate test script generation. Converting
the generator-produced test frames into test
scripts is the most time-consuming part of the
entire testing process. Much of the conversion
could be automated by taking advantage of fre-
quently occurring environment conditions. In us-
ing TSL, the test engineer found that after a few
test cases were completely written by hand, he
could rapidly assemble additional cases by cut-
ting and pasting pieces from the previous ones. A
fixed set of commands could be used to establish
an environment corresponding to a particular
environment category and choice. This idea
could be used to build a version of TSL that
would automatically supply the appropriate sys-
tem commands to establish each test case’s
environment.
SUMMARY
The category-partition method provides the tester with
a systematic method for decomposing a functional spec-
ification into test specifications for individual functions.
The test specification language is a concise, easily mod-
ified representation of tests that can be understood by
programmers, testers, and managers. A particular bene-
fit is that the tester can control the size of a product’s
test suite, not by arbitrarily stopping when a given
number of tests have been written, but by specifying
the interaction of parameters and environments that
determine the tests.
TSL and the category-partition method are a promis-
ing foundation for further automation of specification-
Q n
l I-800-342-6626
ACM’s“OrderExpress”
Service
for ACM Publications.
666 Communications of the ACM
based functional testing, a task that is generally seen as
tedious but necessary for the implementation of high
quality software.
REFERENCES
1. Adrian, W.R., Bran&d, M.A.. and Cherniavsky. J.C. Validation, ver-
ification, and testing of computer software. ACM Camp. Sure. 14, 2
(June 1982). 159-192.
2. Berzins, V.. and Gray, M. Analysis and design in MSG 84: Formaliz-
ing functional specifications. IEEE Trans. Softw. Eng. SE-II, 8 (August
1985). 657-670.
3. Bjorner. D.. and Jones. C.B. Formal Specification and Software Develop-
ment. Prentice-Hall International, En&wood Cliffs, N J., 1982.
4. Elmendorf. W.R. Functional analysis using cause-effect graphs. In
Proceedinys of SHARE XLIII. SHARE. New York. 1974, .567-577.
5. Goodeno;gh; J.B., and Gerhart. S.L. Toward a theory elf test data
selection. 1EEE Trans. Softw. Eng. SE-2. 2 (June 1975), 156-173.
6. Howden, W.H. A survey of dynamic analysis methods In Tutorial:
Program Testing and Validation Techniques, edited by E.F. Miller and
W.H. Howden. IEEE, 1981.
7. Howden, W.H. Errors, design properties. and functional program
tests. In Computer Program Testing Summer School, Soge:ita, Itnly. ed-
ited by B. Chandrasekaran and S. Radicchi. North-Holland. New
York, 1981.
8. Myers, C.J. Software Reliability: Principles and Practices. John Wiley.
New York. 1976.
9. Richardson. D. and Clarke, L. A partition analysis method to in-
crease program reliability. In Proceedings of the 5th lntunafionnl Con-
ference Software Engineering (San Diego. CA, Mar. 9%1211.IEEE. 1981.
pp. 244-253.
10. Weyuker, E.J., and Ostrand, T.J. Theories of program testing and the
application of revealing subdomains. IEEE Trans. Softa. Eng. SE-6 3
[May 1980). 236-246.
CR Categories and Subject Descriptors: D.2.5. [Testing and Debug-
ging]: K.6.3 [Software Management]
Additional Key Words and Phrases: Automatic test generator.
partition. software testing, test specification
Authors’ Present Address: Thomas J. Ostrand and Marc J. Balcer, Soft-
ware Technology Department, Siemens Research and Technology Labo-
ratories, 105 College Road East, Princeton. N.J. 08540.
Permission to copy without fee all or part of this material is granted
provided that the copies are not made or distributed for direct commer-
cial advantage. the ACM copyright notice and the title of lhe publication
and its date appear, and notice is given that copying is by permission of
.
the Association for Comoutine I Machinerv. To cow. _ otherwise. or to
republish, requires a fee and/or specific permission
1
Your credit card and our toll free number provide
quick fulfillment of your orders.
l Journals
l Conference Proceedings
l SIG Newsletters
. SIGGRAPH VIDEO REVIEW
l “Computers in your Life” (An Introductory
Film from ACM)
For Inquiries and other Customer Service
call: (301) 528-4261
acm ASSOCIATION FOR
COMPUTING MACHINERY
-I
June 1988 Volume 31 Number 6