Вы находитесь на странице: 1из 63

Automated Testing of

Object-Oriented
Components using
Intelligent Test Artifacts
Michael Silverstein
Lead Architect
SilverMark, Inc.
(www.javatesting.com)
1 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts, Copyright 2000 SilverMark, Inc.
Component Test
Frameworks
• Level 0: Self tests, usually in static
methods
• Level 1: Test cases implemented as
methods in harness classes
• Level 2: Basic execution framework
(J-Unit, for example)
– TestSuite, TestCase, TestResult
• Level 3: Execution framework includes
objects that encapsulate test specific
behavior, promotes task-configured
reusable test components
2 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,
Example Application

3 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,


Test Design Questions

• How do I create a specifically configured


instance of a component under test?
• How do I manage the application of a
stimulus to the component under test?
• How do I organize stimuli into reusable
groups?

4 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,


Test Design Questions

• How do I validate the state of the


component and system under test after
one or more stimuli?
– How do I manage reference objects?
– How do I compare references objects to
target objects, so that just the states that
are important are compared?
– How do I intelligently traverse the state of
my object under test?
• How do I validate that expected
exceptions are raised?
5 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,
Test Design Questions

• How do I validate that expected


events are fired?
• How do I manage variations on
test inputs in order to drive different
paths with the same test stimuli?
• How do I map development
activities to the tests that validate
their correctness?

6 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,


Pattern Responses
to Design Questions
• Test Component—Implement a
reusable component model to
encapsulate configurable aspects
of automated testing
• Test Sequence—Present a complete
or commonly implemented subset of
a test as a discrete, reusable unit
• Configured Instance—Provide an
instance of an object configured
to a particular state

7 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,


Pattern Responses
to Design Questions
• Test Stimulus—Encapsulate a
stimulus/response validation operation
for a component under test for singly
or multiply threaded systems
• Nested State—Provide a mechanism
for specifying and traversing a chain
of object references to extract a
specific state

8 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,


Pattern Responses
to Design Questions
• Aggregate State—Represent the state of a
component in terms of an aggregate set or
subset of the states of itself and its
subcomponents
• State Validation—Assert that a component
is in the expected state. Define this
assertion as a reusable component
• Object Variation—Organize configured
object variations

9 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,


Test Component

• Intent: Implement a reusable component


model to represent aspects of automated
testing, of arbitrary granularity
• Context: Move common test code out of
test cases and into a framework in a way
that supports creation of reusable test
components

10 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,


Test Component

• Strategy: Create a hierarchy of classes


called Artifacts, each of which embodies
specific test execution behavior, as well
as the properties required to configure
instances for specific applications of
the artifact
– A specification for an artifact describes a
specific configuration of an artifact instance

11 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,


Test Artifact—Properties

• Name—Used to identify and locate


an artifact
• Description—More extensive descriptive
information about the artifact, either as
text or a reference to some external
document (URL)
• Classification of Intent—Specifies intent
of artifact for documentation, (use case
scenario, requirements, validation,
invariant, etc.)

12 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,


Test Artifact—Properties

• Expected exception—A reference


to an exception that is expected to
be signaled as a result of execution
of the artifact. Execution of the artifact
is said to have failed if the exception
is not raised
• Precondition action—Actions to set and/or
verify the state of the system under test
prior to further execution of the artifact

13 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,


Test Artifact—Properties

• Invariant action—A set of assertions that


express general consistency constraints that
apply to every class instance as a whole,
regardless of stimulus. If specified, this is
executed after execution of the artifact
• Synchronization—Strategy for determining
when a response to a stimulus has
completed
– Message completed/returned [synchronous]
– Event(s) fired [synchronous or asynchronous]
– Miscellaneous system state change [asynchronous]

14 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,


What Do Test Artifacts
Look Like?
• An object implemented to provide a specific
test-oriented behavior, configured to act on
some aspect of the system under test
• In many cases, a test artifact is defined in terms
of other test artifact definitions or references to
other test artifacts
• Instantiate test artifact or artifact tree,
then execute it
“Test artifact #2”
Instantia
te eren ce
tifact ref
ar
Test
“Test artifact #1”
Execute

15 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,


Where Do Artifacts
Come From?
• Hierarchy of test classes whose purpose
is to provide test artifacts and supporting
methods, directed toward aspects of the
system under test
– Structural: based on the structural
relationships of the classes under test
– Functional: based on the functional
relationships of classes, clusters and
subsystems of system under test

16 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,


Providing Artifacts
Test hierarchy

Hierarchy under test

17 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,


Test Artifact—Consequences

• Test artifacts provide semantics that


capture important strategic and tactical
decisions about tests
• Implementing typical testing patterns
as a palette of reusable, individually
configurable component types that
embody common testing behavior
enables test developers to focus
less on test implementation and more
on the problem of exercising the
system under test
18 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,
Test Artifact—Consequences

• Artifacts offer arbitrary levels of test


organization beyond the traditional class,
method, or statement level
• Test artifacts support either a bottom-up
or top-down test development process

19 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,


Test Artifact—Consequences

• Test artifacts benefit test tool writers


because artifacts present a façade for
complex testing behavior, and artifacts
may be combined to form a complete
representation of a test which may
then be presented for editing by a
tool user interface
• Test development speed increases
as more reusable components
become available

20 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,


Test Artifact—Consequences

• Defining test artifacts in one and only one


place eases maintenance because
changes to test code due to changes in
the system under test are localized

21 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,


Test Sequence

• Intent:
– Present a sequence of test actions of arbitrary
granularity as a discrete, reusable unit, where
test actions may themselves be references to
predefined reusable components

22 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,


Test Sequence

• Context:
– A prerequisite for creating tests via
composition of components is a container
in which to hold them
– Fine-grained executable units enable
automated testing tools to provide better
definition UIs, better tracking of test execution
progress, and more detailed reports

23 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,


Test Sequence

• Strategy:
– Create a composite test sequence artifact
composed of artifact specifications
– Execution behavior: execute each component
in order

24 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,


Test Sequence

• Consequences:
– The test sequence artifact encourages
construction through composition
– Because composite artifacts are responsible
for sequencing through their components
during execution, they may also embody
general execution rules, such as ceasing
execution on failure

25 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,


Test Sequence

• Example Test Sequences:


– Create new Bank
– Add new account to Bank
– Remove account from Bank
– Insert card
• Account found
• Account not found
– Withdraw cash <= balance
– Withdraw cash > balance (exception)

26 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,


Test Sequence

• Maps nicely to Use Case


Scenario, requirements
– Supports “Uses” and “Extends”

27 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,


Test Stimulus

• Intent:
– Encapsulate a stimulus/response validation operation
for a component under test for singly or multiply
threaded systems
• Context:
– The basic unit of execution of a test case is the
application of a stimulus
– The response to a stimulus should be validated in
terms of a change in the state of the component
under test, or some area of the system under test,
as well as an object returned directly by the stimulus
– In multithreaded systems, the response to a
stimulus may be temporally non-deterministic

28 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,


Test Stimulus—Strategy
Create an artifact that embodies any or all of:
• A stimulus on some object, defined as one of:
– A reference to a target object and the action to perform on the
target object (message send), as well as any arguments.
References to configured instance artifacts serve well for
acquiring argument values
– The selector for a method within a supporting test class
– A reference to an artifact
– Other executable representation of the stimulus
(Java™ technology inner class, Smalltalk block)
• Precondition state validation
• Postcondition state validation
• Invariant validation
• A synchronization mechanism for suspending test
execution until an asynchronous process
has completed, or time limit is exceeded
29 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,
Configured Instance

• Intent: Provide an instance of an


object configured to a particular state
• Context: Creating supporting objects is a
necessary, and often distracting, evil of
component testing. Supporting objects
are required as:
– Inputs to individual stimuli (message
parameters)
– Comparison targets (gold standard objects)
– Starting points for sequences of stimuli for
testing components in a specific state
30 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,
Configured Instance

• Some Configured Instances for


ATM Example:
– New Account
– Account with $0.00/S/M/L balance
– Inactive account
– Debit Transaction for a particular
account and amount
– Credit Transaction for a particular
account and amount

31 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,


Configured Instance

• Strategy: Create an artifact whose purpose


is to return an object in a particular state
– Approaches:
• Instantiation: Create a configured instance of
a class based on:
– The class to instantiate
– A constructor method and parameters
– Additional stimuli
• Materialization: Retrieve serialized object
from some table/database/file, then apply
additional stimuli as necessary

32 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,


Configured Instance

• Strategy: Create an artifact whose purpose


is to return an object in a particular state
– A configured instance may be implemented as
a test sequence or test sequence subclass
• Subclass would have additional specification
for instantiation or retrieval of object, before
applying additional stimuli

33 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,


Configured Instance

• Consequences: Over time, test developers


accumulate a library of supporting object
instances that they may readily apply to
test actions

34 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,


Configured Instance Example

Test Class Artifact Class under test

com.mybank.atm.tests.Account provides instances


of ConfiguredInstance that are set up to return instances
of com.mybank.atm.Account in a particular state

35 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,


Configured Instance
Example—Reusing Preexisting Artifact

Reference to
ConfiguredInstance
(not shown)

36 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,


Configured Instance
Example—Reusing Preexisting Artifact

Reference to
ConfiguredInstance
(not shown)

37 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,


Simple Unit Test Scenario
Instantiate a DebitTransaction setting a particular state, and execute…

38 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,


Example Code—
Configured Instance

// Return a configured instance of a DebitTransaction with amount set to $123

public ConfiguredInstance smallDebitTransaction() {


ConfiguredInstance artifact =
new ConfiguredInstance(“Debit transaction for small amount");

// Set subject via constructor specification


artifact.setConstructorStep(
new Stimulus(“Default").setMessageSpec(
message(com.myBank.atm.DebitTransaction.class)));

// Apply stimulus to subject via Stimulus


artifact.addComponent(new Stimulus(“Set amount").setMessageSpec(
message(subject(),“setAmount",new int[]
{123}));
return artifact; }

39 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,


Example Code—
Artifact Reference
/* Artifact that references an artifact */
public TestSequence requestDebitScenario() {

TestSequence artifact =
new TestSequence(“Debit transaction scenario");

ArtifactReference reference =
new ArtifactReference(“Debit transaction test subject”,
com.myBank.atm.tests.DebitTransaction,”smallDebitTransaction”);

artifact.setConstructorStep(reference);
// Apply stimulus to subject via Stimulus
artifact.addComponent(
new Stimulus(“Execute transaction").setMessageSpec(
message(subject(),“execute)));

return artifact; }

40 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,


Example Test Tool View

41 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,


Nested State

• Intent: Provide a mechanism for specifying


and traversing a recursive chain of object
references to extract a specific state
• Context: Complex objects and object
clusters, especially implementations of
the façade pattern, conceal recursively
held collaborating objects whose individual
states contribute to the whole, and whose
access is often required for state validation
during testing

42 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,


Nested State

• Strategy:
– Implement methods on the root class to provide
nested state
• Adds code bloat to delivered code for the sake
of testability
– Implement code that provides a means to specify an
access scheme for a particular state of a
given object. For example,
return myObject.x().y().z();
– Configure a composite artifact to contain a sequence
of stimuli to be used to access successive states in
an object reference chain
• Less straightforward, but presents opportunity
to build off of a state returned by an artifact
reference
43 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,
Nested State

• Consequences:
– A configured nested state artifact is a
permanent, reusable access strategy
for a particular sub-state within a component
– Maintenance is localized

44 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,


Aggregate State

• Intent: Represent the state of a component


in terms of an aggregate of some subset of
the states of its subcomponents
• Context: The recursive state of an object is
the value set of instance variable values
obtained by recursively expanding each
instance variable value
– A component has one complete recursive
state, but may have many aggregate state
representations, or views, depending on
which states are of interest

45 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,


Aggregate State

• Strategy:
– Populate a composite aggregate state artifact
with components that, when executed, extract
individual states from the object under
consideration
• Nested state artifacts are a prime candidate
for population
• Execution time behavior gathers and returns
a collection of resultant execution values of
each of its components
• Implement comparison operators that take
into account the complete set of states,
like equals()
46 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,
Aggregate State

• A configured aggregate state artifact acts as a reusable


representation of a particular view of an object’s state
• Useful shorthand for comparing objects:

Typical aggregate state comparison


return customer.x().equals(referenceCustomer.x()) &
customer.x().y().equals(referenceCustomer.x().y()) &
customer.x().y().z().equals(referenceCustomer.x().y().z())
… etc. for each state

Artifact aggregate state comparison


(new MyArtifacts().getArtifactNamed(
“General Customer State”)).subject(customer).equals(
(new MyArtifacts().getArtifactNamed(
“General Customer State”)).subject(referenceCustomer));

47 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,


State Validation

• Intent: Assert that a component is in an


expected state. Define this assertion as
a reusable component
• Context:
– State validation involves comparing an object’s
recursive state to some known, expected state,
encoded as:
• A comparison to one or more Gold Standard
reference objects
• Simple comparisons to primitives or literals
• A test for some known object’s inclusion or
exclusion within a collection type
• A comparison between object impressions
48 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,
State Validation—Strategy

• A common strategy: implement an


assert(boolean) test API to log results
of executing an expression that logs the
success or failure of a contained
expression, during test execution
– Simple, flexible, but not very reusable

49 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,


State Validation—Strategy

• Use common postcondition validation


action test artifact property:
– Close, structural relationship between stimulus
and response validation enhances understanding
of the test. Define a validation action as:
• An expression that evaluates to a boolean. This is similar
to the assert API discussed above
• A reference to an artifact that performs the validation
• Where the same validation is performed repeatedly
(invariants, for example) defining and then referencing
an artifact is advantageous, especially if many state
comparisons are involved
• Implement an aggregate state artifact as a
comparison wrapper, implementing equals()

50 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,


Object Variation

• Intent: Organize variations of configured


object instances as a single,
reusable resource
• Context: Need to be able to pass input
variations of test objects to a test for the
purpose of driving execution of the
different paths (data driven tests)
• Variations are themselves reusable units
that may shared among test developers

51 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,


Object Variation

• Strategy:
– Implement an artifact that presents an
enumeration interface for sequentially
accessing its elements, whereby execution
of a given test element is repeated over the
elements returned by the object variation
artifact
• Statically described references to configured
object instances
• Objects dynamically generated by rules
– For example: an Integer interval or
a calculated equivalence set

52 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,


Object Variation

• Strategy:
– For multiple variation artifacts, implement
pluggable value mixing policy (one-by-one,
each-with-every, etc.).

53 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,


Object Variation

• Consequences: Useful variations are


themselves reusable units that may shared
among test developers

54 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,


Object Variation Example

“Account Variations Artifact”

Components Account Variations

Configured Instance
New Account
Artifact Reference
Configured Instance
Account
Balance=$0
Artifact Reference
Configured Instance Account
Balance=$1
Artifact Reference
Configured Instance Account
Balance=$1E6

55 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,


Object Variation Example

56 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,


Object Variations with Reuse

Previously defined
ConfiguredInstance artifact

57 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,


Example Artifact Hierarchy

58 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,


59 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts, Copyright 2000 SilverMark, Inc.
References and
Further Reading
[Beck 94] Kent Beck, Simple Smalltalk Testing. The Smalltalk Report 4(2):16-18,
October 1994

[Binder 99] Robert V. Binder, Testing Object-Oriented Systems: Models, Patterns,


and Tools. Addison-Wesley Object Technology Series. 1999

[Dwyer 99] Graham Dwyer and Graham Freeburn, Business Object Scenarios: a fifth-
generation approach to automated testing. In Software Test Automation : Effective Use of
Test Execution Tools. Edited by Mark Fewster and Dorothy Graham Addison-Wesley Pub
Co. 1999

Firesmith 96] Donald G. Firesmith, Pattern Language for Testing Object-Oriented


Software, Object Magazine, Vol. 5, No. 8, SIGS Publications Inc., New York, New York,
January 1996, pages 32-38.

[Gamma+ 95] Erich Gamma, Richard Helm, Ralph Johnson, John Vlissides, Design
Patterns : Elements of Reusable Object-Oriented Software. Addison-Wesley Professional
Computing. 1995

[Fowler 99] Martin Fowler, A UML Testing Framework. Software Development


Magazine. April, 1999

60 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,


References and
Further Reading
[Fewster 99] Mark Fewster, Dorothy Graham Software Test Automation :
Effective Use of Test Execution Tools. Addison-Wesley Pub Co. 1999

[Hetzel 84] Bill Hetzel, Software Testing, QED Information Sciences, Inc. 1984

[Jeffries 99] Ronald E. Jeffries, Extreme Testing, Software Testing and Quality
Engineering 1(2):23-26 March/April 1999

[Jorgenson 94] Paul C. Jorgenson and Carl Erickson,. Object-Oriented Integration Testing.
CACM, 37(9):30-38 September 1994

[Kaner 93] Cen Kaner, Jack Falk, Hung Quoc, Testing Computer Software, International
Thompson Computer Press 1993

[Marick 95] Brian Marick, The Craft of Software Testing, Prentice Hall. 1995

[McGregor 94] John D. McGregor, Testing Object-Oriented Systems Tutorial Notes In The
Conference on Object-
Oriented Systems, Languages and Applications, October 1994

[McGregor 96a] John D. McGregor and Timothy Korson, Integrating Object-Oriented


Testing and Development Processes. CACM, 37(9):59-77 September 1994

61 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,


References and
Further Reading
[McGregor 96b] John D. McGregor and A. Kare, PACT: An Architecture for Object-
Oriented Component Testing, in the Proceedings of The Ninth International Software
Quality Week in San Francisco, California, SR Institute, San Francisco, 22 May 1996.

[McGregor 99] John D. McGregor Test Patterns: Please Stand By, Journal of Object-
Oriented Programming, 12(3):14-19, June 1999

[Murphy 94] Gail C. Murphy, Paul Townsend, and Pok Sze Wong, Experiences With
Cluster and Class Testing. CACM, 37(9):39-47 September 1994

[Myers 79] Glenford Myers, The Art of Software Testing, John Wiley & Sons, Inc. 1979

[Poston 94] Robert M.Poston, Automated testing from Object Models. CACM, 37(9):48-58
September 1994.

[Siegel 96] Shel Siegel, Object Oriented Software Testing, a Hierarchical Approach. John
Wiley & Sons, Inc. 1996

[Siepmann 94] Ernst Siepmann, A. Richard Newton: TOBAC: A Test Case Browser for
Testing Object-Oriented Software.
ISSTA 1994: 154-168

62 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,


63 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts, Copyright 2000 SilverMark, Inc.