A Publication

PR it Te
BE CTIC ting
A s
Un

ST ES
:
VOLUME 5 • JANUARY 2008 • $8.95 • www.stpmag.com
The Future of Software Testing...

Save $300 With Our

Super-Early Bird Discount
Register Online by January 11

February 26–27, 2008

New York Hilton
New York City, NY

A BZ Media Event
 Brian Behlendorf on
Open Source

Rex Black on ROI

Jeff Feldstein on
Test Teams

Robert Martin on
Craftsmanship

Gary McGraw on
Security

Alan Page on
Centers of Excellence

Stretch your mind at

FutureTest 2008 — Robert Sabourin on
Just-In-Time Testing
an intense two-day
conference for executive
and senior-level
Joel Spolsky on
managers involved Software Testing

with software testing

and quality assurance.
Tony Wasserman on
Our nine visionary
the Future
keynotes and two
hard-hitting panel
Alan Zeichick moderates
discussions will inform two panel discussions on
Test Tools and on the
you, challenge you Application Life Cycle
and inspire you.

www.futuretest.net
 VOLUME 5 • ISSUE 1 • JANUARY 2008

Contents A Publication

12 COV ER STORY
Lights, Camera, ALM 2.0!
When it comes to life cycle management, with its newly automatic
synchronization of metadata and development artifacts, ALM 2.0 is already
a star—and the tester is the director. Roll ‘em! By Brian Carroll

You Can Gauge

18 Performance
Without Requirements
Hang onto your Web app’s users by
analyzing their acceptable wait limits.
Here’s how. By Alexander Podelko

Let Not Your Depar t ments

25 Project Become
A Tragedy of Errors
7 • Editorial
Why we use software to test software.

To bypass critical errors in testing, learn

8 • Contributors
from the flight control industry and
Get to know this month’s experts and the
mind your out-of-range values. best practices they preach.
By Yogananda Jeppu and
Ambalal Patel Offshore
30 Playbook:
Quality Audit 101
9 • Feedback
It’s your chance to tell us where to go.

10 • Out of the Box

New products for testers.
Your company could be looking over-
seas to cut costs and deadlines. Put the
35 • Best Practices
odds in your favor with a quality audit,
Will unit testing go mainstream? Based on
so that your team’s best practices will be
my editorial experience, never! By Geoff Koch
on the agenda—wherever you test.
By Steve Rabin and 38 • Future Test
John Bradway Sync up development with quality assurance
to boost productivity. By Adam Kolawa

JANUARY 2008 www.stpmag.com • 5

 Ed N otes

Using Software
VOLUME 5 • ISSUE 1 • JANUARY 2008
EDITORIAL
Editor Editorial Director
Edward J. Correia Alan Zeichick
+1-631-421-4158 x100 +1-650-359-4763

To Test Software
ecorreia@bzmedia.com alan@bzmedia.com

Copy Editor Contributing Editor

Laurie O’Connell Geoff Koch
loconnell@bzmedia.com koch.geoff@gmail.com

ART & PRODUCTION Happy New Year! As we software? The whole issue
Art Director Art /Production Assistant enter 2008, it’s time once reminds me of an absurdity
LuAnn T. Palazzo Erin Broadhurst
lpalazzo@bzmedia.com ebroadhurst@bzmedia.com
again to contemplate our I came upon while working
accomplishments of the year as a support technician for
SALES & MARKETING gone by, to consider our Windows magazine in the
Publisher goals for the year ahead, and 1990s. We were gathering
Ted Bahr for some, to ponder the requirements for the edito-
+1-631-421-4158 x101
ted@bzmedia.com
many paradoxes that vex rial and production network
our existence. from the staff, which insist-
Associate Publisher List Services
David Karp Lisa Fiske Why, for instance, do we ed on using Windows-based
+1-631-421-4158 x102 +1-631-479-2977 build software to test other hardware and software to
dkarp@bzmedia.com lfiske@bzmedia.com software? This question has Edward J. Correia publish the magazine.
never before occurred to me, nor does it “We’re a magazine about Windows, and
Advertising Traffic Reprints
Phyllis Oakes Lisa Abelson parallel such mysteries as people who are we need to be published on Windows”
+1-631-421-4158 x115 +1-516-379-7097 financially wealthy but short on values. was the philosophy.
poakes@bzmedia.com labelson@bzmedia.com But it does bear some discussion. And while that kind of idealism might
Director of Marketing Accounting
The idea was brought to me by testing have looked good on pages of the maga-
Marilyn Daly Viena Ludewig consultant Elfriede Dustin, who credits a zine, the state of desktop publishing on
+1-631-421-4158 x118 +1-631-421-4158 x110 conference-goer with reminding her of a the Windows platform at the time was
mdaly@bzmedia.com vludewig@bzmedia.com
concept she had pondered many times immature, to be generous. Their operat-
before. So why is it that we develop soft- ing system was Windows for Workgroups,
READER SERVICE
ware to test software? the first such installation in the company.
Director of Circulation Customer Service/
Agnes Vanek Subscriptions The practice of automating software My belief at the time was the same as it
+1-631-443-4158 +1-847-763-9692 testing itself involves a software develop- is today. The magazine should have used
avanek@bzmedia.com stpmag@halldata.com ment life cycle, complete with its own set the best tool available at the time, regard-
of requirements, a design and the actual less of its content. The parent company,
Cover Illustration by P. Avlen development and testing. And as Dustin now called CMP Media, made its bones
points out, the major advances in testing publishing dozens of periodicals, which
tools since the 1990s include the ability to at one time all used a mainframe-style
recognize object properties beyond their publishing system called Atex. And not a
x,y coordinates. This has made automa- single one of its publications was about
tion more viable because scripts can be Atex.
President BZ Media LLC more useful and less fragile. Why? Because one has nothing to do
Ted Bahr 7 High Street, Suite 407 The open source community also has with the other. “We want to eat our own
Huntington, NY 11743
Executive Vice President
+1-631-421-4158
emerged in the last 15 years as a prolific dog food,” they might have said. And for
Alan Zeichick
fax +1-631-421-4130 source of high-quality test automation Windows magazine to use Macintosh
www.bzmedia.com tools. As evidence, consider the FitNesse computers was unthinkable. “Ridic-
info@bzmedia.com
(fitnesse.org) acceptance testing frame- ulous,” I would have said (and probably
work, Watir (wtr.rubyforge.org) Ruby- did, in private). I lost that battle and
Software Test & Performance (ISSN- #1548-3460) is based automated browser testing librar- would ultimately not be involved in the
published monthly by BZ Media LLC, 7 High Street,
Suite 407, Huntington, NY, 11743. Periodicals postage ies, and Python and Perl. deployment. It was just as well, because
paid at Huntington, NY and additional offices.
Just last month, this magazine ran an the team struggled mightily.
Software Test & Performance is a registered trade-
mark of BZ Media LLC. All contents copyrighted excellent tutorial on building your own Software is very good at automating
2008 BZ Media LLC. All rights reserved. The price XML-based test automation framework. things. So when automated testing is the
of a one year subscription is US $49.95, $69.95 in
Canada, $99.95 elsewhere. Other open source test automation need, why not use the best tool for the
POSTMASTER: Send changes of address to Software frameworks are available, such as job? For the practice of automating soft-
Test & Performance, PO Box 2169, Skokie, IL 60076.
Software Test & Performance Subscribers Services STAF/STAX, which provides useful serv- ware testing, the best tool happens to be
may be reached at stpmag@halldata.com or by

staring you right in the face. ý

calling 1-847-763-9692.
ices you don’t have to build from scratch. more software. Sometimes the best tool is
Why shouldn’t software be used to test

JANUARY 2008 www.stpmag.com • 7

 Contributors

BRIAN CARROLL leads the Application Lifecycle Framework

project at Eclipse. His article, which begins on page 12, delves
into ALF and its implementation of ALM 2.0, pertaining to
the tester’s role.
Brian has been developing software professionally for 40
years, the last 25 of which have been focused on developer
tools and infrastructure. Brian has been a popular speaker
and project leader at GUIDE, the OMG and Eclipse. He
co-authored the original OASIS Web Services Distributed
Management (WSDM) specification. He’s also a Fellow at
Serena Software, which develops and markets ALM solutions.

ALEXANDER PODELKO’s engagements as an application

performance expert have included Aetna, Hyperion and
Intel. He is currently serving as a consulting member of
the technical staff at Oracle, part of Performance
Engineering for the Hyperion line of products, which
Oracle acquired in March.
Alexander’s article, which begins on page 18, draws from
his experience in the roles of performance tester, perform-
ance analyst, performance architect and performance engi-
neer to explain how a tester may establish performance
requirements in the absence of documented ones.

YOGANANDA JEPPU and AMBALAL PATEL are scientists at IFCS Aeronautical Development
Agency in Bangalore, India. Beginning on page 25, the colleagues take a Shakespearean
approach to reduction of defects
to help keep your project from
becoming a Tragedy of Errors.
Yogananda has many pub-
lished works, mostly relating to
real-time systems test method-
ologies for performance and
quality assurance in aeronautics-
industry control systems. In his
current post since 2000, Ambalal
holds degrees in mechanical
engineering and instrumentation and a Ph.D. in fuzzy control systems from the
Indian Institute of Technology, Khargapur.

STEVE RABIN has more than 20

years of experience designing and
delivering enterprise software. He’s
currently serving as CTO of soft-
ware and Internet venture capital
firm Insight Venture Partners.
He and JOHN BRADWAY, devel-
opment manager at project man-
agement software maker Primavera
Systems, explain how organizations
can test effectively offshore by keep-
ing the foreign processes in sync with those at home. Their article begins on page 30.

TO CONTACT AN AUTHOR, please send e-mail to feedback@bzmedia.com.

8 • Software Test & Performance JANUARY 2008

Feedback

humans when it comes to the creative

DEVELOP AND TEST process and methods such as explorato-
ry testing. We need software to test soft-
SIMULTANEOUSLY ware simply because, once it is written
properly, it increases the amount of test-
I would like to respond to Prakash Sodhani’s arti-
ing we can do with the limited amount
cle “Navigating Without Requirements” (Software of resources we have.
Test & Performance magazine, Dec. 2007) to suggest Comparing hardware testing to soft-
a much simpler and more effective way for tester to ware testing is an invalid comparison.
test applications being developed using the agile Hardware adheres to fixed rules (laws
of physics, etc.). If you think about it,
approach: namely, be an active participating member
a simulator is a combination of both
of the development team and translate the requirements into test cases at the exact hardware and software. Therefore,
same time as the developers are developing, using the exact same conversations with you’re testing hardware with hardware
the customers.The developers will be able to fix their bugs immediately instead of wait- tools, just as you test software with soft-
ware tools.
ing weeks to be told what the bugs are.
Dale L. Perry
Just because test cases are traditionally developed after the code is complete Tampa, FL
does not mean that this is the only or best way to develop test cases. What better way
to build quality in than to develop the tests in parallel instead of afterward? SOA’S BRAVE NEW WORLD?
Another advantage of the tester being actively involved from the beginning is that Regarding “Changing Tires on a Moving
Car” (Test & QA Report, Nov. 27, 2007),
they will sometimes spot issues with the requirements in the discussions with the cus-
capture and playback strategies have been
tomer that the developers miss. the basis of performance testing tools for
Steven Gordon, Ph.D. years. There is nothing new here. The
Scottsdale, AZ only difference is you are now capturing
and playing back a different stream—
SOAP messages instead of HTML, for
SOFTWARE FOR THE TESTING Communist. They can still grab and example. Rational and Mercury (and oth-
OF SOFTWARE? nationalize all our fabs. We don’t actual- ers) have had this technology in the mar-
“The Software Testing Paradox” by ly “own” anything in China. ket for years.
Edward J. Correia (Test & QA Report, Now if only they could outsource fee- They have also had the ability to vary
Dec. 4, 2007) is an interesting article. I ble-brained editors. That would be justice. the data to make it more realistic. Just
was not so startled by the question “Why Bill Baka playing hundreds of copies of exactly the
develop software to test software?” Via e-mail same message is hardly a good way to val-
We developed a software tool to inject idate an increased load/what-if scenario.
messages into our system under test, and THE HUMAN ADVANTAGE The statement “not having to under-
I do not think this is uncommon in oth- This is in response to Edward J. Correia’s stand the semantics of correctness” is just
er companies/projects. And as for the article “The Software Testing Paradox.” crap. Of course you do. If someone has
hardware world, think of an oscilloscope; There is a simple answer to this ques- changed the code you have to know if
Isn’t that a piece of hardware used to test tion “Why do we build software to use your test failed because it was because the
hardware? And further, think of a hard- for testing other software?” We use soft- code had changed and it was now cor-
ware production line, and consider cus- ware (automated testing tools) to test rectly doing something different, or the
tom-built hardware measuring tools, to software because it is impossible for peo- code had changed and the test has caught
check that the item is within spec. ple to approach effective testing manu- an incorrect behavior.
Mike Arnold ally. Manual testing requires lots of time Why is it every time anyone puts SOA
Cambridge, MA and resources, and is unreliable and can in a sentence, we are supposedly trans-
be very tedious and tiresome. Auto- ported to some brave new world?
SOLD OUT BY OUTSOURCING mated testing has the advantage of Mark McLaughlin
Regarding Edward J. Correia’s being very effective at repetitive tasks. Sydney, NSW, Australia
“Offshoring Strategy: Trust, but Verify” Many of the techniques we use to test
(Test & QA Report, Nov. 20, 2007), can- software are mathematical in nature FEEDBACK: Letters should include the writer’s
cel me. I’m not the CEO selling out his and can be easily automated, making name, city, state, company affiliation, e-mail
country—I’m the EE who got sold out. the task of test design simpler and more address and daytime phone number. Send your
Outsourcing only benefits the business efficient. thoughts to feedback@bzmedia.com. Letters
become the property of BZ Media and may be
execs, not the people who made the busi- However, with all their advantages,
edited for space and style.
ness. Outsource to China. They are still automated testing tools cannot replace

JANUARY 2008 www.stpmag.com • 9

 Out of t he Box

Hoping to Stir The

Heart of the Lonely
Exception Hunter
Red Gate, which specializes in tools for
Microsoft developer technologies, in
December released Exception Hunter,
a US$295-per-user analysis tool for .NET
that it claims can predict exceptions like-
ly to be thrown by methods before
they’re part of a finished application.
“Until now, developers have had to
wait until an error happened to find out
which method throws which exception,”
said Exception Hunter lead developer
Bart Read. “Exception Hunter finds and
reports exceptions early in the devel-
opment process, so they don’t cause
downstream problems for developers
and end users.”
The tool works by permitting the
developer or tester to select an assem-
Exception Hunter
from Red Gate can
identify exceptions
in .NET code before
the app is finished,
allowing handlers to
be written.
bly to analyze,
drilling down to
the method level and reviewing a list of patching after application develop-
the different types of exceptions that ment,” said Read.
can potentially be thrown. Exception Hunter 1.0 is available
With this information, the develop- now for Windows 2000 and later (includ-
er can then write the appropriate excep- ing Vista) and requires the .NET
tion handlers. Framework 2.0 and IE 6 or later. It also
“It’s all about an ounce of preven- couples with the code editing capabili-
tion, rather than the frustrating cure of ties of Visual Studio 2002 or later.

Coverity’s Prevent SQS Now Detects Race

Conditions as Cores Continue to Multiply
With a totally rewritten user interface,
Coverity’s flagship Prevent SQS static
code-analysis tool now can detect race
conditions in software writ-
ten for multicore and mul-
tiprocessor systems.
Released in December,
the latest version of the
scanning tool for C/C++
and Java can now detect the
three major types of code
flaws: race conditions, dead-
locks and thread blocks, the
company claims.
Race conditions occur
when multiple threads
access the same shared data
without suitable locks in
place to prevent accidental
overwrite, loss or corrup-
tion. Deadlocks are defined
as two or more threads that
chase each other’s locks in
Coverity’s Prevent SQS identifies race conditions, now through a new UI.
a circular chain, which usu-
ally leads to a system freeze or crash.
Performance-sapping thread blocks hap-
pen when one thread calls a long-run-
ning operation and blocks other threads
from making any progress.
As the number of processor cores
continues to rise in target sys-
tems, these hard-to-detect and
impossible-to-predict prob-
lems will become an increas-
ingly common issue for
testers. “Race conditions are
particularly difficult for devel-
opers because they are hard
to test for [and] nearly impos-
sible to replicate,” said
Coverity CTO Ben Chelf in a
news release. “The conse-
quences of a race condition in
the field can be disastrous.” In
some instances, they have
even been blamed for deaths
and widespread catastrophes.
Prevent SQS is available
now for all major platforms
and compilers; pricing is
based on project size.

10 • Software Test & Performance JANUARY 2008

 Understand Your
Role as a Tester
When ALM Is

Here’s Your Close-Up:

On the Scene

Get Ready for ALM 2.0

12 • Software Test & Performance

By Brian Carroll
f your organization is seeking
I an application life cycle man-
agement tool, you’d do well to
adopt one that supports ALM
2.0, which improves on prior versions
by automatically synchronizing the
metadata and development artifacts
among tools, while still allowing each
tool to manage much of its own data
storage. Automatic synchronizations
could include, for example, adjusting
the test plan to reflect changes in the
status of a requirement or an issue, or
launching a series of activities to
deploy a build into staging or produc-
tion once that deployment is approved.
Think of the automation of ALM as
replacing the “cut and paste” that
occurs today to keep development
tools in sync, though the newer spec
adds a lot more than that. ALM 2.0
helps all the stakeholders in the soft-
ware development process communi-
cate and collaborate more efficiently.
ALM frameworks provide the glue that
integrates stakeholders and their
development tools across the life cycle
and opens opportunities for process
improvement and quality improve-
ment through the use of better tools.
For more on ALM 2.0, see the sidebar
“ALM’s Second Coming.”
Eclipse Already There
The Eclipse development community is
often quick to exploit new specifications
or development ideas. For example, the
Eclipse Application Lifecycle
Framework (ALF) project (www.eclipse
.org/alf) already implements an open
source ALM 2.0 framework.
Though still in the incubation phase,
ALF provides event-driven tool integra-
tion and orchestration using standards
such as Web services and BPEL, the
OASIS Web Services Business Process
Execution Language OASIS Web
Services Business Process Execution
Language.
Brian Carroll is lead developer on the Eclipse
Application Lifecycle Framework (ALF)
project and a Serena Fellow.
JANUARY 2008
ALF also provides a single sign-on
capability that allows a user’s identity to
be passed securely to all the tools that
participate in an orchestration.
This article explains how ALM sys-
tems—using ALF as a model—can give
development teams, and specifically
testers, the tools they need to perform
most effectively.
To provide development teams with
flexibility and responsiveness, ALF
employs the notion of Event Driven
Architecture (EDA), which allows
•
ALM 2.0 helps
all the stakeholders
in the software
development process
communicate and
collaborate more
efficiently.
• organization upgrades a tool in its
tools to emit an event when something
changes within a tool that may affect
related data in another tool (for exam-
ple, once a build has been completed
or a bug has been approved to be fixed
in the current release).
The events go through the ALF
Event Manager, which filters the
events of interest and routes them to
the appropriate BPEL process. A BPEL
process is like a flowchart that indi-
cates the sequence and data to be
passed among tools to keep them in
sync. ALF also contains a standards-
based identity capability (using WS-
Security, WS-Trust, WS-Federation and
SAML Assertions) that conveys the
identity of the user making the change
that triggered an event to be passed
through the BPEL process to all the
relevant tools. As a result, all the tools
are updated to reflect the change.
ALF also provides common services
that make gluing the tools together eas-
ier. These services include engines to
send e-mail notices to team members
when certain events occur, logging serv-
ices to create an audit trail of the devel-
opment activities, and relationship serv-
ices to establish relationships among
artifacts stored within different tools.
Finally, ALF provides a set of best
practices for tool integration and
vocabularies. Vocabularies define the
core data structures, events and servic-
es that tools should expose in different
stages of the life cycle. An ALF vocab-
ulary describes the essential events
and services for configuration man-
agement, requirements, testing, etc.
Vocabularies make it simpler to
define orchestrations and substitute
development tools with minimal dis-
ruption to the tool integrations.
However, ALF also will work if tools do
not support the vocabularies.
Based on standards and implement-
ed as open source, ALF eliminates the
nightmare that happens when an
development stack and the proprietary
point-to-point integrations break.
This can often bring development
to a grinding halt. Not only is the
underlying source code of the interop-
erability platform available to the
shop, but the logic of the integrations
is expressed in the high-level standard
workflow language of BPEL. The
development team (most likely the
configuration management team) can
make the fixes themselves without hav-
ing to wait for the vendor to get a
patch into the next maintenance
release cycle.
ALM on Testing
Another positive byproduct of ALM
2.0 is that it will make testers’ lives eas-
www.stpmag.com • 13
 ALM 2 IN ACTION

-
es pro
t e ch nologi SDL
t SOA hen W
a b il i ties tha concepts, t and imple-
cap OA of t
rn the ning S nceive opmen
5. Lea art by lear g able to co your devel rhaps
t in r pe
sid er vide. S n BPEL. Be egration” fo easier and
o n e n t b
this, c - and th e “killer i ke your jo .
y p r actice iew to partic th
ment ation will m e recogniti
a on
d
lrea d its purv vel- to
u d on’t a
xte n ftw a r e d e
or g a n iz
y o u so m
t i s c ritical s
o e so o n er e n ge
1. If y he QA team your shop’s ALM soluti e e v en garn m e a surem which chan le
t f r n es , mp
having the design o erwise, you ss, such as o e l l i n itiativ and knowin porating si to
g
n h c e b a r ols
ipate i process. Ot flawed pro changes to 6 .
h s
As wit the proces t ar t by inc n PPM to are
o
n t g a n ving S blow
opme automatin ish betwee an review. impro d which hu orate full- ocess and
r t. QA
e u m
could b esn’t disting that need hu a n
help , then inc o r p
pmen t p r
ting
o
that d ed and thos
e
b y propaga ust r e p o r ts
y o u r develo
at y
autom ficienc eam m how g to
in cr e ase ef ut the QA t ke sense. track ng.
i nt P rocessin
b ma v
2.0 ca
n
ng too
ls,
re they impro ex Eve cess.
2. ALM hanges amo ools to ensu ly in g Compl lopment pro
e
c
simple links among
t of er app he dev ool
r e e x ecution de 7 . Consid quality of t n ’ t like. T -
o t h e o o
monit tomate ch as c
o
improv
e th s you
d
d not-s
2 .0 can au ting tools, su eric code k w i t h tool of great an est-of-
LM ra en uc ix “b
ause A incorpo more g es to get st in a m g to to
3. Bec ols, consider anners and d the resourc 8 . Don’ t uently cont a es switchin rs that tr y
f to s c h a f r e q m a k e n d o h o se
sets o vulnerability ay not have suites ools. ALM w
v
ar y of yourself, “Wame-
it y u m t . B e k
secur ols, that yo s. great tools easier latform. As priet ar y fr
to
quality reviously. s it p roduce p
b r e e d ” i n t o t h e i r ve d b y a p
ro
o r t p s y stem o n ’ t be y o u e r
sup p t he d. D lock s
well as planne level; rest is
pro c ess as ons work as a crooked y’re b e st inte
the rati sing the ”
4. Test re the integ ilds walls u s behave as g and work?
a k e su w h o b u t ra ti o n h a n dli n
M n es r
e maso ur orch to erro
like th nsure that yo cial attention
e e
test to to, giving sp
o se d
supp ns. propagates changes (for example, a
onditio
edge c requirement is deleted) once those
ment. The expected improvement in links are established, but if there are
system quality resulting from that syn- changes to the wording of a require-
chronization is what sells the technolo- ment that subtly change its meaning,
ier. With the new spec, tool integration gy to development managers. the automated transformation rules
is based on the development process, ALM 2.0 can eliminate the rote work won’t catch that change. So the human
such that updates to project require- needed to synchronize data across tools, intelligence of a QA team is still needed
ments are synchronized across all the but tool integrations are generally driv- to interpret a requirement and deter-
tools used in the development cycle. So en by mechanical transformation rules mine which tests are appropriate to ver-
when a requirement is deferred to a and don’t yet understand the semantics ify the requirement has been satisfied
later release, the test management tool (or meaning) of development artifacts. (Director’s Note 1).
will have been automatically notified to Perhaps such understanding will come Certain changes in requirements or
disable tests related to that require- in ALM 3.0. For now, the current spec bug reports will require changes in

14 • Software Test & Performance JANUARY 2008


application code and the tests them-
selves. Improved communication
between tools—and therefore all the
roles in development—may make meta-
data more accessible to testers, but craft-
ing a flexible test and identifying edge
conditions to be tested still requires the
skills of a journeyman tester.
However, with ALM 2.0, the QA
department will have more time to
design and craft tests, and spend less time
on creating custom automations and
reporting on the tests. ALM integrations
will perform those tasks (Director’s
Note 2).
ALM also impacts how integrations
among tools are expressed. Today,
scripting with command-line inter-
faces is the common way to tie togeth-
er a sequence of tools. However, it’s
difficult to integrate tools that run on
different platforms (for example,
Linux or the mainframe), or expose
their interfaces over the Web rather
than the command line.
With ALM 2.0, BPEL can sequence
and integrate tools that run on differ-
ent platforms and that don’t expose
command-line interfaces at all. And
the orchestration can be expressed
using a BPML or BPEL editor rather
than as script code using a text editor.
Unit and Component Testing
The development of unit tests will be
largely unaffected by ALM 2.0.
However, unit tests will be incorporat-
ed into much larger sequences of test-
ing as part of the continuous integra-
tion movement. Why run unit tests
continuously when you can’t incorpo-
rate code scanning, security scanning
and other types of quality assurance
into the process?
An early ALF prototype incorporat-
ed security scanning with traditional
test management, and combined the
reporting from both into a single
“Deploy to Production” issue and
report. It has long been a strange
irony of our industry that the auto-
mated test suite is run manually. With
ALM 2.0, the automated test suite can
run automatically after, say, a success-
ful build (Director’s Note 3).
Integration vs. Application Testing
While the focus of testing will be on
the application delivered to the end
user, the development processes and
its integrations will become more
inclusive and, therefore, will also need
JANUARY 2008
ALM 2 IN ACTION
A
LM 2.0: THE SECOND COMING
The following is an excerpt from Carey Schwaber’s August 2006 Forrester report that
introduced the term ALM 2.0.
Tomorrow’s ALM is a platform for the coordination and management of development activ-
ities, not a collection of life-cycle tools with locked-in and limited ALM features.These plat-
forms are the result of purposeful design rather than rework following acquisitions. The
architectural ingredients of ALM 2.0 are:
· Practitioner tools assembled out of plug-ins. An à la carte approach to product packag-
ing provides customers with simpler, cheaper tools. Far from being a pipe dream, this
approach is a reality today. IBM has done the most to exploit this concept, currently pro-
viding many different grades of development and modeling tools that are all available as
perspectives in Eclipse, as well as the ability to install only selected features packs in each
of these tools.
This approach has not yet been successfully applied outside of development and modeling
tools. For example, today, customers must choose between defect management that’s too
tightly coupled with test management and software configuration management (SCM) or
defect management in a stand-alone tool.
· Common services available across practitioner tools. Vendors are identifying features
that should be available from within multiple practitioner tools—notably, collaboration,
workflow, security and reporting and analytics—and driving them into the ALM platform.
Telelogic has started to make progress on this front for administrative functionality like
licensing and installation. Microsoft has gone even further: Visual Studio Team System
leverages SharePoint Server for collaboration and Active Directory for authentication,
and because it uses SQL Server as its data store, it can leverage SQL Server Analysis
Services and SQL Server Report Builder for reporting and analytics.
· Repository neutrality. At the center of ALM 2.0 sits not one repository, but many. Instead
of requiring use of the vendor’s SCM solution for storage of all life cycle assets, tomor-
row’s ALM will be truly repository-neutral, with close to functional parity, no matter
where assets reside. IBM, for example, has announced that in coming years, its ALM solu-
tion will integrate with a wide variety of repositories, including open source version-con-
trol tools like Concurrent Versions System (CVS) and Subversion. This will drive down
ALM implementation costs by removing the need to migrate assets—a major obstacle for
many shops—and will bring development on mainframe, midrange, and distributed plat-
forms into the same ALM fold.
· Use of open integration standards. Two means of integration—use of Web services APIs
and use of industry standards for integration—will ease and deepen integration between a
single vendor’s tools, as well as between its tools and third-party tools. Many vendors still
don’t offer Web-services-based APIs, but this will change with time. In addition, new stan-
dards for life-cycle integration, including Eclipse projects like the Test and Performance
Tools Project (TPTP) and Mylar, promise to simplify tools integration. One case in point:
SPI Dynamics reports that its integration with IBM Rational ClearQuest Test Manager
took one-third of the time it would have taken if both tools didn’t leverage TPTP.
· Microprocesses and macroprocesses governed by externalized workflow. The ability to
create and manage executable application development process descriptions is one of the
big wins for ALM 2.0. When processes are stored in readable formats like XML files, they
can be versioned, audited and reported upon.This facilitates incremental process improve-
ment efforts and the application of common process components across otherwise discrete
processes. For example, Microsoft Visual Studio Team System process templates are
implemented in XML and contain work-item-type definitions, permissions, project struc-
ture, a project portal and a version control structure.
There is no solution on the market that possesses all of these characteristics, but this is the
direction in which most vendors are moving. However, it will be at least two years before
any vendor offers a solution that truly fulfills the vision of ALM 2.0.
Source: Forrester Research, Inc.
www.stpmag.com • 15
 ALM 2 IN ACTION
to be tested
(Director’s Note 4).
New Skills Needed
SOA is perhaps the most commonly
flaunted buzzword in the industry
today. And for good reason: Beyond
the hype, it’s clear that SOA provides a
platform-independent way to
integrate different software sys-
tems that previously were diffi-
cult or impossible to integrate.
Those systems could be end-
user applications or even sys-
tems of development and test
tools. ALF uses Web services as
the primary interface descrip-
tion and communication mech-
anism.
That means that the WSDL
describes the events and services
of tool interfaces, and BPEL is
used to describe process automa-
tions. And why not? Software
development has been building
SOA-based systems for the business; why
not apply SOA to improve the way soft-
ware development operates? Shouldn’t
the shoemaker’s children have shoes?
(Director’s Note 5)
16 • Software Test & Performance
Management Visibility
ALM 2.0’s automation capabilities pro-
vide the potential for providing QA and
management with far greater visibility
into the development process. Orchestra-
tions can gather data from tools and
refine them for consumption by Project
Portfolio Management (PPM) tools for
• able today in the Eclipse ALF
Open source ALM 2.0 is available
today in the Eclipse ALF project.
•
presentation by QA and development
managers (Director’s Note 6).
Use Complex Event Processing
Prior to ALM 2.0, logged information
was scattered in silos maintained by
each tool. The ALM 2.0 framework can
create a central log of activities that can
be fertile ground for insights that can
improve your processes.
Tools exist that use Complex Event
Processing to look for patterns in
event logs. For example, you may be
able to identify a pattern in which
changes made by a certain developer
or using a certain tool led to a failure
when running automated tests.
Analyzing the logs for such patterns
may lead to insight that can improve
the overall process quality (Director’s
Note 7).
Easier Tool Changes
ALF vocabularies provide another
dimension to the flexibility that ALM
2.0 provides. The idea is that switching
tools today can be a painful experi-
ence—history can get lost, not all data
can be migrated, and integrations will
have to be redone.
ALF vocabularies are developed in
an open and transparent process by
sets of companies agreeing on the
essential data that each tool exposes,
clearly defining the meaning of that
data and segregating essential, com-
mon data from tool-specific data.
Integrations can then be expressed as
a core using the common data elements
and extended to leverage any tool-specif-
ic data. Such an approach increases the
plug-ability of tools. Switching tools then
involves only changes to the tool-specific
data, and then only if that data is con-
sumed by other tools (Director’s Note 8).
Get Started Now
Open source ALM 2.0 is avail-
project. It provides infrastruc-
ture for event routing, guid-
ance for process automation
using the BPEL engine of your
choice and authentication, as
well as conveying identity
among tools integrated with
Web services.
It also provides some vocab-
ularies—and you can partici-
pate in developing new vocabu-
laries for the tools you use, or
try your hand at extending
some existing ones. Download ALF
from www.eclipse.org/alf and get a
head start on improving your develop-
ductivity today. ý
ment organization’s quality and pro-
JANUARY 2008
 By Alexander Podelko

efining performance requirements is

D an important part of system design
and development. If there are no written perform-
ance requirements, it means that they exist in the
heads of stakeholders, but nobody bothered to
write them down and make sure that everybody
agrees on them.
Exactly what is specified may vary significantly
depending on the system and environment, but
all requirements should be quantitative and meas-
urable. Performance requirements are the main
input for performance testing (where they are
verified), as well as capacity planning and pro-
duction monitoring.
There are several classes of performance
requirements. Most traditional are response time
(how fast the system can handle individual
requests) and throughput (how many requests
the system can handle). All classes are vital: Good
throughput with a long response time often is
unacceptable, as is good response time with low
throughput.
Response time (in the case of interactive work) or

processing time (in the case of batch jobs or sched-

uled activities) defines how fast requests should be
processed. Acceptable response times should be
defined in each particular case. A time of 30 min-
utes could be excellent for a big batch job, but
absolutely unacceptable for accessing a Web page
in a customer portal. Response time depends on
workload, so you must define conditions under
which specific response times should be
achieved; for example, a single user, average load or
peak load.
Significant research has been done to define
what the response time should be for interactive sys-
tems, mainly from two points of view: what response
time is necessary to achieve optimal user’s perform-
ance (for tasks like entering orders) and what
response time is necessary to avoid Web site aban-
donment (for the Internet). Most researchers
agreed that for most interactive applications, there
is no point in making the response time faster than
one to two seconds, and it’s helpful to provide an
indicator (like a progress bar) if it takes more than
eight to 10 seconds.
The service/stored procedure response-time
requirement should be determined by its share in
the end-to-end performance budget. In this way,
the worst-possible combination of all required serv-
ices, middleware and presentation layer overheads
will provide the required time. For example, with a
Web page with 10 drop-down boxes calling 10 sepa-
rate services, the response time objective for each
service may be 0.2 seconds to get three seconds
average response time (leaving one second for net-
work, presentation and rendering).
Response times for each individual transaction
vary, so use some aggregate values when specify-
Alexander Podelko is a software consultant currently
engaged by Oracle.

18 • Software Test & Performance JANUARY 2008

ing performance requirements, such as averages or per-
centiles (for example, 90 percent of response times are less
than X). Maximum/timeout times should be provided also,
as necessary.
For batch jobs, remember to specify all schedule-related
information, including frequency (how often the job will be
run), time window, dependency on other jobs and dependent
jobs (and their respective time windows to see how changes in
one job may impact others).
Throughput is the rate at which incoming requests are com-
JANUARY 2008
pleted. Throughput defines the load on the system and is
measured in operations per time period. It may be the num-
ber of transactions per second or the number of adjudicated
claims per hour.
Defining throughput may be pretty straightforward for a
system doing the same type of business operations all the time,
processing orders or printing reports. It may be more difficult
for systems with complex workloads: The ratio of different
types of requests can change with the time and season.
It’s also important to observe how throughput varies with
www.stpmag.com • 19
 GAUGING PERFORMANCE
time. For example, throughput can be
defined for a typical hour, peak hour
and non-peak hour for each particular
kind of load. In some cases, you’ll need
to further detail what the load is hour-
by-hour.
The number of users doesn’t, by
itself, define throughput. Without
defining what each user is doing and
how intensely (i.e., throughput for one
user), the number of users doesn’t
make much sense as a measure of load.
For example, if 500 users are each run-
ning one short query each minute, we
have throughput of 30,000 queries per
hour. If the same 500 users are running
the same queries, but only one query
per hour, the throughput is 500 queries
per hour. So there may be the same 500
users, but a 60X difference between
loads (and at least the same difference
in hardware requirements for the appli-
cation—probably more, considering
that not many systems achieve linear
scalability).
Response Times:
Review of Research
As long ago as 1968, Robert B. Miller’s
paper “Response Time in Man-
Computer Conversational Transactions”
described three threshold levels of
human attention1. J. Nielsen believes
that Miller’s guidelines are fundamental
for human-computer interaction, so
they are still valid and not likely to
change with whatever technology comes
next 2. These three thresholds are:
• Users view response time as instanta-
neous (0.1-0.2 second): They feel
that they directly manipulate
objects in the user interface; for
example, the time from the
moment the user selects a column
in a table until that column high-
lights or the time between typing a
symbol and its appearance on the
screen. Miller reported that thresh-
old as 0.1 seconds. According to P.
Bickford, 0.2 second forms the
mental boundary between events
that seem to happen together and
those that appear as echoes of each
other 3.
Although it’s a quite important
threshold, it’s often beyond the reach of
application developers. That kind of
interaction is provided by operating sys-
tem, browser or interface libraries, and
usually happens on the client side with-
out interaction with servers (except for
dumb terminals, that is rather an excep-
20 • Software Test & Performance
tion for business systems today).
• Users feel they are interacting freely
with the information (1-5 seconds):
They notice the delay, but feel the
computer is “working” on the com-
mand. The user’s flow of thought
stays uninterrupted.
Miller reported this threshold as one
second. Using the research that was avail-
able to them, several authors recom-
mended that the computer should
respond to users within two seconds 1, 4, 5.
Another research team reported that
with most data entry tasks, there was no
advantage of having response times faster
than one second, and found a linear
decrease in productivity with slower
•
An animated watch
cursor was good
for more than a
minute, and a
progress bar kept
users waiting
until the end.
• face (for example, a percent-done
response times (from one to five sec-
onds)6. With problem-solving tasks, which
are more like Web interaction tasks, they
found no reliable effect on user produc-
tivity up to a five-second delay.
The complexity of the user interface
and the number of elements on the
screen both impact thresholds. Back in
1960s through 1980s, the terminal inter-
face was rather simple, and a typical task
was data entry, often one element at a
time. Most earlier researchers reported
that one to two seconds was the thresh-
old to keep maximal productivity.
Modern complex user interfaces with
many elements may have higher
response times without adversely
impacting user productivity. According
to Scott Barber, even users who are
accustomed to a sub-second response
time on a client/server system are
happy with a three-second response
time from a Web-based application7.
P. Sevcik identified two key factors
impacting this threshold8: the number of
elements viewed and the repetitiveness
of the task. The number of elements
viewed is the number of items, fields,
paragraphs etc. that the user looks at.
The amount of time the user is willing to
wait appears to be a function of the per-
ceived complexity of the request.
Users also interact with applications
at a certain pace depending on how
repetitive each task is. Some are highly
repetitive; others require the user to
think and make choices before pro-
ceeding to the next screen. The more
repetitive the task, the better the
expected response time.
That is the threshold that gives us
response-time usability goals for most
user-interactive applications. Response
times above this threshold degrade
productivity. Exact numbers depend
on many difficult-to-formalize factors,
such as the number and types of ele-
ments viewed or repetitiveness of the
task, but a goal of three to five seconds
is reasonable for most typical business
applications.
• Users are focused on the dialog (8+
seconds): They keep their attention
on the task. Miller reported this
threshold as 10 seconds. Anything
slower needs a proper user inter-
indicator as well as a clear way for
the user to interrupt the opera-
tion). Users will probably need to
reorient themselves when they
return to the task after a delay
above this threshold, so productivi-
ty suffers.
A Closer Look At
User Reactions
Peter Bickford investigated user reac-
tions when, after 27 almost instanta-
neous responses, there was a two-
minute wait loop for the 28th time for
the same operation. It took only 8.5
seconds for half the subjects to either
walk out or hit the reboot. Switching
to a watch cursor during the wait
delayed the subject’s departure for
about 20 seconds. An animated watch
cursor was good for more than a
minute, and a progress bar kept users
waiting until the end.
JANUARY 2008

Bickford’s results were widely used
for setting response times requirements
for Web applications. C. Loosley, for
example, wrote, “In 1997, Peter
Bickford’s landmark paper, ‘Worth the
Wait?’ reported research in which half
the users abandoned Web pages after a
wait of 8.5 seconds. Bickford’s paper
was quoted whenever Web site
performance was discussed,
and the ‘eight-second rule’
soon took on a life of its own as
a universal rule of Web site
design.”
A. Bouch attempted to iden-
tify how long users would wait
for pages to load 10. Users were
presented with Web pages that
had predetermined delays
ranging from two to 73 sec-
onds. While performing the
task, users rated the latency
(delay) for each page they
accessed as high, average or
poor. Latency was defined as
the delay between a request for
a Web page and the moment
when the page was fully rendered. The
Bouch team reported the following rat-
ings:
Good Up to 5 seconds
Average From 6 to 10 seconds
Poor More than 10 seconds
In a second study, when users
experienced a page-loading delay
that was unacceptable, they pressed a
button labeled “Increase Quality.”
The overall average time before
pressing the “Increase Quality” but-
ton was 8.6 seconds.
In a third study, the Web pages
loaded incrementally with the banner
first, text next and graphics last. Under
these conditions, users were much
more tolerant of longer latencies. The
test subjects rated the delay as “good”
with latencies up to 39 seconds, and
“poor” for those more than 56 seconds.
This is the threshold that gives us
response-time usability requirements
for most user-interactive applications.
Response times above this threshold
cause users to lose focus and lead to
frustration. Exact numbers vary signif-
icantly depending on the interface
used, but it looks like response time
should not be more than eight to 10
seconds in most cases. Still, the thresh-
old shouldn’t be applied blindly; in
many cases, significantly higher
response times may be acceptable
when appropriate user interface is
JANUARY 2008
implemented to alleviate the problem.
Not-So-Traditional Performance
Requirements
While they’re considered traditional
and absolutely necessary for some
kind of systems and environments,
some requirements are often missed
• increase throughput 10 times
When resource requirements are
measured as resource utilization,
it’s related to a particular
hardware configuration.
•
or not elaborated enough for interac-
tive distributed systems.
Concurrency is the number of simul-
taneous users or threads. It’s impor-
tant: Connected but inactive users still
hold some resources. For example, the
requirement may be to support up to
300 active users, but the terminology
used to describe the number of users
is somewhat vague. Typically, three
metrics are used:
• Total or named users. All registered
or potential users. This is a metric
of data the system works with. It
also indicates the upper potential
limit of concurrency.
• Active or concurrent users. Users
logged in at a specific moment of
time. This is the real measure of
concurrency in the sense it’s used
here.
• Really concurrent. Users actually
running requests at the same
time. While that metric looks
appealing and is used quite often,
it’s almost impossible to measure
and rather confusing: the num-
ber of “really concurrent”
requests depends on the process-
ing time for this request. For
example, let’s assume that we got
a requirement to support up to 20
“concurrent” users. If one request
takes 10 seconds, 20 “concurrent”
requests mean throughput of 120
GAUGING PERFORMANCE
requests per minute. But here we
get an absurd situation that if we
improve processing time from 10
to one second and keep the same
throughput, we miss our require-
ment because we have only two
“concurrent” users.
To support 20 “concurrent” users
with a one-second response
time, you really need to
to 1,200 requests per minute.
It’s important to under-
stand what users you’re dis-
cussing: The difference
between each of these three
metrics for some systems may
be drastic. Of course, it
depends heavily on the nature
of the system.
Performance and Resource
Utilization
The number of online users
(the number of parallel ses-
sion) looks like the best metric
for concurrency (complement-
ing throughput and response time
requirements). Finding the number of
concurrent users for a new system can
be tricky, but information about real
usage of similar systems can help to
make the first estimate.
Resources. The amount of available
hardware resources is usually a vari-
able at the beginning of the design
process. The main groups of resources
are CPU, I/O, memory and network.
When resource requirements are
measured as resource utilization, it’s
related to a particular hardware con-
figuration. It’s a good metric when the
hardware the system will run on is
known. Often such requirements are a
part of a generic policy; for example,
that CPU utilization should be below
70 percent. Such requirements won’t
be very useful if the system deploys on
different hardware configurations,
and especially for “off-the-shelf” soft-
ware.
When specified in absolute values,
like the number of instructions to exe-
cute or the number of I/O per trans-
action (as sometimes used, for exam-
ple, for modeling), it may be consid-
ered as a performance metric of the
software itself, without binding it to a
particular hardware configuration.
In the mainframe world, MIPS was
often used as a metric for CPU con-
sumption, but I’m not aware of such a
www.stpmag.com • 21
 GAUGING PERFORMANCE
widely used metric in the distributed
systems world.
The importance of resource-related
requirements will increase again with
the trends of virtualization and service-
oriented architectures. When you
depart from the “server(s) per applica-
tion” model, it becomes difficult to
specify requirements as resource utiliza-
tion, as each application
will add only incremental-
ly to resource utilization
for each service used.
Scalability is a system’s
ability to meet the per-
formance requirements as
• goals and requirements
the demand increases Using multiple
(usually by adding hard-
ware). Scalability require- performance
ments may include
demand projections such metrics that only
as an increasing number
of users, transaction vol- together provide
umes, data sizes or adding
new workloads. the full picture
From a performance
requirements perspective,
scalability means that you
can complicate
should specify perform-
ance requirements not
your process.
only for one configura-
tion point, but as a func-
tion, for example, of load
or data.
For example, the
• to five seconds, and
requirement may be to
support throughput
increase from five to 10 transactions per
second over the next two years, with
response time degradation not more
than 10 percent. Most scalability
requirements I’ve seen look like “to sup-
port throughput increase from five to
10 transactions per second over next
two years without response time degra-
dation”—that’s possible only with addi-
tion of hardware resources.
Other contexts. It’s very difficult to
consider performance (and, there-
fore, performance requirements)
without context. It depends, for exam-
ple, on hardware resources provided,
the volume of data operated on and
the functionality included in the sys-
tem. So if any of that information is
known, it should be specified in the re-
quirements.
While the hardware configuration
may be determined during the design
stage, the volume of data to keep is
usually determined by the business
and should be specified.
22 • Software Test & Performance
The Difference Between Goals
And Requirements
One issue, as Barber notes, is goals versus
requirements11. Most response time
“requirements” (and sometimes other
kinds of performance requirements) are
goals (and sometimes even dreams), not
requirements: something that we want to
achieve, but missing them won’t neces-
sarily prevent deploying
the system.
You may have both
for each of the perform-
ance metrics, but for
some metrics/systems
,they are so close that
from the practical point
of view, you can use one.
Still, in many cases, espe-
cially for response times,
there’s a big difference
between goals and
requirements (the point
when stakeholders agree
that the system can’t go
into production with
such performance).
For many interactive
Web applications, re-
sponse time goals are two
requirements may be
somewhere between eight
seconds and one minute.
One approach may be
to define both goals and
requirements. The problem? Require-
ments are very difficult to get. Even if
stakeholders can define performance
requirements, quite often go/no-go
decisions are based not on the real
requirements, but rather on second-tier
goals.
In addition, using multiple per-
formance metrics that only together
provide the full picture can compli-
cate your process. For example, you
may state that you have a 10-second
requirement and you took 15 seconds
under full load. But what if you know
that this full load is the high load on
the busiest day of year, that the max
load for other days falls below 10 sec-
onds, and you see that it is CPU-con-
strained and may be fixed by a hard-
ware upgrade?
Real response time requirements
are so environment- and business-
dependent that for many applications,
it’s cruel to force people to make hard
decisions in advance for each possible
combination of circumstances. In-
stead, specify goals (making sure that
they make sense) and only then, if
they’re not met, make the decision
about what to do with all the informa-
tion available.
Knowing What Metrics to Use
Another question is how to specify
response time requirements or goals.
For example, such metrics as average,
max, different kinds of percentiles and
median can be used. Percentiles are
more typical in SLAs (service-level
agreements). For example, “99.5 per-
cent of all transactions should have a
response time less than five seconds.”
While that may be sufficient for
most systems, it doesn’t answer all
questions. What happens with the
remaining 0.5 percent? Does this 0.5
percent of transactions finish in six to
seven seconds or do all of them time
out? You may need to specify a combi-
nation of requirements: for example,
80 percent below four seconds, 99.5
percent below six seconds, 99.99 per-
cent below 15 seconds (especially if we
know that the difference in perform-
ance is defined by distribution of
underlying data). Other examples may
be average four seconds and max 12
seconds, or average four seconds and
99 percent below 10 seconds.
Things get more complicated when
there are many different types of trans-
actions, but a combination of per-
centile-based performance and avail-
ability metrics usually works fine for
interactive systems. While more
sophisticated metrics may be necessary
for some systems, in most cases sophis-
tication can make the process over-
complicated and difficult to analyze.
There are efforts to make an
objective user-satisfaction metric. One
is Application Performance Index
(www.Apdex.org). Apdex is a single
metric of user satisfaction with the per-
formance of enterprise applications.
The Apdex metric is a number
between 0 and 1, where 0 means that
no users were satisfied, and 1 means all
users were satisfied.
The approach introduces three
groups of users: satisfied, tolerating
and frustrated. Two major parameters
are introduced: threshold response
times between satisfied and tolerating
users T, and between tolerating and
frustrated users F 12. There probably is
a relationship between T and the
JANUARY 2008

TABLE 1: THE SEVCIK METHODS
1. Default value (the Apdex methodology
suggests 4 seconds)
2. Empirical data
3. User behavior model (number of
elements viewed/task repetitiveness)
4. Outside references
5. Observing the user
response time goal and between F and
the response time requirement.
Where Do Performance
Requirements Come From?
If you look at performance require-
ments from another point of view, you
can classify them into business, usabil-
ity and technological requirements.
Business requirements come directly
from the business and may be cap-
tured very early in the project life
cycle, before design starts. For a
requirement such as ”A customer rep-
resentative should enter 20 requests
per hour, and the system should sup-
port up to 1,000 customer representa-
tives,” requests should be processed in
five minutes on average, throughput
would be up to 20,000 requests per
hour, and there could be up to 1,000
parallel sessions.
The main trap here is to immedi-
ately link business requirements to a
specific design, technology or usability
requirement, thus limiting the num-
ber of available design choices. If we
consider a Web system, for example,
it’s probably possible to squeeze all the
information into a single page or have
a sequence of two dozen screens. All
information can be saved at once, or
each page of these two dozen can be
JANUARY 2008
GAUGING PERFORMANCE
the chosen design.
For example, if we need to call 10
Web services sequentially to show the
6. Controlled performance experiment
Web page with a three-second response
7. Best time multiple
time, the sum of response times of each
8. Find frustration threshold F first and
Web service, the time to create the Web
calculate T from F (the Apdex method-
page, transfer it through the network
ology assumes that F = 4T)
and render it in a browser should be
9. Interview stakeholders
below three seconds. That may be trans-
10. Mathematical inflection point lated into response-time requirements
of 200-250 milliseconds for each Web
saved separately. We have the same service.
business requirements, but response The more we know, the more accu-
times per page and the number of rately we can apportion overall response
pages per hour would be different. time to Web services. Another example of
Usability requirements (mainly technological requirements can be found
related to response times) also figure in the resource consumption re-
into the performance equation. Many quirements. In its simplest form, CPU
researchers agree that users lose focus and memory utilization should be below
if response times are 70 percent for the chosen
more than eight to 10 hardware configuration.
seconds, and that Business requirements
response times should be
two to five seconds for
maximum productivity.
These usability consider-
• should be elaborated dur-
ing design and develop-
ment, and merge togeth-
er with usability and tech-
ations may influence Small nological requirements
design choices (such as into the final perform-
using several Web pages anomalies ance requirements, which
instead of one). In some can be verified during
cases, usability require- from expected testing and monitored in
ments are linked closely production. The main
to business require- behavior are reason that we separate
ments; for example, these categories is to
make sure that your sys- often signs understand where the
requirement comes from:
of bigger Is it a fundamental busi-
ness requirement or a
problems. result of a design decision
that may be changed if
necessary?
• Determining specific
performance require-
ments is another large
topic that is difficult to for-
malize. Consider the
t e m ’ s approach suggested by Sevcik for find-
response ing T, the threshold between satisfied
times aren’t and tolerating users. T is the main
worse than parameter of the Apdex (Application
response Performance Index) methodology, pro-
times of viding a single metric of user satisfac-
similar or tion with the performance of enterprise
competitor systems. applications. Sevcik defined 10 differ-
The third category, technological ent methods (see Table 1).
requirements, comes from chosen design The idea is to use several (say, three)
and used technology. Some technologi- of these methods for the same system. If
cal requirements may be known from the all come to approximately the same num-
beginning if some design elements are ber, they give us T. While the approach
used, but others are derived from busi- was developed for production monitor-
ness and usability requirements through- ing, there is definitely a strong correla-
out the design process and depend on tion between T and the response time
www.stpmag.com • 23
 GAUGING PERFORMANCE

goal (having all users satisfied sounds as a
pretty good goal) and between F and the
response time requirement. So the
approach probably can be used for get-
ting response time requirements with
minimal modifications.
While some specific assumptions
like four seconds for
default or the F = 4T
relationship may be ip
for argument, the ap-
proach itself conveys the
important message that
there are many ways to
determine a specific per-
formance requirement,
which, for validation
purposes, is best derived
from several sources.
Depending on your sys-
tem, you can determine
which methods from the
above list (or maybe
some others) are appli-
cable, calculate the met-
rics and determine your
become clear.
These two situations look similar,
but are completely different in nature:
1.) The system is missing a require-
ment, but results are consistent: This is
a business decision, such as a cost vs.
response time tradeoff; and 2.) Results
aren’t consistent (while
requirements can even be
met): This may indicate a
• problem, but its scale
isn’t clear until investigat-
ed.
Unfortunately, this view
Usually is rarely shared by devel-
opment teams too eager to
you have finish the project, move it
into production, and
no idea move on to the next proj-
ect. Most developers
what caused aren’t very excited by the
prospect of debugging
the observed code for small memory
leaks or hunting for a rare
symptoms error that’s difficult to
reproduce. So the devel-
these few failed transactions are a view
page for your largest customer, and you
won’t be able to create an order until
it’s fixed?
In functional testing, as soon as you
find a problem, you usually can figure
out how serious it is. This isn’t the case
for performance testing: Usually you
have no idea what caused the observed
symptoms or how serious it is, and
quite often the original explanations
turn out to be wrong.
Michael Bolton described the situa-
tion concisely 13:
As Richard Feynman said in his appen-
dix to the Rogers Commission Report on the
Challenger space shuttle accident, when
something is not what the design expected,
it’s a warning that something is wrong.
“The equipment is not operating as expected,
and therefore there is a danger that it can
operate with even wider deviations in this
unexpected and not thoroughly understood
way.” When a system is in an unpredicted
state, it’s also in an unpredictable state.

requirements.
or how
Requirements
Verification:
serious it is.
Performance vs. Bugs
Requirement verification
presents another subtle
issue: how to differenti-
ate performance issues
• long-running transactions
from functional bugs
exposed under load.
Often, additional investigation is
required before you can determine
the cause of your observed results.
Small anomalies from expected behav-
ior are often signs of bigger problems,
and you should at least to figure out
why you get them.
When 99 percent of your response
times are three to five seconds (with the
requirement of five seconds) and 1 per-
cent of your response times are five to
eight seconds, it usually isn’t a problem.
But it probably should be investigated if 1
percent fail or have strangely high
response times (for example, more than
30 seconds, with 99% three to five sec-
onds) in an unrestricted, isolated test
environment.
This isn’t due to some kind of arti-
opment team becomes
very creative in finding
“explanations.”
For example, growing
memory and periodic
in Java are often explained
as a garbage collection
issue. That’s false in most
cases. Even in the few
instances when it is true, it
makes sense to tune garbage collection
and prove that the problem is gone.
Teams can also make fatal assump-
tions, such as thinking all is fine when
the requirements stipulate that 99 per-
cent of transactions should be below X
seconds, and less than 1 percent of
transactions fail in testing.
Well, it doesn’t look fine to me. It
may be acceptable in production over
time, considering network and hard-
ware failures, OS crashes, etc. But if the
performance test was run in a controlled
environment and no hardware/OS fail-
ures were observed, it may be a bug. For
example, it could be a functional prob-
lem for some combination of data.
When some transactions fail under
load or have very long response times in
Raising Performance
Consciousness
We need to specify performance
requirements at the beginning of any
project for design and development
(and, of course, reuse them during
performance testing and production
monitoring). While performance
requirements are often not perfect,
forcing stakeholders just to think
about performance increases the
chances of project success.
What exactly should be specified—
goal vs. requirements (or both), aver-
age vs. X percentile vs. Apdex, etc.—
depends on the system and environ-
ment, but all requirements should be
both quantitative and measurable.
Making requirements too complicated
may hurt here. You need to find mean-
ingful goals/requirements, not invent
something just to satisfy a bureaucratic
process.
If you define a performance goal as
a point of reference, you can use it
throughout the whole development
cycle and testing process, tracking
your progress from a performance
engineering viewpoint. Tracing this
metric in production will give you valu-

future system releases. ý

ficial requirement, but is an indication
of an anomaly in system behavior or
test configuration. This situation often
is analyzed from a requirements point
of view, but it shouldn’t be, at least
until the reasons for that behavior
the controlled environment and you
don’t know why, you’ve got one or more
problems.
When you have an unknown prob-
lem, why not trace it down and fix it in
the controlled environment? What if
able feedback that can be used for
REFERENCES
1. Miller, R. B. Response Time in User-system
Conversational Transactions, In Proceedings of the
AFIPS Fall Joint Computer Conference, 33, 1968.

24 • Software Test & Performance JANUARY 2008

 Drink Deep or Taste Not Photo Courtesy of The Library of Congress

The Reliability Pool

By Yogananda Jeppu and Ambalal Patel

f Shakespeare were to write a play about the field of safety-critical development, where we do

I software development and testing activities,

he would perhaps call it “The Tragedy of Errors.” And with
most of our work.

Flight Control Software

his use of the language, he would no doubt receive poor “Prove it before these varlets here, thou honourable man; prove it”
marks for usability. may sound dramatic, but those words from Shakespeare’s
Nevertheless, his “Comedy of Errors” offers several
Yogananda Jeppu and Ambalal Patel are scientists at IFCS
parallels to the Tragedy of Errors that is many of today’s
Aeronautical Development Agency in Bangalore, India.
software development projects. That’s true at least in

JANUARY 2008 www.stpmag.com • 25

 TRAGEDY OF ERRORS
FIG. 1: TYPICAL IRON BIRD SETUP
Flight
Dynamic Engineering
Simulator Test Stand
With
Input
Output
Rack &
Actual Signal
Actuators & Processing
Hydraulic
Systems
“Measure for Measure” are a fact of life
for the tester and the project team.
Proving the safety-critical flight control
software on various platforms is a taxing
task for the whole project team. The
safety-critical nature of the software
necessitates a strict discipline in the soft-
ware development process.
Automated code generators, test
case generators and various low-level
tests using automated tools verify the
software and ensure an error-free
development. This ends the software
verification task, which is often carried
out by an independent team. The
process of validating the software and
the system demands an end-to-end test
against the specifications.
In a flight control application, vali-
dating the quadruplex redundancy
built into the computer and the
FIG. 2: AN EVALUATION TOOL
Input Tolerance
Generator Bound
Generator
26 • Software Test & Performance
Digital Flight
Control
Computer
With Control
Laws &
Airdata
Avionics &
Pilot Station
embedded software requires a special-
ized setup to generate typical flight
scenarios. The complete aircraft hard-
ware is placed in a large room con-
nected to custom hardware that emu-
lates the flight conditions. This setup
is normally called the Iron Bird.
What follows is a narration of our
experiences with automatic testing on
this platform as we offer you a look
into the facts of the process. What
should be the automatic pass/fail cri-
teria or bounds for the results in this
noisy (both electronically and audibly)
environment? “To Pass or to Fail: That
Is the Question.” We’re sure you’ll be
quoting us after we bid adieu!
The test procedures detailed here
can’t be found in any textbook. They
were developed by following our
instincts and the knowledge of our sys-
Control
Law & Output
Airdata
Software
with
Tolerance
Processing
Events
tem. The market is full of automated
tools, but most fall short. Sometimes
the only solution is built by hand.
The Iron Bird
Friar John, go hence; Get me an iron crow,
and bring it straight unto my cell.
– “Romeo and Juliet”
A setup like Iron Bird isn’t unique
to the aerospace industry. We’ve visit-
ed automobile plants that use similar
setups for their traction units. What
these and other such systems all have
in common is that the various embed-
ded controllers test actual scenarios in
real time.
A typical Iron Bird setup consists of
actual aircraft actuators with their
associated electrical and hydraulic
setup (see Figure 1). This equipment
produces audible noise, which is easily
controlled by earplugs.
The actuators are connected to an
Engineering Test Stand (ETS). The
ETS has all the electronics to emulate
a four-channel sensor environment
and is electronically noisy. This
noise—the equivalent of what remains
after being reduced to whatever extent
is possible technologically—is the
more dangerous type to applications.
However, it can’t be completely
removed and must therefore be fac-
tored in and accommodated.
The Flight Dynamic Simulator
(FDS) is where the aircraft is simulated
in software. This unit generates the
sensor signals as seen during a typical
flight. The Avionics and Pilot Station
enables the engineer to verify that the
messages displayed to the pilot appear
correctly. It also enables the pilot to fly
the simulator while the test engineer
creates in-flight failure scenarios by
plucking wires and switching off sys-
tems.
This is the final frontier for the
Digital Flight Control Computer
(DFCC). The two critical software
components, namely Control Laws
and the Airdata system, get thoroughly
validated in the presence of simulated
errors in this platform. Any bugs trav-
elling out of this setup will be caught
flying by the pilot.
The Aircraft Is Unstable!
The embedded controller is a software
component that takes in the aircraft
sensor inputs and generates actuator
commands to stabilize the aircraft and
enable flight. This so-called fly-by-wire
JANUARY 2008

technology is seen in action in all
modern fighter aircraft and the jumbo
jets that ferry us around the world
today.
The Airdata system computes air-
craft speed and altitude, and the con-
troller uses this information to pro-
vide uniform operation over the entire
flight envelope. Now these software
components are thoroughly tested at
various levels. But this has to be
proved in an actual environment with
induced errors. What if the wire gets
cut? Can your software handle it?
These questions have to be answered
satisfactorily. This is more like brow-
beating the software—and it really
gets a good beating at Iron Bird.
Many tests are carried out at this
platform and classified into static and
dynamic tests. We’re restricting our
scope to static testing for the discus-
sion here. Thousands of static tests are
conducted on this platform to validate
the embedded controller. This isn’t
possible without automation. We give
the computer the pass/fail criteria by
generating automatic error bounds
specific to each test case. Any numeri-
cal value of the output vari-
ables falling out of these
numerical limits will cause a
test to fail.
The error bounds are gen-
erated taking into considera-
tion the electronic noise in
the sensor inputs and the
As the witches fly through the long
actuator outputs. Here we
consider all the noise in the
tube, they’re battered around
electronic circuits from the
sensor until the point where
based on specific logic, and come
the software takes over at the
input end.
We also take into account
the noise in electronics from
where the software generates
the digital commands until
the actuator begins to drive.
The noise here is from the
sensor electronics, the signal condi-
tioners, the linear/rotary variable dif-
ferential transformers, analog-to-digi-
tal converters and digital-to-analog
converters. The effects of hardware
characteristics such as offset, gain and
biases come into the picture here.
If we inject a rate signal of 10.0
degrees per second, we’re likely to get,
say, 9.123, 11.843 or any random value
between these two bounds. The noise
is also dependent on the amplitude of
the signal. Higher amplitude yields
JANUARY 2008
FIG. 3: THE THREE WITCHES
more problems. The pass/fail criteria
should take into consideration all
these noise factors and the fact that
the embedded controller is going to
generate some noise of its own due to
processing. It’s essential that the width
•
out in a different size.
• into the Control Law and
of the bounds be optimal to catch
bugs. This requires a tool tailor-made
for the application.
A Comparison Tool
A specialized tool has been developed
for the test activity. We call it the
Evaluation Tool, or EVTOOL for
short. It’s a simple name for complex
software. The EVTOOL block
schematic is shown in Figure 2. An
input generator generates a set of val-
ues for all the sensors based on the
TRAGEDY OF ERRORS
The various processes act on the
three witches as they fly through
the tube. When they come out,
they are of different sizes. The
three witches represent the limits
Upper, Nominal and Lower for the
Input and Output Signals.
defined test case. The sensors include
body rates and accelerations, static
and dynamic pressures, pilot inputs,
etc. These input signals are constants,
since we’re considering static tests. A
set of inputs would be to set pitch rate
to 10 deg/s, yaw rate to 0.0
deg/s and normal acceleration
to 3g, etc. The set of inputs,
selected for a test, is injected
into a tolerance bound genera-
tor.
Based on the particular
hardware characteristics and
the senor noise, bias offset and
gain, three values are generat-
ed representing the upper,
nominal and lower limits of the
sensor output.
The three values of each
sensor variable are injected
Airdata module, which
includes an algorithm provid-
ed by the designers. The
Control Law and Airdata mod-
ule defines the embedded controller
functionality in the form of various
paths interconnected, with signals get-
ting added, subtracted, multiplied or
divided, as the case may be.
Each path has controller elements
such as saturation limits, nonlinear
blocks, gains and switches. Since we’re
testing the system in a static mode, we
don’t take into account the filters and
rate limiters. They’re considered as
unity gains for constant inputs.
Neither will we go into laplace trans-
www.stpmag.com • 27
 TRAGEDY OF ERRORS
forms, etc.
Event triggers such as aircraft
switch inputs are injected separately.
The Control Law and Airdata system
outputs are three values of the output
variables that define the pass/fail
bound. When the test case is executed
on the Iron Bird, it should lie within
the upper and lower bounds of these
output values for the test case to pass.
Any anomaly is automatically declared
as a fail requiring further analysis.
The Three Witches
The Tolerance Bound Generator gener-
ates three values of the input signal.
Let’s say we want to test the controller
with a pitch rate input of 10 deg/s. The
output of the module would be some-
thing like 10.897, 10.0 and 9.456.
Imagine these as the three witches in
“Macbeth” who are flying through the
Airdata System and Control Law mod-
ules (as in a game of Quidditch).
Now imagine the Control Law and
FORMULA 1: OUTPUT BOUNDS
z L1 = y0 +
yL = min {zL1 , zL2 , zU 1, zU 2 }
z L 2 = y0 −
y0 = x0 g0 where
zU 1 = y0 +
yU = max {zL1 , zL2 , zU 1 , zU 2 }
zU 2 = y0 −
Airdata module as a factory that bangs on
the items passing through it or stretches
them with tongs. As the witches fly
through the long tube, they’re battered
around based on specific logic, and come
out in a different size (see Figure 3). It’s
possible that the lower limit would come
out as the upper limit. We’ll discuss this
in detail as we go along.
Do the Math
A drop of water in the breaking gulf
And take unmingled that same drop again
Without addition or diminishing.
—“Comedy of Errors”
In the Control Law, two signals can
add up or subtract, as shown in Figure
4. The output is equal to the sum of sig-
nal A and signal B. The bounds for the
output signals are computed from the
bounds of the signals A and B. For
28 • Software Test & Performance
FIG. 4: ADD OR SUBTRACT SIGNALS
Signal A
+ Output Y = A+B
example, say that signal A has three
witches (components): aL the lower
bound, a0 the nominal value and aU the
upper bound. Similarly, the signal B has
its bounds defined by [bL , b0 , bU ]. The
output Y with its three components is
computed by the following equations:
Y = [yL, y0, yU] =[(aL + bL), (a0 +
b0), (aU + bU)] for addition Y = A+B
Y = [yL, y0, yU] =[(aL - bU), (a0 - b0),
(aU - bL)] for subtraction Y = A-B
2 2
(x0 ⋅ (g0 − g L )) + (g0 ⋅ ( x0 − xL )) 
2 2 
(x0 ⋅ (g0 − g L )) + (g0 ⋅ (x0 − xL )) 
 aL/bU, a0/bL, a0/b0, a0/bU, aU/bL,
2 2
(x0 ⋅ ( gU − g0 )) + (g0 ⋅ (xU − x0 )) 
(x0 ⋅ (gU − g0 )) + (g0 ⋅ (xU − x0 )) 
2 2
Please note here that for the differ-
ence of two signals (A-B), the upper
limit of signal B is subtracted from the
lower limit of signal A to give the lower
limit of the output signal. These worst-
case bounds defined above, however,
may give you a wider estimate of the
bound, thus passing a case that should
have failed. This occurs even more fre-
quently in cases where random noise is
present.
We find that using a Root Mean
Square (RSS) representation helps in
these cases. For the case Y = A+B, where
the variables are defined as above, the
RSS is defined as:
2 2
y L = y0 − (a0 − aL ) + (b0 − bL )
y0 = a0 + b0
2 2
yU = y0 + (aU − a0 ) + (bU − b0 ) 
Signal B
In case the signals are subtracted Y =
(A-B), you can change the sign of b to -b
and interchange the upper and lower
limits of B. A “wide band” or “narrow
band” is the question; you’ll have to
decide which formula to use for your
specific problem.
Two signals can divide or multiply;
for example, Y = A/B or Y = AxB. In
cases like this, a Kronecker tensor
product is computed. Leopold
Kronecker incidentally believed that
“God made the integers, all else is the
work of man.” The product, a human
work, is basically a combination of the
upper, lower and nominal values of
signal A and B, as given below.
Z = A ƒ B = [aLbL, aLb0, aLbU, a0bL,
a0b0, a0bU, aUbL, aUb0, aUbU]
Z = A ƒ (1/B) = [aL/bL, aL/b0,
aU/b0, aU/bU]
The bounds of the output are now
defined as the minimum, nominal and
maximum of the product computed
above.
Y = [yL, y0, yU] = [min (Z), {a0b0 or
a0/b0}, max (Z)]
This is the absolute worst-case toler-
ance bound, which is quite wide.
Signal and Gain
The multiplication of a gain with a sig-
nal can be considered a multiplication
of two signals (see Figure 5). This gives
a wider bound. An RSS approach gives a
better result in such cases. Let the signal
X be defined by the three components,
and the gain G be similarly defined by
the three components. The bounds for
the output are then defined as shown in
Formula 1.

 Linear Interpolation

 Nonlinear blocks are normally specified
in a Control Law as two sets of variables;
JANUARY 2008

say, X and Y. Each of this is a vector of
{x1, x2, x3 ..}, {y1, y2, y3 ..}, points known
as the breakpoints of the nonlinearities.
These nonlinear blocks can form a pos-
itive slope or a negative slope, as shown
in Figure 6.
The output is dependent on the
characteristics of this slope. In case of a
positive slope, you’ll observe that the
lower limit of the output corresponds to
the lower limit of the input. But this
isn’t true for a negative slope, where the
upper limit of the output corresponds
to the lower limit of the input. Notice
how the limits were interchanged, as
indicated earlier.
Notice also that the slopes shown in
the figures are linear. There could be two
or three different slopes in the nonlinear
block (see Figure 7). In such cases, the
components would have different ranges
for the output. A large slope for the
upper limit of the input could increase
the upper limit of the output drastically.
This would widen the bounds or reduce
it, changing the shapes of the witches as
they fly through the tube.
Two-Dimensional Lookups
The “G” gain defined in the section
“Signal and Gain” section above could
be a constant or an output of a two-
dimensional lookup table similar to
Table 1. These feedback gains are used
in aircraft to ensure performance over
the flight envelope. The gains are
“scheduled” based on the speed and
altitude. Refer to “Thou Shalt
TABLE 1: A TABLE OF GAINS
Altitude/Speed 100m
0.0 2.354
0.1 3.235
0.2 4.354
Experiment With Thy Software” in the
June 2007 issue of Software Test &
Performance magazine for an article
on the testing of these gain tables.
FIG. 5: SIGNAL MULTIPLICATION
Signal X
JANUARY 2008
TRAGEDY OF ERRORS
A
SHAKESPEAREAN TRAGEDY
If Shakespeare had written a play about software development and testing, the story might
open like this:
ACT I, SCENE 1: A Day in the Software House
Narrator: Diseased Nature oftentimes breaks forth: In strange eruptions. – “Henry IV”
Certification Agent: All is not well; I doubt some foul play. – “Hamlet”
Project Manager: Find out the cause of this effect, Or rather say, the cause of this defect,
For this effect defective comes by cause. – “Hamlet”
Prove it before these varlets here, thou honourable man; prove it –
“Measure for Measure”
Test Lead: I will prove it legitimate, sir, upon the oaths of judgment and reason. –
“Twelfth Night”
Tester A: O hateful error, melancholy’s child, Why dost thou show to the apt thoughts of
men: The things that are not? – “Julius Caesar”
Tester B: When sorrows come, they come not single spies, But in battalions. – “Hamlet”
Tester C: The nature of bad news infects the teller. – (“Antony and Cleopatra”
Whispers in Coffee Room: What’s this, what’s this? Is it her fault or mine? The tempter,
or the tempted, who sins most, ha? - “Measure for Measure”
Project Manager: Condemn the fault, and not the actor of it? – “Measure for Measure”
Test Lead: But since correction lieth in those hands, Which made the fault that we cannot
correct... – “Richard II”
Test Team: Unarm, Eros; the long day’s task is done, And we must sleep. – “Antony and
Cleopatra”
Just don’t get caught sleeping on the job, or your project might end up like the Ariane 5’s
Flight 501.
The two input signals to the lookup interpolated gain output Y = [yL, y0,
table would be altitude and speed, yU] with tolerance bounds is comput-
each signal having its three compo- ed as described below. We consider
nents. The gain thus computed would the set of all combinations of inputs
also have a bound. In these cases we with their three components as below:
normally compute the gains for all C = {(aL, bL), (aL, b0), (aL, bU), (a0,
combinations of the two signals. bL), (a0, b0), (a0, bU), (aU, bL), (aU,
b0), (aU, bU)}
We then compute the gain output
1000m 5000m 10000m using the lookup table for all these com-
4.363 3.456 3.567 binations of inputs as:
Z = {z1, z2, z3, z4, z5, z6, z7, z8, z9}
5.347 4.575 3.567
The nominal value is z5 correspon-
6.474 5.374 3.879
ding to the input combination (a0, b0).
The upper and lower bounds for the
Consider two signals A and B as computed gains are then given by taking
inputs to the lookup table. A = [aL, a0, the maximum and minimum of Z as:
aU] and B = [bL, b0, bU] are defined Y = [yL, y0, yU] =[min (Z), z5, max
with their tolerance bounds. The (Z)]
Non-Monotonic Nonlinearities
We have to take a little care when spe-
cial nonlinear blocks are present in
the path. These nonlinearities can
Output Y = X*G
G exist as shown in Figure 8.
In the convex nonlinearity (a), we
see that the output bounds are not
around the nominal. In cases like this,
continued on page 37 >
www.stpmag.com • 29
 When Your Company Resorts to Sending
Its Testing Overseas, a Quality Audit Can
Help Ensure a Winning Season

30 • Software Test & Performance JANUARY 2008

 By Steve Rabin and John Bradway
utting costs is the goal of every company. And software testing is a practice that
C has been ripe for the offshore harvest. Given the inherent risks of using offshore
resources for testing and adapting offshore teams
to agile practices, it’s important to ensure that the
best practices of the onshore team are being repli-
cated elsewhere.
Through an analysis of the approach used by
one company—Primavera Systems—to align soft-
ware quality efforts around the globe, you can
learn how to perform a quality audit that is equal-
ly applicable to teams inside the four walls of an
organization.
We considered Primavera a good case study
because it has internal development centers in the
U.S. (Bala Cynwyd, Pa. and San Francisco), Israel
and London, as well as offshore development/QA
centers in India and Eastern Europe.
The idea behind a quality audit is to perform a
systematic examination of the practices a team
uses to build and validate software. This is impor-
tant, as issues of quality are represented across the
software development life cycle. The audit aims
not to uncover software defects, but to understand
how well a team comprehends and executes the
defined quality practices.
The quality audit can be used to assess if a
process is working and if things are being done
the way they’re supposed to be. The audit is also
an excellent way of measuring the effectiveness of
implemented procedures. Management can use
audit results as a tool to identify weaknesses, risks
and areas of improvement.
The audit is as much about people as it is about
the procedures in place. The more team members
understand their roles and how that relates to
quality, the more likely the team will grasp and
adhere to the defined practices. The ultimate
objective, of course, is to deliver high-quality soft-
ware (as defined by the organization).
Quality audits are typically performed at regu-
lar intervals. The initial audit develops a quality
procedure baseline and determines areas that
need remediation. Subsequent audits are moni-
toring-oriented to help teams identify and address
gaps in the quality process. Remediation involves
articulating effective actions that address deficien-
cies in the daily process of conducting quality
practices.
A quality audit can be aided by the use of soft-
ware tools but, as stated above, it’s as much about
people mentoring and teaching as anything else.
Any team, offshore or otherwise, will best respond
Photos by Stefan Klein
to an approach that focuses on helping individu-
als meet expectations and generate improvement.
Steve Rabin is CTO at Insight Venture Partners. John
Bradway is development manager at Primavera Systems.
JANUARY 2008
The audit as described throughout this docu-
ment consists of five phases:
Pre-assessment planning. This phase includes
setting expectations, creating a timeline and
getting executive sponsorship for the project
audit. The deliverable for this phase is agree-
ment with and buy-in of the audit process and
a follow-up commitment for improvement.
Typically this is capped by a meeting with the
audit sponsor and key stakeholders on the
audit process and objectives.
Data gathering. This phase involves developing
interview questions and surveys and gathering all
documentation (bug reports, test cases) prior to
the interview process.
Assessment. The assessment phase involves con-
ducting interviews and developing preliminary
findings. Confidentiality is crucial, and team
members must clearly understand the process. A
meeting explaining the process and the reasons
behind it should be held with the entire team.
Post audit. After reviewing documents and
interview notes, the analyzed information is syn-
thesized into a list of findings and prioritized
remediation steps.
Presentation of findings with sponsor and team.
Findings are presented and agreement is reached
on highest-priority improvement areas.
In Primavera’s case, the quality audit and the
entire software development quality management
life cycle are tied to the ISO 9001:2000 standard.
There are a variety of reasons for this, as
explained below.
Aligning Scrum With ISO 9001
Primavera Systems, which has practiced Scrum
since June 2003, found itself with an interesting
dilemma in the beginning of 2005. Increasingly,
perspective customers were inquiring about the
ISO certification level of Primavera’s development
processes. In fact, quality auditing is an important
element in ISO’s quality system standard. The
most recent ISO standard has shifted the focus of
audits from procedural adherence only to meas-
uring the effectiveness of quality practices to deliv-
ered results.
While this makes sense, implementing and
assessing the usefulness of ISO in an agile envi-
ronment is a challenge. After all, the Agile
Manifesto declares “working software over com-
prehensive documentation” and “people
and interactions over process
and tools.” Many
 OFFSHORE PLAYBOOK
agile thought leaders don’t consider ISO
a priority.
Primavera addressed this issue inter-
nally by focusing the team on common
core objectives and the firm’s mission to
deliver the best possible quality software.
The firm sells software to highly regulat-
ed markets, so the need to support cus-
tomer requirements, including ISO, is an
important consideration. Bottom line is
that the quality audit made sense and
provided the opportunity to better align
the on- and offshore teams.
As a first step, Primavera engaged an
outside consultant to assess the current
software development life cycle and
information technology procedures as
they relate to alignment with ISO
9001:2000 standards. Associated with this
was the delivery of a gap analysis between
the current SDLC and the standards.
The results were a little surprising.
While Primavera wasn’t producing all of
the documentation to meet the letter of
the law as it relates to the ISO standard,
existing processes were highly aligned
with the spirit and intent of the standard.
The consultant felt that by organizing
many of the existing artifacts into a qual-
ity management system and creating a
limited amount of additional documen-
tation, the team could safely declare,
“We’re aligned with the ISO 9001:2000
standard” and provide enough docu-
mentation to back this up.
An important side benefit was the
ability to use the assembled information
as a valuable reference for developers
and a great resource for helping new
employees ramp up. The documents,
created by the consultant, described key
processes used in the software develop-
ment life cycle: software testing, configu-
ration management, defect tracking,
internationalization, product mainte-
nance, requirements management and
32 • Software Test & Performance
release management. In other words, the
documentation described Primavera’s
agile methods across development
domains.
During the same time period,
Primavera was engaged in creating two
offshore development Scrum teams in
Bangalore. This emerging documenta-
tion, which articulated development
processes, proved to be a highly useful
resource when ramping up the offshore
team. Since the offshore team was unfa-
miliar with Scrum methodologies in gen-
eral and Primavera’s practices in particu-
lar, being pointed to relevant sections of
the documentation helped orient them
and made them productive more quick-
ly. It also eased communication, since
everyone was using the same practices,
metrics and terminology.
Preparing for the Audit
The quality policy described in the qual-
ity management system called for annual
audits of all quality processes, on- and
offshore. Regardless of the documenta-
tion, development management consid-
ered it important to
ensure that the quality
practices being used off-
shore duplicated those
being used at home.
Better to address quality
issues before there’s an
• responses in offshore
actual problem.
What exactly
Given the number of
development locations,
did the team
the audit team needed to
develop a repeatable want to measure,
process for the project
audit. For example, what what data was
exactly did the team want
to measure, what data was needed, and how
needed, how would the
data be collected and what would it be
rules would be used to
generate the metrics? collected?
Also, would it be necessary
•
to normalize the data?
The team developed a
checklist to qualitatively
measure how well the off-
shore teams were con-
forming to agile practices.
Audit criteria were selected from various
sections of the quality manual along with
input from members representing multi-
ple domains. This resulted in a 43-point
audit covering requirements manage-
ment, design and implementation, con-
figuration management, testing, defect
tracking and release management.
The team decided it was more impor-
tant to audit the quality practices them-
selves vs. the quality of the required arti-
facts. While both are important, the qual-
ity of the artifacts can be improved
through training and education, but if
the processes aren’t in place, not much
can be done. This approach is less threat-
ening to the teams being audited if the
quality of the work is not the primary
focus.
The audit is meant to be foremost a
fact-finding and learning experience.
With a baseline in place, the ongoing
review process can look at the quality of
the artifacts and determine ways to help
the team make improvements. The team
was most interested in objectively look-
ing for evidence that the quality process
was being followed.
Throughout the process, the team
made sure to consult with management,
initially to acquire and maintain leader-
ship buy-in. Meetings were held with
development managers and executives,
both on- and offshore, to discuss the
audit and its goals and to solicit input.
The team discovered
that the word audit evoked
some uncomfortable
teams. Because of this, it
was important to set the
context for the audit as an
exercise in self-improve-
ment and as a retrospective
tool to help drive positive
change. Positioning the
audit in this light helped
ensure the team’s coopera-
tion and active participa-
tion. Since agile methods
generally provide a high
degree of transparency and
are constantly being
refined, the teams were
comfortable with (and
appreciated) the focus.
With management sup-
port and the offshore
team’s understanding of
the context of the audit,
the audit team began gath-
ering and examining some
of the related artifacts, documents that
had been delivered by the offshore team
over time. The idea was to look at exist-
ing requirements and attempt to trace
them through the quality process. In
total, this involved looking for the code,
test plans, automation and test results—
both automated and manual, related to
JANUARY 2008

the delivered requirement.
There were several reasons for doing
this. First, it was valuable to gain an
understanding of how transparent the
process was, based on the exchange of
materials from teams distributed around
the world. This was also a learning exer-
cise to become familiar with the team’s
work processes and communication style.
The team was able to identify a variety of
specific issues that required detailed
examination during the on-site audit.
The Audit Checklist And
Evaluation Scheme
The audit checklist was developed to
measure how effectively a team has
implemented and follows the quality
guidelines. There are a number of ways
Primavera used the results of the audit,
so the checklist had to be synced to the
objectives. This included:
To baseline the current state of devel-
opment within an organization prior to
introducing Primavera’s development
process.
To monitor the progress of adoption
of Primavera’s documented Quality
Management System and identify areas
where the implemented practices were
satisfactory as well as areas that could be
improved.
The criteria used in the audit are
grouped under the following high-level
headings:
• Requirements management
• Design and implementation
• Configuration management
• Testing
• Defect tracking
• Release management
While the overall goal of conducting
a project audit is to provide a qualitative
view of the suitability of and adherence
to the development process, a quantita-
tive view is also necessary. For example,
the auditor conducting the evaluation
may find it useful to flag certain criteria
or results that he feels the need to
emphasize. Similarly, the team being
audited may have certain concerns that
they want to have the auditor scrutinize.
The Primavera audit team adopted
the following scheme for measuring
each of the 43 audit points:
1.00 If the process fully satisfies the
criterion
0.75 If the testing process largely satisfies
the criterion
0.50 If the testing process partially
satisfies the criterion
0 If the testing process does not
JANUARY 2008
satisfy the criterion
This scheme must be used with some
caution and is provided only as a guide-
line. The reason for this is simply that
auditors interpret things differently,
making it difficult to determine the pre-
cise meaning of any particular score. A
degree of subjectivity occurs during the
audit process that must be taken into
account along with the objective meas-
ures.
The audit criteria are listed below.
This is not the full, detailed spreadsheet
used by auditors on-site, but rather the
higher-level categories. Keep in mind
that this checklist was developed to
address the specific needs of Primavera
and the objectives of the audit. The
terms used below along with the refer-
enced tools are also specific to
Primavera’s use of agile practices.
Requirements management
• Primavera PM used to track require-
ments
• Collaboration with Product Owner
• Requirements estimated using
accepted techniques (Ideal Team
Days)
• Sprint 0 to refine estimates
• Sprint -1 to determine initial
requirement estimates
Design and implementation
• Feature specifications (created as
design documents and updated “as
built” when requirement is complet-
ed)
• Code tested and passes unit tests
prior to check-in
• Formal code reviews requested by
programming manager for appro-
priately complex areas
• Peer or buddy review of code for
code check-ins
• Requesting schema changes
through schema change process
• Technical designs where appropri-
ate
• Designs and coding consider inter-
nationalization
• Online help and printed manuals
updated for new requirements
Configuration management
• Builds automated
• Builds automatically deployed twice
daily
• Automated process replicates all
client server, Web and Group Server
code to all four ClearCase servers
• After a compile of the merged code,
OFFSHORE PLAYBOOK
JUnit and FitNesse tests are run. In
the event of a failure, an e-mail noti-
fication is sent and the merge is
rejected and deferred to the next
day.
• FitNesse tests run automatically with
daily builds
Testing
• Acceptance tests cover require-
ments and are automated
• Test procedures documented in
Mercury Quality Center
• Developer unit tests written
• Automated Silk tests run to validate
code integration
• Test cases and test results can be
traced to requirements
• Internationalization cases consid-
ered during testing
• Performance testing
• Test results records stored
• Test cases peer reviewed
• Active system tests conducted
Defect tracking
• In-process defects “scrummed”
appropriately
• Defect reporting using Mercury
Quality Center
• Defect threshold counts used for
sprint entry/exit criteria
Release management
• Sprint review meetings held
• Release management team reviews
• Sprint retrospectives
• Sprint closeout processes (backlogs
updated)
• Tracking progress with burn-downs
• Attend Scrum Master meetings
• Daily team meetings
• Product Owner sets sprint priorities
• Co-located teams
• Obstacles removed daily
• Sprint planning meetings
• Task granularity (e.g., no more than
16 hrs.)
Performing the Audit
It’s worth noting that, except for the
largest organizations, it’s not necessary
or desirable to employ specific auditors.
It’s best to choose auditors from within
the team and across disciplines. In fact,
there are good arguments for cycling the
auditing team so many resources get the
opportunity to interact with their peers
and view the quality process from a dif-
ferent perspective. Auditor training is
recommended so that the results can be
as normalized as possible.
www.stpmag.com • 33
 OFFSHORE PLAYBOOK
By choosing a broad spectrum of
auditors from all levels of the organiza-
tion, you’ll ensure that everyone has
commitment to the Quality System and
gain a better understanding of the over-
all management process. It’s also a good
way for people from various areas to gain
an understanding of how their depart-
ment fits into the organization.
The audit can be done formally or
informally, with every resource or select-
ed resources. The format is based on
the objectives, and in
Primavera’s case ,the approach
chosen was informal. Four
developers, two business ana-
lysts and the entire QA staff par-
ticipated in the first offshore
audit. The team met with these
individuals over a two-day peri-
od focused on looking at indi- documented, and problems reported
vidual processes in addition to
how requirements traced
through the different artifacts.
The mantra for the audit was
not only “tell me,” but “show
me.” Listening to individuals
describe the process and what
was done is important, but so is
viewing the actual artifacts. Since a qual-
ity process is interconnected across the
development life cycle, time was spent
looking at related, adjacent links to
other areas of the process.
It was important to set and then man-
age expectations when reviewing peo-
ple’s work. Reminding everyone that the
focus was on improvement helped us
achieve a balanced auditor/auditee envi-
ronment.
Making sure that all participants gain
value from the experience is also impor-
tant. Everyone involved learned some-
thing new about the processes in use, the
rationale behind them and how provid-
ing traceability adds value. Being able to
take a feature and trace it back through
test results, automation, unit tests and
requirements makes obtaining a detailed
understanding of a feature much easier
than walking in cold.
Teams shouldn’t expect to achieve
a perfect score, so this is another area
that requires expectation manage-
ment. In Primavera’s case, the teams
fared very well. Most of the shortcom-
ings were not unexpected, since the
on- and offshore teams worked closely
together on a daily basis. Interestingly,
several valuable insights into other
areas that needed improvement were
exposed. This was unexpected, but a
34 • Software Test & Performance
good benefit.
At the conclusion of the audit, a
meeting was held with the entire staff
to review the preliminary results of the
audit prior to presenting the results to
management. In each audit, each line
item that did not receive a 1 was ana-
lyzed, and a series of next steps was dis-
cussed. A final remediation plan was
developed and approved based on the
conclusions drawn from analyzing all
of the results.
• remediation
Audit findings need to be
for further action.
•
Wrapping Up the Audit
Audit findings need to be documented
and problems reported for further
action. A date should be established
for the correction, and the next audit
should ensure that issues were reme-
died. The audit documentation does-
n’t need to be complicated, but should
include the audit plan, the audit notes
and the audit report. The notes are the
items the auditor wrote down during
the audit, and can include specific
findings, responses to questions, key
documents reviewed and comments.
The audit report is the “official” doc-
ument used to report the findings of the
audit. A template for this document
should be prepared by the audit team,
amended as necessary and consistently
used by all auditors. The document
should include details of the audit, date,
auditors’ names and findings. Once
agreed to, the audit report should
include the remediation plan.
The process audit is concerned with
both the validity and overall reliability of
the process. For example, does the
process consistently produce the intend-
ed results? It’s important to identify non-
value-added steps that the team may
have added to the process. Once identi-
fied, this needs to be documented. The
team should demonstrate the ability to
perform defined practices consistently,
and it should be noted when this is not
the case.
Bear in mind that the audit has two
modes of operation: appraisal and analy-
sis. Both of these modes involve a combi-
nation of subjective and objective influ-
ences. Appraisal generally involves
resource issues, while analysis is more
procedure-oriented.
Identifying issues across both of these
spectrums is an important audit skill and
needs to be taken into account
and documented. After all, the
plan involves
changes to both how people act
and what the procedure accom-
plishes.
The results of the audit
along with associated docu-
mentation were added to
Primavera’s quality manage-
ment repository. A meeting was
held to discuss the results and
remediation plan with the
same team that met to kick off
the audit process; this included
stakeholders and management.
As in the on-site review, each
area that didn’t score a 1 was dis-
cussed.
Lessons Learned
Setting the stage for the audit as an in-
depth retrospective aligned with the
agile goal of continuous improvement
helped Primavera secure individual buy-
in for the audit.
Since it seemed that people weren’t
prepared for the “show me” mentality
employed during the interview process,
better communication and expectation
settings with the team makes sense.
Discussing the style to be used during the
audit also helps make those involved
more comfortable.
The scoring system is a work in
process and will be improved over time.
It remains subjective, which is accept-
able, but the addition of a weighting sys-
tem is under consideration.
The use of weighting for areas that
are fundamental to the development
process will provide better results.
Since each area of the quality manage-
ment process isn’t equal in impor-
tance, weighting will expose those
areas of concern more visibly.
The bottom line is that the entire
development organization now realizes
the goal of continuous improvement. ý
the value this kind of audit can bring to
JANUARY 2008
 Best Prac t ices
Unit Testing Tools Unlikely
To Transform Life or Industry
Count me among those
skeptical that unit testing
will ever go mainstream.
Granted, I don’t write
code. However, I do churn
out a goodly amount of
tech-related prose. And
like most science and tech-
nology journalists I know, I
more or less loathe the
process of revising and
Geoff Koch
editing what I write.
Why is this relevant? Because the
wrestling that journalists do with para-
graphs, sentences and individual words
is the rough analog to unit testing in
software development.
It’s been said that all journalists are
frustrated novelists. While I don’t have an
unfinished manuscript in my desk drawer,
I am familiar with traditional lore about
great writers. Namely, most of them spend
the lion’s share of their time redoing or
outright ripping up their drafts and start-
ing anew. Here’s former U.S. Poet
Laureate Billy Collins on the subject, from
his poem “Royal Aristocrat.”
“I was a single monkey / trying to
type out the opening lines of my own
Hamlet / but often doing nothing more
than ironing pieces of paper in the plat-
en / then wrinkling them into balls to
flick into the wicker basket.”
But folks, journalism ain’t poetry or
any other type of literature, though I did
have a professor at Stanford who argued
as much. (No doubt he had that unfin-
ished manuscript in his desk drawer.)
Journalism is much more akin to the
process of writing code than to anything
in the so-called higher literary arts. A
journalistic article is quick, approximate
and often destined to have a shelf life—
particularly when the subject is technolo-
gy—measured in weeks or even days
rather than years. Generally, the ideal of
continuous backspacing, tweaking and
polishing is forever yielding to the inter-
JANUARY 2008
rupt-driven work and home
lives that most of us lead.
And, software traditionalists
be damned, I suspect the
same can be said of most
code that’s written in today’s
fast-changing, forever-itera-
tive world of development.
At the risk of being too
solipsistic, dear reader, you
should know that most
months I find myself scram-
bling to file this column with some sem-
blance of respect for the deadline.
(Though for this particular screed, I
failed miserably.) Then, a day or two after
dashing this off to the Long Island–based
editors, I invariably think of several ways
in which I could have more adroitly made
my point. I’m guessing the feeling is quite
close to those experienced by the vast
majority of developers when the code
base they’ve labored to contribute to
finally goes from beta into production.
What Would Alberto Do?
While reporting this column, I heard
from many people with unit testing
experience, and almost all of them
made the same two points. The first is
that unit testing undeniably makes
sense. It’s cheaper and more efficient,
the pros say, to have developers do their
own testing and to automate the process
of running a steadily growing library of
these tests. The second is that the ranks
of developers and teams that actually
practice unit testing are modest indeed,
if not vanishingly small.
In his Jan. 24, 2007, blog entry
“Testing Genes, Test Infection, and the
Future of Developer Testing,” Alberto
Savoia declares that developers who
resist unit testing far outnumber those
who practice the craft, zealously or oth-
erwise. “This spells trouble because I
have found that when developers
become managers, they bring with them
their attitudes about testing,” says
Savoia, founder and chief technology
officer of Agitar Software in Mountain
View, Calif.
Savoia is worth listening to on the sub-
ject of testing and software generally. He
has contributed to products that have
won a bevy of awards, including JavaOne’s
Duke Award and Software Development
magazine’s Productivity Award. And his
résumé includes a list of senior technical
positions at Sun and Google to go along
with that cherished Silicon Valley merit
badge: starting and then selling a compa-
ny for a minor fortune. Savoia’s load test-
ing company Velogic, founded in 1998,
was sold to Keynote just two years later for
$50 million.
Beyond his experience, Savoia brings
one of the most candid and engaging
personalities in the software industry to
an interview. It’s clear after just a few
minutes that the guy is having fun and
has a well-developed sense of humor
about the often stilted and obtuse world
of testing and development. One exam-
ple of this is the chosen moniker,
Crap4j, of the open source project
Savoia spawned with Bob Evans. The
tool helps developers find code that’s
crap to maintain, either because it’s too
complex, doesn’t have enough associat-
ed tests or both.
I can’t help but think as Savoia
enthusiastically describes his career that
in many ways, he’s a case study as to why
unit testing will never reach its tipping
point. We don’t talk specific numbers,
but clearly that résumé of his implies an
enviable degree of financial security. So
why start yet another company in the
ultra-niche testing market? Because, he
explains, he loves this stuff.
In the hyper-logical, left-brained fash-
Seen any good poetry related to programming?
Best Practices columnist Geoff Koch wants to
know: gkoch at stanfordalumni dot org.
www.stpmag.com • 35
 Best
Practices
ion that typifies most successful techies,
Savoia says we’re all constantly faced by
a more or less binary set of choices
about our lives that can be described by
the equation X + Y = total well-being,
financial and otherwise. X, usually a
responsible position at an established
company, is associated with guaranteed
income. Y is associated with choice
imbued with those elusive qualities of
emotional engagement, satisfaction of
intellectual curiosity and overall happi-
ness. Once financial security is achieved,
he continues, it always makes sense to
pursue Y when confronted with an
either-or choice about what to do next.
The logic, which is unassailable, is
not what’s interesting here. Rather, it’s
that for Savoia, happiness is more about
mucking about testing lines of code
than, say, fishing in Montana, going to
cooking school in the south of France,
or in my case, buying courtside seats
and losing myself in Big 10 basketball
from November to March. Savoia’s
choice is sort of akin to a journalist hit-
36 • Software Test & Performance
ting the jackpot with a best-selling book
and then opting to stay at the rewrite
desk for years thereafter—for fun. I’ve
interviewed enough developers to sus-
pect that even among the I-dream-in-
code crowd, Savoia is an outlier.
Limited Commercial Appeal
More concrete reason for skepticism
comes from several of the other inter-
viewees for this column, including one
at industry giant Cisco Systems.
“Right now Cisco is in a latency curve
with unit testing,” says Andy Chessin, a
technical leader at the San Jose, Calif.-
based company. “Unit testing has a
huge barrier to entry. If the group real-
ly isn’t passionate about it, doesn’t
understand it or doesn’t have the time
to get started with it, they probably
won’t be driven to take the plunge.”
“Right now, few people really under-
stand what [API-level] unit testing
involves or how to get started with it,” he
says. “But I think they will soon catch on.”
Maybe, though I wouldn’t bet on it.
Index to Advertisers
Advertiser
Automated QA
Checkpoint Technologies
FutureTest 2008
Hewlett-Packard
iTKO
Seapine
Software Test & Performance
Software Test & Performance
Conference Spring 2008
Software Test & Performance
White Papers
“Simplicity is essential to a core unit
testing framework; for this reason, the
most successful unit test frameworks
tend to stay small, in code size and in
team size,” says David Saff, an MIT doc-
toral student who is now one of the lead
maintainers of JUnit. “In comparison to
other successful open source projects,
this promotes a proliferation of third-
party extensions, while limiting contri-
butions to the core framework. I think
this somewhat limits the chance for a
company to arise that would do the
equivalent for, say, JUnit, that Covalent
has done for Apache.”
In other words, unit testing is des-
tined to remain by and for a modest
group of undoubtedly smart specialists.
So while the worlds of writing and pro-
gramming will always have a small
priesthood obsessed with quality and
constant revision, most of the rest of us
will continue to muddle along as best
we can—at least until we can hang up
tors for other pursuits. ý
our reporter’s notebooks and text edi-
URL Page
www.testcomplete.com/stp 11
www.checkpointech.com/BuildIT 6
www.futuretest.net 2-3
www.hp.com/go/securitysoftware 40
www.itko.com/lisa 8
www.seapine.com/ttstech 4
www.stpmag.com 36
www.stpcon.com 39
www.stpmag.com 17
JANUARY 2008

< continued from page 29
we must take care to see that the apex
point is taken for computing the
bounds. It’s possible that the convex
nonlinearity may be a curve instead of
the inverted V shape shown here. In
such cases, we compute the apex point
as the maximum value that is possible
in the region of interest. In case of
convex nonlinearity, the apex point
defines the upper bound. But in case
of the concave nonlinearity (b), the
apex point defines the lower limit.
We have had numerous problems
with such blocks, with cases failing
when they should have passed. So take
care of these blocks!
Tragedy of an Error
The disastrous failure and tragic self-
destruction of the European Ariane 5
expendable launch system was the
result of a software bug. During Flight
501, its maiden voyage, the data con-
version of a 64-bit floating point num-
ber was too large to be represented by
a 16-bit signed integer value. This out-
of-range value caused a hardware
exception, resulting in total system fail-
ure. In India, we too have had a space
vehicle fail due to out-of-range value.
Because of these experiences, we
now test our systems with 10 percent
higher values than the maximum possi-
ble value of any signal. In such cases, if
the nominal value considered is the
maximum value, the upper bound is set
to 10 percent higher than the maxi-
mum value. The lower bound is as given
by the sensors. In case of the minimum
possible value selected as the nominal,
10 percent lower value is taken as the
lower bound. For example, if the maxi-
mum possible pitch rate signal is 60.0
deg/s, this is selected as the nominal.
FIG. 8: NON-MONOTONICS
a. Convex Nonlinearity
JANUARY 2008
TRAGEDY OF ERRORS
FIG. 6: POSITIVES AND NEGATIVES
Y0 Y0
yU yU
y0 y0
yL
yL
xL , x0 , xU X0 xL , x0 , xU X0
Positive slope Negative slope
Let’s say the upper and lower bounds For example, let’s say that Trigger 1
are 62.4 and 57.5, respectively, as given is set to True based on a speed signal of
by the software module. The upper less than 70.0 km/h. When True, this
bound is taken instead as 66.0, and the trigger further sets the output value of a
lower bound is retained at 57.5 deg/s. second system to 13.0 (a constant), and
when False, it computes a value based
FIG. 7: NON-LINEAR SLOPES on an algorithm.
Let the signal value X = [xL, x0, xU] =
[69.5, 71.3, 73]. Then Trigger 1 will take
Y0 the values [1, 0, 0]. This will cause the sec-
Slope=S2 ond system to have the values [13.0, 4.0,
yU 5.35]. The lower bound is 13.0 as com-
y0 Slope= S1 pared to, say, 3.23 if the trigger was set
yL based only on the nominal value. The
X0 upper bound is 5.35. This would cause a
X=[xL , x0 , xU ] great deal of confusion indeed!
Piecewise Linear (S1 < S2) Avoid Tragedy
Software validation is always associated
with system validation, the most impor-
tant issue being whether the software
Special Events has been designed to cope with system
For events triggered by a particular sig- failures. Proof must be provided for
nal, the trigger is set based only on the this in any safety-critical application,
nominal value. This is important and test cases must be designed to
because when a trigger further triggers cover these aspects. Iron Bird is the
the output of a second system, we can final testbed, and maintaining the soft-
have problems if we don’t set the ware at this level is difficult.
events based on the nominal value. Always experiment with your sys-
tem. The techniques detailed here
have evolved after several experi-
ments in the initial phase of the
project. We modeled the system char-
acteristics to develop the EVTOOL
software. This has paid rich dividends
and continues to do so as new ver-
sions keep coming.
All the effort we put in results in
something fruitful. The challenge of
finding errors in the system gives the
tester a pleasant feeling that is
described beautifully by Shakespeare
in “Much Ado About Nothing”:
a. Concave Nonlinearity The pleasant’st angling is to see the fish
Cut with her golden oars the silver stream,
And greedily devour the treacherous bait. ý
www.stpmag.com • 37
 Future
Future Test
Test
Get Development
In Sync With QA
Development is writing
code for the next release of
the application. QA has a
regression test suite that
tests the previous release,
but they’re waiting for the
end of the current devel-
opment iteration before
they start to update the test
suite to cover the new and
modified functionality. As
Adam Kolawa
a result, the code base is
evolving significantly during the devel-
opment phase, but the regression test
suite is not… so by the time that QA
receives the code from development,
the code and the regression test suite
are totally out of sync.
Once QA has the new version in
hand, they try to run the old regres-
sion test suite against the new version
of the application. It runs, but an over-
whelming number of test case failures
are reported because the code has
changed so much.
At that point, QA often thinks,
“Instead of trying to modify these test
cases or test scripts for the new version,
we might as well go ahead and test it by
hand because it’s the same amount of
work, and even if I update it now, I’ll still
have to update it all over again for the
next version.” So they end up testing by
hand, and typically come to the conclu-
sion that automation is overrated.
That’s how automation goes to hell
in QA.
Keep in Sync
As a result of the divide between QA
and development, the regression test
suite is treated as an afterthought. To
keep the test suite in sync with the
code, the team needs to treat the
regression test suite like it’s a key part
of the application—the part of the
application that verifies whether the
38 • Software Test & Performance
implemented functionality
actually works. This has a
couple of implications.
One is that team leaders
must allocate sufficient
budget and time for regres-
sion test suite development
and maintenance. I’ve
found that the best results
are achieved when there’s
roughly a 50/50 distribu-
tion of effort between writ-
ing code that represents the functional-
ity of the application and writing code
that verifies that functionality.
The other implication is that the
team needs to modify their workflow so
that QA works in parallel with develop-
ment: updating the regression test suite
as the developers update the code base.
To achieve this, QA must become more
tightly integrated with development. To
start, development has to build test
cases as they write code. This means that
the team needs to define and enforce a
policy that every time development
implements a feature or use case, they
add a test case to check that it’s func-
tioning correctly.
The role of QA, then, is to review
these test cases as soon as they’re writ-
ten, as part of the code review proce-
dure. Their goal here is to verify
whether the test case actually repre-
sents the use case that is implemented
in the code. If QA and development
work together in this manner, the test
suite is constantly updated in sync with
the application.
To keep this up, you need to ensure
that the test suite is constantly tested
against the application. This means it
must become part of the nightly build
and test process. Every night, after the
application is built, the regression test
suite executes. If test failures are report-
ed, then the test suite might be growing
out of sync with the application.
Whenever that happens, the developers
need to spend just a few minutes review-
ing any test failures reported for their
code, and then either updating the test
cases (if the test failed because the func-
tionality changed intentionally) or fix-
ing the code (if the test failed because a
modification introduced a defect).
In my humble opinion, this is the
only way that QA can be automated.
Looking Back, Looking Ahead
Having been in this industry for 20 years
now, I’ve witnessed many changes.
Languages have come and gone, the
level of programming abstraction has
elevated, and development processes
have grown increasingly compressed
and iterative. Yet, from assembly code to
SOA, one thing remains the same: the
need for an effective, reliable way to
determine if code changes negatively
impact application functionality.
One way of helping organizations
overcome this challenge is to provide
technologies that enable development
teams to quickly build a regression test
suite that includes unit tests, static
analysis tests, load tests and anything
else that can be used to identify
changes. Our goal here is to help teams
identify and evaluate functionality
changes with as little effort as possible
so that keeping the test suite in sync
with the application is not an over-
whelming chore.
The other part is to optimize the
development “production line” to sup-
port these efforts. This involves imple-
menting infrastructure components
(including source control systems,
nightly build systems, bug tracking sys-
tems, requirements management sys-
tems, reporting systems and regression
testing systems) to suit the organiza-
tion’s existing workflow, linking these
components together to establish a fully
automated building/testing/reporting
process, and mentoring the organiza-
tion on how to leverage this infrastruc-
ture for process improvement. The
software defects. ý
result is greater productivity and fewer
Dr. Adam Kolawa is founder and CEO of
Parasoft.
JANUARY 2008
 April 15-17, 2008
San Mateo Marriott
San Mateo, CA

SPRING

SUPERB SPEAKERS!
Michael Bolton • Jeff Feldstein TERRIFIC TOPICS!
Michael Hackett • Jeff Johnson • Agile Testing • Test Automation

Bj Rollison • Rob Sabourin • UI Evaluation • Java Testing

Mary Sweeney • Robert Walsh • Security Testing • Testing Techniques
AND DOZENS MORE! • Improving Web Application Performance

• Optimizing the Software Quality Process

Register by Jan. 25 To Get • Developing Quality Metrics

The eXtreme Early-Bird Rate! • Testing SOA Applications

www.stpcon.com • Charting Performance Results

• Managing Test Teams

A L T E R N A T I V E T H I N K I N G A B O U T A P P L I C A T I O N S E C U R I T Y:

Hone Your Threat Detection (To A Telepathic Level).

Alternative thinking is attacking your own Web applications, finding
vulnerabilities and destroying them with precision and vengeance—
throughout the life of the application.

It’s looking at application security through the eyes of a hacker

to identify threats to your system and risks to your business.

It’s harnessing the power of SPI Dynamics, recently acquired by HP,

to redefine and expand your security abilities. (Please note: positive
effects on your bottom line.)

It’s assessing security the right way, from development to QA

to operations—without slowing down the business.
(Cue elated cheers.)

Technology for better business outcomes. hp.com/go/securitysoftware

©2008 Hewlett-Packard Development Company, L.P.