Вы находитесь на странице: 1из 40

A Publication

PR va T Ente
BE CTIC ting rise
Fo
Ja the
A es rp
ST ES
r

:
VOLUME 5 • ISSUE 2 • FEBRUARY 2008 • $8.95 • www.stp

Multi-User Testing Can


Be Challenging and Fun

Don’t Drown in Chaos,


Just Pull the Plug
When, Oh When
Will You Profit
From SOA?

A Battle of Open-Source Defect Trackers


April 15-17, 2008
San Mateo Marriott
San Mateo, CA

SPRING

A BZ Media Event

Platinum Sponsors Gold Sponsors Silver Sponsor

www.stpcon.com
SUPERB SPEAKERS!
Michael Bolton • Jeff Feldstein
Michael Hackett • Jeff Johnson
Bj Rollison • Rob Sabourin
Mary Sweeney • Robert Walsh
AND DOZENS MORE!

STPCon is THE best place to


learn the newest techniques
for improving software quality,
but don’t take our word for it—
just listen to your colleagues:

“You’ll find informa- “It solidifies the total


tion outside of your testing experience and
daily activities, and opens your eyes to alter-
options/alternatives to native approaches and
think about new methods that you simply
TERRIFIC TOPICS!
approaches to testing.” cannot get from books.” • Agile Testing • Testing Techniques
—Alex Kang —John Croft • UI Evaluation • Java Testing
Staff Engineer, Tellabs QA Manager, I4Commerce • Security Testing • Test Automation
• Improving Web Application Performance
“Great, informative conference for software testers, • Optimizing the Software Quality Process
leads and managers alike. Useful tutorials and • Developing Quality Metrics
technical classes of wide variety—A must-attend for • Testing SOA Applications
all serious QA/SQE professionals!”
—Alan Abar • Charting Performance Results
Software Quality Engineering Manager, Covad Communications • Managing Test Teams

Register by Feb. 22 To Get The Super Early-Bird


Rate and SAVE OVER $400!
4AKE THE
HANDCUFFS OFF
QUALITY ASSURANCE
Empirix gives you the freedom to test your way.
Tired of being held captive by proprietary scripting? Empirix offers a suite of
testing solutions that allow you to take your QA initiatives wherever you like.
Download our white paper, “Lowering Switching Costs for Load Testing
Software,” and let Empirix set you free.

www.empirix.com/freedom
VOLUME 5 • ISSUE 2 • FEBRUARY 2008

Contents A Publication

14
C OV E R ST ORY
Battle of the Defect Trackers:
Bugzilla Takes On Trac
Two open source defect trackers face off. Learn what makes both of these
tools bug-eating behemoths. By Elfriede Dustin

20 Multi-User
Testing Rocks!
Multi-user testing is crucial for every
application that uses a relational
database. Don’t worry, the process can
be fun and exciting.
By Karen N. Johnson

29
Seeking
Depar t ments
7 • Editorial
SOA Profit Worried about shifty politicians? Maybe
defect-tracking tools can keep them honest.
With SOA, the quest
for quality is the key 8 • Contributors
to a healthy ROI. Get to know this month’s experts and the
By Frank Grossman best practices they preach.

25
9 • Feedback
Making Sense Of It’s your chance to tell us where to go.
Chaos Reports
11 • Out of the Box
Since 1994, the Standish Group has

33
New products for testers.
published its annual Chaos Report,
documenting the IT industry’s in-
36 • Best Practices
efficiencies. Before you panic, read
our own report on where those Manual Test Java Web apps have changed everything about

statistics come from. Naïveté software testing. By Geoff Koch

By Robin Goldsmith In this case study of manual testing 38 • Future Test


gone awry, the trials and tribulations of Early-stage automated source code analysis
a fictional novice tester exemplify what is the next big thing. B y Gwyn Fisher
not to do. By Prakash Sodhani

FEBRUARY 2008 www.stpmag.com • 5


Ed N otes

Defect Tracker
VOLUME 5 • ISSUE 2 • FEBRUARY 2008
EDITORIAL
Editor Editorial Director
Edward J. Correia Alan Zeichick

For Politicians
+1-631-421-4158 x100 +1-650-359-4763
ecorreia@bzmedia.com alan@bzmedia.com

Copy Editor Contributing Editor


Laurie O’Connell Geoff Koch
loconnell@bzmedia.com koch.geoff@gmail.com

ART & PRODUCTION By the time you read this, defects, I wouldn’t vote to
Art Director Art /Production Assistant Super-Duper-Kalamazoop- deploy that product into
LuAnn T. Palazzo Erin Broadhurst
lpalazzo@bzmedia.com ebroadhurst@bzmedia.com
er Tuesday will likely be in the Oval Office.
the history books. I’m As in software, political
SALES & MARKETING referring of course to what defects can be minor or
Publisher we in the U.S. call “Super severe. Minor defects
Ted Bahr Tuesday,” the day in might include crying on
+1-631-421-4158 x101
ted@bzmedia.com
February on which people TV, having an extramarital
of the major political par- affair or experimentation
Associate Publisher List Services
David Karp Lisa Fiske ties from about 20 states with drugs early in life.
+1-631-421-4158 x102 +1-631-479-2977 vote in the presidential pri- While these things might
dkarp@bzmedia.com lfiske@bzmedia.com mary. Primary votes prior Edward J. Correia offer commentary on one’s
to Super Tuesday take place one state at character, they alone would not stop a
Advertising Traffic Reprints
Phyllis Oakes Lisa Abelson a time. It’s all part of the American project from being released.
+1-631-421-4158 x115 +1-516-379-7097 process of deciding who gets to run for But severe bugs might, and would
poakes@bzmedia.com labelson@bzmedia.com president, the nation’s highest office. get top priority when discussing and
Director of Marketing Accounting
As someone who follows national questioning the candidate. Such things
Marilyn Daly Viena Ludewig politics to a flaw (just ask my daughter), might include being caught in a lie
+1-631-421-4158 x118 +1-631-421-4158 x110 I sometimes have a tough time keeping (particularly while under oath), any
mdaly@bzmedia.com vludewig@bzmedia.com
track of candidates’ positions (which type of fiscal malfeasance or too many
are often numerous). Where does each flip-flops on important issues.
READER SERVICE
one stand on the economy, national Just as desired software features
Director of Circulation Customer Service/
Agnes Vanek Subscriptions security and other important issues of change over time, candidates too have
+1-631-443-4158 +1-847-763-9692 the day? And how does their current been known to change their positions
avanek@bzmedia.com stpmag@halldata.com position differ from things they’ve said on the issues, particularly as shifts in
and done in the past? political climate affect public opinion.
Cover Photograph by LuAnn T. Palazzo
Animatronic Godzilla Appears Courtesy of the
As I edited this month’s cover fea- In the days and months after the attacks
Daniel J. and Naomi Pagano Collection, NY ture, it occurred to me that the same of 9/11, most Democrats and
tools we use for tracking software Republicans were in agreement that the
defects could also be applied to tracking U.S. should invade Afghanistan and
politicians. Enter FlakTrak, a new Iraq. Now most disagree.
defect-tracking system I just invented to Similar flip-flops can be seen on
help me keep abreast of where our lead- abortion, POW detention, illegal immi-
President BZ Media LLC
Ted Bahr 7 High Street, Suite 407
ers stand. gration, taxes, global climate change
Executive Vice President Huntington, NY 11743 Here’s how it works. The first time a and “corporate greed.” When a candi-
Alan Zeichick +1-631-421-4158 politician takes a position on an issue, date switches position, either a bug is
fax +1-631-421-4130
www.bzmedia.com it’s entered into FlakTrak and assigned resolved or a feature fails a regression
info@bzmedia.com to that politician. If it’s a position I agree test, and a new defect is logged.
with, it’s resolved immediately and
Software Test & Performance (ISSN- #1548-3460) is
published monthly by BZ Media LLC, 7 High Street,
becomes a feature. Promises to reduce Historically Significant
Suite 407, Huntington, NY, 11743. Periodicals postage taxes, build a border fence and win the For the first time in 80 years, the field of
paid at Huntington, NY and additional offices.
war in Iraq fall into this category. candidates does not include an incum-
Software Test & Performance is a registered trade-
mark of BZ Media LLC. All contents copyrighted If a candidate takes a position I dis- bent president or vice president.
2008 BZ Media LLC. All rights reserved. The price agree with, a bug is created and Interestingly, there’s one candidate who
of a one year subscription is US $49.95, $69.95 in
Canada, $99.95 elsewhere. assigned to that politician. Raising in 1928 was only eight years from being
POSTMASTER: Send changes of address to Software taxes, granting rights to illegal aliens or born. The first few issues he would
Test & Performance, PO Box 2169, Skokie, IL 60076.
Software Test & Performance Subscribers Services calling for surrender, for example. find in his queue would be called
may be reached at stpmag@halldata.com or by
calling 1-847-763-9692.
These all would be classified as defects. McCain-Feingold, McCain-Kennedy and
If a candidate has too many unresolved McCain-Lieberman. !

FEBRUARY 2008 www.stpmag.com • 7


Contributors

ELFRIEDE DUSTIN is a testing consultant currently employed


by Innovative Defense Technologies (IDT), a Virginia-based
software testing consulting company specializing in auto-
mated testing.
In her role as consultant, Elfriede has evaluated scores of
software testing tools. In this month’s cover story, she tack-
les Bugzilla and Trac, two of the industry’s leading issue-track-
ing systems. If you’re about to select a bug tracker or think-
ing about switching, you won’t want to miss her analysis,
which begins on page 14.

This month we’re pleased to welcome KAREN N. JOHNSON


to our pages. Karen is an independent software testing con-
sultant working in the Chicago area, is a frequent confer-
ence speaker and has authored numerous articles and papers.
She is active in numerous testing associations.
In the first of what we hope will be many contributions
to this magazine, Karen explores how and why multi-user
testing typically requires a shorter cycle than other types of
testing. Learn how to do multi-user testing right, begin-
ning on page 20.

ROBIN GOLDSMITH has been president of Go Pro


Management, a software testing and training consultancy,
since 1982. He specializes in business engineering, require-
ments analysis, software acquisition, project management
and quality assurance.
Beginning on page 25, Robin explores the real cause of
chaos, and why most IT projects are late, over budget and
wrong. Learn to avoid these problems by developing objec-
tive factual project measures in an environment where every-
one involved takes responsibility for their results.

FRANK GROSSMAN is co-founder and president of


Mindreef, which offers testing solutions for Web services
and SOA-based applications. Frank, along with Jim Moskun,
developed SoftICE, BoundsChecker and DevPartner, pop-
ular tools that would ultimately be acquired by Compuware.
Drawing from more than 20 years of technical expert-
ise in software tool development, Frank delivers a
thoughtful examination of what companies need to do
to ensure a positive return on their investment in SOA.
Turn to page 29.

As a quality control specialist with a global IT services


organization, PRAKASH SODHANI is involved in a mul-
titude of testing and quality assurance activities. He has
served as a quality professional in numerous capacities
with several top IT organizations.
Prakash holds a master’s degree in computer science
and many testing certificates. Beginning on page 33,
Prakash converts some of his experiences into a
hypothetical case study on the dos and don’ts of software
testing.

TO CONTACT AN AUTHOR, please send e-mail to feedback@bzmedia.com.

8 • Software Test & Performance FEBRUARY 2008


Feedback

CHANGE: THE ONLY CONSTANT


“Traceability: Still a Fact of Life in 2008” KNOWLEDGE FIRST,
(Test & QA Report, Jan. 8, 2008) is still an
interesting read. Having started in an indus-
THEN TOOLS
try where regulatory compliance was Regarding Geoff Koch’s question about what res-
required, it has been interesting to watch onates with his readers (“A Code-Free Look at Change
how things are going back toward compli- and Management,” Best Practices, Software Test and
ance. The culture of an organization needs Performance, Dec. 2007), I agree that tools are sec-
to change and adapt to compliance. It is ondary to knowing what to do.
easy to see why some not-so-good software People seem to forget this when it comes to CM
is produced, as requirements and changes
and process-related issues. For example, I was work-
are not tracked or traced, let alone tested.
ing with a dev manager who wanted me to “enable
Compliance does not mean adding cost
the teams to test more effectively,” but he didn't want me to spend too much time
but being smarter about the processes that
working on current issues or even training people one-to-one, and a new framework
are used. If a process is not reviewed and
wasn’t going to solve the problem. People needed new habits (and some coaching).
modified to incorporate the changes
Ironically, this same manager was a big advocate of use cases and architecture
required for compliance, then the bottom-
line cost for software increases, as stated in (which are tool independent). Testing, CM and deployment are often treated as sec-
the article. Change is the only constant in ondary issues.Testing, CM, deployment and architecture are closely related; your test-
life, and we all need to adapt and change. ing, configuration and architecture can greatly affect how easy it is to deploy and
Gretchen Henrich upgrade. Maybe that's obvious, but...
Another of my common rants is that a team should have a deployment process from
LATE-TERM BUG FIX about day one, so that they can tune the configuration and deployment process. I remem-
Edward J. Correia’s article “Defect Tracker, ber working at places where “build the installer” was scheduled for a few days before
Heal Thyself” (T&QA Report, Dec. 15, ship date.
2007) states the same basic thing found in CM Crossroads at www.cmcrossroads.com and the CM Journal are all about issues
many articles: Look at the defects and eval- related to what Mr. Koch is writing about.
uate them early in the testing process to Thanks for the article.
determine what can be changed in require- Steve Berczuk
ments or further developed requirements
to reduce future defects. Geoff Koch responds:
But what if you are past that stage? How Hi, Steve,
do you evaluate the situation when there I really appreciate you taking the time to correspond. The substance of your com-
are over 2,000 critical defects and devel- ment does validate a conclusion I'm reaching after writing this column for more
opment team is in system test? How do you than two years—namely, there’s a healthy collection of good tools and processes
look at these defects, determine which ones for doing test, CM and deployment. Uber-geeks will argue over the finer points of
are the most important to correct with lim- these tools and processes, but in the bigger picture, these arguments are almost
ited resources, and which ones can be put always irrelevant. That’s because the real stumbling block to sound test & QA is
aside so the application can continue for- almost always human behavior, from the individual developer who finds it too much
ward toward going live? How do you sal- of a hassle to really embrace test-driven development to organizations that peren-
vage the situation to achieve the most nially shortchange test and QA when it comes to resources, schedules, etc. And it’s
impact on making the application work not as if there’s a shortage of information about the cost to developer organizations
and the least impact on data and the users of fixing issues downstream or even the macro cost to society of buggy, unreliable
when the application goes live? software.
Jim Watson
I know that contemporary economic research is making much hay out of finding
examples of how in fact people most often don’t behave rationally; testing in gener-
AUTOMATION FOR FLASH
al seems to be a case in point of this phenomenon.
Regarding “What’s the Best Way to Test
Best regards, and thanks again for reading,
Flash Apps?” (T&QA Report, March 27,
Geoff
2007), my biggest pain point is regression
testing for Flash applications. Manual
testing is time intensive. An automated take a look at this month’s Out of the Box FEEDBACK: Letters should include the writer’s
solution would be helpful. section (page 11) for coverage of how a tool name, city, state, company affiliation, e-mail
Scott Schimanski called Eggplant is being used to automate address and daytime phone number. Send your
Flash testing. thoughts to feedback@bzmedia.com. Letters
become the property of BZ Media and may be
From the editor: According to its creator, Redstone Software,
edited for space and style.
Thanks for writing, Scott. I suggest that you this was an unexpected development.

FEBRUARY 2008 www.stpmag.com • 9


Out of t he Box

Scapa Scraps IT Repeats An SDK for Every


Tired of all those user
Issue Tracker
Perforce in December released an SDK
directory moves, adds
for its Defect Tracking Gateway, a set of
and changes whenev-
components for integrating the compa-
er someone’s hired or
ny’s Perforce 2007.3 Fast SCM with third-
fired? Scrap them with
party issue-tracking systems. First released
Scapa—Scapa ITSA,
in February 2007, Defect Tracking
that is. That’s short for
Gateway integrated only with Hewlett
IT Service Automa-
Packard’s Quality Center 9.0. The SDK
tion, an automation
enables the creation of integration with
tool introduced in De-
any issue tracker.
cember that the com-
Defect Tracking Gateway includes a
pany says can auto-
graphical editor for creating and man-
mate many of the IT
aging field maps between Perforce 2007.3
operations tasks cur-
and a defect management system. A repli-
rently being per-
cation engine moves data between the
formed manually by
two systems and keeps everything in sync.
people.
Synchronization can be either one-way
Using an Eclipse- Scapa ITSA presents a GUI-based environment for automating repet-
or bidirectional. Changelists, created and
based GUI, IT opera- itive tasks now done by people.
used by the Perforce system, describe all
tions staff can create
changes made to files, and also can be
process flows that handle service requests ITSA also includes record and playback
replicated.
coming in through e-mail or Web-based capability. “If you can do something
The SDK and Defect Tracking
forms or other applications. Tasks appear through the GUI, Scapa ITSA can capture
Gateway are included with Perforce
as graphical icons that can be set to per- and automate it,” according to documents
Server 2007.3 for Windows XP, available
form operations selected from a drop-down on the company’s Web site. The solution
now; pricing starts at US$800 per seat.
list. Flows can be run periodically or set to also can be controlled remotely using Citrix,
trigger automatically. VMware/VDI or RDP.
It’s Springtime For
Integration
mValent Integrity Gets Open source and Java solutions provider

Pushy With Provisioning SpringSource in December unveiled


Spring Integration, an extension of the
Spring Framework that the company
Configuration management tools maker tributed systems and detect and manage claims can simplify the creation of mes-
mValent last month released Integrity 5, configuration changes across an enter- sage-driven integration systems.
the latest version of its flagship solution prise. SpringSource, which changed its name
that it claims now enables companies to With the addition of automated deploy- from Interface21 in November, created
push configuration settings to large, dis- ment tools, Integrity 5 now can provision and maintains the Spring application
single or multiple hosts framework.
in unlimited numbers According to the company, Spring
and locations and across Integration simplifies development by
firewalls. Provisioning “handling the message-listening and serv-
can consist of single ice-invoking aspects, [and] applies inver-
components or pack- sion of control principles” to the runtime.
ages. The new version The tool also handles common input and
can also analyze local or output sources through included
remote configurations adapters, and adds to Spring’s core func-
to pinpoint differences tionality with support for JMS, remoting,
and inconsistencies and e-mail and scheduling. Spring Integration
help prevent drift. also takes care of task execution, life cycle
Reporting is simplified management, dynamic languages, aspect-
mValent Integrity 5 can now send configurations in whole or in part
through new automat- oriented programming, event-oriented
to far-flung systems.
ed change dashboards. publishing, and subscription and trans-

FEBRUARY 2008 www.stpmag.com • 11


action management, the company said.
The extensible framework, which is
director Christopher Young, Eggplant was
never intended to test Flash. “We designed
Faster, Better
set to reach version 1.0 by June, allows for it to test or automate anything, but espe- Automation Is The
the creation of custom input and output cially the user experience,” he says.
adapters, content-based routers, content “Anyone who has an issue testing their
Claim From VMware
“enrichers,” and message filters and trans- Flash applications or tests them manual- Virtualization technology maker VMware
lators. ly now” has an easier way. Eggplant works in December released Infrastructure 3,
across multiple browsers and operating which updates its ESX Server to version
systems, including even Symbian and 3.5 and VirtualCenter to version 2.5, and
Klocwork Offers Windows CE, the company says. delivers “new capabilities for increased
Insight Into Desktop levels of automation, improved overall
infrastructure availability and higher per-
Code Analysis StackSafe Virtualizes formance for mission-critical workloads,”
According to Klocwork, maker of auto- The Data Center according to a company document.
“This new release delivers the nonstop
mated source code analysis tools, a tool A new solution from StackSafe uses vir-
introduced in January can prevent bug- virtual infrastructure… and better man-
tualization technology to give IT opera- agement,” says Raghu Raghuram,
gy code from finding its way into an orga- tions teams an easy way to simulate mul-
nization’s main code tree. VMware’s vice president of products and
ti-tiered systems and applications for test- solutions. Performance in ESX Server 3.5
Insight, which began shipping in late
January, is a desktop analysis tool with col- is enhanced through support of paravir-
laboration features that ties in with the tualized Linux and large memory pages,
team’s larger system and identifies defects read company documents, bringing “sig-
in the developer’s local build. Defect nificant performance gains,” particular-
reports can be broken down by compo- ly to Java applications and Oracle data-
nent, team or geography. base workloads.
Although desktop-based source code Performance for Citrix and Windows
analysis tools have been available for years, Terminal Services workloads is improved
Klocwork CEO Mike Laginski claims their thanks to support for TCP segmentation
accuracy was limited since they lacked the offload and jumbo frames, which the com-
context of the overall system. “Conversely, pany claims reduces the CPU overhead
a system that runs only at the system lev- associated with network I/O processing.
el is viewed as an audit tool by develop- Live migration of virtual machine disks
ers and doesn’t give them the ability to between storage systems is now possible
find and fix problems before they check with “little or no disruption or downtime,”
StackSafe’s Test Center, through its browser- using VMware’s Storage VMotion mod-
in their code.” Insight provides both, based interface, permits change testing on virtu-
Laginski claims. ule. The tool also supports dynamic bal-
al systems prior to deployment to production.
Along with Insight, Klocwork intro- ancing of workloads and can help head
duces a declarative language that can be off performance bottlenecks. An Update
ing, performance tuning, upgrades and Manager automates the deployment of
used to extend or customize it and other other activities that might pose risks to
Klocwork products. Insight is available patches and updates for ESX Server hosts
production systems, according to com- and virtual machines.
now in a C/C++/Java version as well as pany claims.
one just for Java. A new Guided Consolidation module
Dubbed Test Center, the new tool assists in the initial setup of VMware, auto-
began shipping late last month and is matically discovering physical servers. It
claimed to enable test teams to “import
Solution for Your [virtual copies of] production systems to
identifies candidates for consolidation,
converts them into virtual machines and
Flash Testing Might conduct staging, testing, analysis and “intelligently places them into the opti-
reporting.” Imported copies are net-
Include Eggplant worked into a working infrastructure
mal VMware server host.”
Also included in the release is an
Hungry for tools to automate your Flash stack that simulates the production con- experimental feature that balances work-
application testing? Perhaps you should figuration, enabling production-safe load to help reduce power consumption
try Eggplant. That’s the word from changes, regression testing, patch test- in the data center, automatically power-
Redstone Software, which reports an ing, security and risk assessment, diag- ing servers up and down based on
increase in usage of the multi-platform UI nostics and root-cause analysis, emer- demand.
automation tool for testing apps written gency change testing, application assem-
in Adobe’s popular environment. bly and validation and compliance report- Send product announcements to
stpnews@bzmedia.com
According to Redstone managing ing, the company says.

12 • Software Test & Performance FEBRUARY 2008


Today! An Epic By Elfriede Dustin

G iven any online testing user group on any


given day, you’ll see this query—“Which is
Battle Between the best tool for xyz?”—where xyz equals any testing catego-
ry, such as automated software testing, performance testing,
defect tracking, etc. No matter which testing category this

Two Top Open


specific query is related to, the answer will generally be “It
depends.”
Finding the best tool for your organization almost always
depends on your specific needs and requirements. In this arti-

Source Defect-
cle, I’ll explore how to evaluate and choose a tool using a
defect-tracking tool as an example, enumerate reasons why
open source tools can be a viable solution, and describe an
example comparison of two of the “top” open source defect

Tracking Tools—
tracking tools: Bugzilla and Trac.
Recently, I was tasked with evaluating defect-tracking tools
for a client. We generally approach any type of tool evaluation
strategically, as follows:

Don’t Miss It!


1. Identify the tool requirements and criteria (in this case,
defect-tracking tool requirements/criteria)
2. Identify a list of tools that meet the criteria
3. Assign a weight to each tool criteria based on importance
or priority
4. Evaluate each tool candidate and assign a score
5. Multiply the weight by each tool candidate score to get
the tool’s overall score for comparison
As with many tool categories, in the case of defect-track-
ing tools, there are a good many choices. Of course, it does-
n’t make sense to implement these steps for a
hundred tools. Instead, it’s a good idea to narrow the broad
list to a select few. This can be done
using such criteria as:
• Requirements. Does the tool meet
the high-level requirements? For
example, if you’re looking for
Web-based tools and several work
only as client/server, those would
not be considered.
• Longevity. Is the tool brand new or
has it been around a while?
Darwin’s principle of “survival of
the fittest” applies to the tool
market. Decide on a number of
years of existence and eliminate
the rest.
• User base. A large user base generally indicates good utili-
Photograph by LuAnn T. Palazzo

ty. With open source, a busy development community also


means more people providing feedback and improve-
ments.
• Experience. Add points if you or your client have had a

Elfriede Dustin works for Innovative Defense Technologies (IDT), which


specializes in automated testing.

www.stpmag.com • 15
BUGZILLA VS. TRAC

FIG. 1: BUGZILLA DEFAULT WORKFLOW (V.3.0) are holding sway with more and more
companies. Advantages of open source
include:
• No licensing fees, maintenance or
New bug from a user with can confirm or
a product without UNCONFIRMED state. restrictions
Unconfirmed • Free and efficient support (though
Bug confirmed or
receives enough votes
varied)
Bug is reopened,
• Portable across platforms
Developer takes was never confirmed • Modifiable and adaptable to suit
possession your needs
New • Comparatively lightweight
Development is • Not tied to a single vendor
Ownership Developer
is changed takes finished with bug Licensing. Trac is distributed open
possession source under the modified BSD
Possible resolutions:
FIXED License (trac.edgewall.org/wiki/Trac
DUPLICATE
Assigned License), a “commercially friendly”
WONTFIX
WORKSFORME
license sometimes known as a “copy-
INVALID Development is center” license (take it to the copy cen-
REMIND finished with bug ter and copy it). It permits changes to
LATER
be made to source code and kept or
Developer takes
distributed commercially with few
possession Resolved restrictions.
Issue is
resolved Bug is closed Bugzilla is covered by the Mozilla
QA verifies
Public License (www.mozilla.org/MPL),
QA is not satisfied
with solution solution worked which is sometimes described as a
BSD/GPL hybrid. This so-called “weak
Bug is reopened
Reopened Verified copyleft” license requires copies and
changes to source code to remain
under the MPL, but permits such
Bug is reopened
Bug is closed changes to be bundled with propri-
etary components.
Closed Support. During my Bugzilla and
Trac evaluations, my questions were
Source: wikipedia.org
often answered within minutes, solv-
ing any issues I ran into from the sim-
good experience with a specific tool ucts, lack of control over improve- ple to the complex.
that meets high-level requirements, ments, and licensing costs and restric- In my experience working with
longevity and user base criteria tions. commercial tool vendors, support is
described. And while a few of those downsides usually more complicated. It can take
Using these criteria, I narrowed the might be applied to some open source days before a support specialist
field from the scores of defect-tracking projects, the advantages of leveraging the addresses an issue, and there is some-
tools to four: two commercial and two open source community and its efforts times an extra cost or annual mainte-

B
open source. This article compares
and contrasts the latter two, Bugzilla
UGZILLA: THE 2007 TESTERS CHOICE
and Trac. Coincidentally, Bugzilla is
widely used by the open source com- Bugzilla was at the top of the 2007 Testers Choice Awards, as noted in the December 2007
munity (in Mozilla, Apache and edition of this magazine.“Bugzilla, competing in a category with commercial products devel-
Eclipse) and was selected “Testers oped by companies with infinitely more money and resources than the open source commu-
Choice” by the readers of this maga- nity from whence it comes,” writes Edward J. Correia of the product. “Originally written in
zine, announced in the December Tcl by Terry Weissman, Bugzilla began its life in 1998 as a replacement for the defect track-
issue (see “Bugzilla: The 2007 Testers er used by Netscape for the Communicator suite (it surely must have been quite loaded).
Choice” sidebar for more).
“Thinking another language might get more traction with the community, Weissman decid-
COTS vs. Open Source ed to port it to Perl, resulting in Bugzilla 2.0. As the Wikipedia story goes, Weissman in April
Commercial off-the-shelf solutions cer- 2000 handed the project off to Tara Hernandez, who succeeded in gaining more participa-
tainly have their advantages, including tion from the development community. She handed it off to its current custodian, Dave
published feature road maps, institu- Miller, and the rest is history. Bugzilla won our top spot in the free test/performance tools
tionalized support and stability category.”
(whether real or perceived). But buy-
ing from a software vendor also has its Download back issues of ST&P, including the December 2007 issue, at www.stpmag.com
downsides, such as vendor lock-in, lack /backissues2007.htm
of interoperability with other prod-

16 • Software Test & Performance FEBRUARY 2008


BUGZILLA VS. TRAC

nance contract to be kept current. according to Wikipedia. It’s employed And with the release of Bugzilla 3.0,
Adaptability. Trac is written in Python for many projects in the public and pri- Trac and Bugzilla both now allow for
(www.python.org). First released in 1991 vate sectors, and is widely used to pro- modification and customized fields.
by Guido van Rossum, this dynamic gram Web applications of all stripes. Community. Trac is hosted at edge-
object-oriented programming language Perl is a high-level programming lan- wall.org and maintained by a communi-
can be used for all types of soft- ty of developers who collabo-
ware development. It offers TABLE 1: DEFECT-TRACKING TOOLS COMPARED rate on projects based on
strong support for integration Python. Edgewall.org is per-
Defect-Tracking Tool Evaluation Criteria Bugzilla 3.0 Trac 0.10
with other languages and tools, haps best known for Trac.
_
comes with extensive standard Installation on Windows + Bugzilla is hosted at
libraries, and can be learned rel- Ease of use + + bugzilla.org and is among
atively quickly by the average Extra features + + the many projects adminis-
developer. tered and maintained by the
Customizable workflow + +
Versions of Python are _
Mozilla Foundation, which is
+
available for Windows, Linux/ Security probably best known for its
Unix, Mac OS X, OS/2 and Firefox browser.
Amiga, as well as for Palm and Nokia guage with an eclectic heritage. Perl’s
(Symbian) mobile-phone operating sys- process, file and text manipulation facil- By Definition
tems. Python has also been ported to ities make it particularly well suited for As defined by their respective Web sites,
run on Java and .NET virtual machines, tasks involving quick prototyping, sys- Bugzilla is “server software designed to
and is used at organizations as diverse as tem utilities, software tools, system man- help you manage software develop-
NASA, Rackspace, Industrial Light and agement tasks, database access, graphi- ment.” Trac is “an enhanced wiki and
Magic, AstraZeneca, Honeywell, cal programming, networking and Web issue-tracking system used for software
Symantec and many others. programming. These strengths make it development projects.”
Bugzilla (www.bugzilla.org), in its especially popular with system adminis- Trac isn’t an original idea. It owes
current iteration, is written in Perl. Perl trators and CGI script authors, but much to the many project management
(www.perl.org) is a stable cross-platform mathematicians, geneticists, journalists and issue-tracking systems that came
programming language first released by and even managers also use Perl, before. In particular, it borrows from
Larry Wall in 1987 that borrows from C, according to a history posted at CVSTrac. The project started as a reim-
shell scripting, AWK, sed and Lisp, trac.edgewall.org. plementation of CVSTrac in Python and
an entertaining exercise, as well as toy-
FIG. 2: TRAC DEFAULT WORKFLOW (V.0.11) ing with the SQLite embeddable data-
base. Over time, the scope of the
endeavor broadened, a goal formed,
* leave new and it was brought on its current
course.
Both Trac and Bugzilla offer brows-
reassign er-based defect tracking with multi-user
access, attachments, e-mail integration,
import/export capability, custom fields,
assigned reassign accept authentication and authorization,
reporting and audit trails.
Once I had identified the two tools I
accept
resolve reassign wished to zero in on, I started compar-
ing features beyond their basic capabili-
ties. Some of their major pluses and
resolve accepted accept minuses are shown in Table 1.
Installation on Windows. One of my
client requirements was that the tool
reassign installs on Windows servers. While most
resolve open source defect-tracking tools are
cross-platform compatible, Bugzilla
accept (which also runs on Linux) focused less
closed
on the Windows environment until its
later releases.
resolve Windows installation is a Trac
reopen strength. It offers bundled installation
and simple setup, installing on
Windows without a hitch. Trac also can
reopened run as a stand-alone Web server;
Source: trac.fuseboxframework.org Bugzilla can’t. There is currently no
Bugzilla bundle installation package for

FEBRUARY 2008 www.stpmag.com • 17


BUGZILLA VS. TRAC

FIG. 3: ADD A BUG sive customization. Trac’s ticket work-


flow customization is implemented
through plugins; Bugzilla’s is done by
editing templates written in HTML, CSS
and JavaScript. Their default workflows
are illustrated in Figures 1 and 2.
Personally, I preferred Trac’s option
to Bugzilla’s, but if your team is skilled
in Web-based languages, your choice
might be different. Both tools received
a plus in this category.
Security is job #1. According to Bug-
zilla’s maintainers: “The current develop-
er community is very much concerned
with the security of your site and your
Bugzilla data. As such, we make every
attempt to seal up any security holes as
soon as possible after they are found.”
As such, a list of the security advi-
sories issued with each release that
Windows; components must be collect- integration with project and test case included security-related fixes is provid-
ed and installed separately. managers, IRC, Web apps and data ed on the Bugzilla homepage. “This is
Ease of use. Once installed, both harvesting. almost every version we’ve ever released
products are easy to use through Web Customizable workflow. The goal of since 2.10,” read a statement, indicative
interfaces that are straight forward and any defect management process is to of the recent attention being paid to
intuitive. Of the two, Trac is more light- solve as many defects as possible as security matters.
weight and requires little overhead, quickly as possible. Independent of the When I asked the Trac develop-
while Bugzilla includes more compo- defect-tracking tool, a process should be ment team about its attention to secu-
nents and options. followed to ensure consistency and rity, I got this response: “I cannot give
Both tools received pluses in this accuracy in the way defects are logged a complete answer, but what I know is
category. and tracked. A sound workflow is neces- that we actively look at code we have
Extra features. In addition to their sary to ensure that a defect follows a from multiple angles to see if there’s a
main role in tracking defects, both Trac consistent path and doesn’t get lost at potential abuse.” I am concerned
and Bugzilla offer numerous additional any point. Therefore, the ability to cus- about such a lax position toward secu-
capabilities. tomize the workflow is an important rity. In our case, lax security was a deal
Trac also includes a wiki, allowing consideration when evaluating a defect breaker. And because of the attention
users to add, remove or edit Web con- tracking tool. to security paid by Bugzilla developers
tent. This tool can be used for project Both Trac and Bugzilla permit exten- of late, the project’s longevity also
management or documentation, played a major part; they’ve
to provide project status to man- FIG. 4: TRAC A TICKET had more time to fix security
agement and testers at log-in flaws.
and be leveraged by other depart- After all, Darwin’s theory of
ments for their own uses. The survival of the fittest also plays a
environment itself is implement- role in the open source arena.
ed as a wiki, making every Trac And the more active the com-
page editable, if the organization munity and the longer an open
so chooses. Trac also is compati- source tool has been around,
ble out of the box with the the bigger the user space and
Subversion as well as other SCM the better the chances that
systems via numerous plugins. security issues have been
Bugzilla also integrates with addressed.
Subversion, but requires help While Trac is lightweight
from one of several plugins for and its plugins and available
the job. Plugins and its extensi- components integrate seam-
ble architecture are among lessly, Bugzilla offers more add-
Bugzilla’s main strengths. Scores ons and utilities. If you’re look-
of plugins have been developed, ing for a large number of inte-
including several desktop and grations, Bugzilla should get
handheld clients, integration the nod. On the other hand, if
with Eclipse, Mylyn and about a you’re seeking a lightweight
dozen SCM systems. Other free system that’s quick and easy to
plugins offer dashboards and install, Trac is the one. !

18 • Software Test & Performance FEBRUARY 2008


5th Annual Gathering
of the Eclipse
Community
Attend Eclipsecon 2008 Keynotes from:
EclipseCon is the premier technical and
user conference focusing on the power of
the Eclipse platform. From implementers
to users, and everyone in between, if
you are using, building, or considering
Eclipse, EclipseCon is the conference
you need to attend.
Sam Cory Dan
Ramji Doctorow Lyons
Over 150 Sessions
and Tutorials Including:
This is your opportunity to get in-depth
- Business
technical information from the Eclipse
- C/C++ Development
experts, learn the latest tips and techniques
- Database Development
for using the tools, network with fellow
- Industry Vertical
enthusiasts and experience the breadth and
- Java Development
depth of the Eclipse community. Attending
- Mobile and Embedded
EclipseCon will expand your knowledge and
- Modeling
make Eclipse work better for you.
- OSGi
- Project Mashups
- Reporting
- Rich Client Platform
March 17th - 20th
- SOA Development
- Test and Performance Santa Clara, California
- Technology and Scripting
- Tools
Register at:
- Web Development
www.eclipsecon.org
By Karen N. Johnson

M ulti-user testing can be fun. That’s true because multi-user apps


are straightforward to test. Bugs in this category appear
either dramatically with splashy data- tional database products have matured,
base errors, or quietly as the applica- and errors in this category may be less
tion and database handle the test con- likely to occur than they did several
ditions gracefully and the test cycle years ago. But database software such as
ends without incident. MySQL has come to market, and new
In either case, multi-user testing releases of databases require testing, just
typically involves relatively short test as new software releases require testing.
cycles because the number of objects Clearly, multi-user testing remains nec-
that need to be tested in multiple user essary.
scenarios has, in my experience, not As many applications have moved to
been large. Also, the errors tend to be the Web, focus has shifted to perform-
less debatable than, say, errors uncov- ance testing, for good reason. We’ve
ered during functional testing. For been focused on dozens, hundreds
these, opinions can vary about what and even thousands of users, not just
the application should do or what the two. The perceived likelihood that two
requirements truly meant. Conversely, users would be accessing and updating
there are no arguments that a dead- the same object at the same time is low,
lock error is unacceptable. low enough to drop multi-user testing
off the list of testing to accomplish.
Overlooked, but Essential But errors from this test cycle reveal
Multi-user testing involves testing an that the impact of errors remains high;
application while simulating two differ- we don’t need to think about dozens of
ent users executing the same transac- users, we just need two users to create
tion at the same time for the purpose of the dreaded deadlock.
discovering application, data and data- Multi-user testing is often mistaken
base errors. Multi- for inexpensive performance testing.
user testing is a Since performance testing and multi-
form of testing fre- user testing both (sometimes) focus on
quently not talked high-frequency, high-volume objects,
about and often the confusion about multi-user testing
overlooked. One persists. But looking at the same
reason this cycle gets
forgotten is that over Karen N. Johnson is a software testing con-
sultant in the Chicago area.
the past decade, rela-
Photograph by Anna Sirotina

20 • Software Test & Performance


Identifying Race
Conditions And
Deadlocks In
Your Apps Can
objects doesn’t mean the testing focus
is the same; multi-user testing is
likely hitting the same transactions at
almost exactly the same time all day Leave You
focused on the concurrency of transac- long.
tions and how concurrency and lock-
ing are handled. Identifying Tests
You can choose from a couple of ways
Smelling Like
Staggered Timing to plan what objects to test. First, look

A Rose
Tests in the multi-user cycle involve at the database in your production
adding, updating or deleting an object environment when you make this
at the same time or with staggered tim-
ing. Let’s break this description down
with an some example. Imagine a Web
application that allows users to man-
age internal documentation for a com-
pany: a library management system.
This system allows internal users to
access company documents and,
depending on their permissions,
enables them to add, update or delete
documents. The application includes
functionality to allow administrative
users to add users and user groups.
This gives us multiple transactions to
work with.
Now imagine in the course of a
workday, two administrative users
attempt to create a new user group at
the same time. One user adds the new
user group with no error and contin-
ues on. The second user encounters a
database error referencing something
about a unique constraint. Unlikely to
happen? Perhaps. But it’s not unlikely
that two users would be adding or edit-
ing documents; in fact, at a large com-
pany with a heavily used library man-
agement system, dozens of users are

www.stpmag.com • 21
MULTI-USER TESTING

assessment—which means you might there’s no additional work in setting testing. If your testing has been more
need a database administrator who has up objects; I cycle through add, then black-box focused or if you haven’t
access to production. Unless develop- edit, and my final test even cleans up included database considerations in
ment and test environments contain a my test data as I delete as the last test. your testing previously, some of these
recent copy of production data, you errors might be new to you. Let’s
won’t get the same assessment as pro- Pair Up or Go Solo? examine each error type one at a time.
duction. Even with a production copy You can test with two people pairing A deadlock occurs when two process-
in test, you can’t be sure the DBA set- up to cycle through various add, es are locked and neither transaction
ting up your dev or test environment update and delete transactions. completes. Deadlocks in production
didn’t trim any tables to save space. Alternately, I execute this type of test- can wreak havoc if two users lock a
A practical way to plan table. If you compare a
testing is to use your knowl- deadlock to traveling down
edge of the application.
What objects are users like-
ly to be “touching” all day
long with high frequency?
• a highway that uses a tunnel
that allows only one car at a
time, you can envision the
lock. As two cars compete to
What are the fastest-grow-
ing tables in the database?
If you compare a pass through the entrance
first, neither allowing the
What objects do those other to pass, the lock is set.
tables contain?
deadlock to traveling down Add a few more cars com-
When planning your ing along, like transactions
testing program, remember a highway that uses a tunnel that continuing on a Web site,
that you don’t need to test and you can envision a
every object. Instead, allows only one car at queue growing with frus-
you’re looking for high fre- trated users (or drivers).
quency and high volume;
high frequency because
a time, you can Deadlocks are ugly.
There are several lock-
these objects are being ing schemas available to
used the most and are envision the lock. prevent deadlocks, and


therefore more likely to more than one database
encounter errors. High vol- vendor on the relational
ume is a likely target database market so there
because these are the are different locking
fastest-growing objects, schemas and concurrency
which also likely makes controls. In fact, there are
them high frequency. Timestamps and ing alone, preferring to arrange two several fascinating problems outlined
version numbers can serve as refer- fairly equal class PCs as I manage two as stories you can find on Wikipedia,
ence points to determine frequency. keyboards and execute transactions. If beginning with the entry on dead-
In the case of volume, you’re looking you choose to go it alone, don’t forget locks. Some of the stories are well
for high table counts. to log in to each PC as a different user. known, the most popular and the start
What is high? Compared to other After all, the purpose is to simulate of the collection is the dining philoso-
objects in the database, these are the two users at the same time—not the phers’ problem (see Edsger W.
objects being added and updated same user on two different worksta- Dijkstra’s work). One type of problem
more often. If you’re conducting per- tions (which, by the way, is another and its related teaching story is
formance testing, you might already form of testing.) For equal-class PCs, referred to as the producer-consumer
be acutely aware of what objects gen- the same timing is easier to accom- problem, which also brings up the
erate the most traffic. Use Table 1 to plish with PCs of equivalent processing point of race conditions.
plan multi-user testing. speeds. Race conditions are a core considera-
Once you identify the objects, think tion in deadlocks. Like the tunnel
about what action is being used the What to Watch For analogy, many traffic issues wouldn’t
most often. Are the objects being Deadlocks. Unique index constraints. take place without a race condition.
added, updated or deleted? A simple Lost edits. Application errors. If multi- Rush hour is a race condition. The
point I’ve learned in executing this user testing didn’t sound exciting at same takes place on the database as
testing is that once I’ve added an first blush, consider these errors in the timing of transactions becomes an
object, I test edit and then delete. This production and you might be willing essential factor.
makes the testing move quickly since to allocate a test cycle to multi-user This is one reason I test both same-

22 • Software Test & Performance FEBRUARY 2008


MULTI-USER TESTING

time and staggered timings. Staggered able to add the record, and the second access the same record at the same
timing can catch errors when a user should be notified of an existing time, with admin users accessing a
process or lock hasn’t been released entry of the same value. If the timing user record, and each user updating
but a user can’t view the lock from the is sequential, the user who attempts to the user record. For the first user to
application front end. access the record, the edits
Testing add, update and TABLE 1: MULTI-USER TEST PLANNING FORM will be saved, but the second
delete transactions with user might not obtain the
Same Timing Staggered Timing
slightly staggered timings necessary lock on the record
can catch these errors. If the Add for their edits to be saved. In
lock hasn’t been released, the worst case, the user’s
Change
the next transaction will edits are lost and the user
encounter an error. Delete isn’t informed. Essentially,
In my experience in a the transaction is lost.
decade of multi-user testing, This is why, in practice,
I’m more likely to encounter dead- add the same record receives an error when I test multi-user editing, I make
locks with the same precise timing on stating a record of the same value a point to know what edit each user
the creation of an object. This is why already exists. In some cases, such as makes, and the edits made are not the
I’d rather operate two keyboards than with MySQL, unless the database has same. In the case of the user record, I
perform pair testing; I can get to the been defined as a transactional data- might edit the last name field, adding
exact same precise moment by my own base, all inserts for the table may be a 1 to the end of the existing name as
two hands better than any other way. admin user 1, and a 2 to the end of the


Plus, I have the patience to execute existing name as admin user 2. In
tests multiple times until I can find the short, knowing exactly which edit is
timestamps that make me convinced being made by each user helps to veri-
I’ve covered the test. fy that both edits made it into the data-
The second most frequent error I base.
encounter is deleting the same object
with slightly staggered timing.
Knowing exactly Too Much Information?
In terms of practical knowledge Another test idea to keep in mind
and more immediately tangible ideas which edit is while executing multi-user tests is
for testing, you might look to know security. Here’s a test you can pick up
more information about the specific at the same time as multi-user testing.
database you’re working with. Are you being made by Review the database errors dis-
working with Oracle, Sybase, SQL played to find the behind-the-scenes
Server, Informix, MySQL or another information of an application. Look
database? Each has different imple- each user helps to for database table names, admin
mentations available, so it’s worth- account information or directory path
while to talk with your DBA about the
concurrency controls that have been
verify that both information being given away on error
messages that share too much infor-
implemented. mation. If your application can trap
If you can’t get the information you edits made it into for database errors, a design decision
need, test to find a deadlock and then needs to be made about how much
you’ll likely get the support and infor- information should be revealed
mation needed—a harsh but effective the database. through error messages.


approach. As most of the database ven- “When two trains approach each
dor products have matured, I haven’t other at a crossing, both shall come to
uncovered as many issues as I did years a full stop and neither shall start up
ago, but multi-user testing still is a test again until the other has gone.”
cycle likely to harvest bugs, and since This Wikipedia entry relating to
the impact can be significant, multi- deadlocks, which quotes a statute
user testing remains a risk area worthy halted. These issues are sometimes passed by the Kansas state legislature
of investigation. referred to as primary key or unique key early in the 20th century, is an excel-
Unique index constraints are database errors. lent way to visualize the importance of
errors that occur when two users A challenge with lost edits is multi-user testing. And now you have
attempt to add the same information whether or not the user is informed. a few techniques to help you imple-
at the same time. One user should be Consider this example: Two users ment it. !

FEBRUARY 2008 www.stpmag.com • 23


The Future of Software Testing...

Save $200 With Our


Early Bird Discount
Register Online by February 8

February 26–27, 2008


New York Hilton
New York City, NY
please note colors:
FUTURE is c 100/m 0/y 0/k 10
2008 is c 98
TEST is k 100

A BZ Media Event Gold Sponsors


Drowning in Chaos?

Pull Yourself Out!


Make Sense
Of the Standish By Robin Goldsmith

Reports And M ost IT projects are late, over budget and deliver some-
thing other than what was expected. Such outcomes are the
consequence of arbitrarily mandated Group’s periodic CHAOS reports
budgets and schedules, inadequately (www.standishgroup.com) about IT
defined business requirements and too- project success rates evoke a certain

Take Control Of little/too-late reactive attention to qual-


ity and testing. Learn to avoid these
ghoulish fascination. Despite, or per-
haps partly because most readers don’t
problems by developing objective factu- recognize the reports’ questionable
al project measures and an environment measures and analysis—including

Your Projects in which everyone involved takes


responsibility for their results.
overlooking what undoubtedly is the
most common real proximate cause of
project failures—there’s wide accept-
Photographs by Kristian Peetz

Looking Into a Train Wreck ance of the reports’ basic findings that
Like a train wreck, the Standish IT projects seldom are on time, on
budget and what the stakeholders
Robin Goldsmith is an author and testing want.
consultant specializing in business engineer- While such an unflattering depic-
ing and requirements.
tion of project effectiveness obviously

FEBRUARY 2008 www.stpmag.com • 25


CONTROL CHAOS

reflects a somewhat contrarian view of ures were made reliably. Glass ques- gle answer fails to take into account
IT, there’s plenty of room for even tioned whether the CHAOS reports the pattern of variation in the source’s
more contrarianism. In fact, the seeds are valid and said his personal experi- actual project data, which I’m sure
of this admittedly contrarian article ence indicated a much lower IT proj- most of the CHAOS sources were high-
were born in a letter to the editor and ect failure rate. ly unlikely to consider, let alone have
subsequent correspondence with (I He may be right, both about his quantified. Thus, the respondent may
believe he’d accept the characteriza- personal experiences and their being have felt the 100-150 range was most
tion) contrarian Bob Glass, who may representative for the broader IT proj- typical, even though perhaps a few
have been the first to ect population. We can’t instances are in the 25-50 and 400-500
publicly question the really tell either for sure, ranges. It’s like the old story of a per-


seemingly sacrosanct though, because the fact son whose left foot is in a bucket of
inviolable CHAOS find- is that nobody else seems boiling water and whose right foot is in
ings that the great major- to have sufficient suitable a bucket of ice. The average of the two
ity of IT projects fail (see objective IT project meas- temperatures might be a comfortable
his “Loyal Opposition” ures that could validate 90 degrees, but neither foot feels at all
article “IT Failure Survey responses whether or not the comfortable.
Rates—70% or 10-15%?” CHAOS reports’ main While CHAOS groups within proj-
in the May-June 2005 in general finding is accurate. ect size categories, it also consolidates
IEEE Software). In fact, this lack of across categories, and it’s unclear
For more than a tend to be objectively measured IT whether its calculations accurately
decade, the Standish project data isn’t just lim- reflect the varying sizes of reported
Group has published a shaky ited to formal study projects. Should a one-month one-per-
series of CHAOS reports reports. It’s highly unlike- son project be given the same weight
that describe embarrass- in both ly that (m)any of the as a one-year 100-person project?
ingly low IT project suc- organizations surveyed Should a project’s size be defined
cess rates, starting with reliability for CHAOS based their based on the original estimate or the
an abysmal 16 percent in answers on reliable and bloated size the report tells us the
1994 and improving to 34 and validity. valid facts, which raises project eventually reached?
percent in 2006 (a num- serious methodological Regardless, to come up with aver-


ber that looks good only reasons to question the ages, it’s necessary to convert each of
in comparison to preced- CHAOS figures’ validity. those 100-150 range responses to an
ing years’ reports—or for The CHAOS data comes average score of 125. Although the
a baseball batter). Even from surveys of IT execu- reports do show distributions of
though it’s now well over tives’ perceptions regard- responses for the various ranges, they
a decade old, the 1994 ing project measures. I focus on single average overrun per-
report seems to continue to be the one think we all recognize that survey cents, which take on an appearance of
cited (and read) most, by me and oth- responses in general tend to be shaky authority and scientific precision.
ers, primarily I assume because it’s the with regard to both reliability and Moreover, even if the CHAOS-
only one available in its entirety for validity. reported average overruns of 189 per-
free (find it at www.standishgroup.com Even when well intentioned, survey cent and 222 percent are absolutely
/sample_research/register.php). Since responses often are guesses or, at best, accurate, it’s unclear exactly how to
subsequent reports are priced prohibi- broad approximations. I don’t know interpret these reported overruns. If
tively (for me), I and presumably most about you, but I’m asked to respond the budget was $100, does an overrun
people know only snippets of them all the time to surveys that ask ques- of 189 percent mean that the project
reported in the trade literature, such tions I don’t have good answers to, actually cost $189 or $289?
as the current 34 percent figure either because I don’t know or On the other hand, despite such seri-
(which was described in SQE’s because none of the answers actually ous methodological issues, a lot of sub-
2/22/2007 Between the Lines e-mail fits my situation. Nonetheless, it does- jective data, including my own, does
newsletter). n’t stop me from answering some- support the general tenor of the
So far as I can tell, though, the thing, probably with vague impres- CHAOS conclusions.
Standish Group’s methodology, find- sions or answers that I know have no For example, when I describe the
ings and analysis have remained fairly basis. 1994 CHAOS numbers in my semi-
consistent over the years, with changes nars/speeches, I usually ask the partici-
essentially only in the absolute figures. Illusion of Precision pants whether the report reflects their
Consequently, concerns raised by the Although I’m sure it’s unconscious, own project experience. Over the years,
1994 report are reasonably likely to statistical manipulations can both dis- thousands of attendees regularly have
remain current. tort reality and imbue an illusory given what seems to be almost always
appearance of precision. For instance, unanimous affirmation. Furthermore,
Is IT Really That Bad? “About 90 percent” sounds much less the reports wouldn’t be so widely cited
Measurements must be both reliable precise than “89.51 percent.” Surveys and accepted unless people do find the
and valid. The reports’ year-to-year often ask people to pick ranges of val- conclusions consistent with their own
consistency indicates that the meas- ues; say 100-150. Having to give a sin- experiences.

26 • Software Test & Performance FEBRUARY 2008


CONTROL CHAOS

It’s All in Your Head scientific community’s agreement that The Real Cause of Failure
However, Glass also has a point. IT global warming will deplete animal life It’s not only pathetic TV wanna-be
projects ordinarily do deliver systems in the oceans and submerge coastal celebrities who respond to dysfunction
that people can and do use. Both Glass cities doesn’t cause individuals to cut with greater dysfunction. Many, if not
and CHAOS are right, to an extent, their gas guzzling. Statistics on the most IT projects are destined from the
and focusing on either’s conclusion damaging health impacts of obesity start for failure because management
alone may itself obscure other signifi- don’t make individuals eat better or has arbitrarily dictated a project budg-
cantly important issues. To understand less. And CHAOS’s reported 84 per- et and schedule that bears no relation-
this balance more fully, certain inter- cent IT project failure rate doesn’t ship to the work to be done.
esting psychological factors also need cause individuals to change how they But that’s just the beginning of a
to be taken into account. do projects. downward spiral. When budgets and
CHAOS is correct that IT projects Perhaps the problem seems over- schedules are nonsense, overrunning
routinely are late, over budget and not whelming, or one’s own power to them becomes nonsense too. So what
entirely what is expected. The aberra- affect it seems so insignificant, but if nonsense budgets and schedules are
tion is large, and it’s really not neces- ultimately it comes down to the nor- overrun 189 percent and 222 percent
sary to quibble over CHAOS’ specific mal psychological defense mecha- of the time? People doing the projects
numbers. Glass is right that usually nisms people enlist unconsciously to don’t take a personal stake in the out-
projects produce some kind of work- protect their self-images. We’ve gotten come because they go into the projects
ing system. Once people can start so good at denying anything that “knowing” the nonsense budgets and
using the system, they get busy with it reflects poorly on us, so unwilling to schedules are impossible to meet,
and tend to forget that it was late, over
budget and perhaps only a portion of
what they’d expected.
That’s fine; life must go on—but
this forgetting can involve some bigger
ramifications. First, the same scenario
recurs, project after project. As an
industry, IT tends not to learn, or per-
haps tends to learn, but not necessari-
ly the right lessons, from our experi-
ence. Instead, we’ve learned to
become quite proficient at a develop-
ment process that reliably repeats fail-
ure—late, over budget and not what’s
expected—and then through denial
essentially accepts that failure as ade-
quate.
Second, by failing to relate our own
actions to our results, we prevent the
personal initiative needed to make
meaningful improvement. Consider a
perceptual paradox phenomenon
that’s evident in the worlds of politics
and culture. For example, survey after
survey finds low approval ratings of
Congress, yet congressional re-elec-
tion rates historically are close to 100
percent, sometimes not even impeded
by incarceration or death. Similarly,
surveys repeatedly find that people say
the American education system does a
terrible job, but their own local school
is just fine. recognize, let alone take responsibility which becomes a self-fulfilling prophe-
People often have an understand- for our results, and so willing to shoot cy, regardless of how “objectively” fea-
able disconnect relating broad-scale the messenger, that we not only fail to sible the budget/schedule may be.
external results to their personal take appropriate corrective actions The more the worker bees grumble
sphere. That’s a major reason why peo- but also sometimes intentionally and miss their targets, the more the
ple seldom change their individual engage in additional self-defeating managers feel compelled to dictate
behaviors in response to well-known behaviors. For example, consider the yet-even-more nonsensical budgets
big problems. Annual reports of lung dysfunctional behaviors that get acted and schedules, thereby ensuring fail-
cancer death statistics tend not to out even more excessively every after- ure and confirming to them that they
cause individuals to stop smoking. The noon on the television scream shows. were right in having to hold the

FEBRUARY 2008 www.stpmag.com • 27


CONTROL CHAOS

troops’ feet to the fire. Dysfunction rent practices won’t produce better tem/software requirements to satisfy
begets more dysfunction. requirements. them, which of course contributes to
The business users don’t know It’s understandable that the report management’s propensity for arbitrar-
about these internal dynamics. They fails to realize this distinction, because ily establishing budgets and schedules.
only know that as an industry, IT the industry is unaware that such a dis- Once a project is set to fail, albeit
almost always blows project budgets tinction exists. Perhaps one reason by management’s actions or absence
and schedules. They perceive that IT why user involvement is low is that thereof, managers quickly distance
doesn’t know what it’s doing, and thus managers may sense that just giving themselves from the project.
they may not believe what IT says, more time by itself often may not pay
which further impedes IT’s ability to off. One More Little Oversight
keep its promises. Similarly, the report mirrors the Roughly half the time spent in IT proj-
Project managers’ psychological industry’s general lack of awareness of ects is taken up by testing, yet the
defense mechanisms pre- the important distinction CHAOS report’s failure factors don’t


vent them from becom- between real, business mention quality or testing.
ing aware, let alone requirements and prod- Again, this probably reflects a lack of
believing, that they may uct/system/software awareness among the surveyed indus-
have credibility issues. So requirements. try executives, which in turn translates
they attribute their diffi- The common use of into the project failures described.
culties to other, often-
Real business the term requirements Inadequate quality clearly causes users
irrelevant factors and refers to the require- not to receive what they expect, and
mistakenly divert atten- requirements ments of the product, sys- the unplanned time needed to fix
tion to these perhaps tem or software that is defects is a major contributor to proj-
non-issues instead of tend not to expected to be created. ect budget and schedule overruns.
addressing their real Said product, system or Without awareness of—and
credibility problems. change nearly software actually is the informed attention to—quality and
This further reduces high-level design of one testing, quality and testing activities
their likelihood of suc- so much of the possible ways how are too little and too late. Too many
cess and their already- to accomplish the pre- defects escape to be found by the
diminished credibility. as people’s sumed real, business- users. Those defects that are found
The business puts more requirements deliverable during development tend to be discov-
pressure on IT manage- awareness whats that provide value ered so late that they’re very expensive
ment, which leads to when delivered/accom- to fix. Proactive testing can turn this
even more nonsense dic- of them. plished/met/satisfied. situation around, actually helping
tates and more overruns; The CHAOS report projects deliver quicker and cheaper


and the cycle perpetu- identified incomplete by catching and preventing more of
ates. and changing require- the errors earlier, when they’re easiest
ments as two separate to fix.
Close, But Missing issues. In fact, the main All in all, in spite of significant
Critical Distinctions reason that product/sys- questions about their measures and
Ultimately, project budgets can’t help tem/software requirements change is analysis, the Standish Group’s CHAOS
being nonsense unless they’re based because they aren’t defined accurately reports seem generally accurate in
on adequately defined real require- and completely enough in the first finding that most IT projects are late,
ments. At first glance, this seems very place, which in turn is mainly due to over budget and not entirely what is
much in line with the project success the failure to adequately discover the expected.
and failure factors that CHAOS analy- real business requirements. Designs To a considerable extent, such out-
sis identifies. (including their product, system comes are the inevitable consequence
Requirements and user involve- and/or software requirements) can of arbitrarily mandated project budgets
ment issues certainly dominate the change frequently and rapidly. Real and schedules, inadequately defined
tops of the factors lists. They indeed business requirements tend not to business requirements and too-
are important, but I fear the report change nearly so much as people’s little/too-late reactive attention to qual-
simply parrots overly simplistic, wide- awareness of them. ity and testing.
spread conventional beliefs that con- Such problems persist in part
tinue to miss the distinctions critical Missing Resources And because few organizations have suit-
for making meaningful improvement. Unrealistic Expectations able or sufficient objective factual
The report focuses only on amount The main remaining failure factors measures of their projects and because
of user involvement. While a certain identified in the report include lack of those responsible for IT haven’t creat-
quantity of user involvement is neces- resources, unrealistic time frames, ed an environment in which they and
sary for discovering the real require- unrealistic expectations, and lack of other involved individuals take
ments, it’s not sufficient. Rather, it’s executive support. The first three are enough personal responsibility for
the quality of that involvement that all largely attributable to not ade- their results.
really matters. Merely subjecting users quately defining the real business The solution? Know, document and
to more of the same ineffective cur- requirements and the product/sys- adhere to the real requirements. !

28 • Software Test & Performance FEBRUARY 2008


Photograph by Alexey Klementiev/Fotolia.com

The Key to ROI’s in The


Quest for Quality
By Frank Grossman

Q uality—you know it when you see it.


Yet the concept of quality has little
meaning unless it’s related to a specific
their investments.
It should come as no surprise that the com-
panies seeing positive returns from their SOA
initiatives are the early adopters in vertical
service, object, experience or desired industries such as financial services, telecom-
munications, energy, insurance and healthcare.
result. As the use of service-oriented architec- These global leaders in technology optimization
ture moves from early adopters to mainstream have embraced initiatives such as SOA as a core
corporate initiatives, companies are struggling business strategy and are realizing positive
to achieve their intended goals of business agili- returns. They also take a pragmatic approach,
ty and cost savings from service reuse. Some knowing that a successful SOA initiative
industry watchers and media surveys report that depends on SOA quality.
many, if not most companies are engaged in
Frank Grossman is president and cofounder of SOA
some form of SOA initiatives—but very few are
test-tool maker Mindreef.
at the point of realizing a positive return on

FEBRUARY 2008 www.stpmag.com • 29


SOA QUALITY

Web Services vs. SOA as the “law enforcement” for an SOA development of rogue, redundant
What exactly is SOA quality? First, let by controlling service implementa- services for specific applications in
me tell you what it’s not. Software tions with the process, procedures and order to meet their project dates. Not
quality initiatives are usually associated standards deemed necessary for only is quality at risk, the fundamental
with testing. And more recently, test- quality. value of the SOA initiative breaks
ing often means testing a single Web While SOA governance solutions down.
service using a waterfall approach. vary in functionality, governance must One approach to achieving SOA
This might work for a Web service that be applied to design time, change time quality is to deploy an SOA quality
has a distinct function with a specific and runtime to be truly effective. The gateway along with governance solu-
life cycle, and can be designed, devel- challenge is that each phase requires a tions. This can complement the con-
oped, debugged and deployed. It then different approach to maximize the trol aspect with interfaces designed for
exists in production until being enablement and adoption of gover- various roles within the process and
different approaches for design time,
change time and runtime. The SOA
quality gateway then becomes the
quality enablement point for reg-
istries.

Collaborative Quality
In any SOA, the more loosely coupled
services are, the more communication
is needed among every team and mem-
ber involved in the design, implemen-
tation and support processes. Agility
and interaction are constantly at risk
in these loosely coupled environments
because of the temptation for each
group to be autonomous.
Collaboration begins at design
time. Architects and business analysts
need to be aware of existing services,
Photograph by Amanda Rohde

what they do and if they’re being used


by other applications. Knowing that a
Web service is already in use provides a
level of trust that is essential to service
reuse and the overall success of an
SOA. At the same time, developers
need to know that quality services exist
revised, updated and replaced with a nance controls. In other words, devel- before building a redundant service.
new service, which is subjected to the opers require education more than To ensure SOA quality, collabora-
same cyclical process to ensure quality. enforcement when it comes to build- tion must continue during runtime
On the other hand, an SOA doesn’t ing compliant services (that is, don’t and change time. Once services are
have a life cycle. SOA is an architec- just tell them a service is not accepted, identified and understood, architects
ture made up of infrastructure and show them why and offer guidance on can begin prototyping an application.
services that must constantly interop- how to fix it). Additionally, to obtain SOA quality,
erate. It doesn’t go offline and it can’t Architects and analysts require business analysts, developers, QA and
be replaced. It can, however, evolve more than just a listing of available support must find an easy way to work
and expand. It can also improve or services; they need translation of the together, regardless of language or
degrade. Therefore, the quality of an XML-formatted descriptions into platform. Traditionally there hasn’t
SOA is reflected by the amount of use something human-readable that they been a way for them to effectively com-
and reuse of the services within it, and can easily understand. municate, since their functions are so
by how well its implementation meets Most registry/repository systems different. A collaborative approach for
the needs of the business, even as focus on design-time governance in SOA implementations will help
those needs evolve. See “SOA Quality the same way they address runtime. increase the agility and reuse of Web
Defined” for Wikipedia’s take on SOA They’re designed to regulate access, services by ensuring quality and build-
quality. security and performance by con- ing trust with service consumers.
A common misconception among sumers. Without varying approaches,
SOA project leaders is that quality con- the strict enforcement becomes detri- The Five Components
cerns can be addressed with a gover- mental to the SOA’s intended goals. If SOA quality doesn’t exist in any single
nance solution alone, typically in the developers grow frustrated and uncon- part of a system, nor is it the sole
form of a registry/repository. Vendors vinced of the standards and policies responsibility of an individual or team.
have positioned governance solutions applied by the registry, it often leads to The architecture must have SOA qual-

30 • Software Test & Performance FEBRUARY 2008


SOA QUALITY

ity throughout. To do this, you must And unlike traditional software appli- stand and resolve problems quickly
build a solid foundation for quality cations, SOA testing occurs while and often need to reproduce scenarios
consisting of five core traits: many services are in various stages of when a failure occurred. When service
• Compliance development or production. Yet with developers need to get involved, com-
• Prototyping service-oriented architectures, testing plete problem data needs to be shared

S
• Testing is unavoidable.
• Diagnostics Therefore, teams need tools that
• Support can provide the necessary user inter- OA QUALITY DEFINED
As with any foundation, if one of face, simulate unavailable services and
the components is weak or missing, ensure that all team members—even
In Wikipedia, SOA quality is defined as
the entire structure is at risk. ones without programming or XML
“a service-oriented architecture that
Compliance. SOA quality starts with language knowledge—can test servic-
meets or exceeds business and techni-
compliance with standards. Non-com- es. It’s also important that test scripts
pliant services pose the highest risk for are accessible so that retesting can be cal expectations by consistently yield-
SOA quality and positive business easily performed when a service policy ing value in the form of cost savings,
returns, and simply can’t exist if an changes or if a new service consumer productivity and time to market.”
SOA is to be successful. Even well-writ- uses the service in a different way.
ten services can’t guarantee broad Diagnostics. Remember, this is still This is achieved through continual
interoperability unless standards and software. So no matter how well Web optimization of all components within
best practices are well designed and services and business processes are the SOA environment to ensure maxi-
adhered to throughout an organiza- tested, problems will still occur. With mum adoption, business agility and
tion. an SOA, problems often need to be service reuse.Therefore, SOA quality is
Developers must embrace stan- solved in real time, and may involve a key component of reaping the intend-
dards, rules and policies by seeing disparate teams and systems. This ed benefits of an SOA—it’s the strate-
value in what they provide. Architects requires collaborative diagnostics. gy needed to achieve maximum busi-
and governance teams must educate Determining if a Web service can per- ness benefit.
others as to what the standards are, form its intended function is often a
why they’re in place and most impor- time-sensitive issue that may require
tantly, how developers can improve fast identification of the root cause of with other team members who can
their projects by becoming compliant. a problem. This also means that all simulate different scenarios to more
Finally, analysts and architects must members of the team are responsible effectively diagnose problems.
embrace compliant serv- for helping to diagnose
ices and leverage them problems. For this to be Communication,Trust and Control


in SOA applications to effective in an SOA, Communication. The ability to commu-
ensure trust and reuse. strong diagnostics are nicate effectively is an essential core
Prototyping. Seeing is essential to providing element in a successful SOA. It’s as
believing. As business and maintaining SOA important to SOA technologies as it is
and IT groups work quality during runtime, to the people involved with the envi-
together on SOA initia- Strong and for preventing run- ronment. Communication in a tech-
tives, prototyping is one time issues by catching nology domain is often referred to as
of the best ways to reach diagnostics are them in design time and interoperation and occurs across differ-
agreement on a WSDL change time. ent platforms, languages, locations,
contract before any code essential Support. The final policies and standards.
is written, and to deliver component in the foun- Communication must take place
SOA quality early. to providing dation of SOA quality is between people, because individual
Prototyping lets busi- support. It is absolutely team members depend on each other
ness analysts, architects and essential to have a mecha- to do their jobs, complete a task, solve
and developers design nism for supporting the a problem or contribute to a project.
and develop very usable maintaining disparate groups that use The communication between business
interfaces early in the services, without over- and IT is a perfect example. This is
process, thereby creating SOA quality whelming the develop- often associated with collaboration.
services designed for ment teams. Also, sup- But communication is more than


reuse. It also allows con- port is fundamentally dif- just interoperability and collabora-
sumers and testers to get ferent in an SOA because tion—it involves the interaction
involved much earlier in it involves two phases that between people and technologies,
the design process, span design time and which is often where quality breaks
reducing the overall runtime simultaneously. down in an SOA.
development cycle. As services are Visibility is an important attribute
Testing. SOA has many moving exposed for use and reuse, consumers offered by a registry, providing a cata-
parts. It’s virtually impossible to test will seek support to assist in the devel- log of services that are available for
every Web service and its interaction opment of their applications. In pro- use. While this helps architects and
with its dependencies within an SOA. duction, support teams need to under- developers to communicate, visibility

FEBRUARY 2008 www.stpmag.com • 31


SOA QUALITY

B
alone is not enough. Services need to
be visible, accessible and easily under- UILDING BLOCKS OF SOA
stood to ensure SOA quality.
Accessibility and understanding When building a foundation for SOA quality, consider the following:
help make visibility a form of commu-
nication in an SOA. Together, they • How will the individual or team responsible for the overall quality of your SOA manage
allow team members to easily view and all the facets of SOA quality?
comprehend what a service can and • How do you ensure that best practices are actually followed, once they’re defined?
can’t do, which can be a challenge
• How do you enable team members to communicate effectively, given diverse development
when interacting with a technical reg-
environments and skill levels?
istry that lists a myriad of XML files
and WSDL contracts. • How do you enable team members, especially those are remote or who don’t have
An SOA quality gateway can pro- advanced coding or XML skills, to test effectively?
vide the accessibility and understand- • How do you efficiently reproduce and solve problems that span disparate systems and
ing attributes to an existing registry disparate teams without finger pointing?
and address the requirements for • How do you achieve the reuse of your SOA service assets across your team?
effective communication within an
SOA.
Look for platforms and tools that allow team members to:
Trust. The primary characteristics
of SOA quality for any business are
agility and reuse. Trust is what drives 1. Define executable standards and specifications early in development that team members
service reuse. It’s critical for SOA must develop to. These help ensure that your Web services will be of high quality.
teams to be aware of existing services 2. Create and run analysis profiles to verify that your services meet the best practices
in the SOA, to understand what they defined by your organization.
can and can’t do, and most important- 3. Save services in reusable workspaces that all team members can run, regardless of skill
ly, to have trust and confidence that
level. This promotes team communication.
the services will execute as intended.
4. Save workspaces, simulations and test scripts that all members of your team can run to
Trust is relevant in nearly every
aspect of an SOA. Architects need to test your services.
trust the services they choose to use. 5. Run previously saved simulations to effectively reproduce, diagnose and solve problems.
Developers need to trust that an exist- 6. Save services simulations for reuse of your existing SOA service assets in new imple-
ing service will be appropriate for an mentations.
application versus building a new one.
They must also trust that the imple-
mented policies and standards are time and runtime. SOA quality.
meaningful and add value. SOA lead- Governance involves policy
ers must trust that services are being enforcement and sometimes requires Achieving Business Objectives
added to the registry and reused. And changes in human behavior, develop- As with any business application, the
QA teams must trust that policies and ment concepts and processes. Control SOA’s ultimate goal is to help an
standards will enable compliance and also involves reducing the number of organization achieve its business
interoperability at runtime. production issues during runtime and objectives. And SOA quality is
Without trust, an SOA will never resolving those issues quickly. required to ensure that the service-ori-
achieve reuse, which leads to redun- Further, control involves under- ented architecture meets or exceeds
dant or rogue services. Trusted servic- standing the dependencies applica- business and technical expectations,
es, however, will lead to “socialized tions have on multiple, disparate serv- whether in the form of cost savings,
quality.” As services gain trust, users ices, and prompts the need to test serv- productivity, better customer service
will share their positive experience ices that depend on other services that or shorter time-to-market. Achieving
with peers—the trust factor grows and might not be directly accessible. this quality means understanding the
reuse increases. This is what we mean overall goals of the SOA strategy and
by socialized quality. SOA quality opti- Quality In = Quality Out optimizing the environment for suc-
mization depends on creating trusted This variation of the old “garbage in, cessful execution.
services and then socializing that qual- garbage out” adage from the comput- SOA quality also requires a top-
ity throughout the SOA. er science field is relevant to the SOA down approach to mapping out those
Control. Governance in an SOA is registry/repository concept. If quality objectives and a bottom-up strategy for
about control. Often what is lacking, isn’t enforced when services are maintaining it. By laying a foundation
however, is the ability to enforce poli- entered into a registry, quality is at risk for quality at the outset of any new
cies. Control without enforcement when services are used from that reg- project, companies can establish a
really isn’t control at all, and will not istry. process for success that will become a
yield SOA quality. Although gover- While registries are an essential ele- standard practice as new services are
nance alone isn’t the answer to SOA ment of SOA governance and gover- added—ensuring consistent trust and
quality, it’s a critical element, and nance is about control, it goes well reuse, and a high-quality SOA at all
comes into play at design time, change beyond simple governance to ensure times. !

32 • Software Test & Performance FEBRUARY 2008


A Case Study That
Shows How Not To
Staff a Project

By Prakash Sodhani

I ’ve long wanted to write an article about man-


ual testing. As an automation specialist, I don’t
think much of testing efforts that include only manual test-
ing. But I firmly believe that a prerequisite for automation
is a somewhat stable application, which can be ensured only
by doing at least some manual testing.
Let’s look in on a manual testing project that went awry.
It involves someone we’ll call Peter working in a company
that sells merchandise over the Internet.
Recently, Peter worked on a project as the lead tester.
The project, referred to internally as “the Rewrite,”
involved an existing application that was being rewritten
using newer technologies. The app was designed to allow
users to come to a Web site and purchase products. Nothing
very special so far; an e-commerce app with lots of process
flows and state combinations to be tested. But since it was
being rewritten, it carried the potential for lots of new
defects in the system and lots of time to test all functionali-
ty from end to end.
Most of the testing was of the manual variety, with a bit
of load testing thrown in for flavor. What follows is a sum-
mary of Peter’s experiences. Each topic is divided into three
parts: What was done, what went wrong, and what I believe
should have been done.

Prakash Sodhani is a quality control specialist at a global IT services


company based in Texas.

www.stpmag.com • 33
DOS AND DON’TS OF TESTING

project. shortened time frame. Testers should


Also, no planning was done. have been asked regularly what they
Management simply assumed that this were testing and how it was going.
sprint would be like any other and fol- Since the testers were new, chances
low the existing plan. But as the proj- were good that many common test sce-
ect progressed, the number of applica- narios would be left out. Trust is good,
tion features increased by about 50 but blind trust can lead to disaster. It’s
percent. To make matters worse, no better to “trust but verify.”
one realized that we’d lose three days
of testing due to holidays. Load Testing
What was done?
What should have been done? In one of his earlier jobs, Peter worked
First and foremost, at least one tester as a load test specialist. He worked on
with experience with the application a variety of projects and learned many
or a similar project should have been of the minute details of load testing.
present at all times. No logic can justi- However, working on this project
fy the decision to assign 1.5 testers to a showed him a different side of the
project that originally required seven. practice.
Managers also should have sought Peter was asked to perform load
input from at least one tester familiar testing just a few days before the appli-
with the original application to deter- cation was scheduled to go live. He was
mine how much work was involved. literally instructed to “write some
Test planning also should have scripts and run them.” No one had any
been made a priority for a project of idea how many users to simulate, what
such importance to the business. kind of scenarios to run, and what was
Scrum Masters and leads should be in expected from the tests. It seemed that
Resource Allocation and Planning the position to know and have exactly all they wanted was to hear that “x
What was done? what they need for such a project number” of users ran successfully with-
The development team followed rather than just attending morning out any errors.
Scrum, an agile methodology. scrums. If test planning had been
Development was divided into differ- done correctly, a viable plan to com- What went wrong?
ent sprints, each one month long. pensate for time lost during holidays What was done is just about the exact
Each tester was assigned to different would have been taken care of, and opposite of what correct load testing is
sprints based on a percentage deter- testers wouldn’t have to spend late supposed to be. The goal of load test-
mined by the testing manger. nights getting the job done. ing is to find the breaking point of the
Prior to the initial launch, the orig- application, using varying real-time
inal application development team Prioritizing Features scenarios. Making the matter worse
included about 14 developers. The What was done? was the assumption that test scripts
Rewrite project was assigned one of In the absence of a plan, testers went take little time to write and can be cre-
the original testers, who’d been testing about testing the application as they ated and executed the same day.
this application for the past couple of thought best. With countless flows to This total lack of knowledge in the
years. Unfortunately, the manager be tested, there was no input from the area of load testing, and the fact that it
approved a month-long vacation for Scrum Master, development leads or was done as a formality, left Peter
him during this time. the test manager as to which should be unmotivated. It became worse and
So we were assigned only two testers a priority. And while experienced when he tried to explain how useful
for this project; one, a new hire who testers might be expected make these load testing can be when done cor-
joined the team a couple of weeks decisions, none were assigned to this rectly; no even responded. And
back, was allocated 100 percent to this project. because the order to run load tests was
project. The other was Peter, who was so close to launch, it was clear to Peter
allocated 50 percent. So in essence, What went wrong? that even though flaws were found, the
the project had 1.5 testers. A lack of guidance and prioritization application would go into production
led minor functionalities being tested anyway. Peter’s outlook changed from
What went wrong? first, leaving little time for major ones, seeking the breaking point to running
Since we were just starting the Rewrite and end-to-end flows were also inade- the load tests without any errors.
project, it was an educated guess as to quately tested. Most of the time was
how many testers and how much of spent on individual features rather What should have been done?
their time would be needed. When this than on the complete solution. Load testing is a specialized form of
application was first developed, seven testing that requires time and plan-
testers were assigned, with 100 percent What should have been done? ning. Just asking someone to write test
of their time allocated to testing. So it Scrum Master and leads should have scripts shows a lack of knowledge.
made no sense to us to assign only two made sure that testers knew what was Also, load testing should have been
testers, both of whom were new to the expected of them, particularly in their included in test plan as a priority item,

34 • Software Test & Performance FEBRUARY 2008


DOS AND DON’TS OF TESTING

and resource allocation should have able to work during weekends. His eran testing level.
been done accordingly. Doing these question was, “Is everyone coming to Granted, some of the testers who
kinds of activities just for formality’s work so that we can get a lot done? were assigned to help had previously
sake is a total waste of time and After all, it’s a UI-based application, worked on the same application, but
resources, and should be and everyone can play all they could do was to go through
avoided, especially when around with it to see if the basic flows. The end result? They
everyone is hard pressed they see something spent only a few hours reviewing the
for time. wrong.” As soon as the application and concluded that every-

Teamwork
Anyone reading this
should be familiar with
• question came out, the
faces of the Scrum Master
and leads became indig-
nant, appearing to say,
thing looked great. Everyone was
happy to hear that, and the launch
remained on schedule. No one had
any idea of what had happened during
the concept of team- It’s essential “Are you kidding? Why the previous month, and Peter was so
work, which, if improp- should we work week- disgusted that he just let it go.
erly managed, can have a to keep ends?”
dramatic effect. In this That was the final What should have been done?
case, a small teamwork people blow, and Peter lost all For help to be effective, it has to be
incident led Peter to interest and concern offered at the right time, or you can
stop testing the applica-
tion and give it a green
motivated about the project. From
that moment on, all of his
foster a false impression that every-
thing is great. Peter knew that the sys-
light for release to pro-
duction.
during the efforts were toward going
to production as soon as
tem still had many defects, but was
totally unmotivated to point them out.
possible, after which Had he noted this early enough in the
What was done?
project. came the wait for the life cycle, the project could have been


Even with Peter’s 50 per- inevitable production rerouted toward an effective course.
cent involvement, he defects to come in. There are lessons to be learned in
spent around 14 hours everything we do. Testing is a subjec-
each day during a two- What should have tive practice, and carries no hard-core
week period. He was, been done? formulas for success. It’s important to
after all, the more expe- The team should have learn from experience; in this case,
rienced tester and was pulled together. Even a from our fictional friend. Peter’s mis-
therefore obliged to short presence or small takes encapsulate many of the pitfalls
work on the more complex functional- show of effort and willingness to pitch I’ve witnessed throughout my career
ities. Most evenings, no developers in on the part of leads and managers in quality control. Keep him in mind
were available to support his issues. In during crunch time goes a long way when you face some of your own. !
every morning scrum, it was repeated- and can motivate the team toward the
ly mentioned how busy everyone was common goal.
and that they couldn’t deliver some of While it’s a good idea to circulate
the requested functionality, even e-mails or memos of praise when
halfway into regression week. everything is said and done, it’s essen-
tial to keep people motivated during
What went wrong? the project. When true effort and com-
Here’s the funny part: In one of the mitment have been maintained
morning scrums, Peter initiated the throughout the project’s life cycle,
notion of working weekends to get after-project commendation is all the
things done. The development lead more sincere and long lasting.
asked his developers who would be
available to work. All that could be Too Little, Too Late
heard was crickets. These are same What was done?
developers who weren’t able to deliver Everyone needs help at one time or
the required functionalities on time another. This project involved many
and were always “busy.” “Busy doing extra hours, and the test manager cor-
what?” Peter wondered. It was an awk- rectly asked Peter if he needed any
ward moment, and I would have help to get his testing done.
expected more concern to get things
done from the leaders and people who What went wrong?
built the product than from a tester Even though Peter finally got some
with less of a stake in the project’s suc- help, it came too late, right before the
cess. application was to go live. It’s difficult
In another meeting just a couple of for someone to walk in to a project
days before the production release, where so much has happened and be
Peter was asked if he would be avail- expected to instantly perform at a vet-

FEBRUARY 2008 www.stpmag.com • 35


Best Prac t ices

Web Services Burden Java


Developers With Testing
Frank Cohen is a 30-year away. But there’s a small hundreds of millions of desktops and,
veteran of the software problem, at least for those according to the official history at
industry and CEO of in charge of the PushTo www.java.com/en/javahistory/timeline
PushToTest, a company Test Web site. .jsp, has even been to Mars by way of the
that earns its bread selling “How do you test any of Java-powered Mars Rover, Spirit. Back
testing tools and services. that?” asks Cohen. “There on Earth, the Web has exploded and is
So he’s an altogether are no standards bodies fast on its way to joining electricity and
unlikely candidate to be standing behind JavaScript phone service as basic utilities in the
fazed by much in the way of or behind these two data developed world. Internet protocols
testing Java-based Web models that this JavaScript have matured to the point where Web
applications. Yet even the program is using. services and SOA are now possible.
Geoff Koch
venerable Cohen had “We never set out to Despite these dramatic changes, it’s
something of an epiphany during a add AJAX capability to our Web site; it still true that, depending on the cir-
recent upgrade of the software that runs just happened to show up with the cumstances, code should be subjected
PushToTest.com. upgrade,” continues Cohen, author of to some combination of unit, integra-
The site runs on Zope, an open the book “Java Testing and Design: tion, functional, system, performance
source application server, and Plone, an From Unit Testing to Automated Web and acceptance testing. However, the
open source content management sys- Tests” (Prentice Hall PTR, 2004). rise of Java Web applications has
tem. Among the many recent enhance- “That’s the epiphany. The world is changed the notion of what it means
ments to the latest version of Plone that marching forward because developers to test software and even the idea of
Cohen’s team installed is a slick AJAX love the rich Internet that AJAX what exactly comprises an application,
interface to facilitate the process of brings. The world is marching forward says Kishore Kumar, director for the
searching the site. Start typing what because enterprises know they can UST Global Java Center of Excellence
you’re looking for in the search box and communicate with other enterprises in India. “Now there are other consid-
up pops a floating window with a list of using things like SOAP, Web services erations, including how to test with
suggested results. Type a little bit more and XML-RPC—and there’s no gate- SOA, how to make sure that exposed
and the list changes, with any luck con- keeper to any of the message formats services are consistent with terms of
verging on your search term. that exist.” any relevant contracts, and that
Say you’re looking for the slides changes to such services comply with
Cohen presented at the Practical Beyond Test-Driven Development the broader regulatory environment,”
Software Quality Testing 2007 confer- Cohen’s story is evidence of the tecton- says Kumar, author of “Pro Apache
ence in September. Type “Prac” in the ic shifts occurring in Java testing. Java Geronimo” (Apress, 2006).
PushToTest.com search box, and the 1.0 was released in 1995, when client- Kumar, who also serves as a senior
first item in the floating window jumps server computing was entrenched and technology evangelist for the Aliso
to an entry on Cohen’s blog where the the Web was more or less in its infancy. Viejo, Calif.-based UST Global, sug-
slides are posted. A well-established division of labor had gests that what’s needed is an altogeth-
This functionality comes from a wee emerged in the world of software. er new mindset about what constitutes
bit of JavaScript associated with that Roughly speaking, developer organiza- successful Java programming. The
search box. One data model allows the tions wrote programs, QA organiza- aim, he says, should be to move
JavaScript program to send queries as tions tested programs, and IT organi- beyond even relatively ambitious prac-
they’re entered to the back-end server. zations supported programs. Indeed, tices such as test-driven development,
Another data model formats the query most early Java programs were accord- where test cases are written before the
results in HTML and displays the results ingly handed off from team to team on production code that implements new
live. From a user’s point of view—at their way from requirements docu- functionality. Soon, he insists, develop-
least that of an impressionable colum- ment to finished product. ers will be measured more by their
nist—the whole experience is slick and Fast forward to today. Java technolo-
yet another sign that the rich Web will gy, now conspicuously open source, is Best Practices columnist Geoff Koch can be
reached at gkoch@stanfordalumni.org.
eventually carry all our browsing blues used by millions of developers, runs on

36 • Software Test & Performance FEBRUARY 2008


Best
Practices

ability to write testable code than to sound, logical argument. Rather, prop- work, which in turn might be repur-
implement new functionality. The fact er incentive is needed, such as fear (do posed for IT to use and run automati-
that the latest Java versions allow this or get fired) or greed (do this and cally as a business service monitor.
developers to easily annotate their make twice as much money as the poor Cohen is right, and his story only
code is a foreshadowing of this future, sap in the cube next to yours). proves that in technology, eventually
he believes. In fact, Kumar insists that Unless I’m missing something, nei- everything old is new again. In the early
each software class might soon come ther such incentive looms on the tech days of computing, at least through the
with notes from the author about just horizon, and as Cohen sees it, there’s mid-1950s, there was essentially no dis-
which combinations of functions can little evidence that the overall level of tinction between testing and debugging.
be used to test it, a change that would coding competence is on the rise. So Developers were expected—required, in
require considerably more skill than how to deal with Java testing in the age fact—to both test and debug. Testing
that used simply to cobble the func- of the rich Web, when you wake up one and debugging were decoupled by 1960,
tion together in the first place. morning and all of a sudden have an and later testing was further subdivided
Granted, Kumar speaks at confer- AJAX-enabled search box on your into the current array of bang-on-the-
ences, writes for a slew of trade publi- home page? code activities, most of which aren’t
cations and leads a team of 400 tech- done by the developers who wrote the
nology specialists, while I’m just a casu- Rx: Repurposing code in the first place.
al industry observer marooned in geo- The only approach, Cohen says, is to Now, the pendulum may be swinging
graphically challenged Michigan. Still, try to combine the efforts of develop- back. Developers, especially in the
I’m more than a little skeptical of his ment, test, QA and IT. For example, open-source, share-and-share-alike
assertion for much the same reason presumably the developer behind the haven of Java programming, should be
that I looked askance at the sunniest search function in the Plone site has expected to promptly debug their new-
unit testing claims in last month’s col- already created a unit test for the fangled Web application as soon as it’s
umn. Here’s the thing: Behavior JavaScript program. Perhaps that unit unveiled as a service—or at least to
change is difficult, and people rarely do test could be repurposed for test and share their secrets for testing it with
what’s best for them on the basis of QA into a functional testing frame- their downstream colleagues. !

Index to Advertisers
Advertiser URL Page

Automated QA www.testcomplete.com/stp 10

EclipseCon www.eclipsecon.org 19

Empirix www.empirix.com/freedom 4

FutureTest 2008 www.futuretest.net 24

Hewlett-Packard www.hp.com/go/securitysoftware 40

iTKO www.itko.com/lisa 8

Software Test & Performance www.stpmag.com 37

Software Test & Performance www.stpcon.com 2-3


Conference Spring 2008

Software Test & Performance www.stpmag.com 39


White Papers

TechExcel www.techexcel.com/stp 13

Test & QA Newsletter www.stpmag.com/tqa 6

FEBRUARY 2008 www.stpmag.com • 37


Future
Future
Test
Test

Who’s Really
While we may expect a rocket guidance
system author to apply 100 percent prov-
ability to his test approach, for the major-
ity of software developers, competitive

Responsible For
market dynamics will dictate when soft-
ware is released.
Given these realities, it’s sensible to
equip people as early in the development
process as possible with tools and process-

Code Quality? es to identify bugs that should never even


make it to testing.
Automated source code analysis
provides one such mechanism by allow-
ing developers to scrub code before
Bugs are easily mass popu- ware testing in today’s mar- check-in and to test for code failures,
lated. In our brave new kets is that it tends to be lim- such as:
Web-connected world, soft- ited to testing what the soft- • Memory and resource leaks
ware vulnerability is a con- ware is supposed to do. • Buffer overflows and code injection
cern of a whole new magni- Today, software is exposed to vulnerabilities
tude. Who’s to blame when all manner of unexpected • Invalid pointer or object reference
that super fridge, which assaults—from the benign • Tainted data propagation and use of
automatically orders your (unexpected use by unex- unsafe libraries or APIs
groceries as they run out, is pected users) to the mali- • Phishing vulnerabilities such as
hacked to run a password- cious (a hacker stealing per- cross-site scripting, request forging
cracking algorithm or file- sonal information). Usage- These aren’t design flaws and they’re
Gwyn Fisher
sharing relay? Who pays driven testing (i.e., testing not baffling architectural failings. They
when that life-saving defibrillator can’t the bits that you expect will be used the are bugs. As such, they should never
power up its paddles because it’s busy most) is a popular model, but it’s increas- make it off the desk of the developer cre-
looking for extraterrestrial life? ingly outmoded as the primary approach ating them.
Although we depend on software to testing software. I believe that automated source-code
from morning until night, it’s riddled analysis at the point of development is
with bugs—many of which are unearthed Expecting the Unexpected the next evolution of software develop-
only after the software is deployed in the What’s needed is an approach that rigor- ment and testing. Ten years ago, the
appliances we purchase and the services ously tests the code under expected and number of developers who routinely
to which we subscribe. The fact is that this unexpected conditions—one that used runtime profiling tools was small.
brilliant multitasking software is devel- ensures all code receives the same test Today there are few professional develop-
oped by people. And people aren’t per- coverage regardless of whether it is expect- ers who haven’t used them at some point.
fect, especially when armed with inher- ed to be executed or not. The approach The smart ones carry these tools with
ently dangerous weapons. must also be thorough, and that’s diffi- them from project to project as a vital
cult to achieve when the testing approach part of their work practices.
Bug Inevitability requires a decision about which paths in Likewise, five years from now, every
The creation of bugs is inevitable, from the software will be investigated and developer will use source code analysis
typing errors to thinking errors, to not-so- which will be overlooked. as a matter of course, and QA depart-
simple design errors and insanely com- A better process would see developers ments will be freed from the tyranny of
plex architectural flaws. Throw in some themselves testing their own code for bug-finding missions to get to the real
garden-variety security nightmares and “simple” bugs. At the developer’s desk, heart of the matter: testing whether
you’ve got a recipe for disaster, or at least agnostic test coverage (typified by pair- products are any good and acting as
a recipe for patches, costly recalls and wise programming and peer review) is full-time customer advocates.
product churn, because the unfortunate possible because the code is available to This approach has another positive
reality is that software development be examined regardless of runtime state. outcome. By making the developers
encourages features and release dates And, since the code isn’t being executed who created the bugs responsible for
over bug-free code. at this stage, thorough coverage of all finding and addressing them, the cre-
It’s time for a new approach to soft- code paths is theoretically possible. ation of those bugs is likely to decline
ware testing and debugging. Today’s mis- The challenge with such approaches over time. !
sion-critical, ubiquitous software must be is obvious: It’s far too costly to have devel-
Gwyn Fisher is CTO of Klocwork, which
defect-free long before it hits the streets. opers testing their own code, and it
makes source code analysis tools.
One of the flaws of traditional soft- would interfere with time-to-market.

38 • Software Test & Performance FEBRUARY 2008


Learn Some
New Tricks!
Get Your Free
White Papers At
www.stpmag.com

Discover all the best software

practices, gain an in-depth

understanding of the

products and services

of leading

software vendors,

and educate yourself

on a variety of topics

directly related to

your industry.