A Publication

PR CM
BE TIC ols
AC To
ST ES
S

:
VOLUME 5 • ISSUE 12 • DECEMBER 2008 • $8.95 • www.stpmag.com

Reeled In By
The Allure
Of Test Tools?
School Your Team On Putting
Test Automation Into Practice page 12

Teach Old Apps

New Testing Tricks
Using Quality Gates To
Prevent Automation
Project Failure
 VOLUME 5 • ISSUE 12 • DECEMBER 2008

Contents A BZ Media Publication

12
COV ER STORY
Reeled In By The Allure Of
Fancy Test Automation Tools?
Here are some real-world examples that can school your team on the ways
of putting test automation into practice.
By Aaron Cook and Mark Lustig

Teach Old Apps

18 Some New Tricks
When redesign’s not an option, and
adding testability interfaces is difficult,
you need ways to improve testability of
existing legacy apps.
By Ernst Ambichl

Depar t ments
5 • Editorial
How many software testers are there in the
world, anyway?

6 • Contributors
Get to know this month’s experts and the
best practices they preach.

7 • ST&Pedia
Industry lingo that gets you up to speed.

9 • Out of the Box

News and products for testers.

11 • Feedback
It’s your chance to tell us where to go.

32 • Best Practices
Quality Gates For the Five Phases
SCM tools are great, but they won’t run
your business or make the coffee.
By Joel Shore
27 Of Automated Software Testing
“You shall not pass,” someone might bellow, as if protecting the team
34 • Future Test
When automating Web service testing, from a dangerous peril. And so might your team if it embarks on the
there’s a right way and a wrong way. journey through the five phases of automated testing.
By Elena Petrovskaya and Sergey Verkholazov By Elfriede Dustin

DECEMBER 2008 www.stpmag.com • 3

 Ed Notes

Editor
VOLUME 5 • ISSUE 12 • DECEMBER 2008
EDITORIAL

Editorial Director
How Many
Edward J. Correia
+1-631-421-4158 x100
ecorreia@bzmedia.com

Copy Desk
Alan Zeichick
+1-650-359-4763
alan@bzmedia.com

Contributing Editors
Software Testers
Adam LoBelia
Diana Scheben
Matt Heusser
Chris McMahon
Joel Shore Are Out There?
ART & PRODUCTION What is the size of the soft- QA] are in the 1-to-3 or 1-to-
Art Director ware testing market? How 5 range.” However, the ratio
Mara Leonardi
many software testers exist in about a third of compa-
in the world? nies is around 1-to-10, he
SALES & MARKETING
I was asked that question said, which skews the aver-
Publisher
recently, and I had to admit age. “We focus on the enter-
Ted Bahr
+1-631-421-4158 x101 that I had no idea of the prise. ISVs tend to have a
ted@bzmedia.com answer. My estimate was higher saturation of testers.
about 250,000, but that was I’ve never seen an enterprise
Associate Publisher
David Karp just a guess. It was not based with a 1-to-1 ratio; maybe
+1-631-421-4158 x102 on research, statistical meas- with certain teams [working
dkarp@bzmedia.com Edward J. Correia
urement or anything at all, on] critical apps. So I would
Advertising Traffic Reprints really. It was just a number plucked from say that 1-to-5 as an industry average
Nidia Argueta Lisa Abelson thin air. would be close.”
+1-631-421-4158 x125 +1-516-379-7097 Numbers I do know for sure are these: Murphy points to what he called the
nargueta@bzmedia.com labelson@bzmedia.com
The circulation of this magazine is about flip-side of that argument. “How many lines
List Services Accounting 25,000. My e-mail newsletter goes to anoth- of code does it take to test a line of code?”
Lisa Fiske Viena Ludewig er 40,000. Our conferences (there are If, let’s say, that ratio was 10-to-1, “how
+1-631-479-2977 +1-631-421-4158 x110 three of them each year) attract another could I have three developers cranking
lfiske@bzmedia.com vludewig@bzmedia.com
few thousand software testers, QA profes- out code and expect one tester to keep up
sionals and senior test managers. Of with that?” Of course, test tools and
READER SERVICE
course, there’s some overlap between mag- automation have the potential to help the
Director of Circulation Customer Service/ azine and newsletter readers and confer- tester here, but his point is well taken;
Agnes Vanek Subscriptions
ence attendees, so let’s say BZ Media is there are not enough testers out there.
+1-631-443-4158 +1-847-763-9692
avanek@bzmedia.com stpmag@halldata.com reaching about 60,000 unique testers. Is Which perhaps helps to explain the
that a lot, or are we just scratching the sur- meteoric rise of test-only organizations
face? cropping up in India. “We’ve seen a huge
To help find the answer, I contacted growth of Indian off-shore groups where
Thomas Murphy, a research analyst with all they do is testing. Tata has spun out
Gartner. While he couldn’t provide even testing as its own entity,” he said, refer-
an approximate body count, he said that ring to the gigantic Indian conglomer-
President BZ Media LLC
sales of software testing tools this year is ate. “They wanted [the testing organiza-
Ted Bahr 7 High Street, Suite 407 expected to yield US$2.2 billion in rev- tion] to be independent, with IV&V char-
Executive Vice President Huntington, NY 11743 enue for the likes of Borland, Hewlett- acteristics.
Alan Zeichick +1-631-421-4158
fax +1-631-421-4130 Packard and other AD tool makers. That “Testing has been a growth business
www.bzmedia.com figure includes about $1.5 billion in test- in India. There’s a huge number of indi-
info@bzmedia.com
tool revenue for so-called “distributed” viduals doing manual testing, [and] build-
platforms such as Linux, Mac OS X and ing regression suites and frameworks for
Software Test & Performance (ISSN- #1548-3460) is Windows, and another $700 million for package testing.” They’ve also had to shift
published monthly by BZ Media LLC, 7 High Street,
Suite 407, Huntington, NY, 11743. Periodicals postage mainframes. their processes to adapt to changes in
paid at Huntington, NY and additional offices. As for people, Murphy estimates that technology. “As they get deeper into test-
Software Test & Performance is a registered trade- for the enterprise—that is, companies ing Web 2.0 and SOA, just to get to the
mark of BZ Media LLC. All contents copyrighted 2008
BZ Media LLC. All rights reserved. The price of a one building applications for internal use— same level of quality they used to have
year subscription is US $49.95, $69.95 in Canada,
$99.95 elsewhere. the most common ratio for developers to requires tremendous increases in their
POSTMASTER: Send changes of address to Software testers is about 5-to-1. That ratio can be own quality proactives.”
Test & Performance, PO Box 2169, Skokie, IL 60076. quite different for ISVs. “At Microsoft, for Still, the number of software testers in
Software Test & Performance Subscribers Services

begun to search. ý
may be reached at stpmag@halldata.com or by instance, the ratio is almost 1-to-1. But the world remains elusive. I’ve only just
calling 1-847-763-9692.
most companies doing a good job [of

DECEMBER 2008 www.stpmag.com • 5

 Contributors

If you’ve ever been handed a software-test automation tool and told that it will make
you more productive, you’ll want to turn to page 12 and read our lead feature. It was
written by automation experts AARON COOK and MARK LUSTIG, who themselves have
taken the bait of quick-fix offers and share their experiences for making it work.

AARON COOK is the quality assurance practice leader at

Collaborative Consulting and has been with the company
for nearly five years. He has led teams of QA engineers and
analysts at organizations ranging from startups to large multi-
nationals. Aaron has extensive experience in the design,
development, implementation and maintenance of QA proj-
ects supporting manual, automated and performance test-
ing processes, and is a member of the American Society for
Quality. Prior to joining Collaborative, he managed the test
automation and performance engineering team for a start-
up company.

MARK LUSTIG is the director of performance engineering

and quality assurance at Collaborative Consulting. In addi-
tion to being a hands-on performance engineer, he special-
izes in application and technical architecture for multi-tiered
Internet and distributed systems. Prior to joining Collaborative,
Mark was a principal consultant for CSC Consulting. Both
men are regular speakers at the Software Test & Performance
conference.

In her upcoming book titled “Implementing Automated

Software Testing,” (Addison-Wesley, Feb. 2009), ELFREDE
DUSTIN details the Automated Software Testing Process best
practices for test and QA professionals. In this third and final
installment on automated software testing, which begins on
page 27, she provides relevant excerpts from that book on
processes describing use of quality gates at each phase of a
project as a means of preventing automation failure.
Elfriede has authored or collaborated on numerous oth-
er works, including “Effective Software Testing” (Addison
Wesley, 2002), “Automated Software Testing,” (Addison Wesley,
1999) and “Quality Web Systems,” (Addison Wesley, 2001). Her latest book “The Art
of Software Security Testing,” (Symantec Press, 2006), was co-authored with security
experts Chris Wysopal, Lucas Nelson, and Dino Dai Zovi.

Once again we welcome ERNST AMBICHL, Borland’s chief

scientist, to our pages. In the March ‘08 issue, Ernst schooled
us on methods of load testing early in development to pre-
vent downstream performance problems. This time he tells
us how to make legacy and other existing applications more
testable when redesign is not an option, and adding testabil-
ity interfaces would be difficult.
Ernst served as chief technology officer at Segue Software
until 2006, when the maker of SilkTest and other QA tools
was acquired by Borland. He was responsible for the devel-
opment and architecture of Segue’s SilkPerformer and
SilkCentral product lines. For Borland, Ernst is responsible for the architecture of
Borland’s Lifecycle Quality Management products.
TO CONTACT AN AUTHOR, please send e-mail to feedback@bzmedia.com.

6 • Software Test & Performance DECEMBER 2008

 ST&Pedia
Translating the jargon of testing into plain English
The Automation Automaton
Even the experts don’t agree on the “right”
way to approach test automation. If there was
one, it would address one specific situation—
which might or might not match yours. What
we might all agree on are which problems are
the most challenging, and on some of the ways
to attack them. We'll start with two of the most
common problems of test automation:
The Minefield
Exploratory testing pioneer James Bach once
famously compared the practice of software
testing to searching for mines in a field. “If you
travel the same path through the field again
and again, you won’t find a lot of mines,” he
said, asserting that it’s actually a great way to
avoid mines.
Automated tests that repeat the same steps
over and over find bugs only when things stop
working. In some cases, as in a prepared demo
in which the user never veers from a script, that
may be exactly what you want. But if you know
that users won’t stick to the prepared test scripts,
the strategy of giving them a list of tests to repeat
may not be optimal.
The Oracle
While injecting randomness solves the mine-
field problem, it introduces another. If there
are calculations in the code, the automated
tests now need to calculate answers to deter-
mine success or failure. Consider testing soft-
ware that calculates mortgage amounts based
on a percentage, loan amount, and payment
period. Evaluating the correct output for a
given input requires another engine to cal-
culate what the answers should be. And to
make sure that answer is correct, you need
another oracle. And to evaluate that...and on
and on it goes.
RECORD/PLAYBACK
In the 1990’s, a common automation strategy
was to record a tester’s behavior and the
AUT’s responses during a manual test by use
of screen capture. Later, these recorded tests
can be played back repeatedly without the
presence of the manual tester. Unfortunately,
these recordings are fragile. They’re subject to
(false) failure due to changes in date, screen
resolution or size, icons or other minute
screen changes.
DECEMBER 2008
Matt Heusser and
Chris McMahon
Q: What would your
answers be?
Can we shrink the project sched-
ule if we use test automation?
Now that the tests are repeat-
able...are they any good?
Which heuristics are you using?
Are we doing an SVT after our
BVT? Does the performance
testing pass?
A: ST&Pedia will help
you answer questions
like these and earn the
respect you deserve.
Upcoming topics:
January
Change Management, ALM
February
Tuning SOA Performance
March
Web Perf. Management
April
Security & Vuln. Testing
May
Unit Testing
June
Build Management
Matt Heusser and Chris McMahon
are career software developers,
testers and bloggers. They’re col-
leagues at Socialtext, where they
perform testing and quality assur-
ance for the company’s Web-
based collaboration software.
KEYWORD DRIVEN
An alternative to record/playback is to isolate
user interface elements by IDs, and to exam-
ine only the specific elements mentioned.
Keyword-driven frameworks generally take
input in the form of a table, usually with three
columns, as shown in Table 1.
TABLE 1
Command Element Argument
type your_name_txt Matthew
click Button Submit
wait_for_text_present_ok Body hello, Matthew
Some versions omit the Command or verb,
where the framework assumes that every verb is
a “click” or that certain actions will always be
performed in a certain order. This approach is
often referred to as data-driven.
Where record/playback reports too many
false failures, keyword driven frameworks do not
evaluate anything more than the exact elements
they are told to inspect, and can report too many
false successes. For example, in Table 1, if the soft-
ware accidentally also included a cached last
name, and wrote “Hello, Matthew Heusser,” this
would technically constitute failure but would
evaluate as true.
MODEL DRIVEN TESTING (MDT)
Most software applications can be seen as list of
valid states and transitions between states, also
known as a finite state machines. MDT is an
approach popularized by Harry Robinson that
automates tests by understanding the possible
valid inputs, randomly selecting a choice and
value to insert. Some MDT software systems
even record the order the tests run in, so they
can be played back to recreate the error. Under
the right circumstances, this can be a powerful
approach.
AUTOMATED UNIT TESTS
Software developers commonly use Test Driven
Development (TDD) and automated unit tests
to generate some confidence that changes to
the software did not introduce a serious
“break,” or regression. Strictly, TDD is unit test-
ing typically developed in the same language of
the code, and does not have the GUI integra-
tion issue challenges associated with automat-
continued on page 33 >
www.stpmag.com • 7
Hardware Makers Get Windows 7 Pre-Beta
Attendees of the Windows Hardware
Engineering Conference (WinHEC) in
Los Angeles in early November received
a pre-beta of Windows 7, the version of
Windows that Microsoft said will replace
Vista. The software, which Microsoft
characterized in a statement as “API-
complete,” introduces a series of capabil-
ities the company says will “make it easi-
er for hardware partners to create new
experiences for Windows PC customers.”
The move is intended to “rally hard-
ware engineers to begin development and
testing” for its nascent operating system.
Windows 7 could be available as soon as
soon as the middle of next year, accord-
ing to a Nov. 7 report on download-
squad.com. Microsoft’s original promise
Zephyr 2.0: Now in the Cloud
Zephyr in late October launched version
2.0 of its namesake software test man-
agement tool, which is now available as a
SaaS in Amazon’s Elastic Compute
Cloud. Zephyr gives development teams
a Flex-based system for collaboration,
resource, document and project man-
agement, test-case creation automation
and archiving, defect tracking and
reporting. The system was previously
available only as a self-hosted system.
“The advantage of a SaaS is that you
need no hardware,” said Zephyr CEO
Samir Shah. “It’s a predeveloped back-end
up and running immediately with 24/7
availability, backup, restore, and high
bandwidth access from anywhere.” The
cost for either system is US$65 per user
per month. If you deploy in-house, you
also get three free users for the first year.
Also new in Zephyr 2.0 is Zbots, which
Shah said are automation agents for inte-
grating Zephyr with other test tools.
“These are little software agents that run
on the remote machines on which you
run Selenium, [HP QuickTestPro] or
[Borland] SilkTest.” They let you run all
your automated tests from within Zephyr,
and bring the results back into Zephyr
for analysis, reporting and communica-
tion to the team. “You build your automa-
tion scripts in those tools,” he explained,
“then within Zephyr, you get compre-
hensive views of all test cases so you can
DECEMBER 2008
for the operating system when it was intro-
duced in 2007 was that of a three-year
timetable that “will ultimately be deter-
mined by meeting the quality bar.”
Word from Microsoft is that develop-
ment of Windows 7 is on schedule. In a
Nov. 5 statement, Microsoft said touch-
sensitivity and simplified broadband con-
figuration were ready.
Helping to afford those opportunities
(Microsoft hopes) is a new component
called Devices and Printers, which report-
edly presents a combined browser for
files, devices and settings. “Devices can
be connected to the PC using USB,
Bluetooth or Wi-Fi,” said the statement,
making no of devices connected via seri-
al, parallel, SCSI, FireWire, IDE, PS/2,
look at the coverage you have across man-
ual and automated tests.”
The system also lets you schedule test
to run. “When it comes time to run auto-
mated scripts, we can kick off automation
scripts on the target machines, bring
results back to Zephyr and update met-
rics and dashboards in real time.” Shah
claims that moving test cases into Zephyr
is largely automatic, particularly if they’re
stored in Excel spreadsheets or Word doc-
uments, which is said is common. “At the
end of the day, you have to give a report
of the quality of your software. That
process is manual and laborious. We auto-
mate that by bringing all that [data] into
The test manager’s ‘desktop’ in Zephyr provides access to information about available human and tech-
nology resources, test metrics and execution schedules, defects and administrative settings.
PCI or SATA. The module also provides
Wizards.
Claiming to simplify connections to
the Internet while mobile, Microsoft has
broadened the “View Available Networks”
feature to include mobile broadband. For
makers of specialized or highly custom
devices, Microsoft provides Device Stage,
which “provides information on the
device status and runs common tasks in
a single window customized by the device
manufacturer.” Microsoft also said that
the “Start menu, Windows Taskbar and
Windows Explorer are touch-ready,” in
Windows 7. It’s unclear if application
developers can access the Windows Touch
APIs. WinHEC attendees also received a
pre-beta of Windows Server 2008 R2.
one place, from release to release, sprint
to sprint, via a live dashboard. We give the
test team and management a view into
everything you are testing.”
Also new in Zephyr 2.0 is two-way inte-
gration with the JIRA defect tracking sys-
tem. “Objects stay in JIRA, and if they’re
modified in one place they’re modified
in the other. We built a clean Web serv-
ices integration, and the live dashboard
also reflects changes in JIRA.”
Version 2.0 improves the UI, Shah said,
making test cases faster and easier to write
thanks to a full-screen mode. “When a
tester is just writing test cases, we expand
that for them and let them speed through
that process. Importing and reporting also
were improved, he said.
www.stpmag.com • 9
Feedback

SOFTWARE IS BUGGED BY BUGS

Regarding “Software is Deployed, Bugs and All,” Test & QA Report,
July 29, 2008, (http://sdtimes.com/link/3299432634):
I was very interested in this article. I will have to track down
the white paper and take a look but there are some comments
that I’d like to make based on the article. First a disclosure of my
own— used to work for a company that made tools in this space
and presently work for a distributor of IBM Rational tools.
Your “paradoxical tidbit” is I think absolutely a correct obser-
vation. It highlights something I’ve observed for some time...that
many businesses seem to think managers in the IT department
don’t need necessarily to have an IT background or they have
an IT background but [no] real management training.
The net result is they either don’t fully understand the impli-
cations of problems with the process, or they understand the prob-
lems but don’t have the management skills to address them. It’s
very rare to find a truly well managed IT department. I think what
you’ve described is really a symptom of that.
Some of the other “conclusions” seem less clearly supportable.
Statistics is a dangerous game. Once you start looking at the low-
er percentage results it is very easy to make conclusions that may
be supportable, but often there are other equally supportable con-
clusions.
For example, the data re time required to field defects... just
because it could take 20-30 days to fix a problem that isn’t neces-
sarily the worst outcome for a business. I can think of several cas-
es where this would be acceptable. To list a few...
1. The cost to the business of not releasing the product with
defects could be even greater
2. There is a work around for the defect
3. The defect is non-critical
4. There is release cycle in place that only allows fixes to be
deployed once a month.
I’m sure there are others.
The takeaways are where the real test of a survey are. From
what you’ve published of the report it seems that the second one
is supportable after a fashion. Clearly they need to fix their process
and it would seem obvious that automated solution should be a
part of that. However I do wonder if they actually got data to
support that from the survey.
The first conclusion says there are “debilitating consequences.”
Again I wonder if the survey actually established that. Clearly there
are consequences, but were they debilitating? Was there any data
about the commercial impact of the defects? Without that it is
hard to say. Yes we all know about the legendary bugs and their
consequences, but that does not automatically imply that all defects
are debilitating.
In any event, it is a topic that should be discussed more often
and I enjoyed the article.
Mark McLaughlin
Software Traction
South Australia

FEEDBACK: Letters should include the writer’s name, city, state, company
affiliation, e-mail address and daytime phone number. Send your thoughts to
feedback@bzmedia.com. Letters become the property of BZ Media and may
be edited for space and style.

DECEMBER 2008 www.stpmag.com • 11

 Beyond Tools:
Test Automation
By Aaron Cook and Mark Lustig
ne of the biggest challenges
O facing quality teams today
is the development and mainte-
nance of a viable test automation
solution. Tool vendors do a great job of
selling their products, but too often lack
a comprehensive plan for putting the
automation solution into practice.
This article will help you undertake a
real-world approach to planning, devel-
oping, and implementing your test
automation as part of the overall quality
efforts for your organization. You will learn
how to establish a test automation envi-
ronment regardless of the tools in your
testing architecture and infrastructure.
We also give you an approach to cal-
culating the ROI of implementing test
automation in your organization, an
example test automation framework
including the test case and test scenario
assembly, and rules for maintenance of
your automation suite and how to account
12 • Software Test & Performance
Once you’ve taken the bait, how
for changes to your Application Under
Test (AUT). We also cover an approach
to the overall test management of your
newly implemented test automation suite.
The set of automation tools an organ-
ization requires should fit the needs, envi-
ronments, and phases of their specific sit-
uation. Automation requires an integrat-
ed set of tools and the corresponding infra-
structure to support test development,
management, and execution. Depending
on your quality organization, this set of
automation tools and technologies can
include solutions for test management, func-
tional test automation, security and vulnera-
bility testing, and performance testing. Each
of these areas is covered in detail.
In most organizations we have seen,
phases and environments typically include
development and unit testing, system inte-
gration testing, user acceptance testing,
performance testing, security and vulner-
ability testing, and regression testing.
Depending on your specific quality
processes and software development
methodologies, automation may not be
applicable or practical for all testing phases.
Additional environments to also con-
sider may include a break-fix/firecall envi-
ronment. This environment is used to cre-
ate and test production “hot fixes” prior
to deployment to production. A training
environment may also exist. An example
of this would be for a call center, where
training sessions occur regularly. While
this is usually a version of the production
environment, its dedicated use is to sup-
port training.
Automation is most effective when
applied to a well-defined, disciplined set
of test processes. If the test processes are
not well defined, though maturing, it may
be practical and beneficial to tactically
apply test automation over time. This
would happen in parallel to the contin-
ued refinement and definition of the test
process. A key area where automation can
begin is test management.
Test Management
Mature test management solutions pro-
vide a repeatable process for gathering
DECEMBER 2008
in Practice
to make the most of your catch
and maintaining requirements, plan-
ning, developing and scheduling tests,
analyzing results, and managing defects
and issues. The core components of
test management typically include test
requirements management (and/or
integration with a requirement manage-
ment solution), and the ability to plan
and coordinate the testing along with
integration with test automation solu-
tions. Many of the test management
processes and solutions available today
also include reporting features such as
trending, time tracking, test results, proj-
ect comparisons, and auditing. Test man-
agers needs to be cognizant of integrat-
ing with software change and configura-
tion management solutions as well as
defect management and tracking.
These core components can be
employed within a single tool or can be
integrated using multiple tools. Using one
tool integrated tool can improve the qual-
ity process and provide a seamless enter-
prise-wide solution for standards, com-
munication and collaboration among dis-
DECEMBER 2008
tributed test teams by unifying multiple
processes and resources throughout the
organization.
For example, business analysts can use
the test management tool(s) (TMT) to
define and store application business
requirements and testing objectives. This
allows the test managers and test analysts
to use the TMT to define and store test
plans and test cases. The test automation
engineers can use the TMT to create and
store their automated scripts and asso-
ciated frameworks. The QA testers can
use the TMT to run both manual and
automated tests and design and run the
reports necessary to examine the test exe-
cution results as well as tracking and
trending the application defects. The
team’s program and project managers
can use the TMT to create status reports,
manage resource allocation and decide
whether an application is ready to be
released to production. Also, the TMT
can be integrated with other automation
test suite tools for functional test automa-
tion, security and vulnerability test
automation, and performance test
automation.
The test management tool allows for
a central repository to store and manage
testing work products and processes, and
to link these work products to other arti-
facts for traceability. For example, by link-
ing a business requirement to a test case
and linking a test case to a defect, the
TMT can be used to generate a trace-
ability report allowing the project team
to further analyze the root cause of iden-
tified defects. In addition, TMT allows for
standards to be maintained which increas-
es the likelihood that quality remains
high. However, facilitating all resources
to leverage this tool requires management
sponsorship and support.
Functional Test Automation
Functional test automation should be
coupled with manual testing. It is not
practical for all requirements and func-
tions to be automated. For example, ver-
ifying the contents of a printed docu-
ment will likely best be done manually.
www.stpmag.com • 13
 HOOKED ON TESTING

However, verifying the print formatting
for the document can easily be automat-
ed and allow the test engineer to focus
efforts on other critical testing tasks.
Functional test scripts are ideal for
building a regression test suite. As new
functionality is developed, the functional
script is added to the regression test suite.
By leveraging functional test automation,
the test team can easily validate function-
ality across all environments, data sets, and
select business processes. The test team
can more easily and readily document
and replicate identified defects for devel-
opers. This ensures that the development
team can replicate and resolve the defects
faster. They can run the regression
tests on upgraded and enhanced appli-
cations and environments during off hours
so that the team is more focused on new
functionality introduced during the last
build or release to QA. Functional test
l
automation can also pro-
vide support for specific
technologies (e.g., GUI,
text-based, protocol specif-
Automation is
ic), including custom con-
trols.
an investment
generation across multiple machines and
data centers/geographic locations.
Security and Vulnerability Testing
Security testing tools enable security
vulnerability detection early in the soft-
ware development life cycle, during the
development phase as well as testing
phases. Proactive security scanning
accelerates remediation, and save both
time and money when compared with
later detection.
Static source code testing scans an
application’s source code line by line to
detect vulnerabilities. Dynamic testing
tests applications at runtime in many
environments (i.e., development, accept-
ance test, and production). A mature
security and vulnerability testing process
will combine both static and dynamic test-
ing. Today’s tools will identify most vul-
nerabilities, enabling developers to pri-
oritize and address these
issues.
Test Environments
When planning the test
automation environment, a
ly and concurrently. The size and scale of
each environment also can be managed
to enable the appropriate testing while
minimizing the overall infrastructure
requirement.
The sample environment topology
below defines a potential automation
infrastructure. This includes:
• Primary test management and coor-
dination server
• Automation test centralizedcontrol
server
• Automation test results repository
and analysis server
• System and transaction monitoring
and metrics server
• Security and vulnerability test
Execution server
• Functional automation execution
server(s)
• Load and stress automation
Execution server(s)
• Sample application under test
(AUT), including:
• Web server
• Application server(s)
• Database server

that yields number of considerations Return on Investment

Performance Testing
Performance testing tools
can be used to measure
the most
load/stress capability and
predict system behavior
significant
using limited resources.
Performance testing tools
benefits
can emulate hundreds
and thousands of concur-
over time.
rent users putting the
application through rig-
orous real-life user loads. IT depart-
ments can stress an application from
end-to-end and measure response times
of key business processes. Performance
tools also collect system and compo-
nent-level performance information
through system monitors and diagnos-
tic modules.
These metrics can be combined to
analyze and allow teams to drill down to
isolate bottlenecks within the architec-
ture. Most of the commercial and open
source performance testing tools avail-
able include metrics for key technologies,
operating systems, programming lan-
guages and protocols.
They include the ability to perform
visual script recording for productivity, as
well as script based viewing and editing.
The performance test automation tools
also provide flexible load distribution to
create synthetic users and distribute load
must be addressed. These
include the breadth of
applications, the demand
for automation within the
enterprise, the types of
automation being executed
and the resource needs
to support all system
requirements. A
l key first step is tool(s) selec-
tion. Practitioners should
consider questions such as:
• Should all tools be from a single vendor?
• How well do the tools interact with the
AUT?
• How will tools from multiple vendors work
together?
That last issue—that of integration—
is key. Connecting tools effectively increas-
es quality from requirements definition
through requirements validation and
reporting.
The next step is to determine the
number of test tool environments
required. At a minimum, a production
test tools environment will be necessary
to support test execution. It is worth con-
sidering the need for a test automation
development environment as well. By hav-
ing both a test automation development
environment and production automation
environment, automation engineers can
develop and execute tests independent-
Defining the ROI of an automation
installation can be straightforward. For
example, comparisons across: time to cre-
ate a defect before and after test automation;
time to create and execute a test plan; and
time to create reports before and after test man-
agement, can all be effective measures.
Test automation is an investment that
yields the most significant benefits over
time. Automation is most effectively
applied to a well defined, disciplined test
processes. The testing life cycle is multi-
phased, and includes unit, integration,
system, performance, environment, and
user acceptance testing. Each of these
phases has different opportunities for cost
and benefit improvements. Additionally,
good test management is a key compe-
tency in realizing improvements and ben-
efits when using test automation.
There are different ways of defining
the return on investment (ROI) of test
automation. It is important to realize that
while ROI is easily defined as benefits over
costs, benefits are more accurately
defined as the benefits of automated testing
versus the benefits of manual testing over costs
of automated testing versus the costs of man-
ual testing. Simply put, there are costs asso-
ciated with test automation including soft-
ware acquisition and maintenance, train-
ing costs, automation development costs
(e.g., test script development), and

14 • Software Test & Performance DECEMBER 2008


automation maintenance costs (e.g., test
script maintenance).
One simple approach to determining
when it makes sense to automate a test
case is:
The benefits of automation are two-
fold, starting with a clear cost savings asso-
ciated with test execution. First, automated
tests are significantly faster to execute and
report on than manual tests. Secondly,
resources now have more time to dedicate
toward other testing responsibilities such
as test case development, test strategy, exe-
cutes tests under fault conditions, etc.
Additional benefits of automation include:
• Flexibility to automate standalone com-
ponents of the testing process. This may
include automating regression tests, but
also actions directly related to test man-
agement, including defect tracking, and
requirements management.
• Increased depth of testing. Automated
scripts can enable more thorough testing
by systematically testing all potential com-
binations of criteria. For example, a sin-
gle screen may be used for searching, with
results formatted differently for different
results sets returned. By automating the
execution of all possible permutations of
DECEMBER 2008
results, a system can be more thorough-
ly executed.
• Increased range of testing. Introduce
automation into areas not currently being
executed. For example, if an organiza-
tion is not currently executing standalone
component testing, this area could be
automated.
Though ROI is a prevailing tool for
conveying the value of test automation,
the less quantifiable business advantages
of testing may be even more important
than cost reduction or optimization of the
testing process itself. These advantages
include the value of increased system qual-
ity to the organization. Higher quality
yields higher user satisfaction, better
business execution, better information,
and higher operational performance.
Conversely, the negative benefits of a low
quality experience may result in decreased
user satisfaction, missed opportunities,
unexpected downtime, poor information,
missed transactions, and countless other
detriments to the company. In this sense,
we consider the missed revenue or wasted
expense of ill-performing systems and soft-
ware, not just the isolated testing process.
Test Automation Considerations
When moving toward test automation, a
number of factors must be considered.
HOOKED ON TESTING
1. What to automate. Automation can
be used for scenario (end-to-end) based
testing, component-based testing, batch
testing, and workflow processes, to
name a few. The goals of automation
differ based on the area of focus. A
given project may choose to implement
differing levels of automation as time
and priorities permit.
2. Amount of test cases to be automat-
ed, based on the overall functionality of
an application: What is the duration of
the testing life cycle for a given applica-
tion? The more test cases that are
required to be executed, the more valu-
able and beneficial the execution of test
cases becomes. Automation is more cost
effective when required for a higher
number of tests and with multiple per-
mutations of test case values. In addi-
tion, automated scripts execute at a con-
sistent pace. User testers pace varies
widely, based on the individual tester’s
experience and productivity.
3. Application release cycles: Are releas-
es daily, weekly, monthly, or annually? The
maintenance of automation scripts can
be more challenging as the frequency of
releases increases if a framework based
approach is not used. The frequency of
execution of automated regression tests
www.stpmag.com • 15
 HOOKED ON TESTING
will yield financial savings as more regres-
sion test cycles can be executed quickly,
reducing overall regression test efforts
from an employee and time standpoint.
The more automated scripts are execut-
ed, the more valuable the scripts become.
It is worth noting the dependency of a
consistent environment for test automa-
tion. To mitigate risks associated with envi-
ronment consistency, environments must
be consistently maintained.
4. System release frequency: Depending
on how often systems are released into the
environments where test automation will
be executed (e.g., acceptance testing envi-
ronment, performance testing environ-
ment, regression testing environments),
automated testing will be more effective
in minimizing the risk to releasing defects
into a production environment, lowering
the cost of resolving defects.
5. Data integrity requirement within and
across test cases: If testers create their own
data sets, data integrity may be compro-
mised if any inadvertent entry errors are
made. Automation scripts use data sets
that are proven to maintain data integri-
ty and provide repeatable expected results.
Synergistic benefits are attained with con-
sistent test data can be used to validate
individual test case functionality as well as
multi-test case functionality. For example,
one set of automated test cases provides
data entry while another set of test cases
can test reporting functionality.
6. Automation engineers staffing and
skills: Individual experience with test
automation varies widely based on expe-
rience. Training courses alone will result
in a minimum level of competency with
a specific tool. Practical experience, based
on multiple projects and lessons learned
is the ideal means to achieve a strong set
of skills, time permitting. A more prag-
matic means is to leverage the lessons
learned from industry best practices, and
the experiences of highly skilled individ-
uals in the industry.
An additional staffing consideration is
the resource permanence. Specifically, are
the automation engineers permanent or
temporary (e.g., contractors, consultants,
outsourced resources). Automation engi-
neers can easily develop test scripts fol-
lowing individual scripting styles that
can lead to costly or difficult knowledge
transfers.
7. Organizational support of test automa-
tion: Are automation tools currently owned
within the organization, and/or has budg-
et been allocated to support test automa-
tion? Budget considerations include
16 • Software Test & Performance
• Test automation software should be embedded in the team. This
• Hardware to host and execute auto- also has implications for how the automa-
mated tests tion framework will be developed.
• Resources for automation scripting and
execution, including test management. Waterfall Methodology
To estimate the development effort for
Test Automation Framework the test automation framework, the
To develop a comprehensive and exten- automation engineers require access to
sible test automation framework that is the business requirements and technical
easy to maintain, it’s extremely helpful if specifications. These begin to define the
the automation engineers understand the elements that will be incorporated into
AUT and its underlying technologies. the test automation framework. From
They should understand how the test the design specifications and business
automation tool interacts with and han- requirements, the automation team
dles UI controls, forms, and the under- begins to identify those functions that will
lying API and database calls. be used across more than one test case.
The test team also needs to under- This forms the outline of the framework.
stand and participate in the development Once these functions are identified, the
process so that the appropriate code automation engineers then begin to
drops can be incorporated into the over- develop the code required to access and
all automation effort and the targeted validate between the AUT and the test
framework development. This comes into automation tool. These functions often
play if the development organization fol- take the form of the tool interacting with
lows a traditional waterfall SDLC. If the each type of control on a particular form.
dev team follows a waterfall approach, Determining the proper approach for
then major functionality in the new code custom-built controls and the test tool
has to be accounted for in the framework. can be challenging. These functions
This can cause significant automation often take the form of setting a value (typ-
challenges. If the development team is ically provided by the business test case)
following an agile or other iterative and validating a result (from the AUT).
approach, then the automation engineers By breaking each function down into the
TABLE 1: GUTS OF THE FRAMEWORK
A business scenario defined to validate that the application will not allow the end
user to define a duplicate customer record.
Components required:
• Launch Application
LaunchSampleApp(browser, url)
• Returns message (success or failure)
• Login
Login(userID, password)
• Returns message (success or failure)
• Create new unique customer record
CreateUniqueCustomer({optional} file name, org, name, address)
• Outputs data to a file including application generated customer ID, org,
name, and address
• Returns message (success or failure)
• Query and verify new customer data
QueryCustomerRecord(file path, file name)
Returns message (success or failure)
• Create identical customer record
CreateUniqueCustomer({optional} file name, org, name, address)
• Outputs data to a file including application generated customer ID, org, name,
and address
Returns message (success or failure)
Handle successful error message(s) from application
• Gracefully log out and return application to the home page
Logout
Returns message (success or failure)
DECEMBER 2008

simplest construct, the test automation
engineer can begin to assemble them
into more complex functions easily, with-
out a great deal of code development.
For example, let’s say the AUT has a
requirement for Login. This is driven
from the business requirement. This can
also be a good example of a reusable test
component. The function Login can be
broken down into four separate functions:
SetUserID, SetPassword, SubmitForm,
and VerifyLogin. Each of these functions
can be generalized so that you end up
with a series of functions to set a value (in
this case, both the UserID and the
Password) and there is a generic submit
function and a generic validate result
function. These three functions form the
basis for the beginning of the develop-
ment of your test automation framework.
Agile Methodology
In the case of an Agile methodology,
often times the development begins by
coding a test case that will fail. Then the
developers write just enough code to
pass the test case. Once this is accom-
plished, then the code is added to the
hourly/daily build and compiled into
the release candidate.
For test automation to be successful
in this type of environment, the test
automation engineers need to be
involved upfront in the decisions of what
tests to automate and what priority to
automate them in. In this case, the team
determines the list of requirements for
the given sprint and assigns priority to
the development items. The testers in
conjunction with the rest of the team
determine which test cases will be can-
didates for this sprint and what order to
begin development. From there, the
actual development begins to follow a
similar pattern to the earlier waterfall
methodology. For example, suppose one
of the scrum team decides to develop
the AUT login function. The testers will
follow the same process to decompose
the high-level business process into
three discrete technical components,
SetValue, SubmitForm, and VerifyResult.
Table 1 shows another example. These
functions will form the basis for the
underlying test automation framework.
After each sprint is accomplished, the
scrum team performs the same evalua-
tion of functions to build and automate,
priority and resources required. From
that point forward, the automation
framework begins to grow and mature
alongside of the application under
DECEMBER 2008
FIGURE 1: TRANSPARENT SCALES
WebTableSearchByRow_v3
‘This Function searches through a webtable to locate any item in the table.
‘WebTableSearchByRow_v3 returns the row number of the correct item.
‘This function takes 6 parameters
‘ @param wtsbr_browser
‘ @param wtsbr_page
‘ @param wtsbr_webtable
‘ @param wtsbr_rowcount
‘ @param wtsbr_searchcriteria
‘ @param wtsbr_columncount
‘ return Row Number for Search Criteria
Example:
rownum = WebTableSearchByRow_v3 (br, pg, wt, rc, sc, cc)
development.
Putting the Pieces Together
When it comes time to execute your
automated tests, it is often the case that
they must be assembled into multiple
technical and business scenarios. By
taking a framework-based approach to
building the automation components,
the job of assembly is straightforward.
The business community needs to dic-
tate the actual business scenario that
requires validation. They can do this in
a number of ways. One way is to pro-
vide the scenario to the automation
engineers (in a spreadsheet, say) and
have them perform the assembly. A
more pragmatic approach is for the
business community to navigate to the
test management system that is used
for the storage and execution of the
test automation components and to
assemble the scenario directly. This
makes the job of tracking and trending
the results much easier and less prone
to interpretation.
Because maintenance is the key to all
test automation, the QA team needs to
standardize the approach and methods
for building all test automation functions.
Each function needs to have a standard
header/comment area describing the
function as well as all necessary parame-
ters. All outputs and return values need
to be standardized. This allows for greater
flexibility in the development of your
automation code by allowing the automa-
tion engineer to focus on the tool inter-
action with your AUT without having to
reinvent the standard output methods.
Any QA engineer can pick up that func-
tion or snippet of code and know what
HOOKED ON TESTING
‘ Browser as defined in the Library
‘ Page as defined in the Library
‘ WebTable as defined in the Library
‘ Total Row Count for the WebTable
‘ String value to search for
‘ Total column count for the webtable
the output message will be. For example,
all WebTableSearchByRow functions
return a row number based on the search
criteria (Figure 1).
To be successful, the automation effort
must have organizational support and be
given the same degree of commitment as
the initial development effort for the proj-
ect, and goals for test automation must
be identified and laid out up front.
Automation should have support and
involvement of the entire team and the
tool(s) should be selected and evaluated
from within your environment.
Your test automation team needs a sta-
ble and consistent environment in which
to develop components of the test
automation framework and the discrete
test cases themselves. The number and
type of tests to be automated need to be
defined (functional UI, system level API,
performance, scalability, security), and
automators need to have dedicated time
to develop the automation components,
store and version them in a source code
control system, and schedule the test runs
to coincide with stable testable software
releases.
Failure in any one of these areas can
derail the entire automation project.
Remember, test automation cannot com-
pletely replace manual testing, and not
all areas of an application are candidates
for automation. But test automation of
the parts that allow it, when properly
implemented, can enhance the ability of
your quality organization to release con-
er cost. ý
sistent, stable code in less time and at low-
www.stpmag.com • 17

TEACH
Your Old Applications
By Ernst Ambichl
n my 15 years of experience in building functional and perform-
I ance testing tools, there is one statement I’ve heard from customers
more than any other: “Changing the
application to make it more testable with
your tool is not an option for my organi-
zation.” If companies don’t change this
mindset, then test automation tools will
never deliver more than marginal
improvements to testing productivity.
Testability improvements can be
applied to existing, complex applications
to bring enhanced benefits and simplic-
ity to testers. This article will teach you
how to enhance testability in legacy appli-
cations where a complete re-design of the
application is not an option (and special
purpose testability interfaces can not eas-
ily be introduced).
You’re also learn how testability
improvements can be applied to an exist-
ing, complex (not “Hello World”) appli-
cation and the benefits gathered through
improved testability. This article covers
testability improvements from the per-
spective of functional testing as well as
from that of performance and load test-
ing, showing which changes are needed
in the application to address either per-
spective.
18 • Software Test & Performance
You’ll also understand the effort need-
ed to enhance testability with the bene-
fits gathered in terms of increased test
coverage and test automation and
decreased test maintenance, and how an
agile development process will help to
successfully implement testability
improvements in the application.
What Do I Mean by ‘Testability?’
There are many ways to interpret the
term testability, some in a general and
vague sense and others highly specific.
For the context of this article, I define
testability from a test automation per-
spective. I found the following defini-
tion, which describes testability in terms
of visibility and control, most useful:
Visibility is the ability to observe the
states, outputs, resource usage and oth-
er side effects of the software under test.
Control is the ability to apply inputs to
the software under test or place it in
specified states. Ultimately testability
means having reliable and convenient
interfaces to drive the execution and ver-
ification of tests.
Stepchild of the Architecture
The likelihood and willingness of devel-
opment teams to provide test interfaces
varies considerably.
Designing your application with testa-
bility in mind automatically leads to better
design with better modularization and less
coupling between modules. For modern
development practices like extreme pro-
gramming or test-driven development,
testability is one of the cornerstones of good
application design and a built-in practice.
Unfortunately for many existing appli-
cations, testability was not a key objective
from the start of the project. For these
applications, it often is impossible to intro-
duce new special purpose test interfaces.
Test interfaces need to be part of the
initial design and architecture of the appli-
cation. Introducing them in an existing
application can cause mature architectural
changes, which most often includes re-writ-
ing of extensive parts of the application
that no one is willing to pay for. Also test
interfaces need a layered architecture.
Introducing an additional layer for the
testability interfaces is often impossible for
monolithic applications architectures.
Testability interfaces for performance
testing need to be able to sufficiently test
the multiuser aspects of an application.
This can become complex as it usually
requires a remote-able test interface.
Thus, special purpose testability inter-
faces for performance testing are even
less likely to exist than testability inter-
faces for the functional aspects of the
application.
To provide test automation for appli-
cations that were not designed with testa-
bility in mind, existing application inter-
faces need to be used for testing purpos-
es. In most cases this means using the
graphical user interface (GUI) for func-
tional testing and a protocol interface or
a remote API interface for performance
testing. This is the approach traditional
DECEMBER 2008
functional and load testing tools are using.
Of course, there are problems using
the available application interfaces for
testing purposes, and you will find many
examples of their deficiencies, especial-
ly when reading articles on agile testing.
Some of those examples are:
• Existing application interfaces are usu-
ally not built for testing. Using them for
testing purposes may use them in a way
the existing client of the interface never
would have intended. Driving tests
through these interfaces can cause unex-
pected application behavior as well as lim-
ited visibility and control of the applica-
tion for the test tool.
Interfaces with Testability Hooks
A key issue with testability using an exist-
ing interface is being able to name and
distinguish interface controls using sta-
ble identifiers. Often it’s the absence of
stable identifiers for interface controls
that makes our life as testers so hard.
A stable identifier for a control means
that the identifier for a certain control is
always the same – between invocations of
the control as well as between different
versions of the application itself. A stable
identifier also needs to be unique in the
context of its usage, meaning that there
is not a second control with the same
identifier accessible at the same time.
This does not necessarily mean that
you need to use GUID-style identifiers
that are unique in a global context.
Identifiers for controls should be read-
able and provide meaningful names.
Naming conventions for these identifiers
will make it easier to associate the iden-
tifier to the actual control.
Using stable identifiers also avoids the
drawbacks of using the control hierarchy
for recognizing controls. Control hierar-
chies are often used if you’re using weak
control identifiers, which are not unique
in the current context of the application.
By using the hierarchy of controls, you’re
providing a “path” for how to find the

To Be More Testable
• A GUI with “custom controls” (GUI con-
trols that are not natively recognized by
the test tool) is problematic because it
provides only limited visibility and con-
trol for the test tool.
• GUI controls with weak UI object recog-
nition mechanisms (such as screen or win-
dow coordinates, UI indexes, or captions)
provide limited control, especially when
it comes to maintenance. This results in
fragile tests that break each time the appli-
cation UI changes, even minimally.
• Protocol interfaces used for perform-
ance testing are also problematic. Session
state and control information is often hid-
den in opaque data structures that are
not suitable for testing purposes. Also,
the semantic meaning of the protocol
often is undocumented.
But using existing interfaces need not
to be less efficient or effective than using
special purpose testing interfaces. Often
slight modifications to how the applica-
tion is using these interfaces can increase
the testability of an application signifi-
cantly. And again, using the existing appli-
cation interfaces for testing is often the
only option you have for an existing appli-
cations.
Ernst Ambichl is chief scientist at Borland.

DECEMBER 2008 www.stpmag.com • 19

FutureTest 2009: QA
Who Should Attend?
The program is designed for high-level test managers,
IT managers or development managers that manage the
test department or consider future test and QA strategies
to be critical to their job and their company. By attending
you will gain ideas and inspiration, be able to think freely,
away from the day-to-day demands of the office, and also exchange ideas with peers after being stimulated
by our inspirational speakers. You will leave this conference with new ideas for managing and solving your
biggest challenges both now and in the future.

Typical job titles at FutureTest include:

CIO/CTO Senior VP, IT Operations Director of Systems Programming
CSO/CISO Senior Software Test Manager VP, Software Engineering
Vice President, Development Security Director Test/QA Manager
Test Team Leader Test Architect Senior Software Architect
Vice President, Test/QA Manager of Quality Assurance
Test Director Project Manager

FutureTest is a
two-day conference • Stay abreast of trends and best practices for managing

created for senior software quality

software test and
QA managers.
FutureTest will
provide practical,
results-oriented
sessions taught by
top industry
professionals.
• Hear from the greatest minds in the software test/QA
community and gain their knowledge and wisdom
• Hear practitioners inside leading companies share their
test management and software quality assurance secrets
• All content is practical, thought-provoking,
future-thinking guidance that you can use today
• Lead your organization in building
higher-quality, more secure software

A BZ Media Event • Network with high-level peers

and the Web
1. You’ll hear great ideas to help
your company save money
with more effective Web applica-
tions testing and quality assurance
and security.
2. You’ll learn how to implement
new test & QA programs and
initiatives faster, so you save money
and realize the benefits sooner.
5 Great Reasons to
3. You’ll listen to how other
organizations have improved
their Web testing processes, so you
5. You’ll be ready to share prac-
tical, real-world knowledge
with your test and development
can adopt their ideas to your own colleagues as soon as you get
projects. back to the office.
4. You’ll engage with the newest
testing methodologies and
QA processes — including some
Add it all up, and it’s where you
want to be in February.
you may never have tried before.
REGISTER by
December 19
Just $1,095
SAVE $350!
February 24 –25, 2009
The Roosevelt Hotel
New York, NY
www.futuretest.net
 OLD DOG, NEW TRICKS
control. Relying your control recognition
on this path just increases the depend-
ency of the control recognition on other
controls and introduces a maintenance
burden when this hierarchy changes.
To improve the testability of your
application, one of the simplest and most
efficient practices is to introduce stable
control identifiers and expose them via
the existing application interfaces. This
practice not only works for functional test-
ing using the GUI but also can be adopt-
ed for performance testing using a pro-
tocol approach. How to accomplish this
for a concrete application is shown next.
Case Study: Testability Improvements
in an Enterprise Application
A major theme for the latest release
of one of our key applications that we are
developing in Borland’s R&D lab in Linz,
Austria, was to make the application ready
for enterprise-wide, global usage. This
resulted in two key requirements:
• Providing localized versions of the appli-
cation.
• Provide significantly increased per-
formance and scalability of the applica-
tion in distributed environments.
Our application is a Web based multi-
user application using a HTML/Ajax
front-end with multiple tiers built on a
Java EE infrastructure with a database
backend. The application was developed
over several years and now contains sev-
eral hundred thousand lines of code.
To increase the amount of test automa-
tion, we wanted to be able to test local-
ized versions of the application with min-
imal or no changes to the functional test
scripts we used for the English version of
the product. Also we wanted to be able
to test the scalability and performance of
existing functionality on a regular night-
ly build basis over the whole release cycle.
These regular performance tests
should ensure that we detect perform-
ance regressions as early as possible. New
functionality should be tested for per-
formance as soon as the functionality was
available, and in combination with exist-
ing functionality to ensure it was built
meeting the scalability and performance
requirements. We thought that by exe-
cuting performance tests on a regular
basis, we should be able to continuously
measure the progress we were making
toward the defined scalability and per-
formance objectives.
Performance Testing
Among the problems we had was frag-
ile performance test scripts. Increasing
22 • Software Test & Performance
FIG. 1: PARSING WORDS
Parse the dynamic control handler from a HTTP response:
WebParseDataBound(<dynamic control handle>, "control_handle=", “!” …)
The function WebParseDataBound parses the response data of a HTTP request
using “control_handle=” as the left boundary and “!” as the right boundary and
returns the result into <dynamic control handle>. You’re tools’ API will vary, of
course, but might look something like this.
Use the dynamic control handler in an HTTP request:
http://host/borland/mod_1?control_handle=<dynamic control handle>!...
FIG. 2: PARSING LOGINS
WebPage(“http://host/borland/login/”);
WebParseDataBound(hBtnLogin, "control_handle=", "!”, 6);
To call the actual “Login” button then might look like this:
WebPage(“http://host/borland/mod_1?control_handle=” + hBtnLogin + “!”);
the amount of test automation for per- URL query string parameters in the fol-
formance tests increases the importance lowing way:
of maintainable and stable performance Each request that triggers an action
test scripts. In previous releases, our per- on a control (such as pressing a button,
formance test scripts lacked stability, clicking a link, expanding a tree node)
maintainability and customizability. Often uses an opaque control handle to iden-
it happened that only small modifications tify the control on the server:
in the application broke performance
test scripts. http://host/borland/mod_1?control_handle=
68273841207300477604!...
Also was it hard to detect the reason
of script failures. Instead of searching for On the server, this control handle is
the root-cause of broken scripts, per- used to identify the actual control
formance test scripts were re-recorded instance and the requested action. As
and re-customized. Complex test cases the control handle references the actu-
were not covered (especially scenarios al instance of a control, the value of the
that change data) as they were hard to handle changes after each server request
build and had a high likelihood to break (as in this case, where a new instance
when changes were introduced in the of the same control will be created). The
application. Also customization of tests control handle is the only information
was complicated, and increased the fragili- the server exposes through its interface.
ty of the tests. This is perfectly fine for the functional-
To get a better insight on the reasons ity of the application, but is a nightmare
for the poor testability, we needed to look for every testing tool (and of course for
at the architecture of the application and testers).
the interfaces we used for testing: What does this mean for identifying
For Web-based applications, the most the actions on controls in a test tool?
common approach for performance test- It means that because of the dynam-
ing is to test at the HTTP-protocol-level. ic nature of the requests (control handles
So let’s take a look into the HTTP-pro- constantly change), a recorded, static
tocol of the application: HTTP-based test script never will work as
Our application is highly dynamic, so it calls control handles, which are not
HTTP requests and responses are dynam- valid. So the first thing testers need to do
ic. This means that they include dynam- is to replace the static control handles in
ic session information as well as dynam- the test script with the dynamic handles
ic information about the active UI con- generated by the server at runtime.
trols. While session information (as in Sophisticated performance test tools will
many other Web applications) is trans- allow you to define parsing and replace-
ported using cookies and therefore auto- ment rules that do this for you and cre-
matically handled by most HTTP based ate the needed modifications in the test
testing tools, the information about script automatically (Figure 1).
actions and controls is represented as An application of course has many
DECEMBER 2008

actionable controls on one page, which
in our case all use the same request for-
mat shown above. The only way to iden-
tify a control using its control handle is
by the order number of the control han-
dle compared to other control handles
in the response data (which is either
HTML/JavaScript or JSON in our case).
E.g.: The control handle for the
“Login” button in the “Login” page
might be the 6th control handle in the
page. A parsing statement needed to
parse the value of the handler for the
“Login” button then might look like that
is Figure 2.
It should be obvious from the small
example in Figure 2 that using the con-
trol handle to identify controls is far from
a stable recognition technique. The prob-
lems that come with this approach are:
• Poor readability: Scripts are hard to read
as they use order number information to
identify controls.
• Fragile to changes: Adding controls or
just rearranging the order of controls
will cause scripts to break or even worse
introduces unexpected behavior caus-
ing subsequent errors which are very
hard to find.
• Poor maintainability: Frequent changes
to the scripts are needed. Detecting the
changes of the order number of control
handles is cumbersome, as you have to
search through response data to find the
new order number of the control handle.
Stable Control Identifiers
Trying to optimize the recognition tech-
nique within the test scripts by adapt-
ing more intelligent parsing rules
(instead of only searching for occur-
rences of “control_handle”) proved not
to be practical. We even found it coun-
terproductive because the more unam-
biguous the parsing rules were the less
stable they became.
So we decided to address the root
cause of the problem by creating stable
identifiers for controls. Of course, this
needed changes in the application code
—but more on this later.
We introduced the notion of a control
identifier (CID), which uniquely identifies
a certain control. Then we extended the
protocol so that CIDs could easily be con-
sumed by the test tool. By using mean-
ingful names for CIDs, we made it easy
for testers to identify controls in
request/response data and test scripts.
Also was then easier for the developers
to associate CIDs with the code that imple-
ments the control.
DECEMBER 2008
CIDs are only used to provide more
context information for the test tool, the
testers, and developers. There is no
change in the application behavior – the
application still uses the control handle
and ignores the CID.
The HTTP request format changed
from:
http://host/borland/mod_1?control_handle=
68273841207300477604!...
to:
http://host/borland/mod_1?control_handle=
*tm_btn_login.68273841207300477604!...
As CIDs are included in all responses
as part of their control handlers, it is easy
to create a parsing rule the uniquely pars-
es the control handle – the search will
only return one control handle as the
CIDs are always unique in the context
of the response.
The same script as above now looks like:
WebPage(“http://host/borland/login/”);
WebParseDataBound(hBtnLogin,
"control_handle=*tm_btn_login.", “!”, 1);
WebPage(“http://host/borland/mod_1?control_
handle==*tm_btn_login.” +
hBtnLogin + “!”);
By introducing CIDs and exposing
CIDs at the HTTP protocol-level, we are
now able to build extremely reliable and
stable test scripts. Because a CID will not
change for an existing control, there is
no maintenance work related to changes
such as introducing new controls in a
page, filling the page with dynamic con-
tent that exposes links with control han-
dlers (like a list of links), or reordering
controls on a page. Also the scripts are
now more readable and the communi-
cation between testers and developers is
easier because they use the same notion
when they were talking about the con-
trols of the application.
Functional Testing
The problem here was the existence of
language-dependent test scripts. Our
existing functional test scripts heavily
relied on recognizing GUI controls and
windows using their caption.
For example, for push buttons, the
caption is the displayed name of the but-
ton. For links the caption is the displayed
text of the link. And for text fields, the
caption is the text label preceding the
text field.
To automate functional testing for all
localized versions of the application, we
needed to minimize the dependencies
OLD DOG, NEW TRICKS
between the test scripts and the different
localized versions of the application.
Captions are of course language depend-
ent, and therefore are not a good candi-
date for stable control identifiers.
The first option would have been to
localize the test scripts themselves by
externalizing the captions and providing
localized versions of the externalized cap-
tions. But still this approach introduces
a maintenance burden when captions
change or when new languages need to
be supported.
Using the caption to identify an
HTML link (HTML fragment of an
HTML page):
<A …
HREF=" http://...control_handle=*tm_btn_login
.6827..."
>Login</A>
Calling the actual “Login” link then
might look like this:
MyApp.HtmlLink(“Login”).Click();
As we already have introduced the con-
cept of stable control identifiers (CIDs)
for performance testing we wanted to
reuse these identifiers also for GUI level
testing. Using CIDs makes the test scripts
language independent without the need
to localize the scripts (at least for control
recognition – verification of control prop-
erties still may need language dependent
code). To make the CID accessible for
our functional testing tool, the HTML
controls of our application exposed a cus-
tom HTML attribute named “CID.” This
attribute is ignored by the browser but
is accessible from our functional testing
tool using the browser’s DOM.
Using the CID to identify a link:
<A …
HREF=" http://...control_handle=*tm_btn_login
.6827..."
CID="tm_btn_login"
>Login</A>
Calling the actual “Login” link using
the CID then might look like this:
MyApp.HtmlLink(“CID=tm_btn_login”).Click();
Existing Test Scripts
We had existing functional test scripts
where we needed to introduce to the
new mechanism how to identify con-
trols. So it was essential that we had sep-
arated the declaration of UI controls
and how controls are recognized from
the actual test scripts using them.
www.stpmag.com • 23
 I’m a person, just like you.

I’ll buy when and how I want.

Do not call me.

Do not make me register to

learn about you.

Don’t try to generate me.

I am not ‘actionable.’

I am Not a Lead
I WILL GET TO KNOW YOU on my own time and
in my own way. I will learn about you myself and
then I may choose to read your white papers,
attend your webinars, or visit your web site. I am
in control of the process. And if I don’t get to
know you first, you will never get my business.

Print Advertising - How Buyers Get to Know You

LEARN MORE AT I-AM-NOT-A-LEAD.COM
Based on study of SD Times readers by Readex, February 2008. How do you prefer to receive marketing information from software
tool companies 61% ads in print magazines, 40% presentations at trade shows, 38% vendor white papers, 29% direct mail, 19% banners.

Therefore we only had to change the declaration of the con-
trols but not their usage in multiple test scripts.
A script that encapsulates control information from actions
might look something like this:
// declaration of the control
BrowserChild MyApp {
…
HtmlLink Login {
tag “CID=tm_btn_login” // before: tag “Login”
…
}
} opers. The code changes needed in the application to add the
// action on the control
MyApp.Login.Click();
This approach works similarly for GUI toolkits such as Java
SWT, Java Swing/AWT, toolkits supporting the Microsoft User
Interface Automation (MSUIA) and Adobe Flex Automation. All
these GUI toolkits allow developers to add custom properties to
controls or are offering a special property that you then can use
to expose CIDs to testing tools. Of course you need to check if
your GUI testing tool is able to work with custom properties of
controls for each toolkit.
Code Changes Needed to Add Testability Hooks
One of the most enlightening experiences in this project was
how easy it was to add the testability hooks to the application
code. Speaking with the developers of the UI framework that
was used in the application and explaining them the needs to
use CIDs for recognizing controls, they immediately found out
that there was already functionality in the framework APIs that
allowed them to add custom HTML attributes to the controls
the UI framework provided.
So no changes in the UI framework code were even needed
to support CIDs for functional testing! Certainly there was work
related to creating CIDs for each UI control of the applica-
tion. But this work was done step-by-step at first introducing
CIDs for the parts of the applications, which we focused our test-
ing efforts on.
Changes in the application code to introduce a CID:
Link login = new Link("Login");
login.addAttribute("CID", "tm_btn_login"); // additional code line to
// create CID the control
More Changes For Performance Testing
For performance testing, we needed to extend the protocol of
the application to also include the CID as part of the control
handler, which is used to relate the actual control instance on
the server with the UI control in the browser. By understanding
what we needed, our UI framework developers immediately fig-
ured out how to accomplish it. Again the changes were mini-
mal and were needed in just one base class of the framework.
Changes in the UI framework of the application code to intro-
duce CIDs in control handlers:
private String generateControlHandle() {
String cid = this.getAttribute(“CID”);
if (cid != null)
return "*" + cid + "." + this.generateOldControlHandle();
else
return this.generateOldControlHandle();
}
The new version of the generateControlHandle method,
which is implemented in the base class of all UI control class-
DECEMBER 2008
OLD DOG, NEW TRICKS
es, now generates the following control handler, which contains
the CID as part of the handler:
*tm_btn_login.68273841207300477604!
Lessons Learned
Costs for enhancing testability using existing application inter-
faces were minimal. One of the most enlightening experiences
in this project was how easy it was to add the testability hooks
at the time we were able to express the problem to the devel-
testability hooks to the existing application interfaces (GUI
and protocol) were minimal.
There was no need to create special purpose testing inter-
faces, which would have caused a major rework of the applica-
tion. And the runtime overhead that was added by the testabil-
ity hooks was negligible. Most effort was related to introducing
stable identifiers (CIDs) for all relevant controls of the appli-
cation. All changes needed in the application summed up to an
effort of about 4 person weeks—this is less than 1% of the
resources working on this application per year!
For the changes and effort to make them, the benefits we
gathered were dramatic. Maintenance work for test scripts
dropped significantly. This was especially true for performance
testing, where for the first time we were able to have a main-
tainable set of tests. Before that we needed to recreate test scripts
whenever the application changed. The performance testing
scripts are now extremely stable and if the application changes
the test scripts are easy to adjust.
What’s more, the changes in the existing functional test
scripts that were needed for testing different localized versions
of the application were small. Here, it helped that we had
already separated the declaration of UI controls and how con-
trols are recognized from the actual test scripts using them.
Being able to also cover the localized versions of the applica-
tion with automated tests not only increased the test coverage
of the application, but it reduced the time we needed for pro-
viding the localized versions of the application. We are cur-
rently releasing all localized versions and the English version
at the same time.
Performance testing is now done by regularly running a
core set of benchmark tests with different workloads against
different configurations on each nightly build. So there is con-
tinuous information how the performance and the scalabili-
ty the application is improving (or degrading). We’re also now
able to detect defects that affect performance or scalability
as soon as they are introduced—which is a huge benefit as per-
formance problems are usually extremely hard to diagnose.
Knowing how performance has changed between two nightly
builds greatly reduces the time and effort needed to find the
root cause.
Moving to an agile development process was the catalyst.
When it’s so easy to improve the testability of existing appli-
ity so significantly, I wonder why it’s not done more often. ý
cations and by that increase test automation and improve qual-
REFERENCES
• Software Test Automation, Mark Fewster, Dorothy Graham, Addison Wesley, 1999
• Design for Testability, Bret Pettichord, Paper, 2002
• Lessons Learned in Software Testing, Cem Kaner et. al, John Wiley & Sons, 2002
• Agile Tester, Internet Blog: Why GUI tests fail a lot?
(From tools perspective), http://developer-in-test.blogspot.com/2008/09/
problems-with-gui-automation-testing.html
www.stpmag.com • 25
By Elfriede Dustin
mplementing a successful Automated Software Testing effort
I requires a well defined and structured, but lightweight technical
process, using minimal overhead and is
described here. It is based on proven sys-
tem and software engineering processes
and consists of 5 phases, each requiring
the “pass” of a quality gate before mov-
ing on to the next phase. By implement-
ing quality gates, we will help enforce
that quality is built into the entire imple-
mentation, thus preventing late and
expensive rework (see Figure 2).
The overall process implementation
can be verified via inspections, quality
checklists and other audit activities, each
of which is covered later in this section.
In my experience in the defense indus-
try, I have modified the Automated Testing
Lifecycle Methodology (ATLM) to adapt
to the needs of my current employer,
Innovative Defense Technologies. See
Figure 1 for an illustration of the process
that’s further defined next.
Our proposed Automated Software
Testing technical process needs to be
flexible enough to allow for ongoing iter-
ative and incremental improvement
DECEMBER 2008
feedback loops, including adjustments
to specific project needs. For example,
if test requirements and test cases already
exist for a project, Automated Software
Testing will evaluate the existing test arti-
facts for reuse, modify them as required
and mark them as “to-be-automated,”
instead of re-documenting the test
requirement and test case documenta-
tion from scratch. The goal is to reuse
existing components and artifacts and
use/modify as appropriate, whenever
possible.
The Automated Software Testing phas-
es and selected best practices need to be
adapted to each task at hand and need
to be revisited and reviewed for effec-
tiveness on an ongoing basis. An
approach for this is described here.
The very best standards and process-
es are not useful if stakeholders don’t
know about them or don’t adhere to
them. Therefore Automated Software
Testing processes and procedures are doc-
umented, communicated, enforced and
tracked. Training for the process will need
to take place.
Our process best practices span all
phases of the Automated Software Testing
life cycle. For example, in the require-
ments phases, an initial schedule is devel-
oped and maintained throughout each
phase of the Automated Software Testing
implementation (e.g. update percentage
complete to allow for program status
tracking, etc). See the section on quali-
ty gates activities related to schedules.
Weekly status updates are also an
important ingredient for successful sys-
tem program management, which spans
all phases of the development life cycle.
See our section on quality gates related
to status reporting.
Post-mortems or lessons learned play
an essential part in these efforts, and are
conducted to help avoid repeats of past
mistakes in ongoing or new development
efforts. See our section on Quality Gates
related to inspections and reviews.
By implementing quality gates and
Elfriede Dustin is currently employed by Inno-
vative Defense Technologies (IDT), a Virginia-
based software testing consulting company
specializing in automated testing.
www.stpmag.com • 27
 QUALITY GATES
related checks and balances along
Automated Software Testing, the team is
not only responsible for the final testing
automation efforts, but also to help
enforce that quality is built into the entire
Automated Software Testing life cycle
throughout. The automation team is held
responsible for defining, implementing
and verifying quality.
It is the goal of this section to provide
program management and the technical
lead a solid set of automated technical
process best practices and recommenda-
tions that will improve the quality of the
testing program, increase productivity with
respect to schedule and work performed,
and aid successful automation efforts.
Testing Phases and Milestones
Independent of the specific needs of
the application under test (AUT),
Automated Software Testing will imple-
ment a structured technical process and
approach to automation and a specific
set of phases and milestones to each
program.
Those phases consist of:
• Phase 1: Requirements Gathering—
Analyze automated testing needs and
develop high level test strategies
• Phase 2: Design & Develop Test Cases
• Phase 3: Automation framework and
test script development
• Phase 4: Automated test execution
and results reporting
• Phase 5: Program review
Our overall project approach to
accomplish automated testing for a spe-
cific effort is listed in the project mile-
stones below.
FIGURE 1: THE ATLM
28 • Software Test & Performance
Phase 1: Requirements Gathering
Phase 1 will generally begin with a kick-
off meeting. The purpose of the kick-off
meeting is to become familiar with the
AUT’s background, related testing pro-
cessing, automated testing needs, and
schedules. Any additional information
regarding the AUT will be also collect-
ed for further analysis. This phase serves
as the baseline for an effective automa-
tion program, i.e. the test requirements
will serve as a blueprint for the entire
Automated Software Testing effort.
Some of the information you gather
for each AUT might include:
• Requirements
• Test Cases
• Test Procedures
• Expected Results
• Interface Specifications
In the event that some information
is not available, the automator will work
with the customer to derive and/or devel-
op as needed. Additionally, during this
phase of automation will generally follow
this process:
1) Evaluate AUT’s current manual test-
ing process and determine
a) areas for testing technique improve-
ment
b) areas for automated testing
c) determine current quality index, as
applicable (depending on AUT state)
d) collect initial manual test timelines
and duration metrics (to be used
as a comparison baseline ROI)
e) determine the “automation index”
—i.e. determine what lends itself
to automation (see the next item)
2) Analyze existing AUT test requirements
for ability to automate
a) If program or test requirements are
not documented, the automation
effort will include documenting the
specific requirements that need to
be automated to allow for a require-
ments traceability matrix (RTM)
b) Requirements are automated based
on various criteria, such as
i) Most critical feature paths
ii) Most often reused (i.e. automat-
ing a test requirement that only
has to be run once, might not be
cost effective)
iii) Most complex area, which is
often the most error-prone
iv) Most data combinations, since
testing all permutations and com-
binations manually is time-con-
suming and often not feasible
v) Highest risk areas
vi) Test areas which are most time
consuming, for example test per-
formance data output and analy-
sis.
3) Evaluate test automation ROI of test
requirement
a) Prioritize test automation imple-
mentation based on largest ROI
b) Analyze AUT’s current life-cycle
tool use and evaluate reuse of exist-
ing tools
c) Assess and recommend any addi-
tional tool use or the required devel-
opment
d) Finalize manual test effort baseline
to be used for ROI calculation
A key technical objective is to demon-
strate a significant reduction in test time.
Therefore this phase involves a detailed
assessment of the time required to man-
ually execute and validate results. The
assessment will include measuring the actu-
al test time required for manually execut-
ing and validating the tests. Important in
the assessment is not only the time
required to execute the tests but also to
validate the results. Depending on the
nature of the application and tests, vali-
dation of results can often take significantly
longer than the time to execute the tests.
Based on this analysis, the automator
would then develop a recommendation
for testing tools and products most com-
patible with the AUT. This important step
is often overlooked. When overlooked
and tools are simply bought up front with-
out consideration for the application, the
result is less than optimum results in the
best case, and simply not being able to
use the tools in the worst case.
At this time, the automator would also
DECEMBER 2008

identify and develop additional software
as required to support automating the
testing. This software would provide inter-
faces and other utilities as required to
support any unique requirements while
maximizing the use of COTS testing tools
and products. A brief description:
• Assess existing automation framework
for component reusability
• GUI record/playback utilities compat-
ible with AUT display/GUI applications
(as applicable in rare cases)
• Library of test scripts able to interface
to AUT standard/certified simula-
tion/stimulation equipment and sce-
narios used for scenario simulation
• Library of tools to support retrieval of
test data generation
• Data repository for expect-
ed results
• Library of performance Automating
testing tools able to sup-
port/measure real time
and non-real time AUTs
inefficient test
• Test scheduler able to sup-
port distributed testing
cases will
across multiple computers
and test precedence
result in poor
The final step for this
phase will be to complete
test program
the configuration for the
application(s), including performance.
the procurement and instal-
lation of the recommended
testing tools and products along with the
additional software utilities developed.
The products of Phase 1 will typically be :
1. Report on test improvement
opportunities, as applicable
2. Automation index
3. Automated Software Testing
requirements walkthrough with
stakeholders, resulting in agree-
ment
4. Presentation report on recom-
mendations for tests to automate,
i.e. test requirements to be auto-
mated
5. Initial summary of high level test
automation approach
6. Presentation report on test tool
or in-house development needs
and associated recommendations
7. Automation utilities
8. Application configuration details
9. Summary of test environment
10. Timelines
11. Summary of current manual test-
ing level of effort (LOE) to be
used as a baseline for automated
testing ROI measurements
DECEMBER 2008
Once the list of test requirements for
automation has been agreed to by the
program, they can be entered in the
requirements management tool and/or
test management tool for documentation
and tracking purposes.
Phase 2: Manual Test Case
Development and Review
Armed with the products of phase 1,
manual test cases can now be devel-
oped. If test cases already exist, they can
simply be analyzed, mapped as applica-
ble to the automated test requirements
and reused, ultimately to be marked as
automated test cases.
It is important to note for a test to be
automated, manual test cases need to be
l automated, as computer
inputs and expectations
differ from human inputs.
As a general best practice,
before any test can be
automated, it needs to be
documented and vetted
with the customer to veri-
fy its accuracy and that the
automator’s understand-
ing of the test cases is cor-
rect. This can be accom-
plished via a test case walk-
through.
l Deriving effective test
cases is important for suc-
cessfully implementing this type of
automation. Automating inefficient test
cases will result in poor test program per-
formance.
In addition to the test procedures, oth-
er documentation, such as the interface
specifications for the software are also
needed to develop the test scripts. As
required, the automator will develop any
missing test procedures and will inspect
the software, if available, to determine
the interfaces if specifications are not
available.
Phase 2 also includes a collection and
entry of the requirements and test cases
into the test manager and/or require-
ments management tool (RM tool), as
applicable. The end result is a populated
requirements traceability matrix inside
the test manager and RM tool that links
requirements to test cases. This central
repository provides a mechanism to
organize test results by test cases and
requirements.
The test case, related test procedure,
test data input and expected results from
each test case are also collected, docu-
mented, organized and verified at this
QUALITY GATES
time. The expected results provide the
baseline to determine pass or fail status
of each test. Verification of the expected
results will include manual execution of
the test cases and validation that the
expected results were produced. In the
case where exceptions are noted, the
automator will work with the customer to
resolve the discrepancies, and as needed
update the expected results. The verifi-
cation step for the expected results
ensures that the team will be using the
correct baseline of expected results for
the software baseline under test.
Also during the manual test assess-
ment, pass/fail status as determined
through manual execution will be docu-
mented. Software trouble reports will be
documented accordingly.
The products of Phase 2 will typically be:
1. Documented manual test cases to
be automated (or existing test cas-
es modified and marked as “to be
automated”)
2. Test case walkthrough and priority
agreement
3. Test case implementation by
phase/priority and timeline
4. Populated requirements traceabil-
ity matrix
5. Any software trouble reports asso-
ciated with manual test execution
6. First draft of “Automated Software
Testing Project Strategy and
Charter” (as described in the
Project Management portion of this
document)
Phase 3 : Automated Framework
and Test Script Development
This phase will allow for analysis and
evaluation of existing frameworks and
automation artifacts. It is expected that
for each subsequent implementation,
there will be software utilities and test
scripts we’ll be able to reuse from previ-
ous tasks. During this phase we deter-
mine scripts that can be reused.
As needed, the automation framework
will be modified, and test scripts to execute
each of the test cases are developed. Scripts
will be developed for each of the test cas-
es based on test procedures for each case.
The recommended process for devel-
oping an automated test framework or
test scripts is the same as would be used
for developing a software application.
Keys to a technical approach to devel-
oping test scripts is that implementations
are based on generally accepted devel-
opment standards; no proprietary imple-
www.stpmag.com • 29
 QUALITY GATES
mentation should be allowed.
This task also includes verifying that
each test script works as expected.
The products of Automated Software
Testing Phase 3 will typically be:
1. Modified automated test frame-
work; reuse test scripts (as appli-
cable)
2. Test case automation—newly
developed test scripts
3. High-level walkthrough of auto-
mated test cases with internal or
external customer
4. Updated requirements traceabil-
ity matrix
Phase 4: Automated Test Execution
and Results Reporting
The next step is to execute automated
tests and develop the framework and
related test scripts. Pass/fail status will be
captured and recorded in the test man-
ager. An analysis and comparison of man-
ual and automated test times and
pass/fail results will be conducted and
summarized in a test presentation report.
Depending on the nature of the appli-
cation and tests, you might also complete
an analysis that characterizes the range
of performance for the application.
The products of Automated Software
Testing Phase 4 will typically be:
1. Test Report including Pass/Fail
Status by Test Case and Require-
ment (including updated RTM)
2. Test Execution Times (Manual and
Automated)—initial ROI reports
3. Test Summary Presentation
4. Automated Software Testing train-
ing, as required
Phase 5: Program Review
and Assessment
The goal of Automated Software Testing
implementations is to allow for contin-
ued improvements. During the fifth
phase, we will review the performance of
the automation program to determine
where improvements can be made.
Throughout the automation efforts,
we collect test metrics, many during the
test execution phase. It is not beneficial
to wait until the end of the automation
project to document insights gained into
how to improve specific procedures.
When needed, we will alter detailed pro-
cedures during the test program, when
it becomes apparent that such changes
are necessary to improve the efficiency
of an ongoing activity.
30 • Software Test & Performance
The test program review also includes ty gates, which apply to Automated
an assessment of whether automation Software Testing milestones.
efforts satisfy completion criteria for the Our process controls verify that the
AUT, and whether the automation effort output of one stage represented in Figure
itself has been completed. The review also 2 is fit to be used as the input to the next
could include an evaluation of progress stage. Verifying that output is satisfacto-
measurements and other metrics col- ry may be an iterative process, and veri-
lected, as required by the program. fication is accomplished by customer
The evaluation of the test metrics review meetings; internal meetings and
should examine how well original test comparing the output against defined
program time/sizing measurements com- standards and other project specific cri-
pared with the actual number of hours teria, as applicable. Additional quality
expended and test procedures developed gates activities will take place as applica-
to accomplish the automation. The ble, for example:
review of test metrics should conclude Technical interchanges and walk-
with improvement recommendations, as throughs with the customer and the
needed. automation team, represent an evalua-
Just as important is to document the tion technique that will take place dur-
activities that automation efforts per- ing and as a final step of each Automated
formed well and were done correctly in Software Testing phase. These evaluation
order to be able to repeat these success- techniques can be applied to all deliver-
ful processes. Once the project is com- ables, i.e. test requirements, test cases,
plete, proposals for corrective action will automation design and code, and other
surely be beneficial to the next project, software work products, such as test pro-
but the corrective actions, applied dur- cedures and automated test scripts. They
ing the test program, can be significant consist of a detailed examination by a per-
enough to improve the final results of the son or a group other than the author.
test program. These interchanges and walkthroughs are
Automated Software Testing efforts will intended to help find defects, detect non-
adopt, as part of its culture, an ongoing adherence to Automated Software Testing
iterative process of lessons learned activi-
ties. This approach will encourage automa- FIGURE 2: AUTOMATED
tion implementers to take the responsi- SOFTWARE TESTING PHASES,
bility to raise corrective action proposals MILESTONES AND QUALITY GATES
immediately, when such actions potentially
have significant impact on test program
performance. This promotes leadership
behavior from each test engineer.
The products of phase 5 will typically be:
1. The final report
Quality Gates
Internal controls and quality assurance
processes verify that each phase has been
completed successfully, while keeping
the customer involved. Controls include
quality gates for each phase, such as tech-
nical interchanges and walkthroughs
that include the customer, use of stan-
dards, and process measurement.
Successful completion of the activities
prescribed by the process should be the
only approved gateway to the next phase.
Those approval activities or quality gates
include technical interchanges, walk-
throughs, internal inspections, examina-
tion of constraints and associated risks,
configuration management; tracked and
monitored schedules and cost; corrective
actions; and more as this section
describes. Figure 1 reflects typical quali-
DECEMBER 2008
 QUALITY GATES

standards, test procedure issues, and oth-
er problems.
Examples of technical interchange
meetings include an overview of test
requirement documentation. When test
requirements are defined in terms that
are testable and correct, errors are pre-
vented from entering the development
pipeline, which would eventually be
reflected as possible defects in the deliv-
erable. Automation design-component
walkthroughs can be performed to ensure
that the design is consistent with defined
requirements, conforms to standards and
applicable design methodology and
errors are minimized.
Technical reviews and inspections have
proven to be the most effective form of
preventing miscommunication, allowing
for defect detection and removal.
Internal automator inspections of
deliverable work products will take place
to support the detection and removal of
defects early in the development and test
cycles; prevent the migration of defect to
later phases; improve quality and pro-
ductivity; and reduce cost, cycle time, and
maintenance efforts.
A careful examination of goals and
constraints and associated risks will take
place, which will lead to a systematic
automation strategy, will produce a pre-
dictable, higher-quality outcome and will
enable a high degree of success.
Combining a careful examination of con-
straints, as a defect prevention technolo-
gy, together with defect detection tech-
nologies will yield the best results.
Any constraint and associated risk will
be communicated to the customer and
risk mitigation strategies as necessary will
be developed.
Defined QA processes allow for con-
stant risk assessment and review. If a risk
is identified, appropriate mitigation strate-
gies can be deployed. We require ongo-
ing review of cost, schedules, processes
and implementation to prevent potential
problems to go unnoticed until too late,
instead our process assures problems are
addressed and corrected immediately.
current environment. The previous ver- the customer immediately and necessary
sion of the tool may have performed cor- adjustments made accordingly.
rectly and a new version may perform fine Tracking schedules on an ongoing
in other environments, but might adverse- basis also contributes to tracking and con-
ly affect the team’s particular environ- trolling costs.
ment. Additionally, using a configuration Costs that are tracked can be controlled.
l
management tool to base- By closely tracking sched-
line the test repository will ules and other required
help safeguard the integri- resources, the automator
ty of the automated test-
By closely tracking assures that a cost tracking
ing process and help with and controlling process is
roll-back in the event of
schedules and followed. Inspections, walk-
failure. throughs and other status
The automator incor- other required reporting will allow for a
porates the use of config- closely monitored cost con-
uration management tools resources, the trol tracking activity.
help control of the Performance is con-
integrity of the automa- automator assures tinuously tracked with
tion artifacts. For exam- necessary visibility into
ple, we will include all that cost tracking project performance,
automation framework related schedule and cost.
components, script files, and controlling The automator manager
test case and test proce- maintains the record of
dure documentation, process is followed. delivery dates (planned
schedules and cost track- vs. actual), and continu-
ing data under a config- l ously evaluates the proj-
uration management sys- ect schedule. This is main-
tem. This assures us that the latest and tained in conjunction with all project
accurate version control and records of tracking activities, is presented at week-
Automated Software Testing artifacts and ly status reports and is submitted with the
products are maintained. monthly status report.
Even with the best laid plans and
Schedules are Defined,Tracked implementation, corrective actions and
and Communicated adjustments are unavoidable. Good QA
It’s also important to define, track processes will allow for continuous eval-
and communicate project schedules. uation and adjustment of task imple-
Schedule task durations are determined mentation. If a process is too rigid, imple-
based on past historical performance mentation can be doomed to failure.
and associated best estimates. Also, any When making adjustments, it’s criti-
schedule dependencies and critical path cal to discuss any and all changes with cus-
elements should be considered up front tomers, explain why the adjustment is rec-
and incorporated into the schedule. ommended and the impact of not mak-
If the program is under a tight dead- ing the change. This communication is
line, for example, only the automation essential for customer buy-in.
tasks that can be delivered on time should QA processes should allow for and sup-
be included in the schedule. During port the implementation of necessary cor-
Phase 1, test requirements are prioritized. rective actions. This allows for strategic
This allows the most critical tasks to be course correction, schedule adjustments,
included and prioritized for completion and deviation from Automated Software
first, and less critical and lower priority Testing Phases to adjust to specific proj-
tasks to be later in the schedule. ect needs. This also allows for continuous

Experience shows that it’s important
to protect the integrity of the Automated
Software Testing processes and environ-
ment. Means of achieving this include the
testing of any new technologies in isola-
tion. This ensures that validating that tools,
for example, perform up to specifications
and marketing claims before being used
on any AUT or customer environment.
The automator also will verify that any
upgrades to a technology still run in the
After Phase 1, an initial schedule is
presented to the customer for approval.
During the technical interchanges and
walkthroughs, schedules are presented
on an ongoing basis to allow for contin-
uous schedule communication and mon-
itoring. Potential schedule risks will be
communicated well in advance and risk
mitigation strategies will be explored and
implemented, as needed. Any potential
schedule slip can be communicated to
a successful delivery. ý
process improvement and ultimately to
REFERENCES
• This process is based on the Automated Testing Lifecycle
Methodology (ATLM) described in the book Automated
Software Testing — A diagram that shows the rela-
tionship of the Automated Software Testing technical
process to the Software Development Lifecycle will
be provided here
• As used at IDT
• Implementing Automated Software Testing,
Addison Wesley Feb. 2009.

DECEMBER 2008 www.stpmag.com • 31

 Best Practices
SCM Tools Are Great,
But Won’t Mind the Store
When it comes to managing
collaborative code-develop-
ment, it is not so much the
choice of tools used, but
rather, ensuring that every-
one plays by the same set of
rules. It takes only one cow-
boy coder to bring disaster
to otherwise carefully con-
structed processes or one
rogue developer to bring a
Joel Shore
project to a standstill.
Whether you’ve implemented an
open-source tool such as CVS, Subversion,
or JEDI; or a commercial tool, experts
agree there had better be a man in a
striped shirt ready to throw a penalty flag
if short cuts are attempted.
“You can’t have a free for all when it
comes to version control; it’s essential to
have clearly defined policies for code
check in and check out,” says David
Kapfhammer, practice director for the
Quality Assurance and Testing Solutions
Organization at IT services provider
Keane. “We see companies installing
sophisticated source code and configu-
ration management systems, but what they
neglect to do is audit their processes.”
And it’s often large, well-established
corporations that should know better.
Kapfhammer says he is working with “very
mature, very big clients” in the health-
care and insurance industries that have
all the right tools and processes in place,
“but no one is auditing, no one is mak-
ing sure everyone follows the rules.”
It boils down to simple human nature.
With enormous pressure to ensure that
schedules are met (though they often are
not), it’s common for developers—in their
well-meant spirit of getting the product out
the door—to circumvent check-in policies.
A simple, low-tech way to avoid poten-
tial disasters, such as sending the wrong
version of code into production, is a sim-
ple checklist attached to developer’s cubi-
cle walls with a push pin. “This method of
managing repetitive tasks and making sure
32 • Software Test & Performance
that the details of the devel-
opment process are followed
sounds completely silly, but
it works,” says Kapfhammer.
That seems simple
enough. But the follow-up
question, of course, is “ok,
but exactly which proce-
dures are we talking about?”
That’s where the idea of a
process model comes into
play.
The current darling of process mod-
els is the Information Technology
Infrastructure Library (ITIL), a compre-
hensive set of concepts and policies
for managing development, operations,
and even infrastructure. There’s serious
weight behind ITIL—it’s actually a trade-
mark of the United Kingdom’s Office of
Government Commerce.
The official ITIL Web site, www.itil-offi-
cialsite.com (apparently they’re serious
about being THE official site), describes
ITIL as the “most widely accepted
approach to IT service management in
the world.” It also notes that ITIL aims to
provide a “cohesive set of best practices,
drawn from the public and private sec-
tors internationally.” That’s especially
important as projects pass among teams
worldwide in an effort to keep develop-
ment active 24 hours a day.
But again, it works only if everyone
plays, and only if development tasks are
granular enough to be manageable.
“If I had to boil everything I know
about source code management and ver-
sion control down to three words, it would
be this,” says Norman Guadagno, direc-
tor of product management for Micro-
soft’s Visual Studio Team System: “Every
build counts.”
Guadagno’s view of the development
world is simple: Build control and process
automation must be ingrained in the cul-
ture of the team and not something that’s
left to a manager. “If it’s not instilled in
the culture of the team, if they don’t
understand branching structures and
changes, it all becomes disconnected
from what makes great software.”
Developers, he notes, often think about
build quality as something that happens
downstream. “The thinking is ‘if I build
and it breaks, it’s no big deal.’ But that’s
not true. Do it up front and you’ll invari-
ably a deliver better product and do it
more timely.”
Branching, while necessary in an envi-
ronment where huge projects are divvied
up among many developers, is the prover-
bial double-edged sword. Handled cor-
rectly, it simplifies development and main-
tenance. But implement a model based
on bad architecture, and projects can col-
lapse under their own weight.
Guadagno recalled a small ISV with a
specialized product in use by five cus-
tomers. Their model of building a custom
version of the app for each customer and
doing that with separate branches was a
good idea, he says. “But what happens if
they grow to 50 or a hundred customers?”
Introducing new functionality eventually
will cause every build to break—and it did.
Spinning off different branches and hop-
ing for the best when it comes to cus-
tomizations simply didn’t work.
“It turns out that this was not a prob-
lem that any source code management
tool or process could solve. The problem
was the original system architecture,
which could handle the addition of a few
new customers, but which never foresaw
the impact of adding dozens.”
Guadagno’s point is that while tools,
processes, and procedures are essential
in assuring a successful outcome, poor
design will still yield poor results. “Good
good architecture.” ý
builds,” he says, “are not a substitute for
Joel Shore is a 20-year industry veteran and
has authored numerous books on personal
computing. He owns and operates Reference
Guide, a technical product reviewing and doc-
umentation consultancy in Southboro, Mass.
DECEMBER 2008
 ST&Pedia
< continued from page 7 100 page loads at the exact same time—
will take some kind of tooling to do eco-
ed acceptance testing. nomically.

AUTOMATED BUSINESS- SMOKE TESTING

FACING TESTS
A final approach is to have some
method of accessing the business logic
outside of the GUI, and to test the
business logic by itself in a standalone
form. In this way, the tests become a
sort of example of “good” operations,
or an executable specification. Two
open-source tools designed to enable
this are FIT and Fitnesse. Business-fac-
ing tests also sidestep the GUI, but
miss any rendering bugs that exist.
AUTOMATED TEST SETUP
Software can be complex and yet not
have simple save—open functionality—
for example, databases. Building import/
export hooks, and a “sample test data-
base” can save time, energy, and effort
that would have to be done over and over
again manually. Automated environment
setup is a form of test automation.
LOAD AND PERFORMANCE
TESTING
Measuring a page load—or simulating
Frequently a team’s first test automation
project, automating a build/deploy/
sanity-check project almost always pays
off. (Chris once spent a few days building
a smoke test framework that increased
efficiency 600% for months afterward.)
COMPUTER ASSISTED TESTING
Jonathan Kohl is well known for using
GUI scripting tools to populate a test
environment with a standard set-up. The
test automation brings the application
to the interesting area, where a human
being takes over and executes sapient
tests in the complex environment.
Any form of cybernetics can make a
person better, faster, and less error prone.
The one thing they cannot do is think.
There’s no shame in outsourcing required,
repetitive work to a computer, but you will
always need a human being somewhere to
drive the strategy. Or, to paraphrase James
Bach: The critical part of testing is essen-
tially a conversation between a human and
to “automate” a conversation? ý
the software. What would it even mean

Discover all the best

Index to Advertisers
software practices, Advertiser URL Page

gain an in-depth Automated QA www.testcomplete.com/stp 8

understanding of the
Empirix www.empirixfreedom.com 4
products and services
FutureTest 2009 www.futuretest.net 20, 21
of leading software
Hewlett-Packard hp.com/go/quality 36
vendors, and educate
I Am Not a Lead www.i-am-not-a-lead.com 24
yourself on a variety of
topics directly related Qualitest www.QualiTest-int.com 4

to your industry. Ranorex www.ranorex.com/stp 15

Reflective Solutions www.stresstester.net/stp 11

Learn Some SD Times Job Board

Seapine
www.sdtimes.com/jobboard

www.seapine.com/stptcm08
26

New Tricks! Software Test & Performance

Software Test & Performance

www.stpmag.com

www.stpcon.com
6, 33

35

Free White Papers at Conference

www.stpmag.com Test & QA Report www.stpmag.com/tqa 10

DECEMBER 2008 www.stpmag.com • 33

 Future
Future Test
Test
Automate Web Service
Testing the Right Way
Web services Web services are perfect
candidates for automated testing. Com-
pared with validating a UI, Web services
Web services testing is quite simple: send
a request, get a response, verify it. But
it's not as easy as it may look. The main
challenge is to identify the formal pro-
cedures that could be used for auto-
mated validation and verification. Other
problems include dynamic data and dif-
fering request formats.
In the project our team was working
on when this was written, there were
many different Web services in need of
testing. We needed to check not only the
format of the request and response, but
also the business logic, data returned by
the services and the service behavior
after significant changes
were made in the archi-
tecture. From the very
beginning we intended to
We are able to
automate as many of these
tests as possible. Some of automate about
the techniques we used for
automated verification of 70 percent of
Web services functionality
were as follows: our test cases
Response schema val-
1. idation. This is the
most simple and formal
[with a]
procedure of response small utility
structure verification. There
are free schema validators built in-house.
out there that can easily
be built into automated
scripts. However, schema validation is not
sufficient of you also need to validate the
Web services logic based on data returned
from a service.
Comparison of the actual response
2. with the expected response. This is
a perfect method for Web service regres-
sion testing because it allows for check-
ing of both response structure and data
within the response. After the successful
34 • Software Test & Performance
Elena Petrovskaya
and Sergey Verkholazov
manual validation, all valid responses are
considered correct and saved for future
reference. The response of all next ver-
sions of that Web service will then be com-
pared with these template files. There is
l
absolutely no need to write
your own script for XML
file comparison; just find
the most suitable free tool
and use it within your
scripts. However, this
method is not applicable
when dealing with dynam-
ic data or if a new version
of Web services involves
changes in the request
structure or Web services
architecture.
Check XML nodes
3. and their values. In
the case of dynamic data, it
can be useful to validate
l only the existence of nodes
using XPath query and/or check the
node values using patterns.
Validate XML response data against
4. original data source. As a “data
source” it’s possible to use database,
another service, XML files, non-XML files
and in general, everything that can be
considered a “data provider” for Web serv-
ices. In combination with response
schema validation, this method could be
the perfect solution for testing any serv-
ice with dynamic data. On the other hand,
it requires additional skills to write your
own scripts for fetching data from data
source and comparing it with service
response data.
Scenario testing. This is used when
5. the test case is not a single request
but a set of requests sent in sequence to
check Web services behavior. Usually the
Web services scenario testing includes
checking one Web service operation using
another. For example, you can check that
the CreateUser request works properly
by sending a GetUser request and vali-
dating the GetUser response. In general,
any of the above techniques can be used
for Web services scenarios. Just be care-
ful to send requests one after another
in the correct order. Also be mindful of
the uniqueness of test data during the
test. The service might not allow you to
create, for example, several users with the
same name. It’s also common for one
request in a sequence to require data
from a previous response. In this case you
need to think about storing that data and
using it on the fly.
All of these techniques were imple-
mented using a home-grown tool which
we were using in our day-to-day work.
Using that tool, we are able to automate
about 70 percent of our test cases. This
small utility that was originally created by
developers for developers to cover typi-
cal predefined test conditions and grad-
ually evolved into rather powerful tool.
Having such a tool in your own organi-
into the future. ý
zation could help your testing efforts well
Elena Petrovskaya is lead software testing
engineer and Sergey Verkholazov is chief soft-
ware engineer working for EPAM Systems in
one of its development centers in Eastern
Europe, and currently engaged in a project for
a leading provider of financial information.
DECEMBER 2008
 Te Be Th
s e
s tin of t Va
g C an lue
Attend on y
fer
en
ce
!

SPRING

March 31 – April 2, 2009

San Mateo Marriott • San Mateo, CA

Speakers 29 Minutes
3 u
Days
’r e b usy, so we
packed the
30
31+
nd-picked
for their
We know yo inimize re h a , CA is
in to 3 days to m Our speak
e rs a
d ability to San Mateo
confere nc e April 2009 xpertise an midway be
tween
ut technical e nowledge co and
your time o ate their k
Monday Tuesday
Wednesday Thursday
San Fran c is
communic
Friday

e . 30 at are
, in ways th g
ic 31
of the off 1 2 Va ll ey.
Silicon
3
Y o u r effectively g ra tin
Mark l for inte
most usefu
6 7
Calendar!
8 9
in to yo ur
dge
that knowle
daily life.

60+ Classes + C o l l eagues 6+ Tutorials

More th a n 60 classes wil
l be offered
/QA and p e rf o
,
rmance
500 ers torials offe
r a comple
te
a re te st rk with pe Full-day tu a ll o w in g
covering so
ft w
n tire applic
ation life
x p e ri e n c es! Netwo , c o ff ee rs io n in to a subjec t,
n si on
ss th e e
a given tim
e Share e ception imme mprehe
issues acro 8 classes in in st ru c to rs at our re c h g re a te r detail, co e o f th e
re a re d and mu ractic
cycle. T h e
’ll always
fi n
m e cases) p ion
means you breaks and a n d (i n so
an a 1-h o u r se ss
slot, which u r needs. m a tt e r th
that fits yo on the subject
something y.
exhibit could conve
floor.

s
31+ Exhibitor
25
$895 Value! t th e m akers of th
e latest so
ll .
ftware
M e e hibit H a
ls in our Ex
Tracks testing too nd
5+ acks The travel
and tu it io n e x p ense of att
rences.
ending STP
Con
Learn
fea tu
a
re
b
s,
o u
te
t th
st
e ir n e w
them out, a
products a
nd talk th
o built th e m .
to e

learning tr n other confe d Southwe

st experts wh
We’ll offer r is less th a
JetB e an
lu
specially fo Check out
designed e r sp ecialists t airfares.
a ge rs and othe for discoun ore you sa
ve.
m a n
u, who mu
st stay
r yo u si g n up, the m Includes:
just like yo hnologies The earl ie
R e gistratio n
e latest tec vent Pa ss p o rt
hnical class
es
on top of th h niques Your Full E ps and tec
pment tec w o rk sh o
and develo • Admissio
n to
ur comp a n y
n to keyno
te Reception
to keep yo • Ad m is si o
a ll a n d Attendee s
ve . n to Exhib it H l Showcase
compe ti ti
• Admissio h tn in g Ta lks and Too
n to Li g
• Admissio rials nch
rence mate
PRODUCED BY
fe aks, and lu
• All c o n
b re a kf a st , coffee bre
ta l
• Continen

For more information,

matiion go to www.stpcon.com
A LT E R N AT I V E T H I N K I N G A B O U T Q U A L I T Y M A N A G E M E N T S O F T WA R E :

Make Foresight 20/20.

Alternative thinking is “Pre.” Precaution. Preparation. Prevention.
Predestined to send the competition home quivering.

It’s proactively designing a way to ensure higher quality in your

applications to help you reach your business goals.

It’s understanding and locking down requirements ahead of

time—because “Well, I guess we should’ve” just doesn’t cut it.

It’s quality management software designed to remove the

uncertainties and perils of deployments and upgrades, leaving
you free to come up with the next big thing.

Technology for better business outcomes. hp.com/go/quality

©2008 Hewlett-Packard Development Company, L.P.