28

December 2014

The Magazine for Professional Testers

The Three Pillars of

Agile Quality and Testing
by Robert Galen

Testing the Internet of

Things The Future is Here
by Venkat Ramesh Atigadda

and many more

110001110001000011100110011110110000010000011110

000100110100101101110011000010111001001111001
 Agile Testing Day in Utrecht
The Agile Testing Day is a one-day conference for and by international soft-
ware testing professionals involved in AGILE WORK PROCESSES. Join this
event with your agile team to LEARN and NETWORK with your peers.

Netherlands, Utrecht March 19, 2015

Catch the
Early Bird
by January 16!

www.agiletestingday.nl
Follow us on Twitter:
@Agile_NL
Dear readers,
We are facing the end of the year 2014 and when we look back we see a great year behind us.

The magazine achieved the expectations set by the editorial team. We have increased the quality
of the articles, we have more readers worldwide, and we made some small changes to the structure
and the website. Even though we are happy with these changes, we want to improve your experience
next year and therefore we are facing new ones.

The conferences we organized ran amazingly well. The first edition of Mobile App Europe was impres-
sive with a lot of new insides on the mobile business and we have already started planning for next
year. Our main conference, Agile Testing Days, was the best ever and we are very proud of having so
many talented speakers at it. The attendees had a lot of fun and learned a lot. It was also a pleasure
to have had the exclusive release of the new book by Lisa Crispin and Janet Gregory. More Agile
Testing is a milestone in the agile world. The sister conference Agile Testing Days Netherlands that
took place earlier this year was a success and we are working hard to give you a great experience
again on March 19, 2015 in Utrecht, The Netherlands. Please visit the website www.agiletestingdays.nl.

I want to thank all the authors, sponsors, and partners for their support in issuing the magazine. A
special thank-you goes to Konstanze who laid-out the magazine for the last time!

Last, but not least, I wish you a Merry Christmas and a Happy New Year!

All the best,

Jos Daz

Testing Experience 28/2014 1

 contents 28/2014

From the Editor...............................................................................................1 The Three Pillars of Agile Quality and Testing.....................................34
by Robert Galen

Agile Is It Worth It? Advantages of Using It........................................4

by Antonio Gonzlez & Rubn Fernndez Serious Games for Testers......................................................................... 37
by Cesario Ramos & Pascal Dufour

Test-Driven Developments are Inefficient;

Behavior-Driven Developments are a Beacon of Hope? Testing the Internet of Things (IoT) the Future is Here................. 40
The StratEx Experience Part II.................................................................8 by Venkat Ramesh Atigadda
by Rudolf de Schipper & Abdelkrim Boujraf

Organize Your Testing Using Test Varieties and Coverage Types...42

A Report about Non-Agile Support for Agile........................................12 by Rik Marselis & Bert Linker
by Martin Uhlig

Reasoning the Next Version of Test Automation Myths................. 46

PERFORMANCE COLUMN: Exploratory Performance Testing.......... 14 by Sujith Shajee
by Alex Podelko

Testing Enterprise Applications: Bring-Your-Own-Crowd.............48

What Developers and Testers Need to Know about the by Philipp Benkler
ISO 27001 Information Security Standard............................................. 16
by Klaus Haller
A Unified Framework for All Automation Needs Part III................50
by Vladimir Belorusets, PhD
What Makes a Good Quality Tester?......................................................20
by Jacqueline Vermette
Book Corner.................................................................................................. 52

Ready for CAMS? The Foundation of DevOps...................................... 22

Masthead......................................................................................................C3
by Wolfgang Gottesheim

Editorial Board.............................................................................................C3
Combinatorial Testing Tools....................................................................24
by Danilo Berta
Picture Credits..............................................................................................C3

A Continuous Integration Test Framework..........................................30

by Gregory Solovey & Phil Gillis Index of Advertisers....................................................................................C3

2 Testing Experience 28/2014

 By Antonio Gonzlez & Rubn Fernndez

Agile
Is It Worth It? Advantages of Using It
Why Did Agility Appear? pivoting and iterating until they reach their expected results is almost
mandatory. Agile development allows small companies to refine their
Around 50 years ago, code was written without any plan and the ar-
products and goals on the go.
chitecture design was determined from many short-term decisions.
This worked well for small systems, but as they grew it became harder Nonetheless, Agile also works well in big companies. Multinationals are
to add new features and to fix bugs. more than ever required to move fast and adapt to the new environ-
ment. Besides, as we all know, the customer is always right. So why
After some years, methodologies were introduced in software develop-
leave them out of the development process? Agile involves the client in
ment to solve these issues. Methodologies are strict processes whose
the project, so companies can understand better and in greater detail
aim is to make software more efficient and predictable. Traditional
what the customer wants.
methodologies (for instance Waterfall) are plan-driven and require
a big effort at the beginning of the project in order to define require-
ments and architecture properly. As you may notice, these processes
Waterfall Agile
can be frustrating and not very open to changes.
Failed
Nowadays, technology and software applications evolve quickly faster Failed Successful Successful
9%
29% 14% 42%
than we expected. Therefore, time-to-market is critical in determining
whether a product will succeed or fail. Reaching the market before your
competitors might actually mean victory. Thus, it is very important
to have the right methodology that embraces and responds to the
continuous changes we are experiencing. That is the main reason why,
in 1975, practices based on iterative enhancements were introduced.
In other words, lets call it agility. Challenged
Challenged
57% 49%

What Are Agiles Main Characteristics?

Reading the Agile Manifesto (www.agilemanifesto.org) it states that
the Agile framework is based on:
Individuals and interactions over processes and tools.
Working software over comprehensive documentation.
Source: CHAOS Manifesto 2012
But these are not the only reasons why Agile is important. There are
plenty more. Below are several explanations of why it is reasonable
and suitable to use agile in software development from different
perspectives and points of view.

Customer collaboration over contract negotiation.

Responding to change over following a plan.
What this means is that the Agile framework focuses on working
software rather than a definition of strict requirements. Another pil-
lar of its philosophy is granted autonomy and shared responsibility of
all individuals within the team. It means not only thinking about the
client, but also motivating and involving programmers, analysts, and
QA engineers to work towards a common goal.
What Advantages Does Agile Have for Product
Managers?
Product managers would like to know exactly what their customers
want, but this is a hard task and unlikely to happen. Agile provides the
appropriate framework for adapting the product to the customers
actual needs. There is no need to define the product perfectly at the
beginning, but, as iterations are done, it is easy to get feedback from
clients and refine the product, implementing only those features that
provide value.

Furthermore, Agile is well-known for its transparency. Product owners

What Is It about Agile That Is so Interesting?
With this brief description, we start to observe why AGILE is so impor-
tant: response to changes. Often, new companies do not know very
well what their clients want or how to define their roadmap, hence
are always aware of what is being done and what actions are being
taken by the development team. With Agile, product owners do not
need to wait until the end of the project to know what the team has
implemented.

4 Testing Experience 28/2014

What Advantages Does Agile Have for Analysts?
Imagine you could gather data and valuable information about a
product before its final version is released. If you are a data scientist,
this may sound perfect to you. That is what Agile gives analysts
continuous information from real clients, and a real product before
it is completely implemented.
What Advantages Does Agile Have for Developers?
Developers are the core of an agile team. Therefore, it is highly im-
portant to provide the right tools and methodologies so they can do
a good job. Agile gives freedom to developers to estimate and write
code as they prefer, and motivates people to share how things are
done and work as a team.
With traditional methodologies, software engineers often feel they
mostly do work that has no meaning for the client, or that this work will
be removed from the final product. Agile focuses on doing tasks that
provide value to the customer, so wasting time and effort on writing
useless code is minimized.
Finally, in Agile there are no senior or junior levels. Everyone is a team
member, so everyones opinion is valuable. Agile helps people to share
their opinions so the whole process may benefit and improve.
What Advantages Does Agile Have for QA
Departments?
With traditional methodologies, QA engineers were left out of the
project until the product was about to be released. QA activities were
understood as a one time action at the end of the project, and this
obviously generates big risks and uncertainty about the products
Source: Gonzalo Vzquez, own illustration.
quality. In traditional methodologies, black box testing was prioritized
over user interface, and every test case and test plan had to be well
documented.
The introduction of Agile frameworks benefitted the role of QA engi-
neers, as they became more relevant. They need to be pro-active from
the start of a project by developing proper measures to assure the
quality of the product. In general, the QA role in Agile should:
Help Business Analysts to define stories and their acceptance
criteria so they to know whether they are satisfying customer
requirements.
Integrate with the development team to assess the adoption of
code standards and the improvement of the code base through
refactoring.
Provide developers with high level test cases and scenarios for the
stories before coding. Respond to change-over following a plan.
Ensure black box testing over the user interface and white box
testing to gain knowledge of the internal workings of the applica-
tion.
Increase automated testing, so the teams speed will increase.
Introduce quality control into every iteration.
What Advantages Does Agile Have for
Management? Are There Any Managers on Agile?
The Agile Manager is responsible for supporting the development
team, clearing any blockages, and keeping the Agile process consistent.
The Agile Manager is the development progress facilitator, and his/
her main task is to maximize the teams effectiveness rather than
Testing Experience 28/2014 5
 controlling how they work, which is more usual in traditional project
management where project managers act as authority figures. In Agile
environments there is no authority role.
The Agile Manager ensures that the right people are on the project,
mentoring, training, guiding, and motivating everyone to reach a
common goal. This important task helps the team to feel ownership
of the project and gives them great motivation to achieve their goals.
These benefits imply a transition to Agile, which is not an easy task.
Teams and companies that do not fully believe in the adoption of
Agile will give up their adoption of it as soon as the first problems are
encountered. A good agile team understands the benefits of Agile
and believes in its adoption, choosing the management and technical
methods provided by the Agile framework that work best for them. If
this occurs, the adoption of Agile is likely to succeed.

Acknowledgements to Gonzalo Vzquez (www.gonvazquez.com)

Then, Is Agile Perfect? for his wonderful illustration.

Of course not. The first thing to note is that Agile is not about perfection,
it is about bringing value to your organization and to your customers in
the most cost-effective and transparent way. For this reason, you need
> about the authors
to be patient when adopting Agile frameworks. It is very important
that the executive team believes in Agile methods. If it happens, your
Antonio Gonzlez Sanchis is a telecommunica-
probability of project success increases considerably.
tions engineer with a Masters degree in Project
Moreover, Agile has several disadvantages. As it focuses strongly on Management and a Diploma in Business and
people, if someone leaves your team, it is likely you will be losing a lot Management from the London School of Eco-
of information and cohesion from the group. Besides, as the team is nomics. He has several years of experience as a
self-managed, maturity is required in all team members and this is software developer and business analyst for
often hard to obtain. companies such as Fraunhofer Institute, Hewlett-
Packard, and Amadeus, always working under agile methodologies.
Finally, there are common mistakes in adopting Agile that cause more
His interests are strongly related to applying agile frameworks to
inconveniences than advantages. One common mistake is trying to
non-related software environments.
adapt your organization to Agile. Agile is a framework that provides
LinkedIn: www.linkedin.com/in/antoniogonzalezsanchis
methods for being more productive, so the trick is to determine which
Website: www.openbmic.com
methods provided by Agile are suitable for your organization and to
tune your methodology to fit your individual or team needs. Another
Rubn Fernndez lvarez has seven years of
error is believing that Agile means that everything can be done at any
experience as a QA test engineer in medical
time. Agile is about flexibility, but if development of a feature has
and industrial environments in different com-
started, it is not possible to change it before it is finished.
panies, including Sogeti, Ingenico, Grifols, and
Aurigae Telefonica R+D. He presently holds the
role of SW QA Manager at Zitro Interactive, where
Does It Only Work in Software Environments?
he is responsible for test management and test
No, it could work in different environments. For example, think of this automation in mobile and web gaming projects. He is a qualified
article that we wrote in a kind of agile way. We first defined what we telecommunications and electronics engineer and is a certified
wanted to do (high level) and we started one week iterations of writing. Scrum Manager.
After every iteration we asked for feedback from experts and people Twitter: @rbnfdez
whom we considered to be important stakeholders. Using this infor- LinkedIn: www.linkedin.com/in/rbnfdez
mation, we improved our article and kept on writing. When we had
completed ten iterations, our stakeholders told us it was good enough
for them, so we decided to close the article. Obviously we were not 100%
agile as we did not release our article before it was finally completed.

As said, Agile does not only work for software projects, but it is true
that it might work better with small budget projects, as refining your
product every iteration may incur unexpected expenses. Moreover,
Agile is very effective in environments where changes in the require-
ments happen quite often due to various business reasons.

Conclusion
Agile is not a one-time aspect in the development process of a company.
It is a development philosophy that helps to deliver frequent releases
with high quality to final customers, through team collaboration,
transparency, and continuous improvement.

6 Testing Experience 28/2014

CABA Certified
Agile Business Analyst
Are you a business analyst who is working in or moving
to an Agile environment? How does the work, and your
approach, differ from the traditional BA role? What are
the recognised practices in Agile business analysis? The
CABA course presents practices and tools for Agile BAs
working in any of the emergent flavours of Agile.

For more information, visit caba.diazhilterscheid.com

Open courses:
or contact us at info@diazhilterscheid.com.

March 1920, 2015 Berlin, Germany Our training courses are available as inhouse courses
June 1112, 2015 Berlin, Germany and outside of Germany on demand.

Daz & Hilterscheid Unternehmensberatung GmbH Phone: +49 (0)30 74 76 28-0

Kurfrstendamm 179 Fax: +49 (0)30 74 76 28-99
10707 Berlin Email: info@diazhilterscheid.com
Germany Website: caba.diazhilterscheid.com
Testing Experience 28/2014 7
Missed Part I?
Read it in
issue No. 27!

By Rudolf de Schipper & Abdelkrim Boujraf

Test-Driven Developments are Inefficient;

Behavior-Driven Developments are a Beacon of Hope?
The StratEx Experience (A Public-Private SaaS and On-Premises Application) Part II

In part 1 of this article, we outlined a number of criticisms and road- software. This is our target audience for this article. We are not pre-
blocks that we have encountered while looking for the best way to tending to describe how to test military-spec applications or embedded
test our application. systems, for example.

We know it is easy to criticize, but this was not for the sake of being
negative. We believe that our experience has led to some valuable
insights, apart from the points we simply do not like. Further, we
believe our experiences are not unique. So, in this second article we
want to take a look at what can be done to test our software efficiently.
What we describe here is not all implemented today. As we said in the
previous article, this is part of a journey, a search for the optimum way
of doing things. You might wonder why? There are a couple of reasons
for this. First and foremost, as a small startup company, our resources
are limited. And, as Google says, scarcity brings clarity[1], so we have
to be careful what we spend our time and energy on. Second, when do-
ing our research on testing methodologies and how to apply these best,
there was one recurring theme: there is never enough time, money, or
resources to test. This can probably be translated into management
does not give me enough, because they apparently believe that what
they spend on testing is already enough. Now here comes the big ques-
tion: what if management is right? Can we honestly say that every test,
every check, every test department even, is super-efficient? We may
argue that test effort may reach up to x% of development effort (30%
has been given as a rough guideline). Well then, if by magic, develop-
ment effort is sliced down to one fifth, is it not logical to assume that
the test effort should be reduced by the same factor? And how would
this be achieved? We want to explore this here.
What do we want to achieve with our tests? In simple terms, we are
not looking for mathematical correctness proofs of our code. Nor are
we looking for exhaustively tested use cases. We want to be reasonably
certain that the code we deploy is stable and behaves as expected. We
are ready to accept the odd outlier case that gives an issue. We believe
our bug-fixing process is quick enough to address such issues in an
acceptable timeframe (between 4 and 48 hours).
Lets look a bit closer at the various types of tests we might need to
devise to achieve such reasonable assurance.
Testing a CRUD Application
The application presents a number of screens to the user, starting
from the home screen, with a menu. The menu gives access to screens,
mostly CRUD-type, while some screens are process-oriented or serve a
specific purpose (a screen to run reporting, for example).
A CRUD screen has five distinct, physical views:
1. The items list (index), i.e., the Contracts list
2. The item details, i.e., the Work Page details
3. The item edition, i.e., edit the Activities details

This is where we came from. We generate large parts of our code. This 4. The item creation, i.e., create a new Type of Risk
reduces development time dramatically. For a small startup this is a
5. And the item deleting, i.e., delete a Project and its related items
good thing. But this also means that we must be able to reduce our
testing time. And that was the reason we had a good and hard look at
Possible actions on each screen are standardized, with the exception
current testing methods, what to test, when to test, and how to test.
of the details screen, where specific, business-oriented actions can
be added. You may think of an action such as closing a record, which
involves a number of things such as checking validity, status change,
Testing a Web Application Deployable as Public-
updating log information, and maybe creating some other record in
Private Cloud and On-Premises Software
the system. In essence, these actions are always non-trivial.
First, lets frame our discussion. The application we are developing is
rather standard from a testing point of view: web-based, multi-tier,
a back-end database, and running in the cloud. The GUI is moderately Testing a Generated vs. a Hand-Coded Piece of
sprinkled with JavaScript (JQuery[2] and proprietary scripts from a set Software
of commercial off-the-shelf (COTS) UI controls like DevExpress[3]and
All CRUD screens are made of fully generated code, with the exception
Aspose[4]). The main way to interact is through a series of CRUD[5]
of the business actions, which are always hand-coded.
screens. We can safely say that there are probably thousands of ap-
plications like this, except that the same piece of code is deployable as The non-CRUD screens are not generated and always hand-coded.
a Private Cloud and Public Cloud application, as well as on-premises Needless to say we try to keep the number of these screens low.

8 Testing Experience 28/2014

We have observed that the generated code and screens are usually
of acceptable initial quality. This is because the number of human/
manual activities to produce such a screen is very low. The code tem-
plates that are used by the generator obviously have taken their time
to be developed. This was however a localized effort, because we could
concentrate on one specific use case. Once it worked and had been
tested (manually!), we could replicate this with minimal effort to the
other screens (through generation). We knew in advance that all the
features we had developed would work on the other screens as well.
An interesting side-effect of this method is that if there is an error in
the generated code, the probability of finding this error is actually very
high, because the code generation process multiplies the error to all
screens, meaning it is likely to be found very quickly.
The hand-code screens are on the other side of the scale. They present
a high likelihood of errors, and we have also found that these screens
are prone to non-standard look and feel and non-standard behavior
within the application. When compared to the approach of generating
highly standardized code, the reasons for this are obvious.
Testing Business Actions
The business actions are the third main concern for testing. These
are non-trivial actions, highly context (data, user) dependent, and
with a multitude of possible outcomes. We have not yet figured out
how to test these artifacts automatically due to the amount of cases
we would need to take into account. Each change in our logic needs a
complete refactoring of those tests that will certainly produce most
of the complaints from of our beloved customers.
Testing the User Interface Using BDD
A final concern is the UI testing. Even with highly standardized screens,
we value this type of testing, for three reasons:
First it is the way to run a partial end-to-end test[6] on your
screens as we test partially the content of the database after a
screen is tested.
Second, it is what the user actually sees. If something is wrong
there, there is no escape.
Third, we like to use such test runs to record documentation,
and demo and training videos using Castro[7], Selenium[8], and
Behave[9] mostly open source software.
We believe that this context is relatively common, with the possible
exception of massive code generation and the use of tests to docu-
ment the application (and we would recommend these as something
to consider for your next project!), so it makes sense to examine how
these different areas can be tested efficiently.
For the generated CRUD screens, tests should be standardized and
generated. Do we need to test all the generated screens? Given the fact
that the tests are generated (so there is no effort involved in creating
the tests), we would say that you must at least have the possibility of
testing all screens. Whether you test them for every deployment is a
question of judgment.
Hand-coded screens, if you have them, probably require hand-coded
tests. Yet, if you have information available that allows you to gener-
ate (parts of) your tests, do it. It reduces the maintenance cycle of your
tests, which means you improve the long-term chances of survival of
your tests. We have not found convincing evidence to state that hand-
coded (non-standardized) screens can be fully described (see Table 1)
by a set of BDD/Gherkin tests or briefly described (see Table 2). The
simple fact is that it would require a large amount of BDD-like tests to
fully describe such screens. One practice we have observed is to have
one big test for a complete screen; however, we found that such tests
quickly become complex and difficult to maintain for many reasons:
1. You do not want to disclose too much technical information to
the business user, e.g., username/password, the acceptable data
types, the URL of the servers supporting the factory acceptance
test (FAT), system acceptance tests[10] (SAT) and the live applica-
tion.
2. You need to multiply the number of features by the number of
languages your system supports.
3. You want to keep the BDD test separate from the code that the
tester writes, as the tests depend on the software architecture
(Cloud, on-premises) and the device that may support the ap-
plication (desktop, mobile).
4. A database containing acceptable data might be used by either
the business user or the tester. The data might be digested by the
system testing the application and reduce the complexity of the
BDD-tests while increasing the source code to test the applica-
tion
1. # file: ./Create_Contract_Request_for_offer.feature
2. Feature: Create a Request for offer
3. As a registered user,
4. I want to create a Request for offer for a project
5. Background:
6. Given I open StratEx "<url>"
7. When I sign up as "<username>"
8. Then I should be signed in as "<user_first_last_name>"
9. Scenario Outline:
10. Then I click on "Contract" menu item
11. Then I click on "Request for offer" menu item
12. Then I click on "Create new" menu item
13. Then I select "<project_name>" from the "Project"
dropdown
14. Then I set the "Title" box with "<project_title>"
15. Then I click on "Save" button
16. Then I check that the "Project" field equals
"<project_name>"
17. Then I check that the "Title" field equals
"<project_title>"
18. And I click on the link "Logout"
19. Examples: staging
20. | url| username
| user_first_last_name| project_name | project_
title |
21. | https://staging.<your application>.com | a Username
| a firstname, a lastname | a project name | a project
title |
Table 1. BDD Definition (Full Description): Create a Request for Offer 11_Create_Contract_
Request_ for_offer.feature
Testing Experience 28/2014 9

1. # file: ./Create_Contract_Request_for_offer.feature
2. Feature: Create a Request for offer
3. As a registered user,
4. I want to create a Request for offer for a project
5. Background:
6. Given I open StratEx "<url>"
7. When I sign up as "<username>"
8. Then I should be signed in as "<user_first_last_name>"
9. Scenario Outline:
10. Then I create one Request for offer
11. And I click on the link "Logout"
12. Examples: staging
13. | url| username|
user_first_last_name|
14. | https://staging.<your application>.com | a Username |
a firstname, a lastname |
Table 2. BDD Definition (Brief Description): Create a Request for Offer 11_Create_
Contract_Request_ for_offer.feature
1. @then(u'I create one Request for offer')
2. def step_impl(context):
3. # click | 'Contract' menu item
4. context.browser.find_element_by_xpath(
"//a[contains(text(),'Contract')]").click()
5. context.browser.dramatic_pause(seconds=1)
6. # click | 'Request for Offer' menu item
7. context.browser.find_element_by_xpath(
"//a[contains(text(),'Request for Offer')]").click()
8. context.browser.dramatic_pause(seconds=1)
9. # click | 'Create New' menu item
10. context.browser.find_element_by_xpath(
"//a[contains(text(),'Create New')]").click()
11. context.browser.dramatic_pause(seconds=2)
12. # select | id=Project | label=StratEx Demo
13. Select(context.browser.find_element_by_id("Project")).
select_by_visible_text(context.testData.find(
".//project_name").text)
14. context.browser.dramatic_pause(seconds=1)
15. # type | id=Title | Horizon 2020 dedicated SME
Instrument - Phase 2 2014
16. context.browser.find_element_by_id("Title").clear()
Table 3. Excerpt from 11_Create_Contract_Request_ for_offer.py
At StratEx, we our current practice is to write brief BDD, after many
attempts to find a right balance between writing code and BDD-tests.
We did choose the Python Programming language (see Table 3) to
implement the tests, because Python[11] is readable even by business
users and can be deployed on all our diverse systems made of Linux
and Windows machines.
Business actions are hand-coded too, but such actions are good can-
didates for BDD-tests, described in Gherkin. As we mentioned before,
Gherkin is powerful for describing functional behavior, and this is
exactly what the business actions implement. So there seems to be a
natural fit between business actions and BDD/Gherkin. The context
10 Testing Experience 28/2014
can usually be described in the same way as for UI tests (see above).
Can such tests be generated? We believe that the effort for this might
outweigh the benefits. Still, using Gherkin to describe the intended
business action and then implementing tests for it seems like a prom-
ising approach.
Conclusion
This broadly covers the area of functional testing, including UI tests. The
question obviously occurs as to what else needs to be tested, because
it is clear that the tests we describe here are not ideal candidates for
development-level (unit) tests they would simply be too slow. In vari-
ous configurations, the tests we described above would run in a pre-
deployment scenario, with more or less coverage: run all screens, run
only the hand-coded screen tests, run some actions as smoke tests, etc.
We believe that the most relevant tests for the development cycle are
the ones related to the work the developer does, i.e., producing code.
This means that generated code can be excluded in principle (although
there is nothing against generating such tests). It focuses therefore on
hand-coded screens and business action implementation.
Starting with the business action implementation, we observe that this
only requires coding in non-UI parts of the application: the model[12]
and the database. It has been shown that it is possible to run unit-like
tests against the model code and against the controller code. Unit tests
against the model can be used to validate the individual functions (as
in normal unit tests), while tests against the controller will actually
validate that the action when invoked by the user from the UI will
yield the expected result. In that sense, this kind of test runs like a UI
test without the overhead (and performance penalty) of a browser.
What is so special about this approach? First, these are not real unit
tests because they do not isolate components. They test an architectural
layer, including the layers below it. This means that, when testing a
model, the database will be accessed as well. This is a deliberate trad-
eoff between the work required to make components unit-testable
and the impact of testing them in context. It means we have to
consider issues such as database content and we need to accept that
these tests run slower than real unit tests. However, because we have
far fewer of this type of test (we only implement the tests for the busi-
ness actions, which is between two and ten per screen), the number
of these tests will be around 100200 for the complete application.
We believe that this is a workable situation, as it allows developing
without having to consider the intricacies of emulated data, such as
mocks or other artificial architectural artifacts, to allow for out-of-
context testing. In other words, we can concentrate on the business
problems we need to solve.
An additional benefit here is that this allows us to test the database
along with the code. Database testing is an area we have not seen
covered often, for reasons that somewhat elude us.
In summary, we have presented here a method for efficiently testing
large parts of web-based software by using elements of code gen-
eration to generate automatable tests, and by using BDD concepts to
model tests for non-generated screens and non-generated business
actions. Further, we have described a method for context-based unit
testing that, when combined with generated code and tests, yields
an acceptable trade-off between development efficiency and time
spent on testing.
> about the authors

This article has not covered other areas of testing, such as performance
and security tests. Currently StratEx has no immediate concerns in
these areas that required us to critically observe how we validate the
application in this respect. ing large multinational projects and likes work-
References
[1] How Google tests software (Whittaker, Arbon et al.)
[2] jQuery: http://jquery.com
[3] DevExpress: https://www.devexpress.com/
[4] Aspose: http://www.aspose.com
[5] Create, Read, Update, Delete
[6] End-to-end test:
http://www.techopedia.com/definition/7035/end-to-end-test
[7] Castro is a library for recording automated screencasts via a
simple API: https://pypi.python.org/pypi/castro
[8] Automating web applications for testing purposes:
http://www.seleniumhq.org
[9] Behave is behavior-driven development, Python style:
https://pypi.python.org/pypi/behave
[10] System Acceptance Test:
https://computing.cells.es/services/controls/sat
[11] Python Programming Language: https://www.python.org/
[12] We assume an MVC or similar architecture is being used here,
or anything that clearly separates the UI from the rest of the
system.
Rudolf de Schipper has extensive experience in
project management, consulting, QA, and soft-
ware development. He has experience in manag-
ing in a team. Rudolf has a strong analytical
attitude, with interest in domains such as the
public sector, finance and e-business.
He has used object-oriented techniques for design and develop-
ment in an international context. Apart from the management
aspects of IT-related projects, his interests span program manage-
ment, quality management, and business consulting, as well as
architecture and development. Keeping abreast with technical
work, Rudolf has worked with the StratEx team, developing the
StratEx application (www.stratexapp.com), its architecture, and the
code generation tool that is used. In the process, he has learned
and experienced many of the difficulties related to serious soft-
ware design and development, including the challenges of testing.
LinkedIn: be.linkedin.com/pub/rudolf-de-schipper/3/6a9/6a9
Abdelkrim Boujraf owns companies developing
software like StratEx (program and project man-
agement) and SIMOGGA (lean operations man-
agement). He has more than 15 years of experi-
ence in different IT positions, from software
engineering to program management in man-
agement consulting and software manufactur-
ing firms. His fields of expertise are operational excellence and col-
laborative consumption. Abdelkrim holds an MBA as well as a
masters in IT & Human Sciences. He is a scientific advisor for the
Universit Libre de Bruxelles where he is expected to provide occa-
sional assistance in teaching or scientific research for spin-offs.
LinkedIn: be.linkedin.com/in/abdelkrimboujraf

Referenced books
The Cucumber Book (Wynne and Hellesoy)

Application testing with Capybara (Robbins)

Beautiful testing (Robbins and Riley)

Experiences of Test Automation (Graham and Fewster)

How Google tests software (Whittaker, Arbon et al.)

Selenium Testing Tools Cookbook (Gundecha)

Referenced articles
Model Driven Software engineering (Brambilla et al.)

Continuous Delivery (Humble and Farley)

Domain Specific Languages (Fowler)

Domain Specific Modeling (Kelly et al)

Language Implementation Patterns (Parr)

Testing Experience 28/2014 11

 By Martin Uhlig

A Report about Non-Agile Support for Agile

Starting Situation
Certainly some developers and testers in the agile environment are
familiar with the following situation. A team had been working on a
product for a long time, but they did not have a dedicated tester. As a
result, quality requirements had rather been neglected. But now just
before the product release everything should get better and the team
would be supported by additional testers.
At the beginning of this year, I became involved in a similar situation
when I was assigned as a tester to a Scrum team that had suffered
from a high turnover of testers. Apart from the unit tests that the
developers had established, there were no automated tests. But as
the product release moved closer, the team had to deal with various
short-term decisions regarding product changes. We had to find a way
to test the products features as well as its non-functional criteria in
a fast and reliable manner.
To start the automation of integration tests and GUI tests during this
stage of the project would have been fatuous. We needed a manual
solution.
test sets had been worked out and iteratively improved by the tester
in the Scrum team (supported by the whole Scrum team). The work
of the QA teams was to be strictly separated from the Scrum team to
avoid lowering the performance of the Scrum team. Therefore the
teams needed an interface to filter the exchange of information in the
direction of the Scrum team, thus avoiding an information overflow
(esp. only actual bugs, no duplicates, etc.). The Scrum team itself would
focus on reproducing and troubleshooting the bugs. We decided to
staff the interface between the teams with the PO and the tester from
the Scrum team. The only exception to the team seperation was the
Daily Scrum meetings. Besides the Scrum team, one agent from each
QA team attended to give their current status.
This plan was refined so the teams were better balanced. One of the
Scrum teams developers moved to a QA team to test with them. Thus,
every QA team had three members, including one experienced tester
who headed the execution of the tests. Furthermore, the Scrum team
had two relatively new members who were not sufficently trained
to do quick and effective bugfixing of this complex software. These
colleagues were given the mission of performing exploratory testing
apart from the test sets and reproducing bugs.

In summary, our final setup was with two QA teams executing a set
The Idea
of tests. These tests included all the positive and the most important
In collaboration with our Product Owner (PO), we created a concept to negative test cases. Every team worked with a different product con-
organise a pure quality assurance sprint (QA Sprint) only for testing, fix- figuration. The QA teams tests were supplemented by explorative tests
ing, and retesting. No feature stories had been planned for this sprint. from the two fresh developers in the Scrum team. Six developers within
the Scrum team took care of bugfixing and deployment. This way, two
But how could we test the whole product in such a short time span?
teams were established and they supported the Scrum team without
It is not possible to perform the test scope needed for a convincing
any signifcant negative effects for the self-organization of the Scrum
and informative result in a two-week iteration not even with all
team (shown in Figure 1). The atypical ratio between developers and
the nine team members. After all, it was necessary to cover different
testers in favour of developers was less problematic because the PO
configurations of the software with tests. But we were very lucky to
had enough old known issues to be fixed in her backlog, keeping the
receive special help from five additional testers and developers who
developers busy till the testers reported the first bugs.
had agreed to support our tests. So we had enough workforce. But
how to manage all these people?

From the outset it was obvious that the team could not simply be
extended. There is no way of conducting an effective Sprint Planning
or Daily Scrum with 14 team members (plus Scrum Master) and the
whole team would have had to reorganise itself. This was no practicable
solution, not even for only one sprint. As a consequence we had to find
another way to integrate the additional testers.

But what approach would work? The answer seems simple we needed
more teams! But a new Scrum team cannot simply be conjured out of
Bugs, issues, etc.
thin air, especially for just one sprint. So we had to distinguish between
the Scrum team and the QA teams. The Scrum team, consisting of the
former team, should basically work as usual in the best Scrum manner.
But to strengthen the Scrum team we needed the additional testers,
but could not get them aboard the Scrum boat due to the reasons
mentioned above. And so we had to create two new teams that were
substantially self-organized but not a fixed part of the Scrum team.
Figure 1. Two small QA teams supporting the Scrum team. The communication between
These QA teams focused on repeatedly running a given set of tests. The
the teams is handled by fixed interfaces and agents (blue).

12 Testing Experience 28/2014

The Execution
The QA sprint started just like any other sprint for the Scrum team,
except that the Sprint Planning was shorter than usual. Only a few
known issues were presented by the PO in Sprint Planning. During the
sprint the PO and the Scrum teams tester evaluated the bugs found
by the QA teams and added them to the Scrum teams sprint backlog
in a prioritized manner.
Besides the Scrum teams Sprint Planning, there was a kick-off meeting
for the QA teams. They were given the instruction to run the positive
and negative test cases from the committed test sets and to document
the bugs. After the execution of the whole test set (duration approx. 2
days) it was reviewed and improved by the teams impressions. Finally,
the test sets were completed by retesting the bug fixes of the current
version. After that procedure, the test sets were ready to be executed
in the QA teams next iteration. As a result, both QA teams always
had the same version of the product for each iteration, but with two
different configurations. Due to the fact that the latest version was
freshly installed for the QA teams in each iteration, the installation
and update mechanism of the software was repeatedly tested by the
PO and the Scrum teams tester.
To thank and to motivate the QA teams, we used some elements from
the concept of gamification. For example, we launched some awards
and small prizes, such as for the most critical bug or the QA team with
the most bugs found.
ing for the correct contact person. On the other hand, the members
of the Scrum team had their workload significantly reduced as a con-
sequence of this interface between the teams. Thus, they could focus
on their work and were only given relevant and revised information
by the QA teams.
However, we underestimated the effort required to create the initial
version of good test plans. The same applied to the time to filter, evalu-
ate, and revise the bugs before we submitted them to the Scrum team.
But we still managed this because we could count on the QA teams
and we knew that we could waive the retests within the Scrum team
apart from the usual ones because the QA teams did the job.
That everything worked out as intended is a big credit to our PO, who
is very open-minded and has a quality assurance background. She
was always open to any suggestions and comments. In other projects
and with another PO this concept would certainly have needed more
negotiation with the PO and other stakeholders. Additionally, we could
benefit from the ability to fall back on our capable and motivated col-
leagues who supported us in the QA teams.
Finally I can report that, the quality assurance in that project is now in a
stable development and the product has been successfully established
on the market. Currently, a large and good choice of automated tests
is available, which are running in continuous integration. The experi-
ences of this QA sprint have certainly had an important influence on
the projects success story.

The kick-off was the official start for the QA teams. For the first complete
execution of the test cases they needed exactly the assumed time of
two days. As an additional advantage, every team had a member who
knew the product in advance. So the other members could benefit from
them in their steep learning curve.
On the first day, the Scrum team worked on the previously known is-
sues until the first bugs were delivered by the QA teams. Additionally,
the two exploratory testers were able to produce some interesting in-
sights during their tests, which could be transformed into reproducible
bugs. After the first two days, the first bugs and issues were fixed. The
retests for these and other improvements (mainly initiated by the QA
teams) were taken into the test sets for the next test iteration. At the
Daily Scrum, the agents from the QA teams reported on their teams
progress and took important information to the QA teams as planned.
> about the author
Martin Uhlig works as a testing consultant for
Saxonia Systems AG in Dresden. Since studying
business informatics, he has been passionate
about agile software development and the cur-
rent trends in the agile community. He has been
working on different projects within the sectors
of logistics, media, and product development.
This also includes agile projects on which he has been working as a
tester and Product Owner.

After each test iteration, the iteration duration of both QA teams

diverged from one another. Therefore we instructed the faster team
to retest old bugs that we had fixed and retested several sprints ago.
The idea panned out as the team actually found some old errors if
only a few that had obviously been reinstalled after fixing.

The Conclusion
The QA sprint was a major success for the project. On the last day of
the sprint, the PO was able to successfully perform the acceptance
test and release the product. As a result of this QA sprint, the team
managed to considerably boost the quality of the product.

The Scrum team and the QA teams appreciated the nature of the
cooperation between the teams. The QA teams benefited from the
clear interfaces because they were given peace of mind whenever
they needed something. Any questions from the QA teams could be
answered very quickly and validly without spending a long time look-

Testing Experience 28/2014 13

 Performance
Column by Alex Podelko
Exploratory
Performance Testing
It looks like exploratory performance testing has started to attract
some attention and is getting a mention here and there. Mostly, I
assume, due to the growing popularity of functional exploratory test-
ing[1]. A proponent of exploratory testing probably would not like my
use of the word functional here, but not much has been written, for
example, about performance exploratory testing and even what has
been written often refers to different things.
There have been attempts to directly apply functional exploratory
testing techniques to performance testing. SmartBear blog posts[2, 3]
contrast exploratory performance testing with static traditional load
testing. My view is probably closer to Goranka Bjedovs understanding
as she described it back in 2007[4].
It was clear to me that a traditional, waterfall-like approach to perfor-
mance testing is very ineffective and error-prone. I presented a more
agile/exploratory approach to performance testing in a traditional
waterfall software development environment at CMG in 2008[5]. I in-
tended to apply the original principals of Manifesto for Agile Software
Development[6] (valuing Individuals and interactions over processes
and tools. Working software over comprehensive documentation. Cus-
tomer collaboration over contract negotiation. Responding to change
over following a plan.) to performance engineering.
Performance testing in projects utilizing specific agile development
methodologies is a separate topic. Having become more involved in the
agile development environment, I added some aspects of it to my pre-
sentation at the Performance and Capacity 2013 conference by CMG[7].
The words agile and exploratory are periodically and loosely used in
relation to performance testing but it does not look like we have any
accepted definition. Both terms are, in a way, antonyms of traditional
waterfall-like performance testing so their meaning may somewhat
overlap in certain contexts. I explained my view of using the word
agile for performance testing in the above-mentioned presentations.
Now it is time to contemplate the use of the word exploratory in the
context of performance testing.
14 Testing Experience 28/2014
If we look at the definition of exploratory testing as simultaneous
learning, test design and test execution1, we can see that it makes
even more sense for performance testing, because learning here is
more complicated, and good test design and execution heavily depend
on a good understanding of the system.
If we talk about the specific techniques used in functional exploratory
testing, some can be mapped to performance testing but definitely
should not be copied blindly. Working with a completely new system,
I found that I rather naturally align my work to a kind of session so
session-related techniques of functional exploratory testing are prob-
ably applicable to performance testing. I would not apply such details
as session duration, for example but the overall idea definitely makes
sense. You decide what area of functionality you want to explore,
figure out a way to do that (for example, create a load testing script)
and start to run tests to see how the system behaves. For example,
if you want to investigate the creation of purchase orders, you may
run tests for different numbers of concurrent users, check resource
utilization, see how the system will behave under stress load of that
kind, or how response times and resource utilization respond to the
number of purchase orders in the database, etc. The outcome would
be at least three-fold: (1) early feedback to development about the
problems and concerns found; (2) understanding the system dynamic
for that kind of workload, what kind of load it can handle, and how
much resource it needs; (3) obtaining input for other kinds of test-
ing, such as automated regression or realistic performance testing to
validate requirements. Then we move to another session exploring
the performance of another area of functionality or another aspect of
performance (for example, how performance depends on the number
of items purchased in the order).
The approach looks quite natural to me and maximizes the amount of
early feedback to development, which is probably the most valuable
outcome of performance testing for new systems. However, there are
different opinions. Objections mostly align along three notions, which,
if taken in their pure form, are not fully applicable to performance
testing of new systems:
Creating a detailed project plan (with test design, time estimates,
etc) in the beginning and adhering to it.
Fully automating performance testing.
Using scientific Design of Experiments (DOE) approaches.
I mention all three of them here because (1) they are often referred as
alternatives to exploratory testing; (2) they all are rather idealistic for
the same reason we do not know much about new systems in the
beginning and every new test provides us with additional information.
And often this additional information makes us to modify the system.
Somehow the point that the system is usually changing in the process
of performance testing is often overlooked.
For example, if your bottleneck is the number of web server threads,
it does not make much sense to continue testing the system once you
realize it. As you tune the number of threads, the systems behavior will
change drastically. And you would not know about it from the begin-
ning (well, this is a simple example and an experienced performance
engineer may tune such obvious things from the very beginning but,
at least in my experience, you will always have something to tune or
optimize that you have no idea about in the beginning).
So, actually, you probably do exploratory testing of new systems one
way or another even if you do not recognize it. And it would probably
be more productive to fully acknowledge it and make it a part of the
process, so you will not feel bad facing multiple issues and will not
need to explain why your plans are changing all the time. I would
concur here with the great post TDD is dead. Long live testing. by
David Heinemeier Hansson[8] discussing, in particular, issues related
to using idealistic approaches. periodically talks and writes about performance-related topics, ad-
References
[1] Exploratory Testing. Wikipedia.
http://en.wikipedia.org/wiki/Exploratory_testing
[2] Ole Lensmar. 2012. Why Your Application Needs Exploratory Load
Testing Today. http://blog.smartbear.com/loadui/why-your-
application-needs-exploratory-load-testing-today
[3] Dennis Guldstrand. 2013. Should Exploratory Load Testing Be Part
of your Process? http://blog.smartbear.com/load-testing/should-
exploratory-load-testing-be-part-of-your-process/
[4] Goranka Bjedov. Performance Testing. 2007. http://googletesting.
blogspot.com/2007/10/performance-testing.html
[5] Alexander.Podelko. Agile Performance Testing. CMG, 2008.
http://alexanderpodelko.com/docs/Agile_Performance_Testing_
CMG08.pdf
[6] Manifesto for Agile Software Development. 2001.
http://agilemanifesto.org/
[7] Alexander Podelko. Agile Aspects of Performance
Testing. Performance and Capacity by CMG, 2013.
http://www.slideshare.net/apodelko/agile-aspects-of-
performance-testing
[8] David Heinemeier Hansson. TDD is dead. Long live testing. 2014.
http://david.heinemeierhansson.com/2014/tdd-is-dead-long-
live-testing.html
> about the author
For the last 17 years, Alex Podelko has worked as
a performance engineer and architect for sev-
eral companies. Currently he is a Consulting
Member of Technical Staff at Oracle, responsible
for performance testing and optimization of
Enterprise Performance Management and Busi-
ness Intelligence (a.k.a. Hyperion) products. Alex
vocating tearing down silo walls between different groups of per-
formance professionals. His collection of performance-related links
and documents (including his recent papers and presentations) can
be found at www.alexanderpodelko.com. He blogs at www.alexan-
derpodelko.com/blog and can be found on Twitter as @apodelko.
Alex currently serves as a director of the Computer Measurement
Group (CMG) www.cmg.org, an organization of performance and
capacity planning professionals.
Testing Experience 28/2014 15
 By Klaus Haller
What Developers and Testers Need to Know
about the ISO 27001 Information Security Standard
Late in 2013, the International Organization for Standardization re-
leased a new version of its ISO 27001 information security standard[1].
The standard covers requirements applying all organizations and ones
relevant only for organizations with in-house software development
and integration projects. They impact testers, developers, and release
managers. This article summarizes the relevant facts and points out
topics that testing and development teams have to work on.
Why Managers Like ISO 27001
Managers are held accountable for security incidents, even if they
have no information security expertise. Gregg Steinhafel illustrated
this involuntarily. He is a former CEO of Target, the second biggest
discount retailer in the USA. Steinhafel was the first CEO of a major
corporation to lose his job due to a data leak[2]. Thus, managers cannot
rely simply on a statement from their Chief Information Officer Se-
curity? Everything is fine! without risking the companys and their
personal future. Here, ISO 27001 comes into play. It brings together,
first, a list of best practices for information security and, second, an
auditing and certification industry. The best practices prevent basic
mistakes or leaving security topics completely unaddressed. External
auditors validate whether the best practices have been implemented.
This external validation gives CEOs and stakeholders extra confidence.
Three Popular Misunderstandings about
Information Security
Information security is a widely used term. Everybody has his own
definition, which can differ from ISO 27001s understanding. The three
most common misunderstandings are:
Misunderstanding 1: Information security focuses
(mainly) on protecting sensitive data
Information security requires the protection of sensitive data. How-
ever, this is only one of the three aspects of the CIA triangle[3], a core
concept in information security. The C represents confidentiality:
protecting sensitive data against unauthorized access. The I stands
for integrity: Information must not be changed inappropriately, either
accidentally or intentionally. Finally, A stands for availability: Users
must be able to retrieve information when needed; no data must get
lost. ISO27001 covers all three aspects of the CIA triangle. Thus, orga-
nizations must address all of them for their certification.
Misunderstanding 2: Information security fights (main-
ly) hackers and malware coming from the outside
Outside hackers and malware pose a threat to every organization, but
employees pose a risk as well. Humans make mistakes, even if they
16 Testing Experience 28/2014
handle sensitive data. Worse, employees might engage into criminal
actions. Snowden illustrated how one single person only can harm a
large organization[4]. Thus, information security must also address
risks from internal employees.
Misunderstanding 3: Information security looks (mainly)
at production systems
Production systems store and process sensitive data, but sensitive data
can also reside in development and test environments. This refers to
the confidentiality aspect. When looking at availability, bad code and
wrong configurations are risks as well. They can harm the stability of
production systems. Thus, IT departments have to address the risk
associated with their change and release process, too.
The ISO 27000 Standard Series
The information security standard consists of a broad document fam-
ily (Figure 1). It is essential to understand the purpose of the various
documents. First, ISO 27000 is the vocabulary standard. It provides
a basic overview and defines the terminology. Second, there are re-
quirement standards: ISO 27001 and ISO 27006. ISO 27006 applies to
auditing organizations only. They are not in the scope of this article.
ISO 27001, however, is the bible for any certification and lists all certi-
fication requirements. The current release dates from late 2013 and is
referred to as ISO27001:2013 to distinguish it from the older version,
ISO27001:2005. ISO 27001 has two parts: a main section and appendix
A. The main section defines a general information security framework.
Topics include top management involvement or the need for an inci-
dent management system. Appendix A lists concrete security topics
(controls) to be implemented.
This ISO 27001 standard is the only normative binding document. In
contrast, guideline standards offer best practices. ISO 27002 helps in
setting up the controls of appendix A of ISO 27001. Other documents
focus on aspects of the main section of 27001. ISO 27003, for example,
looks at information security management systems, and ISO 27005 at
risk management. Industry sector-specific best practices (sector specific
guideline standards) are also available, e.g., for the financial services
industry or for telecommunications. Guideline standards are not man-
datory. Organizations must implement the ISO 27001 requirements
but they are free to follow or not to follow the guideline standards.
ISO 27001 Implementation Responsibilities
As mentioned above, ISO27001 has a main section and an appendix
A. The main section defines an information security framework and
the Chief Security or Chief Compliance Officer has to work on these
topics. In contrast, the IT department in general must implement the
 ISO 27000 Vocabulary
Overview and terminology Standard

ISO 27001 Requirements ISO 27006

Requirements for auditing and Requirement
Main Section Appendix A Standards
certification bodies

ISO 27002 ISO 27003 ISO TR27008

Code of practice for information Implementing information Auditing of information security
security controls security management systems management systems
Guideline
Standards
ISO 27005
Risk Management

ISO 27011 ISO TR 27015 Sector-Specific

Guideline
Telecom Sector Financial Services
Standards

Figure 1. Overview of the 27000 Standard Series. This article mainly deals with ISO 27001 Appendix A with interpretations derived from ISO 27002.

majority of the 114 controls from Appendix A. Some of them are only customer data is sensitive, who can access it, etc. Testers and engineers
relevant to developers, testers, and change managers. They have to are not involved in the classification, although the policies can impact
provide solutions for the controls (see Figure 2), for which they can them. They might forbid, e.g., the storage of credit card numbers of
rely on the next sections. clients in test systems.

Engineering-owned assets have to be classified as well. Potentially, IT has

Does a control define an obligation for to drive this. The main assets are source code and documentation, such
developers, testers, and change managers? as requirements, specifications of new product features, architectural
documents, trading algorithms of hedge funds, etc., and they can be
Yes No as critical as production data. ISO27001 does not declare any asset to
be sensitive or not, it just demands clarification.

Developers/testers/release The policies for handling production data and engineering-owned

Yes managers rely on company-wide assets impact tool decisions. In the last few years, outsourcing re-
Does a Nothing to
solution from the IT department/IT quirements and test or project management tools became popular.
company- be done by
Security/Legal & Compliance, etc. Software-as-a-service is thriving. With ISO 27001, organizations must
wide developers,
solution testers, and ensure that projects store data in externally hosted systems only if this

exist? change does not contradict information security policies.
Developers/testers/change
No managers
managers must provide a solution ISO 27001 also demands secure development environments for the
(focus of this article) complete development cycle (control A.14.2.6). The need for confiden-
tiality, availability, and integrity has a broad impact on access control
Figure 2. ISO27001 Appendix A Standards and the Need for Engineering, Testing and mechanisms, the hiring and contracting of developers and testers,
Change Teams to Come Up with a Solution on their Own. and backup strategies.

A highly critical asset in development and test environments is the

Information Assets in Development and Testing
Information security starts with identifying the valuable information
assets, and classifying and labeling them. The user groups with access
have to be clarified. Data access and protection mechanisms must be
defined (controls A.8.2.1-3).
There are two types of information assets: production data and docu-
ment assets, and engineering-owned assets. In the case of production
data and documents, the business, the legal and compliance depart-
ment, and risk management all work together. They classify the assets
and define policies for handling them. Data privacy laws impact the
policies, especially in Europe. The policies define, for example, whether
test data. Many applications especially business applications in-
corporate databases. Testers need suitable data in the databases in
test environments and ISO 27001 control A.14.3.1 demands that the test
data is protected. When looking at the ISO 27002 guideline, it is clear
that the standard reflects old style test data management. In other
words, test data comes from production. The focus in ISO 27002 is to
mitigate the risks associated with the use of production data, such
as the ability to audit the copy process and strict access rules for test
environments. The trend towards synthetic test data in Europe is not
reflected (see[5] for an in-depth discussion on test data management).
However, ISO 27002 is not normative. Organizations can implement
ISO 27001 in their own way. Especially when organizations test with
synthetic data, many ISO 27002 ideas are obsolete.

Testing Experience 28/2014 17

 Security principles
including rules for
data transfer
Development Testing
Developers work on Testers work on
development systems test systems
Perform security tests Organize user
Security principles acceptance
followed testers
Independent testers check entry
criteria and sign off the testing
including auditable documentation.
Figure 3. How ISO 27001 Influences in the Software Development Processes V-model (Left) and Scrum (Right)
Scrum, V-Model, Security Principles, and ISO 27001
ISO27001 defines three controls for the software development pro-
cesses. First, acceptance testing against the requirements is manda-
tory (control A.14.2.9) against functional and non-functional ones,
the latter including the security requirements. Second, there must be
security tests during the overall development process (control A.14.2.8).
Third, development, test, and operational environments must be
separated (control A.12.1.4).
These controls are compatible with many software development pro-
cesses such as Scrum or the older V-model (Figure 3 ). However, Scrum
requires more thoughts than the V-model. IT departments using the
V-model often have a test center and quality gates. Quality gates define
the criteria for when tests can start and when they succeed. One cri-
terion before the test start can be that developers performed security
tests. One exit criterion for testing can be that user acceptance tests
in the test environment succeeded for functional and non-functional
requirements. When testers work in a test center and are not part of
a development team, development teams cannot put much pressure
on them. Thus, testers can enforce the ISO 27001 requirements more
easily for all projects, even delayed ones.
In the world of Scrum and Agile, there are often no test centers and no
central governance. It might be questionable, but it is reality (see[6]
for better options). Development and testing overlap time-wise. Roles
overlap. In the development methodology, however, this is no excuse
for missing documentation or non-implemented controls in an ISO
27001 audit. All controls mentioned above must exist for all projects,
even delayed ones. This is the key challenge for agile projects in an
ISO 27001 organization.
Besides the controls for the development process, the ISO standard
formulates controls for the software product itself. It demands that
the security needs in the engineering process are reflected by defin-
ing principles and they must be applied to all development projects
(control A.14.2.5). First, the IT department has to write them down.
Second, the IT department must enforce the principles in all projects.
18 Testing Experience 28/2014
Security principles
including rules for
data transfer
Testing
duction systems
Usage on pro-
Usage on
production
systems
Development
Auditable proof/logs needed for:
Developers did develop and run security tests on
development systems.
User acceptance tests passed on test systems
Security principles followed
As part of the requirements analysis for new software or new releases,
information security requirements have to be collected and specified
(control A.14.1.1). The standard requires clarification of the security
needs of data transfers via public networks and electronic transactions
(control A.14.1.2/A.14.1.3). Again, confidentiality alone is not sufficient.
Integrity and availability have to be addressed, too.
ISO 27001 Controls for Release and Change
Management
The ISO 27001 standard emphasizes availability controls, too, i.e., the
A of the CIA triangle. The following controls help to ensure stable
production systems:
Formal change management for changes in IT and business (con-
trol A.12.1.2)
Discouraging changes to vendor-provided software packages
(A.14.2.4)
Strict change control procedures even during development to
prevent unwanted modifications (control A.14.2.2)
Specific testing needs for operation system changes, which require
business critical applications to be tested against a new platform
(control A.14.2.3)
Mandatory rules for software and operating system installation
procedures, and who can do what (A.12.5.1)
Preventing even small changes to circumvent testing or the
change process by separating development, testing, and opera-
tional environments (control A.12.1.4), and by having access control
on the source code (control A.9.4.5)
This is old-time check-box style release management. IT staff have
a list of criteria they check. If all have been met, a change can go to
production. This model fulfills todays needs of many organizations. act accordingly. Chaotic geniuses or non-genius chaotic persons must
Highly agile organizations, however, prefer continuous integration[7] be embedded into teams that ensure ISO 27001 conformity.
and DevOps[8]. They invest in test automation to have quick feedback
Not every developer and tester might appreciate the changes the ISO
loops. There might be even no manual testing before deploying small
27001 standard brings. Thus, we dedicate one Doug Larson quote to
changes into production, which raises the question as to whether this
all who preferred working in IT in the times before standards such as
conforms to ISO 27001.
ISO 27001: Nostalgia is a file that removes the rough edges from the
A clever strategy for dealing with ISO 27001 can help. Discussing good old days.
whether ISO 27001 is outdated and Scrum, DevOps, etc. are state of
the art will result in frustration. ISO 27001 is a top management deci-
sion that overrules any development or test process. Developers and References
testers should invest their time into explaining how they conform to
[1] ISO/IEC 27001:2013 Information technology Security techniques
ISO 27001, even if there is no or minimal human involvement between
Information security management systems Requirements, ISO,
coding and deployment production. Written operational procedures,
Geneva, Switzerland, 2013
archived audit logs, etc. help to tell one story: All ISO 27001 controls are
in place, some with manual check lists, others relying on automated, [2] The Associated Press: Target CEO Gregg Steinhafel resigns follow-
auditable processes. ing last years security breach, 5.5.2014

[3] Wikipedia: Information security, http://en.wikipedia.org/wiki/

Controls for Outsourcing
Outsourcing and offshoring are common in software development
and testing, but pose two information security risks. First, the sourc-
ing partners obtain sensitive data they should not have. Second, their
software development and testing processes might not address the
information security needs properly. ISO 27001 addresses the latter
aspect with control A.14.2.7. It requires the supervision of outsourced
development and testing. The work can be outsourced but the respon-
sibility stays with the organization. In general, ISO 27001 requires sup-
pliers also to be managed with regard to information security (control
A.15). Any supplier management can enforce this. The controls are
not specific to software development and testing, though the checks
might differ slightly.
Information_security, last retrieved July 5, 2014
[4] K. Dilanian: A spy world reshaped by Edward Snowden, Los Ange-
les Times, 22.12.20123
[5] K. Haller: Test Data Management in Practice, Software Quality
Days 2013, Vienna, Austria, 2013
[6] K. Haller: How Scrum Changes Test Centers, Agile Records, August
2013
[7] M. Fowler, M. Foemmel: Continuous integration
http://ww.dccia.ua.es/dccia/inf/asignaturas/MADS/2013-14/
lecturas/10_Fowler_Continuous_Integration.pdf, 2006, last
retrieved July 5, 2014
[8] J. Turnbull: What DevOps means to me, http://www.kartar.net/
2010/02/what-devops-means-to-me/, last retrieved July 5, 2014

Two Main Conclusions on ISO 27001, and

Development and Testing
> about the author
Conclusion 1: Development, testing, and change man-
agement require clear written information security Klaus Haller has worked in IT consulting for more
policies. than nine years, primarily in the banking sector.
His main topics are IT risk and compliance, data
ISO 27001 does not require specific organizational forms or software
loss prevention, test center organization, and
processes. ISO 27001 emphasizes clear rules and policies for the han-
testing of information systems landscapes. He
dling of information assets and the engineering process. First, they
publishes regularly on testing and is a frequent
must clarify what data is sensitive and how to handle it. Second, they
speaker at conferences. He has been with Swiss-
must explain how the organization engineers secure software in a
com since 2006 and is based in Zurich.
secure development area. Third, they must state how to get software
Website: www.klaushaller.net
into production without any risk for production stability.
LinkedIn: www.linkedin.com/pub/klaus-haller/48/a2b/798

Conclusion 2: The organization must enforce the poli-

cies in all projects and have evidence.
ISO 27001 expects policies to be enforced consistently and to have au-
ditable evidence. In other words, there must be a process organization
and all employees must be continuously educated and motivated to
 By Jacqueline Vermette
What Makes a Good Quality Tester?
Twenty five years ago, when I started my career as a young software
engineer, there were very few professional testers. Only large, impor-
tant projects had testing teams. For most projects, software testing
consisted of having the lead analyst review the system just before
delivery. Occasionally, the tests were even performed together with the
client during the acceptance phase, leading to unpredictable results.
After a particularly painful client acceptance experience, the manager
would call a meeting. He would declare: For the next project, we
absolutely need to test before delivery.
But, whos going to test? replied the project leader.
Well, we have Bob and Jackie who are not very busy. They can test
during the two weeks before delivery. Lets ask them to find as many
bugs as they can.
All right, lets give a try.
So in the next project, all the team members would work very hard
during the last two weeks to cap the project. Our new testers, Bob and
Jackie, would do their very best despite their limited experience. How-
ever, Bob did not want to pursue a career in testing. He had no interest
in performing tests all day and would eventually leave the project.
Jackie found more bugs than Bob did. She believed in the process and
would apply her learning experience to improve her contribution in
their subsequent projects.
This was a very typical scenario and reminds me of a Dilbert comic
strip published in March 2010. In it, Dilberts boss asks his help in
quality testing a new software version. Dilbert gives all sorts of silly
reasons to not become a quality tester and concludes his message
by swatting his boss across the face with a binder, to the bosss great
displeasure. In this situation, it is clear that Dilbert has no interest
in testing and most probably does not have the skill set to be a good
tester anyway. Of course, this is a satire, but it is very close to reality.
During the last few decades, development methodology and test-
ing processes have obviously evolved. But even now there is still this
tendency in the IT community to believe that anyone can be a quality
tester but can everyone really test properly? It is incorrect to think
that anyone can produce good software tests. I personally think that
you need the proper genetic make-up to be a great tester, but what
qualities are we talking about here?
A natural-born tester:
1. Has the technical knowledge and deep analytical ability needed to
create extremely complex tests. These characteristics, along with
an innate need to break things down, add to the strength and reli-
ability of the final product. Simple tests can find the most obvious
bugs, such as formatting errors or missing boundary validations.
But it takes more detailed testing scenarios to uncover errors of
logic or cascading effects. For example, going through all cases of
a state diagram, and especially from a state to a forbidden state,
often reveals surprising results. For complex cases, documenting
the tests to execute is essential. Using an old fashioned Excel sheet
is always better than nothing.
20 Testing Experience 28/2014
2. Has an ability to learn. Testers may be asked to go from a limited un-
derstanding of a product to mastering that product in a very short
timeframe. They must be able to memorize details and understand
each modules concepts while maintaining a general overview of
the product. Testers must be willing to review and learn all the
expected system behavior by studying the technical documenta-
tion and spending time with the main analyst. I remember one
particularly complex application for an aluminum smelter where
very few people had an overview of the whole business process at
the beginning. The management was not too sure whether the
test team would be able to test adequately. But by reading all the
available documentation and asking questions and more ques-
tions, we did a great job. Never be shy about asking questions in
order to understand details about the application, especially if the
specifications are not clear enough.
3. Can think outside the box, and takes into account assumptions
and concrete facts. Not all conditions are necessarily stated in the
functional specifications. It is like when you buy a car, you assume
that the hood can easily be opened to check the motor. This crite-
rion is not mentioned in the cars features, but everybody expects
it. Testers should try to test unwritten features. Some unwritten
characteristics could have a significant impact on the quality of
the final product, hence the need to read between the lines. For
example, the system can support some required functionalities,
but what would happen if I tried something a little different? Does
the system support it? Does it crash? Does it corrupt data?
4. Cultivates a keen sense of observation and notes small details.
Their perfectionism can unfortunately annoy programmers and
developers, but good testers can find the biggest bugs in the least
likely situations. If a sequence of system operations is available to
the user, why are they not supposed to perform them, for example?
Why does the screen have labels with different fonts? Report fields
that are not properly aligned or inconsistent use of capitalization are
other examples of small details that can negatively impact the qual-
ity of the product. Some people just notice this type of error more
than others. They are probably like that in all their daily activities.
5. Cares deeply about the final product. They believe in their mission,
which is to protect the companys reputation. They love testing
and are proud to find bugs. Finding a bug is highly satisfying and
finding an especially tricky one surely makes their day.
6. Is organized yet flexible. They pay attention to the manuals and
conduct the tests systematically. This is very important in order to
reproduce a bug. A bug that cannot be properly detailed in order to
be reproduced cannot be corrected. They can also adapt to changes
during the course of a project and are willing to repeat tests over and
over, if necessary. After a bug correction, a test case might need to
be modified and re-executed to validate the quality of the system.
Even with all these attributes, no one can be a good tester if they
cannot bring a positive influence to a development team. A tester
must provide positive feedback, be able to motivate team members to
 improve the quality of their work and, in general, manage each team References
members self-respect.
[1] PAGE Alan, How we test software at Microsoft, Microsoft Press,
The testers role is in constant flux. To stay competitive in todays 2009
market, companies must now produce ever more complex software
[2] PRATAP K.J., The Psychology of testing, The code Project, 2007
solutions, at an ever faster pace, and at a lower cost. Test management
tools, system simulation, and automatic test case execution are now
a must. We must adapt to these changes by developing our program-
ming abilities or by working closely with developers. Promoting more
> about the author
completed unit tests and testing as soon as possible with the developers
helps greatly to reduce errors early in the test process.
Jacqueline Vermette is a quality assurance man-
An effective tester cannot guarantee that a product is completely ager with 25 years of experience in quality assur-
bug-free. However, choosing the right person for the role will lead ance, quality control, functional analysis, and
to the best possible results by drastically reducing the impact of any programming. She also worked to set up quality
remaining bugs. assurance and control methodologies in order
to ensure the quality of deliverables for manu-
In conclusion, for your next project, do not select a Dilbert to perform
facturing industry projects. Jacqueline is a certi-
your quality tests. When selecting a software developer, you are trying
fied software tester (CSTE) and is currently working at Keops Tech-
to choose the right person to take on your project. You want the best
nologies.
one possible. Just apply the same principle when choosing a quality
tester. An efficient software tester will help you to maximize your
return on investment.

Wissen fr Tester

Seit
10 Ja
hren
ein
Best
selle
r

T. Linz E. Hendrickson A. Spillner, T. Linz A. Spillner, T. Roner, H.Stauffer, B. Honegger,

Testen in Scrum-
Projekten
2013 248 Seiten E 34,90 (D)
ISBN 978-3-89864-799-1
Explore It!
2014 196 Seiten E 26,90 (D)
ISBN 978-3-86490-093-8
Basiswissen Softwaretest
5. Auflage
2012 312 Seiten E 39,90 (D)
ISBN 978-3-86490-024-2
M. Winter, T. Linz H. Gisin
Praxiswissen Softwaretest Testen von Data-Warehouse-
Testmanagement und Business-Intelligence-
4. Auflage Systemen
2014 268 Seiten E 69,90 (D)
2014 506 Seiten E 44,90 (D)
ISBN 978-3-86490-052-5 ISBN 978-3-86490-072-3

-Book:
Buch + E unkt.de/
plus
www.dp
dpunkt.verlag GmbH Wieblinger Weg 17 69123 HeidelbergTesting
fon: Experience
0 62 21 / 1483 40
28/2014 21
fax: 0 62 21 / 14 83 99 e-mail: bestellung@dpunkt.de www.dpunkt.de
 By Wolfgang Gottesheim
Ready for CAMS?
The Foundation of DevOps
In a traditional organisation, handoffs between developers and opera-
tors cause friction as these groups follow different goals. Developers
and business drivers within the organization want customers to use
new features and benefit from other improvements, while operators
seek stability and want to provide a stable environment.
One of the groundbreaking books around DevOps, The Phoenix
Project by Gene Kim, Kevin Behr and George Spafford, describes the
practice through The Three Ways of systems thinking, amplifying
feedback loops, and providing a culture of continuous experimenta-
tion and learning. Systems thinking means to focus on overall value
streams and to make sure that defects (for example, broken builds),
are not passed on to downstream units (like the Ops department).
Amplifying the feedback loops translates to providing proper commu-
nication channels between Dev and Ops, and to achieve this without
creating an overly complex framework of processes. Creating a culture
of experimentation and learning encourages everyone to take risks
and learn from failures.
A lot of people see DevOps as the extension of agile practices from
developers towards operations to overcome cumbersome processes.
DevOps a Definition
DevOps aligns business requirements with IT performance, and recent
studies have shown that organizations adopting DevOps practices have
a significant competitive advantage over their peers. They are able to
react faster to changing market demands, get out new features faster,
and have a higher success rate when it comes to executing changes. The
goal of DevOps is to adopt practices that allow a quick flow of changes
to a production environment, while maintaining a high level of stabil-
ity, reliability, and performance in these systems. However, the term
nowadays covers a wide range of different topics and consequently
means different things to different people.
Figure 1. Running tests against the production system gives a better input for capacity planning and uncovers heavy load application issues
22 Testing Experience 28/2014
The Foundation of DevOps
There are a number of definitions and interpretations for DevOps float-
ing around, and a way to look at it is in terms of CAMS: DevOps means
to adopt a Culture of blame-free communication and collaboration,
to embrace Automation to allow people to focus on important tasks,
to introduce continuous Measurements to get feedback on the qual-
ity and usage of features and bug fixes, and to encourage Sharing of
these measurements. This underpins the fact that DevOps is not about
standards or tools, it is about enabling communication and collabora-
tion between departments in an organization.
Plugging Performance into DevOps
When we talk about collaboration, a key aspect is how we prevent
finger-pointing between teams when problems occur. We have to
handle and prevent failures by continuously ensuring high quality,
but while almost every definition of software quality mentions both
functional and non-functional requirements, the non-functional as-
pects like usability, deployability, and performance are only rarely
measured automatically. This becomes a problem as performance
issues are among the hardest to solve they are heavily dependent
on load, deployment, and user behaviour, and Ops teams need help
in identifying these issues and communicating them to Dev in an
actionable way.
In order to focus the entire team on performance, you must plug
performance into the four pillars of CAMS:
Culture: Tighten feedback loops between Dev and Ops
Automation: Establish automated performance monitoring
Measurement: Measure key performance metrics in CI, Test and Ops
Sharing: Share the same tools and performance metrics data across
Dev, Test and Ops
 Figure 2. Automated Tests running in CI also help to detect performance regressions on metrics such as # of SQL calls, page load time, # of JS files or images, etc.
Four Milestones that Companies Should Have in
Mind
Culture Tighten the Feedback Loops between
Dev and Ops
Culture is the most important aspect because it changes the way in
which teams work together and share the responsibility for the end
users of their application. It not only encourages the adoption of agile
practices in operations work, it also allows developers to learn from
real world Ops experience and starts a mutual exchange that breaks
down the walls between teams. From a performance perspective, it
is important to establish a shared understanding of performance
between Dev, Test, and Ops. This enables collaboration based on well-
known measurements and metrics, establishes a shared language
understood by all teams, and allows all teams to focus on the actual
problems. Finger-pointing between teams has to be replaced by a
practice that enables them to get to the root cause of performance
issues, and working together on current issues enables developers to
become aware of performance problems and their solutions.
Automation Establish a Practice of Automated Perfor-
mance Monitoring
Operations and test teams usually have a good understanding of
performance, and they need to educate developers on its importance
in large-scale environments under heavy load. Providing automated
mechanisms to monitor performance in all environments, from CI and
test environments to the actual production deployment, allows the
shared language of performance to be spoken.
Measurement Measure Key Performance Metrics in CI,
Test, and Ops
With performance aspects being covered in earlier testing stages, per-
formance engineers on testing teams have time to focus on large-scale
load tests in production-like environments. This helps them to find
data-driven, scalability, and third party impacted performance prob-
lems. Close collaboration with Ops ensures that tests can be executed
either in the production environment or in a staged environment
that mirrors production, thus increasing confidence when releasing
a new version.
Sharing: Share the Same Tools and Performance Metrics
Data across Dev, Test, and Ops
The more traditional testing teams are used to executing perfor-
mance and scalability tests in their own environments at the end of a
milestone. With less time for extensive testing, their test frameworks
and environments have to become available to other teams to make
performance tests a part of an automated testing practice in a Continu-
ous Integration environment. The automatic collection and analysis of
performance metrics ensures that all performance aspects are covered.
This once again entails defining a set of performance metrics that is
applied across all phases, as this is beneficial to identifying the root
cause of performance issues in production, testing, and development
environments.
Conclusion
The first step in adopting a performance culture is to enable a shared
understanding of performance through a set of key performance
metrics that are accepted, understood, and measured across all teams.
These performance metrics allow all teams to talk about performance
in the same way, and reduce the guesswork and finger-pointing of-
ten associated with troubleshooting performance problems. Once
these metrics have been defined, their automated measurement and
analysis is the next step that makes performance a part of a DevOps
practice.
> about the author
Wolfgang Gottesheim has nearly ten years of
experience as a software engineer and research
assistant for Java Enterprise environments. In his
role as Technology Strategist in the Dynatrace
Center of Excellence he is involved in the strate-
gic development of the Application Performance
Management solutions. His focal points are Con-
tinuous Delivery and DevOps.
Twitter: @gottesheim
Testing Experience 28/2014 23
 By Danilo Berta

Combinatorial Testing Tools

When a software application accepts several inputs, each of which 1-wise testing
can assume different values, it is impossible in general to test all
When the number of combinations is high, it is possible at least verify
combinations of values of input variables, simply because they are too
that at least once each individual value of the variables is given
many. Lets take an example consider a software feature that accepts
as input to the program to be tested. In other words, if the variable A
as inputs three possible values A, B, and C. These values can be chosen
can take the values A1, A2, A3, at least a first test must be executed in
arbitrarily from the following table:
which the variable A=A1, a second test in which A=A2, and a third test
in which the variable A=A3; the same goes for the other variables. This
A B C type of test provides a so-called wise-1 cover, and we will see shortly
A1 B1 C1 the meaning. In practice, we have the following table:
A2 B2 C2
A3 B3 # TEST A B C

A4 1 A1 * *
# Values 2 A2 * *
4 3 2 3 A3 * *
4 A4 * *
Table 1. Variables and Values
5 * B1 *

The total number of possible combinations of variables (A, B, C) is 6 * B2 *

equal to 432=24; in practice, in order to ensure trying all possible 7 * B3 *

combinations of the values of the variables (A, B, C) at least once, 24 8 * * C1
test cases must be carried out. Such combinations are the following: 9 * * C2

14 58 912 1316 1720 2124 Table 3. Max Wise-1 Test Set

A1;B1;C1 A1;B3;C1 A2;B2;C1 A3;B1;C1 A3;B3;C1 A4;B2;C1

A1;B1;C2 A1;B3;C2 A2;B2;C2 A3;B1;C2 A3;B3;C2 A4;B2;C2 A possible first reduction is to set a value for the first variable and assign
A1;B2;C1 A2;B1;C1 A2;B3;C1 A3;B2;C1 A4;B1;C1 A4;B3;C1 a random (but permitted) value to the other variables (stated with *
in Table3) and proceed in this way for all the variables and values. In
A1;B2;C2 A2;B1;C2 A2;B3;C2 A3;B2;C2 A4;B1;C2 A4;B3;C2
this way, we reduce the test cases from 24 to just 9. It is still possible
Table 2. Combinations of Variables for A, B, and C Values to further reduce the number of test cases, considering that instead
of * you can put a value of the variable which can then be excluded
Now, in this particular case, such a number of tests can still be afford- from the subsequent test cases.
able. However, if we consider the general case of N variables X1, X2, ,
Put into practice, for test case #1 in place of B=* put B=B1, instead of
Xk, the first accepting n1 possible values, the second n2 possible values,
C=* put C=C1 and remove test case #5 and test case #8, which are now
the k-th that assumes nk possible values, the total number of combina-
both covered by test case #1.
tions is equal to: n1n2nk. Such a value, even for low values of n1,
n2, , nk is a high number. For example, if k=5 and (n1=3; n2=4; n3=2; Test case #2: in place of B=* put B=B2, and in place of C=* put C=C2,
n4=2; n5=3) we get a number of combinations equal to 34223=1 44. and erase test cases #6 and #9, both of which are now covered by
That is quite a large number of tests to perform if you want to ensure test case #2.
complete coverage of all combinations.
Test case #3: instead of B=* put B=B3, and in place of C=* insert any
In real software applications, the number of values that can assume ni value C1 or C2, considering that the values of the variable C equal to C1
variables is high and it is easy to reach the hundreds of thousands of and C2 have already been covered by test cases #1 and #2; we can let
combinations, making it impossible to perform comprehensive tests C=* and postpone the choice of whether to enter C1 or C2. Now, remove
on all combinations. test case #7, since B=B3 is now covered by test case #3.

How can we carry out an effective test when the number of variables Having understood the mechanism, there remains only test case #4,
and values is so high as to make it impossible to exhaustively test all which covers A=A4; we can let B=* and C=*, postponing the choice of
combinations? What reduction techniques apply? what to actually select when we will really perform the test.

24 Testing Experience 28/2014

The symbol * represents dont care; we can put any value in it and # VARIABLES VALUES
the coverage of the test set does not change, and the values of all PAIRS TOTAL
A B C
variables will be used at least once. Those with * value should be
covered more than once.
(A,B) 4 3 43=12
(A,C) 4 2 42=8
The final minimized test set for wise-1 coverage is the following:
(B,C) 3 2 32=6
# TEST A B C GRAND TOTAL 12+8+6=26

1 A1 B1 C1
Table 5. Counting the Pairs of Values of the Variables A, B, and C
2 A2 B2 C2
3 A3 B3 * Hence, the total of all the pairs of values of the variables A, B, and C
4 A4 * * whose values are reported in Table1 is equal to 26 and they are all
printed in the following table:
Table 4. Minimized Wise-1 Test Set

Table4 is drawn from Table3, moving up the columns of the variables B # # of pairs of values

and C to fill the values w

ith * with specific values; the * values stagnate A, B A, C B, C
in the lines that cannot be covered from specific values (row 3 variable 1 A1,B1 A1,C1 B1, C1
C and row 4 variable B), because the number of values of the variables 2 A1,B2 A1,C2 B1, C2
are different (for example, variable B has just 3 values, while variable A
3 A1,B3 A2,C1 B2, C1
has 4 values; the missing B value with respect to A is replaced by *).
4 A2,B1 A2,C2 B2, C2
Therefore saying that a test set, such as that reported in Table4, pro- 5 A2,B2 A3,C1 B3, C1
vides wise-1 coverage is equivalent to saying that each individual value
6 A2,B3 A3,C2 B3, C2
of each variable is covered at least once.
7 A3,B1 A4,C1
The general wise-1 rule we can deduce is the following: 8 A3,B2 A4,C2
9 A3,B3
N variables X1, X2, , Xk, the first assuming n1 possible values, the
10 A4,B1
second n2 possible values, the k-th nk possible values, the maximum
11 A4,B2
number of tests that provide coverage wise-1 is equal to n1+n2++nk,
while the minimum number of tests is equal to the maximum value 12 A4,B3

among { n1, n2, , nk }. # PAIRS 12 8 6

TOTAL 12+8+6=26
In real cases, what is of interest is always the test set with the minimum
Table 6. Pairs of Values of the Variables A, B, and C
number of test cases that ensures the chosen coverage (and this is for
obvious reasons).
Why should you consider a test set to cover wise-2? Is it not enough to
consider a test set with 1-wise coverage? Here we enter into a thorny
issue, in which opinions are different, concordant, and discordant.
2-wise testing or pairwise testing
Below is the incipit from the site www.pairwise.org[1]:
If the wise-1 test guarantees coverage of every single value for each
variable, it is easy to see that a wise-2 test set ensures that all pairs
Pairwise (a.k.a. all-pairs) testing is an effective test case genera-
of values of the variables are covered at least once. In the case of the
tion technique that is based on the observation that most faults are
variables listed in Table 1, all pairs of variables are as follows: {(A, B), (A,
caused by interactions of at most two factors. Pairwise-generated
C), (B, C)}. In fact, the combinatorial calculation shows that the number
test suites cover all combinations of two, therefore are much smaller
of combinations of N values taken K to K (with NK) is equal to:
than exhaustive ones yet still very effective in finding defects.

N N!
= We mention also the opinion of James Bach and Patrick J. Schro-
K K ! (N K) !
eder about the pairwise method: Pairwise Testing: A Best Practice
That Is Not from James Bach, Patrick J. Schroeder available from
In our example we have three variables (N=3) taken two to two
http://www.testingeducation.org/wtst5/PairwisePNSQC2004.pdf[2]:
(K=2), and applying the combinatorial formula above we get
3 3!
= = 3 ; the three pairs are {(A, B), (A, C), (B, C)}. What do we know about the defect removal efficiency of pairwise
2 2 ! (3 2) !
testing? Not a great deal. Jones states that in the US, on average,
the defect removal efficiency of our software processes is 85%[26].
Wanting to compute all possible pairs of variable values, we need to
This means that the combinations of all fault detection techniques,
consider the following table:
including reviews, inspections, walkthroughs, and various forms of
testing remove 85% of the faults in software before it is released.

Testing Experience 28/2014 25

 In a study performed by Wallace and Kuhn[27], 15 years of failure
data from recalled medical devices is analyzed. They conclude that
98% of the failures could have been detected in testing if all pairs of
parameters had been tested (they did not execute pairwise testing,
they analyzed failure data and speculated about the type of test-
ing that would have detected the defects). In this case, it appears
as if adding pairwise testing to the current medical device testing
processes could improve its defect removal efficiency to a best-in-
class status, as determined by Jones[26].
On the other hand, Smith, et al.[28] present their experience with
pairwise testing of the Remote Agent Experiment (RAX) software
used to control NASA spacecraft. Their analysis indicates that pair-
wise testing detected 88% of the faults classified as correctness and
convergence faults, but only 50% of the interface and engine faults.
In this study, pairwise testing apparently needs to be augmented
with other types of testing to improve the defect removal efficiency,
especially in the project context of a NASA spacecraft. Detecting
only 50% of the interface and engine faults is well below the 85%
US average and presumably intolerable under NASA standards.
The lesson here seems to be that one cannot blindly apply pairwise
testing and expect high defect removal efficiency. Defect removal
efficiency depends not only on the testing technique, but also on
the characteristics of the software under test. As Mandl[4] has
shown us, analyzing the software under test is an important step
in determining whether pairwise testing is appropriate; it is also an
26 Testing Experience 28/2014
important step in determining what additional testing technique
should be used in a specific testing situation.
[4] R. Mandl, Orthogonal Latin Squares: An Application of Experi-
ment Design to Compiler Testing, Communication of the ACM,
vol. 28, no. 10, pp. 10541058, 1985.
[26] Jones, Software Assessments, Benchmarks, and Best Practices.
Boston, MA: Addison Wesley Longman, 2000.
[27] D. R. Wallace and D. R. Kuhn, Failure Modes in Medical Device
Software: An Analysis of 15 Years of Recall Data, Intl Jour. of
Reliability, Quality and Safety Engineering, vol. 8, no. 4, pp.
351371, 2001.
[28] B. Smith, M. S. Feather, and N. Muscettola, Challenges and
Methods in Testing the Remote Agent Planner, in Proc. 5th Intl
Conf. on Artificial Intelligence Planning and Scheduling (AIPS
2000), 2000, pp. 254263
To clarify, we can say that the pairwise or 2-wise test method ensures
that all combinations of pairs of values of the variables are tested,
theoretically ensuring the maximization of the anomalies found,
with percentages ranging from 50% to 98% according to the studies.
In fact, no test can ever guarantee a defined percentage removal of
defects (which can only be calculated ex post for the specific project).
Lets say trying to be realistic that pairwise achieves a valid agree-
GET YOUR
PRINTED COPY
r a ll is s u e s in our shop!
Orde
www.testingexperience-shop.com
ment between the number of tests to be performed and the anomalies
detected, when the number of variables and their values is so high
that it is not possible to test all the combinations (the so called all-wise
testing or N-wise testing, where N is the number of variables we are
playing with).
In the case of a test set covering wise-2 level, it is very simple to know
the maximum number of tests that provides coverage of all pairs of
values of the variables. This value is equal to the number of pairs of
values of the variables themselves. In our example of three variables A,
B and C in Table 1, this number is 26 (calculated as described in Table 6).
The real problem still unsolved is to discover the minimum number
of tests that guarantees wise-2 coverage, although there are a variety
of methods and algorithms that approximate this value for a problem
with an arbitrary number of variables and values. Examples of tools
that use these algorithms are:
1. Microsoft Pairwise Independent Combinatorial Testing tool
(PICT), downloadable from http://download.microsoft.com/
download/f/5/5/f55484df-8494-48fa-8dbd-8c6f76cc014b/pict33.msi
2. NIST Tools: http://csrc.nist.gov/groups/SNS/acts/documents/
comparison-report.html
3. James Bach AllPairs downloadable from
http://www.satisfice.com/tools.shtml
4. Other tools here: http://www.pairwise.org/tools.asp
n-wise Testing with n>2
It is now easy to extend the concept of pairwise or 2-wise testing to the
generic case of n-wise, with n>2. The generic test set provides n-wise
coverage if it is able to cover all the n-tuples (3-tuples if n=3, 4-tuples
if n=4 and so on). As in the pairwise, is always possible to know the
size of the maximum test set, equal to the number n-tuples of values
of the variables, but there is no way to know in the general case the
size of the minimum test set that guarantees coverage n-wise.
Using NIST Tools (ACTS) or Microsoft PICT (or other similar tools), it is
possible to extract a test set that approximates as closely as possible
the minimum test set. It is clear that, given a set of N variables, the
maximum level of wise which you can have is equal to the number
of variables. So, if we have four variables, a 4-wise test set coincides
with all possible combinations, while a 5-wise test set or higher makes
no sense.
The combinatorial testing techniques that we discussed in the first
part of the article are intended to solve a basic problem, which we
have already discussed and we rephrase as follows:
Direct Combinatorial Test Problem: In a software system accepting
N input variables, each of which can take on different values, find the
test set with the smallest number of test cases that guarantees (at least)
coverage of all combinations (2-tuples) of all the values of the variables
involved.
The Pairwise technique and a large number of support tools have
been developed to solve this problem. Once such a test set (the small-
est possible) has been generated, you should run the test cases and
detect (if present) all the defects that arise in the software under test.
There is also a second problem, maybe less popular compared to the
previous, as follows:
Reverse Combinatorial Test Problem: Given a Test Set for which you do
not know the method of generation (if any), calculate what percentage
of nwise coverage the test set ensures, with nwise between 1 and the
number variables in the test set.
An example: tests generated by automatic tools for which you have low
or almost zero process control, or when the test cases are generated
by the automatic flows that feed interfaces between different systems
(think of a system that transmits accounting data from a system A
to B); test data are in general excerpts from historical series over
which you have no control.
For test scenarios in some way related to a combinatorial inverse
problem, it is not easy to find support tools as such tools are not read-
ily available. The only tool I found is NIST CCM, in alpha-phase at the
time I am writing this article. If you like, you can request a copy of
the software: go to http://csrc.nist.gov/groups/SNS/acts/documents/
comparison-report.html as previously reported[3].
In the following we describe a set of tools called Combinatorial Test-
ing Tools (executable under Windows, but, if needed, not difficult to
port under Unix/Linux) that enables the (quasi)minimal test set to
be extracted and the coverage of a generic test set to be calculated,
using systematic algorithms calculation coverage and starting from
all n-tuples of the variables values. Such algorithms should be cat-
egorized as brute force algorithms and should be used (on a normal
supermarket-bought PC) if the number of variables and values is not
too high.
Overview of CTT
Tools in the Combinatorial Testing Tools product try to provide support
to help solve both problems of combinatorial testing previously stated.
Combinatorial Testing Tools do not intend to compete with the ex-
isting tools aimed at solving the direct problem of combinatorial
testing, such as Microsoft PICT, AllPair J. Bach, NIST, or several other
commercial and non-commercial tools already present on the mar-
ket. These tools implement algorithms definitely more effectively
than CTT and therefore should be favorite I repeat to solve the
direct problem.
Regarding the reverse problem of combinatorial tests, to my knowledge
there are no tools on the market (except NIST CCM in alpha release)
and the CTT then attempt to provide a very first solution to the reverse
problem, to be surely improved over time, when we will better under-
stand the logic and the rules that are behind combinatorial testing and
the minimal determination of test sets related to it.
We would like to remember that:
a. Solving the direct problem means determining the smallest
possible test set with a level of WISE coverage agreed (usually
WISE=2) from the set of variables and values.
b. Solving the reverse problem means determining the level of
coverage of a given test set with respect to a reference WISE level
(also here usually WISE=2)
The tools should be categorized as follows:
Testing Experience 28/2014 27
 a. First level tools: batch DOS scripts providing an immediate re- First Level Tools Batch DOS Script
sponse to standard scenarios that typically occur in test projects
requiring combinatorial techniques. Tool runW
b. Second level tools: executable (C++/Perl development language) This is the first tool that is mandatory to run before all the other tools.
and more versatile, giving response to problems that may be less It generates all the n-tuples corresponding to the value of the Wise
common, but sometimes occur in test projects requiring combi- past input (runW = run Wise)
natorial techniques.

Tool runCC and runsCC

First level scripts were thought of as the wrapper around the second
level executables, in order to simplify end user life with a set of Computes the input test set coverage in respect of the WISE passed as
simple commands that enable you to quickly get a number of items of input (runCC = run Circulate Coverage)
standard information. The following table maps the two categories
of tools and the tool with the kind of information it supplies.
Tools runT and runsT
In this article we cannot go in details on the behavior of the tools; what
Get the minimal test set with guaranteed coverage equal to the Wise
follows is a short description of all of the first level tools. More infor-
passed in input (runT = run Test), extracting the test cases from the
mation can be found in the tools user manual which is downloadable
file of the WISE_MAX-tuples (all combinations).
(with the scripts) from: http://www.opensource-combinatorial-soft-
test-tools.net/index.php/download
Tools runTS and runsTS
Anyway, here is a useful summary table that links together first level
and second level tools. Get the minimal test set with guaranteed coverage equal to the Wise
passed in input or equal to the coverage of the test set passed in input
if less than WISE, extracting the test cases from the file of the test set
Second Level
First Level passed in input (runTS = run Test Set)
Tools
1 2 3 4 5 6 7 8 9 10 11 12 13

runW Tools runTSF and runsTSF

runCC Get the minimal test set with guaranteed coverage equal to the Wise
passed in input or equal to the coverage of the test set passed in input
runsCC
if less than WISE, extracting the test cases from the file of the test set
runT passed in input, excluding n-tuples already covered by the partial test
set input file (runTSF = run Test Set Forbidden).
runsT

runTS
Tool runR
runsTS
Extracts a non-minimal test set but still smaller than the maximum
runTSF test set with guaranteed coverage equal to the Wise passed as input
runsTSF (runR = run Reduce).

runC
Tool runC
runR
Applies constraints to n-tuple file (or test set file) passed as input.
Table 7. Mapping of First Level vs. Second Level Tools (Wrapping Map)

Below is the list of second level executables: Second Level Tools Executable
1. calcolaCopertura.exe In the following we describes the second level tools, a little more hard
2. calcolaCoperturaSlow.exe to use but more versatile; may be useful to experienced users for
3. Combinazioni_n_k.exe managing more complex scenarios.
4. contrLimWise.exe
5. count_line.exe
Executable calcolacopertura.exe and
6. creaFileProdCart.exe
calcolaCoperturaSlow.exe
7. creaStringa.exe
8. generaTestSet.exe Performs the coverage calculation of the input test set in respect of
9. generaTestSetSlow.exe the input WISE.
10. ProdCart.exe
11. reduceNple.exe
Executable Combinazioni_n_k
12. runConstrains.pl
13. uniqueRowFile.exe Extracts all K by K combinations of a string of length N passed as input.

28 Testing Experience 28/2014

Executable generaTestSet.exe and variable with a maximum number of values, while N-wise coverage
generaTestSetSlow.exe coincides with all the variable combinations of the value. And what

if we also include the outputs? This could be the material for another
Gets the minimal test set with guaranteed coverage equal to the Wise
article in the future
passed in input or equal to the coverage of the test set passed in input
if less than WISE, extracting the test cases from the file of the test set
passed in input, excluding n-tuples already covered by the partial
Notes of Appreciation
test set input file.
[1] Many thanks to Jacek Czerwonka, owner of the web site
www.pairwise.org, who allowed me to reprint the incipit of the
Executable ProdCart.exe
same. By the way, he wrote to me about some evolution on the subject
Generates all possible combinations of the values of variables as de- of pairwise vs. random testing that you can find in the article An Em-
fined in the input file. pirical Comparison of Combinatorial and Random Testing available at
the link: http://www.nist.gov/customcf/get_pdf.cfm?pub_id=915439
written by: Laleh Sh. Ghandehari, Jacek Czerwonka, Yu Lei, Soheil
Executable reduceNple.exe
Shafiee, Raghu Kacker, and Richard Kuhn.
Squeezes as many n-tuples as possible contained in the input file,
[2] Many thanks to James Bach, owner of the website
replacing the values * with specific values of the variables and thus
www.satisfice.com who allowed me to reprint part of the article
creating a test set from the file of n-tuples. While not the test set
Pairwise Testing: A Best Practice That Is Not from James Bach, Pat-
minimum, it is reduced compared to the test set maximum (coincident
rick J. Schroeder, available from http://www.testingeducation.org/
with all n-tuples). The number of records depends on the sorting of the
wtst5/PairwisePNSQC2004.pdf.
n-tuples input file, in an unknown way. There is definitely a sorting of
the files row to which the test set output contains a minimum number [3] Many thanks to Dr. Richard Kuhn from NIST who kindly sent me a
of test cases with guaranteed WISE-coverage, but finding this sort is copy of the NIST CCM tools. We would like to remind you that you
not feasible from a computational point of view, as it is too onerous. should request a copy: go to http://csrc.nist.gov/groups/SNS/acts/
documents/comparison-report.html as previously reported.
There are six other executables that do not provide direct support to
the generation and/or operation of the test ssets, but are predomi-
nantly used by DOS batch tools to perform secondary operations that
it is impossible or at least very complex to do directly from DOS. > about the author
These utilities may also be of some use, even if they are not to be con-
sidered tout cours test tools. We have not described those utilities Danilo Berta graduated in Physics at Turin Uni-
in this article. versity (Italy) in 1993 and first started work as a
Laboratory Technician, but he soon switched to
the software field as an employee of a consul-
Conclusion tancy company working for a large number of
customers. Throughout his career, he has worked
In the article we gave an overview of a test methodology that uses
in banks as a Cobol developer and analyst, for
combinatorial calculus to find test cases for a software component,
the Italian treasury, the Italian railways (structural and functional
knowing the inputs of the same. Generally speaking a combinatorial
software testing), as well as for an Italian automotive company, the
technique like this generates too many test cases, so we need to define
European Space Agency (creation and testing of scientific data files
a so-called N-wise coverage (with N from 1 to the number of input
for the SMART mission), telecommunication companies, the na-
variables), select a value of N (usually N=2, pairwise testing) and ex-
tional Italian television company, Italian editorial companies, and
tract a subsystem of test cases with the guarantee of N-wise coverage.
more. This involved work on different kinds of automation test proj-
This is the Direct Combinatorial Test Problem and there are a lot ects (HP Tools), software analysis, and development projects. With
of wonderful tools on the market that solve the problem very quickly. his passion for software development and analysis which is not
just limited to testing he has written some articles and a software
We then dealt with the Reverse Combinatorial Test Problem: if you
course for Italian specialist magazines. His work enables him to deal
have a test set build upon the N inputs of a software component
mainly with software testing; he holds both the ISEB/ISTQB Founda-
about which you know nothing, what percentage of N-wise coverage
tion Certificate in Software Testing and the ISEB Practitioner Cer-
does the test set ensure? On the market I just found one tool that
tificate; he is a regular member of the British Computer Society.
addresses this problem: NIST CCM, which is still in alpha phase at the
Currently, he lives in Switzerland and works for a Swiss company.
time I am writing this article. In the article I give an overview of the
Website: www.bertadanilo.name
CTT (Combinatorial Testing Tools) I developed in C++ that, using a
brute-force approach, try to give a very first response to the Reverse
Combinatorial Test Problem.

But there is (at least) a problem whose solution is not still known. For a
software with N inputs, what is the minimum test set (if it exists) that
guarantees the N-level coverage? The solution exists just for a trivial
case: for 1-wise coverage is always equal to the number of values of the

Testing Experience 28/2014 29

 By Gregory Solovey & Phil Gillis

A Continuous Integration Test Framework

Introduction do not have them at all, some teams care about particular resource
details, and some do not know these details.
Continuous Integration (CI) is a software development practice where
developers integrate their work frequently, usually daily, directly on To satisfy these contradictory conditions the following approach was
the projects main stream. CI consists of two activities: build and test. created:
In this article we describe the test aspects of CI. Continuous Integra-
A resource has a name, a pool it can belong to, selection attributes, and
tion Test Framework (CITF) is the Alcatel-Lucent implementation for
ownership attributes. The selection attributes enable resources to be
testing 4G wireless products, developed over several years. We are
identified in order be selected at the execution stage. The ownership
not offering or promoting this particular system, but rather describe
attributes, if defined, assign the resource to a user or group of users.
solutions that can be used to build or select similar CITFs.

Before there was CI, there was test automation. Numerous groups, Attr 1
working on the same large project in several locations, developed their Search
Attr 2
own automation methodology and tools. Unit testing was done by attrs

developers by adding test code; integration and system testers used Attr 3
name
a variety of CLI, GUI, SNMP, and load and stress test tools.
Resource
After the adoption of the CI paradigm, it soon became apparent that pollID
reusing the various independent test solutions at the mainstream level userID
Ownership
posed a challenge. The teams responsible for the build verification attrs
groupID
had to know the specifics of all tools, be able to debug and interpret
their results promptly, and manually build all the permutations of Figure 1. Resource attributes: name, pollID, search set, ownership set

test environments.
A test environment is presented as a set of resource placeholders
We looked at several existing CI test systems, but none seemed suit-
that should be filled with resources when the test task is issued. Each
able for our situation of testing many configurations of embedded
placeholder is defined as a set of search attributes.
systems. We decided to build our own framework using MySQL and a
web interface, with the following goals:
Attr 1

Manage and allocate pools of resources, grouped by configuration. PH 1 Attr 2

Have the ability to dynamically assemble multiple test environ- Attr 3
Test Env
ments for each build, release, and application.
Attr 4
Create testware standards for test tools, or wrappers for all test PH 2
Attr 5
tools, to make them look alike (use the same configuration data
and convert their results to the standard hierarchical CITF format). Figure 2. Test Environment as a set of resource placeholders and respective sets of their

Provide common interfaces for debugging and reporting the build search attributes

status.
A test task description refers to one test environment and describes
Design a quick and easy way to define new software releases and
the ways to select resource(s) for environment placeholders. There are
projects, and integrate new test tools, testware, resource pools,
three ways of finding a resource for a placeholder: by resource name,
and test environments.
from a pool, or from the resources that satisfy the search attributes.
Intelligently select the appropriate test suites (sanity, regression, However, for each resource placeholder only one way should be used.
feature) whenever a build is completed to validate the integrity of
the mainstream, existing, and new functionality.
pool
PH 1
34
Attr 1
1. Resource Management Search
Test Task PH 2 Attr 2
attr
Embedded systems require comprehensive test environments that
Attr 3
include variety of test tools, network elements, load simulators, and a name
PH 3
large selection of end devices. Each build has to be tested on multiple abc23
test environments and resource permutations. It is impossible to have
a dedicated test environment for each release/project/feature. On the Figure 3. Test task as a set of resource placeholders and the respective rules of their

contrary, some teams do not want to share their resources, some teams replacements

30 Testing Experience 28/2014

Every time a build is done, a new instance of a test task (testTaskExe)
is issued. Based on its test task description, all placeholders need to
be filled with real resources. The CITF performs a two stage selection
procedure:
Stage 1: Select available resource candidates based on a selection
method (name, pool, attributes). Resource is considered available if it
is not under test and if it is in an operational state.
Stage 2: From these candidates, select the permitted resources for the
user who issued the request to test. The resource is permitted to be
used if one of three conditions is true:
1. The resource user and group ownerships are not defined
2. If only group ownership is defined, the user should belong to this
group
A use case (UC) represents a subsystem from a behavioral (functional)
point of view and is related to a specific requirement (scenario, algo-
rithm, or feature). A UC consists of test cases.
A test case (TC) is a single verification act. A TC moves the object-to-test
from an initial to a verification state, compares the real and expected
results, and returns it back to its initial state. In most cases, returning
the system to its initial state makes the TCs independent of each other.
A TC is a series of test actions to move the object-to-test through the
above phases (set, execution, verification, reset).
A test action (TA) is a single act of interaction between a test tool and
the object-to-test. It supports the object-to-tests interfaces (CLI, GUI,
SNTP, HTTP). For example, a test action can be the execution of a single
CLI command, a single interaction with the GUI, or sending a single
http request transaction to the client.

3. If user ownership is defined, it should match the user ID of the

TA_1
person who issued the test
TC_1 TA_2

TA_3
Resource 1
TA_1
Test Task Exe
UC_1 TC_2
Resource 2 TA_4
Attr 3
TA_2
Figure 4. Test Task Exe as a set of given resources TC_3
TA_4

A test task is started upon acquiring all the required resources. The TS_1 TA_2
states of its resources will be changed to running. The test task will TC_3
TA_4
start executing component by component, and each component knows UC_2
how to build an environment from the selected resources. Upon com- TC_4 TA_3
pleting the execution of a task, the resources are returned to the avail- TA_2
able state. If the returned resource is unhealthy, recovery mechanisms Test UC_3 TC_3
TA_4
restore it to its initial state, making it ready for subsequent test runs.
TA_1

2. Testware Management
Continuous integration deals with code before the production stage.
This means the testware needs to be adaptive to frequent changes of
many kinds, such as API and command syntax and semantic changes,
and changes in requirements. When these changes have occurred,
hardcoded syntax in testware will be hard to find and correct. The only
way to achieve testware maintenance is by providing a strict relation-
ship between architecture, requirements, and design documents, and
by separating business functionality from implementation details.
2.1 Testware Hierarchy
A hierarchy of testware should reflect the architecture, requirements,
and design documents that describe the object-to-test from the struc-
tural and behavioral views down to the implementation details. The
latter is described below (top-down):
A test (sanity, regression, functional, performance, etc) is a collection
of test sets.
A test set (TS) reflects the structural view of the object-to-test, as de-
scribed by the system architecture. Examples include: a set of hardware
components, a set of services, or a set of network configurations. A TS
is a grouping of use cases.
TC_1 TA_2
TA_3
TS_2 UC_4
TA_2
TC_3
TA_4
Figure 5. Testware Hierarchy
The hierarchical testware presentation is materialized in unconditional
execution of the test cases: all test sets are executed sequentially; all
use cases inside each test set are executed sequentially; all test cases
related to each use case are executed sequentially.
2.2 Testware External Presentation
The testware has to be updated as a result of changing the business
rules, the API syntax, or the GUI appearance. To minimize the number
of changes, the testware is organized externally into configuration
files, test set files, test scripts, and test case libraries. Such presentation
separates the implementation details from the business functions,
making the testware independent of the environment. The test objects
are reusable across releases, projects, and test types.
The relationships between the internal and external presentations
are described below:

Testing Experience 28/2014 31

 A configuration file contains a list of TSs that have to be executed. Each Before starting its tests, each wrapper dynamically creates its configu-
TS in a configuration file points to a test set. ration files, which include references to selected tests, resources to be
used, and the location of the results. The configuration files are built
A test set is a file containing a collection of test script names.
from templates, based on the release, project, and resource parameters.
A test script contains one or more UC descriptions; each of them com- A debug file that is created during a test execution needs to follow a
bines the TC calls. The actual TC descriptions are stored in a test library. standard organization. Upon completion of a test, the results produced
by different test tools are converted into a standard hierarchical format
A test library contains the description of TCs as a sequence of TAs. In
and uploaded to a results repository together with logs and traces.
this manner, various UCs can reuse the same TCs from the test libraries.
The debug file should have a common look for all test tools, with an
A test action is a single act of communication with the interface of the
emphasis on what are the stimuli and responses, and how the com-
object-to-test. It is presented as action words: set, send, push, capture,
parison was made in order to let the tester write CRs that communicate
compare, repeat, etc. The TA parameters are CLI commands or GUI ob-
the problem precisely to the developer.
ject methods and implemented by the language of a specific test tool.
To find a single result from the tens of thousands of test runs per day
UC_1 TC_1 with a few mouse clicks, the results repository area needs to mirror
TC_1 TA_1
TC_2 TA_2 (Figure 4):
Script_1
TC_3 TA_3 TA_1
TS_2 Script_2 UC_2 TC_2 The build infrastructure, as a set of various release/project/
TC_3 TA_1 TA_2 feature/developer streams
TC_4 TA_4
TA_3 The build test infrastructure as a set of test environments and
TS_2 UC_3
TC_3 TC_3 applications to test
Script_3 TA_2 TA_4

UC_4
TA_4 The test structure as a tree of test sets, use cases, and test cases
TC_1
TC_4
TC_3 TA_3
Failed test cases can be filtered for known issues in order to avoid re-
Configu- Test set files: Test scripts: Test libraries: Test
analyzing expected failures. The standard results presentation format
ration file per TS UC(s)/TCs TC definitions actions
supports five levels of test hierarchy, which are presented on the web.
Figure 6. Testware external presentation

The configuration, test set, test script, and library are files that pres-
ent testware. The narrow specialization of each file type serves a
maintenance purpose: a single change in the code of the object-to-test
(on the structural, functional, or syntax level) should lead to a single
change in testware.
4. Test Process
New releases/projects/features, test environments, and testware have
to be created daily. This requires interfaces that work with test related
objects: to create, to edit, to delete, to run, to monitor, and to report.

3. Test Tool Management: Conform or Wrap
Embedded systems applications demand a variety of test tools and
various independent test solutions. It is unreasonable to expect that
the teams responsible for build verification know the specifics of all
tools and are able to debug and interpret their results promptly.
The challenge is to make all test tools (CLI, GUI, load and performance)
look alike, and appear transparent to the tester. The solution is to
require the testware framework to be followed or to create a wrap-
per for each test tool that will serve as an interface between the test
framework and test tools.
The management web interface has to provide a quick and easy way
to define new software releases and projects, and integrate new test
tools, testware, resource pools, and test environments directly into
the database. The build server requests a test for a new build through
an execution interface and is notified of test results and metrics.
The request to run a test is called a verification request (Figure 6). A
verification request specifies a set of test tasks that are independent
and can execute in parallel. Each test task (task for short) specifies a
set of components, which are sets of tests to run sequentially, and is
associated with a test environment, which specifies a set of resources
that must be acquired before execution.

A subdirectory structure that A subdirectory structure that A result file content that
mirrors the build structure mirrors the build test structure mirrors the test structure
Build 1 Test envi- Application 1 Test case 1
Project 1 Use case 1
Build 2 ronment 1 Application 2 Test case 2
Release 1 Test type 1 Test set 1
Build 3 Test envi- Application 3 Test case 3
Project 2 Use case 2
Build 4 ronment 2 Application 4 Test case 4
Results
Build 5 Test envi- Application 5 Test case 5
Project 3 Use case 3
Build 6 ronment 3 Application 6 Test case 6
Release 2 Test type 2 Test set 2
Build 7 Test envi- Application 7 Test case 7
Project 4 Use case 4
Build 8 ronment 4 Application 8 Test case 8

Figure 7. Test results location

32 Testing Experience 28/2014

Each component calls its test tool to start a test. A test task monitors
all components and, based on the database settings, can terminate
it if the execution goes on too long, repeat its execution, or skip the
execution of the subsequent components.
Test Environment 1
Comp 1
Verification Test Task 1 Comp 1 Comp 2
Request 1
Test Task 2
Test Environment 2
Comp 1
Test Task 4
Verification
Request 2 Comp 1 Comp 2
Test Task 5
Test Task 6
Comp 1 Comp 4
Verification
Request 3 Test Task 7
Test Environment 3
Comp 5 Comp 3
Figure 8. A verification request is translated into independent test tasks; each task
executes its components sequentially in its associated test environment.
5. Reliability
A considerable number of failed test cases are not real code errors, but
are environment issues, such as random network glitches, testware
mis-synchronization, or resource failures. The following built-in testing
reliability features help test teams handle such errors:
Starting test cases from an initial state and, in case of failure, a
resource is returned to its initial state by the test cases recovery
sequence.
Using filters to mask expected test failures in case of known
problems.
Automatically rerunning some software components in case of
intermittent failure during test execution.
Recovering a hardware resource from a failure state when the
resource is returned to the pool after a failed test execution.
Manually updating test results after a tester reruns failed test
cases and reporting the build status promptly.
Maintaining redundancy of database and web servers, along with
frequent backups for quick restoration of functionality in the
event of a system outage or database corruption.
6. Metrics
Metrics capture a snapshot of the quality of each build. The primary
metrics are test object pass/fail result counts and test execution times.
These are calculated at the test tool layer.
The second category of metrics is failure reasons, which are used to
identify bottlenecks in the CI process. Sometimes failures unrelated to
the code can occur, such as network glitches, database errors, testware
mis-synchronization, etc. These data, collected and analyzed over time,
can identify areas of CITF or the environment that should be improved.
The third category of metrics is coverage: feature, requirements, and
code coverage. The percentage of code that was touched by tests does
not prove that a test is complete (i.e. covers all possible errors) but rather
reveals areas of the code that were not tested at all. The code coverage
metrics can be useful in CI if the build consists of many independent
layers and modules. An objects quality can be defined as the quality
of the weakest link. Therefore, it is good practice to request the same
percentage of code coverage for all system components. This is espe-
cially important for new code deliveries, since it is the primary proof
(along with requirements traceability) that the necessary automated
tests were added along with the new code.
The fourth category of metrics describes the quality of each build: code
review data, code complexity, warnings, and memory leaks.
Comp 3
Comp 3 7. Conclusions
CI puts heavy demands on testing systems. We found that no com-
mercial solution offered the functionality we needed, which is why we
Comp 6 developed our own CI test framework. The development was driven by
the demands of the test teams responsible for build validation, whose
major constraint was the ability to determine the cause of failure in a
short interval. As a result, we deployed five releases of CITF during the
five years we have been in operation. We currently support four major
releases, approximately thirty projects for each release, ten different
test tools (commercial and in-house), and twenty embedded systems
applications. We support a pool consisting of hundreds of geographi-
cally distributed physical resources. The system verifies tens of builds
daily, by running hundreds of thousands of test cases.
> about the authors
Gregory Solovey is a Distinguished Member of
Technical Staff at Alcatel-Lucent. He has a PhD
in Computer Science (thesis Error Identifica-
tion Methods for High Scale Hierarchical Sys-
tems), more than 25 years of experience, and
over 60 publications and patents in the fields of
test design and automation. He has extensive
experience in establishing a test strategy and process, designing
production-grade test tools, automating the test design, embedding
automation test frameworks, and implementing continuous integra-
tion. He is currently applying and enhancing his approaches to de-
velop a test framework for continuous integration of a hierarchical
embedded system.
LinkedIn: www.linkedin.com/profile/view?id=44354302
Philip Gillis is a Distinguished Member of Tech-
nical Staff at Alcatel-Lucent in Murray Hill, NJ. He
has more than 30 years experience in software
development, including 20 in database design
and administration, and 15 in test automation
systems. Starting with C and Unix, he moved to
Windows programming in the 1990s and to Linux
servers and web browser programming in the late 2000s. He now
designs, implements, and administers a test automation system
using PHP, Perl, JavaScript with jQuery, and MySQL. He holds five US
patents and multiple international patents.
LinkedIn: www.linkedin.com/profile/view?id=5523965
Testing Experience 28/2014 33
 By Robert Galen
The Three Pillars of
Agile Quality and Testing
Introduction
A few years ago, I entered an organization to do some agile-focused
coaching and training. From the outside looking in, it was a fairly
mature agile organization. They had internal coaches in place, had
implemented Scrum while also leveraging Extreme Programming
technical practices for a couple of years, and seemed to be fairly com-
mitted and rigorous in their application of the methods.
It was a global financial firm that delivered their IT projects via highly
distributed teams. There were a wide variety of tools in place for ap-
plication lifecycle management (ALM), software development, and
software testing support. In my first few days, everything I encoun-
tered, while albeit in superficial detail, just felt like a mature agile
organization and I was pleasantly surprised. Heck, I was impressed!
For the purposes of this article, my observations will shift to being
quality, testing, and tester centric.
Too Narrow a Focus
One of the things I noticed is that the firm had gone all in on Behavior-
Driven Development (BDD) leveraging Cucumber. They had invited in
several consultants to teach courses to many of their Scrum teams and,
as a result, everyone was excited and test infected. Teams were liter-
ally creating thousands of BDD-level automated tests in conjunction
with delivery of their software. From their teams perspective, there
was incredible energy and enthusiasm. Literally everyone contributed
tests and they measured the number of increasing tests daily.
These tests were executed as part of their continuous integration
environment, so there were visible indicators of coverage and counts.
It was truly visual and very inspirational. And I could tell that everyone
was focused on increasing the number of automated tests so there
was a unity in goals and in each teams focus.
However, a few days into my coaching, I was invited to a product
backlog refinement session where a team was writing and develop-
ing their user stories. What I expected was to simply be an observer.
What actually happened is that I soon realized that the team did not
know how to write a solid user story. They could barely write one at
all, which shocked me. After this gap became clear, they asked me to
deliver an ad-hoc user story writing class for them. Afterwards, the
team was incredibly appreciative and everyone seemed to get the
place that stories held in developing their BDD-based acceptance tests.
Over the next several days, I started to realize something very im-
portant. The organization was at two levels when it came to its agile
quality and testing practices. People were either all in or they were
unaware or under-practicing specific techniques. For example, they
were all in on BDD and writing automated BDD (Cucumber) tests
and on continuous integration. However, they struggled mightily
with writing basic user stories and literally had no clear or consistent
Definition-of-Done.
34 Testing Experience 28/2014
I also realized that this seesaw effect of focusing on a handful of the
more technical practices was doing them a disservice. Why? Because
it is actually the interplay across practices that influences the effec-
tiveness of your agile testing and the product impact of your quality
practices. I prepared a model for them to illustrate the balance that I
think is critical in agile quality practices.
I called it The Three Pillars of Agile Quality and Testing, and I began to
coach a much more nuanced, broad, and deep approach for their teams.
While this is more of a strategic play and a longer-term focus, the
discussions and the changes they drove had an immediate impact
on the organization. People started to connect the dots between the
technical and softer skill practices. They became much more aware
of the interplay across practices and how this drove an improvement
in quality results.
I want to share the Pillars in this article in the hope that it will help
your agile quality strategy development as well.
A Bit More on Strategy
A really important sub-text to the development of the Three Pillars is
my ongoing observation that agile organizations at best might have a
strategy for their overall transformation. But rarely do they develop or
are able to articulate their overall strategy for agile quality and testing.
I guess the assumption is that these aspects are along for the ride as
part of the development-driven strategies in their agile adoption. But
I could not disagree more with that point of view. I believe that calling
out, making transparent, and focusing on your quality and testing
agile strategies strongly aligns with the Agile Manifesto and makes
for a much more rigorous and successful agile transition.
The ultimate value of the Three Pillars is to help organizations think
about, articulate, and realize their agile strategies in these areas and,
most importantly, to hopefully achieve evolutionary balance.
The Three Pillars of Agile Quality and Testing
As I have said, the driving force behind my creating the Three Pillars
was organizational quality imbalance. As I observed what was hap-
pening at my client, I clearly recognized the imbalance. However, it
was unclear to me how to create a model that would help them. I
eventually came upon the following three critical areas, or Pillars,
where I tried to categorize crucial tactics, strategies, and techniques
that I have found helpful to agile teams as they create a broad and deep
supportive structure for their product quality and testing activities.
Here are the pillars at a high level:
1. Development and Test Automation: This pillar is the technology
side of quality and testing, and it is not simply focused on testing
and testers. It includes tooling, execution of the automation test
 pyramid, continuous integration, XP technical practices, and sup- and typical lack of value. But I AM talking about effective profes-
port for ALM-distributed collaboration tools. sional testing, broadly and deeply applied within agile contexts.

Often it is the place towards which organizations gravitate first

3. Cross-Functional Team Practices: Finally, this pillar is focused on
probably because of our generic affinity for tools solving all of our
cross-team collaboration, team-based standards, quality attitudes,
challenges. An important way to think about this pillar is that it is
and, importantly, on building things properly. Consider this the soft-
foundational, in that the other two pillars are built on top of the
skills area of the three pillars, where we provide direction for how
tooling. And organizations often underestimate the importance,
each team will operate consider them the rules of engagement.
initial cost, and ongoing costs of maintaining foundational agility
in this pillar. Continuous investment is an ongoing challenge here. For example, this is the place where good old-fashioned reviews and
inspections are valued. This would include pairing (across ALL team
Finally, this pillar is not centric to the testing function or group. While
members), but also slightly more formal reviews of architecture, de-
it includes testing, tooling, and automation, it inherently includes
sign, code, and test cases. It is a place where inspection is performed
ALL tooling related to product development across the entire agile
rigorously, as established in the teams Definition-of-Done. Where
organization. It provides much of the glue in cross-connecting
refactoring of the code base and keeping it well kept is also of
tools and automation towards efficiency and quality.
primary importance.

2. Software Testing: This pillar is focused on the profession of testing.
On solid testing practices, not simply agile testing practices, but
leveraging the teams past testing experience, skills, techniques,
and tools. This is the place where agile teams move from a trivial
view of agile software testing (which only looks at TDD, ATDD, and
developer-based testing) towards a more holistic view of quality.
It is a pillar where the breadth and depth of functional and non-
functional testing is embraced. Where exploratory testing is un-
derstood and practiced as a viable testing technique. It is where
the breadth of non-functional testing is understood and applied
to meet business and domain needs, including performance, load,
security, and customer usability testing.
By definition, this is where testing strategy resides, where planning
and governance sit, and where broad reporting is performed. I am
NOT talking about traditional testing with all of its process focus
Speaking of Definition-of-Done, this is the pillar where cross-team
physical constraints, conventions, and agreements are established.
But, more importantly than creating them, it is where the team
makes commitments to consistency and actually holding to their
agreements. Another important focus is on group integrity in con-
ducting powerful retrospectives and fostering continuous improve-
ment in the long term.
Foundational Practices
But beneath the Three Pillars are some foundational principles and
practices that glue everything together. For example, taking a whole-
team view of quality and testing, where it is not just the job of the
testers, but of everyone on the team. I still find far too many agile
teams that relegate the ownership of quality and testing only to testers.

Pillars of Agile Quality

Development & Cross-Functional

Software Testing
Test Automation Team Practices
Pyramid-based Strategy: Risk-based testing:
(Unit + Cucumber + Functional & Team-based Pairing
Selenium) Non-Functional Stop-the-Line Mindset
Continuous Integration Test planning @ Release Code Reviews &
& Sprint levels Standards
Attack technical
infrastructure in the Exploratory Testing Active Done-Ness
Backlog
Standards checklists, Aggressive Refactoring
Visual Feedback templates, repositories of Technical Debt
Dashboards
Balance across manual, User Stories, 3 Amigo
Actively practice ATDD exploratory & based Conversations
and BDD automation

Whole Team Ownership of Quality

Knowing the Right Thing to Build; And Building it Right

Healthy Agile Centric Metrics

Steering via: Center of Excellence or Community of Practice

Strategic balance across 3 Pillars; Assessment, Recalibration, and Continuous Improvement

Figure 1. High-level View of the Three Pillars

Testing Experience 28/2014 35

 Continuously challenging this position and coaching the teams toward Wrapping Up
whole-team ownership is an ongoing focus.
This initial article is intended to introduce The Three Pillars of Agile
Another foundational area is metrics. We all know that what you Quality and Testing as a framework or model for solidifying your agile
measure drives the behavior of the team. As we move towards agil- quality strategies and initiatives.
ity, we need to begin measuring the entire team results rather than
I hope it has increased your thinking around the importance of devel-
functional results, and even showing care when measuring the team,
oping a balanced quality and testing strategy as part of your overall
such as understanding that velocity is probably not the best metric to
agile transformation. As I have observed and learned, it does not simply
measure a healthy team.
happen as a result of going agile, and most of the teams I encounter
One of the core components of the Three Pillars foundation is a thread are largely out of balance in their adoption.
that permeates through the pillars and the foundation. It embraces the
I hope you find the model useful and please send me feedback. If there
core idea that each agile, self-directed team has a basic responsibility
is interest, I may explore more specific dynamics within each pillar in
to build the right things (customer value) and to build them properly
subsequent articles.
(design, construction integrity, and quality).
Stay agile my friends,
Figure 1 is an overview of the types of activity and focus within each
Bob.
pillar. This is a high-level view and there are important nuances that
are missing mostly due to a lack of space.

BTW: I am writing a book based on the Three Pillars model. If you would
like to have more information, participate in the books evolution, or
Cross-cutting Strategy
simply stay tuned in to this topic, please join the books mailing list
Beyond the individual pillars, the value resides in cross-cutting con- here http://goo.gl/ORcxbE
cerns. I will go back to my original story to help make this point. My
client was advanced in BDD practices, but struggling with user story
writing, or even understanding the point of a user story.
> about the author
The results would have been better if they had made the following
cross-Pillar connections:
Bob Galen is President & Certified Scrum Coach
(CSC) at RGCG, LLC a technical consultancy fo-
In Pillar #1 Behavior-Driven Development (BDD) and Acceptance
cused towards increasing agility and pragma-
Test-Driven Development (ATDD) are mostly technical practices. They
tism within software projects and teams. He has
focus on articulating user story acceptance testing in such a way
over 30 years of experience as a software devel-
as to make them automatable via a variety of open source tooling.
oper, tester, project manager and leader. Bob
Unfortunately they have an underlying assumption that you under-
regularly consults, writes, and is a popular speak-
stand how to write a solid story.
er on a wide variety of software topics. He is also the author of the
In Pillar #2 One thing I did not mention in the story was that every books: Agile Reflections and Scrum Product Ownership. He can
team had a different view of what a story should look like and the be reached at: bob@rgalen.com
rules for writing effective stories. There were no norms, consistency Twitter: @bobgalen
rules, templates, or even solid examples. A focus on the software
testing aspects of pillar two would have established these practices,
which would have significantly helped their teams.

In Pillar #3 An important aspect of the user story that my client

failed to realize was the conversation part of the story. If you refer-
ence the 3-Cs of effective story writing as a model, one of the Cs is
having a conversation about or collaborating on the story. It is the
most important C if you ask me. It is where the 3 Amigos of the
story (the developer(s), the tester(s), and the product owner(s)) get
together; leveraging the story to create conversations that surround
the customer problem they are trying to solve.

Do you see the pattern in this case?

You cannot effectively manage to deliver on agile quality practices

without cross-cutting the concerns and focus. In this case, effective
use of user stories and standards, plus BDD and automation, plus the
conversations needed to cross all three pillars. It requires a strategic
balance in order to implement any one of the practices properly.

I hope this example illustrates the cross-cutting nature for effective use
of the Three Pillars, and that you start doing that on your own as well.

36 Testing Experience 28/2014

By Cesario Ramos & Pascal Dufour
Serious Games for Testers
Introduction
The use of ATDD and tools like FitNesse, Cucumber, and Robot Frame-
work makes it necessary to create automated acceptance tests. These
acceptance tests are a natural extension of the acceptance criteria used
in user stories. You use acceptance tests to understand what needs
to be developed so that you can develop the correct functionality. In
order to develop the correct functionality, you need to create a common
understanding of stakeholder value among all team members. You
use story workshops (product backlog refinement meetings in Scrum)
so that everyone can contribute to discovering the whys, hows, and
whats of the stories. You also break down bigger stories (a.k.a. epics)
into smaller stories (a.k.a. user stories), and create new user stories
together with your stakeholders in these workshops.
Creating shared understanding as a team with your stakeholders gives
you the following benefits:
1. You are more likely to create a solution that really delivers business
value. The close collaboration makes you understand the expected
outcomes. You understand why this functionality needs to be cre-
ated and what business value it addresses.
2. You are better equipped to develop the correct solution. You focus
on the solution only after you have understood the needs and prob-
lems from the customers point of view. Without this understand-
ing, agile just helps you create the wrong stuff faster.
3. Your mindset and approach shift from finding defects to preventing
them at the level of business value and development. Remember
that finding defects is a waste from a lean point of view.
4. You are able to write down in unambiguous acceptance tests what
is meant by successfully developing a user story. You develop only
what is needed no more, no less.
Story workshops not only create a shared understanding and accep-
tance test cases, they also open up the opportunity for the whole team
to test. The test cases can be automated, so the developers have work
to do together with the testers. The test cases need to be fleshed out
and extended with new ones, and everybody can do that now.
The common understanding also sets you up for defining exploratory
test charters together with your team. And, as you know, everyone can
execute a manual test as long as an experienced tester is coaching.
That is all very interesting, but how can you actually run a successful
story workshop? What steps are needed, what games can you play,
and how can you facilitate the meeting?
In this article we will tell you about how we do our story workshops
using serious games.
What is a Serious Game?
If you are like us, you have participated in meetings that were boring,
non productive, dominated by a few people, and just a pain to be part
of. You just showed up and said a few things because that was what
management expected you to do. Well, it does not have to be that way.
You can use game mechanics in all your meetings to make them not
only more fun, but much more productive with better results. A way
to successful meetings is through serious games.
A serious game[1] is a game designed to solve a business problem. In a
normal game the purpose is to entertain. In a serious game you use
the game mechanics that engage millions of people playing normal
games all over the world. A serious game puts people into a creative
environment where they are really engaged and can collaborate with
others, moving things around and discussing points of view.
Places where you can use serious games are when understanding
stakeholder needs, clarifying user stories, and distilling test cases. We
do that in a story workshop.
What is a Story Workshop?
An intended outcome of a story workshop is to create a shared under-
standing of user stories for the whole team and to break them down
to the right size. You create a shared understanding by discussing why
they are needed, what problems they solve for the customer, and by
distilling test cases to create that shared understanding.
Outputs of the story workshop for us are:
1. We want to have two or three examples for each user story to be
refined.
2. We want to have an exploratory test charter defined of each user
story to be refined.
3. We want a size estimate in order to make tradeoffs.
4. We want an estimate of the impact the story is going to make on
the intended business value.
A good time slot for a refinement meeting is between one and two
hours. You can always stop the meeting if you finish early, although
this seldom happens because of Parkinsons law[2].
The first problem you address in a story workshop is to better un-
derstand the user stories. A user story is a narrative of a need of a
particular persona. You need to understand what problem and needs
the persona actually has. This is a discovery problem. The question to
be answered is: What problem does the customer have and why does
he/she want it to be solved?
Testing Experience 28/2014 37
 Another problem you address in a story workshop is a design problem. A
user story is also an experiment for which a solution must be designed
by development. Therefore, the question to answer is: What solution
best fits the customers needs? The exact details of the design are not
provided by a story workshop but it does start off the thought process
about a solution.
Finally you have a testing problem. The questions to be answered are:
1. How do we know that the solution solves the customers problem?
Does our solution add more value than his/her current solution?
2. How do we know that we are solving the right problem? Is the prob-
lem we are solving the problem the customer wants to be solved?
3. How do we know we have solved the problem correctly? How do we
quantify the success and failure of the solution?
In a story workshop we try to address all of these questions.
How to Facilitate a Story Workshop
There are a few things you need to take into account if you want a
successful workshop.
First you need to state the goal of the workshop. It is very important
to state the goal at the beginning of the meeting in order for people
to become engaged.
Next you need to discuss the steps of your workshop clearly. So, what
is the agenda and what are we going to do?
38 Testing Experience 28/2014
After that you will have to define the rules of the workshop. How about
ringing phones? Can we read emails during the workshop? What about
interrupting one another while talking? If you have already done some
story workshops with the team, you just quickly remind them of their
own rules and ask if they are still happy with them.
To boost creativity everything is time boxed, so you want to commu-
nicate the time limit.
During the workshop you also want to inform the team of time prog-
ress. You can, for example, let them know every 10 to 15 minutes how
much time is left and discuss whether you are still working on the
most important things.
Finally, a parking lot is very useful to have. If you have any questions
or issues that take up too much time or are not relevant, you can put
them on the parking lot and cover them at the end of the meeting.
A Game Sequence for a Successful Story Workshop
In a story workshop for distilling test cases to create shared under-
standing, you want to perform the following steps:
1. Check in: Explain the goal and agenda of the meeting.
2. Understand the business value: The product owner tells a coher-
ent set of stories with a goal (Sprint Goal if you are using Scrum)
and links them back to the business objective. The team discusses
why we are doing this. We do this using an impact map[3] and
the 5 Whys[4].
3. Understand the customer value: The team breaks up into two [9] Decision tables: http://fitnesse.org/FitNesse.FullReferenceGuide.
sub-teams, and each sub-team gets half of the user stories. The UserGuide.WritingAcceptanceTests.SliM.DecisionTable
sub-teams then create scenarios of the current situation of the
[10] Risk quadrants: http://pascaldufour.wordpress.com/2013/11/19/
persona so that the team understands the personas challenges
user-story-risk-quadrants/
as they are currently. The sub-teams also create scenarios of the
persona with the user stories solution implemented, so that the [11] Testing tours: http://michaeldkelly.com/blog/2005/9/20/touring-
team understands the benefits the persona has from the new solu- heuristic.html and: http://msdn.microsoft.com/en-us/library/
tion. The teams then get together and discuss the scenarios they jj620911.aspx#bkmk_tours
have created with the other subteam. The team discusses Why
does the persona want this? We do this using a storyboard[5]
and a Pain Gain map[6]. This is also the step where you quantify
your goal so you know how much value you have created at the > about the authors
end of the iteration.
Cesario Ramos is the founder of AgiliX Agile Con-
4. Distill acceptance tests: The team then creates acceptance tests
sulting where he works as a professional Agile
for the user stories. Depending on your tools, you create the use
Coach and Scrum trainer @ Scrum.org. He helps
story narratives with Gherkin specifications[7], flow tables[8] and
teams to deliver maximum value within the
decisions tables[9]. The teams break up into sub-teams, and write
constraints of time and budget. Cesario also leads
tables and Gherkin scenarios in collaboration with the product
agile transformation programs at his customers
owner. The sub-teams then get together and discuss the results
through coaching, training and hands-on guid-
with each other. We do this at the whiteboard using tables and
ance of management and development teams. When Cesario is not
scenario writing.
coaching or enjoying family life he enjoys writing and speaking.
5. Define exploratory test charters: Identify risks to target your ex- Cesario is the author of the book EMERGENT Lean Agile adoption
ploratory tests. Once you have identified which stories need manual for an innovation workplace and speaker at international agile con-
testing, you set up an exploratory test charter for each. We do this ferences where he passionately discusses the latest agile topics. You
using a risk impact matrix[10] and we use an exploratory testing can contact Cesario at cesario@agilix.nl.
tour to drive our test[11].

6. Closing: Quick summary of results and final remarks.

The above game sequence is just one way of running a story workshop.
It assumed that you have ready user stories to begin with. The games
explained are the games we use most of the time. There are a lot
more games you can use. We encourage you to try some out in your
workshops to discover which work best in your particular context.
References
[1] Serious game/innovation games:
http://www.innovationgames.com
[2] Parkinsons Law: http://en.wikipedia.org/wiki/Parkinsons_law
[3] Impact Mapping: http://ww.impactmapping.org
Pascal Dufour is a passionate tester and Scrum
coach at Validate-it. Pascal has a passion for Ag-
ile projects, where he tries to implement prag-
matic test strategies combining Agile and con-
text-driven testing. With over ten years of
experience at large international companies,
Pascal has experience with different types of test-
ing from embedded software to system integration. Enthusiastic
and creative, he tries to make testing more fun, making his work
visual and as simple as possible. He helps team members improve
their efforts in Scrum, emphasizing ethics, commitment, and trans-
parency, and motivating people to use a dynamic approach to test-
ing. He believes teams should learn, try, experiment, and work to-
gether to create solutions that solve problems. You can contact
Pascal at pascal@validate-it.nl.

[4] The 5 whys: http://www.gamestorming.com/games-for-problem-

solving/the-5-whys/

[5] Storyboard: http://www.romanpichler.com/blog/agile-scenarios-

and-storyboards/

[6] Pain Gain map: http://www.gamestorming.com/games-for-

design/pain-gain-map/

[7] Gherkin Language: http://cukes.info/gherkin.html

[8] Flow tables: http://fitnesse.org/FitNesse.UserGuide.

FixtureGallery.ImportantConcepts.FlowMode

Testing Experience 28/2014 39

 By Venkat Ramesh Atigadda

Testing the Internet of Things (IoT)

The Future is Here
Introduction Protocol testing Unencrypted network
TCP/IP
The Internet of Things is a scenario in which objects or people are
Wi-Fi
provided with unique identifiers and the ability to transfer data over a Cellular network (3G/4G)
network without requiring human-to-human or human-to-computer
Performance testing Delayed network connection
interaction[1], all these physical objects are connected to one or more Internal computation
sensors via the internet, and each sensor will monitor a specific con- Response time
dition such as temperature, motion, or location. There is huge trans- Embedded testing Timing constraints
formation in the usage of the internet when compared to the past. Software validation and integration
Previously, the internet was used to obtain information, perform
M2M testing SMS vs packet data transmission
transactions, connect to family and friends, and for entertainment, etc., Integration of sub-networks
but today it is also used to connect with physical devices and things, for
Data visualization testing Usability of the device
example a mobile app to locate car keys, a medicine bottle with alerts User experience
on dosage, or temperature control as per climatic changes. Everything
Device testing Device configuration (manual and time-
is connected via the internet for continuous monitoring, and the data based)
is captured and stored in cloud based platforms with a well-defined Alert settings
analytics strategy to support predictive analytics for critical decision Device-to-device integration
making This means there is a significant role for testing these devices
Table 1. Key Verification Aspects
or networks to ensure that quality and standards are maintained.
Below are the reports shared by HP[2], raising significant concerns
regarding user security and privacy:
Business Drivers
80% of devices allowed weak passwords
Business process monitoring Reduced time-to-market
70% did not encrypt data transmissions out to the internet
Rise of smart connected devices Increased customer
satisfaction 60% had cross-site scripting or other flaws in their web interface
Connected network (people, data,
and things) Cost saving 60% did not use encryption when downloading software updates

Development of IP-enabled devices

In light of these reports, there is a greater need for QA services. So in
Table 1 are some of the key testing aspects which need to performed
Key Testing Types as part of IoT testing.

According to a survey by HP[2], 70% of the units in the Internet of

Things segment have serious security flaws, and an average of 25 Additional Verification Aspects
security flaws are observed in each IoT device.
Below are some of the additional verification aspects that can be
considered for testing IoT[3]:
Testing Type Verification Aspects

Security testing Identity authentication

Multi-media auto- Verification of LED flashing and sounds made by the
Password (weak/strong)
mation devices
Vulnerable to XSS
Data leakage Hardware avail- Running verification tests on the hardware directly
Data protection ability without using simulators

Interoperability testing
Conformance testing
Mobile testing
Information exchange between Physical access Verification of the sensors and other hardware inter-
Networks faces during runtime with special access
Components Memory constraints Verification of the system failure due to low RAM
Systems availability
Sensors Behavioral aspects Verification of recompiling embedded code that af-
Smart devices fects the runtime behavior
Gateways
Code security Verification of the embedded system code in the
Mobile apps production version that should not be hacked
Mobile devices
Mobile OS (Android, iOS) Table 2. Additional Verification Aspects

40 Testing Experience 28/2014

Challenges for IoT Testing 3 Are there any security threats in
the usage of the device?
Below are some of the key challenges with regard to IoT testing:
4 Are there any organizational or
Testing multiple combination of radio bands including Bluetooth,
project test processes that must
Wi-Fi and NFC
be adhered to?
Devices with multiple lines of code, which makes life difficult for 5 Has the device been thoroughly
the testers tested with regard to localization/
globalization?
Devices using non-standard communication protocols will require
6 Is the IoT connection tested by as-
significant changes to application testing sessing the performance changes?

Availability of multiple devices and technologies 7 Is the device fixed, mobile, or both
(testing will be different for each)?
Obtaining the complete overview on hardware and software archi-
8 Has the new device been properly
tecture to develop test strategy or test plan
tested?
Limitations with regard to memory, processing power, battery life 9 Are there any verifications and
validations in place?
Complexity of the software and the system
10 Is there any documentation avail-
able with regard to the IoT test
Best Practices strategy, test plan etc.?

Here are some of the best practices that can be practiced for the mutual Table 3. IoT Testing Checklist

benefit of the organization and testers:

The checklist above helps the team to test IoT devices/applications for
Use of clear test specifications to improve quality
maintaining quality[3].
Use of test automation tool to reduce time-to-market

Analyze the products functional requirements and use cases for

References
effective testing
[1] http://whatis.techtarget.com
Determine test metrics that will measure the impact of the IoT
strategy [2] http://www.infosecurity-magazine.com/news/internet-of-things-
laid-bare-25-security-flaws/
Highlight the top priority problems in advance that need to be
tackled [3] http://www.logigear.com

Increase the device connections to gain more IoT value [4] http://www.elektron-technology.com/sites/elektron-technology.
com/files/iot-agar-scientific-white-paper-final.pdf

Advantages [5] http://blogs.clicksoftware.com/clickipedia/five-expected-benefits-

from-the-internet-of-things-the-impact-on-service/
Below are some of the advantages of using the IoT:

Leverage sensor data to analyze and work with management to

improve business processes
> about the author
Delivering detailed information of product in real-life conditions

Help businesses deliver performance, reliability, and interoper- Venkat Ramesh Atigadda has over nine years
ability industry experience in software testing and has
worked in industry verticals such as energy and
Can be controlled from anywhere through their embedded com-
healthcare. Currently he is a solution developer
puters
in the Assurance CoE of the HiTech Industry So-
Can manage things better, from traffic flows to use of energy lution Unit at Tata Consultancy Services Limited.
In his role, he is responsible for test strategy con-
sulting, project technical reviews, writing white papers and analyz-
Checklist for IoT Testing ing the latest testing trends. He has also published articles in Testing-
Experience, TestingCircus magazines and white papers in the IJCEM
and IJRDET forums.
No Activities Yes No Comments

1 Does the device have safety factors

that a tester needs to consider?
2 Are there any functional or non-
functional risks identified for
testing IoT connections?

Testing Experience 28/2014 41

 By Rik Marselis & Bert Linker
Organize Your Testing Using
Test Varieties and Coverage Types
The Struggle
Many test managers often struggle to define the proper way to spread
the testing efforts throughout the project or release activities in such
way that it properly reflects the constraints of quality, risk, time, and
cost.
In recent years, the rise in approaches that use short cycles has made
it even harder to create a balanced approach to testing and translate
that into a test strategy if there is one at all.
We wondered what the reasons for this problem are. In our opinion, one
of the causes lies with confusion about, or even ignorance of, the mean-
ing of the terms test level, test type, and test design technique.
In this age of agile methodologies, test levels are often associated
with hierarchies in testing, and since Agile promotes doing all activi-
ties by one team in a single iteration, there is no hierarchy. The same
reasoning goes for test types. Because all testing happens within
the iteration (which Scrum calls a sprint), the people involved want to
rush their work and do not want to be bothered with the differences
between various types of testing.
Does it really matter? Well, yes! In a survey amongst about 300 projects
over the last 5 years, almost 50% of the people involved said that the
quality delivered by agile IT teams was no better than before they
adopted Agile. So we need to give extra attention to quality, since
quality is supposed to be key in Agile.
Test Varieties
When it comes to testing, as one of the quality measures that can be
taken, we want to make things easy to grasp by introducing something
new: the test variety. This simple term intends to emphasize to all
people involved that testing is not a one-size-fits-all activity. Even when
all testing activities are done by one team within a single iteration, you
will still need to vary the testing. The first variety of testing, of course,
is static testing, i.e., reviewing documents and source code. Static
testing can both be manual (using techniques like technical review
or inspection) and automated (with tools such as static analyzers).
The next view on test varieties relates to the parties involved. The
developers have a technical view of testing, looking to see whether
the software works as designed and properly integrates with other
pieces of software. The designers want to know whether the system
as a whole works according to their specifications (whatever the form
and shape of these specifications may be). And the business people
simply want to know whether their business process is properly sup-
ported. Now, during these various tests there will be different points
42 Testing Experience 28/2014
of view, for example related to quality attributes. Functionality will
seldom be forgotten in testing, but what about looking at maintain-
ability by the developers, installability by the designers, and usability
by the business people?
So in this simple example we can already see at least seven test variet-
ies. During the start-up of a project (for example in a sprint zero),
a test strategy for the project is established. And for each iteration
the test strategy is tuned to the needs in that cycle. This way, all team
members know what their points of focus must be during this itera-
tion. By the way, please be aware that when we say test strategy we
do not necessarily refer to a document, we merely want to emphasize
that there must be an agreed way of assigning the right testing activi-
ties with the right test intensity. And by being aware of the necessary
test varieties you will also have less difficulty in deciding what testing
activities can be done within the sprint and what has to be organized
separately (the common agreement nowadays is that an end-to-end-
test cannot be done by one agile team within their sprint, so this is a
test variety that will often be organized separately).
This is the first step to better testing. By making the people involved
aware of the relevant varieties of testing, it defies the one-size-fits-all
mentality often seen in testing.
Experience-Based and/or Coverage-Based
Approach
The next step in completing the test strategy is to define the proper
approach for the test varieties. Based on the desired quality level and
the perceived risk level, and within the limitations of time and cost, the
team members choose an experience-based and/or coverage-based
approach to testing.
Here is another area of testing that many people struggle with. There
are very many so-called test design techniques. But, in practice, most
testers do not formally apply any technique at all, they just guess
the best test cases in their specific situation. One reason for this is that
there are so many possibilities that they decide not to choose at all. In
our opinion, the choice does not need to be hard. In any given situation
you only need a simple choice of approaches and about a handful of
coverage types to be able to do proper testing.
We distinguish two approaches: experience-based and coverage-based.
For experience-based testing there are a choice of possibilities, of
which exploratory testing is the most well-known and appropriate
approach. These tests are effective at finding defects, but less appro-
priate for achieving specific test coverage, unless they are combined
with coverage types.
Coverage Type Group Coverage Type Description Variation

Process Algorithm Testing the program structure. Statement coverage

Decision coverage (branch testing/arc
testing)

Paths Coverage of the variations in the process in terms of combina- Test depth level 1
tions of paths. Test depth level 2
Test depth level N

Right paths/fault paths
Checking both the valid and invalid situations in every defined Right paths only
error situation. An invalid situation (faulty control steps in the Right paths and fault paths
process or algorithm that precede the processing) should lead
to correct error handling, while a valid situation should be ac-
cepted by the system without error handling.

State transitions Verification of relationships between events, actions, activities, 0-switch

states, and state transitions. 1-switch
N-switch

Conditions/decisions
Decision points Coverage of the various possibilities within a decision point Condition coverage
with the purpose of arriving at the outcomes of TRUE or FALSE Decision coverage
Condition/decision coverage
Modified condition/decision coverage
Multiple condition coverage (per deci-
sion point or across decision points)
Cause-effect graph
Pairwise testing

Data Boundary values A boundary value determines the transfer from one equiva- Light (boundary value + one value)
lence class to another. Boundary value analysis tests the Normal (boundary value + two values)
boundary value itself plus the value directly above and directly
below it.

Equivalence classes The value range of a parameter is divided into classes in which One value per class
different system behaviour takes place. The system is tested Combination with boundary values
with at least one value from each class.

CRUD Coverage of all the basic operations (create, read, update,

delete) on all the entities.

Data combinations Testing of combinations of parameter values. The basis is Right paths/fault paths
equivalence classes. A commonly used technique for data One or some data pairs
combinations is the classification tree. N-wise (e.g. pairwise)
All possible combinations

Data flows Verifying information of a data flow that runs from actor to
actor, from input to output.

Right paths/fault paths
Checking both the valid and invalid situations in every defined
error situation. An invalid situation (certain values or combina-
tions of values defined that are not permitted for the relevant
functionality) should lead to correct error handling, while a
valid situation should be accepted by the system without error
handling.

Appearance Heuristics Evaluation of (a number of) usability principles.

Load profiles Simulation of a realistic loading of the system in terms of

volume of users and/or transactions.

Operational profiles Simulation of the realistic use of the system by carrying out a
statistically responsible sequence of transactions.

Presentation Testing the layout of input (screens) and output (lists, reports).

Usability Validating whether the system is easy to use, understand, and Alpha testing
learn. Beta testing
Usability lab

Table 1. Overview of the coverage type groups, examples of coverage types, and possible variations

Testing Experience 28/2014 43

 Coverage-based testing uses coverage types. A coverage type focuses
on achieving a specific coverage of quality and risks, and on detecting
specific types of defects. Thus a coverage type aims to cover certain
areas or aspects of the test object. Our starting point is that coverage
types not only indicate what is covered, but also provide directions on
how to do so. Coverage types are, as such, the foundation of the many
test design techniques.
Experience-Based Approach
Below we describe three examples of experience-based testing that
may be considered.
Error Guessing
The tester uses experience to guess the potential errors that might
have been made and determines the methods to uncover the resulting
defects. Error guessing is also useful during risk analysis to identify
potential failure modes. Part of this is defect-based testing, where
the type of defect sought is used as the basis for the test design, with
tests derived systematically from what is known about the defect.
Error guessing is often no more than ad hoc testing, and the results of
testing are totally dependent on the experience and skills of the tester.
Checklist-based
The experienced tester uses a high-level list of items to be noted,
checked, or remembered, or a set of rules or criteria against which a
product has to be verified. These checklists are built based on a set of
standards, on experience, and on other considerations. A checklist of
user interface standards used as the basis for testing an application
is an example of checklist-based testing.
Checking of individual elements is often done using an unstructured
list. Each element in the list is directly tested by at least one test case.
Although checklist-based testing is more organized than error guess-
ing, it is still highly dependent on the skills of the tester, and the test
is only as good as the checklist that is used.
Exploratory
Exploratory testing is simultaneous learning, test design, and test ex-
ecution. In other words, exploratory testing is any testing to the extent
that the tester actively controls the design of the tests, as those tests
are performed and use information gained while testing to design new
and better tests. Good exploratory testing is timeboxed based on a
charter that also defines scope and special areas of attention. Since
exploratory testing is preferably done by two people working together
and who apply relevant coverage types for the specific situation at
hand, this approach is preferred over the alternatives mentioned above.
Hybrid approaches
In practice, the use of hybrid approaches is very common. Exploratory
testing, for instance, can be very well combined with the use of coverage
types. And there are test design techniques that may be used within
experience-based as well as coverage-based testing, such as the data
combination test (which uses classification trees).
44 Testing Experience 28/2014
Coverage-Based Testing
In our experience many testers have difficulty in selecting the proper
coverage in a specific situation, which is often caused by confusion
about the coverage type that best matches the specific situation they
want to test. Thats why for coverage based testing we have created
four groups of coverage types. Analyse the type of situation youre in
and select a coverage type from the group that matches this challenge.
Process
Processes can be identified at several levels. There are algorithms of
control flows, event-based transitions between states, and business
processes. Coverage types like paths, statement coverage, and state
transition coverage can be used to test (variations in) these processes.
Conditions/Decisions
In every IT system there are decision points consisting of conditions,
where the system behavior differs depending on the outcome of such
a decision point. Variations of these conditions and their outcomes
can be tested using coverage types like decision coverage, modified
condition/decision coverage, and multiple condition coverage.
Data
Data starts its lifecycle when it is created and ends when it is removed.
In between, the data is used by updating or consulting it. This lifecycle
of data can be tested, as can combinations of input data, and the at-
tributes of input or output data. Some coverage types in this respect
are boundary values, CRUD, data flows, and data combinations.
Appearance
How a system operates, how it performs, and what its appearance
should be are often described in non-functional requirements. Within
this group we find coverage types like heuristics, operational and load
profiles, and presentation.
Coverage Type Table
The Table 1 gives an overview of the coverage type groups, examples
of coverage types, and possible variations.
Although the overview is extensive, it is not exhaustive. Looking at
what can be covered, we could have added aspects like syntax (using
a checklist), semantics and integrity rules (using decision points), au-
thorisation, privacy etc. (using checklists, doing reviews, etc.). However,
we do not want to over-complicate things. We advise you to check
relevant literature for the coverage types and test design techniques
that are suitable in your specific situation.
Test Intensity Table
A main goal of the test strategy is to define the necessary intensity of the
testing, commonly based on risk. High risk requires thorough testing,
low risk may need only light testing. To give you a practical overview
of the coverage types you can select for the different classes, we have
highlighted the most commonly used coverage types and some test
> about the authors
design techniques in which they can be applied. We have not given
an overview for appearance, since the coverage types for appearance
Rik Marselis is one of Sogetis most experienced
are highly interlinked with the aspect to be tested, and we believe that
management consultants in the field of quality
giving a simplified overview would be misleading.
and testing. He is a well-known author, present-
er, and trainer who has assisted many organiza-
Coverage Test Intensity tions throughout the world in actually improving
Type
Group Light Average Thorough their testing and thus achieving fit-for-purpose
quality and increased business success.
Process Statement Decision coverage Paths test depth Twitter: @rikmarselis
coverage and and paths test level2 algorithms
paths test depth level2 Test and paths test
Bert Linker is an experienced test consultant
depth level1 process cycle test depth level3 pro-
within Sogeti. He is (co)author of several books
process cycle cess cycle test
test
and trainer on many test subjects. He has helped
many organizations in traditional and agile en-
Conditions Condition Modified condi- Multiple condition vironments improve their testing and quality
decision tion decision cov- coverage elemen-
processes.
coverage erage elemen- tary comparison
elementary tary comparison test or
comparison test or multiple condition Both Bert and Rik wrote several building blocks for the new TMap
test condition deci- decision coverage HD book that was presented on 28 October 2014.
sion coverage decision table test
decision table
test

Data One or some Pairwise data N-wise or all com-

data pairs combination test binations data
data combina- combination test
tion test

Table 2. Test intensity table

Conclusion
Applying an effective and efficient way of testing does not need to be
bothersome. Using test varieties, a combination of experience-based
and coverage-based testing, and your choice of about five coverage
types that are relevant for your situation, testing in these fast-paced
times will focus on establishing the stakeholders confidence without
tedious and unnecessary work.

Literature
Testing Embedded Software, Bart Broekman & Edwin Notenboom,
Addison Wesley, 2003, ISBN 9780321159861

TMap NEXT for result-driven testing, Tim Koomen, Leo van der
Aalst, Bart Broekman, Michiel Vroon, UTN Publishers, 2006, ISBN
9072194799

TMap NEXT in Scrum, Leo van der Aalst & Cecile Davis, Sogeti, 2012,
ISBN 9789075414646

Neils Quest for Quality; A TMap HD Story, Aldert Boersma & Erik
Vooijs, Sogeti, 2014, ISBN 9789075414837
 By Sujith Shajee
Reasoning the Next Version of
Test Automation Myths
Test automation myths is a discussion topic that echoes around the
validation service areas of the IT industry. Probably the first thought
that flashes though the readers mind would be Why the same old
topic? or What is new to debate about this topic?
For once, everyone undisputedly agrees that test automation is not
what it used to be five or ten years ago. Test automation has evolved in
range and enormity. What started out as simple linear scripting on a
single web application is now a complex hybrid framework architecture
that facilitates test execution on applications developed on diverse
platforms and technologies. Undoubtedly automation progressed, and
so did the myths that are associated with it. A shift in peoples perspec-
tive and knowledge about test automaton has altered the folklores.
This article is the authors viewpoint and experience on how the origi-
nal myth has transformed into a new version, and how derided the
myth still is. The article also provides the authors thoughts on the
new generation of myths.
Myth # 1
Version 1.0 Test automation is to replace manual effort for good
Version 2.0 Test automation is to reduce manual effort for good and
execution is with the click of a button
It was not long ago that getting test automation implemented sent
out a message that it would be eliminating manual effort completely.
Test automation just meant that manual effort would be removed and
automation would take care of the rest. With time, everyone agrees that
manual and automation testing go hand-in-hand. Everyone also agrees
that moving from manual to automation validation is a collaborative
perceptive process, and test automation does not necessarily replace
but certainly reduces manual effort over time.
But with all this in perspective, a new change has arisen in the existing
myth. A new view seems to have altered this legend that automa-
tion would magically run at the click of a button. Once automation is
completed, the expectation is set that you could trigger scripts and
the entire test suite would run without any monitoring 24x7.
While automation does reduce manual effort, it most certainly does not
take away manual intervention in execution completely. Automation
implementation is a strategic advancement in validation, but it does
come with its own set of responsibilities. A few basic tasks include script
execution monitoring, application and defect monitoring, review of
script failures, minor tweaking, and synchronizations of scripts. These
activities are not always introduced due to script quality but are
governed by many external factors like test environment, application
changes, and application performance. So accounting for the factors
mentioned would set a genuine expectation of how automation ex-
ecution would proceed after the trigger and what could be expected
from the implementation.
46 Testing Experience 28/2014
Myth # 2
Version 1.0 Savings through test automation are assured always
Version 2.0 Savings through test automation are assured with well-
structured implementation and can always be achieved in a pre-de-
termined timeframe
When an organization decides to introduce automation into its testing
strategy, the decision is a commitment of huge investment towards
development, maintenance, and other operational costs associated
with the implementation. Return on Investment (ROI) is calculated
and determined, usually prior to kick off of the implementation, and
is considered to be the assured cost savings from implementation.
What everyone seems to understand now is that ROI is indicative of cost
savings associated with a planned and thought-through implementa-
tion. The calculation of ROI alone does not assure cost savings, but you
need to ensure that the implementation is planned and executed with
caution. Hence savings are not always assured.
While there is no question that a well-structured implementation
would yield the projected cost savings, post-implementation main-
tenance and other operational factors have a very great influence on
how and when breakeven is achieved. If a follow-through plan is not
set up for the implementation, achieving benefits from the automa-
tion implementation would be a far-fetched goal for the organization.
Hence the key to a successful implementation would be to govern the
maintenance of the automated test suite with caution.
Myth # 3
Version 1.0 Test automation uncovers more bugs
Version 2.0 Test automation has failed in its implementation if it is
not able to uncover as many bugs as manual testing
Test automation is designed to reduce manual effort and eliminate
human errors in routine test execution activities. A common idea that
prevailed with automation was that it would successfully uncover
more bugs than manual validation. This idea just falls apart when
you realize that automation is only as good as the manual test cases
it was based and built on.
Identifying more application defects is a result of the quality and
coverage of test cases; yet the ability of automation to uncover defects
decides the success of automation implementation
Automation, in most instances, is meant for regression test suites.
Before an application code is moved to a regression environment
for testing, it has already passed the quality checks in the unit, func-
tional, and integration testing phases. In most cases, the applications
stability and quality in the regression phase is quite predictable. So,
in this phase, whether you decide to perform manual or automated
validation, there is only a slim probability of coming across any more
defects than there are. But this most certainly does not imply that the cycle execution. His team is unaware of most of the changes to the
automation implementation is a failure. Let us not forget that when application until afterwards, it is very late when his team has to run
the organization decided to go for automation, uncovering defects was the regression suite, and they are hit with surprising changes in the
not the only goal defined to be achieved as an end result of automation. application, both minor and major.

It is high time we realized that automation maintenance involves effort.

Myth # 4
Version 1.0 Anyone and everything can be automated
Version 2.0 Automation is software engineering, so a developer is the
right fit for implementation
Yes, you are right when you heard that automation is scripting out
your manual test cases in a specific language that is supported by
the automation tool. Now the questions are, does that mean we can
bring in any automation tool and have the manual tester work on
implementing automation? And if the automation tool supports test
case automation, does it mean it should be automated? The answer
to both these questions is most certainly a big no.
With the evolution in the area of automation, organizations have real-
ized that automation is to be considered as a development project and
should follow a well thought-through implementation plan. A proper
automation feasibility analysis together with a cost-benefit analysis
would really help us decide what has to be automated. Just because
it can be, does not mean it should be. Having said that, this realiza-
tion did lead people to get their heads round new options to support
implementation and one of these was development team involvement.
It was not long ago, while working on an automation strategy, that I
was asked to provide a couple of application developers to build out
the automated test suite, replacing my own automation analysts. It
was not that my automation analysts were bad at their job, but when
I asked why we were bringing in someone from the development
team to do this work, the responses were: They have been there and
done this kind of work, They have a better knowledge of software
engineering, and They will be quick because, after all, it is about
scripting the cases.
My response was that if you do not want your development team to
develop and test your application, you should also not want your de-
velopment team to develop and build automation test cases for your
application. They are one and the same thing. Automation is still about
shaping quality test cases to validate the application. You still need
your scripts to find the application bugs. Yes, development skill helps
the implementation process, but that is not the only skill you should
be looking for from an automation engineer who is building it out for
you. An automation engineer is a blend of developer and tester. He/she
should script it out like a developer and still be able to think like a tester.
Unlike manual testing, even some of the minor changes in the applica-
tion have a drastic impact on automation. For instance, an existing field
that was not mandatory on the application form is made mandatory
now. For the manual testing team this means updating the test cases
to populate data in this existing field. While for the automation team
it is about identifying the flow, adding the object, updating scripts
to accommodate this change, and finally testing the change. It would
be bizarre to think that all these minor changes could be fixed during
regression execution and the execution could still be completed. You
need to realize that the regression execution is already on a shortened
timeline due to automation implementation.
The solution is quite simple. Involve your automation team like you
involve your manual team in SDLC. Let them be part of the requirement
analysis and impact analysis sessions. This would help them under-
stand the change better, and even plan the maintenance better. We are
also providing the team with enough time to deal with all the changes.
Good communication is a key to smooth maintenance and execution.
Inference
George Orwell rightly said: Myths which are believed in tend to be
become true. There is a serious need to break the traditional ideas
that are not true about automation from time to time. This will en-
sure impressions that are never intended to be part of an automation
process stay out of it and will help to govern and establish the right set
of standards and practices that will lead to the correct way of looking
at test automation.
> about the author
Sujith Shajee is currently working as a Test Tech-
nical Lead at Infosys Limited (NASDAQ: Infy
www.infosys.com) and is part of the Independent
Validation and Testing Services unit. He has
worked on various projects for strategy develop-
ment, implementation, and delivery in the auto-
mation, performance, and service validation
areas, and has developed expertise in a number of validation tools.
Sujith can be reached at SujithK_Shajee@infosys.com.
LinkedIn: www.linkedin.com/pub/sujith-shajee/8/71b/863

Myth # 5
Version 1.0 Any changes to test automation can be done in no time
because it is automation

Version 2.0 Any changes to test automation can be done in no time

because it is automation

No, it is not a typo, and most definitely you are not reading the ver-
sions wrongly. This is one myth that has just forgotten to evolve over
time. It was just the other day when a colleague of mine was telling
me about the agony he has to go through every automated regression

Testing Experience 28/2014 47

 By Philipp Benkler
Testing Enterprise Applications:
Bring-Your-Own-Crowd
According to Gartner, 25 percent of enterprises will have an enterprise
app store by 20171. Employees, as well as enterprise clients, benefit from
options available on their mobile device. While traditional desktop
applications, such as CRM, are available on smartphones and tablets,
mobile devices offer many possibilities to increase efficiency through
new functionalities. While adapting to those opportunities, companies
face challenges such as mobile device management and bring your
own device (BYOD). No matter how and to which devices enterprise
apps are distributed, they need to work flawlessly and be appealing
enough for people to use them. The following article discusses several
challenges companies face while testing enterprise applications and
offers best practices on how to address them.
Being Aware of Expectations
There is much to learn from the consumer industry when developing
enterprise applications. Smartphones and tablets have become our
constant companions and we have certain expectations of an appli-
cation, no matter whether we use it before, during, or after work. It
needs to solve an issue or facilitate a process, it needs to be easy to use
and it needs to work. Before developing an enterprise app, companies
must be aware that users regardless of whether they are employees
or clients have the same expectations of enterprise apps as they have
of those they use as a consumer. This could be with regard to design,
navigation, or loading time. Even if there is no alternative, an app will
only be used regularly if it meets these expectations.
Pooling Company Knowledge
To succeed at building enterprise applications that users enjoy, thor-
ough analysis is required to define the basic requirements. What kind
of applications do the employees really need? Which processes does
the app need to perform? What problem does it solve? The future
users are the biggest asset in order to find out. Unlike the consumer
industry, employees and even clients are much more accessible for
knowledge gathering, for example through a survey, and learning
more about their expectations. When developing an enterprise app,
their knowledge brings direct value, and this is why they should be
consulted. The employees know how the business is run, are familiar
with existing processes, and will often have very precise ideas as to
which tools would be useful to facilitate daily working life. Further-
more, when integrating employees, security and intellectual property
risks are low, as internal information is less likely to fall into the hands
of unauthorized people.
Creating Acceptance
By integrating employees in the development process, enterprise apps
have a greater likelihood of acceptance and adoption after completion.
In particular, the involvement of opinion leaders has great potential
for stimulating broad acceptance of new mobile business solutions,
and combating reservations towards transformation and change. It is
48 Testing Experience 28/2014
advisable to scale the level of employee involvement as development
progresses. Only key employees offer insights in the early stages, when
most mistakes and defects are discovered. Later on, a wider range of
future users might be integrated once the bulk of mistakes has been
fixed. In practice, there is a fine line between creating acceptance and
fueling objections. It might not be wise to show an application that is
still in its early stages and full of technical issues to a wider audience
just yet. People will form a negative image if they cannot see the whole
picture. The only thing they will see is an app that does not work, is not
intuitive to use, or does not help them to be more efficient. This is also
true for other stakeholders and decision makers. Convincing someone
who is not involved in the development process that the app is going
to become a success while it is still full of issues is much more difficult
than showing them a better version later on. It is therefore important
that all people involved are aware of the current state as well as the
roadmap of an application.
Bug Testing
Extensive functional testing is the basis for any successful app. To
avoid organizational blindness, testers from outside the development
team should be included, so that both staff and independent testers
are assessing the app. In many cases a mix of experienced testers and
unbiased users offers the greatest benefits. While the former know
where to look and discover defects reliably, the latter do the unex-
pected and might find showstoppers that otherwise would have been
missed. Therefore, it makes sense to establish a mix of structured and
explorative testing to cover as many scenarios as possible. Depending
on the stage of the application as well as the availability of internal
resources, this can be done either internally or externally.
Usability Testing
Usability is the key factor for an app to be successful, both in the con-
sumer and in the enterprise sector. Before release, applications should
be tested by the target group to fully understand their wants and needs.
This way, companies can evaluate whether the initial requirements
have been implemented as required at a certain stage in the devel-
opment process. For consumer apps, crowdtesting is an established
approach for quality assurance and usability testing. Crowdtesting
offers access to specific target groups and devices through large pools
of testers. When developing enterprise apps, employees or enterprise
clients take the place of the crowd. They are the future users and know
what works for them and what does not. Instead of carrying out the
testing process independently, companies should think about using
the crowd platform infrastructure of an external service provider
to distribute tests to their own crowd of employees or customers.
This approach is called Bring-Your-Own-Crowd. It reduces project
management time and budget through a managed testing process,
resulting in high-quality results that can be pushed back directly into
the development process. This way, even testing with confidential
data or restricted access on company devices is possible at any time.
1 http://www.gartner.com/newsroom/id/2334015
Device Diversity and Compatibility
Knowing user expectations only helps if the app really works on all
devices necessary. Bring your own device (BYOD) is becoming common
practice in many companies and brings significant challenges when
developing an enterprise app. If there is no standard company device,
applications have to work on all employee devices and platforms that
will be using the app. Taking into account increasing fragmentation,
particularly in the Android market, this is difficult to achieve. In this
case, external crowdtesting enables access to all devices available in a
specific market, provided that access to the app is possible for external
testers, for example through a VPN connection.
Planning and Costs
Extensive testing requires a budget that takes testing costs into con-
sideration. Companies avoiding these costs should bear in mind what
damage can result if the developed applications are not accepted.
Enterprise apps can optimize internal processes, and lead to savings
through efficiency. However, co-existence of digital and analog infra-
structure as a result of a rejected app will actually increase costs. If
employees are not satisfied with the companys official applications,
they might also look for an uncertified alternative, with security and
financial consequences. If companies decide to integrate their employ-
ees into the testing process, appropriate time and resources need to be
allocated for that purpose. Employees will need to use their working
time to test, and therefore possibly postpone other tasks. In addition,
a strong commitment is required from all testers in order to get mean-
ingful and high quality results. Thus, ordering unwilling employees
to test an app is not ideal. If they are not already interested in the app
SAVE THE DATE
November 0912, 2015
in Potsdam, Germany
www.agiletestingdays.com
through early involvement, other incentives such as bug bounties or
other forms of rewards need to be found.
The importance of enterprise applications for business processes will
increase dramatically within the coming years. To create apps that
work and appeal to users, they need to be involved in the development
process. Just like in the consumer industry, user expectations and
device diversity are major challenges that require a detailed testing
strategy before development starts. Involving the users and testing
through the Bring-Your-Own-Crowd approach help to address these
challenges, while offering a framework for effective bug and usability
testing.
> about the author
Philipp Benkler is founder and CEO of the crowd-
testing specialist Testbirds. He is responsible for
sales, the development of the IT infrastructure,
internationalization and quality assurance. Ben-
kler has significant experience in enterprise
environments and as a freelance software devel-
oper. He took part in the elite network masters
program Finance and Information Management at the University
of Augsburg and the TU Munich in Germany, and graduated Master
of Science with honors.
LinkedIn: www.linkedin.com/profile/view?id=55025531
Website: www.testbirds.com
Blog: blog.testbirds.com
Testing Experience 28/2014 49
Missed Part II?
Read it in
issue No. 27!
By Vladimir Belorusets, PhD

A Unified Framework
For All Automation Needs Part III
Introduction 8. // associate a channel with the session

9. Channel channel = session.openChannel("shell");

In the first two parts of this article[1], I described the main principles
10. // create an Expect object
applied in developing a unified test automation (UTA) framework that
11. Expect expect = new Expect(channel.getInputStream(),
serves as a foundation for testing multiple application interfaces.
12. channel.getOutputStream());
The UTA was built on JUnit and JUnitParams. I showed how to test the
13. channel.connect(CHANNEL_TIMEOUT);
browser GUI and REST API within the UTA framework using the open
source Selenium WebDriver and Spring Framework. In this part, I will Listing 1. Establish an SSH connection

describe the details of implementing automated testing of the com-

mand line interface (CLI) when connecting to an SSH server. To verify connection, we can check for the command prompt. The
Expect class contains the expect() method that handles the input
The most popular tool for automating interaction with CLI is Expect. It
stream against a pattern, places the found match in the match string,
was originally written in Tcl, and there are several open source Expect
and updates the isSuccess boolean to true. The pattern can be pre-
implementations in Java. In the UTA, I use the following programs:
sented as a string or a regular expression. The code snippet is shown
in Listing 2.
Expect-for-Java[2] developed by Ronnie Dong. This API is loosely
based on the Perl Expect library
1. expect.expect("#");

JCraft JSch[3] implementation for SSH protocol 2. assertEquals("#", expect.match);

Listing 2. Check for the command prompt

Structure of the CLI tests

For the second operation, Expect provides the method send().
A simple CLI test consists of the following four operations:

1. Establish an SSH connection with a remote server Testing command options

2. Run an input command in the CLI
3. Get and parse the response
4. Assert the actual outcome against an expected pattern for
verification
The first operation is usually performed once per testing class. The
others comprise a block that is repeated multiple times within a test
If the command under test has multiple options, like ls in UNIX or
dir in Windows, it is efficient to test it using a data-driven approach
with JUnitParams library and JUnitParamsRunner. When you need
to match a complex output, then the java.util.regex.Pattern class
is at your service. Listing 3 illustrates how to create a data-driven test
for the command show.
1. @Test

as we issue various commands. 2. @FileParameters(value = "file:c:/DDT/showCommand.csv",

3. mapper = CsvWithHeaderMapper.class)
The SSH connection is easy to accomplish by using the JCraft JSch
4. public void showCommand(String option, String pattern) {
(Java Security Channel) class. Since the session is established once, the
5. // input: show <option>
corresponding statement is placed within the @BeforeClass method
6. expect.send("show " + option + "\n");
(Listing 1).
7. // expect

8. expect.expect(Pattern.compile(pattern, Pattern.DOTALL));
1. JSch jsch = new JSch();
9. // verify
2.
10. assertTrue(expect.isSuccess);
3. // create a connection with an SSH server
11. System.out.println(expect.match);
4. Session session = jsch.getSession(userName, sshHost);
12. }
5. session.setPassword(password);

6. session.setConfig("StrictHostKeyChecking", "no"); Listing 3. Testing command with multiple options

7. session.connect(CONNECTION_TIMEOUT);

The data file showCommand.csv contains two columns: one with the
command options and one with the regex patterns for expected match.

50 Testing Experience 28/2014

CLI tests with decisions The Base class from where all test classes are extended contains the
method commandProcessor(String csvFileName, Expect expect)
Most CLI tests require a next command to be issued based on some
that parses the test scenario file, runs all commands, and verifies pat-
condition expressed in the previous command outcome. In this case,
tern matches. Using JUnitParams library, this test can be presented
you need to create a list of all possible patterns describing the expected
as simply as shown in Listing 6, where cryptoUser.csv is the name
outcomes. When you pass that list into the expect() method, it will
of the test scenario file.
return the index of the pattern that matched. This will let you know
what outcome among the set of multiple outcomes occurred.
1. @Test

Listing 4 provides an example of executing the show hsm status 2. @Parameters({"cryptoUser.csv"})

command that has two possible outcomes: Crypto-user logged in: 3. public void cryptoUserLogin(String file) throws Exception {

yes and Crypto-user logged in: no. 4. commandProcessor(file, expect);

5. }

1. // input
Listing 6. Test described through the CSV file
2. expect.send("show hsm status\n");

3.

4. // expect

5. List<Pattern> list = new ArrayList<Pattern>(); Summary

6. list.add(Pattern.compile("Crypto-user logged in: yes"));
In this last part of the article, I described how to automate testing CLI
7. list.add(Pattern.compile("Crypto-user logged in: no"));
within the UTA. Since the CLI is not as rich as GUI, the structure of the
8. int matchIndex = expect.expect(10, list);
tests is much simpler.
9. // verify

10. assertTrue(expect.isSuccess); The UTA can be extended to include other interfaces as well, and maybe
11. // make a decision someday we will come up with one global test automation framework
12. switch(matchIndex) { that fits all automation needs.
13. // crypto-user logged in

14. case 0:

15. // new command References

16. // crypto-user is not looged in
[1] Vladimir Belorusets. A Unified Framework for All Automation
17. case 1:
Needs Part I, II. Testing Experience, Issue No. 26, pp. 6670, 2014,
18. // new command
Issue No. 27, pp. 913, 2014.
19. }

[2] Expect-for-Java https://github.com/ronniedong/Expect-for-Java

Listing 4. Making decisions

[3] JCraft JSch http://www.jcraft.com/jsch

Descriptive test scenarios

For complex tests with multiple decisions, you can create a test scenario
in a separate text file using the pre-defined conventions. Lets consider
a simple example for illustration. I want to test login/logout commands
for the HSM device. My first command is show hsm status, which gives
me the status of the crypto user. If the crypto user is already logged
in, I issue the log out command. If the crypto user is not logged in, I
want him to log into the HSM. This scenario can be presented in CSV
format as shown in Listing 5.
1. 0, show hsm status, Crypto-user logged in: yes|1, Crypto-
user logged in: no|3
2. 1, hsm logout crypto user, y/\[n\]|2
3. 2, y, Logged out of HSM partition successfully|-1
4. 3, hsm login crypto user, Crypto user successfully logged
into the HSM|-1
Listing 5. A descriptive test
The first column is the number of the command line. The second column
is a command itself. The rest of the columns present patterns (strings or
regex) for all possible command outcomes. Each pattern ends with the
bar that marks the end of the pattern and the next command number
to go. -1 indicates the end of the test.
> about the author
Dr. Vladimir Belorusets is a Director of QA at
Shocase, Inc. a social network for marketers. He
specializes in test automation and testing meth-
odology. Dr. Belorusets is a Certified ScrumMaster
and Certified Tester Foundation Level. He is the
author of various articles published in Testing
Experience, Agile Record, Software Test & Quality
Assurance, Software Test & Performance, and StickyMinds.com. Dr.
Belorusets was a member of the Strategic Advisory Board and Con-
ference Program Board at Software Test Professionals. He was a
speaker at Atlassian Summit, HP Software Universe, Software Test
Professionals, and STARWEST. Vladimir has held development and
QA management positions at Xerox, EMC, Siebel, CSAA, and various
startups. Dr. Belorusets earned his PhD in Control Systems from
Moscow Institute for Systems Analysis, Russian Academy of Sci-
ences and his Masters Degree in Theoretical Physics from Vilnius
State University, Lithuania. Vladimir has taught numerous courses
on functional and performance testing in various San Francisco Bay
Area computer schools.
LinkedIn: www.linkedin.com/pub/vladimir-belorusets-ph-d-csm-
ctfl/0/2/416

Testing Experience 28/2014 51

 Book Corner

Book Review:

Hands-on Mobile App Testing

Authored by Daniel Knott

A few weeks ago, I got my hands on this little same), there is need for more technically-oriented skills, like reading
gem. It is available on Leanpub for a recom- code structures or supporting the team with test automation.
mended price of US$15. It has 340 pages over
Overall, I enjoyed reading through the book and it contains the broad
nine chapters, plus some introduction and
palette of information one would expect.
acknowledgement segments.
Due to the eBook format, it plays to its strengths with a short update
The book is A guide for mobile testers and
cycle for outdated information and a direct access through the URLs
anyone involved in the mobile app business.
to various topics and information.
While non-testing professionals can benefit
from a lot of useful information and getting a If the audience is new to the field, they get a comprehensive, up-to-date
better understanding of mobile needs, it clearly caters to testers (sic). book. Daniel also makes a quick side tour to provide an overview of dif-
ferent test methods and techniques, ranging from the classic preventive
In its initial chapters, the book lays the foundation as to why the mo-
vs. constructive testing with its analytical and dynamic approach (e.g.
bile factor is different to non-mobile, aka our regular software on
white box/black box) to references to Exploratory Testing, BBST and
desktop or even laptop (arguably a kind of mobile) computers. In a
more. If this book is combined with one of the training courses in the
nutshell, this covers user expectations, data networks, fragmentation,
field (CMAP for example), this would be a very good way for a person
and the (often) short release cycles.
to launch themselves into a mobile testing career.
It builds up by showing the landscape which constitutes mobile work,
On the other hand, if the reader is an active mobile testing profes-
including the challenges, and gives practical tips on how to approach
sional, he/she might not take a lot from it. But, as Daniel wrote in his
the different aspects, e.g. how to create mobile device groups to get a
intro: This book is a practical guide on mobile testing. You can read
handle on the huge fragmentation of devices on the market.
it from front to back to get an overview of mobile testing, or you jump
Throughout the chapters there are a lot of useful links, be it statistical straight to the chapters youre most interested in.
sites, which can help identify relevant devices in your local market
For me personally it repeated a lot of known information, which made
region, testing heuristics to provide support in finding more test ideas,
it hard for me to keep reading. But it was worth it, because in each
or links to tools for different purposes.
chapter I found some useful information to take away, which surprised
Another chapter is dedicated to the so-called soft skills of testers. I me in a positive way. Thanks for taking the time to create this book
think that they are not exclusive to mobile testers and every tester can and providing a good guide.
benefit from them. With the advent of the fast paced development
cycles in agile and also in mobile app development (not necessarily the Maik Nogens

New Releases:

Testing Cloud Services Guide to Advanced Software

How to Test SaaS, Paas & IaaS Testing, Second Edition
Authored by Kees Blokland, Jeroen Authored by Anne Mette Hass
Mengerink, Martin Pol
Published by Artech House. 2nd Edition.
Published by Rocky Nook Inc. 1st Edition. 2014. 476 pages. Hardcover. US$114.00
2013. 184 pages. Softcover. US$36.95

52 Testing Experience 28/2014

Masthead
Editor
Daz& Hilterscheid
Unternehmensberatung GmbH
Kurfrstendamm 179
10707 Berlin
Germany
Phone: +49 (0)30 74 76 28-0
Fax: +49 (0)30 74 76 28-99
Email: info@diazhilterscheid.com
Website: www.diazhilterscheid.com
Daz& Hilterscheid is a member of
Verband der Zeitschriftenverleger
Berlin-Brandenburg e.V..
In all of our publications at Daz& Hilterscheid
Unternehmensberatung GmbH, we make every
effort to respect all copyrights of the chosen graphic
and text materials. In the case that we do not have
our own suitable graphic or text, we utilize those
from public domains.
All brands and trademarks mentioned, where ap-
plicable, registered by third parties are subject
without restriction to the provisions of ruling
Editorial Board
A big thank-you goes to the members of
the Testing Experience editorial board for
helping us select articles for this issue:
Maik Nogens, Gary Mogyorodi, Erik van
Veenendaal, Werner Lieblang and Arjan
Brands.
Editorial
Jos Daz
Layout&Design
Lucas Jahn
Website
Konstanze Ackermann
www.testingexperience.com
Marketing & Sales
Subscribe
Annett Schober
subscribe.testingexperience.com
sales@testingexperience.com
Articles&Authors
Price
editorial@testingexperience.com
Online version: free of charge
www.testingexperience.com
ISSN 1866-5705
Print version: 8.00 (plus shipping)
www.testingexperience-shop.com
labelling legislation and the rights of ownership without permission from Daz& Hilterscheid Un-
of the registered owners. The mere mention of ternehmensberatung GmbH, including other elec-
a trademark in no way allows the conclusion to tronic or printed media.
be drawn that it is not protected by the rights of The opinions mentioned within the articles and
third parties. contents herein do not necessarily express those
The copyright for published material created by of the publisher. Only the authors are responsible
Daz& Hilterscheid Unternehmensberatung GmbH for the content of their articles.
remains the authors property. No material in this
publication may be reproduced in any way or form
Index of Advertisers Picture Credits
Agile Testing Days Netherlands........................... C2 iStock.com/akindo..................................................C1
Ranorex............................................................................3 Gonzalo Vzquez (www.gonvazquez.com).....5
CABA Certified Agile Business Analyst..............7
dpunkt.verlag.............................................................. 21
Testing Experience................................................... 26
Rocky Nook, Inc........................................................... 38
Agile Testing Days.....................................................49
CMAP Certified Mobile App Professional......C4
DE December 1516, 2014 Berlin For further information visit
Christmas Special: 200 off! cmap.diazhilterscheid.com or contact us at
info@diazhilterscheid.com.
EN December 1819, 2014 Berlin
Christmas Special: 200 off! All our courses are available as inhouse
courses and outside of Germany on demand!
DE January 1213, 2015 Berlin

CMAP Certified
Mobile App Professional
The new certification for Mobile App Testing
Apps and mobiles have become an important ele-
ment of todays society in a very short time frame.
It is important that IT professionals are up-to-date
with the latest developments of mobile technology
in order to understand the ever evolving impacts on
testing, performance, and security. These impacts
transpire and influence how IT specialists develop
and test software in their everyday work. A Mobile
App Testing certified professional can support the
requirements team in review of mobile application,
improve user experience with a strong understand-
ing of usability and have the ability to identify and ap-
ply appropriate methods of testing, including proper
usage of tools, unique to mobile technology.

Daz & Hilterscheid Unternehmensberatung GmbH Phone: +49 (0)30 74 76 28-0

Kurfrstendamm 179 Fax: +49 (0)30 74 76 28-99
10707 Berlin Email: info@diazhilterscheid.com
Germany Website: cmap.diazhilterscheid.com