Академический Документы
Профессиональный Документы
Культура Документы
Derk-Jan de Grood
TestGoal
Result-Driven Testing
123
Derk-Jan de Grood
Collis B.V.
De Heijderweg 1
2314 XZ Leiden
The Netherlands
grood@collis.nl
ISBN 978-3-540-78828-7
e-ISBN 978-3-540-78829-4
DOI 10.1007/978-3-540-78829-4
ACM Computing Classication (1998): K.6, D.2.5
Library of Congress Control Number: 2008924999
2008 Collis B.V., Leiden, The Netherlands
This work is subject to copyright. All rights are reserved, whether the whole or part of the material is
concerned, specically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting,
reproduction on microlm or in any other way, and storage in data banks. Duplication of this publication
or parts thereof is permitted only under the provisions of the German Copyright Law of September 9,
1965, in its current version, and permission for use must always be obtained from Springer. Violations
are liable to prosecution under the German Copyright Law.
The use of general descriptive names, registered names, trademarks, etc. in this publication does not imply,
even in the absence of a specic statement, that such names are exempt from the relevant protective laws
and regulations and therefore free for general use.
Cover Design: KnkelLopka, Heidelberg
Illustrations: Thijs Geritz, The Hague
Back cover photo: H. de Vries, Rijswijk
Printed on acid-free paper
987654321
springer.com
Focus on business goals. Align your work with those goals. Eliminate work that
does not add value to the business this is todays management mantra. All good
advice. But few in the testing community truly understand what that kind of
alignment means. Derk-Jan de Grood is one of those few.
Today, many variations of testing processes are available to organizations. Some
are tool driven (both commercial and open source); others are document driven
(IEEE 829 Standard for Software Test Documentation); while still others are technique driven (boundary value, state-transition, and pair-wise testing). A myriad of
books are available to help you from the classics by Beizer and Myers to the latest
from Black; Bach, Kaner, and Pettichord; Graham, Evans, and van Veenendaal;
Craig; and Copeland.
TestGoal is different. TestGoal is result-driven. Not the kind of results testers
have historically tried to achievefind all the severity 1 defects, reach 100% statement coverage, or accurately estimate the number of defects remaining. TestGoal
focuses on results that the business cares about. Like it or not, the business does
not care about pair-wise test design and defect taxonomies and defect reports no
matter how pretty our charts and graphs may be. The business cares about business results sales, profit, market share, time to market, product differentiation,
and competitive advantage.
As testers, our goals must not only support the goals of the business, they must be
the goals of the business. Result-driven testing understands those goals, carries out
only those activities that contribute to those goals, and produces information that
enables executive management to achieve those goals. In this way, all of our testing activities support and add value to the organization.
One way TestGoal accomplishes this is by asking us to consider the question
How would you know that this test project has been a success? How many
executive managers would respond, Well, when you have achieved 90% condi-
vi
tion coverage, of course. They dont even know what those words mean. Yet that
is what we measure and report a sign of our disconnectedness from the business.
TestGoal puts us on the right track.
Another way TestGoal guides us is by pointing out that the risks in risk analysis
are those things that prevent the business goals from being achieved profit, image, product differentiation. Yet most testing risk analysis focuses on threats at
a very different levelresource availability, staff availability, and training needs.
TestGoal also emphasizes the central role that testers should play in their organizations. Gone are the days when the testing group was tucked away in the corner
of the organization chart. Todays testers play a central role in the creation of
quality systems, building bridges to all parts of the organization. We add value to
the organization when we influence others in aligning their work to the business.
This book, however, is more than just high-level philosophy. It describes the intricacies of test plans, levels, budgets, strategies, techniques, and designs. It explains
the importance of test environments, execution, reporting, and automation. But
throughout, Derk-Jan reminds us that while Testers see a lot of suboptimal processes and faulty systems This doesnt change the fact that testing is a wonderful
and fun profession. Result-driven testers know they add value to the software
development project. It is a great profession. Im proud to be a part of it. Best
wishes in your testing efforts.
March 2008
Lee Copeland
Lee Copeland has over thirty years experience as an information systems professional and has
held a number of technical and managerial positions with commercial and non-profit organizations in the areas of applications development, software testing, and software development process improvement. He is a well-known and highly regarded speaker at software conferences both
in the United States and internationally. Lee is the author of A Practitioners Guide to Software
Test Design, a compendium of the most effective methods of test case design.
For many organizations, software has become one of the most important pillars of
their business processes. The requirements for the timely availability and quality
of information systems are high by definition. But in spite of all of the innovations
and initiatives to improve processes, its still not easy to develop error-free software. Testing is, and will continue to be, indispensable to identify poor quality in a
timely manner, or in other words, to identify the risks of going live. Learning by
error has taught organizations that testing is not only a must but that it contributes
positively to the companys business goals.
Structured testing began in the United States in the 1960s, and has since developed
into a mature and independent field. Structured testing was first carried out in
military environments and in the computer hardware industry. Pioneers such as
Beizer, Perry, Parnas, Myers, Hetzel and Gelperin have provided a solid methodical and organizational foundation. Throughout the years, the materials they developed have been included in various standards such as IEEE, ISEB and ISTQB.
Over time, these standards have been supplemented with extensions, variations
and special applications.
As the result of a study that was conducted by the General Accounting Office and
due to political pressure, the Dutch Tax and Customs Services was the first (unbelievable but true) to introduce a comprehensive structured test approach in 1985.
As an employee of the Dutch Tax and Customs Services, I was able to build a
bridge between the American developments and the Dutch approach. In 1988, this
approach was published as the Testing Manual, which was quickly sold out. In
the years following its publication, every self-respecting organization, administrative or other, gladly applied this approach, which is the source of every Dutch
testing standard. The Netherlands also have an international reputation to maintain, because their contribution to the worldwide testing community is significant.
Now that testing has conquered a position in information processing, the focus has
shifted to its added value and the relationship between costs and profits, and,
viii
therefore, to the contribution testing makes to the organizations goals. What damage is prevented by testing? What are the risks? And what are the rewards? In
short, the main goal of testing is to minimize risk!
Can testing have more than one goal? It sure can! Testing, of course, is still an
activity aimed at finding errors. But with the improvement of methods, techniques,
and especially of testing tools, the development of reusable testware, expertise and
infrastructure have become valuable by-products. As a result, testing is faster,
cheaper and better, which of course has a positive effect on the costs and the timeto-market. TestGoal takes a result-driven approach to testing.
TestGoal is a very valuable addition to todays broad range of available test literature. The result-driven approach is hotter than ever. This books clear explanations pleasantly point the different disciplines involved in the testing process in the
right direction: What is needed to achieve the anticipated goal?
Experience shows that it is not easy to apply a methodology. Pressures such as
deadlines, low budgets or low quality stop many good intentions dead in their
tracks. Six clearly described steps lead the reader to the goal the test result.
Throughout the book, TestGoal provides readers with pointers that enable them to
focus on the anticipated goal, the test result, and in fact to minimize risk and produce reusable test products. Who does what when? How are risks analyzed and
processes set up? Everything has a purpose.
For the TestGoal tester, this result-driven mindset (the ten test principles) is almost
innate. A unique element of this book is that every letter focuses on these principles, which I interpret as follows: Professional, Result-driven, Confidence-building, Predictable, Supportive, Flexible and Transparent. The flaws of comparable
methodologies are discussed and useful solutions introduced, always with the
same goal in mind the result!
Derk-Jan de Grood has made an exceptional contribution to the world of testing.
His drive for result is now available to everyone.
I wish you a lot of fun applying the approach and am confident that this book will
enable you to achieve your test goals.
Amersfoort, The Netherlands,
February 2007
Martin Pol
Martin Pol is the founder of the first Dutch test approach. He was chairman of the EuroSTAR
conference many times, and was chairman of the Dutch association for testers, TestNet, for five
years. In 1998, he was the first person to receive the European Testing Excellence Award for his
exceptional contribution to the field of testing in Europe.
Acknowledgements
A book does not write itself. A lot of people helped create the TestGoal philosophy described in this book. A word of thanks is therefore more than appropriate;
this book would not be what it is without their contribution.
A lot of colleagues contributed to the first version of TestGoal by writing down
the experiences they gained with specific activities in a test project. Thanks to
their efforts, TestGoal is not only my story, but our story. For their efforts, I would
like to thank the following people:
Albert Los, Albert Witteveen, Colin Lek, Edward van der Pijl, Frodo Wesseling,
Gert de Mooij, Hans van Bommel, Jaap Wijnands, Jos Oudsen, Juul Roelofs,
Leonard Hugenholtz, Marcel van Donge, Marco van der Heide, Mario de Boer,
Mehdi Jalilivandt, Paul van Leeuwen, Roy Miller Sander Panhuyzen, Wilco
Schumacher, Mathilda Banfield and Janneke Klop.
In addition, I would like to thank the following people individually: Trainee class
2005: You very actively used the primal version of this book. Your questions provided me with valuable information, which I have used to expand the TestGoal
story and make it clearer. Ronald Lagendijk: Thank you very much for your review
and your special contribution to the chapter on test automation and the section
on testing conformity and interoperability. Martin Storm: You went through the
manuscript at a number of vital moments, and I took great pleasure in our discussions. Your test expertise was an indispensable contribution. I would also like to
thank you for creating the layout for the chapter on test automation. Harry Vroom:
It took me weeks to process your review comments, all of which were justified.
Your critical examination was a great help. George Leih: I should have known that
you would ask a lot of questions instead of commenting! I had to think about some
of the questions for a while, but you will find the answers in the book. Juul Roelofs:
Your expertise in test automation and test data has been a great help. Thank you for
thinking along with me and for your review. Susan Zarakoviti: When you started
Acknowledgements
your review, you knew nothing about TestGoal, which makes you an ideal reviewer. And even more so because I respect your knowledge of the testing profession. Jaap Wijnands: Thank you for your meticulous thoughts during the review.
Jan Rodenburg: Thank you for your contribution on performance testing. Jasper
Overgaauw: I appreciate how you thought along and helped put exploratory testing
into practice. Thanks go to Cynthia Maasbommel, Jos Oudsen and Herman Rus:
for their indispensable contribution on security testing. Thank you. Vien Sawer of
L&L Bunnick: Thanks to your edit work the text just as I wanted it, understandable
and easy to read. I Couldnt have done without you. Thijs Geritz: You know that
I really like your drawings. Thank you for the pictograms and the other illustrations
you made for us. I am happy to have them in my book.
Henk van Dam and Dirk van den Heuvel: Thank you for your contribution. Our
discussion about the position and content of TestGoal was very intense, but it
resulted in a joint product. I also want to thank you for the support and the freedom that you gave me while I was writing. Dirk Jan, the first time we met I still
had to learn everything about this intriguing profession. I am glad you introduced
me to the world of testing and I am thankful for what you taught me. Without your
lessons, this book would not have come into existence.
Erik Petersen, Donna McLeod, Fariba Marvasti, Julie Gardiner and Thrse
Schoch. Thanks for your interest and pre-reading the English edition.
And finally, I would like to thank the following people:
Martin Pol and Lee Copeland who wrote the prefaces: Thank you for your words
of praise. I am honored that you were took interest in my book. I am thrilled you
liked it.
Hilda and Babette: I am aware of the fact that writing a book is an enormous drain
on the family quality time. Thanks for understanding and for your patience while
I was sitting behind my laptop again.
Derk-Jan de Grood
Content
Result-driven Testing.............................................................................
1.1
The Importance of IT ....................................................................
1.2
A Statement about Quality ............................................................
1.3
The Perception of Testing .............................................................
1.4
A Common Goal ...........................................................................
1.5
Tying in with the Business............................................................
1.6
Result-driven Testing....................................................................
1.7
Focus on the Goal .........................................................................
1
1
3
6
7
10
11
13
15
15
16
17
19
20
22
23
24
25
27
28
29
31
32
32
33
33
33
xii
Content
37
38
42
44
45
46
47
48
49
51
54
56
58
59
61
61
65
65
67
69
71
71
72
73
74
75
77
77
77
79
83
Step 1 Goal
6
87
87
88
90
93
93
94
95
Content
xiii
Step 2 Approach
7
101
101
104
104
105
106
108
111
112
113
113
116
116
117
118
119
119
120
123
125
125
126
126
127
135
136
137
137
138
141
10
Test Plan..................................................................................................
10.1 Introduction...................................................................................
10.2 Description of the Assignment......................................................
10.3 Test Base.......................................................................................
10.4 Test Strategy .................................................................................
10.4.1 Description of the Test Strategy .....................................
10.4.2 Test risk analysis ............................................................
10.4.3 Quality Attributes ...........................................................
10.4.4 Strategy Matrix...............................................................
143
143
145
146
148
149
151
152
155
xiv
Content
10.5
10.6
10.7
10.8
10.9
157
158
159
160
163
164
164
166
167
167
168
169
170
171
171
Step 3 Design
11
175
175
176
178
178
180
181
12
183
183
185
191
191
192
193
193
196
199
203
206
208
210
215
221
223
Content
xv
225
226
227
234
13
237
237
237
238
243
245
246
14
247
247
249
250
251
251
252
253
254
255
256
256
15
Test Environment...................................................................................
15.1 Introduction...................................................................................
15.2 Determine the Requirements of the Test Environment .................
15.2.1 Module Tests and Module Integration Tests ..................
15.2.2 System Tests...................................................................
15.2.3 Functional Acceptation Tests .........................................
15.2.4 User Acceptance Tests ...................................................
15.2.5 Production Acceptance Tests .........................................
15.2.6 Chain Tests.....................................................................
15.2.7 Pilot ................................................................................
15.2.8 Performance Tests ..........................................................
15.2.9 Security Tests .................................................................
15.2.10 Training Purposes...........................................................
15.3 Test Environment Requirements Checklist...................................
15.4 Setting up the Test Environment...................................................
15.5 Configuration and Smoke Test......................................................
15.5.1 Configuring the Test Environment .................................
15.5.2 Smoke Test.....................................................................
257
257
258
258
259
260
261
262
262
263
264
265
266
267
270
271
271
271
12.7
xvi
Content
15.6
272
272
273
274
Step 4 Set up
16
277
277
278
278
278
279
279
280
280
282
282
283
283
285
285
286
288
289
17
Smoke Test..............................................................................................
17.1 Introduction...................................................................................
17.2 Filling out the Checklist................................................................
17.3 Maintaining the Checklist .............................................................
293
293
295
296
Step 5 Execution
18
19
299
299
301
303
305
306
Content
19.3
19.4
20
xvii
321
321
323
323
324
325
326
327
327
328
332
333
334
337
339
340
342
Step 6 Assurance
21
Assurance ................................................................................................
21.1 Introduction...................................................................................
21.2 Evaluating the Test Project ...........................................................
21.2.1 Purpose of the Evaluation...............................................
21.2.2 Points of Attention..........................................................
21.2.3 Lessons Learned Report .................................................
21.3 Determining the Regression Test Set ............................................
21.4 Archiving and Securing the Testware ...........................................
21.5 Handover.......................................................................................
21.6 Discharging the Test Team ...........................................................
349
349
349
349
350
352
353
354
354
355
Introduction
xx
Introduction
Introduction
xxi
other profession, testing encompasses more than the simple application of a methodology. After all, strict adherence to a specific methodology is no guarantee for
success. Success stems from the mindset, enthusiasm, knowledge and skill of the
tester. These factors determine whether a methodology is applied successfully and
whether testing takes on a result-driven character.
Collis is a professional testing organization that has been testing software under
different conditions at both large and small companies for more than ten years.
TestGoal is the result of this extensive experience. TestGoal was developed in the
field and contains practical descriptions combined with familiar examples. TestGoal helps you combine the mindset, enthusiasm, knowledge and skills needed to
successfully streamline your test project.
TestGoal provides a concrete proposal for an efficient test approach. It defines
the activities that must be included in the test project, the sequence in which
they are best run, and the products they produce. The process is efficient because this information enables testers to start testing early on in the project.
TestGoal enables expectations to be aligned and the customer to formulate its
wishes. It stimulates discussion between the tester and the stakeholders about
the approach. Involving stakeholders in the determination of the approach
builds trust and commitment. It also helps align the stakeholders different expectations.
For each of the test activities, the drive to achieve goals and the test principles
are integrated in the practical descriptions. Testers not only learn what their
tasks are, but also how each task contributes to the anticipated goal. The resultdriven tester involves the stakeholders in the test process and ensures that they
can see how the quality of the software improves and how the risks of going live
are reduced one by one. This builds trust and a basis for the go-live decision.
TestGoal provides companies with a generic test approach. Together with a
clear test report, a generic approach makes the test projects more controllable
and comparable to other test projects. This increases the efficiency of the development process and accelerates the time-to-market.
In short: TestGoal is not a methodology, but a philosophy. Its a practical philosophy that gives testers the tools and best practices to make choices and create good
test projects. TestGoal is an integrated combination of these two aspects and helps
testers think and act in a result-driven way.
The Reason for Testing: What are we Going to Test and Why?
A lot of companies are very keen on testing and have a good, structured methodology, but do not have a concrete goal. Some companies have difficulty explaining
why some tests are run and what their goal is. This is, however, a very important
xxii
Introduction
part of successfully delivering quality software. If the goal of the test is not clear,
the tester will not be able to indicate whether the test produces the desired results
or contributes to the organizations business goals. And the latter is crucial to any
company. Testers who know what theyre doing and what their goal is, are not
only able to make the right choices, they are also able to explain their activities
and their goals to others.
The first part of this book explains why and how we should focus on the companys
business goals. It discusses the role business goals play in a test project and how it
helps involve stakeholders in the test activities. The ten test principles are discussed
after the introduction to result-driven testing. They help the tester apply the theory
with the right mindset and are the driving force behind the testers actions. What
characterizes a result-driven tester and what do their actions contribute to?
Most of the literature about test methodologies covers the testing of applications
that are in development, in other words, brand new systems. TestGoal is widely
applicable, and can be efficiently and successfully used in maintenance or line
organizations to test new programs, as well as changes or enhancements to existing applications. The first part of the book takes a look at various levels of testing
and how TestGoal is applied in the different environments. It provides insight into
the approach of a test project and the activities involved. Separate chapters are
dedicated to the testing of new systems, testing in a maintenance environment, and
the testing of performance, security, conformity and interoperability.
Introduction
xxiii
A number of decisions have already been made in TestGoal based on best practices. Choices are always made at the cost of genericness, but this usually benefits the concreteness and applicability of the approach.
Where decisions have not been made, TestGoal suggests a standard option.
Arguments for other options are also provided. This gives readers a starting
point and support should they choose another option.
Each activity is described clearly and in detail. If the activity produces a product, the information the product should contain is indicated. If possible, guidelines and examples are provided. For example, the description of the physical
test design contains tips for the clear formulation of the goal of the test.
Each activity is explained using an action plan. The action plan enables readers
to go through the activities in the test project in a practical sequence, whereby
some activities can be carried out simultaneously. The action plan clarifies the
relationship between the activities and prevents activities being forgotten.
Integrated test principles . Several places in the book refer to relevant test principles. This provides readers insight into the practical application of the principles and into the coherence between the different activities in the test process.
Situations to which best practices apply are indicated. Tips from the field can
be adopted by readers or can serve as an inspiration for devising ones own solution.
The set up of the test process is closely linked to the testers drive to achieve goals
and the test principles. The description of the test approach regularly refers to the
test principles introduced in the first part of the book.
xxiv
Introduction
Students
IT students are a separate target group. Many students are interested in a career as
a tester, but find it difficult to grasp what the profession actually consists of. The
limited amount of time schools spend on testing is partly to blame. This book
demonstrates how testing contributes to the development of an organization, and
thanks to its practical approach, it also provides a clear overview of what testing
actually encompasses. I also hope that the pleasure I get out of this profession
which I hope can be read between the lines will motivate them to become a
member of the great profession testing is.