Software Testing

Principles and Practices

Srinivasan Desikan
Director of Quality Assurance,
Agile Software Enterprise, Pvt. Ltd.
Bangalore, India
Gopalaswamy Ramesh
Consultant and Adjunct Professor,
International Institute of Information Technology,
Bangalore, India
"'$ !;ConteAt!js
\',,;.:::"r- ':," ,:'<;:""~:::)"::":!-' .. f. ",:,:,~",.:~:w.:
Preface
Acknowledgements
Foreword
v
vi
Vll
Part I
IIPrinciples of Testing
.-software Development Life Cycle Models
1. 1
1.2
1.3
1.4
1. 5
1.6
1.7
1.8
1.9
1.10
1.11
1.12
1.13
1.14
2.1
2.2
2. 3
2.4
2.5
Context of Testing in Producing Software
About this Chapter
The Incomplete Car
Dijkstra's Doctrine
A Test in Time!
The Cat and the Saint
Test the Tests First!
The Pesticide Paradox
The Convoy and the Rags
The Policemen on the Bridge
The Ends of the Pendulum
Men in Black
Automation Syndrome
Putting it All Together
References
Problems and Exercises
Phases of Software Project
2.1.1 Requirements Gathering and Analysis
2.1.2 Planning
2.1.3 Design
2.1.4 Development or Coding
2.1.5 Testing
2.1.6 Deployment and Maintenance
Quality, Quality Assurance, and Quality Control
Testing, Verification, and Validation
Process Model to Represent Different Phases
Life Cycle Models
2.5.1 Waterfall Model
2.5.2 Prototyping and Rapid Application Development Models
3
4
6
7
8
9
11
12
12
14
15
16
19
20
22
23
23
25
26
26
26
26
27
27
27
27
29
31
32
32
34
"------- - --- ----------
x Software Testing
2.5.3 Spiral or Iterative Model
2.5.4 The V Model
2.5.5 Modified V Model
2.5.6 Comparison of Various Life Cycle Models
References
Problems and Exercises
36
37
40
43
43
43
Part II
IIWhite Box Testing
3.1 What is White Box Testing?
3.2 Static Testing
3.2.1 Static Testing by Humans
3.2.2 Static Analysis Tools
3.3 Structural Testing
3.3.1 Unit/Code Functional Testing
3.3.2 Code Coverage Testing
3.3.3 Code Complexity Testing
3.4 Challenges in White Box Testing
References
Problems and Exercises
IIBlack Box Testing
47
48
48
49
53
56
56
57
63
67
68
68
73
4.1
4.2
4.3
4.4
4.5
What is Black Box Testing?
Why Black Box Testing?
When to do Black Box Testing?
How to do Black Box Testing?
4.4.1 Requirements Based Testing
4.4.2 Positive and Negative Testing
4.4.3 Boundary Value Analysis
4.4.4 Decision Tables
4.4.5 Equivalence Partitioning
4.4.6 State Based or Graph Based Testing
4.4.7 Compatibility Testing
4.4.8 User Documentation Testing
4.4.9 Domain Testing
Conclusion
References
Problems and Exercises
74
75
76
76
76
82
84
87
90
93
96
99
101
104
104
105
IIIIntegration Testing
5.1 What isIntegration Testing?
5.2 Integration TestingasaTypeof Testing
5.2.1 Top-DownIntegration
5.2.2 Bottom-UpIntegration
5.2.3 Bi-Directional Integration
5.2.4 SystemIntegration
5.2.5 Choosing Integration Method
5.3 Integration TestingasaPhaseof Testing
5.4 ScenarioTesting
5.4.1 SystemScenarios
5.4.2 UseCaseScenarios
5.5 Defect Bash
5.5.1 Choosing theFrequency and Duration of DefectBash
5.5.2 SelectingtheRight Product Build
5.5.3 Communicating theObjectiveof DefectBash
5.5.4 Settingup and Monitoring theLab
5.5.5 TakingActions and FixingIssues
5.5.6 Optimizing theEffortInvolved inDefectBash
5.6 Conclusion
References
Problems and Exercises
107
108
108
111
113
114
115
116
117
118
118
120
122
123
123
123
123
124
124
125
126
126
II' System and Acceptance Testing
6.1
6.2
6.3
6.4
6.5
SystemTestingOverview
Why isSystemTestingDone?
Functional VersusNon-Functional Testing
Functional SystemTesting
6.4.1 Design/Architecture Verification
6.4.2 BusinessVertical Testing
6.4.3 Deployment Testing
6.4.4 BetaTesting
6.4.5 Certification, Standards and Testingfor Compliance
Non-Functional Testing
6.5.1 Settingup theConfiguration
6.5.2 Comingup with Entry/Exit Criteria
6.5.3 BalancingKeyResources
6.5.4 ScalabilityTesting
6.5.5 ReliabilityTesting
6.5.6 StressTesting
6.5.7 Interoperability Testing
127
128
130
131
133
134
135
136
137
140
141
142
143
143
145
149
153
156
6.6
6.7
Acceptance Testing
6.6.1 Acceptance Criteria
6.6.2 Selecting Test Cases for Acceptance Testing
6.6.3 Executing Acceptance Tests
Summary of Testing Phases
6.7.1 Multiphase Testing Model
6.7.2 Working Across Multiple Releases
6.7.3 Who Does What and When
References
Problems and Exercises
158
159
160
161
162
162
164
165
166
166
Performance Testing
169
7.1
7.2
7.3
7. 4
7.5
7. 6
Introduction
Factors Governing Performance Testing
Methodology for Performance Testing
7.3.1 Collecting Requirements
7.3.2 Writing Test Cases
7.3.3 Automating Performance Test Cases
7.3.4 Executing Performance Test Cases
7.3.5 Analyzing the Performance Test Results
7.3.6 Performance Tuning
7.3.7 Performance Benchmarking
7.3.8 Capacity Planning
Tools for Perform'ance Testing
Process for Performance Testing
Challenges
References
Problems and Exercises
170
170
173
174
176
177
177
179
182
184
186
187
188
190
191
192
Regression Testing
8.1 What is Regression Testing?
8.2 Types of Regression Testing
8.3 When to do Regression Testing?
8.4 How to do Regression Testing?
8.4.1 Performing an Initial "Smoke" or "Sanity" Test
8.4.2 Understanding the Criteria for Selecting the Test Cases
8.4.3 Classifying Test Cases
8.4.4 Methodology for Selecting Test Cases
8.4.5 Resetting t1).eTest Cases for Regression Testing
8.4.6 Concluding the Results of Regression Testing
193
194
195
196
197
198
199
200
201
202
205
III'Internationalization (IlSn)Testing
8. 5
9.1
9.2
9.3
9. 4
9.5
9.6
9.7
9.8
9.9
9.10
9.11
Best Practices in Regression Testing
References
Problems and Exercises
Introduction
Primer on Internationalization
9.2.1 Definition of Language
9.2.2 Character Set
9.2.3 Locale
9.2.4 Terms Used in This Chapter
Test Phases for Internationalization Testing
Enabling Testing
Locale Testing
Internationalization Validation
Fake Language Testing
Language Testing
Localization Testing
Tools Used for Internationalization
Challenges and Issues
References
Problems and Exercises
206
208
208
211
212
212
212
213
214
214
215
216
218
219
221
222
223
225
225
226
227
IIAd hoc Testing
10.1 Overview of Ad Hoc Testing
10.2 Buddy Testing
10.3 Pair Testing
10.3.1 Situations When Pair Testing Becomes Ineffective
10.4 Exploratory Testing
10.4.1 Exploratory Testing Techniques
10.5 Iterative Testing
10.6 Agile and Extreme Testing
10.6.1 XP Work Flow
10.6.2 Summary with an Example
10.7 Defect Seeding
10.8 Conclusion
References
Problems and Exercises
228
229
233
234
236
237
237
239
241
242
245
246
248
248
248
300
300
302
275
276
278
281
283
284
285
285
288
291
292
293
295
295
295
299
274
253
254
254
261
261
267
268
269
269
272
273
273
Usability and Accessibility Testing
12.1 What isUsability Testing?
12.2 Approach toUsability
12.3 When todoUsability Testing?
12.4 How toAchieveUsability?
12.5 Quality Factors for Usability
12.6 Aesthetics Testing
12.7 Accessibility Testing
12.7.1 BasicAccessibility
12.7.2 Product Accessibility
Toolsfor Usability
Usability LabSetup
TestRolesfor Usability
Summary
References
Problems and Exercises
Testing of Object-Oriented Systems
11.1 Introduction
11.2 Primer onObject-Oriented Software
11.3 Differences in00Testing
11.3.1 Unit Testingaset of Classes
11.3.2 Putting Classes toWorkTogether-Integration Testing
11.3.3 SystemTestingand Interoperability of 00Systems
11.3.4 Regression Testingof 00Systems
11.3.5 Toolsfor Testingof 00Systems
11.3.6 Summary
References
Problems and Exercises
12.8
12.9
12.10
12.11
Part IV
xiv Software Testing
IIICommon People Issues
13.1 Perceptions and Misconceptions About Testing
13.1.1 "Testingisnot TechnicallyChallenging"
13.1.2 "TestingDoesNot Provide meaCareer Pathor Growth"
Part III
13.2
13.3
13.4
13.1.3 "I AmPut inTesting- What isWrongWithMe?!"
13.1.4 "TheseFolksAreMyAdversaries"
13.1.5 "TestingisWhat I CanDointheEndif I Get Time"
13.1.6 "ThereisnoSenseof Ownership inTesting"
13.1.7 "Testingisonly Destructive"
Comparison between Testingand Development Functions
Providing Career Pathsfor TestingProfessionals
TheRoleof theEcosystemand aCall for Action
13.4.1 Roleof Education System
13.4.2 Roleof Senior Management
13.4.3 Roleof theCommunity
References
Problems and Exercises
303
304
304
306
306
306
307
314
314
315
316
318
318
IIIOrganization Structuresfor Testing Teams 320
14.1 Dimensions of Organization Structures 321
14.2 Structures inSingle-Product Companies 321
14.2.1 TestingTeamStructures for Single-Product Companies 322
14.2.2 Component-WiseTestingTeams 325
14.3 Structures for Multi-Product Companies 325
14.3.1 TestingTeamsasPart of "CTO'sOffice" 327
14.3.2 SingleTestTeamfor All Products 328
14.3.3 TestingTeamsOrganized by Product 329
14.3.4 SeparateTestingTeamsfor Different Phases of Testing 329
14.3.5 Hybrid Models 331
14.4 Effectsof Globalization and Geographically Distributed Teams
onProduct Testing 331
14.4.1 BusinessImpact of Globalization 331
14.4.2 Round theClockDevelopment/Testing Model 332
14.4.3 TestingCompetency Center Model 334
14.4.4 Challenges inGlobal Teams 336
14.5 TestingServicesOrganizations 338
14.5.1 BusinessNeed for TestingServices 338
14.5.2 Differencesbetween TestingasaServiceand Product-
TestingOrganizations 338
14.5.3 Typical Rolesand Responsibilities of TestingServicesOrganization 339
14.5.4 Challenges and IssuesinTestingServicesOrganizations 342
14.6 SuccessFactorsfor TestingOrganizations 344
References 346
Problems and Exercises 346
Part V
Test Planning, Management, Execution, and Reporting 351
15.1 Introduction 352
15.2 TestPlanning 352
15.2.1 Preparing aTestPlan 352
15.2.2 ScopeManagement: DecidingFeaturestobeTested/Not Tested 352
15.2.3 DecidingTestApproach/Strategy 354
15.2.4 Settingup Criteriafor Testing 355
15.2.5 IdentifyingResponsibilities, Staffing,andTrainingNeeds 355
15.2.6 IdentifyingResourceRequirements 356
15.2.7 Identifying TestDeliverables 357
15.2.8 TestingTasks:Sizeand EffortEstimation 357
15.2.9 ActivityBreakdownand Scheduling 360
15.2.10Communications Management 361
15.2.11RiskManagement 362
15.3 TestManagement 366
15.3.1 Choiceof Standards 366
15.3.2 TestInfrastructure Management 369
15.3.3 TestPeopleManagement 372
15.3.4 Integrating withProduct Release 373
15.4 TestProcess 374
15.4.1 PuttingTogether andBaseliningaTestPlan 374
15.4.2 TestCaseSpecification 374
15.4.3 Updateof TraceabilityMatrix 375
15.4.4 Identifying PossibleCandidates for Automation 375
15.4.5 DevelopingandBaseliningTestCases 376
15.4.6 ExecutingTestCasesandKeepingTraceabilityMatrixCurrent 376
15.4.7 CollectingandAnalyzingMetrics 377
15.4.8 Preparing TestSummary Report 377
15.4.9 RecommendingProduct ReleaseCriteria 377
15.5 TestReporting 378
15.5.1 RecommendingProduct Release 379
15.6 BestPractices 379
15.6.1 ProcessRelatedBestPractices 380
15.6.2 PeopleRelatedBestPractices 380
15.6.3 TechnologyRelatedBestPractices 380
Appendix A: TestPlanning Checklist 381
Appendix B:TestPlanTemplate 384
References 385
Problems and Exercises 385
xVii
IIISoftware Test Automation 387
16.1 What isTestAutomation? 388
16.2 TermsUsedinAutomation 390
16.3 SkillsNeeded forAutomation 392
16.4 What toAutomate, Scopeof Automation 394
16.4.1 Identifying theTypesof TestingAmenabletoAutomation 394
16.4.2 Automating AreasLessPronetoChange 395
16.4.3 AutomateTeststhat PertaintoStandards 395
16.4.4 Management AspectsinAutomation 396
16.5 DesignandArchitecturefor Automation 396
16.5.1 External Modules 397
16.5.2 Scenarioand Configuration FileModules 398
16.5.3 TestCasesand TestFramework Modules 398
16.5.4 Toolsand ResultsModules 399
16.5.5 Report Generator and Reports/MetricsModules 399
16.6 GenericRequirements for TestTool/Framework 399
16.7 ProcessModel for Automation 408
16.8 SelectingaTestTool 411
16.8.1 Criteriafor SelectingTestTools 412
16.8.2 Stepsfor Tool SelectionandDeployment 415
16.9 Automation for ExtremeProgramming Model 415
16.10 ChallengesinAutomation 416
16.11 Summary 416
References 418
Problemsand Exercises 419
II' Test Metrics and Measurements 420
17.1
17.2
17.3
17.4
17.5
17.6
What areMetricsand Measurements?
WhyMetricsinTesting?
Typesof Metrics
ProjectMetrics
17.4.1 EffortVariance(PlannedvsActual)
17.4.2 ScheduleVariance(PlannedvsActual)
17.4.3 EffortDistribution AcrossPhases
Progress Metrics
17.5.1 TestDefectMetrics
17.5.2 Development DefectMetrics
Productivity Metrics
17.6.1 Defectsper 100Hours of Testing
17.6.2 TestCasesExecutedper 100Hours of Testing
421
425
427
428
429
430
432
433
434
443
448
450
450
17.7
17.8
17.6.3 Test Cases Developed per 100 Hours of Testing
17.6.4 Defects per 100 Test Cases
17.6.5 Defects per 100 Failed Test Cases
17.6.6 Test Phase Effectiveness
17.6.7 Closed Defect Distribution
Release metrics
Summary
References
Problems and Exercises
450
450
451
452
452
453
455
456
456
IIIllustrations
IIReferences and Bibliography
IIIndex
457
481
483
In this chapter-
./ Context of testing in producing software
./ About this chapter
./ Theincomplete car
./ Dijkstra's doctrine
./ A test in time!
./ Thecat and the saint
./ Test the tests first!
./ Thepesticide paradox
./ Theconvoy and the rags
./ Thepolicemen on the bridge
./ Theends of the pendulum
./ Men in black
./ Automation syndrome
./ Putting it all together
4 Software Testing
1.1 CONTEXT OF TESTING IN PRODUCING SOFTWARE
Almost everything we use today has an element of software in it. In the
early days of evolution of software, theusers of software formed asmall
number compared to the total strength of an organization. Today, in a
typical workplace (andat home), just about everyoneuses acomputer and
software. Administrative staff use officeproductivity software (replacing
the typewriters of yesteryears). Accountants and finance people use
spreadsheets and other financial packages to help them do much faster
what they used to do with calculators (or even manually). Everyone inan
organization and at home uses e-mail and theInternet for entertainment,
education, communication, interaction, and for getting any information
they want. Inaddition, of course, the"technical" people useprogramming
languages, modeling tools, simulation tools, and database management
systems for tasks that they were mostly executing manually a few
years earlier.
Theaboveexamples arejust someinstances wheretheuseof softwareis
"obvious" totheusers. However, softwareismoreubiquitous andpervasive
than seenintheseexamples. Softwaretoday isascommonaselectricitywas
intheearlypart of thelast century.Almost everygadget and devicewehave
at home and at work is embedded with asignificant amount of software.
Mobilephones, televisions, wrist watches, and refrigerators or any kitchen
equipment all haveembedded software.
Another interesting dimension is that software is being used now in
missioncritical situations where failureissimply unacceptable. Thereisno
wayonecansuggest asolutionof "pleaseshutdown andreboot thesystem"
for a software that is in someone's pacemaker! Almost every service we
have taken for granted has software. Hanks, air trafficcontrols, cars areall
powered by softwarethat simply cannot affordtofail. Thesesystems have
torun reliably,predictably, all thetime, everytime.
This pervasiveness, ubiquity, and mission criticality places certain
demands ontheway thesoftwareisdeveloped and deployed.
First, an organization that develops any form of software product
or service must put in every effort to drastically reduce and, preferably,
eliminate any defects in each delivered product or service. Users are
increasingly intolerant of the hit-and-miss approach that characterized
software products. From the point of view of a software development
organization also, it may not be economically viable to deliver products
with defects. For instance, imagine finding a defect in the software
embedded in a television after it is shipped to thousands of customers.
How is it possible to send "patches" to these customers and ask them to
"install the patch?" Thus, the only solution is to do it right the first time,
before sending aproduct tothecustomer.
Second, defectsareunlikely toremain latent for long. Whenthenumber
of userswaslimitedandthewaytheyusedtheproduct wasalsopredictable
Principles of Testing 5
(and highly restricted), it was quite possible that there could be defects in
thesoftwareproduct that would never get detected or uncovered for avery
long time. However, with thenumber of users increasing, the chances of a
defectgoingundetected arebecoming increasingly slim. If adefect ispresent
intheproduct, someone will hit upon it sooner than later.
Third, thenatureofusageofaproduct oraserviceisbecomingincreasingly
unpredictable. When bespoke software isdeveloped for aspecificfunction
for aspecific organization (for example, apayroll package), the nature of
usageof theproduct canbepredictable. Forexample, users canonlyexercise
thespecificfunctionality provided inthebespoke software. Inaddition, the
developers of thesoftwareknow theusers, their business functions, and the
user operations. On the other hand, consider ageneric application hosted
ontheInternet. Thedevelopers of theapplication haveno cOI],trol over how
someone will usetheapplication. Theymay exerciseuntested functionality;
they may have improper hardware or software environments; or they may
not befully trained ontheapplication and thus simply usetheproduct inan
incorrect or unintended manner. Despiteall this "mishandling," theproduct
should work correctly.
Finally,theconsequenceandimpact of every singledefectneeds analysis,
especially for mission critical applications. It may beacceptable to say that
99.9%of defects arefixedin aproduct for arelease, and only 0.1% defects
are outstanding. It appears to be an excellent statistics to go ahead and
releasetheproduct. However, if wemap the0.1%failure inmission critical
applications, thedata will look likethis.
~ A total of 10,000incorrect surgery operations per week.
~ Three airplane crashes every day.
~ No electricity for fivehours every week.
For sure, theabovedataisunacceptable for any individual, organization,
or government. Providing aworkaround, such as "In caseof fire, wear this
dress," or documenting afailure, such as "Youmay lose only body parts
in caseof awrong airplane landing" would not be acceptable in cases of
mission critical applications.
This book focuses on software testing. Traditionally, testing is defined
as being narrowly confined to testing the program code. Wewould like
to consider testing in abroader context as encompassing all activities that
address the implications of producing quality products discussed above.
Producing asoftware product entails several phases (such as requirements
gathering, design, and coding) inaddition totesting (inthetraditional sense
of the term). While testing is definitely one of the factors (and one of the
phases) that contributes toahigh quality product, it alonecannot add quality
to aproduct. Proper interaction of testing with other phases isessential for
agoodproduct. Theseinteractions and their impact arecaptured inthegrid
inFigure 1.1.
Figure 1.1
Relationship of
effectiveness of
testing to quality
of other phases.
Poor upstream activities
Right balance,
Implies heavy
Promotes teamwork,
dependence on testing
Delivers a quality
to detect and correct
product to customers
defects
HIGH REWORK COST
IDEAL STATE
Poor upstream activities AND
May lead to unexpected
poor testing
defects due to poor testing
NOT SUSTAINABLE
RISKY
Quality of Other Phases
If thequality of theother phases islowand theeffectivenessof testing is
low(lower left-hand corner of thegrid), thesituation isnot sustainable. The
product will most likelygoout of business very soon. Tryingtocompensate
for poor quality in other phases with increased emphasis on the testing
phase (upper left-hand corner of the grid) islikely to put high pressure on
everyone asthe defects get detected closer to thetimetheproduct isabout
tobereleased. Similarly,blindly believing other phases tobeof high quality
and having a poor testing phase (lower right-hand side of the grid) will
lead to the risky situation of unforeseen defects being detected at the last
minute. Theideal state of course is when high quality ispresent in all the
phases including testing (upper right-hand corner of thegrid). Inthis state,
thecustomers feel thebenefits of quality and thispromotes better teamwork
and successinanorganization. .
1.2 ABOUTTHIS CHAPTER
In this chapter, we discuss some of the basic principles of testing. We
believe that these principles are fundamental to the objective of testing,
namely, to provide quality products to customers. These principles also
formthe motivation for the rest of the book. Thus this chapter acts as an
anchor for the rest of thebook.
Thefundamental principles of testing areasfollows.
1. Thegoal of testingistofinddefectsbeforecustomers findthemout.
2. Exhaustive testing isnot possible; program testing can only show
thepresence of defects, never their absence.
3. Testingapplies all through thesoftware lifecycleand isnot anend-
of-cycleactivity.
4. Understand thereason behind the test.
Principles of Testing
5. Testthetests first.
6. Testsdevelop immunity and have toberevised constantly.
7. Defects occur in convoys or clusters, and testing should focus on
these convoys.
8. Testingencompasses defect prevention.
9. Testingisafinebalance of defect prevention and defect detection.
10. Intelligent and well-planned automation is key to realizing the
benefits of testing.
11. Testingrequirestalented, committedpeoplewhobelieveinthemselves
andwork inteams.
Wewill takeupeachof theseprinciples inthesubsequent sections. Where
appropriate, wewill illustrate theprinciplewith asimplestory fromoutside
thearenaof information technology todrivehome thepoint.
1.3 THE INCOMPLETE CAR
Eventually, whatever asoftware organization develops should meet the
needs of the customer. Everything elseis secondary. Testingis ameans of
making surethat theproduct meets theneeds of thecustomer.
Wewould liketo assign abroader meaning to the term "customer." It
does not mean just external customers. There are also internal customers.
For example, if aproduct isbuilt using different components fromdifferent
groups within an organization, the users of these different components
shouldbeconsidered customers, evenif they arefromthesameorganization.
Having this customer perspective enhances the quality of all the activities
including testing.
We can take the internal customer concept a step further where the
development teamconsiders thetesting teamas its internal customer. This
way wecanensure that theproduct isbuilt not onlyfor usage requirements
8 Software Testing
but also for testing requirements. This concept improves "testability" of
the product and improves interactions between the development and
testingteams.
Wewould like to urge the reader to retain these two perspectives-
customer perspective and perspective of quality not being an add-on
in the end, but built in every activity and component right from the
beginning-throughout thebook.
If our job is to give a complete car to the customer (and not ask the
customers to paint thecar) and if our intent isto make surethe car works
as expected, without any (major) problems, then we should ensure
that we catch and correct all the defects in the car ourselves. This is the
fundamental objectiveof testing. Anythingwedointesting, itbehovesusto
remember that.
1.4 DIJKSTRA'S DOCTRINE
Consider aprogram that is supposed to accept asix-character code and
ensure that the first character is numeric and rests of the characters are
alphanumeric. How many combinations of input data should we test, if
our goal isto test theprogram exhaustively?
Thefirst character can be filledup in one of 10ways (the digits 0-9).
Thesecondthrough sixthcharacterscaneachbefilledup in62ways (digits
0-9, lower caselettersa-z andcapital lettersA-Z). Thismeans that wehave
a total of 10x (62
5
) or 9,161,328,320valid combinations of values to test.
Assuming that eachcombination takes 10seconds to test, testing all these
validcombinationswill takeapproximately 2,905years!
Therefore, after 2,905years, wemay concludethat all valid inputs are
accepted. But that is not the end of the story-what will happen to the
program when wegiveinvalid data? Continuing theaboveexample, if we
assume there are 10punctuation characters, then we will have to spend
a total of 44,176years to test all the valid and invalid combinations of
input data.
All this just to accept one field and test it exhaustively., Obviously,
exhaustivetestingof areal lifeprogramisnever possible.
All the abovemean that wecanchooseto executeonly asubset of the
tests. Tobeeffective, we should chooseasubset of tests that canuncover
the maximum number of errors. Wewill discuss in Chapter 4, on Black
BoxTesting, and Chapter 3, on WhiteBoxTesting, sometechniques such
as equivalence partitioning, boundary value analysis, code path analysis,
and soonwhichhelp inidentifying subsets of test casesthat haveahigher
likelihoodof uncovering defects.
Nevertheless, regardless of whichsubset of test caseswechoose, wecan
never be100%surethat thereareno defectsleftout. Butthen, toextend an
oldcliche,nothing canbecertainother than deathandtaxes, yetweliveand
doother thingsbyjudiciously managing theuncertainties.
1.5 A TEST IN TIME!
Defects in a product can come from any phase. There could have been
errors while gathering initial requirements. If a wrong or incomplete
requirement forms thebasis for thedesign and development of aproduct,
then that functionality can never be realized correctly in the eventual
product. Similarly, when a product design-which forms the basis for
the product development (a La coding)-is faulty, then the code that
realizes the faulty design will also not meet the requirements. Thus, an
essential condition should be that every phase of software development
(requirements, design, coding, and soon) should catchand correct defects
at that phase, without lettingthedefects seeptothe next stage.
Let us look at the cost implications of letting defects seep through. If,
during requirements capture, somerequirements areerroneously captured
and theerror isnot detecteduntil theproduct isdelivered tothecustomer,
theorganization incursextraexpensesfor
ffi performing awrong designbased onthewrong requirements;
ffi transforming thewrong design into wrong codeduring thecoding
phase;
ffi testing to make sure the product complies with the (wrong)
requirement; and
ffi releasing theproduct with thewrong functionality.
.InFigure1.2thedefectsinrequirementsareshowningray.Thecoloured
figureisavailableonpage457.Asyou cansee, thesegray boxesarecarried
forwardthroughthreeof thesubsequentstages-design, codin~andtesting.
I
10 Software. Testing
Figure 1.2
How defects from early
phases add to the
costs.
Requirement
Phase
Design
Phase
Coding
Phase
Testing
Phase
When this erroneous product reaches the customer after the testing
phase, the customer may incur apotential downtime that canresult inloss
of productivity or business. Thisin tum would reflect as aloss of goodwill
to the software product organization. On top of this loss of goodwill, the
softwareproduct organization would havetoredo all thesteps listed above,
inorder torectify theproblem.
Similarly, when adefect isencountered during thedesign phase (though
the requirements were captured correctly, depicted by yellow), the costsof
all of thesubsequent phases (coding, testing, and soon) havetobeincurred
multiple times. However, presumably, thecostswould belower than inthe
first case, where even therequirements werenot captured properly. This is
because thedesign errors (represented byyellowboxes) arecarried forward
only tothecodingand testing phases. Similarly,adefect inthecoding phase
iscarried forward tothetesting phase (greenboxes). Again, asfewer phases
are affected by this defect (compared to requirements defects or design
defects), wecanexpect that thecost of defects incoding should belessthan
theearlier defects. Ascanbeinferred fromtheabovediscussion, the cost of
adefect iscompounded depending onthedelay indetecting thedefect.
Hence, smaller thelagtimebetween defect injection(i.e.,when thedefect
wasintroduced) and defect detection (i.e.,when thedefect was encountered
and corrected), lesser arethe unnecessary costs. Thus, it becomes essential
Figure 1.3
Compounding effect
of defects on software
costs.
10x
Reqmts Design Coding Testing Post release
Principles afTeSting
to catch the defects as early as possible. Industry data has reaffirmed these
findings. Whilethere isno consensus about thecosts incurred due to delay
in defect detection, adefect introduced during the requirement phase that
makes ittothefinal releasemay cost asmuch asathousand times thecost of
detecting and correcting thedefect during requirements gathering itself.
1.6 THE CATAND THE SAINT
Testingrequires asking about and understanding what you aretrying to
test, knowing what thecorrect outcome is, and why you areperforming any
test. If wecarry out tests without understanding why wearerunning them,
wewill end up inrunning inappropriate tests that do not address what the
product should do. Infact, it may eventurn out that theproduct ismodified
tomakesurethetests arerun successfully, evenif theproduct does not meet
theintended customer needs!
Understanding the rationale of why we are testing certain functionality
leads todifferent types of tests, which wewill cover inPart II of thebook. We
do white box testing to check the various paths in the code and make sure
they areexercised correctly. Knowing which codepaths should beexercised
for agiventest enables making necessary changes toensure that appropriate
paths are covered. Knowing the external functionality of what the product
should do, wedesign black box tests. Integration tests areused tomakesure
that thedifferent components fittogether. Internationalization testing isused
toensure that theproduct works with multiple languages found indifferent
parts of the world. Regression testing is done to ensure that changes work
asdesigned and do not have any unintended side-effects.
12 $oftware Testing
1.7 TESTTHE TESTSFIRST!
Fromthe aboveexample, it is clear that it is the audiologist who has a
hearingproblem, not thepatient! Imagineif thedoctor prescribed atreatment
for thepatient assuming that thelatter couldnot hear at 20feet and 30feet.
Tests are also artifacts produced by human beings, much as programs
and documents are. Wecannot assume that thetests will beperfect either!
It is important to make sure that thetests themselves arenot faulty before
we start using them. Oneway of making sure that thetests aretested isto
document the inputs and expected outputs for agiven test and have this
description validated by anexpert or get it counter-checked by somemeans
outside the tests themselves. For example, by giving aknown input value
and separately tracing out the path to be followed by the program or the
process, one can manually ascertain the output that should be obtained.
Bycomparing this "known correct result" with theresult produced by the
product, theconfidencelevel of thetestandtheproduct canbeincreased. The
practices of reviews and inspection and meticulous test planning discussed
inChapter 3and Chapter 15provide means totest thetest.
1.8 THE PESTICIDE PARADOX
Defectsarelikepests; testing is likedesigning the right pesticides to catch
and kill thepests; and thetest casesthat arewritten arelikepesticides. J ust
like pests, defects develop immunity against test cases! As and when we
write new test casesand uncover new defectsintheproduct, other defects
that were"hiding" underneath showup.
Principle~ of Testing 13
There are two possible ways to explain how products develop this
"immunity" against test cases. One explanation is that the initial tests go
a certain distance into the code and are stopped fromproceeding further
becauseof thedefectsthey encounter. Oncethesedefectsarefixed, thetests
proceed further, encounter newer parts of thecodethat havenot been dealt
with before, and uncover new defects. This takes a"white box" or acode
approach toexplainwhy new defectsget unearthed withnewer tests.
A second explanation for immunity is that when users (testers) start
using (exercising) a product, the initial defects prevent them fromusing
thefull external functionality. As tests are run, defects areuncovered, and
problems arefixed, users get toexplorenewfunctionality that has not been
used beforeand this causes newer defects tobeexposed. This "black box"
view takes afunctionality approach toexplain thecausefor this "more we
test moredefectscorneup" phenomenon.
An alternative way of looking at this problem is not that the defects
developimmunity but thetestsgodeeper tofurther diagnoseaproblemand
thus eventually "kill thedefects." Unfortunately, giventhecomplexnature
of softwareand theinteractions among multiple components, this final kill
happens veryrarely. Defectsstill survivethetests, haunt thecustomers, and
causeuntold havoc.
Software Testing
The need for constantly revising the tests to be run, with the intent
of identifying new strains of the defects, will take us to test planning
and different types of tests, especially regression tests. Regression tests
acknowledge that new fixes (pesticides) can cause new "side-effects"
(new strains of pests) and can also cause some older defects to appear.
The challenge in designing and running regression tests centers around
designing theright teststocombat newdefectsintroduced bytheimmunity
acquired by aprogram against old test cases. Wewill discuss regression
tests inChapter 8.
1.9 THE CONVOY AND THE RAGS
Defects in a program also typically display this convoy phenomenon.
They occur in clusters. Glenford Myers, in his seminal work on software
testing [MYER-79],proposed that theprobability of the existence of more errors
in a section of a program is proportional to the number of errors already found in
that section.
Thismay sound counter-intuitive, but canbelogicallyreasoned out. Afix
for onedefectgenerallyintroduces someinstabilityandnecessitatesanother
fix.All thesefixesproduce side-effectsthat eventually causetheconvoy of
defectsincertainparts of theproduct.
Fromatest planning perspective, this means that if we find defects in
a particular part of product, more-not less-effort should be spent on
testing that part. This will increasethereturn on investments in testing as
thepurpose of testing is findthedefects. Thisalsomeans that whenever a
product undergoes any change, theseerror-prone areasneed tobetested as
they may get affected. Wewill cover theseaspects inChapter 8, Regression
Testing.
Figure 1.4
The number of defects
yet to be found
increases with the
number of defects
uncovered.
"0
c:
~
Q
Q)
.c
,g
~
en
~
o
Number of defects found
A fix for a defect is made around certain lines of code. This fix can
produce side-effects around the same piece of code. This sets in spiraling
changes to the program, all localized to certain select portions of the code.
When welook at the codethat got the fixesfor the convoy 6f defects, it is
likely to look likeapieceof rag! Fixingatear in oneplacein ashirt would
most likely cause damage in another place. Theonly long-term solution in
suchacaseistothrowaway theshirt and createanew one. Thisamounts to
are-architecting thedesign and rewriting thecode.
1.10 THE POLICEMEN ON THE BRIDGE
Testers are probably best equipped to know the problems customers
may encounter. Likethesecondpoliceofficerintheabovestory, they know
peoplefall andthey knowwhy peoplefall. Rather than simply catchpeople
who fall (andthereby beexposedtotherisk of amissed catch), they should
also look at theroot causefor fallingand advise preventive action. It may
not bepossiblefor testers themselves tocarry out preventive action. J ust as
the second policeofficer had to enlist thehelp of an engineer to plug the
hole, testers would havetowork with development engineers tomakesure
theroot causeof thedefectsareaddressed. Thetesters should not feel that
by eliminating theproblems totally their jobs areat stake. Likethesecond
policeman, their careerscanbeenriching andbeneficial totheorganization
if theyharness their defect detectionexperienceand transformsomeof it to
defectprevention initiatives.
Defect prevention is apart of atester's job. A career as a tester canbe
enriching and rewarding, if we canbalance defect prevention and defect
detection activities. Someof thesecareer path possibilitiesareencapsulated
inathree-stage model inChapter 13,CommonPeopleIssues. Wewill now
visit thequestionof what istheright balancebetweendefectprevention and
defect detection.
1.11 THE ENDSOF THE PENDULUM
The eventual goal of any software organization is to ensure that the
customers get products that arereasonably freeof defects. There aretwo
approaches to achieving this goal. Oneistofocus on defect detection and
correction and thesecond is to focus ondefect prevention. Theseare also
calledquality control focus and quality assurance focus.
Testing is traditionally considered as a quality control activity, with
an emphasis on defect detection and correction. Wewould like to take a
broader viewof testing and believethat thereareaspects of testing that are
also defect prevention oriented. For example, one of the aspects of white
box testing, discussed in Chapter 3, is static testing, that involves desk
checking, code walkthroughs, codereviews, and inspection. Eventhough
thesearetraditionally considered as"quality assurance" activities, planning
for overall testing activities with an intent to deliver quality products to
customers, cannot bedoneeffectivelyunless wetakeaholisticviewof what
can be done using quality assurance and what can be done with quality
control (or thetraditional definitionof testing).
Quality assurance is normally associatedwith process models such as
CMM, CMMI, ISO9001,and so on. Quality control, on the other hand, is
associatedwith testing (that formthebulk of the discussionsin this book).
This has caused an unnatural dichotomy between these two functions.
Unfortunately,organizationsviewthesetwofunctionsasmutually exclusive,
"either-or" choices. Wehave even heard statements such as "with good
processes,testingbecomesredundant" or "processesaremereoverheads-we
Principles of Testing
Figure 1.5
Quality control and
quality assurance as
two methods to achieve
quality.
Dp.fect Detection "
,
,
,
,
, ,
. .
. .
. .
. .
, ,
.......
Defect Prevention
.... \,
: ;
. ,
, ,
........
canfind out everything by testing." It is almost asif there aretwo schools
of thought at either extremes of a pendulum-one rooting for defect
prevention (quality assurance) focus and the other rooting for the defect
detection (quality control) focus. It isalsocommon tofind anorganization
swinging from one extreme to another over time, like a pendulum
(Figure1.5).
Rather than view defect prevention and defect detection as mutually
exclusivefunctions or ends of apendulum, webelieveit is worthwhile to
view these two as supplementary activities, being done in the right mix.
Figure 1.6gives a defect prevention-defect detection grid, which views
the two functions as two dimensions. The right mix of the two activities
corresponds tochoosingtheright quadrant inthis grid.
When thefocuson defect prevention islow, theemphasis ontheuse of
appropriate standards, reviews, and processes arevery low. Thisacts asan
ideal "breeding ground" for defects. Most of theeffort in ensuring quality
of aproduct isleft inthehands of thetesting and defect detection team. If
thefocusondefect detectionisalsolow(represented bythelower left-hand
quadrant), thisisabad statefor anorganization tobein. Lackof testing and
Figure 1.6
Relationship between
defect detection focus
and defect prevention
focus.
Last minute rushes
Higher people dependency
Testers as "heroes" and
"adversaries"
Not a healthy state!
Lack of standards foster
"defect breeding"
Lack of testing makes
defects reach the customers
May be resource intensive but
gives better payback
Institutionalizes quality
Makes quality visible to
customers
Double-edged sword!
Excessive process
orientation
Lack of testing makes
defects reach the customers

Defect prevention focus

18 SoAware Te;~ing"
defect detection activities does not "kill" these defects in time; hence the
defectsreach thecustomers. Thisisobviously not ahealthy statetobein.
Evenwhenthedefectdetectionfocusincreases, withcontinued lowdefect
prevention focus (upper lefthand quadrant), thetestingfunctions becomea
high-adrenalin rush, high-pressure job. Most defectsaredetected inthelast
minute-before the product release. Testersthus becomesuperheroes who
"savetheday" by finding all thedefectsjust intime. Theymay alsobecome
adversaries to developers as they always seemto find problems in what
the developers do. Thisquadrant isbetter than theprevious one, but ends
up being difficult to sustain because the last-minute adrenalin rush burns
people out faster.
Preventing anillnessismoreeffectivethan curingit. Peoplewho prevent
defects usually do not get much attention. They are usually the unsung
heroes of anorganization. Thosewho put out thefiresaretheones who get
visibility, not necessarily those who make sure fires do not happen in the
first place. This, however, should not deter the motivation of people from
defect prevention.
Aswesawintheprevious section, defectprevention and defectdetection
arenot mutually exclusive.Theyneedtobebalancedproperly for producing
a quality product. Defect prevention improves the quality of the process
producing theproducts whiledefectdetection andtestingisneeded tocatch
and correctdefectsthat escapetheprocess. Defectprevention isthus process
focused while defect detection isproduct focused. Defect detection acts as
anextrachecktoaugment theeffectivenessof defect prevention.
An increase in defect prevention focus enables putting in place review
mechanisms, upfront standards tobefollowed, and documented processes
for performing thejob. This upfront and proactive focus on doing things
right to start with causes the testing (or defect detection) function to add
morevalue, andenables catchinganyresidual defects(thatescapethedefect
prevention activities) before the defects reach the customers. Quality is
institutionalized with this consistently high focusonboth defect prevention
and defect detection. An organization may have to allocate sufficient
resources for sustaining ahigh level of both defect prevention and defect
detection activities (upper right-hand quadrant inFigure1.6).
However, anorganization should becareful about not relying toomuch
ondefectprevention andreducing thefocusondefectdetection (lower right-
hand quadrant inFigure1.6).Suchahighfocusondefectprevention andlow
focusondefect detection would not createafeelingof comfort amongst the
management onthequality of product released sincethere arelikely tobe
minimal internal defectsfound. Thisfeelingwill giverisetointroduction of
new processes toimprove theeffectivenessof defect detection. Toomuch of
processes andsuchdefectprevention initiativesmay endupbeingperceived
asabureaucraticexercise,notflexibleoradaptabletodifferentscenarios.While
processesbringindisciplineand reducedependency onspecificindividuals,
they-when not implemented in spirit-could also end up being double-
edged swords, actingas adamper to people's driveand initiative. When an
organization pays equally high emphasis to defect prevention and defect
detection (upper right comer inthegrid), it may appear that it isexpensive
but this investment isbound to havearichpayback by institutional quality
internally andmakingthebenefitsvisibleexternallytothecustomers.
An organization should choosethe right place on each of these two-
defect detection and defect prevention- dimensions and thus choose
the right place in the grid. Therelative emphasis to be placed on the two
dimensions will vary with thetypeof product, closenesstothereleasedate,
and the resources available. Making a conscious choiceof the balance by
considering thevarious factorswill enableanorganization toproduce better
quality products. It isimportant for anorganization not to over-emphasize
oneof theseat theexpenseof theother, asthenext sectionwill show.
1.12 MEN IN BLACK
As we can see fromall the above discussions, testing requires abundant
talent inmultiple dimensions. Peopleinthetesting profession should have
a customer focus, understanding the implications from the customer's
perspective. Theyshould haveadequate analytical skillstobeabletochoose
theright subset of tests and beabletocounter thepesticide paradox. They
should think ahead in terms of defect prevention and yet be able to spot
and rectify errors that crop up. Finally (aswewill seeinthenext section),
they must beabletoperform automation functions.
Despiteall thesechallenging technical and inter-personal skillsrequired,
testing still remains a not-much-sought-after function. There was an
interesting experiment that was described by DeMarco and Lister in their
book, Peopleware [DEMA-1987].Thetestingteamwasseededwithmotivated
people who were "freefromcognitivedissonance that hampers developers '.
when testing their own programs." Theteamwas given an identity (by a
black dress, amidst thetraditionally dressed remainder of theorganization)
20 Software Testing
and tremendous importance. All this increased their pride in work and
made their performance grow by leaps and bounds, "almost likemagic."
Longafter theindividual founding members leftand werereplacedby new
people, the"BlackTeam" continued itsexistenceand reputation.
Thebiggest bottleneck intaking up testing as aprofession isthelack of
self-belief. This lack of self-belief and apparent distrust of the existenceof
career options in testing makes people view theprofession as alaunching
pad to do other softwarefunctions (notably, "development," aeuphemism
forcoding).Asaresult, testersdonot necessarilyseekacareer pathintesting
and develop skepticismtowards theprofession.
Wehave devoted an entire chapter in Part III of the book to career
aspirations and other similar issuesthat peopleface.A part of thechallenge
that is faced is the context of globalization-the need to harness global
resources tostay competitive. Weaddress theorganizational issues arising
out of this inanother chapter inPart III.
1.13 AUTOMATION SYNDROME
Principles ofTeiiting 21
If you go through the story closely there appear to be several reasons
for thecropfailuresthat arenot todowiththeautomation intent at all. The
frustration of the farmer should not bedirected at automation but on the
processfollowedfor automation andtheinappropriate choicesmade. Inthe
second cropcycle, thereason for failurewas lack of skillsand inthethird
cycleit isduetoimproper tool implementation.
Inthefirst cropcycle,thefarmer laid off his workers immediately after
thepurchase of motorcyclesand expectedcostand timetocornedown. He
repeated the same mistake for the third crop cycle. Automation does not
yieldresults immediately.
Themoral of the abovestory as it applies to testing is that automation
requires careful planning, evaluation, and training. Automation may
not produce immediate returns. An organization that expects immediate
returns from automation may end up being disappointed and wrongly
blame automation for their failures, instead of objectivelylooking at their
level of preparedness for automation in terms of planning, evaluation,
and training.
A largenumber of organizations fail intheir automation initiatives and
revert to manual testing. Unfortunately, they conclude-wrongly-that
automation will never work.
Testing,bynature, involvesrepetitivework. Thus, itlendsitself naturally
to automation. However, automation is a double-edged sword. Some of
the points that should be kept in mind whileharping on automation are
asfollows.
~ Know first why you want to automate and what you want to
automate, beforerecommending automation for automation's sake.
~ Evaluate multiple tools. before choosing one as being most
appropriate for your need.
~ Trytochoosetoolstomatchyour needs, rather than changing your
needs tomatch thetool'scapabilities.
~ Trainpeople first before expecting themtobeproductive.
~ Donot expect overnight returns fromautomation.
1.14 PUTTING IT ALL TOGETHER
We have discussed several basic principles of testing in this chapter.
These principles provide an anchor to the other chapters that we have
in rest of the book. We have organized the book into five parts. The
first part (which includes this chapter) is Setting the Context, which sets
the context for the rest of the book. In the chapter that follows, we cover
SoftwareDevelopment Ufe Cycle(SDLC)Models inthecontext of testing,
verification and validation activities.
In Part It Types of Testing, we cover the common types of testing.
Chapters 3through 10coverwhiteboxtesting, blackboxtesting, integration
testing, system and acceptance testing, performance testing, regression
testing, internationalization testing, and adhoctesting.
Part III, Select Topics in Specialized Testing, addresses two specific and
somewhat esoterictesting topics-object oriented testing inChapter 11and
usability and accessibilitytesting inChapter 12.
Part IV,People and Organizational Issues inTesting, provides anoftignored
perspective. Chapter 13 addresses the common people issues like
misconceptions, career path concerns and so on. Chapter 14address the
different organizational structures invoguetosetupeffectivetesting teams,
especially inthecontext of globalization.
The final part, Part V, Test Management and Automation, addresses the
process, management, and automationissuestoensureeffectivetestinginan
organization. Chapter 16discussestest planning management and execution.
Thisdiscussesvariousaspectsof puttingtogether atestplan, trackingatesting
projectandrelatedissues.Chapter17goesintodetailsofthebenefits,challenges,
and approaches in test automation-an area of emerging and increasing
importance in the test community. Thefinal chapter, Chapter 18, goes into
details of what data are required to be captured and what analysis is to
be performed for measuring effectiveness of testing, quality of a product
and similar perspectives and how this information canbe used to achieve
quantifiable continuous improvement.
Whilewehaveprovided thenecessary theoretical foundation indifferent
parts of thebook, our emphasis throughout thebookhasbeenonthestateof
practice. Thissectionshould set thecontext for what the reader canexpect
inrest of thebook.
One of the early seminal works on testing is [MYER-79].In particular,
the example of trying to write test cases for verifying three numbers to
bethe sides of avalid triangle still remains oneof thebest ways tobring
forth the principles of testing. [DEMA-87] provides several interesting
perspectives of the entire software engineering discipline. The concept
of black teamhas beenillustrated inthat work. Theemphasis required for
process and quality assurance methodologies and thebalancetobestruck
between quality assurance and quality control arebrought out in [HUMP-
86]. Someof theuniversally applicable quality principles are discussed in
the classics [CROS-80]and [DEMI-86].[DIJ K-72],aTuringAward lecture
brings out thedoctrine of program testing cannever prove the absenceof
defects. [BEIZ-90]discussesthepesticideparadox.
1. Wehave talked about the pervasiveness of software as a reason
why defects leftinaproduct would get detected sooner than later.
Assume that televisions with embedded software were able to
download, install and self-correct patches over the cablenetwork
automatically and the TV manufacturer told you that this would
just takefiveminutes everyweek "at nocosttoyou, theconsumer."
Wouldyou agree? Givesomereasons why this isnot acceptable.
2. Yourorganization hasbeensuccessful indeveloping aclient-server
application that isinstalled at several customer locations. Youare
changing theapplication tobeahosted, web-based application that
anyonecanuseafter asimpleregistration process. Outline someof
the challenges that you should expect froma quality and testing
perspective of thechanged application.
3. Thefollowing were some of the statements made by people in a
product development organization. Identify the fallacies if any
in the statements and relate it to the principles discussed in this
chapter.
a. "The code for this product is generated automatically by a
CASEtool- it istherefore defect - free."
b. "Wearecertifiedaccording to thelatest process models - we
do not need testing."
c. "Weneedtotest thesoftwarewithdot matrix printers because
wehave never released aproduct without testing with adot
matrix printer."
d. "I have run all thetests that I havebeen running for thelast
two releases and I don't need torun any moretests."
e. "This automation tool is being used by our competitors -
henceweshould alsousethesametool."
4. Assume that each defect in gathering requirements allowed to go
to customers costs $10,000,and that the corresponding costs for
designdefectsandcodingdefectsare$1,000and$100,respectively.
Also, assume that current statistics indicate that on average ten
new defectscomefromeachof thephases. Inaddition, eachphase
also lets the defects fromtheprevious phase seep through. What
is the total cost of the defects under the current scenario? If you
put a quality assurance process to catch 50%of the defects from
eachphase not togotothenext phase, what aretheexpected cost
savings?
5. Youaretowrite aprogram that adds two two-digit integers. Can
you test this program exhaustively? If so, how many test cases
are required? Assuming that each test casecan be executed and
analyzed inonesecond, how longwould it takefor you torun all
thetests?
6. We argued that the number of defects left in a program is
proportional to thenumber of defectsdetected. Givereasons why
this argument looks counterintuitive. Also, givepractical reasons
why thisphenomenon causesproblems intesting.
In this chapter-
,/ Phases of software project
,/ Quality, quality assurance, and quality control
,/ Testing, verification, and validation
,/ Process model to represent different phases
,/ Lifecyclemodels
26 Software jesting
2.1 PHASES OF SOFTWARE PROJECT
A softwareproject ismadeup of aseriesof phases. Broadly,most software
projects comprise thefollowing phases.
~ Requirements gathering and analysis
~ Planning
~ Design
~ Development or coding
~ Testing
~ Deployment and maintenance
2.1.1
Requirements Gathering and Analysis
During requirements gathering, thespecificrequirements of the software
to be built are gathered and documented. If the software is bespoke
software, then thereisasinglecustomer who cangivetheserequirements.
If the product is a general-purpose software, then a product marketing
teamwithin thesoftwareproduct organization specifiestherequirements
by aggregating the requirements of multiple potential customers. In
either case, it is important to ensure that the right requirements are
captured at every stage. The requirements get documented in the form
of aSystemRequirements Specification(SRS)document. This document
actsasabridgebetween thecustomer andthedesigners chartered tobuild
theproduct.
2.1.2 Planning
The purpose of the planning phase is to comeup with a schedule, the
scope, and resource requirements for arelease. A plan explains how the
requirements will bemet andby which time. It needs totakeinto account
the requirements-what will be met and what will not be met-for the
current release to decide on the scope for the project, look at resource
availability, and to comeout with set of milestones and release date for
the project. The planning phase is applicable for both development and
testing activities. At theend of this phase, both project plan and test plan
documents aredelivered.
2.1.3 Design
The purpose of the design phase is to figure out how to satisfy the
requirements enumerated in the System Requirements Specification
document. Thedesign phase produces arepresentation that will beused
bythefollowingphase, thedevelopment phase. Thisrepresentation should
serve two purposes. First, fromthis representation, it should bepossible
toverifythat all therequirements aresatisfied. Second, this representation
Sojtware pevelgpmeftt LifeCycle,Models 27
should givesufficient information for the development phase to proceed
with thecodingand implementation of thesystem. Designisusually split
into two levels-high-Ievel design and low-level or adetailed design. The
design step produces thesystemdesign description (SDD)document that
will beused by development teams to produce the programs that realize
thedesign.
2.1.4 Development or Coding
Designactsasablueprint fortheactual codingtoproceed. Thisdevelopment
orcodingphasecomprisescodingtheprograms inthechosenprogramming
language. Itproduces thesoftwarethat meets therequirements thedesign
was meant tosatisfy.Inaddition toprogramming, this phase alsoinvolves
thecreation of product documentation.
2.1.5 Testing
As the programs are coded (inthe chosen programming language), they
are also tested. In addition, after the coding is (deemed) complete, the
product is subjected to testing. Testing is the process of exercising the
softwareproduct inpre-defined ways to checkif thebehavior isthesame
as expected behavior. By testing the product, an organization identifies
and removes asmany defects aspossiblebeforeshipping it out.
2.1.6 Deployment and Maintenance
Onceaproduct istested, it isgiventothecustomers who deploy it intheir
environments. Astheusers start using theproduct intheir environments,
theymayobservediscrepancies between theactual behavior of theproduct
and what they were given to expect (either by the marketing people or
through theproduct documentation). Suchdiscrepancies could end up as
product defects, which need to becorrected. Theproduct now enters the
maintenancephase, wherein theproduct ismaintained or changedtosatisfy
thechangesthat arisefromcustomer expectations, environmental changes,
etc. Maintenance is made up of corrective maintenance (for example, fixing
customer-reported problems), adaptive maintenance (for example, making
the software run on anew version of an operating system or database),
andpreventive maintenance (forexample, changingtheapplication program
codetoavoid apotential security holeinanoperating systemcode).
2.2 QUALITY, QUALITY ASSURANCE, AND QUALITY CONTROL
A software product is designed to satisfy certain requirements of a
given customer (or set of customers). How can we characterize this
phrase-"satisfying requirements"? Requirements get translated into
softwarefeatures, eachfeature being designed tomeet oneor more of the
28 Software Testing
requirements. For eachsuch feature, theexpected behavior is characterized
by aset of test cases. Eachtest caseisfurther characterized by
1. Theenvironment under which thetest caseistobeexecuted;
2. Inputs that should beprovided for that test case;
3. How theseinputs should get processed;
4. What changes should be produced in the internal state or
environment; and
5. What outputs should beproduced.
Theactual behavior of a given software for a given test case, under a
givenset of inputs, inagivenenvironment, and inagiveninternal stateis
characterizedby
1. Howtheseinputs actually get processed;
2. What changes are actually produced in the internal state or
environment; and
3. What outputs areactually produced.
If the actual behavior and the expected behavior are identical in all
their characteristics,thenthat test caseissaidtobepassed. If not, thegiven
softwareissaidtohaveadefect onthat test case.
Howdoweincreasethechancesof aproduct meetingtherequirements
expected of it, consistently and predictably? There are two types of
methods-quality control andquality assurance.
Quality control attempts tobuildaproduct, testitfor expectedbehavior
after it isbuilt, and if theexpectedbehavior is not the sameas the actual
behavior of theproduct, fixestheproduct asisnecessary and rebuilds the
product. Thisiterationisrepeated till theexpectedbehavior of theproduct
matchestheactual behavior for thescenariostested. Thusquality control is
defect-detectionand defect-correctionoriented, and works on theproduct
rather thanontheprocess.
Quality assurance, on the other hand, attempts defect prevention by
concentratingontheprocessof producing theproduct rather than working
on defect detection/correction after the product is built. For example,
instead of producing and then testingaprogramcodefor proper behavior
by exercisingthebuilt product, aquality assuranceapproach would beto
first review the design beforethe product is built and correct the design
errorsinthefirstplace. Similarly,toensuretheproduction of abetter code,
aquality assuranceprocessmay mandate codingstandards tobefollowed
by all programmers. As can be seen fromthe above examples, quality
assurance normally tends to apply to all the products that use aprocess.
Also, sincequality assurancecontinues throughout thelifeof theproduct
it iseverybody's responsibility; henceit is astaff function. Incontrast, the
responsibility for quality control is usually localized to a quality control
team. Table2.1summarizesthekeydistinctionsbetweenqualitycontrol and
quality assurance.
Software Development Life Cycle lVfodeis 29
Table 2.1. Difference between quality assurance and quality control.
Wewill seemore details of quality assurance methods such as reviews
and audits inChapter 3. But thefocus of therest of this book ison software
testing, which is essentially aquality control activity. Let us discuss more
about testing inthenext section.
2.3 TESTING, VERIFICATION, AND VALIDATION
Thenarrow definition of theterm"testing" isthephase that follows coding
and precedes deployment. Testing is traditionally used to mean testing of
the program code. However, coding is adownstream activity, as against
requirements and design that occur much earlier in a project life cycle.
Given that the objective of asoftware project is to minimize and prevent
defects, testing of program code alone is not sufficient. As we saw in the
lastchapter, defectscancreepinduring anyphaseandthesedefectsshouldbe
detectedasclosetothepoint of injectionaspossibleandnot waittill thetesting
of programs. Hence against this, if each phase is "tested" separately as and
when thephase iscompleted (or,better still, asthephase isbeing executed),
then defectscanbedetected early,thereby reducing theoverall costs.
Timely testing increases thechances of aproduct or servicemeeting the
customer's requirements. When a product is tested with appropriate and
realistic tests that reflect typical usage patterns by the intended users, the
chancesof theproduct satisfyingthecustomer's requirement ismuchhigher.
While testing does not guarantee zero defects, effective testing certainly
increases thechances of customer acceptance of thesoftware.
Thepurpose of testing is to uncover defects in the system (and to have
someone fixthedefects). Testingisdonebyaset of peoplewithin asoftware
product (or service) organization whose goal and charter isto uncover the
defects intheproduct beforeit reaches thecustomer (seeSection1.3).Aswe
sawintheprevious chapter, thepurpose of testing isNOT toprove that the
product has nodefects. Thepurpose of softwaretestingistofinddefectsina
softwareproduct. Aswewill seeinthechapters onpeopleandorganizational
Sofrwar~<Iesting
issues (Chapters 13,14), thereward systemsandtheorganization structures
should create and foster an environment that encourages this purpose
of testing.
Testing is NOT meant to replace other ways of ensuring quality (like
reviews). It is one of the methods to detect defects in asoftware product.
There are other methods that achievethe same function. For example, we
will seelater that following well-defined processes and standards reduces
thechances of defects creeping into asoftware. Wewill also discuss other
methods like reviews and inspections, which actually attempt to prevent
defectscomingintotheproduct. Tobeeffective,testingshould complement,
supplement, and augment suchquality assurancemethods discussed inthe
previous section.
The idea of catching defects within each phase, without letting them
reach the testing phase, leads us to define two more terms-verification
andvalidation.
Duringtherequirements gatheringphase, therequirements arefaithfully
captured. TheSRSdocument is theproduct of therequirements phase. To
ensure that requirements arefaithfully captured, thecustomer verifies this
document. The design phase takes the SRSdocument as input and maps
therequirements toadesignthat candrivethecoding. TheSDDdocument
istheproduct of thedesignphase. TheSDDisverifiedby therequirements
teamtoensurethat thedesignfaithfullyreflectstheSRS,whichimposed the
conditions at thebeginning of thedesignphase.
Verification takes care of activities to focus on the question "Are we
building the product right?" and validation takes careof aset of activities to
address thequestion "Are we building the right product?"
Tobuild the product right, certain activities/conditions/procedures are
imposed at thebeginning of the lifecycle. Theseactivities are considered
"proactive" astheir purpose istoprevent thedefectsbeforethey takeshape.
The process activities carried out during various phases for each of the
product releasescanbetermed asverification. Requirements review, design
review, and codereviewaresomeexamples of verificationactivities.
To build the right product, certain activities are carried out during
various phases tovalidatewhether theproduct isbuilt asper specifications.
Theseactivities areconsidered "reactive" astheir purpose isto find defects
that affect theproduct and fixthemas soon as they areintroduced. Some
examples of validation includeunit testing performed toverify if thecode
logicworks, integration testing performed toverifythedesign, and system
testingperformed toverifythat therequirements aremet.
Tosummarize, there aredifferent terminologies that may stand for the
same or similar concepts. For all practical purposes in this book, we can
assumeverificationand quality assurancetobeoneand thesame. Similarly
quality control, validation, and testingmeanthesame.
Software Development Life
2.4 PROCESSMODEL TO REPRESENT DIFFERENT PHASES
A process model is a way to represent any given phase of software
development that effectively builds in the concepts of validation and
verification to prevent and minimize the delay between defect injection
and defect detection (and eventual correction). Inthis model, each phase
of asoftwareproject ischaracterized by thefollowing.
~ Entry criteria, which specify when that phase canbestarted. Also
included aretheinputs for thephase.
~ Tasks, or steps that need tobecarried out inthat phase, along with
measurements that characterize thetasks.
~ Verification, which specifies methods of checking that the tasks
havebeen carried out correctly.
~ Exit criteria, which stipulate the conditions under which one can
consider thephase as done. Alsoincluded arethe outputs for only
thephase.
Thismodel, known as theEntry Task VerificationeXitor ETVXmodel,
offersseveral advantages for effectiveverificationand validation.
1. Clear entry criteria make sure that a given phase does not start
prematurely.
2. The verification for each phase (or each activity in each phase)
helps prevent defects, or at least, minimizes thetimedelay between
defect injectionand defect detection.
3. Documentation of the detailed tasks that comprise each phase
reduces theambiguity ininterpretation of theinstructions and thus
minimizes the variations that can comefromrepeated executions
of thesetasks by different individuals.
4. Clear exit criteriaprovide ameans of validation of thephase, after
thephase isdonebut beforehanding over tothenext phase.
Anexampleof applyingtheETVXmodel tothedesignphaseispresented
inFigure2.1.
Figure 2.1
ETVX model applied
to design. Entry criteria:
Approval of SRS by customer
08
Input:
Approved SRS
Exit criteria:
Complete traceability between
design and SRS
Development team ready to start
programming
D8
*
; ti
,::>:
Output:
Architecture documents
Design documents
Program specifications
32 Software Testing
2.5 LIFE CYCLEMODELS
The ETVXmodel characterizes aphase of aproject. A LifeCyclemodel
describes how thephases combine together to formacompleteproject or
lifecycle. Suchamodel ischaracterized by thefollowingattributes.
The activities performed In any given software project, apart fromthe
most common activities or phases-requirements gathering, design,
development, testing, and maintenance-there couldbeother activities as
well. Some of these activities could be technical activities (for example,
porting) and somecouldbenon-technical (forexample, hiring).
The deliverables from each activity Each activity produces a set of
deliverables, which aretheend products of that activity. For example, the
requirements gathering phase produces the SRSdocument, the design
phase produces theSODdocument, and soon.
Methods of validation of the deliverables Theoutputs produced by a
given activity represent thegoal tobesatisfiedby that activity. Henceit is
necessary tohaveproper validation criteriafor eachoutput.
The sequence of activities The different activities work together in
unison inacertain sequence of steps to achieveoverall project goals. For
example, theprocess of requirements gathering may involvesteps suchas
interviews with customers, documentation of requirements, validation of
documented requirements with customers, and freezing of requirements.
These steps may be repeated as many times as needed to get the final
frozen requirements.
Methods of verification of each activity, including the mechanism of
communication amongst the activities The different activities interact
with one another by means of communication methods. For example,
when adefect is found inoneactivity and istraced back to the causes in
anearlier activity, proper verificationmethods areneeded to retrace steps
fromthepoint of defect tothecauseof thedefect.
Wewill nowlookat someof thecommonlifecyclemodels that areused
insoftwareprojects. Foreachmodel, wewill lookat:
1. abrief description of themodel;
2. the relationship of the model to verification and validation
activities; and
3. typical scenarios where that lifecyclemodel isuseful.
2.5.1 Waterfall Model
IntheWaterfall model, aprojectisdivided intoasetof phases (oractivities).
Eachphaseisdistinct, that is,thereareclearlinesof separation between the
phases, with very clear demarcation of thefunctions of eachof thephases.
Cycle Models
Figure 2.2
Waterfall model.
A project starts with an initial phase, and upon completion of the phase,
moves on to the next phase. On the completion of this phase, the project
moves to the subsequent phase and so on. Thus the phases are strictly
timesequenced.
Wedepict oneexampleof aprojectintheWaterfall model inFigure2.2.
Theproject goesthrough aphase of requirements gathering. At theend of
requirements gathering, aSystemRequirements Specificationdocument is
produced. Thisbecomestheinputto thedesignphase.Duringthedesignphase,
adetailed design isproduced inthe formof aSystemDesignDescription.
WiththeSODas input, theproject proceeds to the development or coding
phase, wherein programmers develop theprograms required to satisfy the
design. Oncethe programmers completetheir coding tasks, they hand the
product tothetestingteam, whotesttheproduct beforeit isreleased.
If there is no problem in a given phase, then this method can work,
going in one direction (likeawaterfall). But what would happen if there
are problems after going to aparticular phase? For example, you go into
thedesignphase and findthat it isnot possibletosatisfy therequirements,
34 Software Testing
going by the current design approach being used. What could be the
possiblecauses and remedies? Youmay try analternative design if possible
and seeif that cansatisfytherequirements. If therearenoalternative design
approaches possible, thentheremust befeedback totherequirements phase
to correct therequirements.
Letustaketheexampleonestepfurther. Supposeadesignwascreatedfor
agiven set of requirements and theproject passed ontotheprogramming/
development phase. Atthispoint of time, itwasfound thatitwasnot possible
todevelop theprograms becauseof somelimitations. What would you do?
Oneapproach would betotryout alternative strategies inthedevelopment
phase sothat thedesign couldstill besatisfied. Another possibility couldbe
that there areflaws indesign that causeconflictsduring d.evelopment and
hence the design has to berevisited. When thedesign phase is revisited-
likein theprevious case-it may happen that theproblemmay have tobe
addressed intherequirements phaseitself.So,aprobleminonephasecould
potentially betracedback toany of theprevious phases.
Sinceeachphasehasanoutput, thelatter canbevalidated against aset of
criteria. Toincreasetheeffectiveness, thecompletion criteriafor eachoutput
canbepublished apriori. Beforeaphasestarts, thecompletioncriteriaforthe
previous phase canbecheckedand this canactasaverificationmechanism
for the phase. This can minimize the kind of delays we discussed in the
exampleabove.
The main strength of the Waterfall Model is its simplicity. The model
is very useful when a project can actually be divided into watertight
compartments. But very few software projects can be divided thus. The
major drawback in the Waterfall model arises fromthe delay in feedback
amongthephases, andthus theineffectivenessof verificationandvalidation
activities. An error in onephase isnot detected till at least thenext phase.
When a given phase detects an error, the communication is only to the
immediately preceding phase. This sequential nature of communication
among thephases canintroduce inordinate delays inresolving theproblem.
Thereduced responsiveness that isinherent inthemodel and thefact that
thesegregation of phases isunrealistic severely restricts theapplicability of
thismodel.
2.5.2 Prototyping and Rapid Application Development Models
Prototyping andRapidApplication Development (RAD)models recognize
and address thefollowing issues.
1. Early and frequent user feedback will increase the chances of a
software project meeting thecustomers' requirements.
2. Changes are unavoidable and the software development process
must beableto adapt itself torapid changes.
Software Development Life Cycle Models 35
ThePrototyping model comprises thefollowing activities.
1. The software development organization interacts with customers
tounderstand their requirements.
2. Thesoftwaredevelopment organizationproduces aprototypetoshow
how the eventual software systemwould look like. This prototype
would havethemodels of how theinput screensand output reports
wouldlooklike,inaddition tohavingsome"empty canfunctionality"
todemonstrate theworkflowand processinglogic.
3. The customer and the development organization review the
prototype frequently so that the customer's feedback is taken very
early inthecycle(that is, during therequirements gathering phase).
4. Basedonthefeedbackandtheprototype thatisproduced, thesoftware
development organization produces theSystemRequirements Spe-
cificationdocument.
5. Once the SRSdocument is produced, the prototype can be dis-
carded.
6. The SRS document is used as the basis for further design and
development.
Thus, theprototype issimply used as ameans of quickly gathering (the
right) requirements. This model has built-in mechanisms for verification
and validation of therequirements. Astheprototype isbeing developed, the
customer'sfrequent feedbackactsasavalidationmechanism. OncetheSRSis
produced, it actsastheverificationmechanismfor thedesignand subsequent
steps. But theverificationand validation activitiesof the subsequent phases
areactually dictated by thelifecyclemodel that isfollowedafter theSRSis
obtained.
Thismodel isobviously advantageous when acustomer canparticipate
by giving feedback. This model is also useful in cases where the feedback
can be easily quantified and incorporated, for example, determining user
interface, predicting performance, and soon.
For a general-purpose product, which is meant for many customers,
there isno single customer whose feedback canbe taken as final. In these
cases, a product manager in the marketing group of the product vendor
usually plays the roleof theeventual customer. Hence the applicability of
this model issomewhat limited togeneral-purpose products. Furthermore,
the prototype is used as a means of capturing requirements and is not
necessarily meant tobeused afterwards. Oftentimes, theprototype (or parts
of the prototype) makes its way to becoming the product itself. This can
have undesirable effects as the prototype usually employs several short
cuts, unstructured methods, and tools to achieveaquick turnaround. Such
short cuts arepotential sources of defectsinliveenvironments and thus can
placeaheavy burden onmaintenance and testing.
The Rapid Application Development model is a variation of the
Prototyping Model. LikethePrototyping Model, theRADModel relies on
feedback andinteractionbythecustomers togather theinitial requirements.
36 Software Testing
However, the Prototyping model differs from the RAD Madelon
two counts.
First, in theRADModel, it isnot aprototype that isbuilt but the actual
product itself.Thatis,thebuilt application(prototype, intheprevious model)
isnot discarded. Hence, itisnamed RapidApplicationDevelopment model.
Second, inorder toensure formalismincapturing therequirements and
proper reflectionof therequirements inthedesignand subsequent phases, a
Computer Aided SoftwareEngineering (CASE)tool isused throughout the
lifecycle,right fromrequirements gathering. SuchCASEtoolshave
ffi methodologies to elicit requirements;
ffi repositories tostorethegathered requirements and all downstream
entities such as design objects; and
ffi mechanisms to automatically translate the requirements stored
in the repositories to design and generate the code in the chosen
programming environment.
Themethodologies provided byaCASEtool canprovideinbuilt means of
verificationandvalidation. Forexample, thetool maybeabletoautomatically
detect and resolveinconsistencies in data types or dependencies. Sincethe
design(and, perhaps, eventheprogramcode)canbeautomatically generated
fromtherequirements, thevalidation canbevery complete, extending toall
thedownstream phases, unlikethePrototyping model.
This method can have wider applicability for even general-purpose
products. Theautomatic generation of thedesign and programs produced
by aCASEtool makes this model more attractive. Thecost of such CASE
toolsisafactor that anorganization would havetoconsider beforedeciding
ontheuseof thismodel for agivenproject. In addition, CASEtools and this
model isgenerally moresuited for applications projectsrather than systems
typeprojects.
2.5.3 Spiral or Iterative Model
TheSpiral or Iterative model follows aprocess inwhich the requirements
gathering, design, coding, and testing are performed iteratively till all
requirements aremet. There is also agood amount of overlap among the
activities of requirements gathering, design, coding, and testing following
this model. What phase the product is in is difficult to conclude as each
requirement can be at adifferent phase. Theonly conclusion that can be
made isat what phase each of therequirements isin. If adefect isproduced
inanyphaseof agivenrequirement, itmay causethat requirement torevisit
an earlier phase. This model enables incremental development whereby
the product evolves, with requirements getting added to it dynamically.
Thisenables theproduct tobedemonstrated, at any point of time, with the
functionality availableatthat point of time. Italsoenables the"increments"
tobesent tothecustomer for approval. Theprogress of theproduct canbe
Software Development Life Cycle Models 37
Table 2.2 Some product requirements and phases.
Figure 2.3
Spiral model.
seenfromthebeginning of theproject as themodel delivers "increments"
at regular intervals. Eventhough it will bevery difficult to plan arelease
date following this model, it allows the progress to be tracked and the
customer approvals to be obtained at regular intervals, thereby reducing
the risk of finding major defects at alater point of time. Table2.2gives an
example of phases for someof therequirements intheproduct.
Figure2.3(thecoloured figureisavailableonpage457)depicts theSpiral
model andthephases involvedinthemodel, for theexampleonTable2.2.As
canbeseen, eachrequirement is"spiraling outwards" through thedifferent
phases astheentireproject evolves.
2.5.4 The V Model
TheWaterfall Model viewed testing as apost-development (that is, post-
coding) activity. The Spiral Model took this one step further and tried
to break up the product into increments each of which can be tested
38
separately. The V Model starts off being similar to the Waterfall Model
in that it envisages product development to bemade up of anumber of
phases or levels. However, the new perspective that the V Model brings
in is that different types of testing apply at different levels. Thus, froma
testing perspective, thetypeof tests that need tobedoneat eachlevel vary
significantly.
Forinstance, consider atypical product development activityrepresented
as a Waterfall Model earlier in Figure 2.2. The system starts with the
overall business requirements fromthepoint of view of customers. These
requirements cover hardware, software, and operational requirements.
Since our focus is on the software, moving from overall requirements
to software requirements becomes the next step. In order to realize the
softwarerequirements, theproposed softwaresystemisenvisaged as aset
of subsystems that work together. This high-level design (of breaking the
systeminto subsystems with identified interfaces) then gets translated to
amore detailed or low-level design. This detailed design goes into issues
like data structures, algorithm choices, table layouts, processing logic,
exceptionconditions, and soon. It results intheidentification of anumber
of components, each component realized by program code written in
appropriate programming languages.
Giventheselevels, what kind of tests apply ineachof these levels?To
beginwith, foroverall businessrequirements, eventuallywhatever software
isdeveloped should fitintoand work inthis overall context and should be
acceptedby theendusers, intheir environment. Thistesting, thefinal proof
of thepudding, isacceptance testing. But, beforetheproduct isdeployed in
thecustomer's environment, theproduct vendor should test it as anentire
unit to make sure that all the software requirements are satisfied by the
product that isdeveloped. Thistestingof theentiresoftwaresystemcanbe
calledsystem testing. Sincehigh-level designviewsthesystemasbeingmade
up of interoperating and integrated (software) subsystems, the individual
subsystems should be integrated and tested together before a full blown
systemtest canbe done. This testing of high-level design corresponds to
integration testing. The components that are the outputs of the low-level
design have tobetested independently beforebeing integrated. Thus, the
testing corresponding to the low-level design phase is component testing.
Finally,sincecodingproduces several programunits, eachof thesesmaller
program units have to be tested independently before trying to combine
themtogether toformcomponents. Thistestingof theprogramunits forms
. unit testing.
Figure2.4depicts thedifferent types of testing that apply toeachof the
steps. For simplicity,wehavenot shown theplanning phase as aseparate
entity sinceit is common for all testing phases. But, it is not possible to
execute any of thesetestsuntil theproduct isactuallybuilt. Inother words,
thestepcalled"testing" isnowbroken down intodifferent sub-steps called
acceptancetesting, systemtesting, andsoonasshowninFigure2.4.So,itis
still thecasethat all thetestingexecution related activitiesaredone only at
theendof thelifecycle.
Figure 2.4
Phases of testing for
different development
phases.
Software Development Life Cycle Models 39
,/
,/
,/
,/
,/
,/
,/
,/
,/
,/
,/
,/
Even though the execution of the tests cannot be done till the product is
built, the design of tests can be carried out much earlier. In fact, if we look at
theaspect of skill sets required for designing each type of tests, thepeople best
suited to design each of these tests arethose who areactually performing the
function of creating the corresponding artifact. For example, the best people
to articulate what the acceptance tests should beare the ones who formulate
the overall business requirements (and, of course, the customers, where
possible). Similarly, the people best equipped to design the integration tests
arethose who know how the system isbroken into subsystems and what the
interfaces between the subsystems are-that is, those who perform thehigh-
level design. Again, the people doing development know the innards of the
program code and thus arebest equipped to design the unit tests.
Not only are the skill sets required for designing these different types of
tests different, but also, there is no reason to defer the designing of the tests
till the very end. As and when each activity on the left-hand side of the "V"
is being carried out, the design of the corresponding type of tests can be
carried out. By performing an early design of the tests and deferring only
the test execution till the end, we achieve three important gains.
~ First, we achieve more parallelism and reduce the end-of-cycle time
taken for testing.
~ Second, by designing tests for each activity upfront, we arebuilding
in better upfront validation, thus again reducing last-minute
surprises.
~ Third, tests are designed by people with appropriate skill sets.
40 softwar( Testing
.................. " 'M_"M.~~WJ ._
'J ; ~ ,*,i;0\,@?% iicW>" ' .. '/ -) :::7";-
;;~:::@~2:::;:~t'(, ':"" ":;, ,'- ,.','.+',{ /: ,'H'
Figure 2.5
V Model.
Acceptance
test design
Unit test design
Thisisthebasisfor theVModel, whichpresents excellentadvantages for
verificationand validation. Asshown inFigure2.5,for eachtypeof test, we
movethedesignof testsupstream, alongwiththeactual activitiesandretain
thetest execution downstream, after theproduct isbuilt.
2.5.5 Modified V Model
TheV Model split the design and execution portion of the various types
of tests and attached the test design portion to the corresponding earlier
phases of thesoftware lifecycle.
An assumption made there was that even though the activity of test
execution was split into execution of tests of different types, the execution
cannot happen until the entire product is built. For a given product, the
different units and components canbeindifferent stages of evolution. For
example, oneunit couldbestill under development and thus beintheunit-
testing phase whereas another unit could be ready for component testing
while thecomponent itself may not beready for integration testing. There
maybecomponents that areready (thatis,component tested) for integration
and being subjected to integration tests (alongwith other modules which
are also ready for integration, provided those modules canbeintegrated).
TheV Model doesnot explicitlyaddress thisnatural parallelismcommonly
found inproduct development.
In the modified V Model, this parallelism is exploited. When eachunit
or component or module is given explicit exit criteria to pass on to the
subsequent stage, theunits or components or modules that satisfy agiven
phase of testing moveto thenext phase of testing where possible, without
Figure 2.6
Modified V Model.
necessarily waiting for all theunits or components or modules to move in
unison fromonephase of testing toanother, asshown inFigure2.6.
J ust as the V Model introduced various types of testing, the modified
V model introduces various phases of testing. A phase of testing has aone-
to-onemapping tothetypes of testing, that is, thereisaunit-testing phase,
component-testing phase, and so on. Once aunit has completed the unit-
testing phase, it becomes part of acomponent and enters the component-
testing phase. It then moves to integration-testing phase and soon. Rather
than view the product as going through different types of tests (as the
V model does), the modified V Model views each part of the product to
go through different phases of testing. Theseare actually two sides of the
samecoinand thus provide complimentary views. Themain advantage the
modified V model brings to the table is the recognition of the parallelism
present indifferent parts of theproduct and assigning eachpart tothemost
appropriate phase of testing that is possible. In Figure 2.6, the columns of
thetablerepresents onesideof V,androws (whicharetest phases) represent
theother sideof V.
InFigure2.6,noticethat different phases of testing aredoneinparallel.
While starting a phase of testing it is important to look at whether the
product is ready for testing. It is determined by aset of entry criteria. The
earliest possiblequality tostart thenext phase of testingisdenoted by entry
criteria, and tostart thenext phaseof testing theearlier phaseneed not have
completed. Thetesting phases arealso associated with aset of exit criteria
to complete the test activities for each phase. They are determined by exit
criteria. Theentry and exit criteria for each of the phases ensure that right
quality of product delivered for starting thetest and right amount of testing
iscompleted for therelease. Eventhough it isindicated inthepicture all of
thetest phases finishat thesametime, practically it canhavedifferent time
lines. Thelongest phase determines thereleasedate.
InFigure2.6,therearetwoadditional activitiesthathavenotbeendiscussed
before. Thecolouredfigureis availableonpage458.Theseare"Component
(1,2...) Complete" and "Components Complete"; these arenot additional
Design Unit test Components Components
IT complete
system
complete complete (1,2...) complete
complete
complete
Unit testing
,
l~l
Component
testing
Entry criteria ______________
Integration
testing
~
System testing
~
Acceptance
Exit criteria
testing
42 SQftwateTesting
phases in a life cycle. They have been introduced just to denote that
integration testing canstart after two components havebeen completed,
and when all components are integrated and tested, the next phase of
testing, that is, systemtesting canstart.
Table 2.3 Model applicability and relevance to verification and validation.
2.5.6 Comparison of Various Life Cycle Models
As can be seen from the above discussion, each of the models has its
advantages and disadvantages. Eachof themhas applicability inaspecific
scenario. Each of them also provides different issues, challenges, and
opportunities for verification and validation. Wesummarize in Table2.3
the salient points about applicability and relevance to verification and
validation for eachof themodels.
TheWaterfall Model was initially coveredin [ROYC-70].Theorigins of the
Prototyping Model comefrom[BROO-75].TheSpiral Model was originally
proposed in [BOEH-88].[GRAO-97]provides somevariations to theSpiral
Model. [RAME-2002],[PRES-97]and [HUMP-86]provide overviews to all
themodels.
1. WhichSOLCmodel wouldbemost suitablefor eachof thefollowing
scenarios?
a. Theproduct isabespokeproduct for aspecificcustomer, who
isalways availabletogivefeedback.
b. Thesameasabove, except that wealsohaveaccesstoaCASE
tool that generates program codeautomatically.
c. A general purpose product, but with a very strong product
marketing team who understand and articulate the overall
customer requirements very well.
d. A product that is made of anumber of features that become
availablesequentially and incrementally.
2. Which of the following products would you say is of "high
quality," based on the criteria we discussed in the book? J ustify
your answer.
a. Three successive versions of the product had respectively 0,
79,and 21defects reported.
b. Threesuccessiveversions of theproduct had respectively 85,
90, and 79defects reported.
3. List three or more challenges fromthetesting perspective for each
of thefollowing models:
a. Spiral Model.
b. V Model.
c. Modified V Model.
I
4. What are some of the challenges that you should expect when
moving fromtheV Model totheModifiedV Model?
5. Figure2.1gavetheETVXDiagramforthedesignphaseof asoftware
project. Draw asimilar ETVXDiagramfor thecoding phase.
6. In the book wehave discussed the Spiral Model as being ideally
suited for aproduct that evolves in increments. Discuss how the
V Model isapplicable for such anincrementally evolving product.
TvpesofTe
a Chapter 6
System and Acceptance Testing
a Chapter 7
Performance Testing
a Chapter 8
Regression Testing
a Chapter 9
Internationalization (1
18
n) Testing
a Chapter 10
Ad hoc Testing
Thispart of thebook discusses various types of tests. Thechapters
progress fromthetypesof testsclosertocodetothoseclosertousers.
Whitebox testing, which tests theprograms by having aninternal
knowledge of program code, is discussed in Chapter 3. Blackbox
testing, which tests the product behavior by only knowing the
external behavior asdictated by therequirements specifications, is
discussed in Chapter 4. As software gets developed in amodular
fashion andthemodules havetobeintegrated together, integration
testing is covered in Chapter 5. System and acceptance testing,
which tests a product completely from a user's perspective in
environments similar to customer deployments, is discussed
in Chapter 6. Performance testing, which tests the ability of the
systemtowithstand typical and excessivework loads, isdiscussed
inChapter 7.Sincesoftwareisalways characterized by changeand
sincechanges should not break what isworking already, regression
testing becomes very important and is discussed in Chapter 8. As
softwarehastobedeployed inmultiple languages acrosstheworld,
internationalization testing, thetopic of Chapter 9, comesinto play.
Finally, adhoc testing, in Chapter 10, addresses the methods of
testing aproduct intypical unpredictable ways that end users may
subject theproduct to.
In this chapter-
./ What iswhite box testing?
./ Static testing
./ Structural testing
./ Challenges in white box testing
3.1 WHAT IS WHITE BOX TESTING?
Everysoftwareproduct isrealized bymeans of aprogram code. Whitebox
testingisawayof testingtheexternal functionalityofthecodebyexamining
and testing theprogram codethat realizes theexternal functionality. This
isalsoknown asclear box, or glass box or open box testing.
Whitebox testing takes into account theprogram code, codestructure,
and internal design flow. Incontrast, black box testing, to bediscussed in
Chapter 4, doesnot lookat theprogramcodebut looksat theproduct from
anexternal perspective.
A number of defects come about because of incorrect translation of
requirements anddesignintoprogramcode. Someother defectsarecreated
by programming errors and programming language idiosyncrasies. The
different methods of white box testing discussed inthis chapter canhelp
reduce the delay between the injection of adefect in the program code
and its detection. Furthermore, sincethe program code represents what
the product actually does (rather than what the product is intended to
do), testing by looking at the program codemakes us get closer to what
theproduct isactually doing.
As shown inFigure3.1, whitebox testingis classifiedinto "static" and
"structural" testing. The corresponding coloured version of Figure 3.1 is
available on page 458. Wewill look into the details of static testing in
Section3.2andtakeup structural testinginSection3.3.
Figure 3.1
Classification of white
box testing.
3.2 STATIC TESTING
Static testing isatypeof testing which requires only thesourcecodeof the
product, not the binaries or executables. Static testing does not involve
Whfte Box Testing 49
executing the programs on computers but involves select people going
through thecodetofindout whether
>I< thecodeworks according tothefunctional requirement;
>I< thecodehasbeenwritten inaccordancewith thedesign developed
earlier intheproject lifecycle;
>I< thecodefor any functionality has beenmissed out;
>I< thecodehandles errors properly.
Statictestingcanbedonebyhumans or withthehelpof specializedtools.
3.2.1 Static Testing by Humans
Thesemethods rely ontheprinciple of humans reading theprogram code
to detect errors rather than computers executing the code to find errors.
Thisprocess has several advantages.
1. Sometimes humans can find errors that computers cannot. For
example, when there aretwo variables with similar names and the
programmer used a"wrong" variableby mistakeinanexpression,
the computer will not detect the error but execute the statement
and produce incorrect results, whereas a human being can spot
such anerror.
2. By making multiple humans read and evaluate the program, we
can get multiple perspectives and therefore have more problems
identified upfront than acomputer could.
3. A human evaluation of the code can compare it against the
specifications or design and thus ensure that it does what is
intended to do. Thismay not always bepossiblewhen acomputer
runs atest.
4. A human evaluation can detect many problems at onego and can
eventrytoidentify theroot causesof theproblems. Moreoftenthan
not, multiple problems canget fixedby attending to thesameroot
cause. Typically,inareactive testing, atest uncovers oneproblem
(or, at best, a few problems) at a time. Often, such testing only
reveals thesymptoms rather than theroot causes. Thus, theoverall
time required to fixall the-problemscanbereduced substantially
by ahuman evaluation.
5. By making humans test the code before execution, computer
resources can be saved. Of course, this comes at the expense of
human resources.
6. A proactive method of testing like static testing minimizes the
delay inidentification of theproblems. AswehaveseeninChapter
1, thesooner adefect isidentified and corrected, lesser isthecost of
fixingthedefect.
50 .Software Testing
7. Fromapsychological point of view, finding defectslater inthecycle
(forexample, after thecodeiscompiled and thesystemisbeing put
together) creates immense pressure onprogrammers. Theyhaveto
fixdefects with lesstimeto spare. Withthis kind of pressure, there
arehigher chances of other defects creeping in.
Therearemultiplemethods toachievestatictestingbyhumans. Theyare(in
theincreasingorder of formalism) asfollows.
1. Desk checkingof thecode
2. Codewalkthrough
3. Codereview
4. Codeinspection
Sincestatic testing by humans is done before the code is compiled and
executed, some of these methods can be viewed as process-oriented or
defect prevention-oriented or quality assurance-oriented activities rather
than pure testing activities. Especiallyasthemethods becomeincreasingly
formal (for example, Fagan Inspection), these traditionally fall under the
"process" domain. They find a place in formal process models such as
ISO9001,CMMI, and soonand areseldomtreated aspart of the "testing"
domain. Nevertheless, asmentioned earlier inthis book, wetake aholistic
view of "testing" as anything that furthers thequality of aproduct. These
methods havebeenincluded inthischapter becausetheyhavevisibilityinto
theprogramcode.
Wewill nowlookintoeachof thesemethods inmoredetail.
3.2.1.1 Desk checking Normally done manually by the author of
the code, desk checking is a method to verify the portions of the code
for correctness. Suchverification is done by comparing the codewith the
design or specificationstomakesurethat thecodedoeswhat itissupposed
to do and effectively. This is the desk checking that most programmers
do before compiling and executing the code. Whenever errors are found,
the author applies the corrections for errors on the spot. This method of
catching and correcting errors ischaracterized by:
1. No structured method or formalismtoensure completeness and
2. No maintaining of alogor checklist.
In effect, this method relies completely on the author's thoroughness,
diligence, and skills. There is no process or structure that guarantees or
verifies the effectiveness of desk checking. This method is effective for
correcting" obvious" codingerrorsbut will notbeeffectiveindetectingerrors
that arise due to incorrect understanding of requirements or incomplete
requirements. Thisisbecausedevelopers (or,moreprecisely, programmers
who are doing the desk checking) may not have the domain knowledge
required tounderstand therequirements fully.
White Box Testing
The main advantage offered by this method is that the programmer
who knows the code and the programming language very well is well
equipped to read and understand his or her own code. Also, since this
is done by one individual, there are fewer scheduling and logistics over-
heads. Furthermore, the defectsare detected and correctedwith minimum
timedelay.
Someof thedisadvantages of thismethod of testing areasfollows.
1. A developer isnot thebest person to detect problems inhis or her
own code. Heor shemay betunnel visioned and haveblind spots
to certain types of problems.
2. Developers generally prefer to write new coderather than do any
formof testing! (Wewill seemore details of this syndrome later in
thesectiononchallenges aswell aswhen wediscuss people issues
inChapter 13.)
3. This method is essentially person-dependent and informal and
thus may not work consistently across all developers.
Owing tothesedisadvantages, thenext two types of proactive methods are
introduced. Thebasic principle of walkthroughs and formal inspections is
toinvolvemultiplepeopleinthereviewprocess.
3.2.1.2 Codewalkthrough Thismethodandformalinspection(described
inthenextsection)aregroup-orientedmethods. Walkthroughsarelessformal
than inspections. The line drawn in formalism between walkthroughs
and inspections isvery thin and varies fromorganization toorganization.
Theadvantage that walkthrough has over desk checking is that it brings
multiple perspectives. Inwalkthroughs, asetof people lookat theprogram
codeand raisequestions for theauthor. Theauthor explains thelogicof the
code, and answers the questions. If the author is unable to answer some
questions, he or she then takes those questions and finds their answers.
Completeness is limited to the area where questions are raised by the
team.
3.2.1.3 Formal inspection Codeinspection-also calledFaganInspection
(namedaftertheoriginalformulator)- isamethod,normallywithahighdegree
of formalism. The focus of this method is to detect all faults, violations,
and other side-effects. This method increases the number of defects
detected by
1. demanding thorough preparation before aninspection/review;
2. enlisting multiple diverse views;
3. assigning specificroles tothemultiple participants; and
4. goingsequentially through thecodeinastructured manner.
A formal inspection should takeplaceonly when theauthor has made sure
the code is ready for inspection by performing somebasic desk checking
and walkthroughs. Whenthecodeisinsuchareasonablestateof readiness,
52 Software Testing
an inspection meeting is arranged. There arefour roles ininspection. First
is theauthor of thecode. Secondis amoderator who isexpected to formally
run the inspection according to the process. Third aretheinspectors. These
arethepeople who actually provides, review comments for thecode. There
aretypically multiple inspectors. Finally,thereisascribe, who takesdetailed
notes during the inspection meeting and circulates themto the inspection
teamafter themeeting.
The author or the moderator selects the review team. The chosen
members have the skill sets to uncover as many defects as possible. In an
introductory meeting, the inspectors get copies (Thesecanbe hard copies
or soft copies) of the code to be inspected along with other supporting
documents suchasthedesign document, requirements document, and any
documentation of applicable standards. Theauthor alsopresents his or her
perspective of what theprogram is intended to do along with any specific
issuesthat heor shemay want theinspection teamtoput extrafocuson. The
moderator informstheteamabout thedate, time, andvenueof theinspection
meeting. The inspectors get adequate time to go through the documents
and program and ascertain their compliance to the requirements, design,
and standards.
The inspection team assembles at the agreed time for the inspection
meeting (alsocalledthedefect logging meeting). Themoderator takestheteam
sequentially through the program code, asking each inspector if there are
any defects inthat part of the code. If any of theinspectors raises adefect,
then the inspection team deliberates on the defect and, when agreed that
thereisadefect,classifiesitintwodimensions-minor/major andsystemic/mis-
execution. Amis-executiondefectisonewhich, asthenamesuggests, happens
because of an error or slip on the part of the author. It is unlikely to be
repeated later, either in this work product or in other work products. An
example of this isusing awrong variablein astatement. Systemic defects,
on the other hand, canrequire correction at adifferent level. For example,
an error such as using some machine-specific idiosyncrasies may have to
removed by changing the coding standards. Similarly, minor defects are
defects that may not substantially affectaprogram, whereas major defects
need immediate attention.
A scribeformally documents thedefectsfound intheinspection meeting
and the author takes care of fixing these defects. In case the defects are
severe, theteammay optionally call for areviewmeetingtoinspect thefixes
toensurethat they address theproblems. Inany case,defectsfound through
inspectionneed tobetracked till completion andsomeoneintheteamhas to
verify that theproblems havebeen fixedproperly.
3.2.1.4 Combining various methods Themethods discussed above
arenot mutually exclusive. Theyneedtobeused inajudicious combination
tobeeffectiveinachieving thegoal of finding defects early.
White Box Testini 53
Formal inspections have been found very effectivein catching defects
early. Some of the challenges to watch out for in conducting formal
inspections areasfollows.
1. Theseare time consuming. Sincethe process calls for preparation
aswell asformal meetings, thesecantaketime.
2. The logistics and scheduling can become an issue since multiple
people areinvolved.
3. It is not always possible to go through every line of code, with
several parameters and their combinations in mind to ensure the
correctness of thelogic, side-effectsandappropriate error handling.
It may also not be necessary to subject the entire code to formal
inspection.
In order to overcome the above challenges, it is necessary to identify,
during theplanning stages, whichparts of thecodewill besubject toformal
inspections. Portions of codecanbeclassifiedonthebasisof their criticality
or complexity as "high," "medium," and "low." High or medium complex
critical codeshould besubject toformal inspections, while those classified
as"low" canbesubjecttoeither walkthroughs or evendesk checking.
Desk checking, walkthrough, review and inspection are not only used
for codebut canbeused for all other deliverables in theproject lifecycle
suchasdocuments, binaries, and media.
3.2.2 Static Analysis Tools
Thereviewandinspection mechanisms described aboveinvolvesignificant
amount of manual work. Thereareseveral static analysis tools availablein
the market that canreduce the manual work and perform analysis of the
codetofindout errors such asthoselisted below.
1. whether there are unreachable codes (usage of GOTO statements
sometimes creates this situation; there could beother reasons too)
2. variables declared but not used
3. mismatch indefinition and assignment of values tovariables
4. illegal or error prone typecasting of variables
5. use of non-portable or architecture-dependent programming
constructs
6. memory allocated but not having corresponding statements for
freeing themup memory
7. calculation of cyclomatic complexity (coveredintheSection3.3)
Thesestaticanalysistoolscanalsobeconsidered asanextensionof compilers
asthey usethesameconcepts and implementation tolocateerrors. A good
compiler isalsoastaticanalysistool. Forexample, most Ccompilersprovide
different "levels" of code checking which will catch the various types of
programming errors givenabove.
54 Software Testing
Some of the static analysis tools can also check compliance for coding
standards as prescribed by standards such as POSIX. These tools can also
check for consistency incoding guidelines (for example, naming conventions,
allowed data types, permissible programming constructs, and so on).
While following any of the methods of human checking-desk checking,
walkthroughs, or formal inspections-it is useful to have a code review
checklist. Given below is checklist that covers some of the common issues.
Every organization should develop its own code review checklist. The
checklist should bekept current with new learning as they come about.
In amulti-product organization, the checklist may be at two levels-first,
anorganization-wide checklist that will include issues such asorganizational
coding standards, documentation standards, and so on; second, aproduct-
or project-specific checklist that addresses issues specific to the product
or project.
CODE REVIEW CHECKLIST
56 Software Testing
3.3 STRUCTURALTESTING
Structural testing takes into account the code, code structure, internal
design, and how they are coded. The fundamental difference between
structural testing and static testing is that in structural testing tests are
actually runbythecomputer onthebuilt product, whereas instatictesting,
the product is tested by humans using just the source code and not the
executables or binaries.
Structural testing entails running the actual product against somepre-
designed test casestoexerciseasmuch of thecodeaspossibleor necessary.
A givenportion of thecodeisexercisedif atest casecauses theprogram to
executethat portion of thecodewhen running thetest.
As discussed at thebeginning of this chapter, structural testing canbe
further classifiedintounit/code functional testing, codecoverage, and code
complexity testing.
3.3.1 Unit/Code Functional Testing
Thisinitial part of structural testing corresponds tosomequick checksthat
a developer performs before subjecting the code to more extensive code
coverage testing or code complexity testing. This can happen by several
methods.
1. Initially, thedeveloper canperform certain obvious tests, knowing
the input variables and the corresponding expected output
variables. This can be a quick test that checks out any obvious
mistakes. By repeating these tests for multiple values of input
variables, theconfidencelevel of thedeveloper togotothenext level
_1~, .. ~~-h. k
:!jfr:1W!t"f\W J 4,o$ <i, *
0/.~,. ~<.-,:,:,:,:,:,. ,.' ,..... .'. " ;~.-';.;h~
White Box Testing 57
increases. This can even be done prior to formal reviews of static
testing sothat thereviewmechanismdoesnot wastetimecatching
obvious errors.
2. For modules with complex logic or conditions, the developer can
builda/I debugversion" of theproduct byputting intermediate print
statements and making sure the program is passing through the
right loops anditerations theright number of times. Itisimportant
to remove the intermediate print statements after the defects
arefixed.
3. Another approach todotheinitial test istorun theproduct under a
debugger or anIntegrated Development Environment (IDE).These
tools allowsinglestepping of instructions (allowing thedeveloper
to stop at theend of eachinstruction, view or modify the contents
of variables, and so on), setting break points at any function or
instruction, andviewingthevarious systemparameters or program
variablevalues.
All the abovefall moreunder the "debugging" category of activities than
under the"testing" category of activities. All thesame, theseareintimately
related to the knowledge of code structure and hence we have included
theseunder the "whiteboxtesting" head. Thisisconsistent with our view
that testing encompasses whatever it takes to detect and correct defects in
aproduct.
3.3.2 Code Coverage Testing
Sinceaproduct is realized in terms of program code, if we can run test
casestoexercisethedifferent parts of thecode, thenthat part of theproduct
realized by thecodegetstested. Codecoveragetesting involves designing
and executing test cases and finding out the percentage of code that is
covered by testing. Thepercentage of code covered by atest is found by
adopting atechnique calledinstrumentation of code. Therearespecialized
tools available to achieve instrumentation. Instrumentation rebuilds the
product, linking the product with aset of libraries provided by the tool
vendors. This instrumented code canmonitor and keep an audit of what
portions of codearecovered. Thetoolsalsoallowreporting ontheportions
of the code that are covered frequently, so that the critical or most-often
portions of codecanbeidentified.
Codecoveragetestingismadeup of thefollowingtypes of coverage.
1. Statement coverage
2. Pathcoverage
3. Condition coverage
4. Function coverage
..~i~::;;):I~;Ii:~
3.3.2.1 Statement coverage Program constructs in most conven-
tional programming languages canbeclassifiedas
1. Sequential control flow
2. Two-way decision statements likeif then else
3. Multi-way decision statements likeSwitch
4. Loops likewhile do, repeat until and for
Object-oriented languages have all of the above and, in addition, a
number of other constructs and concepts. Wewill takeup issues pertaining
to object oriented languages together in Chapter 11. Wewill confine our
discussions heretoconventional languages.
Statement coveragerefers to writing test cases that executeeach of the
programstatements. Onecanstart with theassumption that morethecode
covered, thebetter isthetestingof thefunctionality, asthecoderealizesthe
functionality. Basedonthis assumption, codecoveragecanbeachieved by
providing coveragetoeachof theabovetypes of statements.
For a section of code that consists of statements that are sequentially
executed (that is, with no conditional branches), test casescanbedesigned
to run through fromtop tobottom. A test casethat starts at thetop would
generally have to go through thefull sectiontill thebottomof the section.
However, this may not always be true. First, if there are asynchronous
exceptions that the codeencounters (for example, adivide by zero), then,
evenif westart atest caseat thebeginning of asection, thetest casemay not
cover all thestatements inthat section. Thus, eveninthecaseof sequential
statements, coverage for all statements may not be achieved. Second, a
section of code may be entered frommultiple points. Even though this
points tonot followingstructured programming guidelines, it isacommon
scenarioinsomeof theearlier programming languages.
When weconsider atwo-way decision construct likethe if statement,
then to cover all thestatements, weshould also cover the then and else
parts of the if statement. This means weshould have, for each if then
else, (atleast) onetest casetotest theThenpart and (atleast) onetest case
totest thee1separt.
The multi-way decision construct such as a Switch statement can be
reduced to multiple two-way if statements. Thus, to cover all possible
switch cases, therewouldbemultipletest cases. (Weleaveit asanexercise
for thereader todevelopthis further.)
Loop constructs present more variations to take care of. A loop-in
various forms such as for, while, repeat, and so on-is characterized
byexecutingasetof statements repeatedly until or whilecertainconditions
aremet. A good percentage of thedefectsinprograms comeabout because
of loops that do not function properly. Moreoften, loops fail in what are
called"boundary conditions." Oneof thecommonloopingerrors isthat the
termination condition of theloop is not properly stated. Inorder to make
,'~ite Box '{estin$
sure that there is better statement coverage for statements within a loop,
there should betest casesthat
1. Skip the loop completely, so that the situation of the termination
condition being true before starting theloop istested.
2. Exercisetheloopbetween onceand themaximumnumber of times,
to check all possible "normal" operations of theloop.
3. Try covering the loop, around the "boundary" of n-that is, just
below n, n, and just aboven.
The statement coverage for a program, which is an indication of the
percentage of statements actually executedinaset of tests, canbecalculated
by theformula givenalongside inthemargin.
It is clear from the above discussion that as the type of statement
progresses from a simple sequential statement to if then else and
through to loops, the number of test cases required to achieve statement
coverageincreases.TakingacuefromtheDijkstra'sDoctrinein Chapter I, just
asexhaustivetestingof all possibleinput dataonaprogramisnot possible, so
alsoexhaustivecoverageof all statementsinaprogramwill alsobeimpossible
forall practical purposes.
Evenif weweretoachieveaveryhighlevel of statement coverage, itdoes
not mean that theprogram isdefect-free. First, consider ahypothetical case
when we achieved 100percent codecoverage. If the program implements
wrong requirements and this wrongly implemented codeis "fully tested,"
with 100percent codecoverage, it still is awrong program and hence the
100percent codecoveragedoes not mean anything.
Next, consider thefollowingprogram.
Total = Oi /* set total to zero */
if (code == " M" ) {
stmtli
stmt2i
Stmt3i
stmt4i
Stmt5i
stmt6i
Stmt7i
else percent = value/Total*ldO
i
/* divide by zero */
In the above program, when we test with code = "M," we will get
80percent code coverage. But if the data distribution in the real world is
such that 90percent of the time, the value of codeis not ="M," then, the
program will fail 90percent of the time (becauseof the divide by zero in
thehighlighted line). Thus, evenwith acodecoverageof 80percent, weare
left with adefect that hits theusers 90percent of thetime. Path coverage,
discussed inSection3.3.2.2,overcomes thisproblem.
60 Software Testing
3.3.2.2 Path coverage Inpathcoverage,wesplitaprogramintoanumber
of distinct paths. A program (or apart of aprogram) can start fromthebe-
ginning and takeany of thepaths toits completion.
Let us take anexample of adate validation routine. Thedate isaccepted
asthree fieldsrom, ddand yyyy. Wehaveassumed that prior toentering this
routine, thevalues arechecked tobenumeric. Tosimplify thediscussion, we
haveassumed theexistenceof afunction called leapyear whichwill return
TRUEif the given year is aleap year. Thereis an array called DayofMonth
which contains thenumber of days ineach month. A simplified flow chart
for this isgiveninFigure3.2below.
Ascanbeseenfromthefigure, therearedifferent paths that canbetaken
through the program. Eachpart of the path is shown in red. The coloured
representation of Figure3.2isavailableonpage459. Someof thepaths are
>I<
A
>I< B-D-G
>I< B-D-H
>I< B-C-E-G
>I<
B-C-E-H
>I<
B-C-F-G
>I<
B-C-F-H
Regardless of the number of statements in each of these paths, if we
can execute these paths, then we would have covered most of the typical
scenarios.
False
'1J
'J I False
(j
True
511True
Figure 3.2
Flowchartforadate
validationroutine.