Blackboxtesting

Readings:RonPattonchap5,DanielGalinchap9.5.1 Withmaterialfrom:R.Binder,TestingObjectOrientedSystems

WhatisBlackBoxtesting?

Testingwithouthavinganinsightintothedetails ofunderlyingcode
input Unit output

Testingbasedonspecification

WhatisBlackBoxtesting?
Advantages noneedforsourcecode canbeusedforunitaswellassystemleveltesting Disadvantages don'ttesthiddenfunctions Approaches: Errorguessing EquivalencePartitioning BoundaryValueAnalysis Decisiontables CauseEffectGraphing Testsderivedfromfunctionalrequirements(e.g.usecases)

ErrorGuessing(1)

Adhocapproachbasedonexperience Approach:
1.Makelistofpossibleerrorsorerrorpronesituations

errormodel

2.Designtestcasestocovererrormodel

Developandmaintainyourownerrormodels

ErrorGuessing(2)

Examplesortingarrayfunction Errormodel:

emptyarray alreadysortedarray reversesortedarray largeunsortedarray ...

Generatetestcasesforthesesituations

EquivalencePartitioning

Partitioninputdomaininequivalenceclasses accordingtospecification IdeallyECsshouldbe

suchthateachinputisaclass disjointed elementsofsameclassmappedtooutputsimilarly useofheuristics

DifficultinpracticetofindperfectECs

HeuristicsforIdentifyingEquivalence Classes
Foreachexternalinput:
1. ifspecifiesrangeofvalidvalues,define

1validEC(withintherange) 2invalidEC(oneoutsideeachendoftherange) 1validECand 2invalidECs(noneandmorethanN) 1validEC(withinset)and 1invalidEC(outsideset)

2. ifspecifiesanumber(N)ofvalidvalues,define

3. ifspecifiesasetofvalidvalues,define

HeuristicsforIdentifyingEquivalence Classes
Foreachexternalinput:
4. ifthereisareasontobelievethatprogramhandlevalidinput differently,define

1validECpervalidinput 1validEC(satisfymust)and 1invalidEC(don'tsatisfymust)

5. ifspecifiesamustbesituation,define

6. ifthereisareasontobelievethatelementsinanECarenot handledinanidenticalmannerbytheprogram,

subdivideECintosmallerEcs.

EquivalenceClassPartitioning

Considercreatinganequivalencepartitionthat handlethedefault,empty,blank,null,zero,or noneconditions.

EquivalenceClassesPartitioning TriangleExample(1)

Specification

inputthreeintegers(sidesofatriangle:a,b,c)

eachsidemustbeapositivenumberlessorequalto20. Equilateralifa=b=c Isoscelesif2pairsofsidesareequals Scaleneifnopairofsidesisequalor NotATriangleifa>=b+c,b>=a+c,orc>=a+b

outputtypeofthetriangle:

EquivalenceClassesPartitioning TriangleExample(2)
Accordingtoheuristic#5
InputCondition Sides(a,b,c) ValidEC All>0&<=20(1) InvalidEC a>20(2) b>20(3) c>20(4) a<=0(5) b<=0(6) c<=0(7)

EC#1toobroadcanbesubdivided(heuristic#6)

EquivalenceClassesPartitioning TriangleExample(3)

1.1.a,b,csuchthatthetriangleisequilateral 1.2.a,b,csuchthatthetriangleisisosceles 1.3.a,b,csuchthatthetriangleisscalene 1.4.a,b,csuchthatit'snotatriangle

Basedonthetreatmenttodatahandlingofdata

1.5.a=b=c 1.6.a=b,a<>c 1.7.a=c,a<>b 1.8.b=c,a<>b 1.9.a<>b,a<>c,b<>c

Basedoninput

1.10.a,b,csuchthata>=b+c 1.11.a,b,csuchthatb>=a+c 1.12.a,b,csuchthatc>=a+b

Basedontriangleproperty

EquivalenceClassesPartitioning Identifyingtestcases
1.UntilallvalidECshavebeencoveredbytestcases,

writeanewtestcasethatcoverasmanyofthe uncoveredvalidECsaspossible

2.UntilallinvalidECshavebeencoveredbytestcases,

writeatestcasethatcoversone,andonlyoneofthe uncoveredinvalidECs

EquivalenceClassesPartitioning TriangleExample(4)

Testcasesfortriangle
EC 1.1,1.5 1.2,1.6 1.3,1.9 1.4,1.10 2 3 4 5 6 7 a 12 6 9 8 30 10 7 5 12 8 b 12 6 3 2 15 100 14 10 10 5 c 12 7 7 3 6 99 21 11 10 1 ExpectedOutput Equilateral Isosceles Scalene Notatriangle

EquivalenceClassesPartitioning Nextdate(1)

Input:month,day,yearrepresentingadate

1<=month<=12 1<=day<=31 1812<=year<=2012

Output:dateofthedayaftertheinputdate

Mustconsiderleapyears Leapyearifdivisibleby4,andnotacenturyyear Centuryyearisleapyearsifmultipleof400

EquivalenceClassesPartitioning Nextdate(2)

Equivalenceclasses
InputCondition month day year ValidEcs 1<=month<=12 1<=day<=31 1812<=year<=2012 InvalidEcs month<1 month>12 day<1 day>31 year<1812 year>2012

Refinementcanbedonebasedonhowinputistreated Dayincrementedby1unlessit'slastdayofamonth Dependonmonth(30,31,February) Atendofmonthnextdayis1,monthisincremented Atendofyear,bothdaysandmonthsareresetto1,andyear incremented Problemofleapyear

EquivalenceClassesPartitioning Nextdate(3)

DecompositionofvalidECs
InputCondition month ValidEcs 30days(1) 31days(2) February(3) 1<=day<=28(4) day=29(5) day=30(6) day=31(7) year=1900(8) 1812<=year<=2012and year<>1900and yearmod4=0(9) 1812<=year<=2012and yearmod4<>0(10)

day

year

Bettercoverageobtainedbyselectingtestcasesaccording CartesianproductofECs 36testcases.

EquivalenceClassesPartitioning Nextdate(4)

Testcases
ID 1 2 3 4 5 6 7 8 9 10 11 12 Ecs month 1,4,8 6 1,4,9 6 1,4,10 6 1,5,8 6 1,5,9 6 1,5,10 6 1,6,8 6 1,6,9 6 1,6,10 6 1,7,8 6 1,7,9 6 1,7,10 6 day 14 14 14 29 29 29 30 30 30 31 31 31 year 1900 1912 1913 1900 1912 1913 1900 1912 1913 1900 1912 1913 output 15/06/1900 15/06/1912 15/06/1913 30/06/1900 30/06/1912 30/06/1913 01/07/1900 01/07/1912 01/07/1913 Error Error Error

EquivalenceClassesPartitioning Problems

Specificationdoesn'talwaysdefineexpectedoutputfor invalidtestcases Stronglytypedlanguageseliminatetheneedforthe considerationofsomeinvalidinputs Bruteforceofdefiningatestcaseforeverycombination oftheinputsECs

providesgoodcoverage,but impracticalwhennumberofinputsandassociatedclassesis large

BoundaryValueAnalysis

Errorstendtooccurnearextremevalues (boundaries) BVAenhancesEquivalencePartitioningby

selectingelementsjuston,andjustbeyondthe bordersofeachEC derivingtestcasesbyconsideringECsfromoutputas well

BoundaryConditions

Situationattheedgeoftheplannedoperationallimitsofthe software Datatypeswithboundaryconditions

Numeric,Character,Position,Quantity,Speed,Location,Size First/Last,Start/Finish,Min/Max,Over/Under,Empty/Full, Slowest/Fastest,Largest/Smallest,NextTo/FarthestFrom, Shortest/Longest,Soonest/Latest,Highest/Lowest

Boundaryconditioncharacteristics

BoundaryValueAnalysis Example

Examplesofboundaryvaluesfornextdate

Input month day year

BoundaryValue 0,1,12,13 0,1,30,31,32 1811,1812,2012,2013

BoundaryValueAnalysis Guidelines

Foreachboundarycondition

includeboundaryvalueinatleastonevalidtestcase includevaluejustbeyondboundaryinatleastoneinvalidtest case

Usesameguidelinesforeachoutputcondition

includetestcaseswithinputsuchthatoutputatboundariesare produced

BoundaryValueAnalysis

SubBoundaryConditions

internaltosoftware(notdefinedinspecification) examples:

Powersoftwo:bit,nibble,byte,word,kilo,mega,giga,tera ASCIItable

BoundaryValueAnalysis
Character / lowerbound 0 1 2 3 4 5 6 7 8 upperbound 9 : A a ASCIICode 47 48 49 50 51 52 53 54 55 56 57 58 65 97

Ifnumbersare expected(conversion fromascii),'/'and':' arepossibleboundary values

DecisionTable

Idealforsituationswhere:

combinationsofactionstakenundervaryingsetofconditions conditionsdependsoninputvariables responseproduceddoesn'tdependontheorderinwhichinput variablesaresetorevaluated,and responseproduceddoesn'tdependonpriorinputoroutput

DecisionTableFormat

Conditions

Combinationof conditions(variants)

Actions

Selectedactions

DecisionTableDevelopment
1. Identifydecisionvariablesandconditions 2. Identifyresultantactionstobeselectedorcontrolled 3. Identifywhichactionshouldbeproducedinresponseto particularcombinationsofactions

Example

Supposethefollowinginsurancerenewalrules

0claims,age<=25:raiseby$50 0claims,age>25:raiseby$25 1claim,age<=25:raiseby$100,sendletter 1claim,age>25:raiseby$50 2,3or4claims,age<=25:raiseby$400,sendletter 2,3or4claims,age>25:raiseby$200,sendletter morethan5reclamations:cancelpolicy

DecisionTableTriangleExample

Decisionvariables:sidesa,b,c Conditions

Actions

findingitisnotatriangle findingitisascalenetriangle findingitisanisoscelestriangle findingitisanequilateral triangle findingitisanerror findingvariantimpossible

a<b+c b<a+c c<a+b a=b a=c b=c a>20 b>20 c>20

DecisionTableTriangleExample
c1:a<b+c c2:b<a+c c3:c<a+b c4:a=b c5:a=c c6:b=c c7:a>20 c8:b>20 c9:c>20 a1:notatriangle a2:Scalene a3:Isosceles a4:Equilateral a5:impossible a6:error

1 N N N N X

2 Y N N N N X

3 Y Y N N N N X

4 Y Y Y Y Y Y N N N

5 Y Y Y Y Y N N N N

6 Y Y Y Y N Y N N N

7 Y Y Y Y N N N N N

8 Y Y Y N Y Y N N N

9 Y Y Y N Y N N N N

10 Y Y Y N N Y N N N

11 Y Y Y N N N N N N X

12 Y

13 Y

14 Y

X X X X X

Attentionforinconsistencies

DecisionTableDon'tCare condition

Don'tCarecondition

Maybetrueorfalsewithoutchangingtheaction Simplifiesthedecisiontable Correspondstodifferentimplementationcases:

inputsarenecessarybuthavenoeffectforthevariant inputsmaybeomittedbuthavenoeffectifsupplied TypeSafeexclusion

severalconditionsaredefinedforanonbinary decisionvariable,whichcanholdonlyonevalueat atime

DecisionTableCan'tHappen& Don'tknowconditions
AttentionnottoconfuseaDon'tCareconditionwith

Can'tHappenConditionreflectsassumptionthat

someinputsaremutuallyexclusive, someinputscan'tbeproducedbytheenvironment,or implementationisstructuredsoastopreventevaluation

Don'tKnowConditionreflectsanincompletemodel
testsneededtoexercisetheseundefinedcases

Usuallyindicationofmisspecification

AllExplicitvariants

DecisionTableTestgeneration strategies
atestcaseforeachexplicitvariant atestcaseforeachimplicitvariant
2n variantsfor nconditions

AllVariants

AllTrue

atestcaseforeachvariantwithanoutcome(action) atestcaseforeachvariantwithoutoutcome

AllFalse

DecisionTableTestgeneration Triangleexample

AllExplicitVariants
Variant a 1 4 2 1 3 1 4 5 5 6 7 2 8 9 2 10 3 11 3 12 21 13 6 b 1 4 2 5 b 2 2 4 5 Expectedoutput NotaTriangle NotaTriangle NotaTriangle Equilateral Impossible Impossible Isosceles Impossible Isosceles Isosceles Scalene Error Error

2 3 2 4 6 45

3 2 2 5 12 17

EachCondition/AllConditions

DecisionTableTestgeneration strategies
Heuristictoreducenumberofvariants Foreachvariable,avariantsuchthatthevariableismade truewithallothervariablesbeingfalse,and

Setofvariantsincludes

Onevariantsuchthatallvariablesaretrue(andlogic),or Onevariantsuchthatallvariablesarefalse(orlogic)

DecisionTableEachCondition/All Conditions
S=PorQorR

P F F T F

Q F T F F

R S (action) T x F x F x F

DecisionTableEachCondition/All Conditions
S=PandQandR

P F F T T

Q F T F T

R S (action) T F F T x

DecisionTableEachCondition/All Conditions
Z=(AandBand(notC))or(AandD)
Developvariantsfor(AandBand(notC))andvariantsfor(AandD)

A T F F T T F T

B F T F T -

C T T F F -

D F T T

Z (action)

Don'tCarevariables randomlyor bysuspicion

Valuesmustbeassignedto

DecisionTableEachCondition/All Conditions
Z=(AandBand(notC))or(AandD)
A T F F T T F T B F T F T T T T C T T F F F T T D T F F F F T T Z (action)

fortestsuitegeneration

Exampleofvariantsused

DecisionTableEachCondition/All Conditions

Providesacompacttable

numberoftestsincreaselinearlywithnumberof productterms

Assumesindependenceofconditionevaluation, andabsenceofdefaultactions

DecisionTableBinaryDecision DiagramDeterminants
BinaryDecisionDiagram(BDD)compactrepresentationofa truthtable 1.ConstructBDDfromthetruthtable a)Createadecisiontreefromthetruthtable

NodesrepresentBooleanvariables Leftbranchisalwaysfalse Rightbranchisalwaystrue Eachleafrepresentsresultantvalueforconditionon thepathfromroottoleaf

DecisionTableBinaryDecision DiagramDeterminants
Example:Z=Aand(BorC) DecisionTree A B C 0 0 0 0 C 1 1 C 1

B C 0

DecisionTableBinaryDecision DiagramDeterminants
b)ReducedecisiontreetoaBDD

Fromlefttoright,replaceleafnodeswithequivalent constantsorvariablesandprunethebranches A B C C 0 0 0 0 C 1 1 B C 1

DecisionTableBinaryDecision DiagramDeterminants

A 0 C 0 1 1 B C 1

DecisionTableBinaryDecision DiagramDeterminants

A 0 C 1 B C 1

DecisionTableBinaryDecision DiagramDeterminants

A 0 variant1 B C 1 variant2 variant3

DecisionTableBinaryDecision DiagramDeterminants
2.MapBDDintoBDDdeterminanttable

BDDdeterminantroottoleafpathinBDD
BDD Variant 1 2 3 A F T T B F T C C Z F C T

DecisionTableBinaryDecision DiagramDeterminants
3.GenerateBDDtestsuite
BDD Variant 1 2 2 3 A F T T T B F F T C F T Z

x x

Don'tCarevariables randomlyor bysuspicion

Valuesmustbeassignedto

DecisionTableVariableNegation

StrategydesignedtorevealfaultsresultingfromDon'tCare implementation

BDDdeterminantstrategydoesnotexplicitlyaddressDon't Carevariables

SpecificationmustbegivenassumofproductsBooleanformula ((a11anda12...anda1k)or(a21anda22...anda2l)or...) Producesatestsuitewith:

onevariantforeachproducttermsuchthatnootherproducttermis true(uniquetruepoints) onevariantforeachtermthatresultswheneachliteralineach producttermisnegatedsuchthatthefunctionevaluatestofalse (nearfalsepoints)

DecisionTableVariableNegation

Example:Z=Aand(BorC)=(AandB)or(AandC)

1. Uniquetruepoints
(AandB)true,(AandC)false{A=T,B=T,C=F} (AandB)false,(AandC)true{A=T,B=F,C=T}

DecisionTableVariableNegation

Example:Z=Aand(BorC)=(AandB)or(AandC)

2. Nearfalsepoints
(AandB)

(~AandB){A=F,B=T,C=T} (Aand~B){A=T,B=F,C=F} (~AandC){A=F,B=F,C=T} (Aand~C){A=T,B=F,C=F}

(AandC)

Causeeffectgraphing

Systematicwaytoaidselectingahighyieldset oftestcases

toidentify/analyzerelationshipsinadecisiontable generatebooleanformula CausedistinctinstanceofinputconditionorEC Effectobservableresultofchangeinsystemstate Intermediatenodecombinationofcauses representinganexpressiononinputconditions

Nodes:

Causeeffectgraphinglayout

........ Causes Intermediatenodes Effects

Validlinksfrom causetointermediatenode causetoeffects intermediatenodetointermediatenode intermediatenodetoeffect

Causeeffectgraphingtypeoflinks
directcause ^ AND

OR

NOT

Causeeffectgraphingexample
Afileupdatedependsonthevalueintwofields.Thevalue

infield1mustbean``A''or``B''.Thevalueinfield2must be a digit. In this situation the file update is made. If the value in field 1 is incorrect, error message X12 is issued. If thevalueinfield2isincorrect,errormessageX13isissued.

Causeeffectgraphingexample

Causes

c1:characterincolumn1is''A'' c2:characterincolumn1is''B'' c3:characterincolumn2isadigit e1:updatemade e2:messageX12issued e3:messageX13issued

Effects

Causeeffectgraphingconstraints betweencauses
c1 Exclusivec1andc2cannotbetruesimultaneously E c2 c1 c2 c1 c2 c1 c2 OneandonlyOneoneandonlyoneofc1andc2canbe trueatagiventime Exclusiveoneofc1andc2mustbetrue(can'tbeboth false) Requirec2mustbetrueforc1tobetrue

Causeeffectgraphingconstraints betweeneffects
e1 e2 M Maske1forcedtofalseife2

TestgenerationfromCauseEffect graphs(1)

Dividethespecificationintoworkablepieces. Defineseparategraphsforeachpiece.
1. IdentifyCausesandEffects 2. DeduceLogicalRelationshipsandConstraints representasgraph 3. Drawadecisiontable 4. Converteachvariantinthedecisiontableintoatest case.

TestgenerationfromCauseEffect graphs(2)

Tofindtheuniquecombinationofconditions Workwithasingleeffect(output)atatime
1. Seteffecttotrue(1)state 2. Lookbackwardforallcombinationofinputswhich willforceeffecttothetruestate(constraintslimit numberofcombination) 3. Createacolumnindecisiontableforeach combination 4. Determinestatesofallothereffectsforeach combination

BooleanFormulaGenerationfrom CauseEffectgraphs

Workingfromeffectstowardcauses

1. Transcribenodetonodeformulasfromthegraph
writeformulaforeacheffectanditspredecessors Foreachintermediatenode

writeformulaforintermediatenodeanditspredecessors

2. DerivethecompleteBooleanformula

replaceintermediatevariablesbysubstitutionuntiltheeffect formulacontainsonlycausevariables factorandrewriteformulaintosumofproductsform

BooleanFormulaGenerationfrom CauseEffectgraphs
A B C D E
~
~

^ F

v H v I

I=H G H=FC F=~AB G=D~E

^ G

I=(~AB)C(D~E )

TestCasesderivationfrom requirements

Involveclarificationandrestatementoftherequirements

suchthatrequirementsaretestable enumeratesinglerequirements grouprelatedrequirements Onetestcaseshowingrequirementfunctioning Onetestcasetryingtofalsify

Obtainapointformformulation

Foreachrequirement

e.g.buttryingsomethingforbidden

Testboundariesandconstraintswhenpossible

TestCasesderivationfrom requirements

Example:RequirementsforaVideoRentalsystem
1.Thesystemshallallowhiringandreturningoffilms

1.1. Ifafilmisavailableforhirethenitmaybelenttoacustomer.

1.1.1Afilmisavailableforhireuntilallcopieshavebeensimultaneously borrowed.

1.2.Ifafilmwasunavailableforhire,thenreturningthefilmmakesitavailable. 1.3.Thereturndateisestablishedwhenthefilmislentandmustbeshownwhen thefilmisreturned. 1.4.Itmustbepossibleforaninquiryonahiredfilmtorevealthecurrentborrower. 1.5.Aninquiryonamemberwillrevealanyfilmstheycurrentlyhaveonhire.

TestCasesderivationfrom requirements

Testsituationsforrequirement1.1

Attempttoborrowanavailablefilm. Attempttoborrowanunavailablefilm. Attempttoborrowafilmforwhichtherearemultiplecopies,all ofwhichhavebeenhired. Attempttoborrowafilmforwhichallcopiesbutonehavebeen hired.

Testsituationsforrequirement1.1.1

TestCasesderivationfrom requirements

Testsituationsforrequirement1.2

Borrowanunavailablefilm. Returnafilmandborrowit

Testsituationsforrequirement1.3

Borrowafilm,returnitandcheckdates Checkdateonanonreturnedfilm

TestCasesderivationfrom requirements

Testsituationsforrequirement1.4

Inquiryonhiredfilm. Inquiryonreturnedfilm. Inquiryonafilmwhichhasbeenjustreturned. Inquiryonmemberwithnofilms. Inquiryonmemberwith1film. Inquiryonmemberwithmultiplefilms.

Testsituationsforrequirement1.5

TestCasesderivationfrom UseCases

Forallusecases:
1. DevelopaGraphofscenarios 2. Determineallpossiblescenarios 3. Analyzeandrankscenarios 4. Generatetestcasesfromscenariostomeetacoverage goal 5. Executetestcases

ScenarioGraph
Generatedfromausecase

Nodescorrespondtopointwheresystemwaitsforanevent

environmentevent,systemreaction

Thereisasinglestartingnode Endofusecaseisfinishnode Edgescorrespondtoeventsoccurrence

Mayincludeconditions
Specialloopingedge

Scenario:

pathfromstartingnodetoafinishnode

Title:Userlogin Actors:User Precondition:SystemisON 1:UserinsertsaCard 2:SystemasksforPersonalIdentificationNumber(PIN) 3:UsertypesPIN 4:SystemvalidatesUSERidentification 5:SystemdisplaysawelcomemessagetoUSER 6:SystemejectsCard Alternatives: 1a:Cardisnotregular 1a1:Systememitsalarm 1a2:SystemejectsCard 4a:Useridentificationisinvalid ANDnumberofattempts<4 4a.1GobacktoStep2 4b:Useridentificationisinvalid ANDnumberofattempts>=4 4b.1:Systememitsalarm 4b.2:SystemejectsCard Postcondition:Userisloggedin

ScenarioGraph

Scenarios

Pathsfromstarttofinish Numberoftimeloopscanbetakenneedtoberestrictedforfinite numberofscenarios

Id 1 2 3 4 Events 123456 11a.11a.2 123423 456 12342*3 44b.14b.2 Description Userloginwithregularcard, rightPINonfirsttrial. Normal course of events UserloginwithnonregularCard Userloginwithregularcard, wrongPINonfirsttrial, therightPINonsecondtrial Userloginwithregularcard, wrongPINonfirst,second, thirdandfourthtrials

ScenarioRanking

Iftoomanyscenariostotestall Rankingmaybebasedoncriticallyand frequency Alwaysincludemainscenario

shouldbetestedfirst

TestCasesgeneration

Accordingtocoveragegoal.E.g.

Allbranchesingraphofscnarios(minimalcoverage goal) AllScenarios nmostcriticalscenarios

ExampleofTestCase
TestCase:TC1 Goal:TestthemaincourseofeventsfortheATMsystem. ScenarioReference:1 Setup:CreateaCard#2411withPIN#5555asvaliduser identification,SystemisON Courseoftestcase
1 2 External Event Reaction Comment User insert Card #2411 System asks for Personal Identification Number (PIN) User types PIN #5555 System validates USER identification System displays a welcome message to USER

Passcriteria:Userisloggedin