Академический Документы
Профессиональный Документы
Культура Документы
2004
Academic Edition
Copyright
2003
Scripting
Copyright
2003
Scripting:Anotherscriptformat
Step
#
Check What to
?
do
What to
see
Design
notes
1.
____
Task menu
down
Pull down
task menu
Observation
notes
ThisistheformatthatI(Kaner)usedtouseaWordStar,longago
(backwhenIthoughtscriptingwasagoodidea).
Copyright
2003
Scripting:Perceivedbenefits
Proponents of highly scripted tests believe that scripting offers several
benefits:
Repeatability:Differenttesters,followingthesamescript,willdothesame
thingsandachievethesameresults.Thisprovidesalevelofprocesscontrol
fortesting.
Scalabilityofexpertise:Oneexperttesterwritesthescriptsandjunior
testersexecutethem.Weleverageourbestpeople.
Abilitytoaddinexpensivestaff:Thejuniorscostlessthanthesenior.
Supportsoutsourcedtesting:Cheaplaborfarawaycanfollowthescripts.
Abilitytoaddstafflateintheproject:thescriptsprovidestructureforthe
testerwhoisunfamiliarwiththeproduct.Iftheprojectschedulefalls
behind,wecanaddtestersandkeepthembusywithouttoomuchtraining
cost.
Training:Thescriptactsas"trainingwheels"forthenewtester.After
severalmonthsoffollowingawiderangeofscripts,thenewtesterwill
havelearnedbyexamplealotabouttheprogramandhowtotestit.
Copyright
2003
So,whatswrongwithscripting?
Copyright
2003
Letstryanexample:Asimplearithmeticexercise
Problem
Obtain this
many marbles
21
100
127
Copyright
2003
Asimplearithmeticexercise
Problem
Obtain this
many marbles
21
127
100
14
163
25
99
Copyright
2003
Asimplearithmeticexercise
Problem
Obtain this
many marbles
21
127
100
14
163
25
99
142
121
Copyright
2003
Asimplearithmeticexercise
Problem
Obtain this
many marbles
21
127
100
14
163
25
99
142
121
18
43
10
Copyright
2003
Asimplearithmeticexercise
Problem
Obtain this
many marbles
21
127
100
14
163
25
99
142
121
18
43
10
23
49
20
Copyright
2003
10
TheArithmeticExercise
What we see is that people adapt to procedures.
Once in the mindset of a set procedure
peoplearelikelytomissorignoreblatantlyobvious
informationthattellsthemthat:
a different procedure would be more efficient (in this
case)
this procedure is not applicable to this situation
something else happened while they were following
a procedure, that isnt within the scope of
consideration of this procedure
Copyright
2003
11
Thiscontrastswithacommonassumption
The assumption
Whensomeoneisfollowingastandardprocedure,theyllkeep
theireyesopenandnoticeunusualthings,noticevariation,et
cetera
Copyright
2003
12
Thiscontrastswithacommonassumption
The assumption
Whensomeoneisfollowingastandardprocedure,theyllkeep
theireyesopenandnoticeunusualthings,noticevariation,et
cetera
Have you ever noticed?
Whathappenswhenyoudrivetoanewplace
from a map, or in some other way, making your own way
with a navigator who says, turn left next light etc
Ifyoutrytogothereagain,doyouremembertheroute?
Copyright
2003
13
Thiscontrastswithacommonassumption
The assumption
Whensomeoneisfollowingastandardprocedure,theyllkeep
theireyesopenandnoticeunusualthings,noticevariation,et
cetera
The contrast
Theexamplessuggeststhereverse,thatpeoplegetlockedinto
procedures,focusmoreonfollowingtheinstructionsthanonthe
systembeingworkedthroughthoseinstructions.
My assertion
Whenpeopleembraceprocedures,theyoftenbecomeautomatons.
Copyright
2003
14
Thiscontrastswithacommonassumption
The assumption
Whensomeoneisfollowingastandardprocedure,theyllkeep
theireyesopenandnoticeunusualthings,noticevariation,et
cetera
Inattentional Blindness
Flatlycontradictsourhappyassumption.
seeSightsUnseen,
http://www.apa.org/monitor/apr01/blindness.html
seethedemonstrationsat
http://viscog.beckman.uiuc.edu/djs_lab/demos.html
seethediscussionsathttp://psyche.cs.monash.edu.au/v7/
seetheargumentthatitislackofexpectationratherthanlackof
attentionathttp://psyche.cs.monash.edu.au/v7/psyche706
braun.html
Black Box Software Testing
Copyright
2003
15
AnotherProceduralismExample
HowdoFornixFimbria Study compared normal rats with rats with
fornix-fimbria lesions that destroyed the
LesionsAffectOneWay
hippocampus.
ActiveAvoidance
Behavior?
Through a series of pilot studies, the
experiment simplified and simplified until it
was finally a big white box (with an
CemKaner,Bob
electrical grid) for a floor and a smaller
Osborne,HarveyAnchel,
black box, with a metal door between
MarkHammer&
them.
AbrahamH.Black
Rats were placed in the white box, given a
signal, and had to move to the black box
McMasterUniversity
within 5 seconds or get shocked.
Normals appeared to know where they
86thAnnualConvention
were and where they wanted to go. The
oftheAmerican
lesioned rats appeared to have learned a
PsychologicalAssociation
fallible procedure for escaping.
Toronto,Canada,August <See the paper, and/or the demonstration
in class.>
28,1978
Black Box Software Testing
Copyright
2003
16
Scripting:Thetesters'tourbus
Scripting is the Greyhound Bus of software testing:
Totheexperiencedtester,thetestscriptisatourbus.Whensheseessomething
interesting,shestopsthebusandtakesacloserlook.
The skilled tester breaks away from the script.
The skilled tester can also notice problems that the script writer didn't
anticipate, and make comparisons that the script doesn't call for. An
automated test is very different from a human test,
In effect, the script hides what the skilled tester is doing.
One problem with a bus trip. Its often pretty boring, and you might
spend a lot of time sleeping.
Black Box Software Testing
Copyright
2003
17
Scriptingdoesn'tliveuptoitspromise
Notrepeatable:Differentpeoplefollowingthesamescriptdoandnoticedifferent
things.
Whatexpertiseareyouscaling?Thepersonwhowritesthetestsimaginesasubset
ofthepossibleissuesand,bythenatureofscripting,islimitedtoasmallsetoftests
thatwillberepeatedoverandover.SeeourdiscussionofGUIlevelregressionthe
repeatedtestslackpower.Worse,thescript'sfocusonthesameexpectedresultevery
timesteersthescriptuserawayfromnoticingotherdimensionsof(mis)behavior.
AllofthedisadvantagesofGUIlevelcapture/replayautomation:hardtomaintain,
nomodularity,expensivetocopewithUIchanges,notlikelytoexposemanybugs.
Itmissesthecriticaladvantageofautomation:Ifyou'rerunningabunchoftests
thatareunlikelytoyieldmuchinformation(becauseyou'verunthembefore),atleast
withautomatedtests,youcanminimizethetimeahumanspendsrunningthem
(additionally,theautomatedtestscanbebetterdesignedformaintainability).
Nocommunicationofanalysis:Scriptsvaryonthisissue,butmostfocuson
procedurewithoutexplainingthepointofthetest.Neitherthetesterwhofollowsthe
scriptnortheonewhomaintainsitknowswhatwasinthemindofthetestdesigner.
Ineffectivewaytoeducate:Stepbystepproceduralinstructionisapoorwaytoteach
someoneabouttheprogramorabouttesting.It'snothowpeoplelearn.
Copyright
2003
18
Scripting
Chip Groeder created scripts as pseudocode, in preparation for
test automation.
Won the Best Paper award at STAR 97
But he found most (88%) of the bugs during creation of the
scripts (my experience too).
Maybe the bug-finding benefit lies in structured exploration,
rather than fully-specified regression.
Copyright
2003
19
NeverDoANYkindofScripting?
Checklists (as distinct from scripts) have their place. For
example, think of releasing a product:
Manyverydifferenttasks
Allofthemmustbedone
Thetaskisrarelydone,somanystepsmaybeforgotten.
I prefer to tell testers what to test (what issues to cover), not
how to do the tests. Teaching people how is a matter of
training, not something that I record time and time again in the
test plan. A checklist will sometimes be the right way to present
the list of issues.
I think the level of detail is sufficient if I can successfully pass
the section to a reasonably experienced tester who is a little
familiar with the program and be confident that she can figure
out what the test cases are and how to run them.
Copyright
2003
20