Академический Документы
Профессиональный Документы
Культура Документы
Robert V. Binder System Verification Associates rvbinder@sysverif.com #MoreModelsLessTests Chicago Quality Assurance Association February 25, 2014
Tell Me A Story
ModelBased Testing
Trivial Program
Data space: 21024 Sequences: 21024 Execution space: 22048 1.34 10616
Large subsets of any programs execution space are revealing: if a bug exists in any member of that space, the space is reached it will trigger an observable failure.
But, we rarely know exactly which sub-domains are revealing until we test.
Model Driven Development: Taking BDD/ATDD to the Next Level 4
Medical
IT Mobile Best In Class
4.7
Capers Jones statistics: 5 defects/FP, 95% Removal. Assumes 53 Java LOC/FP .
Model Driven Development: Taking BDD/ATDD to the Next Level 5
Driven to Distraction
Test Driven Development
TDD
What is it?
Model-based Testing
MBT
Model Driven Development: Taking BDD/ATDD to the Next Level 7
Manual Test Cases not executed Automated Test Cases not maintained
Model Driven Development: Taking BDD/ATDD to the Next Level 9
Tell Me a Story
TELL ME A STORY
!!!
Client Endpoint Chat Server
13
Logon Logon
GetStatus GetStatus
Logoff Logoff
15
TestLogon Logon
GetStatus
Logoff
16
TestLogon Logon
GetStatus
Logoff
17
GetStatus
Logoff
18
GetStatus
TestLogoff
Logoff
19
TDD Limitations
Process, Activities
Components, Test Objects
20
Scenario C
Scenario C
23
Overview
http://www.specflow.org
Feature Files SpecFlow
Enter features, scenarios Generates skeleton test code for each step
Extracts step names Can run immediately
Scenario A Scenario B
Scenario C
Test Objects
Nunit Visual Studio Components
25
When
Trigger API call, GUI action
Then
Verification conditions (expected results)
Model Driven Development: Taking BDD/ATDD to the Next Level 26
27
Modelbased Testing
THE VERY MODEL OF A MODERN
What is a Model?
Focus Smaller, simpler In IT systems
Behavior Structure Content
Model Driven Development: Taking BDD/ATDD to the Next Level 29
IT Models
Process, Activities
Behavior, Structure, Content
Technology
30
IT Models
Notional (design) As built (documentation)
Test Models
Focus on aspects that support testing goals
Behavior
Sequence and time
Transformation
Input, output function
Chat Server
Chat Server
Logon
Must be first Cant be repeated Logon
GetStatus
Only after logon Returns number of calls since logon
GetStatus
Logoff
Logoff
Only after logon
Logoff GetStatus Logon assert (401) assert (401) nc++ GetStatus Logoff assert (200) assert (n == nc)
34
Logon
GetStatus
assert (401)
Logoff
List Users
Users: a, b !!! Client Endpoint Client Endpoint Chat Server
35
Users: a, b
Post Pumpkin Latte #mypics
Chat Server
36
1. Create Test Model Actions [Rule] tags a method for exploration Requires defines a exploration precondition must be true to explore
Model Driven Development: Taking BDD/ATDD to the Next Level 38
39
Define machines
Limits Actions used in an exploration
Composable
Very powerful!
Model Driven Development: Taking BDD/ATDD to the Next Level 40
41
Set options to generate test code Use any machine or composite machine
43
44
Explores the test machine Writes out test code using VS testing framework
45
Not intended to be maintained Re-generate any time Add helper code in the model to extend/customize
Model Driven Development: Taking BDD/ATDD to the Next Level 46
Adapters
start/stop chat clients and chat server
47
Spec Explorer
Exploration Interactive model development Generate test suites
Machines Cord Declarations Model Program Model Data Structures Rules Actions Events Adapters
Machines define composable Action sequences Options control exploration and data generation Actions are explored
Rules define exploration conditions
Events define expected SUT response
Options
Data structures support exploration Adapters abstract, control, and observe any interface, any platform
49
Generate
Test Suite
Evaluate
Control
SUT
Observe
Bug
50
Test Execution
52
Test Code
Adapters
Model Driven Development: Taking BDD/ATDD to the Next Level 53
54
Q&A
#MoreModelsLessTests rvbinder@sysverif.com
56
http://robertvbinder.com/wp-content/uploads/rvb-pdf/arts/MBT-User-Survey.pdf
http://www.robertvbinder.com/open-source-tools-for-model-based-testing/
http://www.slideshare.net/robertvbinder/model-basedtestingignite
http://visualstudiogallery.msdn.microsoft.com/271d0904-f178-4ce9-956b-d9bfa4902745
http://msdn.microsoft.com/library/ee620411.aspx Microsoft Open Protocol Project
Model Driven Development: Taking BDD/ATDD to the Next Level 57
http://cacm.acm.org/magazines/2011/7/109908-microsofts-protocol-documentation-program/fulltext#!
http://www.youtube.com/watch?v=SVwBz-ZTliw
https://github.com/cucumber/cucumber/wiki
58
59