Test Automation Lets talk business

Igor Gershovich Connected Testing, Inc. www.connectedtesting.com igershovich@connectedtesting.com

Overview
Test Automation Myths Return on Investment (ROI) Frameworks

Test Automation Myths

Myth #1 - Test Automation is simple, that every tester can do it

This myth is promoted by the tool sales people. They are trying to promote the following test automation process:
Record

the script Enhance the script by adding functions and data driving Run the scripts Report results

Under the influence of this myth the QA manager can proudly report: All our testers are developing test automation.

Reality - Test automation is a software development task

Automation should be designed, developed and tested You need to have some kind of a programming background to implement test automation. Test Automation is not as complex as C++/C#/Java development. Test automation standards should be developed Automated test components are assets that should be treated like application source code

Myth #2 Commercial test tools are expensive

Under the influence of this myth some companies, especially the small ones:
Try

to develop their own test automation tools Use scripting languages like Perl and Ruby Use shareware test tools Do not consider test automation at all

Reality Commercial tools are cheap

Per seat license for the most expensive automation tool is $8K This tool will be used for 5 years. Maintenance/Support fees are 20% of tool cost or $1,800 per year The cost of this tool is $8K/5+$1,800 = $3,100 per year The automation developer cost with overhead is $100K per year The cost of this tool is just 3% of the person who uses it, but productivity gain can be very significant

Commercial Tool Benefits:

Customer support. Many of the open source tools come and go with little to no support Most commercial tools are constantly being updated as technologies change Most commercial tools usually have more functionality (QTP can test various GUI applications: Web, .Net, Java, VB, C/C++, PowerBuilder, etc. vs. WATIR Web only) Commercial tools usually have a large community of users, which translates into better availability of qualified resources Commercial tools require less advanced programming More test automation frameworks are available for commercial tools Commercial tools are integrated with Test Management tools which makes reporting and execution much simpler.

Over 300 Test Tools are available (http://www.softwareqatest.com)

Load/Performance tools 54 Web Functional/Regression 60 Java Test tools - 48 Other Web tools 76

Which tool is right for you?

Absolutely Unscientific research: Employer demand by tool expertise (May, 2008)

Dice search results across US (30 days)

Tool QTP Functional Tester SilkTest TestComplete TestPartner VSTE WATIR Selenium Search string Qtp OR quicktest OR "quick test "rational robot" OR "functional tester silktest OR "silk test testcomplete OR "test complete testpartner OR "test partner VSTE AND test watir Selenium Matches 613 118 86 26 19 45 26 108

Demand per tool

Open Source Microsoft WATIR Open Source Selenium 2% VSTE 10% 4% Compuware TestPartner HP QTP 2% 60% AutomatedQA TestComplete 2% Borland IBM SilkTest Functional 8% Tester 12%

QTP Functional Tester SilkTest TestComplete TestPartner VSTE WATIR Selenium

Excerpt from IDC Report: Worldwide Automated Software Quality 2007-2011 Forecast and 2006 Vendor Shares

Think of yourself like a business

Facts:
Average

IT job tenure is about 2 years Competitive intensity is rising

What you should do:

Invest

your time/money in learning mainstream automation Create a niche for yourself - add Configuration Management, Performance Testing or Project Management to your skillset Market yourself

Return On Investment (ROI)

Classic ROI Calculation

ROI = BENEFIT/COST Automation Cost = Price Of HW + Price of SW + Development Cost + Maintenance Cost + Execution Cost Manual Testing Cost = Development Cost + Maintenance Cost + Execution Cost ROI = (Manual Testing Cost - Automation Cost)/Automation Cost

Looks right, Doesnt it?

Problems with Classic ROI Calculation

You cant compare Automated Testing and Manual Testing. They are not the same and they provide different information about the AUT. You cant compare cost of multiple execution of automated tests vs. manual tests. You would never dream of executing that many test cases manually.

Automation Real ROI

ROI

value IS NOT the value of Automation vs. Cost of executing these tests manually Automation ROI value IS the benefit of this type of testing, and it can be:
Reducing

Time to Market Increased Test Efficiency (Productivity) Increased Test Effectiveness

Reduced Time to Market

Can

get a greater market share Makes people available to work on other projects Higher margins, if no competitive products are currently available

Productivity and Effectiveness

More testing gets done faster, increasing the odds of finding defects Defects found early have better chances of being fixed Manual Testers can concentrate on clever ways to finding defects, instead of typing test inputs and verify output. About 7% of bug fixes create new bugs, sometimes in already tested parts of the system. With automation you can rerun tests for those modules. This almost never happens when testing is done manually.

ROI summary
Each project requires different types of automation - there is often no easy formula available to calculate ROI Performing ROI calculation can help to determine upfront what type of automation, what level of skills, what tools will be required.

Automation Frameworks

Test Automation Framework

A Test Automation Framework is a set of assumptions, concepts and tools that provide support for Automated Software Testing. A correctly implemented Test Automation Framework can further improve ROI by reducing the development and maintenance costs.

Types of Test Automation Frameworks

Modular Data-Driven KeywordDriven Model-Based

Modular framework

The Modular framework is the natural progression derived from Record-and-Playback The modular framework seeks to minimize repetition of code by grouping similar actions into modules (e.g.: login) Test Data is in a script or in an internal table

This is what tool sales people are promoting.

Data-Driven/Keyword Driven
These frameworks are similar in that the data is separated from the test script The script is just a "driver" or delivery mechanism for the data. The difference:

In

keyword-driven testing, the navigation data and test data are contained in the data source In data-driven testing, only test data is contained in the data source.

Model-Based testing
Model-based testing is software testing in which test cases are derived in whole or in part from a model that describes some (usually functional) aspects of the system. Model-based testing for complex software systems is still an evolving field.

Well be concentrating on Keyword-Driven testing since it is the most beneficial framework for large-scale test automation.

Advantages of Keyword-Driven Test automation

This Framework addresses the most common problem with test automation: Automation Engineers do not have domain knowledge and the End Users (Subject Matter Experts/Test Engineers) usually do not have automation expertise. When properly implemented and maintained, it presents a superior ROI because each business event is designed, automated and maintained as a discrete entity. Keywords can then be used to design test cases, but the design and automation overhead for the keyword has already been paid.

Advantages of Keyword-Driven Test automation

Reduced the cost and time spent maintaining and updating tests The modular structure of keyword-driven testing means that new tests can easily be created from pre-existing modules The test team is capable of entirely automating tests, even without programming knowledge Can be easily modified to use with different test tool Reusability across different projects

Classic Keyword-Driven Example

Object Textfield (username) Action Enter Text Data <username>

Recent Success with Keyword-Driven approach

Estimate using Modular Framework: 1 year Estimate with Keyword-Driven Framework: 6 months

Project was completed in 4 months by 1 person, 2 month ahead of schedule. This was the first project for the automation developer utilizing the Keyword-Driven Framework. 170 components were developed (1 component per Web page). 109 Regression Test cases were automated.

Example of Test Data for Keyword-Driven test automation developed by Connected Testing Inc. for TestComplete and QTP

Resources

Everything you want to know about Test Automation by Brian Le Suer Managers Guide to GUI Test Automation by Yury Makedonov The ROI of Test Automation by Michael Kelly What is my ROI? by mVerify Corporation SQA Forums An Overview of Test Automation Frameworks by Nathaniel Ritmeyer The benefits of keyword-based software test automation by David W. Johnson Wikipedia

Igor Gershovich
Company: Connected Testing, Inc. E-Mail: igershovich@connectedtesting.com Phone: 720-933-9395 Website: www.connectedtesting.com