Why Test Automation FAILS

“If you can’t measure it, it doesn’t exists”

Software Testing Conference 2013

Capgemini India Private Limited

Prepared by:
Bharat Mulchandani
Email: Bharat.Mulchandani@capgemini.com
September 2013
  1

Table of Contents

1. Reason for writing White Paper .............................................................2

2. Executive Summary ...................................................................3

3. The Proposed Solution ...............................................................3

3.1 “Pre-requisite before Automation Process” ........................................... 4
3.1.1 Automation Gates in Detail........................................................ 5
3.2 Benefits of Test Automation Success................................................... 8
3.3 Reasons for why Test Automation fails ................................................ 8
3.4 What happens when Automation Fails? ............................................... 8
3.5 MYTHS of Test Automation ................................................................ 9

4. Conclusion .....................................................................................9

5. Appendix..................................................................................... 10
5.1 Case Study .................................................................................... 10

6. References ................................................................................... 11

7. Author’s Biography .......................................................................... 11

Why Test Automation FAILS

  2

1. Reason for writing White Paper

In today's IT business demands “to do more with less” and they're expected to
deliver higher quality systems within stipulated time. Many organizations have
already done research and written several white papers on “why test automation
fails”, “how to make test automation successfully”, “test automation strategy” and so
on. However test automaton failure percentage is still very high and a success
percentage is very low. In spite of following Automation Best Practice, investing time
and efforts result does not yield the expected outcome.
The KEY problem is we are NOT adopting the right process at the right time under
the right circumstances which result in poor ROI and high cost
As per research & survey done from different organization & institute;
My personal experience as well as the results of the IDT survey they conducted with
over 700 responses from all over the world also indicated that many software
automated testing efforts fails due to lack of process.
L. Hayes, However testing automation is uninviting, as the success rate is so low [3].
Gartner states; "Often, functional test automation efforts have failed because of cost
and complexity”.

This White Paper outline has Practical Approach towards test automation and
effortless to follow process as a best practice. This will enable you to explore better
possibilities and make sure this will increase the probability of making precise
decision on test automation approach.
The main objective of white paper expresses how can we make test automation more
cost-effective and improve success percentage with practical study. This approach is
very easy to understand and implement (Please refer Case Study in Appendix
section).

Why Test Automation FAILS

  3

2. Executive Summary

Today’s, IT industry thrives on rapidly changing technologies with accelerating

product lifecycles, many organizations want to develop application to tap the Global
market as soon as possible before any competitor releases it. Most of IT
organizations face many challenges to drive their software development business,
one of the significant challenges is their software/product should be released as soon
as possible and also ensures application is stable with better quality.
To achieve this desired goal manual testing came into picture and organization has to
spend additional money and time to achieve expected quality. Quality is main factor
for driving any product for which one should not be comprised upon.
Manual testing takes more time which can delay the product release – This is the
time when organizations plan easy solution to move from manual to test automation,
so they can release the software promptly and cost-effectively by implementing test
automation.
Test automation has thus become more and more popular in recent years and has
been a constantly increasing activity in the existing software industry. Test
automation is often seen with many benefits e.g. as a way to reduce the costs of
testing, increase test coverage and effectiveness, shorten testing cycles etc. Since
test automation is initially costlier than manual testing, it should be thoroughly
studied before being adopted. Even though most companies believe that test
automation is useful only few companies can claim actual success.

3. The Proposed Solution

Test automation is just like Rome, It wasn't built in a day – similarly it takes
huge efforts to measure the benefits of successful test automation we should be very
careful in adopting and implementing the same. Prior to begin test automating once
should follow well defined process or pre-requisite which contribute and drives to
achieve the desired goal.
As we all are aware that any test automation doesn't produce instant outcome. The
benefits aren't immediate.

We need to solve real business problems with realistic factors. I am sure my

approach will bring fruitful result in your test automation project; this approach does
not need an additional cost.
The Practical Approach for Test Automation is described here based on proven
system/analysis done and achieved.

Before moving from manual to automation we need to cross six different gates, All
six gates are listed below in detail

Why Test Automation FAILS

  4

3.1 “Pre-requisite before Automation Process”

This approach consists of 6 pre-requisite gates before making decision to automate,
It is mandatory to “PASS” each gate before moving on to next gate.

Gate 1; Why to Automate,

Make decision whether to automate or not to automate.

Gate 2; When to Automate,

UAT should be stable & Manual testing process should
be in place, This is most crucial gate.

Gate 3; Which test cases to Automate,

Selection of test case which one to automate and
which test case not to automate based on priority.

Gate 4; How to Automate;

Select right tool for test automation.

Gate 5; Who should Automate;

Expertise level.

Gate 6; Where to Automate;

Test environment for automation.

See Figure, 1.1 Automation Pre-Requisite Gates (APGS).

I ensure “Pre-requisite before Automation Process” is one of the best

automation process and strategy to make rational decision before moving into
automation, this process will save enormous effort, time and cost.

Sequences of all Gates are critical, you cannot overlook any gate for example; one
cannot decide which test case to automate and which test case not to automate
without knowing when to automate.

I am going to detail out each Gate within this planning cycle and how all of it comes
together to help you achieve success within your test automation approach.

Why Test Automation FAILS

  5

Figure 1.1 Automation Pre-Requisite Gates (APGS).

3.1.1 Automation Gates in Detail

Gate 1: Why TO Automate;

Purpose, Benefit & ROI of automation will help us to make accurate
decision whether to automate or not to automate.
This is one of the most important factors before we can start
automation, we need to ask question “why are we automating”.

• Identify what is Purpose & Benefits of test automation (objective to

move from manual to automation should be clear).
• Calculate ROI on parameters of the testing application,
I.e. What will be the age of application, testing frequency and number of
planned release for next 2years, how long it will take to complete test
automation project, tool cost, automation resource cost, current manual
testing cost. (Objective; how effective it will is going to be)

As per experience & research show that it takes at least 8 to 10

releases to recover the effort of test automation (ROI) - An automated
scripts has a finite lifetime.

The main objective are

- Reduce costs of testing (calculate ROI),
- Reduce release cycle time and
- Ensure improved quality & effectively add value to testing process.

IF all above objectives are achievable and measurable, Then we can move
to next phase/gate else we will not go for automation this is not right time

Why Test Automation FAILS

  6

Gate 2: When TO Automate;

When UAT is stable & when manual testing process should be in place.
This is most crucial gate;

• Stability;
How to check for stability of test application?
i) Manual testing process should be in place i.e. all manual test cases
with expected result should be designed and reviewed.
This also includes test steps, test data, expected result, precondition
to validate test scenario
ii) All test cases steps should be executed manually twice on
different build/version.
(Why twice on diff build to make sure application is stable or not)
• Automate those modules where more than 70% of manual test cases
are passed and where there are no major open defects.

Performing manual testing before automation is a must.

Gate 3: Which test case to Automate;

Selection of test case based on priority of which test case should
be automated first

It is important to remember that all test cases cannot be

automated so we need to identify which test case can be
automated and which test case cannot be automated.

Which test cases be automate

• Regression Test cases: Tests that needs to be run on
every build/release/version.
• Data Driven Test: Tests that use multiple data values for the
same actions.
• Test case which have passed while performing manual verification.
• Test cases which have more verification points to test.

Test cases which are more repetitive are Better candidate for
test automation.

Which test cases cannot be automate

• Usability testing - "How easy is the application to use?"
• Test cases without expected result
• Test cases which got failed
• Test cases which are going to execute once
• One-time testing

Certain tests are better when performed manually.

Why Test Automation FAILS

  7

Gate 4: How to automate;

Select right tool for automation; selecting an automated testing tool
is essential for test automation, so it is important to choose the right
tool that best suits your overall automation requirements.

Consider these key points when evaluating automation tool:

• Technology Compatibility (you testing application was developed in
which language i.e. .Net, C# or WPF applications),
• Verify all functionality & feature required for automation exists, i.e.
GUI checkpoints, verify values, databases etc.
• Usability in terms of ease in learning, adapting and trainings,
• Ensuring extensibility, maintainability and support,
• Easy availability of resource,
• License cost.

Gate 5: Who should automate;

Expertise level; we need to have skilled resources based on what type
of automation testing we are planning e.g. Unit Testing, GUI testing,
Database, Web service.
• Automation tester who is having programming logic (skilled
resources – they are similar to developers).
• Automation resource should have experience & knowledge of
manual testing. (it is mandatory to have manual testing skills)
• Automation team should have Readiness/zeal towards automation

It is necessary for automation tester to design test programs, develop

reusable programs, review test scripts and test the test scripts to ensure
the quality of test automation.

Gate 5: Where to automate;

Automation test environment is similar to QA environment.

• Start automating after application GUI and code are almost frozen.
• We should automate on QA environment and not on dev
environment.
• We should automate on same version of code where manual testing
was performed.
• How long test automation project will take to complete and how
effective it will is going to be?
• Automate those modules where there are no major defects open.
• Automate those modules where there are no more major changes.

Note: It’s possible to automate even if the product is not fully ready
but that is a completely different ball game. I am not covering
such exceptional scenarios in this white paper.

Why Test Automation FAILS

  8

3.2 Benefits of Test Automation Success

• Time & Cost Savings

• Improved Testing Quality
• Shortened release Cycle
• Reliable Results
• Increasing test coverage
• Avoid cumbersome repetitive tasks
• Repeatability (Execute same test cases many times)
• Process in Place for Future Success

Automation is long term vision & investments which will

reduces costs and add more value in software testing process

3.3 Reasons for why Test Automation fails

1. Lack of process (Process not followed).

2. Application under test was not stable.
3. Lack of time, budget, and resources
4. Selection of Wrong tool (Tool incompatibility)
5. Lack of expertise. (Automate using record and playback/less programming
capabilities)
6. Automate test cases which failed during manual execution.
7. Poor test coverage
8. Resources do not have testing background
9. Tight Schedule
10. ROI not calculated to make rational decision
11. Automation test script was not reviewed. No Accuracy
12. Every release is having many changes in existing functionality
13. Effectiveness
14. Automated before functional or regression cycle was completed
15. Due to over automation it was NOT delivered on time
16. Poor Test Environment
17. Un-Reliable and Inconsistent

3.4 What happens when Automation Fails?

• Loss of Time & Money

• Stake holders are not satisfied with the expected result
• Overall Reduced Productivity
• Lost Opportunity

Why Test Automation FAILS

  9

3.5 MYTHS of Test Automation

1. Stated goal: “Automate Everything”

- We can automate all test cases i.e. 100% of manual work which will
replace manual testing.
- Automate those test cases also which would be executed only once.
- Automate test case which failed during manual execution.
2. Manual testers are overhead expenditures in organization.
3. Automation test development takes similar time as writing manual test case.
4. Automation reduces the cost and duration of test efforts from day one.
5. We can start developing automation script before manual test execution.
6. Automation primarily is record & playback. (i.e. easy of resources - anyone
can automate)
- Automation does not require any expertise i.e. no programming skills are
required.
7. ROI calculation is not mandatory or necessary (Test Automation should not be
considered as rational decision)
8. Once automated it is for lifetime (no additional cost)
9. It not necessary to review developed automation test script. (scripts will not
have bugs/defects)
10. Automation team can parallel automate with dev team (can automate on dev
environment before unit and functional testing)
11. Automation suite can run/execute entirely unattended.
12. For test script development manual test cases are NOT required. (No
Documentation)
13. Automation is short-term investment.
14. Automation may give you its benefits immediately. (Here and Now)

4. Conclusion

It is our trust that you can achieve your test automation goal and objective once the
above Gates (automation pre-requisite) are followed.

There is no magic formula to make test automation successfully, test automation is a

strategic investment - The key area is in adopting the right process at the right time.

The main goal is to develop best process that highlights the various factors to be
considered while making rational decision to go for test automation.

The objectives of test automation are

1. To reduce costs of testing (ROI),

2. Reduce release cycle,
3. Ensure improved quality,
4. Add value to testing process in order to delivery winning solution which should
make application release as quickly and cost-effectively to drive competitive
advantage.

"Take care with test automation. When done well, it can bring many benefits. When
not, it can be a very expensive exercise in frustration". [2]

Why Test Automation FAILS

  10

5. Appendix

5.1 Case Study

Case Study: Convert from manual testing to automation.

Manual Test Scenario: How to prepare coffee manually by Human being.

Step 1: Verify all Pre-requisites are met

i.e. you should have empty vessel, empty cup, spoon ,working gas stove,
lighter, sugar, coffee powder, milk.
Step 2: Pour one cup milk (125 ml) in empty vessel
Step 3: Light gas stove with lighter (Keep temperature at 110o Celsius)
Step 4: Place milk vessel on the burning stove
Step 5: Wait for 5 min for milk to boil.
Step 6: Add one tea spoon of coffee powder & two tea spoons of sugar in hot milk
Step 7: Mix with spoon and let it boil for 2 min.
Step 8: Switch off the gas stove.
Step 9: Pour HOT coffee in a cup

Expected Result:
Step 10: Verify the Taste, quantity and quality of coffee is same as expected.

Convert above test scenario from Manual TO Automation (using robot).

1. We need to execute above test cases manually to verify everything works

fine before feeding program logic in robot.
We will jot down each and every step and action which is not mentioned in
manual test case.
2. We will execute same test case twice to ensure we are achieving the same
expected result.
If result varies we will not automate.
3. We will feed the program logic into robot.
4. We will execute trial run (dry run) through robot and verify our commands
were correct or not.
5. We will verify robot is doing what was expected.

Why Test Automation FAILS

  11

6. References

[1] http://www.automationanywhere.com/Testing/
Analyst(s): Ian Finley, Nathan Wilson, Gordon Van Huizen.
[2] Howard Fear on Test Automation by Howard Fear (hsf@pageplau.com)
[3] L.Hayes, “Establishing a Test Automation Function”, Journal of Software Testing
Professionals

7. Author’s Biography

Bharat Mulchandani is having more than 10 years of experience in QA who has shaped the
testing solutions and established thought leadership position in the software testing domain.
Currently Bharat is working as a Senior Consultant with Capgemini, He has been directly
involved in providing testing services for a second largest investment bank in US. He is
responsible for QA service, Testing Innovations, Business Development support, accelerating
technology competency and providing thought leadership leveraging to QCOE. Before joining
Capgemini Bharat worked for Sagitec Solutions - USA, He successfully transitioned this small
testing engagement to very large managed services engagement (Horizontal services). He also
is driving the delivery of testing complex systems and applications where he managed delivery
for multiple key client accounts. He also played a key role in proving testing solutions to
various clients. Bharat has received his Master Degree in Commerce from Pune University.

Why Test Automation FAILS