Technical White Paper

Ananthakrishnan J
Architect, Sonata Software
Risk-Based Testing:
Implementation of Risk-Based
Approach for
Quality & Cost Optimization

Author
Kalyanam Kannan

Sonata Software Limited

www.sonata-software.com
Technical White Paper www.sonata-software.com

STATEMENT OF CONFIDENTIALITY

Information included in this document, in its entirety, is considered both confidential and proprietary to
Sonata Software and may not be copied or disclosed to any other party without its prior written
consent.
All logos used in this document are registered trademarks of the respective organizations.

Risk-Based Testing 1 Sonata Software Limited

Technical White Paper www.sonata-software.com

Abstract

As a practiced trend in IT projects, Testing is performed only towards the end of a project. Teams
dedicate hours to test possible risks and flaws after the project is ready to run. As software testing at
this level invites several last minute modifications that can cause discomfort, or sometimes even refute
the very concept of the project, it has become the need of the hour to come up with a way to ensure
detection and reduction of risks, at an early stage of the project. Risk-Based Testing, or RBT as referred
to in this paper, is a procedure in software testing which is used to prioritize the development and
execution of tests based upon the impact and likelihood of failure of the functionality or aspect being
tested based on existing patterns of risk.

Taking a cue from the age-old saying of ‘Precaution is better than cure’, RBT aims to find areas where
possibility of risk or defect is most likely to occur. Through this testing technique, a software test
engineer can now select tests based on risk even before the initiation of the project. Example, through
software testing, one can detect 200 errors by testing 5000 defects. RBT on the other hand, enables the
software tester to pick only 500 probable defects areas and conclude with 190 defects, thereby saving
the effort and time of the software tester.

This paper outlines the Risk-Based Testing approach and describes how Risk-Based Testing can positively

impact the development life-cycle based on business-oriented factors, offering organizations an

actionable plan for starting a Risk-Based Testing approach for projects.

About the Author

Kalyanam Kannan has been in the software industry and testing for the past 14 years and has managed
testing projects using different engagement models, Currently, as a Practice Director in testing, he is
responsible for controlling the quality of releases and delivery with optimum cost. He is also involved in
providing testing solutions using the latest technology, tools and operating models, which enable
projects to minimize cost to quality. His current areas of interest include Risk Based Testing, Test Driven
Development and Open Source Based Testing.

If you would like to interact with the author of this White Paper, feel free to contact us.

Risk-Based Testing 2 Sonata Software Limited

Technical White Paper www.sonata-software.com

Contents
Abstract .................................................................................................................................................... 2
About the Author...................................................................................................................................... 2
Risk-Based Testing .................................................................................................................................... 4
Generic Approach for Risk-Based Testing ................................................................................................. 5
Statistical Models ..................................................................................................................................... 7
Illustration ................................................................................................................................................ 9
Workflow for Risk-Based approach: ......................................................................................................... 10
Results ...................................................................................................................................................... 11
Inferences of the concept ......................................................................................................................... 13
Open Source Test Management................................................................................................................ 13
Summary .................................................................................................................................................. 14

To read more about our views on technology, do visit www.sonatablogs.com

Risk-Based Testing 3 Sonata Software Limited

Technical White Paper www.sonata-software.com

Risk – Based Testing

In today’s scenario, the quality of software is becoming a matter of concern. With this issue creating
conflicting challenges, the industry is testing and trying different measures to tackle it. Innovative
techniques, tools, technologies and ideas are being implemented to ensure availability of standard
software. One of the popular measures adapted is Risk-Based Testing – a technique through which a
certain amount of testing can be done without covering an entire gamut of available test cases.

According to Industry Experts, 80% of applications are either not tested or are manually tested before
being delivered to production. This leaves the quality of such software open for speculation and hence,
several software projects cost high due to the risks related to it.

Although a lot of mandatory regression and end-to-end testing is being done, the earlier the defect is
detected; the lower is the cost of solving the issues. To address these issues, Sonata has developed a
statistical model which would provide us a required methodology for Risk-Based Testing.

Sonata with its unparallel experience of product quality assurance services has understood that Risk-
based testing is vital in today's competitive market. With this Risk-based approach, a reduction in cost
per quality with a faster time to market is achieved.

Diagram 1

White_Paper_Rbt 4 Sonata Software Limited

Technical White Paper www.sonata-software.com

Risk-Based Testing

Risk-Based Testing is a methodology which after identification of risks and their possible impact on
system allows you to prioritize and plan your test strategy in accordance to the risk rating and mitigation
plans.

These provide us with faster time-to-market that gives us more time to fix the defects. The defects are
not detected at the end of the release; in fact the defects can be detected in the early stages of
Application Development itself. This is a scientific and data-based approach which results in cost
optimization and enhancement of quality. It can identify and execute high risk data hence providing
more time for defect fixes.

Generic Approach for Risk-Based Testing

Going ahead with our Risk-Based approach, a Risk Analysis is performed before starting the testing
activities. The prime objective is to take control over the problems before problems take over the
situation.

The following figure shows the activities involved in Risk Analysis when a project is performed. The
diagram below discusses this in detail:

Risk-Based Testing 5 Sonata Software Limited

Technical White Paper www.sonata-software.com

Diagram 2

Diagram2: Risk analysis activity model - This model is taken from Karolak’s book “Software Engineering Risk Management”,
1996 [6] with some additions made (the oval boxes) to show how this activity model fits in with the test process.

The first step is the Test Planning. In this phase, the risks need to be identified and a Risk Strategy should
be created. A risk can be of many types. One of the key important types could be the complexity in the
available applications, the type of resources and available tools. A clear Risk Strategy needs to be
defined before getting into the Test Planning activities.

Subsequently, the Risk Mitigation plan must be prepared. This plan clearly states for a particular type of
risk what the risk mitigation is. For example, if it is going to be a very complex application the risk
mitigation plan would be “dissecting the application into several components as smaller modules, and
fill up each of those components with more capable resources”.

Once the Risk Mitigation plan is completed, other important areas like the Risk Reporting can be
focused on. Risk Reporting is very important because it provides complete transparency across the
entire stakeholders of the project to gauge and act on the risk area. With all of the testing and
inspection techniques and capturing all test metrics, the risks that get reported are identified. At this
stage, one can predict the risks.

Risk-Based Testing 6 Sonata Software Limited

Technical White Paper www.sonata-software.com

In the Risk Prediction stage there is an entire set of data from which the risk-prone area is identified. In
this model risk prediction feedback is again fed into the risk identification area and it is a cyclic process.
By undergoing several iterations of this cyclic process the Risk Strategy as well as Risk Prediction model
can be refined. The areas with certain defects or minimum defects or no defects can easily be predicted.
This is the core idea of Risk Based Technology.

Statistical Models

In the statistical model the importance lies on the characterization of numerical data. Also, it is very
important to estimate the probability in terms of the behavior of system.

The entire testing activity is nothing but probability. It is the probability of finding out a particular defect
or a particular section failing on a particular area or on a particular type of environment. These all may
be useful in deciding the type of testing required and to ascertain the focus in areas of testing. With this
focus, extrapolation or interpolation of the existing data can be conducted and the best fit for that can
be identified. The best fit will provide the critical path for the defects. It will clearly provide areas that
require testing in that particular application or system.

There is another model called Spectral Analysis of data or model generated output which is an industry
standard. Here the focus is on the algorithms that have been used for this Risk-Based Testing. The
algorithm is extended to suit the current situations where the best fit for various factors is exercised.
This helps in calculating the risk as well as the probability of failure.

The statistical model is based on the probability of defects and a consequence of defect. These two are
very critical in defining the Risk Exposure of the system. One of the important parameters is quality of
the code. It may be suffering from poor designing or it may have been coded by an inexperienced
programmer, it may be to a complex functionality. The probability of defects is defined as P(f),
consequence of defect pertaining to the customer C(c) and consequence of defect to the vendor as C(v).

Risk-Based Testing 7 Sonata Software Limited

Technical White Paper www.sonata-software.com

Diagram 3

Consequences of defect for a customer (which is a cost to the customer) may be:

o The probability of a legal threat

o Losing a market place
o Not fulfilling regulations or FDI regulations

The consequence of defect towards the vendor gives a negative credibility to the vendor or it would
increase the maintenance cost because of the functions with faults.

The combination of these two factors leads to a formula: Risk exposure [Re(f)] is calculated as a product
of the probability of failure and the consequence effect.

Diagram 4

The probability of failure again characterized. It is a combination of multiple probabilities of failures.

Normally, the consequences value weighted between 1-3, and it would count the production fault loss
of revenue impact and incurs cost change impact also. The probability of failure is always weighed
between 0-1.

Risk-Based Testing 8 Sonata Software Limited

Technical White Paper www.sonata-software.com

The weighted average of the probability of failure is dependent upon following factors:

• Changed functionality
• New functionality
• Design quality
• Size of the project
• Complexity
• Programmers’ experience

Illustration

The following calculation of Risk-Based matrix is with a live example from one of the Sonata's projects.
For the sample calculation 18 factors of probability were taken into account. The customer is Europe’s
largest holiday company, serves more than 23 million guests every year. Operates own resorts, hotels,
airlines, travel agencies and cruise ships.

The front-end was a Web Selling platform and the back-end is a mainframe system which supports the
entire set of operations. The backend system is capable of running 600-1000 batch programs on a daily
basis and transacts 5-6 million records every day. The entire system has around 10 interfacing systems
(e.g. Amadeus, Alamo etc). They have multiple staging areas where they present their data and do a
focused analysis for the next quarter or the next season coming through.

The Risk-Based Testing was conducted for this particular engagement because there were nearly 8000-
10,000 test cases for the entire set of Enterprise Applications. While doing so, around 2 to 3 releases are
gathered on a monthly basis at the Enterprise Level. If a complete set of testing or end-to-end testing is
required, it is important to cover all sets of test cases across the enterprise chain. In such scenarios, over
2000 to 3000 test cases are run per release. This consumes a lot of effort and hence the cost, as well as a
delay to market.

In order to handle this situation several testing techniques are adopted, test automation in terms of tool
level as well as in terms of data level which would also test the testing data integrity. In terms of
approach a Risk Based Approach is followed because only a certain amount of test cases or certain
types of test cases catch errors, rest of the test cases were defect-free. Based on our observation it was
found that out of 2000-3000 test cases only around 200-300 test cases were capturing defects. This is

Risk-Based Testing 9 Sonata Software Limited

Technical White Paper www.sonata-software.com

because only these test cases are clearly attached to the risk prone areas. Hence, the statistical model
was adopted to capture most of the defects and implement Risk-Based approach.

Various kinds of testing methods have been implemented in this regard; starting with System Testing
which involves testing of the Web selling platform, testing of their core system, testing of their business
intelligence areas, Integration Testing, Regression Testing, Performance Testing to Security Testing etc. A
specialized testing on Data Integrity on volume testing was also conducted. In compiling all these
methods of testing several areas with defects were identified.

The inputs required for fitting the statistical model are:

o The number of defects

o Types of defect: Database defect, Staging Area defect, Web area defect
o Classification of defects: Defects originated of database, originated from the application server
or from the functionality
o Effort required for defect identification
o Effort required for fixing defects
o Weightage for the probability functions in terms of failure and the consequence

All inputs in the algorithms and routines for iterative runs were rigorously followed. This has resulted in
the probability of failure as well as the risk exposure co-efficient.

Workflow for Risk-Based approach:

As the first step, all defects are classified. Once the defects have been classified the probabilities of
various factors which affect the quality of the release are obtained, post which the risk exposure co-
efficient is derived. Once the Risk Exposure co-efficient is identified, the co-efficients are fed into the
iterative algorithms for various values of probability of failures. From this the type and number of test
cases that will be utilized for Risk Based Testing are obtained.

Example, in a live environment, if there are 2000-3000 test cases and it is known from an existing
analysis that the 23rd or 33rd test case are going to yield defects in a particular area which comes under
the sampling techniques, and there are other defects from a different area, then it is convenient to
sample them on a common algorithm and feed it into algorithm. This enables the identification of the

Risk-Based Testing 10 Sonata Software Limited

Technical White Paper www.sonata-software.com

Risk exposure coefficient. Wherever the Risk exposure co-efficient is high, those are the areas that need
100% coverage.

In this specific example the probable failures in terms of change requests, test interfaces, inexperienced
developer, field validations, business rules validations, positive and negative scenarios, third party
interfaces, system integration testing, backend verifications, UI elements testing, content verifications,
content validations, error messages verifications & validations, cross browser testing, platform
compatibility testing, functional end-to-end flow testing have been taken into consideration. Certain
weights have been assigned to these areas, to calculate the Risk exposure for various iterations from a
value of 0.5 to 2.0 and to achieve a constant consequence as 2 to arrive at the probability of failure.

Iterations:
Re(f) {0.5, 1.1, 1.2…..2.0}
having C(c&v) = 2
Note: Open Source TM (Algorithms and IP) was used for this study

Results:

There are 3 different depictions:

Graph 1 Graph 2
Defects in Releases Defects in Releases

Sample: 1200 TC - Continuous Sample: 1200 TC - Random

Risk-Based Testing 11 Sonata Software Limited

Technical White Paper www.sonata-software.com

Graph 3
Defects in Releases

Sample: 1200 TC - RBT

Graph 1: It is a sample of 1200 test cases and it is continuous. A particular release has undergone 10-12
iterations and the test cases have been run 1200 continuously. The test cases have been run by an
automated test suite which has been developed on an Open Source Framework. The defects in Graph 1
found in the different iterations and how the applications are stabilized over a period of iteration. In this
case all the 1200 test cases on all releases in an automated way.
Graph 2: Test cases were selected at random, without any logic or reason. This particular method also
provides the defects but the amount of the defect captured is lesser when compared to the amount of
defect captured while the entire set of test cases is run.

Graph 3: The statistical-based algorithm is run and 400 out of the 1200 test cases are sampled. Only a
minimum number of test cases are run but they capture the maximum number of defects. There is no
variation in results as compared to the number of defects identified when all 1200 test cases are run.
This concept saves a lot of effort and cost that leads to an impressive turnaround time.

Risk-Based Testing 12 Sonata Software Limited

Technical White Paper www.sonata-software.com

Inferences of the concept

As a result, a 50 – 60% reduction on testing effort (400 test cases out of 2000 test cases) is achieved,
generating data for multiple set of defects scenario. These defect scenarios or data will be applicable in
subsequent releases. In a similar project or a similar type of release the respective test cases can be
pulled straightaway. There is no impact on the critical factors that is on the database side or in the
performance side or in the quality side by doing this.

This has resulted in the 40-50% of the testing cost reduction. In the enterprise environment when
multiple projects in multiple streams are run, each product needs to be tested on a particular day or a
particular time segment. To do this testing continuously in all these areas by using risk-based approach a
greater bandwidth is required to run these tests in spite of running less amount of test cases and achieve
more amount of coverage.

The other important advantage is Defect Predictability. Number of defects can be predicted for a certain
size of application. This helps us in estimating the time for defect fixes. In a project lifecycle analysis,
requirements, developments, testing and release need to be planned. Often, the element that is missed
out is time and effort required for the defect fixes. If a decent estimate of defect fixes can be identified,
then it is easier to estimate the time required to complete it.

Since the developers or the programmers required for this program are selected right at the beginning,
there can be optimized use of relevant expertise and hence the risks can be handled efficiently.

Open Source Test Management

In this activity of statistical model or the iterative algorithm and selecting the relevant test cases and
then running those test cases for execution, you require a proper test management system. Either it can
be a quality center or a QA director or any other tool which is capable of doing that. Sonata has
developed an Open Source Test Management System which is integrated with defect and test
management areas. It houses the entire Risk-Based Testing algorithm and the data for various values of
probabilities of failure in different areas in terms of classification. It has become easy to do the
automated test cases predictability and organize the test cases according to the functionality and defect

Risk-Based Testing 13 Sonata Software Limited

Technical White Paper www.sonata-software.com

areas. Customized simulation for various resources can be obtained. This reduces around 70% of the
regression test cost and 50% improvement in controlled releases.

Summary

The statistical approach for Risk Based Testing is a proven model. It is capable of simulating error
injection, analyzed impacts associated with failures. More importantly, it is simple and cost effective. As
an added advantage an Open Source System supporting it is also available. The algorithms are scalable
and iterative and these algorithms can be used or extended for any type of testing (Web testing, Data
Testing, testing of ERP systems). Customized reports in terms of the available number of risk prone test
cases are generated which definitely need to run in a particular release. All the data is available in a
report format. The Open Source Test Management System houses all of these activities and functions
together and being provided to the customer as a package.

Risk-Based Testing 14 Sonata Software Limited

Technical White Paper www.sonata-software.com

CORPORATE OFFICE Singapore

APS Trust Building
Bull Temple Road, N. R. Colony
Bangalore 560 019, India
Tel: 91-80-3097 1999, Fax: 91-80-2661 0972
Email: info@sonata-software.com
1, North Bridge Road, #19-04/05
High Street Center
Singapore – 179094, Singapore
Tel: 65-633-724-72, Fax: 65-633-740-70
Email: info@sonata-software.com

UK
WORLDWIDE OFFICES 5, Churchill Court
Dubai 58, Station Road, North Harrow
Office # 507, Thurraya Tower No.1 Middlesex HA2 7SA, UK
P O Box 502818, Dubai Internet City Tel: 44-20-8863 8833, Fax: 44-20-8863 5533
Dubai, United Arab Emirates Email: info-uk@sonata-software.com
Tel: 971-4-375-4355, Fax: 971-4-424-0132
Email: info-me@sonata-software.com USA
39300 Civic Center Drive,
Germany Suite 270, Fremont, CA 94538, USA
TUI InfoTec GmbH Tel: 510-791-7220, Fax: 510-791-7270
Karl-Wiechert-Allee 4 Email: info-uswest@sonata-software.com
30625 Hannover, Germany
Tel: 49-511-567 5296 2018 156th Ave NE, Suite 100,
Email: sales@tui-infotec.com Building F, Bellevue, WA 98007, USA
Tel: 425-372-2167, Fax: 425 484 7799
India Email: info-usnw@sonata-software.com
6, Richmond Road
Bangalore - 560 025, India 1901 North Roselle Road, Suite 800,
Tel: 91-80-3097 3299, Fax: 91-80-2248 4045 Schaumburg, IL 60195, USA
Email: info@sonata-software.com Tel: 847-517-6310, Fax: 847-517-6313
Email: info-uscentral@sonata-software.com
193, R.V. Road,
Basavanagudi, 11330 Lakefield Drive, Bldg #2, Suite 200
Bangalore - 560 004, India Duluth, GA 30097, USA
Tel: 91-80-3097 2999, Fax: 91-80-2656 7487 Tel: 770-814-4213, Fax: 678-623-0236
Email: info@sonata-software.com Email: info-usse@sonata-software.com

Sonata Towers, Global Village, 275 Grove Street, Suite 2-400

Pattenegere & Mylasandra, Newton, MA 02466, USA
RVCE Post, Mysore Road, Tel: 617-663-4866, Fax: 617-663-6127
Bangalore - 560 059, India Email: info-usne@sonata-software.com
Tel: +91-80-3097 1499
Email: info@sonata-software.com 212, Carnegie Center, Suite 206
Princeton, NJ 08540, USA
1-10-176, Begumpet Main Road Tel: 609-919-6325, Fax: 617-663-6127
Opp. Hyderabad Public School Email: info-useast@sonata-software.com
Hyderabad - 500 016, India
Tel: 91-40-3981 3899, Fax: 91-40-2776 4831
Email: info@sonata-software.com

If you have any experiences related to Risk Based Testing that you would like to share with us, please
write in to us on info@sonata-software.com

Risk-Based Testing 15 Sonata Software Limited