This research note is restricted to the personal use of namit.bajaj.demo@gartner.

com
This research note is restricted to the personal use of namit.bajaj.demo@gartner.com
G00238115
Improve Your Quality Assurance When Using
SaaS
Published: 15 November 2012
Analyst(s): Thomas E. Murphy, Daniel Sholler, Christian Hestermann
Many users hope that moving to SaaS will help to reduce cost of ownership
and shorten time to market. However, loss of control over changes requires
users to spend additional effort improving their quality assurance to avoid
business disruptions.
Key Challenges

Based on marketing messages by vendors, users are lured to assume that using a software as
a service (SaaS) solution requires less testing and quality assurance (QA) efforts than using an
on-premises solution, which is not true.

Users of a SaaS solution have less control over the changes applied to their solutions; in many
cases, they might not even know in detail what was changed, nor do they have visibility into the
QA effort (or lack thereof) that is applied when changes occur.

The changes can theoretically appear at any time, although more vendors offer options to
postpone changes for a limited period of time.
Recommendations

Do not simply expect the vendor to be more accurate in its QA and testing than you are, but
understand where risk areas lie and the contractual or test mitigations you will need to apply.

Make sure you have a good communication channel to vendors to notify them of issues, and to
ensure early access to changes.

Automate regression testing to run as often as possible, or at least as necessary based on your
provider's change patterns.

Establish error-handling and problem resolution processes as a part of the communication plan
when negotiating the SaaS agreement.
This research note is restricted to the personal use of namit.bajaj.demo@gartner.com
This research note is restricted to the personal use of namit.bajaj.demo@gartner.com
Introduction
Software quality and testing are a constant source of grief in software production. When QA finds a
defect, it delays the delivery of the software, and when a defect slips through, QA didn't get the job
done. All in all, everyone looks for ways to reduce the cost of testing and it seems as though they'd
like to forget about it all together. This is driving solid growth in offshore outsourcing of testing, and
has been one of the desired values of utilizing packaged software. Now, SaaS proposes to reduce
the operating cost of software on top of the development and testing savings of packaged software.
However, there is a distinct difference in current SaaS offerings the loss of connection to the
change stream (see "The Impact of the Cloud on ERP and Business Application Planning" and "The
Impact of the Cloud on Architecting an ERP Solution").
While the focus on software testing varies greatly across organizations, with traditional software
delivery the consumer (or implementation partner) has some level of control over changes to the
system. In a traditional upgrade, users should understand that they have to extensively test new
versions before putting them into production. In a SaaS application, it is easy to be misled into
neglecting testing because the upgrade happens in a transparent way. Indeed, vendors tend to
make prospects believe that with a SaaS solution, testing the stability and performance of a newer
version is no longer needed. However, depending on the nature of the changes or how the SaaS or
platform as a service (PaaS) solution is extended or integrated into other systems, customers may
have to spend the same amount of diligence and prudence, or more, as with any other business-
critical application (see "Ten Challenges for Successful Composite Multienterprise SaaS
Integration"). But, without access to the change stream, instead of potentially having changes (and,
therefore, bugs) introduced on a periodic basis, they can be introduced daily (and without your
knowledge). As a whole, traditional package software vendors have not had a great record when it
comes to change management and upgrade support, which does not bode well for the transition
into cloud computing platforms.
Application managers and business users of SaaS-provided business applications face significant
risks when their mission-critical application breaks and no longer delivers correct results.
Depending on the functional scope that is deployed, these risks can affect every area of the
business. In severe cases, this can affect a company's ability to serve its customers or process their
orders.
Analysis
Understand the Costs and Benefits
Testing is a cost-benefit exercise. You can never test everything, because everything would literally
be everything. A combinatorial number of possible use cases, failures, data inputs, etc., would
theoretically need to be tested. The majority of organizations instead test based on the risks they
are trying to control. With SaaS there are hidden costs that you must be aware of which result from
the loss of quality and change management control. Therefore, the question in the SaaS model is
that since you do not know the change stream, is the risk that a vendor-applied change will create
an error in your process something that you want to spend time and money to control, or not? Most
Page 2 of 8 Gartner, Inc. | G00238115
This research note is restricted to the personal use of namit.bajaj.demo@gartner.com
This research note is restricted to the personal use of namit.bajaj.demo@gartner.com
people choose not to make these expenditures, because it means not accepting the change until it
is tested. More SaaS providers give their tenants the option to postpone the activation of a new
version for a limited period of time, but because all tenants in a multitenant SaaS application are
expected to use the same code, no guarantee is given that any found errors will be removed before
the new version is finally deployed. With traditional software packages, you have some access and
control over what is changing and an ability, through your own knowledge or a consulting party, to
understand the risks and how to mitigate them. In SaaS, we are led to believe that all is well, but no
software is bug-free and changes to vendor-provided software have traditionally caused headaches
for users. Thus, make sure you carefully measure the impacts and value of customization and
integration, and how changes will affect the cost of using the software.
Don't Lose QA Control
In a SaaS environment, your organization is paying for outcomes. You are paying the vendor to take
responsibility for many of the common challenges in running applications, at the cost of a limited
ability to modify the deployment and configuration choices. For this reason, it is vital that the focus
of testing shift from feature/function behavior to outcomes. Obviously, in the context of a new
release from a SaaS provider, it would be prudent to test the new things in that release, but that
testing should be done to ensure that the results are what they need to be. Even if a particular
feature does not work as expected, it may not compromise the ability of the system to create
usable results. For example, if tax calculation was done internally in the previous version and is now
done by reaching out to an external service in the new version, the same input data needs to
consistently deliver the same output data, regardless of how the data is derived and regardless of
any additional features provided. Nevertheless, even if the underlying code is provided as a service
and the focus shifts from function to outcome, your focus in QA control still has to be on the
functionality of the solution. You still need to ask: "Does the code provided as a service, when
applied to my data, still produce the expected outcome?"
Critical aspects of remaining in control require you to demand certain behaviors from your SaaS
vendor. The vendor must notify you when changes have occurred, and preferably give you the
option to roll those changes into your system (and back them out) during a transition period.
Second, they must provide some impact analysis for new capabilities, either functional or
nonfunctional. Finally, new capabilities need to be very clearly detailed, so that the "failures" that
are discovered are not failures of expectations. Finally, the vendor must have a clear policy that
allows you to remain operational during a time when an error has been discovered but not yet
remedied.
Ensure Compliance
Vendors will have to take on a greater responsibility for the U.S. Sarbanes-Oxley (SOX) Act and
other regulatory compliance issues in SaaS delivery. However, anywhere you can change a
calculation or a workflow, or anywhere data comes out of the system and then re-enters it, requires
end-user validation. Users will need to add regulatory issues as part of the buying criteria, e.g., has
the vendor been certified? Certification of the vendor's practices, security, load/stress, and
compliance should all be part of your decision on whether to choose that vendor (see "Look Before
You Leap Into Cloud Computing"). You must focus on the fact that purchasing a SaaS solution is
Gartner, Inc. | G00238115 Page 3 of 8
This research note is restricted to the personal use of namit.bajaj.demo@gartner.com
This research note is restricted to the personal use of namit.bajaj.demo@gartner.com
buying a service, not software. This means that you are relying on the vendor to manage the
software correctly, as well as its operation, support and change control. The issue that you are
testing is the suitability of the service as offered to your business and its requirements.
Another potential area of concern is the quality of the vendor's change control practice and how the
provider notifies users. When the large offshore outsourcing vendors started up, they sought out
Capability Maturity Model Integrated (CMMI) Level 5 and International Organization for
Standardization (ISO) 9001 compliance as a way to show they were trustworthy and reliable. If you
are expending the effort to be ITIL-compliant in your own operations, would you not also expect
similar levels of certifications from your service provider, like the Cloud Security Alliance GRC Stack
or Oasis' TOSCA?
With financial systems, another challenge occurs when the fundamental nature of the application is
correct, but something about the specific dataset utilized causes an issue. In software testing,
100% path coverage may be obtained, but it is impossible to test all valid/invalid data inputs. In
general, you are responsible for your data and for ensuring that it works correctly with the software,
not the other way around. Auditors will not accept excuses or claims that you don't know what
happened to the application.
Automate Regression Tests to Run Whenever Change Might Have Occurred
With traditional business application software in the field, the tools to manage change sets and
testing are improving in response to those changes creating upgrade issues for customers. For
instance, SAP Solution Manager or other SAP change tools identify what has changed, which tests
need to be rerun, etc. In SaaS, we will either need to get to that point, or strongly limit customization
and integration. It also means that the test team should have a good set of automation around the
key transactions, integrations and customizations, and should run those potentially every day.
These tests become an internal change notification system. But that still leaves a big window when
a change can occur and your software breaks. This is a risk you are agreeing to live with, unless you
contractually protect yourself from vendor change without notice.
This risk is currently mitigated by service-level agreements (SLAs). Most companies are not
concerned about software quality; they are concerned about risk management, and the goal is to
spend the least you can to have a level of risk you are willing to live with. In moving to SaaS, firms
are trying to reduce testing costs, not increase them. Costs can be controlled by limiting
customization and integrations, although this will limit the application's business value. Limiting
these will curtail the need for concern that vendor updates will break your system and reduce the
need for concern about change management or testing. However, in all cases where the software is
enterprise-deployed, or mission-critical, understanding the vendors' change management and
operational procedures and building a selected set of validation tests around core functionality is
critical.
Be Even More Diligent When Using Application PaaS
Note that PaaS/application platform as a service (aPaaS) means that you take more responsibility
back, and this complicates issues. Rather than just being software that you run (e.g.,
salesforce.com) with minor configuration changes, it is a hosted development platform (e.g.,
Page 4 of 8 Gartner, Inc. | G00238115
This research note is restricted to the personal use of namit.bajaj.demo@gartner.com
This research note is restricted to the personal use of namit.bajaj.demo@gartner.com
Force.com). In this case, you have software developed on top of the platform, and platform version
changes could break functionality. In a traditional development world, this is the same as the
relationship with an application server. The only difference is that in the aPaaS case, you are not
responsible for the deployment and configurations of the application server. This creates the
potential for breaking existing integrations as well as user skills if there isn't a provision for
backward compatibility or support from the vendor to run an older version of the application. There
are connections between this and testing for service-oriented architecture (SOA) and SaaS. This is
especially true if services are primarily externally provided, such as credit authorization, payroll,
shipping, etc. When the vendor changes a service (maybe not even a functional service), the
performance characteristics change, and the system must deal with this (see "APaaS: A Step to a
'Killer App' for Cloud Computing?"). Programs deployed in aPaaS environments must be managed
the same way as any custom program, with the exception of the issues around platform
configuration and deployment.
Take Ownership of Change Discovery and Defect Management
The issues of defects in off-the-shelf software and SaaS-provided solutions are similar in that the
vendor is responsible for the defects in its software, and the user is responsible for the defects in
any customization or integration. However, the difference with SaaS is that the vendor now has
responsibility for platforms and configuration, which means that they are responsible for most of the
nonfunctional requirements of the systems. Because of this transition in responsibility, it is critical
that these vendors provide a method of informing their customers about what defects have been
detected, and what is being done (or can be done) to mitigate them.
Many SaaS providers are more willing and able to provide transparency regarding defect
information than traditional vendors. Because these vendors are trying to disrupt the market and
tend to make use of more-agile practices, and because they need to evolve solutions to manage the
impact of change on software users, it is in the vendors' best interest to make use of a more public
stream of information. However, this will still require user resources to monitor and understand the
impact on help desk trouble tickets, defects and feature requests, and to ensure that changes don't
negatively impact running solutions.
Application managers and key users need to diligently study the information on changes delivered
by the SaaS provider and figure out which of these changes might affect their application. Because
of the fact that all changes are installed into their application, they also need to pay attention to
changes that they do not even intend to activate and use, because any change can potentially have
a negative impact on the correct execution of the deployed software. Some more extensive
changes will require the adaptation and extension of the test cases and test data to ensure that
these changes are being tested in future regression tests.
Should you find that your provider is not providing the needed information or the necessary level of
detail, you must pressure them to correct this. If necessary, the corresponding SLAs need to be
completed in the course of negotiating the next renewal of your subscription.
Gartner, Inc. | G00238115 Page 5 of 8
This research note is restricted to the personal use of namit.bajaj.demo@gartner.com
This research note is restricted to the personal use of namit.bajaj.demo@gartner.com
An Example
SaaS seems to offer the benefit of helping you avoid upgrade pain; you just gain new features
without all the planning and deployment effort involved in on-premises solutions (see "Top Five
Ways to Make Your ERP Upgrade a Success"). For example, you don't worry about your Google
docs working tomorrow, but you hope for improved functionality. One advantage that vendors gain
in Web applications is knowing exactly how many people use specific functionality. Microsoft has
been able to do this with its applications whenever you select the "help provide feedback" option
on product installation. This feedback lets the product manager understand how many people use a
function, how often it is utilized, the potential problems that may occur, etc. This feedback may
show the need to change a feature for usability, or show a feature that isn't widely used (but is still
critical to you), and is not profitable in the vendor's ROI assessment.
A major issue occurred when Microsoft introduced version 4 of Excel. Prior versions supported
macros in foreign languages, e.g., German, so you could say "Berechne <xyz>" instead of
"Calculate <xyz>." But in the next version, this support was dropped, so users had to manually go
through and redo all their macros. Because the software was locally installed, users could control
the rollout of the change and potentially revert to an earlier version while the broken formulas where
updated. Returning to the example of Google docs, basic functionality should be unaffected after a
change. However, if you can use macros, formulas etc., then you hope they still deliver exactly the
same results as before because you don't have a way to revert to a previous version. Thus, when
you move to SaaS, the pace of change is no longer yours to control, and your organization has to
be able to respond to that change as quickly as possible. Because you can't protect yourself from
change (beyond postponing the activation of a new version), a plan should be put in place to ensure
a quick response in addressing issues that arise due to vendor changes. A critical element of this
plan should be the establishment of a key internal contact person who works directly with the
vendor when issues arise.
These mechanisms have evolved. We have seen, in the world of constant betas the Web has
delivered, that there are ways to opt into early access to changes. Vendors need the ability to
understand the effects of these changes, and the only way to get the combinatorial advantage is to
put users on the software. In SaaS business applications, more providers now offer sandboxing
environments where users get access to the next version before it is deployed. Do not simply
expect the vendor to be more accurate than you are, but understand where risk areas lie and the
contractual or test mitigations you will need to apply. In addition, make sure you have a good
channel to vendors to notify them of issues, and ensure early access to changes. Think of SaaS as
a partnership where you are helping the vendor with early feedback and reports, and where they are
building enhancements and resolving issues.
Bottom Line
When moving to SaaS, don't expect defects to magically disappear. Even if you don't know exactly
what has changed and cannot avoid using the defective version until it is corrected, a moderate
testing program will mitigate risks and help avoid damage.
Page 6 of 8 Gartner, Inc. | G00238115
This research note is restricted to the personal use of namit.bajaj.demo@gartner.com
This research note is restricted to the personal use of namit.bajaj.demo@gartner.com
Recommended Reading
Some documents may not be available as part of your current Gartner subscription.
"Software as a Service: Component Development Challenges"
"Look Before You Leap Into Cloud Computing"
"SaaS ERP Only Reduces Part of the Effort Needed to Implement and Operate Your ERP"
"Top Five Ways to Make Your ERP Upgrade a Success"
"Case Study: Deploying SaaS-Based ERP"
"ERP/Business Applications and the Public Cloud: A Life Cycle Assessment Methodology and Key
Focus Areas"
"Oracle, Salesforce.com and SAP Acquisitions Will Impact Your SaaS Strategy"
"The Impact of the Cloud on Architecting an ERP Solution"
"The Impact of the Cloud on ERP and Business Application Planning"
Gartner, Inc. | G00238115 Page 7 of 8
This research note is restricted to the personal use of namit.bajaj.demo@gartner.com
This research note is restricted to the personal use of namit.bajaj.demo@gartner.com
GARTNER HEADQUARTERS
Corporate Headquarters
56 Top Gallant Road
Stamford, CT 06902-7700
USA
+1 203 964 0096
Regional Headquarters
AUSTRALIA
BRAZIL
JAPAN
UNITED KINGDOM
For a complete list of worldwide locations,
visit http://www.gartner.com/technology/about.jsp
2012 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. or its affiliates. This
publication may not be reproduced or distributed in any form without Gartners prior written permission. If you are authorized to access
this publication, your use of it is subject to the Usage Guidelines for Gartner Services posted on gartner.com. The information contained
in this publication has been obtained from sources believed to be reliable. Gartner disclaims all warranties as to the accuracy,
completeness or adequacy of such information and shall have no liability for errors, omissions or inadequacies in such information. This
publication consists of the opinions of Gartners research organization and should not be construed as statements of fact. The opinions
expressed herein are subject to change without notice. Although Gartner research may include a discussion of related legal issues,
Gartner does not provide legal advice or services and its research should not be construed or used as such. Gartner is a public company,
and its shareholders may include firms and funds that have financial interests in entities covered in Gartner research. Gartners Board of
Directors may include senior managers of these firms or funds. Gartner research is produced independently by its research organization
without input or influence from these firms, funds or their managers. For further information on the independence and integrity of Gartner
research, see Guiding Principles on Independence and Objectivity.
Page 8 of 8 Gartner, Inc. | G00238115