Automating Your

The Definitive Guide

Automating Your QA: The Definitive Guide

Automating parts of your QA process is vital to making sure you improve and maintain the quality of your apps
while drastically speeding up your release cycles. It helps keep release cycles on schedule.

It may seem tough to replace the insights Automated QA also provides value for
of veteran software testers, but some
projects are ideal for automated testing,
driven by customizable code that fits your
85
increase
% people on automation teams by letting
them prioritize high-value and exploratory
tests, and helping organizations structure
in test
release cycle like a glove. teams that are set up for growth. It helps
automation over the
past two keep software development processes
Companies automate their testing for
years.1 agile, and has become an essential
many reasons, including increasing speed
DevOps practice.
to market, reducing build turnover, allowing
for monthly app releases, delighting users with new Because of all this, interest in and demand for QA auto-
features without the agitation of bugs, and automating mation is on the rise, with an 85% jump in test automation
ads, analytics and video tracking tests. This lets them use in the last two years across all industries.1 But actual
QA as a competitive advantage to drive growth, deliver knowledge on how to implement the tests is hard to find.
apps to customers more quickly, and focus their developer This eBook will explain the benefits of automated QA,
talent on making the best products possible. provide actionable insights for QA leaders and managers,
and show how partners like Testlio can help implement
automated QA into a companys software development cycle.

1 Haynes, M. Test Automation Usage on the Rise. Retrieved from https://appdevelopermagazine.com/4935/2017/2/8/Test-automation-usage-on-the-rise/

AUTOMATING YOUR QA: THE DEFINITIVE GUIDE 2

Contents

Automating Your QA: The Definitive Guide...................................2

Six Top Benefits of Automated QA.................................................. 4
Enhancing Manual Testing with Automation.................................5
Automating Your Mobile Apps........................................................... 6
Case Study: National Media Company.............................................7
How Automated QA Works................................................................. 8
Running Automated QA Early and Often.....................................10
Maintenance............................................................................................... 1 1
Reaping the Rewards............................................................................. 12
Implementing Automated QAEnter Testlio.............................. 13
Automating with Testlio........................................................................ 14
The QA Partner You Can Trust........................................................... 15
Get Started................................................................................................. 16

AUTOMATING YOUR QA: THE DEFINITIVE GUIDE 3

Six Top Benefits of Implementing QA testing is one of the biggest challenges
to releasing new apps and quickly updating existing
Automated QA ones. When putting together a QA plan, companies face
a difficult quandary: Should they rush through QA, and

1 Products are delivered more quickly, with compromise quality in order to get an app out the door
a much faster time to market. quickly? Or should they slow down their release cycle to
make sure apps are of the highest possible quality?

2 Apps end up with fewer bugs and a

higher quality.
Many companies opt for speed, and they end up sacrificing
quality and risking their reputation in order to get out
quick releases: 72% of IT teams rely on customer com
3 People are freed from performing
repetitive tasks so they can prioritize
high-value and exploratory tests.
plaints to discover user experience issues.2

It doesnt have to be that way, though. A comprehensive

QA plan that includes automation can unlock customer
4 Companies can build QA teams that can
scale as development scales. value and better serve the end user. Companies that
automate the right QA activities in combination with

5 The depth and scope of testing is

dramatically increased. Long tests can be
left unattended and performed thousands
scalable manual testing outperform the competition.
Thats because they deliver products a lot quicker, with
fewer bugs, and with a higher quality. These benefits
of times for each test case.
apply across industries, whether youre in business-
to-business, consumer, media, marketing, customer

6 Software development processes

are kept agile.
relationship management, or customer service. In fact,
the World Quality Report, which focuses on QA, concludes,
The automation of QA activities is not only required but
is the core enabler of increasing throughput and velocity.3

2 https://testlio.com/solutions/automated-software-testing/
3 https://testlio.com/solutions/automated-software-testing/

AUTOMATING YOUR QA: THE DEFINITIVE GUIDE 4

Enhancing Manual Testing with Automation

Any comprehensive QA plan needs to include both automated and manual testing. Thats because automated QA
doesnt replace manual testing, but enhances it. Testing an apps core functions over and over can quickly become
a monotonous task, it takes vital resources away from high-value, exploratory testing. As a result, continual manual
testing of core functions can lead to decreased accuracy, with bugs and errors creeping into apps. For many enterprise
companies, automated QA does a far better job at these tedious tasks than humans do.

By automating the testing of basic, core functions, people can be freed to test more complex, higher-value features.
In addition to improving quality, this also lets companies build QA teams that can scale as development scales.

Automated QA can also improve team morale. Freeing

team members from repetitive tasks so they can spend
The automation of QA activities is not
their time on more challenging and rewarding work
only required but is the core enabler of
improves their skill sets, and makes it more likely they
will stay interested and engaged in their work. The increasing throughput and velocity.
enterprise, in turn, reaps benefits in reduced turnover
and increased productivity.

World Quality
Report

AUTOMATING YOUR QA: THE DEFINITIVE GUIDE 5

Automating Your Mobile Apps

Automation also dramatically increases the depth and scope of testing. Long tests can be left unattended and
performed thousands of times for each test case, something that would be far too costly to test manually.

Automated QA is particularly
well-suited for Android and iOS
apps, because of their device-
specific specifications. That can
mean running tests on dozens
of different devices, with
differing versions of the OS on
each. That is impractical and
costly for manual testing. Its
ideal for automation.

AUTOMATING YOUR QA: THE DEFINITIVE GUIDE 6

Case Study: National Media Company
Automation is not about saving money,
it is about increasing volume, which
means achieving faster build turnover. With
that you can get quarterly app releases down
to monthly app releases. With that you are
able to release new features/patches faster
and respond to the market faster. And that is
where the value starts coming in.
QA Director,
National Media
Company
AUTOMATING YOUR QA: THE DEFINITIVE GUIDE
At Testlio, weve seen the benefits of automated QA close
up. Using Testlios automated QA services, a well-known,
major international media company improved its testing
productivity so much, it was able to expand the work of its
iOS team to test Apple TV as well without adding any staff.
Before using automated QA with Testlio, the company got
new builds out only every two to three weeks. Now, with
automated regression testing done daily, they can get out
builds on a daily basis, and have dramatically shortened
the time upgrades make their way into Apples App Store.
With automated QA, they have been able to more quickly
determine whether newly released content makes its way
into their apps. The tests also automatically determine
whether the apps newest translations made it into the
build. And they have automated ads, analytics and video
tracking tests.
All this has led to better-quality apps released more quickly
and a streamlined QA team built for growth. The end result:
The company has used automation to outpace the competi-
tion when companies are vying for our attention on mobile
devices more than ever.
7
How Automated QA Works

Once youve decided to automate QA, one of the first and most important tasks is to determine which portions of a
QA process can be automated, and which still require manual labor. Automated tests are not a replacement for manual
testers. To find most bugs, human testers are needed to poke around, just as a real user would upon an apps release.
A human testers creativity cannot be replacedwho would think to test an app by shaking the phone, with the back
button held, as the Wi-Fi is switched off?

In practice, only the most basic, core functionality of the app should be
covered by automated tests: Does the app load? Can the user log in? Can
they navigate the menu? Can they watch a video? These types of functional
tests are the lowest-hanging fruit, being well-defined, easily repeatable,
and useful throughout a products entire lifecycle. These mundane tests
can also be prone to human error. Its better to focus manual testers on
creative feature-breaking and bug-hunting, and leave the boring tasks for
the robots.

Targeting basic, high-level functionality means that the tests can be simple,
too. Automated tests are written to model user behavior, often as a sequence
of keywords, actions like clicking on a button or creating an account. This
simplicity keeps tests agile, so they can be written quickly and maintained
efficiently.

AUTOMATING YOUR QA: THE DEFINITIVE GUIDE 8

The key to an effective automated QA system is
being picky about what to automate in the first place.
Here are examples of tests that should be automated, and those better left to manual testers:
Automated
Can a user log into their account?
This is a basic yet critical function of an app, and
straightforward to write an automated test for.
This might be used as a smoke test to alert teams
that network or account code has broken in a
development build.
Are all the content endpoints working?
For apps with large amounts of multimedia content,
making sure its all viewable by users is important. Issues
can stem from the CDN, incorrect metadata, or simply
missing content on your end. When thousands of files
are involved, only automated testing can cover it all.
AUTOMATING YOUR QA: THE DEFINITIVE GUIDE
Manual
Is the app stable?
You could never write enough test cases for this. Automation
may be able to detect crashes found while doing basic
navigation through the app. But the majority of issues will
be stumbled upon by human testers probing the software.
Especially with mobile, many bugs dont manifest until in
real-life situations, when network connectivity, background
device programs, and button-smashing come into play.
Can a user complete a complex task, like leaving a
comment that links to another user?
It may be possible in theory to write an automated test for
this. But it would require a long series of actions, which is
prone to breaking and carries heavy maintenance costs.
Confirming the success is difficult too, as there are side
effects to check, like whether the linked user received
a notification.
9
Running Automated QA Early and Often

After tests are written, it is crucial to continually run them throughout development. It may be tempting to push off
testing, remember the QA mantra: Test Early. Test Often. Test procrastination has been the doom of many go-live
deadlines, as app-breaking bugs go unnoticed until final checks. But detecting regressions through automation can
provide incredible boosts to productivity throughout the entire development cycle.

One component of automated QA More detailed regression tests

is smoke tests. Smoke tests are the
highest-level tests, designed to quickly
reveal that something has broken in
a build. They ensure that bugs are
addressed at their onset, before they
creep into the codebase and increase
in complexity.
are also run often to ensure core
functionality. Smoke tests can
help prioritize which builds are
selected for more rigorous testing.
You dont want to burn money on
heavier testing when users cant even log in or use basic
features. Regression tests are run less frequently, and
on more stable builds, but give deeper insight into the
health of the project.

Because smoke tests are lightweight, they can be run frequently and across multiple buildsoften nightly against all
branches of the codebase. The frequency of smoke tests means that when they do fail, its easier to quickly pinpoint
and address the issue. And when theres no smoke, it allows teams to have confidence in their development builds,
and to focus on pushing forward.

AUTOMATING YOUR QA: THE DEFINITIVE GUIDE 10

Maintenance

Automated QA has huge benefits, but nothing comes for free. As with all types of testing, but especially with automated
testing, constant maintenance is essential to ensure that tests are running smoothly. Changes in an app often cause a test
to fail, not because any real app functionality is broken, but because the test itself is broken. These false positives need to be
investigated by the QA teams, and then the test must be updated to work with the new build to properly test the functionality.

Maintenance is the top priority, and should come before creating additional tests. Before any new tests are written, the
existing tests need to pass. A QA system is complex, and may not always be in perfect order. But technical debt builds up
very quickly in automated QA systems, so its important to keep the machine well-oiled.

Creating an automated QA system is straightforward, but the true test is the

grind of keeping it running effectively. Maintenance can be the breaking point
for many organizations. Automated QA is far from a one-off project, so make
sure you have the resources to keep it operating.

AUTOMATING YOUR QA: THE DEFINITIVE GUIDE 11

Reaping the Rewards

Automated testing will alert you of major breakdownsapp crashes, resolution issues, entire elements no longer
rendering, and missing content endpoints. Its not that these bugs would otherwise go missingautomated QA simply
makes sure these issues are detected and addressed as quickly as possible.

There are some areas where automation can find issues that manual testing might miss.
Testing for the Android platform is notoriously difficult due to the extensive mix of OS
versions, screen sizes, and resolutions found in the wild. Automation can quickly test
many of these permutations. For applications that rely on thousands of different types
and sources of content, an automated approach is the only way to verify that everything
is accessible. Of course, manual testing must still be used to have full confidence that an
app can be released, but automation is an invaluable tool.

A stable development codebase has huge benefits as well. Developers who cant write
code because the build is broken are both frustrated and unproductive. Automated QA
is essential for avoiding this paralysis, and protects developers from the burnout caused
by merge issues inherent in distributed software projects.

AUTOMATING YOUR QA: THE DEFINITIVE GUIDE 12

Implementing Automated QAEnter Testlio

Building your own team from scratch to handle this complex

process is daunting. Some organizations may
already run automated tests on traditional
software projects, but the mobile testing
ecosystem is less mature, and requires more
expertise and know-how. With Testlio, you
leverage the expertise of our dedicated QA
team to enable your developers to focus on their
prime directive: to build and ship features.

Testlios high-quality automated software testing services

are powered by QA managers experienced in developing
automated scripts. We provide a fully managed QA service
that includes test plan creation, testing automation and
timely results. By developing automated scripts around
base features, we help you protect existing functionality.
We bring top QA talent to your business with automated
software testing and processes customized for your
development needs.

AUTOMATING YOUR QA: THE DEFINITIVE GUIDE 13

Automating with Testlio

We develop a comprehensive QA strategy that integrates with the tools and processes you already use. The world standard
for test frameworks is Appium. Testlio uses Appium to run tests against your build, not your source code. That way, we can
get started with your product quickly, and without any concerns about codebase privacy. Bugs often sneak in during the
seemingly innocuous process of preparing a development build for release. By running tests against your real, final build,
Testlio ensures that nothing slips through into the App Store, and out to your users.

Once youre onboard, Testlios team works with you to craft a testing and development plan tailored to
your release cycle and long-term goals, focusing on maintaining consistent product quality. The plan
will ensure you develop at speed, by creating, maintaining, and executing automated test scripts with
immediate results to give insight into the performance of critical features.

Youll get a single point of contact who will work with you through every aspect of setting up
the automated tests, implementing them, and providing ongoing support as new tests
and features are added. Close collaboration is central to the working relationship.
Testlio makes sure that it knows your release schedule and develops a plan that
supports it. Key to the process are the detailed weekly reports youll get from Testlio.
The reports include the results of all tests, and a clear description of the work Testlio
performed that week.

As features are added, Testlio works with client teams to prioritize which are candidates for automation, and which require
manual testing. After areas for automation are identified, Testlio can write specifications, and quickly implement tests to
ensure functionality moving forward.

AUTOMATING YOUR QA: THE DEFINITIVE GUIDE 14

The QA Partner You Can Trust

Keeping an automated QA system functional is difficult, and falling behind can Kick-off and onboarding
We finalize your requirements
quickly lead to chaos. Throughout the ups and downs of release cycles, Testlio
handles the maintenance of your tests, updating scripts to work with new
app versions. Testlio also ensures that the test platform itself is running as Planning & testing strategy
We build your strategy
expectedthis can be an unexpectedly large time sink for inexperienced teams. and testing plan
Although maintenance happens mostly behind the scenes, this is one of the
biggest benefits of choosing to partner with a QA expert like Testlio. Submit your build
Your testing team is ready
Another challenge of managing QA automation in-house is varying workloads. to get started
Things may be calm between sprints, but it is essential to have a full, effective
testing team to push out updates. It can be a challenge to operate a team large Testing magic happens
Truly magical ;-)
enough to handle these surges, and inefficient when outside of a release cycle.

Testlio offers scaling team sizes to address this. Throughout the release cycle,
your Testlio project manager will work with you to determine the resources you Results are back!
24-hour testing cycle
need at that moment. Team size is something that becomes transparentyou
no longer have to think about the number of testers required, just how many
Retrospective meeting
tests need to be run and maintained. This scaling capability is invaluable in What went well,
emergency situations, if for example a serious bug makes it into a release and what did we learn

a hotfix is required. Testlio has a massive community of vetted, experienced

testers, and is flexible to provide the resources you need in a pinch. Test, Rinse, Repeat
Deliver amazing customer
experiences

AUTOMATING YOUR QA: THE DEFINITIVE GUIDE 15

 Once you sign up with Testlio your single point of contact will schedule a meeting.
In that initial meeting, Testlio will ask you a series of questions to help develop
your custom strategy, such as:

What do you hope to gain or test with an automation strategy?

How much time would you otherwise spend testing the project manually?
What type of project management capabilities are you looking for?

At that first meeting, Testlio will help you organize your test team and advise
you on how you need to change your internal processes to take automation QA
into account. Testlio will also clearly delineate where automated testing lets off,
and where manual testing begins. And it will advise you on how to develop the
automated QA specs.

Now that youve seen what Testlios automated QA

can do for you company, whats next?

Get Started

THE QA PARTNER
AUTOMATING YOUR
PLAYBOOK
QA: THE DEFINITIVE GUIDE 16