You are on page 1of 43

Creating Killer Mobile Apps

The keys to developing, testing and launching


mobile applications that your users love

A Free eBook from uTest


Creating Killer Mobile Apps
Introduction……………………………..……………………………….……………………………..………..…………..…......……….2

The Mobile Boom………………………….……………………………………….…….…..….……..…………………….....……......4


Mobile User Growth Rate……………………………………..…………..…..….............................…….……………………..4
Why Mobile? …………………………………………………………….…..…..….............................…….………….....…….…..5
What’s at Stake? …..…..….............................…….…………...……….…………………………………………………………..6
Mobile App Strategy………………………….……………………………………….…….…..………..…………………....……...….7
Mobile Web vs. Native Apps….………………………………….………….…………..…....….……………………..……………..9
What the Users Say …………………………………….………….…………..…....….………….……..………..……………………11
Alternatives: WAC and Hybrid Apps …………………………………….………….…………..….......….…………….………12
Operating Systems, Devices and Carriers…………………………………………………………………..…………………….13
Mobile Testing Types, Tools and Methods……………………………….…….…..….……..………………..….....……....16
Functional Testing…………………………………….………….………………….……..…....….……………….……..…………….17
Usability Testing…………………………………….………….…………..…....….………….……..…………………..………………18
Performance Testing…………………………………….………….…………………………………..…........….………….……..…19
In-House Testing …………………………………….………….…………………………………..…....….…………..……….…….…20
Outsourced Testing …………………………………….………….…………………………………..…....……….………….…....…21
Emulators & Simulators………….…………………….…………………………………..…....….…………………………………..22
Beta Testing…………………………….……….………….…………………………………..…...................….………….……..……23
Key Testing Challenges………………………….……………………………….…….…..….……..…………………....……........24
Crowdsourced Testing………………………….……………………………………….…….…....….……..……………...…….....27
How It Works.……………..…………………………………………………………………….…….……………………………………..29
Challenges …………………………………….………….…………………………………..…....….……………………………..……..30
Benefits…………………………………………………………………………………………………………..……………………………..33
Appendix: Key Players in the Mobile Space…………………………………………………………………………..…..……..35

Table of Contents Page 1


The Challenges of Launching Great Mobile Apps
Launching a killer app is like catching lightning in a bottle – and this is especially true
of mobile applications. In an industry where there is fierce competition for media
attention, VC investment and/or consumer dollars, even the smallest flaw can doom a
mobile app (and the business behind it). With so much at stake, how can so many
mobile app companies ignore the benefits of usability testing, functional testing and
load testing? They want to launch apps that delight their users, right?
This eBook will walk you through the challenges of creating a high-quality mobile
application, and will show you how to overcome them. We will demonstrate how
companies of all sizes – from startups to enterprises – can overcome these obstacles
to build apps that their users love. In doing so, we’ll also cover:
• The Mobile Boom: Learn what’s at stake and also what’s driving the worldwide
mobile application frenzy.
• Mobile App Strategy: We’ll explain how building great apps begins long before
code is written or tested.
• Mobile Testing Types, Tools & Methods: Emulators vs. real testers; the
differences between mobile development platforms and other critical, strategic
decisions that will need to be made.
• Key Testing Challenges: How to navigate through the peaks, valleys, twists and
turns of testing your mobile application.
• Crowdsourced Testing: Discover how tapping into the crowd can help you
launch a killer app, while getting you to market faster and on-budget.
• Key Players: If you’re new to this mobile apps game, this appendix will highlight
the companies, technologies and players that you need to be familiar with
before getting started.

Mobile App Challenges Page 2


THE MOBILE APP BOOM

Page 3
The Mobile Boom
To say that the use of mobile applications is booming would be an obvious understatement. In 2010, mobile app stores
will record an estimated $6.2 billion in overall sales, including 4.5 billion app downloads. By 2013, some analysts expect
mobile app revenues to exceed $21 billion. With charts like this, we wouldn’t bet against them. As they say in the
investing world, the trend is your friend.

So if you think that mobile apps are for kids or that it won’t affect your industry/company/team, think again!

The Mobile Boom Page 4


Why Is The Mobile Market Booming?
The accelerating shift towards mobile applications (and the smart phones
they run on) is really quite simple, but frequently overlooked. Here are just a The Mobile Mantra
few of the reasons why users are flocking toward this technology:
•Accessibility: With mobile devices, the web is always at your side. This
“Much as we love freedom and
includes email, social media, video, games, news, stocks, sports, shopping choice, we also love things that just
and other actions. The ability to stay connected 24/7 has perhaps the most work, reliably and seamlessly. And
widespread appeal of all, and is the number one reason why the use of if we have to pay for what we love,
mobile applications and devices show no signs of slowing down. well, that increasingly seems OK.
•Adaptability: Mobile applications are sometimes thought of as being used Have you looked at your cell phone
primarily by kids and consumers only, but this couldn’t be further from the or cable bill lately?”
truth. Business professionals were, in fact, the earliest adopters of this
technology, and continue to make up a large segment of the overall user - Chris Anderson, Wired
base. This can be seen in the widespread use of business & productivity
apps for CRM, ERP, documents, presentations, email and messaging tools.
Of course, you can also play DoodleJump, too.
•Efficiency: Mobile apps (and to a lesser extent, the mobile web) have
greatly enhanced the efficiency of our everyday online activities. As we will
explain shortly, the goal of any mobile app should be to reduce each feature
to its most basic component. For example, users who want to see the latest
market news will generally download an app that focuses on this
exclusively. They do not want to enter a search term and click through five
pages to see this information. Mobile applications, in other words, cut out
the middleman and give users direct access to the things they want most.

The Mobile Boom Page 5


What’s At Stake?
Still think that mobile apps are for games, music and “entertainment
purposes only”? Do you really think that mobile applications won’t be The Mobile Future
relevant to your company, your team and your job a year from now?
According to a Morgan Stanley analyst,
Think again. As the adoption rate increases, mobile applications will the world is currently in the midst of the
become ever more critical to the success of your business (and your fifth major technology cycle of the past
career). Those who insist on waiting for further confirmation will have half a century. The previous four were
already been left behind. Companies that recognize this trend for what it the mainframe era of the 1950s and 60s,
is – a technological revolution – and take preemptive action to improve the mini-computer era of the 1970s and
the quality and usability of their mobile applications, will find themselves the desktop Internet era of the 80s. The
with a tremendous advantage. current cycle is the era of the mobile
Internet, she says — predicting that
Most companies will simply build a watered-down version of their within the next five years “more users
website. But those who understand the implications of the mobile boom will connect to the Internet over mobile
– and who understand all of the feats this technology is capable of – will devices than desktop PCs.”
build applications with enduring popularity. Which side will YOU be on?
- GigaOm
This mindset is essential. In order to take the steps required for
developing and launching a quality mobile app, you will need to believe
in what you’re doing. If you think mobile apps are simply a passing fad, it
will be reflected in your businesses and products.

Your users are counting on you. Don’t let them down.

Mobile Testing Challenges Page 6


STUFF YOU NEED TO KNOW FIRST

Page 7
Your Mobile App Strategy
One of the first – and most critical – decisions a mobile company will make is whether to develop a native application
or spend their resources optimizing for mobile web browsers.

Mobile Web Native Apps

vs.

Mobile Web vs. Native Apps Page 8


The Case For (and Against) the Mobile Web
Pros:
• Single platform (the web)
• One app to build, launch and maintain
• All that’s needed for some companies

Cons:
• Less controlled user experience
• Slower user experience
• No app store distribution
• Lack of standards across mobile browsers

Mobile Web Page 9


The Case For (and Against) Native Apps
Pros:
• Rich media functionality
• Controlled user experience
• Faster experience for users
• App store distribution

Cons:
• Multi-platform
• Must build, test & maintain multiple apps
• Not necessary for some companies

Native Apps Page 10


Native Apps or Mobile Web? What the Users Say

“There are particular native apps that I use on a regular basis, mainly due to
ease-of-use and simplicity. These include weather apps, travel planning apps
(kayak.com), music apps (Pandora). However, if I’m accessing news and
searching for information, I use mobile sites that automatically render as
mobile versions of their web sites (cnn.com, google.com)…
…I also foresee the future in which mobile web will be much easier to use,
and displace most (not all) of the need for native apps. Instead of downloading
the native app itself, there may be shortcuts on your mobile device for easy
access. Think about the recent progression away from desktop apps and into
purely web apps; mobile is simply a few years behind.”
vs.
**********
“I prefer native apps as they are more focused on your information needs. I
agree that the mobile web is more accessible and affordable, but it takes
longer to wade through all the sites to get to what you want. Mobile web apps
do solve the problem to certain extent, but they’re not on par with the native
apps in my opinion.”

User Testimonials Page 11


Alternatives to Native Apps and Mobile Web
Of course, deciding what form of mobile app you plan to offer isn’t an either/or
decision. There are alternatives, including: Wholesale App Community
•WAC: The emergence of the Wholesale Applications Community (WAC) may end “Though many doubted the long-
up providing software companies with their most flexible solution. WAC is a term viability of the WAC, it looks
consortium of mobile leaders, which has come to include carriers like AT&T, as though the group is gaining the
momentum it needs to make a
Verizon, Sprint, China Mobile, Orange and others, as well as manufacturers like difference.
Sony, Samsung and LG. This community intends to enable developers to develop
one mobile app that will work across ALL carriers, devices and operating systems. There’s a long way to go, and the
concepts the WAC are working
•Hybrid Apps: Similar to the ideas behind developing for the mobile web, hybrid towards are ambitious to say the
apps would allow developers to create mobile apps that share their backend with least, but the steps they’ve taken
their web application. They would create a native app “shell” or “wrapper” which so far seem to be the right ones.”
would utilize the same features, functionality, data and other characteristics.
*****
- MobileMarketing.com

However promising in terms of simplicity and cost, these two options are not
without some major drawbacks, notably that of competing interests. Will these
communities treat all devices, carriers, etc. as equals? So don’t rule out these
alternatives immediately, but proceed with caution.

Mobile Alternatives Page 12


Other Considerations
Mobile devices are evolving into increasingly sophisticated general purpose computers. This has led to the development of a
variety of platforms and operating systems. Today, the operating system inside your mobile device is as equally important as the
device manufacturer (OEM). The various features, GUI, processing speed, and most importantly, the applications available for
your device will depend greatly on the underlying OS, as well as the carrier and OEM.

Device Manufacturers Operating Systems Carriers

Mobile Decisions Page 13


Mobile OS: Past Present and Future
The mobile wars are heating up. This recent study by Quantcast shows that Android is gaining on Apple’s web iOS, although
the former still maintains a huge share of the overall market. Keep this trend in mind when deciding which operating
systems you intend to build an app for first.

Mobile Marketshare Page 14


Mobile Constants
No matter which route you choose – native apps or the mobile web – certain
tasks and backend functionality will remain a constant, including:

•API / web services Mobile Fact


•Architecture Consumer conversion (from surf to
purchase) on iPhone or Android native
apps is 3X - 10X times that of mobile
•Database web sites
The key lesson here is that app quality begins before coding or testing. You Why? Because users are more
should start by reviewing the makeup of your users. What features and comfortable with the device. In most
functionality do they want? What types of information are they trying to access? cases, the usability of the native app,
And most importantly, how can you reach them on their terms? including page load speed, is much
greater than the mobile web.
Other key decisions that every mobile company will have to make include:

•Static content vs. rich content

•Commerce vs. functionality

Mobile Constants Page 15


MOBILE TESTING TYPES AND METHODS

Page 16
Mobile Testing Types: Functional Testing
The most essential testing procedure will be to verify that the basic
functionality of your application. Early on, much of this attention should focus
on the download activities of the prospective user (i.e. that the download Mobile Stats
process is seamless and intuitive). From there, you can begin to verify more
specific functionality. As we’ll explain shortly, this can be done either via a • 51% of iPhone owners have five or
mobile simulator (desktop browser) or by using real-world testers. more games installed on their
devices
Finding device specific bugs: If, for example, you discover links that are not
functional on certain devices, then you should first determine whether it works •37% of iPhone owners play games
on the desktop browser. If it is reproducible on the desktop browser, then it is at least once per day – puzzle games
obviously not a device-specific bug, but rather a bug in the basic application. If
are the most popular
a link is functional in the desktop browser and it doesn’t work in the device,
then it should be identified as a device specific bug. •46% of Blackberry owners have NO
games on their device at all
Such functional testing should occur when the application is in the early
development phase. At that point, testers should perform a quick test on the
devices you deem as the having the highest priority, in order to find any
rendering or any functional issues as soon as possible. This way, development - Source: Mobile Entertainment
can have enough time to fix them or arrive at a proper decision as to whether
to support these devices or not. If fixing the bug involves high dev cost, they
probably won’t.

If this sort of functional testing doesn’t happen at the an early stage (and most
of the time, it doesn’t) there is the risk that the application might not work for
certain devices. This can lead dev/PM to changes the design, which can further
escalate costs.

Testing Types: Functional Testing Page 17


Mobile Testing Types: Usability Testing
Usability testing plays an important role for launching a quality mobile
application. Usability testing should occur as soon as the basic stable for the
application is ready. This is also true in order to test whether the application UI is User Loyalty
user-friendly or not, or whether some part/functionality of the application is
easily discoverable or not. Also, usability studies should be done for the different “While most will tolerate a minor
types of devices, such as touch screen devices, since the user has different glitch, studies show that many
experience for the same application on these devices. won’t hang around if they are
disappointed with the
A user may have a good experience on the iPhone, while another might have bad
performance of a mobile app or
experience on a non–touch screen device. In this instance, changes to the UI
should be considered to improve the experience on the non-touch screen device. website. And the damage doesn’t
stop there. Word-of-mouth will
For example, let’s say there is a form with various fields and it is saved by clicking ensure any bad experiences are
on the save button located at bottom of the screen. For this scenario, someone soon broadcast.”
using iPhone will have no issues to scroll to the bottom to click on the save
button, since the iPhone has a good scrolling mechanism. However, the same - Helen Leggatt, BizReport.com
scenario will give the bad user experience for the non-touch devices users, where
scrolling to the bottom of the page is not as easy.

Needless to say, usability testing is an extremely critical step towards launching a


high-quality mobile application.

Testing Types: Usability Testing Page 18


Mobile Testing Types: Performance Testing
Performance testing is another critical step in the successful launch of a mobile
application. The key factor to be aware of here is mobile carriers, which can Mobile Testing Tip
effect the speed and ease-of-use of the application. If the application
performance is poor for the end user, they will be likely to divert to other apps If the application being developed will be
and services. The application must therefore be tested with different devices dynamic or interactive, then the browser
and carriers depending on the county and region. must support the following:

Most likely, you didn’t build your application for automated users. You built it - JavaScript support: Mobile browsers must
for real users. Why then do so many companies rely solely on automated support JavaScript in order build an
testing results to verify functionality under stress? Until recently, it was interactive application UI.
because there was no such mechanism by which a company could utilize real
testers while a system was under load. - XML HTTP Request Object: XML HTTP
request object support is required to
But as we’ll outline shortly, that obstacle has largely been overcome. By communicate with the back-end server and
leveraging real testers with automated tools, companies now have access to an to update the page with new data without
entirely new layer of data. reloading the page. This will give the user a
smooth browsing experience for the site.
Better yet, they are able to identify and resolve issues that would have been
reported later in the development process. - CSS support: CSS defines how page
elements are to be displayed and enables
you to change appearance and layout of all
of the pages by editing a single file.

Testing Types: Performance Testing Page 19


Mobile Testing Methods: In-House Testing
Pros: In an ideal world, mobile testing could be performed under the watchful
eye of product managers, developers and QA managers, who could observe
firsthand how the application performs under every possible user scenario. The Just how important is global
assembled testing team – consisting of perhaps a dozen or so of the most
talented testers on the market – would have hundreds of handset models at their testing coverage?
disposal: iPhone, Blackberry, Nokia, Samsung, Sony - you name it, they test it. It
would be a testing lab that even the largest software company would envy. “By the end of this year, the mobile
ad market is expected to grow to
With this type of control over the testing process, what could possibly go wrong?
$913.5 million. And by 2013,
Cons: The truth is, for mobile apps, building a comprehensive in-house testing lab Gartner expects mobile ad spending
is prohibitively time-consuming and expensive. This is true for almost every to surpass $13 billion, with the
mobile app company, with very few exceptions. Asia-Pacific region bringing in the
Imagine the expense of building an in-house team and lab capable of assuring the most revenue, followed by North
functionality for iPhone, Blackberry and Android handsets (of all makes and America and Europe.”
models) across wireless carriers in the U.S., U.K, Spain and France. Now imagine
this with 3 apps… 5 apps… Will you send testers around the globe to assure - Meghan Keane,
quality? Of course not. Thus, it’s no surprise that mobile app companies – from www.econsultancy.com
start-ups to large enterprises – don’t rely solely on in-house testing resources.

To be fair, most firms don’t intend to create an in-house team like the one
described above (although some have tried). Worse, they spend their money
trying to get whatever they can from this method. That is, they have 3-5 in-house
testers, with a dozen devices (at most). But their testing coverage is confined to
their own office, and the whole point of comprehensive testing is lost.

Testing Methods: In-House Teams Page 20


Mobile Testing Methods: Outsourcing
Pros: Of course, when mobile application developers discovered that the in-
house route couldn’t provide the testing they needed, they logically said to
themselves: “I’ll just outsource it.” This was not an isolated decision; it was Common User Complaints:
made by thousands of software companies, and few could fault them for their
logic at the time. If comprehensive mobile testing was to be achieved, then •Failed or stalled downloads
surely an offshore firm (with their low-cost labor) would be the only entity •Credit card verification delays
capable of getting the job done at a reasonable price. •No screenshots
•App performance when there is
It’s this line of thinking that continues to define the way in which many mobile
slow connectivity
companies deal with their testing objectives.
•Inconsistent buttons, fonts, etc.
Cons: While outsourcing has certainly proven to be less expensive than hiring •Missing or broken links
an internal team, developers who contracted with such firms quickly ran into •App not supported by device
the same limits as that of their in-house teams – while losing what little control •Outdated versions; bugs still in app
they had over the QA process. And for what? With no insight into the day-to-
day testing activities, how could these developers expect to improve their
application in a timely manner? Developers must remain close to their
application at all times, which is antithetical to the outsourcing mentality.

And as it turns out, even the largest of traditional outsourcing firms don’t have
the resources needed to provide adequate testing coverage required by today’s
mobile environment. Just like their clients, they cannot easily (or affordably)
test an app across dozens of handset makers, models, carriers, languages and
locations. Plus, the testers employed by these offshore firms are unlikely to be
among the target audience for a company’s mobile apps.

Testing Methods: Outsourcing Page 21


Mobile Testing Methods: Simulators/Emulators
Pros: In many ways, the use of mobile emulators and simulators is essential for
developing and testing a quality application. Such technology enables Mobile App Purgatory
developers to verify certain functionality that is not specific to any device,
carrier or operating system. They are also very useful in terms of usability, and Bugs in your mobile app cause
especially design, including data input, screen size, button use, etc. – all from
unintended consequences. Take
the convenience of their own laptop.
this story for example: A mobile
Although many mobile app companies rely on these tools to indentify bugs at app developer is alerted to a
an early stage in development , they often fail to catch bugs at later stages. As iPhone bug, and instead of being
we’ve seen, these type of bugs are always the most expensive to correct. able to fix it right away, he spends
weeks pleading with Apple to
Cons: One of the biggest challenges for mobile developers is that the testing of allow him to proceed with the
applications is occurring in an environment far removed from the real world,
where actual users run and interact with those applications ON their devices.
update. If bugs in your app
Said differently, the gap between “in-the-lab” simulation and “in-the-wild” appear, getting them fixed quickly
usage is far too great to ignore. can be very problematic.

Consider, for instance, an application that is data entry intensive. To assume Of course, if the bug had been
that it can be fully tested on a simulator (i.e. with full keyboard and mouse discovered BEFORE launch, this
access) and be ready for release is a mistake mobile app developers have
would be a non-issue.
continually made for the past five years. The convenience of simulators and
emulators has made it easier than ever to be lured into a false sense of
security. But the advantages of such tools are limited in scope, and should
never be considered a substitute for real-world, on-device testing.

Testing Methods: Simulators and Emulators Page 22


Mobile Testing Methods: Beta Communities
Pros: Mobile companies that can attract a large beta community have a terrific
advantage over their peers, as thousands of users submit bugs and feedback
around the clock…for free.
The Future of Mobile
Needless to say, the allure of an active, engaged group of beta users is too “As mobile matures, more
great for many developers to pass up. Not only do they gauge your application’s
development tools and standards
performance before the big release, they act as advocates for your product,
adopting a strong sense of ownership and helping you to establish a customer will emerge. And in time, testing
base for your product. And so if you’re at all uncomfortable with your app’s tools and bulletproof practices will
performance prior to launch - and don’t want to go to great lengths running evolve. Until then, building and
professional usability tests – beta programs seem like the easiest, safest route. testing mobile apps for the iPhone,
Blackberry, Android, Palm Pre and
Cons: In practice, it is rare for a software company to attract such a large group
others will continue to feel a bit like
of beta testers. After all, not every company can be Google or Mozilla, with its
various (and wildly popular) beta versions, where people line up to test drive the wild west.”
their software. Chances are, your app either lacks enough mass appeal, or is
-Matt Johnston
relatively unknown (for now), and is hence not that attractive to the beta
testing types. VP of Marketing, uTest

But even if you are able to assemble a sizeable beta group, the method still falls
short of providing adequate testing coverage. And the reason for this is simple:
beta testers are more often like users, in that they will try to get your app to
function properly, whereas a real tester would attempt to “break” your app.

In other words: Users use. Testers test.

Testing Methods: Beta Communities Page 23


KEY TESTING CHALLENGES

Page 24
The mobile app testing matrix is infinitely complex – and getting worse

Features

Location
Wireless Carriers
Device Models
OS & Browser

Mobile App Challenges Page 25


The Challenges of Mobile Application Testing
The true cost of bugs: According to a study highlighted on TechCrunch, the average shelf life of an iPhone app is roughly 30
days. It turns out that, for free apps, less than 20% of users return to an app even one day after downloading it. And by day 30,
less than 5% of users are still utilizing the app. And for paid apps, the drop-off is even slightly steeper.

Why the drop-off? For starters: bugs - especially ones that appear within minutes of downloading - are likely to cause users to
abandon the application after a short period of use. With the variety of apps now available, users have an extremely low
tolerance for buggy applications. In a never-ending attempt to appease their user base, developers must therefore test their
products across:

 Handset Maker/Model (Relative cost of defect, by time of discovery)


 Operating System 100
 Browser 80

 Wireless Carrier 60

 Location 40
20
 Language
0
Specification Design Coding Unit Test Integration Release Test Post-Release
Test

No matter what type of mobile applications you develop – whether they’re chat tools, social networking, games, business apps
or others – you face the same fundamental problems encountered by all mobile app developers.

Let’s now take a quick look at how mobile app developers have attempted to deal with these problems…

Mobile Testing Challenges Page 26


CROWDSOURCED TESTING FOR MOBILE

Page 27
Crowdsourced Mobile Testing
Combining the benefits of the aforementioned methods, the use of
community-based testing – or crowdsourcing – has altered the way companies
test their mobile applications. This shift has empowered progressive mobile What is Crowdsourcing?
app companies to make the best decisions, on a per-project basis, without
sacrificing control, costs or accountability. By leveraging a global community of "Crowdsourcing is the act of
professional testers, QA teams are doing the impossible: maintaining app taking a job traditionally
quality, achieving broad testing coverage, meeting launch dates and staying performed by a designated agent
within tighter budgets.
(usually an employee) and
The net effect is that mobile app companies get instant access to a global
outsourcing it to a large group of
community of testers via a platform that enables detailed management of the people in the form of an open call.”
entire test cycle (or even connects to their in-house bug tracking system).
- Jeff Howe
This community can serve as a virtual testing team (for small companies and Author of Crowdsourcing
start-ups), or as an on-demand extension for in-house QA departments.

While the thought of having a virtual QA team from around the globe might
seem like a complicated free-for-all, the reality is just the opposite. With the
right online platform, the process fits neatly into your existing teams and
processes. Here’s a quick look at how it works:

Crowdsourced Mobile Testing Page 28


Crowdsourced Testing: How It Works

Customer Tester Community


Specify Testing Needs Selected testers invited
- Handset Maker, Model & OS
- Carrier, Location & Language to project

Testers report in
Customer approves
real-time

Testers get paid

Crowdsourced Testing: How it Works Page 29


Crowdsourcing Challenges: Quality and Control
Mobile Challenges
Crowdsourcing has a reputation for being noisy and chaotic, without much actual
productive work. Thus’ it’s critical to choose a reputable community that can “Mobile applications come with
point to its customers’ successes, and shares the past performance and their own unique set of challenges.
reputation of community members. In short, make sure you’re choosing a Between Wi-Fi, 3G, Edge and all
community and not a mob. Other common challenges include: their different behaviors, there are
Communication: As with any successful project, communication is key. While so many angles you have to cover.
crowdsourcing doesn't enable face-to-face conversations, it does enable decision- Until recently, testing for this
makers to converse with their virtual QA team in real-time through an easy-to-use criteria would have been
online platform. Also, make sure the testing company you choose provides a impossible, but crowdsourcing has
dedicated project manager to help you manage the community and the process. changed all that.”
Intellectual Property: IP protection is sometimes a concern for customers who
are new to crowdsourcing. While it might seem like your IP could be exposed to a -Travis McElfresh,
community of thousands, top crowdsourcing companies allow customers to -VP of Technology, 1Cast
carefully select their testers and establish non-disclosure agreements (NDA) with
their community members.

Quality: With crowdsourcing, it’s easy to believe that anyone can test your
application. Fortunately, the crowdsourcing model enables customers to be as
selective as they want – in terms of testers’ skill sets, experience, performance
ratings – in determining who gets invited to test their app.

As with other methods, unexpected problems do arise, but what makes


crowdsourcing different is its inherent ability to adapt to changes on-the-fly.

Crowdsourcing Challenges Page 30


Benefits of Crowdsourced Mobile App Testing
Although the advantages of such an approach are numerous, here’s a quick look
at what to expect when implementing crowdsourced QA for your mobile app: Why It Works
Cost Control: Crowdsourcing allows managers to utilize lower-cost outside “Because crowdsourcing is built
support without being tied down by long-term commitments. The fundamental from the open source template,
structure of online communities creates competition, making it more any company looking to leverage
accountable, efficient and cost-effective than traditional outsourcing. computer professionals has the
Dealing with Increased Complexity: Significant quality improvement is achieved advantage of an audience already
when development and testing is done across a wider set of handsets, carriers, familiar with the basic idea of
locations and languages. Crowdsourced mobile app testers can be easily recruited community production.”
based on a wide variety of criteria.
-Jeff Howe
Faster Time-to-Market: Crowdsourcing enables you to ramp up or down to meet
your needs at any given time - eliminating the delays and staffing constraints that Journalist, Wired.com
often stifle mobile app companies around peak release times.

Higher Quality: Customers expect their mobile apps to be fully functional and
bug-free from day one. Companies that believe “good enough” is good enough
when it comes to quality and usability will learn a valuable lesson (the hard way).

Retaining Users: If your product has problems, or doesn’t meet expectations, it’s
easier than ever for your customers (and prospective customers) to find your
competition.

Benefits of Crowdsourcing Page 31


Benefits of “In the Wild” Testing
Mobile Testing Tip
‘Hands On’ Device Interaction
“Even on the same mobile
On a simulator, you still use a mouse to ‘touch’ the screen and simulate gestures. platform, screen sizes and
You also have a full-sized keyboard for data entry. Of course, this is very different
resolutions can vary based on
from using a mobile device.
device type. For instance, the
First, a mobile device sits in your hand. Each of us likely has slightly different ways screen size and resolution on the
of holding and operating the device. For some, it’s done with one-hand using your
thumb or a finger. For others, it might be two hands using both thumbs.
HTC Incredible is different than
that on the HTC EVO 4G.
Second, there’s the act of touching various screen elements like buttons and
Consequently, for an application to
controls. This is much easier to do with a mouse pointer than a pudgy finger.
have a consistent look and feel
Based on the prior experience of many mobile testers, this difference is a critical across both devices and across a
one, and the biggest, for testing application design and function. Using a mouse
with the simulator, you do not get the full effect of having to scroll through a large
variety of other devices, user
list view of items or having to play ‘whack-a-mole’ on the screen with your thumb interface elements and graphics
because button placement for navigating multiple screens is inconsistent. need to be scalable.”
Mobile developers are strongly encouraged to ensure that application testing -Web Trends, Mobile Analytics
begins early, and happens often, on the mobile device itself rather than on a
simulator. The same holds true for tablet devices.

In-the-Wild Testing Page 32


Benefits of “In the Wild” Testing Mobile Testing Tips
Screen Size and Viewing: Interrupts: Test how the device
Even though the simulator application mimics the size of the iPhone screen, it’s and application behaves when it is
still a different experience viewing an application on the device rather than a interrupted by another action, e.g.
monitor. A screen that is readable on a simulator may not be – and usually isn’t – incoming/outgoing call, SMS,
as readable on the device. MMS, video-call, low battery,
Data Entry: low/no connectivity.
No contest here. It would be a terrible mistake to develop an application that’s
“data entry intensive” and test it on a simulator, with full keyboard and mouse
Device Settings: Change the
access, and believe that it’s ready to be released. Trying to type even a sentence is device settings: Date and Time,
a challenge on a touch screen keyboard. Languages (Advanced regional
Functionality: settings), Connections, Shortcuts –
anything you have control over –
In the world of mobile, there are some things you just can’t test on the simulator.,
including the use of accelerometers and location functions to name a few.
and learn how each and combined
settings impact the device.
Today’s mobile devices are pushing the boundaries of traditional application
design and testing methodologies. While fundamentally still the same, those - Bernard Lelchuk
methodologies need to be applied in new ways, especially with regard to testing
application usability.

Screen Size, Data Entry and Functionality Page 33


Conclusion
Testing no longer has to be a neglected phase of mobile application Crowdsourced Capacity
development. There was a time when the testing matrix was indeed too
complex, too burdensome to be completed using standard means. But thanks “Small development firms have
to the rapid evolution of crowdsourced testing, all that has now changed. limited means today to beta test
and peer review their applications
The companies that are now leveraging crowdsourced testing - particularly in with a cross section of
the mobile space - are gaining more of a competitive advantage than they representative users.
perhaps even realize. As the mobile market doubles and triples in size over the
next decade, those that have made testing coverage a priority will enjoy their
ROI in terms of increased market share, profitability and above all, user loyalty. Given the hundreds of thousands
Those who continue to neglect the testing phase will struggle at best, and at of mobile apps, we believe that
worst, cease to exist. It’s that simple. efficient (crowd-sourced) testing
of apps in a global market of users
Either way, the future of mobile applications remains bright. Once reserved for is considerably under-utilized.”
a tech-savvy niche in the business world, the use of mobile applications is now
firmly entrenched in the mainstream. There’s no turning back when it comes to
mobile applications, and that includes methods for testing them. - VisionMobile

For more on how crowdsourcing can complete your mobile testing efforts,
chat with one of our QA gurus by clicking here or by calling 800.445.3914.

Summary Page 34
APPENDIX: MOBILE PLAYERS

Page 35
Mobile Operating Systems
iPhone OS from Apple

The iPhone uses its well known operating system, the iPhone OS, which is
derived from Mac OS X. The iPhone OS has four abstraction layers: the Core OS
layer, the Core Services layer, the Media layer, and the Cocoa Touch layer. The
operating system uses roughly 500 megabytes of the device’s storage. The
iPhone app store is the online market for downloading these applications. Third
party applications were not officially supported until the release of iPhone OS 2.0
on July 11th 2008. Presently, XCode 3.1 is the development environment and the
applications are written in Objective C (a variant of language C). There is no
doubt that the iOs will continue to be among the preferred mobile operating
systems worldwide.

Android from Google

Developed by Google, Android is an open source, Linux-derived platform backed


by Google, along with major hardware and software developers (such as Intel,
HTC, ARM, and eBay, to name a few). Together, they form the Open Handset
Alliance. This OS, though very new, already has a cult following among
programmers eager to develop apps for its flexible, Open Source, back end.
Android promises to give developers access to every aspect of the phone’s
operation. This lends many to foresee the promise of further growth for the
Android platform. Like the iOS, Android figures to remain a prominent figure in
the future of mobile application development.

Mobile Operating Systems: Apple and Android Page 36


Mobile Operating Systems
RIM Blackberry OS

The BlackBerry OS is a proprietary software platform, created by Research In


Motion, for their BlackBerry phones. The OS is focused on easy operation and
was originally designed for business professionals. The Blackberry operating
system provides multi-tasking and supports specialized input devices that have
been adopted by RIM for use in its handhelds, particularly the trackwheel,
trackball, and most recently, the trackpad and touchscreen.

The BlackBerry platform is perhaps best known for its native support for
corporate email, through MIDP 1.0 and, more recently, a subset of MIDP 2.0,
which and allows complete wireless activation and synchronization with
Microsoft Exchange, Lotus Domino, or Novell GroupWise email, calendar, tasks,
notes, and contacts, when used in conjunction with BlackBerry Enterprise Server.
The operating system also supports WAP 1.2. Recently it has seen a surge in
third-party applications and has been improved to offer full multimedia support.

Linux Operating System

The Linux operating system is strongest in China, where it is used by Motorola,


and in Japan, where it is used by DoCoMo. Rather than being a platform in its
own right, Linux is used as a basis for a number of different platforms developed
by several vendors, including Android, LiMo, Maemo, Openmoko and Qt
Extended, which are mostly incompatible. PalmSource (now Access) is moving
towards an interface running on Linux. Another platform based on Linux is being
developed by Motorola, NEC, NTT DoCoMo, Panasonic, Samsung, and Vodafone.

Mobile Operating Systems: RIM and Linux Page 37


Mobile Operating Systems
Palm webOS

Palm webOS is Palm’s next generation operating system. PalmSource traditionally


used its own platform developed by Palm Inc. Underneath the graphical user
interface sits a software stack that has much in common with mainstream Linux
distributions for desktop computers. webOS uses, among other components, the
Linux kernel, Upstart, GStreamer as multimedia framework, libpurple for instant
messaging, Simple DirectMedia Layer (SDL) for game programming, PulseAudio
for audio management, and WebKit Applications for webOS are written in
JavaScript, HTML, AJAX, and some other common ‘web’ technologies. It supports
streaming video in RTSP, H.263, and H.264 formats.

Bada from Samsung

The Bada operating system has a kernel configurable architecture, which allows
the use of either the Linux kernel, or another Real-time operating system (RTOS)
kernel. The Linux kernel is the preferred choice for smartphones, while RTOS is
used for cheaper feature-phones, due to its smaller memory footprint. It has a
web browser based on the open-source WebKit, and also features Adobe Flash.
Both WebKit and Flash can be embedded inside native Bada applications. Bada
offers interactive mapping with Point of interest (POI) features, which can also be
embedded inside native applications.

Bada supports various sensors, such as motion sensing, vibration control, face
detection, accelerometer, magnetometer, tilt, and GPS, which can be
incorporated into applications.

Mobile Operating Systems: Palm and Bada Page 38


Mobile Operating Systems
Windows Mobile from Microsoft

Windows mobile is based on the Windows CE 5.2 kernel, and features a suite of
basic applications developed using the Microsoft Windows API. It is designed to
be somewhat similar to desktop versions of Windows, feature-wise and
aesthetically. The current version is called “Windows Mobile 6.5″.

Third-party software development is available for this OS, with several options
for developers to use when deploying a mobile application. This includes writing
native code with Visual C++, writing managed code that works with the .NET
Compact Framework, or Server-side code that can be deployed using Internet
Explorer Mobile or a mobile client on the user’s device. The .NET Compact
Framework is actually a subset of the .NET Framework and hence shares many
components with software development on desktop clients, application servers,
and web servers which have the .NET Framework installed, thus integrating
networked computing space called the cloud.

The two improved variants of this operating system, Windows Mobile 6


Professional (for touch screen devices) and Windows Mobile 6 Standard, were
unveiled in February 2007. It has been criticized for having a user interface which
is not optimized for touch input by fingers; instead, it is more usable with a
stylus. However, unlike iPhone OS, it does support both touch screen and
physical keyboard configurations. On February 15th, 2010 Microsoft unveiled its
next-generation mobile OS, Windows Phone 7. The new mobile OS includes a
completely new over-hauled UI called “Metro”. It includes full integration of
Microsoft services such as Zune, Xbox Live and Bing.

Mobile Operating Systems: Windows Mobile Page 39


Mobile Development Platforms
BREW (Binary Runtime Environment for Wireless) from Qualcomm

Brew is an application development platform that was originally developed for


CDMA mobile phones, but it also supports GSM for some devices. It debuted in
September 2001 as a software platform. It can download and run small programs
for playing games, sending messages, sharing photos, and the like. The main
advantage of BREW platforms is that the application developers can easily port
their applications between all Qualcomm devices. BREW acts between the
application and the wireless device on-chip operating system in order to allow
programmers to develop applications without needing to code for system
interface or understand wireless applications.

J2me by Sun

Java 2 Micro Edition (J2ME) is Sun’s version of Java aimed at machines with
limited hardware resources such as PDAs, cell phones, and other consumer
electronic and embedded devices. J2ME is aimed at machines with as little as
128KB of RAM and with processors a lot less powerful than those used on typical
desktop and server machines. J2ME actually consists of a set of profiles. Each
profile is defined for a particular type of device — cell phones, PDAs, microwave
ovens, etc. — and consists of a minimum set of class libraries required for the
particular type of device and a specification of a Java virtual machine required to
support the device.

There are now more than 2 billion Java ME enabled mobile phones and PDAs.

Mobile Development Platforms: BREW and J2ME Page 40


Mobile Development Platforms
LiMo

The LiMo Platform is developed by the LiMo Foundation (founded by a group of cellular
handset makers and network operators) for handheld devices, and it uses Linux as its
operating system. It has a modular plug-in architecture, and supports DRM. LiMo
application developers will be able to use SDKs to write managed code running in a Java
virtual machine, browser apps for WebKit, and native code.

Qtopia

Qtopia is a platform for embedded Linux devices that provides the end-user with a set of
core applications, such as PIM applications (Address Book, Calendar and Todo) and a
Media Player. Qtopia provides the developer with a clean object-oriented API to write
applications for an embedded device. Most of the API presented to the developer is the
same API offered by Trolltech for cross-platform development between Windows, Mac
and the UNIX/Linux. The same code can be shared and reused on all of these desktop
platforms and now Qtopia is just another platform to run Qt applications.

The Qtopia platform encompasses both the software that runs on an embedded device
and the software that runs on an end-user desktop. The software that runs on the
desktop, called Qtopia Desktop, allows the user to synchronize their data between the
embedded device and the desktop. Qtopia Desktop has built-in versions of the PIM
application found in Qtopia. Qtopia Desktop allows developers to write plug-ins to
synchronize their information.

Mobile Development Platforms: LiMo and Qtopia Page 41


Acknowledgements: Special thanks to Bernard Lelchuk, Brad Sellick, Jigar Patel and Anuj
Gupta for their contributions to this eBook.
Who’s Crowdsourcing?
About uTest The number of mobile app companies using
crowdsourced testing is growing everyday.
Headquartered near Boston, uTest is the world's largest marketplace for software testing
Here’s a few companies ahead of the curve:
services. The company provides real-world QA services through its community of 30,000+
professional testers from over 165 countries around the world. Thousands of companies -
from web start-ups to enterprise software firms - have signed up to get their apps tested by
the uTest community.

uTest enables companies to launch higher quality products; get their desktop, web and
mobile applications to market faster; and control the cost of testing. Customers specify
their testing requirements for location, language, OS, browser and more – and uTest selects
the testers who best match each project. And because uTest is on-demand, companies pay
for completed test cycles.

A brief online introduction is available at www.utest.com/demo. uTest can be contacted at:

uTest, Inc.
153 Cordaville Road
Southborough, MA 01772
p: 1.800.445.3914
e: info@utest.com
w: www.utest.com

About uTest Page 42