Mobile Handset and Mobile App Testing

Session for Test Management CoP

Date 22 Sep 2011

Disclaimer
The contents of this deck and the session conducted for TMCoP leverages iGATE Patnis past experiences in this space Research references from Industry Research firms Individual opinion and research conducted on this subject over a period of time This is not a snap-shot of iGATE Patnis capability in Mobile Testing For any comments and queries regarding this presentation please contact kapil.chourasiya@igatepatni.com

Proprietary and Confidential September 27, 2011

-2-

Mobile Telephony Trends

More than 60 % of worlds population today has access to a Mobile Phone! Ratio of Feature phone to SmartPhone users was 80:20 ( Q4 2009 ) Projected to become 70:30 by Q4 2011 world-wide
- The Nielsen Company

Worldwide 2011 Q2 Smartphone sales by Operating System - Gartner Market Share Handset Makers ( mn units 2010) : Strategy Analytics 2011

Others, 413.8

Nokia, 453

Apple, 47.5 RIM, 48.8 LG, 116.7

Proprietary and Confidential September 27, 2011
-3-

Samsung, 280.2

Mobility Landscape
Enterprise Systems

Legacy

HIS Supply Chain Financial

Applications

SFA/FFA

Distribution/Retail

Management Dashboard

Patient Monitoring

Credit Card/Claims

Middleware/ Platforms

Sybase Afaria/ Unwired

IBM DB2 Everyplace

Rhomobile

Broadbeam/ Mobileaware

Blackberry Enterprise Server

Sierra/ Motorola Jasper/ Telenor

LAN, PSTN, DSL, GSM, GPRS, CDMA, CDPD, UMTS, WLL, WiFi, Satellite

OS/ Platforms

Mobile Devices

PDA

Smartphones

Mobile phones

Rugged Terminal

POS/ Tablets

Laptop

RS232, USB , Bluetooth, Zigbee, UWB, HomeRF, IrDA

Peripherals

RFID reader

Barcode Scanner

Camera

Printer

Sensors

GPS Receiver

Reference iGP Mobility Deck

Proprietary and Confidential September 27, 2011

-4-

Mobility Landscape and Challenges

Enterprise Systems

Applications

Mobility Challenges Real-time, Off-line, Near-real-time Security, Authentication, Authorization Technology Convergence Provisioning and device management Scalability Services Messaging, location based, notification

Complexity of
Middleware

Applications Range of devices Connectivity options Technology options

OS

Vendors and Platforms

Application Challenges Commercial grade apps Content Aggregation Customizations and personalization Multi-device support Multi-lingual support Application categories

Mobile Devices

Peripherals

Device Challenges Device capabilities Programming models No dominant standard Wide range of target environments

Reference iGP Mobility Deck

Proprietary and Confidential September 27, 2011
-5-

Mobile Testing Landscape

Mobile Handset Testing / Device Qualification Mobile Handset App / Mobile WAP Testing

Handset / Mobile Device

Test Handset from OEM Test Branded PreReleased Handsets supplied by Telecom Operators

Native Applications
Test Apps that reside on the device and may or may not require network connectivity

Web Based Applications

Test Applications that are accessible through mobile browsers

Third Party Applications

Test Third party Apps that are specifically installed on the mobile handset

Examples New / Upgrade of Handset from OEM directly Co-branded Handset with Telecom Operator

Examples Messaging, Multimedia, Games, PIM etc

Examples Mobile Banking Payment solutions

Examples Facebook, Twitter

Proprietary and Confidential September 27, 2011

-6-

Mobile Handset Testing

User Interface Branding Mobile Phone Controls Mobile Handset Applications Partner Applications Multimedia and Graphics Integration with other systems

Types of Testing
Functional Regression Integration Automation Multi-lingual Usability Field / Coverage Quality of Service Performance Testing Security Testing

Focus Areas

The customer in this case could be a Telecom Operator or an OEM for e.g. Nokia
Please note Types of testing highlighted in Italics are not offered as a service in this space by iGATEPatni today
Proprietary and Confidential September 27, 2011
-7-

Mobile Handset Testing Focus Areas

1. User Interface and Branding : Verifying menus, navigations and the soft keys etc. are as per the specifications given in the UI specification document. 2. Mobile phone controls : Verify that all the controls or buttons on the handset are mapped correctly for instance pressing a dedicated camera key should launch the camera application only and not any other application. 3. Mobile Handset application : Verifying the functional correctness of all the applications on the handset for e.g. verifying that the alarm on the handset can be set and that it goes off at the given time. 4. Partner applications : This test is to verify whether all the 3rd party applications present on the handset work properly an example of this could be verifying that the 3rd party application like MSN messenger is launching properly and is working as needed. 5. Multimedia and graphics : This test is to verify whether the graphics on the handset i.e. wallpapers , screen savers, themes (both preloaded and downloaded) appear properly. 6. Integration with other systems : if required

Proprietary and Confidential September 27, 2011

-8-

Network and Non-Network Testing

Mobile Handset Testing

Requires Network DATA

WAP | SMS/MMS | DOWNLOAD CONTENTS DOWNLOAD GAMES WAP deck and 3rd party apps INSTANT MESSAGING (like yahoo,msn.google) EMAIL(3rd part email apps) MAKE/RECEIVE CALLS | VOICE MAIL | SPECIAL NUMBERS RECENT CALL LISTS | INTER CARRIER SWITCH ALARMS AND SCHEDULER APPS. RECEIVING VARIOUS ALERTS (low balance, promo alerts etc.) BLUETOOTH CONTACTS

Doesnt Require Network

TOOLS (calculator,voice memo,stopwatch,etc.) SETTTINGS CAMERA PRE LOADED CONTENTS GAMES HELP & INFORMATION CONTENT MEMORY

CALLS

MEDIA PLAYER (audio and video) LANGUAGE UI (in cases of another language being present like Spanish)
-9-

Proprietary and Confidential September 27, 2011

Sample Handset Specification

Reference Collateral from past engagement

Proprietary and Confidential September 27, 2011

- 10 -

Mobile App Testing

Types of Testing
Functional Compatibility Automation Regression Performance Security Usability

Mobile App Testing

The customer in this case could be an Enterprise OR a content Provider or an App/WAP provider
Please note Types of testing highlighted in Italics are not offered as a service in this space by iGATEPatni today
Proprietary and Confidential September 27, 2011
- 11 -

Popular Testing Tools used in this space

Commercial Device Anywhere Perfecto Mobile Test Quest Squish Jamo Solutions Eggplant Open Source Robotium (Android Apps) FoneMonkey (iphone Apps) TestMaker T-Plan Robot

Please note This is only a sample list of tools. This by no means is a list of most recommended tools by iGATEPatni in this space
Proprietary and Confidential September 27, 2011
- 12 -

Feature Set comparison of Popular Tools

Parameters
Device and Platform Support

Device Anywhere
All Leading Handset OEM s Android, Blackberry, Windows, Symbian, iOS, Win 7, WebOS Drag drop functionality Ability to script in Java Yes Yes No Built-in Test Case Manager Yes HP QTP and QC IBM Rational Quality Subscription based SaaS* [ 12 mth commitment ]

TestQuest
Android, Blackberry, BREW, UTS, Windows No Scripting Yes Yes Yes Built-in Test Mgr Yes Not available Cost

Perfecto Mobile
Android, Blackberry, Nokia, iOS,

Eggplant
Android, iphone, Symbian, Windows Blackberry Yes Yes Yes No No Yes No Cost

Robotium
Android

Scripting Functional Testing Automation Testing Performance Testing Test Management Remote Connectivity Integration with industry std. tools Licensing Model

Yes Yes Yes No No Yes HP QTP and QC Subscription based SaaS*

Yes Yes Yes No No No No Opensource

* Each package is a collection of devices with some common characteristics such as network (e.g. Verizon Package), manufacturer (Samsung Package), operating system (Android package) etc. Reference Collateral from Individual Websites of respective toolset vendors | These are some of the key features ; however one needs to do a thorough POC to arrive at a recommendation
Proprietary and Confidential September 27, 2011
- 13 -

What is Remote Device Testing?

100 s of real devices / phones connected to live network are installed in their Data centre available for use based on a subscription package (SaaS)

Through the DeviceAnywhere Test Centre a tester can remotely connect to these devices / phones via an https web based connection in real time and can remotely perform any action..

Some key benefits of device testing tools

No need not buy all the devices/phones ( including SIM card ) Test multiple combinations of devices / operating systems Leverage feature of the Testing tool to automate and test manage the whole testing process Conduct this testing from a remote location / offshore thus making it further attractive

Proprietary and Confidential September 27, 2011

- 14 -

Approach for Testing

Test Planning Test Design Test Execution Test Closure

Analyze Requirements

Design Test Cases

Functional, Automated

Document Closure Report

Document Test Strategy

Setup Test Environment

NonFunctional

Present Findings

Tool Selection and POC *

Create Test Data

Defect Management

Go Live!

using automated tools OR Emulators using actual devices

Please note this is a technical opinion of the author.
Proprietary and Confidential September 27, 2011
- 15 -

Key considerations for Drafting a Strategy

1. 2. 3. Clearly establish the scope of Testing Functional, Automation etc Agree the number and types of devices that need to be tested for example Blackberry Bold, Blackberry Storm, iPhone etc . Do not leave any loose ends for e.g. Test this on Blackberry which models?? Identify the testing toolset of choice based on the following key criteria 4. 5. 6. 7. 8. 9. Budget for Testing Number of devices / operating systems / browsers ( including versions of each ) to be tested Duration of the programme Results of Technical Proof of Concept

If certifying a common functionality / WAP / APP across devices account for Regression testing of defects across the devices/operating systems/browsers , else you will run out of time during re-tests! Testing report should highlight the features that are working / not working across devices Conduct automation post a thorough ROI calculation only Some tools do not integrate with traditional Test Management and OR defect management tools therefore account for the overhead ( effort and cost ) of managing test cases / defects in a different toolset Defect Management Prioritize defects upfront with the Dev/Vendors and Business In third party testing scenario ( OEM ) account for additional rounds of discussions for defect triage

Please note this is a technical opinion of the author.

Proprietary and Confidential September 27, 2011
- 16 -

Case Studies

Proprietary and Confidential September 27, 2011

- 17 -

Virgin Mobile USA Handset Testing

First MVNO in US and UK Customer base ~ 5 M Pay as you go mobile services and handsets Uses Sprints network Target segment 15-30 yr old Partnered with iGATEPatni for QA and UAT of the IT stack Handset, IVR, WebPortal, CRM, DW and Billing platforms 5 year relationship 52 person strong team in onshore / off
Proprietary and Confidential September 27, 2011
- 18 -

Virgin Mobile USA Customers Internet Security VMU Web Portal Sales
Handset Plans Promos

Contact Center/ IVR Top Ups

Register CC Adhoc Top Up Paypal Top Up Auto Top Up 3rd

Handset Virgin Xtras

Ringtones Wallpapers Mobile Web Messaging Games Party Content

Activations
Port In/Out Change MIN/ESN Promos

Help/Self Care
Services Phones VirginXtras

Gateway CRM Distribution Center Billing Credit Card Transactions CDR Telcordia

Areas of Testing
Handset
Functionality Handset Testing User Interface Branding Mobile Phone controls Partner Applications Multimedia and Graphics

WAP
WAP Enabled Application Third Party Applications Third Party Content

Integration
Integration with Mediation Managers Billing system Customer Care system

Functional Testing | Regression Testing | End to End Testing | Compatibility Testing

Proprietary and Confidential September 27, 2011

- 19 -

Testing Process and Flow

VMU REQUIREMENTS

NOKIA

KYOCERA

UT STARCOMM HANDSET UNDERSTANDING DOCS TEST CASE PREP

UI SPEC & HANDSET HANDSET,UI SPEC ,HRD VMU

PATNI

FLASHING NEW BUILD UNSTABLE END CUTOMER STABLE BUG REPORTING TESTING

Proprietary and Confidential September 27, 2011

- 20 -

Handset Testing
WDIG
HANDSET PORTFOLIO

Offshore
GATEWAY P100 P205

Telcordia DATA Platform

LG200

SCH110

STIC LAB Onsite

Voice Contacts SMS MMS Email

Browsing Settings Calendar Alarm Camera functionalities Hands free Calling

Content Download Family Monitor Family Alerts Family Locator Photo Upload and Download
- 21 -

Proprietary and Confidential September 27, 2011

Proprietary and Confidential September 27, 2011

- 22 -