2013 13th International Conference on Quality Software

PCTF: An Integrated, Extensible Cloud Test

Framework for Testing Cloud Platforms and
Applications
Ganesh Neelakanta Iyer, Member, IEEE, IEEE-CS,
Jayakhanna Pasimuthu, and Ramesh Loganathan
Progress Software Development
iLabs Center, Madhapur
Hyderabad, 500081
Email: {gaiyer,jpasimut,rameshl}@progress.com
testing dimensions commonly applicable across all Cloud platforms/applications. In Section IV, we describe the proposed
Cloud test framework PCTF in detail and in Section V, we
conclude this paper.

AbstractDue to the inherent advantages of Cloud Computing

paradigms, application development, deployment and usage in
Cloud environments has been increased exponentially in the
recent past. This results in reduced time to market, reduced
capital and operational expenses and increased productivity. The
proliferation of Cloud platforms and applications also poses
several challenges for the quality assessment. In this paper, we
analyze various Cloud test approaches and frameworks. Then
we present our research findings on major Cloud dimensions for
testing Cloud platforms and applications. Then we propose an
integrated, extensible Cloud test framework for testing various
Cloud features, called Progress Cloud Test Framework (PCTF)
and describe its components and characteristics.

I.

II.

In this section, we briefly review various Cloud testing

methodologies and practices both from industry as well as
from research literature. Cloud testing can be broadly classified
into Testing the Cloud and Testing in the Cloud (also called as
Test as a Service). Former denotes testing various Cloud-based
infrastructure, platforms and applications whereas the later
term denote the usage of Cloud to develop test frameworks
to test traditional software testing. In this paper we focus on
Testing the Cloud, specifically to test Cloud-based platforms
and applications.

I NTRODUCTION

With the proliferation of Cloud Computing and Web 2.0

technologies, companies are increasingly adopting Cloud platforms (PaaS) for developing and deploying applications and/or
SaaS-based applications. Majority of the information technology companies are either developing and offering new Cloud
solutions or adopting Cloud-based solutions. These Cloudbased solutions have several advantages such as resource
availability on demand, faster time to market and reduced
capital and operational expenses.

In [1], the authors provide a high level overview of Cloud services testing and poses few research challenges in this regard.
In [2], the authors categorize different SaaS testing dimensions
and the challenges associated with it. They describe some high
level view on testing SaaS applications and categorize them in
a comprehensive manner.
There are several frameworks existing for testing some specific
aspects of Cloud testing. In [3], the author describes the framework used by Microsoft to test their Cloud platform Microsoft
Windows Azure. It describes the approach for testing both onpremise and azure run times. In [4], the author describes the
test mechanism for testing Google App Engines API Client
library.

These applications are also inter operable across multiple

platforms and browsers. Other characteristics include multitenancy and auto-elasticity. When such high-end Cloud-based
solutions are developed, it also poses several challenges to assess and quantify the quality of the products developed. These
applications create a unique set of challenges to be addressed
before they are offered for customers. These challenges include
security, high-availability, elasticity and multi-tenancy issues.

There are several research proposals related to Cloud testing

in general. In [5], the authors describe a taxonomy of performance evaluation for Cloud services. They categorize and describe various performance metrics from environmental, physical, capacity and operational point of views. In [6], the authors
propose a performance test framework for Cloud services
called CARE. The measured features in CARE framework
include response time, processing time and DB processing
time. Their test scenarios include Client to Cloud host, Cloudhost to Cloud database and Client to Cloud database.

In this paper, we present our research findings on major issues

to be assessed by any quality assessment team for any Cloudbased platforms/applications developed. We conduct a detailed
study of existing mechanisms and approaches used in practice
and propose a novel framework to test Cloud platforms and
applications.
Remainder of this paper is organized as follows: In Section II,
we describe the existing Cloud test frameworks proposed in
literature as well as those being used in practice for various
Cloud platforms. In Section III, we describe various Cloud
978-0-7695-5039-8/13 $26.00 2013 IEEE
DOI 10.1109/QSIC.2013.65

L ITERATURE R EVIEW

In [7], the authors propose a performance test framework for

135

Fig. 1.

Common Cloud Testing Dimensions

Fig. 2.

provisioning and scalability testing of Cloud platforms. This

framework can measure job wait time, job response time and
cost efficiency. A Hadoop-based distributed test framework is
proposed in [8]. It helps to execute test cases in a concurrent
manner in Cloud. The proposed test execution model is based
on map-reduce programming model and is suitable for testing
traditional applications as well as SaaS-based applications.

traversal vulnerability [14]. It means that one tenant (could be

an intruder) is able to traverse from one Virtual Machine (VM)
client environment to other client environments being managed
by the same hypervisor. This vulnerability might allow a
customer to access the virtual instances of other customers
applications. So adequate testing is required to make sure that
the platform is not vulnerable to such situations.

In [9], a mapping mechanism is proposed for SLA monitoring

and testing. An SQL penetration testing method for Cloud
applications is described in [10]. A model-based risk-driven
security test framework is proposed in [11]. In [12], the
authors propose a framework to test the scalability of SaaS
applications. But these proposals are specifically to test some
specific aspect of Cloud features.
III.

Elastic Load Testing: Load Patterns [13]

With multi-tenant environments, penetration testing [15] is

very important to simulate a malicious user and to test for
all vulnerabilities such as SQL injection, Cross-Site scripting
(XSS) etc. We need to test with a malicious user who has
valid credentials to the Platform under test as a tenant and that
users ability to penetrate the system and view the information
of other tenants. Other security areas to be tested include
checking for the capability of appropriate role-based access
control [17], identity federation management, and appropriate
data management.

C LOUD T ESTING D IMENSTIONS

In this section we briefly describe various Cloud platform

testing dimensions applicable in general.We categorize them
into five and it is illustrated in Figure 1.

C. Performance Testing

A. Elasticity and Scalability Testing

With the unique characteristics of Cloud environments, we

need to perform the accuracy of various data present in the
Cloud, latency and throughput. Further, elastic load testing
and multi-tenant performance testing are other key items to be
considered. Finally, high availability and failover testing are
required to test the behavior of the platform and applications
under resilience scenarios.

One of the major characteristics of Cloud is its support for

auto-elasticity. i.e. resource demand can be satisfied on-the-go
elastically. Hence vertical and horizontal scalability need to
be tested. Vertical scalabaility means that, replacing current
resource with a more powerful resource should satisfy the
increasing demand. On the other hand, horizontal scalability
means that adding more resources of same type should increase
the performance of the platform as per the requirements.

D. Live upgrade testing

Similarly, resources should be provisioned in a seamless

manner whenever the requirements arise. We should also test
if load balancing works properly with changes in demand.
Finally, elastic load testing need to be performed for various
possible user scenarios. In [13], the authors give various
scenarios for elastic load testing. Some elastic load testing
scenarios are illustrated in Figure 2.

This is closely related to understanding the performance

of the system when an upgrade of the software/platform happens and to understand its capability to continue its business
services to users even when the upgrade is going on. i.e. we
need to make sure that business continuity is there even when
software/hardware maintenance/upgrade is performed.

B. Security Testing

E. Multi-tenancy Testing

Cloud platforms and applications are exposed to several

security vulnerabilities [18]. One important security concern is

In addition to multi-tenant penetration testing and multitenant performance testing, we need to test several scenarios

136

Fig. 3.

Proposed Cloud Test Framework (PCTF)

under multi-tenant environments. Various levels of multitenancy are described in [24]. Rigid failure containment between tenants needs to be evaluated during the test phase.
According to [19], it means that failure of one tenant instance
does not cascade to other tenant instances, and that service
transition activities properly apply to individual application
instances rather than inadvertently impacting multiple tenant
application instances.

consists of some components for initiating and managing the

test execution, a data repository and different modules for
different types of testing.

A. Test Management Components

Test engineer can log into the test framework through a Test
Manager module which allows the user to configure various
parameters pertaining to test execution, mechanisms to select
different test suites and options to collect the results and logs
after the test execution. For the log collection, Test Logger is
attached which will be collecting all the test logs and creates
log reports for the test manager.

Multi-tenancy also introduces the risk of correlated or synchronized behaviors [20] that can stress the underlying virtualized
platform, such as when multiple application instances execute
the same recovery action or periodic maintenance actions
simultaneously. Finally, the service transition activity analysis
[19] should verify that no service transition activity impacts
active application instances that are not the explicit target of the
activity. In addition to traditional service transition activities,
the multi-tenancy analysis should also verify that there is no
service impact on other tenant instances when each and every
tenant-specific configuration parameter is changed.

Further, results obtained form the test framework is collected

by a Result Analyzer and is analyzed and stored in an output
repository. These results are available to the user through the
test manager whenever the user wants to access the results.

F. Other test dimensions to be considered

B. Security Testing

In addition to above test dimensions, we need to perform compatibility testing, interoperability testing, integration testing and web portal testing. We need to test the
platform/application across multiple operating systems and
multiple browsers. Further, platforms might provide different
APIs for users to build applications. In such cases we need to
test for API integration.
IV.

We use the SQL injection testing method proposed in [10]

for the SQL injection testing. This method consists of four
steps: Information gathering, Identification of input parameters, generating attacks and results reporting. The approach first
analyzes the Web application to understand its hyperlinks and
input forms. Then it seeds a series of standard SQL attacks in
order to gather error messages. This is matched with a library
of regular expressions related to error messages that databases
can produce. Further attacks are performed based on the mined
data to identify likely table of field names, until it is able to
retrieve the database structure. The framework can be extended
to support other security issues such as Cross-site scripting,
LDAP injection, file path traversal etc.

PCTF A RCHITECTURE

In this section, we describe the proposed Progress Cloud

Test Framework (PCTF) which can be used for testing various
Cloud testing dimensions described before. The framework is
illustrated in Figure 3. The proposed Cloud test framework

137

C. Integration testing

[5]

Zeng Li, Liam OBrien, Rainbow Cai and He Zhang, Towards a Taxonomy of Performance Evaluation of Commercial Cloud Services, 2012
IEEE Fifth International Conference on Cloud Computing
[6] Liang Zhao, Evaluating Cloud Platform Architecture with the CARE
Framework, Software Engineering Conference (APSEC), 2010 17th Asia
Pacific, pp: 60-69,
[7] Nezih Yigitbasi, Alexandru Iosup, Dick Epema and Simon Ostermann,
C-Meter: A Framework for Performance Analysis of Computing Clouds,
9th IEEE/ACM International Symposium on Cluster Computing and the
Grid, 2009
[8] Scott Tilley, T. P. (2012). Software Testing in the Cloud: Migration and
Execution. Springer.
[9] Vincent C. Emeakaroha, Ivona Brandic, Michael Maurer, Schahram
Dustdar: Low level Metrics to High level SLAs - LoM2HiS framework:
Bridging the gap between monitored metrics and SLA parameters in
cloud environments. HPCS 2010: 48-54
[10] Angelo Ciampa, Corrado Aaron Visaggio, and Massimiliano Di Penta.
2010. A heuristic-based approach for detecting SQL-injection vulnerabilities in web applications. In Proceedings of the 2010 ICSE Workshop
on Software Engineering for Secure Systems (SESS 10). ACM, New
York, NY, USA, 43-49.

Many Cloud platforms including Google App Engine [22]

support several APIs for developing custom applications and
accessing various features of the platform. In [21], the authors
propose a way to test the APIs supported by Google App
Engine and we integrate that model into PCTF. It consists of
a client shell and a set of plug-ins to check if different client
requests invokes appropriate APIs.
D. SLA performance testing
Service Level Agreements (SLA) are very important in the
context of Cloud Computing with the pay-as-you-go features.
In [9], the authors propose a comprehensive mechanism to
collect and map different raw SLA metrics to measurable
SLa parameters such as mapping uptime and downtime to
availability and data information to response time. We adopt
that mechanism for SLA monitoring in our framework.

[11]

E. Load testing
For various elastic load testing, we need to generate loads
following different realistic web traffic scenarios. Examples of
such patterns include [23] long-tail traffic, Poisson distribution
and Zipf distribution. The Synthetic Load Generator will
generate the required load based on the configured settings
and perform the load testing.

[12]

[13]

F. Test Repository

[14]

Test repository is a database which contains all the tests

suites, libraries and any other data pertaining to the test
execution. Libraries include SQL string library and error
patterns library for SQL injection security testing and other
library functions required for various tests. There will be
a Service Level Agreement (SLA) mapping table and SLA
metrics database to perform tests for the violation possibility
of all possible SLAs.
V.

[15]

[16]

[17]

C ONCLUSION

[18]

In this paper, we proposed an integrated Cloud test framework called PCTF which allows users to integrate different
independent test components into one framework. Further the
proposed model allows users to extend the model with other
test components as per their requirements without affecting
existing test components. It also allows users to select a
particular test suite or a set of test suites to run on the system
under test and collect consolidated results. We are currently
building this framework to test our Cloud platform and to
conduct various performance evaluation studies.

[19]
[20]

[21]

R EFERENCES
[1]

[2]
[3]

[4]

[22]

Atif Farid Mohammad, Hamid Mcheick, Cloud Services Testing: An

Understanding, Procedia Computer Science, Volume 5, 2011, Pages 513520, ISSN 1877-0509, 10.1016/j.procs.2011.07.066.
Vijayanathan N and Sreesankar S, Overcomign Challenges associated
with SaaS Testing, White paper, Infosys, 2012
Roger Jennings, Azure Storage Service Test Harness,
http://oakleafblog.blogspot.in/2008/11/azure-storage-services-testharness.html
Anthony Vallone, Testing Googles new Infrastructure, http :
//googletesting.blogspot.in/20120 80 1a rchive.html

[23]

[24]

138

Philipp Zech, Michael Felderer, and Ruth Breu. 2012. Towards a Model
Based Security Testing Approach of Cloud Computing Environments.
In Proceedings of the 2012 IEEE Sixth International Conference on
Software Security and Reliability Companion (SERE-C 12). IEEE
Computer Society, Washington, DC, USA, 47-56.
Wei-Tek Tsai, Yu Huang, and Qihong Shao. 2011. Testing the scalability
of SaaS applications. In Proceedings of the 2011 IEEE International
Conference on Service-Oriented Computing and Applications (SOCA
11). IEEE Computer Society, Washington, DC, USA, 1-4.
Ewald Roodenrijs, TMap NEXT Testing Clouds, Sogeti Netherlands,
2011
Dustin Owens. 2010. Securing Elasticity in the Cloud. Queue 8,
5, Pages 10 (May 2010), 7 pages. DOI=10.1145/1794514.1794516
http://doi.acm.org/10.1145/1794514.1794516
Takabi, H.,Security and Privacy Challenges in Cloud Computing Environments, IEEE Security and Privacy, Vol 8, Issue 6, Pages 24-31,
2010
Jayanthi V, Neha M and Nithin D, SaaS Security Testing; Guidelines
and Evaluation Framework, 11th Annual International Software Testing
Conference, 2011, Infosys
Sharma, S, Jena, S K and K, Satyababu, New Approach for Testing
the Correctness of Access Control Policies, IEEE International Advance
Computing Conference 2009, Thapar University, Patiala, Punjab, India
Neil MacDonald, Joseph Feiman, Magic Quadrant for Dynamic Application Security Testing, Gartner Report, 2011
Eric Bauer, Randee Adams, Reliability and Availability of Cloud
Computing, IEEE Press, 2012
Saman Zonouz, Amir Houmansadr, Robin Berthier, Nikita Borisov,
William Sanders, Secloud: A Cloud-based Comprehensive and
Lightweight Security Solution for Smartphones, Computers and
Security, Available online 20 February 2013, ISSN 0167-4048,
10.1016/j.cose.2013.02.002.
William Jenkins, Sergiy Vilkomir, Puneet Sharma, and George Pirocanac. 2011. Framework for testing cloud platforms and infrastructures. In Proceedings of the 2011 International Conference on
Cloud and Service Computing (CSC 11). IEEE Computer Society, Washington, DC, USA, 134-140. DOI=10.1109/CSC.2011.6138511
http://dx.doi.org/10.1109/CSC.2011.6138511
Dan Sanderson, Programming Google App Engine, OReilly Press,
2010
W. Gong, Y.Liu, V. Misra, and D. Towsley. On the Tails of Web Filesize Distributions. In Proceedings of the Thirty-Ninth Annual Allerton
Conference on Communication, Control, and Computing, pp. 192201.
UrbanaChampaign, IL: Coordinated Science Laboratory, University of
Illinois, 2001.
Yefim V Natis, Reference Architecture for Multitenancy: Enterprise
Computing in the Cloud, Gartner Report 2008