STRESS TESTING: PARAMETERS AND LIMITATIONS FOR PROJECT EVALUATION Basit Shahzad, Abdullah S.

Al-Mudimigh & Zahid Ullah Department of Computer Science, King Saud University, Riyadh Saudi Arabia basit.shahzad@gmail.com, mudimigh@ksu.edu.sa, zahid@ksu.edu.sa Abstract Software testing has emerged as an integral component in the software development life cycle. No software can be considered complete unless the testing has been done. Different software testing strategies have emerged over time, that somehow focus on the different levels of the software in order to identify that the software, under consideration is correctly developed and also that it is according to the requirements. Stress testing is used to identify that if the system is capable of affording the load and overwhelming responses, and how does it behave in circumstances. This paper not only unleashes that why and when the stress testing should be done, but also uncovers different strategies of doing the stress testing itself. Finally after a thorough study, investigations have been presented in a tabular format, showing that which testing strategy should be used with respect to the budget, time and specialty of the project under consideration. 1. Introduction Software engineering has emerged as a core discipline over time and the acceptability has been overwhelming. This is not because the fact that people love the world software engineering but because of the fact that both software engineering and software engineers have proved their worth and have made themselves not only acceptable but necessary for the world, specially for Information and Communication Technology (ICT) business. Software testing is the area that ensures that quality software is developed and also the developed software is according to the requirements. This process is called verification and validation of the system (V&V). [8] During testing process a detailed specification of each software component is used by an independent team in order to drive test cases for a system. Over the period of a few decades the science of testing has also evolved. In black Box testing: test cases are derived from the component specification. The whole system works as a black box and the behavior of the system is identified by the inputs and produced against them. In structural testing approach: test cases are derived for the system using the detailed specification of the structural implementation of the system. This approach is also called white box, glass box or clear box testing. Path testing: basically is an approach that uses the white box testing. Path testing strategy ensures that every independent path in a program is tested by using a test case. Hence ensuring that all paths are executed at least once and the finding of errors may be high, although its expensive with respect to time and resources. Cyclomatic Complexity technique is used to identify the independent paths in software [1].After the testing of individual components is complete; they are integrated to make a larger system or a sub-system. The integration process involves in building the software and also ensuring that the integration process has been error free, and also identifying the errors that might have evolved during the integration process. The incremental integration of the system makes it easy to locate the errors Interface testing: takes place only when the system has been integrated to create a larger system. The objective of the interface testing is to identify the possible errors due to the final integration and personal assumptions about the softwares interface.

Proceedings of Regional Conference on Knowledge Integration in ICT 2010

591

Stress testing tries to break the system under test by overwhelming its resources or by taking resources away from it (in which case it is sometimes called negative testing). The main purpose behind this is to make sure that the system fails and recovers gracefully; this quality is known as recoverability. The performance testing has the build-in requirements of sophisticated environment control and the measures that can be repeated with the same parameters, the stress testing has the luxury to be induced the chaos and unpredictability. Despite the minor disadvantages of non-documented system behavior, time constraint, monitoring overhead and large volume of data to be handled [7], stress testing is a choice to have an adequate view of the overall system performance. 2. Why Stress Testing? Stress testing can be used to judge the quality of software by measuring the response time, throughput and cohesiveness of the application modules. As stress testing tests the system as a whole, the whole functionality of the system is tested as a unit, hence enhancing the possibility of any error detection and reducing the possibility of system failure. eBay suffered with a loss of 3$-5$ million, while their site was down for around 22 hours. It has also been observed that the client/customers/users prefer not to establish/continue a relation with the site that has observes continuous problems. The Jupiter Media Matrix survey conducted on 2,000 users found that the user who were unhappy and have broken their relationships with troubling sites is 46%. It reduces the cost of software by helping to provide a more affectionate approach for the application domain architecture. 3. Testing Initiation When to start the testing remains a critical question in any software development life cycle. This certainly is a tradeoff between the software needs and business revenues. Although successfully completed software can greatly enhance the revenues of the development firm yet the decision to start the testing itself is generally delayed. In small setups the testing is delayed until the deadline of delivery is met, and in that case testing just occurs as a fatigue not as a whole activity in the development life cycle. As for as the stress testing is concerned, it is imperative that it starts as early as possible. [4]. The stress testing can be done at different levels, namely: Component Level: Although unit testing has its existence yet it has a disadvantage that in the domain of web services, it cant work to check the concurrency and deadlock of the simultaneous requests, adequately. Therefore it is necessary that each component residing on the web server is tested through the stress testing, in order to check that no deadlock occurs during the simultaneous access, and the consistent position of data is maintained and also no deadlock occurs while the records are being accessed and updated. Environment Level: After the completion of the requirement engineering phase, the development team decides the hardware and software infrastructure that they plan to provide for the development life cycle. The infrastructure may include a database

Proceedings of Regional Conference on Knowledge Integration in ICT 2010

592

application, a front end application, a hardware platform and a load balancer. This infrastructure helps in determining the scalability, reliability and cost of application. Hence, all available infrastructural options are to be reviewed categorically in order to identify and estimate the performance and the cost of performance. Architectural Level: Architectural level stress testing is also called benchmarking. The basic purpose of the stress test on the applications architecture is to measure the cohesiveness of the component residing at the different levels. A well responsive application would ensure that all components at all tires are well associated and working properly. During the development process, the sample components may be taken from each tire of the cohesive modules to detect any flaws during analysis and design of the application. End-to-End : End-to-End stress test has a flavor of real test that may be prolonged to several hours and in some cases even to some days. These End-to-End test (if accurately designed) test the application as whole and at length. This test tries to identify the issues like:

The functional errors that occur only under load conditions. If so, the relevant errors are noted and corrected without disturbing the functional coherence and cohesiveness (if possible). Does the application meet the service levels already defined in the requirement document? Does the application possess the EID (enhanced Information Deployment) behavior [1]? What is the current state of deployment of the application is it immediately deployable or not. If not, what corrections are to be made and what are the possible after effects of those corrections? And if the state of the application remains consistent after the corrections have been made or other inconsistencies are introduced due to the corrections of the initial ones. Is there any module that can be corrected to enhance the overall performance of the application?

4. Strategies for Stress Testing As opting for load testing is a key decision which is governed by the available budget not by the need. As the size of application under consideration grows, the size and cost of effort to do the stress testing also grows. So it is necessary to find a strategy that is right according to the application so that the cost and time spent on the stress testing are minimal and also the benefits are optimized. Many stress testing strategies are mentioned in this regard. Below is a brief discussion of each strategy and the possible problems and advantages that it possesses. Manual stress testing:Manual stress testing is generally considered to be the inefficient and expensive type of testing, as it does not produce repeatable results. So for every action the physically resources have to be present. If e.g. we plan to have 50 users access a resource simultaneously, all 50 users will be required to be present. In case of any error/problem the test is to be re-conducted and thus making the test more expensive. This type of testing is only suitable for very small stand alone application,

Proceedings of Regional Conference on Knowledge Integration in ICT 2010

593

otherwise it is problematic. Locally developed testing applications: As it is expensive to buy the enterprise tools for stress testing, sometimes the decision makers decide to build an in-house application. This kind of approach is just better than nothing. In most such applications test cases are written to test the core functionality of the application and hence the allied modules dont get sufficient attention for testing purposes. The development team is also restricted by budget and certainly cant develop the complete testing mechanism as the cost of testing tool may itself increase than the total revenue of the product being developed [4]. Open source stress testing applications: There are several stress testing tools available in the market. The general problem with these tools is that they dont provide adequate support for the script recorder that allow the tester to record the user actions into a script through a GUI and then again play the recorded script as a stress test. Such web tools are generally not customizable. Some of such tools are: WEBLOAD, FunKLoaD, OpenSTA, JCrawler, pylot, SLOPPY and WebInject [5]. IDE: The current development tools have minimized the needs of separate testing tools by using the Integrated Development Environment (IDE) which also has the option to test the application for any kind of stress living within the application environment. The advantage of this approach is that the developer is not required to learn the use of a new application just to know that how the application is going to be tested, so this way it reduces the time required to develop the test cases and also the stress test itself. While the IDE may be a solution for small application the IDEs are not suitable for critical, heterogeneous or legacy application environment. Web-only stress testing: In the market there are some tools available that are precisely used for the purpose of testing the web applications. The type of applications although cost less but are significantly for the purpose of testing the web applications only. The web testing applications possess limited scalability, accuracy, browser compatibility and reusability. These web tools, generally, dont provide the component level testing and therefore any problem identified at application level cant be identified and fixed. Third party (Hosted) Testing applications: The hosted stress testing applications test a web application over the Internet. The benefit of this approach is that it can be completed with the minimum effort, budget and resources, and does not require any special hardware, software or human on the part of development team. This testing approach is very useful in identifying the possible benefits of the stress testing. The stress testing is recommended to be performed both on the component level and also on the entire application. But sometimes some parts of an application may not be accessible through the public network, just making it difficult to test the application completely. Also the consistent network accessibility, speed and reliable connection

Proceedings of Regional Conference on Knowledge Integration in ICT 2010

594

are mandatory prerequisites of using this approach. Enterprise stress testing application: Enterprise stress testing application is perhaps the only type of application that meets the needs of the projects that are developed at the enterprise level. The enterprise stress testing tool is considered to be highly scalable and can test the heterogeneous environments at the same time and all that in an affordable cost too. The enterprise stress testing tool improves its adequacy by also providing support for the component test and thus helps in detecting the defects that may harm the performance of any application being tested. There are certainly, notable differences within different enterprise testing tools but the enterprise stress testing tools are far more beneficial in terms of performance and cost as compared to other tools.

Fig 1: Testing strategy, w.r.t. Scale, Budget & specialty

5. Limitations of Stress Testing Different web sites and precisely the mail servers have been taken under consideration in the above mentioned three scenarios. Following limitations have been observed through the comprehensive observation of the three scenarios. Stress testing is more suitable for the distributed system where the load is being shared at multiple places and hence the execution of test cases can be done with ease. stress testing is more suitable for the web based online application, as no bandwidth and network issues are involved in stand alone applications.[6] Stress testing is less appropriate for very small application, a there is hardly any inbuilt mechanism to handle stress, therefore a system failure would require more time and effort to recover than what was spent on it for development, hence ensuring extreme financial losses. 6. Conclusion The table above summarizes the discussion by illustrating that in which circumstances which stress testing strategy should be used. For example software can only be stress tested

Proceedings of Regional Conference on Knowledge Integration in ICT 2010

595

manually if its a small application and has adequate budget available. Stress testing however, is not recommended for small application, as stress testing itself is a very expensive activity and its budget may even increase from the budget of software itself. The locally developed test is more suitable for small and medium scale projects as they are not only less expensive but also are adequate for the customizable need of the software. The IDE based test can be conducted to all kind of application and just suitable resources are sufficient to conduct this test as no external cost is involved. The web only testing strategies are suitable for almost all types of software and even for small-budget software. Third-party testing tools can be used for all projects regardless the scale of development but adequate budget should be available to use the third party tools, as they are expensive. The enterprise stress testing strategy can only be used on large scale project having adequate funding available as enterprise testing requires huge financial resource that can just be suitable for large or very large projects. References [1] Ian Sommerville, Software Engineering,6th ed, pp451 [3] Stress Testing in agile computing, http://agiletesting.blogspot.com/2005/02/performancevs-load-vs-stress-testing.html [4] Borland, the open alm company, A Load Testing Strategy, white paper, April 2006,pp6 [5] Web stress testing tools: http://www.wareprise.com/2008/10/29/list-of-top-webapplication-load-and-stress-test-tools/ [6] Web performance checklist: http://key-performance.eu/solutions/openload/download5.html [7] Zhen Min jiang, Ahmed E. Hassan, Automatic identification of load testing problems, IEEE international conference on software maintenance, 2008, pp307-316 [8] Jiantao Pan, Software Testing, Carnegie Mellon University, Dependable Embedded Systems, spring 1999, pp 1-14.

Proceedings of Regional Conference on Knowledge Integration in ICT 2010

596