white paper

number 3 of a series

Making the Case for Test Process

Testing engenders trust
The leading organisations in the testing field are supporting the formation of an independent testing group within Intellect, the UK IT industrys representative body.

The aim is to promote best practice for the benefit of customers and the whole IT industry.

We believe that working together will promote a sense of trust in the testing profession and help leading organisations and customers to improve the effectiveness of their critical IT projects.

This is the third in a series of White Papers that will demonstrate the importance of testing to senior management and will form the basis of a best practice framework.

intellectuk.org

Non-Sponsoring Contributors: IBM (UK) Ltd QBIT Ltd Satyam Computer Services Xansa Plc.

isintegration.com

mercuryinteractive.co.uk

simgroup.com

missiontesting.com

rational.com/ worldwide/uk/

This document is for information purposes only. The authors make no warranties expressed or implied in this paper.

2004

white paper

number 3 of a series

Making the Case for Test Process

Table of Contents
Executive Summary Page 1

Executive Summary
Testing is increasingly being seen as a vital aspect of software development and delivery. However, ineffective testing can be just as damaging as no testing, costing significant project time, effort and money. Ultimately poor testing may fail to address software quality and reliability, with the unacceptable result that your customers are the ones who find and report the defects in your software.

Introduction

Page 2

Making the case for Test Process

Page 3

Process Implementation & Improvement

Page 5

If testing is the right thing to do, how can you ensure that
Ensuring the Successful Adoption of Test Process Page 8

testing is done right? This white paper, the third in a series of white papers from the

Where can I go for Further Information?

Page 10

Intellect Testing Group, will explain how your organisation can leverage the vast body of testing best practice that has been accumulated and used by those organisations that do testing right, helping to ensure you consistently reduce testing time, effort and cost, and deliver higher quality software more quickly.

Conclusion

Page 11

References

Page 11

A brief definition of Test Process

Page 12

The white paper is structured as follows:

Introduction - which reviews the challenges facing organisations engaged in testing and discusses the role of process in addressing them

Making the Case for Test Process - provides a number of success stories of companies adopting test process

Process Implementation and Improvement - explains the key elements of a test process with guidance on how to define and implement the appropriate test process for your organisation

Ensuring the Successful Adoption of Test Process - lists the key dos and donts of successful test process adoption

Where can I go for Further Information - provides guidance on where you can go to find further information on test process.

white paper

number 3 of a series

Making the Case for Test Process

Introduction
When you ask senior managers in organisations developing software if they follow industry best practices in their testing activities, the vast majority of them will confidently assure you that they do. The reality is often far less clear, and even where a formal scheme to follow such best practice has been introduced in an organisation, there are likely to be a variety of different approaches being used that result in varying levels of quality within the organisation. This thought raises more questions: do you know how much time, effort and money this testing costs your organisation? Can you estimate just how much risk is carried in terms of late delivery, with poor testing resulting in the release of poor quality software? Can you afford to run the risk of customers refusing to accept such software, or of serious defects being found during live operation with costly consequences? Can you be sure software is being tested thoroughly where required and not being over tested? You may recall from a previous white paper in this series (Ref: 1) that the US National Institute of Standards and Technology (NIST) reported that for every 1Million spent on software implementations, businesses incur more than 210K of additional cost due to problems associated with the impact of post-implementation faults (Ref: 2). Furthermore, in todays highly competitive IT business, there are massive pressures on companies to be as efficient as possible in developing and delivering software solutions. If you dont find strategies to reduce the cost of software development, your competitors will, allowing them to undercut your prices, offer to develop and deliver products faster, and ultimately, to take business from you. Typically, companies put up with this situation because they take a blinkered short-term view of the projects they run; much better to just get on with it and "make progress" than to take a more enlightened, but longer-term view to actually address the problems and fix them. Every project Just what hard evidence is there that such organisations are actually succeeding with implementing test processes? But talk is cheap; how many silver IT bullets have we seen over the years that have turned out to be blanks? And just because large numbers of organisations have adopted, or are in the course of adopting, a test process, this does not necessarily mean that they are realising genuine tangible benefits. Numerous organisations are now implementing repeatable processes as the solution to these problems. In this context, a process (such as a testing process) provides a means of disseminating industry best practice in software quality and testing to all of the staff in your organisation. Processes define who should do what and when, with standard roles and responsibilities for project staff, and guidance on the correct way of completing their tasks. Processes also provide standard re-usable templates for things like test plans, test scripts and testing summary reports, and even address issues of process improvement (if you are interested in delving deeper, a more comprehensive definition of process can be found later in this paper). has things that were done well and things that were done poorly. An organisation should establish a self-improving process where they learn from mistakes and remember what made a success.

white paper

number 3 of a series

Making the Case for Test Process

Making the case for Test Process

Even where senior managers in organisations appreciate that adopting a test process is the right thing to do, they will still need to convince their colleagues that there is commercial benefit to be gained; bottom line what evidence is there that other organisations are succeeding with a test process, how much will a test process save your organisation, how will it help manage risk, and what return will you accrue from your investment in adopting a test process? This section presents a series of success stories describing prominent organisations that have benefited from the introduction and use of a test process. These (and other success stories available via the Intellect web site) can be used as supporting material to make a case for the benefits of test process.

Success Story 1 Cap Gemini Ernst & Young achieve 50-75% reduction in test time and deliver zero defect system Business Problem: Cap Gemini Ernst & Young (CGE&Y) recognised the need for a standard test process supported by integrated lifecycle tools and test automation to ensure consistent global delivery of software development services in order to reduce risk and ensure quality. Solution: The adoption of a proprietary software development and test process used in conjunction with an integrated set of software development and testing tools delivered a series of tangible benefits:

Adoption of test process enabled better

information sharing and development efficiency, resulting in reduced risk and costs CGE&Y successfully tested and delivered a major software solution to meet their clients precise requirements with no defects even after 5 months of live use Adoption of test process ensured more robust and flexible system architectures to be developed and reused with concomitant benefits in terms of reduced timescales and cost

CGE&Y were able to meet client demands for

consistently-delivered global services Adoption of test process enabled the differentiation of CGE&Ys services and improved their win-rate

Please read more about these and other test process success stories at: www.intellectuk.org/groups/testing.asp

white paper

number 3 of a series

Making the Case for Test Process

Making the case for Test Process

Success Story 2 Test Process Enables Avanade to Manage Successful Relocation of Major High Street Retailer Business Problem: Commercial pressures meant that it was essential for a major high street retailer to relocate to new office premises. The need for a smooth and seamless move was identified as being key to the success of the relocation project, with the staff being able to continue using the companys mission critical software without disruption following the move. As part of the move, it was also decided that the customer would update their core infrastructure to a Windows 2000 solution, both from a client and server perspective. This situation was further complicated by the need to migrate a large number of software applications (some 300 in total), and to ensure their successful integration and operation in the new infrastructure. Solution: Avanade employed testing industry best practice within the relocation project to ensure that the migrated applications would both function correctly in the new infrastructure and interoperate successfully with the other migrated applications. To this end a rigorous test process was defined and documented, covering for example such items as checklists to ensure that the migration success criteria were met. The documented test process was founded on a risk-based approach whereby applications for migration were graded on:

The technical complexity of the changes applied The business criticality of the software The number of users affected

Other aspects of the test process included the use of automated testing tools to ensure that the base build of the Desktop was not affected by the introduction of enhanced/upgraded applications. Furthermore the test process verified that the applications could still co-exist on the target infrastructure. The customer continues to benefit from the successful application of the test process.

Success Story 3 Lloyds TSB Private Banking (LTPB) deliver mission critical private banking software with just three minor problems being observed post launch Business Problem: In the face of growing commercial competition and demand from their customers for internet based private banking, LTPB were keen to investigate strategies for delivering high quality and high reliability web based software as quickly as possible with as little disruption to their business as possible; conservative estimates suggested delivery of poor software could potentially cost LTPB millions of pounds in lost business, possible litigation, as well as the more intangible (but no less damaging) loss of public confidence. Adoption of thorough and effective testing for the new technology solution was identified as being imperative. Solution: A test process was designed that articulated the business-needs that initially were poorly defined. A pragmatic and disciplined approach to identify and eliminate software defects was then implemented. Key support in terms of test automation tools and resource management was incorporated into the process. A risk assessment strategy was also completed at this stage. Adopting test process enabled LTPB to:

Launch the web based offering with only three

low impact problems being discovered post delivery

Achieve a high level of transfer of competencies and

skills throughout their staff, ensuring all staff were able to employ industry best practice in their testing tasks on all projects they were engaged on

white paper

number 3 of a series

Making the Case for Test Process

Process Implementation and Improvement

The first thing organisations often find when they try to understand a common test process is that many projects they have are quite different from each other. Finding a common way to perform testing seems impossible at an initial analysis. This is to be expected as a project is made up of a unique combination of technical platforms, application systems and user organisations. These factors play a large role in determining the right process and test strategy. So how can a common test process be established? Policies and Principles The place to start is to establish the goals for testing and quality within the organisation overall. The extent to which testing will be part of an IT organisation or a business organisation is an important executive decision. The quality goals the organisation believes in are very important as is the visibility the testing groups will have throughout project life cycles. These core beliefs and goals can be distilled into a short set of easy to understand policies and principles for an organisation. This common foundation for quality and testing will apply to all projects. Development and Implementation methods There are various ways to develop and implement systems. The primary ones involve strategic decisions between building systems and buying packages. It would not be uncommon to see both types in an organisation but consciously decided for different systems. Another strategic decision is the extent to which development and implementation is done internally or outsourced. This plays a dramatic role in deciding optimum internal processes. For example the quality and testing processes required for an in house development of a sales system differ significantly from the processes required for the outsourced development of a CRM system. 1. Develop your own test process from first principles 2. Get a specialist testing consultancy to implement your process 3. Buy an off-the-shelf process and customise it to meet your needs Having recognised that your organisation will benefit from the introduction and use of a test process, you will still need to decide on a strategy for adopting and delivering your process. The main options can be summarised as follows: Finding the common process through all of these variables involves starting with policies, then making the policies clearer with development strategies (buy verses build and in source verses out source). This determines the need for quality management and hands on testing processes. The platform and the project determine what the processes should look like. The common process Project or programme instance Each project will then be defined to do a specific set of developments or changes to application systems that impact all or part of an organisation. At this level the testing process should be more prescriptive in determining the best practices to be used by a project but must still be adaptive to the needs of the project. The platforms used by an organisation will also vary according to technical decisions made over a period of time and as new technologies are introduced in the market. Whilst these variations play a lesser role in determining a testing process, they do influence how a testing process is optimised for a given technology. Platform and technical environment

white paper

number 3 of a series

Making the Case for Test Process

Process Implementation and Improvement

Option 1 This option has the advantage that you should end up with a process that closely matches your organisational requirements for developing and testing software. However, it does have a significant disadvantage; you will need to have staff with suitable experience plus their time and availability to define the process. You will also need someone to maintain the process to ensure it keeps up-to-date with changing needs and technologies, and to ensure that you are at least aware of developments in current industry standards and best practice. Option 3 With the best intentions in the world, this strategy frequently fails because staff are simply pulled off the task of developing the process to "fire fight" on higher priority projects (ironically, these problem projects are often precisely the ones that could benefit from process). Option 2 This option involves engaging a specialist-testing consultancy to develop and deliver a test process tailored to your organisations particular testing requirements. Putting the development of your test process in the hands of a reputable testing consultancy means that there is a high probability that the process will end up following industry standards and best practice. Furthermore, the delivered product should provide a very close match to your organisational requirements for such process. This strategy will still require commitment and investment, but it has the major advantage that it is unlikely to tie up valuable internal resources. Typically, such specialist companies will seek to review and understand your testing requirements, identify key areas for improvements, and plan an incremental delivery of elements of the test process. The goal of this approach is to quickly deliver realisable Advice on where to go to find further information on off-the-shelf test process products is provided later in this white paper. In addition to guidance on testing best practice, commercial test process products should also include the means of delivering the process (such as via an intranet browser interface), as well as industry standard downloadable artefacts (document templates, pro-formas, blank test scripts, etc). These features are intended to help those organisations using such products to realise the return on investment as quickly as possible. While such products provide comprehensive guidance on industry best practice, to gain maximum benefit, they should be customised to more closely match the specific testing requirements of your own organisation (such tools typically include guidance on customisation and the facilities to adapt and maintain the process). Assistance can also be obtained from the process vendor or from specialist testing consultancies, who are able to provide guidance and assistance on the rollout and customisation of such products. This option involves the purchase of a commercial-off-the-shelf test process. Advice on where to go to find further information on specialist testing consultancies is provided later in this white paper. To ensure long-term benefits and continuing return on investment, the process should be designed to be easy to maintain in the event of changes in technology and/or the development of new methods or approaches to testing. testing benefit, ensuring rapid return on investment.

white paper

number 3 of a series

Making the Case for Test Process

Process Implementation and Improvement

Process Delivery The most important success factor of delivering a process is to get the right balance between introducing change and commitment from the people that will use the process. During the development of your process, you will need to consider how the information it contains will be delivered to your staff, how they can be encouraged to make use of the process, and how the process will be maintained and improved. Delivery can be as simple as providing paper based manuals, and there are numerous examples of organisations successfully implementing such a strategy. One issue to consider in adopting such an approach is that there is likely to be a significant maintenance cost associated with updating a paper based process, which relies on staff updating their own copies Process Improvement No test process is ever static; technology changes, new testing techniques are introduced, and your organisations test requirements will change. Maintenance of your test process is a necessary part of ensuring it remains effective, efficient and relevant to your testing needs. In parallel to the maintenance of the test process, you should also consider how to improve the process, looking at how defect detection rates can be increased and cost of testing reduced, for example. Process improvement (Ref 8) is a topic that could easily justify a white paper in its own right; however, in the context of this paper we can define process improvement as a strategy that includes both the means of introducing changes to the test process and the means of assessing the success of those changes. One example of a change to the test process could be the introduction of automated testing tools. Assessing the success of changes is typically achieved by the collection and analysis of metrics (Ref 7 provides guidance on the introduction and use of metrics in a testing context plus a suggested metrics starter set). Examples of typical metrics that are collected include: how much effort was expended in completing the testing project, how many defects were found during testing, and how many defects were found by the client following delivery of the tested system. Such information can be used to determine whether the test process is improving over time, by for example, comparing the current effectiveness in finding defects against previously observed results, or for investigating the impact of a specific change to the process. Where your test process has been implemented by a specialist-testing consultancy, you should expect the process to incorporate a scheme for process improvement. Similarly, for those organisations adopting an off-the-shelf process product, the tool should include guidance on process improvement.

with the inevitable risk of missed or lost updates; the challenge of adopting paper based processes is to prevent them ending up on shelves collecting dust rather than being used as a day-to-day reference. Another strategy is to provide access to the process electronically, often by means of your organisations own intranet system. In this approach, the process is held on a central server and delivered to the staff on their own workstations using a browser interface. Updates can be managed centrally with no danger of staff using out of date information. Another powerful feature of such an approach is that electronic copies of standard document templates can be downloaded from a central repository and customised by staff for use on a project-byproject basis.

white paper

number 3 of a series

Making the Case for Test Process

Ensuring the Successful Adoption of Test Process

You may believe you have implemented the best test process in the world, but it may all have been for nothing if nobody in your organisation actually uses the process. And worse, you will have wasted the valuable time, effort and money spent in researching, developing, documenting and rolling out the process. delivery plan. Remember, delivering the training too early will be as ineffective as delivering the training too late; ensure that staff are given instruction in a timely manner so that they can start to use the information as quickly as possible on real projects.

It is just as important to consider strategies for successful adoption and use of the process, as it is to "get the process right". This section provides useful advice in the form of the most common "dos and donts" of process adoption.

Do ensure good access to the information contained within your process; if you are delivering a paper based process ensure it is easily available to all the relevant staff. Where appropriate, consider delivery of your process using your intranet system to improve access and to simplify maintenance and update.

Do ensure commitment of all key decision makers in your organisation; too often process implementations have started-up with the highest ideals and goals, and then foundered when senior management take their eyes off the ball and are distracted by a new endeavour, or are promoted, retire or move sideways in the organisation.

Dont allow the "not invented here" syndrome to block process adoption; there are likely be other groups or individuals in your organisation who already have an interest in process and who may be unwilling to support a new approach. Such groups or individuals should be identified early during the requirements phase of the development of your process, and their views addressed within the process; approached properly such individuals can become powerful advocates for process. Approached badly or ignored, they can work passively or actively against successful adoption of your process.

Dont try to rollout your test process across the whole of your organisation in one go; identify a suitable testing project and consider piloting your process within that project. Get it right on the pilot first, and then, using the lessons learnt, plan a wider rollout.

Do identify a Process Champion; such an individual, typically with a senior management or technical profile and good enthusiastic communication skills can be invaluable in promoting the cause of test process within your organisation. Such an individual should be able to communicate at all levels within the organisation and should be sufficiently technically aware to be able to make the case and argue for the adoption and use of process.

Do promote success; nothing succeeds like success. Consider setting up a test process special interest group that can organise periodic meetings to discuss process issues and present successes. Newsletters, your company intranet, and your process champion can all be used to promote success, reiterate the benefits of, and encourage adoption and use of your process.

Dont forget to plan for process training; ensure you budget for training and mentoring of staff, and incorporate this training into the

white paper

number 3 of a series

Making the Case for Test Process

Ensuring the Successful Adoption of Test Process

Dont give up too soon; although benefits such as improved quality of delivered software may be seen relatively quickly, the overall monetary return on investment will take longer to accrue. Many staff, particularly those with an accountant mentality, may push for process to be dropped in the absence of instant fiscal results. Be careful to manage the expectations of such colleagues; dont present them with an over optimistic view of how soon they will see a return on investment, but instead emphasise the other benefits that can be more quickly realised.

Do implement a process improvement scheme; ensure you collect metrics describing the state of your testing before implementing your test process (otherwise you will have no basis to measure the improvements process will bring). Continue to record metrics in order to understand how continued use of the test process or introduction of new features has impacted your overall testing effectiveness and efficiency.

white paper

number 3 of a series

Making the Case for Test Process

Where can I go for Further Information?

If this white paper has caught your interest in test process and you would like to find additional information on the subject, this section contains a number of useful sources of further information on test process. The following web sites are recommended as useful sources of test process information: There are a number of books that can be referred to for further information on test process:

Ref 7 provides a comprehensive view of the

test process, includes case studies showing how organisations have adopted test process tailored to their own test requirements, as well as a comprehensive set of standard testing templates.

You can obtain general testing guidance,

further details of test process success stories, and access to the other white papers in this series at the Intellect Testing Group pages www.intellectuk.org/groups/it/groups /testing.asp

Ref 8 documents the Capability Maturity

Model (CMM) an example of a scheme that can be applied to test process improvement

References 4 and 5 provide a good source

of information on testing fundamentals.

The British Computer Society Special Interest

Group in Software Testing (BCS SIGiST) provides a vigorous forum for test process discussion - www.sigist.org.uk Details of organisations who can provide specialist test process advice and guidance, and suppliers of off-the-shelf testing products can be found at www.intellectuk.org/groups/it/groups /testing.asp

The Software Process Engineering Metamodel

(SPEM) is a proposal for a standard software development and test process, which is sponsored by the Object Management Group (OMG). Ref 6 provides a specification for SPEM. Also see www.omg.org/technology/documents /formal/spem.htm

10

white paper

number 3 of a series

Making the Case for Test Process

In conclusion
Both this white paper and the previous one in this series (Ref: 1), have made the case for testing as an activity that adds real and quantifiable value to the software development life cycle. As we said in the introduction, ineffective testing can be as damaging as not testing at all, and worse, you will have expended valuable time and money on wasted effort. Test process provides a means to ensure that your testing time and money are spent effectively and efficiently. A test process can encapsulate industry best practice in a form that can be used by all of your staff, giving all members of your organisation the means to perform their testing tasks to the levels achieved by the most effective practitioners in the testing field. At the organisational level, test process provides your company with the means to achieve savings The longest journey, as they say, starts with but a single step; take that step by using the information provided in this white paper to investigate the benefits that test process can bring to your own organisation. This white paper provides an objective view on the benefits that adoption of test process can deliver, as well as the means for you to quantify the value of process in the context of your own company. in time, effort and cost of testing, and improve the quality and reliability of your delivered software. As a consequence, increasing numbers of organisations are adopting test process in order to accelerate delivery of software and to provide themselves with a commercial advantage over their competitors.

References
1 Intellect Testing Group: "Realising the True Value of Testing", Intellect - 2003. Available from the Intellect web site. 2 National Institute of Standards & Technology (NIST), US Dept. of Commerce: The Economic Impacts of Inadequate Infrastructure for Software Testing May 2002 3 IEEE 829-1983 Standard for Software Test Documentation. Software Engineering Standards Committee of the IEEE Computer Society, New York 4 "The Complete Guide to Software Testing Second Edition", Hetzel, B., QED Information Sciences Inc, Massachusetts, 1988. 5 "The Art of Software Testing", Myers, G. J., John Wiley, New York, 1979. 6 "Software Process Engineering Meta Model Specification", the Object Management Group Inc., November 2002. 7 "Testing IT: An Off-the-Shelf Software Test process", Watkins, J., Cambridge University Press, 2000. 8 "Key Practices of the Capability Maturity Model - Version 1.1", Paulk, M. C., et al, Software Engineering Institute - Carnegie Mellon University.

11

white paper

number 3 of a series

Making the Case for Test Process

A Brief Definition of Test Process

A software test process is a documented set of best (or fit for purpose) practices describing how to achieve effective and efficient testing of software applications. A test process should not be a static document , but should be regularly reviewed and maintained in order to incorporate new best practice or to accommodate changes in technology and/or approaches to software development and testing. A test process should incorporate the means to facilitate improvements to the process so that changes to the process can be measured and analysed to determine the benefit, or otherwise, of those changes. A test process should also consider, address and integrate with other aspects of software development, such as requirements management, configuration management, change management and defect tracking. Specifically, a software test process should document (in no significant order): A test process should also demonstrate awareness of other sources of testing information (such as referencing test documentation standards (such as Ref: 3)), definitive texts (such as those by Bill Hetzel (Ref: 4) and/or Glenford Myers (Ref: 5)), and process improvement schemes (such as the Capability Maturity Model scheme (Ref: 8)) for example). 1. Where document can include paper or electronic means of delivering a test process.

A glossary of relevant terms The roles and responsibilities of the testing staff as well as their, reporting and liaison (such as Test Manager, Test Designer, and Tester).

Any artefacts employed in the process, such as:

test plan documents test cases test scripts test logs test summary reports

Specific guidance on best practice (such as how to create realistic and representative test data)

The relationships/interfaces to other disciplines (such as the need for access by the Test Manager and Test Designer to the current requirements for the software under test)

Guidance on the use of specific techniques and tools (such as the use of Boundary Analysis in creating test data, or the role and use of automated testing tools).

The tasks that need to be completed and the order in which they are undertaken (such as the need to develop a test plan)

The inputs and outputs to and from tasks (such as the software to be tested, the date it will be available for testing, and the delivery date for the tested software)

Details of the testing phases (such as their names (Unit/Component test, Integration/link/module testing, System testing, etc), the order they take place in, and the staff responsible for conducting them)

12