Diculties in Establishing a Defect Management Process: A Case Study

Marko Jntti, Tanja Toroi, and Anne Eerola

University of Kuopio, Department of Computer Science, PL 1627, 70211 Kuopio, Finland mjantti@cs.uku.fi

Abstract. A well-organized defect management process is one of the success factors for implementing software projects in time and in budget. The defect management process includes defect prevention, defect discovery and resolution, defect causal analysis, and the process improvement. However, establishing an organization-wide defect management process is a complicated task. The main research question in this paper is what kind of diculties organizations have regarding the defect management process. Our ndings show that problems are related to defect resolution reports, limited project resources for xing defects, and challenges in creating a test environment. Results are based on our observations from four case organizations. The main contribution of this study is to help organizations to identify and avoid typical problems with defect management.

Introduction

Establishing a defect management process is an attractive way to improve the software quality. Early detection of defects provides cost and time savings for software projects because developers need to produce less new product versions and bug xes. Moreover, reduced number of defects in applications increases the level of customer satisfaction, and reliable software is easy to sell to new customers. Several studies have explored defect management activities. For example, different types of defect management models have been described by the Software Engineering Institute (SEI) [1], [2], IBM [3], the IT Infrastructure Library [4], and the Quality Assurance Institute (QAI) [4]. According to the QAI, the defect management process consists of six elements: defect prevention, deliverable baselining, defect discovery, defect resolution, process improvement, and management reporting. The Defect Prevention Model of IBM [3] is focused solely on defect prevention techniques, for example, Defect Causal Analysis method. The causal analysis is used to identify the root cause of the defect [5]. Additionally, there are a number of recent studies that have presented dierent defect classication schemes [6],[5]. Too detailed reporting and complex classication schemes might increase defect processing costs remarkably [7]. Defect management activities are also supported by various quality standards. CMM at Level 5 considers defect management as a key process area with the following

J. Mnch and M. Vierimaa (Eds.): PROFES 2006, LNCS 4034, pp. 142150, 2006. c Springer-Verlag Berlin Heidelberg 2006

Diculties in Establishing a Defect Management Process

143

goals: defect prevention activities are planned, common causes of defects are sought and identied, and common causes of defects are prioritized and systematically eliminated [8]. Currently, many organizations are adopting the Problem Management model described by the IT Infrastructure Library (ITIL) because ITIL has become a de facto standard for IT service management [4]. However, the ITIL model does not dene how to perform testing and defect management activities as a part of IT service management. In fact, it seems to be that there is a need for both a problem management model used by the service support sta and a defect management model used by the application developers and testers. This might cause a communication gap if both counterparts use dierent data repositories for problems and defects. However, few studies have examined the problems that organizations have with defect management. This study continues the work reported in our previous study [9], where we identied the advantages and problems of using an UMLbased test model for creating test cases based on UML diagrams [10]. In this paper the research question is, what kind of problems do organizations have regarding defect management? First, we explore four case organizations goals for defect management and defect management processes. After that, we will investigate what are their problem areas in defect management. Most of the previous research of defect management has focused solely on software companies, although customers are active participants in the defect management process. In our study we are going to examine defect management problems also from the IT customers viewpoint. As main ndings we will show that instead of tool-related diculties major problem areas in defect management are, for example, dealing with defect resolution reports, creating a test environment and a lack of commonly agreed defect management methods. The rest of the paper is organized as follows. In Section 2 we describe the research methods of this study. In Section 3 ndings of the case study are presented. Section 4 is the analysis of ndings. The discussion and the conclusions are given in Section 5.

Research Methods

This case study is a part of the work of the research project SOSE (Service Oriented Software Engineering) at the University of Kuopio, Finland. SOSE is funded by the National Technology Agency TEKES, the European Regional Development Fund (ERDF), and four partner companies. The study was carried out partly with the research project PlugIT (2001-2004), which focused on research into application integration methods in the healthcare domain. The main research purpose of this study was to explore the problems that organizations have in defect management. A case study method was used because it is well suited for the study of information systems in organizations. Yin [11] denes a case study as "an empirical inquiry that investigates a contemporary

144

M. Jntti, T. Toroi, and A. Eerola

phenomenon within its real-life context, especially when the boundaries between phenomenon and context are not clearly evident". Both IT customers and software companies were selected for this study because our objective was to compare the diculties in defect management between these two groups. 2.1 Data Collection Methods

Case A is a large IT service company with over 15 000 employees. It supplies information systems to various industries, such as banking and insurance, energy, telecom and media, and healthcare. The data collection methods in this case included personal interviews with a product manager and a customer support manager. Case B is a project-oriented software company. Its core business is focused on implementing solutions for mobile communication: for example, solutions that enable operators to charge for GSM calls. The company employs 53 people. Their strategic partner is one of the leading GSM operators in Finland. The data collection methods included an interview with an IT manager. Case C is an energy company group consisting of 1) the parent company, which is responsible for the network business and group administration; 2) a company specializing in selling electricity; and 3) the district heating company. They have 430 employees. The data collection methods included personal interviews with an IS manager and a risk manager. Case D is an IS department of a hospital oering specialized services, including specialized nursing services, to the healthcare district. It is also a teaching hospital with medical, nursing science and healthcare students. The data collection methods included interviews with a system tester and a system designer. The interviews with the tester were conducted in the course of the UML-based testing experiment in the PlugIT project [9]. The purpose of the experiment was to explore how UML-based testing [12] helps in nding defects in a healthcare application. Interviews were performed by the rst author and were based on the questionnaire that was based on the research framework developed by the Quality Assurance Institute [13] containing the following questions: 1. Do you have a software quality program, a formal software development lifecycle model, a defect classication method or denition of defects? 2. What kind of defect information is collected? 3. Who collects defect data? 4. What are the sources of defect data? 2.2 Data Analysis Methods

A cross-case analysis technique [14] was used in this study to analyze data from interviews and to compare cases. In the data analysis, we tabulated the data on cases into four categories: existing defect management methods, type of defect information, defect data sources, units responsible for collecting defect data. Then we compared the results, looking for similarities and dierences in defect management processes and problems. Finally, we analyzed how organizations could improve their defect management processes based on the results of this study.

Diculties in Establishing a Defect Management Process

145

Empirical Findings

This section presents our empirical ndings from four cases. In this study, we explored the organizations goals in defect management, their defect management processes, and problems related to the activities in defect management. 3.1 Defect Management Processes

Table 1 shows our ndings regarding the four cases defect management processes. Notes: N = no, Y = yes, - = missing data and P = partially.
Table 1. Defect Management processes in four case organizations Question The case organization has Software Quality Program Formal software development lifecycle Company-wide method of gathering defect information Method of classifying defects/problems Denitions of defects and failures The case organization collects information on Development/Production failures Development phase where a defect originated Activity which originated the defect/problem Type of the defect/problem Cause of the defect/problem Time to resolve the defect /problem Defect/problem resolution costs Sources of defect data are Change Request Form Problem Reports Reviews Inspection measurements Operations/production reports Change Management function Defect data is collected by HelpDesk Quality Assurance Project teams ABCD P Y Y Y Y Y Y N Y Y Y N Y Y Y Y N Y NN YN YY YY NY YY YYY YY NY YY NY NY NY YY YY YY YN N N P Y Y Y Y Y Y N N Y Y Y N N -

YNYN YYYYYYY

Our case organizations goals in defect management were to 1) improve customer satisfaction and decrease costs, 2) to increase software quality, 3) to ensure that customers will get services with as few failures as possible, and 4) to ensure that purchased software works correctly in the system environment. 3.2 Problems Regarding the Defect Management Process

Personal interviews with the case organizations identied following problem areas, bottlenecks and challenges in defect management processes:

146

M. Jntti, T. Toroi, and A. Eerola

1. Dening good metrics for IT service problem management and Service Level Management is a challenge. 2. Creating a large amount of test data for testing is dicult. 3. Load testing and performance testing tools cannot test the whole system. 4. Informing customers of new defects is a challenge because not all customers use the function that containing the defects. 5. Limited resources for xing defects. 6. Establishing a defect management framework takes a lot of time. 7. Teams use dierent methods: the challenge is how to combine methods. 8. IT companies often consider the bugs found by the customer as typical properties of the application. 9. IT companies do not send defect resolution reports to customers. 10. Establishing a test environment is always a challenge. 11. Some software vendors do not provide defect reporting services to customers. 12. No training in using a new defect management tool for reporting defects. 13. Negative attitudes towards defect reporting tools: long report forms. 14. Software vendors deliver applications containing many bugs that should have been found by the developer-side testing.

4
4.1

Analysis
The Analysis of Defect Management Processes

As expected, both customers reported that they do not have software quality programs or a formal software development lifecycle model. Case A stated that some units in their organization have quality programs. All cases had a company wide method for gathering defect data. Case Ds answer was partially yes because they used defect reporting services provided by software vendors. They told that for some vendors defects must be reported by phone or by email. All cases had a method for classifying defects. Case C also used a domainspecic problem classication (low voltage, medium voltage, high voltage faults). Similarly, all cases collected information on development / production failures and the type of defect. Surprisingly, only one of the cases systematically collected information on problem resolution costs, and even this was not related to software problems but to energy faults. Regarding the sources of defect data, on the basis of the interviews it seems that all the cases had used some informal reviews, and three cases had used inspections. Three cases mentioned problem reports. Defect data were collected by project teams in all cases. Cases A and C had a help desk function that was responsible for collecting incidents. Case D emphasized the importance of testers in collecting defects. Testers test each product version, and if the purchased product contains severe defects it will not be installed into the operation environment. In the course of this study an interesting observation was made that Case A and Case D actually use the same defect reporting tool, but in dierent roles (A as an IT service provider and D as an IT customer). According to Case A, the advantages of the tool were an aordable price, a direct web interface for the customer, and good customization possibilities. The system designer of

Diculties in Establishing a Defect Management Process

147

Case D reported that the defect reporting tool is easy to use, and wished that other vendors oered similar tools. Another interesting nding was that while the customers reported that they use defect reporting tools provided by software vendors, the software company B used defect reporting tools provided by their customers. 4.2 The Analysis of the Identied Problems

Regarding the identied problem areas and challenges in defect management, probably the most interesting challenge was reported by an IT service provider that was looking for good metrics for IT service problem management and service level management because they were changing old processes to the processes that are compliant with the IT Service Management standard (ITIL). This was an interesting issue because when the company has a leading position in providing IT services in Scandinavia it can require that their partners and subcontractors must also use ITIL-based processes including problem management. Most of the problems identied in our interviews were not surprising. Two case organizations stated clearly that it is dicult and expensive to create a test environment matching to the real production environment. A big problem is, for example, creating large amounts of suitable test data for testing. It is more expensive and dicult to create 100 Gb test data than only 1 Mb. Secondly, load testing and performance testing tools are often able to test only a part of the system. One of the software companies also stated that IT customers could use the review or inspection methods for the system specication phase in order to point out the issues that are unclear or that include problems. That would probably increase the customers degree of involvement in project issues. One of the companies told that they have not had any problems with defect management tools but the major problem is that their project teams use dierent methods in managing defects. Customers reported as a big problem that IT companies consider the bugs found by the customer often as typical properties of the application. "It is not a bug" has often been a statement from Help Desks of IT providers. An interesting question is why software companies do not admit that their application might have problems and state that problems will be carefully examined and possible improvements are implemented in the next product release. They also stated that IT providers should inform how the defects, reported by a customer, were handled or xed. Often, customers do not understand defect resolution reports that they have received. Lack of resolution reports does not motivate customers to send problem reports to the IT provider in further projects. One of our cases considered as a major problem that all the software vendors do not provide defect reporting systems to customers. In one case, a software vendor had oered a tool for reporting defects but a customer did not use the tool because they had not received any training. Additionally, IT companies reported some other problems, such as high prices of test tools. Especially, load testing is expensive with a large number of virtual users. Limited project resources also aect the use of dynamic and static techniques.

148

M. Jntti, T. Toroi, and A. Eerola

4.3

Lessons Learned

Based on our case study results, we emphasize that organizations should pay more attention to the following issues in performing defect management activities: The defect management process must be an organization-wide process, although Ahonen et al. have argued that it is practically impossible to ensure that all teams use good practices [15]. Henninger [16] has proposed that organizations should build an organizational repository of experiences. Learning from defects is a very good example of experience-based learning. There has to be clear rules and guides that dene who are responsible for recording defect data, how to change a status of the defect, or how to classify a defect. Project teams should be motivated to share information on defects between dierent projects. The support sta of the IT organization must pay more attention to the service quality in the problem situation avoiding this is not a bug service. All problems reported by customers must rst be recorded by the service desk that is usually capable to resolve most of the simple problems. Therefore, programmers will not be disturbed and they have more time to focus on serious problems. The defect/problem resolution reports, that an IT organization sends to customers, must be clear and consistent avoiding dicult IT terms. The organizations management has to allocate sucient resources to defect management and testing teams, and motivate them to use diverse methods for preventing and nding defects, for example, UML-based testing [17], software inspections [18], and defect causal analysis [19]. In the long run, the defect management must be more proactive than reactive. Focusing on preventing defects and problems before they occur is more useful than correcting a large number of repetitive errors. Customers and end users must be trained to use automated defect reporting tools, FAQ sites and known error databases, if available. These tools will remarkably decrease the amount of work spent on processing defects. The defect management model must have clear connections to the neighbor processes such as service desk, change management, testing, and application development. Most of the current models (e.g. IBM, QAI) have ignored this issue. The major drawback of the ITIL problem management model is a poorly documented testing process. One possible method to test the connections between processes is to review the defect life cycle with ten sample defects and identify how they were processed in dierent units of the organization. Suitable metrics should be produced for both the defect management (e.g. defect removal eciency, mean time to failure) [13] and monitoring service levels (e.g. a number of breached service level agreements, the resolution time for a problem) [20].

Diculties in Establishing a Defect Management Process

149

Discussion and Conclusions

This study aimed to explore problems regarding defect management. Our ndings show that dierent stakeholders have dierent problems. A large part of the identied problems were not surprising. A lot of traditional problems were revealed in interviews. IT customers have problems getting defect resolution reports from IT companies, or interpreting them. They dislike being told by IT companies that the bugs they nd are typical features of the application. Moreover, the software they purchase contains defects that should have been found by IT companies testers. IT companies problems are related to decisions about whether to send defect resolution reports and bug xes to customers or not. One major problem is the fact that project teams do not have commonly agreed methods of defect management. The developer side often has limited resources to x defects. Creating a test environment seems to be a common problem for both IT customers and IT companies. Contrary to expectations, problems related to the usability of defect reporting tools were relatively unimportant to the participants in this study. Similarly, static methods for nding defects were used more often than expected. The IT service provider reported a new interesting challenge: how to create suitable metrics for IT service problem management and Service Level Management. This requires further investigation. As with all case studies, there are threats to the validity of this study. First, construct validity is problematic in case study research. Data for the case study should be collected from several sources. We tried to avoid problems with construct validity by using multiple sources of evidence, such as conducting personal interviews with at leasttwo persons per case organization. One potential weakness is the fact that most of the interviewees in our study were managers. In order to get a richer view of the problems in defect management, we need to interview more ordinary programmers and testers. Second, there is the threat to external validity, i.e. to the generalizability of the results. The results of this study might not be generalizable to other organizations, since there were only four cases in this study, and all the cases were in dierent industries. The main contribution of this study lies in helping IT companies and IT customers to identify and avoid typical problems in defect management. Future studies should explore more deeply the problems organizations face with regard to defect resolution reports. In future studies we intend to improve our research framework by exploring how our current defect management model and the problem management framework of the IT Infrastructure Library can be combined, and how service level agreements can be utilized in defect management.

References
1. Florac, W.: Software quality measurement a framework for counting problems and defects. Technical Report CMU/SEI-92-TR-22 (1992) 2. Hirmanpour, I., Schoeld, J.: Defect management through the personal software process. Crosstalk, The Journal of Defense Software Engineering (2003)

150

M. Jntti, T. Toroi, and A. Eerola

3. Mays, R.G., Jones, C.L., Holloway, G.J., Studinski, D.P.: Experiences with defect prevention. IBM Syst. J. 29(1) (1990) 432 4. Oce of Government Commerce: ITIL Service Support. The Stationary Oce, UK (2002) 5. Leszak, M., Perry, D.E., Stoll, D.: A case study in root cause defect analysis. In: ICSE 00: Proceedings of the 22nd international conference on Software engineering, New York, NY, USA, ACM Press (2000) 428437 6. El-Emam, K., Wieczorek, I.: The repeatability of code defect classications. Technical Report. International Software Engineering Research Network, ISERN-98-09. (1998) 7. Humphrey, W.S.: A personal commitment to software quality. In: ESEC. (1995) 57 8. Jalote, P.: CMM in Practise, Processes for Executing Software Projects at Infosys. Addison-Wesley (2000) 9. Jntti, M., Toroi, T.: Uml-based testing: A case study. In: Proceedings of NWUML2004. 2nd Nordic Workshop on the Unied Modeling Language, Turku: Turku Centre for Computer Science (2004) 3344 10. Kruchten, P.: The Rational Unied Process: An Introduction. Addison-Wesley (2001) 11. Yin, R.: Case Study Research : Design and Methods. Beverly Hills, CA: Sage Publishing (1994) 12. Binder, R.: Testing Object-Oriented Systems: Models, Patterns, and Tools. Addison-Wesley (2000) 13. Quality Assurance Institute: A software defect management process. Research Report number 8 (1995) 14. Eisenhardt, K.: Building theories from case study research. Academy of Management Review 14 (1989) 532550 15. Ahonen, J.J., Junttila, T., Sakkinen, M.: Impacts of the organizational model on testing: Three industrial cases. Empirical Softw. Engg. 9(4) (2004) 275296 16. Henninger, S.: Using software process to support learning software organizations. In: 1st International Workshop on Learning Software Organizations, Kaiserlautern (1999) 17. Hartmann, J., Imoberdorf, C., Meisinger, M.: Uml-based integration testing. In: ISSTA 00: Proceedings of the 2000 ACM SIGSOFT international symposium on Software testing and analysis, New York, NY, USA, ACM Press (2000) 6070 18. Gilb, T., Graham, D.: Software Inspection. Addison-Wesley (1993) 19. Card, D.N.: Learning from our mistakes with defect causal analysis. IEEE Software 15(1) (1998) 5663 20. Oce of Government Commerce: ITIL Service Delivery. The Stationary Oce, UK (2002)