Вы находитесь на странице: 1из 92

K i m

H .

P r i e s

&

J o n

M .

Q u i g l e y

Professional Development Series

Saving Software with Six Sigma

volume 1

Saving Software with Six Sigma


by Kim H. Pries & Jon M. Quigley

Edited by: Rich Hand

Copyright 2011 STP LLC All rights reserved. Printed in the United States of America ISBN: 978-0-9831220-0-5 STP 724 South Tejon Street, Suite C Colorado Springs, CO 80904 U.S. and Canada: (877) 257-9531 www.SoftwareTestPro.com STP assumes no liability for error or omission. No part of this publication may be reproduced without the consent Of STP, with the exception of trade publications reporting on the data. In such cases, credit must be given to STP.

Contents
Acknowledgement About the Authors Preface iv v ix

1 Continuous Process Improvement 1 1.1 Same as it ever was........................................................ 1 1.2 Does it have to be this way?........................................... 2 2 3 Six Sigma Dene 5 2.1 Introduction to Six Sigma.............................................. 5 2.2 Customer. ...................................................................... 7 2.3 Administrative............................................................... 8 2.4 Team Charter................................................................. 8 2.5 The Problem Statement. ............................................... 10 2.6 Tools............................................................................ 12 2.7 Problem Identification................................................. 14 2.8 Closure........................................................................ 17 Time to Measure 19 3.1 Measuring the software................................................ 20 3.2 Measuring the software process. ................................... 23 3.3 Conclusion.................................................................. 27

4 Analyze This! 29 4.1 Analyzing the Software Itself........................................ 30 4.2 Analyzing the Software Process.................................... 36 Root cause analysis...................................................... 37 4.3 Whats next?................................................................ 41 5 Implementing 43 5.1 Improving the Software Itself....................................... 43 5.2 Improving the Software Testing................................... 46 5.3 Improving the Software Development Process............. 49 5.4 Conclusion.................................................................. 53

Sav i n g S o f t wa r e w i th S i x S i g m a

Contents
6 Control in Software Development 55 6.1 Controls in Software Code........................................... 55 6.2 Controls in Software Development. .............................. 61 6.3 Controls in Software Development. .............................. 67 7 Final Words 69 7.1 What is Design for Six Sigma?..................................... 69 7.2 Potential Savings . ....................................................... 69 7.3 Potential Obstacles...................................................... 70 Not Invented Here....................................................... 70 Flavor of the Weekit is............................................... 71 Lack of Top-level Management Engagement................ 71 Six Sigma is for Manufacturing Only......................... 72 7.4 Try It,Youll Like it....................................................... 72 Bibliography 74

II

List of Figures
1.1 Off Target......................................................................... 2 1.2 Lessening Time Fighting Fires.......................................... 3 2.1 2.2 2.3 2.4 2.5 2.6 DMAIC Interactions. ....................................................... 10 System Hierarchy. ........................................................... 10 Failure Modes to Causes................................................. 11 IPO Diagram.................................................................. 12 Rayleigh plot.................................................................. 13 Bug Distribution............................................................. 17

3.1 Defect Count.................................................................. 21 3.2 Rayleigh plot.................................................................. 24 3.3 IPO Diagram.................................................................. 26 4.1 Six Sigma Flow............................................................... 29 4.2 Sample two-level factorial table with two responses. ....... 31 4.3 Double the rows, double the work. ................................. 32 4.4  Response surface methodologies make for better analysis.................................................. 33 4.5 Ishikawa Diagram........................................................... 38 5.1 Testing Pyramid.............................................................. 47 5.2 Test-Analyze and Fix...................................................... 49 6.1 Inspections..................................................................... 57 6.2 Weibull Distribution: shape=2, scale=40. ........................ 64 6.3 Severity Chart. ................................................................ 65

III

Sav i n g S o f t wa r e w i th S i x S i g m a

Acknowledgement
winmerge (compare) mikTeX (LaTeX compiler) TeXnicCenter (TeX-specic editor) zscreen (screen capture) vim (programmers editor)

IV

About the Authors


Kim H. (Vajramanas) Pries has four college degrees: B.A. History from the University of Texas at El Paso (UTEP), B.S. Metallurgical Engineer ing from UTEP, M.S. Metallurgical Engineering from UTEP, and M.S. in Metallurgical Engineering and Materials Science from Carnegie-Mellon Uni versity. In addition to the degrees, he has the following certifications: APICS Certied Production and Inventory Manager (CPIM) American Society for Quality (ASQ) Certied Reliability Engineer (CRE) Certied Quality Engineer (CQE) Certied Software Quality Engineer (CSQE) Certied Six Sigma Black Belt (CSSBB) Certied Manager of Quality/Operational Excellence (CMQ/OE) Certied Quality Auditor (CQA) Mr. Pries worked as a computer systems manager (IT), a software engi neer for an electrical utility, a scientic programmer on Defense contract, and for Stoneridge, Incorporated (SRI), he has worked for fteen years as: Software manager Engineering Services Manager Reliability Section Manager Product Integrity and Reliability Director

Sav i n g S o f t wa r e w i th S i x S i g m a

In addition to his other responsibilities, Mr. Pries provides Six Sigma train ing for both UTEP and SRI, cost reduction initatives for SRI. Additionally, in concert with Mr. Quigley, Mr. Pries is the co-founder and principal with Value Transformation, LLC, a training, testing, and product development consultancy. Mr. Pries is also a lay monk in the Soto tradition of Zen Buddhism and functions as an Ino for the Zen Center of Las Cruces while studying for the Soto Zen priesthood. Mr. Priess rst book as Six Sigma for the Next Millennium: A CSSBB Guidebook, revised as Six Sigma for the New Millennium: A CSSBB Guide book, Second Edition form ASQs Quality Press. For Taylor and Francis, Mr. Pries has worked with Jon Quigley to write Project Management of Complex and Embedded Systems and Scrum Project Management as well as this book. With Quigley, he has written well over thirty magazine ar ticles. He has also presented for the Society of Automotive Engineer, the Automotive Industry Action Group, marcus evans, and the Software Test and Performance Convention. Additionally, Mr. Pries is a principal of Value Transformation, a product development training and cost improvement organization. Mr. Pries is also a founding faculty member of Practical Project Management. Email Mr. Pries at kim.pries@valuetransform.com

VI

Jon M. Quigley has three college degrees, B.S. Electronic Engineering Technology from the University of North Carolina at Charlotte, Masters of Business Administration and a M.S. in Project Management from City University of Seattle. In addition to the degrees, he holds the following certications: Project Management Institute Project Management Professional (PMP) International Software Testing Qualifications Board (ISTQB) Certied Tester Foundation Level (CTFL) In addition to the degrees and certications, Mr. Quigley has secured six US patents over the years with another two in various stages at the US Patent Office, one of which is in the pre-grant stage. These patents range from human machine interfaces to telemetry systems and drivers aides:  US Patent Award 6,253,131 Steering wheel electronic interface  US Patent Award 6,130,487 Electronic interface and method for con necting the electrical systems of truck and trailer  US Patent Award 6,828,924 Integrated vehicle communications dis play (also a European patent) US Patent Award 6,718,906 Dual scale vehicle gauge  US Patent Award 7,512,477 Systems and Methods for Guiding Oper ators to Optimized Engine Operation  US Patent Award 7,629,878 Measuring Instrument Having Location Controlled Display Mr. Quigley has also won the Volvo-3P Technical Award in 2005 going on to win the Volvo Technology Award in 2006.

VII

Sav i n g S o f t wa r e w i th S i x S i g m a

Mr. Quigley has more than twenty years of product development ex perience, ranging from embedded hardware and software design through verication, project management and line management. Software Engineer Embedded Engineer (hardware) Test Engineer Project Manager Electrical and Electronic Systems Manager Verication and Test Manager Mr. Quigley is a principal of Value Transformation, a product develop ment training and cost improvement organization. Mr. Quigley is also a founding faculty member of Practical Project Management. Mr. Quigley is the co-author of the book Project Management of Complex and Embedded Systems and Scrum Project Management. Also with Kim H. Pries, he has written more then thirty magazine articles and presented at numerous prod uct development conferences (Software Test and Performance Convention) about various aspects of product development and project management. Mr. Quigley lives in Lexington, North Carolina. Email Mr. Quigley at jon.quigley@valuetransform.com. Additional information can be found at www.valuetransform.com.

VIII

Preface
Continuous process improvement is not as difficult as some would make it sound. It does require an understanding of key metrics, how to acquire this information and an attention to the details of what those metrics are telling you.

IX

Sav i n g S o f t wa r e w i th S i x S i g m a

Chapter 1
Continuous Process Improvement
1.1 Same as it ever was...
Chaos reigns supremedetermining root cause or the precipitating set of events is often nebulous. Yet it is not unheard of, for people to instinctively know the causing action for a particular problem. We have seen that a rapid, knee-jerk response to a situation often does not improve the outcome. Why? At one time, there was a belief, in spontaneous generation (also known as abiogenesis), the conversion of inanimate matter to life. This was a long held belief and not until more modern science, and the microscope, was this debunked. The belief was based at least in part upon the observation that dead animals, generated maggots. This belief was strong enough that it made it into at least one play by William Shakespeare, Hamlet. Hamlet: [Reads to Polonius] For if the sun breed maggots in a dead dog, being a god kissing carrion, Have you a daughter?1 Very few things can be understood with a cavalier approach, especially anything of any complexity. Even subtle interactions can have a significant impact on an outcome. A small mistake, assumption or error can lead to an inaccurate conclusion. The graphic below illustrates how a small misdirection in a selected direction, can result in missing the target or objective.

Sav i n g S o f t wa r e w i th S i x S i g m a

Figure 1.1: Off Target

1.2 Does it have to be this way?


We suggest that a methodical approach to understanding the sources or origins of a particular performance issue is required to determine appropriate actions to resolve the malperformance. Yet many times these discussions of improvement are cavalier or slapdash in approach and dont consider the entirety of the system. It may be surprising to find out how often the continuous improvement initiatives of an enterprise is a collection of ad hoc reactive activities without an overarching vision. These activities take the form of quickly responding to a seemingly ceaseless flamb du jour. Finding the real origins of an outcome often requires time and diligence and a system. The system we recommend is called Six Sigma and represents a bottom-line oriented quality approach with a highly quantitative focus. The goal is to move from forest fires to brush fires to no fire-fighting at all.

Chapter 1

Co n t i n u o u s P roce ss Im prov e m en t

Figure 1.2: Lessening Time Fighting Fires

Sav i n g S o f t wa r e w i th S i x S i g m a

Chapter Notes
1. Shakespeare, William, Easton Press. Shakespear the Tragedies Hamlet page 676

Chapter 2
Six Sigma Define
2.1 Introduction to Six Sigma
Most commonly, the Six Sigma approach is considered a collection of techniques for reducing quality variation in a product with a strong emphasis on statistical methods of analysis and some optimization techniques. While this viewpoint is often true, the Six Sigma approach makes use of some problem-solving tools that are as valid for software testing and performance as they are for hardware design and production lines. Define Spell out the problem (the scope of work) Indicate customer impact Measure Gather data about the problem or situation  Ensure the measurement system records meaningful results Analyze Take the data and recast it as information Use statistical tools when appropriate

Sav i n g S o f t wa r e w i th S i x S i g m a

Improve Optimize at best, simply improve at worst  Frequently uses an implementation and deployment of best practices Control  Put in procedure that provide for monitoring after improvement Use a feedback system to truly control the improvement The use of complex statistics is not a requirement for Six Sigma projects and never has been. The concepts of improvement and control are much more significant and, indeed, represent a hallmark of the Six Sigma approach. Figure 2.1: DMAIC Interactions

Another hallmark of the Six Sigma approach is the pragmatic approach to project selection, which is related to the effect of the project on the bottom line. This line of attack is particularly important when dealing with software since a software failure will exist in every copy of a given release and, sometimes, buried in past releases. Hence, the existence of a significant software issue, particularly product software (embedded), will sometimes force an industrial recall, especially if we are looking at a safetyrelated issue. This situation can be especially exacerbated by variation in customer or field configurations. We then expect delayed detection of field problems until the number of customers with the wayward software is quite large.

Ch a pt er 2

Six Sigma Define

In this book, we will show how powerful this simple problem-solving approach can be and demonstrate its relevance to software testing and performance. Our first chapter explains the define step in great detail.

2.2 Customer
In general, we develop software for customers and not for ourselves. Although the latter is not uncommon, the impact severity is typically less and defect containment is comparatively easy. When a software problem presents, often the best immediate response is to listen to the voice of the customer. This includes asking questions to identify what the problem is and is not. What, from the customers perspective, are the Critical to Quality (CTQ) attributes that are failingthese may not always be obvious software issues. In essence, we begin a diagnosis of the problem by listing all the known factors of the problem as reported by the customer (patient) in order of statistical probability (if known). The benefits of taking a systematic approach are: We can more unambiguously comprehend the situation We can develop a rational forecast of a probable behavior  We can eliminate any immediately regulatory violations (safety issues) with emergency action We can plan for an intervention for the situation  We can sometimes allow the customer to integrate the situation into their product with minimal dissatisfaction until a good reparation time arises.  We can determine future activities and actions based upon fact and not gut or hope. Sometimes a major issue will be the nebulous nature of customer reporting and logistics (getting data from customers customer). Regardless, we should solicit whatever customer data we can get they are experiencing the symptoms.

Sav i n g S o f t wa r e w i th S i x S i g m a

2.3 Administrative
Our Six Sigma Define approach has several components that we will describe. None of the documents needs to be a burden; however, they represent a systematic approach to planning and capturing the issue as it evolves between customer and supplier.

2.4 Team Charter


As in any project, the scope of the work has to be clear to achieve an objective. We accomplish this goal with a team charter. The team charter is a concise document used to summarize information for the solution team in the case where our problem is significant enough to require a team to dig for an answer. This information includes definition of measurable success criteria, including time (number of hours) or desired closure date. Identification of the appropriate team skills needed to define the problem is essential to solving the problem. The team charter will provide the names and roles of the team members, which can also include customer and supplier members; for example, a supplier could be a compiler or operating system manufacturer and a customer could be a downstream commercial buyer. Sometimes we are selling more of a commodity version of software, in which case our customer may simply be an aggregate of unhappy end users. We use human resources to get things done. Experience suggests that people who execute and deliver are frequently tapped for this sort of teamwork, thereby increasing their work load. Given the propensity for over-demand of key talent it is worth mentioning as a warning. The team leader will need to be aware of workload problems as she selects her team for problem resolution (assuming a full team is necessary). The team charter should also provide a general summary of the situation as it is known at the time we write the charter, which is most often the start of the problem-solving project. We will go into more detail a little later on about how to describe or define the situation. In order to solve our problem, we will also need resources: information, money, individuals, a place to work, perhaps some equipment, and special skills.
8

Ch a pt er 2

Six Sigma Define

Our charter will also include simple guidelines for team behavior and for meetings. Meeting information should include required and optional attendees, location, time, and technology needs (if we have a distributed team). Additionally, we should have an agenda and somebody to record minutes. We can include a contact list in the charter that will show email and phone numbers for team members. A preliminary plan is useful, but it does not have to be elephantine. The goal is to solve the problem, not write a burdensome document. Some charters may include operational definitions to ensure the team is literally on the same page with regard to the meaning of critical terms. The charter may also include a description of probable measurement needs, a time line, and a brief assessment of financial needs (although these may have been picked up in the resources section of the charter). Some more ambitious charter may spell out a goal, but in problem situation the goal may be so obvious as to need no further clarification. If we do create a goal, we will spell out what is to be accomplished, the measurable objective, and some kind of deadline. The charter may also contain the communications needs for the activity. This can also be handled within a project communications plan when the activity has a large distribution. Connecting communications needs to the responsibility assignments aides in team mapping. Distributed teams across geography or organizations--make this communications need even greater. Of course, we expect this level of documentation to apply with serious problems and large-scale customer issues. Routine problem-solving will not need a Six Sigma approach. Some people may question whether all this documentation is heavy-handed. Our experience has been with Department of Defense and commercial vehicle product software development and our involvement suggests the more documentation, the better, so long as all of it is relevant to scope, team communication, customer communication, resource allocation and basic problem-solving.
9

Sav i n g S o f t wa r e w i th S i x S i g m a

2.5 The Problem Statement


The problem statement itself, often part of the charter, should describe current behaviors of the software as well as the impact on the customer. This impact should include dollars at stake due to the software non-conformance. Situations of lesser significance like annoyances may not require a team effort or a charter to solve them, but more significant problems should maintain some kind of record for future use and lessons learned. It is critical that we define the problem scope. A good place to start is with Kiplings `six friends, namely: who, what, why, where, when, and how.1 We might also ask `how much? to determine how widespread the problem may be. We can also break the problem down by location if we are using product (embedded) software. An example would be Figure 2.2: Figure 2.2: DMAIC Interactions

We can also build taxonomies based on classes and objects, data structure types and functions. We use any or all of these hierarchical formats to help us isolate where the problem resides. In essence, we are showing where the problem is and, at the same time, discovering where it does not occur. By so doing, we can use a process of elimination to remove components and subsystems that are not relevant to the issue at hand. Our next step is to assess the failure modes. We can organize the failure modes in the following structure:

10

Ch a pt er 2

Six Sigma Define

1. Complete failure (a) Example: software doesnt initialize 2. Partial failure (a) Example: software initialize but resets 3. Aperiodic failure (a)  Example: software occasionally or intermittently fails for no apparent reason and antecedent causes are difficult to determine 4. Periodic failure (a) Example: software resets regularly 5. Failure over time (a)  Example: software becomes corrupt over some quantum of time 6. Too much (a) Example: unspecified features cause problems 7. Too little (a)  Example: software does not apply for some the required capabilities We will use all these failure modes when we make use of one of the defect taxonomies described later in this chapter. We also need to understand that failure modes are not the equivalent of causes. Figure 4 shows the comparison: Figure 2.3: Failure Modes to Causes

11

Sav i n g S o f t wa r e w i th S i x S i g m a

2.6 Tools
The Six Sigma approach also provides us with some easy-touse tools that help during the Define phase of the corrective action. One of these tools is the supplier-input-process-outputcustomer (SIPOC) diagram (see Figure 1). A simpler, useful version of the SIPOC diagram is the Input-process-output diagram or IPO. The left-hand side arrows represent factors that enter a process and the right-hand side arrows represent responses that exit a process. The IPOs can be strung together analogously with a flowchart, with some outputs become inputs to other processes. Please note that although we can model software with these diagrams, they are mostly used to model processes. This process perspective can be helpful if the problem originates from (or lack of) a specific development process. In the case of software performance, we would take a look at the inputs that stimulate a function and observe the reaction of the responses to systematic modifications of the factors. Although, they appear simple, the development of these models takes substantial thought in order to eliminate redundant or meaningless inputs and outputs. Figure 2.4: IPO Diagram

12

Ch a pt er 2

Six Sigma Define

Rayleigh plots allow us to use statistical analysis to determine if we are ready for release. These plots can also be applied after release to model issues discovered in the field. Basically, they are Weibull plots with a shape factor of two, which provides for the distinctive front-loaded appearance of the plot; for example, Figure 2.5 below. Rayleigh plots can also be used to model level of effort for work load analyses. We should also remember that any Weibull plot is, by definition, also a three-parameter extreme value type III plot. Figure 2.5: Rayleigh plot

A time series or run chart can be useful to see if problems especially intermittent problemsarise after a particular time. This situation may be indicative of the effect of a new noise condition or may indicate a deficient release of the software. It is also possible to perform Weibull analysis on this kind of situation to determine the probability of more issues, assuming they are related to time in the field (post-release duration). When we enter into the measurement and analysis phases, Six Sigma practitioners frequently use a graphical device called an Ishikawa diagram (also known as a fishbone diagram or a cause-and-effect diagram). Sometimes, the Ishikawa diagram makes sense during the Define phase when we need some systematic qualitative analyses regarding the problem.
13

Sav i n g S o f t wa r e w i th S i x S i g m a

2.7 Problem Identification


When discovering software issues, we can classify them more systematically by using an existing taxonomy for software defects. Taxonomy is nearly always a tree structure that represents a cognitive hierarchy, the most famous of which is probably the Linnaean taxonomy of living things (kingdom, phylum, class, order, family, genus, species). Two sources of preexisting taxonomies are the IEEE standard and the one defined by Boris Beizer in Software Testing Techniques. The IEEE approach is documented inIEEE Std 1044-1993, IEEE Standard Classification for Software Anomalies Std 1004-1993 and a guide called IEEE Guide to Classification for Software Anomalies Std 1044.1-1995. The IEEE approach is composed of multiple tables that represent a more relational approach to anomaly classification rather than a true taxonomy. Additionally, the standard propose a process composed of Recognition Investigation Action Disposition To each of these are the tasks of Recording Classifying Identifying impact Although the IEEE approach is less of a tree and more of a tabular design, it represents a systematic, rational approach to knowledge management of software defects. Boris Beizer presents his solution in Software Testing Techniques, 2nd Edition (1990)Beizer, Boris, Software Testing Techniques, Second Edition(New York, NY: Van Nostrand Reinhold, 1990) p. 304,1 where he provides the expected taxonomic

14

Ch a pt er 2

Six Sigma Define

hierarchy in an appendix. He also provides a broad overview of statistics related to defect classification. Beizer created his own classification system because he felt the IEEE approach was not sufficiently complete. The taxonomy provides a numerical hierarchy for bugs and a description of each of the many line items. Beizers top-level categories include: 1. Functional requirements (a) Logical (b) Completeness (c) Presentation, documentation (d) Changes 2. Functionality as implemented (a) Correctness (b) Completeness (c) Domain bugs 3. Structural bugs (a) Control flow and sequencing (b) Processing 4. Data (a) Definition, structure, declaration (b) Access and handling 5. Implementation (a) Coding and typographical (b) Standards violation (c) Documentation 6. Integration (a) Internal interfaces (b) External interfaces and timing

15

Sav i n g S o f t wa r e w i th S i x S i g m a

7. System, software architecture (a) O/S call, use bug (b) Software architecture (c) Recovery and accountability (d) Performance (e) Incorrect diagnostic exception (f) Partitions, overlays (g) Sysgen/environment 8. Test definition of execution (a) Test design bugs (b) Test execution bugs (c) Test documentation (d) Test case completeness (e) Other test design/execution bugs This outline provides a quick flavor of Beizers taxonomy. While some people have questioned Beizers relevance to modern software testing approaches, the taxonomy represents a convenient format for setting up the measure phase of the Six Sigma project. This taxonomy is not just a rhetorical or theoretical exercise. It is possible to use this list to understand the origin of the software problems. A software organization can use this list to categorize the problems found in the field from a historical perspective, then use this to extrapolate future conditions. Figure 2.6, is a distribution of faults along this taxonomy from data in Beizers study (Beizer, 1990).2 If an organizations distribution of the problems is known, it is possible to use this information in evaluating the reported problems. More importantly, the list and historical information will provide clues as to any underlying systemic issues in the software or embedded development process that may need addressed.

16

Ch a pt er 2

Six Sigma Define

Figure 2.6: Bug Distribution

2.8 Closure
When the `Define phase is complete, we should be ready for the `Measure phase of the problem-solving algorithm. If we have done our work well, we should know the items that we need to measure in this next phase. Sometimes we learn so much from the data-gathering during `Measure that we will iterate through `Define again and refine the problem statement as well as secure any specialized resources. We should also understand the impact of the software anomaly on both our own development/testing teams and those of the customer. In many cases, it will be possible to put a monetary value on the crisis. If we are repairing a specific software failure, the monetary approach is easy. If we are mutating the software development process itself, we will be faced with the issue that afflicts many process improvement effortsthe inability to assess the cost of a suboptimal process. We should also have a schedule for fixes, whether it is for software itself or for the development process. Additionally, we should also produce a preliminary schedule for testing of the change to ensure timely provision of resources.

17

Sav i n g S o f t wa r e w i th S i x S i g m a

Chapter Notes
1. Kipling, The Elephants Child. The six friends are who, what, when, where, how, why. 2. B  eizer, Boris, Software Testing Techniques, Second Edition (New York, NY: Van Nostrand Reinhold, 1990) p. 304

18

Chapter 3
Time to Measure
In the first chapter, we discussed the Define phase of the Six Sigma quality management process as it relates to software testing and performance. We explained how clearly defining the problem and explicitly describing what needs to be accomplished allows us to set the scope for project activities so well know how and when weve hit the targetthat is, how weve made the correction appropriately. Here, well assume weve already defined a problem issue, whether code-specific or process-specific (process-specific issues are often related to software productivity and failure to meet deadlines or budget targets), and well delve into the second piece of the Six Sigma puzzle: Measure. (In upcoming chapters, well address the next three aspects: Analyze, Improve and Control.) Before doing any measuring, you must: Identify key quality attributes to be measured Devise a measurement plan Understand the limitations of the measurement system Then, measure We will break our discussion into two components, measuring the software itself and measuring the software process.

19

Sav i n g S o f t wa r e w i th S i x S i g m a

3.1 Measuring the software


Some of the common metrics used with software are the following: Lines of code Function points Defect counts Defect types Complexity Lines-of-code is an objective measure of software size. It is important that the developers understand that higher-level languages like C++ do not translate one-for-one into the executable code the machine will use. This situation is also the case when we use dynamic languages like Perl, Ruby, and Python. Java converts into byte-code which is, in turn, converted into executable code. For these reasons, lines-of-code can be a problematic metric. On the positive side, if all development work is being done with one language, the metric is simple to generate and provides a coarse level of measurement. Function points are an alternative to lines-of-code. When using the function point approach, we work with the software specification and attempt to estimate the complexity of the product using a special set of rules. Function points present problems because the estimation process itself is highly subjective. Sometimes, an organization will use some wellknown numbers and backfire the function point value from the lines-of-code; so, why not use the lines-of-code in the first place? The only reason we might use function points in this situation occurs when we are comparing the potential complexity of a product to some standard and we have no existing code. Defect counts are easy to accumulate. The counts are what we call categorical data, which means the Six Sigma practitioner will have to use the statistics of categorical data to make any meaningful statistical analysis. Luckily, this kind of analysis doesnt have to be that difficult: we can use a Pareto chart to display the defects counts by function; for example:
20

Ch a pt er 3

T i m e to M ea s u r e

Figure 3.1: Defect Count

Using our count data and the Pareto chartwhich is organized by count from left-to-right, highest-to lowestwe can easily see that the `read_analog function is a leading source of software defects. By measuring our work using this approach, we can focus our attention on functions that show frequent problems. This does not mean that we will ignore the other functions but, rather, we know we have serious issues with the functions on the left side of the graph. We can use the same approach with software developers to see if we have a human resources challenge with certain individuals. They may require more training, experience, or education or they may be in the wrong position. Using defect types as a metric allows us to check our software to see if we are repeating certain kinds of defects regularly and may provide some direction to our testing group during the analyze phase of our Six Sigma project. A few examples of defect types are as follows: Boundary issues With embedded code, this can be byte boundaries  With other code, we may see array/matrix out-of-bounds issues
21

Sav i n g S o f t wa r e w i th S i x S i g m a

Logic issues  Check for odd expressions, especially negative tests (using not) Bizarre case structures Syntax checking Verify compiler is functioning as expected Data issues Illegal inputs  Incorrect casting to convert from one data type to another One of the benefits of measuring the quantity of each data type is that we can again apply Pareto analysis to the results and use the information during our Six Sigma analyze phase. Also, we can either construct a taxonomy of types or we can use a technical standard such as IEEE 1044-1993. A taxonomy will represent the types in a hierarchical tree format. Software complexity metrics attempt to represent the intrinsic complexity of the software by making calculations. Probably the most well-known approach is the cyclomatic complexity calculation of Thomas McCabe. McCabe initially used a graph theoretical approach, a flow graph, to estimate complexity. The formula for such an approach is Cyclomatic Complexity = E ? N + P, where E = the count of edges of the graph N = the count of nodes of the graph P = the count of connected components Luckily, we dont have to develop the graphs and we dont have to calculate the formula by ourselves. We can use tools such as JavaNCSS for Java or Saikuro for Ruby. These tools will go through the code and make the calculations based on the number of if and case statements as well as other flow-altering code and present the developer with the value. The cyclomatic complexity is a completely objective value. While some people may question the meaning of cyclomatic complexity, the value generated is, at a minimum, suggestive of areas of code that may merit closer attention.
22

Ch a pt er 3

T i m e to M ea s u r e

3.2 Measuring the software process


We can also use our measurement tools to take a look at the development process itself, for example: Project status  Budget  Quality (addressed in the first section of this book)  Delivery  Compliance to international standards  Coding standards  Factor/response analysis When looking at a portfolio of software projects and assessing them over time, we can put the status of the software projects into the following categories: Project status Ahead of budget, schedule On budget, schedule Suspended (may resume) Stopped (candidate for aborted) Unstarted Completed Unknown Aborted (completely dead) Zombie (similar to unknown and still consuming resources) Some of theselike `unknownrepresent serious problems with the process. Basically, the simplest approach would be to build a table of the various development projects in the first column and then list status in the rest of the columns in some convenient increment of time.
23

Sav i n g S o f t wa r e w i th S i x S i g m a

Project budgets are an obvious metric for the development process and fit into a few of the categories we just discussed. The highest cost frequently comes from the payroll for the developers. During a development we might wish to estimate our level of effort so we can put a value on the software engineering we are proposing to do. One way to do this would be to model our efforts using a Rayleigh distribution; for example: Figure 3.2: Rayleigh plot

What we are trying to show with this plot is how we can measure our actual effort (e.g., hours of labor) against the probability distribution function. Front-loading is desirable in order to avoid the `student effectnamed after the tendency of students to put all their effort into the night before the test. Another interesting feature of this Rayleigh distribution is that we can also measure the arrival of defects through the duration of the project and compare our results against the model. Usually real data is noisy (some elements of randomness) but we can still use this approach to potentially provide a basis for releasing the software product as our defect arrival rate slows down and we move into the right tail of the distribution.
24

Ch a pt er 3

T i m e to M ea s u r e

During our software process, we want to measure delivery of functions, modules, and other configuration items to see if we are meeting our schedules. Instead of using the states we defined before, we actually measure our historical progress against development timelines. Tools like Microsoft Project allow the project manager to baseline the initial plan and then produce a secondary plot of the actual dates superimposed on the first timeline. We can to assess the distribution type, the means, the variance, and any other statistical values that would allow us to model the software development project. For example, we could use a tool like @Risk from Palisade Corporation in concert with Microsoft Project to run Monte Carlo simulations of the project. The Monte Carlo approach will generate random values from a given probability distribution; that is, we are using the probability distribution we measured to provide us with a model for each delivery. In so doing, we can provide our software project managers with a method for simulating the delivery times of an entire project or portions of the project. Another method of process improvement is to drive the development team to compliance with an international standard. One such standard is ISO/IEC 15504 in nine parts. ISO/IEC 15504 is a direct descendant of the Software Process Improvement and Capability Evaluation (SPICE) initiative started by European organization in the 1990s. SPICE, in turn, is a variation on the maturity models developed by the Software Engineering Institute in the 1980s: the Capability Maturity Model (CMM) and the Capability Maturity Model Integrated (CMMI). All of these approaches specify a stepwise approach to increasing the capability of software organizations. Best practices are defined and documents for use by auditors have been published by all of these organizations. However, no software organization has to wait for a formal audit in order to establish the practices contained in these standards. We can also measure compliance with coding standards such as MISRA C. MISRA is the Motor Industry Software Reliability Association. They are one group who specified best practices in the C language for embedded systems in the automotive industry.
25

Sav i n g S o f t wa r e w i th S i x S i g m a

An organization does not have to purchase a coding standard to develop one. With enough product history, the organization can develop their own coding standard. The standard developers can refer to the software metrics, especially the taxonomy of failure types and the typical functions that become problem children. One of the favorite tools in the Six Sigma tool box is the IPO diagram: Figure 3.3: IPO Diagram

The reason we use this diagram is provide a model for measuring and correlating factors (inputs) against their effects (outputs). For example, we might use this approach to try and discern which factors are providing our developers with fewer defects: they could be using static analyzers, dynamic analyzers, debuggers, oscilloscopes, logic analyzers, and other tools. We would then conduct experiments to provide ourselves with a meaningful basis for recommending one tool or family of tools over another. Some readers may recognize this approach as Design of Experiments. The downside to this approach is that we usually need special software in order to perform the analysis. Some choices of support software are:
26

Ch a pt er 3

T i m e to M ea s u r e

Minitab Design Expert DOE++ Nutek-4 For something as complex as regression analysis, we can use a free statistical language called R to perform our calculations. R is command-line driven, although a graphical user interface called Rcmdr exists to help ease the shock of using an old-fashioned command line. R even has an interface to Microsoft Excel that allows R to be used as a server to Excel as a client. Designed experiments and regression are powerful tools for analyzing processes.

3.3 Conclusion
As we have seen in this chapter, we can apply the Six Sigma Measure approach to software directly and we can also use it for the software development process. When Six Sigma tools are applied to a process, we usually call it transactional Six Sigma because we are dealing with people. From measure we will move on to analyze in our next chapter and show our readers what we do with the factors we have measured. Do not get frustrated by thinking you have to use statistics as part of the project. In many cases, using Pareto charts and other simple tools are more than enough to provide direction for future analyses and improvements. Sometimes, simply measuring a previously unknown factor reveals the solution without much further effort on the part of the practitioner.

27

Sav i n g S o f t wa r e w i th S i x S i g m a

28

Chapter 4
Analyze This
In previous issues weve discussed the first two phases of the Six Sigma process: Define and Measure. Now its time to address phase 3: Analyze. Six Sigmas five easy pieces make it look like every project is a simple linear progression but, often, this is not the case. We find that projects tend to cycle (see figure 4.1) through the Define, Measure and Analyze stages a few times before settling down. Then, project goals become more well-defined. Figure 4.1: Six Sigma Flow

As we enter the Analyze phase, we find that Six Sigma practitioners frequently sound like they are speaking a new language. For example, in the Six Sigma approach, we often refer to factors and responses. A factor is an input, stimulus or cause; a response is an output, behavior or effect.
29

Sav i n g S o f t wa r e w i th S i x S i g m a

Mathematically, wed say the result y is a function of various subscripted x values (the inputs to the function). Hence, the y is the response and the x values are factors. The factor/ response approach maps nicely to software and hardware inputs and outputs, particularly in black-box testing.

4.1 Analyzing the Software Itself


We can also look at factors and responses with the white-box testing approach by looking at argument lists and functional returns, or we can measure and analyze Web site behavior that responds to human input. Combinatorial testing tries to systematize our approach to analyzing problems by using various mathematical approaches (combinatorial is a mathematical term that refers to the types of combinations of numerical valueswe can use this to our advantage when testing). One of the most common combinatorial approaches is pairwise testing. Pairwise testing involves testings all possible discrete combinations of inputs for each pair. James Bachs program AllPairs (written in Perl) automates this process when fed a table of the inputs. While this program is not optimally efficient at this activity, it is close and the price is right (free!). The Telcordia Automatic Efficient Test Generator can produce an optimal set of pairs, but it costs money (see http://aetgweb.argreenhouse.com/). Pairwise testing can significantly reduce the number of tests required, and the ability to automatically generate tables of pairs is attractive. The problem with this approach lies in the conceptwere looking only at pairs, which means were most likely to find only pairwise problems. Issues that are higher order or that have significant interactions will be completely missed by this approach. But we can reduce our risk by combining this approach with other techniques. Three-wise testing takes the pairwise concept to another level, examining the effect of groups of three (triplets or trinaries). This approach can be efficient but suffers from the same defects
30

Ch a pt er 4

An a ly z e T h is !

as pairwise testing. Also, tool support for this approach is largely lacking. Fractional factorial testing is an approach borrowed directly from manufacturing-style Six Sigma and uses tables much like those used for Designed Experiments. When using Designed Experiments, we execute our tests to a recipe that is usually based on a special kind of matrix called an orthogonal array. Fractional in this context connotes the fact that something is missing. In the case of these experiments, we increase the efficiency of our testing by reducing the number of possibilities of what we can detect. Some fractional tables are so fractional, they show no interactions (see Table 4.2 for an example). Figure 4.2: Sample two-level factorial table with two responses

The 1 is a high value for the input and the -1 is a low value for the input; hence, this table is a two-level factorial with two responseswe measure the responses as we execute each row of factors. For testing and analysis purposes, wed compare our measured responses against our test specification or the software requirements specification. Even with this approach we may still miss some interactions; however, overall it is highly efficient. Full factorial testing allows us to see all effects, including all interactions. It is still efficient, although not as efficient as the fractional factorialwe always have some kind of trade-off during analysis. (Figure 4.3) shows a table using the same two levels as our previous example.)

31

Sav i n g S o f t wa r e w i th S i x S i g m a

Notice that we have twice the number of rows, which means twice the amount of work. This approach is what we must do to see all interactions and it may require many runs. Nonetheless, the full factorial approach is significantly more effective than the one-factor-at-a-time approach. Response surface methodologies (RSM) allow for even greater analytical results, but at a cost. We now have 30 runs for four factors. However, the responses can be modeled as surfaces, which allow us to use a search algorithm to find the sweet spot. This approach can be useful, particularly with timing issues. Figure 4.3: Double the rows, double the work

The RSM approach can generate plane, quadratic and cubic surfaces and, as we have seen, it is much more involved than full factorial. Using a reliability analysis tool like Weibull plots can open another dimension for analysis. The Weibull distribution is part of a family of asymmetric probability distributions (the so-called extreme value distributions). This approach is excellent when a problem happens over time. The shape of this three-parameter distribution can sometimes provide an idea of what kind of failure we are seeing. We also use a special version of the Weibull distribution with a shape factor to two, called the Rayleigh distribution, to assess defect quantities over time. The Rayleigh
32

Ch a pt er 4

An a ly z e T h is !

distribution can provide a statistical rationale for releasing the product. Another benefit of the Weibull family is that it doesnt need more than a handful of data points to yield some information; of course, the more data, the higher the confidence. We can also use a quality tool such as control charts, which are effectively run charts with control limits calculated from the data. The charts are a good choice for time between events analyses. These are used to help discern whether we are seeing random variation or special events. In addition to quality and reliability tools from the Six Sigma toolbox, we have other tools for analysis; for example: Figure 4.4: Response surface methodologies make for better analysis

33

Sav i n g S o f t wa r e w i th S i x S i g m a

In circuit emulators Software emulators Logic analyzers Debuggers Background debuggers In-circuit emulators are part hardware, part software devices used to emulate a specific processor. ICEs are common when developing embedded software, such as the software in a cell phone or microwave oven. They interface directly to live hardware (interaction with external world stimuli) and often have an integrated debugger. They can trace the impact of external work on internal product (software) execution. Additionally, they allow for viewing variables and interactions in the software while under execution by using break points, they can easily view interrupt handling and they provide stack status (especially when we have overflow conditions). These can be expensive and the interface logistics between the ICE and the target system can be challenging. Software emulators are an alternative to ICEs. These software emulators generally run on a PC as an ordinary executable program. They emulate other microprocessors in software the hardware component is usually only the PC. Although they tend to be inexpensive, the quality is only as good as the programming and they are generally not available for the most recent microprocessors. They also have limitations with the replication of hardware influences and specific external events. A logic analyzer is an instrument to display signals in a digital circuit so speedy as to be unobservable. The idea is to capture signals and then present these results to the user in an intelligible format for people. If the analyst is a hardcore hardware person these tools can often be used to trap data in systems that exceed the capacity of an oscilloscope. With add-on modules the logic analyzer can convert this data into:

34

Ch a pt er 4

An a ly z e T h is !

Timing diagrams (when signal high or low) Protocol decodes (example, J1939) State machine traces (state changes are trapped) Assembly language, or A pseudo source-code rendition (no real variable names) Debuggers and background debuggers allow for step-bystep execution of programs or running programs until they reach a break point. In many cases they are integrated with in-circuit emulators. The background debuggers have much of the functionality of in-circuit debuggers, but theyre usually significantly cheaper in price, since much of the functionality is built into the microprocessor. The most common use is for embedded programming debugging. Background debuggers are well-known with certain Freescale microprocessors. Other tools for software analysis include: Instrumentation Profilers Load generators/analyzers Usability profiles Dug tracking tool Instrumentation is the integration of test code into the code under development at compile, to essentially allow the program to monitor and report on itself. This integrated code provides feedback to the developer on the code performance, and is the precursor to profilers. In the olden days, the coder would put instructions directly into the code to provide information on timers and counters. This information would be output in some manner that would allow the information to be readily understood and evaluated.

35

Sav i n g S o f t wa r e w i th S i x S i g m a

Profilers are often used to capture timing and provide analysis of software performance by module or function. Profilers provide insight into the dynamic performance of the software, which makes specific code optimization possible. They can be written by the developer or purchased. The NetBeans integrated development environment contains the NetBeans profiler for Java. We use them to help determine which modules or functions are consuming the most resources, particularly when a performance issue is making the product unacceptable to a customer. The current user base has little patience with slow Web sites (as you no doubt know!). Load generators and analyzers are often used with networks and Web sites. The load generator simulates an increasing multitude of users accessing site. The analyzer examines response times and provides a report. One item for discovery is the point at which the site demonstrates a form of avalanche breakdown and effectively ceases to function. Usability profiles often use test subjects to try out the Web site or application, sometimes with minimal documentation to see how intuitive the software really is. These profiles are largely qualitative, although some tracking of time spent on certain Web pages and screen areas may be collected. Finally, we can use bug-tracking tools. Several open-source and free products are available to do this, including: Bugzilla JitterBug RT Request Tracker TRAC

4.2 Analyzing the Software Process


The Six Sigma methods above are used primarily to find defects in code. Now lets explore options for analyzing problems once the software is in production. We can use many of the same methods we use during development when working with legacy code. We can also analyze the development process itself.
36

Ch a pt er 4

An a ly z e T h is !

Root Cause Analysis Root cause analysis is a technique for diving into the problem and can easily be generalized to software challenges or anything else. Practitioners will often use a well-defined approach such as the eight disciplines to work to the root cause. The eight disciplines are: D1: Create a problem-solving team. This team is often composed of subject matter experts determined by the nature of the problem-solving activity. D2: Define the problem. Just as with the overall Six Sigma approach, we will control the scope of our analysis. This section often contains who, what, when, where, why, how and how much, as well as some clarification about when and where the issue does not happen! D3: Implement and verify interim containment actions (ICAs). The idea of containment is used to control flow of product to a customer. In the case of software, we might recommend continuing to use the previous release due to issues with the new one. If the software is parameter-configurable via production systems, we might choose to suppress this feature (via parameter configuration). Either way, this step requires that we verify containment is working. D4: Identify and verify root causes. During this discipline we use whatever means we have to dig into the problem. For software, we might do traces, in-circuit emulation or debugging activities. Frequently, timing problems are particularly difficult to solve. Again, notice the accent on verification of the root cause. D5: Choose and verify permanent corrective actions (PCAs). At this point, we use our knowledge to create a correction that cannot be undone. D6: Implement and validate PCAs. We take our new corrective action(s), implement and validate. We validate when we test against real customer needs instead of simply meeting the requirements specification. Our fix may require approval from the customer before we proceed. D7: Prevent recurrence. We move to institutionalize the correction, particularly if it is a successful procedural change. With software, we should have in place a defect prevention program that encourages learning among the developers. D8: Congratulate the team. This discipline is often forgotten during the communal sigh of relief we finally resolve the issue.
37

Sav i n g S o f t wa r e w i th S i x S i g m a

Any root cause approach makes some assumptions. One of the most intractable is the very concept of a single root cause. Often, real problems are not so simple that we can dive in and discover one problem that, when corrected, fixes everything. More sophisticated versions of the root cause approach assume that we have condition-making activities as well as a precipitating event or events. We suspect the simple approach is incredibly naive for all but the simplest problems (one example is the Five-Why approach, which continues to ask ``why? until a root cause is reacheda great concept but a poor methodology). Another tool often used during problem-solving is the Ishikawa diagram (also called the cause-and-effect diagram or the fishbone diagramsee Figure 4.5 for an example of a traditional Ishikawa diagram). Figure 4.5: Ishikawa Diagram

Lets look at each of the major fishbones from a software point-of-view: We can improve on the traditional diagram by making some changes to the ``bone headings.

38

Ch a pt er 4

An a ly z e T h is !

Materials Incorrect specification document Errant configuration management Wrong version checked out of version control system Incorrect protocol Machine / Equipment Defective PC Network hardware issue Routers not functioning correctly Damaged network line Untested material Environment Disorderly workplace Job design or layout of work less than acceptable Surfaces poorly maintained Physical demands of the task  Forces of nature (e.g., lightning drops an unbacked up system...yes, we know, nobody else has ever experienced this problem!) Management No or poor management involvement Inattention to task Task hazards uncontrolled Other (horseplay, inattention to details...) Stress demands (deadlines, late deliveries...) Lack of process (Lack of organized process??)
39

Sav i n g S o f t wa r e w i th S i x S i g m a

Method No or poor procedures SOP not followed Poor communication Management system Training or education lacking Poor employee engagement Poor hazard recognition Previously identified hazards not eliminated Other lists of categories exist for the Ishikawa diagram: 4S is Suppliers, Systems, Surroundings and Skills; 8P is Procedures, Policies, Place, Product, People, Processes, Price and Promotion; the McKinsey 7S is Shared vision and values, Staff, Style, Skills, Structure, Systems and Strategy. Of course, problem-solvers dont have to use the same letter or even any of the above examples. These merely serve as memory joggers when going through the process of adding smaller bones to larger bones on the diagram. The primary defect of the Ishikawa diagram is that it is qualitative and only as good as the work put into it; the primary asset of the diagram lies in its simplicity. Another approach to the analysis of the software process involves value stream mapping, where we assess how much of our time is actually spent doing a value-added activity. The standard tools for value stream mapping are strongly oriented toward manufacturing enterprises rather than software development teams. However, our value stream map can be as simple as using ANSI flowcharting or UML action diagrams with two swim lanes: value-added and not value-added. Typically, appraisal activities such as inspections, reviews and testing are considered to be not value-added. We suspect this point is arguable with software development. We can analyze our process for the seven process wastes:
40

Ch a pt er 4

An a ly z e T h is !

Overproduction. The software analog is gold-plating the software, an activity in which we add features we think the client will want. Waiting. We could be waiting for other code to integrate, for test results, for anything (we are not doing productive work here!). Transporting. Perhaps we need to send our code to the test group. Inappropriate processing. We spend a lot of money on special software development tools without considering the most important tool of all, the brain of the software developer. Unnecessary inventory. In the case of software, were talking about work in process building up. Unnecessary/Excess motion. The developers tools are not readily available or we dont have enough to go around. Defects. No discussion necessary.

4.3 Whats Next?


What do we do with all the data weve defined, measured and, now, analyzed? If weve performed our analysis with appropriate diligence, we should have ample information to proceed to the next phase of the Six Sigma approachthe Improve phase. This approach does not have to be heavyhandedin some cases, the entire process may take only a few days to complete. As software problem-solvers, we can tailor the Six Sigma problemsolving algorithm to suit our needs.

41

Sav i n g S o f t wa r e w i th S i x S i g m a

42

Chapter 5
Implementing
We can improve software development in two primary ways: we can improve the software itself (and, possibly, the testing) or we can improve the development process. We have already covered the preceding three Six Sigma Phases (Define, Measure, and Analyze) in previous chapters. Now we take what we have learned and use that information to improve our software development.

5.1 Improving the Software Itself


Training software developers is one obvious method for improving software development. All people may be created equal, but not all programmers are equal. However, simply saying training isnt really good enough. We need to consider what our measurement and analysis phases have told us about our software issues. One typical suite of training classes might include the following for a hypothetical embedded software development group (Our-Department):  Introduction to Our Department coding practices and standard for the C language Our Department specific software handling procedures Use of static testers Use of dynamic testers In circuit emulation Pitfalls in C and how to avoid them

43

Sav i n g S o f t wa r e w i th S i x S i g m a

Memory mapping the processor Lessons learned review Design architectures We call it training but, to some extent, we are really talking about indoctrination into our organizations practices for software development. Some of these practices will fall under the heading of lessons learnedwhere we have captured failures from the past as well as solutions to those failures. It does not do much good to critique failures and understand them if there is no learning from this review. Yet, in our experience, this is often what really happens. Frequent reviews of our lessons learned file or database by the entire team is not wasted time. We have seen what we call the stupidity loopa phenomenon often tied into the turnover rate of developers, especially with senior developers. We can detect the duration of the stupidity loop by the regularity with which we see the same mistakes begin to recur. Hence, the reviews of the lessons learned file allow our entire team to learn from the past without necessarily having all the original players present. This important tool can be quite difficult to implement well. While it is easy to list and document the lesson learned, it is not so easy to put this information into a format that makes for easy searching and retrieval. It is the retrieval that is important. We have seen lessons learned documented in word processors, spreadsheets, databases, and free form storage (for example, Microsoft OneNote). Even with a database, we need to have a good system of search categories and terms or nobody will be able to retrieve the information promptly. If the work to review the lessons becomes a burden, the task will starve away and we will have lost an opportunity to profit from past blunders. If we are using textual information, we can use search engines like Google Desktop to help us find information; an older approach using ASCII files would involve the use of the grep tool common on UNIX-types of operating systems. Once retrieval becomes efficient, we have the beginnings of defect prevention program. We dont think it is inappropriate to make the use of the lessons learned database a part of the software engineers performance evaluation.
44

Ch a pt er 5

I m pl e m e n ti n g

In a sense, while we call it training, what we are really talking about is a continually learning organization SENGE, P.M. 1990. The Fifth Discipline. London: Century Business.1 This is more than learning from the failures. This includes boundary spanning for best practices, which is not necessarily a euphemism for procuring the best practices of other organizations. We may also be able affect the software development process by benchmarking our own organizational practices against those of a well-reputed software house with who we are not in competition. The benchmarking exercise should provide ideas for testing in our own organization (note: we test because many times, the ideas of other organizations are not simply dropin practices). For developers not in the embedded software arena, the same kind of approach applies. We want to capture what we think the developers need to know and we want to indoctrinate the team regarding known development issues that have occurred before. This indoctrination includes instilling the organizations risk philosophy and extends to why some things are done the way they are done; in short, we describe our practice and how to perform them as well as the rationale for doing it that way. The approach applies regardless of whether we are looking at web development, enterprise applications, or database management. In addition to indoctrination/training, we can also reuse known good code as long as we spell out the rules for doing so. In fact, reuse of code (and pieces of hardware for that matter) can become another topic for the training part of software improvement. For code reuse to work well, the library modules we are going to reuse must be well-documented and, well, modular. In some cases, these components can exist as executable code in the form of dynamic link libraries; in other cases, we may have to incorporate source code into our new source code (definitely less desirable since this approach can lead to degradation if the developers touch the library code). As we already know, some languages lend themselves to modularity better than others. Encapsulation (or decoupling) is often a language-based approach to implementing good practices, although we can accomplish decoupling by focusing our attention on older languages that do not explicitly support
45

Sav i n g S o f t wa r e w i th S i x S i g m a

it. If we can encapsulate a software object such that access to its methods is well-controlled, we increase the likelihood of producing a truly robust module. We can practice encapsulation with any language, although the truly object-oriented languages help us to enforce these practices. We also want to be sure that our code will check for simple flaws like overflows and underflowswe cant expect the run-time produced by the compiler/linker to do this for us. With embedded code, we must make sure we are not burdening the microcontroller excessively while maintaining the safety of our code (a sad truth sometimes). The more risky the result of an over- or under-flow, the more important the practice of deliberate protection will be. We have seen substantial protection written into code that controlled a fighter aircraft, since the results of a failure were likely to be catastrophic. The approach worked well.

5.2 Improving the Software Testing


We recommend a multimodal approach to testing, particularly if our measurement and analysis phases suggest that we are letting defects through to our customers. In general, in the embedded world, we use the following approaches most often: Compliance testing: testing to the requirements specification  Combinatorial testing: wherein we stimulate the inputs of the system and observe the outputs of the system using a recipe that exercises all inputs in a particular pattern (for example, pairwise testing tests all pairs)  Stochastic (or exploratory) testing: we allow the test engineer to use their gut and knowledge of the product to explore the behavioral envelope of the software; sometimes, we can use tools to help with this approach (for example, a genetic algorithm)  Severe testing: we challenge the software by doing horrible things to the data bus, the input values, the hardware anything goes with this kind of testing. This testing can exceed the design limits to ferret out marginal performance issues. In fact, it is not out of line for the testing to go all the
46

Ch a pt er 5

I m pl e m e n ti n g

way to the destruct limitby so doing, we establish the true design limit, the failure limit, and the destruct limit, thus characterizing the product. Figure 5.1: Testing Pyramid

Occasionally, an approach like severe testing will produce specious failures; that is, failure we would never expect to see in the field. However, any failure indicates some kind of weakness in the code. We can then make a conscious decision whether or not we should take action on this failure information. This is much better then not looking at what the risk may be; furthermore, in the unlikely event we do see these failures in the field, we already know something about them and have a remedy prepared in anticipation of these dire consequences. Test suites can be just a reusable as reusable software. In fact, for regression testing (testing to verify we havent added defects to already good code); we want to reuse the previous collection of tests to perform our regression testing. Our Six Sigma measurement and analysis phases will tell us if the regression testing performs up to expectations. This is a good candidate for automation, especially
47

Sav i n g S o f t wa r e w i th S i x S i g m a

for organizations that have an iterative product development methodology. In electronics product manufacturing facilities, it is no uncommon to see automated test equipment that is used to exercise the product; in effect, performing extremely high-speed regression tests. We recommend the same kind of approach to regression testingusing technology to speed us through the tedium of ensuring we did not decrease the quality of the software. We can use the results of exploratory testing to add to our existing set of formal tests; in effect, practicing continuous improvement with our test suite. Consequently, we can expect the test suite for a specific product to grow as we move through the development projectwe are learning about the product as we proceed. The expansion of the test suite should be welcomed rather than rejected because it allows us to tighten down on the product with every iteration of the test suite and each release. An additional benefit of the regular growth of the test suite is that it the testing becomes more stringent as we move towards release of our software. Test automation can improve our situation by taking our test suite and applying it to the product at machine speeds as we have already mentioned in our section regression testing. An additional benefit arises because we often do not have to have any kind of human intervention and we can, therefore, run our test suite day and night, thereby improving our throughput on test case achievement. In our case, we are also set up to make a video recording of the testing for subsequent review in the case of a logged failure. Sometimes automation may not be practical for a specific product, but we recommend automation whenever possible. The automation, partially or fully, of a test suite allows our test engineers and the software engineers to move towards a test, analyze, and fix (TAAF) regime. TAAF is a means for using our competitive edge in testing to rapidly cycle through the software testing in the interest of flushing out as many defects as we can. We continue to measure defect arrival rates and, when feasible, we can compare our results to a Rayleigh model, which may provide us with a statistical basis for authorizing a release of a product.
48

Ch a pt er 5

I m pl e m e n ti n g

Figure 5.2: Test-Analyze and Fix

5.3 Improving the Software Development Process


Prioritizing tasks can be difficult, especially when our team is under the hammer to deliver promptly (everything and now). Because many myths surround a highly-cognitive field like software engineering, it can be difficult to determine what the highest priorities are. For example, some organizations find a complete and comprehensive requirements document far outweighs the benefits of design documentation. Other organizations may find or believe the reverse. Additionally, we can use some simple financial tools to help us prioritize the tasks within our development process. We have use a tool that combines net present value with some qualitative assessments and provides a quick pseudo-quantitative calculation of priorities. When mixing factor types, we need to be careful that the value from the NPV calculation does not swamp the qualitative data. We can usually detect a prioritizing issue when we hear our engineers say something like I couldnt do the frimfram function because I didnt have time. Product development process knowledge of the staff is particularly helpful. We are not talking about the knowledge of the players for their particular part of the development puzzle, but knowledge of how the pieces should go together in general. Colleges and universities teach engineers, both hardware and software, about the technical details of the development process.
49

Sav i n g S o f t wa r e w i th S i x S i g m a

Something as rudimentary as configuration management needs are usually not covered and certainly the role of product testing is not. The staffs knowledge of the entire puzzle can come from the use of well-defined software architecture document. The architecture document, if well-crafted, will often see little modification during the development of the software; yet, it is the foundation for understanding the puzzle and all the pieces as a whole. This document can be a key reference when modifications become necessary as new features are required or fixes need to be made. Design tools can be helpful, particularly when it is possible to produce usable code directly from the design tool. We know for example, that the International Telecommunications Union standards Z.100, Z.105, and Z.120 have been implemented in software by Telelogic. This design software produces pseudocode and a graphical representation simultaneously during the software design phase. The code can be tested within the design tool and, when it is ready for outside-the-tool testing, the design can be compiled directly. Other tools such as Matlab/Simulink from Mathworks have similar capabilities. Design tools that are decoupled enough from the code such that they cannot either produce code or a compiled image are generally less useful, although we have seen research indicating that the effort of working in the latest version of the Universal Modeling Language (UML) produced software with fewer issues than similar software developed without the discipline of presenting the design in an abstract format. We need to use these tools with care so that they dont become a collection of pretty pictures to dazzle the customer and upper management. Any reasonably competent modeling tool actually represents a grammar; hence, the structure of the model should be verifiable electronically. If this feature is not available, the results from the design tool will be palliative but not corrective. Design reviews are often touted as a means for improving software. Our experience suggests that reviews may be useful if the participants approach the presented design as if they were hungry piranha and the design was a piece of fresh meat. A
50

Ch a pt er 5

I m pl e m e n ti n g

design review that doesnt have metaphoric blood flowing under the door probably isnt accomplishing much. The primary issue with this approach can be tied to the following items: Lack of use of measurement and analysis information Failure to use the lessons learned file  Inadequate critiques from the participants (the critique should not start with the actual review starts - come prepared) Lack of interest or time put in up front for the review For design reviews to have any chance of having an impact on the end design result, the due diligence in the critique has to be greater then the due diligence during the design work. The consequences for taking this review lightly should have some impact upon the reviewers. Having the people showing up with marked up copies of the design should be a requirement. Those that do not perform this up front work, have not delivered and are evaluated accordingly. Unfortunately, these items suggest that design reviews are hardly deterministic and may have significant qualitative issues. It is when looking back over your shoulder that you notice that a particular failure should have / could have been caught had there been a critical design review. One potential method for improving the design review would be to start with a standardized checklist based, in part, on our lessons learned documentation. At a minimum, this approach will provide a basis for starting the review as well as providing a baseline for problem-checking. Proof of correctness is a rarely used technique, yet a powerful approach to software development. When we prove that software is correct, we assume the specified requirements are themselves correct by some definition of correctness. The requirements may be expressed in a formal language such as Z or the B-Method or some other systematic approach. The requirements are really the Achilles heel of this process. Extensive documentation does not necessarily help. There is still much interpretation by multiple
51

Sav i n g S o f t wa r e w i th S i x S i g m a

parties, common understandings are more difficult to reach, and we may think a common understanding of the documentation exists when such is not the case. Computer scientists have theorized about formal methods for years and both the U.S. Department of Defense and NASA have taken a look at the usefulness of the approach, particularly with safety-critical software. The difficulty of the approach has always been the level of mathematical and logical sophistication required. Once we have code, we can derive the coherence of that code against the requirements. We do not need a doctorate in symbolic logic to be able to do this, but we do need to have a fundamental idea of how to approach the correctness problem. Even so, the correctness approach has the benefit of being less qualitative and more stringent than the design review approach. Increasing tempo helps us especially when we are trying to implement a TAAF (Test, Analyze, And Fix) regimen to our testing and software development. From the project management perspective, we might implement scrum as a simple means of accelerating development and improving project oversight. We have used scrum for line management and we know from experience that the scrum approach will improve the cadence of accomplishment. We can take scrum to the next step and implement a Kanban system. In the Kanban approach we will take all of our user stories (or functions or features) and break them into bite-size chunks. We can then put these items on an electronic white board or a real one and ask the developers to pull the feature from the setup area to the work area to be followed by the complete area when this discrete portion of the project is now complete. The Kanban approach is excellent for job release to the developers but may make for headaches with the planners. The solution to the planning issue lies with reasonably accurate estimation of the work time for each Kanban card/ticket. Regardless of which choices we make, the software development process must be measured: first, to be sure we didnt make it worse and, second, to ensure we have a detectable improvement in performance.
52

Ch a pt er 5

I m pl e m e n ti n g

5.4 Conclusion
There are many ways to improve the quality of the development deliverables. You must start with an understanding of your problem areas and where you can do the most good with the least effort. Making changes to the design environment also means understanding the variables, particularly those that are directly changing because of the improvement actions. These variables we will have derived from the measurement and analyze phases of our Six Sigma problem-solving approach. This means keeping your eyes open for the unintended consequences. The next time we will discuss Control, which really means the approaches we use to make our improvements sustainable.

53

Sav i n g S o f t wa r e w i th S i x S i g m a

Chapter Notes
1. S  ENGE, P.M. 1990. The Fifth Discipline. London: Century Business

54

Chapter 6
Control in Software Development
In this book, we have discussed the relevance of the Six Sigma approach to software development. We have taken close looks at Define, Measure, Analyze, and Improve. Now is the time to examine the Control phase. What is the purpose of adding Control to our list of tasks? One major reason lies in the tendency of processes to lapse back into previous and undesirable behaviors. Our Improve phase options may make for better software or better software development, but it is entirely possible that the new approach hasnt taken root with our developers. By adding some reasonable measures to control regression to the past, we can help see to it that our new approach has a chance to become institutionalized in our organizations, be they large or small or somewhere in between. Additionally, control allows a continuous critique of the performance to some baseline, identifying areas for future improvements, making it possible to asymptotically approach perfection in the development processes.

6.1 Controls in Software Code


We can help ourselves by performing a set of behaviors designed to reduce or eliminate the anticipate relapse: Code reviews Code inspections Code walkthroughs Static analyzers
55

Sav i n g S o f t wa r e w i th S i x S i g m a

Dynamic analyzers White box testing Black box testing Coding standards Practicing safe coding Code reviews and code inspections are related concepts. Inspection will often have a pre-existing checklist, whereas code reviews need not be quite as formal. When discussing control, the actual mechanism of reviews/inspections is of less interest than the frequency of the activity, the critical eye toward the actual activity, and the potential milestones around which they occur. With the exception of very large subroutines/functions, we think it unlikely that the team will review the code in less than functional size; that is, the team reviews the code when the function is complete. Furthermore, the inspection team may set a threshold for the amount of change that can occur to a function before an inspection is scheduleddown to and including one or more changes. Another version of code reviewing is the code walkthrough, with the primary difference being the leadership of the review by the developer or developers of the code. The other formats usually have team leaders who are quasi-independent of the developers. Walkthroughs can be used with specifications of any kind, test documents, models, and prototypes. The idea behind all of these methods of review is to invite other sets of eyes and viewpoints to critically review the software and/or documents for flaws. The defects of these approaches lie in the quality of the review itself and the experience and maturity of the participants. Much of the quality of the review hinges upon the amount of time in the preparing for the review. If a person shows up to the meeting and has not provide sufficient up front critique of the article, the expectation of accomplishing much should be lowered. We suggest, price of entry to the review being a marked up document. Those expected to attend, that can not show a marked up document, have failed to deliver and this should be noted.
56

Chapter 6

Co n t ro l i n S o f t wa r e D ev elo pm e n t

Figure 6.1: Inspections

Configuration management and reviews are equally important. There is abundant anecdotal evidence that suggests configuration management is a key success factor for software and embedded projects. Critiques of the configuration management plan and the details of the configuration through the development process can improve the quality of the system under development. If the project does not have configuration management, they have nothing. We know of at least one case where unknown hardware with unknown software was sent to a customer and, not surprisingly, did not function at the customer site. The four sub-controls that are part of configuration management are:  Configuration identification, wherein we give a name to the components with which we are working  Configuration control, sometimes the most intense and expensive part of the process (and most lacking)  Configuration status accounting, where we give ourselves regular reports about topics of interest in the system
57

Sav i n g S o f t wa r e w i th S i x S i g m a

 Configuration auditing, where we verify that our software function to requirements (functional configuration audit or FCA) and our documentation is correct (physical configuration audit or PCA) Software configuration management systems are generally computer-controlled these days with well-known open source tools like Revision Control System (RCS), Concurrent Versions system (CVS), and subversion. Each of these systems can be persuaded to archive binary files such as those produced by office suites. Additionally, Microsoft Sharepoint provides a level of support for document management and many add-ons exist for it that support workflow and document management. Lastly, Parametric Technology Corporation provides a Product Development System (PDS) that serves the purpose of a change and configuration management system. Static analyzers are tools the developer can use to automatically review the code for potential semantic flaws. One of the software archetypes for this kind of analysis is the tool lint. Lint provides a very detailed assessment of potential flaws, particularly with C code. Lint goes beyond the kind of syntax checking that any competent compiler will perform and adds analysis of items such as dereferencing, odd pointer usage, and bizarre casts. Gimpel Software has produced PC-Lint for twenty-five years and evolved the product over that time by adding more checks and allowing the tool to be used to make C a highly-typed language. Some pundits suggest that the use of the `#assertion pragma constitutes a form of static analysis also. Additionally, custom or proprietary tools can be developed that amount to DRC (Design Rule Checking) tools for the software. These tools evaluate the text of the software code against an organizations standard for structure and use of comments and module layout. This essentially provides some enforcement of an organization coding standards without a great deal of human resources. Dynamic analyzers provide another dimension of code control, but with a tradeoffthey are difficult to use with embedded code. When not developing embedded code, the developer can use a dynamic analyzer to check the stack, key registers,
58

Chapter 6

Co n t ro l i n S o f t wa r e D ev elo pm e n t

interrupts, the heap, watch for memory leaks and corruption, and thread observation. Dynamic analyzers as a group tend to be much more intrusive with the code than the static analyzers. Embedded developers would most likely need to use hardware support tools such as logic analyzers and digital oscilloscopes. White-box testing occurs when a tester, often the developer, works directly with the logic of the function to provide indepth and very detailed testing. For example, the developer might provide a variety of modifications to the arguments of the function as well as modifying values of global variables (yes, eliminating global variables is a form of control!) to observe the effect on the function. In order to accomplish complete observation, the white-box tester must not only observe the values of the overt output of the function, but also take a look at the internal values of the function as well as changes to any other memory location used by the function. With an object-oriented modality, we would expect to see the developers make use of get and set functions as a means of controlling access to values within the function. When we ask the testers to perform black-box testing, we are providing the means to avoid contamination of the testers by the developers. With black-box testing, we do not need to know the inner workings of the code. We provide a realistic set of stimuli to the software and observe for the expected output values all the while noting any unexpected or missing outputs. With embedded software, we can use combinatorial methods to enhance the efficiency of our test approach. The combinatorial technique uses pairwise, three-wise, and orthogonal arrays to provide a mechanically-predetermined list of inputs to the automated test code. The beauty of the combinatorial approach is that we can always estimate our coverage of the inputs and outputs as well as ensuring known levels of efficiency. The downside is that some of the approaches (for example, pairwise) will not observe input interactions except by accident. Simulation is used early to develop the target for the software. We build models that will represent the software targets to be achieved. This will be continuously refined until it represents the
59

Sav i n g S o f t wa r e w i th S i x S i g m a

final product. At which time the code will be generated for the product. The simulation code should fall under configuration management just like all the other code. We use coding standards to help the developers to present a more uniform appearance, at a minimum, and to practice known good behaviors, at an optimum. Just by improving the appearance of the code, the comments, and the general layout, we make the code more amenable to the previously-mentioned code reviews. In short, we are improving the appearance to make life easier for the reviewers and to enhance the probability they will find what they need to find in the code. The coding standard can also specify the use of casts, pointers, array bounds checking and other control techniques to help eliminate stupid mistakes by the developers. On the administrative side, software development managers must enforce the use of standards. Sometimes, the static analyzers can help ensure that the developers are following the coding standard. One well-known coding standard for C is MISRA C, a software coding standard for the C programming language developed by MISRA (Motor Industry Software Reliability Association). Unfortunately, MISRA C is not available for free. We practice safe coding when we follow certain practices that may or may not be part of our coding standard. One method for safe coding is to identify potential candidates for overflow or underflow issues and code checks to ensure that an exception is thrown before the catastrophe occurs or that some kind of correction is enabled. Another safe coding method is to require that all casts be explicit (another place where static analyzers can help us out). Safety critical projects may require that the developers use cleanroom techniques to develop the code: a combination of special diagrams, proof of correctness and statistically-based testing that tends toward zero defect software products. When available, functional languages such as F# and Scala will help to prevent side effects my eliminating the mutability of specific data structures (e.g., strings). Functional languages are generally unavailable for embedded development, but certainly deserve consideration in other venues. We once
60

Chapter 6

Co n t ro l i n S o f t wa r e D ev elo pm e n t

used pointers to achieve improved performance, but modern languages (and some old ones like FORTRAN!) are sufficiently optimized that pointers should not be a necessary. All of the methods we have discussed so far are truly controls on the software itself. However, the software development process itself can be enhanced with controls also.

6.2 Controls in Software Development


We can improve our process by adding the necessary controls to the stages of the process, for example: Lessons learned database Failure mode effects analysis on the process Process control plan Production software approval process Meaningful metrics Defect density tracking Severity assessments Institutionalization for desired behaviors Regular training / indoctrination Design methodologies Model based design Functional coding We have already referred to the lessons learned database in other chapters and, in a sense, it is a kind of holy grail for all kinds of development as we try to answer the oft-repeated and plaintive question of upper management: why couldnt you get it right the first time? Another question often asked is why did we make the same mistakes? The idea behind the lessons learned database is that we record our problems in some kind of structure format and record our solutions to the problems as well. Then, the tool can be used for learning, particularly if we:
61

Sav i n g S o f t wa r e w i th S i x S i g m a

Require new employees to read through the database  Have all members of the development and testing teams review it at least quarterly  Require a special review of specific issues before and during a development Over time we would expect to institutionalize this learning and then the lessons learned database functions as a process control. Over the years, we have the Failure mode and effects analysis (FMEA) recommended for use on software designs or software code. We suspect that this would be a serious misuse of a tool that was never designed to operate on designs at the level of complexity of software. However, we see no reason why we cannot use the FMEA on the process itself (usually called a PFMEA). For each process step, we can analyze the input (cause), the output (effect), and the breakdown in performance (failure mode) and implement contingency plans or anticipatory actions to eliminate high risk failure modes. In another place, we have recommended the use of PFMEA on project plans since project managers usually do not have a convenient tool for the preparation of contingency plans. The Process control plan is a tool used in the automotive, food, and pharmaceutical industries as a means for documenting the steps of the process, associated metrics and requirements, key process characteristics, and reaction plans. Another automotive tool is the Production Part Approval Process (PPAP), which includes a minimum of eighteen documents that support a product release. We recommend the implementation of a software analog called the Production Software Approval Process (PSAP). The PSAP would provide for collection of software-significant documents. For example, one approach might include: The original customer specification if one exists The software development plan

62

Chapter 6

Co n t ro l i n S o f t wa r e D ev elo pm e n t

Software test plans An operational concept description or concept document System/subsystem specifications Software requirements specifications Interface requirements specifications System/subsystem design descriptions Software design descriptions Interface design descriptions Database design descriptions Software test descriptions Software test report  Software product specifications, which might include the executable software, the source files, and information to be used for support  Software version descriptions for each variant of the software release as it evolves Firmware Support Manual for embedded software The PSAP allows the software organization to bundle a complete package of information about the software product for internal review as well as potential customer submission and review. The development team should use meaningful metrics and periodic presentations of metrics to assess their progress. One metric for a given software package (or by function) is defect density tracking, where we look at the ration of defects to lines of code (or function points, feature points, or story points). We would expect and hope that the defect density of the product would decline over the duration of the project. An additional approach might include the use of the Rayleigh model. The illustration that follows shows the Rayleigh distribution (really a Weibull distribution with a shape factor of two):

63

Sav i n g S o f t wa r e w i th S i x S i g m a

Figure 6.2: Weibull Distribution: shape=2, scale=40

As we can see, the defect arrival rate against time is initially relatively low, but as we create more functions and more complex functions, we see a peak value for defects. As the development work begins to subside, the defect arrival rate abates and, ultimately, we have some statistical basis for making a release. Of course, diligence in reviews, inspections, testing, and other control activities strengthen the statistical argument. We use severity assessments as part of a risk mitigation activity. The approach is straightforward: we assess the severity of each function by the amount of damage the function can do if it does not behave correctly. The malfunctions include:

64

Chapter 6

Co n t ro l i n S o f t wa r e D ev elo pm e n t

Too much of a given behavior Too little Static behavior (lockup) Periodic unexpected behavior Aperiodic (random) unexpected behavior No behavior (no startup) In some cases, embedded software will be located in a safety critical application like an anti-locking braking system. Clearly, the undesirable behavior in a vehicle braking system would have a high severity. We might also assign high severities if the malfunction could put us out of business. Regardless, a form of ranking the impact of the risk, whether catastrophic or perception issue it is necessary to develop a course of action. Figure 6.3: Severity Chart

65

Sav i n g S o f t wa r e w i th S i x S i g m a

The institutionalization of desired behaviors is more than the lessons learned database. Some behaviors meriting reinforcement include telling the truth when we discover a major software error, frequent reviews, even more frequent testing, and complete documentation. Over time, institutionalized behaviors should become part of the culture, but may need reinforcement as normal employment turnover occurs. Regular training and indoctrination is a tool for clarifying management expectations with regard to software development. In our experience in both engineering areas and manufacturing plants, we have seen organizational knowledge degrade when not reinforced. Employee turnover will cause this decay to occur naturally. Indoctrination sounds somewhat negative, but what it really means is that we are spelling out the doctrine of how we expect our development, verification, and validation to occur. Regular training and indoctrination, over time, will lead to institutionalized behaviors followed by acculturation of the team. Otherwise, your organization is hoping that the desired behaviors and risk tolerance levels are magically maintained. Our experience suggests that the failure to use this simple and cost-effective control leads to myriad problems as the knowledge of our work forces naturally degrades through turnover and the lack of reinforcement leads to extinction of the desired behavior or application of the knowledge. The software development team may select design methodologies that support better control of software concepts, design, coding, and verification. The use of known good code from libraries is a control in itself. In addition, the team and/ or management may choose to use clean room development, the Rational Unified Process, structured programming, UML, simulation, and a host of other techniques to reduce the risk of bad software. We have seen some evidence that graphical techniques such as UML do increase the quality of the software, meaning we are not simply producing a set of pretty pictures to dazzle our customers. As development techniques evolve, we would expect to see more and more cases where the developer rarely leaves the design mode; that is, the developer will compile
66

Chapter 6

Co n t ro l i n S o f t wa r e D ev elo pm e n t

the design rather than compiling the code. Such is the case with some UML products and it has always been the case with the Telelogic software that supports ITU standards Z.100, Z.105, and Z.120. In fact, parts of UML are derived from the ITU standards. Without control on the coding or the process, we would expect any improvements to degrade after implementation. The controls provide us with some measure of confidence that our improvement will remain until the next wave of improvements and, perhaps, even after that.

6.3 Controls in Software Development


Our exploration of Six Sigma and software test and performance has taken us through the phases of define, measure, analyze, improve, and control (DMAIC). This simple algorithm is a very easy to remember what to do when we discover or recognize a major issue with our software. Although we have discussed some items like project charters and Black Belts that may seem heavy-handed to some practitioners, any of these items may be tailored to fit the needs of the individual development and testing teams. The beauty of the approach lies in the simplicity of the basic algorithm, which ensures that we will never leave out a major step and, perhaps most importantly, we will define (scope) the problem before we dive in and assume we know what is going on with the software or the process. Both of us have substantial experience in embedded development. The problems we see often require an understanding of the electrical, electronic, mechanical, and software behavior of the product. Frequently, problems are timing related and intermittent. These kinds of problems are nightmares to troubleshoot and the concept of defining what is and what is not the problem can be a big help in arriving at the cause of the observable mal-behavior.

67

Sav i n g S o f t wa r e w i th S i x S i g m a

68

Chapter 7
Final Words
In this book, we have moved through the ``standard Six Sigma phases of Define, Measure, Analyzed, Improve, and Control. Please be aware that other sequences exist; for example, the Shewhart cycle of Plan-Do-Check-Act or Boyds OODA loop (observe-orient-decide-act). We know from experience that the Six Sigma sequence is a rational approach to problem solving.

7.1 What is Design for Six Sigma?


Design for Six Sigma employees some new acronyms; for example, Define-Measure-Analyze-Design-Verify (DMADV). We dont think the acronyms are that important as much as the concept of designing for high quality from the onset of development, which implies we have a real basis for identifying best practices--rather than calling the latest glimmer of selfdeception a kind of ``best practice. How often do we see real numbers when referring to the latest software buzzword?

7.2 Potential Savings


What kind of savings can we expect from Six Sigma? Experience suggests one to three percent of gross revenue for a period of two to four yearsafter which, we begin to exhaust the ``lowhanging fruit. We have seen $5 million in savings through Six Sigma projects in roughly a two and half year period for a $50-$80 million division of an electronics company. We must use some caution when anticipating probable savings. For example, we know of a school district that has a roughly $400 million budget and we
69

Sav i n g S o f t wa r e w i th S i x S i g m a

might naively expect to see $8 million a year for four years. Such is not the case, because roughly $300 million of the districts budget lies in payrollthe head count is largely dictated by the number of students plus the required infrastructure. Most of the Six Sigma projects we have seen did not result in head count reductions and we anticipate that we would see little difference with software processes and the software itself. If anything, we would expect a successful Six Sigma program to lead to more software business thanks to increases in quality and efficiency. Lean programs, on the other hand, can lead to head count reductions in manufacturing. We have not seen as much ruthless behavior on the software development side of the house, but lean projects frequently have a sordid history of losing people. We recommend that practitioners approach lean from a Six Sigma point of view, attempting to improve quality and efficiency and removing waste out of the processespecially recoding (rework!) caused inept design and code work.

7.3 Potential Obstacles


Not Invented Here In some enterprises, we may find that the company favors initiatives that originate in the company itself. Outside programs and consultants may be viewed with suspicion. We find this approach to be short-sighted. Yes, consultants sometimes ride into the sunset, leaving behind a mess. However, our experience suggests that this situation is largely a management issue that often starts with a poor choice of consultants. Obviously, it makes sense to vet the consultants who bid on the project for past performance. The consultant of choice must be willing to adapt their generic program to the needs of the client. Anything less implies that a drop-in solution will be adequate to the needs of the clientwhich is nearly always not the case. Consultants who recommend ``drop-in solutions should be dropped out of contention for the contract. We have seen individuals play this game with lean implementationtrying to use the techniques that quasiworked on their last job as a template for the new job.
70

Ch a pt er 7

f i n a l wo r ds

We have to understand that Six Sigma or lean implementations involve a corporate culture--a culture that is often justifiably suspicious of there potential future under the new initiative. Cultures do not change overnight; in fact, cultural change typically only occurs as the result of thousands of actions delivered with a vision. Flavor of the Week-itis If the employees of the enterprise have been through other initiatives, they will be suspicious of the new one. We know of one company that went through Re-engineering ISO 9001 Six Sigma Lean It almost seemed as if the company rolled in the direction of whatever business book one of the high-level executive had most recently read. We suggest that companies implement Six Sigma, especially in the software development environment, by showing success with some easy projects at the very beginning. These early successes can set the tone for future endeavors.

Lack of Top-level Management Engagement


Six Sigma Will Pay for Itself Failure to fund the new initiative is path to failure. We know of one general manager who thought the savings would fund the Six Sigma activities, which is completely backwards. First, we must invest in the new initiative with Training Tools Professional support Experience
71

Sav i n g S o f t wa r e w i th S i x S i g m a

Literature Reinforcement First, the investment, then the savings, then the return on investment. We have also seen lag times from training to real savings of up to three to four years. Hence, do not expect results overnight. We Only Need One Person to Run Six Sigma In one case we know about, the Six Sigma initiative did not take off until the trainer (one person) was complemented by an able administrator (another person). The two people were just enough to get the program to accelerate into savings and finally achieve some return on investment. The program returned roughly five to six million dollars in about three years and only came to a stop when the administrator was moved to a will-othe-wisp lean program that generated largely fantasy savings. Once again, the enterprise will get out of the program more than they put into it when people choices are made with wisdom. Six Sigma is for Manufacturing Only The Define-Measure-Analyze-Improve-Control (DMAIC) algorithm did indeed arise from manufacturing. We see no reason why it must be restricted to manufacturing only. In effect, DMAIC is a problem-solving approach and roughly matches up with the scientific method. The emphasis on measurement is important because both of us have seen much mythology surrounding software development and testing and the measurement system can take some of the twaddle out of that.

7.4 Try It, Youll Like it


We see no good reason not to try the Six Sigma approach with software. A few choice resources (e.g., books) and a dedicated employee with a simple project may be just the thing to show the benefits of this approach. Dip your feet in and try it out!
72

Ch a pt er 7

f i n a l wo r ds

73

Sav i n g S o f t wa r e w i th S i x S i g m a

Bibliography
[Alexander 1979]  Alexander, Christopher. The Timeless Way of Building (Oxford, UK: Oxford University Press, 1979). [Deleuze 1987]  Deleuze, G. and F . Guatarri. A Thousand Plateaus: Capitalism and Schizophrenia. (Minneapolis, MN: University of Minnesota Press, 1987). [French 2008]  French, Simon. \Cynen: repeatability, science and values in Newsletter of European Working Group, Series 3, 17, Spring 2008 [Gawande 2009]  Gawande, Atul. The Checklist Manifesto: How to Get Things Right (New York, NY: Metropolitan Books, 2009). [Krishnamurti 1929]  Krishnamurti, Jiddu Dissolution Speech (August 3, 1929).
http://www.jkrishnamurti.org/aboutkrishnamurti/dissolution-speech.php

(accessed May 1, 2010).

[Nietzsche 1967]  Nietzsche, Friedrich. The Will to Power. Translated by Walter Kaurmann and R. J. Hollingdale. (New York, NY: Random House, 1967). [Nietzsche 1968]  Nietzsche, Friedrich. Twilight of the Idols. Translated by R. J. Hollingdale. (London, UK: Penguin Books, 1968). [Sackett 1979]  Sackett, David L. Bias in analytic research. Journal of Chronic Diseases (Amsterdam, Netherlands: Elsevier Science B.V., 1979) [Scholtes 1998]  Scholtes, Peter R. The Leaders Handbook Making Things Happen, Getting Things Done (New York, NY: McGraw-Hill, 1998).
74

724 S. Tejon Street, Suite C | Colorado Springs, CO 80903 www.softwaretestpro.com

Вам также может понравиться