Вы находитесь на странице: 1из 4

IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS, VOL.

8, NO 2, FEBRUARY 1990 315

Software Costs of Quality


WILLIAM A. MANDEVILLE, MEMBER, IEEE

Abstracf-Our hardware manufacturing cousins have taken advan- and the costs of noncompliance. The costs of compliance
tage of the costs of quality for manufacturing by using it as a lever to
are the costs which are spent to ensure that software is
get needed prevention resources such as new tools, capital, and train-
ing to improve their productivity, quality, and to strategically reduce developed and maintained in accordance with its require-
costs. It is time for the software development community to recognize ments. These costs include the costs to prevent defects
this powerful quality improvement tool and to take full advantage of from occurring in the first place (prevention costs) and the
it. costs o f finding defects (appraisal). A list of typical pre-
vention costs for software follows:
INTRODUCTION the cost of training
the cost of tools
T HERE are very few costs to reduce in the software
world. We cannot reduce inventories or the costs of
materials, and we have little influence over the cost o f
the cost of methods
the cost of standards, policies, and procedures
the cost of consulting
platforms, and labor costs continue to rise. The significant
the cost of quality planning
savings lie in the costs of quality and our ability to reduce
the cost of team prevention meetings
them (by reducing defects). The biggest cost reductions
the cost of fast prototyping
can be gotten by determining what we are spending now
the cost of quality data gathering, analysis, and use
for the costs associated with defects, finding and fixing
the cost of root cause analysis.
the root causes of these defects, and constantly measuring
As you can, see the costs of prevention are the costs to
and monitoring to determine new cost of quality oppor-
prevent defects from occurring (The costs of putting your-
tunities to reduce. Everyone agrees with this philosophy
self into a position of doing things right the first time.)
(preventing defects reduces costs); the problem is how to
The other cost of compliance is appraisal cost. These
identify the problems, get management’s attention, and
costs are associated with the review of the product to en-
develop the projected return on investments so that pre-
sure that it meets requirements. Appraisal costs are usu-
vention resources can be applied to the roots of the prob-
ally associated with the defect removal process. They are
lem.
insurance costs and include the following activities:
This paper defines the costs of quality for software, pre-
the review of requirement specifications
sents a methodology for the collection of costs, and shows
the review of design specifications
you how to use the costs of quality to point the way to-
the review of component and module specifica-
ward quality improvement and significant cost reductions.
tions
There is a big difference between quality control or as-
the review, walk through, or inspection of code
surance and what this paper decribes as quality improve-
the testing process, including unit test, product
ment. The quality function is usually given the charter to
test, system test, and independent verification and vali-
provide assurance that for the company’s product meets
dation test
the requirements of its users.This is different from quality
Beta test, field trial, or parallel test
improvement. Quality improvement is the deliberate,
all product audits (process audits could be consid-
planned approach to preventing defects from occurring.
ered prevention because they are a method to find poten-
Quality improvement responsibility can reside with the
tial problems and remove them, before they cause de-
Quality Organization or it can be part of the development
fects).
organization’s tools or methods group. This paper con-
The costs of noncompliance are failure costs. Included
centrates on using the costs of quality as part of the im-
are the costs to fix errors in software. Failure costs also
provement process.
include the cost of any rework. Rework can be any re-
What are the Costs of Quality for Sofirware? view, audit, inspection, or test which is conducted more
than once. The first review is considered appraisal cost,
Quality costs are categorized into the costs of confor-
any re-review (rework) is considered a failure cost (cost
mance to specifications (or requirements, if you prefer)
of noncompliance). A partial list of failure costs follows:
the cost to fix an error
Manuscript received May 4, 1989; revised September 25, 1989. the cost to re-review a document
The author is with the Carman Group, P.O. Box 867689, Plano, TX
75086. the cost to retest a procedure, case, or script
IEEE Log Number 8932 100. the cost to re-review a module of code

0733-8716/90/0200-0315$01.OO 0 1990 IEEE

Authorized licensed use limited to: Ajay Kumar Garg College of Engineering. Downloaded on January 20, 2009 at 13:07 from IEEE Xplore. Restrictions apply.
316 IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS, VOL. 8. NO. 2. FEBRUARY 1990

0 the cost of computer usage for tests (except the while the cost of fixing a design error found in system test
first) includes the cost of correcting the design documents,
the cost of lab allocations for other than the first changing the code, and retesting (unit, functional, and
tests system).
all expenses associated with fixing problems with Determining which activity is contributing the most to
released systems quality costs is important in root cause analysis. It will
the cost of expediting fixes (FAX, overnight mail, give you a leg up on directing prevention resources to the
E-Mail, overtime, etc.). responsible activity. These can be termed liability costs,
Failure costs are further categorized into the costs of since the originator of the error is responsible for the en-
internal failure and the costs of external failure. The in- tire cost to fix the error.
ternal failures are those failures which occur under the
developer’s control (on the company’s premise). These What Should be Measured?
failures occur in requirements specifications, designs, It is important to measure the relationship among the
code, and the early phases of test. External failures are four costs of quality and the relationship to some base that
those which fail in front of customers or users and are top level managers understand. The importance of mea-
embarrassing. These usually cause reputation damage and suring the costs to each other is that it will dramatically
can cause bad will and loss of sales due to poor quality. show how much is spent on failure costs (fixing defects)
and how dramatically little is spent on prevention costs.
Why Measure the Costs of Quality? Appraisal costs should also be shown so that this relation-
Management (which is responsible for allocating the re- ship can be analyzed. Fig. 1 presents a typical cost of
sources) understands dollars, return on investment, and quality chart.
profit and loss (P & L). No matter how we educate and Notice that as prevention resources are added, total cost
train them, they have difficulty understanding the rela- of quality reduces. This is why quality is considered free.
tionship of defects per thousand lines of code (or on a The relationship of the costs should also be shown in ways
function point base) to real dollar investments in preven- that management can understand and clearly see that the
tion activities. For better or worse, the investment deci- total cost of quality is reducing. This can be accomplished
sion makers were brought up in a business school envi- by measuring the costs of quality to a known base (oper-
ronment. This level of management, however, will ating expenses or budget). Fig. 2 is a typical chart of the
understand that you are spending up to 50% of your op- total costs of quality to budget.
erating expenses finding, fixing, and explaining away Notice that there is cost improvement in the relation-
software defects. You have great a opportunity to show ship. If there is improvement in the relationship of cost to
them the enormous costs of quality and what can be done budget, then the money that would have been spent on
to gain improvements. failures now drops through to the bottom line of the In-
To get funding for improvement projects (prevention), come statement.
it is essential to show the current cost of defects, the cost
of the solution, and the resulting return on investment A Costing Method
(ROI). The only way to continually gain prevention re- There are two methods for collecting the costs of qual-
sources, and thus attack the problems at the source, is to ity: through a defect collection and costing system and
show the problem cost in dollars, the solution cost in dol- through the accounting system. The accounting system
lars, and ROI. The providers of prevention resources un- requires that a labor distribution system be in place and
derstand ROI. that the time spent on reviews, tests, and audits be coded
as appraisal, that the time spent on training, methods, and
COST COLLECTION METHODOLOGY
tools be recorded as prevention, and that the time spent
Where Costs of Quality Must be Measured on fixing defects be recorded as failure costs. The advan-
The costs of quality must be measured at an activity tage to this method is that it is an Accounting Department
level. Activities are defined as a group of tasks relating to function and the numbers are usually not suspect if the
a milestone event. System test would be an activity. Unit right audits and controls are in place. There are also sev-
test would be an activity. So would coding, requirements eral disadvantages.
review, and design review. 1) The Accounting Department has rigid rules and they
Errors occur in each activity. They may either be caused sometimes do not lend themselves to incorporating a cost
by the activity itself or may be the result of an error from of quality system or changes of any kind.
a previous activity. If the error is found and fixed in the 2) The Development Department is not in control of
same activity, say coding, it is much less expensive than the costs of quality, the Accounting Department is.
if it originated in function design and did not manifest Needed changes are sometimes difficult to get because
itself until system test. This is because the cost of cor- they are not the Accounting Department’s first priority.
recting a code error in the coding activity is a matter of 3) It is difficult for developers to accurately record time
changing the code, recompiling, and other administrative spent on different activities.
activities such as reconfiguration and possible re-review, 4) The costs of quality can be shown, but the trail to

Authorized licensed use limited to: Ajay Kumar Garg College of Engineering. Downloaded on January 20, 2009 at 13:07 from IEEE Xplore. Restrictions apply.
MANDEVILLE: SOFTWARE COSTS OF QUALITY 317

140 MONTHLY
120
100
INPUT 4
80
60
40
20 TPUT
0
I 1989

MONTHLY TOTAL EKITRNAL INTERNAL SET UP 4 CALCULATIONS

IAPPRAISAL 0 PREVENTION

Fig. I . Total cost of quality. Summary by cost type for project. Fig. 3. System layout.

The set-up database contains the rates of personnel


working on particular tasks such as functional design re-
views or fixing bugs in a requirements document. These
can be established once and reviewed periodically. The
data entry database is monthly input, where the number
of reviews and tests and the resultant defects are input.
I 1989 I 1990 I
The data in the set-up database and the data entry database
+ MONTHLY TOTAL * DLTERNAL 8 INTERNAL must be kept at an activity level (i.e., code, function de-
-K- APPRAISAL + PREVENTION
sign, system test, etc.) so that the costs can be traced to
Fig. 2 . Percent of quality cost to budget.
the activities. This aids in root cause analysis, by getting
to the activity with the highest cost. On a monthly basis,
which defects are causing the largest costs is dead ended. the calculations should be performed. Monthly is pre-
This is an important disadvantage. You need a trail to the ferred to quarterly because it is more timely or weekly
defects so that root cause analysis can be performed. because the numbers will have more impact. Besides,
A defect collection and costing system uses actual de- managers are used to looking at costs and their impact on
fects and their causes, the labor rates of the personnel fix- a monthly basis.
ing the defects, and the average times to fix defects. These Once the costs of failure are determined (failure cost is
defect costs are used in the calculations for internal and always the largest), we need to look at the defects which
external failure costs. The number of reviews, tests, and are occurring the most frequently Fig. 4 shows the top ten
inspections are used for appraisal costs. defects by type.
Again, there are both advantages and disadvantages to This chart is easily obtained from a defect tracking sys-
a defect costing system. The advantages are that the de- tem. However, if we are interested in strategic cost re-
velopment activity is in control of its cost of quality sys- ductions, we must look beyond the frequency of defects
tem, the rules of accounting are relaxed, and the system and look at the overall cost of the defects. There is a re-
will have the capability to point the way toward the most lationship to the frequency of defects and their costs,
costly defects. The disadvantage is that the numbers are however, costs have a stronger relationship to where a
likely to be less accurate than an accounting system. But, defect was found in the life cycle, versus where it was
the reason you are using a cost of quality system is to inserted. A review of Fig. 5 shows this.
strategically reduce the development costs by getting pre- Notice that defect DO0056 on the defect-by-type chart
vention resources. Accounting accuracy finishes in sec- (Fig. 4) is the most frequent, while defect DO0051 (Fig.
ond place behind root cause analysis and quality improve- 5) is the most costly. To strategically reduce costs, the
ment. most costly defects should be tackled first. This defect
The costs of quality using defects and events (such as strategy is different than trying to get a release out. In that
reviews and tests) is not well publicized and deserves ex- case, the most severe defects must be fixed first. But, fix-
ploration. To start with, a defect collection system can be ing software errors to get a release out does not improve
a good system for root cause analysis and cost reduction. the quality of the software development process. It does
By investigating the top ten defects occurring during an affect the quality of the particular release, but since there
activity and then performing root cause analysis on the is no root cause analysis and corrective action, the same
top offenders, the roots of the problems can be deter- errors will appear in release after release. The root causes
mined. However, to get proper prevention funding, you of defects are rarely paid any attention when you are un-
must know the cost of the problem, the cost of the solu- der release pressure.
tion, and the return. For example, it would be difficult to The improvement in quality and the resultant reduction
obtain an investment of $200 000 in prevention resources in costs of quality come only when we determine the root
if the costs of the problem were not known. And it would causes of defects and take the appropriate corrective ac-
be unwise to invest the funds if the return was less than tion to eliminate the defect, forever.
adequate. Using a cost of quality system not only points us toward
A cost of quality system for software using defects and the most costly defects, but it will also provide liability
events as a base looks like the diagram in Fig. 3. costing. Liability costs are those costs assigned to the
318 IEEE J O U R N A L ON S E L E C T E D A R E A S IN C O M M U N I C A T I O N S . VOL. 8. N O 2. F E B R U A R Y 1990

~~
~

TOTAL COST BY DEFECT


(Top Ten) March/1990

Fig. 4. Total defects by type. Top ten, July 1989

Unique Project Number Goes Here. Proj No.

Fig. 5 . Total cost by defect. Top ten, July 1989.

improvement process is on an even keel and headed in the


originators of the bugs in software. For example, the cause right direction. Opportunities for improvement are always
of a bug inserted by the systems requirements function, there, and the cost improvements to the bottom line fol-
and found in test, is long forgotten when we are trying to low along hand in glove with the quality improvement
make a release schedule. However, if the costs to fix the process.
problem are assigned to the systems requirements func- The three most important things in improving quality
tion, a whole new list of management concerns and ques- are Management Commitment, Management Commit-
tions are likely to be raised. Such as, “How did this prob- ment, and Management Commitment. The way to get and
lem arise?” “How did it get so far without being maintain management commitment is to provide them a
noticed?” and “What are we going to do about prevent- solid return on investment, just as they must provide a
ing it from recurring?” These are excellent questions and return to their management and stockholders. Cost of
the type that quality professionals and designers should quality systems show you what you are spending on de-
ask themselves. But without the added impact of showing fects, what the most expensive defects are, and where they
the defects in costs, the requirements and design errors originate.
are long forgotten. Fig. 6 is a typical liability chart show-
ing the costs of defects assigned to the functions respon- CONCLUSION
sible for them. The time has come for software quality professionals to
This chart brings the right emphasis on problem solv- show their contributions to the company’s well being. We
ing. It also provides the first step in root cause analysis have always known that our activities have positively
and corrective action. We try to solve most problems by contributed, now we can show it. Establishing a software
getting faster test methods (automated testers, faster test cost of quality system is an easy, cost-effective method to
platforms, etc.). This is needed, but the most costly de- start you down the road of long-term cost reductions and
fects are caused by the early life cycle activities (testers real quality improvement.
do not cause defects, designers do).

Root Cause Anulysis and Corrective Action Using


Sofrwure Costs of Quality
Once you have determined a starting place for your root
cause analysis (the most costly defects), you need to de- William A. Mandeville (M‘88) received the B . S .
degree from the University of New Haven. New
termine the cause. After you have determined the root Haven, CT. and the M.S. degree from the Uni-
cause, you need to analyze the solutions. The solutions versity of Southern California.
then need to be costed. With the cost of the problem in He is a nationally recognized expert in the ficld
of software quality assurance and software quality
hand and the cost of the solution, you need to present the costing. His experience spans 20 years in software
return on investment. This is not a trivial task and should engineering and quality management with major
not be taken lightly. There are many activities competing companies including General Dynamics. ITT.
ITEC. and Ericsson. He has prehentcd keveral in-
for investment funds. You must explain the costs of qual- novative papers and has led national discussion
ity, what poor quality is costing the company, the costs panels o n new methodologies in the area of commercial and military soft-
of the solution. and the return that can be provided. ware quality costing and defect collection and reporting. He is currently
the Vice President of Development with the Carman Group. Dallas. T X .
Monitoring the cost of quality improvements is a con- The Carman Group provides software quality aswrance tools. training. and
tinual task. It is the only way to ensure that your quality consulting services.

Authorized licensed use limited to: Ajay Kumar Garg College of Engineering. Downloaded on January 20, 2009 at 13:07 from IEEE Xplore. Restrictions apply.