CPSC 547 Software Measurement

Software Quality Metrics

Professor: Dr. Bing Cong

By:
Aakash Juneja
Jaskaran S. Gandhi
Phong Nguyen
Saba Yaghoobi Saray

Department of Computer Science

California State University Fullerton
Fall 2014

Table of Contents
Abstract ................................................................................................................................................. 4
1.

What is Software Quality? ............................................................................................................ 5

1.1 Why Measure Software? ................................................................................................................. 6

1.2. Software metrics and its objectives ................................................................................................ 7
1.3. The Need for Metrics ...................................................................................................................... 8
1.4 Types of Metrics- ............................................................................................................................10
2.

Background ................................................................................................................................... 11

2.1 Waterfall model .............................................................................................................................. 12

2.2. Spiral model ................................................................................................................................... 13
3.

Software Quality Metrics Classification ...................................................................................... 14

Figure 1. Classification of Software Metrics ........................................................................................16

3.1 Product Quality Metric .................................................................................................................. 17
3.1.1 Scenario ........................................................................................................................................18
Table 1: Requirements Coverage Table ............................................................................................. 20
3.1.2 Goal ............................................................................................................................................. 20
3.1.3 Product Risk Metrics ................................................................................................................... 21
Figure 2: Quality Risk Status (Early Test Execution) ........................................................................ 23
Figure 3: Quality Risk Status (Middle of Test Execution) ................................................................. 24
Figure 4: Quality Risk Status (Late Test Execution) .......................................................................... 25
3.2 In-Process Quality Metric ............................................................................................................. 26
Figure 5: Two contrasting Defect Arrival patterns during Testing ................................................... 28
Figure 6: Defect Removal by Phase for two products ........................................................................ 30
Figure 7: Phase Effectiveness of a Software Project .......................................................................... 32
3.3 Resources ....................................................................................................................................... 32
Table 2 : Comparison between resources ........................................................................................... 33
Table 3: Capacity/Load ....................................................................................................................... 33
Figure 8: Resource Distribution.......................................................................................................... 34
5.

Real World Examples of Software Metrics ................................................................................. 34

a.

Motorola....................................................................................................................................... 34

b.

Hewlett - Packard ........................................................................................................................ 40

Figure 9: Testing Metrics .................................................................................................................... 43

Figure 10: Defect Summary ................................................................................................................ 45

Figure 11: Time Data and Return ....................................................................................................... 45

5.

What are the advantages of using software metrics?.................................................................. 46

5.1 Limitations of Software Metrics .................................................................................................... 48

6.

References ................................................................................................................................. 49

Abstract
Software development is the active conduct of program design systems as they are
retained and improved over their periods. Software systems need to repeatedly change during
their life cycle for numerous causes; adding new features to satisfy user requirements, changing
business needs, introducing novel technologies, correcting faults, improving quality, etc. The
increase of changes, along the development of a software system, can lead to a bad quality. It is,
thus vital to observe how software quality changes so that quality assurance activities can be
correctly scheduled. Software quality is basically a field of study and practice that describes the
necessary qualities of software products.
Software metrics can be used to examine the improvement of the software systems
quality. Empirical sign occurs presenting that there exists a connection among metrics and
software quality. Software metric is a tool for accepting changing phases of the code base, and
the development of the plan. The goal is gaining impartial, reproducible, and computable
measurements, which may have plentiful valued applications in plan and budget planning, cost
estimation, quality assurance testing, software debugging, software performance optimization,
and optimal personnel task assignments.
What seems to be challenging on the way of quality engineering and management is that
the word quality per se is kind of unclear and this problem can consequences to mistake in
delicate processes. There are numerous details behind this mistake. First of all, each person can
have different knowledge about the idea of quality based on their perspectives. Secondly, there
are different levels of concept for each term; while you are debating about quality, we may refer
to the inclusive meaning of quality, or you may refer to obvious sense of it. Moreover, since we
use the term quality in our day to day language it causes difficulties on the technique of using
this term in specialized positions, as these two are so different.
We are planning to discourse the goal of software quality metrics on this paper, while we
explain about metrics classification in software quality namely product metrics, project metrics,
and in-process quality metrics. Meanwhile, we are going to discuss the importance of collecting
software engineering data. In other words, the accumulated data will be the results of welldefined metrics and may help to improve our system quality.

1. What is Software Quality?

Our world uses software everywhere. Every profession depends on it, every cell-phone
needs software, and even each car depend on it. Without software, our world would fall apart.
With this in mind, the quality of that software is important. Because it is widely used, low
quality software is not satisfactory. But what is software quality? High quality software meets
the needs of customers while being reliable, well supported, maintainable, portable, and easily
integrated with other tools. The most basic search in a software quality would be lack of bugs. It
is important that our requirements have as minimum bugs as possible because it will cost a lot of
fix them later on in the software cycle; if the software contains too many defects, the requirement
will be unfulfilled. We express these two ways: defect rate which is no. of defects/million lines
of code and reliability which is no. of failures per n hours of action and mean time failure.

Customer approval is usually measured by taking the percent between (neutral and nonhappy) by the customer satisfaction surveys. To decrease unfairness, methods such as blind
surveys are frequently used. In addition to complete client contentment with the software
product, satisfaction toward specific qualities is also estimated. For example, IBM estimates
satisfaction with their software products in terms of capability, functionality, usability,
performance, reliability, install ability, maintainability, documentation, service, and overall.
Hewlett-Packard focuses on functionality, usability, reliability, performance, and service ability.
Other corporations use related measurements of software customer satisfaction.

We must take these quality attributes into account in the planning and design phases of the
software to increase overall customer satisfaction. However, these quality attributes are not
always mix well with each other. For example, it is hard to attain maintainability when the
complexity of the product is high. Depending on the type of product and customers, different
factors are mandatory for different feature attributes example real-time processing, performance
and reliability might be the most significant attributes. For customers with individual systems
and simple operations, documentation may be more important.

1.1 Why Measure Software?

Determine the quality of the software

Calculate qualities of a software

Improve quality of a software

Estimation and cost of software

Estimate the efficiency effects of new tools and techniques

Establish efficiency developments over time

Improve software quality

Decrease future maintenance needs

1.2. Software metrics and its objectives

Software metrics is used to define the wide-ranging of activities used with measurement. The
practice of metrics involves measures and metrics which leads to long term process
improvement. It assesses the state of the project, tracks potential risks, uncover problem areas,
adjust workflow or tasks, and evaluate teams ability to control quality. Software product metrics
focuses on measuring attributes of the software product, which is any object or document
subsequent from the software development method. Software product objects descent into one of
the four categories: specifications, designs, source code, and test records.
Software metric is simply measurement of software and primary purpose is to help plan and
foresee software growth. Any software metric is an effort to measure or foresee some attribute of
software, process, or resource. Attributes can be internal or external. Internal attributes are those
of the system representations, such as design diagrams, coding or documentation which can be
controlled and measured directly, but their measurement slight off unless there is indication that
the specific metric is related to an external attribute. Examples of internal attributes of source
code are:

Size

Complexity

Coupling

Modularity and

Reuse.

External attributes are usually those influenced by the system in implementation and cannot be
measured directly. They are for example:

Functionality

Reliability

Usability

Efficiency

Maintainability and

Portability

External attributes are important because knowing or foreseeing them plays a vital role in quality
assurance. Some internal attributes are connected with external attributes therefore their metrics
can be used as secondary measures for external attributes.

1.3. The Need for Metrics

Software testing is the process of recognizing the errors or defects in the system and make sure it
is resistant with the customer requirements before releasing it to the market. But testing can
never guarantee 100% bug free software. This might show the unpredictability and
ineffectiveness of testing process or testing procedures followed, it could also account the human
mistakes in case of manual testing or script errors in case of automated testing. This indirectly

influences the quality of the software. Hence, in order to increase the software quality, certain
procedures and guidelines need to be placed down and match the efficiency of the software.

Every software experiences the risk phase. Hence risk management is a very significant factor
which needs to be taken care in order to recover the software quality, consistency and strength of
the software. There should be high amount of clearness as to where the software views in terms
of quality, quantity, development, customer efficiency, obedience to requirements etc. This raises
a distress to have an efficient metric and report based procedure, thus giving growth to test
metrics and reports.

Following are the arguments that support why software testing metrics are necessary.

Metrics helps in tracking of the project status and supports in presenting the statistics to
the senior management in a structured way.

Back tracking can be very stress-free if every action is followed correctly.

Metrics and reports help in accumulating data using which further processes in testing
can be completed more effectively.

Software Metrics and reports help in both project management and process management

Metrics can directly impact both the effectiveness and productivity of a software

Helps in early defect detection and defect removal, thus reducing the cost of defects

Assists the managers in effective decision making

Metrics also act as a benchmark for estimations and provide bottleneck to the testers

Manages risk at ease

Software metrics are used to obtain objective measurements that can be useful for quality
assurance, performance, correcting, managing, controlling and estimating costs. Finding defects
in code (post release and prior to release), foreseeing faulty code, calculating project success, and
predicting project risk. The utility of metrics is limited to quantifying one of the following goals:
Schedule of a software project, Size/complexity of development involved, cost of project, and
quality of software

To summarize, the goal of software metric is to understand the state of the software product.
Hence, it is very much vital to select the correct set of metrics for our product during the testing
process.
1.4 Types of MetricsRequirements metrics

Size of requirements

Traceability

Completeness

Volatility

Product Metrics

Code metrics

Lines of code LOC

Design metrics work out from requirements or design documents before the system has
been implemented

Object oriented metrics- helps in finding faults, and allow team to see directly how to
make their classes and objects simpler.

Test metrics

Communication metrics looking at artifacts i.e. email, and meetings.

Process metrics

Measure the process of software development

Frequently used by management to check the budget and processes for efficiency.
Evaluate and track aspects of the software design process like: human resources, time,
and schedule

2. Background
First thing we need to consider when we are talking about the background of software
measurement is the potential impact of the different software development models on quality.
What seems impossible is discussing software metrics and models without considering the effect
of software development process type on them. Depending on our projects goals and objectives
we designate the preferred software development model among the variety of different processes
and methodologies. Based on which objectives we are pointing at, there are several development

life cycle models to adopt. Each of the known models defines different stages of the process and
also dissimilar set of orders in which the stages should be performed.
Here, we are going to discuss two of the best or the most popular life cycles. Meanwhile, we are
going to explain why it is needed to merge the metrics.

2.1 Waterfall model

During 60s and 70s, software developers were more concerned about controlling and planning
difficulties. At that era, almost all of the software development projects were facing gigantic cost
escalations on one hand and schedule deferrals issues on the other hand. It was exactly when
waterfall process aroused to put an end on the increasing complication of development projects
and its consequences. The waterfall process model raises the development teams sprits to clarify
what are the expectations of this projects to be met before beginning development of the system.
It is simply known as collection and description of the system requirements. The next motivating
asset of waterfall model is its inspiration to break the whole development process into some
reasonable phases such as design, code, test, and etc. which are supposed to deliver some
halfway products leading to the final product. It seemed crucial to make sure of the appropriate
execution and also delivery of good-quality products, hence, each step was required to pass some
criteria such as validation, entry, and exit. This is well known as Entry-Task-Validation-Exit
(ETVX) paradigm which is one of the major specifications of waterfall process.
Waterfall process has various advantages because of this divide and conquer approach. First,
precise tracking of the project evolution is more possible while early recognition of errors is
more likely. Second, by this approach, the software system development organization is obliged
to be more arranged and controllable.

For outsized companies, having huge and complicated development projects, structural tactic
plays a significant role. Using this approach, during the process, bunch of documents must be
created for future use in testing and maintenance arenas. The most important aspect of this
method is making large projects easier to manage and possible for on time delivery without cost
escalations.

2.2. Spiral model

The spiral model which has been developed by Boehm is one of the models of software
development, being fallowed in some organizations. It has been formed after lots of
improvements on the waterfall model while it applied to variety of enormous government
software projects.
The spiral model mostly relies on creating prototypes and using risk management methods.
These features of spiral model increase the spiral models level of flexibility comparing to
waterfall model. As Boehm has explained, the most inclusive application of this model is related
to the expansion of the TRW Software Productivity System, which is known as TRW-SPS.
During these last few years spiral model has been positively recognized between engineers and
project managers due to its high risk management ability.
The need for software metrics had been more palpable as introducing increasing amount of
object oriented projects were occurring. In other words, we can say the available software
development models were effective for the variety of software versions but there were some
restrictions when it came to object oriented softwares. That was when engineers were convinced
the combination of software metrics and software development models is the best solution to the
problem.

3. Software Quality Metrics Classification

Software Quality Metrics are a subclass of software metrics that emphasis on the quality
features of the product, process, and project. In most cases, software quality metrics are more
thoroughly related with process and product metrics rather than with project metrics. Software
metrics can be categorized into 3 classifications:

Product Metrics

Process Metrics

Project Metrics

Product metrics help software engineers to better appreciate the qualities of models and
measure the quality of software by describing the characteristics of the product such as size,
difficulty, design features, presentation, etc. A vision into the design and structure of the
software is provided, and also they help in measuring quality based on a set of distinct rules, that
helps the software engineers. Process metrics are composed over a long period of time through
all projects, and they can be used to help improve software development & maintenance, and aid
in making strategic decisions. The goal is long-term process improvement that could be attained
by providing a set of process indicators. Project metrics are used by project managers and a team
of software designers to adjust project workflow and practical activities that allows them to
define project features and implementation designs. At times, metrics belong to multiple
categories: in-process quality metrics of a project both involve process metrics as well as project
metrics.

Having said above that, software quality metrics are closely associated to process and
product metrics rather than with project metrics; the parameters related to project development
such as the number of developers, their individual skill levels, the size of the project, etc.
certainly affects the quality of the product. The intention of Software Quality Engineering is to
look in to the relationships that exist amongst in-process metrics, project characteristics, and endproduct quality. The findings of this research helps in engineering improvements in both process
and product quality.
In the discussions below, we shall discuss several kinds of metrics in each of the three
classifications of software quality metrics as described briefly above.

Software
Metrics

Product
Metrics

Process
Metrics

Dynamic
Metrics

Static Metrics

LOC

Size Metrics

Design
Metrics

Control Flow
Metrics

Information
Metrics

Token Count

Function
Count

Function
Count

Henry &
Kafura

Figure 1. Classification of Software Metrics

Weighed
Metrics

Data
Structure
Metrics

Software
Science
Metrics

Testability

3.1 Product Quality Metric

Product Quality Metrics help us to better understand the attributes of models and assess
the quality of software under test as well as help us gain insight into the design and construction
of the software. A good test allows us to measure the quality and risk in a system, however
proper product metrics are needed to capture the test measures. Product Quality Metrics provide
an on-the-spot rather than after-the-fact insight into the software development of a product, and
at the same time provide insights to guide where product improvements should occur.

Further discussing the 3 sub-groups of Product Quality Metrics:

Efficiency Product Metrics The degree of measure to which a product can attain that
anticipated level of quality carefully.

Effectiveness Product Metrics The degree of measure to which the product is attaining
anticipated levels of quality.

Elegance Product Metrics The extent of measure to which a product effectively and
efficiently achieves the results in an elegant, and well-executed way.

A Quality Metric is a predictor for Product Quality, and both are used to degree the
properties of software and at the same time help in refining a system module by matching them
with current better-quality systems.

The steps to develop a good quality Product Metrics [2]:

Define test coverage and quality-related objectives for the product.

The effectiveness, efficiency, and elegance with which the product is achieving those
objectives should be taken into consideration.

Devise measurable metrics, either direct or surrogate, for each effectiveness, efficiency,
and elegance question.

Determine realistic goals for each metric, such that we can have a high level of
confidence in the quality and test coverage for the product prior to release.

Monitor progress towards those goals, determining product status, and making test &
project control decisions as needed to optimize product quality and test-coverage
outcomes.

3.1.1 Scenario
We have the following information on a project:

95% of the tests have run

90% of the tests have passed

5% of the tests have failed

4% of the tests are ready to run

1% of the tests are blocked.

It has been assumed that we are on schedule as per the text execution. However, this
information is not accurate enough to confirm that this product will be of standard quality by the
last phase. Since, this information is not accurate, we would need product metrics to assist us in
determining the quality throughout test execution, and at the same time making sure that quality
is on track for successful delivery of the product. Testing Product Metrics are focused on the
quality of the system under test, and it is important to have this information on testing
dashboards so that the stakeholders, participants, and other involved members are on the same
track.

In addition to making sure the quality of the product is at par, Product Metrics also reflect
the entire teams efforts towards quality. Stakeholders, and the management are key as theyre
the ones who determine the software process and its quality capabilities, and the only role of the
testing team is to measure quality, and they cannot control the behaviors of stakeholders or
management with respect to quality.

The objectives for test coverage and quality for every product vary, but often includes
ensuring complete coverage of requirements. A requirements coverage table would help in
monitoring progress towards the goals of complete testing and fulfillment of the requirements, as
shown below [3]:

Table 1: Requirements Coverage Table

3.1.2 Goal
An analytical requirements based test strategy is used, and one or more tests should be
created for every requirement during the test design & implementation phase. If bi-directional
traceability is maintained, complete coverage can then be assured. During test execution, we run
these tests, and we can report the results in terms of requirements fulfilled, and unfilled using the
traceability.

The progress can be monitored towards these goals of complete testing and fulfillment of
requirements by using the metrics analysis as shown in the table above. At the same time, the
table helps in understanding the quality of the product and it eases the decisions to make test and
project control decisions that help us to achieve the best possible test coverage.

The table above depicts quality stands on the major requirements groups for an ecommerce site, and the status of each requirement in each group. Based on the analysis, the
status of each requirement may be determined as follows:

If all the tests related to requirement have been run, then it is said that the requirement
is tested.

If all the tests related to requirement have been run and have passed, then it is said
that the requirement is also classified as passed.

If all the tests relayed to requirement have been run, but one or more have failed, then
it is said that the requirement is classified as failed.

If any of the tests related to requirements are blocked, then it is said that the
requirement is classified as blocked.

If the metrics in a table highlight problems, then it is very helpful for the engineers. This helps to
increase the level of confidence in terms of positive measurements of requirements. Practically,
it is difficult to develop metrics that will predict the quality for software, but with multidimensional coverage metrics, we can have a higher level of confidence when it comes to
satisfaction.

3.1.3 Product Risk Metrics

If we consider product metrics for testing strategies other than requirements based
testing, but now we shall consider risk based testing in which the objective is to typically reduce

product quality risk to an acceptable level, and these two questions can be taken into
consideration:

How effectively are we reducing quality risk overall?

For each quality risk category, how effectively are we reducing quality risk?

Answering the first question, when using a risk based strategy, each quality risk item
deemed sufficiently important for testing should have one or more tests create for it during test
design and implementation. With the help of bi-directional traceability, the coverage can be
assured and measured between the tests and the risk items.

The tests are run and defects are reported during test execution. Bi-directional traceability
is needed between the defects and risk items as well in addition to the test results and risk items.
This enables us to report which risks are fully mitigated, which risks are partially mitigated, and
which risks are unmitigated.

Quality Risk Status

Other Risks

Risks for which at least

one test has failed or
must-fix bug known
Risks for which all tests
were run & passed

Figure 2: Quality Risk Status (Early Test Execution)

The above figure graphically represents all information across all risk items. The region
in green represents risks for which all tests were run and passed. The region in orange represents
risks for which at least one tests has failed or at least one must-fix bug is known. The region in
blue represents other risks, which have no known must-fix bugs but still have tests pending to
run.

Quality Risk Status

Other Risks

Risks for which at least

one test has failed or
must-fix bug known
Risks for which all tests
were run & passed

Figure 3: Quality Risk Status (Middle of Test Execution)

During the second half of the test execution, the green region starts to grow very quickly.
The tests those are most likely to find out bugs in the early half of the test executions have
already been carried out. Testers focus on running confidence-building tests that lower risk
(turning blue region to green), while developers fix the bugs that were found.

Quality Risk Status

Other Risks

Risks for which at least

one test has failed or
must-fix bug known
Risks for which all tests
were run & passed

Figure 4: Quality Risk Status (Late Test Execution)

As mentioned above, the green region completely takes over all the regions during the
end of the testing phase, and the blue region would slowly be faded out that is a good sign since
it represents unmitigated risks.

When a risk-based test strategy is followed, the project management team decides
adequate test coverage. The risks are said to be balance, and quality risk mitigation has been
optimized only if the project management team believes that the quality risks posed by known
defects, test failures or yet un-run tests are acceptable, compared to the schedule and budget risks
associated to continue the testing.

3.2 In-Process Quality Metric

Process metrics are collected over long periods of time and they pretty much cover all
projects involved, and they are used for making strategic decisions. A set of indicators is
provided by process metrics that leads to software process improvement. Organizational
sensitivity is to be kept in mind when interpreting metrics data, and at the same time it is
required to provide regular feedback to engineers who are directly involved with the collection of
measures and metrics.

The basis for in-process quality management is formed by the following metrics:
1. Defect density during Machine Testing: The defect rate is correlated to the defect rate
during formal machine testing. A higher defect rate is an indicator that during the development
phase, the software was injected with a lot of errors. In some cases, the extra-ordinary testing
effort could also be the reason for a high defect rate. A uniform distribution is never followed by
software defect density. If a piece of code has higher testing defects, it is a result of more
effective testing or it could be because of higher latent defects in the code. The team of engineers
can use the following scenarios to judge the release quality [1]:

If the defect rate during testing is the same or lower than that of the previous
release, then ask: Does the testing for the current release deteriorate?
If the answer is no, the quality perspective is positive.
If the answer is yes, extra testing needs to be done.

If the defect rate during testing is substantially higher than that of the previous
release, then ask: Did we plan for and actually improve testing effectiveness?
If the answer is no, the quality perspective is negative. The only
remedial approach in this case is to do more testing, which yields
higher defect rates.
If the answer is yes, then the quality perspective is the same or
positive.

2. Defect arrival pattern during Machine Testing: The defect density during testing is a
summary indicator. More information is given by the pattern of defect arrivals. Even with the
same overall defect rate during testing, different patterns of defect arrivals indicate different
quality levels in the field. Both the defect arrival rate and the cumulative defect rate are shown
by the figure below. The objective of this pattern is always to look for defect arrivals that
stabilize at a very low level, or times between failures that are far apart, before ending the testing
effort and releasing the software to the field. These declining patterns of defect arrival during
testing are indeed the basic assumption of many software reliability models, and the time unit for
observing these patterns is usually weeks and months. [1]

Figure 5: Two contrasting Defect Arrival patterns during Testing

There are three slightly different metrics when we talk about defect arrival patterns
during testing:

The defects reported during the testing phase by time interval. These are just the
raw number of arrivals, not all of which are valid defects.

The true defect pattern is the pattern of valid defect arrivals when problem
determination is done on the reported problems.

The pattern of defect backlog overtime, this is needed because development

organizations cannot investigate and fix all reported problems immediately. A
large defect backlog at the end of development cycle and a lot of fixes has yet to
be integrated into the system, the stability shall be affected. A regression test is
needed to ensure that targeted product quality levels are reached.

3. Phase-based defect removal pattern: The test defect density metric extends to the
phase-based defect removal pattern. It requires the tracking of defects at all phases of the
development cycle (including design reviews, code inspections, and formal verifications) in
addition to testing. Conducting formal reviews or functional verifications to enhance the defect
removal capability of the process reduces error injection, and the overall defect removal ability
of the development process is reflected by the pattern of phase-based defect removal.

Many development organizations use metrics like inspection coverage and inspection
effort for in-process quality management. The figure below shows the patterns of defect removal
of two development projects: Project A was front-end and Project B was heavily testingdependent for removing defects. The different phases of defect removal are high-level design
review (I0), low-level design review (I1), code inspection (I2), unit test (UT), component test
(CT), and system test (ST).

Figure 6: Defect Removal by Phase for two products

4. Defect Removal Effectiveness: Also referred as efficiency,

DRE= (Defects removed during development phase/Defects latent in the product) x 100%
The total number of latent defects in the product at any given phase is not known, the
denominator of the metric can only be approximated. It is usually estimated by:

Defects removed during the phase + defects found later

For each phase, and for the front end, the metric can be calculated. It is known as early
defect removal and phase effectiveness when used for the front end, and other specific phases.
The higher the value of the metric, the more effective the development process and hence, the
fewer the defects are likely to escape to the next phase. The figure below shows DRE by phase
for a software project, the weakest phases were unit test (UT), code inspections (I2), and
component testing (CT). The action plans to improve the effectiveness of these phases were then
established and deployed.

Figure 7: Phase Effectiveness of a Software Project

3.3 Resources
There are certain metrics related to resource distribution measurement such as: utilization
percentage, effort distribution, capacity/load, and response time.

Response Time:
Using response time metric, we will be able to recognize what is the state of our
desired resource. By picking a baseline, we will have a scale to use later on. The next

step is comparing the response times to the baseline whenever we are trying to access
a resource. It is so simple:
Comparison

Resource status

Response time < baseline

Resource is available

Respond time > baseline

Resource is overloaded

Table 2: Comparison between resources

Utilization Percentage
(Total Effort spent by resource) / (Total Budgeted Effort for the resource)
Utilization Percentage simply provides the utilization related to a specific resource in
order to avoid having under-utilized or over-utilized resources. Here, the point to
keep in mind is that the utilization stages should be kept optimal.

Capacity/Load:
It is the best utensil to calculate the maximum load that your project can tolerate and
by using this you can easily figure out if your project is kind of overloaded.

Ratio amount

Project status

Ratio =1

Ideal

Ratio <1

Project can take more load

Ratio >1

Project is overloaded

Table 3: Capacity/Load

Effort Distribution:
It is supposed to show the share of particular resources among a number of
tasks/components. We use pie charts to show the percentage of resources assigned to
each task or component more clearly. For sure, one of the ways to ensure about the
appropriate distribution of resources is using effort distribution measure.

Resource Destribution
Requirement Analysis

0.05

0.05
0.2

Design
Implementation

0.2

Testing
Documentation
0.3

Rework
0.27

Figure 8: Resource Distribution

5. Real World Examples of Software Metrics

a. Motorola
Many companies attempt to use software metrics to better their position, but they often find it too
complex and unnecessary to truly follow-through. On one industry survey that was taken, less
than 10% of the companies surveyed considered the use of metrics as positive and

enthusiastic [100]. One of the few companies that was able to positively turn metrics into
improvement was Motorola.
The managers and engineers at Motorola were yearning for more understanding and insight into
the software development process of their company. They wanted to see if improvement could
be done to quality, productivity, and cycle time.
The first step Motorola did was to define a Metrics Working Group (MWG) in their
organization. This group would be in charge of establishing and defining a company-wide
adopted set of software metrics to be used for improving the quality of the software. This group
was and is very successful in their work, they have been responsible for numerous processes and
metrics in the company. One of these being the Goal/Question/Metric approach.
Measurement is not the goal. The goal is improvement through measurement, analysis, and
feedback. That was the philosophy of the Motorola software metrics initiative. This philosophy
was put in practice when senior management made it a requirement to software development
process (in their Quality Policy for Software Development (QPSD)). The target areas in this
requirement were:

Delivered defects and delivered defects per size;

Total effectiveness throughout the process;

Adherence to schedule;

Estimation accuracy;

Number of open customer problems;

Time that problems remain open;

Cost of nonconformance;

Software reliability.

There actually were some conflicts by all the different business units on deciding the proper
definition on terms like software problem, fault, and many more. These were to be
eventually settled though for the progress of the companys software metrics goals. After the
definitions and other small problems were settled, a Goal/Question/Metric structure was decided,
and this structure had 7 goals:
1. Improve project planning,
2. increase defect containment,
3. increase software reliability,
4. decrease software defect density,
5. improve customer service,
6. reduce the cost of nonconformance
7. increase software productivity
Using these goals, they went on to crease metrics based on the scope.
Below, Ive listed some of the metrics that were borne to address these goals. These metrics are
by no means set in stone and can always change over time due to feedback and effectiveness:
Metric 1.1: Schedule Estimation Accuracy (SEA) = Actual Project Duration / Estimated
Projection Duration (goal 1)
Metric 1.2: Effort Estimation Accuracy (EEA) = Actual Project effort / Estimated Project Effort
(goal 1)

Metric 2.1: Total Defect Containment Effectiveness (TDCE) = Number of pre-release defects /
(number of pre-release defects + number of post-release defects) (goal 2)
Metric 2.2: Phase Containment Effectiveness for phase I (PCEi) = number of phase I errors /
(number of phase I errors + number of phase I defects) (goal 2)
Metric 3.1: Failure Rate (FR) = number of failures / execution time (goal 3)
Metric 4.1a: In-Process Faults (IPF) = in-process faults caused by incremental software
development / assembly-equivalent delta source size (goal 4)
Metric 4.1b: In-Process Defects (IPD) = in-process defects caused by incremental software
development / assembly-equivalent delta source size (goal 4)
Metric 4.2a: Total Released Defects total (TRD total) = number of released defects / assemblyequivalent total source size (goal 4)
Metric 4.2b: Total Released Defects delta (TRD delta) = number of released defects caused by
incremental software development / assembly-equivalent total source size (goal 4)
Metric 4.3a: Customer-Found Defects total (CFD total) = number of customer-found defects /
assembly-equivalent total source size (goal 4)
Metric 4.3b: Customer-Found Defects delta (CFD delta) = number of customer-found defects
found by incremental software development / assembly-equivalent total source size (goal 4)
Metric 5.1: New Open Problems (NOP) = total new post-release problems opened during the
month (goal 5)
Metric 5.2: Total Open Problems (TOP) = total number of post-release problems that remain
open at the end of the month (goal 5)

Metric 5.3: (mean) Age of Open Problems (AOP) = (total time post-release problems remaining
open at the end of the month have been open) / (number of open post-release problems remaining
open at the end of the month) (goal 5)
Metric 5.4: (mean) Age of Closed Problems (ACP) = total time post-release problem closed
within the month were open / number of post-release problems closed within the month (goal 5)
Metric 6.1: Cost of Fixing Problems (CFP) = dollar cost associated with fixing post-release
problems within the month (goal 6)
Metric 7.1a: Software Productivity total (SP total) = assembly-equivalent total source size /
software development effort (goal 7)
Metric 7.1b: Software Productivity delta (SP delta) = assembly-equivalent total source size /
software development effort (goal 7)

Alongside the goals and metrics, Motorola encouraged each software project/business unit to
create their own goals using the agreed-upon metrics based on how they were faring (baseline).
Not to say that one area would be more important than the other, but there were some that stood
out. One of the most important areas was the defect data area. Because if properly analyzed and
addressed, the information gathered would lead to significant improvement. Other metric areas
such as estimation accuracy and software problem-related really helped the units/projects get a
better grip on things.
To complete their engraining of software metrics into their company and its culture, Motorola
wanted to build a software metrics infrastructure, not only would this include the MWG, it would

go to include the Metrics User Group (MUG). This group was established as a forum to share
experiences of software metrics implementation. This group would meet every quarter to share
their experiences with things like tools to automate metrics. Also, there were many additions
and outputs created by this infrastructure. Bullet-pointed below are some of these changes
established by the MWG:

The clarification of metric definition and interpretation.

Two-day training workshops set up for training and consulting support of metric
implementation

A standardized requirements for the automation of data collection and analysis to give to
the tools group making them.

A criteria for the evaluation of metric tracking systems to choose one to buy if necessary.
Also, theres a current list of all the metric tools available if anyone wants to use it.

Support for analysis of the metric data collected and some generic defect classification
scheme and their respective examples for using them to recommend improvements.

Guidelines to whichever unit is interested in creating a function responsible for software

metrics implementation.

A method for software measurement technology assessment and it provides feedback on

priority items that would help the implementation cause further.

Surveys involving customer satisfaction (from the software perspective).

One of MWGs great contribution is their metric and process definition for software review and
test processes. A survey of the software engineers and managers showed that 67% of them use a
software review package made by MWG. The MWG was also faced with requests to centralize

all of the metric data but they decided that it was best to keep the data localized to which ever
entity needed it.
So how did software metrics turn out for Motorola? Wed have to say pretty well! Motorolas
focus on improving their software quality through metrics turned in an astounding 50 time
reduction in software defect density within 3.5 years. Thats amazing. This reduction led
straight to cost reduction and cycle time reduction too! Not to boot that in 1988, the Motorola
Company was awarded the First Malcolm Baldrige national Quality award.
Example 2b. Hewlett - Packard
Another well-known successful metric-user is Hewlett-Packard (HP). One of HPs great story is
how the use of software inspection metrics helped them achieve different levels of success in
finding defects early in the software development life cycle. Its not surprising to say that by
finding those early defects, HP went on to save countless effort and money. This successful
project which they embarked on was called the sales and inventory tracking (SIT) project.
Though there were many various goals for this project, it all involved the same type of data:
computer dealer sales and inventory levels of HP products. Because that was the case, HP
decided to centralize all of the data for the SIT project to make it easy access.
HP followed their normal software life cycle of investigation, design, construction and testing,
implementation and release, and post-implementation review. But in this case, the metrics were
only being used for inspection and testing in the design, and construction and testing phase. In
the inspection process, HP broke it down into an 8 step process:

Step 0: Planning
Step 1: Kickoff
Step 2: Preparation
Step 3: Issue and Question Logging
Step 4: Cause Brainstorming
Step 5: Question and Answer
Step 6: Rework
Step 7: Follow-up
On a funny side note, HP found it much more effective to change the term defect to issue and
question logging because they realized calling something a defect makes the author/engineer
less receptive to the process of improvement.
So which metrics did HP decide to collect for this SIT project? There were three forms used to
collect the inspection metrics. First one was the inspection issue log. The second was the
inspection summary form. And the last was the inspection data summary and analysis form.
From these, HPs team selected for key metrics and those were:
1. Number of critical defects found and fixed
2. Number of noncritical defects found and fixed
3. Total time used by inspections
4. Total time saved by inspections

Their testing process was broken down into test planning, unit testing, module testing, and
system testing. Of course not all of these are created equal and theres only a finite amount of
resources so HP created their Master Test Plan after their assessment to know where to spend the
time and effort at. The system that was to be inspected would be broken down into logical
modules. Each module performed a different specific function. The Master Test Plan told of
how the primary and secondary features were to be tested against their respective product
specifications and how accurate the output of the data was, not to mention the given goals of
optimum speed and efficiency, ease of user use, and system supportability.
It was the risk assessments turn in their test planning phase. They took the modules and their
sub modules and assigned a risk number to it. That number involved their operational
importance to overall system functionality, complexity, and technical difficulty.
The unit testing was done on all the jobs and program streams. Each program and job stream
were counted as an individual entity. HP had the author of those entities to do their own testing
respectively to save money and time. Those authors then filled out the forms for further analysis.
The module testing was more of a double-checking type test. Since after the unit tests were
done, the module testing was there just to ensure that all of them worked together in a nice
cohesive manner. After that came the system testing. HP decided to conduct the system test
along with a pilot test at one of their HP dealers. The values gathered there was then inputted
into their central SIT database. The testing done had their own metrics, and there were two kids
of those used to measure their testing effort:
1) Total number of critical defects and
2) Total testing time for each phase

Which looked like this:

Figure 9: Testing Metrics

So how did HP decide if all of this was worth it or not? HP used an ROI model to measure the
value of inspections and testing in terms of how much time it saved and how quick it got it to the
market. Their reasoning, which is a sound one, is that there should be a value put on how much
time is saved if a defect was found in the inspection/testing phase than if it was found after at the
system test phase.
The model had these metrics to be gathered and analyzed:

Total Time Saved = Total Time Saved by inspection + Total Time Saved by Unit and Module
Testing
Total Time Used = Total Time used by inspection + OTtal Time used by Unit and Module
Testing
Prerelease ROI = Total Time Saved / Total Time Used
Inspection ROI = Total Time Saved by Inspection / Total Time used by inspection
Testing ROI = Total Time Saved by Unit and Module Testing / Total Time used by unit and
module testing
Total Time Saved by Inspection = time saved on critical defects + time saved on noncritical
defects
Total Time used by inspection = Inspection Time + Time to Fix and follow up for defect
resolution
Time Saved on Critical Defects = Black Box Testing Time x Number of Critical Defects Total
Time Used
Mean Total Time to Rework (MTTR) = Time to Find defect + time to Fix defect + Time to
release to Production
Time Saved on Noncritical Defects = MTTR x Number of Noncritical defects
Total Time Saved = Time Saved on Critical defects
Total Time Used = Time to design and build a test + time to execute + time to find and fix a
defect

With those metrics, and the chart below:

Figure 10: Defect Summary

HP was able to conclude that their return on investment in the prerelease was a whopping 355%:

Figure 11: Time Data and Return

Thus another case of software metrics gone right. It is good to have the numbers back up the
practice.
Conclusion
We have just shown two examples of real world applications of software metrics in action. In
both cases we saw how successful they were and if done right, using software metrics can benefit
the project/company in quite a big way. We also see, though, that it takes a lot of effort and
commitment to make software measurement work. These two companies are actually in the
minority of real-life examples gone right. Today, there is still resistance to the implementation
and follow through of solid software metrics. It takes time, effort, cooperation, and a lot of
communication to get it done, but it is well worth it.

5. What are the advantages of using software metrics?

We have gathered a list of all of the possible advantages to software metrics which are as
following:

The basic advantage of software metrics become apparent during the process of
providing feedback to managers about the evolution and quality of the software
during several phases of software development life cycle.

In order to analyze and compare different programming languages, considering their

differences in characteristics, using software metrics is one of the best options.

One of the possible usages of software metrics is in the defining of software quality
specifications.

Software metrics can simply be used to enhance the comparison process between the
varieties of design approaches for software systems.

When it comes to checking software systems requirements in accordance with

specifications, one may benefit from software metrics during the verification process.

Software metrics are beneficial when you need a prediction about the amount of
effort needed to be put in the design and development of the software systems. It may
be possible by analyzing the old data related to some similar standard software
processes.

Software metrics can be used to measure the complexity of the software under
development.

When it comes to comparing and assessing the proficiency of individuals in software

development process, one can definitely take advantage of software metrics
approaches.

In some circumstances you need to divide the complicated modules in your system to
smaller pieces in order to reduce the percentage of softwares complexity. Using
software metrics can help you to decide when the correct time is to stop the division
process.

For appropriate utilization decisions resource managers need to use software metrics
to provide them a correct view.

In situations that you need to make important design decisions for software
developments issues related to maintenance costs you can basically take advantage of
software metrics to make design tradeoffs.

When you are trying to find the best solution for allocating your available resources
in order to test the developed softwares ode, software metrics seem such a useful
tool.

5.1 Limitations of Software Metrics

Software metrics are difficult to apply in some cases, and that increases the cost of
application.

Since software metrics are based on data that has been recorded in the past, its
authenticity is difficult to verify.

Software products can be managed, and their quality could be verified but Software
Metrics are not useful to evaluate the performance of the engineers.

Most of the applications of Software Metrics are based on assumptions, whose definitions
and derivations are not verified.

As you can see above, all the methods and tools of discussed are very old, and empirical.

Estimation is key in Software Metrics, and verifying the authenticity of these estimations
is very hard.

6. References

1. Mishra, Suchitra. "Five Project Management Performance Metrics Key to Successful

Project Execution Operational Excellence." Suchitra Mishra. N.p., 30 Sept. 2013. Web.
03 Dec. 2014.
2.

Subramaniam, Anand. "Project Metrics & Measures." Project Metrics & Measures.
Slideshare.net, 28 July 2009. Web. 03 Dec. 2014.

3. "Earned Value Management." Wikipedia. Wikimedia Foundation, 29 Nov. 2014. Web.

02 Dec. 2014.
4. Jonathan, Louis. "Estimating the Value of Inspections and Early Testing for Software
Projects." Estimating the Value of Inspections and Early Testing for Software Projects
(1994): n. pag. Www.hp.com. Hp, 12 Dec. 1994. Web. 29 Nov. 2014.
5. M, Daskalantonakis. "A Practical View of Software Measurement and Implementation
Experiences within Motorola." IEEE Xplore. Software Engineering, IEEE Transactions
on (Volume: 18, 06 Aug. 2012. Web. 30 Nov. 2014.
6. Kan, Stephen. "Metrics and Models in Software Quality Engineering." Metrics and
Models in Software Quality Engineering. Addison-Wesley Longman Publishing, 12 May
2003. Web. 03 Dec. 2014.
7. Rex Black. Metrics for Software Testing: Managing with Facts Part 4: Product Metrics
http://www.rbcs-us.com/images/documents/Metrics-Article-4.pdf Web. 03 Dec. 2014.