Introduction Quality Assurance (QA): standards Quality Planning Quality Control Product quality Process quality

Introduction to software quality evaluation

Master on Free Software Felipe Ortega, Daniel Izquierdo, Pedro Coca
jfelipe@libresoft.es GSyC/Libresoft

19 de noviembre de 2010

Felipe Ortega, Daniel Izquierdo, Pedro Coca

Introduction to software quality evaluation

Introduction Quality Assurance (QA): standards Quality Planning Quality Control Product quality Process quality

(cc) 2010 Felipe Ortega, Daniel Izquierdo, Pedro Coca. Some rights reserved. This document is distributed under the Creative Commons Attribution-ShareAlike 3.0 licence, available in http://creativecommons.org/licenses/by-sa/3.0/

Felipe Ortega, Daniel Izquierdo, Pedro Coca

Introduction to software quality evaluation

Introduction Quality Assurance (QA): standards Quality Planning Quality Control Product quality Process quality

Index
1 2 3 4 5 6

Introduction Quality Assurance (QA): standards Quality Planning Quality Control Product quality Process quality

Felipe Ortega, Daniel Izquierdo, Pedro Coca

Introduction to software quality evaluation

Introduction Quality Assurance (QA): standards Quality Planning Quality Control Product quality Process quality

What is software quality?

Complex concept, dierent from quality in manufacturing.

Product developed to meet its specication.

But it is dicult to write good software specication. Maintainability, reusability, modularity, security, etc. are usually not included (but required). Quality is linked to meeting end-user expectations (subjective).

Felipe Ortega, Daniel Izquierdo, Pedro Coca

Introduction to software quality evaluation

Introduction Quality Assurance (QA): standards Quality Planning Quality Control Product quality Process quality

Factors that may improve quality

OOP, CASE support. Standards (development and organizational). But bureaucracy is not enough. Quality culture: experience, best practices, management.

Felipe Ortega, Daniel Izquierdo, Pedro Coca

Introduction to software quality evaluation

Introduction Quality Assurance (QA): standards Quality Planning Quality Control Product quality Process quality

Main areas of software quality in large projects

Quality assurance: Dening framework for organizational quality. Quality planning: Selecting elements from the framework, adapted to a particular project. Quality control: Checking that development teams adhere to the plan. For many community FLOSS projects, none of the above are dened. Problems for companies with this methods in place to integrate community FLOSS.

Felipe Ortega, Daniel Izquierdo, Pedro Coca

Introduction to software quality evaluation

Introduction Quality Assurance (QA): standards Quality Planning Quality Control Product quality Process quality

Process and product quality

In traditional manufacturing, process quality is tightly linked to nal product quality. But software involves design, creativity. Subject to past experience and team skills. Dicult to measure some aspects: maintainability, reusability, etc. Obscure links between changes in process and software product quality. Blind application of process quaility is not good for all situations.

Felipe Ortega, Daniel Izquierdo, Pedro Coca

Introduction to software quality evaluation

Introduction Quality Assurance (QA): standards Quality Planning Quality Control Product quality Process quality

Index
1 2 3 4 5 6

Introduction Quality Assurance (QA): standards Quality Planning Quality Control Product quality Process quality

Felipe Ortega, Daniel Izquierdo, Pedro Coca

Introduction to software quality evaluation

Introduction Quality Assurance (QA): standards Quality Planning Quality Control Product quality Process quality

Quality assurance overview

Quality assurance

Product quality

Process quality

Design Requirements Code style Code organiz. Change request

Design Documentation Version release Project plan Testing and UX

Felipe Ortega, Daniel Izquierdo, Pedro Coca

Introduction to software quality evaluation

Introduction Quality Assurance (QA): standards Quality Planning Quality Control Product quality Process quality

Role of software development standards

Try to convey best practices in software development. Oer required framework for QA. Many examples: US DoD, ANSI, IEEE, BSI, NATO, etc. Engineers usually reluctant to adopt standards (bureaucracy).

Felipe Ortega, Daniel Izquierdo, Pedro Coca

Introduction to software quality evaluation

Introduction Quality Assurance (QA): standards Quality Planning Quality Control Product quality Process quality

ISO 9000
International body of standards for quality in any industry. ISO 9001 is the most extended version.
Quality in process, development and maintenance of any product. Mainly concerned about high level procedures. Forces denition of specic metrics to measure quality. Involves periodic reviews to maintain ISO 9001 label award.

Involve denition of documentation templates, and documenting all processes. Does not ensure that standarized process reects best practices in the eld.

Felipe Ortega, Daniel Izquierdo, Pedro Coca

Introduction to software quality evaluation

Introduction Quality Assurance (QA): standards Quality Planning Quality Control Product quality Process quality

Documentation standards

3 dierent categories: Documentation process standards: Establish good process for document production. Documentation standards: Dene structure and presentation of documents. Document interoperability standards: Compatibility between document formats.

Felipe Ortega, Daniel Izquierdo, Pedro Coca

Introduction to software quality evaluation

Introduction Quality Assurance (QA): standards Quality Planning Quality Control Product quality Process quality

Documentation standards

Some examples Document identication: Unique names, easy tracking and identication. Document structure: Homogeneous structure for docs of similar character. Document presentation: Customized layout and look and feel (design and presentation). Document update: Tracking changes/updates/reviews.

Felipe Ortega, Daniel Izquierdo, Pedro Coca

Introduction to software quality evaluation

Introduction Quality Assurance (QA): standards Quality Planning Quality Control Product quality Process quality

Index
1 2 3 4 5 6

Introduction Quality Assurance (QA): standards Quality Planning Quality Control Product quality Process quality

Felipe Ortega, Daniel Izquierdo, Pedro Coca

Introduction to software quality evaluation

Introduction Quality Assurance (QA): standards Quality Planning Quality Control Product quality Process quality

Quality planning overview

5 dierent aspects: Product overview: Description, target market, user expectations. Product plan: Releases, updates, support, distribution, etc. Process description: Product development and management. Quality goals: Including identication of quality attributes to be tracked. Risks and risk management: Risks aecting quality and corrective actions.

Felipe Ortega, Daniel Izquierdo, Pedro Coca

Introduction to software quality evaluation

Introduction Quality Assurance (QA): standards Quality Planning Quality Control Product quality Process quality

Software quality attributes

Safety Security Reliability Resilience Robustness Correctness Clarity Testability Adaptability Complexity Modularity Maintainability Portability Usability Reusability Efficiency Learnability Verifiability

Interoperability
Felipe Ortega, Daniel Izquierdo, Pedro Coca

Installability
Introduction to software quality evaluation

Introduction Quality Assurance (QA): standards Quality Planning Quality Control Product quality Process quality

Index
1 2 3 4 5 6

Introduction Quality Assurance (QA): standards Quality Planning Quality Control Product quality Process quality

Felipe Ortega, Daniel Izquierdo, Pedro Coca

Introduction to software quality evaluation

Introduction Quality Assurance (QA): standards Quality Planning Quality Control Product quality Process quality

Documentation standards

3 components: Quality reviews: Periodic checks of conformity with quality standards dene in the project. Automated assessment: Collecting quality information automatically (product and process). Quality metrics: Measure dierent quality aspects as an input for assessment.

Felipe Ortega, Daniel Izquierdo, Pedro Coca

Introduction to software quality evaluation

Introduction Quality Assurance (QA): standards Quality Planning Quality Control Product quality Process quality

Quality reviews

3 components: We need a dedicated team of reviewers. Systematic methodology and approaches to examine key quality properties dened for the project.
Design or program inspections. Progress reviews (management). Quality reviews: Technical analysis of components and assessment from metrics.

Felipe Ortega, Daniel Izquierdo, Pedro Coca

Introduction to software quality evaluation

Introduction Quality Assurance (QA): standards Quality Planning Quality Control Product quality Process quality

Software measurement and metrics

Sofware metric: Measurement linked to a software system, process or related documentation. Examples: SLOC, complexity, coupling, connascence, etc. Usually, the main problem is how to dene meaningful metrics.
We can measure many dierent aspects and elements. But do they describe the quality aspect we want to track? 2 types: Control metrics (process) and predictor metrics (product).

Key application: predicting possible problems for proactive management and control.

Felipe Ortega, Daniel Izquierdo, Pedro Coca

Introduction to software quality evaluation

Introduction Quality Assurance (QA): standards Quality Planning Quality Control Product quality Process quality

Measurement procedure

Select metrics

Identify interesting components/elements

Corrective actions applied

Perform measurements

Check atypical elements/results

Detection of anomalous results

Felipe Ortega, Daniel Izquierdo, Pedro Coca

Introduction to software quality evaluation

Introduction Quality Assurance (QA): standards Quality Planning Quality Control Product quality Process quality

Index
1 2 3 4 5 6

Introduction Quality Assurance (QA): standards Quality Planning Quality Control Product quality Process quality

Felipe Ortega, Daniel Izquierdo, Pedro Coca

Introduction to software quality evaluation

Introduction Quality Assurance (QA): standards Quality Planning Quality Control Product quality Process quality

Product metrics

Static metrics

Dynamic metrics

Fan-in/Fan-out

Depth inheritance tree

Length of code

Method fan-in/fan-out

Cyclomatic complexity Weighted methods per class Fan-in/Fan-out Number overriding operations

Depth conditional testing

Fog index
Felipe Ortega, Daniel Izquierdo, Pedro Coca Introduction to software quality evaluation

Introduction Quality Assurance (QA): standards Quality Planning Quality Control Product quality Process quality

Product metrics
Static metrics:
Fan-in/Fan-out: Number of functions or methods that call some other function or method (complexity, connascence). Length of code: Size of program. SLOC (LOC or LLOC). Cyclomatic complexity : Metric for control complexity of software. Length of identiers: Average length of identiers used in a program. Supposedly, the longer the ID, the better for readability and maintenance of code. Depth of conditional nesting : Deeply nested statements are harder to be grasped. Fog index: Average length of words and sentences in documents.
Felipe Ortega, Daniel Izquierdo, Pedro Coca Introduction to software quality evaluation

Introduction Quality Assurance (QA): standards Quality Planning Quality Control Product quality Process quality

Product metrics
Dynamic metrics:
Depth of inheritance tree: Number of discrete levels in the tree of classes (OOP). The deeper the tree, the more complex the desgin. Method fan-in/fan-out: Distinction between origin of calls to other methods (from object or from external methods). Weighted methods per class: Number of methods included in a class, weighted complexity of every method. Too complex classes are dicult as for understanding and maintenance. Number of overriding operations: Operations overridden in sub-classes. The higher this number, the less appropriate may be the super-class.

Felipe Ortega, Daniel Izquierdo, Pedro Coca

Introduction to software quality evaluation

Introduction Quality Assurance (QA): standards Quality Planning Quality Control Product quality Process quality

Metrics assessment
It is not enough to collect dierent metrics. We need to infer assessments and conclusions from these results. Most of times, human interpretation of results is needed. Value ranges drawn from multiple case examples may serve as guidelines. Usually, supported through integrated assessment frameworks (Business Intelligence).
Organization of measurement areas and lists of metrics. Automated production of reports summarizing results. Support for elaboration of assessment reports and executive summaries. Other buit-in functionalities: data mining, data warehouse, statistical tools, etc.
Felipe Ortega, Daniel Izquierdo, Pedro Coca Introduction to software quality evaluation

Introduction Quality Assurance (QA): standards Quality Planning Quality Control Product quality Process quality

Index
1 2 3 4 5 6

Introduction Quality Assurance (QA): standards Quality Planning Quality Control Product quality Process quality

Felipe Ortega, Daniel Izquierdo, Pedro Coca

Introduction to software quality evaluation

Introduction Quality Assurance (QA): standards Quality Planning Quality Control Product quality Process quality

Cycle of process improvement

Metrics and reports

Analysis

Measurement

Assessment and recommendations

Change
Improvements to be checked
Felipe Ortega, Daniel Izquierdo, Pedro Coca Introduction to software quality evaluation

Introduction Quality Assurance (QA): standards Quality Planning Quality Control Product quality Process quality

Process improvement

Quality frameworks focused on improving the process, rather than just describing it. 3 stages:
1

Process measurement: Compilation of metrics and indicators related to software process (development and management). Process analysis: Producing meaningful assessment from results, identifying or anticipating problems that must be corrected. Process change: Alter problematic elements or practices to align the process to the quality goals of the project.

Felipe Ortega, Daniel Izquierdo, Pedro Coca

Introduction to software quality evaluation

Introduction Quality Assurance (QA): standards Quality Planning Quality Control Product quality Process quality

Link between process and product

Humphrey, in his book about process management (1988), states that the same techniques applied to any kind of industry are valid for sofware processes. However, a number of caveats are present in sofware projects.
The design phase plays a key role in the whole process and results. Skills and experience of team leaders and experts aect quality of results. Dierent projects may need dierent methodologies and approaches. Great inuence of 3 variables: cost, time frame and schedule.

Felipe Ortega, Daniel Izquierdo, Pedro Coca

Introduction to software quality evaluation

Introduction Quality Assurance (QA): standards Quality Planning Quality Control Product quality Process quality

Process metrics

Process metrics should be aimed to facilitate process improvement. 3 types:

Time to complete the process. Resources required to carry on the process. Number of occurrences of a particular event to be monitored.

The key question is to identify what we need to measure.

Felipe Ortega, Daniel Izquierdo, Pedro Coca

Introduction to software quality evaluation

Introduction Quality Assurance (QA): standards Quality Planning Quality Control Product quality Process quality

The GQM methodology

Frequently applied to develop customized process quality frameworks. Goal-oriented, top-down methodology to measure quality of software processes and products. First developed at University of Maryland (1984) in collaboration with NASA. Hierarchical structure composed by 3 kinds of elements: goals, questions and metrics.

Felipe Ortega, Daniel Izquierdo, Pedro Coca

Introduction to software quality evaluation

Introduction Quality Assurance (QA): standards Quality Planning Quality Control Product quality Process quality

GQM Hierarchy

Goal 1

Goal 2

[...]

Goal N

Q1

Q2

Q3

Q4

Q5

Q6

Metric 2 Metric 8 Metric 1 Metric 3 Metric 4 Metric 5

Felipe Ortega, Daniel Izquierdo, Pedro Coca Introduction to software quality evaluation

Metric 9 Metric 10

Metric 7 Metric 6

Introduction Quality Assurance (QA): standards Quality Planning Quality Control Product quality Process quality

GQM implementation

We need to think at 3 dierent levels:

Conceptual (goals): Goals must focus on an object, is aimed to certain purposes, pursue one or several quality focuses, examine the object from one or dierent points of view and it is dened within a certain context. Operational (questions): Break down one or several roles into concrete issues to be inspected. Quantitative (metrics): Dene measurable quantities or logical aspects that providing information to oer precise answers and assessment in response to the set of questions.

Felipe Ortega, Daniel Izquierdo, Pedro Coca

Introduction to software quality evaluation

Introduction Quality Assurance (QA): standards Quality Planning Quality Control Product quality Process quality

GQM Example

Goal

Object
Product vers. Alpha

Purpose
Understanding

Quality focus Point of view

Reliability

Context

SW develop. team Project A, Company X

Aspects of quality and metrics 1. Number of failure - total number - severity (sorted by criticality) 2. Number of faulty modules - total number - sorted by modules - sorted by lifecycle phases of detection Baseline hypotheses 1. Number of failure - total number: 115 - estimates John: 30% critical, 70% uncritical - estimates Elsa: 15% critical, 60% uncritical, 15% others 2. Number of faulty modules - total number: 200 Modules - AlphaH (40 faults), AlphaD (25), AlphaF (10)...

Influencing factors 1. Degree of Reuse 2. Experience of Development Team Members 3. Adherence to Code Inspection Process

Impact of influencing factors in baseline hypotheses 1. Higher the degree of reuse lower num. failures. 2. The longer the experience of the developement team members, the lower the number of failures and faults. 3. The closer the adherence to the code inspection process, the lower the number of failures and faults in testing and integration

Felipe Ortega, Daniel Izquierdo, Pedro Coca

Introduction to software quality evaluation

Introduction Quality Assurance (QA): standards Quality Planning Quality Control Product quality Process quality

References

Humphrey, W. S. Characterizing the software process. IEEE Software, 5(2), 73-79. 1988. Ian Sommerville, Software Engineering, 8th Edition. Addison Wesley, 2006. Hans van Vliet, Software Engineering: Principles and Practice, 3rd Edition. Wiley, 2008. V. R. Basili, Quantitative Evaluation of Software Engineering Methodology, Proceedings of the First Pan Pacic Computer Conference, Melbourne, Australia, Vol. 1 pp. 379-398, 1985.

Felipe Ortega, Daniel Izquierdo, Pedro Coca

Introduction to software quality evaluation