Enhancing ISO/lEe 25021 Quality Measure

Elements for wider application within ISO 25000

serIes
Dominique St-Louis, Witold Suryn
ETS - Ecole de Technologie Superieure
I d.stlouis@videotron.caJ witold.suryn@etsmtl.cal
Abstract - This article presents an innovative approach to the
determination of a core set of Quality Measure Elements
(QME), and a comparative analysis aimed at enhancing
applicability of ISOII E G 25021, which are critical to the
ISOIIEG 25000 SQuaRE series of standards. The research
presents the methodology, allowing for moving from an outdated
and extensive set of measu res, to a core list of base measu res,
which can be further diversified through contexts of use,
parameters, and considerably simplify the development of
derived measures.
Keywords- Sofware Quality Measure Elements, Base
Measure, QME Categorization, Knowledge Base Repository.
I. INTRODUCTION
This research identifes three major opportunities for
improvement of the current ISO/IEC 25021 standard: better
accessibility for the sofware development industry,
consistency through abidance to ISO sofware and system
engineering vocabulary [19] and ISO metrology standard [6],
and fnally the design of the standard itself.
Complexity of ISO/IEC 25021, a similar concern to the one
expressed about ISO/IEC 9126, makes using this standard
difcult. It is also the reason why sofware quality measures
remain a vague or non-existent concept for many companies
to this day. Making the quality measurement-related part of
ISO/IEC 25000 SQuaRE series of standards readily available,
usefl, inexpensive to implement, and easy to use, requires
the development of a concise and scientifcally sound set of
basic measurement construction modules, as well as Quality
Measurement Elements or QMEs. Consistent adherence to
the measurement theory [6] would improve the credibility of
the whole series of standards, while making it scientifcally
correct, increasing its reliability, and enhancing its
trustworthiness amongst the scientifc community. As
ISO/IEC 25021 makes the basis for the development of
several other standards in the SQuaRE series (ISO/IEC
25022-23 replace [2], [3] and [4]) the importance of its
correct design is crucial.
The objective of the presented research was to determine a
diferent (than in ISO/IEC 25021) approach to defning a
precise set of base quality measures, which would serve as a
basis for further development of the majority of derived
quality measures. The results obtained provide such set of
base measures that are easy to understand by the sofware
industry professionals, compliant with ISO/IEC 15939
standard and propose a design that allows for intuitive
construction of derived measures.
This paper describes the development of the process
allowing for selecting quality measures fom an existing
catalogue and the method used to determine a core set of
QMEs. Also, the development of a digital knowledge base for
sofware quality measures proposed over the course of this
research would be one of its most important results, as it
potentially leads to opportunities to frther develop the
SQuaRE series.
II. LITERA TOE RVW
Several eXlstmg scientifc articles present critical
judgments of ISO/IEC sofware quality and measurement
standards. The most renowned analyses are briefy discussed
below.
ISOIIEG 9126 and its Technical Repors
In 1991, ISO/IEC 9126 was published as the frst
consensus on the terminology for quality characteristics of
sofware evaluation. Between 2001 and 2004, ISO published
an expanded version containing the quality model and the
measures proposed for this model [8]. The current version of
the ISO/IEC 9126 standard [1][2][3][4] consists of one
normative reference, the Quality models [1], and three
technical reports: External metrics [2], internal metrics [3],
and Quality in use metrics [4].
Researchers have found that ISO/IEC 9126 standard has
some major faws, even if it seems the best available so
far. For one, the terminology of ISO/IEC 9126 is not
consistent with the classical terminology of measurement
science and engineering, which causes a real concern for the
validity of proposed measures. Also, measures in ISO/IEC
9126 in many cases are not identifed and/or classifed as
base or derived measures, which stays in confict with
ISO/IEC 15939 measurement information model [8].
ISO standards 24765 [19] and 25000 both refer to ISO/IEC
15939 [6] defnition of terms for base and derived measures
which are:
- Base measure: measure defined in terms of an attrbute and the
method for quantifing it. A base measure is functionally
independent of other measures.
978-1-4673-2421-2/12/$31.00 2012 IEEE 3120
- Derived measure: measure that is defined as a function of two or
more values of base measures.
Abran et al. [8] analyzed the ISO/IEC 9126 standard and
suggested ways to improve it:
The standard should comply with the ISO and general terms
of metrology [10][17].
Measures in [1][2][3] and [4] are merged and should be
reorganized into base and derived measures to comply with
ISO/IEC 15939 [10].
Map the information to the measurement model provided by
ISO/IEC 15939 [6].
Measurement results described by ranges such as [0 .. 1]
could be presented qualitatively. A score of 100% could be
described as excellent, and a score of 80% as very good
[11][12].
Other researchers have also demonstrated the impairments
in the structure of the ISO/IEC 9126 quality model. lung et
al. argued that the use of the six sub-characteristics of
Compliance leads to two kinds of results: (I) each
compliance sub-characteristic belongs to its characteristic as
defned in ISO/IEC 9126, (2) the six compliance sub
characteristics form a separate dimension [13]. The results of
their research showed an ambiguity in the structures of the
characteristics and sub-characteristics in ISO/IEC 9126. As
for Abran et al. [15] they suggest reviewing the structure of
the Usability model by adding Learabilit and Security,
based on their analysis of ISO/IEC 25000 SQuaRE. Chua and
Dayson [14] discovered that the Feasibilit characteristic
adds specifc factors such as Consistency, Simplicity,
Legibilityand Col or use. They also suggest adding the Help
sub-characteristic to Usability characteristic and the global
characteristic of User satisfaction.
In the subject of the implementation of the measures in
industry, Boegh [9] indicated that the measures listed in
ISO/IEC 9126 could not be applied directly because they
entailed a degree of imprecision. In addition, the author
mentioned that engineers in the industry should devote
considerable efforts to redefne the measures and implement
formalized approaches.
ISO/IEC 2500 - SQuaRE
Pushed by its own pace of evolution, the progresses in
information technology [8] and the maturity of the industry,
ISO recognized the need of modifcations of the past quality
standards. As the result, ISO/IEC SC7 developed a new
generation of standards in the sofware quality domain,
creating the Sofware Quality Requirements and Evaluation
(SQuaRE) series of standards.
Being the part of this new series ISO/IEC 25021 has the
purpose "to defne and/or design an initial set of QME ... " In
its 2007 edition of ISO/IEC 25021 [5] ISO/IEC JTCI SC7
addressed several of the issues identifed in ISO/IEC 9126.
For example, the standard is now consistent with the theory
of metrology as well as the sofware and system engineering
vocabulary. Concerning the reorganisation of the measures,
ISO/IEC 25021 has identifed and classifed a set of base
measures in addition to the current development of other
standards concerning derived measures. Moreover, one of its
key goals was the harmonization and coordination with
ISO/IEC 15939 [8]. The standard also provides a set of rules
for the design of QMEs and for verifcation of the design of
existing QMEs. Its content provides the link between
ISO/IEC 9126, and the subsequent SQuaRE series. ISO/IEC
2502n-Quality Measurement Division is made of six
standards:
25020: Measurement reference model and guide
25021: Quality measure elements
.25022: Measurement of quality in use (revision of 9126-4)
25023 :Measurement of system and sofware product quality
(revision of 9126-2 & 3)
.25024: Measurement of data quality
.25025: Measurement of IS service quality
III. RESEARCH APPROACH
Selection of a Repository of measures
The initial research approach was to propose a set of base
measure elements that would be at the same time science
based (i.e. following metrology principles), usefl and
accessible to the sofware industry and utilized as an
international reference. The catalog of measures was to be
built with all the existing quality measures known in
international reference set of ISO/IEC JTCI documents
[2][3][4][5] and [7]. Besides of the SQuaRE series of
standards no other internationally recognized sofware
measurement standards referring to ISO/IEC 15939 principles
were found. The measures published in ISO/IEC 9126
standard [2][3][4] were taken as the basis of this research
because they make the foundation for the development of the
SQuaRE series and at the same time the only available
complete set of measures.
Due to the extensive list of measures that were found,
(including duplicates and synonyms) a turn to a more
simplistic approach was taken. Only measures in ISO/IEC
9126 parts 2, 3 and 4 [2][3][4] were considered for the
experiment. ISO/IEC 25021 was kept in the repository but
only for the fnal research comparison analysis as it was
considered the control group, while [2][3][4] made the
experiment group.
Although ISO/IEC 25021 has already identi fed a set of
base measures, thus dramatically reducing the number of
QMEs proposed by ISO/IEC 9126, QMEs defned in
ISO/IEC 25021 were not used for the experiment. This
decision was taken in order to diferentiate the experiment
and the control group of QMEs as well as obtaining unbiased
experiment results for the selection of base QMEs fom
ISO/IEC 9126 repository. Only the fnal results of this
research, the core set of QMEs generated by the experiment,
was to undergo a comparison analysis with the baseline
QMEs fom ISO/IEC 25021.
3121
Selection of a core set of base QMEs
The base measure elements, even if they primarily apply to
the sofware quality measurement, must be based on the
standard metrology for systems engineering and sofware,
ISO/lEe 15939. Metrology is the "science of measurement
and its applications" [17]. It "includes all theoretical and
practical aspects of measurement, regardless of the
measurement uncertainty and feld of application ... " [17].
The entire list of measures published in ISO/lEe 9126
parts 2, 3 and 4 were identifed, then stored in a repository,
called "QME Reference Base" 9126-2-3-4 [18]. As Abran
mentioned in [17] the technical reports of ISO/lEe 9126
contain over 250 derived measures and 80 base measures.
Afer the development of the repository [18], the research
team identifed 403 measures including derived and base
measures, while ISO/lEe 25021 had identifed and defned
more than 67 base measures. Of these 403 measures, only
eight were clearly identifed as base measures, and fve more
were ambiguous. The numbers presented above (403) are
unrefned and, as there was no discrimination of the
terminology, it included a large number of repetitive
measures with no classifcation in term of base or derived
measures.
The process presented in Figure 2 summarizes the approach
used in this research for selecting a core set of base QMEs
fom the ISO/lEe 9126 standard. Note that step 6 of the
approach was theoretically verifed but not practically
experimented due to absence of published data.
Base QM E selection
(6) Verification and validation of the accuracYI repeatability and
repetitivity of each baseline measurement.
(5) Testing of each base measure found in step (4) with published data from past or
existing project.
(4) Each base measure found in step (3) must pass the criteria of measure element
and measurement method defined in ISO/lEe 15939.
(3) Identify all common measures found in step (2), and distinguish each
measure according to its contexts of use.
(2) Divide each measure fund in step (1 ) in base elements (must be reduced
to the simplest expression) and identif the contexts of use.
(1) list all the base measures set out in 9126 to 2, 3 and 4 and identify if the
measure is a derived measure i.e. a composite or a base measure.
QME selection
Choice of storage and analsis tool
The creation of the repository, called the QME reference
base was developed with the Microsof Excel [18]. The base
and derived measures proposed in ISO/lEe 9126 parts 2, 3,
and 4, as well as those identifed in ISO/lEe 25021 were all
listed in the reference base or repository.
The reference base was also populated with more
information relevant to inventory. For example, elements of
traceability allowing locating the source of the information
fom the technical reports [2][3][4] were added. Information
ofen referenced was added to the database, thus avoiding
searching the paper versions of the standard. To enhance the
communication, follow-up among the research team,
rationale, questions, comments about classifcation or
organisation were recorded in distinctive felds of the base.
The repository tool itself played a major part in the success
of this experiment. It eased the search and comparison of text
defning the QMEs. Although inexpensive, easy to use,
readily available, and understood by all team members, the
choice of this particular data storage, Excel database, had its
limitations. It was not strong enough to make a systematic
analysis of the data it contained. A relational database
management system (RDBMS) would have allowed for more
volume and capacity, supported more complex data structures
and processing, data evolution and development. Information
in ISO/lEe 9126 could have been superposed to the
terminology fom ISO 24765 [19] to assist in the
interpretation of the text, the organization and the
differentiation of the QMEs and the mapping of the
information to the measurement model of ISO/lEe 15939
(see Figure 8), thus improving the analytical capabilities. In
essence, the choice of Excel over a RDBMS was motivated
strictly by time restriction. Nevertheless, a RDBMS would
have been incontestably a superior repository tool option.
Core QME determination
The measures input into the research reference base were
classifed, categorized and sorted according to the vocabulary
used in ISO/lEe 9126 standard. Following this step, QME
contexts of use were deducted using the metrics information
fom ISO/lEe 9126 technical reports. The metrics
information encompassed the QME names, purpose,
application and measurement method, formula, metric scale,
measure type, and target audience. The resulting inventory
allowed for determining and contrasting base measures fom
derived measures (see example in Figure 3).
1
Technical report
Characteristic
Data Interpretation
91262 External
Attractiveness metrics
Sub-characteristic Interface appearance
customisability
Measure X=A/B X is a derived measure, therefore it
must be decomposed to determine
the base measures which it is
composed of, i. e. A and B
A= Number of interface QME base= Number of interface
elements customised in elements
appearance to user's Context 1= customised in appearance
Satisfaction Context 2= to user's Satisfaction
B= Number of interface QME base= Number of interface
elements that the user elements
wishes to customise Context 1= user wishes to customise
Fig. 3. Example of determination during preliminary analysis ofa base QM
As the next step, harmonization of terms was performed:
quotation marks, hyphens and spaces were removed, cases
were replaced with upper or lower case when appropriate, but
words were not changed. Excluding duplicates and using
identical words, the reference base contained:
3122
a set of 121 base measures
a set of 170 derived measures
Afer these adjustments, the reference base underwent
several reviews by the research team, each time, thoroughly
examining its information. Below the detailed steps followed
during the reference base development.
Step 1: Context deteion
Afer each metrics was dissected to extract the base QME,
it had to be determined if this QME had contexts of use. For
example, the QME Number of failure has been identifed as a
base measure. Its contexts were numerous; it was referred
several times as a data element of metrics found in ISO/IEC
9126. Below, some exemplary contexts for that QME:
- Number of failures detected durng defined tral period
- Number of failures emerqed afer failure is resolved
- Number of failures related to transmission
Step 2: Grouping of identical and/or similar QMs
Once all contexts identifed and reviewed, identical QME
were grouped. This grouping, however, called for some minor
QME names adjustment in cases where contexts were similar.
For example, Number of message for liD related errors QME and
Number of messages for Mean liD fulllment ratio QME were all
harmonized with a plural term. A number of words in the
reference base were found to use the singular and others the
plural form.
Afer these initial groupings, there were still several small
groups of base QME remaining. Groupings extension was
created according to the terminology used in ISO/IEC 9126
standards in order to involve more contexts in the existing
QMEs. ISO/IEC 24765 [19] was used as disceming judgment
tool as it is the terminological dictionary known and broadly
accepted in sofware engineering domain.
Step 3: QMs Categorization
The grouping easily identifed a trend: some base QMEs
were original and other base QMEs, without being secondary,
could be extricated fom the original QMEs. Secondary base
QMEs could be related to primary base QMEs with reference
contexts.
A QMEs categorization (see Table 1) was developed by
using the same reference base, a feld allowing differentiation
of the base QME called "roots" fom the other base QME.
The objective was to clearly identif the primary QME on
which ISO/IEC 15939 measurement model could be applied.
TABLE I
QM CATEGORATION
QME Description
Root base Original
Caused by other base QM. They are not derived, but
Contextualized
base QM whose existence is due to the presence of
base
other QME (e.g. number of errors is a base QME and
number of error messages is also a base QME, but an
error is needed to create an error message).
Singleton base Li sted on Iy once in the reference base.
Obsolete base No longer relevant due to technology evolution.
Derived Derived QME
Note: At this step of the research, derived QME were not supposed to be
found since they had already been broken down into base QM .
Step 4: Validation the core with ISOIEC 15939
The previous steps greatly reduced the amount of
validation work using the templates of ISO/IEC 15939 by
focusing on a small core group of QMEs. Validation work
covered options fom selecting Root base QME with the
largest number of contexts to creating the Singleton base
QME (measures less ofen referred to or sparsely
represented).
The research team created a visual model prior to the
measurement model validation (with ISO/IEC 15939), to
better evaluate the work achieved. The visual template that
maps the Root base QMEs and their various contexts has
been called Contextualized base QME model and is presented
in Figure 4. As for Figure 5, it presents a concrete example of
the QME number of failures and its different contexts of use
as described in ISO 9126-2, 3 and 4.
Fig. 4. Base and Contextualized QM model view
Encounter during
operating concurrently
with other software
Fig. 5. QM Number of failures and its diferent contexts of use.
In the following step, for the validation of these Root base
QMEs, measurement models provided by ISO/IEC 15939
were used to prove that the selected base measures met
indeed the requirements of this standard. In order to formally
defne and assess the Root base QMEs the adjusted template
based on 15939 standards was created. The ISO/IEC 15939
template had to be adjusted to the context of this research and
since many sections of the original template were linked to
derived measures rather than to QME, they were removed,
e.g. measurement fnction.
3123
The practical validation of developed QMEs was the
experimental creation of several derived measures. Figure 6,
presents the outcome for the derived measure A((
Aa1//0/l//, Validation of the F00/0 M/by testing the
measures on existing projects in the industry were not
completed due to confdentiality agreements with partners
and in consequence due to time constraints.
Computed confidence limits
based on the standard
result close to the average
_ = .I Access auditability will be
achieved. Very wide
`
confidence limits suggest a
potentially large departure
and the need for contingency
Fig. 6. QME Number of access and with development of Access auditabilit
as a derived measure.
SLe 5: Comparison of the developed core set of QMs
with ISO/C 25021
The research was concluded by a comparative analysis of
the F00/ 0 M/ identifed by this research with those
listed in ISO/IEC 25021. The existence of each base QME
found by the research was verifed in ISO 25021. When
possible, information such as defnition, measurement
method, and so on was compared. See Table 2 for details.
TALE 2
SOM COMASON ELEMNTS OF TH CORE SET OF QMs
Findings
Core size
Base
measure
comparison
This research ISO 25021
21 measures
67 measures (22 initial measures
plus extensions)
Number of failures
Number of failures
Number of system Failures
Number of faults (code)
Number of faults Number of faults (design)
Number of faults (requirements)
N um ber of messages
Number of errors
=Error context
Number of error messages
Number of messages
Number of internal
N/A
functions
Number of external
N/A
functions "feature"
N/A Duration
IV. RESULTS
The presented research applied a different method of
determination of the core set of QMEs than the one used by
ISO/IEC 25021 WG6 (Work Group 6). The resulting core set
of QMEs is also different fom this proposed in ISO/IEC
25021. Some of the base measures identifed by this research
were not presented in ISO/IEC 25021 and vice versa,
however, the goal of reducing the core set of QMEs fom 67
to much smaller number of 21 valid QMEs has been
achieved.
The original measures reference database, which data was
extracted fom ISO/IEC 9126-2-3-4, contained 410 lines, one
line per measure. By limiting the validation work to F00/
0 M/ category and QMEs with numerous contexts 14
base QME were identifed (coverage of 245 lines or 60% of
the reference base, see Figure 7).
Number of
To Com
b0%
Completed
external
It was estimated that around 70 lines still need to be
validated, what explains the 40% fom Figure 7. The 165
remaining lines were rarely referred to by derived measures;
they ofen applied to a single derived measure, for example:
I/O Utilization, Number of instances of data corruption prevention, Size
of sofwa reo
Another interesting fnding was that III derived measures
could be created fom the 14 identifed QMEs. Considering
the total of 178 measures found in ISO/IEC 9126-2-3-4, this
gave 62% of coverage (111/178) of all the measures found in
this standard. Unfortunately, the coverage ratio could not be
verifed against SQuaRE list of derived measures since
standards are still under development.
V. RECOMNDATIONS
A few enhancements could be provided by the 2502n
division standards to make its use more widespread.
Distribution of 2502n division standards in a digital format,
like a knowledge database, would attract a larger population
of sofware quality engineers, and increase its use by the
industry.
Consistent use of ISO/IEC 24765 vocabulary.
Systematic use of ISO 15939 measurement templates,
providing the mapping of each QME to strengthen the
scientifc base and the design of the standard.
The standards provided by ISO are prepared on paper or as
PDF fles. This is considered as a limitation for users, since
they must go through several documents in order to fnd the
necessary measure elements for their projects. The
recommendation of providing a digital knowledge base (see
Figure 8) to the industry is one of the most important fndings
of this research.
3124
Fig. 8. QME Knowledge base.
ISO 25025
Knowledge
base
ISOlEC
Server Creation Server
Tool Server
Knowledge
base
ISOIEC
With this recommendation, some practical issues could be
addressed:
Gather all sofware measurements information on a single
storage platform.
Provide an embedded process to create new measures, base
or derived, associated to a specifc business domain or
project, sofware or component.
Develop graphical engines to build appropriate
measurement models.
Provide secure and remote access to all the information,
thus, letting ISO be the guardian of the data, while allowing
organisations to globally share or protect their own
measurement information to other ISO users.
Other directions for the refnement of this research are:
The application of the same analysis method to the
measures listed in ISO/IEC 25021 in order to determine
other contexts of measurement since the measures provided
in ISO 25021 are also based on ISO 9126-technical reports.
Adding new QMEs that refect current development trends
such as Agile programming. Although this issue came up as
a side effect of this research, it seems logical that the
standard evolves in this direction.
Removing QMEs that are no longer used due to technology
evolution.
Adding the defnition of QME context as a parameter
complementing measurement method and measurement
unit. The 2502n division will surely provide the derived
measures, but adding the method linking derived measures
and QMEs would supplement the information already in the
standards.
CONCLUSION
To improve the ISO/IEC 2502n division standards, the use
of a digital content, as proposed in this paper, would be
benefcial to the IT industry. A database support and user
fiendly applications that allows quality engineers to quickly
navigate and research would be a major improvement to these
standards. The tools, such as an automatic calculation of
derived measures could also be considered. Furthermore,
sofware quality measurement activities could easily be
integrated to existing development processes. Hence, making
it a great tool for improving sofware quality overall.
ACKNOWLEDGMNT
This article has been developed in the context of a research
project realized at Ecole de Technologie Superieure,
Montreal, Canada. The authors would like to thank
Guillaume Bellosguardo, Salah-Eddine Benbrahim and
Charles Hebert for their contribution to this research. Finally,
the authors are grateful to fellow students of class MGL842
(20 II) for their valuable input.
REFERENCES
[1] ISO/IEC 9126-1-2003: Sofware Engineering - Product Quality - Part
1: Quality Model. Geneva. Switzerland: International Organization for
Standardization, 200 I.
[2] ISO/IEC TR 9126-2-2003: Sofware Engineering - Product Quality -
Part 2: External Metrics, Geneva, Switzerland: International
Organization for Standardization, 2003.
[3] ISO/IEC TR 9126-3-2003: Sofware Engineering - Product Quality -
Part 3: Internal Metrics, Geneva, Switzerland: International
Organization for Standardization.
[4] ISO/IEC TR 9126-4-2004: Sofware Engineering - Product Quality -
Part 4: Quality in Use Metrics, Geneva, Switzerland: International
Organization for Standardization.
[5] SC7, ISO/IEC DTR 25021-2011: Sofware Engineering - Sofware
Product Quality Requirements and Evaluation (SQuaRE) - Quality
Measure Elements.
[6] ISO/IEC IS 15939-2007: Sofware Engineering - Sofware
Measurement Process, Geneva, Switzerland: international Organization
for Standardization.
[7] IEEE Std. 1061-1998, Standard for a Sofware Quality Metrics
Methodology, revision, IEE Standards Dept., Piscataway, N.J., 1998
[8] Abran, R. E. Al-Qutaish, J. M. Desharnais, and N. Habra, "An
Information Model for Software Quality Measurement with ISO
Standards," in Proceedings of the International Conference on Sofware
Development (SWDC-REK), Reykjavik, Iceland, 2005. pp. 104-116.
[9] J. Boegh, "Certifying Sofware Component Attributes," IEE Sofware,
Vol. 23, No. 3, 2006, pp. 74 - 81.
[10] A. Abran, R. E. Al-Qutaish, and l-M. Desharais, "Harmonization
issues in the updating of the ISO standards on sofware product
quality," Metrics News: Journal of the Sofware Metrics Community,
Vol. 10, No. 2, 2005, pp. 35-44.
[II] A. Abran, R. E. AI-Qutaish, and l Cuadrado-Gallego, "Analysis of the
ISO 9126 on Sofware Product Quality Evaluation from the Metrology
and ISO 15939 Perspectives WSEAS Transactions on Computers, Vol.
5, No. 11,2006, pp. 2778-2786.
[12] A. Abran, R. E. AI-Qutaish, ad J. Cuadrado-Gallego, "Investigation of
the Metrology Concepts in ISO 9126 on Sofware Product Quality
Evaluation," Proc. Of the 10th international Conference on Computers
(lCComp'06), Vouliagmeni, Athens, Greece, 2006, pp. 864-872.
[13] H.-W. Jung, S.-G. Kim, and c.-S. Chung, "Measuring Sofware
Product Quality: A Survey of ISO/IEC 9126," IEEE Sofware, Vol. 21,
No. 5, 2004, pp. 89-92.
[14] B. B. Chua and L. E. Dyson, "Applying the ISO 9126 Model to the
Evaluation of an e-Learning System," Proc. Of the 21 st Conference of
the Australasian Society for Computers in Learning in Tertiary
Education (ASClLITE'04), Perth, Australia, 2004, pp. 184-190.
[15] . Abran, A. Khelif, W. Suryn, and A. Sefah, "Consolidating the ISO
Usability Models," Proc. Of 11th International Sofware Quality
Maagement Conference, Glasgow, Scotland, U, 2003.
[16] ISO/IEC, International Vocabulary of Basic and General Terms in
Metrology (VIM), Geneva, Switzerland: International Organization for
Standardization, 2008.
[17] A. Abran, Sofware Metrics and Sofware Metrology, IEEE Computer
Society, John Wiley & Sons Inc., New Jersey, 2010.
[18] QM Reference Base V2.0, 2011 (not published).
[19] ISO/IEC FDIS 24765-2009: Systems and sofware engineering -
Vocabulary, Geneva, Switzerland: International Organization for
Standardization.
3125