Discovery

The International Daily journal

ANALYSIS

ISSN 2278 – 5469 EISSN 2278 – 5450

© 2015 Discovery Publication. All Rights Reserved

State of Quality Engineering Practices: Pakistan’s

Perspective
Publication History
Received: 25 August 2015
Accepted: 27 September 2015
Published: 19 October 2015

Citation
Muhammad Naeem Ahemd Khan, Mirza Aamir Mehmood, Qamar Nazir. State of Quality Engineering Practices: Pakistan’s Perspective.
Discovery, 2015, 44(204), 147-153

147
Page
 State of Quality Engineering Practices: Pakistan’s Perspective
Muhammad Naeem Ahemd Khan Mirza Aamir Mehmood1,2
1 2
Department of Computer Science Department of Computer Science
SZABIST BUITEMS
Islamabad, Pakistan Quetta, Pakistan
Abstract— This study analyzes state of the quality
engineering practices being exercised in the software industry of
Pakistan. Statistics have been collected and analyzed to access
four important aspects of quality engineering including quality
policy and review mechanism, quality assurance activities and
practice, quality standards and models, and quality management
systems. For this purpose we prepared an elaborated
questionnaire pertaining to various aspects of quality
management and surveyed more than 30 software houses and
software development organization in the twin cities of
Islamabad and Rawalpindi. The results of survey are reported
herein.
Keywords— Quality Assurance, Software Testing, Software
Quality Models, Software Quality Engineering, Quality
Management System.
I. INTRODUCTION
Software usage is growing constantly throughout the world
and nowadays it is being used in almost every sphere of
human activity. In this prospective, software quality becomes
an important issue not only for software designers and
developers but also for customers, users and communities as a
whole. The quality of software systems is a complex and
multi-faceted concept which is difficult to define and measure.
It cannot be reflected by a one single metric [1], because it
comprises various aspects like functionality, reliability,
usability, efficiency, maintainability and portability. It pertains
to the intrinsic features and characteristics as well as customer
satisfaction which make it hard to precisely define the quality.
Intrinsic quality refers to the quality of the software itself and
customer satisfaction refers to user perception of quality [2].
Designing and following quality models is a common
approach to cope with this problem in a structured manner. A
quality model is defined as an artifact with the objectives to
describe, assess and predict quality [3]. Quality can be
provided implicitly through certain tools or can be provided
explicitly in the forms of international standards, general
guidelines or specific checklists. Since quality models have
been subject to extensive research for many years, therefore, a
number of heterogeneous quality models have emerged e.g.,
domain specific models or test models. Some quality models
express software quality as a whole; others describe certain
quality aspects.
The advancement made with these models and standards
tend to fit for certain approaches that may be optimum for
some projects, but is often ill-suited for others [4]. Since
software industry is experiencing rapid changes due to
advancements in technology, ever-growing user requirements,
Qamar Nazir
PITB
Government of Punjab
Lahore, Pakistan
modernization of business processes, revolutions in network
infrastructure and hardware, sophistication in development
and execution environment etc.; therefore, most of the efforts
to enhance software quality focus on the technical aspects of
the software development process.
During the last four decades, several quality methods and
standards have been developed to steer software developing
organizations by defining and institutionalizing their processes
in an effective manner. These quality models are continually
evolving which has eventually become a complicated exercise
for software industries. In addition to this, many companies
develop indigenous standards and subsequently improve
quality of their systems when their own software development
processes get mature. The current state of the affairs of
software system and software development is categorized by
the increasing complexity of the end product and growing
demands on the cost and time effectiveness as well as overall
quality of the development process [5].
Pakistan's software industry has witnessed steady growth
during the last few years. According to Pakistan Software
Export Board (PSEB) statistics, 144 out of 1500 companies in
Pakistan achieved international process or product quality
certifications [6]. Out of these 144 companies, 110 are ISO
9001 certified, 23 are CMMI certified and 11 are ISO 27001
certified. These statistics are quite alarming and show that
there is strong need to improve the quality of services and
products as well as establishing sound IT-infrastructure in
software industry of Pakistan.
This study focuses on evaluating software quality models
being practiced and explores different methodologies and
practices currently followed by IT organizations in Pakistan.
The study also analyzes the following three main aspects of
quality modeling:
 The employment of specific quality models by the
practitioners,
 The importance of quality attributes and techniques with
respect to their industrial usage,
 The potentials for further improvements of quality models.
This study highlights that the key processes that contribute
towards manufacturing high quality software products under
certain quality models need to be analyzed and defined
appropriately. Further, the study puts emphasis on need for
148
understanding how quality processes and models are applied
and what methods are required to achieve the project goals.
This research also explores the problems and challenges
Page
arising from the current situation in executing these quality
models and standards.
A. Pakistan Software Industry – An Overview
Pakistan's software industry from its nascent beginnings in
the late 1980s has been rising steadily since, and is now ready
for a take-off based on its initial success. Pakistan offers
various competitive advantages over other outsourcing
destinations, such as high quality software development, easy
establishment of business and state-of-the-art
telecommunication and IT infrastructure.
The Government of Pakistan has been proactively
developing IT sector since the last few years. Establishment of
IT Parks with low rent, foreign ownership of equity invested
in IT. Government has offered a tax exemption on the income
from export of computer software and related services till
2016 as well as has allowed 100% ownership of equity and
100% repatriation of profits for foreign investors. Pakistan's
IT industry's global share is estimated at US$2.8 billion and is
offering numerous business opportunities both in public and
private sectors [7]. Table I highlights key facts of software
industry in Pakistan.
TABLE I. KEY FACTS AND FIGURES OF PAKISTAN SOFTWARE
INDUSTRY
Skilled workforce 110000
Software companies 1500
Software Technology Parks (STPs) 9 correspondingly improve the likelihood of pinpointing and
STP Area 700,000 Sq feet
Standards and Certification
ISO 9001 110
CMMI 23
ISO 27001 11
II. RELATED WORK
Nowadays, many organizations are investing huge amount
of money to improve the quality of their software and software
development process; and are using different quality assurance
processes, methodologies and practices to improve the quality
of their software. Code reviews, walkthroughs, standards and
procedures are consistently checked through performance
monitoring, product evaluation, inspection, validation,
verification, audits and allied testing activities.
Davis et al. published the results of a survey on the
practices of software quality. Most of the survey questions
were related to methodical topics rather than the use of models
for product quality. Results of the study showed that 41% of
the respondents claimed not to have any formal quality
assurance method at all, while the majority used their own
technique/method for quality assurance rather than an external
one [8]. T. Hall analyzed the quality-related activities,
particularly developers’ and managers’ attitudes towards
quality. Results of the study showed that 99% of developers
used programming standards, 74% employed code reviews or
inspections and 62% of the respondents collected software
metrics data[9]. Ho-Won et al. published a survey on ISO
9126 with an aim to investigate whether the categorization of
quality attributes was correct and reliable. This resulted in an
improved categorization to provide guidance for revising the
standard. This new categorization includes separating security
from functionality, which is the current proposal of the new
standard called ISO 25010 [10]. A survey in Australia aimed
to determine which testing methodologies and techniques as
well as tools were being used in Australia’s software industry.
The major barriers found for testing were time and cost. The
results showed that only 55.3% of the organizations in the
survey agreed that metrics helped them to improve software
quality. Furthermore, the study discovered that manual static
analysis i.e., formal and informal reviews were more
commonly being used than the automated and tool-based
analyses [11].
To ascertain their quality requirements practices product
managers from Swedish companies were interviewed. The
study found that usability and performance requirements were
being considered as the most important. The study also
identified that a major challenge with quality requirements
was to make it well specified and quantified so that to make it
testable [12]. Survey in Canada showed that inadequate
testing; training and lack of formal criteria for testing increase
the risk of releasing defects into production and release
versions of software products. The ability to detect tendencies
that lead to reduced quality and to identify the root causes of
reductions in product quality suffer from the lack of
exhaustive testing of the product. There is a need for
improvements for quality assurance. Organizations that offer
training on quality assurance and testing procedure to their
developers improve their ability to validate quality and
resolving process and product defects. Cost and lack of
expertise are two major barriers for adaptation of testing
methodology and tools [13].
Research in Brazil conducted with objective to identify
respondents’ perception about the relationship between IT
Governance models and quality instruments adoption. The
results obtained through the survey provided an overview of
the impact of IT-related problems in the organizations, the
degree of knowledge and importance as well as adoption of
different quality instruments[14]. A survey on quality models
being practiced in Germany focused on several domains of
software development such as standard software being used,
custom development methodologies, embedded systems and
company size [3]. Exploratory survey on software practices of
software firms in five ASEAN countries show that software
industry was falling short in the four areas: use of automated
estimation tools (Software Project Planning Practice), use of
requirements traceability matrices (Software Project Tracking
and Oversight practice), use of quantitative quality metrics
(Software Quality Assurance practice), and the use of change
control boards (Software Configuration Management
practice). Organizations that were strong in software quality
assurance practice had in common the best practices of quality
orientation, independent testing teams and Software
Engineering Process Group (SEPGs), peer reviews, and
quality management systems [15].
Agility supports quality objectives and improves quality of
the software development process, hence the overall product
quality. Agile method can be used to handle unpredictable
requirements. Agile is a method of simple planning, end-to-
149
end deliverables with short iterations and earlier releases.
Different activities performed in every iteration, improve the
overall quality of the software and reduce the cost of quality.
Page
One of the main advantages of agile method is that the team
can have the frequent customer feedback. In Agile, on-site
customer, pair programming and continuous integration
activities particularly improve the development process, thus,
improving the overall quality of the software. Extreme
Programming (XP) is one of the software methodologies
within the agile paradigm and it works on many areas of
software development[16]. Through extreme programming,
software quality can be improved to a greater extent. Extreme
programming not only improves the development processes
but also enhances the overall quality of software as it is based
on practices which need to be followed in entire software
development lifecycle [17].
There is a prevalent erroneous assumption that quality
means goodness or shininess. The word “quality” is often used
to signify the relative worth of something which means
different things to different persons. Hence, quality must be
defined as ‘conformance to requirements’. Consequently, the
non-conformance found in the software pertains to the absence
of quality, and in such situation quality problems become non-
conformance problems, and hence quality becomes undefined
[18].
IEEE 729-1983 defines software quality as “the totality of
features and characteristics of a product or service that bears
on its ability to satisfy given needs” [19]. According to
ISO/IEC 9126, software quality has six attributes:
functionality, reliability, usability, efficiency, portability, and
maintainability. Software quality appears to be particularly
problematic when compared to other arenas. The key reasons
for this are that software has no physical existence. Clients do
not know exactly what they need, clients periodically need
changes, advancements in hardware and software are rapid
and customers have high level of expectations [20].
III. RESEARCH METHODOLOGY
This study aims to explore the state of affairs of the quality
models, standards and best practices being used in the
software industry of Pakistan. Literature review was used to
obtain background knowledge about this research area. It
formed the basis for formulating questionnaire to conduct
survey from software companies. Sample size consisted of 50
software houses of various size situated in the cities of
Islamabad and Rawalpindi.
This survey gave us a hands-on knowledge about the
challenges being faced by the software companies. A
comprehensive analysis was performed to identify the problem
areas and challenges faced while implementing the best
practices of quality engineering. Research questions were
addressed using quantitative research method. The solutions
are discussed along with the challenges in the discussion
section as they are connected to them. Figure 1 below gives an
overview of the research process.
Fig. 1. Research Methodology
A. Research Questions
The center of focus of the research is to find answers to the
following questions:
1.Quality objectives and policy
2.Quality assurance activities and practices Quality
attributes and their prioritization
3. Quality and process improvement models
4. Quality management systems
The objective of this research is to identify flaws and
suggest potentials improvements for quality management
system currently being used in Pakistan's software industries.
IV. RESULTS
In this section detail analysis of results derived from
survey are presented.
A. Organizations Quality Policy
Software quality is about conforms of functional and non-
functional requirements. Achieving structural quality which
refers incorporating non-functional requirements is far more
difficult than meeting functional quality. To develop high
quality software, organizations need to define quality
objectives. For this purpose, organizations are required to
develop software quality metrics, define defect target levels,
and identify quality attributes and factors that affect software
quality. It is quite alarming that only 66% organizations
surveyed in this study have the defined quality objectives (Fig.
2).
Fig. 2. Statistics (in %age) showing the state of organizations quality policy
The field of computing is much dynamic as with every
passing day new tools and techniques are developed. On the
other hand, the new tools and techniques bring about new
challenges due to the constantly emerging computing
paradigms. Quality policy and objectives cannot remain
stagnant. They must be reviewed and updated frequently. It
has been observed with concern in this study that only 53%
respondent organizations have framework for reviewing and
updating the quality objective and policy while 30%
Organizations do not have any framework. Rest of the
organizations (16%) did not answer this question. 150
Page
Fig. 3. Statistics (in %age) depicting quality objective and policy review
framework
B. Software Quality Assurance activities and practices
To ensure high quality of software products, it is of
immense importance to have a well-defined organizational
structure and dedicated teams for validation and verification of
developed products. As depicted in Fig 4 it is observed with
serious concern that only 66% participating organizations have
a defined Quality Organizational Structure. Those who do not
have their own quality assurance teams are 20% whereas 13%
have not answered this question.
Fig. 4. Statistics (in %age) showing the state of dedicated V&V teams
Software quality could simply be defined as meeting the
customer requirements; however, it may not have an agreed
deification. Though quality is always required but it is never
stated explicitly and remains a non-functional requirement.
Thus, it is crucial to understand customer perspective about
quality followed by analyzing and recording quality
requirement. This study revealed that 86% organizations
perform quality related analysis, but only 76% organizations
formally record these requirement (Fig 5).
Fig. 5. Statistics (in %age) showing the state of quality related analysis and
documentation
A non-functional requirement specifies criteria for user to
judge system operation. Non-functional requirements are not
explicitly stated but described informally and are often
conflicting. This makes them hard to model, measure and
evaluate. To make them measurable, it is important to discuss
and if required negotiate with the customers. It is observed
that 70% organizations have developed procedures to deal
with non-functional requirements, 23% organizations do not
have any formal procedures and 6% did not answer this
question (Fig 6).
Fig. 6. Statistics (in %age) illustrating state of process and procedures for
NFR
Studies have shown that defects identified at early stage can
be fixed at a lower cost. Therefore quality control activities
should be initiated at very early phase of software
development lifecycle and product quality should be
maintained throughout the SDLC. This requires a meticulous
planning and mechanism to ensure execution of developed
plan. Every phase of SDLC should have a detailed quality
control plan which includes all the required methods, process
and procedures. Among participating organizations, 73% have
quality control plan for design and development phases, 16%
organization do not develop quality plans for these stages and
10% have not answered this question (Fig 7).
Fig. 7. Statistics (in %age) depicting quality control plan for design and
development phases
Validation and verification activities finish with delivery
of products to customers. Validation and verification teams
acts as client's lawyer and ensures that end product meets all
functional and non-functional requirements. Validation and
verification (V&V) teams should develop methods to control
and prevent the delivery of products that do not conform to the
requirements. This study observed that only 60% organization
ensures that no such product should be released, 20%
organizations do not have any mechanism for controlling
delivery of non-conforming products while 13% of
participation organizations have not answered to this question.
Same is depicted in fig 8.
Fig. 8. Statistics (in %age) depicting state of plan to control non-conforming
product's delivery
Software quality metrics are used to measure the fitness for
purpose of the development process, project and product.
Organizations use these metrics to measure the health and if
required take corrective action to fix those areas of
development process, project or product causing defects or
violating targeted defect level/threshold (Fig 9). 151
Page
Fig. 9. Statistics (in %age) depicting state of software quality metrics
 Quality cannot be improved if we cannot measure it. For The following figure depicts the distribution of adopted
this purpose, attributes should be identified and quantified. process improvement model in Pakistan software industry (Fig
Prioritizing these attributes is also important in order to ensure 12).
that neither the most desired attributes are compromised nor
the least priority attributes get more than desired
consideration. Table 2 depicts quality attributes and their
importance for different responding organizations. The study
results indicate that functionality, usability and reliability are
high priority attributes while portability is least important
attribute.
TABLE II. QUALITY ATTRIBUTES AND THEIR PRIORITY

Attribute Priority
1 2 3 4 5 6 07 Fig. 12. Statistics (in %age) Process improvement model
Functionality 16 04 0 01 00 02 00
Reliability 09 05 02 04 01 01 01 Reviews and audits activities ensure that the defined
Usability 03 09 02 03 02 02 01 processes are actively being practiced for all the projects being
Efficiency 02 07 07 03 01 00 02 undertaken within the organization. Any anomaly is identified
Maintainability 03 03 02 05 04 01 03 as minor or major non-conformance and project managers take
Portability 04 03 02 01 03 03 04
2
appropriate corrective actions. It is observed that 70%
No reply
organizations have developed review and audit mechanism,
16% organizations do not conduct audits while 13% have not
answered to this question.
C. Quality Standards and Model
Quality can only be accessed if a minimum threshold and D. Quality Management System
baseline is defined and proper quality standards have been
In complex and large software, defect identification is a
developed or followed. Quality standard could be procedures-
difficult task. Therefore, it is very important to identify defects
specific or resource-specific. It is observed that most of the
as early as possible. Failure to do so results in significant
organizations have defined quality standards either for all
increase in risk of schedule or cost overrun. To prevent this,
resources or all the procedures (Fig 10). .
software companies deploy quality management system.
Quality management system can prevent defects from
occurring, which reduces the risk to a significant level. It is
observed that 63% organization have implemented quality
management system of organization (Fig 13).

Fig. 10. Quality standards

Developing high quality software is a challenging task,

hence main focus of the entire software industry remains on
developing sophisticated processes to produce better quality
products. With changing faces of computing, development
processes need incessant assessment and improvement. It is
observed that 66% companies have implemented models,
standards and best practices for quality products, 10% have
not implemented any model while 20% have not answered this
question. Among those who have developed quality standers
and models, only 63% are satisfied with performance of the
quality models 20% think it is under-performing and 16%
have not answered to this question (Fig 11).
Fig. 13. Quality management system
Review the effectiveness of Quality Management System
of Organization at planned intervals enables organizations to
take preventive and corrective actions, but only 46%
organizations have review process, 20% does not have any
review process at all while 33% did not answer this questions.
V. DISCUSSION
The nature of software quality is dynamic; with changing
face of technology, it should be revisited and upgraded
according to new challenges. To produce high quality
software, it is importance that a realization about it exists at
152

institutional level and organizations develop a quality policy

in accordance with the organizational priorities and measures
of success. Additionally, a mechanism to timely review and
Page

update quality policy should also be established. Quality

Fig. 11. Quality models and standards implemented policy and objectives cannot remain stagnant; these should be
reviewed and updated frequently. This study identifies that
only 66% of the surveyed organizations meet the defined
quality objectives and only 53% respondent organizations
have framework for reviewing and updating the quality
objective and policy.
Quality policy cannot be implemented without a well-
defined organizational quality structure. It includes dedicated
and independent verification and validation teams which
ensure that the developed product meets goals and objectives
defined in the quality policy. The study data shows that only
66% of the participating organizations have a defined quality
organizational structure. Their V&V teams develop methods
to control and prevent the delivery of those products that do
not conform to the requirements. Apparently, only 60% of the
organizations ensure that no such product be released that does
not adhere to the required quality standards.
Quality cannot be achieved unless some ways and means
for measuring the quality are defined. There are many quality
models like ISO 9126 or ISO/IEC 25010:2011 which can be
adopted in this regard. As major source of revenue of
Pakistan’s software industry comes from outsourced projects,
therefore, adopting an internationally recognized quality
model can improve confidence level of the customers on IT
companies and it is observed that only 7% companies have not
adopted international standards. Nevertheless, organizations
may also develop their own quality models to achieve certain
level of confidence about quality of the developed products.
Defect identification and resolution is a costly task.
Quality management system can prevent defects from
occurring, which help overcome the risk of schedule and cost
overruns. It has been observed that 63% of the participant
organizations have adopted and implemented quality
management system. Reviewing the effectiveness of quality
management system at planned intervals enables organizations
to take timely preventive and corrective measures. Data should
be collected to access and improve effectiveness of quality
management system, but the survey results show that only
46% of the organizations have a review process.
VI. CONCLUSION
The objective of this study is to analyze the state of
software quality engineering practices in Pakistan with focus
on four aspects including organization’s quality policy;
software quality assurance activities and practices; quality
standards and model; and quality management system. This
research contributes towards understanding of software quality
assurance best practices in Pakistan and analyzes product
quality models and differentiates them with respect to their
usage as well as possible improvement potentials from the
viewpoint of quality managers and quality model users. This
study can help software development companies to identify
whether these quality models and standards assert
considerable impact to achieve, enhance and sustain quality of
the software systems. Findings of this study can also assist
software houses to increase the level of quality of their
products. It can also help enable organizations to realize the
importance of standards and models in their working
environment and how these contribute to achieve quality in
software products.
VII. REFERENCES
[1] N. E. Fenton and S. L. Pfleeger, Software Metrics: A Rigorous and
Practical Approach, vol. 2. 1997, pp. 38–42.
[2] H. Stephen, Kan, Metrics and Models in Software Quality Engineering,
2nd ed. Boston: Addison-Wesley, 2002.
[3] F. Deissenboeck, E. Juergens, K. Lochmann, and S. Wagner, “Software
quality models: Purposes, usage scenarios and requirements,” in
Proceedings - International Conference on Software Engineering, 2009,
[4] “Software Quality Professional: Are Standards the Answer?,” ASQ
Publ., vol. 8, no. 2, p. 28, 2006.
[5] K. Barbara and P. Shari Lawrence, “Software Quality: The Elusive
Target,” IEEE Softw., vol. 13, no. 1, pp. 12–21, 1996.
[6] “PSEB,” Pakistan Software Export Board, 2013. [Online]. Available:
http://www.pseb.org.pk/newindex.html. [Accessed: 13-Jul-2013].
[7] “P@SHA,” Pakistan Software Houses Association for IT & ITeS, 2013.
[Online]. Available: http://pasha.org.pk/. [Accessed: 13-Jul-2013].
[8] C. J. Davis, A. C. Gillies, P. Smith, and J. B. Thompson, “Current
practice in software quality and the impact of certification schemes,”
Softw. Qual. J., vol. 2, no. 3, pp. 145–161, 1993.
[9] T. Hall, “What do developers really think about software quality?,” in
Software Quality Management III, 1995, vol. 1, pp. 359–368.
[10] J. Ho-Won, K. Seung-Gweon, and C. Chang-shin, “Measuring software
product quality: a survey of ISO/IEC 9126,” Software, IEEE, vol. 21, no.
5, pp. 88–92, 2004.
[11] S. P. Ng, T. Murnane, K. Reed, D. Grant, and T. Y. Chen, “A
preliminary survey on software testing practices in Australia,” in
Software Engineering Conference, 2004. Proceedings. 2004 Australian,
2004, pp. 116–125.
[12] M. Glinz, P. Heymans, R. Berntsson Svensson, T. Gorschek, and B.
Regnell, “Quality Requirements in Practice: An Interview Study in
Requirements Engineering for Embedded Systems,” in Requirements
Engineering: Foundation for Software Quality, vol. 5512, Springer
Berlin Heidelberg, 2009, pp. 218–232.
[13] V. Garousi and T. Varma, “A replicated survey of software testing
practices in the Canadian province of Alberta: What has changed from
2004 to 2009?,” J. Syst. Softw., vol. 83, no. 11, pp. 2251–2262, 2010.
[14] R. S. De Espindola, E. M. Luciano, and J. L. N. Audy, “An overview of
the adoption of IT governance models and software process quality
instruments at Brazil - Preliminary results of a survey,” in Proceedings
of the 42nd Annual Hawaii International Conference on System
Sciences, HICSS, 2009.
[15] A. Imam, S. A. S. Khoja, and I. Shariff, Improving Software Quality: a
Benchmarking Approach, no. 10. CSREA Press, 2007.
[16] M. Huo, J. Verner, L. Zhu, and M. A. Babar, “Software quality and agile
methods,” in Computer Software and Applications Conference, 2004.
COMPSAC 2004. Proceedings of the 28th Annual International, 2004,
pp. 520–525 vol.1.
[17] Y. Dubinsky and O. Hazzan, “Improvement of software quality:
Introducing extreme programming into a project-based course,” in 14th
International Conference of the Israel Society for Quality, 2002, pp. 8–
12.
[18] P. B. Crosby, Quality is free: The art of making quality certain, vol. ix.
1979, p. 309.
[19] D. Ince, ISO 9001 and software quality assurance. McGraw-Hill, 1994.
[20] A. C. Gillies, Software Quality: Theory and Management. London:
Chapman & Hall, Ltd., 1992.
153
Page