Measuring the Success of

Requirements Engineering Processes

Khaled EI Emam *
4
Nazim H. Madhavji

*School of Computer Science, McGill University

t
Centre de Recherche Informatique de Montreal (CRIM)

Abstract improving RE practice, it is t h en of considerable

interest and utility to develop an appropriate instru
Central to understanding and improving require

ments engineering processes is the ability to mea

sure requirements engineering success. This paper
describes a research study whose objective was to
develop an instrument to measure the success of
requirements engineering processes. The instru
ment developed consists of 32 indicators that cover
the two most important dimensions of requirements
ment to measure RE success.
The objective of the research r epo rt ed in this
paper was to develop an instrument for measuring
RE success. RE success is defined as the extent to
which the outcomes of the RE phase serve the
needs of, and provid e a basis for ensuring the suc
cess of, all subsequent activit ies ind i vidua lly and in
,

aggregate, related to the software system through

engineering success. These two dimensions were

out the software system's lifetime. These activities

identified during the study to be: quality of require
ments engineering products and quality of require
ments engineering service. Evidence is presented
demonstrating that the instrume nt has desirable psy
chometric properties, such as high reliability and
validity.
1 Introduction
Existing c laims [6][10] and empirical evidence
[3][5][22] support the notion that an inadequately
performed requirements engineering (henceforth
RE) proc ess is positively associated with software
system failure. There would therefore be an eco
nomic as well as software quality payoff in improving
RE practices. A prerequi si te to improv i ng RE prac
tices, however, is a fundamental understanding of
RE processes and of the fac to rs that lead to their
success or cau se their failure .
An under s t a nding of t he RE p r ocess can be
expressed in the form of a theory, the purpose of
which is to specify the determinants of RE success.
Considering that RE success is a critical unit in any
theory of RE success and that sound theories with
,
empirical support ought to provide a basis for
IT
'!I This research has been supported, in part, by the
Macroscope Project and NSERC Canada.
include: design, coding, testing, putting into opera
tion, and post-deployment evolution. The domain of
analysis of this research is business information sys
tems that are fully customized for individual user
organizations.
Briefly, the main result of this research is a subjec
tive instrument with 32 indicators. The indicators
cover the two most important dimensions of RE suc
cess, which were found to be the quality of RE ser
vice and the quality of RE products. It is further
shown that the instrument has desirable psychomet
ric characteristics, includin g high reliability and
demonstrated content and construct validity.
The significance of this result is that it constitutes
the first, as far as we know, comprehensive effort to
develop an RE success instrument. In terms of
ap plication to practice, the instrument can be used
to gauge the success of RE processes in an organi
zation for assessment and comparison purposes. In
terms of application to research, the instrument can
be utilized for the measurement of RE s ucc e ss in
survey and experimental research.
The next section of this paper presents a summary
review of existing indicators of RE s uc cess . This
review provides the theoretical foundations of the
instrument. Following that, section 3 describes an
extensive empirical investigation of the RE success

204
0-8186-7017-7/95 $04.00 1995 IEEE
concept a n d of indicator s for its assessment.
Section 4 constitutes a detailed account of the
results, and section 5 concludes the paper.
2 Theoretical Foundations
Previous software engineering works that have iden
tified indicators for the assessment of RE success
were reviewed and summarized in our base report
[12]. The purpose of the review was to provide a
sound theoretical foundation for our instrument
development efforts.
It was evident from this review that the predomi
nant conceptualization of RE success in the soft
ware engineering literature has been the quality of
RE products. Thus, a vast majority of the pro
posed/used indicators are assessing some aspect of
RE product quality. Example indicators are: the
extent to which every requirement stated in the
requirements speCification has only one interpreta
tion, the extent to which everything the software is
supposed to do is included in the specification, and
the number of detected errors in a requirements
document that are categorized as being inclusions
of implementation facts.
Although there is a predominant focus on the qual
ity of RE products as the primary dimension of RE
success, a minority of the reviewed articles suggest
that there may be other dimensions. For example,
Dawson [11] writes about process effectiveness and
cycle time. Cordes and Carver [7] discuss necessity,
which is defined as the extent to which information
that is unnecessary for solution development is
included in the requirements document. This is
somewhat analogous to the definition of correctness
that Davis [10] presents. Zagorsky [23] notes that
CASE tool implementation enhances perceived pro
ductivity and effectiveness of RE activities. All of the
above references mention some aspect of the cost
effectiveness of the RE process. Also, 8asili and
Weiss [4] discuss the number of changes made to a
requirements document, which is referred to as
requirements maturity by Farbey [14]. Number of
changes and requirements maturity address another
aspect of cost-effectiveness, namely the amount of
rework. Furthermore, customer satisfaction [11] and
user satisfaction [23] have previously been consid
ered as measures of RE success.
Therefore, it seems that RE success is not uni
dimensional (Le., there are other dimensions apart
from the quality of RE products). More specifically,
can be seen that there are at least two other dimen
Sions, one concerned with user/customer satisfac-
tion and one concerned with productivity/cost-effec
tiveness of the RE process.
3 Research Method
In this research, the RE process studied was the
requirements engineering phase of a software sys
tem development method. The method (henceforth
referred to as method X) has been developed and is
marketed by company Y. Company Y is an informa
tion systems consultancy firm based in Canada with
clients worl(jwide. The ultimate objective of the RE
plhase of method X is to determine the cost-effec
tiveness of the information system to be developed
and make a go/no-go decision based on it.
The resea.rch method followed in the study pre
sented here draws from both, the normative and the
descriptive literature. The normative literature pre
scribes the procedures to be used in instrument
development, for example [16][18][21]. The descrip
tive literature specifies the procedures used by par
tiGular authors for developing instruments, for exam
ple [2][9)[15]1. This research method consisted of two
steps1: (a) define the RE success content domain,
and (b) instrument development and pretest.
3 1 Define the RE Success Content Domain
..
The objective of this step was to identify the dimen
sions of RE success and a representative set of cri
teria for the assessment of RE success. Three dis
tinct investigations were conducted consecutively.
To start, 33 interviews with 30 experts were con
ducted. For the purposes of the results presented in
this paper, the interviews focused on answering one
question: "What criteria are being used to assess
RE success?". Initially, an accumulated list of the
indicators that were mentioned in the reviewed liter
ature was used [12]. This list was subsequently
modified, refined, and reworded as the data collec
tion progressed.
Each expert interviewed has been involved in at
least one RIE phase (and some up to 35+ different
RE phases) in one or more of the following roles:
pl'Oject manager, lead architect, analyst, coach,
and/or auditor. The interviewee characteristics (their
backgrounds and location) are summarized in the
"Interviews" column2 of Figure 1. Figure 1 also
'1 Allintormants In all activities of the research method had a
common understanding Df the RE process since RE phase
objectives and activities are defined in method X.
it 2 It should be noted that an interviewee can be characterized
as having an intersection of ba ckgrounds. Therefore, the
total does not add up to 100%. For example, some senior
analysts take on the role of a project m anager.
205
shows the informant characteristics for some of the
other steps of the research method; these will be
discussed below. The outcome of the first step was
a set of 34 criteria for the assessment of RE suc
cess. These criteria served as inputs to the following
steps.
Subsequently, we checked the completeness of
the criteria, identified the dimensions of RE success,
and prioritized the criteria tapping each dimension.
For this, ten experts were interviewed. The charac
teristics of these ten experts are summarized in the
column labeled "Cat. & Prio." of Figure 1.
The interviewees were initially requested to put the
34 RE success criteria into two or more categories
such that the criteria in each category were most
similar to each other and most dissimilar from the
other categories. The criteria were randomly ordered
before each interview. The interviewees were sub
sequently requested to provide an interpretation of
each category, comment on its completeness and
the existence of any overlapping criteria (i.e., criteria
that were perceived to be exactly the same).
For the prioritization task, the interviewees were
requested to rank order the criteria in each category
in terms of how well they assess their interpretation
of the category. For example, if the category was
interpreted as "user satisfaction", then the intervie
wee was requested to rank order the criteria in
terms of how well they assess "user satisfaction".
The results from all the ten interviews were initially
cluster analyzed, and subsequently prioritized.
For the cluster analysis, a distance matrix was
constructed. The distance matrix was derived from
an incidence matrix as follows: dij 1 - Sij' where dij
=
ence measure. This measure is a direct estimate of
an ordinal scale, and hence can be used for the pur
poses of ranking.
A survey was then conducted to prioritize the
dimensions of RE success. The dimensions were
the outcome of the categorization performed during
the step described above. For the survey, a ques
tionnaire was given to 25 senior company Y consul
tants worldwide. A total of 18 responses were
received, giving a response rate of 72%. A summary
of respondent characteristics is presented in Figure
1 under the heading "Survey". The questionnaire,
among other things, requested that the respondents
rank order the dimensions of RE success in terms of
how important each is perceived to constitute an
indicator of overall RE success.
The rank orderings from the survey were trans
formed into a preference using the total proportion of
respondents who have ranked a dimension higher
than the other dimensions. The data analysis from
this survey indicated which dimension of RE suc
cess is considered the most important, and which
the least important.
3.2 Instrument Development and Pretest
The starting point for instrument development was
the 34 criteria that were already formulated. For this
instrument. a semantic differential scale was utilized
[19]. This scale consists of a concept and adjective
pairs at the extremes of a 7-point scale. Each adjec
tive pair will be referred to as an item.
Each criteria was converted to one or more con
cepts. Two items were developed for each concept.

Each item pair were reverse ordered in the pretest

is the distance between criteria i and j, and Sij is the
instrument to alleviate the tendency of respondents
similarity between criteria i and j. Similarity is
to simply mark straight down a column for the items
defined as the proportion of interviewees who
placed criteria i and j in the same category. Miller
[17] shows that this is a metric distance matrix, and Charac. Interviews Cat. & Prio. Survey
hence satisfies the criteria desirable for the applica Project 70% 70% 78%
tion of numeric cluster analysis algorithms [1]. Mgmt.
Subsequently, multiple hierarchical agglomerative Technical 43% 50% 89%
clustering algorithms [1] were used to identify the
Research & 33% 50% 17%
v a r i o u s d i m e n s i o n s or facets of RE succ ess. Education
Confidence in the emerging dimensions is increased
----------- ----------------------------------

Canada 91% 100% 61%

if similar clusters are extracted using the different
U.S.A. 9% 0% 16.66%
algorithms.
For the criteria in each of these clusters, a prioriti France 0% 0% 11.11%
zation was performed. This prioritization was based Australia 0% 0% 11.11%
on the rank ordering provided by the interviewees.
----------------------------------.----------

Total 30 10 18
For each criterion, the total proportion of intervie
wees who have ranked it higher than all the other Figure 1: Summary of informant characteristics
criteria in the same cluster was used as a prefer- (background & location) by research study activity.

206
covering a particular concept. Standardized instruc
tions were also included with each instrument.
The initial instrument was administered to collect
data on the success of RE processes. A total of 32
data points were collected, each data point repre
senting a particular RE process. The characteristics
of the respondents and the RE processes are sum
marized in Figure 2.
As part of instrument pretest, the reliability, con
struct validity, and effectiveness of the instrument
were evaluated. Each of these is defined below.
Reliability is defined as the extent to which an
experiment, test, or any measuring procedure yields
the same results on repeated trials, and is con
cerned with the problem of random measurement
error [18]. The reliability of the instrument was evalu
ated using the Cronbach alpha coefficient [8].
Construct validity is an operational concept that
asks whether the scales chosen are describing the
true construct(s) [18]. In this study, the construct is
RE success. Construct validity includes two other
concepts: convergent validity and discriminant validi
ty. Convergent validity determines whether the
scales chosen are measuring one underlying con
struct. Discriminant validity determines whether a
scale differentiates between constructs. Construct
validity of the instrument was evaluated using princi-
Charac. Type Attribute Value
pal components analysis and item-total correlations
[18]. For item-total correlations, each scale score
was subtracted from the total to avoid a spurious
part-whole correlation, and the correlation of each
scale with the new score was computed.
Effectiveness refers to the extent to which a scale
is measuring a construct relative to the other scales
that are measuring the same construct. Attaining a
reasonable level of effectiveness is important so as
not to have a lengthy instrument. Multiple criteria
were utilized to determine effectiveness. The pur
pose was to eliminate concepts from the instrument
without negatively affecting its reliability and validity.
The following criteria were utilized:
1. Using the priorities assigned in an earlier step: if a
concept was tapping a dimension that was con
sidered to be of low priority, then it was a strong
candidate for elimination. If a concept had a low
priority in tapping a particular dimension, then it
was a strong candidate for elimination.
2. Using the results of reliability analysis: if removal
of a concept from the instrument resulted in a
large increase in the value of Cronbach alpha
then it was a strong candidate for elimination.
3. Using the results of principal components analy
sis: if a concept loaded relatively low or did not
load on its associated factor, then it was a strong
candidate for elimination.

Location of Canada 62.5% 4. Using the item-total correlations: if a concept cor

Respondents U.S.A 25% related relatively low with the total then it was a
Australia 12.5% strong candidate for elimination.
Position of ProjecVClient Management 41% The above four criteria were used in combination
Respondents in Technical 37.5% to determine which concepts were to be eliminated.
RE Phase Auditing 3%
These criteria will ensure that the instrument retains
Coaching 25%
a good level of reliability and validity. The results of
Main Business Government 37.5% this analysis are presented in the following section.
of Organization Retail, Dis!. & Transp. 15.6%
Aerospace 15.6%
Financialll nsurance 9.4% 4 Results
Communications 6.25%
Other 15.6% The results are presented in three parts: (a) the

Functional Area Finance 22% dimensions of RE success that have been derived,
of Information Purchasing 15.6% (b) the priority of each criterion and dimension within
System Sales/Marketing 12.5% the RE success content domain, and (c) the instru
Inventory Control & Planning 12.5%
Transportation/Logistics 12.5% ment and its characteristics based on the pretest
Other 25% study.

Number of IS Average 5.6

Personnel Range 2-15 4.'1 Dimensions of HE Success
Involved
A total of 34 criteria for assessing the success of the
Number of user Average 11.7
Personnel Range 0-120 RE process have been identified. These are shown
Involved in Figure 3, and are numbered M1-M34.
The cluster analysis of the criteria yielded five clus
Figure 2: Summary of the characteristics of the
ters as shown in Figure 4 (the numbers in Figure 4
respondents and RE processes for instrument pretest.

207
 M1 The awareness of users of the business changes required in order to Mi8 The extent to which key issuesc have been resolved.
implement the recommended solution,
M19The extent to which the users have understood what the new system will
M2 The clarity of the links between the (processa and data) models and the do and will not do,
system objectivesb
M20The extent of user consensus on the recommended solution.
M3 The clarity of the business process in the architecture,
M21 The extent to which top management is convinced that the expected
M4 The thoroughness with which solutions alternative to the recommended benefits are likely to materialize.
solution were explored.
M22The relationship between the users and the requirements engineering
M5 The cost and effort compared to other similar requirements engineering staff.
phases in the same organization or similar organizations.
M23Whether the users have approved all the documentation.
M6 The amount of changes to the RE documentation.
M24The fit between the available funding profile and the necessary funding
M7 The users' reaction to the cost estimate. profile to implement the recommended solution.
M8 The willingness of the users to defend the recommended solution in
M25The fit between the architecture and the way the users work.
front of executive management.
M26The ability of the organization to make the necessary changes to imple
M9 The completeness of coverage of the cost/benefits analysis. ment the recommended solution.

M10The fraction of the cost of the requirements engineering phase com
pared to the (estimated) total system development cost.
M11The amount of deliverables that were not used in formulating the recom
mended solution and the cost/benefits analysis.
M12The amount of benefits that are expected to be brought to the organiza
'"
o tion by implementing the recommended solution compared to altemative
00
solutions.
M13The clarity of the links between the (processa and data) models and the
key issuesc.
M14The adequacy of the diagnosis of the existing system.
M27The fit between the recommended solution and the strategic orientation
of the organization.
M28The fit between the recommended solution and the technical orientation
of the organization.
M29The willingness of the organization to make the necessary changes to
implement the recommended solution.
M30The accuracy of the cost estimates compared to the accuracy required
by the organization.
M31The degree of top management support for changes necessary to imple
ment the recommended solution.

M15The soundness of the approach(es) taken to quantify the intangible ben
efits.
M16The clarity of the links between the weaknesses and strengths of the
existing system and the weaknesses and the strengths of the recom
mended solution.
M17The extent to which the (processa and data) models conform to the rules
of modeling.
M32The fit between the system architecture and the corporate architecture or
information plan.
M33The degree of match between the functionality of the 1st release of the
software system and user expectations.
M34The extent to which the presentation of the costlbenefits analysis follows
the accounting procedures of the organization.

a A process model includes processes of the software system (functions) as well as manual work activities of the business process.
b These are objectives 01 the system not objectives of the software development project, and they are supposed to be specific and measurable. Example objectives are "to
improve the accuracy of the planning process by 10%", "to provide the ability to find a volume in the automated catalogue in less than 5 seconds, 95% of the time", and "to
satisfy 95% of the requests for bibliographic information within 5 minutes",
c Key issues are issues that concern the system not the project. They are critical aspects of the system that require examination; without their resolution the system cannot be
completely defined and developed, When an issue can be resolved without r equir ing a formal examination of alternatives by the decision makers, then it is not a key issue.
Example key issues are "the system must be simple and inviting to use and must enable members to find answers to their queries easily. This is to ensure member accep
tance of the system. Staff assistance should be required only in exceptional cases. ", and "selVlces such as consultation, location of titles, and processing of loans and reser
vations must continue In the event of malfunctions or failure of the automated system. This Is to ensure continuity of service ".

Figure 3: Criteria for assessing RE success.

concern prioritization, and will be described below).
For each cluster, its interpretation is presented.
These interpretations were derived from those pro
vided by the interviewees. Since two of the clusters
directly concerned how good the outputs of the RE
process are (quality of architecture and quality of
cost/benefits analysis), they were grouped into one
general cluster by the authors: quality of RE prod
ucts. Furthermore, since two other clusters seemed
to tap the quality of RE service dimension (user sat
isfaction and commitment, and fit of the recommend
ed solution with the organization), they were also
grouped together by the authors.
The resultant three dimensions of RE success are
as follows:
1. The Cost Effectiveness of The RE Process
This dimension assesses whether a reasonable
amount of resources were consumed during the
RE phase.
3. Quality of RE Service
This dimension concerns the service provided by
the RE staff to the users. The quality of this ser
vice is reflected in the extent of user satisfaction
and commitment, and the extent to which the rec
ommended solution fits the organization. It is criti
cal to provide a service that satisfies the users
and to properly manage their expectations. This is
intended to ensure that they will participate in
later phases and that they will use the system
after it is operational. Furthermore, a software
system is considered as part of a business solu
tion. This may necessitate, for instance, new work
procedures, changes in organizational structure,
and/or changes to relationships with customers
and suppliers. Thus, the products of the RE
process constitute a recommended organizational
change supported by information technology,
which must fit the organization and its capacity to

2. Quality of RE Products
implement it.

The quality of RE products covers the quality of

4.2 Importance of Criteria and Dimensions
the major documents that are produced during the
RE phase: quality of architecture and quality of Tlhe criteria iln each cluster are presented in Figure 4
cost/benefits analysis.

RE SUCCESS

Cost Effectiveness Quality Of

Of RE Process REProducts
(0.438) (1.126)
Quality Of

A
M5(1.599) REService
M10(0.900)
(1.438)
M6(0.80l)
Ml1 (0.300)

Quality Of Quality Of
Architecture CostiBenefits
Analysis
M2(3.402)
M3(2.702) M9(2.700)
M14(2.597) M2l : M24(1.902)
M13(2.499) M12 (1.698)
M30(1.398) User Satisfaction
M16(2.198)
M34 (0.789 ) And Commlttment
M4 (1.799)
M15 (0.702) M19 (3.904)
M18 (1.498) /
M17(0.399) M8(3.800)
Fit Of Recommended
M20(3.304)
Solution With The
M1 (2.600)
Organization
M22(1.800)
M26('1.600) M7(1.600)
M27(1.500) M25 : M23(1.400)
M31(1.200) M33(1.104)
M29(0.900)
M32 : M26(0.600)

Figure 4: The dimensions of RE success, the priorities of the criteria tapping each dimension and the priorities
of each dimension in tapping RE success.

209
in priority order, with the highest priority criteria at
the top. A colon in Figure 4 indicates that the two cri
teria are at the same priority. This prioritization indi
cates how well, compared to other criteria, a particu
lar criterion is a measure of the cluster's interpreta
tion. Next to each criteria/criterion is the preference
measure value that was used for p r ior iti z at io n . It
may be recalled that the total proportion of respon
dents who have ranked a criterion higher than other
criteria in the same cluster was used as a prefer
ence measure.
The data from the survey, which asked respon
dents to rank order the three dimensions of RE suc
cess, were used to determine whether there was
any difference among the perceived importance of
the dimensions of RE success. Figure 4 shows the
preference measures for each of these dimensions.
It is evident that quality of RE service is perceived to
be the most important dimension of RE success
(with a preference measure of 1.438), and the cost
effectiveness of the RE process is perceived to be
the least important dimension of RE success (with a
preference measure of 0.438).
To test the null hypothesis that the three dimen
sions of RE success are of equal importance versus
the alternative hypothesis that the three dimensions
of RE success are ordered in the specific sequence
described above, a nonparametric statistic is used.
The particular statistic is L [20]. The test was con
ducted at an <x=0.01, and resulted in rejecting the
null hypothesis, hence, further supporting the order
ing presented above.
4.3 RE Success Instrument
In particular, it was identified that concepts covering
the SUb-dimension "fit of the recommended solution
with the organization" (M27, M26, M29, M31) and
"user satisfaction and commitment" (M1, M8, M20,
M22) load highly on the first factor and have low
loadings on the second factor. Furthermore, con
cepts covering the "quality of architecture" ( M16,
M14, M13, M2) and "quality of cost/benefits analy
sis" (M9a, M9b, M12, M30) load highly on the sec
ond factor and have low loadings on the first factor.
These results demonstrate good convergent and
discriminant validities.
The factor loadings shown in Figure 5 are in gen
eral very high, lending reasonably strong support to
the instrument's construct validi t y. It should be
noted, however, that construct validity cannot be
claimed until these same results are replicated in
subsequent studies. The results presented here pro
vide some i n i tial evi dence supporting construct
validity, and hence encouraging further studies.
Further evidence of construct validity was obtained
from the results of item-total correlations. Overall, 15
out of 16 correlations are above 0.4 and significant
at an alpha level of 0.05.
5 Conclusions
The research study described in this paper was
based on the premise that it is critical to understand
Quality Qf Quality of RE
RE Service Products
E 1ii
:I Mean 77.6333 79.6250
en iii
Std. Dev. 26.6102 16.7898

The elimination of concepts using the four criteria Qj iii

c: Cronbach a 0.9711 0.9071
a: Std. Error 4.5237 5.1174
mentioned earlier resulted in an instrument with 16
concepts and 32 items (this instrument is available
. M1 0.7739
from the authors). The concepts cover the two m ost I/) M8 0.8079
>-
important dimensions of RE su ccess: quality of RE (ij M20 0.8989
service and qua lity of RE products. c: M22 0.8377
M27 0.6467 {0.5550}
Figure 5 shows the reliability estimates for each of I/) M26 0.6974
-
the two dimensions of RE success. As can be seen, c: M29 0.8755
G>
the reliability estimates (next to Cronbach alpha c: M31 0.8440
0
heading) are sufficiently high to recommend usage c..
E M9a 0.6655
of t h i s i n s t r u m e n t f o r p r ac t i c a l m e a s u r e m e n t 0
(.) M9b 0.6670
(Nunnally [18] suggests values a t least greater than M12 0.3962
0.9 for practical measurement).
(ij M30 0.5097
c..

The results of the principal components analysis 0 M2 0.7205

c:
.;:; M13 0.6801
are also presented in Figure 5 (item composites M14
a. 0.5155
were formed by summing the values on item pairs). M16 0.8003
These results indicate that the structure of the RE ..
success construct matches that originally expected. Figure 5: Summary statistics, results of reliab i l i t y analy
sis, and results of principal components analysis.

210
the RE process in order to improve it. A central con
tributor to this understanding is the development of
an instrument for measuring RE success. An exten
sive research investigation described in this paper
has resulted in such an instrument. This is, as far as
we know, the first comprehensive research effort
that has resulted in an RE success instrument.
The potential applications of these results are dis
cussed from the research and practice perspectives.
From the research perspective, the concern is pri
marily with advancing the state of knowledge about
RE success. The primary significance of this work
for the research community is therefore the exis
tence of a general and standardized instrument for
measuring RE success. It is now an easier task for
researchers to conduct studies (for example, using
survey or experimental empirical research methods)
whose purpose is to test hypotheses (such as those
reported in [13]) about the effect of existing and
emerging practices and tools on RE success.
From the practice perspective, the concern is pri
marily with improving RE practices in order to attain
greater RE success. The instrument for measuring
RE success may be applied by practitioners, for
example, in the evaluation of RE phase pilot pro
jects. Thus, if an organization is adopting a new
requirements engineering method, then the outcome
of a pilot project can be evaluated and compared to
the baseline value of RE success that is more com
mon within the organization. This would allow man
agement to gauge the benefits of the new method.
While the above list of the applications of this
research may not be comprehensive, it is contended
that they address important contemporary issues of
concern to both researchers and practitioners.
As for future research, greater confidence with this
instrument would be established if the evidence sup
porting its reliability and validity can be replicated in
studies conducted by other researchers. It is expect
ed that the instrument can be improved by testing
further, for example, by determining the test-retest
reliability of the instrument, and determining its relia
bility and validity with different samples.
References
[1] M. Aldenderfer and R. Blashfield. "Cluster analysis".
Sage Publications, 1984.
[2] J. Bailey and S. Pearson. "Development of a tool for
measuring and analyzing computer user satisfaction".
In Management Science, 29(5):530-545, May 1983.
[3] V. Basili and B. Perricone. "Software errors and com
plexity: An emp irical investigation".
Communications of the ACM, 27(1):42-52, January
1984.
[4] V Basili and D. Weiss. "Evaluation of a software
requirements document by analysis of change data".
In Proceedings of the Fifth International Conference
on Software Engineering, pages 314-323, 1981.
[5] B. B o ehm. "Software engineering economics".
Prentice Hall, 1981.
[6] F. Brooks. "No silver bullet: Essence and accidents of
software engineering". In C o mp ut er, 20(4):10-19,
April 1987.
[7] D. Cordes and D. Carver. "Evaluation method for user
requirements documents". In Information a nd
Software Technology, 31(4):181-188, May 1989.
[8] L. Cronbach. "Coefficient alpha and the internal con
sistency of tests". In Psychometrika, pages 297-334,
September 1951.
[9] F. Davis. "Perceived usefulness, perceived ease of
u s e , and user acceptance o f Information
Technology". I n MIS Quarterly, 13(3):319-340,
September 1989.
[10] A. Davis. "Software requirements: Objects, functions,
and states". Prentice Hall, 1993.
[11] J. Dawson. "Toronto laboratory requirements process
reference guide". Technical Report (Unpublished),
IBM Canada Ltd. Laboratory, 1991.
[12] K. EI Emam and N. H. Madhavji. "An instrument for
measuring the success of the requirements engineer
ing process in information systems development".
Technical Report SE-94.12, School of Computer
Science, McGill Un iversity, 1994.
[13] K. EI Ernam and N. H. Madhavji. "A field study of
requirements engineering practices in information
systems development". In Proceedings of the Second
IEEE International Symposium on Requirements
Engineering, 1995.
[14] B. Farbey. "Software quality metrics: Considerations
about requirements and requirements specifications".
In Information and Software Technology, 32(1):60-64,
January/February 1990.
[15] B. Ives, M. Olson, and J. Baroudi. "The measurement
of user information satisfaction". In Communications
of the ACM, 26(10):785-793, October 1983.
[16] F. Kerlinger. "Foundations of behavioral research".
Holt, Rinehart, and Winston, 1986.
[17] G. Miller. "A psychological method to investigate ver
b a l concep ts". In Journal of Mathematical
Psychology, 6: 169-191, 1969.
[18] J. Nunnally. " Psychometric theory". McGraw Hill,
1967.
[19] C. Osgood, G. Suci, and P. Tannenbaum. 'The mea
surement of meaning". University of Illinois Press,
1967.
[20] E. Page.. "Ordered hypotheses for multiple treat
ments: A significance test for linear ran ks". In
American Statistical Association Journal, pages 216-
230, March 1963.
[21] D. Straub. "Validating instruments in MIS research".
In MIS Quarterly, pages 147-169, June 1989.
[22] M. Wu. "Selecting the right software application pack
age". In Journal of Systems Management, pages 28-
32, September 1990.
[23] C. Zagorsky. "Case study: Managing the change to
In
C A S E". In Journal of I n formation Systems
Management, pages 24-32, Summer 1990.
211