A Field Study of Requirements Engineering
Practices in Information Systems Developmentll
Khaled EI Emam *
t
Nazim H. Madhavji *
*School of Computer Science, McGill University
tCentre de Recherche Informatique de Montreal (CRIM)
Abstract
To make recommendations for improving require­
ments engineering processes, it is critical to under­
stand the problems faced in contemporary practice.
In this paper, we describe a field study whose gen­
eral objectives were to formulate recommendations
to practitioners for improving requirements engineer­
ing processes, and to provide directions for future
research on methods and tools. The results indicate
that there are seven key issues of greatest concern
in requirements engineering practice. These issues
are discussed in terms of the problems they repre­
sent, how these problems are addressed success­
fully in practice, and impediments to the implemen­
tation of such good practices.
1 Introduction
The requ i rements engineering p rocess is arg uably
one of the most i mportant processes in software
develop ment . Unfortunately, however, the record is
poor. For example, it has been fou nd i n a number of
prev i o us studies that the req ui remen ts engineering
phase is the source of the majority of detected soft­
ware code errors [3][9][14][19]. It is thus im perative
to focus specifically on this process and find ways to
improve it.
As part of our research efforts to make recommen­
d at i o n s for i m prov i n g req u ire ments e n g i neeri n g
processes, this paper describes a field investigation
of c on te mpo rary req u i re ments e n g i neering prac­
tices. T h e g e n eral objectives of the i n vestigation
were to formulate recommendations to practitioners
� This research has been supp orted. in part. by the IT
Macroscope Proi ect and NSERC Canada.
68
0-8186-7017-7/95 $04.00 © 1995 IEEE
for improving req u i rements engineeri ng processes,
and to provide d i rect ions for future research o n
method s a n d too l s t h a t would result i n process
improvements. The specific objectives of the investi­
gation were to identify the most pressing problems
faced by p ract itioners , how these pro blems are
add ressed successful l y in p ractice, and the major
imped iments to the successful implementation of
such good practices.
While there have been previous field investigations
of software engineering p ractice , ours d iffers in a
number of aspects. For i nstance, C u rtis et al. [6]
cond ucted a f i eld s t udy of software e n g i neering
p ractices. Thei r findings covered the requirements
engineering process as well as other software engi­
neering processes. Our study focuses only on the
requirements engineering process, and is t herefore
closer in focus to the field study reported by Lubars
et al. [15]. Moreover, the study described i n this
paper differs from the above two in that: we used
two different data gathering methods (interviews and
inspection of documentation); we collected data on
60 cases (as opposed to 23 and 17 in the previou s
studies respectively); the data we collected consti­
tuted two types of cases: descriptive and prescrip­
tive (as opposed to pri m arily descriptive cases in
both of the previous studies); we included an explicit
val idation step with a subset of th e ori g i n al i nfor­
m a nts; o u r dom ain of an a l y sis w a s bu sin e s s
I nformation Systems that are fully customized for
i n d i v i d u al u s e r organ i z at i o n s; and we e x pl i c itly
attempted to identify impediments to the implemen­
tation of good requirements engineering practices.
These differences were intended to provide a better
understan d i n g of requ irements e n g i neerin g prac­
tices in the chosen domain of analysis.
Briefly, the results of the study i n dicate that, in
addition to technical issues, non-technical issues are research study was the r€lq u i rements engi neering
of major importance d u ri n g the requ i rements engi­ p h ase of a software system development method
neering p rocess in terms of problems, solutions, and (henceforth Method X). Method X has been devel­
i m p e d i m e n t s to i m p l e m e n tin g t h e s o lu t i o n s . oped and is marketed by an I nformation Systems
Furthermore, many o f these issues are o f pri mary consultancy firm with clients worldwide. The ultimate
concern at the planni n g stage of the requirements objective of the requ irements engineering phase i n
engineering p rocess. The significance of these find­ t h i s method is t o determine t h e cost-effectiveness of
i ngs a re t h at t h e y p r o v i d e d i rect i o n f o r f u t u re the Information System to be developed and make a
research targeted at ass isti n g requirements engi­ go/no-go decision based on it. However, prior to the
neering process planning, and they include recom­ p e rformance of such an analys i s , the b u si n ess
mendations for p ractitioners. requ i rements must be elicited and a recommended
I n section 2 t h e research method is d escribed . solution form ulated . The recommended soluti o n
Section 3 is a detailed description of the study's find­ consists o f a requirements specification, a h i g h level
ings. Section 4 concludes the paper with a summary architecture, and an analysis of how the proposed
of the key points and d i rections for future research. system will fit into, and havE� an impact on, the exist­
ing user organization.
2 Research Method The research method consisted of two main activi­
ties: a data gathering step using multiple sou rces,
T h e e m p i rical research method fo l lowed i n t h i s
and a synthesis and validation step.
investigation is t h e multiple case study method as
d e s c r i be d by Yi n [26] , a n d rec o m m e n ded fo r
2.1 Data Gathering Step
research on software deve l opment p rojects [6] .
Furthermore, i t has been asserted that this method The objective of this step was to formulate a set of
is more suitable for research on organ izational, as cases. I n total there were 60 cases. Two types of
opposed to ind ivid ual, problem-solving (Le., multiple cases were sought. The first type of case is descrip­
individuals) [21]. t i v e , and t h e s e c o n d t y p e is p re s c r i p t i v e . T h e
The unit of analysis of this study was the requi re­ descriptive cases represent requirements engineer­
m e n ts e n g i n e e r i n g p rocess. T h e context of o u r ing processes performed at client sites of the com­
pany t h at developed Method X. The p rescriptive
Business of Organization cases represent rules of nood p ractice that were
Pharmaceutical and Health Services 13%
solicited from practitione rs Llsing Method X.
Financial/Insurance 14%
Govemment 33% For req u i rements engineering process (descrip­
Retail and Distribution 27% tive) cases, seventeen individuals were interviewed.
Other 14%
The interviewees provided a description of the prob­
Functional Area of Information System lems faced d u ring the req uirements e n g i ne e r i n g
Purchasing/Sales/Marketing 47% process a n d how these p roblems were handled, if
Finance 33%
20%
they were. Of those i nterviElwed, 35% had a techni­
Other
cal backg round, and 94% h ad a p roject manage­
Figure 1 (a): Characteristics of interview req ui rements ment background1. These interviews resulted in fif­
engineering cases (15 cases). t e e n c as e s2 s u m m a r ized in F i g u re 1 (a)3.
1 It should be note d that an Interviewee can be characterized
Business of Organization as having an intersection of backgrounds. Therefore, the
Financialflnsurance 14%
total does not add up to 100%. For example, some senior
Govemment 40%
analysts take on the role of project manager.
Transportation 7%
2 For 2 of t he descriptive cases, 2 interviewees p r o v i d ed
Engineering 14%
27%
info rm ati on .
Other
3 No information on the actual sizes of the Information
Functional Area of Information System Systems is available. For some cases the system was not
Purchasing/Sales/Marketing 7% actu ally developed yet, and for some other cases the pro­
Finance 27% ject was terminated at the '3nd of the requirements engi­
Production/Manufacturing/Operations 33% neering phase or at some later pha se . For the remaining
Transportation/Logistics 14% cases where a system was actually developed, no data that
Telecommunications 7% can be compared meaningfully across organizations was
Other 14% available. For example, a summary of system sizes using
LOC measures would be misleading given that different
Figure l(b): Characteristics of document requirements o rg aniz at i ons use different progr a m min g languages and
engineering cases (15 cases). use different counting standards.

69
Fu rthermore, the documentation resu lting from fif­
teen other requirements engineering processes was
i nspected . T h e documentat i o n i ncluded req u i re­
ments engineering del iverables (e.g ., p rocess and
data models), as well as some project management
documentation and post-mortem reviews. These fif­
teen cases are summarized in Figure 1 (b).
The second type of case (prescriptive) is a set of
"rules" and their rationale. Each set of rules reflected
the aggregate experiences of a single practitioner.
The experience of the practitioners ranged from 2
req uirements engineering processes and up to 35.
These rules concerned problems that are commonly
faced d uring the requirements engineering process,
how they should be addressed, and why they are
sometimes not add ressed in practice. Example rules
are "users participating in the requirements engi­
neering process should always be at the supervisory
level" and "if the analysts are unfamiliar with the
application domain then consider building a proto­
type, consider software packages, and consider let­
ting the analysts construct models of the current
system (the one being replaced) as a means for
them to learn about the application domain". The
rule sets were solicited from thirty p ractitioners. Of
these, 43% had a technical background, and 70%
had a project management background.
A two-pronged strategy was utilized i n collecting
the data for these cases. The first was to col lect
i nformation about best practices and why they are
• What changes were made to the standard
Method X and why? If no changes were made,
then why not?
• For each activity in Method X, under what cir­
cumstances would this activity be performed
and why? Under what circumstances would you
not perform th is activity and why? What level of
detail is appropriate for the output of this activity
and why?
• For each activity in Method X, what individual,
p roject and/or organizational characteristics
would have an i mpact on its performance, how,
and why?
• What i ndividual, p roject and/or organizational
characteristics would have an impact on the
overal l success of the req uirements engineering
process, how, and why?
• In retrospect, what changes should have been
made to the requirements engineering process
and why?
Figure 2: H i g h level questions used in the data
gathering step.
70
considered best practices. The second was to col­
lect i nformation about i nfidelity or deviations from
the stan d a rd req u i rements e n g ineering process
defined in Method X and the rationale behind such
deviations. Such infidel ity captured the variations
across the req u i rements e n g i n eeri n g p rocesses.
Both of the above app roaches have been recom­
mended as ways for captu ring information that can
be employed in p rocess improvements [17]. The
high level questions that were posed to gather this
data are shown in Figure 2. It should be noted that
these questions were not posed in t h e form of a
questionnaire, but represent the general reasoning
behind the specific questions.
2.2 Synthesis and Validation Step
For the p u rposes of the results presented in t h is
paper, the cases (60 in total) had to be synthesized
into a number of issues that were of greatest con­
cern to practitioners. An issue is denoted by a label
given to a set of related p roblems, the a pproaches
used to address them successfully in practice, and
diff iculties encou ntered impleme n t i n g such good
practices.
The synthesis task proceeded ite ratively. Each
iteration synthesized the information from one more
case. The app roach was to identify the issues in
each case. On su bsequent iterations the synthe­
sized issues were refined and elaborated upon. This
was repeated until all the cases were exhausted.
The authors s u bsequently decided which issues
were of g reatest concern by the number of times
they were repeated and the importance attached to
them in the original cases.
For the validation task, six of the o riginal intervie­
wees were interviewed again . They were p resented
with the set of issues that were identified and asked
to ag ree/disagree with the issues, and comment on
them in terms of thei r correctness and importance.
Furthermore, a version of the discussion presented
in the results section of this paper was sent to three
of the original participants for comment. As well, two
formal presentations were made to three of the origi­
nal interviewees, and the ensuing discussion result­
ed in feedback about the correctness and impor­
tance of the identified issues. This validation exer­
cise cul minated in a refinement of the original set of
issues4, and these are p resented in the following
section.
4 Refinements concemed largely the g rouping of results into
common issues and the clarification of terminology. To a
lesser extent, the relative importance of some problems
was also affected.
 Issue Concern Recommendations Difficulties Implementing Recommendations Successfully
Package Should packages • packages should, in general, be considered in the requirements engi­ • organizational politics and the 'Not Invented Here' syndrome
Consideration be considered in neering process except in the case of unprecedented systems may hinder package consideration
the requirements • when resource constraints exist, packages are considered
engineering with a view to compromising user requirements to avoid pack­
process? age adaptation

Managing the How much func­ • consider the extent of package adaptation necessary and the level of • inexperienced analysts tend to produce models that are too
Level of Detail tional process mod­ uncertainty when deciding when to stop the modeling activity detailed
of Functional eling is necessary • the greater the level of control in the organization, the greater
Process Models in the requirements the pressure to produce too much detail
engineering • when there are resource constraints, there is pressure to cut
phase? down on the modeling activity (i.e., too little detail)

Examining the Should the current • if there is no current system, then there is nothing to examine, other­ • inexperienced analysts may model the current system in too
Current System system be exam­ wise examination of the current system should be done much detail
ined? If so, how • consider the following 4 criteria when determining when to stop: models • some organizations are reluctant to spend more money on the
detailed should this of the existing system should be detailed enough to allow a diagnOSiS, current system and its examination
examination be? more effort is necessary for larger legacy systems, it may be possible • the higher the level of control of the organization, the greater
to reuse models of the current system, junior analysts may leam about the pressure to examine the current system in too much detail
the application domain if they model the current system
User Should users par­ • users should always participate in the requirements engineering • the most desirable users may not be available to participate
Participalion ticipate in the process • there is a history of lack of cooperation between user depart­
requirements engi­ • the mechanisms that can be used to promote user participation include ments and IS
neering process? If face-to-face meetings, workshops with users, communication with • there is open hostility between the user and IS organizations
so, how could they users through electronic mail andlor conference calls, liaison groups
participate and (either based in the user organization or IS organization), off-site visits
.... which users should by senior analysts to user departments, users sitting on a project steer­
.....
participate? ing committee, users reviewing and approving documentation items,
users performing some of the requirements engineering activities them­
selves (e.g., benefits analysis), users paying for the project out of their
own budgets and users being evaluated on the overall success of the
systems development effort
• the appropriate skills of participating users (especially the principal

user) include the list presented in Section 3.4

Managing How can uncertain­ • assi gn appropriately skilled people to analyst and architect positions, in • inappropriately skilled analysts and users are assigned to
Uncertainty ty be alleviated or particular the lead architect. The necessary skill set includes the list these positions due to the common practice of assigning peo­
dealt with in the presented in Section 3.5 ple based on availability rather than capability
requirements engi­ • assign appropriately skilled users to participate in the requirements • a prototyping effort is not managed properly (e.g., not manag­
neering process? engineering process, in particular the principal user. The necessary skill ing user expectations and assigning staff inexperienced with
set includes the list presented in Section 3.4 the prototyping tool to develop the prototype)
• if uncertainty is very high, then consider constructing a prototype

Benefits of How can benefits • an infrastructure must be in place to support the implementation of a • high and unrealistic expectations (e.g., immediate pay-off) by
CASE Tools be gained from CASE tool (e.g., training on methods and tool, and a CASE support management about the benefits of CASE tools
CASE tools in the group) • lack of funding for the implementation of CASE tools
RE process? • an organization must be willing and able to invest in putting such an
infrastructure in place

Project What are the nec­ • always assign a project manager of high capability to the requirements • people are assigned to project management positions based
Management essary skills of a engineering phase. The skill set of a project manager includes the list on their availability rather than on their project management
Capability project manager? presented in Section 3.7 capabilities
• the career track leads inappropriately skilled people to project
management positions

Figure 3: Summary of the seven requirements en ginee ring process issues.

3 Results
T h e res ults of t h i s study i n d icate that t h ere are
seven issues of greatest concern in requirements
engineering practice. These issues are discussed
below in terms of the problems they represent, how
these problems are addressed successfully i n prac­
tice, and impedi ments to the implementation of such
good practices. An overall summary of the issues is
shown in Fig ure 3. This fig ure accompanies the dis­
cussion below.
3.1 Package Consideration
P a c k a g e s o l u t i o n s ( t h e s e are a l s o s o m et i mes
referred to as Components Off The Shelf or COTS)
may be considered during the require ments engi­
neeri ng process for one or a combination of three
reasons. The first is that packages may serve as
existing system s urrogates; t h e second is that a
package solution may form the whole or part of the
recommended solution; and the third is that package
consideration may boost the users' confidence with
the recommended sol ution .
While serving as existing system surrogates, pack­
ages aid in the definition of user needs. Through
their i nteractions with a package, users can more
easi ly identify the fu nctions they need and desire,
and define the functions that are not avai lable in the
package. Davis [8] has also previously suggested
that a package may be used as an anchor and the
new system requirements can be adjusted from it.
Furthermore, when used as existing system surro­
gates, packages have acted as a sou rce of ideas to
analysts in terms of user-interface desi g n , and i n
terms o f how t o structure t h e functions and data i n
their models. These models are ultimately used as
part of the architecture if a software system is to be
developed.
When considered for their potential as the whole of
or part of the solution , packages are perceived to be
less costly business solutions in terms of funds and
t i me to i m plem ent. T h i s is particu larly attract ive
when project f u n d i n g is low o r when the delivery
schedule is tight.
There was strong consensus that packages should
also be considered so as to convi nce the users that
alternative solutions (alternative to building a system
from scratch) were considered, even if they do not
serve as existing system surrogates or become (part
of) the solution. Furthermore, packages have been
used by analysts to convince users of the feasibility
of a part i c u l a r s o l ution, and hence making them
more comfortable with it. For example, in one case
72
where the users were not convinced about the via­
bility of expert system technology, seeing a package
that util izes an expert system convinced them that
the technology was mature enough for their purpos­
es.
The above paragraphs indicate that some benefits
would probably be gained through package consid­
eration. Thus, the g e n eral recommendation has
been that packages should be considered in the
requirements engineering process. I n terms of costs,
the requ irements engi neering phase typically con­
sumes 1 0-1 5% of total system development effort
(this is also supported in the literature [7][20]), and
package consideration is only a small fraction of the
requ irements e n g ineer i n g phase. T h u s t h e total
effort that is consumed i n package conSideration is
relatively small.
I n t h e case of u n preced ented syste m s (with
respect to functionality and/or technology), however,
appropriate packages may not exist in the market.
Therefore, if it is evident that no packages exist, or
that an extensive package adaptation effort is nec­
essary to make use of existing packages, then pack­
age consideration would not n ecessarily be cost­
effective. For example, in one case packages were
not considered because it was very obvious that the
software system was very close to the state-of-the­
art in I nformation Technology.
Two problems were identified that frequently i n hib­
ited package consideration. The first prob l e m is
organ izational pol itics. One project manager with
extensive experience i n government environ ments
elaborates:
In government it is less likely [that packages
are considered]. They are not so concerned
about cost, so they will not accept a package
to save money; there are political reasons for
budgeting. The basic government attitude is
that they don't look at the value of things, it's
not their money.
Another example is when an organization just pur­
chased a mainframe and there existed a PC-based
package worthy of consideration:
Someone up top told them that because they
spent $20M on a mainframe, they had to use
it. They were reluctant to look at any PC­
based solutions. No one wants to rock the
boat!
The secon d pro b l e m is t h e "not i nvented here"
syndrome, whereby analysts believe that their situa­
tion is unique and they want to construct a system
from scratch.
3.2 Managing the Level of Detail of
Functional Process Models
The main purposes of functional process modeling
are to gain an understanding of what the software
system will do, what data it will consume and pro­
d uce, and its i n teractions with other man ual and
au tomated sys t e m s . S uch a n u n d e rsta n d i n g is
employed in performing the cost/benefits analysis
and in formulating and comparing alternative solu­
tions.
Modeling is still necessary even when package
solutions are considered. In such a situation, one
would want to do three things. First, to identify a
suitable set of packages for consideration . Second,
to determine the extent to which package adaptation
is necessary. Third, to compare development costs
with package adaptation costs in order to make a
"build" or "buy" decision. This implies that, at least, a
minimal amount of modeling is essential to identify
packages. Also, if a package is destined to become
the whole of or part of the solution, then a minimal
amount of modeling is necessary to determine the
extent to which package adaptation is required. This
is consistent with both, the p rescriptive and empiri­
cal literature, which promotes modeling as a prereq­
uisite to package consideration [4][24][25].
Therefore, it is always necessary to construct func­
tional process models in the requirements engineer­
ing process. The question , of course, is how much
modeling is necessary (or how much detail) in the
requirements engineering phase? An exiguous level
of detail inhibits the ability to perform an adequate
cost/benefits analysis, and to formulate and com­
pare alternative solution s . An excessive level of
detail is not cost-effective.
Considerable variation in the level of detail of func­
tional process models was observed in the cases
studied. This was evident from docu ment inspec­
tions, where some requirements engineering phases
only p roduced a one page context diagram, and oth­
ers produced a few hund red pages of data flow dia­
g rams. In this study, the level of appropriate detail
was found to be contingent upon the extent to which
package adaptation is necessary and on the amount
of uncertainty.
Package adaptation only becomes important when
a package is considered as part of or the whole of
the recommended solution. Package adaptation was
found to be d riven p rimarily by two factors: the need
for interfaces with other systems and the extent to
which the package covers the system requirements.
T h e g reater t h e extent of adaptation, t h e more
73
d e t � i l e d t he mod e l s s h o u l d b e . F o r e x a m p l e ,
detailed models will facilitatl3 a better "build" or "buy"
decision since they allow for better estimates of (the
package adaptation and implementation) costs and
(package functionality) benElfits, as well as giving the
users a better understanding and awareness of how
the package will fit in with their business needs. A
good example of the benefits of models is described
below by the project managl9r:
The users sometimes focused too much on
the look of the packages and not on whether
the functionality is covered. In the [company's
name] case, after the first demo, the users
were vety happy with thEl package, they want­
ed it right now, and bought it the day after. We
just showed them that the functionality that
was covered in the package represents only
20% of the cost of developing the system.
What do you do with the 80% that is left?
In the above example, the model allowed the ana­
lysts to estimate how much of the desired functional­
ity was covered by the package. T h i s particular
package was later abandoned.
Furthermore, when the context is characterized as
relatively uncertain, then detailed modeling is one
mechanism that would facilitate learning about the
business p rocess and the tec hnology. However,
there are dangers in the employment of modelin g as
the p rimary method for undE!rstanding within a highly
uncertain context. For example, if the users are not
familiar with computerized systems or they do not
understand models, then they will be unable to pro­
vide adequate feedback and validation of the mod­
els, and therefore there is no point in p resenting
them with models. Under such conditions, resorting
to prototyping as the primary method for determining
requirements is preferable.
Functional process mode,ling is one of the most
effort consuming activities in the requirements engi­
neering p rocess. Hence, doing excessive modeling
(where excessive means that the added detail pro­
vides l ittle value for understanding the problem and
the solution, comparing alternative solutions, and for
performing a cost/benefits analysis) leads to a lower
likelihood of a highly cost·effective requirements
engineering p rocess. It should also be recalled that,
just because the requirements engi neering phase
was started, it is not predestined that a system will
be developed or a package purchased and adapted.
One o f the main points o f the requirements engi­
neering p rocess is to mak:e a g o/no-go bus i n ess
decision as to whether to continue or stop the pro-
ject after the requirements engineering phase; thus
the p roject may be stopped at the end of the require­
ments engineering phase. This provides further fuel
to the a rg u me nt that excessive detail is not cost­
effective since the project may be terminated at the
end of the requ i rements engineering phase.
Two factors were frequently identified as leading to
h i g h l y d etailed f u n ctional p rocess models. F i rst,
inexperienced analysts tend to spend a dispropor­
tionate effort on model construction. Judging by the
interviewee comments on this issue, it seems to be
one of the most critical problems in the requirements
engineering process. One p roje ct manager elabo­
rates:
They fall in love with modeling. They lack
experience and are too junior in a business
sense. They produce clever models that are
as close to code as they can get them. They
have too narrow a vision of what the models
are or what they are intended to do. [...] Most
of the time you have to hold back [junior ana­
lysts] from going into details. You have to con­
vince them that the extra effort to put details
will not bring much benefits. You have to
make a business decision and make [junior
analysts] move from one deliverable to anoth­
er, even if they think their baby is not beautiful
enough.
The level of detail of models is a project manage­
ment issue, and it is up to the p roject manager to
terminate modeli ng. To achieve this, an experienced
p roject manage r is necessary to know when to stop.
Second, in some organizations, the level of control
is relatively hi gh. This implies that th ey want much
detail in their models. I n some cases, it is the way
the organization operates, the organization is a
bureaucracy. For example, in one case the require­
ments engineering team had to build detailed mod­
els of all the alternative solutions, to the extent that
they could actually implement all of them if they so
wished. Unfortunately, no particu lar recom menda­
tions for alleviating detail i n such a situation could be
identified.
3.3 Examining the Current System
The main pu rpose of examining the current system
is to understand it and then to assess it. The former
i ncludes u nderstanding what the exist i n g system
does, who are the parties involved with the current
system (e.g., users, suppliers, clients etc.), and why
it o pe rates in the current way. This understanding
would always be translated into an assessment of
74
the current system. The assessment includes the
identification of the strengths and weaknesses of the
cu rrent system, and why the cu rrent system is not
adequate and requires replacement or upgrading.
Understanding and assessing the current system
provides the basis for a business case to construct a
new system . Furthermore, where some of the ana­
lysts are not experienced in the application domain,
modeling the existing system is considered a good
way to g a i n such experience. M o reover, it was
observed that models of the existing system can be
reused in developing the new system where there is
a s i m i larity in fu nctional ity. However, some users
and/or I nformation Systems ( IS) o rganizations p refer
not to examine the cu rrent system due to the atti­
tude that "they have already paid for the current sys­
tem and do not want to spend any more money on
it".
In the cases studied, the existing systems were
inadequate and were being replaced or u pg raded
because they w e re i n efficient or i n effecti v e , o r
because they did not support evolving organizational
strategies and objectives. These inadequacies usu­
ally concerned both the IS organization, as well as
the user organization. Example IS organization inad­
equacies are: the existing system is d ifficult to main­
tain due to unpredictable ri p pl e effects in making
changes, the existing system is not wel l documented
and tu rnover or cutbacks are eliminating IS knowl­
edge about the system, and the I S o rganization
w a n t s t o take a d v a n t a g e o f n e w t e c h n o l o g y .
Examp le user organization i n adeq uacies are: the
existing syste m su ppo rts/automates an inefficient
business process, the existing system has unreliable
or inaccurate outputs which results in extra manual
effort to check and correct its outputs before they
can be used, and the existing system requires the
i n put of too much manually collected data to be use­
ful.
In an ideal world, existing systems would be well
documented, and fu nctional and data models for
these systems shou ld be read ily available at the
begin ning of the requirements engineering process.
However, in many of the cases studied, existing sys­
tem documentation was lacki ng . In the rema i ni ng
cases, models of the existing system were available.
This availabil ity was due to the I S organization keep­
ing docu mentation current, or because the organiza­
tion was going through a business process reengi­
neering exercise . I n the latter situation, existing busi­
ness processes and their supporting I nformation
Systems are modeled as part of the reengineering
approach.
 In cases where existing systems had to be mod­
eled during the requirements engineering phase, the
same p robl e ms with the level of detail of models
mentioned above a rose. These are the lack of expe­
rience of analysts and the high level of control of
some organizations.
3.4 User Participation
The primary benefits of user participation were per­
ceived to be: less rework of the documentation items
since the users took part i n their generation and
approval ; greater fit of the recommended solution to
the organization; participation helps reduce political
conflicts among users; participation ensures early
u ser "b uy-in" i nto the system; and participation
reduces the likeli hood of sabotage or users "trying to
defeat the [recommended solution] or the system".
F u rthermore, s i n ce a critical decision has to be
made after the cost/benefits analysis, user participa­
tion ensures that they are convinced of the estimat­
ed costs and benefits of the recommended solution.
W h i l e it was evident that it is one of the most
i mportant factors that contribute to the success of
the requirements engineering phase, a n umber of
factors in hibited user participation. The most com­
mon ones were a h i story of lack of cooperation
betwe e n I S g r o u p s a n d u se r s , o p e n h o st i l it y
between project members a n d users, a n d u navail­
ability of users.
When there is a h i story of lack of cooperat i on
between users and IS, it is surprisingly difficult to
bring about a change towards g reater user participa­
tion . In one case the project manager explains the
situation:
The IS people have the policy of building
products they feel the users will need and
then try to market them to the users. Business
people are not happy about that. [. .J The IS.
people think they understand the business so
well, better than the user, and so they try to
force their ideas on the users. IS may under­
stand the business, but their perceptions of
the business objectives may be different from
the users.
Open hostility occurred when there were severe
personal ity c lashes between a senior IS person
i n volved in the requ i rements engineering process
and a senior user. In one case the personality clash
was severe enoug h that the user refused to validate
the documentation, with excuses akin to "don't you
know what you are doing, why do you want me to
validate it?" A lack of trust between the two fu nc-
75
tions also inhibits user participation.
Furthermore, the most desi rable users are usually
the least available users. Hence, it is frequently diffi­
cult to g et their time commitment to the req ui re­
ments engineering phase. This is a particularly ardu­
ous issue to resolve in small compani es, and with
users in organizations experiencing rapid g rowth. In
addition, participation in the requirements engineer­
ing phase entails an increase in the users' workload
and responsibility; if they are not appropriately com­
pensated, then t h i s may lead to resentment and
eventually lack of cooperation .
The most desi rab l e users h ave a s k i l l set that
i ncludes the following:
• familiar and experienced with computerized sys­
tems;
• good knowledge of the application domain, the
business processes, and the needs of the user
organization;
• good knowledge of systems development
processes (especially the requ irements engineer­
ing process and modeling techniques) and their
deliverables;
• good knowledge of Information System implemen­
tation planning and management, and change
management concepts; and
• able to deal with people and have good communi­
cation skil ls.
I n addition, it is desirable that participating users
have influence by authority within the user organiza­
tion. This is critical because, in most situations, the
participati ng users must subseq uently go back to
their respective departments and ensure the com­
mitment and buy- i n of the remain i n g end-users.
Moreover, if it is necessary to pull user resources,
especially from other departments, the participating
users must be high enough i n the organ izati onal
h i e rarchy to do it. If the recomme nded solution
encompasses a restructuring of the business, low
level managers usually do not have the power or
capacity to decide at that level, especially if other
departments are concerned. A senior partici pating
user also has the authority (and one of the few in an
organization) to i mplement an I nformation System
that req u i res organ izational changes in the user
organization and to secure the resources to do so;
g etti n g early participation and commitment from
such a user is a prerequisite to a successful imple­
mentation.
3.5 Managing Uncertainty
Uncertainty is one of the most acknowledged situa-
tional variables in the software engineering litera­
ture. It has been recommended that uncertainty
should be considered when deciding whether to pro­
totype [13], when deciding what requirements engi­
neering activities to perform [8], and when deciding
which general systems development life cycle mod­
els to follow [2].
Uncertainty denotes the difference between the
amount of knowledge that is required and that is
available about the problem and solution domains.
The greater the uncertainty, the greater the amount
of changes to the requirements engineering docu­
mentation.
Four dimensions to uncertainty have been identi­
fied: user uncertainty, analyst uncertainty, applica­
tion uncertainty, and utilizing system uncertainty.
User uncertainty is concerned with the ability of the
users participating in the requirements engineering
phase to specify requirements. Analyst uncertainty
is concerned with the analysts' ability to elicit
requirements and use these to formulate a recom­
mended solution. Application uncertainty concerns
the software system to be developed and its associ­
ated technology. Utilizing system uncertainty con­
cerns the business process to be automated or sup­
ported by the software system.
Users who contribute toward uncertainty will gen­
erally not understand models and will not be able to
get the right information. The requirements engi­
neering team will not be building the right system.
This situation is more common in small and rapidly
growing organizations where the most capable
users are not availa ble. One project manager
describes the situation in an organization experienc­
ing rapid growth:
There was very littfe di scipline in the manage­
ment structure because the organization was
built in no time, and before you can build [dis­
cipline] t h e company d o u bled in size.
Everybody in there has been promoted about
three levels beyond their level of competence
as managers. Nobody knows the business.
Lack of knowledge of the business process by the
user cannot be overcome except by either replacing
the user, or if the senior analyst has extensive
knowledge of the business.
Capable analysts are able to elicit requirements
from the users and express these as models reflect­
ing a solution. A capable analyst will have a skill set
including the following:
• good knowledge of the application domain and the
business processes of the user organization;
76
• good knowledge of the hardware and software
technology to be used in the Information System;
• able to deal with people, and has good interview­
ing and verbal communication skills;
• good knowledge of and experience with conceptu­
al and functional modeling;
• good knowledge of and experience with the
requirements engineering process and its deliver­
abies;
• good knowledge of and experience in conducting
a cost/benefits analysis;
• good knowledge and experience of issues related
to system implementation and organizational
change; and
• good knowledge of and experience with software
package analysis and evaluation.
Inexperienced analysts are considered not to be
able to ask the right questions or the "tough" ques­
tions to executives, because they never did. One
commonly encountered problem that leads to inca­
pable analysts taking part in the requirement engi­
neering phase has been careless assignment and
hiring practices. It is common for roles in the require­
ments engineering phase to be instantiated by indi­
viduals who are available rather than those who are
capable of doing the job. The following comments
exemplify the problem of hiring practices:
You have c o mpanies who feel that
Information Systems can be built without any
specific skills. Many organizations do not
understand [the need for specific roles]. They
fee/ that IS people can do any role. [. ..] The
company does not necessarily recruit from the
domain. In fact, the senior members complain
that the recruiting habits are "really bad" and
that they were hiring people that were not
qu alified. Th is was beca u s e Hum a n
Resources usually did the hiring. The prob­
lems were manifested in that there was a high
turnover rate at the junior level.
Also, uncertainty was increased when organiza­
tions were rapidly adopting new technologies in an
attempt to gain a competitive advantage. Such new
technologies included object oriented analysis and
design techniques, client/server architectures, and
sophisticated GUl's. Furthermore, unstable busi­
nesses and users that are unable to commit to the
decisions they make provide the analysts with a
moving target, and hence contribute to uncertainty.
One solution recommended for managing high
uncertainty is prototyping. The primary motivations
for building a prototype were: eliciting requirements,
validating requi rements, and determining the feasi­
bility of particular solutions. This is concordant with
t h e recomme n d a t i o n s in t he l i te rature [1 ][ 1 3].
Different kinds of prototypes were also constructed.
These ranged from the simple sc reen mock-ups,
internal prototypes used only by analysts to test new
technolog ies, to f u l l scale prototypes including a
model office and actors to play the roles for demon­
strating the manual and automated processes and
their interactions.
When uncertainty is low, "prototyping never makes
things bad, it only costs more". However, even when
uncertainty is low, prototypi ng has the advantage of
making the users more comfortable with the solu­
tion; it provides them with g reater visibil ity. This, of
cou rse, is even more beneficial when uncertainty is
high. In addition, as Tozer [23] and Alavi [1J note,
p rototyp i n g can lead to improved relati o n s h i ps
between the users and the analysts.
A classic example where prototyping was critical is
summarized below by one of the team members:
Proto typing was started early because the
system was too much state-of-the-art, not like
designing a payroll system, it was something
that does not exist, with very high ergonomic
requirements. It was very multidisciplinary,
with many different types of users. Each of
these disciplines has their own way of work­
ing, so it is not just mechanizing something
that was never mechanized before, but it is
also to standardize the way the information is
presented. The system is multi-media, it has
voice, text, and image. [. . 1 The practical users
.
on the day-to-day job wanted to criticize every
button on the screens, they're saying there's
too many screens, the navigation is too long,
c a n you combine this screen with that?
They're really in the middle of what's going to
change their way of working. They don't even
like unreal data, it has to make sense.
This exam p l e h i g h l ights two facts. Fi rst, when
technology is new and too fragmented, when the
user i nterface is complex, and when there are many
d ifferent kinds of users, a prototype is useful for
assessing the feasibility of the solution, and ensures
g reater user acceptance of the emerging solution. It
also enables the analysts to learn about the different
implementation alternatives. The second fact is the
importance of constructing prototypes of acceptable
quality using real data, a point previously made by
Tate [22].
W he n u s e rs are not fami l i a r w i t h I nfo rmat i o n
77
Technology, there is the contention among lead
architects that it is easier to prototype than to teach
the users about data and process modeling (for the
users to understand the system and to perform vali­
dation). Furthermore, some project managers assert
that in a number of cases the users were too afraid
to admit that they do not understand the models,
and hence may provide erroneous feedback. When
in doubt, it is therefore more prudent to present to
the users a prototype which they can more easily
relate to. In situations where there are users repre­
senting multiple departments in the organization, a
prototype may also help in attaining a consensus on
the recommended solution.
Three prototype dangers were also h i g h l ighted.
The first one concerns users with little I nformation
Technology experience. A typical comment is:
Users can build misconceptions about soft­
ware development [if you prototype). For
example, software is easy to build and devel­
opment has quick turnaround, or systems
don't work and are full of bugs.
While some authors d iscou nt such fears about
users' misconceptions as something that does not
occur in practice [23], others recognize it as a real
danger [22][1 ]. The issue here is that of managing
user expectations during prototyping efforts.
The second danger is that of the capability of the
person building the prototype. If such a person is not
an expert in the p rototyping tOOl, then prototyping
may not achieve its objectives. Finally, there is the
danger of the prototyping tool itself. One lead archi­
tect elaborates:
Prototyping tools have limitations. They are
good with simple applications. When you
exceed the boundaries of the tool and you
have to modify the generated code, then this
results in much greater effort. It really
depends on the tool and its limitations.
This corroborates the dangers identified by Tozer
[23] who states that "No 4GL meets 100% of the
requirements for building systems such as complex
corporate systems which may have to fit in with
existing mainframe systems". However, she goes on
to add that, from her expmiences, modifying the
generated code is not a problem since the tool that
was used generated readable and structured code,
and the person nel constructing the prototype have
extensive experience with the tool.
3.6 Benefits of CASE Tools
A prerequisite to attai ning significant benefits from a
CASE tool is to implement the tool. I n the require·
ments engineerin g processes we studied, the imple·
mentation of CASE tools was only effective in orga·
n izations that had put in place an infrastructure to
support implementation. This i m plies that at least
the req u i re m e nts e n g i ne e r i n g p rocess was well
defined, the analysts received training on the CASE
tool, and there exists some form of CASE tool sup­
port organization. The literature provides evidence
t hat to i m p l e ment a CASE too l , at a m i n i m u m a
methodology must be in place [5][27]. I n organiza­
tions that did not have an adequate infrastructure,
CASE tools we re bought but were not used, CASE
tools were pu rchased and were being used as draw­
ing tools, other non-CASE tools were used solely for
the pu rpose of drawing models, and/or the analysts
p repared the models by hand.
There was no particular trend noticed in terms of
funding sources for the i mplementation of CASE. I n
some o rganizations these came out of project bud­
gets, while i n othe rs funding was centralized and
CASE was considered an I S fu nction asset for use
by all projects.
However, where the source of f u n d i n g i mposed
resou rce constraints, there was no i ncident where
CASE was implemented. Huff [12] and Hayley and
Layman [11] report on the high dollar and time costs
assoc iated w i t h C A S E i m p l e m e ntat i o n . H e n ce,
resource constra ined p rojects o r I S o rganizations
s i m ply can not afford the expense and/o r time to
i mplement CASE. A typical situation is the organiza­
tion being unable to purchase enough copies of the
tool or p u rchase enough workstations for all the par­
ticipating analysts. In such requ i rements engineering
phases, the CASE tool was not used. Zagorsky [27]
also pinpointed the lack of availability of the CASE
tool to all team members as an obstacle to its imple­
mentation.
I nteresti ngly, many project managers in small IS
organizations, usually characterized by low funding,
e mp h a s i zed the need f o r CASE tools ( o r b etter
CASE tools) and the need for more automation. This
e m p ha s i s is based o n t h e p e rcept ion that more
automation , by itself, will i ncrease productivity. This
perception has been further reinforced by the mar­
keti ng efforts of tool vendors and (independent and
corporate) consultants touting the productivity gains
of CASE.
These p redicted gains not only seem attractive to
small I S o rganizations with low funding, but also to
78
other larger organizations. It was repeatedly stated
by the interviewees that IS o rganizations withi n com­
panies facing strong competition in their traditional
markets, where there is a large backlog of systems
to be developed, where a large percentage of effort
is expended evolving legacy systems, where there is
a lack of job security, and/or where top IS manage­
ment is under p ressu re to deliver resu lts q u ickly,
there is a strong perception that CASE is a critical
key to s o l v i n g p ro d u c t i v i t y p ro b l e m s r a p i d l y .
U nfortunately, the literature does not support such
expectations. Productivity gains from CASE are one
of the long-term benefits of CASE implementation
[11][10][16].
3.7 Project Management Capability
The project manager's role in the requi rements engi­
neering process includes planning and control, and
managing the people in the requirements engineer­
ing team. The project manager also has to manage
relationships with the users. This is i mportant since,
as one interviewee expressed it, "if you have a prob­
lem with a user group, they'll remember it for a long
time ". F u rt h e r m o re , the p roject manager s h o u l d
have a good knowledge o f t h e req u i rements engi­
n ee r i n g p rocess, because s/he has to tell team
members what to do and to make sure that they are
doing what they are supposed to be dOing "so as not
to be taken advantage of".
One of the more important activities that a project
manager must perform is estimating and obtaining
a d e q u ate res o u rces (ti m e and f u n d i n g ) f o r t h e
req ui rements engineering p rocess. The majority of
estimates were based on the experience of the pro­
ject manager and estimation grids for typical require­
ments engineering phases.
A few of the project managers interviewed conced­
ed that t h e re ex i sts p ress u re to u n d e re sti m ate
resources, especially when the I S and/or user orga­
nization is not benefits oriented ( i . e . , decisions are
based main ly on costs rather than on a cost/benefits
ratio) . T h i s l eads to reso u rc e const ra i nts in the
r e q u i re m e n t s e n g i n e e r i n g p ro c e s s . E xa m p l e
sources of pressure to u nderestimate are: political
reasons (for example the politics of budgeting), and
some I S organizations are simply not accustomed to
m a k i n g large i nvest m e nts in I S . Sched u l e c o n ­
straints are also created b y hard deadlines that are
decided outside the IS function. For example, when
one utility had to change its billing process, the sys­
tem that supports billing had to be in place at a spe­
cific date because commitments were already made
to customers. Also, in government agencies, when
there is new legislation, for example new taxes, the
system that suppo rts it must be operational in time
for when the legislation comes into effect.
The consequences of resource constraints (unreal­
istic budgets and schedules) is that they force the
requirements eng ineering team to be "innovative".
This can take the form of cutting down on process
and data modeling activities, to the detriment of the
team's ability to formulate a realistic business case.
A ls o , package s o l u t i o n s a re favored g ive n t h e
resou rce constraints, but the user requ i rements are
c o m p r o m ised to avo i d t h e costs of adaptat i o n .
Furthermore, project management activities are cur­
tailed (having only part-time p roject managers and
reduce p rogress reporting).
Experienced p roject manage rs u nd e rstand the
need to sched ule time for decision making d u ring
the requ i rements eng ineering phase. Fai l u re to do
so may result in g ross underestimates of resources.
Organizations vary widely in the speed with which
they can make decisions. I n some organizations the
decision making p rocess is consensua l , o r every
decision requ i res multiple layers of approval. One
p roject manager commented about a government
�gency where no one actually makes a decision, it
Just emerges. This way no one can be held respon­
sible for a particular decision. It is thus important for
t h e p roject manag e r to est i m ate acc u rately the
delays d u e to decision maki n g lags and incl ude
these in the project plan.
Project managers lacking capabil ity are assigned
to this role for two commonly cited reasons. The first
is the general practice of aSSigning people to roles
based on their availability not capability. The second
is the career track that leads people to project man­
agement positions:
In many cases, people are promoted to the
project manager role based on years with the
company, not based on capability. They don't
understand the importance of project man­
agement, so they assign anyone to project
management, usually an analyst. They do not
understand that a good analyst does not
[always} make a good project manager.
The skill set of a capable project manager should
include the following:
• able to use automated project management tools;
• has good esti mation capabilities:
• able to decompose and monitor activities;
• able to deal with people, and has good negotiation
and verbal communication skills;
79
• g o o d k n ow l e d g e of t h e system d e ve l o p me n t
p rocess (in particular the req u i rements eng ineer­
Ing process) and its deliverables; and
• has a good understanding of emerging req ui re­
ments engineering technologies and tools.
4 Conclusions
It is evident from the foregoing discussion that, as
well as technical issues, non-technical issues are of
! mportance in the requirements engineering process
In IS development. Specifically, the problems faced
concern performing the appropriate activities (e.g.,
whether to consider packages and whether to proto­
type), deciding when to stop particular activities (e.g,
functional process modeling and examination of the
cu rrent system), securing an adequate level of user
partici patio n , and selecting capable perso nnel to
i nstantiate key roles in the requirements engineering
phase (e.g, the p roject manager, lead architect, and
principal user). Furthermore, the main problem with
attaining benefits from CASE tools were i mplemen­
tation p roblems, l ike training the analysts on the tool
and the underlying methodology.
F i n d i ng that non-technical issues are important
supports the beliefs of other researchers in software
process improvement [1 8], and is concordant with
the results of an earlier study of requi rements engi­
neering p ractices where the authors state [ 1 5] : "It
was strikingly obvious that many of the problems
faced by the projects we interviewed were organiza­
tional and non-technical. Moreover, projects tend to
adopt organizational solutions to most of their prob­
lems, whether technical or non-technical". I n addi­
tion, the results of this study complement existing
works by identifyin g i mpedi ments to the i mplementa­
tion of what are considered to be good requ i rements
engineering practices.
F u rt h e r c o n f i d e n c e i n o u r r e s u l t s w o u l d be
obtained if other researchers replicate this study or
cond uct similar studies using a different sample of
projects and informants. Moreover, the employment
of alternative empirical research methods (e.g . , sur­
veys a n d q u a s i - e x p e r i m e n t s ) i n i nvest i g a t i n g
requ i rements engineering p ractices would ascertain
the extent to which our results are affected by the
case study approach that we followed. Such investi­
gations coul d , for i n stance, util ize o u r findings as
hypotheses to be tested .
M a n y o f t h e identified p roblems are o f concern
mainly during the planning of the requi rements engi­
neering process (e.g., deciding whether to prototype
and whether to consider package solutions). Future
research should therefore also focus on providing
automated and non -automated tools and methods
that support decision making d u ring the p la n n i ng
activities. For example, a tool t hat would assist in
deciding what activities to perform and for h ow long.
This would be a requirements engi neering process
design or customization assistance tool . When the
prospect of such a tool was proposed to the intervie­
wees, there was h i g h interest and strong agreement
that it would be beneficial, not only to assist decision
making, but also for trai ning p roject managers on
how to plan requirements engineering p rocesses.
Anoth e r h i g h leve rage area of research t hat i s
identifiable from t h i s study is t h e development of a
method or i nstrument to assess the capability of key
requirements engineering personnel. G iven that pre­
vious studies of req uirements engi neering p ractices
have found that personnel capabil ity and knowledge
h ave a n o n t r i v i a l i mpact on t h e success of t h e
req u i re ments e n g i ne e r i n g p rocess [6] [ 1 5), s u c h
assessments wo uld be invaluable f o r staffing and
training purposes.
Acknowledgements
The authors wish to thank Lionel B riand for his com­
ments on an earlier version of this paper.
References
[1 ] M. Alavi. "An assessment of the prototyping approach
to i n f o rmati o n systems d e ve l o p m e n t " . In
Communications of the A CM, 27(6) : 556-563, J une
1 984.
[2] L. Alexander and A. Davis. "Criteria for selecting soft­
ware process models" . I n Proceedings of the 1 5th
International IEEE COMPSAC, pages 52 1 -528, 1 991 .
[3] V. Basili and B. Perricone. "Software errors and com­
p l e x i ty : A n e m p i r i ca l i n ves t i g a t i o n " . In
Communica tions of the A CM, 27( 1 ) :42-52, Janu ary
1 984.
[4] M. Bryce and T. Bryce. "Make or buy softwa re? " . I n
Journal of Systems Management, pages 6-1 1 , Au gust
1 987.
[5] K. C ulver-Lozo and V. Glezman. "Experiences apply­
i n g methods s u p p o rted by CAS E t o o l s " . I n A T& T
Technical Journal, pa g e s 20-26,
N ovember/December 1 992.
[0] B. C u rtis, H . Krasner, and N . ISGoe. "A field study of
t h e software desi g n process for large systems". I n
Communications of the A CM, 3 1 ( 1 1 ) : 1 26 8 - 1 2 8 6 ,
N ovember 1 988.
[7] E . D a ly. "Management of software d eve l opm e nt" . I n
IEEE Tra nsactions on Software Engineering, S E-
3(3):229-242, May 1 977.
[8] G. Davis. " St rate gi es for i n fo rmat io n req u i reme nts
dete rmination". In IBM Systems Journal, 21 (1 ):4-3 1 ,
1 982.
[9] A. E ndres. "A n analysis of errors and their cau ses i n
80
system programs". In IEEE Transactions on Software
Engineering, SE-1 (2): 1 40-1 49, J un e 1 97 5 .
[ 1 0] M . Gibson, C. S n yde r a n d K. R a i n e r J r . . "CAS E :
C larifying common misconceptions". I n Journal o f
Systems Management, pag es 1 2- 1 9, May 1 989.
[1 1 ] K. Hayley and T. Lyman. "The realities of CASE". In
J ournal of Information Systems Management, pages
1 8-23, Summer 1 990.
[ 1 2] C. HuH. "Elements of a realistic CASE tool adoption
budget". In Communications of the ACM, 35(4):45-54,
April 1 992.
[ 1 3] M . Janson and L. Smith. "P roto ty pi ng fo r systems
deve l o pm en t : A critical app rai sal" . In MIS Quarterly,
pages 305-3 1 6, December 1 985.
[ 1 4] C. Jones. "Assessm ent and control of software risks".
Prentice Hal l , 1 994.
[ 1 5] M. Lubars, C. Potts, and C. Ri chte r . "A review of the
state of t he practice in r e q u i re m e nts mod e l i n g " . I n
Proceedings of the IEEE International Symposium on
Requirements Engineering, pages 2 - 1 4 , J an u a ry
1 993.
[ 1 6] C. Kemerer. "How the learning curve affects CASE
tool adoption". I n IEEE Software, pages 23-28, May
1 992.
[ 1 7] D . Perry. "Human prospects of process design". I n
Proceedings o f the Seventh International Software
Process Workshop, pages 22-27, 1 992.
[1 8] D . Perry, N . Staudenmayer, and L. Votta. "People,
organizations, and p rocess i m pro ve m e n t" . I n IEEE
Software, pages 36-45, J u ly 1 994.
[ 1 9] R . R u be y , J . D a n a a n d P. B i c h e . " Q u a n t i tative
aspects of software validation". I n IEEE Transactions
on Software Engineering, S E - 1 (2 ) : 1 5 0- 1 55 , J u ne
1 975.
[20] S of t w a r e E n g i n e e r i n g L ab o rato r y . " S o f t w a re
En gi neering Laboratory (SEL) relationships, models,
and measurement rules". Technical Report SEL-9 1 -
001, NASA Goddard Space Flight Center, February
1 991 .
[2 1 ] E. Swanson and C. Beath. "The use of case study
data in software management research". In Journal of
Systems and Software, 8(1 ):63-71 , January 1 988.
[22] G. Tate. "Prototyping: Helping to build the rig ht soft­
ware". I n Information and Software Techn ology,
32(4):237-244, May 1 990.
[23] J. T o z e r . " P ro t ot y p i n g as a system develo p m e n t
m e t h o d o l o g y : O p p o rt u n i t i e s a n d p i t f a l l s " . I n
Information and Software Technology, 29(5):265-269,
J u ne 1 987.
[24] S. Vallabhaneni. "Auditin g software development: A
manual with case studies". John W i l ey & Sons, 1 990.
[25] M. Wu. "Selecting the right software application pack­
age". In Journal of Systems Management, pages 28-
32, September 1 990.
[26] R. Yin. "Case study resea rc h , d es i gn an d method s" .
Sage Publications, 1 984.
[27] C. Zagorsky. "Case study: Managing the change to
C A S E" . I n Journal of Information Sys tems
Management, pages 24-32, Summer 1 990.