A Progression Model of Software Engineering

This article has been accepted for
This
publication
is the author's
in a future
versionissue
of an
of article
this journal,
that has
butbeen
has published
not been fully
in this
edited.
journal.
Content
Changes
maywere
change
made
prior
to this
to final
version
publication.
by the publisher
Citation prior
information:
to publication.
DOI 10.1109/TSE.2019.2900213, IEEE
The final version of record
Transactions
is available
on Software
at Engineering
http://dx.doi.org/10.1109/TSE.2019.2900213
JOURNAL OF LATEX CLASS FILES, VOL. 13, NO. 9, SEPTEMBER 2014 1
A progression model of software engineering

goals, challenges, and practices in start-ups
Eriks Klotins, Michael Unterkalmsteiner, Member, IEEE, Panagiota Chatzipetrou,
Tony Gorschek, Member, IEEE, Rafael Prikladnicki, Nirnaya Tripathi, and Leandro Bento Pompermaier
Abstract—Context: Software start-ups are emerging as suppliers of innovation and software-intensive products. However, traditional
software engineering practices are not evaluated in the context, nor adopted to goals and challenges of start-ups. As a result, there is
insufficient support for software engineering in the start-up context.
Objective: We aim to collect data related to engineering goals, challenges, and practices in start-up companies to ascertain trends and
patterns characterizing engineering work in start-ups. Such data allows researchers to understand better how goals and challenges
are related to practices. This understanding can then inform future studies aimed at designing solutions addressing those goals and
challenges. Besides, these trends and patterns can be useful for practitioners to make more informed decisions in their engineering
practice.
Method: We use a case survey method to gather first-hand, in-depth experiences from a large sample of software start-ups. We use
open coding and cross-case analysis to describe and identify patterns, and corroborate the findings with statistical analysis.
Results: We analyze 84 start-up cases and identify 16 goals, 9 challenges, and 16 engineering practices that are common among start-
ups. We have mapped these goals, challenges, and practices to start-up life-cycle stages (inception, stabilization, growth, and maturity).
Thus, creating the progression model guiding software engineering efforts in start-ups.
Conclusions: We conclude that start-ups to a large extent face the same challenges and use the same practices as established
companies. However, the primary software engineering challenge in start-ups is to evolve multiple process areas at once, with a little
margin for serious errors.
Index Terms—software start-up, software engineering practices, progression model
1 I NTRODUCTION lion USD in US start-ups in 2017 alone [8]. Building the first
Software start-ups are small companies created to build and version of a product is a substantial engineering challenge
market a software-intensive product [1]. Start-ups are char- and precedes any market or business related difficulties [6],
acterized by rapid evolution, small teams, uncertainty about [9]. Thus, shortcomings in applied engineering practices
customer needs and market conditions, and a high failure could waste the investment, and hinder any subsequent
rate [2], [3]. However, leveraging cutting-edge technologies, attempts to market the product and to build a sustainable
risk, and speed, start-ups can launch software products business around it. Even if a fraction of start-up failures
fast [4], [5]. could be attributed to engineering failures, that would still
Aims, objectives, and challenges of product engineering present an opportunity for better, start-up specific, engineer-
change quickly as a start-up evolves [6]. State-of-the-art en- ing practices, and a relevant area of research.
gineering methods offer little support for understanding the An increasing number of studies attempt to explore
evolving context and selecting the right practices [2], [7]. A engineering practices in a start-up context, for example,
miscalculation in choosing engineering practices could lead requirements engineering [10], [11], [12], technical debt [13],
to over or under-engineering of the product, and contribute and user experience [14]. Several studies, such as Gia-
to wasted resources and missed market opportunities [4]. rdino et al. [15] and Crowne [6], attempt to explore and
According to industry reports, a record of 19.2 billion present conceptual models of product engineering in start-
EUR of venture capital was invested in European and 80 bil- ups. However, none of these studies provide a full and
detailed answer to what engineering practices start-ups use
• E. Klotins, M. Unterkalmsteiner, and T. Gorschek are with the Software concerning different engineering process areas and start-up
Engineering Research Lab Sweden, Blekinge Institute of Technology, evolution stages. The need to better understand product
Karlskrona, Sweden. engineering in start-ups, and to provide relevant support
• P. Chatzipetrou is with the Software Engineering Research Lab, Blekinge
Institute of Technology, Karlskrona, Sweden, and Department of Infor-
for practitioners, has been highlighted by Unterkalmsteiner
matics, CERIS, Örebro University School of Business, SE-701 82 Örebro, et al. [16] and Klotins et al. [17].
Sweden With this study, we aim to understand how start-ups
• R. Prikladnicki and L. Pompermaier is with Pontifical Catholic University use different engineering process areas, and how utilized
of Rio Grande do Sul, Brazil
• N. Tripathi is with University of Oulu, Finland practices evolve over start-up life-cycle. Through our anal-
ysis, we present a progression model of what engineering
Corresponding author: Eriks Klotins, eriks.klotins@bth.se aspects, that is, goals, practices, and challenges are relevant
Manuscript received XX; revised XX in start-ups in their evolution stage. The model is aimed
0098-5589 (c) 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
AuthorizedCopyright
licensed(c)use
2019limited
IEEE. Personal use is permitted.
to: Universidad AutonomaFor any
deother purposes,
Madrid. permissionon
Downloaded must
May be 02,2020
obtained from the IEEE UTC
at 03:06:32 by emailing pubs-permissions@ieee.org.
from IEEE Xplore. Restrictions apply.
This
publication
is the author's
in a future
versionissue
of an
of article
this journal,
that has
butbeen
has published
not been fully
in this
edited.
journal.
Content
Changes
maywere
change
made
prior
to this
to final
version
publication.
by the publisher
Citation prior
information:
to publication.
DOI 10.1109/TSE.2019.2900213, IEEE
Transactions
is available
on Software
at Engineering
at supporting practitioners in their decision making process Two subsequent literature reviews were published by
and at pinpointing specific engineering challenges for fur- Klotins et al. [24] and Berg et al. [3] aiming to map state-
ther investigation. of-the-art in start-ups with SWEBOK [25] knowledge areas.
We use an adapted case survey method [18] to collect They conclude that there are many gaps and opportuni-
and analyze primary data on engineering practices in 84 ties for developing start-up specific engineering practices.
start-up cases. The cases vary by geographical location, However, they attempt to analyze state-of-the-art in start-
development stage, outcome, and time of operation among ups with SWEBOK that isn’t start-up specific and lacks
other factors, thus presenting a relatively large and diverse several relevant knowledge areas, for example, market-
sample [3]. To explore start-up goals, challenges, and used driven requirements engineering and value driven software
practices, we propose the start-up life-cycle model and engineering.
analyze cases within the same development stage, and with Giardino et al. [7] proposes the Greenfield start-up model
the same outcome. We apply qualitative methods to identify identifying the main engineering categories and their re-
patterns in the data and arrive at explanatory results. The lationships. The model identifies severe lack of resources,
explanatory results are verified and complemented with sta- teamwork, rapid development, little focus to quality, evolu-
tistical analysis providing a firm basis for our conclusions. tionary approach, and technical debt as the key categories.
Our study provides several novel contributions. Firstly, The relationships reveal that development speed is achieved
we present a start-up life-cycle model, aimed at illustrating by capable teams, little focus on quality assurance and
the dynamically evolving nature of start-ups. Secondly, we internal product quality. However, such approach leads to
use the life-cycle model to describe what practices, goals, accumulation of technical debt and hindered performance
and challenges are relevant to start-ups at different life-cycle in the long term.
stages. Thirdly, we present the start-up progression model
aimed at guiding practitioners and at illustrating relevant
2.3 Software engineering practices in start-ups
areas for further exploration.
The remainder of the paper is structured as follows: in Earlier studies point out that start-ups often do not follow
Section 2 we define software start-ups and summarize exist- best engineering practices and opt for an ad-hoc approach
ing work in the area, in Section 3 we describe our research to engineering. Such an approach to engineering is partly
methodology, in Section 4 we report and analyze our results, due to the immaturity of start-ups, rapidly changing envi-
in Section 5 we interpret and discuss our findings, Section 6 ronment, and lack of engineering expertise. However, there
concludes the paper. is also limited support from academia pinpointing engineer-
ing practices that could be suited for such context [16], [17],
[26].
2 BACKGROUND AND RELATED WORK Part of the difficulty to practice and study software
2.1 Software start-ups engineering in start-ups is the lack of knowledge transfer
As early as 1994 Carmel [20] reported on small compa- between start-ups. All knowledge about a domain, ways-of-
nies building and marketing innovative software prod- working, and practices is carried by individuals, thus is lost
ucts. These small companies, or start-ups, prioritize time- when a start-up closes down, and needs to be reinvented
to-market over product quality, teams are small and self- with every new start-up. Thus, a large part of start-up
motivated, and engineering practices informal. Later stud- research is to establish a body of knowledge with the best
ies, for example, Giardino et al. [4], and Sutton et al. [1] engineering practices for start-ups [17], [27].
report the very similar characterization of start-ups. Several studies attempt to explore product engineering
Start-up companies are known for their high failure rate. practices in start-ups. For example, Gralha et al. [11] and
About 75 - 99% of start-up products fail to achieve any Melegati et al. [10] explore requirements engineering prac-
meaningful results in market [21], [22]. The high failure rate tices in start-ups. These studies suggest that requirements
could be explained by market challenges, team issues, diffi- engineering is one of the key engineering process areas in
culties in securing funding and so on. However, the capabil- start-ups, and help to explore the market opportunity and
ity to build software efficiently with limited understanding to devise a feasible solution.
about stakeholder needs and with limited resources is the Hokkanen et al. [28] present a framework guiding user
foremost challenge in software start-ups and precedes any experience design work in a start-up context. They argue
market or business related challenges [23]. that for a product to be successful in a market it has to fulfill
minimal functional and user experience requirements. The
framework identifies and prioritizes various user experience
2.2 What do we know about software start-ups? elements guiding user experience focus.
A broader interest to study software start-ups from software In our earlier work (Klotins et al. [13]) we explore tech-
engineering perspective was launched by publication of a nical debt in start-ups. We found that excessive technical
systematic review on existing literature in the area [2]. Se- debt could be a cause for missed market opportunities and
lected results from this review were also published in IEEE contribute to start-up failures. Furthermore, they identify
Software [4]. In the review, authors point out the potential several strategies that could help to expose unwanted tech-
of software start-ups as vehicles for innovation, and lack nical debt early.
of relevant research in the area. The review lists a number The earlier work suggests that engineering practices are
of contextual challenges to software product engineering in rapidly evolving to accommodate the changing engineering
start-ups compared to established companies. context [7], [11], [23], [28]. Even though, individual areas,
AuthorizedCopyright
licensed(c)use
2019limited
deother purposes,
Downloaded must
May be 02,2020
This
publication
is the author's
in a future
versionissue
of an
of article
this journal,
that has
butbeen
has published
not been fully
in this
edited.
journal.
Content
Changes
maywere
change
made
prior
to this
to final
version
publication.
by the publisher
Citation prior
information:
to publication.
DOI 10.1109/TSE.2019.2900213, IEEE
Transactions
is available
on Software
at Engineering
I Inception II Stabilisation III Growth IV Maturity

A start-up aims to build the first The product is developed further with The aim is set to achieving desired A start-up transitions into an
version of their product customer input and prepared for market share, growth rate established organisation
scaling
Acquired Acquired Acquired

Acquired
(1) (4) (1)
Active Active Active Active

(8) (18) (17) (15)
Closed Closed
Closed Closed
Paused (5) Paused (2)
(4) Paused Paused
(3) (2)
Marketing/growth/maintenance focus
Problem/solution focus
Fig. 1. The start-up life-cycle model based on Crowne [6] and Churchill et al. [19]. The white bubbles indicate “good” and desired states. The shaded
bubbles show the undesirable states. Arrows denote possible transitions between the states. States and transitions that are possible, were however
not observed in this study, are denoted by dashed lines. The numbers in brackets indicate how many cases representing each state were observed.
for example, requirements engineering and user experience 3) Growth - at this stage the focus is set on attaining the
design, has been studied, there is a lack of a coherent view desired market share and growth rate. Although the ef-
of how different engineering areas fit and evolve together. forts shift towards marketing and sales, the engineering
Such complete understanding is needed to analyze why team should cope with a flow of new customer require-
specific practices are applied, and to spot potential misalign- ments, and product variations for different markets.
ment between engineering process areas. 4) Maturity - in this stage a start-up transitions into an
established organization aiming to preserve established
2.4 Start-up life-cycle models market share and to optimize its operations. The engi-
neering team should install routines for operating and
To illustrate the changing objectives of a start-up, there have
maintaining the product.
been attempts to define start-up life-cycle models. Blank [29]
identifies two stages, search for a viable market opportunity, These four stages represent the shift in start-up objec-
and building a viable business around the opportunity. Each tives. Early stages focus on finding a relevant problem, de-
stage consists of several objectives outlining the need to vising a feasible solution. Later, the focus shifts to marketing
explore and validate a potential customer need first, then and improving the efficiency of start-up’s operations [6],
propose and validate a potential solution, and finally scale [31].
up the operations. This model, however, is generic and In case a start-up decides to radically change some
offers little guidance for software engineers. fundamental aspect of its product, that is, to pivot [32],
Inspired by Churchill et al. [19] and Crowne [6] we the product likely moves to an earlier phase in the life-
combine start-up product evolution stages with relevant cycle model. For example, discarding existing features and
organization states (Fig. 1). In this paper, we use the four developing new ones implies abandoning any marketing
stages of a start-up as a basis for explaining start-up evolu- or stabilization efforts related to the abandoned features.
tion and define them as: Developing entirely new features throws the start-up back
1) Inception - a stage between ideation of a product until to inception stage for scoping, validation and piloting.
the start-up releases the first product release to the first At any of the four product stages a start-up strives to: a)
customer. The primary goal of this stage is to scope and remain active and continue operations (that is, not to fail),
build the minimum viable product by balancing needs b) advance to the next stage or c) be acquired by another
of a customer, available resources, and time [30]. company for profit to shareholders. Alternatively, at any of
2) Stabilization - a stage between first product release until these stages, a start-up may be closed down or paused. We
readiness for scaling. In this stage, a start-up aims to en- define organization states as the following:
sure that the product can be decommissioned without 1) Active - the team actively works on the product.
adding extra effort to the development team. That is, 2) Paused - the team has stopped working, however there
the product should be easy to maintain, scale, and sur- is an intention to resume at some point in future.
rounding infrastructure, for example, operations and Reasons for pausing a start-up could be, for example,
customer support, are in place. a temporary shift in founders priorities, temporary
AuthorizedCopyright
licensed(c)use
2019limited
deother purposes,
Downloaded must
May be 02,2020
This
publication
is the author's
in a future
versionissue
of an
of article
this journal,
that has
butbeen
has published
not been fully
in this
edited.
journal.
Content
Changes
maywere
change
made
prior
to this
to final
version
publication.
by the publisher
Citation prior
information:
to publication.
DOI 10.1109/TSE.2019.2900213, IEEE
Transactions
is available
on Software
at Engineering
lack of funding for product development or marketing c) Provide a blueprint of software-intensive product engi-
among other scenarios. neering in a start-up context [17].
3) Acquired - another company acquires the start-up for a We formulate three sub-questions to explore goals,
profit to shareholders. The team is disbanded or merges challenges, and practices specifically.
with the other company.
4) Closed - the team is disbanded or works on something RQ1.1: What goals, relevant to software engineering,
else. can be ascertained in start-up companies?
The combined model of start-up and product life-cycle Rationale: We aim to explore what goals are driving software
states is shown in Fig. 1. We use this model as an input to engineering in start-ups concerning their life-cycle stage,
our study and to capture the state of each studied case. see Fig. 1. A fine-grained understanding of the goals, i.e.
drivers for engineering activities, helps to understand the
context of why certain engineering practices are used, or
3 R ESEARCH METHODOLOGY avoided [17], [37].
We use a case survey method to collect and analyze primary
data from start-up companies. The method is aimed to bal- RQ1.2: What challenges relevant to software engineering
ance studying a few cases in depth with traditional (multi) can be ascertained in start-up companies?
case studies and quantitatively studying many cases [35]. Rationale: Engineering challenges is another context factor,
Case studies offer greater level of detail and internal va- alongside goals, shaping engineering practices in start-ups.
lidity by closely examining multiple data sources. Surveys We aim to explore what specific challenges, associated with
attempt to collect data from a large number of cases, thus start-up life-cycle stages, can be ascertained in start-ups.
achieving a higher degree of generalizability [36].
The main advantages of the case survey method are RQ1.3: What software engineering practices do start-
its ability to collect richer information than conventional ups use?
survey, and extendibility to study more cases. That said, the Rationale: We aim to explore what engineering practices
data collected by a case survey are limited by the scope of start-ups apply as a response to life-cycle stage-specific
the questionnaire instrument [33], [35]. Validity threats of goals and challenges.
our study are discussed in Section 3.3.
The original case survey method description sug-
3.2 Data collection and analysis
gests using coding and statistical methods to analyze the
data [33]. However, we extend the method by adding more 3.2.1 Questionnaire design
steps from the theory building process proposed by Eisen- We base the data extraction on a questionnaire eliciting
hardt [34]. While both methods are compatible, Eisenhardt practitioner experiences about their specific start-up case.
provides more details on inducing a theory from data, that The scope of the survey is inspired by our earlier work the
is, in this study, the start-up progression model. We present Start-up Context Map [17] and covers team, requirements
a mapping between the two methods and the combined engineering, value, quality assurance, architecture and de-
method in Table 1. sign, and project management aspects of start-ups.
During the questionnaire design, we conducted multiple
3.1 Research questions internal and external reviews to ensure that all questions are
relevant, easy to understand and that we receive meaningful
To guide our study, we define the following research
answers.
questions (RQ):
The internal reviews were conducted among the authors
to determine the scope and flow of questions. For the
RQ1: What patterns pertaining software engineering
external reviews we invited 10 members of Software Start-
can be ascertained in start-up companies?
up Research Network1 to provide their input. Firstly, we
Rationale: Very little is known of what software engineering
asked them to fill in the questionnaire and answer all the
practices start-ups use and what is the motivation for using
questions as if they were engineers in a start-up. Then, we
or avoiding specific practices. Earlier studies report the
organized a joint on-line workshop to discuss participants
use of light-weight, ad-hoc practices with emphasis on
reflections and potential improvements to the questionnaire.
requirements engineering [2], [10], [24]. However, most
Their responses were removed from the final dataset.
earlier reports use secondary data, explore only a few cases,
Finally, we piloted the questionnaire with four practi-
and are based on limited understanding of engineering
tioners from different start-ups. During the pilots, respon-
context in start-ups.
dents filled in the questionnaire while discussing questions,
With this research question, we identify commonalities
their answers and any potential issues with the first author
in engineering goals, challenges, and used software en-
of this paper.
gineering practices in start-ups with respect to their life-
As a result of these reviews, we improved the question
cycle stage, see Fig. 1. An understanding of what goals
formulations and removed some irrelevant questions. The
and challenges shape the use of engineering practices are
finalized questionnaire contains, 10 sections, 85 high level
essential to:
questions and 285 sub-questions, 73 of the sub-questions
a) Judge suitability of commonly used practices. are free-text, others are on an ordinal or nominal scale.
b) Devise new engineering practices to navigate specific
challenges and to achieve particular goals. 1. Software Start-up Research Network: http://softwarestartups.org
AuthorizedCopyright
licensed(c)use
2019limited
deother purposes,
Downloaded must
May be 02,2020
This
publication
is the author's
in a future
versionissue
of an
of article
this journal,
that has
butbeen
has published
not been fully
in this
edited.
journal.
Content
Changes
maywere
change
made
prior
to this
to final
version
publication.
by the publisher
Citation prior
information:
to publication.
DOI 10.1109/TSE.2019.2900213, IEEE
Transactions
is available
on Software
at Engineering
TABLE 1
Steps of the research method
Case survey
method Theory building (Eisenhardt [34]) The applied method
Step
(Larsson [33])
We start the study with a broad aim to collect primary data
Define research questions and any a on how start-ups practice software engineering. We define
1 -
priori constructs our research questions in Section 3.1, and present the
start-up life-cycle model in Fig. 1.
We aim to study start-up companies building
Select cases of
2 Specify a population software-intensive products and reach as diverse sample of
interest
start-ups as possible.
Design the data We design a questionnaire for data collection. It is aimed at
Craft instruments and protocols for
3 extraction form for start-up practitioners with practical experience with
data collection
elicitation software engineering in start-up setting, see Section 3.2.1.
We parallelize the final steps of questionnaire design with
Parallelize data collection and
4 - the data collection to identify any issues with the
analysis
questionnaire before scaling up the data collection.
First, we use open coding to extract relevant information
Conduct within case analysis,
from textual data. Secondly, we use the start-up life-cycle
5 Conduct the coding search for cross-case patterns using
model, see Fig. 1, to categorize the cases and conduct
divergent techniques
cross-case analysis, see Section 3.2.3.
Use statistical Iterative shaping of hypotheses, We condense findings from the previous step into more
6 approaches to search evidence for ”why” behind broader patterns, and perform a parallel quantitative
analyze the coding relationships. analysis complementing qualitative results, see Section 3.2.4.
We compare patterns from the previous step with literature.
Comparison with conflicting and
7 - We seek for additional support and explanation for our
similar literature
findings, see Section 3.2.4.
Aim for theoretical saturation when -
8 -
possible
Note that one question in the questionnaire may result in TABLE 2

multiple sub-questions, for example, a high level question Topics covered by the questionnaire
may result in two sub-questions one capturing a Likert-
scale answer, another free text motivation for the answer. Number
Process
The sections cover many software engineering topics, see # Topics Questions of sub-
area
questions
Table 2. Note that through the analysis process some topics
Start-up
were merged, and some lifted out. The last column of the 1 1-8 12 -
demographics
table, process area, shows a mapping between sections of
Product
the questionnaire and the resulting process areas. 2
demographics
9 - 17 18 -
From all the questions, 54 sub-questions focus on captur-
Respondent
ing respondents agreement with statements addressing their 3
demographics
18 - 23 10 I
engineering practices and engineering context. To capture
4 Team demographics 24 - 30 12 I
respondents level of agreement with a statement we use a
Likert scale: not at all (1), a little (2), somewhat (3), very Requirements
5 31 - 47 57 II, III
engineering
much (4). The values indicate the degree of agreement with
a statement. Statements are formulated consistently in a way 6 Software architecture 48 - 55 19 V
that lower values indicate less agreement, however higher 7 User interface design 56 - 59 9 V
values indicate more agreement. We specifically avoid neu- Development
8 60 - 69 80 I - VI
tral (neither agree or disagree) answer option to force re- practices
spondents to state their opinion. However, we provide the 9 Quality assurance 70 - 77 48 IV
“I do not know” option to allow respondents to explicitly
skip the question. Project management 78 - 85 20 VI
10
All the questions and answer options are available as
Total 285
supplemental material2 .
3.2.2 Distribution of the survey and data collection
To distribute the survey we reached out to The Software to use their networks and connections. All authors of this
Start-up Research Network3 and asked other researchers paper actively promoted the survey through their on-line
2. Full questionnaire:
accounts and personal contacts. The first author promoted
http://eriksklotins.lv/files/GCP questionnaire.pdf the study in several start-up themed events. The data col-
3. Software Start-up Research Network: http://softwarestartups.org lection took place between December 1, 2016 and May 15,
AuthorizedCopyright
licensed(c)use
2019limited
deother purposes,
Downloaded must
May be 02,2020
This
publication
is the author's
in a future
versionissue
of an
of article
this journal,
that has
butbeen
has published
not been fully
in this
edited.
journal.
Content
Changes
maywere
change
made
prior
to this
to final
version
publication.
by the publisher
Citation prior
information:
to publication.
DOI 10.1109/TSE.2019.2900213, IEEE
Transactions
is available
on Software
at Engineering
2017. and the second author jointly. In this process, memos with
To support the data collection we created an on-line tool. limited support from the data are removed, or merged
The landing page of the tool contained a short description of with other similar memos. Interesting analysis points were
the study aims, and a promotional video encouraging start- marked for additional analysis. This is an iterative process
ups to share their experiences. The questionnaire was public leading to formulation of our findings.
and freely available to everyone. To screen the responses As the final steps of our analysis, we use descriptive
and gauge their suitability for our study, the questionnaire statistics and contingency tables to identify new and to
contains a multitude of demographical questions about confirm already identified patterns [39]. We illustrate our
the start-up and the respondent. For example, what is re- results with frequency analysis and histograms, and test
spondents relationship with the start-up, their role in the statistical significance of our findings.
company, and how long time ago they were in contact with We use the Chi-Square test of statistical association to
the start-up? test if the associations between the examined variables are
A total of 369 practitioners started to answer the ques- not due to chance. To prevent Type I errors, we used
tionnaire. However, many of the responses were incomplete. exact tests, specifically, the Monte-Carlo test of statistical
We removed responses with less than 50% of questions significance based on 10 000 sampled tables and assuming
answered. We further removed several duplicates and re- (p < 0.05) [40].
sponses from non-software start-ups. The response rate of To examine the strength of associations we use
the questionnaire was 23%, 84 out of 369. Cramer’s V test. We interpret the test results as suggested
by Cohen [41]. That is, we consider thresholds of 0.1, 0.3,
3.2.3 Coding scheme and cross-case analysis
and 0.5 for weak, moderate and strong associations.
We perform the cross-case analysis using both qualitative To explore the specifics of an association, such as which
and quantitative methods. cases are responsible for this association, we perform post-
The responses are already structured by questions, and hoc testing using adjusted residuals. We consider an ad-
for multiple-choice questions, responses are already catego- justed residual significant if the absolute value is above 1.96
rized. Thus, we need to code only responses from open- (Adj.residual > 1.96), as suggested by Agresti [42].
ended questions. Such questions help us to gain a finer
The adjusted residuals drive our analysis on relation-
understanding on how each topic is implemented in start-
ships between start-up demographics and reported engi-
ups. Since there is no established body of knowledge of soft-
neering practices. However, due to the exploratory nature of
ware engineering in start-ups, we use open, in-vivo, coding
our study, we do not state any hypotheses upfront and drive
to gain insights how respondents themselves perceive and
our analysis with the research questions. We document
use software engineering. Thus, we applied open coding
statistically significant findings with field memos for further
to identify described stakeholders, practices, artifacts used
analysis.
or produced, steps taken, and motivations for certain deci-
sions [38]. Each open-ended question addresses a different
3.2.4 Development of patterns
topic, thus we developed an individual coding scheme for
each question. To develop our results further, we revise and group together
The questionnaire is formulated to capture both used similar memos from the previous step to formulate broader
practices, and the practitioners’ experiences with using the findings. This process is aimed at building up evidence
practices. Respondent reflections were captured in separate for supporting a particular finding. As suggested by Eisen-
questions formulated along the lines of: “In hindsight, what hardt [34], we apply the following practices:
would you have done differently?”. In our results, we report • We analyze outlying, extreme, or otherwise surprising
and describe applied practices and respondents’ experiences findings to shape our findings.
with the practices separately. However, the progression • We collect multiple variables on each topic and seek
model in Fig. 4 contains only practices that were reported to triangulate our findings with multiple variables, and
as used. cases.
We use the start-up life-cycle model, see Fig. 1, to • We search for cases that present contradictory evidence
group cases by start-up life-cycle state. We analyze start- to our propositions and shape our propositions to cover
ups within each group and look for recurring engineering the negative evidence.
practices, challenges, goals, and contextual factors in their • To decide if our findings constitute a salient pattern
responses. We document these findings with memos. A we use a combination of criteria. Firstly, we look if
memo describes a finding, start-up cases that were basis a finding appears in more than one case. Secondly,
for the finding, category of the finding, that is, whether it we look if multiple variables support the finding, in
pertains goals, practices, context factors or challenges, and particular, whether respondents have pointed it out in
to what state of the start-up life-cycle model the finding their reflections. Thirdly, if the finding is supported by
pertains to. We developed a total of 1856 codes and 366 statistical analysis.
memos. An example of open coding and memos is available As a result of this step, we identify 55 patterns pertaining
as supplemental material on-line4 . to software engineering in start-ups. Similar to the memos
The initial coding is performed by the first author. The before, a pattern describes a specific practice, challenge,
resulting memos are discussed and refined by the first context factor, or goal, along with information what cases
4. Example of the open coding: were the basis for formulating this pattern, and information
http://eriksklotins.lv/files/GCP open coding.pdf in what context this pattern was observed. The patterns are
AuthorizedCopyright
licensed(c)use
2019limited
deother purposes,
Downloaded must
May be 02,2020
This
publication
is the author's
in a future
versionissue
of an
of article
this journal,
that has
butbeen
has published
not been fully
in this
edited.
journal.
Content
Changes
maywere
change
made
prior
to this
to final
version
publication.
by the publisher
Citation prior
information:
to publication.
DOI 10.1109/TSE.2019.2900213, IEEE
Transactions
is available
on Software
at Engineering
further grouped into 6 engineering areas and categorized Some of the respondents, 36 of 84, 43%, have specified
into 33 patterns illustrating goals, practices, and challenges. that their main area of expertise in their start-up is other
Goals are patterns describing a desirable outcome to- than software engineering, see Fig. 2. There could be a con-
ward which an effort is directed. Such desirable results cern whether they can provide reliable answers to questions
are identified either explicitly by question formulation, e.g. about software engineering. Earlier work suggests that start-
“What is the primary quality goal?”, or by the practitioners’ up teams are closely-knit, team members perform multiple
own reflections on why a certain practice was used. In few roles, and the effort is focused on launching one product [2],
occasions, a goal overlaps with a practice, e.g. to use product [7]. Thus, even if a respondents primary area of expertise
usage metrics to gauge start-up performance, see G16 and is not software engineering, they are closely involved in
P14 in Fig. 4. Such overlap indicates an association between product work.
use of the practice and attainment of the goal. We further explore potential differences in answers due
Practices are engineering activities helping start-ups to to time since the last contact with the start-up and respon-
advance through the life-cycle stages. A practice could be dents area of expertise with statistical tests. As part of the
means-end, such as establishing a team, could help to solve last step of cross-case analysis, see Section 3.2.3, we test if
a problem, e.g. by documenting feature ideas, or help to respondents association with the start-up, age, time since
gather knowledge on the current needs, e.g. by determining the last contact, area of expertise, amount of experience in
“good-enough” level of quality [43]. the area of expertise, and amount of experience working
Challenges are difficulties in practicing software engi- with start-ups, has any effect on their responses. Significant
neering and progressing through the life-cycle stages. We results from this analysis are presented in respective process
identify challenges from respondents reflections on how areas.
practices were applied and what they would do differently There is a possibility that respondents are unwilling to
the next time. Some challenges overlap with practices, e.g. to provide honest responses, or twist the responses to what
establish a feedback loop with customers, see for example, they believe researchers hope to hear. In the particular case
C4 and P5 in Fig. 4. Such overlap indicates an association of startups, and especially when respondents are answering
between the practice and the challenge, i.e. that start-ups at- about a failed case, it may be hard for them to admit were
tempt to use a specific practice, however find it challenging. they failed and if they were not following what is perceived
As a result of this step, we identify patterns pertaining to as the best practices in software development.
software engineering in start-ups. Similar to memos before, Participation in the study was voluntary, and we adver-
a pattern describes a specific practice, challenge, context tised the questionnaire as means to help other start-ups by
factor, or goal, along with information what cases were sharing what worked and what did not in their start-ups.
the basis for formulating this pattern, and information in Thus, we minimize biases stemming from participants being
what context this pattern was observed. The patterns are forced to participate and to provide dishonest answers. The
further grouped into engineering process areas, resulting in questionnaire contains a mix of multiple-choice, Likert scale,
the start-up progression model. and free text providing multiple means of capturing the ex-
periences, and mitigating the chance of extreme responses.
We offer “What would you do differently next time?” ques-
3.3 Threats to validity tions to capture respondents own lessons learned.
Larsson [33] identifies a number of validity concerns for case Interpretive validity, objectivity of the researcher is concerned
survey research. with potential bias stemming from researchers misinterpret-
Descriptive validity, factual accuracy could be compro- ing the data. To address this threat, the first three authors of
mised if responses from start-ups lack important informa- this paper frequently met and discussed any intermediate
tion leading to an incorrect interpretation of the cases. To findings as part of the analysis process. As a result, find-
address this threat we iterated the questionnaire instrument ings with weak support from the data were identified and
with both researchers and start-ups to assure that all im- removed, see Steps 4-6 in Table 1.
portant aspects of software engineering in start-ups are cov- Generalizability of our results is determined by the num-
ered and our questions are understandable to practitioners. ber and diversity of the studied cases. We explore a diverse
Furthermore, we provided an explicit option to capture “I set of 84 start-up cases varying by geographical location, do-
do not know” answers and, at the end of each section, main, product life-cycle stage, the extent of team expertise,
asked “What would you do differently next time?” question and start-up outcome. That said, operational companies are
to capture practitioner reflections on the most important overrepresented in our sample, see Fig. 3, and could bias our
lessons learned. conclusions towards active, that is, to some extent successful
Respondent bias stems from participants’ inability or un- start-ups. To compensate for this potential bias, we analyze
willingness to provide accurate responses. operational and closed companies separately, compare the
We collect respondents experiences and estimates of results, and apply statistical methods to determine signifi-
events that may have occurred in the past. Thus, the quality cance of any conclusions.
of the responses depend greatly on respondents memory Our sample mostly contains start-ups from Europe and
and ability to reconstruct past events. Respondent responses South America, and start-ups from North America and
suggest that majority of them, 67 out of 84, 80%, are cur- Asia are underrepresented. For example, start-ups in the
rently involved with their start-ups or have been involved in U.S. may have access to a broader and more homogeneous
the last 12 months. Thus, the majority of responses concern market than their European counterparts who need to adapt
relatively fresh experiences. their products to the diversity of Europe’s markets. Such
AuthorizedCopyright
licensed(c)use
2019limited
deother purposes,
Downloaded must
May be 02,2020
This
publication
is the author's
in a future
versionissue
of an
of article
this journal,
that has
butbeen
has published
not been fully
in this
edited.
journal.
Content
Changes
maywere
change
made
prior
to this
to final
version
publication.
by the publisher
Citation prior
information:
to publication.
DOI 10.1109/TSE.2019.2900213, IEEE
Transactions
is available
on Software
at Engineering
differences could have an effect on software engineering

10 or more
goals, challenges, and practices. Main area of expertise
years
Non-software engineering
Repeatability of case surveys increases the objectivity of
Software engineering
the findings. To strengthen repeatability of this study, we 7 - 9 years
provide the data extraction form, full demographical infor-
mation of the studied cases, and raw data5 as supplemental 4 - 6 years
material.
1 - 3 years
4 R ESULTS AND ANALYSIS 6 - 12

months
After removing incomplete and irrelevant responses, we
analyze 84 responses from start-ups building software- Less than
6 months
intensive products.
Some of the first questions in the questionnaire col- 40 20 0 0 20 40
lects demographical information about the start-up, such as Working in the area of expertise Working in start-ups
current state, state of the product, when the team started

working on the product. The responses show that our Fig. 2. Distribution of respondent background and prior experience
sample consists of start-ups established between 2004 and
2017, with the majority (60 of 84, 71%) of start-ups actively
Paused Acquired Closed

working on their products at the time of the survey; 24 are
closed, paused or acquired by other companies, see Fig. 3.
Responses on the product state show that our sample
contains start-ups in all life-cycle stages, inception - 15
(18%), stabilization - 24 (29%), growth - 26 (30%), and
maturity - 15 (18%), however 4 companies have not specified
their product state. Most start-ups have their main office
in South America (39 of 84, 46% ) and Europe (33 of 84,
Operational
39%), the rest are located in Asia and North America. Such
underrepresentation of North America and Asia could be
explained by origin of the authors. The authors represent
Europe and South America and were actively promoting
the study in their networks.
2004 2006 2008 2010 2012 2014 2016
With the questionnaire we collect respondents demo-
graphical information, such as age, experience, area of ex-
pertise, relationship with the start-up, and how recent they Fig. 3. Gantt chart illustrating operation time and outcome for studied
have been involved in the start-up. Responses show that companies. 9 respondents had not answered when they started working
on the product, thus these cases are not shown in the figure
most of the respondents (62 of 84, 74%) are founders, others
are employed by start-ups (16 of 84, 19%), or are otherwise
associated. The respondents are about evenly distributed
present the start-up progression model. To maintain trace-
regarding whether their area of expertise is software engi-
ability between our description and the figure we enumerate
neering or not, an how much prior experience they have in
our findings in the following way: goals (Gx), challenges
their area of expertise (left side of Fig. 2). However, most
(Cx) and practices (Px). On the top-right corner of each bar
have little experience with start-ups before the current case
we denote how many cases were basis for formulating the
(right side of Fig. 2).
finding.
Other details characterizing our sample and engineering
The purpose of the progression model is to provide an
context in start-ups are presented along with their respective
overview of the critical stages of product development, main
process areas, discussed in the remainder of this section. A
concerns, and relevant practices. For researchers, the model
list of studied cases, respondents, and their demographical
summarizes the key engineering concerns for further inves-
information is available as supplemental material6 .
tigation and provides a structure for adding new results.
We structure our results into 6 process areas: team,
For practitioners, the model helps to identify the current
requirements engineering, value, quality assurance, archi-
product stage, the key objectives to be attained and lists the
tecture and design, and project management, see Table 2.
key engineering practices for attaining said objectives. We
Within each process area we consider goals, challenges, and
discuss each process area, goal, challenge and practice in
practices, i.e., engineering aspects, see the legend in Fig. 4.
the following sections.
We analyze the process areas in relation to the four start-
To present respondent estimates on Likert scale ques-
up evolution stages, inception, stabilization, growth, and
tions, we illustrate the distribution of answers with in-line
maturity. In Fig. 4 we provide an overview of the results and
histograms, for example, ( , “To what extent it is
true that most product/service features are invented rather than
5. Upon request to the first author, Eriks Klotins, eriks.klotins@bth.se
6. All cases and their demographical information: discovered?”).
http://eriksklotins.lv/files/GCP demographics.pdf The four bars denote the distribution of respondent
AuthorizedCopyright
licensed(c)use
2019limited
deother purposes,
Downloaded must
May be 02,2020
This
publication
is the author's
in a future
versionissue
of an
of article
this journal,
that has
butbeen
has published
not been fully
in this
edited.
journal.
Content
Changes
maywere
change
made
prior
to this
to final
version
publication.
by the publisher
Citation prior
information:
to publication.
DOI 10.1109/TSE.2019.2900213, IEEE
Transactions
is available
on Software
at Engineering
agreement with a statement on a Likert scale (“not at all”, “a product altogether. As one start-up reflected on their lessons
little”, “somewhat”, and “very much’). The above example learned from their team formation:
shows that most respondents “somewhat” agree with a “We should have looked for more developers with
statement (shown after the histogram), and the distribution experience in delivering products fast, and do not rely on
of responses is skewed towards agreeing with the statement. corporate “experts” that deliver anything but completed
software.”
4.1 Team process area
Statistical analysis shows a strong association between
The team process area concerns start-up teams with respect domain knowledge and engineering skills in the team
to formation of a team, individual skills, attitudes, capabili- (Cramer0 s V = 0.513, p < 0.05). Teams with less domain
ties, and coordination between individuals. knowledge also lack engineering knowledge, and teams
with adequate domain knowledge are likely to have suffi-
4.1.1 Goals cient engineering skills. The level of expertise is also associ-
Establishing a team posing sufficient skills and expertise ated with the state of the start-up (Cramer0 sV = 0.316, p <
is one of the first goals of a start-up (G1). Earlier studies 0.05). Operational start-ups estimate their skills and knowl-
suggest that the team is the catalyst for product develop- edge significantly higher than paused or closed companies.
ment in start-ups [7]. Team characteristics such as cohesion, The level of team skills and experience is reported as one
coordination, leadership, and learning are recognized as of the key success factors in software projects [46], and
essential for software project success [44]. new business creation [47]. Thus, there could be a causal
relationship between the level of team skills and start-up
Across our sample, the median team size is 4 - 8 people
outcomes.
and 1 - 3 of them primarily work on software engineering.
From the 41 start-ups in growth or maturity stages, 80%,
We observe a tendency of growing median team size over
or 33 companies, reflect on the need for specialist skills and
the start-up life-cycle, from 1 - 3 in the inception stage, to 4
the challenge to acquire them (C3). For example, finding
- 8 in the stabilization and growth stages, and to more than
employees with domain-specific experience, or engineers
20 in the maturity stage.
with specific technical skills. Often start-ups reflect, that
The responses suggest that to build a team, start-ups
it would have been beneficial to acquire such expert skills
need to address communication issues, shortages of the do-
earlier. However, the responses do not reveal the cause for
main and engineering expertise, commitment issues, and to
the difficulty.
create accountability. Statistical analysis on accountability in
The shortage of experts with domain-specific experience
the teams shows an association (Cramer0 sV = 0.334, p <
could be explained by short supply of experts in a narrow
0.05) between closed companies and a lack of accountability.
and potentially new area. Such explanation highlights the
Thus, forming a unit that poses relevant and complementary
importance of education and training in start-ups. However,
expertise, and that can work together efficiently with little
an alternative explanation could be that experts choose not
overhead is an essential early goal in start-ups, see G1 in
to work with start-ups due to, for example, an uncertain
Fig. 4.
future of the company.
By comparing responses from start-ups at different life-
By analyzing the responses on the team’s attitude to-
cycle stages, we found that establishing a team is a concern
wards good engineering practices and their engineering
in the early stages of a start-up. We observe that start-ups at
skills we found that less skilled teams are less predis-
the inception and stabilization stages reflect on issues associ-
posed towards following good engineering practices such
ated with creating a team, see G1 and C2 in Fig. 4. However,
as avoiding code smells (Cramer0 s V = 0.342, p < 0.05) or
start-ups in the maturity stage reflect on challenges related
thorough testing of their product (Cramer0 sV = 0.366, p <
to managing a large team (C1).
0.05). Further analysis reveals that the attitude towards fol-
lowing good engineering practices is associated with start-
4.1.2 Challenges up outcomes. Operational start-ups recognize benefits from
Looking into respondent reflections on their team formation, following good engineering practices, such as maintaining
we found that the majority, 61 out of 84, 72%, reports good software architecture (Cramer0 s V = 0.400, p < 0.05)
some team-related challenges. The challenges concern team and avoiding code smells (Cramer0 s V = 0.320, p < 0.05).
formation, management, expertise, leadership, and coordi- Paused start-ups report seeing fewer benefits from good
nation, see C1 - C3. engineering practices. This finding suggests an association
Start-ups at all life-cycle stages report shortages in en- between start-up outcomes and attitudes towards utilizing
gineering skills and domain expertise. The most severe good engineering practices.
shortages are reported by start-ups at the inception stage Interestingly, respondents with more individual experi-
where only 3 out of 15, 20%, respondents, estimate their en- ence estimate their team attitudes towards good engineer-
gineering, and 1 out of 15, 6%, rate their domain knowledge ing practices and quality of engineering work significantly
as sufficient. However, we observe a tendency of estimates worse than less experienced respondents (Cramer0 s V =
improving over life-cycle stages. A potential explanation 0.384, p < 0.05). This finding could be interpreted as fol-
could be survivor bias and start-ups who managed to ac- lows: a) inexperienced engineers overestimate the quality
quire the necessary competencies were able to advance [45]. of their and their teams’ work, or b) experienced engineers
We also observe a tendency that less skilled teams spend underestimate their work. Kruger and Dunning [48] have
more time in the inception stage and often fail to release the studied cognitive biases in estimating own abilities. Their
AuthorizedCopyright
licensed(c)use
2019limited
deother purposes,
Downloaded must
May be 02,2020
This
publication
is the author's
in a future
versionissue
of an
of article
this journal,
that has
butbeen
has published
not been fully
in this
edited.
journal.
Content
Changes
maywere
change
made
prior
to this
to final
version
publication.
by the publisher
Citation prior
information:
to publication.
DOI 10.1109/TSE.2019.2900213, IEEE
Transactions
is available
on Software
at Engineering
Legend
Gx Goals direct practitioners attention and effort Px Practices, specific activities, methods, Cx Challenges, barriers, issues experienced
towards relevant objectives, and guide selection of frameworks, and models aiding software by practitioners affecting product
relevant practices. engineering engineering
Start-up life-cycle stages

INCEPTION S TA B I L I Z AT I O N GROWTH M AT U R I T Y
Build and launch the first version of Develop the product further and Aim to achieve desired market share Transition into an established
the product prepare for scaling and growth rate organisation
Engineering process areas
12 cases 14 cases
G1 Build an effective team
C1 Manage a large team
P1 Assemble a small team of capable and motivated individuals
I TEAM
14 cases 33 cases
C3 Need for specialist skills
C2 Lack of team expertise, engagement, coordination, and leadership
P2 Use of external experts for specialist tasks
6 cases
P3 Outsource engineering work
32 cases
G2 Validate invented ideas 35 cases C4 Establish a feedback loop
II R E Q U I R E M E N T S
P4 Consult with customers P5 Customer input drives product work

ENGINEERING
6 cases 19 cases
G3 Balance customer value with time-to-market G4 Support business needs
56 cases 7 cases
P6 Scope the MVP C5 Feature creep
57 cases
P7 Manage changing requirements, document feature ideas for further analysis
56 cases
G5 External, customer perceived value with emphasis on functionality and user experience
14 cases
III V A L U E
Internal market potential value

FOCUS
G6
Proportions of a full page fig
9 cases
G7 Financial value, revenue
3 cases
G8 Internal, differentiation value
27 cases
G9 Functionality
IV Q U A L I T Y G O A L S
14 cases
G10 Maintainability
TESTING
13 cases 4 cases
G11 Time-to-market G12 Portability
&
23 cases 16 cases
G13 Determine “good-enough” quality level
C6 Manual regression testing takes substantial effort
P8 Elicit and validate quality requirements
25 cases 14 cases
P9 Informal, manual, exploratory testing P10 QA process, tester role
V ARCHITECTURE
28 cases 22 cases
G14 Mitigate technology risks
C7 Technical debt slows down development
DESIGN
P11 Use established technologies, frameworks, and components

&
41 cases
P12 Continuously review and improve user interfaces
7 cases 34 cases
Lack of a performance G15 Gauge product performance in the market
MANAGEMENT
C8
benchmark
VI P R O J E C T
P13 Use external product usage metrics
4 cases G16 Gauge internal efficiency 14 cases

C9
Manage external
stakeholder dependencies P14 Use internal team performance metrics
34 cases 14 cases
Maturing project
P15 Minimal control over schedule and resources P16
management process
Fig. 4. The start-up progression model outlining software engineering goals, challenges, and practices in start-ups
AuthorizedCopyright
licensed(c)use
2019limited
deother purposes,
Downloaded must
May be 02,2020
This
publication
is the author's
in a future
versionissue
of an
of article
this journal,
that has
butbeen
has published
not been fully
in this
edited.
journal.
Content
Changes
maywere
change
made
prior
to this
to final
version
publication.
by the publisher
Citation prior
information:
to publication.
DOI 10.1109/TSE.2019.2900213, IEEE
Transactions
is available
on Software
at Engineering
findings suggest that low-ability engineers cannot objec- A potential pitfall is to select the technical product
tively evaluate their actual competence or incompetence, leaders based their executive powers and not professional
and are likely to overestimate their abilities. competences. This could be the case when, for example, a
We have also found that early start-ups do not imple- less competent founder refuses to give away the CTO role
ment any objective metrics to assess their team performance, to a more suited employee [6].
see section 4.6. Therefore, implementation, evaluation, and An association between good teamwork practices and
improvement of engineering practices depend on compe- software project cost and quality is studied in the context
tence and gut-feeling of early start-up engineers. of established companies [51]. Higher team capabilities re-
Individual skills, competencies and teamwork capabil- garding technical skills and domain knowledge, are strongly
ities have been recognized as essential success factors in associated with a lower number of discovered defects and
software engineering projects [25], [46], [49]. Earlier studies lower software maintenance costs. Moreover, commitment
suggest that start-ups rely on implicit knowledge, and have to a shared goal and internal team communication mecha-
very little, if any, organizational capital [7], [50]. Thus, estab- nisms are essential for project success. Our results indicate
lishing a small and efficient team is essential to compensate that such findings are relevant in the start-up context as
for the lack of organization. Our results show that early team well.
issues could be a reason for stalling product engineering and We looked into how the respondents’ relationship with
collapse of a company even before the product is launched the start-up affects their responses. We found that founders
to market. of start-ups are significantly more optimistic about the qual-
Another challenge reported by 67%, 56 of 84 companies, ity of planning (Cramer0 s V = 0.381, p < 0.05), and quality
is to engage the team and coordinate the product work. The of product engineering (Cramer0 s V = 0.385, p < 0.05),
difficulty stems from team members having other priorities compared to hired engineers and external contractors. Such
outside the start-up in combination with a bloated, poorly results suggests a potential fault-line in the teams between
organized, and distributed team. The reported issues in such founders and employees in terms of how they perceive the
teams are poor communication, lack of clear responsibilities engineering context.
and unbalanced skill set with further effects on productivity, As shown by Chow [46], joint decision making and
degrading motivation, and poor execution of multiple paral- knowledge sharing, critical to project success, depend on
lel activities, e.g., product engineering work and marketing, efficient communication. However, fault-lines splitting a
among other challenges. team into two or more sub-groups, result in impaired com-
From all team related concerns we distill 2 main chal- munication and further adverse effects from stemming from
lenges. The first challenge pertains team building and communication issues. Causes and effects of team fault-
comprises of a lack of team expertise, engagement, and lines are observed and studied in the context of globally
coordination at the inception and stabilization stages, see distributed teams, for example Gopal, et al. [52] and Staats
C2. The second challenge, see C1, is to manage a large et al. [53].
and potentially distributed team at the maturity phase. Earlier studies suggest that start-ups have small, flat
Respondents mention difficulties to coordinate and main- and empowered teams. Empowerment of individuals is
tain efficient teamwork across multiple teams and time- supposed to reduce the need for bureaucracy and improve
zone. Such findings suggests that start-up principals need flexibility [2]. However, our results suggest that even though
to recognize and adapt for different teamwork challenges as teams are small, there still exists a communication gap
the company moves forward. between founders and employees. An explanation could
Part of the challenge to initiate teamwork is the absence be that principal decisions from founders could be ill-
of leadership to establish an engineering team and to drive communicated, thus perceived by employees as unjustified,
the product engineering work. As stated by one start-up: and degrading motivation and trust in a team [54].
“All of the founders had other occupations. 3 pro- 4.1.3 Practices

fessors (2 of them located in the USA), and one busi- We looked at practitioner responses to identify how team
ness owner located in Turkey. There was a lack of formation challenges are addressed in their start-ups. We
communication. The developers were hired part-time. recognize two general scenarios how start-up teams are
There was no-one for whom this start-up was their formed (P1). One scenario is creating a new team from peo-
primary occupation. Maybe hiring a manager would ple without previous joint experience. The second scenario
have helped.” is that a team originates from a former organization and
already has some teamwork experience. Not every start-up
Looking more into leadership we found 4 cases explicitly
could have the opportunity to reuse an existing team, thus
pointing out the need for technical leadership. A technical
we consider these both strategies as variations of the same
leader, or CTO, is needed to set up an engineering team
practice to establish a team.
and to lead product engineering work. As one respondent,
The responses suggest that newly created teams start
employed by a start-up, stated:
with a few founders and new people are added when there
“I would fire the current CTO and hire a new CTO, is a need for additional skills or human resources. Such orga-
who is more technically sound. In the startup, the most nizations have yet to establish teamwork practices, acquire
prominent thing is that the CTO should be technically domain knowledge, and vet their engineering capabilities.
sound. Otherwise, it is hard to drive the product forward Thus, new teams are prone to teamwork issues, shortages of
and to motivate the development team.” skills and expertise.
AuthorizedCopyright
licensed(c)use
2019limited
deother purposes,
Downloaded must
May be 02,2020
This
publication
is the author's
in a future
versionissue
of an
of article
this journal,
that has
butbeen
has published
not been fully
in this
edited.
journal.
Content
Changes
maywere
change
made
prior
to this
to final
version
publication.
by the publisher
Citation prior
information:
to publication.
DOI 10.1109/TSE.2019.2900213, IEEE
Transactions
is available
on Software
at Engineering
Teams originating from earlier projects are slightly larger efforts, and encourage loyalty to the team [55].
and have more diverse competencies (compared to new Suggested reading: Bubshait et al. [56] presents critical
teams), shared history concerning established ways of work- concepts influencing team performance and lists build-
ing, roles, responsibilities, and had already ironed out initial ing blocks for establishing highly performing teams.
team formation issues. While we do not know the exact rela- 2) Engineering and domain expertise are important for
tionships between team members in all cases, 9 respondents efficient teamwork. Engineering expertise is essential
mentioned that their teams have been working on earlier to build the product fast. However, domain under-
projects suggesting that they have shared experience before standing helps to identify and interpret software re-
the current project. An example of this scenario would be quirements [57]. Start-up teams are often formed by
a small consultancy company, i.e., offering customized ser- inexperienced people or work in new domains, thus
vices, that identifies an opportunity to develop a product for knowledge sharing and mutual learning is essential.
mass-market. They keep the consultancy business going to Suggested reading: Eppler et al. [58] presents processes,
support the start-up endeavor, eventually aiming to become tools, and factors for enabling team knowledge man-
a product organization. agement. Cockburn and Highsmith [59] explore the role
Making use of an existing team helps to alleviate initial of individual competences in agile team performance.
team formation challenges, see C2 in Fig. 4, and minimize
the risk of the team breaking apart. Moreover, an existing
team likely has experience in relevant product engineering 4.2 Requirements engineering process area
technologies, markets, and the product domain. Therefore, The requirements engineering process area concerns the
such teams have an advantage over recently formed teams elicitation, analysis, validation, documentation and scoping
with no shared background and experience. Similar re- of software requirements. The identification and validation
sults, pointing out that start-up founders’ earlier experience of a relevant product idea, that is requirements identifica-
shapes their skills and attitudes helping to cope with un- tion, validation, and scoping, are some of the most impor-
certainty are reported by Politis [47]. However our results tant activities in start-ups [29].
show that skills and expertise of the team as a whole plays
an important role. 4.2.1 Goals
Start-ups report different tactics to address the lack of
One of the first steps in any software project is to identify
engineering competences. As an alternative to establishing
needs and constraints placed on a software product [25].
an own engineering team, 3 start-ups mention outsourcing
Respondents from start-ups at the inception stage agree
product development work to another company (P3). The
that product features are to a large extent invented, and
motivation for outsourcing is to quickly build the first
are based on founders experience, and understanding about
version of the product without the effort of creating an own
the domain ( , “To what extent it is true that most
team. However, outsourcing the engineering work comes
product/service features are invented rather than discovered?”).
with challenges to negotiate requirements and communicate
efficiently. As one respondent stated: As a consequence of invented requirements, one of the
first objectives in a start-up is to break down invented ideas
“We started the development offshore with an exter- into software requirements and to validate these require-
nal company, now development is in-house. With fewer ments, optimally with inputs from target customers (G2).
people, we have the same cost, but we at least tripled the As one practitioner stated:
productivity.”
“The real purpose of requirements elicitation is
Start-ups at all life-cycle stages mention the use of exter- to invalidate requirement ideas as quickly as possible
nal consultants to help with specialist tasks (P2) in addition without unnecessary effort on meta documentation or
to their engineering team. For example, 6 start-ups mention implementation.”
that they have used external user interface specialists to
help with product design. Some mention using external Responses show that as soon as the product is launched
developers for mobile application development, security, to market, start-ups put more emphasis on using their
and optimization related tasks. customers as requirement sources. Requirements invention
becomes less common. Responses, to what extent prod-
uct requirements are invented, shifts towards disagreement
4.1.4 Lessons learned
when start-ups mature. ( , “To what extent it is
Our results support earlier findings that team is the catalyst true that most product/service features are invented rather than
for product engineering [7]. Team issues could hinder start- discovered?”). Thus, the goal to quickly invalidate own ideas
up potential to advance through the life-cycle stages. is most relevant at the inception stage and before start-ups
For start-ups, our findings present several implications: have established a feedback loop and could use customer
1) Team formation is an essential early activity. To attain a input for feature ideas, see C4 and P5 in Fig. 4.
highly performing team, a team building program must Requirements validation is mainly about what function-
be implemented, focusing on establishing respect for ality and quality to offer. However, it is essential to differen-
everyone in the organization, identify and communi- tiate between the validation of the idea itself (requirements)
cate individual performance standards, develop ways and how it is provided (design) [60]. The differentiation is
of efficient communication, identify clear individual significant. A great idea can be received poorly if the design
and group goals, reward teamwork, and team-building of it, regarding how the solution is offered, is bad.
AuthorizedCopyright
licensed(c)use
2019limited
deother purposes,
Downloaded must
May be 02,2020
This
publication
is the author's
in a future
versionissue
of an
of article
this journal,
that has
butbeen
has published
not been fully
in this
edited.
journal.
Content
Changes
maywere
change
made
prior
to this
to final
version
publication.
by the publisher
Citation prior
information:
to publication.
DOI 10.1109/TSE.2019.2900213, IEEE
Transactions
is available
on Software
at Engineering
Results from established companies working on market- “We informally asked our friends and family about
driven products are similar. Mature companies, alike start- existing solutions, then we brainstormed with the infor-
ups invent or discover requirements indirectly through anal- mation collected about how to design the product and
ysis and observations. Established companies face similar what features it should have. We never used a formal
challenges to validate requirements before a product is method to elicit requirements, we just informally decided
launched. This is compensated by internal requirements the features our product should have.”
analysis and frequent releases [61], [62].
To achieve the goal of the inception stage and to release The importance of stakeholder involvement in new
the first version of the product, see Fig. 1, start-ups must product development as a success factor has been pointed
determine the scope of the minimum viable product (MVP). out by nearly every study on requirements engineering,
The MVP is a trade-off between features, quality, time, for example, Chow [63], Karlsson [64], Hoyer [65], and
and cost [30] used to gauge market interest in the product Blank [31]. However, as our results show, finding the right
and to establish an early customer base justifying further sample of customers and convincing them to invest time in
investments in the product. collaboration is challenging.
When asked how the MVP was scoped and what quality Establishing relationships with potential customers is
attributes were considered necessary, the majority of respon- a goal of sales activities as well. A recent study on new
dents, 56 out of 84, 67%, pointed out that the priority is product development from the sales perspective highlights
to maximize customer value through functionality, usabil- the overlap between requirements engineering and sales
ity, and user experience, while keeping engineering effort processes, pointing out that getting to know customers is
minimal, see P6. Practice of scoping the MVP is related to essential for both technical and commercial outcomes of
the goal of balancing customer value with minimal develop- the project. Involving both engineering and sales roles in
ment effort (G3). As one respondent reflects the MVP should establishing customer contacts helps to ensure continuity
be scoped by the current needs, and not by wishful thinking: between requirements engineering and commercial relation-
ships [66].
“Any requirement that is essential for validating the Some, 7 of 24, 30%, start-ups at the stabilization stage
growth or value hypotheses is priority. Anything that is reflect that they had believed that adding more features to
to support the business when the user base is larger than the product would improve their chances of success. That
100 users is not a priority. Scoping is an exercise of un- constitutes feature creep, see C5, and stems from difficulties
prioritizing features from being added to the MVP.” to elicit useful feedback from customers, and generalizing
Responses on release scoping goals from start-ups in customer specific requirements. Start-ups report that feature
stabilization, growth and maturity stages suggest a shift in creep drained financial resources and added extra complex-
scoping goals compared to the inception stage. The release ity to the product. As one company reflected on their lessons
scoping goals of inception and stabilization stages are to learned:
maximize customer value and to validate the product idea, “We should have done usability tests with real
however later goals shift to supporting business goals, such customers with a prototype of the product and iterate
as monetization and growth (G4). We present related results faster based on user feedback and actual usage, not just
on value focus and product quality goals in Sections 4.3 guessed what customers might want.”
and 4.4.
Feature creep is reported as a general challenge in devel-
4.2.2 Challenges oping new software products. It is caused by adding new
Comparing responses from active and closed start-ups at requirements late in the product development without ap-
stabilization and growth stages we observe that internal propriate analysis. The new requirements could stem from
sources, such as brainstorming and invention of require- external stakeholders, thus appear very appealing to include
ments, are the most popular requirement sources and used in a release, or could be discovered and self-approved by
by 94% of active, and 71% of closed start-ups. the team during the development process. Either way, a
Other requirement sources are similar products, used by company should analyze the impact and assure that higher
83% active and by 43% closed start-ups, and market trends, business goals are not compromised [67].
used by 57% active and by 29% closed start-ups. However,
only 43% of closed start-ups have used input from potential 4.2.3 Practices
and existing customers compared to 91% of active start-ups. Responses suggest that start-ups use a mix of requirements
This difference is statistically significant (Cramer0 s V = sources. Across all cases, internal sources such as require-
0.463, p < 0.05). ments invention and brainstorming are the most reported
Looking at the free text responses for an explanation, by 76 out of 84, 90%, start-ups, followed by potential and
we found that all closed start-ups, and 10 out of 35, 29% of existing customers (66 cases, 79%), and analysis of similar
the active start-ups had difficulties establishing contact with products (59 cases, 70%). Market trends and business goals
their potential customers and involving them in the product are less utilized, standards, laws, and regulations are used
work (C4, P5). The common shortcomings are involving by start-ups in regulated domains, such as medicine.
customers too late in product work, for example, only after Respondents suggest that they have used their previous
release to market, and sampling of potential customers for experience in the domain, both professional and from using
requirements elicitation and validation. As one respondent similar products, to identify “obvious” requirements and
reflected: requirements sources. However, customers are the most
AuthorizedCopyright
licensed(c)use
2019limited
deother purposes,
Downloaded must
May be 02,2020
This
publication
is the author's
in a future
versionissue
of an
of article
this journal,
that has
butbeen
has published
not been fully
in this
edited.
journal.
Content
Changes
maywere
change
made
prior
to this
to final
version
publication.
by the publisher
Citation prior
information:
to publication.
DOI 10.1109/TSE.2019.2900213, IEEE
Transactions
is available
on Software
at Engineering
valuable source of requirements, as stated by one respon- “Yes, requirements change! We happily abandon
dent: the obsolete requirements and remove any code from
“The most important source is existing and potential the product if there is any. New requirements are re-
customers where we have continuous dialog in place. prioritized with a goal to validate our growth and value
This is somewhat self evident. Understanding of compe- hypotheses.”
tition, business models and regulations is secondary to From our sample, only 7%, 6 start-ups, have explicitly
that. ” stated that they do not document their requirements in any
Elicitation, validation: Start-ups report using observation way. The most used specification format is informal notes
(64%) and interviews (61%) to elicit requirements from and drawings, reported by 56%, 47 respondents, others
customers. On-site customer and surveys are less used, 32% report more formal techniques, such as the use of templates
and 46% respectively. Prototypes and mock-ups are used by and formal specifications, for documenting requirements.
61% of respondents to support brainstorming and to elicit Most commonly, start-ups document requirements on a
feedback in customer interviews. feature level (43%, 35 cases), and a function/action level
Responses suggest that elicitation is triggered by internal (21%, 18 cases). Requirements are written down as ideas
ideas that are further elaborated and iterated with input which are later elaborated (P7). As one respondent reflected
from customers. As one company reflected: on their practice:
“Elicitation is a cyclical process of getting info from “We used an on-line tool (Trello) to write all the
customers/products, analyzing and then brainstorming features we wanted to implement and, at the same time,
to feed into prototypes and mock-ups. Early in the to organize their development. Trello cards served as the
project, this was at a very high level. Later when we requirements to be implemented. There were no steps
concentrated development on specific features, the steps in documenting requirements per-see, we just talked
were repeated in a more detailed manner.” informally about what to implement and then wrote it
down in order to not forget it.”
Requirements elicitation overlaps with requirements
validation (P4). Or, as one respondent put it: “the With further statistical analysis, we found that the un-
elicitation techniques are used to IN-validate require- derstandability of requirements is associated with the state
ments/ideas/features”. The most frequently reported val- of a company (Cramer0 s V = 0.319, p < 0.05). Significantly
idation technique is internal reviews used by 55%, 46 start- more closed companies, compared to active ones, report
ups, followed by prototype demonstrations to customers that even though requirements were written down, they
reported by 49%, 41 start-ups. A/B tests to measure cus- were difficult to understand and use in practice. Teams
tomer reaction on new features are used by 25%, 21 start- with insufficient domain knowledge are also more likely
ups. Use of A/B testing increases over the start-up life-cycle not to document their requirements. However, teams with
from 6% at inception stage to 34% at the maturity stage, po- adequate domain knowledge are more likely to use tem-
tentially because A/B tests and other data-driven methods plates for documenting their requirements (Cramer0 s V =
require a significant number of customers interacting with 0.345, p < 0.05). These findings suggest that more rigorous
the product and such numbers may not be available at early requirements documentation could be a way to acquire,
stages [68]. document and distribute critical domain knowledge in the
By looking into associations between difficulties in re- team. Alternatively, teams with better skills and domain
quirements elicitation and demographical information we knowledge see the upside of documenting requirements.
found that younger respondents, 25 - 34 years old, report Either way, we observe an association between understand-
more difficulties in collecting and prioritizing requirements ability of requirements, improved domain knowledge, and
than older, 35 - 44 years old, respondents (Cramer0 s V = progression of a start-up.
0.599, p < 0.05). An explanation for this finding could Requirements documentation enables to create a plan,
be that older age is associated with a broader network of outlining what features to implement when, i.e., develop a
personal contacts and more extensive domain experience product road-map. The responses suggest that 46 start-ups,
supporting identification of relevant ideas for product fea- 55%, have such a road-map. From start-ups at inception,
tures. stabilization and growth stages, about half, 40 - 50% of
Similar results are reported by Azoulay et al. [69] sug- the star-ups, report having a road-map. However, product
gesting that with older age comes more experience, busi- road-maps are used by nearly all, 87%, of mature start-ups.
ness acumen, broader social network, and greater access to Between closed and active start-ups, 67% of operational, 40
financial resources, thus older founders are more likely to cases, have road-maps. However, only 2 or 18% of closed
succeed commercially. start-ups had road-maps. Respondents reflections suggest
Start-ups in all life-cycle stages report that requirements that road-maps are used as planning documents internally
are changing and they have an informal process to manage and often synced with primary stakeholders. Parts of a road-
changes (P7). The most common source of changes is input map that concerns near future are more detailed, however
from customers. Responses suggest that start-ups aim to long-term plans are described at a higher, milestone level.
work in short iterations and frequently re-prioritize their As suggested by the start-up life-cycle model, the first
backlogs. Thus, requirements changes do not have any sig- significant milestone for start-ups is to release the minimum
nificant adverse effects. As one respondent described their viable product, see Fig 1. Responses suggest many strategies
change management process: for scoping the MVP (P6), such as using their domain
AuthorizedCopyright
licensed(c)use
2019limited
deother purposes,
Downloaded must
May be 02,2020
This
publication
is the author's
in a future
versionissue
of an
of article
this journal,
that has
butbeen
has published
not been fully
in this
edited.
journal.
Content
Changes
maywere
change
made
prior
to this
to final
version
publication.
by the publisher
Citation prior
information:
to publication.
DOI 10.1109/TSE.2019.2900213, IEEE
Transactions
is available
on Software
at Engineering
knowledge, input from potential customers and partners, requirements engineering practices and state of a start-up.
soft launch by releasing the product to one customer at the We present the following implications for practitioners:
time, time-boxing, and “gut-feeling”. With the MVP, start- 1) Starting to collect input from potential customers as
ups aim to deliver the essential features in a shortest time early as possible is the key to identifying the most
possible. As one practitioner described the MVP scoping relevant requirements. By involving customers in new
process: product engineering, companies can achieve a higher
“We carefully removed any feature or function not degree of efficiency, ensure product fit with customer
essential to testing growth and value hypotheses. Of needs, and achieve higher customer engagement and
which two, the value hypotheses is prioritized.” satisfaction [65]. Moreover, early customer relationships
are a basis for sales activities when the product is
Value is specified as the primary prioritization goal by launched.
85%, 71 of 84, of respondents. Implementation time is the Suggested reading: Cui et al. [72] compiles earlier work
secondary goal reported by 38%, 32 cases. Requirements from marketing and innovation literature and presents
prioritization is done by consulting customers and other practical guidelines on how to involve customers and
stakeholders. We observe that at the inception and stabiliza- use their knowledge in developing innovative prod-
tion stages, start-ups consider customer needs, while at the ucts.
growth and maturity stages, business requirements, such as Hoyer et al. [65] present a framework for value co-
a need for revenue and growth, are discussed as well. As creation in product development comprising of moti-
one practitioner described their prioritization process: vators, outcomes, and potential impediments.
“To prioritize we use this question: how much money 2) Feature creep can be managed by requirements analysis
or new customers we will have if we implement the new focusing on how many customers will find a feature
feature?” useful. Only features that concern the majority of the
customers should be included in the roadmap.
In their reflections, nearly all respondents from both Suggested reading: Elliot [67] proposes to use the Pareto
active and closed start-ups, suggest that they should have principle in deciding whether a feature is part of the
spent more time with customers to understand and analyze core product or is customer specific. He argues that
their needs better. As one respondent describes their lessons about 20% of features are used by 80% of customers,
learned: and the key to avoiding scope creep is to pinpoint the
“In the start we let innovations lead our goals too 20%. Also, the paper lists several strategies for handling
much. Then we got our first customers and did not listen feature creep.
to them much. We did not track in a enough detailed way 3) Writing down requirement ideas, their source and ra-
what the customer is exactly doing with our product. tionale can help to acquire, maintain and distribute do-
We let just one dedicated person to be in contact with main knowledge in the team. For example, even a basic
certain customers and did not share details in the team. requirements specification helps to establish a common
Even a small company needs to setup a program to make vocabulary and avoid delays and time-consuming in-
interaction with customers transparent and actionable.” teractions caused by confusion and misunderstandings
between stakeholders [73].
Comparing our results on requirements engineering in Suggested reading: Hadar et al. [57] explore the role
start-ups with results from established market-driven com- of domain knowledge in requirements elicitation. They
panies we observe many similarities. For example, in both analyze both positive and negative effects of prior
contexts requirements are primarily invented, used practices domain knowledge in elicitation interviews. Domain
are light-weight and informal, and focused around quick knowledge helps to ask more focused questions, pro-
releases to elicit customer feedback [61], [62]. However, we vides a common language, and saves time in learning
observe a difference in prioritization practices. Established the basics.
companies rely more on effort estimates in prioritizing 4) Release scoping, especially scoping of the minimum
requirements, while start-ups use value as the primary viable product, should be done carefully and optimally
prioritization target [70]. with clear goals and input from all stakeholders. Our
The discrepancy in prioritization goals could be ex- results from start-ups are similar to results from es-
plained by a lack of unified and quantifiable view on value. tablished companies suggesting that facing uncertainty
Thus, established companies opt for scalable and straight- companies are likely to overscope their product re-
forward prioritization criteria [71]. Moreover, established leases [74]. Suggested reading:
companies are likely to operate within a set budget and Suggested reading: Bjarnason et al. [74] presents a root-
schedule constraints further motivating the need to adhere cause analysis and effects of release overscoping in
to effort estimates. However, start-ups are more customer- software projects.
centric, flexible, and work on a smaller number of features,
thus can use value as prioritization target [4].
4.3 Value focus
4.2.4 Lessons learned The software value concept characterizes the broader aims
Our results support earlier findings that requirements en- of a company and aligns all activities in an organization
gineering is one of the key engineering activities in start- towards defined value goals [75]. Respondent responses
ups [10]. Moreover, our results show an association between often mention value as a criterion for prioritizing require-
AuthorizedCopyright
licensed(c)use
2019limited
deother purposes,
Downloaded must
May be 02,2020
This
publication
is the author's
in a future
versionissue
of an
of article
this journal,
that has
butbeen
has published
not been fully
in this
edited.
journal.
Content
Changes
maywere
change
made
prior
to this
to final
version
publication.
by the publisher
Citation prior
information:
to publication.
DOI 10.1109/TSE.2019.2900213, IEEE
Transactions
is available
on Software
at Engineering
ments and scoping product releases. However, a value is ranking of value is similar, customer value being the most
a vague term and could mean different things to different important, followed by internal value and financial aspects,
stakeholders. To explore how start-ups interpret the phrase we observe differences in the definition of these values. For
we map their definitions of value to software value aspect example, established companies interpret customer value as
taxonomy [71]. We summarize these different views as en- delivery time and perceived quality. However, none of the
gineering goals. start-ups in our sample have mentioned time-to-market or
product quality in their value definitions. The internal value
4.3.1 Goals in established companies is understood as internal product
Across the sample, the dominant view on value is the quality, technical debt, supporting tools and processes. Start-
customer perspective, reported by 48% or 40 start-ups. Re- ups, in turn, focus on market potential and differentiation
spondents define customer value as perceived benefits, re- value.
garding functionality, user experience, and hedonistic value, Differences in the studied samples can explain these
derived from the product, and potential for the company to discrepancies in value focus. Established companies could
capitalize this value, see G5-G8. As one respondent phrased see more value in consistently delivering quality features to
it: existing customers and keeping technical debt low. How-
ever, start-ups aim to identify an unmet customer need in a
“With value we understand the benefit for a cus- high-potential market [1], [22].
tomer to use our product. Higher value strengthens
our position in the market compared to competitors and 4.3.2 Lessons learned
helps to increase the price of the product.” Our results show shifting views on value definition among
start-ups in different life-cycle stages. However, there is a
The second most reported interpretation of value stems
shortage of similar results (except for Alahyari et al. [76]) for
from the internal business perspective, i.e., how the com-
comparison and deeper analysis. Nevertheless, we present
pany estimates product value from their own, internal
the following implications for practitioners and gaps for
perspective (23% or 20 start-ups). Respondents define this
further investigation:
perspective as both market potential, e.g., how many cus-
tomers could benefit from a particular feature (G6), and • Understanding of value focus can help practitioners
differentiation value, e.g., how a feature will help to stand to the scope and align their engineering and business
out in the market (G8). As one respondent put it: activities to maximize specific types of value. Align-
ment of activities helps to ensure that technology ac-
“Value is something that can be used to mar- tually supports specific business goals and deliver the
ket/distinguish the product from competition.” intended value to stakeholders [77].
• Our results show that at least two value perspectives
Financial value concerning revenue is the third most re-
are relevant at any life-cycle stage. The value focus
ported interpretation of value, by 11% or 9 respondents (G7).
shifts from customer value and internal market value
However financial value is often defined in combination
at inception stage to financial and differentiation value
with other value perspectives, such as customer or internal
at the maturity stage, see Fig. 4. Understanding of
value. As one respondent described their multi-faceted view
the multi-faceted nature of value can help to facilitate
on value:
communication between stakeholders [71].
“Value is what could give more revenue to the Suggested reading: Khurum et al. [71] present a taxon-
company and make the product easier to use for users omy of software value aspects establishing a common
and motivate them to purchase, so that we could have vocabulary and understanding on different value as-
more transactions each day.” pects.
Carlson and Wilmot [77] present a practical guide on
We compare value definitions between start-ups at dif- how to identify, use and develop an understanding of
ferent life-cycle stages and observe several tendencies. Cus- the value and use value to align organizational efforts
tomer value pertaining perceived benefits of the product of in building innovative products.
is the dominant value perspective in all life-cycle stages,
reported by 23 - 46% of respondents. Internal value captur-
ing product’s market potential value, for example, ability 4.4 Quality goals and testing process area
to serve more customers efficiently and to access broader This process area concerns product quality goals and prac-
markets, is reported by 20% of respondents at inception and tices to attain these goals. Product quality is a mix of
growth stages, 12% at the growth stage, and not reported functionalities, non-functional attributes, and broader con-
at all by start-ups at the maturity stage. Financial value of straints determining commercial success of a product (e.g.,
the product is not reported at all by companies at inception cost of product development vs. returns from marketing
stage, however it grows over start-up life-cycle and peaks at the product). We aim to explore what aspects of software
the maturity stage where 20% of start-ups have reported it. quality are considered significant by practitioners and what
Internal, differentiation value are more reported by start-ups practices are used to attain such aims.
at inception and maturity stages, and appear less relevant at
stabilization and growth stages. 4.4.1 Goals
A similar analysis conducted in established product Respondent answers suggest that product functionality is
companies shows somewhat different results [76]. While the the most common quality goal (G9) across all life-cycle
AuthorizedCopyright
licensed(c)use
2019limited
deother purposes,
Downloaded must
May be 02,2020
This
publication
is the author's
in a future
versionissue
of an
of article
this journal,
that has
butbeen
has published
not been fully
in this
edited.
journal.
Content
Changes
maywere
change
made
prior
to this
to final
version
publication.
by the publisher
Citation prior
information:
to publication.
DOI 10.1109/TSE.2019.2900213, IEEE
Transactions
is available
on Software
at Engineering
stages, reported by 32%, 27 respondents. Time-to-market is Alternative methods, often used in parallel, are exploratory
the second most common objective (G11), indicated by 15%, testing, or scenario-based testing, reported by 21% and 19%
13 start-ups. Even though time-to-market is not a product start-ups respectively.
quality goal intrinsically, it has a profound influence on the Respondents estimates suggest that test case documen-
product decisions [7], [20]. The frequency of other quality tation varies from informal to systematic without any clear
goals varies across life-cycle stages. tendency ( , “To what extent it is true that test cases
Maintainability is frequently reported quality goal in are not systematically documented?”). Estimates on the test
stabilization, growth and maturity stages (G10). Portabil- case coverage are biased towards agreeing with less than
ity is reported only by start-ups at the maturity stage complete coverage ( , “To what extent it is true that
(G12). Reliability is mentioned by few cases in stabilization test cases do not fully cover the product functionality?”)
and maturity stages. Shifting quality goals indicate suggest We observe little differences in testing practices between
changing priorities and adds support for start-up life-cycle start-ups in different life-cycle stages. However, start-ups
model, see Section 2.4. at growth and maturity stages reflect that more test au-
Looking further into how the required level of product tomation, more skills regarding software testing, and more
quality was determined we found that start-ups reflect on a systematic testing would have been helpful. An explanation
“good-enough” level of their quality goals (G13). This goal for such results could be that consequences of the informal
is related to scoping of the MVP (P6) and focus on non- testing surface only when a product gains a user base. Two
functional features of the product. Wrongly estimating the start-ups at the maturity stage reflect that a dedicated tester
required quality level (G13, P8) could lead to poor market role, responsible for performing testing tasks, is needed
reception due to less than acceptable quality, or waste by (P10). As stated by one of the respondents:
providing excessive quality [78]. “I think we need more structured testing, a man-
ager of testing that coordinates the efforts, all being
4.4.2 Practices
responsible [for testing the product] is NONE being
Start-ups report using different strategies, based on in-house responsible.”
expertise, user feedback, and iterative development (P8) to
set their quality targets. Answers to questions on the use of automated testing
One strategy is not to set any specific quality targets and show that a third, 26 out of 84, 31%, of start-ups are attempt-
iteratively identify, and improve relevant quality aspects. As ing to implement automated testing, and only 17 start-ups,
stated by one company: 20%, have explicitly stated that no test automation is used.
“We take a continuous iterative refinement approach Responses suggest that regression testing is an increas-
rather than setting a fixed goal, so we periodically assess ing concern over start-up life-cycle (C6). Start-ups at incep-
which areas are in need of improvement, evaluate and tion and maturity stages report that they spend substantial
prioritize the options” effort on manually testing the entire product ( ,
“To what extent it is true that manual testing of the entire prod-
As an alternative, one start-up at the inception stage uct/service is required to make sure that a release is defect free?”).
reported aiming for the simplest solution that can be im- At the same time, respondents report that few defects slip
proved, if needed: through testing and are reported by customers ( ,
“We look for what is the simplest, regarding size and “To what extent it is true that customers often report defects
complexity, solution to this problem. Will that solution that could have been captured earlier?”). Two respondents from
be able to support a couple of hundred users? How many start-ups at the maturity stage have stated explicitly that
different ways can the solution be improved to support they are working to improve and automate their product
more users?” testing.
Start-ups could benefit from more rigorous testing prac-
Regnell et al. [78] describe a quality requirements tices. Efficient software testing enables faster product re-
roadmapping model for determining the required quality leases, thus allowing the teams to reduce time-to-market
level. The method proposes to identify relevant quality and to iterate new features faster. More rapid release cycles
metrics and defines useless, useful, competitive, and ex- contribute to faster requirements validation (G2, P5) and are
cessive quality ranges for each. Then, it uses these ranges known to improve customer satisfaction [79].
to compare the own product with competition and to spot Good testing practices and use of automated tests can
opportunities for improvement. Using such a model in a support the on-boarding of new developers. Automated
start-up could help in determining essential qualities, a tests provide a safety net for inexperienced developers to
minimum quality level for a product to be useful, and discover any problems with their code quickly on their
opportunities to differentiate among similar products. This own. Automated test definitions serve as means of docu-
is aligned with the focus of internal value start-ups typically mentation to learn how different components of a product
have (see Section 4.3.2). work [80]. Therefore, having good testing practice and test
Respondent answers suggest that informal manual test- automation have benefits beyond a defect-free software.
ing is the most common practice for making sure that the
product has an acceptable level of quality at all life-cycle 4.4.3 Lessons learned
stages, reported by 33%, 28 start-ups (P9). However, the Our results show that start-ups primarily focus on deliver-
responses suggest that informal testing is gradually replaced ing relevant functionality. Other quality aspects change as
with an organized QA process (P10) at the maturity stage. a start-up advances through the life-cycle. Software defects
AuthorizedCopyright
licensed(c)use
2019limited
deother purposes,
Downloaded must
May be 02,2020
This
publication
is the author's
in a future
versionissue
of an
of article
this journal,
that has
butbeen
has published
not been fully
in this
edited.
journal.
Content
Changes
maywere
change
made
prior
to this
to final
version
publication.
by the publisher
Citation prior
information:
to publication.
DOI 10.1109/TSE.2019.2900213, IEEE
Transactions
is available
on Software
at Engineering
aren’t a concern. However, the internal quality and testing “We don’t use unstable components or new 3rd
practices are important to support the sustainable evolution party services without evaluating them thoroughly and
of the product. We identify the following implications for running a small pilot project.”
practitioners:
For at least one start-up in our sample, poor choice of
• Our results suggest that external quality isn’t a concern
a technology platform hindered quality, delayed product
in start-ups, potentially due to relatively small products
releases, and was the leading cause for discontinuing the
and early adopters being more tolerant towards defects.
product. Therefore, selecting a technology stack is an im-
That said, quality becomes a concern in growth and
portant goal early in the start-up life-cycle.
maturity stages when commercial success of a product
Earlier studies have pointed out that start-ups leverage
depends on quality service.
on cutting-edge technologies to develop innovative prod-
• We observe that maintainability is an increasing con-
ucts and gain competitive advantages [1], [15]. However,
cern over a start-up’s life-cycle. Maintainability helps a
we could not find support for such results in our dataset.
start-up to remain fast in launching new features and
Looking into how organizations make technology deci-
sets a foundation to enable portability of a product.
sions in other contexts, we found that performance regard-
Suggested reading: Regnell et al [78] propose a
ing total time and resources efficiency, maintainability and
lightweight method for quality requirements roadmap-
reliability are the top criteria for software component selec-
ping. The method helps to establish a frame of reference
tion in established companies. Similar results are reported
for assessing current product quality and spotting op-
by Petersen et al. [35], and Badampudi et al. [83]. Start-up
portunities for improvement.
companies could be considering similar criteria and opting
• Respondent answers suggest a lack of test automation
for more stable components to save development time and
and superficial testing practices. However, our earlier
resources. Furthermore, start-ups could be leveraging on
study on technical debt in start-ups could not find
established, well-known technologies to create new and
any significant association between testing debt and
innovative products [84].
product quality or team performance issues [13]. Nev-
ertheless, good testing practices help to have faster
product releases and speed up the on-boarding of new 4.5.2 Challenges
developers. Respondent estimates suggest that technical debt is preva-
Suggested reading: Collins et al. [81] present an ex- lent in start-ups, especially late in the life-cycle, see C7.
perience report on test automation practices in an ag- Nearly all start-ups report some signs of technical debt, such
ile development context. They present several lessons as shortcomings in knowledge distribution, code smells,
learned from implementing test automation and what suboptimal architecture decisions, and lack of automated
types of test automation bring the most benefit. testing. Testing debt, associated with lack of automated
regression testing and a need to manually re-test the entire
product before releases, is the most common type of techni-
4.5 Architecture and design process area cal debt. Documentation, architecture debt, and code smells
The architecture and design process area concerns the in- are also a concern, albeit to a lesser extent.
ternal product structure, selection and use of components, In our earlier study with the same dataset, we found
construction technologies, interfaces and other aspects sup- documentation, architecture debt, and code smells to be
porting the construction of the product. associated with impaired team productivity and product
Earlier studies suggest that start-ups leverage on open- quality. However, we could not find any significant asso-
source components, third-party services and cutting-edge ciation between testing debt, productivity or quality [13].
technologies to construct their products [7]. We aim to Statistical results showed a significant association be-
explore how start-ups choose the components and how they tween start-up team size, team skill level and level of
design their product architectures. technical debt. Larger, less skilled teams are more likely to
The visual appearance of graphical user interfaces in- suffer from consequences of technical debt (Cramer0 s V =
fluences the usability of a product and affects stakeholder 0.386, p < 0.05).
perception about the product. Look and feel of product Comparing reports on the technical debt between start-
user interfaces is known to have an impact on project ups at different life-cycle stages we found a pattern of
success [82]. We explore how start-ups design user interfaces architecture debt growing over time and peaking at the
for their products. growth stage (C7). By looking into respondent reflections,
we found that start-ups at growth and maturity stages face
challenges stemming from earlier architecture decisions. As
4.5.1 Goals stated by one respondent:
We inquired respondents on the use of cutting-edge tech-
“Initially, the product was started by an inexperi-
nologies and found that start-ups aim to minimize risks
enced developer and many design decisions were incor-
from using immature technologies by using well-known,
rect. Fixing them early would have been easy, however
stable technologies in the product development (G14). How-
with the product and user base growing, changing core
ever, 39%, 33 of 84, of the respondents report using or
things became increasingly difficult. Correcting simple
experimenting with some cutting-edge components on the
architecture mistakes that could have been trivial to fix
side. As one start-up reflected:
early, can take weeks now.”
AuthorizedCopyright
licensed(c)use
2019limited
deother purposes,
Downloaded must
May be 02,2020
This
publication
is the author's
in a future
versionissue
of an
of article
this journal,
that has
butbeen
has published
not been fully
in this
edited.
journal.
Content
Changes
maywere
change
made
prior
to this
to final
version
publication.
by the publisher
Citation prior
information:
to publication.
DOI 10.1109/TSE.2019.2900213, IEEE
Transactions
is available
on Software
at Engineering
Our results match earlier findings suggesting that techni- Our results on user interface design are aligned with
cal debt in start-ups is caused by a need to develop a product studies investigating user experience practices in a start-up
fast and an inadequate team skill level [7]. However, our context. Start-ups use a set of practices to iterate prototypes,
respondent answers indicate that external pressures, such collect feedback and continuously improve their user in-
as competition do not cause the need for speed. Instead, terfaces. [85]. Quality criteria for good user interfaces and
the need for fast product development is primarily caused user experience are a clear message, visual attractiveness,
by internal considerations to validate the product idea as intuitiveness, and credibility [28].
quickly as possible with minimal waste.
4.5.3 Practices Our analysis contradicts earlier results that start-ups ex-
Start-ups report leveraging on best practices and established tensively use cutting-edge technologies [4]. Rather, start-
frameworks in creating software architectures (P11), thus ups opt for a stable technology platform with standard
avoiding the effort of inventing complex solutions them- architecture to alleviate technology risks. We formulate the
selves. Nearly all start-ups report using open source or following implications for practitioners:
free to use tools and components. Only few report use • Start-ups mitigate technology risks by selecting stable
of commercial off-the-shelf components. Such a strategy technologies and following best practices associated
reduces the effort of software engineering, potential lock-in with the technologies. Stable technologies and best
effects with specific vendors, and need to reinvent already practices improve team performance and provide a
existing functionality. As stated by one respondent: rigid platform for innovative features.
Further reading: Petersen et al. [35] explore how com-
“We used a standard MVC architecture for the back-
panies make decisions to select between open-source,
end and the web front-end, and the standard architecture
in-house, or COTS components. Although the study is
for iOs/Android. The back-end and web front-end was
performed among established companies, it presents
implemented with Ruby on Rails and we followed its
different strategies and factors influencing the deci-
architecture conventions to guide the design.”
sions, thus providing an overview of the process.
Few start-ups in the maturity stage, 4 out of 15, 27%, Baskerville et al. [5] present a study exploring reactive
report using in-house developed, innovative technologies. engineering practices in start-up companies and argue
For example, highly customized deployment configurations, that software engineering in start-ups mainly focuses
audio and video processing tools, and a custom graph on integration and interoperability of components de-
database, to support their product development. Note that veloped elsewhere.
such innovations in technologies are done late in the start- • Our findings suggest that user interface design is a
up life-cycle to support further evolution of an already significant activity. However, it must be done by con-
established product. tinuously monitoring and tweaking the product.
Graphical user interface design is interrelated with user Suggested reading: Hokkanen [28] presents a frame-
experience design. User interfaces could be the primary work for minimum viable user experience pinpointing
touch point between users and the product, thus largely essential qualities and practices of developing good
contributing to users perception of the product. In our study, user interfaces.
we explore on how are the product user interface designs
created without inferring a connection to a broader practice 4.6 Project management
of user experience design.
Respondent responses on user interface (UI) design The project management process area concerns planning
practices suggest that start-ups use mock-ups, utilize de- and control of engineering activities in a start-up. Planning
sign frameworks and borrow ideas from other products and control are known as important to optimize resource
to develop their product UI (P12). Three start-ups report usage and attainment of specific goals [86]. We aim to
outsourcing the UI design work. Companies reflect that explore how start-ups plan and control their activities.
user interface design is an ongoing process of continuous
improvement. Thus, user interface design is not a signifi- 4.6.1 Goals
cant concern at any particular stage. At the inception and The start-up life-cycle model, see Fig. 1, outlines the main
stabilization stages, UI is tested by internal reviews and by milestones, releasing a minimum viable product, stabiliz-
observing users interacting with the UI. At later stages of ing the product for growth, attaining market share, and
the start-up life-cycle when a large number of customers are transitioning into an established organization. We inquired
interacting with the product, start-ups use analytical tools start-ups on how they control and measure their progress
and A/B tests to monitor how customers interact with the towards success.
UI and to identify opportunities for improvement. The responses show that start-ups primarily use exter-
The main reported goals of the UI are usability, 75%, nal metrics, such as revenue, number of customers, and
63 cases, and usefulness, 35%, 29 cases. When reflecting customer satisfaction, to measure their success (G15, P13).
on their user interface development practices, respondents However, start-ups at growth and maturity stages also
suggest that investing more time in customer tests instead of consider using internal metrics, such as team performance,
inventing the UI internally, and having clear user interface adherence to deadlines, and budget plans as performance
guidelines, would have been helpful. measures (G16, P14).
AuthorizedCopyright
licensed(c)use
2019limited
deother purposes,
Downloaded must
May be 02,2020
This
publication
is the author's
in a future
versionissue
of an
of article
this journal,
that has
butbeen
has published
not been fully
in this
edited.
journal.
Content
Changes
maywere
change
made
prior
to this
to final
version
publication.
by the publisher
Citation prior
information:
to publication.
DOI 10.1109/TSE.2019.2900213, IEEE
Transactions
is available
on Software
at Engineering
We observe differences between start-ups in different what extent such practices are used to guide work in early
life-cycle stages. Start-ups at the inception stage, before they start-ups remains to be explored.
have launched their product, do not report the use of any Respondents estimate that their goals are rather clear
metrics to measure their progress. However, they aim to use and change rarely ( , “To what extent it is true
external metrics, e.g., number product users, as soon as the that goals in the start-up are rather unclear and change often?”).
product is launched. At the stabilization stage, just after the As the source for uncertainty respondents report specific
product is launched, the primary success metrics are exter- requirements from customers, demands from partners, and
nal and aimed to monitor general product adoption rates. commercial results (C9). Such factors are out of control for
As start-ups progress through the life-cycle, metrics become start-ups. Thus, any plans should have a room for uncer-
more specific, attached to high-level business milestones, tainty and adjustments.
and consider both internal and external aspects jointly. In- When inquired to what extent respondents experience
ternal team performance metrics are monitored and used to time and financial resources shortages, most responses fall
gauge start-up performance, in addition to external, market between “A little” and “Somewhat”, indicating that avail-
adoption metrics. ability financial resources and time is a concern, however
not to an extreme extent, ( , “To what extent it is
4.6.2 Challenges true that there is a constant time pressure?”) and ( ,
“To what extent it is true that there is a constant resources
Our results show that start-ups aim to use internal and
pressure?”)
external metrics to gauge their progression, see G15-16 in
Statistical analysis shows a linear correlation between
Fig. 4. However, external metrics are not available in the
time and resources shortages indicating that companies
inception stage, before the product is launched.
struggling with finances also struggle with time. However,
Start-ups at the inception stage do not report using any
the free-text answers does not contain any mention of spe-
specific metrics to control their progress. Potentially, due
cific difficulties stemming from time or financial resources
to unclear, changing plans, and immature project manage-
shortages. The responses are consistent across the sample,
ment practices. Thus, start-ups at the inception stage lack
and we could not identify any particular cohort where time
a benchmark to gauge their progression (C8). As stated by
and resource shortages would be more (or less, thereof) of a
one practitioner:
challenge.
“We work towards the goal to improve our plat- Such results contradict earlier studies stating that ex-
form, priorities changed, and but there was no control treme lack of resources is one of the key characteristics of
over schedule. Sometimes tasks took a lot longer than start-ups [1], [4]. The trade-off between project scope and
planned.” budget, and project management are reported as a concern
in nearly every study exploring software project success
In their reflections, practitioners mention a need for factors, for example, Chow et al. [46], Junk [30], Reel [91],
tighter control over product engineering work, deadlines and Nasir [92]. Thus, time and budget constraints, and
and budget. As one practitioner reflected on lessons learned: associated trade-offs are not specific to start-ups exclusively.
“First and foremost, I would have defined short term Moreover, a majority of software projects require more re-
and long term goals of the start-up. That would have sources to complete than initially estimated [93]. Therefore,
helped us to align all resources and activities.” resource shortages alone are not a differentiating factor
between start-ups and established companies. That said,
Results from the statistical analysis show that with the start-ups could differ from established organizations with
advantage of hindsight, practitioners estimate the perfor- thinner margins for resource overruns, less rigorous control
mance of their teams more critically by a significant margin. over resource utilization, and could have more difficulty to
Such findings suggest that it is challenging to assess per- access additional resources [94].
formance from within a company objectively, and objective A study of 1000 Finnish start-ups between 2010 – 2013
early performance indicators could be useful (C8). Objec- reveals that external funding has no association with start-
tive performance indicators could be helpful to establish a up outcome. That is, start-ups with more resources at hand
performing team (C2). do not have more chances of survival and success than start-
Setting goals and metrics to measure the progress to- ups with more limited resources. Moreover, start-ups with-
wards goals is a common practice in project manage- out external funding are likely to generate more revenue in
ment [86], [87]. However, as shown by practitioner re- the long term [95]. Such findings highlight the importance
sponses, start-ups at the inception stage do not measure of scoping, planning and control to optimize utilization of
their progression. Lack of measurement could lead to pitfalls any amount of resources. lack of clear goals
such as the late realization of resource overruns, and scope
creep. 4.6.3 Practices
Looking at the literature, we found several models aimed The responses suggest that start-ups at the inception stage
to guide early start-ups [31], [88], [89]. These models pro- use elementary, if any, practices to schedule their work and
pose first to define and validate the customer need, then keep control over time and budget, see C8. Some start-ups
validate a solution to this need, followed by validating are constrained with a fixed budget or a hard deadline,
product feasibility through small and large-scale prototypes. however control over utilization of time and budget is based
A somewhat similar approach focused on continuous ex- on “gut-feeling”. As one respondent from a start-up at the
perimentation is proposed by Fagerholm [90]. However, to inception stage reflects on their planning practices:
AuthorizedCopyright
licensed(c)use
2019limited
deother purposes,
Downloaded must
May be 02,2020
This
publication
is the author's
in a future
versionissue
of an
of article
this journal,
that has
butbeen
has published
not been fully
in this
edited.
journal.
Content
Changes
maywere
change
made
prior
to this
to final
version
publication.
by the publisher
Citation prior
information:
to publication.
DOI 10.1109/TSE.2019.2900213, IEEE
Transactions
is available
on Software
at Engineering
“We have something on the budget but it’s very latter stages. Such evolution regarding practice maturity is
informal and it’s solely based on projected product rev- also noted by earlier studies, e.g., Giardino et al. [7]. How-
enue, and user retention and acquisition. In hindsight, ever, we present empirical results of specific practices, goals,
we should have done better in terms of scheduling and challenges at each life-cycle stage, and discuss the results
scoping the work.” in the context of related work from similar contexts. This
way we provide actionable support for practicing software
Following from our earlier discussion on control over engineering in start-ups.
resource utilization, lack of control over time and financial Our results show that domain knowledge, technical ex-
resources is a potential pitfall for early start-ups. pertise, and teamwork are the key components in the early
Start-ups at the stabilization stage report improving stages of a start-up. Shortages of any of these components
project management practices, such as setting boundaries can be compensated with specific practices. For example,
for certain expense positions, assigning a budget to meet scarcity of domain knowledge in the team can be compen-
specific goals, and attempting to estimate their cash-flow sated with more rigorous requirements engineering prac-
(P15). Start-ups at maturity stage report using processes, tices helping to identify, acquire, and distribute information
supported by tools, for resource planning and control over in the organization. Similarly, lack of technical expertise in
resource utilization (P16). As one respondent reflects on the core team can be compensated by outsourcing engineer-
their planning practices in a mature start-up: ing work to another team.
“We initially created a budget plan to understand In the later stages, technical debt, growing team and lack
if our business is viable at all and to apply for a of processes (e.g. quality assurance, project management,
loan. Nowadays we use budget estimates as a tool for managing a large, distributed team) hinder engineering
planning.” work. However, these challenges can be anticipated and
addressed with appropriate practices.

Our analysis shows that clear goals, control over resources 5.2 Evolution of software engineering practices in
and schedule are essential to track progress over start-up start-ups
life-cycle. However, there is a challenge to objectively assess By looking at the patterns, we can identify the key concerns
own performance before a product is launched to market. at each life-cycle stage.
Lack of planning and control could contribute to budget At the inception stage, the most important concern is
and schedule overruns leading to wasted opportunities. to assemble a small team of few individuals with suffi-
Lack of resources does not have a substantial effect on cient domain knowledge, technical expertise and adequate
engineering practices and start-ups’ prospects of advancing financial resources to build the first release of a product.
through the life-cycle. However, lack of planning and con- As our results show, different practices can compensate for
trol over resource utilization, especially on early stages of shortcomings in domain knowledge (e.g., by more actively
a start-up is a pitfall. Start-ups need to set clear goals and involving stakeholders and more rigorously documenting
metrics to assess their performance from the very beginning requirements), lack of technical expertise (e.g., by using
objectively. external engineering team), and limited budget (e.g., by
Suggested reading: Dybå et al. [96] present an overview adjusting the scope of the release). Releasing the first version
of agile project management practices and provide practical of a product depends on how efficiently a team can use their
tips on how to organize project management in uncertain understanding of the domain and engineering expertise
and changing environments. to produce software. Excessively large teams and difficult
communication drains resources, time and motivation, and
could lead to the closure of the company even before the
5 D ISCUSSION
launch of a product.
5.1 Reflections on the research questions At the stabilization stage, we observe two main concerns.
Our primary research question explores what engineering The first is related to team building and establishing an
patterns, that is, common goals, practices, challenges, con- efficient team with defined areas of responsibility, trust,
textual factors, can be ascertained in start-up companies. and coordination. After a start-up releases the first product
To identify the patterns, we use the start-up life-cycle version to customers, the company need to balance between
model. The model supported clustering of cases by product developing new features and providing quality service for
life-cycle stage and outcome. Thus, observing how engineer- the existing customers. Aforementioned requires the team to
ing focus changes over start-up life-cycle and enabling a be more diverse, handle a broader range of responsibilities
discussion of what practices contribute to desirable state and more stringent internal structure. The second concern
transitions. Furthermore, with the model and our results, is related to establishing a feedback loop with customers.
we aim to provide a blueprint for studying start-ups as Our results show an association between using customer
dynamic and multi-faceted entities. Such blueprint could feedback and technical and commercial success. Identifying
enable compatibility of results from different studies, and early customers and involving them in the engineering work
contribute towards a coherent view of software engineering is one of the essential tasks in a start-up.
in start-ups. At the growth stage, we observe that concerns related
In most process areas we observe evolving practices, to business, monetization, and marketing become relevant
from rudimentary at the early stages, to more mature in the and have an influence on product engineering. For example,
AuthorizedCopyright
licensed(c)use
2019limited
deother purposes,
Downloaded must
May be 02,2020
This
publication
is the author's
in a future
versionissue
of an
of article
this journal,
that has
butbeen
has published
not been fully
in this
edited.
journal.
Content
Changes
maywere
change
made
prior
to this
to final
version
publication.
by the publisher
Citation prior
information:
to publication.
DOI 10.1109/TSE.2019.2900213, IEEE
Transactions
is available
on Software
at Engineering
companies’ own goals such as the need for monetization R EFERENCES

and expansion, must be taken into account in addition to
[1] S. M. Sutton, E. C. Cubed, and M. Andretti, “The Role of Process
customer needs. Serving a growing number of potentially in a Software Start-up,” IEEE Software, vol. 17, no. 4, pp. 33–39,
diverse customers require the product to be flexible, scal- 2000.
able, and reliable. Providing such qualities require expert [2] N. Paternoster, C. Giardino, M. Unterkalmsteiner, T. Gorschek, and
engineers. Growing complexity of the product creates a P. Abrahamsson, “Software development in startup companies: A
systematic mapping study,” Information and Software Technology,
need for more robust testing and deployment practices, vol. 56, no. 10, pp. 1200–1218, oct 2014.
alleviating a need for manual work to prepare every release. [3] V. Berg, J. Birkeland, A. Nguyen-Duc, I. Pappas, and L. Jaccheri,
Furthermore, increasing complexity of the organization and “Software startup engineering: A systematic mapping study,”
the product expose technical debt which must be addressed Journal of Systems and Software, 2018.
[4] C. Giardino, M. Unterkalmsteiner, N. Paternoster, T. Gorschek, and
to support further growth. P. Abrahamsson, “What Do We Know about Software Develop-
At the maturity, stage start-ups are concerned with ment in Startups?” IEEE Software, vol. 31, no. 5, pp. 28–32, sep
managing a growing, and potentially, distributed team. 2014.
At this point, more rigorous project management practices [5] R. Baskerville, B. Ramesh, L. Levine, J. Pries-Heje, and S. Slaughter,
“Is internet-speed software development different?” IEEE Soft-
are needed to organize and control engineering work, and ware, vol. 20, no. November 2015, pp. 70–77, 2003.
more thorough engineering processes could be introduced [6] M. Crowne, “Why software product startups fail and what to do
marking a transition into an established organization. about it,” in Engineering Management Conference. Cambridge, UK:
IEEE, 2002, pp. 338–343.
[7] G. Carmine, N. Paternoster, M. Unterkalmsteiner, T. Gorschek, and
6 C ONCLUSION P. Abrahamsson, “Software Development in Startup Companies:
The Greenfield Startup Model,” IEEE Transactions on Software
In this paper, we investigated how 84 software start-ups Engineering, vol. X, no. September, p. 233, 2016.
utilize software engineering to build innovative software- [8] Y. Wijngaarde, D. Paliksha, J. Puls, M. Squarci, and I. Anihi-
movskaya, “Annual European Venture Capital Report, 2017,”
intensive products. To frame the results we proposed start- Dealroom.co, Tech. Rep. February, 2018.
up life-cycle model and looked into team, requirements [9] C. Giardino, S. S. Bajwa, and X. Wang, “Key Challenges in Early-
engineering, value focus, quality and testing, architecture Stage Software Startups,” in Agile Processes, in Software Engineering,
and design, and software project management process areas. and Extreme Programming, vol. 212, 2015, pp. 52–63.
[10] J. Melegati, A. Goldman, and S. Paulo, “Requirements Engineering
This framing highlights stage and process area specific in Software Startups : a Grounded Theory approach,” in 2nd Inter-
goals, challenges, and practices. We have discussed our national Workshop on Software Startups, Trondheim, Norway, 2016.
findings in the context of related work and formulated [11] C. Gralha, D. Damian, A. I. T. Wasserman, M. Goulão, and
practical lessons learned aimed at practitioners. J. Araújo, “The evolution of requirements practices in software
startups,” pp. 823–833, 2018.
We conclude that all explored process areas are relevant [12] N. Tripathi, E. Klotins, R. Prikladnicki, M. Oivo, L. B. Pomper-
for start-ups and, in essence, not different from established maier, A. S. Kudakacheril, M. Unterkalmsteiner, K. Liukkunen,
companies. That said, potentially the key difference and and T. Gorschek, “An anatomy of requirements engineering in
software startups using multi-vocal literature and case survey,”
difficulty to practice software engineering in start-ups is to
Journal of Systems and Software, 2018.
manage the evolution of practices in all the process areas at [13] E. Klotins, M. Unterkalmsteiner, T. Gorschek, and R. Prikladinicki,
once with only a little room for error. Even though start-ups “Exploration of Technical Debt in Start-ups,” in International Con-
are experimental by nature and making errors in the process ference of Software Engineering, 2018.
[14] L. Hokkanen and K. Väänänen-Vainio-Mattila, “Ux work in star-
is inevitable, there is a distinction between taking calculated tups: Current practices and future needs,” in Agile Processes
risks, and neglecting the best engineering practices. in Software Engineering and Extreme Programming, C. Lassenius,
To further understand and support software engineering T. Dingsøyr, and M. Paasivaara, Eds. Cham: Springer Interna-
in start-ups we aim to conduct a series of structured work- tional Publishing, 2015, pp. 81–92.
[15] M. Unterkalmsteiner, R. Feldt, and T. Gorschek, “A taxonomy
shops to conduct an assessment of software engineering for requirements engineering and software test alignment,” ACM
practices in start-ups using results obtained in this study. Transactions on . . . , vol. V, no. 212, pp. 1–39, 2014.
With the results of the workshops, we aim to refine further [16] M. Unterkalmsteiner, P. Abrahamsson, X. Wang, A. Nguyen-Duc,
the patterns and a methodology for using the patterns in S. Shah, S. S. Bajwa, G. H. Baltes, K. Conboy, E. Cullina, D. Den-
nehy et al., “Software startups–a research agenda,” e-Informatica
improving engineering practices in start-ups. Software Engineering Journal, vol. 10, no. 1, 2016.
[17] E. Klotins, M. Unterkalmsteiner, and T. Gorschek, “Software-
intensive product engineering in start-ups: a taxonomy,” IEEE
7 ACKNOWLEDGMENTS Software, vol. 35, no. 4, 2018.
The authors of this paper would like to thank all practition- [18] K. Petersen, D. Badampudi, S. Shah, K. Wnuk, T. Gorschek,
E. Papatheocharous, J. Axelsson, S. Sentilles, I. Crnkovic,
ers who found time and motivation to share their experi- and A. Cicchetti, “Choosing Component Origins for Software
ences. Reaching this diverse population of start-ups would Intensive Systems: In-house, COTS, OSS or Outsourcing?
not be possible without help and support from Software – A Case Survey,” IEEE Transactions on Software Engineering,
vol. 44, no. 2, pp. 237–261, 2017. [Online]. Available: http:
Start-up Research Network7 community, and specifically //ieeexplore.ieee.org/document/7870688/
Nana Assyne, Anh Nguyen Duc, Ronald Jabangwe, Jorge [19] N. Churchill and V. Lewis, “Five Stages of Small Business
Melegati, Bajwa Sohaib Shahid, Xiaofeng Wang, Rafael Ma- Growth,” Harvard Business Review, vol. 61, no. 3, pp. 30–40, 1983.
tone Chanin, and Pekka Abrahamsson. [20] E. Carmel, “Rapid development in software package startups,” in
Proc. 27th Hawaii Int’l Conf. System Sciences, 1994.
Work of Rafael Prikladnicki is partially funded by CNpq [21] Atomico/SLUSH, “The State of European Tech 2017,”
and FAPERGS (process 17/2551-0001205-4). Atomico, Tech. Rep., 2017. [Online]. Available: https:
//2017.stateofeuropeantech.com/
7. The Software Start-up Research Network, [22] S. Blank, “Embrace failure to start up success.” Nature, vol. 477,
https://softwarestartups.org/ no. 7363, p. 133, sep 2011.
AuthorizedCopyright
licensed(c)use
2019limited
deother purposes,
Downloaded must
May be 02,2020
This
publication
is the author's
in a future
versionissue
of an
of article
this journal,
that has
butbeen
has published
not been fully
in this
edited.
journal.
Content
Changes
maywere
change
made
prior
to this
to final
version
publication.
by the publisher
Citation prior
information:
to publication.
DOI 10.1109/TSE.2019.2900213, IEEE
Transactions
is available
on Software
at Engineering
[23] E. Klotins, M. Unterkalmsteiner, and T. Gorschek, “Software En- [49] C. Wohlin, D. Šmite, and N. B. Moe, “A general theory of soft-
gineering Anti-patterns in start-ups,” In review by IEEE Software, ware engineering: Balancing human, social and organizational
2017. capitals,” Journal of Systems and Software, vol. 109, pp. 229–242,
[24] ——, “Software Engineering Knowledge Areas in Startup Compa- 2015.
nies : a mapping study,” in Lecture Notes in Business Information [50] P. Seppänen, K. Liukkunen, and M. Oivo, “Little big team: Acquir-
Processing. Springer, 2015, pp. 245–257. ing human capital in software startups,” in International Conference
[25] IEEE, Guide to the Software Engineering Body of Knowledge Version on Product-Focused Software Process Improvement. Springer, 2017,
3.0 (SWEBOK Guide V3.0), 2014. pp. 280–296.
[26] A. Yau and C. Murphy, “Is a Rigorous Agile Methodology the Best [51] M. S. Krishnan, “The role of team factors in software cost and
Development Strategy for Small Scale Tech Startups?” University quality: An empirical analysis,” Information Technology & People,
of Pennsylvania Department of Computer and Information Sci- vol. 11, no. 1, pp. 20–35, 1998.
ence, Tech. Rep., 2013. [52] A. Gopal, J. A. Espinosa, S. Gosain, and D. P. Darcy, “Coordination
[27] D. Leonard-Barton, “Wellsprings of knowledge: Building and sus- and performance in global software service delivery: The vendor’s
taining the sources of innovation,” 1995. perspective,” IEEE Transactions on Engineering Management, vol. 58,
[28] L. Hokkanen, K. Kuusinen, and K. Väänänen, “Minimum viable no. 4, pp. 772–785, 2011.
user experience: A framework for supporting product design in [53] B. R. Staats, K. L. Milkman, and C. R. Fox, “The team scaling
startups,” in International Conference on Agile Software Development. fallacy: Underestimating the declining efficiency of larger teams,”
Springer, 2016, pp. 66–78. Organizational Behavior and Human Decision Processes, vol. 118,
[29] S. Blank, “Why the Lean Start Up Changes Everything,” Harvard no. 2, pp. 132–142, 2012.
Business Review, vol. 91, no. 5, p. 64, 2013. [54] K. Boies, J. Fiset, and H. Gill, “Communication and trust are
[30] W. S. Junk, “The Dynamic Balance Between Cost, Schedule, Fea- key: Unlocking the relationship between leadership and team
tures, and Quality in Software Development Projects,” Computer performance and creativity,” The Leadership Quarterly, vol. 26, no. 6,
Science Dept., University of Idaho, vol. SEPM-001, 2000. pp. 1080–1094, 2015.
[31] S. Blank, The four steps to the epiphany. K&S Ranch; 2nd edition, [55] D. D. Tippett and J. F. Peters, “Team building and project man-
2013. agement: How are we doing?” Project Management Institute,
1995.
[32] S. S. Bajwa, X. Wang, A. N. Duc, and P. Abrahamsson, “How do
software startups pivot? empirical results from a multiple case [56] A. A. Bubshait and G. Farooq, “Team building and project suc-
study,” in International Conference of Software Business. Springer, cess,” Cost engineering, vol. 41, no. 7, pp. 34–38, 1999.
2016, pp. 169–176. [57] I. Hadar, P. Soffer, and K. Kenzi, “The role of domain knowledge
[33] R. Larsson, “Case Survey Methodology: Quantitative Analysis of in requirements elicitation via interviews: an exploratory study,”
Patterns Across Case Studies.” Academy of Management Journal, Requirements Engineering, vol. 19, no. 2, pp. 143–159, 2014.
vol. 36, no. 6, pp. 1515–1546, 1993. [58] M. J. Eppler and O. Sukowski, “Managing team knowledge:
core processes, tools and enabling factors,” European Management
[34] K. M. Eisenhardt, “Building Theories from Case Study Research,”
Journal, vol. 18, no. 3, pp. 334–341, 2000.
vol. 14, no. 4, pp. 532–550, 1989.
[59] A. Cockburn and J. Highsmith, “Agile software development, the
[35] K. Petersen, D. Badampudi, S. Shah, K. Wnuk, T. Gorschek,
people factor,” Computer, vol. 34, no. 11, pp. 131–133, 2001.
E. Papatheocharous, J. Axelsson, S. Sentilles, I. Crnkovic, and
[60] A. van Lamsweerde, From System Goals to Software Architecture.
A. Cicchetti, “Choosing component origins for software intensive
Berlin, Heidelberg: Springer Berlin Heidelberg, 2003, pp. 25–43.
systems: In-house, cots, oss or outsourcing?–a case survey,” IEEE
[Online]. Available: https://doi.org/10.1007/978-3-540-39800-4 2
Transactions on Software Engineering, 2017.
[61] Å. G. Dahlstedt, L. Karlsson, A. Persson, J. Natt och Dag, and
[36] P. Runeson, M. Höst, A. Rainer, and B. Regnell, Case study research
B. Regnell, “Market-Driven Requirements Engineering Processes
in software engineering. Hoboken, NJ, USA: John Wiley & Sons,
for Software Products - a Report on Current Practices,” in Interna-
Inc., mar 2012.
tional Workshop on COTS and Product Software, RECOTS 2003, 2003.
[37] D. Kirk and S. G. MacDonell, “Categorising software contexts,”
[62] C. Alves, S. Pereira, and J. Castro, “A study in market-driven
20th Americas Conference on Information Systems, AMCIS 2014, no.
requirements engineering,” Workshop em Engenharia de Requisitos
April, 2014.
WER06, pp. 2–3, 2006.
[38] J. M. Corbin and A. Strauss, “Grounded theory research: Pro- [63] T. Chow and D.-B. Cao, “A survey study of critical success factors
cedures, canons, and evaluative criteria,” Qualitative Sociology, in agile software projects,” Journal of Systems and Software, vol. 81,
vol. 13, no. 1, pp. 3–21, 1990. no. 6, pp. 961–971, jun 2008.
[39] S. J. Haberman, “The analysis of residuals in cross-classified [64] L. Karlsson, Å. G. Dahlstedt, B. Regnell, J. N. och Dag, and
tables,” Biometrics, pp. 205–220, 1973. A. Persson, “Requirements engineering challenges in market-
[40] A. C. Hope, “A simplified monte carlo significance test proce- driven software development–an interview study with practition-
dure,” Journal of the Royal Statistical Society. Series B (Methodolog- ers,” Information and Software technology, vol. 49, no. 6, pp. 588–604,
ical), pp. 582–598, 1968. 2007.
[41] J. Cohan, “Statistical power analysis for the behaviour sciences,” [65] W. D. Hoyer, R. Chandy, M. Dorotic, M. Krafft, and S. S. Singh,
1988. “Consumer cocreation in new product development,” Journal of
[42] A. Agresti, An introduction to categorical data analysis. Wiley New service research, vol. 13, no. 3, pp. 283–296, 2010.
York, 1996, vol. 135. [66] A. La Rocca, P. Moscatelli, A. Perna, and I. Snehota, “Customer
[43] S. Sheppard, A. Colby, K. Macatangay, and W. Sullivan, “What involvement in new product development in b2b: The role of
is engineering practice?” International Journal of Engineering Educa- sales,” Industrial Marketing Management, vol. 58, pp. 45–57, 2016.
tion, vol. 22, no. 3, p. 429, 2007. [67] B. Elliott, “Anything is possible: Managing feature creep in an
[44] T. Dingsøyr, T. E. Fægri, T. Dybå, B. Haugset, and Y. Lindsjørn, innovation rich environment,” in Engineering Management Confer-
“Team performance in software development: research results ence, 2007 IEEE International. IEEE, 2007, pp. 304–307.
versus agile principles,” IEEE Software, vol. 33, no. 4, pp. 106–110, [68] H. H. Olsson and J. Bosch, “From opinions to data-driven software
2016. r&d: A multi-case study on how to close the’open loop’problem,”
[45] M. Shermer, “How the survivor bias distorts reality,” Scientific in Software Engineering and Advanced Applications (SEAA), 2014 40th
American, vol. 1, 2014. EUROMICRO Conference on. IEEE, 2014, pp. 9–16.
[46] T. Chow and D.-B. Cao, “A survey study of critical success factors [69] J. D. K. J. M. P. Azoulay, B. Jones, “Research: The average age of
in agile software projects,” Journal of Systems and Software, vol. 81, a successful startup founder is 45,” Harvard Business Review, Jul
no. 6, pp. 961–971, 2008. 2018.
[47] D. Politis, “Does prior start-up experience matter for en- [70] J. R. F. Dos Santos, A. B. Albuquerque, and P. R. Pinheiro, “Re-
trepreneurs’ learning? a comparison between novice and habitual quirements prioritization in market-driven software: A survey
entrepreneurs,” Journal of small business and Enterprise Development, based on large numbers of stakeholders and requirements,” in
vol. 15, no. 3, pp. 472–489, 2008. Quality of Information and Communications Technology (QUATIC),
[48] J. Kruger and D. Dunning, “Unskilled and unaware of it: How 2016 10th International Conference on the. IEEE, 2016, pp. 67–72.
difficulties in recognizing one’s own incompetence lead to inflated [71] M. Khurum, T. Gorschek, and M. Wilson, “The software value
self-assessments.” pp. 1121–1134, 1999. map—an exhaustive collection of value aspects for the develop-
AuthorizedCopyright
licensed(c)use
2019limited
deother purposes,
Downloaded must
May be 02,2020
This
publication
is the author's
in a future
versionissue
of an
of article
this journal,
that has
butbeen
has published
not been fully
in this
edited.
journal.
Content
Changes
maywere
change
made
prior
to this
to final
version
publication.
by the publisher
Citation prior
information:
to publication.
DOI 10.1109/TSE.2019.2900213, IEEE
Transactions
is available
on Software
at Engineering
ment of software intensive products,” JOURNAL OF SOFTWARE- [96] T. Dybå, T. Dingsøyr, and N. B. Moe, “Agile project management,”
EVOLUTION AND PROCESS, no. July 2010, pp. 481–491, 2012. in Software project management in a changing world. Springer, 2014,
[72] A. S. Cui and F. Wu, “Utilizing customer knowledge in innovation: pp. 277–300.
antecedents and impact of customer involvement on new product
performance,” Journal of the academy of marketing science, vol. 44,
no. 4, pp. 516–538, 2016.
[73] J. Buchman and C. H. Ekadharmawan, “Barriers to sharing do-
main knowledge in software development practice in smes,” in
Proceedings of the 3rd international workshop on knowledge collabora- Eriks Klotins is a Ph.D. student of Software
tion in software development (KCSD2009). Citeseer, 2009, pp. 2–16. Engineering at Blekinge Institute of Technology
[74] E. Bjarnason, K. Wnuk, and B. Regnell, “Overscoping: Reasons (BTH). He received his Master’s degree from
and consequences—a case study on decision making in software University of Latvia in 2011. He has over nine
product management,” in Software Product Management (IWSPM), years of experience in software engineering in
2010 Fourth International Workshop on. IEEE, 2010, pp. 30–39. both large companies and start-ups. His re-
[75] B. Boehm, “Value-based software engineering: reinventing,” SIG- search is focused on software start-ups and bet-
SOFT Softw. Eng. Notes, vol. 28, no. 2, pp. 3–, 2003. ter engineering practices for start-ups. Specifi-
[76] H. Alahyari, R. B. Svensson, and T. Gorschek, “A study of value in cally, what engineering practices are best suited
agile software development organizations,” Journal of Systems and for start-ups to support their evolution, and com-
Software, vol. 125, pp. 271–288, 2017. mercial and technical success.
[77] C. R. Carlson and W. W. Wilmot, Innovation: The five disciplines for
creating what customers want. Crown Business, 2006.
[78] B. Regnell, R. B. Svensson, and T. Olsson, “Supporting roadmap-
ping of quality requirements,” IEEE Software, vol. 25, pp. 42–47,
2008. Dr. Michael Unterkalmsteiner is a senior lec-
[79] L. Chen, “Continuous delivery: Huge benefits, but challenges too,” turer in the Blekinge Institute of Technology’s
IEEE Software, vol. 32, no. 2, pp. 50–54, 2015. Software Engineering Research Laboratory. His
[80] R. Pham, S. Kiesling, L. Singer, and K. Schneider, “Onboarding research interests include data-driven software
inexperienced developers: struggles and perceptions regarding engineering, software measurement and test-
automated testing,” Software Quality Journal, vol. 25, no. 4, pp. ing, process improvement, and requirements en-
1239–1268, 2017. gineering. He primarily conducts empirical re-
[81] E. F. Collins et al., “Software test automation practices in agile search in close collaboration with industry part-
development environment: An industry experience report,” in ners. Michael received a PhD in software engi-
Proceedings of the 7th International Workshop on Automation of Soft- neering from the Blekinge Institute of Technol-
ware Test. IEEE Press, 2012, pp. 57–63. ogy.
[82] P. Ralph and P. Kelly, “The dimensions of software engineering
success,” in Proceedings of the 36th International Conference on Soft-
ware Engineering. ACM, 2014, pp. 24–35.
[83] D. Badampudi, C. Wohlin, and K. Petersen, “Software component
decision-making: In-house, oss, cots or outsourcing-a systematic
literature review,” Journal of Systems and Software, vol. 121, pp. 105– Dr. Panagiota Chatzipetrou is an Assistant Pro-
124, 2016. fessor at the department of Informatics at Örebro
[84] S. Maranville, “Entrepreneurship in the business curriculum,” University School of Business in Örebro, Swe-
Journal of Education for Business, vol. 68, no. 1, pp. 27–31, 1992. den, where she belongs to the Centre for empir-
[85] L. Hokkanen and K. Väänänen-Vainio-Mattila, “Ux work in star- ical research on information systems (CERIS).
tups: current practices and future needs,” in International Confer- She is also part of the Software Research En-
ence on Agile Software Development. Springer, 2015, pp. 81–92. gineering Lab (SERL) at Blekinge Institute of
[86] C. S. Snyder, “A guide to the project management body of knowl- Technology in Karlskrona, Sweden.
edge: Pmbok ( R ) guide,” Project Management Institute: Newtown She received her BSc degree in Informatics,
Square, PA, USA, 2014. MSc in “Informatics and Business Administra-
[87] A. Shahin and M. A. Mahbod, “Prioritization of key performance tion” and Ph.D. in Informatics from the Depart-
indicators,” International Journal of Productivity and Performance ment of Informatics, Aristotle University of Thessaloniki (AUTh), Greece.
Management, vol. 56, no. 3, pp. 226–240, 2007. Her doctoral dissertation has the title: “Statistical methods in information
[88] E. Ries, The Lean Startup: How Today’s Entrepreneurs Use Continuous systems project planning”.
Innovation to Create Radically Successful Businesses, crown busi ed., As a researcher, she mainly focuses on empirical studies under the
2011. different perspectives of software development. Her research interests
[89] J. Bosch, H. Olsson, J. Björk, and J. Ljungblad, “The early stage include applications of statistical methods to quality problems in soft-
software startup development model: A framework for opera- ware engineering and especially to requirements engineering and the
tionalizing lean principles in software startups,” in Lean Enterprise exploitation of human factor and the different views that ultimately de-
Software and Systems, 2013, ch. The Early, pp. 1–15. termine the quality of a software product and the product development.
[90] F. Fagerholm, A. S. Guinea, H. Mäenpää, and J. Münch, “The Also, she has been working with decision support systems for the de-
right model for continuous experimentation,” Journal of Systems velopment of software-intensive systems, large-scale agile (and global)
and Software, vol. 123, pp. 292–305, 2017. software development, and behavioural software engineering.
[91] J. S. Reel, “Critical success factors in software projects,” IEEE
software, vol. 16, no. 3, pp. 18–23, 1999.
[92] M. H. N. Nasir and S. Sahibuddin, “Critical success factors for
software projects: A comparative study,” Scientific research and
essays, vol. 6, no. 10, pp. 2174–2186, 2011. Prof. Dr. Tony Gorschek is a professor of Soft-
[93] K. Molokken and M. Jorgensen, “A review of software surveys on ware Engineering at Blekinge Institute of Tech-
software effort estimation,” in Empirical Software Engineering, 2003. nology (BTH) and part time at Chalmers. He
ISESE 2003. Proceedings. 2003 International Symposium on. IEEE, has over ten years industrial experience as a
2003, pp. 223–230. CTO, senior executive consultant and engineer,
[94] N. Bocken, “Sustainable venture capital–catalyst for sustainable but also as chief architect and product manager.
start-up success?” Journal of Cleaner Production, vol. 108, pp. 647– In addition he has built up five startups in fields
658, 2015. ranging from logistics to Internet based services.
[95] A. Suominen, S. Hyrynsalmi, K. Still, and L. Aarikka-stenroos,
“Software Start-up failure An exploratory study on the impact of
investment,” pp. 55–64, 2017.
AuthorizedCopyright
licensed(c)use
2019limited
deother purposes,
Downloaded must
May be 02,2020
This
publication
is the author's
in a future
versionissue
of an
of article
this journal,
that has
butbeen
has published
not been fully
in this
edited.
journal.
Content
Changes
maywere
change
made
prior
to this
to final
version
publication.
by the publisher
Citation prior
information:
to publication.
DOI 10.1109/TSE.2019.2900213, IEEE
Transactions
is available
on Software
at Engineering
Rafael Prikladnicki is an associate professor at

the School of Technology and the director of the
Science and Technology Park (Tecnopuc) at the
Pontifical Catholic University of Rio Grande do
Sul, Brazil, where he also leads the MuNDDoS
research group. He is the chair of IEEE Soft-
ware’s advisory board.
Nirnaya Tripathi is a doctoral student at M3S

research unit, University of Oulu since 2014. He
received his Master’s degree in Information Pro-
cessing Science at University of Oulu in 2012.
Prior to that, he received his bachelors degree
in Information Technology at Manipal Institute
of Technology in 2009. His research interest
includes entrepreneurship, startup ecosystem,
software engineering in startup, large-scale lean
and agile software development and global soft-
ware development. He is currently doing re-
search on startup ecosystem and requirement engineering in software
startups.
Leandro Bento Pompermaier is an assistant

professor at the School of Technology at the
Pontifical Catholic University of Rio Grande do
Sul (PUCRS) - Brazil. In addition, he is the
leader of the startup area of the PUCRS Science
and Technology Park (TECNOPUC). He holds
a master degree in computer science from the
Federal University of Rio Grande do Sul and is
currently a PhD student at PUCRS. He is also
an entrepreneur and angel-investor in some IT
companies and software startups.
AuthorizedCopyright
licensed(c)use
2019limited
deother purposes,
Downloaded must
May be 02,2020

A Progression Model of Software Engineering

Загружено:

Сведения о документе

Оригинальное название

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

A Progression Model of Software Engineering

Загружено:

Авторское право:

Доступные форматы

This article has been accepted for

A progression model of software engineering

Index Terms—software start-up, software engineering practices, progression model

I Inception II Stabilisation III Growth IV Maturity

Acquired Acquired Acquired

Active Active Active Active

Note that one question in the questionnaire may result in TABLE 2

differences could have an effect on software engineering

4 R ESULTS AND ANALYSIS 6 - 12

current state, state of the product, when the team started

Paused Acquired Closed

Start-up life-cycle stages

P4 Consult with customers P5 Customer input drives product work

Internal market potential value

P11 Use established technologies, frameworks, and components

P13 Use external product usage metrics

4 cases G16 Gauge internal eﬃciency 14 cases

“All of the founders had other occupations. 3 pro- 4.1.3 Practices

4.6.4 Lessons learned

companies’ own goals such as the need for monetization R EFERENCES

Rafael Prikladnicki is an associate professor at

Nirnaya Tripathi is a doctoral student at M3S

Leandro Bento Pompermaier is an assistant

Вам также может понравиться