The Journal of Systems and Software 86 (2013) 872889

Contents lists available at SciVerse ScienceDirect

The Journal of Systems and Software

journal homepage: www.elsevier.com/locate/jss

25 years of software engineering in Brazil: Beyond an insiders view

Paulo Anselmo da Mota Silveira Neto a,c,f, , Jos Sousa Gomes b , Eduardo Santana de Almeida b,c,e ,
Jair Cavalcanti Leite d , Thais Vasconcelos Batista d , Larissa Leite d
a
Federal University of Pernambuco (UFPE) Recife, Pernambuco, Brazil
b
Federal University of Bahia (UFBA) Salvador, Bahia, Brazil
c
Reuse in Software Engineering (RiSE), Brazil
d
Federal University of Rio Grande do Norte (UFRN), Brazil
e
Fraunhofer Project Center (FPC) for Software and Systems Engineering
a
Universidade Federal Rural de Pernambuco (UFRPE), Brazil

a r t i c l e i n f o a b s t r a c t

Article history: The software engineering area is facing a growing number of challenges due to the continuing increase
Received 20 January 2012 in software size and complexity. The challenges are addressed by the very relevant and high quality
Received in revised form 17 October 2012 publications of the Brazilian Symposium on Software Engineering (SBES), in the past 25 editions. This
Accepted 18 October 2012
article summarizes the ndings from two different mapping studies about these 25 SBES editions. It
Available online 27 October 2012
also reports the results of an expert opinion survey with the most important Brazilian researchers in
the software engineering (SE) area. The survey reinforces the ndings of the mapping studies. It also
Keywords:
provides guidance for future research. In addition, the studies report several ndings that conrmed
Mapping study
Expert opinion survey
the validity of the research methods applied. All of these ndings are important input to the current
Software engineering Brazilian SE scenario. Our ndings also suggest that greater attention should be given to the SE area,
by improving researchers interaction with industry and increasing collaboration between researchers,
especially internationally.
2012 Elsevier Inc. All rights reserved.

1. Introduction empirical evidences can contribute for a better understanding of

In the last decades with the growth of software complexity
and size, sometimes increasing beyond being manageable, differ-
ent conferences related to software engineering (SE) were created
around the world in order to address such challenges. These con-
ferences have focused on specic issues such as requirements
engineering, software architecture, software testing, software evo-
lution and so on, as well as, general SE topics. The International
Conference in Software Engineering (ICSE) and the European
Software Engineering Conference (ESEC/FSE) are important confer-
ences in this area. In Brazil it could not be different and in 1987 the
Brazilian Symposium on Software Engineering (SBES) was created
to promote and discuss the state of the art and practice.
Promoted by the Brazilian Computer Society (SBC), the event
typically attracts more than 500 people, including students, profes-
sors, researchers and members from the industry. Its main objects
of study are processes, methods, techniques and tools for software
development, maintenance and evolution.
In this context, this study presents a set of ndings regarding
to the past, present and future of the SE area in Brazil. The set of
Corresponding author at: Federal University of Pernambuco (UFPE) Recife,
Pernambuco, Brazil.
E-mail address: pauloadmsn@gmail.com (Paulo.A.da.M. Silveira Neto).
the area by presenting its main authors, universities, research top-
ics, as well as, problems and deciencies to be further addressed.
The research design of this study is composed of two different
steps. First, two mapping studies (Gomes et al., 2011; Leite et al.,
2011) were performed, focusing on the SBES publications and retro-
spectives, which aimed to analyze their relevance to its community
and industry, as well as, to identify the type of articles, the research
topics, and so on. Both mappings were normalized, combined and
one more year was included in their analysis. Next, based on these
two studies, a survey involving some of the most important Brazil-
ian experts was conducted in order to investigate the SE area in
Brazil. It combined ideas from Kitchenham and Peeger (2008) with
good practices from Fink (2003), providing an expert perspective
regarding to the SE area.
The results show interesting ndings, problems and challenges
regarding the Brazilian SE community, such as: difcult interaction
with companies, the bureaucracy and inefciency of the funding
agencies, high student turnover to industry, pressure for high pro-
ductivity, and problem with evaluation based on some referees.
In addition, other ndings related to the symposium evolution
regarding to number of submissions, article per universities, top
ten authors can be found in this article.
This article is organized as following: Section 2 describes the
related work in the area. Section 3 presents the research method-
ology applied in the study. Section 4 discusses the mapping studies

0164-1212/$ see front matter 2012 Elsevier Inc. All rights reserved.
http://dx.doi.org/10.1016/j.jss.2012.10.041
 Paulo.A.da.M. Silveira Neto et al. / The Journal of Systems and Software 86 (2013) 872889
design describing the research questions, as well as, the data col-
lection, analysis, and validity procedures. Section 5 summarizes the
expert survey results. Finally, Section 6 presents the summary of
the work and future directions.
2. Related work
The state of SE research from different points of view is dis-
cussed in Glass et al. (2002). In this study, the authors address
different topics, such as: the most investigated SE topics, meth-
ods and approaches; and the levels of analysis in which the SE
researchers conduct research. This way, they analyzed 369 papers
from six leading research journals in the SE eld: Information and
Software Technology (IST), Journal of System and Software (JSS),
Software Practice and Experience (SPE), IEEE Software, ACM Trans-
actions on Software Engineering and Methodology (TOSEM), and
IEEE Transactions on Software Engineering (TSE).
Zannier et al. (2006) empirically analyzed the papers published
at ICSE, presenting quantitative and qualitative results of a quasi-
random experiment of empirical evaluations over the lifetime of
the conference. They randomly sampled ICSE publications across
the 29 years not including invited talks, panels, workshops and
tutorials in the population. The studies were examined to deter-
mine rst if it contained an empirical evaluation component and
next to determine if the empirical evaluation was sound. Next, they
were classied according to the study type and sampling type and
show the improvements or decline of these results over the lifetime
of ICSE.
Davis et al. (2007) presented the results of a demographic analy-
sis by date, type, outlet, author, and author afliation for an existing
database of over 4000 requirements engineering publications. It
aimed to provide an objective overview of the advances and current
state of the requirements engineering discipline.
Montesi and Lago (2008) classied and described the arti-
cle types published in SE emphasizing on what is required for
publication in journals and conference proceedings. They also pro-
posed a classication of different SE article types, which helps the
reader through the SE literature, and makes the researcher reects
on directions for improvements. A recent keynote Ghezzi (2009)
was presented in the 31st International Conference on Software
Engineering (ICSE) by Carlo Ghezzi, where he was interested in
describing the reections on 40+years of SE research. In this work,
some interesting questions were dened such as: What did we pro-
duce?, How can their success (impact) be dened?, How can it be
measured?, What can we learn from the past?, Can we do better? and
Will the paradigms followed in the past survive in the future?. In order
to reect an answer for the questions, all conference editions were
analyzed from 1976 to 2009 in different perspectives. The publi-
cations were analyzed regarding to if it was performed in industry
or academia, the number of accepted papers per year, geographi-
cal distribution and different research topics (Testing and Analysis,
Design and Software Architecture, Specication and Verication,
Tools and SDEs, Management and Cost, Empirical Studies, Existing
Systems and others).
Although our study instantiated some of the ideas from the
related work in order to use a well establish classication or data
collection procedure, we also used the same classication as used
in Ghezzi (2009) to compare the SBES and ICSE conferences.
In order to present the main ndings and the state-of-the-
art regarding to SE in Brazil, two mapping studies (Gomes et al.,
2011; Leite et al., 2011) were performed combining ideas from
two other works (Kitchenham and Charters, 2007; Petersen et al.,
2008), which improve the reliability and allow the repeatability of
the study, when compared with the previous ones. Those studies
presented how the Brazilian Symposium on Software Engineering
873
Fig. 1. Multi-method approach.
(SBES) has being evolved regarding to: research topic, study type,
geographical distribution, analyze their relevance to its community
and to industry.
3. Research methodology
A multi-method approach was used in order to understand the
past, present and future of the SE area in Brazil. Firstly, two mapping
studies (Gomes et al., 2011; Leite et al., 2011) were performed to
gure out how the area is evolving and if it suffers inuences from
international research. It mapped out 25 years of SBES to synthesize
available evidences to suggest important implications for practice,
as well as, to identify research trends, open issues, and areas for
improvements. After that, an expert opinion survey was applied
with some important Brazilian SE researchers to conrm the previ-
ous ndings and also provide new insights. Fig. 1 summarizes the
overall research method applied.
3.1. Mapping study
The experimental SE community is working toward the deni-
tion of standard and reliable processes for conducting systematic
literature reviews (SLR) (MacDonell et al., 2010; Kitchenham et al.,
2011; Zhang et al., 2011; Cruzes and Dyb, 2011). In this study,
we applied the approach dened by Petersen et al. (2008) in order
to map out the Brazilian SE area analyzing the 25 SBES editions.
While systematic reviews (Kitchenham and Charters, 2007) aim
to provide answers to questions from relatively narrow range of
quality-assessed studies, a mapping study is rather focused on
examining the extent, range and nature of research activity, pro-
viding an overview of a specic area (Cruzes and Dyb, 2011).
In our study, we combined mapping study and systematic
review good practices, such as protocol denition, to take advan-
tage of both methodologies. Fig. 2 shows the set of steps applied in
the study.
3.2. Expert opinion survey
The most common research method is the survey based on
expert opinion (Kitchenham and Peeger, 2008) which was also
applied in this study. Although it is a popular data collection
method, it must be used in appropriated conditions (Fink, 2003).
According to Fink (2003), a survey is not just the questionnaire
or checklist for gathering information, it is a research method used
to collect information to describe, compare or explain knowledge,
attitude and behaviors. While expert opinions are a series of sci-
entic endeavors, which are employed to interpret data, predict
systems behaviors, and assess uncertainties (Li and Smidts, 2003).
In our study, the methods were combined in order to take advan-
tage from both.
4. Analyzing the SBES history: a mapping study
In order to understand how the software engineering area is evolv-
ing in the Brazilian scenario, a mapping study (Budgen et al., 2008)
was performed considering the 25 editions of the SBES symposium.
Based on the mapping results, we were able to gure out what are
the main investigated areas in SE and how they are evolving, who
874 Paulo.A.da.M. Silveira Neto et al. / The Journal of Systems and Software 86 (2013) 872889

Fig. 2. Scoping study approach.

Adapted from Petersen et al. (2008)

are the main researchers and companies, that collaborated with
important studies in the area, and nally, what are the main gaps.
In addition to provide the state-of-the-art regarding to the
Brazilian SE area, the study points out some interesting areas which
need to be carefully investigated as well as areas where limited
evaluation was performed. All these results can be useful for both
academic and industry professionals interested in solving their
everyday problems.
Inclusion criteria: Research papers published in the main confer-
ence track.
Exclusion criteria: All communications, posters, tutorials, panels
and tool session papers were not included in our analysis.
The inclusion and exclusion criteria over the 25 SBES
proceedings resulted in 528 studies to be further analyzed and
classied.

4.3. Data extraction

4.1. Research questions
A collection form was designed consisting of a number of
The general questions which guide this mapping study and
attributes for collecting and storing all the required information
reect our goals are: How the software engineering area is evolving in
to address the research questions and categorize them accord-
Brazil?, What are the most investigated research topics and how they
ing to a research type facet. In order to answer the research
are evolving? and How the SBES conference is evolving when compared
questions, we conducted a quantitative analysis of the extracted
with ICSE? In order to answer these questions, a set of perspectives
data.
were considered, as described following:
Regarding data collection, the abstract, introduction and con-
RQ1 How the software engineering area is evolving in
clusion paper sections were read and the following information
Brazil? Rationale: It aims to analyze the SE area, regarding differ-
were extracted from each study and proceedings: authors; univer-
ent point of views, such as: number of national and international
sities and companies; publication year; number of submitted and
authors, number of papers per authors, which are the top uni-
accepted studies; city and state; language; and all the information
versities (public and private) and regions. It is also interesting to
needed to categorize the study using the previously dened facets.
analyze if the SBES studies are being referred by other studies pub-
In addition, the studies references were examined in order to col-
lished in the same conference, and identify some trends regarding
lect papers published in previous SBES editions. This information
to the SE area and how they are evolving when comparing with the
was used to dene the most cited SBES papers.
International Software Engineering area and conference (ICSE).
RQ2 What are the most investigated research topics and
4.4. Classication scheme
how they are evolving? Rationale: It aims to analyze the SBES stud-
ies regarding to the type of research and SE sub-areas. It is also
The key-wording process dened in Petersen et al. (2008) was
important to analyze if there are international inuences over the
used in order to classify the studies according to different points of
Brazilian research, in order to point some similarities and differ-
view. It is used to reduce the time needed in developing the clas-
ences.
sication scheme and ensuring that the scheme takes the existing
RQ3 How the SBES conference is evolving when compared
studies into account. In addition, the classication applied in the
with ICSE? Rationale: It aims to analyze the relationship between
related work studies were also considered in order to establish
international research and the papers published in the SBES sym-
some relationship between ICSE and SBES.
posium, comparing with the ICSE results. This comparison was
All authors were involved during the classication scheme de-
performed using different points of view, such as: the number of
nition, in which the studies were further classied. Three different
accepted papers and ICSE inuences on the topics addressed by
facets were used in the study as following:
SBES. It is also important to analyze how it happens over the years.
RQ4 How the QUALIS CAPES inuences on the SBES submis- Study type: Publishable papers: general, extended versions of con-
sions? Rationale: It analyzes the QUALIS CAPES (Brazilian Federal
ference papers, empirical research reports: (i) observational studies,
Agency for the Improvement of Higher Education) impact on the
(ii) case studies, (iii) eld studies, (iv) experimental papers, (v)
publications since the QUALIS consists in one of the main instru-
meta-analyses; experience papers, theoretical papers, surveys,
ments to evaluate graduate programs in Brazil.
papers oriented toward practice: (i) educational papers, (ii) appli-
RQ5 What is the relationship between the number of pub-
cation papers, opinion papers. (It was adapted from Montesi and
lications written in English and in Portuguese? Rationale: It aims
Lago (2008) and some studies types were not considered in the
to analyze the number of international inuences and diffusion of
classication, since they are reported as exclusion criteria. For
SBES publications.
example, short, tutorial and tool report papers.)
ACM classication: general, requirements and specications,
4.2. Search strategy, data sources and study selection design tools and techniques, coding tools and techniques, soft-
ware and program verication, testing and debugging, programming
A manual search was conducted by two authors in all SBES environments and construction tools, distribution, maintenance,
proceedings in order to collect the studies, examining the stud- and enhancement, metrics and measurement, management, design,
ies title and abstract. The following inclusion and exclusion criteria software architectures, interoperability, reusable software, human
were used: factors in software design, software and system safety, conguration
 Paulo.A.da.M. Silveira Neto et al. / The Journal of Systems and Software 86 (2013) 872889 875

Fig. 3. Accepted, Submitted and acceptance rates (SBES).

management, software construction, software engineering process,

software quality and SQA and miscellaneous. (A more detailed Fig. 4. Accepted and submitted studies on ICSE (Ghezzi, 2009).

description of each classication used in our study is available

at ACM WebSite.1 ) Table 1
Research topics: Testing and Analysis, Design and Software Archi- Number of authors per paper.

tecture, Specication and Verication, Tools and SDEs, Management Number of papers Number of authors
and Costs, Empirical Studies, Existing Systems and Other. These 212 2
research topics were the same addressed by ICSE Keynote pre- 168 3
sentation (Ghezzi, 2009), and it will be useful to compare ICSE 74 4
with SBES. 35 1
19 6
15 5
The ACM classication is commonly used in the SE area to clas- 2 7
sify and identify some specic sub-areas and also to identify the 1 8
main keywords in journals and conference papers. 1 9
1 10

4.5. Outcomes
In this section, each research question is answered from differ-
ent point of views, highlighting evidence gathered from the data
extraction process. All of these results populated the classication
scheme, which evolves while doing the data extraction.
4.5.1. How the software engineering area is evolving in Brazil?
In order to understand the popularity and rigor of the SBES con-
ference over the 25 years, Fig. 3 shows the number of submitted
and accepted papers, as well as, the acceptance rates. Analyzing
this graph, we see that the number of submissions is increasing
over the years, reaching 134 submitted papers in 2006. During more
than one decade (19871999), the average was 72.07, during the
second one, it increased to 106.91 (20002011). Although the sub-
missions are increasing, the average of accepted papers decreased
from 30.61 to 20.41 in the second decade. It indicates the increasing
interest in SBES and at same time the increasing on quality, since
the acceptance rate is still around 21.12%.
In 1994, the number of submitted papers exceeded the average
(89.12). From 2002, the submitted papers are still over the average.
It did not happen with the accepted papers, which remained above
the average since 1992.
On the other hand, if we compare with ICSE data (Ghezzi, 2009),
a huge difference can be noticed, since ICSE recently receives an
average of 350 papers while SBES received 100120, and the SBES
acceptance rate has an average of 21.12%. Based on these numbers,
we can hypothesize that both conferences are in the same way, fol-
lowing the same tendencies, except for the fact that, after 2006, the
number of submitted papers tends to increase over the recent years,
as showed Figs. 3 and 4. It is important to mention that the keynote
(Ghezzi, 2009) did not provide the data related to the submitted
papers before 1996.
Number of authors per publication. Analyzing all the published
papers in the SBES editions, Table 1 shows the number of authors
Table 2
Number of universities per paper.
Number of universities Number of papers
1 342
2 140
3 38
4 7
6 1
per paper. We noticed that about 71% of papers published over
the years have at least two and three authors (about 40.1% and
31.8%, respectively). However, there are published papers with
more authors, reaching a maximum of ten authors in 2011 with
the paper titled IT Project Variables in the Balance: A Bayesian
Approach to Prediction of Support Costs published by Dalmazo
et al. (2011).
Table 2 shows the number of universities per paper. In addition,
no patterns were found regarding to the number of universities per
year.
Number of citations. A deep investigation over the SBES publica-
tions shows the numbers of SBES referred papers over the years.
All papers reference list was analyzed in order to identify papers
that had already been published in SBES. Based on Fig. 5, it possi-
ble to see that SBES papers are serving as good sources to dene

1
http://www.computer.org/portal/web/publications/acmsoftware. Fig. 5. Number of citations per year.
876 Paulo.A.da.M. Silveira Neto et al. / The Journal of Systems and Software 86 (2013) 872889

Table 3
Top 10 papers according to Google Scholar (2012 January).

Year Title Authors Number citations

(January 2012)

2003 On the Reuse and Maintenance of Aspect-Oriented SantAnna, C., Garcia, A., Chavez, C. von F.G., Lucena, 191
Software: An Assessment Framework C.J.P., von Staa, A.
2001 CRE: A Systematic Method fot COTS Components Alves, C., Castro, J.F.B. 96
Selection
2000 Domain Oriented Software Development Oliveira, K., Travassos, G.H., Rocha, A.R.C. 52
Environments (Ambientes de Desenvolvimento de
Software Orientados a Domnio)
1994 Constrained Mutation in C programs Wong, W.E., Maldonado, J.C., Delamaro, M.E., 37
Mathur, A.
2001 Using Objects and Patterns to Implement Domain Guizzardi, G., Falbo, R.A., Filho, J.G.P. 32
Ontologies
1992 Data-driven Strategy in Software Development Nunes, D.J., 31
(Estratgia data-driven no desenvolvimento de
software)
1997 Um modelo Fuzzy para avaliaco da qualidade de Belchior, A.D., G.B.X., Rocha, A.R.C. 30
software
2002 Formalizing Object-Oriented Design Metrics upon Baroni, A., Abreu, F.B.e. 25
The UML Meta-Model
2003 Adapting the NFR Framework to Aspect-Oriented Sousa, G.M.C., Silva, I.G.L., Castro, J.F.B. 24
Requirements Engineering
2005 Taming Heterogeneous Aspects with Crosscutting Chavez, C. von F.G., Garcia, A., Kulesza, U., 24
Interfaces SantAnna, C., Lucena, C.J.P.
2001 An Aspect-Based Approach for Developing Garcia, A., Silva, V.T., Lucena, C.J.P., Milidiu, R. 23
Multi-agent Object-Oriented Systems
2006 Aspectual Connectors: Supporting the Seamless Batista, T.V., Chavez, C. von F.G., Garcia, A., 23
Integration of Aspects and ADLs. SantAnna, C., Kulesza, U., Lucena, C.J.P.

and evolve the already proposed techniques, methods, processes Table 4

Top 11 authors.
and so on. An interest point was that the papers which are being
referred decrease when comparing both decades. From 1987 to Author University Number of
1999, the average was 10.53 and from 2000 to 2011 it was 5.25. publications
In 1993, it achieved the biggest number of citations (18), followed Jos Carlos Maldonado USP 36
by 1997 (15). Even with this difference, no reason was found for Paulo Csar Masiero USP (UFSCar) 27
this phenomenon, and it is a topic to be further investigated. Silvio Romero de Lemos Meira UFPE 25
Carlos J. P. de Lucena PUC/RJ 23
The most referred paper considering the studies published in
Roberto Tom Price UFRGS (UFSC) 22
the 25 SBES editions was an international study titled Evaluation of Julio C.S. do Prado Leite PUC/RJ 20
the Cost of Alternate Mutation Strategies published in 1996. It was Jaelson F.B. de Castro UFPE 18
cited six times and authored by Aditya Mathur and W. Eric Wong, Alessandro Garcia PUC/RJ 15
(Lancaster
both from Purdue University, West Lafayette, IN, USA.
University)
Table 3 shows the number of citations according to Google Arndt von Staa PUC/RJ 15
Scholar2 (2012 January). According to this search 42 studies had Cludia Maria Lima Werner UFRJ 15
at least one citation. Guilherme Horta Travassos UFRJ 15
The most active researchers. From a total of 820 authors who
published papers on SBES editions, Jos Carlos Maldonado from
ICMC-USP is leading the list with 36 published papers, followed
by Paulo Csar Masiero (27) from ICMC-USP, and Silvio Romero de Authors citation. Table 6 shows the number of times in which
Lemos Meira (25) from CIn-UFPE. The list with the top eleven SBES a particular author is cited without taking into account the self-
authors is shown in Table 4. It is important to mention that the data citations. Analyzing this dataset, it is clear that it changed when
show the number of times in which an author is co-authoring and compared with Table 5. Jos Carlos Maldonado (8) loses the lead to
authoring a study. In cases where the author published using two Julio C.S. do Prado Leite with 12 citations, and go into fourth position
different liations both were considered, for example, the author with Antnio F. do Prado (7), W. Eric Wong (8) and Aditya Mathur
Paulo Csar Masiero represent two different universities (USP and (8). Paulo Csar Masiero gets the second position with 11 cita-
UFSCar). tions followed by Arndt von Staa (10). Mrio Jino and Roberto Tom
Analyzing the contributions of each author, Table 5 shows the Price also dropped position and they stayed at sixth and seventh,
number of times in which a specic author was cited, considering respectively.
the 25 SBES editions. Jos Carlos Maldonado is still leading the list Program committee. By analyzing the program committee most
with 29 citations, followed by Julio C.S. do Prado Leite (20), Antnio of the members are from the Brazilian community (62.5%) against
F. do Prado and Paulo Csar Masiero (17). Antnio F. do Prado (37.5%) from international community. It is important to highlight
(17) and Mario Jino (15) (which does not appear in the top eleven that this international participation has been increasing over 25
authors) are one of the most cited in the conference. Table 5 shows years, as shown in Fig. 6. We believe it is important to the SBES
also the number of self-citation and cases in which the author had internationalization and recognition.
publication in a previous university. A deeply investigation in the international presence through the
25 SBES editions, the top 10 countries which collaborates with the
conference can be viewed in Table 7.
From the 134 universities involved in the program committee
2
http://scholar.google.com.br/. 53 are national and 81 are international. It is curious that although
 Paulo.A.da.M. Silveira Neto et al. / The Journal of Systems and Software 86 (2013) 872889 877

Table 5
Top 11 cited authors.

Author University Number of Number of

citations self-citations

Jos Carlos Maldonado USP 29 21

Julio C.S. do Prado Leite PUC/RJ 20 8
Antnio F. do Prado UFSCar 17 10
Paulo Csar Masiero USP (UFSCar) 17 6
Arndt von Staa PUC/RJ 15 5
Roberto Tom Price UFRGS (UFSC) 14 10
Mrio Jino UNICAMP 15 9
Carlos J.P. de Lucena PUC/RJ 11 4
Alessandro Garcia PUC/RJ (Lancaster University) 10 4
W. Eric Wong Purdue University 9 1
Aditya Mathur Purdue University 9 1

Fig. 7. Number of researchers in the program committee per university.

Fig. 6. Evolution of international researchers in the program committee.

A closer investigation on these universities show the following

Table 6
Top 11 cited authors without self-citations.
researchers as the main contributors (see Table 8).
Number of publications per region. The rst two top authors are
Author University Number of citations from the Southeast region, which contributed with 339 studies. It
without self-citations
shows that the most economically developed regions appear with
Julio C.S. do Prado Leite PUC/RJ 12 the big amount of publications. It can be explained by the available
Paulo Csar Masiero USP (UFSCar) 11
economic and technological resources, as well as, the amount of
Arndt von Staa PUC/RJ 10
Jos Carlos Maldonado USP 8 researchers. However, we cannot consider this reason as absolute
Antnio F. do Prado UFSCar 7 truth and more information is required to conclude it. Figs. 8 and 9
Aditya Mathur Purdue University 8 show the distribution of publications over the regions and per
W. Eric Wong Purdue University 8 states, respectively. Two interesting points can be identied. The
Carlos J.P. de Lucena PUC/RJ 7
Mrio Jino UNICAMP 6
number of international studies is larger than two Brazilian regions,
Alessandro Garcia PUC/RJ (Lancaster 6 including the midwest and north. Although the northeast is not
University) one of the most economically developed, it appears in the sec-
Roberto Tom Price UFRGS (UFSC) 4 ond position, since in the last decade, the government has made
explicit incentives to the region providing economic and techno-
logical resources in every request for proposals.
it is a national conference, the number of international universities The SBES has shown a growing interest by international
is bigger than national ones. It can indicate that we need to increase researchers, having the rst international publication in 1992 with
the participation of different universities in Brazil. Considering the four different authors and achieving fourteen in 1996. It is impor-
national environment, Fig. 7 shows the number of occurrences tant to mention that some Brazilian researchers were working on
regarding the researchers per university.

Table 8
Table 7 Top 10 researchers involved in program committee.
Top 10 countries involved in program committee.
Researcher Number of
Country Number of Occurrences occurrences

England 56 Julio C.S. do Prado Leite 21

United States 54 Daltro Jos Nunes 20
Italy 25 Paulo Csar Masiero 20
Portugal 23 Jos Carlos Maldonado 19
Canada 20 Ana Regina Cavalcanti da Rocha 18
Argentina 15 Daniel M. Berry 18
Spain 15 Jaelson F.B. de Castro 17
France 7 Arndt von Staa 16
Germany 7 Itana Maria de Sousa Gimenes 16
Chile 3 Cludia Maria Lima Werner 15
878 Paulo.A.da.M. Silveira Neto et al. / The Journal of Systems and Software 86 (2013) 872889
Fig. 8. Number of studies by region.
international universities and research centers and still publishing
on SBES. In this case, it was counted as international publication.
A mobility program proposed by Brazilian Government called Sci-
ence without Borders (Cincia sem Fronteiras) can contribute to
draw the attention of the international researchers in SBES since
this is a way to meet potential Brazilian students and starting col-
laborations with researchers that will be nanced by the program.
Fig. 10 shows the evolution of the publications per year by each
of the ve Brazilian geographic regions. The Southeast contains the
largest amount of publications in almost every year. The number
of publications of the South region is similar to the Northeast in
average but the later has an increase in the number of publications
in the last 10 years. North and Midwest have the lowest number
of published papers in SBES. No particular pattern was identied
regarding to the number of publications per region.
In order to identify the cause of the increase or decrease on the
amount of publications by region throughout the 25 years, we ana-
lyzed the relationship with the number of masters and doctoral
courses per region during such period of time.
Before 1987, the Southeast region already had 7 masters and 3
doctoral courses. This scenario could explain why it had the largest
quantity of papers since the beginning of SBES. The southeast kept
this statistic and continues to have the largest amount of publica-
tions and graduation programs since then.
For a long period from 1987 to 2000 the South had more or
almost the same number of graduate programs that the Northeast
region and that statistics was reected in the similar quantity of
publications. However, after 2001, the amount of published papers
by Northeast researchers increased signicantly and in 2008 it
overcame the Southeast. A possible reason is due to the start of
M.Sc. and Ph.D. courses in the Northeast, especially after 2005. In
Fig. 9. Number of studies per state.
Fig. 10. Number of studies by regions per year.
Fig. 11. Evolution of master program per year in each region.
the last edition (2011), the Northeast region had 7 papers published
followed by Southeast and South, with 6 and 3 papers, respectively.
The same behavior can be observed in the North and Midwest
region, although in a smaller scale. After 2001, both the amount
of publications and masters courses have had a litter growth. After
2008, three Ph.D. programs were created in the Midwest and one in
the North. Simultaneously, both regions have had an increase, even
small, of accepted papers. Therefore, the expectation is that this fact
implied on more publications from these regions. Figs. 11 and 12
summarize the evolution of graduate courses per year in each
region. No papers from North and Midwest regions were published
in 2011.
Although the most economically developed Brazilian regions
had received a good score in publications, the Federal University
of Pernambuco (UFPE), that had at least one published paper per
year, was the university with the greatest number of publications
at the SBES conference. The others seven universities are mainly
from South and Southeast regions, representing 64.6% of region
contribution.
Table 9 shows the university classication regarding to the
CAPES criteria. CAPES plays a key role in the expansion and consoli-
dation of graduate (stricto sensu) programs, which is an assessment
 Paulo.A.da.M. Silveira Neto et al. / The Journal of Systems and Software 86 (2013) 872889 879

in consecutive years. After 2003, the university number of papers

accepted became more constant having at least 1 or 2 publications
during the last 7 years. In 2011, nobody from UFRJ published on the
main track. However, no discrepancy on the values was observed on
this situation, so no remarkable reason was found for this scenario.
Until 1993, UFSCar researchers had no publications in SBES.
The reason for this is that the two main researchers of this uni-
versity nished their Ph.D. in 1992 and 1996. After that, almost
every year the professors had papers published but the amount
was unstable, ranging from 0 to 4. Later, in 2003, the amount of
publications became a little bit more constant, even though there
was no increase on the number of published papers.
A similar situation can be observed analyzing the UFRN publi-
Fig. 12. Evolution of Ph.D. programs per year in each region.
cations. Before 2001, the researchers from UFRN had only 2 papers
published at SBES. Afterwards, until 2003, the amount of publica-
Table 9
tions was erratic or null. On the other hand, from 2005 to 2009 the
Universities and CAPES.
number of papers increased. Even though the number did not rise
University Ph.D. M.Sc. CAPES in the last two years, in every year since 2005 there was at least
UFPE Yes Yes 6 one publication from UFRNs researchers. One of the main reasons
PUC-RJ Yes Yes 7 for this is the consolidation of the master program, started in 1998,
UFRGS Yes Yes 6 and the creation of the doctoral program in 2008. Moreover, after
USP Yes Yes 6
UFRJ Yes Yes 7
2006, the university hired a considerable number of professors in
UNICAMP Yes Yes 6 the SE area.
UFRN Yes Yes 4 A total of 114 different universities participated at the SBES con-
UFSC No Yes 4 ference by publishing studies. From the 114 universities, 47 are
UFPB No Yes 3
international and 67 are national. By analyzing the national sce-
UFSCar Yes Yes 4
UFBA Yes No 4 nario, only 24 are private universities.
Authors publications per university and region. In order to eval-
uate the inuence of the top 11 authors on the amount of papers
instrument (ranging from 1 to 7) for the university community, per region and their importance to the university, their publica-
interested in achieving academic excellence degree for the national tion patterns were analyzed throughout the 25 years. Our goal is
M.Sc. and Ph.D. programs. Many of the top 11 universities have a to analyze if the performance of the region or the university is due
Ph.D. program and they are qualied by CAPES in a range from to a consistent and large number of researchers or if it is mainly
4 to 7 levels. Although UFSC had no Ph.D. program until 2011, it because of a single researcher.
was classied by CAPES as level 4 and it appears among the top In our analysis, we have identied that most of the top 11
ten universities. UFPB also appears among the top ten universities, authors are responsible for the highest percentage of publications
however, a different context was seen evaluating UFPB publica- of their university and, as a consequence, of the region.
tions over the years. Until 2002, UFPB had a campus in two cities: The Southeast region has the highest number of publication in
Joo Pessoa and Campina Grande. Even though there were two dif- SBES. The researchers that have published most of the papers are
ferent computer science courses, it was the same university. The from PUC-Rio (number 2 in ranking of universities), UFRJ (number
highest amount of papers was published by the researchers from 5 in ranking) and USP-So Carlos (number 4).
Campina Grande, specially because they already had an old gradu- The top two authors, who have published more than 1 paper
ate program. After 2002, there was a separation and the campus at per year in average, work at USP-So Carlos. They have 91% of the
Campina Grande became UFCG. In the following years, there were publication of their institution and that is the main reason to the
only 2 publications from UFPB. classication of USP-So Carlos in the top 5.
Number of publications per university. Analyzing the publications PUC-Rio is in the second place in the ranking of the top uni-
of the top 10 universities throughout the 25 years of SBES by cal- versities and its regularity is the main explanation for that score.
culating the average and standard deviation, a particular behavior Another reason for that performance is that 4 of the 11 top authors
of the quantity of publications of 6 universities was observed. USP work there. In the rst 15 years, 2 of these authors, Arndt von Staa
is the most irregular university in the top 10, as it has the high- and Julio C.S. do Prado Leite, specially the second one, kept the bal-
est standard deviation, but no specic pattern or reason was found ance on the amount of publications accumulating 32 publication
for explaining this behavior. UNICAMP had a regular amount of works. In the last decade Carlos J.P. de Lucena and Alessandro Gar-
publications, while UFRJ was also unstable. Professors from UFSCar cia have kept the university in the top considering the number of
and UFRN only started to have more publications after a few years, publications in SBES, gathering 42 papers published.
and UFPB had a huge decrease on the number of papers in the last The last two authors in the raking of top 11 authors are from UFRJ
decade because of the separation and creation of another university and they are the major authors from that institution, with 54% of
(UFCG) in the same state. the entire amount of papers published in SBES. Other two authors
In the 1990s, there were a regular amount, between 3 and 5 per Ana Regina C. da Rocha and Marcos Borges are also signicant
year, of papers published by UNICAMP researchers. After 2001, they considering the amount of publications of this institution.
kept some regularity on the amount of papers, but it decreased to 1 Northeast is the second most productive region and there are
or 2 per year in recent edition and to zero in the last one. One of the three main universities in the top 10 ranking. UFPE is the top 1
presumable reasons for this is that two of the main researchers of university considering the quantity of papers published, and it has
the university stopped to publish papers in the last decade. In the two authors in the top 11 ranking. These two authors have written
last year they have no paper. 52% of the papers. The rst of them, Silvio Romero de Lemos Meira,
Also in the 1990s, particularly after 1994, the amount of papers has had a high number of papers, but his publications are concen-
published by UFRJ had an irregularity ranging from 0 or 1 to 4 or 5 trated in the rst 16 years. In the last decade, he has only one paper
880 Paulo.A.da.M. Silveira Neto et al. / The Journal of Systems and Software 86 (2013) 872889

Table 11
Top 8 industries.

Industry Number of publications

IBM 4
CPqD Telebrs 3
ENEA/Italy 2
UNICEN 2
Petrobrs 2
EMBRAPA 2
CTI 2
FUEM 2

Fig. 13. Industry vs. academia.

evolution regarding each SE topic was examined in order to iden-

published. His colleague, Jaelson F.B. de Castro, has started publish-
ing in 1993 and after that he became fairly regular, with 0.83 papers
per year in average. We can conclude that these 2 authors have a
signicant contribution to put UFPE as the top 1. However, there
are other important and slightly regular professors that have been
publishing in SBES throughout the 25 years, such as Paulo Borba,
Augusto Sampaio, and Paulo Cunha.
The other two universities from this region are UFRN and UFPB.
Although there is no outstanding researcher from any of these insti-
tutions, as no professor gures on the top 11 ranking, the two of
them together accumulate 21% of the papers from the Northeast.
In the South, there are two main universities contributing to
publications in SBES. UFRGS is the third university having the high-
est quantity of papers published, but it has only one author guring
on the top 11. Roberto Tom Price is, indeed, the main contributor to
that ranking position, scoring 22 papers published and an average
of 0.92 papers per year. He is the author of 30% of publications from
UFRGS. Some other authors, such as, Leila Ribeiro and Daltro Nunes
had a remarkable contribution as well, particularly in the rst 15
years of the conference. UFSC has some publication that contributes
to put the South in the third place but there is no author in the top
10. The last two regions, North and Midwest, have no university in
the top 10 and no authors in the top 11 rankings.
Industry impact. In order to understand the industry participa-
tion in the SE research, Fig. 13 shows the number of publications
from industry and academia over the SBES editions. Despite the
slight participation by industry at the conference, this participa-
tion is constant, around 1.96 studies per year. Tables 10 and 11
show the number of studies per university and industry, respec-
tively. When comparing these data, it is clear that we need more
results from industry. It can be achieved by encouraging industry to
participate and also by reducing the existing gap between industry
and academia.
4.5.2. What are the most investigated research topics and how
they are evolving?
All the papers were analyzed in order to evaluate the topics
regarding to the ACM research classication, ICSE Keynote topics
(Ghezzi, 2009), and the Study types (Montesi and Lago, 2008). The
tify any trend in the area and try to nd any relationship with ICSE
topics.
Table 12 shows the top ten most investigated ACM research top-
ics by universities. The research topics investigated by industries
will be presented in another section, when a comparison between
both data will be done. A subsequent investigation in each topic
shows how the sub-topics are distributed. All of them are described
in Table 13.
A deeper analysis of the areas on the SBES papers classied
according to ACM topics reveals the inuence of others related SE
events in Brazil. The number of papers in Software Architecture
(SA), an area that emerged in 1993, is null in the rst ve edi-
tions, but they began to appear in that year. From 2004 to 2006,
a total of ten papers in SA was published in the main track. How-
ever, with the rst edition of SBCARS, the Brazilian Symposium on
Components, Architecture and Reuse in 2007, the number of papers
dropped down to zero so far.
The ACM topics Software and Program Verication includes For-
mal Methods and SBES had 2 papers per edition in average from
1989 to 2003, with the highest number of 5 papers in 1994. Since the
rst edition of the Brazilian Symposium on Formal Methods (SBMF)
in 2004 that number dropped down to 0.71 papers per edition in
average.
It is worth to say that the very low number of papers in Software
Quality, Software and System Safety, and in Human Factors in Soft-
ware Design is due to specic Brazilian conferences in these areas.
The Brazilian Symposium on Software Quality (SBQS) had its rst
edition in 2002, but several previous editions of the Workshop on
Software Quality were co-located with SBES since 1994. There was
also the Workshop on Fault Tolerance, since 1999, and a Brazilian
Symposium in Human Factors in Computing Systems since 1998,
where many Brazilian researchers from these areas have published
their papers.
Since 1998, the Brazilian Workshop on Requirements Engineer-
ing has attracted many researchers from Brazil and it could explain
the decrease in the number of papers about this subject at SBES in
the last decade. The number of papers per edition in average was
6.2 in the rst ten editions and dropped down to 2.8 in the last ten
editions.

Table 10 Table 12
Number of publications per university. ACM topics.

University Number of publications ACM topic Number of

occurrences
UFPE 86
PUC/RJ 75 Requirements and Specications 133
UFRGS 73 Testing and Debugging 67
USP 71 Design Tools and Techniques 41
UFRJ 56 Design 37
UNICAMP 37 Software Architectures 38
UFSCar 31 Distribution, Maintenance and Enhancement 38
UFRN 20 Programming Environments and Construction Tools 34
UFSC 16 Software and Program Verication 32
UFBA 13 General 24
UFPB 13 Software Engineering Process 24
 Paulo.A.da.M. Silveira Neto et al. / The Journal of Systems and Software 86 (2013) 872889 881

Table 13
ACM subtopics.

ACM topics Number of ACM topics Number of

occurrences occurrences

Requirements and specications 133 Testing and debugging 67

Languages 13 Reliability 2
Methodologies 41 Test Documentation 4
Analysis 8 Testing Strategies 39
Management 5 Debugging Aids 2
Specication 38 Error Handling and Recovery 5
Elicitation Methods 8 Test Coverage of Code 1
Process 2 Test Design 1
Tools 12 Test Execution 1
Validation 6 Test Management 1
Design Tools and Techniques 41 Testing Tools 7
Distributed and Internet Based Software Engineering Tools and 22 Tracing 3
Techniques
CASE 7 Usability Testing 1
Modules and Interfaces 4 Design 37
Programmer Workbench 8 Object-Oriented Design Methods 9
Distribution, Maintenance and Enhancement 38 Rapid Prototyping 2
Enhancement 1 Design Concepts 3
Maintenance Process 2 Methodologies 18
Restructuring, Reverse Engineering, Reengineering 17 Representation 4
Extensibility 5 State Diagrams 1
Maintenance Measurement 1 Programming Environments and Construction 34
Tools
Maintainability 2 Graphical Environments 5
Version Control 7 Interactive Environments 2
Portability 3 Environments for Multi-Processor Systems 1
Software and Program Verication 34 Integrated Environments 11
Formal Methods 10 Programmer Workbench 15
Assertion Checkers, Assertion Languages, Performance 3 General 24
Model Checking 9 Software Psychology 2
Reliability 3 Software Engineering for Internet Projects 6
Validation 9 Standards 16
Software Engineering Process 34 Software Architectures 37
Process Denition 2 Patterns 16
Process Measurement 3 Domain-Specic Architectures 18
Software Process Models 13 Languages 3
Process Implementation and Change 7 Reusable Software 2
Process Infrastructure 2 Reuse Models 2
Qualitative process models 1 Managements 1
Qualitative Process Analysis 6 Cost estimation 1

Regarding to the study type, the studies were classied based on
Scheme dened by Montesi and Lago (2008). The following aspects
were used to classify the studies (Table 14):
Theoretical papers: tend to be methodological or computational
paper, i.e., presenting computational methods and algorithms.
Published papers: general papers are the typical software engi-
neering papers which has a three-section structure: introduction,
main body and conclusion, and extended versions of conference
papers are papers that passed through a double round of reviews.
Empirical studies: can be divided in ve categories: case studies;
experimental papers; eld studies; observational studies, which
include all empirical research strategies that impose little control
Table 14
Studies type.
over the developmental process, and meta-analysis, which allows
to combine results of various empirical studies.
Experience report: papers have the sense of experience, and
can be called by lessons learned reports or industry experience
reports.
Tool report: they are studies that perform specic development
tasks or on how to use specic tools.
It is important to mention that a study is classied as empirical
if it presents at least one section with some empirical evaluation.
Fig. 14 shows the increasing interest on empirical research, since
the second SBES edition, when appeared the rst empirical study.
Initially, the studies were proposing something with none or few
validations; however, as the SE area is becoming more mature, the

Study type Number of

occurrences

Theoretical papers 183

Publishable papers general 120
Case studies 82
How to tool report papers 73
Experimental papers 47
Experience report papers 18
Field studies 14
Observational studies 8
Meta-analysis 2
Extended versions of conference papers 0
Fig. 14. Theoretical vs. empirical.
882 Paulo.A.da.M. Silveira Neto et al. / The Journal of Systems and Software 86 (2013) 872889

Table 15
ACM topics for empirical studies.

ACM topic Experimental Case studies

Coding Tools and Techniques 1 0

Design 3 8
Design Tools and Techniques 1 9
Distribution, Maintenance and 6 6
Enhancement
General 2 1
Human Factor in Software 0 1
Design
Managements 1 1
Metrics and Measurement 1 3
Programming Environments 0 5
and Construction Tools
Requirements and 7 21
Specication
Fig. 15. Empirical methods. Reusable Software 1 6
Software and System Safety 0 1
Software Architectures 3 4
importance to validate the studies, in some way, increases consid- Software Construction 1 1
Software Engineering Process 4 4
erably.
Software Quality and SQA 1 1
Performing a more systematic analysis on the empirical stud- Software and Program 4 5
ies or how the methods, techniques, processes and tools are being Verication
evaluated over the years, we faced with ve different types of stud- Testing and Debugging 11 5
ies. The studies were classied according to ve different empirical
methods (Montesi and Lago, 2008). Fig. 15 shows the distribution
Table 16
of each evaluation method. ACM topics per university.
In this conference, more than a half of empirical studies pub-
University Main ACM topics
lished had some section presenting a case study or experiment.
Over the past 25 years, these two types of article were varied, as PUC/RJ Requirements and Specication
shown in Fig. 16. In the rst year of the conference, we did not UFRJ Requirements and Specication
UFRN Reusable Software
have any empirical paper, however, in 1988, Ana Luzia Goncalves UFPE Requirements and Specication
Sarino, from USP, published the rst paper titled A JSD Method UFSC Requirements and Specication
to support Modular and Object Oriented Programming (the original UFSCar Requirements and Specication
title O Mtodo JSD como Apoio a Programaco Modular e Orientada a UFRGS Requirements and Specication
UNICAMP Testing and Debugging
Objetos) which seems to be experimental. The rst appearance of a
USP Testing and Debugging
case study was in 1989 with a paper titled Formal Methods for Soft- UFBA Mestrics and Measurement
ware Development: A Case Study (the original title Mtodos Formais UFPB Design and Software and Program Verication
para Desenvolvimento de Software: Um estudo de caso) published
by de O. Annarumma et al. (1989). However, despite being the rst
empirical paper, none of them were cited by other SBES papers. Testing and Debugging representing exactly 23.4% of experimental
From this paper, in each year, there was at least one paper which papers. On the other hand, the same topic represents only 6.1% of
reports a case study, except 2000 and 2010. case studies.
The number of experimental papers was increasing in the past The aforementioned ndings could indicate the maturity of the
ten years but it was not always like that. This type of paper had a lot SE area, since there are a lot of methods, techniques and processes
of highs and lows as shown in Fig. 16. With the exception of 1989, in those areas, which is being evolved and assessed over the years.
1993 and 2001, there was, at least, one paper published with this In order to understand what are the main interest of research
kind of article each year. originated from industry and academia, Table 17 shows the top ten
Table 15 shows the distribution of case studies and experiments investigated topics from industry and Table 16 the most investi-
over the ACM areas. Most of them addressed the Requirements gated by universities. Although the signicant difference in number
and Specications and testing and debugging areas. The use of case
studies is more signicant in the requirements and specications Table 17
ACM topic. This topic represents about 25.6% of the case studies, ACM topics addressed by industry.
while it represents about 14.9% of the experimental papers. How-
ACM topic Number of studies
ever, this research topic is not the greatest interest of experimental
papers. The topic that has the most experimental researchers is Requirements and Specication 14
Testing and Debugging 9
Reusable Software 6
Programming Environments and Construction Tools 4
Conguration Management 4
Software Engineering Process 4
General 4
Software and Program Verication 3
Software Construction 3
Design 3
Distribution, Maintenance and Enhancement 3
Design Tools and Techniques 2
Metrics and Measurement 2
Software Architectures 2
Managements 1
Fig. 16. Case studies vs. experimental papers.
 Paulo.A.da.M. Silveira Neto et al. / The Journal of Systems and Software 86 (2013) 872889
Fig. 17. Comparison between ICSE and SBES (tool and SDEs).
Fig. 18. Comparison between ICSE and SBES (Specication and Verication).
of publications over the 25 editions, the studied topics are similar,
both of them have more emphasis on Requirements and Specica-
tions and Testing and Debugging.
4.5.3. How the SBES conference is evolving when compared with
ICSE conference?
In order to understand the relation of the international research
in the conference, the SBES studies were classied in the same clas-
sication proposed by Ghezzi (2009). In his keynote, he analyzed
how each topic is evolving. Based on this analysis, we compared
both schemes as shown in Figs. 1719. The topics that showed some
similarity were Specication and Verication, Tools and SDEs, and
Empirical Studies. In addition, analyzing the gures, we realized
that the national researchers are not following the international
research topics.
The research interest topics have been changing over the years.
First, we identied that the most investigated topic was Tools and
SDEs, as shown in Fig. 17. However, over the years, this topic was
losing focus on research interest, and the quantity of publications
decreased. The same happened to Specication and Verication.
This drop in interest in tools and SDEs is also found in ICSE when
analyzing the same interval (Fig. 17). However, the Specication
and Verication (Fig. 18) topic did not increase or decrease, it has
always maintained an average of published papers.
When comparing the number of empirical studies (Fig. 19) and
how they are evolving, we noted that this type of study is increasing
in both scenarios, national and international. A possible explanation
Fig. 19. Comparison between ICSE and SBES (empirical studies).
883
is that once the methods, processes and techniques are proposed,
they are now being more validated.
4.5.4. How the QUALIS CAPES inuences on the SBES submissions?
The CAPES evaluation system for the Brazilian graduate pro-
grams was created in 1976. In 2001, it was the rst time that
CAPES implemented the triennial evaluation of the programs using
QUALIS, a document that classies conference and periodicals.
QUALIS was rst used in the triennial evaluation of 19982000 and
has been applied since then. The scientic production classied by
QUALIS consists in one of the main instruments to evaluate grad-
uate programs. Thus, it is important to analyze the behavior of the
publications in SBES since the beginning of the use of QUALIS.
In 2002, the number of submissions to SBES raised to more than
100 papers. At that time, SBES had the higher classication level,
considering only Brazilian Conferences in QUALIS (level A). Until
2009, the number of submissions was always superior to 100 papers
per year. In 2010, this number decreased to less than 90 papers per
year. According to our survey (Section 5), this fact was inuenced
by the changes in the classication of the events in the new QUALIS
document, which was released in 2009. The new QUALIS classied
the publications according to the impact factor of the periodicals or
the conferences. This factor is mainly inuenced by the frequency
in which a paper is cited. In this scenario, SBES was classied as a
B2 level, the fourth level in a seven strata of the classication.
4.5.5. What is the relationship between the number of
publications in English and in Portuguese?
The ofcial idiom used in SBES has been an issue discussed by
the community for many years. Nowadays, English and Portuguese
are the ofcial idioms. Considering all of the 25 years of SBES, 69.5%
of the papers were written in Portuguese. In the rst 13 years, the
amount of papers written in Portuguese was almost twice, or even
three times the number of papers written in English. From 1987
to 1999, 85.15% of the papers were in Portuguese. However, in
2000, suddenly there were more papers in English than in Por-
tuguese. Since that year, there has been no clear pattern 127
(51.62%) papers were written in Portuguese and 119 (48.37%) were
written in English. The number of papers in both languages has
altered. In some years, there are more English papers and in others,
papers in Portuguese were the majority. Fig. 20 shows that in some
years there are more papers in English but then it changes without
clear evidence. Still, the absolute difference between the number
of papers has not been so different.
In addition, researchers started concerning about creating more
interactions with other people around the world working on the
same topics, and about making possible that the paper can be read
by any researcher interested in the topic.
However, there is no consensus if English should be the ofcial
language. Some of the researches argument that SBES should accept
papers in English and Portuguese. Others consider that as SBES is
884 Paulo.A.da.M. Silveira Neto et al. / The Journal of Systems and Software 86 (2013) 872889
Fig. 20. Relationship between English and Portuguese publications.
the most important Brazilian conference, students, professors, and
researchers in general ought to have the possibility of writing their
ideas in the language they feel more comfortable. However, all of
them agree that articles and specially presentations in English must
be motivated, mainly because it brings more international visibility
to the papers and to the symposium.
4.6. Main ndings
A summary of the main ndings, as well as, some topics to be
further investigated are discussed as following:
F1: The number of submissions over the 25 editions is increas-
ing. It starts with 40 in 1987, reaching a peak of 134 in 2006,
an average of 89.125. More investigation is required in order to
understand the rates in number of papers submitted over the
years.
F2: Although the SBES papers are being referred, the citation aver-
age of the rst and second decade, 10.535.25, shows a decrease
in SBES papers citations. 1993 and 1997 had the major number of
citations 18 and 15, respectively. A deep investigation is required
to understand how it happens on ICSE, and from which data it
becomes signicant.
F3: Although SBES is a Brazilian conference, the most cited study,
was Evaluation of the Cost of Alternate Mutation Strategies pub-
lished in 1996, by Aditya Mathur and W. Eric Wong, both from
Purdue University, West Lafayette, IN, USA.
F4: The top three contributors which collaborates with the biggest
number of publication were: Jos Carlos Maldonado (ICMC-USP),
Paulo Csar Masiero (ICMC-USP), and Silvio Romero de Lemos
Meira (UFPE), with 36, 27 and 25, respectively. In the context of
the most cited author, Jos Carlos Maldonado (29 citations) and
Paulo Csar Masiero (17 citations) are still in the list, but Mrio
Jino (15 citations) appears in the list. Silvio Romero de Lemos
Meira has 4 citations, which put him the thirty-rst position. A
further analysis should be performed in order to understand the
authors interest, regarding to research areas.
F5: UFPE, PUC/RJ, UFRGS, USP, and UFRJ were the universities
with more publication at SBES. An investigation to identify trends
regarding to how the number of publications evolves increasing
or decreasing is required. For example, if it increases due to the
growth in the amount of hired professors in a particular SE area.
F6: The rst two top authors are from the Southeast region which
contributes with 339 publications. One possible reason is that it
is the most developed region. However, it is important to analyze
other possible factors, if the rst researcher in the area is from
that region or the rst M.Sc. and Ph.D. programs are located in
that region.
F7: The number of international studies is greater than three
Brazilian regions, including the South region, which is one of the
most economically developed. It had the rst international pub-
lication in 1992 and achieved seven publications in 1994. In this
dataset, Brazilian authors that were working abroad had their
papers counted as international. Another future investigation can
analyze Brazilian and foreign researchers in each contribution.
F8: Even with the active participation of industry over the 25 edi-
tions, it is still discrete with average of 1.91 papers per year. A
deep investigation is required to understand it, as well as, the
main reasons which makes the companies interested in the con-
ference and specic topics.
F9: Most of the top 11 universities have a Ph.D. program and are
qualied as CAPES from 4 to 7. Although UFSC had no Ph.D. pro-
gram until 2011. It was classied as CAPES 4 and appears among
the top ten universities. Nevertheless, an investigation in order
to analyze how the graduate program evolves and its impact on
number of publication is required.
F10: The number of studies that report some kind of empirical
study (experiments, case studies and so on), is increasing over
the years. It seems that the conference is becoming more mature,
which impacts on the quality of the studies. The same happens
with ICSE submissions (Ghezzi, 2009). An analysis regarding the
quality of those empirical studies is important to identify some
evolution, and also to investigate how it behaves when comparing
the different SE research areas.
F11: Requirements and specication and testing and debug-
ging were the most addressed research topics. Considering the
Requirements & Specication areas the most investigated sub-
topic was methodology, while in testing and debugging the
testing strategies were the most investigated. An analysis should
be performed in order to understand if the main SBES researchers
are interested in these areas or if there are many researchers
working on those areas.
F12: When comparing the SBES results with ICSE, some simi-
larity were found regarding to tools and SDEs, Specication and
Verication and empirical studies occurrence. A deep analysis is
required to understand the reasons for that similarity, as well as,
the differences found when comparing another areas.
4.7. Threats to validity
There are some threats to the validity of our study, which we
briey describealong with the mitigation strategy for them.
Classication scheme: During the classication process, the stud-
ies were classied based on our judgment. However, despite
double checking of some studies, some of them could have been
classied incorrectly. In addition, as the abstracts are not so reli-
able, some sections were also read. Regarding ACM classication,
in some cases, a study could be classied in more than one facet,
when it involves different aspect in SE. For this study, we consid-
ered all the classications for the paper.
Manual data collection: As the data collection was performed
manually, since most of the proceedings were not in the digi-
tal format, we considered it as an error prone activity. In order to
mitigate this issue, some classications were double checked.
Authors and co-authors distinction: No distinction was made
regarding the co-authoring of studies. The main author receives
the same score as co-authors.
Universities and industries publications: In some cases, a study
could be published by a university and an industry. When this
happens, the study was counted in both categories (university
and industry).
Authors nationality: When a study is authored by national and
international authors, it was counted in both categories (inter-
national and national publications) in order to have a fair
classication.
 Paulo.A.da.M. Silveira Neto et al. / The Journal of Systems and Software 86 (2013) 872889 885

Fig. 21. The overall survey methodology (Kasunic, 2005; Mosleh and Apostolakis, 1987; Li and Smidts, 2003).

Brazilian regions: In cases in which the studies have more than one Table 18
List of study experts.
author from different regions, each region received a publication,
it holds true for the international authors. Experts University
Number of accepted and submitted papers on ICSE: Most of the
Alessandro Garcia PUC-RIO
data regarding to ICSE, were obtained from the keynote pre- Arndt von Staa PUC-RIO
sentation (Ghezzi, 2009). When available, the ICSE data were Carlos J.P. de Lucena PUC-RIO
cross-checked. Cludia Maria Lima Werner UFRJ
Guilherme Horta Travassos UFRJ
Marcos Borges UFRJ
5. Expert opinion survey Paulo Borba UFPE
Paulo Csar Masiero USP
Roberto Tom Proce UFRGS
This survey is a data-gathering and analysis approach in which
respondents answer questions or answer statements previously
developed. It allows the generalization about the beliefs and opin-
our study, a set of 23 experts were selected out of over 30 potential
ions of many people by studying a subset of them (Kasunic, 2005).
candidates, however 9 experts answered our invitation. Table 18
The expert opinion method is not new (Chhibber et al., 1992), how-
shows the list of experts which collaborates with our study.
ever, the idea that the musings, brainstorming, guesses and experts
speculations, which is considered a signicant input in a structured
decision process is relatively recent. The widespread use of expert 5.2. Survey questionnaire and evaluation
opinion stems from the fact that knowledge in different elds which
involves probabilistic analysis and decision-making process is gen- Once the research objectives were dened and the target audi-
erally rare and incomplete. ence was identied and characterized, the survey instrument
In this context, both methods were combined in order to take design comes into place. The questionnaire, which composes the
advantage of them. Fig. 21 shows the overall process applied in this survey, was dened based on the steps dened in Kitchenham and
study. Peeger (2008), Fink (2003): search the relevant literature; con-
struct an instrument; evaluate the instrument; and document the
instrument.
5.1. Survey objectives, design, and target audience
According to Fink (2003) the questions can be done in two differ-
ent forms. When they require respondents to use their own words,
This study aims to show a retrospective and prospective view
they are called open and when the answers or responses are pre-
regarding the SE area in Brazil, in order to provide new insights to
selected, the questions are termed as closed. Both forms present
researchers and practitioners. It considers the SE expert perspec-
benets and limitations. Although Kitchenham and Peeger (2008)
tives regarding to the results of our preliminary studies (Gomes
advocate the use of closed questions in self-administered question-
et al., 2011; Leite et al., 2011), which was performed to gure out
naires, our set of questions is also composed of open questions,
how the SE area is evolving and if it suffers inuences from inter-
which aims to capture the researcher opinion. Thus, the study
national research.
questionnaire is composed of 1 closed question and 12 opened
Different survey design involves different objectives, so it is
questions. The complete set of questions is available at the
important to dene a concise design in order to clearly estab-
website3 .
lish all goals. Fink (2003) proposes two different survey designs:
In order to avoid interpretation difculties or technical prob-
experimental and descriptive. The former compares two or more
lems, a pilot test was performed to simulate the survey application
groups, at least one is experimental. The latter produces informa-
on a small scale considering different members of our target
tion regarding to existing groups and phenomena. Concerning to
audience. It is important to reinforce that the authors were not
the way in which the survey was carried out, Kitchenham and
considered during the pilot test.
Peeger (2008) propose three different ways: self administered
questionnaires, telephone surveys, and one-to-one interviews. In
our context, the descriptive self-administered questionnaires were 5.3. Expert opinion synthesis
used since the SE experts involved were located in different places.
The design-sampling plan was based on the results provided by Cruzes and Dyb (2011) state that research synthesis is a col-
our previous work (Gomes et al., 2011; Leite et al., 2011) in which lective term for a family of methods that are used to summarize,
the main SE researchers were identied, enabling results gener- integrate, compare and combine the ndings of different studies
alization and also increasing the survey precision and condence. on a specic topic or research question. One of these research syn-
Since there is a chance that the expert will make a mistake or that thesis methods is called Narrative Synthesis (Rodgers et al., 2009),
his knowledge is inadequate, it is more safe to use more than one
expert. This tendency to seek as many experts as possible, stems
from a perception of safety in numbers (Li and Smidts, 2003). In 3
http://paulosilveira.com/SBES25/form.php.
886 Paulo.A.da.M. Silveira Neto et al. / The Journal of Systems and Software 86 (2013) 872889
which is characterized by the adoption of a narrative (as opposed
to statistical) summary of ndings of answers. This way, the nar-
rative synthesis analysis was used in order to gather all expert
answers.
5.4. Data collection and analysis
In order to collect the data the experts lled an online form avail-
able on a Website. The form was published in November 2011. From
the 23 invited researchers only 9 reported their answers.
5.5. Collected data
From the nine experts, six were from Rio de Janeiro, one from
So Paulo, one from Rio Grande do Sul and one from Pernambuco
and all their answers were gathered. The experts are from academia
and are Ph.D. in computer science.
In addition, all the experts worked in SE projects and were
general chairs, program chairs, program committee members or
keynote speakers in one of the 25 SBES editions.
5.6. Survey results
In this section the analysis of the collected data are presented,
discussing the given answer for each question. Two of the thirteen
questions were made to contextualize the researcher, the way they
were used in the previous section.
5.6.1. Main problems and challenges for the Brazilian researcher
Some problems and challenges were identied by the
researchers, they are described as following:
Difcult interaction with companies: For the researchers that work
in public universities, the interaction with companies in order to
establish high level projects is not easy. In some cases, there is low
integration between the academia and the commercial or indus-
trial software community. In addition, the industry demands few
research, which makes the academic research isolated in some
cases, and it is not the main goal to solve a real scenario problem.
The Informatics Acts (Acts n 8.248/91; n 10.176/01 e
11.077/04), rst published, in 1991, in Brazil intends to foster ICT
(Information and Communication Technology) industry to invest
in research and development (R&D) and to bring together Brazil-
ian researchers and industry practitioners. According to a report
from the Brazilian Science, Technology and Innovation Minis-
ter, 74% of the investments in R&D are concentrated in private
research companies. Those companies accounted for almost 20
times more resources than the public and private education insti-
tutions where the main authors of the SBES publications are. The
investments grew quite markedly from 2003, primarily as a result
of initiatives of some multinational companies that have estab-
lished their own R&D institutes in Brazil. The report reveals that
more than 60% of R&D is in software development, particularly
in commerce and banking software. However, those R&D invest-
ments were not reected in scientic publications. Only 23% have
published results and one company is responsible for 90% of pub-
lications. It means that the Informatics Act was important to
improve technological development in software industry but not
in software research. The integration of industry and academia
is still a goal to be achieved. (CGEE-GEOPI, 2010, Final report of
Informatics Policies Evaluation.4
4
http://www.mct.gov.br/index.php/content/view/333859.html, in 28th April.
The bureaucracy and inefciency of the funding agencies: Unstable
sources and bureaucracy were pointed as being the main prob-
lems that the research should face in the Brazilian scenario.
High student turnover: The high number of jobs in industry, offer-
ing salary four or ve times bigger than the master and Ph.D.
scholarships, have contributed to the students turnover. In this
way, the academic environment is not too attractive for them.
Pressure for high productivity: The researcher is involved with
too many activities at the same time, which makes it difcult
to maintain a good publication average.
Unfair and erroneous evaluation: The unfair evaluation by some
national conferences, was also pointed as a problem. In some
cases, the paper is not properly reviewed or the feedbacks are
not good enough to improve the paper. It is important to men-
tion that this problem is independent of country. Thus, it should
not be considered a problem or challenge for Brazilian authors.
5.6.2. National or international publications and citations
Five experts (55.55%) usually publish in both international
events and national events, 33.33% in international and only 11.11%
in national ones.
Some of the researchers stated that although international pub-
lications are more valuable in terms of productivity evaluations, it
is still very important to publish inside the country.
The international publications availability and the hard access
for the Brazilian publications are the main reasons to the authors
prefer to cite international than national papers. Other researchers
pointed the credibility of international papers as being higher than
the national ones. In addition, they mentioned the worldwide repu-
tation of international conferences and journals. Some researchers
also mentioned that they always cite important papers being inter-
national or not.
5.6.3. Reasons that some authors cite more international papers
than the national ones
Different reasons were pointed regarding to this aspect, such as:
(i) the limited number of researchers in some areas in Brazil and
in some cases, they investigate different areas; (ii) the proceedings
availability can also impact when searching for related research;
(iii) some sub-areas are inuenced by international researchers;
(iv) the low academic value attributed to the local events can also
contribute to the reduced number of national papers citation; and
(v) based on the conventional wisdom that the credibility of those
(international) papers is higher than national ones.
5.6.4. English use
In order to understand that variability, we interviewed program
chairs and some of the most active authors of SBES papers. Accord-
ing to them, one of the main reasons for the grown of English paper
in recent years are the need to be mentioned worldwide. Moreover,
Brazilian Government funding agencies dened evaluation criteria
of the publications based on their citation to evaluate graduating
programs and courses.
Also, according to the interviewed researchers, since 2001, when
CAPES adopted QUALIS to evaluate the scientic publications of the
graduating programs, many Brazilian researchers, became aware
that writing in English is very important. Two of the interviewed
researchers also considered that students should always write in
English to become a successful researcher. By doing that, they could
write a preliminary version of the paper to publish in SBES, and then
write an extended version to submit to a journal.
Some reasons were listed by the researchers, which can justify
the difference in English publications, after 2000:
Best SBES papers award: For some researchers (11.11%), one aspect
that can contribute with this increase is the invitation of best
 Paulo.A.da.M. Silveira Neto et al. / The Journal of Systems and Software 86 (2013) 872889
SBES papers to be submitted (as extensions) to an international
journal and the papers would only be nominated if it was written
in English.
CAPES and digital libraries: Some researchers (33.33%) believe that
the inuence of digital libraries and the pressure from funding
agencies such as CNPq, CAPES, with QUALIS as a target, and others
evaluation committee changed the landscape of scientic publi-
cations worldwide.
English popularization: 55.55% of the researchers stated that
English is the most popular language in the academia and there
is a pressure for all papers to be written in this language.
Most of the researchers (55.55%) agreed that the papers should
be written in English, however there is no common sense regarding
how to encourage it. Some researchers also defended the use of both
languages: English brings visibility; on the other hand, Portuguese
provides more dissemination in Brazil, particularly, for Brazilian
undergraduate students. Therefore, 44.44% believe that papers in
either language should be accepted, perhaps for different tracks.
5.6.5. Academia and industry gap
For the researchers (44.44%), this gap happens due to the lack
of communication between academia and industry. Industry deals
with problems, real problems, and academia deals mainly with aca-
demic topics, and abstract problems. Thus, the solution to reduce
this gap is to connect the academia and industry with the creation
of research institutes, such as the Software Engineering Institute
(SEI) and Department of Defense (DoD) in the U.S. This way, the
SE researchers can get more involved in collaboration projects
industry-academia and motivate industry to write a paper together
and contribute to the SE area.
The industry panels and sections that were dedicated to indus-
trial research in the last SBES editions have also contributed in order
to reduce this gap, since the academic research is aware about the
real problems faced by the industrial practitioners.
5.6.6. Citation decreasing rst and the second decades
Different reasons were identied regarding to this aspect. They
are described as follows:
The availability of different international conferences and jour-
nals increased in the last years can be a factor that contributed to
the reduction of SBES papers citations.
The reduced QUALIS score that SBES received can make the stu-
dents to publish their papers in other international conferences,
which are better ranked. This way, the lower number of papers
published in SBES, the lower number of papers to cite.
The specicity of the papers can also be a reason for this reduction
in citations.
Another reason is the gap between the work conducted in Brazil
and that performed in other countries.
Several aspects contributed to this difference. First, 33.33% of the
experts attributed it to the fact that the way in which the country is
being economically and politically organized, with more developed
states in the Southeast. For example, So Paulo, Rio de Janeiro, and
Minas Gerais represent 70% of the Brazilian Gross National Prod-
uct (GNP). However, it has been changing recently. Second, 66.67%
of the experts stated that the bad income distribution is another
reason for this difference. The number of graduate programs and
university maturity in these regions are also factors that contribute
to the number of publications.
887
5.6.7. Requirements and specication and testing and debugging
research areas
Although the number of researchers in the testing area is not too
big, a number of senior researchers has invested in this area. This
way, their students also published in this area, justifying this big
number. The reason is not the amount of researchers in this area,
but the time invested. A possible explanation can be the fact that
these two topics represent the starting and ending point of the
development cycle, which can impact on the quality.
5.7. Main ndings
Brazilian researchers have faced different challenges in their
careers. One of them is the difcult interaction with companies,
since the topics studied by the academic researchers most of the
time are not applied in industrial projects. We believe that the
industrial track recently organized in SBES is the rst attempt to
minimize this gap between the academia and industry. The quality
of the reviews is also a problem that could be improved with a blind
review or a process with a rebuttal system already used in some
conferences. In addition, the value the scholarship has contributed
with the high students turnover. The funding agency itself makes
mistakes when it decides the values of the scholarships. For exam-
ple, a master student working in a project involving the academia
and industry receives a scholarship higher than a Ph.D. student. A
planning and restructuring should be done about the career and
salary plan applied to the Brazilian researchers (students). Finally,
the unfair and erroneous evaluation of Brazilian conferences has
discouraged the Brazilian researchers themselves to publish and
cite national publications. This real problem can be divided into two
different ones. First, we believe that the evaluation of the papers
submitted to conferences is not reliable enough to encourage the
national publication. We believe that the blind review can min-
imize this problem. Next, the way in which the conferences are
evaluated could be improved. We agree that the QUALIS mecha-
nism is an improvement in the evaluation system such as one used
in Australia (Excellence in Research for Australia (ERA) Journal &
Conference Rankings), however, it presents some inconsistencies
since there are conferences and journals with grades that have
been contested by the research community. Nevertheless, we fully
agree that its creation was the rst step in improving the evaluation
system.
All the aforementioned problems can directly or indirectly
impact in the way in which the SBES papers are being written (lan-
guage) and cited. According to our study, the SBES papers citation
rate is not reecting the importance of this event to the Brazil-
ian research. Although the international visibility is important to
increase the SBES papers citation by writing papers in English and
uploading the SBES proceedings in international sources, our study
pointed out that the Brazilian researchers themselves are not citing
the SBES papers.
5.8. Threats to validity
There are some threats to the validity of our study, which were
briey described and discussed:
Sample size: The most obvious threat to internal validity is the
sample size, which is small when applying statistical inference-
based analysis methods.
Research questions: The research questions we dened cannot
provide complete coverage of the SE area. We had considered
this as a feasible threat; however, we had several discussions with
experts to validate the questions.
Reliability: It shows the measured stability across the observation
units, i.e., that a respondent answers the same or approximately
888 Paulo.A.da.M. Silveira Neto et al. / The Journal of Systems and Software 86 (2013) 872889
the similar questions the same way each time. We utilized pilot
tests to improve the reliability of our survey instrument. In par-
ticular, we make sure that all experts could understand the
questionnaire items. In order to improve it, some technical terms
were also explained.
6. Conclusion and future work
In order to understand how the SE area has evolved, this study
presented some ndings resulted from a mapping study (Budgen
et al., 2008) considering the 25 editions of SBES conference and a
survey involving some of the main Brazilian researchers. The results
of this study provided important information regarding the past,
present, and future of the SE area in Brazil.
By analyzing the number of submitted articles over the years,
it is clear the growing interest in SBES symposium, however the
acceptance rate is still around 21.12%. When comparing with ICSE,
the numbers show that both conferences are following the same
tendencies.
Although the industry had a slight participation in the events,
they are still contributing with their research over the SBES edi-
tions. The creation in the last years of the industrial track has
contributed to this participation.
It is important to highlight that the empirical research is increas-
ing. It improves the reliability and quality of the articles ndings,
since it presents an assessment of the proposed approach, method-
ology, method, or so on. The number of articles written in English
is also increasing. This fact contributes with the conference inter-
nationalization.
Although the numbers state that SBES and the SE area in Brazil
are becoming more mature, there are some points and challenges
that need to be addressed. The bureaucracy of funding agencies,
scholarship values, and the lack of reliability on the SBES reviews
are the main problems that need to be solved.
For the next 25 years we hope that the conference achieve matu-
rity in terms of: (i) a broader participation of the industry, which
can be fostered by improving cooperation between academy and
industry in Brazil; (ii) the growth in the number of outstanding
researchers coming from different regions, which has potential to
be achieved as the ongoing expansion program of the superior
education by the Brazilian Government has been paving the way
for the spinoff development of several graduate and undergrad-
uate courses in all regions of the country; and (iii) the increase
of the international visibility and impact factor of the conference,
which can be achieved by broadening the number of publications in
English and attracting the interest of the international community
in SBES.
Finally, it is important to classify and report something about
one of the results. Future work includes extending the interest in
each area of SE to a systematic literature review to deeply classify
and analyze the research area. A comparison with international SE
area can be also be a good way to understand the Brazilian trend
throughout the years.
Acknowledgements
This work was partially supported by the National Institute of
Science and Technology for Software Engineering (INES5 ), funded
by CNPq and FACEPE, grants 573964/2008-4 and APQ-1037-1.03/08
and CNPq grants 305968/2010-6, 559997/2010-8, 474766/2010-1
and FAPESB.
5
http://www.ines.org.br
References
Budgen, D., Turner, M., Brereton, P., Kitchenham, B.,2008. Using mapping studies in
software engineering. In: PPIG: Annual Meeting of the Psychology of Program-
ming Interest Group. Lancaster University, pp. 195204.
Chhibber, S., Apostolakis, G., Okrent, D., 1992. A taxonomy of issues related to the
use of expert judgments in probabilistic safety studies. Reliability Engineering
Systems Safety 38 (12), 2745.
Cruzes, D.S., Dyb, T., 2011. Research synthesis in software engineering: a tertiary
study. Information and Software Technology 53 (5), 440455, Special section on
best papers from XP2010.
Dalmazo, B.L., de Sousa, A.L.R., Cordeiro, W.L., Wickboldt, J.A., Lunardi, R.C., dos San-
tos, R.L., Gaspary, L.P., Granville, L.Z., Bartolini, C., Hickey, M.,2011. IT project
variables in the balance: a Bayesian approach to prediction of support costs.
In: SBES: Brazilian Symposium on Software Engineering. IEEE Press, So Paulo,
Brazil, pp. 224232.
Davis, A., Hickey, A., Dieste, O., Juristo, N., Moreno, A.,2007. A quantita-
tive assessment of requirements engineering publications 19632006.
In: REFSQ: International Working Conference on Requirements Engineer-
ing: Foundation for Software Quality. Springer-Verlag, Berlin, Heidelberg,
pp. 129143.
de O. Annarumma, G., Carneiro, L., de Lima, A.V., Molinari, P., de la Quintana, E., Stern,
R., 1989. Mtodos Formais para Desenvolvimento de Software: Um estudo de
caso. In: SBES: Brazilian Symposium on Software Engineering, Recife, Brazil, p.
201.
Fink, A., 2003. The Survey Handbook. Sage Publications, Los Angeles, EUA.
Ghezzi, C., 2009. Reections on 40+years of software engineering research and
beyond: an insiders view. In: ICSE: Keynote Speaker Presentation at Interna-
tional Conference on Software Engineering, Vancouver, Canada, pp. 158.
Glass, R.L., Vessey, I., Ramesh, V., 2002. Research in software engineering: an analysis
of the literature. Information and Software Technology 44 (8), 491506.
Gomes, J.S., da Mota Silveira Neto, P.A., Cruzes, D.S., de Almeida, E.S.,2011.
25 years of software engineering in brazil: an analysis of sbes his-
tory. In: SBES: Brazilian Symposium on Software Engineering. IEEE Press,
So Paulo, Brazil, pp. 413.
Kasunic, M., 2005. Designing an effective survey. Technical report, handbook
CMU/SEI-2005-HB-004. Software Engineering Institute, Carnegie Mellon Uni-
versity.
Kitchenham, B., Charters, S., 2007. Guidelines for performing systematic literature
reviews in software engineering. Technical report EBSE 2007-001. Keele Uni-
versity and Durham University Joint report.
Kitchenham, B., Brereton, P., Li, Z., Budgen, D., Burn, A., 2011. Repeatability of sys-
tematic literature reviews. In: EASE: International Conference on Evaluation and
Assessment in Software Engineering, University of Durham, pp. 4655.
Kitchenham, B.A., Peeger, S.L., 2008. Personal Opinion Surveys. Springer, London.
Leite, J., Batista, T., Leite, L.,2011. Software engineering research in Brazil: an analysis
of the last ve editions of SBES. In: SBES: Brazilian Symposium on Software
Engineering. IEEE Press, So Paulo, Brazil, pp. 2429.
Li, M., Smidts, C., 2003. A ranking of software engineering measures based on expert
opinion. IEEE Transaction Software Engineering 29 (9), 811824.
MacDonell, S.G., Shepperd, M.J., Kitchenham, B.A., Mendes, E., 2010. How reliable
are systematic reviews in empirical software engineering? IEEE Transaction
Software Engineering 36 (5), 676687.
Montesi, M., Lago, P., 2008. Software engineering article types: an analysis of the
literature. Journal System Software 81 (10), 16941714.
Mosleh A.V.B., Apostolakis, G., 1987. Methods for the elicitation and use of expert
opinion in risk assessment. Technical report NUREG/CR-4962, T Pickard, Lowe
and Garrick, Inc., Newport Beach, CA.
Petersen, K., Feldt, R., Mujtaba, S., Mattsson, M.,2008. Systematic mapping studies
in software engineering. In: EASE: International Conference on Evaluation and
Assessment in Software Engineering. University of Bari, Italy, pp. 6877.
Rodgers, M., Sowden, A., Petticrew, M., Arai, L., Roberts, H., Britten, N., Popay, J.,
2009. Testing Methodological guidance on the conduct of narrative synthesis
in systematic reviews: effectiveness of interventions to promote smoke alarm
ownership and function. Evaluation 15 (1), 4973.
Zannier, C., Melnik, G., Maurer, F.,2006. On the success of empirical studies in the
international conference on software engineering. In: ICSE: International Con-
ference on Software Engineering. ACM, New York, NY, USA, pp. 341350.
Zhang, J., Babar, M.A., Bai, X., Li, J., Huang, L., 2011. An empirical assessment of
a systematic search process for systematic reviews. In: EASE: International
Conference on Evaluation and Assessment in Software Engineering, Durham,
England, pp. 5665.
Paulo A. da Mota Silveira Neto is a PhD student at Federal University of Pernambuco
(UFPE), he is also a researcher from Reuse in Software Engineering (RiSE) Labs. His
main research interest is Experimental Software Engineering, Software Reuse and
Software Quality. Contact him at pauloadmsn@gmail.com.
Jos Gomes is bachelor at Computer Science from Federal University of Bahia (2012).
Has experience in projects management from undergraduate students at Federal
University of Bahia (2012). Has experience in Computer Science, acting on the fol-
lowing areas: software engineering, mapping study, literature review and scoping
study.
Eduardo Santana de Almeida is an assistant professor of software engineering at
Federal University of Bahia, Brazil, where he leads the Reuse in Software Engineering
 Paulo.A.da.M. Silveira Neto et al. / The Journal of Systems and Software 86 (2013) 872889
(RiSE) Labs. He is also the Head of Engineering and Development at Fraunhofer
Project Center (FPC) in Salvador, Brazil. Dr. Almeida received a PhD in computer
science from Federal University of Pernambuco, Brazil. His research interests include
methods, processes, metrics, and tools to develop reusable software. Contact him at
esa@dcc.ufba.br.
Prof. Dr. Jair Cavalcanti Leite is an Associate Professor at the Computer Science
Department of the Federal University of Rio Grande do Norte (UFRN), Brazil, since
1993 working as lecturer and researcher. He holds a B.Sc. in Computer Science from
the Federal University of Paraba (1988), Brazil, M.Sc. in Computer Science from the
Pontical Catholic University of Rio de Janeiro (PUC-Rio), Brazil, 1991, and is Doctor
of Science in Computer Science from the Pontical Catholic University of Rio de
889
Janeiro (PUC-Rio), Brazil, 1998. His research interests are Software Design, Software
Architecture, User Interface Design, and Human-Computer Interaction.
Thais Batista is an Associate Professor at the Computer Science Department of the
Federal University of Rio Grande do Norte (UFRN), Brazil. She received her Ph.D. and
M.Sc. degrees in Computer Science from the Pontical Catholic University of Rio de
Janeiro (PUC-Rio), Brazil. Her current research interests include software architec-
ture, aspect-oriented development, software product lines, cloud computing, and
distributed systems.
Larissa Leite is an undergraduate student of software engineering at Federal Uni-
versity of Rio Grande do Norte, Brazil. Her main research interest is software testing.