2014 40th Euromicro Conference on Software Engineering and Advanced Applications
Towards Automated Process and Workflow Management: A Feasibility Study on
Tool-Supported and Automated Engineering Process Modeling Approaches
Dietmar Winkler Michaela Schnbauer Stefan Biffl
Vienna University of Technology, Institute of Software Technology & CDL-Flex*
Favoritenstrasse 9-11/188, A-1040 Vienna, Austria
<firstname.lastname>@tuwien.ac.at
Abstract [Context] Engineering workflows and processes typi-
cally include a wide range of organization specific best-practices
required for successful project execution. In contrast to business
administration processes, engineering processes are highly specific
and can include various heterogeneous engineering tools to be
integrated in an overall process and project environment.
[Challenge] High complexity and volatility of engineering
processes require specific solutions to define, model, implement,
and evaluate individual engineering process steps. In this paper we
focus on the first step of process management, i.e., the process
definition phase. Based on the well-established Business Process
Management Notation (BPMN) various tools have been developed
that aim at supporting process modeling and automation. The key
question is to what extent different tools support process automa-
tion in a defined context, i.e., in heterogeneous engineering envi-
ronments. [Method] Following a common tool evaluation ap-
proach, we developed an evaluation framework and conducted a
feasibility study to investigate process modeling and automation
capability of tools based on defined requirements and scenarios.
[Results] The developed tool evaluation process was found useful
for analyzing candidate process modeling tools. In addition, results
of the evaluation process highlighted a set of specific tools that can
support modeling and process automation. [Conclusions] The
evaluation framework can enable benchmarking for process mod-
eling tools and support (a) process engineers in selecting most
valuable tools and (b) tool vendors to better understand complex
process needs with respect to improving and/or extending current
solutions.
Keywords software management, process configuration,
process automation, workflow modeling, tool evaluation,
feasibility study, BPMN
I. INTRODUCTION
Business processes and workflows represent the back-
bone of successful organizations. A variety of tools exist for
correct implementation of those processes and workflows. In
ISO standards [10] several processes support organizations
in their business, e.g., management processes, business
processes with focus on customer product development,
customer relationship management, supply chain manage-
ment, and supporting processes such as quality management
[10]. However, in software and systems development organi-
zations engineering processes are success-critical processes
within the organization [5] for product life-cycle support [9].
Systems engineering processes, especially in automation
systems engineering (ASE) e.g., hydro power plant engi-
*
CDL-Flex: Christian Doppler Laboratory for Software Engineering
Integration for Flexible Automation Systems. http://cdl.ifs.tuwien.ac.at.
978-1-4799-5795-8/14 $31.00 2014 IEEE
DOI 10.1109/SEAA.2014.60
neering or manufacturing system development where sev-
eral disciplines have to collaborate, process management is a
key challenge to support successful engineering projects
[11][23]. Various stakeholders, coming from different discip-
lines, use different and highly specific tools and data models
that are the foundation for collaboration and data exchange.
However, observations in industrial settings, e.g., at a large-
scale hydro power plant systems development and integra-
tion organization, highlighted additional challenges to make
processes work; engineering processes must overcome limi-
tations caused by various specific tools that are loosely con-
nected and data models that are incompatible to each other
[13]. However, engineering processes and workflows have to
support this heterogeneity of tools and data models in engi-
neering projects.
Figure 1: Sequential Engineering Process in Automation Systems
Engineering Projects according to [23].
Figure 1 presents a sequential engineering process illu-
strated by the project flow that should manage various
stakeholders and different tools and data models within a
heterogeneous engineering environment [23]. A strict separa-
tion of process steps might be useful for project management
and control, e.g., by following a plan-driven approach [3].
However, in industry practice individual engineering discip-
lines have to work concurrently because of time, budget, and
project constraints. Thus, there is strong need to synchronize
individual (and heterogeneous) data models to enable a
sound foundation for technical development. As engineers
work in parallel and distributed, changes in engineering
artifacts, e.g., in electrical, mechanical, and software plan,
are success-critical issues for project managers and engineers
[23], which have to be supported by an appropriate process
approach.
The automation service bus (ASB) aims at bridging the
gap between various disciplines by providing a middleware
platform that is capable of integrating heterogeneous tools
and data models in a flexible way to support engineering
projects in complex and distributed environments [1]. To
address change management needs for engineers and project
managers, there is the need to implement process rules and
guidelines within the communication platform. Initially, the
ASB approach used hard-coded process rules, derived from
102
discussions with industry partners, to enable collaboration els, e.g., Activiti [18]. The process engine is able to detect
and data exchange between different tools and data models. events and to process them according to guidelines and rules
However, different organizations include different re- defined in business process models. For example, if an initial
quirements in terms of applied tools and data models, and event in a process models occurs, the process engine starts a
involved stakeholders. Thus, there is a strong need to support new process instance and executes this process based on its
flexible process management to address (a) system integra- model [22]. Figure 2 presents an example of a model of
tion needs (e.g., enabling an efficient and effective collabora- business process management systems architecture.
tion), (b) individual requirements of engineers (e.g., using
well-known and established tools and data models even in a
heterogeneous and distributed environment), and (c) project
managers (e.g., enabling efficient and effective project moni-
toring and control in such ASE environments).
In this paper we present the concept of an automation
supported process and workflow approach that enables mod-
eling, implementation and evaluation of engineering
workflows and processes in the ASE domain (see Figure 3).
Further we focus on the first step of the workflow modeling
approach, i.e., how to derive process requirements, how to
model these processes and how to prepare processes for Figure 2: Model of a Business Process Management
implementation, verification, and validation. Because the System Architecture [22].
Business Process Modeling Notation (BPMN) [18][20] has
In Automation Systems Engineering (ASE), process
been identified as a proven and well-established approach for
models include additional challenges, especially if heteroge-
process and workflow modeling, the goal of the first step in
neous tools and data models and a variety of stakeholders are
this research direction focuses on a systematic evaluation of
involved, e.g., mechanical, electrical, and software engineers
tools that are (a) capable of modeling engineering
as well as project and quality managers that are responsible
workflows, (b) support a technical implementation of these
for the overall engineering project. Thus, we see process
workflows, and (c) provide mechanisms for simulation. To
management including tool support in heterogeneous engi-
support this goal, we derived three main research questions:
neering environments a crucial challenge in ASE projects.
RQ1: What are the requirements for tools to automate
Figure 3 presents a basic workflow management approach
workflow definition steps in ASE projects? RQ2: How can
with the ASB including four main parts: (1) process and
BPMN tools be evaluated effectively and efficiently? RQ3:
workflow definition and modeling, e.g., based on customer
To which extent do selected tools support the definition of
input; (2) process implementation that should be supported
executable processes and workflows? To identify the most
by the process and workflow engine; (3) process evaluation
promising tools that support workflow definition and auto-
with focus on verification based on captured process events
maton we developed an evaluation framework and conducted
and data (3a) and process validation (3b).
a tool feasibility study for selected candidate BPMN tools.
The remainder of this paper is structured as follows: Sec-
tion 2 presents related work on business process modeling,
BPM tool support and introduces to the tool evaluation
process. Section 3 presents the research issues. We present
the tool evaluation process approach in Section 4 and the
results in Section 5. Section 6 summarizes validity consid-
Tech. Interop.
erations. Finally, Section 7 concludes and identifies future
work with focus on the next step towards automation sup-
ported process and workflow modeling.
II. RELATED WORK
This section summarizes related work on business
process modeling in the context of automation ASE projects
(Section II.A), BPMN and tool support (Section II.B), and Figure 3: Process and Workflow Management Cycle
tool evaluation processes (Section II.C). in ASE projects based on ASB [21].
A. Business Process Modeling in the ASE context In this paper we focus on the investigation of tools that
Modeling tools have to support specific modeling lan- support process modeling (see Figure 3, step 1) as founda-
guages that can be used for creating graphical representations tion for automation supported transition, verification and
of business processes according to a defined standard, e.g., validation of process entities, relationships, and actions to the
using BPMN [20]. Tools that allow the execution of business process engine. However, it remains unclear which tools
processes include a so-called process engine (or workflow support basic requirements of ASE projects to which extent.
engine) that works with the graphical business process mod-

103
B. BPMN and Tool Support
A key challenge of process engineers is to provide a
process and workflow description that (a) is easy to under-
stand for customers and engineers and (b) include the capa-
bility to transfer process designs to a running process or
workflow engine. The Unified Modeling Language (UML) is
a well-established approach for modeling the static structure
and the dynamic behavior of a system [1]. For instance,
activity diagrams aim at describing processes and workflows
systematically. However, UML primary focuses on software
engineers rather than on business people. In addition there is
no specification for mapping UML Activity Diagrams to a
business process execution language [14].
BPMN is a well-established approach for business
process modeling [20] representing business processes sys-
tematically in the form of business process diagrams. The
graphical representation of BPMN enables stakeholders, who
might not be familiar with formal process modeling ap-
proaches, to design business processes as well as engineering
processes more effective and efficient [8][15]. Because of
the formal definition of BPMN it aims at bridging the gap
between process definition and implementation.
However, tools are needed to support process engineers
in systematically designing and modeling processes and
workflows. In literature, several tool studies have been re-
ported. For instance, Ramakrishnan1 provides results of an
automation-supported tool study which is based on vendor
information and defined selection criteria. It remains open
how the tools perform in a practical environment. Pawar
investigated several BPMN modeling tools with respect to
testability and interoperability. However, it remains open
how BPMN results can be used for automation-supported
workflow implementation [16]. Nevertheless, existing tools
studies can be used as a starting point for evaluating selected
tools in context of ASE projects in the ASB context.
C. Tool Evaluation Process
A key question is how to evaluate tools in an effective
and efficient way. Poston et al. proposed a tool evaluation
process [17] which has been applied for evaluating software
testing tools. Figure 4 illustrates four steps of the tool evalua-
tion process: (1) Identification of requirements needed by the
user with focus on expected tool features; (2) Definition and
prioritization of selection criteria to focus on the most prom-
ising tool candidates; (3) Search for candidate tools to iden-
tify available solutions for evaluation, and (4) Tool evalua-
tion that focuses on the execution of the evaluation process
according to identified selection and evaluation criteria.
Figure 4: Tool evaluation process based on Poston et al. [17].
This general approach can represent the foundation for
evaluating tools in different areas, e.g., evaluating BPMN
tools in ASE contexts. However some adaptations are re-
1
Tool study reported by Ramakrishnan available at
http://psymphony.wordpress.com/2013/06/04/top-ten-bpm-tools-you-
cannot-ignore/; accessed in March 2014.
104
quired to focus on specific requirements in the ASE context.
Thus, we adapt the basic process approach, proposed by
Poston et al. [17], to meet individual criteria in context of
our feasibility study.
III. RESEARCH ISSUES
Based on the need for tool-supported definition processes
and workflows in BPMN, aligned with the process automa-
tion framework, we derived three main research issues:
RQ1. What are the requirements for tools to automate
workflow definition steps in ASE projects? Based on a series
of workshops with industry partners, developers, and expe-
riences collected during process development for ASE
projects in the hydro power plant domain, we elicited and
analyzed the most important requirements as input for tool
evaluation.
RQ2. How can BPMN tools be evaluated effectively and
efficiently? Based on Poston et al. [17], we developed a
context-specific evaluation framework that focuses a syste-
matic analysis of candidate BPMN tools in the context of the
ASE project based on the ASB platform.
RQ3. To which extent do the identified tools support the
definition of executable processes and workflows? We ap-
plied the framework to selected and most promising BPMN
modeling tools, i.e., tools that aim at supporting process and
workflow modeling requirements (see Section V.C) towards
business process automation, and analyzed the tools accord-
ing to the evaluation framework.
IV. TOOL EVALUATION PROCESS
This section summarizes the adapted tool evaluation
process based on the basic process proposed by Poston el al.
[17] and applied by Winkler et al. [24] (see Figure 5). Adap-
tations of the initial tool evaluation process include (a) de-
velopment of scenarios relevant for the ASE domain
(process step 4a), (b) development of the evaluation frame-
work (process step 4b), and (c) evaluation of the selected
candidate tools (step 4c). Note that 4c is similar to the origi-
nal process step 4 proposed by Poston et al. [17]. We believe
that these additional steps (4a and 4b) can help to (a) better
evaluate tools in a given domain or context and (b) provide a
systematic and repeatable approach towards benchmarking
of tools.
Figure 5: Adapted Tool Evaluation Process.
Figure 5 presents the adapted tool evaluation process
based on [17] and [24], including six steps:
Step 1: Identification of Requirements. Based on existing
work (see Section II.C) and a sequence of workshops ex-
ecuted by a team of process experts and ASB developers, we
derived a set of requirements as input for BPMN tool evalua-
tion. Note that we followed the EasyWinWin process ap-
proach [4] for the workshops. In addition we categorized
collected requirements and prioritized them according to
individual benefits in context of process and workflow mod-
eling, simulation, and the capability for automation-
supported implementation, verification, and validation. Prior-
ities follow the estimated importance according to process
design needs and include four levels: (a) Critical require-
ments (weight 10) that are mandatory for every tool; (b)
Important requirements (weight 7 = high); (c) less important
requirements (weight 3.5 = medium); and (d) nice-to-have
features (weight 1 = low) that are considered as optional.
Step 2: Define and prioritize selection criteria. To effi-
ciently focus on the most relevant tools to be investigated,
we defined a set of success-critical criteria for every re-
quirements category. Note that these criteria are based on
identified and prioritized requirements, i.e., critical require-
ments (weight 10). See
for details in the context of our feasibility study.
Step 3: Identification of available tools. Based on list of
current BPMN tool vendors provided by the OMG Group2,
sources provided in Section II.B and a quick web search we
derived a set of 76 candidate tools that are promising candi-
dates for evaluation. See Schnbauer et al. [19] for a com-
plete list of candidate tools. However, evaluating 76 tools
takes a considerable effort. Thus, we applied the selection
criteria, defined in Step 2 of the adapted evaluation process
to focus on the most promising candidate tools.
Step 4a. Scenario development. According to our pre-
vious work in the area of tool evaluation [24] common and
typical scenarios derived from individual application do-
mains help analyzing candidate tools effectively and effi-
ciently [24]. Scenarios include typical activities from user
perspective to better identify benefits and weaknesses of the
tool under investigation. In context of this work, one key
process derived from our industry partner has been selected
for evaluation, i.e., a change management workflow. Note
that scenario brainstorming can be supported by established
methods and tools, e.g., EasyWinWin [4].
Step 4b. Evaluation framework definition. The evaluation
framework typically consists of a table of categorized re-
quirements and corresponding priorities in context of the
scenario and/or application domain, a set of individual tools
to be evaluated, assessment results, and the overall score.
Figure 6 illustrates an example for an evaluation matrix.
Individual assessment results (per requirement) have been
derived by expert estimation regarding the degree of fulfill-
ment, i.e., 0 if the tool does not support the requirement and
1 if the tool fully supports the requirement. However, de-
pending on the requirement different values in the range of
0% to 100% might apply. Based on weighted requirement
(i.e., assessment results multiplied by priority) a Score (S) is
2
Object Management Group (OMG): http://www.bpmn.org
105
calculated per requirement and requirement category. The
results are normalized by dividing them by the maximum
total score and can be interpreted as percentage values that
indicate how well each tool scored. Calculated and norma-
lized total scores can be compared and aggregated to achieve
a comprehensible ranking of the evaluated tools. See Section
V for the evaluation results of the feasibility study.
Tool C
...
...
Figure 6: Tool evaluation Framework based on [24].
Step 4c. Tool evaluation. By applying test scenarios, e.g.,
the change management process approach, and the evalua-
tion framework, we evaluated and analyzed every selected
candidate tool. Note that the scenarios (from user perspec-
tive) are typically based on user interaction with the tool.
V. BPMN TOOL EVALUATION RESULTS
We applied the evaluation process by conducting an ini-
tial tool study (i.e., feasibility study) for selected BPMN
tools with focus on the capability of tools in the ASE and
ASB context. The main goal of this study was to (a) elicit
requirements related to BPMN modeling, simulation, and the
capability towards automation-supported workflow imple-
mentation, verification and validation, (b) develop an evalua-
tion framework as initial benchmarking approach for users
(e.g., to support tool selection) and tool vendors (e.g., to
improve and evolve existing tools), and (c) the identification
of most promising candidate tools for further usage in the
business process modeling life-cycle in the ASB context.
A. Tool Study and Arrangements
Basically the feasibility study followed the defined
process steps, described in Section IV. However, different
groups of stakeholders are involved in the individual steps of
the study, e.g., in the requirements elicitation and prioritiza-
tion step a set of workshops with different stakeholders, i.e.,
customers, developers, and quality and process experts, has
been executed; tool evaluation framework design and tool
evaluation have been done by one master student supported
by a supervisor and reviewed by two process experts from
industry and research partners.
Requirements elicitation and prioritization. During
workshops with industry partners a set of process require-
ments, mainly with focus on the process workflow, were
defined. We applied the EasyWinWin process approach [4]
to elicit, classify, and rate individual requirements. These
workshops have been moderated by a quality manager and
process expert, i.e., two of the authors. Result was a basic set
of requirements (from customer perspective) and the evalua-
tion scenario to be implemented. The second part of the
requirements elicitation step includes the identification of
additional requirements and process constraints from devel-
opment perspective. Note that one goal is to support process
modeling, simulation, and automation-supported implemen-
tation of the workflow (see Figure 3). The second series of
workshops involved 3 researcher and 5 developers to find
additional requirements for process implementation as input
for evaluating candidate tools. We used results of the indus-
try partner workshops as input for the workshops.
Tool evaluation design and evaluation execution. Based
on the workshop results, the evaluation framework has been
defined and the tool study was executed by a master student,
who has experience in process modeling, supported by a
supervisor. The results of the study were reviewed by two
process experts from industry and research partners.
B. Requirements and Tool Properties (TP)
Expected tool requirements and properties were collected
in two series of workshops involving different groups of
stakeholders, (a) from engineering process perspective and
(b) from development perspective. Overall, 39 requirements
in 6 requirements categories were identified. Table 1 sum-
marizes the identified requirements categories and the share
of requirements. The basic requirements classes focus on
general requirements (10%) (e.g., availability or platform
support), tool usability (28%), and the support of BPMN 2.0
(3%)). Because our main goals is to investigate the applica-
bility of candidate tools with respect to data exchange and
the capability for automation supported implementation, we
introduced the requirements categories Export Functionality
and Interoperability (23%) and BPMS Activiti support, a
promising tool for automating processes (5%). BPMS refers
to a Business Process Modeling System or Suite. Finally,
modeled processes have to be verified and validated; simula-
tion plays an important role in this context. Thus, we intro-
duced a requirement category for this purposes, i.e., Process
Simulation Capability (31%). See Schnbauer et al. for a
detailed list of all requirements and categories [19].
Table 1: Requirements & Requirement Category Overview.
Requirements No of Critical
Requirement Category No % Requirements
General Requirements 4 10%
BPMN 2.0 Support 1 3%
Export Functionality and Interoperability 9 23%
Usability 11 28%
BPMS Activiti Support 2 5%
Process Simulation Capability 12 31%
Total 39 100% 5 (13%)
Collected requirements include different prioritities ac-
cording to the application area and user needs. As introduced
in Step 1 of Section IV we applied four priority levels (i.e.,
critical, important, less important, and nice-to-have features)
and used critical requirements for a pre-selection of candi-
date tools. Out of 39 requirements, 5 were identified as crit-
ical (13%). See Table 1 and [19] for details.
C. Identification and Selection of Candidate Tools
As described in Step 3 of Section IV we used several
sources to collect individual tools and came up with a total
number of 76 candidate tools. In the previous step we eli-
cited 5 critical requirements for pre-selection. One of the
most important criteria was the availability of tools for test-
ing purposes. Thus, we had to focus on available open source
tools and on commercial tools, where test versions were
available for evaluation. Table 1 summarizes the number of
requirements per category and Table 2 presents selected
individual requirements that are used for pre-selecting re-
lated tools. However, we are aware that alternative tools
might fulfill defined requirements on a high level; neverthe-
less, they have been excluded in this study because of avail-
ability reasons. Note that we did not include BPMS Activiti
Support in the list of critical requirements because there are
alternative tools usable for workflow implementation pur-
poses. However, BPMS Activiti has been initially selected
by our developers as candidate tool for the next step of the
process automation approach and, thus, was included in the
evaluation section. Nevertheless, Activiti does not support
simulation and was, therefore, excluded from evaluation.
Table 2: Critical Tool Selection Criteria (weight 10).
Requirement Category Individual Critical Requirement
General Requirements Availability for Testing Purpose
BPMN 2.0 Support Support of the BPMN 2.0 standard
Export Functionality and Inte- Export model in xml that can enable
roperability interoperability with other tools.
Usability Tool installation performance
Simulation Support Offering UI for specifying simulation
parameters and executing a simulation
We applied selected critical requirements on all of the 76
tools based on individual information provided by the tool
vendors. Missing information or information contrary to the
defined critical selection criteria lead to an exclusion of the
tool from further evaluation. After this step in the selection
process of candidate tools a test version of each selected tool
was downloaded and installed. The final result of the tool
selection process included eight tools that were considered
for further evaluation (see Table 3; in alphabetical order).
Table 3: Selected Tools for Evaluation.
Tool Tool
1. Abacus3 5. iGrafx Process4
2. AccuProcess Modeler5 6. inubit BPM Suite6
1 3. Agilian7 7. Logizian8
1
4. Bonita BPM Suite9 8. Signavio Process Editor10
1
1
0 D. Scenario Brainstorming
1 Scenarios, i.e., typical use cases in a defined application
domain or discipline, can be used for an effective and effi-
cient evaluation of tools in a given context. Based on our
experiences with scenarios in tool evaluations [24], scenarios
fit also well to the purposes of evaluating BPMN tools and
their applications. Tool requirements and application scena-
rios were elicited during a series of workshops with our
3
Abacus: http://www.avolution.com.au/
4
iGrafx Process: http://www.igrafx.com
5
AccuProcess Modeler: http://www.accuprocess.com
6
inubit BPM Suite: http://www.bosch-si.com
7
Agilian: http://www.visual-paradigm.com/product/ag/
8
Logizian: http://www.visual-paradigm.com/product/lz/
9
Bonita BPM Suite: http://www.bonitasoft.com/
10
Signavio Process Editor: http://www.signavio.com
106
industry partner, a large-scale systems integrator in the hydro
power plant domain (see Figure 1 for the basic engineering
process, implemented at our industry partner).
However, in practice, engineers coming from different
disciplines, work concurrently using a variety of different
tools and data models that need to be synchronized frequent-
ly to enable a sound technical foundation for system devel-
opment and a project that is less risky. This synchronization
process requires a defined workflow that is able to handle
changes in an efficient and effective way. See [23] for a
detailed description and a pilot application of an initial ver-
sion of the change management process, derived from our
industry partner. After a pilot application we refined the
change management process (see Figure 7 for a process
overview). The synchronization process starts with the avail-
ability of a new version, represented by a simple CSV table.
Query EDB refers to a central data base where the current
versions of the overall project are located, i.e., new docu-
ment versions have to be compared to that baseline by the
system. Depending on the comparison results, individual
actions have to be executed (see subtask Signal Compari-
Figure 7: Basic Use Case: Change Management Process Overview in BPMN 2.0 according to [19] and [23].
E. Results of the Tool Evaluation
The main goal of the tool evaluation was to identify the
strengths and weaknesses of selected tools with respect to
defined requirements and scenarios relevant for defining and
designing engineering and business processes and
workflows. Based on the evaluation framework (Section IV)
every selected tool was evaluated according to requirements
and requirements categories. The rating schema follows a
level of fulfillment on a scale von 0 to 100% or 0/1, depend-
ing on the scale to be applied. See Schnbauer et al. [19] for
a detailed description of the different rating schemes, how
they were applied to each requirement, and the complete set
of evaluation results.
Figure 8 presents the summarized results of selected re-
quirements based on the identified categories. Based on the
results of the tool evaluation Logizian turned out to be the
tool with the best overall support for the defined require-
ments. It offers comprehensive simulation support, several
import and export functionalities, and a quite good usability.
According to its design and functionalities Logizian is quite
similar to Agilian, another tool provided by Visual Para-
digm. However, the user group is different: While Agilian
107
son). Finally, human interaction tasks enable the decision
whether to accept or to reject the change (see subtask Hu-
man Interaction). These steps are executed sequentially for
all available signals. The workflow concludes with a sum-
mary report of the synchronization process. See Schnbauer
et al. [19] for a detailed description of the change manage-
ment process and related subtasks.
Based on this engineering workflow we focus on a set of
three basic scenarios for evaluation related to the identified
requirements and categories:
Definition of the workflow/process, i.e., the change
management process derived from our industry partner.
Import and Export that focus on interoperability and
data exchange between similar tools, e.g., electrical, me-
chanical, and software planning tools.
Simulation Test Scenario that can enable verification
and validation of the process, including the definition of
a simulation sequence of the workflow and an observa-
tion of the simulated behavior.
addresses enterprise architects, Logizian is designed for
business analysts and therefore specialized on modeling,
analyzing and improving business processes. Hence, they
also differ in some of their functionalities. Note that these
functional differences are not in the scope of the tool study.
The main weakness of all evaluated tools can be found in
the field of interoperability. Only one half of the selected
tools offer acceptable support for interaction with other tools.
More than fifty percent of the tools offer over-average sup-
port for the actual BPMN 2.0 standard. The other tools like
for example Bonita BPM Suite or AccuProcess Modeler
support at least most of the important BPMN 2.0 elements.
According the aspect of process validation by simulation
more than two thirds of all selected tools could score with
quite comprehensive simulation features. The tool that re-
ceived the lowest score in this requirement category was
Abacus. It only offered simulation-similar features like cost
and performances analysis instead of the expected interface
for specifying simulation parameters. Figure 9 summarizes
the aggregated results of all requirements categories and the
average score of all tools under investigation. The order of
the tools according to the overall results represent the rank-
ing (i.e., from left to right in Figure 9).
 Figure 8: Evaluation of Selected Requirements of Related Categories.
Figure 9: Summarized Results based on Requirements Categories.
VI. LIMITATIONS AND VALIDITY CONSIDERATIONS
Every study has to deal with limitations and threats to va-
lidity. We identified a set of threats and addressed them:
Requirements and Scenarios have been developed in a
set of workshops involving a set of different stakehold-
ers, i.e., industry people, quality and process experts,
and developers. Thus, we believe that we covered a
wide range of stakeholders that addresses critical issues
properly based on a real world setting.
Tool evaluation process. The evaluation is based on a
well-defined and established tool evaluation process in-
troduced by Poston et al. [17] and already applied by
Winkler et al. [24] in a different application domain.
Thus, we see the evaluation process as appropriate.
Tool selection. In context of our initial feasibility study
the selection of tools is based on the availability of exist-
ing solutions, i.e., as open source tools and test versions
of commercial products. We applied selection criteria to
focus on the most promising tools that are available for
evaluation. Thus, the list of evaluated tools is a snapshot
in the given context and cannot be considered as com-
plete. Extending the list of evaluated tools will remain
for future work.
Data collection. Some estimated evaluation results, e.g.,
usability concerns, can be seen as subjective. However,
we referred to best practices to enable an objective as-
sessment in academic environment. In addition, the re-
108
sults, i.e., requirements, priorities, and results, have been
reviewed by experts from industry and research partners.
Study participants. The requirements were elicited in
two series of workshops with industry partners and de-
velopers. The evaluation framework has been developed
by a master student supported by a supervisor. To over-
come a possible bias two process experts from industry
and research partners reviewed the evaluation frame-
work and the results. This setting might not be repre-
sentative for an industrial setting. Nevertheless, this will
not be a strong limitation to evaluate the tool features
with respect to BPMN requirements as the scenarios
have been derived from an industrial environment.
BPMN Limitations. We selected the BPMN notation as a
well-established and proven approach for process and
workflow modeling [18][20], being aware of the limita-
tions regarding notation, semantics, and process interac-
tion of concurrent execution [6]. More recent approach-
es, e.g., Subject-oriented Business Process Management
(S-BPM) [7] or Adaptive Case Management (ACM)
[12] are promising candidates that aims at overcoming
BPMN limitations.
VII. DISCUSSION, CONCLUSION AND FUTURE WORK
Engineering and development processes are success-
critical issues in software and systems engineering processes.
However, additional challenges arise in a heterogeneous
environment where stakeholders coming from different dis-
ciplines have to collaborate and exchange data. In addition
different workflows and processes are applicable that have to
be managed in an effective and efficient way to make
projects successful. In order to support flexible engineering
processes within the process management life cycle,
processes have to be designed, implemented, verified, and
validated according to current needs. Figure 3 presented a
process and workflow management life-cycle in ASE
projects that can be supported by the ASB approach [2].
However, a first step towards automation supported
process and workflow management is the definition of
processes and workflows capable for automation-supported
implementation within a workflow environment. In a first
step of tool support for process and workflow management,
the main research question focused on the systematic inves-
tigation of tools for process and workflow management that
(a) enable the effective and efficient definition of processes
workflows, (b) provide mechanisms for import and export
functionality to support tool interoperability, and (c) enable
the simulation of processes and workflows for verification
and validation purposes.
RQ1. What are the requirements to support automation-
supported workflow definition steps in ASE projects? In a set
of workshops and discussions with industry partners, process
and quality management experts we identified an overall
number of 39 requirements in 6 requirements categories that
are used for evaluating selected candidate tools (see Table 1)
and a set of 5 success-critical requirements for setting up the
tool evaluation. The workshop format (based on the Easy-
WinWin approach [4]) was found helpful in addressing and
eliciting important requirement from different perspectives,
i.e., from industry partner and developer perspective.
RQ2. How can BPMN tools be evaluated effectively and
efficiently? Based on the work by Poston et al. [17] and
according to [24] we developed and extended a tool evalua-
tion process based on a defined set of scenarios. The evalua-
tion framework and the scenarios, derived from real world
settings, represent a systematic and adjustable approach to
focus on most relevant aspects from different perspectives.
According to the evaluation framework, the systematic tool
study enabled an efficient, effective, and traceable evaluation
of selected BPMN modeling tools. Scenarios helped to focus
on most relevant industry partner needs leading to traceable
evaluation results. Alternative contexts or needs can lead to
additional scenarios which can be addressed within the
framework. Further identifying critical selection criteria
helped to reduce evaluation effort as less promising tools can
be excluded early.
RQ3. To which extent do the identified tools support the
definition of executable processes and workflows? The third
research question focused on the results of the tool evalua-
tion. The results of the BPMN tool study showed that there
are several tools available that offer support for the BPMN
2.0 standard as well as other required features for supporting
interoperability and simulation. However, the results also
showed needs for improvement: there are only few tools that
are fully compatible to each other (i.e., limitation of interope-
rability). Furthermore, it seems that there is no uniform BPM
format that can be interpreted by all tools which can also
109
hinder effective data exchange. See Section V.E and [19] for
details on the evaluation process and the results.
Table 4: List of evaluated tools ranked by their calculated
and normalized total scores.
Rank Tool Score
1 Logizian 89 %
2 Agilian 87 %
3 Signavio Process Editor 80 %
4 Bonita BPM Suite 78 %
5 iGrafx Process 76 %
6 inubit BPM Suite 70 %
7 AccuProcess Modeler 62 %
8 Abacus 37 %
Table 4 summarizes the final score of the tool evaluation
study of the 8 selected tools. Logizian received the highest
score, followed by Agilian, and the Signavio Process Editor.
Following these results, Logizian seems to be the most
promising tool to support effective and efficient process
definition, date exchange and interoperability and simulation
based on the results of the tool study. During the evaluation
we observed strengths and weaknesses related to all tools
which offer space for improvement and extensions.
Future Work will include two main research directions, (a)
towards extending and improving the evaluation framework
including a larger set of tools that will be considered in the
evaluation study; and (b) the next step of the process man-
agement framework, i.e., the automation-supported imple-
mentation of process modeling outputs in the ASB environ-
ment (see Figure 3).
Extension and improvement of the evaluation process.
This research was based on preliminary studies (see Section
II.C) and an additional quick search on existing tools. How-
ever, a formal research protocol and a systematic mapping
study could extend the currently available tool sets and lead
to more comprehensive results. In a next step we will also
include commercial tools to enable a more complete view on
BPMN tools. Additional scenarios and a refinement of the
evaluation process will also lead to more comprehensive
results towards a benchmarking approach of BPMN model-
ing tools. We believe that this initial feasibility study can
support tool vendors to improve and extend existing tools
and users to select appropriate tools in their individual con-
texts.
Process and Workflow Management Cycle. Finally, and
the most important future work aspect will focus on the au-
tomation-supported process and workflow management life-
cycle, i.e., the implementation, verification and validation of
designed processes and workflows. Results of this research
direction will support flexible process and workflow man-
agement, can enable automation-supported implementation
in the ASB context and provide mechanism to automatically
verify and validate designed and implemented workflows.
ACKNOWLEDGMENT
This work was supported by the Christian Doppler For-
schungsgesellschaft, the Federal Ministry of Economy, Fam-
ily and Youth, and the National Foundation for Research,
Technology and Development - Austria.
 REFERENCES
[1] Ambler S.: Elements of UML 2.0 Style, Cambridge Univ Press,
2005.
[2] Biffl S., Schatten A., Zoitl A.: Integration of Heterogeneous
Engineering Environments for the Automation Systems Lifecycle,
In: Proc. of Industrial Informatics (Indin), 2009.
[3] Biffl S., Mordinyi R., Raidl G., Steininger H., Winkler D.: An SME
Transition from Plan-Driven to Hybrid Project Management with
Agile Software Development Methods; In: Proc. of EuroSPI,
Industrial Experience Paper, Luxemburg, 2014 (accepted).
[4] Boehm B., Grnbacher P., and Briggs R.: Easy-WinWin: A
Groupware-Supported Methodology for Requirements Negotiation,
In: Proc. of ICSE, 2001.
[5] Bondi A.B.: Foundations of Software and Systems Performance
Engineering: Process, Performance Modeling, Requirements, Testing,
Scalability, and Practice, Addison Westley, 2014.
[6] Brger E.: Approaches to modeling business processes: a critical
Analysis of BPMN, workflow patterns and YAWL, In: Journal of
Software and Systems Modelling, 11(3), pp.305-318, 2012.
[7] Fleischmann A., Schmidt W., Stary C., Obermeier S., Brger E.:
Subject-Oriented Business Process Management, Springer, 2012.
[8] Geambau C.V.: BPMN vs. UML Activity Diagram for Business
Process Modelling, In: Accounting and Management Information
Systems, 11(4), pp.637-651. Bucharest Univ. of Economic Studies,
Romania, 2012.
[9] Grieves M.: Product Lifecycle Management Driving the next
generation of lean Thinking, McGraw-Hill Professional, 2005.
[10] ISO 9001:2008: Quality Management Systems -- Requirements, ISO
Standard, 2008.
[11] Kossiakoff A., Sweet W.N., Seymour S., Biemer S.M.: Systems
Engineering Principles and Practices, John Wiley & Sons, 2011.
[12] Kurz M., Lederer M.: Subject-Oriented Adaptive Case
Management, In: Proc. S-BPM One, pp.123-132, Germany, 2014.
[13] Moser T., Mordinyi R., Winkler D., Melik-Merkumians M., Biffl S.:
Efficient Automation Systems Engineering Process Support Based
110
on Semantic Integration of Engineering Knowledge, In: Proc. of
ETFA, Toulouse, France, 2011.
[14] Peixoto D.C.C., Batista V.A., Atayde A.P., Borges E.P., Resende
R.F., Pdua C.I.P.S.: A Comparison of BPMN and UML 2.0
Activity Diagrams, In: Proc. of VII Simpsio Brasileiro de
Qualidade de Software, Vol. 56, 2008.
[15] OMG: Business Process Model and Notation (BPMN), Version
2.0, Object Management Group, 2011.
[16] Pawar S.N.: BPMN Tools A Comparative Analysis to Improve
Interoperability, MA-Thesis, Purdue Univ., USA. 2011.
[17] Poston R.M., Sexton MP.:. Evaluating and selecting testing tools,
In: IEEE Software, 9(3), pp. 33-42, 1992.
[18] Rademakers T.: Activiti in Action: Executable Business Processes in
BPMN 2.0, Manning, 2012.
[19] Schnbauer M., Winkler D.: A Feasibility Study on Tool-Supported
and Automated Business Process Modeling Approaches, Technical
Report, QSE-CDL 14-02, TU Vienna, March 2014. Available at:
http://qse.ifs.tuwien.ac.at/publication/IFS-CDL-14-02.pdf.
[20] Shapiro R., White S.A., Bock C., Palmer N., zur Muehlen M.,
Brambilla M., Gagne D.: BPMN 2.0 Handbook Second Edition:
Methods, Concepts, Case Studies and Standards in Business Process
Modeling Notation, Future Strategies Inc., 2011.
[21] Sunindyo W.D., Moser T., Winkler D.: Process Model Validation
for Heterogeneous Engineering Environments, In: Proc. of the
Software Quality Days, Practical Track, Vienna, Austria, 2012
[22] Weske M.: Business Process Management: Concepts, Languages,
Architectures, Springer, 2007.
[23] Winkler D., Moser T., Mordinyi R., Sunindyo W.D., Biffl S.:
"Engineering Object Change Management Process Observation in
Distributed Automation Systems Projects", Proc. of EuroSPI,
Roskilde, Denmark, 2011.
[24] Winkler D., Biffl S., Kaltenbach A.: Evaluating Tools that Support
Pair Programming in a Distributed Engineering Environment, In:
Proc. of EASE, Keele, Great Britain, 2010.