Академический Документы
Профессиональный Документы
Культура Документы
Anton Jansen
thijmen@acm.org
Heiko Koziolek
anton.jansen@se.abb.com
Anne Koziolek
Department of Informatics,
University of Zurich
Zurich, Switzerland
heiko.koziolek@de.abb.com
ABSTRACT
1.
koziolek@ifi.uzh.ch
INTRODUCTION
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise, to
republish, to post on servers or to redistribute to lists, requires prior specific
permission and/or a fee.
ICPE12, April 2225, 2012, Boston, Massachusetts, USA
Copyright 2012 ACM 978-1-4503-1202-8/12/04 ...$10.00.
205
2.
2.3
Our case study consisted of three major activities: performance measurement (Section 3), performance modeling
(Section 4), and design space exploration (Section 5). Fig. 1
provides an overview of the steps performed for the case
study. The following sections will detail each step.
3.
PERFORMANCE MEASUREMENT
Measurements are needed to create an accurate performance model. To accurately measure the performance of
the RDS, a number of steps needs to be performed. First,
tools have to be selected (Section 3.1). Second, a model
should be created of the system workload (Section 3.2). Finally, measurements have to be performed (Section 3.3).
3.1
206
use cases: the periodic uploading of diagnostic/error information by devices and the interaction of Service Engineers
(SE) with the system. Surprisingly enough, the customer related use cases were relatively low in frequency. Most likely
this is due to customers being only interested in interacting
with the system when the devices have considerable issues,
which is not often the case.
The RDS thus executes two performance-critical usage
scenarios during production: periodic uploading of diagnostic status information from devices and the interaction of
service engineers (SE) with the system. We approximated
the uploads with an open workload having an arrival rate
of 78.6 requests per minute. Furthermore, we characterized
the SE interactions with a closed workload with a user population of 39.3 and a think time of 15 seconds. All values
were derived from the production logs of the system.
We decided to run load tests with the system on three
different workload intensities: low, medium, and high. The
workloads are specified in Table 1 as the number of sustained
uploads received from the devices per minute, and the number of concurrent service engineers requesting internal web
pages from the system.
The medium workload approximates the current production load on RDS. The low workload was used as an initial
calibration point for the performance model and is approximately half the production load. The advantage of the low
workload is that the system behavior is more stable and
consistent, making it easier to study. The high workload
represents a step towards the target capacity and enables
us to study how the resource demands change at increasing
loads.
Performance Measurement
Measurement
Tool Selection
(Section 3.1)
Workload
Modeling
(Section 3.2)
Measurement
Execution
(Section 3.3)
Usage Model
Performance
Measures
Performance Modeling
Model
Calibration
(Section 4.4)
Model
Construction
(Section 4.3)
Modeling
Tool Selection
(Section 4.1)
Performance
Model
Cost Model
Construction
(Section 5.3)
Degree of
Freedom Mdl.
(Section 5.4)
Manual
Predictions
(Section 5.1)
Cost Model
Degree of
Freedom
Model
Design
Space
Exploration
Pareto-optimal
candidates
Automated
Design Space
Exploration
(Section 5.5)
workload
low
medium
high
Table 1:
uploads/min
41.0
78.6
187.9
SE requests/min
20.5
39.3
93.9
3.3
Measurement Execution
207
4.
PERFORMANCE MODEL
4.2
The PCM is based on the component-based software engineering philosophy and distinguishes four developer roles,
each modeling part of a system: component developer, software architect, system deployer, and domain expert. Once
the model parts from each role have been assembled, the
combined model is transformed into a simulation or analysis
model to derive the desired performance metrics. Next, we
discuss each roles modeling tasks and illustrate the use of
the PCM with the ABB RDS model (Fig. 2).
The component developer role is responsible for modeling
and implementing software components. She puts models of
her components and their interfaces in a Component Repository, a distinct model container in the PCM. When a developer creates the model of a component she specifies its
resource demands for each provided service as well as calls
to other components in a so-called Service Effect Specification (SEFF).
As an example consider the SEFF on the right of Fig. 2. It
shows an internal action that requires 100 ms of CPU time
and an external call to insert data into the database. Besides mean values, the PCM meta model supports arbitrary
208
Uploads/min = 5
Service
Engineer
Device
CPUs = 8
#Replicas = 1
HWCost = 40
Users = 3
Think time
= 15 s
CPUs = 8
#Replicas = 1
HWCost = 40
<<Provided
SystemInterface>>
Service
Engineer
Website
Data Access
CPUs = 12
#Replicas = 1
HWCost = 15
Database Server
DMZ Server
RDS
Connection
Point
(web services)
Parser
Data Mining
and Prediction
Computation
Device Data
Database
<<SEFF>>
<<InternalAction>>
CPU demand = 100 ms
<<ExternalCall>>
DataAccess insert
If prediction
else
<<ExternallCall>>
DataMining mine
<<implements>>
Figure 2: Palladio Component Model of the ABB remote diagnostic system (in UML notation)
probability distributions as resource demands. Other supported resource demands are, for example, the number of
threads required from a thread pool or hard disk accesses.
The component developer role can also specify component
cost (CCost), for example, our database component has cost
10 (CCost = 43) (cf. Section 5.3).
The software architect role specifies a System Model by
selecting components from the component repository and
connecting required interfaces to matching provided interfaces. The system deployer role specifies a Resource Environment containing concrete hardware resources, such as
CPU, hard disks, and network connections. These resources
can be annotated with hardware costs. For example, application server AS1 in Fig. 2 has 8 CPU cores (CPUs = 8),
costs 40 units (HWCost = 40, cf. Section 5.3), and is not
replicated (#Replicas = 1).
Finally, the domain expert role specifies the usage of the
system in a Usage Model, which contains scenarios of calls
as well as an open or closed workload for each scenario.
The Palladio workbench tool currently provides two performance solvers: SimuCom and PCM2LQN. We chose
PCM2LQN in combination with the LQNS analytical solver,
because it is usually much faster than SimuCom. Automatic
design space exploration requires to evaluate many candidates, so runtime is important. PCM2LQN maps the Palladio models to a layered queueing network (LQN). This does
not contradict our former decision against LQNS, since here
the LQN models remained transparent to the stakeholders
and were only used for model solution. The LQNs analytic solver [26] is restricted compared to SimuCom, since it
only provides mean value performance metrics and does not
support arbitrary passive resources such as semaphores.
4.3
Model Construction
209
<<ServiceEffectSpecification>>
<<ExternalCall>>
Call container
<<ExternalCall>>
Call web service
<<Internal Action>>
CPU demand = 57 ms
HDD demand = 0.1 ms
p=0.83
<<ExternalCall>>
Call web service
p=0.17
<<Internal Action>>
CPU demand = 1733 ms
HDD demand = 0.1 ms
<<ExternalCall>>
Call web service
<<Internal Action>>
CPU demand = 67.24 ms
0.4
<<Internal Action>>
CPU demand = 33.15 ms
HDD demand = 0.7 ms
0.45
<<ExternalCall>>
Call web service
0.15
0.157
0.843
<<Internal Action>>
HDD demand = 0.4 ms
<<Internal Action>>
CPU demand = 1921 ms
HDD demand = 0.4 ms
<<ExternalCall>>
Call web service
<<ExternalCall>>
Call container
<<Internal Action>>
CPU demand = 31.25 ms
4.4
5.
Model Calibration
5.1
Manual Exploration
Initially, we partially explored the design space by manually modifying the baseline model [23]. First, we used the
AFK scale cube theory, which explains scalability in three
fundamental dimensions, the axes of the cube. Scalability
can be increased by moving the design along these axes by
cloning, performing task-based splits or performing requestbased splits. We created three architectural alternatives,
210
each exploring one axis of the AFK scale cube [19]. Second,
we combined several scalability strategies and our knowledge about hardware costs to create further alternatives to
cost-effectively meet our capacity goal. Finally, we reflected
several updates of the operational software in our model,
because the development continued during our study.
The first of our AFK-scale cube inspired alternatives,
scales the system by assigning each component to its own
server. This complies to the Y-axis in the AFK scale cube.
However, some components put higher demands on system
resources than others. Therefore, it is inefficient to put each
component on its own server. The maximum capacity of this
model variant shows that network communication would become a bottleneck.
A move along the X-axis of the AFK scale cube increases
replication in a system (e.g., double the number of application servers). All replicas should be identical, which requires database replication. We achieved this by having
three databases for two pipelines in the system: one sharedwrite database and two read-only databases. This scheme
is interesting because read-only databases do not have to be
updated in real-time.
The AFK scale cube Z-axis also uses replication, but additionally partitions the data. Partitions are based on the data
or the sender of a request. For example, processing in the
RDS could be split on warning versus diagnostic messages,
or the physical location, or owner of the sending device.
All alternatives did not consider operational cost. Therefore, we also developed an informal cost model with hardware cost, software licensing cost and hosting cost. Hardware and hosting costs are provided by ABBs IT provider.
A spreadsheet cost model created by the IT provider captures these costs. For software licensing an internal software license price list was integrated with the IT providers
spreadsheet to complete our informal cost model.
We further refined the alternatives with replication after
finding a configuration with a balanced utilization of the
hardware across all tiers. In the end, we settled on a configuration with one DMZ server running the connection point
and parser component, one application server running the
other components and one database server only hosting the
database, i.e., a 1:1:1 configuration.
To scale up for the expected high workload (scen. 1), we
first replicated the application server with an X-split (i.e.,
two load-balanced application servers, a 1:2:1 configuration).
For further workload increase (scen. 2+3), this configuration could be replicated in its entirety for additional capacity (i.e., a 2:4:2 configuration). This resulting architecture
should be able to cope with the load, yet it is conservative. For example, no tiers were introduced or removed, and
there was no separation based on the request type to different pipelines (i.e., z-split).
The potential design space for the system is prohibitively
large and cannot be explored by hand. Thus, both to confirm our results and to find even better solutions, we conducted an automatic exploration of the design space with
PerOpteryx, as described in the following.
Initial candidate
Initial and
random
candidates
2. Evolutionary Optimisation
a
Cost
with quality properties
Set of candidates
3. Present results
211
freedom instances as the starting population. Then, iteratively, the main steps of evaluation (step 2a), selection (step
2b), and reproduction (step 2c) are applied.
First, each candidate is evaluated by generating the PCM
model from the genome and then applying the LQN and
costs solvers (2a). The most promising candidates (i.e. close
to the current Pareto front, fulfilling the requirements, and
well spread) are selected for further manipulation, while
the least promising candidates are discarded (2b). During reproduction (2c), PerOpteryx manipulates the selected
candidate genomes using crossover, mutation, or tactics
(cf. [30]), and creates a number of new candidates.
From the results (step 3), the software architect can identify interesting solutions in the Pareto front fulfilling the user
requirements and make well-informed trade-off decisions. To
be able to apply PerOpteryx on the RDS Palladio model, we
first created a formal PerOpteryx cost model (Section 5.3)
and a degree of freedom instances model (Section 5.4) as
described in the following two subsections.
5.3
Resource container replication clones a resource container including all contained components. In our model, all
resource containers may be replicated. We defined the upper
limits for replication based on the experience from our manual exploration [23]. If database components are replicated,
an additional overhead occurs between them to communicate their state. This is supported by our degree of freedom
concept, as it allows to change multiple elements of the architecture model together [31]. Thus, we reflected this synchronization overhead by modeling different database component versions, one for each replication level.
Number of (CPU) cores can be varied to increase or
decrease the capacity of a resource container. To support
this degree of freedom type, the cost model describes hardware cost of a resource container relative to the number of
cores. The resulting design space has 18 degree of freedom
instances:
5 component allocation choices for the five components
initially allocated to application server AS1: They may
be allocated to any of the five application servers and to
either the DMZ server or the database server, depending
on security and compatibility considerations.
6 number of (CPU) cores choices for the five available application servers and the DMZ server, each instance allows
to use 1, 2, 3, 4, 6, or 8 cores as offered by our IT provider.
7 resource container replication choices for the five application servers (1 to 8 replicas), the DMZ server (1 to 8
replicas), and the database server (1 to 3 replicas). The
resource container replication degree of freedom instance
for the database server also changes the used version of the
database component to reflect the synchronization overhead of different replication levels.
The PerOpteryx cost model allows to annotate both hardware resources and software components with the total cost
of ownership, so that the overall costs can be derived by
summing up all annotations. For our case study, we model
the total costs for a multiple year period, which is reasonable since the hosting contract has a minimum duration of
several years. In total our cost model contained 7 hardware
resource and 6 software component cost annotations. The
hardware resource costs were a function depending on the
number of cores used.
However, the cost prediction cannot be fully accurate.
First, prices are re-negotiated every year. Second, we can
only coarsely approximate the future disk storage demands.
Finally, we do not have access to price information for strategic global hosting options, which means that we cannot explore the viability of replicating the RDS in various geographical locations to lower cost and latency.
Furthermore, we are unable to express between different
types of leases. The IT provider offers both physical and
virtual machines for lease to ABB. The two main differences
are that virtual machines have a much shorter minimum
lease duration and that the price for the same computational
power will drop more significantly over time than for physical servers. While these aspects are not of major impact on
what is the best trade-off between price and performance,
it has to be kept in mind that a longer lease for physical
machines that have constant capacity and price (whereas
virtual machines will become cheaper for constant capacity)
reduces flexibility and may hurdle future expansion.
5.4
5.5
212
1
0.8
0.7
0.9
0.8
CPU Utilization
CPU Utilization
0.9
0.6
0.5
0.4
0.7
0.6
0.5
0.4
0.3
0.3
0.2
0.2
0.1
0.1
0
0
100
300
500
700
900
1100
1300
100
1500
300
500
700
RDS
Connection
Point
(web services)
Device Data
Database Server
CPUs = 8
#Replicas = 1
HWCost = 40
Service
Engineer
Website
RDS
Connection
Point
(web services)
Max# = 3
CCost = 43
CPUs = 3
#Replicas = 1
HWCost = 27
Database
CPUs = 1
#Replicas = 1
HWCost = 22
Parser
Data Mining
Prediction
Computation
Data Access
1300
1500
Service
Engineer
Website
CPUs = 8
#Replicas = 5
HWCost = 200
Database Server
.Max# = 3
CCosts = 60
Device Data
Synching
Database
Parser
Data Mining
Prediction
Computation
Data Access
CPUs = 12
#Replicas = 2
HWCosts = 30
CPUs = 12
#Replicas = 1
HWCost = 15
5.5.1
1100
900
Costs
Costs
5.5.2
For the higher workload scenario, we first ran 3 PerOpteryx explorations of the full design space in parallel on a
quad-core machine. Each run took approx. 8 hours. Analyzing the results, we found that the system does not need many
servers to cope with the load. Thus, to refine the results,
we reduced the design space to use only up to three application servers, and ran another 3 PerOpteryx explorations.
Altogether, 17,857 architectural candidates were evaluated.
Fig. 5 shows all architecture candidates evaluated during
the design space exploration. They are plotted for their costs
and the maximum CPU utilization, which is the highest utilized server among all used servers.
Candidates marked with a cross () have a too high CPU
utilization (above the threshold of 50%). Overloaded candidates are plotted as having a CPU utilization of 1. The
response time requirements are fulfilled by all architecture
candidates that fulfill the utilization requirement.
Many candidates fulfilling all requirements have been
found, with varying costs. The optimal candidate (i.e. the
candidate with the lowest costs) is marked by a square.
This optimal candidate uses three servers (DMZ server, DB
server, and one application server) and distributes the components to them as shown in Fig. 6. Some components are
moved to the DMZ and DB server, compared to the initial
candidate. No replication has to be introduced, which would
lead to unnecessarily high costs. Furthermore, the number
of cores of the DMZ server are reduced in the optimal candidate to save additional costs.
Note, that we did not consider the potentially increased
reliability of the system due to replication. A reliability
model could be added to reflect this, so that PerOpteryx
213
1
0.8
CPU Utilization
0.9
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
100
300
500
700
900
1100
1300
1500
Costs
Parser
CPUs = 4
#Replicas = 5
HWCost = 150
Database Server
.Max# = 3
CCosts = 43
Service
Engineer
Website
Synching
Database
Device Data
Data Mining
Prediction
Computation
Data Access
6.
CPUs = 12
#Replicas = 1
HWCosts = 30
5.5.3
5.5.4
LESSONS LEARNED
214
conclude this type of projects are too expensive for small systems. Short-term fixes may turn-out to be cheaper, despite
their inefficiency. We are therefore not surprised that performance fire-fighting is still common practice. More support
for performance modelers would be required to decrease the
needed effort, e.g. by automatically creating initial performance models based on log data.
They constructed a very large LQN with more than 20 processors and over 100 tasks. After benchmarking the system,
they analyzed the throughput of the system for massively
higher workloads. Huber et al. [27] built a Palladio Component Model instance for a storage virtualization system from
IBM. They measured performance of the system and analyzed the performance for a synchronous and asynchronous
re-design using the model. All of the listed case studies analyze only a single degree of freedom and/or changing workload and do not explicitly address the performance and costs
trade-offs for different alternatives.
Concerning design space exploration approaches in the
performance domain, three main classes can be distinguished: Rule-based approaches improve the architecture by
applying predefined actions under certain conditions. Specialized optimization approaches have been suggested for
certain problem formulations, but they are limited to one
or few degree of freedom types at a time. Meta-heuristic
approaches apply general, often stochastic search strategies
to improve the architecture. They use limited knowledge
about the search problem itself.
Two recent rule-based approaches are PerformanceBooster and ArchE. With PerformanceBooster, Xu et al. [40]
present a semi-automated approach to find configuration
and design improvements on the model level. Based on a
LQN model, performance problems (e.g., bottlenecks, long
paths) are identified in a first step. Then, mitigation rules
are applied. Diaz-Pace et al. [25] have developed the ArchE
framework. ArchE assists the software architect during the
design to create architectures that meet quality requirements. It provides the evaluation tools for modifiability or
performance analysis, and suggests modifiability improvements. Rule-based approaches share the two limitations of
being restricted to the pre-defined improvement rules and
the potential of getting stuck in local optima.
Two recent meta-heuristic approaches are ArcheOpteryx
and SASSY. Aleti et al.[20] use ArcheOpteryx to optimize
architectural models with evolutionary algorithms for multiple arbitrary quality attributes. As a single degree of
freedom, they vary the deployment of components to hardware nodes. Menasce et al. [37] generate service-oriented architectures using SASSY that satisfy quality requirements,
using service selection and architectural patterns. They
use random-restart hill-climbing. All meta-heuristic-based
approaches to software architecture improvement explore
only one or few degrees of freedom of the architectural
model. The combination of component allocation, replication, and hardware changes as supported by PerOpteryx is
not supported by the other approaches, and furthermore
PerOpteryx is extensible by plugging in additional model
transformations [31].
In addition, the other approaches target to mitigate existing performance problems or improve quality properties,
while our study targets to optimize costs while maintaining acceptable performance (still, other quality properties
can also be optimized with our approach, if reasonable in
setting at hand).
7.
RELATED WORK
8.
CONCLUSIONS
215
9.
[1]
[2]
[3]
[4]
[5]
[6]
[7]
[8]
[9]
[10]
[11]
[12]
[13]
[14]
[15]
[16]
[17]
[18]
[19]
[20]
[21]
[22]
[23]
[24]
[25]
[26]
[27]
[28]
[29]
[30]
REFERENCES
[31]
[32]
[33]
[34]
[35]
[36]
[37]
[38]
[39]
[40]
216