Cluster Comput
DOI 10.1007/s10586-015-0460-x
Agent-based load balancing in Cloud data centers
J. Octavio Gutierrez-Garcia1 · Adrian Ramirez-Nafarrate2
Received: 5 March 2014 / Revised: 31 January 2015 / Accepted: 14 May 2015
© Springer Science+Business Media New York 2015
Abstract Cloud data centers are generally composed of
heterogeneous commodity servers hosting multiple virtual
machines (VMs) with potentially different specifications and
fluctuating resource usages. This may cause a resource usage
imbalance within servers that may result in performance
degradation and violations to service level agreements. This
work proposes a collaborative agent-based problem solving
technique capable of balancing workloads across commodity,
heterogeneous servers by making use of VM live migra-
tion. The agents are endowed with (i) migration heuristics
to determine which VMs should be migrated and their des-
tination hosts, (ii) migration policies to decide when VMs
should be migrated, (iii) VM acceptance policies to deter-
mine which VMs should be hosted, and (iv) front-end load
balancing heuristics. The results show that agents, through
autonomous and dynamic collaboration, can efficiently bal-
ance loads in a distributed manner outperforming centralized
approaches with a performance comparable to commercial
solutions, namely Red Hat, while migrating fewer VMs.
Keywords Load balancing · Virtual machine live
migration · Data centers · Cloud computing ·
Intelligent agents · Multiagent systems
B J. Octavio Gutierrez-Garcia
joseogg@gmail.com; octavio.gutierrez@itam.mx
Adrian Ramirez-Nafarrate
adrian.ramirez@itam.mx
1
Department of Computer Science, Instituto Tecnológico
Autónomo de México, 1 Rio Hondo St., Progreso Tizapan,
01080 Mexico City, DF, Mexico
2 should be conducted dynamically. In fact, balancing work-
Department of Industrial & Operations Engineering,
Instituto Tecnológico Autónomo de México, 1 Rio Hondo St.,
Progreso Tizapan, 01080 Mexico City, DF, Mexico
1 Introduction
A Cloud data center is a networked server farm generally
composed of heterogeneous commodity hosts endowed with
a wide variety of computing resources [26]. In Cloud data
centers, virtualization technology hides server heterogeneity,
enables server consolidation, and increases server utiliza-
tion [14,39,41]. A virtualized host is capable of hosting
multiple virtual machines (VMs) with potentially different
resource specifications and different, time-varying workload
types [8]. Servers hosting heterogeneous VMs (each requir-
ing different computing resources) whose workload may
vary unpredictably may cause a resource usage imbalance
within hosts. This results in performance deterioration, and
as a consequence, violations to service level agreements.
Accumulated resource usage imbalance within hosts can be
caused by hosting VMs with unbalanced computing require-
ments and/or unbalanced workloads. For instance, a VM
running a compute-intensive application with low memory
requirements (see [21]) may cause a CPU usage imbalance
with respect to memory usage. Then, the mapping between
VMs and hosts should take into account both (i) computing
resource specifications, e.g., number of cores and memory
size, and (ii) time-varying workload. Live migration of VMs
[12] has emerged as a means to dynamically balance time-
varying workloads in virtualized Cloud data centers because
it enables VMs to be migrated from an (overloaded) host to
another host with near zero service downtime [45]. Neverthe-
less, whereas the mapping of VMs to hosts based on (static)
computing resource specifications can be performed when
a VM allocation request is received, the mapping based on
the time-varying workload of VMs is a challenging task and
loads across heterogeneous hosts by dynamically mapping
VMs to a set of hosts is an NP-hard problem [19].
123

Previous load balancing mechanisms [1,6,32,46] have
disregarded that Cloud data centers are commonly composed
of highly heterogeneous hardware and that VM resource
usages may be highly volatile. In addition, load balancing
mechanisms in Cloud data centers are commonly supported
by centralized front-end entities that route requests to allo-
cate VMs to a set of hosts based on their current load, e.g.,
the Red Hat Enterprise Virtualization Suite [32]. However,
centralized load balancing mechanisms [32,43] rely on a sin-
gle entity to monitor and balance loads across the entire data
center, becoming a system bottleneck. Moreover, to take into
account different resource usage patterns within the Cloud
data center, load balancing mechanisms supported by fore-
casting models have been developed to distribute VMs across
a set of hosts [3,9,20,34,46,47]. However, Cloud data cen-
ters are highly dynamic and unpredictable due to (i) irregular
resource usage patterns of consumers constantly requesting
VM (de)allocations [19], (ii) fluctuating resource usages of
VMs, and (iii) unstable rates of arrivals and departures of data
center consumers. Furthermore, the performance of hosts
when handling different load levels may vary greatly.
The inherent dynamicity of Cloud data centers accentu-
ates the need for the agent-based Cloud computing paradigm
[36] to automate Cloud resource management. As shown in
[36], the agent-based Cloud computing paradigm has proved
to be efficient and effective in managing Cloud resources
(see [17,18] for examples of agent-based Cloud comput-
ing approaches). Agent-based Cloud computing consists of
designing, implementing, and exploiting agent-based prob-
lem solving techniques to achieve complex objectives (e.g.,
load balancing) in continuously evolving Cloud-computing
environments.
This paper proposes an agent-based Cloud load balancing
mechanism to cope with (i) continuously evolving work-
loads of Cloud data centers, (ii) fluctuating resource usages
of VMs, and (iii) irregular resource usage patterns of con-
sumers. The agent-based load balancing approach makes use
of (i) live migration of VMs and (ii) the capabilities of agents
to interact and cooperate among themselves to achieve a
design objective, e.g., load balancing, in a distributed manner.
In this work, agents are endowed with a novel agent-based
problem solving technique, the multi-agent protocol for load
balancing, capable of balancing workloads across a set of
commodity, heterogeneous hosts. In the multi-agent proto-
col for load balancing, roles are defined for three types of
agents: (i) front-end agents, (ii) server manager agents, and
(iii) virtual machine agents, which dynamically cooperate
among themselves to balance different workload types (e.g.,
CPU and memory loads) by using migration policies, migra-
tion heuristics, and a set of load balancing heuristics. The
load balancing heuristics are distributed, i.e., on the one
hand, decisions about (i) when a VM should be migrated
and (ii) which VM should be migrated are made by each
123
Cluster Comput
server manager agent, on the other hand, decisions about
selecting destination hosts are made collaboratively between
server manager agents that need to migrate VMs and server
manager agents willing to host VMs (based on a set of VM
acceptance policies).
The contributions of this work are as follows.
(1) An agent-based load balancing architecture for Cloud
data centers capable of monitoring and balancing differ-
ent workload types in a distributed manner.
(2) A cooperative agent-based protocol capable of dynam-
ically balancing loads to cope with (i) continuously
evolving workloads, (ii) volatile resource usages of VMs,
and (iii) irregular resource usage patterns of consumers.
(3) An agent-based load balancing mechanism composed of
(i) migration heuristics to determine which VM should
be migrated and its destination host, (ii) migration poli-
cies to enable agents to establish preferences over when
VMs should be migrated, and (iii) VM acceptance poli-
cies to determine whether to participate in the bid for
hosting a given VM from a host facing either an over-
load or a resource usage imbalance.
The remaining sections of the paper are organized as fol-
lows. Section 2 formalizes the load-balancing problem from
a perspective of VM live migration. Section 3 describes
the agent-based Cloud architecture. Section 4 presents the
proposed agent-based load balancing mechanism. Section 5
provides experimentation and analysis of results. Section 6
presents a related work comparison, and Sect. 7 includes
some concluding remarks and future research directions.
2 Problem statement
Consider a Cloud data center whose state is evaluated at
discrete times t = 1, 2, . . . , T . These times can be evenly
distributed through the policy evaluation horizon T or based
on events (e.g., VM (de)allocations). The problem consists
of finding a policy that balances workload across the hosts
in a Cloud data center while minimizing the number of VM
migrations. The policy determines (i) when to migrate a VM
and (ii) where to migrate the VM to. Therefore, the problem
can be represented using the following mathematical model:
Indices
i: index of VMs in the Cloud data center (i = 1, 2, . . . , n)
j: index of hosts in the Cloud data center ( j = 1, 2, . . . , m)
t: index of migration decision time points (t = 1, 2, . . . , T )
Parameters
αi : Number of cores required by VMi
αit% : Percentage of CPU usage of VMi at time t
Cluster Comput

βi : Number of gigabytes of memory required by VMi of VMs change dynamically over time, which increases the
βit% : Percentage of memory usage of VMi at time t complexity of the problem.
C jt : Available number of cores of host j at time t
M jt : Available memory (expressed in gigabytes) of host j
at time t 3 Agent-based load balancing architecture

Decision variable The agent-based architecture for load balancing (Fig. 1) is

 composed of virtual machine agents, server manager agents,
1 if VMi migrates to host j at time t and a front-end agent.
xi jt =
0 otherwise Virtual machine agents (VMAs) are in charge of (i) mon-
itoring VM resource usages, and (ii) sending monitoring
Objective functions reports containing the percentages of CPU usage and mem-
   ory usage to server manager agents (which manage hosts).
Min f T, αi , αit% , βi , βit% , xi jt , C jt , M jt (1) VMAs are designed to be deployed on VMs.
Server manager agents (SMAs) manage hosts and collab-
orate among themselves to evenly distribute loads across a

n 
m 
T
set of potentially heterogeneous hosts by using a distributed
Min xi jt (2)
problem solving technique, namely the multi-agent proto-
i=1 j=1 t=1
col for load balancing (see Sect. 4.5), which is supported by
Subject to VM live migration. SMAs trigger VM migrations based on
a set of server-centric migration policies (see Sect. 4.1), e.g.,

m
workload thresholds. In doing so, an SMA, deployed on a
xi jt ≤ 1 ∀i ∀t (3)
commodity host, can be configured by a data center adminis-
j=1
trator to trigger VM migrations whenever the performance of
the host deteriorates. SMAs are endowed with a set of migra-

n
tion heuristics to determine both (i) the destination host to
αi xi jt ≤ C jt ∀ j∀t (4)
i=1
which the VM can be migrated (Sect. 4.5) and (ii) the VM
to be migrated (Sect. 4.3). In addition, SMAs are provided

n
with VM acceptance policies (Sect. 4.2) to determine which
βi xi jt ≤ M jt ∀ j∀t (5) VMs (in the process of being migrated by other SMA) can
i=1
be hosted, e.g., an SMA with a current heavy CPU workload
cannot host a CPU-intensive VM, but it may host a memory-
xi jt ∈ {0, 1} (6)
intensive VM. Whenever an SMA is unable to migrate a VM
Equation (1) represents the objective of minimizing a mea- by collaborating with other SMAs (due to their VM accep-
sure related to the imbalance of resource usage calculated as tance policies), the SMA sends a call for assistance to the
the standard deviation of resource usages across hosts in a front-end agent to ask for a suggestion regarding where to
Cloud data center, which depends on the resource require- migrate a given VM, see Sect. 4.5 for details. SMAs are also
ments and resources usages of all the VMs and their current in charge of (i) allocating VMs in response to VM allocations
locations. Eq. (2) represents the objective of minimizing the received from the front-end agent, (ii) instantiating VMAs
total number of migrations during the policy evaluation hori- that monitor resource usages of VMs on hosts, (iii) monitor-
zon. ing resource usages of hosts by summarizing the monitoring
Constraint set in Eq. (3) implies that a VM can be migrated reports received from the VMAs, and (iv) replying to requests
only to one host or not being migrated at all. Constraint sets from the front-end agent about the overall resource usages of
in Eqs. (4) and (5) implies that a VM can be migrated only to hosts. SMAs are designed to be deployed on hosts in Cloud
a server with sufficient cores and memory to host it, respec- data centers, one SMA per host.
tively. Finally, constraint set in Eq. (6) represents the binary The front-end agent (FA) is in charge of routing VM allo-
nature of the decision variable. cation requests (from Cloud data center consumers) to SMAs
Even in a small Cloud data center, this problem may based on a set of load balancing heuristics (Sect. 4.4). When-
become intractable because of the large number of decision ever a VM allocation request arrives, the FA samples current
variables, constraints, and non-linearity of the first objec- resource usages of hosts by broadcasting a resource informa-
tive function (Eq. 1). The mathematical model is similar to tion request to SMAs to determine the best server (according
the two-dimensional bin-packing problem [4], an NP-hard to a load balancing heuristic) to host the VM. In addition,
problem [13], with the difference that the resource usages whenever overloaded SMAs are unable to find a destination

123

Fig. 1 Agent-based load balancing architecture
host (due to their VM acceptance policies), by request of the
overloaded SMAs, the FA suggests an SMA that can host the
VM. If no host has sufficient computing resources, the FA
suggests keeping the VM (see Sect. 4.5). The FA is designed
to be deployed on front-end servers of Cloud data centers.
4 Agent-based load balancing
The agent-based load balancing mechanism is supported by:
(1) VM migration policies to enable SMAs to establish pref-
erences over when VMs should be migrated to other
SMAs (Sect. 4.1).
(2) VM acceptance policies to enable SMAs to establish
preferences over which VMs should be hosted/accepted
from other (overloaded) SMAs (Sect. 4.2).
(3) VM migration heuristics to enable SMAs to select VMs
to be migrated when the SMAs are overloaded (Sect.
4.3).
123
Cluster Comput
(4) A set of load balancing heuristics of the FA to select the
initial hosts of VMs (Sect. 4.4).
(5) A distributed problem solving technique to evenly dis-
tribute loads across a set of hosts in a collaborative
manner (Sect. 4.5).
4.1 VM migration policies
A VM migration policy indicates when to trigger a VM
migration from one SMA to another SMA. A VM migration
policy consists of a migration threshold and an activation
threshold.
A migration threshold indicates the resource usage level
at which a host degrades its performance. Then, VM migra-
tion thresholds are used by SMAs to trigger VM migrations
whenever a certain level of resource usage is reached by
their corresponding hosts on which SMAs are deployed on.
SMAs are provided with both CPU-based and memory-based
migration thresholds. SMAs can enable either migration
Cluster Comput
threshold or both of them simultaneously. In addition, if
both migration thresholds are enabled, SMAs can be con-
figured to trigger VM migrations when either any migration
threshold is violated or when both migration thresholds
are violated simultaneously. If only one migration thresh-
old, either the CPU-based or the memory-based migration
threshold, is enabled, a VM migration will be triggered when-
ever either the CPU-based or the memory-based migration
threshold is violated, respectively. Configuring SMAs to trig-
ger VM migrations according to either a single migration
threshold or both migration thresholds should be based on
the specific characteristics of the hosts where SMAs are
deployed on. For instance, the performance of a commod-
ity host may considerably deteriorate only when the CPU
usage is relatively high; as a consequence, its SMA may be
configured to migrate VMs using the CPU-based migration
threshold.
An activation threshold prevents SMAs from migrating
VMs unnecessarily in the presence of abrupt but short-lived
resource usage peaks. The activation threshold indicates the
maximum number of accumulated times (based on the mon-
itoring rate of hosts) that resource usages can be above the
migration thresholds before attempting to migrate VMs as in
[32].
Both the migration threshold and the activation thresh-
old are decided by a data center administrator based on the
computing capabilities of each host as in Red Hat [31]. For
instance, a CPU-intensive host may be configured with a rel-
atively high threshold on CPU usage, while a commodity
host may be configured with a relatively low threshold on
CPU usage.
4.2 VM acceptance policies
A VM acceptance policy indicates which VMs (based on
their resource usages) should be accepted from other over-
loaded SMAs in the process of conducting collaborative load
balancing among SMAs by using VM live migration.
A VM acceptance policy consists of (i) a resource usage
level, (ii) an associated resource usage type either CPU or
memory, and (iii) a category either above or below a certain
resource usage level. The resource usage level indicates the
maximum (when the category of the VM acceptance pol-
icy is below) or the minimum (when the category of the
VM acceptance policy is above) resource usage level that
a VM may have in order be hosted in a host managed by
an SMA. Then, VM acceptance policies are used by SMAs
(during the execution of the multi-agent protocol for load
balancing) to determine whether to bid for hosting a given
VM, see Sect. 4.5 for details. SMAs are provided with both
CPU-based and memory-based VM acceptance thresholds.
By using independent VM acceptance thresholds, SMAs can
be configured to consolidate or avoid either CPU or memory
load. For instance, an SMA managing a CPU-intensive host
may consolidate VMs running compute-intensive applica-
tions with low memory requirements (e.g., applications for
texture sampling [21]) by setting the VM acceptance poli-
cies with a memory resource level below a certain level, e.g.,
30 %, and a CPU usage level above a certain level, e.g., 60 %,
i.e., the SMA would only accept VMs with a low memory
usage and a high CPU usage.
The resource usage level of a VM acceptance policy indi-
cates the resource usage profile of the VMs that should be
accepted by a given SMA according to the profile of its host,
e.g., a CPU-intensive server may be better suited to host
VMs running compute-intensive applications. In this work,
a data center administrator should determine the resource
usage levels of the VM acceptance policies of hosts. How-
ever, dynamic profiling of hosts (like in [33]) can be used
to automatically determine the resource usage level of VM
acceptance policies, i.e., what type of VMs with respect to
resource usage should be hosted in a given physical server.
Nevertheless, profiling hardware infrastructure of a Cloud
data center is out of the scope of this paper.
4.3 VM migration heuristics
A VM migration heuristic guides (overloaded) SMAs to
select VMs to be migrated to other (underloaded) SMAs by
ranking VMs based on their resource usages. An SMA makes
use of its migration heuristics whenever VM migration poli-
cies (Sect. 4.1) are violated, i.e., when both the migration
threshold and the activation threshold are violated.
SMAs are provided with a set of nine migration heuristics
that make use of information from CPU and memory usages
to rank VMs. The VM with the highest rank (according to
the migration heuristic) is selected for migration. The set of
nine VM migration heuristics are as follows:
{κ:CPU-μ:MEM} ∪ {κ:CPU} ∪ {μ:MEM} ∪ {R}
where CPU indicates that the migration heuristic takes into
account the ranking of CPU usage, MEM indicates that the
migration heuristic takes into account the ranking of memory
usage, R indicates that VMs are randomly selected, κ and
μ ∈ {H, L} and define whether the VM with the highest
(H) or the lowest (L) CPU usage and memory usage receives
the highest weight for migration, respectively. For instance,
an SMA adopting an L:CPU-H:MEM migration heuristic
selects the VM with a combination of the lowest ranking of
CPU usage and the highest ranking of memory usage.
An SMA adopting a κ:C PU -μ:MEM migration heuristic
selects the VM to be migrated based on the algorithm shown
in Fig. 2. In this algorithm, the SMA sorts VMs based on CPU
usage in ascending or descending order (line 1), as indicated
by parameter κ, and assigns them a weight according to their
123

Fig. 2 Selection of VMs for migration
CPU rank order (line 2). Then, the SMA sorts VMs again, but
this time based on memory usage in ascending or descend-
ing order (line 3), as indicated by parameter μ, and assigns
them a weight according to their memory rank order (line
4). Afterwards, the SMA sums both CPU and memory rank
orders (line 5) and selects the VM with the highest overall
rank order (line 6).
An SMA adopting a VM migration heuristic based on the
usage of a single resource either κ:CPU or μ:MEM uses a
reduced version of the algorithm shown in Fig. 2. In order
to select a VM based on κ:CPU, the SMA sorts VMs based
on CPU usage in ascending or descending order (line 1), as
indicated by parameter κ, and assigns them a weight accord-
ing to their CPU rank order (line 2). Then, the SMA selects
the VM with the highest value of CPU Rank Or der . In a similar
manner, in order to select a VM based on μ:MEM, the SMA
sorts VMs based on memory usage in ascending or descend-
ing order (line 3), as indicated by parameter μ, and assigns
them a weight according to their memory rank order (line
4). Then, the SMA selects the VM with the highest value of
MEM Rank Or der .
An SMA using VM migration heuristics that assign high
overall rank orders to VMs with either low CPU usage
(L:CPU) and/or low memory usage (L:MEM) aims to find
more potential SMAs that could accept a given VM (based on
their VM acceptance policies) when conducting collabora-
tive load balancing. An SMA using VM migration heuristics
that assign high overall rank orders to VMs with either high
CPU usage (H:CPU) and/or high memory usage (H:MEM)
aims to quickly reduce its workload while reducing the num-
ber of VM migrations (at least from its perspective).
123
Cluster Comput
4.4 Load balancing heuristics of FAs
A total of four commonly used load balancing heuristics [7,
24,40,49] were implement into FAs to determine the initial
destination hosts of newly arrived VMs:
(1) Round-robin heuristic consisting of allocating newly
arrived VMs evenly across a pool of hosts by assigning
each VM to the next available host of the pool. See [49]
for an example of a round-robin load balancing heuristic.
(2) Greedy heuristic consisting of allocating newly arrived
VMs to the host with the smallest sum of percentage of
CPU usage and percentage of memory usage. See [7] for
an example of a greedy load balancing heuristic.
(3) First-fit heuristic consisting of allocating a newly arrived
VM to the first host of the pool with sufficient available
resources to meet the VM’s requirements. See [24] for
an example of a first-fit load balancing heuristic.
(4) Best-fit heuristic consisting of allocating a newly arrived
VM to the host with the fewest number of available cores.
The number of available cores was selected over the
amount of free memory because virtualized hosts may
assign considerably more (virtual) memory than their
available physical memory to VMs [35]. See [40] for an
example of a best-fit load balancing heuristic.
4.5 Multi-agent protocol for load balancing (MapLoad)
The Multi-Agent Protocol for Load balancing (MapLoad)
adopted by agents has three roles: initiator, participant, and
facilitator (see Figs. 3, 4). Whenever an overloaded SMA, i.e.,
Cluster Comput
Fig. 3 Multi-agent protocol for
load balancing (MapLoad)
an SMA whose resource usage violates a migration threshold,
triggers a VM migration, the overloaded SMA playing the
initiator role broadcasts a call for proposals containing the
description of the VM (that the initiator SMA is trying to
migrate) to n participant SMAs (Fig. 3), i.e., SMAs within
the same Cloud data center, in order to look for potential
SMAs able to host the VM. The participant SMAs may reply
with (i) a propose message containing their current resource
usage information, meaning they can host the VM or (ii) a
refuse message (Fig. 3) meaning they cannot host the VM at
least without compromising their migration thresholds.
In order to evaluate the received k propose messages from
the participant SMAs, the initiator SMA is endowed with a
greedy proposal selection mechanism. The proposals con-
taining CPU and memory usage information are evaluated
according to the migration thresholds enabled (Sect. 4.1). If
the initiator SMA makes use of only the CPU-based migra-
tion threshold, then the participant SMA of the proposal with
the lowest CPU usage is selected. In a similar manner, if the
initiator SMA makes use of only the memory-based migra-
tion threshold, then the participant SMA of the proposal with
the lowest memory usage is selected. In addition, if the initia-
tor SMA simultaneously makes use of both, the CPU-based
and the memory-based migration thresholds, then the par-
ticipant SMA of the proposal with the smallest sum of both
CPU and memory usage is selected. By adopting the greedy
proposal selection mechanism, whenever an overloaded, ini-
tiator SMA attempts to migrate a VM, the initiator SMA
selects the participant SMA that may be less likely to migrate
a VM based on the current resource usage information.
Once the initiator SMA has selected the best participant
SMA to migrate the VM based on its proposal evaluation
123

Fig. 4 Agent interaction
protocol for getting suggestions
from the facilitator
mechanism, the initiator SMA sends (i) an accept-proposal
message to the best participant SMA and (ii) k-1 reject-
proposal messages to the remaining participant SMAs (Fig.
3). Then, the selected participant SMA may reply with (i) a
confirm message that automatically reserves the computing
resources to host the VM or with (ii) a cancel message (Fig.
3) if the participant SMA no longer has sufficient comput-
ing resources due to either a sudden resource usage increase
or because the participant SMA has accepted to host other
VM as a result of a concurrent execution of MapLoad with
other initiator SMA. Once the initiator SMA receives a con-
firm message from the selected participant SMA, the initiator
SMA migrates the VM to the selected participant SMA and
sends an inform-done message if successful, or a failure mes-
sage otherwise (Fig. 3).
If after sending a call for proposals to k participant SMAs,
the initiator SMA receives k refuse messages from the par-
ticipant SMAs, i.e., if no participant SMA proposes itself
to host the VM, the initiator SMA sends a request message
containing the description of the VM to the FA (playing the
facilitator role of MapLoad) in order to get a suggestion about
a participant SMA able to host the VM (Fig. 3). The facilita-
tor FA broadcasts a request message to the participant SMAs
asking for their resource usage information (Fig. 4). Then,
the participant SMAs reply with an inform message contain-
ing their resource usage information (Fig. 4). Subsequently,
the facilitator FA, based on its load balancing heuristic (Sect.
123
Cluster Comput
4.4), selects the best participant SMA to migrate the VM and
sends a request message to the selected SMA to reserve the
computing resources to host the VM (Fig. 4), to which the
selected participant SMA may reply with (i) a confirm mes-
sage reserving the computing resources or with (ii) a failure
message (Fig. 4) when the participant SMA no longer has
sufficient computing resources due to the concurrent nature
of MapLoad. On the one hand, if the facilitator FA receives a
confirm message from the participant SMA, the FA sends an
inform message to the initiator SMA indicating the selected
participant SMA able to host the VM (Fig. 4). On the other
hand, if the facilitator FA receives either a failure message
from the selected participant SMA or the facilitator FA is
unable to find a participant SMA able to host the VM, the FA
sends an inform message containing a suggestion to keep the
VM to the initiator SMA (Fig. 4).
It should be noted that when the FA is contacted by an
overloaded SMA (looking for a recommendation for a poten-
tial target host of a VM), the overloaded (initiator) SMA
already have tried to migrate the VM to other (participant)
SMAs, which sent refuse messages (to the initiator SMA)
meaning (i) they are unable to host the VM due to insuffi-
cient resources or (ii) their migration thresholds would be
compromised by hosting the VM. If no participant SMA
has sufficient resources to host the VM, the FA instructs the
initiator SMA to keep the VM. However, if some partici-
pant SMAs sent refuse messages because hosting the VM
Cluster Comput
would have compromised their migration thresholds, the FA
(based on its load balancing heuristics) may determine that it
is better to migrate the VM to a given participant SMA even
compromising its migration thresholds than keeping the VM
in the initiator SMA. In such a case, the FA instructs the
selected participant SMA (which is assumed to be collab-
orative) to allocate resources to host the VM and sends an
inform message to the initiator SMA indicating the selected
participant SMA. It is worth mentioning that by hosting the
VM, the selected participant SMA may suddenly violate their
migration thresholds. Nevertheless, this does not result in an
immediate VM migration due to the activation threshold that
prevents the SMA from migrating VMs unnecessarily in the
presence of abrupt but short-lived resource usage peaks as
stated in Sect. 4.1. Moreover, if the resource usage of the
selected participant SMA decreases, the SMA may not have
to migrate the newly arrived VM or other VM. Finally, it
should be noted that overloaded SMAs contact the FA only
when the Cloud data center is completely overloaded, i.e.,
when the resource usages of all the SMAs are above or near
their migration thresholds.
It is noteworthy that agents, namely SMAs and the FA,
are capable of concurrently handling multiple instances of
MapLoad.
4.6 Message complexity: MapLoad versus Red Hat
A performance measure of distributed problem solving
techniques is the number of messages exchanged by the
underlying distributed algorithm in the worst case scenario
[15].
The number of messages exchanged by MapLoad to
migrate a VM depends on the number of participant SMAs
whose resource usages are sampled. In the worst case sce-
nario (assuming that the resource usages of all the participant
SMAs in a Cloud data center are sampled), the number of
messages exchanged by agents adopting MapLoad can be
calculated as follows.
(1) Firstly, the initiator SMA sends a call for proposals (to
host a VM) to n − 1 participant SMAs, i.e., all the SMAs
in a Cloud data center except the initiator SMA. Number
of messages exchanged: n − 1.
(2) Assuming that this is the worst case scenario, the partic-
ipant SMAs reply with refuse messages meaning their
migration thresholds would be compromised by hosting
the VM. Number of messages exchanged: n − 1.
(3) In response, the initiator SMA sends a request message
(asking for a recommendation) to a FA playing the facili-
tator role of MapLoad. Number of messages exchanged:
1.
(4) Next, the FA sends n − 1 request messages, one for each
participant SMA, to sample their resource usages. Num-
ber of messages exchanged: n − 1.
(5) In response, all the participant SMAs reply with an
inform message containing their resource usages. Num-
ber of messages exchanged: n − 1.
(6) Afterwards, the FA sends a request message to the
selected participant SMA with the lowest resource usage,
which replies with a confirm message meaning it has
allocated sufficient resources to host the VM. Number
of messages exchanged: 2.
(7) Finally, the FA sends an inform message (indicating the
selected participant SMA) to the initiator SMA, which,
in response, migrates the VM and sends an inform-done
message to the selected participant SMA. Number of
messages exchanged: 2.
Thus, in the worst case scenario, the number of mes-
sages exchanged by the underlying distributed algorithm of
MapLoad is bounded by Eq. (7).
(n − 1) + (n − 1) + 1 + (n − 1) + (n − 1) + 2 + 2
= 4(n − 1) + 5 = 4n + 1 messages (7)
As n tends to infinity, the worst-case message complexity of
MapLoad is O(n). In contrast, in a centralized load balanc-
ing mechanism such as the Red Hat Enterprise Virtualization
Suite [32], a central manager has to monitor resource usages
of hosts (see [30]) either to (i) initially allocate VMs based
on the current load of the hosts or to (ii) balance the load of
the Cloud data center based on VM live migration. Then, the
central manager of Red Hat is periodically receiving mes-
sages from the hosts according to a monitoring frequency.
Thus, the overall number of messages exchanged among the
central manager and the hosts is (at least) bounded by mn
messages where m and n stand for monitoring frequency and
number of hosts, respectively. As m and n tend to infinity, the
worst-case message complexity of Red Hat is O(n 2 ). More-
over, it is noteworthy that Red Hat periodically exchanges
messages regardless whether it needs to migrate VMs or not,
while agents adopting MapLoad exchange messages only
when (i) an overloaded SMA needs to migrate a VM or (ii)
the FA needs to initially allocate a VM.
5 Evaluation and empirical results
Three series of experiments were conducted for evaluat-
ing the performance of agents adopting MapLoad using the
agent-based testbed described in Sects. 3 and 4. The testbed
was implemented in java and the JADE agent platform [5].
In the agent-based testbed, each SMA controls a JADE agent
123

container, i.e., an instance of a JADE runtime environment
using a unique network port. VM migration was simulated
by means of live agent migration between JADE agent con-
tainers. The experiments were conducted using a computer
with the following specifications: Intel Core 2 Duo E8400,
3.00 GHz, 4 GB RAM, with a 32-bit Windows 7 Enterprise
OS, SP1.
5.1 Agent-based load balancing versus centralized load
balancing
Objective The experiment was designed to evaluate and com-
pare the efficiency of SMAs adopting MapLoad (with and
without initial allocation assistance from FAs) and a set of
commonly used benchmark heuristics [7,24,40,49] in bal-
ancing loads in heterogeneous Cloud data centers.
Benchmark With the objective of comparing and vali-
dating the empirical results, a centralized load balancing
approach (where a single central entity decides and selects
hosts for allocating VMs) was implemented. The central
entity was endowed with the same four commonly used load
balancing heuristics implemented in FAs, namely, round-
robin, greedy, first-fit, and best-fit heuristics (see Sect. 4.4).
It should be noted that the set of commonly used benchmark
heuristics [7,24,40,49] does not make use of live migration
of VMs, and thus, once the central manager has allocated a
VM to a host, the VM is not migrated. The architecture of
the centralized approach consists of a pool of hosts (within a
Cloud data center), and a central manager that selects the host
to which VM allocation requests should be routed based on
a commonly used load balancing heuristic. The central man-
ager was programmed to continuously monitoring resource
usages of every host, and thus, the manager had complete
knowledge of CPU and memory usages of every host within
its Cloud data center.
To focus the experiment on evaluating the effectiveness of
MapLoad with respect to the centralized approach in balanc-
ing Cloud data centers, a FA was programmed to route newly
arrived VMs to a randomly determined host with sufficient
available resources to meet VMs’ requirements, e.g., number
of cores, so as to SMAs had to balance the load without initial
allocation assistance from a FA. However, whenever SMAs
adopted MapLoad and received no proposals for hosting a
VM, as indicated by MapLoad (see Sect. 4.5), SMAs con-
tacted the FA for a host suggestion to which migrate the VM
to, and if there was a set of SMAs with sufficient resources,
the FA randomly selected an SMA to host the VM, and sent
the suggestion to the SMA adopting the initiator role. In addi-
tion, to explore the efficiency of SMAs adopting MapLoad
fully assisted by FAs, FAs and the central manager were pro-
vided with the same four commonly used load balancing
heuristics.
123
Cluster Comput
Table 1 Load-related input data for Experiments 5.1, 5.2, and 5.3
Input parameter Possible values
Number of VMs 300
Specifications of No. of cores {1, 2, 4, 6, 8}
VMs Memory sizes {1, 2, 3, 4, 5, 6, 7, 8, 9, 10,
11, 12, 13, 14, 15, 16,
17, 18, 19, 20} GBs
CPU usage profiles {LowC PU (50 %),
MediumC PU (75 %),
HighC PU (100 %)}
Memory usage profiles {Low M E M (50 %),
Medium M E M (75 %),
High M E M (100 %)}
Mean inter-arrival time of VMs 500 ms
Mean inter-departure time of VMs 35,000 ms
A centralized load balancing approach was selected for
comparison because, firstly, centralized approaches are com-
monly used [7,24,40,49], and secondly, because comparing
a centralized load balancing approach with agents adopting
MapLoad is a challenge. For instance, whereas centralized
approaches (i) have full knowledge of the Cloud data center,
e.g., a central manager knows the number of hosts and their
latest resource usages [32], and (ii) all the hosts in the Cloud
data center are fully controlled by a central manager such as
in [32], agents adopting MapLoad (i) have a partial view of
the Cloud data center, e.g., an SMA may not be aware of the
number of hosts and/or other SMAs’ resource usages, and (ii)
agents (in this work) should dynamically coordinate among
themselves to balance a load across a set of hosts.
Experimental settings The testbed has three different sets
of input parameters: (i) Cloud data center’s input data, (ii)
load-related input data (Table 1), and (iii) agent-based testbed
parameters (Table 2).
The simulated Cloud data center consisted of 6 hosts with
20 cores and 48 GBs, 8 hosts with 40 cores and 96 GBs, and
6 hosts with 60 cores and 144 GBs. In doing so, the hetero-
geneous Cloud data center was composed of an arbitrarily
distributed number of relatively weak hosts, regular hosts,
and relatively powerful hosts. The host specifications were
based on [11]. The number of hosts (simulated by SMAs)
was set to 20 to have sufficient hosts to handle the amount of
load described in Table 1.
Load-related input data (Table 1) consists of number of
VMs (simulated by VMAs), specifications of VMs, CPU and
memory usage profiles of VMs, and mean inter-arrival and
mean inter-departure time of VMs. The number of VMs was
set to 300 based on [44] to explore the efficiency of MapLoad
using a larger number of VMs than the maximum number of
VMs that can be successfully instantiated (at the same time)
using existing open source frameworks for Cloud data cen-
ters, e.g., OpenNebula can successfully instantiate (at the
Cluster Comput

Table 2 Agent-based testbed’s input data for Experiments 5.1, 5.2, and 5.3
Input parameter Possible values

Experiment 5.1 Experiment 5.2 Experiment 5.3

FA’s load balancing heuristics {Round-robin, greedy, Greedy heuristic None

first-fit, best-fit} heuristics
SMAs’ proposal selection Greedy selection
mechanism
SMAs’ CPU-based acceptance Above
policy
SMAs’ memory-based acceptance Above
policy
SMAs’ CPU acceptance threshold 0%
SMAs’ memory acceptance 0%
threshold
SMAs’ VM migration heuristics H:CPU-H:MEM heuristic {H:CPU-H:MEM, H:CPU-L:MEM,
L:CPU-H:MEM, L:CPU-L:MEM,
H:CPU, L:CPU, H:MEM, L:MEM, R}
heuristics
SMAs’ CPU-based migration 50 %
threshold
SMAs’ memory-based migration 50 %
threshold
SMAs’ activation threshold 300 threshold violations
SMAs’ monitoring frequency 100 ms

same time) up to 148 VMs (see [44, p. 740]). In order to
create a load consisting of heterogeneous VMs, the number
of logical cores and memory size were randomly assigned
from the possible values reported in Table 1. In addition, to
create asymmetrical, heterogeneous resource usage profiles
among the VMs, the resource usage profiles of VMs were
randomly assigned from nine predefined resource usage pro-
files resulting from {LowC PU (50 %), MediumC PU (75 %),
HighC PU (100 %)} x {Low M E M (50 %), Medium M E M
(75 %), High M E M (100 %)} uniformly distributed. CPU and
memory usage of VMs were modeled using first-order
autoregressive random processes [27] to correlate resource
usage over time, similarly as in [46]. The first-order autore-
gressive process has a long-term mean that was used to
simulate random resource usage levels that, in the long-term,
tended to 50 %, 75 %, and 100 %, for low, medium, and high
resource usage, respectively. The mean inter-arrival time and
the mean inter-departure time of VMs were set to 500 and
35,000 ms, respectively. Both the mean inter-arrival time and
the mean inter-departure time of VMs were determined based
on an experimental tuning to increasingly overload the Cloud
data center to its maximum capacity to host VMs, and thus,
(i) forcing SMAs to migrate VMs, and (ii) having VMs with
a relatively long lifetime such that they may migrate several
times.
The input parameters of the agent-based testbed (Table
2) consist of the FA’s load balancing heuristics and SMAs’
parameters: VM acceptance policies, VM migration heuris-
tics, VM migration thresholds, an activation threshold, and a
monitoring frequency.
SMAs’ VM acceptance policies were set to above for both
the CPU-based and the memory-based acceptance policies.
The CPU and the memory acceptance thresholds were set to
0 %. In doing so, SMAs were configured to respond to all
the calls for proposals (while SMAs have sufficient comput-
ing resources to host VMs) from other SMAs attempting to
migrate VMs.
Some of the benchmarking load balancing heuristics take
into account both CPU and memory usage, e.g., greedy and
first-fit heuristics. Then, to establish a comparison, SMAs
were provided with a combined migration heuristic, i.e., a
migration heuristic that takes into account both CPU and
memory usage, namely, an H:CPU-H:MEM heuristic, which
was selected because it keeps a relatively small number of
VM migrations while achieving a relatively good perfor-
mance balancing loads (see Sect. 5.3, for an evaluation of
the VM migration heuristics adopted by SMAs to balance
loads).
The CPU-based and memory-based migration thresh-
olds were set to 50 % to (i) equally trigger VM migrations
based on either CPU and memory usage, and to (ii) activate
MapLoad in the presence of a relatively heavy load (greater
than a 50 % of SMAs’ computing resources). In addition,
the activation threshold of SMAs (i.e., the amount of time

123
 Cluster Comput

SMAs are allowed to have computing resource usages above

their migration thresholds) was set to 300 threshold viola-
tions. Since the SMAs’ monitoring frequency of computing
resources was set to 100 ms, SMAs attempted to migrate
VMs after having computing resource usages above their
migration thresholds for 30 s. SMAs were able to determine
300 consecutive threshold violations after 30 s by monitoring
their computing resources every 100 ms. In doing so, SMAs
disregarded abrupt but short-lived resource usage peaks. Pro-
vided that (i) the activation threshold of SMAs corresponds
to 3 % of the overall average length of the experiment runs
(1019.1 s), and (ii) SMAs were increasingly overloaded, but
subsequently followed by a gradual load decline, SMAs were
not constantly migrating VMs from the beginning to the end
of experiment runs, i.e., SMAs migrated VMs only when
their migration thresholds were continuously exceeded for
an amount of time determined by the VM activation thresh-
old. Hence, the value of the activation threshold was set to
300 based on an experimental tuning in order to disregard
abrupt but short-lived resource usage peaks, but at the same
time, forcing SMAs to migrate VMs.
For each configuration of the testbed, 10 independent
experiment runs were conducted for a total of 90 simula-
tions, which involved instantiating 27,000 VMAs.
Performance measure The performance measure used to
compare the efficiency of the centralized load balancing
approach and agents adopting MapLoad is average sample
standard deviation of percentage of resource usage, which
Fig. 5 Load balancing comparison of only SMAs, FAs & SMAs, and
is calculated as the arithmetic mean of the standard devia- central managers
tions of nresource usage samples expressed as percentages. A
resource usage sample is the sum of CPU usage and memory
usage. The sum of CPU usage and memory usage was used Analysis As shown in Fig. 5, SMAs endowed with
because both SMAs and the central manager had to balance MapLoad attained a lower average sample standard devia-
a heterogeneous load composed of VMs with asymmetri- tion of percentage of resource usage (Fig. 5, 1st column)
cal resource usage profiles. The lower the average sample than central managers adopting commonly used load bal-
standard deviation of percentage of resource usage, the bet- ancing heuristics (Fig. 5, 2nd, 4th, 6th, and 8th columns),
ter balanced a Cloud data center is, i.e., the percentage of namely, best-fit, first-fit, greedy, and round-robin load bal-
computing resource usages is uniform across (potentially ancing heuristics. SMAs distributed the load across hetero-
heterogeneous) hosts. Provided that the central manager can- geneous hosts more evenly than central managers because
not migrate VMs, the performance of both agents adopting whereas central managers routed newly arrived VM allo-
MapLoad and the central manager cannot be compared using cation requests to hosts based on (i) the resource usages
the number of VM migrations. See Sect. 5.2 for a comparison of hosts at that particular time and on (ii) the VMs’ initial
between SMAs adopting MapLoad and the Red Hat Enter- resource requirements, SMAs, through distributed collabo-
prise Virtualization Suite (that balances loads based on live ration, dynamically migrated VMs based on their continu-
migration of VMs) with respect to the number of VM migra- ously changing resource usages to the most suitable hosts
tions. (according to the SMAs’ migration heuristic). In addition,
Results Experimental results are shown in Fig. 5. From overloaded SMAs (adopting the initiator role of MapLoad)
these results, three observations are drawn. autonomously migrated VMs to other SMAs (adopting the
Observation 1 SMAs endowed with MapLoad (without participant role of MapLoad) that replied to the call for pro-
initial allocation assistance from FAs) were capable of dis- posals to host a given VM. The participant SMAs replied
tributing loads across heterogeneous hosts more evenly than to call for proposals only if hosting the VM did not violate
central managers endowed with commonly used load balanc- their migration thresholds. In doing so, participant SMAs
ing heuristics. autonomously proposed themselves as potential hosts. More-

123
Cluster Comput
over, given that participant SMAs’ proposals consisted of
their resource usages, the initiator SMA was able to select
the participant SMA with the lowest resource usage.
Equipped with partial knowledge of a Cloud data center,
through collaboration, SMAs can autonomously and dynam-
ically balance loads across a set of heterogeneous hosts
outperforming central managers endowed with commonly
used load balancing heuristics with full knowledge of the
Cloud data center.
Observation 2 For each load balancing heuristic: round-
robin, greedy, first-fit, and best-fit load balancing heuristics,
SMAs with initial allocation assistance from FAs and using
MapLoad were capable of distributing loads across hetero-
geneous hosts more evenly than central managers using the
same load balancing heuristics as FAs.
Analysis SMAs receiving assistance from FAs by ini-
tially allocating VMs (Fig. 5, 3rd, 5th, 7th, and 9th columns)
outperformed central managers (Fig. 5, 2nd, 4th, 6th, and
8th columns) in uniformly balancing loads across a pool
of heterogeneous hosts. Since both FAs and central man-
agers were provided with the same load balancing heuristics:
best-fit, first-fit, greedy, and round-robin, SMAs assisted by
FAs attained lower average sample standard deviation of per-
centage of resource usage than central managers because (i)
SMAs benefited from the initial allocation of VMs (as central
managers did), and (ii) SMAs also balanced loads by migrat-
ing VMs taking into account dynamic fluctuations of VMs’
resource usages.
Whereas front-end load balancers, either FAs or central
managers, may help to balance loads based on (i) static VMs’
computing requirements and (ii) hosts’ resource usages at
the moment of VM arrivals, balancing loads also taking
into account continuously changing resource usages of VMs
(as done by SMAs) can achieve a more uniformly balanced
Cloud data center.
Observation 3 Through collaboration, SMAs and FAs
(using the round-robin load balancing heuristic) achieved the
most balanced Cloud data centers.
Analysis As observed in Fig. 5, the best (i.e., the lowest)
average sample standard deviation of percentage of resource
usage was achieved by SMAs adopting MapLoad assisted
by FAs using the round-robin heuristic (Fig. 5, 9th column)
while the second best balanced load was achieved by SMAs
assisted by FAs using the greedy heuristic (Fig. 5, 7th col-
umn). In addition, the third lowest average sample standard
deviation of percentage of resource usage was achieved by
SMAs without assistance from FAs (Fig. 5, 1st column).
SMAs adopting MapLoad assisted by FAs using the
round-robin heuristic outperformed central managers and
other SMAs and FAs because (i) both the computing require-
ments and the resource usage profiles of VMs were assigned
randomly using a uniform distribution, and (ii) FAs allo-
cated newly arrived VMs uniformly (in a round-robin basis)
across the hosts. Thus, due to the characteristics of the
randomly generated workload consisting of uniformly dis-
tributed resource usage profiles and computing requirements,
FAs (by themselves) by uniformly allocating VMs attained
a relatively evenly balanced Cloud data center. This state-
ment is confirmed by the overall fifth best balanced Cloud
data center achieved by central managers using a round-robin
heuristic (Fig. 5, 8th column). In addition, as explained in
Observation 2 (Sect. 5.1), SMAs may benefit from the initial
allocation of VMs by FAs, and as explained in Observation
1 (Sect. 5.1), by dynamically taking into account dynamic
fluctuations of VMs’ resource usages, SMAs are capable of
balancing loads more evenly than front-end balancers.
Combining SMAs (endowed with MapLoad) with FAs
using a suitable load balancing heuristic according to the
characteristics of the workload results in the most evenly
balanced Cloud data center.
5.2 MapLoad versus Red Hat
Objective The experiment was designed to evaluate the
performance of SMAs adopting MapLoad (with initial allo-
cation assistance from FAs) regarding the number of VM
migrations.
Benchmark The load balancing mechanism adopted by
the Red Hat Enterprise Virtualization Suite [32] was imple-
mented and selected for comparison because (i) it balances
loads based on VM live migration (as in this work), and (ii)
whenever a VM allocation request arrives, Red Hat selects
the host to allocate the VM based on the current load of the
hosts (as in this work). In addition, the load balancing mech-
anism adopted by the Red Hat Enterprise Virtualization Suite
achieved record-setting benchmark results [38].
Experimental settings The testbed has four different sets
of input parameters: (i) Cloud data center’s input data, (ii)
load-related input data (Table 1), (iii) agent-based testbed
parameters (Table 2), and (iv) the Red Hat Enterprise Virtu-
alization Manager’s input data (Table 3).
The input parameters of the load balancing mechanism
adopted by the Red Hat Manager (Table 3) consist of a front-
end load balancing heuristic, a VM migration heuristic, a
maximum service level, an activation threshold, and a mon-
itoring frequency of resource usage.
The Red Hat Enterprise Virtualization Suite is equipped
with a manager (a central entity) that selects the host for allo-
cating new VMs. In addition, whenever workload balancing
is required, the Red Hat Manager decides and selects both
the VM to be migrated and the host for allocating the VM.
The front-end load balancing heuristic adopted by the Red
Hat Manager [32] to determine the host for allocating a new
VM is even distribution [29], which selects the host with
the lowest CPU usage to allocate the VM. The migration
heuristic adopted by the Red Hat Manager [29] consists of
123
 Cluster Comput

Table 3 Input data of the Red

Hat Enterprise Virtualization
Manager’s load balancing
mechanism
Input parameter Possible values
Load balancing heuristic Even distribution. The manager selects the host with
the lowest CPU usage for allocating both newly
arrived VMs and VMs migrated from overloaded
hosts.
VM migration heuristic Lowest CPU usage heuristic
Maximum service level per host 50 % of CPU usage
Activation threshold 300 threshold violations
Monitoring frequency of resource usage 100 ms

migrating VMs with the lowest CPU usage to the host with
the lowest CPU usage. The Red Hat Manager’s activation
threshold is the length of time that a host is allowed to con-
tinue above its maximum service level. Then, to establish
a comparison, the activation threshold and the monitoring
frequency of resource usage were set equally for both the
Red Hat Manager (Table 3) and the agent-based testbed. In
addition, the maximum service level (i.e., the CPU-based
migration threshold) per host of the Red Hat Manager and
SMAs were set both to 50 % to establish a comparison.
For each configuration of the testbed, 10 independent
experiment runs were conducted for a total of 20 simula-
tions, which involved instantiating 6000 VMAs.
Performance measures The performance measures used
to compare the efficiency of the load balancing mechanism
adopted by the Red Hat Enterprise Virtualization Suite [32]
and agents adopting MapLoad were (i) average number of
VM migrations per host, and (ii) average sample standard
deviation of percentage of CPU usage.
The average number of VM migrations per host is calcu-
lated as follows.

n
mi
i=1
m̄ =
n
where m i stands for total number of migrations from host i
during a simulation run, and n stands for number of hosts
in the Cloud data center. The smaller the average number of
VM migrations per host, the lower the overhead of the load
balancing mechanism, and the better the performance. Fig. 6 Load balancing results: MapLoad versus Red Hat
The average sample standard deviation of percentage of
CPU usage was used as a performance measure because the
load balancing mechanism adopted by the Red Hat Enterprise of the Red Hat Enterprise Virtualization Suite dominated the
Virtualization Suite [32] balances loads based only on CPU other when taking into account both (i) the average number of
usage. The lower the average sample standard deviation of VM migrations per host and (ii) the average sample standard
percentage of CPU usage, the better balanced a Cloud data deviation of percentage of CPU usage.
center is (regarding CPU usage). Analysis As shown in Fig. 6, whereas agents adopting
Results Experimental results are shown in Fig. 6. From MapLoad achieved a lower average number of VM migra-
these results, one observation is drawn. tions per host than the Red Hat manager, the Red Hat manager
Observation 1 Neither the load balancing mechanism of attained a lower average sample standard deviation of CPU
agents adopting MapLoad nor the load balancing mechanism usage than agents adopting MapLoad. Both SMAs and FAs

123
Cluster Comput
adopting MapLoad were capable of balancing the Cloud
data center with a relatively small difference of 1.74 % (in
comparison to Red Hat) for the average sample standard devi-
ation of CPU usage while migrating (9.45 %) fewer VMs
than the load balancing mechanism of the Red Hat Enter-
prise Virtualization Suite. This is because whenever SMAs
detected threshold violations, the initiator SMAs adopted
MapLoad with underloaded SMAs (playing the participant
role of MapLoad), i.e., SMAs with computing resource
usages below their migration thresholds. In addition, when-
ever no SMA replied to call-for-proposals messages to host
a VM, the initiator SMA sent a request to the FA to ask for
a suggestion for a specific host where to migrate the VM
to. If no host had sufficient computing resources to host the
VM, the FA replied with a suggestion to retain the VM. In
addition, whereas the Red Hat Manager’s migration heuris-
tic was based only on CPU usage, the migration heuristic of
SMAs was based on both CPU and memory usage simulta-
neously. Then, agents adopting MapLoad required that both
the CPU-based and the memory-based migration thresholds
were violated in order to trigger the migration of a VM. This
suggests that the SMAs’ migration thresholds were violated
fewer times than the Red Hat Manager’s migration thresh-
olds, and as a consequence, the Red Hat Manager migrated
more VMs. By migrating more VMs and at the expense
of overhead, the load balancing mechanism of the Red Hat
manager attained a slightly more balanced Cloud data center
with a relatively small difference of 1.74 % (in comparison
to agents adopting MapLoad) for the average sample stan-
dard deviation of CPU usage. This empirical evidence also
suggests that for both, agents adopting MapLoad and the
Red Hat Manager, there is a tradeoff between the number of
VM migrations and the effectiveness in balancing Cloud data
centers.
The load balancing mechanism of the Red Hat Manager
achieved lower average sample standard deviation of CPU
usage (than agents adopting MapLoad) because (i) the Red
Hat Manager continuously monitored the performance of
all the hosts and had full knowledge of the hosts’ resource
usages, and (ii) the even distribution load balancing heuris-
tic outperformed the H:CPU-H:MEM heuristic adopted by
SMAs. In addition, as indicated in [31], when the Red Hat
Manager migrates VMs from a host to another (destination)
host, if the destination host’s CPU usage suddenly crossed
its CPU usage threshold, the VMs (recently attached to the
host) did not start until the host’s CPU usage decreased below
its threshold. Every monitoring interval, the Red Hat Man-
ager verified whether the host was able to start the asleep
VMs, and if it was the case, the Red Hat manager started the
VMs. Since, in the experiment runs, the Cloud data center
was overloaded to its maximum capacity to host VMs, the
Red Hat manager migrated some VMs that were not imme-
diately started when they arrived to their destination hosts.
Consequently, the CPU usages of the asleep VMs were not
taken into account to compute the average sample standard
deviation of CPU usage attained by the Red Hat Manager.
The asleep VMs may have improved both (i) the average
number of VM migrations per host and (ii) the average sam-
ple standard deviation of percentage of CPU usage attained
by the Red Hat Manager. It should be noted that when SMAs
migrated VMs, the VMs were immediately started when they
arrived to their destination hosts.
It is noteworthy that the results shown in Fig. 6 suggest
that there is a trade-off between workload balance across
hosts and the number of VM migrations. This trade-off arises
because balancing workload across hosts over time requires
frequent VM migrations. Then, the sooner an SMA migrates
a VM (causing a resource usage imbalance), the more VM
migrations are done and the more balanced the Cloud data
center is.
In conclusion, through collaboration, agents adopting
MapLoad achieved better performance (regarding average
number of VM migrations) than the Red Hat Manager while
achieving a load balancing comparable to Red Hat Manager
with a relatively small difference of 1.74 % in the average
sample standard deviation of CPU usage. It should be noted
that SMAs adopting MapLoad balanced the Cloud data center
in a distributed and collaborative manner by interacting with
the FA only when all the SMAs were overloaded. Moreover,
in contrast to the load balancing mechanism adopted by the
Red Hat Enterprise Virtualization Suite [32], agents adopting
MapLoad do not require continuously monitoring computing
resource usages because agents autonomously trigger VM
migrations whenever their individual migration thresholds
are violated.
5.3 Evaluating the agent-based VM migration heuristics
Objective The experiment was designed to evaluate the effi-
ciency of the different VM migration heuristics adopted by
SMAs.
Experimental settings Experiment runs were conducted
using SMAs provided with each one of the VM migration
heuristics described in Sect. 4.3 (namely H:CPU-H:MEM,
H:CPU-L:MEM, L:CPU-H:MEM, L:CPU-L:MEM, R,
H:CPU, L:CPU, H:MEM and L:MEM) to explore and com-
pare their efficiency.
The parameters of (i) the Cloud data center’s input data,
e.g., specification of hosts, (ii) load-related input data (Table
1), e.g., resource usage profiles of VMs, and (iii) agent-based
testbed parameters (Table 2), e.g., SMAs’ monitoring fre-
quency, have the same justification presented in Sect. 5.1.
For each configuration of the testbed, 10 independent
experiment runs were conducted for a total of 90 simula-
tions, which involved instantiating 27,000 VMAs.
123

Fig. 7 Load balancing results
of the VM migration heuristics
implemented into MapLoad
Fig. 8 Average number of VM
migrations per host for each VM
migration heuristic
Performance measures The performance measures are (i)
average sample standard deviation of percentage of resource
usage and (ii) average number of VM migrations per host.
The lower the average sample standard deviation of percent-
age of resource usage, the better balanced a Cloud data center
is. The lower the average number of VM migrations per host,
the lesser overhead derived from migrating VMs.
Results Empirical results are shown in Figs. 7, 8, and 9.
From these results, four observations are drawn.
Observation 1 SMAs using VM migration heuristics that
take into account information from both CPU and mem-
ory usage achieved better load balance than SMAs using
either only CPU-based or only memory-based VM migra-
tion heuristics.
Analysis As shown in Fig. 7, columns 1–4, the smallest
average sample standard deviation was achieved by SMAs
123
Cluster Comput
selecting VMs to be migrated based on the sum of the
percentage of CPU and memory usage, namely, L:CPU-
L:MEM, H:CPU-H:MEM, H:CPU-L:MEM, and L:CPU-
H:MEM. The best load balance was achieved by SMAs
using the L:CPU-L:MEM heuristic (Fig. 7, 1st column),
in which the VM with the lowest resource usage was
selected to be migrated. Then, as in the knapsack prob-
lem, small items are easier to fit (in a host), and as a
consequence, the load can be uniformly distributed across
hosts.
SMAs using VM migration heuristics based only on either
CPU or memory usage were outperformed by SMAs using
VM migration heuristics based on combined information
from CPU and memory usage (Fig. 7) because by migrat-
ing VMs taking into account only one resource usage type,
e.g., CPU usage, may have caused a load imbalance in the
Cluster Comput
Fig. 9 Average number of FA’s
suggestions to keep VMs.
resource usage type that was not taken into account, e.g.,
memory usage.
In conclusion, providing SMAs with VM migration
heuristics that take into account both CPU and memory usage
results in a better overall load balance for both CPU and mem-
ory workloads on each host.
Observation 2 SMAs using an L:CPU migration heuris-
tic achieved better CPU load balance than SMAs using an
H:CPU or MEM-based VM migration heuristics. In addi-
tion, SMAs using either L:MEM o R VM migration heuristics
achieved better memory load balance than SMAs using an
H:MEM or CPU-based VM migration heuristics.
Analysis As observed in Fig. 7, whereas the best (i.e.,
the lowest) average sample standard deviation of percent-
age of CPU usage was achieved by SMAs using an L:CPU
heuristic, the worst (i.e., the highest) average sample stan-
dard deviation of percentage of CPU usage was achieved by
SMAs using memory-based VM migration heuristics. This
is because whereas SMAs using CPU-based VM migration
heuristics, namely, L:CPU and H:CPU, focused on balanc-
ing CPU load, SMAs using memory-based VM migration
heuristics focused on balancing memory load. In addition,
SMAs using an H:CPU migration heuristic were outper-
formed (regarding CPU load balance) by SMAs using an
L:CPU migration heuristic. This is because SMAs using
an H:CPU heuristic migrated VMs with a high CPU usage
causing large variations in CPU usage in both the host that
migrated the VM and the host that received the VM. In a
similar manner, SMAs using an L:MEM heuristic are better
balancing memory loads (Fig. 7) than SMAs using CPU-
based VM migration heuristics. In addition, SMAs using
either L:MEM or R achieved better memory load balance
than SMAs using an H:MEM heuristic because migrating
VMs with either a low memory usage (as in L:MEM) or a
(long-run) average memory usage (as in R) prevented the
Cloud data center from large variations in memory usage.
By using VM migration heuristics focused on a given
resource usage type, SMAs can autonomously balance
resource-specific (either CPU or memory) loads, which com-
bined with the resource-specific (either CPU or memory)
migration thresholds (see Sect. 4.1) endow SMAs with capa-
bilities to consolidate resource-specific loads, e.g., SMAs
with a CPU migration threshold set to 90 % may consolidate
and balance CPU loads in CPU-intensive hosts.
Observation 3 In general, SMAs using VM migration
heuristics taking into account both CPU and memory usage
balanced the loads involving a smaller number of VM migra-
tions than SMAs using VM migration heuristics taking into
account either memory or CPU usage.
Analysis As shown in Fig. 8, the SMAs using H:CPU-
H:MEM, L:CPU-H:MEM, and H:CPU-L:MEM heuristics
balanced the loads migrating fewer VMs than SMAs balanc-
ing loads based on only one resource usage type either CPU
or memory usage. This is because SMAs using VM migra-
tion heuristics that take into account information from both
CPU and memory usage continuously kept an evenly balance
load across the hosts (as shown in Observation 1 of Sect.
5.3). Then, SMAs using combined VM migration heuristics
needed to migrate fewer VMs than SMAs using VM migra-
tion heuristics based on either CPU or memory usage, which
needed to migrate more VMs to keep a balanced load across
hosts.
Endowing SMAs with VM migration heuristics that take
into account both CPU and memory usage results in (i) bal-
ancing loads more efficiently by migrating a relative small
number of VM migrations (comparable to commercial sys-
tems, see Observation 1 of Sect. 5.2) and (ii) a better overall
load balance (see Observation 1 of Sect. 5.3).
Observation 4 SMAs using either an L:CPU or an L:MEM
migration heuristic, received the fewest suggestions from the
FA to keep VMs and the largest number of suggestions from
the FA to migrate VMs.
123

Analysis As shown in Fig. 9, the FA (when requested
by SMAs) sent the smallest number of suggestions to keep
VMs to SMAs that balanced loads based on an L:CPU or
an L:MEM migration heuristic. This is because both L:CPU
and L:MEM migration heuristics selected the VM with the
lowest resource usage, which are easier to accommodate as
in the knapsack problem, where it is easier to accommodate
small items in a finite-capacity host, than large items. Sim-
ilarly, as shown in Fig. 8, when the SMAs used either an
L:CPU or an L:MEM migration heuristic, they received the
largest number of suggestions from the FA. This is because
the FA was able to find a server with sufficient resources to
host a VM with low resource usages.
Through interaction and collaboration SMAs and FAs
are capable of efficiently balancing loads without requir-
ing centralized, continuous monitoring by means of adopting
MapLoad where SMAs (when required) ask for suggestions
to the FA, which samples resource usages of the other SMAs.
In summary, by using autonomous and dynamic collabo-
ration, agents adopting MapLoad are capable of efficiently
balancing loads in a fully distributed manner across a
set of heterogeneous hosts outperforming central managers
endowed with commonly used load balancing heuristics with
full knowledge of the Cloud data center. In addition, SMAs
are capable of balancing data centers with a performance
comparable (with a relatively small difference of 1.74 %)
to the mechanism adopted by the Red Hat manager while
migrating fewer VMs. Moreover, SMAs are capable of using
their different VM migration policies to balance resource-
specific loads accordingly.
6 Related work
Cloud load balancing has been approached from differ-
ent perspectives ranging from energy-aware load balancing
[23] and network traffic load balancing [28] to balanc-
ing mechanisms focused on stream processing [22]. This
present load balancing mechanism is focused on balanc-
ing CPU and memory resources taking into account both
server heterogeneity and VM resource usage heterogeneity
simultaneously by making use of live migration of VMs.
In this regard, previous works [1,6,32,46] have disregarded
that Cloud data centers are commonly composed of highly
heterogeneous commodity hardware and that VM resource
usages may be highly volatile.
Centralized load balancing mechanisms [32,43] rely on
a single entity to monitor and balance loads across a set of
hosts, becoming a bottleneck and a single point of failure.
In contrast, in this work, agents endowed with MapLoad are
capable of dynamically sampling resource usages to monitor
hosts only when an (overloaded) host needs to migrate a VM.
Moreover, unlike VMware [43] and Red Hat [32] where all
123
Cluster Comput
the decisions about VM migrations are taken by a single
entity, agents, in this work, autonomously coordinate among
themselves to select the best (e.g., the least overloaded) host
to receive a given VM.
Placing VMs on a set of heterogeneous hosts can be
related to multi-dimensional bin packing problems [10] to
which mathematical approximations have been developed
[37]. Nevertheless, adopting such mathematical solutions to
balance data centers based on VM live migration is unfeasi-
ble [19] because (i) the solutions for bin packing problems
assume empty containers (i.e., hosts without any VMs), and
(ii) they are not concerned with the number of movements
(i.e., VM migrations) required to balance the load.
Hyser et al. [19] proposes a virtual machine placement
service that monitors and predicts (based on a simulated
annealing algorithm) resource usages of hosts to determine
the source and destination hosts of VMs according to a given
heuristic. Hyser et al. [19] acknowledge that the centralized
architecture of their system limits the scalability of the load
balancing mechanism, unlike the agents in this work that
balance the Cloud data center in a distributed manner. In
addition, the virtual machine placement service [19] aims to
keep resource usages of hosts as close to the overall aver-
age resource usage as possible. However, given that Cloud
data centers are commonly composed of heterogeneous com-
modity hosts, maintaining all the hosts with similar resource
usages disregarding their computer specifications may lead
to situations where compute-intensive hosts may have the
same CPU load than memory-intensive hosts. In contrast, in
this present work, SMAs can balance loads (in a distributed
manner) taking into account server heterogeneity by speci-
fying migration thresholds for different computing resource
types.
Load balancing mechanisms supported by VM live migra-
tion commonly make use of policies to control the tradeoff
between load balance and the number of VM migrations.
The Red Hat Enterprise Virtualization Manager [32] imple-
ments load balancing based on VM live migration. Red Hat’s
load balancing mechanism is endowed with a VM migration
threshold based merely on CPU to specify the maximum ser-
vice level at (i) which the host may start migrating VMs and
(ii) no more VMs are assigned to the host. Red Hat adopts
an even distribution load balancing heuristic consisting of
migrating VMs with the lowest CPU usage to the host with
the lowest CPU usage. Whereas in Red Hat a single central
entity selects both the VM and its target host, in this present
work, an overloaded SMA (i) decides which VM should be
migrated based on its migration heuristic that can be config-
ured to its specific needs, and (ii) selects a target host among
a set of self-nominated, underloaded SMAs. Agents, in this
work, collaborate among each other to balance both CPU and
memory loads in a distributed manner as opposed to Red Hat
that is a centralized system that only balances CPU loads. In
Cluster Comput
addition, agents, in this work, can adopt migration heuristics
focused on their specific commodity hardware, for instance,
an SMA managing a commodity host with limited memory
resources may opt to migrate VMs with high memory usage.
Moreover, upon receiving a proposal to host a VM, agents
are capable of nominating themselves to host the VM based
on their VM acceptance policies that aim to make a better
use of the hosts managed by SMAs, for instance, an SMA
managing a host with limited memory resources may opt to
propose itself to host VMs with high CPU usage, but very
low memory usage.
VMware provides a resource scheduler [43] for balanc-
ing loads. The load balancing mechanism is supported by
affinity rules. An affinity rule indicates that (i) two VMs
should be allocated either in the same host or in different
hosts or that (ii) a given VM should be assigned to a specific
group of hosts. Whereas in the resource scheduler of VMware
[43] affinity rules are defined based on (i) the assumption
of prior knowledge about the resource usage of the VMs
and (ii) the matching of VMs and hosts is conducted in a
static manner, agents, in this work, assume no prior knowl-
edge of VMs’ resource usages and the matching between
VMs and hosts is performed dynamically based on the cur-
rent resource usages of both VMs and hosts. The resource
scheduler [43] also provides a recommendation system [42]
to migrate VMs to balance either CPU or memory loads.
Unfortunately, VMware [42,43] does not disclose informa-
tion about how destination hosts and candidate VMs are
selected. However, patents filed by VMware (see [9,20]) sug-
gest that the selection of VMs and destination hosts are based
on resource usage prediction.
Load balancing mechanisms based on resource usage pre-
diction [3,9,20,34,46,47] suffer from the highly dynamic
performance variability of Cloud environments composed
of heterogeneous commodity hardware. The overall perfor-
mance of a Cloud depends on network conditions, workload
types, current number of VMs, VM resource usage patterns,
VM allocation request rates, actual matching of VMs and
hosts, etc. Then, estimating the performance of a Cloud
data center is, by itself, an extremely challenging prob-
lem. Moreover, load balancing mechanisms supported by
resource prediction [47] require a representative sample of
(i) the Cloud data center’s workload, (ii) of VM resource
usages, and (iii) of the performance profiles of hosts. As
a consequence, load balancing approaches supported by
resource prediction make use of behavioral trends [3], his-
torical data of workload [20], and VM resource usage traces
[9]. In contrast, this present work requires profiles of neither
workload nor VM resource usage patterns. Agents, in this
work, react to the performance variability of Cloud environ-
ments and can be configured to deal with the heterogeneous
commodity hardware that composes Clouds by using migra-
tion thresholds and VM acceptance policies. However, it is
acknowledged that the proposed agent-based load balancing
mechanism can be complemented by adding a resource usage
prediction mechanism.
With respect to agent-based load balancing approaches, in
the work presented in [48], agents deployed on VMs are capa-
ble of monitoring the location of consumer requests. Then,
whenever a number of consumer requests coming from a
foreign network overpasses (based on a threshold) the num-
ber of local requests, the VM (that receives the requests) is
migrated to the foreign network. In [48], there is no agent
interaction to guarantee whether migrating the VM based
merely on the location of its consumers will be better to the
overall load balance of the Cloud. The agent-based load bal-
ancing mechanism proposed by Anderson et al. [2] makes use
of VM live migration endowed with agent interaction proto-
cols enhanced with balancing heuristics, which are specified
using lightweight coordination calculus. In [2], the agents
managing hosts are classified as idle, overloaded, and under-
loaded. Whereas idle agents are assumed to be balanced and
take no part in load balancing, underloaded and overloaded
agents interact with each other to migrate VMs from over-
loaded to underloaded agents. Given that Cloud data centers
are commonly composed of heterogeneous commodity hard-
ware, classifying hosts in three levels may result in situations
where underloaded agents managing a host with limited com-
puting resources can pass from underloaded to overloaded
by accepting a VM with high resource usage. This is because,
in [2], no prior sampling of the resource usage of hosts is con-
ducted for selecting the best destination host according to a
criterion. In this regard, agents, in the present work, sam-
ple hosts’ resource usages to select the best destination host
according to a set of heuristics and policies. In addition, in
this work, a FA conducts an initial load balancing by distrib-
uting VM allocations across a set of hosts.
It is acknowledged that this paper is a significantly and
considerably extended version of the work reported in [16].
Whereas in [16], the idea of using policy-based agents to bal-
ance loads in Cloud data centers was introduced, this present
work extends [16] by:
(1) Devising a multi-agent protocol for load balancing
(MapLoad) where both FAs and SMAs collaborate
among themselves to evenly distribute loads (Sect. 4.5)
with a worst-case message complexity of O(n) (Sect.
4.6).
(2) Providing FAs with a set of load balancing heuristics
(Sect. 4.4).
(3) Designing a mechanism to provide overloaded SMAs
with suggestions from FAs (Sect. 4.5).
(4) Including a new VM migration policy that allows SMAs
to migrate VMs when either the CPU-based or memory-
based migration threshold is violated (Sect. 4.1).
(5) Defining VM acceptance policies (Sect. 4.2).
123

(6) Implementing an activation threshold into SMAs to
reduce the number of VM migrations (Sect. 4.1).
(7) Conducting extensive experiments to compare the effi-
ciency of MapLoad in balancing loads with respect to a
set of commonly used benchmark heuristics (Sect. 5.1),
to compare MapLoad with respect to the Red Hat Enter-
prise Virtualization Suite (Sect. 5.2), and to evaluate the
efficiency of the VM migration heuristics (Sect. 5.3).
7 Conclusion and future work
The novelty of this work lies in devising and implementing
a novel agent-based problem solving technique (MapLoad)
enhanced with VM migration policies, VM acceptance poli-
cies, VM migration heuristics, and a set of load balancing
heuristics to dynamically balance loads while taking into
account both server heterogeneity and VM resource usage
heterogeneity simultaneously.
The significance of this work lies in the fact that agents
adopting MapLoad are capable of balancing loads in a dis-
tributed and scalable manner while coping with continuously
evolving workloads and volatile resource usages of VMs by
making use of VM live migration. In addition, as shown in the
empirical evaluation (Sect. 5), through collaboration, agents
equipped with partial knowledge of the Cloud data center can
autonomously and dynamically balance loads outperform-
ing central managers endowed with commonly used load
balancing heuristics with full knowledge of the Cloud data
center. Moreover, agents adopting MapLoad achieved better
performance (regarding average number of VM migrations)
than the Red Hat Manager while achieving a load balancing
comparable to the Red Hat manager (a suite that achieved
record-setting benchmark results [38]).
This work contributes with (i) an agent-based load balanc-
ing architecture for Cloud data centers capable of monitoring
and balancing different workload types in a distributed
manner and (ii) with a novel agent-based problem solving
technique (MapLoad) capable of balancing workloads across
a set of commodity, heterogeneous hosts in a cooperative
manner. Moreover, this work also contributes with: (i) VM
migration policies to establish preferences over when VMs
should be migrated, (ii) VM acceptance policies to establish
preferences over which VMs should be hosted/accepted from
other hosts, and (iii) a set of VM migration heuristics to guide
the selection of VMs to be migrated.
Future work will focus on estimating VM migration
overhead [25] and developing a resource usage prediction
mechanism to complement the decision making process of
agents. In addition, whereas this work has focused on show-
ing the desirables properties of an agent-based Cloud load
balancing approach (e.g., efficient, distributed load balanc-
ing through agent collaboration), experiments using a much
123
Cluster Comput
larger number of hosts should be conducted to empirically
evaluate the scalability of the proposed agent-based Cloud
load balancing approach. However, it should be noted that, in
Sect. 4.6, it is formally demonstrated that the worst-case mes-
sage complexity of MapLoad is in the order of O(n) where
n stands for number of hosts.
Acknowledgments The authors would like to thank the Editor-in-
Chief and the anonymous referees for their comments and suggestions.
This work has been supported by Asociación Mexicana de Cultura A.C.
References
1. Amazon EC2: Elastic load balancing. http://aws.amazon.com/
elasticloadbalancing/ (2015). Accessed 1 Feb 2015
2. Anderson, P., Bijani, S., Vichos, A.: Multi-agent negotiation of
virtual machine migration using the lightweight coordination cal-
culus. In: Jezic, G., Kusek, M., Nguyen, N.-T., Howlett, R.J., Jain,
L.C. (eds.) 6th KES International Conference on Agent and Multi-
agent Systems: Technologies and Applications. Lecture Notes in
Computer Science, vol. 7327, pp. 124–133. Springer, Berlin (2012)
3. Andreolini, M., Casolari, S., Colajanni, M., Messori, M.: Dynamic
load management of virtual machines in cloud architectures. In:
Avresky, D.R., Diaz, M., Bode, A., Ciciani, B., Dekel, E. (eds.)
Cloud Computing. Lecture Notes of the Institute for Computer
Sciences, Social-Informatics and Telecommunications Engineer-
ing, vol. 34, pp. 201–214. Springer, Berlin (2010)
4. Bansal, N., Lodi, A., Sviridenko, M.: A tale of two dimensional
bin packing. In: Proceedings of the 46th Annual IEEE Symposium
on Foundations of Computer Science, pp. 657–666 (2005)
5. Bellifemine, F., Caire, G., Poggi, A., Rimassa, G.: JADE—a white
paper. EXP Search Innov. 3(3), 6–19 (2003)
6. Bobroff, N., Kochut, A., Beaty K.: Dynamic placement of virtual
machines for managing sla violations. In: Proceedings of the 10th
IFIP/IEEE International Symposium on Integrated Network Man-
agement, pp. 119–128 (2007)
7. Breitgand, D., Marashini, A., Tordsson, J.: Policy-driven service
placement optimization in federated clouds. Technical Report, IBM
Research Division (2011)
8. Buyya, R., Beloglazov, A., Abawajy, J.: Energy-efficient man-
agement of data center resources for cloud computing: a vision,
architectural elements, and open challenges. In: Proceedings of the
2010 International Conference on Parallel and Distributed Process-
ing Techniques and Application, pp. 12–15 (2010)
9. Canturk, I., Chengwei, W., Bhatt, C., Shanmuganathan, G., Holler,
A.M.: Process demand prediction for distributed power and
resource management. U.S. Patent Application 13/281,234, 2011
10. Chekuri, C., Khanna, S.: On multi-dimensional packing problems.
SIAM J. Comput. 33(4), 837–851 (2014)
11. Cisco systems: Cisco USB B440 M2 high-performance blade
server data sheet. http://www.cisco.com/en/US/prod/collateral/
ps10265/ps10280/ps11584/data_sheet_c78-646960.html (2015).
Accessed 1 Feb 2015
12. Clark, C., Fraser, K., Hand, S., Hansen, J.G., Jul, E., Limpach,
C., Pratt, I., Warfield, A.: Live migration of virtual machines. In:
Proceedings of the 2nd Conference on Symposium on Networked
Systems Design & Implementation, USENIX Association, vol. 2,
pp. 273–286 (2005)
13. Coffman, E.G., Garey, M.R., Johnson, D.S., Tarjan, R.E.: Perfor-
mance bounds for level-oriented two-dimensional packing algo-
rithms. SIAM J. Comput. 9(4), 808–826 (1980)
Cluster Comput
14. Daniels, J.: Server virtualization architecture and implementation.
Crossroads 16(1), 8–12 (2009)
15. Das, B., Bharghavan, V.: Routing in ad-hoc networks using min-
imum connected dominating sets. In: Proceedings of the IEEE
International Conference on Communications, pp. 376–380 (1997)
16. Gutierrez-Garcia, J.O., Ramirez-Nafarrate, A.: Policy-based agents
for virtual machine migration in cloud data centers. In: Proceedings
of the IEEE International Conference on Services Computing, pp.
603–610 (2013)
17. Gutierrez-Garcia, J.O., Sim, K.M.: A family of heuristics for agent-
based elastic cloud bag-of-tasks concurrent scheduling. Future
Gener. Comput. Sys. 29(7), 1682–1699 (2013)
18. Gutierrez-Garcia, J.O., Sim, K.M.: Agent-based Cloud Service
Composition. Appl. Intell. 38(3), 436–464 (2013)
19. Hyser, C., Mckee, B., Gardner, R., Watson, B.J.: Autonomic virtual
machine placement in the data center. Hewlett Packard Laborato-
ries, Technical Report HPL-2007-189, 2007–189 (2007)
20. Ji, M., Waldspurger, C.A., Zedlewski, J.: Method and system for
determining a cost-benefit metric for potential virtual machine
migrations. U.S. Patent No. 8,095,929, 2012. Washington, DC:
U.S. Patent and Trademark Office
21. Kerr, A., Diamos, G., Yalamanchili, S.: A characterization and
analysis of GPGPU kernels. Technical Report GIT-CERCS-09-06,
Georgia Institute of Technology (2009)
22. Kleiminger, W., Kalyvianaki, E., Pietzuch, P.: Balancing load in
stream processing with the cloud. In: Proceedings of the 2011 IEEE
27th International Conference on Data Engineering Workshops, pp.
16–21 (2011)
23. Lang, W., Patel, J.M., Naughton, J.F.: On energy management,
load balancing and replication. ACM SIGMOD Rec. 38(4), 35–42
(2010)
24. Lin, C.C., Liu, P., Wu, J.J.: Energy-efficient virtual machine pro-
vision algorithms for cloud systems. In: Proceedings of the 2011
Fourth IEEE International Conference on Utility and Cloud Com-
puting, pp. 81–88 (2011)
25. Liu, H., Xu, C.Z., Jin, H., Gong, J., Liao, X.: Performance
and energy modeling for live migration of virtual machines. In:
Proceedings of the 20th International Symposium on High Perfor-
mance Distributed Computing, pp. 171–182 (2011)
26. Metri, G., Srinivasaraghavan, S., Shi, W., Brockmeyer, M.: Exper-
imental analysis of application specific energy efficiency of data
centers with heterogeneous servers. In: Proceedings of IEEE
5th International Conference on Cloud Computing, pp. 786–793
(2012)
27. Montgomery, D.C., Jennings, C.L., Kulahci, M.: Introduction to
Time Series Analysis and Forecasting. Wiley, Hoboken (2011)
28. Patel, P., Bansal, D., Yuan, L., Murthy, A., Greenberg, A., Maltz,
D. A., Kern, R., Kumar, H., Zikos, M., Wu, H., Kim, C., Karri, N.:
Ananta: Cloud scale load balancing. In: Proceedings of the ACM
Conference on SIGCOMM, pp. 207–218 (2013)
29. Red Hat: Cluster general properties—administration guide.
https://access.redhat.com/site/documentation/en-US/Red_
Hat_Enterprise_Virtualization/3.1/html/Administration_Guide/
Cluster_general_entities.html (2015). Accessed 1 Feb 2015
30. Red Hat: Monitoring—administration guide. https://access.redhat.
com/documentation/en-US/Red_Hat_Enterprise_Virtualization_
for_Servers/2.2/html/Administration_Guide/Administration_
Guide-Monitoring.html (2015). Accessed 1 Feb 2015
31. Red Hat: Red Hat Enterprise Virtualization 3.2—explanation of set-
tings and controls in the Edit cluster policy window. https://access.
redhat.com/site/documentation/en-US/Red_Hat_Enterprise_
Virtualization/3.1/html/Administration_Guide/Cluster_general_
entities.html (2015). Accessed 1 Feb 2015
32. Red Hat: Red Hat Enterprise Virtualization 3.2—technical
reference guide. https://access.redhat.com/site/documentation/
en-US/Red_Hat_Enterprise_Virtualization/3.2/html/Technical_
Reference_Guide/index.html (2015). Accessed 1 Feb 2015
33. Ren, G., Tune, E., Moseley, T., Shi, Y., Rus, S., Hundt, R.:
Google-wide profiling: a continuous profiling infrastructure for
data centers. IEEE Micro 30(4), 65–79 (2010)
34. Ren, X., Lin, R., Zou, H.: A dynamic load balancing strategy for
cloud computing platform based on exponential smoothing fore-
cast. In: Proceedings of the 2011 IEEE International Conference on
Cloud Computing and Intelligence Systems, pp. 220–224 (2011)
35. Siebert, E.: Understanding VMware ESX virtual machine physical
and virtual hardware. http://searchvmware.techtarget.com/tip/Und
erstanding-VMware-ESX-virtual-machine-physical-and-virtual-
hardware (2015). Accessed 1 Feb 2015
36. Sim, K.M.: Agent-based cloud computing. IEEE T. Serv. Comput.
5(4), 564–577 (2012)
37. Skiena, S.: The Algorithm Design Manual: Text, vol. 1. Springer,
Berlin (1998)
38. SPEC: All published SPECvirt_sc2010 performance results
SPEC. http://www.spec.org/virt_sc2010/results/specvirt_sc2010_
perf.html (2015). Accessed 1 Feb 2015
39. Speitkamp, B., Bichler, M.: A mathematical programming
approach for server consolidation problems in virtualized data cen-
ters. IEEE T. Serv. Comput. 3(4), 266–278 (2010)
40. Tsai, Y.L., Huang, K.C., Chang, H.Y., Ko, J., Wang, E.T., Hsu,
C.H.: Scheduling multiple scientific and engineering workflows
through task clustering and best-fit allocation. In: Proceedings of
the 2012 IEEE Eighth World Congress on Services, pp. 1–8 (2012)
41. Vaquero, L.M., Rodero-Merino, L., Caceres, J., Lindner, M.: A
break in the clouds: towards a cloud definition. ACM SIGCOMM
Computer Communication Review 39(1), 50–55 (2008)
42. VMware: Distributed resource scheduler—recommendation
page. http://pubs.vmware.com/vsphere-4-esx-vcenter/index.jsp#
viewing_drs_cluster_information/c_the_drs_recommendations_
page.html (2015). Accessed 1 Feb 2015
43. VMware: VMware Infrastructure—Resource management with
VMware distributed resource scheduler. http://www.vmware.com/
pdf/vmware_drs_wp.pdf (2015). Accessed 1 Feb 2015
44. Von Laszewski, G., Diaz, J., Wang, F., Fox, G.C.: Comparison
of multiple cloud frameworks. In: Proceedings of the 2012 IEEE
5th International Conference on Cloud Computing, pp. 734–741
(2012)
45. Voorsluys, W., Broberg, J., Venugopal, S., Buyya, R.: Cost of vir-
tual machine live migration in clouds: a performance evaluation.
Cloud Computing, pp. 254–265. Springer, Berlin (2009)
46. Wood, T., Shenoy, P., Venkataramani, A., Yousif, M.: Black-box
and gray-box strategies for virtual machine migration. In: Proceed-
ings of the Fourth USENIX Symposium on Networked Systems
Design & Implementation, pp. 701–708 (2007)
47. Wu, Y., Yuan, Y., Yang, G., Zheng, W.: Load prediction using
hybrid model for computational grid. In: Proceedings of the 8th
IEEE/ACM International Conference on Grid Computing, pp. 235–
242 (2007)
48. Xing, L.: A self-management approach to service optimization
and system integrity through multi-agent systems. Master’s Thesis,
University of Oslo (2008)
49. Zhang, H., Jiang, F., Yoshihira, K., Chen, H., Saxena, A.: Intelligent
workload factoring for a hybrid cloud computing model. In: Pro-
ceedings of the 2009 World Conference on Services-I, pp. 701–708
(2009)
123

J. Octavio Gutierrez-Garcia
received his Ph.D. in Electri-
cal Engineering and Computer
Science from CINVESTAV and
Grenoble Institute of Technol-
ogy, respectively. Dr. Gutierrez-
Garcia is an Associate Profes-
sor in the Department of Com-
puter Science at Instituto Tec-
nológico Autónomo de México.
He has served as a reviewer for
numerous international confer-
ences and journals. His current
research interests include Cloud
computing, distributed artificial
intelligence, service-oriented computing, and agent-based modeling.
123
Cluster Comput
Adrian Ramirez-Nafarrate is
an Associate Professor in the
Department of Industrial & Oper-
ations Engineering at the Insti-
tuto Tecnológico Autónomo de
México. His research interests
include modeling, simulation
and analysis of service systems.
He received a Ph.D. degree in
Industrial Engineering from Ari-
zona State University, an MS
in Manufacturing Systems from
ITESM and a BS in Industrial
Engineering from Universidad
de Sonora.