Analysis of Virtualization

Technologies for High Performance

Computing Environments
Andrew J. Younge, Robert Henschel, James
T. Brown, Gregor von Laszewski, Judy Qiu,
Geoffrey C. Fox

Indiana University Bloomington
https://portal.futuregrid.org
1
Outline
Introduction
Related work
Feature comparison
FutureGrid experimental setup
Performance comparison
HPCC
SPEC
Discussion

2
https://portal.futuregrid.org
Introduction
What is Virtualization?
A method of partitioning a physical computer into multiple
virtual computers, each acting independently as if they were
running directly on hardware.
What is a Hypervisor?
A technique used to run multiple operating systems
simultaneously on a single resource.
Also called a Virtual Machine Monitor (VMM).
What is a Virtual Machine?
A software implementation of a machine that executes as if it
was running on a physical resource directly.
Why does it matter?
Cloud Computing!!!



3
https://portal.futuregrid.org
Motivation
Most Cloud deployments rely on
virtualization.
Amazon EC2, GoGrid, Azure, Rackspace Cloud
Nimbus, Eucalyptus, OpenNebula, OpenStack
Number of Virtualization tools or Hypervisors
available today.
Xen, KVM, VMWare, Virtualbox, Hyper-V
Need to compare these hypervisors for use
within the scientific computing community.
4
https://portal.futuregrid.org
Current Hypervisors
5
https://portal.futuregrid.org
Hypervisors
Evaluate Xen, KVM, and VirtualBox hypervisors
against native hardware
Common, well documented
Open source, open architecture
Relatively mature & stable
Cannot benchmark VMWare hypervisors due
to proprietary licensing issues.
6
https://portal.futuregrid.org
Related Research
Some work has already been done to evaluate
performance
Karger, P. & Safford, D. I/O for virtual machine monitors: Security and performance issues.
Security & Privacy, IEEE, IEEE, 2008, 6, 16-23
Koh, Y.; Knauerhase, R.; Brett, P.; Bowman, M.; Wen, Z. & Pu, C. An analysis of performance
interference effects in virtual environments. Performance Analysis of Systems & Software,
2007. ISPASS 2007. IEEE International Symposium on, 2007, 200209.
K. Jackson, L. Ramakrishnan, K. Muriki, S. Canon, S. Cholia, J. Shalf, H. Wasserman, and N.
Wright, Performance Analysis of High Performance Computing Applications on the Amazon
Web Services Cloud, in 2nd IEEE International Conference on Cloud Computing Technology
and Science. IEEE, 2010, pp. 159168.
P. Barham, B. Dragovic, K. Fraser, S. Hand, T. L. Harris, A. Ho, R. Neugebauer, I. Pratt, and A.
Warfield, Xen and the art of virtualization, in Proceedings of the 19th ACM Symposium on
Operating Systems Principles, New York, U. S. A., Oct. 2003, pp. 164177.
Adams, K. & Agesen, O. A comparison of software and hardware techniques for x86
virtualization. Proceedings of the 12th international conference on Architectural support for
programming languages and operating systems, 2006, 2-13.
7
https://portal.futuregrid.org
Features
Xen KVM VirtualBox VMWare
Paravirtualization Yes No No No
Full Virtualization Yes Yes Yes Yes
Host CPU X86, X86_64, IA64 X86, X86_64, IA64,
PPC
X86, X86_64 X86, X86_64
Guest CPU X86, X86_64, IA64 X86, X86_64, IA64,
PPC

X86, X86_64 X86, X86_64
Host OS Linux, Unix Linux Windows, Linux, Unix Proprietary Unix
Guest OS Linux, Windows, Unix Linux, Windows, Unix Linux, Windows, Unix Linux, Windows, Unix
VT-x / AMD-v Opt Req Opt Opt
Supported Cores 128 16* 32 8
Supported Memory 4TB 4TB 16GB 64GB
3D Acceleration Xen-GL VMGL Open-GL Open-GL, DirectX
Licensing GPL GPL GPL/Proprietary Proprietary
8
https://portal.futuregrid.org
Usability
KVM and VirtualBox trivial to install & deploy.
Xen requires special kernel, leading to more
complications.
VMWare ESX runs as a standalone hypervisor.
All are supported under Libvirt API.
Used by many IaaS frameworks.
Xen & Virtualbox have nice CLI, VMWare has
an advanced web based GUI.
9
https://portal.futuregrid.org
Performance Analysis
In order to assess various performance
metrics, benchmarks are needed.
Provide a fair, apples-to-apples comparison
between each hypervisor.
Comparisons can be made across other
benchmark submissions on different machines.
Reproducible and verifiable results.
Open standards, no special optimizations or tricks
available (hopefully).
10
https://portal.futuregrid.org
HW Resources at: Indiana University, San Diego Supercomputer Center, University of Chicago /
Argonne National Lab, Texas Applied Computing Center, University of Florida, & Purdue
Software Partners: University of Southern California, University of Tennessee Knoxville, University of
Virginia, Technische Universtitt Dresden
FutureGrid
NID 10GB/s
10GB/s
10GB/s
10GB/s
10GB/s
1GB/s
Router
11 6
5
12
4
7
2
7
Germany
France
IU: 11 TF IBM 1024 cores
6 TF Cray 672 cores
TACC: 12 TF Dell 1152 cores
UCSD: 7 TF IBM 672 cores
UC: 7 TF IBM 672 cores
UF: 3 TF IBM 256 cores
Interet 2
TeraGrid
FutureGrid is an experimental grid and cloud test-bed with
a wide variety of computing services available to users.
https://portal.futuregrid.org
Testing Environment
All tests conducted on the
IU India cluster as part of
FutureGrid.
Identical nodes used for
each hypervisor, as well
as a bare-metal (native)
machine for the control
group.
Each host OS runs RedHat
Enterprise Linux 5.5.
India: 256 1U compute nodes
2 Intel Xeon 5570 Quad core
CPUs at 2.93Ghz
24GB DDR2 Ram
500GB 10k HDDs
InfiniBand DDR 20Gbs
12
Testing Environment (cont)
Guest Operating System:
RedHat Enterprise Linux 5.5 x86_64
Kernel 2.6.18-194.8.1.el5
Minimal server installation (Base, Core)
Default settings and services
Full virtualization using Intel VT-x
Each VM gets 16GB of memory
Virtualization Add- ons:
Xen: version 3.1.0 with 2.6.18-194.8.1.el5xen kernel
KVM: version 83
VirtualBox: version 3.2.10_66523_rhel5
Bare-Metal OS = Host OS


13
https://portal.futuregrid.org
Benchmark Setup
HPCC
Industry standard HPC
benchmark suite from
University of Tennessee.
Sponsored by NSF, DOE,
DARPA.
Includes Linpack, FFT
benchmarks, and more.
Targeted for CPU and
Memory analysis.
SPEC OMP
From the Standard
Performance Evaluation
Corporation.
Suite of applications
aggrigated together
Focuses on well-rounded
OpenMP benchmarks.
Full system performance
analysis for OMP.
14
https://portal.futuregrid.org
15
https://portal.futuregrid.org
16
https://portal.futuregrid.org
17
https://portal.futuregrid.org
18
https://portal.futuregrid.org
19
https://portal.futuregrid.org
Performance Roundup
Hypervisor performance in Linpack reaches roughly 70% of native
performance during our tests, with Xen showing a high degree of
variation.
FFT benchmarks seem to be less influenced by hypervisors, with
KVM and VirtualBox performing at native speeds yet Xen incurs a
50% performance hit with MPIFFT.
With PingPong latency KVM and VirtualBox perform close to native
speeds while Xen has large latencies under maximum conditions.
Bandwidth tests vary largely, with the VirtualBox hypervisor
performing the best, however having large performance variations.
SPEC benchmarks show KVM at near-native speed, with Xen and
VirtualBox close behind.
20
https://portal.futuregrid.org
Conclusion
Big Question: Is Cloud Computing viable for
scientific High Performance Computing?
Our answer is Yes (for the most part).
Features: All hypervisors are similar.
Performance: KVM is fastest across most
benchmarks, VirtualBox close.
Overall, we have found KVM to be the best
hypervisor choice for HPC.
Currently moving to KVM for all of FutureGrid.
21
https://portal.futuregrid.org
THANK YOU!
22
https://portal.futuregrid.org
Virtual Machines
! " # ! " #
! " " #
$%&' ( %&) #
*+, - .%/) ' #
0%&' ( %&) #
12) &%345##
*67/) , #
8229# 8229#
*+, - .%/) ' #
0%&' ( %&) #
12) &%345##
*67/) , #
8229# 8229#
$: 7/#; ) 7: - &<) #
http://futuregrid.org 23
Feature Roundup
All hypervisors evaluated have acceptable
level of features for x86 virtualization.
Xen provides best expandability, supporting
up to 128 CPUs and 4TB of RAM.
Can remove CPU limit for KVM.
VirtualBox needs to add support for >16GB RAM.
All have API plugins to allow for simplified IaaS
usage.


24
https://portal.futuregrid.org
Need to learn more from a performance comparison.
Benchmarks
SPEC Benchmarks
specCPU 2006 CPU bound benchmark
specMPI 2007 MPI: cpu, memory, network
specOMP 2001 Open MP: cpu, memory, IPC
specVIRT 2010 CPU, memory, disk I/O, network
HPCC Benchmarks
HPL (Linpack), DGEMM, STREAM, PTRANS,
RandomAccess, FFT, etc
http://futuregrid.org 25