You are on page 1of 44

M ASARYKOVA UNIVERZITA FAKULTA INFORMATIKY

}w!"#$%&123456789@ACDEFGHIPQRS`ye|
B ACHELOR THESIS

Virtual machine management software

Ondrej Fam ra e

Brno, Spring 2011

Declaration
I hereby declare, that this paper is my original authorial work, which I have worked out by my own. All sources, references and literature used or excerpted during elaboration of this work are properly cited and listed in complete reference to the due source.

Ondrej Fam ra e

Advisor: RNDr. Jan Kasprzak ii

Acknowledgement
I would like to thank my supervisor for support and feedback. Also I would like to thank Z.F. for grammar check and moral support.

iii

Abstract
The goal of the work is to introduce concept of virtualization management software (VMS), compare existing solutions in this area and introduce architecture of libvirt, virtualization API library. The practical output of this thesis is to design VMS for use at Faculty of Informatics at Masaryk University.

iv

Keywords
virtualization, qemu, kvm, libvirt

Contents
1 2 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . About Virtualization . . . . . . . . . . . . . . . . . . . . . . . . 2.1 Common terminology . . . . . . . . . . . . . . . . . . . . 2.2 Types of virtualization . . . . . . . . . . . . . . . . . . . . 2.2.1 Full hardware emulation . . . . . . . . . . . . . . 2.2.2 Hardware assisted/accelerated emulation . . . . 2.2.3 Paravirtualization . . . . . . . . . . . . . . . . . . 2.2.4 OS-level virtualization/containers virtualization . 2.3 Linux-specic technologies used in virtualization . . . . 2.3.1 Huge pages . . . . . . . . . . . . . . . . . . . . . . 2.3.2 Kernel Samepage Merging (KSM) . . . . . . . . . 2.3.3 sVirt . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3.4 Control Groups (cgroups) . . . . . . . . . . . . . . 2.3.5 VirtIO paravirtualized I/O drivers . . . . . . . . . Virtualization management software . . . . . . . . . . . . . . 3.1 Architecture of VMS . . . . . . . . . . . . . . . . . . . . . 3.1.1 VM lifecycle management . . . . . . . . . . . . . . 3.1.2 VM conguration management . . . . . . . . . . . 3.1.3 Storage management . . . . . . . . . . . . . . . . . 3.1.4 Virtual network management . . . . . . . . . . . . 3.1.5 User management . . . . . . . . . . . . . . . . . . 3.1.6 VM migration . . . . . . . . . . . . . . . . . . . . . 3.1.7 VM provisioning . . . . . . . . . . . . . . . . . . . 3.1.8 VM access control . . . . . . . . . . . . . . . . . . . 3.2 Built-in VMS . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.1 VirtualBox . . . . . . . . . . . . . . . . . . . . . . . 3.3 Stand-alone VMS . . . . . . . . . . . . . . . . . . . . . . . 3.3.1 Ganeti . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.2 libvirt . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 Cloud computing . . . . . . . . . . . . . . . . . . . . . . . 3.4.1 OpenNebula . . . . . . . . . . . . . . . . . . . . . . 3.4.2 Amazons Elastic Compute Cloud (Amazon EC2) Libvirt virtualization API . . . . . . . . . . . . . . . . . . . . . 4.1 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1.1 Guest conguration management . . . . . . . . . 4.1.2 Storage management . . . . . . . . . . . . . . . . . 4.1.3 Networking management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 5 5 6 6 6 7 7 8 8 8 9 9 9 10 10 10 11 11 12 12 12 13 13 14 14 15 15 16 17 17 18 19 19 20 21 21 1

4.1.4 Access and user management . . . . . . . . . . . . . . 4.2 API bindings for other languages . . . . . . . . . . . . . . . . 5 VMS for Faculty of Informatics at Masaryk University (FI MU) 5.1 Faculty Administration (fadmin) . . . . . . . . . . . . . . . . 5.2 State of virtualization deployment . . . . . . . . . . . . . . . 5.2.1 PV175/PV176 MS Windows Systems Management I./II. 5.2.2 LaSArIS lab . . . . . . . . . . . . . . . . . . . . . . . . 5.2.3 Generic servers for research projects . . . . . . . . . . 5.3 General requirements . . . . . . . . . . . . . . . . . . . . . . . 5.4 Special requirements . . . . . . . . . . . . . . . . . . . . . . . 5.4.1 PV090 UNIX - Seminar of System Management . . . 5.4.2 LaSArIS lab . . . . . . . . . . . . . . . . . . . . . . . . 5.5 Web application vs. Desktop application . . . . . . . . . . . . 6 fadmin/virt . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.1 Database design . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 Custom scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2.1 VM serial console script . . . . . . . . . . . . . . . . . 6.2.2 VM IP discovery script . . . . . . . . . . . . . . . . . . 6.2.3 VM hibernation script . . . . . . . . . . . . . . . . . . 6.3 Limiting resource usage . . . . . . . . . . . . . . . . . . . . . 6.4 Front-end . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4.1 one-click VM creation . . . . . . . . . . . . . . . . . . 7 Use case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.1 Virtualization server Adikia . . . . . . . . . . . . . . . . . . . 7.1.1 Networking . . . . . . . . . . . . . . . . . . . . . . . . 7.1.2 Disks . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.1.3 Software . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2 Current deployment . . . . . . . . . . . . . . . . . . . . . . . 7.3 Stats and graphs . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3.1 CPU . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3.2 Memory . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3.3 KSM . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.1 Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . A Appendix - Screenshots . . . . . . . . . . . . . . . . . . . . . . . .

22 22 23 23 23 24 24 24 25 25 25 25 26 27 27 28 28 28 29 29 30 30 31 31 31 31 32 32 32 33 33 34 35 35 37

1 Introduction
Virtualization is a technique that allows simultaneous execution of applications on single physical computer providing them with isolated environment called virtual machine (VM). The concept of VM was in existence since 1960s when it was rst developed by IBM to provide concurrent, interactive access to a mainframe computer. [3] Since then several new techniques of virtualization has been developed to improved VMs performance. These techniques are discussed in chapter 2. Improvements specic to the virtualization on Linux systems are discussed in section 2.3. With increasing popularity of virtualization, question of VM administration emerges. In order to simplify the management of growing number of VMs, virtualization management software (VMS) appeared. not only they offer VM management, but they go beyond that and provide management of activities tightly connected to virtualization, such as network and storage management. General architecture of these systems is discussed in chapter 3 Later in the chapter, the overview of architecturally interesting VMS is provided. In chapter 4 VMS called libvirt is introduced. It provides a good building block for developing VMS on the top of it. Architecture of guest conguration management, storage management, network management, access and user management is discussed here, aiming to show how this is provided by libvirt. Chapter 5 introduces concept of VMS for Faculty of Informatics at Masaryk University (FI MU). As the potential platform for VMS, Faculty Administration (fadmin) system is briey introduced. After that current state of penetration of virtualization and VMS at FI MU is analysed. Based on the analysis we dene general as well as some specic requirements for universal VMS at FI MU.

1. I NTRODUCTION The output of the work, VMS for FI MU, is introduced in chapter 6. Here the architecture of system built on libvirt library as well as custom additions that adds functionality missing in libvirt is showed. In chapter 7 we take look at deployment of the work in real world environment. In conclusion (8) we summarize the achieved goals and suggest possible future improvements. Screenshots of created application can be found in appendix A.

2 About Virtualization
The primary goal of virtualization is to separate software from the available hardware resources in order to allocate them optimally to individual isolated systems[6]. Compared to non-virtualized systems it adds another layer between running system and hardware. This way virtualization allows the placement of specic virtual machines, each containing only the necessary services for their task. This improves security of these services as it brings another layer of isolation.

2.1

Common terminology

Host computer is a real physical machine that runs hypervisor and posses hardware resources we want to use for virtualization. Guest computer, also called virtual machine (VM), is an isolated virtual environment within host Operating system (OS). Emulation in computing is referred to as functionally simulating a system by another.[6] It creates the same data, executes the same programs, achieves same results as original system. Hypervisor also called virtual machine monitor (VMM) is a program that creates virtual environment and schedules resources of host for VMs. 5

2. A BOUT V IRTUALIZATION There are two types of hypervisors: type 1 hypervisor runs directly on hardware and allocates resources as requested by guests. type 2 hypervisor runs within OS, which allocates resources to hypervisor and then it allocates them to VMs.

2.2

Types of virtualization

Depending on hardware capabilities of virtualization host and guest operating system (OS) compatibility we have several virtualization types. Sorting them from the slowest one but the most compatible with various guest OSs and broad range off processor architectures, to the fastest one but restricted to run only on host processor architecture and to use only host kernel calls. 2.2.1 Full hardware emulation This type uses only the software emulation to simulate guest hardware. The advantage of this approach is ability to emulate virtually any kind of hardware. This way we can run applications that were intended for hardware different from one we have. However, with this compatibility comes the price of poor performance, because every instruction sent to emulated hardware must be processed by software running on the host. Compared to the non-virtualized application, performance is signicantly worse. 2.2.2 Hardware assisted/accelerated emulation If we have hardware that supports virtualization we can ofoad some work from software emulation to hardware to get better performance. Limitation of this approach is, that not all hardware provides virtualization support, so some of it still needs to be emulated for guest by the software. Till this point guest OS could run in emulated environment without modications.

2. A BOUT V IRTUALIZATION 2.2.3 Paravirtualization Because fully emulated VMs might not be aware of the virtual environment they run in, it may be a good idea to improve their performance to make the aware of this environment. This technique is called paravirtualization. The basic idea is to ensure co-operation between guests and host system, in order to schedule resources of host more effectively among other guests. The only problem, that arises here, is that guest OS must be modied to be able to run and to communicate with hypervisor. 2.2.4 OS-level virtualization/containers virtualization The fastest technique, which is still considered to be a virtualization, is deployed only by isolating guest on le system level. Linux systems only need two parts, which creates functional system, kernel and root le system. OS-level virtualization isolates guest in its own le system namespace and provides it with access to running host kernel. The example of this kind of virtualization is Linux command chroot, which creates le system isolation for guest. Performance of this solution is native-like, but guest can only use kernel calls of currently running host kernel.

2. A BOUT V IRTUALIZATION

2.3

Linux-specic technologies used in virtualization

2.3.1 Huge pages Linux uses by default 4KB memory pages, but in vast majority of nowadays kernels there is also support for larger pages called huge pages. Depending on the architecture, these pages can be 2MB or 4MB in size. The main advantage of using huge pages is that it can lower the number of cache misses in processors TLB (Translation Look-aside Buffer) , what makes "memoryhungry" applications run faster. Since virtualization is a "memory-hungry", it can benet from this. Performance improvement while using huge pages with KVM1 is around 20 %.[7] 2.3.2 Kernel Samepage Merging (KSM)

Even while running Linux without virtualization, there is probability of allocating the memory page containing duplicate content. Most of these duplicities can be suppressed by using copy-on-write memory pages 2 but this requires the system to recognize which memory pages are the same. To mitigate memory page duplicities the technique, called KSM, is deployed. This technique periodically scans memory searching for duplicate memory pages. Once these are found, they are converted into copy-on-write pages freeing up the memory.
1. Kernel virtual machine 2. copy-on-write memory page doesnt take up any space in memory, in page table it only refers to memory page holding the same content. This content is copied in case it need to be changed in one of pages.

2. A BOUT V IRTUALIZATION Because for virtualization its not anything rare to have multiple guests running OS or applications that stores the same data in memory, this can lower memory usage of guests on host. 2.3.3 sVirt sVirt is pluggable security framework for libvirt. 3 It deploys Mandatory Access Control (MAC) to isolate guests from host and others guests. This adds another level of isolation in case the guest would manage outbreak from virtual environment, due to aw in hypervisor. This solution is designed to work with SELinux4 and SMACK5 security drivers. 2.3.4 Control Groups (cgroups) Control Groups provide a mechanism for aggregating/partitioning sets of tasks, and all their future children, into hierarchical groups with specialized behaviour.[2] In virtualization its mainly used for limiting and accounting of basic resources (CPU, RAM, disk, network). It can also be used for fair processor scheduling among guests if used with CFQ6 Linux I/O scheduler. 2.3.5 VirtIO paravirtualized I/O drivers VirtIO is an API for paravirtualized drivers. [4] It allows use of paravirtualized drivers within hypervisor, which doesnt provide paravirtualization. On supported guest OSs it brings better performance comparing to their emulated counterparts but reducing compatibility with guest OSs. The second limitation of these drivers is that they have to be supported by the host system as well. (this requirement comes out from the fact that they are paravirtualized).

3. 4. 5. 6.

libvirt is VM management system which will be covered later in chapter 4 Security Enhanced Linux Simplied MAC Kernel Completely Fair Queuing

3 Virtualization management software


Since hypervisors provides virtual environment for guests, they are not usually accessed directly. Instead virtualization management software (VMS), which provides interface that makes management of guests easier, is used. Some of the features usually required from VMS include: VM lifecycle management VM conguration management management of storage used by VMs management of virtual networks user management VM migration from one host to another VM provisioning VM access control

Interface provided by VMS can be in form of: command line interface (CLI), application with GUI1 or application programming interface (API).

The main goal remains the same, to abstract work with hypervisor and related utilities.

3.1

Architecture of VMS

3.1.1 VM lifecycle management The fundamental property of VMS is ability to keep the track of VMs life. This includes who, when and how created, modied and started VM as well as information about VMs uptime before its crash. VMS should also provide feature for pausing or hibernating VMs - dumping their active memory to le on hosts for later restoration and shutting them effectively down.

1. Graphical user interface

10

3. V IRTUALIZATION MANAGEMENT SOFTWARE 3.1.2 VM conguration management As the VM is something that exists only virtually, there needs to be a place where its conguration is stored. Depending on VMS, the conguration of VM can be variably abstracted. In case of more abstract conguration it brings better interoperability across different VMS and hypervisors. While in case of less abstract one, it can be more optimized for selected hypervisor, as it may consists only of command line arguments that needs to be passed to emulator in order to start VM. Storing the conguration is only the question of implementation and it usually includes using les and databases to store it. 3.1.3 Storage management VM is not aware of virtual disks location on host. This is provided by VMS, which knows where the virtual disk is and passes this information to VM. Possible virtual disks locations on host are: a le (on hosts or network le system), a partition on physical disk on host or a physical disk attached to the host.

Depending on VMs storage type, this subsystem provides basic services such as: resizing the virtual disk, space allocation on host (in case of le-based virtual disks), virtual disk compression or encryption provided by host,

This subsystem keep track of disks and their attachment to VMs, in order to provide optimization in case of simultaneous access by multiple VMs to the disk by disabling caching. It also can provide features such as disk provisioning, which populates empty virtual disk with selected data, or snapshoting which allows to create a snapshot of the virtual disk in selected moment and adds possibility to return its state to this point.

11

3. V IRTUALIZATION MANAGEMENT SOFTWARE 3.1.4 Virtual network management VM can be connected to physical network in different ways, depending on hosts level of control over this connection. These are: host-only access which allows the VMs network to access only the host, NAT-ed access to hosts physical network, direct access to hosts physical network (using routing or network bridge on host) or exclusive access to network card of the host.

While the last option grants VM the same access to network as the host have with nearly the same performance as host, it leaves host minimum control over the network card. Other approaches enables host to use virtual networks to have control over trafc that ows through it. Virtual networks provides ability to interconnect VMs with themselves, host or hosts physical network. Virtual network management stores information about these virtual networks and also keep track of their lifecycle. 3.1.5 User management For multi-user environment deployment the user management is the most important feature, as it provides way to limit access to actions provided by VMS by users and groups. Authentication of users can be performed by VMS or it can be provided by external mechanism such as LDAP2 or Kerberos. 3.1.6 VM migration Process of moving VM from one host to another we call migration. This process provides temporarily or permanently transferring the VM conguration along with storage (such as virtual disks, ISO images) connected to VM. VMS can provide two types of migration: ofine migration which is done while VM is not running during migration and online migration that transfers VM without stopping it.

In second case this can be very difcult because the VMs memory and storage may be in unpredictably changing state that makes online migration sometimes impossible.
2. Lightweight Directory Access Protocol

12

3. V IRTUALIZATION MANAGEMENT SOFTWARE 3.1.7 VM provisioning To simplify creation of VMs, VMS can provision template of VMs hardware conguration or even disk template containing beforehand prepared OS and applications. Also it can provide the option to upload users disk or VM conguration to system. This rapidly speeds up deployment of VMs serving the common tasks such as web or database servers. 3.1.8 VM access control Several ways how to access VM include access using: graphical console to VM on host or over network, serial console to VM on host or over network, direct connection to VMs OS over network (for example using ssh).

VMS deploys access restrictions depending on used technology they can vary from simple le system access restrictions (for example in case of connecting to local socket on which serial console of guest resides), to certicate based authentication using RDP3 . The common way is to use password to protected access to VM.

3. Remote Desktop Protocol

13

3. V IRTUALIZATION MANAGEMENT SOFTWARE

3.2

Built-in VMS

The model in which hypervisor is inseparable part of VMS is called Built-in VMS. Advantage of this approach is that it can fully exploit all the feature of hypervisor. Its limitation is that it often can manage only the built-in hypervisor. Usual deployment of this model is used on workstations as virtualization solution for single user or as part of bigger VMS where it servers the role of node. 3.2.1 VirtualBox Supported platforms: Supported hypervisors: used version: Management of Storage: Network: Users: VM provisioning: VM migration: Linux, Mac, Windows built-in VirtualBox hypervisor 4.0.6 OSE simple, built-in built-in none, relies on host system none built-in

VirtualBox (VB) is hypervisor bundled with set of tools acting as VMS in one package. Among this tools we can nd GUI named VirtualBox Manager and CLI utility named VBoxManage for advanced users. The GUI compared to CLI contain only subset of features that VB provides. Access control is based on "all or nothing" principle and it does not provide user management. Storage management is rather simple one as it only keeps track of disk and ISO images4 that are used by VMs. Network management is quite simple and limits VMs to use 4 network cards at max. The hypervisor provides support for VM migration, which it calls teleportation, however it involves use of CLI and so it provides solution to advanced user only.

4. ISO image is an "image" of an entire CD, DVD, or BD. The entire contents of a disc can be perfectly represented in a single ISO le.[1]

14

3. V IRTUALIZATION MANAGEMENT SOFTWARE

3.3

Stand-alone VMS

Opposed to built-in VMS they come without bundled hypervisor and serves purpose of an alternative VMS for built-in VMS or universal VMS capable to manage more than one hypervisor.

Ability to manage multiple hypervisors usually incorporates abstraction of VMs conguration in VMS. However, this abstraction can be limiting for some hypervisor-specic features which may break portability of VMs conguration.

3.3.1 Ganeti Supported platforms: Supported hypervisors: used version: Management of Storage: Network: Users: VM provisioning: VM migration: Linux Xen, KVM 2.4.2 built-in built-in built-in, limited built-in built-in

Ganeti represents cluster-based 5 VMS that can manage Xen and KVM hypervisor. It is accessed through CLI or API that provides management by other applications such as Ganeti Web Manager6 . Primary goal is deployment on 1-40 hosts with easy setup for redundancy of VMs. Storage management provides as interesting feature ability to setup virtual disk on network RAID17 disk facilitating DRBD89 technology. This allows easy migration of VMs between hosts, called nodes, across cluster.

5. cluster = set of machines that cooperate in order to bring better performance or redundancy 6. http://code.osuosl.org/projects/ganeti-webmgr 7. Redundant Array of Independent Disks (in mirroring mode) 8. Distributed Replicated Block Device 9. http://www.drbd.org/

15

3. V IRTUALIZATION MANAGEMENT SOFTWARE 3.3.2 libvirt Supported platforms: Supported hypervisors: used version: Management of Storage: Network: Users: VM provisioning: VM migration: Linux, Mac, Windows Xen, KVM, VMware, LXC, VirtualBox, ... 0.8.2 built-in built-in none built-in built-in

Libvirt is an API for wide range hypervisors that tries to provide unied management of VMs on different hosts using different hypervisors. To demonstrate all capabilities it also provides CLI utility that can be used for access to this API. Main aim of libvirt is to provide building block for other VMS. More about libvirt can be found in chapter 4.

16

3. V IRTUALIZATION MANAGEMENT SOFTWARE

3.4

Cloud computing

Cloud computing represents VMS that tries to simplify whole concept about VMs in order to allow less technically skilled user to use virtualization on large scale. It aims not only to provide access to hypervisors but also to manage whole infrastructure around hosts lowering the maintenance cost of virtualization. This infrastructure is then called a cloud. Before continuing we establish some additional terminology: instance is name for VM running inside cloud. public cloud means that whole infrastructure runs on hardware over which we have no control. private cloud means that we provide hardware for cloud infrastructure. hybrid cloud is mix between public and private cloud where we use own hardware but we also can use public infrastructure, which can help us in times we need more hardware than we have and we dont want to run all our instances in public cloud. To access cloud-based VMS, web GUI is mostly deployed. Main advantage of these VMS is that they are aimed to provide easy deployment of virtualization. They have already built-in network, storage and user management. 3.4.1 OpenNebula Supported platforms: Supported hypervisors: Linux Xen, KVM, VMware

OpenNebula aims to provide a open, exible, extensible, and comprehensive management layer to automate and orchestrate the operation of virtualized data centers by leveraging and integrating existing deployed solutions for networking, storage, virtualization, monitoring or user management. [5] It builds on the top of other VMS and provide cloud infrastructure for private, hybrid and public clouds. Compared to Amazon EC2 it provides only VMS without infrastructure on which it will run. Similarly it uses web based GUI to manage the whole infrastructure.

17

3. V IRTUALIZATION MANAGEMENT SOFTWARE 3.4.2 Amazons Elastic Compute Cloud (Amazon EC2) Supported platforms: Supported hypervisors: Linux, Mac, Windows Xen

Amazon EC2 is paid public cloud that provides not only VMs web GUI and API to access VMs but also underlying infrastructure. Conguration of VMs is dened by templates and cannot be fully customized. Pricing the usage of this service is based on resources consumed by running instances. This solution is example of how the cloud-based VMS can look-like a worklike, but because it is limited to be used only on infrastructure provided by Amazon it may not be solution for everybody.

18

4 Libvirt virtualization API


The goal of libvirt is to provide a common and stable layer sufcient to securely manage domains on a node, possibly remote. Libvirt is intended to be a building block for higher level management tools and for applications focusing on virtualization of a single node (the only exception being domain migration between node capabilities which involves more than one node). [6] In our terminology node is a host computer, and domain is a guest computer.

4.1

Architecture

Figure 4.1: libvirt architecture diagram Libvirt as shown in gure 4.1 consists of public API for external applications and driver API that includes drivers for communication with various hypervisors. Drivers implement same API for communicating with libvirt through which they provide access to hypervisor they represent. Goal is to provide one way how to do something in libvirt, so when the external application calls libvirt function that starts the VM, this function will be called 19

4. L IBVIRT VIRTUALIZATION API same way no matter what underlying driver is used to access hypervisor. The only exception is when application starts to communicate with libvirt, it rstly uses a special URI1 which identies which driver to use. Driver can refer not only to hypervisor running locally but also to remote hypervisor either running on its own, or one provided by remote libvirt as shown in gure 4.2.

Figure 4.2: libvirt remote architecture diagram

4.1.1 Guest conguration management Guests conguration is described by XML2 les. It can be persistent or transient. While persistent guest congurations are dened in libvirt permanently until user decides to undene them from it, transient congurations are dened only for time the guest is running or until the host on which they are dened is restarted. To ease initial understanding of how libvirt works it provides virsh, CLI management utility, that is shipped with libvirt. virsh implements features that libvirt API provides in simple shell environment allowing user to interact with libvirt API directly. Besides VM conguration management it provides also virtual network and storage management.

1. Uniform Resource Identier 2. Extensible Markup Language

20

4. L IBVIRT VIRTUALIZATION API 4.1.2 Storage management Concept of storage volumes and storage pools is used to provide management of VMs storage. Following terminology applies: volume is a single storage volume which can be assigned to a guest, or used for creating further pools.[6] pool provides a means for taking a chunk of storage and carving it up into volumes.[6] Pool can be used to manage: a directory on hosts containing les that represents virtual disks, a physical disk, LVM3 group, iSCSI4 target or host adapter.

Conguration of storage pools as well as volume denitions are represented by libvirt in XML format. 4.1.3 Networking management Conguration of virtual networks is represented by XML les. In gure 4.1.3 we can see physical diagram of all possible virtual network congurations on host. Dividing by connectivity of these networks to physical network connected to hosts we differentiate the consecutively: Bridged networking: Guest A is connected to the same physical network as the host is connected using bridge on host. NAT-ed networking: Guest B is connected with interface eth0 to virtual network VNET0 which is forwarded by host to physical network using NAT5 isolated networking: Guest C is connected to virtual network VNET1 which isnt forwarded by host to any physical network directly. Guest C can so communicate only with network VNET1. routed networking: similar to NAT-ed networking this can provide access to physical network but without need to use NAT.

3. Logical Volume Manager 4. Internet Small Computer System Interface 5. Network address translation

21

4. L IBVIRT VIRTUALIZATION API

Virtual networks in libvirt on Linux are created by using Linux Ethernet bridge 6 which is protocol independent. 4.1.4 Access and user management The main way how to communicate with libvirt is using sockets that libvirt creates. There are two sockets from which one provides read only and second full access to libvirt. Beyond this there is in the time of writing no user management provided by libvirt.

4.2

API bindings for other languages

To develop own VMS on top of libvirt, it provides support for C and C++ directly and for some other languages it provides API bindings: C# - http://libvirt.org/csharp.html Java - http://libvirt.org/java.html OCaml - http://libvirt.org/ocaml/ Perl - http://search.cpan.org/dist/Sys-Virt/ PHP - http://libvirt.org/php Python - http://libvirt.org/python.html Ruby - http://libvirt.org/ruby/

6. http://www.linuxfoundation.org/collaborate/workgroups/ networking/bridge

22

5 VMS for Faculty of Informatics at Masaryk University (FI MU)


5.1 Faculty Administration (fadmin)

Fadmin is web based administration system that also provides user management at FI MU. Its purpose is to provide framework into which other applications that require user authentication at FI MU can be integrated. All data, except passwords of users, are stored in fast database which enables easy development of new applications. The application can be found at https://fadmin.fi.muni.cz/auth/.

5.2

State of virtualization deployment

At FI MU in time when we were starting with development of our software1 already existed some VMS. However usability of these systems was not very good. As main problems we have identied: lack of documentation on how to use these systems, underestimated propagation of services they provide, very limited number of conguration options for VMs, overall complexity of management system.

We have also deeply analyzed the two examples of their deployment in production. PV175, PV176 1 105/160 10/20 LaSArIS lab 2 7/10 4/5

virtualization hosts total VMs (active/inactive) VMs not used for teaching (project VMs) (active/inactive)

Figure 5.1: VMs deployment

1. around summer 2010

23

5. VMS FOR FACULTY OF I NFORMATICS AT M ASARYK U NIVERSITY (FI MU) 5.2.1 PV175/PV176 MS Windows Systems Management I./II. Deployed VMS was Microsoft System Center Virtual Machine Manager 2008 (MSCVMM) which manages guests running on Microsofts (MS) Hyper-V hypervisor. Primary goal of MSCVMM is to provide management of VMs for students attending this courses2 . On demand VMs were provided for staff and other students conducting research. Solution was optimized for MS Windows platform and was not very suitable for generic guest OS hosting. Despite effort of administrators the management of VMs is sometimes clumsy and lacks variety of options such as serial console for VMs or access to graphical console other than through MS ActiveX plug-in. 5.2.2 LaSArIS lab At Lab of Software Architectures and Information Systems (LaSArIS) we found virtualized infrastructure using two lab servers running VMware Server hypervisor. Aim of this solution was to provide ability to create VMs on demand as they were needed for research at lab. VMS was accessible only through web and was operated only by administrators of lab. On one server we found web server running along with hypervisor because of performance issues in virtual environment. 5.2.3 Generic servers for research projects During research we noticed non-neglectable number of physical servers which primary purpose was to host web of project and results of research. This web cannot be placed elsewhere because it requires special (sometimes obscure) software which possibly could cause troubles along with other webs on existing web servers. Another problem was need to have administrator privileges for some of these webs to get setup and maintain. Many of these servers were even operated outside designed server rooms where they can be easily physically compromised or can suffer from instability due environment. All these problems could be solved by use of VMs. Additionally this approach provides easier access to functions such as remote machine reset in case of guest OS freeze or other problem which could be done using VMS.
2. MS Windows Systems Management I./II. PV175 - https://is.muni.cz/predmet/FI/PV175 PV175 - https://is.muni.cz/predmet/FI/PV176

24

5. VMS FOR FACULTY OF I NFORMATICS AT M ASARYK U NIVERSITY (FI MU)

5.3

General requirements

From previous analysis we created list of following requirements: easy access to VMs graphical console, serial console and life cycle control, ability to use various processor architecture including x86, x86_64 and ARM 3 , easy to use intuitive user interface, VMs owned by group of users not only one user.

5.4

Special requirements

Additionally some special requirements raised. 5.4.1 PV090 UNIX - Seminar of System Management PV090 course4 , which is similar to PV175 and PV176 courses, plans to start using VMs for teaching. Main requirements were: custom network topology accessible only by students attending the seminar, nested virtualization support5 make use of own hardware for hosting VMs.

5.4.2 LaSArIS lab Main goal was simplication of access to VMs and ability to provide free capacities for other members of lab to run own VMs without assistance of administrators.

3. Advanced RISC(Reduced instruction set computing) Machine 4. https://is.muni.cz/predmet/FI/PV090 5. Ability to use assisted virtualization (2.2.2) inside VM.

25

5. VMS FOR FACULTY OF I NFORMATICS AT M ASARYK U NIVERSITY (FI MU)

5.5

Web application vs. Desktop application

To provide this application to widest range of users without limiting them on used OS, the only reasonable way was to deploy web based applications. Compared to desktop based ones which provides in general better performance in this case was neglectful compared to ability to access the application easily from wide range of devices. Along with universal access we gained easier maintainability of application.

26

6 fadmin/virt
Our implementation of management system is divided into two parts: libvirt as back-end for VM management on virtualization servers fadmin as front-end for VM management in database

Libvirt cannot store VM congurations in database and it leaves us two options where to store these data and how to retrieve them back, either by storing all information in libvirt on virtualization server and every time they are needed we retrieve them from virtualization server or storing all information in database and use ability to store information on virtualization servers only as cache but not as primary source.

The rst option provides simpler design and easy to achieve consistency of data, because they are stored only once. Second option although provides better performance, because data are stored in fast database and retrieving them from it is much quicker than querying virtualization server and parsing its response. However this decision brings consistency problem in which data on virtualization server and in database can be different. Keeping in mind that only right data are in database and not on virtualization server we can ignore this problem and implement cache-like behaviour of virtualization server.

6.1

Database design

VM congurations are stored in database provided by fadmin along with other virtualization related information. These are stored in tables separately from fadmins core tables. To provide integration with fadmins user management some tables are interconnected with fadmins core tables. While user management is handled by fadmin, our application takes care of limiting resources that users may use for their VMs. VM hardware conguration is stored in several tables which has to be joined in order to provide enough information to create VM conguration. Separate tables for disks and network adapters are used. This enables us to add multiple instances of same type of hardware without database layout modication. Other tables are used to store logs, list of virtualization servers, storage pools and other generic system and performance data. 27

6. FADMIN / VIRT

6.2

Custom scripts

To overcome some missing functionality in libvirt we have implemented this functionality using single purpose custom scripts. Some of them may render useless in future as the libvirt will implement their functionality. 6.2.1 VM serial console script Version of libvirt we were using1 was not able to provide reasonably secure and easy to setup way to access serial console of running VM over network we created own solution. Our goal was to provide serial console access over ssh which is both secure and comfortable for users (as they could login using same login and password as they use on other machines at FI MU). Libvirt creates names of devices that holds access to serial console of VM unpredictably, but after creation of such device it stores information about its location into conguration le of VM. As another part of solution we provided users with ssh access to virtualization server, but for security reasons we did not want them to have shell access on this server. To address these problems we divided solution into these two sub-problems: script that waits for newly started VM to create its temporary conguration from which we can read out path to serial console of guest on host,change permissions on this le to user whose VM it is and write down this info to the le and script that after successful login of user through ssh to virtualization server will jail user in restricted environment which will provide only our interface to access users serial consoles that are active2 .

Both script can be found on attached CD in directory scripts/serial_console/. 6.2.2 VM IP discovery script This script tries to guess the IP address of VM by parsing DHCP3 servers leases le on machine on which it is used and provides mapping of MAC addresses to IP addresses if it nds a match. Script is deployed on main DHCP server for FIs physical network to provide access to DHCP leases for VMs that are connected to network directly and

1. 0.8.2 2. list of these VMs is stored in le created by rst script 3. Dynamic Host Conguration Protocol

28

6. FADMIN / VIRT on all virtualization servers to provide information about IP address of VMs behind NAT or in isolated networks which are served by DHCP server running on virtualization hosts.

Script relies on fact that VM uses DHCP to congure its IP address. It can be found on attached CD in directory scripts/dhcp_leases_parser/. 6.2.3 VM hibernation script To protect virtualization servers from being overwhelmed by inactive VMs that are only consuming resources, we setup script to periodically check if such VMs exists. Detection of these machines is based on value of VMs lifetime provided by database. This values is by default 7 days from VMs start and can be indenitely times extended to 7 days from the time this was requested. If VMs lifetime reaches zero or negative values periodically running script will request VMs hibernation. By hibernation we mean pausing VM, dumping its memory to disk and effectively shut it down to free up resources of virtualization server. Next time the VM is started, the dumped memory loads back and VM execution is restored from the point when this happened. Sometimes guests OS may not survive resume from hibernation as great change in time of guests hardware clock occurs, but this happens only occasionally

6.3

Limiting resource usage

Ensuring virtualization servers stability we limit amount of memory that VMs can use to total of 110 % of host memory. This technique is called overcommiting and it wouldnt be safe if we hadnt used KSM that lowers memory usage by deduplicating memory pages. As not fully implemented feature that should protect virtualization servers from free memory exhaustion we deploy cgroups. When memory limit of group of processing that are used for virtualization is reached OOM4 killer tries to kill process inside this group instead of some other processes that may be important for hosts OS to function properly.

4. Out Of Memory

29

6. FADMIN / VIRT

6.4

Front-end

To make users feel familiar with interface we chose fadmin as it is already known by users. Simplication of whole interface was a bit harder but in the end it resulted in only three page design consisting of main page with list of users VMs and form to create a new VM, VM details page with details about selected VM and ability to change them and disk manager page which lists users virtual disks and shows whether they are attached to VMs.

Screenshots of pages can be found in Appendix A. Use of this interface is well documented in online help 5 which is referred from main page of application. 6.4.1 one-click VM creation For users the most interesting feature we have implemented is single-click VM creation. It allows creation of VM by just typing its name and optionally description to form on main page and then just hit submit button to create it. VM is instantly ready to be started with default conguration or to be modied by user.

5. https://fadmin.fi.muni.cz/auth/virth/help.mpl

30

7 Use case
To test usability and stability of our application we introduced it to PhD students and staff of FI and granting them access to use it. We extended resource limits for those interested in further testing.

7.1

Virtualization server Adikia

As virtualization server we have used abandoned server, named Adikia, with 4 dual-core Xeon processors, 32 GB RAM, around 2 TB of raw disk space and 2 Gigabit network cards. 7.1.1 Networking Adikia is connected to the network using 2 network cards: eth0 provides access for management and administration of server, eth1 provides networks for VMs and acts as default route for trafc from Adikia.

This way we separated management trafc (backups, VM migrations) from trafc of virtual machines and also put all VMs to network which is specificaly assigned to VMs at FI. 7.1.2 Disks Disks spins at 15000 RPM1 and are attaches using SAS2 interface. Currently used disks are: 2x 36GB disks, 4x 300GB disks and 4x 136GB disks.

Disks were arranged in RAID5 conguration to provide more capacity rather than more performance.
1. rotations per minute 2. Serial Attached SCSI(Small Computer System Interface)

31

7. U SE CASE 7.1.3 Software Fedora 12 as an operating system (OS), QEMU/KVM 0.12.3 as hypervisor and libvirt 0.8.2 for VM management.

We enabled KSM, which deduplicates the same memory pages and in order to enhanced security of both host and guests we congured SElinux with sVirt extension in enforcing mode. To access serial consols of running VMs we allowed users to connect to ssh server on host with restriction to run only application for connection to these consols.

7.2

Current deployment

In time of writing the work around 50 VMs were dened in database by around 30 users and 14 VMs were running on Adikia. Most of these machines run much longer than the standard lifetime of the VM and are meant as research machines for students and staff.

7.3

Stats and graphs

Since summer of 2010 we monitor Adikias memory usage, cpu usage and KSM effectiveness with MRTG 3 .

3. Multi Router Trafc Grapher - http://oss.oetiker.ch/mrtg/

32

7. U SE CASE 7.3.1 CPU

Figure 7.1: CPU usage in % (orange - system CPU, red - non-idle CPU) Looking at CPU usage graph (7.3.1) we can observe constant usage of CPU by system which is caused by KSM daemon scanning the memory. This high CPU usage by KSM was lowered in newer kernels. The graph also shows that VMs utilize at average same amount of CPU as the systems tasks. 7.3.2 Memory

Figure 7.2: Memory in bytes(light blue - inactive memory, green - active memory) Memory graph (7.3.2) shows us how much of memory pages were active (used) and how much of it was inactive (unused). While memory overcommit was set to 110 % of total memory there was still at least 5-8 GB of inactive memory that could be used. 33

7. U SE CASE 7.3.3 KSM Effectiveness of KSM was monitored in two graphs. First one (7.3.3) shows amount of unique memory pages used by VMs compared to shared memory pages. This graphs doesnt show how much memory we saved using KSM, but only shows how much memory was duplicit.

Figure 7.3: KSM memory usage in bytes (light blue - shared memory pages, orange - unshared memory pages) On second graph (7.3.3) we have amount of shared memory and amount of memory that these pages would take if they werent converted to copyon-write pages by KSM. Difference between these two values represents memory savings brought by KSM.

Figure 7.4: dark khaki - sharing of pages, green - shared pages

34

8 Conclusion
We have provided an introduction to the world of virtualization on Linux systems aiming to show virtualization management software (VMS), that provides easy access for regular users, that can benet from advantages of virtualization. We have also implemented VMS for users at FI MU hoping it will be useful and that it will unify process of implementing VMs at FI MU. As we showed in Use case chapter (7), users are apparently interested in our solution.

8.1

Future work

While fadmin interface is still not in nal state of its development, we will continue our work on nishing it and bringing it into nal release as soon as possible. At some point we would like to supersede VMS of Windows administration seminar and migrate users to our interface. Since the beginning of the applications development many new and interesting ideas appeared and we would like to implement them in future releases. World of software moves very fast and so does the development of libvirt API, that brings new features in every new version. We would also like to catch up with it and implement some of its new features.

35

Bibliography
[1] Tim Fisher. Iso le. http://pcsupport.about.com/od/termsi/ g/isofile.htm, May 2011. [2] Paul Menage. Cgroups. http://www.kernel.org/doc/ Documentation/cgroups/cgroups.txt, May 2011. [3] Susanta Nanda and Tzi cker Chiueh. A survey of virtualization technologies. Technical report, 2005. [4] Rusty Russell. virtio: towards a de-facto standard for virtual i/o devices. SIGOPS Oper. Syst. Rev., 42:95103, July 2008. [5] Borja Sotomayor, Ruben S. Montero, Ignacio M. Llorente, and Ian Foster. Virtual infrastructure management in private and hybrid clouds. IEEE Internet Computing, 13:1422, 2009. [6] Robert Warnke and Thomas Ritzau. qemu-kvm & libvirt. Books on Demand GmbH, Norderstedt, Germany, 2010. [7] Chris Wright and John Cooper. Kvm huge page backed memory. http://fedoraproject.org/wiki/Features/KVM_Huge_ Page_Backed_Memory, September 2009.

36

A Appendix - Screenshots

Figure A.1: Disk manager page

37

A. A PPENDIX - S CREENSHOTS

Figure A.2: Main page of application

38

A. A PPENDIX - S CREENSHOTS

Figure A.3: Page with VM details and conguration

39