The Essentials Series

Virtual Security
Concerns &
Solutions
sponsored by

by Greg Shields
Article 1: Understanding & Improving Hypervisor Security ..........................................................1 
What Is the Hypervisor? ......................................................................................................1 
Why Is Hypervisor Security Important? ..............................................................................2 
The Security Implications of Hypervisors ...........................................................................3 
Hypervisor-Based Attacks: An Example .............................................................................3 
Preventing Hypervisor-Based Attacks .................................................................................4 
Virtual Machine Eggs in Your Hypervisor Basket ..............................................................5 
Article 2: Understanding & Improving Virtual Machine Security ..................................................6 
The Added Issues of Virtual Machine Dormancy ...............................................................7 
Agentless and Agent-Based Tools .......................................................................................9 
New States Equals New Needs ............................................................................................9 
Article 3: Understanding & Improving Virtual Network Security ................................................10 
The Effect of Machine Migrations on Networking............................................................12 
Implications of Cross-Virtual Machine Traffic .................................................................13 
External Agent-Based Approaches Overcome Virtual Transience ...................................13 

i
Copyright Statement
© 2008 Realtime Publishers, Inc. All rights reserved. This site contains materials that
have been created, developed, or commissioned by, and published with the permission
of, Realtime Publishers, Inc. (the “Materials”) and this site and any such Materials are
protected by international copyright and trademark laws.
THE MATERIALS ARE PROVIDED “AS IS” WITHOUT WARRANTY OF ANY KIND,
EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE,
TITLE AND NON-INFRINGEMENT. The Materials are subject to change without notice
and do not represent a commitment on the part of Realtime Publishers, Inc or its web site
sponsors. In no event shall Realtimepublishers.com, Inc. or its web site sponsors be held
liable for technical or editorial errors or omissions contained in the Materials, including
without limitation, for any direct, indirect, incidental, special, exemplary or consequential
damages whatsoever resulting from the use of any information contained in the Materials.
The Materials (including but not limited to the text, images, audio, and/or video) may not
be copied, reproduced, republished, uploaded, posted, transmitted, or distributed in any
way, in whole or in part, except that one copy may be downloaded for your personal, non-
commercial use on a single computer. In connection with such use, you may not modify
or obscure any copyright or other proprietary notice.
The Materials may contain trademarks, services marks and logos that are the property of
third parties. You are not permitted to use these trademarks, services marks or logos
without prior written consent of such third parties.
Realtime Publishers and the Realtime Publishers logo are registered in the US Patent &
Trademark Office. All other product or service names are the property of their respective
owners.
If you have any questions about these terms, or if you would like information about
licensing materials from Realtime Publishers, please contact us via e-mail at
info@realtimepublishers.com.

ii
Article 1: Understanding & Improving Hypervisor Security
A successful virtualization deployment delivers a host of compelling benefits to the IT
organization—reduced costs from power and cooling, a right-sizing of resource demands to
supply, and improved operational agility. But there’s a hidden risk that lurks in, around, and
through virtualization environments that only the most observant are aware of today. That quiet
killer is virtualization’s hypervisor itself.

What Is the Hypervisor?

Although not all virtualization solutions leverage a hypervisor, those most commonly used for
the processing of production workloads typically leverage the use of one. Virtualization
platforms today that make use of a hypervisor are VMware ESX and Virtual Infrastructure,
Microsoft Hyper-V, and Citrix XenSource, among others. Within these virtualization
architectures, the hypervisor is a thin layer of code that resides between the physical hardware
and any virtual machines that are hosted on the virtual server itself.

Figure 1: A hypervisor is used by many virtualization solutions; it is positioned between physical hardware
and residing virtual machines.

Figure 1 shows a simplistic graphical representation of this layering. With virtualization

solutions that make use of them, the job of the hypervisor is effectively to “proxy” requests by
virtual machines to resources that are available on the physical hardware of the server.

1
Why Is Hypervisor Security Important?
Organizations that leverage virtualization do so because virtualization’s hypervisors enable a
level of commonality between virtual machines across all hypervisors of the same platform.
When a virtual machine is created atop a virtualization solution’s hypervisor, that virtual
machine is functionally similar to other virtual machines in terms of its emulated hardware
composition. This commonality allows virtual machines from one host to function when
relocated to another. It also allows virtual machines collocated on a single host to share the
physical resources of that host while maintaining hard boundaries between individual virtual
machines.
Yet at the same time this introduction of a singular hypervisor across the IT environment adds a
common codebase atop which resides all the virtualized computing resources of that IT
environment. Should a security vulnerability in that common codebase be exploited through the
use of malicious code, it would put each and every virtual machine at risk for failure. The
pervasive nature of the hypervisor across all virtual hosts means that a single piece of malicious
software that has the capability to compromise one hypervisor instance can easily compromise
others.
Figure 2 illustrates how the introduction of this single instance of replicating malware can
quickly exploit each hypervisor in the IT environment. The result of this replication is the
potential for failure of all the virtual machines atop each hypervisor.

Virtual Virtual Virtual Virtual Virtual Virtual Virtual Virtual Virtual

Machine Machine Machine Machine Machine Machine Machine Machine Machine
Replicating
Malware Hypervisor Hypervisor Hypervisor

Physical Hardware Physical Hardware Physical Hardware

Figure 2: The hypervisor’s pervasive nature within the virtualized environment makes it a single point of
failure in the case of malicious compromise.

0 In short, the move to virtualization effectively puts an IT environment’s virtual machine eggs in a
single hypervisor basket.

2
The Security Implications of Hypervisors
Although hypervisor-based security exploits today remain mostly within the realm of academic
research, their potential for massive environment failure makes them an important consideration
for environments that have made the jump to virtualization. Hypervisors are by nature software-
based code, which infers that they require occasional updating to patch security holes and protect
against discovered vulnerabilities. Examples of this need have already been seen with
hypervisors in production use today. For example, the security group Secunia has reported as of
the time of this writing:
• 19 advisories and 128 vulnerabilities for VMware ESX Server 3.x
• 7 advisories and 14 vulnerabilities for XenSource Xen 3.x
As with traditional operating systems (OSs), most hypervisor-based vulnerabilities are
constrained by a need for administrative rights in order to accomplish their mission. Without the
proper administrative rights, the hypervisor remains off-limits to exploitive code. Yet the level of
maturity associated with rights and privileges in virtualization environments may not be to the
same level as those already in place for Windows. If IT environments do not properly lock down
privileges and console access to virtual environments with the same level of care seen with
directory services, this situation exacerbates the potential for successful code exploitation
through malicious software.

0 Essentially, IT environments must use the same level of due diligence with virtual environment
security as with OS and directory services security. If security controls for the virtual environment are
lax, there is a greater chance for infection.

Hypervisor-Based Attacks: An Example

The situation described up to this point is perhaps best illustrated through a real-world example.
In this example, let us examine a typical healthcare environment that has recently completed a
substantial migration of server resources to virtualization. This company has standardized on a
single virtualization platform due to the associated cost savings as well as the benefits to systems
management.
All hosts run the same version of the virtualization solution’s software. However, due to the
timing of the virtualization project, hosts were brought online over a period of months. Not
planned for in this environment was the need to monitor the versions of the hypervisor code
itself. Although all hosts run the same version of the virtualization software, hosts that were
installed later in the project quietly run a later build of the version. The changes associated with
the newer build versions were incorporated to protect against a known network-based
vulnerability.

3
As with many healthcare organizations, the example network here operates with a centralized
data center but a federated operating model. Multiple offices and hospitals share the use of the
network and services on that network. Due to this federated model, security controls are not
cohesively applied in all areas. At some point during its operation, an instance of replicating
malicious software was introduced into the environment which infected earlier builds of the
virtualization software. The end result was a loss of virtual machines atop those hosts that were
not properly secured.

Figure 3: Patch levels are critical even with hypervisors. On the left is an up-to-date hypervisor that survives
the attack, while on the right is one that does not because it is not up to date.

Preventing Hypervisor-Based Attacks

The previous example is a simplistic one that is quickly resolved by applying the right patches.
But hypervisor-based attacks can come from many possible vectors. Adding to this complexity,
vulnerabilities are likely to impact every virtualization solution available on the market today.
With many environments leveraging the use of multiple virtualization solutions—each requiring
its own vendor-specific tool for scanning—administrators are likely to see a significant added
workload associated with verifying their virtual security.
In order to best recognize and prevent hypervisor-based attacks, IT organizations should consider
the following suggestions for improving their security posture for virtualized environments:
• Identifying hypervisor vulnerabilities with vulnerability assessments. Hypervisor-based
attacks exist on-disk and on the network in much the same way that traditional malware
does with physical OSs. Thus, locating potentially unwanted software means identifying
their known heuristics. Using an effective vulnerability assessment solution,
organizations can assess hypervisors for missing updates and properly plan change
control windows to mitigate risks. Although the proper defense against a hypervisor-
based attack is the installation of necessary patches, assessing their risk to the
environment is equally critical.
• Prevent network exposure. Virtualization environments enjoy a much greater level of
agility in terms of network configuration. With essentially all virtualization solutions
including virtual networking components inside the virtual server, it is possible to restrict
traffic to those connections that have access to the hypervisor. Preventing all but the most
critical of network connectivity to the hypervisor itself goes far in preventing network-
based attacks from impacting the hypervisor.

4
 • Segregate management networks. One way in which the previous points can be
accomplished is through a logical separation of management networks from those used
by virtual machines themselves. Virtual machines and their network traffic typically have
no need for direct network access to those networks used for hypervisor management.
Segregating traffic to networks exclusively used for hypervisor management limits
hypervisor exposure.
• Consider agent-based approaches. Although agentless approaches may involve fewer
requirements for management, there are certain needs that can only be fulfilled through
agent-based approaches. Scanning, patching, reporting, and suggesting improvement
actions work best when agents leverage administrative access for digging deep into
system processes. This is particularly the case when other securing mechanisms—such as
those discussed in the earlier bullet points—are being used in the environment.

Virtual Machine Eggs in Your Hypervisor Basket

Virtualization indeed promises huge improvements to cost and management flexibility but with
the silent risk associated with its common hypervisor code across the environment. Thus, hosting
all your virtual workloads atop that singular entity brings the need for added diligence in terms of
security for the hypervisor itself. Environments that don’t move now to protect this pervasive
interface will see the risk of large-scale virtual machine failure in the case of an infection event.

5
Article 2: Understanding & Improving Virtual Machine
Security
Article 1 of this series discusses some of the new and powerful hypervisor-based vecAtors for
infection that arrive when an IT organization makes the move to virtualization. That article
discusses how the hypervisor easily becomes a quiet risk to the unprepared organization. But the
hypervisor itself is only one facet of the story. Along with its benefits, the move to virtualization
brings about added risks associated with your virtual machines themselves.
Physical servers traditionally tend to operate in the Powered On state for essentially their entire
operational life cycle. Once built, a physical server is rarely rebooted and almost never down for
an extended period of time due to the always-on needs for its services. On the contrary, while
virtual machines can be considered functionally equivalent with traditional physical machines,
their easy-to-create nature and single-file composition makes them much more likely to exist in
states other than On and Operational. For example:
• Virtual machine templates, which are machines themselves, are rarely powered on.
• Single-use virtual machines may linger on-disk past the point of their usefulness. Their
presence on-disk may ultimately be forgotten over time.
• Virtual machines based on templates rather than built from the ground up tend to not
exist in the Build state like what is usually required in the early stages of physical server
creation.
Figure 1 shows a graphical representation of these differences in states between the two machine
types.

Figure 1: Virtual machines and physical machines tend to spend their time in much different states.

The most problematic of these states is the situation in which a virtual machine for one reason or
another finds itself in an extended state of powered down. While powered down, a virtual
machine is little more than a file on a disk. The accumulation of these files across the IT
environment can grow to become a critical issue for organizations that lack the capability to
inventory their state and keep them patched.

6
The Added Issues of Virtual Machine Dormancy
With virtualization, most IT organizations are aware of the ease of creating new virtual
machines. Virtualization’s “copy and paste” process for accomplishing this task speeds the
process of bringing on new services to meet the demands of business. But at the same time, it
introduces a set of risks to the computing environment.
First is the problem of virtual machine spread. When the creation process grows absurdly simple,
IT organizations can quickly find themselves awash in dozens or hundreds of new virtual
machine instances to manage. Dealing with the management and licensing issues of a quickly
growing server count can be a major hurdle for the unprepared organization.
This first problem of spread is obvious but merely administrative. There is a more critical yet
often overlooked security-related issue associated with virtual machine inventory growth. That
issue is virtual machine dormancy. When virtual machines can be created quickly, there comes
the increased likelihood that some will be created for short-term uses and then later discarded.
These virtual machines when powered off exist as benign files in a data storage location but can
later become a vector for compromise when powered on.
The ever-changing nature of security threat prevention requires that functioning computers
operate with a specific and up-to-date configuration to protect them against malicious threats in
the environment. Consider the situation shown in graphical form in Figure 2. There it is easy to
see how a virtual machine can be created and used, then powered off and forgotten, only to be
later powered on without the proper security configurations needed to protect itself from
compromise.
As this example shows, protections were implemented for every operational computer in the
environment. But those protections were missed on the machine that was powered off during the
update. The dormant virtual machine—powered off and missing the update—when later powered
on, becomes a risk for compromise.
Time Line
Virtual Machine Virtual Machine Virtual Machine Virtual Machine
Created & Used Powered Off Powered On Compromised

Security Config Exploit

Updated Released

Figure 2: In the timeline of virtual machine dormancy, forgotten virtual machines are likely to be powered
down during the period when critical security configurations are updated. This results in the virtual machine
being unprepared for an exploit should it later be powered on.

7
When thinking about this issue of virtual machine dormancy, consider the answers to five
questions that probe how your environment handles security and configuration updates in
support of security:
• How are you patching? When your organization undergoes its regular patching process,
is that process done using manual tools or through automated systems? More importantly,
are those systems regularly probing all endpoints on the network to identify computers
that have missed the deployment of a particular patch? Effective patching systems have
the capability to regularly scan endpoints across the environment without the need for
installed agents on each computer. By not relying on the need for installed clients, rogue
computers that are not built to specification can be quickly identified and patched when
they arrive in the network. For dormant virtual machines, regular scanning and patch
updating ensures that any offline virtual machine is quickly updated before it can be
exploited.
• How are you updating security configurations? Security for computer systems is more
than simply ensuring that the correct vendor patches are correctly installed. Firewall
configurations, file system permissions, desktop and application lockdowns, and
allowed/disallowed executables are all examples of security configurations that must be
set and maintained over the life cycle of each computer in the environment. As with
patches, your process for updating security configurations must have the capability to
quickly identify and resolve areas of gap.
• How are you monitoring your baseline? For both of the previous needs, there is the
critical necessity of creating and maintaining a baseline across all computers. The process
to monitor for that baseline and individual machine alignment with that baseline must
include regular scanning irrespective of location, directory services membership, and
operating system (OS) type.
• How are you verifying that entitlements are current? One major factor in ensuring that
baseline is in validating the users and entitlements on machines in the environment. For
security as well as compliance purposes, IT organizations must ensure that dormant
virtual machines are not hosting expired users or entitlements assigned to those users.
Your management solution must have the minimal capability to peer into virtual
machines as they power on to identify and remove expired accounts before they can be
exploited.
• How are you identifying when machines are powered on? The most crucial component of
all these questions is in identifying when computers are brought online in the
environment. When machines both virtual and physical can be identified and adjudication
actions completed at the point of their power on, much of the risks associated with
dormancy disappear. Active identification of machines going from Off or Extended Off
states to On and Operational is a key need for environments that leverage virtualization.

8
Agentless and Agent-Based Tools
In answering these questions, IT environments that make use of virtualization must incorporate
tools that assess the risk of dormant virtual machines. That assessment requires two separate but
linked phases. First, the location of dormant virtual machines—those that exist only as files on a
disk—must be identified and inventoried prior to being powered on. By identifying dormant
virtual machines while they remain benign, it is possible to move to the second phase.
In the second phase, virtual machines must be protected from the point they are powered on until
they are considered healthy. This protection prevents the effect of external attacks from
impacting the unprotected machine until the proper protections can be put in place. In
accomplishing this mission, both agentless and agent-based tools can be used. Agentless
management platforms provide a mechanism to scan entire network environments irrespective of
location. Agentless management platforms integrate with common management frameworks at
the OS level as well as the virtual platform level to interrogate network endpoints for specific
information. When otherwise unmanaged network endpoints become active, only through the use
of agentless mechanisms can these endpoints be quickly identified.

 Part of that risk identification process should include the mapping of environment elements—virtual
machines being only one example—to their relevance to the business. Best-in-class tools provide the
ability to map network elements such as virtual machine composition to business applications. This
gives the IT organization an easy way to identify potential threats and prioritize them based on risk,
impact, and business priority.

Agent-based tools enable richer management capabilities of predetermined IT assets. With

agents installed to virtual machines at the time of their build, the agents have the onboard ability
to identify when they have been powered on. They can provide protections from within the
virtual machine while incorporating the necessary configuration updates as identified by
management servers.

New States Equals New Needs

With the potential for entirely new states of operation associated with virtual machines comes the
need for new tools. These tools ensure the proper configuration of machines even during
extended periods of being powered off. Only by leveraging the right set of tools that enables
monitoring for dormant machines can IT environments truly protect themselves against the
accidental exposure that can occur by simply powering on a forgotten or expired virtual machine.

9
Article 3: Understanding & Improving Virtual Network
Security
A virtual network is not the same thing as a physical network. You’ve seen virtual networks
before. These are the networks created inside most enterprise-class virtualization solutions for
connecting virtual machines to each other and the outside world. They create linkable virtual
switches that leverage a virtual host’s internal bus rather than its network card to connect virtual
machines to each other. Virtual switches can also be attached to physical interfaces in the server
in complex ways to route virtual machine traffic over multiple external networks. A graphical
representation of this is seen in Figure 1.

VM VM VM VM VM VM

Virtual Host Virtual Host

Figure 1: Virtual switches connect virtual machines to each other and the outside world.

Yet these virtual networks are indeed not the same as their physical equivalents. Though they
appear at first blush to provide much of the same functionality as seen with traditional networks,
these networks are special. The level of functionality they provide is quite different than you’re
used to seeing with your Cisco or Juniper devices. For example:
• Hubs vs. switches. Many virtual switches actually function as network hubs rather than
network switches. This support varies by vendor but can be a critical difference in how
you want to set up your virtual networking infrastructure within a virtual host. Switches
by nature have the ability to segregate traffic across ports, isolating traffic to only its
destination host. This process reduces the effect of improperly configured network cards
while eliminating the ability to promiscuously sniff for traffic intended for other hosts. If
your virtual network environment is not fully trusted, a hub-based virtual switch may not
provide the level of security you require.
• Layer 2 vs. layer 3. The virtual switch itself in most virtualization platforms today
operates at the OSI model’s layer 2 rather than layer 3. Most physical infrastructures
today have elevated their network configuration beyond the limitations of layer-2
switching due to the need for layer 3’s configurable virtual routing. With virtual switches
operating at layer 2 instead of layer 3, it can be difficult for virtual routing to be correctly
applied. Many virtual platforms leverage external mechanisms for the support of VLANs,
yet these tend to be proprietary in nature.

10
 • Access control. Virtual switches are often wide open in terms of access control, with no
capabilities for the assignment of per-port access control lists (ACL). This lack of
effective access control means that external traffic destined for virtual machines within a
virtual host can often only be controlled to the point of the virtual host itself—though
again some proprietary VLAN support is often available. This lack of internal ACLs also
means that virtual machine-to-virtual machine traffic within a virtual host is similarly
uncontrollable. When virtual machines need to interconnect with other virtual machines
via a controlled interface, often the only mechanism to support this need is through the
creation of a third firewall-type appliance that controls that traffic between each. This
firewall-type appliance can impact available resources needed for its processing and its
positioning must be carefully controlled as the migration of virtual machines occurs over
their operational life cycle.
• Switches vs. routers. Virtual switches are simply that: switches. Thus, if routing support
is necessary, an external device or virtual appliance that supports routing is necessary. As
discussed earlier, the same elements associated with router appliances can used for virtual
firewalling. As virtual switches, their native functionality often lacks the advanced
protocol support used by routers for the routing of network traffic.
• Layer-4 protection. Lastly, as layer-2 devices, virtual hardware today cannot protect
individual virtual machines from traffic arriving over specific ports. Layer-4 ACLs allow
traffic to route to a network endpoint but restrict that traffic to network ports that are
appropriate for the services being hosted atop that endpoint. Without built-in layer-4
protection, it is challenging to lock down the traffic routed to virtual machines to only
appropriate ports.
Care must be taken when incorporating the use of virtual machines in the environment when
security configurations require advanced network configurations. Those network configurations
may not be possible using virtual networking due to these inherent limitations.

11
The Effect of Machine Migrations on Networking
Another facet of virtual networking that must be considered is how virtual networking is
impacted by the effects of virtual machine migration. All enterprise-class virtualization platforms
include the ability to hot migrate virtual machines from one host to another. This hot migration
may occur for resource load-balancing purposes or as an automated action that occurs with the
loss of a virtual host. In either case, that migration has the potential to impact networking in a
number of different ways:
• Source and target host virtual networking mismatch. When a virtual machine is migrated
from one host to another, there is the possibility that elements of its networking
configuration at the virtual switch layer may not follow. Testing must be done to ensure
that the virtualization platform successfully transfers the virtual switch configuration with
the virtual machine during a migration event.
• External networking configurations. When external networking configurations are
targeted on a per-virtual host basis, there is the potential that migrated virtual machines
will begin service after a migration at their target virtual host without the correct
networking as seen by external network equipment. External networking equipment must
be configured with the flexibility to quickly “find” the relocated virtual machine and
reconfigure to support its new location.
• Speed of convergence and name resolution update. A situation that is arguably more
problematic in migrations that occur across subnets—such as in a disaster recovery
situation—the timing of network convergence and name resolution must occur at a rate
that is acceptable to connecting clients. Convergence embodies the amount of time
required for the network as a whole to recognize the changes to routing associated with a
change to the topology. Name resolution refers to the need for resolution mechanisms
such as DNS to properly update across the organization with the virtual machine’s new
location.
• Inappropriate cross-virtual machine communication. Lastly are the concerns of virtual
machines that should not have the ability to directly communicate with each other. A
more detailed example of this type of communication is provided in the next section.
These may be one business unit’s application that must not communicate with another’s
due to line-of-business regulatory or legal considerations. Organizations that leverage
virtualization and who operate under these restrictions must be aware of and compensate
for the potential for inappropriate cross-virtual machine communication.

12
Implications of Cross-Virtual Machine Traffic
The best way to explain the final bullet in the previous section is through the use of an example.
Consider a financial institution that runs an internal payroll application in addition to its front-
end finance tools. Although these may leverage the same technology for their operation, legal
and compliance regulations may require the isolation of these two applications from each other.
The internal finance application must never have direct—for example, non-firewalled or
uncontrolled—communication with the front-facing application.
When these two applications are hosted in a physical environment, it is relatively easy to ensure
that their communication remains controlled at the network layer. Setting ACLs within the
network infrastructure to prevent cross-communication is a trivial matter. Neither physical
instance has the need or the ability to relocate its position on the network.
Yet when those applications are virtualized, the situation changes significantly. Their initial
positioning within the virtual infrastructure may keep them segregated from each other.
However, over time, the virtual infrastructure’s load-balancing or high-availability features may
later result in both virtual machines landing on the same virtual host. Assuming that external
networking protections are put in place to appropriately isolate traffic as the machines migrate,
once both virtual machines are collocated on the same host, they may begin cross-
communicating via the internal virtual switch. Considering the limitations discussed earlier, this
situation could violate the required segregation of these two virtual machines. To resolve this
situation, techniques can be implemented such as advanced TCP/IP security on the host or the
use of a third-party agent-based solution. These solutions enable administrators to ensure that
cross-communication does not occur even as virtual machines migrate from host to host.

External Agent-Based Approaches Overcome Virtual Transience

Necessary in these instances are external tools that enable advanced networking functions similar
to those discussed earlier. When native tools are not enough to provide the level of network
agility required by the complex business, external tools may be necessary to manage network
connectivity between virtual machines. Segregated solutions that incorporate vision outside the
confines of the virtual infrastructure ensure that no matter where virtual machines go they are
always under management. Agent-based solutions within the virtual host ensure that actions can
be implemented based on the monitoring data gathered through the segregated tool.

13