Where Logs Hide: Logs in Virtualized Environments

By Dr. Anton Chuvakin

WRITTEN: 2008

DISCLAIMER:

Security is a rapidly changing field of human endeavor. Threats we face literally

change every day; moreover, many security professionals consider the rate of
change to be accelerating. On top of that, to be able to stay in touch with such
ever-changing reality, one has to evolve with the space as well. Thus, even
though I hope that this document will be useful for to my readers, please keep
in mind that is was possibly written years ago. Also, keep in mind that some of
the URL might have gone 404, please Google around.

This paper describes log management in virtualized environments-its

challenges and opportunities. We will cover the similarities and differences in
logging for virtualized environments versus physical environments.

Introduction to Logging

A beaten maxim proclaims that “knowledge is power,” but where do we get our
knowledge about information technology (IT) components such as computers,
networking gear, application frameworks, SOA web infrastructure and the like?
The richest sources of such information that is always available but often
overlooked are the logs and audit trails that are produced by these systems and
applications. Through logs, audit trails and various alerts, information systems
often give signs that something is amiss, or an event logged in the log files
provides insight into future problems. Logs can also reveal larger weaknesses
–that may affect regulatory compliance and even IT governance, and, by
extension, corporate governance. However, more often than not, it’s difficult to
extract information from log files and distil the data into useful and usable or
actionable information.

To start from the very high level, logs equal accountability. Wikipedia
defines accountability as " a concept in ethics with several meanings…often
used synonymously with such concepts as answerability, enforcement,
responsibility, blameworthiness, liability and other terms associated with the
expectation of account-giving." There are many other mechanisms for
accountability in an organization, but logs are the most prevalent. And if your IT
staff is not accountable, neither is your business. Unless you take logs
seriously, you may be sending out the message that your organization shuns
accountability. Along the same lines, logs are also immensely valuable for
meeting regulatory compliance. Many recent US laws including HIPAA, GLBA,
Sarbanes-Oxley (SOX) and others have requirements related to log auditing and
the handling of those logs (see my papers “Log management in the age of
compliance” and “Six Mistakes of Log Management”)

Let’s take a look at virtualization and what it means in terms of log collection
and retention.

Introduction to Virtualization

Server virtualization makes it possible to combine multiple diverse systems onto a single
hardware platform, thus shrinking server, storage and networking costs, reducing
power requirements (through a direct decrease in consumed energy and
cooling costs), increasing utilization of existing computing resources and
improving productivity. The impact is significant; Garter reports savings of up to
25 percent due to server consolidations and decreased hardware purchases.

Virtualization also simplifies server provisioning, increases the average

workload per server and shrinks server administration workloads, reducing the
amount of required hardware purchases. Organizations save money through
better hardware utilization. Simplified backup and recovery is also possible,
because virtual machines can be brought back online much faster than physical
machines. Virtual platforms and their management tools enable the smooth
transition from a physical to a virtual environment.

It all sounds good, but what happens to logs, logging and log management
when IT environments are virtualized?

Logging Meets Virtualization

As one can guess, virtualization platforms present new sources of logs to

manage. In addition to having new log information to collect and analyze, we
new challenges to logging and log analysis arise, such as the potential need to
review access logs collected while virtual machine images were inactive. In
addition, new opportunities for log management are also present, such as
ensuring new virtual images are pre-configured with central logging
capabilities. There may be ways to use logs to solve new problems, such as
monitoring health and uptime status of virtual platforms and application stacks.
The ubiquitous nature of log management allows the development of new
operational, security and compliance solutions for virtual infrastructures using
the tools we already have.

What stays the same?

First, let’s review what stays the same. A virtual server is still a server –
complete with operating system and applications, and logs that must be
collected, retained (for security and compliance reasons) and analyzed, just as
they do in “physical” environments. The rest of IT infrastructure stays the
same: Routers still route network traffic, switches perform switching, firewalls
and other network security devices perform their functions on network traffic,
etc. In other words, IT infrastructure with virtual platforms, hosts systems and
guest systems are largely the same as those with all physical elements; with all
the usual logging that needs to be managed. Similarly, networking between
guest systems running on a single virtual platform resembles networking
between physical machines, and needs to be monitored and audited just like on
a physical network.

In a virtual environment, servers are still provisioned, modified and configured

by system administrators, and of course accessed and utilized by end users.
Such activities create audit trails that are collected and reviewed in just the
same manner as are physical environments. For example, if an MS SQL
database server is running on Windows 2003 operating systems, but this
Windows system itself sits atop of a Linux-based VMWare host, both Windows
logs and MS SQL audit trails must be collected and analyzed for access
violations, new user accounts, data access attempts or unauthorized changes
to database structures.

In short, the advent of virtualization is not a reason to throw away tools that
work for you in physical environments. They will continue to deliver value and
help your IT and business to operate efficiently, be secure and compliant with
relevant regulations, especially given the fact that the future belongs to a mix
of physical and virtual environments.

What changes?

On the other hand, virtualization has brought a lot of new technologies (all with
their own logs) as well as new problems for IT departments to solve. Such
problems might not have any equivalent in the physical world, where “a server”
always meant “a piece of hardware” plus “an operating system” plus “one or
more of user applications” running on it—a worldview that virtualization is
making obsolete.

A virtual platform comprises a hardware platform, operating system and a

hypervisor, or virtual machine software that enables other systems to run on
top of it. Such a setup gives way to several major changes:

1) New logs include hypervisor application logs, record virtualization-specific

activity logs (new guest image creation, guest operating systems startup, patch
access, etc). These logs must be understood by log management tools as well
as the virtual machine administrators.

2) Aggregation of servers on one hardware platform calls for stricter availability

monitoring. Indeed, recovering a virtual machine image from backups might be
relatively simple, but availability monitoring must still be stringent. Log
management tools and possibly other monitoring tools must be deployed with
real-time alerting to notify the administrators of impending fault and possible
crashes or problems.

3) Stricter host platform security monitoring will help reduce the risk of
breaches into the virtual infrastructure world. Extensive logging, log collection
and analysis will allow thorough incident investigation. Such logs include
security incident response and forensics activity across virtual farms, as well as
across massive SAN arrays that house virtual machine images.

4) Management tools that enable organizations to deploy and control virtual

server farms introduce their own logs and logging challenges. For example,
logging the activities of server administrators means recording the provisioning,
configuration and status changes of virtual machines performed via such
management tools.

5) As virtual machines proliferate across an enterprise’s IT infrastructure,

physical hosts are retired, an new technologies must be used to secure and
manage the virtual machines. Activity such as patching, management,
configuration and deployment and migration of virtual machines must be
logged and monitored, just like in a physical environment.
Controlling and auditing these virtualization-specific activities makes another
excellent use case for logs.

Beware of Rogue Virtual Machines

Finally, “rogue” virtual machines pose a unique security problem. If users
provision their own virtual machines and their own guest systems, tracking
such activities across the organization, presents a worthy challenge – for
example, if a unauthorized application, that would otherwise be banned, runs in
its own virtual image, enforcing the security policy becomes harder since
endpoint monitoring tools might not see through the virtualization veil. Rogue
machines deployed “in the cloud” via Amazon web services, for example,
present the ultimate challenge of this type. If a system resides on somebody
else’s virtual platform in the cloud, the chances of getting evidence of activities
on such systems becomes next to impossible.

Logging and Virtualization—The Good, the Bad and the Ugly

At this point it should be clear that changes that IT staff must face as
virtualization becomes a reality in the datacenter are indeed massive. For IT
staff tasked with logging activity across the infrastructure, these changes can
be good, bad or ugly:

1) They’re good because it’s easier to provision systems with centralized

logging already enables. IT staff can also retrofit other systems by adding
logging to the virtual image of that system. Moreover, current logging tools
such as LogLogic will still work – a major good point.

2) They’re bad—or partly bad—because there are new logs to collect and
analyze and new activities to track and monitor. Virtual machines must be
closely watched for availability and security issues and to ensure they comply
with policies and regulations.
4) They’re “ugly”—sometimes, because unmanaged virtual machines can pop
up on the organization’s systems or even in the cloud, violating IT policies and
presenting significant enforcement and investigation challenges.

Logs Help Virtualization

In addition to being affected by it, logging and log management can also
augment virtualization projects, especially in the areas of security, compliance
and manageability..

Security: Logging creates a trail of accountability for users and, especially,

those privileged to access the underlying hypervisor. Tracking access to virtual
machine hosts system and inactive guest images creates a trail that can be
used for monitoring and auditing, as well as investigations for cybercrime or
insider abuse. Perusing logs for security-relevant failures, such as missing
controls, unauthorized access or unapproved changes is just as helpful in a
virtual environment as it is in a physical environment.

Compliance: Recent mandates such as PCI DSS and others require logging, log
collection and retention, log analysis and review, and log protection. For
example, logging is one of the 12 PCI requirements (Requirement 10), whether
the environment is physical or virtual. Hence, logs from virtual machines must
be given at least as much importance as logs from physical environments

Manageability: Administrators and system operators benefit from logging, as

well. Monitoring for failures and errors as well as general virtual machine health
is not possible without effective log management.

Conclusion
Along with all the promise and benefits of a virtual infrastructure comes
significant change, requiring new ways for organizations to collect and manage
logs. However, existing log management tools such as LogLogic log
management appliances can still be leveraged to address these new logging
challenges, and to optimize, secure and bring into compliance newly virtualized
IT infrastructures.

ABOUT THE AUTHOR:

This is an updated author bio, added to the paper at the time of reposting in
2009.

Dr. Anton Chuvakin (http://www.chuvakin.org) is a recognized security expert in

the field of log management and PCI DSS compliance. He is an author of books
"Security Warrior" and "PCI Compliance" and a contributor to "Know Your
Enemy II", "Information Security Management Handbook" and others. Anton
has published dozens of papers on log management, correlation, data analysis,
PCI DSS, security management (see list www.info-secure.org) . His blog
http://www.securitywarrior.org is one of the most popular in the industry.

In addition, Anton teaches classes and presents at many security conferences

across the world; he recently addressed audiences in United States, UK,
Singapore, Spain, Russia and other countries. He works on emerging security
standards and serves on the advisory boards of several security start-ups.

Currently, Anton is developing his security consulting practice, focusing on

logging and PCI DSS compliance for security vendors and Fortune 500
organizations. Dr. Anton Chuvakin was formerly a Director of PCI Compliance
Solutions at Qualys. Previously, Anton worked at LogLogic as a Chief Logging
Evangelist, tasked with educating the world about the importance of logging for
security, compliance and operations. Before LogLogic, Anton was employed by
a security vendor in a strategic product management role. Anton earned his
Ph.D. degree from Stony Brook University.