Understanding “Host” and “Guest”

Memory Usage and Related

Memory Management Concepts
Kit Colbert
Sr. Staff Engineer, VMware
Disclaimer
This session may contain product features that are
currently under development.

This session/overview of the new technology represents

no commitment from VMware to deliver these features in
any generally available product.

Features are subject to change, and must not be included in

contracts, purchase orders, or sales agreements of any kind.

Technical feasibility and market demand will affect final delivery.

Pricing and packaging for any new technologies or features

discussed or presented have not been determined.
“These features are representative of feature areas under development. Feature commitments are
subject to change, and must not be included in contracts, purchase orders, or sales agreements of
any kind. Technical feasibility and market demand will affect final delivery.”
Agenda

Motivation
Define host and guest memory usage and ask some questions
Memory management concepts
Answer our questions
Best practices
Summary
Q&A
Motivation
Host and Guest Memory Usage
Using Host and Guest Memory Usage

Useful for quickly analyzing a VM’s status

– Coarse-grained information
– Important for prompting further investigation
Requires an understanding of memory management concepts
– Many aspects of host/guest memory interaction are not obvious
Define host and guest memory and
ask some questions
Host Memory Usage Defined

Host Memory Usage

– The amount of physical host memory in megabytes allocated to a
guest
• Includes virtualization overhead (e.g. hypervisor data)
Guest Memory Usage Defined

Guest Memory Usage

– The amount of memory in megabytes actively used by a guest
operating system and it’s applications
– The percent of total guest memory actively used by a guest
 Memory Terminology
memory size
total amount of memory
presented to a guest

allocated memory unallocated memory

memory assigned to memory not assigned
applications
Host memory usage
measures this, sorta…
active memory inactive memory
allocated memory recently allocated memory not
accessed or used by recently accessed or
applications used

Guest memory usage measures this

Some commonly asked questions

Why is host memory usage so high?

Why is host memory usage greater than guest memory usage?
Why is host/guest memory usage different than what I see inside
the guest?
Memory Management Concepts
Discussion of Concepts

Focus on high-level concepts, not implementation details

– Simpler and easier to understand
– Keep the fundamental ideas and drop the unimportant ones
Other presentations may present this information differently
– May discuss more implementation than concept
– That’s ok – different viewpoints for same thing!
Virtual Memory
“virtual” memory
Creates uniform memory address space
– Operating system maps application virtual guest
addresses to physical addresses
– Gives the operating system memory “physical” memory
management abilities that are transparent to
the application
hypervisor
Hypervisor adds extra level of indirection
– Maps guest’s physical addresses to “machine” memory
machine addresses
– Gives the hypervisor memory management
abilities that are transparent to the guest
Virtual Memory

VM
“virtual” memory “virtual”
Application App
guest
memory
guest

“physical” memory Operating “physical”

OS
System hypervisor
memory

hypervisor

Hypervisor “machine”
“machine” memory Hypervisor
memory
Application Memory Management

Starts with no memory

Allocates memory through syscall App
to operating system
Often frees memory voluntarily
through syscall
OS
Explicit memory allocation
interface with operating system

Hyper
visor
Operating System Memory Management

Assumes it owns all physical memory

No memory allocation interface App
with hardware
– Does not explicitly allocate or free
physical memory
OS
Defines semantics of “allocated” and
“free” memory
– Maintains “free” list and “allocated”
lists of physical memory
Hyper
– Memory is “free” or “allocated” visor
depending on which list it resides
How “Allocated” and “Free” Lists Work

Allocated List Free List

App

OS

Memory becomes “allocated” or

“free” based on which list (piece
of memory) has a pointer to it
Hypervisor Memory Management

Very similar to operating system

memory management App
– Assumes it owns all
machine memory
– No memory allocation interface
with hardware OS

– Maintains lists of “free” and

“allocated” memory
Hyper
visor
VM Memory Allocation

VM starts with no physical

memory allocated to it App
Physical memory allocated on
demand
– Guest OS will not explicitly
OS
allocate
– Allocate on first VM access
to memory (read or write)
Hyper
visor
VM Memory Reclamation

Guest physical memory not “freed”

in typical sense App
– Guest OS moves memory to its
Guest
“free” list Free List

– Data in “freed” memory may

not have been modified OS

Hypervisor isn’t aware when guest

frees memory
Hyper
– Freed memory state unchanged visor
– No access to guest’s “free” list
– Unsure when to reclaim “freed”
guest memory
VM Memory Reclamation Cont’d

Guest OS (inside the VM) Inside the VM

– Allocates and frees… Guest App

free list

– And allocates and frees… VM

– And allocates and frees…
VM OS
– Allocates…
– And allocates…
– And allocates… Hyper
visor
Hypervisor can’t reclaim memory
through guest frees!
What to do about VM memory
reclamation?
Why not just watch the free list?

Idea
– Hypervisor could identify memory used by guest for it’s free list
– Watch that memory area for changes and free physical memory that
backs guest memory added to free list
Problem
– Free list may be hard to identify
• Depends on guest type and may change from release to release
– “Free” is not always free
• “Free memory” may be used in buffer cache or elsewhere by guest
– Guest may not free memory when hypervisor wants it to
What can the hypervisor do to reclaim
VM memory?
Nothing
– Route taken by pretty much everyone except VMware
– No memory overcommitment! (Inefficient use of memory)
– May try to convince you memory overcommitment isn’t important
Something
– VMware-innovated techniques
– Supports memory overcommitment
• Efficient use of memory
• Strict resource control guarantees
Memory Overcommitment
VM 1 VM 2 VM 3

Hyper
visor

Aggregate guest memory greater than host memory

– Not enough physical memory to satisfy all VM memory needs
– Eventually VM may want physical memory when none is left
Why is memory overcommitment important?

Ensures physical memory is actively used as much as possible

– Guest VMs may have lots of unused or inactive memory
– No reason to back one VM’s unused/inactive memory with physical
memory if other VMs can use it
– Hypervisor can reclaim unused/inactive memory and redistribute to
other VMs who will actively use it
Increases VM-to-host consolidation ratio
– Each VM has a smaller physical memory footprint since only its
active memory is resident
– You can fit more VMs in the same amount of physical memory
VM Memory Reclamation
Techniques
Transparent Page Sharing (TPS)
VM 1 VM 2 VM 3
Simple idea: why maintain many
copies of the same thing?
– If 4 Windows VMs are running,
there are 4 copies of Windows
code Hyper
visor
– Only one copy is needed
Share memory between VMs when
possible VM 1 VM 2 VM 3
– Background hypervisor thread
identifies identical sets of memory
– Points all VMs at one set of
memory, frees the others
Hyper
– VMs are unaware of the change visor
Ballooning

Hypervisor wants to reclaim memory VM 1 VM 2

Guest OS is not aware of this App App
– Thinks it owns all physical memory
– Sits inside its own “box”, unaware it’s
running in a VM or that other VMs
are running
Goal: make the guest aware so it OS OS
frees up some of its memory
Solution: artificially create memory memory pressure
pressure inside the VM
Hyper
– “Push” memory pressure from the visor
hypervisor into the VM
– Use “balloon” driver inside the VM to
create memory pressure
Inflating Balloon
Guest: 3
6 pieces of
1. Balloon driver allocates memory are allocated
memory
2. Balloon driver pins Guest Balloon
App Driver
allocated memory
Guest
3. Guest may reclaim other free list
memory
OS
4. Balloon driver tells
hypervisor what List of
memory it allocated memory

5. Hypervisor frees machine Hyper

memory backing memory visor
allocated by balloon driver
6. Hypervisor now has more Hypervisor: 8
5 pieces of
free physical memory memory are allocated
Inflating Balloon Cont’d
Why can the hypervisor reclaim ballooned memory?
What in the VM assumes the memory has a specific value?
1. Balloon driver doesn’t Balloon
Driver
Because of a contract with
the hypervisor

2. OS doesn’t OS
Because the memory
allocated to an app List of
memory
3. Nothing in the VM relies Hyper
on the memory’s value for visor
correctness
Hypervisor can safely reclaim ballooned memory because the
VM does not rely on a particular value for that memory.
Inflating Balloon Cont’d
Guest OS swapping, a possible side effect of ballooning
Two possibilities for guest free memory:
Guest Guest Guest Balloon
free list free list App Driver

2. Needs to
swap!
OS

1. No swapping
necessary!
Hyper
visor

1. VM has lots 2. VM doesn’t

of free have much Guest OS chooses whether
memory free memory to swap or not!
Swapping

Hypervisor can swap VM memory App

– Swaps out to a per-VM swap file
– Transparent to the VM

Hypervisor swapping is a last resort OS

– Both TPS and ballooning preferred
• Low overhead for TPS
• Guest swapping due to ballooning
performs better than hypervisor swapping Hyper
• But both take time visor
– Hypervisor swapping quickly reclaims
memory, but is more expensive overall
When to reclaim memory
Quick Review

Physical memory is not reclaimed on a guest that is “free”

– VM may accrue lots of physical memory
Transparent page sharing is always running
– May or may not help reduce a VM’s physical memory consumption
Ballooning or swapping is the only other way to reclaim memory
– Both have performance overhead
– Want to use only when necessary
Question: when to invoke ballooning or swapping?
When To Reclaim Memory

Memory overcommitment
– Aggregate guest memory is greater than host memory
Not memory overcommitted
– Never reclaim memory from guest through ballooning or swapping
• No need to!
• (Transparent page sharing is always running)

Memory overcommitted
– Only reclaim once free physical memory drops below threshold
• Start ballooning when memory starts to fall toward threshold
• Start swapping as memory nears/passes threshold

* Assumes no VM or resource pool memory limits are set!

Answer our questions
High Host Memory Usage

Why is host memory usage so high?

– VM may have used lots of memory in the past
• On boot, Windows zeros all guest memory
• Starting several guest applications together, e.g., at boot time or when
building a big project, can lead to high memory usage
• Guest workload spike
– Host memory usage represents “high water mark” of guest’s memory
usage on non-memory overcommitted hosts
• No reason to reclaim memory from VM when not memory overcommitted
Host Memory Greater Than Guest Memory

Why can host memory be greater than guest memory?

– At some point in the past, the guest consumed a larger amount of
memory than it is actively using now
– With little to no memory pressure on the physical host, there is no
reason for hypervisor to reclaim memory from the guest
– The guest maintains a high host memory usage even though its
guest memory usage is small
Expected behavior
– Nothing to worry about!
Host/Guest Memory Usage Different
Inside Guest
Why is host/guest memory usage different than what I see inside
the guest OS?
– Guest memory
• Guest has better visibility while estimating “active” memory
• ESX active memory estimate technique can take time to converge
– Host memory
• Host memory usage doesn’t correspond to any memory metric within the
guest
• Host memory usage size is based on a VM’s relative priority on the
physical host and memory usage by the guest

Again, this is expected!

Best Practices
Host Memory Usage

Large host memory usage OK

– Expected behavior much of the time!
Host memory usage shouldn’t limit guest memory usage
– Host memory usage should be large enough to accommodate guest
workload
– Don’t over commit host memory forcing the guest to continuously
swap
Use shares to adjust relative priorities among VMs when memory is
overcommitted
– Host memory usage is based on shares when overcommitted
Guest Memory Usage

VM’s memory size should be slightly larger than the average guest
memory usage
– Accommodates workload spikes without guest swapping
– Hypervisor will handle excess host memory usage
– Larger VM memory size means more overhead memory
Summary
Summary

Host/guest memory usage is not as simple as it looks!

– Represents complex set of data
– Requires understanding of memory management concepts
– Many aspects that are not obvious!
Memory Management Concepts
– Guest doesn’t “free” memory in a typical sense
– Hypervisor isn’t aware of memory a guest has “freed”
– Hypervisor uses transparent page sharing, ballooning, and swapping
to reclaim memory
– Ballooning and swapping used only when host is memory
overcommitted!
Summary Cont’d

Host/guest memory can be very useful with proper knowledge

– Quick way to determine the status of VM
– Important tool to prompt further investigation
– Requires good mental model of memory management concepts
• Hopefully we’ve helped you with that today!
Q&A
Kit Colbert
Sr. Staff Engineer
VMware
 Thank you for coming.

Rate your session and

watch for the highest scores!
Backup Slides
Measuring Active Guest Memory

Guest self-measurement won’t work

– Each guest uses different method to estimate active memory
– Can’t compare activeness data from Windows guest with Linux guest
– Comparable estimate important for making allocation decisions
Statistical Sampling
– Select a subset of memory at random
– Compute the percentage of pages accessed in a minute
– Average percentage gives an estimate of “active” guest memory
Ballooning and Swapping

Ballooning
– Provide adequate swap space in guest OS
• Ballooning can lead to high guest memory swapping
• Ballooning not effective if a guest can’t swap
– Limit maximum balloon size if lots of guest memory pinned
• Pinned memory cannot be swapped
• Use sched.mem.memctl.max to set maximum balloon size

Swapping
– Ensure adequate swap space on VMFS datastore
• Swap size = VM memory size – VM memory reservation