Bash on Ubuntu on macOS
Takaya Saeki Takahiro Shinagawa
Yuichi Nishiwaki Information Technology Center
takaya.saeki@is.s.u-tokyo.ac.jp The University of Tokyo
nyuichi@is.s.u-tokyo.ac.jp shina@ecc.u-tokyo.ac.jp
Department of Computer Science
The University of Tokyo
ABSTRACT
Linux is a popular operating system (OS) as a production en-
vironment, while many developers prefer to use macOS for
their daily development. One way to deal with this situation
is running Linux in a virtual machine and the other is porting
development environments from Linux to macOS. However,
using a virtual machine has a resource sharing problem,
and porting environments is costly and often incomplete.
A promising approach to low-cost and seamless resource
sharing is to develop a Linux compatibility layer for macOS.
Unfortunately, existing methods of implementing OS com-
patibility layers lack robustness or flexibility. In this paper,
we propose a new architecture of OS-compatibility layers.
It allows user-space implementation of the core emulation
layer in the host OS to improve robustness while maintaining
the flexible and powerful emulation ability without heavily
depending on the host OS kernel by exploiting virtualization
technology. We implemented our approach and confirmed
that Ubuntu’s userland runs on macOS. Our experimental
results show that our approach has reasonable performance
for real world applications.
KEYWORDS
Operating System Compatibility, Virtualization
ACM Reference format:
Takaya Saeki, Yuichi Nishiwaki, Takahiro Shinagawa, and Shinichi
Honiden. 2017. Bash on Ubuntu on macOS. In Proceedings of APSys
’17, Mumbai, India, September 2, 2017, 8 pages.
https://doi.org/10.1145/3124680.3124733
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies
are not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. Copyrights
for components of this work owned by others than the author(s) must
be honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior specific
permission and/or a fee. Request permissions from permissions@acm.org.
APSys ’17, September 2, 2017, Mumbai, India
© 2017 Copyright held by the owner/author(s). Publication rights licensed
to Association for Computing Machinery.
ACM ISBN 978-1-4503-5197-3/17/09. . . $15.00
https://doi.org/10.1145/3124680.3124733
Shinichi Honiden
Department of Computer Science
The University of Tokyo
National Institute of Informatics
honiden@nii.ac.jp
1 INTRODUCTION
Linux is one of the most popular operating systems (OSs). It
is widely used as not only a desktop environment but also
a production environment. For example, 37% of the top 10
million websites were hosted by Linux [21], and more than
90% of 371,132 Amazon EC2 instances were Linux (56.4%
were Ubuntu) [15]. Therefore, many real-world applications
are developed for Linux and there exist a large number of
Linux binaries and distributions. On the other hand, a certain
number of software developers prefer to use macOS instead
of Linux as a development environment [17]. For these soft-
ware developers, there is a huge gap between the production
environment and development environment.
To fill the gap, two different approaches are taken. One is
to install Linux in a virtual machine (VM) on macOS. How-
ever, resource sharing between the guest and host OSs has
difficulties because resources are managed by the guest and
host OS separately. For example, the guest and host file sys-
tem trees are different. Inter-process communications (IPCs)
between the guest and host processes is not supported; there-
fore, pipe-based communication is impossible, for example.
Memory management is also performed by both the guest
OS and host OS independently, and therefore users have to
decide how much memory they “give” to a VM. The other
approach is to port applications and development environ-
ments from Linux to macOS. Although various kinds of tools
initially developed for Linux were ported to macOS, porting
software is very costly and often incomplete. For example,
Valgrind, a popular dynamic analysis tool, required almost a
year to be ported to macOS Sierra.
Apart from Linux on macOS, there exists a third approach:
using an OS compatibility layer. It allows applications imple-
mented for an OS to run on another OS natively with less
effort. This approach solves the problems that the above ap-
proaches have. Since the OS compatibility layer absorbs the
environment differences, developers will not be burdened
with porting efforts. In addition, since guest application’s
resources are managed by the host OS, resource sharing
between the guest and host OS is achieved smoothly. For
example, a guest application can use the current free mem-
ory of the host OS as much as possible, rather than a fixed
APSys ’17, September 2, 2017, Mumbai, India
amount of memory pre-allocated to the VM. After the appli-
cation terminates, the used memory will be freed naturally,
rather than the guest kernel keep the memory in the VM.
Windows Subsystem for Linux (WSL, also known as Bash
on Ubuntu on Windows) [8] and Linuxulator [9] are exam-
ples of this approach. They enable unmodified Linux applica-
tions to run on Windows and FreeBSD respectively. They are
both in-kernel subsystems that handle Linux system calls at
the privileged level. Since privileged software has full control
of software interrupts and page tables, they have enough
flexibility to increase compatibility as much as the original
kernel. However, they still have a problem. They lack robust-
ness since in-kernel subsystems are often unstable and they
are not isolated from the host kernel. In fact, WSL sometimes
causes the blue screen of death of Windows [7].
The easiest way to realize such robustness is to implement
OS compatibility layers in the user space. Cygwin [2] and
MinGW [16] are such products. These compatibility layers
are implemented in purely user space, therefore the bugs or
crashes are safely isolated from the host kernel. However, the
fact they live in the use space causes another big problem;
inflexibility due to the lack of kernel privilege. Since they do
not have privileged ability such as page table management
or interrupt handling, they give up binary compatibility and
adopt API compatibility. Therefore, they require guest OS’s
application to be recompiled with their tool-chains. They
also have performance issues. Taking Cygwin for example,
its fork implementation is struggling because of the inability
of the copy-on-write technique. MinGW gives up the full
compliance to the Linux kernel to gain performance. This
observation indicates that robustness and flexibility are in a
trade-off relationship in traditional approaches.
Library OSs, such as OSv [12], implement a kind of OS
compatibility layer that allows a guest binary to run on a
VM interface. However, they do not allow seamless commu-
nication between the guest and host processes. NOVA [18]
exploits a microkernel-like approach to run a part of a virtual
machine monitor (VMM) in user space to improve robust-
ness. However, it offers low-level machine interface rather
than system-call level abstract interface, therefore its imple-
mentation becomes more complicated than OS compatibility
layers. Barrelfish [3] uses user-level monitor processes to
implement system calls. However, its architecture does not
use an host OS, therefore the OS functionalities must be
implemented from scratch.
In this paper, we present a new architectural design of OS
compatibility layers. This design realizes both robustness
and flexibility by utilizing virtualization technology. In our
execution model, an individual VM is launched per guest pro-
cess, and a guest binary runs in a VM without the OS kernel.
System calls issued by the guest process are trapped and em-
ulated in a host process, called a monitor process, created for
T. Saeki, Y. Nishiwaki, T. Shinagawa, and S. Honiden
each guest process. The monitor process issues host system
calls to emulate guest system calls, and leverages virtualiza-
tion technology to trap software interrupts and manipulate
page tables for the guest process. This design allows most of
the emulation layer to be implemented in a user-space host
process, while having the flexible and powerful emulation
ability to achieve full binary compatibility. It also achieves
seamless communication between guest and host processes
and high portability of the emulation layer.
We implemented a Linux compatibility layer for macOS,
called Noah, based on our proposed design. Noah can run
unmodified ELF binaries for x86-64 Linux 4.6 on macOS
10.12 Sierra. We confirmed that the userland of Ubuntu 16.04
and Arch Linux run on Noah. We implemented emulation
for many Linux subsystems such as process management,
memory management, virtual file systems, networks and
signals. Noah currently supports 172 out of 329 Linux system
calls. Although the implementation is still in progress, Noah
can build Linux kernels on it and run several X11 applications.
Noah uses Hypervisor.framework [1] for its virtualization
component, so we do not need to modify the macOS kernel.
Our experimental results showed that the overhead of
Linux kernel build time on Noah was around 7.2% and the
exec system call was 2.4 times faster than that of macOS.
This paper is organized as follows. Section 2 shows re-
lated work. Section 3 explains the architectural design of
our approach and Section 4 describes the implementation of
our Linux compatibility layer for macOS. Section 5 presents
experimental results. Section 6 summarizes this paper.
2 RELATED WORK
Xax [6] abstracts an execution environment of native code
as a lightweight process, called picoprocess. A picoprocess is
created and mediated by an OS-specific monitor program,
and communicates with it via highly restricted system calls.
To set up a restricted execution environment, a picoprocess
has a boot loader and trampoline code inside it to communi-
cate with the monitor. A picoprocess is similar to our guest
process in that system calls are mediated by a host process.
However, the boot loader and trampoline code inside the
picoprocess incur ahead-of-time or just-in-time patching
procedure to restrict system calls. The implementation of
picoprocesses is also different from ours. Their Linux imple-
mentation uses ptrace to restrict system calls, suffering from
performance hit and complication on memory management
because ptrace does not allow direct manipulation of memory
map of the target process. Their Windows implementation
uses a kernel driver to mediate system calls, suffering from
kernel dependency and robustness decline. Our monitor pro-
cesses can directly trap system calls and other privileged
events without depending on the host kernel by exploiting a
hardware-based virtualization technology.
Bash on Ubuntu on macOS
Embassies [10] extends picoprocesses by adding rich func-
tions such as the IP protocol and user interface APIs. A suc-
cessor work [11] proposed an architecture to run POSIX
applications in it. In this architecture, a POSIX emulator
running inside a picoprocess offers POSIX ABI. The POSIX
emulator consists of several subsystems such as the virtual
file system and IP multiplexer, and is implemented based on
the Embassies ABI. This approach needs a large emulation
layer to realize the POSIX ABI by the Embassies ABI, which
is narrower than normal system calls. Embassies inherits the
same benefits and drawbacks of the picoprocess’s work.
Foreign LINUX (flinux) [20] is an emulation software to
run unmodified Linux binaries on Windows. It performs
binary translation against Linux binaries to allow its user-
space implementation without losing flexibility. In flinux,
system calls are intercepted via translated trampoline code.
However, memory layout configuration is not so flexible
because a guest process shares the memory space with the
corresponding host process. Additionally, it is significantly
slower than ours due to the online scan-and-patch process.
Dune [4] resembles our work in that they both run guest
programs in VMs with higher-level interfaces than the ma-
chine architecture. However, their goals are different; Dune
aims at providing user programs direct access to hardware
features, whereas we emulate the kernel interface of a dif-
ferent OS. We only use ring 3 in VMX non-root mode for
running its guest process, whereas Dune consists of pro-
cesses running in different rings and VMX modes.
Multiverse [13] has a goal similar to that of Dune. It gives
Linux applications the ability to utilize privileged Hybrid
Runtime (HRT) environment. Multiverse leverages a VMM
to trap privileged operations and emulate Linux behaviors in
HRT just like our architecture. However, Multiverse has no
host OS under a VMM and emulates privileged operations by
the actual Linux kernel that is running cooperatively with the
HRT. In contrast, our architecture handles them in the host
OS to realize resource sharing and seamless communication.
Barrelfish [3] is an OS that adopts the multikernel model.
Barrelfish has a similar architecture to our work in that the
core kernel component is implemented as multiple user-
space processes called monitor processes. However, Bar-
relfish adopts the message passing model for IPC to im-
prove scalability on multi-core and heterogeneous systems,
whereas our architecture uses the traditional shared memory
model to improve communication performance between a
guest process and the OS compatibility layer.
NOVA [18] is a redesign of VMMs from the viewpoint
of microkernels. Both Nova and our work put complicated
components, such as page table management, in user space
in order to improve robustness. However, they differ in that
Nova only isolates complex parts of VMMs from the host
APSys ’17, September 2, 2017, Mumbai, India
The host OS Guest VMs
load &
monitor process guest process
manage
user emulate trap
space system
callls system callls
kernel upcall
monitor
space
(no kernel)
kernel
VMM module
Figure 1: The design of our OS compatibility layer
kernel, whereas we isolate all kernel components from the
guest kernel and put them in a host process.
OSv [12] and our work are similar in that they both are
aiming at constructing from scratch a lightweight kernel
interface of Linux. However, their goals and implementations
are distinct. On one hand, OSv is a mere operating system for
VMs. OSv focuses on performance improvement rather than
compatibility; it even exposes non-POSIX interface to user
programs and is optimized to run faster with executables
specially modified for OSv. On the other hand, ours is not
an operating system but an OS compatibility layer. Its main
aim is to accomplish full compatibility with Linux, without
giving up as much performance as possible.
3 DESIGN
Figure 1 shows the design of our OS compatibility layer. It
consists of three components: a VMM module, guest VMs,
and monitor processes. The VMM module is a component
in the host OS kernel that provides a VM management in-
terface to user-space applications. We can exploit several
OS-standard VMM modules; for example, Linux has KVM,
FreeBSD has vmmapi, and macOS has Hypervisor.framework.
The guest VMs work as containers of guest binaries and are
used to trap access from guest processes. The monitor pro-
cesses are regular processes of the host OS that emulate
system calls and manage VMs through the VMM module.
A guest application is executed in the following way. First,
a monitor process corresponding to the guest application
is launched. It creates a VM through the VMM module and
loads the guest binary image into the memory space of the
VM. Then, the monitor process asks the VMM module to start
execution of the VM from the entry point of the guest binary,
thereby the control is passed to the guest process. While the
guest process is running, it will issue system calls. These
system calls are trapped by the VMM module and the VMM
module up-calls the monitor process. The monitor process
emulates the trapped system calls by using the system calls
of the host OS. The monitor process then returns the result
APSys ’17, September 2, 2017, Mumbai, India
of emulated system calls to the VM by way of the VMM
module, and the control is returned to the guest process.
In our design, a guest process runs in a VM and a monitor
process is created for each VM. Therefore, when a guest
process tries to create another process, the corresponding
monitor process first creates another monitor process, and
the new monitor process creates a new VM. In the case of
processes having a parent-child relationship like in UNIX,
the new monitor process clones the VM states of the original
guest process and set the states to the newly created VM.
Then, the original monitor process returns the result of the
process creation to the original guest process, and the new
monitor process passes the control to the new guest process.
One advantage of this design is that process scheduling can
be left to the host OS. A guest process can get control when
the corresponding monitor process is scheduled by the host
OS, therefore, in effect, the guest process is schedule by the
host OS as an ordinary process. This allows fair scheduling
among multiple guest and host processes. Another advantage
is that resource sharing between guest and host processes
becomes seamless. For example, they can share a file system
tree because both processes access the file system of the host
OS. They can also use a pipe to communicate with each other.
A pipe access from the guest process is converted to that of
the host OS by the monitor process, and the communications
are handled by the host OS. Therefore, the guest and host
processes can be communicated with each other as if they
communicates on the same OS, not on different OSs.
An important characteristic of our design is that OS com-
patibility layers can be implemented in user space. This char-
acteristic leads to two advantages. The first is robustness.
Existing OS compatibility layers with ABI compatibility are
usually implemented in kernel space, therefore a bug in this
layer might cause a kernel crash. In contrast, bugs in the
monitor process in our design do not lead to kernel crashes.
Although the VMM module runs in the kernel, it is relatively
robust because it is small and well-maintained as a part of
the standard kernel. The second is portability. The monitor
process is implemented in a user-space process and loosely
coupled with the internal APIs of the host OS kernel. User-
space processes are relatively easy to be ported to another
OS, and it can also use any cross-platform utility libraries or
even high-level languages such as Rust, Go, and Ruby.
While our design has the robustness by implementing OS
compatibility layers in user space, it also has the flexible and
powerful emulation ability thanks to virtualization technol-
ogy. By running a guest process inside a VM, system calls
and other privileged events such as page faults are trapped
by the hardware and the control is passed to the monitor
process via the VMM module. The monitor process has the
total control of the memory layout with the ability to ma-
nipulate page tables. Therefore, it is possible to implement
T. Saeki, Y. Nishiwaki, T. Shinagawa, and S. Honiden
complex memory management of the target OS such as copy-
on-write page mappings between processes. Therefore, this
approach can achieve ABI compatibility without requiring
modifications to either guest processes or host kernels.
4 IMPLEMENTATION
We have implemented a Linux compatibility layer for macOS
based on our design. The implementation, Noah, targets
x86-64 Linux 4.6 and macOS 10.12 Sierra or later. For the
virtualization foundation, we utilized Hypervisor.framework,
a built-in library of macOS that provides a set of user space
APIs to create and manage VMs. By relying on the built-in
library of the OS, we could avoid writing kernel modules,
improving robustness and reducing implementation costs.
Note that macOS only runs on Intel CPUs, not AMD’s ones.
4.1 Boot Process
A guest Linux process is created when the noah command
is executed or a guest process issues a fork(2) system call.
When a guest process is being created, the monitor process
first launches a new VM using Hypervisor.framework. Then,
to avoid injecting custom boot code into the VM, the monitor
process manipulates the VM registers so that the VM directly
enters x86-64 long mode. Additionally, to prevent any code
from running in privileged mode in the VM, the monitor
process initializes some control registers (such as CR0 and
CR4) and model-specific registers (including IA32_EFER)
with empty settings. In particular, to trap system calls issued
by the guest Linux process, the IA32_EFER.SCE bit is cleared,
thereby the SYSCALL instruction is disabled.
In x86-64, some system registers need to hold physical
memory addresses of memory data structures (e.g., page ta-
bles and segment descriptor tables). To allocate such data
structures, we reserved a region of 1 GB in the physical ad-
dress space in the VM, which is not mapped in user space.
The monitor process allocates data structures from this re-
gion and initializes them with empty settings, except for
page tables (see Section 4.3 for detail memory management).
4.2 ELF Loader
We implemented our own ELF loader to load Linux ELF
executable files into VMs. The loader is invoked just after
the noah command is executed or a guest Linux process
issues the execve(2) system call. When the loader is given
a path to a Linux ELF executable file, it first opens the Linux
ELF loader file (ld.so) through the virtual file system in the
monitor process (see Section 4.5 in detail). It then uses the
internal version of mmap() to map the content of the loader
file into the guest Linux address space in the VM. After
setting up the execution environment of ld.so, it passes
the control to ld.so with the Linux ELF executable file as
Bash on Ubuntu on macOS
an argument. Finally, ld.so loads the Linux ELF executable,
constructs memory segments, and resolves dynamic linked
libraries in the emulated environment as usual.
The ELF loader also supports the setuid bit. If the monitor
process is executed as root and the setuid bit of the target
ELF executable file is set, the monitor process changes the
effective user ID of the guest Linux process to that of the file
owner. For example, a setuid-root Linux command can write
to files owned by root. Note that guest Linux processes share
the same user name and ID with the host macOS.
4.3 Memory Management
Since we use a VM to run a Linux guest process, we need to
manage page tables by ourselves. In general, a VM involves
two page tables: the VM’s page table and the Extended Page
Table (EPT). To avoid the cost of handling two page tables,
we should fix one page table and manipulate only the other.
Which one should be fixed is a design choice.
We chose to fix the VM’s page table and manipulate the
EPT. One reason of this is performance. When a VM is
switched to another, the TLB of the VM’s page table is flushed
because the page table is changed. On the other hand, the
TLB of EPT is not flushed on VM switches, if the tagged TLB
for EPT, called Virtual Processor ID (VPID), is supported.
Therefore, we can reduce the number of page walks by using
huge pages in the VM’s page table. Another reason is to
make debugging easier. At the early stage of development,
we designed a VM and its monitor process to share the same
mapping from virtual addresses to physical addresses for a
particular region. To do so in the VM’s page table, we need
to obtain the physical address corresponding to the virtual
address. Unfortunately, macOS does not support such opera-
tion. On the other hand, Hypervisor.framework supports an
API to set the physical address of an EPT entry by specifying
a virtual address of the monitor process.
In the VM’s page table, we use straight (identity) mapping
where the virtual address is identical to the physical address.
This mapping is simple but has a limitation due to the hard-
ware bus width. The physical address width of the current
Intel CPU series is no more than 39 bits, whereas the virtual
address width is 48 bits. Therefore, we cannot map the upper
9-bit virtual address space in a VM with this mapping. Fortu-
nately, current Linux does not use this part. In addition, we
have never observed any application exhausting 256 GB (i.e.,
maximum addressable size with 39 bit) of virtual address
space. The exception is the top 1-GB region. No physical
page is mapped here in the virtual address space to hide it
from user space. It is used to locate system data structures
such as page tables and segment descriptor tables (described
above in Section 4.2). Consequently, the VM’s page table has
511-GB straight mapping and 1-GB empty mapping.
APSys ’17, September 2, 2017, Mumbai, India
The implementation of the other part of memory manage-
ment subsystem is surprisingly simple. We manage memory
regions with the vm_area_struct structures like Linux. When a
guest Linux process issues mmap(2) or other memory-related
system calls, the monitor process manipulates them as well
as EPT by way of Hypervisor.framework. Since the API of
Hypervisor.framework to manage EPTs accepts a virtual ad-
dress, physical page management can be left to macOS.
To support multiple guest processes, monitor processes
need to communicate with each other through an IPC mech-
anism. We chose to use our own shared memory allocator,
mainly for performance improvement. When the noah com-
mand starts, we pre-allocate a few gigabytes of memory
region using mmap(2) with the MAP_SHARED flag in the mon-
itor process. Data structures to be shared among monitor
processes are allocated from it. Note that the pre-allocated
buffer consumes little memory owing to lazy page allocation.
4.4 Process Management
We implemented a subset of the clone(2) system call; only
simple forking and thread creation are supported. Unfortu-
nately, Hypervisor.framework does not support a fork of a
process holding a VM, so we need a work around. The actual
handling of a fork is as follows. First, the monitor process
saves the current VM state and destroys the VM. Second,
the monitor process forks. Third, each of the two processes
launches a new VM. Finally, they restore the saved state and
start the execution. We synchronize these VM restarts using
condition variables to avoid race condition. Thread creation
is straightforward because Hypervisor.framework provides
such APIs. Process-specific data, such as memory region map-
pings, futex structures, signal handlers, and vfs structures,
are stored in the shared memory area (see Section 4.3).
4.5 Virtual File System
File access from guest Linux processes is basically forwarded
to the host macOS file system. However, to support Linux-
style file system trees, we emulate virtual file system (VFS)
of Linux in the monitor process. The VFS consists of a path
translator and object-oriented programming (OOP) compo-
nent. The path translator converts a virtual path to the host
path by resolving symbolic links and virtual mount points.
The OOP component provides an interface to install custom
file systems. The VFS allows us to expose macOS’s root file
system to Linux programs without breaking the Linux user
space. Also, virtual file systems like sysfs and procfs can be
implemented upon this facility. The implementation of the
VFS is designed to be independent from the host OS archi-
tecture for the most part. This design significantly reduces
the cost of porting Noah to other platforms than macOS.
APSys ’17, September 2, 2017, Mumbai, India
Table 1: Macro benchmark results on macOS and Noah
Benchmark name macOS Noah % UnixBench [14] and compress-7zip, sqlite, postmark
Dhrystone (LPS) 34438960.2 38265208.1 -10%
compress-7zip (MIPS) 9724 9277 4.80%
sqlite (sec) 3.55 4.11 15.8%
postmark (TPS) 2308 929 148%
kernel build (sec) 106.2 113.8 7.2%
4.6 Other Subsystems
Most of the other Linux system calls are passed-through to
macOS with flag conversion and structure adjustment. For
example, the monitor process handles getpid(2) by simply
calling the equivalent system call of macOS and returning
the value to the guest process. Most signals sent to a monitor
process are routed to the corresponding VM with proper con-
version. Signal handlers of the monitor process just record
the arrivals of the signals. Before the next entrance to the
VM, the monitor process checks the records, and if signal
arrivals are recorded, it creates a new signal stack frame in
the VM and set the IP to the registered signal handler.
5 EVALUATION
This section shows evaluations of Noah performance and
compatibility. For performance, we ran a couple of bench-
marks on Noah and on native macOS. The benchmarks are di-
vided into two groups: macro benchmarks and micro bench-
marks. The former shows the overall performance of Noah
in real-world applications, and the latter shows the overhead
on system calls. For compatibility, we describe the current
compatibility status of Noah with Linux kernel.
We carried out all evaluations on MacBook Pro
Early 2015 model with 2-core / 4-thread 3.1GHz
Intel Core i7, 16GB DDR3 memory, and 512 GB
SSD. It runs the userland of Ubuntu 16.04 on ma-
cOS Sierra 10.12.5. The Noah’s git commit revision is
2bfd3bb4244d9b171091ee21188c2048fff93be0.
Before evaluations, we need to refer to an important bug
of the current Hypervisor.framework. It is a performance
degradation bug about VM creation speed. This bug causes
the slowdown of VM creation as more VMs are created. In
addition to the creation speed, kernel_task, a core kernel
thread of macOS, consumes more memory and never releases
it, and eventually macOS freezes. Therefore, we suspect that
this is a memory leak bug. Due to this bug, performance of
fork on Noah degrades as processes are forked. Therefore,
the performance results will be worse than it actually is. This
will be fixed in the future version of Hypervisor.framework.
T. Saeki, Y. Nishiwaki, T. Shinagawa, and S. Honiden
5.1 Macro Benchmarks
We measured the performance of Dhrystone of
of Phoronix Test Suite [19]. We ran all benchmarks from
UnixBench on a single core because some benchmarks got
unstable with multiple cores on Noah. Other benchmarks ran
on 4 cores. Dhrystone and compress-7zip are CPU-bound,
and sqlite and postmark are I/O bound. We also measured
the Linux kernel build time on a single core. The Linux
kernel is based on version 3.4.113. On macOS, we manually
modified the kernel so that the minimal configuration
of it can be compiled with cross tool chains. The kernel
configuration is “allnoconfig”. We built “vmlinux” instead of
the usual “bzImage” because even more troublesome work
was needed for building it on macOS.
Table 1 shows the result. The units are Loop Per Second
(LPS) for Dhrystone, Million Instructions Per Second (MIPS)
for compress-7zip, Transaction Per Second (TPS) for post-
mark, and seconds for sqlite and kernel build. For LPS, MIPS,
and TPS, higher is better, and for seconds, lower is better.
The results show that Noah incurred low overheads on CPU-
bound applications. This is reasonable because the Linux
process directly run on processors in hardware-assisted VMs.
We are not sure why Dhrystone is faster on Noah than on
macOS, but it might be caused by the binary differences. On
the other hand, I/O-bound applications had relatively large
overheads. This would be caused by system call emulation
(see next section for detail). Especially, postmark incurred
high overhead of 148%, but this benchmark uses server work-
load for production environments and we believe this is
an acceptable overhead in development environments. The
overhead of Linux kernel build is within a reasonable range
(7.2%). This suggests that Noah incurs reasonable overhead.
5.2 Micro Benchmarks
We ran most benchmarks from UnixBench that could run on
Noah. In addition, we wrote micro benchmarks for “fork +
exec” and “System call” by ourselves. “fork + exec” measures
the number of fork() and exec() system calls executed in
a second. “System call” measures the number of getpid()
system calls, without caching, executed in a second.
Table 2 shows the results. They show that “System call”
incurred the highest overhead (1121%). This is the expected
result because Noah incurs six context switches for each
system call: the guest VM to the VMM, the VMM to the
monitor process, the monitor process to the host kernel, and
three reverse directions. This benchmark measured only the
context switch overhead and was the worst case scenario.
On the other hand, in “execl”, Noah outperforms macOS (2.4
times faster) since “execve” in Noah just replaces the con-
tents of VMs without depending on macOS while macOS
Bash on Ubuntu on macOS APSys ’17, September 2, 2017, Mumbai, India

Table 2: Micro benchmark results on macOS and Noah

Benchmark name macOS Noah Unit Overhead

System call 8279169.8 738597.3 LPS 1121%
fork + exec 408.7 393.9 LPS 3.77%
UnixBench/execl 546.2 1325.1 LPS -58.8%
UnixBench/file read 1024 bufsize * 2000 1698850.0 455603.0 KBps 273%
UnixBench/file read 4096 bufsize * 8000 4062016.5 1468246.2 KBps 177%
UnixBench/file write 1024 bufsize * 2000 1502291.3 445372.9 KBps 137%
UnixBench/file write 4096 bufsize * 8000 4271771.7 1530728.9 KBps 179%
UnixBench/pipe 1209016.7 246907.7 LPS 390%
UnixBench/pipe based context switching 175200.2 75834.3 LPS 131%

requires microkernel-based complicated process replacement
and memory management operations. “fork + exec” has al-
most the same performance on Noah and macOS (3.77%).
This indicates that the speed of a VM creation, snapshot
and restore for fork emulation is fast enough. Note that this
number will get even better if the performance degradation
bug of Hypervisor.framework is fixed.
The “file” and “pipe” showed modest overheads. Since we
used an SSD, both benchmarks were relatively CPU bound
rather than I/O bound. Since Noah’s VFS is a thin emula-
tion layer, the overheads mostly came from those of system
calls. In fact, the file benchmark measured file access per-
formance by calling read() or write() system calls many
times. Therefore, the file access speed is limited by the buffer
size of each system call multiplied by the number of “System
call” execution per seconds. For example, in the file read ex-
periment with 1024 buffer size, the performance in KBps will
be the same as the number of read() system calls issued per
second. Since Noah can issue system calls at most 738597.3
times per second, its upper limit is the same, although ac-
tual value is slower than that. There are non-negligible cost
in file access or other micro benchmarks, but we also note
that macro benchmarks indicated that such overhead for
real-world applications was in an acceptable range.
Though still many features are unimplemented, a lot of
real-world applications already work; package managers
such as apt and pacman, development tools such as gcc, vim,
make, and Ruby, daily commands such as bash and ls, and
even X applications including xeyes, xfwrite, and DooM3.
Network tools like nc also works. Much more applications
will work as the development proceeds.
6 SUMMARY AND FUTURE WORK
This paper described the design and implementation of a
novel OS compatibility layer. Our design improves robust-
ness, compatibility, seamlessness, and portability by exploit-
ing virtualization technology. In our design, every virtual
guest process runs inside a VM and the system calls are
trapped by the corresponding monitor process. The imple-
mentation, Noah, demonstrated that unmodified Linux exe-
cutables, including gcc and X applications, run on macOS.
The implementation still lacks a large part of the full Linux
kernel interface. Some of them should require hard work or
new techniques to keep the implementation fast and concise.
For example, ptrace(2) is such a system call that is left
unimplemented because of difficulties in implementing inter-
process synchronization with satisfactory performance. We
will complete the implementation in our future work.

5.3 Compatibility with Linux Kernel ACKNOWLEDGMENTS

This work is partly supported by Mitoh, a financial assistance
How compatible Noah is with the Linux kernel is an impor-
program by the government of Japan for outstanding young
tant question. To measure the quality of Linux compatibility
students and engineers.
layers, Linux Test Project [5] can be used. It runs many test
cases for all system calls to make sure that the system imple-
AVAILABILITY
ments expected behaviors of system calls. However, we did
not carried out such experiments because the current imple- Noah is publicly available from (under the dual MIT / GPL
mentation of Noah is still in early stage. The implementation licenses) https://github.com/linux-noah/noah.
lacks a large part of the full Linux kernel interface; there
are 157 unimplemented system calls among 329 currently.
Hence, we would not have insightful result yet. Thorough
quality evaluation will be executed in our future work.
APSys ’17, September 2, 2017, Mumbai, India T. Saeki, Y. Nishiwaki, T. Shinagawa, and S. Honiden

REFERENCES [20] Xiangyan Sun. 2015. Foreign LINUX - Run unmodified Linux applica-
[1] Apple. 2017. Hypervisor | Apple Developer Documentation. https: tions inside Windows. https://github.com/wishstudio/flinux. (2015).
//developer.apple.com/documentation/hypervisor. (2017). [accessed [accessed 2017-06-14].
2017-06-14]. [21] W3Techs. 2017. World Wide Web Technology Surveys. https://w3techs.
[2] Cygwin authors. 2017. Cygwin. https://www.cygwin.com. (2017). com/. (2017). [accessed 2017-06-01].
[accessed 2017-06-14].
[3] Andrew Baumann, Paul Barham, Pierre-Evariste Dagand, Tim Harris,
Rebecca Isaacs, Simon Peter, Timothy Roscoe, Adrian Schüpbach, and
Akhilesh Singhania. 2009. The Multikernel: A New OS Architecture
for Scalable Multicore Systems. In Proceedings of the ACM SIGOPS
22nd Symposium on Operating Systems Principles. ACM, 29–44. https:
//doi.org/10.1145/1629575.1629579
[4] Adam Belay, Andrea Bittau, Ali José Mashtizadeh, David Terei, David
Mazières, and Christos Kozyrakis. 2012. Dune: Safe User-level Access
to Privileged CPU Features. In Proceedings of the 10th USENIX Sym-
posium on Operating Systems Design and Implementation (OSDI 2012).
335–348.
[5] LTP developers. 2012. LTP - Linux Test Project. https://
linux-test-project.github.io/. (2012). [accessed 2017-06-17].
[6] John R. Douceur, Jeremy Elson, Jon Howell, and Jacob R. Lorch. 2008.
Leveraging Legacy Code to Deploy Desktop Applications on the Web.
In Proceedings of the 8th USENIX Symposium on Operating Systems
Design and Implementation (OSDI 2008). 339–354.
[7] gyf304. 2016. MLton crashes and BSODs. https://github.com/Microsoft/
BashOnWindows/issues/847. (2016). [accessed 2017-06-01].
[8] Jack Hammons. 2016. Windows Subsystem for Linux
Overview. https://blogs.msdn.microsoft.com/wsl/2016/04/22/
windows-subsystem-for-linux-overview/. (2016). [accessed
2017-06-14].
[9] Brian N. Handy, Rich Murphey, and Jim Mock. 2017. Chapter 10.
Linux Binary Compatibility. https://www.freebsd.org/doc/handbook/
linuxemu.html. (2017). [accessed 2017-06-14].
[10] Jon Howell, Bryan Parno, and John R. Douceur. 2013. Embassies:
Radically Refactoring the Web. In Proceedings of the 10th USENIX
Symposium on Networked Systems Design and Implementation (NSDI
2013). 529–545.
[11] Jon Howell, Bryan Parno, and John R. Douceur. 2013. How to Run
POSIX Apps in a Minimal Picoprocess. In Proceedings of the 2013
USENIX Annual Technical Conference. 321–332.
[12] Avi Kivity, Dor Laor, Glauber Costa, Pekka Enberg, Nadav Har’El,
Don Marti, and Vlad Zolotarov. 2014. OSv - Optimizing the Operating
System for Virtual Machines. In Proceedings of the 2014 USENIX Annual
Technical Conference. 61–72.
[13] Conor Hetland Kyle C. Hale and Peter Dinda. 2017. Multiverse: Easy
Conversion of Runtime Systems into OS Kernels via Automatic Hy-
bridization. In Proceedings of the 14th IEEE International Conference on
Autonomic Computing (ICAC 2017).
[14] Kelly Lucas and developers. 1989. Byte-UnixBench. https://github.
com/kdlucas/byte-unixbench. (1989). [accessed 2017-06-17].
[15] The Cloud Market. 2017. EC2 Statistics. http://thecloudmarket.com/
stats. (2017). [accessed 2017-06-01].
[16] MinGW.org. 2017. MinGW | Minimalist GNU for Windows. https:
//www.mingw.org. (2017). [accessed 2017-06-14].
[17] Stack Overflow. 2016. Developer Survey Results. https://insights.
stackoverflow.com/survey/2016. (2016). [accessed 2017-06-01].
[18] Udo Steinberg and Bernhard Kauer. 2010. NOVA: A Microhypervisor-
Based Secure Virtualization Architecture. In Proceedings of the 5th
European Conference on Computer Systems (EuroSys 2010). 209–222.
https://doi.org/10.1145/1755913.1755935
[19] Phoronix Test Suite. 2017. Phoronix Test Suite. https://www.
phoronix-test-suite.com/. (2017). [accessed 2017-06-17].