Oracle on vSphere

Nutanix Best Practices

Version 4.0 • June 2019 • BP-2000

 Oracle on vSphere

Copyright
Copyright 2019 Nutanix, Inc.
Nutanix, Inc.
1740 Technology Drive, Suite 150
San Jose, CA 95110
All rights reserved. This product is protected by U.S. and international copyright and intellectual
property laws.
Nutanix is a trademark of Nutanix, Inc. in the United States and/or other jurisdictions. All other
marks and names mentioned herein may be trademarks of their respective companies.

Copyright | 2
 Oracle on vSphere

Contents

1. Executive Summary.................................................................................5

2. Introduction.............................................................................................. 6
2.1. Audience.........................................................................................................................6
2.2. Purpose.......................................................................................................................... 6

3. Nutanix Enterprise Cloud Overview...................................................... 8

3.1. Nutanix Acropolis Architecture.......................................................................................9
3.2. Nutanix Era.................................................................................................................... 9

4. Nutanix Cluster Best Practices............................................................ 10

4.1. Nutanix Platform Guidance.......................................................................................... 10
4.2. Storage Configuration.................................................................................................. 11
4.3. Networking....................................................................................................................12
4.4. High Availability............................................................................................................13

5. ESXi Recommended Settings.............................................................. 14

5.1. Virtualization and Compute Configuration................................................................... 14

6. Linux Operating System Recommended Settings............................. 16

6.1. Kernel Settings for Oracle........................................................................................... 16
6.2. Configure HugePages for Oracle Database................................................................ 17
6.3. Network Configuration..................................................................................................18
6.4. Configure Volumes (VMDKs)....................................................................................... 18
6.5. VMDKs......................................................................................................................... 20
6.6. Nutanix Volume Groups............................................................................................... 20
6.7. Configure Logical Volume Manager (LVM).................................................................. 22
6.8. Kernel Settings for the Proc File System.................................................................... 23

7. Oracle Database Best Practices.......................................................... 25

7.1. ASM Settings............................................................................................................... 25

3
 Oracle on vSphere

7.2. Oracle Clusterware CSS Timeout Settings..................................................................25

7.3. Database Settings for Oracle Single Instance and RAC............................................. 26

8. Conclusion..............................................................................................27
8.1. References................................................................................................................... 27
8.2. About Nutanix...............................................................................................................28

List of Figures................................................................................................................ 29

List of Tables.................................................................................................................. 30

4
 Oracle on vSphere

1. Executive Summary
The Nutanix Enterprise Cloud provides a complete datacenter infrastructure solution for Oracle
databases, eliminating the complexities and inefficiencies of traditional multitier datacenter
environments. Whether you are virtualizing critical or tier-1 Oracle databases or running them on
bare metal, Nutanix solutions bring the predictable performance, scalability, and cost benefits of
web-scale architecture to your transactional and analytical Oracle database environments.
The hypervisor-agnostic Nutanix solution (VMware vSphere, Microsoft Hyper-V, or Nutanix AHV)
delivers powerful self-healing, data protection, and disaster recovery capabilities to keep your
databases and applications running and your critical data well protected. The Nutanix Enterprise
Cloud provides near-instantaneous local and remote backups using snapshots for offloading
RMAN (Recovery Manager) backups to tape and disk or WORM (write once, read many) for
offsite backup. You can also use a Nutanix clone snapshot to easily refresh production-based test
and development Oracle instances.
This best practices guide recommends Nutanix cluster settings for running Oracle databases, as
well as ESXi settings for VMs, networking, volumes (VMDKs), Linux OS, and Oracle ASM and
database.
Nutanix is an Oracle Platinum Partner.

1. Executive Summary | 5
 Oracle on vSphere

2. Introduction
Unless otherwise stated, the solution described in this document is valid on all supported AOS
releases.

2.1. Audience
This best practices guide is part of the Nutanix Solutions Library for Oracle. We wrote it for IT
administrators and solutions architects responsible for designing, managing, and supporting
Oracle Database deployments on Nutanix infrastructures. Readers should be familiar with
VMware vSphere, Oracle Database, the Linux operating system, and the Nutanix Enterprise
Cloud.

Tip: For more information on Linux system administration or Oracle installation and
setup, please refer to documentation from the appropriate software vendor.

2.2. Purpose
This document provides design, configuration, and optimization guidelines for a single instance of
Oracle Database or RAC running on Nutanix with VMware ESXi. In this document, we cover the
following topics:
• Overview of the Nutanix solution.
• Nutanix cluster best practices.
• VMware ESXi best practices.
• Linux operating system best practices.
• Oracle Database best practices.
Upon completing this document, the reader should be able to design, architect, and deploy a
high-performing and highly available Oracle Database solution on the Nutanix platform.
For information specific to Oracle on Nutanix AHV, please refer to the Oracle on AHV best
practices guide.

2. Introduction | 6
 Oracle on vSphere

Table 1: Document Version History

Version
Published Notes
Number
1.0 September 2014 Original publication.
2.0 February 2017 Updated with current Nutanix platform information.
Updated provisioning recommendations and noted
2.1 May 2017 VMware ESXi 6.0.0 support for adding disks online in
Oracle RAC environments.
Updated platform overview and Nutanix Platform
2.2 January 2018
Guidance section.
3.0 November 2018 Updated product information and Best Practices section.
3.1 December 2018 Updated Best Practices section.
Added Nutanix Era section and updated the OLTP
3.2 March 2019
Scenario Detail and OLAP Scenario Detail tables.
4.0 June 2019 Major updates throughout.

2. Introduction | 7
 Oracle on vSphere

3. Nutanix Enterprise Cloud Overview

Nutanix delivers a web-scale, hyperconverged infrastructure solution purpose-built for
virtualization and cloud environments. This solution brings the scale, resilience, and economic
benefits of web-scale architecture to the enterprise through the Nutanix Enterprise Cloud
Platform, which combines three product families—Nutanix Acropolis, Nutanix Prism, and Nutanix
Calm.
Attributes of this Enterprise Cloud OS include:
• Optimized for storage and compute resources.
• Machine learning to plan for and adapt to changing conditions automatically.
• Self-healing to tolerate and adjust to component failures.
• API-based automation and rich analytics.
• Simplified one-click upgrade.
• Native file services for user and application data.
• Native backup and disaster recovery solutions.
• Powerful and feature-rich virtualization.
• Flexible software-defined networking for visualization, automation, and security.
• Cloud automation and life cycle management.
Nutanix Acropolis provides data services and can be broken down into three foundational
components: the Distributed Storage Fabric (DSF), the App Mobility Fabric (AMF), and AHV.
Prism furnishes one-click infrastructure management for virtual environments running on
Acropolis. Acropolis is hypervisor agnostic, supporting two third-party hypervisors—ESXi and
Hyper-V—in addition to the native Nutanix hypervisor, AHV.

Figure 1: Nutanix Enterprise Cloud

3. Nutanix Enterprise Cloud Overview | 8

 Oracle on vSphere

3.1. Nutanix Acropolis Architecture

Acropolis does not rely on traditional SAN or NAS storage or expensive storage network
interconnects. It combines highly dense storage and server compute (CPU and RAM) into a
single platform building block. Each building block delivers a unified, scale-out, shared-nothing
architecture with no single points of failure.
The Nutanix solution requires no SAN constructs, such as LUNs, RAID groups, or expensive
storage switches. All storage management is VM-centric, and I/O is optimized at the VM virtual
disk level. The software solution runs on nodes from a variety of manufacturers that are either
all-flash for optimal performance, or a hybrid combination of SSD and HDD that provides a
combination of performance and additional capacity. The DSF automatically tiers data across the
cluster to different classes of storage devices using intelligent data placement algorithms. For
best performance, algorithms make sure the most frequently used data is available in memory or
in flash on the node local to the VM.
To learn more about the Nutanix Enterprise Cloud, please visit the Nutanix Bible and
Nutanix.com.

3.2. Nutanix Era

Nutanix Era makes Nutanix Enterprise Cloud the ideal platform for running databases. Nutanix
Era is a software suite that automates and simplifies database administration, bringing one-click
simplicity and invisible operations to database provisioning and life cycle management (LCM).
With one-click database provisioning and copy data management (CDM) as its first services,
Nutanix Era enables DBAs to provision, clone, and refresh their databases to any point in time.
The API-first Nutanix Era architecture can easily integrate with your preferred self-service tools,
and every operation has a unique ID and is fully visible for auditing.
For more information, read our Nutanix Era solution brief.

3. Nutanix Enterprise Cloud Overview | 9

 Oracle on vSphere

4. Nutanix Cluster Best Practices

This section discusses selecting Nutanix nodes for better performance, in terms of CPU
clock speed, while lowering Oracle licensing costs. We also discuss networking and storage
best practices for a Nutanix cluster, as a proper setup eliminates infrastructure performance
bottlenecks.
For AHV-specific guidance, refer to the Oracle on AHV best practices guide.

4.1. Nutanix Platform Guidance

• Use a single replication factor 2 container with inline compression enabled.
• Select an appropriate model that meets your compute, storage, and licensing requirements:
⁃ Ideally, use all-flash or NVMe storage. Alternatively, size the platform so the working set fits
in the SSD tier of a hybrid system.
⁃ Nutanix supports database VMs that are larger than the storage capacity of an individual
node, as all storage is pooled at the block level across an entire Nutanix cluster.
⁃ Select higher-memory node models for I/O-heavy Oracle Database workloads.
⁃ Use nodes that each have double the memory size of the largest single VM.
⁃ Choose nodes that have a CPU speed of at least 2.6 GHz for all Oracle Database
workloads.
⁃ Ensure that the nodes you select conform to your organization’s licensing constraints.
• Create a dedicated consistency group for Oracle Database VMs and applications. When
using Nutanix snapshots and replication for Oracle RAC, all RAC nodes must be in the same
consistency group.
• If running Oracle Database on Windows, use application-consistent snapshots on the
consistency group to invoke VSS when snapshotting. Otherwise, follow the guidance in Oracle
MOS ID 604683.1 (Oracle account required).
• Nutanix CVMs should always be in the hypervisor cluster root, not in a child resource pool.
• The maximum recommended I/O size is 1 MB for best performance.
• The Nutanix Shadow Clones feature is enabled by default. However, you should disable this
feature when deploying Oracle RAC databases. For more information on disabling Shadow
Clones, see KB 1828.
• Erasure coding is not recommended for Oracle, but you can enable it if space is a constraint.

4. Nutanix Cluster Best Practices | 10

 Oracle on vSphere

4.2. Storage Configuration

The Nutanix Enterprise Cloud provides an ideal combination of high-performance compute with
localized storage to meet any demand. True to the capabilities inherent in the platform’s design,
there is no need to reconfigure or customize the Nutanix product to optimize for the Oracle use
case.
The figure below shows a high-level example of the relationship between a Nutanix block, node,
storage pool, and container.

Figure 2: Nutanix Component Architecture

The following table shows the Nutanix storage pool and container configuration. Always ensure
that your environment has ample storage capacity, so it can tolerate the loss of a node during
failure or maintenance.

Tip: Enable inline compression for Oracle databases.

Note: Do not enable deduplication for Oracle databases. Only enable erasure
coding if space is a constraint.

Table 2: Nutanix Storage Configuration

Name Role Details

SP01 Main storage pool for all data All disks
CTR-RF2-VM-01 Container for all VMs and data ESXi: datastore

Tip: Recommended maximum I/O size is 1 MB.

4. Nutanix Cluster Best Practices | 11

 Oracle on vSphere

4.3. Networking
• Use low-latency 10 GbE, 25 GbE, or 40 GbE switches.
• Establish redundant 10 GbE, 25 GbE, or 40 GbE uplinks from each Nutanix node.
• Ensure adequate throughput between Nutanix nodes and Oracle database VMs.
• Check for any pause frames that could impact replication and VM communication.
• Use dedicated NICs (at least 10 GbE) for Oracle RAC Cache Fusion (heartbeat) networks on
each Nutanix node.
• Do not enable jumbo frames unless absolutely necessary.
• If you are using Nutanix Volumes for Oracle data or log disks:
⁃ Set up an IP address for iSCSI data services on the Nutanix cluster.
⁃ Set up a tagged vLAN to isolate Nutanix Volumes iSCSI traffic.
⁃ Use iSCSI CHAP if required.

Note: If you are upgrading AOS, be sure to check the release notes for your
target AOS version. Consider engaging Nutanix Support if you encounter network
connectivity issues when hosting or migrating Oracle RAC VMs.

Nutanix recommends a leaf-spine network architecture, which is designed for linear scaling.
A leaf-spine architecture consists of two network tiers: an L2 leaf and an L3 spine based on
nonblocking switches. This architecture maintains consistent performance without any throughput
reduction due to a static maximum of three hops from any node in the network.
The figure below shows the design of a scale-out leaf-spine network architecture, which provides
20 GbE active throughput from each node to its L2 leaf and scalable 80 GbE active throughput
from each leaf to its spine switch, allowing you to grow from one Nutanix block to thousands
without any impact to available bandwidth.

4. Nutanix Cluster Best Practices | 12

 Oracle on vSphere

Figure 3: vMotion Test Scenarios Overview

4.4. High Availability

• Size clusters for N+1 redundancy.
• Use a percentage value for HA admission control: when using VMware vSphere, obtain this
percentage value by dividing one by the number of hosts per cluster.
• Set VM antiaffinity rules to prevent multiple Oracle RAC VMs in an Oracle RAC cluster from
running on a single Nutanix node.

4. Nutanix Cluster Best Practices | 13

 Oracle on vSphere

5. ESXi Recommended Settings

The following table lists the recommended settings for Oracle database VMs.

Table 3: ESXi Recommended Settings

Parameter
Network adapter
Storage adapter
OS and app disks
Database (ASM) disks for
standalone*
Configuration
VMXNET3
Minimum of 3 PVSCSI (OS + DB + redo), 4 for larger or high-
performance databases
Thin provisioned, disk mode = dependent
Thin provisioned, disk mode = independent persistent

Thick provisioned eager zeroed, disk mode = independent

Database (ASM) disks for persistent
RAC*
(see VMware KB 1034165)

VMware tools Latest installed

Memory Locked (preferred)
Disk.ENABLEUUID Enable in VMX file for ASM vDisks
Advanced VM configuration Name: ethernetX.coalescingScheme (where X = VMNIC number)
option (with RAC Cluster
Interconnect) Value: disabled

* Using in-guest iSCSI requires Nutanix Volumes. For more information, refer to the Nutanix
Volumes best practices guide.

5.1. Virtualization and Compute Configuration

Host and VM Sizing (vCPUs, RAM, Storage)
• For tier-1, mission-critical workloads, maintain a one-to-one ratio of physical cores to vCPUs.

5. ESXi Recommended Settings | 14

 Oracle on vSphere

• For small Oracle Database VMs, keep vCPUs <= to the number of cores per each physical
NUMA node.
• Keep vCPU numbers simply divisible by NUMA node sizes for easy scheduling.
• Leave hyperthreading sharing at the default policy (any).
• For tier-1 workloads, lock Oracle Database VM memory at a minimum reserve sufficient to
cover both the system global area (SGA) HugePages and the Program Global Area (PGA).
• Use the following calculation to size Oracle Database VM memory:
⁃ VM Memory = Oracle SGA + Oracle PGA + OS memory + VM overhead
• Use Paravirtual SCSI controllers (PVSCSI) and VMXNET3 NICs.
• If you are using Nutanix Volumes for Oracle data or log disks:
• Use separate volume groups for data and logs.
• Use four to eight vDisks for data per volume group, and two to four vDisks per volume group
for logs.
• If you are using Oracle ASM for volume management, map a disk group to a Nutanix volume
group.
• Do not use resource pools unless absolutely necessary. If using resource pools, ensure that
shares are sized correctly.
• Use antiaffinity rules to prevent multiple Oracle RAC nodes in the same RAC cluster from
running on the same Nutanix node.

Host and VM Configurations

• Follow VMware performance best practices.
• When using vSphere Distributed Switch, select route based on physical NIC load for the
teaming policy. Otherwise, select route based on virtual port ID.
• Use multi-NIC vMotion when deploying Oracle database with large RAM configurations.
• Use a separate vSphere cluster or DRS antiaffinity rules to keep Oracle Database VMs on
licensed hosts and to keep related Oracle Database VMs apart within the licensed group of
hosts.
• Although the multiwriter flag is supported for Oracle RAC with vSphere on up to a maximum
of eight hosts concurrently (see VMware KB 1034165 for more information), Nutanix does
not recommend using it. We recommend using Nutanix Volumes with the in-guest iSCSI
initiator. Only use the multiwriter flag as a last resort. For more information, refer to the Nutanix
Volumes best practices guide.

5. ESXi Recommended Settings | 15

 Oracle on vSphere

6. Linux Operating System Recommended

Settings
When running Oracle Database on Linux, there are many areas to consider, such as
performance, data protection, and efficiency. This section discusses the best practices for Oracle
Linux, Red Hat Enterprise Linux, and SUSE Linux.

6.1. Kernel Settings for Oracle

This section details the parameter settings for Oracle Database 12cR1. These settings derive
from the Oracle preinstall package for 12cR1. If you are using a different Oracle Database
version, select the preinstall package for that version. For example, if you want to install version
18c, use oracle-database-preinstall-18c. For more information about these settings, refer to
Oracle Database 12cR1 installation documentation and Configuring Kernel Parameters and
Resource Limits. Be sure to consult the documentation specific to your version of Oracle
Database.

Note: The vm.nr_hugepages setting is reserved for enabling HugePages. Please

refer to the Configure HugePages for Oracle Database section for more information.

limits.conf Settings
Following are the shell settings for the “grid” and “oracle” users. You can use the settings as
shown or adjust them to fit your environment. For more information about these settings, refer to
Oracle Database 12cR1 installation documentation and the Oracle kernel parameters page cited
previously.

Note: The memlock setting is reserved for enabling HugePages. Set it to a number
slightly lower than the actual memory. Following is an example for a VM with 32
GB of physical memory. Please refer to the later Configure HugePages for Oracle
Databases section for information on enabling HugePages.

6. Linux Operating System Recommended Settings | 16

 Oracle on vSphere

Default Profile Settings

Append the following to /etc/profile:
if [ $USER = "oracle" ] || [ $USER = "grid" ] ; then
if [ $SHELL = "/bin/ksh" ]; then
ulimit -p 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
fi

6.2. Configure HugePages for Oracle Database

Nutanix recommends using HugePages to improve performance, especially in a data warehouse
environment. For large SGA (system global area) sizes, HugePages substantially improves
virtual memory management. Without HugePages, the SGA memory is divided into 4 K pages
that the Linux kernel must manage. With HugePages, the page size increases anywhere from 2
MB to 256 MB, depending on the kernel version and the hardware architecture, which reduces
the total number of pages the kernel manages. In addition, the HugePages stay resident; they
cannot be swapped out. Please refer to Oracle documentation for more information.
• Disable Automatic Memory Management (AMM) on the database instance if it is enabled.
Although AMM is the simplest way to manage instance memory because it automatically
manages and tunes the database, it cannot be used with HugePages.
• Set the SGA_TARGET and PGA_AGGREGATE_TARGET parameters. For example, if you
decide to reserve 20 percent of the total physical memory for the operating system and other
non-Oracle applications, you can allocate the remaining 80 percent of memory as described in
the table and bullet list below.

Table 4: Guidelines for Oracle SGA and PGA

OLTP DSS
SGA_TARGET = (total_mem * 0.8) * 0.8, SGA_TARGET = (total_mem * 0.8) * 0.5,
where total_mem is the total amount of where total_mem is the total amount of
physical memory available on the system physical memory available on the system

6. Linux Operating System Recommended Settings | 17

 Oracle on vSphere

OLTP DSS
PGA_AGGREGATE_TARGET = (total_mem * PGA_AGGREGATE_TARGET = (total_mem *
0.8) * 0.2, where total_mem is the total amount 0.8) * 0.5, where total_mem is the total amount
of physical memory available on the system of physical memory available on the system

Note: Use these recommendations as guidelines only. You must monitor the
memory usage and adjust accordingly based on your workload.

• If you are running Oracle 11.2.0.2 or later, you can set USE_LARGE_PAGES to ONLY, which
prevents the database from starting if it is not backed up by HugePages.
• Disable the Transparent HugePages option if it is not already disabled. Transparent
HugePages differs from HugePages in that it dynamically allocates memory via a
khugepaged kernel thread, rather than at boot time, as HugePages does. Transparent
HugePages does not work well with Oracle databases, however; it can cause performance
problems and node reboots in RAC installations.
• Set transparent_hugepage=never.
• Reboot.

6.3. Network Configuration

The table below shows the required networking for setting up Oracle servers.

Table 5: Network Settings

Oracle Speed Purpose

Oracle Public Network 10 GbE Public or management network
Oracle Private Network 10, 25, or 40 GbE RAC Interconnect
Oracle Data Network 10, 25, or 40 GbE iSCSI network with Nutanix Volumes

On the Oracle VM, add NOZEROCONF=yes to the /etc/sysconfig/network file.

6.4. Configure Volumes (VMDKs)

In a traditional three-tier architecture, DBAs often work with storage administrators to create a
custom storage design to suit their Oracle environment, including designating the RAID level and

6. Linux Operating System Recommended Settings | 18

 Oracle on vSphere

block size for the Oracle datafiles or online redo logs. This process can become cumbersome to
manage if there are multiple databases.
The Nutanix Enterprise Cloud eliminates all such problems associated with RAID and block
size. Once you create the required number of volumes for your Oracle database, you’re ready to
deploy.
The table below shows the minimum recommended number of VMDKs for Oracle. The
appropriate number of vDisks depends on the expected size of the Oracle database and the
number of Nutanix nodes in your cluster. We based the following example on a four-node Nutanix
cluster.

Table 6: Nutanix Volume Configuration for Oracle

Number
Purpose Comment
of VMDKs
1 Boot disk Can be used with LVM or Standard partition
Database datafiles / control Can be used with Oracle ASM or file system
8
files / redo log files with LVM
Can be used with Oracle ASM or file system
4 Database archive log files
with LVM
Database RMAN backup Can be used with Oracle ASM or file system
4
files with LVM

Note: Nutanix recommends using multiple VMDKs, as this approach provides

the best performance and load balancing for high I/O workloads such as Oracle
databases.

The following table provides Oracle ASM disk group layouts.

Table 7: ASM Disk Group Layout

Number of VMDKs Purpose Comment

Use with VMDKs or
2 volumes OCR ASM DG (Normal Redundancy)
volume group with iSCSI
Use with vDisk or
8 volumes DATA ASM DG (External Redundancy)
volume group

6. Linux Operating System Recommended Settings | 19

 Oracle on vSphere

Number of VMDKs Purpose Comment

Use with vDisk or
4 volumes FRA ASM DG (External Redundancy)
volume group

There are two ways to create volumes on Nutanix for your Oracle Database VM with ESXi:
1. VMDKs.
2. Nutanix volume groups for in-guest iSCSI.

6.5. VMDKs
The VMDKs option simplifies VM administration, as it does not require in-guest iSCSI setup with
volume groups. When creating a VM, simply add VMDKs and assign them to the proper PVSCSI
based on the information provided previously in the ESXi Recommended Settings section. This is
the best option if you have a database that does not have intensive I/O requirements.

6.6. Nutanix Volume Groups

Alternatively, you can attach Nutanix volume groups, which are a collection of volumes, to the
VMs. Volume groups enable you to separate the data volumes from the VM’s boot volume.
This separation allows you to put the volume group that contains only the database volumes
into a Nutanix protection domain that you can then snapshot and clone. In addition, volume
groups let you configure clustering for shared-disk access across multiple VMs. Such clustering
includes Oracle RAC. To attach the volume group to multiple RAC VMs, you must create it with
an attribute of shared=true, using the command line interface (aCLI). After you have created the
volume groups, you can use in-guest iSCSI and the DSIP (Data Services IP) to attach them to
the Oracle RAC VMs.

Note: If the volume group you want to share is already attached to a VM, you must
detach it, change the shared parameter to true, and reattach it.

For more information and how-to guidance on the iSCSI recommended settings, refer to the
Linux on Nutanix AHV best practices guide.

Note: If you want to use Nutanix Volumes for your Oracle Database, please refer to
the Nutanix Volumes best practices guide. As an additional resource, the Linux on
Nutanix AHV best practices guide provides detailed instructions and covers setting
up iSCSI ifaces.

6. Linux Operating System Recommended Settings | 20

 Oracle on vSphere

Linux Disk Device Settings

To optimize your Oracle Database performance, change the following disk parameters from their
default settings.
• Change the max_sector_kb to 1024 (the default is 512). With newer Linux distributions, the
default may already be set to 1024. Also change the disk timeout to 60 (default is 30).
[root@oracle1 ~]# lsscsi | grep NUTANIX | awk '{print $NF}' | awk -F"/" '{print $NF}' | grep
-v "-" | while read LUN
do
echo 1024 > /sys/block/${LUN}/queue/max_sectors_kb
done
[root@oracle1 ~]# lsscsi | grep NUTANIX | awk '{print $NF}' | awk -F"/" '{print $NF}' | grep
-v "-" | while read LUN
do
echo 60 > /sys/block/${LUN}/device/timeout
done

Either put this command in the /etc/rc.local file, so that it runs the next time the server reboots,
or use UDEV rules. For UDEV rules, you can create a file with the below content under the /etc/
udev/rules.d directory.
ACTION=="add", SUBSYSTEMS=="scsi", ATTRS{vendor}=="NUTANIX ", ATTRS{model}=="VDISK", RUN+="/
bin/sh -c 'echo 1024 >/sys$DEVPATH/queue/max_sectors_kb'"
ACTION=="add", SUBSYSTEMS=="scsi", ATTRS{vendor}=="NUTANIX ", ATTRS{model}=="VDISK", RUN+="/
bin/sh -c 'echo 60 >/sys$DEVPATH/device/timeout'"

• After Oracle installation, change the disk io_scheduler to noop. The default is deadline.
[root@oracle1 ~]# lsscsi | grep NUTANIX | awk '{print $NF}' | awk -F"/" '{print $NF}' | grep
-v "-" | while read LUN
do
echo noop > /sys/block/${LUN}/queue/scheduler
done

If you put this command in the /etc/rc.local file, it runs the next time the server reboots.
Alternatively, you can put it in the grub configuration file. Use the following steps for GRUB2
configuration.
• Edit the /etc/default/grub file, add “elevator=noop” to the GRUB_CMDLINE_LINUX line, and
save the file.

Figure 4: Add elevator=noop to /etc/default/grub File

6. Linux Operating System Recommended Settings | 21

 Oracle on vSphere

• Disable transparent_hugepage.
• If the Linux kernel supports the blk_mq (block multiqueue) option (kernel versions 4.12
and later), add the parameter “scsi_mod.use_blk_mq=1” to enable it and remove the
elevator=noop option.

Figure 5: Enabling Block Multiqueue

• Run grub2-mkconfig to generate a new grub file and reboot.

[root@oracle1 ~]# grub2-mkconfig –o /boot/grub2/grub.cfg

Note: Linux kernel versions 4.12 and later offer new scheduler options within
blk_mq: mq-deadline, BFQ, and Kyber. Based on Nutanix performance testing, we do
not currently recommend using any of these schedulers. Select the “none” option.

6.7. Configure Logical Volume Manager (LVM)

This section discusses Linux Logical Volume Manager (LVM) best practices. If you are creating
a file system for databases instead of using Oracle ASM, be sure to use LVM striping across all
volumes for a specific mount point.
The following example shows how to create an LVM volume group with eight volumes for
a database datafiles mount point. For more information, please refer to Nutanix Volumes
documentation.
• Create the physical volumes.
• Create an LVM volume group.
• Create a logical volume. Make sure to use the -i (lowercase i) option to specify the number
of volumes to stripe across and the -I (uppercase i) option to specify the stripe size. The
recommended stripe size is 512 KB.
[root@localhost ~]# lvcreate -l 383994 -i 6 -I 512 -n vol1 vgdata
Logical volume "vol1" created.

• Create the file system. In this example, we are creating an ext4 file system.
[root@localhost ~]# mkfs.ext4 /dev/vgdata/vol1

• Mount the ext4 file system. If you are creating an XFS file system, please refer to the Mount
Options table below.
[root@localhost ~]# mount /dev/vgdata/vol1 /u01/oradata -o
noatime,nodiratime,discard,barrier=0

6. Linux Operating System Recommended Settings | 22

 Oracle on vSphere

Note: You can use xfs_admin for xfs or e2label for ext4 to label your file systems
with a friendly name. Use the LABEL= option in the /etc/fstab file for ease of
management.

The table below lists the recommended mount options for ext4 and XFS file systems.

Table 8: Mount Options for ext4 and XFS File Systems

File System Type Mount Options

ext4 noatime,nodiratime,discard,barrier=0
XFS noatime,nodiratime,discard,nobarrier,logbufs=8

6.8. Kernel Settings for the Proc File System

In addition to configuring LVM, you must also set the following kernel parameters in the /
etc/sysctl.conf file for optimal performance. These settings can be found at the /proc/sys/vm
documentation page.

Table 9: Proc File System Kernel Parameters

Settings Value Purpose

vm.overcommit_memory
vm.dirty_background_ratio
vm.dirty_ratio
vm.dirty_expire_centisecs
vm.dirty_writeback_centisecs
1 Disable memory overcommit handling.
As a percentage. Contains the number of pages at
5 which the background kernel flusher threads will
start writing out dirty data.
As a percentage. Contains the number of pages at
15 which a process that is generating disk writes will
start writing out dirty data itself.
This tunable is used to define when dirty data is
500 old enough to be eligible for writeout by the kernel
flusher threads.
The kernel flusher threads periodically wake up and
100 write old data out to disk. This tunable expresses
the interval between those wakeups.

6. Linux Operating System Recommended Settings | 23

 Oracle on vSphere

Settings Value Purpose

This tunable defines how aggressively the kernel

swaps memory pages. A value of 0 instructs the
kernel not to initiate swap until the amount of free
and file-backed pages is less than the high-water
mark in a zone.
vm.swappiness 0
Note: With vm.swappiness set to 0, encountering
OOM (out of memory) error messages indicates
that you should reconsider the sizing for your
Oracle VM. For any Oracle database VMs, the
kernel should not be swapping.

6. Linux Operating System Recommended Settings | 24

 Oracle on vSphere

7. Oracle Database Best Practices

In this section we discuss Oracle best practices regarding storage (disk I/O), specifically Oracle
ASM settings and Oracle Database. For information on Oracle parameters not related to storage,
please visit the Oracle documentation page.

7.1. ASM Settings

The following table shows the recommended settings for all ASM disk groups.

Table 10: Oracle ASM Settings

Settings Values
ASM Allocation Unit (AU) size 1 MB
ASM OCR disk group redundancy Normal/High
ASM DATA disk group redundancy External
ASM FRA disk group redundancy External

Nutanix supports ASM disks using UDEV devices, ASMLib devices, or ASMFD (ASM Filter
Driver) devices. ASMLib is available with Oracle 11g and 12c; ASMFD is available with Oracle
12cR1 and later versions. The ASMFD feature rejects write I/O requests that are not issued by
Oracle software. This filter ensures that users with administrative privileges cannot inadvertently
overwrite Oracle ASM disks, thus preventing corruption in those disks and files within the disk
groups.
Nutanix performance testing shows that ASMFD and UDEV yield similar results, so you can
deploy whichever type of device you are most comfortable with on your Nutanix cluster. Refer to
Oracle documentation for more information on how to deploy Oracle ASM.

7.2. Oracle Clusterware CSS Timeout Settings

The CSS misscount parameter represents the maximum time, in seconds, that the system can
tolerate a missed network heartbeat before beginning to reconfigure the cluster to evict the
node. In the Linux operating system, this parameter is set to 30 seconds by default. You can

7. Oracle Database Best Practices | 25

 Oracle on vSphere

change the CSS misscount timeout to 60 seconds to avoid node eviction in case of network or I/
O contention. For more information, please refer to Oracle MOS ID 294430.1 and KB 6843.

7.3. Database Settings for Oracle Single Instance and RAC

The following table shows the recommended settings for Oracle single instances and RAC
databases.

Table 11: Oracle Database Settings

Settings Value Purpose

This parameter allows directIO and
AsyncIO when using a file system such
filesystemio_options setall
as ext. It is not required when using
Oracle ASM.
Refer to
SGA_TARGET
HugePages section
Refer to
PGA_AGGREGATE_TARGET
HugePages section
Private interconnect between RAC
RAC Interconnect 10, 25, or 40 GbE
nodes.

DB_FILE_MULTIBLOCK_ This parameter minimizes I/O during

128 table scans. Refer to Oracle Doc ID
READ_COUNT 1298860.1 for more information.
Degree of parallelism. On Nutanix, it is 1
PARALLEL_THREADS_PER_CPU 1
per vCPU.

7. Oracle Database Best Practices | 26

 Oracle on vSphere

8. Conclusion
The Nutanix platform offers the capability to run almost any workload. You have the ability to run
both ORADB and other VM workloads simultaneously on the same platform. The database’s
CPU and storage requirements drive density for Oracle Database deployments. Test validation
has shown that increasing the number of ORADB VMs on the Nutanix platform is preferable
to scaling the number of Oracle instances, to take full advantage of Nutanix performance and
capabilities.
From an I/O standpoint, the Nutanix platform handled the throughput and transaction
requirements of a demanding Oracle database given DSF-localized I/O, server-attached flash,
and intelligent tiering. When you need to virtualize more Oracle databases, you can simply scale
out the Nutanix platform and add more database VMs to gain capacity and performance. During
the load tests, we also subjected the Oracle RAC environment to VMware vMotion migrations to
demonstrate the robustness of the platform and to prove that convergence of all networking, even
when pushed to the saturation point, does not disrupt client connections.
We determined pod sizing after carefully considering performance as well as accounting for
the additional resources needed for N+1 failover capabilities. The Oracle on Nutanix solution
provides a single high-density platform for Oracle, VM hosting, and application delivery. This
modular, pod-based approach enables such deployments to easily be scaled. You can start small
and pay as you grow for any workload.
The appendix includes detailed validation and benchmarking for Oracle Database on Nutanix. It
also includes the scripts and testing configuration required to reproduce the successful results.
For further discussion of the best practices for Oracle on Nutanix, please contact us on our twitter
@Nutanix, or visit with us on the Nutanix NEXT Community.

8.1. References
1. VMware vSphere Networking best practices guide
2. VMware vSphere Storage best practices guide
3. Nutanix Volumes best practices guide
4. Oracle on AHV best practices guide
5. Oracle Contracts
6. Understanding Oracle Certification, Support, and Licensing on VMware Environments
7. VMware Expanded Oracle Support Policy
8. TSANet

8. Conclusion | 27
 Oracle on vSphere

8.2. About Nutanix

Nutanix makes infrastructure invisible, elevating IT to focus on the applications and services that
power their business. The Nutanix Enterprise Cloud OS leverages web-scale engineering and
consumer-grade design to natively converge compute, virtualization, and storage into a resilient,
software-defined solution with rich machine intelligence. The result is predictable performance,
cloud-like infrastructure consumption, robust security, and seamless application mobility for a
broad range of enterprise applications. Learn more at www.nutanix.com or follow us on Twitter
@nutanix.

8. Conclusion | 28
 Oracle on vSphere

List of Figures
Figure 1: Nutanix Enterprise Cloud................................................................................... 8

Figure 2: Nutanix Component Architecture......................................................................11

Figure 3: vMotion Test Scenarios Overview.................................................................... 13

Figure 4: Add elevator=noop to /etc/default/grub File..................................................... 21

Figure 5: Enabling Block Multiqueue............................................................................... 22

29
 Oracle on vSphere

List of Tables
Table 1: Document Version History................................................................................... 7

Table 2: Nutanix Storage Configuration........................................................................... 11

Table 3: ESXi Recommended Settings............................................................................14

Table 4: Guidelines for Oracle SGA and PGA.................................................................17

Table 5: Network Settings................................................................................................ 18

Table 6: Nutanix Volume Configuration for Oracle.......................................................... 19

Table 7: ASM Disk Group Layout.................................................................................... 19

Table 8: Mount Options for ext4 and XFS File Systems..................................................23

Table 9: Proc File System Kernel Parameters.................................................................23

Table 10: Oracle ASM Settings........................................................................................25

Table 11: Oracle Database Settings................................................................................ 26

30