PowerEdge VRTX Dual PERC Failover

How to configure a PowerEdge VRTX to use the Dual Shared PERC as a Failover
prior to the release of the firmware for redundancy.
Dell Solutions Center and Dell Software Group
February 2014
Revisions (required)
Date Description

February 2014 Initial release

THIS WHITE PAPER IS FOR INFORMATONAL PURPOSES ONLY, AND MAY CONTAIN TYPOGRAPHICAL ERRORS AND
TECHNICAL INACCURACIES. THE CONTENT IS PROVIDED AS IS, WITHOUT EXPRESS OR IMPLIED WARRANTIES OF
ANY KIND.
2014 Dell Inc. All rights reserved. Reproduction of this material in any manner whatsoever without the express
written permission of Dell Inc. is strictly forbidden. For more information, contact Dell.

Dell, the DELL logo, and the DELL badge are trademarks of Dell Inc. Symantec, NetBackup, and Backup Exec are
trademarks of Symantec Corporation in the U.S. and other countries. Microsoft, Windows, and Windows Server are
registered trademarks of Microsoft Corporation in the United States and/or other countries. Other trademarks and
trade names may be used in this document to refer to either the entities claiming the marks and names or their
products. Dell disclaims any proprietary interest in the marks and names of others.
Trademarks used in this text:
Dell, the Dell logo, Dell Boomi, Dell Precision ,OptiPlex, Latitude, PowerEdge, PowerVault,
PowerConnect, OpenManage, EqualLogic, Compellent, KACE, FlexAddress, Force10 and Vostro are

trademarks of Dell Inc. Other Dell trademarks may be used in this document. Cisco Nexus, Cisco MDS , Cisco NX-

0S , and other Cisco Catalyst are registered trademarks of Cisco System Inc. EMC VNX , and EMC Unisphere are

registered trademarks of EMC Corporation. Intel , Pentium , Xeon , Core and Celeron are registered trademarks of

Intel Corporation in the U.S. and other countries. AMD is a registered trademark and AMD Opteron, AMD

Phenom and AMD Sempron are trademarks of Advanced Micro Devices, Inc. Microsoft , Windows , Windows

Server , Internet Explorer , MS-DOS , Windows Vista and Active Directory are either trademarks or registered

trademarks of Microsoft Corporation in the United States and/or other countries. Red Hat and Red Hat Enterprise

Linux are registered trademarks of Red Hat, Inc. in the United States and/or other countries. Novell and SUSE are

registered trademarks of Novell Inc. in the United States and other countries. Oracle is a registered trademark of

Oracle Corporation and/or its affiliates. Citrix , Xen , XenServer and XenMotion are either registered trademarks or

trademarks of Citrix Systems, Inc. in the United States and/or other countries. VMware , Virtual SMP , vMotion ,

vCenter and vSphere are registered trademarks or trademarks of VMware, Inc. in the United States or other

countries. IBM is a registered trademark of International Business Machines Corporation. Broadcom and

NetXtreme are registered trademarks of Broadcom Corporation. Qlogic is a registered trademark of QLogic
Corporation. Other trademarks and trade names may be used in this document to refer to either the entities claiming
the marks and/or names or their products and are the property of their respective owners. Dell disclaims proprietary
interest in the marks and names of others.
Table of contents
Revisions (required) .......................................................................................................................................................................... 2
Executive summary .......................................................................................................................................................................... 4
1 Summary ..................................................................................................................................................................................... 5
2 Dual SPERC in PowerEdge VRTX ............................................................................................................................................ 6
2.1 Customer Situation ......................................................................................................................................................... 6
2.1.1 PowerEdge VRTX Solution ............................................................................................................................................ 7
3 What needs to happen for this to work ................................................................................................................................. 8
3.1 Ordering the Parts for a Secondary PERC .................................................................................................................. 9
A Configuration details ................................................................................................................................................................ 11
B PowerEdge VRTX Storage Infrastructure............................................................................................................................. 12
C Additional resources................................................................................................................................................................ 13
Executive summary
The firmware necessary to provide the Dell PowerEdge VRTX redundant PERC controller is not available
until late Q1 FY15 or early Q2 FY15. This documentation entails the information of how, in the Dell
Solution Center, 4 different Proof Of Concepts with customers we were able to determine that the current
firmware and PERC controllers can provide a fail-over level of protection in the current Dell PowerEdge
VRTX.

This process requires the customer to obtain a secondary SPERC controller that can be kept installed in
the secondary SPERC bay within the VRTX appliance. Until release of the firmware activating the
secondary SPERC and providing HA SPERC failover this secondary controller slot will not interact or
attempt to manage the VRTX shared storage.

Taking advantage of this secondary SPERC in the event of a PRIMARY SPERC failure, all actions taken
within the VRTX appliance will be a production disruption level event.
1 Summary
This Technical White paper pertains to the customer use cases, tested configurations and the results of
our testing with the Dell PowerEdge VRTX and the Dual SPERC controller in a live Proof Of Concept
environment within the Dell Solution Centers. As we have all been informed, the Dell PowerEdge VRTX
will have a firmware capability of providing an Active-Passive Shared PERC controller later in FY15. Our
first customer to come to the Dell Solution Centers was impressed with the Dell PowerEdge VRTX but was
unwilling to deploy his 70 sites with this product until there was a way to protect the local shared storage
with a redundant PERC controller. We installed the secondary SPERC controller and tested the hardware
failure of the primary SPERC controller. The secondary SPERC was able to import the Virtual Disk
configuration(s) (meta data) from the physical drives within the VRTX local shared storage After replacing
the primary SPERC with the secondary SPERC and bringing the VRTX appliance back online. None of the
local shared storage drives were considered Foreign due to their Virtual Disks creation being handled by
the same model of SPERC controller.

What this paper is and is not

This technical white paper contains the testing and methods of validation with customers via the Dell
Solution Center.

Table 1
Is Is not

Tested and validated claim results Untested/not validated claims (Marketing

claims only)
Product user manual
Product configuration guide
Blog/Wikis
2 Dual SPERC in PowerEdge VRTX
The current Firmware versions of the CMC and Shared PERC controllers do not support Active-Passive
nd
failover. The current recommendation is to not install the 2 Shared PERC controller and Riser card in a
PowerEdge VRTX as the results did not appear to be conclusive as to what would occur during initial
testing. The concern was that the Shared PERCs would fight for control of the data and that introducing a
secondary shared PERC after the fact would require a rebuild of the subsystem.

Note: These tests were all performed on A Rev PowerEdge VRTX hardware with current levels of
firmware and software available from support.dell.com at time of testing. Understand that many minor
changes have occurred since beta release and into the current A Rev configuration that may have caused
many of the previous results prior to our tests.

2.1 Customer Situation

The initial customer coming into the Dell Solution Centers was concerned with a redundant access to the
storage located on the PowerEdge VRTX which is accessed by all 4 nodes in the server chassis. The
systems were slated to ship to remote locations where IT support staff would not be able to provide direct
support. Preconfiguring these systems with an installation check-list for non-IT staff was the end goal.

1. The main points covered regarding a redundant controller in a Dell Server.

a. At the cost point you can use software and a second chassis if the customer is insistent on
redundant hardware.
b. Servers dont generally have a redundant controller. Dell is making an Active-Passive second
PERC available later this year FY15.
Note: These points were covered with the customer - How many servers have you seen that have a
second controller? And how many servers does Dell sell with 2 controllers? The PowerEdge R720 has
2 PERCs, but they are not redundant.
c.A Second SPERC for customer replacement can be purchased as spare parts today. It can be
placed on a shelf and if the first controller fails it takes little time to unseat primary SPERC
controller and replace with backup SPERC controller.
d. And last, we covered the Mean Time Between Failure Rate (MTBF) of a PERC, as it is a large
number of hours of continuous use.
2. Finally the customer noted they would not be purchasing the system and would purchase a
competitive product and backup software. They asked Dell to let them know when a system
capable of handling the redundant controller would be available.
th
Note: The Secondary PERC for the PowerEdge VRTX will launch with the Q1 block, March 11 , 2014
2.1.1 PowerEdge VRTX Solution
The customer was informed that the PowerEdge VRTX motherboard was already pinned, wired, and ready
for a secondary controller. The only thing remaining was the firmware to support the Active-Passive
configuration of the secondary redundant shared PERC controller as well as additional SAS backplane with
correct cross-connect cabling between the two SAS backplanes.
b. It was determined, after some consternation, that while customer was onsite at the Dell Solution
Centers the hosting Solutions Architect would build out the VRTX environment to match their desired
configuration while having the secondary SPERC controller installed within the VRTX appliance.
c. The secondary Shared PERC and riser to attach to the shared storage and brought the PowerEdge
VRTX online.
d. Once VRTX had secondary SPERC installed, ESX5.1 U1 was installed on all blades then creating a central
DataStore from the VRTX local shared storage so each ESX install could see the same
DataStore. Virtual Machines were created on each ESX host with respective VMDKs residing on
the DataStore from VRTX shared storage. With all VMs running the customer physically pulled
the power cables of the VRTX appliance.

e. Customer then performed the physical replacement of primary SPERC with backup SPERC controller
and reconnected power to VRTX appliance.

f. Once VRTX was online and ESX hosts were backup, a simple Rescan within each ESX host
Configuration->Storage area brought back the DataStore created from VRTX local shared storage.

Figure 1 PowerEdge VRTX simplified Block view Tested solution.

PERC 2 Expander 2 and

SAS Cables
3 What needs to happen for this to work
PowerEdge VRTX can support up to four servers (M520 or M620s or two M820s) that connect to the
infrastructure through a mid-plane. These are the same blades found within the M1000e chassis with 2
minor modifications:
a. The DRAC firmware has been flashed to communicate with the Chassis Management
Controller (CMC) in the PowerEdge VRTX.
b. Spare Part 6YCPB to attach the Mezzanine slots of the M-series blade to the PCIe bus
architecture of the PowerEdge VRTX. (1 Part per Mezzanine / fabric) Without these adapters,
the M-series blade will only communicate on the LOM fabric to and from the chassis.

The system Planar provides dual switches to direct traffic from the server to the other PCIe based
elements: the PERC controllers and PCI-e slots.

As with the PowerEdge M1000e chassis, the CMC plugs into the planar and serves as the main entity to
control switch paths, sequence the power-on of blades, but in the PowerEdge VRTX it also controls the
PCIe cards etc.

You will need to have two (2) Shared PERC8 controllers for this to work. They both dock into the planar
and control the shared storage subsystem. This subsystem supports up to two (2) expanders and supports
either 2.5 or 3.5 disk options.

Figure 2
 The PowerEdge VRTX appliance provides a converged infrastructure platform for server and storage
through the use of a dual shared PCIe Switch domain, which used Multi-path IO to increase the
throughput within the infrastructure.

Each server/node looks like a server with standard PCIe slots and direct attached storage, rather than a
blade with mezzanine cards attached to SANs. Server Fabrics B and C use PCIe Mezzanine cards (spare
part number 6YCPB you will need 1 for each fabric connection (B or C)) instead of the typical blade
Ethernet, Fibre Channel or InfiniBand cards. This helps reduce the complexity and increase the
deployment configurations as in a standard rack or tower server.

Each of server/node PCIe mezzanine cards attach to two PCIe Switches located on the VRTX planar.
These switches also utilize a Multi-path IO chip to increase throughput and connectivity of multiple
devices to a single PCIe path.

The switch chips on the planar of the PowerEdge VRTX are 2 PCIe chips, both with (multipath IO) MR IOV
chips for multipath functionality. The theoretical limit of each PCIe chip is 8 lanes of 10Gb of traffic bi-
directional. The actual limit is closer to 20-40% of the theoretical limit. With the MR IOV chips we see
above the standard 20-40% limits and closer to the theoretical limits of the full 10Gb per lane.

These PCIe switches connect the servers to the PCIe slots, as well as to 2 (PowerEdge RAID Controller)
PERC-mini slots. The PERC cards are shared across all four server nodes, through SR over MR-IOV
technology enabled with the PCIe Switches.

The PERC cards connect to SAS Expanders and a shared backplane for fault tolerant SAS based storage,
similar to an external PERC to SAS JBOD architecture on any existing Dell PowerEdge architecture. The
PCIe and SAS topologies maintain a full dual-path from the servers to the PERC cards to the storage array.

3.1 Ordering the Parts for a Secondary PERC

Ordering the Secondary PERC can be done today but only as Spare Parts. You can set it on a shelf or
install it as we did with our customers.

Table 2 Ordering Secondary PERC for PowerEdge VRTX prior to March 11, 2014.
SKU Description

P3WV4 Assembly, Card, CTL, SPERC8, 1GB, VRTX

TJ2VK Backplane Expander

And one of these depending on the chassis configuration

T2XF6 MiniSAS cable for single expander config in 2.5 chassis

JC49N MiniSAS cable for single expander config in 3.5 chassis

Once the firmware is available for the Active Passive firmware support of the PERC H810, you will be able
to order a simple customer kit and onsite support / installation.
Table 3 Ordering Secondary PERC for 2.5 backplane option - PowerEdge VRTX after March 11, 2014.
(Release of redundant PERC Firmware)
SKU Description

386-BBBN 2nd SPERC8 Ctl for VRTX 2.5 CK

P1V45 Customer Kit,Kit,SPERC8,DUAL,2.5

Recommended services for secondary PERC

996-3219 Deployment Consulting 1 Yr 1Case Remote Consulting Service

984-1037 Installation of a Dell Server, Storage or Peripheral Device, PE Server HWT

Table 4 Ordering Secondary PERC for 3.5 backplane option - PowerEdge VRTX after March 11, 2014.
(Release of redundant PERC Firmware)
SKU Description

386-BBBM 2nd SPERC8 Ctl for VRTX 3.5 CK

P1V45 Customer Kit,Kit,SPERC8,DUAL,3.5

Recommended services for secondary PERC

996-3219 Deployment Consulting 1 Yr 1Case Remote Consulting Service

H66M6 Installation of a Dell Server, Storage or Peripheral Device, PE Server HWT

A Configuration details
Table 5 Component table for PowerEdge VRTX tested in POCs.

Component Description

Operating system ESXi 5.1 U1 was the hypervisor installed on each blade. Each blade then
had two Virtual Machines respectively installed, one VM running RHEL
and one VM running Windows Server 2008 R2

Driver version NIC driver, storage drivers, etc.

Main-Board firmware 1.0.A00.201307102026

Cabling If applicable power, network, and/or other cabling

Server PowerEdge VRTX, CPU type, memory, chassis disks, etc.

M620, CPU type, memory, internal disks, etc.
M520, CPU type, memory, internal disks, etc.

CMC active & standby 1.00.A00.201306122025

firmware

Storage Controller PowerEdge VRTX chassis drives

SPERC 8, 25 x 1.2 TB 10K RPM SAS drives

Switch 1GB VRTX Switch

B PowerEdge VRTX Storage Infrastructure
One of the key differentiators for PowerEdge VRTX is the integrated shared storage. This was designed in
response to customer requests for virtualization support at remote sites, which requires shared storage for
VM migration. To date, most shared storage solutions have been too expensive for these types of
applications. By leveraging the PERC SRIOV and PCIe Switch SR over MR features enables this low cos
6Gb/s DAS solution and virtualization support.

The elements for the PowerEdge VRTX shared storage is shown in Figure 3 below. The PCIe Switch
provides the SR over MR functionality, mapping virtual functions to multiple servers through virtual PCIe
hierarchies. Another differentiator is the ability to build multiple RAID sets by selected specific physical
hard drives, and at the host level, the PERC drivers are assigned to virtual functions which map to
corresponding virtual drives in the storage array.

In this architecture, the host has no control over physical storage array devices; instead, the CMC and
MPC control the physical drive mappings and status.

Figure 3 PowerEdge VRTX Shared Storage Infrastructure

Note: Storage sharing does not require any high availability features and can be done with a single
PERC card and SAS expander. Shared storage is supported with either single or dual PERC/Expander
configurations.
C Additional resources
Support.dell.com is focused on meeting your needs with proven services and support.

DellTechCenter.com is an IT Community where you can connect with Dell Customers and Dell employees
for the purpose of sharing knowledge, best practices, and information about Dell products and
installations.

Referenced or recommended publications:

Standard URL for information and specifications

PowerEdge VRTX http://www.dell.com/PowerEdge/VRTX

PMBus http://pmbus.info/specs.html
Power System Management
Protocol Specification, v1.2
SAS http://www.t10.org/ftp/t10/drafts/sas1/sas1r10.pdf
Serial Attached SCSI, v1.1
SATA https://www.sata-io.org/secure/spec_download.asp
Serial ATA Rev. 2.6; SATA II, http://www.sata-io.org/docs/S2Ext_1_2_Gold.pdf
SATA 1.0a Extensions, Rev.
1.2
SMBIOS http://www.dmtf.org/standards/smbios/
System Management BIOS
Reference Specification,
v2.7
TPM https://www.trustedcomputinggroup.org/downloads/s
Trusted Platform Module pecifications/tpm/tpm
Specification, v1.2
UEFI http://www.uefi.org/specs/
Unified Extensible Firmware
Interface Specification, v2.1
USB http://www.usb.org/developers/docs/
Universal Serial Bus
Specification, Rev. 2.0
Windows Logo http://www.microsoft.com/whdc/winlogo/hwrequirem
Windows Logo Program ents.mspx
System and Device
Requirements, v3.10