EMC XtremIO SolVe Generator

Solution for Validating your engagement

Topic
XtremIO Installation Reference

Selections
Select Cluster Version of XtremIO for OCE:: XtremIO 4.0.4 (4.0.4-41)
Select an XtremIO Online Cluster Expansion Resource:: Single-to-Dual Online Cluster Expansion
Preparation Procedure

Generated:

1:03 PM >
September 20, 2016

SolVe Generator Updated:

REPORT PROBLEMS
If you find any errors in this procedure or have comments regarding this application, send email to
SolVeFeedback@emc.com
Copyright 2017 EMC Corporation. All rights reserved.
Publication Date: September, 2016
EMC believes the information in this publication is accurate as of its publication date. The information is subject to
change without notice.
THE INFORMATION IN THIS PUBLICATION IS PROVIDED AS IS. EMC CORPORATION MAKES NO
REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS
PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS
FOR A PARTICULAR PURPOSE.
Use, copying, and distribution of any EMC software described in this publication requires an applicable software
license.
For the most up-to-date regulatory document for your product line, search for regulatory on the applicable product
page at https://support.EMC.com
For the most up-to-date listing of EMC trademarks, see the list of EMC Corporation Trademarks on EMC.com.
All other trademarks used herein are the property of their respective owners.

EMC CONFIDENTIAL

version: 1.2.0.0

1 of 12

EMC XtremIO SolVe Generator

Solution for Validating your engagement

Contents
How to Prepare a Single-to-Dual Online Cluster Expansion............................................3
Introduction.................................................................................................................................................. 3
Remote Online Cluster Expansion Planning Tasks......................................................................................4
Hardware Requirements.............................................................................................................................. 5
Verify Cabling of the Existing Cluster........................................................................................................... 5
Confirm Expansion X-Brick Compatibility..................................................................................................... 5
Rack Expansion X-Brick.............................................................................................................................. 7
Cable Expansion X-Brick -- IPMI, MGMT, SAS & Power.............................................................................7
Apply Power to Expansion X-Brick and Ready Hardware............................................................................8
Cable Expansion X-Brick -- InfiniBand Switches.......................................................................................... 8
Perform Network Pre-Checks for Cluster Expansion...................................................................................9
Perform Cluster Health Check.................................................................................................................... 11
Transfer Activity to Professional Services.................................................................................................. 11
Appendix A: Accessing the Storage Controller for Tunnel Usage..............................................................12

EMC CONFIDENTIAL

version: 1.2.0.0

2 of 12

EMC XtremIO SolVe Generator

Solution for Validating your engagement

How to Prepare a Single-to-Dual Online Cluster Expansion

Introduction
This document describes how to prepare an XtremIO single X-Brick cluster to be available for online
cluster expansion into a dual X-Brick cluster. This procedure is non-disruptive.
When expanding a cluster from a single to dual X-Brick topology, the active cluster must be running
version 4.0.4-41 (or above).
Exception: If the XtremIO cluster undergoing an OCE has previously been upgraded to 4.0.4-28, the
OCE may proceed as scheduled (no need to NDU from 4.0.4-28 to 4.0.4-41 or above).
If the cluster is running a version below 4.0.4-28, open a Service Request (SR) with RPS to schedule and
perform an NDU to upgrade the existing cluster to version 4.0.4-41 (or above).
In addition, the Storage Controllers of the expansion X-Bricks must be re-imaged with version 4.0.0-49.
MAPORTN:

OnlyoeCE/NDUFRaptircbdn.mTheskoprfialwtneh.

MAPORTN:

ToexpandsglicX-Bukrtm e,yofisndxpatulX-Brcek.Yohnxpadticlusrf,oeghX-Bk.tAspcduraiehogtnv-slxpa.i

MAPORTN:

InordetxpaSX-Bick,yufsonvertalc,byxdigune25SDs

MORTANP:

oSeratCgnsl()vdmfhciXtreIOuaspolmnFRUk()cetb-usrhdixpaoBXfmnOCE.

MAPORTN:

IfRecoPvsinrptahu'mXeIOoslnti,RcvrPu*bpaedghOCEnmt.TorifRecPvszd/nthlur,eXMCLIcomadsw-intrevy,adcsohinftrgw50:124RecPovintFCsrq9.-0mec:ovpintfRPSrCI.dscvey,ohiutmpa efsconiygrudtpalhwecsringOCEmt.IfueoqsiranchwdgpRovePt,lsacinrGbThSupot.Ifecsmnalrhopi,tdcewOCE.nhasbuelyfpcotdm,RPvnriabesu.ofEtMCKBA#4792radinlsef.

MAPORTN:

Foraelsin-tgduOCEvRPQfperclso,tinavS(P)mudhesclxpaonwitE eg(L3r).N:DlvyofthOCEmusapkceSndrFgyitbusaho;reIlSndT(im),Esta orPcfSdneTim()asp.t

Remote Online Cluster Expansion Planning Tasks

In advance of the Online Cluster Expansion event, complete the following tasks remotely:
1. [ ]
Ensure you have downloaded / updated the XtremIO SolVe Generator to the latest
available release. This is to ensure you utilizing the latest documentation, procedures and scripts
throughout the OCE engagement.
2. [ ]
Perform a cluster health check, collect a log file bundle, and upload each to the Service
Request (SR).The initial log file bundle and health check should be performed to ensure our baseline
cluster state for the OCE is stable. These tasks can be performed remotely via an ESRS or WebEx
connection.

EMC CONFIDENTIAL

version: 1.2.0.0

3 of 12

EMC XtremIO SolVe Generator

Solution for Validating your engagement
hIfetclusrakindv<yEcoprtCablend><ypioalsdDeb>,nuthrAESipocsmeuyrtnhfxpaiov.ReXmIOtrMdl&Cpaibyxofnset.Iuvrpacbmilyh,texnsovadcPjrM(g)sinetohamdrpeicotvans.
Example:
General Cluster Configuration Information:
-----------------------------------------XtremIO Version 4.0.4-28 running on vXMS on local vxmsxbrickhpk33
XtremIO Version 4.0.4-28 running on Cluster xbrickhpk33
XMS Management Port Configuration: Online 1000Mb/s Full Duplex
Cluster Size and Capacity: 1 X 40TB
Encryption: Capable and Enabled or Encryption: Capable and Disabled
vRAM:8GB vDisk:900GB
PSNT: XIO00161300777

In the above example, if the active cluster is < Encryption Capable and Enabled > or <
Encryption Capable and Disabled > a DAE containing SSDs of P/N 005050377 cannot be used
for the expansion event per the XtremIO Model & Compatibility Matrix.
3. [ ]
You must execute an encryption recovery procedure on the XtremIO cluster before
performing the Online Cluster Expansion (OCE) event in order to ensure the cluster is fully protected
from the conditions outlined within EMC Technical Advisory (ETA) # 484910. For a detailed procedure
explaining how to perform the encryption recovery procedure, refer to EMC Knowledgebase Article
(KBA) # 482666. HIGHLY IMPORTANT: Failure to follow this step may lead to a Data Loss (DL)
scenario on an affected XtremIO cluster.
4. [ ]
Ensure the Project Manager (PM) for this activity has provided a completed XtremIO
Installation Summary Form containing the IP addresses to be used for the cluster expansion.

Hardware Requirements
Expanding from a single to a dual X-Brick cluster requires converting the existing cabling infrastructure of
the single X-Brick cluster to support additional X-Bricks.
Make sure to locate the Single X-Brick Expansion Kit, which should be used for this purpose. Ensure the
expansion kit includes the following:

2 x C13-C14 power cables

4 x InfiniBand Switch cables
1 x DB9-RJ45 cable
1 x RJ45-RJ50 cable
1 x Label sheet set
1 x Black cable tie

The expansion X-Brick configuration kit is described in the Hardware Installation and Upgrade Guide. For
purposes of this expansion activity, an X-Brick 2 Configuration Kit is used.

Verify Cabling of the Existing Cluster

Perform the following actions to validate the physical cabling of the existing single X-Brick cluster adheres
to XtremIO cabling guidelines.
AOTN:IU

Duringsthweopcld, i.Dntoyecabslhurinectodsh,puwbrl.

EMC CONFIDENTIAL

version: 1.2.0.0

4 of 12

EMC XtremIO SolVe Generator

Solution for Validating your engagement
1 Make sure that the cluster components are properly connected to PWR-A and PWR-B, as
described in Chapter 5 of the Hardware Installation and Upgrade Guide.
5. [ ]

Verify that Battery Backup Unit 1 (X1-BBU1-Input) is connected to PWR-A.

6. [ ]

Verify that Battery Backup Unit 2 (X1-BBU2-Input) is connected to PWR-B.

7. [ ]
Verify that each BBU has exactly two power cables connected to its power feeds and that
no power cables are connected to the red sockets, as can be seen in the image below:

If any of the backend cabling is not correct, perform corrective action to bring the clusters cabling into
compliance with XtremIO guidelines. If needed, engage a local XtremIO Field Support Specialist for
assistance.

Confirm Expansion X-Brick Compatibility

ARNING: Ensure you complete this step prior to proceeding with any further tasks as part of the Online
Cluster Expansion (OCE). Failure to perform the below validation efforts may result in a DU event. Ensure
the expansion DAE SSDs support encryption if the active cluster is encryption capable. If you observe a
compatibility mismatch, halt all expansion activity and contact the Project Manager (PM) assigned to this
engagement to determine the appropriate corrective actions.
Visually inspect both the existing cluster and the expansion X-Bricks storage controller part numbers
(P/Ns) and DAE SSD part numbers (P/Ns). Confirm they match and are compatible per XtremIO Model &
Compatibility Matrix.
The P/Ns of the SCs can be checked using the available pull tab to reveal a label containing the P/N.
via the front of the SC.

Example: P/N derived from pull tab of SCs is equal to: 100-586-017-01

EMC CONFIDENTIAL

version: 1.2.0.0

5 of 12

EMC XtremIO SolVe Generator

Solution for Validating your engagement
The P/Ns of the DAE SSDs can be checked by examining the SSDs P/N via the clear filter pane via
the front of the DAE.

Example: P/N derived from front of DAE for SSDs is equal to: 005050674
In the example above, SCs of P/N 100-586-017-01 and SSDs of P/N 005050674 will comprise the cluster
as part of the expansion event. Per the XtremIO Model & Compatibility Matrix this is acceptable.
If you observe a compatibility mismatch, contact the Project Manager (PM) assigned to this engagement
to determine the appropriate corrective actions.

Rack Expansion X-Brick

1 Add the new X-Brick to the rack, as described in Chapter 3 of the Hardware Installation and
Upgrade Guide.
8. [ ]
Add the two InfiniBand Switches to the rack, as described in Chapter 3 of the Hardware
Installation and Upgrade Guide.
ARNING:

Do not connect the InfiniBand Switch to the active cluster.

Cable Expansion X-Brick -- IPMI, MGMT, SAS & Power

AOTN:IU

Dontceayblshurinotceds,hprwublBy.ignX2-SC1PUR,adBOMcblingeortphas cedur.
1 Connect the Ethernet (IPMI & MGMT) and SAS cables of the new X-Brick, as described in
Chapter 5 of the Hardware Installation and Upgrade Guide.
2 Connect all DAE and IBSW power cables of the new X-Brick, as described in Chapter 5 of
the Hardware Installation and Upgrade Guide.
9. [ ]
Prepare the two C13-C14 power cables for the new X-Brick, supplied in the X-Brick
Expansion Kit and connect them to the PWR-B rail. Leave the cables (hanging) in such a way that
they will be easy to connect to the following Storage Controller ports:
X1-SC1-PSU-L
X1-SC2-PSU-L

EMC CONFIDENTIAL

version: 1.2.0.0

6 of 12

EMC XtremIO SolVe Generator

Solution for Validating your engagement
ote: The intent of this action is to pre-stage the cables in such a manner so they may be easily connected to
the cluster at a future point during the expansion process. Ensure the cables can reach X-Brick 1 of the cluster
from their pre-staged location.
ARNING:

Do not connect these power cables to the active cluster; only to the PWR-B rail.

10. [ ]
Connect the two power cables of the new X-Brick (X-Brick number two) to the correct
power sources, according to the table below:
Source

Destination

X2-SC1-PSU-L

PWR-A

X2-SC2-PSU-L

PWR-A

This is described in Chapter 5 of the Hardware Installation and Upgrade Guide.

Apply Power to Expansion X-Brick and Ready Hardware

On each of the new Storage Controllers of the new X-Brick, do the following:
1. [ ]
Power up the new X-Bricks Storage Controllers as described in the EMC XtremIO
Storage Array User Guide.
2. [ ]
Re-image the new Storage Controllers with software version 4.0.0-49 using the USB
rescue image as described in the Software Installation and Upgrade Guide.

Cable Expansion X-Brick -- InfiniBand Switches

1 Prepare the four (4) InfiniBand Switch cables, supplied in the X-Brick Expansion Kit, and connect
them to the InfiniBand Switches according to the table below. Ensure to leave the cables (hanging) in
such a way that they will be easy to connect to the storage controllers at future point during the
expansion process.

IBSW1-P01
IBSW1-P02
IBSW2-P01
IBSW2-P02

Example:

EMC CONFIDENTIAL

version: 1.2.0.0

7 of 12

EMC XtremIO SolVe Generator

Solution for Validating your engagement

ARNING:

Do not connect the InfiniBand Switch to the active cluster.

3. [ ]
Connect the InfiniBand Switch cables according to the table below and as described in
Chapter 5 of the Hardware Installation and Upgrade Guide.

ARNING:

Source

Destination

X2-SC1-IB1

IBSW1-P03

X2-SC1-IB2

IBSW2-P03

X2-SC2-IB1

IBSW1-P04

X2-SC2-IB2

IBSW2-P04

IBSW1-P17

IBSW2-P17

IBSW1-P18

IBSW2-P18

Do not connect the InfiniBand Switch to the active cluster.

Perform Network Pre-Checks for Cluster Expansion

1 If needed, use one of the Storage Controllers of the cluster (in the existing X-Brick) to gain CLI
access, as described in the section entitled Appendix A: Accessing the Storage Controller for Tunnel
Usage below.
4. [ ]
Confirm SSH Firewall is unlocked on XMS using the following XMCLI command: showclusters-info vertical.
xmcli (tech)> show-clusters-info vertical

EMC CONFIDENTIAL

version: 1.2.0.0

8 of 12

EMC XtremIO SolVe Generator

Solution for Validating your engagement
Cluster-Name: xbrickhpk2-3
Index: 1
State: active
Conn-State: connected
Activation-Time: Thu Jan 28 18:09:11 2016
Start-Time: Thu Jan 28 18:09:11 2016
SW-Version: 4.0.4-28
PSNT: XIO00142200018
Encryption-Mode: disabled
Encryption-Supported: False
Encryption-Mode-State: none
Compression-Mode: enabled
SSH-Firewall-Mode: unlocked
OS-Upgrade-Ongoing: false
Cluster-Expansion-In-Progress: no
Upgrade-State: no_upgrade_done
5. [ ]
If it is < locked >, disable XMS SSH Firewall before starting procedure using the following
XMCLI command:
modify-ssh-firewall cluster-id=< cluster name > ssh-firewall-mode=unlocked
xmcli (tech)> modify-ssh-firewall cluster-id="xbrickhpk2-3" ssh-firewallmode=unlocked
Modified SSH Firewall
6. [ ]
Confirm TCP port 22 is opened between X1-SC1 and X1-SC2 and XMS using the
following XMCLI commands:
test-xms-tcp-connectivity port=22 server="<X1-SC1-IP>"
test-xms-tcp-connectivity port=22 server="<X1-SC2-IP>"
xmcli (tech)> test-xms-tcp-connectivity port=22 server="10.XXX.XXX.XX"
Done!
Connectivity checked successfully
7. [ ]
Obtain the list of IPs that will be assigned to the management port of the new storage
controllers and verify there are no duplicate IP addresses detected for the new storage controllers:
ote:
The IP addresses being assigned to the management ports of the new storage controllers must reside on
the same subnet of the existing cluster.
ote: Do not assign the IP addresses to the new storage controllers using the Easy Install CLI. The new IP
addresses will be assigned during the cluster expansion process.

Use the following XMCLI commands to perform this network validation:

test-ip-connectivity interface="sc_management" ip-addr="< X2-SC1-IP >" cluster-id="<
cluster name >" sc-id="X1-SC1"
test-ip-connectivity interface="sc_management" ip-addr="< X2-SC2-IP >" cluster-id="<
cluster name >" sc-id="X1-SC1"

EMC CONFIDENTIAL

version: 1.2.0.0

9 of 12

EMC XtremIO SolVe Generator

Solution for Validating your engagement
xmcli (tech)> test-ip-connectivity interface="sc_management" ipaddr="10.XXX.XXX.XX" cluster-id="xbrickhpk2-3" sc-id="X1-SC1"
1 * * *
2 * * *
3 * * *
4 * * *
5 * * *
6 * * *
7 * * *
8 * * *
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
Traceroute done
xmcli (tech)>
The expected response from this command is no successful traceroute responses as evident by
twenty (20) rows of three (3) asterisk ( * * * ) marks and a closing statement of Traceroute done.
8. [ ]
Verify with customer the TCP ports required for OCE are open between the XMS and
existing/expansion storage controllers.
TCP 11000 through 11032
TCP 22000 through 22032
TCP 23000 through 23032
The above ports are assigned two at the time for each storage controller so the range
accommodates a theoretical sixteen (16) X-Bricks.
It is not possible to verify ports of the expansion storage controllers (since they don't have an IP
address configured). However, a good indication may be obtained by checking the existing storage
controllers.
Use the following XMCLI command to validate X1-SC1: test-xms-tcp-connectivity
test-xms-tcp-connectivity port=11000 server="<X1-SC1-IP>"
test-xms-tcp-connectivity port=11001 server="<X1-SC1-IP>"
test-xms-tcp-connectivity port=22000 server="<X1-SC1-IP>"
test-xms-tcp-connectivity port=22001 server="<X1-SC1-IP>"
test-xms-tcp-connectivity port=23000 server="<X1-SC1-IP>"
test-xms-tcp-connectivity port=23001 server="<X1-SC1-IP>"
xmcli (tech)> test-xms-tcp-connectivity port=11000 server="10.XXX.XXX.XX"
Done!
EMC CONFIDENTIAL

version: 1.2.0.0

10 of 12

EMC XtremIO SolVe Generator

Solution for Validating your engagement
Connectivity checked successfully
The expected response from this XMCLI command for each port is: Connectivity checked
successfully.

Perform Cluster Health Check

1 Verify XtremIO cluster health using the latest Health Check Script (HCS) as available within the
XtremIO SolVe Generator. Ensure you use the < --full > parameter when executing HCS. Resolve any
failed items before proceeding with the remainder of this procedure.
xmcli (tech)> run-script script="system_health-vxxx.x.x-s4.0.0.py" arguments-list=["-cluster-id=< cluster name >", "--full"]
2 Confirm no alerts are outstanding on the cluster using the following XMCLI command: showalerts cluster-id=<cluster name>. Resolve any outstanding alerts before proceeding with the
remainder of this procedure.
xmcli (tech)> show-alerts cluster-id=xbrickhpk2-3
No matches found

Transfer Activity to Professional Services

DontprceduayhfwOliCsExon()tka.hcePjrMgsinodtOCEveharmfopcduebyamrPfsinlSvc().

Appendix A: Accessing the Storage Controller for Tunnel Usage

1 Confirm SSH Firewall is unlocked on XMS using the following XMCLI command: show-clustersinfo vertical.
xmcli (tech)> show-clusters-info vertical
Cluster-Name: xbrickhpk2-3
Index: 1
State: active
Conn-State: connected
Activation-Time: Thu Jan 28 18:09:11 2016
Start-Time: Thu Jan 28 18:09:11 2016
SW-Version: 4.0.4-28
PSNT: XIO00142200018
Encryption-Mode: disabled
Encryption-Supported: False
Encryption-Mode-State: none
Compression-Mode: enabled
SSH-Firewall-Mode: unlocked
OS-Upgrade-Ongoing: false
Cluster-Expansion-In-Progress: no
Upgrade-State: no_upgrade_done
9. [ ]
If it is < locked >, disable XMS SSH Firewall before starting procedure using the following
XMCLI command:

EMC CONFIDENTIAL

version: 1.2.0.0

11 of 12

EMC XtremIO SolVe Generator

Solution for Validating your engagement
modify-ssh-firewall cluster-id=< cluster name > ssh-firewall-mode=unlocked
xmcli (tech)> modify-ssh-firewall cluster-id="xbrickhpk2-3" ssh-firewallmode=unlocked
Modified SSH Firewall
10. [ ]
Open an SSH tunnel for Technician to XMS. The tunnel can only be established from the
Technician port on X1-SC1 or X1-SC2. Use the appropriate < cluster name > and < sc-id > to open
the tunnel from the correct SC. In the following example SSH tunnel will be open from < X1-SC2 > of
cluster < xbrickhpk2-3 > to the XMS. You may need the customer to assist you with this task if direct
access to the XMS is not available.
modify-technician-port-tunnel cluster-id=xbrickhpk2-3 sc-id=2 open
xmcli (tech)> modify-technician-port-tunnel cluster-id="xbrickhpk2-3" scid=2 open
Opened Tech port tunnel
11. [ ]

Connect the laptop to a Technician port and use an SSH client to login into X1-SC2.

ssh -l xmsadmin 169.254.254.1 -p 10022

xmsadmin@169.254.254.1's password:
xxxxxxxx
Username: tech
Password: xxxxxxxx
12. [ ]
Close Technician SSH tunnel after work is completed using the following XMCLI
command:
modify-technician-port-tunnel cluster-id=xbrickhpk2-3 sc-id=2 close
xmcli (tech)> modify-technician-port-tunnel cluster-id="xbrickhpk2-3" scid=2 close
Closed Tech port tunnel

EMC CONFIDENTIAL

version: 1.2.0.0

12 of 12