EBG Server Education: GX71155C This Course Is Owned and Published by EBG Server Education

EBG Server Education
Servicing the NeXtScale System

n1200 Enclosure (5456)
and
nx360 M4 Node (5455)
April 2014
Study guide
GX71155C
This course is owned and published by

EBG Server Education.
___________________________________________________________________________________
Copyright © 2014, Lenovo Corporation.
Lenovo
8001 Development Drive
Morrisville, North Carolina, 27560
Lenovo reserves the right to change product information and specifications at any time
without notice. This publication might include technical inaccuracies or typographical
errors. References herein to Lenovo products and services do not imply that Lenovo
intends to make them available in all countries. Lenovo provides this publication as is,
without warranty of any kind, either expressed or implied, including the implied
warranties of merchantability or fitness for a particular purpose. Some jurisdictions do
not allow disclaimer of expressed or implied warranties. Therefore, this disclaimer may
not apply to you.
Data on competitive products is obtained from publicly obtained information and is
subject to change without notice. Contact the manufacturer for the most recent
information.
Lenovo and the Lenovo logo is a trademark or registered trademark of Lenovo
Corporation or its subsidiaries in the United States, other countries, or both. Intel and
the Intel logo is a trademark or registered trademark of Intel Corporation or its
subsidiaries in the United States, other countries, or both. Other names and brands are
the property of their respective owners.
The following terms are trademarks, registered trademarks, or service marks of Lenovo:
Access Connections, Active Protection System, Automated Solutions, Easy Eject Utility,
Easy-Open Cover, IdeaCentre, IdeaPad, ImageUltra, Lenovo Care, MaxBright,
NetVista, New World. New Thinking, OneKey, PC As A Service, Rapid Restore, Remote
Deployment Manager, Rescue and Recovery, ScrollPoint, Secure Data Disposal,
Skylight, Software Delivery Center, System Information Gatherer, System Information
Reporter, System Migration Assistant, System x, Think Pad, ThinkAccessories,
ThinkCenter, ThinkCentre, ThinkDisk, ThinkDrive, ThinkLight, ThinkPad, ThinkPlus,
ThinkScribe, ThinkServer, ThinkStation, ThinkStore, ThinkVantage, ThinkVision,
ThinkWorld, TopSeller, TrackPoint, TransNote, UltraBase, UltraBay, UltraConnect,
UltraNav, VeriFace.
For more information, go to: http://www.lenovo.com/legal/copytrade.html.
The terms listed for the following partners are the property of their respective owners:
AMD
Intel
IBM
Microsoft
NVIDIA
April 2014 2 GX71155C

Servicing the IBM NeXtScale System n1200 Enclosure (5456) and nx360 M4 Node (5455) – Table of
contents
Table of contents
Objectives .................................................................................................................. 4
Servicing the NeXtScale System n1200 Enclosure (5456) ......................................... 5

Overview .............................................................................................................. 5
NeXtScale System n1200 Enclosure .................................................................... 5
Product front view ................................................................................................. 6
Product inside view............................................................................................... 7
Product rear view.................................................................................................. 9
Product description ............................................................................................. 10
IBM NeXtScale System n1200 Enclosure features and specifications..................... 16
System configurations and diagrams ....................................................................... 18
Diagrams ............................................................................................................ 18
FPC web interface .............................................................................................. 19
Problem determination and troubleshooting ............................................................ 27
Servicing the IBM NeXtScale System nx360 M4 Node (5455) .................................. 29
Overview .................................................................................................................. 29
NeXtScale System nx360 M4 Node ................................................................... 29
Product front view ............................................................................................... 30
Product inside view............................................................................................. 31
Product rear view................................................................................................ 32
Product description ............................................................................................. 33
IBM NeXtScale System nx360 M4 features and specifications ................................ 35
System configurations and diagrams ....................................................................... 37
Memory considerations ...................................................................................... 37
HDDs considerations .......................................................................................... 38
IBM NeXtScale Storage Native Expansion (NeX) Tray ...................................... 40
Connecting to the IBM NeXtScale Storage Native Expansion (NeX) Tray ......... 42
IBM NeXtScale PCIe Native Expansion (NeX) Tray ........................................... 46
Connecting to the IBM NeXtScale PCIe Native Expansion (NeX) Tray .............. 49
Diagrams ............................................................................................................ 51
Problem determination and troubleshooting ............................................................ 54
Helpful links ............................................................................................................. 55
Summary ...................................................................................................................... 56
April 2014 3 GX71155C

Servicing the IBM NeXtScale System n1200 Enclosure (5456) and nx360 M4 Node (5455) – Objectives
Objectives
After completing this course, you will be able to:
1. Provide an overview of the IBM NeXtScale System n1200 Enclosure and nx360
M4 Node.
2. Describe the features of the IBM NeXtScale System n1200 Enclosure and nx360
M4 Node.
3. Describe the components within the IBM NeXtScale System Enclosure and Node
and their locations.
4. Describe the problem determination steps and explain how to troubleshoot the
IBM NeXtScale system.
April 2014 4 GX71155C

Servicing the IBM NeXtScale System n1200 Enclosure (5456) and nx360 M4 Node (5455) – Overview
Servicing the NeXtScale System n1200

Enclosure (5456)
Overview
NeXtScale System n1200 Enclosure

The NeXtScale system is the next generation of an IBM iDataPlex System with a 6U
enclosure, 12 horizontal node bays in the front, and can hold up to six power supplies.
Ten fans and one FPC (Fan/Power Control) module are in the rear. The system
provides better density and more flexibility, as well as cost efficiency by sharing power,
fan, and mechanical resources.
Figure 1: IBM NeXtScale System n1200 Enclosure
April 2014 5 GX71155C

Product front view

Figure 2 shows the IBM NeXtScale System n1200 Enclosure with 12 nx360 M4 Nodes.
Figure 2: IBM NeXtScale System n1200 Enclosure front view
April 2014 6 GX71155C

Product inside view
Figure 3: IBM NeXtScale System n1200 Enclosure, front inside view
April 2014 7 GX71155C

Figure 4: IBM NeXtScale System n1200 Enclosure, rear inside view
April 2014 8 GX71155C

Product rear view

Figure 5 shows the NeXtScale System n1200 Enclosure with six power supplies and 10
fans.
Figure 5: IBM NeXtScale System n1200 Enclosure rear view
April 2014 9 GX71155C

Product description
The NeXtScale System Enclosure has an outer skeleton with x86 Intel compute nodes
inside. Unlike Blade and Pure System, the NeXtScale System does not have internal
I/O switches. Use standard switches with the NeXtScale System. Aside from FPC,
which provides management to the fan and power, the system does not have extra
management mechanisms.
Figure 6 shows the IBM NeXtScale System 1U, half-wide compute Node
Figure 6: IBM NeXtScale System nx360 M4 Node
April 2014 10 GX71155C

Figures 7 and 8 display front and rear enclosure locations for the IBM NeXtScale
System.
Bay 1 starts at the lower left of the chassis and ends with bay 12 at the upper right front
side corner. At the rear, it displays locations for 10 fans, six power supplies, and one
FPC module.
Figure 7: IBM NeXtScale System n1200 Enclosure bay locations
All power supply modules are combined into a single power domain within the
enclosure, which distributes power to each of the compute nodes and ancillary
components through the enclosure midplane. The midplane is a highly reliable design
with no active components. Each power supply is designed to provide fault isolation and
is hot swappable.
April 2014 11 GX71155C

Figure 8: IBM NeXtScale System n1200 Enclosure rear power supply, fan, and FPC module
locations
The FPC module is the management component to control the power bank of the entire
chassis and gives the power permission to each compute node. The FPC web interface
can be accessed through an Ethernet connection. The default IP address is
192.168.0.100. The default User Name is USERID (in capital letters) and the default
password is PASSW0RD (in capital letters) where 0 is zero.
Figure 9 shows the IBM NeXtScale System n1200 Enclosure FPC information tag at
the rear of the enclosure next to FPC module.
April 2014 12 GX71155C

Figure 9: FPC information tag
Figure 10 shows the IBM NeXtScale System n1200 Enclosure rear components
number tag and screw driver at the rear of the enclosure in between fan 1 and fan 7.
The screw driver is used for disassembling the HDD cage that is shown in Figure 11.
Figure 10: Rear components number tag and screw driver
April 2014 13 GX71155C

Figure 11: Disassembling a hard disk drive cage (1.8 inches)
Figure 12 shows the FPC Module with the following LEDs, control, and connector.
Figure 12: FPC Module
Power-on LED: When this LED is lit (green), it indicates that the fan and power
controller has power on.
Heartbeat LED: When this LED is lit (green), it indicates that the fan and power
controller is actively controlling the chassis.
Locator LED: When this LED is lit (blue), it indicates the chassis location in a rack.
April 2014 14 GX71155C

CHK LED: When this LED is lit (yellow), it indicates that a system error occurred. Check
the event log for more information.
Ethernet port activity (RJ-45) LED: When this LED is flashing (green), it indicates that
there is activity through the remote management and console (Ethernet) port over the
management network.
Ethernet port link (RJ-45) LED: When this LED is lit (green), it indicates that there is
an active connection through the remote management and console (Ethernet) port to
the management network.
Remote management (Ethernet) connector: The remote management connector (RJ-
45) is the management network connector for the FPC Module.
Figure 13 shows the FPC web interface.
Figure 13: FPC web interface
Use the FPC web interface to perform chassis management functions within a web
browser. It contains the following functions:
 Node status report.
 Chassis power and fan status report.
 Chassis power and fan configuration management.
 Chassis VPD information report.
 Chassis event log display, backup, and restore.
 FPC module management, setting, backup, reset, and restore.
Note that the FPC currently supports only the following browsers:
IE6 32 bit, IE7 32/64 bit, IE8 32/64 bit, Firefox 2.0.x 32/64 bit, Firefox 3.0.5 32/64 bit,
Firefox 4.0.1 32/64 bit and Firefox 21 or later.
April 2014 15 GX71155C

Servicing the IBM NeXtScale System n1200 Enclosure (5456) and nx360 M4 Node (5455) – Features
and specifications
IBM NeXtScale System n1200 Enclosure features and

specifications
The NeXtScale enclosure is 914.5 mm x 263.3 mm x 447 mm (36 x 10.36 x 17.59
inches). It is 6U high and implements 12 compute nodes. The basic form factor of the
compute node is 1U half-wide, up to six 900 W or 1300 W power supplies and 10 80-
mm fans. Both of these components are hot swappable.
Figure 14 shows the major components of the IBM NeXtScale System n1200
Enclosure. Remove the lift handle before pushing the enclosure into the rack.
Figure 14: IBM NeXtScale system n1200 Enclosure components
Sep 2013 16 GX71155

and specifications
Table 1 shows the comparison between the NeXtScale System Enclosure and Flex
System chassis.
Table 1: NeXtscale n1200 Enclosure and Flex System Enterprise Chassis
NeXtScale n1200 Flex System
Height (U) Six Ten
Compute node Twelve (1U half-wide) Seven full-wide or 14 half-

bays wide compute nodes
Switch bays None Four high-speed scalable

switches
Fan module Ten 80 mm (3.14 inches) Ten 80 mm x 8 + 40 mm x 2

maximum, dual motor (3.14 x 8 + 1.57 inch x 2)
maximum
Power supply Six (900 W or 1300 W) Six (2500 W) maximum

maximum
Control module Fan/Power Control (FPC) Chassis Management

Module (CMM)
Chassis height 263.3 mm (10.37 inches) 440 mm (17.32 inches)
Chassis depth 914.5 mm (36 inches) 801 mm (31.5 inches)
Chassis width 447 mm (17.6 inches) 447 mm (17.6 inches)
System weight 112 kg (247 lb.) verify 220.45 kg (486 lb.)

(maximum)
Note: Mixing 900 W and 1300 W power supplies into the same chassis is
prohibited, and 1300 W power supply only supports highline Vin (ac 220 V – 240
V).
Sep 2013 17 GX71155

Servicing the IBM NeXtScale System n1200 Enclosure (5456) and nx360 M4 Node (5455) – System
configurations and diagrams
System configurations and diagrams

Diagrams
Figure 15 shows the NeXtScale n1200 Enclosure midplane and its connecting
components. Midplane provides 12 V standby power to FPC, and 12 V power to all
other devices. For the FPC and IMM server nodes to communicate with each other,
there are 12 I2C bus connections in between them and the IMM is the bus master. FPC
can obtain the status of the computer nodes through the I2C bus.
Figure 15: The NeXtScale System n1200 Enclosure block diagram
April 2014 18 GX71155C

FPC web interface

Figure 16 shows the FPC function tabs and reset button.
Figure 16: FPC function tabs and reset button
There are six function tabs in FPC.
Summary: The Summary page displays the overall status and chassis information
including Chassis Front Overview and Chassis Rear Overview.
Power and cooling: There are five sections under power tab that is shown in Figure 17.
Figure 17: FPC power sections
 Power Overview: Displays the chassis level power consumption, node level power
consumption, and subsystems power consumption includes power supplies and fans.
 PSU Configuration: For the user to setup power supply redundancy mode shown in
Figure 18.
April 2014 19 GX71155C

Figure 18: FPC PSU configuration
Redundancy Mode: There are three different redundancy modes to choose from.
 No Redundancy: System could be throttled or shut down if one or more power
supplies are in faulty condition.
 N+1: Have one of the properly installed PSUs as redundant power supply so that
there is no impact to system operation or performance if any one of the PSUs is in
faulty condition, given that Oversubscription mode is not enabled.
 N+N: Have half of the properly installed PSUs as redundant power supplies so that
there is no impact to system operation or performance if half of the PSUs have
failed, given that Oversubscription mode is not enabled. That is, six PSUs properly
installed and three PSUs could fail without any impact when N+N and no
Oversubscription is applied.
PSU Oversubscription Mode: This function allows users to take advantage of the
extra power from the redundant power supply when the power supplies are in good
condition. When the redundancy fails, the PSU will shut down within one second if
system power loading is not corrected after the time limit. The FPC takes the action of
node throttling at such power emergency. Chassis performance could be impacted even
in redundancy mode if Oversubscription is also enabled. Note that Oversubscription is
applied only with N+1 or N+N redundancy mode. When enabled, Oversubscription
mode with N+1 PSU configuration, the total power available is equivalent to No
redundancy mode.
 Power Cap: For users to setup power capping/saving.
Figure 19 shows the power capping policy of the FPC Module. Users can choose either
the chassis or node level capping/saving through power capping policy. Power capping
allows users to set a wattage limit on power consumption.
April 2014 20 GX71155C

Figure 19: FPC power capping policy
There are four different power saving modes shown in Table 2.
Table 2: FPC power saving mode
Function Power saving mode Description
Disable Static maximum The system runs at full

performance speed (no throttling)
regardless of the workload.
Mode 1 Static minimum power The system runs in a

throttled (defined by the
implementation) regardless
of the workload.
Mode 2 Dynamic favor performance The system adjusts throttling

level that is based on
workload, attempting to favor
performance over power
saving.
Mode 3 Dynamic favor power The system adjusts throttling

level that is based on
workload, attempting to favor
power saving over
performance.
 Voltage Overview: Monitors the voltage rail on the FPC.
April 2014 21 GX71155C

 Power Restore Policy: For users to enable power restore policy as shown in Figure
20. When power restore policy is enabled, FPC remembers nodes that are already
powered on before the ac is abruptly lost and automatically turns on those nodes
when the ac is recovered.
Figure 20: FPC power restore policy
Cooling: There are three sections under cooling tab that is shown in Figure 21.
Figure 21: FPC cooling sections
 Cooling Overview: System fan speed is displayed in rpm. Error log is asserted when
fan speed is below lower critical threshold (1472 rpm).
Figure 22 shows the FPC Cooling overview. The NeXtScale System n1200 Enclosure
system fan is equipped with a dual motor and normally operates at 2000 to13000 rpm.
April 2014 22 GX71155C

Figure 22: FPC cooling overview
 PSU Fan Speed: Shows the power supply fan speed. PSU fan speed normally
operates at 5500 to 23000 rpm.
 Acoustic Mode: Users can set acoustic mode. There are three acoustic modes to
choose from as shown in Figure 23.
Figure 23: FPC acoustic mode selection
The three acoustic modes to choose from are:

 Mode 1: System fan speed is capped at 20% duty.
April 2014 23 GX71155C

System information: The system information tab contains the fixed VPD data. They are
Chassis VPD, midplane VPD, and FPC VPD. Figure 24 shows the Chassis VPD.
Figure 24: Chassis VPD
Event Log: This tab lets users view the SEL (System Event Log) and perform
backup/restore/restore to default operations. There are two sections that are shown in
Figure 25.
Figure 25: event log sections
 Event Log: SEL logs chassis level events so that user can get some clues of what
is going on in the chassis. A maximum of 512 event entries can be logged.
 USB Recovery: One USB storage device is used for FPC to preserve or migrate
SEL and user configurations (shown in Figure 12). This USB key must be mounted
on the FPC board to function correctly. When there is no data that is stored in the
USB key, factory default settings are applied for all configurations. User
configurations are automatically backed up to the USB key when they are set or
modified.
Figure 26 shows the USB Recovery functions. There are three functions on the USB
Recovery page.
 Backup: Backup SEL and chassis configurations to USB key. Chassis
configurations include power supply redundancy policy, oversubscription mode,
chassis or node capping/saving setting, acoustic mode setting and power restore
policy.
 Restore: Restore and apply the configurations that are stored in USB to FPC.
 Restore to Default: Return all settings to factory default setting.
April 2014 24 GX71155C

Figure 26: FPC USB Recovery function
Configuration: The configuration tabs have settings that are used to manage FPC
module. There are eight sections that are shown in Figure 27.
Figure 27: configuration sections
 Firmware Update (to update FPC Firmware. Shown in Figure 28.)
Figure 28: FPC Firmware Update page
 SMTP (To configure e-mail alert and SMTP Authentication. Shown in Figure 29.)
April 2014 25 GX71155C

Figure 29: SMTP Setting
 SNMP (To configure IPv4 and IPv6 Destination List. Shown in Figure 30.)
Figure 30: SNMP Setting
 PEF (Platform Event Filter) (Configured SMTP and SNMP traps allow user to
monitor chassis for selected events. SMTP/SNMP trap event types can be set in the
PEF page.)
 Network Configuration (allows user to modify networking parameters).
 Time Setting (to configure system time).
 User Account (to configure user roles).
 Web Service (web service lets user configure different HTTP/HTTPS ports for
connection and the web page timeout period.
April 2014 26 GX71155C

Servicing the IBM NeXtScale System n1200 Enclosure (5456) and nx360 M4 Node (5455) – Problem
determination and troubleshooting
Problem determination and troubleshooting

The NeXtScale System n1200 Enclosure problem determination methodology is similar
to the BladeCenter environment. Users can obtain the Event log information and
monitor hardware status from the FPC web interface and also check the hardware
(Fans and PSUs) error LED in the back of the NeXtScale n1200 system by sight. The
most efficient way to populate the chassis with nodes is from the bottom up. For
hardware, ensure that all connectors are intact and the correct parts are in the proper
locations according to installation and configurations guides.
Figures 31 and 32 shows the error led location of the power supply and the fan
modules.
Figure 31: Enclosure PSU error LED location
April 2014 27 GX71155C

Figure 32: Enclosure fan module fault LED location
April 2014 28 GX71155C

Servicing the IBM NeXtScale System

nx360 M4 Node (5455)
Overview
NeXtScale System nx360 M4 Node
The IBM NeXtScale System nx360 M4 is a 1U high and half-wide compute node
designed to fit the standard PCI card form factor and open standard management tools.
It is especially suitable for HPC (High Performance Computing) Technology, Enterprise
Grid, and Public Cloud environment.
Figure 33: nx360 M4 Compute Node
April 2014 29 GX71155C

Product front view

Figure 34 shows the front view of the IBM NeXtScale System nx360 M4. The IMM
management port is only for the IMM web interface connection.
Figure 34: IBM NeXtScale System nx360 M4 Node front view
April 2014 30 GX71155C

Product inside view

Figure 35 shows the IBM NeXtScale System nx360 M4 inside (exploded view). The
compute node supports up to one 3.5-inch, two 2.5-inch, or four 1.8-inch easy-swap
SAS or SATA hard disk drives.
Figure 35: IBM NeXtScale System nx360 M4 Node inside (exploded view)
April 2014 31 GX71155C

Figure 36: IBM NeXtScale System nx360 M4 Node inside view
Product rear view

Figure 37 shows the IBM NeXtScale System nx360 M4 compute node rear view. The
power connector is connected to midplane.
Figure 37: IBM NeXtScale System nx360 M4 Node rear view
April 2014 32 GX71155C

Product description
The NeXtScale System nX360 M4 compute node is a newly designed server with a
Romely platform and Ivy Bridge-EP processor. It supports up to two 130 W CPUs and
eight of 16 GB DIMMs for a total capacity of 128 GB. It is designed to support an
onboard Mezzanine card and one FHHL (Full Height Half Length) PCIe slot.
Figure 38: IBM NeXtScale System nx360 M4 Node
Figure 39 shows the Top Cover Ruler of the NeXtScale System nx360 M4 Node that is
used to indicate how far the node has been pulled out. The number on the ruler ranges
from seven to one, where digit one indicates the end of the node.
April 2014 33 GX71155C

Figure 39: IBM NeXtScale System nx360 M4 Node, top cover ruler
April 2014 34 GX71155C

and specifications
IBM NeXtScale System nx360 M4 features and

specifications
The NeXtScale nx360 M4 compute node is 658.8 mm x 41 mm x 216 mm (25.93 x 1.61
x 8.5 inches). The specifications are as follows:
 CPU/Chipset:
o Dual CPU/Socket R/Up to 130 W
o Processors: Intel Ivy Bridge-EP (6, 8, 10, 12 core)
o Chipset: Intel Patsburg - J with Intel node manager 2.0 supportability
o 8 GT/s QPI (Quick Path Interconnect)
 Memory:
o Eight DIMM slots, DDR3
o Support UDIMMs or RDIMMs
o 128 GB maximum capacity (16 GB per DIMM), one slot per channel
o Support memory mirroring and Chipkill independent mode (for x4-based
memory DIMMs)
 HDD:
o One 3.5-inch LFF SATA HDDs or two 2.5-inch SFF SATA/SAS HDDs or
four 1.8-inch SSD (3.3 V only) HDDs
o Two 2.5-inch SFF HDDs and four 1.8-inch SSD all support hardware RAID
function when a hardware RAID card is present
o Software RAID supports two 2.5-inch SATA and four 1.8-inch SSD with
RAID 0/1/10 function
o HDD Controllers: Intel Patsburg – J
 RAID:
o ServerRAID H1110 SAS/SATA Controller for IBM System X
o ServerRAID M1115 SAS/SATA Controller for IBM System X
 NIC:
o Intel Powerville LAN Controller supporting 2 x 1 GbE ports
 I/O:
o Two 1 GB Ethernet connector
o One management Ethernet
o Two PCIe slots
April 2014 35 GX71155C

and specifications
 X16 front PCIe slot on the riser card
 X16 Rear PCIe Slot (supports the GPU tray only)
o The onboard KVM console connector (with two USB, one COM port and
one VGA output ability)
 Firmware:
o BIOS: IBM UEFI
o BMC: IBM IMM V2 with FOD
o DIAG: DSA, Preboot DSA in UEFI environment
 System Management
o IMM V2
o UpdateXpress service package
o Tivioli Provisioning Manager for OS deployment
o Toolscenter support (ASU, BoMc, UXSPI, DSA, Server Guide and Scripting
Toolkit)
o xCAT
April 2014 36 GX71155C

Servicing the IBM NeXtScale System nx360 M4 Node – System configurations and diagrams
System configurations and diagrams

The NeXtScale nx360 M4 is a compute node of The NeXtScale n1200 Enclosure. It
connects to the midplane of the chassis with a single power connector. The FPC
module is inserted from the rear of the chassis and has connections with the IMMs in
the 12 compute nodes. All I/O connections are in the front of the node, including NIC
and KVM connections.
Figures 40 and 41 show the IBM NeXtScale System nx360 M4 Node console cable. It
provides two USB (USB 2.0), one COM port and one VGA port.
Figure 40: IBM NeXtScale System nx360 M4 Node, KVM console cable
Figure 41: IBM NeXtScale System nx360 M4 Node, KVM console cable front view
Memory considerations
The NeXtScale nx360 M4 node supports only one DIMM per channel. The DIMM’s
population rules should follow the population sequence as shown in Figure 42 and
April 2014 37 GX71155C

43.
DIMM Population Sequence (Independent Mode)- RDIMM, UDIM
Installed microprocessors DIMM connector population sequence
Microprocessor 1 4 -> 3 -> 1 -> 2
Microprocessor 1 and
4 -> 5 -> 3 -> 6 -> 1 ->8 -> 2 -> 7
Microprocessor 2
Figure 42: Independent Mode
DIMM Population Sequence (Mirrored Mode)- RDIMM, UDIM

Installed microprocessors DIMM connector population sequence
Microprocessor 1 (4, 3) -> (1, 2)
Microprocessor 1 and
(4, 3) -> (5, 6) -> (1, 2) -> (8, 7)
Microprocessor 2
Figure 43: Mirrored Mode
HDDs considerations
The NeXtScale nx360 M4 supports three types of HDDs cages. Figure 44 shows the
HDDs numbering of the different HDD cages.
Figure 44: HDDs numbering
April 2014 38 GX71155C

Figure 45 shows the HDDs numbering of the 1.8-inch SSD HDDs cages.
Figure 45: 1.8-inch SSD HDDs numbering
The NeXtScale nx360 M4 supports 2.5/3.5-inch HDDs and 1.8-inch SSD, the connector
type of the cable assembly also different if connect to planar, PCI card or internal
storage tray (3.5-inch HDDs only).
Table 3 shows the three different types of connectors for the HDDs cage while
connecting to the planar, PCI card or storage tray (for 3.5-inch HDDs only). The internal
HDDs numbering of the 3.5-inch HDDs cage will change from HD0 to HD7 when
connecting to the storage tray.
Table 3: NeXtscale nx360 M4 node HDDs cage and connectors
Cage/cable Connectors
assembly
3.5-inch HDDs Connects to planar Connects to PCI Connects to

cage card storage tray
(HD07)
April 2014 39 GX71155C


(HD07)
Does not support

(HD07)
Does not support
IBM NeXtScale Storage Native Expansion (NeX) Tray

The 3.5-inch HDDs cage model of NeXtScale nx360 M4 node also provides an internal
storage tray configuration which is called IBM NeXtScale Storage Native Expansion
(NeX) Tray shown in Figure 46.
April 2014 40 GX71155C

Figure 46: IBM NeXtScale Storage Native Expansion (NeX) Tray front and rear view
Figures 47, 48 and 49 show the IBM NeX Tray top view, which includes seven 3.5-inch
HDDs, and the bottom view.
Figure 47: IBM NeXtScale Storage Native Expansion (NeX) Tray top view
Figure 48: IBM NeXtScale Storage Native Expansion (NeX) Tray top view with 7 x 3.5-inch HDDs
Figure 49: IBM NeXtScale Storage Native Expansion (NeX) Tray bottom view
April 2014 41 GX71155C

The IBM NeXtScale Storage Native Expansion (NeX) Tray is a 1U height, half wide
storage node. The specifications are as follows:
 Supports 3.5-inch SATA/SAS HDDs node only
 Supports up to seven 3.5-inch LFF SATA/SAS HDDs
 Hardware RAID card should be connected from the computer node via riser1 PCIe
interface to support RAID functions (RAID 0/1/5)
Connecting to the IBM NeXtScale Storage Native Expansion (NeX) Tray

In order to connect the IBM NeXtScale Storage Native Expansion (NeX) Tray with the
nx360 M4 node. Procedures needed are as follows:
 Remove the top cover of the NeXtScale nx360 M4 compute node

 Route the HDDs cage signal cable of the NeXtScale nx360 M4 node as shown in
Figure 50 to ensure the connector can be retrieved when the IBM NeXtScale
Storage Native Expansion (NeX) Tray is attached.
Figure 50: IBM NeXtScale nx360 M4 node cable routing
 Place the IBM NeXtScale Storage Native Expansion (NeX) Tray on top of the
compute node, and align the guide pin of the NeX tray to the nx360 M4 compute
node, then close up the compute node with the IBM NeXtScale Storage Native
Expansion (NeX) Tray as shown in Figure 51.
April 2014 42 GX71155C

Figure 51: IBM NeXtScale nx360 M4 node connects to the NeX tray
 Connect the signal cable to the socket on the PCI riser card assembly in the
compute node, and route the signal cable as shown in Figure 52.
Figure 52: The NeX tray signal cable and cable routing
 Open the cover by pressing the latch.

 Connect the HDD 7 cable in the IBM NeXtScale Storage Native Expansion (NeX)
Tray to the signal cable from the 3.5-inch HDDs cage in the compute node, then
connect the SAS 0 and SAS 1 cable to the RAID card on the PCI rise card
assembly in the compute node as shown in Figure 53. Note that the SAS 0 cable
controls HDD 0 to HDD 3 and the SAS 1 cable controls HDD 4 to HDD 7.
April 2014 43 GX71155C

Figure 53: Connects the cables in the NeX tray to the compute node
 Route the cables properly to avoid interference. Install the HDDs to the NeX tray if
required.
 Place the top cover on the IBM NeXtScale Storage Native Expansion (NeX) Tray
and closed.
Figure 54 shows the IBM NeXtScale Storage Native Expansion (NeX) Tray connecting
to the NeXtScale nx360 M4 node.
Figure 54: IBM NeXtScale Storage Native Expansion (NeX) Tray connecting to the NeXtScale
nx360 M4 node
Figure 55 shows the release latch of the IBM NeXtScale Storage Native Expansion
(NeX) Tray.
April 2014 44 GX71155C

Figure 55: IBM NeXtScale Storage Native Expansion (NeX) Tray release latch
The IBM NeXtScale Storage Native Expansion (NeX) Tray has its own light path to
identify the location of failed HDDs.
Figure 56 shows the HDDs numbering when the IBM NeXtScale Storage Native
Expansion (NeX) Tray connects to the NeXtScale nx360 M4 node.
Figure 56: HDDs numbering of the IBM NeXtScale Storage Native Expansion (NeX) Tray when
connects to the nx360 M4 node
Figure 57 shows the HDDs error indicator and light path button of the IBM NeXtScale
Storage Native Expansion (NeX) Tray. Press the light path button after remove the IBM
April 2014 45 GX71155C

NeXtScale Storage Native Expansion (NeX) Tray out of the chassis, the HDD ERR
indicator will illuminate and point out which number of HDDs has failed.
Figure 57: IBM NeXtScale Storage Native Expansion (NeX) Tray HDDs error indicator and light
path
IBM NeXtScale PCIe Native Expansion (NeX) Tray

The NeXtScale system also comes with an optional PCIe expansion tray configuration,
which can provide more PCIe slots for GPU and ServerRAID card. The IBM NeXtScale
PCIe Native Expansion Tray also called the GPU tray is a 1U design attached to nx360
M4 compute node and transform to a 2U form factor.
Figure 58: IBM NeXtScale PCIe Native Expansion (NeX) Tray front and rear view
Figures 59 and 60 show the GPU tray top view and the bottom view.
April 2014 46 GX71155C

Figure 59: IBM NeXtScale PCIe Native Expansion (NeX) Tray top view
Figure 60: IBM NeXtScale PCIe Native Expansion (NeX) Tray bottom view
April 2014 47 GX71155C

The GPU tray supports up to two full high/full length/double width GPU cards. The riser
cards are used to expand the capacity and change the direction of the PCIe slot on
mainboard to Support GPUs. Note that CPU 2 should be installed before using riser 2.
The two risers are shown in Figure 61 and 62.
Figure 61: IBM NeXtScale PCIe Native Expansion (NeX) Tray riser 1
April 2014 48 GX71155C

Connecting to the IBM NeXtScale PCIe Native Expansion (NeX) Tray

Before installing the GPU tray, it is suggested to place the GPU tray on the foam pad
that came with the packaging in order to protect the PCIe riser-cage connectors as
shown in Figure 63.
To install the GPU tray into compute node, procedures needed are as follows:
 Remove the top cover and PCI riser bracket of the NeXtScale nx360 M4 compute
node
 Remove all PCI riser-cage assembly from the GPU tray and set aside
 Place the GPU Tray on top of the compute node, and align the guide pin of the GPU
tray to the nx360 M4 compute node, then close up the compute node with the IBM
GPU Tray as shown in Figure 64.
April 2014 49 GX71155C

Figure 64: IBM NeXtScale nx360 M4 node connects to the GPU tray
 If PCI adapters are available, install the adapters to the riser-cage assembly by
following the rules shown in Table 4.

Table 4: PCI adapter installation rules
GPU tray Riser 1 Riser 2 (CPU2 required)
Slot 1 GPU adapter
Slot 2 RAID card
Slot 4 GPU adapter
 Install the PCI riser-cage assembly to the compute node and connects related
cables as shown in Figure 65.
Figure 65: IBM NeXtScale PCIe Native Expansion (NeX) Tray with 2 GPU cards
 Check if both Riser1 and Riser 2 are properly connected to the compute node, and
then close the top cover as shown in Figure 66.
April 2014 50 GX71155C

Figure 66: IBM NeXtScale PCIe Native Expansion (NeX) Tray connecting to the NeXtScale nx360
M4 node
Note: 1300 W power supply is required when IBM NeXtScale PCIe Native
Expansion (NeX) Tray is in the configuration.
Diagrams
Descriptions for items that are listed in Figure 67 are as follows:
Figure 67: IBM NeXtScale System nx360 M4 Node block diagram
April 2014 51 GX71155C

DDR3 Bus 0-3:1DPC= one DIMM per channel. The Bus is composed of signals
(Address, Command, Control, and Data). The main function is to communicate between
the CPU and memory.
X16 PCIe3, X24 PCIe3 and X8 PCIe3: It supports PCIe1 (2.5 Gbps), PCIe2 (5 Gbps),
and PCIe3 (8 Gbps). The main function is to communicate between the CPU and PCIe
slot. These slots support up to 8 Gbps. CPU 1 is responsible for X24 PCIe3 and X8
PCIe3, while CPU2 is responsible for X16 PCIe3. CPU 2 should be installed before
using X16 PCIe3.
Two SATA 6 Gbps and 3 Gbps: PCH (Platform Communication Hub) supports SATA
Gen 3 (6 Gbps) only on ports 0 and 1, and supports 3 Gbps on Ports 2 and 3. When
installing four 1.8-inch SSD HDDs (two are SATA 6 Gbps and the other two are 3 Gbps)
with RAID function, due to the different pairs of RAID group, the performance of the
array will be downgraded to 3 Gbps. For example, if pair SSD A and C in a mirror
shown in Figure 68, its performance is lower than if pair SSD A and B in a mirror.
Figure 68: SATA transfer rate in different Mirror pair
DMI2 4Gbps: A DMI bus is similar to a PCIe bus and is used for communication
between the CPU and PCH.
April 2014 52 GX71155C

Figure 69 shows the internal connectors of the NeXtScale System nx360 M4 node.
Figure 69: IBM NeXtScale System nx360 M4 internal connectors
April 2014 53 GX71155C

Problem determination and troubleshooting

Since the NeXtScale System n1200 enclosure only provides power distribution and
cooling with no other node management functions to each node, each nx360 M4
compute node server is an independent System X server. The troubleshooting
techniques and tools are the same as the other System X servers.
When hardware or system errors are encountered, the System error light is illuminated.
Users can check the errors through the IMM web interface or collect DSA logs for
further analysis.
When the Check Log light is illuminated, it indicates that some events have occurred
and have been logged in the IMM event log.
Remember to power off the node before removing it from the NeXtScale System n1200
enclosure.
Figure 70 shows the front panel of the NeXtScale System nx360 M4 node.
Figure 70: IBM NeXtScale System nx360 M4 Node front panel
April 2014 54 GX71155C

Servicing the IBM NeXtScale System n1200 Enclosure (5456) and nx360 M4 Node (5455) – Helpful links
Helpful links
Table 1: Prerequisite courses
Title Link to materials
IBM Announcement Letter http://www-01.ibm.com/common/ssi/cgi-

bin/ssialias?subtype=ca&infotype=an&appname=iSource
&supplier=877&letternum=ENUSZG13-0248
IBM Redbooks http://www.redbooks.ibm.com/abstracts/tips1051.html
Installation and Service https://www-

Guide 947.ibm.com/support/entry/myportal/docdisplay?brand=5
000008&lndocid=MIGR-5093698
The Power Configurator tool http://www-

03.ibm.com/systems/bladecenter/resources/powerconfig.
html
YouTube n1200:
http://www.youtube.com/channel/UCo3OO3gVr1ScdyDq
O62G4Hg/videos?view=1&flow=list
nx360:
https://www.youtube.com/channel/UC53n0DrNorOmj6oX
XrtuucA/videos?flow=list&view=1
YouKu n1200:
http://u.youku.com/IBMNeXtScalen1200
nx360:
http://u.youku.com/IBMNeXtScalenx360M4
April 2014 55 GX71155C

Servicing the IBM NeXtScale System n1200 Enclosure (5456) and nx360 M4 Node (5455) – Summary
Summary
This course enabled you to:
1. Describe the features of the IBM NeXtScale System n1200 Enclosure, nx360 M4
Node and the Storage Native Expansion (NeX) Tray
2. Describe the components within the IBM NeXtScale System n1200 Enclosure,
nx360 M4 Node, the Storage Native Expansion (NeX) Tray and their locations
3. Highlight some similarities and differences between the IBM Flex System and the
IBM NeXtSWcale System
4. Describe the new features of the FPC module
5. Describe the problem determination steps and explain how to troubleshoot the
IBM NeXtScale System n1200 Enclosure, nx360 M4 Node and the Storage
Native Expansion (NeX) Tray.
April 2014 56 GX71155C

EBG Server Education: GX71155C This Course Is Owned and Published by EBG Server Education

Загружено:

Сведения о документе

Оригинальное название

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

EBG Server Education: GX71155C This Course Is Owned and Published by EBG Server Education

Загружено:

Авторское право:

Доступные форматы

EBG Server Education

Servicing the NeXtScale System

This course is owned and published by

April 2014 2 GX71155C

Servicing the NeXtScale System n1200 Enclosure (5456) ......................................... 5

Summary ...................................................................................................................... 56

April 2014 3 GX71155C

April 2014 4 GX71155C

Servicing the NeXtScale System n1200

NeXtScale System n1200 Enclosure

Figure 1: IBM NeXtScale System n1200 Enclosure

April 2014 5 GX71155C

Product front view

Figure 2: IBM NeXtScale System n1200 Enclosure front view

April 2014 6 GX71155C

Product inside view

Figure 3: IBM NeXtScale System n1200 Enclosure, front inside view

April 2014 7 GX71155C

Figure 4: IBM NeXtScale System n1200 Enclosure, rear inside view

April 2014 8 GX71155C

Product rear view

Figure 5: IBM NeXtScale System n1200 Enclosure rear view

April 2014 9 GX71155C

Figure 6: IBM NeXtScale System nx360 M4 Node

April 2014 10 GX71155C

Figure 7: IBM NeXtScale System n1200 Enclosure bay locations

April 2014 11 GX71155C

April 2014 12 GX71155C

Figure 9: FPC information tag

Figure 10: Rear components number tag and screw driver

April 2014 13 GX71155C

Figure 11: Disassembling a hard disk drive cage (1.8 inches)

Figure 12: FPC Module

April 2014 14 GX71155C

Figure 13: FPC web interface

April 2014 15 GX71155C

IBM NeXtScale System n1200 Enclosure features and

Figure 14: IBM NeXtScale system n1200 Enclosure components

Sep 2013 16 GX71155

Table 1: NeXtscale n1200 Enclosure and Flex System Enterprise Chassis

NeXtScale n1200 Flex System

Height (U) Six Ten

Compute node Twelve (1U half-wide) Seven full-wide or 14 half-

Switch bays None Four high-speed scalable

Fan module Ten 80 mm (3.14 inches) Ten 80 mm x 8 + 40 mm x 2

Power supply Six (900 W or 1300 W) Six (2500 W) maximum

Control module Fan/Power Control (FPC) Chassis Management

Chassis height 263.3 mm (10.37 inches) 440 mm (17.32 inches)

Chassis depth 914.5 mm (36 inches) 801 mm (31.5 inches)

Chassis width 447 mm (17.6 inches) 447 mm (17.6 inches)

System weight 112 kg (247 lb.) verify 220.45 kg (486 lb.)

Sep 2013 17 GX71155

System configurations and diagrams

Figure 15: The NeXtScale System n1200 Enclosure block diagram

April 2014 18 GX71155C

FPC web interface

Figure 16: FPC function tabs and reset button

There are six function tabs in FPC.

Figure 17: FPC power sections

April 2014 19 GX71155C

Figure 18: FPC PSU configuration

April 2014 20 GX71155C

Figure 19: FPC power capping policy