Вы находитесь на странице: 1из 11

HBA Provisioning and Replacement Guide

How to use this Guide


This guide is intended to provide instructions for the MHS Intel team regarding the
provisioning of any server with HBAs. There is a separate cluster document with special
instructions regarding the correct way to mount external, shared storage devices on a
cluster.

If you have are not familiar with Storage Area Networks then please see the included
glossary before starting. There is a great deal of terminology.

The purpose of this guide is to explain the proper configuration of server side hardware
and software needed to connect a Windows server to a Storage Area Network or Storage
device. SCSI devices, such as the MSA500, are not covered in this guide. Additional
instructions for replacing bad server side hardware, or hardware required to attach a
server to a Storage Area Network and some diagnostic steps, are also included.

The Basics (Verbose.)


Provisioning a Server (Hit the ground running.)
Replacing a HBA/Troubleshooting a Down Link
Glossary

Initial Provisioning of a Server with a Host Bus Adapter


(HBA)
Item(s) to check when taking ownership of any provisioning ticket with a networked
storage component:
Confirm that HBAs are installed on the server. Use Device Manager to
determine installed hardware.
If this server has a single HBA stress to the account team and the customer the
risks involved with this configuration.
o Multiple single points of failure.
o Downtime in the event there is a fault on the storage device, even one
the storage device recovers from gracefully.
o Risk of data corruption as a result of any of these failures.
o Try and get multipathing.
Have datacenter confirm cabling. If not complete inform the Project Manager
(or Service Delivery Manager), document the missing cabling in your ticket.
Negotiate next step with the project team. Obtaining and installing cabling
can take days, the DC may not have the cabling infrastructure or parts
required. It is the responsibility of the project management team to obtain the
parts and infrastructure. Without cabling it is not possible to install current
multipath software packages. Make this clear to PMs.
Check for information regarding final setup and document.
o Will the server be a member of a cluster?
If this server will be a cluster member review cluster
documentation for special instructions regarding storage.
o Shared or dedicated storage device?
o If shared, please be aware that shared storage is allocated in 10GB
chunks per presented LUN and you will lose a very modest amount of
space to overhead.
o Obtain desired size of final volume(s). This is often not the same as
space requested in the configuration workbook (CW) or even the space
requested for the logical unit name (LUN a logical disk presented by
the storage device). Frequently the CW will contain a lump sum of
all the space required and the LUN size copies that number, even
though it may be desired to have smaller separate volumes each on
their own LUN. Always consult owner of the application that will be
using the external storage devices.

Basic Steps For Installation (See Detailed Instructions Below for software location
and instructions specific to vendors.)

1. Install OS and Hardware patches, hotfixes, etc. to prevent known problems with HBA
required software. (This section is subject to frequent changes as it may contain
recent patches or workarounds.)
2. Install/Update HBA controller Storport drivers.
3. Install HBA controller software. This allows for troubleshooting of, and updates to,
the physical layer.
4. Update HBA controller firmware.
5. Install/Update SAN Communication software. (Not available for all vendors.)
6. Install Multipathing software.
7. Submit HBA zoning request and LUN provisioning to appropriate storage team.
8. Enable and format disks.

Detailed Instructions and Troubleshooting

1) Install OS and Hardware patches, hotfixes, etc. to prevent known problems with
HBA required software.
a) Hardware Requirements for Drivers, etc.
i) For HP Proliant Servers
(1) Update HP array drivers past 6.6.2.32 to prevent BSOD
(2) http://support.microsoft.com/kb/940015/
b) OS Requirements
i) Windows 2003
(1) Service Pack 1 or higher
(2) Microsoft Storport Hotfix 932755
ii) Windows 2000
(1) To be determined on case-by-case basis.
c) Not required. Oth hotfixes (as needed/case-by-case)
i) http://support.microsoft.com/default.aspx?scid=kb;[LN];Q940467
2) Install/Update HBA device drivers for Storport.
a) Emulex HP Provided.
i) Most common current configuration
(1) HP Provided FC2142SR/A8002A 4GB Adapter.
(2) \software$\HBA_Prov\ 2 -- HBA Storport Drivers\ X:\HBA_Prov\2 --
HBA Storport Drivers\HP Card\FC2142SR_A8002A\cp007110.zip
ii) Download/Install latest Storport drivers from HP for the card displayed under
device manager.
iii) First check the software share \HBA_Prov\ 2 -- HBA Storport Drivers for
comparison.
iv) Check HP website and confirm these are the latest drivers.
(1) http://h18006.www1.hp.com/storage/saninfrastructure/hba.html
(2) Select correct product and navigate to driver download.
(3) The HP website may present you with multiple driver version for similar
dates because it divides up driver downloads based on OS and HBA
speed, not exact designation. At the moment the only known way to make
sure you have the right driver is to find the matching HBA from the OEM
(ex. Emulex), extract the install files for the driver pack, read the
relnotes.rtf file and make sure your HBA is in the supported list.
(4) Compare against current set on software share by date.
(a) \software$\HBA_Prov\ 2 -- HBA Storport Drivers
v) Run the installer, accept defaults.
(1) In case of error extract executable, extract files from executable, then run
the installer from the unzipped files. Often this will provide you with
error messages, other methods may not.
vi) Do not add the drivers through Device Manager. This is not supported by HP.
b) Emulex Non-OEM,3rd party.
i) http://www.emulex.com/support/index.jsp
ii) Select your downloads based on Storage vendor. This is critical! They may
vary slightly depending on Storage vendor. In case of mixed environment,
consult Emulex.
c) QLogic HP Provided.
(1) http://h18006.www1.hp.com/storage/saninfrastructure/hba.html
(2) Select correct product and navigate to driver download.
(3) The HP website may present you with multiple driver version for similar
dates because it divides up driver downloads based on OS and HBA
speed, not exact designation. At the moment the only known way to make
sure you have the right driver is to find the matching HBA from the OEM
(ex. Emulex), extract the install files for the driver pack, read the
relnotes.rtf file and make sure your HBA is in the supported list.
(4) Compare against current set on software share by date.
(a) \software$\HBA_Prov\ 2 -- HBA Storport Drivers
ii) Run the installer, accept defaults.
(1) In case of error extract executable, extract files from executable, then run
the installer from the unzipped files. Often this will provide you with
error messages, other methods may not.
iii) Do not add the drivers through Device Manager. This is not supported by HP.
d) QLogic Non-OEM, 3rd part.
i) Check Qlogic website for instructions.
3) Install HBA controller software. This allows for troubleshooting of and updates
to the physical layer.
a) Most common current configuration
(1) HP Provided FC2142SR/A8002A 4GB Adapter.
(2) HBAnyware will need to be installed.
(3) \software$\ HBA_Prov\3 -- HBA Controller Software\Emulex\HP
OEM Installed\cp007453_02012007.zip
b) Emulex card -- HP Provided
i) Install HP HBAnyware package
(1) Software share: HBA Prov (Iain's Draft Guide)\2 -- HBA Controller
Software\Emulex\HP OEM Installed\cp006584.exe
(2) HP Website:
http://h20000.www2.hp.com/bizsupport/TechSupport/ProdSearch.jsp?
prod=Utility%20-%20FC%20HBA&lang=en&cc=us
c) Emulex card Non-OEM supplied (3rd party vendor)
i) Install Emulex HBAnyware package
(1) Software Share: HBA Prov (Iain's Draft Guide)\2 -- HBA Controller
Software\Emulex\Emulex or 3rd party Currently none.
(2) Emulex Website:
d) QLogic card HP Provided
i) Install HP Sansurfer package
(1) Software share:
(2) HP Website:
e) QLogic card Non-OEM supplied (3rd party vendor)
i) Install HP Sansurfer package
(1) Software share:
(2) QLogic Website:
f) Other Obtain specific documents from tech lead, PM, or create as needed.
4) Update HBA controller firmware.
a) Most common current configuration.
i) HP Provided FC2142SR/A8002A 4GB Adapter.
ii) \software$\HBA_Prov\4 -- HBA Firmware\HPA8002A\ wf270a5.zip
(1) Unzip files
(2) Open HBAnyware
(3) Select ONE HBA
(4) Select Firmware Tab
(5) Select Update Firmware
(6) Follow wizard, browse to unzipped files when indicated.
b) Use HBA Controller Software installed in step 2. Check helpfiles, usually
obvious.
c) Download latest firmware updates from HBA vendor, OEM does not usually
supply firmware updates.
5) Install/Update SAN Communication software. (Not available for all vendors.)
a) EMC Clariion devices.
i) Install Navisphere/Navicli
(1) \software$\HBA_Prov\5 -- Storage Device Client Software (not all
vendors)\Navisphere
(2) Accept defaults
ii) Install Navisphere/Naviagent
(1) Obtain IPs of Storage device from magma
(2) Privileged users will be. Add both IPs
(a) system@IP
(3) Otherwise accept defaults
(4) Note: Network connectivity to the SAN is a requirement over 4.34.x.x
network for this to function.
iii) If Navisphere Agent cannot be installed it will NOT break the SAN or the
client. Instead, the SAN is unable to automatically read and control approved
paths to the SAN from the client and more manual work will need to be done
by the SAN management team.
b) HP EVA
i) Unknown/None.
6) Install Multipath Software.
a) Common configuration: Shared SAN CVA (HP EVA 8000)
i) \software$\HBA_Prov\6 -- Multipathing and or DSM\EVA4-6-8000-32-
64-2003-MPIO\HPMpioFFEVAXLv2.01.01.zip
ii) Check for updates
(1) http://h20000.www2.hp.com/bizsupport/TechSupport/DriverDownloa
d.jsp?
prodNameId=421494&lang=en&cc=us&prodTypeId=18964&prodSer
iesId=421492&taskId=135
(2) HP MPIO Full Featured DSM (Device Specific Module) for EVA4000/6000/8000 Disk Arrays
iii) Do not install load balancing module for clustering.
iv) The DSM package will detect MPIO and if not present install it.
v) Do not install load balancing module for clustering.
vi) The DSM package will detect MPIO and if not present install it.
vii) Start Programs HP DSM DSM Cli
(1) This will open a command prompt.
(2) Hpdsm devices will list LUNs and the number of paths to them.
This will show no devices if nothing is zoned, nothing is allocated, or all
physical connection was lost and the server was subsequently rebooted
without it being restored
b) HP EVA Sole customer.
i) Obtain/Install latest Device Specific Module (DSM) package for your OS and
EVA. This may differ between 32 and 64 bit.
(1) HP Website:
ii) Do not install load balancing module for clustering.
iii) The DSM package will detect MPIO and if not present install it.
iv) Start Programs HP DSM DSM Cli
(1) This will open a command prompt.
(2) Hpdsm devices will list LUNs and the number of paths to them. This
will show no devices if nothing is zoned, nothing is allocated, or all
physical connection was lost and the server was subsequently rebooted
without it being restored.
c) EMC Powerpath
i) License required. Customer supplied.
(1) AON licenses maintained by Srinivas M. Reddy
ii) software$\HBA_Prov\6 -- Multipathing and or DSM\EMC Powerpath
iii) See latest numerical version for 32 or 64 bit OS. Clearly labeled.
iv) Additional versions and patches available on
(1) http://powerlink.emc.com
(2) Account is required.
(a) ibabeu@csc.com : CSC1ntel
v) Note: unless LUNs are assigned and all networking is done correctly between
the client server and the storage unit you will NOT see anything in multipath
software.
7) Additional Steps (may be conducted out of order).
a) Put in LUN and switch zoning request.
i) Open ticket with UNIX/Storage support team
ii) http://tshare1.ops.cscehub.com/Unix/cgi-bin/san-req.cgi
iii) Submit request, copy and paste details into ticket.
8) Enable and format disks.
a) ABSOLIUTELY CRITICAL STEP!
i) Make sure any LUN assigned to multiple systems is clustered (or otherwise
prevented from coming online in both places using such features as LUN
masking).
b) ABSOLUTELY CRITICAL STEP! Do not forget to assign offset on new
partition.
i) Cmd prompt
ii) Diskpart
iii) List disk (pulls up a list of disks
iv) Select disk #
v) Set primary align=64
vi) To check alignment on already formatted disk
vii) Cmd prompt
viii) Diskpart
ix) List Disk
x) Select disk
xi) List partition (check offset column)
Replacing an HBA and Troubleshooting Hardware
Connections to the Network
1) Confirm it is a bad HBA and nothing else
a) Using HBAnyware, SANSurfer, or another vendors tool for looking at HBAs
directly check to see connection status for that HBA. If the HBA shows as
connected but is not available in multi-path software this means the HBA is
working and connecting, but settings are incorrect on the SAN or Storage Server.
b) Have DC check the cable, sometimes the Input/Output ports are reversed
c) Change switch ports
i) The HBA plugs into what is called a GBIC (it does optical to electrical
conversions for the switch) and it often goes bad.
d) Confirm zoning on SAN switch.
e) Check SAN for LUN binding.
f) Check SAN for paths being activated
i) Requires engineering mode password
ii) Not needed when using Navisphere agent.
2) Preparation work
a) Zone the new HBA
i) Have the DC check for the IEEE number on the new HBA box or the card
itself.. This is translated into a WWWN and WWPN addresses (think of them
as MAC addresses). If the DC cannot find the new WWWN you may obtain
it after the card has been installed by checking HBAnyware, SANSurfer, or
Systems Management Homepage.
(1) 20:00 + IEEE# = WWPN
(2) 10:00 + IEEE# = WWWN
ii) Give the IEEE number to the SAN and switch administrators
(1) Make sure it is zoned onto the right switch. In many cases switches are in
individual fabrics (a logical collection of switches) to create redundancy.
The new card should be zoned on the same switch as the old one.
(2) SAN administrator will need to bind the IEEE/WWWN number to the
storage group for the server.
(a) Check for procedure details
b) If this is an HBA upgrade copy all required drivers and patches to the server.
c) Always download drivers from the section of the HBA providers websites
devoted to the correct SAN provider (e.g. on the Emulex driver download section
go to the EMC section, NOT Windows).
3) Physical work
a) It may be possible to do a Hot Add on some systems but please be aware that
HBAnyware will freeze if this is done and will need to be restarted. Other
software may need to be restarted. A hot add is not currently recommended.
b) Replace the HBA, preferably using the same slot.
i) The replacement HBA will NOT show up in powerpath or other multipath
software.
ii) The replacement HBA MAY show up in HBAAnyware
iii) The replacement HBA MAY show up in HBAAnyware but NOT bound to any
storage groups. Meaning no LUNs appear under that HBA in the software.
(1) In this event contact the SAN administrator to activate the paths
(a) This requires the engineering password
(b) In Navisphere got to Storage Groups Properties Hosts
Advanced Check all appropriate paths.
iv) At this point the HBA should show up in PowerPath properly. If not further
troubleshooting may be required.

The Basics of Storage Area Networks


A Storage Area Network, or SAN, is a specialized type of network designed to support
traffic to external storage devices. These networks are designed to be high bandwidth,
low latency, and redundant as the provide the ability for a server to use a remote group of
disks as if they were a local high speed device.

A SAN consists of several components in order to achieve the effect of allowing remote,
high speed disks. On the server side a Host Bus Adapter (HBA), correct system drivers,
correct device drivers, and optional software to manage redundant connections to the
same disk resources (multipath software). The HBA can be thought of as a special
network card. It has a unique hardware address that is used to configure networking on
the rest of the SAN and often uses fiber optic cables. Once past the server portion there
is cabling (obvious), data switches for networking, and final storage devices themselves.

From the server perspective both system level drivers and device level drivers are
required. Much like SCSI, the system needs system level drivers to understand the
general control language of the disks. Current standard is Storport, these are Windows
system level drivers using a language native to Fibre Channel. Other driver types
actually translate between SCSI and Fibre commands, a less than optimal approach.
Individual device drivers are required to run the actual Host Bus Adapters, just as device
drivers are required for any other hardware device.

HBA cards also come with additional management software. The most common
examples are HBAnyware (Emulex), and SAN Surfer (QLogic). This software provides
an interface to check the health status of the HBA(s), update firmware, check connection
status, alter network speed, or check other HBA driver specific settings. It is here you
would check to see if an HBA believes it is connected or disconnected from the network,
not within multipath software.

The HBA is used to connect with the Storage Area Network and requires a physical
cabling media to get this done. As fiber optic cabling is a common standard these cables
may be moderately expensive and somewhat fragile. They often have their own patch
panel infrastructure. It is possible for their to be copper interconnects, but none are
currently used within MHS. Fiber optic cables distinguish between input and output
ports, they must be matched correctly between the switch and the HBA.
Next in the network is the Storage Switch, or Data Switch. These provide the logic and
physical routing of signals between individual ports on the switches themselves and the
correct storage devices. Configuration is either entirely or semi-manual and the hardware
address for any HBA must be added to the switch and set correctly before data can pass
to the correct places. Of special note is what is called the GBIC. Each port on a switch
must translate signals from optical, to electrical, pass them to the switching gear, and
translate back to optical on the other end. Current optical switches are not practical
technology for datacenters, all switching is done on electrical signals. GBICs are prone
to failure.

Data switches provide switching and network access to any Storage Devices or Servers
on the network. A SAN may have multiple Storage devices present. Storage devices are
large collections of physical disks which are then shared out to multiple clients as
Logical Unit Numbers. A Storage administrator takes a bunch of physical disks,
determines desired RAID, etc. and then presents the logical disks to the clients. Each
LUN number is unique on the Storage device, but the device may present a LUN as a
different number to the client than it shows internally. So LUN 463 on the Storage device
may be LUN 3 on the target client(s). Storage devices are designed to have redundant
connections (sometimes called Storage Processors) and connect to redundant or multiple
networks,

In a redundant setup Data Switches are setup in groups and those groups do not talk to
the redundant group. You have, in essence, two separate networks that do not talk to each
other. This is due to the way error propagation is done in data networks. Instead,
redundant HBAs talk to both networks and can communicate with the storage device over
multiple, redundant interfaces. In order to avoid conflicts, multipath software is used.

Microsoft has garnered loathing and respect for the way they decided to handle multipath
software. Until recently, multipath drivers were provided by multiple storage vendors.
This approach could lead to conflicts if one server was to speak with multiple storage
devices from different vendors. Microsoft chose to release Multipath IO drivers (MPIO)
in order to deal with this problem. Simply put, multipath is one driver and all storage
devices must go through that driver, this avoids multiple multipath drivers being required.
Any functionality specific to a storage device is then handled in a Device Specific
Module or DSM. Some vendors view this as an attempt to lock vendors into an MS
solution, but these also tend to be the vendors who sold multipath software for large
amounts of money.

Multipath software should really be required for any server with an HBA, even if there is
only one HBA. In the event of a problem on the storage device a machine with a single
HBA will fail, even if the storage device handles the fault in an expected and graceful
manner. Although a reboot may fix the issue, intervention by the storage team may be
required to re-establish the link and access to the disks.
Glossary
SAN Storage Area Network.

A specialized network designed to carry data traffic between


servers and attached tape devices or disk storage arrays. This is
disk level traffic and requires a very high performance.

Also used to refer to the device that houses all the disks, but that
is technically inaccurate.
Storage Device or The physical device housing physical disks that is responsible
Storage Server for handling RAID, multipathing, logical disks, etc. Ex.: EMC
(sometimes called Clariion CX600, HP EVA 8000
SAN, see above)
LUN Logical Unit Number A way of describing and identifying a
logical disk present by a Storage Device to a client server.
Data Switch Much like a network switch, but specialized for SAN traffic.
Ex. Brocade.
Switched Fabric A SAN configuration where there are data switches (much like
network switches) that allow multiple clients to access multiple
storage devices. This is used when there are more than 2 clients
connecting to a single storage device.
Point-to-Point or Fibre A method of connecting servers to a Fiber based external
Channel Arbitrated storage device that does NOT rely on a storage network and
Loop does not require a Data Switch. Instead each server is
connected directly. Limited to 2 servers per storage device.
HBA Host Bus Adapter. Usually a Fibre Card. It is a PCI card that
sits between the host computer and a storage area network and
handles the physical traffic.
Fibre Card Its called a fibre card instead of a fiber card because there
are copper based transport mechanisms in addition to optical
ones. This is the most common Host Bus Adapter type.
GBIC Gigabit Interface Converter. Converts signals from optical to
electrical and back again. Also allows data switches to serve
both optical and copper based HBAs as the GBIC is usually a
module that can be swapped out.
Storport Drivers Current standard. Microsoft written drivers that are
specifically designed to support SANs.
SCSIPort Drivers A driver that translates between SCSI commands and SAN
commands on the fabric. Being discontinued.
FC Port Drivers For point-to-point networks, being discontinued.
PowerPath An EMC product that does multipathing. This technique
allows the host operating system to seamlessly communicate
with a storage device over multiple HBAs and switches
simultaneously for increased performance and
Navisphere Refers to the web based management software EMC uses to
configure and monitor SANs.
Navicli Refers to the command line management software EMC uses to
configure and monitor SANs. Note: requires network
connection.
WWPN world-wide port name Used by the SAN as a physical address
to bind an HBA to various switch ports and specify which
LUNs of a storage device a client with an HBA should have
access to.

Ex. 10:00:00:00:c9:42:c5:9d
WWN World Wide Name Similar to a WWPN but refers to a slightly
different function. Identical to the WWPN, merely varying in
the first two octets.

Ex. 20:00:00:00:c9:42:c5:9d