BadBoy:MDC:/root> oslevel -s 5300-08-08-0943 BadBoy:MDC:/root> lslpp -l | grep -i sddpcm devices.sddpcm.53.rte 2.5.0.

0 COMMITTED IBM SDD PCM for AIX V53 devices.sddpcm.53.rte 2.5.0.0 COMMITTED IBM SDD PCM for AIX V53 chaapmpu001:MDC:/root> pcmpath query version SDDPCM VERSION 2.5.0.0 (devices.sddpcm.53.rte) BadBoy:MDC:/root> Active/Passive? Active/Active? What? Just licking the surface Active/Passive paths management method allows the operating system to detect what SCSI targets were really different names of the same physical device allowing host to detect a failed path and to select another one to failover to. By the way, this is the earliest method of paths management. The weak point of this method is the apparent waste of paths only one can be the active path at a time! Active/Active paths management software allows data to flow simultaneously through all available paths! Be aware that each category, as always, has several subcategories. Excluding rootvg disks, each node has the following ones in its inventory. BadBoy:MDC:/root> lsdev -Cc disk . hdisk2 Available 06-00-02 MPIO FC 2145 hdisk3 Available 06-00-02 MPIO FC 2145 hdisk4 Available 06-00-02 MPIO FC 2145 hdisk5 Available 06-00-02 MPIO FC 2145 hdisk6 Available 06-00-02 MPIO FC 2145 hdisk7 Available 02-00-02 MPIO FC 2145 hdisk8 Available 02-00-02 MPIO FC 2145 hdisk9 Available 03-08-02 IBM MPIO DS4500 Array Disk hdisk10 Available 03-08-02 IBM MPIO DS4500 Array Disk hdisk11 Available 03-08-02 IBM MPIO DS4500 Array Disk hdisk12 Available 03-08-02 IBM MPIO DS4500 Array Disk hdisk13 Available 03-08-02 IBM MPIO DS4500 Array Disk hdisk14 Available 03-08-02 IBM MPIO DS4500 Array Disk hdisk15 Available 03-08-02 IBM MPIO DS4500 Array Disk hdisk16 Available 03-08-02 IBM MPIO DS4500 Array Disk Hdisk2 through hdisk8 are delivered from our SVC (dev.code 2145). Hdisk9 throughhdisk16 are delivered from one of our DS4500s. Look again at the output above and notice that it shows relationships between a single FC adapter and a single disk. Funny, isnt it? This is despite the fact that two FC adapters provide access to the SVC storage and another pair of FC adapters connects DS4500 storage. When cross examined with other commands (pcmpath/lspath) it looks that lsdev shows disks attached to the preferred controller. I wonder if this is really true? Till morning, hdisk9 through hdisk16 had four paths to connect to their respective LUNs. Overnight, SAN administrator re-cable a SAN switch (planned event) and as the result one path from each DS4500 disks was removed from each of its two FC adapters. Notice the differences/similarities in output of the lspath and the datapath query devicecommands. Notice also the new Dual Active and Active/Asymmetric andActive/Passive headers bellow. BadBoy:MDC:/root> pcmpath query device Total Dual Active and Active/Asymmetrc Devices : 7 DEV#:2 DEVICE NAME: hdisk2 TYPE: 2145 ALGORITHM: Load Balance SERIAL: 60050768018100314800000000000397 ================================================================ Path# Adapter/Path Name State Mode Select Errors 0 fscsi1/path2 OPEN NORMAL 131357 0 1* fscsi1/path3 OPEN NORMAL 106 0 2 fscsi0/path0 OPEN NORMAL 131991 0 3* fscsi0/path1 OPEN NORMAL 106 0 .......................

....................... DEV#:8 DEVICE NAME: hdisk8 TYPE: 2145 ALGORITHM: Load Balance SERIAL: 6005076801830037C00000000000042F ================================================================ Path# Adapter/Path Name State Mode Select Errors 0 fscsi0/path0 OPEN NORMAL 506 0 1* fscsi0/path1 OPEN NORMAL 8 0 2 fscsi1/path2 OPEN NORMAL 485 0 3* fscsi1/path3 OPEN NORMAL 8 0 Total Active/Passive Devices : 8 DEV#:9 DEVICE NAME: hdisk9 TYPE: 1742-900 ALGORITHM: Load Balance SERIAL: 600A0B800017BF6100000000441FB9B5 =============================================================== Path# Adapter/Path Name State Mode Select Errors 0* fscsi2/path0 OPEN NORMAL 8208724 0 1* fscsi2/path1 FAILED NORMAL 8210579 0 2 fscsi4/path2 OPEN NORMAL 78732380 0 3 fscsi4/path3 FAILED NORMAL 74943946 0 ....................... ....................... DEV#:16 DEVICE NAME: hdisk16 TYPE: 1742-900 ALGORITHM: Load Balance SERIAL: 600A0B800017B78D00003F264CF8D0B8 ================================================================ Path# Adapter/Path Name State Mode Select Errors 0 fscsi2/path0 OPEN NORMAL 127496 0 1 fscsi2/path1 FAILED NORMAL 135190 0 2* fscsi4/path2 OPEN NORMAL 4899 0 3* fscsi4/path3 FAILED NORMAL 4289 0 BadBoy:MDC:/root> The character * above denotes the nonferred path. Earlier versions of PCM or SDD are not provisioned to show this marker. The MPIO way to list the state of disks paths follows bellow. BadBoy:RDC:/root> lspath ................................... ................................... Enabled hdisk8 fscsi0 Enabled hdisk8 fscsi1 Enabled hdisk8 fscsi1 Failed hdisk9 fscsi2 Failed hdisk10 fscsi2 Failed hdisk11 fscsi2 Failed hdisk12 fscsi2 Enabled hdisk9 fscsi2 Enabled hdisk10 fscsi2 Enabled hdisk11 fscsi2 Enabled hdisk12 fscsi2 Failed hdisk13 fscsi2 Failed hdisk14 fscsi2 The paths to DS4500 disks need cleaning the failed paths need to be removed. Since I have done it many times in the past I used the same steps as always. Since sddpcm code is loaded I executed the pcmpath query adapter command (in the sdd case, I would executedatapath query adapter) to learn the numerals assigned to the host FC adapters by thepcmdrv driver. Look bellow, what you see is 100% correct. This is not a copy/paste mistake on my part and it does make sense. BadBoy:MDC:/root> pcmpath query adapter Total Dual Active and Active/Asymmetrc Adapters : 2 Adpt# Name State Mode Select Errors Paths 0 fscsi1 NORMAL ACTIVE 1622701 0 14 1 fscsi0 NORMAL ACTIVE 1621135 0 14 Active 14 14

Total Active/Passive Adapters : 2 Adpt# Name State Mode Select Errors 0 fscsi2 NORMAL ACTIVE 137879 0 1 fscsi4 NORMAL ACTIVE 8 0 BadBoy:MDC:/root> Paths 8 8 Active 8 8

]Notice, that the listing is broken between two different groups based on the storage array type. What? The term Asymmetrc is most likely a result of IBMs outsourcing software development overseas. I bet it should read Asymmetric. OK, what is important here is the following: each group has its own two adapters 0 and 1. There are no adapters 0, 1, 2, 3. This is the part that made me gasp for air, this is whats new to me and what I have not expected to see I got served. Shortly later, I recognize that I have been in this situation before (SDDPCM and MISSING LUN PATHS) there are two ways to remove the nonexistent paths a disk by disk or all at once. Before proceeding, let demonstrate a few ways to identify the paths leading from hdisk to its LUN from PCM and MPIO points of view. PCM has two options. The long one being pcmpath query device and the short one calledlspcmcfg. The difference is obvious. The second one produces a simplified version of the first one. Missing from its output are paths identifiers and fscsi#. The output of the first command (pcmpath has been shown earlier in this post. Bellow is the output of the second command. FatBoy:RDC:/root> lspcmcfg .................................................................... hdisk9 (Avail pv chop3_vg) 600A0B800017BF6100000000441FB9B5 = path0 (Enabled) (Enabled) hdisk10 (Avail pv chop3_vg) 600A0B800017BF6100000003448C5DA5 = path0 (Failed) (Enabled) path2 (Enabled) path3 (Failed) hdisk11 (Avail pv chop3_vg) 600A0B8000213DE6000041294CF8D471 = path0 (Failed) (Enabled) path2 (Enabled) path3 (Failed) hdisk12 (Avail pv chop3_vg) 600A0B800017BE3E00003F464CF8D2A0 = path0 (Failed) (Enabled) path2 (Enabled) path3 (Failed) hdisk13 (Avail pv chop3_vg) 600A0B800017B78D00000000441FBB58 = path0 (Failed) (Enabled) path2 (Enabled) path3 (Failed) hdisk14 (Avail pv chop3_vg) 600A0B800017B7DE00000001448C6197 = path0 (Failed) (Enabled) path2 (Enabled) path3 (Failed) hdisk15 (Avail pv chop3_vg) 600A0B800017B7DE000041024CF8E40B = path0 (Failed) (Enabled) path2 (Enabled) path3 (Failed) hdisk16 (Avail pv chop3_vg) 600A0B800017B78D00003F264CF8D0B8 = path0 (Failed) (Enabled) path2 (Enabled) path3 (Failed)

path1 path1 path1 path1 path1 path1 path1 path1

To generate the MPIO point of view for the identified disks, enter the following on the command line adjust accordingly to your own case). Note that the output has been shortened to increase its clarity. BadBoy:RDC:/root> for i in 9 10 11 12 13 14 15 16 do lspath -l hdisk$i -H -F"name parent path_id connection done name parent path_id connection name parent path_id connection hdisk10 hdisk10 hdisk10 hdisk10 name fscsi2 fscsi2 fscsi4 fscsi4 0 1 2 3 200300a0b817be3f,1000000000000 200300a0b817be40,1000000000000 200200a0b817be3f,1000000000000 200200a0b817be40,1000000000000 status status Failed Enabled Enabled Failed status

parent path_id connection

hdisk11 hdisk11 hdisk11 hdisk11

fscsi2 fscsi2 fscsi4 fscsi4

0 1 2 3

200300a0b817be3f,2000000000000 200300a0b817be40,2000000000000 200200a0b817be3f,2000000000000 200200a0b817be40,2000000000000

Failed Enabled Enabled Failed

name parent path_id connection status name parent path_id connection status hdisk16 fscsi2 0 200300a0b817b78e,3000000000000 Failed hdisk16 fscsi2 1 200300a0b817b78f,3000000000000 Enabled hdisk16 fscsi4 2 200200a0b817b78e,3000000000000 Enabled hdisk16 fscsi4 3 200200a0b817b78f,3000000000000 Failed In both cases, it is easy to establish the pattern. Each hdisk has one missing path per its adapter two in total. The disks use fscsi2 and fscsi4 which are the logical adapters representing physical adapters fcs2 and fcs4. The details of both types may be produced with the lsattr El fscsi# and lscfg vl fcs# respectively. The first solution disk by disk from a given adapter: for i in 9 10 11 12 13 14 15 16 >do > do > rmpath -l hdisk$i -p fscsi2 -d > done paths Deleted paths Deleted paths Deleted paths Deleted paths Deleted paths Deleted paths Deleted paths Deleted cfgmgr l fcs2 The second solution simultaneously removing all paths from all disks attached to a given adapter : rmpath p fscsi2 d cfgmgr l fscs2 Do not proceed to the next adapter without checking/verifying that the paths have been reactivated over the just processed adapter.