[Brocade]SAN Troubleshooting(11) - Common SAN Issue Part5 - Mar...

http://windowspeople.com/brocade/brocade-san-troubleshooting-11-...

[Brocade]SAN Troubleshooting(11) - Common

SAN Issue Part5 - Marginal Links
Marginal Links caused by physical connectivity includes physically hardware issue, transmissin of light/signal,
negotiation of speed and synchronization of Transmission characters and words.

Physical Components
Marginal Links involves the connection between the switch and the edge device. Isolating the exact cause of a marginal
link involves analyzing and testing many of the components that make up the link.
1. SwitchShow

Use switch to confirm switch is online, SFP installed.

2. portErrShow
check portErrShow to see if there is high number of errors on CRC or ENC_OUT, You should always reset(clear) port
counters to assist the problem determination:
Clear port counters using portStatsClear. you can now clear the stats by slot, if Error counters increasing in a short period
of time, could indicate issue.
To clear hardware statistics for all ports on slots 3-5:
switch:admin> portstatsclear -s 3-5
To clear hardware statistics for all ports on slots 3-5 and 7-10:
switch:admin> portstatsclear -s 3-5 7-10
here is sample portErrShow output,
DCX01:admin> porterrshow
frames
tx

rx

enc
in

crc
err

crc

too

too

g_eof

shrt

long

bad
eof

enc

disc

out

c3

link
fail

loss

loss

sync

sig

frjt

fbsy

=========================================================================================================
0:

336.0m

1.8g

1:

898.0m 116.9m

2:

38.6m

60.2m

3:

4:

5:

6:

(output truncated)

porterrshow can be divided in 3 sections

Section1 - Encoding Statistics

enc_in - Encoding errors inside frames
1 of 8

02-01-2017 14:02

[Brocade]SAN Troubleshooting(11) - Common SAN Issue Part5 - Mar...

http://windowspeople.com/brocade/brocade-san-troubleshooting-11-...

- Incremented when 8b/10b encoding errors are detected within a frame (2112 transmission characters)
- Detected on the ingress port, it means that when this counter gets incremented, it points to the traffic coming from the
remote end
- Traffic is only impacted when enc_in gets incremented, it means that a systems is trying to send frames through a
channel with errors. The crc_err counter are frames with CRC errors. If this counter goes up, then the physical path
should be inspected. Check the cables to and from the switch, patch panel, and other devices. Check the SFP by
swapping it with a known good working SFP.
- The enc_out are errors that occur outside the frame and usually indicating a bad primitive. Incremented during speed
negociation and link transitions prior to login. Locking a port to the speed supported by the end device can be used to
isolate enc_out issues. If you notice the crc_err counter go up, you have a bad or damaged cable, or a bad or damaged
device in the path.
Section2 - Frame Statistics
- crc_err means frames with CRC errors which Indicates corruption of byte(s) within the data frame. Because switch is
cut-through, CRC is checked and validated as frames leaves the switch. Due to this, CRC errors from a single device
maybe seen on multiple ports in a fabric.
To find the source of CRC errors, look at the crc_err and crc_g_eof counters:
a). The crc_err counter represents frames with CRC errors. This simply means that a port received a frame with a bad
CRC. It does not, by itself, indicate that the link to that port was the source of the problem. In isolation, this usually
indicates that a problem exists upstream of that link.
b).The crc_g_eof counter is used to locate the source of CRC problems. It represents frames with CRC errors and a
good EOF marker. This means that it was the first port detecting the CRC error. It will mark the frame with a bad EOF
and pass the frame on to its destination, so that the destination can proactively request retransmission. Subsequent ports
in the path will also detect the CRC error, so the crc_err counter will increment on these ports. However, since the first
port marked the frame with a bad EOF, crc_g_eof on the subsequent ports will not increment. The link associated with the
port with an increasing crc_g_eof counter is the marginal link and the source of the errors. Look for marginal SFPs on
each side of the link and problems in the optical pathway between the SFPs. If these are all verified to be good, the
problem could be a failing switch port.
- The disc_c3 errors are discarded class 3 errors, which means that the switch is holding onto the frame longer than the
hold time allows. below situation could cause disc_c3 error,
a).it is not routable to a destination address
b).Destination ID (DID) is not online
c).Timeout exceeded in the ASIC (500 ms)
d).Also could caused by FC node or Switch port status change(online/offline)continuously. further investigate with
fabriclog -s output
Note:One of the common cause is related to is ISL over-subscription.
Section3 - Signal/Sync/Link Statistics
loss_sig: Loss of signal, indicate electrical signal is lost on a link.
loss_sync: Loss of synchronization, indicate a bit/word synchronization on a link is lost.
frjt: Frames rejected with F_RJT
fbsy: Frames rejected with F_BSY
link_fail: Link failures (LF1 or LF2 states)
2 of 8

02-01-2017 14:02

[Brocade]SAN Troubleshooting(11) - Common SAN Issue Part5 - Mar...

http://windowspeople.com/brocade/brocade-san-troubleshooting-11-...

If a Port remains in the LR Receive State for a period of time greater than R_T_TOV (Receiver-Transmitter Timeout Value,
100ms per default), a Link Reset Protocol Timeout will be detected that results in a Link Failure condition and the port
enters the NOS (Not Operational State) Transmit State.
The link failure also indicates that loss of signal or loss of sync lasting longer than the R_T_TOV value was detected while
not in the Offline state. As for loss_sync, verify that the link was not brought down manually after setting the baseline.
Link Failure state consists of:
LF1 => No operational Receive
LF2 => No operational Transmit
To conclude, if found enc_in and/or crc_err, it means a possible bad media (SFP, cable, path panel). Loss of Sync, Loss of
Signal and Enc_out are errors expected every time a user brings a port up or down (reboots, power cycles,
disconnections, portdisable/portenable).
3. portflagsshow
Use portflagsshow to check if LED is seen

SAN01:FID128:admin> portflagsshow
Slot Port SNMP

Physical

Flags

-----------------------------------1

0 Online

In_Sync

PRESENT ACTIVE F_PORT G_PORT U_PORT LOGICAL_ONLINE LOGIN NOELP LED ACCEPT FLOGI

1 Online

In_Sync

PRESENT ACTIVE F_PORT G_PORT U_PORT LOGICAL_ONLINE LOGIN NOELP LED ACCEPT FLOGI

2 Online

In_Sync

PRESENT ACTIVE F_PORT G_PORT U_PORT LOGICAL_ONLINE LOGIN NOELP LED ACCEPT

3 Online

In_Sync

PRESENT ACTIVE F_PORT G_PORT U_PORT LOGICAL_ONLINE LOGIN NOELP LED ACCEPT

4 Online

In_Sync

PRESENT ACTIVE F_PORT G_PORT U_PORT LOGICAL_ONLINE LOGIN NOELP LED ACCEPT FLOGI

5 Online

In_Sync

PRESENT ACTIVE F_PORT G_PORT U_PORT LOGICAL_ONLINE LOGIN NOELP LED ACCEPT FLOGI

4. sfpshow
Use sfpshow to check
- wavelength and speed,
- also verify the SFP is not faulty
- and Tx/Rx Power is inside recommended values according to SFP datasheets. make sure Optical Modules (GBIC/SFP
/SFP+/XFP) must match cables.
Note that If the switch running on FOS7.x, you can also use sfpshow -health.
sfpshow all
sfpshow <port>

3 of 8

02-01-2017 14:02

[Brocade]SAN Troubleshooting(11) - Common SAN Issue Part5 - Mar...

http://windowspeople.com/brocade/brocade-san-troubleshooting-11-...

SAN01:FID128:admin> sfpshow

Slot 1/Port 0: id (sw) Vendor: BROCADE Serial No: UYA109491147982 Speed: 200,400,800_MB/s

Slot 1/Port 1: id (sw) Vendor: BROCADE Serial No: UYA109491157932 Speed: 200,400,800_MB/s
Slot 1/Port 2: id (sw) Vendor: BROCADE Serial No: UYA109501179322 Speed: 200,400,800_MB/s
Slot 1/Port 3: id (sw) Vendor: BROCADE Serial No: UYA109531208472 Speed: 200,400,800_MB/s
Slot 1/Port 4: id (sw) Vendor: BROCADE Serial No: UYA109531213772 Speed: 200,400,800_MB/s
Slot 1/Port 5: id (sw) Vendor: BROCADE Serial No: UYA109521204452 Speed: 200,400,800_MB/s
SAN01:FID128:admin> sfpshow 1/1
Identifier:

SFP

Connector:

LC

Transceiver: 540c402000000000 200,400,800_MB/s M5,M6 sw Inter_dist

Encoding:

8B10B

Baud Rate:

85

(units 100 megabaud)

Length 9u:

(units km)

Length 9u:

(units 100 meters)

Length 50u:

(units 10 meters)

Length 62.5u:3

(units 10 meters)

Length Cu:

(units 1 meter)

Vendor Name: BROCADE

Vendor OUI:

00:05:33

Vendor PN:

57-1000046-01

Vendor Rev:

Wavelength:

850

Options:

001a Loss_of_Sig,Tx_Fault,Tx_Disable

BR Max:

BR Min:

Serial No:

UYA109491157932

Date Code:

091203

DD Type:

0x68

(units nm)

Enh Options: 0xf0

Status/Ctrl: 0x90
Alarm flags[0,1] = 0x0, 0x0
Warn Flags[0,1] = 0x0, 0x0
Alarm
low

Warn
high

low

high

Temperature: 35

Centigrade

-15

90

-10

85

Current:

6.832

mAmps

2.000

8.500

2.000

8.500

Voltage:

3294.2

mVolts

2800.0

3800.0

2970.0

3630.0

RX Power:

-3.0

dBm (506.0 uW) 0.0

uW

6550.0 uW

49.0

uW

1100.0 uW

TX Power:

-2.6

dBm (546.7 uW) 50.0

uW

800.0

100.0

uW

700.0

uW

uW

SAN01:FID128:admin>

4 of 8

02-01-2017 14:02

[Brocade]SAN Troubleshooting(11) - Common SAN Issue Part5 - Mar...

http://windowspeople.com/brocade/brocade-san-troubleshooting-11-...

Port Speed & Synchronization

Brocade best practice for the port speed configuration:
- Auto Negotiate for Initiators, Targets and local ISLs
- Manually set port speeds for long distance ISLs
- Targets port speeds must be manually set to their own speed from the device point of view.
- If Targets port speed couldn't set to manual, It should be enforced at the switch port.

Port Parameters
use portcfgshow to verify all port parameters . See below sample of output

5 of 8

02-01-2017 14:02

[Brocade]SAN Troubleshooting(11) - Common SAN Issue Part5 - Mar...

http://windowspeople.com/brocade/brocade-san-troubleshooting-11-...

SAN01:FID128:admin> portcfgshow
.., output truncated,
Ports of Slot 4

10

11

12

13

14

15

-----------------+---+---+---+---+-----+---+---+---+-----+---+---+---+-----+---+---+--Speed

AN

AN

AN

AN

AN

AN

AN

AN

AN

AN

AN

AN

AN

AN

AN

AN

Fill Word

AL_PA Offset 13

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

Trunk Port

ON

ON

ON

ON

ON

ON

ON

ON

ON

ON

ON

ON

ON

ON

ON

ON

Long Distance

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

VC Link Init

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

Locked L_Port

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

Locked G_Port

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

Disabled E_Port

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

Locked E_Port

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

ISL R_RDY Mode

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

RSCN Suppressed

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

Persistent Disable ..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

LOS TOV enable

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

NPIV capability

ON

ON

ON

ON

ON

ON

ON

ON

ON

ON

ON

ON

ON

ON

ON

ON

NPIV PP Limit

126 126 126 126

126 126 126 126

126 126 126 126

126 126 126 126

QOS E_Port

AE

AE

AE

AE

AE

AE

AE

AE

AE

AE

AE

AE

..

..

..

..

EX Port

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

Mirror Port

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

Rate Limit

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

Credit Recovery

ON

ON

ON

ON

ON

ON

ON

ON

ON

ON

ON

ON

ON

ON

ON

ON

Fport Buffers

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

Port Auto Disable

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

CSCTL mode

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

Fault Delay

Ports of Slot 4

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

-----------------+---+---+---+---+-----+---+---+---+-----+---+---+---+-----+---+---+--Fill Word

AL_PA Offset 13

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

Trunk Port

ON

ON

ON

ON

ON

ON

ON

ON

ON

ON

ON

ON

ON

ON

ON

ON

Long Distance

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

VC Link Init

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

Locked L_Port

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

Locked G_Port

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

Disabled E_Port

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

Locked E_Port

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

ISL R_RDY Mode

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

RSCN Suppressed

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

6 of 8

02-01-2017 14:02

[Brocade]SAN Troubleshooting(11) - Common SAN Issue Part5 - Mar...

http://windowspeople.com/brocade/brocade-san-troubleshooting-11-...

Persistent Disable ..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

LOS TOV enable

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

NPIV capability

ON

ON

ON

ON

ON

ON

ON

ON

ON

ON

ON

ON

ON

ON

ON

ON

NPIV PP Limit

126 126 126 126

126 126 126 126

126 126 126 126

126 126 126 126

QOS E_Port

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

EX Port

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

Fport Buffers

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

Port Auto Disable

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

..

Fault Delay

where AE:QoSAutoEnable, AN:AutoNegotiate, ..:OFF, NA:NotApplicable,

??:INVALID,
LM:L0.5
.., output truncated,

See below spreadsheet on the command to use to make changes to the port parameters:
Command

Comments

portcfgspeed

Speed Displays AN for auto speed nego a on mode, or a specic speed of 1, 2, 4, or 8

Parameter
Port Speed

Gbits/sec. This value is set by the portcfgspeed command.

Reset to Defaults

portcfgdefault

Port Type (L_Port)

portcfglport

Locked L_Port Displays ON when the port is locked to L_Port only. Displays (..) or OFF
when L_Port lock mode is disabled and the port behaves as a U_Port). This value is set
by the portcfglport command.

Port Type (E_Port

portcfgeport

disabled)

Disabled E_Port Displays ON when the port is not allowed to be an E_Port.

Displays (..) or OFF when the port is allowed to func on as an E_Port. This value is set
by the portcfgeport command.

Port Type (E or

portcfggport

F_Port only)

Locked G Port Displays ON when the port is locked to G Port only Displays ( )
TS300 End Device Connec vity G_ G_only. (..) or OFF when G_Port lock mode is
disabled and the port behaves as a U_Port. This value is set by the portcfggport
command.

Long Distance and

VC Link Init

7 of 8

portcfglongdistance

Long Distance Displays (..) or OFF when long distance mode is o; otherwise,
displays long distance levels as shown below. This value is set by the
portcfglongdistance command.
LE The link is up to 10 km
LD The distance is determined dynamically
LS The distance is determined sta cally by user input

02-01-2017 14:02

[Brocade]SAN Troubleshooting(11) - Common SAN Issue Part5 - Mar...

ISL R_RDY Mode

portcfgislmode

http://windowspeople.com/brocade/brocade-san-troubleshooting-11-...

ISL R_RDY Mode Displays ON when ISL R_RDY mode is enabled on the port.
Displays (..) or OFF when ISL R_RDY mode is disabled. This value is set by the
portcfgislmode command.

QoS

portcfgqos

QOS E_Port Displays ON when Quality of Service (QoS) is enabled on the port.
Displays (..) or OFF when QoS is disabled. By default, QoS is enabled by best eort based
on availability of buers. This value is set by the portcfgqos command.

Credit Recovery

8 of 8

portcfgcreditrecovery

Credit Recovery Displays ON when Credit Recovery is enabled on the port or (..) or
OFF when disabled. This value is set by the portcfgcreditrecovery
command. The credit recovery feature is enabled by default, but only ports
congured as long distance ports can u lize this feature.

02-01-2017 14:02