Nexus Technology Labs

Nexus Technology Labs - Fabric Extenders
(FEX)
FEX Active Standby
Task
Configure N5K1 to pair with the Fabric Extender N2K1 as follows:
Enable the Fabric Extender feature.
Configure N5K1's link connecting to N2K1 as a FEX port.
N2K1 should be module number 101.
Configure N5K2's link connecting to N2K2 as a FEX port.
Configure the links between N5K1 & N5K2 as 802.1q trunk links.
Configure N5K1's links to Server 1 and the Emulex CNA Server in VLAN 10.
Configure Server 1 with the IP address 10.0.0.1/24 on this link.
Configure the Emulex CNA Server to do Active Standby NIC teaming as follows:
Use the IP address 10.0.0.10/24 for the NIC Team.
Use the link to N2K1 as the primary active path and the link to N2K2 as the
secondary standby path.
Verify that both Server 1 and Server 2 have connectivity to the Emulex CNA Server,
and that traffic to the server is flowing only through N2K1.
Disable the FEX port from N5K1 to N2K1, and verify that connectivity to the CNA
Server is maintained by using the backup path through N2K2.
Configuration
N5K1:
feature fex
!
vlan 10
!
interface Ethernet1/1
switchport access vlan 10
spanning-tree port type edge
speed 1000
!
interface Ethernet1/3 - 5
switchport mode trunk
spanning-tree port type network
!
switchport mode fex-fabric
fex associate 101
!
interface Ethernet101/1/1
N5K2:
feature fex
!
vlan 10
!
speed 1000
!
!
fex associate 102
!
Verification
In Active/Standby FEX topologies, hosts are physically attached to multiple FEXes,
but only actively forward on one path. Note that this topology is not related to vPC,
as vPC is used to achieve active/active forwarding, not active/standby. This
topology also requires the end hosts support of teaming through software. In this
particular case, the teaming is achieved through the Emulex OneCommand utility,
which manages the NIC Team/Port Channel config of the end adapter.
First, the end host is configured to team its links together, with the type defined as
failover in this case. Some other utilities call this active/standby or
primary/secondary, but they essentially mean the same thing. Note that the first
connection is listed as Primary, which is the link to N2K1 (hence N5K1), whereas
the second connection goes to N2K2 (hence N5K2).
IP addressing is configured on the logical Team adapter, similar to how IOS or NXOS puts logical configuration on a port channel interface.
To test the traffic flows, you can use the iPerf application to generate bulk TCP or
UDP traffic. In the output below, we see the CNA Server receiving two TCP streams
of approximately 1Gbps each, one from Server 1 and one from Server 2.
From the network side, the interface counters indicate that both of these flows are
going through the link to N2K1/N5K1, while the backup link through N2K2/N5K2 is
unused.
N5K1# show interface e101/1/1 | include rate
30 seconds input rate 38455104 bits/sec, 75106 packets/sec
30 seconds output rate 1819517120 bits/sec, 149849 packets/sec
input rate 38.46 Mbps, 75.11 Kpps;
output rate 1.82 Gbps

, 149.85 Kpps
input rate 1.07 Kbps, 2 pps; output rate 200 bps
, 0 pps
A failure of the FEX port from N5K1 to N2K1 signals a link-down event to the end
host.
N5K1# config t
Enter configuration commands, one per line.
End with CNTL/Z. N5K1(config)# int e1/10
N5K1(config-if)# shut
2013 Mar
2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_CFG_CHANGE: Interface Ethernet1/10 is down(Config change)
2013 Mar
2 19:19:46 N5K1 %FEX-5-FEX_PORT_STATUS_NOTI: Uplink-ID 1 of Fex 101 that is connected with Ethernet1/10 ch
2013 Mar
2 19:19:46 N5K1 %NOHMS-2-NOHMS_ENV_FEX_OFFLINE: FEX-101 Off-line (Serial Number SSI16330GT8)
2013 Mar
2 19:19:46 N5K1 %PFMA-2-FEX_STATUS: Fex 101 is offline
2013 Mar
2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_MODULE_REMOVED: Interface Ethernet101/1/1 is down (module removed)
2013 Mar
2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_MODULE_REMOVED: Interface Ethernet101/1/2 is down (module removed)
2013 Mar
2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/3 is down (Interface removed
2013 Mar
2013 Mar
2013 Mar
2013 Mar
2013 Mar
2013 Mar
2013 Mar
2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/10 is down (Interface remove
2013 Mar
2013 Mar
2013 Mar
2013 Mar
2013 Mar
2013 Mar
2013 Mar
2013 Mar
2013 Mar
2013 Mar
2013 Mar
2013 Mar
2013 Mar
2013 Mar
2013 Mar
2013 Mar
2013 Mar
2013 Mar
2013 Mar
2013 Mar
2013 Mar
2013 Mar
2013 Mar
2013 Mar
N5K1(config-if)# 2013 Mar

2013 Mar
2 19:19:47 N5K1 %FEX-5-FEX_PORT_STATUS_NOTI: Uplink-ID 1 of Fex 101 that is connected wit
2 19:19:47 N5K1 %ETHPORT-5-IF_DOWN_ADMIN_DOWN: Interface Ethernet1/10 is down (Administratively down)
The end hosts NIC Teaming software detects the primary link failure and begins to
forward via the backup path.
From the network view, we see that traffic is now re-routed through the backup path
via N2K2/N5K2.
, 140.58 Kpps

(FEX)
FEX Active Active Host vPC
Task
Configure N5K1's link to Server 1 in VLAN 10.
Configure FEX support as follows:
Configure N5K1 to pair with N2K1 using FEX number 101.
Configure N5K2 to pair with N2K2 using FEX number 102.
Configure a vPC between N5K1 and N5K2 as follows:
Configure vPC domain 1 on the vPC peers N5K1 and N5K2.
Use the mgmt0 port as the vPC Peer Keepalive link.
Use LACP for negotiation of all port channels.
Configure all links between the vPC peers as Port-Channel 1, and use this
as the vPC Peer Link.
Configure N5K1 and N5K2's links to the Emulex CNA Server as PortChannel 10 and vPC 10.
Port-Channel 10 should be an access port in VLAN 10.
Configure the Emulex CNA Server with LACP NIC Teaming, and use the IP address
10.0.0.10/24 for the NIC Team.
and that traffic to the server is being load balanced across both links through
N2K1/N5K1 and N2K2/N5K2.
Configuration
N5K1:
feature lacp
feature vpc
feature fex
!
vlan 10
!
vpc domain 1
peer-keepalive destination 192.168.0.52
!
interface port-channel1
vpc peer-link
!
vpc 10
!
speed 1000
!
channel-group 1 mode active
!
fex associate 101
!
N5K2:
feature lacp
feature vpc
feature fex
!
vlan 10
!
vpc domain 1
!
vpc peer-link
!
vpc 10
!
speed 1000
!
!
fex associate 102
!
Verification
Fabric Extender (FEX) and vPC topologies currently come in three forms. The first is
a vPC from the FEX southbound to the end server, sometimes called a Host vPC;
the second is a vPC from the FEX northbound to the parent switches, sometimes
called a Fabric vPC; and the third is both a southbound and northbound vPC from
the FEX, which is considered an Enhanced vPC or EvPC. Note that EvPC is only
supported on newer hardware platforms with corresponding newer software
releases. This particular configuration is considered the first variation, a Host vPC.
This example uses the same physical topology as before, except now the server
attached to the Fabric Extenders can do active/active forwarding. This is
accomplished by configuring a vPC between the parent switches of the FEXes.
Logically, this topology would be the same as if the CNA server were physically
wired to N5K1 with one link of its NIC, and then to N5K2 with the other link. This is
again because the FEX simply acts as a remote line card of the parent switch and
behaves just like a module of a modular switch. Because of the vPC configuration,
N5K1 and N5K2 appear to be the same upstream switch from the CNA servers
perspective; therefore, it can do active/active forwarding and load balancing just as
if it was dual attached to a single switch.

From the server side, the NIC Teaming software is configured to form an LACPbased team. Note that although the terms LACP and 802.3ad are normally
interchangeable, some variations of NIC Teaming software use one term to define a
channel as mode on and the other as mode active. In the case of the Emulex
OneCommand, if you choose 802.3ad teaming, you would need to configure the
channel-group 10 mode on on the NX-OS side, while LACP means that the
channel mode can be active. As shown below, the load balancing method can also
be chosen based on load, IP address, or MAC address.
Like before, the IP address goes on the logical team adapter, not the physical links.
From the network side, the first major verification is to ensure that the vPC peering
is up between the 5Ks. Only after the keepalive is confirmed and the vPC peer link
is formed can the vPC to the end host actually form. Note that the same vPC
consistency rules apply to FEX-based vPCs as to regular vPCs.
N5K1# show vpc
Legend:
(*) - local vPC is down, forwarding via vPC peer-link
vPC domain id
: 1
Peer status
: peer adjacency formed ok
: peer is alive
vPC keep-alive status
Configuration consistency status: success

Per-vlan consistency status
: success
Type-2 consistency status
: success
vPC role
: primary
Number of vPCs configured
: 1
Peer Gateway
: Disabled
Dual-active excluded VLANs
: -
Graceful Consistency Check
: Enabled
vPC Peer-link status

--------------------------------------------------------------------id
Port
Status Active vlans
--
----
------ -------------------------------------------------- 1 Po1
up
1,10
vPC status
---------------------------------------------------------------------------id
Port
Status Consistency Reason
Active vlans
------ ----------- ------ ----------- -------------------------- ----------- 10 Po10

success
success
up
10
The FEX configuration itself it technically unrelated to the vPC, as the FEX Fabric
Ports are configured the same as before.
N5K1# show interface fex-fabric
Fabric
Fex
Port
Fabric
Port State
Fex
Uplink
FEX
Model
Serial
--------------------------------------------------------------- 101
1
N2K-C2232PP-10GE
Eth1/10
Active
Eth1/11
Active
SSI16330GT8

Fabric
Fex
Port
Fabric
Port State
Fex
Uplink
FEX
Model
Serial
--------------------------------------------------------------- 102
2
N2K-C2232PP-10GE
SSI15030C1R
For final verification, generate traffic flows between the servers and note the
interface statistics of the FEX host ports to the Emulex CNA server. In the below
output, iPerf is used to generate bulk TCP flows from Server 1 and Server 2 to the
CNA server.
These two flows are near line-rate for the 1GigE attached Server 1 and Server 2.
The difference between this example and the last one, however, is that these 2 x
1Gbps flows are distributed between the vPC member ports to the CNA server. This
can be verified as seen through the interface counters below:
output rate 934.56 Mbps

, 76.97 Kpps

, 76.98 Kpps
In the case of a link failure, traffic will automatically be rerouted to the other
available member links after LACP detects the fault. As shown below, when N5K2's
link to the downstream N2K2 FEX goes down, both 1Gbps traffic flows are rerouted
to the other FEX.
N5K2# config t
2013 Mar
2013 Mar
2 22:10:07 N5K2 %FEX-5-FEX_PORT_STATUS_NOTI: Uplink-ID 2 of Fex 102 that is connected with Ethernet1/11 ch
2013 Mar
2 22:10:07 N5K2 %NOHMS-2-NOHMS_ENV_FEX_OFFLINE: FEX-102 Off-line (Serial Number SSI15030C1R)
2013 Mar
2 22:10:07 N5K2 %PFMA-2-FEX_STATUS: Fex 102 is offline
<snip>
, 151.41 Kpps

(FEX)
Fabric Extenders (FEX)
Task
Configure N5K1's link connecting to N2K1 as a FEX port
These links should both be STP Edge Ports.
Configure the Emulex CNA Server with the IP address 10.0.0.10/24 on this link.
When complete, Server 1 and the Emulex CNA Server should have IP reachability to
each other.
Configuration
N5K1:
feature fex
!
vlan 10
!
speed 1000
!
fex associate 101
!
Verification
Fabric Extenders (FEXes) are access switches that behave as remote line cards of
a parent switch. After the FEX is paired with the parent switch, such as a Nexus 5K
or 7K, all configuration occurs on the upstream parent. From the parent switchs
perspective, the FEX is simply another module or line card, and is configured as
such.
In the output below, we can see that a Nexus 2232PP FEX is paired with the parent
switch N5K1 as FEX number 101. This means that the FEX is simply treated as
module 101 from the 5Ks perspective.
N5K1# show fex
FEX
Number
FEX
FEX
Description
State
FEX
Model
Serial
-----------------------------------------------------------------------101
FEX0101
Online
N2K-C2232PP-10GE
SSI16330GT8
The detailed output below shows the specifics of this FEX, such as the software
version downloaded from the parent and what the state is. When the state is online,
the most important portion of the output shown below is how the downstream FEX
ports are pinned to the upstream Fabric Ports. In this topology, there is only one
physical uplink from the FEX to N5K1, so all FEX ports are pinned to the Fabric Port
E1/10. In a case in which more physical links are used, the pinning of FEX ports can
be controlled with the pinning max-links command under the global FEX
configuration, or the Fabric Port can be configured as a port-channel, essentially
dynamically pinning all FEX ports to all ports in the channel at the same time. In the
latter case, traffic is then load balanced based on the Port-Channel load balancing
method.
N5K1# show fex detail
FEX: 101 Description: FEX0101 state: Online
FEX version: 5.1(3)N1(1a) [Switch version: 5.1(3)N1(1a)]
FEX Interim version: 5.1(3)N1(1a)
Switch Interim version: 5.1(3)N1(1a) Extender Serial: SSI16330GT8
Extender Model: N2K-C2232PP-10GE
,
Part No: 73-12533-05

Card Id: 82, Mac Addr: 54:78:1a:30:3d:c2, Num Macs: 64
Module Sw Gen: 12594
[Switch Sw Gen: 21]
post level: complete pinning-mode: static
Max-links: 1
Fabric port for control traffic: Eth1/10

FCoE Admin: false
FCoE Oper: true
FCoE FEX AA Configured: false
Fabric interface state:
Fex Port
Eth101/1/2
State
Up
Eth1/10 - Interface Up. State: Active

Fabric Port Eth101/1/1
Eth1/10
Eth101/1/3
Down
None
Eth101/1/4
Down
None
Eth101/1/5
Down
None
Eth101/1/6
Down
None
Eth101/1/7
Down
None
Eth101/1/8
Down
None
Eth101/1/9
Down
None
Eth101/1/10
Down
None
Eth101/1/11
Down
None
Eth101/1/12
Down
None
Eth101/1/13
Down
None
Eth101/1/14
Down
None
Eth101/1/15
Down
None
Eth101/1/16
Down
None
Eth101/1/17
Down
None
Eth101/1/18
Down
None
Eth101/1/19
Down
None
Eth101/1/20
Down
None
Eth101/1/21
Down
None
Eth101/1/22
Down
None
Up
Eth1/10
Eth101/1/23
Down
None
Eth101/1/24
Down
None
Eth101/1/25
Down
None
Eth101/1/26
Down
None
Eth101/1/27
Down
None
Eth101/1/28
Down
None
Eth101/1/29
Down
None
Eth101/1/30
Down
None
Eth101/1/31
Down
None
Eth101/1/32
Down
None
Logs:
03/02/2013 18:17:32.337586: Module register received
03/02/2013 18:17:32.339470: Registration response sent
03/02/2013 18:17:32.465539: Module Online Sequence 03/02/2013 18:17:35.611664: Module Online
When pairing between the FEX and the parent switch is complete, further
configuration of the FEX ports is the same as any other physical link. Note that there
are some behavioral differences between FEX host ports and other physical links;
for example, the FEX ports always run as STP Edge Ports with BPDU Filter and
Guard enabled. This can be seen below; although the spanning-tree port type edge
is not configured on the FEX port, it still operationally runs in that mode.
N5K1# show spanning-tree vlan 10
VLAN0010
Spanning tree enabled protocol rstp
Root ID
Priority
32778
Address
000d.eca2.edbc
This bridge is the root
Bridge ID
Interface
Hello Time
sec
Priority
32778
Address
000d.eca2.edbc
Hello Time
sec
Role Sts Cost
Max Age 20 sec
Forward Delay 15 sec
(priority 32768 sys-id-ext 10)
Max Age 20 sec
Prio.Nbr Type
---------------- ---- --- --------- -------- -------------------------------Eth1/1
Desg FWD 4
128.129
Eth101/1/1
Desg FWD 2
128.1153 Edge P2p
Edge P2p
N5K1# show spanning-tree interface e101/1/1 detail
Port 1153 (Ethernet101/1/1) of VLAN0010 is designated forwarding

Port path cost 2, Port priority 128, Port Identifier 128.1153
Designated root has priority 32778, address 000d.eca2.edbc

Designated bridge has priority 32778, address 000d.eca2.edbc
Designated port id is 128.1153, designated path cost 0
Timers: message age 0, forward delay 0, hold 0
Number of transitions to forwarding state: 1 The port type is edge
Link type is point-to-point by default Bpdu guard is enabled
Bpdu filter is enabled by default
BPDU: sent 11, received 0

(FEX)
FEX Active Active Fabric vPC
Task
Configure FEX support as follows:
Configure N5K1 and N5K2 to pair with N2K1 using FEX number 101.
Configure a vPC between N5K1 and N5K2 as follows:
Configure vPC domain 1 on the vPC peers N5K1 and N5K2.
Use the mgmt0 port as the vPC Peer Keepalive link.
Configure the FEX Fabric ports from N5K1 and N5K2 to N2K1 as PortChannel 101, and as vPC 101.
Configure the Emulex CNA Server to do Active Standby NIC teaming as follows:
Use the link to N2K2 as the primary active path and the link to N2K1 as the
secondary standby path.
Use the IP address 10.0.0.10/24 for the NIC Team, and assign its links to
VLAN 10.
and that traffic to the CNA Server is flowing only through N2K2.
Disable the link from the CNA Server to N2K2, and verify that connectivity is
maintained by using the backup path through N2K1.
Configuration
N5K1:
feature lacp
feature vpc
feature fex
!
vlan 10
!
vpc domain 1
!
vpc peer-link
!
fex associate 101
vpc 101
!
fex associate 102
vpc 102
!
speed 1000
!
!
fex associate 101
channel-group 101 mode on
!

fex associate 102
!
!
N5K2:
feature lacp
feature vpc
feature fex
!
vlan 10
!
vpc domain 1
!
vpc peer-link
!
fex associate 101
vpc 101
!
fex associate 102
vpc 102
!
speed 1000
!
!

fex associate 101
!
fex associate 102
!
!
Verification
Fabric Extender (FEX) and vPC topologies currently come in three forms. The first is
a vPC from the FEX southbound to the end server, sometimes called a Host vPC,
the second is a vPC from the FEX northbound to the parent switches, sometimes
called a Fabric vPC, and the third is both a southbound and northbound vPC from
the FEX, which is considered an Enhanced vPC, or EvPC. Note that EvPC is only
supported on newer hardware platforms with corresponding newer software
releases. This particular configuration is considered the second variation, the Fabric
vPC.
In the Fabric vPC, the end host may be single or dual attached to one or more
FEXes, but the FEX does not perform port channeling southbound to the end host.
Instead, the FEX forms a vPC northbound to multiple parent switches. Although this
does not contribute to any load distribution between the end server and the FEX, it
does more evenly distribute the load from the FEXes northbound to their parents.
The potential danger with this design, however, is that multiple parent switches,
each with separate management and control planes, are referencing the same FEX
host ports. This means that if the configuration becomes out of sync between the
parent switches, there could be a problem in the data plane of the FEX host ports. A
possible resolution to this problem is to use the Configuration Synchronization
feature, which is demonstrated in a separate scenario.
The configuration of this scenario is similar to the other FEX pairings, with the
exception that the port channel southbound from the parent switch to the FEX does
not run LACP. This is because the FEX uplink ports do not support LACP: they only
support static channels. When complete, both parent switches must agree on
identical configurations down to the FEX Fabric Ports and to the FEX Host Ports.
The consistency of the parent switches configurations is protected against using the
vPC consistency check. Note that the show vpc output below indicates that each of
the FEX Host Ports now participates in the vPC, even though there is not
channeling configured on the host ports. Like other vPC configurations, the first
verification should be that the vPC Peer Keepalive is up and the vPC Peer Link
adjacency has been formed.
N5K1# show vpc
Legend:
vPC domain id
: 1
Peer status
: peer is alive

: success
: success
vPC role
: primary
Peer Gateway
: Disabled
: -
: Enabled
: 66

--------------------------------------------------------------------id
Port
Status Active vlans
--
----
------ -------------------------------------------------- 1
Po1 up
1,10
vPC status
---------------------------------------------------------------------------id
Port
Active vlans
------ ----------- ------ ----------- -------------------------- ----------- 101

success
success
success
success
102
Po102
102400 Eth101/1/1
up
success
success
10
102401 Eth101/1/2
up
success
success
102402 Eth101/1/3
down*
Not
Consistency Check Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
102403 Eth101/1/4
102404 Eth101/1/5
down*
down*
up
Po101
up
102405 Eth101/1/6
102406 Eth101/1/7
102407 Eth101/1/8
102408 Eth101/1/9
down*
down*
down*
down*
102409 Eth101/1/10 down*
102410 Eth101/1/11 down*
102411 Eth101/1/12 down*
102412 Eth101/1/13 down*
102413 Eth101/1/14 down*
102414 Eth101/1/15 down*
102415 Eth101/1/16 down*
102416 Eth101/1/17 down*
102417 Eth101/1/18 down*
102418 Eth101/1/19 down*
102419 Eth101/1/20 down*
102420 Eth101/1/21 down*
102421 Eth101/1/22 down*
102422 Eth101/1/23 down*
102423 Eth101/1/24 down*
102424 Eth101/1/25 down*
102425 Eth101/1/26 down*
102426 Eth101/1/27 down*
102427 Eth101/1/28 down*
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
102428 Eth101/1/29 down*
102429 Eth101/1/30 down*
102430 Eth101/1/31 down*
102431 Eth101/1/32 down*
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
103424 Eth102/1/1
up
success
success
10
103425 Eth102/1/2
up
success
success
103426 Eth102/1/3
down*
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
103427 Eth102/1/4
103428 Eth102/1/5
103429 Eth102/1/6
103430 Eth102/1/7
103431 Eth102/1/8
103432 Eth102/1/9
down*
down*
down*
down*
down*
down*
103433 Eth102/1/10 down*
103434 Eth102/1/11 down*
103435 Eth102/1/12 down*
103436 Eth102/1/13 down*
103437 Eth102/1/14 down*
103438 Eth102/1/15 down*
103439 Eth102/1/16 down*
103440 Eth102/1/17 down*
103441 Eth102/1/18 down*
103442 Eth102/1/19 down*
103443 Eth102/1/20 down*
103444 Eth102/1/21 down*
103445 Eth102/1/22 down*
103446 Eth102/1/23 down*
103447 Eth102/1/24 down*
103448 Eth102/1/25 down*
103449 Eth102/1/26 down*
103450 Eth102/1/27 down*
103451 Eth102/1/28 down*
103452 Eth102/1/29 down*
103453 Eth102/1/30 down*
103454 Eth102/1/31 down*
103455 Eth102/1/32 down*
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
N5K2# show vpc

Legend:
vPC domain id
: 1
Peer status
: peer is alive

: success
: success
vPC role
: secondary
Peer Gateway
: Disabled
: -
: Enabled

--------------------------------------------------------------------id
Port
Status Active vlans
--
----
------ --------------------------------------------------
: 66
1 Po1
up
1,10
vPC status
---------------------------------------------------------------------------id
Port
Active vlans
------ ----------- ------ ----------- -------------------------- ----------- 101

success
success
success
success
102
Po102
102400 Eth101/1/1
up
success
success
10
102401 Eth101/1/2
up
success
success
102402 Eth101/1/3
down*
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
102403 Eth101/1/4
102404 Eth101/1/5
102405 Eth101/1/6
down*
down*
down*
102406 Eth101/1/7
down*
102407 Eth101/1/8
down*
102408 Eth101/1/9
down*
102409 Eth101/1/10 down*
102410 Eth101/1/11 down*
102411 Eth101/1/12 down*
102412 Eth101/1/13 down*
102413 Eth101/1/14 down*
102414 Eth101/1/15 down*
102415 Eth101/1/16 down*
102416 Eth101/1/17 down*
102417 Eth101/1/18 down*
up
Po101
up
102418 Eth101/1/19 down*
102419 Eth101/1/20 down*
102420 Eth101/1/21 down*
102421 Eth101/1/22 down*
102422 Eth101/1/23 down*
102423 Eth101/1/24 down*
102424 Eth101/1/25 down*
102425 Eth101/1/26 down*
102426 Eth101/1/27 down*
102427 Eth101/1/28 down*
102428 Eth101/1/29 down*
102429 Eth101/1/30 down*
102430 Eth101/1/31 down*
102431 Eth101/1/32 down*
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
103424 Eth102/1/1
up
success
success
10
103425 Eth102/1/2
up
success
success
103426 Eth102/1/3
down*
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
103427 Eth102/1/4
down*
103428 Eth102/1/5
down*
103429 Eth102/1/6
down*
103430 Eth102/1/7
103431 Eth102/1/8
103432 Eth102/1/9
down*
down*
down*
103433 Eth102/1/10 down*
103434 Eth102/1/11 down*
103435 Eth102/1/12 down*
103436 Eth102/1/13 down*
103437 Eth102/1/14 down*
103438 Eth102/1/15 down*
103439 Eth102/1/16 down*
103440 Eth102/1/17 down*
103441 Eth102/1/18 down*
103442 Eth102/1/19 down*
103443 Eth102/1/20 down*
103444 Eth102/1/21 down*
103445 Eth102/1/22 down*
103446 Eth102/1/23 down*
103447 Eth102/1/24 down*
103448 Eth102/1/25 down*
103449 Eth102/1/26 down*
103450 Eth102/1/27 down*
103451 Eth102/1/28 down*
103452 Eth102/1/29 down*
103453 Eth102/1/30 down*
103454 Eth102/1/31 down*
103455 Eth102/1/32 down*
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Note that both N5K1 and N5K2 are pairing with the same downstream FEXes.
Fabric
Fex
Port
Fabric
Port State
Fex
Uplink
FEX
Model
Serial
--------------------------------------------------------------101
Eth1/10
Active
N2K-C2232PP-10GE
SSI16330GT8
102
Eth1/11
Active
N2K-C2232PP-10GE
SSI15030C1R

Fabric
Fex
Port
Fabric
Port State
Fex
Uplink
FEX
Model
Serial
--------------------------------------------------------------101
Eth1/10
Active
N2K-C2232PP-10GE
SSI16330GT8
102
Eth1/11
Active
N2K-C2232PP-10GE
SSI15030C1R
From the end servers perspective, their links have been configured in
active/standby failover teaming, with the primary path being to N2K2.
For final verification of traffic distribution, Server 1 and Server 2 generate TCP flows
toward the CNA server. The end result is multiple flows for a total nearing 2Gbps,
which is the combined line rates of Server 1 and 2.
From the network side, both N5K1 and N5K2 see that all flows towards the CNA
server exit via N2K2.
, 1 pps

, 160.63 Kpps
, 1 pps

, 160.63 Kpps
Note that these outputs are nearly identical on both parent switches, as they are
both referencing the same physical FEX host ports. The key difference with this
configuration design, however, is that traffic is load balanced from the parent
switches down to the N2K2 FEX. This can be verified by viewing the counters of the
FEX Fabric Ports of the parent switches, as seen below.
N5K1# show interface e1/10 - 11 | include rate|Ethernet1
Ethernet1/10 is up
input rate 18.55 Kbps, 2 pps;
output rate 2.94 Kbps

, 2 pps
Ethernet1/11 is up

, 80.56 Kpps
Ethernet1/10 is up

, 2 pps
Ethernet1/11 is up

, 80.64 Kpps
Note that neither parent switch is sending traffic to FEX Fabric Port E1/10 because
this is the link to N2K1, which the CNA server is using as the standby connection.
Although the total of flows from Server 1 and 2 are 2Gbps, they are nearly equally
split between the FEX Fabric Ports going from N5K1 and N5K2 southbound to the
N2K2 FEX.
In the case of a failure of the servers primary uplink, traffic is automatically rerouted to the backup link via FEX N2K1, as shown below.

Ethernet1/10 is up
30 seconds output rate 987383384 bits/sec

, 80800 packets/sec
input rate 17.62 Kbps, 2 pps; output rate 518.97 Mbps, 42.43 Kpps
Ethernet1/11 is up
30 seconds input rate 8624 bits/sec, 1 packets/sec 30 seconds output rate 3128 bits/sec
, 2 packets/sec
input rate 16.26 Kbps, 2 pps; output rate 463.30 Mbps, 37.85 Kpps
Ethernet1/10 is up
, 81015 packets/sec
input rate 24.29 Mbps, 40.97 Kpps; output rate 478.05 Mbps, 39.09 Kpps
Ethernet1/11 is up
, 2 packets/sec
input rate 25.69 Mbps, 43.28 Kpps; output rate 505.58 Mbps, 41.32 Kpps

(FEX)
FEX and N5K Config Sync
Task
Enable the vPC, FEX, and LACP features on N5K1 and N5K2.
Enable Cisco Fabric Services over IP (CFSoIP) distribution between N5K1 and N5K2.
Configure vPC domain 1 between N5K1 and N5K2, and use the mgmt0 link for the
vPC Peer Keepalive.
Create a Config Sync session on both N5K1 and N5K2, and use the switch profile
name N5K.
Use the mgmt0 IP addresses as the config sync peers destination.
Verify that N5K1 and N5K2 can reach each other over CFSoIP for the config sync
session.
Without making any additional changes on N5K2, use the switch profile on N5K1 to
replicate the following configuration to both switches:
Pre-provision FEX modules 101 and 102, both of type N2K-C2232P.
Create VLAN 10.
All links to Server 1 and Server 2 should be access ports in VLAN 10.
Configure the links to the Emulex CNA Server in VLAN 10.
Commit the config and verify that both N5K1 and N5K2 identically accept it into their
running configuration.
Configuration
N5K2:
feature vpc
feature fex
feature lacp
cfs ipv4 distribute
!
vpc domain 1
peer-keepalive destination 192.168.0.51 vrf management
!
end
config sync
switch-profile N5K
sync-peers destination 192.168.0.51
verify
N5K1:
feature vpc
feature fex
feature lacp
cfs ipv4 distribute
!
vpc domain 1
peer-keepalive destination 192.168.0.52 vrf management
!
end
config sync
switch-profile N5K
sync-peers destination 192.168.0.52
verify
show switch-profile status
slot 101
provision model N2K-C2232P
!
slot 102
!
vlan 10
!

vpc peer-link
!
fex associate 101
vpc 101
!
fex associate 102
vpc 102
!
speed 1000
!
!
fex associate 101
!
fex associate 102
!
!
!
commit
Verification
Configuration Synchronization, also known as Config Sync for short or Switch
Profiles, is a way to apply a template of configuration onto multiple Nexus switches
at the same time. This feature is especially useful between vPC peers, or when FEX
deployments are used in active/active, where a downstream FEX peers with more
than one upstream parent switch. This feature helps to ensure that configurations
stay consistent between the vPC peers or FEX parents (or both as in the case
shown below), and avoid problems such as vPC failure caused by a consistency
check error.
Note that of current releases, this feature is not supported on the Nexus 7K.
Additionally, not all commands are supported in the switch profile mode, and instead
must be configured in regular global config. In this particular case, the unsupported
commands are the feature enablement of vPC, FEX, and LACP, as well as the vPC
domain creation, as the configuration is different between the vPC peers
(specifically the peer keepalive destination address).
To use config sync, the switches must first be configured to use Cisco Fabric
Services over IP (CFSoIP), as this is the control plane protocol that is actually used
to sync the config between the switches. This is enabled simply as follows:
N5K1# conf t
End with CNTL/Z. N5K1(config)# cfs ipv4 distribute
N5K1(config)# show cfs peers
Physical Fabric
------------------------------------------------------------------------Switch WWN
IP Address
------------------------------------------------------------------------- 20:00:00:0d:ec:a2:ed:80
192.168.0.51
[Local] N5K1
20:00:00:0d:ec:a4:74:00 192.168.0.52
Total number of entries = 2
Next, start a config sync session, create an identically named switch profile on each
switch, specify the IP address of the peer to sync with, and then verify their
connectivity.
N5K1# config sync
End with CNTL/Z. N5K1(config-sync)# switch-profile N5K
Switch-Profile started, Profile ID is 1 N5K1(config-sync-sp)# sync-peers destination 192.168.0.52

N5K1(config-sync-sp)# verify
Verification Successful
N5K1(config-sync-sp)#
vN5K2# config sync
End with CNTL/Z. N5K2(config-sync)# switch-profile N5K
Switch-Profile started, Profile ID is 1 N5K2(config-sync-sp)# sync-peers destination 192.168.0.51

N5K2(config-sync-sp)# verify
Verification Successful
If the switch profile is in sync between the peers, they should both agree on the
profile revision number and show the sync status as in sync.
N5K1(config-sync-sp)# show switch-profile status
switch-profile
: N5K
----------------------------------------------------------
Start-time: 382018 usecs after Sun Mar

End-time: 441035 usecs after Sun Mar
3 15:44:48 2013
3 15:44:50 2013
Profile-Revision: 1
Session-type: Initial-Exchange
Session-subtype: Init-Exchange-All
Peer-triggered: Yes
Profile-status: Sync Success
Local information:
---------------- Status: Commit Success
Error(s):
Peer information:
---------------- IP-address: 192.168.0.52
Sync-status: In sync
Status: Commit Success
Error(s):
switch-profile
: N5K
----------------------------------------------------------

3 16:35:38 2013
3 16:35:40 2013
Profile-Revision: 1
Session-type: Initial-Exchange
Session-subtype: Init-Exchange-All
Peer-triggered: No
Profile-status: Sync Success
Local information:
Error(s):
Peer information:
---------------- IP-address: 192.168.0.51
Sync-status: In sync
Status: Commit Success
Error(s):
Now the switches are ready to accept the configuration changes to synchronize.
Commands are entered just like in global config, but they are not immediately
applied. Instead they are sent to the switch profile buffer, as shown below. Before
the buffer is committed, the buffer can be deleted or modified as desired. The line
numbers of the buffer show how the config will sequentially be applied. Therefore,
any configurations that are sensitive to order of operations must have the correct
line numbering in the buffer before a commit is executed.
N5K1(config-sync-sp)# slot 101
N5K1(config-sync-sp-slot)#
N5K1(config-sync-sp-slot)# !
N5K1(config-sync-sp-slot)# slot 102
N5K1(config-sync-sp-slot)#
N5K1(config-sync-sp-slot)# !
N5K1(config-sync-sp-slot)# vlan 10
N5K1(config-sync-sp-vlan)# !
N5K1(config-sync-sp-vlan)# interface port-channel1
N5K1(config-sync-sp-if)#
vpc peer-link
N5K1(config-sync-sp-if)# !
N5K1(config-sync-sp-if)# interface port-channel101
fex associate 101
vpc 101
N5K1(config-sync-sp-if)# interface port-channel102
fex associate 102
vpc 102
N5K1(config-sync-sp-if)# interface Ethernet1/1 - 2
N5K1(config-sync-sp-if-range)#
speed 1000
N5K1(config-sync-sp-if-range)# !
N5K1(config-sync-sp-if-range)# interface Ethernet1/3 - 5
N5K1(config-sync-sp-if-range)# !
N5K1(config-sync-sp-if-range)# interface Ethernet1/10
fex associate 101
N5K1(config-sync-sp-if)# interface Ethernet1/11
fex associate 102
N5K1(config-sync-sp-if)# interface Ethernet101/1/1
N5K1(config-sync-sp-if)# interface Ethernet102/1/1
N5K1(config-sync-sp-if)# show switch-profile buffer
switch-profile
: N5K
---------------------------------------------------------Seq-no
Command
---------------------------------------------------------1
1.1
2
2.1
slot 101
slot 102
vlan 10
4.1
4.2
4.3
vpc peer-link
5.1
5.2
fex associate 101
5.3
vpc 101
6.1
6.2
fex associate 102
6.3
vpc 102
interface Ethernet1/1-2
7.1
7.2
7.3
speed 1000
8.1
8.2
8.3
9.1
9.2
fex associate 101
9.3
10
10.1
10.2
fex associate 102
10.3
11
11.1
12
12.1

When the commands in the buffer are acceptable, the profile is committed. During
the commit procedure, the config is synchronized across to the other peer using
CFSoIP, and applied sequentially. If there is an error in applying the config, all
commands in the buffer are rolled back and the commit fails. In other words, either
the commit succeeds 100 percent, or no config is applied to either peer. In the
output below, we see that the commit was successful, and syslog messages begin
to appear as config changes, link up/down events, etc. occur just as if you had
applied the commands manually on each switch individually.
N5K1(config-sync-sp-if)# commit
Verification successful...
Proceeding to apply configuration. This might take a while depending on amount of configuration in buffer.
Please avoid other configuration changes during this time.
2013 Mar
3 15:49:31 N5K1 %ETH_PORT_CHANNEL-5-CREATED: port-channel1 created
2013 Mar
3 15:49:31 N5K1 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel1 is down (No operatio
2013 Mar
2013 Mar
2013 Mar
3 15:49:33 N5K1 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel101 is down (No operat
2013 Mar
3 15:49:33 N5K1 last message repeated 2 times
2013 Mar
<snip> Commit Successful
N5K1(config-sync)#
2013 Mar
2013 Mar
2013 Mar
2013 Mar
2013 Mar
3 16:40:24 N5K2 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel101 is down (No operat
2013 Mar
3 16:40:24 N5K2 last message repeated 2 times
2013 Mar
<snip>
When the commit is successful, N5K1 automatically exits out of the switch profile
configuration mode. If additional config changes are required, a new switch profile
session must be started, using the same session name as before. Note in the output
below that both switches agree on the switch profile revision number, and the
switches are in sync.
N5K1(config-sync)# show switch-profile status
switch-profile
: N5K
----------------------------------------------------------

3 15:49:28 2013
3 15:49:38 2013
Profile-Revision: 2
Session-type: Commit
Session-subtype: Peer-triggered: No Profile-status: Sync Success
Sync-status: In sync Status: Commit Success
Error(s):
switch-profile
: N5K
----------------------------------------------------------

3 16:40:18 2013
3 16:40:29 2013
Profile-Revision: 2
Session-type: Commit
Session-subtype: Peer-triggered: Yes Profile-status: Sync Success
Local information:
Error(s):
Peer information:
---------------IP-address: 192.168.0.51
Sync-status: In sync Status: Commit Success
Error(s):
From N5K2s perspective, the configuration commands appear in the running config
just as if they had been entered manually in global configuration.
N5K2# show run interface
!Command: show running-config interface

!Time: Sun Mar
3 17:12:09 2013
version 5.1(3)N1(1a)
vpc peer-link
fex associate 101
vpc 101
fex associate 102
vpc 102
<snip>
Further verification shows that the configured features such as the vPC, FEX Fabric
Ports, VLANs, etc. are functional.
N5K2# show vpc
Legend:
vPC domain id
: 1
Peer status
: peer is alive

: success
: success
vPC role
: secondary
: 66
Peer Gateway
: Disabled
: -
: Enabled

--------------------------------------------------------------------id
Port
Status Active vlans
--
----
------ --------------------------------------------------
Po1
up
1,10
vPC status
---------------------------------------------------------------------------id
Port
Active vlans
------ ----------- ------ ----------- -------------------------- ----------101
Po101
up
success
success
102
Po102
up
success
success
up
success
success
10
102400 Eth101/1/1
102401 Eth101/1/2
up
success
success
102402 Eth101/1/3
down*
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
Applicable
Performed
Not
102403 Eth101/1/4
102404 Eth101/1/5
down*
down*
102405 Eth101/1/6
down*
102406 Eth101/1/7
down*
102407 Eth101/1/8
102408 Eth101/1/9
down*
down*
<snip>
VLAN0010
Root ID
Bridge ID
Interface
Priority
32778
Address
000d.eca2.edbc
Cost
Port
4096 (port-channel1)
Hello Time
Priority
32778
Address
000d.eca4.743c
Hello Time
sec
sec
Role Sts Cost
Max Age 20 sec
Max Age 20 sec
Prio.Nbr Type
---------------- ---- --- --------- -------- -------------------------------Po1
Root FWD 1
128.4096 (vPC peer-link) Network P2p
Eth1/1
Desg FWD 4
128.129
Edge P2p
Eth1/2
Desg FWD 4
128.130
Edge P2p
Eth101/1/1
Desg FWD 1
128.1153 (vPC) Edge P2p
Eth102/1/1
Desg FWD 1
128.1281 (vPC) Edge P2p
Nexus Technology Labs - Virtual Port

Channels (vPC)
vPC and HSRP
Task
This task applies only to INE Bootcamp partipants. Load
balancing is not included in the self-paced training curriculum at
this time.
Configure vPC between N7K1 and N7K2 as follows:
N7K1 and N7K2 are the vPC Peers.
Configure all available F1 ports between the vPC peers as Port-Channel 1,
and use this as the vPC Peer Link.
Use the mgmt0 ports for the Peer Keepalive Link.
Configure all available links from N7K1 and N7K2 to N5K1 in Port-Channel
51, and as vPC 51.
All portchannels should be trunks, STP Network Ports, and use LACP for
negotiation.
Configure VLAN assignments and the servers as follows:
Configure the link from N5K1 to Server 1 as an access port in VLAN 10.
Configure the link from N5K1 to Server 2 as an access port in VLAN 20.
Server 1 should use the IP address 10.0.0.1/24, and a default gateway of
10.0.0.254.
Server 2 should use the IP address 20.0.0.2/24, and a default gateway of
20.0.0.254.
Configure Inter-VLAN Routing and HSRP on N7K1 and N7K2 as follows:
Create interfaces VLAN 10 and VLAN 20 on N7K1 and N7K2, using the IP
address 10.0.0.X/24, where X is the last octet of the IP address on their
mgmt0 interfaces.
Configure HSRP group 10 for VLAN 10 on N7K1 and N7K2 using the virtual
address 10.0.0.254/24.
address 20.0.0.254/24.
Set the Port Channel load balancing method on the Nexus switches to include the
source and destination layer 4 port numbers.
When complete, Server 1 and Server 2 should have IP reachability to each other,
and traffic between them should be load distributed across all links in the vPC.
Configuration
N5K1:
feature lacp
!
vlan 10,20
!
port-channel load-balance ethernet source-dest-port
!
switchport mode access
speed 1000
!
speed 1000
no shutdown
!
N7K1-1:
feature vpc
feature lacp
!
vlan 10,20
!
port-channel load-balance src-dst ip-l4port-vlan
!
vpc domain 1
!
no shutdown
!
switchport
vpc peer-link
!
no shutdown
!
switchport
vpc 51
!
interface Vlan10
no shutdown
ip address 10.0.0.71/24
hsrp 10
ip 10.0.0.254
!
interface Vlan20
no shutdown
ip address 20.0.0.71/24
hsrp 20
ip 20.0.0.254
N7K2-1:
feature vpc
feature lacp
!
vlan 10,20
!
!
vpc domain 1
!
no shutdown
!
switchport
vpc peer-link
!
no shutdown
!
switchport
vpc 51
!
interface Vlan10
no shutdown
ip address 10.0.0.75/24
hsrp 10
ip 10.0.0.254
!
interface Vlan20
no shutdown
ip address 20.0.0.75/24
hsrp 20
ip 20.0.0.254
Verification
This scenario demonstrates how the forwarding pattern of vPC and HSRP combined
differs from that of just HSRP on its own. The results of this scenario would be
similar if either VRRP or GLBP were used, because all of the First Hop Redundancy
Protocols (FHRPs) have special behavior that interacts with vPC.
First, the layer 2 only switch N5K1 has access ports in VLANs 10 and 20, and a
trunking port channel that carries both VLANs. From N5K1s perspective, this port
channel logically connects to just one upstream switch, but in reality it is the two
physical vPC Peers, N7K1 and N7K2.
N5K1# show spanning-tree vlan 10,20
VLAN0010
Root ID
Bridge ID
Priority
4106
Address
68bd.abd7.6041
Cost
Port
Hello Time
Priority
32778
Address
000d.eca2.edbc
Hello Time
Interface
sec
sec
Role Sts Cost
Max Age 20 sec
Max Age 20 sec
Prio.Nbr Type
---------------- ---- --- --------- -------- -------------------------------- Po51

Root FWD 1
128.4146 Network P2p Eth1/1
Desg FWD 4
128.129
P2p
VLAN0020
Root ID
Bridge ID
Interface
Priority
4116
Address
68bd.abd7.6041
Cost
Port
Hello Time
Priority
32788
Address
000d.eca2.edbc
Hello Time
sec
sec
Role Sts Cost
Max Age 20 sec
Max Age 20 sec
Prio.Nbr Type
---------------- ---- --- --------- -------- -------------------------------- Po51

Root FWD 1
Desg FWD 4
128.4146 Network P2p Eth1/2

128.130
P2p
According to normal layer 2 switching vs. layer 3 routing logic, any hosts in VLAN 10
that want to talk to hosts in VLAN 20 must have their traffic switched up to the
default gateway and have the layer 2 header re-written with a new source and
destination MAC address, then switched to the final destination. In this case, there
are two default gateways for each VLAN 10 and 20, both N7K1 and N7K2 that share
the HSRP virtual IP address. In the output below we can see that N7K1 is the active
HSRP router for both groups.
N7K1-1# show hsrp
Vlan10 - Group 10
(HSRP-V1) (IPv4) Local state is Active
, priority 100 (Cfged 100)
Forwarding threshold(for vPC), lower: 1 upper: 100
Hellotime 3 sec, holdtime 10 sec
Next hello sent in 1.595000 sec(s)
Virtual IP address is 10.0.0.254 (Cfged)
Active router is local
Standby router is 10.0.0.75 , priority 100 expires in 5.957000 sec(s)
Authentication text "cisco"
Virtual mac address is 0000.0c07.ac0a (Default MAC)
4 state changes, last state change 00:48:39
IP redundancy name is hsrp-Vlan10-10 (default)
Vlan20 - Group 20
Active router is local
Standby router is 20.0.0.75 , priority 100 expires in 6.264000 sec(s)
Virtual mac address is 0000.0c07.ac14 (Default MAC)
N7K2-1# show hsrp
Vlan10 - Group 10
(HSRP-V1) (IPv4) Local state is Standby
Active router is 10.0.0.71, priority 100 expires in 9.872000 sec(s)
Standby router is local
Virtual mac address is 0000.0c07.ac0a (Default MAC)

Vlan20 - Group 20
Active router is 20.0.0.71, priority 100 expires in 5.152000 sec(s)
Standby router is local
Virtual mac address is 0000.0c07.ac14 (Default MAC)
The potential problem with this design is that if traffic is switched to N7K2, the
standby HSRP router, because of the Port Channel load balancing method of N5K1,
it would have to be sent to the active HSRP router, N7K1, to be routed. This means
that traffic would have to transit the vPC Peer Link, which is undesirable because
the aggregate of flows from vPC Member Ports would quickly overwhelm the vPC
Peer Link. To prevent this from being necessary, vPC changes the behavior of the
FHRPs so that the standby router can forward the same as the active router. The
result of this can be seen below.
Server 2 generates bulk TCP flows to Server 1 using the iPerf app. The aggregate
of flows nears 1Gbps.
When the access switch, N5K1, receives these flows, they have the destination
MAC address of the virtual HSRP address. This MAC address is reachable via the
port channel to the upstream 7K, and is then load balanced based on the layer 4
port information of the flows as configured.
N5K1# show port-channel summary
Flags:
D - Down
P - Up in port-channel (members)
I - Individual
H - Hot-standby (LACP only)
s - Suspended
r - Module-removed
S - Switched
R - Routed
U - Up (port-channel)
M - Not in use. Min-links not met
-------------------------------------------------------------------------------Group Port-
Type
Protocol
Member Ports
Channel
-------------------------------------------------------------------------------51
Po51(SU)
Eth
LACP Eth1/6(P)
Eth1/7(P)
Eth1/8(P)
Eth1/9(P)
N5K1# show interface e1/6 - 9 | include rate|Ethernet

Ethernet1/6 is up
Hardware: 1000/10000 Ethernet, address: 000d.eca2.ed8d (bia 000d.eca2.ed8d)
input rate 2.96 Mbps, 5.28 Kpps; output rate 111.59 Mbps
, 13.04 Kpps
Ethernet1/7 is up
Hardware: 1000/10000 Ethernet, address: 000d.eca2.ed8e (bia 000d.eca2.ed8e)
, 13.19 Kpps
Ethernet1/8 is up
Hardware: 1000/10000 Ethernet, address: 000d.eca2.ed8f (bia 000d.eca2.ed8f)
, 28.20 Kpps
Ethernet1/9 is up
Hardware: 1000/10000 Ethernet, address: 000d.eca2.ed90 (bia 000d.eca2.ed90)
, 28.28 Kpps
In the output above, we see that some traffic goes from N5K1 to N7K1, and some
from N5K1 to N7K2. Without the vPC modification to HSRP, this traffic should have
to be switched from N7K2 to N7K1 before it can be routed, because N7K2 isnt the
active HSRP router. However, the interface counters of the vPC Peer Link, as seen
below, indicate that the flows are not switched in that direction, and instead N7K2 is
routing them itself even though it is HSRP standby.
N7K2-1# show interface port-channel 1 | include rate
30 seconds output rate 1544 bits/sec, 1 packets/sec input rate 1.56 Kbps, 1 pps; output rate 1.54 Kbps
, 1 pps
This behavior can be further verified by disabling the uplinks from N5K1 to N7K1, as
shown below.
N5K1# show cdp neighbors
Capability Codes: R - Router, T - Trans-Bridge, B - Source-Route-Bridge
S - Switch, H - Host, I - IGMP, r - Repeater,
V - VoIP-Phone, D - Remotely-Managed-Device,
s - Supports-STP-Dispute
Device-ID
Local Intrfce Hldtme Capability
Platform
Port ID
Nexus-MGMT-SW
mgmt0
175
S I
WS-C3550-48
N5K2(FLC12480280)
Eth1/3
140
S I s
N5K-C5020P-BF Eth1/3
N5K2(FLC12480280)
Eth1/4
140
S I s
N5K2(FLC12480280)
Eth1/5
140
S I s
N7K1-1(JAF1510CMLQ)
Eth1/6
170
R S s
N7K-C7010
Eth2/3
169
R S s
N7K-C7010
Eth2/4
Fas0/31
N7K1-1(JAF1510CMLQ)
N7K2-1(TBM14311481)
Eth1/8
167
R S s
N7K-C7010
Eth2/5
N7K2-1(TBM14311481)
Eth1/9
167
R S s
N7K-C7010
Eth2/6
Eth1/7
N5K1# config t
End with CNTL/Z. N5K1(config)# int e1/6 7
N5K1(config-if-range)# shutdown
2013 Mar
5 21:54:43 N5K1 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel51: Ethernet1/7 is down
2013 Mar
2013 Mar
2013 Mar
2013 Mar
2013 Mar
With the links from N5K1 to N7K1 disabled, the only way for them to be switched
northbound is via N7K2, as shown below.

vN5K1# show interface e1/6 - 9 | include Ethernet1|rate
Ethernet1/6 is down (Administratively down)
input rate 0 bps, 0 pps; output rate 0 bps, 0 pps
Ethernet1/7 is down (Administratively down)
input rate 0 bps, 0 pps; output rate 0 bps, 0 pps
Ethernet1/8 is up
, 42.86 Kpps
Ethernet1/9 is up
, 42.84 Kpps
Because N7K1 still has the vPC Peer Link forwarding VLANs 10 and 20, it is still the
HSRP Active router.
N7K1-1# show hsrp | include state|Group
Vlan10 - Group 10
4 state changes, last state change 02:02:24 Vlan20 - Group 20
N7K2-1# show hsrp | include state|Group
Vlan10 - Group 10
7 state changes, last state change 01:36:48 Vlan20 - Group 20
If N7K1 is the router that is doing the layer 2 header re-write, the vPC Peer Link
should show 1Gbps input and output, which it does not according to the output
below.
N7K2-1# show interface port-channel 1 | include rate

30 seconds output rate 1504 bits/sec, 1 packets/sec input rate 1.58 Kbps, 1 pps; output rate 1.50 Kbps
, 1 pps
Note that this behavior, in which both the active and standby HSRP routers are able
to forward traffic, is the default. There is no additional configuration needed to
accomplish this. As long as HSRP/VRRP/GLBP is configured in conjunction with
vPC, this behavior will be seen.

Channels (vPC)
Back-to-Back vPC
Task
Configure a vPC domain between N7K1 and N7K2 as follows:
Configure vPC domain 70 between the vPC Peers, and use their mgmt0
interfaces for the vPC Peer Keepalive Link.
Configure all available links from N7K1 and N7K2 to N5K1 and N5K2 in PortChannel 70, and as vPC 70.
All port channels should be 802.1q trunks, be configured as STP Network
Ports, and use LACP for negotiation.
Configure a vPC domain between N5K1 and N5K2 as follows:
Configure vPC domain 50 between the vPC Peers, and use their mgmt0
interfaces for the vPC Peer Keepalive Link.
Configure all available ports between the vPC peers as Port-Channel 1, and
use this as the vPC Peer Link.
Configure all available links from N5K1 and N5K2 to N7K1 and N7K2 in PortChannel 50, and as vPC 50.
Port Channels 1 and 50 should be 802.1q trunks, be configured as STP
Network Ports, and use LACP for negotiation.
Configure a vPC from N5K1 and N5K2 to Server 1 as follows:
Use LACP negotiation for all port channels and NIC Teams.
Configure the links from N5K1 and N5K2 to Server 1 as Port-Channel 10,
vPC 10, and as an access port in VLAN 10.
Configure the links from N5K1 and N5K2 to Server 2 as Port-Channel 20,
vPC 20, and as an access port in VLAN 20.
Server 1 should do NIC Teaming for its links to N5K1 and N5K2, use the IP
address 10.0.0.1/24 for the NIC Team, and have a default gateway of
10.0.0.254.
Server 2 should do NIC Teaming for its links to N5K1 and N5K2, use the IP
address 20.0.0.2/24 for the NIC Team, and have a default gateway of
20.0.0.254.
Configure Inter-VLAN Routing on N7K1 and N7K2 as follows:
Create interfaces VLAN 10 and VLAN 20 on N7K1 and N7K2, using the IP
address 10.0.0.X/24, where X is the last octet of the IP address on their
mgmt0 interfaces.
address 10.0.0.254/24.
address 20.0.0.254/24.
Set the Port Channel load balancing method on the Nexus switches to include the
source and destination layer 4 port numbers.
When complete, Server 1 and Server 2 should have IP reachability to each other,
and traffic between them should be load distributed across all links in the vPCs.
Configuration
N5K1:
feature lacp
feature vpc
!
vlan 10,20
!
!
vpc domain 50
!
vpc peer-link
!
speed 1000
vpc 10
!
speed 1000
vpc 20
!
vpc 50
!
speed 1000
!
speed 1000
!
!
N5K2:
feature lacp
feature vpc
!
vlan 10,20
!
!
vpc domain 50
!

vpc peer-link
!
speed 1000
vpc 10
!
speed 1000
vpc 20
!
vpc 50
!
speed 1000
!
speed 1000
!
!
N7K1:
feature vpc
feature lacp
feature hsrp
feature interface-vlan
!
vlan 10,20
!
!
vpc domain 70
!
no shutdown
!
switchport
vpc peer-link
!
switchport
no shutdown
!
switchport
vpc 70
!
interface Vlan10
no shutdown
ip address 10.0.0.71/24
hsrp 10
ip 10.0.0.254
!
interface Vlan20
no shutdown
ip address 20.0.0.71/24
hsrp 20
ip 20.0.0.254
N7K2:
feature vpc
feature lacp
feature hsrp
!
vlan 10,20
!
!
vpc domain 70
!
no shutdown
!
switchport
vpc peer-link
!
switchport
no shutdown
!
switchport
vpc 70
!
interface Vlan10
no shutdown
ip address 10.0.0.75/24
hsrp 10
ip 10.0.0.254
!
interface Vlan20
no shutdown
ip address 20.0.0.75/24
hsrp 20
ip 20.0.0.254
Verification
Back-to-Back vPC is not a specific configuration, but instead refers to a design in
which two vPC peers have a vPC configured toward another set of vPC peers,
which likewise have a vPC configured back the other direction. In this particular
design, the Back-to-Back vPC consists of the vPC from the Nexus 7Ks southbound
to the Nexus 5Ks, and then from the Nexus 5Ks northbound back to the Nexus 7Ks.
The resulting logical topology is a single logical 8-way port channel between the
boxes.
From the Nexus 7K's perspective, this simply looks like a normal vPC configuration,
with Port-Channel 70 being the vPC Member Port.
N7K1# show vpc
Legend:
vPC domain id
: 70
Peer status
: peer is alive
Configuration consistency status
: success
: success
: success
vPC role
: secondary
: 1
Peer Gateway
: Disabled
: -
: Enabled
Auto-recovery status
: Disabled

--------------------------------------------------------------------id
Port
Status Active vlans
--
----
------ --------------------------------------------------
Po1
up
1,10,20
vPC status
---------------------------------------------------------------------id
Port
Active vlans
--
----
------ ----------- ------
------------
70
Po70
up
success
success
1,10,20
From the Nexus 5K's perspective, this likewise looks like a normal vPC
configuration, with Port-Channels 10 and 20 going southbound to the end-hosts,
and Port-Channel 50 going northbound to the 7Ks.
N5K1# show vpc
Legend:
vPC domain id
: 50
Peer status
: peer is alive

: success
: success
vPC role
: primary
: 3
Peer Gateway
: Disabled
: -
: Enabled

--------------------------------------------------------------------id
Port
Status Active vlans
--
----
------ --------------------------------------------------
Po1
up
1,10,20
vPC status
---------------------------------------------------------------------------id
Port
Active vlans
------ ----------- ------ ----------- -------------------------- ----------10
Po10
up
success
success
10
20
Po20
up
success
success
20
50
Po50
up
success
success
1,10,20
To test the final distribution of traffic flows, Server 2 uses the iPerf app to generate a
number of TCP flows to Server 1, as shown below. The aggregate of flows is near
1Gbps. Although you might expect the aggregate to be closer to 2Gbps, which is
both links in the NIC Team of the end servers, the load balancing method of the Intel
NIC card is always choosing the same member of the NIC Team because the
source and destination IP address is the same for all flows. If two or more
destinations had flows going to them, both links of the team could be utilized.
When the traffic flows enter the 5Ks, their more granular method of load balancing
based on both source and destination TCP ports allows the flows to be more evenly
split among the members of the vPCs. As shown below, N5K1 is receiving the
majority of the traffic in from Server 2.
N5K1# show interface e1/2 | include rate
30 seconds output rate 2781568 bits/sec, 5402 packets/sec input rate 950.13 Mbps
, 78.26 Kpps; output rate 2.78 Mbps, 5.40 Kpps
N5K2# show interface e1/2 | include rate
30 seconds output rate 4094712 bits/sec, 7938 packets/sec input rate 592 bps
, 0 pps; output rate 4.09 Mbps, 7.94 Kpps
N5K1 then distributes the load of these ~ 1Gbps flows between the members of vPC 50, which go northbound to the 7Ks.
N5K1# show port-channel summary interface po50
Flags:
D - Down
I - Individual
s - Suspended
r - Module-removed
S - Switched
R - Routed
-------------------------------------------------------------------------------Group Port-
Type
Protocol
Member Ports
Channel
-------------------------------------------------------------------------------50
Po50(SU)
Eth
LACP
Eth1/6(P)
Eth1/7(P)
Eth1/9(P)
N5K1# show interface E1/6 - 9 | include Ethernet1|rate
Eth1/8(P)
Ethernet1/6 is up

, 22.35 Kpps
Ethernet1/7 is up

, 23.81 Kpps
Ethernet1/8 is up

, 22.03 Kpps
Ethernet1/9 is up
input rate 962.17 Kbps, 1.75 Kpps;

, 22.37 Kpps
When the 7Ks receive the flows in, they likewise load-balance them back down to
the 5Ks after the routing process occurs on the layer 3 VLAN interfaces. Note that if
the traffic was Intra-VLAN, the flows would not need to go all the way north to the
7Ks, but instead would be directly locally switched by the 5Ks. Also note that the
distribution of flows is not equal from the 7Ks back down to the 5Ks, simply because
of the uneven hashing results of the flows' input src/dst IP addresses and TCP
ports. Over a longer-term average and with a more diverse distribution of flows, the
distribution of traffic among the vPC member ports would be more even.
N7K1# show port-channel summary interface port-channel 70
Flags:
D - Down
I - Individual
s - Suspended
r - Module-removed
S - Switched
R - Routed
-------------------------------------------------------------------------------Group Port-
Type
Protocol
Member Ports
Channel
-------------------------------------------------------------------------------70
Po70(SU)
Eth
LACP
Eth2/3(P)
Eth2/4(P)
Eth2/6(P)
N7K1# show interface e2/3 - 6 | include Ethernet2|rate
Eth2/5(P)
Ethernet2/3 is up

, 20.76 Kpps
Ethernet2/4 is up

, 10.81 Kpps
Ethernet2/5 is up
input rate 696 bps, 0 pps;

, 6.00 Kpps
Ethernet2/6 is up

, 9.05 Kpps
N7K2# show port-channel summary interface port-channel 70
Flags:
D - Down
I - Individual
s - Suspended
r - Module-removed
S - Switched
R - Routed
-------------------------------------------------------------------------------Group Port-
Type
Protocol
Member Ports
Channel
-------------------------------------------------------------------------------70
Po70(SU)
Eth
LACP
Eth2/3(P)
Eth2/4(P)
Eth2/5(P)
Eth2/6(P)
Ethernet2/3 is up

, 37.38 Kpps
Ethernet2/4 is up
, 2.02 Kpps
Ethernet2/5 is up

, 3.68 Kpps
Ethernet2/6 is up
, 1.72 Kpps

Channels (vPC)
vPC Peer Switch
Task
Configure a vPC between N7K1, N7K2, and N5K1 as follows:
51, and as vPC 51.
All port channels should be trunks, and use LACP for negotiation.
Create VLAN 10 on N7K1, N7K2, and N5K1, and set the spanning-tree priority to
4096 on both N7K1 and N7K2 for this VLAN.
Using the vPC peer-switch feature, configure N7K1 and N7K2 so that they both
share the STP root bridge role for VLAN 10.
Configuration
N5K1:
feature lacp
!
vlan 10
!
no shutdown
N7K1-1:
feature vpc
feature lacp
!
vlan 10
spanning-tree vlan 10 priority 4096

!
vpc domain 1
peer-switch
!
no shutdown
!
switchport
vpc peer-link
!
no shutdown
!
switchport
vpc 51
N7K2-1:
feature vpc
feature lacp
!
vlan 10
!
vpc domain 1
peer-switch
!
no shutdown
!
switchport

vpc peer-link
!
no shutdown
!
switchport
vpc 51
Verification
Before applying the peer-switch feature, both N7K1 and N7K2 share the same STP
priority value, but the switch with the lower MAC address portion of their STP BridgeID is elected the root bridge. This is the normal expected behavior of STP.
N7K1-1# show spanning-tree vlan 10
VLAN0010
Address
Cost
Port
Hello Time
sec
4106
Max Age 20 sec
68bd.abd7.6041
Hello Time
Interface
4106
0026.980c.2141
Bridge ID Priority
Address
Root ID Priority
sec
Max Age 20 sec
Role Sts Cost
Prio.Nbr Type
---------------- ---- --- --------- -------- -------------------------------Po1
Root FWD 1
Po51
Desg FWD 1
128.4146 (vPC) P2p
VLAN0010
Address
Root ID Priority
4106
0026.980c.2141

Hello Time
Bridge ID Priority
2
4106
sec
Max Age 20 sec
Address
0026.980c.2141
Hello Time
Interface
sec
Max Age 20 sec
Role Sts Cost
Prio.Nbr Type
---------------- ---- --- --------- -------- -------------------------------Po1
Desg FWD 1
Po51
Desg FWD 1
128.4146 (vPC) P2p
N5K1# sh spanning-tree vlan 10
VLAN0010
Root ID Priority
Address
Bridge ID
Interface
Cost
Port
Hello Time
Priority
32778
Address
547f.ee79.137c
Hello Time
sec
Max Age 20 sec
0026.980c.2141
sec
Max Age 20 sec
Role Sts Cost
Prio.Nbr Type
---------------- ---- --- --------- -------- -------------------------------Po51
Root FWD 1
128.4146 P2p
After enabling the peer-switch feature, both N7K1 and N7K2 share the same STP
Bridge-ID, so both see themselves as the STP Root Bridge. This is an exception to
normal STP rules that is specific to the vPC.
vN7K1-1# show spanning-tree vlan 10
VLAN0010
Address
Root ID Priority
4106
0023.04ee.be01

Hello Time
Bridge ID Priority
Address
sec
4106
Max Age 20 sec
0023.04ee.be01
Hello Time
Interface
sec
Role Sts Cost
Max Age 20 sec
Prio.Nbr Type
---------------- ---- --- --------- -------- -------------------------------Po1
Root FWD 1
4106
Po51
Desg FWD 1
128.4146 (vPC) P2p
VLAN0010
Address
Root ID Priority
4106
0023.04ee.be01

Hello Time
Bridge ID Priority
Address
sec
4106
Max Age 20 sec
0023.04ee.be01
Hello Time
Interface
sec
Role Sts Cost
Max Age 20 sec
Prio.Nbr Type
---------------- ---- --- --------- -------- -------------------------------Po1
Desg FWD 1
Po51
Desg FWD 1
128.4146 (vPC) P2p
N5K1# sh spanning-tree vlan 10
VLAN0010
Address
Bridge ID
Interface
Root ID Priority
4106
0023.04ee.be01
Cost
Port
Hello Time
Priority
32778
Address
547f.ee79.137c
Hello Time
sec
sec
Role Sts Cost
Max Age 20 sec
Max Age 20 sec
Prio.Nbr Type
---------------- ---- --- --------- -------- -------------------------------Po51
Root FWD 1
128.4146 P2p

Channels (vPC)
vPC Customization
Task
51, and as vPC 51.
All port channels should be trunks and use LACP for negotiation.
Customize the vPC configuration on N7K1 and N7K2 as follows:
The vPC peers should use the vPC system MAC address of
1234.5678.9ABC and an LACP system priority of 1000.
N7K1 should be the primary vPC peer with a role priority of 10, and N7K2
should be the secondary vPC peer with a role priority of 20.
In the case that one of the vPC peers is reloaded, vPCs should be delayed
from coming up for 60 seconds after the vPC peer adjacency is reestablished.
In the case that a Type 1 mismatch occurs for a particular VLAN, that VLAN
should be suspended on the vPCs of both the primary and the secondary
vPC peers.
In the case that the vPC Peer Link flaps, both the IPv4 ARP Cache and IPv6
ICMP ND Cache should be syncronized between the vPC peers upon
reconvergence of the vPC Peer Link.
In the case that both vPC Peers fail but only one of them is able to be
restored, the newly restored peer should wait five minutes before declaring
itself the active vPC Peer and bringing its vPCs up.
Configuration
N5K1:
feature lacp
!
no shutdown
N7K1-1:
feature vpc
feature lacp
!
vpc domain 1
role priority 10
system-mac 12:34:56:78:9a:bc
system-priority 1000
delay restore 60
no graceful consistency-check
auto-recovery reload-delay 300
ipv6 nd synchronize
ip arp synchronize
!
no shutdown
!
switchport
vpc peer-link
!
no shutdown
!
switchport
vpc 51
N7K2-1:
feature vpc
feature lacp
!
vpc domain 1
role priority 20
system-mac 12:34:56:78:9a:bc
system-priority 1000
delay restore 60
no graceful consistency-check
auto-recovery reload-delay 300
ipv6 nd synchronize
ip arp synchronize
!
no shutdown
!
switchport
vpc peer-link
!
no shutdown
!
switchport
vpc 51
Verification
N5K1# show lacp neighbor interface port-channel 51
Flags:
S - Device is sending Slow LACPDUs F - Device is sending Fast LACPDUs

A - Device is in Active mode
port-channel51 neighbors
P - Device is in Passive mode
Partner's information
Port
Partner
Partner
System ID
Port Number
0x4203
2590
Partner
Age
Flags Eth1/6
1000,12-34-56-78-9a-bc
SA
LACP Partner
Partner
Partner
Port Priority
Oper Key
Port State
32768
0x8033
0x3d
Partner
Partner
Partner
System ID
Port Number
Port
0x4204
2590
Age
Flags Eth1/7
1000,12-34-56-78-9a-bc
SA
LACP Partner
Partner
Partner
Port Priority
Oper Key
Port State
32768
0x8033
0x3d
Partner
Partner
Partner
System ID
Port Number
Port
0x205
2591
Age
Flags Eth1/8
1000,12-34-56-78-9a-bc
SA
LACP Partner
Partner
Partner
Port Priority
Oper Key
Port State
32768
0x8033
0x3d
Partner
Partner
Partner
System ID
Port Number
Port
0x206
2590
Age
Flags Eth1/9
1000,12-34-56-78-9a-bc
SA
LACP Partner
Partner
Partner
Port Priority
Oper Key
Port State
32768
0x8033
0x3d
N7K1-1# show vpc role
vPC Role status

---------------------------------------------------- vPC role
Dual Active Detection Status
: 0 vPC system-mac
N7K2-1# show vpc role
: 12:34:56:78:9a:bc
: 1000
vPC system-priority
vPC local system-mac
: primary
: 68:bd:ab:d7:60:41
vPC local role-priority
: 10
vPC Role status

---------------------------------------------------- vPC role
Dual Active Detection Status
: secondary
: 0 vPC system-mac
vPC system-priority
: 12:34:56:78:9a:bc
vPC local system-mac
: 1000
: 00:26:98:0c:21:41
vPC local role-priority
N7K1-1# show vpc

Legend:
vPC domain id
: 1
Peer status
: peer is alive
: success
: success
: success
vPC role
: primary
: 1
Peer Gateway
: Disabled
: - Graceful Consistency Check
: Enabled (timeout = 300 seconds)

--------------------------------------------------------------------id
Port
Status Active vlans
--
----
------ --------------------------------------------------
Po1
up
vPC status
---------------------------------------------------------------------id
Port
Active vlans
--
----
------ ----------- ------
------------
51
Po51
up
success
success
N7K1-1# show ip arp vpc-statistics

ARP sync Enabled
ARP vPC global statistics

Ignore to send pull request using CFSoE as the feature is off : 6
Ignore to send push message using CFSoE as the feature is off : 8
Response sent via CFSoE : 1
: Disabled
: 20
Received message via CFSoE : 1

Channels (vPC)
Nexus 7K Virtual Port Channels (vPC)
Task
51, and as vPC 51.
All port channels should be trunks, and use LACP for negotiation.
Configuration
N5K1:
feature lacp
!
no shutdown
N7K1-1:
feature vpc
feature lacp
!
vpc domain 1
!
no shutdown
!
switchport
vpc peer-link
!
no shutdown
!
switchport
vpc 51
N7K2-1:
feature vpc
feature lacp
!
vpc domain 1
!
no shutdown
!
switchport
vpc peer-link
!
no shutdown
!
switchport
vpc 51
Verification
Flags:
D - Down
I - Individual
s - Suspended
r - Module-removed
S - Switched
R - Routed
-------------------------------------------------------------------------------Group Port-
Type
Protocol
Member Ports
Channel
-------------------------------------------------------------------------------- 51 Po51(SU)
Eth
LACP Eth1/6(P)
Eth1/7(P)
Eth1/8(P)
Eth1/9(P)
N7K1-1# show vpc

Legend:
vPC domain id
: 1
Peer status
: peer is alive
: success
: success
Type-2 inconsistency reason
: Consistency Check Not Performed
vPC role
: secondary
: 1
Peer Gateway
: Disabled
: -
: Enabled
: Disabled

--------------------------------------------------------------------id
--
Port
Status Active vlans
----
------ -------------------------------------------------- 1
Po1 up
vPC status
---------------------------------------------------------------------id
Port
Active vlans
--
----
------ ----------- ------
------------ 51
success
Po51 up
success
N7K2-1# show vpc

Legend:
vPC domain id
: 1
Peer status
: peer is alive
: success
: success
vPC role
: primary
: 1
Peer Gateway
: Disabled
: -
: Enabled
: Disabled

--------------------------------------------------------------------id
Port
Status Active vlans
--
----
------ -------------------------------------------------- 1
Po1 up
vPC status
---------------------------------------------------------------------id
Port
Active vlans
--
----
------ ----------- ------
------------ 51
success
Po51 up
success

Channels (vPC)
Nexus 7K Virtual Port Channels (vPC) with Layer 3
Routed Keepalive
Task
Configure all available M1 ports between the vPC peers as Port-Channel 2,
and use this as the Peer Keepalive Link. This link should be a native layer 3
routed interface in its own VRF.
51, and as vPC 51.
All layer 2 port channels should be trunks, and all channels should use
LACP for negotiation.
Configuration
N5K1:
feature lacp
!
no shutdown
N7K1-1:
feature vpc
feature lacp
!
vrf context VPC_KEEPALIVE
!
vpc domain 1
peer-keepalive destination 169.254.0.2 source 169.254.0.1 vrf VPC_KEEPALIVE

!
switchport
vpc peer-link
!
vrf member VPC_KEEPALIVE
ip address 169.254.0.1/30
!
switchport
vpc 51
!
no shutdown
!
no shutdown
!
no shutdown
N7K2-1:
feature vpc
feature lacp
!
!
vpc domain 1
!
switchport
vpc peer-link
!
ip address 169.254.0.2/30
!
switchport
vpc 51
!
no shutdown
!
no shutdown
!
no shutdown
Verification
Flags:
D - Down
I - Individual
s - Suspended
r - Module-removed
S - Switched
R - Routed
-------------------------------------------------------------------------------Group Port-
Type
Protocol
Member Ports
Channel
-------------------------------------------------------------------------------- 51 Po51(SU)
Eth
LACP Eth1/6(P)
Eth1/7(P)
Eth1/8(P)
Eth1/9(P)
N7K1-1# show vpc

Legend:
vPC domain id
: 1
Peer status

: peer is alive
: success
: success
vPC role
: secondary
: 1
Peer Gateway
: Disabled
: -
: Enabled
: Disabled

--------------------------------------------------------------------id
Port
Status Active vlans
--
----
------ -------------------------------------------------- 1
Po1 up
vPC status
---------------------------------------------------------------------id
Port
Active vlans
--
----
------ ----------- ------
------------ 51
success
Po51 up
success
N7K2-1# show vpc

Legend:
vPC domain id
: 1
Peer status
: peer is alive
: success
: success
vPC role
: primary
: 1
Peer Gateway
: Disabled
: -
: Enabled
: Disabled

--------------------------------------------------------------------id
Port
Status Active vlans
--
----
------ --------------------------------------------------
Po1 up
1
vPC status
---------------------------------------------------------------------id
Port
Active vlans
--
----
------ ----------- ------
------------ 51
success
Po51 up
N7K1-1# show port-channel summary

Flags:
D - Down
I - Individual
s - Suspended
r - Module-removed
S - Switched
R - Routed
-------------------------------------------------------------------------------Group Port-
Type
Protocol
Member Ports
Channel
-------------------------------------------------------------------------------1
Po1(SU)
Po2(RU)
51
Eth
Eth
Po51(SU)
LACP
LACP
Eth
Eth2/1(P)
Eth1/1(P)
LACP
Eth2/2(P)
Eth1/2(P)
Eth2/3(P)
Eth2/4(P)
N7K1-1# show ip route vrf VPC_KEEPALIVE

IP Route Table for VRF "VPC_KEEPALIVE"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>
169.254.0.0/30
, ubest/mbest: 1/0, attached
* via 169.254.0.1, Po2
, [0/0], 00:12:45, direct

169.254.0.1/32, ubest/mbest: 1/0, attached
*via 169.254.0.1, Po2, [0/0], 00:12:45, local
N7K1-1# ping 169.254.0.2 vrf VPC_KEEPALIVE
PING 169.254.0.2 (169.254.0.2): 56 data bytes

64 bytes from 169.254.0.2: icmp_seq=0 ttl=254 time=1.125 ms
success

Channels (vPC)
Nexus 7K Virtual Port Channels (vPC) with SVI
Keepalive
Task
Configure all available M1 ports between the vPC peers as a layer 2 trunk,
and as Port-Channel 2.
Configure a layer 3 VLAN 999 interface for the Peer Keepalive of the vPC
peers. This SVI should be in its own VRF, and should not forward over the
vPC Peer Link or vPC member ports.
51, and as vPC 51.
All layer 2 port channels should be trunks, and all channels should use
LACP for negotiation.
Configuration
N5K1:
feature lacp
!
no shutdown
N7K1-1:
feature lacp
feature vpc
!

!
vlan 1,999
!
vpc domain 1
!
interface Vlan999
no shutdown
ip address 169.254.0.1/30
!
switchport
switchport trunk allowed vlan 1-998,1000-4094
vpc peer-link
!
switchport
!
switchport
vpc 51
!
switchport
no shutdown
!
no shutdown
!
no shutdown
N7K2-1:
feature lacp
feature vpc
!
!
vlan 1,999
!
vpc domain 1
!
interface Vlan999
no shutdown
ip address 169.254.0.2/30
!
switchport
vpc peer-link
!
switchport
!
switchport
vpc 51
!
switchport
no shutdown
!
no shutdown
!
no shutdown
Verification
N7K1-1# show vpc
Legend:
vPC domain id
: 1
Peer status
: peer is alive
: success
: success
: success
vPC role
: secondary
: 1
Peer Gateway
: Disabled
: -
: Enabled
: Disabled

--------------------------------------------------------------------id
Port
Status Active vlans
--
----
------ --------------------------------------------------
Po1
up
vPC status
---------------------------------------------------------------------id
Port
Active vlans
--
----
------ ----------- ------
------------
51
Po51
up
success
success
N7K1-1# show vpc peer-keepalive
: peer is alive
--Peer is alive for
: (1825) seconds, (714) msec
--Send status
: Success
--Last send at
: 2013.02.12 19:01:15 528 ms
--Sent on interface
: Vlan999
--Receive status
: Success
--Last receive at
: 2013.02.12 19:01:15 941 ms
--Received on interface
: Vlan999
--Last update from peer
: (0) seconds, (221) msec
vPC Keep-alive parameters

--Destination
: 169.254.0.2
--Keepalive interval
: 1000 msec
--Keepalive timeout
: 5 seconds
--Keepalive hold timeout
: 3 seconds
--Keepalive vrf
: VPC_KEEPALIVE
--Keepalive udp port
: 3200
--Keepalive tos
: 192
Note that VLAN 999 does not forward over the vPC Peer Link or the vPC member
ports because VLAN 999 was removed from the trunking allowed list.
VLAN0999
Root ID
Bridge ID
Interface
Priority
33767
Address
0026.980c.2141
Cost
Port
Hello Time
Priority
33767
Address
68bd.abd7.6041
Hello Time
sec
sec
Max Age 20 sec
Max Age 20 sec
Role Sts Cost
Prio.Nbr Type
---------------- ---- --- --------- -------- -------------------------------Po2
Root FWD 1
128.4097 P2p
N7K1-1# show interface trunk
<snip>
-------------------------------------------------------------------------------Port
Vlans Allowed on Trunk
-------------------------------------------------------------------------------Eth1/1
1-4094
Eth1/2
1-4094
Eth2/1
1-998,1000-4094
Eth2/2
1-998,1000-4094
Eth2/3
1-998,1000-4094
Eth2/4
1-998,1000-4094
Po1
1-998,1000-4094
Po2
1-4094
Po51
1-998,1000-4094
<snip>
-------------------------------------------------------------------------------Port
Vlans in spanning tree forwarding state and not pruned
-------------------------------------------------------------------------------Eth1/1
none
Eth1/2
none
Eth2/1
none
Eth2/2
none
Eth2/3
none
Eth2/4
none
Po1
Po2
999
Po51
<snip>

Channels (vPC)
Active Standby NIC Teaming without vPC
Task
Configure the links connecting N5K1 and N5K2 as Port-Channel1.
This port channel should be an 802.1q trunk link and be configured as an STP
Network Port.
Configure VLAN 10 on N5K1 and N5K2.
Assign VLAN 10 as the access VLAN for all of N5K1 and N5K2's connections to
Server 1 and Server 2, and configure these as STP Edge Ports.
Configure Active/Standby NIC Teaming on Server 1 and Server 2 as follows:
Server 1 should use the link to N5K1 as primary, and the link to N5K2 as a
backup.
Server 2 should use the link to N5K2 as primary, and the link to N5K1 as a
backup.
Server 1 should use the IP address 10.0.0.1/24, and Server 2 should use
10.0.0.2/24.
When complete, ensure that Server 1 and Server 2 have IP connectivity to each
other, and that they maintain connectivity if either of the primary links of the NIC team
goes down.
Configuration
N5K1:
vlan 10
!
speed 10000
!

speed 1000
!
speed 1000
!
channel-group 1
!
channel-group 1
!
channel-group 1
N5K2:
vlan 10
!
speed 10000
!
speed 1000
!
speed 1000
!
channel-group 1
!
channel-group 1
!
channel-group 1
Verification
In this design, the end servers are dual attached to separate access switches, N5K1
and N5K2. Because the 5Ks do not have vPC configured, the end host cannot form
a port channel to the switches. Instead, the NIC Teaming software of the server is
configured to use one link as primary and the other link as backup. This is then
considered an Active/Standby design, because only one link can forward at a time.
From the network point of view, there is no special configuration. The links
connecting to the servers are simply configured as normal access ports or trunk
ports, and then the load distribution is up to the end host.
In this case specifically, Server 1 and Server 2 both have Intel NICs, so the teaming
is configured under the NIC properties and then the Teaming tab to get to the Intel
Advanced Networking Services (ANS) properties.
Create the team and add the physical adapters to it, and then define the team type
as Fault Tolerance. This is what Intel ANS calls Active/Standby teaming, in which
one link of the team forwards and the other links are backup.
Go back to the NIC Team properties and choose the adapter that you want to be the
primary or secondary.
IP configuration goes under the new logical Team adapter, similar to how NX-OS or
IOS uses the logical Port-Channel interface to configure properties of the physical
members of a channel.
After connectivity is established between the servers, you can generate bulk TCP or
UDP flows by using the iPerf application. In the following output, Server 2 is listening
for a TCP connection from Server 1, and Server 1 is generating a TCP flow near its
line rate of 1Gbps. Note that the flows do not total 2Gbps, as only one adapter in the
NIC Team is actively forwarding.
From the network side, we can verify which links are being used by checking the
interface counters. Below we see that N5K1 is receiving the near 1Gbps flow in from
Server 1 on Ethernet1/1.
Ethernet1/1
is up
, 78.59 Kpps; output rate 3.59 Mbps, 6.94 Kpps
Ethernet1/2 is up

input rate 432 bps, 0 pps; output rate 1.61 Kbps, 2 pps
N5K1 then sends this flow over the trunk to N5K2, Port-Channel1.
N5K1# show interface port-channel1 | include rate

, 78.28 Kpps
N5K2 then forwards the traffic to Server 2 via the local link Ethernet1/2.
Ethernet1/1 is up
Ethernet1/2 is up

, 77.93 Kpps
The key point about the above output is that the standby links to the servers,
specifically E1/2 on N5K1 and E1/1 on N5K2, are essentially unused. Regardless of
the amount of flows being sent between the servers, they cannot exceed 1Gbps as
an aggregate, because only one of the physical links of the team is being used. The
only real advantage of this type of design is that there is fault tolerance for the NICs.
If N5K1s link to Server 1 goes down, as shown below, the traffic will re-route to
N5K2.
N5K1# config
End with CNTL/Z.
N5K1(config)# int e1/1

2013 Mar
2013 Mar

Ethernet1/1
is up
, 77.65 Kpps; output rate 3.52 Mbps, 6.87 Kpps Ethernet1/2
is up


, 77.65 Kpps
Now that the connection from Server 1 to N5K1 is down, N5K2 receives all traffic
between the servers.

Channels (vPC)
Active Active NIC Teaming with vPC
Task
Configure a vPC Domain between N5K1 and N5K2 as follows:
Create vPC Domain 1 on the peers, and use the mgmt0 ports for the vPC
Peer Keepalive Link.
The vPC Peer Link should use LACP negotiation, be an 802.1q trunk link,
and be an STP Network Port.
Configure vPCs from N5K1 and N5K2 to Server 1 and Server 2 as follows:
Configure N5K1 and N5K2's links to Server 1 as Port-Channel 101.
Port-Channel 101 should be configured as an access port in VLAN 10, an
STP Edge Port, and as vPC 101.
Configure N5K1 and N5K2's links to Server 2 as Port-Channel 102.
Port-Channel 102 should be configured as an access port in VLAN 10, an
STP Edge Port, and as vPC 102.
Configure Active/Active NIC Teaming on Server 1 and Server 2 as follows:
Configure a NIC Team on Server 1 using 802.3ad (LACP); both links to
N5K1 and N5K2 should be in this team, and it should use the IP address
10.0.0.1/24.
Configure a NIC Team on Server 2 using 802.3ad (LACP); both links to
N5K1 and N5K2 should be in this team, and it should use the IP address
10.0.0.2/24.
When complete, ensure that Server 1 and Server 2 have IP connectivity to each
other, and that traffic between them uses both uplinks to N5K1 and N5K2
simultaneously.
Configuration
N5K1:
feature lacp
feature vpc
!
vlan 10
!
vpc domain 1
!
vpc peer-link
!
vpc 101
!
vpc 102
!
speed 1000
!
speed 1000
!

N5K2:
feature lacp
feature vpc
!
vlan 10
!
vpc domain 1
!
vpc peer-link
!
vpc 101
!
vpc 102
!
speed 1000
!
speed 1000
!
Verification
In this design, the end servers are dual attached to separate access switches, N5K1
and N5K2. Additionally, N5K1 and N5K2 are configured for Virtual Port Channel
(vPC), which is a type of Multi-Chassis EtherChannel (MEC). vPC means that the
downstream devices, Server 1 and Server 2 in this case, see the upstream switches
(the vPC Peers) as a single switch. In other words, while the physical topology is a
triangle, the logical topology is a point-to-point port channel.
vPC configuration is made up of three main components, the vPC Peer Keepalive
Link, the vPC Peer Link, and the vPC Member Ports. The vPC Keepalive Link is any
layer 3 interface, including the mgmt0 port, that is used to send UDP pings between
the vPC peers. If the UDP ping is successful over the keepalive link, the peers are
considered to be reachable. The second portion, the vPC Peer Link, is used to
synchronize the control plane between the vPC Peers. The Peer Link is used for
operations such as MAC address table synchronization, ARP table synchronization,
IGMP Snooping synchronization, and so on. The Peer Link is a port channel made
up of at least two 10Gbps links, and it should be configured as a layer 2 trunk link
that runs as STP port type network. The final portions, the vPC member ports, are
the port channel interfaces that go down the end hosts or downstream devices.
The first step in vPC verification is to ensure that the vPC Peer Keepalive is up and
that the vPC Peer Link is up, as shown below.
N5K1# show vpc
Legend:
vPC domain id
: 1
Peer status
: peer is alive

: success
: success
vPC role
: primary
: 2
Peer Gateway
: Disabled
: -
: Enabled

---------------------------------------------------------------------
id
Port
Status Active vlans
--
----
------ --------------------------------------------------
Po1
up
1,10
<snip>
Next, the vPC Member Ports are configured to the end hosts. In the output below,
Port-Channel101 to Server 1 shows its vPC as down, because the vPC has been
configured on the switch side but not yet on the server side. The end result is that
the link runs as a normal access port, as indicated by the Individual flag of the
show port-channel summary.
N5K1# show vpc 101
vPC status
---------------------------------------------------------------------------id
Port
Active vlans
------ ----------- ------ ----------- -------------------------- ----------- 101

*
success
success
Po101

Flags:
D - Down
I - Individual
s - Suspended
r - Module-removed
S - Switched
R - Routed
-------------------------------------------------------------------------------Group Port-
Type
Protocol
Member Ports
Channel
-------------------------------------------------------------------------------1
Po1(SU)
Eth
LACP
Eth1/3(P)
101
Po101(SD)
Eth
LACP
Eth1/1(I)
102
Po102(SU)
Eth
LACP
Eth1/2(P)
Eth1/4(P)
Eth1/5(P)
Next, the end server is configured for NIC Teaming. In the case of the Intel ANS
software, an LACP-based channel is called 802.3ad Dynamic Link Aggregation.
down
After the server signals the switch with LACP, the channel can form and the vPC
comes up, as shown below.
N5K1#
2013 Mar
3 18:58:39 N5K1 %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet1/1 is down (Initializing)
2013 Mar
3 18:58:39 N5K1 %ETH_PORT_CHANNEL-5-PORT_INDIVIDUAL_DOWN: individual port Ethernet1/1 is down
2013 Mar
3 18:58:39 N5K1 %ETHPORT-5-SPEED: Interface port-channel101, operational speed changed to 1 Gbps
2013 Mar
3 18:58:39 N5K1 %ETHPORT-5-IF_DUPLEX: Interface port-channel101, operational duplex mode changed to Full
2013 Mar
3 18:58:39 N5K1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface port-channel101, operational Receive Flow Control
2013 Mar
3 18:58:39 N5K1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface port-channel101, operational Transmit Flow Contro
2013 Mar
3 18:58:42 N5K1 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel101: Ethernet1/1 is up
N5K1# 2013 Mar
3 18:58:51 N5K1 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel101: first operational port changed fro
2013 Mar
3 18:58:51 N5K1 %ETHPORT-5-IF_UP: Interface Ethernet1/1 is up in mode access
2013 Mar
3 18:58:51 N5K1 %ETHPORT-5-IF_UP: Interface port-channel101 is up in mode access
N5K1# show vpc 101
vPC status
---------------------------------------------------------------------------id
Port
Active vlans
------ ----------- ------ ----------- -------------------------- ----------- 101

success
success
Po101
up
10
The IP configuration of the server goes on the logical NIC Team interface, similar to
how NX-OS and IOS use the logical Port-Channel interface to reference the
physical members of the channel.
Testing the traffic flows over the vPC in the data plane becomes a little difficult in
this case. Each device that has a port channel configured ultimately controls the
decision of how its outbound traffic flows. For example, if a traffic flow is moving
from Server 1 to Server 2, Server 1 first determines which links to send the flows out
on, and then the upstream switches choose which outbound links to send the flows
out on, until the final destination is reached. This is an issue because you will not
see an even distribution of traffic among the NIC Team and vPC Member Ports
unless there is a sufficiently large number of flows from diverse source and
destination addresses. Although the port-channel load balancing method can be
changed on the Nexus switches, it cannot be changed in the Intel NIC drivers in this
design. Therefore, to fully verify that Active/Active forwarding is working, we need
more than one destination address to send to. This is achieved below by configuring
a secondary IP address on the NIC Team of Server 1.
Next, Server 2 is configured to send separate UDP flows to each of the addresses
on Server 1 with the iPerf app, as shown below.
On the network side, the traffic flows in the data plane can be verified by looking at
the interface counters of the vPC Member Ports. If the input bandwidth counter from
Server 2 is split between both N5K1 and N5K2, we would then know that Server 2 is
distributing the load between both members of its NIC Team in an Active/Active
manner. Furthermore, if we see that the output bandwidth counters from N5K1 and
N5K2 to Server 1 is split between them, we would also know that the switches are
doing Active/Active forwarding to the destination. This can be seen in the output
below.
N5K1# show interface e1/1-2 | in rate|Ethernet
Ethernet1/1 is up

, 926 pps
Ethernet1/2 is up
, 926 pps; output rate 947.38 Kbps, 199 pps
N5K2# show interface e1/1-2 | in rate|Ethernet
Ethernet1/1 is up
Hardware: 1000/10000 Ethernet, address: 000d.eca4.7408 (bia 000d.eca4.7408)

, 975 pps
Ethernet1/2 is up
Hardware: 1000/10000 Ethernet, address: 000d.eca4.7409 (bia 000d.eca4.7409)
input rate 6.21 Mbps

, 975 pps; output rate 144 bps, 0 pps
Note that on N5K1 the input rate of E1/2, which connects to Server 2, matches the
output rate of E1/1, which connects to Server 1. Likewise, on N5K2 the input rate of
E1/2, which connects to Server 2, matches the output rate of E1/1, which connects
to Server 1. Also note that these traffic flows do not cross the vPC Peer Link
between the Nexus 5Ks, because this link is excluded from the data plane under
normal correct operations. Verification of the counters of Port-Channel1, the vPC
Peer Link, show little to no traffic being sent or received on the port.
N5K1# show interface port-channel 1 | include rate

The output shown above indicates the normal forwarding logic of vPC, which is that
the vPC Peer will first attempt to forward traffic to a local vPC Member Port instead
of crossing the vPC Peer Link. The only time that this rule is normally broken for
known unicast traffic is if the local vPC Member Port is down. For example, if a
failure occurs between N5K1 and Server 1, N5K1s traffic received from Server 1
going to Server 2 must be sent over the vPC Peer Link; otherwise it would be
blackholed. This can be seen below.
Normally this detection is immediate based on link failure, but in this topology design
Server 1 is a Virtual Machine that is not directly physically connected to N5K1.
When the LACP timer expires, N5K1 detects that the vPC peer is gone, and the
vPC Member Port goes down.
N5K1#
2013 Mar
2013 Mar
3 22:54:34 N5K1 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel101: port-channel101 is down
<snip>
N5K1# show vpc
Legend:
vPC domain id
: 1
Peer status
: peer is alive

: success
: success
vPC role
: primary
: 2
Peer Gateway
: Disabled
: -
: Enabled

--------------------------------------------------------------------id
Port
Status Active vlans
--
----
------ --------------------------------------------------
Po1
up
1,10
vPC status
---------------------------------------------------------------------------id
Port
Active vlans
------ ----------- ------ ----------- -------------------------- ----------- 101

success
102
Po102
success
up
Po101
success
success
10
Now any traffic that comes in on N5K1 from Server 2 that is going toward Server 1
must transit the vPC Peer Link.
down*
N5K1# show interface port-channel 1 | include rate


, 856 pps
This situation normally only happens during a failure event. It is highly undesirable
for vPC because the vPC Peer Link is usually much lower bandwidth (such as 20
Gbps) than the aggregate of the vPC Member Ports (such as 400Gbps+, depending
on port density), so the Peer Link can quickly become overwhelmed if it needs to be
used in the data plane.
Nexus Technology Labs - Storage Area

Network (SAN)
FCoE QoS
Task
Configure N5K1 to pair with the Fabric Extender N2K1 using FEX number 101.
Configure FCoE support for the Emulex CNA server on N5K1 as follows:
Enable the FCoE feature set on N5K1.
Create VSAN 101 on N5K1.
Configure VLAN 1101 on N5K1, and map it to the FCoE VSAN 101.
Configure N5K1's link to the Emulex CNA server as an 802.1q trunk and as
STP port type edge trunk.
Configure VLAN 10 on N5K1, which will be the server's access VLAN for
regular LAN traffic (that is, not FCoE traffic).
Configure N5K1's link to the CNA server with VLAN 10 as the native VLAN,
and limit the trunk link to only VLANs 10 and 1101.
Create interface Virtual Fibre Channel 111 on N5K1, and bind this to N5K1's
link to the CNA server.
Assign this interface VFC111 to VSAN 101, and edit the trunking allowed list
so that it only includes VSAN 101.
Modify the QoS policy on N5K1 so that the FCoE traffic to and from the
Emulex CNA server is reserved 75% of the link bandwidth, while normal
LAN traffic is reserved 25%.
When complete, verify that the Emulex CNA server is registered to the fabric and has
learned the QoS policy from the N5K.
Configuration
N5K1:
feature fex
feature fcoe
!
vlan 10
vlan 1101
fcoe vsan 101
!
vsan database
vsan 101
!
fex associate 101
!
switchport trunk native vlan 10
switchport trunk allowed vlan 10,1101
spanning-tree port type edge trunk
no shutdown
!
interface vfc111
bind interface Ethernet101/1/1
switchport trunk allowed vsan 101
no shutdown
!
policy-map type queuing FCoE_75_PERCENT_POLICY
class type queuing class-fcoe
bandwidth percent 75
class type queuing class-default
!
system qos
service-policy type queuing input FCoE_75_PERCENT_POLICY
service-policy type queuing output FCoE_75_PERCENT_POLICY
Verification
One important behavioral difference between native Fibre Channel and Ethernet
networks is that Fibre Channel is by design a lossless protocol and Ethernet is not.
To provide guaranteed delivery, Ethernet networks rely on upper-layer protocols
such as TCP to do flow-control and retransmission. In Fibre Channel networks,
these functions are built directly into the equivalent of the layer 2 transport.
Therefore, to meet the application requirements of running SCSI reads and writes
over an Ethernet transport (FCoE), Ethernet had to be extended to allow for
guaranteed delivery.
These Ethernet extensions are grouped together under the terms Data Center
Ethernet (DCE), Data Center Bridging (DCB), or Converged Enhanced Ethernet
(CEE), depending on which vendors documentation you are reading. All of these
terms essentially mean the same thing, which in one form or another is QoS support
for FCoE traffic. The relevant standards that emerged from these protocols are
802.1Qbb Priority-based Flow Control (PFC), 802.1Qaz Enhanced Transmission
Selection (ETS) & Data Center Bridging Capabilities Exchange Protocol (DCBX),
and 802.1Qau Quantized Congestion Notification (QCN). Of these, DCBX is the
negotiation protocol that runs between the FCoE-capable switch (ithe FCF) and the
end host (the ENode), PFC provides the pause functionality so that FCoE doesnt
get dropped, ETS provides the QoS scheduling (similar to CBWFQ), and QCN
performs end-to-end flow control. Ciscos NX-OS implementation of FCoE supports
DCBX, PFC, and ETS, but not QCN.
From an implementation point of view, we must ensure that all FCFs and ENodes
have DCBX enabled, and that through DCBX they are able to negotiate PFC and
ETS. The actual signaling of the DCBX exchanges happens through the Link Layer
Discovery Protocol (LLDP), which is essentially the open standard version of CDP.
This means that the first step of FCoE support is that the Nexus switch and the end
hosts CNA become LLDP adjacent, as shown below.
N5K1# show lldp neighbors interface e101/1/1
Capability codes:
(R) Router, (B) Bridge, (T) Telephone, (C) DOCSIS Cable Device
(W) WLAN Access Point, (P) Repeater, (S) Station, (O) Other
Device ID
Local Intf
Hold-time
OCe11102-FX FWVer:4.1.402.18Eth101/1/1
120
Capability
S
Port ID
0000.c9bb.199f
Total entries displayed: 1
The device ID shown above indicates that the end station has an Emulex
OCe11102-FX Converged Network Adapter (CNA), along with the particular
firmware version. This particular CNA supports the standardized FCoE negotiation
protocols, including FCoE Initialization Protocol (FIP), DCBX, PFC, and ETS. This is
important to note because some older generation 1 CNAs dont support the new
standards, which means they cannot negotiate FCoE with all switches. Specifically,
this CNAs support for these standard protocols can be verified as follows:
N5K1# show system internal dcbx info interface ethernet 101/1/1
Interface info for if_index: 0x1f640000(Eth101/1/1)
Interface info for if_index: 0x1f640000(Eth1/1)

tx_enabled: TRUE
rx_enabled: TRUE dcbx_enabled: TRUE
DCX Protocol: CEE

DCX CEE NIV extension: disabled
<snip>
Decoding of the output of this command is not very intuitive; it assumes you know
the specific Type Length Values (TLVs) that LLDP and DCBX use to negotiate the
PFC and ETS features. If for some reason negotiation fails between the FCF and
the ENode, and you need to pinpoint the exact reason, visit the Troubleshooting
FCoE Issues section of the Nexus 5K documentation at
http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/troubleshooting/guide/n5K_ts_
. For example, in this case we would want to know that the CNA has the PFC
feature on, and has agreed to mark the FCoE traffic with the particular Class of
Service (CoS) value that the switch has requested. This is verified from the switchs
side per the output below.
N5K1# show system internal dcbx info interface ethernet 101/1/1 | include TLV|willing
LLDP TLV's
LLDP TLV type:Chassis ID
LLDP TLV type:Port ID
LLDP TLV Length: 7
LLDP TLV Length: 7
LLDP TLV type:Time to Live
LLDP TLV Length: 2
LLDP TLV type:Port Description

LLDP TLV type:System Name
LLDP TLV Length: 8
LLDP TLV Length: 28
LLDP TLV type:System Description
LLDP TLV Length: 45
LLDP TLV type:System capabilities
LLDP TLV Length: 4
LLDP TLV type:LLDP Organizationally Specific

LLDP TLV type:END of LLDP PDU
LLDP TLV Length: 55
LLDP TLV Length: 0
Peer's DCX TLV:

DCBX TLV Proto(1) type: 1(Control) DCBX TLV Length: 10 DCBX TLV Value
sub_type 0, error 0, willing 0, enable 0, max_version 0, oper_version 0
DCBX TLV Proto(1) type: 2(PriGrp)
DCBX TLV Length: 17 DCBX TLV Value
sub_type 0, error 0, willing 1, enable 1

, max_version 0, oper_version 0 DCBX TLV Proto(1) type: 3(PFC)
DCBX TLV Length: 6 DCBX TLV Value
sub_type 0, error 0, willing 1, enable 1
, max_version 0, oper_version 0
DCBX TLV Proto(1) type: 4(App) DCBX TLV Length: 10 DCBX TLV Value
sub_type 0, error 0, willing 1, enable 1, max_version 0, oper_version 0
Feature Register Params: max_version 0, enable 1, willing 0 advertise 1
lldp feature register params max_version 0, enable 1, willing 0 advertise 1, disruptive_e
rror 0 mts_addr_node 0x2201 mts_addr_sap 0x179
lldp feature register params max_version 0, enable 1, willing 0 advertise 1, disruptive_e
rror 0 mts_addr_node 0x2201 mts_addr_sap 0xaf
Total TLVs unrecognized: 0
In reality, this negotiation is much more easily verified from the end station itself,
because the CNAs management application, such as the Emulex OneCommand
Manager, will simply tell you whether negotiation was successful. In the output
shown below, we see that FCoE negotiation was successful and the switch has
informed the CNA to do a 50/50 split between LAN and SAN traffic. SAN traffic is
reserved the Class of Service (CoS) 3, whereas LAN traffic gets the other CoS
values 0, 1, 2, 4, 5, 6, and 7.
Note that on the first-generation Nexus 5000s (5010 and 5020), this QoS policy is
enabled by default. However, in the second-generation Nexus 5500s (5548 and
5596), this policy is not on by default and must be manually enabled. On a Nexus
5500, this can be verified as follows:

Nexus-5548UP# show run all | section "system qos"
system qos
service-policy type queuing input default-in-policy
service-policy type queuing output default-out-policy
service-policy type qos input default-in-policy service-policy type network-qos default-nq-policy
Nexus-5548UP# show policy-map type network-qos
Type network-qos policy-maps

===============================
policy-map type network-qos default-nq-policy

class type network-qos class-default
mtu 1500
multicast-optimize
policy-map type network-qos fcoe-default-nq-policy

class type network-qos class-fcoe
pause no-drop
mtu 2158
mtu 1500
multicast-optimize
Note that there is a default QoS policy that would support FCoE, the fcoe-default-nqpolicy, but it is not the one that is enabled. In the case of a Nexus 5000, this policy
differed as follows:
Nexus-5020# show run all | section "system qos"
system qos
service-policy type queuing input default-in-policy
service-policy type queuing output default-out-policy
service-policy type qos input default-in-policy service-policy type network-qos default-nq-policy
Nexus-5020# show policy-map type network-qos

===============================

pause no-drop
mtu 2158
mtu 1500
The specific relevant output above is the fact that there is a pause no-drop class,
which is enabling Priority Flow Control (PFC) for the FCoE class. The actual
reservation of SAN vs. LAN traffic, however, does not use this network-qos policy,
but is instead controlled by the queuing policy. The default queuing policies can be
seen below.
N5K1# show policy-map type queuing
Type queuing policy-maps

========================
policy-map type queuing default-in-policy

policy-map type queuing default-out-policy

These separate input and output policies both call FCoE and default classes, and
assign them 50% of the bandwidth. The matches in these classes can be verified as
follows:
N5K1# show class-map type queuing
Type queuing class-maps

=======================
class-map type queuing class-fcoe
match qos-group 1
class-map type queuing class-default

match qos-group 0
class-map type queuing class-all-flood

match qos-group 2
class-map type queuing class-ip-multicast
The queuing class-map class-fcoe is matching qos-group 1, and then in turn is

guaranteed 50% bandwidth from the queuing policy-map. The qos-group, however,
is only a temporary internal placeholder, not an actual marking in the packet. Where
this qos-group 1 comes from is as follows:
N5K1# show class-map type qos
Type qos class-maps

===================
class-map type qos match-any class-fcoe
match cos 3
class-map type qos match-any class-default

match any
class-map type qos match-any class-all-flood

match all flood
class-map type qos match-any class-ip-multicast

match ip multicast
N5K1# show policy-map type qos
Type qos policy-maps

====================
policy-map type qos default-in-policy

class type qos class-fcoe
set qos-group 1
class type qos class-default

set qos-group 0
These qos type class-maps and policy-maps are where the actual classification of
FCoE occurs. The workflow of the QoS policy is then as follows:
Match cos 3 with the qos type class-map called class-fcoe.
Set the qos-group to 1 from the qos type policy-map called default-in-policy .
Match qos-group 1 with the queuing type class-map called class-fcoe.
Perform the bandwidth reservation on qos-group 1 with the queuing type policy-maps
called default-in-policy and default-out-policy.
You may ask, Why is this workflow so important? It is because you must ensure that
only your FCoE traffic lands in these particular classes; otherwise, other LAN traffic
types could potentially break the lossless behavior that we are trying to achieve for
FCoE. With the default built-in policies (assuming they are enabled on Nexus 5500),
this behavior will be achieved, but if you are modifying the policy for other LAN
traffic types, you must be careful not to accidentally mark anything as CoS 3 or qosgroup 1, because these land in the FCoE classes.
To meet the requirements of this specific task, if we want to change how much
bandwidth is allocated to the SAN traffic vs. the LAN traffic, we simply need to
create a new queuing policy-map that calls the already in-place FCoE queuing classmaps. In our case, this new policy is defined as follows:
N5K1# show policy-map system type queuing
Service-policy (queuing) input

:
FCoE_75_PERCENT_POLICY
policy statistics status:
Class-map (queuing):
disabled
class-fcoe
(match-any)
Match: qos-group 1
class-default (match-any)
Match: qos-group 0
Service-policy (queuing) output
:
FCoE_75_PERCENT_POLICY
policy statistics status:
disabled
class-fcoe
(match-any)
Match: qos-group 1
class-default (match-any)
Match: qos-group 0
From the end hosts CNA point of view, these new values should be learned through
the DCBX negotiation, and then enforced in the data plane as the QoS policy of the
NIC itself. The changes can be seen below: Now the CoS 3 traffic is guaranteed
75%.

Network (SAN)
Fibre Channel Zoning
Task
Create VSAN 101 on MDS3 and N5K1, and VSAN 102 on MDS4 and N5K2.
Configure MDS3 and MDS4's links to the SAN as 2Gbps F_Ports in VSANs 101 and
102, respectively.
Configure trunking on the links between MDS3 and N5K1 and MDS4 and N5K2 as
2Gbps TE_Ports for VSANs 101 and 102, respectively.
Configure FCoE support for the Emulex CNA Server on N5K1 and N5K2 as follows:
Enable the FCoE feature set on N5K1 and N5K2.
Configure N5K1 and N5K2's links to the Emulex CNA server as 802.1q
trunks and as STP port type edge trunk.
Configure VLAN 10 on N5K1 and N5K2, which will be the server's access
VLAN for regular LAN traffic (that is, not FCoE traffic).
Configure N5K1's link to the CNA Server with VLAN 10 as the native VLAN,
Configure N5K2's link to the CNA Server with VLAN 10 as the native VLAN,
Configure zoning in the Fibre Channel fabric for VSAN 101 as follows:
Configure enhanced zoning for VSAN 101 on MDS3 and N5K1.
Configure a zoneset for VSAN 101 named SAN_A_ZONESET.
Create a zone in this zoneset named EMULEX_TO_NTFS1_SAN_A.
Add the pWWNs of the Emulex CNA attached to N5K1 and the SAN
attached to MDS3's port FC1/9 to the zone.
Activate the zoneset.
Commit the VSAN 101 enhanced zoning session.
Configure zoning in the Fibre Channel fabric for VSAN 102 as follows:
Configure enhanced zoning for VSAN 102 on MDS4 and N5K2.
Configure a zoneset for VSAN 102 named SAN_B_ZONESET.
Create a zone in this zoneset named EMULEX_TO_NTFS1_SAN_B.
Add the pWWNs of the Emulex CNA attached to N5K2 and the SAN
attached to MDS4's port FC1/9 to the zone.
Activate the zoneset.
Commit the VSAN 102 enhanced zoning session.
When complete, the Emulex CNA server should be able to mount only one volume,
NTFS1, out both SAN A and SAN B.
Configuration
N5K1:
feature fex
feature fcoe
!
vlan 10
vlan 1101
fcoe vsan 101
!
vsan database
vsan 101
!
interface fc2/1 - 2
switchport speed 2000
switchport mode E
no shutdown
!
fex associate 101
!
no shutdown
!
interface vfc111
no shutdown
!
vsan database
vsan 101 interface vfc111
!
zone name EMULEX_TO_NTFS1_SAN_A vsan 101
member pwwn 21:00:00:1b:32:07:32:23
member pwwn 10:00:00:00:c9:bb:19:9f
!
zoneset name SAN_A_ZONESET vsan 101
member EMULEX_TO_NTFS1_SAN_A
!
zoneset activate name SAN_A_ZONESET vsan 101
!
zone commit vsan 101
N5K2:
feature fex
feature fcoe
!
vlan 10
vlan 1102
fcoe vsan 102
!
vsan database
vsan 102
!
interface fc2/1 - 2
switchport mode E
no shutdown
!
fex associate 102
!
no shutdown
!
interface vfc211
no shutdown
!
vsan database
!
zone default-zone permit vsan 102
MDS3:
interface fc1/9 - 10
switchport mode F
no shutdown
!
vsan database
vsan 101
vsan 101 interface fc1/9
!
interface fc1/1 - 2
switchport rate-mode dedicated
switchport mode E
no shutdown
!
member pwwn 21:00:00:1b:32:07:32:23
!
!
!
MDS4:
switchport mode F
no shutdown
!
vsan database
vsan 102
!
interface fc1/1 - 2
switchport rate-mode dedicated
switchport mode E
no shutdown
!
zone name EMULEX_TO_NTFS1_SAN_B vsan 102
member pwwn 21:02:00:1b:32:47:32:23
member pwwn 10:00:00:00:c9:bb:19:a3
!
zoneset name SAN_B_ZONESET vsan 102
member EMULEX_TO_NTFS1_SAN_B
!
zoneset activate name SAN_B_ZONESET vsan 102
Verification
Zoning in Fibre Channel is analogous to Access Lists in the IP world, meaning that
zoning is used to control which devices can talk to each other through the Fibre
Channel Fabric. Specifically, zoning is used to control which FC Initiators can talk to
which FC Targets. Zoning helps to prevent problems with file locking if multiple
hosts are mounting the same volumes, or data corruption if hosts are mounting the
wrong types of volumes. For example, if a Windows host accidentally mounts a
volume formatted as VMFS for ESXi hosts, it might corrupt the datastore, or if an
ESXi host mounts a NTFS formatted volume, it would be corrupted.
A very important point to remember about zoning is that it is required. It is not an
optional step in the configuration of the Fibre Channel Fabric. MDS and Nexus
switches run what is known as hard zoning, which means that the zoneset is
enforced not only in the control plane during the Fabric Discovery process of an
initiator, but also in the data plane on all switches in the fabric. In the Ethernet world,
this would be analogous to having a VLAN ACL (VACL or VLAN Access Map)
configured on every single switch that knows about the VLAN in question. This also
means that for zoning to work properly, all switches in the fabric must agree on the
zoneset configuration. The easiest and most commonly implemented way of having
the switches agree on the zoning configuration is to use what is known as enhanced
zoning.
Enhanced zoning allows zoning configuration changes to be made on one device in
the fabric, and then have these changes dynamically advertised to other switches in
the fabric. Additionally, while a zoning configuration change is in progress, a lock is
put on the fabric to ensure that someone else doesnt make a separate zoning
change that would conflict. Assuming that the fabric is already end-to-end (VSANs
are created, Trunking Expansion ports are active, etc.), the zoning mode can be
changed from basic to enhanced on one switch in the fabric, and then advertised to
the other switches. In the output below, we see that MDS3 sets the zoning mode to
enhanced for VSAN 101, and then N5K1 learns about this change. Likewise, N5K2
sets VSAN 102 to enhanced zoning mode, and MDS4 learns about it.
MDS3# conf t
End with CNTL/Z. MDS3(config)# zone mode enhanced vsan 101
WARNING: This command would distribute the zoning database of this switch throughout the fabric. Do you want to cont
Set zoning mode command initiated. Check zone status
N5K1# show zone status vsan 101
VSAN: 101 default-zone: deny distribute: full Interop: default mode: enhanced
merge-control: allow
session: none
hard-zoning: enabled broadcast: enabled
Default zone:
qos: none broadcast: disabled ronly: unsupported
Full Zoning Database :
DB size: 44 bytes
Zonesets:0
Zones:0 Aliases: 0 Attribute-groups: 1
Active Zoning Database :

Database Not Available Status: Set zoning mode success
at 19:47:54 UTC Mar 20 2013
N5K2# conf t
End with CNTL/Z. N5K2(config)# zone mode enhanced vsan 102
WARNING: This command would distribute the zoning database of this switch throughout the fabric. Do you want to cont
Set zoning mode command initiated. Check zone status
MDS4# show zone status vsan 102
VSAN: 102 default-zone: deny distribute: active only Interop: default mode: enhanced
merge-control: allow
session: none
hard-zoning: enabled broadcast: enabled
Default zone:
qos: none broadcast: disabled ronly: disabled

DB size: 44 bytes
Zonesets:0
Zones:0 Aliases: 0 Attribute-groups: 1

Database Not Available Status: Set zoning mode success
at 02:24:13 UTC Jan 13 1982
When all devices in the fabric agree that the zoning mode is enhanced, any of the
switches can start an enhanced zone session. This is where a lock is advertised
through the fabric to other switches, zoning configuration changes are made, the
changes are committed and advertised to other switches in the fabric, and finally the
lock is released. Below we see MDS3 making zoning changes and then N5K1
learning about them after the changes are committed.
MDS3# conf t
End with CNTL/Z. MDS3(config)#

Enhanced zone session has been created. Please 'commit' the changes when done.
MDS3(config-zoneset)# zone name EMULEX_TO_NTFS1_SAN_A
MDS3(config-zoneset-zone)# member pwwn 21:00:00:1b:32:07:32:23
MDS3(config-zoneset-zone)# member pwwn 10:00:00:00:c9:bb:19:9f
MDS3(config-zoneset-zone)# exit
MDS3(config-zoneset)# exit MDS3(config)# zoneset activate name SAN_A_ZONESET vsan 101
MDS3(config)# zone commit vsan 101
Commit operation initiated. Check zone status
N5K1# show zoneset active vsan 101
* fcid 0x630000 [pwwn 21:00:00:1b:32:07:32:23]
N5K1# show run zone
!Command: show running-config zone

!Time: Wed Mar 20 20:06:40 2013
version 5.1(3)N1(1a)
zone mode enhanced vsan 101
!Active Zone Database Section for vsan 101
member pwwn 21:00:00:1b:32:07:32:23

* fcid 0xef0000 [pwwn 10:00:00:00:c9:bb:19:9f]

do clear zone database vsan 101
!Full Zone Database Section for vsan 101
member pwwn 21:00:00:1b:32:07:32:23

At any given time if the zoning database is synchronized (which it should be), all
devices that are registered to the fabric for a particular VSAN should see the same
active zoneset. The active zoneset is the one that is actually being applied as the
filter in hardware, whereas the full zoneset is the one in the local configuration. In
the output below, we see that the VSAN 101 devices and the VSAN 102 devices
agree on the active zonesets for those particular VSANs.
MDS3# show zone active vsan 101
* fcid 0x630000 [pwwn 21:00:00:1b:32:07:32:23]
N5K1# show zone active vsan 101
* fcid 0x630000 [pwwn 21:00:00:1b:32:07:32:23]
MDS4# show zone active vsan 102
* fcid 0xef0000 [pwwn 21:02:00:1b:32:47:32:23]
* fcid 0x280000 [pwwn 10:00:00:00:c9:bb:19:a3]
N5K2# show zone active vsan 102

* fcid 0xef0000 [pwwn 21:02:00:1b:32:47:32:23]
* fcid 0x280000 [pwwn 10:00:00:00:c9:bb:19:a3]
Whenever an initiator registers to the fabric, a Fabric Login (FLOGI) and Port Login
(PLOGI) are performed. Through these login processes, the fabric learns about the
initiators pWWN, assigns them an FCID, and advertises to them with Registered
State Change Notifications (RSCNs) their zoning configuration. In other words, an
initiator will be able to discover only the targets in the fabric that zoning permits. In
our particular case, this means that the Emulex CNA server can mount the NTFS1
volume, but not the NTFS2 volume, because it is not zoned to permit that particular
pWWN. The final verification of this can be seen in the Windows Disk Management
utility below.
Note that although the LUN for the NTFS1 volume exists on both SAN A and SAN
B, separate VSAN numbers and pWWNs are used on the SAN A side vs. the SAN B
side. This means that separate zonesets are needed on A vs. B with the appropriate
numbers. If this configuration is successful, the end host should be able to use the
LUN on both SAN A and SAN B, and as long as one of these sides is reachable,
read/writes to the volume will be successful.
To verify this, the server is configured to write to the volume with the Disk Speed
Test application.
Interface counters on both N5K1 and N5K2 indicate that the server is using
Multipath IO (MPIO) to load balance SCSI read and write traffic to both fabrics.
MPIO is preconfigured on the server and should work as long as the same volume
is seen on both SAN A and SAN B by Windows Disco Management.

N5K1# show interface vfc111 | include rate
1 minute input rate 283775808 bits/sec, 35471976 bytes/sec, 16610 frames/sec
1 minute output rate 209903280 bits/sec, 26237910 bytes/sec, 12577 frames/sec

Next, a failure of a link to the server is simulated by shutting a port down on the SAN
A side. Note that the rest of the FC Fabric is informed about this with a Fabric
Logout message.
N5K1# config t
End with CNTL/Z. N5K1(config)# int e101/1/1
N5K1(config-if)# shutdown
2013 Mar 20 21:02:45 N5K1 %ETHPORT-5-IF_DOWN_CFG_CHANGE: Interface Ethernet101/1/1 is down(Config change)
2013 Mar 20 21:02:45 N5K1
%FLOGI-5-MSG_PORT_LOGGED_OUT: %$VSAN 101%$ [VSAN 101, Interface vfc111: mode[TF]] Nx Port 10:00:00:00:c9:bb:19:9f lo
2013 Mar 20 21:02:45 N5K1 %LLDP-FEX101-5-SERVER_REMOVED: Server with Chassis ID 0000.c9bb.199f Port ID 0000.c9bb.199
2013 Mar 20 21:02:45 N5K1 %PORT-5-IF_DOWN_NONE: %$VSAN 101%$ Interface vfc111 is down (None)
N5K1(config-if)# 2013 Mar 20 21:02:45 N5K1 %PORT-5-IF_DOWN_NONE: %$VSAN 101%$ Interface vfc111 is down (None)
2013 Mar 20 21:02:45 N5K1 %ETHPORT-5-IF_DOWN_ADMIN_DOWN: Interface Ethernet101/1/1 is down (Administratively down)
All traffic from the server is now re-routed to SAN B. This can be verified below, as
the I/O rate of N5K2s link to the CNA is effectively double that of before:
N5K1(config-if)# show interface vfc111 | include rate
1 minute input rate 0 bits/sec
, 0 bytes/sec, 0 frames/sec 1 minute output rate 0 bits/sec
, 0 bytes/sec, 0 frames/sec

Network (SAN)
Fibre Channel over Ethernet (FCoE)
Task
Create VSAN 101 on MDS3 and N5K1, and VSAN 102 on MDS4 and N5K2.
Configure MDS3 and MDS4's links to the SAN as 2Gbps F_Ports in VSANs 101 and
102, respectively.
Configure trunking on the links between MDS3 and N5K1, and MDS4 and N5K2, as
2Gbps TE_Ports for VSANs 101 and 102, respectively.
Configure FCoE support for the Emulex CNA server on N5K1 and N5K2 as follows:
Enable the FCoE feature set on N5K1 and N5K2.
Configure N5K1 and N5K2's links to the Emulex CNA server as 802.1q
trunks and as STP port type edge trunk.
Configure VLAN 10 on N5K1 and N5K2, which will be the server's access
VLAN for regular LAN traffic (that is, not FCoE traffic).
link to the CNA Server.
Configure N5K1 and MDS3 so that the default zoning policy for VSAN 101 is permit.
Configure N5K2 and MDS4 so that the default zoning policy for VSAN 101 is permit.
When complete, the Emulex CNA server should be able to mount two volumes,
NTFS1 and NTFS2, out both SAN A and SAN B.
Configuration
N5K1:
feature fex
feature fcoe
!
vlan 10
vlan 1101
fcoe vsan 101
!
vsan database
vsan 101
!
interface fc2/1 - 2
switchport mode E
no shutdown
!
fex associate 101
!
no shutdown
!
interface vfc111
no shutdown
!
vsan database
!
N5K2:
feature fex
feature fcoe
!
vlan 10
vlan 1102
fcoe vsan 102
!
vsan database
vsan 102
!
interface fc2/1 - 2
switchport mode E
no shutdown
!
fex associate 102
!
no shutdown
!
interface vfc211
no shutdown
!
vsan database
!
MDS3:
switchport mode F
no shutdown
!
vsan database
vsan 101

!
interface fc1/1 - 2
switchport mode E
no shutdown
!
MDS4:
switchport mode F
no shutdown
!
vsan database
vsan 102
!
interface fc1/1 - 2
switchport mode E
no shutdown
!
Verification
Similar to how a physical FC port on an MDS or Nexus switch is used to reference a
server that is physically attached with a regular Fibre Channel HBA, a Virtual Fibre
Channel (VFC) interface is used to reference FCoE attached hosts. Each VFC is
bound to one underlying physical link, or potentially a port channel in the case of
Virtual Trunking Expansion Ports (VTE_Ports) for multi-hop FCoE. In the output
below, we see the interface VFC111 that is bound to N5K1s physical Ethernet link
to the Emulex CNA Server.
N5K1# show interface vfc111
vfc111 is trunking
Bound interface is Ethernet101/1/1
Hardware is Ethernet
Port WWN is 20:6e:00:0d:ec:a2:ed:bf Admin port mode is F, trunk mode is on
snmp link state traps are enabled Port mode is TF
Port vsan is 101
Trunk vsans (admin allowed and active) (101)
Trunk vsans (up)
(101)
Trunk vsans (isolated)
()
Trunk vsans (initializing)
()

72491639 frames input, 151963572580 bytes
0 discards, 0 errors
63800061 frames output, 132994718596 bytes
0 discards, 0 errors
last clearing of "show interface" counters never
Interface last changed at Sat Mar
9 16:10:22 2013
The VFC interface above is configured as a Virtual Fabric Port (VF_Port), or more
specifically in this case, a Virtual Trunking Fabric Port (VTF_Port) because trunking
is on. Similar to how a Fabric Port (F_Port) on the switch side connects to a Node
Port (N_Port) on the end host side (either the FC Target or FC Initiator), the VF_Port
connects to a Virtual Node Port (VN_Port). The VN_Port is the Converged Network
Adapter (CNA) of the end server, or what is sometimes also called the Ethernet
Node (ENode).
The significance of this output is that the VSAN is up on the VFC port. If the VSAN
is stuck in either initializing or isolated mode, this normally indicates some sort of
negotiation error with the end host. For example, if the FCoE Forwarder (FCF),
which is the switch, and the ENode, which is the end host, did not properly negotiate
Data Center Bridging Exchange (DCBX), the VSAN would not be up.
There are several hardware and software reasons that this could occur, which are
outside the scope of this scenario. For more information on problems that cause the
VFC to fail, and their resolution, see Troubleshooting FCoE Issues (
http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/troubleshooting/guide/n5K_ts_
) in the Nexus 5000 documentation.
In the output below, the vfc interface appears just like any other Fibre Channel
interface, but it doesnt tell us which physical Ethernet port it is actually bound to.
N5K1# show interface fc2/1 - 2 brief
------------------------------------------------------------------------------Interface
Vsan
Admin
Admin
Mode
Trunk
Status
SFP
Oper
Oper
Port
Mode
Speed
Channel
Mode
(Gbps)
------------------------------------------------------------------------------fc2/1
on
trunking
swl
TE
--
fc2/2
on
trunking
swl
TE
--
N5K1# show interface vfc111 brief
------------------------------------------------------------------------------Interface
Vsan
Admin
Admin
Mode
Trunk
Status
SFP
Oper
Oper
Port
Mode
Speed
Channel
Mode
(Gbps)
------------------------------------------------------------------------------vfc111
101
on
trunking
--
TF
auto --
During the FCoE Initialization Protocol (FIP) exchange between the FCoE
Forwarder (FCF) and the ENode, the ENode is assigned a Fabric Provided MAC
Address (FPMA) as well as a regular Fibre Channel Identifier (FCID). The FCID can
be verified through either of the below outputs.
N5K1# show fcoe database
------------------------------------------------------------------------------INTERFACE
FCID
PORT NAME
MAC ADDRESS
------------------------------------------------------------------------------- vfc111
0xef0000
10:00:00:00:c9:bb:19:9f 00:00:c9:bb:19:9f
Total number of flogi count from FCoE devices = 1.

N5K1# show flogi database
-------------------------------------------------------------------------------INTERFACE
VSAN
FCID
PORT NAME
NODE NAME
-------------------------------------------------------------------------------vfc111
101
0xef0000
10:00:00:00:c9:bb:19:9f 20:00:00:00:c9:bb:19:9f
The FPMA of the host is the combination of the FC-MAP of the FCF, and the FCID
of the Enode. Below we see that N5K1s FC-MAP is 0e:fc:00, and from above we
see that the ENode has the FCID 0xef0000. This means that their MAC address for
FCoE data plane traffic is 0e:fc:00:ef:00:00.
N5K1# show fcoe
Global FCF details
FCF-MAC is 00:0d:ec:a2:ed:80 FC-MAP is 0e:fc:00
FCF Priority is 128
FKA Advertisement period for FCF is 8 seconds
VFC MAC details

vfc111 FCF-MAC is 00:0d:ec:a2:ed:80
Now that the ENode has been verified as logged in to the fabric, we want to make
sure that the SAN itself is logged in, that the N5K can see the SAN in the fabric, and
that the MDS can see the ENode in the fabric. All of these can be verified by viewing
the FCNS database, as shown below.
N5K1# show fcns database
VSAN 101:
-------------------------------------------------------------------------FCID
TYPE
PWWN
(VENDOR)
FC4-TYPE:FEATURE
-------------------------------------------------------------------------0x630000
21:00:00:1b:32:07:32:23 (Qlogic)
scsi-fcp:target
0x630100
21:01:00:1b:32:27:32:23 (Qlogic)
scsi-fcp:target
0xef0000
10:00:00:00:c9:bb:19:9f (Emulex)
ipfc scsi-fcp:init

MDS3# show fcns database
VSAN 101:
-------------------------------------------------------------------------FCID
TYPE
PWWN
(VENDOR)
FC4-TYPE:FEATURE
-------------------------------------------------------------------------0x630000
21:00:00:1b:32:07:32:23
scsi-fcp:target
0x630100
21:01:00:1b:32:27:32:23
scsi-fcp:target
0xef0000
10:00:00:00:c9:bb:19:9f (Emulex)
ipfc scsi-fcp:init
In VSAN 102 on the SAN B side, a similar output would be seen, but with different
pWWNs and FCIDs, as the FC domain services, FCNS, FLOGI, etc. happens
separately on a per-VSAN basis.
VSAN 102:
-------------------------------------------------------------------------FCID
TYPE
PWWN
(VENDOR)
FC4-TYPE:FEATURE
--------------------------------------------------------------------------
0x280000
10:00:00:00:c9:bb:19:a3 (Emulex)
ipfc scsi-fcp:init
0xef0000
21:02:00:1b:32:47:32:23 (Qlogic)
scsi-fcp:target
0xef0100
21:03:00:1b:32:67:32:23 (Qlogic)
scsi-fcp:target

VSAN 102:
-------------------------------------------------------------------------FCID
TYPE
PWWN
(VENDOR)
FC4-TYPE:FEATURE
-------------------------------------------------------------------------0x280000
10:00:00:00:c9:bb:19:a3 (Emulex)
ipfc scsi-fcp:init
0xef0000
21:02:00:1b:32:47:32:23
scsi-fcp:target
0xef0100
21:03:00:1b:32:67:32:23
scsi-fcp:target
Another important point about FCoE is that the switches must ensure a lossless
fabric for the ENodes. In other words, the FCoE switches need a QoS policy to
make sure that FCoE isnt dropped. On the Nexus 5000s, this policy is on by
default, as shown below.
N5K1# show policy-map system type network-qos

===============================

match qos-group 1
pause no-drop
mtu 2158

match qos-group 0
mtu 1500
On the Nexus 5500 switches, this QoS policy is not enabled by default and must be
configured before the FIP negotiation is successful. Configuring and modifying this
policy is covered in a later scenario.
One more step on the network side now remains before the end hosts can mount
their LUNs on the SAN. Fibre Channel Zoning must be configured so that the N5Ks
and MDSes allow traffic in the data plane between the hosts and their LUNs. Zoning
is analogous to an Access List in the IP world, and the default zoning behavior is to
deny all traffic. For the purpose of simplicity in this scenario, the zoning policy was
changed to explicit permit, which can be seen below. In a real design, zoning would
be manually configured to associate which servers can map which storage. Zoning
techniques are covered in a separate scenario.
N5K1# show zone status vsan 101
VSAN: 101 default-zone: permit
distribute: active only Interop: default
mode: basic merge-control: allow
session: none
hard-zoning: enabled broadcast: disabled
Default zone:
qos: none broadcast: disabled ronly: unsupported
DB size: 4 bytes
Zonesets:0
Zones:0 Aliases: 0

Database Not Available Status: Default zoning policy changed to permit
at 18:42:25 UTC Mar 20 2013
The next step in verification is to check the end host itself. In the output below, in the
Emulex OneCommand Manager app we can see that both ports of the CNA have
logged into the fabric and detected LUNs available from the SAN. If Automapping is
turned on in OneCommand, which it is below, no extra steps are needed in the
management app.
The LUNs on the SAN should now be reported to Windows Disk Management so
that the volumes can actually be mounted. In the output below, we see that the
LUNs are mounted as the NTFS1 and NTFS2 volumes.
If you want to further verify storage traffic in the data plane, you can use a Disk
Speed Test application to stress test reads and writes to the disks, as shown below.
Because the end host is pre-configured to do Multipath I/O (MPIO), we should see
traffic coming in from the ENode on both the SAN A and SAN B sides, as shown
here.
N5K2# show interface vfc 211 | include rate
Both of the 5Ks should then be trunking the SCSI reads and writes out to the
MDSes, as shown below.
N5K1# show interface fc2/1 - 2 | include fc|rate
fc2/1 is trunking 1 minute input rate 102600368 bits/sec

N5K2# show interface fc2/1 - 2 | include fc|rate

Note that all four uplinks to the MDSes, two from N5K1 and two from N5K2, are
being used because the 5Ks have multiple equal cost paths to reach the SAN
targets in the fabric.

Network (SAN)
SAN Port Channeling
Task
Enable the FCoE feature on N5K1 and N5K2.
Create VSAN 101 on MDS3 and N5K1.
Configure MDS3's links to the Fibre Channel SAN as follows:
Hardcode the links to 2Gbps.
Set the links to be a Fabric Ports (F_Ports).
Assign VSAN 101 to these links.
Configure MDS4's link to the Fibre Channel SAN for your rack as follows:
Configure the links between MDS3 and N5K1 as follows:
Aggregate these links together into Port Channel 11 using the Port Channel
Protocol for negotiation.
Set the port channel to be a Trunking Expansion Port (TE_Port).
Limit the VSANs on the trunk to just VSAN 101.
Statically aggregate these links together into Port Channel 12.
Set the port channel to be a Trunking Expansion Port (TE_Port).
When complete, ensure that the MDSs see the FC SAN ports in the FLOGI database
and that the N5Ks see the FC SAN ports in the FCNS database.
Configuration
N5K1:
feature fcoe
!
vsan database
vsan 101
!
interface fc2/1 - 2
switchport mode E
channel-group 11 force
no shutdown
!
interface san-port-channel 11
channel mode active
switchport mode E
N5K2:
feature fcoe
!
vsan database
vsan 102
!
interface fc2/1 - 2
switchport mode E
no shutdown
!
interface san-port-channel 12
no channel mode active
switchport mode E
MDS3:
switchport mode F
no shutdown
!
vsan database
vsan 101
!
interface fc1/1 - 2
switchport mode E
no shutdown
!
interface port-channel 11
channel mode active
switchport mode E
MDS4:
switchport mode F
no shutdown
!
vsan database
vsan 102
!
interface fc1/1 - 2
switchport mode E
no shutdown
!
interface port-channel 12
no channel mode active
switchport mode E
Verification
Like Ethernet port channels, SAN port channels are used to aggregate the
bandwidth of multiple physical links. Fibre Channel traffic is then load balanced over
the member links of the SAN port channel using the source ID, destination ID, and
exchange ID. Note that the Nexus 5000 switches denote these as san-portchannels, whereas the MDS denotes them simply as port-channels.
Below we see that the SAN port channel has formed between N5K1 and MDS3
using the Port Channel Protocol for negotiation.

N5K1# show san-port-channel summary
U-Up D-Down B-Hot-standby S-Suspended I-Individual link
summary header
-------------------------------------------------------------------------------Group
Port-
Type
Protocol
Member Ports
Channel
-------------------------------------------------------------------------------11
San-po11
FC
PCP
(U)
FC
fc2/1(P)
fc2/2(P)
MDS3# show port-channel database interface port-channel 11

port-channel 11
Administrative channel mode is active Operational channel mode is active
Last membership update succeeded
First operational port is fc1/2
2 ports in total, 2 ports up Ports:
fc1/2
fc1/1
[up]
[up]
*
N5K1# show san-port-channel database interface san-port-channel 11
san-port-channel 11
Last membership update is successful
2 ports in total, 2 ports up
First operational port is fc2/2
Age of the port-channel is 0d:00h:03m:06s Ports:
fc2/2
fc2/1
[up]
[up]
Below we see that the port channel is running as a Trunking Expansion Port
(TE_Port) and is trunking VSAN 101. The speed of the channel is 4Gbps, the
aggregate of 2 x 2Gbps member ports.
N5K1# show interface san-port-channel 11 brief
-------------------------------------------------------------------------------Interface
Vsan
Admin
Status
Trunk
Oper
Oper
IP
Mode
Speed
Address
Mode
(Gbps)
-------------------------------------------------------------------------------san-port-channel 11
on
trunking
TE
-N5K1# show interface san-port-channel 11 trunk vsan

san-port-channel 11 is trunking
Vsan 101 is up (None)
From a Fibre Channel routing point of view, N5K1 sees MDS3 (Domain ID 0x51)
reachable through the single san-port-channel 11 interface.
N5K1# show fspf internal route vsan 101
FSPF Unicast Routes

--------------------------VSAN Number
Dest Domain
Route Cost
Next hops
----------------------------------------------- 201
0x51(81)
250 san-port-channel 11
The Fibre Channel SAN sends a Fabric Login (FLOGI) to MDS3 to join the fabric,
and is assigned an FCID.
MDS3# sh flogi database vsan 101
-------------------------------------------------------------------------------INTERFACE
VSAN
FCID
PORT NAME
NODE NAME
-------------------------------------------------------------------------------fc1/9
101
0x510000
21:00:00:1b:32:07:32:23 20:00:00:1b:32:07:32:23
fc1/10
101
0x510100
21:01:00:1b:32:27:32:23 20:01:00:1b:32:27:32:23
Total number of flogi = 2.
N5K1 can see the pWWN and FCID of this target as registered to the Fibre Channel
Name Server database, which indicates that the fabric communication is end-to-end
for VSAN 101.
N5K1# show fcns database vsan 101
VSAN 101:
-------------------------------------------------------------------------FCID
TYPE
PWWN
(VENDOR)
FC4-TYPE:FEATURE
-------------------------------------------------------------------------0x510000
21:00:00:1b:32:07:32:23 (Qlogic)
scsi-fcp:init
0x510100
21:01:00:1b:32:27:32:23 (Qlogic)
scsi-fcp:init

Network (SAN)
VSANs and VSAN Trunking
Task
Enable the FCoE feature on N5K1 and N5K2.
Configure MDS3's links to the Fibre Channel SAN as follows:
Configure MDS4's link to the Fibre Channel SAN for your rack as follows:
Set the links to be Trunking Expansion Ports (TE_Ports).
Set the links to be Trunking Expansion Ports (TE_Ports).
When complete, ensure that the MDSs see the FC SAN ports in the FLOGI database
and that the N5Ks see the FC SAN ports in the FCNS database.
Configuration
N5K1:
feature fcoe
!
vsan database
vsan 101
!
interface fc2/1 - 2
switchport mode E
no shutdown
N5K2:
feature fcoe
!
vsan database
vsan 102
!
interface fc2/1 - 2
switchport mode E
no shutdown
MDS3:
switchport mode F
no shutdown
!
vsan database
vsan 101
!
interface fc1/1 - 2
switchport mode E
no shutdown
MDS4:
switchport mode F
no shutdown
!
vsan database
vsan 102
!
interface fc1/1 - 2
switchport mode E
no shutdown
Verification
This scenario shows how to establish basic VSAN connectivity between the MDS
and Nexus 5000 switches. First, we verify that the VSAN is assigned to the correct
Fabric Ports to the storage array and that the VSAN is trunking between the
switches.
MDS3# show vsan 101 membership
vsan 101 interfaces
: fc1/9
fc1/10
MDS3# show interface fc1/1 - 2 trunk vsan

fc1/1 is trunking
Vsan 101 is up
(None)
fc1/2 is trunking
Vsan 101 is up (None)
The port types can also be verified with
show interface brief
, as you can see below.
MDS3# show interface fc1/1-2,fc1/9-10 brief
------------------------------------------------------------------------------Interface
Vsan
Admin
Admin
Mode
Trunk
Status
SFP
Oper
Oper
Port
Mode
Speed
Channel
Mode
(Gbps)
------------------------------------------------------------------------------fc1/1
on trunking
swl TE
-- fc1/2
on trunking
-- fc1/9
101
on
up
swl F
-- fc1/10
101
on
up
swl F
--
swl TE
If vsan 101 is end-to-end through the fabric, the switches should know about each
others Domain IDs. In the output below, we see that MDS3 is assigned the Domain
ID 0x51 and N5K3 is assigned 0x12.
MDS3# show fcdomain domain-list vsan 101
Number of domains: 2
Domain ID
---------
WWN
----------------------- 0x51(81)
20:65:00:0d:ec:26:ea:81 [Local]
[Principal]
0x12(18)
20:65:00:0d:ec:a2:ed:81
N5K1# show fcdomain domain-list vsan 101
Number of domains: 2
Domain ID
WWN
---------
-----------------------
0x51(81)
20:65:00:0d:ec:26:ea:81 [Principal] 0x12(18)
20:65:00:0d:ec:a2:ed:81 [Local]
Because MDS3 and N5K3 have two trunk links connected between them, they
should both be seen as equal cost paths to reach the remote switchs Domain ID, as
shown below.
N5K1# show fspf internal route vsan 101
FSPF Unicast Routes

--------------------------VSAN Number
Dest Domain
Route Cost
Next hops
----------------------------------------------- 101
0x51(81)
500
fc2/1
fc2/2
Now that the inter-switch communication has been verified, we can focus on the
Fibre Channel Target and Initiator verification. The FC Target is the SAN directly
attached to MDS3. If this communication is successful, we should see that the
SANs FC HBAs have sent a Fabric Login (FLOGI) into the switch fabric. This can
be verified below, which shows the ports that the SAN is connected on, its VSAN
assignments, its Fibre Channel Identifiers (FCID), and both the Port World Wide
Names (pWWN/WWPN) and the Node World Wide Names (nWWN/WWNN).
MDS3# show flogi database vsan 101v
-------------------------------------------------------------------------------INTERFACE
VSAN
FCID
PORT NAME
NODE NAME
-------------------------------------------------------------------------------fc1/9
101
0x510000
21:00:00:1b:32:07:32:23 20:00:00:1b:32:07:32:23
When FLOGI is complete, targets and initiators register with the Fibre Channel
Name Server/Service (FCNS). Unlike the FLOGI database, which is only local to the
switch where the initiator/target is directly connected, the FCNS database is fabricwide. This means that if the Nexus 5K and the MDS are properly communicating,
the N5K should see the FCIDs and pWWNs of the SAN in the FCNS database, as
shown below.
N5K1# show fcns database vsan 101
VSAN 101:
-------------------------------------------------------------------------FCID
TYPE
PWWN
(VENDOR)
FC4-TYPE:FEATURE
-------------------------------------------------------------------------0x510000
21:00:00:1b:32:07:32:23
(Qlogic)
scsi-fcp:init 0x510100
(Qlogic)
scsi-fcp:init
21:01:00:1b:32:27:32:23

Network (SAN)
Fibre Channel Device Aliases
Task
Configure MDS3's links to the SAN as 2Gbps F_Ports in VSAN 101.
Configure trunking on the links between MDS3 and N5K1 as 2Gbps TE_Ports for
VSAN 101.
Configure FCoE support for the Emulex CNA server on N5K1 as follows:
Enable the FCoE feature set on N5K1.
Configure N5K1's link to the Emulex CNA Server as an 802.1q trunk and as
STP port type edge trunk.
Configure VLAN 10 on N5K1, which will be the server's access VLAN for
regular LAN traffic (not FCoE traffic).
Configure device aliases in the Fibre Channel fabric as follows:
Change the Device Alias mode to Enhanced.
Start a device-alias database session on MDS3.
Create a device alias named ARRAY_1_SAN_A_PORT_1 for the pWWN
learned on MDS3's first link to the SAN.
Create a device alias named ARRAY_1_SAN_A_PORT_2 for the pWWN
learned on MDS3's second link to the SAN.
Create a device alias named EMULEX_CNA_SERVER_SAN_A for the
pWWN of the CNA server.
Create a device alias named NEW_INITIATOR for the pWWN

10:00:00:00:00:00:00:01.
Commit the Device Alias database, and ensure that N5K1 learns the name
to pWWN mappings.
Configure zoning for VSAN 101 as follows:
Enable enhanced zoning for VSAN 101.
Create a Zoneset named VSAN_101_ZONESET.
Add a Zone named ZONE_ONE to this set with members
ARRAY_1_SAN_A_PORT_1 and EMULEX_CNA_SERVER_SAN_A.
Add another zone named ZONE_TWO to this set with members
ARRAY_1_SAN_A_PORT_2 and NEW_INITIATOR.
Zone membership should be based on the device aliases instead of directly
on the pWWNs.
Commit the Zoneset, and verify that both MDS3 and N5K1 have activated
the Zoneset.
When complete, the Emulex CNA server should be able to mount only one volume,
NTFS1.
Configuration
N5K1:
feature fex
feature fcoe
!
vlan 10
vlan 1101
fcoe vsan 101
!
vsan database
vsan 101
!
interface fc2/1 - 2
switchport mode E
no shutdown
!
fex associate 101
!
no shutdown
!
interface vfc111
no shutdown
!
vsan database
!
device-alias mode enhanced
device-alias database
device-alias name NEW_INITIATOR pwwn 10:00:00:00:00:00:00:01
device-alias name ARRAY_1_SAN_A_PORT_1 pwwn 21:00:00:1b:32:07:32:23
device-alias name EMULEX_CNA_SERVER_SAN_A pwwn 10:00:00:00:c9:bb:19:9f
!
device-alias commit
!
zone name ZONE_ONE vsan 101
member pwwn 21:00:00:1b:32:07:32:23
!
[ARRAY_1_SAN_A_PORT_1]
[EMULEX_CNA_SERVER_SAN_A]
!
zone name ZONE_TWO vsan 101
member pwwn 21:01:00:1b:32:27:32:23
!
member pwwn 10:00:00:00:00:00:00:01
[NEW_INITIATOR]
!
zoneset name VSAN_101_ZONESET vsan 101
member ZONE_ONE
member ZONE_TWO
!
zoneset activate name VSAN_101_ZONESET vsan 101
member pwwn 21:00:00:1b:32:07:32:23

!
!
member pwwn 21:01:00:1b:32:27:32:23
!
member pwwn 10:00:00:00:00:00:00:01
[NEW_INITIATOR]
!
member ZONE_ONE
member ZONE_TWO
!
MDS3:
switchport mode F
no shutdown
!
vsan database
vsan 101
!
interface fc1/1 - 2
switchport mode E
no shutdown
!
!
device-alias commit
!
member pwwn 21:00:00:1b:32:07:32:23

!
!
member pwwn 21:01:00:1b:32:27:32:23
!
member pwwn 10:00:00:00:00:00:00:01
[NEW_INITIATOR]
!
member ZONE_ONE
member ZONE_TWO
!
member pwwn 21:00:00:1b:32:07:32:23
!
!
member pwwn 21:01:00:1b:32:27:32:23
!
member pwwn 10:00:00:00:00:00:00:01
[NEW_INITIATOR]
!
member ZONE_ONE
member ZONE_TWO
!
Verification
Fibre Channel device aliases are loosely analogous to Fully Qualified Domain
Names (FQDNs) and DNS in the IP world. The goal of using aliases is to give a
human-friendly name to an FC initiator or targets hardware address, the Port World
Wide Name (pWWN). Enhanced device aliases, similar to Enhanced Zoning, can be
administered from any switch participating in the fabric and can have changes
dynamically advertised through Cisco Fabric Services (CFS), while at the same time
preventing multiple people from editing the database at the same time by adverting
a lock to the fabric. Like zoning, the device alias mode can be verified as follows:
N5K1# show device-alias status
Fabric Distribution: Enabled Database:- Device Aliases 4 Mode: Enhanced
Checksum: 0x161758502cec8377ff1ffed99edf6bf
After a device-alias database session is started, a lock is advertised to the fabric

and no one else can make changes. As shown below, N5K1 starts a session by
creating an alias; MDS3 is unable to start a session because N5K1s changes have
not yet been committed.
N5K1# config t
End with CNTL/Z.
N5K1(config)# device-alias database

N5K1(config-device-alias-db)# show device-alias status
Fabric Distribution: Enabled
Database:- Device Aliases 4
Mode: Enhanced
N5K1(config-device-alias-db)# device-alias name ALIAS1 pwwn 11:22:33:44:55:66:77:88
N5K1(config-device-alias-db)#
MDS3# config t
End with CNTL/Z.
MDS3(config)# device-alias database

MDS3(config-device-alias-db)# device-alias name ALIAS2 pwwn AA:BB:CC:DD:00:11:22:33
Operation failed. Fabric is already locked
MDS3(config-device-alias-db)# show device-alias status
Fabric Distribution: Enabled
Database:- Device Aliases 4
Mode: Enhanced
Locked By:- User "CLI/SNMPv3:admin" SWWN 20:00:00:0d:ec:a2:ed:80
After they are committed, the changes are advertised to everyone in the fabric.
N5K1(config-device-alias-db)# device-alias commit
N5K1(config)# show device-alias database
device-alias name ALIAS1 pwwn 11:22:33:44:55:66:77:88

MDS3# show device-alias database
device-alias name ALIAS1 pwwn 11:22:33:44:55:66:77:88

Another advantage of using device aliases is that these names will now appear in
various show commands to more clearly identify a pWWN, as shown below.
N5K1# show zoneset active
* fcid 0x630000 [pwwn 21:00:00:1b:32:07:32:23] [ARRAY_1_SAN_A_PORT_1]
* fcid 0xef0000 [pwwn 10:00:00:00:c9:bb:19:9f] [EMULEX_CNA_SERVER_SAN_A]
* fcid 0x630100 [pwwn 21:01:00:1b:32:27:32:23] [ARRAY_1_SAN_A_PORT_2]
pwwn 10:00:00:00:00:00:00:01 [NEW_INITIATOR]

N5K1# show flogi database
-------------------------------------------------------------------------------INTERFACE
VSAN
FCID
PORT NAME
NODE NAME
-------------------------------------------------------------------------------vfc111
101
0xef0000
10:00:00:00:c9:bb:19:9f 20:00:00:00:c9:bb:19:9f

VSAN 101:
-------------------------------------------------------------------------FCID
TYPE
PWWN
(VENDOR)
FC4-TYPE:FEATURE
-------------------------------------------------------------------------0x630000
21:00:00:1b:32:07:32:23 (Qlogic)
scsi-fcp:target
0x630100
21:01:00:1b:32:27:32:23 (Qlogic)
scsi-fcp:target
0xef0000
10:00:00:00:c9:bb:19:9f (Emulex)
ipfc scsi-fcp:init

Network (SAN)
iSCSI Virtual Targets
You may download or view the physical topology for your rack by
clicking the Resources button in the upper-right.
Task
Create VSAN 101 on MDS3, and configure its links to the SAN as 2Gbps F_Ports in
VSAN 101.
Establish IP connectivity between MDS3 and Server 1 as follows:
Configure MDS3's Gig1/1 interface with the IP address 10.0.0.63/24.
Configure Server 1's link to N5K2 with the IP address 10.0.0.11/24.
Configure 3750G-2's link the MDS3's Gig1/1 in VLAN 101.
Configure VLAN 101 trunking between N5K2 and 3750G-2.
Enable Jumbo Frame support on MDS3's Gig1/1, Server 1's link the N5K2,
and N5K2.
Verify that Server 1 can ping MDS3 in the 10.0.0.0/24 network.
Configure iSCSI Virtual Target support on MDS3 as follows:
Enable the iSCSI feature set on MDS3 for module 1.
Create interface iSCSI1/1, and assign it to VSAN 101.
Configure MDS3 to dynamically import all Fibre Channel targets into iSCSI.
Create an iSCSI Virtual Target using Server 1's IP address 10.0.0.11 as the
initiator.
MDS3 should represent this iSCSI initiator to the SAN as a Fibre Channel
initiator in VSAN 101 with pWWN 10:00:00:00:15:c5:10:01.
Configure Fibre Channel Zoning so that Server 1 has access to LUNs on
MDS3's first link to the SAN.
When complete, open the Windows iSCSI Initiator on Server 1, enter the target
address 10.0.0.63, and click Quick Connect. If successful, Server 1 should mount the
volume named "iSCSI1" from the Fibre Channel SAN.
Configuration
3750G-2:
system mtu jumbo 9000
!
vlan 101
!
interface GigabitEthernet1/0/5
description To MDS3 Gig1/1
!
description To N5K2 E1/14
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 101

N5K2:
speed 1000
!
switchport trunk allowed vlan 101
speed 1000
!
policy-map type network-qos JUMBO_FRAMES
mtu 9216
!
system qos
service-policy type network-qos JUMBO_FRAMES
MDS3:
feature iscsi
iscsi enable module 1
!
iscsi import target fc
iscsi initiator ip-address 10.0.0.11
static pWWN 10:00:00:00:15:c5:10:01
vsan 101
!
switchport mode F
no shutdown
!
zone name iSCSI_VIRTUAL_TARGET vsan 101
member pwwn 10:00:00:00:15:c5:10:01
member pwwn 21:00:00:1b:32:07:32:23
!
member iSCSI_VIRTUAL_TARGET
!
!
interface GigabitEthernet1/1
ip address 10.0.0.63 255.255.255.0
switchport mtu 8000

no shutdown
!
interface iscsi1/1
no shutdown
!
vsan database
vsan 101
vsan 101 interface iscsi1/1
Verification
In addition to normal Fibre Channel switching, Cisco MDS switches support IP
storage (IPS) services. One of these IPS services is the iSCSI Virtual Target,
sometimes called the iSCSI Gateway feature, which allows the MDS to be a proxy
between an iSCSI-based initiator and a Fibre Channel-based target. The advantage
of this feature is that the MDS can attach to a normal Fibre Channel-based SAN and
then support any iSCSI initiator, such as the Microsoft iSCSI Initiator, as shown in
this scenario, or a NIC that supports boot from iSCSI, without needing native Fibre
Channel infrastructure end to end. From the perspective of the FC SAN, the MDS is
the FC initiator, and from the perspective of the iSCSI initiator, the MDS is the iSCSI
target.
The first step in configuring this feature is obtaining IP reachability between MDS
and the end server that is the iSCSI initiator. When considering iSCSI vs. native
Fibre Channel, it is important to remember the frame and payload size. Fibre
Channel has a max frame length of 2148 bytes, with a usable max payload of 2112
bytes. Ethernet transports normally default to a 1500-byte MTU. Running IP and
TCP on top of this adds an additional 20 bytes of overhead each, which means that
the usable TCP payload called the Maximum Segment Size (MSS) - is 1460
bytes. Using this default lower MTU with iSCSI can result in poor performance, or
dropped frames in the worst case scenario. Therefore it is normal best practice to
enable Jumbo Frame support for Ethernet whenever iSCSI is being used. On the
MDS, this is configured as the switchport MTU, and verified below.
MDS3# show interface gigabitethernet 1/1
GigabitEthernet1/1 is up
Hardware is GigabitEthernet, address is 000d.bd86.a4ec
Internet address is 10.0.0.63/24 MTU 8000
Port mode is IPS

Speed is 1 Gbps
bytes
Beacon is turned off

Auto-Negotiation is turned on
5 minutes input rate 48 bits/sec, 6 bytes/sec, 0 frames/sec
5 minutes output rate 8 bits/sec, 1 bytes/sec, 0 frames/sec
2745528 packets input, 5823955862 bytes
5119 multicast frames, 0 compressed
0 input errors, 0 frame, 0 overrun 0 fifo
4149531 packets output, 4783045498 bytes, 0 underruns
0 output errors, 0 collisions, 0 fifo
0 carrier errors
On the Nexus 5K, Jumbo Frame support is enabled by changing the Network QoS
policy, as shown below.
N5K2# show policy-map system type network-qos

===============================
policy-map type network-qos JUMBO_FRAMES
match qos-group 1
pause no-drop
mtu 2158
match qos-group 0
mtu 9216
When IP connectivity is established, the MDS presents itself as an iSCSI virtual

target to the end host. Below we see that the Windows machine has initiated an
iSCSI session to the MDS, and the MDS has assigned it virtual pWWN of
10:00:00:00:15:c5:10:01. Typically, you would statically assign a pWWN to an iSCSI
initiator if you wanted to do your Fibre Channel Zoning based on pWWNs, as shown
in this scenario. Zoning can also be configured based on the iSCSI Qualified Name
(IQN), the IP address of the initiator, or even login authentication information. In
these cases the pWWN assignment would be arbitrary, assuming that the SAN itself
does not need to enforce some type of LUN Masking.
MDS3# show iscsi initiator
iSCSI Node name is 10.0.0.11
iSCSI Initiator name: iqn.1991-05.com.microsoft:rack1server1
iSCSI alias name:

Configured node (iSCSI)
Node WWN is 22:02:00:0d:ec:26:ea:82 (dynamic)
Member of vsans: 101
Number of Virtual n_ports: 1 Virtual Port WWN is 10:00:00:00:15:c5:10:01 (configured)
Interface iSCSI 1/1, Portal group tag: 0x3000

VSAN ID 101, FCID 0x630200
Below we see that the MDS appears to the iSCSI initiator as the iSCSI target with
IQN iqn.1987-05.com.cisco:05.mds3.01-01.2100001b32073223.
MDS3# show iscsi virtual-target
target: iqn.1987-05.com.cisco:05.mds3.01-01.2100001b32073223
* Port WWN 21:00:00:1b:32:07:32:23 , VSAN 101

Auto-created node (iSCSI)
MDS3# show iscsi session
Initiator 10.0.0.11 Initiator name iqn.1991-05.com.microsoft:rack1server1
Session #1 Target iqn.1987-05.com.cisco:05.mds3.01-01.2100001b32073223

VSAN 101 , ISID 400001370000, Status active
, no reservation
This IQN in what appears in the Windows iSCSI Initiator when the session is
established, as shown below.
When the volume is mounted, you can use the Disk Speed Test app shown below to
generate bulk SCSI read and write traffic to the disk.
From the network side, the MDS maintains interface counters for the logical iSCSI
interface.
MDS3# show iscsi stats iscsi1/1
iscsi1/1
iSCSI statistics
64734 packets input, 1377350972 bytes
Command 43407 pdus, Data-out 21269 pdus, 1374170112 bytes, 2723133 fragments, unsolicited 1489841152 bytes
output 64733 packets, 260484838 bytes
Response 43390 pdus (with sense 9), R2T 21285 pdus
Data-in 70674 pdus, 257373134 bytes
A data plane capture from the end server indicates that Jumbo Frames are being
used for the iSCSI reads and writes. Below we see a Wireshark output with an IP
packet length of 4184, which without our MTU modification would be limited to 1500
bytes.

Network (SAN)
Fibre Channel over IP (FCIP)
Task
Configure basic Fibre Channel reachability throughout the fabric as follows:
Create VSAN 101 on N5K1, MDS3, and MDS4.
Configure MDS4's first link to the SAN as a 2Gbps F_Port in VSAN 101.
Configure trunking on the links between MDS3 and N5K1 as 2Gbps
TE_Ports for VSAN 101.
Configure N5K1 to pair with the Fabric Extender N2K1 using FEX number
101.
Configure FCoE support on N5K1 to the the Emulex CNA server.
The CNA server should use VLAN 1101 for FCoE traffic, which should map
to VSAN 101.
When complete, N5K1 and MDS3 should see the CNA server registered to
the fabric, and MDS4 should see the SAN.
Configure Fibre Channel over IP between MDS3 and MDS4 as follows:
Configure 3750G-2 so that the Gig1/1 ports of MDS3 and MDS4 are in one
VLAN, and the Gig1/2 ports are in a separate VLAN.
Configure the Gig1/1 ports on MDS3 and MDS4 with the IP addresses
169.254.1.63/24 and 169.254.1.64/24, respectively.
Configure the Gig1/2 ports on MDS3 and MDS4 with the IP addresses
169.254.2.63/24 and 169.254.2.64/24, respectively.
Enable Jumbo Frame support on the Gig ports of MDS3, MDS4, and 3750G2.
Enable the FCIP feature on MDS3 and MDS4.
Configure FCIP profile 1 on MDS3 and MDS4 that uses the source IP
address of their Gig1/1 interfaces.
Configure interface FCIP1, which uses FCIP profile 1, and references the
remote peer's IP address in the 169.254.1.0/24 subnet.
Configure FCIP profile 2 on MDS3 and MDS4 that uses the source IP
address of their Gig1/2 interfaces.

Configure interface FCIP2, which uses FCIP profile 2, and references the
remote peer's IP address in the 169.254.2.0/24 subnet.
Both of these FCIP interfaces should be TE_Ports that carry VSAN 101.
Configure device aliases so that the pWWN of the SAN connected to MDS4 gets the
alias ARRAY_1_SAN_B_PORT_1, and the pWWN of the Emulex CNA server gets
the alias EMULEX_CNA_SERVER_SAN_A.
Configure zoning in VSAN 101 using these aliases so that the CNA server can talk to
the SAN.
When complete, the CNA server should mount a volume called FCIP1, and SCSI
reads and writes to this volume should be load balanced across both FCIP tunnels
between the MDS switches.
Configuration
3750G-2:
vlan 1691,1692
!
!
!
!
N5K1:
feature fex
feature fcoe
!
vlan 1101
fcoe vsan 101
!
vsan database
vsan 101
!
interface fc2/1 - 2
switchport mode E
no shutdown
!
fex associate 101
!
no shutdown
!
interface vfc111
no shutdown
!
vsan database
!
!
device-alias name ARRAY_1_SAN_B_PORT_1 pwwn 21:02:00:1b:32:47:32:23
device-alias commit
!
!
zone name CNA_SERVER_TO_SAN_OVER_FCIP vsan 101
member device-alias EMULEX_CNA_SERVER_SAN_A
member device-alias ARRAY_1_SAN_B_PORT_1
!
zoneset name VSAN101_ZONESET vsan 101
member CNA_SERVER_TO_SAN_OVER_FCIP
!
zoneset activate name VSAN101_ZONESET vsan 101
MDS3:
vsan database
vsan 101
!
!
interface fc1/1 - 2
switchport mode E
no shutdown
!
feature fcip
!
ip address 169.254.1.63 255.255.255.0
switchport mtu 8000
no shutdown
!
ip address 169.254.2.63 255.255.255.0
switchport mtu 8000
no shutdown
!
fcip profile 1
ip address 169.254.1.63
!
fcip profile 2
ip address 169.254.2.63
!
interface fcip1
use-profile 1
peer-info ipaddr 169.254.1.64
no shutdown
!
interface fcip2
use-profile 2
no shutdown
MDS4:
vsan database
vsan 101
!
!
interface fc1/9
switchport mode F
no shutdown
!
feature fcip
!
ip address 169.254.1.64 255.255.255.0
switchport mtu 8000
no shutdown
!
ip address 169.254.2.64 255.255.255.0
switchport mtu 8000
no shutdown
!
fcip profile 1
ip address 169.254.1.64
!
fcip profile 2
ip address 169.254.2.64
!
interface fcip1
use-profile 1
no shutdown
!
interface fcip2
use-profile 2
no shutdown
Verification
Fibre Channel over IP (FCIP) is a tunneling technique that is used to transport
native Fibre Channel traffic inside of TCP and IP. Unlike the iSCSI Virtual Target
feature of the MDS, an FCIP tunnel does not perform any sort of protocol
conversion, but instead is more analogous to how a GRE tunnel works in the IP
world. FCIP is sometimes called SAN Extension, because by adding the
retransmission and resequencing capabilities native to TCP on top of Fibre Channel,
the SAN can be extended over further physical distances while at the same time
guaranteeing reliability.
The FCIP tunnel configuration is very straightforward, as shown in this example.
First, IP connectivity is established between the switches over their GigE ports. In
this case the switches are in the same IP subnet, but if routing was required it would
be configured simply with static routes. Jumbo Frame support should be enabled,
just like in the iSCSI Virtual Target configuration, to allow for the larger native FC
packets to be fit inside a TCP payload.
MDS3# sh run interface gigabitethernet 1/1
version 4.1(3a)
ip address 169.254.1.63 255.255.255.0
switchport mtu 8000
no shutdown
MDS3# ping 169.254.1.64

PING 169.254.1.64 (169.254.1.64) 56(84) bytes of data.
--- 169.254.1.64 ping statistics --5 packets transmitted, 5 received, 0% packet loss, time 3996ms
rtt min/avg/max/mdev = 0.381/0.413/0.456/0.039 ms
Next, an FCIP profile is created that references the IP address on the GigE link as
the source. This is the logical equivalent of the tunnel source command for a GRE
tunnel in IOS. Note from the output below that other TCP parameters, such as the
keepalive and retransmission time, can be modified, but the default values are fine
for this example.
MDS3# show fcip profile 1
FCIP Profile 1 Internet Address is 169.254.1.63
(interface GigabitEthernet1/1)
Tunnels Using this Profile: fcip1
Listen Port is 3225

TCP parameters
SACK is enabled
PMTU discovery is enabled, reset timeout is 3600 sec
Keep alive is 60 sec
Minimum retransmission timeout is 200 ms
Maximum number of re-transmissions is 4
Send buffer size is 0 KB
Maximum allowed bandwidth is 1000000 kbps
Minimum available bandwidth is 500000 kbps
Configured round trip time is 1000 usec
Congestion window monitoring is enabled, burst size is 50 KB

Auto jitter detection is enabled
Now that we know from where to source the FCIP tunnel, a logical FCIP interface is
created that calls the profile and specifies the other end of the tunnel. The peer-info
command below is then the logical equivalent of the tunnel destination command
for a GRE tunnel.
MDS3# show run interface fcip1
version 4.1(3a)
interface fcip1
use-profile 1
no shutdown
Assuming that the switch on the other end of the tunnel is properly configured, a
TCP session should now form between the endpoints, as shown below.
MDS3# show fcip counters
fcip1
TCP Connection Information
2 Active TCP connections
Control connection: Local 169.254.1.63:3225, Remote 169.254.1.64:65500
Data connection: Local 169.254.1.63:3225, Remote 169.254.1.64:65502
32 Attempts for active connections, 6 close of connections

TCP Parameters
Path MTU 8000 bytes
Current retransmission timeout is 200 ms
Round trip time: Smoothed 2 ms, Variance: 1 Jitter: 195 us
Advertized window: Current: 32 KB, Maximum: 31 KB, Scale: 5
Peer receive window: Current: 25 KB, Maximum: 25 KB, Scale: 5
Congestion window: Current: 43 KB, Slow start threshold: 112 KB
Current Send Buffer Size: 31 KB, Requested Send Buffer Size: 0 KB
CWM Burst Size: 50 KB
Measured RTT : 0 us Min RTT: 0 us Max RTT: 0 us
5272 frames input, 779640 bytes
2288 Class F frames input, 232548 bytes
2984 Class 2/3 frames input, 547092 bytes
0 Reass frames
0 Error frames timestamp error 0
309765 frames output, 649335820 bytes

2207 Class F frames output, 260232 bytes
307558 Class 2/3 frames output, 649075588 bytes
0 Error frames
Although the FCIP tunnel transport is IP, it is now treated like a normal Fibre
Channel link. Specifically, it is treated as a Trunking Expansion Port (TE_Port), just
like a native Fibre Channel link between the switches.
MDS3# show interface fcip1
fcip1 is trunking
Hardware is GigabitEthernet
Port WWN is 20:10:00:0d:ec:26:ea:80
Peer port WWN is 20:10:00:0d:ec:3f:8a:00
Admin port mode is auto, trunk mode is on
snmp link state traps are enabled Port mode is TE
Port vsan is 1
Speed is 1 Gbps
Trunk vsans (admin allowed and active) (1,101) Trunk vsans (up)
Trunk vsans (isolated)
()
Trunk vsans (initializing)
()
Interface last changed at Sat Sep
(1,101)
4 04:24:23 1982
<snip>
The link also participates in normal Fabric Services, such as FCNS, FSPF, Zoning,
Device Aliases, etc., just like any other native FC link in the topology. Because we
configured two separate FCIP tunnels, they are seen as equal-cost paths between
the MDS switches.
MDS3# show fspf internal route vsan 101
FSPF Unicast Routes

--------------------------VSAN Number
Dest Domain
Route Cost
Next hops
----------------------------------------------- 101
fcip2
101
0xef(239)
500
fc1/1
fc1/2
0xee(238)
1000
fcip1
If the fabric is end to end, we should see the pWWNs of the target and initiator in the
FCNS database of the MDSes and the N5K.
VSAN 101:
-------------------------------------------------------------------------FCID
TYPE
PWWN
(VENDOR)
FC4-TYPE:FEATURE
-------------------------------------------------------------------------0xee0000
21:02:00:1b:32:47:32:23
scsi-fcp:target
[ARRAY_1_SAN_B_PORT_1]
0xef0000
10:00:00:00:c9:bb:19:9f (Emulex)
ipfc scsi-fcp:init

VSAN 101:
-------------------------------------------------------------------------FCID
TYPE
PWWN
(VENDOR)
FC4-TYPE:FEATURE
-------------------------------------------------------------------------0xee0000
21:02:00:1b:32:47:32:23
scsi-fcp:target
0xef0000
10:00:00:00:c9:bb:19:9f (Emulex)
ipfc scsi-fcp:init

VSAN 101:
-------------------------------------------------------------------------FCID
TYPE
PWWN
(VENDOR)
FC4-TYPE:FEATURE
-------------------------------------------------------------------------0xee0000
21:02:00:1b:32:47:32:23 (Qlogic)
scsi-fcp:target
0xef0000
10:00:00:00:c9:bb:19:9f (Emulex)
ipfc scsi-fcp:init
Before the initiator can actually talk to the target, zoning must also be configured.
MDS4# show zoneset active vsan 101
zoneset name VSAN101_ZONESET vsan 101

zone name CNA_SERVER_TO_SAN_OVER_FCIP vsan 101
* fcid 0xef0000 [device-alias EMULEX_CNA_SERVER_SAN_A]
* fcid 0xee0000 [device-alias ARRAY_1_SAN_B_PORT_1]
The Emulex CNA server should now automatically mount the LUN presented by the
SAN, as shown below.
Traffic can be generated in the data plane using the Disk Speed Test app on the
server.
Throughput to the disks should be higher than the iSCSI Virtual Target
configuration, as less processing is required to simply encapsulate the traffic into
TCP and IP than to do an upper-layer protocol conversion, but it should be a little
slower than native 2Gbps Fibre Channel because some overhead is still incurred by
the tunneling. Verification of the counters of the FCIP tunnel interfaces should
indicate that both links are being used.
MDS4# show interface fcip1 - 2 | include "fcip|input rate|output rate"
fcip1 is trunking
fcip2 is trunking
Nexus Technology Labs - Classical Ethernet

Switching
Core Nexus LAN Switching
Task
Configure N7K1, N7K2, N5K1, and N5K2 as follows so that IP connectivity is
established between Server 1 and Server 2:
Create VLAN 10 on N7K1, N7K2, N5K1, and N5K2.
Assign VLAN 10 as the access VLAN for N5K1's connection to Server 1;
Server 1 should use the IP address 10.0.0.1/24.
Assign VLAN 10 as the access VLAN for N5K2's connection to Server 2;
Server 2 should use the IP address 10.0.0.2/24.
Statically set these links to 1Gbps on N5K1 and N5K2, respectively.
Configure 802.1Q Trunking between the switches as follows:
N5K1's links to N7K1 should all be trunks.
Edit the trunking allowed list so that only VLANs 1 and 10 will forward over
these links.
Disable all other interconnections between the switches.
When complete, Server 1 and Server 2 should have IP connectivity to each other.
Configuration
N5K1:
vlan 1,10
!
speed 1000
no shutdown
!
shutdown
!
no shutdown
!
no shutdown
!
shutdown
N5K2:
vlan 1,10
!
shutdown
!
speed 1000
no shutdown
!
shutdown
!
no shutdown
!
no shutdown
!
shutdown
N7K1:
vlan 1,10
!
interface Ethernet1/1 - 2, Ethernet2/1 - 4
switchport
no shutdown
N7K2:
vlan 1,10
!
switchport
no shutdown
Verification
Server 1 is able to reach Server 2 on the local LAN segment:
Only the required interfaces are connected. Links to the end servers run at 1Gbps,
whereas the trunks between the switches run at 10Gbps:
N5K1# show interface status
-------------------------------------------------------------------------------Port
Name
Status
Vlan
Duplex
Speed
Type
-------------------------------------------------------------------------------Eth1/1
--
connected 10
full
1000
SFP-1000BAS
Eth1/2
--
disabled
full
10G
SFP-1000BAS
Eth1/3
--
disabled
full
10G
SFP-H10GB-C
Eth1/4
--
disabled
full
10G
SFP-H10GB-C
Eth1/5
--
disabled
full
10G
SFP-H10GB-C
Eth1/6
--
connected trunk
full
10G
SFP-H10GB-C Eth1/7
SFP-H10GB-C
--
connected trunk
full
10G
Eth1/8
--
disabled
full
10G
SFP-H10GB-C
Eth1/9
--
disabled
full
10G
SFP-H10GB-C
VLAN 10 is created and forwarding on the required interfaces:

N5K1# show vlan brief
VLAN Name
Status
Ports
---- -------------------------------- --------- ------------------------------1
default
active
Eth1/2, Eth1/3, Eth1/4, Eth1/5

Eth1/10, Eth1/11, Eth1/12
Eth1/13, Eth1/14, Eth1/15
Eth1/16, Eth1/17, Eth1/18
Eth1/19, Eth1/20, Eth1/21
Eth1/22, Eth1/23, Eth1/24
Eth1/25, Eth1/26, Eth1/27
Eth1/28, Eth1/29, Eth1/30
Eth1/31, Eth1/32
10
VLAN0010
active
Eth1/1, Eth1/6, Eth1/7
VLANs 1 and 10 are trunking on the required interfaces, and other VLANs have
been removed from the allowed list:
N7K1# show interface trunk
-------------------------------------------------------------------------------- Port
Native
Status
Port
Vlan
Channel
-------------------------------------------------------------------------------- Eth1/1
-- Eth1/2
1 trunking
-- Eth2/1
1 trunking
-- Eth2/2
1 trunking
-- Eth2/3
1 trunking
-- Eth2/4
1 trunking
1 trunking
--
-------------------------------------------------------------------------------- Port
Vlans Allowed on Trunk
-------------------------------------------------------------------------------- Eth1/1
Eth1/2
1,10
Eth2/1
1,10
Eth2/2
1,10
Eth2/3
1,10
1,10
-------------------------------------------------------------------------------Port
Vlans Err-disabled on Trunk
-------------------------------------------------------------------------------Eth1/1
none
Eth1/2
none
Eth2/1
none
Eth2/2
none
Eth2/3
none
Eth2/4
none
-------------------------------------------------------------------------------- Port STP Forwarding

-------------------------------------------------------------------------------- Eth1/1
Eth1/2
none
Eth2/1
none
Eth2/2
none Eth2/3
Eth2/4
1,10
1,10
1,10
-------------------------------------------------------------------------------- Port
Vlans in spanning tree forwarding state and not pruned
-------------------------------------------------------------------------------- Eth1/1
Eth1/2
none
Eth2/1
none
Eth2/2
none Eth2/3
Eth2/4
1,10
1,10
<snip>
All four switches agree on who the root of the Spanning-Tree is for VLAN 10, and
alternate paths to the root bridge are blocked:
VLAN0010
Spanning tree enabled protocol rstp Root ID
Address
Bridge ID
Interface
Priority
32778
0026.980c.2141
Cost
Port
134 (Ethernet1/6)
Hello Time
Priority
32778
Address
547f.ee79.137c
Hello Time
sec
sec
Role Sts Cost
Max Age 20 sec
Max Age 20 sec
Prio.Nbr Type
1,10
---------------- ---- --- --------- -------- -------------------------------- Eth1/1

4
128.129
P2p Eth1/6
Root FWD
128.134
P2p Eth1/7
Altn BLK
128.135
P2p
Desg FWD
VLAN0010
Address
Priority
32778
0026.980c.2141
Bridge ID
Cost
Port
134 (Ethernet1/6)
Hello Time
Priority
32778
Address
547f.ee7a.4d7c
Hello Time
Interface
sec
sec
Role Sts Cost
Max Age 20 sec
Max Age 20 sec
Prio.Nbr Type
---------------- ---- --- --------- -------- -------------------------------- Eth1/2

4
128.130
P2p Eth1/6
Root FWD
128.134
P2p Eth1/7
Altn BLK
128.135
P2p
Desg FWD
VLAN0010
Address
32778
0026.980c.2141
Bridge ID
Interface
Priority
Cost
Port
129 (Ethernet1/1)
Hello Time
Priority
32778
Address
68bd.abd7.6041
Hello Time
sec
sec
Role Sts Cost
Max Age 20 sec
Max Age 20 sec
Prio.Nbr Type
---------------- ---- --- --------- -------- -------------------------------- Eth1/1

2
128.129
P2p Eth1/2
Altn BLK
128.130
P2p Eth2/1
Altn BLK
128.257
P2p Eth2/2
Altn BLK
128.258
P2p Eth2/3
Desg FWD
128.259
P2p Eth2/4
Desg FWD
128.260
P2p
VLAN0010
Root FWD

Address
Priority
32778
0026.980c.2141
Bridge ID
Interface
Hello Time
sec
Max Age 20 sec
Priority
32778
Address
0026.980c.2141
Hello Time
sec
Max Age 20 sec
Role Sts Cost
Prio.Nbr Type
---------------- ---- --- --------- -------- -------------------------------- Eth1/1

2
128.129
P2p Eth1/2
Desg FWD
128.130
P2p Eth2/1
Desg FWD
128.257
P2p Eth2/2
Desg FWD
128.258
P2p Eth2/3
Desg FWD
128.259
P2p Eth2/4
Desg FWD
128.260
P2p
Server 1 and Server 2's MAC addresses are learned in the VLAN 10 CAM table:
N5K1# show mac address-table dynamic vlan 10
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link
VLAN
MAC Address
Type
age
Secure NTFY
Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+------------------ *
10
* 10
000c.29d0.a588
000c.29f8.57ec
dynamic
dynamic
180
250
F
F
Eth1/6
F
Eth1/1
Desg FWD

Switching
VLAN Trunking Protocol (VTP)
Task
Configure trunking between N7K1, N7K2, N5K1, and N5K2 as follows:
Configure VLAN Trunking Protocol (VTP) on the switches as follows:
Enable VTP on all switches.
Use the VTP domain name NEXUS.
N5K1 should run in VTP client mode.
N5K2 should run in VTP off mode.
N7K1 should run in VTP transparent mode.
N7K2 should run in VTP server mode.
Configure VTP authentication on N5K1 and N7K2 using the password
VTPPASSWORD.
Configure the following VLANs on N7K2:
VLAN 10 named TEN.
VLAN 20 named TWENTY.
VLAN 1000 named ONETHOUSAND.
VLAN 2000 named TWOTHOUSAND.
Configure VLAN 1500 named FIFTEENHUNDRED on N5K1.
Configure VLAN 3000 named THREETHOUSAND on N7K1 and N5K2.
Verify the resulting VLAN databases on the switches.
Configuration
N5K1:
feature vtp
vtp domain NEXUS
vtp mode client
vtp password VTPPASSWORD
!
vlan 1500
name FIFTEENHUNDRED
!
shutdown
!
no shutdown
!
shutdown
N5K2:
feature vtp
vtp domain NEXUS
vtp mode off
vlan 3000
name THREETHOUSAND
!
shutdown
!
no shutdown
!
shutdown
N7K1:
feature vtp
vtp domain NEXUS
vtp mode transparent
!
vlan 3000
name THREETHOUSAND
!
switchport
no shutdown
N7K2:
feature vtp
vtp domain NEXUS
vtp mode server
vtp password VTPPASSWORD
!
vlan 10
name TEN
vlan 20
name TWENTY
vlan 1000
name ONETHOUSAND
vlan 2000
name TWOTHOUSAND
!
switchport
no shutdown
Verification
Like Catalyst IOS, Nexus NX-OS supports VLAN Trunking Protocol (VTP) to ease in
the administration of VLAN creation, deletion, and naming throughout the LAN.
However, there are a few key functional differences between NX-OSs
implementation of VTP and most other platforms.
The first, as shown in this example, is that NX-OS supports VTP mode off. In this
mode, all received VTP updates are filtered, and the VLAN database is only locally
significant. This is unlike VTP Transparent mode, which passes VTP updates
between interfaces, even though its database is locally significant as well. Another
difference, as of the version in this example, is that Nexus 5K does not support VTP
pruning. Also note that although extended VLANs can be configured in all four VTP
modes, they still cannot be advertised by either servers or clients because there is
currently no support for VTP version 3.
The end result is that it is fairly easy for the VLAN databases to become out of sync
when there is a mix of VTP modes in the network, not to mention the same potential
problems as in Catalyst IOS of mistakes in the configuration of VTP servers possibly
deleting VLANs, or the entire VLAN database, in the entire network. For this reason,
it is very typical that VTP is not used in Data Center LAN environments, although
the feature is still technically supported.
Verification of VTP is similar to that of Catalyst IOS, as follows:
N5K1# show vtp status
VTP Status Information

---------------------VTP Version
: 2 (capable) Configuration Revision
: 16
Maximum VLANs supported locally : 1005

Number of existing VLANs
: 8 VTP Operating Mode
VTP Domain Name
: NEXUS
VTP Pruning Mode
: Disabled (Operationally Disabled)
VTP V2 Mode
: Disabled
VTP Traps Generation
: Disabled MD5 Digest
: Client
0xEC 0x88 0x7C 0xB6 0x3E 0x9E 0x73 0xF2

Configuration last modified by 0.0.0.0 at 2-13-13 17:47:19
VTP version running
: 1
VLAN Name
Status
Ports
---- -------------------------------- --------- ------------------------------1
default
active

Eth1/10, Eth1/11, Eth1/12
Eth1/13, Eth1/14, Eth1/15
Eth1/16, Eth1/17, Eth1/18
Eth1/19, Eth1/20, Eth1/21
Eth1/22, Eth1/23, Eth1/24
Eth1/25, Eth1/26, Eth1/27
Eth1/28, Eth1/29, Eth1/30
Eth1/31, Eth1/32 10
active
active
active
Eth1/1, Eth1/6, Eth1/7 20
TEN
TWENTY
Eth1/6, Eth1/7 1000 ONETHOUSAND
Eth1/6, Eth1/7
1002 fddi-default
suspended Eth1/6, Eth1/7
1003 token-ring-default
1004 fddinet-default
1005 trnet-default
suspended Eth1/6, Eth1/7 1500 FIFTEENHUNDRED

active
Eth1/6, Eth1/7
N5K2# show vtp status

---------------------VTP Version
:0

VTP Domain Name
: NEXUS
VTP Pruning Mode
VTP V2 Mode
: Disabled
: Disabled
: Off
MD5 Digest
: 0x04 0x6C 0xBC 0xB5 0x6F 0xCE 0x92 0xC9

VTP version running
: 1
VLAN Name
Status
Ports
---- -------------------------------- --------- ------------------------------1
default
active

Eth1/10, Eth1/11, Eth1/12
Eth1/13, Eth1/14, Eth1/15
Eth1/16, Eth1/17, Eth1/18
Eth1/19, Eth1/20, Eth1/21
Eth1/22, Eth1/23, Eth1/24
Eth1/25, Eth1/26, Eth1/27
Eth1/28, Eth1/29, Eth1/30
Eth1/31, Eth1/32 3000 THREETHOUSAND
active
Eth1/6, Eth1/7
N7K1-1# show vtp status

---------------------VTP Version
:0

VTP Domain Name
: NEXUS
: Transparent
VTP Pruning Mode
VTP V2 Mode
: Disabled
: Disabled
MD5 Digest
: 0x93 0x5A 0xCB 0xBB 0xDA 0xC8 0x15 0x37

VTP version running
: 1
N7K1-1# show vlan brief
VLAN Name
Status
Ports
---- -------------------------------- --------- ------------------------------1
default
active

Eth2/7, Eth2/8
3000 THREETHOUSAND
active

Eth2/3, Eth2/4N7K2-1# show vtp status

---------------------VTP Version

: Server
: 16
VTP Domain Name
: NEXUS
VTP Pruning Mode
VTP V2 Mode
: Disabled
: Disabled MD5 Digest
0xEC 0x88 0x7C 0xB6 0x3E 0x9E 0x73 0xF2

Local updater ID is 0.0.0.0
VTP version running
: 1
N7K2-1# show vlan brief
VLAN Name
Status
Ports
---- -------------------------------- --------- ------------------------------1
default
active

Eth2/7, Eth2/8 10
active

Eth2/3, Eth2/4 20
active
TEN
TWENTY

Eth2/3, Eth2/4 1000 ONETHOUSAND
active

Eth2/3, Eth2/4
1002 fddi-default
suspended Eth1/1, Eth1/2, Eth2/1, Eth2/2
1003 token-ring-default
Eth2/3, Eth2/4
Eth2/3, Eth2/4
1004 fddinet-default

Eth2/3, Eth2/4
1005 trnet-default

Eth2/3, Eth2/4 2000 TWOTHOUSAND
active

Eth2/3, Eth2/4

Switching
Port Channels
Task
Configure port channels between N7K1, N7K2, N5K1, and N5K2 as follows:
The links between N5K1 and N7K1 should be grouped together in PortChannel 1. Do not use a negotiation protocol for this channel.
The links between N7K1 and N7K2 should be grouped into Port-Channels 2
and 3. Both switches should initiate LACP negotiation for both of these
channels and use LACP fast timers.
The links between N5K2 and N7K2 should grouped together into PortChannel 4. N7K2 should initiate LACP negotiation and N5K2 should
respond.
Configure all of these port channels as 802.1Q trunk links.
Configure N7K2 with an LACP priority of 16384 so that it is the preferred device for
managing negotiation of its port channels.
Configure all switches to use source and destination TCP/UDP ports for load
balancing flows across the port channel members.
Configure N5K1's link to Server 1 and N5K2's link to Server 2 in VLAN 10.
Server 1 should use the IP address 10.0.0.1/24 and Server 2 should use the IP
address 10.0.0.2/24. When complete, Server 1 and Server 2 should have IP
reachability to each other.
Use the iPerf application to verify that traffic flows between Server 1 and Server 2 are
being distributed among the member links of the port channels.
Configuration
N5K1:
vlan 10
!
!
speed 10000
!
speed 1000
!
shutdown
!
channel-group 1
!
channel-group 1
!
shutdown
N5K2:
feature lacp
!
vlan 10
!
!
speed 10000
!
shutdown
!
speed 1000
!
shutdown
!
channel-group 4 mode passive
!
channel-group 4 mode passive
!
shutdown
N7K1:
feature lacp
!
vlan 10
!
port-channel load-balance src-dst l4port
!
switchport
!
switchport
!
switchport
!
lacp rate fast
switchport
no shutdown
!
lacp rate fast
switchport
no shutdown
!
lacp rate fast
no shutdown
!
lacp rate fast
no shutdown
!
channel-group 1
no shutdown
!
channel-group 1
no shutdown
N7K2:
feature lacp
!
vlan 10
!
lacp system-priority 16384
port-channel load-balance src-dst l4port
!
switchport
!
switchport
!
switchport
!
lacp rate fast
switchport
no shutdown
!
lacp rate fast
switchport

no shutdown
!
lacp rate fast
no shutdown
!
lacp rate fast
no shutdown
!
no shutdown
!
no shutdown
Verification
Port channels in NX-OS, just like in Catalyst IOS and other platforms, require that
the member interfaces first have compatible parameters for the channel to form. In
NX-OS, these parameters can be verified with the command show port-channel
compatibility-parameters . Some of these parameters can be seen below:
N7K1-1# show port-channel compatibility-parameters | include \*
* port mode
* speed
* MTU
* MEDIUM
* Span mode
<snip>
In this topology, the links between the Nexus 7000 switches include both M series
and F series modules. Because these modules have different port level capabilities,
they are not compatible to channel together. This is why the above solution has one
port channel for the M1 ports and another for the F1 ports. The NX-OS parser will
detect this and return an error message if you attempt to channel together
incompatible port types, as shown below:
N7K1-1# config t
End with CNTL/Z.
N7K1-1(config)# int e1/1 - 2, e2/1 - 2

N7K1-1(config-if-range)# channel-group 10 mode active
N7K1-1 %ETH_PORT_CHANNEL-3-COMPAT_CHECK_FAILURE: rate mode is not compatible
command failed: port not compatible [rate mode]
** You can use force option to override the port's parameters

** (e.g. "channel-group X force")
** Use "show port-channel compatibility-parameters" to get more information on failure
After the channels are successfully formed, the show port-channel summary output
should indicate that the member links are "Up in the port-channel" with flag (P). This
output also shows whether LACP negotiation was used or not.
Flags:
D - Down
I - Individual
s - Suspended
r - Module-removed
S - Switched
R - Routed
-------------------------------------------------------------------------------Group Port-
Type
Protocol
Member Ports
Channel
-------------------------------------------------------------------------------1
Po1(SU)
Eth
NONE
Eth1/6(P)
Eth1/7(P)

Flags:
D - Down
I - Individual
s - Suspended
r - Module-removed
S - Switched
R - Routed
-------------------------------------------------------------------------------Group PortChannel
Type
Protocol
Member Ports
-------------------------------------------------------------------------------4
Po4(SU)
Eth
LACP
Eth1/6(P)
Eth1/7(P)

Flags:
D - Down
I - Individual
s - Suspended
r - Module-removed
S - Switched
R - Routed
-------------------------------------------------------------------------------Group Port-
Type
Protocol
Member Ports
Channel
-------------------------------------------------------------------------------1
Po1(SU)
Eth
NONE
Eth2/3(P)
Eth2/4(P)
Po2(SU)
Eth
LACP
Eth1/1(P)
Eth1/2(P)
Po3(SU)
Eth
LACP
Eth2/1(P)
Eth2/2(P)

Flags:
D - Down
I - Individual
s - Suspended
r - Module-removed
S - Switched
R - Routed
-------------------------------------------------------------------------------Group Port-
Type
Protocol
Member Ports
Channel
-------------------------------------------------------------------------------2
Po2(SU)
Eth
LACP
Eth1/1(P)
Eth1/2(P)
Po3(SU)
Eth
LACP
Eth2/1(P)
Eth2/2(P)
Po4(SU)
Eth
LACP
Eth2/3(P)
Eth2/4(P)
Spanning-Tree Protocol should see the port channels as one logical link, as shown
below. Separate channels that point the same direction in the spanning-tree, such
as Port-Channels 2 and 3 below, are still subject to the normal forwarding and
blocking rules.
VLAN0010
Root ID
Priority
32778
Address
0026.980c.2141
Bridge ID
Cost
Port
Hello Time
Priority
32778
Address
68bd.abd7.6041
Hello Time
Interface
sec
Max Age 20 sec
sec
Max Age 20 sec
Role Sts Cost
Prio.Nbr Type
---------------- ---- --- --------- -------- -------------------------------- Po1

1
128.4096 P2p Po2
Root FWD
128.4097 P2p Po3
Altn BLK
128.4098 P2p
Desg FWD
LACP neighbors of N7K2 should see that its System Priority (the first portion of the
System ID) has been reduced to a more preferred value of 16384. The output below
also shows whether the neighbor is running LACP in active or passive mode, and
whether slow or fast LACP hellos are being used.
N7K1-1# show lacp neighbor interface port-channel 2
Flags:
S - Device is sending Slow LACPDUs F - Device is sending Fast LACPDUs
A - Device is in Active mode

P - Device is in Passive mode
port-channel2 neighbors
Partner's information Partner
Port System ID
Eth1/1 16384,0-26-98-c-21-41
Partner Partner
Port Number
0x101
Age Flags
2077 FA
LACP Partner
Partner
Partner
Port Priority
Oper Key
Port State
32768
0x1
0x3f
Partner
Partner
Partner
Port
System ID
Port Number
Age
Flags
Eth1/2
16384,0-26-98-c-21-41
0x102
2076
FA
LACP Partner
Partner
Partner
Port Priority
Oper Key
Port State
32768
0x1
0x3f
To verify the configured load balancing method of the switches, use the
channel load-balance command, as seen below.
show port-
Note that on the Nexus 7000 the load balancing method can only be
changed in the default VDC, as this change is chassis-wide between
all VDCs.
N5K1# show port-channel load-balance
Port Channel Load-Balancing Configuration:

System: source-dest-port
Port Channel Load-Balancing Addresses Used Per-Protocol:

Non-IP: source-dest-mac IP: source-dest-port source-dest-ip source-dest-mac
N7K1# show port-channel load-balance
Port Channel Load-Balancing Configuration:

System: src-dst l4port
Port Channel Load-Balancing Addresses Used Per-Protocol:

Non-IP: src-dst mac IP: src-dst l4port
To verify load distribution among member ports of the channels in the data plane,
Server 1 generates multiple TCP flows to Server 2 using the iPerf testing app.
Verification of the port channel from N5K1 to N7K1 shows a 30-second average
output rate of around 750Mbps. This, however, does not tell us which particular
member interfaces of the channel are being used.
N5K1# show interface port-channel 1 | include "output rate"

, 61.97 Kpps
By looking at the individual counters of the member interfaces, we can see that the
load is being distributed among them. Note that it is not a 1:1 even distribution,
because not every TCP flow is getting the same about of throughput based on other
variables such as burstiness, queueing, etc.
N5K1# show interface e1/6 - 7 | include "Ethernet1/|output rate"
Ethernet1/6 is up

, 39.26 Kpps
Ethernet1/7 is up
, 22.71 Kpps

Switching
Rapid PVST Traffic Engineering
Task
Configure all links connecting N7K1, N7K2, N5K1, and N5K2 as 802.1Q trunk ports.
Configure the links connecting N5K1 and N5K2 as a port channel.
Create VLANs 10 and 20 on all switches, and assign them as follows:
Server 1s link to N5K1 should be in VLAN 10 and use the IP address
10.0.0.1/24.
20.0.0.1/24.
20.0.0.2/24.
10.0.0.2/24.
Configure Spanning-Tree Protocol between the switches as follows:
All switches should use 32 bits for spanning-tree port path costs.
N7K1 should be the STP Root Bridge for VLAN 10, with N7K2 being the
backup Root Bridge.
N5K2 should be the STP Root Bridge for VLAN 20, with N5K1 being the
backup Root Bridge.
Server 1s VLAN 10 traffic to Server 2 should follow the path of N5K1 ->
N7K2 -> N7K1 -> N5K2 -> Server 2.
Server 2s VLAN 20 traffic to Server 1 should follow the path of N5K1 ->
N7K1 -> N7K2 -> N5K2 -> Server 1.
Configuration
N5K1:
vlan 10,20
!
spanning-tree pathcost method long

!
spanning-tree vlan 10,20 cost 99999
speed 10000
!
speed 1000
!
speed 1000
!
channel-group 1
!
channel-group 1
!
channel-group 1
!
spanning-tree vlan 10 cost 99999
!
!
!
N5K2:
vlan 10,20
!

!
speed 10000
!
speed 1000
!
speed 1000
!
channel-group 1
!
channel-group 1
!
channel-group 1
!
!
!
!
N7K1:
vlan 10,20
!
!
switchport
no shutdown
!
switchport
no shutdown
!
no shutdown
!
no shutdown
!
no shutdown
!
no shutdown
!
no shutdown
!
no shutdown
N7K2:
vlan 10,20
!
!
switchport
no shutdown
!
switchport

no shutdown
!
no shutdown
!
no shutdown
!
no shutdown
!
no shutdown
!
no shutdown
!
no shutdown
Verification
NX-OS runs Rapid Per-VLAN Spanning-Tree Protocol by default. This means that
for each VLAN that is created, a separate instance of STP is created, with each of
these running the 802.1w RSTP algorithm. Beyond this, the default behavior of STP
on NX-OS is essentially identical to that of Catalyst IOS.
For the purposes of traffic engineering, the Root Bridge election can be modified on
a per-VLAN basis by changing the STP Bridge-ID Priority value (lower is preferred),
and the Root Port election can be modified on a per-port per-VLAN basis by
changing either the STP Port Cost or STP Port Priority. Changing the Port Cost is
the more common modification, as the Port Priority only comes into play when you
are choosing between multiple links with the same Root Path Cost, and the links
connect to the same upstream switch.
Note that in this example the range of STP cost values was increased from a 16-bit
field to a 32-bit field (the long pathcost method), as newer higher-speed links such
as 10GigE, 40GigE, 100GigE, and beyond start to look the same from a cost point
of view when the default short pathcost method is used. Also note that this
command is applicable only when running Rapid PVST on NX-OS, as Multiple
Spanning-Tree (MST) always uses the longer pathcost length.
Verification of this task can be performed by viewing the Root Bridge and Root Port
election on a per-switch basis, or by viewing the MAC address table, as the STP
topology ultimately controls which interfaces can participate in MAC address
learning. Below we see that for VLAN 10, N7K1 is elected the Root Bridge. This
implies that all of its VLAN 10 links will be Designated ports in the Forwarding state.
VLAN0010
Root ID
Bridge ID
Interface
Priority
4106
Address
68bd.abd7.6041 This bridge is the root
Hello Time
Priority
4106
Address
68bd.abd7.6041
Hello Time
sec
Max Age 20 sec
sec
Max Age 20 sec
Role Sts Cost
Prio.Nbr Type
---------------- ---- --- --------- -------- -------------------------------- Eth1/1 Desg FWD

2000
128.129
P2p Eth1/2 Desg FWD
2000
128.130
P2p Eth2/1 Desg FWD
2000
128.257
P2p Eth2/2 Desg FWD
2000
128.258
P2p Eth2/3 Desg FWD
2000
128.259
P2p Eth2/4 Desg FWD
2000
128.260
P2p Eth2/5 Desg FWD
2000
128.261
P2p Eth2/6 Desg FWD
2000
128.262
P2p
MAC addresses for VLAN 10 are being learned in ports Eth2/5 and Eth1/1, which
implies that N5K2 and N7K2 on the other end of these links, respectively, have
chosen those ports as their Root Ports.
N7K1-1# show mac address-table dynamic vlan 10
Legend:
VLAN
MAC Address
Type
age
Secure NTFY Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+-----------------* 10
000c.29d0.a588
dynamic
30
F Eth2/5
* 10
000c.29f8.57ec
dynamic
30
F Eth1/1
Per the output below, N7K2 chose E1/1 as the Root Port to reach N7K1. Although
all ports have the same cost of 2000, E1/1 has the lowest Port ID (port priority and
port number) on the other end of the link.
VLAN0010
Root ID
Bridge ID
Interface
Priority
4106
Address
68bd.abd7.6041
Cost
2000
Port
129 (Ethernet1/1)
Hello Time
Priority
8202
Address
0026.980c.2141
Hello Time
sec
sec
Role Sts Cost
Max Age 20 sec
Max Age 20 sec
Prio.Nbr Type
---------------- ---- --- --------- -------- -------------------------------Eth1/1

128.129
Root FWD 2000

P2p
Eth1/2
Altn BLK 2000
128.130
P2p
Eth2/1
Altn BLK 2000
128.257
P2p
Eth2/2
Altn BLK 2000
128.258
P2p
Eth2/3
Desg FWD 2000
128.259
P2p
Eth2/4
Desg FWD 2000
128.260
P2p
Eth2/5
Desg FWD 2000
128.261
P2p
Eth2/6
Desg FWD 2000
128.262
P2p
Per the view of the CAM table below, we see that N7K2 learns MAC addresses for
VLAN 10 in Eth1/1, its root port, and Eth2/5, the downstream link connecting to
N5K1.
Legend:
VLAN
MAC Address
Type
age
---------+-----------------+--------+---------+------+----+-----------------* 10
000c.29d0.a588
dynamic
30
Eth1/1
* 10
000c.29f8.57ec
dynamic
30
Eth2/5
On the next downstream switch, N5K1, we see that it has chosen Eth1/8, a link to
N7K2, as its Root Port. This is because other possible paths to the Root Bridge
have had their cost raised to 99999. The end result is that traffic received from
Server 1 in VLAN 10 going to Server 2 is first forwarded to N7K2, then to N7K1,
then to N5K2, and finally to Server 2.
VLAN0010
Root ID
Bridge ID
Interface
Priority
4106
Address
68bd.abd7.6041
Cost
4000
Port
136 (Ethernet1/8)
Hello Time
Priority
32778
Address
547f.ee79.137c
Hello Time
sec
Max Age 20 sec
sec
Role Sts Cost
Max Age 20 sec
Prio.Nbr Type
---------------- ---- --- --------- -------- -------------------------------Po1
Altn BLK 99999
128.4096 P2p
Eth1/1
Desg FWD 20000
128.129
P2p
Eth1/6
Altn BLK 99999
128.134
P2p
Eth1/7
Altn BLK 99999
128.135
P2p Eth1/8
128.137
P2p
128.136
Eth1/9
Root FWD 2000
P2p
Altn BLK 2000
N5K1# show mac address-table dynamic vlan 10v

Legend:
VLAN
MAC Address
Type
age
Secure NTFY
Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+------------------ *
10
000c.29d0.a588
* 10
000c.29f8.57ec
dynamic
130
dynamic
130
F
F
Eth1/8
F
Eth1/1
Likewise, traffic in VLAN 20 from Server 2 can be verified to follow the path of
N5K1 -> N7K1 -> N7K2 -> N5K2 -> Server 1 by the CAM tables below.
Legend:
VLAN
MAC Address
Type
age
Secure NTFY
Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+------------------ *
20
000c.29d0.a57e
* 20
000c.29f8.57f6
dynamic
10
dynamic
10
F
F
Eth1/2
F
Eth1/6

Legend:
VLAN
MAC Address
Type
age
---------+-----------------+--------+---------+------+----+------------------ *
20
000c.29d0.a57e
* 20
000c.29f8.57f6
dynamic
dynamic
F
30
F
F
Eth2/3
F
Eth1/1

Legend:
VLAN
MAC Address
Type
age
---------+-----------------+--------+---------+------+----+------------------ *
20
000c.29d0.a57e
* 20
000c.29f8.57f6
dynamic
dynamic
30
30
F
F
Eth1/1
F
Eth2/3

Legend:
VLAN
MAC Address
Type
age
Secure NTFY
Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+------------------ *
20
* 20
000c.29d0.a57e
000c.29f8.57f6
dynamic
dynamic
20
20
F
F
Eth1/6
F
Eth1/1

Switching
MSTP Traffic Engineering
Task
Configure the links between N7K1, N7K2, N5K1, and N5K2 as port channels and
802.1q trunk links as follows:
Use Port-Channel 1 on the F ports between N7K1 and N7K2.
Use Port-Channel 2 between N7K1 and N5K1.
Disable all other ports.
Create VLANs 10, 20, 30, 40, 50, and 60 on all switches.
Configure Multiple Spanning-Tree Protocol between the switches as follows:
Configure the switches in MST region 100 named MST100.
VLANs 10, 20, and 30 should be in MST instance 1.
VLANs 40, 50, and 60 should be in MST instance 2.
N7K1 should be the root bridge for MST instance 1.
N7K2 should be the root bridge for MST instance 2.
N5K2 should be the CIST root bridge.
Configure MST traffic engineering as follows:
N5K2 should forward traffic through N5K1 to get to N7K2.
N7K2 should forward traffic through N5K2 to get to N7K1.
Configuration
N5K1:
feature lacp
!
vlan 10,20,30,40,50,60
!
spanning-tree mode mst

spanning-tree mst configuration
name MST100
revision 100
instance 1 vlan 10,20,30
!
!
!
N5K2:
feature lacp
!
vlan 10,20,30,40,50,60
!
spanning-tree mst 0 priority 4096
name MST100
revision 100
!
!
!
!
spanning-tree mst 2 cost 99999
!
N7K1:
feature lacp
!
vlan 10,20,30,40,50,60
!
name MST100
revision 100
!
!
!
N7K2:
feature lacp
!
vlan 10,20,30,40,50,60
!
name MST100
revision 100
!
!

!
!
!
Verification
N5K2# show spanning-tree mst 0
##### MST0
vlans mapped:
Bridge
address 000d.eca4.743c
Root
this switch for the CIST
1-9,11-19,21-29,31-39,41-49,51-59,61-4094
priority
4096
(4096 sysid 0)
Regional Root this switch

Operational
hello time 2 , forward delay 15, max age 20, txholdcount 6
Configured
hello time 2 , forward delay 15, max age 20, max hops
Interface
Role Sts Cost
20
Prio.Nbr Type
---------------- ---- --- --------- -------- -------------------------------Po3
Desg BLK 1000
128.4098 P2p
Po5
Desg FWD 1000
128.4100 P2p
Po6
Desg FWD 660
128.4101 P2p
N7K1-1# show spanning-tree mst 1

##### MST1
vlans mapped:
Bridge
address 68bd.abd7.6041
Root
this switch for MST1
Interface
10,20,30
Role Sts Cost
priority
4097
(4096 sysid 1)
Prio.Nbr Type
---------------- ---- --- --------- -------- -------------------------------Po1
Desg FWD 1000
128.4096 P2p
Po2
Desg FWD 1000
128.4097 P2p
Po3
Desg FWD 1000
128.4098 P2p
##### MST1
vlans mapped:
Bridge
address 0026.980c.2141
priority
32769 (32768 sysid 1)
Root
address 68bd.abd7.6041
priority
4097
port
cost
2000
Po5
10,20,30
(4096 sysid 1)
rem hops 18
Interface
Role Sts Cost
Prio.Nbr Type
---------------- ---- --- --------- -------- -------------------------------Po1
Altn BLK 99999
128.4096 P2p
Po4
Altn BLK 99999
128.4099 P2p Po5
FWD 1000
Root
128.4100 P2p

##### MST2
vlans mapped:
Bridge
address 0026.980c.2141
Root
this switch for MST2
Interface
40,50,60
Role Sts Cost
priority
4098
(4096 sysid 2)
Prio.Nbr Type
---------------- ---- --- --------- -------- -------------------------------Po1
Desg FWD 1000
128.4096 P2p
Po4
Desg FWD 1000
128.4099 P2p
Po5
Desg FWD 1000
128.4100 P2p
N5K2# show spanning-tree mst 2
##### MST2
vlans mapped:
Bridge
address 000d.eca4.743c
priority
32770 (32768 sysid 2)
Root
address 0026.980c.2141
priority
4098
port
cost
1660
Interface
40,50,60
Po6
Role Sts Cost
(4096 sysid 2)
rem hops 18
Prio.Nbr Type
---------------- ---- --- --------- -------- -------------------------------Po3
Altn BLK 99999
128.4098 P2p
Po5
Altn BLK 99999
128.4100 P2p Po6
660
128.4101 P2p
Root FWD

Switching
STP Bridge Assurance
Task
Configure the links between N7K1, N7K2, N5K1, and N5K2 as port channels and
802.1q trunk links as follows:
Use Port-Channel 1 on the F ports between N7K1 and N7K2.
Disable all other ports.
Create VLANs 10, 20, 30, 40, 50, and 60 on N7K1 and N7K2.
Create VLANs 10, 20, and 30 on N5K1.
Create VLANs 40, 50, and 60 on N5K2.
Configure N7K1 as the STP root bridge for all VLANs.
Configure all trunk links as STP port type network.
Configuration
N5K1:
feature lacp
!
vlan 10,20,30
!
!

!
N5K2:
feature lacp
!
vlan 40,50,60
!
!
!
N7K1:
feature lacp
!
vlan 10,20,30,40,50,60
!
spanning-tree vlan 10,20,30,40,50,60 priority 4096
!
!
!

N7K2:
feature lacp
!
vlan 10,20,30,40,50,60
!
!
!
Verification
Spanning-tree Bridge Assurance is an STP enhancement to help prevent against
unidirectional links, and to also automatically prune unneeded VLANs from trunk
links. Like STP Loopguard or UDLD, STP Bridge Assurance uses a keepalive, the
STP BPDU in this case, to make sure that both ends of the link can both send and
receive packets. Unlike UDLD, though, this keepalive is on a per-VLAN basis (or perSTP instance basis in the case of MST). The advantage of using this feature is that
it not only prevents against unidirectional links, but has a behavior similar to VTP
pruning, which means that you do not need to manually edit the trunking allowed list
to remove unneeded VLANs off trunks. Bridge Assurance is configured by setting
the spanning-tree port-type network at the link level.
In the below output on N7K1, we see that VLAN 10 is not forwarding toward N5K2
on Port-Channel 3, and that VLAN 40 is not forwarding toward N5K1 on PortChannel 2. Note that in addition to the port being in the blocking state, it is denoted
as Bridge Assurance Inconsistent. This essentially means that when N7K1 sent a
BPDU keepalive for VLAN 10 out Po3, it did not receive a response back in. The
end result is that the VLAN is blocked, and hence pruned.
VLAN0010
Root ID
Priority
4106
Address
68bd.abd7.6041
Bridge ID
Hello Time
Priority
4106
Address
68bd.abd7.6041
Hello Time
Interface
sec
sec
Max Age 20 sec
Max Age 20 sec
Role Sts Cost
Prio.Nbr Type
---------------- ---- --- --------- -------- -------------------------------Po1
Desg FWD 1
128.4096 Network P2p
Po2
Desg FWD 1
Po3
Desg BKN*1
128.4098 Network P2p *BA_Inc
VLAN0040
Root ID
Priority
4136
Address
68bd.abd7.6041
Bridge ID
Hello Time
Priority
4136
Address
68bd.abd7.6041
Hello Time
Interface
sec
sec
Role Sts Cost
Max Age 20 sec
Max Age 20 sec
Prio.Nbr Type
---------------- ---- --- --------- -------- -------------------------------Po1
Desg FWD 1
Po2
Desg BKN*1
128.4097 Network P2p *BA_Inc
Po3
Desg FWD 1
If a new VLAN is added, Bridge Assurance automatically reacts to it by either

pruning or un-pruning the VLAN, depending on whether the keepalive is received bidirectionally, as seen below.
N7K1-1# conf t
End with CNTL/Z. N7K1-1(config)# vlan 100
N7K1-1(config-vlan)# end
N7K1-1# 2013 Mar
2 17:41:56 N7K1-1 last message repeated 1 time
2013 Mar
2 17:41:56 N7K1-1
%STP-2-BRIDGE_ASSURANCE_BLOCK: Bridge Assurance blocking port port-channel1 VLAN0100.

2013 Mar
2 17:41:56 N7K1-1

2013 Mar
2 17:41:56 N7K1-1
After the VLAN is configured on the other side of a trunk link, BA unblocks it.
N7K2-1# config t
End with CNTL/Z. N7K2-1(config)# vlan 100
N7K2-1(config-vlan)# end
N7K2-1#
N7K1-1# 2013 Mar
2 17:42:24 N7K1-1
%STP-2-BRIDGE_ASSURANCE_UNBLOCK: Bridge Assurance unblocking port port-channel1 VLAN0100.
This automatic pruning behavior can also be verified by checking the VLANs
forwarding on the trunk links through the show interface trunk command, as shown
below.
N5K1# show interface trunk | begin Forwarding
Port
STP Forwarding
-------------------------------------------------------------------------------Eth1/3
none
Eth1/4
none
Eth1/5
none
Eth1/6
none
Eth1/7
none
Eth1/8
none
Eth1/9
none Po2
Po3
1,10,20,30
Po6
1,10,20,30
<snip>
N5K2# show interface trunk | begin Forwarding
Port
STP Forwarding
-------------------------------------------------------------------------------Eth1/3
none
Eth1/4
none
Eth1/5
none
Eth1/6
none
Eth1/7
none
Eth1/8
none
Eth1/9
none Po3
Po5
1,40,50,60
Po6
1,40,50,60
<snip>
N7K1-1# show interface trunk | begin Forwarding
Port
STP Forwarding
-------------------------------------------------------------------------------Eth1/1
none
Eth1/2
none
Eth2/1
none
Eth2/2
none
Eth2/3
none
Eth2/4
none
Eth2/5
none
Eth2/6
none Po1
Po2
1,10,20,30
Po3
40,50,60
10,20,30,40,50,60
<snip>
N7K2-1# show interface trunk | begin Forwarding
Port
STP Forwarding
-------------------------------------------------------------------------------Eth1/1
none
Eth1/2
none
Eth2/1
none
Eth2/2
none
Eth2/3
none
Eth2/4
none
Eth2/5
none
Eth2/6
none Po1
Po4
Po5
none
<snip>
1,10,20,30,40,50,60

Switching
STP Edge Ports
Task
Configure the links between N7K1, N5K1, and N5K2 as follows:
Configure the ports between N7K1 and N5K1 as 802.1q trunks.
Configure the ports between N7K1 and N5K2 as 802.1q trunks.
Configure these links as STP Network Ports.
Disable all other links between the switches.
Configure VLANs between N7K1, N5K1, and N5K2 as follows:
Create VLAN 10 on these three switches.
Configure N5K1's link to Server 1 as an access port in VLAN 10.
Configure N5K2's link to Server 2 as an access port in VLAN 10.
Configure N5K1 and N5K2's links to the servers as STP Edge Ports.
When complete, Server 1 & 2 should have IP reachability to each other, and should
not be subject to the STP forwarding delay if one of their links flap.
Configuration
N5K1:
vlan 10
!
speed 1000
!
N5K2:
vlan 10
!
speed 1000
!
N7K1:
vlan 10
!
Verification
Rapid STP Edge Ports are the logical equivalent of the previously Cisco proprietary
PortFast feature. Syntax-wise, the spanning-tree port type edge is the equivalent of
Catalyst IOS spanning-tree portfast. Like PortFast ports, STP Edge Ports are not
subject to the STP Forwarding Delay, because they are the leaf nodes of the tree.
The end result of using Edge Ports is two-fold. The first is that when an edge port
comes up, it immediately transitions to the forwarding state without having to wait
the default 30-second forwarding delay. Second, if the link goes down, a Topology
Change Notification (TCN) is not generated. TCNs in RSTP means that the CAM
table must immediately be flushed. Frequent TCN generation in RSTP could mean a
lot of inefficient flooding in the network, as MAC addresses would constantly have to
be re-learned if TCNs are causing the CAM table to flush. To prevent this, all links
that go to non-STP devices, such as end servers, storage arrays, etc., should run as
STP edge ports.
Verification that a link is running as edge is shown below.
N5K2# show spanning-tree interface e1/2
Vlan
Role Sts Cost
Prio.Nbr Type
---------------- ---- --- --------- -------- -------------------------------VLAN0010
Desg FWD 4
128.130 Edge P2p
The default behavior is that all ports run as STP port type normal, as seen by the
Type P2p from the show spanning-tree output below.
N5K1# show spanning-tree interface e1/1
Vlan
Role Sts Cost
Prio.Nbr Type
---------------- ---- --- --------- -------- -------------------------------VLAN0010
Desg FWD 4
128.129 P2p
When normal ports are activated, RSTP initiates the sync process to attempt rapid
convergence. Because the device on the other end of the link, a server in this case,
is not running RSTP, no sync response is received. The result is that RSTP must fall
back to legacy STP rules and wait for the forward delay to expire before the link can
transition to forwarding state. From the debug output and the timestamps below, we
see that the link is activated at 16:52:44 and begins in the blocking state, transitions
to learning, and then finally to forwarding at 16:53:14, which is 30 seconds after the
link up event.
N5K1# debug spanning-tree event interface e1/1
N5K1# conf t
N5K1(config-if)# no shut
2013 Mar
2 16:52:39 N5K1 %ETHPORT-5-IF_ADMIN_UP: Interface Ethernet1/1 is admin up .
N5K1(config-if)# 2013 Mar
2 16:52:44 N5K1 %ETHPORT-5-SPEED: Interface Ethernet1/1, operational speed changed to 1 G
2013 Mar
2 16:52:44 N5K1 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/1, operational duplex mode changed to Full
2013 Mar
2 16:52:44 N5K1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/1, operational Receive Flow Control sta
2013 Mar
2 16:52:44 N5K1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/1, operational Transmit Flow Control st
2013 Mar
2 16:52:44 N5K1 %ETHPORT-5-IF_UP: Interface Ethernet1/1 is up in mode access
2013 Mar
2 16:52:44.824150 stp: stp_im_l2_protocol_state_change(341): IM L2 Port Proto UP Notif : port_mode 0x80000
2013 Mar
2 16:52:44.824484 stp: stp_mcs_set_slave_mcec_vlans_multi_port_state(928): Ignore - Not a vPC Master
2013 Mar
2 16:52:44.824514 stp: vb_vlan_shim_set_vlans_multi_port_state(2738): Req (type=9) to PIXM vdc 1, inst 0,
2013 Mar
2 16:52:44 .824700 stp
: Success: pixm_send_set_mult_cbl_vlans_for_multiple_ports, num ports 1 VDC 1, state BLK

, rr_token 0x1f65af msg_size 544
2013 Mar
2013 Mar
2013 Mar
2 16:52:59 .765901 stp
: Success: pixm_send_set_mult_cbl_vlans_for_multiple_ports, num ports 1 VDC 1, state LRN

, rr_token 0x1f6686 msg_size 544
2013 Mar
2013 Mar
2013 Mar
2 16:53:14 .772174 stp
: Success: pixm_send_set_mult_cbl_vlans_for_multiple_ports, num ports 1 VDC 1, state FWD

, rr_token 0x1f6741 msg_size 544
2013 Mar
2 16:53:14.772296 stp: stp_send_pvlan_port_state_change_notif[4224]:idx=0:state=4:if_index=Ethernet1/1:pri
When the link is configured as edge, the transition from down to forwarding happens
immediately, as shown below.
N5K1# debug spanning-tree event interface e1/1
N5K1# conf t
End with CNTL/Z.

N5K1(config-if)# shut N5K1(config-if)# spanning-tree port type edge
Warning: Edge port type (portfast) should only be enabled on ports connected to a single
host. Connecting hubs, concentrators, switches, bridges, etc... to this
interface
when edge port type (portfast) is enabled, can cause temporary bridging loops.
Use with CAUTION
Edge Port Type (Portfast) has been configured on Ethernet1/1 but will only
have effect when the interface is in a non-trunking mode.
N5K1(config-if)# no shut
2013 Mar
2 17:07:44 N5K1 %ETHPORT-5-IF_ADMIN_UP: Interface Ethernet1/1 is admin up .
2013 Mar
2 17:07:49 N5K1 %ETHPORT-5-SPEED: Interface Ethernet1/1, operational speed changed to 1 Gbps
2013 Mar
2 17:07:49 N5K1 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/1, operational duplex mode changed to Full
2013 Mar
2 17:07:49 N5K1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/1, operational Receive Flow Control sta
2013 Mar
2 17:07:49 N5K1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/1, operational Transmit Flow Control st
2013 Mar
2 17:07:49.516552 stp: stp_im_l2_protocol_state_change(341): IM L2 Port Proto UP Notif : port_mode 0x80000
2013 Mar
2013 Mar
2013 Mar
2 17:07:49.517105 stp: Success: pixm_send_set_mult_cbl_vlans_for_multiple_ports, num ports 1 VDC 1, state
2013 Mar
2 17:07:49 N5K1 %ETHPORT-5-IF_UP
: Interface Ethernet1/1 is up in mode access

2013 Mar
2013 Mar
2013 Mar
2 17:07:49.525715 stp
: Success: pixm_send_set_mult_cbl_vlans_for_multiple_ports, num ports 1 VDC 1, state FWD

, rr_token 0x1fb7db msg_size 544
Nexus Technology Labs - Overlay Transport

Virtualization (OTV)
Overlay Transport Virtualization (OTV)
Task
Configure basic layer 2 reachability as follows:
Configure Server 1s link to N5K1 with the IP address 10.0.0.1/24.
Configure N5K1s link to Server 1 in VLAN 10.
Configure VLAN 10 trunking on the links between N7K1 and N5K1, and
between N7K2 and N5K2.
Configure N7K1 and N7K2 as the STP root bridges for VLAN 10.
Disable all other unused ports on all four switches.
Configure Overlay Transport Virtualization (OTV) between N7K1 and N7K2 to tunnel
traffic between Server 1 and Server 2 as follows:
Enable the OTV feature on N7K1 and N7K2.
Create VLAN 999 on N7K1 and N7K2 and configure it as the OTV Site
VLAN.
Configure N7K1 with the OTV Site Identifier 0x101.
Configure the first M1 port connecting N7K1 and N7K2 as a native layer 3
routed interface using the addresses 169.254.0.71/24 and 169.254.0.72/24,
respectively. This will be the OTV Join Interface.
Enable IGMPv3 on the OTV Join Interface.
Configure interface Overlay 1 on both N7K1 and N7K2. Use the routed
interface between them as the OTV Join Interface, 224.1.1.1 as the OTV
Control Group, 232.1.1.0/24 as the OTV Data Group range, and VLAN 10 as
an Extend VLAN.
When complete, N7K1 and N7K2 should form an OTV IS-IS adjacency, and traffic
between Server 1 and Server 2 should be bridged over the OTV tunnel.
Configuration
N5K1:
vlan 10
!
speed 1000
!
N5K2:
vlan 10
!
speed 1000
!
N7K1:
feature otv
!
vlan 10
!
vlan 999
name OTV_SITE_VLAN
!
otv site-vlan 999
otv site-identifier 0x101
!
interface Overlay1
otv join-interface Ethernet1/1
otv control-group 224.1.1.1
otv data-group 232.1.1.0/24
otv extend-vlan 10
no shutdown
!
ip address 169.254.0.71/24
ip igmp version 3
no shutdown
!
N7K2:
feature otv
!
vlan 10
!
vlan 999
name OTV_SITE_VLAN
!
otv site-vlan 999
!
interface Overlay1
otv extend-vlan 10
no shutdown
!
ip address 169.254.0.72/24
ip igmp version 3
no shutdown
!
Verification
Overlay Transport Virtualization (OTV) is an Ethernet over IP tunnel that allows you
to span the broadcast domain of two Data Center sites together over an IP
transport. In essence, OTV accomplishes the same goal as other L2 tunneling
protocols such as L2TPv3, Any Transport over MPLS (AToM), or Virtual Private
LAN Services (VPLS), but it includes special enhancements specific to Data Center
Interconnect (DCI) environments. The first of these enhancements is that OTV is
transport agnostic.
In this example, our OTV Edge Devices N7K1 and N7K2 are directly connected over
a point-to-point layer 3 routed interface. However, OTV can run over any transport
as long as both unicast and multicast IP connectivity exist. This means that OTV
can run over Dark Fibre, MPLS L2VPN/L3VPN, DMVPN, etc., as long as
reachability is there. In later examples, we will see how to lessen this connectivity
requirement so that only unicast IP reachability is required, as opposed to both
unicast and multicast.
The layer 3 routed interface used to form the OTV tunnel is called the Join Interface.
Note that as of the release used in this example, the Join Interface cannot be an SVI
or a Loopback; it must be a native layer 3 routed physical interface, subinterface, or
port channel. This link is essentially what you would think of as the tunnel source
in a normal GRE tunnel configuration. The logical interface where traffic is OTV
encapsulated is called the Overlay Interface, which is analogous to the interface
tunnel in normal GRE tunnel configurations. A concise verification of the state of
both the Join and Overlay interfaces is shown below.
N7K1# show otv
OTV Overlay Information

Site Identifier 0000.0000.0101
Overlay interface Overlay1
VPN name
: Overlay1
VPN state
: UP
Extended vlans
: 10 (Total:1)
Control group
: 224.1.1.1
Data group range(s) : 232.1.1.0/24

Join interface(s)
: Eth1/1 (169.254.0.71)
Site vlan
: 999 (up)
AED-Capable
: Yes
Capability
: Multicast-Reachable
The above output shows us both the Unicast and Multicast transport addresses
used, as well as other pertinent information such as the VLANs extended (bridged)
over the tunnel, whether the local Site VLAN is up (which it must be), and whether
the edge router has elected itself as Authoritative (which it must be). Another
important output from above is the Site Identifier, which must be the same for Edge
Devices in the same DC site, and different between different sites (as is this case).
The specific need for the Site VLAN and Site Identifier will be explored in more
detail in later scenarios.
For the transport addresses, Unicast reachability is simple to verify. If the OTV Edge
Devices can ping the IP address on their Join Interfaces, thats basically all there is
to it. For multicast reachability, we would, of course, require a multicast-capable

transport, such as a point-to-point link like in this case, or MPLS L3VPN that has
MDT service enabled. Note that the multicast transport must be both ASM (*,G) and
SSM (S,G) capable, because the discovery of other OTV edge devices uses shared
trees (*,G), whereas actual multicast data plane over the tunnel uses shortest path
trees (S,G). Because SSM is used for certain flows, IGMPv3 support must be
enabled on the Join Interface as well as the actual PIM router on the other side of
the link (if applicable).
Without doing a verification in the DCI transit network itself, multicast reachability
can be quickly verified from the OTV edge routers themselves simply by checking
whether the OTV IS-IS adjacencies have formed. IS-IS traffic is encapsulated in the
OTV control-group multicast, which is the (*,G) feed. This means that if IS-IS is up,
multicast likewise is up. This verification is shown below.
N7K1# show ip mroute
IP Multicast Routing Table for VRF "default"
(*, 224.1.1.1/32)
, uptime: 00:02:22, otv ip
Incoming interface: Ethernet1/1
, RPF nbr: 169.254.0.71 Outgoing interface list

: (count: 1) Overlay1
, uptime: 00:02:22, otv
N7K1# show otv isis adjacency
OTV-IS-IS process: default VPN: Overlay1
OTV-IS-IS adjacency database:
System ID
SNPA
Level
State
Hold Time
Interface Site-ID
N7K2
0026.980c.2141
UP
00:00:25
Overlay1 0000.0000.0102
Unlike FabricPath, which uses IS-IS simply to establish a shortest path tree between
the FabricPath Spine and Leaf switches, the OTV IS-IS process is used to advertise
reachability information about the end hosts. Before traffic can actually flow over the
tunnel, the end devices MAC addresses must be advertised into IS-IS, and can be
verified as follows.
N7K1# show otv route
OTV Unicast MAC Routing Table For Overlay1
VLAN MAC-Address
Metric
Uptime
Owner
Next-hop(s)
---- --------------
------
--------
---------
-----------
10 000c.2922.167d
10 000c.29bc.5bea
42
00:26:16
00:26:17
overlay
site
N7K2
Ethernet2/3
MAC addresses local to the site should be reachable out a physical link or portchannel, whereas MAC addresses in other sites will be reachable out the overlay
tunnel interface. The detailed advertisements in the IS-IS database are shown here.
N7K1# show otv isis database detail
OTV-IS-IS Process: default LSP database VPN: Overlay1
OTV-IS-IS Level-1 Link State Database

LSPID
Seq Number
.00-00
0x00000006
Checksum
0x2014
Lifetime
751
0/0/0/1
Instance
0x00000003
Area Address
00
NLPID
0xCC 0x8E
Hostname
N7K2
Length : 4
Extended IS
N7K1.01
Metric : 40
MAC Address
Vlan
: 10 : Metric
: 1
Vlan
: 10 : Metric
: 1
: 000c.2922.167d
Digest Offset :
.00-00
A/P/O/T N7K2
0 N7K1
* 0x00000006
0x8E3B
790
0/0/0/1
Instance
0x00000006
Area Address
00
NLPID
0xCC 0x8E
Hostname
N7K1
Length : 4
Extended IS
N7K1.01
Metric : 40
: 000c.29bc.5bea
MAC Address
Digest Offset :
N7K1.01-00
* 0x00000003
0xBC4E
645
Instance
0x00000003
Extended IS
N7K2.00
Metric : 0
Extended IS
N7K1.00
Metric : 0
Digest Offset :
0/0/0/1
The database output above indicates that N7K1 is advertising Server 1s MAC
address 000c.29bc.5bea in VLAN 10, whereas N7K2 is advertising Server 2s MAC
address 000c.2922.167d in VLAN 10.
Another indication that the data plane is working in OTV is to check the ARP and
ICMP ND cache on the OTV Edge Devices. As a flooding optimization, the AEDs
cache ARP and ND responses that they have learned so that further ARP/ND
requests do not need to flood over the overlay tunnel. This can be seen below.
N7K1# show otv arp-nd-cache
OTV ARP/ND L3->L2 Address Mapping Cache
Overlay Interface Overlay1
VLAN
MAC Address
Layer-3 Address
Age
Expires In
10
000c.2922.167d
10.0.0.2
00:00:01
00:07:58
From the above output, we see that N7K1 knows about Server 2 with the IP address
10.0.0.2 across the Overlay1 interface. If Server 1 were to clear its ARP cache and
send a new ARP request for Server 2, we should see N7K1 respond locally instead
of forwarding the request over the Overlay, as shown here.
N7K1# debug otv arp-nd
N7K1#
otv: IPv4 ARP Request packet received from source 10.0.0.1 on interface Ethernet2/3
otv: Found ORIB mac route next hop Ethernet2/3 for target IP 10.0.0.2 with source MAC 0010-000c.29bc.5bea , hence se
otv: Send proxy ARP-Reply to 10.0.0.1 (0010-000c.29bc.5bea) for target IP 10.0.0.2 with target MAC 000c.2922.167d on
An important point to remember about OTV and data plane traffic is that the OTV
tunnel adds 42 bytes of overhead and does not support fragmentation. This means
that if the DCI does not support Jumbo Frames, it is possible that traffic flows that
work over other interconnects such as dark fiber wont work over OTV. From the
end host perspective, this can be seen here.
ICMP PINGs from Server 1 to Server 2 with the Dont Fragment bit set work up to
1430 bytes. With the ICMP header of 8 bytes and IP header of 20 bytes, it means
that the entire IP packet length is 1458. An additional 42 bytes of OTV bring us to a
total length of 1500 bytes, which is the default MTU. The next ping with an ICMP
payload of 1431 is dropped, as the OTV edge device silent discards the packet
instead of fragmenting it. To increase the payload size that the tunnel supports, the
DCI links (the Join Interface and everything else in the transit path) must support
Jumbo Frames. On Nexus 7K this is simply one command at the interface level,
shown here.
N7K1# conf t
End with CNTL/Z.

N7K1(config-if)# mtu 9216
Now the servers can exchange packets up to 1500 bytes, as shown above. Jumbo
Frames above this size are still dropped, but now due to the normal layer 2
infrastructure such as the 5Ks and the Server NIC cards themselves not having
Jumbo Frames enabled, not the OTV tunnel or the DCI itself.

OTV Authentication
Task
traffic between Server 1 & Server 2 as follows:
VLAN.
Enable IGMPv3 on the OTV Join Interface.
Configure interface Overlay 1 on both N7K1 and N7K2. Use the routed
interface between them as the OTV Join Interface, 224.1.1.1 as the OTV
Control Group, 232.1.1.0/24 as the OTV Data Group range, and VLAN 10 as
an Extend VLAN.
Configure OTV Authentication as follows:
Authenticate the OTV IS-IS adjacency between N7K1 and N7K2 using an
MD5 hash of the key-string ADJACENCY.
Authenticate the OTV IS-IS LSP advertisements between N7K1 and N7K2
using an MD5 hash of the key-string UPDATES.
Configuration
N5K1:
vlan 10
!
speed 1000
!
N5K2:
vlan 10
!
speed 1000
!
N7K1:
feature otv
!
vlan 10
!
vlan 999
name OTV_SITE_VLAN
!
otv site-vlan 999
!
key chain OTV_ISIS_ADJACENCY
key 1
key-string 0 ADJACENCY
!
key chain OTV_ISIS_LSP
key 1
key-string 0 UPDATES
!
interface Overlay1
otv isis authentication-type md5
otv isis authentication key-chain OTV_ISIS_ADJACENCY
otv extend-vlan 10
no shutdown
!
otv-isis default
vpn Overlay1
authentication-type md5
authentication key-chain OTV_ISIS_LSP
!
ip address 169.254.0.71/24
ip igmp version 3
no shutdown
!
N7K2:
feature otv
!
vlan 10
!
vlan 999
name OTV_SITE_VLAN
!
otv site-vlan 999
!
key chain OTV_ISIS_ADJACENCY
key 1
key-string 0 ADJACENCY
!
key chain OTV_ISIS_LSP
key 1
key-string 0 UPDATES
!
interface Overlay1
otv isis authentication-type md5
otv isis authentication key-chain OTV_ISIS_ADJACENCY

otv extend-vlan 10
no shutdown
!
otv-isis default
vpn Overlay1
authentication key-chain OTV_ISIS_LSP
!
ip address 169.254.0.72/24
ip igmp version 3
no shutdown
!
Verification
OTV uses IS-IS in the control plane behind the scenes to advertise MAC address
reachability between different DC sites. This means that many of the features native
to IS-IS are also inherently native to OTV, such as authentication. Specifically, IS-IS
supports two different types of authentication, authentication of the IS-IS adjacency,
and authentication of the Link State Packet (LSP) exchange.
The configuration of OTV IS-IS adjacency authentication is very similar to what you
would see in regular IOS or in NX-OS for the regular IPv4/IPv6 IS-IS process. A key
chain is defined globally and then applied at the link level where the adjacency is
occurring. In our case, the OTV IS-IS adjacency forms on the logical Overlay1
interface, as shown below.
N7K1# show otv adjacency
Overlay Adjacency database

Overlay-Interface Overlay1
:
Hostname
System-ID
Dest Addr
N7K2
0026.980c.2141 169.254.0.72
Up Time
State
04:11:11
UP
The authentication configuration at the link level can be verified as follows.

N7K1# show otv isis interface overlay 1
Overlay1, Interface status: protocol-up/link-up/admin-up
IP address: none
IPv6 address: none
IPv6 link-local address: none
Index: 0x0001, Local Circuit ID: 0x01, Circuit Type: L1
Level1
Adjacency server (local/remote) : disabled / none
Adjacency server capability : multicast Authentication type is MD5
Authentication keychain is OTV_ISIS_ADJACENCY
Authentication check specified
LSP interval: 33 ms, MTU: 1400

Level
Metric
CSNP
40
10
Level
1
Adjs
1
Next CSNP
AdjsUp Pri
1
Hello
00:00:08
64
Circuit ID
N7K1.01
Multi
Next IIH
00:00:01
Since
* 04:12:58
In the case of an authentication failure, the IS-IS adjacency would fail to form. If
authentication is enabled, as verified above, and the adjacency is forming, we can
assume that the authentication is successful.
Troubleshooting a mismatch of authentication can be seen below, where N7K1
modifies its key chain so that the password is not the same as the remote OTV edge
device, and the end result is that the IS-IS adjacency is torn down.
N7K1# config t
N7K1(config)# key chain OTV_ISIS_ADJACENCY
N7K1(config-keychain)# key 1
End with CNTL/Z.
N7K1(config-keychain-key)# key-string WRONGPASSWORD

N7K1(config-keychain-key)# end
N7K1#
OTV-IS-IS adjacency database: System ID
Interface Site-ID N7K2
Level State
SNPA
0026.980c.2141
1 UP
Hold Time
00:00:10
Overlay1 0000.0000.0102
N7K1# 2013 Apr 19 21:14:21 N7K1 %ISIS_OTV-5-ADJCHANGE:
isis_otv-default [2770]
LAN adj L1 N7K2 over Overlay1 - DOWN (Hold timer expired)

on MT-0

OTV-IS-IS adjacency database: System ID
Interface Site-ID N7K2
SNPA
0026.980c.2141
Level State
1 LOST
Hold Time
00:05:19
Overlay1 0000.0000.0102
By debugging the OTV IS-IS process, we can see that the authentication
configuration is the reason that the adjacency is not forming.
N7K1# debug otv isis iih
<snip> isis_otv: default [2770] Receive L1 LAN IIH over Overlay1 from N7K2 (0026.980c.2141)
len 1397 prio 7
isis_otv: default [2770] Receive link cap tlv, site cap sub-tlv, len 9 isis_otv: default [2770]
LAN IIH authentication failure on Overlay1 from 0026.980c.2141
<snip>
LSP authentication, like in regular IOS, is configured under the global process,
which in our case is the global OTV IS-IS process. The below output shows us that
LSP authentication is enabled.
N7K1# show otv isis
ISIS process : default

VPN: Overlay1
System ID : 68bd.abd7.6041
SAP : 439
IS-Type : L1
Queue Handle : 11
Maximum LSP MTU: 1392

Graceful Restart enabled. State: Inactive
Last graceful restart status : none
Metric-style : advertise(wide), accept(narrow, wide)
Area address(es) :
00
Process is up and running

VPN ID: 214
Incremental update routes during SPF run
Stale routes during non-graceful controlled restart
Interfaces supported by OTV-IS-IS :
Overlay1
Level 1 Authentication type is MD5
Authentication keychain is OTV_ISIS_LSP
Authentication check is specified

Address family IPv4 unicast :
Number of interface : 1
Adjacency check disabled
Distance : 115
Address family IPv6 unicast :
Distance : 115
Address family MAC unicast :
Distance : 115
L1 Next SPF: Inactive
A problem in LSP authentication is a little harder to troubleshoot, because a

mismatch in authentication does not prevent the adjacency from forming, but it will
prevent traffic in the data plane from actually forwarding. To illustrate this, N7K1
changes its LSP authentication key chain to be incorrect, as shown below. Note that
before changing the authentication, N7K1 has OTV routes installed via the overlay.
VLAN MAC-Address
---- -------------10 000c.2922.167d
10 000c.29bc.5bea
Metric
Uptime
-----42
-------01:26:12
01:34:46
Owner
Next-hop(s)
--------overlay
----------N7K2
site
Ethernet2/3
End with CNTL/Z.
N7K1# config t
N7K1(config)# key chain OTV_ISIS_LSP

N7K1(config-keychain)# key 1
N7K1(config-keychain-key)# key-string 0 WRONGPASSWORD
N7K1(config-keychain-key)# end
N7K1#
Now that the authentication is mismatched, we try to send traffic between the
servers.
The result is a little different than expected, because even though the OTV AEDs
dont match their authentication, traffic is still allowed to pass. In reality, whats
happening is that because the topology has not changed at all, and neither of the
AEDs has had to do IS-IS LSP flooding, they dont yet know that a mismatch as
occurred. Traffic will not begin to drop until a topology event occurs that. To illustrate
this, Server 1 changes the MAC address of its NIC card, which will cause a new
OTV IS-IS advertisement.
After Server 1 generates traffic to Server 2, the routing table of the AEDs is as
follows.

VLAN MAC-Address
Metric
Uptime
Owner Next-hop(s)
---- --------------
------
--------
---------
00:00:13
-----------
10 0000.0000.0001
site Ethernet2/3
10 000c.29bc.5bea
01:53:08
site Ethernet2/3

VLAN MAC-Address
Metric
Uptime
Owner Next-hop(s)
---- --------------
------
--------
---------
01:56:46
site Ethernet2/3
10 000c.2922.167d
-----------
Note that the AEDs only know about MAC addresses local to their site, because the
Overlay1 interface is not listed as a next-hop for any MAC address. Additionally, we
see that on N7K1 the new MAC address of Server 1 was learned. A debug of the
OTV IS-IS process will now tell us what the problem is.
N7K1# debug otv isis lsp flooding
isis_otv: default [2770] Receive Overlay1 L1 LSP
0026.980c.2141.00-00 Seq 0x0000003A Chksum 0x705F Lifetime 1199 over Overlay1 from 0026.980c.2141 prio 7
isis_otv: default [2770] LSP authentication failure
This output indicates that N7K2 is trying to send N7K1 an IS-IS update over the
OTV tunnel, but the authentication is failing. Note, however, that it does not stop the
adjacency from forming.

OTV-IS-IS adjacency database:
System ID
SNPA
Level
State
Hold Time
Interface Site-ID
0026.980c.2141
0026.980c.2141
UP
00:00:30
Overlay1 0000.0000.0102

OTV Adjacency Server
Task
traffic between Server 1 and Server 2 as follows:
VLAN.
Configure interface Overlay 1 on both N7K1 and N7K2, using the routed
interface between them as the OTV Join Interface.
Configure VLAN 10 as an Extend VLAN on Overlay 1.
Instead of configuring multicast control and data groups, configure N7K1 as
the OTV Adjacency Server.
Configuration
N5K1:
vlan 10
!
speed 1000
!
N5K2:
vlan 10
!
speed 1000
!
N7K1:
feature otv
!
vlan 10
!
vlan 999
name OTV_SITE_VLAN
!
otv site-vlan 999
!
interface Overlay1
otv adjacency-server unicast-only
otv extend-vlan 10
no shutdown
!
ip address 169.254.0.71/24
no shutdown
!

N7K2:
feature otv
!
vlan 10
!
vlan 999
name OTV_SITE_VLAN
!
otv site-vlan 999
!
interface Overlay1
otv use-adjacency-server 169.254.0.71 unicast-only
otv extend-vlan 10
no shutdown
!
ip address 169.254.0.72/24
no shutdown
!
Verification
With the default operation, OTV requires that the Data Center Interconnect (DCI)
transit network between the DC sites be multicast capable, both for Any Source
Multicast (ASM) and for Source Specific Multicast (SSM). This can potentially limit
the deployment options for OTV if the DCI transit network doesn't natively support
multicast, such as the Internet or a DMVPN tunnel, for example. To alleviate these
design constraints, newer versions of the OTV implementation introduce the OTV
Adjacency Server feature.
The OTV Adjacency Server is one or more OTV Authoritative Edge Devices (AEDs)
that can be used to discover all other endpoints of the OTV tunnel without the need
for multicast. Without the Adjacency Server, all AEDs join the ASM (*,G) OTV
control group. This group is used to discover the other AEDs in the network,
because all AEDs join the same PIM shared tree. In addition to endpoint discovery,
the control group is used to transmit certain control plane packets such as the IS-IS
routing protocol traffic. To see the Adjacency Server in action, first lets review how
the control and data planes of OTV are built in the default design.
Below we see N7K1 configured with both an OTV control group and data group.
Additionally, the interface facing toward the DCI (the OTV Join Interface) is running
IGMPv3.
N7K1# show run interface overlay 1
!Command: show running-config interface Overlay1

!Time: Thu Apr 25 23:53:10 2013
version 6.0(2)
interface Overlay1
otv join-interface Ethernet1/1 otv control-group 224.1.1.1
otv extend-vlan 10
no shutdown
N7K1# show running-config interface Ethernet1/1
!Command: show running-config interface Ethernet1/1

!Time: Fri Apr 26 00:15:33 2013
version 6.0(2)
ip address 169.254.0.71/24 ip igmp version 3
no shutdown
Next, both unicast and multicast connectivity is verified over the DCI. Server 1
sends unicast ICMP PINGs to Server 2, as well as a traceroute. The PINGs test
basic unicast connectivity, and the traceroute shows that they are on the same layer
2 segment.
To use the iPerf app for sending and receiving multicast traffic, the
Windows routing table must know which link to send the multicast
UDP traffic out and where to send the IGMP Report (Join) messages
out. This is what the static multicast route is accomplishing. This is
only needed here because there are multiple NICs on the Virtual
Machines, and we need to make sure these messages go out the
correct link.
Next, iPerf is used to generate and receive multicast traffic. R1 is the multicast
server and chooses the group address 239.0.0.1. R2 is the multicast receiver, and
joins 239.0.0.1. Note that the UDP packet length must be lowered because the OTV
tunnel does not support fragmentation.
After the multicast tree is built end to end, multicast data plane traffic through the
OTV tunnel is encapsulated in the multicast data group that was configured.
Essentially, this is the multicast UDP packets generated with iPerf from the end
server being encapsulated across the DCI as multicast GRE (the OTV tunnel). From
the below output, we see that N7K1 is a sender for the (S,G) pair (169.254.0.71/32,
232.1.1.0/32), which is this new multicast GRE tunnel. The address 232.1.1.0/32
comes from the fact that the multicast data group range of interface Overlay1 is
232.1.1.0/24, so simply the first address is chosen.
(*, 224.1.1.1/32), uptime: 03:47:33, otv ip

Incoming interface: Ethernet1/1, RPF nbr: 169.254.0.71
Outgoing interface list: (count: 1)
Overlay1, uptime: 03:47:33, otv
(169.254.0.71/32, 232.1.1.0/32)
, uptime: 00:00:55, igmp ip
Ethernet1/1, uptime: 00:00:55, igmp, (RPF)
The show otv mroute output below tells us what specific multicast traffic is being
encapsulated into the new multicast tunnel. In this case we see that in VLAN 10,
there is a sender 10.0.0.1 using the group address 239.0.0.1.
N7K1# show otv mroute
OTV Multicast Routing Table For Overlay1
(10, *, 239.0.0.1), metric: 0, uptime: 00:00:57, overlay(r)

Overlay1, uptime: 00:00:57, isis_otv-default
(10, 10.0.0.1, 239.0.0.1)
, metric: 0, uptime: 00:02:07, site Outgoing interface list
: (count: 1) Overlay1
, uptime: 00:00:57, otv
The OTV AED on the other end of the tunnel, N7K2, sees the new multicast GRE
tunnel coming in, as shown below.
(*, 224.1.1.1/32), uptime: 04:19:02, otv ip

(169.254.0.71/32, 232.1.1.0/32)
, uptime: 00:00:31, otv ip
In summary, we see that the control plane traffic such as the IS-IS routing protocol
hellos use the OTV control group (224.1.1.1/32 in this case), whereas actual data
multicast feeds use a new address in the 232.1.1.0/24 range. Now the AEDs are
changed so that an OTV Adjacency Server is specified, and the multicast related
configuration is removed.

!Time: Fri Apr 26 00:31:57 2013
version 6.0(2)
interface Overlay1
otv extend-vlan 10 otv adjacency-server unicast-only
no shutdown
N7K1# show run interface e1/1
!Time: Fri Apr 26 00:32:02 2013
version 6.0(2)
ip address 169.254.0.71/24
no shutdown

!Time: Fri Apr 26 00:27:14 2013
version 6.0(2)
interface Overlay1
otv extend-vlan 10 otv use-adjacency-server 169.254.0.71 unicast-only
no shutdown
N7K2# show run interface ethernet 1/1

!Time: Fri Apr 26 00:27:15 2013
version 6.0(2)
ip address 169.254.0.72/24
no shutdown
N7K1 is configured as the Adjacency Server, and N7K2 is configured with the
unicast address of the Adjacency Server. This means that N7K1 will learn N7K2s
unicast address when N7K2 registers. Also, if there were more AEDs at the remote
site or if there were multiple remote sites, they would all register with N7K1, then
N7K1 would distribute the endpoint addresses to all other AEDs. Additionally, there
is an option to configure redundant Adjacency Servers, as with the current
configuration shown in this task; a single failure of the Adjacency Server would
effectively cause a loss in connectivity of the entire OTV network.
From the end servers perspective, nothing changes. Both unicast and multicast
reachability are maintained; however, in the core of the DCI, the traffic pattern
changes. The first change of note is that there are no longer any entries in the
multicast routing table over the DCI, as seen below.
The AED does still know about the multicast sender in its local site, as seen with the
show otv mroute below.
N7K1# show otv mroute
OTV Multicast Routing Table For Overlay1
(10, *, 239.0.0.1), metric: 0, uptime: 00:01:29, overlay(r)

Overlay1, N7K2, uptime: 00:01:29, isis_otv-default
(10, 10.0.0.1, 239.0.0.1)
, metric: 0, uptime: 00:06:08, site
Overlay1, N7K2, uptime: 00:01:29, otv
An inline EthAnalyzer capture shows us that control plane traffic is unicast only
between the AEDs, where previously this would have been multicast using the
control group.
On Nexus 7K, EthAnalyzer can only be run by a user with the admin
role in the default VDC.
N7K2# ethanalyzer local interface inband capture-filter "ip proto 47" limit-captured-frames 1 detail | no-more
Capturing on inband
Frame 1 (228 bytes on wire, 196 bytes captured)
Arrival Time: Apr 26, 2013 01:52:50.984764000
[Time delta from previous captured frame: 0.000000000 seconds]
[Time delta from previous displayed frame: 0.000000000 seconds]
[Time since reference or first frame: 0.000000000 seconds]
Frame Number: 1
Frame Length: 228 bytes
Capture Length: 196 bytes
[Frame is marked: False]
[Protocols in frame: eth:ip:gre:mpls:pwethheuristic:pwethcw:eth:mdshdr:fc]
Ethernet II, Src: 68:bd:ab:d7:60:41 (68:bd:ab:d7:60:41), Dst: 00:26:98:0c:21:41 (00:26:98:0c:21:41)
Destination: 00:26:98:0c:21:41 (00:26:98:0c:21:41)
Address: 00:26:98:0c:21:41 (00:26:98:0c:21:41)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
.... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
Source: 68:bd:ab:d7:60:41 (68:bd:ab:d7:60:41)

Address: 68:bd:ab:d7:60:41 (68:bd:ab:d7:60:41)
Type: IP (0x0800)
Internet Protocol, Src: 169.254.0.71 (169.254.0.71), Dst: 169.254.0.72 (169.254.0.72)
Version: 4
Header length: 20 bytes
Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
0000 00.. = Differentiated Services Codepoint: Default (0x00)
.... ..0. = ECN-Capable Transport (ECT): 0
.... ...0 = ECN-CE: 0
Total Length: 182
Identification: 0x0824 (2084)
Flags: 0x02 (Don't Fragment)
0.. = Reserved bit: Not Set
.1. = Don't fragment: Set
..0 = More fragments: Not Set
Fragment offset: 0
Time to live: 254 Protocol: GRE
(0x2f)
Header checksum: 0x1f69 [correct]
[Good: True]
[Bad : False] Source: 169.254.0.71 (169.254.0.71)
Destination: 169.254.0.72 (169.254.0.72)
Generic Routing Encapsulation (0x8848 - unknown)

Flags and version: 0000
0... .... .... .... = No checksum
.0.. .... .... .... = No routing
..0. .... .... .... = No key
...0 .... .... .... = No sequence number
.... 0... .... .... = No strict source route
.... .000 .... .... = Recursion control: 0
.... .... 0000 0... = Flags: 0
.... .... .... .000 = Version: 0
Protocol Type: Unknown (0x8848)
MultiProtocol Label Switching Header, Label: 42, Exp: 0, S: 1, TTL: 254
MPLS Label: 42
MPLS Experimental Bits: 0
MPLS Bottom Of Label Stack: 1
MPLS TTL: 254
PW Ethernet Control Word
Sequence Number: 1
Ethernet II, Src: 5b:ea:86:dd:60:00 (5b:ea:86:dd:60:00), Dst: 00:02:00:0c:29:bc (00:02:00:0c:29:bc)
Destination: 00:02:00:0c:29:bc (00:02:00:0c:29:bc)
Address: 00:02:00:0c:29:bc (00:02:00:0c:29:bc)

Source: 5b:ea:86:dd:60:00 (5b:ea:86:dd:60:00)
Address: 5b:ea:86:dd:60:00 (5b:ea:86:dd:60:00)
.... ...1 .... .... .... .... = IG bit: Group address (multicast/broadcast)
.... ..1. .... .... .... .... = LG bit: Locally administered address (this is NOT the factory default)
Type: Unknown (0x0000)
MDS Header(SOFi2/)
MDS Header
...1 0001 0000 0001 = Packet Len: 4353
.... 0000 0000 00.. = Dst Index: 0x0000
.... ..00 0000 0000 = Src Index: 0x0000
.... 0010 0000 1000 = VSAN: 520
Fibre Channel
[Exchange First In: 0]
[Exchange Last In: 0]
R_CTL: 0xc1(Link_Control Frame/ACK0)
Dest Addr: 16.13.dc
CS_CTL: 0xff
Src Addr: 02.00.00
Type: Basic Link Svc (0x00)
F_CTL: 0x000000 Exchange Originator Seq Initiator CS_CTL
0... .... .... .... .... .... = ExgRpd: Exchange Originator
.0.. .... .... .... .... .... = SeqRec: Seq Initiator
..0. .... .... .... .... .... = ExgFst: NOT exchg first
...0 .... .... .... .... .... = ExgLst: NOT exchg last
.... 0... .... .... .... .... = SeqLst: NOT seq last
.... ..0. .... .... .... .... = Pri: CS_CTL
.... ...0 .... .... .... .... = TSI: NOT transfer seq initiative
.... .... 00.. .... .... .... = LDF: Last Data Frame - No Info (0x000000)
.... .... ..00 .... .... .... = A01: no ack required (0x000000)
.... .... .... ..0. .... .... = RetSeq: NOT retransmitted sequence
.... .... .... .... ..00 .... = AA: ABTS - Cont (0x000000)
.... .... .... .... .... 0... = RelOff: rel offset NOT set
SEQ_ID: 0x00
DF_CTL: 0x00
SEQ_CNT: 0
OX_ID: 0x0001
RX_ID: 0x0002
Parameter: 0x02220223
1 packet captured
Program exited with status 0.
Nexus Technology Labs - Data Center

Technology Lab Resources
Data Center Rack Rental Access Guide
This guide describes how to use the INE CCIE Data Center Rack Rentals that
complement the CCIE Data Center Technology Lab Online Workbook.
Important Notes
To rent Data Center racks, you must have purchased the Data Center Online
Workbook. You can read about, preview, and purchase the workbook here.
You are limited to 3 concurrent scheduled sessions with a maximum time of 9 hours
per session.
Common Problems
Scheduling a Rack Session
Passwords
Scheduling Data Center Rack Add-ons
Loading and Saving Configurations
Canceling a Rack Session
Connecting to NX-OS Devices CLI
Using the Rack Control Panel
Connecting to Windows Virtual Machines
Connecting to UCS Blade ESXi Instances
Common Problems
Telnet Connection Warning
Some Telnet clients will close the Telnet window if the connection cannot be
established. This behavior prevents you from seeing error messages that indicate
the line is in use, so its a good practice to disable this behavior in your Telnet client.
If you do not see a command prompt when you establish a Telnet connection, you
may need to press Enter to wake up the device.
Important Configuration Changes
When you load a saved configuration into the UCS, you must change the interfaces
in use to match the Data Center rack you are using; our automation cannot make
this modification for you. You will find the necessary configuration changes in the
reminder email you receive before your rack session.
Port Speed Information
If you receive an error message stating, SFP validation failed, you must manually
set the port speed to 1G (1000) because Nexus does not support auto-negotiation
from 10G to 1G.
Scheduling a Rack Session

You schedule your Data Center rack session through your Members account.
1. Sign in to the Members site, and then click Rack Rentals on the left side of the page.
2. Find the rack rental type you want, and click Schedule to see the rack session
booking page.
3. On this page, select a preferred range of start and end dates and start and end times
for your rack rental session.
4. Click Search to find and select available sessions. Click the Book button to reserve
your time.
Data Center rack sessions start at designated, set intervals every
three hours. You cannot choose a custom start time for Data Center
rack rentals at this time. The last half hour of your session is for
intersession (the rack is reset for the next customer). For example, if
you reserve a 3-hour session, your session will last 2.5 hours; if you
reserve a 6-hour session, your session will last 5.5 hours. You are not
charged for the intersession portion of your rack rental.
FULLY BOOKED means that the session is not available to rent.
Book means that you may reserve this session. It also displays the number of
tokens required to reserve this session.

Rent Now means that the session is available for immediate rental.
Before booking or reserving a session, select any add-ons you want
by using the check boxes above the calendar tool.
In the Reserved Sessions area, the Active tab displays any sessions that have
already started. These sessions cannot be cancelled. The Upcoming tab displays
your reserved sessions, which can be cancelled at any time before the start time of
the session.
Passwords
Device
IP Address
Username/password (casesensitive)
ESXi1
10.0.115.11
root / cciedc01
ESXi2
10.0.115.12
root / cciedc01
ESXiVMFEX
10.0.115.21
root / cciedc01
Win2k8-BM
10.0.114.21
Administrator / Cciedc01
N7K and
N5K
cisco / cisco
MDS
cisco / cisco (role = cisco)
UCS
192.168.0.100
cisco / rackpassword (role =

admin)
The UCS password is now dynamically changed to match your

current rack session password.
Scheduling Data Center Rack Add-ons

Data Center Rack Rentals are modular.
By default, each 'Base 5K/7K' rental includes access to 2 x Nexus 7000 VDCs, 2 x
Nexus 5000s, and 2 x Windows Server Virtual Machines.
By selecting the UCS/SAN Add-On, you also get access to the UCS Blade Server
chassis, UCS 6200 Fabric Interconnects, MDS 9200 SAN Switches, and the Fibre
Channel SAN.
By selecting the N2K/SAN Add-On, you also get access to the Nexus 2000 Fabric
Extenders with a 10GigE attached server, MDS 9200 SAN Switches, and the Fibre
Channel SAN.
Note that not all labs require each Add-On. You can determine which labs need
which Add-Ons by comparing the Data Center Rack Rental Access Guide CCIE DC
Physical Diagram, located in the Resources section in the upper-right corner of this
page, with the diagram for the individual workbook section that you are working on.
Use the following general guidelines to help determine which add-ons you need:
The Nexus labs require only the Base N5K/N7K rental, unless you are working on
the Fabric Extender or SAN labs, in which case they require the N2K/SAN Add-On.
The UCS labs require the Base N5K/N7K rental plus the UCS Add-On.
Loading and Saving Configurations

When you load a saved configuration into the UCS, you must change the interfaces
in use to match the Data Center rack you are using; our automation cannot make
this modification for you. You will find the necessary configuration changes in the
reminder email you receive before your rack session.
Before saving any configuration, make sure that all of your devices are at the
command prompt or a login prompt.
Canceling a Rack Session

You can cancel a rack session in your Members account.
1. Click Rack Rentals on the left side of the page, and then click the Upcoming
Rentals tab at the top.
2. To cancel a session, click the red Cancel button in the Upcoming Rentals window.
3. You will be asked to confirm your cancellation. Another message states the number
of tokens that have been returned to your account. Click OK. Your session will be
cancelled and your tokens refunded.
Connecting to NX-OS Devices CLI

When your session begins, find your login information on the Rack Rentals page as
shown above. Open your terminal software, such as PuTTY, SecureCRT, etc., and
telnet to dc.ine.com. Log in with the provided username and password.
When you are logged in, a menu displays the devices that you have access to. The
example shown below indicates that this session is assigned the Base 5K/7K rental,
with both the UCS/SAN and N2K/SAN add-ons.
Choose the device that you want to connect to, such as N7K1, and log in with the
username cisco and password cisco. Note that the default ACE
username/password is cisco/ciscocisco.
Certain commands, such as erasing and reloading the device or

editing the management IP address, are purposely restricted to allow
the automation system to function properly.
Using the Rack Control Panel

The Rack Control Panel allows you to reset devices to their default configurations
and access the remote desktops of the virtual machines. To access the control
panel during your active rack session, click Rack Rentals on the Members site, find
your session by clicking the Upcoming Rentals tab, and then click the Control
Panel button.
Save config and load config functionality will be added to the DC
control panel shortly.
Connecting to Virtual Machines

Each rack session is assigned access to certain virtual machines, based on your
rack number assignment and the add-ons that you chose. To access the virtual
machines:
1. Go to the Rack Control Panel on the Members site.
2. Click the Remote Desktop Access tab to see a list of available virtual machines
below. If this page asks you for a username and password, use your rack username
and password, e.g. username dcrack1 password abcdef.
3. Click the icon for the VM that you want to access, and the remote desktop will
automatically open in another window. The remote desktop's resolution automatically
resizes to your browser window size, so if you simply refresh this page the resolution
will update to the browser size.
If for some reason you lock yourself out of a Virtual Machine, you can reset them
from the control panel. All changes you made during your rack session will be
reverted when you use this option.
Connecting to UCS Blade ESXi Instances

When it becomes necessary to connect to the UCS blade ESXi (or WindowsBareMetal) instances, follow the directions in the task to connect to them, using the
VMWare Infrastructure Client provided (preinstalled on any of the Windows VM
desktops). The IP addresses for the UCS blade ESXi and Windows Win2k8R2 bare
metal instances are:
ESXi1: 10.0.115.11
ESXi2: 10.0.115.12
ESXi-VMFEX: 10.0.115.21
Win2k8-BM: 10.0.114.21
The username and password for all ESXi instances are root and cciedc01.
The username and password for the Win2k8R2 instance is Administrator and
cciedc01 or Cciedc01.
Nexus Technology Labs - Release Notes

Nexus Technology Labs: January 3, 2014
Changes made January 3, 2014
This workbook was updated with new rack number associations, and redundant
MDS links were removed.
Nexus Technology Labs - FabricPath

FabricPath
Task
Configure basic layer 2 connectivity between the Nexus 7Ks and 5Ks as follows:
Create VLANs 10 and 20 on N7K1, N7K2, N5K1, & N5K2.
Create SVIs for VLANs 10 and 20 on N5K1 & N5K2. Use IP addresses
10.0.0.5Y/24 and 20.0.0.5Y/24, where Y is the 5K's device number per the
diagram.
Configure trunk links between N7K1 & N5K1, and between N7K2 & N5K2.
All other links on the 5Ks should be disabled.
Configure FabricPath on N7K1 & N7K2 as follows:
Install and enable the FabricPath feature set.
Configure VLANs 10 & 20 as FabricPath VLANs.
Set the FabricPath Switch-ID to by 7Y, where Y is the 7K's device number
per the diagram.
Enable FabricPath on the F1 ports between the 7Ks; the links down to the
5Ks will be running Classical Ethernet.
Ensure that the 7Ks are the root of the Spanning-Tree for both VLANs 10 &
20.
Once complete, N5K1 & N5K2 should have IP reachability between their VLAN 10 &
20 interfaces, and the 7Ks should be routing this traffic over FabricPath between
each other.
Configuration
N5K7:
shutdown
!
!
vlan 10,20
!
interface Vlan10
no shutdown
ip address 10.0.0.57/24
!
interface Vlan20
no shutdown
ip address 20.0.0.57/24
!
no shutdown
!
no shutdown
N5K8:
shutdown
!
!
vlan 10,20
!
interface Vlan10
no shutdown
ip address 10.0.0.58/24
!
interface Vlan20
no shutdown
ip address 20.0.0.58/24
!
no shutdown
!
no shutdown
N7K7:
feature-set fabricpath
!
vlan 10,20
mode fabricpath
!
spanning-tree vlan 10,20 priority 0
!
fabricpath switch-id 77
!
switchport mode fabricpath
no shutdown
!
no shutdown
!
no shutdown
!
no shutdown
N7K8:
!
vlan 10,20
mode fabricpath
!
!
!
no shutdown
!
no shutdown
!
no shutdown
!
no shutdown
Verification
Note that the installation of the FabricPath feature in the
Admin/Default VDC is already done for you in the INE rack rentals.
For other topologies the following steps would also be required before
running FabricPath.
DC2-N7K1# conf t
End with CNTL/Z.
DC2-N7K1(config)# install feature-set fabricpath
feature set is installed already(0x40aa0011)DC2-N7K1(config)# vdc N7K7

DC2-N7K1(config-vdc)# allow feature-set fabricpath
DC2-N7K1(config-vdc)# end
DC2-N7K1#
Once FabricPath is enabled at the interface level in the user VDC, the 7Ks should
form an IS-IS adjacency, as seen below.
N7K7# show fabricpath isis adjacency
Fabricpath IS-IS domain: default Fabricpath IS-IS adjacency database:
System ID
SNPA
Level
State
Hold Time
Interface
N7K8
N/A
UP
00:00:25
Ethernet2/25
N7K8
N/A
UP
00:00:24
Ethernet2/26
The show fabricpath switch-id command shows how many switches (both Leaf and
Spine nodes) are in the FabricPath topology, and what their FabricPath Switch-IDs
are (essentially the IS-IS Router-ID). This would also be where we would see any
switches running vPC+ or Anycast HSRP.
N7K7# show fabricpath switch-id
FABRICPATH SWITCH-ID TABLE
Legend: '*' - this system
'[E]' - local Emulated Switch-id
'[A]' - local Anycast Switch-id
Total Switch-ids: 2
=============================================================================
SWITCH-ID
SYSTEM-ID
FLAGS
STATE
STATIC
EMULATED/
ANYCAST
--------------+----------------+------------+-----------+--------------------
* 77
64a0.e742.8dc5
Primary
Confirmed Yes
No
4055.390b.c145
Primary
Confirmed Yes
No
78
The FabricPath routing table shows how the local Leaf or Spine makes its path
selection to the other Switch-IDs in the Fabric.
N7K7# show fabricpath route
FabricPath Unicast Route Table
'a/b/c' denotes ftag/switch-id/subswitch-id
'[x/y]' denotes [admin distance/metric]
ftag 0 is local ftag
subswitch-id 0 is default subswitch-id
FabricPath Unicast Route Table for Topology-Default
0/77/0, number of next-hops: 0

via ---- , [60/0], 0 day/s 00:02:32, local 1/ 78
/0, number of next-hops: 2 via Eth2/25, [115/40]
, 0 day/s 00:02:24, isis_fabricpath-default via Eth2/26, [115/40]
, 0 day/s 00:02:24, isis_fabricpath-default
The details of the IS-IS database show how the switches are specifically connected
together in the SPF graph, their FabricPath Switch-ID, which Topologies are
enabled, and which switches are the Multi-Destination Root for those trees.
N7K7# show fabricpath isis database detail
Fabricpath IS-IS domain: default LSP database
LSPID
Seq Number
0x00000006
0x5E1F
Instance
0x00000002
Area Address
00
NLPID
0xC0
Hostname
N7K8
Extended IS
Lifetime
A/P/O/T N7K8.00-00
0/0/0/1
Length : 4
N7K7.00
Extended IS
Capability
Checksum
1157
Metric : 40
N7K7.00
Metric : 40
: Device Id: 78 Base Topology
Base Topo Trees :

Trees desired: 2
Nickname
Trees computed: 2
Trees usable: 2
Priority: 0 Nickname: 78 BcastPriority: 64

Version
Version: 1 Flags: 0
Nickname Migration :
Swid: 78 Sec. Swid: 0
0 N7K7
Digest Offset :
.00-00
* 0x00000009
0xB2C5
Instance
0x00000009
Area Address
00
NLPID
0xC0
Hostname
N7K7
Extended IS
N7K8.00
Extended IS
0/0/0/1
Length : 4
Capability
1158
Metric : 40
N7K8.00
Metric : 40
Base Topo Ftag
Graph 1: Root: N7K7 Primary: 1, Secondary: 0 Nickname 77
Base Topo Trees :

Trees desired: 2
Trees computed: 2
Trees usable: 2
Base Topo Roots :

Graph 1: Root Nickname: 77
Nickname

Version
Version: 1 Flags: 0
Digest Offset :
From the devices in the Classical Ethernet domain (the 5Ks in this case), they
simply see the network as a normal Ethernet switched LAN.
VLAN0010
Root ID
Bridge ID
Priority
10
Address
c84c.75fa.6000
Cost
Port
134 (Ethernet1/6)
Hello Time
Priority
32778
Address
547f.ee79.137c
Hello Time
sec
sec
Max Age 20 sec
Max Age 20 sec
Interface
Role Sts Cost
Prio.Nbr Type
---------------- ---- --- --------- -------- -------------------------------- Eth1/6

2
128.134
Eth1/7
Root FWD
P2p
Altn BLK 2
128.135
P2p
N5K7# ping 10.0.0.58

PING 10.0.0.58 (10.0.0.58): 56 data bytes
Request 0 timed out
--- 10.0.0.58 ping statistics --5 packets transmitted, 4 packets received, 20.00% packet loss
round-trip min/avg/max = 3.038/4.441/4.976 ms
Legend:
VLAN
MAC Address
Type
age
Secure NTFY
Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+------------------ *
10
547f.ee7a.4d7c
dynamic
10
Eth1/6
When frames are received by the FabricPath Leaf nodes, a MAC address table
lookup is performed, and the destination FabricPath Switch-ID is found. This field is
used as the Outer Destination Address (ODA) when the Ethernet frame is
FabricPath encapsulated. The source address of the FabricPath frame is the local
Switch-Id, which is then considered the Outer Source Address (OSA).
Note: MAC table entries displayed are getting read from software.
Use the 'hardware-age' keyword to get information related to 'Age'
Legend:
age - seconds since last seen,+ - primary entry using vPC Peer-Link,
(T) - True, (F) - False ,
VLAN
MAC Address
~~~ - use 'hardware-age' keyword to retrieve age info
Type
Secure NTFY Ports/ SWID.SSID.LID
age
---------+-----------------+--------+---------+------+----+-----------------* 10
547f.ee79.137c
dynamic
~~~
dynamic
~~~
Eth2/27
10 547f.ee7a.4d7c
F 78.0.26
In this example there are no pure Spine switches in the FabricPath topology, but in
a real deployment the Spine switches will not populate the MAC address table for
destinations reachable via FabricPath. Instead they simply use the FabricPath
routing table to forward towards the ODA address (destination Switch-ID).
Note that the FabricPath Leaf Switches must be the root of the STP
for the Classical Ethernet domain, otherwise their CE facing ports will
be disabled, as seen below. This is similar to the STP RootGuard
feature.
N7K7# config
End with CNTL/Z.
N7K7(config)# spanning-tree vlan 10 priority 61440

N7K7(config)#
2015 Feb 13 20:32:14 N7K7 %STP-2-L2GW_BACKBONE_BLOCK:
L2 Gateway Backbone port inconsistency blocking port Ethernet2/27 on VLAN0010
.
2015 Feb 13 20:32:14 N7K7 %STP-2-L2GW_BACKBONE_BLOCK: L2 Gateway Backbone port inconsistency blocking port Ethernet2
N7K7(config)# show spanning-tree vlan 10
VLAN0010
Root ID
Priority
61450
Address
c84c.75fa.6000

Hello Time
sec
Max Age 20 sec
Bridge ID
Interface
Priority
61450
Address
c84c.75fa.6000
Hello Time
sec
Role Sts Cost
Max Age 20 sec
Prio.Nbr Type
---------------- ---- --- --------- -------- -------------------------------Eth2/27
Desg BKN*2
128.283
P2p *L2GW_Inc
Eth2/28
Desg BKN*2
128.284
P2p *L2GW_Inc

FabricPath with Port-Channels
Task
diagram.
per the diagram.
Aggregate the F1 ports between the 7Ks as interface Port-Channel1.
Enable FabricPath on Port-Channel1 of the 7Ks; the links down to the 5Ks
will be running Classical Ethernet.
20.
each other.
Configuration
N5K7:
shutdown
!
!
vlan 10,20
!
interface Vlan10
no shutdown
ip address 10.0.0.57/24
!
interface Vlan20
no shutdown
ip address 20.0.0.57/24
!
no shutdown
!
no shutdown
N5K8:
shutdown
!
!
vlan 10,20
!
interface Vlan10
no shutdown
ip address 10.0.0.58/24
!
interface Vlan20
no shutdown
ip address 20.0.0.58/24
!
no shutdown
!
no shutdown
N7K7:
!
vlan 10,20
mode fabricpath
!

!
!
switchport
!
channel-group 1
no shutdown
!
channel-group 1
no shutdown
!
no shutdown
!
no shutdown
N7K8:
!
vlan 10,20
mode fabricpath
!
!
!
switchport
!
channel-group 1
no shutdown
!

channel-group 1
no shutdown
!
no shutdown
!
no shutdown
Verification
As seen in previous examples, the underly switchport mode (access, trunk, layer 3,
etc.) is independent of the Port-Channel configuration. Per the output below we can
see that Port-Channel1 is running in FabricPath mode.
N7K7# show int port-channel1 status
-------------------------------------------------------------------------------Port
Name
Status
Vlan
Duplex
Speed
Type
-------------------------------------------------------------------------------Po1
connected f-path
-full
a-10G
--
The difference between this example and the previous one is that the switches will
only form one FabricPath IS-IS adjacency over the Port-Channel, instead of one
adjacency for each underlying member port. From a control-plane point of view, the
port-channel design is more efficient, because if one of the member interfaces is
lost, IS-IS does not need to reconverge. Also the IS-IS state machine does not need
to maintain multiple adjacencies between the same switches.
System ID
SNPA
Level
State
Hold Time
Interface
N7K8
N/A
UP
00:00:26
port-channel1
From a traffic forwarding point of view, packets are still load balanced on the links
between the 7Ks, but now it uses the port-channel hashing mechanism instead of
the IS-IS load balancing method.

via ---- , [60/0], 0 day/s 00:23:10, local 1/ 78
/0, number of next-hops: 1 via Po1, [115/20]
Note that the cost of the route between the 7Ks is now 20, where in the previous
example it was two links each with a cost of 40. This is because FabricPath IS-IS
uses a linear reference bandwidth scale, that if the bandwidth of the links is doubled
(2 x 10GigE), the cost is halved. This also means from a path selection point of
view, FabricPath IS-IS would prefer a 2 x 10GigE channel vs. a single 10GigE link.
LSPID
Seq Number
.00-00
0x0000000F
0xA796
Instance
0x00000008
Area Address
00
NLPID
0xC0
Hostname
N7K8
Extended IS
Capability
Checksum
Lifetime
1032
A/P/O/T N7K8
0/0/0/1
Length : 4
N7K7.00
Metric : 20
Base Topo Trees :

Trees desired: 2
Nickname
Trees computed: 2
Trees usable: 2

Version
Version: 1 Flags: 0
Digest Offset :
N7K7
.00-00
* 0x00000012
0x9601
Instance
0x00000012
Area Address
00
NLPID
0xC0
Hostname
N7K7
Extended IS
0/0/0/1
Length : 4
Capability
1026
N7K8.00
Metric : 20
Base Topo Ftag

Base Topo Trees :
Trees desired: 2
Trees computed: 2
Trees usable: 2
Base Topo Roots :

Nickname

Version
Version: 1 Flags: 0
Digest Offset :
From the MAC address table perspective, the destination MAC addresses are still
associated with the destination Switch-ID, so the Port-Channel configuration does
not impact this operation.
Note: MAC table entries displayed are getting read from software.
Use the 'hardware-age' keyword to get information related to 'Age'
Legend:
age - seconds since last seen,+ - primary entry using vPC Peer-Link,
(T) - True, (F) - False ,
VLAN
MAC Address
~~~ - use 'hardware-age' keyword to retrieve age info
Type
age
Secure NTFY Ports/ SWID.SSID.LID
---------+-----------------+--------+---------+------+----+-----------------* 10
547f.ee79.137c
dynamic
~~~
Eth2/27
10
547f.ee7a.4d7c
dynamic
~~~
F 78.0.26

FabricPath Hello Authentication
Task
diagram.
per the diagram.
Enable FabricPath on the F1 ports bewteen the 7Ks; the links down to the
Configure the 7Ks to authenticate each others IS-IS hellos with an MD5
hash of the string CCIEDC.
20.
each other.
Configuration
N5K7:
shutdown
!
!
vlan 10,20
!
interface Vlan10
no shutdown
ip address 10.0.0.57/24
!
interface Vlan20
no shutdown
ip address 20.0.0.57/24
!
no shutdown
!
no shutdown
N5K8:
shutdown
!
!
vlan 10,20
!
interface Vlan10
no shutdown
ip address 10.0.0.58/24
!
interface Vlan20
no shutdown
ip address 20.0.0.58/24
!
no shutdown
!
no shutdown
N7K7:
!
vlan 10,20
mode fabricpath
!
!
!
key chain FPATH_HELLO_AUTH
key 1
key-string CCIEDC
!
fabricpath isis authentication-type md5
fabricpath isis authentication key-chain FPATH_HELLO_AUTH
no shutdown
!
no shutdown
!
no shutdown
!
no shutdown
N7K8:
!
vlan 10,20
mode fabricpath
!
!
!
key chain FPATH_HELLO_AUTH
key 1
key-string CCIEDC
!

no shutdown
!
no shutdown
!
no shutdown
!
no shutdown
Verification
FabricPath IS-IS supports two types of authentication, hello authentication
configured at the interface level, and Link State Packet (LSP) authentication
configured under the global IS-IS process.
In the below output, one side of the link is configured with authentication, while the
other is not. After the IS-IS hold timer expires, the adjacency is torn down.
N7K8# sh run int e2/25 - 26
!Command: show running-config interface Ethernet2/25-26

!Time: Fri Feb 13 21:59:27 2015
version 6.2(10)
no shutdown
no shutdown

System ID
SNPA
Level
N7K7
N/A
1 UP
00:00:12
Ethernet2/25 N7K7
00:00:10
Ethernet2/26
State
Hold Time
Interface
N/A
1 UP
N7K8# 2015 Feb 13 21:59:42 N7K8 %ISIS_FABRICPATH-5-ADJCHANGE:
isis_fabricpath-default [906]
P2P adj L1 N7K7 over Ethernet2/26 - DOWN

(Hold timer expired) on MT-0
2015 Feb 13 21:59:43 N7K8 %ISIS_FABRICPATH-5-ADJCHANGE:
isis_fabricpath-default [906]
P2P adj L1 N7K7 over Ethernet2/25 - DOWN

(Hold timer expired) on MT-0
System ID
SNPA
Level
N7K7
N/A
1 LOST
00:05:31
Ethernet2/25 N7K7
00:05:13
Ethernet2/26
State
Hold Time
N/A
Interface
1 LOST
Debugging the IS-IS hello exchange between peers can be used to troubleshoot
authentication issues, as seen below.
N7K8# debug fabricpath isis iih
2015 Feb 13 22:02:22.852373 isis_fabricpath: default [906] Receive P2P IIH over Ethernet2/25 from N7K7 len 1500 prio
2015 Feb 13 22:02:22.852412 isis_fabricpath: default [906]
P2P IIH authentication failure over Ethernet2/25
2015 Feb 13 22:02:22.852428 isis_fabricpath: default [906] P2P IIH auth failed!
2015 Feb 13 22:02:23.048618 isis_fabricpath: default [906] IIH timer expired for interface Ethernet2/25
2015 Feb 13 22:02:23.048699 isis_fabricpath: default [906] isis_add_auth_tlv: keychain : , type 36, handle0x0x896d44
Interface level FabricPath authentication can be verified as follows:

N7K7# sh fabricpath isis interface e2/25
Fabricpath IS-IS domain: default
Interface: Ethernet2/25
Status: protocol-up/link-up/admin-up
Index: 0x0002, Local Circuit ID: 0x01, Circuit Type: L1 Authentication type MD5
Authentication keychain is FPATH_HELLO_AUTH
Authentication check specified
Extended Local Circuit ID: 0x1A098000, P2P Circuit ID: 0000.0000.0000.00

Retx interval: 5, Retx throttle interval: 66 ms
LSP interval: 33 ms, MTU: 1500
P2P Adjs: 1, AdjsUp: 1, Priority 64

Hello Interval: 10, Multi: 3, Next IIH: 00:00:05
Level
Adjs
AdjsUp
Metric
CSNP
40
60
Next CSNP
Last LSP ID
Inactive
ffff.ffff.ffff.ff-ff
Topologies enabled:
Level Topology Metric
MetricConfig Forwarding
40
no
UP
40
no
UP

FabricPath LSP Authentication
Task
diagram.
per the diagram.
Enable FabricPath on the F1 ports bewteen the 7Ks; the links down to the
Configure the 7Ks to authenticate each others IS-IS LSPs with an MD5 hash
of the string CCIEDC.
20.
each other.
Configuration
N5K7:
shutdown
!
!
vlan 10,20
!
interface Vlan10
no shutdown
ip address 10.0.0.57/24
!
interface Vlan20
no shutdown
ip address 20.0.0.57/24
!
no shutdown
!
no shutdown
N5K8:
shutdown
!
!
vlan 10,20
!
interface Vlan10
no shutdown
ip address 10.0.0.58/24
!
interface Vlan20
no shutdown
ip address 20.0.0.58/24
!
no shutdown
!
no shutdown
N7K7:
!
vlan 10,20
mode fabricpath
!
!
!
key chain FPATH_LSP_AUTH
key 1
key-string CCIEDC
!
fabricpath domain default
authentication key-chain FPATH_LSP_AUTH
!
no shutdown
!
no shutdown
!
no shutdown
!
no shutdown
N7K8:
!
vlan 10,20
mode fabricpath
!
!
!
key chain FPATH_LSP_AUTH
key 1
key-string CCIEDC
!
authentication key-chain FPATH_LSP_AUTH
!
no shutdown
!
no shutdown
!
no shutdown
!
no shutdown
Verification
FabricPath IS-IS supports two types of authentication, hello authentication
configured at the interface level, and Link State Packet (LSP) authentication
configured under the global IS-IS process.
An LSP authentication mismatch can be difficult to troubleshoot, as it appears as
though the switches form an IS-IS adjacency, as seen below.
System ID
SNPA
Level
4055.390b.c145
N/A
1 UP
State
00:00:32
Ethernet2/25 4055.390b.c145
00:00:26
Ethernet2/26
Hold Time
N/A
Interface
1 UP
However, when LSP authentication fails, the switches will not install routes learned
from each other into the routing table. Without routes installed, the end result is a
failure in the data plane of the actual user traffic sent over the fabric.


via ---- , [60/0], 0 day/s 00:03:30, local
LSP authentication failure can be debugged as follows:

N7K7# debug fabricpath isis lsp flooding
2015 Feb 13 20:58:14.289898 isis_fabricpath: default [6331] Receive default L1 LSP
4055.390b.c145.00-00 Seq 0x00000007 Chksum 0x5C20 Lifetime 1139 over Ethernet2/26 prio 6
2015 Feb 13 20:58:14.289937 isis_fabricpath: default [6331] LSP authentication failure
2015 Feb 13 20:58:14.609373 isis_fabricpath: default [6331] Receive default L1 LSP 4055.390b.c145.00-00 Seq 0x000000
2015 Feb 13 20:58:14.609407 isis_fabricpath: default [6331] LSP authentication failure
2015 Feb 13 20:58:17.439426 isis_fabricpath: default [6331] Receive default L1 GM-LSP 4055.390b.c145.00-00 Seq 0x000
2015 Feb 13 20:58:17.439462 isis_fabricpath: default [6331] GM-LSP authentication failure
2015 Feb 13 20:58:18.576989 isis_fabricpath: default [6331] Retx L1 GM-LSP N7K7.00-00 Seq 0x00000002 Chksum 0xA670 L
After the authentication problem is corrected, normal adjacencies form and the
routing table is populated.

System ID
SNPA
Level
00:00:30
Ethernet2/25 N7K7
00:00:28
Ethernet2/26
State
Hold Time
N/A
Interface N7K7
1 UP


via ---- , [60/0], 0 day/s 00:03:10, local 1/ 77
/0, number of next-hops: 2 via Eth2/25
, [115/40], 0 day/s 00:00:15, isis_fabricpath-default via Eth2/26
, [115/40], 0 day/s 00:00:15, isis_fabricpath-default
N/A
1 UP

FabricPath Root Tree Election
Task
diagram.
per the diagram.
Configure N7K1 to be the root of the first multi-destination tree for the
default FabricPath topology.
Configure N7K2 to be the root of the second multi-destination tree for the
default FabricPath topology.
20.
each other.
Configuration
N5K7:
shutdown
!
!
vlan 10,20
!
interface Vlan10
no shutdown
ip address 10.0.0.57/24
!
interface Vlan20
no shutdown
ip address 20.0.0.57/24
!
no shutdown
!
no shutdown
N5K8:
shutdown
!
!
vlan 10,20
!
interface Vlan10
no shutdown
ip address 10.0.0.58/24
!
interface Vlan20
no shutdown
ip address 20.0.0.58/24
!
no shutdown
!
no shutdown
N7K7:
!
vlan 10,20
mode fabricpath
!
!
!
root-priority 255
!
no shutdown
!
no shutdown
!
no shutdown
!
no shutdown
N7K8:
!
vlan 10,20
mode fabricpath
!
!
!
root-priority 254
!
no shutdown
!
no shutdown
!
no shutdown
!
no shutdown
Verification
The FabricPath Multi-Destination tree is used for forwarding broadcast, unknown
unicast, and multicast traffic over the FabricPath network. Each FabricPath topology
has two trees, each denoted with a unique ftag. The first of these trees is used to
forward broadcast and unknown unicast, while the second tree is for forwarding
multicast. In a real deployment, the roots of the trees should be placed on the
FabricPath Spine nodes.
To influence the tree election, the root-priority can be set under the global
FabricPath process. The device with the highest priority is elected root for tree 1,
and the next-highest priority device is elected root for tree 2. The result of the
election can be verified as follows:
N7K7# show fabricpath isis topology summary
FabricPath IS-IS Topology Summary
Fabricpath IS-IS domain: default MT-0
Configured interfaces:
Max number of trees: 2
Ethernet2/25
Ethernet2/26
Number of trees supported: 2
Tree id: 1, ftag: 1, root system: 64a0.e742.8dc5, 77

Tree id: 2, ftag: 2, root system: 4055.390b.c145, 78
Ftag Proxy Root: 64a0.e742.8dc5
The field MT-0 means Multi-Topology 0, which is the default topology that all
FabricPath links belong to. The root of the trees are denoted by their FabricPath
Switch-IDs, which in this case have been statically assigned. The below output
shows the specific IS-IS cost values to reach the roots of the trees. Note that a
metric of 0 to the root means that device is the root for that tree.
Per the below output we can see that Switch-ID 78 has a cost of 40 to reach the root
of tree 1, while 78 is the root for tree 2 (it has a cost of 0 to itself).
N7K7# show fabricpath isis trees
Fabricpath IS-IS domain: default

Note: The metric mentioned for multidestination tree is from the root of that tree to that switch-id
*:directly connected neighbor or link
P:Physical switch-id, E:Emulated, A:Anycast
MT-0 Topology 0, Tree 1

, Swid routing table
78, L1
via Ethernet2/26, metric 40
Topology 0, Tree 2
, Swid routing table 78
, L1 via Ethernet2/26, metric 0

FabricPath Path Selection
Task
diagram.
per the diagram.
Modify the FabricPath path selection so that 7K1 prefers to route to 7K2
over their first link, while 7K2 prefers to route to 7K1 over their second link.
20.
each other.
Configuration
N5K7:
shutdown
!
!
vlan 10,20
!
interface Vlan10
no shutdown
ip address 10.0.0.57/24
!
interface Vlan20
no shutdown
ip address 20.0.0.57/24
!
no shutdown
!
no shutdown
N5K8:
shutdown
!
!
vlan 10,20
!
interface Vlan10
no shutdown
ip address 10.0.0.58/24
!
interface Vlan20
no shutdown
ip address 20.0.0.58/24
!
no shutdown
!
no shutdown
N7K7:
!
vlan 10,20
mode fabricpath
!
!
!
fabricpath isis metric 20
no shutdown
!
no shutdown
!
no shutdown
!
no shutdown
N7K8:
!
vlan 10,20
mode fabricpath
!
!
!
no shutdown
!
fabricpath isis metric 20
no shutdown
!
no shutdown
!

no shutdown
Verification
Modifying FabricPath path selection works the same way as in OSPF or IS-IS for
regular IP routing, where the cost value is a function of the link (i.e. the Link State)
as opposed to the route itself (as in EIGRP or BGP). Therefore in order to influence
path selection, we can simply raise or lower the IS-IS metrics on a per-link basis.
Since we are currently using only one default topology, modifying the
cost of a link affects the path selection of all VLANs being routed over
FabricPath.
Prior to modifying metrics, the 7Ks install ECMP routes to each others Switch-IDs.

via ---- , [60/0], 0 day/s 00:02:37, local 1/ 78
, 0 day/s 00:01:29, isis_fabricpath-default via Eth2/26, [115/40]
After modification of the metrics, only the lowest cost path is installed in the routing
table.
N7K7# config t
End with CNTL/Z.N7K7(config)# int e2/25
N7K7(config-if)# fabricpath isis metric 20

N7K7(config-if)# end
N7K7# 2015 Feb 13 21:04:37 N7K7 last message repeated 1 time


via ---- , [60/0], 0 day/s 00:03:09, local 1/ 78
The result of this change can be seen in the FabricPath IS-IS database.
LSPID
Seq Number
.00-00
0x00000010
0x6324
Instance
0x0000000A
Area Address
00
NLPID
0xC0
Hostname
N7K8
Extended IS
Checksum
1138
N7K7.00
Capability
A/P/O/T N7K8
0/0/0/1
Length : 4
Extended IS
Lifetime
Metric : 40
N7K7.00
Metric : 20
Base Topo Trees :

Trees desired: 2
Nickname
Trees computed: 2
Trees usable: 2

Version
Version: 1 Flags: 0
Digest Offset :
.00-00
0 N7K7
* 0x00000010
0x9AEA
Instance
0x00000010
Area Address
00
NLPID
0xC0
Hostname
N7K7
Extended IS
:
:
0/0/0/1
Length : 4
N7K8.00
Extended IS
Capability
1139
Metric : 20
N7K8.00
Metric : 40
Base Topo Ftag

Base Topo Trees :
Trees desired: 2
Trees computed: 2
Trees usable: 2
Base Topo Roots :

Nickname

Version
Version: 1 Flags: 0
Digest Offset :

FabricPath Multi Topology Routing
Task
Create VLANs 10, 20, and 30 on N7K1, N7K2, N5K1, & N5K2.
Create SVIs for VLANs 10, 20, and 30 on N5K1 & N5K2. Use IP addresses
10.0.0.5Y/24, 20.0.0.5Y/24, and 30.0.0.5Y/24, where Y is the 5K's device
number per the diagram.
Configure VLANs 10, 20, & 30 as FabricPath VLANs.
per the diagram.
Ensure that the 7Ks are the root of the Spanning-Tree for VLANs 10, 20, &
30.
Configure FabricPath Multi-Topology Routing as follows:
Create two new topology on N7K1 and N7K2.
VLAN 10 should belong to the first new topology.
VLAN 20 should belong to the second new topology.
All other VLANs should belong to the default topology.
The first new topology should only route over the first link between the 7Ks.
The second new topology should only route over the second link between
the 7Ks.
Once complete, N5K1 & N5K2 should have IP reachability between their VLAN 10,
20, & 30 interfaces; the 7Ks should be installing only one route for VLAN 10, only
one route for VLAN 20, but two ECMP routes for VLAN 30 over FabricPath.
Configuration
N5K7:
shutdown
!
!
vlan 10,20,30
!
interface Vlan10
no shutdown
ip address 10.0.0.57/24
!
interface Vlan20
no shutdown
ip address 20.0.0.57/24
!
interface Vlan30
no shutdown
ip address 30.0.0.57/24
!
no shutdown
!
no shutdown
N5K8:
shutdown
!
!
vlan 10,20,30
!
interface Vlan10
no shutdown
ip address 10.0.0.58/24
!
interface Vlan20
no shutdown
ip address 20.0.0.58/24
!
interface Vlan30
no shutdown
ip address 30.0.0.58/24
!
no shutdown
!
no shutdown
N7K7:
!
vlan 10,20,30
mode fabricpath
!
fabricpath topology 1
member vlan 10
!
member vlan 20
!
spanning-tree vlan 10,20,30 priority 0
!
!
fabricpath topology-member 1
!
!
no shutdown
!
no shutdown
N7K8:
!
vlan 10,20,30
mode fabricpath
!
member vlan 10
!
member vlan 20
!
spanning-tree vlan 10,20,30 priority 0
!
!
!
!
no shutdown
!
no shutdown
Verification
This feature comes from the legacy Multi-Topology Routing (MTR) feature of link
state protocols such as OSPF and IS-IS, which was originally designed to allow for
class-based forwarding without the need for MPLS Traffic Engineering. In the case
of FabricPath, Multiple Topologies allow you to control path selection on a per-VLAN
basis or per a group of VLANs, analogous to how Multiple Spanning-Tree path
selection modification works.
A FabricPath Topology is made up of links and VLANs assigned to the topology.
SPF is run separately for each topology, which means that path selection can be
modified from one topology independent of another. The end result of this
configuration is that there are multiple routing tables, one for each topology, as seen
below:
N7K7# show fabricpath route topology all

via ---- , [60/0], 0 day/s 00:14:12, local 1/ 78
, [115/40], 0 day/s 00:10:33, isis_fabricpath-default via Eth2/26
FabricPath Unicast Route Table for Topology 1

via ---- , [60/0], 0 day/s 00:14:12, local 3/ 78
FabricPath Unicast Route Table for Topology 2

via ---- , [60/0], 0 day/s 00:14:12, local 5/ 78
Topologies, their associated links, and SPF metrics can be verified as follows.
All links running FabricPath always belong to the Default topology,
MT-0.
N7K7# show fabricpath isis topology

FabricPath IS-IS Topology
Fabricpath IS-IS Graph 0 Level-1 for MT-0 IS routing table
N7K8.00, Instance 0x00000020
, metric 40
, metric 40
MT-1
*via N7K8, Ethernet2/25

N7K8.00, Instance 0x00000009
, metric 40
MT-2
N7K8.00, Instance 0x00000009
, metric 40
VLAN to topology mappings can be verified as follows.

N7K8# show fabricpath topology vlan
Topo-Description
Topo-ID
Configured VLAN List
-------------------------------- --------- ------------------------------------0
1-9, 11-19, 21-4095
10
20
For each new topology in FabricPath, a new Root Tree election is performed. Note
that each tree gets a unique ftag value, which is used to classify packets in the data
plane as part of the FabricPath header encapsulation.
N7K7# show fabricpath isis topology summary
FabricPath IS-IS Topology Summary
Ethernet2/25
Ethernet2/26

Ftag Proxy Root: 64a0.e742.8dc5 MT-1
Ethernet2/25

Ftag Proxy Root: 64a0.e742.8dc5 MT-2
Ethernet2/26

Ftag Proxy Root: 64a0.e742.8dc5

Nexus Technology Labs

Загружено:

Сведения о документе

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

Nexus Technology Labs

Загружено:

Авторское право:

Доступные форматы

Nexus Technology Labs - Fabric Extenders

input rate 38.46 Mbps, 75.11 Kpps;

output rate 1.82 Gbps

input rate 1.07 Kbps, 2 pps; output rate 200 bps

End with CNTL/Z. N5K1(config)# int e1/10

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_CFG_CHANGE: Interface Ethernet1/10 is down(Config change)

2 19:19:46 N5K1 %NOHMS-2-NOHMS_ENV_FEX_OFFLINE: FEX-101 Off-line (Serial Number SSI16330GT8)

2 19:19:46 N5K1 %PFMA-2-FEX_STATUS: Fex 101 is offline

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_MODULE_REMOVED: Interface Ethernet101/1/1 is down (module removed)

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_MODULE_REMOVED: Interface Ethernet101/1/2 is down (module removed)

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/3 is down (Interface removed

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/4 is down (Interface removed

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/5 is down (Interface removed

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/6 is down (Interface removed

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/7 is down (Interface removed

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/8 is down (Interface removed

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/9 is down (Interface removed

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/10 is down (Interface remove

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/11 is down (Interface remove

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/12 is down (Interface remove

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/13 is down (Interface remove

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/14 is down (Interface remove

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/15 is down (Interface remove

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/16 is down (Interface remove

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/17 is down (Interface remove

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/18 is down (Interface remove

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/19 is down (Interface remove

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/20 is down (Interface remove

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/21 is down (Interface remove

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/22 is down (Interface remove

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/23 is down (Interface remove

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/24 is down (Interface remove

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/25 is down (Interface remove

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/26 is down (Interface remove

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/27 is down (Interface remove

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/28 is down (Interface remove

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/29 is down (Interface remove

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/30 is down (Interface remove

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/31 is down (Interface remove

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/32 is down (Interface remove

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/1 is down (Interface removed

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/2 is down (Interface removed

N5K1(config-if)# 2013 Mar

2 19:19:47 N5K1 %FEX-5-FEX_PORT_STATUS_NOTI: Uplink-ID 1 of Fex 101 that is connected wit

2 19:19:47 N5K1 %ETHPORT-5-IF_DOWN_ADMIN_DOWN: Interface Ethernet1/10 is down (Administratively down)

input rate 36.07 Mbps, 70.45 Kpps;

Nexus Technology Labs - Fabric Extenders

if it was dual attached to a single switch.

N5K1# show vpc

: peer adjacency formed ok

vPC keep-alive status

Configuration consistency status: success

Type-2 consistency status

Number of vPCs configured

Dual-active excluded VLANs

Graceful Consistency Check

vPC Peer-link status

Status Active vlans

------ -------------------------------------------------- 1 Po1

Status Consistency Reason

------ ----------- ------ ----------- -------------------------- ----------- 10 Po10

N5K2# show interface fex-fabric

input rate 19.75 Mbps, 38.58 Kpps;

output rate 934.56 Mbps