SPAR

Valentin Enescu

© 2013 IBM Corporation

 System Networking

SPAR

What it is: Switch Partition (SPAR) is a BLADE proprietary technology developed to facilitate the
creation of multiple partitions within an embedded switch to form a virtual switching context with
respect to the Data Plane partition of a physical switch.

Why: The requirements for data center administrators today include increasing performance, reducing cost
and complexity, and simplifying management. SPAR addresses the need by providing a simple Ethernet
interface connectivity option for connecting the Blade-Server chassis to the network infrastructure.
Administrative effort and networking skills required to connect the Blade-Center to the network is minimized
and simplified with the SPAR functionality.

Ideally, with the factory config that resides on Eagle it should be as simple as plug-and-play.

2 IBM Confidential © 2013 IBM Corporation

 System Networking

General considerations
SPAR itself is implemented as a dedicated VLAN or a set of VLANs which are generally
defined by a set of internal server ports and a single external uplink port or LAG
(static or LACP).

Since there is only one uplink there is no risk to create loops. STP is turned OFF on all
SPAR ports.

Port memberships are mutually exclusive between SPARs and with regular vlans.

SPAR ports can’t be members in regular VLANs, VLANs used by SPARs can’t be used
In other context

SPARs use the global VLAN space of the switch.

SPARs can’t communicate between them or with other VLANs on local switch, the traffic
must be bridged/routed by uplink switch.

3 IBM Confidential © 2013 IBM Corporation

 System Networking

Local Domain SPAR

Local domain processing inside SPAR reflects regular 802.1Q bridge characteristics where
VLAN classification and assignment occurs on the customer VLAN (C-VLAN).

Duplicate customer VLAN (C-VLAN) IDs across SPAR partitions can be defined, where
isolation between different customer traffic within similar sets of VLANs across SPARs is
achieved through egress mask programming on the Broadcom switching ASIC.

L2 switching is based on the MAC & C-VLAN within the SPAR local domain.

4 IBM Confidential © 2013 IBM Corporation

 System Networking

Local Domain SPAR

5 IBM Confidential © 2013 IBM Corporation

 System Networking

Passthrough SPAR

Pass-through domain processing inside SPAR reflects tunneling characteristics where

VLAN classification and assignment occurs on the tunnel service VLAN (S-VLAN) to isolate
different customer traffic within similar sets of VLANs. Passthrough is not customer VLAN
aware.

 Each switching entity is identified by its unique domain VLAN ID which enables the
ability to tunnel the entire 4K VLAN range that exists outside of the switch.

L2 switching is based on the MAC & S-VLAN within the SPAR pass-through domain.

In this case, different VLAN traffic will mix together in a single broadcast domain.

Servers in the SPAR will receive other VLAN traffic and drop it if it does not belong to the
VLAN.

6 IBM Confidential © 2013 IBM Corporation

 System Networking

Passthrough SPAR

7 IBM Confidential © 2013 IBM Corporation

 System Networking

Configuration example
This example includes configuration of SPAR 1 in 7. Create SPAR 2
passthrough mode and SPAR 2 in local domain mode:
>>Configuration# spar 2
1. Create SPAR 1
8. Add uplink port to SPAR 2
>>Main# /cfg/spar 1
>>SPAR 2 Configuration# uplink port Ext 2
2. Add uplink port to SPAR 1
9. Configure domain mode
>>SPAR 1 Configuration# uplink
>>SPAR 1 Uplink Configuration# port Ext 1 >>SPAR 2 Configuration# domain
>>SPAR 1 Uplink Configuration# .. >>SPAR 2 Domain Configuration# mode local

3. Configure domain mode 10. Configure SPAR VLAN

>>SPAR 1 Configuration# domain >>SPAR 2 Domain Configuration# dvlan

>>SPAR 1 Domain Configuration# mode passthrough >>SPAR 2 Default VLAN Domain Configuration# sparvid 4082

4. Configure SPAR VLAN 11. Add member ports

>>SPAR 1 Domain Configuration# dvlan >>SPAR 1 Default VLAN Domain Configuration# addsport INTA11-INTA14
>>SPAR 1 Default VLAN Domain Configuration# sparvid 4081 >>SPAR 1 Default VLAN Domain Configuration# ..

5. Add member ports 12. Configure local domain 1

>>SPAR 1 Default VLAN Domain Configuration# addsport INTA5-INTA10
>>SPAR 1 Default VLAN Domain Configuration# ..
>>>SPAR 1 Domain Configuration# ..
6. Enable SPAR 1
>>SPAR 2 Domain Configuration# lvlan 1
>>SPAR 2 Local VLAN Domain 1 Configuration# vid10
>>SPAR 2 Local VLAN Domain 1 Configuration# addsport INTA11-INTA14
>>SPAR 2 Local VLAN Domain 1 Configuration# ena
>>SPAR 2 Local VLAN Domain 1 Configuration# ..

>>SPAR 1 Configuration# ena

>>SPAR 1 Configuration# ..
8 IBM Confidential © 2013 IBM Corporation
 System Networking
Configuration Example

13. Configure local domain 2

>>SPAR 2 Domain Configuration# lvlan 2

>>SPAR 2 Local VLAN Domain 2 Configuration# vid20
>>SPAR 2 Local VLAN Domain 2 Configuration# addsport INTA11-INTA14
>>SPAR 2 Local VLAN Domain 2 Configuration# ena
>>SPAR 2 Local VLAN Domain 2 Configuration# ..

14. Configure local domain 3

>>SPAR 2 Domain Configuration# lvlan 3

>>SPAR 2 Local VLAN Domain 3 Configuration# vid30
>>SPAR 2 Local VLAN Domain 3 Configuration# addsport INTA11-INTA14
>>SPAR 2 Local VLAN Domain 3 Configuration# ena
>>SPAR 2 Local VLAN Domain 3 Configuration# ..
>>SPAR 2 Domain Configuration# ..

15. Enable SPAR 2

>>SPAR 2 Configuration# ena

9 IBM Confidential © 2013 IBM Corporation

 System Networking

Unsupported Features with SPAR

•Hot Links
•IGMP
•L3 configuration on SPAR (i.e. /c/l3/if <#>/vlan != SPAR VLAN)
•Management VLAN
•Private VLAN
•Protocol VLAN
•QBG
•sFlow
•SLP
•Stacking
•STP, RSTP, MRSTP, PVST
•UFP
•vLAG
•VMAP (VLAN mapped ACLs)
•VMReady
•vNIC

10 IBM Confidential © 2013 IBM Corporation

 System Networking

Supported features and restrictions

Feature Supported features on SPAR (restrictions and interactions may apply)

FCoE Support available only within the SPAR Local Domain mode.
  •No FCoE support on the SPAR Pass-through Domain mode.

•No FCoE support on SPAR using the FCoE auto VLAN option.
 
Static/LACP trunk Single uplink trunk (static/LACP) definition on SPAR:
  •Support static/LACP trunk from SPAR to single switch uplink.

•Support FCoE.
 
Local Domain SPAR Local Domain:
  •Allow duplicate customer VLAN IDs across partitions and maintain isolation between them.
  •Local L2 switching based on MAC & C-VLAN within the SPAR local domain.

•Support FCoE.
 
Pass-through
Domain SPAR Pass-through Domain:
  •TAGIPVID approach to provide tunneling functionality within SPAR.
  •VLAN classification and assignment occurs on the tunnel service VLAN (S-VLAN).
  •Local L2 switching based on MAC & S-VLAN within the SPAR pass-through domain.

  •Unsupported with FCoE.

11 IBM Confidential © 2013 IBM Corporation

 System Networking
Supported features and restrictions

Feature Supported features on SPAR (restrictions and interactions may apply).

L2 Failover L2 Failover Feature Support for SPAR:
•Depending on uplink monitor port status, the L2 failover feature will set the internal physical port link
  state up/down.
  •SPAR support available with L2 Failover MMON options only.
•No SPAR support available with L2 Failover AMON options which depends on the VLAN monitor
  option.

Port Mirroring Port Mirroring Feature Support for SPAR:

  •Packet can be mirrored to any port inside the same SPAR or to ports not part of any SPAR.

  •Packet cannot be mirrored to ports belonging to different SPARs.

ACL Access Control List (ACL) Feature Support for SPAR:
•ACL mirroring will have the same behavior as physical port mirroring, where a packet can be mirrored
to any port inside the same SPAR or to ports not part of any SPAR. Packet cannot be mirrored to
  ports belonging to different SPARs.
•ACL with general filtering criteria can be installed on ports belonging to multiple SPARs or outside of
  any SPAR.
  •ACL configured with the "egrport" option will be limited on ports within the same SPAR.

•ACL configured with the vmap option is unsupported on SPAR.

 
VLAN VLAN creation is limited to the range of <2 to 4094> on SPAR.
SPAR VLANs (including its own default domain VLAN) are created using the same 4K VLAN space on
the global switch. As a result, VLANs that are used by a SPAR cannot be used by the global switch or
  other applications.
12 IBM Confidential © 2013 IBM Corporation
 System Networking

Limitation of Local Domain SPAR

MAC learning occurs on the global switch and is limited to learning a unique MAC/VLAN
combination on a per port basis, and NOT on a per SPAR basis.

This limitation is a result of sharing the same global FDB table. Detection of the same
MAC/VLAN combination on different ports (more specifically across different SPAR ports) will
appear as a station move.

This limitation affects the SPAR Local Domain topology by restricting the deployment within
distinct physical networks.

13 IBM Confidential © 2013 IBM Corporation

 System Networking
Limitation of Local Domain SPAR
Limitations with sharing the same physical network across SPARs

Consider the following scenario:

1) Client ARPs for Server 1

2) ARP broadcast on VLAN10 ingress both EXT1 and EXT4.

3) Assume ARP BC packet arrives on EXT1 ahead of EXT4.

a)When the ARP BC packet arrives initially on EXT1,
CMAC/VLAN 10 is learned on port EXT1.
b)When the ARP BC packet arrives on EXT4, the same
CMAC/VLAN10 is detected and treated as a station
move. The FDB table is then updated to reflect this
change and CMAC/VLAN10 is now learned on port
EXT4.

4) The ARP BC is prevented from leaking across the Local

Domain SPARs using the egress block mask, so no loop is
created.

5) Both Servers 1 and 4 receive the ARP BC packet from the

Client within their respective SPAR Local Domains.

6) Since the ARP request target is for Server 1, Server 1 responds

by sending an ARP reply to the Client.
7) L2 switching will do an FDB lookup and attempt to forward the
packet to port EXT4, however, the SPAR egress block mask
prevents the packet from leaking across the SPARs.
8) The Client fails to receive an ARP response from Server 1 in
this scenario © 2013 IBM Corporation
9) SPAR Local Domain should be deployed on distinct physical
 System Networking

Eagle customizations
To overcome the aforementioned limitation, on Eagle, host-mode has been introduced. When it is enabled:

L2 learning is disabled on uplink ports

ACL is installed on server ports to flood DLF to uplink. Each Local Domain SPAR has it’s own ACL.

Host mode affects just the Local Domain SPARs. Passthrough SPARs and the rest of the ports on the switch
remain unaffected.

These 2 settings can be disabled only when host mode is disabled.

The problem with host mode,is that when FDB ages out, traffic between servers will become unkown and will
be forwarded to uplink, if uplink switch doesn’t support link bridge, the traffic will never return, communications
between peers will become lost.

A solution? Static MACs defined on internal ports.

15 IBM Confidential © 2013 IBM Corporation

 System Networking
Eagle customizations

ARP Monitor Uplink

In an environment where the VMs MACs are statically bound, if a GARP is received on uplink for that MAC,
then the static entry is removed.

This works only when host-mode is enabled.

16 IBM Confidential © 2013 IBM Corporation

 System Networking
Factory config on Eagle
version "7.8.0.31"
!
switch-type "IBM Flex System Fabric SI4093 System Interconnect Module(Upgrade2)"
interface port EXT2
iscli-new
lacp mode active
!
lacp key 1000
!
!
interface port EXT3
!
lacp mode active
spar 1
lacp key 1000
uplink adminkey 1000
!
domain default member INTA1-INTA14
interface port EXT4
enable
lacp mode active
exit
lacp key 1000
!
!
spar 2
interface port EXT5
uplink adminkey 1001
lacp mode active
domain default member INTB1-INTB14
lacp key 1000
enable
!
exit
interface port EXT6
!
lacp mode active
spar 3
lacp key 1000
uplink adminkey 1002
!
domain default member INTC1-INTC14
interface port EXT7
enable
lacp mode active
exit
lacp key 1000
!
!
portchannel 65 lacp key 1000 suspend-individual
interface port EXT8
portchannel 66 lacp key 1001 suspend-individual
lacp mode active
portchannel 67 lacp key 1002 suspend-individual
lacp key 1000
!
!
!
interface port EXT9
interface port EXT1
lacp mode active
lacp mode active
lacp key 1000
lacp key 1000
!
interface port EXT10
lacp mode active
lacp key 1000
17 IBM Confidential ! © 2013 IBM Corporation
 System Networking
Factory config on Eagle
interface port EXT11
lacp mode active !
lacp key 1002 interface port EXT20
! lacp mode active
interface port EXT12 lacp key 1001
lacp mode active !
lacp key 1002 interface port EXT21
! lacp mode active
interface port EXT13 lacp key 1001
lacp mode active !
lacp key 1002 interface port EXT22
! lacp mode active
interface port EXT14 lacp key 1001
lacp mode active !
lacp key 1002 failover enable
! failover trigger 1 mmon monitor admin-key 1000
interface port EXT15 failover trigger 1 mmon control member INTA1-INTA14
lacp mode active failover trigger 1 enable
lacp key 1001 !
! failover trigger 2 mmon monitor admin-key 1001
interface port EXT16 failover trigger 2 mmon control member INTB1-INTB14
lacp mode active failover trigger 2 enable
lacp key 1001 !
! failover trigger 3 mmon monitor admin-key 1002
interface port EXT17 failover trigger 3 mmon control member INTC1-INTC14
lacp mode active failover trigger 3 enable
lacp key 1001 !
! !
interface port EXT18 !
lacp mode active !
lacp key 1001 !
! !
interface port EXT19 cee enable
lacp mode active
lacp key 1001

18 IBM Confidential © 2013 IBM Corporation

 System Networking

Miscellaneous

SPAR is supported on Eagle/Compass/CompassFC/Pollux

There can be configured a maximum of 8 SPARs. A local domain SPAR can have up to
256 local VLAN IDs on Eagle while on the rest of the platforms it can have up to 32.

19 IBM Confidential © 2013 IBM Corporation