A Quick Look at Cisco FabricPath

Cisco FabricPath is a proprietary protocol that uses ISIS to populate a routing table that is used for layer 2
forwarding.
Whether we like or not, there is often a need for layer 2 in the Datacenter for the following reasons:

Some applications or protocols require to be layer 2 adjacent

It allows for virtual machine/workload mobility

Systems administrators are more familiar with switching than routing

A traditional network with layer 2 and Spanning Tree (STP) has a lot of limitations that makes it less than
optimal for a Datacenter:

Local problems have a network-wide impact

The tree topology provides limited bandwidth

The tree topology also introduces suboptimal paths

MAC address tables dont scale

In the traditional network, because STP is running, a tree topology is built. This works better for for flows that
are North to South, meaning that traffic passes from the Access layer, up to Distribution, to the Core and then
down to Distribution and to the Access layer again. This puts a lot of strain on Core interconnects and is not
well suited for East-West traffic which is the name for server to server traffic.
A traditional Datacenter design will look something like this:

If we want end-to-end L2, we could build a network like this:

What would be the implications of building such a network though?

Large failure domain

Unknown unicast and broadcast flooding through large parts of the network

A large number of STP instances needed unless using MST

Topology change will have a large impact on the network and may cause flooding
2

Large MAC address tables

Difficult to troubleshoot

A very brittle network

So lets agree that we dont want to build a network like this. What other options do we have if we still need
layer 2? One of the options is Cisco FabricPath.
FabricPath provides the following benefits:

Reduction/elimination of STP

Better stability and convergence characteristics

Simplified configuration

Leverage parallell paths

Deterministic throughput and latency using typical designs

VLAN anywhere flexibility

The FabricPath control plane consists of the following elements:

Routing table Uses ISIS to learn Switch IDS (SIDs) and build a routing table

Multidestination trees Elects roots and builds multidestination trees

Mroute table IGMP snooping learns group membership at the edge, Group Member LSPs (GM-LSPs)
are flooded by ISIS into the fabric

Observe that LSPs has nothing to do with MPLS in this case and that this is not MAC based routing, routing is
based on SIDs.
FabricPath ISIS learns the shortest path to each SID based on link metrics/path cost. Up to 16 equal (ECMP)
routes can be installed. Choosing a path is based on a hashing function using Src IP/Dst IP/L4/VLAN which
should be good for avoiding polarization.
FabricPath supports multidestination trees with the following capabilities:

Multidestination traffic is contained to a tree topology, a network-wide identifier (Ftag) is assigned to

each tree

A root switch is elected for each multidestination tree

Multipathing is supported through multiple trees

3

Note that root here has nothing to do with STP, think of it in terms of multicast routing.

Multidestination trees do not dictate forwarding for unicast, only for multidestination packets.
The FabricPath data plane behaves according to the following forwarding rules:

MAC table Hardware performs MAC lookup at CE/FabricPath edge only

Switch table Hardware performs destination SID lookups to forward unicast frames to other switches

Multidestination table A hashing function selects the tree, multidestination table identifies on which
interfaces to flood based on selected tree

The Ftag used in FabricPath identifies which ISIS topology to use for unicast packets and for multidestination
packets, which tree to use.
If a FabricPath switch belongs to a topology, all VLANs of that topology should be configured on that switch to
avoid blackholing issues.
FabricPath supports 802.1p but can also match/set DSCP and match on other L2/L3/L4 information.
With FabricPath, edge switches only need to learn:
4

Locally connected host MACs

MACs with which those hosts are bidirectionally communicating

This reduces the MAC address table capacity requirements on Edge switches.
FabricPath Designs
There are different designs that can be used together with FabricPath. The first one is routing at the Aggregation
layer.

The first design is the most classic one where STP has been replaced by FP in the Access layer and routing is
used above the Aggregation layer.
This design has the following characteristics:

Evolution of current design practices

The Aggregation layer functions as FabricPath spine and L2/L3 boundary

FabricPath switching for East West intra VLAN traffic
5

 SVIs for East West inter VLAN traffic

Routed uplinks for North South routed flows

Access layer provides pure L2 functions

FabricPath core ports facing Aggregation layer
CE edge ports facing hosts
Optionally vPC+ can be used for active/active host connections

This design is the simplest option and is an extension of regular Access/Aggregation designs. It provides the
following benefits:

Simplified configuration

Removal of STP

Traffic distribution over all uplinks without the use of vPC

Active/active gateways

VLAN anywhere at the Access layer

Topological flexibility
Direct-path forwarding option
Easily provision additional AccessAggregation bandwidth
Easily deploy L4-L7 services
Can use vPC+ towards legacy Access switches

There is also the centralized routing design which looks like the following:

Centralized routing has the following characteristics:

Traditional Aggregation layer becomes pure FabricPath spine

Provides uniform any-to-any connectivity between leaf switches
In simplest case, only FabricPath switching occurs in spine
Optionally, some CE edge ports exist to provide external router connections

FabricPath leaf switches, connecting to spine, have specific personality

Most of the leaf switches will provide server connectivity, like traditional access switches in Routing
at Aggregation designs
Two or more leaf switches provide L2/L3 boundary, inter-VLAN routing and North-South routing
Other or same leaf switches may provide L4-L7 services

Decouples L2/L3 boundary and L4-L7 services provisioning from Spine

Simplifies Spine design

The different traffic flows in this design looks like the following:

Another design is the multi-pod design which can look like the following:

The multi-pod design has the following characteristics:

Allows for more elegant DC-wide versus pod-local VLAN definition/isolation

No need for pod-local VLANs to exist in core
Can support VLAN id reuse in multiple pods

Define FabricPath VLANs -> map VLANs to topology -> map topology to FabricPath core port(s)
8

Default topology always includes all FabricPath core ports

Map DC-wide VLANs to default topology

Pod-local core ports also mapped to pod-local topology

Map pod-local VLANs to pod-local topology

This post briefly describes Cisco FabricPath which is a technology for building scalable L2 topologies, allowing
for more bisectional bandwidth to support East-West flows which are common in Datacenters. To