#CLUS

How to setup an ACI

Multi-Site with single
Pod and Multi-Pod
Max Ardica – Principal Engineer - DCNBU
Ramses Smeyers – Principal Consulting Engineer - CX

BRKACI-2291

#CLUS
Agenda
• Brief Multi-Pod/Multi-Site Review and Positioning
• Prerequisites
• Hardware Inspection and Installation
• Installing the First Site
• Expanding the Single Pod into a Multi-Pod Fabric
• Simplified Tenant Management through MSO
• Adding the DR Site on MSO
• MSO Additional Functionalities

#CLUS BRKACI-2291 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Cisco Webex Teams
Questions?
Use Cisco Webex Teams to chat
with the speaker after the session

How
1 Find this session in the Cisco Live Mobile App
2 Click “Join the Discussion”
3 Install Webex Teams or go directly to the team space
4 Enter messages/questions in the team space

Webex Teams will be moderated cs.co/ciscolivebot#BRKACI-2291

by the speaker until June 16, 2019.

#CLUS © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
Brief Multi-Pod /
Multi-Site Review and
Positioning

BRKACI-2291

5
Multi-Pod or Multi-Site?

That is the question…

#CLUS BRKACI-2291 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
And the answer is…

BOTH!

#CLUS BRKACI-2291 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
 ACI Multi-Pod For More Information on
ACI Multi-Pod:
Overview BRKACI-2003
VXLAN
Inter-Pod
Pod ‘A’ Network
Pod ‘n’

MP-BGP - EVPN

…
Up to 50 msec RTT

APIC Cluster
IS-IS, COOP, MP-BGP IS-IS, COOP, MP-BGP

Availability Zone

• Multiple ACI Pods connected by an IP Inter-Pod L3
network, each Pod consists of leaf and spine nodes
• Up to 50 msec RTT supported between Pods
• Managed by a single APIC Cluster
• Single Management and Policy Domain
#CLUS
• Forwarding control plane (IS-IS, COOP) fault
isolation
• Data Plane VXLAN encapsulation between Pods
• End-to-end policy enforcement
BRKACI-2291 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
 ACI Multi-Pod
Most Common Use Cases
 Need to scale up a single ACI fabric above
Pod
200 leaf nodes supported in a single Pod Inter-Pod
 Handling 3-tiers physical cabling layout Leaf Nodes Network
(for example traditional N7K/N5K/N2K
deployments)
Spine Nodes

 True Active/Active DC deployments

Pod 1 Pod 2
Single VMM domain across DCs (stretched ESXi
Metro Cluster, vSphere HA/FT, DRS initiated
workload mobility,…)
Deployment of Active/Standby or Active/Acive
clustered network services (FWs, SLBs) across DCs APIC Cluster
DB Web/App Web/App
Application clustering (L2 BUM extension across
Pods)

#CLUS BRKACI-2291 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
 ACI Multi-Site VXLAN
Overview Inter-Site
Network

MP-BGP - EVPN
Multi-Site
Orchestrator

Site 1 Site 2
REST
GUI
API Availability Zone ‘B’
Availability Zone ‘A’
Region 1

• Separate ACI Fabrics with independent APIC clusters • MP-BGP EVPN control plane between sites
• No latency limitation between Fabrics • Data Plane VXLAN encapsulation across
• ACI Multi-Site Orchestrator pushes cross-fabric sites
configuration to multiple APIC clusters providing • End-to-end policy definition and
scoping of all configuration changes enforcement
#CLUS BRKACI-2291 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
 ACI Multi-Site
Most Common Use Cases

• Scale-up model to build a • Data Centre Interconnect (DCI) • ACI Multi-Cloud

very large intra-DC network Integration between on-prem and
Extend connectivity and policy
(above 400 leaf nodes) public clouds
between ‘loosely coupled’ DC sites
Disaster Recovery and IP mobility use
cases

#CLUS BRKACI-2291 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
Prerequisites

BRKACI-2291

12
 For More Information on starting

Prerequisites an ACI fabric from scratch:

BRKACI-2004

• Before starting, you should have:

• For each APIC a routable IP addresses for OOB mgmt and CIMC
• Functional NTP server
• Serial number of all leaf and spine nodes
• Optionally but recommended:
• 1 IP per leaf and spine for OOB
• SCP / FTP / HTTP server (software)
• Console / serial server
• Infrastructure VLAN / VTEP pool
• vCenter IP address and credentials

#CLUS BRKACI-2291 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
Hardware
Inspection and
Installation
#CLUS BRKACI-2291 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
Our Setup for Today (High Level View)
Single external network used
for IPN and ISN

IPN/ISN

Site1-Pod 1 Site1-Pod 2 Site2-Pod 1

WAN

Site 1 Site 2

#CLUS BRKACI-2291 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
Our Setup for Today (Detailed View)
Bru ACI Fabric POD 37 & 38
bdsol-aci37-multipod1 bdsol-aci37-multipod3

bdsol-aci38-router2

multipod2 multipod4

bdsol-aci38-multisite1

BDSOL-ACI37-SPINE1 BDSOL-ACI37-SPINE2 BDSOL-ACI37-SPINE3 BDSOL-ACI37-SPINE4

BDSOL-ACI38-SPINE1 BDSOL-ACI38-SPINE2

BDSOL-ACI37-LEAF3 BDSOL-ACI37-LEAF4 bdsol-aci38-server2

BDSOL-ACI37-LEAF1 BDSOL-ACI37-LEAF2

BDSOL-ACI38-LEAF2
BDSOL-ACI38-LEAF1

BDSOL-ACI37-APIC1 BDSOL-ACI38-APIC1

BDSOL-ACI37-APIC2 bdsol-aci37-server2 BDSOL-ACI38-APIC2

bdsol-aci37-server1 bdsol-aci38-server1
bdsol-aci38-router1
bdsol-aci37-router2
bdsol-aci37-router1

BDSOL-ACI37-APIC3 BDSOL-ACI38-APIC3
bdsol-aci38-router3

#CLUS BRKACI-2291 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
Installing the First Site
Site1-Pod1 Configuration
Installing the First Site
Site1-Pod1: Initial Fabric Setup (Already Done)

• APIC initial configuration (S1P1-APIC1) [only the 1st one for now]
• 1st leaf discovery
• Spines discovery
• 2nd leaf discovery
• S1P1-APIC2 configuration
• Verification
• OOB mgmt  IPs for leaf and spine nodes

#CLUS BRKACI-2291 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
Installing the First Site
Site1-Pod1 Fabric

Site1-Pod 1
S1P1-Spine201 S1P1-Spine202
vCenter
Server 1

S1P1-Leaf101 S1P1-Leaf102

ESXi Cluster

192.168.1.1 S1P1-APIC1 S1P1-APIC2

.101
WAN
192.168.200.100/30
.102

#CLUS BRKACI-2291 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
Installing the First Site
Parameters for the APIC Initial Setup Script

S1P1-APIC1 S1P1-APIC2

Fabric name Fabric1 Fabric1

Fabric ID 1 1

Active controllers 3 3

Pod ID 1 1

Controller ID 1 2

TEP Pool 10.0.0.0/16 10.0.0.0/16

Infra VLAN 3937 3937

#CLUS BRKACI-2291 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
Installing the First Site
Site1-Pod1: Remaining Configuration Steps to Do

• NTP configuration
• Route Reflector for intra-BGP VPNv4 sessions
• VMM integration
• Tenant configuration with ‘Ecommerce’ running application
• ‘Ecommerce’ app connectivity verification
• L3Out creation and external connectivity verification

#CLUS BRKACI-2291 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
Demo 1
NTP, VMM, L3Out Configuration and Pod Verification
Expanding the
Single Pod into a
Multi-Pod Fabric
 Expanding the Single Pod into a Multi-Pod Fabric
Adding the IPN and Site1-Pod2

• Step 1: setup the Inter-Pod Network (IPN)

• Step 2: create the Multi-Pod fabric using the APIC Wizard
 Add Site1-Pod1
 Add Site1-Pod2
 Discovery of Pod2’s leaf and spines nodes
• Step 3: S1P2-APIC3 in Pod2 joins the APIC cluster
• Step 4: extend ‘Ecommerce’ Tenant to Pod2 (L3Out, ESXi host, access policies)
• Verification Steps:
• Verify that the existing tenant configuration is extended into the Multi-Pod fabric
• Verify East-West and North-South connectivity

#CLUS BRKACI-2291 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
Expanding the Single Pod into a Multi-Pod Fabric
Adding the IPN and Site1-Pod2
BDSOL-ACI37-Multipod1 BDSOL-ACI37-Multipod3

IPN
BDSOL-ACI37-Multipod2 BDSOL-ACI37-Multipod4

Site1-Pod 1
S1P1-Spine202 S1P2-Spine401 S1P2-Spine402
Site1-Pod2
S1P1-Spine201

vCenter
Server 1

S1P1-Leaf101 S1P1-Leaf102 S1P2-Leaf301 S1P2-Leaf302

BDSOL-ACI37-APIC2
BDSOL-ACI37-APIC3
BDSOL-ACI37-APIC1

#CLUS BRKACI-2291 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
 Expanding the Single Pod into a Multi-Pod Fabric
Step 1: Setup the Inter-Pod Network (IPN)
Pod1 External TEP-Pool Pod2 External TEP-Pool
172.16.2.0/24
S1P1-Spine201
172.16.1.0/24
OSPF Area 0 S1P2-Spine401

IPN3, .18
.2 IPN1
.1
Primary RP Backup RP
PIM .17
.10 .26
.101 .102

1/33 1/33
.21
.5
1/35 1/48 1/48 1/36
TEP Pool: PIM PIM
TEP Pool:
10.0.0.0/16 .9 1/35 1/36 .25 10.1.0.0/16
1/34 1/34
S1P1-Spine202 .6 .22 S1P2-Spine402
.13 .109 .110 .29

1/48 PIM 1/48

.14 .30
IPN2 IPN4

Site1-Pod1 Site1-Pod2
IPN Infra Address Space: 172.16.101.0/24

#CLUS BRKACI-2291 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
Demo 2
Setup the Inter-Pod Network
Expanding the Single Pod into a Multi-Pod Fabric
Step 2: Create the Multi-Pod Fabric Using the APIC Wizard and import Pod2
Spine and Leaf Nodes

Nodes automatically discovered in Site1-Pod2 that

need to be added to the APIC fabric membership table

Node ID Pod ID Name S/N

301 2 S1P2-Leaf301 FDO224702ET

302 2 S1P2-Leaf302 FDO223007J4

401 2 S1P2-Spine401 FDO22472FCV

402 2 S1P2-Spine402 FDO22391NP2

#CLUS BRKACI-2291 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
Demo 3
Create the Multi-Pod Fabric Using the APIC Wizard
 Expanding the Single Pod into a Multi-Pod Fabric
Step 3: S1P2-APIC3 in Pod2 Joins the APIC Cluster

Pod2 uses TEP Pool

S1P1-APIC1 S1P1-APIC2 S1P2-APIC3 10.1.0.0/16 but
S1P2-APIC3 resides
Fabric name Fabric1 Fabric1 Fabric1
in TEP Pool of Pod1
Fabric ID 1 1 1

Active controllers 3 3 3

Pod ID 1 1 2

Controller ID 1 2 3

TEP Pool 10.0.0.0/16 10.0.0.0/16 10.0.0.0/16

Infra VLAN 3937 3937 3937

#CLUS BRKACI-2291 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
Demo 4
S1P2-APIC3 in Pod2 Joins the APIC Cluster
 Expanding the Single Pod into a Multi-Pod Fabric
Step 4: Extend ‘Ecommerce’ Tenant to Pod2 (L3Out, ESXi Host to VDS, etc.)

Site1-Pod1 Site1-Pod2

1/11 Site1-L3Out
1/11 1/11
1/17 .1 .5 1/19 1/11 .9 .13 1/19
1/17

.14
.2 .6 Stretched ESXi Cluster .10
1/17 1/19 1/17 1/19

.101 .105

Connectivity to the 1/47 WAN

1/47
.106
WAN network is .102

pre-provisioned 1/7 1/9

Ecommerce Tenant External Address Space
1/1 192.168.200.0/24
192.168.100.0/24

#CLUS BRKACI-2291 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
Demo 5
Extend ‘Ecommerce’ tenant configuration to Pod2
Simplified Tenant
Management
through MSO
Simplified Tenant Management through MSO
Configuration Steps

• Initial setup of MSO

• Adding the Multi-Pod fabric as first site on MSO
• Importing existing ‘Ecommerce’ tenant configuration on MSO

#CLUS BRKACI-2291 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
 ACI Multi-Site
BGP Inter-Site Peers

Inter-Site
Network
• EVPN-RID, O-UTEP and O-MTEP addresses
Anycast VTEP Addresses:
O-UTEP & O-MTEP
are assigned from the Multi-Site
Orchestrator and must be routable across
the ISN

EVPN-RID 4 • Inter-site communication always happens

encapsulating traffic to one of the Anycast
EVPN-RID 1
EVPN-RID 2 EVPN-RID 3 TEP address (O-UTEP for L3/L3 unicast
forwarding, O-MTEP for BUM forwarding)

#CLUS BRKACI-2291 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 37
 Adding the DR Site on MSO
Assign Routable TEP Addresses and BGP EVPN Router-IDs

O-UTEP-S1P1: 172.16.100.101
O-UTEP-S1P2: 172.16.100.102
O-MTEP-S1: 172.16.100.100 IPN/ISN
BGP Speaker 1: 172.16.100.201
BGP Speaker 2: 172.16.100.202
Site1-Pod 1 Site1-Pod 2
BGP Speaker 2
O-UTEP-S1P1 O-UTEP-S1P2
O-MTEP-S1

BGP Speaker 1
Site1-L3Out

WAN

Site 1

#CLUS BRKACI-2291 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 38
Demo 6
Simplified Tenant Management through MSO
Adding the DR Site
on MSO
Adding the DR Site on MSO
Configuration Steps

• Installing the DR fabric (already done)

• Adding the DR fabric as a second site on MSO (assign routable TEP
addresses and BGP EVPN Router-IDs)
• Verifying IPN connectivity
• Extending the tenant ‘Ecommerce’ to the DR site
• Create a local L3Out in the DR fabric
• Extending the existing ‘Ecommerce’ tenant configuration to the DR site
• Verify external connectivity

#CLUS BRKACI-2291 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
Adding the DR Site on MSO
Site2-Pod1 Fabric

Site2-Pod1
S2P1-Spine201 S2P1-Spine202

S2P1-Leaf101 S2P1-Leaf102

ESXi Cluster 2

BDSOL-ACI38-APIC1 BDSOL-ACI38-APIC2 BDSOL-ACI38-APIC3

#CLUS BRKACI-2291 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
Adding the DR Site on MSO
Parameters for the APIC Initial Setup Script (Already Done)

S2P1-APIC1 S2P1-APIC2 S2P1-APIC3

Fabric name Fabric2 Fabric2 Fabric2

Fabric ID 1 1 1

Active controllers 3 3 3

Pod ID 1 1 1

Controller ID 1 2 3

TEP Pool 10.2.0.0/16 10.2.0.0/16 10.2.0.0/16

Infra VLAN 3937 3937 3937

#CLUS BRKACI-2291 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 43
 Adding the DR Site on MSO
Assign Routable TEP Addresses and BGP EVPN Router-IDs

O-UTEP-S1P1: 172.16.100.101 O-UTEP-S1P1: 172.16.200.101

O-UTEP-S1P2: 172.16.100.102 O-MTEP-S1: 172.16.200.100
O-MTEP-S1: 172.16.100.100 BGP Speaker 1: 172.16.200.201
IPN/ISN BGP Speaker 2: 172.16.200.202
BGP Speaker 1: 172.16.100.201
BGP Speaker 2: 172.16.100.202 BGP Speaker 2
Site1-Pod 1 Site1-Pod 2 Site2-Pod 1
BGP Speaker 2
O-UTEP-S1P1 O-UTEP-S1P2 O-UTEP-S1P1
O-MTEP-S1 O-MTEP-S2

BGP Speaker 1 BGP Speaker 1

Site1-L3Out Site2-L3Out

WAN

Site 1 Site 2

#CLUS BRKACI-2291 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 44
Adding the DR Site on MSO
Verifying IPN connectivity
IPN1 IPN3
S2P1-Leaf201

1/48 .9
.1 1/48
.2
IPN2 IPN4
IPN Site2 Infra Address Space:
172.16.102.0/24
.13
TEP Pool:
1/48
.5 1/48
10.2.0.0/16
.10 1/3
1/2 .2 1/1
.14
1/4 .1
.6 .18 .17 .6
1/5 .5
1/5
S2P1-Leaf202
WAN Infra Address Space: IPN5
172.16.110.0/24
Site2-Pod1

#CLUS BRKACI-2291 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 45
 Adding the DR Site on MSO
Create a Local L3Out in the DR Site
Site2-Pod1

Site2-L3Out

1/17 .17 .21 1/19

.22
.18
Ecommerce Tenant External Address Space 1/17 1/19
192.168.200.0/24
.109 1/47

.110 1/11

1/1
192.168.100.0/24

#CLUS BRKACI-2291 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 46
Demo 7
Adding the DR Site on MSO
MSO Additional
Functionalities
MSO Additional Functionalities
• End host connectivity verification
• Host route advertisement (inbound traffic optimization)
• Enabling CloudSec encryption between sites

#CLUS BRKACI-2291 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
ACI Multi-Site
CloudSec Encryption for VXLAN Traffic Encrypted Fabric to Fabric Traffic
[GCM-AES-256-XPN (64-bit PN)])
CloudSec = “TEP-to-TEP MACSec”

VTEP IP MACSec VXLAN Tenant Packet

VTEP Information
in Clear Text
Inter-Site Network

MP-BGP - EVPN

Multi-Site
Orchestrator

Supported from ACI 4.0(1) release for FX line cards and 9332C/9364C platforms

#CLUS BRKACI-2291 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 50
Demo 8
MSO Additional Functionalities
Complete your
online session • Please complete your session survey
evaluation after each session. Your feedback
is very important.
• Complete a minimum of 4 session
surveys and the Overall Conference
survey (starting on Thursday) to
receive your Cisco Live water bottle.
• All surveys can be taken in the Cisco Live
Mobile App or by logging in to the Session
Catalog on ciscolive.cisco.com/us.
Cisco Live sessions will be available for viewing
on demand after the event at ciscolive.cisco.com.

#CLUS BRKACI-2291 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 52
Continue your education

Demos in the
Walk-in labs
Cisco campus

Meet the engineer

Related sessions
1:1 meetings

#CLUS BRKACI-2291 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 53
Thank you

#CLUS
#CLUS