Вы находитесь на странице: 1из 34

ACI deployment and migration:

Design considerations and best practices


Part 2: ACI Multi-Pod and ACI Multi-Site
Max Ardica
Principal Engineer - DCNBU
Partner DC VT (Amsterdam, 26/03/19)
Agenda

• Multi-Pod and Multi-Site Positioning


• Multi-Site Orchestrator Deployment Considerations

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Multi-Pod and Multi-Site
When to Position One vs. the Other?

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
When to Position Multi-Pod?

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
ACI Multi-Pod
Overview
VXLAN
Inter-Pod
Network
Pod ‘A’ Pod ‘n’

MP-BGP - EVPN


Up to 50 msec RTT

APIC Cluster
IS-IS, COOP, MP-BGP IS-IS, COOP, MP-BGP

Availability Zone

• Multiple ACI Pods connected by an IP Inter-Pod L3 • Forwarding control plane (IS-IS, COOP) fault
network, each Pod consists of leaf and spine nodes isolation
• Up to 50 msec RTT supported between Pods • Data Plane VXLAN encapsulation between Pods
• Managed by a single APIC Cluster • End-to-end policy enforcement
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
• Single Management and Policy Domain
Single AZ with with Tenant Isolation
Isolation for ‘Virtual Network Zone and Application’ Changes

Inter-Pod
Network

ACI Multi-Pod
Fabric

APIC Cluster

Tenant ‘Prod’ Configuration/Change Domain Tenant ‘Dev’ Configuration/Change Domain

• The ACI ‘Tenant’ construct provide a domain of application and associated virtual network policy
change
• Domain of operational change for an application (e.g. production vs. test)

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

6
ACI Multi-Pod
Most Common Use Cases
▪ Need to scale up a single ACI fabric above
200 leaf nodes supported in a single Pod Pod
▪ Handling 3-tiers physical cabling layout (for Inter-Pod
Leaf Nodes Network
example traditional N7K/N5K/N2K
deployments)
• Alternative to the Multi-Tier fabric Spine Nodes
(supported from ACI release 4.1(1))

▪ True Active/Active DC deployments


Pod 1 Pod 2
Single VMM domain across DCs (ESXi
Metro Cluster, vSphere HA/FT, DRS,…)
Deployment of clustered network services
(FWs, SLBs)
Application clustering (L2 BUM extension DB
APIC Cluster
Web/App Web/App
across Pods)

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
ACI Multi-Pod and VMM Integration
IPN

Pod 1 Pod 2

VMM Domain
DC1

HV HV HV vSwitch1 HV HV HV

• Cluster of Hypervisors stretched across Pods


➢ Single VMM domain created across Pods
• Logical switch extended across the hypervisors part of the same stretched cluster
• Support for all intra-cluster functions (vSphere HA/FT, DRS, etc.)
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Connecting Multi-Pod to the External L3 Domain
Use of Local or Remote L3Outs

▪ A Pod does not need to have a dedicated WAN


connection (i.e. can offer transit services to other
Pods) MP-BGP - EVPN

▪ Multiple WAN connections can be deployed


across Pods
▪ Outbound traffic: by default VTEPs always select Pod 1 Pod 2

WAN connection in the local Pod based on


preferred metric
WAN WAN
▪ Leaf nodes in Pods without local L3Outs will
load-balance traffic between L3Outs in remote Pod 3

Pods

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Typical options for an
Multi-Pod and Network Services Active/Active DC use case

Integration Models
ISN

▪ Active and Standby pair deployed across Pods


▪ No issues with asymmetric flows

Active Standby

ISN • Active/Active FW cluster nodes stretched across Sites


(single logical FW)
• Requires the ability of discovering the same MAC/IP info in
separate sites at the same time
Active/Active Cluster • Supported from ACI release 3.2(4d) with the use of
Service-Graph with PBR

ISN
▪ Independent Active/Standby pairs deployed in separate
Pods
▪ Use of Symmetric PBR to avoid the creation of
asymmetric paths crossing different active FW nodes
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Active/Standby Active/Standby
When to Position Multi-Site?

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
ACI Multi-Site
Overview VXLAN
Inter-Site
Network

MP-BGP - EVPN
Multi-Site
Orchestrator

Site 1 Site 2
REST
GUI
API Availability Zone ‘B’
Availability Zone ‘A’
Region 1
• Separate ACI Fabrics with independent APIC clusters • MP-BGP EVPN control plane between sites
• No latency limitation between Fabrics • Data Plane VXLAN encapsulation across sites
• ACI Multi-Site Orchestrator pushes cross-fabric configuration to • End-to-end policy definition and enforcement
multiple APIC clusters providing scoping of all configuration
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
changes
ACI Multi-Site
Most Common Use Cases
• Scale-up model to build a very • Data Center Interconnect (DCI)
large intra-DC network (above • Extend connectivity and policy between
the number of leaf nodes ‘loosely coupled’ DC sites
supported in an ACI fabric) • Disaster Recovery and IP mobility main use
cases
✓ Tight control on BD extension across sites

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
ACI Multi-Site Connectivity Options Should be the behavior for the
majority of BDs with Multi-Site
Per Bridge Domain Behavior
Layer 3 only across sites IP Mobility without BUM flooding Layer 2 adjacency across Sites
1 2 3
ISN ISN ISN
Site Site Site Site 2
Site Site Site
1 2 1 2 1 2

▪ Bridge Domains and subnets not ▪ Same IP subnet defined in ▪ Interconnecting separate sites for
extended across Sites separate Sites fault containment and scalability
Support for IP Mobility (‘cold’ and reasons
▪ Layer 3 Intra-VRF or Inter-VRF ▪
communication (shared services ‘live’* VM migration) and intra- ▪ Layer 2 domains stretched across
across VRFs/Tenants) subnet communication across sites Sites, support application
clustering
▪ No Layer 2 BUM flooding across
sites ▪ Layer 2 BUM flooding across sites

MSO GUI MSO GUI MSO GUI


(BD) (BD) (BD)
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

*’Live’ migration officially supported from ACI release 3.2 14


ACI Multi-Site and VMM Integration
ISN

Site 1 VMM Domain VMM Domain Site 2


DC1 DC2

vCenter vCenter
Server 1 Server 2

SRM SRM
HV HVVDS1 HV
EPG1 HV HVVDS2 HV
EPG1

Live vMotion/Cold Migration


• Live virtual machines migration across sites is supported only with vCenter deployments
(both for single or multiple vCenter options)
▪ Requires vSphere 6.0 and newer, no support for DRS, vSphere HA/FT
• Use of Site Recovery Manager (SRM) or similar higher level orchestrator for workload
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
recovery across sites
Connecting Multi-Site to the External L3 Domain
Restricting the Use of Local L3Outs
Supported Design
✓ Not Supported Design
(planned for ACI 4.2 release) ❌
Inter-Site Network Inter-Site Network
X

L3Out L3Out L3Out L3Out


Site 1 Site 2 Site 1

Site 2

WAN WAN

Note: the same consideration applies to both Border Leaf L3Outs and GOLF L3Outs
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Deployment options fully
Multi-Site and Network Services supported with ACI Multi-Pod

Integration Models
ISN

• Active and Standby pair deployed across Pods


• Currently supported only if the FW is in L2 mode or in
L3 mode but acting as default gateway for the
endpoints
Active Standby

Active/Active FW cluster nodes stretched across Sites


ISN

(single logical FW)
• Requires the ability of discovering the same MAC/IP info
in separate sites at the same time
Active/Active Cluster
• Not supported

ISN • Recommended deployment model for ACI Multi-Site


• Option 1: supported from 3.0 for N-S if the FW is
connected in L3 mode to the fabric  mandates the
deployment of traffic ingress optimization
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential • Option 2: supported from 3.2 release with the use of
Active/Standby Active/Standby Service Graph with Policy Based Redirection (PBR)
Multi-Pod and Multi-Site
Complementary and not Alternative

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Foundational Requirement
Creation of Two Independent Fabrics/AZs

Fabric ‘A’ (AZ 1)

Fabric ‘B’ (AZ 2)

Application
workloads deployed
© 2019 across Cisco
Cisco and/or its affiliates. All rights reserved. availability
Confidential
zones
Foundational Requirement
Creation of Two Independent Fabrics/AZs

Multi-Pod Fabric ‘A’ (AZ 1)

‘Classic’ Active/Active

Pod ‘1.A’ Pod ‘2.A’

ACI Multi-Site

Multi-Pod Fabric ‘B’ (AZ 2)

‘Classic’ Active/Active

Pod ‘1.B’Application Pod ‘2.B’


workloads deployed
© 2019 across Cisco
Cisco and/or its affiliates. All rights reserved. availability
Confidential
zones
Multi-Site Orchestrator
Deployment Considerations

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
ACI Multi-Site
MSO Schema and Templates
Schema
▪ Template = ACI policy definition
(ANP, EPGs, BDs, VRFs, etc.)
▪ Schema = container of Templates
sharing a common use-case
• As an example, a schema can be dedicated
to a Tenant

▪ The template is currently the atomic


unit of change for policies
• Such policies are concurrently pushed to
one or more sites Site 1 Site 2
▪ Scope of change: policies in different EFFECTIVE
POLICY
EFFECTIVE
POLICY
templates can be pushed to
separate sites at different times
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
ACI Multi-Site
Schema and Templates Definition for the DR Use Case
Future
Schema Schema Schema

Template 1 Template 1 Template 2 Template 1


EP1 EP2 EP1 EP2 EP1 EP2 EP1 EP2
C C C C
EPG EPG EPG EPG EPG EPG EPG EPG

t1 t1 t1 t2 t1 t2

Prod Site DR Site Prod Site DR Site Prod Site DR Site

▪ Single Template associated to Prod ▪ Separate Template associated to Prod ▪ Single Template associated to Prod
and DR Sites and DR Sites and DR Sites
▪ Any change applied to the template ▪ Changes made to a template can be ▪ Capability of independently apply
is pushed to both sites applied only to the mapped site changes to each site
simultaneously
▪ Requires sync between the two ▪ Brings together the advantages of
▪ Easiest way to keep consistent
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
templates (manual or performed by an the previous two options
policies deployed across sites higher level Orchestration tool)
How to Define Schemas, Templates
and the Mapping to ACI Sites?

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Schema Design
One Template per Site, plus a Common Template

Schema
ANP1 Site 1
Template
Site 1
EPG1 EPG2 BD1 BD2

Site 2
ANP1
Template
Site 2
EPG3 EPG4 BD3 BD4

ANP1 Site 3 Site 3


Template
EPG5 EPG6 BD5 BD6

ANP1 VRF
BD7 C1 C2
EPG7
Contracts

Common Template (for stretched objects)


© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Schema Design
Deployment Considerations

▪ All objects defined inside the schema are visible and can be referenced via the
drop-down list
• This is not the case for object referenced across schemas  for those it is required to digit at least 3
letters of their names to be displayed and then create references
▪ Current support limited to 5 templates per schema
• With four sites you could have a template per site and one stretched template (would not scale to support
other combinations)
▪ Be aware of the maximum object limit in the same schema (500 objects is the
current limit)
• Every object that can be defined in a template counts (EPGs, BDs, VRFs, Contracts, etc.)
• May make sense to locally define on APIC objects that are only used locally in a site
▪ Note: increasing both the number of templates and number of objects in a schema
is planned for a future ACI release (2HCY19)
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Multiple Schemas Design
One Application per Schema, plus a Schema for Common Objects

App1 APN 1
Schema
Site 1
Template
Site 1
EPG1 EPG2 BD1 BD2

APN 2 Site 2 Site 2


Template
EPG3 EPG4 BD3 BD4

App2
Site 3
APN 3 Site 2 & 3
Schema Template
EPG5 EPG6 BD5 BD6

VRF VRF 1 VRF 2


schema C1 C2

Contracts
Common Template
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Multiple Schemas Design
Deployment Considerations

▪ Allows to easily identify application in the schema window and verify their health in
the MSO dashboard
▪ Applications can easily be localized or stretched across sites
▪ Less likely to hit the current 500 objects limit per schema
▪ Cross-schema objects can’t be referenced through the drop-down menu, need to
perform a manual search (using at least 3 letters from their name)

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
How to Define the Policies inside a
Template for a Given Tenant?

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
ACI Multi-Site Orchestrator
Defining Policies in a Template
Green Field Deployment Import Policies from an Existing Fabric

Site 1
Site 1 Site 1 Site 1

1a 1b 2b
2a
Site 2
Site 2 Site 2 Site 2

2 2 1

Site 1 Site 2 Site 1 Site 2


Green Field Green Field Existing Fabric Green Field
1a. Model new tenant and policies to a common template on MSO 1. Import existing tenant policies from site 1 to new common and
and associate the template to both sites (for stretched objects) site-specific templates on MSO
1b. Model new tenant and policies to site-specific templates and 2a. Associate the common template to both sites (for stretched objects)
associate them to each site 2b. Associate site-specific templates to each site
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
2. Push policies to the ACI sites 3. Push the policies back to the ACI sites
ACI Multi-Site Orchestrator
Defining Policies in a Template (2)
Import Policies from Multiple Existing Fabrics

▪ In the current implementation, MSO does not


Site 1
Site 1

2a 2b
Site 2
Site 2 allow diff/merge operations on policies from
different APIC domains
1 1 ▪ It is still possible to import policies for the
3 same tenant from different APIC domains,
under the assumption those are no
conflicting
Site 1 Site 2 • Tenant defined with the same Name
Existing Fabric Existing Fabric • Name and policies for stretched objects are
1. Import existing tenant policies from site 1 and site 2 to new
also common
common and site-specific templates on ACI MSO
2a. Associate the common template to both sites (for stretched objects)
2b. Associate site-specific
© 2019 Cisco templates
and/or its affiliates. to each
All rights reserved. site
Cisco Confidential

3. Push the policies back to the ACI sites


ACI Multi-Pod
Where to Go for More Information

✓ ACI Multi-Pod White Paper


http://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-
infrastructure/white-paper-c11-737855.html?cachemode=refresh

✓ ACI Multi-Pod Configuration Paper


https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-
infrastructure/white-paper-c11-739714.html

✓ ACI Multi-Pod and Service Node Integration White Paper


https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-
infrastructure/white-paper-c11-739571.html

✓ BRKACI-2003 @ Cisco Live Barcelona 2019


https://ciscolive.cisco.com/on-demand-library/?search=BRKACI-
2003#/session/1532112828758001tmf6

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
ACI Multi-Site
Where to Go for More Information

✓ ACI Multi-Site White Paper


https://www.cisco.com/c/en/us/solutions/collateral/data-center-
virtualization/application-centric-infrastructure/white-paper-c11-739609.html

✓ Deploying ACI Multi-Site from Scratch


https://www.youtube.com/watch?v=HJJ8lznodN0

✓ BRKACI-2125 @ Cisco Live Barcelona 2019


https://ciscolive.cisco.com/on-demand-library/?search=BRKACI-
2125#/session/1532112831071001tQVx

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Вам также может понравиться