Brkaci 2642

ACI Troubleshooting
Layer 3 Out (L3Out)
Takuya Kishida – Technical Marketing, DCBU ACI
BRKACI-2642
Cisco Webex Teams
Questions?
Use Cisco Webex Teams to chat
with the speaker after the session
How
1 Find this session in the Cisco Events Mobile App
2 Click “Join the Discussion”
3 Install Webex Teams or go directly to the team space
4 Enter messages/questions in the team space
BRKACI-2642 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Agenda
• L3Out Key Components

• Learning routes (Routing Protocol)
• Distributing routes within ACI (MP-BGP)
• Advertising ACI subnet
• Contract on L3Out (prefix based EPG)
• L3Out Subnet scope options

• Summary of all options
• Export Route Control Subnet example
Acronyms/Definitions Reference Slide
Acronyms Definitions
ACI Application Centric Infrastructure
APIC Application Policy Infrastructure Controller
EP Endpoint
EPG Endpoint Group
BD Bridge Domain
VRF Virtual Routing and Forwarding
L3Out Layer 3 Out (External Routed Network)
L3Out EPG Layer 3 Out EPG, Prefix Based EPG (External Network Instance)
MP-BGP Multi Protocol BGP
VPNv4 Virtual Private Network Version 4
RT Route Target
RD Route Distinguisher
Why L3Out?
Why L3Out?
• What is EPG for ?
➢ Endpoint (EP) = MAC & /32 IP (or /128)
VRF • What is BD Subnet for ?

➢ To be a default gateway
➢ For ACI Fabric to know a subnet for EPs in a
BD Subnet A BD
This is for Spine-Proxy
Please check BRKACI-3545 for details
EPG EPG
EP A1 EP A2 EP A3 R1 MAC R1
IP R1
MAC A1 MAC A2 MAC A3 A network device (ex. router,

IP A1 IP A2 IP A3
IP A4 loadbalancer) as an endpoint?
IP X.X.X.X/8
IP Y.Y.Y.Y/8
IP Z.Z.Z.Z/8
Why L3Out?
• What is EPG for ?
➢ Endpoint (EP) = MAC & /32 IP (or /128)
VRF • What is BD Subnet for ?

➢ To be a default gateway
➢ For ACI Fabric to know a subnet for EPs in a
BD
BD Subnet A BD Subnet R, X, Y, Z
A network device as an endpoint?

EPG EPG EPG ➢ All IPs as /32 in a single endpoint
leaf1# show endpoint vlan 84
R1 84 vlan-5 0000.0000.R1R1 L po3
EP A1 EP A2 EP A3 MAC R1 TK:VRF1 vlan-5 R.R.R.1 L po3
IP R1 TK:VRF1 vlan-5 X.X.X.1 L po3
IP X1 – X999 TK:VRF1 vlan-5 X.X.X.2 L po3
MAC A1 MAC A2 MAC A3 IP Y1 – Y999 TK:VRF1 vlan-5 X.X.X.3 L po3
IP A1 IP A2 IP A3 IP Z1 – Z999 .....
IP A4 TK:VRF1 vlan-5 Y.Y.Y.1 L po3
.....
TK:VRF1 vlan-5 Z.Z.Z.1 L po3
.....
IP X.X.X.X/8 ※ One endpoint can have up to 1024 IPs in ACI

IP Y.Y.Y.Y/8 This does not scale and efficient.
IP Z.Z.Z.Z/8 No need to learn each IP as /32.
Why L3Out? • What is L3Out for ?
➢ To connect ACI with other network domain
= devices with multiple subnet behind it
VRF • How is L3Out different from EPG?

➢ Speak Routing Protocol
➢ No IP learning as endpoint
➢ Next-hop IP is stored in ARP table
BD Subnet A L3Out Routing Protocol
= Same as normal routers
Next-hop MAC in endpoint table

EPG EPG L3Out EPG leaf1# show endpoint vlan 84
84/TK:VRF1 vxlan-14876665 0000.0000.R1R1 L po3
EP A1 EP A2 EP A3 R1 MAC R1 Next-hop IP in ARP table (only for L3Out)
IP R1
leaf1# show ip arp vlan 84
MAC A1 MAC A2 MAC A3 Address Age MAC Address Interface
IP A1 IP A2 IP A3 R.R.R.1 00:07:51 0000.0000.R1R1 vlan84
IP A4
Other routes via Routing Protocol
leaf1# show ip route vrf TK:VRF1
X.0.0.0/8, ubest/mbest: 1/0
*via R.R.R.1, vlan84, [110/5], 2d00h, ospf-default, intra
IP X.X.X.X/8 Y.0.0.0/8, ubest/mbest: 1/0
IP Y.Y.Y.Y/8 *via R.R.R.1, vlan84, [110/5], 2d00h, ospf-default, intra
IP Z.Z.Z.Z/8 Z.0.0.0/8, ubest/mbest: 1/0
*via R.R.R.1, vlan84, [110/5], 2d00h, ospf-default, intra
L3Out
Key Components
L3Out Key Components
1. Learn external routes
Distribute ➢ Routing Protocol in L3Out
External Routes
VRF Overlay-1
2. Distribute external routes to other
leaves
non-BLEAF BLEAF
➢ MP-BGP
VRF1 VRF1 3. Advertise internal routes (BD subnet)

BD L3Out to outside
➢ Redistribution
L3Out
EPG and
Advertise EPG Learn
Internal Routes External Routes
➢ Contract
(Export) (Import)
4. Allow traffic with contracts
➢ L3Out EPG (Prefix Based EPG)
and
Allow traffic ➢ Contract
1. Learn External Routes = Routing Protocol
Configurations
External Routed Networks (L3Out)
• VRF to deploy Routing Protocol Routing Protocol
• Routing Protocol parameters Information
ex. OSPF area 0.0.0.1 nssa Only on
configured nodes
Node Profile (Border LEAF)
• Node(s) to deploy Routing Protocol
• Static Route (if any)
Interface Profile
• I/F(s) to deploy Routing Protocol
• Routing Protocol I/F parameters
VRF1 VRF1 VRF1
ex. OSPF hello interval
BD L3Out L3Out
Networks (L3Out EPG)
• Contract
• Advanced Route Control ※ Details for L3Out EPG are in later sections
ex. route-map BRKACI-2642 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
Verification Examples (OSPF)
1. Is OSPF enabled on a correct I/F?
border-leaf# show ip ospf int bri vrf TK:VRF1
Interface ID Area Cost State Neighbors Status
Same CLI verifications
Vlan58 134 backbone 4 BDR 2 up are as useful in ACI too
border-leaf# show vlan id 58 extended
VLAN Name Encap Ports If anything is not as
---- -------------------------------- ---------------- ---------------------- expected, check config
58 TK:VRF1:l3out- vxlan-15695748, Eth1/3, Po2
L3OUT_OSPF:vlan-1425 vlan-1425
or any faults in APIC GUI.
2. Are OSPF parameters matching with neighbors?

border-leaf# show int vlan 58 | grep MTU
Is MTU matching?
MTU 1500 bytes, BW 10000000 Kbit, DLY 1 usec
Is Network Mask matching?
border-leaf# show ip ospf int vlan 58 | egrep 'IP|State|Timer|auth' Is Area matching?
IP address 15.0.0.3/24, Process ID default VRF TK:VRF1, area backbone Is Timer matching?
State BDR, Network type BROADCAST, cost 4 Is Network Type expected?
Timer intervals: Hello 10, Dead 40, Wait 40, Retransmit 5
etc.
No authentication
3. Are OSPF neighbors established correctly? Can they ping to each other?
border-leaf# show ip ospf neighbors vrf TK:VRF1 leaf# iping –V <VRF> <target IP>
Neighbor ID Pri State Up Time Address Interface
※OSPF DBD requires unicast reachability
4.4.4.4 1 FULL/DR 2d06h 15.0.0.4 Vlan58
9.9.9.9 1 FULL/DROTHER 2d06h 15.0.0.1 Vlan58 etc.
Verification Examples (EIGRP)
1. Is EIGRP enabled on a correct I/F?
border-leaf# show ip eigrp int bri vrf TK:VRF1
Xmit Queue Mean Pacing Time Multicast Pending
Same CLI verifications
Interface Peers Un/Reliable SRTT Un/Reliable Flow Timer Routes are as useful in ACI too
vlan92 2 0/0 1 0/0 50 0
border-leaf# show vlan id 92 extended If anything is not as

VLAN Name Encap Ports expected, check config
---- -------------------------------- ---------------- ----------------------
92 TK:VRF1:l3out- vxlan-14712828, Eth1/3, Po2
or any faults in APIC GUI.
L3OUT_EIGRP:vlan-1426 vlan-1426
2. Are EIGRP parameters matching with neighbors?

border-leaf# show int vlan 92 | grep MTU border-leaf# show ip eigrp vrf TK:VRF1 | egrep 'AS|K'
MTU 1500 bytes, BW 10000000 Kbit, DLY 1 usec IP-EIGRP AS 1 ID 3.3.3.3 VRF TK:VRF1
Metric weights: K1=1 K2=0 K3=1 K4=0 K5=0
border-leaf# show ip int vlan 92 | grep 'IP addr'
IP address: 16.0.0.3, IP subnet: 16.0.0.0/24
Is MTU matching?
Is Network Mask matching?
3. Are EIGRP neighbors established correctly? Is AS matching?
border-leaf# show ip eigrp neighbors vrf TK:VRF1 Is K value matching?
H Address Interface Hold Uptime SRTT RTO Q Seq etc.
(sec) (ms) Cnt Num
0 16.0.0.4 vlan92 12 2d06h 1 50 0 10
1 16.0.0.1 vlan92 13 2d06h 1 50 0 346
Verification Examples (BGP)
1. Is BGP neighbor session configured as expected?
border-leaf# show ip bgp neighbors vrf TK:VRF1 | egrep 'BGP nei|Using|Opens|hops'
BGP neighbor is 17.0.0.1, remote AS 65001, ebgp link, Peer index 1
Using Loopback6 as update source for this peer
External BGP peer might be upto to 2 hops away Is it correct remote AS?
Opens: 1 1 Is it using correct source I/F with
correct IP?
border-leaf# show ip int lo6 | grep 'IP addr' Is enough multi-hop configured
IP address: 3.3.3.3, IP subnet: 3.3.3.3/32 for eBGP?
Is Open message exchanged?
2. Is there IP reachability ?
border-leaf# iping -V TK:VRF1 17.0.0.1 -S 3.3.3.3
PING 17.0.0.1 (17.0.0.1) from 3.3.3.3: 56 data bytes
64 bytes from 17.0.0.1: icmp_seq=0 ttl=255 time=0.76 ms
64 bytes from 17.0.0.1: icmp_seq=1 ttl=255 time=0.639 ms Is there an IP reachability to the BGP neighbor
=== snip === from the correct source IP?
--- 17.0.0.1 ping statistics ---
5 packets transmitted, 5 packets received, 0.00% packet loss
3. Are BGP neighbors established correctly? Is it receiving BGP routes?

border-leaf# show ip bgp summary vrf TK:VRF1 Is ACI BGP using expected local AS?
BGP router identifier 3.3.3.3, local AS number 65003
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd

17.0.0.1 4 65001 3300 3302 78 0 0 2d06h 2
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
2. Distribute External Routes = MP-BGP in infra
Implement all steps except
Configurations for step 1 (user L3Out)
Pod Profile
MP-BGP in Route Reflectors
Pod Policy Group VRF Overlay-1
BGP Route Reflector Policy 5 4 3 2

Import To Route Reflector Export
• default To other LEAFs
back to VRF1 to MP-BGP
from MP-BGP 10.0.0.0/8 (VRF1) 10.0.0.0/8 (VRF1)
-> LEAF2 -> Local
System Settings
10.0.0.0/8 10.0.0.0/8
VRF1 VRF1
BGP Route Reflector -> LEAF2 -> local
• ACI BGP AS number EPG L3Out

(for both MP-BGP and L3Out BGP)
• MP-BGP Route Reflector Spines 1
user L3Out
10.0.0.0/8 (Routing Protocol
or Static Route)
2. Distribute External Routes = MP-BGP in infra
1. Select ACI BGP AS and Route Reflector SPINEs 2. Apply Route Reflector policy to Pod Policy Group
Use default
3. Apply Pod Policy Group

※ L3Out BGP share this same AS with the internal MP-BGP to Pod Profile
※ Check appendix for MP-BGP details
CLI Verification
1. Do both border leaf and non-border leaf have BGP sessions with RR spines?
leaf# show bgp sessions vrf overlay-1
Neighbor ASN Flaps LastUpDn|LastRead|LastWrit St Port(L/R) Notif(S/R)
10.0.184.65 65003 0 2d07h |never |never E 37850/179 0/0
10.0.184.66 65003 0 2d07h |never |never E 45089/179 0/0
leaf# acidiag fnvread | grep spine

1001 1 spine1 FGE10000000 10.0.184.65/32 spine active 0
1002 1 spine2 SAL10000000 10.0.184.66/32 spine active 0
2. Is the external route learned on a border leaf? ✓ BGP neighbors are RR spines TEP IPs
border-leaf# show ip route vrf TK:VRF1
10.0.0.0/8, ubest/mbest: 1/0
*via 15.0.0.1, Vlan58, [110/5], 2d08h, ospf-default, intra
✓ Next-hops are border Leaf TEP IPs
3. Does non-border leaf show the expected border leaf as next-hop? ✓ Learned via iBGP in ACI AS# (65003)
non-border-leaf# show ip route vrf TK:VRF1
10.0.0.0/8, ubest/mbest: 2/0
*via 10.0.184.67%overlay-1, [200/5], 2d08h, bgp-65003, internal, tag 65003
non-border-leaf# acidiag fnvread

ID Pod ID Name Serial Number IP Address Role State
LastUpdMsgId
--------------------------------------------------------------------------------------------------------
103 1 leaf3 SAL10000003 10.0.184.64/32 leaf active 0
104 1 leaf4 SAL10000004 10.0.184.67/32 leaf active 0
3. Advertise BD subnet
Configurations
Bridge Domain (BD)
VRF Overlay-1
BD Subnet Redistribution
• Subnet A Direct (Subnet A) BLEAF non-BLEAF
✓ “Advertised Externally” -> L3Out Protocol
VRF1 VRF1
Associated L3Out
• Target L3Out(s) L3Out Subnet A BD
to advertise BD subnets
L3Out
EPG
EPG
No BD Subnet A on BLEAF yet

➢ MP-BGP is only to distribute external routes
➢ MP-BGP never distributes BD subnets
Configurations
Bridge Domain (BD)
VRF Overlay-1
BD Subnet Redistribution
• Subnet A Direct (Subnet A) BLEAF non-BLEAF
✓ “Advertised Externally” -> L3Out Protocol
VRF1A
Subnet VRF1
Associated L3Out
• Target L3Out(s) L3Out Subnet A BD
to advertise BD subnets
L3Out
EPG
EPG
External Routed Networks (L3Out)
Static Route (subnet A)
Networks (L3Out EPG) on BLEAF via MO (object) Pushed by APIC. Not MP-BGP.
• Contract to EPG Please check “pervasive gateway”
in BRKACI-3545
1. L3Out Association from BD (for redistribution) 2. Contract

border-leaf# show ip ospf vrf TK:VRF1 192.168.0.0/24, ubest/mbest: 1/0, attached, direct, pervasive
*via 10.0.184.64%overlay-1, [1/0], 04:32:27, static
Redistributing External Routes from
direct route-map exp-ctx-st-2326530 ip prefix-list Make sure the other end of contract is configured correctly as well
➢ 192.168.1.0/24 21
BRKACI-2642 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
CLI Verification (OSPF, EIGRP)
1. Does the border leaf have BD subnet to advertise?
border-leaf# show ip route vrf TK:VRF1 If not, check the contract between the L3Out EPG
192.168.1.0/24, ubest/mbest: 1/0, attached, direct, and the EPG for the BD.
pervasive This should be pushed by APIC. Not via MP-BGP.
2. Check a route-map name used by the routing protocol on the border leaf for redistribution
border-leaf# show ip ospf vrf TK:VRF1 border-leaf# show ip eigrp vrf TK:VRF1
Redistributing External Routes from Redistributing: Check next page for BGP
direct route-map exp-ctx-st-2097152 direct route-map exp-ctx-st-2097152
3. Does the route-map have expected BD subnet?

IP prefix-list should have the BD
border-leaf# show route-map exp-ctx-st-2097152 subnet.
route-map exp-ctx-st-2097152, deny, sequence 1
Match clauses:
If not, check APIC config and any
tag: 4294967295 faults.
Set clauses: ✓ Is “Advertise Externally” on the
route-map exp-ctx-st-2097152, permit, sequence 15804 BD subnet checked?
Match clauses: ✓ Is L3Out associated to the BD?
ip address prefix-lists: IPv4-st49158-2097152-exc-int-inferred-export-dst
ipv6 address prefix-lists: IPv6-deny-all
Set clauses:
border-leaf# show ip prefix-list IPv4-st49158-2097152-exc-int-inferred-export-dst
ip prefix-list IPv4-st49158-2097152-exc-int-inferred-export-dst: 1 entries
seq 1 permit 192.168.1.254/24 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
CLI Verification (BGP)
1. Does the border leaf have BD subnet to advertise?
--- snip ---
2. Check a route-map name used by BGP outbound rule for each neighbor
border-leaf# show bgp process vrf TK:VRF1
Information for address family IPv4 Unicast in VRF TK:VRF1 BGP redistributes all direct routes first,
Redistribution then limit the routes with an outbound route-map.
direct, route-map permit-all
border-leaf# show ip bgp neighbors vrf TK:VRF1 | egrep '^BGP|Out'

Outbound route-map configured is exp-l3out-L3OUT_BGP-peer-2097152, handle obtained
3. Does the BGP outbound route-map have the expected BD subnet?

IP prefix-list should have the BD
border-leaf# show route-map exp-l3out-L3OUT_BGP-peer-2097152 subnet.
route-map exp-l3out-L3OUT_BGP-peer-2097152, permit, sequence 15801 If not, check APIC config and any
Match clauses:
ip address prefix-lists: IPv4-peer49157-2097152-exc-int-inferred-export-dst faults.
ipv6 address prefix-lists: IPv6-deny-all ✓ Is “Advertise Externally” on the
Set clauses: BD subnet checked?
route-map exp-l3out-L3OUT_BGP-peer-2097152, deny, sequence 16000 ✓ Is L3Out associated to the BD?
Match clauses:
route-type: direct
Set clauses:
border-leaf# show ip prefix-list IPv4-peer49157-2097152-exc-int-inferred-export-dst
ip prefix-list IPv4-peer49157-2097152-exc-int-inferred-export-dst: 1 entries
seq 1 permit 192.168.1.254/24
4. Prefix based Contract
Learning
10.0.0.0/8 and 20.0.0.0/8
through routing protocol Should be able to
talk with 10.0.0.0/8
10.0.0.0/8
L3Out
? EPG
20.0.0.0/8
Should NOT be able

to talk with 20.0.0.0/8
How do we accomplish this ??

Learning
10.0.0.0/8 and 20.0.0.0/8
through routing protocol
10.0.0.0/8 L3Out EPG A
Subnet 10.0.0.0/8
✓ External EPG
L3Out EPG
L3Out EPG B
Subnet 20.0.0.0/8
20.0.0.0/8 ✓ External EPG
Prefix Based EPG (= L3Out EPG)
Configurations
External Routed Networks (L3Out) VRF
Node Profile L3Out BD

L3Out
EPG
Interface Profile EPG
Networks (L3Out EPG) EPG (Security Group)

Classification
• A subnet with scope EPG Classification
“External Subnets for based on prefix Prefix Mapping
the External EPG”
VLAN + I/F
This scope is VRF wide.
No overlapping with other L3Out EPGs in the same VRF
Traffic from LEAF front panel port

VRF1 – 10.0.0.0/8 => pcTag 49158
leaf# show vrf TK:VRF1 detail extended | grep vxlan

Encap: vxlan-2097152
leaf# vsh_lc -c 'show system internal aclqos prefix' | egrep 'Vrf|10.0.0.0'

Vrf-Vni VRF-Id Table-Id Addr Class Shared Remote Complete
2097152 8 0x8 10.0.0.0/8 49158 0 1 No
=== use this command from 3.2 ===

leaf# vsh –c ‘show sytem internal policy-mgr prefix’
“External Subnets for the External EPG” is
4. Prefix based Contract to declare this subnet belongs to this
L3Out EPG
➢ To create prefix to pcTag mapping
NOTE:
It has nothing to do with routing table or
routing protocol behavior unlike other
Route Control Subnet scopes
A common mistake is selecting both

“External Subnets for the External EPG”
and “Export Route Control Subnet” for the
same subnet, which implies a conflicting
situation where the subnet behind the
L3Out but the same L3Out is also
expected to advertise/redistribute the
subnet back to where it came from. It may
not cause an immediate issue but
unnecessary redistribution should always
be avoided.
Check L3Out Subnet scope section for
details.
L3Out
192.168.1.1 EPG
CLI Verifications Contract Drop on this leaf shows up in this command.
EPG 10.0.0.0/8
1. Check if there is any contract drops Check both ingress/egress leaf just in case,
or see appendix for Policy Control Enforcement Direction
leaf# show logging ip access-list internal packet-log deny
[ Wed May 8 18:34:31 2019 155907 usecs]: CName: TK:VRF1(VXLAN: 2719744), VlanType: FD_VLAN, Vlan-Id: 26, SMac: 0x0050569185d1,
DMac:0x0022bdf819ff, SIP: 192.168.1.1, DIP: 10.0.0.1, SPort: 58968, DPort: 80, Src Intf: port-channel1, Proto: 6, PktLen: 74
2. Check VRF VNID pcTag/contract is per VRF

leaf# show vrf TK:VRF1 detail extended | grep vxlan except for shared service (VRF route leaking)
If your source/destination is an endpoint, it should

3. Check source (or destination) EPG pcTag be in here.
sclass = pcTag = EPG ID for contract
leaf# show system internal epm endpoint ip 192.168.1.1 | egrep 'VRF|sclass'
Vlan id : 30 ::: Vlan vnid : 9025 ::: VRF name : TK:VRF1
BD vnid : 16318374 ::: VRF vnid : 2097152 Make sure the external IP is not here.
Flags : 0x80005c04 ::: sclass : 49100 ::: Ref count : 5 This pcTag takes precedence over “prefix-pcTag
EP Flags : local|IP|MAC|host-tracked|sclass|timer| mapping table”. If it is, check the traffic path that
caused ACI to learn the external IP as an endpoint.
4. Check destination (or source) L3Out prefix based EPG pcTag
leaf# vsh_lc -c 'show system internal aclqos prefix' | egrep 'Vrf|10.0.0.0'
2097152 8 0x8 10.0.0.0/8 49200 0 1 No “External Subnet for the External EPG”
config is reflected here.
=== use this command from 3.2 === This is Longest Prefix Match.
leaf# vsh –c ‘show sytem internal policy-mgr prefix’ © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
L3Out
192.168.1.1 EPG
CLI Verifications 49100
EPG 10.0.0.0/8
49200
5. Check contracts between two pcTags scope = VRF VNID
leaf# show zoning-rule scope 2097152 | egrep ‘Rule|49100|49200’
Rule ID SrcEPG DstEPG FilterID operSt Scope Action Priority
4165 49100 49200 5 enabled 2097152 permit fully_qual(7)
leaf# show zoning-filter filter 5

FilterId Name EtherT ArpOpc Prot ~snip~ SFromPort SToPort DFromPort DToPort ~snip~
======== ====== ====== ========= ======= ~snip~ ======= ==== ==== ==== ~snip~
5 5_0 ip unspecified icmp ~snip~ unspecified unspecified unspecified unspecified ~snip~
6. Check ELAM to see if the traffic is using correct src pcTag and dst pcTag
https://dcappcenter.cisco.com/elam-assistant.html
L3Out Contract
Common Issue (L3Out EPGs with 0.0.0.0/0)
VRF 1 L3Out A
L3Out EPG A
10.0.0.0/8
0.0.0.0/0
✓ External EPG 0.0.0.0/0 should cover
10.0.0.0/8
EPG X
L3Out B
L3Out EPG B
20.0.0.0/8
0.0.0.0/0
✓ External EPG 0.0.0.0/0 should cover
20.0.0.0/8
Both 10.0.0.0/8 and 20.0.0.0/8 can talk to EPG X

even though there is no contract between L3Out EPG B and EPG X Do not overlap
➢ Prefix-pcTag mapping is per VRF. External EPG
subnets
0.0.0.0/0 for L3Out A and B ends up in the same entry.
L3Out
L3Out Contract 192.168.1.1 EPG X

EPG A
L3Out
10.0.0.0/8
Common Issue (L3Out EPGs with 0.0.0.0/0) EPG B 20.0.0.0/8
1. Check VRF VNID

“0.0.0.0/0 -> 15” is the only pcTag entry
in this VRF.
2. Check source (or destination) EPG pcTag ➢ Both L3Out A & B will share it since
there is no other granular LPM entries
leaf# show system internal epm endpoint ip 192.168.1.1 | egrep 'VRF|sclass'
Vlan id : 30 ::: Vlan vnid : 9025 ::: VRF name : TK:VRF1
BD vnid : 16318374 ::: VRF vnid : 2097152 NOTE:
Flags : 0x80005c04 ::: sclass : 49100 ::: Ref count : 5 • 0.0.0.0/0 always use pcTag 15
• This is not a routing table. It doesn’t matter even if
the routing table has more granular routes
3. Check destination L3Out 0.0.0.0/0 EPG pcTag
leaf# vsh_lc -c 'show system internal aclqos prefix' | egrep ‘Vrf|2097152'
2097152 8 0x8 0.0.0.0/0 15 0 0 No
This contract is due to “EPG X <-> L3Out A”

4. Check contracts between pcTags But any traffic that hits 0.0.0.0/0 in the prefix table
leaf# show zoning-rule scope 2097152 | egrep ‘Rule|49162’
can use this rule
L3Out Subnet Scope
L3Out Subnet Scope
L3Out Subnet Scope
Route Control for Routing Protocol
• Export Route Control Subnet
• Import Route Control Subnet
• Shared Route Control Subnet
Traffic Classification for Contract

Grouping by
• External Subnets for the External EPG functionality
• Shared Security Import Subnet
Aggregate
• Aggregate Export
• Aggregate Import
• Aggregate Shared Routes
L3Out Subnet Scope Summary
Only for contracts
No impact in routing table
Route Control for Routing Protocol Subnet Classification

Export Route Control Subnet Import Route Control Subnet Shared Route Control Subnet External Subnet for
(Mainly for Transit Routing) (Only for OSPF or BGP) the External EPG
L3Out L3Out VRF1 VRF2 L3Out
shared-filter
Protocol Protocol L3Out L3Out L3Out
Database Database EPG1 EPG2
Protocol
export-filter Database
import-filter
10.0.0.0/8 20.0.0.0/8
Advertise the route from ACI Receive the route from outside Leak the external route to Group subnets into
to outside (by default, receive all) different VRF each L3Out EPG (pcTag)
aggregation aggregation aggregation
Aggregate Export Aggregate Import Aggregate Shared Route Shared Security Import
VRF1 VRF2
L3Out L3Out VRF1 VRF2
X.X.X.X/X le 32
Protocol Protocol L3Out
L3Out
Database Database L3Out L3Out
Protocol
EPG1 EPG1
0.0.0.0/0 le 32 Database
0.0.0.0/0 le 32
10.0.0.0/8
Advertise all routes from ACI Receive all routes from Leak multiple external Leak prefix-pcTag mapping
to outside outside routes to different VRF
© 2020 to different VRF
Cisco and/or its affiliates. All rights reserved. Cisco Public
Route Control Enforcement
Import is disabled by default.
➢ Receive all routes by
default.
No import route control.
Export is always enabled.
Available
Route only
Control for when
Routing enabled
Protocol
Export Route Control Subnet Import Route Control Subnet Shared Route Control Subnet
(Only for OSPF or BGP)
VRF1 VRF2
L3Out L3Out
shared-filter
Database Database Protocol
Database
export-filter
import-filter
Advertise the route from ACI Receive the route from outside Leak the external route to
to outside (by default, receive all) different VRF
aggregation aggregation aggregation
Aggregate Export Aggregate Import Aggregate Shared Route
L3Out L3Out VRF1 VRF2
X.X.X.X/X le 32
Database Database Protocol
Database
0.0.0.0/0 le 32
0.0.0.0/0 le 32
Advertise all routes from ACI Receive all routes from Leak multiple external
to outside outside routes to different VRF
Export Route Control (OSPF)
VRF overlay-1
MP-BGP (vpnv4)
ACI Border LEAF
User VRF Route Import Route Export

BGP (IPv4) Route-maps
Redistribute
permit-all
L3Out 1
permit-all
OSPF Protocol Database
External Router BRKACI-2642 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 38
Export Route Control (OSPF)
VRF overlay-1
MP-BGP (vpnv4)
Creates a route-map
ACI Border LEAF

Redistribute export
permit-all
IP prefix-list
L3Out 1 10.0.0.0/8
permit-all
== NOTE ==
Be careful when deploying
multiple L3Outs in one VRF.
Route maps are shared with
other protocols (L3Out) in the
same VRF on the same LEAF.
10.0.0.0/8
Export Route Control User

VRF
L3Out
2
Protocol
Database
(OSPF) Another Border LEAF

BGP (IPv4)
VRF overlay-1
MP-BGP (vpnv4)
ACI Border LEAF

From another L3Out BGP (IPv4) Route-maps
on different LEAF Redistribute
using MP-BGP export
permit-all
L3Out 1 export
Redistribute
permit-all
10.0.0.0/8

VRF
L3Out
2
Protocol
Database

BGP (IPv4)
VRF overlay-1
MP-BGP (vpnv4)
Advertise external routes
ACI Border LEAF from other L3Out(s)
➢ Transit Routing
From another L3Out BGP (IPv4) Route-maps
on different LEAF Redistribute
using MP-BGP export
permit-all
L3Out 1 export L3Out 3

Redistribute
permit-all
export
OSPF Protocol Database Protocol
Database
Redistribute
From another L3Out
or Area-filter on same LEAF
10.0.0.0/8
10.0.0.0/8

VRF
L3Out
2
Protocol
Database

BGP (IPv4)
VRF overlay-1
Advertise BD subnets MP-BGP (vpnv4)
➢ 2nd method Advertise external routes
NOTE: ACI Border LEAF from other L3Out(s)
From 3.0, “Advertised Externally” ➢ Transit Routing
on BD subnet is also required with
User VRF
this method Route Import Route Export
Redistribute
export
permit-all
BD L3Out 1 export L3Out 3

Redistribute
permit-all
export
Protocol
export
Subnets RIB OSPF Protocol Database

Database
Redistribute
Redistribute or Area-filter
10.0.0.0/8
CLI Verification (OSPF/EIGRP)
1. OSPF/EIGRP Redistribution route-map
border-leaf# show ip ospf vrf TK:VRF1 It shares the same route-map with other
Redistributing External Routes from protocols in the same VRF on the same LEAF
static route-map exp-ctx-st-2097152
direct route-map exp-ctx-st-2097152
route-map naming:
bgp route-map exp-ctx-proto-2097152
eigrp route-map exp-ctx-proto-2097152 exp-ctx-st-<vrf vnid> or
Area (backbone) exp-ctx-proto-<vrf vnid>
Area-filter in 'exp-ctx-proto-2097152'
border-leaf# show ip eigrp vrf TK:VRF1
Redistributing: EIGRP doesn’t support Transit Routing on a same LEAF.
static route-map exp-ctx-st-2097152 ➢ No equivalent filter like OSPF area-filter in EIGRP
ospf-default route-map exp-ctx-proto-2097152
bgp-65003 route-map exp-ctx-proto-2097152
All Export Route Control subnet on a
2. route-map and ip prefix-list same LEAF is added here
border-leaf# show route-map exp-ctx-proto-2097152
route-map exp-ctx-proto-2097152, permit, sequence 15801 Same goes to exp-cxt-st-2097152
Match clauses:
ip address prefix-lists: IPv4-proto49158-2097152-exc-ext-inferred-export-dst
Set clauses:
border-leaf# show ip prefix-list IPv4-proto49158-2097152-exc-ext-inferred-export-dst
tag 4294967295
ip prefix-list IPv4-proto49158-2097152-exc-ext-inferred-export-dst: 1 entries
seq 1 permit 10.0.0.0/8 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
CLI Verification (BGP) BGP has a route-map per L3Out
➢ A bit more granular control
route-map naming:
1. BGP outbound route-map exp-l3out-<bgp l3out name>-peer-<vrf vnid>
border-leaf# show ip bgp neighbors vrf TK:VRF1
Outbound route-map configured is exp-l3out-L3OUT_BGP-peer-2097152, handle obtained
2. route-map and ip prefix-list

All Export Route Control subnets from
border-leaf# show route-map exp-l3out-L3OUT_BGP-peer-2097152
the same BGP L3Out is added here
route-map exp-l3out-L3OUT_BGP-peer-2097152, permit, sequence 15804
Match clauses:
ip address prefix-lists: IPv4-peer49157-2097152-exc-ext-inferred-export-dst
Set clauses:
tag 4294967295
route-map exp-l3out-L3OUT_BGP-peer-2097152, deny, sequence 16000
Match clauses:
route-type: direct
Set clauses:
border-leaf# show ip prefix-list IPv4-peer49157-2097152-exc-ext-inferred-export-dst
ip prefix-list IPv4-peer49157-2097152-exc-ext-inferred-export-dst: 4 entries
seq 1 permit 10.0.0.0/8
BD Subnet and Export Route Control
192.168.1.0/24, ubest/mbest: 1/0, attached, direct, pervasive “Advertised Externally” removes VRF
*via 11.0.248.0%overlay-1, [1/0], 00:00:05, static, tag 4294967295
tag from BD subnet

192.168.1.0/24, ubest/mbest: 1/0, attached, direct, pervasive
IP prefix-list from
“Export Route Control”
Prior to 3.0 border-leaf# show route-map exp-ctx-st-2097152
route-map exp-ctx-st-2097152, permit, sequence 15804 for 192.168.1.0/24
Match clauses:
Set clauses:
From 3.0 border-leaf# show route-map exp-ctx-st-2097152

route-map exp-ctx-st-2097152, deny, sequence 1
Match clauses:
New rule to prevent BD subnets tag: 4294967295 IP prefix-list from
without “Advertised Externally” Set clauses: “Export Route Control”
from being advertised route-map exp-ctx-st-2097152, permit, sequence 15804 for 192.168.1.0/24
Match clauses:
Set clauses:
Import Route Control (OSPF)
VRF overlay-1
MP-BGP (vpnv4)
Creates a route-map
ACI Border LEAF

Redistribute
export
permit-all
import
L3Out
IP prefix-list
10.0.0.0/8 permit-all
Import Route Control (OSPF)
VRF overlay-1 If it’s not allowed to be imported, it should

MP-BGP not be exported to other L3Outs.
(vpnv4)
When L3Out2 is not OSPF
ACI Border LEAF Redistribution is blocked by table-map
(only routes in RIB can be redistributed)
When L3Out2 is OSPF
User VRF Limit external routesRoute Import Route Export
Block advertisement to another OSPF area
to be used in RIB. BGP (IPv4) Route-
via area-filter
It is still in OSPF LSDB. Redistribute maps
export
permit-all
import
BD L3Out L3Out 2
permit-all
import
import
export
Subnets RIB Protocol
Database
Table-map area-filter
out Still need export in L3Out2
(OSPF) on top of import in
L3Out1 (OSPF)
CLI Verification (OSPF) • Table-map to prevent the routes from being
border-leaf# show ip ospf vrf TK:VRF1
Table-map using route-map exp-ctx-2097152-deny-external-tag
used in RIB
• “Area-filter out” to prevent the routes from
Area (backbone) being advertised to another OSPF area on a
Area-filter out 'imp-ctx-ospf-area20971520' same LEAF (Transit Routing)
border-leaf# show route-map exp-ctx-2097152-deny-external-tag

route-map exp-ctx-2097152-deny-external-tag, deny, sequence 1
route-map for table-map
Match clauses: 1. blocks any routes with VRF tag
tag: 4294967295 2. allow routes with Import Route Control
Set clauses: subnet in OSPF area X
route-map exp-ctx-2097152-deny-external-tag, permit, sequence 15801 3. block any routes from OSPF area X
Match clauses:
ip address prefix-lists: IPv4-ospf-49158-2097152-exc-ext-inferred-import-dst-rtpfx
ospf-area: backbone
Set clauses:
Match clauses:
ospf-area: backbone
Set clauses:
route-map exp-ctx-2097152-deny-external-tag, permit, sequence 20000 A prefix configured by “Import Route
Match clauses: Control Subnet”
Set clauses:
border-leaf# show ip prefix-list IPv4-ospf-49158-2097152-exc-ext-inferred-import-dst-rtpfx
ip prefix-list IPv4-ospf-49158-2097152-exc-ext-inferred-import-dst-rtpfx: 1 entries
seq 1 permit 10.0.0.0/8
CLI Verification (OSPF) cont. • Table-map to prevent the routes from being
border-leaf# show ip ospf vrf TK:VRF1
used in RIB
• “Area-filter out” to prevent the routes from
Area (backbone) being advertised to another OSPF area on a
Area-filter out 'imp-ctx-ospf-area20971520' same LEAF (Transit Routing)
border-leaf# show route-map imp-ctx-ospf-area20971520

route-map imp-ctx-ospf-area20971520, permit, sequence 15801 route-map for area-filter
Match clauses:
ip address prefix-lists: IPv4-ospf-rt-ospf-import49158-2097152-exc-ext-inferred-import-dst-
Set clauses:
border-leaf# show ip prefix-list IPv4-ospf-rt-ospf-import49158-2097152-exc-ext-inferred-import-dst-

ip prefix-list IPv4-ospf-rt-ospf-import49158-2097152-exc-ext-inferred-import-dst-: 1 entries
seq 1 permit 10.0.0.0/8
A prefix configured by “Import Route

Control Subnet”
CLI Verification (BGP)
BGP uses an inbound route-map (per L3Out) instead of table-map
border-leaf# show ip bgp neighbors vrf TK:VRF1
Inbound route-map configured is imp-l3out-L3OUT_BGP-peer-2097152, handle obtained
border-leaf1# show route-map imp-l3out-L3OUT_BGP-peer-2097152

route-map imp-l3out-L3OUT_BGP-peer-2097152, permit, sequence 15801
Match clauses:
ip address prefix-lists: IPv4-peer49157-2097152-exc-ext-inferred-import-dst
Set clauses:
border-leaf# show ip prefix-list IPv4-peer49157-2097152-exc-ext-inferred-import-dst

ip prefix-list IPv4-peer49157-2097152-exc-ext-inferred-import-dst: 1 entries
seq 1 permit 10.0.0.0/8
A prefix configured by “Import Route

Control Subnet”
Shared Route Control Subnet
(VRF Route Leaking) Configuration in VRF1 L3Out
VRF overlay-1
MP-BGP (vpnv4)
User Route Import Route Export User Route Import Route Export
VRF1 VRF2
BGP (IPv4) BGP (IPv4)
Redistribute
Import RT
permit-all <AS>:<VRF2 VNID> Route-maps
L3Out == default ==
Only import L3Out routes for
OSPF Protocol Database same VRF (VRF2) from other
LEAF
VRF overlay-1
MP-BGP (vpnv4)
VRF1 VRF2
Redistribute
Import RT
L3Out <AS>:<VRF1 VNID>
Contract across VRFs
OSPF Protocol Database EPG ➢ Import VRF1
routes as well
VRF overlay-1
MP-BGP (vpnv4)
shared
VRF1 VRF2
Redistribute
Import RT
L3Out <AS>:<VRF1 VNID>
shared
IP prefix-list
OSPF Protocol Database EPG 10.0.0.0/8
Limit routes to be
imported (leaked)
CLI Verification
1. MP-BGP Import rule with another VRF VNID route-target and a route-map
leaf# show bgp process vrf TK:VRF2
Information for address family IPv4 Unicast in VRF TK:VRF2

• It always has Import and Export RT for its
Import route-map 2588672-shared-svc-leak own VRF2 VNID (65003:2588672)
Export RT list:
65003:2588672 • VRF Route Leaking is handled by Import RT
Import RT list:
65003:2097152
and Import route-map (highlighted ones)
65003:2588672
Label mode: per-prefix
leaf# show vrf TK:VRF1 detail extended | egrep 'RD|vxl'

RD: 10.0.184.64:2
Encap: vxlan-2097152 VRF VNID can be checked with this
command to confirm Import RT is correct
leaf# show vrf TK:VRF2 detail extended | egrep 'RD|vxl'
RD: 10.0.184.64:13
CLI Verification
2. A route-map for shared service (VRF Route Leaking) 1. Prevent BD subnet (pervasive
route) from being imported via
leaf# show route-map 2588672-shared-svc-leak MP-BGP.
route-map 2588672-shared-svc-leak, deny, sequence 1 BD subnet distribution should be
Match clauses:
pervasive: 2 done by APIC instead of MP-BGP.
Set clauses:
route-map 2588672-shared-svc-leak, permit, sequence 2 2. Allow importing any routes from
Match clauses: the same VRF.
extcommunity (extcommunity-list filter): 2588672-shared-svc-leak Extended community list has RT
Set clauses:
route-map 2588672-shared-svc-leak, permit, sequence 1000 for the same VRF VNID.
Match clauses:
ip address prefix-lists: IPv4-2097152-32771-18-2588672-shared-svc-leak 3. Allow importing certain routes
ipv6 address prefix-lists: IPv6-deny-all from another VRF
Set clauses:
leaf# show ip extcommunity-list 2588672-shared-svc-leak RT for the same VRF VNID

Standard Extended Community List 2588672-shared-svc-leak
Not for VRF Route Leaking
permit RT:65003:2588672
leaf# show ip prefix-list IPv4-2097152-32771-18-2588672-shared-svc-leak IP Prefix-List from Shared

ip prefix-list IPv4-2097152-32771-18-2588672-shared-svc-leak: 1 entries
seq 1 permit 10.10.10.0/8 Route Control Subnet
Shared Security Import Subnet
Routing table is leaked via MP-BGP
and “Shared Route Control Subnet”
MP-BGP
User User
VRF1 VRF2
Prefix <-> pcTag mapping is leaked
via APIC and “Shared Security Import
Subnet”
VRF1 RIB Prefix – pcTag mapping VRF2 RIB Prefix – pcTag mapping
10.0.0.0/8 -> Local VRF1: 10.0.0.0/8 -> pcTag X 10.0.0.0/8 -> LEAF 1 in VRF 1 10.0.0.0/8 -> pcTag X
Routing
Protocol
CLI Verification
1. Check VRF VNID
leaf# show vrf TK:VRF2 detail extended | grep vxlan pcTag (class) for shared route
prefix-pcTag mapping is sahred
2. Prefix – pcTag mapping table to VRF2 (VNID 2588672)
1st-gen-leaf# vsh_lc –c ‘show system internal aclqos prefix’ | egrep '^Shared|52.52.52'

Shared Addr Mask Scope Class RefCnt
10.0.0.0 ffffff 0 18 1
2nd-gen-leaf# vsh_lc -c 'show system internal aclqos prefix' | egrep 'Shared|52.52.52'

2097152 8 0x8 10.0.0.0/8 18 0 1 No
2588672 11 0xb 10.0.0.0/8 18 1 1 No
leaf# vsh -c 'show system internal policy-mgr prefix'

Vrf-Vni VRF-Id Table-Id Table-State VRF-Name Addr Class Shared Remote Complete
2097152 8 0x8 Up TK:VRF1 10.0.0.0/8 18 True True False
2588672 11 0xb Up TK:VRF2 10.0.0.0/8 18 True True False
From 3.2 release, use this command

regardless of leaf generations
Aggregate Route Control
== Export == == Import == == Shared ==
Route-maps Route-maps Route-maps
export import shared

IP prefix-list
IP prefix-list IP prefix-list
10.0.0.0/8 le
0.0.0.0/0 le 32 0.0.0.0/0 le 32
32
permit-all permit-all permit-all
Only “Aggregate Shared Routes” support non-0.0.0.0/0 aggregation

OSPF route-map
Export Route Control Subnet xxxxxxxx
Import Route Control Subnet xxxxxxxxx
Per
VRF/LEAF
VRF overlay-1
MP-BGP (vpnv4)
ACI Border LEAF

BGP (IPv4) Route-
Redistribute maps
export-proto
permit-all export-static
Redistribute or
export-
BD L3Out proto Redistribute area-filter in L3Out 2 deny-external
import-ospf
export-
export-
proto
static
Protocol permit-all
Subnets RIB Redistribute
OSPF Protocol Database Database
import-
export
external
deny-
opsf
Still needs export in
Table-map area-filter
out L3Out2 on top of
import in L3Out1
EIGRP route-map
Export Route Control Subnet xxxxxxxx
Import is not supported for EIGRP deny-external

Used only for VRF Tag
Per
VRF overlay-1 VRF/LEAF
MP-BGP (vpnv4)
ACI Border LEAF

BGP (IPv4) Route-
Redistribute maps
export-proto
permit-all export-static
export-
BD L3Out proto Redistribute L3Out 2 deny-external
Redistribute
export-
static
permit-all
export-
Protocol
proto
Subnets RIB Redistribute
EIGRP Protocol Database Database
external
deny-
Table-map
BGP route-map
Export Route Control Subnet export-l3out
Import Route Control Subnet import-l3out
Per
VRF overlay-1 L3Out/LEAF
MP-BGP (vpnv4)
ACI Border LEAF

Route-
L3Out maps
export-l3out
import-l3out
Redistribute
BGP (IPv4) L3Out 2
permit-all
Protocol
Database permit-all
export-l3out import-l3out
outbound route-map Inbound route-map
Routing Loop
Loop Avoidance for OSPF/EIGRP – VRF Tag
EIGRP and MP-BGP Redistribution Issue
Routing Loop Avoidance - VRF tag (OSPF/EIGRP)
Set VRF tag Block routes with its own VRF tag
(By default 4294967295)
when exporting routes It may overwrite the original route
(By default 4294967295 for all VRF) “10.0.0.0/8 => Router 1”
User VRF
L3Out 1 L3Out 2 L3Out 3
Protocol Database Protocol Database Protocol Database
export
10.0.0.0/8 => Router 3
10.0.0.0/8 => Router 1 (tag 4294967295)
10.0.0.0/8 => L3Out 2

Router 1 Router 2 (tag 4294967295) Router 3
10.0.0.0/8
※ VRF tagging for exported routes and blocking routes with VRF tag are always enabled
It will be blocked in other VRFs
Set VRF tag as well since all VRF use same
when exporting routes VRF tag by default
(By default 4294967295 for all VRF)
User VRF 1 User VRF 2
export
10.0.0.0/8 => Router 3
10.0.0.0/8 => Router 1 (tag 4294967295)
10.0.0.0/8 => L3Out 2

10.0.0.0/8
※ VRF tagging for exported routes and blocking routes with VRF tag are always enabled
VRF2’s tag is 200 not 100.
➢ Routes are not blocked
Set VRF tag 100
when exporting routes
User VRF 1 User VRF 2
export
10.0.0.0/8 => Router 3
10.0.0.0/8 => Router 1 (tag 100)
10.0.0.0/8 => L3Out 2

10.0.0.0/8
VRF 1 tag – 100 Do this when routes need to be

VRF 2 tag – 200 advertised back to ACI
VRF tag can be

configured per VRF
In this example VRF1’s tag is 100
--- snip ---
※ VRF tag is only for OSPF and EIGRP

leaf# show ip ospf vrf TK:VRF1 | egrep 'route-map|Redis'

eigrp route-map exp-ctx-proto-2097152
bgp route-map exp-ctx-proto-2097152
Export routes with VRF tag
leaf# show route-map exp-ctx-proto-2097152

route-map exp-ctx-proto-2097152, permit, sequence 15802
Match clauses:
ip address prefix-lists: IPv4-proto49158-2097152-exc-ext-inferred-export-dst
Set clauses:
tag 100
leaf# show ip prefix-list IPv4-proto49158-2097152-exc-ext-inferred-export-dst
ip prefix-list IPv4-proto49158-2097152-exc-ext-inferred-export-dst: 1 entries
seq 1 permit 10.0.0.0/8
Always there with VRF tag
note:
Import Route Control Subnet is added
here after VRF tag deny rule when
Import Route Control subnet is used.
leaf# show ip ospf vrf TK:VRF1 | egrep 'route-map|Redis'
eigrp route-map exp-ctx-proto-2097152
bgp route-map exp-ctx-proto-2097152 Block routes with
VRF tag
leaf# show route-map exp-ctx-2097152-deny-external-tag

Match clauses:
tag: 100
Set clauses:
route-map exp-ctx-2097152-deny-external-tag, permit, sequence 200
Match clauses:
Set clauses:
EIGRP & MP-BGP Redistribution Issue
1. 10.0.0.0/8 from L3Out 1 via EIGRP on two Border LEAFs
VRF overlay-1
MP-BGP (vpnv4)
User Route Import BGP (IPv4)

Route Export
User Route Import
BGP (IPv4)
Route Export
VRF1 VRF1
Redistribute Redistribute
L3Out1 L3Out1
EIGRP Topology EIGRP Topology
10.0.0.0/8 FD 100000 10.0.0.0/8 FD 100000
10.0.0.0/8 BRKACI-2642 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 69
2. L3Out 1 exports all routes (including 10.0.0.0/8)
VRF overlay-1
MP-BGP (vpnv4)

Route Export
User Route Import
BGP (IPv4)
Route Export
VRF1 VRF1
Redistribute Redistribute Redistribute Redistribute
L3Out1 0.0.0.0/0 le 32 L3Out1 0.0.0.0/0 le 32
10.0.0.0/8 FD 100000 10.0.0.0/8 FD 100000
3. Redistributed routes have lower metric than the original
VRF overlay-1
MP-BGP (vpnv4)

Route Export
User Route Import
BGP (IPv4)
Route Export
VRF1 VRF1
Redistribute Redistribute Redistribute Redistribute
L3Out1 0.0.0.0/0 le 32 L3Out1 0.0.0.0/0 le 32

10.0.0.0/8 FD 51200 10.0.0.0/8 FD 51200
Overwrite the 10.0.0.0/8 FD 100000 Overwrite the 10.0.0.0/8 FD 100000

original original
EIGRP & MP-BGP Redistribution Solution1
Export only necessary routes
VRF overlay-1
MP-BGP (vpnv4)

Route Export
User Route Import
BGP (IPv4)
Route Export
VRF1Redistribute VRF1 Redistribute

Redistribute Redistribute
L3Out1 20.0.0.0/8 L3Out1 20.0.0.0/8

Not Redistributed Not Redistributed
back back
10.0.0.0/8 FD 100000 10.0.0.0/8 FD 100000
Add “set metric” rule to export route-map
VRF overlay-1
MP-BGP (vpnv4)

Route Export
User Route Import
BGP (IPv4)
Route Export
VRF1Redistribute Set metric VRF1 Set metric

Redistribute Redistribute Redistribute
L3Out1 0.0.0.0/0 le 32 L3Out1 0.0.0.0/0 le 32

10.0.0.0/8 FD 2588162 10.0.0.0/8 FD 2588162
Lower metric Lower metric
Not used 10.0.0.0/8 FD 100000 Not used 10.0.0.0/8 FD 100000
Add “set metric” rule to export route-map
L3Out Contract
deep dive
L3Out Contract
pcTag (policy control Tag) in normal EPG
Source EP Forwarding
Learning
On APIC Lookup
EPG A EPG B Source EPG Destination EPG

ICMP Check Check
VLAN + I/F Forwarding Result
On LEAF
get pcTag
source destination Filter

pcTag A pcTag B ICMP pcTag A pcTag B
Contract Filter Check
L3Out Contract
Src: Subnet A -> Dst: Subnet B
pcTag (policy control Tag) in L3Out EPG
Learning
On APIC Lookup
L3Out EPG A L3Out EPG B

Subnet A Subnet B Source EPG (pcTag) Destination EPG (pcTag)
✓ External EPG ✓ External EPG Check Check
ICMP
On LEAF VLAN + I/F Forwarding Result
VRF subnet pcTag pcTag VRF
VRF1 subnet A pcTag A

VRF1 subnet B pcTag B Prefix To pcTag mapping for L3Out
VRF1 0.0.0.0/0 pcTag 15
Hit subnet A
pcTag A pcTag B
source destination Filter Hit subnet B
pcTag A pcTag B ICMP

default catch all
not used in this example BRKACI-2642 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 77
L3Out Contract
pcTag (policy control Tag) in L3Out EPG with 0.0.0.0/0
Learning
On APIC Lookup

0.0.0.0/0 Subnet B Source EPG (pcTag) Destination EPG (pcTag)
ICMP
VRF1 subnet B pcTag B Prefix To pcTag mapping for L3Out

VRF1 0.0.0.0/0 pcTag 15
No Hit for subnet A
pcTag VRF Keep pcTag from pcTag B
source destination Filter VLAN + I/F
pcTag VRF pcTag B ICMP
default catch all
not used in this example BRKACI-2642 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 78
L3Out Contract
pcTag (policy control Tag) in L3Out EPG with 0.0.0.0/0
Learning
On APIC Lookup

Subnet A 0.0.0.0/0 Source EPG (pcTag) Destination EPG (pcTag)
ICMP
VRF1 subnet A pcTag A

Prefix To pcTag mapping for L3Out
VRF1 0.0.0.0/0 pcTag 15
pcTag
pcTag A
source destination Filter No Hit for subnet B 15
Use default pcTag 15
pcTag A pcTag 15 ICMP
L3Out Contract
Common Issue (L3Out EPGs with 0.0.0.0/0) Prefix-pcTag entry is per VRF.
Default catch all (0.0.0.0) is shared
On APIC with everyone in the VRF.
VRF 1 L3Out A On LEAF

L3Out EPG A
0.0.0.0/0 VRF subnet pcTag
10.0.0.0/8
✓ External EPG
VRF1 0.0.0.0/0 pcTag 15
EPG X ICMP
L3Out B source destination Filter
L3Out EPG B pcTag X pcTag 15 ICMP
0.0.0.0/0 pcTag VRF pcTag X ICMP
20.0.0.0/8
✓ External EPG
These contracts are from EPG X and L3Out A
However, traffic from/to L3Out B (20.0.0.0/8) will also use
default pcTag (VRF or 15) due to 0.0.0.0/0 config.
No overlap of External EPG L3Out subnets in same VRF

Use 0.0.0.0/0 (External subnet for the external EPG) only for one L3Out EPG per VRF
How to get pcTag for L3Out
How to get pcTag for normal EPG
== Policy tab ==
➢ Check EPG’s pcTag
== Operational tab ==
➢ Check if the endpoint is learned on
the expected EPG
How to get VRF pcTag
• From APIC
admin@apic1:~> moquery -c fvCtx -f 'fv.Ctx.name=="VRF1"' | egrep '#|dn|pcTag'
# fv.Ctx
dn : uni/tn-TK/ctx-VRF1
pcTag : 49153
• From LEAF
leaf# vsh_lc -c 'show system internal eltmc info vrf TK:VRF1' | grep sclass
scope: 4 ::: sclass: 49153
Under VRF
L3Out Contract
Policy Control Enforcement Direction
A feature to save contract TCAM usage on border LEAF
No effects on
APIC L3Out EPG X EPG <-> EPG traffic
EPG A
Subnet X
EPG B ✓ External EPG
ICMP
Egress Policy Enforcement Ingress Policy Enforcement default

from 1.2
Non-Border LEAF(s) Border Non-Border LEAF(s) Border

LEAF(s) LEAF(s)
with EPG A with EPG A
source destination Filter source destination Filter
pcTag A pcTag X ICMP source destination Filter pcTag A pcTag X ICMP source destination Filter
pcTag A pcTag X ICMP
with EPG B with EPG B - none -
pcTag B pcTag X ICMP
source destination Filter source destination Filter
pcTag B pcTag X ICMP pcTag B pcTag X ICMP
Under VRF
L3Out Contract
Policy Control Enforcement Direction
How does it affect traffic flow and contract?
Egress Policy Enforcement Ingress Policy Enforcement
EPG -> L3Out Contract is applied Contract is applied EPG -> L3Out
on Egress LEAF on Ingress LEAF
EPG L3Out EPG EPG L3Out EPG
Otherwise if remote EP exists,

Contract is applied
EPG <- L3Out EPG <- L3Out
Contract is applied Contract is applied
on Egress LEAF on Ingress LEAF on Egress LEAF
EPG L3Out EPG EPG L3Out EPG
L3Out Contract EPG -> L3Out
Contract is applied on
Egress LEAF
CLI Verification
Ingress Compute LEAF Verification EPG L3Out EPG
for pcTag 49162 pcTag 16391
Egress Policy Enforcement
Points to the
correct border leaf
TEP Prefix-pcTag mapping table
Routing Table
leaf# show ip route 52.52.52.0/24 vrf TK:VRF1 leaf# show vrf TK:VRF1 detail extended | grep vxlan
52.52.52.0/24, ubest/mbest: 1/0 Encap: vxlan-2097152
*via 11.0.64.64%overlay-1, [200/5], 00:00:14, bgp-65003, internal,
tag 65003 leaf# vsh -c 'show system internal policy-mgr prefix' | grep 2097152
recursive next hop: 11.0.64.64/32%overlay-1 leaf# ---- no output ----
No prefix-pcTag for a pure compute LEAF

note: before 3.2 release, use
Hardware table vsh_lc –c ‘show system internal aclqos prefix’
(2nd gen or later LEAF –EX, -FX, -FX2 etc.)
leaf# vsh_lc -c 'show system internal eltmc info vrf TK:VRF1' | grep hw_vrf_idx
vrf_id: 8 ::: hw_vrf_idx: 4614 check hw_vrf index pcTag (CLSS) is 1 to
bypass contract
leaf# vsh_lc -c 'show platform internal hal l3 routes’ | egrep ‘VRF|52.52.52’
| VRF | Prefix/Len| RT| RID | LID | Type| PID | FPID/| HIT |N| NB-ID | NB Hw | PID | FPID/| TBI |TRO|Ifindex|CLSS|CLP| AI |SH|DH| Flags |
| 4614| 52.52.52.0/ 24| UC| 1a2| 117| TCAM| 1823| 0| 1823|E| 1a| 12| NA| NA| NA| NA| 0| 1| 3| 0| 0| 0|
Check contract on LEAF (zoning-rule) Zoning-rule on compute

LEAF in egress mode BRKACI-2642
leaf# show zoning-rule scope 2097152 | egrep 'Rule|16391'

(not used in this case)
4142 49162 16391 5 enabled 2097152 permit© 2020 fully_qual(7)
Cisco and/or its affiliates. All rights reserved. Cisco Public 86
L3Out Contract
Contract is
EPG -> L3Out applied on Egress
LEAF
CLI Verification
EPG L3Out EPG
Egress border LEAF Verification for pcTag 49162 pcTag 16391
Egress Policy Enforcement
Points to the
correct external
next-hop IP Routing Table Prefix-pcTag mapping table
Bleaf# show ip route 52.52.52.0/24 vrf TK:VRF1 Bleaf# show vrf TK:VRF1 detail extended | grep vxlan
*via 15.2.2.1, vlan37, [110/5], 00:25:42, ospf-default, intra
Bleaf# vsh -c 'show system internal policy-mgr prefix' | grep 52.52.52
Vrf-Vni VRF-Id ~snip~ VRF-Name Addr Class Shared Remote
Complete
2097152 8 ~snip~ TK:VRF1 52.52.52.0/24 16391 False False False
Hardware table prefix-pcTag mapping for L3Out EPG

(2nd gen or later LEAF –EX, -FX, -FX2 etc.) note: before 3.2 release, use
vsh_lc –c ‘show system internal aclqos prefix’
Bleaf# vsh_lc -c 'show system internal eltmc info vrf TK:VRF1' | grep hw_vrf_idx
vrf_id: 8 ::: hw_vrf_idx: 4614 check hw_vrf index pcTag (CLSS) is
16391 (0x4007)
Bleaf# vsh_lc -c 'show platform internal hal l3 routes’ | egrep ‘VRF|52.52.52’
| 4614| 52.52.52.0/ 24| UC| 121| 68| TCAM| 80f| 0| 80f|A| 7afd| 80df| NA| NA| NA| NA| 0|4007| 2| 0| 1| 0|spi,dpi
Check contract on LEAF (zoning-rule)

Zoning-rule on border BRKACI-2642
LEAF in egress mode
Bleaf# show zoning-rule scope 2097152 | egrep 'Rule|16391'
L3Out Contract Contract is applied on
Ingress LEAF
EPG -> L3Out
CLI Verification
EPG L3Out EPG
Ingress Compute LEAF Verification pcTag 49162 pcTag 16391
for
Points to the Ingress Policy Enforcement
correct border leaf
TEP Prefix-pcTag mapping table
Routing Table
leaf# show ip route 52.52.52.0/24 vrf TK:VRF1 leaf# show vrf TK:VRF1 detail extended | grep vxlan
*via 11.0.64.64%overlay-1, [200/5], 00:00:14, bgp-65003, internal,
tag 65003 leaf# vsh -c 'show system internal policy-mgr prefix' | grep 52.52.52
recursive next hop: 11.0.64.64/32%overlay-1 Vrf-Vni VRF-Id ~snip~ VRF-Name Addr Class Shared Remote
Complete
2097152 8 ~snip~ TK:VRF1 52.52.52.0/24 16391 False True False

leaf# vsh_lc -c 'show system internal eltmc info vrf TK:VRF1' | grep hw_vrf_idx
16391 (0x4007)
leaf# vsh_lc -c 'show platform internal hal l3 routes’ | egrep ‘VRF|52.52.52’
| 4614| 52.52.52.0/ 24| UC| 1a2| 117| TCAM| 1823| 0| 1823|A| 7c82| 830a| NA| NA| NA| NA| 0|4007| 2| 0| 0| 0|spi,dpi

Zoning-rule on compute BRKACI-2642
LEAF in egress mode
leaf# show zoning-rule scope 2097152 | egrep 'Rule|16391'
L3Out Contract EPG -> L3Out
Contract is applied on
Egress LEAF
CLI Verification
EPG L3Out EPG
Egress border LEAF Verification for pcTag 49162 pcTag 16391
Ingress Policy Enforcement
Points to the
correct external
next-hop IP Routing Table Prefix-pcTag mapping table
Bleaf# show ip route 52.52.52.0/24 vrf TK:VRF1 Bleaf# show vrf TK:VRF1 detail extended | grep vxlan
*via 15.2.2.1, vlan37, [110/5], 00:25:42, ospf-default, intra
Bleaf# vsh -c 'show system internal policy-mgr prefix' | grep 52.52.52
Vrf-Vni VRF-Id ~snip~ VRF-Name Addr Class Shared Remote
Complete
2097152 8 ~snip~ TK:VRF1 52.52.52.0/24 16391 False False False

Bleaf# vsh_lc -c 'show system internal eltmc info vrf TK:VRF1' | grep hw_vrf_idx
16391 (0x4007)
Bleaf# vsh_lc -c 'show platform internal hal l3 routes’ | egrep ‘VRF|52.52.52’
| 4614| 52.52.52.0/ 24| UC| 121| 68| TCAM| 80f| 0| 80f|A| 7afd| 80df| NA| NA| NA| NA| 0|4007| 2| 0| 1| 0|spi,dpi

No zoning-rule on border BRKACI-2642
LEAF in ingress mode
Bleaf# show zoning-rule scope 2097152 | egrep 'Rule|16391'
--- none --- © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 89
MP-BGP Deep Dive
MP-BGP MP-BGP is automatically deployed once Route
Reflector (and MP-BGP AS) is configured
VRF overlay-1 MP-BGP
MP-BGP table (vpnv4)
Non-BLEAF Border LEAF
VRF BGP table (IPv4) VRF BGP table (IPv4)
Redistribute
permit-all
Route-maps
L3Out export
RIB
Redistribute export
Protocol Database
permit-all
External Router
MP-BGP with L3Out BGP MP-BGP is automatically deployed once Route
Reflector (and MP-BGP AS) is configured
VRF overlay-1 MP-BGP
MP-BGP table (vpnv4)
Non-BLEAF Border LEAF
VRF BGP table (IPv4) VRF
BGP table (IPv4)
Route-maps
RIB
L3Out export
(BGP)
export permit-all permit-all
External Router
CLI Verifications
1. BGP process in your VRF with expected Redistribution and Route-Target
border-leaf# show bgp process vrf TK:VRF1 Automatically created regardless of routing
VRF RD : 10.0.184.64:2 protocol used in L3Out. If not, check Route
Reflector policy on APIC
Information for address family IPv4 Unicast in VRF TK:VRF1
Redistribution
direct, route-map permit-all • VRF RD (Route Distinguisher) is based on TEP IP
static, route-map imp-ctx-bgp-st-interleak-2097152
eigrp, route-map permit-all • BGP redistributes (almost) all external routes to
ospf, route-map permit-all
Export RT list: export them into MP-BGP vpnv4 by default.
65003:2097152 Check a later page for the exception on BD subnets
Import RT list: (direct routes).
65003:2097152
Information for address family IPv6 Unicast in VRF TK:VRF1 • RT (Route Target) is based on ACI BGP AS and
--- snip --- VRF VNID.
2. External routes are redistributed/exported into VPNv4 in VRF overlay-1

border-leaf# show bgp vpnv4 unicast vrf overlay-1
MP-BGP VPNv4 table can be checked via
Network Next Hop Metric LocPrf Weight Path normal CLI in vrf overlay-1
Route Distinguisher: 10.0.184.64:2 (VRF TK:VRF1) NOTE:

* i5.5.5.0/24 10.0.184.67 5 100 0 ?
This example shows two routes are learned
*>r 0.0.0.0 5 100 32768 ?
* i15.0.0.0/24 10.0.184.67 0 100 0 ? locally (r with next-hop 0.0.0.0) and also from
*>r 0.0.0.0 0 100 32768 ? another leaf with TEP 10.0.184.67.
CLI Verifications
3. MP-BGP on all leaves should have all external routes in VPNv4 format in VRF overlay-1
non-border-leaf# show bgp vpnv4 unicast vrf overlay-1
5.5.5.0/24 is advertised from border-leaf1
Route Distinguisher: 10.0.184.64:2 (10.0.184.64) and border-leaf2
*>i5.5.5.0/24 10.0.184.64 5 100 0 ? (10.0.184.67)
* i 10.0.184.64 5 100 0 ?
Route Distinguisher: 10.0.184.67:1 Two entries with the same next-hop LEAF TEP
*>i5.5.5.0/24 10.0.184.67 5 100 0 ? means there are two Route Reflectors.
* i 10.0.184.67 5 100 0 ?
non-border-leaf# show bgp vpnv4 unicast 5.5.5.0/24 vrf overlay-1

Route Distinguisher: 10.0.184.67:1
BGP routing table entry for 5.5.5.0/24, version 598 dest ptr 0xaa7e840c Each VPNv4 route in VRF overlay-1 has RT
AS-Path: NONE, path sourced internal to AS with its original VRF VNID
10.0.184.67 (metric 3) from 10.0.184.65 (1.1.1.101)
Extcommunity:
RT:65003:2097152
4. BGP process is running also in your VRF on non-border-leaf for MP-BGP

non-border-leaf# show bgp process vrf TK:VRF1
Information for address family IPv4 Unicast in VRF TK:VRF1 IPv4 BGP imports all the external routes for
Export RT list: its own VRF based on RT from VPNv4 table
65003:2097152
Import RT list:
65003:2097152 If BGP is not running in your VRF on non-border-
Information for address family IPv6 Unicast in VRF TK:VRF1 leaf, check Route Reflector Policy config on APIC
--- snip ---
CLI Verifications
5. The external routes are imported in IPv4 BGP table based on RT
non-border-leaf# show bgp ipv4 unicast vrf TK:VRF1
Network Next Hop Metric LocPrf Weight Path

*>i5.5.5.0/24 10.0.184.64 5 100 0 ?
*|i 10.0.184.67 5 100 0 ?
6. The routing table shows border leaves as next-hop learned from iBGP
non-border-leaf# show ip route 5.5.5.0/24 vrf TK:VRF1
5.5.5.0/24, ubest/mbest: 2/0
recursive next hop: 10.0.184.67/32%overlay-1
recursive next hop: 10.0.184.64/32%overlay-1
CLI Verifications for BD subnet exception
border-leaf# show bgp vpnv4 unicast neighbors vrf overlay-1
BGP neighbor is 10.0.184.65, remote AS 65003, ibgp link, Peer index 1
For address family: VPNv4 Unicast

Outbound route-map configured is deny-pervasive, handle obtained
BD subnets and Null0 I/F should not be

BGP neighbor is 10.0.184.66, remote AS 65003, ibgp link, Peer index 2
distributed via MP-BGP.
For address family: VPNv4 Unicast ➢ Outbound route-map to BGP Route
Outbound route-map configured is deny-pervasive, handle obtained Reflector (spines) limits BD subnets
(pervasive routes) and Null0.
border-leaf# show route-map deny-pervasive
route-map deny-pervasive, deny, sequence 1 In this example, 10.0.184.65 and 10.0.184.66
Match clauses: are RR spines.
pervasive: 2
Set clauses:
route-map deny-pervasive, deny, sequence 2
BD subnets are deployed based on object
Match clauses: policies from APIC instead of routing protocol
interface: Null0
Set clauses:
route-map deny-pervasive, permit, sequence 3
Match clauses:
Set clauses:
Reference
ACI Fabric L3Out Guide -

https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/guide-c07-743150.html
Complete your
online session
survey • Please complete your session survey
after each session. Your feedback
is very important.
• Complete a minimum of 4 session
surveys and the Overall Conference
survey (starting on Thursday) to
receive your Cisco Live t-shirt.
• All surveys can be taken in the Cisco Events
Mobile App or by logging in to the Content
Catalog on ciscolive.com/emea.
Cisco Live sessions will be available for viewing on

demand after the event at ciscolive.com.
Continue your education
Demos in the
Walk-in labs
Cisco campus
Meet the engineer

Related sessions
1:1 meetings
Thank you

Brkaci 2642

Загружено:

Сведения о документе

Оригинальное название

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

Brkaci 2642

Загружено:

Авторское право:

Доступные форматы

ACI Troubleshooting

Layer 3 Out (L3Out)

Takuya Kishida – Technical Marketing, DCBU ACI

• L3Out Key Components

• Distributing routes within ACI (MP-BGP)

• Advertising ACI subnet

• Contract on L3Out (prefix based EPG)

• L3Out Subnet scope options

• Export Route Control Subnet example

VRF • What is BD Subnet for ?

MAC A1 MAC A2 MAC A3 A network device (ex. router,

VRF • What is BD Subnet for ?

A network device as an endpoint?

IP X.X.X.X/8 ※ One endpoint can have up to 1024 IPs in ACI

VRF • How is L3Out different from EPG?

Next-hop MAC in endpoint table

VRF1 VRF1 3. Advertise internal routes (BD subnet)

2. Are OSPF parameters matching with neighbors?

border-leaf# show vlan id 92 extended If anything is not as

2. Are EIGRP parameters matching with neighbors?

3. Are BGP neighbors established correctly? Is it receiving BGP routes?

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd

BGP Route Reflector Policy 5 4 3 2

• ACI BGP AS number EPG L3Out

3. Apply Pod Policy Group

leaf# acidiag fnvread | grep spine

non-border-leaf# acidiag fnvread

No BD Subnet A on BLEAF yet

border-leaf# show ip route vrf TK:VRF1

3. Does the route-map have expected BD subnet?

border-leaf# show ip bgp neighbors vrf TK:VRF1 | egrep '^BGP|Out'

3. Does the BGP outbound route-map have the expected BD subnet?

Should NOT be able

How do we accomplish this ??

10.0.0.0/8 L3Out EPG A

Prefix Based EPG (= L3Out EPG)

Node Profile L3Out BD

Networks (L3Out EPG) EPG (Security Group)

Traffic from LEAF front panel port

VRF1 – 10.0.0.0/8 => pcTag 49158

leaf# show vrf TK:VRF1 detail extended | grep vxlan

leaf# vsh_lc -c 'show system internal aclqos prefix' | egrep 'Vrf|10.0.0.0'

=== use this command from 3.2 ===

A common mistake is selecting both

2. Check VRF VNID pcTag/contract is per VRF

If your source/destination is an endpoint, it should

leaf# show zoning-filter filter 5

Both 10.0.0.0/8 and 20.0.0.0/8 can talk to EPG X

L3Out Contract 192.168.1.1 EPG X

Common Issue (L3Out EPGs with 0.0.0.0/0) EPG B 20.0.0.0/8

1. Check VRF VNID

This contract is due to “EPG X <-> L3Out A”

Traffic Classification for Contract

Route Control for Routing Protocol Subnet Classification

L3Out L3Out VRF1 VRF2 L3Out

Aggregate Export Aggregate Import Aggregate Shared Route

L3Out L3Out VRF1 VRF2

ACI Border LEAF

User VRF Route Import Route Export

User VRF Route Import Route Export

Export Route Control User

(OSPF) Another Border LEAF

ACI Border LEAF