Академический Документы
Профессиональный Документы
Культура Документы
2.5 1st July 2015 John Support portal login credentials added
2.7 25th Nov 2015 John Adding bullets to the O365 design
3.0 4th April 2016 John New site addition (Ashburn, SJ)
Distribution
GIMEC, LOB, SE, SA, SD, IT, Service Planning and Realization
Table of Contents
1. Introduction .................................................................................................................................... 5
D. IP addressing................................................................................................................................. 11
H. Multicast ....................................................................................................................................... 12
L. Troubleshooting ............................................................................................................................ 19
v. DESIGN .................................................................................................................................. 22
P. References .................................................................................................................................... 29
Q. Annexures..................................................................................................................................... 29
• This model is based on Layer-3 VPN NNI. In this model the traffic will flow over the GVPN
Backbone.
• The IZO Private network will connect to NGE network on an ENNI connection to the nearest
NGE equipment and the GVPN ASBR will connect to the nearest NGE equipment on the ENNI
connection. NGE network will provision an EPL service between the two ports.
• These VLANs will be pre-provisioned or transparently passed and an aggregate bandwidth
will be allocated for all these vlans at the NGE network.
• Layer-3 BGP session will be established between the GVPN ASBR and the Cloud provider’s
Router during customer provisioning. The IP Addressing will be mutually agreed between the
cloud provider and TCL during the customer turn-up.
• PE-CE BGP sessions will be established between the Enterprise Customer’s CPE and the
GVPN PE router.
Customer will order a circuit to connect to Microsoft Azure Services through connectivity
partner TATA Communications. A “circuit” represents a redundant pair of connections
between the customer and Windows Azure configured in Active-backup configuration.
Azure services are categorized as Private, Public to represent the IP addressing schemes.
o Windows Azure compute services, namely virtual machines (IaaS) and cloud services
(PaaS) deployed within a virtual network are considered Private.
o Services such as Windows Azure Storage, SQL databases and Websites (to name a
few) are considered Public.
o Services such as O365, Exchange Online, SharePoint Online, Skype for Business, and
CRM Online are Microsoft Online Services or O365 services. Please refer section
(N. Microsoft/O365 Services) for details
A sub-interface will be created for each service on both the interconnects i.e. 3 sub-interface
per interconnect per customer if a customer subscribes to all 3 services.
For Private service requirement the Azure facing interface can be placed under the existing
VPN as it is extension his GVPN
For Public and O365 create a new VRF/VPN as these are hosted on Public IIP’s and NAT is
performed at the CPE.
Public and O365 interface can be placed under the same VPN as both are on Public IP’s
BGP sessions will be setup across these 6 sub-interfaces so we can have redundancy and
routes exchanged.
A remote site in this VPN can now reach the Azure Public/ Private/ O365 services via TCL IZO
Private.
TCL will use Q-in-Q VLAN tagging to hand off customers to Windows Azure. Microsoft will
provide the carrier with the port names for the two ports and an STAG to uniquely identify a
customer’s circuit. S-Tag will be unique across the physical link and unique per customer.
Customer circuit may have 3 pairs of routing session enabled (one pair for public services,
one pair for private services and one pair for O365 services).
o Sub-interface for Private services is identified by a unique CTAG 113
o Sub-interface for public services is identified by a unique CTAG 114
o Sub-interface for O365 services is identified by CTAG 115
Public and O365 interface can be placed under the same VPN as both are on Public IP’s
The VLAN IDs are selected by the carrier and passed on to Windows Azure for configuring
routing.
The 3 C-Tags used will be same for all customers and will be agreed upon by Microsoft and
TCL.
Customers are required to have a pairs of BGP for every service in order to have a highly
redundant service.
Local Preference and AS-PATH bgp attributes will be used to make one of the sessions as
primary and the other as a backup.
Microsoft will use AS 12076 and TCL will use AS 4755
Microsoft has set prefix limits for every peer, bgp session will be dropped if this limit
exceeds
o 4000 prefixes per BGP session for a Private peer
o 200 prefixes per BGP session for a Public peer
o 200 prefixes per BGP session for a O365 peer
TCL will set a prefix limit of 2000 on every BGP session for all service types
D. IP addressing
TCL must allocate a /30 CIDR block for assigning IP addresses to interconnect interfaces, these /30 IP
address block must not conflict with the range reserved by the customer for Azure VNET/ on-
premises LAN.
Private Services:
TCL will allocate IP’s for the Cloud facing interface i.e. 2*/30 Private IP
TCL will allocate IP’s for the Customer facing interface i.e. atleast 1*/30 Private IP
TCL will allocate IP’s for CPE management i.e. 1*/32 Private IP
Public Services:
TCL will allocate IP’s for the Cloud facing interface i.e. 2*/30 Private IP
TCL will allocate IP’s for the Customer facing interface i.e. atleast 1*/30 Private IP
TCL will allocate IP’s for CPE management i.e. 1*/32 Private IP
TCL will allocate Public non-routable IP’s for NAT i.e. 1*/32 Public non-routable IP for user
traffic
Customer can also bring his own Public IP’s, in this case TCL will not allocate Public IP’s.
O365 Services:
TCL will allocate Public IP’s for the Cloud facing interface i.e. 2*/30 Public non-routable IP
TCL will allocate IP’s for the Customer facing interface i.e. atleast 1*/30 Private IP
TCL will allocate IP’s for CPE management i.e. 1*/32 Private IP
TCL will allocate Public non-routable IP’s for NAT i.e. 1*/32 Public non-routable IP for user
traffic
Additional Public non-routable IP’s for Active Directory or any other Application which
requires static NAT will be allocated by TCL on request.
Customer can also bring his own Public IP’s, in this case TCL will not allocate Public IP’s.
E. VPN Topologies
The IZO Private NNI will be very similar to an Option-A NNI. And any VPN topology (Full Mesh, Hub
and Spoke, Partial Mesh and Extranet) can be supported over the NNI.
F. Quality of Service
Microsoft supports QOS only for Microsoft Online/ O365 services.
TCL offers a single class QOS for Public and Private services customers, wherein customers
can opt to place all their traffic on any one COS queue
For Microsoft Online/ O365 TCL will support multi-queue COS as for any unmanaged
customer. Refer Section (N. Microsoft Online/ O365 Services) for details
G. CPE Management
Since there will be no CPE at the cloud providers, there will be no CPE Management component for
the Cloud facing site, however remaining sites of the VPN can be managed.
H. Multicast
Multicast is not supported by Microsoft
1. Customer after signing into his Azure Portal requests a circuit and creates a ServiceKey.
Customer will use a PowerShell cmdlet to make this request.
For example we’ll use TATA as the service provider and will specify a 10 Mbps ExpressRoute circuit
in Hong Kong.
2. Once customer makes a request, it is visible on the Partner Portal under Expressroute tab
3. Click on the correct request to enter the provisioning workflow.
This page gives details of the S tag allocated for this circuit, the location, BW and circuit status.
The S tag allocated here has to be entered into the TCL provisioning systems so that the team
configuring the router interface has the vlans readily available.
NOTE: C vlan is allocated by TCL and has been fixed as 113 for Private Peer and 114 for Public peer
4. Click on start provisioning, enter the configure tab and enter details
Local AS number, IP subnets for the interconnect, C vlans.
NOTE: C vlan is allocated by TCL and has been fixed as 113 for Private Peer and 114 for Public peer
4. Configure the TCL router, Refer the Annexure for Sample configuration
5. BGP comes up and TCL is able to ping the Azure side IP.
Microsoft will start announcing prefixes on the Public peer immediately.
Prefixes on the Private peer will be seen only after customer maps a VNET to this circuit.
6. Come back to provision tab and click finish provisioning, now the state should be
provisioned
7. Customer sees the status of this circuit as Provisioned too
9. MS will start advertising on the Public Peer once the BGP comes up.
Prefixes will be seen on the Private Peer after customer maps the VNET to the circuit
10. Customer Creates a VM and maps it to a Virtual network
You cannot reduce the bandwidth of an ExpressRoute circuit without disruption. Downgrading
bandwidth will require you to de-provision the ExpressRoute circuit, and then re-provision a new
ExpressRoute circuit.
1. Customer sends a request to TCL asking termination. Customer will share the Service-Key
2. Partner logs into the portal finds the circuit for this S-Key and clicks on clear settings for both the
private and public services
3. Then click on Deprovision Circuit Option available on the lower task bar, this will clear the VRF
configured for this customer at Microsoft
5. Once customer deletes the circuit, the request is removed from the partner portal and the S vlan
allocated is released
L. Troubleshooting
Circuit Down:
Connection is the logical circuit assigned to each customer. Here customer link might not be pingable
or BGP down
Interconnect Down:
Interconnect is the physical link connecting Microsoft and TCL, Interconnect going down means all
customer on this link will be down.
1. TCL and Microsoft are connected via an Ethernet backhaul provided by NGE
2. Check TCL port configuration and status.
3. Contact NGE support as we do now for any backbone/access links and check Ethernet link
status
4. Open a case with Microsoft if issue persist.
2. Upon accessing, the first view will be a summary of recent incidents as well as regional contact
information and announcements.
3. To Open a New Incident, Click on the “Submit New Incident” button in the middle of the screen.
Fill the fields as in the below snapshot.
Select sub-topic (Availability, Deployment, Development, Other, Partner Portal, Runtime) based on
the problem and clink NEXT
4. Select severity and describe the problem
Severity Options:
i. Severity C incidents have a response time of 4 business hours
ii. Severity B incidents have a response time of 2 business hours.
iii. Severity A incidents have a response time of 1 business hour and must be submitted over the
phone by referring to TCL Access ID : 001793888
iv. PRE-REQUISITES
A valid and active Microsoft Azure account. An Azure subscription is a requirement even if
connectivity is limited to non-Azure Microsoft cloud services, such as Office 365 services.
An active Office 365 subscription.
Customer must have an existing business relationship with TCL ie customer should have an
existing GVPN connection.
Internet connection for DNS resolution and CDN acceleration.
Traffic destined to O365 services must be from a valid public IPv4 addresses.
v. DESIGN
The Following diagrams explain the Control plane and Data plane setup for typical Azure customer
New sub-interface will be created on the MS interconnect, towards customer new links or
new sub-interface on existing last-mile will be created.
Customer requires Internet for DNS and Akamai (MS uses Akamai for application
acceleration)
A new VRF/VPN for O365/Public service to be created. (Existing VPN can also be used if the
pre-requisites are met)
IZO private CPE at the customer location will connect to this new VPN with the O365
ExpressRoute circuit at the other side.
MS will announce global O365 and regional Public service prefixes over this circuit.
So the customer will now be preferring the more specific prefixes to a default route from
internet or if he has an Internet link with full routes then he has to prefer the IZO
connection.
User traffic will be PAT’ed at the CPE
Some applications in Azure will not work with PAT so a Static NAT has to be configured for
such applications eg Active Directory.
Traffic to O365 will get NAT’ed on the CPE and flow via the O365 VPN to MS
If customer requires a Public Services connectivity then the new circuit can also be placed in
the same O365 VRF as both are on Public IP’s
Traffic to Internet will take the default route to Internet.
VLAN
S-vlan is allocated by Microsoft for every service key generated by the customer.
C-vlan for O365 services is fixed as 115.
IP Addressing
TCL will allocate IP’s for the Cloud facing interface i.e. 2*/30 Public non-routable IP
TCL will allocate IP’s for the Customer facing interface i.e. atleast 1*/30 Private IP
TCL will allocate IP’s for CPE management i.e. 1*/32 Private IP
TCL will allocate Public non-routable IP’s for NAT i.e. 1*/32 Public non-routable IP for user
traffic
Additional Public non-routable IP’s for Active Directory or any other Application which
requires static NAT will be allocated by TCL on request.
Customer can also bring his own Public IP’s, in this case TCL will not allocate Public IP’s.
QOS
The following table provides a list of DSCP markings used by Microsoft for Skype for Business
Application.
Microsoft when sending traffic to TATA will DSCP tag packets based on the workloads.
For traffic from Customer to Microsoft, customer has to specify port ranges for each of the
workloads and create QOS policies on the client computer. Refer the link
“ https://technet.microsoft.com/en-gb/library/jj205371.aspx ” on how customer can set
Qos.
Customer can they subscribe for QOS from TATA and select a profile from the 6COS model.
Routing exchange will be over eBGP protocol. EBGP sessions are established between the
MSEEs and TCL routers
Microsoft will use AS 12076 and TCL 4755
EBGP will be configured to achieve active/backup using bgp LP and AS-PATH prepend
MS will accept 200 prefixes per BGP session.
Default route advertisement over Public/ O365 peering is not permitted
Once BGP is established MS will announce all the regional O365 routes.
Microsoft will tag prefixes advertised through public peering and Microsoft peering with
appropriate BGP community values indicating the region the prefixes are hosted in. This will
be available only for customers subscribing to ExpressRoute Premium.
MS has not enabled bgp community tagging till now. Below are the details of the regional
communities expected from MS:
In addition to the above, Microsoft will also tag prefixes based on the service they belong to.
This applies only to the Microsoft peering. MS has not enabled bgp community tagging till
now. The table below provides a mapping of service to BGP community value expected from
MS:
2. Customer will place an order with cloud type as O365 and service key details.
3. Partner to login to the Partner-portal and locate the service key under the ExpressRoute tab
4. Click on start provisioning, enter the configure tab and enter details
Peer AS number, IP subnets for the interconnect, C vlans.
5. TCL should also enter the prefixes that need to be announced to Azure via the O365 peer.
Microsoft will validate the owner of the Public against the regional routing internet registry (RIR) or
an internet routing registry (IRR)
7. BGP comes up immediately and TCL will start receiving regional O365 prefixes
o Increased route limits for Azure private peering from 4,000 routes to 10,000 routes.
o Global connectivity for services. An ExpressRoute circuit created in any region will have
access to resources across any other region in the world. For example, a virtual network
created in West Europe can be accessed through an ExpressRoute circuit provisioned in
India.
o Increased number of VNet links per ExpressRoute circuit from 10 to a larger limit (20-
100), depending on the bandwidth of the circuit.
P. References
https://azure.microsoft.com/en-gb/documentation/services/expressroute/
Q. Annexures
i. Template for IP validation
Content:
Hi Team,
Please validate IP's for the Microsoft Peer, details below:
Below are the Interface configs for Cisco and Alcatel Lucent routers.
VLAN 555 is S-TAG
VLAN 113 is C-TAG
For Alcatel Lucent Primary router:
port 2/2/9
ethernet
mode access
encap-type qinq
no autonegotiate
exit
no shutdown
bgp
group "AZURE"
family ipv4
as-override
type external
import "IMPORT_AZURE"
export "PE-CE_EXPORT"
local-as 4755
peer-as 12076
disable-communities extended
prefix-limit 2000
neighbor x.x.x.x
exit
exit
policy-statement "IMPORT_AZURE"
entry 10
action accept
local-preference 200
Alcatel Secondary Router
port 2/2/9
ethernet
mode access
encap-type qinq
no autonegotiate
exit
no shutdown
bgp
group "AZURE"
family ipv4
as-override
type external
import "PE-CE_IMPORT"
export "EXPORT_AZURE"
local-as 4755
peer-as 12076
disable-communities extended
prefix-limit 2000
neighbor x.x.x.x
exit
exit
policy-statement "EXPORT_AZURE"
entry 10
action accept
as-path-prepend 4755 1
For Cisco router
Primary router
ip vrf AZURE
rd 4755:
route-target export 4755:
route-target import 4755:
interface GigabitEthernet9/0/2.555113
description Microsoft Cloud Connect @ Hong Kong - Microsoft Private Services POC
encapsulation dot1Q 555 second-dot1q 113
ip vrf forwarding AZURE
ip address 172.30.100.9 255.255.255.252
no ip redirects
no ip proxy-arp
service-policy input Policy
service-policy output Policy
!
address-family ipv4 vrf AZURE
no synchronization
redistribute connected
neighbor 172.30.100.10 remote-as 12076
neighbor 172.30.100.10 activate
neighbor 172.30.100.10 as-override
neighbor 172.30.100.10 route-map AZURE-IN in
neighbor 172.30.100.10 maximum-prefix 2000
exit-address-family
ip vrf AZURE
rd 4755:
route-target export 4755:
route-target import 4755:
interface GigabitEthernet3/0/1.555113
description Microsoft - Private Service POC
encapsulation dot1Q 555 second-dot1q 113
ip vrf forwarding AZURE
ip address 172.30.100.13 255.255.255.252
no ip redirects
no ip proxy-arp
shutdown
service-policy input Policy
service-policy output Policy
!
address-family ipv4 vrf AZURE
no synchronization
redistribute connected
neighbor 172.30.100.14 remote-as 12076
neighbor 172.30.100.14 activate
neighbor 172.30.100.14 as-override
neighbor 172.30.100.14 route-map AZURE-OUT out
neighbor 172.30.100.14 maximum-prefix 2000
exit-address-family
!
end