Customer Related Complaints on day today basis.

1. Packet Drops (ILL/ VPN)

2. Latency (ILL/ VPN)
3. Destination Not reachable (ILL / VPN)
4. Site not opening / slow response.
5. Link down / flapping (ILL/ VPN)
6. L2 MPLS VPN link down.
7. Retail Services.
(a) Packet Drops: ILL
Check for any Packet drops between PE-CE.





Check any packet drops observed due to Service Policy / Rate limiting
Check the geographical location on the IP address. Use "cqcounter.com" URL to check this.


Check source based trace to destination provided.










Start pinging the destination with the source, if packet drops observed follow next step.

Check source based ping response to second last hop in the trace.

If no packet drops observed, this mean there are packet drops only to destination i.e.
destination end issue.

If packet drops observed, check ping response to above hop and repeat the step till the hop
you dont observe any packet drop














If packet drop observed to hop on which we have visibility, then concentrate on that segment and troubleshoot
accordingly.
If packet drops observed to 6453 network IP , escalate the case to 6453.
Meanwhile, ask SOC team to share the source based reverse trace to have more visibility on the issue.
Check the Forward and Reverse path. And Make it symmetric and optimal.

(a) Packet Drops: VPN

Check for any Packet drops between PE-CE.
Check any packet drops observed due to Service Policy / Rate limiting.
Check any drops observed Between Egress PE from Ingress PE.
Check any uplink link is having any drops.
Check is there any link on the core is having issue (Interface Flap, Physical Issue, High Utilizationetc)
Check if all the uplinks and downlinks on the PE is having drops, more probably due to SUP card
issue/Malfunctioning.
Check any errors observed in Line card and Fabric level
Escalate the issue with IPTAC for further analysis with CISCO TAC

(a) Latency : ILL
Check for any latency observed between PE-CE (Last mile, Policy, Chocking)
Check the geographical location of the destination IP address on site "www.cqcounter.com".
Check what Service category customer belongs (PILL, STDILL, SILL, CILL..Etc).




Check source based trace response for the destination.
If destination belongs to US East coast location, forward trace should go via MLV session and if it belongs to
US west coast, trace should go via CXR session.
Check if the latency is under SLA commitment based on customers service category.
If forward trace is going via best path, ask customer for reverse trace.
If forward and reverse traces are asymmetric, try to make them symmetric.
Same step to be followed for domestic destinations.

(b) Latency: VPN
Check for any latency observed between PE-CE (Last mile, Policy, Chocking)
Check the latency commitment based on the Class of service (COS1, COS2, COS3...Etc.)
Check the latency commitments in intercity Core to core location.
For customer and in backbone as well.
Check any uplink or any link in the backbone is having any issue.( Tx. Related/Tx layer switching/ High
Utilization / Load Balancing / Backbone Link down/flap / Any Hardware related..Etc)
Check the traffic is going on the right path, any changes in IGP OSPF default metric, which is causing the
traffic to take the longer path.
Check if forward and reverse traces are symmetric. If not, try to make them symmetric with coordination
with Senior members in team.

(a) Destination Not reachable : ILL
Check the source based trace for internet IP from PE.
Check if the destination IP address is reachable from our Gateway router. ( ICR )
Check the destination IP address is reachable from Route-servers.
To check on route server, pls do telnet route-views.on.bb.telus.com.










If trace is successful from route server, it is confirmed that there is issue in TCL BB/reverse trace.
If trace is not initiating on from particular source, pls check route for source IP from core box.













If route is learning for another PE where this customer is not connected. This means IPs are
conflicting. Take up accordingly.
If route is not available in the routing table, you need to originate the prefix with service community.
Take the issue with our upstream providers.
Check If customer is a BGP customer and Multi-homed with two different ISP,
Ask customer to share forward and revere trace from his LAN.
Check for any Routing issues.
(b) Destination Not reachable (CE-CE).
Check Customer link is up and not flapping. CE-PE dynamic protocol is UP(BGP, OSPF, Eigrp) if any
Ask customer to send source based trace from his CE. And check where it gets drops.
Check MPLS Ping and Trace between PE-PE.


Check whether VRF table has got the routes. Use command sh ip route vrf <vrf name> <IP address>:.
Check customer VRF got corrupted or what.
Check the VRF CEF and forwarding table to make sure label are properly tagged to customer routes.
Use sh ip bgp vpnv4 vrf <vrf name> <IP address>. Pls check RTs attached are correct or not.
Exported RT for route must be imported at B-end. Need to check sh run vrf <vrf name>.







Check if any uplink is having any issue,
Try removing traffic from uplink if the link is coming on OSM module and check.
Check any reverse routing at customer end and inform accordingly.
Link down (ILL / VPN).
Check any configuration issue at our PE router end.
Check if interface config is correct or not, interface is up or down.
Check customer Vlan is passed on the complete ring, in case last mile is on Ethernet.
If SVI interface is there, vlan should be passed in complete ring as below.
Router (Gi1/1)Switch-1(Gi0/1)(Gi0/1)Switch-2(Gi0/2)(Gi0/1)Switch-3(Gi0/2)(Gi2/1)Router
Check the Line card /Module is having any issue and all other links are UP and observing traffic on other
customer links coming on the same slot/module.
Try giving a reset to the ports, if required reset on controller level and check.
Do shut no shut to interface
Clear arp at router
Clear mac table at router and switch
If you are able to see the mac from downlink, still arp is not getting at router. Pls do laptop testing.
If link is up on laptop, ask FTR team to check media. If not, go for port change.
L2 MPLS VPN link down
Cross-Check the L2 configuration.
Check the VC status, Labels and packet sent and received counters.
Use command sh mpls l2transport vc <VC ID> detail.
Check the outgoing interface from PE. For 7600 outgoing interface should be OSM or SIP/SPA
SFP interface doesnt support for L2 VPN traffic
Check the outgoing interface and its MTU.
Check the pseudowire ping End to End VC.
ping mpls pseudowire <x-connect IP> <VC ID>
Check the MAC learning.
A-end mac should be learned t B-end and vice versa
Try recreating the VC, if still problem observed; try to shift uplinks IGP traffic to another interface.
Sometimes, uplink cards might have gone faulty in respective of carrying L2 VPN traffic