BASIC TROUBLE SHOOTING OF VPN & LEASE LINE INDEX

Sr. No 1 2 3 4 5 6 7 8 TOPIC PVN Link DOWN PVN Link FLUCTUATION VPN Link PACKET LOSS VPN LATENCY ISSUE BASIC CISCO / JUNIPER COMMAND Lease Line Link Down Lease Line FLAPPING / ERRORS / RELIABLITY BASIC TRANSPORT COMMAND ( NORTEL ) Page No 2-4 5 6-7 8-10 11 12-13 14 15

LINK DOWN CASE

E1-Interface 1) Check the interface description DLHICAG1B> show interfaces e1-0/1/0:30 Physical interface: e1-0/1/0:30, Enabled, Physical link is Up Interface index: 184, SNMP ifIndex: 150 Link-level type: PPP, MTU: 1504, SDH mode, Speed: 1984kbps, Loopback: None, FCS: 16, Framing: G704 Device flags : Present Running Interface flags: Point-To-Point SNMP-Traps Internal: 0x4000 Link flags : Keepalives Keepalive settings: Interval 10 seconds, Up-count 1, Down-count 3 Keepalive: Input: 146187 (00:00:10 ago), Output: 146495 (00:00:08 ago) LCP state: Opened NCP state: inet: Opened, inet6: Not-configured, iso: Not-configured, mpls: Notconfigured CHAP state: Closed CoS queues : 4 supported, 4 maximum usable queues Last flapped : 2008-03-19 19:37:22 IST (01:49:00 ago) Input rate : 6728 bps (10 pps) Output rate : 97976 bps (10 pps) DS1 alarms : None SDH alarms : None DS1 defects : None SDH defects : None Logical interface e1-0/1/0:30.0 (Index 689) (SNMP ifIndex 74) Description: TO_NIIT_OMBB200709136658160 Flags: Point-To-Point SNMP-Traps 0x4000 Encapsulation: PPP Bandwidth: 256kbps Protocol inet, MTU: 1500 Flags: None Addresses, Flags: Is-Preferred Is-Primary Destination: 97.184.38.172/30, Local: 97.184.38.174, Broadcast: 97.184.38.175 2) Check the terse response DLHICAG1B> show interfaces e1-0/1/0:30 terse Interface Admin Link Proto Local Remote e1-0/1/0:30 up up

e1-0/1/0:30.0 up up inet 97.184.38.174/30 a. If the physical protocol both up ping the client end router DLHICAG1B> ping 97.184.38.173 source 97.184.38.174 routing-instance NIIT rapid count 100 PING 97.184.38.173 (97.184.38.173): 56 data bytes !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! --- 97.184.38.173 ping statistics --100 packets transmitted, 100 packets received, 0% packet loss round-trip min/avg/max/stddev = 18.558/23.512/68.392/11.769 ms b. If the physical and protocol are both down check loop with last mile NE (network element), also check the alarm at last mile NE. Alarms on last mile may be PPI - PDH Physical Interface , comes from electrical droppings side PPI-AIS PPI Alarm Indication signal (client side) PPI-LOS PPI loss of signal alarm (client side) PPI-TF PPI Transmit Fail (client side) TU-AIS Tributary Unit (network side) TU-PLM Payload level mismatch (network side) TU-RDI -(network side) LP-PLM-lower path payload level mismatch (network side) c. In PPI alarm get the router and converter resetted; also check the power status at Client end. If last mile is UBR or LMDS get the respective LED status For UBR (IDU, IDA, ITA) IDU, ODU, Power, AIR, Service For LMDS (TSB, TS3) INT, EXT, Power 3) Arrange the Er accordingly in respective CMP following the escalation matrix as FA , Er,CMP Lead , Customer Activation Head ,ISP Lead ,FA Lead 4) When Er on site First check the converter status, and its configuration If link doesnt come up isolate the issue of issue of physical connectivity by giving remote loop from juniper / MCN end TX NE in case of CISCO Gigabit Interface / fast Ethernet interface 1) Check the interface description Same as E1/Ds 2) Check the terse response

Same as E1/Ds 3) Ge/Fe port will always show physical and protocol up Same as E1/Ds 4) Ping the link OR 5) Check whether MAC address of client end is getting learnt on juniper. MUMBCAG1B> show arp | match ge-3/2/0.286 00:02:17:fd:48:1a 97.208.15.125 97.208.15.125 ge-3/2/0.286 none 6) Ping the client end IP by assigning to last mile Cisco device.

LINK FLUCTUATION ISSUE

E1/Ds-interface 1) Check fluctuation on juniper DLHICAG1B> show log messages | match e1-0/1/0:30 | last 2) Check alarm logs on last mile. 3) Go for BERT/V.35 testing activity if required 4) If required last mile NE port can be changed Ge /Fe port 1) In this case fluctuation logs cannot be seen on Juniper so check alarm logs on last mile NE 2) Coordinate with concerned team accordingly 3) Last mile may be MEN, TULIP, and HECL

PACKET LOSS ISSUE

1) Check ping response 2) Check utilization a. Check the Bandwidth DLHICAG1B> show configuration interfaces e1-0/1/0:30 e1-options { timeslots 2-32; framing g704; } unit 0 { description TO_NIIT_OMBB200709136658160; bandwidth 256k; family inet { policer { input 256k-policer; output 256k-policer; } address 97.184.38.174/30; } family mpls; } b. Check the utilization DLHICAG1B Seconds: 7 Time: 22:07:32 Delay: 4/0/30 Interface: e1-0/1/0:30, Enabled, Link is Up Encapsulation: PPP, Keepalives, Speed: 1984kbps Traffic statistics: Current delta Input bytes: 2105358423 (400 bps) [0] Output bytes: 13725133617 (376 bps) [0] Input packets: 12335273 (1 pps) [0] Output packets: 13786346 (0 pps) [0] 3) Check MRTG graph a. For juniper 97.253.1.42

b. For CISCO 97.253.106.21 If the traffic is below 85% -E1-interface 1) Get client end resetting done 2) Go for field activity same as above Ge /Fe port 1) Check packet loss by assigning IP to last mile NE 2) For last mile 3) MEN: go for field activity. 4) TULIP/HECL: book FT

LATENCY ISSUE
Latency values for different last mile 1) TX / DLC 2) Wireless (UBR/LMDS) -3) MEN -4) TULIP/HECL If the value exceeds as mentioned above take all the steps as for packet loss. Reach ability issue Suppose client want to reach an IP of location DLHI from MUMB 1) Check WAN link ping status for MUMB MUMBCAG1B> ping 10.234.232.21 source 10.234.232.22 routing-instance ACC_HUB rapid count 100 PING 10.234.232.21 (10.234.232.21): 56 data bytes !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! --- 10.234.232.21 ping statistics --100 packets transmitted, 100 packets received, 0% packet loss round-trip min/avg/max/stddev = 0.697/0.872/1.451/0.148 ms 2) Check the routes for DLHI IP in concern MUMBCAG1B> show route 10.234.236.85 table ACC_HUB ACC_HUB.inet.0: 357 destinations, 653 routes (357 active, 0 hold down, 0 hidden) + = Active Route, - = Last Active, * = Both 10.234.236.84/30 *[BGP/170] 00:08:57, localpref 100, from 220.224.191.77 AS path: I > to 220.224.184.9 via ge-1/0/0.0, Push 101664, Push 721607(top) [BGP/170] 00:08:57, localpref 100, from 220.224.191.81 AS path: I > to 220.224.184.9 via ge-1/0/0.0, Push 101664, Push 721607(top) 3) If not getting routes , inform the client 4) If getting routes , check where this route is hosted

a. MUMBCAG1B> show route 10.234.236.85 table ACC_HUB detail ACC_HUB.inet.0: 357 destinations, 653 routes (357 active, 0 hold down, 0 hidden) 10.234.236.84/30 (2 entries, 1 announced) *BGP Preference: 170/-101 Route Distinguisher: 220.224.181.254:4 Next-hop reference count: 471 Source: 220.224.191.77 | | Announcement bits (2): 0-KRT 2-BGP RT Background AS path: I (Originator) Cluster list: 220.224.191.77 220.224.191.13 AS path: Originator ID: 220.224.181.254 Communities: target:125:326 b. MUMBCAG1B> show host 220.224.181.254 254.181.224.220.IN-ADDR.ARPA-domain name pointer DLHICAG1.RDN.RIC.NET.IN 5) Here it will show DLHI RDN Reliance router. 6) Check the DLHI WAN link ping status . DLHICAG1B> ping 10.234.236.85 source 10.234.236.86 routing-instance ACC_SPOKE rapid count 100 PING 10.234.236.85 (10.234.236.85): 56 data bytes !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! --- 10.234.236.85 ping statistics --100 packets transmitted, 100 packets received, 0% packet loss round-trip min/avg/max/stddev = 62.508/84.320/152.107/19.095 ms 7) Check routes of MUMB WAN ips in DLHI router under proper vrf DLHICAG1B> show route 10.234.236.85 table ACC_SPOKE ACC_SPOKE.inet.0: 180 destinations, 257 routes (180 active, 0 holddown, 0 hidden) + = Active Route, - = Last Active, * = Both 10.234.236.84/30 *[Direct/0] 2w3d 19:21:13 > via ge-3/1/0.140 [BGP/170] 09:01:32, MED 0, localpref 100 AS path: 64828 ?

> to 10.234.236.85 via ge-3/1/0.140 8) Check ping for DLHI PE and CPE from MUMB router MUMBCAG1B> ping 10.234.236.85 source 10.234.232.22 routing-instance ACC_HUB rapid count 100 PING 10.234.236.85 (10.234.236.85): 56 data bytes !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!^C --- 10.234.236.85 ping statistics --77 packets transmitted, 76 packets received, 1% packet loss round-trip min/avg/max/stddev = 81.891/108.318/169.754/22.280 ms 9) Check ping for MUMB PE and CPE from DLHI router DLHICAG1B> ping 10.234.232.21 source 10.234.236.86 routing-instance ACC_SPOKE rapid count 100 PING 10.234.232.21 (10.234.232.21): 56 data bytes !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! --- 10.234.232.21 ping statistics --100 packets transmitted, 100 packets received, 0% packet loss round-trip min/avg/max/stddev = 20.569/28.588/84.679/14.695 ms 10) Case may arise a. Both ping are perfect -- no issue in reliance n/w b. DLHI PE to MUMB PE is not pinging DLHI PE to MUMB CPE not pinging where as DLHI CPE pinging from MUMB PE -- return routes needs to be added at MUMB end pointing towards DLHI c. MUMB PE to DLHI CPE not pinging where as MUMB CPE pinging from DLHI PE -return routes needs to be added at DLHI end pointing towards MUMB

10

BASIC CISCO / JUNIPER COMMAND

Cisco Command show ip interface brief show ip route show ip bgp summary show ip bgp neighbors neigh received-routes show ip ospf interface show ip ospf interface show ip route ospf show ip ospf database show version show processes cpu show logging Juniper Command show interface terse show route show bgp summary show route receiveprotocol bgp neigh show route sourcegateway neigh protocol bgp show ospf neighbor show ospf interface show ospf route show ospf database show version, show system uptime Shows whether a neighbor supports the route refresh capability Co-Ordinating Definition displays the status of interfaces configured for IP displays summary information about entries in the routing table displays the status of all Border Gateway Protocol (BGP) connections

shows neighbor ID, Priority, IP, & State if the neighbor router, dead time. shows neighbor id, pri, state, dead time, address and interface display the current state of the routing table display list of information related to the OSPF database for a specific communication server display the system hardware config., software version, and name and source of configuration files and boot images display the state of logging to the syslog

show system process displays utilization statistics show log messages

11

BASIC TROUBLE SHOOTING OF LEASE LINE

Leased Lines: Definition: Leased line is a point to point link of dedicated bandwidth between two different locations. For eg, it can be between two locations such as Mumbai to Delhi or it can be between Bang to Bang , etc. In this Service, the customer is given a fixed Bandwidth. Leased which are carried internationally to locations such as UK are referred as International Leased lines. Basic Diagnosis: Check at Last Mile(LM) nodes on both ends the alarms present. If no alarms are present, then check the alarm logs. Always confirm with the customer the link status of the circuit. If link is down, then only thru online loop testing with customer we are able to check the path. Various Equipments Used: Transport Equipments: 16X, 4XE, 01C, 01X, N61, N62, N63,N64, N65, Marconi Equipment. DLC elements:: ANR001 -----RT and ANC001 ------CT Fibcom Elements: F21, F24. Wireless Equipments Used: UBR:: I2D, I2A, IDD , LMDS :: TSB,BSB , TS3 Various Types Of Alarms Observed:: Alarm observed From Customer End:: PPI-AIS, PPI-LOS Transport related Alarm observed:: TU-AIS, LP-SF, TU-RDI , TU-PLM ,RFI, SSF When path alarms are observed, then payload at transport Mux, is 0, 7 or 255. Otherwise it is 2 or 5 depending on mux equipment. Diagnosis of Leased Line Circuits:

12

LINK DOWN
Consider a Circuit as shown below : HCGRGNHUDAY128N61101 E1-1-2-4(j1-k263) <=====> MCNMUMVASIY155N63101 -E1-1-4-17(j1-k162) ::: GRGNMUMBANTLL08 Always Check alarms on both ends. If path alarms are observed , then co-ordinate with Transport Team. Suppose we are getting Ais or Los alarm at Mumbai end, then give a loop from Mumbai to Gurgaon and get it checked by the customer .Once loop is detected at Gurgaon , then arrange a Field engineer at Mumbai end. Similarly if alarm is observed at gurgaon end, then check loop from Gurgaon to Mumbai and once detected, then arrange field engineer at Gurgaon end. If wireless devices are involved, then circuit can be like:: Wireless device<= = =>HCGRGNHUDAY128N61101 E1-1-2-4(j1-k263) <=====> MCNMUMVASIY155N63101 -E1-1-4-17(j1-k162) . In such cases, check alarms at Both ends transport equipments. If path alarms are observed , then co-ordinate with Transport Team. If AIS or LOS is present at gurgaon end, then first check loop Gurgaon to Mumbai and once loop is detected, then co ordinate with wireless team

13

LINK FLAPPING / ERRORS / RELIABLITY ISSUE

In such cases, always check the alarms and alarm logs at both ends. Sometimes due to path alarms fluctuation or due to ais or los alarm fluctuation, customer gets CRC errors, or reliability issue or Link flaps. Consider a Circuit as shown below : HCGRGNHUDAY128N61101 E1-1-2-4(j1-k263) <=====> MCNMUMVASIY155N63101 -E1-1-4-17(j1-k162) ::: GRGNMUMBANTLL08 First check with customer with soft loop testing. Provide loop from one end to another and ask the customer and aks him to observe the loop for around 30 minutes or 1 hr. If in a loop given from Mumbai to Gurgaon detected clean and loop from Mumbai to Mumbai is detected with errors, then arrange field engineer with Ber meter for meter testing at Mumbai end If in a loop given from Gurgaon to Mumbai detected clean and loop from Gurgaon to Gurgaon is detected witrh errors, then arrange field engineer with Ber meter for meter testing at Gurgaon end. If in a loop given from Mumbai to Gurgaon or in a loop given from Gurgaon to Mumbai errors are observed by the customer, then check loop from Mumbai to Mumbai and at the same time from Gurgaon end. If no errors are observed then co ordinate with Transport team to check the path. In some cases of NSA faults, End to End ber testing is done to check the entire path from one point to another by sending field Engineer at both engineer and do end to end testing. Entire Reliance path is covered in such testing.

14

BASIC TRANSPORT COMMAND ( NORTEL )

1. TO CHECK THE CONNECTION ~ C N V C PORT DETAILS 2. TO CHECK ALARM ~ V AA 3. TO CHECK ALARM LOG ~ V AL 4. TO GIVE LOOP ~ D L L PORT DETAILS ------ LOCAL LOOP ~ D L R PORT DETAILS ------- REMOTE LOOP 5. TO SWITCH THE PATH ON PROTECTION ~ M O P A OR B PORT ( A and B ARE THE PATHS ON WHICH YOU NEED TO PUT THE LINK ) ~ M O P S A OR B FROM A OR B ( A and B ARE THE PATHS ON WHICH YOU NEED TO PUT THE LINK OR FROM WHERE )

THANK YOU

15