Ticket Editado

20/05/2015 07:34:51 CGW
Author:ALANSO BESSMAN
This site is down and not responding to poll please investigate.
INCIDENT CONTACT DETAILS:

---------------------------------------------------Name: 'ALANSO BESSMAN'
Tel No: '1-6517954300-'
Email:
'ALANSO.BESSMAN@ECOLAB.COM'
20/05/2015 07:34:53 CGW
Ticket submitted at 20/05/15 10:34:51 GMT
Service Impact changed to 3 - Medium.
CUSTOMER UPDATE:
'Ticket Created' email sent to: ALANSO.BESSMAN@ECOLAB.COM
emec@ecolab.com
rachel.maskew@bt.com
jason.reed@bt.com
eric.rotunda@bt.com
20/05/2015 07:38:06 ANALISTA 1
Operational Categorisation values Changed to:
Tier 1 - BAU FAILURE
Tier 2 - BAU FAILURE
Tier 3 - UNAVAILABLE/NO RESPONSE
Product Categorisation Values Changed to:
Tier 1 - DEVICE PRODUCT
Tier 2 - ROUTERS
Tier 3 - CISCO 1941
Element changed to na-ecola-car70-dal-am at NA-CARROLTON-TX-1368.
CUSTOMER UPDATE:
'Modify Ticket' email sent to: ALANSO.BESSMAN@ECOLAB.COM
emec@ecolab.com
jason.reed@bt.com
eric.rotunda@bt.com
20/05/2015 07:38:29 ANALISTA 1
Assignee changed tO ANALISTA 1
20/05/2015 07:40:07 ANALISTA 2
Assignee changed to ANALISTA 2
Severity changed to P3.
20/05/2015 09:00:58 ANALISTA 3
A: Inbound call from Ecolab, spoke with Sca. He informed that there is power on
location
A: Upgraded to P1
N: Engage PTT
Severity changed to P1.
Service Impact changed to 1 - Severe.
CUSTOMER UPDATE:
emec@ecolab.com
jason.reed@bt.com
eric.rotunda@bt.com
20/05/2015 11:41:11 ANALISTA 4
20/05/2015 11:52:24 ANALISTA 5
20/05/2015 12:30:29 ANALISTA 5
D - Link for the circuit BCRL0493 is down
T PE14-DAL-AM#show interface Serial2/0/1/25:0
Serial2/0/1/25:0 is down, line protocol is down
Hardware is PMC FREEDM, PMC TEMUX, TDK LIU
Description: Service:IGNITE_LL MPLS,CID:01007554,SR:1398381,CE:na-ecola-car70dal-am,Speed:3072,on:14-05-2014,by:indalkv
MTU 1500 bytes, BW 1536 Kbit, DLY 20000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation FRAME-RELAY, crc 16, loopback not set
Keepalive not supported
Bundle Link of bundle MFR985
Last input 05:27:57, output 05:27:55, output hang never
Last clearing of "show interface" counters 44w5d
Input queue: 0/75/0/7 (size/max/drops/flushes); Total output drops: 2
Interface Serial2/0/1/25:0 queueing strategy: PXF Class-based
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
427279385 packets input, 3839553248 bytes, 0 no buffer
Received 0 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
1143 input errors, 215 CRC, 464 frame, 0 overrun, 0 ignored, 464 abort
531023186 packets output, 1296049121 bytes, 0 underruns
7 output errors, 0 collisions, 19 interface resets
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions alarm present
Timeslot(s) Used: 1-24, subrate: 1536Kb/s, transmit delay is 0 flags
non-inverted data
PE14-DAL-AM#
D - Link is down for for the circuit BCRL0494

Last clearing of "show interface" counters 44w5d
non-inverted data
PE14-DAL-AM#
C - Ticket has been raised with LEVEL 3 for both circuit
N - Follow up with LEVEL 3 for updates
CUSTOMER UPDATE:
emec@ecolab.com
jason.reed@bt.com
eric.rotunda@bt.com
20/05/2015 13:49:48 ANALISTA 5
C - No updates from LEVE 3 yet
111111111111111111111111111111111111111111
C - 1st level escalation since there is no updates from the Level 3 yet
N - Follow up with Level 3 for updates
11111111111111111111111111111111111111111111
CUSTOMER UPDATE:
emec@ecolab.com
jason.reed@bt.com
eric.rotunda@bt.com
20/05/2015 16:05:30 ANALISTA 2
A - Inbound call from customer, spoke to TOM, he want BT to escalate the last mi
le PTT.
C - Tom requested to be updated over phone: +16303052404
A - While checking the PE, interface was looped.

S - Link down
A - Customer escalated BT to 1st level.
N - Escalate PTT for updates.
Description: CHAMELEON-LOOPTOOL locked by user 603394964 @ Wed May 20 19:00:41
2015
Internet address is 10.10.11.29/30
Encapsulation HDLC, crc 16, loopback not set
Keepalive set (10 sec)
Last clearing of "show interface" counters 00:00:05
non-inverted data
PE14-DAL-AM#
Assignee changed to Edion Dias.
CUSTOMER UPDATE:
emec@ecolab.com
jason.reed@bt.com
eric.rotunda@bt.com
20/05/2015 16:52:03 ANALISTA 6
Assignee changed to Felipe Jacob.
20/05/2015 17:30:01 ANALISTA 6
############################################################################
############################################################################
D:
- Circuit is still down/down
A:
- BT-NOC SP checked the PE and the technical logs show the circuit still
DOWN/Down (looped)
- BT-NOC SP also checked the tickets raised with Level3 in the web porta
l and found new info.
C:
- Level3 updates:
----------9251305
----------Carrier Escalation Management is reviewing your case at this tim
e and we have reached out to the local carrier to be informed the last mile carr
ier has this set up for a dispatch. I have requested for them to find out if a d
ispatch is even needed or if this can just be dropped remotely. I will be updati
ng this in the next hour.
***************************************
The local carrier is seeing a loop on this that they can not rel
ease. There is a last mile carrier that is being involved at this time. Level 3
will provide another update in about 1 hour.
***************************************
***************************************
***************************************
----------9251337
----------Opened up a ticket with the local exchange carrier/LEC. They were able to test g
ood to the smart jack but not able to loop end user equipment.
N:
Next:
- BT-NOC SP to follow up with Level3 for updates about both tickets rais
ed with them.
=================================================================
----------------------------------------------------------------T:
- Technical Logs:
-------------------------------------------------------------------------------PE14-DAL-AM - Serial2/0/1/25:0
---------------PE14-DAL-AM#show interface Serial2/0/1/25:0
Serial2/0/1/25:0 is down, line protocol is down (Looped)
Description: CHAMELEON-LOOPTOOL locked by user 606198606 @ Wed May 20 20:21:38
2015
Internet address is 10.10.11.21/30
Encapsulation HDLC, crc 16, loopback not set
Keepalive set (10 sec)

non-inverted data
PE14-DAL-AM#
=================================================================
############################################################################
############################################################################
CUSTOMER UPDATE:
emec@ecolab.com
jason.reed@bt.com
eric.rotunda@bt.com
20/05/2015 17:53:14 ANALISTA 6
############################################################################
############################################################################
A:
- Level 3 escalated to 2 level.
C:
2222222222222222222222222222222222
This ticket has been escalated and is currently at a Level 2 escalation. Request
ed escalation is currently being reviewed.
2222222222222222222222222222222222
############################################################################
############################################################################
CUSTOMER UPDATE:
emec@ecolab.com
jason.reed@bt.com
eric.rotunda@bt.com
20/05/2015 19:06:05 ANALISTA 6
############################################################################
############################################################################
A:
- Bt-NOC contacted Matt from Level 3 and ask him to get a new update for
this ticket ASAP.
He informed that could not reach an available technician to update us and he wil
l call us back in at least 1 hour with new info.
N:
- BT-NOC SP to follow up with Level 3 for updates.
C:
- Level 3 escalated to 3 level.
3333333333333333333333333333333333
This ticket has been escalated and is currently at a Level 3 escalation. Request
ed escalation is currently being reviewed.
3333333333333333333333333333333333
############################################################################
############################################################################
CUSTOMER UPDATE:
emec@ecolab.com
jason.reed@bt.com
eric.rotunda@bt.com
20/05/2015 20:52:54 ANALISTA 6
############################################################################
############################################################################
A:
- BT-NOC tried to contact LCON in order to follow the PTT suggestion and
reboot the modem, but no answer to the phone call.
- BT-NOC contacted Level 3 and schedule a FE dispatch on next business h
ours (9:00 am) to the customer site to get the modem checked.
C:
- Last update in Level 3 ticket:
Circuit status :
The Lec carrier is now good to the smart jack , but cannot loop the customer csu
.
I called customer X , he said the customer power and equipment was verified. I
informed of the Lec testing results. X will contact the end-user to have them r
eset the equipment and check the circuit cabling on their end. Holding the tick
et 24 hours to assist the customer further if needed.
N:
Next:
- BT-NOC SP to follow up with Level 3 for the FE dispatch results.
=================================================================
----------------------------------------------------------------T:
- Technical Logs:
-------------------------------------------------------------------------------PE14-DAL-AM - Serial2/0/1/25:0
---------------PE14-DAL-AM#show interface Serial2/0/1/25:0
Serial2/0/1/25:0 is down, line protocol is down (looped)

non-inverted data
PE14-DAL-AM#
=================================================================
############################################################################
############################################################################
CUSTOMER UPDATE:
emec@ecolab.com
jason.reed@bt.com
eric.rotunda@bt.com
20/05/2015 20:57:44 ANALISTA 6
Ticket Suspended at 20/05/2015 19:30:22. Reason: Awaiting Downtime from Cust. St
atus changed to SUSPEND.
CUSTOMER UPDATE:
emec@ecolab.com
jason.reed@bt.com
eric.rotunda@bt.com
20/05/2015 21:03:06 ANALISTA 6
Assignee changed to DISPATCH.
21/05/2015 10:41:18 ANALISTA 5
Ticket returned from Suspension at 21/05/2015 10:41:12. Status changed to WORK I
N PROGRESS.
Ticket at Escalation Level 6. Customer Update due at 22/05/15 19:41:17 GMT
CUSTOMER UPDATE:
emec@ecolab.com
jason.reed@bt.com
eric.rotunda@bt.com
21/05/2015 10:51:44 ANALISTA 5
A - Called Level 3 and spoke with Donna
C - Donna informed that LEVE 3 was able to loop until the smart jack and custome
r need to reboot the equipment

A - Email has been sent to LCON request to reboot the equipment
N - Follow up mailbox for updates from LCON
Assignee changed to Vitor Oliveira.
CUSTOMER UPDATE:
emec@ecolab.com
jason.reed@bt.com
eric.rotunda@bt.com
21/05/2015 10:52:41 ANALISTA 5
Ticket Suspended at 21/05/2015 10:52:30. Reason: Cust To Supply More Info. Statu
s changed to SUSPEND.
CUSTOMER UPDATE:
emec@ecolab.com
jason.reed@bt.com
eric.rotunda@bt.com
21/05/2015 13:07:47 ANALISTA 7
21/05/2015 14:43:05 ANALISTA 7
New Subcase XF38112689 submitted for NL_AMS_NCCA_IA-MPLS.
21/05/2015 14:44:41 ANALISTA 7
Update from Parent XF38105843 : A - BT NOC received a call from Customer Tom st
ating NO FE on site.
A - BT NOC called PTT in order to have updates, informed that circuit is looped
and needs further investigation.
A - BT NOC was informed that Level 3 will reach last mile for checks because the
y have only on node in this service and have noremotely managemeng for it.
A - BT NOC need that last mile remove loop that probably they have leaved on the
service.
A - BT NOC received a call from LEvel 3 stating no loop placed from their manage
d environment.
A - BT NOC requested FE on site in order to have HARD LOOP test for entire envir
onment validation.
A - BT NOC RAISED a case with NCCA Team in order to health check PE device with
configuration and physical connections.
N - Follow up with NCCA for PE device health checks.
N - Follow up with PTT for updates.
N - KCI Customer ASAP.
C - Dear NCCA Team,
We have service provider stating no loop and no evideces of errors on th
eir managed environment, we would like to ask for your assistance in orde dro ha
ve the PE equipment health checked physically and configuration because we have
interface hard down with a loop placed, Please refer to the below tech logs.
Please let us know in case you have any concerns.
Device details:
***PE INFORMATION***
PE Node : PE14-DAL-AM
PE Port : Serial2/0/1/25:0
PE Port : Serial2/0/1/26:0
Since now, we would like to thank you in advance for your time and atten
tion.
C LCON:
ALANSO BESSMAN
1-6517954300ALANSO.BESSMAN@ECOLAB.COM
emec@ecolab.com
tdassette@nalco.com - TOM
non-inverted data
PE14-DAL-AM#
PE14-DAL-AM#show interface Serial2/0/1/25:0

non-inverted data
PE14-DAL-AM#
C CUSTOMER UPDATE:
emec@ecolab.com
jason.reed@bt.com
eric.rotunda@bt.com
21/05/2015 14:47:16 ANALISTA 7
Ticket returned from Suspension at 21/05/2015 14:47:12. Status changed to WORK I
N PROGRESS.
CUSTOMER UPDATE:
emec@ecolab.com
jason.reed@bt.com
eric.rotunda@bt.com
21/05/2015 14:59:34 RESOLVER 1
D:Controller is UP
PE14-DAL-AM#sh controller T3 2/0/1
Load for five secs: 2%/1%; one minute: 6%; five minutes: 6%
Time source is NTP, 17:53:46.636 GMT Thu May 21 2015
T3 2/0/1 is up. Hardware is C10K Half Height CT3 line card
Applique type is Channelized T3
Controller is in channelized mode
D: Multiple T1's are also UP UP on the controller indicating a issue with the in
dividual T1 on the Providers network
PE14-DAL-AM#sh int desc | i 2/0/1
Se2/0/1/1:0
up
up
Service:IGNITE_LL MPLS,CI
D:01006074,SR:1316074,CE:bi-bkleu-dal1-dal-am,Speed:1024,on:29-04-2013,by:darade
a
Se2/0/1/1:0.101
up
up
D:01006074,SR:1316074,CE:bi-bkleu-dal1-dal-am,Speed:1024,on:29-04-2013,by:darade
a
Se2/0/1/2:0
up
up
D:1142115,SR:1197446,CE:fx-1086900-cs2-us-dal,Speed:,on:05-08-2010,by:shirsar
Se2/0/1/2:0.100
up
up
System:VCATT,CID:1142115,
SiteID:1086900,CE:fx-1086900-cs2-us-dal,VPN:GREY,Speed:32,OrderID:1218509,on:0902-2011,by:mutcht,VER:43_10
Se2/0/1/2:0.101
up
up
SiteID:1086900,CE:fx-1086900-cs2-us-dal,VPN:FE,Speed:232,OrderID:1420165,on:02-0
7-2014,by:payasir,VER:43_10
Se2/0/1/3:0
up
up
Service:IGNITE_LL,Cust:CA
SENEWHOLLA,CID:1144385,SRN:1058567,Status:configured on 16-01-2008 ,by User:KUMA
RIP
Se2/0/1/3:0.101
up
up
Service:IGNITE_LL,Cust:CA
SENEWHOLLA,CID:1144385,SRN:1058567,Status:configured on 16-01-2008 ,by User:KUMA
RIP
Se2/0/1/4:0
up
up
D:1141058,SR:1433346,CE:gr-basf-pho71-dal-am,Speed:1536,on:13-10-2014,by:dholakn
Se2/0/1/4:0.101
up
up
D:1141058,SR:1433346,CE:gr-basf-pho71-dal-am,Speed:1536,on:13-10-2014,by:dholakn
Se2/0/1/5:0
down
down
D:10361,SR:1198766,CE:gr-nestl-den1-dal-am,Speed:1536,on:17-09-2010,by:kocerm2
Se2/0/1/5:0.101
down
down
D:10361,SR:1198766,CE:gr-nestl-den1-dal-am,Speed:1536,on:17-09-2010,by:kocerm2
Se2/0/1/6:0
up
up
D:1144385,SR:1215076,CE:sc-case-spr1-dal-am,Speed:1536,on:29-12-2010,by:shahpa
Se2/0/1/6:0.101
up
up
D:1144385,SR:1215076,CE:sc-case-spr1-dal-am,Speed:1536,on:29-12-2010,by:shahpa
Se2/0/1/7:0
up
up
D:01004347,SR:1252053,CE:na-mars-wac71-dal-am,Speed:6144,on:28-09-2011,by:patwar
a5
Se2/0/1/8:0
up
up
a5
Se2/0/1/9:0
up
up
a5
Se2/0/1/10:0
up
up
a5
Se2/0/1/11:0
down
down
Service:IVS MPLS,ICID:103
61,ISID:S1092549,IAID:1-HEKZ21,Status:configured on 07-07-2011 ,by User:calisoi
Se2/0/1/12:0
down
down
Se2/0/1/13:0
up
up
D:01007989,SR:1435598,CE:na-jabil-el70-dal-am,Speed:1024,on:31-10-2014,by:dholev
Se2/0/1/13:0.101
up
up
D:01007989,SR:1435598,CE:na-jabil-el70-dal-am,Speed:1024,on:31-10-2014,by:dholev
Se2/0/1/14:0
up
up
D:1142857,SR:1431215,CE:ch-clari-mid70-dal-am,Speed:1536,on:12-05-2015,by:kariek
tm
Se2/0/1/14:0.101
up
up
D:1142857,SR:1431215,CE:ch-clari-mid70-dal-am,Speed:1536,on:12-05-2015,by:kariek
tm
Se2/0/1/15:0
down
down
Se2/0/1/16:0
down
down
Se2/0/1/17:0
up
up
D:1137537,SR:1450930,CE:ch-synge-wes70-dal-am,Speed:128,on:27-03-2015,by:dumrev
Se2/0/1/17:0.101
up
up
D:1137537,SR:1450930,CE:ch-synge-wes70-dal-am,Speed:128,on:27-03-2015,by:dumrev
Se2/0/1/18:0
up
up
D:1144147,SR:1268128,CE:na-exide-irv71-dal-am,Speed:256,on:14-03-2012,by:sayyedj
2
Se2/0/1/18:0.101
up
up
D:1144147,SR:1268128,CE:na-exide-irv71-dal-am,Speed:256,on:14-03-2012,by:sayyedj
2
Se2/0/1/21:0
up
up
D:1100200,SR:1265271,CE:be-us350-mce1-dal-am,Speed:768,on:21-01-2012,by:nurseca
Se2/0/1/21:0.101
up
up
D:1100200,SR:1265271,CE:be-us350-mce1-dal-am,Speed:768,on:21-01-2012,by:nurseca
Se2/0/1/22:0
up
up
D:01007490,SR:1381878,CE:al-prysm-hut01-dal-am,Speed:1536,on:30-12-2013,by:orabo
nl
Se2/0/1/22:0.101
up
up
D:01007490,SR:1381878,CE:al-prysm-hut01-dal-am,Speed:1536,on:30-12-2013,by:orabo
nl
Se2/0/1/23:0
up
up
D:1142115,SR:1278282,CE:fx-1110021-ce1-us-dal,Speed:,on:03-04-2012,by:zalarx
Se2/0/1/23:0.100
up
up
SiteID:1110021,CE:fx-1110021-ce1-us-dal,VPN:GREY,Speed:32,OrderID:1278282,on:0304-2012,by:zalarx,VER:54_10
Se2/0/1/23:0.101
up
up
SiteID:1110021,CE:fx-1110021-ce1-us-dal,VPN:FE,Speed:88,OrderID:1278282,on:26-04
-2012,by:enukorg2,VER:56_10
Se2/0/1/24:0
up
up
D:01007554,SR:1395800,CE:na-ecola-dal70-dal-am,Speed:1536,on:19-03-2014,by:nagar
sa
Se2/0/1/24:0.101
up
up
D:01007554,SR:1395800,CE:na-ecola-dal70-dal-am,Speed:1536,on:19-03-2014,by:nagar
sa
Se2/0/1/25:0
down
down
D:01007554,SR:1398381,CE:na-ecola-car70-dal-am,Speed:3072,on:14-05-2014,by:indal
kv
Se2/0/1/26:0
down
down
D:01007554,SR:1398381,CE:na-ecola-car70-dal-am,Speed:3072,on:14-05-2014,by:indal
kv
Se2/0/1/27:0
up
up
D:01003084,SR:1406957,CE:aa-tatac-dal70-dal-am,Speed:768,on:05-09-2014,by:paswan
r
Se2/0/1/27:0.101
up
up
D:01003084,SR:1406957,CE:aa-tatac-dal70-dal-am,Speed:768,on:05-09-2014,by:paswan
r
Se2/0/1/28:0
up
up
D:37646,SR:1412488,CE:br-emr-new70-dal-am,Speed:1536,on:30-06-2014,by:autim2
Se2/0/1/28:0.101
up
up
D:37646,SR:1412488,CE:br-emr-new70-dal-am,Speed:1536,on:30-06-2014,by:autim2
N:Please complete loop testing at PE POP site and prove the fault is not on the
providers network
Incident Ended at 21/05/15 17:59:30 GMT and Customer Informed of Clear at 21/05
/15 17:59:31 GMT. Status Changed to AWAITING CLOSURE.
*** END OF UPDATE
21/05/2015 15:04:38 ANALISTA 9
Internal Use ONLY:
C - Keep following up with the case
21/05/2015 15:12:06 DUTY MGR 1
======================================= << Duty Manager >> ====================
===================
T -
S - Both links down. One of them is showing looped.

A - SD was on the call with Level 3 and performed many tests. Need to have now a
FE on site for a hard loop.
C - According to Level 3, the FE and only be dispatched tomorrow @ 13:00PM. Not
acceptable.
A - I escalated Level 3 to have the dispatch today.
A - Customer and SRM informed on email chain.
N - Follow up with Level 3 for the FE ETA and keep customer informed.
C - Level 3 tickets escalations levels : NOW
9251305 ---> 4th level
9251337 ---> 2nd level
======================================= << Duty Manager >> ====================
===================
CUSTOMER UPDATE:
emec@ecolab.com
jason.reed@bt.com
eric.rotunda@bt.com
21/05/2015 15:41:53 DUTY MGR 1
======================================= << Duty Manager >> ====================
===================
T
S
D
A
N
C
- I Spoke with Terry from Level 3. ( Tech that is working on this case)
- Keep checking updates with Level 3.
- Terry will call their LEC now and try to arrange a best ETA.
======================================= << Duty Manager >> ====================

===================
CUSTOMER UPDATE:
emec@ecolab.com
jason.reed@bt.com
eric.rotunda@bt.com
21/05/2015 17:04:42 DUTY MGR 2
======================================= << Duty Manager >> ====================
===================
A - Engaged Global escalation team.
A - Will escalate it further with Level 3
Dear Global Escalation Management Team,
We need your help in order to proceed with further escalation with Level
3 on the tickets below.

"
"
9251305 ---> 4th level

9251337 ---> 3rd level
Facts :
"
"
Site is isolated over 24 hours now;

Level 3 does not provide consistent updates.
Thank you.
======================================= << Duty Manager >> ====================
===================
CUSTOMER UPDATE:
emec@ecolab.com
jason.reed@bt.com
eric.rotunda@bt.com
21/05/2015 17:09:20 ANALISTA 10
21/05/2015 17:11:10 DUTY MGR 2
======================================= << Duty Manager >> ====================
===================
Silvaeo
A - Called Global escalation team talked with Dean and requested help, he will l
ook into the case and and will asist us.
C - Dean requested us to escalate to the next level and copy them so they can pr
oceed with the case.
======================================= << Duty Manager >> ====================

===================
CUSTOMER UPDATE:
emec@ecolab.com
jason.reed@bt.com
eric.rotunda@bt.com
21/05/2015 17:30:32 DUTY MGR 2
======================================= << Duty Manager >> ====================
===================
Level 3,
Can you please advise what is the action plan to repair the links under
the tickets below, our customer is down over 24 hours now, and we do not have c
onsistence updates on what is going on, please advise us what is the latest upda
tes on the case.

"
"
9251305
9251337
Thank you.
======================================= << Duty Manager >> ====================
===================
CUSTOMER UPDATE:
emec@ecolab.com
jason.reed@bt.com
eric.rotunda@bt.com
21/05/2015 17:31:45 ANALISTA 10
A - Called Level 3 for status, spoke with Terrel, informed they have tech to be
dispatched still today or tomorrow in the morninh to test on the smartjack. I co
ntacted customer, on the phone 1 -972 416 1080, John Garze, but hes not avaialble
, ring no answer.
N - To follow-up with Level 3 for updates, confirmation on the dispatch and info
rm Lcon.
CUSTOMER UPDATE:
emec@ecolab.com
jason.reed@bt.com
eric.rotunda@bt.com
21/05/2015 18:09:47 DUTY MGR 2
A - Called customer Tom and left him a message informing the link was restored.
A - Called Alonso and informed him the link was restored, we will monitor the li
nk for 24 hours and get the RFO.
N - Follow up with Level 3 on the RFO and monitor the link for 24 hours.
C - Customer contact details.
Name: 'ALANSO BESSMAN'
Tel No: '1-6517954300-'
Email:
'ALANSO.BESSMAN@ECOLAB.COM'
KCI
Ecolab team,
I just spoke with Alonso and he confirmed the services were restored, I
will pursue the reason for outage and will advise you once carrier provide such
details.
Thank you.
CUSTOMER UPDATE:
emec@ecolab.com
jason.reed@bt.com
eric.rotunda@bt.com
21/05/2015 18:15:00 ANALISTA 11
A - Received a call fromTerry/Level 3 about the following circuits:
BCRL0493 - Technician went to the site and found a problem with the router and f
ixed it. He was not able to provide us more information. - Circuit is up and run
ning
BCRL0494 - tech went to the site and found AIS alarm coming from CE. He informe
d we need to check the equipment
N - Follow up for updates
CUSTOMER UPDATE:
emec@ecolab.com
jason.reed@bt.com
eric.rotunda@bt.com
21/05/2015 19:19:55 DUTY MGR 2
======================================= << Duty Manager >> ====================
===================
A - Called Level 3 duty manager and they informed the issue is on our router whi
ch I disagree, as the link was up over 45 minutes
I reengaged our Global escalation team, and requested a new router and a case to
be raiase with GCS, X is currently
raising the tickets.
======================================= << Duty Manager >> ====================
===================
CUSTOMER UPDATE:
emec@ecolab.com
jason.reed@bt.com
eric.rotunda@bt.com
21/05/2015 19:54:08 ANALISTA 11
A - I have called Level 3 Terrel, he has the Lcon Name and number ( 214 876 5092
), hes going to send a FE engineer to work on the circuit at the smartjack.
N - To follow-up with Level 3 for updates.
CUSTOMER UPDATE:
emec@ecolab.com
jason.reed@bt.com
eric.rotunda@bt.com
22/05/2015 00:50:29 ANALISTA 12
Assignee changed to ANALISTA 12.

22/05/2015 01:08:49 ANALISTA 12
+++++++++++++++++++++++++++++++++++++++++
D - Link for the circuit BCRL0493 is down
T:
D - Link is down for for the circuit BCRL0494
C:
--- Latest update is that BT Escalation and Service Desk Management has arrange
d with PTT Management for a vendor meet (with the LEC to be onsite) tomorrow mo
rning, ETA by 9am local time.
--- Lcon Name and number ( 214 876 5092 ), hes going to send a FE engineer to wo
rk on the circuit at the smartjack.
N - To follow-up with Level 3 for updates.

++++++++++++++++++++++++++++++++++++++
CUSTOMER UPDATE:
emec@ecolab.com
jason.reed@bt.com
eric.rotunda@bt.com
22/05/2015 01:16:09 DUTY MGR 2
======================================= << Duty Manager >> ====================
===================
Team,
I tried several times which different analysts and I finally managed to
create a ticket with Unisys.
"
Talked with Unisys Analyst Akshay REF# 33334939
======================================= << Duty Manager >> ====================

===================
CUSTOMER UPDATE:
emec@ecolab.com
jason.reed@bt.com
eric.rotunda@bt.com
CUSTOMER UPDATE:
emec@ecolab.com
jason.reed@bt.com
eric.rotunda@bt.com
22/05/2015 05:31:22 DUTY MGR 3
======================= << Duty Manager >> =======================
A - Unisys confirmed part ETA to 6 am local time.
A - Replied to Unisys asking to confirm part will not return as customer confirm
ed 9 am local time for vendor meeting.
N - Get confirmation from Unisys and follow up with vendor meeting with Level 3
and Unisys for circuit troubleshooting at 9 am local time.
======================= << Duty Manager >> =======================
CUSTOMER UPDATE:
emec@ecolab.com
jason.reed@bt.com
eric.rotunda@bt.com
22/05/2015 07:56:28 ANALISTA 7
22/05/2015 10:59:26 ANALISTA 7
A - BT NOC called HM/Unisys in order to get updates, spoke with abishek who stat
ed FE details and ETA.
A - BT NOC called PTT/Level3 spoke Joseph who stated No ETA for FE dispatch avai
lable.
A - BT NOC engaged Duty manager for further assistance.
N - Follow up with Level3 for ETA updates.
C - Level3 FE Details:
PENDING.
C - Unisys FE Details:
Robert Chatter
Phone: +1 2144 2544 61
ETA: 09:00 LT.
C - Level 3 to raise a conference call once on site
Ready-Access Number:
8007471915
Secondary Ready-Access Number:
+1 3032481285
7-Digit Access Code:
4164931

non-inverted data
PE14-DAL-AM#
PE14-DAL-AM#show controller T3 2/0/1
T3 2/0/1 is up. Hardware is C10K Half Height CT3 line card
Applique type is Channelized T3
Controller is in channelized mode
Description: USDALB002-USDALB003-45MB-010(BTGS-WW)
No alarms detected.
MDL transmission is disabled
PE14-DAL-AM#show log | include Serial2/0/1/25:0
May 20 09:15:22.679 GMT: %LINK-3-UPDOWN: Interface Serial2/0/1/25:0, changed sta
te to down
May 20 09:15:23.679 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial2
/0/1/25:0, changed state to down
te to up
/0/1/25:0, changed state to up
te to down
te to up
te to down
te to up
te to down
te to up
te to down
te to up
te to down
te to up
te to down
te to up
te to down
te to up
te to down
May 20 20:05:34.131 GMT: %LINK-5-CHANGED: Interface Serial2/0/1/25:0, changed st
ate to administratively down
te to down
te to up
te to down
te to up
te to down
te to up
te to down
te to up
.May 21 02:21:37.334 GMT: %LINK-3-UPDOWN: Interface Serial2/0/1/25:0, changed st
ate to down
te to up
te to down
te to up
te to down
May 21 20:23:04.761 GMT: %CLEAR-5-COUNTERS: Clear counter on interface Serial2/0
/1/25:0 by autoloop on vty0 (212.31.215.247)
te to up
te to down
te to up
te to down
te to up
te to down
May 22 13:27:15.812 GMT: %LINK-5-CHANGED: Interface Serial2/0/1/25:0, changed st
ate to administratively down
te to down
PE14-DAL-AM#
CUSTOMER UPDATE:

emec@ecolab.com
jason.reed@bt.com
eric.rotunda@bt.com
22/05/2015 14:30:19 DUTY MGR 1
======================================= << Duty Manager >> ====================
===================
T
S
D
A
A
N
C
We have now Unisys and Level 3 FE on site working on the issue.

Chat with SRM/ 3rd line and Global Escalation for updates.
Follow up
last email from Level 3 escalation.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Tony/BT Team,
We've had everyone on site are working together at this time. We are going to va
lidate the demark again while your teams are actively replacing their router/equ
ipment.
I'll update again once we have some new info.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
======================================= << Duty Manager >> ====================
===================
CUSTOMER UPDATE:
emec@ecolab.com
jason.reed@bt.com
eric.rotunda@bt.com
22/05/2015 15:35:11 ANALISTA T2 1
Incident Ended at 22/05/15 18:15:58 GMT. Status changed to CLEARED.
CUSTOMER UPDATE:
emec@ecolab.com
jason.reed@bt.com
eric.rotunda@bt.com
22/05/2015 15:35:32 ANALISTA T2 1
Assignee changed to ANALISTA T2 1
22/05/2015 15:39:24 ANALISTA T2 1

========================
A:Worked with Unisys FE on site to replace the router which restored service.
A: On site customer advised that service is back online running clean.
A: Validated that service is up and stable.
N: Incident closure.
T:Logs:
RFO: Router chassis failure
RES: Router chassis replaced
Neighbor
/PfxRcd
62.6.6.58
V
4
AS MsgRcvd MsgSent
12641
631
103
TblVer InQ OutQ Up/Down State

2049
0 00:15:47
May 22 18:18:37.561 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet0/0, changed s

tate to down
May 22 18:18:41.597 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet0/0, changed s
tate to up
May 22 18:18:42.597 UTC: %LINEPROTO-5-UPDOWN: Line protocol on Interface Gigabit
Ethernet0/0, changed state to up
May 22 18:18:42.617 UTC: %DUAL-5-NBRCHANGE: EIGRP-IPv4 10: Neighbor 10.9.103.249
(GigabitEthernet0/0) is up: new adjacency
na-ecola-car70-dal-am#sh ip int brief

Interface
IP-Address
ocol
Embedded-Service-Engine0/0 unassigned
GigabitEthernet0/0
10.9.103.250
GigabitEthernet0/1
unassigned
Serial0/0/0:0
unassigned
Serial0/0/1:0
unassigned
Loopback10
147.152.37.248
MFR985
unassigned
MFR985.101
147.152.37.248
OK? Method Status

YES
YES
YES
YES
YES
YES
YES
YES
unset
manual
NVRAM
unset
unset
NVRAM
unset
TFTP
Prot
administratively down down

up
up
administratively down down
up
up
up
up
up
up
up
up
up
up
na-ecola-car70-dal-am#ping 62.6.6.58 r 1000 si 1458 df

Type escape sequence to abort.
Sending 1000, 1458-byte ICMP Echos to 62.6.6.58, timeout is 2 seconds:
Packet sent with the DF bit set
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (1000/1000), round-trip min/avg/max = 20/23/96 ms
================
CUSTOMER UPDATE:
emec@ecolab.com
jason.reed@bt.com
eric.rotunda@bt.com
22/05/2015 16:24:58 ANALISTA T2 1
A: Mailed Unisys FE Robert:
@@@@@@@@@@
To:'Chatter, Robert L' <Robert.Chatter@unisys.com>; BTARCE SERVICE G <btarce.ser
vice@bt.com>
Cc:
Subject:RE: txf38105843
Hello Robert,
I have had the chance to collect the traceback logs from defective router, could
you please manage to have someone from Unisys to work and find out the root cau
se for the error?
Attached you have the full collected log, I will expect your feedback upon this.
Thanks!
@@@@@@@@@@
N: Follow up with Unisys FE for updates.
CUSTOMER UPDATE:
emec@ecolab.com
jason.reed@bt.com
eric.rotunda@bt.com
22/05/2015 16:40:04 ANALISTA T2 1
A: Advised customer of service restoral:
@@@@@@@@@@
To:Maskew,R,Rachel,JSE R <rachel.maskew@bt.com>; Johnson, Sandra <Sandra.Johnson
@ecolab.com>; Rotunda,ET,Eric,JKL R <eric.rotunda@bt.com>; Ried, Bill <Bill.Ried
@ecolab.com>; dutymanager americas G <dutymanager.americas@bt.com>; EMEC Service
Account (emec@ecolab.com); Custodio,ES,Emerson,JSW3 R <emerson.silva@bt.com>
Cc:Johnson, Jason (IS) <jason.r.johnson@ecolab.com>; Dissette, Thomas <tdissette
@nalco.com>; Denton, Scott <Scott.Denton@ecolab.com>; Ruppel, Sharon <Sharon.Rup
pel@ecolab.com>
SubjectRE: BT Ticket XF38105843 RE: Incident INC0792620 -- opened on your beha
lf
Dear customer,
Currently the incident have been cleared once the defective router have been rep
laced with a new unity which got properly configured thus completely restoring n
etwork availability for the site. Unisys engineer Robert surveyed the whole loca
tion speaking with the majority of users and got a validation that service is up
and running clean, fact that has been also technically validated by captured lo
g's that can be found at the body of the BT incident ticket.
If you have any further questions or concerns please let me know.
Thanks!
ANALISTA T2 1
@@@@@@@@@@
CUSTOMER UPDATE:
emec@ecolab.com
jason.reed@bt.com
eric.rotunda@bt.com
24/05/2015 18:15:16 ANALISTA 9
Customer Informed of Clear at 24/05/15 21:15:08 GMT. Status Changed to AWAITING
CLOSURE.
CUSTOMER UPDATE:
emec@ecolab.com
jason.reed@bt.com
eric.rotunda@bt.com
24/05/2015 18:16:13 ANALISTA 9
Customer accepted closure at 24/05/2015 22:15:48 GMT . Status changed to CLOSED.
24/05/2015 18:16:29 ANALISTA 9
CLOSURE CONFIRMED
CUSTOMER UPDATE:
'Close Ticket' email sent to: ALANSO.BESSMAN@ECOLAB.COM
emec@ecolab.com
jason.reed@bt.com
eric.rotunda@bt.com

Ticket Editado

Загружено:

Сведения о документе

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

Ticket Editado

Загружено:

Авторское право:

Доступные форматы

20/05/2015 07:34:51 CGW

INCIDENT CONTACT DETAILS:

MTU 1500 bytes, BW 1536 Kbit, DLY 20000 usec,

A - While checking the PE, interface was looped.

- Circuit is still down/down

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

- Level 3 escalated to 2 level.

- BT-NOC SP to follow up with Level 3 for updates.

- Level 3 escalated to 3 level.

- Last update in Level 3 ticket:

5 minute input rate 0 bits/sec, 0 packets/sec

r need to reboot the equipment

Received 0 broadcasts (0 IP multicasts)

S - Both links down. One of them is showing looped.

======================================= << Duty Manager >> ====================

3 on the tickets below.

9251305 ---> 4th level

Site is isolated over 24 hours now;

======================================= << Duty Manager >> ====================

tes on the case.

Assignee changed to ANALISTA 12.

N - To follow-up with Level 3 for updates.

Talked with Unisys Analyst Akshay REF# 33334939

======================================= << Duty Manager >> ====================

0 output buffer failures, 0 output buffers swapped out

'Modify Ticket' email sent to: ALANSO.BESSMAN@ECOLAB.COM

We have now Unisys and Level 3 FE on site working on the issue.

22/05/2015 15:39:24 ANALISTA T2 1

TblVer InQ OutQ Up/Down State

May 22 18:18:37.561 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet0/0, changed s

na-ecola-car70-dal-am#sh ip int brief

OK? Method Status

administratively down down

na-ecola-car70-dal-am#ping 62.6.6.58 r 1000 si 1458 df

Вам также может понравиться