BTS+ restart interim report

Telkomsel, Indonesia
Martin Holecek, Manish Handa, Erawan Sukma
23st December 2011

Strictly confidential
1
Nokia Siemens Networks

BTS+ restarts / Martin Holecek , Manish Handa, Erawan Sukma/ 23rd December 2011

BTS+ restart
Content
Executive summary
Progress on corrective activity
Root cause analyses
Root cause
Final Solution
Statistical BTS restart comparison
COBA2P8 HW replacement status

Strictly confidential
2
Nokia Siemens Networks

BTS+ restarts / Martin Holecek , Manish Handa, Erawan Sukma / 23rd December 2011

BTS+ with COBA2P8

Executive summary
Case description
BTS+ with high TRX configuration sporadically restarted under high traffic
load. This was due to watch dog expiry as a result of lack of processing
power in COBA2P8 HW unit due to OS scheduling design SW fault
Current situation
After deployment of final BTS SW correction 11-20 the BTS restarts are
resolved

Strictly confidential
3
Nokia Siemens Networks

BTS+ restarts / Martin Holecek , Manish Handa, Erawan Sukma / 23rd December 2011

BTS+ with COBA2P8

Progress on corrective activity
Corrective plan (1st December 2011, Batam)
BTS+ SW update to 11-20 (priority for site with TRX configuration >= 20 TRX)
Estimated update duration 4 working days (400 sites), 3 weeks for all CA sites
CR submitted on 1-Dec-11, estimated start 5-Dec-11
All BTS+ under CommonAbis are updated to latest BTS+ SW 11-20 by 20th Dec 2011

NSN will preventively replace all COBA2P8 &COSA6P16 on all sites with TRX configuration >= 20 TRX
Estimated delivery of replacement units 83 units by CW49; 317 by CW 03 2012
Estimated replacement on site 83 units by CW50, priority to be set by Telkomsel
Batch#1 delivered and partially replaced on site (details next slide)
Estimated replacement of remaining 317 units by CW7 2012 is canceled
Further COBA2P8 replacement are not required as final SW solution is available

NSN Root cause analyses

R&D in Jakarta since 11th Dec 2011
Root cause identified and final correction deployed
Deployed correction was monitored since 1st December 2011

Strictly confidential
4
Nokia Siemens Networks

BTS+ restarts / Martin Holecek , Manish Handa, Erawan Sukma / 23rd December 2011

BTS+ with COBA2P8

Root cause analyses
Root cause
Observed COBA2P8 card reset was due to watchdog timer expiry and OS scheduling design
Watchdog Timer triggers a system reset or other corrective action if the main program, due any condition (such as
a hang), neglects to regularly service the watchdog .The intention is to bring the system back from the
unresponsive state into normal operation
What situation caused Watch dog expiry
Priority task A always preempts low priority task B
After priority task A gets processed, low priority task B gets control again
If there are more then one high priority tasks the low priority task B will wait till all high priority tasks finish
in high load situation, low priority task B never gained control and at one point watchdog timer expired
Root cause analyses history
Priority FR opened on 20th June 2011 (NA04948089)
Correction 11-12 provided on 15th September 2011 (restart reduction not measured, 1 BTS deployed)
Correction 11-16 provided on 31st October 2011 (restart reduction ~75%, some BTS doesnt restart at all)
Correction 11-20 provided on 25th November 2011 (67 BSC, 3843 BTS loaded*)
Monitoring ongoing until 23rd December 2011
Final solution

BTS+ SW correction 11-20, that modifies OS scheduling design

Strictly confidential
5
Nokia Siemens Networks

BTS+ restarts / Martin Holecek , Manish Handa, Erawan Sukma / 23rd December 2011

BTS+ with COBA2P8

Statistical BTS restart comparison
Before 11-20 Build
20

15

Before BTS+ SW 11-20 deployment

SW reset (watch dog expiry) is major contributor
HW reset present (PLN, Transmission)*
Other (manual O&M procedures)*

10

Restarts of all BTS+ are shown

5

This includes BTS with COBA2P and COBA4P

0

SW reset

HW reset

Other

After 11-20 Load

20

15

10

After BTS+ SW 11-20 deployment

SW reset (watch dog expiry) is zero
HW reset present (PLN, Transmission)*
Other (manual O&M procedures)*

100% reduction Watch dog expiry reset

54% overall reset reduction

SW reset

HW reset

Other

* NSN have no control over PLN, Transmission and O&M activities, these restarts occur on all BTS types and HW configurations

Strictly confidential
6
Nokia Siemens Networks

BTS+ restarts / Martin Holecek , Manish Handa, Erawan Sukma / 23rd December 2011

BTS+ with COBA2P8

Additional information on current BTS restarts
Other cause
O&M activity

Count Solution

Remark

N/A

User interventions on BCF

OCVCXO

COBA board
replacement (FLM)

* See next slide for details

Unknown

logs missing

Suspected OCVCXO

Physical link

FLM on tranmission

BTS restarts due to "Other"

6
5
4
3

BSS init

2
Activity on BSS (BSC init, units
restarts)

N/A

1
0
O&M

HWR cause
Known

Uknown

Count Solution
138

38

Remark

FLM on
Infrastructure
N/A

Main failure, Rectifier failure,

Fuse alarm and Low battery
HW induced restart with
missing detail indication,
potentially not created ENVA
alarms

OCVCXO

Unknown Physical link

BTS restarts due to "HW"

unknown
22%

known
78%

Strictly confidential
7
Nokia Siemens Networks

BSS init

BTS+ restarts / Martin Holecek , Manish Handa, Erawan Sukma / 23rd December 2011

BTS+ with COBA2P8

Additional information on current BTS restarts
How to recognize OCVCXO error

Strictly confidential
8
Nokia Siemens Networks

BTS+ restarts / Martin Holecek , Manish Handa, Erawan Sukma / 23rd December 2011

BTS+ with COBA2P8

COBA2P8 HW replacement status
Temporary Workaround
Temporary workaround is COBA2P8 &COSA6P16 replacement with COBA4P12 &COSA4P12
List of replaced and delivered units per region
Config >= 20 TRXes

Sumbags
el

Kalimantan

Bali Nunsra

Sumbagteng

Sulmalpua*

Total

Replaced till 1st Dec 2011

22

12

46

Batch #1 delivery

47

49

Batch #1 replacement

15

17

Batch #2 delivery**

N/A

N/A

N/A

N/A

N/A

N/A

Batch #2 replacement**

N/A

N/A

N/A

N/A

N/A

N/A

Not replaced

102

11

38

61

214

* After COBA2P8 audit in Sulmalpua region requirements reduced from 231 to 61units
** Batch #2 delivery on hold, final SW correction deployed and under monitoring

Strictly confidential
9
Nokia Siemens Networks

BTS+ restarts / Martin Holecek , Manish Handa, Erawan Sukma / 23rd December 2011

Strictly confidential
10
Nokia Siemens Networks

BTS+ restarts / Martin Holecek , Manish Handa, Erawan Sukma/ 23rd December 2011