1

Rev A
KPI CSR Workflow Description

Introduction
This description is to be used by engineers as a reference when investigating CSRs reporting KPI degradation. It outlines an investigation strategy but
does not cover every scenario due to the complexity of the WRAN system. It is not exhaustive and currently covers the standard Ericsson WRAN
Retainibility and Accessibility KPIs.


Ericsson KPI formulae
The following are the KPI formulae these workflows are based on.
Speech Accessibility:

Contents

Work Flows
KPI CSR Workflow
CSR Common_WF
Common_WF
Dropped_calls
RRC Setup Failure
Iu Sig Setup Failure
NAS_WF
Rab Setup Failure

Investigation
Procedures
Subrack_IP
Tn_IP
ModuleMP_IP
Iublink_IP
Utrancell_IP
Alarms_IP
LoadRejs_IP
UehExc_IP
Admission_Denied
2

Rev A
Packet Interactive Accessibility:

Speech Drop Rate

Packet Interactive Drop Rate

Contents

Work Flows
KPI CSR Workflow
CSR Common_WF
Common_WF
Dropped_calls
RRC Setup Failure
Iu Sig Setup Failure
NAS_WF
Rab Setup Failure

Investigation
Procedures
Subrack_IP
Tn_IP
ModuleMP_IP
Iublink_IP
Utrancell_IP
Alarms_IP
LoadRejs_IP
UehExc_IP
Admission_Denied
3

Rev A


CSR Common
Workflow
When a CSR is first received a number of
initial actions should always be performed,
1. Check the product the CSR is
written on is still supported and
has not entered ML4. If the
product is no longer supported the
CSR should be de-escalated stating
this, after consultation with the
relevant gatekeeper.
2. Check that the CSR contains the
mandatory and if applicable
problem specific information. If
the DCG logs are not present then
they should be requested either by
email or returning the CSR with a
note.
3. After becoming Familiar with the
CSR problem / request, update the
primus case and perform a search.










DCG Link:
https://ericoll.internal.ericsson.com/sites/PLM/WRAN/Pages/
DataCollectionGuidelines.aspx


Primus Link:
http://e-support.ericsson.se/iview/ui/eserver.asp
CSR escalated
D
e
-
e
s
c
a
l
a
t
e
C
S
R
1. Product
Supported?
2. DCG
Attached?
3. Primus case update and
Search
No
No
Investigation Procedures
Common Workflows
4

Rev A

4. If an existing primus case is found
that answers the CSR request /
problem link that case to the CSR
and fetch it into the CSR. De-
escalate the CSR.


4. Existing
primus case
found?
A

Yes
D
e
-
e
s
c
a
l
a
t
e
C
S
R
Common Workflows Investigation Procedures
5

Rev A

KPI Common
Workflow

1. If the CSR reports KPI degradation
continue with this workflow.
Otherwise the CSR is beyond the
scope of this description.

2. Confirm the customer is using
standard KPI formula. Analyse the
ROP files to confirm the level of
degradation reported. Also
establish when the degradation
first began (when comparing KPIs
time of day and day of week
should be considered).
3. Check Alarm, Event, and System
logs for entries that coincide with
the start of KPI degradation.
4. Check the System logs for OAM
activates carried out on or in the
day before the degradation began.
Moshell Commands







RNC> pmr -r 3 -m <hours>
RNC> pmx utrancell <counter> -m <hours>



RNC> lgaesrm

RNC> lguoqlrm
1. CSR reports
KPI
degradation?

2. Establish exact level and
date / time when KPI
degradation began.
A
B
e
y
o
n
d

S
c
o
p
e
3. Check the System logs
around the time the
degradation started.
4. Check System logs for
OAM activities.
No
Investigation Procedures
Common Workflows
6

Rev A


5. Did degradation occur after RAN
upgrade. If so was it after the RNC
or the RBS upgrade.
6. Check the Network Impact Report
and release notes for changes,
new features or TRs that could be
responsibe.
7. Check for changed parameter
values, newly enabled features
etc.. (Note: Use complete MoM).
8. Breakdown the affected KPI(s) per,
Subrack
Transmission link (ET Board)
Module MP
Iublink
Utrancell level

9. If there is a common component
that is causing the degradation
execute that investigation
procedure. If more than one
component is affected but it is not
widespread start with worst
affected component. If necessary
return and continue with step 10.






RNC> diff . kget_before_upgrade.txt


RNC> pmr -r 5,50-53 -m <hours>
RNC> pmr -r 46-49,56-57 -m <hours>
RNC> pmr -r 4,42-45 -m <hours>
RNC> pmr -r 33-37 -m <hours>
RNC> pmr -r 24,27-28,29-31 -m <hours>
Yes
9. Common
component
found?

MMP
Subrack
Iublink
TN
Cells
8. Check for worst affected
Subrack, Transmission link,
MMP, Iublink, Cells
6. Check Release notes
& NIR for new SW.
7. Perform kget diff on
before and after upgrade
kgets.
5.Degraded
after
upgrade?
Yes
Common Workflows Investigation Procedures
7

Rev A





10. If it is a Retainibility KPI that is
affected choose Drop calls. If
multiple KPIs are degraded choose
the KPI that was first affected.

11. If it is an accessibility KPI that is
affected choose degradation type,
RRC Success Rate
Iu Signalling Setup Success
Rate.
Standalone Srb Success Rate
(NAS).
Rab Success Rate

12. Currently the scope of this
description is limited to
Retainibility and Accessibility KPI
degradation
.



RNC> pmr -r 3 -m <hours>




RNC> pmr -r 3 -m <hours>

RNC> pmxhn utrancell nassignrelease -m
<hours> -a

11. Is
accessibility
affected?
RRC

Rab
Iu Sig
NAS
Yes
No
12. Beyond scope of
this description
Yes
10. Is
Retainibility
affected?
Drop calls
Common Workflows
Investigation Procedures
8

Rev A
Dropped Calls
1. For the worst cell extract the Rab
specific counters and determine if it is
a single Rab type that is affected.
pmNoSystemRabReleaseSpeech
pmNoSystemRabReleaseAmrNb
pmNoSystemRabReleaseAmrWb
pmNoSystemRabReleaseCs64
pmNoSystemRabReleaseCsStream
pmNoSystemRabReleasePacket
pmNoSystemRabReleasePacketStrea
m
pmNoSystemRabReleasePacketStrea
m128
pmNoSystemRabReleasePacketUra
pmNoSystemRabReleasePsStreamHs
pmNoSystemRbReleaseEul
pmNoSystemRbReleaseHs

2. Extract Iurlink counters. Check if one
specific Iurlink is affected or the drop
rate over Iur has causing the increase.
pmNoSystemRabReleaseCs64
pmNoSystemRabReleaseCsStream
pmNoSystemRabReleasePacket
pmNoSystemRabReleasePacketStrea
m
pmNoSystemRabReleaseSpeech
Commands
RNC> pmxh utrancell=<worst cell>
pmnosystemrabrelease -m <hours> -a










RNC> pmr -r 39 -m <hours>
RNC> pmxnh iurlink pmnosystemrabrelease -m
<hours> -a
Drops

1. Check if one Rab Type is
affected.
2. Check if increased drops
are related to one or more
Iurlinks.
Common Workflows
Investigation Procedures
9

Rev A



3. Check for changes to TNL and RNL
QoS.




4. Check with CN concerned for any QoS
changes related to Rab Request.




5. Check Node Synch in RNC and RBS.
and Frame Synch.



6. Check that Frame synch has not
degraded between RNC and RBS.
Check for increase in the number of
discarded or control frames being
sent.
pmNoDchDlTimingAdjContrFrames
pmNoDchUlDataFramesOutsideWindo
w
pmNoDlDchDiscardedDataFramesE
pmNoDlDchDiscardedDataFramesL
pmNoUlDchDiscardedDataFramesE
pmNoUlDchDiscardedDataFramesL
RNC> lguoqlrm













RNC> sts
RBS> sts




RNC> pmxh DchFrameSynch Frames -m <hours> -
a
RBS> pmxh IubDataStreams late -m <hours> -a


3. Check the OAM logs
specifically from changes to
QoS.
4. Check for QoS changes
performed in CN.
UEH Exception
Analysis.

5. Check Node Synch in
RNC and RBS.
6. Check Frame Synch.
Common Workflows
Investigation Procedures
10

Rev A


RRC setup failure
1. Check RRC Rej counters due to
feature RRLC Early Filter, CC Sp
flow control and MMP load.
pmNoRejRrcConnRrcLc
pmNoRejRrcConnSpFlowCtrl
pmNoRejRrcConnMpLoadC

2. For the worst affected Utrancell
check if admission denied counters
have increased,
pmNoRrcReqDeniedAdm
pmNoRrcCsReqDeniedAdm
pmNoRrcPsReqDeniedAdm

3. Are the admission denied counters
stepped due to loadsharing,
pmNoLoadSharingRrcConn
pmNoLoadSharingRrcConnCs
pmNoLoadSharingRrcConnPs

4. Have RL setup failure increased
due to Transport Network (TN)
congestion,
pmNoRrcConnReqBlockTn
pmNoRrcConnReqBlockNode

Commands
RNC> pmxh utrancell=<worst cell> pmnorejrrc -m
<hours> -a



RNC> pmr -r 21 -m <hours>
RNC> pmxh utrancell=<worst cell>
reqdeniedadm$ -m <hours> -a

RNC> pmxh utrancell=<worst cell>
pmnoloadsharingrrcconn -m <hours> -a


RNC> pmxh utrancell=<worst cell>
pmnorrcconnreqblock -m <hours> -a

RRC

1. Found
RRC
Rejects?
Yes
Load Rejs

3. Denied
due to load
sharing?
AdmDenied

No
2. Check for
Adm denied?
End
Check for Admission denied counters?
Yes
Yes
4. TN
Congestion?
TN

Yes
Common Workflows
Investigation Procedures
11

Rev A

5. Check RBS counters for Radio Link
setup failures,
pmSetupFailuresSf128
pmSetupFailuresSf64

6. Request the traces from affected
RBS to investigate failures.
Request support from RBS

7. If issue cannot be resolved with
trace information, request support
from RBS.

RBS> pmxh uplinkbasebandpool
pmsetupfailuressf64 -m <hours>
RBS> pmxh downlinkbasebandpool
pmsetupfailuressf128 -m <hours>

RBS> lh mp te e all NBAP_RESOURCE
RBS> lh mp te e all RADIOLINK_REJECT

5. RL Failures
in RBS?
6. Enable tracing in RBS
to investigate failures.
UEH Exception
Analysis.

Yes
End
7. Request RBS support.
Common Workflows
Investigation Procedures
12

Rev A

Iu Signalling
setup failure
1. Check Alarm, Event, and System
logs for entries relating to RANAP
or SCCP. If alarm found execute
Alarm IP then return to next step if
necessary.
2. Check the te log on RANAP and
SCCP MPs for ERRORs or Info
traces. Execute Error IP then
return to next step if necessary.
3. Are Initial Direct Transfers being
discarded due to RANAP Overload,
pmIdtDiscarded
pmIdtDiscardedCs
pmIdtDiscardedPs
Check MP load during time of
degradation. Also RncFunction
Refused Request counter for SCCP,
pmRefusedRequestSccpMpLoad.





RNC> lgaesrm egrep -i ranap|sccp


RNC> lh sccpmp te log read
RNC> lh ranapmp te log read

RNC> pmxh cnoperator pmidtdiscarded -m
<hours>


RNC> pmr -r 7 -m <hours>
RNC> pmxh rncfunction
pmrefusedrequestsccpmpload -m <hours>

Iu Sig
3. Check RANAP/SCCP MPs
for overload?
1. Found
Alarms &
Events?
Alarm
2. Found
Errors in te
log?
Errors

Yes
Yes
Common Workflows Investigation Procedures
13

Rev A


4. Check for RANAP or SCCP
congestion. DCG logfile <RNC
name>_dcg_m.log, printouts
ranap cong
rof_congestion_info. Also check
that connected Ids are not
reaching maxconn value in
command ranap conn.
5. If MMP Ueh Exception log are
available check for exceptions that
relate to RANAP or SCCP. If the
problem is still present targeted
analysis can be performed.
Execute IP then return to next step
if necessary.
6. Check for ongoing Reset Resource
procedures between RNC and CN
no RANAP MP.







RNC> lh modX te e all UEH_EXCEPTION.


RNC> lh ranapmp te filter set "([1] = 26)"
RANAP_ASN
RNC> lh ranapmp te e bus_send bus_receive
RANAP_ASN

Yes
5. UEH
Exceptions
logs
Available?
4. Check RANAP/SCCP
Congestion?
Exception
Analysis.

6. Check for ResetResource
messages on RANAP MP.
Investigation Procedures Common Workflows
14

Rev A



7. Update the primus case with new
information from investigation to
date.
8. If an existing primus case is found
that answers the CSR request /
problem link that case to the CSR
and fetch it into the CSR. De-
escalate the CSR.
9. Request support from CN to
investigate this problem in Core.


Yes
7. Primus case update
and Search
8. Existing
primus case
found?
D
e
-
e
s
c
al
a
t
e
-
C
S
R
9. Create Support
case towards CN.
Common Workflows
Investigation Procedures
15

Rev A
NAS Sig failure
General: Covers failures during the
NAS signalling phase, which is the
period after the Iu Signalling
connection has been established up
until the Rab Assignment Req is
received by the RNC.
1. Check Alarm, Event, and System
logs for entries relating to
SCCP/RANAP. If alarm found
execute Alarm IP then return to
next step if necessary.
2. Check the te log on the SCCP &
RANAP MPs for Errors/Info traces.
3. Check counters, compare with
values before degradation on RNC
and then cell level. Sort so worst
affected cells appear first.
pmNoSystemReleaseSrbOnly136
pmNoSystemReleaseSrbOnly34
pmNoSystemReleaseSrbOnlyEul
pmNoSystemReleaseSrbOnlyHs
pmNoCellDchDisconnectAbnorm
Commands








RNC> LGAESRM egrep -i sccp|ranap





RNC> lh sccpmp te log read
RNC> lh ranapmp te log read


RNC> pmxhn utrancell
SystemReleaseSrbOnly|DisconnectAbnorm
-m <hours>
RNC> pmx utrancell
pmNoSystemReleaseSrbOnly136 -m
<hours> -a | sort -k3 -n -r
RNC> pmx utrancell
pmNoCellDchDisconnectAbnorm -m
<hours> -m <hours> | sort -k3 -n -r

NAS
1. Found
Alarms &
Events?
Alarm
Yes
2. Found
SCCP Errors?
Errors

Yes
3. Check SrbOnly and
Abnormal Rel counters?
Common Workflows Investigation Procedures
16

Rev A


4. Check the MMP logs for UEH
Exceptions relating to RANAP
(Pertinent ECs disconnectInd &
Release Command with non
Normal cause value). Compare
with logs from before degradation
began. Execute IP then return to
next step if necessary.
5. Request support from CN to
investigate this problem in Core.


4. UEH Exc
logs available?
Exception
Analysis.

Yes
5. Create Support
case towards CN.
Common Workflows Investigation Procedures
17

Rev A
Rab Setup failure
1. Check for increase in Rab mapping
or UE capability failures on node
level.
pmNoInvalidRabEstablishAttemps
pmNoRabEstablishFailureUeCapa
bility
2. Check for changes to the support
Rab Combinations in the RNC. Also
check for any changes to QoS
feature/parameters.
3. Check if the degradation coincides
with the release of a new UE in the
market concerned.
4. Request support from Core wrt
any changes made to QoS
parameters sent in Rab Req.
5. If increase in Adm denied for Rabs
execute IP. (From W12B the
Number of Rab Adm denied is
calculated as total Req denied
minus Rrc Req denied)
pmNoReqDeniedAdm
pmNoRrcReqDeniedAdm

Commands



RNC> pmxhn utrancell
failureUecap|invalidrabest -m <hours


RNC> lgoqlnrm













RNC> pmxhn utrancell [co]reqdeniedadm$ -m
<hours>

RNC> pmxh utrancell
pmNoReqDeniedAdm$|pmNoRrcReqDenie
dAdm$ -m <hours> -a | sort -k3 -n -r
1. Failures in
UE Cap /
mapping?
Rab
2. Check RNC OAM log
for changes to Rab
Combination or QoS?
3. Request info from
customer wrt new UE
releases?
4. Create Support
case towards CN.
Yes
5. Adm
Denied
increased?
AdmDenied

Yes
Common Workflows Investigation Procedures
18

Rev A




6. For Speech Rabs check if number
of directed retries has changes.
pmNoDirRetryAtt

7. Check for increase in Rab failures
due to TN congestion or failure.
pmNoRabEstBlockTnCs57
pmNoRabEstBlockTnCs64
pmNoRabEstBlockTnPsIntHs
pmNoRabEstBlockTnPsIntNonHs
pmNoRabEstBlockTnPsStrHs
pmNoRabEstBlockTnPsStrNonHs
pmNoRabEstBlockTnSpeech

8. Check RBS counters for Radio Link
setup failure increase,
pmSetupFailuresSf256
pmSetupFailuresSf128
pmSetupFailuresSf64
pmSetupFailuresSf32
pmSetupFailuresSf16
pmSetupFailuresSf8
pmSetupFailuresSf4

9. Request the traces from affected
RBS to investigate failures.

10. If issue cannot be resolved with
trace information, request support
from RBS.





RNC> pmxhn utrancell pmNoDirRetryAtt -m
<hours>
RNC> pmxh utrancell pmNoDirRetryAtt -m
<hours> -a | sort -k3 -n -r

RNC> pmxhn utrancell pmNoRabEstBlockTn -m
<hours>
RNC> pmxh utrancell pmNoRabEstBlockTn -m
<hours> -a | sort -k3 -n -r





RBS> pmxh uplinkbasebandpool
pmsetupfailuressf -m <hours>
RBS> pmxh downlinkbasebandpool
pmsetupfailuressf -m <hours>







RBS> lh mp te e all NBAP_RESOURCE
RBS> lh mp te e all RADIOLINK_REJECT

7. TN
Congestion?
TN

8. RL Failures
in RBS?
9. Enable tracing in RBS
to investigate failures.
10. Request RBS
support.
UEH Exception
Analysis.

End
Yes
Yes
6. Enable tracing in RBS
to investigate failures.
Investigation Procedures
Common Workflows
19

Rev A





INVESTIGATION
PROCEDURES
(IP)

















20

Rev A
IP: Subrack
General: This investigation Procedure (IP) is
only to be used where KPI degradation is
found to relate to a single Subrack (not a
single Transmission Link or MMP within the
subrack).
1. Check for ISL/ESL faults and Overload.
- Check for crashes on Scb(x) boards
- Check for errors on Scb(x) boards
- Check for ISL Overload
- Check the controlling MMP and ATM
port for Iublinks are in same Subrack

2. Check Subrack ETs for faults.
3. Check MMP for usage and faults.


4. Check the Subrack DcDevice
allocation, usage and faults.
5. Check the Subrack CcDevice
allocation, usage and faults. For ATM if
Ccdevice allocated in different Subrack than
ET termination can cause ISL load.
6. If ATM is used on Iu, check that
atmUserPlaneTermSubrackRef is set in IuLink
MO, if not this can cause ISL load.
7. Check RSTP configuration.
8. Additional spas related information
can be gathered with dcgx command.
Commands






RNC> lgs
RNC> lgt -g scx | egrep -i
error|restart
RNC> pmx . pmPeakBwLevel -m
<hours>
RNC> lkra


RNC> lgt -g et | egrep -i
error|restart

RNC> lh modxx te log read | egrep -i
error|restart (xx =
ms,es1,es2)


RNC> std
RNC> lgt -g dcxx (xx = ms,es1,es2)

RNC> lgt -g ccxx (xx = ms,es1,es2)
RNC> cedr


RNC> get IuLink
atmUserPlaneTermSubrackRef



RNC> steg

RNC> dcgx


Contents



Work Flows
KPI CSR Workflow
CSR Common_WF
Common_WF
Dropped_calls
RRC Setup Failure
Iu Sig Setup Failure
NAS_WF
Rab Setup Failure

Investigation
Procedures
Subrack_IP
Tn_IP
ModuleMP_IP
Iublink_IP
Utrancell_IP
Alarms_IP
LoadRejs_IP
UehExc_IP
Admission_Denied
21

Rev A
IP: Transport Net
General: This investigation is only to be used
where KPI degradation is found to relate to
Transport Network.
1. Check TN/RSTP configuration.
2. Check alarm log from bouncing links.
3. Check the System logs for OAM config
changes associated with the TN.
4. Check the ET board(s) for restarts or
errors.
5. Check for Aal2 setup failures
6. Perform ping / ete loopback test on
suspected TN Links.
7. Check intermediate TN nodes
(Routers, RXIs etc..) for alarms or errors.
8. Check intermediate TN nodes for OAM
config changes.
9. If a single far end node is involved
(MSC, SGSN, RBS) check this for alarm or
errors.
10. Check far end node for OAM config
changes.
11. Sometimes the nature of a TN fault
means isolating the faulty transmission link
requires removing it from service. This can be
done by locking/unlocking TL in rotation until
the faulty link is found.

Commands




RNC> stv
RNC> sti
RNC> steg

RNC> lga

RNC> lguoqlrm


RNC> lgt -g et | egrep -i
error|restart
RNC> lgs

RNC> pmr -r 57 -m <hours>

RNC> stip
RNC> acc <VclTp> eteLoopBack

RXI> lga
RXI> lgt -g et | egrep -i
error|restart
RXI> lgs

RXI> lguoqlrm


RBS> lga
RBS> lgt -g et | egrep -i
error|restart
RBS> lgs

RBS> lguoqlrm


Contents



Work Flows
KPI CSR Workflow
CSR Common_WF
Common_WF
Dropped_calls
RRC Setup Failure
Iu Sig Setup Failure
NAS_WF
Rab Setup Failure

Investigation
Procedures
Subrack_IP
Tn_IP
ModuleMP_IP
Iublink_IP
Utrancell_IP
Alarms_IP
LoadRejs_IP
UehExc_IP
Admission_Denied
22

Rev A
IP: Module MP
General: This investigation is only to be used
where a KPI degradation is found to relate to
a Module MP.
1. Check the MMP for restarts or errors.

2. Check MMP load.



3. Check the System logs for OAM
config changes associated with this MMP.

4. Check DCs associated with this MMP
for overload and faults.




5. Check if large Location/Routing Areas
could be causing high load due to paging.
Commands




RNC> lgs
RNC> lgt -g modx | egrep -i
error|restart


RNC> pmr -r 7 -m <hours>



RNC> Lguoqlrm


RNC> std
RNC> pmr -r 9 -m <hours>
RNC> pmxh dcdevice=<dc>
MeasLoad -m <hours>
RNC> lgt -g dcx | egrep -i
error|restart
RNC> pmr -r 8 -m <hours>



RNC> get locationarea reserved
RNC> get Routingarea reserved




Contents



Work Flows
KPI CSR Workflow
CSR Common_WF
Common_WF
Dropped_calls
RRC Setup Failure
Iu Sig Setup Failure
NAS_WF
Rab Setup Failure

Investigation
Procedures
Subrack_IP
Tn_IP
ModuleMP_IP
Iublink_IP
Utrancell_IP
Alarms_IP
LoadRejs_IP
UehExc_IP
Admission_Denied
23

Rev A
IP: Iublink
General: This investigation is only to be used
where a KPI degradation is found to relate to
all cells in a specific IubLink(s).
1. Check the System logs for OAM config
changes associated with this site (RNC+RBS).

2. Check system logs for recent entries
concerning Iublink/Utracells (RNC+RBS).

3. Check for errors related to this Iublink
on terminating MMP.
4. Check if Iublink credit usage going
over 80%, AvgDlCredits/AvgUlCredits.
5. Check resource usage and hanging
resources.

6. Check for high Ul RSSI in the cells for
the site.
7. In RBS check license state and
capability.
8. Check that Iublink is residing in same
Subrack as user plane resources.

9. Check for changes in nodesynch delay
and also the RBS Synch status. Also counters
for discarded frames in RBS.

10. Check NTP synch in RBS and RNC.
Commands





RNC> Lguoqlrm
RBS> Lguoqlrm


RNC> lgaesrm
RBS> lgaesrm


RNC> lgt -g modx


RNC> pmxh IubLink=<Iublink>
pmdlCredits|pmulCredits -m
<hour>

RNC> cedh
RNC> lh ranapmp ueregprint age
3600

RNC> pmr -r 32 -m <hours>


RBS> get NodeBFunction license


RNC> strt
RNC> pmxh
IubLink=<Iublink>,NodeSynch
delay -m <hours>

RBS> get Synchronization
RBS> pmxh IubDataStreams
pmDchFrames.*late -m
<hours>

RNC> ntpconfig info
RBS> ntpconfig info


Contents



Work Flows
KPI CSR Workflow
CSR Common_WF
Common_WF
Dropped_calls
RRC Setup Failure
Iu Sig Setup Failure
NAS_WF
Rab Setup Failure

Investigation
Procedures
Subrack_IP
Tn_IP
ModuleMP_IP
Iublink_IP
Utrancell_IP
Alarms_IP
LoadRejs_IP
UehExc_IP
Admission_Denied
24

Rev A
IP: Utrancell
General: This investigation is only to be used
where a KPI degradation is found in a
Utrancell(s) but not a complete site.
1. Check system logs for recent entries
concerning the Utracell (RNC+RBS).
2. Check the System logs for OAM config
changes associated with this cell (RNC+RBS).
3. Check for errors related to this cell on
terminating MMP.
4. Check in the RBS for errors/restarts
on RAX/TX boards for this cell.
5. Check for high Ul RSSI in this cell.
Commands




RNC> lgaesrm
RBS> lgaesrm

RNC> Lguoqlrm
RBS> Lguoqlrm

RNC> lgt -g modx | egrep -i <Cell
Id>

RBS> lgt egrep -i error|restart



RNC> pmr -r 32 -m <hours



Contents



Work Flows
KPI CSR Workflow
CSR Common_WF
Common_WF
Dropped_calls
RRC Setup Failure
Iu Sig Setup Failure
NAS_WF
Rab Setup Failure

Investigation
Procedures
Subrack_IP
Tn_IP
ModuleMP_IP
Iublink_IP
Utrancell_IP
Alarms_IP
LoadRejs_IP
UehExc_IP
Admission_Denied
25

Rev A
IP: Alarms
This investigation is to be used when relevant
Alarm(s) is found.
Alarms:
Where a relevant alarm(s) is found the
corresponding Operational Instruction (OPI)
found in CPI Store should be executed. The
correct OPI can be found by using the Specific
Problem text from the alarm to perform a
search in the relevant library (e.g. search for
Fach_InternalResourceUnavailable in library
WCDMA RNC 3820 W12.0)

IP: Error codes
SCCP Errors:
1. SCCP Error codes can be decoded at
cello work support tool, they are also
described in Maintenance Instructions SCCP
(1/1541-CAA 901 437 Uen).
2. It may be necessary to contact CPP
support where an SCCP Error has occurred as
the Error code description is often cryptic.






CPI Store Link:
http://cpistore.internal.ericsson.com/alex








Cello Work support tool Link:
http://support.dach.ericsson.se/worksupport/t
ools/Cello_err.php3


Contents



Work Flows
KPI CSR Workflow
CSR Common_WF
Common_WF
Dropped_calls
RRC Setup Failure
Iu Sig Setup Failure
NAS_WF
Rab Setup Failure

Investigation
Procedures
Subrack_IP
Tn_IP
ModuleMP_IP
Iublink_IP
Utrancell_IP
Alarms_IP
LoadRejs_IP
UehExc_IP
Admission_Denied
26

Rev A
IP: Load Rejects
1. If pmNoRejRrcConnMpLoadC is being
stepped, check if down switch due to soft
congestion is occurring. This can result in high
load on mMP.
pmNoOfSwDownEulCong
pmNoOfSwDownHsCong
pmNoOfSwDownNgCong
pmSoftCongRbsUlHw
pmNoOfSwDownNgHo
If down switch due to congestion is
confirmed, consider enabling feature RRLC
Early Filter on the affected cells.

2. If pmNoRejRrcConnRrcLc is being
stepped check alarms in RNC related to this
feature
Check also alarms in RBS related to HW
failures, refer to IP Alarms.

3. If pmNoRejRrcConnSpFlowCtrl is
being stepped, check the number of cells
allocated to that CC device.
4. Check that the CcDevice used by this
cell is in the same Subrack as the Iublink TN
termination.

5. Check the CC device load.




RNC> pmxh utrancell NoOfSwDown -
m <hours> -a














RNC> altk

RBS> altk



RNC> std cc



RNC> str -I <Iublink Id>



RNC> pmr -r 8 -m <hours>


Contents



Work Flows
KPI CSR Workflow
CSR Common_WF
Common_WF
Dropped_calls
RRC Setup Failure
Iu Sig Setup Failure
NAS_WF
Rab Setup Failure

Investigation
Procedures
Subrack_IP
Tn_IP
ModuleMP_IP
Iublink_IP
Utrancell_IP
Alarms_IP
LoadRejs_IP
UehExc_IP
Admission_Denied
27

Rev A
IP: UEH Exception

UEH Exception Identification
There are a number of stratages that can be
used when analysing UEH Exception to
identify the exception associated with the
degradation. The strategy used will depend
on a number of factors including,
The size of degradation, is it 0.1% or
10%.
Nature of the degradation, is it
localised to cluster/site/area or widespread .
Available data, is data availabe from
before degradation began for comparison
purposes.
In general it is good to narrow the area of
investiagtion as much as possible. Use
targated tracing on,worst affected cells,sites
or mMP, only tracing on worst affected
UeRc(s), etc.
General analysis:
This is where all UEH Exception are included
and an analysis is done to breakdown the
number of exceptions per Exception code,
Cause code, Transition type, etc ... This is
most useful when the size of the degradation
is relatively large compared to the normal
number of failures that occur for that KPI, and
data is available from before the problem
began to compare against.
This analysis is performed in 3 steps,
1. Parse exception logs from before
degradation.
2. Parse exception logs taken during
degradation.
3. Using uehExcAnalyser.xlsm import
and analysis the data in excel.
Commands





















RNC> lh modx te e all
UEH_EXCEPTION

Cygwin$ java -jar
ExcAnalyser.jar <Raw Exc file before>
<Parsed Exc file before>
Cygwin$ java -jar Analyser.jar
<Raw Exc file during> <Parsed Exc file
during>
Contents



Work Flows
KPI CSR Workflow
CSR Common_WF
Common_WF
Dropped_calls
RRC Setup Failure
Iu Sig Setup Failure
NAS_WF
Rab Setup Failure

Investigation
Procedures
Subrack_IP
Tn_IP
ModuleMP_IP
Iublink_IP
Utrancell_IP
Alarms_IP
LoadRejs_IP
UehExc_IP
Admission_Denied
28

Rev A

Targeted analysis:
This is where only UEH Exceptions that have
been associated with the stepping (or not
stepping in case of attempt/success counters)
of the counter that resulted in the KPI
degradation are included in the analysis.
Additional counter specific tracing are
required along with the standard UEH
Exception trace. Due to the counter tracing
requirement useful trace logs are usually not
availabe from before the problem began for
comparison.
This analysis is performed in 3 steps,
1. Identify what counter(s) within the
KPI formula that has changed resulting in the
degradation. Extract the values for the
counters concerned individually and compare
against those from before degradation.

2. Request counter specific traces to be
taken for the counter(s) identified in step 1 (if
it is an attempt without a success, trace both
counters).





3. Parse the logfile to extract only the
UEH Exceptions that are associated with the
releveant counter(s).



4. Using ExcAnalyser.xlsm import and
analysis the data in excel.



















RNC> pmx <mo> <counter> -m
<hours>




RNC> lh modx ue_pm print -counter
| egrep -i <counter>
RNC> lh modx ue_pm enable -
counter <counter1 Id>
RNC> lh modx ue_pm enable -
counter <conter2 Id>
RNC> lh modx te e trace6
PM_OBS_IND

Cygwin$ java -jar
ExcAnalyser.jar <Raw Except file
name> <Parsed Exc file name>
<counter name>
Cygwin$ java -jar
ExcAnalyser.jar <Raw Exc file name>
<Parsed Exc file name> <attempt
counter name> <^success counter
name>


Contents



Work Flows
KPI CSR Workflow
CSR Common_WF
Common_WF
Dropped_calls
RRC Setup Failure
Iu Sig Setup Failure
NAS_WF
Rab Setup Failure

Investigation
Procedures
Subrack_IP
Tn_IP
ModuleMP_IP
Iublink_IP
Utrancell_IP
Alarms_IP
LoadRejs_IP
UehExc_IP
Admission_Denied
29

Rev A
Filtering:
Where the degradtion has been identified as
affecting a particular Utrancell(s) or UeRc,
then it is useful to narrow the tracing,
outlined in step 2 above, to the worst
affected cell and/or UeRc. This is done using
UE Random tracing.



Examples:
1. Increase in Speech Drop Rate KPI,
with Utrancell cid 4217 being worst affected .





2. Degradation of Eul Accessibility with
UeRc=25 in Utrancell cid=4200 worst
affected. Worst affect UeRc found using
Attempt and Success UeRc MO counters.

RNC> lh modx traceCounter on
<counter1 Id>
RNC> lh modx traceCounter on
<counter2 Id>
RNC> lh modx te e trace9
UE_COUNTERS_GENERAL
RNC> lh modx te e all
UEH_EXCEPTION
RNC> lh modx uerandtrace on -cell
cid <cid> -uerc <uerc>
RNC> lh modx uerandtrace max -
unlim

Worked Example: Speech drop rate
degradation.
RNC> lh mod80 traceCounter on 4284
RNC> lh mod80 te e trace9
UE_COUNTERS_GENERAL
RNC> lh mod80 te e all UEH_EXCEPTION
RNC> lh mod80 uerandtrace on -cell cid
4217
RNC> lh mod80 uerandtrace max -unlim
Cygwin$ java -jar ExcAnalyser.jar Inputfile.log
outputfile.csv
PMNOSYSTEMRABRELEASESPEECH

Worked Example: Eul Accessibility
degradation.
RNC> lh mod8 traceCounter on 4564
RNC> lh mod8 traceCounter on 4565
RNC> lh mod8 te e trace9
UE_COUNTERS_GENERAL
RNC> lh mod8 te e all UEH_EXCEPTION
RNC> lh mod8 uerandtrace on -cell cid 4200
-uerc 25
RNC> lh mod8 uerandtrace max -unlim
Cygwin$ java -jar ExcAnalyser.jar Inputfile.log
outputfile.csv
PMNORABESTABLISHATTEMPTPACKETINTERAC
TIVEEUL -crossreference
"^PMNORABESTABLISHSUCCESSPACKETINTERA
CTIVEEUL"

Contents



Work Flows
KPI CSR Workflow
CSR Common_WF
Common_WF
Dropped_calls
RRC Setup Failure
Iu Sig Setup Failure
NAS_WF
Rab Setup Failure

Investigation
Procedures
Subrack_IP
Tn_IP
ModuleMP_IP
Iublink_IP
Utrancell_IP
Alarms_IP
LoadRejs_IP
UehExc_IP
Admission_Denied
30

Rev A
UEH Exception Meaning
Where a relevant Ueh Exception is found
there are a number of options for further
investigation,
1. The exception mean and description
of the scenario in which it would occur, along
with further trace suggestion can be found in
the Common Exception Handler (CEH).
2. If the Exception is not present in the
CEH (not all Ueh Exception are added to this
tool yet) then the exception meaning can be
found in UEH Exception Definition document
(available in CDM).
3. For further explanation and details it
is also possible to submit a support request
to the UEH Subsystem within RNC.



CEH Link:
https://wcdma-
commonexception.rnd.ki.sw.ericsson.se

CDM Link UEH Exception Definition:
32/1551-CRA 403 38/1


Contents



Work Flows
KPI CSR Workflow
CSR Common_WF
Common_WF
Dropped_calls
RRC Setup Failure
Iu Sig Setup Failure
NAS_WF
Rab Setup Failure

Investigation
Procedures
Subrack_IP
Tn_IP
ModuleMP_IP
Iublink_IP
Utrancell_IP
Alarms_IP
LoadRejs_IP
UehExc_IP
Admission_Denied
31

Rev A
IP: Admission
Denied
RRC Related:
1. Check the following resource specific
counters for indication of RBS HW, Power or
downlink code congestion. Compare values
with those from before degradation began.
pmNoRrcReqDeniedAdmDlHw
pmNoRrcReqDeniedAdmUlHw
pmNoRrcReqDeniedAdmDlPwr
pmNoRrcReqDeniedAdmDlChnlCode

2. Check for hanging resources in the
RBS
Rab Related:
1. Check the following resource specific
counters for indication of RBS HW, Power or
downlink code congestion. Compare values
with those from before degradation began.
pmNoFailedRabEstAttemptExceedConnLi
mit
pmNoFailedRabEstAttemptLackDlAse
pmNoFailedRabEstAttemptLackUlAse
pmNoFailedRabEstAttemptLackDlChnlCo
de
pmNoFailedRabEstAttemptLackDlHw
pmNoFailedRabEstAttemptLackUlHw
pmNoFailedRabEstAttemptLackUlHwBest
pmNoFailedRabEstAttemptLackDlHwBest
pmNoFailedRabEstAttemptLackDlPwr
pmNoFailedRabEstAttEulRateCong

2. Check for hanging resources in the
RBS.

Commands


RNC> pmxh utrancell=<worst cell>
pmNoRrcReqDeniedAdm -m
<hours>









RNC> cedh
RNC> lh ranapmp ueregprint age
3600


RNC> pmxh utrancell=<worst cell>
FailedRabEstAt-m <hours>














RNC> cedh
RNC> lh ranapmp ueregprint age
3600

Contents



Work Flows
KPI CSR Workflow
CSR Common_WF
Common_WF
Dropped_calls
RRC Setup Failure
Iu Sig Setup Failure
NAS_WF
Rab Setup Failure

Investigation
Procedures
Subrack_IP
Tn_IP
ModuleMP_IP
Iublink_IP
Utrancell_IP
Alarms_IP
LoadRejs_IP
UehExc_IP
Admission_Denied