KPI Definitions in TEMS Products

Copyright Infovista Sweden AB 2017

TEMS is a trademark of Infovista. All other trademarks are the property of their respective holders.
No part of this document may be reproduced in any form without the written permission of the copyright holder.

The contents of this document are subject to revision without notice due to continued progress in methodology, design and
manufacturing. Infovista shall have no liability for any error or damage of any kind resulting from the use of this document.

www.Infovista.com | 1
Contents
1 Introduction....................................................................................................................................................... 3
2 General Aspects of KPI Data Collection and Computation ............................................................................... 4
2.1 KPI Reporting ............................................................................................................................................ 4
2.2 ETSI Compliance ...................................................................................................................................... 4
2.3 Complete vs. Timed Measurements .......................................................................................................... 4
2.4 IP Capture Options .................................................................................................................................... 5
3 KPIs by Service ................................................................................................................................................ 6
3.1 KPIs for FTP Download ............................................................................................................................. 6
3.1.1 Timed Measurements ........................................................................................................................ 6
3.1.2 SFTP (Secure Shell FTP) .................................................................................................................. 6
3.2 KPIs for FTP Upload ................................................................................................................................. 6
3.2.1 Timed Measurements ........................................................................................................................ 7
3.2.2 SFTP (Secure Shell FTP) .................................................................................................................. 7
3.3 KPIs for HTTP Get/Download ................................................................................................................... 7
3.3.1 Timed Measurements ........................................................................................................................ 7
3.4 KPIs for HTTP Post/Upload....................................................................................................................... 7
3.4.1 Timed Measurements ........................................................................................................................ 8
3.5 KPIs for Ping ............................................................................................................................................. 8
3.5.1 Timed Measurements ........................................................................................................................ 8
3.6 KPIs for CS Voice...................................................................................................................................... 9
4 Notes on Other Services: Email, Iperf, Streaming, CS Fallback ..................................................................... 10
4.1 Email ....................................................................................................................................................... 10
4.2 Iperf ......................................................................................................................................................... 10
4.2.1 TCP ................................................................................................................................................. 10
4.2.2 UDP ................................................................................................................................................. 11
4.3 Streaming over HTTP .............................................................................................................................. 11
4.3.1 Streaming Player Download ............................................................................................................ 11
4.3.2 Start of Streaming Video Replay...................................................................................................... 12
4.3.3 Streaming Video Session ................................................................................................................ 12
4.4 CS Fallback ............................................................................................................................................. 15

www.Infovista.com | 2
1 Introduction
This document describes the implementation of ETSI circuit-switched and packet-switched service KPIs, as well
as some closely related non-ETSI KPIs, in TEMS products.

The document is valid for the following TEMS product versions:

• TEMS Automatic 10.1 and later

• TEMS Investigation 16.0 and later

• TEMS Symphony 7.4 and later

WARNING: This legacy document has not been updated for several years. It should be read for background
information and historical interest only. Please contact Infovista before relying on any technical details in this
document.

www.Infovista.com | 3
2 General Aspects of KPI Data Collection and Computation

2.1 KPI Reporting

For PS data services, current TEMS products report KPI events, each of which contains either

• an instance of the KPI itself (such as a session duration or throughput figure), or

• an instance of an occurrence whose probability is measured by the KPI (such as a session failure).

These in turn underlie the aggregation of KPI statistics in TEMS Discovery or some other post-processing tool.
The “probability” KPIs obviously need to be based on a fair number of sessions for a good estimate of the
probability in question to be obtained. The data-collecting TEMS products do not in themselves output values of
these latter KPIs; rather, they need to be calculated during post-processing.

TEMS Symphony users should note that earlier versions of TEMS Symphony (version 6.x and older) did not
report data in this fashion, but only trigger points on which to base the KPI computation: for example, session start
and end times.

Regarding CS voice, see section 3.6.

2.2 ETSI Compliance

In TEMS products, timestamps for KPI computation are captured as close to the source as possible: that is, from
the packet capture driver within the operating system kernel.

Calculations adhere to the specification  ETSI TS 102 250-2 V2.2.1 (2011-04) unless otherwise stated. This
specification is hereafter referred to as “ETSI 102 250-2”.

2.3 Complete vs. Timed Measurements

The standardized KPIs have been defined with complete transactions in mind. Transactions that are aborted
after a fixed duration specified by the TEMS product user (timed measurements) have not been considered. In
case of timed measurements, a best-effort solution is provided. Where triggers do not fully match ETSI 102 250-2,
they have been aligned as closely as possible to their ETSI counterparts in a manner compliant with  ETSI TS
102 678 V1.2.1, except that the following item stated in section 4.1 of that specification has not been
implemented:

The connection should be checked to be still alive at the end of the transfer period Δt d. The reception of
any data packet sent by the server on the data connection after the end of the transfer period is a valid
indicator that the data connection is still alive. An appropriate timeout of e.g. 3 to 5 multiples of typical RTT
in the measured network might be used while waiting for the desired packet.

“Session Time” KPIs, of course, make sense only for sessions that have been allowed to run to completion.

For details, see the various subsections of chapter 3.

www.Infovista.com | 4
2.4 IP Capture Options
The default IP data capture that is done by TEMS products during PS data service testing is designed to obtain
sufficient input for KPIs. No special activity needs to be used in Service Control scripts for this purpose.

It is however also possible to conduct a more thoroughgoing IP sniffing by means of the “Start IP Sniffing” script
activity. This activity has a “Filter” parameter whose possible settings are given here for convenience.

Filter: Type of filtering to apply to IP packets.

• Optimized Performance: Capture of IP packets is reduced to the minimum needed to compute KPIs. (Packets
are filtered with respect to IP address, protocol [TCP/UDP], and port.) Not supported for on-device
measurement.

• Headers Only: The first 54 bytes of every IP packet are captured.

• None: No filtering of IP packets. Note: This means that all packets belonging to any service used by the PC
will be captured; packets are filtered with respect to IP address only. Be aware that full packet capturing can
cause crashes on high speed networks where sustained high throughput is achieved.

www.Infovista.com | 5
3 KPIs by Service
This chapter deals with TEMS product KPIs for each service.

3.1 KPIs for FTP Download

ETSI-to-TEMS Mapping Table
ETSI KPI “FTP {Download|Upload}” Sect. TEMS KPI Event
Service Non-Accessibility 6.1.1 FTP Download Service Not Accessible
Setup Time 6.1.2 FTP Download Setup Time
IP-Service Access Failure Ratio 6.1.3 FTP Download IP Service Access Failure
IP-Service Setup Time 6.1.4 FTP Download IP Service Setup Time
Session Time 6.1.6 FTP Download Data Transfer Time
Mean Data Rate 6.1.7 FTP Download Mean Data Rate
Data Transfer Cut-off Ratio 6.1.8 FTP Download Data Transfer Cutoff

Note: “Method A” and “Method B” values are differentiated wherever applicable. See ETSI 102 250-2
section 4.2.

3.1.1 Timed Measurements

For timed measurements, the end trigger is defined as “Last incoming packet containing content”. This trigger is
valid for both completed and aborted transfers. Since this definition conforms to the ETSI standard, timed FTP
downloads will have ETSI compliant KPIs.

3.1.2 SFTP (Secure Shell FTP)

If SFTP is used for the file transfer rather than FTP proper, no KPIs are obtained. ETSI 102 250-2 does not define
any KPIs for SFTP, and some metrics would not be possible to obtain owing to the SSH encryption.

3.2 KPIs for FTP Upload

ETSI-to-TEMS Mapping Table
ETSI KPI “FTP {Download|Upload}” Sect. TEMS KPI Event
Service Non-Accessibility 6.1.1 FTP Upload Service Not Accessible
Setup Time 6.1.2 FTP Upload Setup Time
IP-Service Access Failure Ratio 6.1.3 FTP Upload IP Service Access Failure
IP-Service Setup Time 6.1.4 FTP Upload IP Service Setup Time
Session Time 6.1.6 FTP Upload Data Transfer Time
Mean Data Rate 6.1.7 FTP Upload Mean Data Rate
Data Transfer Cut-off Ratio 6.1.8 FTP Upload Data Transfer Cutoff

Note: “Method A” and “Method B” values are differentiated wherever applicable.

www.Infovista.com | 6
3.2.1 Timed Measurements
For timed FTP uploads, the end trigger is defined as “Reception of ACK for last packet containing content”. This
trigger is not ETSI compliant: ETSI 102 250-2 (section 6.1.6.3) defines the trigger as “Stop: Reception of the [FIN,
ACK] for the last data packet containing content”. That is, the “FIN” indicator has been removed from the
condition. This is because the FIN indicator is sent during connection teardown, a procedure which does not
necessarily take place during a timed measurement.

3.2.2 SFTP (Secure Shell FTP)

If SFTP is used, no KPIs are obtained; compare section 3.1.2.

3.3 KPIs for HTTP Get/Download

This section deals primarily with HTTP downloads conducted with the TEMS browser. The same KPI values apply
also for Internet Explorer browser usage.

ETSI-to-TEMS Mapping Table

ETSI KPI “HTTP” Sect. TEMS KPI Event
Service Non-Accessibility 6.8.1 HTTP Service Not Accessible
Setup Time 6.8.2 HTTP Setup Time
IP-Service Access Failure Ratio 6.8.3 HTTP IP Service Access Failure
IP-Service Setup Time 6.8.4 HTTP IP Service Setup Time
Session Time 6.8.6 HTTP Data Transfer Time
Mean Data Rate 6.8.7 HTTP Mean Data Rate
Data Transfer Cut-off Ratio 6.8.8 HTTP Data Transfer Cutoff

Note: “Method A” and “Method B” values are differentiated wherever applicable.

3.3.1 Timed Measurements

There is no difference between timed and complete measurements; the same triggers can be used in both
scenarios.

3.4 KPIs for HTTP Post/Upload

ETSI does not define KPIs for HTTP Post/Upload. The TEMS product KPIs have been designed to closely
resemble the ETSI KPIs for other services, such as HTTP Get and FTP (compare sections 3.1–3.3).

www.Infovista.com | 7
 ETSI-to-TEMS Mapping Table
ETSI KPI “HTTP” (Get) Sect. TEMS KPI Event
Service Non-Accessibility 6.8.1 HTTP Service Not Accessible
Setup Time 6.8.2 HTTP Setup Time
IP-Service Access Failure Ratio 6.8.3 HTTP Post IP Service Access Failure
IP-Service Setup Time 6.8.4 HTTP Post IP Service Setup Time
Session Time 6.8.6 HTTP Post Data Transfer Time
Mean Data Rate 6.8.7 HTTP Post Mean Data Rate
Data Transfer Cut-off Ratio 6.8.8 HTTP Post Data Transfer Cutoff

Trigger points for the HTTP Post session are as follows:

• Start: First packet containing content.

• End: Last incoming ACK packet for content.

No “Method A” vs. “Method B” distinction exists for this service.

3.4.1 Timed Measurements

Since ETSI does not define any KPIs for HTTP Post, the session end trigger was defined in such a way as to
work for both timed and complete measurements.

3.5 KPIs for Ping

ETSI-to-TEMS Mapping Table
ETSI KPI “Ping” Sect. TEMS KPI Event
Round trip time 6.3.1 Ping Roundtrip Time

Trigger points for Ping round trip:

• Start: ICMP Echo Request sent.

• End: ICMP Echo Reply received by the sender.

The round-trip time is that calculated by the Windows API function IcmpSendEcho2Ex. This is the lowest-latency
way available in Windows to send Ping requests and capture replies. Reference:

msdn.microsoft.com/en-us/library/windows/desktop/aa366050(v=vs.85).aspx

The difference between the round-trip time measured by Windows and that derived from the network trace is in
the range ±0.5 ms. Since the round-trip time is reported in milliseconds in the KPI, the difference after rounding is
not noticeable.

TEMS products also define a timeout error message “Ping Timeout” signifying a failed Ping request.

3.5.1 Timed Measurements

Not applicable.

www.Infovista.com | 8
3.6 KPIs for CS Voice
ETSI-to-TEMS Mapping Table
ETSI KPI: “Telephony” Sect. TEMS Data (see also below)
Service Non-Accessibility 6.6.1 Blocked Call event
Setup Time 6.6.2 Call Setup event with call setup time
Speech Quality on Sample Basis 6.6.4 PESQ/POLQA information elements
Cut-off Call Ratio 6.6.5 Dropped Call event

For CS voice, no special KPI events are generated. However, call setup time is carried as extra information by the
Call Setup event, and speech quality scores are reported in information elements belonging to the “Media Quality”
category. Supported speech quality measures are PESQ, POLQA NB and POLQA SWB.

TEMS products also have an additional, non-ETSI speech quality measure called Speech Quality Index (SQI).

CS voice KPIs are computed for mobile-originated (MO) as well as mobile-terminated (MT) calls.

www.Infovista.com | 9
4 Notes on Other Services: Email, Iperf, Streaming, CS Fallback
For the following services, TEMS products do not have any KPIs defined. However, they do exhibit certain
information elements and/or events with similar content.

Average throughputs and all percentages apply to the period following the latest network connect (triggering of
Network Connect event) and are reset at network disconnect (when the Network Disconnect activity has
completed).

4.1 Email
IE Name Range/Unit Description
Email Receive 0 ... 350000
Average throughput for receiving of email.
Average Throughput kbit/s
0 ... 172800
Email Receive Elapsed time for current email receive session. (Not
s
Transfer Time an average.)
(= 48 h)
Email Send Average 0 ... 350000
Average throughput for sending of email.
Throughput kbit/s
Email Send Transfer 0 ... 172800 Elapsed time for current email send session. (Not an
Time s average.)

4.2 Iperf
Iperf testing is conducted with the Network Bandwidth activity in Service Control scripts. The testing can be done
over either TCP or UDP.

4.2.1 TCP

IE Name Range/Unit Description

TCP Download
0 ... 350000
Average Throughput Average throughput for TCP download.
kbit/s
(kbit/s)
TCP Download 0 ... 172800 Elapsed time for current TCP download session.
Transfer Time s (Not an average.)
Retransmissions (in percent) on the downlink over
the TCP protocol during the last second. (Not an
0 ... 100 average.)
TCP Packet Loss
% Note: For this percentage to be correct, IP sniffing
must not be set to “Optimized Performance”. See
section 2.4.
TCP Upload Average 0 ... 350000
Average throughput for TCP upload.
Throughput (kbit/s) kbit/s
TCP Upload Transfer 0 ... 172800 Elapsed time for current TCP upload session. (Not
Time s an average.)

www.Infovista.com | 10
4.2.2 UDP

IE Name Range/Unit Description

UDP Download
0 ... 350000
Average Throughput Average throughput for UDP download.
kbit/s
(kbit/s)
UDP jitter: the mean deviation (in ms) of the
UDP Download Jitter Text difference in packet spacing at the receiver
compared to the sender, for a pair of packets.
Retransmissions (in percent) on the downlink over
UDP Download 0 ... 100
the UDP protocol. This measurement is obtained
Packet Loss %
once every second. (Not an average.)
UDP Download 0 ... 172800 Elapsed time for current UDP download session.
Transfer Time s (Not an average.)

UDP Upload Average 0 ... 350000

Average throughput for UDP upload.
Throughput (kbit/s) kbit/s

UDP Upload Transfer 0 ... 172800 Elapsed time for current UDP upload session. (Not
Time s an average.)

4.3 Streaming over HTTP

4.3.1 Streaming Player Download

These information elements relate to the task of downloading the streaming player.

Please note that not all of the “Streaming Player” elements are obtained if the streaming server is an HTTPS
server, since the relevant packets are then encrypted.

IE Name Range/Unit Description

Streaming Player Percentage of attempts to download the streaming
0 ... 100
Service IP Access player that failed before the first packet of the
%
Failure Ratio download was received (IP access failure).
Streaming Player Time taken for IP access when downloading the
0 ... 172800
Service IP Access streaming player (from user request to receipt of first
s
Time packet).
Percentage of attempts to download the streaming
Streaming Player
0 ... 100 player that failed at the data transfer stage (after
Download Data
% receipt of first packet and before receipt of last
Transfer Failure Ratio
packet).
Streaming Player
0 ... 172800 Time taken to download the streaming player (from
Download Data
s receipt of first packet to receipt of last packet).
Transfer Time
Percentage of attempts to download the streaming
Streaming Player 0 ... 100
player that failed at some stage (IP access or data
Session Failure Ratio %
transfer).
Streaming Player 0 ... 172800 Total session time for streaming player download
Session Time s (from user request to receipt of last packet).

www.Infovista.com | 11
4.3.2 Start of Streaming Video Replay

IE Name Range/Unit Description

Streaming Service 0 ... 172800 The duration of a service access from requesting the
Access Time s stream at the portal until the reception of the first
stream data packet.
Streaming Service 0 ... 100 Percentage of attempts to request streaming that
Non Accessibility % failed (never receive response from streaming host).
Streaming The duration of the delay after streaming request
0 ... 100
Reproduction Start was responded and before the video replay was
%
Delay started.
Streaming Indicate that there is error occurs after streaming
Reproduction Start Text request was responded and before the video replay
Failure was started.
Streaming Video Play 0 ... 100 Percentage of attempts to start streaming video
Start Failure Ratio % replay that failed (playing never started).
Streaming Video Play 0 ... 172800 Time from user video replay request until playing
Start Time s started.
Streaming
0 ... 100 Percentage of failures throughout the video playing
Reproduction Cut off
% session.
Ratio

4.3.3 Streaming Video Session

Overall

IE Name Range/Unit Description

Percentage of streaming video replay session that
Streaming Video 0 ... 100
failed at some point before the last video packet was
Session Failure Ratio %
received.
Streaming Video 0 ... 172800 Total time of streaming video replay session from
Session Time s user request to receipt of last video packet.
VQmon Metrics: Video and Audio Perceptual Quality

IE Name Range/Unit Description

Streaming Absolute 0 ... 5
Average absolute video MOS for the stream.
MOS-V MOS
Streaming 0 ... 5
Average audio MOS for the stream.
MOS-A MOS

Streaming 0 ... 5
Average audio–video MOS for the stream.
MOS-AV MOS
Average relative video MOS for the stream. The
difference between relative and absolute video MOS
Streaming Relative 0 ... 5
is that the relative metric does not consider frame
MOS-V MOS
resolution, thus constituting a score which is relative
to the ideal for the current video format.
These average scores are also reported in a “Streaming Quality MOS” event at the end of a streaming session.

www.Infovista.com | 12
Bandwidth

IE Name Range/Unit Description

Streaming Average
0 … 350000 Average audio bandwidth excluding transport packet
Audio Received
kbit/s overhead, FEC, and retransmissions.
Bandwidth
Streaming Average
0 … 350000 Average bandwidth for video transport packets,
Video Received
kbit/s excluding IP overhead, FEC, and retransmissions.
Bandwidth

Packet Loss, Packet Errors

IE Name Range/Unit Description

Streaming Average
0 ... 100 Percentage of audio packets lost or discarded
Audio Effective Packet
% (minus those corrected).
Loss Rate
Streaming Average
0 ... 100 Percentage of video packets lost or discarded
Video Effective Packet
% (minus those corrected).
Loss Rate
Streaming Audio 0 ... 100 Percentage of audio packets received with errors,
Packet Corrected Rate % but corrected using error correction algorithms.
Streaming Audio
0 ... 100 Percentage of audio packets discarded by the
Packet Discarded
% receiving jitter buffer.
Rate
Streaming Audio Percentage of audio packets identified as out-of-
0 ... 100
Packet Out Of sequence, possibly due to high jitter levels or the
%
Sequence Rate use of load-sharing devices.
Streaming Video 0 ... 100 Percentage of video packets received with errors,
Packet Corrected Rate % but corrected using error correction algorithms.
Streaming Video
0 ... 100 Percentage of video packets discarded by the
Packet Discarded
% receiving jitter buffer.
Rate
Streaming Video Percentage of video packets identified as out-of-
0 ... 100
Packet Out Of sequence, possibly due to high jitter levels or the
%
Sequence Rate use of load-sharing devices.

www.Infovista.com | 13
Buffering and Rebuffering

IE Name Range/Unit Description

Streaming Video Number of times the video stream replay was
Text
Interruption Count interrupted for rebuffering.
Streaming Video 0 ... 172800 Total duration of video stream replay interruptions for
Interruption Duration s reasons of rebuffering.
Streaming Playout 0 ... 350000 Equal to Streaming Playout Buffer Empty Rate
Buffer Delta Rate kbit/s minus Streaming Playout Buffer Fill Rate.
Streaming Playout
0 ... 100 Percentage of the total session time during which
Buffer Empty
% the playout buffer was empty.
Proportion
Streaming Playout 0 ... 350000 Rate at which data is emptied from the streaming
Buffer Empty Rate kbit/s player’s playout buffer.
Streaming Playout 0 ... 350000 Rate at which the streaming player’s playout buffer is
Buffer Fill Rate kbit/s filled with new data.
Streaming Playout
0 ... 100 Percentage of the total session time that was spent
Buffer Rebuffering
% rebuffering.
Proportion

www.Infovista.com | 14
4.4 CS Fallback
The following events report on various aspects on CS fallback procedures:

Event Name Description

A CS fallback call was blocked. This can happen in several ways:

• The Extended Service Request timed out without any response
from network, or the network responded with Service Reject. In
this case the CS fallback procedure never reaches the RAT
change stage.

• RAT change to UTRAN/GERAN failed, as indicated by the event

CSFB Blocked Call EUTRAN RRC Connection Release Redirected Failure.

• After successful RAT change to UTRAN/GERAN, the CS call

setup failed (CS Blocked Call event generated).

Extra information:
• Call direction (MO/MT)

• Block type

A CS fallback call was set up. Triggered by the Layer 3 message

Alerting.
Extra information:
• Call direction (MO/MT)

• Target technology (WCDMA/GSM)

CSFB Call Setup • Call setup time (measured from CSFB Call Attempt, i.e. the first
call attempt)

• User setup time (measured from CSFB Call Initiation, thus more
accurately reflecting the user-perceived setup time. Obtained
only for MO calls in scripts. If no CSFB Call Initiation event was
generated, the user setup time cannot be computed.)

This event carries performance information on the switch back to

LTE after hangup of a CS fallback call.
Extra information:

EUTRAN Reselection Time • Idle to LTE time: Time in seconds from entering idle mode (in
After CSFB Call UMTS) to reception of System Information Block on LTE.

• SIB 19 to LTE time: Time in seconds from reception of System

Information Block Type 19 (UMTS) to reception of System
Information Block on LTE.

This event reports the IP interruption time during RAT change to

UTRAN/GERAN due to initiation of a CS fallback call.
PS Data Interruption Time
Due To CSFB Extra information: Interruption time in ms. Measured from last
received IP packet in EUTRAN to first received IP packet in
UTRAN/GERAN.

www.Infovista.com | 15