Towards Quality and QOS Assessment For VoLTE - IMSWF

TOWARDS QUALITY AND QOS
ASSESSMENT FOR VOLTE IN ETSI

TC INT chairman Giulio Maggiore - IMS World Forum 30th April 2014
ETSI 2013. All rights reserved
Outline
ETSI Technical Committee TC
INT for Core Network and
Interoperability Testing
ETSI INT technical work
specifications
Quality and QOS Assessment
for VoLTE in ETSI
Whats next?
CORE NETWORK AND

INTEROPERABILITY TESTING
(INT)
3
TC INT (Core Network and

Interoperability Testing)
Member of EP E2NA
In 2012 TC INT participated in the creation of the ETSI
E2NA project contributing with its expertise in network
testing as described in the E2NA ToRs approved by the
Board#89.
TC INT change of ToRs and name

According to E2NA ToRs approved by Board#89
The modification of the ToRs in order to align its activities
and contributions with EP E2NA.
Change of name with Core Network and
Interoperability Testing (INT) to better reflect the
areas of activity.
No change of acronym that will remain INT
ETSI/BOARD(13)94_041
ETSI INT SPECIFICATIONS

Interoperability , Performance , Security and Conformance
Production of test specifications

Production of test specifications:
(11) Interoperability
(4) IMS/NGN Performance Benchmark
(1) Security
Production of IMS Conformance:

(32) Test Suite Structure and Test Purposes
(TSS&TP)
(24) Abstract Test Suite (ATS)
(24) Protocol Information Conformance
Statement (PICS)
(24) Protocol Implementation eXtra
Information for Testing (PIXIT)
INT full work programme

QUALITY AND QOS ASSESSMENT

ETSI APPROACH
7
Quality: the new approach
Quality is the most important factor in client choice and churn.
In order to offer the best user experience its necessary to shift in the quality
monitoring: from a Network Centric point of view to Customer Centric
Network Centric
Customer Centric
Core
APN
APN
Access
Core
APN
Access
Customer Centric monitoring requires new

KPIs
Moving from the network to the user centric point of view a new set of
KPIs are required able to analyze the experience of the single user
In order to monitor the e2e network chain the KPIs have to be

Technical KPI
technology/access
Service KPI independent
bricks
of service KPI
monitor specific network elements
Immediate overview of the service
Critical Indicators
Localized
Access layer
Operative System
Device
Network section
Core
9
10
2013 VOLTE QOS ASSESSMENT

TECHNOLOGY EVALUATION
EVENT
VoLTE QoS Assessment event overview

Scope: Technology evaluation Plugtests aiming at
validating the different methodologies for end to
end QoS Assessment for VoLTE described in ETSI
TS 103 189 over a deployed LTE/IMS network.
Dates: 18-20 November 2013
Venue: ETSI, Sophia Antipolis, FR
Web: http://www.etsi.org/news-events/events/698-volte-qosassessment-2013
Participants:
Supporter:
11
VoLTE QoS Assessment event scope

End to end tests performed with several different UEs
over a deployed LTE / IMS test network.
4 Test Setups
Acoustic to acoustic
Acoustic to electrical
Electrical to acoustic
Electrical to electrical
Voice quality tests based on instrumental assessment
of speech samples performed on different
combinations of the acoustic and electrical interfaces
which allows evaluating the different methodologies
and setups.
Parallel network performance evaluation
12
2013 VoLTE QoS Assessment Technology

Evaluation event test description TS 103 189
Sub-parameters delivered by the

particular tests:
End-to-end Tests based on Instrumental Assessment
of Speech Samples (POLQA model deployed):
Mean Opinion Score (MOS)-Listening Quality Objective

Delay variation over time
Difference in loudness level
MOS stability
End-to-end Tests based on Instrumental Assessment

of Video Samples (PEVQ model deployed):
DMOS (Difference Mean Opinion Score)

Delay and Quality variation over time
Jerkiness and Blockiness
Blur, Chrominance and Luminance
Temporal Distortion
Frozen and Skipped Frames

Evaluation event test description TS 103 189
Sub-parameters delivered by the

particular tests:
Other end-to-end tests in the voice channel:
End-to-end frequency response

Overall loudness rating
End-to-end delay
Quality of echo cancellation
Double talk performance
End-to-end Tests based on functional parameters:

Telephony Call Setup Time
Streaming Service Access Time
Network Performance Parameters

UNI-to-UNI delay over time
UNI-to-UNI jitter
UNI-to-UNI packet loss

Evaluation event test description
General description of tests defined in ETSI TS
103 189:
QoS tests based on the ETSI INT specifications as
described in the latest versions of:
ETSI
ETSI
ETSI
ETSI
TS
TS
TS
TS
186
186
102
103
011-1 (Test purposes for IMS NNI Interoperability)

011-2 (Test descriptions for IMS NNI Interoperability)
901 (IMS NNI interoperability test descriptions for RCS)
029 (IMS & EPC Interoperability).
QoS tests divided into two parts:

First one focusing on the instrumental assessment of speech
15
and video quality by using real speech and video samples

Second one dealing with functional parameters of voice and
video over LTE connections and other parameters in
the end-to-end voice channel
On the top of it, selected network performance parameters
listed in the TS 103 189
16
VoLTE QoS Assessment Setup 1

Acoustic-to-Acoustic
ITU-T Specifications for Quality

measurements
Recommendation ITU-T J.247: "Objective perceptual multimedia video quality
measurement in the presence of a full reference".
Recommendation ITU-T P.50: "Artificial voices".
Recommendation ITU-T P.57: "Artificial ears".
Recommendation ITU-T P.58: "Head and torso simulator for telephonometry".
Recommendation ITU-T P.64: "Determination of sensitivity/frequency characteristics of
local telephone systems".
Recommendation ITU-T P.79: "Calculation of loudness ratings for telephone sets".
Recommendation ITU-T P.340: "Transmission characteristics and speech quality
parameters of hands-free terminals".
Recommendation ITU-T P.380: "Electro-acoustic measurements on headsets".
Recommendation ITU-T P.501: "Test signals for use in telephonometry".
Recommendation ITU-T P.505: "One-view visualization of speech quality measurement
results".
Recommendation ITU-T P.502: "Objective test methods for speech communication
systems using complex test signals".
Recommendation ITU-T P.581: "Use of head and torso simulator (HATS) for hands-free
terminal testing".
Recommendation ITU-T P.800.1: "Mean Opinion Score (MOS) terminology".

Acoustic to Electrical
18

Electrical to Acoustic
19

Electrical to Electrical
20
21
MAIN OUTCOME OF THE VOLTE

QOS ASSESSMENT AND NEXT
STEPS
Main event outcome

Validation of 76% of the test cases
in TS 103189
A number of gaps and potential
improvements in TS 103189 have
been identified
A new work item has been open for
the update of TS 103 189 with the
new setup, acoustic-to-electrical
identified during the event
TS 103 189 has been published and
is available on the ETSI portal
22
Further steps with regard to ETSI TS

103 189
Verification of remaining voice quality test
cases
tbd
Verification of video quality test cases
(RTP-based video streaming):
Test houses willing to do so are needed
Extension of video quality test cases
towards MPEG-DASH
ITU-T SG12 Question 14 currently running a WI
called P.NATS
23
Maputo - Mozambique - 14 - 16 April 2014

Evaluation event test results
24

Evaluation event test results
25
Verification of video quality test cases

(RTP-based video streaming)
What we are looking for?
What are the parameter we want to
measure?
What are the tools expected?
Parameters
Covered by
TS 183.189
26
test tools
focused on
video quality
Prototipization
and validation
Customers perception KPIs
VoLTE
Browsing
Non Accessibility
Session Setup Failure
Setup time
Cut off Ratio
Packet loss rate
Latency
Jitter
MOS Audio/Video Quality
Number of HTTP
Sessions
Average Duration of
HTTP Sessions
Average HTTP
Throughput
Peak HTTP
Throughput
HTTP Setup Time
HTTP Dropped
Connections
HTTP Sessions
Unavailability
Retransmission Rate
Packet loss rate
Latency
Jitter
Loading Page
27
Gaming
Streaming
(HTTP YouTube sessions included)
Number of Gaming
Sessions
Gaming Sessions
Average Duration
Gaming Average
Throughput (DL & UL)
Gaming Overall
responsiviness [ms]
Retransmission Rate
Packet loss rate
Latency
Jitter
Number of streaming sessions
Average duration of streaming
sessions
Average Streaming Throughput
Peak Streaming Throughput
Setup Streaming Time
Streaming Dropped Sessions
Streaming Unavailability
Re-Buffering Time during Streaming
Streaming Synchronization
Packet loss rate
Latency
Jitter
MOS Audio/Video Quality
Networks KPIs/KQIs
09/09/15
Zhilabs 2014, Confidential Information
28
Next steps
Get comments on TS 103
189
Extend TS 103 189
addiming more Customer
Perception KPIs
Get proposal from Video
Test Houses
Next event will be
launched when VoLTE
Native Handset and PEVQ
tools will be available for
testing
29
Contact Details:
Giulio Maggiore
Telecom Italia
ETSI TC INT Chairman
Giulio.Maggiore@telecomitalia.it
Thank you!
30
TC INT TERMS OF REFERENCE

31
TC INT (Core Network and

Interoperability Testing) Terms of
Reference
The scope of INT includes:
Development of Core Network test
specifications (interoperability,
conformance, performance, security)
based on 3GPP and E2NA specifications.
To initiate and supervise interoperability
events (such as Plugtests) related to Core
Networks as well as other events
(workshops and seminars)
To coordinates interoperability efforts with
other organisations GSMA, OMA, IMS/NGN
Forum, IMTC, etc.
Endorsement of test specifications from/to
other SDOs e.g. ITU-T.
32
ETSI INT SPECIFICATIONS

Interoperability , Performance , Security and Conformance
33
TC INT Interoperability Specification

Work
R6
TS 186 011
V 1.1.1
2007
R1
34
R7
TS 186 011
V 2.1.1
Late R7
TS 186 011
V 2.2.1
2008
2009
R2
Late R2
R8
TS 186 011
V 3.1.1
TS 102 901
V 1.1.1
2010
R9
R10
TS 186 011
V 4.1.1
TS 102 901
V 2.1.1
TS 186 011
V 5.0.2
TS 102 901
V 5.0.6
TS 103 129
TS 103 129
V2.0.4
2011
2012 2013
E2NA
Conformance Test Specifications

IMS Basic Call Specifications and Supplementary
Services;
Diameter Conformance testing for Rx interface TS
101 580;
Diameter Conformance testing for Gx interface TS
101 606;
Diameter Conformance testing for S6a and S9
interface just started;
35
Interoperability Test Specifications

IMS NNI Interworking
Test Specifications TS
186 011
RCS TS 102 901
IMS&EPC
interoperability TS
103 129
36
IMS Interworking
Test Specifications
IMS Interconnection
Test Specifications
INTEROPERABILITY EVENTS
37
Interoperability events
2007 The first event examined interconnection aspects of
3GPP IMS Release 6, including basic call on the Mw interface.
2008 - The second event had a wider scope that included the
testing of 3GPP IMS Release 7 interworking, roaming, border
control, and integration of application servers executing
selected Multimedia Telephony supplementary services.
2009 - The third ETSI IMS event assessed the interoperability
as well as conformance of IMS core networks (composed of
P/I/S-CSCF, IBCF, AS, DNS and HSS) which are implemented on
the basis of ETSI TS 124 229/3GPP TS 24.229 (Version 7.14.0)
(3GPP late release 7).
2012 RCS&VoLTE interoperability event in cooperation with
GSMA and MSF held in Kranj (Slovenia) and Beijng (China)
38
Press Coverage: RCS VoLTE Test Event
39
2007 and 2008 IMS interoperability

events
The first event examined
interconnection aspects of 3GPP IMS
Release 6, including basic call on the
Mw interface.
The second event had a wider scope

that included the testing of 3GPP IMS
Release 7 interworking, roaming,
border control, and integration of
application servers executing selected
Multimedia Telephony supplementary
services.
40
2009 - IMS interoperability events

The third ETSI IMS event assessed the
interoperability as well as conformance of
IMS core networks (composed of P/I/SCSCF, IBCF, AS, DNS and HSS) which are
implemented on the basis of ETSI TS 124
229/3GPP TS 24.229 (Version 7.14.0)
(3GPP late release 7).
41
2012 RCS & VoLTE interoperability event
The forth event endorsed and tested a

number of GSMA PRDs and validated the
Testing Scenarios developed by MSF and
ETSI to facilitate multi-vendor
deployment of LTE./EPC/IMS technology
for RCS and VoLTE services.
http://www.etsi.org/newsevents/past-events/642-2012-rcsvolte-interop-event
42

Towards Quality and QOS Assessment For VoLTE - IMSWF

Загружено:

Сведения о документе

Исходное описание:

Оригинальное название

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

Towards Quality and QOS Assessment For VoLTE - IMSWF

Загружено:

Авторское право:

Доступные форматы

TOWARDS QUALITY AND QOS

ASSESSMENT FOR VOLTE IN ETSI

ETSI 2013. All rights reserved

CORE NETWORK AND

ETSI 2013. All rights reserved

TC INT (Core Network and

TC INT change of ToRs and name

ETSI INT SPECIFICATIONS

ETSI 2013. All rights reserved

Production of test specifications

Production of IMS Conformance:

INT full work programme

QUALITY AND QOS ASSESSMENT

ETSI 2013. All rights reserved

Quality: the new approach

Quality is the most important factor in client choice and churn.

Customer Centric monitoring requires new

In order to monitor the e2e network chain the KPIs have to be

Immediate overview of the service

2013 VOLTE QOS ASSESSMENT

ETSI 2013. All rights reserved

VoLTE QoS Assessment event overview

ETSI 2013. All rights reserved

VoLTE QoS Assessment event scope

ETSI 2013. All rights reserved

2013 VoLTE QoS Assessment Technology

Sub-parameters delivered by the

Mean Opinion Score (MOS)-Listening Quality Objective

End-to-end Tests based on Instrumental Assessment

DMOS (Difference Mean Opinion Score)

2013 VoLTE QoS Assessment Technology

Sub-parameters delivered by the

End-to-end frequency response

End-to-end Tests based on functional parameters:

Network Performance Parameters

2013 VoLTE QoS Assessment Technology

011-1 (Test purposes for IMS NNI Interoperability)

QoS tests divided into two parts:

and video quality by using real speech and video samples

VoLTE QoS Assessment Setup 1

ETSI 2013. All rights reserved

ITU-T Specifications for Quality

VoLTE QoS Assessment Setup 2

ETSI 2013. All rights reserved

VoLTE QoS Assessment Setup 3

ETSI 2013. All rights reserved

VoLTE QoS Assessment Setup 4

ETSI 2013. All rights reserved

MAIN OUTCOME OF THE VOLTE

ETSI 2013. All rights reserved

Main event outcome

ETSI 2013. All rights reserved

Further steps with regard to ETSI TS

Maputo - Mozambique - 14 - 16 April 2014

2013 VoLTE QoS Assessment Technology

ETSI 2013. All rights reserved

2013 VoLTE QoS Assessment Technology

ETSI 2013. All rights reserved

Verification of video quality test cases

ETSI 2012. All rights reserved

Customers perception KPIs

ETSI 2012. All rights reserved