See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/311532272

Quality of Experience analysis for VoLTE services through Artificial Neural

Network fitting

Conference Paper · October 2016

DOI: 10.1109/WINCOM.2016.7777208

CITATION READS

1 176

2 authors:

Alessandro Vizzarri Fabrizio A. M. Davide

University of Rome Tor Vergata Università Telematica Guglielmo Marconi
13 PUBLICATIONS   41 CITATIONS    111 PUBLICATIONS   2,522 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

CROSS crossprogramme.eu View project

PASION View project

All content following this page was uploaded by Alessandro Vizzarri on 31 August 2017.

The user has requested enhancement of the downloaded file.

 Quality of Experience Analysis for VoLTE Services
through Artificial Neural Network Fitting
Alessandro Vizzarri
Department of Enterprise Engineering
University of Rome Tor Vergata
Rome, Italy
alessandro.vizzarri@uniroma2.it
Abstract —We consider Quality of Experience (QoE),
measured in terms of Mean Opinion Score Listening Quality
Subjective (MOS_LQS) for a Voice over LTE (VoLTE)
application in realistic situations. A set of service scenarios have
been identified and the network performance simulated. We
organized a listening panel to measure MOS according to the
standard procedures. MOS-LQS results and QoS metrics have
been correlated using a set of Artificial Neural Network (ANN)
models. As a clear result the ANN accurately models the
relationship. This confirms again the need for an approach that is
non linear and capable of generalization, as made by the human
judgment. Future research is necessary to generalize the ANN
model to a far larger scale while controlling the combinatorial
explosion of scenarios and technical parameters.
Keywords—LTE; VoIP; VoLTE; end-to-end QoS; QoE; LTE
KPIs; IP cloud; ANN; MOS; MOS-LQS.
All the authors contributed equally to this work.
I. INTRODUCTION
Voice over LTE (VoLTE) is a key service for Long Term
Evolution (LTE) networks, and needs specific attention for
quality of service (QoS). End-to-end approach for QoS is
strongly recommended by the standard not only for data-based
services but also for delay sensitive services VoLTE [1].
MNOs analyze and monitor continuously the main network
parameters (Key Performance Indicators, KPI) at network level
and try to estimate Quality of Experience (QoE) perceived by
end user at the application level.
The present work considers QoE measured in terms of
Mean Opinion Score Listening Quality Subjective for a VoLTE
application in realistic situations. We are interested in studying
the behavior for MOS (as QoE parameter) versus the network
KPIs (i.e. QoS parameters). Research presents extensive
samples of LTE networks modelled in realistic usage
conditions through scenario making. We propose the approach
of modelling multiuser and multiservice scenarios according to
reported conditions, and simulate the network performance
using the OPNET simulation software (as described in Sect. 2).
Many authors applied polynomial, exponential and logarithmic
functions to fit the objective MOS value (see references in [2]).
Then we attacked the subjective measurements for MOS. This
required to invest a certain amount of resources to train and
employ a panel of listeners for an extensive series of
Fabrizio Davide
Department of Engineering of Innovation
University of G. Marconi
Rome, Italy
f.davide@unimarconi.it
experiments, reported in Sect. 4. In Sect. 5 QoS metrics have
been used as predictors, together with a feed-forward Neural
Network compared with a linear multi-regression technique.
This is motivated by a shared evidence of nonlinear behavior
for MOS versus network parameters in realistic scenarios. Sect.
6 will draft the most relevant conclusions out of the present
approach.
II. QOE ANALYSIS FOR A VOLTE APPLICATION
Since LTE is fully IP-based wireless standard it only
enables entire wireless transmission over Packet Switching (PS)
paths using the Internet Protocol (IP) protocol as the network
protocol. All applications delivered over LTE systems are IP-
based, including voice applications as Voice Over LTE
(VoLTE) [3], [4], [5]. VoLTE is delivered in best effort
modality. This implies VoLTE, as a best effort service, requires
an effective service management in order to guarantee both an
acceptable end-to-end QoS at network layer and an acceptable
QoE perceived by end-users. Vizzarri in [6] presents a review
of the most important papers focused on the end-to-end QoS in
case of applications delivered over LTE networks. Reference
[7] analyses the QoS for a VoLTE service through and end-to-
end approach. Reference [8] studies the impact of network
congestions on the VoLTE end-to-end performance. Major
standardization entities already treated the QoS issue in LTE
through an end-to-end approach. In [9] ETSI provides end-to-
end QoS reference architecture for LTE and a description of
relevant management functions. Implementation of QoS
policies and strategies are left to MNOs. QoE of a voice
application is usually represented by the Mean Opinion Score
(MOS). MOS is a scalar variable, whose values are limited in
the range from 1 (worst case) to 5 (best case) [10], which
indicates an average level of service acceptance for the end
user. The MOS value is strictly related to the R factor provided
by the ITU E-Model [11]. The E-Model combines a number of
different impairments to be considered for an overall quality
measure.
There are various different MOS measures, as shown in Fig.
1. We can distinguish objective measurement that rely on
estimates of conversational and listening quality, based on
technical measures and subjective measurements that rely on
collaboration of human subjects. In case measures are
performed before the receiver we speak of MOS
Conversational Quality Estimated (MOS-CQE) and MOS
Listening Quality Estimated (MOS-LQE). In case measures are
performed after the receiver we speak of MOS Listening
Quality Objective (MOS-LQO) and MOS Listening Quality
Subjective (MOS-LQS), respectively objective and subjective.
In the present paper we are interested in subjective
measurement of MOS.
Fig. 1. MOS and voice quality estimation schemes for VoIP Services.
Available methods can be grouped in three classes. The
“Conversational Opinion Test” requires a preparation in the
laboratory in terms of creation of impairments in the VoLTE
testbed system such as loss, delay and echo. Each test requires
two subjects who have to sit in separate sound-proof rooms and
then talk to each other before finally rating the individual score,
using 5-point scale. Also, each condition requires participants
at least 24-32 subjects to test. The final result is the MOS
Conversational Quality Subjective (a variant of MOS-LQS
shown on Fig. 1). The “LQS Test” requires a laboratory too
[12]. Preparation is somehow more complicate, because
participants are taken individually and provided each with
audio files that is the result of a high fidelity recording
transmitted along the VoLTE service chain, including the
receiver. Each participant has to give the individual score,
using the usual 5-point scale. Measurement of each scenario
should be conducted with at least 16 subjects [13].
In the following we will focus on the listening test MOS-
LQS. We based on a transmission test carried out with pre-
recorded phrases; the goal of this test is to obtain the absolute
quality of the voice sample after transmission, through the
direct hearing of the sample, without a reference sample. There
are various listening tests, i.e. Absolute Category Rating
(ACR), Degradation Category Rating (DCR) and Comparison
Category Rating (CCR) [14]. The quality must be evaluated
through different opinion scales: Listening quality scale,
Listening-effort scale and Loudness-preference scale. After
collection of each participant’s opinion of quality (expressed in
the usual range), the MOS is computed as the mean of the test
panel. The ITU-T Rec. P.800 (1996) defined the requirements a
MOS test has to comply with [15].
The final objective of our research is to correlate the MOS-
LQS with the QoS metrics of the VoLTE applications. Here we
adopt as the QoS metrics relevant for a VoLTE service three
Key Performance Indicators (KPIs): end-to-end Delay, Packet
Loss rate (PLR) and Jitter (other quality of service measures for
a generic VoIP application can be found in [16]). Delay is
expressed as the amount of time a packet sent by the source
(caller) takes to reach destination (callee). An excess of delay
can make an audio conversation very difficult: both caller and
the callee might start to talk at the same moment or interrupt
each other. Further an excess of delay might generate Packet
Loss because the delayed voice packet could be dropped. Jitter
is the variability in the arrival time between packets, caused by
network congestion or route changes [17]. Negative effects of
Delay and Jitter are voice echoes, while high value of Packet
Loss can produce overlapping of words with a strong negative
impact on voice intelligibility. The effects of these QoS metrics
on QoE are well known, though we are interested in correlating
the QoE with the measured QoS metrics via neural networks.
There are two set of reasons. First, neural networks are a well-
known tool to fit non-linear input-output functions, and may be
easily used in our problem. Second, neural networks were
demonstrated to have a great potential in modeling human
behavior. On consequence we are interested in exploiting this
subtler property to find a way to reduce the expensive process
of full MOS-LQS measurement.
III. RELATED WORK
The approach of the authors is to correlate the objective
QoS KPIs at network level with MOS as QoE subjective
indicator. We will apply in the following different statistical
regression techniques: Multiple Linear Regression (MLR) and
Artificial Neural Network (ANN) with different training
algorithms. In [18] the authors propose fuzzy-logic for
QoS/QoE mapping. QoS/QoE mapping is carried out
correlating Mean Opinion Score (MOS) as QoE parameter to
network KPIs as QoS parameter. In particular Delay, Jitter and
Packet Loss Rate are considered as KPIs. A comparison of
different approaches for the best fitting curve of the simulation
results is in [19] [20].
In [21] the authors conducted experimental tests in order to
evaluate the QoS of a generic VoIP service. On the basis of the
obtained test results, the authors tried to correlate the metrics
measured at network level (Packet Loss and Packet Delay) to
the corresponding MOS-CQS resulting from the subjective test.
They proceeded to fit the obtained values through a MLR and
identified a polynomial relationship. Finally they calculated the
related Mean Absolute Percent Error (MAPE).
QoE/QoS mapping can be also analyzed and modelled
through the involvement of the ANNs [22] [23], since a
desiderable output (QoE in our case) can be predicted on the
basis of multiple inputs (network QoS metrics at network
level). In [24] the authors try to correlate the QoE to QoS in
case of a mobile data service through an ANN. After training,
the ANN model gives two different types of relationship for the
QoE: direct if QoE is estimated against the bandwidth; inverse
in case of estimation of Delay. In [25] V. A. Machado et alii
defined an ANN-based network model for QoS/QoE correlation
in case of a video service delivered over a WiMAX network.
Test results confirm the goodness of the ANN model, which
exibits an acceptable error. In [26] the authors apply the ANN
to the estimation of the volume traffic matrix in a large scale [P
network. They compare different training algorithms (as
Levenberg-Marquardt and Bayesian Regularization) in terms of
robustness and accuracy in order to perform a good prediction
of traffic. Results confirm the Levenberg-Marquardt training
algorithm to have the best error robustness.
 IV. EXPERIMENTAL QCI 1 (GBR) and Allocation and Retention Priority (ARP 1).
The VoLTE application has been launched with a start offset of
A. Methodology 20s till the end of simulation period. In the HTTP application,
Our rationale is as follows: concerning scenarios from n.25 to n.48, UE_2 can download 1
∑ Define realistic scenarios for VoLTE application, to KB web page, n. 5 medium images with dimension up to 2 KB
be simulated for computing KPIs. and two short videos with dimension up to 350 KB. HTTP
∑ Perform MOS LQS testing per each scenario. application is launched with a start offset of 40s. Since modeled
∑ Attempt multi parameter fitting of QoE vs QoS KPIs profiles add a start offset around 40s, VoLTE and HTTP
through the ANN. applications start after 80s from simulation initiation.

B. Simulated scenarios
Realistic scenarios are simulated on the basis of network
impairments that disturb VoLTE calls. Impairments are
represented by:
∑ mixed traffic: VoLTE application is delivered over the
LTE network together with HTTP web browsing
application.
∑ insertion of IP cloud: additive delay and IP packet
discard ratio are produced across entire end-to-end
transmission chain.
This framework is typical when a backbone network
section is involved in service delivery. The HTTP application
is the application considered for modeling a set of multiuser a) b)
and multiservice scenarios. We made the exercise of
considering different LTE network topology (with or without
IP cloud) and traffic flows (single or mixed), as shown in Fig.
2, and identified a series of 48 scenarios.
Scenarios from n.1 to n.12 are characterized by two UEs:
UE_1 (caller) is performing a VoLTE call to UE_2 (callee)
using a direct link. Traffic flow is single. Topology is shown in
Fig. 2.a.
Scenarios from n.13 to n.24 are characterized by three UEs
and one HTTP web server: UE_1 and UE_2 are performing a
VoLTE call while UE_3 is performing an HTTP web browsing
session. Traffic flow is mixed: VoLTE and HTTP web
browsing services are performed by UEs (see Fig. 2.b). c) d)
Fig. 2. LTE network topology simulated for VoLTE application. One UE for
Scenarios from n.25 to n.36 are similar to those in the group VoLTE without IP cloud (a), two UEs for VoLTE and HTTP browsing
1-12: UE_1 is performing a VoLTE call to UE_2, but link without IP cloud (b), One UE for VoLTE with IP cloud (c), two UEs for
among them is affected by insertion of IP cloud. It adds 1% VoLTE and HTTP browsing with IP cloud (d).
packet discard ratio and 0.1s delay between caller (UE_1) and
callee (UE_2) (refer to Fig. 2.c). The simulation period is equal to 5 (3mins plus offset) for
each scenario.
Scenarios from n.37 to n.48 are similar to those in the group
13-24: UE_1 and UE_2 are performing a VoLTE call using a TABLE III. SCENARIO CONFIGURATION
direct link interrupted by IP cloud, UE_3 is performing HTTP
web session (refer to Fig. 2.d). Table III resumes the main LTE LTE S1 eNB
Scen. IP User Hop
Serv. Serv. Capacity BW
characteristics for each simulated scenario. No.
Type No.
cloud
[%]
No. No.
[MHz]
C. Simulation settings 1-12 VoLTE 1 NO
100; 75;
2 2
5; 10;
50;30 20
Scenarios have been simulated using the LTE network
model provided by OPNET 17.5 PL6. The UE’s antenna gain 100; 75; 5; 10;
13-24 VoLTE 1 YES 2 3
is -1 dBi with a receiver sensitivity of -200 dBm. eNodeB uses 50;30 20
10 MHz LTE bandwidths and FDD Duplex Mode. Link among
LTE network nodes is of type PPP D3, with a data rate of VoLTE 100; 75;
44.736 Mbps. The simulation area is a typical campus area 25-36 2 NO 3 2 5; 10; 20
+ HTTP 50;30
(100 Km-square wide). We employed the Voice Codec GSM
EFR and one voice frame per packet. As per the LTE standard,
the VoLTE application is carried out over the EPS bearer with
 scenario number) versus each of the KPIs (which show abrupt
VoLTE 100; 75; 5; 10; changes among contiguous scenarios).
37-48 2 YES 3 3
+ HTTP 50;30 20

D. Experimental Setup
The experimental setting is based on the OPNET simulation
tool configured in the System-in-the-loop (SITL) modality.

Fig. 4. MOS-LQS (leftmost) and KPIs (rightmost) represented along the

series of scenarios. On the x-axis there is the scenario order number.

V. ANALYSIS OF RESULTS
A. Multi Linear Regression
In [2] we studied the relationship between QoS KPIs and a
MOS, intended as a QoE metrics, fitting the linear model with
bias shown in (1) :
Fig. 3. The experimental testbed.

OPNET SITL module is installed on a fixed PC, shown in = + ∗ DEL + ∗ + ∗ + (1)

the middle of Fig. 3. This provides the emulation of the LTE
network with two different UEs, e.g. the Sender UE (caller, on where β0, β1, β2 and β3 are the unknown regression
the left) and the Receiver UE (callee, on the right), to perform coefficients and εi ~ (N,σi2) is the error for each observation.
an unidirectional VoLTE call. The two UEs are emulated by In [2] we used the same data as in Table IV and got results
two laptops equipped with a single Network Interface Cards through MLR as shown in Table V.
(NICs). In this setting the Sender transmits a high quality voice
TABLE V. MULTI LINEAR REGRESSION
track, three minutes long, to the Receiver through the OPNET
SITL. The resulting voice track (RVT) is recorded at the Multi Linear Regression’s Statistical Indicators
Receiver and stored. For each scenario we prepared three
Factors β Std. Error tStat pValue
RVTs using a GSM-EFR audio codec. Finally, the authors estimated
performed an ACR listening test, employing a panel of 25 Intercept 5.7371 0.48221 11.898 2.4177e-15
participants, chosen in the age range from 20 to 50. Each panel
participant was provided with a RVT per scenario randomly DEL -15.802 4.8986 -3.2258 0.0023718
chosen out of three. At the end of each run the participant gave JIT -1.7824 3.1521 -0.56547 0.57463
a numeric score (from 1 to 5) to evaluate the quality (in terms PLR 1.5226 2.4779 0.61447 0.54207
of subjective MOS) of the listened RVT.
E. Results Statistical indicators for MLR are: Root Mean Squared
Table IV shows results we have got, in terms of the Error (RMSE) equal to 0.489; Mean Squared Error (MSE)
network QoS metrics (end-to-end Delay, Jitter and Packet Loss equal to 0.699285. Then R-squared is 0.798 and Adjusted R-
Rate) and the QoE metrics MOS-LQS. Squared 0.784. Looking at the p-values for JIT e PLR we see
the null hypothesis cannot be rejected for these variables.
TABLE IV. SIMULATION RESULTS Patterns for residuals, as shown in Fig. 5, reveal a poor fitting
quality, both for size of residuals and the asymmetric
QoS and QoE results [min; max] distribution around their median.
Packet Loss
Scen. Delay Jitter MOS-LQS B. Neural Network Fitting
Rate
No. [s] [s] [1-5]
[%]
We selected a 3-10-1 feed-forward network: 3 linear input
[0.11; [0.06; [0.20; [3.10;
1-12
0.12] 0.11] 0.50] 4.55] neurons, 10 sigmoid neurons in the hidden layer and a linear
[0.13; [0.09; [1.05; [3.39; output neuron. For the training process, we adopted 34 samples
13-24
0.13] 0.13] 1.50] 3.96] (70% of the data set) as the training set, 7 samples (15% of the
25-36
[0.19; [0.15; [7.10; [1.95; whole data set) as the validation set, 7 samples (15% of the
0.21] 0.19] 9.23] 2.71] whole data set) as the test set. The choice of only ten hidden
[0.26; [0.21; [19.92; [1.03;
37-48
0.27] 0.27] 26.06] 1.80]
neurons is a guarantee against overfitting. Then we employed
some training strategies as in the following.
Fig. 4 makes evident at first inspection a fairly non-linear
behavior of MOS-LQS (which varies smoothly with the
 validation performance at epoch 61. Coefficient of correlation
R ends at 0.95 putting this network at the same performance
level (on the test set) as the NNLM, as shown in Table VII but
farer from unwanted effects, such as overfitting.

Fig. 6. Error histogram for the ANN trained with the Levenberg-Marquardt
Algorithm.
Fig. 5. Statistical indicators of the MLR approach.

1) Levenberg-Marquardt Training Algorithm

TABLE VI. NNLM REGRESSION RESULTS

Statistical Indicators
Dataset
Samples MSE R-factor

Training 34 0.0689156 0.94922

Validation 7 0.279969 0.96953
Testing 7 0.0945887 0.97147

Let us start with the Levenberg-Marquardt algorithm for

training of the NNLM. We repeated the training ten times and
picked up the best network, which reached the best validation
performance after eight training epochs. Table VII shows the
mean sum-squared error that is fairly good and largely lower
than the MSE for MLR. Fig. 6 the error histogram for NNLM:
we learn that errors are more in the negative range, as for MLR Fig. 7. Performance of the ANN trained with the the Levenberg-Marquardt
residuals, though their size is lower. This issue improves Algorithm.
substantially for testing and validation samples (a view that we
cannot have in the usual MLR approach that uses all the data 3) Scaled Conjugate Gradient Training Algorithm
for fitting). Fig. 7 shows the regression plots of the network We also tried the Scaled Conjugate Gradient training and
outputs with respect to targets for training, validation, and test achieved the best validation performance for the NNSCG at
sets, and finally the whole data set. Coefficient of correlation R epoch 34, and R still equal to at 0.95.
indicates the proportionate amount of variation in the target
value explained by the network response, considered as the C. Discussion
independent variables in the linear regression model between The statistical indicators of performance for the four
the two ones. Nearer the R is to 1, better the network fits the adopted methods are reported in Table VII. The mean squared
MOS-LQS. Indeed, the NNLM has a fairly good RALL=0.95. error gets the minimum with the neural network trained by the
Levenberg-Marquardt algorithm. The MLR exhibits greater
2) Bayesian Regularization Training Algorithm
error, even if it has benefitted of 30% more data points for
We tried the Bayesian Regularization for training because
fitting. A specific remark is due on the three learning
our dataset is small. This Bayesian regularization takes place
approaches for the neural network. The most interesting
within the Levenberg-Marquardt algorithm modifying the
capability of a model for MOS is that the model output, though
optimization function. It utilizes all the data for training and
affected by error, still remains within the thresholds for the
requires longer training periods. In our case it gets the best
MOS target (that are discrete). This is generally the case for
NNLM as can be seen in the right lowermost plot of Fig. 7, with
an exception for the errors occurring around the target value
3.3 that is confused as 4 by NNLM. This is an issue solved by
the NNBR (comparing both the plots: here one is not reported
for brevity). The fact that NNBR has a slightly higher MSE is
meaningless. Further it requires more intense training, and this
is tolerable because training is a negligible cost if compared
with the cost of the listening test.
TABLE VII. MODEL COMPARISON
Main Statistical Indicators
Model
MSE RMSE RAll R2All
MLR 0.23912 0.48900 0.6371 0.7982
NNLM 0.10442 0.32314 0.9534 0.9764
NNBR 0.17862 0.42263 0.9524 0.9759
NNSCG 0.18338 0.42823 0.9527 0.9760
VI. CONCLUSIONS
VoLTE is a key service for LTE networks, and still needs
improvement and operalization of quality measures. Research
is strongly engaged in modelling LTE networks in realistic
usage conditions and modelling the quality perceived by the
users as a consequence of measurable technical parameters.
Here we modelled the network through scenario making,
produced a limited number of cases, simulated the network
performances. Then we attacked the subjective measurements
for MOS. This required to invest a certain amount of resources
to train and employ a panel of listeners. Thanks to a good
organization testing was performed at sustainable costs.
We studied how MOS-LQS results can be modelled using a
set of Artificial Neural Network models. The ANN model
outperforms the MLR, and this result is crystal clear, due to the
inner nonlinearities in the data (as per Fig. 4) and the filtering
action made by the human judgment. This confirms again the
need for an approach that is non linear and capable of
generalization.
However panel testing cannot be extensively repeated, as
would be necessary for a MNO that might face a incredible
number of operational situation. Our future research is aimed at
generalising the ANN model to a far larger scale (three orders
of magnitude at least) thanks to a fully distributed smart
approach. We have at stake the search for a viable way to make
quality nearer to user perception while controlling the
combinatorial explosion of scenarios and technical parameters.
REFERENCES
[1] 3GPP, Release 8 V0.0.3, “Overview of 3GPP Release 8: Summary of all
Release 8 Features” November 2008.
[2] A. Vizzarri, F. Davide, “Quality of Experience Model for VoLTE
Services through Statistical Analysis”, submitted to GLOBECOM 2016.
[3] 3GPP TS 23.401, “3rd Generation Partnership Project; Technical
Specification Group Services and System Aspects; General Packet Radio
Service (GPRS) enhancements for Evolved Universal Terrestrial Radio
Access Network (E-UTRAN) access (Release 8)”.
[4] 3GPP TS 23.228, “3rd Generation Partnership Project; Technical
Specification Group Services and System Aspects; IP Multimedia (IM)
Subsystem - Stage 2 (3G TS 23.228 version 2.0.0)”.
View publication stats
[5] 3GPP TS 23.203, “PCC – Policy and Charging Control Architecture”.
[6] A. Vizzarri, S. Forconi “Review of Studies on End-to-End QoS in LTE
Networks”, in Proc. AEIT Congress, Mondello, Italy, 2013, pp. 1-6.
[7] A. Vizzarri, “Analysis of VoLTE end-to-end quality of service using
OPNET” in Proc. European Symposium on Computer Modeling and
Simulation, Pisa, Italy, 2014, pag. 452 – 457.
[8] A. Vizzarri, “Analysis of VoIP Over LTE End-To-End Performances in
Congested Scenarios” in Proc. Artificial, Intelligence, Modelling and
Simulation, Madrid, Spain, 2014, pp. 393-343.
[9] 3GPP TS 123.207 “Digital cellular telecommunications system (Phase
2+); Universal Mobile Telecommunications System (UMTS); LTE; End-
to-end Quality of Service (QoS) concept and architecture”.
[10] ITU Recommendation P.800.2, “Series P: Terminals and Subjective and
objective assessment methods”.
[11] ITU Recommendation G.109 “Definition of categories of speech
transmission quality”
[12] A. Lakaniemi, J. Rosti, V. I. Raisanen, “Subjective VoIP speech quality
evaluation based on network measurements”, IEEE ICC 2001;
[13] Telchemy. Voice Quality Measurement.
http://www.telchemy.com/appnotes/TelchemyVoiceQualityMeasureme
nt.pdf, Mar. 2008 (accessed: May 12th, 2016).
[14] F. D. Rango, M. Tropea, P. Fazio, and S. Marano, “Overview on VoIP:
Subjective and Objective Measurement Methods”, IJCSNS, vol. 6, pp.
140-153, Jan. 2006.
[15] ITU-T Recommendation P.800 (08/96), "Methods for subjective
determination of transmission quality".
[16] L. Monacelli, R. Francescangeli, “Fault Management for VoIP
Applications over wireless and wired NGN Networks: an operational
prospective” in Proc. 7th IEEE International Workshop on Performance
and Management of Wireless and Mobile Networks, Bonn, Germany,
2011, pp. 711-718.
[17] J. Yu and I. Al-Ajarmeh, “Call admission control and traffic engineering
of VoIP. The Second International Conference on Digital
Telecommunications”, San Jose, California, USA, July 2007, pp. 11–16.
[18] M. Alreshoodi, J. Woods, “An empirical study based on a Fuzzy Logic
System to Assess the QoS/QoE Corelation for Layered Video Streaming”
in Proc. 2013 IEEE International Conference on Computational
Intelligence and Virtual Environments for Measurement Systems and
Applications, Milan, Italy, 2013.
[19] M. Alreshoodi, J. Woods, “Survey on QoE/QoS correlation models for
multimedia services”, International Journal of Distributed and Parallel
Systems (IJDPS) Vol.4, No.3, May 2013.
[20] S. Khorsandroo, R. Md Noor “Stimulus-Centric versus Perception-
Centric Relations in Quality of Experience Assessment”, in Proc.
Wireless Telecommunications Symposium, London, UK, 2012, pp. 1-6.
[21] N. Khitmoh, P. Wuttidittachotti, T. Daengsi “A Subjective - VoIP
Quality Estimation Model for G.729 Based on Native Thai Users” in
Proc. 16th International Conference on Advanced Communication
Technology, Pyeongchang, South Corea, 2014, pp. 48-53.
[22] R. E. Uhrig, “Introduction to artificial neural networks” in Proc. 1995
IEEE IECON 21st Int. Conference on Industrial Electronics, Control,
and Instrumentation, Orlando, Florida, 1995, pp. 33-37.
[23] G. Dreyfus, Neural Networks: methodology and applications, Ed. Berlin,
Springer, 2005.
[24] S. Rivera et alii, “QoS-QoE Correlation Neural Network Modeling for
Mobile Internet Services” in Proc. 2013 Int. Conference on Computing,
Management and Telecommunications, Ho Chi Minh City, Vietnam,
2013, pp. 75-80.
[25] V. A. Machado et alii, “A New Proposal to Provide Estimation of QoS
and QoE over WiMAX Networks” in Proc. 2011 IEEE Third Latin-
American Conference on Communications, Belem do Para, Brasile, 2011,
pp. 1-6.
[26] C. Benhamed “A large scale IP Network Traffic matrix Estimation based
on ANN: A comparison study on training algorithms” in Proc. 2015 4th
Int. Conf. on Electrical Engineering, Boumerdes, Algery, 2015, pp. 1-5.