End-to-End Speech Quality Testing

in a Complex Transmission Scenario

F. Kettler*, H.W. Gierlich*,
J. Berger**, H. Klaus**, I. Kliche**, K.-D. Michael**,
T. Scheerbarth**, R. Scholl***, J.-L. Freisse****
* HEAD acoustics GmbH, Herzogenrath,
** T-Nova Dt. Telekom Innovationsgesellschaft mbH Berkom,
*** European Telecommu. Standards Institute ETSI (now Siemens AG)
**** European Telecommunications Standards Institute ETSI
Overview
n Introduction
1
st
ETSI Speech Quality Test Event, Oct. 2000
n Test Setup for End-to-end Speech Quality Testing
n Test Methods, Parameters and Results
n One-way Speech Quality
n Echo and Double Talk Performance
n Background Noise Transmission Quality
n Summary
Introduction
IP network
designed for pure data
transmission
delay is high, unpredictable and time-variant
influence on echo and conversational quality
speech coders
specific kind of disturbances (packet loss, jitter...)
+ differentiation between terminal and network is no
longer possible!
Telephone network
real time transmission
properties
ETSI Speech Quality Test Event
Organized by ETSI, Sophia Antipolis, France, Oct. 2000
Sponsored by Texas Instruments, Alcatel, T-Nova and HEAD acoustics
Test labs T-Nova and HEAD acoustics
n Evaluation of end-to-end (mouth to ear) speech quality
using real equipment in a controlled VoIP environment
n One exclusive testing day for each manufactures
n Objective tests and subjective test (offline)
n Detailed test report and an anonymous report
containing all results
n Goals: identification of system improvements and
current status of achievable VoIP quality
Overview
n Introduction
1
st
ETSI Speech Quality Test Event, Oct. 2000
n Test Setup for End-to-end Speech Quality Testing
n Test Methods, Parameters and Results
n One-way Speech Quality
n Echo and Double Talk Performance
n Background Noise Transmission Quality
n Summary
Typical Signal Processing
Decoder
Buffer,
Packe-
tization
Buffer,
PLC
H(f )
- Coder
NLP NR
+
CN
VAD
EC
IP
IP
loudspeaker
jitter buffers,
PLC
decoder
comfort
noise
microphone
EC
level
switching
coder VAD
noise
reduction
delay,
jitter,
packet
loss
Speech Quality Parameters
speech
quality
talking
situation
listening
situation
conversational
situation
from the users perspective
Listening Situation
voice activity detection
speech coders
packet loss
packet loss concealment
...
IP
terminal A terminal B
Talking Situation
round trip delay
echo level and echo characteristic
implementation of speech echo cancellers
quality of background noise transmission
IP
terminal A terminal B
...echo?
Conversational Situation
propagation delay - dynamics impairments
double talk detection (EC implementation)
echo during double talk
level variations during double talk
IP
terminal A terminal B
Test Set-up
Acoustical
acoustical
test set-up
with IP terminals
mounted to the
HATS (Head And
Torso Simulators)
NISTNet
Packet
MONITOR
Packet Loss
DELAY
Output
HEAD acoustics
Testsystem CAS
Input
IP
IP
IP-Phone
PC
IP-Terminal
Test Set-up
Acoustical
electrical
test set-up
with one IP phone
and one gateway
(accessed through
a 4-wire ISDN line)
IP
NISTNet
Packet
MONITOR
T1 / E1 / BRI
Packet Loss
DELAY
Output
HEAD acoustics
Testsystem CAS
Input
Gateway
IP
ISDN DSS1
ALCATEL PABX 4400
IP
IP Terminal
(Phone or PC)
Overview
n Introduction
1
st
ETSI Speech Quality Test Event, Oct. 2000
n Test Setup for End-to-end Speech Quality Testing
n Test Methods, Parameters and Results
n One-way Speech Quality
n Echo and Double Talk Performance
n Background Noise Transmission Quality
n Summary
Listening Situation
Analysis methods: perceptual speech quality models
IP
terminal A terminal B
feeding
recording
and analysis
Instrumental Measures
Realizations
PESQ (P.862) Perceptual Evaluation of Speech Quality
TOSQA Telecommunications Objective Speech
Quality Assessment
TOSQA 2001 Extension to acoustical measurements
Typical Processing Steps (Schematic):

Adaptation
Hearing Model
Comparison,
Reference,
Reduction,
Signal Value
Results of
Listening Tests!
S index
Q
Speech
Signal
Test Object Hearing Model
processed
signal
reference
signal
1
st
ETSI SQTE - Test Conditions
IP
terminal A terminal B
c1: clear condition
c2: 1%packet loss
c3: 2%packet loss
c4: 3%Packet loss
c5: 5%packet loss
c6: 1%packet loss + 20 ms jitter
1
st
ETSI SQTE - Results
C1 C3 C5 C2 C4 C6
G.711 G.729
MNRU
18 30
C1 C3 C5 C2 C4 C6 C1 C3 C5 C2 C4 C6
G.723.1@6.3
24 12 06
direkt
References
1
2
3
4
5
G.711
PL=20...30ms
VAD on, PLC on/off
G.723.1@6.3
PL=30ms
VAD on, PLCon/off
G.729
PL=10...30ms
VAD on, PLC on/off
G.711 reference
G.723.1@6.3 ref.
G.729 ref.
TMOS
Analysis of packet loss and PLC
Occurrence of packet loss
(20 ms packet length)
Packet loss concealment
(typical implementation)
Detailed analysis of PLC implementations
Cross correlation analysis
vs. time
Relative Approach analysis
vs. time and frequency
Overview
n Introduction
1
st
ETSI Speech Quality Test Event, Oct. 2000
n Test Setup for End-to-end Speech Quality Testing
n Test Methods, Parameters and Results
n One-way Speech Quality
n Echo and Double Talk Performance
n Background Noise Transmission Quality
n Summary
Analysis methods: echo measurements based on
Composite Source Signals (CSS) under single and
double talk conditions
IP
terminal A terminal B
...echo?
feeding,
recording
and analysis
Talking Situation
Noise part
Voiced part
Pause
Signal description
in ITU-T P.501
Used e.g. in G.168
(Echo Cancellers)
Composite Source Signal
Sequence length 700 ms
SLR JLR JLR RLR
7dB 0dB 0dB 3dB
TELR a(Echo) a(Echo) TELR
RLR JLR JLR SLR
3dB 0dB 0dB 7dB
OLR = 10dB
Requirement on echo loss
depends on transmission
delay
Echo under Single Talk Conditions
Analysis methods: double talk measurements based
on two de-correlated Composite Source Signals
IP
terminal A terminal B
feeding feeding,
recording
and analysis
Conversational Situation
Signal description
in ITU-T P.501
Analysis methods
in ITU-T P.502
Combination of two CSS
Typical double talk
sequence (speech)
English, male and female
voices
Combination of CSS
incl. single and double
talk periods
Comparison to Real Speech
measured signal
nearly perfectly
matches original
test signal
green: measured signal
red: original test signal
yellow: overlap
Double Talk Evaluation: Result #1
green: measured signal
red: original test signal
yellow: overlap
Double Talk Evaluation: Result #2
short term echo
components
during pauses
and jitter
echo jitter
green: measured signal
red: original test signal
yellow: overlap
Double Talk Evaluation: Result #3
strong echo
components
(occurred only
during double
talk!)
strong echo
green: measured signal
red: original test signal
yellow: overlap
Double Talk Evaluation: Result #4
clipping at
beginning and
end of most
bursts and
comfort noise
during pauses
clipping
comfort noise
Overview
n Introduction
1
st
ETSI Speech Quality Test Event, Oct. 2000
n Test Setup for End-to-end Speech Quality Testing
n Test Methods, Parameters and Results
n One-way Speech Quality
n Echo and Double Talk Performance
n Background Noise Transmission Quality
n Summary
Test Signal for Noise Transmission
Noise signal
(Hoth spectrum)
with increasing
level vs. time
Signal description
in ITU-T P.800
Comparison of Test Results
1
2
3
4
Comparison of
three typical
implementations
(1) red: original test signal
(2) yellow: VAD threshold
(3) blue: AGC gain
(4) magenta: comfort
noise injection
Overview
n Introduction
1
st
ETSI Speech Quality Test Event, Oct. 2000
n Test Setup for End-to-end Speech Quality Testing
n Test Methods, Parameters and Results
n One-way Speech Quality
n Echo and Double Talk Performance
n Background Noise Transmission Quality
n Summary
n Identification of improvements and current status of
achievable VoIP quality - success
n Combination of various test procedures!
n Overall quality rating given by the participants
excellent
n A short report can be found under etsi.org/plugtests
n ETSI strongly recommended to continue the process
of end-to-end speech quality tests
n 2
nd
SQTE planned for April 2002
Summary