Академический Документы
Профессиональный Документы
Культура Документы
Contents
1
Introduction
1.1
System Components
2.2
2.3
2.5
2.3.1
2.3.2
2.3.3
2.3.4
2.4.2
2.4.3
2.4.4
15
QoS Measurements
17
4.1
4.2
Independent Channels.................................................................................................... 17
4.3
4.4
4.5
4.6
2.1
2.4
4.5.2
4.5.3
4.5.4
4.5.5
4.5.6
4.5.7
4.5.8
Real-Time Display........................................................................................................21
4.6.2
4.6.3
4.6.4
4.6.5
4.6.6
4.7
4.6.8
4.6.9
4.6.10
Streaming Test.............................................................................................................27
Messaging ...................................................................................................................... 32
4.7.1
4.7.2
MMS Tests...................................................................................................................32
4.8
4.9
4.10
Summary ........................................................................................................................ 37
38
5.1
5.2
5.3
42
6.1
6.2
46
7.1
7.2
Parameters Recorded..................................................................................................... 49
52
8.1
8.2
8.3
10
4.6.7
56
57
10.1
10.2
LTE Scanner................................................................................................................... 57
10.3
11
59
12
Appendix
61
12.1
Glossary ......................................................................................................................... 61
Introduction
1.1
TEMS Symphony
To attract, satisfy, and retain customers, mobile operators must combat the number one
source of dissatisfaction: poor quality stemming directly from inadequate services or weak
network performance. The TEMS Symphony wireless benchmarking solution gives insight
into the customers experience of both voice and IP-based services from operators and their
competition. This information is critical to the operators for making network improvements,
changes, and investments that will improve customer satisfaction.
Based on Ascom Network Testings powerful and flexible new MTP-4 hardware platform,
TEMS Symphony 6.1 offers a single solution that meets all of mobile operators
benchmarking needs whether vehicular, indoor, stationary, or nomadic testing. TEMS
Symphony interfaces with a wide range of user equipment, including standard handsets,
USB devices, and PCIe MiniCards, and provides voice, data, and video service testing. It can
be paired with our TEMS Discovery software for optimal post-processing.
Also thanks to the MTP-4 platform, TEMS Symphony 6.1 utilizes a modular configuration for
easy installation and maintenance. The flexibility of the MTP-4 system assures that users
have the right test tool for the task at hand, with the ability to easily reconfigure the unit when
confronted with different measurement requirements.
This document gives a description of the features and functions of TEMS Symphony 6.1.
NT11-17590, 1.0,8/29/2011
System Components
2.1
Measurement Principle
Test connections are built up between two end points. These connections can be speech
telephony calls, video telephony calls, VoIP calls, multimedia/video applications, mobile data
applications (for example, e-mail, FTP, UDP, HTTP, WAP, Ping) or messaging (SMS, MMS).
The performance of the network is then measured during these test calls/connections
The measurements are in two groups:
Subscriber view.
Quality of service (QoS) measurements represent network performance as seen by the
subscriber. These include real speech and video quality, dropped call rates, failed call rates,
SMS/MMS/e-mail success rates and delivery times, FTP/UDP/HTTP throughputs, etc.
Engineering view.
These measurements deliver detailed air interface information such as RxLev, RxQual,
interference, serving/neighbor cells, Layer 3 messages, etc. These are invaluable in
troubleshooting/optimizing the radio network.
2.2
Measurement Part
2.3
TEMS Symphony can cope with the different requirements of benchmarking networks all
over the world. It is very powerful and can be configured in very flexible ways.
2.3.1
Highly Modular
With the highly modular design, measurement systems can be ideally adapted to customer
requirements. Between one and 12 measurement channels are possible per MTP-4 chassis.
Chassis may be daisy-chained, to expand the system up to 24 measurement channels.
With four measurement processors per chassis, performance is entirely scalable. If one
measurement processor is available per channel, maximum data rates can be evaluated in
parallel (several video streams). Up to four chassis may be connected to support up to 16
processors. This provides an unparalleled level of capacity with which to measure highspeed data services.
2.3.2
MTP-4 Chassis:
6
NT11-17590, 1.0,8/29/2011
NT11-17590, 1.0,8/29/2011
RF Combiner Platform
To help reduce the number of external antennas, and control the RF isolation of the test
devices, an RF combiner platform is provided as an available option. Each stackable RF
combiner platform contains 3 x 4 port combiners, and provides front-mounted connectors for
all RF inputs and outputs.
RF Scanner Modules
The TEMS Symphony MTP-4 platform provides support for 3G and 4G EX scanning
receivers from PCTEL. EX scanners can be plugged into the MTP-4 platform. This allows a
very compact solution that can complete QoS measurements with RF scanner data.
2.3.3
Infinite Symphony
The flexibility and modularity of TEMS Symphony is unprecedented. It can be configured in
practically an infinite number of ways to suit the measurement job at hand.
There is no other measurement system with the unique features of TEMS Symphony:
Multi-Processor Test Platform with 4 powerful CPUs
Since the TEMS Symphony GUI is connected to the measurement system via a LAN or a
wireless connection, the user can decide where to be during a measurement.
Several TEMS Symphony GUIs at the same time
While the TEMS Symphony GUI PC controls the measurement, other read-only GUI PCs can
also be connected, showing the real-time measurement results. This enables,
10
NT11-17590, 1.0,8/29/2011
for example, the technical expert in his office to assist the driver in case of any problems or
questions during drive tests.
Time shift and replay of measurement data
Yet another innovation is the ability to replay the measurement data during the actual
measurement. For example, if the measurement has been in process for five minutes, the results
of the previous minutes can be replayed on the GUI device without having to stop the
measurement itself.
To make life even simpler, a SIM card slot is provided on the front of the MTP-4 chassis for all
applicable devices. This means that changing the SIM card on a test mobile phone can be done
easily, without opening the cover or removing the battery of the mobile phone. Another very useful
feature is that there are no cable connections at the back of the electronic unit, making for easy
mounting in vehicles/portable systems.
2.3.4
Portable Version
A portable kit is available to allow TEMS Symphony to be used for indoor/walk tests. The system is
housed in a backpack, which may be carried or rolled as a trolley. The portable kit provides
support for a single TEMS Symphony chassis, with up to eight measurement channels.
The portable kit provides:
A discreet backpack which is convertible to a rolling trolley
Battery pack to provide several hours of operation
RF combiner
Patch antennas
NT11-17590, 1.0, 8/29/2011
11
GPS antenna
Lightweight components for easy transport and setup
Hot swappable battery functionality for continuous operation
The total weight of the MTP-4 in the portable kit is approximately 15kg.
Combined with the portable kit, TEMS Symphony with the MTP-4 platform may be used in
any pedestrian areas, including trains, airports, offices, or stadiums.
In addition, TEMS Symphony can also be quickly and easily installed in a vehicle for drive
testing.
2.4
The optional TEMS Symphony Land Unit provides a land-based termination point for
measuring audio quality and placing land-to-mobile test calls. Calls can be placed between
the TEMS Symphony Land Unit and the MTP-4 benchmarking platform.
2.4.1
PSTN/ISDN Interfaces
MOC/MTC calls
Uplink/downlink speech quality (built-in DSPs)
Up to 12 PSTN and ISDN interfaces can be placed in the 19" TEMS Symphony Land Unit
equipment.
Mobile phones
The TEMS Symphony Land Unit can also be used for stationary measurements with mobile
phones. The phones are typically used for speech calls in cases of mobile-to-mobile call
testing, video telephony tests, or for SMS and MMS tests.
VoIP calls via LAN
Connected to a LAN interface, the TEMS Symphony Land Unit can be used for testing VoIP
sessions against mobile equipment. Different VoIP agents can concurrently run on one
system, allowing MOC/MTC calls and speech quality evaluation.
2.4.2
NT11-17590, 1.0,8/29/2011
When used as a data server, the TEMS Symphony Land Unit may be either connected to the
Gi interface or to the public Internet.
The TEMS Symphony Land Unit is a PC that has been enhanced with the necessary
supplementary cards. It has a 19" housing with a height of four units.
The TEMS Symphony Land Unit is designed for use without operator assistance.
Even in cases of point-to-point tests, all configurations of the Slave channels in TEMS
Symphony Land Unit are performed by the full automatic configuration from the Master
channels in drive and walk test equipment.
2.4.3
The measured data can be replayed in the GUI of the measurement component, thus
enabling the network engineer to re-examine the data in detail. All of the powerful functions
of the GUI during measurement are available in data replay (e.g., drag and drop, multiple
text/graphics windows, and forward/reverse replay, etc.).
2.4.4
TEMS Discovery
TEMS Discovery is a highly configurable and user-friendly post-processing solution for air
interface measurement data. It allows engineers to easily assess wireless performance and
quickly pinpoint network problems. With TEMS Discovery, the user has the flexibility to
configure a wide range of items, from simple view layouts to sophisticated report templates
and user-defined key performance indicators (KPIs).
13
2.5
The following illustration show the different components of TEMS Symphony, the worlds
leading cellular measurement system:
14
NT11-17590, 1.0,8/29/2011
With more than 15 years of continuous development, TEMS Symphony has many distinctive
features that enable users to carry out measurements easily, and to analyze and present the
results quickly.
Below is a short list of features, which will be expanded in more detail in the following
chapters.
3.1
15
16
Mobile application protocol stack (e.g., MMS/WAP), so measurement trigger points can
be precisely defined (e.g., as per ETSI definition). The protocol messages are recorded
simultaneously with air interface data (e.g., Layer 3). Therefore, the user has a complete
view of different layers, which is invaluable in troubleshooting/optimization
Combining test mobile phone and scanner based measurements, resulting in powerful
diagnostic functions
Subscriber view of QoS in uplink and downlink, with numerous features to simulate
subscriber behavior (e.g., redial, pause, MOC/MTC/mobile-to-mobile calls, etc.)
Detailed decoding of engineering data: Layer 3, RLC/MAC messages
Flexibility in running measurements over different air interfaces (e.g., the same FTP tests
can be run over GPRS, EDGE, WCDMA, HSPA, CDMA2000 EV-DO, WIMAX, LTE, etc.)
Technology-specific features (e.g., TBF usage), QoS fulfillment, time used/time wasted
for transmission for GPRS)
The IP traces captured per channel allow in-depth analysis in case of weak performance
or problems with IP-based services
VoIP sessions between two mobile phones or between a mobile and a fixed Internet user
can be set up and evaluated in the same way as circuit-switched calls, including speech
MOS in uplink and downlink
NT11-17590, 1.0,8/29/2011
QoS Measurements
QoS measurements are a true representation of what subscribers experience in the network.
QoS measurements can be made in a mobile operators network or competitors networks,
and the results can be viewed in real time during measurement or in post-processing
systems.
4.1
An important advantage of TEMS Symphony is that the QoS measurements can be made on
almost any air interface technology, simultaneously. For example, the same speech call tests
can be made on GSM and WCDMA and provide the dropped call/failed call rates, real
speech quality measurements, etc. Similarly, mobile data tests of FTP, UDP, and e-mail can
be made on GSM, GPRS, EDGE, WCDMA, HSPA, HSPA+, CDMA2000, EV-DO (including
Rev. A) WiMAX, and LTE, and give invaluable results for each network/technology.
This also enables the benchmarking of different networks (irrespective of which technologies
are in use) or the technologies themselves.
4.2
Independent Channels
Each channel in the TEMS Symphony measurement system can have its own measurement
task, and it will run independently of what other channels are measuring (e.g., speech, IP
data, SMS, MMS tests).
4.3
Measurement Configuration
The setup and configuration of a test measurement can be done very quickly and easily.
There are dedicated measurement programs for the different types of tests (e.g., speech and
FTP). Creation of the measurement programs is supported by special editors and can also
be exchanged among different users and systems.
17
Subscription parameters (e.g., SIM subscriptions and phone numbers), as well as destination
parameters, can be separately configured and used for the final measurement configuration
of the system.
The system configurations can be prepared by the drive test manager and distributed to the
different Symphony systems in use.
4.4
Master/Slave Configurations
In peer-to-peer tests such as speech calls or SMS, both involved parties have to know about
the test parameters. The parties sitting on the TEMS Symphony system used in drive or walk
tests and controlled by an operator are the so-called Master channels. The Master channels
will contact their corresponding peers (called Slave channels) before measurement and
inform them about the measurement parameters (e.g., call length and type of call sequence).
18
NT11-17590, 1.0,8/29/2011
No operator assistance is needed for the TEMS Symphony Land Unit, even in the case of
peer-to-peer tests.
4.4.1
EMC
Server
Internet
GPRS EDGE /
WCDMA HSPA /
CDMA 1xEV-DO
TCP
Company LAN
TC
P
Master
Slave
TEMS
Symphony MTP-4
Data Card
GPS / NTP Time
TEMS
Symphony
Land Unit
4.5
4.5.1
Measurement Configurations
19
Speech channels can be synchronized in time (that is, speech calls on the TEMS Symphony
systems will start at exactly the same time).
4.5.2
When a real speech sample has been transmitted through the network being tested, the
PESQ algorithm (according to ITU-T recommendation P.862) evaluates the speech quality
as heard by a human ear in real time, and shows the speech quality on a scale from 1
(bad) to 5 (excellent).
In addition, the user can detect other in-band problems like silence, audio-level jump, and
echo, which can lower the speech quality as heard by the subscriber. Gaps in speech such
as those caused by handovers are measured in terms of length and position within the
speech sample.
Networks using GSM Full Rate, Enhanced Full Rate, and Half Rate codecs, as well as AMR
networks, can be measured.
Support for POLQA, the Latest Speech MOS Algorithm
Support has been added for the new voice quality testing standard algorithm, POLQA (ITU-T
recommendation P.863). POLQA provides new opportunities to benchmark the voice quality
of competing mobile networks. With enhanced MOS accuracy, POLQA is suitable for highdefinition voice, 3G, and 4G/LTE technologies.
Wideband Speech Testing
To accommodate the testing of new voice codecs like AMR-WB, a wideband speech testing
option has been added. Based on the POLQA algorithm, the wideband speech option
provides new speech samples that are optimized for testing either narrowband speed (up to
3.2kHz) or wideband speech (up to 7kHz).
4.5.3
Live Recording
This is a unique feature of TEMS Symphony that the user can employ to set the thresholds
by which the received speech sample will be recorded.
For example, if the received speech sample has a quality between 3.5 and 1.0, it will be
recorded.
4.5.4
Entire calls can be audio-recorded and can be used for investigation of problems (e.g., in
cases of congestion, busy called party, and diversion to speech box).
4.5.5
Call Statistics
During speech tests, other QoS parameters that affect the subscriber perception of network
quality are also measured dropped call rates, blocked calls (call status failed but network
was available), no service (no usable network found), successfully completed call rates (can
be defined as completed calls or completed calls with speech quality greater than x), setup
times, and more.
20
NT11-17590, 1.0,8/29/2011
4.5.6
4.5.7
No-Coverage Areas
TEMS Symphony allows the user to specify whether to make calls in no-coverage areas.
This is vitally important for benchmarking statistics because, for example, if calls are made in
an area where network A has coverage but not network B, then the call success rate of A
would be much higher than B. This could be misleading if not interpreted carefully.
4.5.8
Real-Time Display
During measurement, the measured results e.g., MOS and other values can be shown in
real-time. The user can construct preferred displays using graphics/text/rolling text windows.
21
4.6
Mobile data tests measure the performance of the network in transferring data/messages. A
great variety of tests are available FTP, UDP, Ping, e-mail, and others and the user can
set up a measurement program consisting of different tests.
4.6.1
In mobile data tests, the TEMS Symphony measurement system acts as the client, and, as
such, needs a server with which to communicate. The server can be:
TEMS Symphony Land Unit-Data, connected at the Gi interface or on the public Internet
Other server (i.e., a public server)
Possibilities with TEMS Symphony Land Unit
Using a TEMS Symphony Land Unit provides optimal control over IP data services, enabling
Ping, UDP, and FTP uplink testing to be carried out. This may not be possible with a public
server.
Even FTP downloads, normally not a problem with a public server, could sometimes fail
because the target file has been moved to another directory or deleted. In addition, a public
server may be temporarily overloaded and hence bias the measurement result.
However, sometimes it may be desirable to use a public server, such as in benchmarking
exercises where a connection via a competitors network to ones own TEMS Symphony
Land Unit-Data may not be possible.
TEMS Symphony Land Unit at the Gi interface
Another advantage of TEMS Symphony Land Unit-Data is that it can be connected to the Gi
interface, thus eliminating the (highly variable) impact of the public Internet.
22
NT11-17590, 1.0,8/29/2011
This gives the operator a more accurate picture of the performance of its network and the
elements under its control.
4.6.2
FTP Tests
FTP measurements (active or passive) can be configured using file types of zip, jpg, text, or
binary, and the file size can be 15 KB to 1 GB. This is another advantage of TEMS
Symphony, where the slow-start mechanism of TCP (which can lower the data throughput)
can be investigated or eliminated by using different file sizes.
FTP download (get) or upload (put) measurements can be performed.
Sample time specifies how often the FTP throughput is reported or recorded. Below is a
typical screen shot during FTP measurements:
Example of an FTP test: grid views (tables), message browser, and line chart
4.6.3
UDP Tests
TEMS Symphony UDP measurements can be used to test the maximum capacity/throughput
of the network (without TCP retransmission), or the performance in most streaming
applications.
They can be set up with message sizes of 200 to 5,000 bytes and the number of messages
to be sent can be controlled by the count parameter (10 to 10,000). The throughput sample
time can also be changed.
NT11-17590, 1.0, 8/29/2011
23
Bandwidth limit
The bandwidth limit is another unique feature of TEMS Symphony.
In the measurement path between the TEMS Symphony MTP-4 client and the TEMS
Symphony Land Unit-Data, or public, server, there may be bottlenecks that limit the UDP
performance (e.g., intermediate nodes with limited buffers) and it is useful to be able to steer
the test program to send UDP packets up to a certain throughput limit, but not beyond (which
could lead to loss of datagrams).
UDP tests can be done in uplink and downlink. In the case of uplink, the TEMS Symphony
Land Unit-Data measures the UDP uplink throughputs and stores the uplink results.
4.6.4
Ping Tests
Ping is a useful measurement for testing round-trip delays and hence its impact on
handshake-based protocols (e.g., TCP) or interactive classes of applications (reaction time).
TEMS Symphony enables the user to define the message size (20 to 5,000 bytes) because it
is an important factor for some investigations, such as:
The relationship between message size and round-trip delay
Network sensitivity to message size (some real-life measurements show that some
networks tend to lose a lot of messages of a particular size)
The impact of the maximum packet size on the network under test
If the maximum echo time is greater than the timeout specified (500 to 20,000 ms), then the
message is considered as lost.
The pause (100 to 10,000 ms) is another important advantage of TEMS Symphony. It
specifies the pause between receiving the echo from the previous message to sending the
next message.
TEMS Symphonys ability to vary the pause duration (in addition to the message size)
enables the user to investigate:
The behavior of opening/closing/delayed closing of TBF in GPRS
The inter-relationship between the Ping frequency and the echo time
In addition, TEMS Symphonys Ping tests (unlike, for example, console Ping in Microsoft)
can be run simultaneously on all measurement channels (i.e., benchmarking is possible with
Ping tasks).
An Ascom Network Testing Application Note is available to explain the details of the Ping
timing and its impact on measured results.
24
NT11-17590, 1.0,8/29/2011
4.6.5
HTTP download tests are done with connections to a destination, which can be configured as
follows:
For testing the retrieval of Web pages, for example, HTTP tests are configured with a count
specifying how many times the Web page is downloaded. A timeout gives the maximum time
25
allowed until the end of the transmission. If the same Web page is downloaded more than once, a
pause specifies the wait time between retrievals. HTTP tests can also be run over a proxy server.
Number of threads
Number of threads is another unique feature of TEMS Symphony that allows the user to control
the number of parallel downloads.
For special investigations of network performance, the number can be set to, say, 1 (one file
download at a time). To emulate performance of commercial browsers, the number can be set
higher, to, say, 6.
A sample result, shown in real time with signaling messages, is:
4.6.6
This type of test allows for the download of Web pages from the Internet via Microsofts
Internet Explorer. By using Internet Explorer, the experience of common users will be
reflected.
4.6.7
This type of test simulates mobile users uploading their data (such as photo books) to a
network server on the Internet.
26
NT11-17590, 1.0,8/29/2011
A dedicated TEMS Symphony application running on a server will communicate the HTTP
upload test running on the TEMS Symphony system. As with the HTTP download, files of
different sizes can be selected. Upload time and throughput are measured and will be stored
on the TEMS Symphony system.
4.6.8
WAP Tests
WAP tests perform end-to-end quality measurements of the WAP service, including all
elements involved: mobile network, WAP proxy server, and Web server. These tests help
pinpoint the detailed reasons for service faults across various protocol layers. The WAP
browser runs on the TEMS Symphony system. The mobile phone is used as a data modem
only.
4.6.9
E-Mail Tests
Sending and retrieving e-mail (with or without attachments) can be programmed, with the text
size of the e-mail between 10 to 10,000 bytes.
Wait time (the delay before the e-mail is retrieved) can also be set by the user.
Count is the number of times the e-mail is sent and retrieved.
Timeout is the maximum delay for the anticipated response time (TEMS Symphony will
suggest a suitable value, depending on the size of the attachment).
Pause is between 0 to 1,000 seconds and represents the pause between sending e-mail.
A random pause can also be specified.
Attachment, if activated, can be a text, jpg, or zip file of 10 KB to 5 MB.
The measured results of the e-mail test include success/failed operations (send/retrieve),
send times, and retrieve times (minimum, maximum, and mean).
4.6.10
Streaming Test
TEMS Symphony can test streaming multimedia services delivered over packet-switched
bearers. Multimedia clips can be streamed via the mobile Internet in an automated repetitive
way and all relevant streaming QoS parameters will be measured and presented online.
Perceptual streaming video quality measurement will result in video MOS values. The values
are mapped on a scale ranging from 1 (bad) to 5 (excellent).
From TEMS Symphony 5.2 onwards, the multimedia clip can be observed in a dedicated
window and heard via the audio output. Live recording of the video stream for later
documentation and investigation is also possible.
When using TEMS Symphony GPRS/EDGE or WCDMA test mobile phones, all engineering
traces including Layer 3 messages will be presented online and logged for post-processing.
TEMS Symphony's streaming test is optimized for wireless video and supports all major
streaming players (Windows Media, RealMedia, QuickTime). Different 3GPP codecs (ITU-T
H.263, MPEG-4) can be tested.
Perceptual Quality Measurement
TEMS Symphony uses absolute metrics. The perceptual metrics take into account the image
content and frame data of the video resulting from the given coding and transmission
conditions.
27
Quality metrics can be divided into relative (full-reference) metrics and absolute (noreference) metrics. Relative metrics compare a compressed or otherwise processed video
directly with the original, whereas absolute metrics analyze any given video without the need
for a reference, using only the data contained in the clip under test. The figures below show
the different application scenarios for relative and absolute metrics:
System
under test
System
under test
Relative
metrics
Absolute
metrics
NT11-17590, 1.0,8/29/2011
simply be retransmitted. If substantial parts of the video stream are not available when they
are needed for display, decoders often show the last good picture until they can resume
playback. An illustration of this is shown below. Jerkiness can also be introduced by the
encoder dropping frames in an effort to achieve the given bit rate constraints. Finally, a
reduced or varying frame rate can also create the perception of jerky motion.
Video Playback
Time
Time
Blockiness is a perceptual measure of the block structure that is common to all DCT-based
image and video compression techniques (see the examples below). The DCT (discrete
cosine transform) is typically performed on 8x8 blocks in the frame, and the coefficients in
each block are quantized separately, leading to discontinuities at the boundaries of adjacent
blocks. Due to the regularity and extent of the resulting pattern, the blocking effect is easily
noticeable. Blockiness can also be caused by transmission errors, which often affect many
blocks in a video frame. The agents blockiness metric looks for these characteristic block
patterns in the video.
Blur is a perceptual measure of the loss of fine detail and the smearing of edges in the video
(see figures below for examples). It is due to the attenuation of high frequencies by coarse
quantization, which is applied in every lossy compression scheme. It can be further
aggravated by filters, such as for de-blocking, which are sometimes used in the decoder to
reduce the noise or blockiness in the video. In certain compression schemes, transmission
errors or packet loss can also induce blur. Another important source of blur is low-pass
filtering (e.g., digital-to-analog conversion). The agents blur metric analyzes the video for
these types of distortions. Subjective experiments with images containing different types of
blur show a correlation of up to 96% between the agents blur metric and perceived blur.
29
Original
Blockiness
Blur
Blur
30
NT11-17590, 1.0,8/29/2011
Using the data from these tests, the agents metrics achieve a correlation with subjective
MOS of around 90%. As a comparison, PSNR (which is a relative metric) has a correlation of
only 40% for these experiments.
Cause analysis
The different types of perceptual artifacts described above and common video problems
causing them are summarized in the following table:
Metric overview and cause analysis
Perceptual
metric
Type
Description
Common causes
Jerkiness
Temporal
Blockiness
Spatial
Blur
Spatial
MOS
prediction
Spatial and
temporal
31
4.7
Messaging
4.7.1
SMS Tests
SMS tests are carried out between two channels/test mobile phones. One sends the SMS
(and records the sent time and whether it was successfully sent), and the receiver records
the receive time and whether it was successfully received. The delivery time (send to
receive) is calculated.
The measured results include: number of sent requests (of which how many are successful)
and send times (minimum, maximum, mean).
4.7.2
MMS Tests
The MMS test consists of sending/retrieving MMS between two channels (Master/Slave) in
the same equipment, measuring the success/failure of the MMS transmission, the
send/retrieve times, and detecting any failures during the WAP/MMS message exchange.
This is implemented by having the WAP stack and the MMS software running in TEMS
Symphony, with the test mobile phone providing the connection.
32
NT11-17590, 1.0,8/29/2011
Integration of the WAP stack in TEMS Symphony software offers many advantages:
During MMS/WAP tests, the test mobile phone can still deliver engineering information
like Layer 3 messages.
Precise trigger points (per ETSI definitions) can be defined (i.e., starting the time
measurement at a particular WAP/MMS message, and ending the measurement at a
specific message).
The exchange of MMS/WAP messages can be monitored and any problems detected.
The MMS test is independent of which type of test mobile phone is in use (e.g., a GPRS
or a WCDMA test mobile phone).
The measured results of the MMS test include: send and delivery success rates, numbers of
send requests (and how many were successful), send times (minimum, maximum, mean),
notification times (minimum, maximum, mean), retrieve times (minimum, maximum, mean),
and unexpected receives.
MMS trigger point definitions
Due to the availability of the MMS stack in TEMS Symphony, the trigger points for the
various measurements can be precisely defined.
For send-receive-delivery indication (T1 to T9 as per ETSI TS 102-250-2), see figure below:
Master
ng
e
tS
ir
nf
r
ta
Send Timeout
T1
Slave
i
nd
T2
T3
d
en
Notification Timeout
T4
PDP Cont.
PDP Cont.
Activation Send MM Deactiv.
n
g
tio vin
ca trie
i
d
I n Re
y
f
rt
i
ot Sta
N
Retrieve Timeout
T5a T5b
Notify MM
T6
PDP Cont.
Activation
T7
Retrieve
MM
ify
ot
Master
se
on
p
s
T8
PDP Cont.
Deactiv.
T9
Delivery
Indication
4.8
33
Since the video quality at the receiving end is impacted by the quality of the radio link and the
demand placed on it (e.g., how fast is the video picture changing), TEMS Symphony
provides four different video samples to ensure that truly representative tests can be done.
Provided video content
Grandma is a fairly stable video clip, with the subject talking without much movement.
Salesman has due to waving arms, etc. more movement built in.
Car phone represents a typical video telephony usage, with someone talking in a moving
vehicle.
The captured video is then evaluated by a referenced video quality algorithm PEVQ, giving
the following measured values:
Video MOS score
[1 to 5]
Effective frame rate
[fps]
Number of frozen and skipped frames
[%]
Jerkiness
[0 to 10]
Blur
[0 to 10]
Blockiness
[0 to 10]
Brightness reference and captured vide
[0 to 10]
Contrast reference and captured video
[0 to 10]
PSNR of luminance (Y) and chrominance
(Cb, Cr)
[dB]
At the same time as video transmission, speech signals are alternately sent in both directions
(half duplex) and their quality is evaluated at both ends. The measurement parameters are
the same as with speech tests described previously.
The video telephony test with video and speech MOS can be carried out with sending and
receiving ends on the same equipment (TEMS Symphony) or on two different systems.
4.9
VoIP Tests
VoIP sessions can be set up between two mobile subscribers (2) or between a mobile and a
subscriber connected to a stationary LAN/Internet connection (1):
34
NT11-17590, 1.0,8/29/2011
VoIP
Internet
Cellular
Network
EMC Server
Data Card
for EMC
TEMS
Symphony MTP-4
M = Master
Data Card
for EMC
M
S =Slave
TEMS
Symphony MTP-4
EMC
TEMS
Symphony
Land Unit
After building up an IP bearer to the Internet and registration of both parties, VoIP sessions
can be configured with the same flexibility as circuit-switched speech calls. The number of
MOC/MTC calls and the speech mode (uplink, downlink, half duplex) can be configured per
test.
By using VoIP agents installed on TEMS Symphony modules and on the TEMS Symphony
Land Unit, VoIP sessions can be set up and quality can be measured on different levels:
35
IP-trace log
Jitter
Delay
Packet Loss
R-factor (MOS)
NB/WB MOS
Special effects
PESQ
ETSI KPI
VQMon
VoIP AGENT
VQMon
Metrics
IP Trace
CODEC
Jitter Buffer
Emulator
Jitter
Buffer
RTP
Call ctr
SIP
UDP
IP
Call statistics
All trigger points for calculations of the call statistics as defined in ETSI TS 102 250:
Registration failure ratio
Registration time
Session setup failure ratio
Session setup time
Session completion failure ratio
Speech quality, based on ITU P.862.1
Speech transmission delay
Speech quality
Speech quality MOS can be measured in uplink or downlink and is currently based on ITU
P.862.1.
Network transmission quality
VQMon measures the quality of the IP network and codec performance using synchronized
video and speech samples, providing metrics for:
Jitter
Packet loss
Delay
36
NT11-17590, 1.0,8/29/2011
IP trace
Logging the full IP trace can be very helpful for analyzing problems during registration and
session setup.
Choice of codecs
TEMS Symphony customers can select from several different types of codecs:
G711
G723.1
G729
WB-AMR
GSM
4.10
Summary
TEMS Symphony can test the network QoS performance from a subscribers point of view,
measuring:
Speech calls with speech quality, in-band audio problems, and call statistics
Mobile data applications with a great variety of protocols (with special configuration
possibilities for in-depth investigations) and popular applications like SMS, e-mail, WAP,
and MMS
Video connections for video streaming and video telephony, with unique features for
feeding/capturing video signals and seeing the video during tests
VoIP sessions including speech MOS quality, call statistics parameters, and network
transmission quality metrics
These tests can be mixed and matched on a single system, with each channel independently
running different measurement programs.
The channels can be of different air interface technologies: GSM, GPRS, EDGE, WCDMA,
HSPA, CDMA2000, EV-DO, WiMAX, and LTE.
37
While it is performing the QoS measurements described above, TEMS Symphony can
simultaneously measure the engineering parameters in the air interface. This chapter covers
the engineering measurements which are invaluable for network tuning and troubleshooting.
5.1
5.2
Scanning results can be monitored online in different specific optimized views. The box plot
view is an all-channel monitor showing the signal strength of all the channels selected in the
38
NT11-17590, 1.0,8/29/2011
system. The RSSI is color-coded and gives a good idea of the signal strength. Detailed RSSI
results are displayed in dBm together with results for BSIC, C/I, and BCCH. All of these
scanning results can be sorted by channel number or signal level.
In the box plot view, numerical details are highlighted by hovering over the channel numbers
In addition, interesting results can be analyzed in the field in time shift mode and can be
combined with all types of other results in different windows.
39
5.3
40
NT11-17590, 1.0,8/29/2011
C1
Dedicated reports
Cell Name
Speech quality
Quality
Silence
Signal Level
Echo Amplitude
Echo Delay
Gap in speech
Channel report
BSIC
CHTYPE
TN
SUBCH
TSC
MAIO
HSN
BCCH_ARFC
C/I values
GSM Layer 3 Info
TA
Actual Timing Advance
TxPwr
Actual MS TxPower
RxLevFull
Serving Cell RxLevFull
RxQualFull
Serving Cell RxQualFull
RxLevSub
Serving Cell RxLevSub
RxQualSub
Serving Cell RxQualSub
FER
Frame Erasure Rate
RLCinit
Radio Link Counter init
RLActual
Radio Link Counter actual
RLCStatus
Radio Link Counter status
DTX UL
Uplink DTX status
DTX DL
Downlink DTX status
Neighbor Cell Count
ARFCNx
Frequency Channel neighbor x
RxLevNx
RxLev of neighbor x
BSICNx
BSIC of neighbor x
Idle reports
RxLevFull
Serving Cell RxLevFull
TxPowerMax
Tx max power serving cell
RxLevAccMin
RxLev min access lev serving
cell
Power Class
MS Power Class
DSC
init, actual, status change
C1
C1 from serving Cell
C2
C2 from serving Cell
Neighbor Cell Count
ARFCNx
Frequency Channel neighbor x
RxLevNx
RxLev of neighbor x
BSICNx
BSIC of neighbor x
C1Nx
C1 of neighbor x
C2Nx
C2 of neighbor x
RACNx
RA Color Code neighbor x
The actual list of measured parameters may differ slightly depending on the model of the test
mobile phone.
41
GPRS engineering measurements are carried out at the same time as the mobile data QoS
tests described above. The test mobile phones in TEMS Symphony will carry out tests (such
as UDP) while delivering GSM/GPRS engineering data. Since GPRS is based on GSM
technology, the aforementioned GSM engineering parameters are also available during
GPRS measurements.
Additional configuration possibilities include:
APN
Bearer type
Packet-switched
Dial-up mode
Auto attach
On/off
Band forcing
850/900/1800/1900
Do detach/attach
On:
detach with every disconnect
6.1
Attach
with
every
connect
and
Similar to the GSM displays, GPRS real-time display during measurement shows in-depth
information on the operation of the network:
Which measurement program is in progress (e.g., an HTTP task)
Serving and neighbor cell information
GPRS radio environment TA, C value, etc., and dedicated channel information
Layer 3 messages (rolling window)
GPRS idle information (BCCH RxLev, C1, C2, etc.)
Layer 2 information including LLC frames and throughput, RLC frames and throughput,
PDP context information, status of the GPRS layers (SM, GM, RR)
QoS fulfillment and TBF information
EDGE parameters like coding schemes
A typical screen shot during testing shows the wealth of information available in the TEMS
Symphony real-time display:
42
NT11-17590, 1.0,8/29/2011
43
6.2
C_Value
RxQual
SIGN_VAR
I_LevelTS0
I_LevelTS1
:
I_LevelTS7
Timing Advance
TxPower
Frame Erasure Rate
Block Error Rate
44
Delay
Messages Transferred
Packet Loss
Coding Scheme UL
Coding Scheme DL
Time Slot Count UL
Time Slot Count DL
RLC Throughput UL
RLC Frames Transmitted UL
RLC Frames Repeated UL
RLC Throughput DL
RLC Frames Transmitted DL
RLC Frames Repeated DL
RLC Acknowledge Mode
LLC Throughput UL
LLC Frames Transmitted UL
LLC Frames Repeated UL
LLC Throughput DL
LLC Frames Transmitted DL
LLC Frames Repeated DL
LLC Acknowledge Mode
Time Slot Allocation UL
Time Slot Allocation DL
Direction
TFI (Temporary Flow Identity)
NT11-17590, 1.0,8/29/2011
SM Layer State
GM Layer State
RR Mode
Pages Loaded
Pages Timed Out
Download Time
Page Size
Data Q Class
Actual Throughput
Actual Block Errors
Total Throughput
The actual measured parameters may differ slightly depending on which test mobile phone
models are used.
45
Similarly to GSM/GPRS engineering tests, the WCDMA tests are run simultaneously with the
QoS measurements, so if TEMS Symphony is carrying out a speech test call or a UDP
transfer, the WCDMA engineering data will be simultaneously measured and recorded.
Several test devices for WCDMA/HSDPA/HSUPA+ testing equipped with engineering trace
software are available for different WCDMA frequency bands
7.1
46
NT11-17590, 1.0,8/29/2011
47
48
NT11-17590, 1.0,8/29/2011
7.2
Parameters Recorded
RAB info
Network Type
Network State
MCC, MNC
LAC
RAC
URAid
Neighbor info
Neighbor Count
Downlink UARFCN
Downlink UARFCN of 2nd freq
UTRA Carrier RSSI
UTRA Carrier RSSI of 2nd freq
Cell ID (per neighbor cell)
Cell Type (per neighbor cell)
Cell Status (per neighbor cell)
Primary Scrambling Code (per neighbor cell)
P-CPICH Ec/Io (per neighbor cell)
P-CPICH RSCP (per neighbor cell)
PS Layer States
SM State
GMM State
PMM State
RLC info
RLC Entity Count
Radio Access Bearer ID (per RLC entity)
Radio Bearer ID (per RLC entity)
Logical Channel Type (per RLC entity)
UL Transport Channel ID (per RLC entity)
DL Transport Channel ID (per RLC entity)
RLC Mode (per RLC entity)
PRACH info
PRACH Initial Tx Power
PRACH Preamble Offset
PRACH Preamble Tx Count
PRACH Message Tx Power
PRACH Max. Preamble Count
49
HSPA+ Configuration
Uplink Tx Power
Uplink Interference
PC Algorithm
PC Step Size
DPC Mode
DL 64QAM Modulation
MIMO
MIMO Antenna2 CPICH Type
MAC Mode
HS-SCCH Less Operation
Layer 1 info
L1 Channel State
STTD on P-CCPCH
Number of S-CCPCHs
HSDPA info
Serving HSDPA RL
HSDPA Activity
HS-PDSCH Decoding Success Rate
No. of HS-PDSCH
HS-PDSCH Gross Bit Rate
HS-PDSCH Re-transmission Rate
Modulations QPSK, 16QAM
UL HS-DPCCH Statistics (CQI, CRC results)
HSUPA info
Serving E-DCH Cell
RG Index
Max. Spreading Factor
Max. Layer 1 Bit Rate
Happy Bit Delay Condition
2- and 3-Index Step Threshold
Requested
Minimum
Negotiated
QoS Version: R97
R99
R97&99
Precedence Class
Delay Class
Reliability Class
Peak Throughput
Mean Throughput
Radio Priority
Traffic Class
Delivery Order
Delivery of Erroneous SDU
Max. SDU Size
Max. Bit Rate UL
Max. Bit Rate DL
Residual BER
SDU Error Ratio
Transfer Delay
Traffic Handling Priority
Guaranteed Bit Rate UL
Guaranteed Bit Rate DL
Markers
Handover
Cell Reselection
RRC State Transition
URA Update
Location Area Update (same as in GPRS)
Routing Area Update (same as in GPRS)
PS Attach/Detach (same as in GPRS)
PDP Context (same as in GPRS)
Layer 3 Messages
50
NT11-17590, 1.0,8/29/2011
Please note that the parameters recorded depend on the make and software release of the
test mobile phone and can therefore vary somewhat from the above.
51
At the same time the QoS measurements described above are performed, TEMS Symphony
can measure the engineering parameters in the air interface.
This chapter covers the engineering measurements for CDMA2000 technology, including
EV-DO and EV-DO Rev. A, which are invaluable for network tuning and troubleshooting.
TEMS Symphony CDMA2000 tracing functionality also covers AMPS in case of a handoff to
the analog network. Supported measurement devices include various models of handsets
and data cards from vendors such as Samsung, Kyocera, Sierra Wireless, and others.
8.1
52
NT11-17590, 1.0,8/29/2011
53
8.2
Serving info
Network Type
ARFCN Channel Number
SID/NID
Technology Mode (1x, EV-DO, EV-DO Rev. A)
T_Add
T_Drop
T_Comp
TT_Drop
Network State
Vocoder Type
Phone State
Tx Power
Tx Power Adjust
Rx0 and Rx1 Power
Serving PN
Serving SINR
DRC Requested/Current
Reverse Rate Limit (Rev. A)
Current Reverse Rate (Rev. A)
Markers
Handoff
Layer 3 Messages
Finger info
Finger Count
PN Code per finger
Ec/Io per finger
Power Control
Tx Power
Tx Power Adjust
Rx1 Power
Forward FCH, SCH0 and SCH1 FER
Forward FCH, SCH0 and SCH1 Frame Rate
Reverse FCH, SCH0 and SCH1 Frame Rate
54
Access Channel
Reverse Channel Traffic
Sync Channel
Paging Channel
Forward Channel
Forward Dedicated Control Channel
Reverse Dedicated Control Channel
Broadcast Control Channel
Reverse Enhanced Access Channel
Forward Common Control Channel
EV-DO Signaling
NT11-17590, 1.0,8/29/2011
8.3
CDMA Scanner
The scanner is an important tool for the user to check coverage and pilot pollution on CDMA
technology.
A CDMA RF scanner is integrated into TEMS Symphony covering the 800 MHz and
1900 MHz frequency bands. The scanner can operate on CDMA2000 1x and EV-DO carriers
reporting a variety of measurements including:
CDMA RSSI scan
Pilot and Pilot TopN scan, with the possibility to highlight T_Add and T_Drop thresholds
Time domain scan
Walsh code scan
55
While it is performing the aforementioned QoS measurements for mobile data, TEMS
Symphony can simultaneously measure the engineering parameters in the air interface.
For WiMAX mobile (IEEE 802.16e) technology, supported measurement devices are data
cards based on chipsets of Beceem.
9.1
56
Network info
Physical metrics
Network Type
Network State
Subscriber Station State
Base Station ID
Operator ID
Cell ID
Mean RSSI
Mean CINR
Center Frequency
Channel
Bandwidth
TxPower
PER
UL Data Rate
DL Data Rate
NT11-17590, 1.0,8/29/2011
10
At the same time the QoS measurements described above are performed on mobile data,
TEMS Symphony can measure the engineering parameters in the air interface.
For LTE technology, supported measurement devices are data cards from Samsung and LG,
for the 2600 MHz and 700 MHz bands.
10.1
10.2
LTE Scanner
The LTE scanner is an important tool for checking LTE coverage, and identifying potential
sources of interference in the LTE band.
An LTE baseband scanner is available for the 700 MHz upper and lower bands. It is capable
of a variety of measurements, including:
Primary and secondary sync
SIR
Power
Ec/No
Cell ID
57
10.3
Continuous Expansion
Ascom Network Testings policy is one of continuous expansion, and therefore the company
reserves the right to change the functions in the TEMS Symphony system without advanced
notice.
58
NT11-17590, 1.0,8/29/2011
11
Electromagnetic compatibility
ENV 50204:1995
EN 55022:1998 + A1 + A2
CISPR 22:1997 + A1 + A2
2004/104/EG
EN 61000-4-2:1995 + A1 + A2
IEC 61000-4-2:1995 + A1 + A2
EN 61000-4-3:2002
IEC 61000-4-3:2002
ENV 50204
EN 61000-4-4:2004
IEC 61000-4-4:2004
EN 61000-4-6:1996 + A1
IEC 61000-4-6:1996 + A1
MIL-STD-810,M514:2000
EN 60068-2-56:1990
IEC 68-2-1
IEC 68-2-2
Environmental testing
IEC 68-2-6
Vibrations
59
EN55022:1998, class B
EN 55024
Electromagnetic compatibility
Niederspannungsrichtlinie
RL 89/336/EWG
Electromagnetic compatibility
RL 93/68/EWG
CE-Kennzeichnung
RL 95/54/EC
ECE 10 R-02
Car-mounted equipment
60
EU-WEEE
EU-RoHS
NT11-17590, 1.0,8/29/2011
12
Appendix
12.1
Glossary
AMPS
AMR
APN
(GPRS)
ARFCN
BCCH
BLER
BSIC
C/I
C1
(GSM, GPRS)
C2
(GSM, GPRS)
CDMA
Channel Report
CRC
CS
Circuit-switched
DCH
Dedicated Channel
DCT
Dedicated Report
Downlink
Normally:
Radio channel from base station to mobile phone.
With TEMS Symphony:
Slave transmits, Master analyses.
DPC Mode
DSP
Ec/Io
EDGE
ETSI
FACH
FER
FTP
(GSM)
(GPRS, WCDMA)
(GSM)
(WCDMA)
(Streaming)
(WCDMA Layer 1)
(WCDMA)
(IP)
61
62
GMM
GPRS
GPS
GSM
GUI
Handover
HCS
HSDPA
HSN
HSPA
HSDPA + HSUPA
(WCDMA)
HSUPA
(WCDMA)
HTTP
ID
Identifier
Idle Report
IP
ISDN
KB
LAC
LTE
Master
Master always makes the first call. Master can call a Slave/a response station
or set up a connection to a server.
MCC
MECCB2
Message Browser
MIA
MMS
MNC
MOC
MOS
ms
Millisecond
MTC
(WCDMA)
(IP)
(IP)
NT11-17590, 1.0,8/29/2011
MUC
NAS
Non-Access Stratum
NCC
P.862
P.862.1
P.862 mapped on the MOS rating scale (see ITU-T Rec. P.862.1).
Rating scale: 1 to 4.5
P.863
P-CCPCH
P-CPICH
PDP
PESQ
PEVQ
Ping
PMM
P.OLQA
PRACH
Proxy
Proxy servers provide protection against attacks from the Internet and
optimize the data transfer.
PS
Packet-switched
PSNR
PSTN
QoS
Quality of Service
QVP
QVS
R-Factor
RAB
RAC
RF
Radio frequency
RLC
RLC/MAC
RRM
(WCDMA Layer 3)
(GSM)
(Video tel.)
(IP)
(WCDMA)
(Video tel.)
(VoIP)
(WCDMA)
(GPRS)
63
64
RRC
(WCDMA Layer 3)
RSCP
RSSI
RxLev
RxQual
Second
SBC
SDU
Serving Cell
SIM
SIP
SIR
SIRA
SM
Session Management
SMS
SQL
TA
Timing Advance
To compensate the signal delay.
TBF
TCP
TDMA
TFCI
(WCDMA)
TPC
(WCDMA)
UARFCN
UDP
UMTS
Uplink
Normally:
Radio channel from mobile phone to base station.
With TEMS Symphony:
Master transmits, Slave analyzes.
URAid
URL
USB
UTRA
(IP)
(GSM)
(GPRS)
(IP)
(UMTS)
(IP)
(IP)
(UMTS)
NT11-17590, 1.0,8/29/2011
VoIP
Voice over IP
WAP
WCDMA
(IP)
65