White Paper

After 14 Years of Manual

Interoperability Testing,
Finally the Process has
been Automated

Teraquant Corporation
370 Stirrup Trail
Monument
CO 80132

ph: +1 719 488 1003

fx: +1 719 488 1005
www.teraquant.com

In partnership with
Automation of the Interoperability Test Process Page 2/12

Multivendor Interworking and the Interoperability Process

To build network based services such as Unified Communications and Web

Conferencing, best-in-class network devices are connected together in a multivendor
environment. Much effort is expended in Interoperability Testing to ensure devices from
different vendors interwork and provide seamless services. The testing workflow
process can be categorized as follows:

Development Tests (Unit, Module, Component, …)

Standardized Functional Testing (Testing against a standard)
Interoperability Tests
(Implementation against each other, n x m testing)
Performance Tests (High, long-lasting load)

Classical Conformance Testing, i.e. testing for conformance of protocol implementation

against an industry standard or specification, is losing ground due to the extensive effort
required to check all protocol message exchange scenarios. Conformance Testing uses a
piece of test equipment which generates all the test scenarios and checks for received
message syntax, timing etc. Once the test suite has been implemented in the testing,
process is automated and repeatable. However, the end goal is interoperability with
other vendor's devices in order to build a service. Interoperability testing, pair wise
testing, is accepted as a solution to improve interoperability and the standards bodies
and industry forums arrange regular events, where vendors may test interoperability
with fellow industry participants. Examples of such interoperability events include:

ETSI - Test Specifications, PlugTests

OMA – Test Specifications, Test Fest
SIP Forum – SIPit, SIMPLEt
IMS forum - Plugfests

Enter “Interoperability Testing” into a search engine and you will receive Over 700,000
hits on Google, more than 1.3 million hits with Yahoo.

Interoperability Testing involves inter-connecting two devices. Stimulation of these two

devices to implement the test plan is manual and automation and repeatability is
difficult to achieve. In addition, a more time consuming and intense phase is analyzing
the behavior of the devices afterwards i.e. which implementation between the two
devices is correct, according to the standard or RFC and which implementation is not
correct, therefore preventing interoperability.

© Teraquant Corporation and Testing Technologies ● All rights reserved ● www.teraquant.com

Automation of the Interoperability Test Process Page 3/12

Some InterOp events analyze each test or interaction between the two interoperating
devices to understand in protocol terms for correct implementation or which party did
not follow the recommendation. Other InterOp events just determine if the intended
test or session was successful e.g. the voice call went through or the message arrive
error free and analyze the details only in case of failures.

Where Costs Are Eaten Up In Interoperability Testing

Traces captured with tools like Wireshark are key to understanding why devices from
different vendors do not interoperate. They are used at most Interoperability Events
worldwide such as InterOp, Testfests & PlugFests. The problem is analyzing hundreds of
traces by visual inspection is manual, time-consuming, error-prone and requires an
expert.

Test
Design
< 50%
Test
Specification
Test
Efforts
Test Verification

> 50%

Test Implementation

Figure 1 – The current test effort problem

At a recent ETSI IMS InterOp Plugtest event,

[http://www.etsi.org/plugtests/ims2/IMS2.htm] the following analysis was conducted of
the time and effort expended during the test event:

The setting

43 InterOp test cases defined by ETSI

6 vendors participating at the Plugtest
30 test sessions
Approx. 900 test runs
In 3 days

© Teraquant Corporation and Testing Technologies ● All rights reserved ● www.teraquant.com

Automation of the Interoperability Test Process Page 4/12

The effort

150 hours of testing and (40%)

225 hours of validation of traces to analyze behavior (60%)
Total: 375 hours! (100%)
48 ManDays

Test design and test specification is the innovative phase that cannot be automated. The
test runs and verifying the results and looking at problems of unsuccessful test are the
phases occupying by far the most time and should be automated. The challenge is how
to achieve this time-saving, so we can concentrate on what should and should not be
tested, not how to test it.

Reducing Costs in Interoperability Testing

The key to efficiency and reduced time and costs is to increase reusability. Reusability
across different types of testing, reusability of information between System Architects,
Developers, Testers and Managers, reusability of test implementations and reusability
of know-how. This can all be captured for the purposes of computer automation in a
common test language across the different phases i.e. Development Tests, Standardized
Functional Testing, Interoperability Tests and Performance Tests. Such a test language
developed by the Standards Institutes is TTCN (Test and Testing Control Notation).
Having being used by the telecommunications and

networking industry for 24 years, TTCN is now in its third release, TTCN-3.

What is TTCN-3? TTCN-3 is a testing technology widely used by the Telecommunication,

Software, Automotive, Energy, Electrical Distribution Industry & other industries to
achieve test automation and repeatability.

The test language has been specifically developed for testing purposes, full-featured and
is open. i.e. allows for test scripts to be used on a variety of multivendor test platforms
supporting a TTCN-3 compiler.

TTCN-3 possesses a textual and graphical test scripting language look-and-feel and is
used to describe test purposes and also to implement test scripts as a scripting
language. Don’t reinvent the wheel! Use the test definitions produced in development
for Automation in your QA department. In addition, complete test suites useful here are
available for free download for example from the ETSI web site.

For more on TTCN-3, visit the ETSI web site

http://www.ttcn-3.org/Benefits.htm

© Teraquant Corporation and Testing Technologies ● All rights reserved ● www.teraquant.com

Automation of the Interoperability Test Process Page 5/12

Figure 2 - Reusability between different types of testing

So to optimize efficiency, automate the part of Interoperability Testing that takes most
of the time –Verification. We need to reuse existing test frameworks and Profit from
investments already done. Extensive Interoperability Test cases already exist for many
SIP, IMS, HSS, IPv6, SS7, SIGTRAN & WiMax scenarios, as mentioned above. And these
Interoperability Test scenarios exist in an industrial grade test automation framework
and mature, well accepted processes, workflows and tools. That framework is TTCN-3.

© Teraquant Corporation and Testing Technologies ● All rights reserved ● www.teraquant.com

Automation of the Interoperability Test Process Page 6/12

Figure 3 - The full InterOp test run at a glance

How Those Costs Savings were Proven

So how was this framework applied to the ETSI IMS Interop Plugtest event in Slovenia in
November, 2008? This solution addresses the heart of the problem.

© Teraquant Corporation and Testing Technologies ● All rights reserved ● www.teraquant.com

Automation of the Interoperability Test Process Page 7/12

Figure 4 - Configuration of the Trace Analyzer

At this last particular IMS Workshop [http://www.etsi.org/plugtests/ims2/IMS2.htm]

Theofanis Vassiliou-Gioles, CEO at Testing Technologies, talked about "How to increase
efficiency in interoperability testing by using test automation frameworks", presenting
methods to significantly reduce the time spent in verifying traces. The techniques were
put to the test at the event in Slovenia, successfully reducing the time to evaluate each
Interoperability Test up to 50% with 100% accuracy!

After twelve hours of trace reading, even the most versatile expert encounters his
limitations. This manual offline analysis of traces requires meticulous concentration, is
error-prone and depends on experts who are able to understand and validate content
and sequence of data captured in each interoperability session.

By using Automated Trace Analysis, not only is the time spent to evaluate the tests
reduced to minutes but the validation accuracy is always 100% compared to manual
analysis.

© Teraquant Corporation and Testing Technologies ● All rights reserved ● www.teraquant.com

Automation of the Interoperability Test Process Page 8/12

Figure 5 - Pinpointing to the problem with a single click

The configuration of the 30 test sessions were captured in 30 configuration files. Each
configuration file provided the necessary parameters for the 43 test scenarios. The
participating Devices under Test *DUT’s+ were interconnected and these test scenarios
initiated. Wireshark software monitors the message flow between the DUT’s. These
traces are imported into Testing Technologies’ TTworkbench which applies Automatic
Intelligent Trace Verification based on the same 43 TTCN-3 based test cases used for the
Plugtest event.

The Testing Technologies solution, built on top of the TTworkbench product, not only
automatically validates traces, but also visualizes the interoperability scenarios
graphically and produces human readable test reports. - All this, at a touch of a button.

The panel discussion at the ETSI IMS Workshop identified the lack of thorough
Conformance Testing as one of the main issues when evaluating interoperability test
results. However, it has also been accepted that classical Conformance Testing, as
performed in the days of ISDN, ATM, and GSM, is far beyond any economic reasoning.

© Teraquant Corporation and Testing Technologies ● All rights reserved ● www.teraquant.com

Automation of the Interoperability Test Process Page 9/12

Testing Technologies' goal is to bring two worlds together - the Conformance Testing
world with its typically high degree of test automation driven by a Tester Simulator and
the Interoperability Testing world, where devices are simply plugged together and usage
scenarios are used as test criteria.

Results of the Trial

The TTworkbench Automated Validation Solution has been successfully applied at the
2nd IMS Plugtest hosted by ETSI in Bled, Slovenia. Stephan Schulz, team member of ETSI
Plugtest, summarizes Testing Technologies' tool support as follows:

"All in all, we have been very satisfied with your support and performance of the tool.
We have been able to significantly speed up conformance analysis. Results from the first
test sessions included 100% matches of tool verdicts and manual analysis, and that is a
strong statement."

What did we Gain in Quantitative Terms?

The total time to run tests was not affected because that is still manual. However,
validation of the traces after the test session was drastically short circuited through the
Intelligent Auto Trace Analysis done by TTworkbench. The first time this was done, time
was allowed to tailor the Auto Trace Analysis. So total time to complete test execution
and validation benefitted by a time reduction of 25%. The second time running these
IMS test scenarios were used, time to execute and validate was reduced by 53% as the
Auto Trace Analysis tool can be reused as-it-is.

© Teraquant Corporation and Testing Technologies ● All rights reserved ● www.teraquant.com

Automation of the Interoperability Test Process Page 10/12

400

350

300

250
Total time for
200 validation
150

100 Total time for

50 execution

0
Manual Validation Automated Validation

Check out the website of the IMS Workshop to obtain presentations and meeting
minutes of workshops.

[http://www.fokus.fraunhofer.de/en/fokus_events/ngni/ims_ws_08/index.html]

400

350

300

250 Total time for

preparation
200 Total time for
validation
150
Total time for
100 execution

50

0
Manual Validation Automated Validation w Prep

© Teraquant Corporation and Testing Technologies ● All rights reserved ● www.teraquant.com

Automation of the Interoperability Test Process Page 11/12

How can this be used in a QA Laboratory?

As mentioned above, TTCN-3 Test Suites covering many of your QA test purposes are
available for free download from the ETSI web site. Ref
www.etsi.org/WebSite/technologies/ConformanceTesting.aspx

These test cases can be run on any TTCN-3 compiler (such as TTworkbench) to provide a
stimulus to a Device under Test (DUT) in a QA or Test lab, outside of an InterOp event,
without the involvement of the other DUT to be interoperated with. These Test Suites
prepare a device for the interoperability events, ensuring it has minimum conformance
capabilities and thus increasing the likelihood to interoperate with other devices. These
Test Suites also form the basis for test automation for Regression and Soak Testing
processes.

© Teraquant Corporation and Testing Technologies ● All rights reserved ● www.teraquant.com

Automation of the Interoperability Test Process Page 12/12

Further Information

TTworkbench is distributed in the USA & Canada by Teraquant Corporation, a Test

solutions and Test Systems Integrator based in Colorado. Teraquant provides highly
focused Test Solutions, integrating Best-in-Class test products from industry leaders to
meet specific test objectives and test plans. Teraquant works with many of the leading
Telecom Vendors and Service Providers, providing test automation and Network
Monitoring Systems for SS7, ISDN, wireless & VoIP.

Teraquant can be reached at admin@teraquant.com located at:

370 Stirrup Trail

Monument
CO 80132
ph: +1 719 488 1003
fx: +1 719 488 1005
www.teraquant.com

© Teraquant Corporation and Testing Technologies ● All rights reserved ● www.teraquant.com