Test Report

Multi-Vendor Carrier Ethernet

Interoperability Event

Madrid, September 2006

 Carrier Ethernet World Congress 2006
Editors Note
Since 1995, our multi-vendor
interoperability events have
always been technology
focused: ATM, MPLS, or Voice
over IP.
In contrast, Carrier Ethernet
has proven to be an interdisci-
plinary challenge exercis-
Carsten Rossenhoevel ing our knowledge in diverse
Managing Director
physical, transport, and appli-
cation layer technologies.
For this event, we have created a full-fledged MPLS
backbone, tested 10Gigabit Ethernet optics between
core router vendors, verified interoperability of
various new IEEE standards for Ethernet resilience
and OAM, and provided mock users with multicast
IPTV content.
Since our last interoperability test at CEWC 2005,
Carrier Ethernet marketing success has been
outstanding. Last year we saw a few showstoppers
for Carrier Ethernet deployment: lack of implementa-
tion of resilience and OAM mechanisms and lack of
end-to-end provisioning and fault management. It is
a very positive surprise to witness many improve-
ments in most of these areas:
Six vendors sucessfully tested multi-vendor
interoperability of Ethernet OAM "Dying Gasp"
(802.3ah) and continuity checks (802.1ag draft)
for the first time ever, worldwide.
The existence of pre-standards implementations
shows that vendors are eager to deliver OAM
and fault detection solutions in their Carrier Ether-
net products.
13 implementations passed the Ethernet Link
Aggregation (802.3ad) tests, proving that native
Ethernet link resilience is widely supported by
now. Carriers urged us to address native Ethernet
based node resilience in future events.
MPLS, a widespread deployed foundation for
large Carrier Ethernet networks, is a mature and
reliable solution. We did not see any issues what-
soever for basic Ethernet services with seven
vendors. Only small interoperability issues
remained in the MPLS OAM space (LSP ping).
Support for advanced traffic engineering in MPLS
is slowly growing (the standards are of
unequalled complexity); three vendors mastered
our very challenging DiffServ-TE tests.
We built a state-of-the-art backbone, aggregation
and edge network of massive scale and diversity to
2
Public Interoperability Event
support Carrier Ethernet services. The preparations
took three months and the hot-staging event kept
over 40 engineers from 16 vendors busy for eight
testing days. Over 50 devices, 300 cables, 2.5
metric tons of devices were installed and untold
amounts of coffee consumed
The interoperability event was very successful and
the experience gained is priceless. We hope you
find the results useful.
Introduction
Our interoperability events enclose two targets. For
the participating vendors the event is a unique
opportunity to test implementations against other
vendors. Vendors gather information about the
nature of any problem and in many cases receive
new code updates from developers before the event
is over. Interoperable vendors use the results to gain
footprint in new markets and display their level of
technological readiness. A second target is provid-
ing carriers and service providers with a realistic
picture of available technical solutions, potential
deployment issues and best practices network
design. In recent years carriers involvement in our
interoperability events has grown.
A resulting effect of the interoperability event is feed-
back to technical committees and standard bodies
regarding the readiness of standards for real world
deployment.
Figure 1: Hot-staging at EANTC, Berlin
The test plan was developed by EANTC between
June and August 2006. The test areas were finalized
after extensive review with participating vendors
and service providers.
Based on EANTCs experience in organizing and
executing multi-vendor interoperability events an
eight days, closed doors, hot-staging event was
conducted in EANTCs lab in Berlin, Germany. The
results are summarized in this white paper.
 Carrier Ethernet World Congress 2006 Public Interoperability Event

Participants and Devices Carrier Involvement

Germanys T-Systems and Colt Telecom reviewed the
Actelis Networks ML 130 test plan and provided detailed comments. Service
ML 628 provider participation ensured that the test areas
were realistic and matched their requirements.
ADVA Optical FSP 150CCf
Networking In addition, three T-Systems engineers participated
FSP 150M
for the duration of the whole EANTC hot-staging
Alcatel 7750 SR1 event. T-Systems engineers were responsible for
carrying out the service performance tests and the
7750 SR7
5620 SAM
10Gigabit Ethernet physical layer tests.
1850 TSS-40
Network Design
Ciena DN7100
The interoperability test network resembled a next-
Circadiant Systems OST-ST generation end-to-end carrier infrastructure designed
for residential IPTV, Triple Play, and business
Corrigent Systems CM-21 PTS
services.
CM-106V PTS
Given a total of 16 participating vendors, we
IXIA 1600T created a multi-vendor network with many interoper-
ability test opportunities. This amount of vendors
Huawei ME60 would be quite unusual in a real service provider
Technologies S8500 network; carriers often use heterogeneous Carrier
Ethernet networks from a few vendors typically
Juniper Networks M320 Multi Service Edge Router two vendors for the core, aggregation and CPEs.
M10i Multi Service Edge Router
All services in the test network were based on
Lucent Technologies Metropolis AMU Carrier Ethernet as specified by the Metro Ethernet
Forum (MEF) with technical standards as defined by
Metropolis ADM
the IEEE 802 committee and ITU.
MultiService Mux
The MEF is agnostic to the underlying network tech-
Ethernet Router 15202
nology used to deliver Ethernet services. The
Ethernet Router 15800 creation and support of such services is, however,
Ethernet Router 15102 an essential component of the interoperability test
event. Participating vendors were required to facili-
Nortel MERS 8600 tate the establishment of Ethernet services starting
RAD Data ETX-202
from their customer facing ports (User-Network Inter-
Communications face, UNI) over a diverse number of access technol-
RICi-8E1
ogies.
RICi-E1
Figure 2 illustrates the five logical parts of the
IPMUX-14
network that were the focus of the tests:
Egate-100

Redback Networks SmartEdge 400

Document Structure
Stratex Networks Eclipse Microwave Link
Participants Page 3
Telco Systems, T-Metro-100 Network Design Page 3
a BATM Company T-Metro-200 Test Areas Page 4
T-Marc-250 Interoperability Results Page 6
T-Marc-254 Topology Diagram Page 13
Problem Summary Page 14
Tellabs 6325 Edge Node
References Page 15
6315 Metro Ethernet Node

3
 Carrier Ethernet World Congress 2006 Public Interoperability Event

1. Backbone. Core infrastructure technologies developed a suite of tests to assess the ability of
suitable for transport Ethernet services and a variety of access devices (Aggregation and
service attributes were demonstrated here. We CPE) to seamlessly extend both business and resi-
evaluated the support for private line and trans- dential "triple-play" Ethernet services over
parent LAN services together with QoS and diverse access media including fiber and voice
Traffic Engineering, OAM, and multicast. grade copper infrastructure.
During the preparation phase, the technologies 4. Network Services Area. Triple play applica-
for the backbone were selected. Participating tions such as VoIP and IPTV (IP-Multicast based)
vendors suggested to use MPLS and Provider servers were positioned in this area.
Backbone Bridges. Both technologies were 5. Residential Customers. CPEs targeted at resi-
selected; multi-vendor interoperability tests in the dential users were connected to the edge of the
backbone were carried out over MPLS technol- network. A diverse number of access technolo-
ogy only. gies such as Ethernet in the first mile, Ethernet
2. Aggregation. Diverse technical solutions to over copper and Ethernet over E1 bundles were
implement aggregation and access networks for tested and demonstrated. Residential services
Carrier Ethernet services were demonstrated in were to be received from applications residing in
this area. The tests evaluated both user-network the network services area.
interface (UNI) interoperability at network cloud 6. Business Customers. MEF defined E-LAN
boundaries and technology-specific multi-vendor services were to be configured for usage by
interoperability. Specifically, we evaluated mock business customers. Business services such
VLAN-based Ethernet transport solutions, Ethernet as Voice over IP (VoIP) and Internet access would
provider bridges, provider backbone bridges, receive different levels of classification and will
Ethernet over resilient packet rings (RPR), Ethernet be treated in the backbone according to different
over SDH rings, and Ethernet-based wireless SLA levels.
rings.
3. Ethernet Access. The wide range of access
technologies and media for extending the reach
Test Areas and Test Plan
of Ethernet services has become the focus of the The number of test areas in our interoperability
Metro Ethernet Forum Objective 3 Ethernet events is being expanded every year: A growing
Access. This year, for the first time, EANTC number of vendors apply the Carrier Ethernet stan-
dards for different types of
products and applications.
This year, our tests ranged
Business/Residential Applications

from physical layer (10GigE

Demarcation/Managed CPEs

optical performance) up to
the application layer (IPTV
Aggregation
Network quality), from data plane
Services
Area (link aggregation) and
control plane (MPLS) to the
Backbone
management plane (OAM).
Aggregation This is very promising from
Services
the user point of view.
Carrier Ethernet is becom-
ing a complete network
Customers protocol family for end-to-
end networks.
Backbone (P) Router Business VPN Backbone and Aggregation
In detail, our tests covered
areas using various the following areas. Custo-
Backbone (PE) Router Set-top Box
technologies mer services were emulated
Aggregation Switch VoIP Phone using point-to-point Ethernet
10Gigabit Ethernet
Gigabit Ethernet Private Lines (EPL) and busi-
Demarcation / Managed CPE Fast Ethernet
ness-oriented Ethernet Trans-
parent LAN services at the
Figure 2: Schematic Network Design

4
 Carrier Ethernet World Congress 2006
periphery of the network. At the service provider
area Network Operations Center (NOC) functional-
ity was tested and residential services such as IPTV
were demonstrated while at the Carrier Ethernet
Network several technologies were simultaneously
used to depict the richness in possible ways of real-
izing MEF user services.
E-Line and E-LAN support: The Metro Ether-
net Forum (MEF) defines two fundamental Ether-
net service types. E-Line is defined as point-to-
point ethernet virtual connection while E-LAN is
defined as multipoint-to-multipoint virtual connec-
tion. These service types are used to create Ether-
net services.
The Ethernet Private Line (EPL) service is specified
using an E-Line service type and is created from
a point-to-point Ethernet Virtual Connection (EVC)
connecting two dedicated User-Network Inter-
faces (UNIs). Similarly Ethernet Virtual Private
Line is specified using an E-Line service type,
however, it does not require UNI exclusivity and
a lower level of transparency is expected from
the service provider.
Quality of Service. To properly cope with
congestion in the carrier network and to maintain
service-level agreements, mechanisms to priori-
tize traffic classes are required. This test area
was defined with three traffic classes in the core
of the network: one for voice; video; and Internet
traffic, and verified the different per-service quali-
ties facilitated by the traffic classes.
Traffic Management. Some Carrier Ethernet
transport technologies implement advanced
traffic management beyond traffic prioritization.
A test of Differentiated Services Traffic Engineer-
ing (DiffServ-TE) in the MPLS cloud was sched-
uled to verify class-based routing, preemption,
and prioritization of paths for E-LANs and E-Lines
based on administrative settings.
Operation, Administration and Maintenance (OAM)
support is required to operate a Carrier Ethernet
network efficiently. Recently, OAM protocols both for
Ethernet physical layer and for VLAN-based connec-
tivity have been defined. Our interoperability tests
were of these procedures were, to the best of our
knowledge, conducted for the first time ever.
Ethernet Physical Layer OAM. The IEEE
Standard 802.3ah-2004 is commonly referred to
as Ethernet in the First Mile (EFM). The document
defines services and protocols that are meant to
be used between stations in a subscriber access
network. Given the time constraints for the hot-
5
Public Interoperability Event
staging event a single function defined in the
IEEE 802.3ah was outlined for testing: Ethernet
Dying Gasp. Dying Gasp is a bit defined in EFM
OAMPDUs that informs a neighboring Ethernet
station that its peer station suffered a local unre-
coverable failure condition. The vendors partici-
pating in this test generated Dying Gasp
messages when the station was rebooted from
the management interface; when the station was
shutdown from the management interface and in
several cases when the power was disconnected
(using a a battery or a capacitor to enable the
station to send a Dying Gasp message before
shutting down).
This mechanism enables service providers to
extend their OAM domain all the way to the
customer premise and by monitoring first mile
OAM messages to offer the customers a high
level service quality and proactive customer
care.
Connectivity Fault Management. The IEEE
standard 802.1ag "Connectivity Fault Manage-
ment" is still in draft form, however, several
vendors expressed interest is testing their imple-
mentations against other vendors. Connectivity
Fault Management (CFM) defines capabilities to
detecting, verifying and isolating connectivity
failures in virtual bridges networks. The standard
provides both providers and customers with high
level of visibility into the status of their Ethernet
network. The protocol is meant to be used across
Ethernet domains and does not require physical
connectivity between the stations.
For the purpose of the interoperability event one
aspect of the CFM protocol was tested: Continu-
ity Check (CC). CC messages are used for
connectivity failure detection and are a useful
complement to the EFM Dying Gasp message.
Each station belonging to a maintenance domain
can be configured to periodically send continuity
check messages to other stations within its main-
tenance domain. When the station fails to
receive three consecutive CC messages or
receives a CC message with incorrect informa-
tion (Such as interval or maintenance domain ID)
the peer is considered faulty.
Service Performance. The MEF defines three
performance attributes for Ethernet Private Lines:
Frame Delay Performance, Frame Delay Varia-
tion Performance, and Frame Loss Ratio Perfor-
mance. The test plan called for service perfor-
mance verification from end to end.
Ethernet Link Aggregation. Carrier Ethernet
infrastructures use link aggregation primarily for
 Carrier Ethernet World Congress 2006
link resiliency of Fast Ethernet and Gigabit Ether-
net connections, by grouping of several inter-
faces into a single logical unit. If traffic across the
link aggregation group (LAG) is kept at half the
available bandwidth (by provisioning for exam-
ple) the LAG can guarantee that when a member
of the group fails, the rest of the LAG members
will carry the traffic load without permanent loss.
TDM Support. Legacy TDM application
support is important in a next-generation packet
network. We verified whether circuit emulation
solutions worked through the backbone. TDM
traffic has high requirements on latency, jitter,
and packet loss, so connectivity was established
using services with the highest quality of service
available.
Bandwidth Profiles at the UNI. Service
providers often use traffic classification and
bandwidth limitation enforcement (Committed
and Excess Information Rate) for different appli-
cations at the user-network interface (UNI) facing
the customer. These functions were verified
according to MEF standards per Ethernet Virtual
Connection (EVC).
10 Gigabit Ethernet Physical Layer
Performance. In order to confirm vendor
compliance to the 10GigE (IEEE 802.3ae) speci-
fication, a test of physical layer impairments and
simulated real world conditions was scheduled.
The effects of optical noise and signal degrada-
tion over long fiber links and aging optical trans-
ceivers were observed in relation to core network
rerouting and traffic engineering functions.
6
Public Interoperability Event
Interoperability Test Results
After eight intense days of hot-stage testing we
collected a substantial amount of results. The results
provide a detailed insight into the current status of
Carrier Ethernet solutions. The following sections
summarize our findings.
Results: Ethernet E-Line / E-LAN
Services
As in a real Carrier Ethernet network, point-to-point
and multipoint-to-multipoint services were created in
the test environment.
Point-to-point Ethernet private line (EPL) services were
created between mock customer pairs. The connec-
tions were created according to MEF6 which defines
EPL as a dedicated port for a single user on each
end of the Ethernet Virtual Connection (EVC). We set
up a total of 18 end-to-end virtual connections
between all the participating managed CPE and
demarcation devices. Each of the EVCs traversed the
backbone, so EVC support of the aggregation and
core network elements was tested at the same time.
In the aggregation network, EVCs were realized as
two-port VLANs. They were delivered to the
emulated customer premise equipment (CPE) without
tags. The Ethernet Provider Bridges standard (IEEE
802.1ad; Q-in-Q) was not employed due to
limited time. In the MPLS backbone, EVCs were
transported as Ethernet pseudowires see MPLS
section below for more details.
In addition, we configured E-LAN services with multi-
ple network access locations. Basic E-LAN interoper-
ability was reached between all CPE vendors includ-
ing across all participating access technologies.
There were no technical interoperability issues with
E-Lines; however, manual provisioning of each indi-
vidual E-Line on every node is a time consuming
process. A common edge-to-edge provisioning plat-
form based on standardized configuration mecha-
nisms would have sped testing in this multi-vendor
network and would have mostly eliminated configu-
ration issues.
Most configuration problems were associated with
Ethernet auto-negotiation, Maximum Transmission
Unit (MTU) mismatch, and VLAN tag mismatch
(where a vendor is supposed, but fails, to pop a
VLAN tag, so that the next hop accidentally stacks
multiple tags).
 Carrier Ethernet World Congress 2006 Public Interoperability Event

Results: MPLS Backbone Services

Lucent
Huawei 15800 Lucent
15202
ME60 Redback
Juniper SmartEdge 400 Huawei
Lucent
15800 M10i ME60

Lucent Ciena
15202 Alcatel
DN7100
1850
Alcatel
Juniper 7750 SR1
Alcatel M10i
7750 SR7
Alcatel
Alcatel 7750 SR7 Juniper
Alcatel 1850 TSS M10i
7750 SR1 Ciena
Redback Alcatel
SmartEdge 400 DN7100
7750 SR1
Alcatel
7750 SR7
Figure 3: MPLS RSVP-TE Signaling
Connections
MPLS Router Ethernet Pseudowire

Routing and Signaling. After all core routers
had been installed, the MPLS backbone was quickly
established. RSVP-TE was used for transport signal-
ing, OSPF-TE as the routing protocol. Most MPLS
devices participating in this test were used both as
PE (Provider Edge) and P (Provider) routers,
forwarding and terminating label-switched paths.
There were no connectivity issues for basic signaling
and routing interoperability between Alcatel 1850
TSS, Alcatel 7750, Ciena DN7100, Huawei ME60,
Juniper M10i, Juniper M320, Lucent 15800, Lucent
Figure 4: MPLS Pseudowires
In addition, we established a BGP-based VPLS
domain for the same three services. Huawei ME60
and Juniper M10i participated in these services.
Note that BGP Router Reflectors can be used to
establish a hierarchy in order to scale the control
plane for BGP-VPLS (this has not been tested due to
limited time).
Telco Systems
Lucent
T-Metro

15202, and Redback SmartEdge 400. 15102 Lucent

15800 Lucent
Alcatel 15202
MPLS Point-to-Point And Multipoint 1850 TSS Redback Huawei
SmartEdge 400 ME60
Services. Ethernet pseudowires were preconfig-
ured for all E-Lines required in the end-to-end H-VPLS LDP
MTUs Alcatel BGP
network. There were no interoperability limitations 7750 SR7
VPLS
VPLS
other than configuration issues in the beginning. All Juniper
M10i
Huawei
pseudowires were successfully established. ME60
Juniper
M10i
Three Virtual Private LAN Services (VPLS) domains Telco Systems Alcatel Alcatel
7750 SR7
were created in the MPLS backbone for multipoint T-Metro 7750 SR1

triple play services video, voice, and Internet Alcatel

access. These domains were interconnected with the
E-LAN services in the aggregation and edge areas.
We used two different IETF-specified protocols for
VPLS service creation: LDP and BGP. Alcatel,
Huawei, Lucent, Redback, and Telco Systems partici-
pated in the LDP-VPLS service. A hierarchical
domain was created with H-VPLS to improve network
scalability and to demonstrate the different roles of
the systems under test. The Alcatel 1850 TSS,
Alcatel 7750 SR1, Huawei ME60, Lucent 15102,
and Telco Systems T-Metro participated as Multi-
Tenant Units (MTUs) while Alcatel 7750 SR7, Lucent
15800, Lucent 15202, and Redback
SmartEdge 400 took the roles of VPLS PE devices.
7750 SR1
Telco Systems
T-Metro
Figure 5: VPLS Domains
All VPLS services were quickly configured and full-
mesh connections were established, proving that
VPLS is a well-understood and widely interoperable
industry solution by now. The only fundamental issue
is that BGP-VPLS and LDP-VPLS are incompatible
because of the different signaling protocols used. As
a temporary solution, we established a single
pseudowire per broadcast domain interconnecting
the LDP- and BGP-VPLS. The Alcatel 7750 SR7
(Video domain) and Redback SmartEdge 400
(Voice domain) provided the pseudowire intercon-

7
 Carrier Ethernet World Congress 2006 Public Interoperability Event

nection on the LDP side and the Juniper M10i on the
BGP side. The interconnection worked well.MPLS
Connectivity Verification
LSP ping tests were carried out in the MPLS back-
bone with Alcatel 7750 SR1 and SR7, Ciena DN
Maximum Allocation Model (MAM) while others
used the Russian Dolls Model (RDM). Juniper was the
only vendor supporting both models in the same
implementation.

Bandwidth
Constraint
7100, Huawei ME60, Juniper M10i, Juniper M320,

Resv Request
(Mbit/s)
Lucent 15202, Lucent 15800, Redback

Setup/Hold
Traffic Type

Traffic Eng.

Tunnel BW
Class Type
SmartEdge 400 and Telco Systems T-Metro. See

(Mbit/s)
Priority
figure 6 for a detailed logical test topology.

classes
We discovered a total of three interoperability MAM RDM
issues; this calculates to a 88 % success rate and is
a great improvement over the last MPLS LSP ping CT0 EFa TE0 0/0 450 900 10
tests we carried out a year ago.
TE1 1/1 50
Test System Attachment. The Ixia 1600T was
connected with Gigabit Ethernet to every provider TE2 2/2 100
edge router and full-mesh connectivity within the
VPLS domain was successfully verified. CT1 AF1b TE3 3/3 450 450 200

Ciena a. Voice traffic

DN7100
Lucent b. Video traffic
Lucent
15800
15202

Alcatel
One tunnel was configured per TE-class for each
7750 SR-1 direction and each link, resulting in a total of eight
tunnels (four per direction) between each pair of PE
routers. As a result of the network configuration,
Alcatel Juniper tunnels of different priority had to compete for the
7750 SR-7 M10i
allocated bandwidth and used different paths
through the backbone. We verified that the path
selection worked properly.
Redback Huawei
SE400 ME60

Juniper Telco Systems

M320 T-Metro
MPLS Router Successful LSP Ping
Figure 6: MPLS Ping
Results: Quality of Service and Traffic
Engineering
Ciena DN7100, Juniper M10i, Juniper M320 and
Huawei ME60 participated in the DiffServ-TE tests
and successfully established DiffServ-TE enabled
tunnels.
DiffServ-TE is complex to implement and suffers from
many options. Its complexity led to a misunderstand-
ing in the implementation of OSPF-TE details in one
case. The fact that the IETF specified three different
allocation models did not help interoperability either
all network elements should implement the same
allocation model if DiffServ-TE is to be used. Unfortu-
nately some participating vendors implemented
Results: Aggregation and Access
Services
As expected, all devices in the multi-vendor VLAN-
based (IEEE 802.1q) aggregation network interwor-
ked smoothly. There were only few configuration
issues related to Ethernet auto-negotiation (we
believe it should always be disabled between
network elements).
In the CEWC 2005 event, multi-home connection of
VLAN clouds using the Spanning Tree protocol was
a major source of headache. This year we accepted
that Spanning Tree is not a carrier-grade solution
and connected all equipment to the MPLS backbone
in a star topology.
A substantial number of carrier-class aggregation
technologies interworked with VLAN-based infra-
structures in the test network (in no specific order):
Carrier Ethernet Services over RPR over
SDH. Corrigent used a packet-optimized optical
transport solution using standards-based Resilient
Packet Ring (RPR) technology (IEEE 802.17),

8
 Carrier Ethernet World Congress 2006
combining standard MPLS with RPR to provide a
bandwidth-efficient universal transport solution.
The CM-106V PTS and CM-21 PTS provided
connectivity through the MPLS core for Ethernet
services supporting multiple applications. The Corri-
gent Packet Transport Solution participated in the
service performance tests.
Ethernet over SDH rings. Lucent ADM and
Tellabs 6325 demonstrated Gigabit Ethernet over
SDH using GFP (Generic Framing Procedure, ITU-T
G.7041) and LCAS (Link Capacity Adjustment
Scheme ITU-T G.7042). LCAS allows dynamic
change of bandwidth if, for instance, impaired SDH
links cause some virtual containers (VCs) to fail.
This was tested in an STM-16 ring between Lucent
ADM, Lucent AMU, and Tellabs 6325 switches. For
extra resiliency some of the VCs were SNC
protected over an STM-4 link.
T-MPLS rings. T-MPLS based ring protection was
used between the two Alcatel 1850 TSS participat-
ing in the test. The solution is a pre-standard imple-
mentation based on a draft ITU-T recommendation.
T-MPLS ring protection further extends resilience
capabilities. The Alcatel 1850 TSS participated in
interoperability tests both as an Aggregation Switch
and as a PE Router.
For the same purpose of increasing the network resil-
ience, Telco Systems employed a ring solution based
on H-VPLS and MPLS rerouting protocols.
Resilient Wireless Packet Ring. Stratex
Networks took part in the EANTC Interoperability
test with their Wireless Ethernet Product Eclipse.
Three Eclipse 18 GHz Microwave Radios were
deployed as an access ring with a capacity of
155 Mbit/s carrying native Ethernet traffic.
Each node was configured to operate first as an
Aggregation device and then as a CPE. Wireless
ring protection was configured using Stratex
Networks Resilient Wireless Packet Ring Protocol
(RWPR). The Eclipse Ring participated in link aggre-
gation and service performance tests.
Ethernet in the First Mile Over Copper. A messages were sent by CPEs upstream to the aggre-
bonded group of eight pairs of voice-grade copper
was connected between an Actelis ML 130 Carrier
Ethernet over Copper aggregation switch and an
Actelis ML 600 Ethernet Access Device (EAD), to
create a 45 Mbit/s Ethernet service to residential
subscribers.
We verified that the Ethernet over Copper service
was able to support the high performance, low
latency service requirements for successfully deliver-
9
Public Interoperability Event
ing the IPTV video stream to a set-top box connected
to the EAD and the RAD IP-MUX Circuit Emulation
over Ethernet device which was also connected to
the ML 130.
Results: OAM Ethernet in the First Mile
Five vendors participated in the first public multi-
vendor IEEE 802.3ah test ever. ADVA, Huawei,
RAD, Telco Systems and Tellabs provided several
implementations of physical layer OAM.
According to the standard, devices were assigned
active or passive roles. Typically, the aggregation
device will serve as an active station, initiating the
OAM discovery process, sending variable requests
and loopback control OAMPDUs. The CPE device is
set to passive mode, enabling it to participate in the
discovery mechanism and to send event notifica-
tions.
Telco Systems
T-Metro
Telco Systems
CPEs T-Marc
Huawei
ME60
ADVA
FSP 150CCf
Huawei Access
S8500
RAD
ETX-202
Tellabs
6315 ADVA
FSP 150M
Tellabs
6325
Neighbor Detection Dying Gasp Messages
Figure 7: OAM Physical Layer Tests
We were able to complete the discovery process
between all but one paired devices in the test. In
several of the test pairs, both stations had to be
configured in active mode and in one instance both
stations were able to exchange OAMPDUs only
when both were set to enhanced mode (to support
organization specific OAMPDUs).Dying Gasp
gation network upon an electrical power loss. (Typi-
cally, only CPEs are expected to generate these
messages.) Once OAMPDUs were exchanged
between the two stations, the CPE was shutdown
and the aggregation device was expected to inform
the operator about the neighbors condition. All
devices receiving dying gasp messages notified the
user by way of log messages, command line notifi-
cation or SNMP traps.
 Carrier Ethernet World Congress 2006 Public Interoperability Event

The level of basic OAM 802.3ah support was prom-
ising given the fact that the standard is still young.
We look forward to seeing more complete 802.3ah
implementations in the future.
Results: VLAN-Based OAM
Connectivity Messages
In addition to physical layer OAM we evaluated
Ethernet-based connectivity messages as defined in
IEEE 802.1ag. Nortel, RAD, Telco Systems, and
Tellabs participated in these tests. It is important to
mention that 802.1ag is still in the draft status.
Results: Service Performance
With support from T-Systems, we carried out end-to-
end service performance tests over the complete
network and all technologies deployed.
The measurements were carried out in accordance
with MEF specifications. An Ixia 1600T load gener-
ator was used to emulate bi-directional customer
traffic on point-to-point Ethernet Private Lines (EPLs).
Packet sizes of 64, 512, and 1500 Bytes (without
Ethernet headers) were used to verify service perfor-
mance for small and large packets and to ensure
there were no MTU size issues.

The challenge in testing CFM proved to be in the
many configuration parameters and options defined
in the draft standard. In addition the exact version of
the draft and the corresponding ITU document
(Y.1731) added complexity to the testing.
Several vendors implementations offered predefined
configured parameters that were not tunable. For
example, a simple predefined Continuity Check
(CC) message interval parameter set to 100 ms in
one vendors implementation did not allow testing
against another vendor who had a predefined CC
message interval of one second. This issue and a
few further early implementation problems were
resolved during the hot-stage event. In addition, RAD
Service performance is unrelated to backbone or
device performance. In this interoperability test,
different transport technologies limited the end-to-
end bandwidth. We measured all end-to-end rela-
tionships with a fixed bandwidth of 5 Mbit/s per
direction.
The table below summarizes the results of 17 Ether-
net Virtual Circuits (EVCs). Each of the end-to-end
flows traversed at least four, in average seven
switches.
End-To-End Latency

Low BW
demonstrated 802.1ag-based performance Packet Packet

GigEth
measurements based on the MEF defined parame- Size Loss

Max

Max
(ms)

(ms)

(ms)
Min
ters for Delay, Delay Variations and Frame Loss. We (Bytes) (%)
were impressed with the level and commitment to the
success of the 802.1ag related tests. 64 0.0 0.04 0.16 1.00

512 0.0 0.13 0.63 1.94

Telco Systems
Nortel T-Marc 1500 0.0 0.32 0.74 6.80
8600

The minimum latency across the backbone was

Telco Systems RAD
T-Metro ETX-202 outstanding (0.04 ms), but even the maximum
latency at Gigabit Ethernet edges was far below
RAD Tellabs expectations (0.74 ms). This result shows that multi-
RICi-8E1 6325
vendor Carrier Ethernet backbones and aggregation
networks can keep up in service quality with any
Continuity Check Messages other legacy network technologies.
At the edge, the majority of results were similar to
Figure 8: OAM Connectivity Tests the core. The maximum delay of a few EVCs was
influenced by physical layer multiplexing technolo-
gies (Ethernet over n x E1) or by radio network
Lucent ADM provided proprietary pre-standard
access latency. However, even over an E1 link the
802.1ag OAM used for round trip delay measure-
maximum latency was 6.80 ms which is still much
ments, Ethernet loopbacks and continuity check
below xDSL latency.
within the Lucent MSPP domain.

10
 Carrier Ethernet World Congress 2006
Results: Bandwidth Profiles
The Metro Ethernet Forum defines Bandwidth Profile
as the characterization of the lengths and arrival
times for ingress service frames at the User Network
Interface (UNI). In the Ethernet Services Attributes
Phase 1 technical specification (MEF10) three
models are defined for the level of granularity at
which UNI bandwidth profiles are applied.
For the purpose of this test we chose one of the
models -- Bandwidth Profile per EVC. In this model
each EVC across the UNI is applied with a different
Bandwidth Profile. For service providers, this level of
granularity allows each customer to receive a differ-
ent level of service while all customer connections
traverse the same access device.
The goal of the test was to show that participating
vendors can apply different Bandwidth Profiles per
EVC. In our case two EVCs were configured across
each capable UNI, each with a different bandwidth
profile.
Vendors were instructed to configure two parameters
for the Bandwidth Profile Committed Information
Rate (CIR) and Excess Information Rate (EIR). Once
the Bandwidth Profiles were configured and applied
to the EVCs we generated traffic across both EVCs
and measured the loss rate. We expected loss at the
aggressive Bandwidth Profile and zero loss at the
second Bandwidth Profile.
All participants that could apply a per-EVC Band-
width Profile passed the test. Some of the partici-
pants could not apply a per-EVC bandwidth profile,
however, these vendors products are aimed at
different network topologies or market segments, all
of which would enforce bandwidth profiles at differ-
ent areas of the network.
The parameters to be configured for Bandwidth
Profile have also proven to be difficult to translate
from the MEF technical specifications to various
vendors implementations. The concept of EIR has, in
more than one case, been consumed by the aggre-
Traffic Generator
Provider Edge
CPE Aggregation
EVC With Bandwidth Profile
Figure 10: Bandwidth Profile Reference Model
Public Interoperability Event
gation of Committed Information Rate and Peak
Information Rate (the difference between the two
resulting in EIR), lacking an implicit definition of EIR.
Results: Link Aggregation
We tested link aggregation according to IEEE
802.3ad for the purpose of link resiliency.
Link Aggregation Groups (LAG) containing pairs of
Gigabit Ethernet interfaces were defined between
participating vendors. Once the pairs established
connectivity, we used the Ixia 1600T traffic genera-
tor to send traffic at 25% of the aggregated link
capacity. Redundancy switchovers were forced by
disabling each of the links in turns.
All participating devices passed the test in general:
Alcatel 1850 TSS, Alcatel 7750 SR1 and
7750 SR7, Corrigent CM-106V PTS,
Huawei ME60, Juniper M320, Lucent 15102,
Lucent 15202, Lucent 15800, Lucent ADM,
RAD ETX-202, Redback SmartEdge 400, Stratex
Networks Eclipse, and Telco Systems T-Metro.
We noticed differing switchover times between
vendors on physical link disconnection. Packets
were lost for intervals between 0.5 ms and 99 ms
while the ingress switch was calculating the new
link. The rerouting time is related to the hash algo-
rithm that controls traffic distribution to each of the
links. These aspects should be investigated further.
In one case, Ethernet auto-negotiation was miscon-
figured over the link aggregation group: It was
enabled on one side and disabled on the other. This
situation resulted in a rerouting time of 372 ms, until
both ends had (correctly) completed the timer-based
semi-autonegotiation. Again we recommend to
disable Ethernet auto-negotiation on network links.
Standards committees should consider making this a
default.
Provider Traffic Generator
Provider Edge
Aggregation CPE
Network
Core
EVC Without Second Bandwidth Profile
11
 Carrier Ethernet World Congress 2006 Public Interoperability Event

Results: TDM over Ethernet

TDM over Ethernet connection (as per MEF 8) was
created between the RAD IPMUX-14 and Telco
Systems T-Marc. An Ethernet pseudowire with high
priority (Class Type 0, see MPLS) was created to
forward encapsulated E1 frames. The connection
worked properly.

Results: Impact of Physical Layer

Impairments on 10GigE Systems
In order to confirm vendor compliance to the IEEE
802.3ae specification a Circadiant Systems Optical
Standards Tester (OST-ST) was connected in-line into
the core network. This provided the facility to create
physical layer impairments and simulate real world
conditions in the captive lab environment.
Stressed eye conditions were applied, see the
figures below.
Figure 12: 10GigE Stressed Optical Eye
(Resulting in Packet Loss)
All tested 10 Gigabit Ethernet transmitters and
receivers were confirmed to exceed the requirements
of the IEEE 10GigE standard. For example, the
receiver of the link under test allowed for an extra
budget of 3 dB for the vertical eye closure penalty
(VECP) before packet loss occurred, as shown in
figure 13.

Frame Loss vs VECP

Lost Frames
10000000
1000000
100000
10000
1000
100
10
1
1.129 1.82 2.68 3.4 4.7 6.97 8.3 9.6
IEEE 802.3ae Threshold2.7dB VECP (dB)

Figure 11: 10GigE Optical Eye

(Stressed but Acceptable) Figure 13: Frame Loss versus Vertical Eye
Closure Penalty (VECP)
At the same time we ran packet throughput and loss
measurements. Margin testing was also made by
increasing stress beyond the specification require-
ment to the point where packet loss occurred, and
ultimately rerouting took place (see figure 12).

12
 Corrigent IXIA 1600T
CM-21 PTS

RAD RICi-E1 Corrigent RAD

Stratex ETX-202
CM-106V PTS Eclipse Stratex
Eclipse
MPLS-RPR RWPR
Telco Systems Adva
OC-192 FSP 150CCf
T-Marc-250

Tellabs 6325 Corrigent Stratex

MTU CM-106V PTS Eclipse Telco Systems

Aggregation Switch
Backbone (P) Router
Telco Systems T-Metro-200 RAD

Backbone (PE) Router

T-Metro-100 RICi-8E1
Telco Systems
T-Marc-254 RAD
Alcatel
OP1551

Demarcation / Managed CPE

1850 TSS Alcatel Huawei
T-MPLS 7750 SR7 S8500
RAD Adva
Carrier Ethernet World Congress 2006

Ring MTU Egate-100 FSP 150M

Alcatel
1850 TSS MTU
Lucent 15102
Huawei Lucent
Final Integrated Test Network

IXIA 1600T ME60 15800 Adva

Juniper FSP 150CCf
M10i MTU
Telco Systems

13
Circadiant
T-Marc-254 OST-ST Juniper Alcatel
Adva FSP 150M M10i 7750 SR1
Lucent

VoIP Phone
Set-top Box
Nortel ADM
8600 Juniper Alcatel

Video Headend
Redback M320 7750 SR7 Actelis

Traffic Generator
SmartEdge 400 Lucent STM-16 ML628
AMU SDH
RAD ETX-202 Lucent Ring
Alcatel 15202 Tellabs
Nortel 8600 7750 SR1 6325
Telco
MTU Systems
Tellabs Ciena Lucent
H-VPLS T-Metro-200
6315 DN7100 Ring AMU

E1
Actelis Nortel
IXIA 1600T

Other
RAD ETX-202 ML130
8600

CPE Area
Telco Systems
Telco Systems T-Metro-200

Fast Ethernet
RAD ETX-202
T-Marc-250

Gigabit Ethernet
Actelis

Aggregation Area

10Gigabit Ethernet
ML628
Nortel 8600
RAD IXIA 1600T

Backbone/Provider Edge
IPMUX-14
Public Interoperability Event
 Carrier Ethernet World Congress 2006 Public Interoperability Event

Problem Summary
Problem Description Temporary Recommendation
Area Solution, if any

MPLS DiffServ-TE Order of RSVP-TE objects in the RESV Tolerate different Clarify RFC2205 with regards to
message matter for some implementa- order recommended object order
tions

Compatibility MAM / RDM not clearly IETF should state in how far MAM
defined and RDM can be used together

Vendors implement different allocation IETF should invalidate all but one
models allocation model

MPLS Ping Router Alert flag not taken into account Fix code
when sending replies

Ethernet Link Aggregated link connections converge Disable auto-negoti- IEEE should recommend to
Aggregation very slowly (> 500 ms) when auto-negoti- ation on all network disable auto-negotiation for GigE
ation is involved links / 10GigE connections per
default

Not all aggregated links are used in a Implement multiple hash algo-
balanced way rithms for different network appli-
cations

Fault Maintenance domain name format was Implement a configu- Resolve preferred formats
Management incompatible and not configurable. IEEE ration option between IEEE and ITU
802.1ag prefers format #2; ITU supports format
#32.

Continuity check (CC) transmit interval Add more flexibility to implemen-

was different and not configurable tations of CC parameter control

OAM Fault Several stations could only act as Active Implement passive role for all
Management role CPE equipment
802.3ah
Some stations could not generate Dying Add support for Dying Gasp for
Gasp messages all CPE equipment as this is one
of the main reasons to implement
802.3ah

Bandwidth Parameter units defined wrong (CIR = Implement EIR according to MEF:
Profiles 2 Mbit/s requires EIR 2 Mbit/s) As additional bandwidth (in
excess of CIR)

Small EIR/CIR values not supported MEF Ethernet Service Attributes

(128 kbit/s) spec requires support

Ingress/per-VLAN policing not supported

E-LAN Number of E-LANs per uplink stream MEF should define minimum
limited to 1 (one) performance recommendations
as part of the certification
program

14
 Carrier Ethernet World Congress 2006 Public Interoperability Event

Acknowledgments
We would like to thank Ralf-Peter Braun, Thomas Kessler, Manuel Paul, and Sabine Szuppa from T-Systems for
the extensive support during the hot-staging event, guidance during test plan development and network design.
This document has been edited by Gabriele Schrenk, Carsten Rossenhoevel, Jambi Ganbar, and
Sergej Kaelberer (EANTC).

References
Ethernet Services Attributes Phase 1 (MEF 10)
Requirements and Framework for Ethernet Service Protection in Metro Ethernet Networks (MEF 2)
Metro Ethernet Network Architecture Framework - Part 1: Generic Framework (MEF 4)
Ethernet Service Definitions (MEF 6)
Abstract test suite for Ethernet Services at the UNI (MEF 9)
Abstract Test Suite for Traffic Management Phase 1 (MEF 14)
Media Access Control (MAC) Bridges (IEEE 802.1D)
Media Access Control (MAC) Bridges Amendment 2:Rapid Reconfiguration (IEEE 802.1W)
Virtual Bridged Local Area Networks (IEEE 802.1Q)
Link Aggregation Control Protocol (IEEE 802.3ad)
10Gb/s Ethernet Task Force (IEEE P802.3ae)
Carrier Sense Multiple Access with Collision Detection (CSMA/CD) Access Method and Physical Layer
Specifications. Amendment: Media Access Control Parameters, Physical Layers, and Management (IEEE Std
802.3ah-2004)
Virtual Bridged Local Area Networks - Connectivity Fault Management (IEEE 802.1ag/D7)
OAM functions and mechanisms for Ethernet based networks (ITU-T Y.1731)
Traffic Engineering (TE) Extensions to OSPF Version 2 (RFC 3630)
RSVP-TE: Extensions to RSVP for LSP Tunnels (RFC 3209)
Pseudowire Setup and Maintenance Using the Label Distribution Protocol (LDP) (RFC 4447)
BFD For MPLS LSPs (draft-ietf-bfd-mpls-03.txt)
Bidirectional Forwarding Detection (BFD) (draft-ietf-bfd-base-05.txt)
Detecting Multi-Protocol Label Switched (MPLS) Data Plane Failures (RFC 4379)
Protocol Extensions for Support of Diffserv-aware MPLS Traffic Engineering (RFC 4124)
Requirements for support of Diff-Serv-aware MPLS Traffic Engineering (RFC 3564)
Requirements for Traffic Engineering Over MPLS (RFC 2702)
Maximum Allocation Bandwidth Constraints Model for Diffserv-aware MPLS Traffic Engineering (RFC 4125)
Russian Dolls Bandwidth Constraints Model for Diffserv-aware MPLS Traffic Engineering (RFC 4127)

15
EANTC AG IIR Conferences
European Advanced Networking Test Center

Einsteinufer 17 29 Bressenden Place

10587 Berlin, Germany London SW1E 5DR, UK
Tel: +49 30 3180595-0 Tel: +44 (0) 20 7017 7483
Fax: +49 30 3180595-10 Fax: +44 (0) 20 7017 7825
info@eantc.de registration@iir-conferences.com
www.eantc.com www.carrierethernetworld.com

This report is copyright 2006 EANTC AG. While every reasonable effort has been made to ensure accuracy and
completeness of this publication, the authors assume no responsibility for the use of any information contained herein.
All brand names and logos mentioned here are registered trademarks of their respective companies in the United
States and other countries.