Вы находитесь на странице: 1из 7

Home . September 2010 Issue . Cover Story .

 Ethernet for IEC 61850

Ethernet for IEC 61850


Authors:
Roger Moore, RuggedCom, Canada and Maciej Goraj, RuggedCom, Spain
INTRODUCTION
Substation Communication Networks
This article looks at the key issues and considerations when designing an Ethernet network for
substation automation applications. Specific topics addressed are environmental robustness, copper
cables and fiber optics, network switching and latency, topologies, redundancy, quality of service, virtual
LANs, priority tagging for GOOSE and Sampled Values, operations and maintenance, and migration
strategy for connecting legacy device to substation LAN.
Environmental Robustness
A key requirement of most substations IEDs and the Ethernet LAN is that they must operate properly
under the influence of a variety of EMI phenomena commonly found in the substation. IEC 61850-3
specifies a variety of type withstands tests designed to simulate EMI phenomena such as inductive load
switching, lightening strikes, electrostatic discharges from human contact, radio frequency interference
due to personnel using portable radio handsets, ground potential rise resulting from high current fault
conditions within the substation and a variety of other EMI phenomena commonly encountered in the
substation.
IEEE 1613, a standard for “Environmental and Testing Requirements for Communications Networking
Devices in Electric Power Substations” goes one step further by defining “Class 2” operation that
requires no communications errors, delays or interruptions occur during the application of the type tests.
Often the Ethernet switches will be installed in the same compartment or even on the same rack as
protective relaying IEDs. Therefore, it is necessary that the Ethernet equipment be “substation
hardened”, from an EMI immunity perspective, to the same level as protective relaying IEDs. The need
for environmental robustness becomes extremely imperative when a LAN based tripping schemes via
GOOSE is implemented; one lost message could be the difference between success and failure. The
designer of the automation system must ensure that Ethernet equipment vendors demonstrate
conformance to IEC 61850-3 type tests.
The use of Fiber or Copper at Physical Media Level
Ethernet supports both fiber optics and the ubiquitous CAT5/CAT6 cabling and RJ45 connector
combination seen in virtually every home and office around the globe. The difficult decision facing the
designer is when to use fiber and when to use copper. The technical advantages of fiber optics are many
ranging from immunity to electrical interference and the ability to span long distances and maintain
immense bandwidth for packet hungry technologies like video streaming.
The designer of the system must weigh up front cost versus reliability and criticality of the electrical
system being protected. A compromise sometimes made is to use copper interconnections within a
substation bay to connect IEDs to switches and to use fiber to connect the switches between bays. This
is however specific to each utility, some may have strict requirement that all Ethernet connections at
process and bay level must be fiber optic and only station devices such as RTUs, gateways or
substation computers are allowed to use copper.
Once a decision is made for where to use fiber, there are still decisions to make regarding the type of
fiber, connectors, and transceivers. There are two basic types of fiber: multi-mode and single-mode. The
former can use inexpensive LEDs to impart light onto the cable but has bandwidth and distance
limitations: 2 km at 100 Mbps, and 300 m at 1Gbps. The latter requires a higher quality laser light but
allows almost infinite bandwidth and distances exceeding 100 km. Multi-mode fiber is generally suitable
within the substation for the majority of applications and is often used. However, the general trend is
towards single-mode as cost of single-mode fiber is actually less, and single-mode provides some
degree of ‘future proofing.’ Connector choice is another item on the menu as vendors of IEDs support a
variety ranging from ST, SC, LC, and MTRJ. The MTRJ and LC connectors have two major benefits over
ST and SC connectors as they permit higher port density and avoid typical mistakes when RX and TX
are wrongly connected. The trend of the future in fiber optic substation LAN seems to be LC connector.
Another interesting standard worth mentioning is IEEE 802.3 100BaseBX which is based on bidirectional
transmission of light signal over a single fiber. This solution uses two different wavelength for data
transmission in each direction, being 1.3 and 1.5 um. Each transceiver has special lenses that permit
separation of wavelengths between transmission and reception. IEEE 802.3 100BaseBX permits the
reduction of fiber cables by 50% but its support in substation IEDs is very limited.
Functions of an Ethernet Switch
All 61850 IEDs must be physically connected to an Ethernet switch. Ethernet is a packet based
communications technology where an IED may start transmitting a data packet at any time. The function
of a switch is to prevent collisions of these packets and to send the packet in the direction of the desired
recipient. This is done using the descriptively named ‘store and forward’ process where received packets
are buffered in memory on ingress, placed in a queue for the egress port, and then transmitted once the
packet reaches the front of the queue. It is the queuing mechanism that eliminates collisions and allows
full duplex operation.
This is in contrast to repeaters or hubs of the past that used CSMA/CD to detect that a collision occurred
and then retransmitted a random amount of time later. Determination of the egress port is done via MAC
address lookup and learning of addresses which makes this entire operation automatic. The basic
functionality described above can be found in what is termed an unmanaged switch.
A managed switch offers additional functionality for managing and optimizing the network. Some of these
features include:

◾ User interface via RS232, Telnet, SNMP, HTTP


◾ Status, statistics, and troubleshooting facilities
◾ Rapid Spanning Tree (IEEE 802.1D-2004) for fault tolerant topologies
◾ VLANs (802.1Q)
◾ Class of Service - CoS (802.1p)
◾ SNMPv2, SNMPv3, RMON
◾ IGMP(Internet Group Messaging Protocol)
◾ MMRP (Generic Multicast Registration Protocol)
◾ GVRP(Generic VLAN Registration Protocol)
◾ Link aggregation (IEEE 802.3ad)
◾ Port Mirroring
◾ Optional support for SNTP or IEEE 1588v2

Substation hardened switches usually implement several advanced features to ensure cyber security,
some examples of these features are:

◾ Multi-level user passwords


◾ SSH/SSL encryption
◾ Enable/disable ports, MAC based port security
◾ Port based network access control (802.1x)
◾ VLAN (802.1Q) to segregate and secure network traffic
◾ RADIUS centralized password management

Latency in a Switched Network
Latency in a communications network is defined as the time it takes for a message to traverse the
network from the transmitter to the receiver. In critical applications such as GOOSE the network must
guarantee a certain maximum latency or the application may fail. Switched Ethernet networks have
several sources of latency:

◾ Store and forward


◾ Switch fabric processing
◾ Wireline transmission
◾ Frame queuing

All of these latencies except for queuing are deterministic and yet the effects of frame queuing can also
be calculated providing one knows the nature of all sources of traffic on the network.

a)  Store and Forward Latency (LSF) 
Store and forward refers to the basic operating principle of an Ethernet switch. The term is descriptive of
its actual operation: the switch stores the received data in memory until the entire frame is received. The
switch then transmits the data frame out the appropriate port(s). The latency this introduces is
proportional to the size of the frame being transmitted and inversely proportional to the bit rate as
follows:
LSF = FS/BR
where LSF is the store and forward latency, FS is the frame size in bits, and BR is the bit rate in bits/s.
For the maximum size Ethernet frame (1500 bytes) at 100 Mbps the latency is 120 µs. For comparison,
the minimum size frame (64 bytes) at Gigabit speeds has a latency of just 0.5 µs.
b)  Switch Fabric Latency (LSW)
The internals of an Ethernet switch are known as the switch fabric. The switch fabric consists of
sophisticated silicon that implements the store and forward engine, MAC address table, VLAN, and CoS,
among other functions. The fabric introduces delay when executing the logic that implements these
functions. The switch fabric latency in utility grade hardened switches is in a range of few µs.
c)  Wireline Latency (LWL) 
Bits transmitted along a fiber optic link travel at about ⅔ of the speed of light (3x108 m/s) . When very
long distance Ethernet links are deployed, this delay can become significant. The one way latency for a
100 km link works out to:
LWL = 1x105 m / (0.67 × 3×108 m/s) ≈ 500 µs
For the distances involved in substation local area network, this delay becomes trivial compared with the
other contributions to latency.
d)  Queuing Latency (LQ) 
Ethernet switches use queues in conjunction with the store and forward mechanism to eliminate the
problem of frame collisions that used to exist on broadcast Ethernet networks. Queuing introduces a
non-deterministic factor to latency since it can often be very difficult to predict exact traffic patterns on a
network.
Class of Service (CoS) introduces a priority scheme to Ethernet frames to help mitigate queuing latency.
It is a best-effort service, however, and cannot guarantee quality of service, since multiple frames at the
highest priority level must still be queued relative to one another. Another consideration is that if a lower
priority frame has already started transmission, then that frame must be completed before the switch
may begin transmitting the higher priority frame.
Calculating with absolute certainty the worst case latency for any Ethernet frame can be challenging. It
requires detailed knowledge about all sources of traffic on the network. Specifically, one must know the
maximum frame size transmitted by any device, the CoS priority of frames, and the time distribution and
rate of frames. In an arbitrary communications network, little of this information is known and some
assumptions have to be made.
Estimating the average latency for an Ethernet frame is simple. For a network with no traffic load, the
queuing latency for a frame will be nil. For a loaded network, one can assume that the likelihood of a
frame already in the queue is proportional to the network load.
e)  Total Worst-Case Latency Calculation (LTOTAL) 
The latency sources described above are duplicated for every switch that an Ethernet frame must
traverse on its journey from source to destination. Hence the general calculation for worst-case latency
in a switched Ethernet network is expressed as:
LTOTAL = [LSF + LSW + LWL+ LQ] x NSWITCHES
where each contribution to latency is considered separately for each switch in the path. The calculation
may be simplified considerably if one considers the case where only one traffic source has a high priority
and is infrequent enough that multiple frames of that type need never be queued at any switch in the
network. In this situation, the worst-case queuing latency is exactly that due to one maximum sized
frame in each switch in the path. The worst-case latency then simplifies to:
LTOTAL = [(FS/BR) + LSW + (FSMAX /BR) ] x NSWITCHES + LWL(total)
where FS is the size of the high-priority frame being considered, in bits, and LWL(total) is the latency due
to the cumulative wireline distance from transmitter to receiver.
Network Topologies
There are multiple network topologies possible for IEC 61850 based LAN in electrical substation. The list
below shows the most popular topologies as used today:

◾ Non-redundant Star Topology


◾ Ring Topology
◾ Multiple Ring or Mesh Topology
◾ Ring of Switch Nodes Topology

There is no one best network topology for all substation automation applications. The same applies to
redundancy protocols. All topologies and redundancy mechanisms have their benefits and weaknesses
depending on the use case.
a) Star Topology
Star topology is the most basic architecture found in switched networks. It can even be implemented with
unmanaged switches as there is no need for redundancy protocols such as RSTP. Its most important
advantage is simplicity, ease of configuration and scalability. There is an obvious drawback as it is not
resilient to single point of failure. Star topology does not have inherent redundancy and is not
recommended for high voltage substations or for critical applications (Figure 1).
b) Ring Topology
Ring topology introduces redundancy in the network as there is alternative communication path between
all Ethernet switches in the ring. IEDs are typically connected to switches with single links. Ring topology
requires managed switches with redundancy protocol that will assure loops are eliminated and prevent
infinite transmission of data frames in the network. The most popular and widely deployed redundancy
protocol is RSTP, however there exist also other proprietary protocols that in certain cases may have
faster network convergence time. Such proprietary solutions limits the topology to just one switch vendor
because interoperability is not guaranteed. Ring topology has limitation dictated by the maximum
number of hops in the ring. The latest RSTP standard limits this number to 40 hops. The bigger the
number of switches is the longer the convergence time is. Assuming an efficient RSTP implementation
with 5 ms delay per hop then in a large RSTP the total convergence time can be up to 200 milliseconds
(Figure 2).
c) Multiple Ring or Mesh Topology
Depending on the application or the size of the network instead of single ring a multiple ring topology can
be used. A good criteria for designing this topology is separating the network into different rings based
on substation voltage levels or based on division into bays. For example devices at each voltage level
may pertain to a different communication ring or each bay can have a separate Ethernet fiber optic ring.
The advantage of this topology is logical separation of the network that follows the physical topology of
the substation or the natural separation of IEDs into different groups driven by protection and control
application. The logical separation of traffic can be achieved by more sophisticated mechanisms such as
VLANs or multicast filtering. There is also a possibility to use later 3 switches or routers as demarcation
points between different voltage levels, as it permits stronger access control and filtering mechanism
between different network segments. Multiple ring topology can be seen as mesh topology (Figures 3
and 4).
d) Ring of Switch Nodes Topology

Another topology used in today’s substation


is the ring of IEDs with embedded Ethernet
switch module. This topology is sometimes
referred as ring of switching end nodes. The
embedded switch modules shall implement
RSTP protocol and other typical features of
managed switches. This topology has some
benefits for non-critical applications as it
offers potential cost reduction by elimination
of number of communication links and
number of standalone switches. One of the
shortcomings of this topology is reliability as
in case of a software or hardware failure in
the IED the switching node may be lost
disrupting the network. This problem may
be reduced by proper implementation of the
switching module and proving it with independent power supply from the power supply of the IED. Ring
of switching nodes can also introduce maintenance and upgradability issues. Upgrading the firmware of
the embedded switch in the device may require using IED configuration software. Having an embedded
switch in the IED reduces the freedom of the utility to make decisions of a replacement in any moment of
a particular protection relay by the equivalent device from different vendor. (Figure 5).
Network Redundancy
Protection and control system perform critical application in electrical substations and in IEC 61850
based systems the communications network should always be available, fast and secure. Network
redundancy is required for substation automation system in order to guarantee that the network is
resilient to communication failures.
To achieve network redundancy more than one path is required from source to destination which implies
physical loops in the network. However, if a true loop were to occur in an Ethernet network, the first
broadcast frame would circulate endlessly, consuming all available bandwidth resulting in a ‘broadcast
storm.’ Layer 2 redundancy protocols like Rapid Spanning Tree Protocol (RSTP) prevent this problem by
quickly forming a logical tree network that spans all switches on the network – hence the name of the
protocol; at the base of the tree is found the ‘root bridge’ which is elected by all the switches. Spanning
tree protocols ensure that certain links in the network are put into a backup state so that no traffic may
flow across the link thus breaking any physical loops in the network. The backup links are re-enabled as
needed when network problems occur to restore connectivity of all devices.
The automatic network fault recovery shall be fast in order to minimize data losses and ensure proper
functioning of the system. Some redundancy protocols permit achieving recovery times in range of
dozens of seconds while others can even recover the network within few milliseconds.
Quality of Service
The recent international standard IEC 62439 introduces new concept of high availability networks and
defines protocols like PRP or HSR that enables “zero-time” recovery. PRP and HSR are perfect
solutions when no data loss is permitted upon the network failures. Examples of applications when “zero-
time” recovery or “bumpless redundancy” may be required are tripping via GOOSE messages or IEC
61850 Process Bus in high voltage substations. The ultimate goal of deploying redundant networks is to
guarantee packet delivery for high priority traffic classes. PRP or HSR protocols duplicate all the data
frames but are not lossless networks by definition. By duplicating a frame those protocols only increase
the delivery probability of a packet. By itself, the protocol does not guarantee packet delivery even for
duplicate packets. Therefore for PRP and HSR a managed, state of the art Quality of Service (QoS)
engine is the key in order to provide guaranteed packet delivery for high priority traffic classes. Without
it, guarantee delivery can not be achieved. Even short term congestion caused by certain data source
consuming all the available bandwidth can create packet drop and affect higher level applications.
VLANs and Traffic Prioritization with Class of Service
Virtual LAN (VLAN) defined in IEEE 802.1Q and Class of Service (CoS) defined in IEEE 802.1p share a
common tag header in Ethernet frame. These two mechanisms provide two basic features for IEC 61850
based communications network:

◾ Traffic segregation
◾ Traffic prioritization

Traffic segregation permits restriction of certain types of traffic or data streams to determined segments
of the network or to defined group of receivers. Traffic prioritization allows increasing performance of the
critical applications such as GOOSE or Sampled Measured Values. A virtual LAN allows the advantages
of a logically separate network while sharing cabling and equipment infrastructure with other VLANs to
reduce cost. Each VLAN has its own broadcast domain, meaning that Ethernet frames from one VLAN
will not be transmitted onto another VLAN. This segregation provides a powerful security mechanism;
users and IEDs on one VLAN cannot communicate with other VLANs unless a router is deployed to
route between the VLANs (Figure 6).

There are many traffic flows in the IEC 61850 substation network that merit segregating into separate
VLANs.
For example:

◾ Substation LAN management


◾ SCADA/Engineering Access
◾ GOOSE Messages
◾ Process bus
◾ Synchrophasors
◾ Protection A vs. Protection B
◾ Video surveillance and access control
◾ VoIP

Separating these functions into different VLANs has many advantages. Devices with high volume traffic
output such as merging units or video encoders do not flood other devices with traffic they cannot
tolerate. Secure access to different VLANs by personnel is easily controlled at a central router
demarcation point. Access to real time VLANs like GOOSE and process bus can be restricted entirely to
the devices involved to prevent misoperation. Protection A+B schemes can be implemented with a single
physical network thus reducing cost but without sacrificing reliability (Figure 7).
By default all traffic received at ingress port queues of an Ethernet switch is processed according to
FIFO (First In, First Out) mechanism. Prioritization with Class of Service allows that higher priority traffic
from multiple ingress traffic queues can be sent first. Whenever a frame arrive at a switch’s ingress port
its priority tag is analyzed and if determined that it belong to high priority level this frame can be
processed and forwarded before other frames with lower priority that have arrived earlier and were
already waiting in the queue buffer. This permits that application-critical or time-sensitive traffic like IEC
61850-8-1 GOOSE, IEC 61850-9-2 Sampled Values or VoIP can have reduced jitter and latency
introduced by Ethernet switch (Figure 8).
Operations, Administration, and Management

Ethernet switches are IEDs just like protection relays


and require configuration and monitoring to achieve
optimum performance of the substation LAN. This is
sometimes forgotten when planning, commissioning,
and during normal operation. Switches are mostly
“plug and play” but when using more advanced
features like VLANs and security some level of
expertise is needed and training for staff is highly
advisable. During commissioning and normal
operation, Ethernet switches can provide valuable
diagnostic information such as link status and traffic
flows.
At minimum, automation systems should include
monitoring of alarms generated by switches so that
preventive maintenance can be performed before a
major event occur. For example, with monitoring
alarms, link failures could go undetected because RSTP restores all connectivity; however, a second link
failure could cause many IEDs to be cut off from the network with subsequent misoperation of protection
schemes and loss of data.
Advanced substation switches provide a failsafe relay that indicates if any alarms have occurred which
can be connected to an IED input to trigger an operator response. The Simple Network Management
Protocol (SNMP) can also be used for this purpose.

Migration of Legacy 
Protocols and Serial 
IEDs
The IEC 61850
substation does not
exclude other legacy
protocols and most
systems will have a
mixture of
communications
protocols. Ethernet easily
accommodates having
DNP/TCP/UDP, IEC
60870-5-104,
Modbus/TCP, and others
simultaneously. Legacy
serial IEDs can also be transported over the same Ethernet network by using serial device servers that
convert these older protocols to their Internet Protocol equivalent. Going further, protocol converter
gateway products are emerging that map data points from serial IEDs to the equivalent logical node
representation of IEC 61850 so such devices can integrate into IEC 61850 automation. This all makes
migration to an IEC 61850 solution possible without having to abandon existing systems.
Biographies
Roger Moore received his Bachelor of Applied Science degree majoring in computer science and
physics from the University of Toronto in 1990. He has over 17 years of experience in the electric utility
industry and substation automation and holds patents related to advances in communications and
protective relaying. Roger worked as project manager for GE Multilin where he developed advanced
protective relaying systems and substation automation technology. In 2002 he joined RuggedCom
company and is currently the Vice President of Engineering. Roger is member of IEC TC57 WG10 the
standardization body for IEC 61850 communications.
Maciej Goraj received his B.Sc. and M.Sc. degrees from the Warsaw University of Technology in
2000, 2001 respectively. After graduation Maciej moved to Spain and joined General Electric Company
in 2001 where he worked in different positions. Maciej has been involved in the design and
implementation of multiple communication protocols and has field experience as he commissioned
network systems in high voltage substations. In 2009 Maciej joined RuggedCom where is currently
working as Utility Market Manager for Europe, Middle East and Latin America. He is member of IEC
TC57 WG10.

Home | Current Issue | Tutorials | White papers | Books | Tools | Events | Advertising | Classified | Forum
Terms and Conditions of Use and Privacy Policy
© PAC World - Last updated: 05 Nov 2010