NERC CIP Compliance: New Challenges for Asset Management

Professionals

Gowri Rajappan, grajappan@doble.com

Doble Engineering Company, 85 Walnut St, Watertown, MA 02472

ABSTRACT

Recent cyber attacks on the power grid and new regulations are fundamentally changing the asset
management job requirements. Considerable time is now spent on ensuring cyber security in general and
NERC CIP compliance in particular. However, early attempts have revealed many challenges in
complying with the asset management-related NERC CIP regulations. Building a compliant IT system and
trying to tweak it to the Operations Environment (OT) environment leads to broken work processes. The
better approach is to design a well considered OT solution with tailored IT security measures. In this
paper, we discuss the compliance challenges and solutions.

INTRODUCTION

Cyber security is fast becoming a leading challenge to grid reliability. Recent power grid cyber attacks
have demonstrated the dangers posed by cyber threats, especially Nation State-backed Advanced
Persistent Threats (APTs). Increasingly, asset management professionals are in the vanguard of the
defense against such threats. In addition to their regular testing duties, they are being asked to perform
Information Technology (IT) tasks such as updating the software and firmware on relays and other
microprocessor-based devices. Due to their regular contact with relays and other Bulk Electric System
Cyber Assets (BCAs), they are targets to APTs as well. In particular, the laptops and tablets they use to
work with the BCAs are a means for malicious software (malware) transmission to the BCAs.

This paper will discuss recent power grid cyber attacks and the variety of ways in which the attackers
have attempted or managed to enter the grid systems. Laptops and tablets that come into contact with
BCA are one of the main ways that attackers can get in. The paper will also describe new NERC CIP
requirements that help protect the laptops and tablets used to work with the BCA. These requirements go
into effect on the 1st of April 2017. This paper also describes some possible IT security solutions to meet
the requirements. Traditional IT solutions, while effective against cyber threats, have considerable issues
when faced with the challenges of asset testing and often prevent the work from getting done. A more
effective approach is to design an Operations Technology (OT)-centric solution that is tailored to fit the
asset management work processes and fortified with IT security measures. This type of approach and
recent experiences with it are also presented.

POWER GRID CYBER ATTACKS

Cyber threats have received a great amount of attention recently. It is fair to ask whether this is justified or
whether it is hype. The world’s preeminent insurance and reinsurance marketplace, Lloyd’s, conducted a
study on the potential effects of a cyber attack on the US power grid [1]. They studied a fictional attack
where a malware infects electric generation control rooms in the Northeast US and simultaneously
sabotages 50 generators. This attack temporarily destabilizes the Northeastern United States regional
grid and causes some sustained outages. While power in this simulation was restored to some areas
within 24 hours, other parts of the region remained without electricity for a number of weeks. This
scenario, while improbable, is technologically possible. Lloyd’s estimated that the GDP at risk could be up
to $1 trillion, and the total value of claims paid by the insurance industry would be $21.4bn, rising to
$71.1bn in the most extreme version of the scenario.

© 2017 Doble Engineering Company – 84th International Conference of Doble Clients

All Rights Reserved
Lloyd’s judgment of the technical feasibility of such an event is based on the increasing sophistication of
real-world attacks. As recent events have illustrated, there are a variety of ways that malware can enter
industrial facilities and critical infrastructure. Social engineering techniques such as phishing, watering
hole attacks such as those that target industry resource websites, and compromised USB devices are the
most common ways. This paper will provide real world examples of these as well as some of the
challenges that the defenders face.
Social Engineering: Many in the utility industry are aware of the cyber attacks against Ukrainian utilities
in December 2015 [5], which involved the attackers compromising SCADA and opening circuit breakers in
at least 30 substations, causing interruption of service to over two hundred thousand customers. This is
the first known incident in which a mass outage was caused purely through a cyber attack. Due to the
compromise to SCADA (Supervisory Control and Data Acquisition), the utilities had to switch to manual
operating mode in order to the reclose the circuit breakers and restore power. Forensic analysis showed
that the hackers worked on this attack over several months, and it involved multiple coordinated layers of
compromise, including malware and Denial of Service (DoS). As shown in Figure 1, the attackers got their
first toehold in the network by means of phishing emails with weaponized Microsoft Office files. This is a
type of Social Engineering. If a laptop that is used to work with BCA is affected through such a means, it
is even easier for the attackers to get to critical grid assets than the long sequence of actions involved in
the Ukraine attack.

ICS Kill Chain Mapping to the Ukraine Attack

Figure 1

Watering Hole: One well-studied APT is the 2014 attack by the group known as Energetic Bear, which
was thought to target energy sector entities. The purpose of the attack was cyber espionage (i.e., data
theft) and so its visible impact wasn’t nearly as severe as the Ukraine attack. The attackers compromised
the software update servers and web servers of several Industrial Control System (ICS) vendors and
replaced software updates and installers for the vendors’ products with trojanized versions [4]. When the
industrial customers of these products downloaded and executed any of this software, it infected their
computers with the malware. This is a type of attack that the laptops used to work with BCA are

© 2017 Doble Engineering Company – 84th International Conference of Doble Clients

All Rights Reserved
2-9
susceptible to, since the asset testers may have to download a variety of software and firmware from
vendor websites.

USB Devices: There have been several well-known instances of USB devices propagating malware. In
2012, the Department of Homeland Security Industrial Control System Cyber Emergency Response
Team (ICS-CERT) discovered sophisticated malware in the control environment of a power generation
facility. A USB drive used for backing up control systems configurations had been the culprit, and several
engineering workstations critical to the operation of the control environment were affected. The ICS-
CERT team noted that, while the implementation of an antivirus solution presents challenges in a control
system environment, it would have been effective in stopping the malware discovered on the USB drive
and the engineering workstations [1]. USB drives are quite frequently used in asset testing – for instance,
to transfer or back up the test files; therefore, it is a key vulnerability.

Challenges Faced by Defenders: Some of the challenges in combating cyber threats is illustrated by the
following case study. On December 30, 2016, the Department of Homeland Security US Computer
Emergency Response Team (US-CERT) put out an alert about a Russian malware campaign, called
Grizzly Steppe, which was targeting US Government agencies and industrial entities. As shown in
Grizzly Steppe spearphising for cyber intrusion against target systems [3]
Figure 2
, this attack involved Russian APTs using spearphishing (i.e., targeted phishing) to deliver the malware
and extract data from target systems.

Grizzly Steppe spearphising for cyber intrusion against target systems [3]
Figure 2
Upon receiving this alert, most US utilities searched their computers for the Indicators of Compromise
(IOC) associated with this malware, as is now common practice. This practice is called threat hunting and
it involves looking through the computers for known IOCs such as communication with certain IP
addresses and files that match the malware signature. Burlington Electric Department in Vermont found
one computer that had one of the IOCs, namely communication with an IP address associated with
Grizzly Steppe. News of this initial detection was leaked and, due to the prevailing political climate, widely
reported in the media as an instance of Russians having successfully infiltrated the grid, causing panic in
the industry. Upon further investigation, the IP address turned out to be a Yahoo cloud server that the
computer communicated with when the user checked Yahoo! email.

© 2017 Doble Engineering Company – 84th International Conference of Doble Clients

All Rights Reserved
3-9
This incident illustrates the challenges faced by cyber defenders, which is akin to that faced by the
intelligence community. Threat hunting across the tens of thousands of computers owned by
organizations is a challenge, especially given the time constraints. For the laptops that are used to test
BCA, which are even more critical due to their proximity to the grid assets, it is even more of a challenge
to look for incidents. These devices are used in the field and may only have intermittent network
connectivity. Timely assessments are needed, but premature release of information may lead to
misreporting and paranoia.

NERC CIP REGULATIONS

In response to the cyber threat landscape, NERC has led the development of Critical Infrastructure
Protection (CIP) cyber security standards. The first version of this standard was released in 2008, and the
standards have rapidly evolved since then. The CIP Version 5 and 6 went into effect July 2016. This new
version is a significant expansion of the requirements and the entities affected. As illustrated in Figure 3,
there were fewer than 100 entities that were subject to CIP V3 (“Type 2” and “Type 3” in the figure). In
CIP V5/6, the CIP requirements apply to more than 1000 entities, as a result affecting nearly every grid
asset owner and operator.

Expansion of NERC CIP Entities in CIP V5/6

Figure 3

In CIP V5/6, the criteria for what constitutes a Bulk Electric System (BES) Asset are broader than the
criteria for Critical Assets in V3. Groupings of such BES Assets, called BES Cyber Systems, are
categorized into High, Medium and Low Impact. As shown in Figure 3, many new control centers are
identified as High Impact under the new classification and more substations identified as Medium Impact
(“Type 1” in Figure 3). A large number of small distribution providers are now newly classified as having
Low Impact assets (“Type 4” in Figure 3).

The experience of Sacramento Municipal Utility District (SMUD) is illustrative of the NERC CIP impact [7].
As noted in [7], SMUD’s Distribution Control Center (DCC) was newly classified as a high-impact facility
because it had possible connectivity to the transmission operations computers when performing certain
distribution functions (via SCADA). NERC determined that, while actual connectivity is not apparent, if
there is the capability of any potential connectivity to equipment or software that involves control of the
BES, then the utility’s distribution control center can have a high impact on the BES, and it must be

© 2017 Doble Engineering Company – 84th International Conference of Doble Clients

All Rights Reserved
4-9
identified as such. Altogether, the DCC and four newly identified medium-impact substations cost SMUD
10,000 person hours and $5 Million in cyber security upgrades.

Among the newest NERC CIP requirements are the Transient Cyber Asset (TCA) requirements, which go
into effect April 2017. The TCA requirements are meant to secure one of the most vulnerable links – the
laptop computers and tablets that are brought in to sensitive environments such as substations. As FERC
noted in Order 791, such transient devices can move between electronic security perimeters and could
spread malware across BES Cyber Systems. The TCA requirement primarily affects asset management
professionals, because they rely on devices such as laptops and tablets to perform their test and
maintenance functions.

A notable aspect of CIP V5/6 is that the TCA requirements apply even to laptops that are used to connect
to a BCA exclusively via a serial port. While serial communications was excluded from CIP V3 due to the
protocol being non-routable, the BES Cyber Asset definition is CIP V5/6 is more general and includes any
programmable electronic device, immaterial of the means of communication.

The current TCA requirements, effective April 2017, apply only to High and Medium Impact BCS.
However, NERC has also developed TCA requirements for Low Impact systems, which will go into effect
upon FERC approval. These requirements would apply at the facility level. This is so that the utilities don’t
have to inventory and keep track of what low impact systems are out there and instead just need to know
which facilities have low impact systems. An implication of this is that, in the case of system-level
requirements (as with medium and high impact), the requirements only apply when an asset tester
connects to the systems (e.g., a test laptop connecting to protection relay). In contrast, for the low impact
requirements, the requirements need to be met while in the facility. An example is, in order to work in a
substation that has low impact systems (which is majority of substations with 100 kV or above
transformers), the requirements need to be met. This facility-level approach means that the requirements
apply to anyone who is in a facility that has low impact systems – transformer and circuit breaker test
technicians, maintenance and commissioning personnel, etc. Ultimately, any laptop or tablet taken into a
substation will have to meet the NERC CIP TCA requirements.

Sacramento Municipal Utility District (SMUD) CIP V5 Facilities [7]

Figure 4

© 2017 Doble Engineering Company – 84th International Conference of Doble Clients

All Rights Reserved
5-9
IT SOLUTIONS TO NERC CIP TCA COMPLIANCE

In an IT-centric approach to meet the NERC CIP TCA requirements, one starts with existing IT solutions
and attempts to tailor it to the OT work processes. In the case of laptops that are used to test BCA, there
are three existing IT solution templates that could be and have been used. Figure 5 shows these solution
templates. The first solution is to secure the device, the second is to secure the device’s access to the
assets, and the third is to secure the facility.

In the first solution, various hardening technologies are useful: in particular, restricting user privileges and
restoring the device to a known good state upon reboot. These are technologies that are commonly found
in kiosk computers provided for public use. The user privileges in terms of ports and services access are
highly restricted. Furthermore, any files created by the user are deleted upon reboot. In the extreme case,
the device could be reimaged before and after working with BCS. However, these measures make the
device extremely limited and, given the challenges testers typically face in the field, not useful in many
cases. For instance, a tester may have to use a USB-to-Serial converter on a USB port in order to
connect to a BCA with a serial port. If the USB ports were locked down as part of the hardening to
prevent malware incursion, this would not be possible. Furthermore, as part of testing, setting files and
test results have to be stored. These stored files would be lost if the machine is restarted for any reason.

IT NERC CIP TCA Compliance Approaches

Figure 5

In the second approach, namely securing access, technologies used for Mobile Device Management
(MDM) to facilitate Bring Your Own Device (BYOD) apply. In this approach, the testing device is not
trusted and is audited prior to use in order to verify that it is sufficiently secure so as not to compromise
the sensitive environment. The auditing could be through automated means such as a cyber security
kiosk in the substation. The user would connect the computer to the kiosk for a security scan and
approval. The network within the facility should be locked down with technologies such as Network
Access Control in order to ensure that only devices that are audited and approved have access. This is a
time consuming approach, since the pre-use scans might take a long time and hence hold up work.
Furthermore, the kiosks themselves would have to be maintained with malware definition updates, etc.

In the third approach, namely securing the facility, the BCS site may be treated as a secure facility with
practices common to large control centers. In this case, external electronic devices aren’t allowed in the
facility. Asset testing and maintenance operations are performed using a computer that is permanently
placed at the facility. The drawback of this approach is that the laptop that is permanently placed in the
substation needs to be maintained, e.g., application of patches and virus definition updates. While this

© 2017 Doble Engineering Company – 84th International Conference of Doble Clients

All Rights Reserved
6-9
may work for facilities such as control centers with on-site IT personnel, it is not a realistic approach for
substations.

All these approaches are cumbersome and ultimately ineffective when applied to substation
environments. The BCA are varied and hundreds of different software applications are needed to test this
diverse portfolio. Due to the age of some of the BCA, the software applications used to test these facilities
are very old as well and, in many cases, no longer supported by the vendor. Devices that are used to run
such software would resist the measures described in this section. Moreover, unforeseen circumstances
come up while in the field, such as the need to connect to serial ports through USB, which cannot be
easily solved in the remote facilities where the testing occurs without providing administrative rights to the
tester.

OT SOLUTION TO NERC CIP TCA COMPLIANCE

In an OT-centric approach, one starts with the OT work processes that need to be protected. The work
process of interest is the testing and maintenance of grid assets. The following are some of the elements
of this work process:

 The correct test plans and configurations are needed for the task at hand.
 The test results need to be provided to the appropriate storage systems.
 The facilities may be remote without easy access to technical support for issues that arise.
 Various ports such as Ethernet, USB, and Serial are needed to connect to the BCA and the test
instrument.
 Some of the BCA are old and can only be tested using old, unsupported software.

In order the facilitate these features, the following elements are needed:

 Transparent communication that automatically syncs the test plans and results on the device with
the relevant servers and does it securely.
 Communication management that disables external communication while connected to BCA.
 Secure remote support that meets the NERC CIP requirements.
 Port management that enables the ports appropriate for the testing task at hand, while keeping
the unnecessary ones disabled.
 Secure environment for executing old, unsupported software that are needed to test the aged
BCA.

Figure 6 illustrates the various components of such a solution developed by Doble in close collaboration
with utility partners. This solution, known as the Doble Universal Controller (DUC), is accessed through a
ruggedized tablet or laptop that has been custom designed for asset test and maintenance tasks. The
software applications installed on the device are chosen to fit the maintenance and test needs; these
could include Doble software such as Doble Test Assistant (DTA) and ProtectionSuite, Doble ENOSERV
software such as RTS, and third-party applications such as SEL acSELelator. Furthermore, old,
unsupported testing software are run in a secure environment with special security controls.

This device connects with the test apparatus, runs the appropriate software to execute the approved test
plan and acquires data from the tests. The test plans and the test results are automatically kept in sync
with the servers – i.e., updated test plans and associated documentation are automatically downloaded to
the device, and the test results are automatically uploaded to the servers. This data synchronization
occurs through secure cellular data communication, facilitated by a 4G LTE data card. Through the
secure 4G LTE interface, the device is connected to the servers that stores the test results in the relevant
database such as the DTAWeb. The data center connection is also used by the device to automatically
obtain and keep current the approved test plans, setting files, and other associated documentation. The
connection between the device and the data center(s) is based on a store-and-forward approach, so that
if the 4G LTE coverage is poor or unavailable in the field, the test results are stored and automatically
uploaded when the connection is reestablished.

© 2017 Doble Engineering Company – 84th International Conference of Doble Clients

All Rights Reserved
7-9
 Illustration of the Doble field force automation solution
Figure 6

The entire DUC infrastructure can be isolated from the utility IT infrastructure, thereby preventing malware
propagation between the two environments and from the public Internet, thereby eliminating the main
source of cyber compromise. This is achieved by configuring the DUC 4G LTE communication to occur
over a private MPLS network. The DUCs, in this case, are able to only communicate with the servers that
are used to manage the DUCs and the database servers and network fileshares that need to exchange
information such as relay settings and test results with the DUCs. General email and web browsing are
disabled, thereby preventing the two most common means of malware propagation.

Furthermore, the ports such as USB that are generally not needed are disabled. But the USB port may be
needed to connect to test apparatus for some testing scenarios, in which case a special test mode is
available that enables the USB port in a device mode, while at the same time disabling other capabilities
such as external communication. This precludes the possibility of DUC becoming a bridge between the
asset under test and an external network. The selective enabling of USB port in a device mode also
prevents the use of USB storage devices, thereby eliminating another common means of malware
propagation.

The 4G LTE connection on the DUC is also utilized to provide on-demand support. If the DUC user has a
technical issue while in the field, he can launch the support app on the DUC to connect with Doble Client
Service personnel. The Doble support app, if permitted by the DUC user, allows the Doble support person
to be virtually immersed in the field context through capabilities such as viewing the DUC screen and
viewing the output of the camera on the DUC. In addition to support provision by Doble personnel, the
DUC on-demand support facility can be utilized for remote support by the utilities. For instance, less
experienced field technicians can request and receive remote support from seasoned personnel back in
the office to troubleshoot maintenance and testing issues in the substations. This on-demand support
capability has proven to be a timely force multiplier to overcome the challenges of aging workforce and
constrained resources.

CONCLUSION

Cyber security is a discipline that is here to stay and grow in importance with each newly discovered
cyber attack. It is important for asset management professionals to recognize this and become more
aware of security threats and countermeasures as it applies to their job function. In 2017 and 2018, the
NERC CIP TCA requirements are expected to fundamentally change asset testing and maintenance.
Doble has made an organizational commitment to work with utilities to develop OT-centric solutions that
enhance security and productivity while meeting regulatory requirements such as NERC CIP.

© 2017 Doble Engineering Company – 84th International Conference of Doble Clients

All Rights Reserved
8-9
REFERENCES

[1] Lloyd’s, “Business Blackout: The Insurance Implications of a Cyber Attack on the US Power Grid,”
July 2015,
https://www.lloyds.com/~/media/files/news%20and%20insight/risk%20insight/2015/business%20blac
kout/business%20blackout20150708.pdf

[2] Department of Homeland Security Industrial Control Systems Cyber Emergency Response Team
(ICS-CERT), ICS-CERT Monitor October-December 2012, https://ics-cert.us-
cert.gov/sites/default/files/Monitors/ICS-CERT_Monitor_Oct-Dec2012.pdf

[3] Department of Homeland Security United States Computer Emergency Response Team (US-CERT),
GRIZZLY STEPPE – Russian Malicious Cyber Activity, 29 December 2016, https://www.us-
cert.gov/security-publications/GRIZZLY-STEPPE-Russian-Malicious-Cyber-Activity

[4] Erik Hjelmvik on NETRESEC blog, “Full Disclosure of Havex Trojans,”

http://www.netresec.com/?page=Blog&month=2014-10&post=Full-Disclosure-of-Havex-Trojans

[5] ICS-CERT Alert, “Cyber-Attack Against Ukrainian Critical Infrastructure,” 25 February 2016,
https://ics-cert.us-cert.gov/alerts/IR-ALERT-H-16-056-01

[6] Tobias Whitney (Manager of CIP Compliance, NERC), “CIP Standards and Compliance Update: 2016
Plan,” December 2015.

[7] Michael Gianunzio and James Leigh-Kendall, “Critical Infrastructure Protection: Lessons Learned
From a Unique NERC Pilot Project,” Western Energy Magazine, 17 February 2016,
http://www.westernenergy.org/news-resources/unique-nerc-pilot-project/

BIOGRAPHY

Dr. Gowri Rajappan is a smart grid consultant at Doble Engineering Company, in which capacity he
contributes to cyber security and asset management areas. He is an expert in information security,
analytics, and enterprise integration. He the lead for the cyber security activities at Doble and he is the
chair of the IEC TC57 task group for developing a Common Information Model (CIM) standard for Asset
Management. He holds a Ph.D. in Electrical Engineering from Northwestern University and has authored
several peer-reviewed technical papers and patents.

© 2017 Doble Engineering Company – 84th International Conference of Doble Clients

All Rights Reserved
9-9