Data Analytics

GROUP TECHNOLOGY & RESEARCH, WHITE PAPER 2018
DATA ANALYTICS IN THE

ELECTRICITY SECTOR
SAFER, SMARTER, GREENER

DATA ANALYTICS IN THE ELECTRICITY SECTOR
CONTENTS
1 INTRODUCTION: WHAT IS IT ALL ABOUT? ____________________________________________ 4

1.1 Data: the new oil of the 21st century? ........................................................................................ 6
1.2 Convergence of structured and unstructured data ................................................................. 7
1.3 Reading guide .............................................................................................................................. 9
2 THE POWER SYSTEM AND ITS CHALLENGES _________________________________________ 10

2.1 The use of data in the power system ........................................................................................ 14
3 THE IMPACT OF DATA ANALYTICS ON FORECASTING AND PLANNING ____________________ 16

3.1 Asset management: investments and maintenance ............................................................... 17
3.2 Renewable generation: resource planning and dispatch ....................................................... 23
3.3 Demand forecasting .................................................................................................................... 24
3.4 Summary for forecasting and planning .................................................................................... 25
4 THE IMPACT OF DATA ANALYTICS ON OPERATIONS ____________________________________ 26

4.1 Operations in the power system ............................................................................................... 27
4.2 The impact of analytics on grid operations ............................................................................. 30
4.3 The impact of analytics on commercial wholesale operations ............................................. 32
4.4 The impact of analytics on retail operations ........................................................................... 33
4.5 Aggregating demand response in smart energy systems .................................................... 34
4.6 Summary for operations ............................................................................................................ 37
5 THE IMPACT OF DATA ANALYTICS ON SETTLEMENT ______________________________ 38

5.1 Settlement in unbundled markets ......................................................................................................................
38
5.2 The impact of data analytics on wholesale settlement ........................................................................ 39
5.3 The impact of smart metering ................................................................................................................
40
5.4 Advanced energy contracts ........................................................................................................................
41
5.5 Summary for settlement .................................................................................................................
46
6 CONCLUSION AND SUMMARY _______________________________________________________ 48

6.1 Applying data analytics in the electricity sector ....................................................................... 49
6.2 Final relection ............................................................................................................................ 50
7 REFERENCES/SOURCES _______________________________________________________ 51
8 END NOTES ___________________________________________________________________ 52

1 - INTRODUCTION:
WHAT IS IT ALL ABOUT?
Data is a buzz word in many industries, and indeed advanced data analytics — combining data
sources in creative ways, automatic decision making through machine learning, etc. — is at the
pinnacle of this applied scientiic development that is currently transforming many industries.
I n marketing and (online) sales, data analytics is

booming. Knowing and serving customers as
A well-known example is Amazon, which generates
almost half of its sales through recommendations it
best as you can, remains the core of any business and makes to its customers based on data about their buying
being able to use huge amounts of consumer data, ind behaviors and previous purchases as well as purchases
correlations, improve hit rates and the value offered to from comparable customers.
customers is increasingly becoming a strong source of
income. The use of data analytics to reduce costs, improve
eficiency and quality is increasing rapidly in operational
processes. This ranges from applications in logistics
planning and optimization to automation of
manufacturing using machine learning.
04 ENERGY Data analytics in the electricity sector

Sometimes, just the access to data already has the Most examples of big data and data analytics are
potential to transform a whole industry. The success of about consumers, and consumers act as the ‘linking pin’
Airbnb for example, is primarily based on giving between different sets of data. In the business to
‘amateur’ tourist accommodation providers access to business context data analytics already has a history and
the market by disclosing information to tourists about wide spread use, especially in logistics and operational
available lodgings. Thus, by drastically lowering entry management.
barriers for new entries (private accommodations) to But new analytical methodologies and algorithms, for
enter the market, Airbnb has disrupted the hotel and example machine learning, are having a huge impact
tourist accommodation markets. Uber has done the same in these ields. In fact, most examples in this paper are
with UberPOP/UberX, by allowing self-employed taxi about applications in business processes.
drivers entry to the market and generally charging much
lower fares than normal taxis1. Data and smart algorithms So, it is quite logical that data analytics and the
allow Uber drivers eficient access to clients, driving possibilities of ‘Big Data’2 are a major strategic issue in
down prices and disrupting the existing market. At the business communities. Data analytics can be a game
same time, clients perceive the information they get changer, either by setting costs to new unprecedented
when the taxi arrives, its fares, etc. as an increase of the levels, by offering new value to customers or by creating
quality of service. new markets or business models that do both. So, how
will this impact the electricity industry?3
The potential of data analytics is huge. Keeping to the
example of the taxi industry: a study from MIT using
Ride-Share Vehicles and data analytics showed that, in
the City of New York “3,000 four-passenger cars could
serve 98 percent of taxi demand in New York City, with
an average wait-time of only 2.7 minutes”. Currently there
are nearly 14,000 taxis that perform this task (1).
Data analytics in the electricity sector ENERGY 05

1.1 Data: the oil of the 21st Combining data with the data of only a selected number
of competitors will give a competitive advantage
century? over those that do not participate in the exchange4.
This can be an effective strategy if this results in an
Some see data as the new ‘oil’: a valuable ‘raw material’ increase in market share of all collaborating participants
that will be vital for the survival of companies in the at the cost of the non-participating companies.
coming decades. They see access to data as a For the non-participating competitors, there will be an
commodity that needs to be secured, not unlike incentive to follow the same path and start a ‘data
countries that try to secure access to energy and raw consortium’ among themselves to remain competitive.
materials to fuel their economies.
The forces driving cooperation between companies to
st
Whether it is justiied to view data as the ‘oil’ of the 21 share data and information thus look like those that are at
century, i.e. a commodity that needs to be secured, play when industrial standards are established. However
remains to be seen. Contrary to this view, many others unlike with standardization, presence is not enough in
argue that sharing data creates even more value. Data ‘data consortia’; the partner that would like to join most
is not becoming scarce; it is becoming abundant. The likely must bring its own data, or something else that will
term ‘Big Data’ implies that there is no shortage of data beneit the consortium. So, securing data for yourself will
and—unlike oil—data is not being consumed during its likely remain important.
use. It can be distributed, shared and copied indeinitely.
The saying “To share knowledge is to multiply it” applies Sharing huge amounts of even non-sensitive data does
equally to data. The value of combining two different have challenges. As it is not economical to send over
datasets might be much greater than each individual set huge amounts of data back and forth, this either implies
on its own. This applies especially to the ‘traditional’ Big a shared data warehouse with the consortium, which
Data (if you already can call it this way). Big Data is often implies a huge commitment and lock in; or it implies
being characterized by having more volume, velocity, allowing the consortium partners access to your data
variety and veracity than can be processed with systems with their analytical algorithms, which requires
traditional means, and is mostly concerned with inding quite some trust in the intent and capability between all
correlations in large quantities of consumer related data. consortium partners. Nevertheless, many companies are
More data means inding more useful correlations and putting effort into building platforms that will allow
making already known correlations more reliable and sharing data with others5.
speciic and thus more valuable.
Will access to data truly function as the new ‘oil’ of the
However, there is a catch. In a competitive market 21st century? Unlike oil, data is not an economically
success is determined by relative advantage. You don’t scarce consumable resource. However, (access to) data
have to be good; you have to be better than your does represent a huge economic asset that can be used
competitors. Having data your competitors do not have to get access to even larger sets of data. This reasoning
can be more beneicial than sharing this data with them. applies to all forms of data: from generic data sets about
Sharing data will likely result in a better offer to your consumer behavior and preferences; to large volumes
customers, but so will the offer of your competitors. data about assets.
Thus, there is a possibly that you will lose your
competitive advantage through sharing data.

1.2 Convergence of structured Analytics of structured data has a history much longer
than that of unstructured data, but also this ield is
and unstructured data undergoing rapid developments. An example of analysis
of structured data is Process Mining (2). Process mining
One of the largest drivers behind the Big Data trend is analyses business processes based on actual records
the ability to draw conclusions from unstructured data, generated during the process, to create a ‘picture’ of how
like texts, pictures, videos, social media and behavior. the business process functions, including all hick ups,
This is basically being done by adding structure to this bottlenecks and loops. The results of the analysis are then
data, for example linking it to each other through used to optimize the process and signiicantly increase
correlations, or quantifying it through the weights in eficiency and the quality of its output.
neural networks. There are many applications where
unstructured data is being analyzed, from self-learning In asset-rich environments sensors generate the major
algorithms translating texts form one language into portion of structured data. Sensors of all kinds are
another without a beforehand ixed set of grammar rules becoming so inexpensive that they can be used in
and dictionary to the automatic generation of juridical virtually everything, from sensors in clothes to
advice and self-driving cars. monitor health and the effectiveness of a sports training,
to sensors in lower pots that can be used to administer
A huge application is marketing. For the irst time in the optimal amount of water. The amount of data
history creating structure in consumer data can be used generated by these sensors is huge. While this sensor
to create an increasingly complete picture of (potential) data may consist of ‘structured’ time series, the context
customers, customer groups and their preferences. and metadata is often missing or dificult to trace, adding
This offers huge beneits compared to the traditional an unstructured element to this data.
approaches to consumer proiling based on the often
infrequent direct contact between thebusiness and its
customers. In a world where competition is ever
increasing, this customer knowledge is increasingly
becoming a vital asset to companies.

THE PROBLEM THE PROCESS THE RESULT
BIG DATA DATA ANALYTICS SECURING BENEFITS
Too big, too complex, Collecting, cleaning, processing, Insight, optimisation,

too fast analysing, visualising automated decision-making
Big data will underpin new To extract meaningful value Make the data work:
waves of productivity, growth from big data, you need Leveraging data in the core
and consumer surplus optimal processing power, process to create new services,
analytics capabilities and gain efficiency, reduce costs
skills and improve performance.
McKinsey 2011 IBM 2014
Integrate analytics in
Secure data access! Develop smart tools! core processes!
Figure 1 - Big data and data analytics aim to create value. Many applications are generic.
Application of data analytics to the core processes requires more specific effort but has a higher potential.
The difference between structured data and Many tools and applications of Big Data and data
unstructured data is disappearing. For example, in asset analytics for the analysis of both structured and
management, structured and metadata will not only unstructured data are generic and are applied in
improve knowledge, but will also ‘institutionalize’ different business sectors. They concern generic aims
practical knowledge6, often considered as the most like marketing and sales, procurement or process
unstructured knowledge there is and something management that have similar goals for all industries.
traditional tools like expert systems have struggled with. The challenge that many companies are now facing is
This will result in more reined asset management how to apply data analytics to their business speciic and
strategies that will take more practical aspects into technical processes, products and services in order to
consideration. The use of real time sensor data in stay competitive.
maintenance allows for increasingly ‘leaner and meaner’
maintenance strategies without sacriicing on quality. Companies start to implement data analytics to optimize
their core processes, increase service levels and/or
develop revolutionary new ways of working, new
products and new services. This step is challenging
because it requires a combination of ICT knowledge,
data analytics skills as well as domain knowledge to know
where data analytics brings most value and to interpret
the results, as illustrated in igure 1.

Data analytics has such a huge impact on businesses
that it becomes easy to lose track of the obvious:
no matter how much data and how advanced the
analytics, the value of data lies in the conclusions arising
from the analysis. To reuse the example of Amazon;
it is the appropriate recommendation that saves the
customer time and reduces the risk of a bad buy that
encourages him/her to make another purchase.
1.3 Reading guide
This paper is organized into 6 chapters. While this

chapter (chapter 1) relects on high over developments
concerning data analytics in general, chapter 2
introduces the most relevant developments within the
electricity sector, its stakeholders and the use of data in
the power system. Chapter 2 ends with a global process
segmentation of the electricity sector that is used to
segment the impact of data analytics on different
elements of the power industry. These are described in
chapters 3, 4 and 5.
Chapter 3 highlights some examples of the impact data

analytics is having on forecasting and planning in
different aspects of the sector: investment decisions and
maintenance; renewable generation forecasting and
demand forecasting. Chapter 4 analyses the impact on
respectively network, wholesale and retail operations,
coming together in how these work together in smart
energy operations involving demand response.
Chapter 5 starts with a short introduction on settlement
(e.g. resolving and remunerating of contracts,
agreements & billing) in unbundled energy markets and
analyses how data analytics might create new markets
and how these translate into the retail market with the
help of smart meters and advanced energy contracts
possibly involving block chain technologies.
Chapter 6 eventually summarizes the impact of data

analytics on the electricity sector overall and highlights
speciic aspects for the different stakeholders in the
electricity value chain.

2 - THE POWER SYSTEM
AND ITS CHALLENGES
While data analysis always was an important part of econometrics and logistics, the availability
of huge amounts of data through the Internet and social media, data analytics - and related
associated aspects like machine-learning - is becoming a force that is transforming industries.
T o understand the transformational effect of data

analytics on an industry, it is important to know
‘Digitalization’ is the main topic of this paper.
‘Decarbonization’ refers to the shift from fossil fuels
the structure, processes and other developments and to renewables. For the power sector, this shift towards
challenges in that industry. So, a short introduction on the renewable generation translates into an increasing
structure, processes and developments of the electricity amount of variable i.e. ‘non-controllable’ power sources
industry is described. like wind and solar, and by a corresponding shift to
smaller and more dispersed power generation (often
The electricity sector is going through a transition called DER: Distributed Energy Resources8). This trend is
towards a more renewable electricity system, while trying referred to as ‘decentralization’.
to remain reliable and affordable. This process of the
energy transition and the balancing act between The power sector, including utilities as well as system and
renewable, reliable and affordable is referred to as ‘the network operators, are used to making huge decisions
energy trilemma’7. This transition is also often described with investment horizons that span over decades.
by the trends 'decarbonization', 'decentralization' and However, in many regions in the world, the stable
'digitalization'. environment in which such decisions are possible is
disappearing (3) because of the energy transition.

Governmental policies are driving the large-scale However, at the at the end of the value chain,
implementation of renewables with mixed success9. competition for consumers is ierce and might
Developments such as electric mobility might speed eventually result in a shakeout, if the utilities fail to
up, or stall; the use of residential solar the use of diversify and create new value. Data analytics will be a
solar panels on residential rooftops is expanding part of this new value, as will be demonstrated in the
exponentially; market models around lexibility, designed following chapters.
to cope with (local) variable renewable energy sources
and that include the network are being designed (4). Network and system operators often operate in a strict
And because of the increase of power generation by regulatory framework and have an assigned task to fullil.
wind and solar, the electricity system itself is becoming However, the dimensions and metrics they use in
more volatile. designing and operating their grid will need to change.
This is due to the emergence of distributed generation,
All these unknowns make long term investment less predictable and more correlated loads, and load
decisions hazardous. So, in many regions in the world, situations that can change much faster than before, which
stakeholders in the electricity sector—utilities as well as may be caused by massive installations of solar panels or
network and system operators—are forced to make charging stations of electric vehicles in a neighbourhood.
fundamental choices. In the following chapters the role of data analytics in
changing grid operations is analyzed.
Traditionally, utilities, at least the ones in Europe, used
to used to make their largest proit with their power In summary, both utilities and grid operators face
generation portfolio. Since 2008 that has changed challenging times as their environment is changing and
drastically10 and they are forced to rethink their they are forced to cope with growing uncertainty and
complete business models, acknowledging the fact complexity, changing policies, regulations, market
that the liberalization of generation, combined with the models and fast technological developments. Because
increase of renewable generation result in a lasting shift of variable renewable power generation, the system is
of proitability from the beginning of the electricity value becoming more volatile and these developments will
chain towards the end. The split between the traditional accelerate the adoption of data analytics in both network
power generation and the retail and grid activities of operations and emerging energy parties, such as the
E.ON into respectively Uniper and E.ON; and of RWE into aggregators and energy service companies.
respectively RWE and Innogy are examples of this on an
organization level.

FLEXIBILITY, POWER SYSTEM GOVERNANCE AND THE
UNIVERSAL SMART ENERGY FRAMEWORK
In unbundled energy markets, there is a strict distinction between commercial stakeholders and
market facilitating and ‘shared infrastructure’ management parties, such as distribution network
operators (DSOs49) and transmission system operators (TSOs). Commercial stakeholders are ‘governed’
by a competing market: competition should ensure their cost eficiency and their customer focus.
Grid operators have a natural monopoly. To ensure their cost eficiency and proper execution of their
tasks, they are governed by regulation and monitored by external (governmental) agencies, referred
to as the ‘Regulators’.
Transmission and distribution system/network operators (TSOs and DSOs) have the task to facilitate
the market by transporting and distributing the electricity through their grids and by maintaining the
integrity of the system through several technical and administrative functions. These include
maintenance and grid expansion to provide suficient capacity; the procurement of ancillary services
to keep the system stable; and facilitate market processes that are necessary for the market to function.
Examples of these facilitating services include providing metering services and facilitating the allocation
and reconciliation of energy use and generation.
However, regulating TSOs and DSOs by strictly deining their tasks, services and price levels does have
drawbacks. DSOs and TSO are by regulation not allowed to beneit speciic market parties—including
end—users, they cannot differentiate their offering to speciic individual wishes of end-users. All market
parties—including end users—should be treated equally. For relative simple needs, that all end-users
share, like a grid connection with a ixed capacity for a ixed price, this works well. However sometimes
more complex situations arise, like a congestion in the grid that affects only a limited number of users.
This strict approach leads to major ineficiencies as the TSO/DSO are not allowed to make use of the dif-
ferent needs and wishes of individual customers and therefore have to fall back to the ‘one size its’
all approach.
By redeining this ‘one size its all’ service level as the best ‘collective service’ possible to which
end-users are entitled but can negotiate about, this paradox can be broken. TSO/DSOs still have an
obligation to offer this ‘best collective service’ if end users if requested. However, it becomes
negotiable. In case of network capacity problems, a TSO/DSO can offer to compensate end-users in the
problem area to reduce (or increase) their load/production through an auction mechanism. End-users
that accept, will offer the lexibility while being compensated, while end-users that do not or cannot
accept, will not be affected by the capacity problems. This mechanism ensures an optimal allocation of
scarce lexibility, not only for the DSO, but also for the TSO and market parties.
Within the USEF framework (4) this concept has been extensively developed, including fail safe
mechanisms for DSOs; the deinition of stakeholders like aggregators that facilitate the interaction with
end-users; and deinitions of processes and protocols in a reference implementation that has been and
is continued to be tested in several demonstration projects among Europe.

STAKEHOLDERS IN THE POWER SYSTEM
The power system knows many stakeholders and demand and supply and seeks the most
the most important ones are described below. economical solution for the energy to be supplied.
The BRP can source the requested energy on
The role of the Transmission System Operator behalf of the Supplier in two ways: directly,
(TSO) is to transport energy from centralized by dispatching power plants with which it has a
producers to dispersed industrial prosumers and contractual agreement, or indirectly, by trading on
Distribution System Operators over its high-voltage the energy markets.
grid. The TSO also operates interconnectors that
link to other high-voltage grids in neighbouring The role of the Supplier is to source, supply, and
regions and countries. The TSO is responsible invoice energy to its customers. The Supplier and
for keeping the system in balance by deploying its customers agree on commercial terms for the
regulating capacity, reserve capacity, and incidental supply and procurement of energy. In this paper
emergency capacity (In some countries this task is often utility is used to describe a supplier that also
performed by an Independent System Operator, owns generation.
the ISO).
The role of the Aggregator is to accumulate
The role of the distribution network operator lexibility (controlled changes in power demand or
(DNO) or Distribution System Operator (DSO) is generation) from Prosumers sell it to the BRP, the
to cost-effective transfer and distribute energy in a TSO and/or the DSO. The Aggregator’s goal is to
given region over the distribution grid to and from maximize the value of that lexibility, taking into
end users and for the connections to and from the account customer needs, economic optimization,
transmission grid. The difference between a DSO and grid capacity.
and a DNO is that the DSO is able to perform
grid capacity management (i.e. activate demand An Energy Service Company (ESCo) offers auxiliary
response and local generation to support the energy-related services to Prosumers but is not
distribution grid). directly active in the electricity value chain.
An ESCo may provide for example data related
A Balance Responsible Party (BRP) is responsible services like insight services and energy
for actively balancing supply and demand for its management services, but may also offer
portfolio of producers, aggregators, and installation services (like condensing boilers,
prosumers. The BRP forecasts its portfolio’s energy

2.1 The use of data in the The power system started more than one hundred years
ago, and has been expanding ever since. It is possible
power system that there are still some components in the existing
power grid that are more than 80 years old. Companies
Knowing what customers value is indispensable in any have been merged into large utilities and a lot of data
industry, even for an industry delivering a commodity has been lost or conined in legacy systems. This data, so
like power. This holds true for the commercial players in called ‘dark data’ is very dificult and expensive to access.
the electricity value chain and even for regulated
monopolies like grid operators. Data analytics is a huge Most new data on the other hand, are sensor data from
enabler for this. Data analytics applied to other generic sensors within the grid and at its edges. For example,
business areas; such as the redesign and optimization Phasor Measurement Units (PMUs) and other sensors,
business processes (‘process mining’), logistics, smart metering at end users and even directly from
procurement, pricing, risk management, and many more equipment of customers if they allow it (like solar panel
become increasingly relevant to both utilities and grid inverters, combined heat and power installations, etc.).
operators. A complicating factor with this kind of data is that it is
often context speciic, thus not easily shared or
But, as mentioned in section 1.2, ‘Big Data’ and data interpreted. It usually consists in the form of time series.
analytics will also have speciic beneits for speciic To get the most out of this kind of data it is not suficient
sectors. For a sector that is going through a to just know what is being measured, but also consider
fundamental change like the power sector, this is the physics of the measured devices, its location and
especially true. Given the challenges the power system relation to other devices, its environment, logic, business
and its stakeholders are facing, it is not dificult to context like contracts, market prices, etc.
understand the huge interest from power industry
stakeholders in Big Data and data analytics as a source Figure 2 shows a schematic diagram of the basic process
for possible answers, solutions and tools that will help in the power industry. Data plays an important role in all
during the transition. of them. These processes use data to give information
about either the future, the current situation (now) or
the past. Processes dealing with the ‘future’ are about
predicting what might happen and to plan accordingly;
processes about ‘now’ are about operations and control
and act on what currently is happening; and processes
about the ‘past’ are mainly about settling of what has
happened between stakeholders, like for example billing.
While it is possible to make this ‘rough’ distinction

between processes, it should be noted that this is a
simpliication and that the impact of data analytics will
link these processes closer together. For example,
‘Settlement’ might be based on real time dynamic prices
that are inluenced by operations. ‘Operations’ might be
based on probabilistic calculations of a set of forecasted
scenarios and ‘Forecasting and Planning’ will make use of
real-time operational data and historical data collected
from the system. But nevertheless, it is a very practical
distinction that will be used to structure this paper.

Past Now (real time) Future
TIME
ENERGY SYSTEM
Regulated:
OPERATION FORECASTING AND
Network and SETTLEMENT PLANNING
system operators (dispatch, control,
(data allocation, support, maintenance, (loads, asset status,
Non-regulated: contracts & billing) workforce management) contracting, investment
Generator, retailer, decisions)
aggregator
CUSTOMER VALUE
CUSTOMER CUSTOMER
PERCEPTION
CUSTOMER EXPECTATION
(residential, industrial, SATISFACTION AND
producer) RETENTION (products, services) (information, marketing)
THIRD PARTY
(energy service
company, independent DATA ANALYSIS, COMBINING DATA SOURCES
service provider)
Figure 2 - Overview of the purpose of the use of data in the power system set in a timeframe: is data used for past actions (settlement);
is data used for future actions (forecasting and planning). The 'effect' on other stakeholders (customers and third parties) is
taken into account. Customer satisfaction is in the end a result of his expectation and what he/she perceives they're receiving.
Mapping the same approach of future’, now and past value chain: utilities, retailers, aggregators and grid and
regarding services to the customer gives an indication of system operators. It is however important to notice
how these speciic processes might help in creating value that there are more companies connected to the
to the customer (the end users of the electricity system). electricity value chain, like independent service providers
Carefully managing customer expectations, as well as and energy service companies.
meeting them when the moment is due, will ultimately
will lead to satisfaction and customer retention11 (see The next section will address the basic processes as
igure 2). mentioned in igure 2:
 forecasting and planning
This paper is about data analytics in the electricity sector  operation and
and it mostly considers stakeholders in the electricity  settlement

3 - THE IMPACT OF DATA
ANALYTICS ON FORECASTING
AND PLANNING
One of the main applications of data analytics is forecasting. In the electricity sector the
application of forecasting ranges from predicting power lows minutes ahead, to economic
developments in the energy market spanning decades; and, from lifetime predictions of large
assets, to predictions of behaviors and preferences of consumers.
D ata analytics in the power system usually is

supplemented by physical models or scenarios
This chapter does not have the ambition to give an
extensive overview of all developments regarding
that describe (possible) external developments, such as forecasting in the electricity sector. It does however
new technologies, changing regulation, growth rates or discuss the impact and opportunities data analytics is
disruptive events. having on a selected couple of examples that are
indicative for the wider developments. For each example,
Figure 3 shows an overview of common processes in the the role of data analytics regarding the example is
energy industry linked to their associated time scale. described, followed by a discussion of some new
A general rule of thumb is that the shorter the time developments and opportunities.
horizon the more automated and integrated the
forecasting will be into the process, driven by both
necessity and availability of data.

A network operator is expected to deal adequately
3.1 Asset management: with all types of uncertainties associated with these
investments and maintenance developments. That is, a network operator must be
effective in forecasting and in least-regret
decision-making regarding future utilisation of his
Introduction and current situation network. For example, decision-making regarding
In power systems, asset management is usually deined investments, reinforcements, replacements and
as the activities and practices through which a network depreciations.
operator (TSO and/or DSO12) develops and manages its
assets and their associated performance, risks and More data is becoming available due to the extensive
expenditures over their lifecycle to fulil the requirements monitoring of existing assets and the development of
and needs of grid users. Asset management is concerned new data collection techniques/technologies13.
with policy development, decision-making and The tools used to interpret this data are becoming more
managing of grid investments and maintenance. While sophisticated and data-driven. This data is helping in
the focus of this section is mainly on the grid operators, inding new correlations between environmental data
most of it equally applies to other stakeholders in the and asset utilization, and remaining lifetime as well as
power system which manage assets with a long lifetime. forecasting imminent failures.
Network operators, as well as other asset managers, must Early experiences are helping to prioritize the
deal with increasing uncertainties related to regulations, deployment of the next sensors. Unfortunately, from
developments in the energy market and the use of a data point of view—assets have a technical lifetime
electricity. Examples include the changes of grid codes of multiple decades and most of the healthy asset
and market design, the emergence of local energy population is still not being monitored. So, in practice
markets, integration of renewables (solar and wind power most information about assets and asset lifetime comes
generation) and storage, and the increase of electric from investigations of incidental failures.
transportation.

Investments
Maintenance
Resource planning
Dispatch planning
Execution
Balancing
minutes
MINUTES hours
HOURS days
DAYS months
MONTHS years
YEARS decades
DECADES
Figure 3 - Typical timing scope for important processes in the power system related to forecasting
This data is supporting the execution of asset New developments and opportunities
maintenance, for which there are many philosophies. Within the power industry, especially the network
Most of these philosophies can be categorized into three operators, there are some hurdles for asset
basic types, each having their own merits and drawbacks. management to become more data driven. One major
This is dependent on the assets age and use, the hurdle is that grid assets have a very long (technical)
probability and consequence of failure, the availability lifetime, some ranging over 60 years. Most failures are
of data, costs, etc.: incidents with external causes, e.g. by storms, loods,
building and digging activities, although aging and wear
 Corrective maintenance (for example ‘run to failure’) caused by increasing loads are beginning to have a
 Preventive maintenance (for example periodic larger impact. There is extensive knowledge and
maintenance) information about failure mechanisms and core failure
 Predictive maintenance (for example maintenance; causes available, gained by power failure investigations.
reliability-centered maintenance) However, there is surprisingly little ‘raw’ data about asset
failures. The cause of this is the long lifetime of assets and
The available data and data analytics basically play a the fact that most of them are not monitored. Especially
role on two levels. First, they allow for selecting the little data is available of the ‘large healthy population’,
right approach for each individual asset and/or which makes the application of many analytical
group/cluster of assets. Second, they play a major role algorithms like ‘rare event’ statistics dificult.
in the maintenance methodologies themselves.
With the more advanced methodologies, aspects that
have predictive qualities about future possible failures
are monitored and help decide about preventive
maintenance and when (plannable) corrective
maintenance measures are required.

Failures are often dificult to compare. Grid assets may
operate under very different circumstances and in very ONLINE SMART MONITORING:
different environments. In combination with the limited
amount of failures, it is a challenge to gain enough
SMART CABLE GUARD
statistical relevant data to use common ‘Big Data’
techniques to correlate data of asset failures with data
about their historical use and environment. While the
number of assets being monitored is increasing,
eventually solving the blind spot of the ‘healthy
population’. However, the number of failures, and thus
data about failures, will remain limited.
So, while there is no doubt that asset management

—especially asset maintenance—will beneit from data
analytics, it needs to face two main challenges to be able
for data analytics to fulil the promise many people in the
sector believe:
SMART CABLE GUARD EQUIPMENT:
a cable sensor and a receiver
1. How to gain more relevant data about asset failures
and especially near ‘misses’—to optimize maintenance
If you know what the future holds, you can
(obviously without creating more failures).
anticipate on what it will bring.
2. How to maximize the learning from the data that is
available by ‘generalizing’ it somehow, so it can be
linked to data from assets operating in different
situations under circumstances, as well as generally A medium voltage network that is continuously
be applied to all assets considering all different uses monitored by a fully automated system which
and circumstances. immediately gives a warning when problems
arise. This is a dream long held by many grid
The irst challenge can be greatly reduced if grid operators. Part of this dream is now realized by
operators would share data about failures—as well as Smart Cable Guard®.
their healthy asset population—to some extent.
Section 1.1 ‘outlined that there is a lot of value in
sharing data if that gives a competitive advantage. This is the irst sentence of the DNV GL website on Smart Cable Guard (11).
For grid operators (who do not directly compete Smart Cable Guard is a monitoring and prediction solution that has been
developed over a period of more than 10 years by DNV GL in cooperation
against each other on the reliability of their grids),
with the Technology Foundation of the Dutch government and Dutch grid
there would be great value in sharing information operators and STW. It is using continuous measurements at both ends of a
about asset failures. Sharing data with (energy-intensive) medium voltage cable to discover patterns in frequency, location, and
industry may help increase both the (healthy) population magnitude of partial discharges within the cable. These patterns are used to
data and the failure data. Standardization, recommended predict the remaining life time of the cable, and more precisely can give an
indication of where and when a fault will occur in the cable.
practices and independent IT-platforms may support
such developments. The oil industry recognized decades Smart Cable Guard is an example that shows that the application of data
ago that they were not competing on reliability of their analytics can be very speciic and that applying data analytics has to be one
assets, and are now sharing reliability data together with care. For each type of cable, Smart Cable Guard requires speciic
in the OREDA organization (Offshore and onshore algorithms tweaked to that speciic type of cable. For example, in XLPE cables
the interval between the irst sign of partial discharge activity and the average
Reliability Data)14.
failure is 2 months, while in old PILC cables this is 16 years, because of its
self-healing properties. Using the advanced pattern recognition algorithms of
Smart Cable Guard, the precision for forecasting a failure can be increased to
respectively 10 days for XLPE cables and 3 years for PILC cables (15).
Another additional feature of the measurements is that the location of a fault

can pinpointed at the moment it occurs. So, if a cable is being cut by for
example digging activities, it is immediately known exactly where the fault has
occurred and thus how to isolate it to save precious customer minutes lost.

Impact of asset data (its condition, health, use and
environment) is translated into probabilities and
RISK
eventually into risks to the integrity of the system.
ASSETS PROBLEM SOLUTION RANKING
AT IONS
LIMIT
ASSET RANKING SOLUTION
LIM
ITAT
ION
S
A
DAT
HEALTH
Figure 4 - A risk-based asset management decision process (5) and the application of (historical) failure data by factorizing
this into core probabilities and risks
The second challenge can only be addressed by Although data analytics promises to ind predictive
recognizing that the availability of (failure) data will structures in the data itself, combining it with
remain an issue and thus a different approach is required fundamental knowledge about the system has huge
than with normal big data applications, that presume an beneits. This “knowledge analytics” can be represented
‘abundance’ of data. The required methodologies need by a Bayesian network, which is essentially a complete
to make optimal use of the relative small amount that is representation of the interconnections between different
available. For example, as assets will be more and more features of a system that lead to performance failures.
monitored, new tools, such as ‘rare event’ statistics, can The interconnections in a Bayesian network are
be applied, giving better predictions of the asset lifetime. represented by conditional probabilities (e.g., the
probability of an interconnect failure given the
Increased monitoring also means that failure data and probabilities of encapsulation, metal frame, and glass
data about the circumstances and environment before failures), which are derived from data, experience, and
and during the failure, will become available. However, models. Using Bayesian networks in this way offers
for this to become truly valuable, this needs to be several advantages: they can be used to perform
generalized to be applicable to the whole asset sensitivity analyses, thus providing leading indicators
population, i.e. assets operating under different of failures; they can be used to conduct dynamic risk
circumstances (see igure 5 for an example). management using sensor inputs.

With the help of Bayesian networks, raw data from In addition, the opportunity to combine data from
among others power failure investigations, as well as multiple sources (e.g. multiple cooperating grid
other sources, like sensors, can be unraveled or operators from all over the world), stakeholders
factorized into root causes and root mechanisms that multiplying knowledge on their existing assets might
impact the probability of failures. These factors can prove invaluable to speed up the learning experience
then be compared with factors of other failures and of new, relatively unfamiliar asset types, like DC
recombined and used to predict the probability of equipment and batteries, making their application
failures of other assets16, even assets in different in the future much more likely to be successful.
circumstances. The advantage of this methodology is
that these ‘root’ factors (i.e. core aspects affecting failure)
can be updated with new speciic data as it becomes
available (using for example Bayesian statistics) and thus
an ever increasing ‘knowledge base’ is created.
Humidity Temperature
Vapor phase
salt
Manufacturing
Material infant defect
susceptibility Impact in
transportation
Atmospheric
corrosion
Galvanic Impact in
coupling to Mechanical installation
steel force
UV
Metal frame
failure
Encapsulant Glass failure

failure
Interconnect
failure
Figure 5 - Example of a Bayesian network for the failure mechanisms of solar panels, able to combine data with fundamental
knowledge of the system15. 'Backpropagation' techniques might be used to reverse experience data to 'root causes'.
These kinds of 'empirical' models can also be used to establish links between data of assets operating in completely
different circumstances.

3.2 Renewable generation:
resource planning and
dispatch
Introduction and current situation New developments and opportunities

Probably the most forecasted system is the weather. Weather forecasts will continue to become more accurate
Weather prediction has a history as old as humanity itself. as the amount of input data computing power is
In the electricity sector, extensive use is being made of increasing. More detailed data is becoming available.
weather forecasts, historically mostly for load forecasting, Examples of this are the Sentinel Satellites from the EU’s
but forecasting the generation of weather dependent Copernicus program; more local sensors and weather
renewable energy is becoming more and more crucial. stations, allowing much higher detail of the f
Generally, weather forecasts from meteorological orecasting models (from 5 km to 2 km resolution); high
institutes are used as the basis for the forecasting of quality imagery of cloud coverage and waves between
renewable energy as well as demands. offshore wind turbines, giving information about wakes
behind the turbines, etc.
It is not dificult to notice how many ways of forecasting
the weather there are; from detailed physical modelling However, being a chaotic system, each slight
to empirical methods and looking at trends. Nowadays improvement of forecasting will require an exponentially
practically all of them use a lot of different sources of increasing amount of data and computing power and
digitalized data, such as pressure, wind speeds, large jumps in the increase of accuracy of weather
temperatures, data from satellites etc. predictions are not expected, not even through the
application of new methodologies or forecasting
Numerical Weather Prediction (NWP) is a container for techniques like deep learning. Though especially for
many computer aided predictions based on physical predicting ‘rare’ events, like storms or loods that
modelling, parameterization, statistics etc. An interesting might put the electricity system at risks, these ‘slight’
observation concerning NWP is that the best results are improvements still might prove to be quite signiicant.
achieved when using models with many slightly different The use of weather predictions to calculate the power
initial states, known as ‘ensemble forecasting’, and even output proile of a wind turbine or solar panel, and
using different models, such as ‘super ensemble especially a whole wind or solar farm is a much younger
forecasting’. science, which is increasingly becoming more critical for
wind farm operations in recent years. Wind (and solar)
Power generation from wind and solar is obviously energy is affecting the power system and power markets,
highly dependent on the weather. Data from specialized and operations are shifting from maximizing energy yield
weather forecasts, (for example for a wind farm these to integrating the farm into the electricity system and
could be wind speeds, wind speed variations, wind market17.
direction, pressures, temperatures, etc.) is combined with
characteristics of the speciic wind farm, like historical The need to translate weather forecasts to power output
performance, reactiveness to wind variations (ramping), is fast developing and is gaining importance.
probability of icing dependent on wind and Some examples are the re-evaluation and adjustments of
temperatures. All this data and models are combined assumptions made in the irst generations of models;
to form a generation proile for the next period (which the development of advanced models, able to accurately
could be 15 minutes ahead to a week). model turbulence and wakes and thus the inluence
of wind turbines on each other; and models and
methodologies to estimate the effect of clouds and
cloud movement on the power output of solar farms.

3.3 Demand forecasting day approach’ based on historical data, prove to be
surprisingly accurate for a large number of consumers.
Introduction and current situation These methods take a ‘top down’ approach (see igure 6),
As with renewable generation, electricity demand is using (well-known) patterns in overall loads as a basis for
highly inluenced by the weather, especially, heating and forecast. These methods prove to be powerful.
cooling. Variations in demand used to be the main source The demand of even a few hundreds of households
of uncertainty in the electricity system, thus forecasting follows a regular pattern, and on a scale of millions of
load always has been an important topic in electricity households, industry and commercial buildings, the
systems. Both short term forecasting, necessary for demand is regular and shows a well-known pattern that
contracting and planning of dispatch of generation; is easy to forecast using the prementioned methods.
as well as long term demand, inluencing investment Size (i.e. the number of customers) is an important tool to
decisions in power generation and transmission and manage risks for balance responsible parties, and thus a
distribution networks. source of competitive advantage.
Long term forecasting in general relies on scenario However, demand is becoming more correlated due to
planning, based on general and local economic the ongoing electriication of heat (heat pumps,
developments, correlated with data from industry, air-conditioning) and transport (electric vehicles).
commercial and residential demand and possible Local (solar) generation is also increasingly occurring in
technological developments like for example energy distribution grids, so the statistical averaging out will
savings, the use of electric vehicles and solar energy. become less in the future. More relevant, because
expanding the network capacity to cope with this
Short term load forecasting depends on statistical synchronization of demand and generation in the
learning algorithms, regression models, neural networks capillaries of the network is very expensive, these
and other pattern recognition methods18. capacity limitations will lead to the necessity for local
Because aggregated demand has a very periodical ‘balancing’ (i.e. congestion management), and thus
(i.e. day/ week/season) and stable behavior, even relative for local demand forecasts consisting of much smaller
simple methods that exploit this behavior, like the ‘similar numbers of end users.
POWER SYSTEM BIG DATA AGGREGATION
Top system level Top system level
Consumers Consumers
Figure 6 - Traditional demand forecasting in power systems relies on known patterns overall (i.e. aggregated) loads.
'Big data' - as applied in for example marketing - is trying to create insights in unknown patterns by aggregating
data from users. As local forecasting becomes more important because of infrastructure constraints, this will lead
to an increased application of 'Big data' techniques in demand forecasting.

New developments and opportunities
An alternative approach to top down forecasting
3.4 Summary for forecasting
considers the use and demand of individual appliances, and planning
correlated with other demand, events and local weather
forecasts, before being aggregated into a forecast of Forecasting in power systems is applied to a wide ield of
(part of) the demand. The main advantage of this so issues and different time horizons. This chapter touched
called ‘end-use method’ is that patterns and correlations upon a few applications: Long term forecasting
can be found on appliance level. Therefore, forecasts necessary for investment decisions and maintenance
can be made much more local and lexible, for example strategies (often asset management); forecasting the
considering local weather variations, events and output of wind and solar farms; and demand forecasting.
scenarios. As more demand data becomes available
for analysis, due to the presence of smart meters and In practice, combining datasets still turns out to be hard,
the metering of individual devices among others, this due to different regional conventions, different time
approach is becoming more accurate and therefore scales and differences in meta information. Nevertheless,
feasible. in each of the examples raised, data becomes
increasingly available as data from different sources are
The ‘top down’ methodologies in load forecasting will disclosed. Also, more sensors from ranging from usage
remain dominant for a long time. However, because in network equipment, to weather sensors in the ield
the aforementioned increase in correlation in electric to smart meters measuring small scale demand are
demand and importance of local forecasting of smaller deployed. All these added together consequently i
numbers of end-users, it is likely that they will be mprove the integrity of data.
gradually being complemented by bottom up
approaches, like the end-use method and ‘big data’ While the data analytic tools that are applied in each case
approaches; where correlations in the use of appliances are comparable, the developments in each area are
and external factors are identiied by combining a lot of facing different challenges. Where asset management
data from individual users, building up to a picture of faces increasing amount of data, only a very limited
the total system (see igure 6). number of incidents happen. Data analytics is like
searching for a needle in a hay stack. With the exception
The combination of such a top down and a bottom up of predicting extreme weather conditions, forecasting
approach is much more lexible than the traditional top renewables and demand is a more 'continuous' process.
down approach alone, because it can handle forecasting Demand forecasting potentially will face increasingly
smaller number of end users and can consider changing synchronized demand from charging electric vehicles
local circumstances like the local weather, events and and heat pumps, including possible ‘feedback loops’
local trends, while still retaining the accuracy in caused by demand response reacting on energy prices
forecasting large number of end users and local network constrains, leading to a relative
increase of bottom up modelling of demand at the
expense of the traditional top down statistical approach.

ANALYTICS ON OPERATIONS
The area that probably will be inluenced most by data analytics is 'Operations'. Historically
analytics of operational processes was manual work and thus labour intensive and expensive.
This meant that it was predominantly a tool for analysts in staff departments to underpin
strategic decisions or to design more eficient operational processes.
A s developments go on, data gathering becomes

more automated, continuous and 'complete';
The result of these exercises is an eficient and optimized,
but rather ‘frozen’ process that does not respond well to
methods and tools become more sophisticated and changes in the process’ environment and ‘non-iltered’
standardized (e.g. Six Sigma) and more and more disturbances. A developing trend in manufacturing
processes are optimized. (e.g. in the semi-conductor industry or the solar panel
manufacturing industry) is that not only the data
In many (automated) manufacturing facilities a lot of gathering is automated and continuous, but also the
data is being gathered continuously within the process. analytics. By continuously analysing the data and
This data is periodically analysed by experts and used to adjusting the processes accordingly, the analytics and
redesign or ‘tweak’ the operational processes to optimize optimization becomes part of the system itself and able
it, e.g. by insulating the process from disturbances. to adopt to changing circumstances.

At the same time, more sensors in the system and
new ‘smart’ control algorithms and better and faster
4.1 Operations in the power
simulation techniques allow sub-processes (and system
components) to become more ‘aware’ of what is
happening outside their direct environment, and take
this into account in their control and optimization actions. The developments towards more autonomous
operational decision making, as described in the
So, on the one side data analytics will allow for previous introduction, makes sense only if there are
automating parts of the (previously manual) ‘top down’ continuously changing circumstances to which the
optimization and tweaking of processes and systems. operations need to adapt to, as is the case in the power
This becomes an integral part of the system or process sector.
itself, as is shown in igure 7. On the other side process
control (e.g. security of components in grids) is Among the largest challenges the power system is facing,
continuously expanding its scope and able to take is the incorporation of more renewable energy sources,
more and more information into account. while remaining dependable and affordable. While this
will include controllable renewable energy sources like
The result of both these developments will be the biomass and geothermal, the bulk of the energy will be
automation of operational decision making and in provided only when there is wind and sunshine. Wind
processes that will become more robust, lexible and and solar are energy sources that are geographically
adaptive to external inluences. Increasing eficiency distributed and vary with changing weather conditions.
does no longer automatically come at the expense of At the same time, the electricity demand is changing,
loss of lexibility. mostly due to the electriication of transport and heating.
This means that the variables in the power system, i.e. the
size and direction of power lows, the relative balance,
and the power quality will change continuously. Also, the
long-term developments in both demand and generation
are much more uncertain than they used to be.

PERIODIC ANALYTICS
PERIODIC ANALYTICS
(e.g. business process redesign, Six Sigma etc.)
Process improvements,
tweaks to reduce disturbances Continuous analytics, adaptive control, situational
awareness to continuously optimize the process or system
ADAPTIVE ADAPTIVE ADAPTIVE

CONTROL CONTROL CONTROL CONTROL CONTROL CONTROL
Subprocess or Subprocess or Subprocess or Subprocess or Subprocess or Subprocess or

component component component component component component
PROCESS PROCESS
Figure 7 - A significant part of the 'manual' process analytics will be automated and become part of the process itself.
At the same time component/step control will become more aware of its environment and other components
in the system (situational awareness).
The traditional solution to cope with such uncertainty is The power system can therefore beneit greatly from
to increase the network capacity and/or the generation the integration of data analytics supported optimization
capacity. However, this solution is becoming increasingly into operational processes, as described in the
expensive for the described uncertainty, because of two introduction of this chapter. However, it can only do so if
reasons: certain conditions are met. The most important condition
is that there should be enough (near) real-time data from
 Variable renewable generation (i.e. wind and solar) sensors in the system to assess the situation and status.
is highly correlated. New electric demand Second, there should be enough possibilities or ‘buttons
(electric vehicle charging as well as heat pumps) to press’ to adapt the system to cope with the changing
is also potentially highly correlated. To accommodate environment, i.e. ‘actors. Finally, the right models and
peaks caused by this correlated demand and this ‘meta-models’ should be available to respectively give
(local) generation would require an enormous the gathered sensor data a context and relevance; issue
increase in grid transmission and distribution capacity, the right command to the ‘actors’; and adapt the models
which would have a low degree of utilization. themselves to learn and keep the system optimal under
 As the future becomes less predictable, the risks that changing circumstances. All these conditions are
traditional large and long term investments in both beginning to be satisied.
generation as transmission capacity become ‘stranded
assets’ are increasing drastically, leading to a relative The number of sensors in the grid is rapidly increasing.
higher attractiveness of more lexible and short term Today, smart meters for example can act as the receptors
solutions.19 on the outer edges of power systems, and an increasing
number of smart devices will do the same. Within the
grid, Phasor Measuring Units (PMUs) are installed at
strategic locations, measuring all aspects of the power
low.

At the same time ‘actors’—the tools to inluence and FRAUD DETECTION USING
optimize the grid and power system—are being
developed. Automated demand response (DR) is gaining SMART METER DATA
renewed interest, for example the possibility for smart
charging electric vehicles. Load Tap Changers (LTC),
smart transformers and the control of the reactive power
through inverters (of e.g. solar panels) can make the grid
much more bi-directional. Concepts like self-healing
grids are being developed. For example, by
automatically reconiguring a (meshed or ring) grid as a
response to very fast fault passage indicators, the effects
of a failure in the grid can be reduced to a minimum,
or even avoided. Grid maintenance, as well as the
maintenance of power plants, is more and more planned
and executed based on the outcome and analytics of
operational and other data (such as market data).
So, both the development and deployment of sensorsin

the power system as the development of ‘actors’ is In (12) some practical applications of data
ongoing. To interpret and relate sensor measurements
analysis in distribution grids are analysed.
and other data from many different sources and
locations, to draw conclusions from them and One typical example considers the detection of
eventually translate these into appropriate ‘actions’, “non-technical losses” (NTL), in particular electricity
models are inevitable. theft by tampering with or bypassing the meters.
This includes use of electricity for illegal drugs
For power transmission and distribution systems, ‘state cultivation. Typical indirect methods use periodic
estimation’20 and load low models come to mind, but
behavior of currents (aligned with growing
depending on the application also other kinds of models
will be used. Utilities and aggregators are using models patterns), using data mining techniques. Ideally
relating weather data, data about loads, measured grid the exact location of the perpetrators is identiied.
frequency and other sources to predict imbalance and The newly proposed methods use well established
intraday market prices and optimize their generation load low analysis to estimate “normal conditions”
and load portfolio accordingly, e.g. through demand in terms of active and reactive power and voltage
response. Many other examples of models are available,
and expected (technical) losses. This can be done
especially for specialized situations or local decision
making21. per distribution substation and per feeder, based
on smart meter data. Once deviations in the voltage
While these models themselves often are based on proiles exceed a certain threshold localisation is
physics, they need to be adjusted and tweaked to adapt done. Note that even the smart meters do not
to constantly changing (often unmeasured) parameters. identify the power that is withdrawn illegally
This process relies heavily on automated data analytics
(by deinition) but they still measure voltages.
and machine learning and assures that the automated
decision making is adapting constantly to the changing
circumstances. Examples are the constantly changing Petr Kadurek based his method on smart meter data,
portfolios of retailers, aggregators and trading in particular voltage data. Newer methods
companies; the changing load and/or generation at may have to rely on or include PQ-data (Power
grid connections because consumers install solar panels, Quality), since illegal growing tends to use LED
a heat pump or an electric vehicle. The models used for
lighting, using much less power (so smaller power
automated decision making will adapt to this. They will
‘learn’ and adapt to these changing circumstances. deicits and smaller voltage drops), but added
harmonics due to the electronics, THD (Total
Harmonic Distortion) may be used as a base for
analysis, comparing simulations and measurements.

There are already many different techniques that have
been developed and are now available. For example,
the Bayesian method to ‘tune’ model parameters and
evolutionary algorithms for various outcomes, such as
minimizing risks by optimizing over possible hazardous
scenarios, neural nets for identifying and recognizing
patterns and anticipating accordingly, etc.
These ‘meta- models’ sometimes will be a separate

model or algorithm ‘wrapped’ as a separate layer around
the system model, and sometimes will be an integrated
part of the model of the—to be managed—system itself.
In the next three sections examples of the possible

impact of data analytics on grid operations, commercial
wholesale operations and retail operations are given
respectively.
4.2 The impact of analytics on

grid operations
For both utilities as grid operators, the ability to optimize

and ine-tune their operations is ever increasing as more
buttons and dials become available. However, as they
control increasingly detailed aspects of the power
system, they increasingly require more and different
kinds of data and more advanced models and data
analytics to do so. So, to fully beneit from data
analytics, it is inevitable that the analytical processes,
as well as their eventual results—i.e. the decisions and
actions—are automated. A typical example is the ELVIS
information system of FinGrid, applying among others,
IBM’s Dr. Watson, to enhance many grid related
applications22, in operations as well as in asset
management.
DSOs need to automate data analytics for optimizing

operations to a much larger degree than utilities or TSOs.
This is due to the size and numbers of the issues they
might face. Instead of facing a few ‘global’ problems,
DSOs face multiple relatively small local problems.
For example, capacity problems due to the intermittence
of solar power generation or the sudden charging of
electrical vehicles will likely appear in many parts of the
grid simultaneously. Each of these individual problems
will be too small to justify manual decision making.
So, what kinds of operational decisions can be made?

Grid automation includes tap changers and smart
transformers, managing voltage levels in the grid so
power can low ‘two ways’. It includes automated grid

reconiguration of loops to reduce both grid losses as possible to incorporate scenarios and risk in
well as reduce the effects of potential outages and the operations23, gaining large operational beneits both in
effects of outages themselves, in the context/case of cost reductions as in quality or risk reduction (e.g. less
self-healing grids. It might include activating automated customer minutes lost). Basically, operations can be
demand response and local storage, that will help DSOs optimized using several forecast scenarios and potential
to reduce peak loads to avoid grid losses and capacity risks, including risks associated to using ICT itself.
problems. Also, the management of the maintenance
work force will be optimized using more data and smart The simulation model(s) itself can be very fast, and do
analytics. not need to be very accurate or complicated. This may
be due to several factors. For example, data from
The operations department within the DSO will get a strategically placed measurements will be used to
more strategic role as grid automation and the use of the continuously recalibrate them, and they can draw from
lexibility of customers’ assets will gain importance and previously compiled more detailed simulations. These
prove to be a viable alternative to investments in excess models will be able to prevent grid instabilities that
capacity to accommodate the increasing amount of otherwise might emerge because of unexpected
variable renewable generation and demand. interference of behavior of smart devices24.
This behavior is simulated by 'digital twins', digital
This will be achieved by incorporating more automated models that are compiled beforehand and then tested
analytics and decision support, both on a local and on and certiied so they can be trusted to react the same as
a global level, to utilize the potential of lexible demand as their real counterparts and thus can be used to study
of electric vehicles, heat pumps, building climate and examine their behavior on the grid and on other
installations and some industrial processes to balance digital twins. They can be published for use by other grid
against the variations in the power generation of operators, for example using blockchain technology to
renewables, reduce resistive losses and implement c ensure their integrity.
oncepts like self-healing grids. A schematic of how
this might work is shown in igure 8. For other stakeholders in the power system, data
analytics and automated decision making can play a
Through ‘faster-than-real-time’ simulations e.g. based similar role in operations. Automation already plays a
on load low simulations and state estimation it will be major part in power trading (similar as in stock market
trading) and in control of power plants, and is likely to
expand further.
NETWORK RISKS
Activities to reduce risks
GRID OPERATIONS
(DMS, switching, DATA ACQUISITION STRATEGIC THREATS
MODEL
disturbances, events, (measurements) AND SCENARIOS
workforce development)
OTHER PROCESSES
Activities to optimize operations
NETWORK STATUS (strategy, asset
management
investments)
Figure 8 - Risk-based grid operations: a schematic of how faster than real time simulations, continuously calibrated with
measurements and data from the grid can be used to continuously assess and minimize risk and optimize the
grid configuration, work force deployment, etc.

4.3 The impact of analytics An example of such a product might be a kind of
‘reinsurance’ of imbalance risks that might appear around
on commercial wholesale the balancing and program responsibility mechanisms.
operations Traders trade energy in time blocks called ‘Imbalance
Settlement Period’ (ISP, in most countries this is 15
minutes). If the physical demand and generation in an
Like network operation, commercial operations will ISP does not match, they must compensate the TSO for
be highly affected by data analytics, even more than it solving the created problem. To solve these imbalances
already is. Commercial operations include (renewable) the TSO has contracted power generation, and in some
power generation, trading, electricity supply and countries, the TSO has organized an ‘automatic
aggregation. The commercial domain in the semi-real-time auction’ (the imbalance market) where
electricity sector can roughly be divided into a wholesale market parties can offer power to the TSO to restore the
part, where energy parties interact with each other, and balance. The cost the TSO incurs (and forwards to the
a retail part, where energy is sold to (or bought from) responsible party that caused the imbalance) is only
essentially non-energy companies and citizens. While this known after the ISP is closed. For many smaller
division is blurring as companies and citizens are getting parties25 there are beneits to lock this price in a trade
more involved in ‘traditional’ wholesale activities, such with another party (which can more easily bear or hedge
as power generation; and changing business models of this risk, e.g. because of aggregation or available
retailers and aggregators are integrating wholesale and lexibility) for a price before or even during an ISP.
retail processes more and more, this division is still quite
insightful. Numerous other possible new products and markets
will emerge as a response to data analytics and data
As discussed in chapter 3, data analytics play a major part exchange. For example, around Power Purchase
in forecasting renewable generation and demand, as well Agreements (PPAs) between renewable generators,
as in market price forecasting. As forecasting is being storage and industry (6), especially industry able to
developed to become increasingly sophisticated, it response to luctuations in renewable output: Industry
results in temporary advantages to the companies who would secure renewable energy, while the renewable
forecast best. However, markets and especially market generator would beneit from being shielded from the
prices, are dependent on the actions of other luctuations of the power market by synchronous
stakeholders and their forecasts. So, while market generation of renewable energy. Other examples might
forecasting is tremendously important to the relative be around differentiation of energy by source, location
competitive advantage of the participants, it is an arms and/or even application; or around products where
race that does not really change the system overall. risks of intermittency of renewables are shared and/or
mitigated through ‘private’ capacity products; or—as will
In operations, as in other domains, it is the actions that be discussed in section 5.4 'Advanced energy contracts'—
count. More data, and especially more real time data will integrate the value chain by utilizing new retail products
give traders, retailers and aggregators a better insight based on demand response and local storage owned by
in their current and future market positions and risks, end-users.
leading to a need to develop the tools to swiftly act upon
these insights. To optimize their market positions, they
will need to adjust their portfolio of generation, demand
and contracts, by trading among each other. This will
lead to new tradable products and more differentiated
markets.
Just like the increase in solar energy will likely lead to a

further differentiation of tradable futures beyond the
current ‘base’, ‘peak’ and ‘16h-peak’ products, more
(real time) data will lead to a differentiation short term
tradable products (and possibly new market places) to
utilize the results of these analytics.

4.4 The impact of analytics on
retail operations
In the electricity sector retail is generally reserved to

indicate electricity sales to commercial and residential
end-users, and often also includes energy services and
energy data services to end-users. Although through
developments such as local generation (the emergence
of prosumers26) and demand response (end-users
offering lexibility to the electricity sector), the distinction
between retail and wholesale is becoming blurred.
The impact of data analytics on retail operations will be

much more visible to the public than the impact of data
analytics on wholesale and grid operations. Like in all
other consumer sectors, data analytics and big data
allow retailers and aggregators to much better tweak
their propositions to individual consumers, creating
higher marketing eficiencies and sales,27 and
eventually leading to smart energy contracts, that respect
the individual wishes and requirements of consumers
instead of the ‘one size its all’ energy contracts that are
dominant today.
On appliance level, smart thermostats exist already for

a number of years, monitoring the behavior of residents
in a building and optimize heating accordingly28. ‘Boxes’
(physical or virtual cloud-based) that analyse smart meter
data go way beyond merely showing energy use and are
Eventually consumers might become a
able to disaggregate smart meter data into the load of
individual appliances using self-learning (non-intrusive) digital Gladstone Gander, the lucky bird
load monitoring algorithms29. from the Donald Duck cartoons. ‘Gladstone
Gander services’ imply that eventually your
The change in retail operations that will likely have the ‘digital’ life will become that of Gladstone
biggest impact on the other parts of the electricity value Gander. Artificial Intelligence modelled
chain, will likely come from retailers and aggregators
around you, will ‘tweak’ cyber space around
aggregating lexibility from end users, using demand
response. They might use this lexibility to optimize you so it seems everything will work in your
trading on electricity markets; sell it to grid operators as favour. All information (like product and
emergency power, or to solve congestions - adopting the service propositions) will be fed to you
concept of lexibility as a new commodity. Alternatively, exactly when you subconsciously want it.
they might use it to tune their client's individual energy The more information is known about you
use to their speciic wishes, thus using lexibility as
the better and more easy your (digital) life
diversiication opportunity in the retail market. In section
4.5 ‘Aggregating demand response in smart energy will become (at a price).
systems’ and 5.4 ‘Advanced energy contracts’, this is
further discussed.

4.5 Aggregating demand While DR has a huge potential, for it to be responsive
enough to add value to the continuously changing power
response in smart energy system, demand response will need a much higher
systems degree of automation and automated decision making
than what is commonly commercially available now.
This needs to be on all levels – from the individual device
level, to the decision to dispatch hundreds of megawatts
In the previous sections, a number of subjects have been by changing the charging behavior of electric vehicles.
addressed that often are associated with ‘smart grids’, Behavior. Therefore, data analytics will play a major part
‘smart markets’ or ‘smart energy systems’, including in the future of demand response.
automated demand response. The previous paragraphs
did not address an important aspect of demand As mentioned in the introduction of this section, the
response (DR) , which is what demand response involvement of multiple stakeholders makes demand
involves—devices and processes of—end users that have response complicated. Flexibility is a resource owned
their own requirements and conditions. They likely will by the customer that must be bought (back) from the
demand compensation for their services to the system, customer32. This is such a complicated and specialized
even if these conditions are met. The exchange of these task that often this is tasked to a dedicated role: the
DR services are transactions and smart grid concepts role of aggregator. An aggregator aggregates demand
that take this into account are often referred to as response and other sources of lexibility to sell to other
‘transactional’ smart grids). However, these requirements stakeholders in the power system and to make sure
and conditions are not ixed and will depend on the end that all possible obligations with these stakeholders are
users’ continuously changing needs and circumstances. fulilled (such as balance responsibility and capacity
constrains).
Demand response is the most economically feasible
solution to provide lexibility to accommodate higher Aggregators that focus on aggregating many small
penetrations of renewable variable energy from the wind end-users need to be able to monitor and control
and the sun30. It is in many ways already an established hundreds of thousands to millions of devices,
business. For decades, day and night tariffs have existed considering the speciic circumstances of each individual
in many countries. Direct control of both devices and device in (near) real time. This requires an enormous
processes, like air conditioners and aluminum smelters, amount of very fast data analytics and communication.
has been done for a long time. More recently, However, depending on the design of such an
specialized independent aggregators are emerging, aggregator system, a lot of this data analytics can be
acting as ‘brokers’ between (industrial) end users and done by the devices themselves (a phenomenon called
TSOs for ancillary services like emergency power31. ‘edge computing’).

A demand response system of an aggregator must Dispatchable DR thus can feel quite intrusive to
address two issues: customers, although many adaptions have been made
to soften this. This is especially true for the United States
 Control: How to control the demand (or local where dispatchable DR is much more common than in
generation) of all the customers most parts of the world. Examples range from remotely
 Value allocation: How to reward these preferably controlled air-conditioning to load shedding contracts
in relation to merit with large aluminium smelters. In the Unites States
experience shows that the reliability of dispatchable de-
These two aspects of demand response are highly mand response is as reliable as controlled generation.
intertwined as the control mechanism must control
demand response in such a way that most value is Non-dispatchable demand response, sometimes referred
created, meaning customers and devices that are least to as implicit demand response, focuses on the value
disadvantaged (i.e. require least compensation) will be allocation system, often relying on the behavior of
dispatched irst. customers to adjust their demand in response to price
incentives. This makes non-dispatchable demand
Historically the emphasis of demand response was either response programs not reliable enough to rely on for
on control (e.g. direct control) or on the value allocation grid safety and stability, limiting its use and value to a
(e.g. day/night tariffs), leading to respectively ‘support’ role. Examples are day/night tariffs and critical
dispatchable and non-dispatchable demand response. peak pricing.
Dispatchable demand response, sometimes called However, this division between dispatchable and
explicit demand response, focuses on the control system, non-dispatchable DR is disappearing. Dispatchable and
often simplifying the value allocation system to a ixed non-dispatchable DR, where respectively a control signal
fee to the participating customers. It is dificult to adjust or a price signal is communicated are essentially
dispatchable DR programs to changing desires and 'one-way communication' concepts. Two-way
circumstances of customers without involving a more communications in combination with data analytics and
advanced value allocation system. A customer that does local intelligence is changing this rapidly.
not mind the air-conditioning to be switched off one day,
might ind it uncomfortable the next.

LOAD SCHEDULING
(ELECTRIC MOBILITY)
EXAMPLE: Dispatchable demand response
Load (e.g. electric boilers) is scheduled to the - for the DSO, TSO
or BRP - optimal time
ADVANTAGES
Power (kW)
 Technically relatively easy to implement

 Direct control of loads
DISADVANTAGES
 Dificult to evaluate the value for the power system and the individual customer
Time
 Dificult to apply to more than one use of lexibility
TIME OF USE PRICING

EXAMPLE: Non-dispatchable demand response
Time of use pricing
ADVANTAGES
(Value to suupplier)
 Relatively straightforward
Price (€)
 Lots of experience with day/night tariffs

 Customers are free to react on incentive
DISADVANTAGES
 No direct control of loads (uncertainty of the response and ability to 'ine tune')
 Possible instability problems if automation is applied on a large scale
Time
REAL TIME MARKET PRICING

EXAMPLE: An integrated approach
Market-based coordination (day-ahead markets, for small scale: PowerMatcher,
Intelligator)
Result of bidding of supply and demand
ADVANTAGES
Value (€)
Supply  'Multi-goal optimization' ane enumeration possible between stakeholders
Price
DISADVANTAGES
Demand  Timing: anticipation to (global) changes in the system should be
implemented locally (through bids), making local intelligence more
Power (kW) crucial
Figure 9 - Examples of demand response concepts: each of them communicates on two of three possible axes: power, value and time.
Each concept is missing one of these axes, which is the root of its disadvantages. Dispatchable demand response misses value;
non-dispatchable demand response misses power and a market approach misses time, resulting in planning challenges.

Traditional dispatchable demand response controls For most parts of the power system the necessary
power directly: For each point in time the amount of prerequisites are available or soon will be:
controllable power is controlled by the central control
system (as shown in the irst graph in igure 9). Traditional  Sensors to sense the ‘state’ of the system,
non-dispatchable demand response sets the value of environment and relevant conditions
power directly (for example hourly prices, shown in  Actors that affect the system and can optimize
the second graph in igure 9). One solution for the operation
integration of dispatchable and non-dispatchable  Models to translate sensor data into the system and
demand response lies in recognizing that combining derive the appropriate actions
them results in a mechanism very much resembling a  Meta models to adjust and the models to capture
market (see third graph in igure 9). higher order changes (machine learning techniques)
Because the wishes and requirements of individual For network operations, this will result in a much deeper
consumers can be implemented in the demand response control at the edges of the network, incorporating
scheme, retailers or aggregators will be able to offer demand response and control of decentral generation.
much more tailored energy contracts to their customers. Inluencing demand and local generation means that not
They can offer services that optimize the electricity use only the grid itself is considered, but also the speciic
of a customer to make greatest use of the solar energy circumstances of end-users’ processes and energy
from his neighbourhood, while offering his neighbour markets (either directly or through intermediaries like
a service that minimizes his bill (see also chapter 5). aggregators). This requires very fast models utilizing for
At the same time the remaining lexibility can be used to example precompiled simulations of more complicated
offer services to the grid operator to avoid congestions aspects of the system system, such as using certiied
and to optimize the aggregator’s position on the markets. ‘digital twins’.
An example of a pilot project where these propositions
were implemented is discussed on page 42 Data analytics in wholesale operations also will utilize
(PowerMatchingCity). sensors such as smart meters (and many other sensors),
but then to create and utilize this data to formulate and
optimize opportunities to trade eventually leading to
new tradable products and new markets; for example by
4.6 Summary for operations activating demand response and small scale residential
storage to optimize intraday and balancing positions.
In general, the impact of data analytics on operations can
be characterized by leaving behind the old ‘paradigm’ These products coming from wholesale optimization,
that a choice needs to be made between eficiency and are materializing in new retail products. Already energy
lexibility, or at least bring this paradigm to a next level. suppliers are diverging their product range (e.g. offering
Periodic analyses made by staff departments to optimize hourly wholesale prices to their clients). Data analytics
processes are replaced by machine learning algorithms, will tweak retail propositions to redistribute costs and
using online data that continuously will tweak and risks between the energy supplier/aggregator, wholesale
optimize the processes. traders/generators and end-users. It will incorporate
demand response, possibly automatically dispatched
For the power system, which used to handle variations in through ‘smart contracts’, making sure to incorporate the
demand and disturbances with capacity and redundancy, speciic wishes of the consumer, for example using real
this change comes just in time. As the variations in the time renewable energy from a speciic wind turbine.
system will become dominated by renewable
generation as well as demand with very synchronous
behavior, this traditional response of increasing capacity
becomes increasingly more expensive, both in regard
to variable loads caused by variable renewables and
synchronous demand as with regard to the risk of
stranded assets, that turn out to be obsolete before
their anticipated end of life, due to the fast changing
(economic or regulatory) environment.

ANALYTICS ON SETTLEMENT
Energy settlement is the process by which the difference between electricity generated and
electricity sold is reconciled. The process consists mainly of allocating metering data to the
responsible market parties and associated contracts and, afterwards, the associated payments.
D ata analytics in forecasting is mostly concerned

with using existing data to forecast what this
5.1 Settlement in unbundled
data will be in the future. Data analytics in operations is markets
mostly concerned with translating both existing data and
forecast data into control and optimization actions. In an unbundled electricity market, the buying and selling
of electricity and gas is done by commercial companies,
Like operations, data analytics applied to settlement is while the transport and distribution of the energy is done
mostly concerned with optimization, but on a longer through a public network, managed by independent
timeframe: optimizing energy transactions and creating transmission and/or distribution system operators that
new competitive energy services and propositions. are regulated and monitored by government agencies
(Regulators). Besides managing the system and network,
transmission and distribution system operators are
usually tasked to facilitate the market.

Because energy lows through the public network, 5.2 The impact of data analytics
following the laws of physics and not necessarily
according to contractual arrangements, these lows
on wholesale settlement
cannot be physically traced back to the buyers and s
ellers of this energy. Therefore, settlement in an Data analytics is a mathematical toolbox. It is the use
unbundled electricity market is an elaborate and of these tools that bring the beneits. So, how does
data intensive administrative process. this relate to settlement, being basically the process of
calculating (and remunerating) the difference between
Besides the billing to energy consumers by energy previously negotiated contracts and agreements and the
suppliers, it consists of processing the meter data of all actual measured energy lows using metering data?
generation feeding into the grid and all demand drawn
from it and allocate this to the responsible commercial Obviously, a lot of data is being processed during
parties, as well as using this data to settle the deviations settlement, and data analytics on settlement data can
of the physical power lows with all commercial contracts extract more data that brings value for other processes,
and obligations that were traded bilaterally or on the like forecasting, marketing and sales. Settlement itself
energy markets prior to the physical delivery (often however is an administrative process, and there appears
referred to as 'imbalance'). to be little room for applying new insights that might
result in beneits within the wholesale process itself.
Small, usually residential users have a special place in this Which of course does not imply that data analytics
process. In most countries, their energy use is considered cannot be used to make this process itself more eficient,
through an artiicial standardized proile, which once faster and secure.
a year is ‘scaled’ based on their total measured annual
consumption and then settled between the responsible However, at the boundaries of the settlement processes
energy suppliers in a process called reconciliation. things are changing. The most evident being the use of
Smart meters (see also section 5.3) will allow the energy 15-minute smart meter data in the allocation process,
consumption and generation of these small end-users with most of the potential impact likely to be most felt
to be included into the ‘real time’ allocation of the on retail billing and retail contracts. More and faster data
electricity. In most countries, the energy market is might lead to a better insight in the positions and risks
discretized into 15-minute time slots called ‘Imbalance of electricity traders, especially in relation to imbalance
Settlement Period’ or ISP33. settlement as mentioned in section 4.3.

This scenario may lead to new risk mitigation and trade Settlement based on smart meter data (i.e. ‘billing’)
options, for example pre-, or real-time settlement involves huge streams of data and requires
imbalance markets, where imbalance risks can be corresponding investments in ICT infrastructure,
hedged; and new demand response contracts targeting but processing this data for settlement still can be
speciic residential end-users, such as electric vehicle considered ‘standard’ ICT.
owners.
The impact of allocation on smart meter data instead
And inally, the emergence of (independent) of settlement on standardized average proiles can be
aggregators, which focus on lexibility instead on bulk huge, because this means that it becomes possible to
energy sales, imply thatimplies that a separation needs directly link the settlement between energy suppliers and
to be made between the electricity supplied by the their consumers (billing) with the settlement of the rest
traditional energy supplier and the lexibility created of the electricity system (allocation and reconciliation).
by the aggregator. Sometimes installing a separate In other words, it becomes possible to create innovative
meter will be suficient to distinguish between the energy energy contracts for consumers based on real prices
supplied by different suppliers, but in the future, more and risks on energy markets (instead on merely yearly
often than not data analytics will need to be applied. averages). So, smart metering allows for advanced time
An example would be when there is a need to determine of use contracts, dynamic pricing and other contracts
a base line for explicit demand response sold by an based on optimizing energy use on energy price and
aggregator to the transmission system operator as risks or for example, contractually link the energy use to
emergency power. The base line being the energy renewable intermittent renewable energy sources like a
demand (or generation) that would have been supplied nearby wind farm as will be discussed in the next section.
by the energy supplier if the aggregator would not have
activated the demand response (7).
5.3 The impact of smart

metering
Smart metering is often seen as a way to make
consumers aware of their energy use and, hopefully,
create the interest to reduce it. Data analytics is seen as a
tool to give consumers the necessary insight and options
to reduce their energy use. These include various
visualization tools; load disaggregation (NIALM34);
benchmarking with peer groups; concrete and consumer There are already a number of energy suppliers that pass
speciic proposed actions35; etc. And indeed, smart day-ahead market prices to their customers, and earn
boxes, thermostats and monitors that offer such their margins from ixed fees.36. Because they ‘spread’
functionality ind their way towards consumers. the risks of trading on the day ahead market over their
customers instead of hedging this risk by spreading it
Data analysis of smart meter data also has a potential over their sources/procurement contracts, they can
beyond the energy domain: Smart Cities. Detailed data be structurally cheaper than traditional suppliers as
of energy consumption and generation might play a end-users, are generally less affected by higher energy
role in monitoring and optimizing public services other prices for a short duration than the energy supplier is,
than energy supply, for example: trafic control, security, thus an overall smaller ‘insurance cost’ should be paid.
healthcare, etc. However, this is beyond the scope of this Still, the marketing of these kind of energy contracts to
paper. risk-averse end-users still is a major challenge.

Propositions based on sourcing take the source of the
5.4 Advanced energy energy into account. Originally end users could only
contracts choose one type of energy, which was differentiated into
‘green’ and ‘grey’ energy. Currently, more variations are
emerging. Speciically for electricity, these can be traced
As mentioned in the previous section, smart meter back to either only power only from national wind farms;
data combined with data analytics will pave the way for or to a speciic farm with solar panels chosen by the
new advanced energy contracts. Besides the previous consumers themselves. For electricity that is generated
mentioned inancial and risk-based energy contracts under the appropriate criteria a ‘guarantee of origin’ is
(such as retail prices based on the day ahead market), issued by a certiication body to the generator.
smart meter data combined with data analytics will Users buying this energy will receive the appropriate
allow energy suppliers and aggregators to further certiicates40 and when the energy is consumed, the
differentiate their offerings on other values than price certiicates are destroyed. This certiication system
and risk. In liberalized energy markets, where energy ensures that the total amount of electricity used matches
suppliers ind it dificult to differentiate their offers, this the total amount of certiied electricity produced within a
is a major issue. To avoid direct competition on price, certain predeined period, usually a year.
in a retail market that is becoming more and more
transparent, energy suppliers need to differentiate their Big Data and data analytics can make this system much
products from their competitors and ind niches in the more reined. By combining generation data with
market37. consumption data, it becomes possible to differentiate
this certiication not only on source, but also in time41,
While the amount of possible energy supply services is so consumers can choose to buy real-time solar power
limitless, there are only two basic parameters that are and, when the sun sets, supplement it by the local wind
connected to the energy supply itself that can be used farm or battery storage. Consumers thus can choose
to differentiate services from those of competitors. to become truly ‘independent’ from fossil ired power
These are the price (actually price/risk) and the source plants. Energy suppliers/aggregators might differentiate
of the energy38, which data analytics will have a major themselves by becoming facilitators of inter-customer
impact on. trade, using timed dependent information of origin and
destination to allow customers to exchange electricity
Propositions based on price/risk try to optimize the among each other including their collectively owned
energy bill for the customer, considering his risk appetite wind turbine42.
and ability to reduce this risk in other ways. The rule of
thumb is that the higher the risk, the lower the expected While these examples might seem a little exotic, they
bill will be. An example is dynamic pricing, where the were demonstrated in quite some smart energy pilots.
consumer price is varying every 15 minutes based on For example, in PowerMatching City II43, automatic
actual market prices or renewable generation. demand response based on either dynamic pricing
Consumers that are very aware, or have lexible loads that or the availability of local energy from solar panels—
might be adjusted automatically (e.g. EV charging) would including wholesale settlement—was demonstrated,
beneit with a variable, but on average lower energy using heat pumps, micro CHP’s and electric vehicles
bills39. (8) (9). And while the current public interest remains low,
the interest from some companies and public bodies
On the risk-averse extreme, this could be a ixed price wanting to go beyond ‘merely’—on average—renewable
contact, which does not change regardless of the amount to fully—real-time—renewable, is increasing.
of energy used (e.g. a capacity contract, only limited by a
special maximizing switch in the connection to the grid).
The consumer then knows exactly how much he is going
to pay irrelevant of how much energy he uses. The risk for
the energy supplier will be higher (maybe limited by the
special digital or physical ‘fuse’) and this will be included
in the price. Of course, energy suppliers will be able to
use data analytics to keep this risk as low as possible.

POWERMATCHING CITY:
RECONCILING CONFLICTING GOALS IN A SMART ENVIRONMENT
As mentioned in paragraph 4.5, dispatchable demand response and non-dispatchable demand response
can be integrated through a market mechanism in such a way that demand can be operated as a virtual
power plant, while still enabling the participating clients the freedom not to participate when they are not
lexible enough or if the price is too low.
PowerMatching technology as applied in PowerMatchingCity (part of the EU FP6 project INTEGRAL),

basically communicates the energy need of a device to the system based on an economic measure.
For example, a heat pump/heat buffer combination can offer to use energy for a price that depends on
the ill level of the heat buffer. If the buffer is almost empty, it would make a bid to ‘buy’ energy even for
a high price. If near full, it would offer to buy energy only if the price is low.
One of the advantages of such a system is that all devices will make a bid to use or produce electricity just
a few minutes in advance. This means that the system is basically a ‘feed forward control system’, with all
the beneits for stability. The bid curves can easily be aggregated to form bid curves on higher levels, like
households, aggregated to district level, to aggregator or retailer portfolio level etc. Control or dispatch
can then be done by communicating the price associated with the required power demand or generation.
For traders, this system acts as if dispatching through a virtual power plant. Access to the bid curve of the
whole portfolio provides a relation between power change and price change. By changing the electricity
price of the consumers (the retail price through the aggregator) traders unlock this lexibility. It can be used
to optimize the trader’s portfolio or it can be sold to the intraday or balancing markets.
For the aggregator or retailer, this demand response opens a new dimension for the development of
energy products and contracts based on the preferences their customers, up to the point to facilitate their
consumers to organize their own energy (sub)community.
The ‘default’ option is that devices are optimized on the retail price set by the aggregator. This price is
the effect of the aggregator’s effort to optimize the load and local generation on behalf of its customers.
Consumers and local generators that are the aggregator’s customers which (automatically) react on this
price will receive a lower energy bill.
In addition, the aggregator’s customers can react on other information as well. Consumers that would like
to maximize the fraction of a certain kind of energy (for example solar) or certain sources such as locally-
generated energy can do so. If this energy is available they can change their bids accordingly, by using
real time certiicates, or just measuring data.

For grid operators, short term capacity management is made available. The aggregated bid curve can be
used to set local prices to make sure that the total load (or generation) will be within all capacity boundaries.
Within an area with an expected congestion area there is another price than outside? Communicating this
price to the devices around the (expected) congested area will avoid overloading.
The project PowerMatchingCity has already demonstrated that smart grids are technically feasible.
 Flexibility has an economic value for the economy
 Energy services can be created that meet the needs of consumers
 Market barriers that impede the monetization of lexibility can be eliminated relatively cheaply
MARKET OPTIMIZATION
HOME OPTIMIZATION
NETWORK OPTIMIZATION

The role of blockchain in retail
As in other sectors, blockchain is receiving a lot of Blockchain (or alternatively tangle45) is considered as
attention in the power and utility sector as well. It can an important piece of the puzzle in the how the power
be used in many applications, from cryptocurrencies to sector might develop. Just as the shift from fossil to
facilitate energy trade among citizens to digital models renewable electricity generation is breaking the business
of power electronic equipment for use in stability logic of ever-increasing scale in central (thermal) power
assessments by grid operators as discussed in paragraph generation, blockchain based transactions might break
4.2 ‘The impact of analytics on grid operations’. Although the business logic in energy supply and billing46,
the main application is seen in energy or power making blockchain a potential game changer, especially
transactions between end-users (or their equipment in energy retail, which allows the emergence of new
when talking about the internet of things) and/or players in the market, among which (cooperatives of)
stakeholders in the electricity system such as retailers of their own clients—active consumers—themselves.
the TSOs.
Consumers are actively organizing themselves into
Blockchain basically is a shared, distributed ledger of cooperatives, installing solar panels on their roofs or
blocks of transactions, or other entries to the ledger, collectively build wind farms to generate and supply
that are chained to each other. Each block has a hash themselves with renewable energy. Blockchain (and
(basically a digital ingerprint that depends on the tangle) technologies have many characteristics that seem
content of the block) and contains the hash of the like a natural it with this development. Organizing a
previous block. So, once an entry is recorded in a block, cooperative requires a high degree of organization and
it cannot be altered anymore. Most blockchains are dedication. Not only in initiating such initiatives, but
permissioned blockchains, meaning that all participants especially in maintenance and administration.
are known and trusted. There are some applications—like Blockchain might facilitate a major part of this
Bitcoin or Ethereum—that are permissionless (open) administration, such as the automatic registration (and
and everyone can make transactions as well as add new remuneration) of energy transactions. Many applications
blocks to the chain, as long as you can ‘prove that you using blockchain already have been developed that are,
have an interest in upkeeping the integrity of the chain’, or can be used for energy applications; ranging from
by showing a proof of effort44 (or e.g. a proof of stake) trusted peer to peer energy transactions and billing
and the newly added block has been veriied by others. applications47 to certiication.
This process of adding new blocks to the chain is called
‘mining’ and successful adding a new block to the chain
is rewarded, for example by gaining bitcoins.

For a successful and secure permissionless open Using a blockchain to facilitate peer to peer trade
blockchain application quite a lot of decentral between end users to circumvent the role of an energy
computing power is required. Creating a separate supplier that does not take this ‘imbalance risk’ into
blockchain for each application seems impractical and account, would still require a contract with large party
not very secure. It seems plausible that many blockchain that can insure these risks. Such a blockchain will be an
applications will share a common blockchain, and a interesting tool for the energy supplier/aggregator/
limited number of platforms/’tokens’ that support balance responsible party to make their processes more
users to make their own blockchain applications will eficient and secure, but it will be just an evolution of the
appear. Examples of such platforms that support existing business model of energy supplier/aggregator,
multiple blockchain applications with one blockchain enhancing their services to facilitate their customers to
that is large enough to ensure the security of the have more choice in selecting the source of the energy
application are Ethereum and Electron. they use.
There are more hurdles to overcome before blockchain However, it is only a matter of time before blockchain
can break ‘the business logic in energy supply and applications appear that might be able to create
energy billing’. For example; exchanging electricity portfolios and incorporate data analytics to calculate
between ‘peers’ uses the electricity network. The network and share this risk among all participants, thus creating
requires a continuously balance of the power in and a virtual balance responsible party. It might even
output. So, if the two ‘peers’ are not exactly matching incorporate instructions for controlled demand response
their power generation and use, others that use the and local generation providing the lexibility and control
network are affected. to offset the imbalance risks of both demand, for
example, to incorporate the variable supply of local
In most countries, it is obligatory to be ‘insured’ against renewable energy supply. Blockchain used in such a
this risk of not complying with your agreed demand and fashion has the potential to put the power system
supply contracts (thus creating an administrative completely on a new footing.
imbalance). Short imbalances are collectively taken care
of by the TSO, longer imbalances are taken care of by the
energy supplier (and corresponding balance responsible
party)48.

5.5 Summary for settlement
Settlement consists mainly of allocating meter data Data analytics will enhance this trend and will allow
and their corresponding energy exchange to the suppliers or aggregators to enter a business niche that is
responsible market parties and associated contracts and, focussing on energy of ‘higher value’ to their customers,
afterwards, the associated payments. While data analytics for example by operating demand response or (local)
can achieve beneits in optimizing these processes, it is in storage tweaked to the wishes of these customers, using
these contracts and agreements between electricity data from many sources, among which data from smart
traders, energy suppliers and end-users where the major meters.
developments will be.
So, the development of new business models, away from
The main application of data analytics will be driven by the economies of scale of traditional energy suppliers,
the changes at the edges of the settlement processes, focussing on ‘premium’ lexible energy and the
caused by the development of smart metering and the application of data analytics go hand in hand and it is
rise of new business models focusing on alternative value dificult to see which one enables the other. Applications
models rather than the economies of scale of traditional like blockchain, tangle and the internet of things likely
energy suppliers. There are already some energy will play a role in this, likely to make existing settlement
suppliers who have started to make use of smart meter processes much more eficient and secure, but there is
data to build their retail products on, like charging their a small chance that blockchain technology will disrupt
customers wholesale prices. the existing business models of energy suppliers by
automating their tasks and enabling end-users to trade
among themselves, skipping the middle man.

THE INTERNET OF THINGS AND TRANSACTIONAL ENERGY
Another major transformational trend is the

development of the Internet of Things (IoT).
It has many faces and applications, for example
optimizing supply chains and logistics50 and
media (sharing data between different devices
etc.). Energy is seen as an important application,
especially in unlocking large amounts of small
scale lexibility from devices such as heat pumps
and electric vehicles.
The Internet of Things is about devices communicating with each other. Of course, for the IoT to create
value, this communication has to have sensible content, that has meaning to the ‘receiver’. In other words,
the IoT devices need to have something in common to talk over. While this value can be derived from the
end user (e.g. streaming media, remotely controlled thermostats or an IoT coffee machine connected to the
alarm clock), it becomes interesting if this communication becomes more autonomous, such as optimizing
logistic chains in the industrial IoT, or indeed exchanging energy and optimizing the power system.
Energy use, and especially lexibility in energy use, is something all devices share, and provided it is a
scarce resource is an ‘interesting’ topic between devices to communicate. While the relative increase in
value of this energy IoT will diminish as more appliances offer lexibility, it still might offer a kick-start to a
much wider IoT.
The value of connectivity vs

the value of optimization
Internet of Things:
Internet of Things:
Value
value proportional
proportional toton·log(b)
n· log(n)
The value of connectivity vs. the value of optimization
network
thenetwork
Added
Addedvalue of of
value
smart
smart energy
energy to
to the
valueofofthe
the internet
Internet of things
of Things
Scaling up networks. As energy is a 'topic' all devices share,
its initial value is relatively high. However, as networks grow,
Totalvalue
its relative value will decline. Contrary to Metcalfe's law51,

Total
all devices offer the same thing: lexibility in energy use that
Energy optimization: can be used to optimize the power system.
Value proportunal
Energy opimization: to
nnxx valuelog(n)
proportional to log(n)
Number of connected devices n

Number of connected devices n

6 - CONCLUSION AND
SUMMARY
Data analytics has, and will continue to have, an enormous impact on the electricity system
and its stakeholders. To beneit from it, companies will strive to connect data sources, sources
within their company, external public sources and also sources from other companies, including
sharing data with competitors.
I n all the examples mentioned, data analytics is

bridging borders between types of data such as
Competing in a data and data analytics environment
therefore has a completely different logic compared to
structured and unstructured data; between departments competing over scarce resources, and seeing data as a
and functions within companies; between companies scarce resource to be secured is a recipe for failure.
and their customers; and even between competitors. Competing in such an environment is about using data
In this sense data (and data analytics) is more of a and data analytics to create a bond and relationship with
‘binding force’ that binds businesses (and other other stakeholders, and not only customers.
stakeholders) together into a much larger network than Synergy between data is the cement (or duct tape) that
it is a fuel that drives the business itself. ties everything together. Companies should
So, coming back to the initial question – ‘will data be evaluate their ambition in what role they want to take in
the oil of the 21st century?’ As a transformational force, this ‘structure’ bound together by data, and what value
the impact of data might be comparable to that of oil in they want to add to it, compared to companies with
the 20th century. However, it will do this by binding comparable ambitions; instead of merely looking at
stakeholders together instead of stakeholders competing linear value chains and their added value to their direct
over scare resources. customers.

Network operators, as well as energy suppliers, need to
make fundamental choices:
6.1 Applying data analytics in
Be an essential—but rather unglamorous ‘brick’ in the the electricity sector
foundation of the building, offering essential basic
platform services (which are likely to be regulated In the electricity sector the rise of data and data analytics
eventually if they are essential). Or, instead, going for a coincides with the transition towards a more renewable
glamorous, high risk, high reward strategy in ‘the top of energy system, though the speeds of both developments
the building’, offering ‘Gladstone Gander’ kind of differ radically. Data and data analytics will play a major
services using state of the art advanced artiicial part in how, and how fast—the energy transition will
intelligence; or will it be something completely develop. While many of the data analytic methods and
unprecedented and new? algorithms are built upon similar principles, their
application in different domains face different challenges
and hurdles and domain knowledge is invaluable in the
development and application of these algorithms.
Nevertheless, in all applications it will allow a much more
responsive behavior, by binding together all elements
from the electricity value chain.
Because of the dificulties to store electricity in large

quantities, processes in the electricity sector are—more
than in other sectors—divided into processes
anticipating and preparing for the future; operatingthe
system in (near) real time; and settling the difference
between the two with hindsight. Data and data analytics
do make the distinction between these processes less.

Data analytics is most widely applied in forecasting
on many different time scales, and developments in
6.2 Final reflection
forecasting are developing fast. Forecasting is widely
used in the electricity sector, and its application will only
increase and become more sophisticated. Important themes such as the availability of data, quality
Better forecasting will allow the operation of the power of data, privacy and cyber security should not be
system to become more effective and eficient. neglected. They pose important requirements that need
to be satisied for data analytics to reach its potential.
It will be operations that will be transformed the most However, as such, in this paper they are considered
by data analytics and its associated technologies. solvable, and thus outside the focus of this paper, which
Data analytics allows for the optimization activities, aims to give a sense of this potential and its impact on
traditionally conducted ‘ad hoc’ or by staff departments, the electricity sector. This is illustrated with several
to become part of the core processes itself. Thus, the examples.
core processes can be continuously ‘tweaked’ to the
changing environment, though, as will all specialization, This paper tries to show in which direction developments
the necessary infrastructure likely will reduce the are heading, though it remains unclear where it will end.
lexibility to react to ‘higher’ level strategic changes. In 2008 a very interesting observation was made in an
Nevertheless, as the electricity sector goes through a article called ‘The end of theory’ (10), stating a quite
transition, the lexibility through data analytics is a huge convincing argument that data and data analytics would
merit for all its stakeholders. It allows retailers and make scientiic theory obsolete: “Given enough data, the
aggregators to implement automatic demand response data will speak for itself”.
and generation that reacts to variable renewable
generation, as well as grid operators to ‘tune in’ to these Naturally this caused a lot of discussion, but also raised
developments and together with energy suppliers and new questions. The development of theory started with
aggregators, to include transmission and distribution in trying to explain observations, irst through myths and
optimizing electricity supply. legend, and the development of the ‘scientiic method’
through veriication and especially falsiication of
The boundaries between operations and settlement will observations and correlations, making theoretical
become smaller and smaller. Especially at the ‘edges’ hypotheses that pass more trustworthy. In principle, the
of the settlement processes, commercial stakeholders, same mechanism can be applied to data, ever expanding
such as energy suppliers and aggregators will make use the database giving more and more context and inding
of data analytics to diversify and escape the ‘commodity and reining more and more relations between
trap’ by developing new energy products, services and parameters.
business models, giving end users more choice.
Decentralized ledger technologies, such as blockchain Nevertheless, hypotheses also act as a guide to scientiic
and tangle, might facilitate this, mostly by adding cyber research. It gives something to verify or falsify by
security, though the main potential of these technologies experiments (i.e. by observations). So, a question that
lies in administrating with n x m transactions, where a emerges is what will be the new guide to the
‘central authority’ (like an energy supplier or grid development of knowledge? Or will it merely be a
operator) no longer is necessary. process of illing in the empty spots in the databases
and inding more and more complicated correlations,
using more and more advanced artiicial intelligence?

7 - REFERENCES/SOURCES
(1) Rus, Daniela. Study: Carpooling apps could reduce trafic 75%. www.csail.mit.edu. [Online] 3 January 2017
https://www.csail.mit.edu/ridesharing_reduces_trafic_300_percent
(2) Aalst, prof. dr. WMP van der. http://www.processmining.org/. Process Mining. [Online] 2014
http://www.processmining.org/
(3) DNV GL. 10 Technology Trends Creating a New Power Realy. www.dnvgl.com. [Online] 2016
https://www.dnvgl.com/energy/publications/download/technology-outlook-2025-energy.html
(4) USEF. www.USEF.info. http://www.usef.info/Downloads.aspx. [Online] 2016.
(5) The application of health and risk indices as a decision-support tool for utilities.
Vermeer, M.E., Schuddebeurs, J.D. and Wetzer, J.M. London : IET Digital Library, 2016
Asset Management Conference (AM 2016), 2016 page 31
(6) World business council for sustainable development. Corporate Renewable Power Purchase Agreements
www.wbcsd.org. [Online] 26 October 2016
http://www.wbcsd.org/Clusters/Climate-Energy/Resources/Corporate_Renewable_PPAs_Scaling_up_globally
(7) USEF. USEF: Workstream on aggregator implementation models
https://www.usef.energy. [Online] September 2017. https://www.usef.energy/news-events/publications/
(8) Kamphuis, R., et al. Real-time trade dispatch of a commercial VPP with residential customers in the
PowerMatchingCity SmartGrid living lab. [Online] 2013
IET Conference Publications. 2013. 1-4. 10.1049/cp.2013.0666
(9) Bliek, F., et al. PowerMatchingCity, a living lab smart grid demonstration. Innovative Smart Grid Technologies
Conference Europe (ISGT Europe), 2010 IEEE PES. 1 - 8. 10.1109/ISGTEUROPE.2010.5638863, 2010
(10) Anderson, Chris. The end of theory: the data deluge makes scientiic method obsolete. Wired. June 2008
(11) DNV GL. https://www.dnvgl.com/publications/smart-cable-guard-2-0-17603
www.dnvgl.com [Online] DNV GL, 2018
(12) Kadurek, Petr. Data Applications for Advanced Distribution Networks Operation (PhD thesis)
Enschede: Ipskamp drukkers, 2013
(13) DNV GL. https://www.dnvgl.com/publications/making-renewables-smarter-104362
https://www.dnvgl.com/publications/. [Online] 11 2017
(14) Sridhar, Narasi. 2017 Frank Newman Speller Award: Knowledge-Based Predictive Analytics
in Corrosion. 2017, Vol. 74, 2
(15) Qian, Y., et al. Risk on failure, based on PD measurements in actual MV PILC and XLPE power cables
Jicable'15. Versailles, France, June 2015

8 - END NOTES
1
This comes of course at the expense of the taxi drivers. Whether this is completely ethical is another issue.
Around Uber, but also around Airbnb there are many controversies. They are transforming their industries to such
an extent that many think that new regulation is required.
2
‘Big Data’ is usually characterized by Volume, Velocity, Variety and sometimes Veracity. In other words, the data is
too much, too fast, too divers and with a too low ‘integrity’ to be handled with ‘traditional’ ICT technology.
3
Note: A fundamental difference between these markets and electricity is that besides better service and more choice
Uber and Airbnb main breakthrough is a much more eficient matching between demand and supply that includes
new participants that did previously not participate. The electricity system is characterized by a per deinition (near)
perfect match of demand and supply, that includes all grid connected generation and demand. So, while the Uber
and Airbnb business models can and likely will be transposed to the energy sector, their impact will be different.
Data analytics does enable other business models that might have a similar impact on the electricity markets as
Uber did on the taxi market.
4
Management of the shared data might also be an issue, as this can lead to distrust and if one of the participating
partners would take this role. Most likely this would be done by an independent third party (e.g. a joint venture of
the participants, or an independent service provider).
5
For example, DNV GL has set up its Veracity data analytics platform for external clients to store, analyse and share
data among each other, even across industries and beneit from the data analytics capabilities it offers.
6
In 2008 Chris Anderson published an article in ‘Wired’ predicting the End of Theory. “With enough data, the numbers
speak for themselves.” For practical experience this would be even truer.
7
More information about the challenges the electricity sector is facing can be found in other DNV GL white and
position papers, among which the Energy Transition Outlook: https://eto.dnvgl.com.
8
DER (distributed energy resources) can be generation, like CHP units, solar and wind generation. It includes also
potential lexible loads, like charging of electric vehicles and demand response (both industrial as residential).
9
The main message is that policies are often changing every few years. Still successful examples are the feed in tariffs
boosting solar in Germany and the current tenders for off-shore wind projects in the North Sea, driving down costs.
On the other side there is the ETS scheme that still does not really take off.
10
Another dimension in the transformation of the energy supply is formed by the effects of the unbundling of
distribution and production/supply function of utilities and the liberalization of the latter since 2000.
This has contributed to an overinvestment and consequent crisis in the traditional fossil fuelled power generation.
11
Customers’ expectations as well as the perceived value of the services they received are inluenced by many factors,
like past experiences, information from competitors, reactions from friends and family, the media, etc.
The reversed mechanism: a dissatisied customer ‘running away’, because the service/product did not live up to their
expectation, is even stronger.
12
DSO stands for Distribution System Operator. In this document the term DSO is chosen over DNO
(Distribution Network Operator), because Distribution Network Management will require more and more active
control of assets and services, for example for capacity management, voltage control and other power quality issues.
13
For example the use of drones for automatic inspections of wind turbine blades, see (13).
14
See www.oreda.com
15
See (14). The graph is based on internal analysis by Narasi Sridhar.
16
A (simpliied) example could be: P(failure near sea shore) = P(failure because of salt) + P(failure because of
humidity) – P(failure salt - failure humidity). By factorizing these probabilities in elements and recombining the
elements, data from different failures can be reused to anticipate failures in other situations and possible other
kinds of assets.

17
For a wind/solar farm that is rewarded per MWh in a feed in or subsidy scheme, it is more important to maximize
the output using the available wind, than it is to know exactly what this output will be in the next four hours
(for investment decisions is important to know the average yield of a farm). A wind or solar farm that is (partly)
participating in the electricity market, where energy is traded beforehand, this becomes much more important.
18
For example ARIMA models (autoregressive integrated moving average) as well as neural networks applied to
aggregated data demand data to extract trends and periodic patterns.
19
This also applies to long term energy contracts. This is for example illustrated by the increase in intraday trading.
In parts of Western Europe TSO’s are inding it increasingly challenging to include all the changes in the BRP’s energy
programs to the transport prognosis that is caused by intraday trading.
20
State estimators basically estimate the state of a grid (i.e. the complex voltages of all nodes and the related currents)
by minimizing the errors between the results of load low calculations and actual (synchronized) measurements.
Both state estimation and load low simulations are widely used, but for distribution grids mostly ‘off line’ situations
once per e.g. month. And not continuously as input for active grid management.
21
For example, DNV GL developed ‘Kermit’ and ‘Pulse Chlorination’. ‘Kermit’ is a model for grid stability, based on the
simulation of inertia and frequency control. ‘Pulse Chlorination’ is a commercial product DNV GL able to minimize the
amount of chlorine necessary to avoid bio fouling in cooling water outlets of power plants and factories, based on the
monitoring and modelling of clams.
22
Reference: http://www.ingrid.i/en/grid_projects/ELVIS/Pages/default.aspx; and
https://www.ingrid.i/en/pages/news/news/2016/ingrids-elvis-project-completed
23
With the help of mathematical developments like stochastic collocation; splitting techniques (rare event simulations)
able to immensely speed up Monte Carlo kind of simulations; and dimensionality reduction of scenario’s; it becomes
much easier to quickly identify and assess potential critical situations.
24
For this the behaviour of smart devices need to be known, and especially how smart devices might react to one
another. It is viable that the model based control includes a model of these devices, merely that this model can identify
predeined ‘instability states’ from more detailed simulations using digital twins (certiied digital models) of the smart
devices and inverters that behave exactly like the physical original (within the simulated scope). More on the veriication
of these digital models and their physical counterpart, using PHIL (Power Hardware In the Loop) and CHIL (Control
Hardware In the Loop) methodologies can be found in the DNVGL position paper ‘Grid Cybernetics’.
25
Like ‘lexibility providers’ such as greenhouse CHP units; industrial and agricultural cooling houses as well as other
demand response and small lexible generation.
26
Consumers that also produce energy e.g. with solar panels.
27
For example, there supposedly is a correlation between switching from energy supplier and the installation of
solar panels 2 months earlier, so information of solar panel installation provides a perfect lead to energy retailers.
28
NEST is probably the most well-known.
29
Called Non-Intrusive Load Monitoring (NILM) or Non-Intrusive Appliance Load Monitoring (NIALM).
30
This depends of course on the demand. While in principle all demand is lexible for the right price, this applies
mostly to processes where the ‘objective’ can be decoupled from the power use. Examples are heating and cooling
processes (like air conditioners), where the heat or cold can be buffered, or charging electric vehicles, where the
charging and driving happen on different times.
31
See e.g. https://www.enernoc.com/ and http://www.energy.actility.com/
32
Assuming of course a basic level of quality and reliability of the power system, where power is considered a commodity
and not as a luxury.
33
This used to be called Program Time Unit or PTU. In most countries this is 15 minutes, is some this is 30 minutes
(e.g. the United Kingdom) or 1 hour (e.g. Spain).

34
NIALM: Non-Intrusive Appliance Load Monitoring: disaggregation of metering data into the energy use of individual
appliances that themselves are not monitored.
35
For example to replace an old refrigerator when it becomes economical beneicial to replace it.
36
E.g. Easy Energy: https://www.easyenergy.com.
37
While a traditional energy supplier’s business model focusses on economics of scale and quantity, An (independent)
aggregator can be seen as an energy ‘supplier’ that is trying to differentiate its energy based special (time dependent)
requirements: i.e. lexibility. This lexibility is created by aggregating and controlling lexible demand and generation.
38
Of course besides the energy itself there are other differentiation values. For example providing excellent customer
contacts; providing information and advice on energy use; give welcome or retention presents for new resp. existing
customers; create a corporate image itting the life style of customers; and many more all are important and are used
in creating ‘value’ to consumers, but all of these services and/or attributes are not directly related to the energy itself.
39
Because the consumer is taking the risk of the luctuating market prices, this does not necessarily mean that the energy
supplier will receive lower margins. Normally the supplier would have much better capabilities to manage this risk,
because of the large portfolio he has. However, if the consumer can adjust his demand to the luctuating prices
(Demand Response) it will result in a net gain.
40
The certiicates and the energy can be traded independently.
41
See also GPX (www.gpx.eu) and PowerMatching City II (www.powermatchingcity.nl)
42
The energy supplier becomes an aggregator and—until blockchain will take over—will perform the settlement and
responsibilities to the system (like balance responsibility).
43
See www.powermatchingcity.nl
44
With proof of effort, a successful miner will create a new block of entries including a hash, that miners only can solve
by trial and error in order to win. If the new solution in veriied, the winner will add a new block with entries to the
blockchain, etc.
45
A tangle—as developed and applied in Iota—is basically a network of hashed ‘transactions’ instead of blocks of
transactions in a single chain. This allows parallel veriication of transactions done by the parties doing the transactions
themselves, instead of being performed by separate ‘miners’. This allows for smaller transactions and overall less mining
costs (in Iota transaction can be done at a cost of verifying two other transactions). This might avoid the huge energy
consumption currently required by bitcoin miners to supply the ‘proof of effort’ (i.e. solve the hash puzzle).
See https://iota.org/ and https://iota.org/IOTA_Whitepaper.pdf
46
Among others creating economies of scale with huge volumes and centrally standardized processes.
47
E.g. PowerPeers, GPX, Innogy EV Charging, Grid+. Though most of these are based on a private blockchain, i.e.
blockchains where transactions (or the information) are stored in a blockchain for security reasons, but instead of
competing miners there are only a couple of central authorities (or just one) that validates all transactions and puts them
in the chain.
48
The TSO procures ancillary services like frequency containment reserve (FCR) to compensate balance disturbances
resulting in frequency deviations. If the imbalance is longer than the imbalance settlement period (ISP, in most countries
15 minutes), then one or more balance responsible parties (BRP, often an energy supplier) did not act according previous
negotiated contracts and this mismatch is produced/consumed by order of the TSO and the bill forwarded to the
responsible BRP. All demand and generation fed into, or drawn from the grid has to be accounted for by a BRP.
A contract with a BRP can be considered as an insurance against creating imbalance.
49
DSO stands for Distribution System Operator. In this document the term DSO is chosen over DNO, because Distribution
Network Management will require more and more active control of assets and services, for example for capacity
management, voltage control and other power quality issues.
50
The Internet of things allows to optimize value chains and logistics up to the devices of the end users (like the ‘fridge,
after noticing that it is about to run out of milk, will order new milk).
51
Metcalfe’s law states that the value of a network scales with the number of connections between nodes, this scales with
the number of nodes squared: n2, or, discounting for declining values of connections, scales to n*log(n).
See https://en.wikipedia.org/wiki/Metcalfe%27s_law

SAFER, SMARTER, GREENER
DNV GL Netherlands B.V.

Utrechtseweg 310-B50
6812 AR Arnhem
The Netherlands
Tel: +31 26 356 9111
Email: contact.energy@dnvgl.com
www.dnvgl.com
Author: Marcel Eijgelaar
DNV GL
DNV GL is a global quality assurance and risk management company. Driven by our purpose of safeguarding life, property
and the environment, we enable our customers to advance the safety and sustainability of their business. We provide
classiication, technical assurance, software and independent expert advisory services to the maritime, oil & gas, power and
renewables industries. We also provide certiication and supply chain services to customers across a wide range of industries.
Operating in more than 100 countries, our experts are dedicated to helping customers make the world safer, smarter and
greener.
In the power and renewables industry

DNV GL delivers world-renowned testing and advisory services to the energy value chain including renewables and
energy eficiency. Our expertise spans onshore and offshore wind power, solar, conventional generation, transmission
and distribution, smart grids, and sustainable energy use, as well as energy markets and regulations. Our experts support
customers around the globe in delivering a safe, reliable, eficient, and sustainable energy supply.
Group Technology & Research

The objective of strategic research is through new knowledge and services to enable long term innovation and business
growth in support of the overall strategy of DNV GL. Such research is carried out in selected areas that are believed to be
of particular signiicance for DNV GL in the future. A white paper from Group Technology & Research is intended to highlight
indings from our research programmes.
The trademarks DNV GL and the Horizon Graphic are the property of DNV GL AS. All rights reserved.
This publication or parts thereof may not be reproduced or transmitted in any form or by any means, including photocopying
or recording, without the prior written consent of DNV GL.
©DNV GL 10 /2018

Data Analytics

Загружено:

Сведения о документе

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

Data Analytics

Загружено:

Авторское право:

Доступные форматы

GROUP TECHNOLOGY & RESEARCH, WHITE PAPER 2018

DATA ANALYTICS IN THE

SAFER, SMARTER, GREENER

1 INTRODUCTION: WHAT IS IT ALL ABOUT? ____________________________________________ 4

2 THE POWER SYSTEM AND ITS CHALLENGES _________________________________________ 10

3 THE IMPACT OF DATA ANALYTICS ON FORECASTING AND PLANNING ____________________ 16

4 THE IMPACT OF DATA ANALYTICS ON OPERATIONS ____________________________________ 26

5 THE IMPACT OF DATA ANALYTICS ON SETTLEMENT ______________________________ 38

6 CONCLUSION AND SUMMARY _______________________________________________________ 48

8 END NOTES ___________________________________________________________________ 52

I n marketing and (online) sales, data analytics is

04 ENERGY Data analytics in the electricity sector

Data analytics in the electricity sector ENERGY 05

06 ENERGY Data analytics in the electricity sector

Data analytics in the electricity sector ENERGY 07

BIG DATA DATA ANALYTICS SECURING BENEFITS

Too big, too complex, Collecting, cleaning, processing, Insight, optimisation,

08 ENERGY Data analytics in the electricity sector

1.3 Reading guide

This paper is organized into 6 chapters. While this

Chapter 3 highlights some examples of the impact data

Chapter 6 eventually summarizes the impact of data

Data analytics in the electricity sector ENERGY 09

T o understand the transformational effect of data

10 ENERGY Data analytics in the electricity sector

Data analytics in the electricity sector ENERGY 11

12 ENERGY Data analytics in the electricity sector

Data analytics in the electricity sector ENERGY 13

While it is possible to make this ‘rough’ distinction

14 ENERGY Data analytics in the electricity sector

Data analytics in the electricity sector ENERGY 15

D ata analytics in the power system usually is

16 ENERGY Data analytics in the electricity sector

Data analytics in the electricity sector ENERGY 17

18 ENERGY Data analytics in the electricity sector

So, while there is no doubt that asset management

Another additional feature of the measurements is that the location of a fault

Data analytics in the electricity sector ENERGY 19

ASSETS PROBLEM SOLUTION RANKING

ASSET RANKING SOLUTION

20 ENERGY Data analytics in the electricity sector

Encapsulant Glass failure

Data analytics in the electricity sector ENERGY 21

Introduction and current situation New developments and opportunities

Data analytics in the electricity sector ENERGY 23

POWER SYSTEM BIG DATA AGGREGATION

Top system level Top system level

24 ENERGY Data analytics in the electricity sector

Data analytics in the electricity sector ENERGY 25

A s developments go on, data gathering becomes

26 ENERGY Data analytics in the electricity sector

Data analytics in the electricity sector ENERGY 27

ADAPTIVE ADAPTIVE ADAPTIVE

Subprocess or Subprocess or Subprocess or Subprocess or Subprocess or Subprocess or

28 ENERGY Data analytics in the electricity sector

So, both the development and deployment of sensorsin

Data analytics in the electricity sector ENERGY 29

These ‘meta- models’ sometimes will be a separate

In the next three sections examples of the possible

4.2 The impact of analytics on

For both utilities as grid operators, the ability to optimize

DSOs need to automate data analytics for optimizing

So, what kinds of operational decisions can be made?

30 ENERGY Data analytics in the electricity sector