A DESCRIPTIVE STUDY OF BIG DATA ANALYSIS IN
SMART GRID INFRASTRUCTURE
Prof. Mahesh K1
maheshkgowda@gmail.com
Department of Electrical and Electronics
New Horizon College of Engineering
ABSTRACT:
Smart grid is a complete automation system, where large
pool of sensors is deployed in conventional power grids for
controlling and monitoring it on remote basis. It enables a
two-way flow of power and data between suppliers and
consumers in order to facilitate the power flow
optimization in terms of economic efficiency, reliability
and sustainability. Smart Grids (SGs) are emerging as a
promising technology meant to cope with the energy
efficiency issue, currently witnessed in legacy electrical
grids, by disseminating relevant information in a real-time
mode among the different SG components.The data
collected from these sensors is complex and have all the
characteristics to be called as Big Data.There is a
requirement for real time processing of data in Smart grid.
robust data analytics, high performance computing,
efficient data network management, and cloud computing
techniques are critical towards the optimized operation of
Smart Grids Hence, this paper will give a platform to
discuss about suitable methods for storing, performing Big
Data analytics pertaining to Smart grid.
Keywords:Big Data, Smart Grid, Energy, Infrastructure.
1. INTRODUCTION
Smart grid is a major research and development direction in
today's energy industry. Smart grid delivers electricity
between suppliers and consumers so as to form a
bidirectional electricity and information flow
infrastructure[1][2]. It fulfills the demands of each
stakeholder, functionality coordination in electric power
generation, terminal electricity consuming and power
market. It also improves the efficiency in each part of the
system operation and reduces the cost and environmental
impact. With the growing trend in information and
communication technologies, the conventional power grid
is evolving into a sensor-embedded network (Smart Grid),
which has the ability to control and automate the entire
processes. Sensors of various types are deployed across the
length and breadth of the smart grid. All these sensors
produce heterogeneous type of data, which are then
collected at the utility data-centers. The amount of data
collected from all the different connected components and
sensors in a very short interval of time is huge. Therefore, it
is out of scope for conventional storing techniques and
computing facilities to handle such a big data available.
The variability, variety and velocity are the characteristics
Darshan Kumar N
Drashanyad19@gmail.com
Department of Electrical and Electronics
New Horizon College of Engineering
of Big Data. It can be analyzed to reveal the knowledge of
unseen patterns which are hidden in large data sets and
utilized for making strategic, tactical and operational level
decisions.
In smart grids, there are various sources that huge amount
of data can be generated through diverse measurements
acquired by Intelligent Electronic Devices (IEDs) in the
smart grid:
Data from power utilization habits of users;
 Data from Phasor Measurement Units (PMUs)
for situation awareness;
 Data from energy consumption measured by the
widespread smart meters;
 Data from energy market pricing and bidding
collected by Automated Revenue Metering
(ARM) system;
 Data from management, control and maintenance
of device and equipment in the electric power
generation, transmission and distribution in the
grid;
 Data from operating utilities, like financial data
and large data sets which are not directly
obtained through the network measurement.
1. BACKGROUND
1.1 Smart grid
Smart grid is a form of electric power network. It is a
combination of all the power utilities resources, both of
renewable and non-renewable. As a significant innovation,
smart grid provides automatic monitoring, protecting and
optimizing for the operation of the interconnected systems.
The smart grid is bidirectional system Characterized by the
bidirectional connection of electricity and information sets,
the smart grid has formed an automated, widely distributed
delivery power network. Taking advantages from modern
communications, it can not only deliver real-time data and
information but also implement the instantaneous
management balance of demand and supply.
Operations on energy storage demand response and
communication between terminal users and power
companies will make a real large scale data-communicated
grid system.This adds complexity in the conventional grid
and promote offering sustainable progresses to utilities and
customers. In order to match its complexity, many
technologies are exploited in the smart grid domain.
Especially, big data is attracting more and more attention in
the power grid domain because it can help extract valuable
information from data sources of wide diversity and high
volume.
1.2 Smart Grid Data Flow
Figure.1: Types of data in SG
 Power generated oriented data
In power plants, Figure 1.Electricity is generated from
different sources like water, coal, tides, wind, nuclear etc
[3] . A proposed approach was wireless sensors and actor
networks which can be employed seamlessly over wind
farms to extract the data about the dynamic state behavior
of the wind turbines. A coal-based power plant fault
analysis anddiagnosis system using Association rule
mining. Foreseeing the predicted load paves theway for
power plants, to plan for their future needs. This helps for
utility industries to save millions of dollars a year.
 Power transmission distribution data
In General, distributed control system (DCS) and
supervisory control and data acquisition (SCADA) are
propriety control and monitoring systems used in power
transmission and distribution. A list of monitoring sub
systems are provided, whichcan be used for monitoring the
grid activities.
 Power Consumption data
The distributed electricity will be consumed by consumers
from various zones like Residential (Individual housesand
Apartments), Commercial (e.g., Insurance), Industrial (e.g.,
Factories), Transportation (Railways), Emergencyservices
(e.g., hospitals) and governmental services (e.g., school),
etc. Smart meters are equipped at customer endpoints,
which sense and broadcast utilization data to the service
providers at regular interval of period. These data are of
two types, either in disaggregated data (break up data for
every single component or group of numerouscomponents
in the single oriented electrical circuits) or in aggregated
data (collective data of all appliances).Customer profiling is
helpful in their behavior prediction and in providing
dynamic pricing that meets their requirements.
2. ENERGY BIG DATA:
ARCHITECTURE
2.1 Three-tier energy big data analytics architecture
A widely accepted analytical framework of big data is
shown in Figure. 2, which has three layers including data
access and computing, data privacy, domain knowledge
and big data mining algorithm.
The inner core data mining platform is mainly responsible
for data access and computation process. With the
increasing growth of the data volume, distributed storage of
large scale data need to be taken into account while
computing. That is to say, data analysis and task processing
are divided into multiple sub tasks and executed on a large
number of computing nodes through a parallel program.
The middle layer of the structure plays an important role to
connect inner layer and outer one. The data mining
technology in the inner layer provides a platform for data-
related work in middle layer such as the information
sharing, privacy protection, and knowledge acquisition
from areas and applications, etc. In the whole process, the
information sharing is not only the guarantee of each phase,
but also the goal of processing and analyzing with the big
data in smart grid.
In the outer layer of the architecture, preprocessing is
necessary for the heterogeneous, uncertain, incomplete, and
multi-source data through data fusion technology. After
preprocessing, complex and dynamic data will be
excavated, and then, pervasive smart grid global knowledge
can be obtained through local learning and model fusion.
Finally, model and its parameters need adjustment
according to the feedback.
Figure.2:Three-tier architecture
2.2 Random matrix theory
Random matrix theory (RMT) has emerged as a
particularly useful framework for many theoretical
questions, especially for those concerning multivariate data.
[4] There are two frameworks for RMT: one assumes the
asymptotic convergence, and the dimensions are infinite;
the other one is the non-asymptotic solution, assuming
finite matrix size. For the asymptotic results, in theory, we
require the infinite size of the matrix, which is infeasible in
practical world. However, the results are remarkably
accurate, even for relatively moderate matrix sizes such as
tens. This is the very reason why this random matrix model
is penetrating so many areas from financial engineering to
wireless network. Our initial motivation for this model was
from large-scale wireless network. The new trends for
RMT are (1) finite matrix and (2) non-Gaussian matrix
entries.

2.3 An integrated big data architecture in the smart

grid

Based on big data analytics and cloud technology, an

integrated architecture can be used in the smart grid in
many ways, such as optimizing power transmission,
controlling power consumption, keeping the balance
between power demand and supply, etc. This architecture
takes advantages of the three technologies including big
data analytics, smart grid and cloud computing, which
composes an enhanced version of smart grid.The improved
smart grid architecture is illustrated in Figure 3, which
consists of the smart grid, the HDFS and the related cloud
environment. In order to manage the storage and retrieval
Figure.3:The integrated architecture
of massive data, HDFS is used in this system. HDFS has
two types of nodes: 1. Namenode (the master), which
manages the file system by keeping relevant files metadata
and namespace entries, and 2. Datanodes (workers), which 3. ENERGY BIG DATA: KEY
are the real workhorses of the system. They store and TECHNOLOGIES
retrieve files’ blocks upon request from the master node
(i.e., the namenode). The communications are done via the
3.1 Types of data acquired
RPC (Remote Procedure Call) protocol and itconcentrates
(i) Operational data—Represents the electrical behavior of
on distributed storage to nodes in racks. This architecture
the grid. [5] It includes data such as voltage and current
also contains a database including consumer behavior
phasors, real and reactive power flows, demand response
pattern, historic data, details in power supply and demand.
capacity, distributed energy capacity and power flows, and
Each time the system estimates the demand and calculates
forecasts for any of these data items.
the supply, it will refer to the consumer behavior patterns
(ii) Non-operational data—Represents the condition, health
and historic data. These data are stored in a cloud-based
and behavior of assets. It includes master data, data on
Cassandra Data Base (Cassandra DB). This improved smart
power quality and reliability, asset stressors, utilization, and
grid utilizes a prediction algorithm to estimate the demand
telemetry from instruments not directly associated with grid
and supply of electric power. In a distributed environment,
power delivery.
the smart grid uses distributed power resources like solar,
wind, nuclear sources, and is applied in many areas, such as
(iii) Meter usage data—Includes data on total power usage
industrial production and social infrastructure.
and demand values such as average, peak and time of day.
It does not include data items such as voltages, power
flows, power factor or power quality data, which are
sourced at meters but fall into other data classes.

(iv) Event message data—Consists of asynchronous event

messages from smart grid devices. It includes meter voltage
loss/restoration messages, fault detection event messages
and event outputs from various technical analytics. As this
data is triggered by events, it tends to come in big bursts.

(v) Metadata—Is the overarching data needed to organize

and interpret all the other data classes. It includes data on
grid connectivity, network addresses, point lists, calibration
constants, normalizing factors, element naming and
network parameters and protocols. Given this scope,
managing metadata for a smart grid is a highly challenging
task.
While the first three of these classes are relatively familiar
to utilities, the last two have been less prominent to date—
and are likely to present more problems as utilities adapt to
the smart grid world.
3.2 Data Acquisition and Storing
Data acquisition and storing is the initial problem in big
data. The big data technologies in acquisition and storing
stage gather data from various information in energy
system. The collected data are of different sources,
different formats and different features that is stored in data
repositories. The acquisition and storing technology of big
data belongs to data management which involves data
fusion, data integration, data management, and data
transforming which is usually called Extract Transform
Load (ETL) technology Figure 4.
Figure 4: Shows the data acquisition and storing flow using
ETL technology.
Data acquisition involves data access and collection. Since
data have private information, its confidentiality and
security should be considered during accessing and
transmitting.
4. ENERGY BIG DATA:
APPLICATIONS
4.1 Smart meters
Smart meter system employs several control devices,
various sensors to identify parameters and devices to
transfer the data and command signals [6][7] .When we
adapt to the smart grid system, we’re moving from one
meter reading per month to smart meter readings every 30
minutes transforms into 48 million readings for every
million consumers, resulting in a massive volume of data to
manage. The data is collected in frequent time periods and
if technology is available for near real time analysis, many
advantages could be achieved. The analytics technologies
will have to deal with not only consumption data, but
information about consumers, weather as well as many grid
behavior based readings. To manage and use this
information to gain insight, utility companies must be
capable of managing high-volume data and using advanced
analytics to transform data into actionable insights.Utilities
that build up this capability can gain insight into their
operations and assets, enabling them to take proactive
action rather than simply reacting to events after they
happen. As highlighted in the results may include increased
profitability, a reduced carbon footprint, increased safety,
enhanced regulatory interaction and improved customer
satisfaction. Although the increase in volume, especially in
consumption data capture is the most highlighted big data
aspect with smart meters the other key aspects of velocity
and variance are equally important. This is more relevant
for data generated by sensors and new grid instrumentation.
For example in real-time tasks such as equipment reliability
monitoring, outage prevention or security monitoring,
overnight is not good enough. Although several research
level techniques have been reported on analyzing streaming
data, much work still needs to be done in making these
commercially viable. Variety signifies the increasing array
of data types, which are collected not only from traditional
sources like industrial control systems but also from
security cameras; weather forecasting systems, maps,
drawings and pictures; and the web. The variety of data is
likely to become increasingly important to utilities as they
begin to analyze social media and call center dialogues and
to integrate such information in to smart meter and grid
generated data as part of their decision-making and
planning processes.
Technologies for smart metering
Majority of the metering intelligence related analytics work
reported uses time varying power consumption data to
generate consumption (or load) patterns showing the
consumers usage behavior. Clustering usage patterns makes
it possible to identify typical behaviors called typical load
profiles (TLP). TLPs could be used for load forecasting,
load estimation, load control, abnormal electricity
consumption detection, designing electricity tariff offers,
developing market strategies or demand side response
policy. As in figure 5, clustering based load profiles
provide the base for several other key analytics applications
which are described below.
Figure.5:Typical load profile
 4.2 Dynamic energy management
Dynamic energy management requires power flow
optimization, system monitoring, real-time operation, and
production planning [8] [9]. There are six levers of
effective DSM: rates, incentives, access to information,
technology and control, education and marketing, and
customer insight and verification. Dynamic energy
management in a Smart Grid is a complicated,
multivariable procedure, since the latter enables an
interconnected power distribution network by allowing a
two-way flow of both power and data. Thanks to the bi-
directional flow of information and power between
suppliers and consumers, the grids become more adaptive
to the increased penetration of distributed energy resources,
encouraging also users' participation in energy savings and
cooperation through the demand resource mechanism.
Demand response can be applied to both residential (e.g.,
cooling, heating, electric vehicles (EVs) charging, etc.) and
industrial loads and includes three different concepts; i)
energy consumption reduction, ii) energy consumption
shifting to periods of low demand, andiii) Efficient
utilization of storage systems. It should be noticed here that
plug-in EVs can be considered as storage devices, while the
careful scheduling of their charging and discharging can
benefit both their owners and the utilities [10]. With the
help of advanced energy storage will be able to increase the
reliability and security of power systems. In the short term,
fast-response storage has already been demonstrated as a
viable means of managing grid imbalances and volatility
through the regulation service. In the longer term, storage
can be deployed to shift energy in time to smooth the
output of renewable generation or reduce the peak load on
constrained transmission assets.Obviously, this further
increases the parameters that the dynamic energy
management algorithms have to take into account, such as
the EVs charging profiles. Consequently, the associated
complexity is also increased, creating at the same time
storage capacity prediction problems. Thus, a crucial issue
in Smart Grids is how to manage demand response in order
to reduce peak electricity load, utilizing at the same time
renewable energies and storage systems more efficiently.
Finally, effectiveness of demand response algorithms
depends critically on demand, price, load, and renewable
energy forecasting, which highlights the need for
sophisticated signal processing techniques.
In order to deal with the high level of uncertainties in
dynamic energy management, the extreme size of data, and
the need for real-time learning/decision making, the smart
grid demands advanced data analytic techniques, big data
management, and powerful monitoring techniques.
4.3 Renewable energy
The new and renewable energy are integrated in the electric
power generation, which is different from the traditional
power generation mode. This difference causes the
measurement and management of generated data from
electric power become increasingly difficult and complex.
Since the big data technology helps to make better
prediction, management and processing complex big data
in the energy domain, it has become increasingly popular
and widely applied in renewable energy companies.All
researches in carried out in this area have a common
purpose: cost minimization. Kung and Wang proposed a
recommender system for the best combination of renewable
energy resources with cost-benefit analysis, which include
analytical module, cloud data base, and user interface. This
study used Markov Chain to investigate the influences of
decision-making related torenewable energy and electricity
demand in random time. Since the historical electricity data
is recorded in continuous time series, Continuous Markov
Chain can be applied to analyze energy big data in order to
help power enterprises make optimal investment of
renewable energy and evaluate optimal energy
configuration.
5. CONCLUSION
In this paper, we gave an introduction on big data and
power grid, and reviewed the related work, techniques of
big data analysis. Then, we discussed the important role of
big data, which brought efficiency and accuracy to the
power system. Furthermore, an integrated architecture was
introduced for the big data analysis, with the help of these
architectures, we can perform predictive analysis, prevent
outages, improve customer services and ensure service
reliability. By the integration of various distributed
generating sources, refined forecasting, load planning, and
unit commitment it would be possible to avoid inefficient
energy transmitting or dispatching extra generation which
in turn would help operate the smart grid in an efficient and
reliable way. The excessive renewable energy which is
generated can transmitted to the demand location with the
help of these architectures. Various applications of energy
big data in field of smart meters, dynamic energy
management, and electric vehicleswereillustrated.
6. REFERENCES:
1. DrissBenhaddou, Mohamed RiduanAbid,
OuidadAchahbar, Nacer Khalil, TajjeeddineRachidi
and Maen Al Assaf, “BIG DATA PROCESSING
FOR SMART GRIDS”- IADIS International Journal
on Computer Science and Information Systems Vol.
10, No. 1, pp. 32-46 ISSN: 1646-3692
2. Huijiang, kunwang, yihuiwang, min gao and
yanzhang, “ENERGY BIG DATA: A SURVEY”-
digital object identifier 10.1109/access.
2016.2580581
3. Shyam R, Bharathi Ganesh HB, Sachin Kumar S,
PrabaharanPoornachandranb, Somkan K P “Apache
Spark a Big Data Analytics Platform for Smart
Grid”- SMART GRID Technologies, August 6-8,
2015
4. Xing He, Qian Ai, Member, IEEE, Robert C. Qiu,
Fellow, IEEE, Wentao Huang, LongjianPiao,
Haichun Liu, “A Big Data Architecture Design for
Smart GridsBased on Random Matrix Theory”.
5. “Achieving high performance in smart grid data
management” by Accenture,2012.
6. Soma ShekaraSreenadh Reddy Depuru, Lingfeng
Wang, Vijay Devabhaktuni and Nikhil Gudi, ” Smart
 Meters for Power Grid – Challenges, Issues,
Advantages and Status”
7. DammindaAlahakoon, xinghuo Yu, “advanced
analytics for harnessing the power of smart meter big
data” conference paper:
Nov2013,DOI:10.1109/IWIES.2013.6698559
8. Panagiotis D. Diamantoulakis, Vasileios M.
Kapinas, George K. Karagiannidis, “Big Data
Analytics for Dynamic Energy Management in Smart
Grids”
9. Brandon davito, humayun tai, and Robert uhlaner,
“The smart grid and the promise of demand-side
management”
10. MahnooshAlizadeh, Xiao Li,
ZhifangWang,AnnaScaglione, and RonaldMelton,
“demand-side management insmart grids” –IEEE
signal processing magazine September 2012