Managing your

Assets with Big Data

Tools
Karthigai Muthu
Agenda

Big Data value proposition

Big Data Technology Stack
Hype Cycle for Emerging Technologies

Source: Wikipedia
Sources of
data
30 billion
12+ TBs RFID tags
of tweet data today 4.6
every day (1.3B in billion
2005) camera
phones
world
wide
data every day

100s of
? TBs of

millions of
GPS
enabled
devices
sold
25+ TBs of annually
log data
every day 2+
billion
people
on the
76 million smart
Web
meters in 2009
by end
200M by 2014
2011
 What makes Data Big
Characteristics Description Attributes Drivers

Volume The amount of data generated or
intensify that must be ingested,
analyzed and managed to make
decision based on complete data
analysis
Exabyte (EB) Increase in data sources
Zettabyte (ZB) Higher resolution sensors
Yottabyte (YB) Scalable infrastructure

Velocity How fast the data is being Batch Improved throughput connectivity
produced and changed and the Near real time Competitive advantage
speed at which is transformed into Real time and Streams Pre-computed information
insight Rapid feedback loop

Variety The degree of diversity of data Degree of structure M2M/IoT

from sources both inside and Complexity Social Media
outside an organization Genomics
Video and Mobile

Veracity The quality and provenance of data Consistency Cost

Completeness Need of traceability and justification
Ambiguity
Integrity
Big Datas Greatest Power: Predictive
Analytics
Whats driving Big Data

- Optimizations and predictive analytics

- Complex statistical analysis
- All types of data, and many sources
- Very large datasets
- More of a real-time

- Ad-hoc querying and reporting

- Data mining techniques
- Structured data, typical sources
- Small to mid-size datasets
The Evolution of
Business Intelligence

Interactive
Speed Business Big Data:
Scale
Intelligence & Real Time &
Single View
BI Reporting
OLAP & In-memory
Graph Databases
Data warehouse RDBMS Speed
QlikView, Tableau,HANA
Scale
Business Objects,
SAS, Informatica,
Cognos other SQL
Reporting Tools

Big Data:
1990s 2000s Batch Processing &
Distributed Data Store
Hadoop/Spark;
HBase/Cassandra/MongoDB

2010s
Solving business problem with big
data
Formulation of big data strategy

People; 31%
Tools; 33%

Data; 16% intent; 20%

Companies Market share in Big Data
Big Data Investments
Priority for big data across industry
Are you aware the risk of not implementing
Big Data in your company
Big data changed connected things to
Internet of Everything(IoE)
How the industry can leverage from big
data
Challenges in implementing the big
data
Returns of Investment(ROI)
How do companies get MORE from big
data
Merge
Optimize
Respond
Empower
Are you planning to launch your new
product.
Customer 360`

Social Banking
Media Finance

Our
Known
Gaming
History

Purchase
Entertain
Entertain

Customer
 Real-Time Analytics/Decision
Requirement

Product
Recommendations Friend Invitations
that are Relevant to join a
& Compelling Game or Activity
that expands
business

Influence
Behavior
Improving the
Marketing Customer
Effectiveness of a
Promotion while it Learning why Customers
is still in Play Preventing Fraud Switch to competitors
as it is Occurring and their offers; in
& preventing more time to Counter
proactively
IoT+Big Data = IoE(Internet-of-
Everything)
Role of Big Data in M2M/IoT
Big Data is a factor that will, to a large extent, determine the
future growth rate in the M2M industry
M2M will connect increasingly more nodes that will provide
data from endpoints.
Data will be more granular, more frequent, and more
accurate, with bigger data sets or even live data streams
Large volume of endpoint connections IPv4 addressing
scheme cant accommodate everything(sensors, smart
phones, smart factories, smart grids, smart vehicles,
controllers, meters ) that it requires IPv6
IoE= Convergence of IoT, Big Data Analytics ,Cloud
Computing and other technologies is collectively called as
Internet of Everything
Challenges of Big Data in M2M/IoT

Meeting the need for speed

Data understanding
Maintaining data quality
Displaying the meaningful result
IoT/M2M Applications..
Big Data Use Cases IoT/M2M
Personal IoT: the scope is a single person, such as a
smartphone equipped with GPS sensor or a fitness device that
measures the heart rate. This is one of the fastest growing,
consumer-oriented areas of IoT.
Group IoT: the scope is a fairly small group of people, such as
a family in a smart house, co-workers in a van or a group of
tourists. This is one of the most challenging areas and is still in
its early phase.
Community IoT: the scope is a large group of people,
potentially thousands and more; usually this is in a public
infrastructure context, such as smart cities or smart roads.
This is a young and potentially promising IoT area.
Industrial IoT: the scope can be within an organization
(smart factory) or between organizations (retailer supply
chain). This is arguably the most established and mature part
Big Data Use cases IoT/M2M
Agriculture - sensors can be deployed on farm machinery in order to provide
data about the equipment, soil temperature, moisture, etc.
Buildings/Smart Homes - Building sensors be used to help facility managers
become more proactive about ensuring that their buildings operate at peak
efficiency.
Communities Smart cities make use of parking space availability systems,
intelligent traffic monitoring systems, intelligent highways, weather-adaptive
street lighting, and more.
Healthcare Infant monitors, smart diapers, pills with ingestible sensors are
just some of the IOT-based devices.
Manufacturing factories with sensors can improve operations, product
quality, and decrease safety hazards.
Smartphones can control everything from door locks, thermostats, light
bulbs, vacuum cleaners, and more.
Utilities smart water meters can be used to reduce water leaks. Smart
electric grids can adjust rates depending on usage.
Wearables Smart watches, fitness trackers and health monitors may
become primary source for human-related data, and can also be used in
Benefits of Big Data Analytics in
M2M/IoT
1. Device Maintenance:
a. Time for next patch upgrade
b. Energy management
c. Inventory management and track replacement
2. Proactive Healthcare:
Capture and analyze real time data from medical monitors to
predict potential health problems before patients manifest
clinical signs of infection.
3. Monetize Machine Data:
a. Monitor performance, usage and capacity details to uncover
up-sell and cross-sell opportunities
b. Maximize the lifespan and performance of high value
medical assets
Benefits of Big Data Analytics cont..

4. Optimize Support Operations:

a. Reduce MTTR and support escalations
b. Preempt failures with proactive support
c. Troubleshoot with accurate information
d. Proactive consultation to customers on
approaching expiry dates
Big Data Analytics Stack
Lamda Architecture
Batch vs. Real-Time processing

Batch processing
- Gathering of data and processing as a group at one time.
- Jobs run to completion
- Data might be out of date

Real-time processing
- Processing of data that takes place as the information is
being entered.
- Run for ever
Storm

Apache Storm is a free

and open source distributed
real-time computation system.

Storm makes it easy to reliably

process unbounded streams of
data, doing for real-time
processing what Hadoop did for
batch processing
Storm Is

Stream Processing
Fast
Scalable
Fault Tolerant
Reliable
Tuple
Streams
Spouts
Bolts
Topologies
Reliable Processing
Reliable Processing
Stream Grouping
Groupings are used to decide to which task in the
subscribing bolt (group) a tuple is sent.

Possible Groupings:
- Shuffle
- Fields
- All
- Global
- None
- Direct
- Local or Shuffle
Storm Cluster View
Fault Tolerance
Fault Tolerance
Fault Tolerance
Fault Tolerance
Fault Tolerance
Parallelism
Parallelism
Apache Storm Real-time -Use cases
Segment Prevent Use Cases Optimize Use Cases

Financial Services Securities fraud Order routing

Operational risks & compliance Pricing
violations
Telecom Security breaches Bandwidth allocation
Network outages Customer service

Retails Shrinkage Offers

Stock outs Pricing

Manufacturing Preventative maintenance Supply chain optimization

Quality assurance Reduced plant downtime

Transportation Driver monitoring Routes

Predictive maintenance Pricing

Web Application failures Personalized content

Operational Issues
The End