Big Data Review 2015

A detailed investigation into the maturing of Big Data analytics

June 2015 3,995

Sponsored by
 Contents
Executive summary p3
Research overview p4
Big Data goes mainstream p5

What do we want from Big Data? p8

The democratisation of data analytics p 11
Democratisation and best practice p 14
Turning data into knowledge p 17
Factors for success p 20
The Internet of Things p 24
Conclusions p 27
About Computing p 28

This document is property of Incisive Media Services Ltd. Reproduction and distribution of this
publication in any form without prior written permission is forbidden.

2 Big Data Review 2015

1 Executive Summary 2015 was the
2015 was the year that Big Data went from being something that a few bigger
organisations were doing to being something that a majority of organisations year that Big
were either doing or at the very least actively considering. The maturing Data went
of cloud-based Big Data services has made Big Data analytics a feasible
reality for organisations of all sizes. The benefits of Big Data are more widely from being
understood driven by the increasing numbers of case studies available.
Vendors of Big Data software and services have become better at explaining something that
exactly how their solutions benefit businesses. a few bigger
The Computing Big Data Review 2015 summarises the results of a organisations
comprehensive research programme undertaken by Computing during the
first quarter of 2015. The findings are compared with those published in the were doing
Computing Big Data Review 2014 in order to establish the degree to which to being
the market has developed in this short space of time and the degree to which
Big Data and the democratisation of data analytics is changing business something that
organisations. The review contains some unique insights from high-ranking
IT decision makers into how organisations are approaching and deploying a majority of
Big Data solutions and some of the specific challenges they face. organisations
Key highlights from the research include: were either
Live Big Data deployments are increasing fast in number as are the doing or at the
actionable insights that they generate. Next-generation data warehouses/
analytics databases and cloud-based Big Data services have shown very least
particularly strong growth in the last 12 months. actively
The maturing of real-time analytics is encouraging greater numbers of considering
organisations to look at Big Data.
The focus for businesses is analysing internal data for the purposes of
finding efficiencies rather than the speculative analysis of external data to
potentially open new markets.
The pace of democratisation of data analytics is increasing with huge
implications for businesses and traditional hierarchies.
Evangelists for data-driven decision-making are encountering some
significant obstacles as they continue to democratise data analytics
technical, legal and cultural hurdles all must be overcome.
Even bigger obstacles are encountered in the drive to turn raw data into
actual knowledge and actionable insight knowing the right questions
to ask, determining what the useful data was in among the noise and
reforming corporate culture were some of the biggest issues.
Business buy-in remains the number one priority for those initiating
and driving through Big Data projects. CIOs are the logical owners of
Big Data but projects have to be underpinned by a vision of the possible,
knowledge of the data available and a willingness and ability to push
through the cultural changes required.

Big Data Review 2015 3

 An average measure of data quality of approximately 85 per cent was
deemed sufficient for most to make an important business decision.
Quality and provenance of data is key in determining the confidence to
make decisions but the fact that 100 per cent accuracy is unlikely to be
feasible allows organisations to bridge the cultural gap between decisions
made on gut instinct and those driven purely by data.
The Internet of Things (IoT) is already affecting nearly one third of our
research participants with an even larger proportion expecting it to do so
soon. While the predicted explosion in data is already starting to happen,
growth is being impeded by real concerns about privacy and the ability of
security technology to keep up.

2 Research overview
The key objective of the Computing Big Data Review 2015 was to compare
research with that undertaken 12 months ago and published in the
Computing Big Data Review 2014, in order to establish the degree to which
the Big Data market has matured along with the causes and consequences.

Key areas of research included:

Big Data technologies actually in use or being actively considered
Changes in the last 12 months in the Big Data market that are impacting
businesses
What organisations want from Big Data
Realities, possibilities and implication of the democratisation of data
analytics
The challenges of turning data into knowledge and actionable insight
Factors for likely success in Big Data projects
The Internet of Things and Big Data
2.1 Methodology
The research project was conducted in five phases, using a combination of
qualitative and quantitative methods.

Phase 1 Telephone interviews with IT decision-makers who participated

in the 2014 Big Data research in order to establish key changes that have
occurred in the last 12 months. These participants were from sectors
including retail, technology, utilities, financial services, education and
government.

Phase 2 Four face-to-face interviews with IT decision-makers who had

adopted Big Data solutions enabling researchers to uncover the key benefits
being realised from Big Data implementations and challenges along the way.

Phase 3 A nationwide, online quantitative study completed by 390

IT decision-makers representing organisations ranging in size from 100
employees to many thousands (Fig 1).

4 Big Data Review 2015

Phase 4 A round of face-to-face interviews were conducted with
companies at various stages of implementing Big Data solutions, in order to
verify the quantitative results and gain further insights.

Phase 5 A final round of telephone interviews were conducted with other

IT decision-makers to provide further feedback and validation. Slides setting
out findings from earlier stages of research were shared at this point.

FIG. 1 Quantitative survey, participants by sector

Banking/Financial/Insurance
8%
14% Charity/Third sector
Media, Print & Publishing
12%
5% Healthcare
Business & Professional Services
4%
Manufacturing
10% 3% Distribution & Transportation
Retail & e-tail
9% Leisure & entertainment
Telecoms/Technology
14%
Public Sector/Government
12%
Education
5% 3%
1% Other

3 Big Data goes mainstream This year, 18

The market for Big Data analytics has continued to mature in the 12 months
since the publication of the Computing Big Data Review 2014. In the two per cent of
years since Computing began its research into this area, perceptions of Big respondents
Data have undergone a transformation.
have
In 2013, it was viewed as an impossibly large undertaking that less than 10
per cent of our research participants had even begun to look into. About one implemented
quarter of respondents at the time believed that Big Data would come and go Big Data
because the tools were simply too complex for most organisations.
solutions at
However, by 2014 there were more case studies available of real-world
implementations. This, the increasing maturity of Infrastructure as a Service the operational
(IaaS) and the growing number of open source solutions available had led 11 level
per cent of respondents to implement Big Data techniques and solutions on
an operational level with another third of participants either looking at or
actively evaluating possible solutions.

This year, 18 per cent of respondents have already implemented Big Data
solutions at the operational level. When asked what technologies they
were using or considering and presented with a list of likely options, the
proportion opting for none of the above plummeted from 33 per cent in
2014 to 16 per cent this year.

Big Data Review 2015 5

 A full comparison of Computings 2015 findings compared with last years
can be seen in Figure 2. All categories have seen a significant increase,
particularly next-generation data warehouses/analytics databases and
cloud-based Big Data services.

FIG. 2 Which Big Data technologies are you considering

using/have skills in-house?
2015
Next-generation data warehouses/ 41%
analytics databases 22%
Cloud-based Big Data services 41%
17%
Data integration and data quality 35%
platforms and tools 37%
Application development platforms 32%
applied to Big Data 22%
Analytic and transactional 32%
applications and services, as applied 24%
to Big Data use cases
NoSQL databases
32%
24%
Advanced analytics and data science 32%
platforms as applied to Big Data 23%
Hadoop software 29%
15%
In-memory databases as applied to 29%
Big Data 17%
Non-Hadoop Big Data platforms
24%
13%
None of these 16%
0% 10% 20% 30% 40% 50%
Considering or using Have skills in-house

2014
Next-generation data warehouses/ 24%
analytics databases 13%
Cloud-based Big Data services 26%
13%
Data integration and data quality 24%
platforms and tools 22%
Application development platforms 17%
applied to Big Data 11%
Analytic and transactional 15%
applications and services, as applied 13%
to Big Data use cases
NoSQL databases
25%
15%
Advanced analytics and data science 15%
platforms as applied to Big Data 13%
Hadoop software 21%
8%
In-memory databases as applied to 23%
Big Data 11%
Non-Hadoop Big Data platforms
12%
6%
None of these 33%
0% 10% 20% 30% 40% 50%
Considering or using Have skills in-house

6 Big Data Review 2015

Our research uncovered various reasons for this increased takeup. The first
is that Big Data technologies are no longer perceived as being something that Companies

only large organisations can implement. The cloud has ensured that even the
smallest of businesses can utilise Big Data. have become
much more
Cloud computing You dont have to have the same
potentially makes Big number of staff around tooling attuned to the
Data refining and analysis so businesses can take more advantage
much easier even for of services. The service-based nature problem they
modest and small firms of everything is totally changing our are trying to
IT manager, Research consumption of IT CIO, Services
solve and how
There have been other factors too. As increasing numbers of case studies that data might
become available the understanding of the benefits of Big Data has increased.
This process of education has been pushed along by the vendors themselves help them
who have become much better at explaining benefits for businesses. solve it
Vendors have also addressed issues of usability, smoothing off some of the
rough edges and offering training to users as part of enterprise subscription
packages. However, much of the increased awareness and takeup of of

Big Data has been driven by factors unique to individual industries, as the
following words from our interviewees demonstrate.

Were now bringing Theres a big push now

consumer data into our within the NHS on
own Big Data, so we get a more integrated patient records, with
rounded picture of individuals information from GPs, from

Head of research information & intelligence, acutes, from social care

Government department CIO, Local authority

Companies have become much more attuned to the problem they

are trying to solve and how that data might help them solve it
Software director, Utilities

Computing asked: What changes have you noticed over the last 12 months
with respect to Big Data that are most likely to impact on your business or on
the market in general?
The most frequently chosen answer was Moving from analysing historical
data to real-time analytics. The commercial benefits of real-time data
analytics for businesses include better understanding of their customers and a

chance to proactively improve services. Two of our interviewees shared how
they were using this technology.

Real-time analytics are very compelling. Its ensuring we have

Its very compelling for a team who have real-time controls in
developed a product as it shows the immediate place in terms of avoiding any
impact of the work theyve been doing. It is used fraudulent activities. Were
very heavily at time of launch of products and looking at high-volume, high-risk
features, and at very time-sensitive occasions like transactions. In retail banking
the release of TV commercials, so you can see the you have to mitigate risk in
immediate impact. Then you can make decisions real-time. When a transaction
sooner, without having to wait for those rolling is made we should know if its
cycles of analytics to come in. You can make a fraudulent or not Director data
decision almost immediately CIO, Online services infrastructure global technology, Banking

Big Data Review 2015 7

 Computing also wanted to establish whether the departments and individuals
initiating and owning Big Data projects were still the same as the ones
doing so a year ago. There had been no change. IT departments were still
initiating more Big Data projects than any other department. This is not
surprising ownership of data and the knowledge of where it lies and how
to process it still predominantly resides within IT teams, so they are a natural
starting point for Big Data investigation. However, many of our respondents

reiterated the message that a multi-departmental approach was necessary as
projects were initiated.

We had the The drive has come up from the

idea, we put it basement. One way we have made this
into the IT strategy happen is by bringing tools in and empowering
but the business has the business analysts to be able to understand
grabbed it and run their area of the business, whereas the
with it. So theyre the traditional route would have been I think I
ones engaging the need some reports, Ive got to talk to IT. Its
rest of the business the analysts who have created this information
CIO, Local authority culture Head of enterprise analytics, Large retail

The final two participants highlight the fact that while IT teams may be
driving Big Data projects, the long-term goal of these projects is to make data
analytics widely accessible for the rest of the business.

Speculative 4 What do we want from Big Data?

The first step for any organisation at the beginning of their Big Data journey
analysis with is to decide exactly what they want out of it. Computing asked: What do you
a view to think would be the main benefits to your organisation of a more advanced
(Big) Data processing and analytics solution?
uncovering new
As Figure 3 shows, the largest single proportion of respondents (54 per
insights and cent) said that quicker and better decision-making was the biggest benefit
opportunities they hoped for. Forty-seven per cent stated that improved responsiveness
to customer needs was on their wish list an increase on last year when
was viewed as this factor polled 34 per cent. Forty-six per cent hoped for sharper
competitive edge and 41 per cent the better identification of new trends and
a nice to have opportunities. However, 36 per cent were looking at internal processes and
by the majority hoped to use Big Data to make these more efficient.
of our
respondents

8 Big Data Review 2015

FIG. 3 What do you think would be the main benefits to
your organisation of a more advanced (Big) Data processing
and analytics solution?
Quicker and better decision-making 54%
44%
Improved responsiveness to 47% 2015
customers needs 34% 2014

Competitive edge 46%

40%
Identifying new trends/opportunities 41%
44%
Better efficiency of internal processes
(e.g. automation, more logical 36%
business processes) 30%
36%
Better accuracy of data 27%
Better understanding of market 33%
trends/forecasting 29%
Better understanding of customers 32%
and communication with them 38%
28%
Lower operational costs 26%
28%
Better risk management 27%
22%
Cross-selling opportunities 21%
18%
Optimised marketing to consumers 20%
0% 10% 20% 30% 40% 50% 60%

However, when asked to choose a statement to describe analytics in their

organisation, a huge 76 per cent of respondents chose We are focused
primarily on analysing operational data to obtain insights and improve
efficiency, as opposed to 24 per cent who told us that they are focused
primarily on analysing external data for opportunities and to increase
revenue (Fig. 4).

FIG. 4 If you had to choose one of the following to

describe analytics in your organisation, which would it be?

24% We are focused primarily on analysing

operational data to obtain insights and
improve efficiency

We are focused primarily on analysing

external data for opportunities and to
increase revenue
76%

Big Data Review 2015 9

 The findings of this question suggest that hoped-for benefits of Big Data
analytics to competitive edge and the identification of new commercial
opportunities may be a little further down the line for businesses than

a first glance would suggest. Some of our interviewees explain why this
may be the case.

We started to do a lot of I dont know if I will make

work with external data,
but we ended up wanting to
get our own data right first
We tried to jump to the new
opportunities too quickly
Solutions manager, NGO
money by introducing
more products, but I know that
if I get my house in order, at
least I can immediately save
some money Director data
infrastructure global technology, Banking

These responses are indicative of the fact that many organisations are still
in the early phases of Big Data deployment. A logical first step is to start
by making efficiencies and proving the value of the project. Once that is
achieved you might look at external data with a view to providing new
opportunities.

Certainly, speculative analysis with a view to uncovering new insights and

opportunities was viewed as a nice to have rather than an essential feature
by the majority of our respondents. Only eight per cent placed it at the top
of the scale for importance, with nine per cent placing it one step below
(Fig. 5). However, certain industries such as Telecoms/IT, Banking, Charity/
Third Sector and Retail in particular were more likely to undertake
speculative analysis.

FIG. 5 How important is speculative data analysis for

your organisation (looking at data from different sources
with the aim of discovering new/unexpected insights)?
Not important Very important

6% 22% 17% 20% 19% 9% 8%

We want the speculative. We want to move away from the
routine-based last year I sold these amounts of things; this year
I am going to sell five per cent more approach. There is nothing there
that says that five per cent is the right number, or that people are still

going to want to buy this stuff, or that a different segment of people
might want to buy this stuff Head of enterprise analytics, Large retail

[Speculative analysis] is enormously important for us as we are a

business that deals with data. The ability to use data better than
our competitors is a huge differentiator. Its finding better ways to draw
inferences from that, to make predictions that will improve our products
in ways that we dont know yet CIO, Services

10 Big Data Review 2015


The concept of disruptive analytics is one that several of our interviewees
were intrigued by.

Analytics was very much descriptive Disruptive

before. There was nothing dynamic analytics is much
about it. Today we are looking at disruptive more interesting.
analytics. Disruptive analytics is really whats What can you do to
happening now. Its no longer looking at the disrupt the current
past and understanding what youre going flow of the business
to do; its actually looking at the past and in order to create an
knowing what is happening in the future improvement Head of
to predict what youre going to sell Director research information & intelligence,
digital, analytics & innovation, Business services Government department

5 The democratisation of data analytics The

When business organisations are deciding exactly what they want to achieve
from Big Data analytics, they need to decide just how democratic their democratisation
analytical tools should be. of data analytics
As Figure 6 illustrates, at present data analytics are only available to creates the
specialists and heads of department in the majority of organisations. Only
15 per cent have invested in self-service analytics tools which a majority potential for an
of employees are encouraged to access. Another two per cent take data explosion of
democratisation very seriously and are in the process of rolling out training
and tools across the organisation. creative thinking
as boundaries
FIG. 6 Accessibility of data and analytics, now and in are torn down
three years
and individuals
Access to datasets and specialist tools 41%
restricted to specialists only 7% obtain a new
Specialists, heads of department and 29% picture of the
others have access 21%
A few business users have access but 12%
organisation
not encouraged 20% and their role
Invested in self service analytics tools
with most employees encouraged to
use them
15% 44%
37%
within it
Rolling out training and tools across the 2%
organisation
14%
Other 2%
1%
0% 10% 20% 30% 40% 50%
Now In 3 years

Big Data Review 2015 11

 However, when asked what they expected the picture to look like three years
from now, respondents answered very differently (Fig. 6). Only seven per
cent expected that access to data sets and analytics tools will remain limited
to specialists essentially, IT departments and business analysts. A little over
one fifth (21 per cent) expected that such tools will be available to heads of
departments with another fifth predicting that people who are interested will
be able to access core data and tools.

However, no fewer than 37 per cent of respondents expected their

organisations to have invested in self-service analytics tools and that most
employees will be encouraged to use them for decision-making. A further
14 per cent predicted that training and tools for self-services analytics will
have been rolled out across their entire organisation.

The development of new data integration, discovery and visualisation tools

along with more agile storage and the ability to analyse huge data sets

combine to drive this process. Unlike traditional business intelligence (BI)
tools, anybody (within reason) can use them.

BI tools are not very agile. With the new Big Data visualisation tools
I dont need to build anything. I dont need any semantic layer or an
object-driven interface in order to grab data out, impose an hierarchy thats
relevant and then visualise something Head of enterprise analytics, Large retail

The impact of enhanced data visualisation tools on business decision-making

is difficult to overstate. In contrast to the static tools of old, current data
visualisation tools offer the ability to look back at data points over time and

to drill down into the data at points of interest. Different employees can
interrogate the same pool of data in very different ways.

The more visually you can Visualisation is so

display something, the
better. Instead of being a bunch of
numbers or a line graph you can
turn it into a heat map for energy
use and overlay that on a city
map that will really grab peoples
key. When [a large
customer] finally put shipping
on a map, they saved 100m
by preventing goods being
shipped to the wrong place.
You can spot that very quickly

attention and they will be much through visualisation tools
more engaged... IT manager, Research Director, Consultancy

The managers of the business are much more interested in how well
they are performing, how they are complying with regulations, so
they will look at data through a completely different lens to the engineer
who will be looking at the fine details Software director, Utilities

12 Big Data Review 2015

If the predictions made by the research participants come to pass, the
implications for organisations (not to mention wider society) are enormous.
The democratisation of data analytics creates the potential for an explosion of
creative thinking as boundaries are torn down and individuals obtain a new
picture of the organisation and their role within it. New ideas can be tested
against data models as what-if? scenarios. The decision-making process
can be speeded up. Successful ventures can be expanded more quickly
and unsuccessful ones dropped sooner. In short, a data-driven business

can be more agile, rational and efficient, from top to bottom. Many of our
interviewees were excited by the possibilities.

We want to use more evidence- In the past, officers

based decision-making.
Everyone should have an equal voice
in putting forward something to test.
It could be a user test or a piece of
analysis on our customers. This has
worked very well in our product
development, but we are now trying
would sit there with
lots of paper and charts and
PowerPoint presentations.
What theyve started doing now
and this is a business-led
decision theyre actually using
the dashboard and allowing

to take that approach across our members to click in and ask

whole business CIO, Online services questions CIO, Local authority

The new way of analytics allows us to dig in, visualise, re-profile, cut,
slice and dice, which is a very dynamic and different way of managing
performance from the factory reports of how many employees youve got.
The second thing for us was the opportunity to bring information together
from multiple sources and apply these tools on top to give us insights that
otherwise we wouldnt be able to do CIO, Local authority

An interesting premise put forward by some of our participants was that

putting information and the ability to test and query data in the hands of the
business could lead to a less hierarchical organisation and a flatter decision-

making process, ultimately leading to a high level of democratisation and
empowerment.

We want to allow anybody Theres more scope for data-

in the business to put
their hypotheses forward. We
have mechanisms for anyone
to suggest their ideas. It
relieves the pressure on senior
management, the people that
have the responsibility to come
up with great ideas. It then
becomes their responsibility to
support these ideas getting built
in the most appropriate order
CIO, Services
driven decisions and more
scope for taking localised decisions
to the business units with data
so they have some predictive
positioning on it. As finances get
tighter my concern is that you get
more control put back in So yes, it
may produce flatter decision-making,
but only in those businesses where
theyve gone through the pain barrier
and are cash rich Head of research
information & intelligence, Government department

Big Data Review 2015 13

 What of the role of the IT department in the new data democracy? For
many reasons, the days of IT departments as gatekeepers to technology are
coming to an end. The move towards the democratisation of data analytics
has developed in tandem with vastly enhanced mobility and a transformation
in the nature of security threats. The Computing Data Security & Risk
Management Review 2014 explored in detail how IT departments have had
to devolve significant responsibility for data security to other departments
and employees. The democratisation of data analytics is simply another

component of this wider transformation of business organisations and the
role of IT within them.

The role of IT is Data analytics is allowing us to

changing but it is not
diminishing. As data grows
exponentially and becomes
more prevalent through the
business then the role for
IT will continue to increase
CIO, Media
have those insightful and informed
conversations between the data analytic
teams and business management
teams I also think the IT manager is
becoming more of a resource for the
business now Head of research information &
intelligence, Government department

One-hundred 6 Democratisation and best practice

If businesses are to realise the exciting possibilities of the democratisation of
per cent data data analytics, most will have to overcome a few hurdles not least selling

accuracy is a the benefits of data-driven decision-making to organisations resistant to
change. Some of our interviewees had experienced some real challenges here.
goal that is out
of reach for a Traditionally, leaders have been leaders for a reason and 99 per cent
of that is down to their gut instinct. What we are saying is we want

majority of to run our future using something else thats quite disruptive
organisations Director digital, analytics & Innovation, Business services

given budget Our field is still heavily based How do you move
in engineering. Theres been from gut feel to data-
limitations. a tradition of trusting the engineers based decision-making? In
However, the judgment and that the engineer is retail, you would think they
irreplaceable. With technology we are know their customer and
fact that a not trying to replace the engineer, but their local people: When the
to aid the engineers decision-making sun comes out I must put
majority of and allow the engineer to concentrate this in front of the store
respondents on the problems we cant solve But you cant work like
analytically or by machine learning. that nowadays, you cant
told us that That has required a cultural change. have a national chain thats
Its like the first time a pilot had to turn run by 700 different store
somewhere the auto-pilot on. Its about trusting it managers Head of enterprise
around 80 to Software director, Utilities analytics, Large retail

90 per cent
was good
enough was
telling

14 Big Data Review 2015

Part of corporate culture is the actual ownership of data and it is still
causing issues. The traditional boundaries of ownership are being broken by
the possibilities of Big Data analytics but the traditional data owners are, in

many cases, still reluctant to share it. Many organisations are still grappling
with the difficulties in persuading these parties to play nicely.

Its breaking some of the organisational silo culture as you are

suddenly enabling data in all sorts of the different places that
werent joined up before Solutions manager, NGO

Even if the cultural difficulties of bringing together disparate sources of data

can be overcome, the actual technicalities remain challenging. Different
countries and continents have differing data laws and regulations. EU Safe
Harbour, for example, inhibits the movement of personal data to jurisdictions
it deems inadequate in terms of data protection. The fact that the US

and the EU have different approaches to data protection causes significant
compliance headaches for CIOs.

We find more and Theres a huge global divergence

more that people
are sensitive to where
their data is stored. Some
will not let their data
out of US borders; some
German banks wont let
between the different legislative
regimes of North America, Europe and the
rest of the world with Europe having some of
the better privacy standards. America doesnt
care about it, then theres China and India
and other places that dont really understand

their data outside of the what the issues are and occasionally get very
EU CTO, Technology rabid about it CIO, Education

We operate in 55 countries, each with their own rules. Our main

challenge is with raw data because of the sheer number of
systems. If I look at transaction data alone, there are about 50 different
services CIO, Banking

In addition to the challenges inherent in reforming corporate culture and

actually making data available centrally, another problem is quality. The
implications of poor data quality can be far-reaching.

Provenance is a Its very difficult to re-build confidence.

huge issue. If you
dont have that chain of
data and know where
it has come from and
you see a data set in
isolation, you have no
idea if its still valid
CIO, Government agency
If data is found not to be correct, it
then becomes very damaging. It only has to
happen once to a user for him to question
every subsequent piece It is difficult to
create trust when data comes from so many
disparate sources. Even if the data has been
checked and re-checked, its coming from so
many different sources CIO, Services

However, our research indicated that the challenge (and expense) of

maintaining data quality is what is helping companies to bridge the divide
between data-driven decision-making and the old-school approach.

Big Data Review 2015 15

 While some operational data domains need to be close to 100 per cent
accurate for finance, invoices and purchase orders, for example for
analytics lower levels of data quality are generally acceptable.

Computing asked about how accurate data would need to be in order for
it to be used as the basis for an important decision.

Thirty-six per cent of respondents chose 80 per cent accuracy, 32 per cent 90
per cent accuracy, and an optimistic 11 per cent went for 100 per cent (Fig. 7).

FIG. 7 In order to make a business decision you need

to feel confident in the data; however it may not be worth
the effort to ensure data is 100 per cent accurate. At what
percentage of data accuracy would you be comfortable
in terms of using it to make an important decision (e.g. a
major marketing campaign, the launch of a new product
line, a new traffic control system, etc)?
36%
32%

11% 11%
2% 4% 3%
0% 0% 1%
10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Data accuracy required

One-hundred per cent data accuracy is a goal that is out of reach for a
majority of organisations given budget limitations. However, the fact that

a majority of respondents told us that somewhere around 80 to 90 per cent
was good enough was telling.

Its about supporting gut feeling and experience, not replacing it.

If data is 80 per cent accurate you leave room to make the actual
decision Data infrastructure director, Global bank

I always tend to hedge towards the If you were

80 per cent mark. Its the 80/20 rule confident that
for me. I know if 80 per cent of the data is the data was 90 per
relatively accurate then I have a good chance cent accurate, for most
of making a good decision You can waste business decisions you
a lot of time getting that 20 per cent Head of would probably be
research information & intelligence, Government department OK CIO, Services

The clear message is that when it comes to the data analytics and their wider
use in organisations, perfectionism is the enemy of progress.

16 Big Data Review 2015

7 Turning data into knowledge

I dont
Having explored the challenges inherent in the democratisation of data
analytics, we now turn to some of the wider issues organisations face when believe in this
setting out a strategy to realise the potential benefits of Big Data. claptrap that
When asked what the main types of difficulty were, the largest single data is the
proportion of survey respondents (38 per cent) replied that a lack of
resources be they skills or time was still the biggest barrier. Our new currency;
interviewees provided some insight into this pressing and ever-present its not. Data is

problem. The need to train people internally in the relevant skillsets is
undoubtedly slowing down progress. the new raw
material and
We went out to The candidates that you need to
market to look for do this properly are largely PhD- information is
experienced Spark and holding post graduates. If you look for
Scala developers and they people doing ecommerce theyre incredibly
the new
are as rare as hens teeth expensive. We are now looking at academic currency
or they were looking for programmes, whereas before we would
salaries beyond what we be looking at someone with commercial
are able to offer, so we experience. Its the academic skills which
have gone and trained will make the use of our data in much
people up internally more interesting ways and thats going
CTO, Technology back to academia CIO, Online services

In addition to squeezed resource, 17 per cent of respondents had concerns

about integration and 15 per cent over analytical issues such as complexity
and transparency of data. Full results are shown in Figure 8 and show
remarkably little change from responses to the same question in 2014.
Big Data may be maturing but the barriers to its wider adoption remain
remarkably consistent.

FIG. 8 Which is the main type of difficulty standing

in the way of successfully managing and exploiting the
potential opportunities offered by Big Data?
38%
Resources (lack of skills or lack of time)
39%
Integration (formatting, merging 17%
different types of data) 18%
Analytical issues (complexity of data, 15%
lack of visibility) 15%
Storage and logistical issues (including 14%
data transfer, data existing in many
different locations) 10%
Having the right solution to meet 14%
our needs 12%
4%
Other
6%
0% 10% 20% 30% 40%
2015 2014

Big Data Review 2015 17

 In order to gain further insight, Computing asked a follow-up question:
Which of these produces the most challenges in terms of achieving success
with Big Data?

Interestingly, only 13 per cent stated that the raw data was the biggest issue.
The biggest challenges facing the architects dealing in raw data are those
already discussed simply knowing what is there and what is needed,
skills, knowing where to start and stop, the collation of data from numerous
locations, etc (Fig. 9).

FIG. 9 Which of these produces the most challenges in

terms of achieving success with Big Data?

13%
Data the raw data

39%
Information turning raw data into
something useful

Knowledge providing actionable insights

48%

When these hurdles have been jumped, respondents immediately run

into another pair of related obstacles namely how to turn raw data into

something meaningful (48 per cent) and derive actionable insights from it
(39 per cent). As one of our interviewees succinctly stated:

I dont believe in this claptrap that data is the new currency; its not.
Data is the new raw material and information is the new currency
CIO, Government agency

Indeed. However, turning raw data into something useful tends to be the
preserve of BI teams and this can prove to be a procedural bottleneck.

The handover of One of the things that weve found

the information and
understanding from the BI
teams to business analysts
is not as seamless as it
could be, although they
work pretty much as one
team CIO, Online services
quite tricky is that information to
knowledge bit where there are shifting
sands around requirements. We find it
quite difficult sometimes because the
dashboards are so flexible to actually tie
down how we want to use them to drive
[decisions] CIO, Local authority

18 Big Data Review 2015

Computing asked survey respondents to rank their biggest challenges when it

came to turning data into information. The biggest problems were asking the
right questions and determining the real value of the information available.

With Big Data, theres always been too much data and not enough
science and its how to sift the wood from the trees. With the advent
of analytics and visualisation its getting a lot better, but I still think the
bottleneck is the volume of data against the question you are trying to
answer And the bottleneck is more about the skillsets than anything else
Head of research information & intelligence, Government department

Trickier still is turning information into knowledge because here the issues
are cultural as much as they are technical. It may be the analysts job to
translate information for the upper echelons of the business, but for true
data-driven decision-making this needs to go far wider. Changing the
organisational culture and building confidence in data-driven decision-
making were the two biggest challenges faced by respondents here.

As we have already touched upon, the implementation of cultural reforms

requires certain skills and, at times, sheer managerial clout. Information is
often synonymous with power and changes to its distribution will inevitably
become political. There are likely to be losers as well as winners. These
findings bring us back full circle to the key message that has occurred
throughout all of our research on Big Data three years in a row the
importance of Big Data initiatives being multi-disciplinary endeavours.

Only the business can decide what questions need to be asked and only the
business can ultimately push the cultural changes required.

If you look at the C-suite 15 years ago, did half the people today exist
within that boardroom as they do now? Thats all come from the
technology. Now data and the governance around it are all going to change
it. It will have a lasting impact across the board Director, Consultancy

Big Data Review 2015 19

 Big Data can 8 Factors for success
It follows that if Big Data is to be a success within a business it is a
only be a programme that needs to be led from the top of an organisation rather than
success if a from a technical perspective. Our survey participants agreed with this line
of reasoning with 40 per cent stating that a CIO was the best person to lead
business has a Big Data strategy. The CIOs nearest competition was the CTO who polled
17 per cent of the vote. It was felt among our interviewees that being on the
vision of what

board was a key attribute for a project initiator. There was a tangible concern
it wants to that Big Data would be perceived as an IT issue and left on the side.
achieve, Its a C-suite and CEO There is still a view that IT
knows what conversation versus a CDO- puts laptops on peoples
CIO conversation. It will ultimately desks, so they sometimes
data it has and sit with a chief data or monitoring forget to include us in the
has decided officer, but clear strategy and
vision needs to be brought in from
conversation. We are missing a
presence on the board; the CIO
how people the top as its going to drive the is not on the board, he is one
business forward Director digital, step down Data analytics manager,
are going to analytics & innovation, Business services Infrastructure

use it. And,

indeed, how it FIG. 10 Which job role is best suited to lead a Big
Data strategy?
is going to
persuade them CIO - chief information officer 40%
CTO - chief technology officer 17%
to do so
CDO - chief data officer 11%
CEO - chief executive officer 10%
COO - chief operations officer 10%
CMO - chief marketing officer 5%
CFO - chief finance officer 4%
Other 3%

Big Data (and indeed any other technology) projects can be broken into three
phases planning, implementation and exploitation of outcomes. Computing
asked participants in both qualitative and quantitative parts of our research
what they thought were the key factors for success in each stage.

20 Big Data Review 2015

8.1 Planning
The top three factors for success in the planning stage of a Big Data
programme were business buy-in (45 per cent), knowing what data you have
(36 per cent) and having a core understanding of the business (33 per cent).
Only slightly fewer respondents (32 per cent) considered having a vision of
the opportunities available a key touchstone for success, closely followed by
data governance and having the right skills available (Fig. 11).

FIG. 11 Please pick up to three essential success

factors when planning a Big Data project
Business buy-in 45%
Knowing what data you have 36%
Having a core understanding of the business 33%
Having the vision to see the opportunities 32%
Data governance 31%
The right skillsets 30%
The ability to centralise data from different sources 27%
The ability to store data in an agile and flexible way 23%
A culture that welcomes change 23%
Data provenance (tracing and recording the origins of data

and its movement between databases) 11%

I am a firm believer Theres no point doing it unless

that there is too much
data out there, so you need
a clear identification of what
the problem is youre trying to
solve or overcome Director digital,
analytics & innovation, Business services
you have a grand vision, which
is a business vision as opposed to a
technology vision. The vision of the
opportunity has to be a technology
interpretation of the business vision
CIO, Government agency

Big Data Review 2015 21

 8.2 Implementation
When it came to implementation the most important factor for our survey
respondents was also business buy-in. This was ranked as important by 48
per cent of respondents. Having a core understanding of the business came
next at 35 per cent and the importance of a multi-disciplinary implementation
team was underlined by 34 per cent, as was the requirement to know what
data you have. Resource was raised again by 31 per cent of respondents who
stated that the right skillsets were crucial to success (Fig. 12).

FIG. 12 Please pick up to five essential factors for

success when implementing a Big Data project
Business buy-in 48%
Having a core understanding of the business 35%
Implementation team including both business and IT staff 34%
Knowing what data you have 34%
The right skillsets 31%
Simple end-user tools 30%
The right technology 30%
Data governance 28%
Culture that embraces change 21%
The ability to store data in an agile and flexible way 21%
Showing results quickly 19%
Ability to process different formats of data 19%
Building confidence among business users 19%
Integration with current BI tools 17%
Refining/filtering the data for use 14%
Data provenance (tracing and recording the origins of data
and its movement between databases) 13%
Central tools to manipulate data 13%
Predictive models 8%
Adapting strategies 8%
Change internal processes 8%
Ability to scale out fast 7%

Machine learning/automation 5%

The combined Put together an approach which

functions of business
and IT people are really
quite powerful as long as we
dont get too precious about
where the boundaries are
Head of enterprise analytics, Large retail
brings together multi-disciplinary
teams of people and empowers them
to be self-organised and self-controlling
and motivated to achieve goals in a
repeatable, metronomic and iterative
fashion Software director, Utilities

22 Big Data Review 2015

8.3 Exploiting outcomes
We asked survey participants to pick up to five success factors when
implementing Big Data. Again, business buy-in was the top-scoring factor
chosen by 51 per cent of respondents. Thirty-nine per cent stated that speedy
results were a key factor in the continuing success of the project and an equal
proportion mentioned the trust in the information to enable data-driven
decision-making. Having the vision to identify the opportunities ranked
fourth at 38 per cent.

FIG .13
Please pick up to five essential factors for
success in exploiting the outcomes of a Big Data project
Business buy-in 51%
Showing results quickly 39%
Trust in the information to enable data-driven
decision-making 39%
Having the vision to see the opportunities 38%
Building confidence among business users 37%
Simple end user tools 35%
Culture that embraces change 35%
Having a core understanding of the business 30%
Visualisation/engaging the audience/self-service tools 27%
The right skillsets 25%
Adapting strategies 23%
Data governance 23%
Integration with current BI tools 21%
Data provenance (tracing and recording the origins of data
and its movement between databases) 14%
Change internal processes 12%

Machine learning/automation 4%

People bang on about data quality and data governance. Visualising

data is great, but people need the confidence that what they are

seeing cant be disputed. You need a tool which makes it easier to prove
Head of enterprise analytics, Large retail

Its getting the trust and that can take Its getting
quite a long time. There can be a the results in
lack of trust because no one had seen this quickly that brings
before or had seen it in a different way you the buy-in
Solutions manager, NGO Solutions manager, NGO

Big Data Review 2015 23

 Essentially, Big Data can only be a success if a business has a vision of what
it wants to achieve, knows what data it has and has decided how people are

going to use it. And, indeed, how it is going to persuade them to do so. Our
interviewees had more advice:

When you build an engine, you want a high-performance engine

so you know the value of the engine as the vision in your head
is a driver driving faster down the road. Youve got to think about the
cubic capacity of the engine, how you are going to aspirate it At an
engineers level, youve really got to consider the holistic engine. Whats
happening at the moment with Big Data is that people are homing in
on cam belts and the best cam belt you can buy, but is it going to make
your engine go any faster? Have you thought about visualisation? You
can buy the best Big Data technology kit there is, but if the end users
dont understand what the information is then you have wasted your

time. Its about balancing the attention, balancing the investment that
is how to mature CIO, Government agency

Business buy-in. That is the principle of analytics, working in

an information-based culture is important but this has to be
business-led. If the business says I need to be able to do this, how
do I do it Mr IT Person?; thats where we step up Head of enterprise
analytics, Large retail

The vast 9 The Internet of Things

The Internet of Things (IoT) is at the height of the hype curve right now
majority of the but is already beginning to increase the volume of data in existence to levels
data generated beyond that with which, arguably, current data infrastructures will soon be
unable to cope. Industries such as manufacturing and transport are at the
by the IoT will vanguard of the IoT but five years down the line most analysts predict that
the majority of industries will be working on projects in this area. The vast
just be noise majority of the data generated by the IoT will just be noise but businesses are

but businesses beginning to grapple with how they isolate the useful data and turn it into
knowledge and action.
are beginning
to grapple with The Internet of Things is going to produce a huge data explosion
over the next couple of years. I think CIOs have had their say on
how they where thats going and we are going to see more and more data from
connected devices, and there will be a requirement to analyse some of
isolate the that IT manager, Research
useful data
and turn it into Computing asked survey respondents about the relevance of the IoT to their
industries. The answers are shown in Figure 14. A striking finding was the 29
knowledge per cent stating that it was already making an impact not bad for a concept
at the top of the hype curve. A further 38 per cent stated that IoT would
and actionable affect the way that they worked but just not yet. Representatives from the
insight distribution and transport, telecoms/IT and public sectors were most likely
to state that they were already witnessing an impact on their organisations.

24 Big Data Review 2015

Survey respondents predicted that domestic/leisure industries would be the
one on which IoT had the greatest impact followed by retail, health, logistics,

transport/automotive and utilities. Our interviewees were excited about the

possibilities offered by IoT.

The potential around IoT is enormous. IoT to me means sensors and

everything that is to do with the construction or maintenance of the
railway should be included in the design Data analytics manager, Infrastructure

Parking bay sensors is another There are health benefits

one thats happening in the
public realm in some councils now.
I think the difficulty with public
authorities is we havent quite
cottoned on to the way in which
the IoT can completely redefine the
paradigm because weve tended to
think about delivering services by
automation of existing processes
CIO, Local authority
as more and more people
wear wearables which will
become more capable in terms
of what they can measure and
transmit. We will see a rise in
GPs monitoring their patients
through devices and being able
to say Its time you made an
appointment for a check-up
IT manager, Research

FIG. 14 How relevant to your business is the

Internet of Things?

22% 10%
Huge it is changing everything

Its already making an impact

19%
38%
It will change the way we work but not yet

I cant see it making a huge difference

Its not relevant

11%

However, there are a few challenges to resolve before the IoT takes a firm
hold on the popular imagination and day-to-day life, as Figure 15 shows.

Big Data Review 2015 25

 FIG. 15 Which of the following areas require most
work before the Internet of Things really takes off in your
industry sector?
Privacy/data protection technology 51%

Privacy/data protection legal framework 38%

Educating employees 31%

Educating customers/clients 31%

Connectivity protocols 29%

Analytics 27%

Bandwidth 23%

Storage 15%

Power supplies 4%

N/A it wont 15%

Respondents could select multiple answers

A huge 51 per cent said that privacy/data protection technology needed

further development, and 48 per cent said the actual privacy/data protection
legal framework gave them cause for concern. Certainly many individuals
and organisations are imagining the possibilities of IoT and some of those
possibilities look decidedly dystopian. With FitBit evidence now admissible in
courtrooms1 there is considerable concern that both security technology and

data protection law simply hasnt caught up with a world in which sensors are
proliferating at a lightning rate.

There has been a lot of work in the health sector to ensure

privacy. The biggest concern is who in the commercial sector
gets access and this has been taken very seriously Head of research
information & intelligence, Government department

Some concerns are perhaps a little overplayed but it doesnt require a huge
leap to imagine personal medical data being used as a tool for some very
aggressive marketing or the prospect of critical public infrastructure being
used by those with nefarious intentions particularly given the spectacular
rise in targeted attacks that we have seen in the last 18 months.
Education was the second biggest area blocking further development of IoT,
according to our respondents both in terms of educating employees and
clients. However, that education can only be effective if security concerns
have been properly addressed. If they are not, misinformation will continue
to circulate and effective education becomes infinitely more difficult.

1
http://www.forbes.com/sites/parmyolson/2014/11/16/fitbit-data-court-room-personal-injury-claim

26 Big Data Review 2015

10 Conclusions Big data hype
Computings 2015 research into Big Data has found that it is continuing
to mature and evolve. Live deployments are increasing in number as are has now been
the actionable insights they generate. Big Data hype has now been firmly
replaced by real case studies and usage of advanced analytics, agile storage firmly replaced
and data centralisation. In particular, more developed real-time analytics are by real case
pushing increasing numbers of organisations to take the Big Data plunge,
driven by both the prospect of speedy gains from internal efficiencies and a studies and
longer term prospect of speculative analysis of external data to potentially
open up new markets and opportunities. usage of
The pace of democratisation of data analytics is increasing, with huge
advanced
implications for businesses. When decision-making is truly data-driven and analytics, agile
access to analytics is not restricted to a few personnel, everything can happen
faster, making businesses more agile and efficient. Greater empowerment for storage and
a greater number of employees could be a very positive thing and potentially data
lead to changes in the traditional corporate hierarchies with which we are all
so familiar. centralisation
However, new hierarchies will create both winners and losers and corporate
culture tends to be reluctant to embrace change. IT departments will have
to evolve, and decision makers need to be persuaded of the benefits of
data-driven decision-making over decisions based on instinct. Ownership
of data is rarely relinquished without a struggle and the sheer complexities
involved in gathering and determining the accuracy and quality of raw data
held in differing legal jurisdictions and physical locations makes the task a
formidable one.

Turning this raw data into actual knowledge and action are the next steps
and it is in these areas rather than with gathering raw data that respondents
were experiencing their biggest challenges. Knowing which were the right
questions to be asking and determining what information was of genuine
value were big challenges, although not as big as the changes required in
organisational culture to build confidence in data-driven decision-making.

Given the importance of corporate culture to the success of Big Data

projects, it follows that they must be led from the top. Big Data being viewed
as an IT problem was a concern as it has been in previous years, although
most of our respondents believed that the CIO was still the best person to
own a Big Data project. Business buy-in was the key to a successful project,
and that buy-in needs to be driven by a vision of the possible, knowledge of
the data available and an acceptance that data-driven decision-making will
involve significant change in attitudes and culture.

The Internet of Things (IoT) was changing organisations for nearly one
third of our research participants and the predicted explosion in data was
starting to happen. Although progress down the other side of the hype curve
is being impeded by real concerns about privacy and the ability of security
technology to protect it, the IoT is something we are likely to be exploring in
our Big Data Reviews in the years ahead.

Big Data Review 2015 27

 About Computing
Computing is published by Incisive Media, one of the worlds leading B2B
information providers and the publisher behind some of the UKs most well
respected IT media brands, most notably CRN, The Inquirer, IT Hound and
V3.co.uk.

Incisive Medias database of hundreds of thousands of IT decision maker

subscribers puts it in an unrivalled position to provide valuable research and
commentary on the latest business IT purchasing trends and processes.

For more information about our research, surveys and content creation
services please call Tom Wright, publisher, on (+44) 20 7316 9529 or John
Leonard, research and content editor, on (+44) 20 7316 9776.

28 Big Data Review 2015