BIG DATA MINING: A CHALLENGE AND
HOW TO MANAGE IT
DINESH
CSA Deptt. PDMCE
dineshgomber@gmail.com
ABSTRACT
Big Data is a new term used to identify the datasets
that due to their large size and complexity, We call them
BIG DATA because we can not manage them with our
current methodologies or data mining software tools. Big
Data mining is the capability of extracting useful
information from these large datasets or streams of data,
that due to its volume, variability, and velocity, it was not
possible before to do it. The Big Data challenge is
becoming one of the most exciting opportunities for the
next years. We present in this issue, a broad overview of
the topic, its current status on Big Data mining. This
paper shows the challenge and tools to manage
heterogeneous information frontier in Big Data mining
research
INTRODUCTION
Data Mining is an analytic process designed to explore
data in search of consistent patterns and/or systematic
relationships between variables, and then to validate the
findings by applying the detected patterns to new subsets
of data. The ultimate goal of data mining is prediction -
and predictive data mining is the most common type of
data mining and one that has the most direct business
applications. The process of data mining consists of three
stages: (1) the initial exploration, (2) model building or
pattern identification with validation/verification, and (3)
deployment (i.e., the application of the model to new data
in order to generate predictions).
Applications where data collection has grown
tremendously and is beyond the capability of commonly
used software tools to capture, manage, and process
within a tolerable elapsed time. The most fundamental
challenge for Big Data applications is to explore the large
volumes of data and extract useful information or
knowledge for future actions . In many situations, the
knowledge extraction process has to be very efficient and
close to real time because storing all observed data is
nearly infeasible.
DATA MINING AND BIG DATA
Big data and data mining are two different things. Both of
them relate to the use of large data sets to handle the
collection or reporting of data that serves businesses or
JITENDER
CSA Deptt. PDMCE
Jitender_13@rediffmail.com
other recipients. However, the two terms are used for two
different elements of this kind of operation.
Big data is a term for a large data set. Big data sets are those
that outgrow the simple kind of database and data handling
architectures that were used in earlier times, when big data
was more expensive and less feasible. For example, sets of
data that are too large to be easily handled in a Microsoft
Excel spreadsheet could be referred to as big data sets.
Data mining refers to the activity of going through big data
sets to look for relevant or pertinent information. This type
of activity is really a good example of the old axiom
"looking for a needle in a haystack." The idea is that
businesses collect massive sets of data that may be
homogeneous or automatically collected. Decision-makers
need access to smaller, more specific pieces of data from
those large sets. They use data mining to uncover the pieces
of information that will inform leadership and help chart the
course for a business.
Data mining can involve the use of different kinds of
software packages such as analytics tools. It can be
automated, or it can be largely labor-intensive, where
individual workers send specific queries for information to
an archive or database. Generally, data mining refers to
operations that involve relatively sophisticated search
operations that return targeted and specific results. For
example, a data mining tool may look through dozens of
years of accounting information to find a specific column of
expenses or accounts receivable for a specific operating
year.
In short, big data is the asset and data mining is the
"handler" of that is used to provide beneficial results.
Data Mining Challenges with Big
Data
Data is being produced at an ever increasing rate. There has
also been an acceleration in the proportion of machine-
generated and unstructured data (photos , videos, social
media feeds and so on) compared to structured data such
that 80% or more of all data holdings are now unstructured
and new approaches and technologies are required to access,
link, manage and gain insight from these data sets. The
origin of the term Big Data is due to the fact that we are
creating a huge amount of data every day.
Volume: there is more data than ever before, its size
continues increasing, but not the percent of data that our
tools can process
Variety: there are many different types of data, as text,
sensor data, audio, video, graph, and more
Velocity: data is arriving continuously as streams of
data, and we are interested in obtaining useful information
from it in real time.
Visualization. A main task of Big Data analysis is
how to visualize the results. As the data is so big, it
is very difficult to find user-friendly visualizations.
New techniques, and frameworks to tell and show
stories will be needed,
Hidden Big Data. Large quantities of useful data are
getting lost since new data is largely untagged
filebased and unstructured data. The 2012 IDC
study on Big Data [10] explains that in 2012, 23%
(643 exabytes) of the digital universe would be
useful for Big Data if tagged and analyzed.
However, currently only 3% of the potentially
useful data is tagged, and even less is analyzed.
T soft-ware environment designed for statistical
TOOLS: OPEN SOURCE
REVOLUTION
The Big Data phenomenon is intrinsically related to the
open source software revolution. Large companies as
Face-book, Yahoo!, Twitter, LinkedIn bene t and
contribute work-ing on open source projects. Big Data
infrastructure deals with Hadoop, and other related
software as:
Apache Hadoop [1]: software for data-intensive dis-
tributed applications, based in the MapReduce pro-
gramming model and a distributed le system called
Hadoop Distributed Filesystem (HDFS). Hadoop al-
lows writing applications that rapidly process large
amounts of data in parallel on large clusters of
compute nodes. A MapReduce job divides the input
dataset into independent subsets that are processed by
map tasks in parallel. This step of mapping is then fol-
lowed by a step of reducing tasks. These reduce tasks
use the output of the maps to obtain the nal result of
the job.
Apache Hadoop related projects [2]: Apache Pig, Apache
Hive, Apache HBase, Apache ZooKeeper, Apache Cas-
sandra, Cascading, Scribe and many others.
Apache S4 [3]: platform for processing continuous
data streams. S4 is designedspeci cally for managing
data streams. S4 apps are designed combining streams
and processing elements in real time.
Storm [4]: software for streaming data-intensive dis-
tributed applications, similar to S4, and developed by
Nathan Marz at Twitter.
In Big Data Mining, there are many open source
initiatives. The most popular are the following:
Apache Mahout [5]: Scalable machine learning and
data mining open source software based mainly in
Hadoop. It has implementations of a wide range of
machine learning and data mining algorithms:
clustering, clas-si cation, collaborative ltering and
frequent pattern mining.
R [6]: open source programming language and
computing and visualization. R was designed by
Ross Ihaka and Robert Gentleman at the
University of Auckland, New Zealand beginning
in 1993 and is used for statistical analysis of
very large data sets.
MOA [7]: Stream data mining open source
software to perform data mining in real time. It
has imple-mentations of classi cation,
regression, clustering and frequent item set
mining and frequent graph mining. It started as
a project of the Machine Learning group of
University of Waikato, New Zealand, famous for
the WEKA software. The streams framework [6]
provides an environment for de ning and
running stream pro-cesses using simple XML
based de nitions and is able to use MOA,
Android and Storm. SAMOA [1] is a new
upcoming software project for distributed stream
mining that will combine S4 and Storm with
MOA.
 Vowpal Wabbit [8]: open source project started at More specic to Big Graph mining we found the
following open source tools:

Yahoo! Research and continuing at Microsoft

Research to design a fast, scalable, useful
learning algorithm. VW is able to learn from
terafeature datasets. It can exceed the
throughput of any single machine network
interface when doing linear learning, via
parallel learn-ing.
Pegasus [9]: big graph mining system built on
top of MapReduce. It allows to show patterns
and anomalies in massive real-world graphs. See
the paper by U. Kang and Christos Faloutsos in
this issue.

Conclusion
[5] Apache Mahout, http://mahout.apache.org.

[6] R Core Team. R: A Language and Environment for

Due to increasing the size of data day by day, Big
Data is going to continue growing during the next
years and becoming one of the exciting opportunities
in future. Because todays we have an overwhelming
growth of data in terms of volume, velocity and
variety. So from a security and privacy standpoint,
the threat landscape and security and privacy risks
have also seen an unprecedented growth. Now we
are in a new era where Big Data mining will help us
to discover knowledge that no one has discovered
before In order to explore Big Data, we have
analyzed several challenges at the data, model, and
system levels. To support Big Data mining, high
performance computing platforms are required
which impose systematic designs to unleash the full
power of the Big Data.
Statistical Computing. R Foundation for Statistical
Computing, Vienna, Austria, 2012. ISBN 3-900051-07-0.
[7] A. Bifet, G. Holmes, R. Kirkby, and B. Pfahringer.
MOA: Massive Online Analysis http://moa.
cms.waikato.ac.nz/. Journal of Machine Learning
Research (JMLR), 2010.
[8] D. Laney. 3-D Data Management: Controlling Data
Volume, Velocity and Variety. META Group Research
Note, February 6, 2001
[9]U. Kang, D. H. Chau, and C. Faloutsos. PEGASUS:
Mining Billion-Scale Graphs in the Cloud. 2012.
[10]J. Gantz and D. Reinsel. IDC: The Digital Universe in
2020: Big Data, Bigger Digital Shadows, and Biggest
Growth in the Far East. December 2012.

REFERENCES

[1] Apache Hadoop, http://hadoop.apache.org.

[2] P. Zikopoulos, C. Eaton, D. deRoos, T. Deutsch, and G.

Lapis. IBM Understanding Big Data: Analytics for
Enterprise Class Hadoop and Streaming Data. McGraw-
Hill Companies,Incorporated, 2011

[3] L. Neumeyer, B. Robbins, A. Nair, and A. Kesari. S4:

Distributed Stream Computing Platform. In ICDM
Workshops, pages 170177, 2010

[4] Storm, http://storm-project.net.

.