2015 IEEE International Conference on Big Data (Big Data)
Automotive Big Data: Applications, Workloads and
Infrastructures
Andre Luckow , Ken Kennedy , Fabian Manhardt , Emil Djerekarov , Bennie Vorster , Amy Apon
Innovation Lab, BMW Group IT Research Center, Information Management Americas
Greenville, South Carolina, USA
Clemson University, Clemson, South Carolina, USA
AbstractData is increasingly affecting the automotive indus-
try, from vehicle development, to manufacturing and service
processes, to online services centered around the connected
vehicle. Connected, mobile and Internet of Things devices and
machines generate immense amounts of sensor data. The ability
to process and analyze this data to extract insights and knowledge
that enable intelligent services, new ways to understand business
problems, improvements of processes and decisions, is a critical
capability. Hadoop is a scalable platform for compute and storage
and emerged as de-facto standard for Big Data processing at
Internet companies and in the scientic community. However,
there is a lack of understanding of how and for what use cases
these new Hadoop capabilities can be efciently used to augment
automotive applications and systems. This paper surveys use
cases and applications for deploying Hadoop in the automotive
industry.
Over the years a rich ecosystem emerged around Hadoop
comprising tools for parallel, in-memory and stream processing
(most notable MapReduce and Spark), SQL and NOSQL engines
(Hive, HBase), and machine learning (Mahout, MLlib). It is
critical to develop an understanding of automotive applications
and their characteristics and requirements for data discovery,
integration, exploration and analytics. We then map these re-
quirements to a conned technical architecture consisting of core
Hadoop services and libraries for data ingest, processing and
analytics. The objective of this paper is to address questions,
such as: What applications and datasets are suitable for Hadoop?
How can a diverse set of frameworks and tools be managed
on multi-tenant Hadoop cluster? How do these tools integrate
with existing relational data management systems? How can
enterprise security requirements be addressed? What are the per-
formance characteristics of these tools for real-world automotive
applications? To address the last question, we utilize a standard
benchmark (TPCx-HS), and two application benchmarks (SQL
and machine learning) that operate on a dataset of multiple
Terabytes and billions of rows.
I. I NTRODUCTION
The increasing digitalization of the automotive industry
driven by mobile and connected devices that carry an in-
creasing number of sensors is creating a signicant increase
in demand for data storage, processing and analytics. Enter-
prises are overwhelmed with large data volumes and analytics
requirements as machine and sensor data generated by the
Internet of Things (IoT) (soon the Internet of Everything) is
collected in ner granularities and higher frequencies. Exam-
ples of such applications in the automotive industry are: the
connected vehicle, autonomous driving, smart manufacturing,
the Industrial Internet, and mobility services. The requirements
of Big Data applications are often described using the 5 Vs of
978-1-4799-9926-2/15/$31.00 2015 IEEE
volume, variety, velocity, veracity, and value (see [1] for the
original 3 Vs): Volume: Industry sources estimate that on aver-
age about 480 TB of data were collected by every automotive
manufacturer in 2013 (IHS) [2]. It is expected that this size will
increase to 11.1 PB per year in 2020. Variety: Heterogeneous
data sets generated by different sources stored in different
formats impose signicant challenges during the ingest and
integration process; their fusion enables sophisticated analytic
approaches ranging from self-learning algorithms for pattern
detection to dimension reduction approaches for complex
predictions. Velocity: Data ingest rates and processing require-
ments vary widely from batch processing to real-time event
processing of online data feeds, inducing high requirements
on data infrastructure. Veracity: describes the uncertainty in
the data, in the data provenance and the analytical modeling.
Value: describes the ability to extract meaningful, actionable
business insights from data. Further important dimensions
include: data quality, provenance & lifecycle management.
The term data lake is used to refer to the ability to retain
large volumes of data in its original, raw form to enable
agile analytics on it. The phrase is commonly used to de-
scribe Hadoop deployments. Hadoop [3] was developed as an
open source implementation of the MapReduce [4] abstraction
introduced by Google. Hadoop is based on a parallel and
distributed architecture and commodity hardware providing
economical storage and processing capabilities. A vibrant
ecosystem of tools for various data-related tasks, for data in-
gest (Flume [5], Kafka [6]), data processing (e. g. MapReduce,
Spark [7], Flink [8] etc.) and advanced data analytics (e. g.
MLlib, Mahout) emerged. Hadoop is increasingly competing
with enterprise datawarehouse systems (Oracle, SQL Server,
Teradata, HANA, etc.).
While there is vast research on the usage of Hadoop in
Internet companies [9] and in sciences [10], an understanding
on the trade-offs of automotive Big Data applications and
the suitability for a Hadoop data lake is missing. In this
paper, we investigate the characteristics of automotive data
applications to understand the variety of automotive data
sources, their volume and velocity requirements, and derive an
understanding how these requirements can be addressed using
Hadoop and Hadoop-based tools. We propose and validate a
Hadoop-based architecture for an automotive data lake: First,
we must understand the automotive use cases and requirements
for the infrastructure. Second, the tools and middleware stack
1201
representing the foundation of the platform must be selected.
This is particular challenging as the variety of tools is very
large Fox [11] lists >350 tools in the Big Data ecosystem.
Based on the results of the technical platform, services for
managing and curating data need to be established. Finally,
business requirements must be carefully balanced with data
governance and compliance considerations.
Further, there is lack of understanding of the performance of
Hadoop and other Hadoop ecosystem tools for workloads, such
as data extraction, transformation, SQL, and machine learning
tasks. While various benchmark suites emerged (e. g. TPCx-
HS, TPC-DS, BigDataBench), they often lack applicability
to real-life workloads and are often misused for marketing
purposes making it difcult to generalize the results to real-
world applications (see Baru [12] and Fox [13] for discussions
on Big Data benchmarks). We propose the usage of a hybrid
approach consisting of two automotive application and a
standard benchmark (TPCx-HS [14]). This will enable us to
understand real-life performance trade-offs, which is relevant
to the architecture and sizing of the automotive data lake. In
particular, we investigate: (i) the performance/capability trade-
off of processing frameworks such as MapReduce, Tez [15],
Spark [7] and Flink [8] using Terasort/TPCx-HS, (ii) the SQL
performance of Hive [9] and Spark-SQL on an automotive
dataset with 120 billion records, and compare the performance
to a state-of-the-art in-memory relational data management
system (RDBMS), and (iii) the performance and scalability
of Hadoop-based machine learning using Spark MLLib [16]
in comparison to tools, such as R. For (ii) and (iii), we utilize
two real world automotive datasets to derive benchmarks
with different scale factors using the synthetic data generator
developed by Kennedy [17].
This paper is structured as follows: In section II, we
provide an overview of automotive use cases and analyze their
characteristics and requirements with respect to a Big Data
infrastructure. We propose a data lake architecture comprising
multiple logical layers and frameworks/tools in section III.
We particularly investigate processing and execution man-
agement frameworks and higher-level frameworks for SQL
and advanced analytics. Further, we evaluate the security
requirements and mechanisms available for Hadoop. We con-
clude this section with a discussion of how the proposed
architecture addresses automotive application requirements. In
section IV we present our experiences with Hadoop using
selected automotive workloads.
II. AUTOMOTIVE A PPLICATIONS
Scientic disciplines, such as physics and biology, along
with Internet companies, have been processing large volumes
of data for some time. Now, traditional industries, such as the
automotive, are facing similar challenges as digitalization of
their business processes and models increases. Developing and
characterizing use cases is critical for designing an automotive
Big Data platform.
1202
A. Connected Vehicle and Intelligent Transport Systems
Gartner estimates that by 2020 most vehicles in mature
automotive markets will have data connectivity [18]. An
increasing number of vehicles are already online and utilize
telematic services, such as OnStar, Ford Sync, and BMWs
Connected Drive. In addition to 3G/LTE based services, it
can be expected that vehicle-to-vehicle (V2V) and vehicle-
to-infrastructure (V2I) based services will transition from
research to production in the near future.
The number of sensors per vehicle and the data produced by
these is rapidly increasing. Today, the electronic control units
(ECUs) of a vehicle can generate multiple GBs of data per
second. The majority of this data is transient; only a fraction
of data is typically retained and used by telematic services,
e. g. for trafc prediction, safety warnings, vehicle diagnostics,
location-based services (e. g. local search), entertainment ser-
vices, and autonomous driving. By 2018 it is expected that for
BMW alone more 10 million vehicles will have connectivity
and will each transmit more than 1 TB of data each day [19].
Trafc prediction is an example of a service that relies
on crowdsourced sensor data, collected from vehicle eets,
smartphones, and trafc infrastructures. In addition, this data
is valuable for improving the energy efciency of vehicles.
The prediction of key trafc indicators, such as trafc ows
or trafc signal states, is a challenge. Often, highly dynamic
sources (e. g. map data, current and historic trafc ows, and
weather) need to be fused to enable accurate prediction. For
example, we investigate the utilization of GPS probe data for
predicting trafc light phases and to for improving the energy
efciency of the vehicle [20]. While optimizing the travel of
an individual vehicle yields benets, the data can also be used
to understand and improve trafc infrastructure.
Autonomous driving requires complex sensor data fusion
(e. g. infrared, Lidar and camera-based sensors), which pro-
vides the basis for the generation of an environmental model
and for making real-time decisions. The Google self-driving
car generates about 750 MB of sensor data per second [21].
Autonomous driving requires sophisticated machine learning
techniques, e. g. deep learning methods for image classication
and understanding and for learning trafc situations.
While connected vehicle applications and services are often
centered around a single vendor, even greater benets occur
when data are combined from public and private sources,
including automotive, smart phone, parking, energy, and public
transportation. Intelligent transport systems services enabled
by this data include intermodal routing, advanced parking
services, car sharing, electric vehicle related services, and the
optimization of transportation infrastructure usage to increase
throughput, and to reduce emissions and accidents.
B. Manufacturing & Quality
The Industrial Internet [22] describes the increase of con-
nected machines in industrial environments and the resulting
ability to optimize global operations using data generated
during the production process. The amount of data collected
during the production of a vehicle from OEMs and suppliers
is rapidly increasing. New opportunities are being created
to monitor, analyze, and optimize the manufacturing process
and the supply chain. Currently, the majority of this data is
semi-structured produced by IT systems and humans, but an
increasing amount is machine-generated (e. g., from PLCs or
SCADA systems). The fusion of this data with other data
sources, such as diagnostic data from the eld and engineering
data, enables predictive analytics to understand and prevent
quality issues in the plant. By combining this data with
diagnostic data from the eld, a broader perspective on quality
can be obtained that will inform the design of future vehicle
models.
C. Deep Learning for Image/Video Analytics
A particular challenge is the large amount of unstructured
data (videos, images, text), e. g. from camera-based sensors
on the vehicle or machines in the manufacturing process.
To effectively utilize this kind of data new machine learning
methods like deep learning are essential. Deep learning [23]
refers to a set of machine learning algorithms that utilize neural
networks for feature extraction/learning, classication and pre-
diction. Deep Learning is powering Internet services, e. g. the
voice recognition and dialog systems of Siri, Google Now and
Microsoft Cortana, and will have many applications within the
automotive industry, such as computer vision for autonomous
driving and robotics, for optimizations in the manufacturing
process (e. g. monitoring for quality issues), connected vehicle
and infotainment services (e. g. voice recognition systems).
D. Discussion
The automotive industry is transforming: digital services are
center of innovation. In addition to the described use cases,
there are many more automotive applications as the digital
footprint of customers is increasing. More digital touch points
(e.g., web, mobile devices, social media, online services) gen-
erate more and often complex unstructured data that requires
machine learning techniques to extract knowledge and insight.
Existing datawarehouse solutions inhibit to scale to meet the
immense data volume, velocity and analytics requirements of
the described applications. Initially explorations of these uses
cases typically focused on augmented relational data manage-
ment systems (RDBMS) with the capabilities of a scalable,
distributed storage and compute platform, such as Hadoop.
This allows us to combine medium-size data sources (e. g.
transactional data from system of records systems) with large
volume data sources, e. g. sensor and probe data. Supporting
ad-hoc queries on a dynamic set of datasets in an agile way
using higher-level tools, such as SQL and SQL-based business
intelligence tools is a critical requirements. Further, machine
learning techniques for extracting knowledge from data is
critical for different tasks: for assisting data discovery by
automatic identication of patterns, for understanding com-
plex interactions and correlations in data and for predictive
modeling. Scalable machine learning techniques are required
to deal with large volumes of data and also with the inherent
high dimensionality of unstructured data, such as text, images
1203
Business Intelligence
(Reporting, OLAP, Data
Advanced Analytics
(ETL, Exploration, Prediction, Clustering,
Data Applications
(Recommendations, Applications
Discovery) Search) Predications)
Higher-Level
Security & Management
SQL Data Frame Custom APIs
Processing
Hadoop Processing Hadoop Streaming Processing &
Relational
Databases (RDMS)
(HBase, MapReduce, Spark, Flink, (Spark Streaming, Execution
Custom Jobs) Storm, Flink) Management
(Oracle, SQLServer,
PostgreSQL)
Data/Compute
Hadoop Filesystem/YARN
Resources
Data Ingest, Loading and Integration
(API Access, Flume, Sqoop, Kafka) Data
Data Sources Ingest
(Transactional/ERP data, sensor data, machine-generated data, log data)
Fig. 1. Data Lake Architecture: The architecture comprises ve layers: In
the application layer the focus is to support advanced analytics and custom
data applications. The majority of applications will access the data lake via
SQL on the API layer. Data processing is done using Hadoop frameworks on
top of HDFS/YARN.
and video. Such applications have a very different workload
characteristic than traditional Big Data workloads that were
primarily I/O bound; neural networks used for deep learning
are e. g. compute-intensive. A Big Data platform must account
for these complex requirements.
III. A PACHE H ADOOP FOR AUTOMOTIVE A PPLICATIONS
Hadoop was conceived as a scalable infrastructure for data-
intensive applications providing an open source implemen-
tation of the MapReduce programming model [4]. With the
introduction of the YARN resource manager [24], Hadoop has
evolved to a general-purpose cluster computing framework for
heterogeneous tasks, supporting interactive and batch tasks as
well as data- and compute-intensive tasks. We refer to the
Hadoop Filesystem (HDFS) and YARN as core Hadoop and
to higher-level frameworks and tools as the Hadoop ecosystem
or environment.
Figure 1 shows the data lake architecture. The main ob-
jective of the architecture is the storage and processing of
large volumes supporting ETL, analytics and machine learn-
ing workloads. The Hadoop ecosystem differs signicantly
from traditional enterprise environments, which are typically
characterized by long release cycles, well-dened product
dependencies and monolithic solutions. While the Hadoop
ecosystem is evolving to higher-level abstractions, it requires
a deep expertise in distributed computing, data storage and
processing. This main objective is to understand this land-
scape of Hadoop tools, the different trade-offs und evaluate
how these tools can be successfully deployed, operated, and
integrated in an enterprise environment. An important concern
is interoperability with enterprise datawarehouse environments
based mainly on relational technologies. As the majority of
structured data resides in existing relational environments, a
stable integration with these environments is essential.
The architecture comprises ve layers: In the application
layer the focus is to support advanced analytics and custom
data applications. The majority of applications will access the
data lake via SQL on the API layer. For advanced analytics
the dataframe abstraction is provided (see section III-C). Direct
access via specic framework API will be available to some
special applications. In the processing layer we distinguish
between data processing (batch and interactive) and stream-
ing frameworks (realtime). The data/compute resource layer
encapsulates the core Hadoop services YARN and HDFS.
While we currently envision that the majority of interactions is
directed by users, in the future the degree of automation will
improve and the number of automated machine connections
to the cluster will increase conducting data loads, discovery,
integration and analytics tasks without user interactions.
In the following we survey tools for the different layers:
In section III-A we analyze the processing and execution
management layer for supporting batch and stream computing;
we continue with the analytics layer investigating SQL and ad-
vanced analytics/machine learning capabilities in section III-B
and III-C. In section III-D we will analyze the current state of
security and governance in the Hadoop ecosystem.
A. Processing and Execution Management
Hadoops original MapReduce uses a disk-oriented ap-
proach for storage and processing of large volumes of data.
It will remain important for archival storage and computing,
but can have slow access speeds for interactive or real-time
analytics requiring queries and for iterative processing for
machine learning. To address these issues various processing
and execution frameworks evolved, e. g. Spark [7], Flink [8],
Tez [15], HARP [25]. Typically, a processing framework
comprises a programming model and an execution engine that
is responsible for lower-level resource management tasks, such
as scheduling, optimizations etc.
In particular, Spark rapidly gained popularity. It utilizes
in-memory computing, which makes it particularly suitable
for iterative processing. The programming model is based on
an abstraction referred to as Resilient Distributed Datasets
(RDD). RDDs are immutable, read-only data structures on
which the application can apply transformations. The run-
time handles the loading and distribution of the data into
the memory of the cluster nodes. The immutable nature of
RDDs enables efcient recoveries after a failure by simple
re-applying transformations. A constraint of Spark is that
its in-memory capabilities are limited to single application
jobs. There are multiple on-going efforts to extract in-memory
capabilities into a separate runtime layer that is usable by
multiple frameworks and not restricted to caching data within
a single framework and/or job. Tachyon provides a HDFS
compatible distributed in-memory lesystem. HDFS supports
since Hadoop 2.3 an in-memory cache.
An emerging class of data-driven applications, e. g. trafc
prediction and autonomous driving, requires realtime data pro-
cessing and analytics. The Lambda architecture [26] aims to
support data-driven applications that have high data volumes,
ingest rates, and require both realtime and batch analytics. For
this purpose, the Lambda architecture denes a batch layer,
a serving layer and a speed layer. Hadoop while initially
designed for MapReduce-style batch computing supports
more stream processing via different frameworks and tools:
The core of a realtime pipeline is a message broker, such as
1204
Kafka [6], which decouples message delivery and processing,
enabling the separation of the realtime and batch processing
pipeline. A message broker enables applications to support
multiple data consumers, e. g. a batch consumer for large
historical analytics and realtime consumers for incremental
model updates. While in the past several processing engines
specialized for Streaming emerged (e. g. Storm [27]), program-
ming models between batch and streaming are harmonizing.
Spark Streaming and Flink allows users to utilize the same
API on both batch and streaming data sources. Sparks MLlib
supports the update of models inside the Spark Streaming
environment.
B. SQL Frameworks
While Hadoop MapReduce and Spark provide a set of well-
dened, scalable abstractions for efcient data processing,
higher-level abstraction to structured data - either based on
SQL or document/object-store capabilities - enable a higher
productivity.
Many automotive use cases rely on SQL as common gram-
mar for data extraction. SQL is particularly useful for querying
columnar data that has been derived from less structured
data sources. In general, two architectures emerged: (i) the
integration of Hadoop with existing relational datawarehouse
systems and (ii) the implementation of SQL engines on top of
core Hadoop services, i. e. HDFS and YARN. Architecture (i)
is often used to integrate data from Hadoop into an existing
datawarehouse. The scalability of this architecture is limited
as queried data always needs to be processed in the database
(which is typically a magnitude smaller than the Hadoop
cluster). In the following we focus on Hadoop SQL engines.
Inspired by Googles Dremel [28], various SQL query
engines running over Hadoop have emerged: Hive [9],
HAWQ [29], Impala [30] and Spark-SQL [31]. Hive was the
rst and is one of the most used SQL engine for Hadoop.
Early versions of Hive compiled SQL into MapReduce jobs
which often yielded non-optimal performance. Thus, Hive was
extended to multiple runtimes, e. g. Tez [15] and Spark in addi-
tion to MapReduce. Tez generalizes the MapReduce model to
a generic dataow oriented processing model while providing
an improved runtime system that supports in-memory caching
and container re-use. Spark-SQL [7] is a SQL-engine that is
part of the Spark distribution. A particular advantage of Spark-
SQL is the tight integration with other analytic tools provided
in the Spark ecosystems - results of SQL queries can be loaded
into a dataframe and then further processed with advanced
analytics tools, such as MLlib (see section III-C).
In contrast to traditional databases that often store data
in proprietary formats tightly coupled to their processing
engine, open data storage formats are essential in the Hadoop
environment in order to maintain interoperability between the
various tools and achieving a maximum level of exibility.
Very commonly there are multiple access paths and tools that
access the same data. Thus, various le formats optimized for
different use cases emerged: ORC [32] and Parquet [33] are
particularly designed for SQL workloads using a columnar
data structure.
In summary, SQL processing on Hadoop provides more
exibility compared to monolithic RDBMS, but requires a
deeper understanding of the trade-offs: An application can
optimize storage format (columnar storage for analytics, ran-
dom access), processing engine (MapReduce, Tez, Spark),
SQL frontend (Hive, Spark-SQL), the optimizer, to meet the
requirements of the particular workload. The optimizer plays
a critical role in achieving an optimal performance. Project
Calcite used e. g. by Hive provides a data-format and
processing engine agnostic optimizer that can be used to
implement federated SQL engines on top of heterogeneous
data sources (e. g. Hive, HBase, MongoDB).
C. Advanced Analytics
The term analytics is broadly used to describe systems
for extract-transform-load (ETL), business intelligence for
data exploration and reporting (e. g. QlikView, Tableau) and
advanced analytics tasks, such as data mining and machine
learning. In the following, we focus on programmatic ad-
vanced analytics tools. The most known abstraction for data
analytics is the dataframe abstractions originally introduced in
R (respectively it predecessor S) exposes data in a tabulated
format to the user and supports the efcient expression of
data transformations and analytics [34]. A dataframe provides
a two-dimensional table abstraction that can store different
types of data (e. g. numeric, text data and categorical data) in
its columns. Depending on the specic implementation, the
dataframe abstraction typically supports various functions for
data transformations, e. g. for subsetting, merging, ltering and
aggregating data. In addition to R, several other dataframe
implementations exist, e. g. Pandas [35], NumPy [36], and
Dato SFrames [37] for Python. Traditionally, implementations
of dataframe abstraction lacked the ability to scale-out. In
the following we investigate different analytic environments
with respect to their expressiveness, ease-of-use, scalability
and advanced analytics/machine learning support.
Figure 2 shows the landscape of analytics tools. The R
language [34] is a popular programming environment for data
analytics. While R provides an ecosystem comprising of a
manifold set of packages for statistics and machine learning,
it has severe limitation with respect to scalability [38]. Various
approaches for supporting better scalability via out-of-core
and distributed processing emerged. Often, R or Python serves
as an interface to these scalable analytics environments (e. g.
Spark, H2O, or in-database analytics). In the following we
focus in particular on the R and Python ecosystems as these are
the most prevalent programming languages for data science.
Scaling machine learning and analytics techniques requires
careful considerations of two forms of parallelism: data par-
allelism typically can easily explored using programming
models, such as MapReduce or Spark. Linear algebra solvers
often used for machine learning typically demand a more ne-
grained parallelism, which is more difcult to scale. However,
1205
Hadoop
Cluster
Analytics
Mahout Spark/MLLib
H2O
In-Database
Analytics
PivotalR Madlib
Scale
Oracle Enterprise
R
Workstation
Workstation-based
Python/Dato R
Analytics
Open Source Python/Scikit pbdR Python
R
Revolution R Other (Java, Scala, SQL)
None Data Data & Compute
Parallelism
Fig. 2. Scalable Data Analytics: We identify three categories of tools:
workstation-based tools typically provide the most comprehensive set of
features and best ease-of-use. In-database analytics provide a more scalable
environment, but are typically less scalable and exible than Hadoop-based
tools.
various parallel libraries for these tasks, e. g. ScaLAPACK [39]
or ARPACK [40], already exist.
Several R packages for out-of-core and parallel processing,
either via explicit implementation (e. g. pbdR [41]) or im-
plicitly within a higher-level package, emerged. pdbR re-uses
ScaLAPACK for linear, while the open source R relies on the
non-parallel LAPACK and LINPACK. Further, various higher-
level packages for R exist that provide dataframe compatible
abstraction, but delegate the implementation data transforma-
tions and analytics to a distributed environment, e. g. Hadoop
or Spark. For example, Spark-SQL and MLLib [16] provide a
dataframe abstraction for Python, Scala and R the dataframe
is implemented using the well-known RDD abstraction of
Spark. MLLib relies on ARPACK encapsulated by netlib-
java for linear algebra operations [42]. Revolution R provides
data-frame level implementations of analytics functions that
are implemented by utilizing lower-level data-parallelism for
ltering, transformations etc. (e. g. rhadoop) and data/compute
parallelism for advanced analytics (e. g. for generalized linear
models, decision trees and KMeans clustering). H2O [43]
utilizes a similar architecture. A challenge is the need to copy
the data between the local R environment and the cluster.
Further, the exposed dataframe APIs mimic R idioms, but
cannot be used interchangeably with the standard R dataframe.
Another approach is the usage of in-database analytics. The
levels of integration between R and databases can vary. In the
simplest case the data-parallelism can be exploited using PL/R
code. Other systems integrate Revolution R (e. g. Teradata)
or custom analytics implementations (e. g. Oracle). Another
example is Madlib [44] a library that was designed for in-
database use. It runs embedded in databases, such as Postgres,
HAWQ and Impala, enabling the usage of analytics functions
via SQL.
In general, two kind of integrations of analytic tools with
Hadoop exist: (i) tools that can load/store data from Hadoop
and (ii) tools that are tightly integrated with Hadoop and
process data in-situ. Nearly all analytical tools support access
to data stored in Hadoop either via HDFS or via an ODBC
connection to columnar data stored in Hive. Data discovery
tools (such as QlikView, Tableau), BI and analytical tools
(e. g., R, Scikit-Learn, SPSS and SAS) fall into this category.
While this approach enables the quick processing of small
to medium size data, it also has some disadvantage: data
is duplicated, which increases the management and security
overhead, and the scalability is often limited. The number and
maturity of higher-level tools that support scenario (ii) and
bridge the various layers and natively run analytics inside the
cluster is still limited.
D. Data Governance and Security
The data lake as central platform for storing diverse data sets
supporting ad-hoc data integration, querying and analytics is a
challenging environment with respect to data governance and
security. Often, the precise nature and privacy implications of
the data at hand are unknown and requires analysis before
the data can be efciently governed. Furthermore, the process
of linking data sources can potentially lead to unanticipated
privacy issues. Thus, it is critical to dene security measures
and governance processes for data access, retention, linkage
and quality. The default security level of Hadoop is very low.
As Hadoop evolved security mechanisms have been retrotted
into different frameworks, however, it still lacks a coherent
security architecture. The Kerberos feature is critical to prevent
the easy impersonation of other user accounts. However, the
maturity of Kerberos support is in many tools very low -
some frameworks, e. g. Kafka, have no Kerberos support at
all, others, e. g. Spark, have difculty to interact with certain
secured services.
It is best practice to deploy Hadoop behind a rewall and
only expose a small, selected set of services, either direct or
via a gateway service, such as Apache Knox. Accordingly,
the number of users with direct access to HDFS and YARN
can be kept small. Knox allows the integration with single-
sign-on environments (e. g. Siteminder) and can forward the
authenticated user or map it to specic service users. In non-
Kerberos clusters, these gateway services are valid approaches
for minimizing the number of users with direct access to HDF-
S/YARN even though only a small number of use cases can be
sufciently supported. The usage of Kerberos authentication is
essential for large-scale, multi-tenant data lakes and sensitive
data.
Another important concern is authorization. Many frame-
works chose to implement their own authorization infrastruc-
ture: Hives SQL grants/revokes or cell-level authorization for
HBase and Accumolo are examples of this approach. These
tools rely on a shared service user for all HDFS interaction.
While these silo-ed security schemes simplify the security
in their domain, they increase the management complexity
and the difculty of implementing consistent policies. Two
approaches for solving this issue have been proposed: (i)
Sentry adds a common authorization handler for adding role-
based authorizations to different frameworks (currently Hive
and Cloudera Impala). (ii) The usage of a shared directory
1206
for user group and role management (e. g. based on LDAP)
and a security management tool, such as Argus, to ensure that
privileges are correctly translated to the different frameworks.
The support for encryption of both network trafc and
HDFS data is required for sensitive data. While network trafc
encryption is supported by many Hadoop services, no built-
in encryption for HDFS data is provided at the moment. The
setup of network encryption is complex: encryption must be
enabled for all daemons, thus, management support for cong-
uration and key management is important. HDFS encryption
is supported since Hadoop 2.6; several products attempt to
simplify and enhance this feature, e. g. Vormetric, Voltage or
Clouderas Navigator Encrypt and Key Trustee.
Data lifecycle management describes the ability to track
data provenance and lineage across the different phases: data
ingest, storage and processing. Data sources and heteroge-
neous data processing infrastructures contribute to the com-
plexity of this task. Often, it is required to move data between
different data stores with different, potentially inconsistent,
access policies. An important part of the data lifecycle is
the assessment of the data criticality and the assignment of
access policies. Hadoops data lifecycle solutions are still in
their infancy with products, such as Clouderas Navigator
and Apache Falcon. As part of the data lifecycle it is often
necessary that data is masked (i.e. parts of the data need to be
removed or pseudonymized) to meet the security level of the
platform [17].
E. Discussion
Automotive Big Data applications in the different domains:
manufacturing, quality, connected vehicle and intelligent trans-
port systems share common characteristics: they bring together
a variety of data sources well structured data from transac-
tional systems and less structured data generated by machines,
mobile devices, social networks etc. Loading and integrating
these datasets and making it explorable via higher-level tools,
such as SQL is critical. Understanding the trade-offs of the
different SQL on Hadoop frameworks in conjunction with the
different data formats is therefore crucial. As machine learning
techniques get deployed in the various phases of data science
projects, e. g. for data cleaning, data discovery, and different
form of analytics (text analytics, pattern mining, predictive
modeling, etc.). Specialized tools for deep learning e. g. often
require an specialized infrastructure (using e. g. GPU) and
often do not integrate with the Hadoop scheduler YARN.
YARN decouples cluster resource management and appli-
cations and allows to deploy most frameworks on user-level
without administrative involvement. However, in particular
frameworks that require a tighter integrations, e. g. Spark-
SQL/Hive, some pitfalls in particular with secure cluster
exist. Also, as the implementation of YARN applications is
a complex endeavor, it is typically not a rst order consid-
erations; YARN lacks the exibility and easy-of-use of other
schedulers such as Mesos [45] as it was primarily designed
for Hadoop applications. There are some attempts to integrate
these systems, e. g. Myriad provides the ability to dynamically
spawn YARN clusters inside Mesos [46]. Another concern is
security basic security mechanisms, such as Kerberos, are
available; however, in particular emerging frameworks, such
as Spark, have some issue interfacing with secure Hadoop
cluster. Further, the data management and governance remains
a manual process. Often, it is e. g. very difcult to assign
access rights to raw data as the real content of the data remains
opaque.
Another important consideration is the selection of a
Hadoop distribution trading off capabilities, performance and
vendor lock-in. Hadoop distributions bundle the core Hadoop
with components to simplify deployment, monitoring and
upgrade. In the optimal case, the Hadoop distribution should
provide tools that cover all layers of the architecture. The
most well-known distributions are Cloudera, Hortonworks and
MapR. Each distribution follows a philosophy of releases.
Cloudera typically adopts selected upstream features to their
stable version. Hortonworks follows closely the open source
release cycle. Also, many vendors substitute their own compo-
nents at different levels of the Hadoop ecosystem. MapR re-
places HDFS with their own distributed lesystem, promising
better performance and reliability. The participation of many
vendors in the Hadoop ecosystem has led to maturity, but
also to fragmentation. While the lower layers of Hadoop (i. e.
HDFS and YARN) have reached stability, vendors have moved
to higher layers for differentiation, such as SQL, machine
learning and security.
IV. E XPERIMENTAL E VALUATION
While a set of benchmarks for Big Data workloads is
emerging (e. g. TPC-DS [47], TPCx-HS [14]), often difcult to
compare results generated by different vendors are available.
These results are typically obtained using highly optimized
setups and carefully selected numbers. For the design and
sizing of our data lake it is critical to understand the trade-
off and optimization parameters and efforts for the different
frameworks. This is not possible by relying on published
benchmark results. Thus, we created a custom benchmark
suite based on two real-world datasets of an automotive
applications. We augment these results the TPCx-HS standard
benchmark Terasort to obtain a baseline performance of our
cluster. The application workloads we study are SQL and
a logistic regression and SVM for text mining. For these
experiments, we use our 48 node Hadoop data lake with 10
Gigabit Ethernet connectivity, Hortonworks HDP 2.2 with
Hadoop 2.6, Hive 0.14, Spark-SQL 1.3., Flink 0.9 and Tez
0.5.2.
A. Terasort
Figure 3 investigates the Terasort performance for different
Hadoop processing engines. We sort 1 TB of data using
32 nodes. All jobs are run via YARN, which introduces some
challenges: due to the memory footprint of YARN, not the
full memory is available for Spark; further, heterogeneous
nodes with different amounts of cores/memory are not op-
timally supported. We use the Terasort implementation in the
1207
2500
2000
Time (in sec)
1500
1000
500
0
Flink MapReduce Spark Tez
Framework
Flink MapReduce Spark Tez
Fig. 3. Terasort Benchmark for Flink, MapReduce, Tez and Spark: Run-
ning on YARN out of the box. Flink worse performance and resource usage.
Spark more memory usage; MapReduce more CPUs, Tez best performance
Hadoop examples for MapReduce and Tez, the Flink example
implementation and the un-optimized Spark implementation
provided by Min [48].
The results demonstrate the inherent trade-off between pro-
gramming model capabilities, easy-of-use and performance.
Both Flink and Spark provide a higher-level and a more
expressive programming model than MapReduce - while Spark
outperforms MapReduce, Flink show the worst performance
of all frameworks. Furthermore, it must be noted that the
performance improved signicantly from Spark 1.3 to 1.4
(>30 %). Nevertheless, the Tez implementation shows the
best performance. This is particularly interesting since Spark
showed the best performance in the Terasort Graysort bench-
mark in 2014 [49] - there are several potential explanations:
to ensure comparability and real-world applicability, we run
all scenarios through YARN, which is associate with some
overhead. The Spark/YARN integration is still evolving. In
particular, the startup times for Spark on YARN with a high
number of containers is signicant. Tez is much better opti-
mized for YARN. Finally, the environment for the benchmark
was different - the Graysort was run on EC2 instances with
SSDs, while we utilize disk-based storage.
B. SQL on Hadoop
SQL is one of the most important workloads that users cur-
rently use on data stored in Hadoop for ETL, data preparation
and simple analytics tasks. Our benchmark consists of several
large tables with up to 120 billion rows with an uncompressed
data size of 3.2 TB. The benchmark comprises ve queries. In
the following, we evaluate the suitability of Hive (with Tez)
and Spark-SQL using different storage formats: Text, Parquet,
and ORC. In both cases, we run the queries via the Hive
respectively the Spark-SQL shell and YARN. Data is loaded
from HDFS in both cases.
Figure 4 shows the results of the benchmark. We tested
a total of ve queries - in the following we particularly
investigate three of these: select, aggregate with groupby, and
join using a dataset consisting of 20 million vehicles and
 Select Aggregate Join Data Size (in GB) Number of Nodes
(32 Nodes) (8 GB Data)
10000 600

Time (in sec)

400

1000









Time (in sec)

200

100

0

1 2 4 8 16 8 16 32

10

Logistic Regression (SGD) SVM (SGD) Logistic Regression (LBFGS)

Feature Extraction Classification

Fig. 6. Scalability of Different MLLib Algorithms: Both the feature

4 8 16 32 4 8 16 32 4 8 16 32 extraction and classication scale linear. The performance of the training phase
depends on the used algorithm and optimizer: Both stochastic gradient descent
Number of Nodes based approaches (logistic regression and SVM) show a worse performance
Hive/Text Hive/ORC Hive/Parquet Spark/ORC Spark/Parquet than the L-BFGS optimizer that is available in Spark for logistic regression.

Fig. 4. Hadoop SQL Performance: The gure shows the scale-out perfor-
mance of our automotive SQL benchmark based on queries of datawarehouse
the join query Hive is in average 50 times faster than Spark-
system. SQL. We had to optimize the join query for Spark, because
of memory issues in particular for the four node scenario.
Also, it must be noted that window queries are not supported
by Spark at the moment (it is part of our benchmark, but not
6000 depicted in graph). The advantage of the data lake architecture
Time (in sec)

is that different technologies can be combined trading off

4000
2000
0 Hive and Spark to a commercial analytical database system
Select Aggregate
Query
RDBMS Hive
Fig. 5. Hive/Spark versus RDBMS: On a comparable infrastructure 4 nodes
with 128 GB memory vs. 1 nodes with 512 GB, we compare Hive/Spark to a
commercial RDBMS. Generally, the RDBMS can outperform Hive and Spark
however, both deliver a solid performance at a lower cost.
120 billion rows. For 20 million vehicles the columnar data
formats ORC and Parquet compress the data signicantly,
reducing the size from 3.2 TB for text to 1.3 TB for Parquet
and to 193 MB for ORC. For Hive, ORC exhibits a better
performance compared to the other formats; For Spark-SQL,
Parquet shows the best performance. It must be noted that
data conversion to Parquet and ORC is associated with some
overhead (e. g. it took e. g. 40 min to create an ORC table based
on the text le). As the results indicate this effort typically
pays-off.
We compare three queries: a select, aggregate and join
query. The select query does not require a reduce phase,
whereas both the aggregate and join require one. The shufe
trafc is the highest for the join query. Both Hive and
Spark (when comparing Hive/ORC and Spark/Parquet) deliver
a comparable performance for the select and join query;
nevertheless, Hive has a small advantage in most cases. For
capabilities and performance: while lacking in SQL support
and performance, Spark-SQL has advantages with respect to
integration with higher-level analytics and machine learning
frameworks, such as dataframes and MLlib.
In the next scenario we compare the performance of
Join
(RDBMS). Figure 5 shows the time for each system broken
down by query. We utilize a similar number of cores and
Spark
memory for all three systems: Hive/Spark are run on 4 nodes
with 12 cores and 128 GB memory each and the RDBMS is
run on a 40 core/512 GB system. We use a reduced dataset
consisting of 60 billion rows and Hive/ORC and Spark/Parquet
(50 % of the size for the previous scenarios). While the
RDBMS has an advantage as it runs in non-distributed mode,
it outperforms both Hive and Spark-SQL. Spark-SQL memory
usage and thus, the necessity to page-out data from the RDD
during the shufe phase causes the performance deterioration
during the join query. Nevertheless, both frameworks delivery
an acceptable performance considering that they are available
as open source tools and run on commodity hardware.
C. Advanced Analytics: Text Classication
Analytical challenges arise in particular from unstructured
data, such as text data. Machine learning approaches are
commonly used to address this challenge. Scalable approaches
for machine learning are necessary to support large volume
and high dimensional data. Text classication is an example
of a machine learning problem and describes the process of
grouping documents into categories. In this project we utilize
Support Vector Machines (SVM) and Logistic Regression
(LR) to classify verbatim data from customer surveys. Our
analysis particularly focuses on the MLlib implementation of

1208
these machine learning algorithms inside of Spark. For this
purpose, we utilize a synthetic dataset of up to 110 million
records (16 GB scenario).
Figure 6 shows the results of this benchmark. A scalable
machine learning implementation is the key for dealing with
high-dimensional data and large volumes (larger number of
rows and columns). The application uses TF-IDF for feature
extraction and a SVM and logistic regression for classication.
As shown in the gure, TF-IDF scales linearly, while the
machine learning algorithms exhibit an increased overhead
with larger data sizes due to the higher number of columns
in the TF-IDF matrix and the need for synchronization after
each iteration. Both learning algorithms SVM and logistic
regression perform similarly when using stochastic gradient
descent (SGD) as solver. A signicant speedup can be ob-
served by replacing the SGD with the Limited-memory Broy-
denFletcherGoldfarbShanno (LBFGS) solver provided by
MLlib for Logistic Regressing - both solvers showed a com-
parable accuracy in the results. The results demonstrate the
scalability of Spark MLlib compared to other machine learning
frameworks our initial version of this SVM classier was
built in R and was constrained to about 100,000 records.
V. C ONCLUSION AND F UTURE W ORK
Hadoop enables enterprises to handle the different Vs
of Big Data - in particular the data volume and velocity
requirement are manageable with a Hadoop cluster. With this
capability, Hadoop lls a gap in many existing enterprise
landscapes - most commercial database vendors have realized
that und started to support Hadoop. However, we believe that
the data lake has the potential to accommodate many more
workloads as the Hadoop ecosystem evolves. This is most
visible in the different SQL on Hadoop frameworks, which
improved signicantly over the past years as seen in the
presented benchmarks. Due to these developments, analytics
on large datasets is becoming the norm. A challenge remains
the productivity and the ability to generate insight from data.
Scalable machine learning approaches are essential to extract
knowledge from data. Intelligent, data-driven services are the
foundation of future innovations. A single vertical vendor and
closed stack cannot provide the capabilities needed to fulll
all requirements. Hadoop provides efcient and economical
storage in conjunction with a vast set of processing and ana-
lytics frameworks enabling a wide range of novel automotive
applications. In comparison to traditional data warehouses, it
provides greater agility supporting relational data, unstructured
data and schema-on-read data organization. The vibrant open
source ecosystem functions as a supplier for capabilities that
are beyond the scope of a single vertically integrated system.
A Hadoop data lake is suitable for supporting typical
automotive application workloads, such as SQL and machine
learning applications. In our benchmarks we showed interac-
tive response time for SQL-based data warehouse workloads
on 120 billion records, while also illustrating the weaknesses
of Hadoop SQL frameworks, such as the processing of join
1209
queries. Furthermore, we showed the potential of Hadoop-
based machine learning frameworks, such as Sparks MLlib;
using this tool, we were able to process a signicantly larger
dataset of high-dimensional text data consisting of 110 million
records than with our previous R solution. However, several
challenges remain: the maturity of the platform needs to
increase to meet the requirements of the increasing number of
applications and users. Many available tools are low-level and
often immature and requiring highly advanced skills to operate
them. Finding and developing these skills and establishing
a culture and analytical mindset for supporting Big Data
services remains challenging. Higher-level tools are emerging
and hopefully can address this issue.
With the uptake of the data lake infrastructure, high stan-
dards for data governance and the security are required. Fur-
ther, sophisticated workload management systems for resource
and data provisioning are needed to support the exible and
agile infrastructure usage required by data scientists. The
complexity of analytical workloads will further increase by
including, e. g. more unstructured data (images, videos) and
spatial data critical for location-based services. Many IoT
applications require close to realtime processing of incoming
data feeds. Emerging machine learning algorithms, such as
deep learning, are increasingly complex and demand even
more memory and compute resources. An area of research
is the use of hybrid query engines and the support for
analytics across data residing on different platforms. While
the current state of the art in analytics is the usage of hand-
crafted features and supervised learning, we believe that more
advanced analytical methods are required to handle the data
deluge, e. g. topic modeling and deep learning.
ACKNOWLEDGMENTS
We acknowledge our colleagues at BMW and Clemson for
providing valuable input and discussion to the paper: Matthew
Cook, Sandeep Jeereeddy, Linh Ngo, and Jason Anderson.
This work was supported in part by NSF Grant #1228312.
R EFERENCES
[1] D. Laney, 3D data management: Controlling data volume, velocity, and
variety, META Group, Tech. Rep., February 2001.
[2] J. Dorsey, Big data in the drivers seat of con-
nected car technological advances, http://press.
ihs.com/press-release/country-industry-forecasting/
big-data-drivers-seat-connected-car-technological-advance, 2013.
[3] Apache hadoop, http://hadoop.apache.org/, 2014.
[4] J. Dean and S. Ghemawat, MapReduce: Simplied Data Processing
on Large Clusters, in OSDI04: Proceedings of the 6th conference on
Symposium on Opearting Systems Design & Implementation. Berkeley,
CA, USA: USENIX Association, 2004, pp. 137150.
[5] Apache Flume, https://ume.apache.org/.
[6] Jay Kreps and Neha Narkhede and Jun Rao, Kafka: a distributed
messaging system for log processing, in 6th International Workshop
on Networking Meets Databases, Athens, Greece, 2011.
[7] M. Zaharia, M. Chowdhury, T. Das, A. Dave, J. Ma, M. McCauley,
M. J. Franklin, S. Shenker, and I. Stoica, Resilient distributed
datasets: A fault-tolerant abstraction for in-memory cluster computing,
in Proceedings of the 9th USENIX Conference on Networked
Systems Design and Implementation, ser. NSDI12. Berkeley, CA,
USA: USENIX Association, 2012, pp. 22. [Online]. Available:
http://dl.acm.org/citation.cfm?id=2228298.2228301
 [8] A. Alexandrov, R. Bergmann, S. Ewen, J.-C. Freytag, F. Hueske,
A. Heise, O. Kao, M. Leich, U. Leser, V. Markl, F. Naumann, M. Peters,
A. Rheinlnder, M. J. Sax, S. Schelter, M. Hger, K. Tzoumas, and
D. Warneke, The stratosphere platform for big data analytics, The
VLDB Journal, vol. 23, no. 6, pp. 939964, Dec. 2014. [Online].
Available: http://dx.doi.org/10.1007/s00778-014-0357-y
[9] A. Thusoo, J. S. Sarma, N. Jain, Z. Shao, P. Chakka, S. Anthony,
H. Liu, P. Wyckoff, and R. Murthy, Hive: A warehousing
solution over a map-reduce framework, Proc. VLDB Endow.,
vol. 2, no. 2, pp. 16261629, Aug. 2009. [Online]. Available:
http://dx.doi.org/10.14778/1687553.1687609
[10] S. Jha, J. Qiu, A. Luckow, P. K. Mantha, and G. C. Fox, A tale of
two data-intensive paradigms: Applications, abstractions, and architec-
tures, Proceedings of 3rd IEEE Internation Congress of Big Data, vol.
abs/1403.1528, 2014.
[11] Hpc-abds kaleidoscope of 350 apache big data stack and hpc tecnolo-
gies, http://hpc-abds.org/kaleidoscope/, May 2015.
[12] C. Baru, M. Bhandarkar, C. Curino, M. Danisch, M. Frank, B. Gowda,
H.-A. Jacobsen, H. Jie, D. Kumar, R. Nambiar, M. Poess, F. Raab,
T. Rabl, N. Ravi, K. Sachs, S. Sen, L. Yi, and C. Youn, Discussion
of BigBench: A Proposed Industry Standard Performance Benchmark
for Big Data, in Sixth TPC Technology Conference on Performance
Evaluation Benchmarking, ser. LNCS. Springer Berlin Heidelberg,
2014.
[13] G. C. Fox, S. Jha, J. Qiu, S. Ekanazake, and A. Luckow, Towards a
Comprehensive Set of Big Data Benchmarks, 2015.
[14] R. Nambiar, M. Poess, A. Dey, P. Cao, T. Magdon-Ismail, D. Qi Ren,
and A. Bond, Introducing tpcx-hs: The rst industry standard for
benchmarking big data systems, in Performance Characterization
and Benchmarking. Traditional to Big Data, ser. Lecture Notes
in Computer Science, R. Nambiar and M. Poess, Eds. Springer
International Publishing, 2015, vol. 8904, pp. 112. [Online]. Available:
http://dx.doi.org/10.1007/978-3-319-15350-6_1
[15] Apache tez, http://hortonworks.com/hadoop/tez/, 2013.
[16] X. Meng, J. K. Bradley, B. Yavuz, E. R. Sparks, S. Venkataraman,
D. Liu, J. Freeman, D. B. Tsai, M. Amde, S. Owen, D. Xin, R. Xin,
M. J. Franklin, R. Zadeh, M. Zaharia, and A. Talwalkar, Mllib:
Machine learning in apache spark, CoRR, vol. abs/1505.06807, 2015.
[Online]. Available: http://arxiv.org/abs/1505.06807
[17] J. W. Anderson, K. E. Kennedy, L. B. Ngo, A. Luckow, and A. W. Apon,
Synthetic data generation at scale with hadoop, in IEEE International
Conference on Big Data, 2014.
[18] Thilo Koslowski, How technology is ending the automotive industrys
century-old business model, Gartner Research, 2012.
[19] K. Fehrenbacher, Cloudera gets proactive with hadoop management,
GigaOM http://bit.ly/bmw-10m-connected-cars, 2013.
[20] S. A. Fayazi, A. Vahidi, G. Mahler, and A. Winckler, Trafc signal
phase and timing estimation from low-frequency transit bus data, under
review, 2014.
[21] A. D. Angelica, Googles self-driving car gath-
ers nearly 1 GB/sec, http://www.kurzweilai.net/
googles-self-driving-car-gathers-nearly-1-gbsec, 2013.
[22] J. Bruner, The industrial internet the machines are talking, http:
//radar.oreilly.com/2013/03/industrial-internet-report.html, 2013.
[23] Y. Bengio, I. J. Goodfellow, and A. Courville, Deep learning,
2015, book in preparation for MIT Press. [Online]. Available:
http://www.iro.umontreal.ca/~bengioy/dlbook
[24] V. K. Vavilapalli, Apache Hadoop YARN: Yet Another Resource
Negotiator, in Proc. SOCC, 2013.
[25] B. Zhang, Y. Ruan, and J. Qiu, Harp: Collective communication on
hadoop, in "Technical Report Indiana University", 2014.
[26] N. Marz, Big data : principles and best practices of scalable realtime
data systems. Manning Publishing, 2014.
[27] Twitter, Storm: Distributed and fault-tolerant realtime computation,
http://storm-project.net/.
[28] S. Melnik, A. Gubarev, J. J. Long, G. Romer, S. Shivakumar, M. Tolton,
and T. Vassilakis, Dremel: Interactive analysis of web-scale datasets,
in Proc. of the 36th Intl Conf on Very Large Data Bases, 2010, pp.
330339. [Online]. Available: http://www.vldb2010.org/accept.htm
[29] M. A. Soliman, L. Antova, V. Raghavan, A. El-Helw, Z. Gu, E. Shen,
G. C. Caragea, C. Garcia-Alvarado, F. Rahman, M. Petropoulos,
F. Waas, S. Narayanan, K. Krikellas, and R. Baldwin, Orca: A
modular query optimizer architecture for big data, in Proceedings of the
2014 ACM SIGMOD International Conference on Management of Data,
1210
ser. SIGMOD 14. New York, NY, USA: ACM, 2014, pp. 337348.
[Online]. Available: http://doi.acm.org/10.1145/2588555.2595637
[30] M. Kornacker, A. Behm, V. Bittorf, T. Bobrovytsky, C. Ching,
A. Choi, J. Erickson, M. Grund, D. Hecht, M. Jacobs, I. Joshi,
L. Kuff, D. Kumar, A. Leblang, N. Li, I. Pandis, H. Robinson,
D. Rorke, S. Rus, J. Russell, D. Tsirogiannis, S. Wanderman-Milne,
and M. Yoder, Impala: A modern, open-source sql engine for
hadoop. in CIDR. www.cidrdb.org, 2015. [Online]. Available: http:
//dblp.uni-trier.de/db/conf/cidr/cidr2015.html#KornackerBBBCCE15
[31] M. Armbrust, R. S. Xin, C. Lian, Y. Huai, D. Liu, J. K. Bradley,
X. Meng, T. Kaftan, M. J. Franklin, A. Ghodsi, and M. Zaharia,
Spark SQL: relational data processing in spark, in Proceedings of
the 2015 ACM SIGMOD International Conference on Management of
Data, Melbourne, Victoria, Australia, May 31 - June 4, 2015, T. Sellis,
S. B. Davidson, and Z. G. Ives, Eds. ACM, 2015, pp. 13831394.
[Online]. Available: http://doi.acm.org/10.1145/2723372.2742797
[32] Optimized Row Columnar (ORC) Format, http://docs.hortonworks.
com/HDPDocuments/HDP2/HDP-2.0.0.2/ds_Hive/orcle.html, 2014.
[33] Parquet Columnar Storage Format, http://parquet.io/, 2014.
[34] R Core Team, R: A Language and Environment for Statistical
Computing, R Foundation for Statistical Computing, Vienna, Austria,
2013, ISBN 3-900051-07-0. [Online]. Available: http://www.R-project.
org/
[35] Pandas: Python Data Analysis Library, 2015. [Online]. Available:
http://pandas.pydata.org/
[36] S. van der Walt, S. Colbert, and G. Varoquaux, The numpy array: A
structure for efcient numerical computation, Computing in Science
Engineering, vol. 13, no. 2, pp. 2230, March 2011.
[37] GraphLab Create User Guide, 2015. [Online]. Available: https:
//dato.com/learn/userguide/
[38] S. Sridharan and J. M. Patel, Proling r on a contemporary processor,
Proc. VLDB Endow., vol. 8, no. 2, pp. 173184, Oct. 2014. [Online].
Available: http://dl.acm.org/citation.cfm?id=2735471.2735478
[39] Scalapack scalable linear algebra package, http://www.netlib.org/
scalapack/, 2014.
[40] Arpack, http://www.caam.rice.edu/software/ARPACK/, 2014.
[41] G. Ostrouchov, W.-C. Chen, D. Schmidt, and P. Patel, Programming
with big data in r, 2012, http://r-pbd.org/.
[42] Distributing the singular value decomposition
with spark, https://databricks.com/blog/2014/07/21/
distributing-the-singular-value-decomposition-with-spark.html, 2014.
[43] H2O Scalable Machine Learning, 2015. [Online]. Available:
http://h2o.ai/
[44] J. M. Hellerstein, C. R, F. Schoppmann, D. Z. Wang, E. Fratkin,
A. Gorajek, K. S. Ng, C. Welton, X. Feng, K. Li, and A. Kumar,
The madlib analytics library: or mad skills, the sql, Proc. VLDB
Endow., vol. 5, no. 12, pp. 17001711, Aug. 2012. [Online]. Available:
http://dl.acm.org/citation.cfm?id=2367502.2367510
[45] B. Hindman, A. Konwinski, M. Zaharia, A. Ghodsi, A. D. Joseph,
R. Katz, S. Shenker, and I. Stoica, Mesos: a platform for ne-grained
resource sharing in the data center, in Proceedings of the 8th USENIX
conference on Networked systems design and implementation, ser.
NSDI11. Berkeley, CA, USA: USENIX Association, 2011, pp. 2222.
[Online]. Available: http://dl.acm.org/citation.cfm?id=1972457.1972488
[46] M. Soni and R. DelValle, Myriad: Running yarn
alongside mesos, 2014, https://speakerdeck.com/mohit/
running-yarn-alongside-mesos-mesoscon-2014.
[47] TPC-DS, TPC Benchmark (TPC-DS): The New Decision Support
Benchmark Standard, http://www.tpc.org/tpcds/, 2015.
[48] D. Y. Min, Spark terasort, https://github.com/DrakeMin/spark-terasort,
2015.
[49] R. Xin, P. Deyhim, A. Ghodsi, X. Meng, and M. Zaharia,
Graysort on apache spark by databricks, http://sortbenchmark.org/
ApacheSpark2014.pdf, 2014.