Select Language ▼

Translation Disclaimer (/termsofuse.html#translation)

Search this site Search

(https://www.datanami.com)

Home (https://www.datanami.com/) About (https://www.datanami.com/about/) Follow Datanami: 

Resources (https://www.datanami.com/whitepaper/) Events (https://www.datanami.com/events/) (https://www.facebook.com/pages/Datanami/124760547631010)

Subscribe (https://www.datanami.com/subscribe/)  (https://www.twitter.com/datanami) 

(https://www.linkedin.com/groups/Big-Data-News-

Network-4166980) 
(https://www.datanami.com/feed/)

HOME (HTTPS://WWW.DATANAMI.COM/) FEATURES SECTORS APPLICATIONS TECHNOLOGIES VENDORS

JOB BANK (HTTPS://WWW.DATANAMI.COM/JOB-BANK/) EVENTS

December 13, 2018 This Just In Most Read

Top
2018: A Big Data Year in Review
Stories July 11, 2019
Alex Woodie
On Alluxio Delivers First Data Orchestration
As 2018 comes to a close, it’s worth taking some time Platform Powering Multi-Cloud Analytics and AI
to look back on the major events that occurred this (https://www.datanami.com/this-just-in/alluxio-
year in the big data, data science, and AI space. delivers-first-data-orchestration-platform-
There’s a lot that occurred – and a lot that we powering-multi-cloud-analytics-and-ai/)
chronicled in the virtual pages of Datanami. We hope Talend Announces Availability of Stitch Data
you enjoy this retrospective as you prepare for 2019. Loader on AWS Marketplace
(https://www.datanami.com/this-just-in/talend-
Data security continued to be a major topic in 2018
announces-availability-of-stitch-data-loader-on-
(https://www.datanami.com/2018/04/18/creating-a-
(gguy/Shutterstock) aws-marketplace/)
modern-information-security-foundation/), particularly
Iron Mountain Expands Data Services to Support
as the rash of big data breaches continued
(https://www.datanami.com/2018/12/11/data-breaches-and-abuses-abound-but-regulatory-
appetite-appears-lacking/). The IT world had a rude awakening following the 2018 New Year’s
celebration when the Spectre and Meltdown vulnerabilities were discovered in practically every
computer processor (https://www.datanami.com/2018/01/08/meltdown-spectre-will-impact-
analytics/) on the planet. Apparently, chipmakers took some shortcuts in developing the
speculative execution methods that boost multi-threaded performance in modern CPUs. Failure to
apply the patches (which robbed the chips of performance) put vulnerable data at risk.

The sophistication of machine learning automation

tools increased a good deal during 2018, which is
not surprising. Data scientists who are looking to
boost their own productivity
(https://www.datanami.com/2018/05/09/the-10-
step-guide-to-mastering-machine-learning/) – or
data analysts and power users who wanted to swing
above their weight – had a smorgasbord of ML
automation tools available from a raft of vendors like
(https://2s7gjr373w3x22jf92z99mgm5w- Visit additional Tabor Communications publications
Cloudera, DataRobot, H2O.ai, Anaconda, Domino
wpengine.netdna-ssl.com/wp-
Data Labs, IBM, SAS, and Alteryx – not to mention
content/uploads/2018/01/meltdown_shutterstock_Jaiz-
cloud offerings from AWS
Anuar.jpg)
(https://www.datanami.com/2018/02/28/aws-
looks-demystify-machine-learning/), Azure, and GCP (Jaiz Anuar/Shutterstock) (https://hpcwire.com/)
(http://www.enterpriseai.news/)
(http://www.hpcwire.jp/)
or open source kits
(https://www.datanami.com/2018/08/30/new-open-
source-projects-emerge-for-machine-learning/), including those for Python, R, Java, and Scala. Sponsored Whitepapers
Data governance isn’t a new concept, but it sometimes seemed that way this year, particularly
Wide Temperature for Industrial Memory
with the General Data Protection Regulation (https://www.datanami.com/2018/05/24/data-
Applications
regulation-d-day-arrives-final-thoughts-on-gdpr/) (GDPR) threatening big sanctions on companies (https://tci.taborcommunications.com/l/21812/2019-
that were careless with data. The growing concerns around data security – along with the 06-28/6jktlv)
difficulty data science teams were facing in just corralling and making sense of data – combined
to put data governance on the map (https://www.datanami.com/2018/01/24/techs-hottest-new-
The Big Data Transformation
trend-data-governance/) in a big way. (https://2s7gjr373w3x22jf92z99mgm5w-wpengine.netdna- (https://tci.taborcommunications.com/l/21812/2019-
ssl.com/wp-content/uploads/2016/12/AWS_amazon.jpg) 05-14/6j9nqg)
 The rapid growth of the cloud as a platform for storing View the Whitepaper Library
big data and analyzing it (https://www.datanami.com/whitepaper/)
(https://www.datanami.com/2018/01/22/cloud-looms-
answer-things-analytic/) was a huge story for 2018. At
the end of the year, AWS had a $27 billion yearly run rate,
and was growing at 46% annually. Azure and GCP were Sponsored Multimedia
growing even faster, although they weren’t even close to
matching AWS in the revenue department. Nobody was

‹ ›
surprised when AWS unveiled a slew of new ML functionality
(https://www.datanami.com/2018/11/28/aws-bolsters-machine-learning-services-at-reinvent/) at
its annual re:Invent show at the end of the year

Let’s face it: Big data, as a defining name and concept, is on its last legs. The new paradigm that’s
emerging is being defined with an old-ish phrase, artificial intelligence – but one that’s being
(http://tci.taborcommunications.com/l/21812
(http://tci.taborcommu
infused with new meaning. As the big data bubble shrinks, AI’s just keeps getting bigger. The
03-05/67p7rm) 11-14/65lrgg)
smart money folks on Sand Hill Road poured money into AI startups
(https://www.datanami.com/2018/01/19/vc-firms-invest-heavily-ai-pushing-hype-bubble-bigger/') Real-Time Improving
at a rapid clip in 2018, and there’s no sign that it’s abating. Connected Manufacturing
(https://2s7gjr373w3x22jf92z99mgm5w- Customer Quality and Asset
wpengine.netdna-ssl.com/wp-
Experiences – Performance with
Easier Than You Industrial Internet of
Think Things
(http://tci.taborcommunications.com/l/21812
(http://tci.taborcommu
03-05/67p7rm) 11-14/65lrgg)

content/uploads/2017/10/GDPR_shutterstock_Pe3k.jpg)

The shortage of data scientists has been talked about for years. But as technology evolved,
organizations found that what they really needed was more data engineers
(https://www.datanami.com/2018/09/03/engineers-vs-scientists-who-to-hire-in-data-now/).
Those folks who are skilled at using modern tools like Apache Spark to manipulate data and build
data pipelines for machine learning and advanced analytics use cases are in even hotter demand
(https://www.datanami.com/2018/02/05/2018-will-year-data-engineer/) that data scientists.

As the early months of 2018 rolled on, the May 25 deadline for GDPR compliance loomed large for
chief data officers. Some worried that the new European Union law would kill artificial intelligence
(https://www.datanami.com/2018/02/27/gdpr-artificial-intelligence-killer/), while others thought
that it could be beneficial by forcing companies to adopt
(https://www.datanami.com/2018/04/16/the-enlightening-side-of-gdpr-compliance/) good data
management practices.

If 2017 gave us a small taste of blockchain technology

and new crypto currencies like Bitcoin, then 2018 blew
out the doors with a seven-course meal. The possibility
of having a secure distributed ledger of transactions
triggered the imaginations of technologists
(https://www.datanami.com/2018/04/09/blockchain-
starting-to-feel-its-way-into-the-artificial-intelligence-
ecosystem/) in all sorts of ways, including making the
insurance industry
(https://www.datanami.com/2018/06/28/how-
(https://2s7gjr373w3x22jf92z99mgm5w- blockchain-is-impacting-data-and-processes-in-
wpengine.netdna-ssl.com/wp- insurance/) more efficient, and even helping streamline
ML and AI processes
content/uploads/2018/04/blockchain_shutterstock_dencg.jpg)
(https://www.datanami.com/2018/06/13/can-
(dencg/Shutterstock) blockchain-help-ml-and-ai/), not to mention managing
big data
(https://www.datanami.com/2018/06/28/blockchain-solutions-usher-in-era-of-trusted-big-data/).
However, as 2018 draws to a close, most of the blockchain promises have been unfulfilled.

The fortunes of Nvidia (https://www.nvidia.com) continued to soar in 2018 as demand for its
GPUs skyrocketed due to expanding deep learning and high performance computing (HPC)
workloads. Nvidia CEO Jensen Huang cemented his reputation as a high tech rock star
(https://www.datanami.com/2018/03/29/reporters-notebook-the-2018-gtc-experience/) during
Contributors
his two-and-a-half hour keynote at the GPU Technology Conference in March. While the
company’s stock was down at year’s end (along with the rest of the stock market) there’s no signs
of a slowdown for Nvidia. Alex Woodie
Editor in Chief
Cloud data warehouses had a giant year as
companies warmed up to the prospect of
adopting ready-to-use analytic databases that
can analyze petabytes of data in milliseconds George Steve Conway
Leopold
Hyperion
Contributing Research
Editor

Tiffany Trader

(https://2s7gjr373w3x22jf92z99mgm5w- Contributing Editor

wpengine.netdna-ssl.com/wp-
 content/uploads/2016/04/Jen-Shun-
Huang.png)

Nvidia CEO Jensen Huang

(https://www.datanami.com/2018/04/24/googles-bigquery-gaining-steam-as-cloud-warehouse-
wars-heat-up/). AWS RedShift reportedly has 6,500 customers, while the amount of data
customers had stored in BigQuery doubled from 2016 to 2017. Snowflake Computing, meanwhile,
Featured Events
landed a mammoth $450 million funding (https://www.datanami.com/this-just-in/snowflake-
Strata Data Conference
closes-450-million-in-growth-funding-to-accelerate-u-s-and-global-expansion/) round, and cloud New York 2019
analytics will never be the same. (https://www.datanami.com/event/s
OSCON 2019 data-conference-new-
Dogged by technological complexity and the rise of big data environments in the cloud, the york-2019/)
(https://www.datanami.com/event/oscon-
Hadoop market continued to disintegrate. The surprise merger of Hortonworks and Cloudera 2019/)
(https://www.datanami.com/2018/10/03/cloudera-and-hortonworks-to-merge/) emphasized the
smaller role that Hadoop is playing in the enterprise as other alternatives continue to evolve. The
merger led some to speculate that Hadoop is officially dead
(https://www.datanami.com/2018/10/03/cloudera-and-hortonworks-to-merge/), but others point September 23 -
out that clouds can’t deliver hybrid on-premise and multi-cloud data management strategies, July 15 - July 18 September 26
which (not coincidentally) is the new Cloudera’s focus Portland OR United States New York United States
(https://www.datanami.com/2018/10/24/new-cloudera-plots-a-course-toward-a-unified-future/). Data Leaders Summit
Europe 2019
Tabor Communications held its inaugural Advanced Scale Forum (https://www.datanami.com/event/data-
(https://www.advancedscaleforum.com/) (ASF) event this May in Austin, Texas. The event leaders-summit-europe-
brought together leaders in HPC, big data (https://www.datanami.com/2018/05/09/emerging- 2019/)
advanced-scale-tech-trends-focus-of-annual-tabor-conference/), and AI, and featured a number of
speakers from leading companies like Gulfstream, Moonshot Research, Accenture, Comcast,
Nimbix, The BioTeam, Dell EMC, and others. TCI also held its first HPC on Wall Street one-day
seminar in September, capping a successful year of on-site events. Planning is underway for the
October 23 - October 24
second ASF, which will be back in Florida next April.
Berlin Germany
(https://2s7gjr373w3x22jf92z99mgm5w-
wpengine.netdna-ssl.com/wp-
View More… (https://www.datanami.com/events/)
content/uploads/2018/03/kubernetes.png)

Driven by the rise of cloud container architectures,

Kubernetes emerged as the new “it” technology in
2018. The rush was on to retrofit existing systems
(https://www.datanami.com/2018/05/03/want-kafka-
on-kubernetes-confluent-has-it-made/), from Hadoop
and Kafka, to adopt Kubernetes for cluster scheduling
and management. Much of the activity at the fall
Strata Data Conference revolved around vendors expressing support for Kubernetes, which looks
set to supplant YARN in next-gen big data cluster architectures.

Fueled by demand to train machine learning models on ever-bigger collections of data – not to
mention Nvidia’s meteoric rise — 2018 saw a surge of activity in new silicon
(https://www.datanami.com/2018/05/15/a-wave-of-purpose-built-ai-hardware-is-building/)
development. Google led the way with its tensor processing units
(https://www.datanami.com/2018/05/21/google-unleashes-cloud-ml-engine/) (TPUs), while
startups like Wave Computing, Groq, Flex Logix, and Graphcore explored other types of
processors, including GPUs, FPGAs, and ASICs.

As more companies adopted neural networks and deep learning techniques to automate decision
making, it also raised tought questions about exactly how it works
(https://www.datanami.com/2018/08/15/bright-skies-black-boxes-and-ai/).  AI’s explainability
problem (https://www.datanami.com/2018/07/06/ai-youve-got-some-explaining-to-do/), as its
been called, has bolstered technologists to find solutions. One of the first was the LIME project,
which emerged from Unviersity of California. But firms like FICO and SAS discovered that simply
exercising the models (https://www.datanami.com/2018/05/30/opening-up-black-boxes-with-
explainable-ai/) and reporting their actions could be the way forward.

The improvement in machine learning automation

and the shortage in classically trained data scientists
combined to turbo-charge the rise of the citizen data
science
(https://www.datanami.com/2018/05/29/citizen-
data-scientists-take-on-bigger-roles/) movement in
2018. The trend has the backing of Gartner, which
predicts that citizen data science positions will grow
five times faster

(https://2s7gjr373w3x22jf92z99mgm5w-
wpengine.netdna-ssl.com/wp-
content/uploads/2018/05/black-
box_shutterstock_amasterphotographer.jpg)

(amasterphotographer/Shutterstock)

(https://www.datanami.com/2018/08/13/empowering-citizen-data-science/) than jobs for full-

fledged data scientists. Being a citizen never felt so good!
As more companies adopt machine learning techniques to automate decision making, the thorny
issue of AI ethics keep cropping up. Plenty of abuses of data
(https://www.datanami.com/2018/10/16/do-amazons-biased-algorithms-spell-the-end-of-ai-in-
hiring/) and AI have been documented, which led some to observe that the data science
community needs to take charge and build ethics
(https://www.datanami.com/2018/06/22/ethical-data-science-is-good-data-science/) into their
work. It’s also renewed calls to find ways to use big data tech to actually help people
(https://www.datanami.com/2018/08/08/the-humanity-in-artificial-intelligence/), as opposed to
just making more money.

Despite all the talk about getting ahead of the competition with AI and big data science, the fact
remains that few organizations are actually having rip-roaring success with it yet, and the use
cases remain narrow (https://www.datanami.com/2018/10/03/deep-learning-is-great-but-use-
cases-remain-narrow/). Study after study show that, outside of tech giants, Web startups, and
Fortune 100 firms with billion-dollar IT budgets, there just aren’t that many examples
(https://www.datanami.com/2018/08/07/its-still-early-days-for-machine-learning-adoption/) of
companies hitting it out of the ballbark. The sage advice for succeeding at AI
(https://www.datanami.com/2018/11/01/focus-on-business-processes-not-big-data-
technology/) is the same as it was for big data: start small and grow smartly
(https://www.datanami.com/2018/06/11/want-to-win-at-ai-aim-high-but-start-small/).

It was good year for startups raising money.

Among those closing rounds were Dremio ($25
million Series B

(https://2s7gjr373w3x22jf92z99mgm5w-
wpengine.netdna-ssl.com/wp-
content/uploads/2017/01/shutterstock_engineer.jpg)

Demand for citizen data scientists is growing

(Dean Drobot/Shutterstock)

(https://www.datanami.com/2018/01/23/dremio-accelerates-growth-plans-following-25m-series-
b/)); InfluxData ($35 million Series C (https://www.datanami.com/2018/02/13/time-series-leader-
influxdata-raises-cash/)); Scality ($60 million Series E
(https://www.datanami.com/2018/04/18/scality-adds-new-investor-in-latest-funding-round/));
ThoughtSpot ($145 million Series D (https://www.datanami.com/2018/05/08/thoughtspot-eyes-
exit-with-latest-funding-round/)); MemSQL ($30 million Series D
(https://www.datanami.com/2018/05/16/memsql-gains-traction-new-investors/)); Immuta ($20
million Series B (https://www.datanami.com/2018/06/20/immuta-cashes-in-on-data-privacy-
scramble/)); CrateDB ($11 million Series A (https://www.datanami.com/2018/06/19/cratedb-
raises-cash-upgrades-machine-data-platform/)); Tamr ($18 million Series D
(https://www.datanami.com/2018/07/11/tamr-preps-for-growth-with-18m-round/)); Cloudian
($94 million Series E (https://www.datanami.com/2018/08/29/cloudian-sees-94m-for-object-
funding/)); StreamSets ($24 million Series C
(https://www.datanami.com/2018/09/11/streamsets-gets-35m-for-dataops/)); DataRobot ($100
million Series D (https://www.datanami.com/2018/10/25/hot-datarobot-raises-a-bundle/)); and
Looker ($100 million Series E (https://www.datanami.com/2018/12/06/looker-announces-vc-
haul-to-fund-platform-push/)).

We also saw some companies going public in this sector. Among those having IPOs were: Pivotal
(raised $555 million (https://www.datanami.com/2018/04/20/pivotal-software-is-latest-tech-
ipo/)) and Elastic (raised $252 million (https://www.datanami.com/2018/10/05/elastic-ipo-
expected-to-raise-250m/)). The acquisitions desk was also busy.  Infogix bought LavaStorm
(https://www.datanami.com/2018/03/26/infogix-acquires-data-prepper-lavastorm/) in March;
Cloudian bought Infinity Storage (https://www.datanami.com/2018/03/15/cloudian-buys-file-
storage-pioneer/) in March; Actian was bought by equity firm
(https://www.datanami.com/2018/04/12/hcl-equity-investor-acquire-actian/) HCL in April;
 Microsoft buys Bonsai (https://www.datanami.com/2018/06/21/microsoft-acquires-ai-
specialist-bonsai/) in June; Qlik bought Podium Data
(https://www.datanami.com/2018/07/24/qlik-acquires-data-manager/) in July; MariaDB bought
Clustrix (https://www.datanami.com/2018/09/20/with-eye-on-oracle-mariadb-buys-clustrix/) in
September; Talend bought Stitch (https://www.datanami.com/2018/11/07/21615/) in November;
Cloudera and Hortonworks agreed to merge (https://www.datanami.com/2018/10/04/reaction-
to-hortonworks-cloudera-mega-merger/) in October; and IBM agreed to buy RedHat
(https://www.datanami.com/2018/10/29/what-does-ibms-acquisition-of-red-hat-mean-for-open-
source/) in October.

We saw some big changes in strategy for key players. DataTorrent threw in the towel
(https://www.datanami.com/2018/05/08/datatorrent-stream-processing-startup-folds/) with its
streaming data platform. Teradata announced it would no longer focus on analytics, but providing
answers (https://www.datanami.com/2018/10/09/new-teradata-focuses-on-answers-not-
analytics/). Cloudera announced some strategic shifts
(https://www.datanami.com/2018/10/24/new-cloudera-plots-a-course-toward-a-unified-future/)
 following its announced merger with
Hortonworks; Chris Lynch promised to scale
AtScale after being named CEO in June
(https://www.datanami.com/2018/06/27/former-
vertica-ceo-takes-helm-at-atscale/).

2018 is almost over, but it’s not too late to make

news. If you have a hot tip, please share it by
contacting us
(https://www.datanami.com/about/#contact).
(https://2s7gjr373w3x22jf92z99mgm5w-
Related Items:
wpengine.netdna-ssl.com/wp-
2018 Predictions: Opening the Big Data content/uploads/2018/04/Screen-Shot-2018-
Floodgates 04-20-at-11.59.26-AM.png)
(https://www.datanami.com/2017/12/19/2018- (iQoncept/Shutterstock)
predictions-opening-big-data-floodgates/)

What 2017 Will Bring: 10 More Big Data Predictions

(https://www.datanami.com/2017/01/04/2017-more-big-data-predictions/)

2016: A Big Data Year in Review (https://www.datanami.com/2016/12/19/2016-big-data-year-

review/)

Share this:

 (https://www.datanami.com/2018/12/13/2018-a-big-data-year-in-review/?
share=twitter)

 (https://www.datanami.com/2018/12/13/2018-a-big-data-year-in-review/?
share=facebook)

 (https://www.datanami.com/2018/12/13/2018-a-big-data-year-in-review/?
share=linkedin)

 (https://www.datanami.com/2018/12/13/2018-a-big-data-year-in-review/?
share=reddit)

 (https://www.datanami.com/2018/12/13/2018-a-big-data-year-in-review/?
share=email)

Tags: AI (https://www.datanami.com/tag/ai/), analytics

(https://www.datanami.com/tag/analytics/), big data (https://www.datanami.com/tag/big-
data/), cloud (https://www.datanami.com/tag/cloud-2/), governance
(https://www.datanami.com/tag/governance/), GPU (https://www.datanami.com/tag/gpu/),
Hadoop (https://www.datanami.com/tag/hadoop/), IPO (https://www.datanami.com/tag/ipo/),
machine learning (https://www.datanami.com/tag/machine-learning/), security
(https://www.datanami.com/tag/security/), year in review
(https://www.datanami.com/tag/year-in-review/)

© 2019 Datanami. All Rights Reserved. A Tabor Communications Publication Contact (https://www.datanami.com/about/contact/) Back to Top
Privacy Policy (https://www.datanami.com/about/privacypolicy/)
Cookie Policy (https://www.datanami.com/about/cookie-policy/)
About Datanami (https://www.datanami.com/about/)