Академический Документы
Профессиональный Документы
Культура Документы
Dr. Vijay Srinivas Agneeswaran, Director and Head, Big-data R&D, Innovation Labs, Impetus
1
Contents
Big Data Computations
Introduction to ML Characterization
Programming Abstractions
GraphLab
Recommender systems
Medical decision aids Robot navigation systems
3
Outlook
Sunny
Humidity
Overcast
Yes
Rain
Wind
High
No
Normal
Yes
Strong
No
Weak
Yes
cons
Low prediction accuracy, High variance, Size VS Goodness of fit
K-means Clustering
Data Mining
Supervised learning (Classification/regression) random forests, support vector machines (SVMs), logistic regression (LR), Nave Bayes
Credit scoring, fraud detection, market basket analysis, medical diagnosis, manufacturing optimization
10
Computations/Operations
Logistic regression, kernel SVMs, conjugate gradient descent, collaborative filtering, Gibbs sampling, alternating least squares.
Data sets not embarrassingly parallel? Machine vision from Google [3] Deep Learning Artificial Neural Networks Speech analysis from Microsoft Giant 5 Graph processing GraphLab, Pregel, Giraph
[1] National Research Council. Frontiers in Massive Data Analysis . Washington, DC: The National Academies Press, 2013. [2] Richter, Yossi ; Yom-Tov, Elad ; Slonim, Noam: Predicting Customer Churn in Mobile Networks through Analysis of Social Groups. In: Proceedings of SIAM International Conference on Data Mining, 2010, S. 732-741 [3] Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Quoc V. Le, Mark Z. Mao, Marc'Aurelio Ranzato, Andrew W. Senior, Paul A. Tucker, Ke Yang, Andrew Y. Ng: Large Scale Distributed Deep Networks. NIPS 2012: 11
Iterative ML Algorithms
What are iterative algorithms? Those that need communication among the computing entities Examples neural networks, PageRank algorithms, network traffic analysis Conjugate gradient descent Commonly used to solve systems of linear equations [CB09] tried implementing CG on dense matrices DAXPY Multiplies vector x by constant a and adds y. DDOT Dot product of 2 vectors MatVec Multiply matrix by vector, produce a vector. Communication Overhead 1 MR per primitive 6 MRs per CG iteration, hundreds of MRs per CG computation, leading to 10 of GBs of communication even for small matrices. Other iterative algorithms fast fourier transform, block tridiagonal
[CB09] C. Bunch, B. Drawert, M. Norman, Mapscale: a cloud environment for scientific computing, Technical Report, University of California, Computer Science Department, 2009.
Second Generation
Third Generation
Examples
Mahout, Pentaho, Revolution R, SAS Inmemory Analytics (Hadoop) Horizontal (over Hadoop) Small subset sequential logistic regression, linear SVMs, Stochastic Gradient Descent, k-means clustering, Random Forests etc. Vast no. Kernel SVMs, Multivariate Logistic Regression, Conjugate Gradient Descent, ALS etc. Most tools are FT, as they are built on top of Hadoop
Spark, HaLoop, GraphLab, Pregel, SAS In-memory Analytics (Greenplum/Teradata), Giraph, Golden ORB, Stanford GPS, ML over Storm Horizontal (Beyond Hadoop) Much wider including Conjugate Gradient Descent (CGD), Alternating Least Squares (ALS), collaborative filtering, kernel SVM, belief propagation, matrix factorization, Gibbs sampling etc. Multivariate logistic regression in general form, K-means clustering etc. work in progress to expand the set of algorithms available. FT HaLoop, Spark Not FT Pregel, GraphLab, Giraph Spark giant 2, 3 and 4. GraphLab giant 5.
Practically Nothing
FaultTolerance Giants
Vijay Srinivas Agneeswaran, Pranay Tonpay and Jayati Tiwari, Paradigms for Realizing Machine Learning Algorithms, Big Data Journal (Libertpub), 1(4), 207-214.
13
Contents
Big Data Computations
Introduction to ML Characterization
Programming Abstractions
GraphLab
14
15
16
Conviva
Uses Hive for repeatedly running ad-hoc queries on video data.
Yahoo
Advertisement targeting: 30K nodes on Hadoop Yarn
Optimized ad-hoc queries using Spark RDDs found Spark is 30 times faster than Hive
Moved to Shark for on-the-fly queries C* OLAP aggregate queries on Cassandra 130 secs, 60 ms in Spark
17
BDAS: Spark
Transformations/Actions Map(function f1) Filter(function f2) flatMap(function f3) Union(RDD r1) Sample(flag, p, seed) groupByKey(noTasks) Description Pass each element of the RDD through f1 in parallel and return the resulting RDD. Select elements of RDD that return true when passed through f2. Similar to Map, but f3 returns a sequence to facilitate mapping single input to multiple outputs. Returns result of union of the RDD r1 with the self. Returns a randomly sampled (with seed) p percentage of the RDD. Can only be invoked on key-value paired data returns data grouped by value. No. of parallel tasks is given as an argument (default is 8). Aggregates result of applying f4 on elements with same key. No. of parallel tasks is the second argument. Joins RDD r2 with self computes all possible pairs for given key. Joins RDD r3 with self and groups by key.
reduceByKey(function f4, noTasks) Join(RDD r2, noTasks) groupWith(RDD r3, noTasks) sortByKey(flag) Sorts the self RDD in ascending or descending based on flag. Reduce(function f5) Aggregates result of applying function f5 on all elements of self RDD Collect() Return all elements of the RDD as an array. Count() Count no. of elements in RDD take(n) Get first n elements of RDD. First() Equivalent to take(1) saveAsTextFile(path) Persists RDD in a file in HDFS or other Hadoop supported file system at given path. saveAsSequenceFile(path Persist RDD as a Hadoop sequence file. Can be invoked only on key-value paired RDDs ) that implement Hadoop writable interface or equivalent. foreach(function f6) Run f6Chowdhury, in parallel on Tathagata elements of self RDD. [MZ12] Matei Zaharia, Mosharaf Das, Ankur Dave, Justin Ma, Murphy McCauley, Michael
J. Franklin, Scott Shenker, and Ion Stoica. 2012. Resilient distributed datasets: a fault-tolerant abstraction for inmemory cluster computing. In Proceedings of the 9th USENIX conference on Networked Systems Design and Implementation (NSDI'12). USENIX Association, Berkeley, CA, USA, 2-2.
Representation of an RDD
Information Set of partitions HadoopRDD 1 per HDFS block FilteredRDD Same as parent JoinedRDD 1 per reduce task Set of dependencies None 1-to-1 on parent Shuffle on each parent
Function to compute data set based on parents Meta-data on location (preferredLocaations) Meta-data on partitioning (partitioningScheme)
None HashPartitioner
19
4.
}
println("Result: " + w)
http://spark-project.org
24
25
Contents
Big Data Computations
Introduction to ML Characterization
Programming Abstractions
GraphLab
26
27
Contents
Big Data Computations
Introduction to ML Characterization
Programming Abstractions
GraphLab
29
PMML Primer
PMML offers a standard to define a model, so that a model generated in tool-A can be directly used in tool-B.
May contain a myriad of data transformations (pre- and post-processing) as well as one or more predictive models.
30
Likelihood
Prior
Given features X1,X2,,Xn, predict a label Y by calculating the probability for all possible Y value
31
Normalization Constant
32
Model Model specifi attributes Mining Schema Treatment for missing and outlier values Targets Prior probability and default Outputs List of computer output fields Post-processing Definition of model architecture/parameters.
33
34
35
36
38
39
2.0
10 25
21
129 310
40
1.0
10
35 mins 17 sec
More than 3 hrs
41
Contents
Big Data Computations
Introduction to ML Characterization
Programming Abstractions
GraphLab
42
Data associated with edges (weights, for instance) and vertices (user profile data, current interests etc.).
Consistency is important in ML algorithms (some do not even converge when there are inconsistent updates collaborative filtering).
GraphLab provides varying level of consistency. Parallelism VS consistency.
Implemented several algorithms, including ALS, K-means, SVM, Belief propagation, matrix factorization, Gibbs sampling, SVD, CoEM etc.
Co-EM (Expectation Maximization) algorithm 15x faster than Hadoop MR on distributed GraphLab, only 0.3% of Hadoop execution time. [YL12] Yucheng Low, Danny Bickson, Joseph Gonzalez, Carlos Guestrin, Aapo Kyrola, and Joseph M. Hellerstein. 2012. Distributed GraphLab: a framework for machine learning and data mining in the cloud. Proceedings of the VLDB Endowment 5, 8 (April 2012), 716-727.
GraphLab could not scale to Altavista web graph 2002, 1.4B vertices, 6.7B edges.
Most graph parallel abstractions assume small neighbourhoods low degree vertices But natural graphs (LinkedIn, Facebook, Twitter) power law graphs. Hard to partition power law graphs, high degree vertices limit parallelism.
[1] Joseph E. Gonzalez, Yucheng Low, Haijie Gu, Danny Bickson, and Carlos Guestrin (2012). "PowerGraph: Distributed Graph-Parallel Computation on Natural Graphs." Proceedings of the 10th USENIX Symposium on Operating Systems Design and Implementation (OSDI '12).
Contents
Big Data Computations
Introduction to ML Characterization
Programming Abstractions
GraphLab
45
R2: Multi-tenancy
Addressed by Hadoopon-Demand Security, Quotas
R5: Reliability/Availability
Job Tracker bugs
Vinod Kumar Vavilapalli, Arun C Murthy , Chris Douglas, Sharad Agarwal, Mahadev Konar, Robert Evans, Thomas Graves, Jason Lowe , Hitesh Shah, Siddharth Seth, Bikas Saha, Carlo Curino, Owen O'Malley, Sanjay Radia, Benjamin Reed, and Eric Baldeschwieler, Apache Hadoop YARN: Yet Another Resource Negotiator, ACM Symposium on Cloud Computing, Oct 2013, ACM Press. 46
47
YARN Internals
Application Master Sends ResourceRequests to the YARN RM Captures containers, resources per container, locality preferences.
YARN RM Generates tokens and containers Global view of cluster monolithic scheduling.
Node Manager Node health monitoring, advertise available resources through heartbeats to RM.
48
Contents
Big Data Computations
Introduction to ML Characterization
Programming Abstractions
GraphLab
49
Programming Abstractions
PMML
XML based representation of the analytical model
Spark
Scala collection over a distributed shared memory system
GraphLab
Gather-ApplyScatter
Forge
Domain Specific Language
50
51
Forge [AKS13] a meta DSL for high performance DSLs. 40X faster than Spark! OptiML DSL for machine language
[Arvind K. Sujeeth, Austin Gibbons, Kevin J. Brown, HyoukJoong Lee, Tiark Rompf, Martin Odersky, and Kunle Olukotun. 2013. Forge: generating a high performance DSL implementation from a declarative specification. In Proceedings of the 12th international conference on Generative programming: concepts & experiences (GPCE '13). ACM, New York, NY, USA, 145-154.
Conclusions
Beyond Hadoop Map-Reduce philosophy
Optimization and other problems. Real-time computation
PMML scoring
Spark for batch computations
Spark streaming and Storm for real-time. Allows traditional analytical tools/algorithms to be
re-used.
52
Thank You!