ARM and Machine Learning

Jem Davies
ARM Fellow & VP Technology
Media Processing Group, Imaging & Vision Group

December 12 2016

© ARM 2016
 Who am I?
 Jem Davies - ARM Fellow and VP of Technology - Media
Processing and Imaging & Vision Groups
 Working on multimedia, computer vision and machine learning
 13 years at ARM
 Mathematician turned chemist, turned software engineer…
 … turned architect
 … turned tech future predictor
 Glider pilot instructor, fireworks lighter
and scuba diver…
 … what next?

2 © ARM 2016
 Machine Learning overview
 Machine Learning makes smart connections to previously
encountered concepts
 It’s useful when:
• We don’t have algorithms…
• but we do have a lot of data

Image recognition Robotics Home security Speech recognition

3 © ARM 2016
 We use Machine Learning technology every day

Speech Digital
recognition assistant

Computational
Predictive text
photography

Face tracking
Fingerprint ID
camera

4 © ARM 2016
 Definitions and
recent developments

5 © ARM 2016
 The AI landscape
Machine
Learning Artificial
Intelligence

Constraints
Computer
and control Machine
Vision
theory
Learning

AI Deep
Learning

Natural
Search and
language
Optimization
processing
Algorithms:
CNNs,
RNNs, etc.
Reasoning

6 © ARM 2016
 Key terms and definitions
Artificial Intelligence
• The broadest term - applying to any technique enabling computers to mimic
human intelligence

Machine Learning
• A subset of AI including techniques enabling computers to improve at tasks
with experience. Includes deep learning

Deep Learning/Neural Networks

• A branch of machine learning that attempts to model real life ideas in data by
using a deep graph with multiple processing layers

Algorithms
• DNNs, CNNs, RNNs, SGEMM etc.

Computer Vision
• An interdisciplinary field that allows computers to gain understanding from
digital images or videos. Many computer vision applications use ML algorithms.

7 © ARM 2016
 Recent developments in deep learning
 The image detection benchmark ImageNet has
reached nearly 100% accuracy
 Largely due to Neural Networks

 Research groups feel it’s not useful to work on

ImageNet anymore (it’s a solved problem)
 GoogLeNet, Inception, etc.

 The successful approach used for ImageNet has

spread to other domains, yielding great
improvements

8 © ARM 2016
 Why did this happen ?
The Machine Learning learning loop

9 © ARM 2016
 Basic neural network

Squashing function

x1 Sigmoid: y=1/(1+e^(-z))
w1
x2 w2 n
z = ∑ wi xi
[Inputs] x3 w3 i =1 y [Output]
wn-1 y = H ( z)
xn-1
wn
xn
10 © ARM 2016
 Machine Learning computation
in the cloud and on-device

11 © ARM 2016
 Dissecting the Machine Learning process

Cloud side Training engine Generate inference engines

Inference engine

Heterogeneous tools
Device side Distribute & optimize
ARM Cortex + Mali + Others

Platform acceleration libraries

12 © ARM 2016
 Machine Learning process on ARM
Cloud services / applications Cloud applications

Cloud service SDK

Cloud APIs
+ analytics
Cloud side Training engine
Cloud Metadata streams, thumbnails etc.

Device
Applications

Inference engine Middleware

Heterogeneous tools
Device side SW
ARM Cortex + Mali + Others HW
CPU GPU CV/ML IP HW Cores

Platform acceleration libraries (Cache-coherent interconnect) HW Interconnect

13 © ARM 2016
 It is easy to recognise speech
It is easy to wreck a nice beach

14 © ARM 2016
Automatic speech recognition is not easy
 Active research area for over 50 years!
 Significant attention recently due to large amount of
cloud computing
 Neural Networks has improved performance
 Siri, Google Now, Cortana, Amazon Echo now creating
usable applications

 Applications
 Safety and security
 Interactions with smaller or hands free devices (e.g. wearables)
 Improved accessibility for hearing-impaired
 Allows indexing of spoken words

15 © ARM 2016
 ASR use cases and Machine Learning
 Large Vocabulary Continuous Speech Recognition (LVCSR)
 Dictation/transcription, virtual assistant
 Requires dictionary, knowledge of grammar

 Keyword spotting of simple commands

 “OK Google”, “Set alarm for 7”

 Sound monitoring
 Early/automatic anomaly detection

16 © ARM 2016
 Keyword spotting Speech Signal

 Listen for certain words/phrases

Feature extraction
 Only need to learn certain words
 “Okay Google” Features
 “Play music”
Acoustic model
 Simpler algorithm
Words Play music
 No knowledge of grammar
 Only needs acoustic model

 Algorithm must run locally on device

 Potentially always listening
 Power consumption vitally important

17 © ARM 2016
 Computer Vision

18 © ARM 2016
 Computer vision pipeline

Output
Application

High-level
Detection and processing
segmentation
Feature
Pre-processing extraction
Image acquisition

ISP/GPU GPU/CPU/DSP/Spirit CV CPU

19 © ARM 2016
 ARM and Machine Learning

20 © ARM 2016
 Machine Learning runs on ARM-powered client
devices today

21 © ARM 2016
 Caffe framework deep learning on ARM

Video hosted on youtube

https://www.youtube.com/watch?v=k4ovpelG9vs&t=13s

22 © ARM 2016
 Deep learning frameworks on ARM
 DNN used for real-time detection of objects 2.7 5.5
64% 20%

 Based on the Caffe deep learning framework

 Detection entirely on the mobile device – not

cloud based

 Optimized for ARM Cortex CPU or Mali GPU

 Running Machine Learning on the GPU frees

up the CPU for other tasks

 Optimised libraries also being developed for

TensorFlow and OpenVX

23 © ARM 2016
 Conclusions Innovation Growth Maturity
Machine
Learning
 The rapid uptake in Machine Learning is
going mainstream, affecting compute
everywhere
Platforms
 Machine Learning dramatically increases & Tools
compute demands
 As much as possible, Machine Learning
workloads should run locally on device,
not on remote servers
 Machine Learning is driving demand for advanced ARM
processors and accelerator IP
 Machine Learning is having a significant impact on ARM’s
roadmap for future processors and architectures
24 © ARM 2016
 Thank you for listening

The trademarks featured in this presentation are registered and/or unregistered trademarks of ARM Limited
(or its subsidiaries) in the EU and/or elsewhere. All rights reserved. All other marks featured may be
trademarks of their respective owners.
Copyright © 2016 ARM Limited

© ARM 2016