The Evolution of Personal Assistants:

from Science Fiction to Reality to the Transparent Brain

Mobile Voice Conference San Francisco, March 3-5, 2014
Marsal Gavald marsal@expectlabs.com

Summary

Thesis Human behavior is getting easier to observe and predict. First Corollary Personal assistants will become uncannily good. Second Corollary Deviating from the recommended course of action will need to be enshrined as an unalienable right.

The evolution of taxi receipts

Measuring productivity

Tracking shoppers

Source: The New York Times

Source: Berry & Reed

Magnifying motion

Source: Wu, Hao-Yu, et al. Eulerian video magnification for revealing subtle changes in the world. ACM Transactions on Graphics (TOG) 31.4 (2012): 65.\

Predicting user location from historical cell tower data: 93% accuracy

Source: Science 327, 1018 (2010)

Amazons Anticipatory shipping

Source: Amazon

Source: US PTO

Mechanical chess-playing Turk (Wolfgang von Kempelen, 1770)

Source: Wikipedia

Singing automaton Olimpia

E.T.A. Hoffmanns The Sandman (1816) Jacques Offenbachs Tales of Hoffmann (1881)

Source: YouTube

10

Apples Knowledge Navigator (1987)

Messages and calendar Document search Related articles Video conferencing Article sharing Data mashup

Source: Apple

11

Spike Jonzes Her (2013)

Source: herthemovie.com

13

Artificial Intelligence slowly but inexorably improving

Progress in Automatic Speech Recognition ! ! ! Dynamic speaker adaptation Deep/recurrent neural networks Ultra large language models

Natural Language Understanding ! Conversation and topic modeling ! Knowledge Graph ! 570 million entities ! 18 billion facts & relationships Machine Learning ! Latent factor models for recommender systems leads to improved understanding of natural, human-to-human conversations.
Source: Google Source: NIST

14

Deep learning applied to Automatic Speech Recognition

Microsoft
Context-dependent deep neural network, hidden Markov model: -33% WER Deep tensor neural networks: -8% WER
Source: Large vocabulary speech recognition using deep tensor neural networks, Yu et al. (2012)

IBM
Deep neural network with linear feature-space maximum mutual information discriminative objective function: -5% WER
Source: Discriminative feature-space transforms using deep neural networks, Saon et al. (2012)

Google
Context-dependent ANN/HMM hybrid system pretrained with deep belief networks: -5% WER

Source: Application of pretrained deep neural networks to large vocabulary speech recognition, Navdeep Jaitly et al. (2012)

Deep learning applied to Natural Language Understanding

Stanford Sentiment Treebank
From semantic vector spaces to recursive neural tensor networks: -25% sentiment accuracy error

Source: Recursive deep models for semantic compositionality over a sentiment treebank, Socher et al. (2013)

Knowledge Graphs
From disembodied strings to grounded entities
! ! ! ! ! ! ! ! Yahoo! 10 M entities, 30 M properties, 10 M connections Microsoft 300 M entities, 800 M connections Google 570 M entities, 18 B properties and connections Wikipedia 4 M entities Freebase 40 M topics, 2 B facts Factual 66 M local businesses and POIs in 50 countries LinkedIn 225 M people Facebook 1.15 B people

Cf. ! Cyc 239 K concepts, 2 M facts ! OpenCyc 6 K concepts, 60 K facts

Source: Yahoo!

Exploiting users location, calendar, email

! Local search
! Tempo AI ! EasilyDo ! Cue (Apple) ! Google Now !

Source: Google

Recommender systems
! Amazon
! Netflix ! Pandora ! Spotify ! FourSquare ! LinkedIn ! Facebook !

Source: Tempo AI

Mobile devices capture data via many sensors

Cameras, microphones, Wi-Fi, LTE, GPS, and

Source: funf open sensing framework

20

Source: Samsung

Context awareness helps modulate response

Source: Entourage by HBO

Youre on speakerphone and my wife is in the car

Even more sensors are coming

Google Glass 3 axis gyroscope 3 axis accelerometer 3 axis magnetometer ambient light and proximity sensors bone conduction transducer camera, Wi-Fi, Bluetooth

Speech recognition from facial muscle activity captured by electrodes (Schultz & Wand, 2010)

Emotiv EPOC EEG headset

Coupling an electronic skin tattoo to a mobile communication device (U.S. Pat. Application 20130297301)

Quantified Self

Source: Giga OM

Source: Giga OM

Source: Rachelle DiGregorio

Source: Facebook

1949

2013

WAR IS PEACE FREEDOM IS SLAVERY IGNORANCE IS STRENGTH

ALL THAT HAPPENS MUST BE KNOWN SHARING IS CARING SECRETS ARE LIES PRIVACY IS THEFT

The possibility of total digital surveillance touches the essence of our life. It is thus an ethical task that goes far beyond the politics of security [ to the] freedom and dignity of the individual.
Angela Merkel On Freedom and Security January 29, 2014

Source: The New York Review of Books (March 20, 2014)

Literally Transparent Brain

Source: National Geographic Magazine

Source: National Geographic Magazine

26

Towards mind reading

Source: Science 320.5880 (2008)

Source: Carnegie Mellon University

27

Literally Transparent Brain

Source: National Geographic Magazine

Source: National Geographic Magazine

28

Literally Transparent Brain

Source: National Geographic Magazine

Source: National Geographic Magazine

29

When one has weighed the sun in the balance, and measured the steps of the moon, and mapped out the seven heavens star by star, there still remains oneself. Who can calculate the orbit of his own soul?!
Oscar Wilde
De Profundis (1897)

Source: Risdall