Вы находитесь на странице: 1из 85

# A Gentle Introduction

to Neural Networks
(with Python)
Tariq Rashid @postenterprise
EuroPython Bilbao July 2016

Background
Ideas
DIY
Handwriting
Thoughts

and a

live demo!

Background

+
+
+
+
+
+

2403343781289312
2843033712837981
2362142787897881
3256541312323213
9864479802118978
8976677987987897
8981257890087988
= ?

AI is Huge!

Googles and Go

Ideas

## Simple Predicting Machine

Kilometres to Miles

## random starting parameter

try a model - this one is linear

Kilometres to Miles

not great

Kilometres to Miles

better

Kilometres to Miles

worse !

Kilometres to Miles

best yet !

Key Points

1.

2.

xactly? Try
e
s
rk
o
w
g
in
how someth
w
o
n
k
.
t
n
o
D
parameters
le
b
ta
s
ju
d
a
model with
.
parameters
e
th
e
n
fi
re
or to
Use the err

Garden Bugs

Classifying Bugs

Classifying Bugs

Classifying Bugs

Classifying Bugs

Key Points

ings.

1.

Classifying

thing

th
predicting
e
k
li
a
d
in
k
s is

Example

Width

Length

Bug

3.0

1.0

ladybird

1.0

3.0

caterpillar

## shift the line up

just above the training data point

## error = target - actual

E = (A + A)x - Ax
A = E / x

Hang On!

Oh no!
each update ignores
previous examples

A = L (E / x)

learning rate

Key Points

1.

you
d - ensures
o
o
g
is
g
in
your learn
pact of
im
s
e
c
u
d
Moderating
re
nd
your data, a
ll
a
m
o
fr
rn
lea
.
training data
y
is
o
n
r
o
outliers

Boolean Logic

## IF I have eaten my vegetables AND I am still hungry

THEN I can have ice cream.

go to the park.

Input A

Input B

AND

OR

Boolean Logic

Boolean Logic

XOR Puzzle!

Input A

Input B

XOR

XOR Solution!

Key Points

gle
ith just a sin
w
d
e
lv
o
s
e
nt b
roblems ca
p
e
m
o
S
1.
r classifier.
simple linea
g together to
in
rk
o
w
s
e
nod
e multiple
s
u
n
a
c
u
o
s.
2. Y
se problem
e
th
f
o
y
n
a
solve m

Brains in Nature

Brains in Nature

11,000 neurons

## natures brains can eat, fly, navigate, fight,

communicate, play, learn
.. and theyre

resilient

302 neurons

37 billion neurons
(humans 20 billion)

https://en.wikipedia.org/wiki/List_of_animals_by_number_of_neurons
https://faculty.washington.edu/chudler/facts.html

Brains in Nature

Brains in Nature

logistic function
y = 1 / (1 + e-x)

Brains in Nature

Artificial Neuron

Pause.

...

link weight?

Key Points

things, and
d
te
a
c
ti
is
h
do sop
ct
brains can
l
ra
and imperfe
tu
e
a
g
a
N
m
a
d
1.
to
ly resilient
are incredib
omputing.
c
l
a
n
io
it
d
a
like tr
signals .. un
d
artly inspire
p
s
in
ra
b
l
a
logic
to copy bio
g
in
ry
T
.
2
etworks.
n
l
a
r
u
e
n
l
artificia
its
arameter p
le
b
ta
s
ju
e ad
ights are th
e
w
k
in
L
s.
3.
ing happen
rn
a
le
e
th
where

## Feeding Signals Forward

Matrix Multiplication

Matrix Multiplication

weights

incoming signals

WI = X
dot product

Key Points

e
tions can b
la
u
lc
a
c
rd
a
no
iplication,
any feedforw
lt
m
u
e
m
h
T
ix
tr
a
.
1
as m
concisely
d
e
s
s
re
p
x
e
network.
e
th
e
p
a
h
s
t
matter wha
n do matrix
a
c
s
e
g
a
u
g
lan
.
rogramming
p
e
m
nd quickly
o
a
S
y
tl
n
2.
ie
ic
n really eff
multiplicatio

Network Error

Network Error

Internal Error

Internal Error

Matrices Again!

Key Points

we
guide how
to
r
o
r
r
e
e
th
ber we use
m
e
m
e
k weights.
R
n
li
.
r
1
te
e
m
dels para
refine a mo
e
is easy - th
s
e
d
o
n
t
u
tp
or at the ou
nd actual
a
d
e
ir
s
2. The err
e
d
etween the
difference b
outputs.
obvious. A
t
n
is
s
e
d
o
n
l
or at interna
portion to
rr
o
r
e
p
e
h
in
T
it
t
li
.
p
3
to s
pproach is
a
c
ti
is
r
u
e
h
hts.
the link weig
be
e error can
th
g
n
ti
a
g
a
back prop
too!
4. and
ultiplication
m
ix
tr
a
m
as a
expressed

Aaarrrggghhh !!

## landscape is a complicated difficult mathematical function ..

with all kinds of lumps, bumps, kinks

Gradient Descent

## smaller gradient .. youre closer to the

bottom take smaller steps?

Key Points

e
of finding th
y
a
w
l
a
c
ti
c
a pra
t descent is
n
ie
d
a
r
G
1.
nctions.
fu
lt
u
c
fi
if
d
f
minimum o
by
rshooting
e
v
o
f
o
e
c
n
ha
shallower.
avoid the c
ts
n
e
a
g
c
t
u
n
o
ie
Y
d
ra
2.
eg
r steps if th
e
ll
a
m
s
g
in
tak
tion
ifficult func
d
a
is
rk
o
etw
nt
f a neural n
o
r
o
rr
dient desce
e
ra
e
g
h
e
b
y
a
3. T
m
eights so
of the link w
will help ...

## We need to find this gradient

Error Gradient

E = (desired - actual)2

## school level calculus (chain rule)

dE/dwij = - ej . oj . (1 - oj) . oi
previous node

## A gentle intro to calculus

http://makeyourownneuralnetwork.blogspot.co.uk/2016/01/a-gentle-introduction-to-calculus.html

the slope

DIY

Neural Network
Class

Initialise

Train

Query

do the learning

## Python has Cool Tools

matrix maths

numpy
scipy
matplotlib
notebook

Function - Initialise

## # initialise the neural network

def __init__(self, inputnodes, hiddennodes, outputnodes, learningrate):
# set number of nodes in each input, hidden, output layer
self.inodes = inputnodes
self.hnodes = hiddennodes
self.onodes = outputnodes
# link weight matrices, wih and who
# weights inside the arrays are w_i_j, where link is from node i to node j in the next layer
# w11 w21
# w12 w22 etc
self.wih = numpy.random.normal(0.0, pow(self.hnodes, -0.5), (self.hnodes, self.inodes))
self.who = numpy.random.normal(0.0, pow(self.onodes, -0.5), (self.onodes, self.hnodes))
# learning rate
self.lr = learningrate
# activation function is the sigmoid function
self.activation_function = lambda x: scipy.special.expit(x)
pass

numpy.random.normal()

Function - Query

## combined weighted signals into hidden

layer
then sigmoid applied
# query the neural network
def query(self, inputs_list):
# convert inputs list to 2d array
inputs = numpy.array(inputs_list, ndmin=2).T
# calculate signals into hidden layer
hidden_inputs = numpy.dot(self.wih, inputs)
# calculate the signals emerging from hidden layer
hidden_outputs = self.activation_function(hidden_inputs)
# calculate signals into final output layer
final_inputs = numpy.dot(self.who, hidden_outputs)
# calculate the signals emerging from final output layer
final_outputs = self.activation_function(final_inputs)
return final_outputs

numpy.dot()

Function - Train

## # train the neural network

def train(self, inputs_list, targets_list):
# convert inputs list to 2d array
inputs = numpy.array(inputs_list, ndmin=2).T
targets = numpy.array(targets_list, ndmin=2).T

## # calculate signals into hidden layer

hidden_inputs = numpy.dot(self.wih, inputs)
# calculate the signals emerging from hidden layer
hidden_outputs = self.activation_function(hidden_inputs)
# calculate signals into final output layer
final_inputs = numpy.dot(self.who, hidden_outputs)
# calculate the signals emerging from final output layer
final_outputs = self.activation_function(final_inputs)

## output layer errors

hidden layer errors

## # output layer error is the (target - actual)

output_errors = targets - final_outputs
# hidden layer error is the output_errors, split by weights, recombined at hidden nodes
hidden_errors = numpy.dot(self.who.T, output_errors)
# update the weights for the links between the hidden and output layers
self.who += self.lr * numpy.dot((output_errors * final_outputs * (1.0 - final_outputs)), numpy.
transpose(hidden_outputs))
# update the weights for the links between the input and hidden layers
self.wih += self.lr * numpy.dot((hidden_errors * hidden_outputs * (1.0 - hidden_outputs)), numpy.
transpose(inputs))
pass

update weights

Handwriting

## Handwritten Numbers Challenge

MNIST Datasets

MNIST dataset:
60,000 training data examples
10,000 test data examples

MNIST Datasets

label
784 pixels
values

28 by 28 pixel image

Experiments

## 96% is very good!

weve only used simple ideas
and code
random processes
do go wonky!

More Experiments

98%

is amazing!

Thoughts

often

Thanks!
live demo!

## Finding Out More

makeyourownneuralnetwork.blogspot.co.uk

github.com/makeyourownneuralnetwork
www.amazon.co.uk/dp/B01EER4Z4G

twitter.com/myoneuralnet
slides goo.gl/JKsb62

Raspberry Pi Zero

## It all works on a Raspberry Pi Zero

and it only costs 4 / \$5 !!