Machine learning tools, Keras

25.09.2019
Kristijan Šarić
EXACT BYTE d.o.o.
About me
● Java (desktop, web, mobile) applications
● Python (ML, web applications, scripts)
● JavaScript (Angular, React, …)
● Scala (web), purely functional programming
● Haskell (IOHK - explorer, wallet, cardano-shell, consulting for foreign companies)
● My own products (using ML)
○ https://www.emprovio.com/
○ https://contetino.com/
○ https://alenn.ai
○ Croatian sign language (in progress)
AI
● (Most) General form/term for anything relating to machines “thinking for
themselves”
● Artificial Intelligence is the broader concept of machines being able to carry
out tasks in a way that we would consider “intelligent” [1]
Machine learning
● “Rather than teaching computers everything they need to know about the
world and how to carry out tasks, it might be possible to teach them to learn
for themselves” [1]
● Probabilistic reasoning - what is the connection between inputs and outputs?
● Interesting field that emerged is Probabilistic programming which emphasizes
“reasoning under uncertainty”
Machine learning

Reinforcement

Supervised Unsupervised
AI vs Machine learning
● Deep Blue, the AI that defeated the world’s chess champion in 1997, used a
method called tree search algorithms to evaluate millions of moves at every
turn
● Prolog (Zlatko presented), first order logic - "there exists x such that x is
Socrates and x is a man"
● General intelligence, which is very broad and uses a sort of “transfer learning”
over different domains is more general than ML - imagine a neural network
that learns how to understand the language by understanding pictures
Neural networks

0.75 Magic 0.32

Neural networks

[0.75, 0.35, 1.7, -4.5] Magic [0.63, -3.56, 0, -10.3]

Neural networks

[ 0.75 [ 0.63
, 0.35 , -3.56
, 1.7 Magic ,0
, -4.5 , -10.3
] ]
Neural networks

[ 0.63
[ [ 0.75 ]
, -3.56
, [ 0.35, 1.7 ]
Magic ,0
, [ -4.5 ]
, -10.3
]
]
Neural networks

[ 0.63
[ [ [ 0.75 ] ]
, -3.56
, [ [ 0.35, -1.5 ], [ 1.7 ] ]
Magic ,0
, [ [ -4.5 ], [ -6.4 ]
, -10.3
]
]
 Neural networks

[ 0.63
[ [ [ 0.75 ] ]
, -3.56
, [ [ 0.35, -1.5 ], [ 1.7 ] ]
Function ,0
, [ [ -4.5 ], [ -6.4 ]
, -10.3
]
]

● Universal Approximation Theorem

● http://neuralnetworksanddeeplearning.com/chap4.html
Linear function

plt.plot([0,1,2,3], [0,1,2,3])
Non - linear function
xs = np.linspace(-10, 10, 100).tolist()

plt.plot(xs, list(map(lambda x: x**2 - 1, xs)))

Linear vs non-linear
● Linear is a straight line, the slope is constant
● Non-linear is not a straight line, the slope is not constant
Images, Text, Music?

Magic

[ 0.63
[ [ [ 0.75 ] ]
, -3.56
, [ [ 0.35, -1.5 ], [ 1.7 ] ]
Magic ,0
, [ [ -4.5 ], [ -6.4 ]
, -10.3
]
]
Neural networks (some) details

Input Output
(Neuron) (Neuron)
 Neural networks (some) details

0-1 Input Output 0-1

 Neural networks (some) details

0.35 Input Output 0.35

 Neural networks (some) details

is_human_frozen(X)

-100 Input Output 0

 Neural networks (some) details

is_human_frozen(X)

100 Input Output 1

 Neural networks (some) details

import numpy as np

# 1÷(1+(2.71828^-100))
def sigmoid(X):
return 1/(1+np.exp(-X))
-100 Input Output 0
np.round(sigmoid(-100), 2) == 0
np.round(sigmoid(100), 2) == 1
 Neural networks (some) details

import numpy as np

# 1÷(1+(2.71828^100))
def sigmoid(X):
return 1/(1+np.exp(-X))
100 Input Output 1
np.round(sigmoid(-100), 2) == 0
np.round(sigmoid(100), 2) == 1
 Neural networks (some) details

Thermostat was born!

Not the most useful one, but it would serve to check
if a person is frozen.

Output
100 Input ACTIVATION Output 1
FUNCTION
 Neural networks (some) details

is_frozen
= i1 * w1
w1

Output
100 Input (i1) ACTIVATION Output 1
FUNCTION
w2
is_burned
= i1 * w2
 Neural networks (some) details
Bias
Hidden 1
= i1 * w1
w1

Output
100 Input (i1) ACTIVATION Output 1
FUNCTION
w2
Hidden 2
= i1 * w2

This is what LEARNS!

Bias is missing, yes.
Bias
https://hackernoon.com/everythi
ng-you-need-to-know-about-ne
ural-networks-8988c3ee4491
Neural networks (some) details

Hidden

Input Output
Hidden

Hidden
Neural networks (some) details

Hidden

Input Output
Hidden

Hidden
Neural networks (some) details

Hidden

Input Output
Hidden

Hidden
Neural networks (some) details

Hidden Hidden

Input Output
Hidden Hidden

Hidden Hidden
Neural networks (some) details

Hidden Hidden

Deep neural
Input Output

network, Deep
Hidden Hidden

learning!
Hidden Hidden
Neural networks (some) details

Hidden Hidden

Deep neural
Input Output

network, Deep
Hidden Hidden

learning!
Hidden Hidden
Features, why neural networks

Input Output
Features, why neural networks

Input
Output (Is
(Image of
it a cat?)
animals)
Features, why neural networks

Input
Algorithm Output (Is
(Image of EXPERT
(Classification) it a cat?)
animals)
Features, why neural networks

Input Algorithm
(Feature Output (Is
(Image of extraction + it a cat?)
animals) classification)

That’s deep. Duude.

Keras
● High level library, integrated into Tensorflow 2.0
● There is a lot of tools/libraries to work with (Tensorflow, Pytorch, …)
● They tend to be quite similar
● Similarity comes from the fact that most of the actual computations are
operations on Tensors (which are arrays for our purposes)
● Tensors are mathematical objects which can be studied like many fields in
mathematics (set theory, calculus, tensor calculus, …)
● The original library that used to run a lot of these computations was NumPy
● NumPy is limited in the sense that it can execute only on CPU and not on
GPU
● GPU computation is faster these days since the calculations fit nicely into the
framework of neural networks (a lot of small/simple calculations)
Keras, initial example
# Keras imports

from keras.models import Sequential

from keras.layers import Dense

# Create a Sequential model

model = Sequential()
Keras, initial example
# Add an input layer and a hidden layer with 2 neurons

model.add(Dense(2, input_shape=(1,), activation='tanh'))

# Add a 1-neuron output layer

model.add(Dense(1, input_shape=(2,)))

# Summarise your model

model.summary()
Keras, test
# Add an input layer and a hidden layer with _ neurons
model.add(Dense(_, input_shape=(_,), activation='tanh'))

# Add a neuron output layer

model.add(Dense(_, input_shape=(_,)))

# Summarise your model

model.summary()

3 INPUT, 5 HIDDEN, 1 OUTPUT ?!

Keras, test
# Add an input layer and a hidden layer with _ neurons
model.add(Dense(_, input_shape=(_,), activation='tanh'))

# Add a neuron output layer

model.add(Dense(_, input_shape=(_,)))

# Summarise your model

model.summary()

5 INPUT, 20 HIDDEN, 2 OUTPUT ?!

Keras, test
# Add an input layer and a hidden layer with _ neurons
model.add(Dense(_, input_shape=(_,), activation='tanh'))

# Add a neuron output layer

model.add(Dense(_, input_shape=(_,)))

# Summarise your model

model.summary()

5 INPUT, 20 HIDDEN, 20 HIDDEN, 2 OUTPUT ?!

Keras, training
# Compile your model

model.compile(optimizer='adam',loss='mse')

# Fit your model on your data for 5 epochs

model.fit(Xs,Ys, epochs=5)

# Evaluate your model

print("Final loss:",model.evaluate(Xs,Ys))
Keras, training
X Y

-100 -1

-50 -1

-30 -1

0 1

30 1

50 1

60 -1

100 -1

150 -1
Keras, training
# Compile your model

model.compile(optimizer='adam',loss='mse')

# Fit your model on your data for 5 epochs

model.fit(Xs,Ys, epochs=5)

# Evaluate your model

print("Final loss:",model.evaluate(Xs,Ys))
Keras, simple classification example
● MNIST
● https://en.wikipedia.org/wiki/MNIST_database
Keras, training
from __future__ import print_function

import keras

from keras.datasets import mnist

from keras.models import Sequential

from keras.layers import Dense, Dropout, Flatten

from keras.layers import Conv2D, MaxPooling2D

from keras import backend as K

Keras, training
batch_size = 128

num_classes = 10

epochs = 12

# input image dimensions

img_rows, img_cols = 28, 28

(x_train, y_train), (x_test, y_test) = mnist.load_data()

Keras, training
1. x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)
2. x_test = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)
3. input_shape = (img_rows, img_cols, 1)
4.
5. x_train = x_train.astype('float32')
6. x_test = x_test.astype('float32')
7.
8. x_train /= 255
9. x_test /= 255
10. print('x_train shape:', x_train.shape)
11. print(x_train.shape[0], 'train samples')
12. print(x_test.shape[0], 'test samples')
Keras, training
1. # convert class vectors to binary class matrices
2. y_train = keras.utils.to_categorical(y_train, num_classes)
3. y_test = keras.utils.to_categorical(y_test, num_classes)
4.
5. model = Sequential()
6. model.add(Conv2D(32, kernel_size=(3, 3),
7. activation='relu',
8. input_shape=input_shape))
9. model.add(Conv2D(64, (3, 3), activation='relu'))
10. model.add(MaxPooling2D(pool_size=(2, 2)))
11. model.add(Dropout(0.25))
12. model.add(Flatten())
13. model.add(Dense(128, activation='relu'))
14. model.add(Dropout(0.5))
15. model.add(Dense(num_classes, activation='softmax'))
Keras, training
1. model.compile(loss=keras.losses.categorical_crossentropy,
2. optimizer=keras.optimizers.Adadelta(),
3. metrics=['accuracy'])
4.
5. model.fit(x_train, y_train,
6. batch_size=batch_size,
7. epochs=epochs,
8. verbose=1,
9. validation_data=(x_test, y_test))
10. score = model.evaluate(x_test, y_test, verbose=0)
11. print('Test loss:', score[0])
12. print('Test accuracy:', score[1])
13.
Keras, and others, problems
● Your tensor is not the right size - “ValueError: Error when checking target:
expected dense_3 to have 4 dimensions, but got array with shape (774, 27)”
● “I expected tensor to be of this size. For some reason the tensor is not that
size. You figure out where and when that happened.“
● Same as “Car stopped. Figure it out.” *
● Where is the context? Where are the programming best practices?
● I end up replacing values to see where the error is - which is the same as
trying to find a bug in your application by providing different inputs.
● Larger architectures are not that easy to debug. The tools we have for
debugging are not that impressive.
● Keras/Tensorflow are especially notorious for these debugging issues. Static
graph...
Conclusion
● Neural networks are not for everything, use for “unstructured data”
● Keras is nice and simple for simple examples
● If you try to do something complex it ramps up pretty fast to a point where it
doesn’t matter what tool/library you are using and where you should be
relying on understanding specific architectures/techniques/research
● (NN) Architectures are an art as much as they are reasoning
● I don’t think there will be AutoML tools if you want to do something right (high
accuracy, normal number of examples, …)