Neural Networks

CIS Faculty
Department of Information Systems

Mansoura University
Artificial Neural Networks
Dr. Mohammed Elmogy
Mansoura University
Faculty of Computers & Information Sciences
elmogy@gmail.com
5. October 2011
Artificial Neural Networks 1

CIS Faculty
Mansoura University
Outline
Introduction
Natural and Artificial Neurons
ANN Learning
ANNs Types
Where are NN Used?

CIS Faculty
Mansoura University
Introduction Artificial Neural Networks
Outline
Introduction
ANN Learning
ANNs Types
Where are NN Used?

CIS Faculty
Mansoura University
A New Sort of Computer
What are computer systems good at... and not so good at?

CIS Faculty
Mansoura University
Artificial Neural Network (ANN)

I An ANN is a mathematical model or computational model that
is inspired by the structure and/or functional aspects of
biological neural networks.
I An ANN consists of an interconnected group of artificial

neurons, and it processes information using a connectionist
approach to computation.
I In most cases an ANN is an adaptive system that changes its

structure based on external or internal information that flows
through the network during the learning phase.

CIS Faculty
Mansoura University
Artificial Neural Network (ANN) (cont.)

I Modern neural networks are non-linear statistical data modeling
tools. They are usually used to model complex relationships
between inputs and outputs or to find patterns in data.
I Structure: large number of highly interconnected processing

elements (neurons) working together.
I Like people, they learn from experience (by example).

CIS Faculty
Mansoura University
Where Can ANNS Help?

I When we cannot formulate an algorithmic solution.
I When we can get lots of examples of the behavior we require

(learning from experience).
I When we need to pick out the structure from existing data.

CIS Faculty
Mansoura University
Natural and Artificial Neurons Artificial Neural Networks
Outline
Introduction
ANN Learning
ANNs Types
Where are NN Used?

CIS Faculty
Mansoura University
Inspiration from Neurobiology

I A neuron: many-inputs / one-output unit.
I output can be excited or not excited.
I incoming signals from other neurons determine if the neuron
shall excite (fire).
I Output subject to attenuation in the synapses, which are
junction parts of the neuron.

CIS Faculty
Mansoura University
Inspiration from Neurobiology (cont.)

I Dendrites: Carry signals in.
I Synapse: Size chages in response to learning. The synapse

resistance to the incoming signal can be changed during a
”learning” process.
I Axon:Carries signals away
I Nucleus: How to deal with the input?

CIS Faculty
Mansoura University
Artificial Neurons
I The neuron calculates a weighted sum of inputs and compares
it to a threshold. If the sum is higher than the threshold, the
output is set to 1, otherwise to 0.

CIS Faculty
Mansoura University
A Simple Perceptron

CIS Faculty
Mansoura University
Example: A Simple Single Unit Adaptive Network

CIS Faculty
Mansoura University
ANN Learning Artificial Neural Networks
Outline
Introduction
ANN Learning
ANNs Types
Where are NN Used?

CIS Faculty
Mansoura University
Learning
I From experience: examples / training data.
I Strength of connection between the neurons is stored as a

weight-value for the specific connection.
I Learning the solution to a problem = changing the

connection weights.

CIS Faculty
Mansoura University
Operation Mode
I Fix weights (unless in online learning).
I Network simulation = input signals flow through network to

outputs.
I Output is often a binary decision.
I Inherently parallel.
I Simple operations and threshold: fast decisions and real-time

response.

CIS Faculty
Mansoura University
Operation Mode (cont.)

I Adaptive interaction between individual neurons.
I Power: collective behavior of interconnected neurons.

CIS Faculty
Mansoura University
Evolving Networks
I Continuous process of:
I Evaluate output.
I Adapt weights.
I Take new inputs.
I ANN evolving causes stable state of the weights, but neurons

continue working: network has learned dealing with the
problem.

CIS Faculty
Mansoura University
ANNs Types Artificial Neural Networks
Outline
Introduction
ANN Learning
ANNs Types
Where are NN Used?

CIS Faculty
Mansoura University
Learning Performance
I Network architecture.
I Learning method: Unsupervised, Reinforcement learning, and
Backpropagation.

CIS Faculty
Mansoura University
Unsupervised Learning
I No help from the outside.
I No training data, no information available on the desired
output.
I Learning by doing.
I Used to pick out structure in the input:
I Clustering
I Reduction of dimensionality =⇒ compression
I Example: Kohonen’s Learning Law.

CIS Faculty
Mansoura University
Competitive learning: Example

I Example: Kohonen network or Self-organizing Map (SOM)
(Winner takes all).
I Only update weights of winning neuron:

I Network topology.
I Training patterns
I Activation rule
I Neighbourhood
I Learning

CIS Faculty
Mansoura University
SOM

CIS Faculty
Mansoura University
SOM (cont.)

CIS Faculty
Mansoura University
Competitive learning: Example (cont.)

I SOM is trained using unsupervised learning to produce a
low-dimensional (typically two-dimensional), discretized
representation of the input space of the training samples, called
a map.
I SOMs use a neighborhood function to preserve the topological

properties of the input space.
I The goal of learning is to cause different parts of the network

to respond similarly to certain input patterns. This is partly
motivated by how visual, auditory or other sensory information
is handled in separate parts of the cerebral cortex in the human
brain.
CIS Faculty
Mansoura University
Reinforcement Learning
I Teacher: training data.
I The teacher scores the performance of the training examples.
I Use performance score to shuffle weights randomly.
I Relatively slow learning due to randomness.

CIS Faculty
Mansoura University
Back Propagation
I It is a supervised learning method. It requires a teacher that
knows, or can calculate, the desired output for any input in the
training set.
I It is most useful for feed-forward networks. The term is an

abbreviation for backward propagation of errors.
Backpropagation requires that the activation function used by
the artificial neurons (or nodes) be differentiable

CIS Faculty
Mansoura University
Back Propagation (cont.)

CIS Faculty
Mansoura University

CIS Faculty
Mansoura University

I Desired output of the training examples.
I Error = difference between actual & desired output.
I Change weight relative to error size.
I Calculate output layer error, then propagate back to previous

layer.
I Improved performance, very common!

CIS Faculty
Mansoura University
Online / Offline
I Offline
I Weights fixed in operation mode.
I Most common.
I Online
I System learns while in operation mode.
I Requires a more complex network architecture.

CIS Faculty
Mansoura University
Generalization vs. Specialization
Optimal number of hidden neurons

I Too many hidden neurons: you get an over fit, training set is
memorized, thus making the network useless on new data sets.
I Not enough hidden neurons: network is unable to learn problem

concept. The network’s language isn’t able to express the
problem solution.
I Overtraining: Too much examples, the ANN memorizes the

examples instead of the general idea.

CIS Faculty
Mansoura University
Where are NN Used? Artificial Neural Networks
Outline
Introduction
ANN Learning
ANNs Types
Where are NN Used?

CIS Faculty
Mansoura University
Where are NN Used?

I Recognizing and matching complicated, vague, or incomplete
patterns.
I Data is unreliable.
I Problems with noisy data.

I Prediction
I Classification
I Data association
I Data conceptualization
I Filtering
I Planning

CIS Faculty
Mansoura University
Where are NN Used? (cont.)

I Prediction: learning from past experience, pick the best stocks
in the market, predict weather, identify people with cancer risk.
I Classification: Image processing, Predict bankruptcy for credit

card companies, Risk assessment.
I Recognition: Pattern recognition: SNOOPE (bomb detector in

U.S. airports), Character recognition, Handwriting: processing
checks.

CIS Faculty
Mansoura University
Where are NN Used? (cont.)

I Data association: Not only identify the characters that were
scanned but identify when the scanner is not working properly.
I Data Conceptualization: infer grouping relationships e.g.

extract from a database the names of those most likely to buy
a particular product.
I Data Filtering: e.g. take the noise out of a telephone signal,

signal smoothing.
I Planning: Unknown environments, Sensor data is noisy, Fairly

new approach to planning.
CIS Faculty
Mansoura University
Strengths of a Neural Network

I Power: Model complex functions, nonlinearity built into the
network.
I Ease of use: Learn by example, Very little user domain-specific

expertise needed.
I Intuitively appealing: based on model of biology, will it lead to

genuinely intelligent computers/robots?
I Neural networks cannot do anything that cannot be done using

traditional computing techniques, BUT they can do some
things which would otherwise be very difficult.
CIS Faculty
Mansoura University
General Advantages
I Advantages:
I Adapt to unknown situations.
I Robustness: fault tolerance due to network redundancy.
I Autonomous learning and generalization.
I Disadvantages:
I Not exact.
I Large complexity of the network structure.
I For motion planning?

CIS Faculty
Mansoura University
Status of Neural Networks

I Most of the reported applications are still in research stage.
I No formal proofs, but they seem to have useful applications

that work.

Neural Networks

Загружено:

Сведения о документе

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

Neural Networks

Загружено:

Авторское право:

Доступные форматы

CIS Faculty

Department of Information Systems

Artificial Neural Networks

Dr. Mohammed Elmogy

Artificial Neural Networks 1

Artificial Neural Networks 2

Artificial Neural Networks 3

A New Sort of Computer

Artificial Neural Networks 4

Artificial Neural Network (ANN)

I An ANN consists of an interconnected group of artificial

I In most cases an ANN is an adaptive system that changes its

Artificial Neural Networks 5

Artificial Neural Network (ANN) (cont.)

I Structure: large number of highly interconnected processing

I Like people, they learn from experience (by example).

Artificial Neural Networks 6

Where Can ANNS Help?

I When we can get lots of examples of the behavior we require

I When we need to pick out the structure from existing data.

Artificial Neural Networks 7

Artificial Neural Networks 8

Inspiration from Neurobiology

Artificial Neural Networks 9

Inspiration from Neurobiology (cont.)

I Synapse: Size chages in response to learning. The synapse

I Axon:Carries signals away

I Nucleus: How to deal with the input?

Artificial Neural Networks 10

Artificial Neural Networks 11

Artificial Neural Networks 12

Example: A Simple Single Unit Adaptive Network

Artificial Neural Networks 13

Artificial Neural Networks 14

I Strength of connection between the neurons is stored as a

I Learning the solution to a problem = changing the

Artificial Neural Networks 15

I Network simulation = input signals flow through network to

I Output is often a binary decision.

I Simple operations and threshold: fast decisions and real-time

Artificial Neural Networks 16

Operation Mode (cont.)

Artificial Neural Networks 17

I ANN evolving causes stable state of the weights, but neurons

Artificial Neural Networks 18

Artificial Neural Networks 19

Artificial Neural Networks 20

Artificial Neural Networks 21

Competitive learning: Example

I Only update weights of winning neuron:

Artificial Neural Networks 22

Artificial Neural Networks 23

Artificial Neural Networks 24

Competitive learning: Example (cont.)

I SOMs use a neighborhood function to preserve the topological

I The goal of learning is to cause different parts of the network

I The teacher scores the performance of the training examples.

I Use performance score to shuffle weights randomly.

I Relatively slow learning due to randomness.

Artificial Neural Networks 26

I It is most useful for feed-forward networks. The term is an

Artificial Neural Networks 27

Back Propagation (cont.)

Artificial Neural Networks 28