1

Intelligent Driver Assistance System Using

Artificial Neural Networks
Adarsh V R (B100291EC), Adeeb P A (B100118EC), Ashwin S M (B100422EC),
Athul M Das (B100123EC), Dileep R (B100004EC)
S8 ECE A BATCH
Project Advisor: Dr. Praveen Sankaran

AbstractRoad accidents are an issue of growing concern

today. Proper following of traffic rules and signboards can help
reduce the accidents to a good extent. We consider designing
a system which can assist in safe driving. A system which can
give prompt messages to the driver in advance about the traffic
signs ahead, pedestrians crossing the road etc. can help reduce
accidents. Natural environments in which traffic signs are present
makes artificial neural networks a suitable choice for dealing with
the problem. In this paper, we provide a report of our initial work
in this direction. A brief explanation of artificial neural networks
is provided, followed by problem definition, proposed solution,
initial simulations, work schedule and conclusions.

axis is not perpendicular to the sign, it could cause a change

in the scale for each axis causing a change in the aspect ratio.
5. The physical condition of the traffic sign could change due
to age, accidents etc.
All these indicate that we cannot use a simple algorithmic
method for traffic sign recognition in a real world scenario.
A more efficient way would be to use a technique that can
tolerate some differences in the data acquired with respect to
the reference data for a traffic sign.

Index Termsmachine learning, neural network, perceptrons,

back propagation, road accident, traffic signs.

B. Human Nervous System

I. I NTRODUCTION

ncreasing number of road accidents point towards the

need for strict adherence to traffic rules. Traffic signs and
signals act as an assistance for safe driving. Following these
signs and signals can help reduce the number of accidents to a
great extent. A system which can alert the driver of upcoming
traffic signs and signals appropriately can help him/her drive
safely. Such a system would involve continuous monitoring
of the environment in front of the vehicle and detection of
traffic signs and signals. After detecting these, they should
be properly recognised and the driver alerted appropriately.
Future intelligent vehicles would take decisions about the
speed, trajectory etc. on the basis of the signs detected. Such
a system would automatically limit the speed of the vehicle,
warn illegal manoeuvres or indicate the presence of traffic
signs to the driver.
The detection and recognition of the traffic signs present
difficulties due to the variabilities of the natural environment
in which these are present.

A. Difficulties in traffic sign recognition

1. Lighting conditions are not under our control. It can change
with the time of the day, season, cloudiness etc.
2. Partial occlusions, shadows etc. due to other objects.
3. Object size depends on distance to the object.
4. It is not possible to generate an off-line model for the
traffic sign under consideration due to the numerous degrees
of freedom of viewing the traffic sign. If the camera optic

A digital computer is algorithm based and makes use of

the stored program concept proposed by Von Neumann[5].
With this, we first obtain a step by step procedure for solving
a problem known as algorithm. This is then converted into a
program which is then executed sequentially by the computer.
This approach is particularly suitable for data manipulation,
mathematical computations etc. As such, digital computers
are faster than human beings in such numerical computations.
Instructions are executed in a modern digital computer at the
rate of around an instruction per nanosecond.
Humans and other animals have a different mechanism for
data storage and computation. The most important difference
is that we humans can learn from our past experiences. So
we have the ability to store acquired knowledge and use
it for taking decisions. Also human beings have a more
distributed system for computation. It is made up of a
number of processing units known as neurons networked
and interconnected together. All the neurons work in parallel
making our nervous system a heavily parallel architecture.
Knowledge is stored in the strength of the interconnections
between neurons which is adapted and changed with our
experiences. Human beings are slower than digital computers
in performing individual computations. We can complete
computations on the order of around a computation per
millisecond.
However the heavily parallel architecture of our nervous
system combined with the fact that we can learn from our
experience means that we can perform certain tasks far more
quickly compared to a digital computer. These tasks include
pattern recognition (for example face recognition, object

recognition), perception etc.

C. Need for Artificial Neural Networks

We can tolerate some variations in the data received
compared to that of the original data set that we have
learned. We can also perform these tasks quickly than an
algorithm based approach. Machine leaning, a branch of
artificial intelligence is an area that deals with systems which
can learn from past experiences and take decisions based
on that. There are various machine learning algorithms like
decision tree learning, association rule learning, artificial
neural networks, inductive logic programming etc [2]. Among
these artificial neural network is an idea inspired by the
biological neural networks of human beings. With artificial
neural networks, computations are structured in terms of
an interconnected structure of artificial neurons. Knowledge
acquired is stored in the weights of the interconnections
of the neurons. So artificial neural networks have the
ability for storing acquired knowledge and use it for taking
decisions. They also have a parallel architecture similar to
the biological neural networks.Hence, they are more suited
for perceptual tasks than a normal algorithmic based approach.

II. L ITERATURE S URVEY

There are different types of Artificial Intelligence (AI)
technologies available today like
1. Fuzzy logic
2. Artificial Neural Network
3. Genetic Algorithms
4. Decision tree learning
Each of these is motivated by the aspiration to go beyond the
capabilities of normal algorithm based approach. Each has its
own advantages and disadvantages.

A. Artificial Neural Networks

As mentioned earlier, artificial neural network is a type
of machine learning algorithm. Artificial neural networks are
based on the biological nervous system of human beings.
Similar to that of a biological neural network, an artificial
neural network is made up of a network of processing units.
In this case, biological neurons are replaced by artificial
neurons. Presence of a number of parallel processing elements
makes artificial neural networks a heavily parallel architecture.
Each interconnection in the artificial neural network has an
interconnection weight which resembles the strength of the
interconnections in a biological neural network. These weights
are adapted with the learning process to suit our application
at hand.
1) Model of a neuron: A neuron is the basic information
processing unit of both the biological and artificial neural
network. A biological neuron consists of a cell body, axon,
synapses and dendrites. A neuron receives its inputs from

other neurons through dendrites and gives inputs to other

neurons through synapses. Strength of an interconnection input
determines the weight of that input. We try to model our
artificial neuron in a similar way.
Following figure gives the model of an artificial neuron

Fig. 1. Model of an artificial neuron

The above model consists of a set of synapses, each characterised by a weight of its own, a bias, an adder, an
activation function and finally the output. x1, x2, x3, ..., xm
denote the inputs to the neuron. These inputs are weighted by
wk1 , wk2 , wk3 , ..., wkm where k is a number characterising a
neuron. These weighed inputs are added together along with a
bias bk by an adder and fed as input to the activation function
of the neuron. Activation function has basically the job of
limiting the amplitude of the weighted sum computed above.
So activation function is sometimes also called squashing
function. A particular case of activation function is a normalised activation function in which the output is limited to
the amplitude range of [1, 1].
From the above model, we can describe a neuron mathematically in terms of the following two equations.
uk =

m
X

wkj xj

(1)

yk = (uk + bk )

(2)

j=1

where yk is the output of the k th neuron. Here bias bk is an

external parameter of the neuron. We can reformulate (2) as
yk = (vk )

(3)

Pm

where vk = j=0 wkj xj , wk0 = 1 and x0 = bk . vk is also

called the induced local field of the neuron.
The commonly used activation function is
Sigmoid function:- Sigmoid function is an s-shaped function[1]. It is probably the most common type of activation
function under use in artificial networks. It takes on continuous
values. One eaxmple of a sigmoid function is the logistic
function defined as
1
(vk ) =
(4)
1 + exp(avk )
Here a is known as the slope parameter. Changing the value
of a changes the slope of the sigmoid function.

hence again a feedforward network.

Multilayer feedforward networks are one of the most
widely used network architectures.

Fig. 2. Sigmoid function for different slopes[1]

Another commonly used sigmoid function is the tansigmoid

function defined as
(vk ) = tanh(vk )
2) Network Architectures: Architecture of a neural
network refers to the structure of the neural network.
Network architectures may vary in basic layout as well as
in complexity. Choosing a particular network architecture
directly influences the learning algorithm used for training
the neural network and the complexity of the application for
which the neural network is being used.
In general, there are three differnent classes of network
architectures.
1. Single layer feedforward networks: A layered neural
network is one in which inputs, neurons and outputs are
spread over different layers and are interconnected with
each other. Simplest of the layered neural networks are the
single layer feedforward networks. These networks have an
input layer of source nodes followed by a layer of output
neurons[1]. Connections flow only in the forward direction,
hence the name feedforward network.

Fig. 3. A single layer feedforward network

2. Multilayer feedforward network:- A multi layer feedforward

network is different from a single layer feedforward network
in that it has hidden layer(s) of neurons in between the input
and output layers. Neurons in the hidden layer are called
hidden neurons. These neurons intervene between the input
layer of source nodes and output in some useful manner.
These can hence be more complex than single layer neural
networks and can be used for more involved applications.
Here again connections flow only in the forward direction,

Fig. 4. A multilayer feedforward network

3) Learning: One of the most attractive features of artificial

neural network is the ability to learn. We can train our neural
network for a data set. Training has to be optimal to get
good results. The data set used for training is crucial. Network
architecture used is critical in choosing the learning paradigm.
In supervised training, a set of input output pairs is given
to train the neural network. Weights of the neural interconnections are adjusted properly to give the optimum results.
One of the popular supervised learning paradigms is the
back propagation algorithm used with multilayer feed forward
networks. In back propagation algorithm, error of the network
(difference between desired and actual output) is computed
during each stage of training and the weights are adjusted to
minimise the error.
Unsupervised learning and reinforcement learning are two
other types of learning paradigms used for training neural
networks.
4) Back Propagation Algorithm and Feed Froward
Network: The back propagation algorithm is a common
method of training artificial neural networks which is used
to learn the weights of a multilayer neural network with
fixed architecture. It tries to minimize the sum squared error
between networks output and given target value.
It is a supervised learning method, and is a generalization of
the delta rule. It requires a dataset of the desired output for
many inputs, making up the training set. It is most useful for
feed-forward networks (networks that have no feedback, or
simply, that have no connections that loop). Back propagation
requires that the activation function used by the artificial
neurons (or nodes) be differentiable.
Feed forward networks often have one or more hidden layers
of sigmoid neurons followed by an output layer of linear
neurons. Multiple layers of neurons with non-linear transfer
functions allow the network to learn non-linear and linear

relationships between input and output vectors.

The linear output layer lets the network produce values
outside the range -1 to +1. On the other hand, if you want to
constrain the outputs of a network (such as between 0 and 1),
then the output layer should use a sigmoid transfer function
(such as logsig).
To illustrate the back propagation process consider a three
layer neural network with two inputs and one output, which
is shown in the picture below:
Each neuron is composed of two units. First unit adds

Fig. 5. 3-layer neural network

products of weights coefficients and input signals. The

second unit realise non-linear function, called neuron transfer
(activation) function. Signal e is adder output signal, and
y = f (e) is output signal of non-linear element. Signal y is
also output signal of neuron. To teach the neural network

Propagation of signals through the hidden layer: Symbols

Wmn represent weights of connections between output
of neuron m and input of neuron n in the next layer.
Y4 = f4 (W14 Y1 + W24 Y2 + W34 Y3 )
Y5 = f5 (W15 Y1 + W25 Y2 + W35 Y3 )
Propagation of signals through the output layer:
Y = f6 (W46 Y4 + W56 Y5 )
In the next step the output signal of the network, Y is
compared with the desired output value Z (the target), which
is found in training data set. The difference is called error
signal of output layer neuron.
=Y Z
The idea is to propagate error signal (computed in single
teaching step) back to all neurons.
The weights Wmn used to propagate errors back are equal
to the ones used for computing the output value. Only the
direction of data flow is changed (signals are propagated from
output to inputs one after the other). This technique is used
for all network layers. If propagated errors come from more
than one neuron, they are added. The illustration is below:
4 = W46
5 = W56
1 = W14 4 + W15 5
When the error signal for each neuron is computed, the weight
coefficients of each neuron input node may be modified. In
(e)
represents derivative of neuron activation
formulas below dfde
function.

Fig. 6. A neuron

we need training data set. The training data set consists of

input signals (X1 and X2 ) assigned with corresponding
target (desired output) Z. The network training is an iterative
process. In each iteration weights coefficients of nodes are
modified using new data from training data set. Modification
is calculated using algorithm described below: Each teaching
step starts with forcing both input signals from training set.
After this stage we can determine output signals values for
each neuron in each network layer.
Let us consider that signals are propagating through input
layer. Symbols W(xm )n represent weights of connections
between network input Xm and neuron n in input layer.
Symbols Yn represents output signal of neuron n.
Then,
Y1 = f1 (W(X1 )1 X1 + W(X2 )1 X2 )
Y2 = f2 (W(X1 )2 X1 + W(X2 )2 X2 )
Y3 = f3 (W(X1 )3 X1 + W(X2 )3 X2 )

Fig. 7. Algorithm of Back Propagation

1 (e)
0
W(X
= W(X1 )1 + 1 dfde
(X1 )
1 )1
1 (e)
0
W(X
= W(X2 )1 + 1 dfde
(X2 )
2 )1
Similarly other weights coefficients also modified.

III. P ROBLEM D EFINITION

There has been significant increase in road accidents over
the years. Human error is a major contributing factor to
this situation. It is a common case that drivers often neglect
traffic signs due to various distractions. The driver may be
seeing the sign but the brain may fail to register this due to a
variety of reasons. This can lead to accidents. For example if
a driver fails to observe a Sharp Curve Ahead sign, he may
not decrease the speed of the vehicle and this may lead to an
accident.
A machine interface that can analyse the surroundings and

then raise alarms if necessary can be used as a counter

measure to this. This project deals with the smaller part of
the problem in which we design a system to monitor the
surroundings for traffic signs and classify them accordingly.

IV. P ROPOSED S OLUTION

An algorithmic approach may not be a good strategy since
the image of the traffic sign may be taken from different
angles and the vehicle may be moving at different speeds.
It is difficult to compare the input images with our data set
as the images might be different from the images to which
comparisons are made. Thus we need a way to identify images
even when they are not exactly similar to the image data set
at our hands.

A. Work Done in seventh Semester

This project involved selecting suitable methods for image
processing as well as neural network sections, which required
determining many parameters like the number of hidden
neurons, the learning rate and so on in the neural network
section. In image processing section, the best features to be
used to detect the traffic signs had to be finalized. All of these
could only be done using simulations. So, the seventh semester
was mainly dedicated for building our system in MATLAB,
ensuring its working and determining the parameters to be
used for building the driver assistance system.
1) Image Processing Section: In this section mainly the
traffic sign detection from video was done in MATLAB.
Traffic sign detection was done based on features like colour
and shape of traffic signs. Colour is a dominant visual feature
and is undoubtedly a key piece of information for drivers to
identify traffic signs. Colour information is widely used in
traffic sign detection systems, especially for segmentation of
traffic sign images from the rest of the scene. Shape is another
important parameter that can be used to successfully detect
traffic signs. A technique which can detect various shapes in
the image can help us find out traffic signs present in the
image. Hough Transform, which involves making use of the
parametric equation of the shape of interest, was used to find
out shapes of interest present in the image. [3]

Fig. 8. Traffic Sign Detection Problems: a. Reflections b. Not controlled lighting c. Shadows d,e. Partial occlusions f. Sign Rotation g. Shape Deformation

Neural networks are a good choice in such cases where

we need to identify a pattern. Training data can be obtained
by taking images of the traffic signs under various practical
scenarios with the vehicle moving at different speeds and at
different angles of viewing w.r.t traffic sign. This can then
be used to train the neural network. Since the processing is
complex, we cannot use a single layer perceptron but rather a
multi-layer perceptron may have to be used.
A video recorder can be used to gather video data which is
then processed so as to separate the target area from the overall
input frame. This is then given as input to the neural network
which then classifies the input accordingly.

Fig. 9. Input video used for Traffic Sign Detection based on colour

V. W ORK D ONE
Our project can be mainly divided into two sections. The
traffic sign detection part is done using image processing and
the recognition part is done using neural networks.The video is
sent to the image processing section which does the processing
and detects the traffic sign and send the detected traffic sign
to neural network section which in turn recognises the sign
and give the alert to the driver.

Fig. 10. Result of Traffic Sign Detection based on colour:Traffic Sign detected

Initially, detection based on colour was done. As part of this

colour based recognition, a video was taken as input the and
number of frames in it were calculated. Here a video with
29 frames/second was used. Each frame was then fed to a
function that detected the traffic signs using colour. Initially
the regions containing red color were determined. The image
was read into a 3 dimensional matrix that was used to store the
corresponding intensities of Red, Green and Blue. The pixel
values were then traversed one by one. After that, regions with
the intensity value of the Red component greater than those
of the corresponding green and blue components were found.
Thus what we did in effect was to find the regions that are
dominant in red component. Once the starting region of the
red colour was found the corresponding matrix indices were
stored so that the region could be demarcated later.
Using Hough Transform, detection using shape was done. The
parametric equation of circle was used to detect circular traffic
signs from the video. After trying out the detection of traffic

circular regions was computed. If it is above a set threshold,

the circular region is detected as a traffic sign. This method
provided better results. Figure 22 below illustrates the method
which was done in MATLAB. It can be noted that the same
traffic sign could not be detected simply based on colour due
to problems in proper edge detection. Converting image to
YCbCr space and then performing edge detection solves this
problem.

Fig. 12. Traffic sign detection based on both colour and shape. Images in
clockwise direction from top left are 1.Original image 2.Red difference of
chroma of the image 3. Edge detected image 4. Detected traffic sign 5. Binary
image of 2

Fig. 11. Traffic sign detection based on shape. The images in clockwise
from top left position are as follows. 1. Colour image 2. Image converted to
grayscale 3. Circular shape detected from edge detected image 4. Traffic sign
detected and separated from the image 5. Binary image

signs in separate using colour and shape, both of the methods

were combined together to provide a more efficient detection
mechanism. For this the image was first converted from the
RGB space to YCbCr space(Y-luminance, Cb- blue difference
chroma component, Cr- red difference chroma component).
Here detection of traffic signs with a red boundary was done.
Cr component in the YCbCr space was used to detect red
colours in the image more easiy. So after converting the image
to YCbCr space, the image corresponding to Cr component
was obtained. This image was then converted to a binary
image using proper thresholding. Edge detection was then
carried out on this image. Hough transform was then applied
to find cicular shapes in the edge detected image. Once the
circular shapes were obtained, percentage of red colour in the

Thus, it was concluded that shape and color features of

traffic sign combined could be used for detection purposes.
Also working in a colour space other than RGB colour space
could improve the detection capability of the system.
2) Neural Network Section: In order to understand how
neural network works and in order to familiarize with the
neural network toolbox in MATLAB, a simulation was done
in MATLAB. Since our project deals with the recognition of
images, as an initial step, recognition of character images
using ANN was done in MATLAB. First, a training data
set was created which contained images of different fonts
of the alphabets. Characters A-F was chosen and images of
10 different fonts of these six characters were created. (Total
60 characters). These images were read from MATLAB and
were reshaped to 100 X 100 pixels. Then these images were
converted to single dimensional arrays of 10000 size and
stored as columns of the input vector. For each letter, different
classes were defined for different fonts. This is to be used as
the training set. A simple complete multilayer neural network
was created with a single hidden layer having 15 neurons.
Using these letters as the training set, the neural network
was trained using back propagation algorithm. Random noise
was then added to the each of the letters previously used for
training. A new neural network was created similar to the
earlier one and this new network was trained using the new
data set (the one obtained after adding random noise to the

first training set). It has more number of train vectors. Now

the two trained networks were tested with the first training set
itself as the test set and percentage of recognition of errors
were plotted in both cases. 15% of the training set was used
as the validation set. Number of hidden layers of the neural
network was then changed and the performance change was
evaluated.
It is seen in figure 13 that as the number of neurons in the
hidden layer increases, the classification error decreases and
finally becomes zero. It is seen in figure 14 that the percentage
of error in recognition is zero for the second neural network
whereas for the first network, it is non-zero.
By doing character recognition using neural networks, a brief
idea was obtained about the effect of the parameters like
learning rate, number of hidden layers, number of neurons
in a hidden layer, size of training, testing and validation test.

trying to interpret all that MATLAB code. Then it turns it into

Java, and then finally executes the code. OpenCV, on the other
hand, is basically a library of functions written in C/C++. It
is specific purpose library for image processing and computer
vision. It covers many function for image/video processing,
pattern recognition as well as other well known technique that
are usually used in computer vision. We can directly provide
machine language code to the computer to get executed.
So ultimately we get more image processing done for your
computers processing cycles, and not more interpreting. As a
result of this, programs written in OpenCV run much faster
than similar programs written in MATLAB. That is OpenCV
is faster when it comes to speed of execution. For example,
In MATLAB, we would typically get 3-4 frames analysed per
second from a video. Whereas In OpenCV, we would get at
least 30 frames per second, resulting in real-time detection.
Also due to the high level nature of MATLAB, it uses a lot of
your systems resources. A MATLAB code requires over a giga
byte(GB) of RAM to run through video. In comparison, typical
OpenCV programs only require appproximately 70MB of
RAM to run in real-time. So for our system which requires real
time video processing, OpenCV provides a better choice than
MATLAB. Hence the system is implemented in OpenCV.[6]
B. Traffic Sign Detection

Fig. 13. Performance Dependence on number of neurons in Hidden Layer

Fig. 14. Percentage Error Recognition using two different data sets

VI. W ORK DONE IN EIGHTH S EMESTER

A. Use of OpenCV
MATLAB is built on Java, and Java is built upon C. So
when you run a MATLAB program, your computer is busy

Image Processing work done involves detecting traffic signs

from the captured video signal. It involves detecting the
regions of interest using colour and shape of the traffic signs
of interest.
The captured video signal is processed one frame at a time.
As a first step in processing, the frame of interest is converted
from RGB colour space to HSV colour space. YCbCr colour
space used for processing during last semester was replaced
with HSV space since it was later learned that HSV space
provides better performance under varying lighting and environmental conditions in which the traffic sign may be captured.
At present it was decided to correctly recognise three traffic
signs namely school ahead, no parking and no entry using the
system designed. It can be noted that all these traffic signs have
red coloured boundaries. Hence the processing done involves
identifying the regions in the frame having red colour. For this
the frame now in HSV space is thresholded by specifying the
proper range of red colour pixels in HSV space. This range
takes into account the variations produced by various lighting
and environmental conditions. The thresholded image will then
have white in the regions of red colour in the frame and black
colour in other regions.
The thresholded image is then subjected to contour detection
to detect contours in the thresholded image. After detecting
contours, those contours having area less than a minimum
specified area are neglected. This is to reduce the number
of contours to be considered for recognition. After this the
remaining contours are filled with white pixels. This is then
given as input to shape detection process which detects contours having triangular or circular shape. These two shapes are
considered since our traffic signs of interest have only these
two shapes. Those regions in the image having this shape are

then identified from the original input frame and cropped out.
These are then reshaped properly and given as input to the
Artificial Neural Network for recognition.
Following figure shows the results of detection of traffic sign
using the above mentioned techniques. Here the figures in the

clockwise order are: 1. Input image, 2. HSV image, 3.Detected

contours, 4. Thresholded image. The last figure shows the
detected traffic sign.
C. Traffic Sign Recognition
Most of the work done during the previous semester was
based on MATLAB where we used a neural network to do
some basic tasks like digit recognition. This was done so as to
get acquainted with the implementation of a neural network
at the basic level. Some image processing tasks which were
required as preprocessing steps were also done. During the
start of the current semester the efforts were mainly on
implementing all the things that were done in MATLAB
with the help of OpenCV. As such we implemented the digit
recognition using neural network with the help of OpenCV.
In order to create the training dataset for the neural network,
hundreds of images corresponding to various traffic signs
were taken and they were converted into a matrix form inside
which the corresponding pixel values were stored. The test
set was also created like this. These values would then be fed
to the neural network to train them. As a next step, a neural
network was created and it was trained using the traffic data
set corresponding to three traffic signs. The number of input
layers corresponds to the number of attributes of the data
which is the number of pixels stored per each image. The
output layer corresponds to three outputs, the final value
predicted was taken to be the maximum of the three values
at the output.
Once the neural network was trained, the weights
corresponding to each neuron in the hidden layer were
then stored inside an xml file. This would make it easier to
load the neural network parameters faster in the future as
we dont need to train the network again. We then checked

if this was able to differentiate between the three traffic

signs. Initially we received an accuracy of 67.6%. However
after trying with different number of hidden layers, this was
brought up to around 92.3%. The corresponding number of
neurons in the hidden layer was 125.
The results obtained are shown in the figure below. The

testing was done initially with images that were already cut
out from the original images. Later on the image processing
part was also combined which would give the required region
of interest as input to the neural network while the neural
network would then recognize the traffic sign. One of the
major problems corresponding to this is that it is that it is
quite possible for the input to the system to be something
similar to a traffic sign (a flux board for instance). This may
result in erroneous output as the neural network will give an
output corresponding to any of the three values no matter
what the input is.
One way to counter the above mentioned problem is to
train the neural network intentionally with such images (flux
boards and so on). Thus the system can then identify this as
a deviation from the normal traffic signs and give output as
required. However this is still in development stage.

D. Combined System
Traffic sign detection and traffic sign recognition stages
can be combined together to form a complete traffic sign
recognition system. Results of the combined system for no
entry traffic sign are shown below. Here the no entry traffic

signs have been detected from the frame. The region is cropped
out and then given as input to the artificial neural network.
The recognised traffic sign result can be seen in the command

prompt window which in this case correctly recognises the

traffic sign.
E. Challenges
1) Real time processing: One of the most important
challenges that needs to be addressed is the real time
data processing. The video needs to be continuously
monitored and further processing needs to be carried
out fast so that the delay is minimal. This is very
important as with high speed vehicles there is a
chance that the traffic signs may become detected
after some delay. This can render the system useless.
It was with this intention that OpenCV was used
in the project. A future scenario where we can use
better embedded support is also possible because of
this. For example the software can be embedded into
some platform that needs only minimal processing
power but can do the required job at hand with elegance.
2) Dataset: The collection of possible traffic signs is
numerous. However not all the signs need to be given
equal importance. We need to select only some critical
signs like School ahead, deep curve ahead; and so
on and the system can neglect some signs like those
informing things like the location of nearest hotel and
so on(This might be added into the system in the future
in order to enhance the intelligence of the system and
to make the system work as an assistance for things
other than driving also). Thus the photos corresponding
to some important traffic sign were taken to be used.
3) Improving accuracy: In the image processing part
the major technique by which the region of interest
is cropped was by using various attributes like the
colour of the region and also its shape.However
this method has its own disadvantages as well. For
example it is quite possible that the system might
erroneously detect flex boards that are similar to
traffic signs. As already discussed this problem is
being tackled by using some other training sets for
this kind of data sets so that the system can ignore them.

VII. C ONCLUSION
As it can be seen, even though the system was able to detect
and recognize a small set of traffic signs, there are still many
problems that needs to be addressed like erroneous detection,
lighting conditions and so on. There is also a chance for a
decrease in accuracy with an increase in the number of traffic
sign classes.
R EFERENCES
[1] Simon Haykin , Neural Networks: A comprehensive Foundation, 3rd
ed.Pearson Educaton, 2005.
[2] A. de la Escalera, J.M Armingol, M. Mata, Traffic sign recognition and
analysis for intelligent vehicles, 2003.
[3] Rafael.C.Gonzalez, Richard.E.Woods, Digital Image Processing, 3rd
ed.Pearson Education.

[4] Xiaoguang HU, Xinyan ZHU, Deren LI,Traffic Sign Detection Based On
Biologically Visual Mechanism, 2012
[5] David A Patterson, John L Hennessy, Computer Organisation and DesignThe Hardware Software Interface,Elsevier, 2003
[6] The OpenCV Reference Manual Release 2.4.8.0