LANE DETECTION AND AUTONOMOUS DRIVING CONTROL SYSTEMS
USING NEURAL NETWORK ON VIDEO GAMES SIMULATION
Abstract— Autonomous driving is one of the most
popular topics in research related to vehicle safety.
One of the most useful technologies in autonomous
driving is lane detection that uses longitudinal marks
(e.g. dotted and straight lines) as a reference to keep
the vehicle running on track. The main purpose of
previous research focused on two topics. One of them
is how to detection lane effectively using various
image processing methods (i.e. edge detection).
Various operators on edge detection are proposed to
obtain the best possible accuracy detection results.
However, previous research only did the simulation
and could not figure out how the results of the system
had been made if implemented directly on the
vehicle. Therefore, in this paper we will do the
research to do the simulation in real-time to know
how the system driving a vehicle with a system that
has been proposed.
Keywords— Autonomous Driving, Neural Network,
Lane Detection, Edge Detection.
I. INTRODUCTION
Autonomous driving is one of the most popular
topics in research related to vehicle safety. If humans use
the eye to see the conditions around the vehicle, this
system also uses various sensors to detect the conditions
around the vehicle to make decisions like humans. The
camera is one of the important tools as a substitute for
the human eye, the camera itself can be installed in
various vehicle positions. Many systems such as
adaptive cruise control, blind spot detection, collision
avoidance system, night vision and traffic sign detection
using the camera on the vehicle. The purpose of these
systems is to detect the road markers and provide advice
to drivers who will leave the track.
Lane detection is a process of detecting markers on
the path that will be used as the basis of intelligent
systems in the vehicle to make decisions. There are
various techniques that can be used to detect markers on
the lane, in previous research there are two very popular
techniques [11]. One is the path detection effectively
using various image processing methods or image
processing.
Image processing is a form of processing or
processing the signal with the input of a picture or image
and transformed into another image as output with a
particular technique. Edge detection or commonly called
edge detection is one technique that can be used to
extract an information from an image and is one of the
techniques of image processing.
In this paper we will focus on path detection by using
edge detection. Various operators on edge detection are
proposed to obtain the most accurate detection results
[1-4], [8], [9]. Edge detection has many operators, each
operator has different functions and results depending
on the object under study [7]. Canny is the most
commonly used operator on road mark detection, such
as [1-5].
First, we will discuss the lane itself and the methods
or techniques that can be used to detect road markers.
II. LANE DETECTION
Lane detection can be done with a variety of
methods, common methods are often used to detect the
path of taking the image of the road with the help of
cameras mounted outside or inside the vehicle. Then do
preprocessing to change the color mode on the grayscale
image, this stage aims to reduce processing time. If the
image is not converted to grayscale then the system will
process 3 gray color channel on the image that is (R) red,
(G) green and (B) Blue which cause the process takes
longer time. If the image to be processed has noise, it
will make the image obscure and block the actual edge
of the image. Therefore, a smoothing filter will be
applied such as gaussian, median, mean, etc. to reduce
noise on the image. Then it will do edge scanning using
edge detection and convert the image to binary or black
and white. After that will scan the boundary line on the
image using hough 1
Road Image 𝐷 = min⁡[𝑎𝑏𝑠( ⁡w – Xh)] (1)
2
transform. And generate a 𝐸𝑥−𝐵𝑥
𝑋ℎ = 𝐵𝑥 + (ℎ − 𝐵𝑦) (2)
𝐸𝑦−𝐵𝑦
boundary line on the path Grayscaling
and then display it again in
D is the distance between detected lines and the center
the initial image by
Smoothing
of field. Xh is the absolute of the intersection point of
coloring the detected line
the lane and the image boundary line. where w is the
markings.
Edge width of image while h represents the height; B and E
Enhancement
From the algorithm represent two end points of detected lane; Bx & By is
above the edge detection is the coordinate of original point and Ex & Ey is the
the most important step to Canny Edge coordinate of terminal point. While the minimum value
Detection
get the line on the road of D, the detected lines are the inside lane lines that
marker. Edge detection also vehicle driving on. The principle of eq.3 is shown in
has a variety of techniques Extract the one third at
bottom of the image Fig.6 and the final test results shown in Fig.8.
(operator) in it that has
influence also on the
results. Here shown some Hough Transform

results of the lane detection

using edge detection Extract lane lines
techniques are different. detection

Figure 6. The schematic diagram of calculating inside

lane line

Figure 1. Results lane detection using edge detection [7]

Figure 7. Results lane detection using Hough
III. SIMULATION
Transform
The most important stage is lane detection. we will
use edge detection to extract road markings, then use
hough transform to find the line curves on edge
detection results. But there is a proble, the result of the
lane detection must have 2 lines for each side left and
right as shown in Figure 7. to solve the problem, in order
to more precise detection result this research only to use
the inside line using the following equation:

Figure 8. The final result of lane detection

After getting the line from both sides. System must
know the position of the vehicle. The formula to find the
vehicle position estimation of the markers on equations
3 and 4 while to know the range can be seen in equation
4:
𝑅1 = (middle − B1) − 𝑃1 (3)
𝑅2 = P2 − (middle + B2) (4)
𝑅 = R1 + R2 (5)
While P si Point x of y maximum; B1 & B2 is Car
border; R1 & R2 are Range B to P and Middle is
Length of curve x / 2. The formula to find where the
direction of moving vehicles left or right mark, using
equation 5:
First, we must describe about the ROI it self, figure 10
explain about region. Our system used 800 x 600 px
resolution. Because the number of disturbances in the
area that is not needed, will perform the ROI process,
can be seen in Figure 10, here is the region selected and
will be processed. after getting the image ROI, then will
be done lane detection algorithm implementation. can be
seen in figure 11 of the lane detection algorithm before
using hough transform. and in figure 12 is the result after
applied hough transform.

𝑅1−𝑅2
𝑅𝑜𝐿 = 100 (6)
𝑅

In equation 6, the value multiplied by 100 is to find the

value in percentage form. Here is the rule to find the
vehicle moving towards the left and right markers: Figure 10. Schematic flowchart system

- If the direction of car = R1> R2, (R1-R2) / R x100

then the direction of the vehicle will be turned to the
right of the mark
- If the direction of car = R1 <R2, (R2-R1) / R x100
then the direction of the vehicle will be turned to the
left of the mark. If not, then the value is 0. The
direction of the vehicle is in the middle. Figure 11. Results of lane detection

Now we can start to build system for simulation.

Simulation using 3D game open-world. Open world
mean, the game such as the real world, the character can
running, driving, eating, fighting, die, etc. we need
driving feature for implementing the system. GTA V
have all of the feature, vehicle, driving, road. Flowchart
system showed in figure 9.

Lane Figure 12. Final results of lane detection

ROI
Detection
It’s time to build the self-driving system. After we have
lines in both side. We can predict the vehicle position
Neural Network Self Driving System
Control System using equation 6. Code for controling vehicle in game
showed in table 1.
Figure 9. Schematic flowchart system
 Tabel 1. Define control system IV. NEURAL NETWORK CONTROL
For input, position of vehicle and range car to the left
def straight(): def right():
PressKey(W) PressKey(D)
and right lines and output is where the vehicle must turn,

ReleaseKey(A) ReleaseKey(A) left, right or straight. After define the input, now system
ReleaseKey(D) ReleaseKey(W) need the training data, in previous research they use
def left(): ReleaseKey(D) more than 50.000 data training. Training data use for
PressKey(A) def slow(): system to improve the speed of process. If use lane
ReleaseKey(W) ReleaseKey(W)
detection the system need 0.2xxx seconds for process
ReleaseKey(D) ReleaseKey(A)
the algoritm. If frame rate is 30 fps. System just can
ReleaseKey(A) ReleaseKey(D)
process 5/30 frame per second it will be affect the
if R1 < 0 and R2 < 0:
right() system for make a decision.
elif R1 > 0 and R2 > 0:
left()
else:
straight()

V. REFERENCE

[1] Erin Welling, CONVOLUTIONAL NEURAL NETWORKS IN AUTONOMOUS VEHICLE CONTROL

SYSTEMS.
[2] Oleksandr Bailo et.al. 2017.Robust Road Marking Detection and Recognition Using Density-based Grouping
and Machine Learning Techniques. Applications of Computer Vision (WACV), 2017 IEEE Winter
Conference.
[3] Mariusz Bojarski, Ben Firner, Beat Flepp, Larry Jackel, Urs Muller and Karol Zieba. 2016. End-to-End Deep
Learning for Self-Driving Cars
[4] Rishal H. 2017. Deep Learning Convolutional Neural Networks in Autonomous Vehicle Control
[5] N. Mechat, N. Saadia, N.K. M’Sirdi, N. Djelal. 2012.Lane Detection and Tracking by Monocular Vision
System in Road Vehicle. Image and Signal Processing (CISP), 2012 5th International Congress.
[6] Raghuraman Gopalan, Tsai Hong, Mike Shneier, and Rama Chellappa. 2014. Video-based Lane Detection
using Boosting Principles
[7] Shengyan Zhou, Jianwei Gong, Guangming Xiong, Huiyan Chen and Karl Iagnemma. 2010. Road Detection
Using Support Vector Machine based on Online Learning and Evaluation.
[8] FAN Chao, XU Jing-bo, DI Shuai. 2014. Lane Detection Based on Machine Learning Algorithm.
TELKOMNIKA Indonesian Journal of Electrical Engineering.
[9] M. Foedisch. A.Tekuchi, 2004. Adaptive real-time Road Detection using Neural Network. Intelligent
Transportation Systems, 2004. Proceedings. The 7th International IEEE Conference.
[10] Jian Wang. Zhong Ji. Yu-Ting Su. 2009. Unstructured Road Detection Using Hybrid Features. Machine
Learning and Cybernetics, 2009 International Conference.
[11] Sha Yun, Zhang Guo-ying, Yang Yong. A Road Detection Algorithm by Boosting Using Feature
Combination. Intelligent Vehicles Symposium, 2007 IEEE
[12] Zu Whan Kim. 2008. Robust lane Detection and Tracking in challenging Scenarios. IEEE Transactions on
Intelligent Transportation Systems
[13] P. Conrad, M. Fodisch. Performance Evaluation of Color Based Road Detection Using Neural Nets and
Support Vector Machine. Applied Imagery Pattern Recognition Workshop, 2003
[14] Yasamin Alkhorshid, Kamelia Aryafar, Sven Bauer, Gerd Wanielik. 2016. Road Detection through
Supervised Classification. Computer Vision and Pattern Recognition (cs.CV)
[15] U. Qayyum and M.Y. Javed. 2005. Lane Change Detection using Fusion of IPM and SVM. Intelligent
Systems and Control (ISC 2005)
[16] Ella (Jungsun) Kim, Shane Soh. Detecting Lane Departures Using Weak Visual Features. FINAL PROJECT
REPORT.
[17] Raghuraman Gopalan, Tsai Hong, Michael Shneier. Et al. A Learning Approach Towards Detection and
Tracking of Lane Markings. IEEE Transactions on Intelligent Transportation Systems 2012.
[18] Michael Smart, Steven L. Wasiander. 2015. Stereo Augmented Detection of Lane Marking Boundaries.
Intelligent Transportation Systems (ITSC), 2015 IEEE 18th International Conference
[19] Taewon Han, Yangyhun Kim, Kisung Kim. 2014. Lane Detection & Localization For UGV in Urban
Environment. Intelligent Transportation Systems (ITSC), 2014 IEEE 17th International Conference.
[20] ALHADI ALI ALBOUSEFI. Vehicle Lane Departure Prediction Based On Support Vector Machines. Final
Project.