Robot Detection and Localization Based on Deep
Learning
Sha Luo, Huimin Lu, Junhao Xiao, Qinghua Yu, Zhiqiang Zheng
College of Mechatronic Engineering and Automation
National University of Defense Technology
luoshasha1992@gmail.com, lhmnew@nudt.edu.cn, junhao.xiao@hotmail
yuqinghua163@163.com, zqzheng@nudt.edu.cn
Abstract—Real-time and accurate robot detection and
localization is important for the RoboCup Middle size
League (MSL) soccer robots. In the current robot
detection methods used by most of the teams, the black-
color-based information is used to distinguish robots from
the environment, which is not robust if the robot changes
its makers’ color according to the current rule.
Considering the good performance of deep learning on the
feature extraction and object detection, in this paper, we
propose a novel approach for robot detection and
localization based on Convolutional Neural Networks
(CNNs) for RoboCup MSL soccer robots. The approach is
composed of two stages: robot detection using the RGB
image, and robot localization using the depth point cloud.
The high accuracy and mean average precision (mAP)
verify that the proposed method is suitable for robot
detection during the MSL competition, which will benifit
the following strategy design and obstacle avoidance
procedures. The proposed approach can be easily
exploited to deal with different objects and adapted to be
used in other RoboCup leagues. The acquired dataset is
made available for the community.
Keywords—Robot Detection and Localization; Deep learning;
RoboCup MSL; Convolutional Neural Networks (CNNs);
I. INTRODUCTION
Robot Soccer World Cup (RoboCup 1 ) is a worldwide
competition and academic event for promoting the research
and development of artificial intelligence (AI) and robotics by
providing a challenging and public testing platform. The
Middle Size League (MSL) is one of the most important
events of RoboCup. For a completely autonomous soccer
robot, the accurate and real-time detection of robots plays an
important role in the robot's entire system, and helps the robot
to perceive the environment as a basis for realizing
autonomous capabilities such as motion planning and
decision-making.
In the past decade, most MSL teams used the color-based
methods for robot detection [1, 2] because the main color of
the robots is black, which is obvious and easy to be
1 RoboCup Homepage: http://www.robocup.org/
This research was supported by National Natural Science Foundation of
China (61403409, 61503401), China Postdoctoral Science Foundation
(2014M562648).
‹,(((
recognized. According to the RoboCup MSL 2017 new rule,
the robots are allowed to wear clothes of different colors [3],
which introduces a new challenge to those methods. Recently,
deep learning has attracted increasing attentions on the object
detection and recognition as it allows computational models
composed of multiple processing layers to learn the
representations of data with multiple levels of abstraction.
These methods have dramatically improved the state-of-the-art
in speech recognition, visual object recognition, object
detection and so on [4]. Since the early 1990s, CNNs have
been applied with great success to the detection, segmentation
and recognition of objects and regions in images. In 2012,
Krizhevsky et al. [5] applied deep CNNs to a million of
images and showed high image classification accuracy on the
ImageNet Large Scale Visual Recognition Challenge
(ILSVRC) [6, 7] which made the CNNs well-known by
computer vision researchers.
Usually, the increased detection speed is achieved at the
cost of the decreased detection accuracy. Ribeiro et al used
deep learning on the MSL for object detection [12] where the
classifiers implemented by artificial neural networks are
trained using the images obtained from an omnidirectional
vision sensor. Different objects can be detected although there
are imaging distortions due to the use of omnidirectional
vision sensor. A Deep Learning method for NAO robot
detection was proposed in the RoboCup Standard Platform
League [13], which performs very well in terms of accuracy.
However, the processing time is too long to meet the real time
requirement.
Robot 1
Vision
Kinect
Industrial computer
Communication ROS_MASTER
Jetson TX2
Decision
LAN
(a) (b)
Fig. 1 Illustration of NuBot implementation; (a) NuBot robot (b) Connection
architecture of Kinect, Jetson TX2 and robot’s onboard industrial computer

 There are some state-of-the-art object detection systems
such as SSD [8], Faster-RCNN [9], YOLO v2 [10, 11] and so
on. YOLO v2 applys a single neural network to the full image
and the network divides the image into regions and predicts
bounding boxes and probabilities for each region. These
bounding boxes are weighted by the predicted probabilities.
According to the Table. 3 in [11], YOLO v2 is faster and more
accurate than previous detection methods. It can also run at
different resolutions for an easy tradeoff between the speed
and accuracy (67 FPS with mAP 76.8% on VOC2007 test).
In this paper, we used the Kinect v22 as the vision sensor
for robot detection and localization which is an active 3D
depth estimation setup employing IR laser structured patterns
for depth calculation. This characteristic makes it more
accurate in object localization. However, the huge amount of
data from the Kinect will increase the onboard industrial
computer’s CPU burden which needs to run motion control,
omnidirectional vision, and decision control and
communication modules at the same time. Considering the
weight and real time requirements, we decided to use the
NVIDIA TX2 3 embedded development board to process the
data from the Kinect for robot detection and localization and
transmit the localization results to the onboard industrial
computer for later decision making and obstacles avoidance.
The detection and localization system is equipped on our
NuBot soccer robot (Fig. 1(a)), and the communication
between TX2 and robot’s onboard industrial computer is
handled by ROS master messages as illustrated in Fig. 1 (b).
For employing the accurate and fast features of the YOLO
detection system, we adjusted the framework for the fast and
accurate robot detection during the detection stage. We also
built a novel dataset for robot detection which contained fully
annotated images acquired from MSL competitions on a
regular field. The dataset is publicly available at:
https://github.com/Abbyls/robocup-MSL-dataset. After the
robot detection step, we obtained robot’s 2D positions from
the RGB image. Then we registered the RGB image to the
depth image and produced the depth point cloud to obtain
robot’s 3D positions.
The structure of the paper is organized as follows. Section
2 introduces the proposed approach for the robot detection and
localization in detail. Section 3 presents the experiments
conducted on our NuBot soccer robot, and analyzes the
accuracy and efficiency of the approach. Section 4 concludes
this paper and discusses the future work.
accurate ground-truth annotations. Considering the lack of
open source dataset about the RoboCup MSL, we decided to
collect a set of images taken under varying conditions with
different vision sensors from different point of view. We also
made the dataset publicly available and encourage the similar
research in robotics/vision community.
Pipeline
Detection
Dataset creation Image pre-process acceleration Robot detection
Localization
Image
Noise elimination 3D position
registration
Depth point cloud
Fig. 2 The pipeline of the proposed approach
A typical image from our dataset is shown in Fig. 3, where
there are some robots with different shapes and colors on the
green soccer field. These robots have some common features
such as having a black base, the upper body being thinner than
the bottom body and having similar background. We need to
detect the robots and consider annotating the robots’ relative
positions with respect to the image. All the annotations are
provided in an XML file named "annotations.xml" that
contains the robot’s Class, Width, Height, Xmin, Xmax,
Ymin, Ymax and so on. The origin of the image coordinate is
placed in the upper-left corner. Since usually more than one
robot are presented in an image, bounding boxes could
overlap. There are 1456 images in total without any pre-
processing in our dataset, and each image has different
dimensions. We selected 1000 images for training and 456
images for testing.
X
(Xmin,Ymin)

II. PROPOSED APPROACH

height

The pipeline of our proposed approach is shown in Fig. 2

which includes two stages: robot detection and 3D position
localization.
(Xmax,Ymax)
A. Dataset building width
To train the network model for robot detection, a huge Y
amount of data from a real scenario are needed which require
Fig. 3 One of the annotated images from our dataset

2
Kinect Homepage: https://developer.microsoft.com/en-us/windows/kinect
3
Jetson TX2 introduction: http://elinux.org/Jetson_TX2


B. Image pre-processing acceleration
As mentioned above, the YOLO detection system is
extremely fast and accurate in comparison with other detection
systems when it is used on the NVIDIA Titan X4, which is the
strongest graphics processing unit at that time. Obviously, the
embedded development board Jetson TX2 we used is weaker
than the NVIDIA Titan X which makes the detection process
slower for the robot detection.
For fast and convenient prediction, we transformed and
resized (at the ratio) the input image before the prediction
process and then used a fixed sized box to constrain the
resized image to a fixed width and height image as the input
data to the predict process. The image pre-processing process
and the prediction process need about 58ms and 60ms
respectively, failing to meet the real-time requirements in the
real competition.
CPU
Transmit the RGB image
to YOLO framework GPU
Image data processing
Copy the boxed
image to CPU
Fig. 4. Data transmission between the CPU and GPU for the image pre-
processing
The pipeline of the image pre-processing algorithm offers
pixel-level data parallelism which can be easily exploited on
the CUDA (Compute Unified Device Architecture)
architecture. To accelerate the detection process, we decided
to parallelize the data pre-processing step using the CUDA.
Since the GPU consists of multiple cores, it allows
independent thread scheduling and execution, and is suitable
for the computation of independent pixels. Therefore, in the
image pre-processing step, we use m × n threads for an image
with the dimension of m×n, and use blocks of the appropriate
size running on multiple cores. The pipeline of the data
transmission between the CPU and GPU is shown in Fig. 4.
C. Robot Detection
After the image pre-processing, we put the processed box
images into the test network to predict the robot’s box position
and the corresponding probability. Considering the real-time
requirement in robot detection process, we use a tiny version
model which has 9 convolutional layers and 1 region detection
layer. The first 6 convolution layers of the network extract
features from the image, and the pooling layer is for the
subsampling and enhancing the rotational invariance while the
region detection layer predicts the boxes’ coordinates and
4
NVIDIA Titan X Homepage: https://www.nvidia.com/en-us/geforce/
products/10series/titan-x-pascal/

classifying probabilities. The network architecture is shown in
Fig. 5.
Conv Conv Conv Conv Conv Conv
Conv Conv Conv
Region
Pool Pool Pool Pool Pool Pool Detection
Fig. 5 The network architecture of Convolutional Neural Networks used for
D. Image Registration and Depth
robot Point Cloud Production
detection.
We register the RGB image to the depth image and
produce the depth point cloud, and then find the depth value
according to the box’s center position in the registered image.
After that, we can get the 3D position in the depth point cloud.
The open source driver libfreenect2 [14, 15] is used here, but
it does not provide parallel registration of the color and depth
image. The Jetson TX2 has relatively weaker CPU comparing
with the mainstream CPU processors. To improve the real
time performance of the algorithm and for the convenience of
robot detection to be discussed later, we parallelized the image
registration and depth point cloud production on the GPU.
(a) (b)
(c) (d)
Fig. 6 Image registration and depth point cloud production (a) RGB image (b)
Depth image (c) registered image (d) depth point cloud
The bottlenecks of this step implemented on the GPU are
that the amount of data to process is huge, the global storage
bandwidth is small, and the number of shared storage and
constant storage is limited. To get around the bottlenecks,
texture storage provided by the GPU architecture is utilized. It
is cached on the chip and provides more effective bandwidth
by reducing memory requests to off-chip DRAM. We use the
texture memory to store depth-to-color-mapping data. The
computation speed is greatly improved on each thread since
memory access time is significantly reduced. The registered
image and depth point cloud is shown in Fig. 6.
E. Noise Elimination
Sometimes we will get the false positive results because of
the blurred features and noise in the image. To avoid these
situations, a validation process is employed to decide whether
the box includes a real robot.
The robot’s size is limited in a fixed range, and all of the
robots have similar sizes. So, we can get the relationship
between the box’s size and the box’s center to the Kinect.
Using this knowledge, we estimate the number of pixels that a
robot box should have according to its distance to Kinect. We
can get the relation function as equation (1) in which the x
represents the distance and f(x) represents the box’s size. The
curve of f(x) is plotted in Fig. 7. Then we can eliminate the
box the size of which deviates much from the curve.
f ( x) 3.67e  09 * x  1.972
x 10
5 relations_between_ball_pixels_and_distance
6
5
4 conclude that we can detect visible robots in the MSL
ball_pixels
0
0 1000 2000 3000 4000
distance(mm)
Fig. 7 The relation between robot box’s size and its distance to the Kinect
sensor.
F. Get the 3D Positions
We get the most probable candidate robot’s regions after
validating the targets, and then we search every box’s center
position in the depth point cloud to get the 3D position of each
detected robot. If the box’s center has no depth information,
we adjust it by adding a small offset on the center.
At last, we can get the robot’s 3D position in the depth
point cloud as shown Fig. 8(b), where the grey ball represents
the detected robot’s center position.
.
(a)
Fig. 8 (a) Detected robots in registered image (b) The robots’ 3D positions in
depth point cloud.
III. EXPERIMENTAL RESULTS
This section presents quantitative experimental results
obtained by our approach. In order to test both the real-time
performance and measurement accuracy of robot detection and
localization, we used our NuBot soccer robots as the
experiment platforms which were equipped with Kinect v2
sensor and employed Jetson TX2 as the processor. Jetpack 3.0
was used here as the programming interface which uses
CUDA 8.0 for parallel computing and CUDNN 5.0 for the
CNNs acceleration.
A. Detection precision and mAP
As there is no other public dataset about MSL robots
available for us to measure our dataset performance, we tested
the dataset using the 400 testing images contained in the
dataset. The testing results are shown in TABLE I.
From the table we can find that the precision is
approximately to 90% which meets the detection accuracy
requirements in the RoboCup competition. The mAP results
indicate the good performance of the MSL robots datsaset. We
encourage other people to use our dataset and compare the
(1) algorithm performance based on the dataset in the robot vision
community. Furthermore, we tested the detection performance
ball_pixels vs. distance using some images acquired in the 2017 RoboCup
fit curve competition, Nagoya with different viewpoints, and the
detection results are shown in Fig. 9. From the results we can
competition using our proposed approach.
5000 6000 7000 8000
Fig. 9 Detection examples on the 2017 RoboCup competition
TABLE I. DATASETS PERFORMANCE
IOU Recall Precision mAP
72.76% 94.79% 89.09% 70.65%
(b)
B. Real-time performance after the parallel computing
The detection process includes two steps: image pre-
processing and prediction. The prediction process employs the
GPU parallel computing technique which leads to little
acceleration space because of the hardware restriction. The
image pre-processing process impacts the speed a lot, and the
computation time to process each frame of image is
approximately 58ms. We accelerated this process by
parallelizing it based on CUDA.
We tested the processing time of parallelized image pre-
precessing under different scenes in the robot soccer field. The
improved performance is shown in TABLE II. We can realize
real-time robot detection in the RoboCup MSL competiton
using such approach with the YOLO v2 architecture. The
statistics about the computing time in the image pre-
precessing after the acceleration and in the prediction process
are shown in Fig. 10, where the red lines represent the median
computing time, grey lines represent the maximal and minimal
computing time, and the blue boxes are the Box-whiskerPlot
acquired by Matlab.


Fig. 10 Statistics of processing time.
TABLE II. PROCESSING TIME AFTER ACCELERATION
Time Pre-processing Prediction
TX2 51-60 54-60
After acceleration 0-10 54-60
C. Lcalization error
After the accurate robot detection in the registered RGB
image, we need to localize the robots in the soccer field for
decision-making and obstacles avoidance procedures. The
accuracy of the localization has a great impact on the robot’s
competition performance during the game.
As mentioned above, we combined the depth information
with RGB information to get the detected robot’s 3D position
in the field. To measure the localization error, we set an
experiment in which two soccer robots are placed at positions
(-1660, 1800) and (1750, 3280) respectively in our soccer
field, and the robots’ positions are measured from the position
(0, 0) by a robot with the proposed detection and localization
system. The localization results are shown in Fig. 11 (red
points: detected robots’ positions, blue points: robots’ ground
truth positions, maroom point: NuBot’s position with a Kinect
sensor for robot detection and localization) and the average
Euclidean distance measurement error is shown in TABLE III.
Considering the physical dimensions of the robot
(approximately 52 cm*52 cm) and the error of placing the
robot at a specific position, the localization error (30.567 mm)
of our proposed system is acceptable.
robots localization
6000
5500
5000
4500
4000
3500
Z(mm)
3000
2500
2000
1500
1000
500
0 with NAO Soccer Robots," Robocup Symposium: Poster presentation.
-3000 -2500 -2000 -1500 -1000 -500 0 500 1000 1500 2000 2500 3000
X(mm)
Fig. 11 robot localization results in our soccer field
TABLE III LOCALIZATION ERROR (mm)
Robot positions mean min max
(-1660,1800) 24.140 0.568 77.967
(1750,3280) 36.994 0.832 181.65
8
IV. CONCLUSION AND FUTURE WORK
This paper proposed a novel approach used for the robot
detection and localization in the RoboCup MSL competition
using the Kinect v2 and Jetson TX2 as the hardware platform.
We built a dataset about the MSL robots as there is no
public dataset for the research of MSL robot detection and we
shared it on the github for the development in the robot vision
community. We accelerated the image pre-precessing process
in the YOLO detection system to make it meet the real-time
requirements of RoboCup MSL competition. More
importantly, we combined the detected robots’ 2D positions in
the RGB image with the depth point cloud to obtain the 3D
positions, so the accuracy of robot localization was improved.
We think the real-time performance will be further improved
FPS
if we use a more powerful GPU.
8.3-9.2
14.2-18.5 In the future, we will maintain and improve our dataset of
MSL robots and continue the research on the accurate and
real-time detection and localization of the robots, ball and
referee.
REFERENCES
[1] Neves, A.J., Pinho, A.J., Martins, D.A., and Cunha, B, "An Efficient
Omnidirectional Vision System for Soccer Robots: From Calibration to
Object Detection," Mechatronics. 21 (2), 2011, pp.399-410.
[2] Lu, H., Yang, S., Zhang, H., and Zheng, Z, "A Robust Omnidirectional
Vision Sensor for Soccer Robots," Mechatronics. 21(2), 2011, pp. 373-
389.
[3] MSL Technical Committee. "Middle Size Robot League Rules and
Regulations for 2017," 2017.
[4] LeCun, Yann, Yoshua Bengio, and Geoffrey Hinton. "Deep learning,"
Nature 521.7553, 2015, pp. 436-444.
[5] A. Krizhevsky, I. Sutskever, and G. Hinton, "ImageNet classification
with deep convolutional neural networks," In NIPS, 2012.
[6] J. Deng, A. Beng, S. Satheesh, H. Su, A. Khosla, and L. Fei-Fei, " Image
Net Large Scale Visual Recognition Competition 2012 (ILSVRC2012),
" 2012. http://www.image-net.org/challenges/ LSVRC/2012/.
[7] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, " ImageNet:
A large-scale hierarchical image database," In CVPR, 2009.
[8] Liu, Wei, et al, "Ssd: Single shot multibox detector," European
conference on computer vision. Springer, Cham, 2016.
[9] Ren, Shaoqing, et al, "Faster R-CNN: Towards real-time object
detection with region proposal networks," Advances in neural
detected robot 1 position information processing systems. 2015.
detected robot 2 position
robots groundtruth [10] Redmon, Joseph, et al, "You only look once: Unified, real-time object
positions detection," Proceedings of the IEEE Conference on Computer Vision
NuBot position and Pattern Recognition. 2016.
[11] Redmon, Joseph, and Ali Farhadi, "YOLO9000: better, faster, stronger,"
arXiv preprint arXiv:1612.08242, 2016.
[12] de Almeida Ribeiro, P.R., Lopes, G., and Ribeiro, F, "Neural Network in
Computer Vision for RoboCup Middle Size League," Journal of
Software Engineering and Applications. 9(07), 319, 2016.
[13] Albani, D., et al, "A Deep Learning Approach for Object Recognition
2016.
[14] Lawin, F.J., Forssén, P.-E., and Ovrén, H, "Efficient Multi-frequency
Phase Unwrapping Using Kernel Density Estimation. In European
Conference on Computer Vision," pp. 170-185. oct 2016.
[15] Lingzhu Xiang, F.E., Christian Kerl, Thiemo Wiedemeyer, Lars,
hanyazou, Alistair: libfreenect2: Release 0.2 [Data set],Zenodo.
http://doi.org/10.5281/zenodo.50641, 2016.