Proceedings of the IFAC Workshop on Navigation, Guidance

Proceedings
and Control ofof
Proceedings the
the IFAC
IFAC Workshop
ofUnderwater Vehicles on
Workshop on Navigation,
Navigation, Guidance
Guidance
Proceedings
and of the IFAC Workshop on Navigation, Guidance
and Control
April of
of Underwater
28-30, 2015.
Control Vehicles
Girona, Spain
Underwater Vehicles Available online at www.sciencedirect.com
and Control
April 28-30, of Underwater
2015. Girona, Vehicles
Spain
April 28-30, 2015. Girona, Spain
April 28-30, 2015. Girona, Spain
ScienceDirect
IFAC-PapersOnLine 48-2 (2015) 008–013
A
A benchmarking
benchmarking perspective
perspective of
of underwater
underwater
A
A benchmarking
benchmarking perspective
perspective
intervention of
of underwater
systems underwater
intervention
intervention systems
systems 
∗
intervention
∗ ∗
systems∗ ∗
P. J. Sanz ∗ J. Pérez ∗ J. Sales ∗ A. Peñalver ∗ J. J. Fernández ∗
P.
P. J. Sanz
Sanz ∗ J.D.Pérez ∗ J. ∗ Sales ∗ A. ∗Peñalver ∗ J. J. ∗ Fernández
Fernández ∗
P. J.
J. Sanz ∗ J. J.D.Pérez
Fornas
Pérez
Fornas
∗ J. Sales
J.∗∗ R.
Sales
R.
∗ A. ∗Peñalver
Marı́n
Marı́n
J. C. Garcı́a
A. ∗Peñalver
J. C. Garcı́a
∗ J. J.
J. J.∗ Fernández ∗
∗
D. Fornas ∗ R. Marı́n ∗ J. C. Garcı́a ∗
D. Fornas R. Marı́n J. C. Garcı́a
∗
∗ Computer Science and Engineering Department,
∗ Computer Science and Engineering Department,
∗ Computer Science and
University
University
of Jaume-I,
Computer
of Jaume-I, Science and Engineering
Castellón,
Castellón,
Spain. (e-mail:
Engineering
Spain.
Department,
sanzp@uji.es)
Department,
(e-mail: sanzp@uji.es)
University
University of Jaume-I, Castellón, Spain. (e-mail: sanzp@uji.es)
of Jaume-I, Castellón, Spain. (e-mail: sanzp@uji.es)
Abstract: This paper presents recent progress concerning benchmarking issues in the underwa-
Abstract:
roboticsThis
Abstract: This paper
paper presents recent progress
a veryconcerning benchmarking issues in the underwa-
ter
Abstract:
ter robotics paper presents
manipulation
This
manipulation
context.
presents
context.
recent
recent After
After
progress
progress
aa very
concerning
concerning
intensive
benchmarking
intensive 6-years period of
benchmarking
6-years period
issues
work,
issues
of work,
in the
the underwa-
in under
under
several
underwa-
several
ter
ter robotics
funding
robotics manipulation
research projects,
manipulation context.
all of
context. them After
After in the
a very intensive
aforementioned
very intensive 6-years
area,
6-years period
a strong
period of
of work,
know-how
work, under
under hasseveral
been
several
funding
funding research
research projects,
projects, all
all of
of them
them in
in the
the aforementioned
aforementioned area,
area, a
a strong
strong know-how
know-how has
has been
developed.
funding
developed.
As
research
As
part
part
of
projects,
of
this
this
expertise,
all of themainnew
expertise,
a new underwater
theunderwater
aforementioned simulation tool
area, atool
simulation
has
strong
has
been
know-how
been has been
implemented.
implemented. been
developed.
This platform
developed. As
As part
enables
part of
of this
the expertise,
this integration,
expertise, aa new underwater
simulation,
new underwater simulation
comparative
simulation tool
analysis
tool has
has and been
been implemented.
experimentation
implemented.
This
This platform
platform enables
enables the
the integration, simulation,
of the comparative
results, usinganalysis as inputand experimentation
on
This
on
real
real
data and
platform
data and a the integration,
a detailed
enables detailed
characterization
integration,
characterization
simulation,
simulation,
of the
comparative
comparative
results, using
analysis
analysis
as input
and
and
aa has
experimentation
a simple web-based
experimentation
simple web-based
on
user
on real
real data
interface.
data and
In
and a
a detailed
fact, our
detailed characterization
previous experience
characterization of
of the
in
the results,
related
results, using
research
using as
as input
projects
input a simple
simple web-based
evidenced
web-based the
user
user interface.
interface.
necessity of In
In
testing, fact,
fact, our
our
comparing previous
previous
and experience
experience
evaluating in
in related
related
different research
research
algorithms inprojects
projects
similar has
has evidenced
evidenced
conditions. So, the
the
user interface.
necessity of In fact,
testing, our previous
comparing and experiencedifferent
evaluating in related researchinprojects
algorithms similar has evidenced
conditions. So, the
the
necessity
similarity
necessity of
of testing,
between
testing, comparing
the virtual
comparing and
scenario,
and evaluating
where
evaluating different
the algorithms
benchmarking
different algorithms willin
in similar
be performed,
similar conditions.
conditions.and So,
the
So, the
real
the
similarity
similarity
one, will between
between
determine the
the
the virtual
virtual
quality scenario,
scenario,
of the where
where
results. the
the
In benchmarking
benchmarking
that sense, a will
will be
be
methodology performed,
performed,
is and
and
presented the
the real
real
which
similarity
one, will between the
determine the virtual
quality scenario,
of the where the
results. In benchmarking
that sense, a will be performed,
methodology is and the
presented real
which
one,
one, will
updates determine
will the scenariothe
determine with
the quality of
of the
the results.
real information
quality results. eachIn that
that sense,
In time a real aatrial
sense, methodology
is performed.
methodology is
is presented
Benchmarking,
presented which
which
aupdates
updates
very
updates
the
the
active
the
scenario
scenario
robotic
scenario
with
with
area
with
real
real
in
real
information
information
nowadays,
information is each
each
the
each
time
time aa real
underlying
time a real
real
trial
trial
problem
trial
is
is
is
performed.
performed.
to solve.
performed. In Benchmarking,
Benchmarking,
summary,
Benchmarking, the
a
a
mainvery
very active
active robotic
robotic
functionalities forarea
area in
in nowadays,
nowadays,
benchmarking is
is
availablethe
the underlying
underlying
in the problem
problem
simulation to
to
platform solve.
solve. will In
In besummary,
summary,
highlighted, the
the
a veryfunctionalities
main active roboticfor area in nowadays,
benchmarking is the underlying
available in the problem
simulation to solve.
platform will Inbe summary,
highlighted, the
main
by
main functionalities
using some case
functionalities for
studiesbenchmarking
concerning
for benchmarking available
object
available in the
tracking simulation
under
in the under
simulation platform
visibility
platformchanges,will be highlighted,
object
will object tracking
be highlighted,
by
by using
using some
some case studies concerning object tracking visibility changes, tracking
under
by using
under some case
variable
variable
waterstudies
case
water
currentconcerning
studies
current concerning
and 3D
object
object tracking
and 3D reconstruction
reconstruction tracking under
subject
underto
subject to
visibility
differentchanges,
visibility
different
optical object
changes,
optical object tracking
conditions.
tracking
conditions.
under
under variable
variable water
water current
current and
and 3D
3D reconstruction
reconstruction subject
subject to
to different
different optical
optical conditions.
conditions.
© 2015, IFAC (International Federation of Automatic Control) Hosting by Elsevier Ltd. All rights reserved.
Keywords: Underwater simulator; Benchmarking; Underwater intervention; Robotics.
Keywords: Underwater
Keywords: Underwater simulator; Benchmarking; Benchmarking; Underwater intervention; intervention; Robotics.
Keywords: Underwater simulator; simulator; Benchmarking; Underwater Underwater intervention; Robotics. Robotics.
1. INTRODUCTION Regarding benchmarking in robotics, a big effort has
1. INTRODUCTION
1. INTRODUCTION Regarding
been made benchmarking
Regarding benchmarking in
in robotics,
robotics, aa big effort has
1. INTRODUCTION Regarding
been made
over the last few
benchmarking
over the last few in years.
robotics,
years. In a big
In fact,
fact,
effort
bigsome
effort
some
has
recent
has
recent
been
European made
During the last 6-years period (i.e. 2009-2014) our re- been made over the last few years. In fact, some recent over
projects,the last
like few
FP7-BRICS years. In fact,
(Best some recent
Practice in
During group
the lastlast 6-yearshas period (i.e. 2009-2014) our re- re- European
European
Robotics), projects,
projects,
were devoted like
like FP7-BRICS
FP7-BRICS
to this specific (Best
(BestcontextPractice
Practice
(Nowak in
in
During
search
During the
the last 6-years
IRS-Lab
6-yearshas period
been
period (i.e.
very
(i.e. 2009-2014)
active
2009-2014) working our
our re- in European projects, like toFP7-BRICS (Best Practice in
search group IRS-Lab been very active working in Robotics),
Robotics),
et al., 2010).were
were devoted
devoted
Moreover, to this
this
following specific
specific
previous context
context (Nowak
(Nowak
research in
search
the
search group
underwater
group IRS-Lab
robotics
IRS-Lab has
has been
manipulation
been very
very active
field
active working
under
working three in
in Robotics), wereMoreover,
devoted to this specific context (Nowak
the underwater
underwater robotics manipulation field under three et
et
this al.,
al., 2010).
2010).
context Moreover,
(DEXMART, following
following previous
2009), it previous
is clear that: research
research
“In thein
in
the
different
the research
underwater robotics
projects:
robotics manipulation
RAUVI
manipulation (Sanz field
et
field al.,under
2010)
under three
and
three et al., 2010). Moreover, following previous research in
different research projects: RAUVI (Sanz et al., 2010) and this
this
domain context
contextof (DEXMART,
(DEXMART,
robotics research, 2009),
2009),
it is it
it is
is clear
clear
extremely that:
that:
difficult“In
“In the
the
not
different
TRITON research
(Sanz
different research etprojects:
al., RAUVI
2013a), funded (Sanz
by et al.,
Spanish 2010)
Ministry, and this context (DEXMART, 2009), it is clear that: “In the
TRITON (Sanz et etprojects:
al.,(Sanz RAUVI
2013a), funded (Sanz et al., 2010)
by Spanish
Spanish Ministry, and domain
domain of
of robotics research, it is extremely difficult not
TRITON
and
TRITON (Sanz
(Sanz et al.,
FP7-TRIDENT 2013a),
al.,(Sanz
2013a), etfunded by
al., 2013b),
funded funded
by Spanish by the only
Ministry,
Ministry, domain
only to of robotics
to compare robotics
compare
research,
results
research,
results from
it
it is
from different is extremely
different
approaches,
extremely
approaches,
difficult
difficult
but
not
but also
not
also
and
and FP7-TRIDENT
FP7-TRIDENT
European Commission. (Sanz
All et
et
these al.,
al., 2013b),
2013b),
projects funded
funded
have been by
by the
the
coor- only
to to
assess compare
the results
quality of from
the different
research. approaches,
This is but
especially also
true
and FP7-TRIDENT (Sanz et al., 2013b), funded by the only
to to compare
assess the results
quality of from
the differentThis
research. approaches,
is but true
especially also
European Commission. All these
these projects have complexity
been coor- coor- if to one
assess the
European
dinated
European withCommission.
several partners
Commission. All and,projects
All these have
with a high
projects been
been coor- to
have complexity if assess
one the quality
wishes
wishes quality
to
of
of the
to evaluate
evaluate the research.
the performance
research.
the
This
This is
performance is especially
especially
of
true
of intelligent
true
intelligent
dinated
dinated
in both, with
with several
several
hardware partners
partners
and and,
and,
software with
with a
a
components.high
high complexity
Moreover, if one
robotic wishes
systems to evaluate
interacting the
with performance
the real of
world.” intelligent
Several
dinated with severaland partners and,components.
with a high complexity if one
robotic wishes
systems to evaluate
interacting the
with performance
the real of
world.” intelligent
in
in both,
both, hardware
hardware and software Moreover, robotic
dealing definitions systems interacting
of benchmarks have with beenthe real
real world.”
theproposed, but Several
Several
we will
these
in
these
projects
both, hardware
projects were and software
were targeted
targeted
to common
software
to
components.
common
objectives
components.
objectives
Moreover,
Moreover,
dealing
robotic
definitions
definitions
systems
of
of
interacting
benchmarks
benchmarks have
have
with been
been proposed,
proposed,
world.”but
but
Several
we
we will
these
with
these projects
underwater
projects werewere targeted
intervention to
targeted tosystems,common
systems,
common to objectives
to be
objectivesvalidated dealing
dealing in use the
definitions
use the
one
one
stated
of stated
benchmarks at
at
Dillman
have been
Dillman
(2004): “adds
proposed,
(2004): “adds but we will
numerical
numericalwill
with
with
sea underwater
underwater
conditions intervention
at intervention
the end. systems, to be
be validated
validated in
in use the
evaluation one of stated
results at Dillman
(performance (2004): “adds
metrics) asnumerical
key ele-
with underwater intervention systems, to be validated in use the oneofstated
evaluation results at(performance
Dillman (2004): “addsasnumerical
metrics) key ele-
sea conditions at the end. evaluation of results (performance metrics) as
sea
sea aconditions
As conditions
consequence,
at
at the
the end.
the partners need to be sure that evaluation
allend.
ment.
ment.
The
The
main
of results
main
aspects
aspects
are
(performance
are
repeatability,
metrics)
repeatability, as key ele-
key
independency,
independency, ele-
As a consequence, all the partners need to be sure that ment.
and
ment. The main
unambiguity”.
The main aspects
A short
aspects are
are repeatability,
state-of-the-art
repeatability, independency,
can be found
independency,
As
As a consequence,
their part of the system
theira consequence,
partwhenof the
the
all
all the
system
partners
theand their need
partners
and their need
to
to be
algorithms
algorithms
sure
be will
sure work
will that and
that
work and unambiguity”.
(Pérez et A
and unambiguity”.
elsewhere
unambiguity”. al.,short
A
A short
2014b).
short
state-of-the-art
state-of-the-art
state-of-the-art can
can be
can be
found
be found
found
their
properlypart
their partwhen of
of the the system
system and
is their
assembled algorithms
and will
tested. work
With elsewhere
elsewhere (Pérez
(Pérez et
et al.,
al., 2014b).
2014b).
properly
properly
this aim, when
a thesystem
the
simulator system
system
that
and
is
allows
their algorithms
is assembled
assembled
the and tested.
and
researchers
will With
tested. to
work elsewhere (Pérez et al., 2014b).
With
intro- In order to simulate the experiments, the UWSim simula-
properly
this aim,
aim, when the system is assembled and tested. With In In order to etsimulate the experiments,
experiments, the UWSim
UWSim simula-
this
duce
this aa
the model
aim, a
simulator
simulator
simulator
that
of thethat whole
that
allows
allows
system
allows
the as
the
the
researchers
researchers
well as a to
researchers
to
to
intro-
intro-
realistic
intro- In order
tor (Prats
order to
to etsimulate
al., 2012)
simulate the and a benchmarking
the experiments, the
the UWSim toolsimula-
which
simula-
duce the model of the whole system as well as a realistic tor
tor
is (Prats
(Prats
highly et al.,
al.,
integrated 2012)
2012) with and
and the aa simulator
benchmarking
benchmarking were tool
tool which
which
developed
duce
duce the
scenario
the model
for
model of
testing
of the
the whole
their
whole system
algorithms
system as
aswaswell
well as
as a
considered
a realistic
realistic an tor (Prats et al., 2012) and a benchmarking tool which
scenario for testing their algorithms was considered an is
is
(seehighly
highly
Figure integrated
integrated
1). Moreover, with
with athe
the simulator
simulator
methodology were
were
that developed
developed
allows re-
scenario
extremely for testing
important
scenario forimportant their
testing their tool. algorithms
In addition
algorithms was
wasto considered
the simulator,
considered an
an (seeis highly integrated with the simulator were developed
extremely tool. In addition to the
the simulator, (see Figure
Figure 1).
1). Moreover,
Moreover, aa methodology
methodology that
that allows
allows re-
extremely
benchmarking
extremely important
facilities
important tool.
can
tool. In
Inhelpaddition
the
addition to
researchers simulator,
to the simulator,to com- searchers
(see to
Figureto 1).
searchers
work
work
in
Moreover,
in
different
different
conditions
a methodology
conditions
and
that
and allows re-
increasing
increasingre-
benchmarking
benchmarking
pare facilities
facilities
different algorithms can
can help
andhelp
better the
the researchers
researchers
understand to
to
their com-
com- searchers
lim- gradually to work
thework in different
level inof different conditions
difficultyconditionshas been and and increasing
designed. This
benchmarking facilities can help the researchers to com- searchers to
gradually the level of difficulty has been increasing
designed. This
pare different
different algorithmsfor andenabling
better understand
understand their lim- lim- methodology
gradually the level of
pare
itations
pare different algorithms
and robustness,
algorithmsfor and better
andenabling their improvement.
better understand their
their lim- gradually methodology the also
also of difficulty
levelhelps
helps
to improve
difficulty
to improve
has been
has the
the
designed.
beenscenarios
designed.
scenarios
forThis
for
the
This
the
itations and robustness,
robustness, their improvement.
improvement. methodology also helps to improve the scenarios for
itations and for enabling
itations and robustness, for enabling their improvement.their benchmarking,
methodology
benchmarking,
thus
also
thus
obtaining
helps to improve
obtaining
increasingly a
the scenarios
increasingly a
more
more for the
realistic
the
realistic
 This work was partly supported by Spanish Ministry of Re- benchmarking,
one.
benchmarking, thus
thus obtaining
obtaining increasingly
increasingly aa more
more realistic
realistic
 one.
one.
 This
search work
and was
was partly
Innovation supported
supported by
DPI2011-27977-C03 by Spanish
(TRITON Ministry
Project), of
of Re- by one.
 This work
This and
work
partly
was partly
Spanish
supported by Spanish
Ministry
Ministry of Re-
Re- The proposed methodology can be summarized in Fig-
search
Foundation Innovation DPI2011-27977-C03
Caixa Castelló-Bancaixa,
search and Innovation DPI2011-27977-C03 (TRITON
Universitat
(TRITON Jaume Project),
I grants
Project), by
by The proposed methodology can be asummarized in Fig-
search and
Foundation Innovation DPI2011-27977-C03 (TRITON Project), by The
ure
The 2. proposed
This figure
proposed methodology
is representing
methodology can
can be be asummarized
roadmap enabling
summarized in
in Fig-
Fig-
Foundation Caixa
PI.1B2011-17 and Castelló-Bancaixa,
Caixa PID2010-12, Universitat
Castelló-Bancaixa, Universitat
JaumeJaume
Universitat I PhDII grants
Jaume grants ure
ure 2.
2. This
This figure
figure is
is representing
representing a roadmap
roadmap enabling
enabling
Foundation Caixa
PI.1B2011-17 Castelló-Bancaixa, Universitat Jaume I grants the experimental validation, independently of the kind of
PI.1B2011-17 and
PREDOC/2012/47 and PID2010-12,
PID2010-12, Universitat
and PREDOC/2013/46,
Universitat and Jaume
Jaume II PhD
PhD grants
by Generalitat Va-
grants ure
the 2. This figure
experimental is representing
validation, a roadmap
independently of the enabling
kind of
PI.1B2011-17 and PID2010-12,
PREDOC/2012/47
lenciana PhD grant and and Universitat and
PREDOC/2013/46,
ACIF/2014/298.
Jaumeby I PhD grants
Generalitat Va- the
the experimental
running intervention
experimental validation,
(e.g.
validation, independently
“search &
independentlyrecovery”, of the
of theor kind
“panel
kind of
of
PREDOC/2012/47 PREDOC/2013/46, and by Generalitat Va- running intervention (e.g. “search & recovery”, or “panel
PREDOC/2012/47
lenciana PhD grant and PREDOC/2013/46,
ACIF/2014/298. and by Generalitat Va- running intervention (e.g. “search & recovery”, or “panel
lenciana PhD grant ACIF/2014/298.
lenciana PhD grant ACIF/2014/298. running intervention (e.g. “search & recovery”, or “panel
Copyright
2405-8963 ©© 2015,
IFAC 2015
IFAC (International Federation of Automatic Control)8 Hosting by Elsevier Ltd. All rights reserved.
Copyright
Copyright
Peer review© IFAC
©under 2015
IFAC responsibility
2015 of International Federation of 8
8 Control.
Automatic
Copyright © IFAC 2015 8
10.1016/j.ifacol.2015.06.002
 P. J. Sanz et al. / IFAC-PapersOnLine 48-2 (2015) 008–013 9

able comparison between different algorithms that share

a common robotic platform. For this reason, the tool has
modelling capabilities that enable scenario customization.
Different use cases will be explained also, where the fea-
tures of the benchmarking infrastructure are shown in
detail.
The next section describes the UWSim architecture, de-
tailing the benchmarking module and scenarios. Then, in
Section 3 the different use cases will be explained and some
conclusions are provided in Section 4.

2. UWSIM: A 3D UNDERWATER SIMULATION

TOOL FOR BENCHMARKING
Fig. 1. UWSim underwater simulation: panel manipulation
scenario. Girona 500 I-AUV in water pool conditions UWSim 1 (Prats et al., 2012) is an open source software
manipulating an underwater panel using the ARM5E tool for visualization and simulation of underwater robotic
light-weight robotic arm. missions that offers benchmarking capabilities through
a specific module (see Figure 1). The software is able


to visualize underwater virtual scenarios that can be
configured using standard modeling software and can be
connected to external control programs by using the ROS

interfaces. UWSim is currently used in different ongoing
 
 

projects funded by European Commission (MORPH 2 and
PANDORA 3 ) in order to perform HIL (Hardware in the
Loop) experiments and to reproduce and supervise real
missions from the captured logs.
 

 


The simulator has been implemented in C++ and makes
use of the OpenSceneGraph 4 (OSG) and osgOcean 5 li-


braries. OSG is an open source 3D graphics application


 


 
!

Fig. 2. Cyclic development methodology for continuous
integration, enabling the experimental validation, in-
dependently of the kind of running intervention.
intervention”). First of all, the software is developed and
the hardware is designed and modeled to be used in the
UWSim. Then, using the simulator and the benchmarking
module described below, the system is tested virtually,
evaluating its robustness as well as its limitations. Sec-
ondly, a water tank is used to check each part of the
system individually in a real but controlled scenario, where
problems like salt water or underwater currents do not
appear. For the third step, the whole system is integrated
and tested in a water pool. This scenario is also controlled,
but some problems like underwater currents or different
kinds of visibility can artificially be produced. In the fourth
step, the system is drawn to the sea, where the final inter-
vention is executed. During the sea trials, information of
the scenario is collected to create a virtual scenario for the
simulator as similar as possible to the real environment.
Finally, the algorithms can be improved, or even new ones
can be implemented using the real information, and then
the cycle restarts.
The aim of this paper is to present our recent results gen-
erating a benchmarking tool for underwater intervention
contexts and a simulation tool which enables the suit-
programming interface (API) used by application devel-
opers in fields such as visual simulation, computer games,
virtual reality, scientific visualization and modeling. Os-
gOcean is another open source project that implements
realistic underwater rendering using OSG and was devel-
oped as part of an EU funded research initiative called the
VENUS project (Alcala et al., 2008).
The UWSim is divided into different modules (see Fig-
ure 3): there is a Core module in charge of loading the
main scene and its simulated robots; an Interface module
that provides communication with external architectures
through the Robot Operating System (ROS); a Dynamics
module that implements underwater vehicle dynamics; a
Physics module that manages the contacts between objects
in the scene; the osgOcean, in charge of rendering the
ocean surface and special effects; the GUI module, that
provides support for visualization and windowing toolkits;
an User Interface Abstraction Layer (UIAL), which is in
charge of improving the Human-Robot Interaction, the 3D
immersion using a Head-Mounted Display and filtering the
information to be shown to the user; and the benchmark-
ing module explained in the following section.
1 Available online: http://www.irs.uji.es/uwsim
2 FP7-MORPH “Marine Robotic System of Self-Organizing,
Logically Linked Physical Nodes (MORPH)” Available:
http://morph-project.eu
3 FP7-PANDORA, “Persistent Autonomy through learNing, aDap-
tation, Observation and Re-plAnning (PANDORA)” Available:
http://persistentautonomy.com
4 R. Osfield, D. Burns et al., Available:
http://www.openscenegraph.org
5 Available: http://code.google.com/p/osgocean
9
10 P. J. Sanz et al. / IFAC-PapersOnLine 48-2 (2015) 008–013

!" 
 
 



   

 #!" 
 $  

   






 
  

 


 
!  "
!"









 
  

 
 
%!%
' 
 


 

 
  

  

"
  


 

  
  
  !  "


( (
#
"  #
" 
)&
* 
)

Fig. 4. Benchmarking module flow diagram: a benchmark

$)

 $) +  )



%


&


$)
&
configuration is loaded into the benchmark module
and a scene is loaded into the simulator. Then, the
benchmark module produces some results that can
be logged for posterior analysis.
Fig. 3. UWSim modules diagram and its interconnec-
tions: Core, Interfaces, Physics, Dynamics, osgOcean, changes. This feature allows the possibility to not only
Graphical User Interface (GUI), User Interface Ab- testing algorithms in a fixed condition setup, but also in a
straction Layer (UIAL) and Benchmarks. range of possible scenarios, and checking the performance
of the software depending on the parameters variation.
2.1 Benchmarking module Measures and scene updaters can be controlled through
triggers to start and stop evaluation depending on events
A benchmarking module for UWSim is also available such as reaching a position, receiving a message or elapsed
(Pérez et al., 2013). This module uses ROS to interact time.
with other external software, as UWSim does. The ROS
interface permits users to evaluate an external program In order to increase the benchmarking module usability, a
which can communicate both with the simulator (which benchmarking web tool has been developed to execute and
can send commands to perform a task) and with the supervise different predefined benchmarks (Pérez et al.,
benchmarking module (which can send the results or data 2014b). This tool acts as an interface, allowing to easily
needed for evaluation). Detailed information on how to configure, execute and download results from a set of
configure and run a benchmark in UWSim can be found available benchmarks, using ROSBridge and ROSlibjs.
online 6 . The web for benchmarking is also available online 7 and
is currently under development to add brand new features
For the development of the module, two important objec- such as software uploading, benchmark definition, teleop-
tives were taken into account. The first one is to be trans- eration, etc.
parent to the user, in other words, that it does not require
major modifications to the algorithm to be evaluated. The 2.2 Real Benchmarking scenarios
other objective of the module is that it must be adaptable
to all kind of intervention tasks in the underwater robotic
The benchmarking procedure presented in this paper
environment.
makes use of virtual scenarios that describe the environ-
Benchmarks are defined in XML (eXtensible Markup ment in an accurate manner, including water conditions,
Language) files. Each file will define which measures are terrain, objects or even visibility, to face different prob-
going to be used and how they will be evaluated. This lems such as black-box recovery, panel intervention, etc.
allows the creation of standard benchmarks defined in Moreover, the use of information from real environments,
a document to evaluate different aspects of underwater adds to the benchmark unexpected situations that enrich
robotic algorithms, being able to compare algorithms the robotic experience. Real modelled scenarios offer much
from different origins. Each of these benchmarks will be more interesting environments to compare algorithms and
associated with one or more UWSim scene configuration to improve the research procedure.
files, being the results of the benchmark dependent on
Following the methodology previously mentioned, the sce-
the predefined scene. The whole process is depicted in
narios used for the benchmarking increase their realism at
Figure 4.
each iteration of the cycle. The first step for the generation
Two main aspects define a benchmark in UWSim: mea- of the benchmarking scenario, consists of modelling all the
sures and scene updaters. Measures are the different things previous knowledge related to the possible characteristics
to measure in a benchmark, for instance position error, of the real scenario. For this scenario, the limitations and
elapsed time or 3D reconstruction coverage. Scene up- robustness of the algorithms are obtained. Then, after each
daters create a controlled environment change through physical experiment in which real data is acquired, the vir-
the benchmark execution, allowing multiple tests to be tual scenario is updated with the new real information in
performed, such as visibility, water current force or light order to obtain the closest modelled scenario to the reality.
6 The UWSim Benchmarks Workspace. Available online: 7 The UWSim online Underwater Simulator for benchmarking.

http://sites.google.com/a/uji.es/uwsim-benchmarks. Available: http://robotprogramming.uji.es/UWSim/config.html

10
 P. J. Sanz et al. / IFAC-PapersOnLine 48-2 (2015) 008–013 11

 



 
 




Fig. 5. Scenario improvement: (top) initial scenario based
on previous knowledge; (bottom) scenario updated
with the real acquired data.
This updated scenario can be used to develop algorithms
that can cope with its real counterpart even in the simplest
feasible manner, avoiding oversized complexity. Moreover,
this scenario could be provided to the scientific community
that perhaps do not have the resources (e.g. robots) or the
permissions (environment) to benchmark their algorithms
in it.
As an example of the described benchmarking scenario
cycle, in the TRIDENT project, a first scenario (see
Figure 5 top) was developed where the kinematic model of
the robotic system and the black-box mockup dimensions
were equivalent to the real ones, while the environment was
a generic one. After the final sea trials carried out in Port
de Sóller (Mallorca, Spain; 1-5 Oct 2012), the previous
scenario was updated and improved with a 3D textured
terrain from the real acquired bathymetry data and the
obtained seafloor images (see Figure 5 bottom).
3. USE CASES FOR UNDERWATER
BENCHMARKING
The benchmarking functionality has recently been used
in three use cases: (1) tracking algorithms evaluation
11

 


 
 
Fig. 6. Software schema used in Visibility use case. Or-
ange: benchmark outputs, Green: evaluated software
available to configure online.
subject to water currents at different forces, (2) tracking
algorithms evaluation with varying degrees of visibility and
(3) 3D reconstruction methods under different visibility
and noise conditions. Finally, the reconstruction has also
been done in real conditions and then evaluated using
the simulator. Future use cases, such as manipulation
benchmarks, are still under development.
Each benchmarking use case is formed by two main
blocks and the benchmarked algorithm. The first block is
UWSim, which simulates the scene and provides ground
truth to the second main block benchmarking module.
This benchmarking module evaluates the software using
UWSim ground truth and modifies UWSim scene accord-
ing to benchmarking needs. Finally, the evaluated soft-
ware, that user should provide, may use a structure as
complex as it is required by the problem to solve. The
evaluated software acquires data from UWSim sensors as
it would do in a real scenario and it generates the results
for the benchmarking module.
3.1 Object tracking evaluation subject to visibility changes.
In Pérez et al. (2014a), a visibility benchmark for track-
ing algorithms is presented. The benchmark measures the
tracker error while following a target in a camera as can
be seen in Figure 6. The tracker must find target’s corners
and centroid of a target object as visibility conditions get
worse. The results will show the minimum visibility envi-
ronment that the tested algorithm requires to accurately
track an object.
In this experiment, many different tracker configurations
were tested, using as reference the ESM (Malis, 2004) and
ViSP algorithms (ViSP, 2010). This configuration includes
two similarity functions, four methods, and five different
warps. The goal of the study was to choose the best
algorithm for a tracking target and further manipulation.
Results showed that ZNCC (Zero mean Normalized Cross-
Correlation) trackers from ViSP were much better for
visibility changes while SSD (Sum of Square Differences)
ones, also from ViSP, are affected by them. Moreover, as
there was no movement at all restrictive warps performed
better than general ones. A larger discussion on results is
provided on Pérez et al. (2014a), and the whole experiment
can be repeated in the online benchmarking platform.
12 P. J. Sanz et al. / IFAC-PapersOnLine 48-2 (2015) 008–013

   

    
  
       
  ! $
    # 


! $   



" %  
   


  





!"  Fig. 8. Software schema used in the 3D reconstruction use
 case. Orange: benchmark outputs, Green: evaluated
'    software available to configure online.
!"   &
  be seen in Figure 8. The 3D reconstruction algorithms
# 
$%&  are evaluated while the light and noise conditions are


 
dynamically changed.

&
 
The stereo reconstruction is done using a block matching
algorithm that uses previously calibrated stereo cameras
Fig. 7. Software schema used in tracker current use case. to match points from the left and right cameras. The
Orange: benchmark outputs, Green: evaluated soft- displacement between each point pair is used to compute
ware available to configure online. the 3D coordinates of the corresponding point. With the
3D point list, a disparity map and a point cloud can be
3.2 Object tracking and control evaluation under current built.
variations.
The second algorithm requires the use of a laser stripe
projector attached to the forearm of a robotic arm (i.e.
In this use case, a controller and tracking algorithm is
the ARM5E manipulator) and a camera. Then, the elbow
evaluated in simulation. The goal of the experiment is to
link of the arm moves to scan the scene while the camera
maintain the vehicle (the Girona 500 I-AUV) in a relative
captures images. For each frame, a segmentation algorithm
position to a target (black-box), so it is able to start a
is used to detect the laser projection points. Using trian-
manipulation intervention. The metrics used are the error
gulation, the 3D position of each point can be estimated
on the tracked object, measuring the difference between
and used to build a point cloud.
the real object position and the tracker estimation, and
the vehicle positioning error with respect to the desired To compare these methods in real (see Figure 9) and
position. virtual environments, a precise model of the object was
used as a ground truth to develop 3D reconstruction
The experiment is available online 8 for further testing.
quality metrics. The algorithms were executed in this
The software architecture used to solve the problem can
scenarios where a box was placed in a flat surface. This
be observed in Figure 7. A tracker is used to find the
benchmark was done in simulation and in real hardware,
target in the camera, using that information the visual
more precisely, in a water tank with the ARM5E arm.
station keeper decides the optimal position, and position
The defined metrics used to compare the performance are:
and velocity controllers maintain the vehicle there.
mean box reconstruction error, standard deviation of the
In the experiments, several trackers were tested using box reconstruction error, box surface coverage and number
P (proportional) and PI (proportional-integral) position of outliers.
controllers under varying current force conditions. As
The results of the benchmark confirm that the laser re-
expected, PI achieved better results in every situation.
construction works better in darker environments, while
For a deeper results analysis Pérez et al. (2014b) can be
the stereo reconstruction requires objects with a well lit
consulted and, it is also available on the online platform.
textured surface to be reconstructed properly. Laser recon-
struction obtains good results in scenes with more light,
3.3 Evaluation of 3D Reconstruction techniques with and its performance increases with the darkness, being
varying light conditions slightly affected by noise. Although stereo reconstruction
gets the best results in ideal conditions, noise affects neg-
In the last use case, two 3D reconstruction methods are atively the reconstruction in a great extent and it can not
compared in order to obtain the best possible results before reconstruct the environment without decent illumination.
the manipulation stage of an underwater robot. The first
one consists of a stereo pair performing stereo reconstruc- 4. CONCLUSIONS
tion, while the second one reconstructs the scene using
a laser projector, a camera and segmentation techniques.
As it has been demonstrated along the aforementioned
The resulting point clouds have been evaluated under
projects, when complexity is very high and, a lot of human
different light conditions. The benchmark structure can
and different mechatronic resources must be combined to
8 The UWSim online Underwater Simulator for benchmarking. achieve the expected objectives, integration and bench-
Available: http://robotprogramming.uji.es/UWSim/config.html marking procedures are crucial aspects looking for succeed.

12
 P. J. Sanz et al. / IFAC-PapersOnLine 48-2 (2015) 008–013 13

 DEXMART (2009). Specification of benchmarks. In Deliv-


erable D6.1 from FP7-DEXMART Project (DEXterous

  and autonomous dual-arm/hand robotic manipulation
with sMART sensory-motor skills: A bridge from natural

 to artificial cognition. URL http://www.dexmart.eu.
 Dillman, R. (2004). KA 1.10 Benchmarks for Robotics
Research. Technical report, University of Karlsruhe.
Malis, E. (2004). Improving vision-based control using effi-
cient second-order minimization techniques. In Robotics
and Automation, 2004. Proceedings. ICRA ’04. 2004
IEEE International Conference on, volume 2, 1843 –
1848 Vol.2. doi:10.1109/ROBOT.2004.1308092.
Nowak, W., Zakharov, A., Blumenthal, S., and Prassler, E.

  (2010). Benchmarks for mobile manipulation and robust
obstacle avoidance and navigation. In Deliverable D3.1
 from FP7-BRICS Project (Best Practice in Robotics).
 URL http://www.best-of-robotics.org/home.
Fig. 9. Physical benchmarking system: water tank, Light-
weight ARM5E manipulator, stereo camera, laser
stripe projector and an object to be reconstructed and
grasped (i.e. an amphora).
Concerning the benefits of this platform, it is notice-
able that some current EU projects like PANDORA and
MORPH have been using this tool in their work plan. As
has been highlighted previously, the developed method-
ology has consisted on a simulated trial in first place,
secondly the test in the water tank scenario, and at the
end, in the seabed.
Moreover, the platform has been recently extended to in-
clude a benchmarking module, which allows the character-
ization of the results from the real scenario, and using these
datasets for further research, allowing the comparison of
different algorithms for the given real conditions of the
experiments.
Currently, the tool presents a web-based user interface
that enables the researcher to select the experimentation
scenario, while focusing on particular test algorithms (e.g.
2D vision-based grasping control, 3D object reconstruc-
tion, etc.). Moreover, the benchmarking tool incorporates
the possibility to add specific conditions to the testbed
scenario, such as low visibility and water currents.
As conclusion, the benchmarking tool has demonstrated to
be and excellent software for integration and experimental
validation, for both, virtual and real scenarios. It allows
the definition of ordered datasets from real experiments,
and their further use for comparing specific scientific al-
gorithms such as pattern recognition and 3D object re-
construction for manipulation, taking into account specific
visibility and current conditions.
Further work will focus on enhancing the tool to provide
more datasets and use cases that enable measuring and
comparison of specific robotics algorithms.
REFERENCES
Alcala, F. et al. (2008). VENUS (Virtual ExploratioN
of Underwater Sites) Two years of interdisciplinary
collaboration. In 14th International Conference on
Virtual Systems and Multimedia. Limassol, Cyprus.
13
Prats, M., Pérez, J., Fernández, J., and Sanz, P. (2012). An
open source tool for simulation and supervision of un-
derwater intervention missions. In Intelligent Robots and
Systems (IROS), 2012 IEEE/RSJ International Confer-
ence on, 2577–2582. doi:10.1109/IROS.2012.6385788.
Pérez, J., Sales, J., Marı́n, R., and Sanz, P.J.
(2014a). Web-based configuration tool for bench-
marking of simulated intervention autonomous un-
derwater vehicles. In Autonomous Robot Sys-
tems and Competitions (ICARSC), 2014 IEEE In-
ternational Conference on, 279–284. Espinho, Por-
tugal. doi:10.1109/ICARSC.2014.6849799. URL
http://dx.doi.org/10.1109/ICARSC.2014.6849799.
Pérez, J., Sales, J., Marı́n, R., and Sanz, P. (2014b).
Online tool for benchmarking of simulated intervention
autonomous underwater vehicles: Evaluating position
controllers in changing underwater currents. In 2014
Second International Conference on Artificial Intelli-
gence, Modelling and Simulation (AIMS2014), 246–251.
Madrid, Spain.
Pérez, J., Sales, J., Prats, M., Martı́, J.V., Fornas, D.,
Marı́n, R., and Sanz, P.J. (2013). The underwater
simulator UWSim: Benchmarking capabilities on au-
tonomous grasping. In 11th International Conference
on Informatics in Control, Automation and Robotics
(ICINCO).
Sanz, P.J., Peñalver, A., Sales, J., Fornas, D., Fernández,
J.J., Perez, J., and Bernabé, J.A. (2013a). GRASPER:
A multisensory based manipulation system for under-
water operations. In 2013 IEEE International Confer-
ence on Systems, Man, and Cybernetics (SMC). IEEE,
Manchester, UK.
Sanz, P.J., Prats, M., Ridao, P., Ribas, D., Oliver, G.,
and Orti, A. (2010). Recent progress in the RAUVI
project. a reconfigurable autonomous underwater vehi-
cle for intervention. In 52-th International Symphosium
ELMAR-2010, 471–474. Zadar, Croatia.
Sanz, P.J., Ridao, P., Oliver, G., Casalino, G., Petillot, Y.,
Silvestre, C., Melchiorri, C., and Turetta, A. (2013b).
TRIDENT: An european project targeted to increase
the autonomy levels for underwater intervention mis-
sions. In OCEANS’13 MTS/IEEE conference. San
Diego, CA.
ViSP (2010). Visual Servoing Platform. Available online:
http://www.irisa.fr/lagadic/visp/visp.html.