ELECTRONICS AND ELECTRICAL ENGINEERING

ISSN 1392 1215 20XX. No. X(XXX)

ELEKTRONIKA IR ELEKTROTECHNIKA
AUTOMATION, ROBOTICS
T 125
AUTOMATIA!IMAS, ROBOTECHNIKA
Real-Time 3D Hand Gestures Recognition for Manipulation of
Industrial Robots
T. Cerlinca, G. Pentiuc, . lad
Stefan cel Mare University of Suceava,
13 University Street, RO-720229 Suceava, Romania, phone !"0230#2"$01, e-mails tu%or&c'ee%(usv(ro)
pentiuc'ee%(usv(ro) vla%v'ee%(usv(ro
http://dx.doi.org/10.5755/j01.eee
Introduction
Nowadays, the Human Robot ntera!tion is a "ery
attra!ti"e resear!h area with mu#tip#e and $uite interesting
app#i!ations to the industry %ie#d. &he intera!tion between a
human operator and a roboti! system !an be a!hie"ed in
se"era# ways, as %o##ows: using the robot's interna# !ontro#
pane#, using a remote !ontro##er, through "oi!e !ommands,
through a dedi!ated so%tware app#i!ation whi!h a##ows
remote !ontro# and e"en through hand gestures. (s the
"oi!e !ommands and the hand gestures a##ow %or a natura#
!ommuni!ation, there is an in!reasing resear!h interest in
designing and de"e#oping new methods and a#gorithms that
wi## ma)e the intera!tion as re#iab#e as it !an be. n the
human*to*human !ommuni!ation, "oi!e and hand gestures
are pra!ti!a##y inseparab#e: the hand gestures a#ways
strengthen the spea)er's ideas+ moreo"er, peop#e
sometimes !ommuni!ate with ea!h other on#y through
hand gestures ,e.g.: (meri!an -ign .anguage/. n the HR
area, "oi!e and gestures are usua##y seen as independent o%
ea!h other, but some systems use a !ombination o% them,
aiming %or an in!reased re#iabi#ity. t has to be mentioned
that most o% the !urrent HR based systems are on#y meant
%or mo"ing the robot or its mobi#e arm %rom one p#a!e to
another. 0n#y a re#ati"e#y sma## number o% HR based
systems approa!h the prob#em o% !ontro##ing a robot
through hand gestures so that it %u#%i##s rea# tas)s. Hand
gestures re!ognition prob#em is not simp#e either in theory
or in pra!ti!e. 1asi!a##y, this prob#em is redu!ed to the
re!ognition o% the hand traje!tories whi!h are a!$uired in
rea# time. 2epending on the re$uirements o% the pra!ti!a#
app#i!ation, the hand gestures re!ognition !an be done in
32 or e"en in 42.
5urrent#y, there are many methods and a#gorithms %or
42 hand gestures re!ognition ,a good sur"ey is pro"ided in
61, 37/, but some o% them seem to ha"e on#y a pure#y
theoreti!a# nature and there%ore !annot be integrated into
spe!i%i! HR app#i!ations. 8or examp#e, some a#gorithms
assume that a## traje!tories ha"e the same #ength. t is
ob"ious#y, that su!h a !onstraint #imits the use o% these
a#gorithms. 8or instan!e, when a human operator ma)es
the same gesture 1000 times, the asso!iated traje!tories
wi## a#ways ha"e a di%%erent #ength. &his drawba!) !an be
o"er!ome by using a 2&9 ,2ynami! &ime 9arping/
based distan!e whi!h a#so assures time and speed
in"arian!e. 2&9 was su!!ess%u##y used in 647 to !ontro#
the mo"ement o% an omni*whee# robot through gestures.
:et, the system has %ew disad"antages as %o##ows: it is not
adapti"e ,%or di%%erent en"ironments, the thresho#d needs to
be manua##y set/ and the obje!t being dete!ted and tra!)ed
is not the human hand but a sma## red ba##.
n 6;7 the authors propose a HR based system %or
industria# robot manipu#ation through hand gestures. &hey
used .2& distan!e ,.e"enshtein 2istan!e on &raje!tories/
%or traje!tories mat!hing and H<<s ,Hidden <ar)o"
<ode#/ %or re!ogni=ing di%%erent a!tion se$uen!es. &he
hand dete!tion is !arried out using a spatio*tempora# 42
mode# o% the hand*%orearm #imb. Hand tra!)ing is a!hie"ed
with an impro"ed "ersion o% the !#assi!a# -hape 8#ow
a#gorithm. &he authors !#aim a >0? g#oba# re!ognition
rate. (n impro"ed "ersion o% the !#assi!a# 52&9
,5ontinuous 2ynami! &ime 9arping/ a#gorithm is
proposed in 657.
H<<s and 1ayes !#assi%iers were used in 6@7 to
!reate a 42 gesture re!ognition system that is !apab#e o%
re!ogni=ing 1@ di%%erent !#asses o% gestures per%ormed with
one or both hands. &o dete!t and tra!) the obje!ts o%
interest ,hands and head/, the system ana#y=es the !o#or
!omponent o% the 32 images and uses the A<
,Axpe!tation*maximi=ation/ a#gorithm.
( %u##y %un!tiona# HR based system whi!h re#ies on
a rea#*time 42 hand gestures re!ognition a#gorithm is
presented in 677. &he authors a#so propose a human
dete!tion method whi!h in!orporates s)in, %a!e and #eg
dete!tion. n 6B7 the authors propose a rea#*time 42
pointing gesture re!ognition a#gorithm whi!h a##ows %or a
natura# human*robot intera!tion. Hands are dete!ted on the
base o% the !o#or distribution o% the %a!e region and the
%a!e was dete!ted with an impro"ed "ersion o% the we##
)nown Cio#a and Dones dete!tor. &ra!)ing was a!hie"ed
with a 42 parti!#e %i#ter and the estimation o% the pointing
dire!tion re#ies on a !as!ade o% H<<s.
(n inno"ati"e haar#et*based 42 hand gestures
re!ognition system whi!h !ombines both RE1 and &o8
!ameras is presented in 6>7. n 6107 the authors !ombine
both gesture traje!tories and a!!e#eration signa#s aiming
%or a no"e# approa!h to 42 hand gestures re!ognition
prob#em. (nother no"e# approa!h whi!h !#aims to be "iew*
in"ariant is presented in 6117. (## gestures are mode#ed
through H<<s and their !orresponding traje!tories are
assumed to be in a p#ane whi!h is estimated through .east
-$uares <ethod. (!!ording to the authors, the re!ognition
rate ranges %rom >4.;? and >4.7?.
&his paper addresses the prob#em o% !ontro##ing an
industria# robot in a "ery natura# way, through 42 hand
gestures, so that it wi## be ab#e to %u#%i## rea# tas)s
!omprising o% a #ist o% spe!i%i! operations.
!"stem arc#itecture
&he manipu#ation o% the industria# robot through 42
hand gestures is a 7 stages pro!ess, as %o##ows: image
a!$uisition, obje!t dete!tion and 42 "irtua# en"ironment
!onstru!tion, 42 hands dete!tion and tra!)ing, traje!tory
smoothing, hand gesture re!ognition, obje!tsF manipu#ation
within the 42 "irtua# en"ironment and obje!tsF
manipu#ation by the industria# robot. &he system
ar!hite!ture is presented in 8ig. 1.
n the %irst stage, the system grabs a series o% depth
maps %rom the stereos!opi! !amera system and tries to
dete!t the obje!ts that wi## be manipu#ated by the industria#
robot. &he #ayout %or this app#i!ation !omprises o% a series
o% di%%erent obje!ts whi!h !an a#so be o% di%%erent shapes
,!uboids, !y#inders et!./ and si=es.
n the next stage, the system automati!a##y !reates a
42 "irtua# en"ironment in whi!h the user !an intera!t with
the dete!ted obje!ts through hand,s/ gestures. &he "irtua#
en"ironment is intended to be a pre!ise !opy o% the rea#
en"ironment in whi!h the industria# robot operates. 8or
!ertain app#i!ations ,e.g.: a %ood repository, where the
obje!ts are represented by big !ontainers with %ood and/or
drin)s/ it is hard i% not a#most impossib#e to dete!t a##
obje!ts at on!e, there%ore the #ayout needs to be manua##y
!onstru!ted.
0n!e the "irtua# en"ironment has been bui#t, the
human operator !an intera!t with it through the hand
gestures to a!!omp#ish the %o##owing main tas)s:
manipu#ation o% obje!ts ,se#e!tion, grabbing, mo"ing and
#ea"ing/, rotation o% the 42 s!ene and =oom in/out. &he
intera!tion ends when the human operator per%orms a
spe!i%i! gesture with both hands. (t this point, the system
iterates through the #ist o% operations per%ormed by the
human operator in the "irtua# en"ironment and !reates a
series o% spe!i%i! !ommands that wi## be trans%erred to the
industria# robot. 9hen the robot has %inished re!ei"ing the
#ist o% operations that need to be done it wi## immediate#y
start operating on the rea# #ayout, mo"ing the rea# obje!ts
%rom one p#a!e to another. 2uring this time, the system
wi## %ree=e the intera!tion with the 42 "irtua# en"ironment.
ntera!tion wi## start again on!e the robot !omp#eted the
tas) and the system re!ei"es a !on%irmation in this regard.
$ig. %. -ystem ar!hite!ture
&he !ommuni!ation between the main system and the
industria# robot is done "ia a spe!ia#i=ed midd#eware. &his
way, the system does not need to )now anything about the
robot and the way it operates. t !an be any robot whi!h is
!apab#e o% grabbing, mo"ing and #ea"ing obje!ts at spe!i%i!
!oordinates.
Image ac&uisition
(## the a#gorithms that we ha"e imp#emented within
the industria# robot manipu#ation system ,gestures
re!ognition, 42 obje!t dete!tion et!./ pro!ess depth maps
,42 in%ormation/ grabbed %rom a stereo"ision !amera
system. n the experiments we ha"e !ondu!ted, two
di%%erent "ideo input de"i!es were used. &he %irst one is
1umb#ebee3 AAA*14>; 8ire9ire stereo"ision !amera
system ,pro"ided by Goint Erey Resear!h/ and the se!ond
one is the <i!roso%t Hine!t sensor.
3D 'b(ect detection
(s we a#ready mentioned, the system we ha"e
de"e#oped automati!a##y dete!ts a## the obje!ts that wi## be
manipu#ated by both the human operator ,in the 42 "irtua#
en"ironment/ and the industria# robot. In#i)e the !#assi!a#
a#gorithms, ours not on#y that it dete!ts 42 obje!ts but it
a#so pro!esses on#y the depth in%ormation grabbed %rom a
stereos!opi! !amera system. &he depth map !ontains
in%ormation re#ated to the distan!e between di%%erent
obje!ts in the s!ene and the stereos!opi! !amera system.
&he a#gorithm !onsists o% three stages, as %o##ows:
depth map grabbing+
42 region growing. &his is an origina# 42 extension to
the !#assi!a# region growing a#gorithm whi!h operates
on the !o#or !omponent o% the 32 images. ( detai#ed
des!ription o% this a#gorithm !an be %ound in 6137. n
this stage, the a#gorithm dete!ts a## the obje!ts in the
repository, no matter what shape they may ha"e. 8or
ea!h obje!t, the a#gorithm dete!ts the top*side shape
and the distan!e to the ground. t has to be mentioned
that the s!ope o% this a#gorithm is not to pro"ide a %u##
42 obje!t re!onstru!tion, but a basi! 42
representation o% ea!h dete!ted obje!t so that the
industria# robot wi## )now how to wor) with it+
shape dete!tion. n this stage, the a#gorithm dete!ts a##
the !uboids and !y#indri!a# obje!ts. &he dete!tion
wor)s %ine no matter i% the obje!ts are rotated or not.
-hape dete!tion was a!hie"ed with the a#gorithm
pro"ided by the 0pen5C ,0pen 5omputer Cision/
#ibrary.
&his approa!h has %ew important ad"antages, as %o##ows:
an obje!t whi!h shows an une"en distribution o% the
!o#or !omponent wi## be !orre!t#y dete!ted and #abe#ed
6137 and not sp#it into se"era# sma##er obje!ts+
i% dea#s with !hanges o% #ighting+
it gi"es a basi! 42 representation o% ea!h dete!ted
obje!t as %o##ows:
( ) ( ) { }
i i i i i i i i
*+ , y - , y - S , , , , , , ,
3 3 1 1
,1/
where:

i
S is the shape o% the obje!t ,!uboid or !y#inder/+
( )
i i i
, y - , ,
1 1
and ( )
i i i
, y - , ,
3 3
are the 42
!oordinates %or the upper*#e%t and #ower*right
!orners o% the top*side shape+

i
*+ is distan!e to the ground+
&he resu#ts o% this stage are presented in 8ig. 3. &he
%irst image shows an examp#e o% the robotFs wor)ing
en"ironment: the board on whi!h se"era# !uboids are
p#a!ed at di%%erent !oordinates. &he se!ond image shows
the %i#tered depth map and the third one, the top*side shape
dete!tion. &he %ourth image shows the 42 "irtua#
en"ironment whi!h was bui#t on the base o% the dete!ted
obje!ts.
$ig. ). 0bje!t dete!tion. 42 Cirtua# en"ironment
3D Hand detection and trac*ing. Tra(ector" smoot#ing
&his is probab#y one o% the most important stages
be!ause a## subse$uent stages depend on the resu#ts
pro"ided by this one. 8or instan!e, i% hands are not %ast and
a!!urate#y dete!ted and tra!)ed, their !orresponding
traje!tories wi## be a%%e!ted by errors and there%ore, the
gesture re!ognition a#gorithm may %ai# to !orre!t#y
re!ogni=e the gestures per%ormed by the human operator.
&hus, a re#iab#e 42 hand tra!)er is needed. (s we a#ready
mentioned, we wor)ed with two di%%erent stereo"ision
!amera systems and there%ore with two di%%erent tra!)ers
,one %or the 1umb#ebee3 and another one %or <i!roso%t
Hine!t sensor/.
1oth tra!)ers pro"ide the 42 position o% the hands in
rea# time. &hey are a#so !apab#e o% remo"ing the %a#se
dete!tions by !onstant#y ana#y=ing the hands traje!tories.
&he dete!tion is !onsidered to be erroneous when the
%o##owing !ondition is %u#%i##ed:
( ) .hMa-*ist /01 /01 **ist
prev curr
t t
< 4
,3/
where:

curr
t
/01
is the !urrent 42 hand position+

prev
t
/01
is the 42 hand position %or the #ast
su!!ess%u# dete!tion+
.hMa-*ist is the maximum a##owed distan!e
between the 42 !oordinates o% two !onse!uti"e
dete!tions. &his thresho#d is not stati! but dynami!a##y
!omputed on the base o% the timestamp prev curr
t t
.
&he %irst a#gorithm, whi!h promotes one o% our
origina# ideas is main#y based on a !#assi!a# 32 image
segmentation a#gorithm name#y, region growing 6137. &he
a#gorithm is !apab#e o% dete!ting the hands %rom #ong
distan!es and it a#so dea#s with !hanges o% #ighting 6137.
(#though it pro"ides good resu#ts, the a#gorithm has a
sma## disad"antage: it needs an additiona# stage o% head
dete!tion and tra!)ing whi!h may s#ow down the entire
pro!ess. ( detai#ed des!ription o% this a#gorithm !an be
%ound in 6137.
&he se!ond a#gorithm whi!h wor)s with <i!roso%t
Hine!t sensor pro"ides the best resu#ts when the distan!e
between the human operator and the sensor ranges %rom
1.3 to 4.5 meters. % the human operator is #o!ated too !#ose
to the sensor the resu#ts tend to be noisy. &he dete!tion
resu#ts are shown in 8ig. 4 and 8ig. ;.
$ig. 3. Hands dete!tion. Axamp#e no. 1
$ig. +. Hands dete!tion. Axamp#e no. 3
9e a#so ana#y=ed the 42 traje!tories a!$uired in rea#
time by both a#gorithms and %or di%%erent gestures and it
seems the se!ond a#gorithm pro"ides s#ight#y more
a!!urate resu#ts. Howe"er, the hand traje!tories are not yet
suitab#e as inputs %or the gesture re!ognition a#gorithm. n
order to smooth the traje!tories, in the next stage, we app#y
a 42 Ha#man %i#ter. &he resu#t o% the smoothing pro!ess %or
a simp#e traje!tory whi!h was a!$uired in rea# time is
shown in 8ig. 5. &he origina# traje!tory is mar)ed with red
points and the smoothed one with b#ue ones.
$ig. ,. &raje!tory smoothing
3D Hand gestures recognition
9ithin the %ramewor) o% our system, the hand
gestures are used %or two di%%erent purposes, as %o##ows: to
manipu#ate di%%erent obje!ts in the 42 "irtua# en"ironment
and to initiate/!omp#ete a spe!i%i! a!tion. (s %or the %irst
purpose, we ha"e imp#emented a 42 "irtua# mouse whi!h
operates in the "irtua# en"ironment and is entire#y
!ontro##ed through hands. n order to be ab#e to dea# with
the three basi! operations ,hand mo"ement within the
"irtua# en"ironment, se#e!ting/mo"ing an obje!t and
#ea"ing the se#e!ted obje!t at the !urrent !oordinates o% the
"irtua# mouse/ we ha"e used 4 di%%erent 42 "irtua# mode#s
o% the hand, as shown in the 8ig. @. (n examp#e o% how an
obje!t !an be manipu#ated within the "irtua# en"ironment
is gi"en in 8ig. 3.
$ig. -. 42 "irtua# mode#s o% the hand
&o !ope with the se!ond goa# ,se#e!tion o% an a!tion/
we ha"e %irst !reated a di!tionary o% B di%%erent gestures
and then !reated a training set by a!$uiring 400 samp#e
hand traje!tories %or ea!h gesture. &hese are the re%eren!e
traje!tories under#ying the hand gestures re!ognition. Aa!h
traje!tory in the training set is represented as %o##ows:
{ } [ ]
m 2 2
3 3 3 4 R ,..., , ,
3 1

,3/
where:

5
4 is the gesture represented by the traje!tory+
{ }
m
3 3 3 ,..., ,
3 1
are the traje!tory points+
&he di!tionary !ontains the %o##owing gestures:
-&(R& N&AR(5&0N * initiates the intera!tion
with the "irtua# en"ironment so that the human
operator !an start hand#ing the 42 obje!ts+
-&0G N&AR(5&0N J stops the intera!tion and
sends a## the !ommands to the industria# robot+
-&(R&/-&0G R0&(&0N J initiates/stops the
rotation o% the "irtua# en"ironment+
-&(R&/-&0G K00< J initiates/stops the =ooming o%
the "irtua# en"ironment+
-A.A5& 01DA5& J se#e!ts an obje!t %rom the "irtua#
en"ironment+
G.(5A 01DA5& J p#a!es the se#e!ted obje!t to the
!urrent !oordinates o% the "irtua# mouse.
&he gesture re!ognition prob#em, whi!h u#timate#y
redu!es to that o% traje!toriesF mat!hing, raises %ew
important issues, as %o##ows:
gestures genera##y di%%er in regard to the starting
position. 9hen ma)ing the same gesture se"era#
times, the starting position wi## be a#ways di%%erent,
whi!h means ea!h time there wi## be a di%%erent o%%set
[ ]
, y -
t t t , ,
to the starting position o% the re%eren!e
traje!tory. &he o%%set wi## propagate through a## the
points in the traje!tory and there%ore wi## a%%e!t the
traje!tories !omparison resu#t. &he higher is the o%%set,
the bigger wi## be the distan!e between two
traje!tories whi!h !orrespond to the same gesture.
Eenera##y, this prob#em !an be so#"ed through a
simp#e 42 trans#ation as %o##ows:
1
1
1
1
]
1

1
1
1
1
]
1

1
1
1
1
]
1

1 1 0 0 0
1 0 0
0 1 0
0 0 1
1
'
'
'
,
y
-
t
t
t
,
y
-
,
0
-
,4/
gestures are per%ormed at di%%erent speeds. &his issue
a%%e!ts on#y those systems in whi!h the ma!hine being
!ontro##ed by gestures needs to operate at di%%erent
speeds in a!!ordan!e with the a!tua# "e#o!ity o% the
hands being tra!)ed.
di%%erent #engths o% traje!tories. &he #ength o% a
traje!tory a!$uired in rea#*time wi## a#ways be
di%%erent %rom that o% the re%eren!e traje!tory. &his
prob#em !an be so#"ed through a 42 s!a#e
trans%ormation as %o##ows:
1
1
1
1
]
1

1
1
1
1
]
1

1
1
1
1
]
1

1 1 0 0 0
0 0 0
0 0 0
0 0 0
1
'
'
'
,
y
-
S
S
S
,
y
-
,
y
-
,;/
where
-
S , y
S
and
,
S are the s!a#e %a!tors %or ea!h
o% the 4 axes.
&he hand gestures re!ognition method that we ha"e
designed and de"e#oped uses the 2&9 ,2ynami! &ime
9arping/ a#gorithm %or traje!tories mat!hing ,ensuring
time and speed in"arian!e/ and HNN ,)*nearest neighbor/
a#gorithm %or !#assi%i!ation. Ei"en two traje!tories, . o%
#ength n and R o% #ength m ,the %irst one a!$uired in rea#
time and the se!ond one, a re%eren!e traje!tory %rom the
gesture di!tionary/, the 2&9 a#gorithm wi## !ompute the
simi#arity between R and ., by %inding an optima# mat!h
between their !orresponding points.
n the %irst step, the a#gorithm !omputes the distan!e
matrix, as %o##ows:
( ) ( ) ( )
( ) ( ) ( )
( ) ( ) ( )
1
1
1
1
]
1

m n n n
m
m
R . % R . % R . %
R . % R . % R . %
R . % R . % R . %
*.6
, ... , ,
... ... ... ...
, ... , ,
, ... , ,
3 1
3 3 3 1 3
1 3 1 1 1
,5/
where
( )
7 i
R . % ,
is the Au!#idean distan!e between the
points
i
. and 7
R
. 1ased on this matrix, in the next step,
the 2&9 a#gorithm wi## %ind the best path ,with the
minimum !ost/ whi!h starts at [ ] 1 , 1 * and ends at
[ ] m n * , . &he a#gorithm imposes the %o##owing
!onstraint: the path, whi!h is a se$uen!e o% mat!hing
points, needs to be monotoni!a##y and !ontinuous. &hus, at
ea!h step the new *.6 matrix, wi## be !omputed as
%o##ows:
[ ] [ ]
[ ]
[ ]
[ ]

,
_

1 , 1
1 ,
, 1
, ,
min
7 i
7 i
7 i
7 i 7 i
*.6
*.6
*.6
*.6 *.6
,@/
9hen the a#gorithm %inishes, the minimum !ost wi##
be %ind in [ ] m n * , . &he traje!toriesF mat!hing is
i##ustrated in 8ig. @.
$ig. .. Axamp#es o% 2&9 mat!hing
(s we a#ready mentioned, %or traje!toryFs
!#assi%i!ation ,gesture se#e!tion/ we ha"e used the )*NN
a#gorithm whi!h runs against the training set des!ribed in
,3/. 8irst, the 2&9 a#gorithm determines the best mat!h
between . and ea!h
i
R traje!tory be#onging to the
training set. &hen, the )*NN a#gorithm se#e!ts the %irst 5
,usua##y, 5L10/ traje!tories ha"ing the shortest distan!e to
.. 8ina##y, the a#gorithm !hooses that gesture whi!h is
most %re$uent among the %irst 5 traje!tories.
T#e industrial robot
&he assemb#y !e## !onsists o% a -5(R( robot ,-ony
-RM*@11/ with ; degree o% %reedom, and a set o%
transportation and %ixturing de"i!es ,8ig. B/. (s %or the
!ontro# is !on!erned, the !e## is e$uipped with two
!ontro##ers, one represented by the robotFs !ontro##er
,-RM*5@1/ and the other one by a G.5 ,0mron 5300HM/
whi!h runs spe!i%i! app#i!ations %or the !ontro# o%
transportation and %ixturing de"i!es. 1oth !ontro##ers ha"e
R-*343 !ommuni!ation ports a##owing them point*to*point
!ommuni!ation with a de"i!e whi!h is externa# to the !e##,
by using a user*de%ined proto!o#.
$ig. /. -ony -RM*@11 industria# robot
&he main system ,!omprising o% a 42 "irtua#
en"ironment and a hand gestures re!ognition modu#e/
!ommuni!ates with the !e##Fs !ontro##ers "ia a hardware
independent inter%a!e running on an embedded de"i!e o%
A#sist Netmaster type. &he inter%a!e has the ro#e to abstra!t
the !ommuni!ation with !ontro##ers, pro"iding the main
system with a set o% high #e"e# !ommands. &hese
!ommands are stru!tured as M<. e#ements, in whi!h the
e#ementFs tag represents the !ommand type and its
attributes !ontain the asso!iated data. 8or examp#e, the
!ommand %or mo"ing a !uboid %rom a !ertain #o!ation to
another one may ha"e the %o##owing %orm: 8Move
Source9:1; *est9:2; <=.
&he embedded de"i!e on whi!h the hardware
independent inter%a!e runs !an be seen as an additiona#
!ontro##er atta!hed to the !e## ensuring both physi!a#
!ommuni!ation inter%a!ing as we## as the #ogi!a# one. &he
hardware independent inter%a!e ,H/ was de"e#oped using
an A5 @1;>> mode#ing approa!h. A5 @1;>> is a new#y
adopted standard %or the use o% %un!tion b#o!)s ,81/ in
distributed systems o% embedded de"i!es %or industria#*
pro!ess measurement, !ontro# and automation. &he H
was de"e#oped as a -er"i!e nter%a!e 8un!tion 1#o!) ,o%
HN-onyN5e## type/, en!apsu#ated a#ong with se"era#
!ommuni!ation %un!tion b#o!)s into a !omposite 81. 8ig.
> presents the !ontent o% this 81.
$ig. 0. 5ontent o% the !omposite 81 en!apsu#ating the hardware
independent inter%a!e
&he -er"er 81 a##ows re!ei"ing and sending o%
messages %rom/to the main system, whi#e the 81s pairs
O.o3ort>, ?rom3ort>P and O.o3ort@, ?rom3ort@P
a##ows the H to !ommuni!ate with the !e##Fs !ontro##ers
"ia the seria# ports ,> and @/ o% Netmaster.
Conclusions
&he HR based system that we ha"e designed and
de"e#oped promotes the use o% hand gestures %or a more
natura# intera!tion between humans and industria# robots.
&he intera!tionFs goa# is not to mo"e the robot or its
mobi#e arm %rom one p#a!e to another, but to !ontro# the
robot in su!h a manner so that it wi## be ab#e to a!!omp#ish
!ertain tas)s. &he intera!tion is done "ia a dynami!a##y
!reated 42 "irtua# en"ironment. &he ; major ad"antages o%
our approa!h are:
it e#iminates the dead*times whi!h are spe!i%i! to those
systems whi!h promotes a dire!t intera!tion+
the human operator does not need to be in the
proximity o% the industria# robot+
a se$uen!e o% operations whi!h simu#ates an industria#
pro!ess %#ow needs to be done on#y on!e. (%ter that,
the pro!ess !an ta)e p#a!e numerous times, within the
robotFs wor)ing en"ironment+
it !an be extended, so as to a!!omp#ish more !omp#ex
tas)s ,e.g.: mo"ing and p#a!ing !ontainers in a rea#
%ood warehouse/.
8uture wor) wi## be %o!used on the design and
de"e#opment o% a more !omp#ex system whi!h wi## a##ow
users to manipu#ate di%%erent me!hani!a# parts and to bui#d
rea# produ!ts, on#y through hand gestures.
1c*no2ledgements
&his paper was supported by the proje!t QGrogress
and de"e#opment through post*do!tora# resear!h and
inno"ation in engineering and app#ied s!ien!es* GRi2A *
5ontra!t no. G0-2RI/B>/1.5/-/570B4Q, proje!t !o*%unded
%rom Auropean -o!ia# 8und through -e!toria# 0perationa#
Grogram Human Resour!es 3007*3014.
References
1. 1n*it C., Ra#e(a 3., 4aren D., !onia R. nte##igent
(pproa!hes to intera!t with <a!hines using Hand Eesture
Re!ognition in Natura# way: ( -ur"ey // nternationa#
Dourna# o% 5omputer -!ien!e R Angineering -ur"ey
,D5-A-/, 3011. JNo. 3,1/.
3. Pragati G., 5a6een 1., !an(ee6 !. Cision 1ased Hand
Eesture Re!ognition // 9or#d (!ademy o% -!ien!e,
Angineering and &e!hno#ogy, 300>. JNo. ;>.
4. Indra !., Gama 4. Eesture Re!ognition (p#i!ation based
on 2ynami! &ime 9arping ,2&9/ 80R 0mni*9hee#
<obi#e Robot // AAG- 8ina# Groje!t, 3011.
;. Ha#n M., 4ruger 7., 8o#ler C., 4ummert $. 42 (!tion
Re!ognition in an ndustria# An"ironment // Gro!eedings o%
4rd nternationa# 9or)shop on Human*5entered Roboti!
-ystems ,H5R-F0>/, 1ie#e%e#d, Eermany, 300>.
5. 9ang 9., 7i 9. ( New 2es!riptor %or 42 &raje!tory
Re!ognition // &he Ninth nternationa# -ymposium on
0perations Resear!h and ts (pp#i!ations ,-0R(F10/,
5hengdu*Diu=haigou, 5hina, (ugust 1>J34, 3010.
@. 1gnes 3., !ebastien M., 'li6ier :., 3ean-;mmanuel .
H<< and 0H<< %or the Re!ognition o% <ono* and 1i*
<anua# 42 Hand Eestures // 8E Net 9or)shop on Cisua#
0bser"ation o% 2ei!ti! Eestures, 300;.
7. an den :erg# M., Carton D., De 5i(s R., Mitsou 5.,
7andsiedel C., 4ue#nlen< 4., 8oll#err D., an Gool 7.,
:uss M. Rea#*time 42 hand gesture intera!tion with a robot
%or understanding dire!tions %rom humans // AAA
nternationa# symposium on robot and human intera!ti"e
!ommuni!ation, 3011. J G. 457J4@3.
B. C#ang-:eom Par*., M"ung-C#eol Ro#., !eong-8#an
7ee. Rea#*time 42 pointing gesture re!ognition in mobi#e
spa!e // Bth AAA nternationa# 5on%eren!e on (utomati!
8a!e R Eesture Re!ognition, 300B. J G. 1*@.
>. an den :erg# M., an Gool 7. 5ombining RE1 and &o8
!ameras %or rea#*time 42 hand gesture intera!tion // AAA
9or)shop on (pp#i!ations o% 5omputer Cision ,9(5C/,
3011. J G. @@*73.
10. 3un C., Can =., 8ei :., Dac#eng T. 8eature %usion %or 42
hand gesture re!ognition by #earning a shared hidden
spa!e // Gattern Re!ognition .etters, 3013. JNo. 44,;/. J G.
;7@J;B;.
11. Ruifeng 9., 3un C., Pengc#eng 7., Guang C., Can =., >i
=. Ciew in"ariant hand gesture re!ognition using 42
traje!tory // Bth 9or#d 5ongress on nte##igent 5ontro# and
(utomation ,955(/, 3011. J G. @415*@430.
13. Cerlinca T., Pentiuc G. Robust 42 Hand 2ete!tion %or
Eestures Re!ognition // -pringer, nte##igent 2istributed
5omputing C * -tudies in 5omputationa# nte##igen!e, 3013.
JNo. 4B3,3013/. J G. 35>J3@;.

3013 04 13
T. Cerlinca, G. Pentiuc, . lad. Real-Time 3D Hand Gestures Recognition for Manipulation of Industrial Robots ?? ;lectronics
and ;lectrical ;ngineering. @ 4aunasA Tec#nologi(a, )B%). @ 5o. =C===D. @ P. ==@==.
&his paper presents an inno"ati"e system %or industria# robot manipu#ation through 42 hand gestures. &he intera!tion between the
human operator and the roboti! system is done "ia a dynami!a##y !reated 42 "irtua# en"ironment whi!h is a pre!ise !opy o% the robot's
rea# wor)ing en"ironment. &he "irtua# en"ironment enhan!es the manipu#ation o% di%%erent obje!ts and the se#e!tion o% the desired
a!tion, in a natura# way, on#y though hand gestures. In#i)e most o% the HR based systems, ours is not meant %or mo"ing the robot or its
mobi#e arm %rom one p#a!e to another, but to per%orm a spe!i%i! tas) !omprising o% a #ist o% spe!i%i! operations. &his )ind o% intera!tion
has an important ad"antage: it e#iminates the dead*times whi!h are spe!i%i! to the dire!t*intera!tion based systems. &he gesture
re!ognition a#gorithm pro!esses depth maps grabbed %rom a stereos!opi! !amera system and uses 2ynami! &ime 9rapping to !ompute
the simi#arity between the hand traje!tories a!$uired in rea# time and those %rom the gestures di!tionary.
1utoriusC-iaiD. !traipsnio pa6adinimas ?? ;le*troni*a ir ele*trotec#ni*a. @ 4aunasA Tec#nologi(a, )B==. @ 5r. =C===D. @ P.
==@==.
-antrau)a. #. M, bib#. M, #ent. M ,ang#S )a#ba+ santrau)os ang#S ir #ietu"iS )./.