4th International Conference on Electrical Engineering (ICEE 2015)

IGEE, Boumerdes, December 13th -15th, 2015

3D Interaction techniques using gestures recognition

in virtual environment
A.Messaci1,2
1
2

N.Zenati1

A.Bellarbi1

M.Belhocine1

Centre de Dveloppement des Technologies Avances CDTA Alger, Algerie.

Dpartement des Mathmatiques et de l'Informatique Universit Larbi Ben Mhidi,
Oum-El-Bouaghi. Algerie
1

{amessaci, nzenati, abellarbi,mbelhocine}@cdta.dz

Abstract
In virtual and augmented reality (VR/AR), the interaction is
used to designate a set of rules and techniques that allows the
user to perform interaction tasks in a virtual/augmented
environment. The use of hand gestures is the most natural way to
perform an interaction, for this reason gestures recognition has a
part in the word of interaction. In this context, the aim is to study
the different interaction techniques that are used to interact with
3D objects in the case of virtual/augmented reality and the
different methods for vision based hand gesture recognition. In
this paper we develop a 3D interaction technique
"selection/manipulation" using gestures recognition in virtual
environment, the choice of the interaction technique and the
method of gesture recognition are explained. This implemented
system provides more flexible, natural and intuitive interaction
possibilities, and also offers an economic and practical way of
interaction.
Keywords:
Virtual reality, augmented reality, 3D interaction, gesture
recognition.

I. INTRODUCTION
To exploit virtual and augmented reality, the concept of
interaction was introduced. However, experiments in this
field have shown that the use of 2D interfaces in virtual
environments was not effective at all from the point of view
of the interaction that is what led to the 3D interaction. The
interaction plays a very important role in the effective use of
a computer and its different applications. The interaction can
be defined as a language of communication between man
and machine. This language is the set of actions/reactions
loop and computer through sensor and motor interfaces and
interaction techniques. The fundamental tasks of 3D
interaction are: (i) Navigation, it includes all view point
movements of the users in virtual environments. (ii)
Selection, it is the task of acquisition or designation of a
target. (iii) Manipulation, it represents the process that allows
to change the properties of an object or a set of objects.
Virtual world and application control allow users to run a
command to achieve a specific goal or objective.

These3D interactions techniques are often performed by

tools or devices brought by hand, which need a special
learning because it is not natural, In this paper, we are
interested in the selection and the manipulation of virtual
objects in a virtual environment using a vision based hand
gesture recognition technique. Interacting by hand is the
most natural and intuitive way for a person to communicate
with its environment, it would be interesting to change the
computer systems to take into account this form of
interaction.

II. RELATED WORK

Much research has examined hand-gesture-based
interaction with objects in virtual environment. Mechanical
data gloves such as the CyberGlove were used to obtain
information on hand motions [1, 2]. In this approach, users
had to wear burdensome, uncomfortable data gloves that
limited the natural movements of their hands. To overcome
this drawback, vision-based approaches are driven by gesture
recognition, using a video camera or webcam instead of a
wearable device. One such approach is DesIRe [3], which
enables users to create, move, rotate, scale, and select objects
in VEs by using predefined hand gestures by means of an
infrared LED. Dardas and Alhaj [1] implemented a technique
that uses hand gestures to generate commands for interacting
with objects in games, combining Bag-of-Features and
Support Vector Machine (SVM). Some searchers have used
colored markers placed on hands [4,5], the fudicial markers
have used as an interaction tools [6]. More recently,
recognition-based approaches using Microsoft Kinect [7]
have attracted greater interest than techniques that employ
general 2D cameras (such as webcams, video cameras, or
digital cameras) to obtain a sequence of hand motions
because of Kinects superior real-time image-processing
features, which enable greater accuracy and enhanced
performance [8]. Slambekova [9] developed a more natural
technique for controlling objects in a virtual world, using
Kinect to combine gaze direction with hand gestures. Several
university research groups have used Kinect to acquire

2015 IEEE

information on hand motions. The MXR research lab at the

University of Southern California has uploaded a video that
demonstrates avatar actions (such as shaking hands, bowing,
hugging, and flying) and virtual camera movements in a VE
controlled by real-world hand, arm, and body motions using
Kinect and Open Natural Interaction [10, 11]. In the present
work, we develop a technique of selection/manipulation of
objects in virtual spaces based on Microsoft Kinect using a
hand gestures.
III. SYSTEM DESIGN

interaction technique and sends the results

interface.

to the user

User Interface
3D interaction
technique(selection/manipulation)

A. Hardware system description

Glass : work
Kinect

3D Gestures
Modelisation

3D Gestures
Recognition

(Nimble SDK)

Tracking Gestures
Videoproject

RGB &Depth Images

Kinect Capture
Mirror

Fig .2. Application software

1) 3D
interaction
/manipulation)

Fig .1. System setup

The system is presented as follows (see Figure 1): The

workspace of the table is a semitransparent glass with a
projector under the table, users are sitting / standing behind
the table. At the top, a Kinect [12,13] is fixed, and oversees
the work area and the users hands. Visual feedback of the
gestures reproduction and the result of the control action are
projected directly onto the table and sent to the user. The aim
of this system is to allow to user to select and manipulate
virtual objects contained in virtual scene through the screen ,
and this by hand 3d interaction. In the following we give our
choices about gestures recognition.
B. Software system description
As illustrated in Figure 2, the application is composed of
several module. The Kinect capture module captures and
transmits the depth image to the tracking gestures module,
which is composed of two parts:3D gestures modelisation,
and recognition. This module is the main part of the system,
it allows firstly the3D hand recognition and description in
order to interpret the done gestures; then, it applicate the

technique

(selection

A various 3D interaction techniques are proposed, each

one has its advantages and limitations. Therefore, the choice
of the adequate interaction technique must be established
basing the application field, and taking in account its limits.
Moreover, the limitations are not the only parameter to
consider. In fact, the modality of interaction is also an
important criterion.
We have three classes of interaction:
a) Gestural: this class includes all interaction techniques
by gestures, there are several techniques in this part of
modality: Hand direction, Hand gesture, Virtual Hand,
Homer, RayCasting[14,15], FlashLight, SWG, Go-Go[16],
Voodoo Dolls
b) Visual: these techniques use the point of view of the
user, interaction techniques related to this modality: Gaz
direction[16], Voodoo Dolls, SWG.
c) Hardware: these techniques use a stylus or any other
material, some interactions techniques are corresponding to
this modality: Voodoo Dolls, WIM, Displacement using a
map.
In our approach, we choose the gestural modality, which
offers natural way in interaction by hand gestures. We note
that the user interaction field is limited by the capture field of
Kinect thereby virtual environment is limited , in order to
interact naturally the technique chosen is "virtual hand", used

for selecting and moving objects it reproduces the hand of

user's hand movements linearly.

2) Tracking gesture
a)

3D gesture recognition

There are two gesture recognition approaches: based

vision and based sensors [14, 15]. The choice between these
approaches depends on various parameters. To distinguish
between the two approaches, hardware tool is one of the
most important criteria. In fact , if the material used is known
before the choice will be guided by the material because it is
obvious to opt for sensors based gesture recognition
techniques if we use data gloves or opt for techniques based
vision if the capture tool is a camera[16].We are interested in
techniques of gesture recognition based vision because our
system is based on Kinect for gesture recognition. These
techniques of gesture recognition are divided into two
representative families: recognition of 3D model-based
gestures [17,18] and gesture recognition based appearance.
These approaches consist of several representations[19].In
our approach we need some precision to the gestures
recognition, the Kinect gives us depth information which
leads us to favor gestures based on 3D models recognition
techniques. we choose geometrical 3D model this approach
contain essential information of the skeleton.
b)

B. Results

1) The Virtual environment

Our tests have been realized in a simple virtual
environment containing a cube, sphere and cylinder, using
Unity3D (see Figure 3).

3D Gestures modelisation

Using Nimble SDK system developed by 3Gear provides

3D hand tracking for 2 hands, specifically for desktop use.
The 3Gear system runs as a server application. Developers
wanting to develop their own software based on the
interaction captured by 3Gear should let their software
connect to the locally running 3Gear hand tracking server.
The 3Gear server sends messages to connected client, most
of these messages are regarding the detection of hand poses.
These messages contain the global orientation of each
detected hand, as well as the positions of all the joints per
hand. we consider a 3Dskeleton model for each hand
(left/right).The hands joints are considered as local
landmarks. Using Nimble SDK library we get the 3D
position and orientation of each joints. The wrists joint is
also positioned relatively to the Kinect. The used 3D
skeleton model simplify the 3D hand reconstruction or
reproduction on the worker side, it also gives a robust
gestures tracking and recognition.

Fig .3. Virtual environment

2) Hands gesture
Users hands gesture are reproducing using Nimble SDK
(see Figure 4)

IV. IMPLEMENTATION AND RESULTS

A. Virtual environment
The virtual environment are generated by Unity3D,
Unity3D is a cross-platform game creation system developed
by Unity
Technologies,
including
a game
engine and integrated development environment (IDE). It is
used to develop video games for web sites, desktop
platforms, consoles, and mobile devices. First announced
only for Mac OS, at Apples Worldwide Developers
Conference in 2005, it has since been extended to target
more than fifteen platforms.

Fig .4. Hands gestures

3) Object selection
If the hand approaches the volume surrounding the
object, the object change color; the color blue was associated
for the selection phase (see Figure 5).

Fig .5. selection phases

4) Manipulation

interaction tasks in a virtual or augmented environment. In

this paper, we have listed 3D interaction and gesture
recognition techniques, advantages and drawbacks of each.
Then, we have chosen both adequate 3D interaction and
gesture recognition techniques according to a specific
application in order to develop an interactive virtual reality
application, in which we have used the "virtual hand" as 3D
interaction techniques (selection/manipulation) and the
vision based 3D Geometric Model technique for gesture
recognition. In addition, we have integrated the user view
point using an HMD as a display device, in order to give
more degrees of freedom to the user. As the first achieved
results shown, this developed system allows the user to be
more flexible when he performs a set of tasks. In addition,
the chosen techniques reduces errors due to bad detection
and the brightness variation and offers an economic and
practical way of interaction.

As soon as we manipulate the object the hand change

color (visual indication) in this phases the position of the
object is changed (see Figure 6).

VI. REFERENCES
[1]

Fig .6. Manipulation of object

5) Navigation
We define two gestures (i) moving forward, if an object
is far (see Figure 7.a), and (ii) moving backward(see Figure
7.b)

Fig .7. (a) Move Forward (b)Move Backward

V. CONCLUSION
The concept of interaction is now used to refer to a set of
rules and techniques that allows the user to perform

N. H. Dardas and M. Alhaj, Hand Gesture Interaction with a 3D

Virtual Environment, The Research Bulletin of Jordan ACM, vol. 2,
no. 3, (2011), pp. 86-94.
[2]
S. S. Rautaray and A. Agrawal, Real Time Hand Gesture
Recognition Systems for Dynamic Applications, International
Journal of UbiComp (IJU), vol. 3, no. 1, (2012), pp. 21-31.
[3]
M. A. Kavakli, Designing in virtual reality (DesIRe): a gesturebased interface, Proceedings of the 2nd international conference on
Digital interactive media in entertainment and arts, (2007), pp. 131136.
[4] A. Bellarbi, S., Benbelkacem, N., Zenati-Henda, M.
Belhocine.
Hand Gesture Interaction Using Color-Based Method For Tabletop
Interfaces. IEEE 7th International Symposium on Intelligent Signal
Processing (WISP), 2011.
[5] Wang, R. Y., & Popovi, J. (2009). Real-time hand-tracking with a
color glove.ACM Transactions on Graphics (TOG), 28(3), 63.
[6] A.Bellarbi, H.Belghit, S.Bwenbelkacem, N.Zenati, and M.Belhocine.
"Hand gesture recognition using contour based method for tabletop
surfaces." In Networking, Sensing and Control (ICNSC), 2013 10th
IEEE International Conference on, pp. 832-836, 2013.
[7]
Microsoft Corporation, Kinect for Windows & Kinect for Windows
SDK
Beta
1
Programming
Guide
Version,
http://www.microsoft.com/en-us/kinectforwindows/,
Kinect
for
Windows
SDK,
http://msdn.microsoft.com/enus/library/hh855347.aspx (accessed on March 2013).
[8] J. O. Kim, C. Park, J. S. Jeong, N. Baek and K. H. Yoo, A Gesture
Based Camera Controlling Method in the 3D Virtual Space,
International Journal of Smart Home, vol. 6, no. 4, (2012), pp. 117126.
[9]
T. Phan, Using Kinect and OpenNI to Embody an Avatar in Second
Life,
http://www.youtube.com/watch?v=ehTvtkybubM
(http://projects.ict.usc.edu/mxr/) (accessed on March 2013)
[10]
D. Slambekova, Gaze and Gesture Based Object Interaction in
Virtual World, Master thesis, Rochester Institute of Technology.
Rochester, New York, (2011).
[11]
J.O.Kim, M.Kim, K.H.Yoo"Real-Time Hand Gesture-Based
Interaction with Objects in
3D Virtual Environments",
communication of the International Journal of Multimedia and
Ubiquitous Engineering Vol.8, No.6 (2013), pp.339-348
http://dx.doi.org/10.14257/ijmue.2013.8.6.33.
[12] Melgar, E. R. and C. C. Diez. Arduino and Kinect Projects: Design,
Build, Blow Their Minds, Apress.2012

[13] Martin, J. Reconnaissance de gestes en vision par ordinateur, Institut

National Polytechnique de Grenoble-INPG.2000
[14]
Knecht, Martin, Georg Tanzmeister, Christoph Traxler, and
Michael Wimmer. "Interactive BRDF estimation for mixed-reality
applications." (2012).
[15] Ouramdane, N., Otmane, S., Davesne, F., &Mallem, M. FOLLOWME: a new 3D interaction technique based on virtual guides and
granularity of interaction. In Proceedings of the 2006 ACM
international conference on Virtual reality continuum and its
applications.(2006, June).
[16]
Ouramdane, N., Davesne, F., Otmane, S., &Mallem, M. 3D
interaction technique to enhance telemanipulation tasks using virtual
environment. In IEEE/RSJ International Conference on Intelligent
Robots and Systems, (pp. 5201-5207). 2006.
[17]
Roth, S. D. Ray casting for modeling solids. Computer graphics
and image processing, 18(2), 109-144. 1982.
[18]
Quek, F. K. Toward a vision-based hand gesture interface. Virtual
Reality Software and Technology Conference. 1999
[19]
Rautaray, S. and A. Agrawal. "Vision based hand gesture
recognition for human computer interaction: a survey." Artificial
Intelligence Review: 1-54. 2012