International Journal of Engineering Research & Technology (IJERT)

ISSN: 2278-0181
Vol. 2 Issue 12, December - 2013

Sixth Sense Technology : A Brief Literary Survey

Manab Kumar Saha1, Sirshendu Hore2

1,2
Asst. Prof, Department of CSE
HETC
Hooghly, India

information regarding some object very effectively,

Abstract easily and quickly.

Gestures are used widely in sixth sense technology

This paper is a brief survey in the field of Sixth
because they are the most vital and significant type
Sense Technology. Sixth Sense is an interface of
of human communication. Gestures offer several
wearable gestures which upgrade the physical
advantages over traditional human-computer
world around us with digital information and let us
interaction system as it offers touch less system
use the natural gestures to interact with that
therefore it is particularly helpful for physically
information. The advent of technology over the last
challenged person, elder users as well as in health
decade has established a new dimension in the field
care system while maintaining the sterility.
of Human computer Interaction (HCI). The aim of
this paper is to review various gestures, Gestures can be of different type such as
components used, methodology adopted in sixth
RT
sense technology and its application.  Facial expression
 Hand
 Eye
IJE

1. Introduction  Symbolic Responses

We are sensitive on the five basic senses i.e.  Speech
feeling, seeing, tasting, smelling and hearing.  Sound
Apart from those basic senses there is also one type
of sense known as ‘Sixth Sense’. “Sixth Sense  Gait
Technology”, it is the newest vernacular that has Figure 1 illustrates different types of Gesture: -
demonstrates its presence in the technical field.
This newly introduced technology has developed,
which has its relationship to the influence of these
six senses. Our everyday computing machines will
soon be able to sense the different feelings expand
in the surroundings and it is nothing but a gift of the
“Sixth Sense Technology” Sixth Sense is a
wearable “gesture based” device that broaden the
real world with information which are of digital
type and let’s people use a different types of
gestures to interact with that information and there Figure 1. Different types of gesture
by design a environment sometime known as
virtual environment. Information is generally Gestures in sixth sense can be used to
written on paper or computer screen but sixth sense
technology frees the data from that limitation and 1) Generate an event or command for practical
seamlessly integrates information & reality. With interaction.
sixth sense technology the gap between the digital 2) Control object directly.
world & human world is being bridged. The
human-computer interaction becomes very flexible Virtual environment (VE) gives a three-
due to this technology. This technology helps us to dimensional (3D) environment in which a person is
virtualizes real world things & know much more able to interact directly with virtual objects
naturally and powerfully with the environment.

IJERTV2IS120921 www.ijert.org 2889


Machine learning & Intelligence: Combines a wide
variety of advanced technologies to give machines
the ability to learn, adapt, make decisions, and
display behaviours not explicitly programmed into
their original capabilities. Some machine
intelligence capabilities, such as neural networks,
expert systems, and self-organizing maps, are plug-
in components – they learn and manage processes
at a very high level. Other capabilities, such as
fuzzy logic, Bayes Theorem, and genetic
algorithms are building blocks – they often provide
advanced reasoning and analysis capabilities that
are used by other machine reasoning components.
Machine Intelligence capabilities add powerful
analytical, self-tuning, self-healing, and adaptive
behaviour to client applications. They also
comprise the core technologies for many of
Scianta’s advanced data mining and knowledge
discovery services.
2. Literature Review
A lot of research is being done in the fields of
Human computer Interaction (HCI) and its
RT
application in virtual environment. Researchers
have tried detecting the virtual object to control
system environment using video devices for HCI.
IJE
By using the web cameras as the input device,
various natural gestures can be detected, tracked
and analyzed. To help achieve those gestures we
used various image features and gesture Templates.
Cootes et al [7] used Active Shape Models (ASM)
to track deformable objects. M. Isard et al
introduced random sampling filters [8] to address
the need of represent multiple hypotheses while
tracking. G. Kitagawa [9] applied Condensation
algorithm in factored sampling to solve the
problem of visual tracking in clutter. Hojoon Park
[10] used index finger for cursor movement and
angle between index finger and thumb for clicking
events. Chu-Feng Lien [11] used only the finger-
tips to control the mouse cursor and his clicking
method was based on image density, and required
the user to hold the mouse cursor on the desired
spot for a short period of time. A.Erdem et al [12],
used finger tip tracking to control the motion of the
mouse. A click of the mouse button was
implemented by defining a screen such that a click
occurred when a user’s hand passed over the
region. Robertson et al [13], used another method
to click. They used the motion of the thumb (from a
‘thumbs-up’ position to a fist) to mark a clicking
event thumb. Movement of the hand while making
a special hand sign moved the mouse pointer.
Shahzad Malik [14] developed a real-time system
which will trace the 3D position and 2D orientation
IJERTV2IS120921 www.ijert.org
International Journal of Engineering Research & Technology (IJERT)
ISSN: 2278-0181
Vol. 2 Issue 12, December - 2013
of the thumb and index finger of each hand without
the use of special colour object or gloves. In 3D
gaming Nasser H. Dardas et al. [15] developed a
finger based gesture recognition system to control
3D game.
3. Evolution
The WUW stands for (Wear UR World) a
wearable gesture used in Sixth Sense which is a
user friendly interface that provides links with the
physical world around us with digital information
and it uses hand gestures to interact with them. The
initial approach Radio Frequency Identification
(RFID) which is an electronic tagging technology
that allows for detection and tracing of the tags, and
consequently Sixth Sense focuses on the fixed
objects applying on RFID to an enterprise setting,
such as a university department or corporate office.
In short Sixth Sense represents a form of mobile
computing which is applied to non-computing
entities.
Figure 2. Initial Approaches
The initial approach was with the projector
attached to the helmet but there are various
drawbacks of this process, e.g. the person had to
stay still so that the images could be projected on a
suitable surface. And further it was difficult to
manage the mobile computing device kept in the
user’s pocket. This approach definitely needed few
changes. Thus the approach changed when a
smaller projector has been introduced and created
the pendant prototype to be shabby around the
neck. The prototype Sixth Sense is composed of a
camera, a mirror and pocket projector. The
hardware components are grouped in a pendant-like
mobile wearable device. Both the camera and the
projector are connected to the mobile computing
device in the user’s pocket.
2890

Figure 3. Final approach
The prototype was built from a powered - battery
3M projector and a ordinary web cam, with an
attached mirror — all connected to an internet-
enabled mobile phone. The cost of the setup which
is less than $350, allows the user to project
information from the phone onto any surface —
user hand, the body of another person or even wall.
4. Components
Camera: Camera behaves as a digital eye of
the system. It basically used to captures the scene
RT
that the user is looking at. The stream of video
captured by the camera is passed to mobile
computing device which does the appropriate
IJE
compute vision computation. The major functions
of the camera are:
 Captures user’s gestures and movement
(used in reorganization of user gestures)
 Captures the scene in front and objects the
user is interacting with (used in object
reorganization and tracking)
 When the user performs a ‘framing’
gesture, it took a photo of the scene from
the front.
Projector: The key output device of the Sixth
Sense system is projector. The projector visually
augments physical objects, wall and surfaces the
user is interacting with by projecting graphical user
interfaces and digital information. Projector
provides by the mobile computing device with the
content to be projected. The projector unit used in
prototype runs on a rechargeable battery. The
important functions of the projector have been
listed below:
1. Augments related information from the
internet and the physical objects the user
interacting with by projecting just-in-time.
2. Projects graphical user interface of the
selected application onto walls or surfaces
in front.
IJERTV2IS120921
International Journal of Engineering Research & Technology (IJERT)
ISSN: 2278-0181
Vol. 2 Issue 12, December - 2013
Mirror: Mirror used to reflect the projection
emergence from the projector and thus helps in
projecting onto the desired locations on walls or
surfaces. The user can change the gradient of the
mirror manually to change the location of the
projection. For example in application where the
user wants the projection to go on the surface of the
wall instead of the ground in front, he can change
the tilt of the mirror to change the projection. Thus,
the mirror in the Sixth Sense helps to overcome the
limitation of the limited projection space of the
projector.
Microphone: The microphone is an elective
component of the Sixth Sense. When using a paper
as a computing interface, microphone is required.
When the user wants to use a sheet of paper as an
interactive surface, he or she clips the
microphone to the paper. The microphone attached
this way captures the sound signals of user’s
touching the paper. After that the data is passed
to computing device for processing. Later,
combined with the tracking information about
user’s finger, the system is able to identify precise
touch events on the paper. Here, through the
microphone sound signal has been capture along
with the time information whereas the camera
performs tracking.
Mobile Computing Device: The Sixth
Sense system uses a mobile computing device in
user’s pocket as the processing device. This device
can be a mobile handset or a small device where
connection can be wireless or wired.
5. Applications
The Numerous applications on sixth sense
technology already have taken place. Some of the
applications are given bellow: -
Radio Frequency Identification: The radio
frequency identification is the platform which is
based on the enterprise intelligence which
combines Radio Frequency Identification events
with information from other enterprise systems and
sensors to automatically make inferences about
objects, people, workspaces and their interactions.
This is basically an electronic class technology that
allows tracking of tags & detection & consequently
the objects that they are affixed to. It is used for
tagging & remote detection with the low cost
having widespread adoption of RFID.
Washing Machine: A remarkable front loader
that incorporates the unparalleled sixth Sense
www.ijert.org 2891

technology has been placed in a Whirlpool AWOE
8758 White Washing Machine, gives more
optimization of resources and also increased saving
in terms of energy, water and time.
Virtual Call: You can use the Sixth Sense to
project a keypad onto your hand, and then use that
virtual keypad to make a call.
Figure 4. Virtual keypad
Virtual Map: We can see the map virtually on
a physical surface & can find our destination. One
can use his/her index fingers and thumbs to
navigate the map, for example, to zoom in and out
and do other controls.
RT
IJE
Figure 5. Virtual map
Virtual Watch: With Sixth Sense all user can
draw a circle on their wrist with their index finger
to get a virtual watch that gives us the correct time.
The computer track the red marker piece of tape or
cap, recognizes the gesture, and instructs the
projector to flash the image of a watch onto his/her
wrist.
Figure 6. Virtual wrist watch
Virtual Newspaper in Multimedia Mode:
Using this technology we can see more information
IJERTV2IS120921
International Journal of Engineering Research & Technology (IJERT)
ISSN: 2278-0181
Vol. 2 Issue 12, December - 2013
about the topic by video streaming system directly
on the newspaper. A Newspaper can provide live
video news or dynamic information can be
provided on a regular piece of paper. Thus a piece
of paper converted into a video display.
Figure 7. Newspaper in multimedia mode
Figure 8. Take picture without camera
3D Gaming in virtual environment: Using
different hand gesture as arrow key one can
control popular 3D game as show in figure 9.
Figure 9. Playing “Fly over the city” game
Touch based interface-“Sparsh”: Sparsh is
an interaction method that lets users conceptually
transfer media from one digital device to their
body, and pass it to the other digital device by
simple touch gestures.
Misc App: Other than these applications there
are some applications like drawing a picture using
hand movement on a physical object, checking the
flight details only having the flight information or
the boarding pass, we can zoom a picture, get
production information while buying a product we
can also a get any book information and Amazon
rating of that book, a more powerful application
www.ijert.org 2892

regarding sixth sense is while interacting with
someone we can get his/her personal details.
Figure 10 - 11 shows some Misc application.
Draw a picture Zooming Book detail
Figure 10. Misc application
Flight updates People Info Product details
RT
Figure 11. Misc application
6. Software used & Methodology adapted
IJE
To develop an application based on sixth sense
technology one can use
1) Language: Java, C #, C++, OpenCV, JavaCV.
2) Image processing Software: MatLab,
Mathematica, Cylab
3) Powerful Electronic Gazette: WUW v0.1 beta,
Hand Gloves
Methods:
Initialization: It is basically a static approach.
Different gestures are captured, enhanced, features
are extracted and finally a gesture template or
cluster model is created using different algorithm
of artificial intelligence such as MLP with Feed
Forward Network, Visual Memory, and K-Means
Clustering etc. Figure 12 illustrate typical hand
gesture training process [15] while figure 13
illustrates the use of k-means cluster algorithm.
IJERTV2IS120921
International Journal of Engineering Research & Technology (IJERT)
ISSN: 2278-0181
Vol. 2 Issue 12, December - 2013
Figure 12. Hand gesture training process
Figure 13. K-means cluster with two cluster
Acquisition: It is a real time approach where frames
are captured using webcam or from a video input
file.
1) Segmentation: In this technique each of the
frames is processed separately.
Before analysis: The image is smoothed, skin
pixels are labeled, noise is removed and small gaps
are filled. Image edges are found, and then a region
detection algorithm is used to segment the target
gesture from other background information. Finally
features are extracted using popular methodology
[Skin detections, Sift, SURF, FAST algorithms].
2) Pattern Recognition: Once the user’s gestures
has been segmented and related features are
extracted, it is compared with stored gesture
templates or clustering model using different
matching algorithm such as Hausdorff matching,
Euclidean distance, hidden Markova model, Bag of
Words, Hamming Distance, correlation based
approach etc. Figure 14 illustrate the typical
process of testing used for hand gesture
recognition.
Figure 14. Hand gesture recognition process.
www.ijert.org 2893

Execution: Finally, the system carries out the
corresponding action according to the recognized
gesture.
Figure 15; illustrate the complete methodology
adopted for sixth sense technology.
RT
IJE
Figure 15. Block diagram of methodology adapted
for gesture based sixth sense application.
7. Acknowledgement
We would also like to express our appreciation
and gratitude to HETC college authority for
helping us to develop this paper.
8. Conclusion
Sixth sense technology is an emerging
technology in the field of computer vision and HCI.
As a result, it’s a common fallacy that Sixth Sense
is a completely solved problem in this area. Some
of the directions for the future research work in the
field to users, evaluators and vendors can be
1) To make our environment more robust
especially for low quality and degraded
images which is captured using economical
image acquisition device?
2) How to overcome different lighting condition
(day light and artificial light) which has a great
impact in this area.
3) Improve Machine learning & Intelligence for
gesture recognition through comparative studies.
IJERTV2IS120921
International Journal of Engineering Research & Technology (IJERT)
ISSN: 2278-0181
Vol. 2 Issue 12, December - 2013
9. References
[1] Dong A Bowman et. al, ‘The Virtual Venue: User
Computer Interaction in Information – Rich Virtual
Environments’, 1998, Vol.7, pp. -478 – 493.
[2] Viola and M.Jones, ‘Robust Real – time Object
Detection’. International Journal of Computer
Vision, 2(57) 2004, pp. 137 – 154.
[3] T.Kirishima, K.Sato and K.chihara, ‘ Real – time
gesture recognition by learning selective control of
visual interest point’. Paper presented at the IEEE
Trans. on Pattern Analysis and Machine
Intelligence, Vol. 27, No. 3, March [2005], pp. 351
– 364.
[4] Y.Wu and T.Huang, ‘Hand modelling analysis and
recognition for the vision – based human computer
interaction’. Paper presented at the IEEE Signal
Processing Magazine, Special Issue on Immersive
Interactive Technology, Vol.18, No. 3, 2001,pp.-
51 – 60.
[5] R.Bolt, “Put – that – there”,:‘Voice and gesture at
the graphics interface’. In Proc.SIGGRAPH, 1980.
[6] T.Schlomer, B. Poppinga, N. Henze and S. Boll,
‘Gesture recognition with a Wii controller’, In
Proceedings of the second – international
conference on Tangible and embedded interaction,
2008, pp. 11 – 14. Available from: ACM Portal:
ACM Digital Library.
[7] Cootes, A.Hill, C.J.Taylor and J. Haslam, 1994, ‘
The Use of Active Shape Models For Locating
Structures in Medical Images’. Image and Vision
Computing, Vol.16, No.6, July [1994], pp. 355 –
366.
[8] M.A.Isard and A.Blake, ‘Visual tracking by
stochastic propagation of conditional density’. In
the 4th Proceedings European conference
computer vision, pp.343 – 356, 15 April [1996],
Cambridge, England.
[9] G.Kitawaga, ‘Monte Carlo Filter and Smoother for
the Non – Gaussian Nonlinear State Space
Models’. Journal of Computational and Graphical
Statistics, Vol. 5, No. 1, 1996, pp.1 – 25.
[10] Hojoon Park, ‘ A Method for Controlling Mouse
Movement using a Real – Time Camera’ 2008.
[11] Chu – Feng Lien, ‘ Portal Vision – Based HCI – A
Real Time Hand Mouse System on the Handheld
Devices’.
[12] A. Erdem, E. Yardimci, Y. Atalay, V. Cetin,
‘Computer vision based mouse, A. E. Acoustics,
Speech, and Signal Processing’. Paper presented at
the proceedings, (ICASS). IEEE International
Conference, 2002.
[13] Robertson P., Laddaga R., Van Kleek M., ‘Virtual
mouse vision based interface’, In the Proceedings
of the nineth – international conference on
intelligent user interfaces, pp. 177 – 183. Available
from: ACM Portal: ACM Digital Library. [January
2004]
www.ijert.org 2894
 International Journal of Engineering Research & Technology (IJERT)
ISSN: 2278-0181
Vol. 2 Issue 12, December - 2013

[14] Shahzad Malik, ‘Real-time Hand Tracking and

Finger Tracking for Interaction’,CSC2503F Project
Report. [18 December 2003].
[15] Nasser H. Dardas et al, ‘Hand Gesture Interaction with a
3D Virtual Environment’, The Research Bulletin of
Jorden, ACM , ISSN: 2078 - 7952 , Vol II(III) , Page -
86.
[16] Azriel Rosfeld and Avinash C. kak, Digital Picture
Processing, Academic Press, In. Orlando,FL,USA, 1982.
[17] Olivier Faugeras, ‘Three – Dimensional Computer
Vision, A Geometric viewpoint, MIT press, 1993.

RT
IJE

IJERTV2IS120921 www.ijert.org 2895