c






We've evolved over millions of years to sense the world around us. When we encounter
something, someone or some place, we use our five natural senses to perceive information about
it; that information helps us make decisions and chose the right actions to take. But arguably the
most useful information that can help us make the right decision is not naturally perceivable with
our five senses, namely the data, information and knowledge that mankind has accumulated
about everything and which is increasingly all available online. Although the miniaturization of
computing devices allows us to carry computers in our pockets, keeping us continually
connected to the digital world, there is no link between our digital devices and our interactions
with the physical world. Information is confined traditionally on paper or digitally on a screen.
Most of the people have some portable technology like smart phones, laptops, e-book and
iPods. But you need to carry all the stuff with you. We grew up interacting with the physical
objects around us. And there are enormous numbers of them that we use every day. Unlike our
most computing devices these objects are much more fun to use. When you talk about objects
one another thing that automatically comes to that thing is gestures how we manipulate the
objects. We use gestures not only to interact with each other but also to interact with the physical
world around us.

è
 èè

 

Fig 1: Five senses and the sixth sense

c c‘u



Yhe exploration of this sixth sense technology started with a question why can¶t we use a
computer to interact with physical world.Yhis literally started with amouse that comes with a ball
in it and YWO ROLLERS that actually guides the movement.So taking out rollers from another

c
mouse totally we have four rollers.Using these a system interface device was formed.With this
whatever you do in physical world will be reflected in the digital world.

Fig 2: Virtual Demonstration of Sixth Sense

With this initially a pen to draw 3-dimensions in digital world was made that can help designers
and architects not only think but they can actually draw so that it is more intuitive to use that
way. Yhe goal is to bring part of physical world to digital world to make computing devices more
intuitive. Not only this, the reverse process is made possible.

Initially the device started with a helmet consisting of the camera, projector and color
markers. Later it was converted into a form so that it is easy to use it by wearing around the neck.

Sixth Sense bridges this gap, bringing intangible, digital information out into the tangible world,
and allowing us to interact with this information via natural hand gestures. µSixth Sense¶ frees
information from its confines by seamlessly integrating it with reality, and thus making the entire
world your computer.

 
c




u


From thousands of years mankind used the five senses to gain information. But now as this
device is completely portable and it can provide information about anything, it is named as
µSIXY SENSE DEVICE¶.


 



Yhe Sixth Sense prototype is comprised of a pocket projector, a mirror and a camera. Yhe
hardware components are coupled in a pendant like mobile wearable device. Both the projector
and the camera are connected to the mobile computing device in the user¶s pocket.

Fig 3: Working Principle of Sixth Sense

Yhe projector projects visual information enabling surfaces, walls and physical objects
around us to be used as interfaces; while the camera recognizes and tracks user's hand gestures
and physical objects using computer-vision based techniques. Yhe software program processes
the video stream data captured by the camera and tracks the locations of the colored markers
(visual tracking fiducially) at the tip of the user¶s fingers using simple computer-vision
techniques. Yhe movements and arrangements of these fiducially are interpreted into gestures
that act as interaction instructions for the projected application interfaces. Yhe maximum number
of tracked fingers is only constrained by the number of unique fiducially, thus Sixth Sense also
supports multi-touch and multi-user interaction.

·
 ‘‘‘ ‘‘ ‘

‘‘CPU & MEMORY WIY 3G‘PROJECYOR‘CAMERA

·
 u


· c


   

Gesture recognition is a topic in computer science and language technology with the goal of
interpreting human gestures via mathematical algorithms. Gestures can originate from any bodily
motion or state but commonly originate from the face or hand. Current focuses in the field
include emotion recognition from the face and hand gesture recognition. Many approaches have
been made using cameras and computer vision algorithms to interpret sign language. Gestures
can exist in isolation or involve external objects. Free of any object, we wave, beckon, fend off,
and to a greater or lesser degree (depending on training) make use of more formal sign
languages. With respect to objects, we have a broad range of gestures that are almost universal,
including pointing at objects, touching or moving objects, changing object shape, activating
objects such as controls, or handing objects to others.

Gesture recognition can be seen as a way for computers to begin to understand human body
language, thus building a richer bridge between machines and humans than primitive text user
interfaces or even GUIs (graphical user interfaces), which still limit the majority of input to
keyboard and mouse. Gesture recognition enables humans to interface with the machine ( MI)
and interact naturally without any mechanical devices. Gestures can be used to communicate
with a computer so we will be mostly concerned with empty handed semiotic gestures. Yhese can
further be categorized according to their functionality.

·

 



Symbolic Gestures are gestures that, within each culture, have come to a single meaning. An
Emblem such as the ³OK´ gesture is one such example, however American Sign Language
gestures also fall into this category.

M
· ·
  



Deictic Gestures are the types of gestures most generally seen in CI and are the gestures of
pointing, or otherwise directing the listeners attention to specific event or objects in the
environment.

· 
 




Ionic Gestures, as the name suggests, these gestures are used to convey information about the
size, shape or orientation of the object of discourse. Yhey are the gestures made when someone
says ³Yhe plane flew like this´, while moving their hand through the air like the flight path of the
aircraft.

· 
  




Yhese are the gestures typically used in showing the use of movement of some invisible tool or
object in the speaker¶s hand. When a speaker says ³I turned the steering wheel hard to the left´,
while mimicking the action of turning a wheel with both hands, they are making a pantomimic
gesture. Using the concept of gesture recognition, it is possible to point a finger at the computer
screen so that the cursor will move accordingly. Yhis could potentially make conventional input
devices such as mouse, keyboards and even touch-screens redundant. Gesture recognition can be
conducted with techniques from computer vision and image processing. Yhe literature includes
ongoing work in the computer vision field on capturing gestures or more general human pose
and movements by cameras connected to a computer.

· 
 



Computer vision is the science and technology of machines that see. As a scientific discipline,
computer vision is concerned with the theory behind artificial systems that extract information
from images. Yhe image data can take many forms, such as video sequences, views from
multiple cameras, or multi-dimensional data from a medical scanner.
Computer vision, on the other hand, studies and describes the processes implemented in software
and hardware behind artificial vision systems. Yhe software tracks the user¶s gestures using
computer-vision based algorithms. Computer vision is, in some ways, the inverse of computer
graphics. While computer graphics produces image data from 3D models, computer vision often
produces 3D models from image data. Yhere is also a trend towards a combination of the two
disciplines, e.g., as explored in augmented reality. Yhe fields most closely related to computer

›
vision are image processing, image analysis and machine vision. Image processing and image
analysis tend to focus on 2D images, how to transform one image to another. his characterization
implies that image processing/analysis neither require assumptions nor produce interpretations
about the image content. Computer vision tends to focus on the 3D scene projected onto one or
several images, e.g., how to reconstruct structure or other information about the 3D scene from
one or several images. Machine vision tends to focus on applications, mainly in manufacturing,
e.g., vision based autonomous robots and systems for vision based inspection or measurement.

·
   
!  

Yhe computer vision system for tracking and recognizing the hand postures that control the
menus is based on a combination of multi-scale color feature detection, view based hierarchical
hand models and particle filtering. Yhe hand postures or states are represented in terms of
hierarchies of multi-scale color image features at different scales, with qualitative inter-relations
in terms of scale, position and orientation. In each image, detection of multiscale color features is
performed. Yhe hand postures are then simultaneously detected and tracked using particle
filtering, with an extension of layered sampling referred to as hierarchical layered sampling. Yo
improve the performance of the system, a prior on skin color is included in the particle filtering.



"
# $

#
%
 & 

M‘ m
m  
 
 

M‘  

M‘ 

M‘  

M‘ 

M‘  

^

! !

Yhe sixth sense prototype implements several applications that demonstrate the usefulness,
viability and flexibility of the system .sixth sense technology has got a wide range of applications
in different fields. Yhe following can explain the areas in which this amazing technology can be
applied.

 c


'

You hold up your left hand, fingers pointing to the right. Yhe system recognizes that you want to
make a call, and projects a dialing pad onto your fingers. You tap the virtual keypad with your
right hand to dial the call.

Fig 5: Check time

ü


'(
!
 

If you want to make a call to anyone and you find yourself with no such calling device, it is
possible to make a call with sixth sense technology. It¶s enough to stretch your palm and then a
virtual dialing pad will be projected. In this way you can call with no external device.

Fig 6: Make a call

 ·
(




If you find beautiful scenery and want to capture it, you hold your fingers out at arm¶s length
forming square. Yhe system snaps a photo of what¶s enclosed by your fingers. Later, you can
sort, resize and fiddle with these photos by projecting them onto any wall and dragging their
images with your fingertips, à la Microsoft Surface.

Fig 7: Yake Snaps

Yhat¶s it you captured it and you can project it on any surface say a wall, and you can even edit
it and mail to a friend.

o
 
 
 

 


 o  
 
 
 

When you go to a book store to buy a new book, as soon as u hold a book in your hand
the sixth sense device immediately projects a rating of the book enabling you to choose the best
one and moreover it provides you with other additional information. Not only a book any
purchase you make this device can help you. You hold up your boarding pass; the words
DELAYED 20 MINUYES or ON YIME are beamed onto it.

Fig 9: Flight Updates

 

 % 



 

 


While you¶re reading a newspaper, depending on the news and related photos you can
immediately watch the related videos on the news papers itself and make it useful and interesting
rather than going for other sources.

 c 
    


 

 

Your newspaper will show you the live of weather information rather than starting your
computer to know this.

Fig 11: Live Information


 

  

 $



Using the global positioning system, you can project maps anywhere which provide information
on your position depending on your moves and find a way by using zoom out and zoom in
options.

 c    



When this topic is explained to our former president DR.A.P.J .ABDUL KALAM ,he suggested
a new use saying that a person who is dumb can use speaker system to express his thoughts using
this technology.many other processes are going on to make this technology reach masses.

 )
  



Attach the microphone available with the webcam to a paper. Now the sound of touch is exactly
getting a where a person is touching the paper.

c
 Fig 13: Video Playing

 *
 
!
  

You run into a guy at a party. Yhe system projects his name and Face book keywords onto his Y-
shirt, so you can remember who he was and start a conversation.

Fig 14: People Information

 *
 
(

And why only computers, we can play with papers. We can take a part of a document putting
over another paper and you can even modify the paper and print it. So the work flow is more
intuitive. In a more dynamic way you can pinch the information on the paper to your system.

cc
 Fig 15: Paper Work


!!!



SixthSense is an user friendly interface which integrates digital information into the physical
world and its objects, making the entire world your computer.

SixthSense does not change human habits but causes computer and other machines to adapt to
human needs. It uses hand gestures to interact with digital information.
1. Supports multi-touch and multi-user interaction.

2. Data access directly from machine in real time.

3. It is an open source and cost effective and we can mind map the idea anywhere.
4. It is gesture-controlled wearable computing device that feeds our relevant information and
turns any surface into an interactive display.

5. It is portable and easy to carry as we can wear it in our neck.

6. Yhe device could be used by anyone without even a basic knowledge of a keyboard or mouse.

7. Yhere is no need to carry a camera anymore. If we are going for a holiday, then from now on
wards it will be easy to capture photos by using mere fingers

c 
 c


Sixth Sense Yechnology May Change ow We Look at the World Forever. Yhe true power of
Sixth Sense lies on its potential to connect the real world with the Internet, and overlaying the
information on the world itself. Clearly, this has the potential of becoming the ultimate
³transparent´ user interface for accessing information about everything around us.

Yeachers may find that Sixth Sense, because of its unique ability to work with many of
Gardner¶s multiple intelligences, will be a catalyst for students who might otherwise remain
unengaged in learning. Sixth Sense uses the visual, kinesthetic, intrapersonal and interpersonal
among others.

If this technology does nothing else, the idea that students can simply use their hands to draw
images on walls and take pictures with their hands will be (at least initially) quite motivating

Sixth Sense will give new meaning to ³looking something up.´ Since we are essentially talking
about the internet, students will be given the opportunity to portably research the world. Yhey
can all go on a virtual field trip and project the images on their own desks, for example. Students
will be given an opportunity through this technology to feel more connected to others.

Although there does seem to be the sense that individuals could walk through life wired and
lonely, I like to think the classroom would serve as the environment in which students and
teachers using Sixth Sense would want to share.

Not only could the sharing take place in the classroom, I suppose since the users are connected
online, sharing could take place around the world.

"
u!'

Yo get rid of color markers. Yo incorporate camera and projector inside mobile computing
device. Whenever we place pendant- style wearable device on table, it should allow us to use the
table as multi touch user interface. Applying this technology in various interest like gaming,

c·
education systems etc. Yo have 3D gesture tracking. Yo make sixth sense work as fifth sense for
disabled person.

)
 

Imagination is the only limit for technologies like this one. It augments the physical world
around us with digital information and lets us use natural hand gestures to interact with that
information. It is a sense that would give seamless and easy access to Meta information or
information. So, with this technology we can have our digital world with us wherever we go.
And we can start using any surface around us as an interface. It will change the way we interact
with people not only with physical world. Actually the hardware is not that hard to manufacture.
Yhe open source software is going to be made available .It costs around $350.So the technology
can reach masses than staying to laboratories or environments.

cM

" 

M‘ P. Maes, P. Mistry. Unveiling the "Sixth Sense," game-changing wearable tech. YED
2009. Long Beach, CA, USA 2009

M‘ GIZMODO - Sixth Sense Yechnology May Change ow We Look at the World Forever

M‘ SOFYPEDIA - Next-Gen 'Sixth Sense' Device Created at MIY

M‘ PCWORLD - MIY Wearable Gadget Gives You Sixth Sense

M‘ http://www.pranavmistry.com/projects/sixthsense/

M‘ BBC - Sixth Sense blurs digital and the real.

c›