Erik DeVito, MBA, CSEP-Acq

INCOSE Chesapeake Chapter

January 2012
What is Gesture Recognition?
Historical Highlights
Technology Overview
Industry Landscape
Spotlight on Kinect
What is Disruptive Technology?
Is Gesture Recognition a Disruptive
Technology?
Implications for Systems Engineering?

Gesture Recognition is a technology that
achieves dynamic human-system
interactions that do not require physical,
touch, or contact based input mechanisms

Types of Gesture Recognition:
Hands
Full Body
Historical Highlights
1919- The Theremin is an electronic
musical instrument that does not require
touch but accepts user gestures to create
music (Billingshurst, 2011).


Image Source: io9.com


1970s-1980s Videoplace used real time
image processing of live video to enable
user interaction with a responsive display
(Billingshurst, 2011).


Image Source: Myron Krueger



Image: Maes et al., 1995


1993- ALIVE system, used computer and
video overlays that enabled user and virtual
agents to interact in same display space
(Maes, Darrell, Blumberg, & Pentland,1995).


Image: Sony Computer Entertainment of America


2003- The EyeToy, A 2D camera peripheral
for the Sony Playstation 2 that translated
body movements into game controls
(Marks, 2010).


Image: roeder-johnson.com

2005- Canesta EP-DP200 DevKit, 1
st
3D
Gesture Recognition imaging device using
CMOS-based single chip used IR to sense
depth and motion of user (Canesta, 2010).

Technology Overview
Typically software based-Not requiring special
cameras or sensors
Requires 3 steps
Segmentation
Separates pixels around the hands or body from background
scene
Feature detection
Statistical- based on center of equivalent rectangle
Contour- based on the outline of users hands or body
Gesture recognition
Directional (Left, Right, Up, Down)
Action Based (Point, Reach, Grab, and Select)
Custom libraries



(Du & Li, 2000)
Uses embedded or separate device based cameras,
projectors, and sensors to sense depth and motion
Microwave
Low-power, 9-15 GHz, approx. 2-3cm wavelength
Gaze, Eye and Facial Gesture tracking
Light Waves
Multi-Camera Triangulation
Time of Flight (CW or Modulated IR)
Structured Light (Grid or Pseudo Random IR)
Laser (Object Tracking and Scanning)
Ultrasonic Waves
Basic low resolution/Single Pass
Uses shifts in Doppler frequency to recognize movement



(Castaneda & Navab, 2011; Kalgaonkar & Raj, 2009)
(Castaneda & Navab, 2011)
Measures depth by recognizing return
differences in infrared (IR) light that has been
reflected off the user
Continuous Wave
Measures phase shift of return wave
Pulse Modulated
Measures absolute pulse return time




Image: Jeff Ludvick

Projects IR light into a known pattern and
internal triangulation enables the device to
recognize users place within scene
Produces a 3D map of the reflected IR returns and
re-calibrates it to projected pattern
Depth and movement are obtained by continuously
remapping the current scene and computing the
shift from the previous scene
Can use a combination of Grid and/or Pseudo
Random Speckles



(Castaneda & Navab, 2011)
This video was shot with night vision and
shows the Microsoft Kinect projecting IR
Structured Light into a room. Shows
combination of Grid and Pseudo Random
Speckles
http://www.youtube.com/watch?v=6CFoOFZ
6ifc





Industry Landscape
Companies Chip Development
Software/Middleware
Development Hardware Device Development
AMD X X
ASUS X
Ceva X
Cognivue Corp X X
Eedoo X
Extreme Reality X
eyeSight X
GestureTek X X
H2i Technologies X X
Intel X X
JDSU X
Mesa X
Microsoft X X
Movea X
Oblong Industries X X
Omek Interactive X
Panasonic X
PMD Technologies X
Pointgrab X
PrimeSense X X X
Qualcomm X
Reactrix
Samsung X X
SideKick
SoftKinetic X X X
Texas Instruments X X

Images: Asus, PMD Tech, H2ii Inc., Lenovo Corp., SoftKinetic, Panasonic Corp


Image: Vemea.com

Spotlight on Kinect
A Structured Light Gesture Recognition device
Uses PrimeSense PS 1080 image processor
connected to an IR light and 2 CMOS sensors to
facilitate object acquisition and recognition
Contains 2 microphones and connects to the Xbox
360 via USB 2.0


Image: Microsoft
(Microsoft Kinect Teardown , 2010)

Image: ifixit.com
Released on Nov 4
th
2010 for $149.99
On the same day, Adafruit Industries announces
$2,000 reward to the first person who can deliver a
fully workable Open Source Driver
Later that day after an article on CNET.com
Microsoft responds with significant disdain and
threatens that theyre going to work closely with
law enforcement and product safety groups to keep
Kinect tamper-resistant
Adafruit raises the prize to $3,000 stating the
companys response as the reason for the increase


(The Open Kinect Project, 2010, Terdiman, 2010)
On Nov 10
th
2010 Adafruit Industries announces a
winner and releases open source drivers for Kinect
On Dec 20
th
2010 Microsoft denies that the Kinect
was hacked but instead asserts that unencrypted
USB was intentional
Jan 2011, Kinect breaks world record for Fastest-
Selling Consumer Electronics Device with 8m units
in 60d (Announced Later)
Hacker/Project sites for the Kinect sprout up and
gain popularity with videos and software
downloads posted in abundance



(Adafruit Industries, 2010; Bishop, 2010; Kinect Confirmed, 2011)
On Feb 21
st
2011 Microsoft announces Kinect for
Windows Software Development Kit will be released
in Spring 2011Former employee and key Kinect
developer confesses (on his blog) that he put up
the $3,000 reward
Kinect for Windows beta SDK released in June 2011
Enabled non-commercial development of applications for
Kinect.
Company creates official project pages, blog, and
Kinect Website to highlight what new and
innovative projects people are creating with the
Kinect.







(Lee, 2011)
Kinect sales for Xbox 360 steamed forward with
robust holiday 2011 salesAt the time some
estimated 11-14m sold
Kinect Accelerator-Startup incubator program
announced-10 companies with innovative Kinect
solutions/products/business plans will be selected
on Jan 25
th
2012 for $20k funding with mentoring
by Microsoft developers and managers
Company announced @ CES 2012 that Kinect is
coming to Windows on February 1st and that sales
to date have reached 18m units







(Kinect Accelerator Program, 2011; Use the Power of Kinect , 2012)
Health Care
Education
Mobile Phones
Automotive
Robotics/Computer Vision
Home Media/Appliance Interaction
Personal Computer Interaction
Interactive Vending, Signage and Displays
Security/Identity Management
Motion Capture for graphics and film effects
Video Games







Image: Jeff Ludvick

This video shows a person using a delta robot
to grasp metals balls using only gestures and
body movements. The robot is actuated by
three servo motors which are controlled by an
AVR microcontroller.
http://www.youtube.com/watch?v=-
Ub8LKPkhos&feature=player_embedded





This video shows a person controlling a
remote controlled quad copter.
http://youtu.be/A52FqfOi0Ek





This video shows the OmniTouch Natural
User Interface created by Chris Harrison @
Carnegie Melon. It uses a modified Microsoft
Kinect and Pico Projector to create a unique
projection based user interface.
http://youtu.be/Pz17lbjOFn8





What is Disruptive Technology?
Have very different package of
attributes mainstream customers
historically valueoften perform far worse
along one or two dimensions important to
those customersmainstream customers
are unwilling to use a disruptive product in
applications they know and
understanddisruptive technologies tend
to be used and valued only in new markets
or new applications; in fact, they generally
make possible the emergence of new
markets

(Bower & Christensen, 1996)
The following were disruptive because they
enabled people to do things that they
couldnt previously do with existing
technologies:
Telephone
Transistor Radio
Photocopier
Personal Computers
Digital Photography
Each had significant trade-offs, as the
disruption occurred they were overcome

(Christensen, Anthony, & Scott, 2003)
Is Gesture Recognition a
Disruptive Technology?
1. Signals of
Change
3. Strategic
Choices
2. Competitive
Battles
(Christensen, Anthony, & Scott, 2003)
(Christensen, Anthony, & Scott, 2003)
Description Disruptive Technology Theory Evaluation
Are there signs that someone is
capitalizing on opportunities for
change?
Serving nonconsumers by
introducing a disruption that
was previously not convient, too
expensive or more
technologically advanced for
them to acquire.
As a new market innovator
Microsoft has introduced a
packaged that is convient,
inexpensive, and does not
require the user to have special
skills to use it.
(Christensen, Anthony, & Scott, 2003)
Description Disruptive Technology Theory Evaluation
What is the likely result of head
to head battles?
The new market entrant has
unique and/or abundant
resources and is applying them
in a way that causes incumbents
to fail.
Microsoft is flush with
resources. One of the top
software firms in the world. This
is no start up. Incumbents are
scrambling for investments and
partnership with other industry
powerhouses.
(Christensen, Anthony, & Scott, 2003)
Description Disruptive Technology Theory Evaluation
Are firms making decisions that
increase or decrease their
ultimate chances for success?
Successful disruptors harness
the forces of disruption by
selecting the right value network
and marketplace to make their
stand.
Microsoft used the Xbox 360,
the most popular video gaming
device on the market as their
launch pad for Gesture
Recognition. Additionally, they
have adopted a previously
successful community
development approach to help
this technology further gain a
foothold. Incumbents have been
forced into "also ran position
within the marketplace.
Is Gesture Recognition a Disruptive
Technology?
Yes!!!
It is highly likely that Gesture Recognition is in fact a
Disruptive Technology.
The Kinect could act as the vehicle for this disruption.
This is inline with both the definition of and theory
behind Disruptive Technology
Microsoft satisfies the 3 Step Process that helps
determine if an industry is ripe for disruption.





(Christensen, Anthony, & Scott, 2003)
Implications for Systems
Engineering?
Exponential growth is deceptive. It starts
out almost imperceptibly and then explodes
with unexpected furyunexpected, that is,
if one does not take care to follow its
trajectory

(Kurzweil, 2006)
Requirements Definition & Analysis
Understand thresholds for Gesture Recognition performance as it relates
to stakeholder requirements or expectations.
Ground Truth vs. MarketingI want to be able to use an Xbox Kinect to!
Determine parameters will be used to define performance?
Architectural Design
Identify architectural implications of adding Gesture Recognition into the
system architecture.
Are Gesture Recognition interface requirements creating an issue for the
systems affordability/ROI?
Does Gesture Recognition play well with othersInteroperability Issues?
Implementation & Integration
Gesture Recognition may evolve quickly so a balance must be maintained
between selecting system element solutions too early or too late in the
process.
Too early and you negate enhanced capabilities and opportunities for cost
savings
Too late and you risk adopting technology that has advanced past your
architectural interfaces. REWORK!
In general, international standards and best
practices do not exist for Gesture Recognition
This poses some challenge for Systems Engineers
Bright spots do exist
OpenNi is an industry-led organization to promote
compatibility and interoperability of Natural Interaction
(NI) devices applications and middleware.
Point Cloud Library (PCL) is a cross platform set of
algorithms for creating data structures that represent a
collection of multi-dimensional points to recreate or
represent 3D objects.

(Introducing OpenNI, 2011)
This video created in 2009 by Microsoft to
showcase their vision for the future of
productivity. Gesture Recognition tech is
shown along with other technologies and
provides compelling view of how things might
evolve in the workplace.
http://youtu.be/t5X2PxtvMsU





This video is from www.theverge.com and the
shows host Josh Topolsky visiting with
Microsoft Advanced Research to get a tour of
their Building 99 and see new innovations
with the Xbox Kinect.
http://youtu.be/4DS_zGm_K84






Questions?
Billingshurst, M. (2011). Gesture based interactions. Retrieved from http://www.billbuxton.com/input14.Gesture.pdf
Bishop, T. (2010, November 20). Microsoft: Kinect wasnt hacked, USB port left open by design. TechFlash.com. Retrieved
from http://www.techflash.com/seattle/2010/11/microsoft-kinect-not-hacked-left.html
Bower, J. L., & Christensen, C. M. (1996). Disruptive technologies: Catching the wave. Harvard Business Review (January
February 1995), pp. 4353. Journal of Product Innovation Management, 13(1), 75-76. Retrieved from
http://www.cc.gatech.edu/~spencer/courses/ethics/misc/bower.pdf
Castaneda, V., & Navab, N. (2011). Time-of-flight and Kinect imaging [PowerPoint slides]. Retrieved from
http://campar.in.tum.de/twiki/pub/Chair/TeachingSs11Kinect/2011-DSensors_LabCourse_Kinect.pdf
Christensen, C.M., Roth, E.A., Anthony, S.D. (2004). Seeing what's next: Using theories of innovation to predict industry
change. Boston, MA: Harvard Business School Press.
Du, W., & Li, H. (2000). Vision based gesture recognition system with single camera. Signal Processing Proceedings, 2000.
WCCC-ICSP. 5th International Conference, August 21-25, 2000. Los Alamitos, CA: IEEE Computer Society. Retrieved from
http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=891796
Kalgaonkar, K., & Raj, B. (2009). One-handed gesture recognition using ultrasonic doppler sonar. Acoustics, speech and
signal processing, 2009. ICASSP. IEEE International Conference, April 19-24, 2009. Los Alamitos, CA: IEEE Computer
Society. Retrieved from http://ieeexplore.ieee.org/xpl/mostRecentIssue.jsp?punumber=4912736
Kinect Accelerator Program Seeking Innovators. (2011, December 28). Microsoft. Retrieved from
http://blogs.msdn.com/b/kinectforwindows/archive/2011/12/28/kinect-accelerator-program-seeking-innovators.aspx
Kinect confirmed as fastest selling consumer electronics device. (2011) Guinness World Records.
http://community.guinnessworldrecords.com/_Kinect-Confirmed-As-Fastest-Selling-Consumer-Electronics-
Device/blog/3376939/7691.html
Kurzweil, R. (2006) The singularity is near: When humans transcend biology. New York, NY: Penguin Group
Lee, J.C. (2011, February 21). Windows drivers for Kinect, finally!. Retrieved from
http://procrastineering.blogspot.com/2011/02/windows-drivers-for-kinect.html






Maes, P., Darrell, T., Blumberg, B., & Pentland, A. (1995). The Alive System: Full-body interaction with autonomous agents.
Computer Animation Conference, Geneva, Switzerland, April, 1995. Los Alamitos, CA: IEEE Computer Society . pp. 11-18.
Retrieved from http://ieeexplore.ieee.org/xpl/mostRecentIssue.jsp?punumber=3144
Marks, R. (2010). EyeToy, innovation and beyond. Sony Computer Entertainment America. Retrieved from
http://blog.us.playstation.com/2010/11/03/eyetoy-innovation-and-beyond/comment-page-2/#comment-478157
Microsoft Kinect Teardown . (2010). ifixit.com. Retrieved from http://www.ifixit.com/Teardown/Microsoft-Kinect-
Teardown/4066/1
Starting February 1, 2012: Use the Power of Kinect for Windows to Change the World. (2012, January 09). Microsoft.
Retrieved from http://blogs.msdn.com/b/kinectforwindows/archive/2012/01/09/kinect-for-windows-commercial-
program-announced.aspx
The Open Kinect Project- THE OK PRIZE- get $3,000 bounty for Kinect for Xbox 360 open source drivers. (2010, November
04). Adafruit Industries. Retrieved from http://www.adafruit.com/blog/2010/11/04/the-open-kinect-project-the-ok-
prize-get-1000-bounty-for-kinect-for-xbox-360-open-source-drivers/
Terdiman, D. (2010, November 04) Bounty offered for open-source Kinect driver. CNET.com. Retrieved from
http://news.cnet.com/8301-13772_3-20021836-52.html
Terdiman, D. (2011, February 21) Kinect developer claims credit for hack bounty idea. CNET.com. Retrieved from
http://news.cnet.com/8301-13772_3-20034579-52.html
Van Camp, J. (2010, October 19). Microsoft reveals Kinect launch games. Digital Trends.com. Retrieved from
http://www.digitaltrends.com/gaming/microsoft-reveals-kinect-launch-games/
WE HAVE A WINNER Open Kinect driver(s) released Winner will use $3k for more hacking PLUS an additional $2k goes
to the EFF. (2010, November 10). Adafruit Industries (Blog). Retrieved from
http://www.adafruit.com/blog/2010/11/04/the-open-kinect-project-the-ok-prize-get-1000-bounty-for-kinect-for-
xbox-360-open-source-drivers/