NUI - GSoC 09

Multitouch improved MouseDriver

April - 2009

Natural User Interface Group

Summer of Code Application

Tiago Serra

GSoC 09

serratiago@gmail.com

http://vimeo.com/hci

GSoC 09 - Multitouch improved MouseDriver

Abstract
A multi operating system MT mouse emulator that can simulate multiple cursors and give visual cues on the mouse events red.

About me
Name: Tiago Jorge de Carvalho Limpo Serra Email:serratiago@gmail.com Location/Timezone:Coimbra, Portugal - GMT 00:00 Website:http://vimeo.com/serratiago Age:27 Education/Qualications: Universidade de Coimbra - Departamento de Engenharia Informtica - Computer Enginnering Major. Master's student Academic and Industry Background - HCI work: Universidade de Coimbra - Departamento de Engenharia Informtica - Game development research lab Instituto Pedro Nunes, Coimbra - HCI research lab NUI member: <serratiago> Since August 2008.

GSoC 09

serratiago@gmail.com

http://vimeo.com/hci

GSoC 09 - Multitouch improved MouseDriver

Features
Manipulate existing non multitouch software through a MT-display Ability to handle more than one cursor by fast switching through points of contact (blobs) Drag&Drop/Selection, Left Click, Right click, Double-click and scroll-wheel gestures wrapper for mouse events Fire mouse events when specic gestures are made Visual interface overlay for mouse events feedback

Detailed description
Multitouch displays are a very interesting technology, yet there's few applications that truly can use its potencial. The MouseDriver comes as the best way to accommodate current desktop environments and applications in this human computer interface. This project proposal aims to accomplish this by using various techniques that emulate mouse cursor events in multiple points of the screen for environments where only one cursor is available. By fast switching the mouse cursor through all available surface points of contact (blobs) we can successfully fake mouse events as though they happen simultaneously. For instance, if I want to drag and drop two items simultaneously I can do it with a nger in each one and the underlying invisible mouse cursor would jump at a high rate trough those two points on the screen giving the impression we are really manipulating both simultaneously. There is another possibility here that is to map higher level events like the zoom "pinch" and others to mouse events like the mouse wheel up/down. Once we accomplish this we can use the MouseDriver in a multi-user surface to interact with previously single-user and single-cursor only applications. Since we have to make the mouse cursor invisible and there is really no available cursor but a touchable surface, we also need to create some visual feedback that informs the user of what events were red and where. For instance, a double-tap would re a double-click mouse event on the particular screen position and show two circles as visual feedback there. A single tap would re a mouse-click event and show a one circle visual feedback. An interesting research possibility would be mapping multitouch interactions into composed mouse events. For instance, a selection gesture using four ngers could be recognized by the driver and re the appropriate composite single point events. All MouseDriver work will be based upon the currentMartin Kaltenbrunner Java MouseDriver implementation.

GSoC 09

serratiago@gmail.com

http://vimeo.com/hci

GSoC 09 - Multitouch improved MouseDriver

Justication and why me:

Although current MouseDriver implementation are sufcient for handling basic desktop mouse functionality, much has to be done to truly exploit multitouch capabilities in not so multitouch aware applications.

The current MouseDriver suggest a one cursor interaction limitation but I feel that this doesn't need to be the case.

One good application where we can use this MouseDriver are RTS games. In these games we can usually manipulate everything using the mouse cursor. Here we could make troop selections and command soldiers, for instance, with simple and powerful gestures and interactions.

This proposal comes as a personal drive to use current multitouch setups in actual desktop application and games without having to modify the underlying operating system and/or application.

GSoC 09

serratiago@gmail.com

http://vimeo.com/hci

GSoC 09 - Multitouch improved MouseDriver

Development Methodologies:
The methodology that I propose for this project is based on 7 stages: Technological basis research - Research of the technological fundaments for the mouse driver which includes: OSs mouse event system Developing preliminary driver which includes the following functionalities mouse move mouse left button click Proof of concept demonstration demonstration of the preliminary driver capabilities writing of a short report of tests and observations Draft technical specication at modeling level development of technical specication diagrams Implementation nal development of the mouse driver application with complete set of features which includes visual event feedback system inclusion of src les headers and function headers in the form of comments for extraction for Detailed Design Testing and bug xing demonstration and validation of the complete driver capabilities through RTS gaming and driver demoing Final documentation writing of a short Report of Tests and Observations for all drivers functionalities and capabilities writing of a Architectural Design Document based on technical specication diagrams writing of a Detailed Design Document based on the extraction of le and function headers from the source code writing of a User Manual for the mouse driver application

Brief list of deliverables:

1st Month Deliverable: Preliminary mouse driver 1st to 3rd Week: Implementation of preliminary mouse implementation 4th Week: Report and Observations. Simple interaction demo. 2nd Month Deliverable: High-Level multitouch interactions to mouse events 1st to 3rd Week: Implementation of high level interactions wrapper. 4th Week: Detailed Design Documentation extracted. Simple interaction demos. 3rd Month Deliverable: Visual feedbacks. Documentation. 1st and 2nd Week: Implementation of visual feedbacks for basic mouse events. 3rd Week: Architectural Design Documentation and User Manual 4th Week: Final test with an RTS game.

GSoC 09

serratiago@gmail.com

http://vimeo.com/hci

GSoC 09 - Multitouch improved MouseDriver

References
- Cross platform TUIO mouse [ http://nuigroup.com/forums/viewthread/4402/ ]

GSoC 09

serratiago@gmail.com

http://vimeo.com/hci