Gesture-Based Virtual Music Conducting using Microsofts Kinect

Sachi A. Williamson Bachelor of Science, Computer Science Faculty Mentor: Laurie Murphy CSCE 499 - Fall 2012
Abstract The open-sourced availability of Microsoft Kinect development tools and the broad accessibility of the hardware enable the suitable development of gesture-based software designed to teach basic musical conducting skills to users of any age and background. This project will interactively guide players through the basic patterns and essential elements of conducting using embedded videos with music professors and other teaching methods using the Kinects gesture-based functionality. In addition, the culmination of the lessons will be the opportunity to conduct a diverse repertoire of works while following along with a simplied score. The Kinect library and the project additions will provide possibilities for substantial personal learning experience and growth. Finally, the project will also serve as a model for an interdisciplinary capstone project combining computer science and the arts and may also contribute to the teaching methods of instrumental and choral conducting.

Introduction

There has always existed a subtle dichotomy between technology (specically computer science) and the arts due partially to tensions of the importance of logical over creative skills. However, recent research has suggested that these skills in fact complement each other, and there has been an increasing number of computer scientists who are also musicians or artists, especially evident in numerous colleges around the nation. As a child, my mother dreamt that I would be a professional musician. I have played the viola since the ripe age of ve and the oboe since age ten, and I arrived to Pacic Lutheran University expecting to contribute to the arts community while majoring in my second passion, computer science. However, it became apparent that participating in ensembles and pursuing a Bachelor of Science was infeasible, and I had to choose which discpline to drop. But how could I do that? I loved music more than anything, but I recognized that employment in music was very slim in contrast to living in one of the largest technology 1

meccas in the nation. Although I still have been able to minor in Music, I would like to prove that music and programming can thrive harmoniously. My hope is that this project will ultimately change the degree system so that students may not be faced with choosing one discipline like myself and other students with similar interests did. Following the release of the Kinect and its SDK by Microsoft, interests for pursuing this technology beyond its initial capacity for games blossomed. Eugene Goosens, an English conductor, stated: With a perfect orchestra you can do what you like . . . You can draw a sort of immense emotional throb out of the air merely by curving your hand. You can get brilliant waves of sound merely by a twist of the wrist. You can make sudden and absolute silence by a gesture. It is the most wonderful of all sensations that any man can conceive. It is through this statement that it seems rather evident that the Kinect may be designed to enhance gesture-based activities considerably, especially conducting. Condescension unjustiably emerges from both the arts and science communities, and this project is designed to bridge the gap between the music and science elds. Thus, the project is targeted towards both musicians and non-musicians alike for a mutual appreciation of the compatibility between the arts and technology. The tentative structure of the conducting application is that in each module, a music professor will demonstrate to the player proper conducting techniques, then interactively ease the player into becoming condent enough to independently conduct a short excerpt of music. The lessons themselves will be simple enough for the player to conduct on a rudimentary level, thus allowing exibility for prior musical background. I have owned the Kinect hardware for use with my Xbox 360 since its release in 2010 and this has given me considerable familiarity with the technologys capabilities. Frankly, I have contemplated about this idea for a capstone for over two years already. To my dismay, however, I recently discovered an application designed for the Kinect called Kinductor, which was created by a team of three students and professionals in a 24-hour time period for Code Camp.1 However, I have condence that this capstone project will have not only a similar underlying structure with gesture-based control and time-stretching algorithms but also more features such as interactive videos and a dierent design for conducting. Additionally, I hope to improve my software engineering knowledge as well as improving my ability to learn independently. Although I still have a long process ahead, by researching the Kinect SDK and familiarizing myself with the core elements involved for the creation of the application, I feel well equipped thus far for future research.

2
2.1

Goals
Functional Goals
To bridge the gap between science and art by combining my interests of music and programming To create a conducting application using the Kinect SDK that will gradually teach the user basic conducting patterns and gestures
1

http://kinductor.codeplex.com

 To embed videos of interviews and lessons with PLU music professors to teach users how to conduct correctly To introduce the user to a diverse repertoire of works (hopefully with previous recordings of the ensembles to be conducted) Potential modules: patterns, styles, cues, fermatti, dynamics, tempo changes, facial expressions, eye contact, drawing with gestures, choral conducting

2.2

Learning Goals

To gain knowledge of C#, the Kinect SDK and open-source libraries utilizing various features (e.g. DirectX) To learn how to more eciently and eectively research using online materials To improve problem-solving (debugging) skills and software engineering methods and processes To gain more in-depth experience in conducting techniques

Predicted Requirements
Microsoft Kinect device (already own) Kinect AC Adapter (approximately $10) Programming with the Kinect for Windows Software Development Kit: Add gesture and posture recognition to your applications by David Catuhe (available at Amazon.com for $18.45 on Oct. 2, 2012) Meet the Kinect: An Introduction to Programming Natural User Interfaces by Sean Kean (available as an eBook at the PLU library)

Predicted Challenges
Working independently without the assistance of other students The relatively new technology of the Kinect and general unfamiliarity of the SDK and C# personally as well as among the Computer Science faculty Organizing each step in the process of creating the application feasibly while meeting deadlines Continuing communication and collaboration with the music professors at PLU

5
5.0.1

Approximate Timeline
September

Research of the Kinect Toolkit library and similar additions as well as basic implementation of Kinect applications and gesture-based control; collaboration with PLU Music professors on structure of application relative to feasibility. Create project website and developer blog. 5.0.2 October

Continued experimentation with the Kinect SDK and C#. Research on DirectX and timestretching algorithms for audio clips. Further research on possibilities of gestures in the SDK as well as more-detailed idea of the layout of the application. Requirements Document started/completed. 5.0.3 November

More specic research and more complex experimentation of the Kinect SDKs features and limitations. Experiment with reections in SDK. Design documents related to layout of application (including menus and features) completed. 5.0.4 December

Preparation for ocial implementation of conducting application using Kinect SDK and supported libraries. Design presentation and planning with PLU music professors. 5.0.5 January

Occasional coding for recognizing specic gesture patterns for both arms and tempo changes. 5.0.6 February

Continued coding for gesture pattern recognition, tempo changes, and fermatti. Implementation of time-stretching algorithms in audio clips. 5.0.7 March

Continued coding for cues, styles, and dynamics. Implementation of DirectX for videos of music professors. 5.0.8 April

Finalizing menu and layout as well as additional features. Clean-up of code and optimization. Preparation for nal presentation. 5.0.9 May

Preparation for nal presentation with plans for further development of application 4

Potential Works to be Used in the Project

Luther, Martin. A Mighty Fortress is Our God (Choir of the West) Tchaikovsky, Pyotr Illyich. Fifth Symphony: Second Movement (University Symphony Orchestra) Marquez, Arturo. Danzon No. 2 (Simon Bolivar Youth Orchestra) http://www. youtube.com/watch?v=3vwZAkfLKK8 Shostakovich, Dmitri. Festive Overture, Op. 86 (University Wind Ensemble) Dvorak, Antonin. New World Symphony: Movement Four (Minnesota Youth Symphonies) Beautiful Savior (Combined Choirs of PLU)