BMUS 33181244

Making Music with Motion : Virtual Instrument with Musical Expression

Abstract Virtual instrument has become a promising area in the music industry and is currently being developed for expressive musical purposes. For my final project based on this area, I will be using Kinect software. Kinect is a motion capture interface which is capable of capturing and analysing the recognised 3D gestural motions of a body; in this case the movements of a performer. In turn, sound is generated by the Max/MSP environment by translating and mapping the received motion data into other data such as MIDI parameters and visual signal processing.
1

In the project, the interaction between the motion gesture and

musical performance and sound mapping will be explored in detail. For example, this essay will discuss if there is such distinction between New Interfaces for musical expression and new interfaces for controlling sound by analysing the meaning of 'musical expression' and how it requires not only a good control of interface but also virtuosic mastery of the instrument it controls. Moreover, through an actual implementation of the system, research and analysis, I hope to find other areas in which one could further explore in the future. Introduction The possibility of easily creating new interfaces to control music has become a reality since the introduction to computer generated sound. Due to the development of and fast growing modern technology, many technically oriented musicians have experimented with these technologies to change various aspects and ideas of musical performance. On one
1

Max/MSP/Jitter: http://cycling74.com/products/maxmspjitter/

BMUS 33181244

hand, much effort has gone into designing new ways of sound generation. On the other hand, some effort has gone into designing new controllers or physical devices that implement a motion sensing technology and translate that motion (generated by the performer) into a (MIDI) signal that, in its turn, controls a sound synthesis device. However, one must realise that such new interfaces which make sound does not necessarily mean that they are music. In other words, the controller of the interface is technically a musician who must have the ability to play the instrument in a meaningful way in order to create the qualities of expression in computer music performance. This area has become an international interest and new methods are continuously being explored in order to enhance the expressiveness of computer music.2 A well-designed instrument supports the ability to play the music by allowing the user to have enough control freedom to make music while also being sufficiently simple and userfriendly to allow one to actually learn to play the instrument.

Background Virtual Instrument Recently, much of the research has been focused on the creation of new musical instruments such as the LightHarp MIDI controller. Examples of performances on the instrument can be easily found on YouTube.3 It is a versatile instrument that allows a varied

Dobrian, C. and D. Koppelman, The E in NIME: musical expression with new computer interfaces in Proceedings of New Interfaces for Musical Expression (NIME) (IRCAM, Centre Pompidou Paris, France, 2006 ) 277282
3

http://www.youtube.com/watch?v=sLVXmsbVwUs

BMUS 33181244

combination of simultaneous continuous control modes.4 A different instrument, SoundSaber is a musical instrument based on optical infrared marker-based motion capture technology. This is most probably influenced by the emergence of modern, electronic instruments (specifically those based on computers) and the general boom in electronic musical genres. The computer is used to create arbitrary mappings between gesture and sound; thereby providing the possibility of computer supported sound and directed musical interaction. Therefore in fact, the computer allows the creation of new interfaces and sounds which were never before possible. Unfortunately, with so much freedom to design the mapping between gesture and sound, there has not been a strong set of design criteria developed. In this particular issue, M. Wanderley and P. Depalle provide an excellent review of the state of the art of gestural control of sound synthesis and discuss strategies and guidelines for the design of these mappings.
6 5

With only a few exceptions, much design work has focused on a single

person who will be the first one to play the instrument with the optimistic outlook that other players will emerge. One particular instrument that I personally find interesting and fascinating is the Akai EWI series. EWI stands for Electronic Wind Instrument controller and at present, there are two versions in the series; EWI USB and EWI4000s. I believe the most recent EWI4000s is an evolution of wind instruments. It features multiple fingering modes allowing the performer to be able to play pretty much all the wind and brass instruments. The musician is also able to plug a wireless MIDI or audio set up into the base of the EWI, and move freely around the

S. Favilla, Real-time Control of Synthesis Parameters for LightHarp MIDI Contro ller in Proceedings of the 1997 ACMA Conference.Auckland, New Zealand (1997)
5 6

http://www.youtube.com/watch?v=XEmhGACZ-O8

M. Wanderley and P. Depalle, Gestural control of sound synthesis in Proc. IEEE, vol. 92 (Apr. 2004) 632 644

BMUS 33181244

stage without the unnecessary bother by wires.7 One can further find examples of performances on the EWI also on YouTube.8 I first saw this instrument on a TV program, played by one of member the group called House Rulez. Their performance was very impressive and it was amazing how the wind instrument controller generated an electronic sound. Apart from the MIDI keyboard, it is the first MIDI controller that I have come across and it has intrigued me so much that it has given me dreams to create an even more advanced MIDI controller likewise with the latest technology available in the future. Nevertheless, one cannot think little of the difficulties and issues that designers of new electronic instruments deal with. A couple major issues include: the performer's conceptualisation of the instrumental interface, the kinds of sounds that are to be made and the qualities of variability available in those sounds. These issues can be very difficult as the idea of the instrumental interface includes everything from the performer's physical gestures to the connection of these gestures with the resultant sounds and furthermore, each problems need to be addressed through new techniques for creating mapping between controllers and the sonic results. Most of novel musical controllers that have already been made are to fall into disuse simply due to the fact that there are only few musicians other than the original designers that are committed enough and able to master the intricacies of each interface. In order to tackle and overcome these problems, certain elements will be considered in my project that can be embodied through the development of intimacy between the performer/controller and the device.

7 8

www.akaipro.com/ewi4000s http://www.youtube.com/watch?v=w65uVO3KOFU

BMUS 33181244

The two most dominant elements that will be covered, dealing with the creation of these new interfaces as musical instruments include the following: 1) The necessity for these new interfaces to support a virtuosic performance but at the same time being accessible and simple 2) The definition of an appropriate mapping between the players control of the instrument and the sound produced For the first issue, as already mentioned, the difficulty arises because it is relatively simple to create an interface that allows for any user to create musical sounds with little practice and the appropriate computer support. However, it is open to debate whether these instruments allow the player to produce much musical expression. Debate over the second, defining the mapping in creating a new interface, generally centres on what types of sounds should be mapped for the musician to control. In many ways, this is as complex as the notions of what is musical and what is not. I believe that it can be said that many people refuse to play or even try out an instrument due to its many difficulties to do so. Furthermore, most of the time, the idea of music itself is given more importance over the musician or his tools. However, it is obvious that music grows and changes with the perfection of the physical means of the instruments and the invention of playing styles. This goes for not only computer musical interfaces but for any musical instrument such as the piano. Nevertheless, for most musicians, this sort of experimentation or project is seen to be of the historic and golden age sort, with no possibility or need to be resumed. So far the art of the interface between physical gesture and abstract function is respected only by aerospace and sports equipment designers. At STEIM the

BMUS 33181244

emphasis is on what could be called 'instrumentation'.

In music system design, the meaning

of instrumentation can be used for many things; from the mechanical design of sensors and controllers, through the electronics and software of interfaces and finally to the modelling of the higher level relations between performer and composition. The issues here are many and the need for computer instrumentation varies from the simple need for a controller for the articulation of a well-defined musical parameter to analytic interfaces which hear musically and provide data to influence rather than to control. About Kinect Before discussing in more detail about the virtual instrument and its musical expression, let me review the background of Kinect and share some information on how the device actually works. Kinect is a product made by Microsoft and it is a full body tracking and motion sensing input device for Microsofts Xbox 360 game console. It is among the recent advances of human computer interaction applications after its first release in November 2010.10 The device is basically a depth camera. If we take a look at a normal camera, it collects the light that bounces off the objects that are in front of the device. Then, they turn the lights into an image that resembles what is seen through the camera lens. In comparison, Kinect records the distance of the objects that are placed in front of it. This is done by using an infrared light to create a depth image that captures not what the objects look like, but where they are in space.11

9 10

http://www.steim.org/steim/texts.php?id=3

Microsoft Xbox 360 Kinect Launches November 4- http://gizmodo.com/5563148/microsoft-xbox-360kinect-launches-november-4

11

Greg Borenstein, Making Things See (O'Reilly Media January 2012) 440

BMUS 33181244

The interface enables players to control the console with their own bodily motions and voice.12 For example, in a game of golf on Kinect Sports 2, the player can change their club by saying 'change club to ' and then stand in front of the device and create a gesture just as if they were holding a real club. This freedom is achieved by the depth camera and microphone of the device capturing and analyzing the image and sound of the users. Since this controller-free interface has the ability to extend the degree of freedom and expressiveness of the users, similar to my own project, many researchers and developers have tried to apply the interface in a way that Kinect is not only for the game console, but also for controlling their own applications. Various applications of the motion detecting device have now been presented to the world as open source drivers for Kinect have recently developed through their own efforts. In the case of my project, it is the key application for a virtual instrument. In terms of other examples, I read an article also based on a similar theme to my own using the Kinect for creating musical expression. In this article, the creators introduce their system which was designed to create and control music according to the user's body motion via Kinect. 'Through Kinects ability to interpret a users whole motion, users can control the sound generation system with the rich range of movement of the body.'13 In this work of Interactive Music, they used the OpenNI framework and NITE Middleware from PrimeSense.

12 13

http://www.xbox.com/kinect

Yoo, Min-Joon, Jin-Wook Beak, In-Kwon Lee, Creating Musical Expression using Kinect in Proceedings of the International Conference on New Interfaces for Musical Expression (NIME 2011) 324-325

BMUS 33181244

Musical Expression Defining expression The subject of 'musical expression' itself, is in fact a debatable matter. Nevertheless, one can define its meaning as the art of playing music with communication and feeling. A performer aims to communicate and manipulate such feelings and musical variety through the way they play their instrument. For example, elements such as dynamics (forte and piano), phrasing, differing qualities of touch and articulation on the instrument, colour of sound, intensity and energy. As a result, he/she aims to allow the audience to enjoy or relate to the emotional, musical experience. The performer may only interpret the composer's expressive intentions or have an interpretation of their own of the music. One could further suggest that musical expression is the element which separates the performance to the composition. Other means of producing musical expression is by the human body itself. For example, expression can be closely related to breath, and the voice's natural ability to express feelings of deep emotions. A closer link to my own project, expression has to be made through bodily language. In other words, a performer may move or sway with the music in order to express the certain emotions they wish to portray through the music. In consideration of my instrumental interface, the body movements of the performer are in fact the instrument to create sound and music. However, one must understand that the emotional content of musical expression is something different from the emotional content of specific sounds. For example, a loud 'bang' noise may startle a person but the specific sound in itself cannot really be described as expressive. In contrast, if a sudden loud sound was incorporated into a song with the intention of startling the audience at that point of music, it may be described as musical expression. Furthermore,

BMUS 33181244

the components of musical expression continue to be the subject of extensive and unresolved dispute. More specifically in discussion of electronic music and my project, there are certain issues dealt with the relationship between the performer's physical gestures and the sonic results. In comparison to acoustic instruments, this is where issues of consistency, continuity, and coherence come into play. For example, one may question if the same physical gesture will always produce the same kind of sound or if a slight change in gesture would also result in a slight change in sound. We refer to this as the cognitive retraining problem, because to solve it, performers have to retrain how they conceive of the sounds being produced. Nevertheless, one can build an instrument that is consistent, continuous, and coherent in both gesture and sound space and yet is a poor instrument if it does not allow for sufficiently nuanced sounds. In other words, simple mappings from a control value to a synthesis parameter can result in instruments lacking in rich sounds and controllable musical variety. The problem with such simple mappings is that, no matter how well you play it, the sounds get tiring and frankly quite boring. In the case of pitch, for example, as the pitch is changed in an acoustic instrument, the timbre automatically changes in various subtle, coordinated ways due to the physics of strings and resonances. If we simply change the pitch of most simple synthesis algorithms, such as FM, without making any adjustments to timbre, the result sounds insufficiently characteristic, that is, it leaves the resultant acoustic gesture underdefined and therefore sounding too simplistic. For example, this can be found in the case with using pitch-bend on an FM synthesizer. However, even in the case where one parameter is used to drive several, such as when the single parameter of MIDI note-on velocity is mapped to many timbral parameters, it can still be perceived as too simplistic: after extended listening,

BMUS 33181244

the timbre is understood to lie along only one continuum.14 Control and Expression Simplicity

I believe simplicity is a very important entity in this project. The controls and the mappings of gesture to sound programmed into the computer that mediates between the controller and the sound generator must be simple and direct in order that the audience is able to perceive the cause-effect relationship effectively. Variety

Another point which was taken into consideration is the variety of controllers and its relationships. It must be recognised that when one works with such simple gesture-sound relationships, the (performance) may be slightly boring for the audience. Therefore, there is a need for more complex mappings and a variety of controllers in order for the performer to make as many different sounds and combinations as possible to create a maximum musical performance.

14

Garnett, G., and C. Goudeseune, Performance Factors in Control of High-Dimensional Spaces In Proceedings of the 1999 International Computer Music Conference (San Francisco International Computer Music Association, 1999) 268 - 271

BMUS 33181244

Implementation Installation For my project I have used the external program 'Synapse'. For this software to run on a system, one must install both OpenNI and PrimeSenses Natural Interaction Middleware (NITE).
15 16

Furthermore, in order to allow the processing sketch to run, one must also install Processing with the following packages installed in the Libraries folder: SimpleOpenNI a simple OpenNI and NITE wrapper for processing oscP5 - an implementation of the OSC protocol for processing controlP5 - GUI and controller library for processing System Description Once all are installed and ready, to connect the Kinect software with my Apple MacBook, I used the modules developed in OpenNI. The PrimeSenses NITE provided useful APIs for the manipulation of naive data. Moreover, I also used Synapse to access the joint data extracted from the body motion; the Synapse streams the users skeleton data and sends the joint positions via OSC which allows you to easily use Kinect to control Ableton Live. Afterwards, I then implemented the Processing Sketch that I have made which listens to the OSC messages from the Synapse and responds to Max for live patch in Ableton Live.17

15 16 17

OpenNI - http://www.openni.org/ PrimeSense- http://www.primesense.com/?p=515 Ableton Live - http://www.ableton.com/

BMUS 33181244

Program Basics In a very short summary, there are three layouts on the Processing Sketch that contains Controllers such as Sliders, Buttons, Toggles and Knobs. While most of these controllers in the Processing sketch such as Toggles send OSC messages received by the movements sensed on Kinect to Max/Msp Patch. Other controllers such as the buttons on Layout 1 send the OSC messages to the Osculator Porgram which translates the OSC to MIDI messages. The Max Patch and Osculator Program then finally sends messages to control specified features on Ableton Live and Osculator sends the translated MIDI messages to Live also. The individual mapping strategies on each layout and Max Patch will be discussed in detail later on in this essay. How To Use In order to map the skeleton to your body, one must stand in front of the camera and hold the psi pose for several seconds. After doing this, a red skeleton will show on the screen bound to yourself. When this is done, he or she is ready to open the Processing Sketch and use the controllers by moving their hands over them. On the Sketch, a red and blue dot will act like a curser representing the left and right hand respectively sensed from Synapse. Furthermore, hand gestures moving up and down, side to side will be read by Max Patch. The Processing Sketch and its Design Before looking into the mapping details of Max Patch and its relations with Ableton Live, I will first discuss the design of my Processing Sketch. In my Processing Sketch, there are three layouts which the performer can choose to use. The different layouts show various facilities which are specifically designed depending on its purpose and in a way easy for the performer to understand and follow. (The screen shots of the full processing source codes are

BMUS 33181244

attached at the very end of the essay.) Layout 1: The first layout represents a drum pad. The performer may use this to improvise and in order play a melodic line. Each box is assigned as numbers from 1 to 12. (Incidentally, one may assign different names to the pads so long as the set up in the Osculator Program follows.) When the performer hits the blue dot on a box, the numerical OSC messages are then sent to the Osculator which is then translated into MIDI notes chromatically from C2 to B2. As the performer hits at a box, the OSC message is sent to the Osculator program which changes the message into MIDI. However, in order for this to work, one must make sure that the 'Record' button at the top of the layout is switched on. This will be shown by turning red. Above the drum pad (and in every other layout also) there are buttons which control the playing and stopping of individual loops in the different channels in Ableton Live. The Left and Right arrows can be used to move from one channel to another and the + and - buttons can be used for selecting the individual loops within the chosen channel.

BMUS 33181244

Layout 2: In this layout, I have set up individual buttons for the different instruments within the Ableton Live channels. For example, the loop sample sound of a drum is already set up in Live which will play when the performer hits at Drum 1. Therefore one is able to choose a specific instrument they wish to play. If they wish to stop the loop to play, similar to Layout 1, the performer can choose the loop by the buttons on top, and press at the Stop button.

Layout 3: Layout 3 has several functions which need explaining. First of all, it represents a 2 octave piano keyboard. The first octave is on the top and the lower at the bottom. Similar to an electric keyboard, the sound made by hitting the keys are not necessarily a piano sound. This can be pre-setup by the performer on Live to a sound of their choice. Furthermore, on the side of the two octaves, I have created two buttons which the performer is able to use if they wish to change the key, or pitch of the notes. For example, if the key is changed to D, the first note of the keyboard would change to D.

BMUS 33181244

Secondly, in order to allow a more musical expressive performance, I have created buttons on this layout which can aid towards more variable sounds and effects. It allows the performer to control certain effects on Live which again, need to be pre-set up. I have created switches which will enable the record and launch buttons of each channel. These can be found in between the piano keys. These were made possible by linking the switches to Max Patch. The switch on the layout opens the Max Patch and hence enables the set ups on Max Patch to actually work. Therefore, when the switch is turned on, the performer can simply use their hand movements to control the buttons. For example, as the performer makes a leftward gesture with their left hand, the record button will be turned on and as he makes a rightward gesture with their right hand, the launch button will launch. The increasing processing speed of computing technology means that real-time operation is now possible.

BMUS 33181244

Mapping Strategies Definition of OpenSound Control OpenSound Control (OSC) was originally developed to facilitate the distribution of control structure computations to small arrays of loosely coupled heterogeneous computer systems. A common application of OSC is to communicate control structure computations from one client machine to an array of synthesis servers. The abstraction mechanisms built into OSC (a hierarchical name space and regular expression message targeting) have also proven to be useful in implementations running entirely on a single machine.
18

In the

context of my instrumental interface, the basic strategy can be understood as the following: Step 1- All the incoming gestural data received by Kinect software is translated into OSC messages with descriptive addresses Step 2- Then all other controllable parameters in the rest of the system are made OSCaddressable Therefore the gestural performance mapping is simply a translation of one set of OSC messages to another. This gives performers greater scope and facility in choosing how best to effect the required parameter changes.19

18 19

OpenSoundControl (OSC): http://opensoundcontrol.org/

Wright, M., A. Freed, A. Lee, T. Madden, and A. Momeni, Managing Complexity with Explicit Mapping of Gestures to Sound Control in OSC In Proceedings of the 2001 International Computer Music Conference (San Francisco, International Computer Music Association)

BMUS 33181244

Types of Mapping In discussion of the link between the human instrumental actions or gestures with the generation of sound, one can compare mappings in interactive systems to the playing of acoustic instruments. The role of mapping can be seen in two main different points of views as discussed in the article 'Towards a Model of Instrumental Mapping in Expert Musical Interaction'20. The first idea is understanding mapping as a specific feature of composition. In comparison to acoustic musical instruments, pre-set ups of effects and sounds can be made through mappings. The second is that it is in fact a part of the instrument. In the creation of the virtual instrument, it is clear that the mappings in Max Patch play a big role in actually allowing sounds to be generated. This in mind, it easily makes sense to think of mapping as a very important part of the system/instrument. For instance, as a violin player approaches his instrument in different ways to create different sounds, an interactive system performer also can make use of the mappings within different contexts to produce a variety of sounds; the more complex the mappings that are set up results in maximising the various musical possibilities for the performer. The different types of mapping settings can be categorised into three. J. Rovan has identified these as one-to-one mapping, divergent-mapping and convergent mapping.21 One-to-One Mapping: This type of mapping is when each independent gestural output is assigned to one musical parameter. (Usually via a MIDI control message.) This is the simplest mapping scheme but unfortunately, it is consequently the least expressive. It takes

20

A. Hunt, M. Wanderley, and R. Kirk, Towards a Model for Instrumental Mapping in Expert Musical Interaction. In Proceedings of the 2000 International Computer Music Conference (San Francisco, International Computer Music Association, 2000) 209 - 212
21

J. Rovan, M. Wanderley, S. Dubnov, and P. Depalle, Instrumental Gestural Mapping Strategies as Expressivity Determinants in Computer Music Perfo rmance. In Proc. of the Kansei Workshop (Genova, 1999) 68-73

BMUS 33181244

direct advantage of the MIDI controller architecture. Divergent Mapping: This is when one gestural output is used to control more than one simultaneous musical parameter; one-to-many. Although it may initially provide a macrolevel expressivity control, this approach nevertheless may prove limited when applied alone, as it does not allow access to internal (micro) features of the sound object. Considering the violin again, the violin bow actually influences many aspects of the sound such as volume, timbre, articulation and (to some extent) pitch. This is therefore an example of a one-tomany mapping.22

Convergent Mapping: In this case, many gestures are coupled to produce one musical parameter. Therefore, opposite to divergent mapping, this type can be simply understood as many-to-one. However, for this scheme to work, it requires previous experience with the system in order to achieve effective control. Although this is in fact harder to master, consequently it proves to allow more musical expressiveness. If we explain convergent mapping in terms of the acoustic violin again, one could consider the idea of controlling the volume of the instrument in many ways. In other words, there is no single control but rather a combination of approaches such as bow-speed, pressure on the bow and even finger positioning. Therefore this can be seen as many inputs controlling a single parameter. If we take a look at previous works related to mapping, one example can be found published by I. Bowler, A. Purvis, P. Manning and N. Bailey.23 In this article, they present a general way to map N articulation parameters to M synthesiser control parameters through
22 23

J. Rovan, Instrumental Gestural Mapping Strategies 69

I. Bowler, A. Purvis, P. Manning and N. Bailey, 'On Mapping N Articulation onto M Synthesiser-Control Parameters.' in Proc (1990) 181-184

BMUS 33181244

interpolation of the M control parameters placed at coarsely quantised positions in the N dimensional performance space.24 On my own system, I have mostly used the Divergent Mapping to control the parameters within the Ableton live. By using the X or Y axis of the hand positions, I was able to control these values. Synapse, sends OSC (OpenSound Control) values of the skeletal points to Max Patch. (The spider diagram like mappings which I created in Max Patch will be discussed and shown in detail later in the essay.) The points that are traced from the body are at the points of joints: right-hand, left-hand, right-elbow, left-elbow, right-foot, left-foot, right-knee, left-knee, head, torso, neck, left-shoulder, right-shoulder, left-hip and right-hip. All of these points also have the ability to move up, down, left, right, forward and back as the performer moves around in front of the sensor. In some sense, the body becomes both the performer and instrument.

The Max/Msp Patch In order to explain my implementation on Max Patch, I have made a few screen shots which demonstrate how all the applications are linked together when the performer makes a gestural command. In the first screen shot, the object 'udprecieve' receives OSC messages either from the Processing Sketch or directly from Synapse and 'udpsend' sends the received message back to the specific OSC port. In this case, '12347' and '12003'.

24

J. Rovan, Instrumental Gestural Mapping Strategies 70

BMUS 33181244

Second example will show what happens when the buttons at the top of each layout in Processing Sketch is pressed by the performer. If we take the stop button for an example, the OSC message which I named 'stop' will be sent to Max Patch which 'udpreceive' receives. Thereafter, the object 'match stop' is activated and track that was playing in Ableton Live will stop.

BMUS 33181244

In the Processing Sketch, buttons in layout2 are assigned as 'b1' to 'b24'. These are linked to its relative objects in Max Patch, so when the buttons are pressed, the related object 'match' is activated.

When the buttons in layout3 in the processing sketch are pressed by the performer, it enables and activates the Max Kinect events in Max Patch. These events can be pre-set up in various combinations. For example, I may decide for a certain channel to turn off and a particular loop/track in another channel to turn on with certain effects by creating the mappings in Max Patch. These commands will all activate as the 'Target State' is switched on by the buttons in

BMUS 33181244

Layout 3.

The complex mappings shown in the last screen shot are also shown in a simpler version in Ableton Live as shown below.

BMUS 33181244

THE GESTURE-SOUND RELATIONSHIP The Performer-Instrument Relationship Finally, now that I have discussed all the technical ideas and the layout of my interactive system, I will now refer back to the idea of musical expression. Referring back to the idea of musical expression discussed in the background, I bring back the idea that music is not only about the pitches. In other words, there are many different ways to create musical expression by manipulating the sounds. This is possible through performing gestures which will change the tempo, sound level, timing, intonation, articulation, timbre, vibrato and more. Furthermore, continuous control of musical elements such as portamento, timbre, dynamics are often more expressive and more satisfying dramatically than simply mapping motion to pitch. The concept of gesture can be dened as 'a movement of part of the body to express an idea or a meaning', and the extensive presentation of its relationships with sound in music has emerged.25 Therefore, new technologies used for musical performance should build meaningful relationships between sound and movement, in order not only to ensure coherence in the musical experience of the performer, but also to preserve the musical communication between musicians and listeners.26 In an attempt to incorporate some of these ideas in to my system, I have demonstrated some gestural effects by using the X and Y axis messages on Max Patch and further sound effect controls such as cut off frequency (which changes the sound level/dynamic of the sound) and delay in the final Layout of the Processing Sketch. The idea of using the X and Y axis does not only effect the sound but also up to a certain
25

R. I. Godoy and M. Leman, Why Study Musical Gestures? In: Musical Gestures: Sound, Movement, and Meaning, ed. by R. I. Godoy and M. Leman (2009) 29
26

C. Cadoz, Instrumental Gesture and Musical Composition In Proc. of the 14th International Computer Music Conference (Kln 1988) 112

BMUS 33181244

point, allows some kind of visual/gestural expression to be projected onto the audience In other words, a gesture-sound or performer-instrument relationship is established on the stage through the performance gestures. One could further suggest that it is this idea that separates a live performance to a recording. In the words of J. Davison, (gestures) 'are often emphasized in some manner to heighten the ritualistic qualities of the communal experience that is live music performance.'27 One could suggest that the performer and instrument becomes a single entity which is one of the ideas of this project. However, there is another kind of gesture and sound relationship apart from the idea of musical expression. This idea is perhaps a bit more straight forward and can be discussed in terms of an acoustic instrument. Physical gesture takes in fact a huge role in the process of not only creating musical expression but in creating the actual sounds in the first place. In other words, musical instruments such as the violin require the performer to blow, pluck, strum, squeeze, stroke, hit or bow. As a result, the acoustic instrument vibrates in a manner determined by the energy transmitted onto it which is generated by the performer's gestures. These then in turn, determine the amplitude, pitch and timbre of each sound event, whilst also visually engaging an audience in the moment of performance. In terms of my project however, this idea becomes slightly different. For there is no touching involved such as the blowing of an wind instrument and the plucking of a stringed instrument and with the idea that the body becomes the instrument itself, the performer only needs to make gestures in mid-air and then Synapse does the 'clicking' or 'pressing' instead. However, as a violin would not sound if not struck, this idea is the same within the interactive system as the set buttons are not pressed, no sound will be generated.
27

J. Davidson, The Social in Musical Performance, in The Social Psychology of Music, ed D. Hargreaves & A. North (Oxford: OUP,1997) pp. 20928

BMUS 33181244

Nevertheless, in terms of both acoustic and electronic instrumental performances, one can acknowledge that musicians use gesture both as a means to engage the production of sound on an instrument and as an expression of an deliberate intention. In other words, this facet of gesture is intended to convey something of the emotional interpretation the performer wishes to convey through the medium of the musical instrument. 'The author of the instrumental interface proposes that an experienced musician develops a proprioceptive relationship with his or her instrument that is, a largely unconscious perception of movement and stimuli arising within the body from the relationship between the human body and the instrument during performance; a direct relationship is established between the physical gesture, the nature of the stimuli and the perceived outcome. The resulting awareness is multifaceted and has been at the centre of musical performance of all genres for centuries.28

EVALUATION During the initial stages of creating the instrumental interface, I had issues with choosing the most suitable application for my project and went through a couple of trial and errors. First of all, I had in fact considered using a finger tracking application 'KinectCoreVision' rather than the Synapse skeletal tracking which I chosen in the end. Not so different from Synapse, the finger tracking application which was written in C++ code by Patricio Gonzalezvivo, detects the edge of the hand shape and sends OSC messages from the positions of those edges, i.e. fingers. However, the problem I had with this was that the process of detecting the fingers were not stable and easy. Even the slightest points of the hand
28

G. Paine, Towards unified design guidelines for new interfaces for musical expression In Organised Sound 14/2 (Aug. 2009) 142-155

BMUS 33181244

were treated as fingers and the tracking was messy. Furthermore, I tried a second method which was the gestural recognition using GIDE (Gesture Interaction Designer) created by Bruno Zamborlin, Phd student from Digital Studio at Goldsmiths, University of London.29 However, I had problems with frequent errors using this application also. The issues with applications using gestural recognition can be found in many articles such as the article 'Glove-Talk II: A neural network interface which maps gestures to parallel format speech synthesizer controls' by S. Fels and G. Hinton30. The article discusses that since humans do not reproduce their gestures very precisely, natural gesture recognition is rarely sufficiently accurate. Classification errors and segmentation ambiguity cause many of the problems with gesture recognition. Only when gestures are produced according to a well-defined formalism, such as sign language, does automatic recognition have acceptable precision and accuracy.31 Although my final choice of using the Kinect software is satisfactory, one little downside is that the device has to be set up about 1.5 metres away from the performer. Therefore, it cannot really be used on a small stage or performance area. Nevertheless, the device picks up gestures accurately and is links well with the other applications.

For my Processing Sketch, I had again initially had something slightly different and made changes during the process. Instead of three layouts, there were four in which the third had the keyboard and the fourth had the extra effects buttons. However, after some use of the system myself, I felt that the navigation around the four layouts were a bit too much of a hassle and so decided that putting layout 3 and 4 together was more convenient. This

29 30

http://www.brunozamborlin.com/home/

S. Fels and G. Hinton, Glove-TalkII: A neural network interface which maps gestures to parallel formant speech synthesizer controls, IEEE Trans. Neural Networks, vol. 9, (Jan. 1998) 205212
31

J. Kramer and L. Leifer, The Talking Glove: A speaking aid for nonvocal deaf and de af-blind individuals, in Proc. RESNA 12th Annu. Conf.,(1989) 471472

BMUS 33181244

allowed me to add in extra effects much more easily while improvising a melody on the keyboard. I believe that the three different finalised layouts are designed as user friendly in their own different ways which allow the performer to actually enjoy playing. However, if I was to change something or improve on it, perhaps it would have been much easier and simpler to design the keyboard layout just like a real keyboard. The piano keyboard that I have designed was perhaps a bit unnecessary and users may have understood the original design much faster. Nevertheless, the overall design is quite easy to follow but practice is needed to understand the system and to navigate around the Processing Sketch with freedom.

One of the key ideas of my interactive system was to make musical expression possible which I believe I have succeeded up to a certain extent with some complex mapping ideas. For example, the cutting frequency effect allows the change of small and big sounds (like dynamics), there is a delay effect which changes the tempo momentarily. The different combinations of Max Kinect events also allowed me to set up various timbres and textures of sounds which I could change with a single button. However, the disadvantage of this is that the combinations do need to be made in advance. Furthermore, even more complex mapping ideas and structures would probably allow the performer to have maximum, precise control.

If I was to suggest further improvements to the system and performance for future references, I would suggest a faster laptop processor would be necessary in order to reduce the latency while pressing the notes/buttons on the Processing Sketch or triggering the audio loops. Moreover, if there was a better and more accurate application for finger detecting, I would consider it more useful for my virtual instrument as it would allow more fluidity and flexibility in controlling the individual knobs and buttons.

BMUS 33181244

CONCLUSION

Some interfaces may require a longer practice time in order to achieve good results but this is no different to learning an acoustic musical instrument. It would be close to impossible to create an interface which can be understood and learnt straight away. In light of this idea, I propose an argument that perhaps the preconceptions about computers interfaces expected to be easy and simple. This is somewhat ironic and misleading. As already discussed many times throughout this essay, simple controls or mappings do not allow a varied and expressive outcome. Therefore it does not make sense to expect an interface to be easy as such as it can be mastered in minutes or that the performer not needing much skill to use it. In other words, I believe it is more suitable to say that an interface should be complex but user friendly, as in having a easily recognisable design and names of buttons and toggles, but not necessarily easy to master the actual use of them. Complex mappings need time to learn but they allow users to develop strategies and skills in order to master controlling the parameter spaces.

Overall, I believe my instrumental interface is both quite effective in complexity for musical expression but also has a user friendly design. The design is easy to follow and understand once you get used to it. However, although some practice will be needed and it still has its limits, to some extent, one can suggest that my processing sketch is perhaps easier to learn than trying to learn all the individual musical instruments used in the interactive system, such as the drum kit or the keyboard. From this point of view, it can be said that I have succeeded in creating a 'simple' virtual instrument which also allows much freedom for musical expression.

BMUS 33181244

Bibliography

Borenstein, Greg, Making Things See (O'Reilly Media January 2012) 440Dobrian, C. and D. Koppelman, The E in NIME: musical expression with new computer interfaces in Proceedings of New Interfaces for Musical Expression (NIME) (IRCAM, Centre Pompidou Paris, France, 2006 ) 277 282 Bowler, I., A. Purvis, P. Manning and N. Bailey, 'On Mapping N Articulation onto M Synthesiser-Control Parameters.' in Proc (1990) 181-184 Cadoz, C., Instrumental Gesture and Musical Composition In Proc. of the 14th International Computer Music Conference (Kln 1988) 112 Davidson, J.,The Social in Musical Performance, in The Social Psychology of Music, ed D. Hargreaves & A. North (Oxford: OUP,1997) pp. 20928 Favilla, S. Real-time Control of Synthesis Parameters for LightHarp MIDI Controller in Proceedings of the 1997 ACMA Conference.Auckland, New Zealand (1997) Fels, S. and G. Hinton, Glove-TalkII: A neural network interface which maps gestures to parallel formant speech synthesizer controls, IEEE Trans. Neural Networks, vol. 9, (Jan. 1998) 205212 Garnett, G., and C. Goudeseune, Performance Factors in Control of High-Dimensional Spaces In Proceedings of the 1999 International Computer Music Conference (San Francisco International Computer Music Association, 1999) 268 - 271 Godoy, R. I. and M. Leman, Why Study Musical Gestures? In: Musical Gestures: Sound, Movement, and Meaning, ed. by R. I. Godoy and M. Leman (2009) 29 Hunt, A., M. Wanderley, and R. Kirk, Towards a Model for Instrumental Mapping in Expert Musical Interaction. In Proceedings of the 2000 International Computer Music Conference (San Francisco, International Computer Music Association, 2000) 209 - 212 Kramer, J and L. Leifer, The Talking Glove: A speaking aid for nonvocal deaf and de af-blind individuals, in Proc. RESNA 12th Annu. Conf.,(1989) 471472 Microsoft Xbox 360 Kinect Launches November 4- http://gizmodo.com/5563148/microsoft-xbox-360-kinectlaunches-november-4 Paine, G., Towards unified design guidelines for new interfaces for musical expression In Organised Sound 14/2 (Aug. 2009) 142-155 Rovan, J., M. Wanderley, S. Dubnov, and P. Depalle, Instrumental Gestural Mapping Strategies as Expressivity Determinants in Computer Music Performance In Proc. of the Kansei Workshop (Genova, 1999) 68-73 Wanderley, G. and P. Depalle, Gestural control of sound synthesis in Proc. IEEE, vol. 92 (Apr. 2004) 632644 Wright, M., A. Freed, A. Lee, T. Madden, and A. Momeni, Managing Complexity with Explicit Mapping of Gestures to Sound Control in OSC In Proceedings of the 2001 International Computer Music Conference (San Francisco, International Computer Music Association) Yoo, Min-Joon, Jin-Wook Beak, In-Kwon Lee, Creating Musical Expression using Kinect in Proceedings of the International Conference on New Interfaces for Musical Expression (NIME 2011) 324-325

BMUS 33181244

Websites

Max/MSP/Jitter: http://cycling74.com/products/maxmspjitter/ http://www.youtube.com/watch?v=sLVXmsbVwUs http://www.youtube.com/watch?v=XEmhGACZ-O8 www.akaipro.com/ewi4000s http://www.youtube.com/watch?v=w65uVO3KOFU http://www.steim.org/steim/texts.php?id=3 http://www.xbox.com/kinect OpenNI - http://www.openni.org/ PrimeSense- http://www.primesense.com/?p=515 Ableton Live - http://www.ableton.com/ OpenSoundControl (OSC): http://opensoundcontrol.org/ http://www.brunozamborlin.com/home/

BMUS 33181244

Screen shots of the processing source codes for the interactive system

12 buttons in layout1 (b1 ~ b12)

24 buttons in layout2 (tt1 ~ tt24)

8 buttons in layout3 (t1~t8)

BMUS 33181244

BMUS 33181244

BMUS 33181244

BMUS 33181244

BMUS 33181244

BMUS 33181244

BMUS 33181244

BMUS 33181244

BMUS 33181244

BMUS 33181244

BMUS 33181244