Вы находитесь на странице: 1из 8

Gestural Imagery in the Service of Musical Imagery

Rolf Inge Godøy

Section for musicology, University of Oslo P.O.Box 1017 Blindern, N-0315 Norway r.i.godoy@imt.uio.no tel. (+47)22854064, fax. (+47)22854763

Abstract. There seem to be strong links between gestural imagery and musical imagery, and it is suggested that gestural imagery can be instrumental in trig- gering and sustaining mental images of musical sound. Gestural images are seen as integral to most experiences of music, and several practical and theo- retical musical disciplines could profit from focusing on these gestural images. Research in support of this is reviewed, and some topics for future research are presented.



The topic of this paper is how gestural imagery, meaning imagining or mentally simu- lating various gestures, can trigger, sustain, and enhance images of musical sound in our minds. This is part of a long-term research project, Motor-mimesis, which aims at enhancing our means for thinking musical sound in various tasks (composing, arrang- ing, improvisation, performance, analysis, music education, etc.) through mental images of sound-associated actions, including both sound-producing actions such as hitting, stroking, plucking, bowing, blowing, etc., and other sound-related body movements such as dancing and various kinds of sound tracing gestures. In the call for papers prior to a conference on musical imagery some years back, we tentatively defined musical imagery as 'our mental capacity for imagining musical sound in the absence of a directly audible sound source, meaning that we can recall and re-experience or even invent new musical sound through our 'inner ear'.' [1]. It could now be useful to propose a similar definition here of gestural imagery as 'our mental capacity for imagining gestures without seeing them or actually carrying them out, meaning that we can recall and re-experience or even invent new gestures through our 'inner eye' and inner sense of movement and effort.' It must be empha- sized that imagery is not just a matter of daydreaming or arm-chair contemplation, but is in fact the very basis for thinking and feeling: Memory, and hence, imagery, is at work in all perception and cognition, and, as pointed out already by the phenome- nologists at the end of the 19th century [2], there simply would be no perception and cognition at all without mental images of past and expected (future) events. Although there has been a number of studies of musical imagery during the last decades [3], the most intriguing, and also most practically oriented question is in my opinion that of what actually triggers and sustains images of musical sound in our minds. This question of what is the 'engine' of musical imagery has oddly enough

A. Camurri and G. Volpe (Eds.): GW 2003, LNAI 2915, pp. 55–62, 2004. © Springer-Verlag Berlin Heidelberg 2004

56 Rolf Inge Godøy

been given little attention by researchers, however, the prime suspect for this trigger- ing agent which emerges from various studies of musical imagery, is actually gestural imagery. In other words: Images of gestures seem to be efficient in evoking images of sounds. The challenge now is to substantiate this claim, both by providing evi- dence from available research reports on the neurophysiological and cognitive bases for the gesture-sound interaction in imagery, and to design schemes for practical ap- plications which can demonstrate that imagining gestures is actually an efficient strat- egy for evoking lucid and vivid images of musical sound in various music related tasks.

2 Gesture-Sound Links in Perception and Cognition

Many musicians are familiar with the experience that recall of music is facilitated by mimicking the sound-producing movements or other kinds of movements associated with a musical work, e.g. moving hands and fingers as if actually playing when recall- ing a piano piece, playing ’air guitar’ or ’air drum’ when recalling a song, making dance movements when recalling a dance tune, etc. In other words, there can hardly be any doubt that gestures are effective in triggering images of musical sound, some- thing which has in fact been documented through a series of experiments by Mariko Mikumo [4], and something which has been remarkably depicted by David Sudnow in his introspective account of jazz improvisation [5]. Generally, imagery research in any domain, be that visual, auditive, motor, etc., has inherently significant methodological problems because it is difficult to study what goes on within the mind, i.e. we can not place an 'observer' in people’s minds to register what is going on when people imagine various scenes, gestures and sounds. However, the last couple of decades has seen the emergence of some clever methods for capturing what is assumed to go on in processes of imagery, such as measuring reaction times, and effects of mental preparation and rehearsal, for various tasks [6]. In addition to such indirect methods, advances in methods for non-invasive neuro- physiological observation, such as fMRI, PET and ERP [7], have given us more in- formation on the workings of mental imagery. One significant finding here is that of 'functional equivalence', meaning the close resemblance between the neuronal appara- tus involved in actual actions and/or perceptions and in the imagined actions and/or perceptions. One possible interpretation of this could be that imagery does share many features with actual experience, hence, that studying the links between gestures and sound in actual experience could also give us clues as to the links between ges- tures and sound in imagery. Also, another interesting finding here is that certain mo- tor areas of the brain seem to be activated when imagining sound, hence that action imagery is activated concurrently with tasks of musical imagery. Neurophysiological research will in the near future no doubt come up with inter- esting findings on gesture-sound links in imagery. However there is also a consider- able amount of existing audition research which suggests close links between gesture and sound in perception and cognition. This research could be summarized as 'eco- logical' in the sense that its basic paradigm is to understand human audition as em- bedded in evolutionary constraints, hence, as trying to understand human audition as a holistic and cross-modal phenomenon where the different sense-modalities cooperate in order to extract meaning from what we hear in our environment. There are many

Gestural Imagery in the Service of Musical Imagery


instances of this in the domain of Auditory Scene Analysis [8], and just to mention one classic example, the so-called ’McGurk effect’ demonstrates that we may be tricked when seeing a certain sound-producing gesture to believe that we have heard something else than we actually have been presented with in the acoustic signal [9]. However, the most persistent project to explore and exploit gesture-sound links is that of the so-called ’motor theory’ of perception [10]. Going back more than three decades, the advocates of this theory have claimed that the various features in the sig- nal alone are not sufficient to account for perception and discrimination of language sounds, and that the listener actually mentally mimics the sound-producing gestures when trying to make sense of what is spoken, projecting gestural images on to the sound input in a top-down manner. The motor theory thus claims that perception in- volves a gestural simulation in the mind of the listener, and that learning to under- stand a language is actually a process of learning to imagine the sound-producing gestures of that particular language. The motor theory has been controversial. How- ever, the above-mentioned advances in neurophysiological research is now increas- ingly giving support for this theory [11]. Ideas from motor theory are fundamental here, not only for exploring gesture- music links, but also for structuring our images of musical sound as gesture units (see section 5 below). In this connection, research into the role of gestures in language is also relevant [12], as this indicates how gestures not only amplify certain rhetorical elements of speech, but suggests that gestures may have been the evolutionary basis for spoken language [13], and may be instrumental in actually generating utterances [14], i.e. that gestures in speech may play a cognitive role similar to the role of ges- tures in music. As to this last point, there is of course also a fast growing body of research into practical applications of gesture control in music [15], and experiences here can no doubt tell us more about the links between gestural imagery and musical imagery as well.

3 Separating Gestures and Sound

Although the classical separation of sense-modalities such as vision, hearing, balance, etc. may now seem questionable from what is known about the cooperation of many channels of sensory input in the brain [16], it will for strategic reasons be useful to conceptually separate gestures and sound here [17]. This means separating images of what we do from images of the effects of what we do, or separating the silent choreog- raphy of sound-producing and sound-accompanying actions from sonorous images. This means furthermore that we separate ecological knowledge of actions that we all have accumulated since our births (or even from a pre-natal stage), i.e. knowledge of action-trajectories and effort in stroking, blowing, hitting, kicking, etc., from ecolo- gical knowledge of resonant features of whatever sounding body (strings, tubes, vocal tract, etc.) and environment (rooms, ambient sounds, transducing medium, etc.) that we similarly have experienced by being in the world [18, 19]. First of all, this separation of gestures and resultant sound reveals quite clearly that what we often (or in most cases) think of as the meaning or content of music is actu- ally a matter of gestural images: Images of effort, velocity, contours, trajectories, gait, etc., could all be understood as gestural phenomena, as gestural images transmitted by sound and ’decoded’ in listening back to a gestural language as in ballet or pantomime

58 Rolf Inge Godøy

[20]. For example, when listening to the ferocious beating of drums, it is probably almost impossible to avoid having images of an equally ferocious activity of hands and mallets hitting the drum membranes, and conversely, listening to a slow, quiet, and contemplative piece of music would probably evoke images of slow and smooth sound-generating gestures. For disciplines such as musical aesthetics and musical semiotics, a focus on gesture could be very useful to characterize musical meaning. In our context here, such a shift of focus reveals that gestural scripts are deeply em- bedded in what we think of as musical imagery, making the ’pure’ sound a kind of ’residue’ which is left over once we remove gestural images. The consequences of this is that a cultivation of gestural images is not only legitimate, but can have some very attractive features in practical applications as well (see section 5 below). Secondly, this separation of gesture and sound may encourage us to understand music as organized not only by traditional principles for pitch, harmony, consonance- dissonance, motives, melodies, thematic development, etc. but equally well by prin- ciples for motor cognition, i.e. action-units or action-gestalts, motor programmes, motor-hierarchies, coarticulation, etc., as I shall briefly present in the next section. There can be no doubt that human motor cognition has a number of constraints on music making, e.g. need for alternating between effort and rest, need to conserve energy, thresholds for velocity, phase transitions with changes in tempi [21], etc., and that human motor cognition exerts schematic constraints on melodic, rhythmic and textural grouping [22, 23]. Thirdly, separating gesture and sound opens up the fascinating domain of motor imagery [24], hence, opens up for thinking musical imagery as simulation of gestures. This in turn opens up for all the advantages inherent in the dynamics of motor im- agery, such as imagining actions at variable resolution (mentally fast running through

a sequence of actions or slowly replaying all the details) and variable acuity (vague,

approximate, sketch-like recollections of actions and effort or very precise, salient images of trajectories). Musical sound can not (in principle) unfold faster than what it does without distorting its features, but motor programmes may create compressed overview images of gestures in the form of scripts or lists of highlights, similar to trailers in the promotion of films where the point is to pack as many salient scenes as possible (often the most spectacular and/or violent) into a 30 second advertisement. In fact, scores in common practice western notation are partially such gesture scripts, allowing for rapid running through or ’at a glance’rough impression of what the music is like (making some suspicious conductors prefer a quick glance at a score rather than spending time on actually listening through a work). This separation of gesture and musical sound prepares the ground for talking about mental simulation as the substance of musical imagery, meaning that musical imagery

is not a kind of abstract representational or propositional system, but is a matter of re-

enacting from a first-person, egocentric perspective what we have perceived in the world. This simulation view of cognition [16, 25] seems fortunately now to be gain- ing plausibility, and means that whatever kind of music we imagine, and regardless our level of expertise and/or the complexity of the music, there is always the possibil- ity of mentally mimicking gestures we think belong to the music (although perhaps only in a low-acuity, sketchy manner).

4 Motor Cognition

Gestural Imagery in the Service of Musical Imagery


Accepting that gestural images are integral to musical imagery, it could be useful to have a brief look at some principles of motor cognition [26]. There are physiological constraints on human action which are obvious in musical situations (e.g. singers, brass and woodwind players need to breathe, velocity of bow, hand and finger move- ments are limited within a certain range, etc.), but most of all, there are some cogni- tive constraints and principles of organization which are highly relevant for gestures in music, hence also for gestural imagery in music. The most important concept here is that of motor programmes. A motor pro- gramme is a mental image of an action or a sequence of actions, such as opening a door, sitting down, walking to work, or hitting a drum, playing a tune on the flute, or playing a long piece on the piano. A motor programme is thus a kind of script for how something is to be done, and this script may, as mentioned above, have a vari- able resolution, e.g. I can very quickly envisage the walk to my place of work, or I can go through the entire trajectory quite slowly, envisaging every little part of the trajectory and how my feet will be moving at any point of time. This variable resolu- tion is interesting for musical purposes, as it shows how it is possible to quickly run through the gestural script of some music, or to go through it very slowly, focusing on every single detail. Also, motor programmes are flexible and allow for alternatives when needed. For instance, I can push open the door with my shoulder or my foot when I am carrying something in my hands, or to take a more musical example, I can play a melody on the piano with my left hand if I have injured my right hand. This possibility of alter- native executions is called motor equivalence, and is important for music cognition as it may be a model for understanding how alternative renderings of similar musical ideas are possible. The idea of motor equivalence could allow for generalizations of musical gestures, e.g. a rapid glissando on a harp would be quite similar to a rapidly played series of tones on a piano, although in detail the actions would be rather differ- ent in these two cases. Furthermore, motor programmes are hierarchical in the sense that they provide overviews of entire actions, i.e. provide executive control, yet at the same time allow for control of detail through ’sub-routines’. With practice and increased level of skill in various tasks, there is a tendency to group actions into larger units, i.e. into chunks, making detail actions automatic and attention to these details unnecessary. These action-based groupings are significant in musical contexts, and it can be shown that rhythmical groupings often follow these principles of action-chunking [22, 23]. Fur- thermore, chunkings of action seem also to have a basis in physiological op- timalization, meaning that with practice, there is usually a tendency towards maxi- mum efficiency or energy conservation, resembling a mass-spring model with alternations between peaks effort and phases of relaxation. Related to chunking of musical sound on the basis of physiological and cognitive optimalization, is the phenomenon of coarticulation, meaning that several otherwise separate actions are fused or embedded into larger scale action-units. The concept of coarticulation was probably first used in linguistics to denote the preparation of the next phoneme by shaping the vocal apparatus in advance, but it has now also been applied to other areas of human movement, such as when my arm is moving towards an object to pick it up, my fingers simultaneously move to anticipate the optimal way

60 Rolf Inge Godøy

of grasping the object [27]. The occurrences of coarticulation in music are numerous,

as in phrasing, articulation, and even in intonation, and naturally occurring coarticula- tion in vocal and instrumental music is in fact that which makes naturally sounding melodic phrases on sampled wind, brass and string instruments practically impossible.

5 Advantages and Applications of Gestural Imagery

A focus on gestural imagery in music will have many advantages. As I have tried to

suggest, the gestural component is a significant component of musical listening in general, and deserves much more attention as a carrier of meaning in studies of musi- cal aesthetics, semiotics, analysis, performance, music education, etc., and it could be tempting to speak of a ’gestural hermeneutics’ in musical thinking, meaning gesture as the mediating element between music and other modes of thought. Furthermore, ges- tural imagery could also shed light on enigmatic issues in music cognition and aes- thetics such as temporality and experience of musical form through the possibility of thinking gestures at various levels of resolution as gesture scripts. Actually, a cultiva- tion of gestural imagery could be seen as a contribution to phenomenological ap- proaches to music theory, related to the project of Pierre Schaeffer [28], with mental gestural tracing or sketching of the shapes of sound objects as an extension of Schaeffers typo-morphological characterizations. Furthermore, gestural images may

also help us understand categorization in music because gestures can be easily recog- nized yet at the same time be variation-tolerant in detail, meaning there can be variant versions of a gesture such as in hitting a drum, yet the gesture retains the general feature of a sudden, ballistic movement. This possibility of categorizations by actions was suggested by Elanor Rosch and co-workers [29], and is similarly reflected in the motor theory of perception [10]. Finally, gestural imagery can also have some more specific, practical applications, and now just to mention some:

Thinking melodic contours as gestures, hence, as holistic entities and not as collec- tions of pitches, and thus enhance our understanding of melodic phenomena.

Thinking musical textures as complex, multi-dimensional patterns of gestures, all seen from the egocentric, first-person perspective, and thus help us grasp the work- ings of different texture types.

Thinking orchestration (and score reading) as similarly complex, multi- dimensional patterns of gestures, however with the addition of the resultant sound from these gestures, generating more lucid predictions of how scores are going to sound.

Providing a better basis for music education in general, and ear-training in particu- lar, by specifically cultivating the musical imagery triggering capabilities of ges- tural imagery.

Helping to understand and master rhythmical grouping, expressivity, and phrasing in musical performance by mental practice of gesture chunks.

Helping to understand the mental and motor processes of improvisation, i.e. how it is possible to simultaneously have images of the immediate past and images of what is to come in the immediate future, by simultaneously thinking different ges- tural scripts.

Gestural Imagery in the Service of Musical Imagery


6 Conclusion and Ideas for Further Research

Altogether, there is reasonable grounds for assuming that there are quite close links between gestural images and images of musical sound, and it is easy to see how these links could be exploited in a number of theoretical and more practical challenges in music. Needless to say, there are a large number of unanswered questions here, and in the overseeable future, research within the following domains could be useful to substantiate our knowledge of the links between gestural and musical imagery:

Compiling relevant data from existing observational (non-invasive) neurophysi- ological studies. There is a fast growing body of research here, and increasingly so, there seems to be much interest in the subjects of motor imagery and mental simulation.

Gathering data on performance gestures, meaning that we need to know more about both the physiological and cognitive constraints and schemata which are at work in performance gestures, as well as developing suitable representations for this data.

Physical model synthesis with incremental changes of the excitatory gestures fol- lowed by listening judgements, i.e. an ’analysis by synthesis’ approach to studying the relationship between gestures and timbral qualities.

Various kinds of experiments on the links between gestural and musical memory, e.g. along the lines of [4], and on the priming and mental practice effects of ges- tural imagery in practical tasks such as orchestration, improvisation and perform- ance. All through this gathering of data from various sources, the unifying principle here should be that of mental simulation, i.e. that of understanding perception, cognition and imagery in music as an active, embodied re-enactment of the gestures we believe belong to musical sound.


1. Godøy, R. I., Jørgensen, H. (eds.): Musical Imagery. Swets and Zeitlinger, Lisse (Holland) (2001) p. ix

2. Husserl, E.: Vorlesungen zur Phänomenologie des inneren Zeitbewusstseins. Max Nimey- er Verlag, Tübingen (1980)

3. For an overview, see Godøy, R. I., Jørgensen, H. (eds.): Musical Imagery. Swets and Zeit- linger, Lisse (Holland) (2001)

4. Mikumo, M.: Encoding Strategies for Pitch Information. Japanese Psychological Mono- graphs No. 27 (The Japanese Psychological Association) (1998)

5. Sudnow, D.: Ways of the Hand. Harvard University Press, Cambridge, Mass. (1978)

6. Kosslyn, S. M.: Image and Brain. The MIT Press, Cambridge, Mass. (1994)

7. Janata, P.: Neurophysiological Mechanisms Underlying Auditory Image Formation in Mu- sic. In: Godøy, R. I., Jørgensen. H. (eds.): Musical Imagery. Swets and Zeitlinger, Lisse (Holland) (2001) pp. 27-42

8. Bregman, A. S.: Auditory Scene Analysis. The MIT Press, Cambridge, Mass. & London


9. McGurk, H., MacDonald, J.: Hearing Lips and Seeing Voices. Nature, 264 (1976) 746-748


Rolf Inge Godøy


Liberman, A. M., Mattingly, I. G.: The Motor Theory of Speech Perception Revised. Cog- nition, 21 (1985) 1-36


Fadiga, L., Craighero, L., Buccino, G., Rizzolatti, G.: Speech Listening Specifically Modulates the Excitability of Tongue Muscles: a TMS Study. European Journal of Neu- roscience, Vol. 15 (2002) pp. 399-402


McNeill, D.: Hand and Mind: What Gestures Reveal About Thought. University of Chi- cago Press, Chicago, IL (1992)


Rizzolatti, G., Arbib, M. A.: Language Within Our Grasp. Trends in Neuroscience Vol. 21 (1998) 188–194


Kita, S.: How representational gestures help speaking. In: McNeill, D. (ed.): Language and Gesture. Cambridge University Press, Cambridge (2000) pp. 162-185


Cadoz, C., Wanderley, M.: Gesture-Music. In: Wanderley, M. M., Battier, M. (eds.):

Trends in Gestural Control of Music (CD-ROM). IRCAM, Paris (2001) pp. 28-65


Berthoz, A.: Le sens du mouvement. Odile Jacob, Paris (1997)


Godøy, R. I.: Imagined Action, Excitation, and Resonance. In: Godøy, R. I., Jørgensen. H. (eds.): Musical Imagery. Swets and Zeitlinger, Lisse (Holland) (2001) pp. 239-252


Freed, D. J.: Auditory Correlates of Perceived Mallet Hardness for a Set of Recorded Per- cussive Sound Events. Journal of the Acoustical Society of America, 87 (1990) 311-322


Handel, S.: Timbre Perception and Auditory Object Identification. In: Moore, B. C. (ed.):

Hearing. Academic Press, San Diego (1995) pp. 425-461


Laban, R.: The Mastery of Movement (fourth edition by Lisa Ullmann). Northcote House Publishers Ltd., Plymouth (1980)


McAngus Todd, N. P., O'Boyle, D. J., and Lee, C. S.: A Sensory-Motor Theory of Rhythm, Time Perception and Beat Induction. Journal of New Music Research,Vol. 28, No. 1 (1999) pp. 5-28


Godøy, R. I.: Knowledge in Music Theory by Shapes of Musical Objects and Sound- Producing Actions. In M. Leman (ed.): Music, Gestalt, and Computing. Springer Verlag, Berlin (1997) pp. 106-110


Godøy, R. I.: Cross-modality and Conceptual Shapes and Spaces in Music Theory. In I. Zannos (ed.): Music and Signs. ASCO Art & Science, Bratislava (1999) pp. 85-98


Jeannerod., M.: Mental Imagery in the Motor Context. Neuropsychologia, Vol. 33, No. 11 (1995) 1419-1432


Dokic, J. & Proust, J. (eds.): Simulation and Knowledge of Action. John Benjamins Pub- lishing Company, Amsterdam/Philadelphia (2002)


Rosenbaum, D. A.: Human Motor Control. Academic Press, Inc, San Diego (1991)


Rosenbaum, D. A., Loukopoulos, L. D., Meulenbroek, R. G. J., Vaughan, J., Engelbrecht, S. E.: Planning Reaches by Evaluating Stored Postures. Psychological Review, Vol 102, No. 1 (1995) 28-47


Schaeffer, P.: Traité des objets musicaux. Éditions du Seuil, Paris (1966)


Rosch, E., Mervis, C. B., Gray, W. D., Johnson, D. M., Boyes-Braem, P.: Basic Objects in Natural Categories. Cognitive Psychology, 8 (1976) 382-436