Timbre Space
as a Musical Control Structure
David L. Wessel
IRCAM
31, Rue St. Merri
75004 Paris, France
Introduction
Researchon musical timbre typically seeks represen-
tations of the perceptualstructureinherentin a set of sounds
that have implicationsfor expressivecontrol over the sounds
in composition and performance.With digital analysis-based
sound synthesis and with experimentson tone quality per-
ception, we can obtain representationsof sounds that suggest
ways to provide low-dimensionalcontrol over their percep-
tually importantproperties.
In this paper, we will describe a system for taking
subjective measures of perceptual contrast between sound
objects and using this data as input to some computer
programs. The computer programs use multidimensional
scalingalgorithmsto generategeometricrepresentationsfrom
the input data. In the timbralspaces that result from the
scaling programs,the various tones can be representedas
points and a good statistical relationshipcan be sought be-
tween the distancesin the space and the contrastjudgments
between the correspondingtones. The spatial representation
is given a psychoacoustical interpretationby relating its
dimensionsto the acousticalpropertiesof the tones. Controls
are then applied directly to these propertiesin synthesis.
The control schemes to be describedare for additivesynthesis
and allow for the manipulationof the evolvingspectralenergy
distribution and various temporal features of the tones.
* Taped examples of various timbralmanipulationschemes
discussedin this article are availablefrom IRCAM;get cassette
order informationfrom IRCAMat the addressabove.
D. Wessel: Timbre Space as a Musical Control Structure
Tests of the control schemeshave been carriedout in musical
contexts.* Particularemphasiswill be givenhere to the con-
structionof melodiclinesin wlhichthe timbreis manipulatedon
a note-to-note basis. Implicationsfor the design of human
control interfacesand of software for real-timedigitalsound
synthesizerswill be discussed.
Musical Timbre
Timbre refers to the "color" or quality of sounds,
and is typically divorced conceptually from pitch and
loudness. Perceptual researchon timbre has demonstrated
that the spectral energy distributionand temporalvariation
in this distributionprovidethe acousticaldeterminantsof our
perceptionof sound quality (See (Grey: 1975) for a thorough
review).Withone notable exception (Erickson: 1975), music
theorists have directed little attention towards the compo-
sitional control of timbre. The primaryemphasishas been
on harmonyand counterpoint.The reasonfor this probably
lies in the fact that most acoustical instrumentsprovide for
very accuratecontrol over pitch but providelittle in the way
of compositionally specifiablemanipulationof timbre. With
the potential of electroacousticinstrumentsthe situation is
quite different. Indeed one can now think in terms of pro-
viding accurate specifications for, by way of example,
sequences of notes that change timbre one after another.
It is this note-to-note changeof timbrethat will concern us
in this paper.
Page 45
Synthesis Technology
Digitaltechnologyofferspowerful,general,andflexible
soundsynthesizers.A numberof suchsynthesismachines
have alreadybeen constructedand are producingmusical
results.Notableexamplesincludethe digitalsynthesisand
processingsystemdesignedby PeterSamson(1977) now in
operationat the StanfordCenterfor ComputerResearchin
MusicandAcoustics;the 256 digitaloscillatorbankdesigned
andconstructedby G. DiGiugno(1976) andthe digitalsyn-
thesizerof HalAllesandDiGiugno(1977), bothin operation
at IRCAM; the Alles(1977 a, b) synthesizerat BellLabs;and
the Dartmouthdigitalsynthesizer(Alonsoet al.: 1975).Some
of thesedevicesofferthe alluringpossibilityof usinga "brute
force"additiveapproachto the synthesisof complexand
musicallyrichtime-variant spectra.In this reportwe will
concentrateon this formof additivesynthesis,becauseof
its generality,andwith the accompanying problemof pro-
vidingdirectcontroloverthe perceptualpropertiesof the
synthesizedsounds.
Beforebeginninga descriptionof a procedurefor
developingcontrolsthat could facilitatethe musicallyex-
pressivemanipulation of complextime-variant spectra,we
willexaminethe natureof boththe acousticalandperceptual
databasesinvolved.Additivesynthesisrequiresa considerable
if not overwhelming amountof explicitinformation,andwe
shallexplorewaysto reducethis quantityof datawithout
sacrificingrichnessin the sonic result.On the otherhand,
the datathatwe canobtainaboutourperceptual experience
of timbrehas quitea differentcharacterfromthe physical
dataof acoustics,andso we shallalsoexaminesuchnotions
as subjectivescales,perceptualdimensions,and structural
representations of subjectivedata.We shall then see the
extent to whichwe can givean accountof the subjective
experienceby examiningthe relationshipbetween the
acousticaland the perceptualdatabases.Weseek a psycho-
acousticsof timbrethathasimplications for timbralcontrol
in musicalcontexts.
Additive Synthesis and Possibilities for
Data Reduction
In the additivemodelfor soundsynthesis,a tone is
represented by the sumof sinusoidalcomponents,each of
whichhas time-varying amplitudeand frequency.Moorer
(1977) givesanexcellentaccountof the details.To synthesize
a sound,one specifiesa numberof softwareor hardware
sinusoidaloscillators,each with its own amplitudeand
frequencycontrolenvelopes.Additivesynthesisof thisform
hastwo importantadvantages. First,it is general,thatis, with
a sufficientnumberof independently controllableoscillators
a verylargeandhighlyvariedclassof signalscanbe generated.
At somesacrificein computational efficiencyandwith an
increasein the quantityof datafor specifyingthe envelopes,
one canmimicFMsynthesis(Chowning:1973)andothernon-
lineartechniques(Arfib: 1977;Beauchamp:1975;Le Brun:
1979;Moorer:1976).Of course,one canproduceeffectsnot
possiblewith thesetechniques.Second,one can analyze
existing sounds and obtain data that can be used to resyn-
thesize exactly the signal that was analyzed. Moorer(1976)
has describedthe applicationof the phasevocoder (Portnoff:
Page 46 Computer Music Journal, Box E, Menlo Park, CA 94025
1976) to the analysis and synthesis of musical sounds. The
phasevocoder is an advanceover methods like the heterodyne
filter (Beauchamp: 1969; Moorer: 1975) in that the analysis
does not have to be pitch synchronousnor do the waveforms
have to contain more or less harmoniccomponents, thus
permittingthe analysis of tones with pitch variationas well
as inharmonictones like those produced by percussionin-
struments.The method also guaranteesthat when the analysis
data is not modified, the originalsignalis recoveredexactly.
Let us say that we want to synthesize,using data from
phase vocoder analysis, a musical instrument timbre with
25 harmonics.In this case 25 amplitude and 25 frequency
envelopes will be required.Storing these functions in full
detail demandsconsiderablememory, and if we are using
a computer-controlleddigitalsynthesizerlike those mentioned
at the start of this article,then transferof the envelopesmay
exceed the bandwidthof the link between the computerand
the synthesizer. Furthermore,if the shapes of the envelope
functions are to be modified, the computationtime required
to rescaleevery point of the functions may easily exceed the
capabilitiesof real-timemanipulation.Clearlysome form of
data reductionis required,if one wants to work in real time.
A particularlyattractive procedure that produces a
significantreduction in the quantity of data involved is to
approximate curvilinearenvelope functions with functions
composed of a seriesof straightline segments(such as those
given in (Moorer:1977)). Such straight-line-segmentapproxi-
mations can be stored in terms of the coordinatesof the
break points of the function, thus greatly reducingmemory
demands. Furthermore,the digital synthesizersconstructed
by Alles, DeGiugno, and Samson provide digital oscillators
that include straight-lineramp controls for both amplitude
and frequency.In workingwith these oscillators,one provides
as data the startingvalue for a ramp,its slope, and a terminat-
ing value or time that indicateswhen a new slope is required.
The actual generationof the values along the specified line
segments is provided within the oscillator itself. In sup-
plying control data from the computer'smemory to these
synthesizers,the break points of the line-segmentapproxi-
mations can be passed directly from the computer to the
synthesizer, thus greatly reducingthe data rate demandson
the interface. Finally, the straight-line-segmentapproximations
make possible rapid modification of the function shapes, as
only the coordinatesof the break points need be modified.
But can we get away with such a drastic data reduction?
If we approximatecurvilinearfunctions with a smallnumber
of connected straightlines, will high audio quality and timbral
richnessbe maintained?
Indications that the straight line segment approxi-
mations would providesatisfactoryresultshave been obtained
for brasstones describedin (Risset and Mathews: 1969) and
by (Beauchamp: 1969). Grey (1975) carriedout a carefully
controlledperceptualdiscriminationexperimentto determine
the extent to which the tones with completely detailed ampli-
tude and frequency functions could be discriminatedfrom
those with line-segmentapproximationsconsisting of from
five to seven segmentsper envelope (also Grey and Moorer:
1977). Grey used 16 different orchestralinstrumenttones,
and in generalhe found the discriminationsto be extremely
difficult. His findingsstronglysuggestthat it is not necessary
to retainthe highly complex temporalmicrostructureof the
amplitude and frequency functions in order to preserve
timbral quality. It would appear that the line-segment
Volume 3 Number 2
 approximationscan be made with little harm. Besides the
resulting data reduction, an important advantageof such
approximationsis that the resultingtones have more clearly
defined acoustical properties. This will prove especially
important when we wish to determine those physical pro-
perties that are especially important for perception.
Representation of Timbre Dissimilarities
In the next sections we will first describe a method
for characterizingthe structureof the relationshipsinherent
in a set of sounds differingin timbre.We will then show how
such representationsof the timbres can be related to the
underlyingacoustical propertiesof the tones. We will also
show that the representationscan be used to compose timbral
patterns with perceptual properties predictable from the
structureof the representation.Finally it will be arguedthat
when this is accomplishedwe have in some sense designeda
systematic control scheme for the perceptually important
acoustic propertiesof the sounds.
From a quantitativepoint of view, data from subjective
judgments has a peculiar if not uncertain status (Luce:
1972). The notion of a unit of measurementsuch as the
decibel or hertz is difficult if not impossibleto establishfor
subjectivescales. We can of course choose units for subjective
scales like "sones" or "mels" as Stevens (1959) has done,
but the so-called unit derived in one experimentalcontext
fails to remain fixed in other contexts. In fact, the "sone,"
the unit for subjectiveloudness, is not invariantacross the
two ears of individualswith normal hearing(Levelt, et al.:
1972). Such units are useful in that they providea common
languagefor discussingthe auditoryabilities of a population
of listeners, but they cannot justifiably be treated with the
algebraof dimensionalanalysis that underliesmeasurement
in the physical sciences. I think it right to be pessimistic
about the possibility of subjectivescales being elevatedto the
same form of measurementas physical measurement.But
subjectivejudgments,if collected over a sufficient number
of objects, in this case sounds, can have a representable
structure,and this structurecan in turn be relatedto various
acousticalparameters.
Perceptualjudgments tend to be relative in nature.
With few exceptions, we tend to judge an object in terms
of the relationshipsit has with other objects. Relational
judgments are of great interest in music, because music
involvespatternscomposed of a varietyof sounds, and it is the
relational structure within and between the patterns that
is of primaryimportance. Judgmentsof the extent of per-
ceptual similarityor dissimilaritybetween two sounds can
be made in a very intuitive fashion. One can say that sound
A is more similarto sound B than to sound C without having
to name or otherwise identify explicitly the attributes that
were involved in the judgment. Researchgroups at IRCAM,
at MichiganState University,and at the StanfordCenterfor
Researchin Musicand Acoustics have been usingperceptual
dissimilarityjudgmentsin a varietyof musicaland otherwise
audio-relatedcontexts. One of the generaltechniques is to
representthe perceptualdissimilaritiesas distancesin a spatial
configuration.One begins with a set of dissimilarityjudgments
typically taken for all pairsof sounds that can be formed from
the set. This matrix of dissimilaritiesis then processedby one
of a variety of multidimensionalscaling programssuch as
KYST (Kruskal: 1964 a,b ). The multidimensionalscaling
D. Wessel: Timbre Space as a Musical Control Structure
programsproduce an n-dimensionalspatial arrangementof
points that representthe varioussound objects. The programs
operate to maximize a goodness-of-fitfunction relatingthe
distancesbetween the points to the correspondingdissimilarity
ratingsbetween the sounds.
Perhapsat this point it would be best to show how the
multidimensionalscalingexperimentswere carriedout. At
IRCAMwe have recently developed a set of programsthat
greatly facilitate the design, execution, and interpretation
of such experiments.In the following example we use the
same set of sounds used in both (Grey : 1975) and (Grey and
Gordon: 1978) and presented in the series "Lexicon of
AnalyzedTones" (Moorer,et al.: 1977, 1978). This set con-
sists of 16 synthetic orchestralinstrumenttimbresand a group
of eight hybrid instrumenttimbres producedby exchanging
spectral envelopes between members of the original set.
The goal of our experiment will be to provide a represen-
tation of these 24 timbres as points in a Euclidianspace as
well as an interpretationof this representationin terms of
the acousticalpropertiesof the tones.
General Method
The procedurethat providesan interpretedrepresenta-
tion of the soundsinvolvesthe following five steps: 1. Selec-
tion of materialsfor study; 2. Collection of the dissimilarity
judgments; 3. Representationof the dissimilarityjudgments
with spatial and other schemes such as clusters and graphs;
4. Psychoacousticalinterpretationof the structure; 5. Veri-
fication of the interpretationin musicalsituations.
Selecting the Materials for Study
In preparingthe sounds that we wish to represent,
attention must be paid to (A) the numberof sound elements,
(B) the problem of equalizingthe sounds with respect to
parameterswe wish to ignore, and (C) the rangeof variation
within the set of sounds.
A. The Numberof Sound Elements
To obtain a meaningfulrepresentation,a certainminimal
number of sound elements is required,in order that the
dissimilarltiesimpose a sufficient amount of constraintfor
fixing accuratelythe locations of the points in a space. Some
of the multidimensionalscalingprograms,like KYST,operate
with only qualitativeor "ordinal"constraintson the distances
in the space. These algorithimsseek arrangementsof the points
such that the rank order of the interpoint distancesin the
space matches, in terms of a well-defined goodness-of-fit
measure,the rank order of the dissimilaritiesbetween the
correspondingstimuli. If we begin with interpoint distances
from a known configurationof points, the programsprovidea
very accuraterecovery of the positions of the points even
when the distancesare subjectedto radicalmonotonic trans-
formations or perturbationsdue to random error(Shepard:
1966). It is hard to set down a precise rule of thumb for
determiningthe minimumnumber of elements to use. The
choice dependson the numberof dimensionsand the distribu-
tion of the points in the space, an issue to which we shall
return below. In most psychologicalresearchusing multi-
dimensionalscaling, 10 points have seemed sufficient for
two dimensionsand 15 for three. In our recent researchwe
have tended to encouragethe use of between 20 to 30 points
for spacesof two and three dimensions,respectively.
Page 47
B. Equalizingthe Tones with Respect to
ExtraneousParameters
If possible, the tones should be equalizedwith respect
to the propertiesthat are not to influence the judgments.
When studying timbre, the usual procedureis to equalize
the pitch, subjectiveduration,loudness, and room information
aspects of the tones. On the other hand, if we are studying
room information (that is, reverberationstructure),then we
probablywant to use a standardsource and manipulateonly
the reverberationparameters.Attention should also be paid
to just what is being equalized. If we are equalizingwith
respect to loudness, for example, and the tones in the set have
differentspectralshapesandattackrates,then simply matching
sound pressurelevel (in terms of decibels) will not provide
the appropriateequalization. In this case we are without a
satisfactory model for the perception of the loudness of
complex time-variantspectra (Moorer: 1975) and we must
resort to making empiricalmatches. Grey (1975) provides
a good example of such subjective matching proceduresfor
pitch, duration,and loudness.
C. Controllingthe Rangeof VariationWithinthe Set
The rangeof variationin the timbresof the sounds will
certainlydiffer from one type of study to another. In some
instanceswe will want to investigatea timbraldomainhavinga
broad range of variation including perhaps inharmonic
percussionsounds as well as sounds with more or less har-
monic components. In other situations more restrictionsare
imposed on the range of variation,as in, for example, the
study to be describedin this paperwhere we use only sounds
derived from standard orchestralinstruments played in a
conventionalmanner.For even more refinedinvestigationsof
timbralnuance one might use a very limited rangeof sounds.
Once a generalrangeof variationhas been determined,
some attention must be paid to the homogeneity of variation
within the set. Considerthe following example. If we choose
eight distinctively percussivetimbres and eight distinctively
non-percussion timbres but provide no linking elements be-
tween the two domains,then it is likely that all the subjective
dissimilaritiesthat are made for pairsthat cut acrossthe two
classesof sounds are largerthan all the intra-classdissimilari-
ties. In situationslike this, some of the multidimensionalscal-
ing programsgive what are called degeneratesolutions and the
intra-class structureis not fully displayed. Shepard (1974)
providesa detailed discussionof this problemand prospects
for its solution. In addition, in such a situation I find that in
makingthe judgmentsI have a difficult time concentratingon
the relatively subtle differences for pairs within a class in
the context of the much largerdissimilaritiesfor the pairs
spanningthe two categories.Unfortunately, since we will
most often deal with timbraldomainsabout which we have
little knowledge, clear rules for the preliminaryselection
of the variationin the materialare hard to set down. Selection
proceduresdepend ultimately on our specific interests and
desiresfor control.
To facilitate the preliminaryscreeningand selection of
the sounds, we have developedan interactiverandomaccess
audio playback programon IRCAM'sDEC-10 computer.
This programis called KEYS and was written by Bennett
Smith (Wesseland Smith: 1977). One suppliesKEYSwith a
list of the sounds written out as a list of the names of the files
Page 48 Computer Music Journal, Box E, Menlo Park, CA 94025
in which each sound element is stored. Each sound file in the
list is then related to a characterthat can be typed on the
terminalkeyboard. The relationshipsbetween the characters
and the sounds are listed on the computer terminal'sCRT
display. When a characteris typed the correspondingsound
file is played throughthe digital-to-analogconvertersand the
playing action is indicated by an increasein the brightness
of the character-filename entry in the table displayedon the
CRT. This programpermitsrapidauditorycomparisonsamong
a largenumberof different sounds. Currentlythe limit is 100
files of arbitrarylength.
Collecting the Timbre Dissimilarity Judgements
Thoughthere exist a varietyof ways to collect percep-
tual dissimilarities,we have found simple ratingsof the extent
of dissimilarityto be the most efficient and least tedious way
to make the judgments.At IRCAMwe have been using direct
estimatesof the dissimilaritybetween two tones. Ourlistening
judge, using a programwritten by Bennett Smith called
ESQUISSE,sits before a CRT displayterminaland an audio
system fed by the computer's digital-to-analog converters.
The two soundsin a pairare relatedto the terminalkeys "m"
and "n", allowingthe listener to play the sounds at will. After
listening,the judge enters a ratingwith one of the keys "0"
through"9" on the terminal.Immediatelyafter the judgment
is entered, the next pair of tones is ready to be played from
the keyboard. The sequencing of the judgment trials is
random,and all of the n (n- 1)/2 pairsare used, wheren
is the numberof sounds understudy. After all the pairshave
been judged, the random sequence is unscrambledand a
matrix of dissimilaritiesis formed.
The data collection programincludes what we call a
"coffee-break" feature that allows the judgment session to
be interruptedeither by machinefailureor human fatigue.
The listener can then returnto the experimentat a later
time and continue from the point in the sequencewhere the
interruptionoccurred.Withthis program,one is able to listen
to the sounds rapidly and freely, and the judgments can be
enteredwith ease.
A Two-dimensional Representation of 24 Orchestral
Instrument Timbres
I servedas a judge using the dissimilaritydata collection
programon a set of 24 orchestralinstrumenttimbresthat were
obtained from John Grey. These soundswere synthesized
usingline segmentenvelopesand were equalizedsubjectively
for pitch, loudness, and duration.A two-dimensionalrepre-
sentationof the soundsprovidedby the KYSTprogramis shown
in Figure 1, along with an interpretationof the dimensionsof
this space. The verticalaxis is relatedto the spectralenergy
distributionof the tones, and the horizontal,to the natureof
the onset transient.The sounds at the top of plot are bright
in character,and as one moves towardsthe bottom the timbres
become progressivelymore mellow. In a numberof studies on
timbrespaces(Wedenand Goude: 1972; Wessel:1973; Grey:
1975; Ehresmanand Wessel: 1978; Grey and Gordon: 1978;
Wesseland Grey: 1978) this dimensionrelatedto the spectral
energy distributionhas appeared.A consistent, quantitative,
acousticalinterpretationhas also been providedby calculating
an excitation patternfor the spectrumprovidedby Zwicker's
model for loudness (Zwicker and Scharf: 1965). This trans-
Volume 3 Number 2
formationon the acousticalspectrumcompensatesfor certain
propertiesof the auditory system like criticalbands and the
asymmetricspreadof maskingfrom low to high frequencies.
The centroid or mean of this compensatedspectralenergy
distributionis then calculatedand correlatedwith projections
of the points on the axis assumedto be relatedto brightness.
In all of the studies these correlationshave been very high.
mTM
UTPZ C02
"BRIGHTER"
bol
tBCZ
VTMZ
dEH fc,
"MORE BITE"
Os2Z
IITP j S2Z
w
XS2
rojz
Wslz
nsl
eBN kFL ix2
JX3
qFHZ
aFH P53
SBNZ
FIGURE1. Two-dimensionaltimbrespace representationof
24 instrument-likesounds obtained from Grey. The space was
producedby the KYSTmultidimensionalscalingprogramfrom
dissimilarityjudgementsmadeby Wessel.The lower-case letters
identify the sounds as recordedon a cassette-tape preparedto
accompanythis paperand distributedby IRCAM.The upper-
case subscriptsat each point identify the tones as reportedin
(Grey: 1975, 1977), (Greyand Gordon: 1978) and (Gordon
and Grey: 1978). The originalanalyzed tones upon which
these synthesized versions are based are being presented
in the ComputerMusicJournalseries "Lexicon of Analyzed
Tones."
Abbreviationsfor stimuluspoints: 01, 02 = oboes, FH =
French horn, BN = bassoon, C1 = E-flat clarinet, C2 = bass
clarinet, FL = flute, X1, X2, X3 = saxophones, TP = trumpet,
EH = English horn, S1 = cello played sul ponticello, S2 =
cello played normally, S3 = cello played muted sul tasto,
FHZ = modified FH with spectralenvelope, BNZ =
modified BN with FH spectralenvelope, S1Z = modified S1
with S2 spectralenvelope, S2Z = modified S2 with S1 spectral
envelope, TMZ = modified TM with TP spectral envelope,
BCZ = modified C2 with 01 spectral envelope, 01Z =
modified 01 with C2 spectralenvelope.
D. Wessel: Timbre Space as a Musical Control Structure
The horizontal dimension is related to the quality of
the "bite" in the attack. Possibilitiesfor quantificationof this
dimensionare discussedin the next section on timbrepatterns
formed in the space.
Predictions About Timbre Patterns
To a largeextent, music consists of syntactic patterns.
It is the natureof the relationshipsamongthe elements of the
patternsthat is of primaryimportancein their perception.In
the next seriesof examplesI wouldlike to show that when note -
to-note timbralchangesare organizedin terms of the timbre
space which we just examined, then predictableperceptual
organizationsof timbre patternscan be obtained.
First we will examine some auditoryeffects that we can
relateto the dimensionsof the spaceandthe distancesspanned.
In the following patternsthe sequenceof notes will alternate
between two differing timbres, but otherwise the pitch se-
quence and rhythmictimingwill remainfixed. The pitch se-
quence is the simple, repeating,three-note ascendingline
shown in Figure2. The alternatingtimbresequenceis shown
by the alternatingnotes markedrespectivelywith "0" and
hX "X". When the timbral distance between the adjacentnotes
is small, the repeatingascendingpitch lines dominate our
perception.However,when the timbre differenceis enlarged
along the "spectralenergy distribution"axis, the perceptual
organizationof the patternis radicallyaltered.The line now
splits at the wide timbralintervalsand for many listenerstwo
g
interwovendescendinglines areformed,eachwith its own tim-
bralidentity. This type of effect is called "melodicfission" or
"auditorystreamsegregation"in the psychoacousticliterature
(Bregmanand Campbell: 1971; Van Noorden: 1975) and
is a consequence of the large spectral energy distribution
between the alternatingtimbres.
FIGURE2. Ascending pitch patterns in "three" with two
alternatingtimbres ("0" and "X"). If the timbraldifference
between adjacent notes is large,then one tends to perceive
interleaveddescendinglines formed by the notes of the same
timbraltype.
A different effect is obtainedby movingalong the di-
mension we interpretedas relatingto the onset characteris-
tics of the sounds. Whenthis is done we obtain a perceptu-
ally irregularrhythm even though the acousticalonset times of
the notes are the same. This observationhas some important
implicationsfor the control of sound in synthesis.Whenwe
alter the propertiesof the attack of the tone, we are also
likely to influence the temporal location of the perceived
onset of the tone. This lack of accordbetween physicalonset
time and subjectiveonset time has been observedwith speech
sounds by Morton,et al. (1975). Morton'sexperimentalpro-
cedure offers the possibility of determiningthe relative per-
ceived onset times for a set of notes. The procedureuses a
simpleABAB... AB alternatingsequence similarto those
just described.The listener adjusts the shift in onset for all
the B's in the sequence until the sequence is perceivedas
regular,and the temporaldisplacementin the physicalonset
Page 49
is then noted. Perhapswith the applicationof such a method
to musicaltimbresand with the employment of a good model
of auditory temporal integrationof complex time-varying
spectra,we will be able to predictmore preciselywhere the
perceivedonsets of tones with differing spectralevolutions
will occur. Nevertheless, both the fine tuning of rhythm
in music and psychoacoustic researchwill benefit greatly if
the control software of our synthesissystems allows easy and
flexible adjustment of physical onset times in complex
musicalcontexts.
Withthe previousexampleswe have furtherverifiedthe
interpretationof the timbre space and have demonstrated
that to some extent the propertiesof the space retain their
validity in richer musical situations. In the next example
involvingtimbralanalogiesI hope to demonstratethat other
propertiesof the geometry of the timbre space allow us to
make predictionsabout patternperceptionas well.
!

Timbral Analogies

Composers frequently make transpositionsof pitch

patterns.It seemednaturalto ask if transpositionsof timbral
sequenceswork as well. To get some preliminaryindications
regardingthis possibility David Ehresmanand I (Ehresman
and Wessel: 1978) tested a parallelogrammodel of analogies
developed by Rumelhartand Abramson(1973). The basic
idea is illustratedin Figure3. If we make a two-note timbral
pattern,the sequenceA,B in Figure3, and wish to make an
analogous(or, for our purposes,transposed)sequencebegin-
ning on timbreC, then we choose the note D that best com-
pletes a parallelogramin the space.
To test this idea we presentedlisteners with four dif-
ferent solutions to timbral analogiesof the form A -+ B
FIGURE3. Parallelogrammodel of timbre analogies.A + B
as C -> Di. They were asked to orderthe alternativeanalogies
is a givenchange in timbre;C + D is a desiredtimbralanalogy,
indicatingthe best formed, the next best formed, and so forth. with C given. D is the ideal solution point. D1, D2, D3, and
1,r

The idea was to check if the listeners'rankingof the goodness

of the analogy would be inversely related to the distance D4 are the actual solutions offered to the listeners.
between the variousalternativesolutions,Di's, and the ideal
solution point specified by the parallelogramin the timbre
space. In constructingthe analogy problemswe chose the
alternativesso they would fall at graded distances from the Table 1. Rankorderdata averagedover nine listenersand all
ideal solution point. We selected 40 different analogy prob- 40 analogies. Cf. Figure3.
lems from a timbre space similarto the one just described.
Our laboratorycomputer system allowed us to synthesize
and store the tones and then to automate and analyze the Listener-AssignedRank
Rank Distanceof (J)
experiment. One important feature of the system was that the Alternativefrom I
the listenershad essentiallyrandomaccess to the four alter-
the IdealSolution 1 2 3 4
native formations of an analogy problem and were able to
make rapidauditorycomparisonsamong them.
1 .422 .303 .156 .119
Table 1 shows the results of this experiment for nine 2 .322 .283 .217 .178
listeners who each ordered the solutions for 40 different (/) 3 .169 .267 .358 .206
analogyproblems.The entriesin the table indicate the propor- 4 .086 .147 .269 .497
tion of times, averagedover listenersand analogies,for which
the Ith closest alternativeto the ideal analogy point was
ranked as the Jth best solution, where I is the row index
and J is the column index. Column 1 of this table shows The idea of transposingtimbralpatternssuggestsanother
that the predictionwas indeed fulfilled. In fact, the distance procedurefor representingimportantperceptualrelationships.
between an alternativeand ideal analogy point predicts not Usinga technique called simultaneouslinearequation scaling
only the best solution but the rankorderingof most of the (Carrolland Chang:1972) one can derivedistanceestimates
alternatives.Though more researchneeds to be done, the directly from the analogy quality judgments and then use
notion of transposinga sequence of timbres by forming the standardmultidimensionalscaling algorithms on these
anothersequence geometricallyparallelto it in timbrespace distances. Though we have yet to try this scheme, it seems
thus appearsto be a reasonableand musically viable idea. promising.
Page 50 Computer Music Journal, Box E, Menlo Park, CA 94025 Volume 3 Number 2
 Designing Control Systems from the Perceptual
Representations
The timbre space representation suggests relatively
straightforwardschemes for controlling timbre. The basic
idea is that by specifying coordinates in a particulartimbre
space, one could hear the timbre representedby those co-
ordinates. If these coordinates should fall between existing
tones in the space, we would want this interpolated timbre
to relate to the other sounds in a mannerconsistent with the
structureof the space. Evidencethat such interpolatedsounds
are consistent with the geometry of the space has been
providedby Grey (1975). Grey used selected pairsof sounds
from his timbrespace and formed sequencesof interpolated
sounds by modifying the envelope break points of the two
soundswith a simple linearinterpolationscheme. These inter-
polated sequencesof sounds were perceptuallysmooth and did
not exhibit abruptchangesin timbre.Membersof the original
set of sounds and the newly createdinterpolatedtimbreswere
then used in a dissimilarityjudgmentexperimentto determine
a new timbre space. This new spacehad essentially the same
structureas the original space with the interpolated tones
appropriatelylocated between the sounds used to construct
them. It would appear from these results that the regions
between the existing sounds in the space can be filled out,
and that smooth, finely graded timbral transitionscan be
formed.
The most naturalway to move about in the timbral
space would be to attach the handles of control directly to
the dimensionsof the space. I examinedsuch a control scheme
in a real-time context (Wessel: 1976). A two-dimensional
timbrespace was representedon the graphicsterminalof the
computer that controlled the DiGiugno oscillator bank at
IRCAM.One dimensionof this space was used to manipulate
the shape of the spectralenergy distribution.This was accom-
plished by appropriatelyscaling the line segment amplitude
envelopesaccordingto a shapingfunction. The other axis of
the space was used to control either the attack rate or the
extent of synchronicityamongthe variouscomponents. Over-
all, the timbraltrajectoriesin these spaceswere smooth and
otherwise perceptually well-behaved. To facilitate more
complex forms of control, an efficient computerlanguagefor
dealingwith envelopesis needed. Grey and his colleaguesat
Stanfordhave developeda languagefor this purpose(Kahrs:
1977), and our group at IRCAMis making a similareffort.
The basic idea behind such a languageis to providea flexible
control structure that permits specification, sequencing,and
combination of variousproceduresthat creafe and modify
envelopes. These procedureswould include operationslike
stretching or shorteningduration, changingpitch, reshaping
spectrum, synchronizingor desynchronizingspectral compo-
nents, and so forth. Withsuch a languageit will be possible
to tie the operationson the envelope collections directly to
the properties of the perceptual representationsof the
material.
Acknowledgments
I would like to thank John Grey and John Gordon for
their tones and comments, and GeraldBennett, Andy Moorer,
Wayne Slawson, and John Strawn for their comments and
criticalreadingof the manuscript.
D. Wessel: Timbre Space as a Musical Control Structure
References
Alles, H. G. (1977a) "A Portable Digital Sound Synthesis
System." ComputerMusic Journal, Vol. 1, No. 4,
pp. 5-6.
Alles, H. G. (1977b) "A ModularApproach to Building
Large Digital Synthesis Systems." ComputerMusic
Journal,Vol. 1, No. 4, pp. 10-13.
Alles, H. G., and DiGiugno, G. (1977c) "A Onr Card,64-
ChannelDigitalSynthesizer."Computer usic Journal,
Vol. 1, No. 4, pp. 7-9.
Alonso, S., Appleton, J. H., Jones, C. (1975) "A Special-
Purpose Digital System for the Instruction, Compo-
sition, and Performanceof Music."Proc. of the 1975
Conferenceon Computersin UndergraduateCurricula,
6, 17-22.
Arfib, D. (1977) "Digital Synthesis of Complex Spectraby
Means of Non-Linear Distortion of Sine Waves and
Amplitude Modulation."Paper presented at the 1977
InternationalComputerMusic Conference,Center for
MusicExperiment,Universityof Californiaat San Diego,
La Jolla.
Beauchamp,J. W. (1969) "A Computer System for Time-
VariantHarmonicAnalysis and Synthesis of Musical
Tones," in Music by Computers,edited by H. von
Foerster and J. W. Beauchamp.John Wiley and Sons,
Inc., New York.
Beauchamp,J. (1975) "Analysisand Synthesis of Comet
Tones Using Nonlinear InterharmonicRelationships."
Journal of the Audio EngineeringSociety, Vol. 23,
pp.778-795.
Bregman,A. S., and Campbell,J. (1971) "PrimaryAuditory
Stream Segregationan(i Perception of Orderin Rapid
Sequencesof Tones."Journalof ExperimentalPsychol-
ogy, Vol. 89, pp. 244-249.
Carroll,J. D., and Chang,J. J. (1972) "Simules:Simultaneous
Linear Equation Scaling."Proceedings, 80th Annual
Convention of American Psychological Association.
Chowning,J. M. (1973) "The Synthesis of Complex Audio
Spectra by Meansof Frequency Modulation."Journal
of the Audio EngineeringSociety, Vol. 21, pp. 526-
534. Reprintedin ComputerMusic Journal, Vol. 1,
No. 2, pp. 46-54, 1977.
DiGiugno,G. (1976) "A 256 DigitalOscillatorBank."Paper
presented at the 1976 InternationalComputerMusic
Conference, MassachusettsInstitute of Technology,
Cambridge.
Ehresman,D., and Wessel, D. L. (1978) "Perception of
TimbralAnalogies."IRCAMTechnicalReport No. 13.
Erickson,R. (1975) Sound Structurein Music. Universityof
CaliforniaPress,Berkeleyand Los Angeles.
Grey, J. M. (1975) Explorationof MusicalTimbre.Stanford
Univ. Dept. of MusicTech. Rep. STAN-M-2.
Grey, J. M., and Moorer,J. A. (1977) "PerceptualEvaluations
of Synthesized MusicalInstrumentTones." Journal of
the Acoustical Society of America, Vol. 62, pp. 454-
62,1977.
Grey, J. M. (1977) "MultidimensionalPerceptualScalingof
MusicalTimbre."Journalof the Acoustical Society of
America,Vol. 61, pp. 1270-1277, 1977.
Gordon, J., and Grey, J. M. (1978) "PerceptualEffects of
SpectralModificationson OrchestralInstrumentTones."
ComputerMusic Journal, Vol. 2, No. 1, pp. 24-31.
Page 51
Kahrs,M. (1977) "A ComputerLanguagefor Psychoacoustic
Study and MusicalControl of Timbre."Paperpresented
at the 1977 InternationalComputerMusicConference.
Centerfor MusicExperiment,Universityof Californiaat
San Diego, La Jolla.
Kruskal,J.B. (1964a) "MultidimensionalScalingby Optimiz-
ing Goodness of Fit to a Nonmetric Hypothesis."
Psychometrika,Vol. 29, pp. 1-27.
Kruskal,J.B. (1964b) "NonmetricMultidimensionalScaling:
A NumericalMethod." Psychometrika, Vol. 29, pp.
115-129.
Le Brun,M. (1979) "Waveshaping Synthesis,"Journalof the
Audio EngineeringSociety, Vol. 27, No. 4, pp. 250-
266.
Levelt, W.J. M., Riemersma,J.B., and Bunt, A.A. (1972)
"BinauralAdditivity of Loudness,"British Journal of
Mathematicaland StatisticalPsychology, Vol. 25, pp.
51-68.
Luce, R. D. (1972) "WhatSort of Measurementis Psycho-
physicalMeasurement?"AmericanPsychologist, Febru-
ary, pp. 96-106.
Moorer,J.A. (1975) "On the Loudness of Complex, Time-
Variant Tones." Stanford University, Department of
Music,Tech Report STAN-M-4.
Moorer, J. A.(1976) "The Synthesis of Complex Audio
Spectra by Meansof Discrete Summation Formulae."
Journal of the Audio EngineeringSociety, Vol. 24,
pp.717-727.
Moorer,J. A. (1976) "The Use of the PhaseVocoder in Com-
PuterMusicApplications."Presentedat the 55th Con-
vention of the Audio EngineeringSociety; availableas
PreprintNo. 1146 (El).
Moorer,J. A. (1977) "SignalProcessingAspects of Computer
Music- A Survey."Proceedingof the IEEE, Vol. 65,
No. 8, pp. 1108-1137, and ComputerMusicJournal,
Vol. 1,No. , pp. 4-38.
Moorer, J.A., Grey, J.M., and Snell, J.M., (1977) "Lexicon
of AnalyzedTones (PartI : Violin Tone)," Computer
MusicJournal,Vol. 1, No. 2, pp. 39-45.
Moorer, J.A., Grey, J.M., and Strawn, J. (1977) "Lexicon
of AnalyzedTones (Part II :Clarinetand Oboe Tones),"
ComputerMusic Journal, Vo. 1, No. 3, pp. 12-29.
Moorer, J.A., Grey, J.M., and Strawn, J. (1978) "Lexicon
of Analyzed Tones (Part III: the Trumpet),"Computer
MusicJournal,Vol. 2, No. 2, pp. 23-31.
Morton, J., Marcus,S., and Frankish,C. (1976) "Perceptual
Centers (P-Centers)"Psychological Review, Vol. 83,
No. 5, pp. 405-408.
Page 52 Computer Music Journal, Box E, Menlo Park, CA 94025
Portnoff, M.R. (1976) "Implementationof the DigitalPhase
Vocoder Using the Fast Fourier Transform."IEEE
Transactionson Acoustics, Speech, and Signal Proces-
sing, Vol. ASSP-24, pp. 243-248.
Risset, J. C. and Mathews, M.V. (1969) "Analysisof Musical
InstrumentTones."Physics Today, Vol. 22, pp. 23-30.
Rumelhart, D.E., and Abrahamson,A.A. (1973) "Toward
a Theory of AnalogicalReasoning."CognitivePsycho-
ogy, Vol. 5, pp. 1-28.
Samson,Peter (1977) "SystemsConceptsDigitalSynthesizer
Specifications."Availablefrom Systems Concepts,520
ThirdStreet, San Francisco,California94107.
Shepard,R. N. (1966) "MetricStructuresin OrdinalData."
Journal of MathematicalPsychology, Vol. 3, pp. 287-
315.
Shepard, R. N. (1972) "PsychologicalRepresentationof
Speech Sounds," in Human Communication,edited
by E.E. Davisand P.B. Denes. Mc Graw-Hill,New York.
Shepard,R.N. (1974) "Representationsof Structurein
Similarity Data: Problems and Prospects." Psycho-
metrika39, 373-421.
van Noorden, L. (1975) "TemporalCoherencein the Percep-
tion of Tone Sequences." Instituut voor Perceptie
Onderzoek,Eindhoven,Holland.
Wedin,L. and Goude, G. (1972) "DimensionAnalysisof the
Perception of Instrumental Timbre." Scandinavian
Journ.Psych.,Vol. 13, pp. 228-240.
Wessel,D.L. (1973) "Psychoacousticsand Music: A Report
from MichiganState University."PAGE:Bulletin of the
ComputersArts Society, Vol. 30.
Wessel,D.L. and Grey, J.M. (1978) "ConceptualStructures
for the Representationof MusicalMaterial."IRCAM
TechnicalReport #14.
Wessel, D.L. and Smith, B. (1977) "PsychoacousticAids
for the Musician'sExplorationof New Material."Paper
presentedat the 1977 InternationalComputerMusic
Conference,Center for Music Experiment,University
of Californiaat San Diego, La Jolla.
Wessel,D.L. (1976) "PerceptuallyBased Controls for Ad-
ditive Synthesis."Paperpresentedat the 1976 Interna-
tional Computer Music Conference, Massachusetts
Institute of Technology.
Zwicker, E. and Sharf, B. (1965) "A Model of Loudness
Summation."Psych. Review, Vol. 72(1), pp. 3-26.
Volume 3 Number 2