Using a Theoretical Multimedia Taxonomy

Framework
RACHELLE S. HELLER, C. DIANNE MARTIN, NUZI HANEEF, and SONJA
GIEVSKA-KRLIU
The George Washington University

Multimedia (MM) is a polysemous term, a term with many definitions, and in this case, many
roots. In this paper, multimedia is defined as the seamless integration of two or more media.
Each ancestor brings another requirement, muddying the field and making it difficult to work
through. A multimedia taxonomy based on a previous media taxonomy is proposed to help
organize the discipline. The taxonomy helps to classify the space called multimedia and to
draw attention to difficult issues. The paper outlines the forms contributing to multimedia
text, sound, graphics, and motionand aligns them with probable formats elaboration,
representation, and abstractionand sets them within a contextaudience, discipline, interactivity, quality, usefulness, and aesthetics. The contexts are more clearly defined in two
areas: interactivity and the information basis for a discipline. Examples are presented
describing the use of the taxonomy in the design and evaluation of student projects in a
computer science-based multimedia course.
Categories and Subject Descriptors: H.5.0 [Information Interfaces and Presentation]:
General
General Terms: Design, Theory
Additional Key Words and Phrases: Computer uses in education, evaluation, multimedia

1. INTRODUCTION: WHAT IS MULTIMEDIA?

Multimedia (MM) is a polysemous terma term with many definitions, and
in this case, many roots. In this paper, multimedia is defined as the
seamless integration of two or more media. If two or more media are
attached to each other, but not in a seamless way, we refer to them as
multiple media. Depending on its roots, multimedia takes on different
characteristics. If the root is education, then the focus of the multimedia
discussion is the delivery of education, and the media are analyzed in terms
of their effectiveness in delivering information. If the root is human
computer interaction, the focus is the interface and the navigation afforded

Authors address: The George Washington University, Washington, DC 20052.

Permission to make digital / hard copy of part or all of this work for personal or classroom use
is granted without fee provided that the copies are not made or distributed for profit or
commercial advantage, the copyright notice, the title of the publication, and its date appear,
and notice is given that copying is by permission of the ACM, Inc. To copy otherwise, to
republish, to post on servers, or to redistribute to lists, requires prior specific permission
and / or a fee.
2001 ACM 1531-4278/01/0300 0004 $5.00
ACM Journal of Educational Resources in Computing, Vol. 1, No. 1, Spring 2001, Article #4, 22 pages.

R. S. Heller et al.

by the use of media. Similarly, if the root is computer graphics, then all of
the issues inherent in computer graphicsissues such as frame rate,
refresh, and lightingare applied to multimedia. Each ancestor brings
another requirement, introducing more complexity in the field and making
it difficult to work through. Every designer wants to build a high-quality
system, but actually doing so and evaluating it is a difficult task given the
polysemous context.
We propose a multimedia taxonomy to help organize the discipline on the
basis of a previous media taxonomy [Heller and Martin 1995]. The taxonomy helps to classify the space called multimedia and helps to draw
attention to difficult issues. While multimedia is an emerging field, the fact
that we can provide a taxonomy indicates that it has reached some level of
stability and maturity. Due to our outlook, we can see the directions
imposed on the field by its many roots. As we classify the field, we also
move forward to a deeper analysis.
2. THE TAXONOMY
Multimedia taxonomy can be visualized as a three-dimensional matrix
(Figure 1). Media type pertains to the various media involved: text, sound,
images, motion, and multimedia. Media expression, i.e., elaboration, representation, and abstraction, refers to the level of abstraction using the
media. Context, the third dimension, reveals the various roots of multimedia: disciplines, interactivity, audience, aesthetics, quality and usefulness.
Media type is arranged as a series of individual media of increasing
complexity (i.e., storage). The various media should be clear. Text is the
presentation of information using an alphabetical symbol system. This
includes prose in various languages as well as presentations in such forms
as mathematics or other symbol systems. Sound includes spoken words as
well as generated tones forming music or other audible information. Images
include photos and hand-drawn items, while motion can be motion pictures
or animation. Multimedia, as defined above, is the combination of any two
or more of these.
The media expression categories require some explanation. These move
from concrete to abstract. Elaboration is where no information is edited
out. So that if we are examining a textual presentation of a speech, the
elaboration presentation would include the fully rendered speech. An image
would be the complete image of something like a room with all the details.
Elaboration presentations are the most concrete and require the least effort
on the part of the user in decoding the symbol set. This does not mean that
the level of information is simple, but that the medium is not hiding or
subsuming any information. Representational categories provide some editing or ellipses of information. For example, in the textual category, the
speech might be represented as a series of bullets or outlines or even
power-point slides, while the graphic image might be a blueprint of the
room. The following questions then arise: How does one classify a compressed image? Is it elaboration or representation? One answer is to
ACM Journal of Educational Resources in Computing, Vol. 1, No. 1, Spring 2001.

Theoretical Multimedia Taxonomy Framework

Multimedia
MEDIA TYPE

Sound
Text

Audience
CONTEXT

Abstraction

Elaboration

Graphics

(increasing
complexity)

Representation

Motion

MEDIA EXPRESSION
(increasing abstraction)

Discipline
Interactivity
Quality

Usefulness
Aesthetics
Fig. 1.

Original multimedia taxonomy.

compare the question to one of lossy or lossless compression. Finally, the

abstraction category is the abstract presentation of information, often
iconic in form. This category is related to metaphor. For example, if the
speech is about acid rain, the text as abstraction might be written as falling
rain in the style of the French poet, Apollinaire. Such use of text is not
newilluminated manuscripts often used micrography to present information in text formed into patterns. Information about the room could be
presented in a icon showing merely sketches of the room.
The final dimension, context, is an attempt to respond to the fact that
interactive multimedia applications do not exist in a vacuum. They are
expressions that are meant to be used by an audience within a topic area or
discipline, and they are judged well done, or not, depending on criteria of
usefulness, quality, and aesthetics. These measures reflect the polysemous
nature of multimedia. Education has claims on multimedia, so purpose,
interactivity, and learning become integral to it; and since art also claims
multimedia, aesthetics is added; multimedia is also claimed by technical
fields like signal processing and computer science, so issues of quality are
included. For example, the category quality refers to the technical aspects
of application delivery, and might be described by the attributes of clarity
of presentation or fidelity of reproduction. Appropriate concerns within this
category might include clarity of characters on the screen as well as the
technical synchronization of two media. Additionally, using the attributes
within this category as part of an evaluation protocol will prompt an
ACM Journal of Educational Resources in Computing, Vol. 1, No. 1, Spring 2001.

R. S. Heller et al.
Table I.

Media Form

Sample Questions from the Quality Dimension

Elaboration

Text

Word choice
unclear/clear

Sound

Speech captured
without stutters
Image details clear

Graphics

Motion

Media Expression
Representation

Abstraction

Math functions correct

Font for bold and italics
readable
Sound effects recognizable
Quality of image (low to high)
Size of image (too large/small
to just right)

Icons are clear

Iconic metaphor is
sharp

Quality of motion
(jerky to smooth)

Multimedia

Media technical
synchronization

evaluator to ask questions like: Is the motion of the lips synchronized to the
speech? Table I contains a sampling of questions that might be prompted
by the quality dimension in the multimedia taxonomy.
3. USING THE TAXONOMY
The multimedia taxonomy can be used for design and evaluation of multimedia products and applications. The taxonomy aids brainstorming . Using
it for design allows implementers, as they develop new materials, to
consider the available media forms and formats. Designers often consider
issues of metaphor, interaction, cognitive accessibility, and the use of
media. For example, in an application on a visit to the island of Goa (an
island community in India), the designer wanted to present most of the
information using text. However, textual presentations are often boring,
dense, and not attractive to the user, especially to someone browsing
through a lot of material. Using the multimedia taxonomy, it becomes clear
that text does not have to consist of full sentences. It can be used as a diary
with printed notes and a series of post-its, underlines, and marginalia to
indicate additional ideas. The diary is lively, and the information is given
in its entirety (Figure 2). In another application, in an attempt to present
the history of one of the oldest churches in Detroit, the application
combines church bells, choir music, and a voice-over replete with echoes
reminiscent of a thundering preacher. In an application on a visit to the
Mayan culture, using a graphical map as a navigational guide and a
graphic of a Mayan leader to indicate the source of the information allows
the developer to provide multiple avenues of information for a user.
In the Cite dArte presentation included in this paper, the student
creator uses the metaphor of a magazine to present the place called Paris.
He uses various forms of text to get his metaphor across, including full
textual descriptions as well as representational text for tables of contents
and shorter descriptions of wine making. While his graphics are not always
of high artistic quality, his series on wine making uses three simple,
ACM Journal of Educational Resources in Computing, Vol. 1, No. 1, Spring 2001.

Theoretical Multimedia Taxonomy Framework

Fig. 2.

Use of text based on taxonomy.

representational drawings to show the process. While he could have used a

full motion video sequence to portray this process, in keeping with the
magazine metaphor, he chose the animation sequence. The student also
uses a variety of sound, speech, and music. The opening sequence includes
a waltz-like score with animated text to introduce the CD. In all, the
student uses various aspects of the taxonomy to provide a variety of media
presentations and yet ensure the consistency of the message.
To use the taxonomy as an aid in evaluation, it is best to review
evaluation as a stage setting for using the taxonomy. Experimental opportunities, field evaluations, system testing, comparative testing, expert
evaluations, and checklists are typical formative and summing-up evaluation experiences. Each can be enhanced by using the multimedia taxonomys form and format. Controlled experiments can be enhanced as they ask
questions about particular media expressions, for example: Is this color
better (or worse) than that color? In the more open field evaluations, we can
use the taxonomy to help develop questions related to the purpose or the
users. Wu and Martin [1997] used the taxonomy in their field study of
media preferences. By observing the taxonomy categories, they were able to
design an experiment in which they asked shoppers in a mall how they
preferred to receive information. Reeves and Harmon [1993]) provide a
checklist for multimedia evaluation that fits the taxonomy well, and using
the taxonomy can be expanded or made more specific. In asking about
aesthetics, they offer an analysis continuum from pleasing to displeasing as
an evaluation check. The taxonomy suggests a series of additional categories for aesthetic evaluations.
When using the taxonomy as an evaluation tool, the multimedia category
is the richest. It allows us to reflect on the difficult issues of synchronization. For example: Do text and images synchronize temporally and spatially
and well as cognitively? For two media to synchronize temporally, we want
to ask if they are timed to appear at the right moment. Text could appear
ACM Journal of Educational Resources in Computing, Vol. 1, No. 1, Spring 2001.

R. S. Heller et al.

before, at the same time, or after an image (depending on the designer).

Spatial synchronization involves asking about the physical layout of two or
more media: Is the text positioned correctly to the right (or left, or above or
below) the image? Positioning could be related to style as required by an
outside source or by ease of use for a reader of left to right or right to left
languages. Cognitive synchronization can be thought of in terms of how
well we matchi the intellectual level of the material presentation with the
audiences receptive ability.
Originally, the taxonomy had only three categories in media expression.At the start of the work on evaluation, it was observed that questions
often repeated themselves, regardless of the form of medium or expression
format. For example, questions about text size are appropriate no matter
the tests media expression. Based on this observation, the taxonomy was
expanded to provide a category called general in the format. This category
is used for questions that are constant over the format space. The newly
expanded taxonomy becomes a floor, not a ceiling, for a series of guidelines
that can be used to generate a series of questions about an application.
Contextual categories that are not applicable to the specific product being
evaluated can be eliminated from the protocol. The questions generated
could be framed either as a Likert scale question with five gradations or
open-ended questions requiring a free-form user response. For example, to
evaluate use of color, one can generate a question for the form:
Not
Pleasing
Pleasing
Use of color 1................2............... 3............... 4.............. 5

The discipline aspect of the context is considered here. Selecting the

media and presentation techniques for the multimedia presentation depend
to a large extent on the information to be communicated. Given the
information content, it is the designers job to choose the most effective way
of conveying the important messages of the multimedia product. Content
may be categorized, analyzed, and represented in different ways for different purposes. There are often many methods to present a certain kind of
information, but there is no general guidance that can help designers to
select the media. In other words, better understanding of the informations
characteristics, their relation to the characteristics of the media, and the
identification of users and tasks will help multimedia developers to make
critical decisions. For example, the presentation design will vary depending
on whether we are dealing with physical entities in three-dimensional
space or with abstract and logical entities in arbitrary, cognitive space.
What would the designers choice be if the relations among the data in the
set are temporal, spatial, or some particular kind of semantic relationship?
A research-based categorization of different types of data across many
disciplines is presented to help designers, developers, and evaluators of
multimedia software. Such a categorization must identify the data characteristics that may impact the effectiveness with which a given medium
expresses the information. The system should be as comprehensible as
ACM Journal of Educational Resources in Computing, Vol. 1, No. 1, Spring 2001.

Theoretical Multimedia Taxonomy Framework

possible, so that any number of other categories could be fit into the
system.
Selecting the way to convey information does not only depend on the type
of information. In cases where more than one medium and more than one
presentation technique could be used, other factors will determine the final
decision. The intent of the communication, user characteristics, and the
tasks that the presentations are designed to support are important issues
as well. Many design decisions will be affected by factors like hardware and
software characteristics, conflicts in coordination imposed by multiple
media, and time and space restrictions in the environment where the
interaction is to take place. Cross-factor and more comprehensive analyses
are needed to find the answers for many design decision-making processes
(e.g., media selection, presentation mode, and content selection) by taking
all the relevant issues and their interdependencies into account.
4. CONTENT CHARACTERISTICS
It is a great challenge to identify the general information characteristics
critical for multimedia design. How can we define and characterize unique
properties of different types of information across many disciplines and
applications? Is there any existing classification that we can use for
multimedia development? Should we have a holistic approach to the
content or try to categorize smaller content units or even data entities? The
parallel between MM designers and painters given in Reiser and Reiser
[1995] could be the answer. Painters look at each detail separately and also
step back to look at the composition as a whole. Designers, just like
painters, may find it difficult to not focus on details, and hence to forget the
purpose of those details in conveying the message as a whole. Thus, the
description must identify general categories of data objects as well as
relationships among them that may affect the decision-making process.
A review of the classification systems already in existence may suggest
some of the information characteristics worth considering. Several research
areas address the problem of classifying information content. Perhaps the
most systematic of these in terms of usefulness for multimedia development are the systems in artificial intelligence (AI). Characterizing information is a prerequisite to defining the expressiveness of presentation techniques and automating the process of technique selection in intelligent
multimedia user interfaces [Feiner 1991; Feiner and McKeown 1993;
Maybury 1992; Roth et al. 1993]. The type of information or the nature of
the content was identified as one of the characteristics important for the
media-selection process. Various classifications of information types have
been proposed [Andre et al. 1993; Roth et al. 1993; Arens et al. 1991], but
there is no general classification that could be used across many disciplines
and applications. Some of the differences among the content type categories
in these systems are due to the characteristics of the discipline or domainspecific information the investigators were concerned with. The differences
in the classification systems were also affected by the different applications/
ACM Journal of Educational Resources in Computing, Vol. 1, No. 1, Spring 2001.

R. S. Heller et al.

tasks for which the systems were devised. Many categories proposed in the
AI literature seem to fit fairly well into the categories of our taxonomy,
although few appeared to be examples of one category.
An exhaustive body of the literature on content can also be found in the
area of instructional design. Here the analysis of the content is oriented
more toward educational objectives. The content structure and organization
are discussed in terms of creating interactive and effective learning environments that can accommodate different learning styles and teaching
approaches, and the impact on the content structure and organization
when applied to instructional design, in this case content structure, is
driven by the application as well as the discipline [Jonassen 1988; 1991;
Mayer and Sims 1994]. Learning theory has given a lot of attention to the
cognitive processes in the reception and organization of information [Tennyson 1977; Gregg and Farnham-Diaggory 1975]. The implication of this
research is relevant for future comprehensive analyses in multimedia
taxonomy when complex interdependencies of many categories and attributes (e.g., content type, perceptual problems, users cognitive capabilities and limitations, their previous knowledge of the content being taught,
users goals, intent, and communication environment) will be analyzed.
5. INFORMATION TYPE TAXONOMY
Information type appears to be a necessary, or at least a useful, dimension
in categorizing content information. The proposed categorization should be
considered as a first step in the development of a content taxonomy.
Information type taxonomy is presented on Table II. As with any aspect of
a taxonomy, that of information type can be expected to expand with use.
The taxonomy appears to be general and applicable to different units of
informationfrom single data objects, to sets with different relational
connections, to classes, and even systems. It is built upon the foundation of
language (e.g., objects (nouns, abstract or compound actions), verbs, etc.), a
commonsense view of several application domains, and already existing
classifications found in the literature. In an attempt to find a suitable way
to present the relationship between the information type and the media
taxonomy, a three-point scale (*, **, and ***) is chosen to rate the
appropriateness of different media for presenting different types of information [Heller and Martin 1995; 1998].
The effectiveness with which each medium achieves its purpose varies.
Each medium makes use of different skills and capabilities. Graphics might
be preferred to text when presenting concrete information, which involves
physical objects, persons or places with certain visual properties like shape,
size, or color. Images convey information in ways different from, and
supplementary to, text.
It is a well-known fact that text is the most appropriate medium for
presenting abstract concepts and logic [Berinstein 1997; Roth et al. 1993].
Still, there are few well-established symbols and icons that express abstract information and can be used for some abstract objects or terms.
ACM Journal of Educational Resources in Computing, Vol. 1, No. 1, Spring 2001.

Theoretical Multimedia Taxonomy Framework

Table II.

Information Type Taxonomy

Information Type

Media
Type

Concrete
Physical
objects,
persons,
places

Abstract
Abstract
concepts

Spatial
Temporal Quantitative Covariant
Location,
Temporal
Quantifying
Semantic
spatial
relationships information, relationship
relationships
relational
between
between
facts
different
objects
pieces of
information

Text
Examples

*
**
Description Definition
of an object of
psychology

***
Timeline of
events

Sound
Examples

*
**
Description Speaker
of an object giving/telling
definition of
psychology

***
Speaker
reporting

Graphics
Examples

**
Picture of
an object

***
Map of an
area; picture
of furnished
room

***
Distances
between
sites in
miles

**
Speaker
reporting
causes of an
environmental
disaster and
its effects
**
**
Bar chart of Picture of a
distances
place before
between
and after an
cities
event
Speaker
reporting
production
growth
figures

**
Pictures of
events,
states
displayed
from top to
bottom (or
left to right)
Motion
**
***
**
Examples
Video
Video clip of Video
Animated
sequence of
a room with showing
graphs of
an object,
zooming
chronological production
place or
effects
order of
growth of a
person
depicting
events
product
each object
Multimedia
***
***
***
***
***
Examples
Combination Combination Combination Any
Combination
of graphics of graphics of graphics
appropriate of graphics
and text
and text or and text, or combination and text
sound
motion and
sound
Appropriateness:

low *

medium **

**
Math
problem and
solution

**
Video
sequence of
an action
and its
effects
***
Any
appropriate
combination

high ***

Conceptual graphs are a good vehicle for presenting abstract concepts and
logic. Text should be used to convey quantitative and relational facts,
although some graphic expressions like charts, tables, and graphs could be
used for large numbers of quantitative or relational facts.
Spatial information shows where things reside in relation to one another
in space or explain how we can get from one place to another. Location
information and composition of objects are best depicted by pictures and
ACM Journal of Educational Resources in Computing, Vol. 1, No. 1, Spring 2001.

10

R. S. Heller et al.

maps [Andre et al. 1993]. Actions and movements usually have a spatial
attribute, too. Text or sound could be added for completeness.
Temporal specifications and temporal relations among states, events, or
actions could be communicated by text. The sequential nature of text has
been exploited to tell stories and the history of events. Graphics can also be
used for temporal information, but has limited applicability. Overlapping
events and certain time specifications (e.g., mostly, periodically, or in the
future) are hard to express using only graphics [Andre et al. 1993]. The
organizational scheme may reflect different functionalites such as task
characteristics, domain knowledge relationships, or logical ordering and
connections to accommodate user and designer goals. In addition, many
units could be decomposed into more granular entities that do not necessarily belong to the same scheme. For example, information about the United
States could be divided geographically (e.g., east coast, west coast) and
then presented in alphabetical order (e.g., their names) or numerical order
(e.g., area size).
Different semantic relationships between data objects (e.g., cause/effect,
logical ordering, and problem/solution) that must be presented as one piece
of information, called covariant information. The selection of the media
here depends on the nature of the data we are dealing with and their
relationships. Some pieces of information may fall into more than one
category. For example, some of the covariant data types (e.g., cause/effect,
and action/result) have spatial and/or temporal attributes. It became apparent that in trying to fit specific information types into the media
taxonomy, some information characteristics are described differently. Thus,
this classification scheme contains some nonexclusive categories. The presentation of artistic types of data (e.g., poems, songs, movies, dramas or
ballets, performance, sculpture, and painting) usually depend on the original art medium. They could be represented as a separate category or may
fall into one of the existing categories (e.g., painting is concrete; ballet is
spatial; poem is abstract).
When the CD for Cite dArte was being created, the types of information
described in the taxonomy provided guidance on presentation. The presentation includes concrete and abstract information about the painters lives
as well as their motifs. The references to the wine producing sections of
France revolve around spatial and temporal information is addressed in the
sequencing of wine production. There was an attempt, guided by the
taxonomy, to present covariate information in directing wine choices.
6. ORGANIZING CONTENT
People organize information in order to understand, to explain, or to control
presentations. The content structure and organization influences the way
people comprehend the information presented [Rosenfeld and Morville
1998]. Structuring (i.e., segmentation or chunking) refers to partitioning
the content into discrete units that are linked for the presentation. This
involves a selection of the pattern (i.e. organizational scheme) according to
ACM Journal of Educational Resources in Computing, Vol. 1, No. 1, Spring 2001.

Theoretical Multimedia Taxonomy Framework

Table III.

11

Content Structuring Pattern

Content Organization

Media Type
Text

Sound

Graphics

Motion

Multimedia

Alphabetical/
numerical

Geographical/ Chronological/
spatial
temporal

Logical
(e.g., causal)

Topical
(e.g., by subject)

Text
information
about states in
US in
alphabetical
order
Speaker
presenting
information in
alphabetical
order

US state
Chronology of Instructions
information
events
for preparing
divided into
a meal
East and West
coast group

Different
section of a
lesson

II. (sound
version)

Speaker
presenting
events in
chronological
order

Pictures of
animals in
name
alphabetical
order
Movie actors
in alphabetical
order

Image map of
geographic
area with
clickable
places
Video
sequence of a
house interior,
one room at a
time

Images of
events on a
timeline

Sound clips
categorized by
performer or
kind of music
(e.g., country,
rock, jazz)
Images of
animals divided
into groups
based on region

Any
combination

Any
combination

Speaker
presenting a
story

Sequence of
pictures on
installing
kitchen
appliances
Video
Video
sequence of
sequence of a
the same
tornado
location at
followed by a
different times sequence on
damage
Any
Any
combination
combination

Movie clips
divided
according to
director

Any
combination

which the content information will be divided into smaller units. The
heterogeneous and ambiguous nature of multimedia makes it difficult to
apply highly structured systems to the content. It is important to determine the attributes of the data elements that will be used to partition or
aggregate along some dimension.
The functional dependencies among the units will determine the appropriateness of different schemes. Consideration should be given to the
arrangement of the content(s) and how the arrangement fits the designers
perspectives and overall plan for delivery. Several research studies show
that changes in the semantic content or variation in structure are reflected
in differences in recall [Last 1998; Mayer and Sims 1994]. The organizational scheme may reflect different functionalites such as task characteristics, domain knowledge relationships, or logical ordering and connections to
accommodatea user and designer goals. In addition, many units could be
decomposed into more granular entities that do not necessarily belong to
the same scheme. For example, information about the United States could
be divided geographically (e.g., East coast, West coast) and then presented
in alphabetical order (e.g., according to their names) or numerical order
(e.g., by area size).
ACM Journal of Educational Resources in Computing, Vol. 1, No. 1, Spring 2001.

12

R. S. Heller et al.

Organizational patterns like task-driven or metaphor-driven organization [Rosenfeld and Morville 1998] are considered logical patterns of
content. In this case there is always some kind of reason/logic underlying
the relations among the functions, actions, and other parts of some task.
Metaphors are also used to organize logical patterns of content in an
intellectually meaningful way.
Content organization is a set of modules and the possible paths that
users may take in traversing the content. In other words, the designer
determines which modules should be next and when the user should shift
to the next module. Domain knowledge is stored in the knowledge base that
is structured by a particular data model (e.g., entity-relationship model,
object-oriented paradigm, and expert system). The data model defines the
organization of the information contained in the knowledge base. The
organization defines the logical relationships among the content units in
the knowledge base.
There are three possible types of content organization: sequential/linear;
hierarchical (i.e., tree structured); and Web structured (i.e., nonlinear,
associatively structured information). Systems can be static or dynamic in
terms of content variability. Interactive expansion and modification of
content organization is a feature of the generative organization. It is clear
that having adopted a certain interactivity style, we can choose an appropriate organization type [Aleem 1997]. Hierarchical and Web organizations
are appropriate for a reactive type of interactivity, while Web and generative organization seems to be a natural choice for the productive interactivity style.
7. MULTIMEDIA TAXONOMY FOR EVALUATION
Multimedia taxonomy is best understood in its application. In evaluation
studies directed at graduate computer science classes, students typically
identify an expanded set of questions. In this course [Heller and Martin
1998] students examine the media taxonomy at length and are then
assigned to teams to develop an evaluation protocol in one of the context
categories for a specific application such as AdaMentor (website), a site for
teaching Ada online, or the Clinique promotional CD. For example, the
aesthetics category produced the following concerns: In the general expression of text, we might ask: Is the font size appealing? While in the
elaboration expression of text, it is appropriate to ask about literary
expression on the continuum from dry to poetic. In the representational
expression of text, students questioned whether the format of the outline is
logically well structured, and in considering the abstract expression within
the aesthetic categories, they asked if the textual image as an icon is
pleasing.
Similarly, when considering questions of the audience for text, students
asked if the instructions were clearin elaborative or representational text
expressions, they asked evaluative questions about the appropriatness of
the feedback to the audience and the compatibility between the reading
ACM Journal of Educational Resources in Computing, Vol. 1, No. 1, Spring 2001.

Theoretical Multimedia Taxonomy Framework

Table IV.

13

Typical Aesthetic Evaluation Questions

Media Expression

Media Type
Text

General

Elaboration

Representation

Font size appealing Literary expression Format (i.e. outline)

Free of stereotypes (dry to poetic)
well placed
(media overuse?)
Position of
Overall impression
messages
Screen design
appropriate
Screen spacing
Use of color
Sound
Sound
Background sound
(annoying to helpful)
(pleasing)
Sound
(noisy to tuneful)
Volume (too soft/
loud to just right)
What did you think
of the sound?
Graphics
Still images
Background
attractive?
images (too faint)
Use of color
pleasing?
Motion
Video or animation
(not pleasing to pleasing)
Multimedia Overall design
(chaotic to organized)
Overall design
(colorless to colorful)
Overall design
(dull to exciting)
Overall design
(dark to bright)
Overall design
(cold to warm)

Abstraction
Metaphors
consistent within
application
Text image or icon

Sound effects (not

pleasing to pleasing)
Sound effects
(clear to unclear)

Icons pleasing?

level of the audience and the level of expression in the description. The
usefulness category prompted them to assess the value added by a media
type or expression. The extended multimedia taxonomy elicited questions
about sound quality as well as sound impact.
Under the category of usefulness, students generated questions about
whether a medium like sound was functional by asking users to rate the
sound on a Likert scale that ranged from annoying to helpful. They also
prepared protocols with open-ended questions about what the user remembered about the sound in the application being evaluated. Thus, the newly
expanded multimedia taxonomy became a floor, not a ceiling, for a series of
guidelines that can be used to generate evaluation questions about a
multimedia application. Parts of the taxonomy not relevant to the specific
product being evaluated were ignored by the students when developing the
evaluation protocol and data-gathering instruments.
ACM Journal of Educational Resources in Computing, Vol. 1, No. 1, Spring 2001.

14

R. S. Heller et al.
Table V.

Typical Audience Evaluation Questions

Expression

Type

General

Elaboration

Text

Intended audience
is clear
Cognitive load
(unmanageable to
manageable)
Knowledge space
compatibility
Media overuse1
Pace controlled by
the user
Program of interest
to intended user?

Appropriate level
of difficulty in
sentence structure
and vocabulary

Representation
Feedback is
audienceappropriate
Instructions are
clear (no to yes)

Sound
Graphics

Motion

Abstraction

Metaphor
appropriate to
user?
Purpose of motion
is clear

Multimedia
1
In this category media overuse relates to whether or not the audience is being overstimulated by the use of media. For example, for field-dependent thinkers, the use of too much
media might be a distraction from the message.

7.1 Aesthetics
Table IV presents student-generated formative evaluation questions on the
aesthetics, appearance (the artistic look or impression) of the presentation.
Questions related to design, rather than function, are the focus here.
7.2 Audience
Audience guidelines are intended to direct the evaluator to issues of how
the IMM relates to the audience and how the audience might process the
media form in a specific media format. Before preparing an evaluation
protocol for a particular IMM, the evaluator should know who the intended
audience is. For example, in text as elaboration (Table V), it is necessary to
ask whether the text is at the reading level appropriate for the intended
audience. On the other hand, while text as representation does require the
user to read, there are fewer words and grammatical constructions in this
category; but the representation (e.g., outline) might not be familiar to the
particular audience.
7.3 Discipline
Guidelines here are intended to identify the content-specific material in
each of the media forms within a specific media format (Table VI). As
students prepared questions in this category, they used an expanded
ACM Journal of Educational Resources in Computing, Vol. 1, No. 1, Spring 2001.

Theoretical Multimedia Taxonomy Framework

Table VI.

15

Evaluations Related to Discipline

Expression

Type
Text

General
Major points are
easily recalled
Appropriate level of
difficulty
Knowledge space is
compatible with
purpose
Text is too
much/little
Media selection
inadequate

Sound

Elaboration
Information is
missing
Easy to get to
central point
Information is
factually correct
Information content
(irrelevant)
Information
(confusing)
Information (boring)
Information
(too little)
Information lacks
credibility
Information
presentation obtuse
Sound conveys
(too much/little)
information

Graphics
Motion
Multimedia

Representation

Abstraction
Metaphor intuitive
for purpose

Sound effects
convey information
Icons convey
information

Media integration
(uncoordinated to coordinated)
Sound with stills
(ineffective to effective)
Sound with motion
(ineffective to effective)
Text with sound
(ineffective to effective)
Text with stills
(ineffective to effective)
Text with motion
(ineffective to effective)

version of the media taxonomy that included the work described in the
content characteristics section described above.
For example, for concrete topics such as the description of Clinique
cosmetics, students queried whether the video presentation was clear and
well sequenced. Since the CD also included logical content on how to use a
series of cleansing products, students asked whether the textual checklist
was synchronized with the graphic presentation.
7.4 Interactivity
In this category, evaluators should address aspects of control, navigation,
and linking. Aleem [1997] expanded the relationship between the interactivity
ACM Journal of Educational Resources in Computing, Vol. 1, No. 1, Spring 2001.

16

R. S. Heller et al.

Table VII.
Media type

Examples of Media Expression In the Interactive Category

Passive

Reactive

Text

No user control;
sequential
presentation

Page turner
Linear spacing

Graphics

No user control;
sequential
presentation

Able to make
predefined
changes in
graphics

Sound

No user control;
sequential
presentation

Motion

Multimedia

Able to make
predefined
changes, adjust
volume
No user control;
Able to make
sequential
predefined
presentation
changes such as
path or target of
motion
No user control of Any combination
any combination of of types
types listed above

Proactive
Browsing,
hypertext, fixed
anchors, paths
with choices for
user
User initiated
changes to
graphics (size,
shape, color,
position)
Changes to
stations, tracks,
fast-forward, loop

Directive
Dialog-based,
creative writing,
word processing

User created
graphics

User created
sounds

User controlled
User created
start, stop, pause, animated
forward, reverse
sequences

Any combination
of types

Any combination of
types

attribute and the media type and expression. Further, he subdivided the
attribute of interactivity into four categories: passive, reactive, proactive,
and directive. With passive interactivity, the user has no control, instead
all control is embodied in the application (e.g., automated setting in a
PowerPoint presentation). Reactive interactivity provides limited response
for the user within a scripted sequence. Proactive interactivity allows the
user to play a major role in the design and construction of situations,
typically by manipulating values. Multimedia that has interactivity at the
directive level allows the user to both respond and initiate actions within
the application, as well as to tailor aspects of the environment such as
selection of color choice, feedback choice, and so on. Table VII (excerpted
from Aleem [1997], and used with permission) identifies examples of media
expression within each of these categories for the media types of text and
sound.
7.5 Quality
Table VIII refers to the technical, reproductive aspects of the IMM, and
lists some of the questions prepared by the students in the project evaluation.
7.6 Usefulness
This category refers to the value of the material presented in the IMM as
well as its ease of use. Since interactivity covers navigational issues and
ACM Journal of Educational Resources in Computing, Vol. 1, No. 1, Spring 2001.

Theoretical Multimedia Taxonomy Framework

Table VIII.

17

Evaluation Questions Related to Quality

Expression

Type

General

Text

Word choice
clear/unclear

Sound

Graphics

Elaboration
Is it clear how to
reach goal
Data is correct (no
to yes)
Speed appropriate/
inappropriate
Grammar correct
Reading characters
on screen (very hard
to very easy)
Spelling correct (no
to yes)

Representation
Math functions
correct
Is it clear which
input device to use

Abstraction
Position of icon and
buttons on screen
(inappropriate to
appropriate)

Sound quality
(unclear to clear)
Sound speed
(too fast/slow to just right)
Sound volume
(too loud/soft to just right)
Quality of image
(unclear to clear)
Quality of image
(low to high)
Quality of motion
(jerky to smooth)
Size of image
(too large/small to just right)

Motion
Multimedia

usefulness covers the ease with which a user can operate the application,
one might think there is some confusion between the two categories. But
there is a distinction. In addition to asking whether the user can operate
the equipment necessary to make the application useful, one should ask
whether the application is worthwhile to the user. Questions like: Can the
user operate the head tracker or the roller ball? and Is there a need for
external devices to run the IMM? are appropriate in the ease of use
category, and even more so are questions on the value of the application to
the user. Table IX includes some of the class questions. Students asked
questions about the usefulness of the application on the basis of the tasks
accomplished. For example, in the AdaMentor evaluation, students queried
whether the representational instructions enabled the users to get to the
specific content area.
As can be judged from the foregoing, not all of the areas within the
three-dimensional multimedia taxonomy are complete. Some are being
examined in detail, while others are still to be developed. Students in the
Seminar on Multimedia Evaluation in the Department of Electrical Engineering and Computer Science at The George Washington University
ACM Journal of Educational Resources in Computing, Vol. 1, No. 1, Spring 2001.

18

R. S. Heller et al.
Table IX.

Evaluations Related to Usefulness

Expression

Type

General

Text

Have you used the system

before (No to Yes)
Did you view the entire
system?
How many features have
you used?
(Did you get what you
wanted?)
Overall function
(dysfunctional to functional)
Ease of use
(difficult to easy)
Easy to get to central point
in program? (No to Yes)
Unity of action supports
central theme (No to Yes)
Program does not cause
disorientation (No to Yes)
Would you use the system
again?
How much did the presence
of a specific medium help in
your use of the system
(very little to very much)
Specific medium information
(not interesting to interesting)
Specific medium information
(not helpful to helpful)
What specific medium did
you remember the most?
Overall impression
(confusing to clear)
Overall impression
(disorganized to organized)

Sound

Elaboration Representation

Graphics
Motion
Multimedia

Abstraction
Was the metaphor
intuitive to the
application?

Media integration
(uncoordinated to
coordinated)

[Heller and Martin 1995] have been successful in designing evaluation

protocols using the multimedia taxonomy as a guideline. Table X gives a
sample of the survey comments from the formative evaluation of AdaMentor, an online web site for the study of the Ada programming language.
Multimedia taxonomy helped pinpoint such problems such as too many
forms within the web site and the inability for a user to see the entire
page. On the bases of the evaluation, students were able to suggest that
the Ada Mentor developers reduce the navigational capabilities and clarify
the functionality of navigational icons.
ACM Journal of Educational Resources in Computing, Vol. 1, No. 1, Spring 2001.

Theoretical Multimedia Taxonomy Framework

Table X.
Keyword

Actual Comments of Concern

Hard to read
Could be bigger
Too much technical language
Navigation
Hard to find exercises
Buttons didnt load in some cases
Need to move inherent icons
Need better button description
Trouble getting back to where I was
Speed
Took forever to load
Slow on T1 line
Slowness more trouble than its worth
Screen design Cant see entire page
Screen size for exercises is too small

Content

19

AdaMentor: Selected Responses Using the Taxonomy Protocol

Text

Interactivity

Actual Comments of Praise

Not too wordy

Link design is efficient

Clean setup
Easy to look at
Nice graphic layout

Not much better than a book

Book could have been read anywhere
Exercises are not interactive
Need better explanations
Information on each topic
Too much technical language
was sufficient
Include security information only when necessary
Make it more fun

8. CONCLUSION
We believe that multimedia taxonomy is a reasonable organizing framework. The extensibility of the taxonomy has already been demonstrated by
the fact that it supports the work by Aleem [1997] and by the extensions
detailed in this paper. Admittedly, there is even more room for extending
the multimedia taxonomy if new ways of thinking about multimedia
context are considered e.g., motivation or stimulation. For example, it is
worth considering whether the context dimension should be expandedit
may be useful to add a category for a computing/presentation system. Such
a context dimension category might ask whether a product should be
presented on a web page or a kiosk, and how that decision will affect the
rest of the design.
Another extension is to treat each category in the context dimension as a
minidimension itself and to identify values for it. The extensibility of the
taxonomy is demonstrated by the ability to take a category and form new
detailed subtaxonomies. For example, the discipline category, which includes the content concept, could be enhanced to consider the following
roles that MM may play in MM products:
Content: The information to be presented; for example, the body of a web
page.
Presentation mechanism: Those elements of the presentation whose purpose is to facilitate communication of the content (i.e., the user interface,
control mechanisms, metacontent); for example, navigation buttons on a
web page and browser controls.
ACM Journal of Educational Resources in Computing, Vol. 1, No. 1, Spring 2001.

20

R. S. Heller et al.
Table XI.

Examples of Roles for Each Media Type

Media Role

Media type

Content

Text

Paragraphs in a book
Email message text
Newspaper headline or
article

Sound

Recorded weather
information
Sound track in CBT
module

Graphics

Pie chart
Image from a web-based
photo library

Motion

Motion video or
animation of planetary
movement in a CBT
module on astronomy
Multimedia CBT module with sound,
video, and animation

Presentation Mechanism

Merged content &

Mechanism

Book page numbers

Labels of email fields,
e.g., From
Continuation note at the
end of a newspaper
column, e.g., Continued
on page A4 column 2
Telephone menu
Earcons
Presentation note in CBT
module, e.g., This is the
end of Lesson 1.
Icons on navigation
button
Browser logo
GUI elements such as the
scroll bar and cursor or
mouse pointer
Revolving Netscape icon
showing that download is
in progress3

Book index or table of

contents;2

(Any of the examples in

other rows in this
column)

Interactive lesson
Virtual reality
presentation

You have mail

announcement by a web
portal application

Video game
Image map (graphic with
embedded links)
Hierarchical catalog of
web-based library with
links to catalogued items
Talking head of instructor
in a CBT module on
astronomy

2
Indexes and catalogs facilitate communication of primary content but also constitute
information in their own right since they show the structure of the primary information.
3

In terns of the Media Type dimension this is motion and in terms of Media Expression, this
is an abstraction. if, instead of this, we had a status message saying Download is in progress,
it would be text in Media Type and elaboration in Media Expression. In bolth cases, in terms
of Role, it is part of the presentation mechanism rather than content.

Merged content and mechanisms: Integrated or merged content and

presentation mechanisms or self-presenting material.
We could populate the space of this subtaxonomy using role and media
type as its two dimensions, as shown in Table XI. For purposes of design
and evaluation, it may be useful to differentiate between MM elements
acting as content versus those acting as presentation mechanisms.
Multimedia taxonomy is an attempt at the formalism needed to provide
both a theoretical and practical framework for the new and rapidly growing
field of interactive multimedia. Currently, we are starting usability studies
using the taxonomy. Specifically, we are interested in whether evaluators
can use the taxonomy as a guideline to improve their existing evaluation
protocols or does the fragmentation of the taxonomy interfere with, or
distract from, the evaluation. In continuing efforts to establish quality
evaluation products, we are also starting to review the use of objectACM Journal of Educational Resources in Computing, Vol. 1, No. 1, Spring 2001.

Theoretical Multimedia Taxonomy Framework

21

oriented design techniques and metaphors as a possible format for multimedia evaluation. As as a start, we are considering each medium (text,
sound, stills and motion) as a specific object, and each object has its specific
attributes (size, color, form declaration, representation, and form), and
behaviors (interactions). Objects can form clusters, and it is the clusters
that can have different relations and interdependencies. Investigations of
cluster analysis as a methodology for multimedia evaluation is just beginning.
As an organizing principle, the multimedia taxonomy presented here can
be used to understand both design and content messages. Another future
step is to review the taxonomy in light of various studies on the psychological and cognitive aspects of multimedia applications to determine whether
this taxonomy can shed light on these areas. Questions such as how we
come to understand an image and how that understanding is different from
our understanding of text can be answered in part by using the taxonomy.
We present a few examples of the impact of the multimedia taxonomy on
the design and implementation of evaluation protocols for multimedia
products. More work in this area remains to be done.
REFERENCES
ALEEM, T. A. 1997. World Wide Web site. http://www.erols.com/aleem/mediatax.html or
http://www. erols.com/aleem/interact.html,
ANDRE, E. ET AL. 1993. WIP: The automatic synthesis of multimodal presentation. In
Intelligent Multimedia Interfaces, M. Maybury, Ed. MIT Press, Cambridge, MA, 7593.
ARENS, Y., MILLER, L., AND SONDHEIMER, N. 1991. Presentation design using an integrated
knowledge base. In Intelligent User Interfaces, J. W. Sullivan and S. W. Tyler, Eds. ACM
Press, New York, NY, 241258.
BERINSTEIN, P. 1997. Moving multimedia: Information value in images. Searcher, 40 49.
FEINER, S. 1991. An architecture for knowledge-based graphical interfaces. In Intelligent
User Interfaces, J. W. Sullivan and S. W. Tyler, Eds. ACM Press, New York, NY, 259 279.
FEINER, S. K. AND MCKEOWN, K. R. 1993. Automating the generation of coordinated
multimedia explanations. In Intelligent Multimedia Interfaces, M. T. Maybury, Ed. Amer.
Assn. for Artificial Intelligence, Menlo Park, CA, 117138.
GREGG, L. W. AND FARNHAM-DIAGGORY. 1975. Content and Structure in Learning.
HELLER, R. S. AND MARTIN, C. D. 1995. A media taxonomy. IEEE MultiMedia 2, 4, 36 45.
HELLER, R. S. AND MARTIN, C. D. 1998. Multimedia taxonomy for design and evaluation. In
Handbook on Multimedia Computing, B. Fruht, Ed. CRC Press, Inc., Boca Raton, FL.
JONASSEN, D. H. 1988. Instructional Design for Microcomputer Courseware. Lawrence
Erlbaum Associates Inc., Hillsdale, NJ.
JONASSEN, D. H. 1991. Hypertext as instructional design. Educ. Technol. Res. Dev. 39, 1,
8392.
LAST, D. A. 1998. Effects of prior knowledge and goal strength. In Technology and Teacher
Education Annual. 476 479.
MAYBURY, M. T. 1992. Communicative acts for explanation generation. Int. J. Man-Mach.
Stud. 37, 2 (Aug.), 135172.
MAYER, R. E. AND SIMS, V. K. 1994. To whom is a picture worth a thousand words? Extensions
of dual-coding theory of multimedia learning. J. Educ. Psychol. 86, 3, 389 401.
REEVES, T. C. AND HARMON, S. W. 1993. Systematic evaluation procedures for instructional
hypermedia/multimedia. In Proceedings of the Annual Meeting of the AERA.
REISER, H. AND REISER, B. 1995. Aesthetic consideration unique to interactive multimedia.
IEEE Comput. Graph. Appl. 15.
ACM Journal of Educational Resources in Computing, Vol. 1, No. 1, Spring 2001.

22

R. S. Heller et al.

ROSENFELD, L. AND MORVILLE, P. 1998. Information Architecture for the World Wide Web.
OReilly & Associates, Inc., Sebastopol, CA.
ROTH, S. F. AND HEFLEY, W. E. 1993. Intelligent multimedia presentation systems: research
and principles. In Intelligent Multimedia Interfaces, M. T. Maybury, Ed. Amer. Assn. for
Artificial Intelligence, Menlo Park, CA, 1358.
TENNYSON, R. D. 1977. Content structure as a design strategy variable: I content acquisition.
ED 137 314.
WU, M. H.-P. AND MARTIN, C. D. 1997. An exploratory study of user media preferences in a
public setting. J. Educ. Multimedia Hypermedia 6, 1, 321.
Received: August 2000;

revised: November 2000;

accepted: December 2000

ACM Journal of Educational Resources in Computing, Vol. 1, No. 1, Spring 2001.