Академический Документы
Профессиональный Документы
Культура Документы
2013
visual communication
ARTICLE
Cross-media communication in
advertising: exploring multimodal
connections between television
commercials
and websites
A b stract
I ntroduction
Figure 1. From the YouSee television commercial (12 May 2011). Reproduced courtesy
of IK Jensen, YouSee.
438
media instead of doing something clever. The father then turns on the television, and we witness from his face how the television programme instantly
holds his attention (Figure 1b). Consequently, when his wife comes home after
a run and asks her husband what he is doing, he gets annoyed and tells her not
to disturb him ( do you mind!). The television commercial concludes with
a graphic still in which the characteristic green, white and yellow stripes of
the YouSee brand are displayed as background. First, the stripes appear with a
transparent television screen in front, co-branding a myriad of channel logos,
and with a male voice-over encouraging the viewer to order now (Figure 1c).
Next, the YouSee logo is displayed together with the top-level domain name
.dk (Figure 1d).
If we now visit the announced website (see Figure 2), the green, white
and yellow stripes appear as background, and the YouSee logo appears in
the top left corner of the page. In a semi-transparent box a special offer is
announced in written text, followed by an orange button to click and purchase.
The user is addressed in a direct manner by means of imperatives (get, order
now) and personal address ( you save / what are you waiting for?), and,
on the right, the message free of charge is visualized by a three-dimensional
zero in which a collage of still pictures from different television channels is
displayed.
From the brief description of the YouSee example, it appears that the
television commercial and the website connect in various ways. The website is
announced as a place to order now, but also other kinds of connections seem
to exist, most remarkably, the characteristic stripes that make up the final
graphic still in the television commercial and the background of the website.
Yet, the television commercial and the website also differ in many respects.
The characters from the drama sequence are absent on the website as well as
the implicit mode of address put forward by the drama.
The YouSee example illustrates the kind of cross-media communication to be examined in this article. In particular, we aim to explore how television commercials and websites relate by means of multimodal connections
across media and how these connections can be described and analyzed. To
our knowledge, there exists no fully fledged model for describing and analyzing connections between advertising content appearing in different media.
Therefore, the following study is explorative in the sense that it (1) synthesizes
perspectives from different research areas, most significantly those of advertising research and multimodal analysis; and (2) draws on a sample of actual,
observed multimodal practices (i.e. recent examples of connected advertising content appearing in different media). Our study will focus on the television commercial and the website as two separate texts (following Geertz,
1973: 9ff.) which relate in various ways via what we shall coin multimodal
connections (see more below). Hence, in this article, we will neither include
perspectives of production (e.g. the intention of producers) nor include actual
reception patterns (e.g. viewer experience, use and effect). Instead we will
439
Figure 2. Front page of the YouSee website (12 May 2011). Reproduced courtesy of IK Jensen, YouSee.
440
441
442
While the conceptual distinction between informational and transformational strategies identifies the type of content and argument involved
in advertising, it does not detail the ways in which the addressee is in fact
spoken to. The issue of modes of address can be further specified by identifying differences in the ways in which the addressee is spoken to in terms
of verbal address. In television commercials, verbal address is predominantly
auditory and this is indicative of the importance of sound in television commercials more generally. It is not unusual for television commercials to appear
intelligible even to viewers looking elsewhere. Speech and music are the types
of sounds that ensure that glancing or distracted viewers to some extent are
kept informed on what is going on in the television commercial sound can
function as a sort of umbilical cord between the television commercials and
viewers.
In television commercials, the auditory verbal direct or explicit address
can take the form of a presenter, a testimonial, or a voice-over (Stigel, 2001:
332ff.). The speech of the presenter and the testimonial are both visualized
through appearances of speech performing characters. They differ in the way
in which the speech is directed: the presenter has eye contact with the camera and is thus demanding something from the viewer (cf. the conception of
demand-pictures in Kress and Van Leeuwen, 2006: 117ff.), whereas the testimonial e.g. a celebrity endorser usually has eye contact with an imagined interviewer sitting behind the camera. Contrary to the visually immersed
speech of the presenter and testimonial, the speech of the voice-over originates from outside the visualized context. The three types of verbal direct
address are all primarily factual in the sense that the speech is identifying
and commenting on i.e. providing information on products and brands.
Accordingly, they all three correspond to the broad category of informational
commercials. In television commercials, speech might also appear in a more
indirect and implicit manner as is the case in dramas and montages (Stigel,
2001). In small-scale dramas, for example, speech takes the form of a dialogue
between characters in a dramatic situation (as in the YouSee example), and
usually the role of the product and brand is not declared explicitly. Small-scale
dramas and montages tend to correspond to the broad category of transformational commercials.
Generally, a restricted focus on strategy may lead us to ignore otherwise
important aspects. Thus, we suggest that an analysis of strategy or what we
in the following will coin address strategy should be (1) nuanced by a focus
on the way in which the addressee is spoken to, and (2) supplemented with an
analysis of other types of multimodal connections between the television commercial and the website considered as degrees of similarity and/or dissimilarity
between one or more modes across media (to be expanded on below).
The referred examples of relevant advertising research on cross-media
communication indicate that knowledge about how connections are established is restricted. To obtain a more distinct focus on the organization of texts
443
and their connections we will now present a selection of concepts that we find
applicable from the field of multimodal analysis.
When pointing at similarity and dissimilarity (see Kim et al., 2005) to describe
connections between television commercials and websites, we find resemblance to the concepts of connection and disconnection related to the multimodal principle of framing (Kress and Van Leeuwen, 2006) and Kresss (2009,
2010) description of modal fixing and modal framing. Where modal fixing
concerns the applied modes used for representation (e.g. a written text represents an advertised product in one way, an image in another, etc.), modal
framing is about the ways pieces of information are connected to each other
so that we know what to put together with what, what to read in relation to
what (Kress, 2010: 149).
As pointed out by Kress and Van Leeuwen (2001: 67), modes can appear
in different media, and the materiality of the medium will affect how modes
appear. For instance, written text on the screen in a television commercial
and written text on a website have different qualities. It follows that connections can be established in many ways between the same mode across media,
but connections between different modes across media are also possible, for
example as a written factual piece of information on the website specifying a
dream-like audio-visual scenario from the television commercial.
Based on the linguistic works of MAK Halliday and JR Martin, Van
Leeuwen (2005) presents the linking of items of information as an example of
multimodal cohesion. He categorizes information linking as related either to
the concept of elaboration that repeats or restates information for the purposes
of clarification or to the concept of extension that adds new information, linking it to the existing information in a particular way for example, temporally,
or logically (p. 222). These basic categories are further sub-categorized into
systems of verbal, visual as well as visualverbal linking. Information linking
is considered as a link between two items of content appearing in one and the
same text. For instance, the linking between a text and an image can be realized as an explanation (a sub-category of elaboration) when the text makes the
image more specific (p. 230). Also hypertext links can be considered from the
perspective of information linking (p. 240).
The general categories of information linking seem useful in describing the links between television commercials and their announced websites
as regards the overall functions of the two connected texts. Even though the
announced web address in the television commercial is currently a nonclickable link, it seems reasonable to consider the two texts as linked to each
other (p. 219). The website can, for instance, be characterized as an extension
of the television commercial in that it provides additional information to the
444
445
Announcements
As mentioned, the ad pairs in the sample were determined by the presence of
an explicit reference from the television commercial to the website. This kind of
reference to the connected text as such arguably establishes an important multimodal connection. The question is now how the television commercial refers
to the website, and how the website might refer to the television commercial.
The television commercial most typically refers to the website via
a spoken or written announcement of the web address, i.e. by verbal references. The web address can be presented more or less significantly (by means
of size, volume, duration and placement), and it can be further accompanied
by instructive co-texts that offer arguments and reasons for visiting the website (see Graakjr and Jessen, 2012, for a typology of arguments). In terms of
information linking (Van Leeuwen, 2005), the website is usually presented as
an extension of the television commercial. Visual references to specific web-
446
sites are not represented in the sample, solely the more conventional features
of the digital medium are remediated, e.g. by showing a moving cursor in the
television commercial.
References from websites to television commercials as presented on
television usually appear in a(n) (audio-)visual mode. (Audio-)visual references from websites to television commercials are identified as three kinds of
remediation: firstly, and most distinctively, remediation of television commercials on websites occurs as a repetition of the television commercial. The television commercial takes up nearly the whole space of the web page and loads automatically. Although the television commercial is used as a background on the
web page with other items of information in front of it, the television commercial
is highly foregrounded by the automatic playing of moving images and sound.
Secondly, remediation is identified when the television commercial appears side
by side with other items of information on the web page. The television commercial is not loaded or played automatically; instead it has to be initiated by
the user. Sometimes the television commercial appears as an item of information without further introduction, sometimes it is launched by a welcome or a
slogan. Thirdly, remediation occurs as stills from the television commercial, and
these stills function either as links to a subsidiary page from which the television
commercial can be played or as mood enhancement or product presentation.
Since television commercials appear unannounced during the flow of television
and with no specific destination to refer to, verbal references from the websites
to television commercials are more unusual and are characterized by unspecific
references such as the products from the television campaign.
Participants
In the following, we will examine the appearances of particular participants in
the television commercials and on the announced websites. Informed generally by the communicative characteristics of the genre of advertising, we have
selected four participants: logo, product, character and setting. While other
genres may prompt a different selection of participants, these present what
we consider to constitute the primary subject matter of the communication
(to paraphrase Kress and Van Leeuwen above) when it comes to advertising.
Logo A logo is a graphic mark used by commercial enterprises and organiza-
447
The logo might also appear elsewhere as an integral part of a display of products.
In the periphery, the logo functions as an assurance that the user has arrived, as
a permanent reminder of where the user is (on the particular website), and as
an identification of what and where the user can come back to, namely the front
page of the website.
In television commercials, the logo might also appear small-scaled permanently in the periphery of the screen, for example in the lower left corner
of the screen. More often, though, the logo appears only sporadically during the commercial, for example as an integral part of a display of products
(diegetic or non-diegetic). However, most significantly, the logo in television
commercials appears large and in full screen at the end of the commercial as a
kind of exclamation mark. Here the logo functions to conclude the commercial by identifying and underlining the sender. Sometimes the underlining of
the sender is emphasized by animation of the logo, and often the visual logo is
anchored and enhanced by an accompanying sound signature. In television
commercials that link to websites, the logo at the end also functions as a kind
of colon. Thus, the logo presentation at the end signals that the preceding
information can in some way be extended when visiting a website (cf. information linking). Typically, this possibility is spelled out via an accompanying
verbal incitement to visit the website for example: see [website] for further
information. Whereas the logo on the website has a user-anchoring function
the user is informed and guided on the website, not away from it the logo
in the television commercial can function so as to lead the viewer away from
the television commercial (and television programming, more generally) and
to the website.
Product As expected, connections between television commercials and
websites are often established through the advertised product (or brand). The
product is either shown (in moving images or still pictures) or mentioned (in
speech or writing) in the television commercial as well as on the website. The
product is not necessarily represented in the same mode in the two media,
but there seems to be a common reference point to what is advertised. If we
take a closer look at how the product is shown in the sample, we find that the
product in the television commercial first and foremost is represented in an
audio-visual mode, sometimes followed by cut-out photographic stills of the
product (a so-called pack shot) at the end, whereas on the website the product most often is represented in a photographic mode as still pictures. In the
television commercial the product may appear as part of a narrative, for
instance showing the product in use and the values associated with it. On the
websites the product both appears as a detached or decontextualized object
on a neutral background (this seems to be the case on websites facilitating
purchase) and as implied in different kinds of settings, for instance in pictures showing the product in a use situation or in pictures embellishing the
product by means of, e.g. light and juxtaposition with pleasurable objects
such as flowers and fruit.
448
In television commercials that communicate through a dramatic composition, the product is sometimes hidden to begin with, only to be revealed
later as part of a (humorous) punch line. The hiding of the product serves
as a means to activate the viewer who is supposed to speculate about what is
advertised and how the product is related to the story. In contrast, the representation of the product on websites is often highlighted and pointed out
(if the user needs to search for the product on the website it is probably due
to navigational problems and not resulting from a calculated communicative
choice). The emphasis and pictorial concretization of the product also seems
to be a common trait on websites accompanying television commercials in
which the product is not shown, but only mentioned and implied.
It seems reasonable to assume that the same product will appear in
both the television commercial and on its announced website. However, this is
not always the case. For example, an announcement of a particular product in
the television commercial is sometimes followed by a presentation of another
product on the website. In such cases, either the logo or other kinds of markers
of the brand or the company serve to connect the television commercial and
the website.
As mentioned, the product is not only shown but also represented verbally in speech or writing. In the television commercials the product is mentioned by voice-overs, presenters, testimonials, or by fictional characters in
a dramatic universe. Except for websites on which the television commercial
is remediated in an audio-visual format, the websites do not include speech
(or other kinds of sound that may be argued to represent qualities related
to the product). The verbal representation of the product on the website is
mainly in writing, and the writing appears as slogans, headlines, show cards,
menu items, hyperlinks, and body text. In the television commercials the written representation of the product is limited to slogans, headlines and show
cards (and perhaps conditions of purchase in a small font size). The websites
thus include much more written information about the product than the television commercial; on the other hand, the writing on the websites seems to
be imbued with a distinct oral character (Janoschka, 2004: 125ff.) echoing the
verbal representation of the product by the voice-over.
Character In addition to the visual or verbal representation of the advertised
449
commercials (e.g. models showing the product) who are depicted in stills on
the website, often with simulated eye contact to the user. However, we also
find examples in which the presenter function is shared between different
people in the television commercial and on the website respectively. In such
cases, the website presenter appears more unobtrusive and anonymous than
the presenter in the television commercial. The I/yourelation that is proposed and emphasized by the presenter function is thus framed differently in
the two media. In addition, the presenter function is often represented by
means of written text on the website.
Setting The settings represented in the television commercial and on the
website are usually very different. Except for when the television commercial
is remediated on the website, products are presented in dissimilar settings, for
instance a highly articulated product setting in the television commercial in
contrast to a minimalistic product setting on the website. The product can also
be shown in different stages of the life cycle of the product, e.g. a representation of the product in use in the television commercial as opposed to a representation of the product in the process of production on the website. Usually,
the settings are represented in different modes. Most common is the difference between an audio-visual and a graphic or photographic representation of
settings. It seems, however, that the television commercial in some way anticipates the different setting on the website. We have observed that television
commercials often involve breaks in the composition by way of shifts from an
audio-visual mode to a graphic mode, for example in the final still. In many
cases these stills seem to match the settings of the website. So, despite the obvious differences in the representation of settings in the two media, a variety of
connections can also be identified, for instance in the use of color schemes as
well as in the recurrence of distinctive qualities related to the setting, e.g.
typography or graphic style.
Address strategy
Address strategy refers to ways in which the addressee is positioned. Both television commercials and websites, of course, more or less explicitly position
their addressees as consumers. However, the specific modes of address tend
to differ when it comes to how the consumer is positioned in television commercials compared to websites.
The television commercials in our sample represent a wide spectrum of
address strategies. In general, the sample mirrors the heterogeneous distribution of address strategies in previous, comparable samples (e.g. Stigel, 2001).
While the present sample does not exactly correspond to the distribution of
predominantly transformative address strategies in the above sample of Kim
et al. (2005) e.g. transformative address strategies tend to be more characteristic of the highly profiled commercials broadcast during the Super Bowl
than of commercials during everyday broadcasts transformative address
strategies occur more often in the present samples television commercials
compared to the samples websites.
450
Based on the examination of the sample, we have described a range of multimodal connections between the television commercials and their announced
websites. We have identified how connections can be established by the presence of different kinds of references to the connected text. These kinds of references are located in the text as an announcement of the other. Moreover, we
have examined multimodal connections between texts in terms of different
degrees of similarity or dissimilarity between what is represented and how it
is represented. What is represented concerns the participants in the television
commercial and on the website respectively, and how it is represented concerns mode of representation of participants (visual, verbal, or auditory) and
address strategy (informational or transformational). Different participants
can be represented in the same mode or in different modes across media (and
451
sometimes also in different modes within the same media); moreover, different modes of representation can be related to the same or to different modes
of address. Evidently, how multimodal texts are related across media can be
a very complex matter to describe. During the sequence of a television commercial the product may be represented in different modes, for instance the
product can be represented by means of moving images in the first part of
the commercial and by still photography and speech at the end. Likewise, the
website may involve multiple modes in the representation of the product, and
may implement a mixed strategy as regards address strategy.
Figure 3 summarizes the identified possible multimodal connections
specified as three analytical dimensions that should be taken into consideration when analyzing cross-media communication between television commercials and websites.
The framework allows us to map how multimodal connections are
actualized in specific ad pairs. According to the three analytical dimensions,
we begin by noticing the direction of the announcement from the television
to the website and/or from the website to the television commercial if such
announcement is present. The bi-directional arrows in Figure 3 indicate the
possible kinds of announcements to the connected text. In regard to the participants, two kinds of observations should be made. Firstly, we should examine whether the participants (logo, product, character and setting) are more or
less similar or dissimilar with regard to what is represented (e.g. is it the same
product that is advertised?). Secondly, the represented participants should be
examined according to whether they can be characterized as more or less similar
or dissimilar in regard to how (in what modes) the participants are presented
(e.g. is the product presented in the same way?). As illustrated by the different
452
We will now revisit the introductory example of YouSee to illustrate how the
proposed framework can be applied to describe and analyze multimodal connections (see Figure 4). Considering the announcement of connected texts,
the uni-directional arrow indicates that the television commercial explicitly
announces the website by a verbal (written) reference (see Figure 1b). The
website is not explicitly referring to the television commercial, neither is
any kind of visual reference in terms of remediation registered. In order to
deal with the complexity of participants and address strategy, the television
commercial is divided into two units of analysis. First the drama part will be
addressed, then the final still.
In the analysis of multimodal connections between the main drama
sequence in the television commercial and the front page of the website, we
begin by noticing what is similar in regard to what and how participants are
represented. In the television commercial the product is described by means
of moving images, yet in a very implicit way as something we hear (we do
not see the screen, only the sound from the television set is heard) and as
something mentioned (by the father in his critical attack on teenage media
culture). In contrast, the same product is described in a more factual way on
453
the web, partly in writing, partly by means of the still pictures in the zero in
which we see examples of what YouSee can offer on the screen (see Figure 2). It
follows, that neither the characters nor the settings appear similar. The fictive
characters as well as the home settings of the drama are absent on the website.
Instead, we are confronted with a graphic setting and an explicit identification of the sender (television from us). Accordingly, we notice a difference in
address strategy, namely a transformational address strategy in the television
commercial and an informational address strategy on the website.
When considering the final still of the television commercial and the
front page of the website, a different set of connections appear. First and foremost, the logo appears similar even though the .dk is added in the television
commercial. As regards mode of representation, the product is represented dissimilarly by verbal speech in the television commercial and in written text on
the web. Also the characters are dissimilar as we encounter a voice-over in the
television commercial and a written identification of sender or seller (us) on
the website, yet, as to what is represented (the sender or the seller), the characters appear dissimilar. Finally, the presentation of the setting and the characteristic stripes in a three dimensional room in the television commercial (see
Figure 1d) differ from the presentation of a two-dimensional wall on the website (see Figure 2). Yet, the colors and patterns of stripes are similar. The stripes
of the television commercial seem to allude to a path, while the stripes of the
websites seem to assure that the user has arrived. Unlike the drama part, what
is now becoming similar is the overall informational address strategy.
In Figure 4, the differences in the television commercial, i.e. the drama
part and the graphic still, are taken into account by acknowledging that the two
parts propose different multimodal connections (see the grids in the boxes of
settings and address strategy indicating both similarity and dissimilarity).
However, the framework does not provide consideration of how the
transition from dissimilarity to similarity as regards settings and address strategy is composed. In the YouSee television commercial, the setting of the final
still is, in fact, represented by hints in the drama part by way of two paintings
in which the characteristic stripes are displayed (see one example in Figure
1b). In addition, the colors of some of the house interior seem to match the
colors of the stripes, for instance, the yellow bookcase and the green lamp in
Figure 1a. In this way the drama part seems to tune in on the final still as
well as on the website a level of analytical detail which the proposed general
framework does not easily capture.
C oncluding R emar k s
454
websites in our sample. The three dimensions have been presented in a general
framework (see Figure 3) and illustrated by way of an examination of a specific example (YouSee). Whereas multimodal connections established by the
announcements concern how the two texts are referred to as texts in the two
texts respectively, the participants and address strategies in both texts form
multimodal connections to varying degrees of similarity and dissimilarity.
In regard to participants and address strategy, the connection between
a television commercial and a website can rarely be described unambiguously
as either similarity or dissimilarity (as the YouSee example has shown). It is
also important to stress that similarity does not necessarily imply a stronger
or more coherent connection. Unlike similarity that seems to highlight that
parts belong together or are identical, dissimilarity seems to suggest that the
parts belong to different domains (Van Leeuwen, 2005: 12), yet still supplement each other. Dissimilarity can even be explained via the announcement
of the website which typically informs or indicates what the user should expect
to find on the website, e.g. acquire information about the product. Hence, in
some cases, dissimilarity can be interpreted as expressions of a communicative
intent involving a semiotic division of labor between the television commercial and the website (e.g. the television commercial showing the product in use
and the website facilitating transaction as in the YouSee example), arguably
mirroring characteristic features (i.e. push or pull) of different media involved.
While this article has presented ways of analyzing multimodal connections in relation to three analytical dimensions, it has not examined whether
a typology of multimodal connections between television commercials and
websites can be developed on a general level. Such an examination should
preferably involve an even larger sample of ad pairs, which could then be analyzed and categorized on the basis of the framework presented in Figure 3.
Also, diachronic and synchronic studies i.e. of possible developments and
differences between comparable samples could profitably be performed to
examine the representativeness of the present study and its results. It is beyond
the scope of this article to further examine these questions, but hopefully the
proposed framework can inspire future research on these and similar subjects.
At least, in relation to current research on cross-media effect between
television commercials and websites, the proposed analytical dimensions provide a framework for a more detailed description of the stimuli to which the
consumer is (potentially) responding. Moreover, the framework may serve to
generate specific hypotheses about the relation between types of connections
(a possible typology) and cross-media effects. In the realm of multimodal
analysis, the framework contributes with a particular focus on the complex
nature of multimodal connections by rethinking the concept of modal fixing and modal framing (Kress, 2009, 2010) in a cross-media perspective. The
framework allows us to explore and consider what to read in relation to what
(Kress, 2010: 149) and to make that consideration across media platforms.
455
This research received no specific grant from any funding agency in the public,
commercial, or not-for-profit sectors.
R eferences
456
457
458