The Processed Video Body as a Supernatural Being
Todd Winkler
Brown University
Providence, RI USA
Todd_Winkler@brown.edu
Abstract
Realtime video processing enables a live sensing body to be
depicted on screen endowed with supernatural qualities. An
exploration of these ideas through the perspective of multiple
art forms shows common techniques that enhance the
experience for participants and viewers. Close attention paid to
the altered self starts a feedback loop that creates new sensation
and kinesthetic response. The mystical depiction of the
expressive human body, by itself and in relation with others,
drives the research and content of this study. Video processing
serves as deep content and inspiration for the work.
Keywords
Video processing, supernatural, digital double, performance
Introduction
Digital doubles representing spiritual emanations or
incarnations of the body relate to notions of ghosts, astral
bodies, out-of-body experiences, and soul projection.
The double may be depicted as a gaseous figure
composed of particles, or in another, more liquid-like
ethereal form, luminous and transparent. Steve Dixon
In his book, Digital Performance (2006), Steve Dixon
suggests several different theatrical functions of the digital
double. Most pertinent to the work discussed here is his
notion of the digital double as a spiritualized or
supernatural being [1]. Live video processing offers a
transformation and a transcendence of the physical body.
Even though the video projection is a result of computation
and manipulation of pixels, it's uncanny attachment to a
performer imparts it with the expressive language of a
moving, sensing being.
The author has explored these concepts in several
artistic disciplines and offers a comparison study here
gleamed from insights creating audio/video installations,
dance pieces and theatre productions that all share
similarities. Careful attention is paid to creating a life-sized
processed body and to its relationship with the living body.
Even in more abstract works, the processing has meaning,
suggesting a narrative to participants as they negotiate their
newfound status as mystical beings. These techniques also
find their purpose as characters in storytelling found in
theatrical work.
The life-sized processed image is important to obtain
balance and parity between the two performers. The
familiar experience of mirrors and shadows paves the way
for the believability of two different but simultaneous
bodies. Even when a process leans so far towards
abstraction as to blur the connection to the original, the
gestures are still felt and read as tethered to a
spontaneously acting person.
We have a great ability for recognizing and
understanding human movement, even from the most
minimal of detail. Perception experiments that abstract the
moving body image down to just a few point lights
(illuminated dots representing the joints of the body) show
that the distinctive timing, articulation and pathways of
human movement are easily recognizable, even when
distorted by masking, timing or viewing angle. Point-light
experiments have also shown the inherent readability of
the expressive body, with our ability to recognize such
things as gender, intent and emotions solely from
abstracted movement information [2]. These studies
confirm the expressivity of human movement, even in a
highly abstracted supernatural form.
Along with the proper scale, synchronization is
important in assuring that participants locate their body
simultaneously in the real world and in the virtual world.
Even small time delays will disrupt this effect. This not
only allows the participants to see their processed image
glowing, shimmering or erupting in fire, but to become that
image to feel what it is like to have that body. This is the
one of the most interesting facets of this work: the
interactive loop of observing and identifying with the
processed body informs the actual movement and
sensations in the real body. Full attention focused on the
digital double produces new sensations and a new
kinesthetic vocabulary refined and directed by the process.
The author has written elsewhere about studies in the
mislocation of the body and its relevance to video
installations - under the proper conditions, participants may
perceive the location of their bodies shift from the actual
location into the projection world [3]. Especially in
interactive work that allows free fully-body movement, the
actions of the user provides a coherent sense of bodily
self in the projected image [4]. This sense of body
ownership encourages installation participants and
performers to explore the physical potential of the virtual
worlds, portraying unnatural physics, strange colors, or
space and time distortions.
1
 Common Techniques
The first step in the process is to isolate a participants
figure from the background. This is done with a single
camera input using some form of background subtraction,
either lumakey (selecting a lit figure in front of a black
background) or difference sensing (taking a shot of the
empty space and looking for any pixel changes when a
subject enters). Using visible light captures more details of
facial expression and clothing. Infrared lighting with an
infrared camera yields an excellent silhouette, with the
added feature of being able to ignore any lighting changes
on a stage or projected video images that might otherwise
be picked up by the camera and cause video feedback (the
projected video is outside of the IR range, and is thus
ignored). The IR technique on stage can be used to meld
the performers real bodies and their supernatural Fig. 1 A processed figure immersed in moving light.
processed bodies into a unified scene. Video processing
and output is handled using Jitter (Cycling 74), with To enhance and represent connections between friends
simple projection mapping and additional effects handled and strangers, a special effect is activated when there is
by the OpenGL component. With work featuring multiple more than one participant; a fine webbing attaches to the
video outputs, the hardware devices used are extremities of two or more silhouettes when they get close
TripleHead2Go (3 projectors) and DataPath wall (within 3 feet), shooting out from one body to connect with
controllers (5 projectors). and touch another. Sometimes traces of the body are left
Three primary techniques will be examined here: visible on screen, encouraging users to paint the walls
keying, connecting, and traces (painting). In keying, the with their interactions and gestures. A hypnotic soundtrack
silhouette is filled with a moving video image, showing the completes the world and assists in pulling viewers in to a
body as light, water, clouds, or fire. Connecting uses some more complete and immersive experience.
objects in the cv.jit collection (computer vision) to create With the loss of physical detail that might create a
rays of light, an energy body, and webbing that connects feeling of separation or body self-consciousness,
two or more participants. Traces allows users to draw participants notice their similarity and connection with
pathways of their movements, simply allowing each frame other freely moving, and sensing, virtual bodies. This
to build up without erasing the previous, or using pixel shared experience leads to the dissolution of the barrier
averaging over time to capture an image with stillness, or a between self and others. Everyone is in the same boat
ghostlike figure with movement. and is made up of the same material. The relationships are
These techniques and ideas are now followed over the complex: people isolated with friends or strangers in a
course of several years of development in three different darkened room, a feeling of being in two bodies at once,
projects. the location or mislocation of sensation, the intense images
of flowing water and light, and the overlap and interaction
Glint (2010) Audio/Video Installation with real and virtual bodies.
Glint imagines an ethereal space where participants as
spiritualized beings meet and interact. The original video
source material for Glint is high-contrast shots of various
forms of sunlight reflected off of the ocean. The long shots
capture a shimmer of light, whereas close-ups produce an
intense ray or large sparkles of changing light. Viewers see
their life-sized bodies filled with this intense light, and/or
floating on top of it, or portrayed as shimmering rays of
energy. Some report feelngs of movements when filled
with moving water, or a feeling of warmth in their body
from the light filling the torso.
One or more people enter a specially built 10 x 14
room with large projections completely filling three walls,
and an infrared camera used to capture participants
silhouettes. There, they are free to move and explore their
spiritualized bodies made of this magical light.
Fig. 2 Bright reflected light inside the torso.

2

 Bali Ghost Dance (2013) Dance Improvisation
Bali Ghost Dance was a collaboration with choreographer
Nyoman Catra to explore improvised Balinese dance
movements and live video processing. A painting mode
produced a colorful collaborative image by accumulating
successive video frames of an energy body image, rather
than just showing a single current frame. This effect shows
the traces of dance movements, a Muybridge-type result
with the addition of fine filigree textures and changing
colors. This technique gave the dancers agency and
creativity inside the virtual world, where they developed
methods for realizing amazingly complex collaborative
pictures from their gestures and movements.
Fig. 3 Collective painting showing body traces.
One section feature a more abstract movement of just
the extremities of the limbs by having a solo dancer wear
all black except for his hands, feet and head. With the
difference sensing technique, using a black background,
any black clothing worn by a dancer becomes invisible
to the process. In this section the dancers paints a beautiful
counterpoint of head, hands and feet as clean trails on the
screen, clearly differentiated by height, without the torso
there to erase the previous images. Other sections used
black cloth to show various dancers magically appear or
disappear on screen.
Similar techniques of transparency and invisibility
were used in an abstract connected-line environment. This
technique translated the body and its movements into long
lines, rendering ever-changing geometrical forms. Finally,
a technique was discovered whereby a dancers image
could be frozen on screen, but the dancer could continue to
interact with the frozen image, making the two come to life
with connected webbing. This is an especially useful
technique for a solo artist, whereby they can interact in a
duet or an ensemble of their previous selves.
The video capturing process used a color camera to
separate the dancers from a black background. The color
image was essential for many of the effects. Unlike the
infrared techniques, the color camera has to be oriented
away from the projection screen or video feedback would
occur, but the tradeoff is that is can pick up and process a
full colored image (the IR technique is good for getting a
silhouette, and possible some facial features, but results in
odd black and white images).
Fig. 4 Traces of the head, hands and feet.
Time's Up: Love, Friendship, and
Transformation Across the Fourth Dimension
(2015) Multimedia Theatre
Times Up is the evening-length theatre piece that is the
culmination of three years of research and development
with writer/director Kym Moore, presented by the Anti-
Gravity Theatre Project. The author created the video
design and original music for the production. The script
and the technology evolved organically together through
numerous workshops, so that the depiction of supernatural
and spiritualized beings became an integral aspect of the
storytelling. A complete account of this large project is
beyond the scope of this paper, but knowledge of the story
and set design is necessary for understanding the essential
role video processing played in creating characters and
action.
The Story (Press release by Kym Moore)
Across moments of time - from medieval France to
present-day Harlem - and even in spaces outside of time,
two souls keep brushing up against one another, intuitively
sensing their past and future lives together. They
alternately fight to remember the intimacy of their souls
and yearn to forget in the hopes of skirting pain.
Throughout, conflicting cosmological forces look on and
sometimes assume physical bodies, with which they thread
these two souls through discord and unity. What is
revealed is a relationship existing outside of time and
space, in a place without culture, gender or politics.
3
 The two souls, Jason and Sienna, are dropped into
three different historical settings and their idealized
relationship must encounter the assumptions, limitations,
and prejudices of each time and place. The piece weaves
between a feudal village during the Cathar crusade in
medieval France where religious dissenters are burned at
the stake; a whites only diner in 1960s Selma during the
chaos of police attacks on civil-rights demonstrators; and a
trendy soul food restaurant in contemporary Harlem where
Sienna (the owner) is reluctant to hire Jason because he is
white. Gradually Jason and Siennas ethereal bodies
become unstuck in time and space, laying bare the arbitrary
yet very real devices of racism, politics and religion. This
true connection between their souls is manifested by
transformational live video processing of the actors
bodies, projected throughout the space. Times Up seeks to
reconsider "otherness and disrupt our habitual methods of
perceiving one another that prevent us from connecting
with each other in meaningful and productive ways.
The main body of the piece explores the relationship
of Jason and Sienna in three historical eras in the real
world, interspersed with sections where their ethereal
bodies become unstuck in time and space. These sections
rely on video processing to capture and alter the actors live
images, placing them inside a fantastical screen
environment. The processing furthers the conceptual goals
of the piece by literally stripping away superficial
differences. These poetic worlds are filled with intense
sound, music and color, with the actors onscreen bodies
appearing to float, fracture and dissipate. Towards the end,
it is through these sequences that the couple remembers
and their spiritualized bodies become unstuck in time, able
to go back and forth to traverse the previous real world
scenes. This realization not only transforms their past
relationships but also opens them up to a whole new way
of living in the current moment.
Set Design, Projection Design
The set for Times Up, designed by Sara Brown, features
two grey vinyl rear-projection screens in a V formation at
the back of the stage, and a curved scrim on rollers at the
front of the stage (front projection on a split curtain). This
allows for the actors to be cocooned in projections with a
more typical video backdrop in back, and the semi-
transparent scrim providing a curved projection surface
that the actors can be viewed through in front (with proper
lighting). Above the scrim is a long white banner, also used
for projection from the main front projector. Using
projection mapping, the front projector serves multiple
purposes: projecting onto the actors bodies, projecting
onto the banner with the scrim open, projecting only on the
scrim, and all of the above. This flexible projection design
offers numerous opportunities to enhance the story by
quickly establishing unique relationships between the
images, the actors and the audience.
A recent iteration of the piece adds two large
projections to the side walls of the theatre, expanding the
on-stage images into the entire theatre space to immerse
the audience. In the opening scene in Medieval France, for
example, Sienna is accused of witchcraft and is burned at
the stake. A burst of fire onstage grows in intensity,
spreading into the audience and engulfing the theatre. The
Selma riots, as another example, moves from the stage to
the walls of the theatre with archival footage and intense
sound. Still images are also used to create scenography in
each real world era, differentiated by image quality and
color: grainy black and white imagery for 1965 Selma;
intense high-saturation colors for present day Harlem; and
gauzy and darkened images for Medieval France.
The video projections support the story with three
main functions: establishing each of the three historical
eras; depicting a place outside of time; and showing the
supernatural transformation of the actors bodies, the
primary topic here, of which a just a few examples follow.
Processing Examples
Selma Dream
During a riot in Selma (1965), Jason has protected Sienna
(a stranger) by taking her into his parents whites-only
diner, where they hide in a storage room while the riots
rage. After some tense moments, they open up to each
other with revelatory conversation. Since it is too
dangerous for Sienna to walk home, they both fall asleep in
the storage room. They dream together in a place where
they can be connected, happy, and free as children playing.
The video effect in this section shows a pulsating golden
outline of their bodies, with webbing that stretches to
connect them together whenever they get close to one
another. They play freely with this elastic connection. In
this world, the processing shows what they could be to
each other without the constraints of politics, race and the
historical era in which they are caught. The spiritualized
body is displayed not only as a separate being, but also as a
combined entity that connects and separates as the
relationship and dance unfolds. Just before dawn, they
return to their sleeping positions as one unified body, but
they must confront the reality of their dangerous situation
once they awaken.
4
 material (fire, air, water), and imbued with supernatural
powers. In one scene, she appears as a warrior, doing a
fierce dance to the sounds of pounding drums. As she stops
with her hands overhead, she leaves traces of her arms on
screen (using video averaging), depicting the multi-arm
goddess Kali, filled with the images of explosions on the
sun (taken from NASA public domain footage).

Fig. 5 Selma Dream with lines connecting actors.

Afterlife
After Sienna is burned at the stake in the Medieval France
scene, she discovers that she exist as a body of sparkling
light, floating in a star field. She is finally free and marvels
at her new ethereal body. She discovers that Jason is there,
too.

Fig. 7 Sun explosions filling the body of Kali

Another scene ends after a huge storm and a struggle,

with the clearing blue sky and light clouds finding their
way inside of her body, and her hand elongated to connect
to the others by extending across the entire back wall.

Fig. 6 Afterlife showing sparkling ghostly image.

Kali, The Goddess

A character encountered outside of time, The Goddess, Fig. 8 Cloud body reaching a long hand to others.
appears in several scenes, her body filled with elemental

5

Jason and Sienna Remember
At some point towards the end of the performance, Jason
and Sienna remember all of their pasts together, and all
time collapses. Two scenes show this transformation by
combining the spiritualized bodies, seen previously
isolated, composited with the scenography of the real
world scenes. The first such moment takes place in the
Selma diner, where Siennas colorful pulsing light body is
mixed with the previously seen black and white diner
image. Jason walks through the wall to greet her in the
diner, also appearing as an energy body with saturated blue
and deep brown colors There, they embrace and they
remember everything. At that moment, they are able to
travel through time at will, and they jump to Harlem. There
they test the reality of the video backdrop and find that
by touching it, it turns to liquid, revealing its transient
nature and the falseness of the fixed world.
References
1. S. Dixon, Digital Performance (Cambridge, MA: MIT Press,
2007), 253 254.
2. M. Shiffrar, People Watching: Visual, Motor, and Social
Processes in the Perception of Human Movement, Cognitive
Science, 2 (2011): 68 78.
3. T. Winkler, ``Flying, Spinning, and Breaking Apart: Live
Video Processing and the Altered Self,'' Proceedings for the
International Symposium and Electronic Art (2010).
4. M. Tsakiris, G. Prabhu, and P. Haggard, Having a body
versus moving your body: how agency structures body-
ownership, Conscious and Cognition, 15 (2006): 423 432.
Authors Biography
Todd Winkler is a composer and video artist, whose work over
the past 30 years has explored ways in which human actions can
influence sound and images produced by computers, in
multimedia dance and theatre productions, interactive video
installations, and concert pieces for computers and instruments.
He is the author of Composing Interactive Music (MIT Press), as
well as papers that bridge the fields of music, computer science,
video art, cognitive science and dance/theatre. Winkler's creative
projects have received international attention at festivals and
venues throughout the US, Australia, Europe, and Asia. He is the
recipient of awards from National Endowment for the Arts,
American Composers Forum, Irish Arts Council, Fulbright
Scholar Program, Meet the Composer, and National Endowment
for the Arts. He is a professor in the Brown University
Department of Music, where he is Director of the PhD program in
Multimedia and Electronic Music Experiments (MEME).

Fig. 9 A translucent spiritualized body in the Selma diner.

Conclusion
Video processing offers unlimited malleability and a
wildly expanded range of possible meanings of human
movement. The realtime and life-size aspects of these
works insures that participants feel an ownership and
participation in the expressive force of their newfound
supernatural bodies. A concerted effort has been made to
make these effects absolutely essential to each piece it is
the material from which they are made. Storytelling,
spiritual beings, connection to others, and magic are all
revealed in this one very personal perspective of this
readily available technology.