See

discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/220876899

Foldable interactive displays

Conference Paper January 2008
DOI: 10.1145/1449715.1449763 Source: DBLP

CITATIONS

READS

38

286

3 authors:
Johnny C. Lee

Scott E. Hudson

Carnegie Mellon University

Carnegie Mellon University

14 PUBLICATIONS 938 CITATIONS

216 PUBLICATIONS 6,769 CITATIONS

SEE PROFILE

SEE PROFILE

Edward Tse
SMART Technologies
40 PUBLICATIONS 637 CITATIONS
SEE PROFILE

All in-text references underlined in blue are linked to publications on ResearchGate,

letting you access and read them immediately.

Available from: Edward Tse

Retrieved on: 09 September 2016

Foldable Interactive Displays

Johnny Chung Lee, Scott E. Hudson
Human-Computer Interaction Institute
Carnegie Mellon University
5000 Forbes Ave., Pittsburgh, PA 15213
{johnny, scott.hudson}@cs.cmu.edu

Edward Tse
Smart Technologies
1207-11 Avenue SW, Suite 300
Calgary, Alberta, Canada T3C 0M5
edwardtse@smarttech.com

ABSTRACT

Modern computer displays tend to be in fixed size, rigid,

and rectilinear rendering them insensitive to the visual area
demands of an application or the desires of the user.
Foldable displays offer the ability to reshape and resize the
interactive surface at our convenience and even permit us to
carry a very large display surface in a small volume. In this
paper, we implement four interactive foldable display
designs using image projection with low-cost tracking and
explore display behaviors using orientation sensitivity.
Author Keywords

Foldable displays, interactive, mobile, projection,

augmented reality, orientation sensitivity, privacy.
ACM Classification Keywords

H5.2 [Information interfaces and presentation]: User

Interfaces. H5.1 [Multimedia Information Systems]:
Augmented Reality.

Figure 1 Foldable fan display with stylus input.

this paper, we explore this concept of inactive foldable

displays and create a number of working prototypes such as
the one shown in Figure 1.

INTRODUCTION

In the realm of science fiction, future display technology

often depicted as holographic surfaces that float in thin air.
Sometimes these displays can be summoned at will in
proximity to a persons body, can be changed in size and
shape to fit the desired usage, can be collapsed or dismissed
in an instant if the user needs to tend to some other activity,
and of course, can support interactive input. While modern
displays have become thinner, higher in resolution, and
provide input using a stylus or touch sensitivity, we still are
a long way from achieving these technological visions.

Emerging technologies such as electronic paper and organic

light emitting diode (OLED) displays are expected to
provide some degree of flexibility. However, current
prototypes remain quite rigid and are typically rectilinear.
This prevents them from becoming truly foldable in the
sense that we think of paper as being foldable. Additionally,
performing input on such flexible displays is an entirely
separate technological hurdle. The approach we use in
exploring flexible displays is by augmenting the appearance
of passives surfaces with image projection [1, 2]. This
allows us to combine the flexibility and minimal weight of
plain paper or fabric with the dynamic content capabilities
of a computer display creating a coherent and fully
functional user experience. PaperWindows utilized a
similar approach to explore interaction techniques with
sheets of paper as if they were digital displays [4], but
focused mostly on flat paper interactions in a tabletop
scenario and relied on a high-cost motion tracking system
for location discovery. Our work demonstrates foldable and
re-shapeable semi-rigid hand-held displays using an
affordable, low-cost location tracking technology.

Many of the displays we see in hand-held devices today are

small LCD displays of fixed shape and size. In this respect,
they are insensitive to the desires of a user or the needs of
an application. Ideally, we would like displays that we can
dynamically reshape or resize to suit our desired usage,
similar to the way we might handle a newspaper, or simply
so that we are able to fit a large display into our pocket. In

Permission to make digital or hard copies of all or part of this work

for personal or classroom use is granted without fee provided that
copies are not made or distributed for profit or commercial advantage
and that copies bear this notice and the full citation on the first page.
To copy otherwise, or republish, to post on servers or to redistribute
to lists, requires prior specific permission and/or a fee.
UIST08, October 1922, 2008, Monterey, California, USA.
Copyright 2008 ACM 978-1-59593-975-3/08/10...$5.00.

287

Figure 2 Foldable display shapes (left to right): newspaper, scroll, fan, and umbrella.
Red dots indicate the location of the LEDs use for tracking in these prototypes.
Tracking

Tracking is accomplished using infrared (IR) LEDs

embedded into the foldable surface and the infrared dot
tracking capabilities of the PixArt camera within the
Nintendo Wii remote. This camera contains hardware blob
tracking up to four points simultaneously with a resolution
of 1024 by 768 at 100Hz. Since the computer vision
tracking is performed by hardware in the camera, this is a
low-cost, easy to implement solution that provides highresolution, low-latency tracking. When positioned adjacent
to a projector, the camera tracking data can be calibrated to
the image location resulting in spatially augmented reality
on moving surfaces [5]. However, camera based tracking
has limitations such as the number of distinct points that
can be reliably tracked simultaneously, the inability to
provide point identity, and manual calibration with the
projected image.
The four point limit of the PixArt chip reduces the
complexity of our foldable geometries requiring simplifying
assumptions, such as requiring shapes to have fairly high
folding symmetry or equally sized square panels. We take
advantage of the semi-rigid properties of our designs and
place the IR emitters at strategic locations such that we can
fit a geometry model to the data. This model provides
estimations of the display edges and corners which may not
be explicitly tracked. Assumptions are made regarding
display starting orientation, point ID, and the range of
movement allowing the model to fit the surface using a
small number of points. The fewer points that are available,
the stronger these estimations and assumptions must be.
However, as the technology evolves providing larger
numbers of trackable points, many of these assumptions can
be eliminated resulting in more robust performance. If the
projector was capable of presenting both visible application
content and non-visible structured light patterns [6], large
numbers of points could be tracked simultaneously without
ambiguous identity further reducing the number of
assumptions and spatial limitations.

Figure 3 Foldable display prototypes at various stages of

expansion (top to bottom): newspaper, scroll, fan, & umbrella
FOLDABLE SHAPES

In this section, we present four foldable display designs.

This is, of course, not an exhaustive list. However, we
believe they present a number of expansion and collapsing
behaviors likely to be used in a typical foldable display. For
each description, please refer to Figure 2 for an illustration
and Figure 3 for images of the working prototype.

The LEDs in the display surface run for several hours using
a small rechargeable battery pack. Since infrared LEDs
emit non-visible light, the LEDs appear as small black dots
5mm in diameter. The LEDs can also be placed beneath a
translucent surface to hide their visual presence entirely. As
with any optical based tracking system, occlusions are a
problem but could be address using additional cameras or
location estimation using remaining visible points.

Newspaper

One of the most common formats in which we interact with

large sheets of printed material is a typical newspaper.

288

Figure 4 Orientation sensitivity behaviors (left to right): double-sided display surfaces can react differently depending on the
direction they are flipped, simulated lenticular can change the document view depending on the angle of viewing in a hand-held
display, or a tabletop scenario where tilt angle may correspond to different privacy states: private, public, excluded.

Sometimes referred to as the broadsheet format, these large

sheets of paper are folded in half vertically and then again
horizontally allowing a variable visual area ratio of 4 to 1.
Additional folds can be added to further increase the
magnitude of variability in visual area. In our prototype, we
use two folds to support viewing half a page up to two full
pages side by side. The user can gracefully increase or
decrease the viewing area simply by unfolding or folding
the display. Tracking, when viewing a half-page, uses only
two IR emitters defining the top edge of a rigid square
panel. The first fold is defined by a third point in the bottom
left corner becoming visible below. The second fold is
defined by a fourth point appearing in the top right corner.

display surface minimizing distortion. Two IR emitters

define the outer and inner radius of the display area. The
hinge location is estimated using the known physical
proportions of the fan. A third IR emitter identified the
sweep of the display along an ellipsoid.
Umbrella

Another common example of expanding and collapsing a

large surface is a parasol or umbrella. These surfaces can
frequently be operated by one hand using spring loaded
designs and can produce a very large surface area very
quickly. Depending on the culture of origin and intended
purpose, umbrella and parasol designs vary from parabolic
bell shapes, to conical, to nearly planar. Distortions due to
non-planar surfaces can be compensated for if the geometry
is known before hand. An umbrella design may perhaps not
be the most ideally suited shape for interactivity due to the
central perpendicular column of the handle. However, the
surface area change ratio is very dramatic making it
potentially attractive for certain applications and lends itself
to rotational input. The handle also provides an optically
plausible location for a projection and tracking device for
true mobility. When all four IR emitters are visible, a plane
is defined which orients a parabolic model.

Scroll

While less common today, large printed material was once

transported and viewed in the format of scrolls that could be
unrolled. This allowed individuals to not only customize the
amount of visible area, but also the location of that area
within a long document hence the concept of scrolling a
window in a typical GUI environment. By creating a digital
display scroll we can change the size and aspect ratio of the
viewable area quickly and easily exposing more of the
application content. This design can also be collapsed into a
relatively small form factor for storage. This is tracked
using an IR emitter in each corner and the display is
assumed to begin in an upright orientation.

ORIENTATION SENSITIVITY

Though we track gross movements for the purposes of

projection and folding, we can also respond to flipping or
subtle display movements to create orientation sensitive
behavior. The following behaviors are illustrated in Figure
4 and images of the working prototype are shown in Figure
5.

Fan

Folding fans are perhaps one of the best examples of a

device that must be very large in surface area to be
effective, usable by one hand so that the other hand is free
to perform a task, and collapsible for easy storage in a
pocket. Coincidentally, these properties are also desirable in
a mobile display technology. As a result, this design may be
one of the most practical for foldable displays in a mobile
scenario. In our prototype the ratio of display area from a
fully expanded to fully collapsed configuration is
approximately 20 to 1 ranging from 100 square inches to a
small strip. The elongated strip can be used to display status
messages, rolling text, or progress bars similar to a portable
music player. Some folding fans designs allow full 360
degree expansion creating a circular display area. The fan
format can either be used in full or partial expansion to vary
the amount of screen area desired. While we used a pleated
folding fan for our prototype, folding fans can also be
composed of parallel slats resulting in a nearly planar

By monitoring the persistent visibility of edges, we can

detect when the display has been flipped and in which
direction. This is accomplished by tracking emitter
visibility and motion modeling to determine probabilistic
movement. Emitters are placed along the edge such that
they remain visible when viewed from the side but back
facing edges will result in disappearance. This provides a
way to create lightweight double-sided display surfaces
simply by projecting different content on each side.
Flipping the surface in different directions (left, right, up,
down, or diagonally) can trigger events for document
navigation or view manipulation [3].
We can react to more subtle tilting movement of the surface
altering the view depending on the angle at which is it held.

289

This is done by monitoring the ratio of the edges of a semrigid surface. We refer to this as a simulated lenticular
screen and can respond to multiple axes of orientation
change.
In a multi-user scenario, horizontally tilting or turning the
display to another person (assuming the computer is aware
of viewer locations) can trigger different document views.
If placed on a table top, vertical tilting may correspond to
the implicit privacy states of the display depending on its
expected visibility. Tilting the display toward yourself
might indicate a private state, placing it flat would indicate
a public state, and tilt it away might indicate an excluded
state, possibly useful for presentation or gaming scenarios.
Use of this capability will depend on the application
scenario of the table top system.
While it would be possible to have the system react to
additional degrees of rotational freedom as well as three
degrees of translational movement, having such complex
display behavior would be highly application dependent.
For example, six degree-of-freedom display tracking would
be appropriate for creating a view portal into a virtual 3D
environment but unnecessary for many 2D GUIs.
Figure 5 Orientation sensitivity prototypes: (top) doublesided flip direction, (middle) vertical and horizontal simulated
lenticular in a hand-held surface, and (bottom) a tabletop
scenario where tilt correlates to different privacy states.

INTERACTIVITY

By tracking additional dots over those embedded into the

display surface, we can track a stylus for input shown in
Figure 1. Since our fan design used only three IR emitters,
we were able to use the fourth point to represent an input
stylus. The stylus contains a button to activate the emitter
providing a passive method of detecting clicking and
dragging. The location of the cursor is mapped to the 2D
location on the foldable display. Alternative style designs
could utilize a tip switch, but this will vary performance
when using different display materials. This technique
provides an easy way to obtain interactivity on all of the
surfaces described including their double-sided variants.
High-speed IR data transmission could be used to transmit
additional data from the stylus such as left and right click,
or orientation data. While it would be possible to support
multiple cursors, a camera based approach may lead to
tracking confusion when the pens approach each other or
the emitters defining display boundaries. However,
projector-based tracking would allow many cursors to be
used simultaneously without ambiguity [6].

manner. As tracking technologies improve, surface

modeling assumptions can be reduced improving
robustness. With the emerging flexible display and portable
projection technologies, foldable designs may become
practical in portable consumer electronics in the near future.
ACKNOWLEDGMENTS

This work was supported in part by grants from General

Motors, the Intel Research Council, and the National
Science Foundation under Grant IIS-0713509
REFERENCES
1. Badyopadhyay, D., Raskar, R., Fuchs, H., Dynamic Shader Lamps:
2.
3.

CONCLUSION

In this paper, we explored the concept of interactive

foldable displays by presenting four functional prototype
designs featuring different folding behaviors. The displays
react to the users movement, expansion state, orientation,
and input styli. Foldable displays offer the ability store
large display surfaces in a small volume allowing users to
quickly expand and collapse them when convenient.

4.

5.

6.

While the current state of projection technology is not yet

practical for supporting spatially augmented reality in truly
mobile scenarios, this exercise allows us to explore
interacting with high-fidelity prototypes in a low-cost

290

Painting on Real Objects. IEEE ACM Int. Symposium on Augmented

Reality (ISAR), New York, 2001.
Brown, L. et al, "A Widget Framework for Augmented Interaction in
SCAPE" Proceedings of the ACM Symposium on User Interface
Software and Technology, October 2003.
Chen, N., Guimbretiere, F., Dixon, M., Lewis, C., Agrawala, M.,
"Navigation Techniques for Dual-Display E-Book Readers" CHI
2008.
Holman, D., Vertegall, R., Altosaar, M., Troje, N., Johns, D., "Paper
windows: interaction techniques for digital paper" Proceedings of the
SIGCHI conference on the human Factors in computing systems,
2005.
Gupta, S. Jaynes, C., "Active Pursuit Tracking in a Projector-Camera
System with Application to Augmented Reality" IEEE Computer
Society Conference on Computer Vision and Pattern Recognition,
2005
Lee, J., Hudson, S., Dietz, P. "Hybrid Infrared and Visible Light
Projection for Location Tracking" Proceedings of the ACM
Symposium on User Interface Software and Technology, October,
2007.