This article was downloaded by: [Escuela Sup Pol del Ejercito ]

On: 02 September 2013, At: 06:40

Publisher: Taylor & Francis
Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House,
37-41 Mortimer Street, London W1T 3JH, UK

International Journal of Human-Computer Interaction

Publication details, including instructions for authors and subscription information:
http://www.tandfonline.com/loi/hihc20

Usability Principles for Augmented Reality Applications

in a Smartphone Environment
a a a
Sang Min Ko , Won Suk Chang & Yong Gu Ji
a
Yonsei University , Seoul , Korea
Published online: 10 Jun 2013.

To cite this article: Sang Min Ko , Won Suk Chang & Yong Gu Ji (2013) Usability Principles for Augmented Reality
Applications in a Smartphone Environment, International Journal of Human-Computer Interaction, 29:8, 501-515, DOI:
10.1080/10447318.2012.722466

To link to this article: http://dx.doi.org/10.1080/10447318.2012.722466

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) contained
in the publications on our platform. However, Taylor & Francis, our agents, and our licensors make no
representations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of the
Content. Any opinions and views expressed in this publication are the opinions and views of the authors, and
are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon and
should be independently verified with primary sources of information. Taylor and Francis shall not be liable for
any losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoever
or howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use of
the Content.

This article may be used for research, teaching, and private study purposes. Any substantial or systematic
reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any
form to anyone is expressly forbidden. Terms & Conditions of access and use can be found at http://
www.tandfonline.com/page/terms-and-conditions
 Intl. Journal of Human–Computer Interaction, 29: 501–515, 2013
Copyright © Taylor & Francis Group, LLC
ISSN: 1044-7318 print / 1532-7590 online
DOI: 10.1080/10447318.2012.722466
Usability Principles for Augmented Reality Applications
in a Smartphone Environment
Sang Min Ko, Won Suk Chang, and Yong Gu Ji
Yonsei University, Seoul, Korea
Through the rapid spread of smartphones, users have access to
Downloaded by [Escuela Sup Pol del Ejercito ] at 06:40 02 September 2013
many types of applications similar to those on desktop computer
systems. Smartphone applications using augmented reality (AR)
technology make use of users’ location information. As AR appli-
cations will require new evaluation methods, improved usability
and user convenience should be developed. The purpose of the cur-
rent study is to develop usability principles for the development
and evaluation of smartphone applications using AR technology.
We develop usability principles for smartphone AR applications
by analyzing existing research about heuristic evaluation methods,
design principles for AR systems, guidelines for handheld mobile
device interfaces, and usability principles for the tangible user
interface. We conducted a heuristic evaluation for three popularly
used smartphone AR applications to identify usability problems.
We suggested new design guidelines to solve the identified prob-
lems. Then, we developed an improved AR application prototype
of an Android-based smartphone, which later was conducted a
usability testing to validate the effects of usability principles.
1. INTRODUCTION
Augmented reality (AR) is a view of a real-world envi-
ronment that is modified by a computer. It is a subset of
virtual reality (VR) but differs from VR in that it offers a
greater sense of realism to its users. The difference between
AR and VR is that AR overlaps virtual information with reality,
whereas VR substitutes reality. This technology offers informa-
tion delivery, acquisition, and application by observing objects.
Early AR-related technologies were developed and studied for
use with industry, military readiness, surgery training, com-
puter games, and computer-supported collaboration. Research
on AR-related technology is ongoing as the spread of small
handheld devices and smartphones increases (Azuma et al.,
A preliminary version of this article was published in Korean in
the Journal of Korean Institute of CALS/EC, 16, 35–47, 2011. This
work was supported by Basic Science Research Program through the
National Research Foundation of Korea funded by the Ministry of
Education, Science and Technology (2012R1A1A2006949).
Address correspondence to Yong Gu Ji, Department of Information
and Industrial Engineering, Yonsei University, 262 Seongsanno,
Seodaemun-gu, Seoul, 120-749, Korea. E-mail: yongguji@
yonsei.ac.kr
501
2001; Billinghurst, Belcher, Gupta, & Kiyokawa, 2003; Bruns,
Brombach, & Bimber, 2008; Livingston et al., 2002; Nassar &
Meawad, 2010; Thomas et al., 2002; Wiedenmaier, Oehme,
Schmidt, & Luczak, 2003).
The latest high-end smartphones offer global positioning
systems (GPS), which allow the user to track their location,
as well as geo-magnetic sensors, G-sensors, and digital com-
passes. They also include high-performance processors and
high-resolution digital cameras and displays. Moreover, these
phones can retrieve information regarding a user’s current loca-
tion. These properties of smartphones satisfy the requirements
for an AR system, which needs to recognize objects using a
camera by detecting object properties, provide related content
by obtaining information related to the users’ location informa-
tion, and handle many types of data at the same time. Unlike
the traditional personal computer environment, the AR envi-
ronment requires new considerations for user. AR is usually
used while the user is mobile and subject to many distractions,
unlike the traditional personal computer (Chincholle, Goldstein,
Nyberg, & Eriksson, 2002). When using AR on mobile phones,
users can be distracted by disordered displays, complicated
execution, inappropriate functions, and inconsistency between
manipulation methods. However, most of these distractions are
manageable (Shneiderman, 2010).
Studies regarding AR usually include precision devices and
software technologies for special situations. However, because
these studies are led by technologies, usability researchers point
out that they lack of consideration of user convenience and
usability (Dünser, Grasset, & Billinghurst, 2008; Livingston,
2005; Nilsson & Johansson, 2006). Existing studies have usu-
ally evaluated the aspects of cognition and performance (Dünser
et al., 2008) and have not considered the special contexts of AR
environments (Ullmer & Ishii, 2000).
In this research, we studied usability principles for
smartphone AR applications based on user location infor-
mation. We then conducted heuristic evaluations using these
principles with people in Korea using smartphone-augmented
applications. Through this evaluation, we developed improved
smartphone AR application prototypes and validated results
from the heuristic evaluation by conducting usability testing.
 502
2. LITERATURE REVIEW
2.1. Augmented Reality
Concepts and Characteristics of Augmented Reality
AR overlaps virtual information with reality, whereas VR
provides three-dimensional (3D) images based on computer
information. Thus, AR provides a greater sense of realism
than VR can. Studies on AR began after Ivan Sutherland pub-
lished his 1968 article “A Head-Mounted Three Dimensional
Display.” The term “augmented reality” is reputed to have been
used for the first time in the early 1990s by Tom Caudell at
Boeing in his system development research of wire assembly
work for flight manufacturers (Caudell & Mizell, 1992).
When comparing the concept of VR to the concept of
AR, AR is closer to reality because VR can only offer vir-
Downloaded by [Escuela Sup Pol del Ejercito ] at 06:40 02 September 2013
tual information to users. There have been many arguments
for and against the efficiency of AR. In 1994, Paul Milgram
differentiated reality from virtuality. He divided AR from aug-
mented virtuality (AV) between virtuality and reality. Milgram
and Kishino (1994) distinguished “virtuality,” “reality,” and
“in the middle of virtuality and reality” by using the term the
“reality–virtuality continuum” (Milgram & Colquhoun, 1999;
Milgram & Kishino, 1994; see Figure 1). Since then, AR has
been recognized as an independent field of study.According to
the concept of the reality–virtuality continuum, users recognize
less and less information in reality as they get closer to virtual-
ity. Mixed reality (MR), which is between virtuality and reality,
can be roughly divided into two categories: AR and AV. AV is
distinguished from AR by the fact that insertion and background
in AV is reflected by the computer-generated environment.
In 1997, Ronald Azuma published a survey on AR and he
defined AR in 2001 as follows: “An AR system supplements
the real world with virtual (computer-generated) objects that
appear to coexist in the same space as the real world” (p. 34).
He also defined three representative properties of AR: the neces-
sity of combination between reality and virtuality, the necessity
of interaction in real time, and the necessity of observing the
real world in 3D. To date, the definition of AR by Milgram
and Azuma is accepted by many researchers and is the most
important standard for AR.
The concept of AR can be applied to many fields, but most
research has focused on human eyesight. Through AR systems,
FIG. 1. Reality–Virtuality Continuum (modified from Milgram & Kishino, 1994; 1999) (color figure available online).
S. M. KO ET AL.
users are provided with additional real-time 3D graphics with
an original view of reality. That is, the system provides ren-
dered virtual images to users while maintaining spatial relation
between reality objects and virtual objects followed by the
user view (Henrysson, 2007). Until the mid-2000s, AR stud-
ies remained limited to R&D and trial applications. However,
the appearance of smartphones with highly efficient cameras,
graphic-processing capabilities, wireless communication func-
tion, and GPS elevated the importance of AR studies (Föckler,
Zeidler, Brombach, Bruns, & Bimber, 2005; Miyashita et al.,
2008; Morrison et al., 2009; Nassar & Meawad, 2010).
Augmented Reality in a Mobile Environment
There is no commonly used formal definition for mobile AR.
In earlier studies, an AR system referred to a system constructed
on a laptop-based system that provided mobility to users. Later,
the definition expanded to a system constructed on a handheld
device. Nowadays the definition of mobile AR has expanded
to include other mobile devices (e.g., smartphones, tablets, and
other devices) that have AR systems (Jeon & Lee, 2011).
AR services provided in the mobile environment can be
classified based on their object-tracking methods as similar,
marker-based, and vision-based AR methods (Shin, Oh, Suh, &
Woo, 2010). The similar AR method identifies augmented loca-
tions and visualizes relevant contents by using the sensors of
a mobile device such as GPS or a digital compass. This is
the most common method, because smartphones are usually
equipped with digital sensors. The origin of mobile similar
AR is MARS (Mobile Augmented Reality System). MARS is
a mobile tour guide AR service system that provides a vari-
ety of information based on the users’ location and direction
through a Head-Mounted Display (Höllerer, Feiner, Terauchi,
Rashid, & Hallaway, 1999). Marker-based AR methods aug-
ment digital information by identifying objects and locations
using markers associated with the objects to be recognized by
the camera in a mobile device. A disadvantage of marker-based
AR is that the marker cannot be recognized by other sensors
such as digital compasses or GPS. Invisible-marker-based AR
such as that using infrared marker has been studied as a sup-
plement to current marker-based AR problems (Park & Park,
2010). Vision-based AR methods are ways to augment digital
 USABILITY FOR AUGMENTED REALITY APPLICATIONS
information by identifying the properties of objects and track-
ing them after identification. These methods have the advantage
of being able to track objects by recognizing them without a
marker but have the disadvantage of being difficult to imple-
ment because they require real-time image processing and
augmentation.
Smartphone AR applications are used in providing location-
based services (LBS), which make use of the location infor-
mation of the mobile device. Virrantaus et al. (2001) defined
LBS as “services accessible with mobile devices through the
mobile network and utilizing the ability to make use of the
location of the terminals” (p. 66). These characteristics of
smartphone AR applications have some limitations. In the
marker-based AR method, the marker should be installed addi-
tionally. In the vision-based AR method, there are some tech-
Downloaded by [Escuela Sup Pol del Ejercito ] at 06:40 02 September 2013
nical limitations such as real-time video processing and object
recognition.
One of the currently best-known applications is called Layer.
This application provides LBS through smartphones and is
used worldwide in iPhones and Android phones. This applica-
tion augments digital contents related to camera images using
location and direction information (Shin et al., 2010; Vaughan-
Nichols, 2009). Each layer refers to a function such as allowing
users to search for a coffee shop nearby. It also allows users to
create custom layers and share them with other users.
2.2. Usability of Augmented Reality Applications
Related Studies
Swan and Gabbard (2005) compiled usability research on
AR and found that only 14.3% (38 papers out of 266) mentioned
user-centered design in the perspective on human–computer
interaction (HCI) and only 7.9% (21 papers out of 266) had
conducted experiments in terms of general-use users (Swan &
Gabbard, 2005). Swan and Gabbard categorized user-centered
experiments related to AR into three categories. The first cat-
egory included experiments of which the main purpose was
to understand the recognition and perception of users’ AR
operations. The second category of experiments was those mea-
suring the task performance of users during use of specific
AR applications. The last category of experiments included
those seeking to interact and communicate with AR users.
They pointed out that there is a lack of research regarding
user interface and interaction from a user-centeric point of
view. They also pointed out the necessity for additional studies
regarding AR.
Dünser expanded on Swan and Gabbard’s work even fur-
ther. He examined the AR-related studies that had been done
up until 2007 (Dünser et al., 2008). Only 10% of 161 stud-
ies had conducted user-performed AR experiments. This review
also found that there were only 41 studies on actual usability,
excluding studies on recognition and perception point of view
and performance. The authors pointed out that the lack of
studies regarding the AR user experience has led to there being
503
insufficient training regarding how to evaluate AR experiences,
how to design experiments, how to select and apply proper
methods, and how to analyze the results from experiments.
Usability Features of Augmented Reality Applications
We studied existing research on AR applications and cat-
egorized existing problems into four different categories. The
studies reviewed involved smartphone AR applications, which
provide services based on users’ location information and the
usability problems of these applications.
Augmented reality interacting with a small display device.
There have been many studies regarding improving usability
while reading and interacting with small-display devices (Jones,
2006). Gabbard and Swan Ii (2008) performed user tests on
information expression for implementing AR in a mobile envi-
ronment. They observed problems due to the small size of the
displays on mobile devices. They found that in the expression
of information over reality, a small display gives too much
information compared with the small size (Gabbard & Swan
Ii, 2008). To implement AR, actual reality and the augmented
digital contents should be expressed on the display. There exists
the possibility that AR may be too complex for current mobile
devices because the interface might have too many contents and
menus to manipulate.
Multimodal interface. AR is closely related to tangible user
interfaces (TUIs) while still maintaining properties of a graphic
user interface (GUI) for the purpose of information searching
and transmission (Nilsson & Johansson, 2006). For users to use
AR applications on their smartphones, they will need to acquire
reality information and track objects with their smartphone
cameras to obtain images. Therefore, consideration of various
user interfaces, including GUI and TUI, will be necessary.
Limited manipulation. According to Henrysson (2007), the
typically studied AR methods have distinguishing features.
Virtual information in these AR systems could be manipu-
lated with both hands with the use of a head-mounted display.
However, in a smartphone environment, one hand is required for
holding the device and the other hand is used for manipulating
the display (Henrysson, 2007). Therefore, it will be necessary to
develop new interaction methods for mobile devices that make
use of one-handed manipulation.
User on the go. In the process of conducting tasks such as
searching for directions, users’ recognition and attention could
be dispersed because of the properties of location-based ser-
vices described by Chincholle et al. (2002) as “user on the
go” properties. They also studied how smartphones can pro-
vide location-based services, and pointed out that mobile users
do not have sufficient patience to learn new operating systems
(Chincholle et al., 2002). Location-based services should not
obstruct the ongoing tasks (e.g., driving, walking, etc.) of users.
During these activities, time is an extremely important factor to
the user. Therefore, the manipulation of location-based services
should be simple.
 504 S. M. KO ET AL.

3. DEVELOPMENT OF USABILITY PRINCIPLES FOR
AUGMENTED REALITY APPLICATION
In this study we developed usability principles for
smartphone AR application and conducted a heuristic eval-
uation to identify usability problems. Then, we developed a
prototype and conducted a usability testing to validate the
effects of usability principles. The research framework of this
study is illustrated in Figure 2. The research framework can be
divided into two phases.
In Phase 1, we collected usability principles through the lit-
erature review and conducted an expert meeting to discuss the
collected usability principles. Next, we classified usability prin-
ciples through the principle component analysis. In Phase 2, we
conducted heuristic evaluation of smartphone AR applications
using the classified usability principles developed in Phase 1.
Downloaded by [Escuela Sup Pol del Ejercito ] at 06:40 02 September 2013
applications as discussed in section 2.2. To do so, we collected
usability principles by reviewing and analyzing existing studies
in order to develop suitable usability principles.
The purpose of this study is to develop suitable usability
principles for smartphone AR application services based on
users’ location information. We collected existing usability
principles from the studies of Atkinson, Bennett, Bahr, and
Nelson (2007); Dünser, Grasset, Seichter, and Billinghurst
(2007); Gong and Tarasewich (2004); and Kim et al. (2007).
See Table 1. Atkinson et al. (2007) reorganized the heuristic
evaluation methods of four authoritative researchers (Nielsen,
Shneiderman, Tognazzini, and Tufte) and developed a mul-
tiple heuristics evaluation table. They identified overlap and
combined conceptually related concepts in order to simplify a
comprehensive set of heuristics. We considered this study as

Then, we implemented an improved prototype of a smartphone
AR application, which later was conducted a usability testing.
3.1. Usability Principles for Augmented Reality
Application
In this study, we present principles for new forms of usability
evaluation by considering the properties of smartphone AR
an adequate preliminary study of heuristic evaluation involv-
ing smartphone AR and usability principles. In our study, we
selected 12 usability principles, including software–user inter-
action, learnability, and cognition facilitation (Atkinson et al.,
2007).
Dünser et al. (2007) investigated general HCI principles
related to AR application design. They found that AR-related
studies have been primarily technology driven and may focus

FIG. 2. Research framework. Note. AR = augmented reality.

References
Atkinson et al. (2007)
Dünser et al. (2007)
Gong and Tarasewich
(2004)
Kim et al. (2007)
TABLE 1
Collected Usability Principles
Usability Principles
Cognition facilitation, Consistency, Defaults, Error management, Graphic design, Help and
documentation, Learnability, Navigation and exiting, Software-user interaction, System-real
world match, System-software interaction, User control and software flexibility
Affordance, Error tolerance, Flexibility in use, Learnability, Low physical effort, Reducing
cognitive overhead, Responsiveness and feedback, User satisfaction
Allow for personalization, Consistency, Design dialogs to yield closure, Design for “top-down”
interaction, Design for enjoyment, Design for limited and split attention, Design for multimodal
interfaces, Design for multiple and dynamic contexts, Design for speed and recovery, Enable
frequent users to use shortcuts, Error prevention and simple error handling, Offer informative
feedback, Reduce short-term memory load, Reversal of actions, Support internal locus of control
Accuracy, Arrangement, Attractiveness, Consistency, Controllability, Direct manipulation,
Durability, Ease, Effectiveness, Efficiency, Error indication, Familiarity, Feedback, Graspability,
Learnability, Match between system and real world, Pleasantness, Predictability, Prevention,
Safety, Simplicity, Size, Subject satisfaction, Tolerance principle, User control
 USABILITY FOR AUGMENTED REALITY APPLICATIONS
excessively on hardware and software issues. They suggest that,
to develop successful AR systems, studies need to be properly
focused on user-centered design principles as well as technical
issues. They searched various fields for HCI design principles
and guidelines to improve AR usability and chose appropri-
ate design principles for AR systems based on these findings.
They then validated their studies by providing existing studies
and examples that explain the relationship between the design
principles and properties of AR systems. For these reasons, the
Dünser et al. (2007) study is suitable to present adequate pre-
liminary data for researchers wanting to study the usability of
AR applications. In this study, we selected eight usability prin-
ciples, including affordability, reducing cognitive overhead, low
physical effort, and so on (Dünser et al., 2007).
Gong and Tarasewich (2004) discussed the features and lim-
Downloaded by [Escuela Sup Pol del Ejercito ] at 06:40 02 September 2013
itations of mobile device interfaces and compared them to a
desktop environment. They presented guidelines for a hand-
held mobile device interface based on desktop user interfaces
and research on mobile user interfaces and usability. These
design guidelines are helpful to HCI researchers and practition-
ers who are working on mobile usability and interface design.
Their study is a precursor to our study on the usability of
smartphone AR applications in a mobile environment. For our
study, we selected 15 usability goals, including “enable fre-
quent users to use shortcuts,” “offer informative feedback,”
and “design dialogs to yield closure” (Gong & Tarasewich,
2004).
TUIs imitate basic human senses and behavior, including
touch and feel (Ishii, 2008). AR connects digital information
to objects between actual space and virtual space and con-
trols digital information through the use of real objects. When
considering these properties, TUI is an interface adequate for
AR. The gesture, which expands or reduces objects on a touch
screen, is a representative example of interaction with a TUI.
Kim et al. (2008) suggested usability principles and an eval-
uation framework for TUIs to evaluate usability and identify
design issues of TUIs. In their study, the researchers reviewed
existing usability-related studies and suggested TUI-adequate
principles through retrospective analysis. Their study is suit-
able to present suitable preliminary data for considering the
properties of TUIs for use with smartphone AR. In our study,
we selected 25 usability principles, including user control,
controllability, and feedback (Kim, Kim, Chio, & Ji, 2008).
We conducted an expert meeting to discuss standards for
61 usability principles from Atkinson et al. (2007), Dünser
et al. (2007), Gong and Tarasewich (2004), and Kim et al.
(2008). We selected 61 usability principles that directly impact
the properties and performance of smartphone AR applica-
tions. We applied the following standards to these 61 selected
usability principles to decide on the final usability principles.
We selected and integrated these usability principles according
to deletion standards, which depended on whether the principles
were objective or subjective and whether they were repeti-
tive (see Table 2). Finally, we selected 22 usability principles
505
TABLE 2
Deleted and Integrated Usability Principles
Availability, Consistency, Context-based, Defaults, Direct
manipulation, Enjoyment, Error management, Exiting,
Familiarity, Feedback, Help and documentation, Hierarchy,
Learnability, Low physical effort, Multimodality,
Navigation, Personalization, Predictability, Recognition,
Responsiveness, User control, Visibility
by reviewing the relationships among these principles. In the
example of error management, users could easily perform
supported tasks if predictable errors were prevented. Error man-
agement directly affects work performance and should be part
of selection standards. Next, we developed a classification sys-
tem for usability principles. First, we developed an interrelation
matrix to prove relationships between each usability principle.
If there was a relationship between a pair of usability principles,
it was indicated with a 2. If the relationship was ambiguous, this
was indicated with a 1. A 0 indicated that no relationship existed
between a pair of usability principles. Ten experts who each had
at least 2 years of experience in UI and UX fields participated.
The result of this process is shown in Table 3.
In this study, each factor included principles with factor
loading of at least 0.6. The low physical effort (factor loading
<0.28) principle was accepted by a majority of the partici-
pants. It was agreed that this factor it is essential and should be
included due to the properties of AR applications, which must
be controlled by smartphone users.As a result of the analysis
of the main factors, usability principles were classified into five
different groups: user-information, user-cognitive, user-support,
user-interaction, and user-usage (see Figure 3). The definitions
of these groups are as follows.
1. User-information: The user-information group consists of
principles related to providing information to users such as
visual information, classified menu structure, and familiarity
with expression.
2. User-cognitive: The user-cognitive group consists of princi-
ples related to cognitive aspects required for users to mini-
mize memory loads, react as expected, and learn applications
easily.
3. User-support: The user-support group consists of princi-
ples related to user support: providing useful information,
reducing errors, handling, and personalizing.
4. User-interaction: The user-interaction group consists of
principles related to interaction between users and appli-
cations, such as providing feedback with minimum
manipulation.
5. User-usage: The user-usage group consists of principles
related to actual usage, which include reactions appropriate
for surrounding situations and methods to use or stop the
applications easily.
 506 S. M. KO ET AL.

TABLE 3
Results Obtained From a Principal Component Analysis With
Varimax Rotation
Factors

Principles 1 2 3 4 5
Multimodality .884
Enjoyment .819
Familiarity .817
Visibility .789
Hierarchy .681
Defaults .619
Recognition .905
Predictability .886
Downloaded by [Escuela Sup Pol del Ejercito ] at 06:40 02 September 2013

Learnability .880
Consistency .853
Error management .870
Help and .853
documentation
User control .744 FIG. 3. Structured usability principles.
Personalization .652
Feedback .925 element follows established usability principles (Nielsen &
Direct manipulation .707 Mack, 1994). There are some studies that validate effectiveness
Responsiveness .701 of heuristic evaluation, organize usability principle, and con-
Low physical effort .282∗ duct a heuristic evaluation with experts to suggest applicability
Context based .822 on the result of the study (Desurvire, Caplan, & Toth, 2004;
Exiting .816 Hvannberg, Law, & Lárusdóttir, 2007; Kostaras & Xenos, 2007;
Navigation .603 Kurniawan & Zaphiris, 2005; Sutcliffe & Gault, 2004; Tan,
Availability .603 Liu, & Bishu, 2009; Tang, Johnson, Tindall, & Zhang, 2006).
∗
We developed usability principles specialized for the evalu-
Remaining principles after adjustment. ation of smartphone AR applications. As Law and Hvannberg
(2004) pointed out, expertise is required to avoid omitting
The usability principles classification system is shown in problems or having failed deliveries when the usability eval-
Table 4. uation is completed by someone unfamiliar with the process
(Law & Hvannberg, 2004). Therefore, it is necessary to classify
heuristic evaluations using general-use heuristic evaluations.
3.2. Validation for the Developed Usability Principles In this study, we defined evaluations that were found by experts
From the obtained usability principles from this research, we to be “heuristic evaluations.” Usability principles refer to prin-
investigated the usability problem of smartphone AR applica- ciples developed for the evaluation of smartphone AR applica-
tion based on heuristic evaluation. Moreover, we proposed a tions. We also conducted heuristic evaluations of smartphone
prototype by applying improvements from usability problem AR applications using the 22 usability principles developed in
obtained by heuristic evaluation. Then user-based evaluation the previous section.
was conducted to validate the research. The validation of this For the heuristic evaluation, we selected three applications
research was done based on previous research methodology that provide services based on users’ location information.
which mostly conducted heuristic evaluation and user-based These three applications use AR technology now in use in
evaluation of the improvements obtained from the evaluation Korea. The applications have been developed for both Android
(Allen, Currie, Bakken, Patel, & Cimino, 2006; Rau & Liang, and iOS. They are all recently launched, popular applications
2003). in Korea that provide various functions based on users’ location
information. These three applications are free downloads from
Heuristic Evaluation With Developed Usability Principles Android Market (Google Play) or iTunes. They provide search
Heuristic evaluation is a usability inspection method that services, share information, and provide AR functions based on
helps to identify usability problems. This is an informal method search information. Some applications provide SNS functions
that involves usability specialists’ judge where each dialogue such as finding and recommending related people through
 USABILITY FOR AUGMENTED REALITY APPLICATIONS 507

TABLE 4
Usability Principles Developed for the Evaluation and for the Development of AR Application
Principles Definition
User-Information Defaults The initial establishment should be operated easily by the users. Also, the
frame designating the input space and the instances related to the form of
input should be provided.
Enjoyment An aesthetic design including colors should be used to provide exciting
experiences to users.
Familiarity Not only familiar metaphors and icons but also user-centered languages
should be used.
Hierarchy If the quantity of information is large, the information should provide a
phased design to the users to make it easier to utilize.
Multi-modality A modality such as sound as well as a visual screen should be provided
when information is provided.
Downloaded by [Escuela Sup Pol del Ejercito ] at 06:40 02 September 2013

Visibility The graphic factors should be designed properly.

User-Cognitive Consistency The generally used terms and interfaces should be maintained consistently
to prevent confusion.
Learnability The functions and the features of the application should be effective and
easy to learn for users.
Predictability The functions and the features of the application should be effective and
easy to learn for users.
Recognition Necessary information should be provided properly so users don’t have to
use short-term memory.
User-Support Error management The errors that occur while using applications should be supported by the
method of prevention and solution.
Help and documentation Appropriate Help should be provided to support user applications.
Personalization The interface should be easy enough to be modified by the users’ tastes and
specialties.
User control It should make the users feel that they are controlling the system and the
system is responding to their actions.
User-Interaction Direct manipulation When the users control the device, the information appearing on the screen
and the action of the users should be matched intuitively.
Feedback The sequence of the process and the state of system should be consistently
provided to the users.
Low physical effort It should minimize the effort of operating the application and the tiredness
of the users.
Responsiveness It should react quickly to the action of the users.
User-Usage Availability The application should have rapid initial operating speed and the previous
working state and options should remain the same after reoperation.
Context-based The user-interface should be designed considering various kinds of
environments and designed to correspond to the using environments.
Exiting To stop working or to go back to previous working parts should be easy.
Navigation The users should be allowed to navigate freely on the application when they
are performing their work.

pattern analysis. They also provide linkage to smartphones
and social commerce. According to Android Market, these
three applications have been downloaded at least 1 million
times and possibly up to 10 million times. This study focused
on conducting a heuristic evaluation of the functions and
properties of popularly used AR applications. We selected three
applications for Android and iOS (OVJET, ScanSearch, and
Qook Town).
According to previous studies three to five evaluators were
sufficient to find the majority of usability problems (Faulkner,
 508 S. M. KO ET AL.

2003; Nielsen & Landauer, 1993). With this in mind, we
choose seven participants (five male, two female; average age =
29.1) who have used smartphones and had at least 2 years
of work experience in the field of UI design or usability
evaluation.
We applied usability principles specialized for smartphone
AR environments (domain specific). Experts conducted
heuristic evaluation using an evaluation form that included
the seven attributes suggested by Cockton and Woolrych in
2001. Prior to the evaluation, we carried out an orientation so
the experts would understand each usability principle and the
evaluation form (see Table 5).
TABLE 5
Downloaded by [Escuela Sup Pol del Ejercito ] at 06:40 02 September 2013
As a result of heuristic evaluation, we drew 208 prob-
lems from the three applications. After exclusion of duplicated
problems, 96 problem lists remained. The importance of each
problem was estimated by scoring its seriousness: 3 (serious), 2
(normal) and 1 (trivial). The top problems of the applications in
order of importance are as shown in Table 6.
The three applications had six common problems out of
the top ten problems. The six common problems are shown in
Table 7.
In this study, we presented design guidelines to improve
the problems analyzed from the experts’ problem lists. These
design guidelines were made based on the cause of the prob-
lems along with context and related usability principles. The
completed guidelines are as in Table 8.
The purpose of this study was to develop usability principles

Forms for Organizing the Problems for smartphone AR application properties based on previous
usability-related studies. To accomplish this, we reviewed pre-
1 Divisions of the problem numbers
vious HCI usability studies and smartphone environments. Our
2 Simple explanation of the problems
study also referred to studies that applied previous usability
3 Difficulties expected for the users
principles to mobile devices. To understand the TUI properties
4 Specific problem situations (context)
of AR, we reviewed studies of TUI usability and smartphone
5 Problem factors
GUIs. We developed usability principles that will take on two
6 Usability principles which should be applied
roles in the application development process. One role will
7 Seriousness of problems (serious, normal, trivial)
be to determine the direction of smartphone AR application

TABLE 6
Final Usability Problem Lists and Obtained Measurement of Importance Drawn From Heuristic Evaluation
Applications

No. Problem Lists A B C

1 Difficulties of selection and expression on icons and text occur when presenting the result of 15 17 11
searching positional information because the application shows duplicate results on the screen.
2 Only limited essential information is provided to the users when they see the result of searching 8 10 8
positional information.
3 Confusion can result due to lack of explanation of icons which are unfamiliar to the users. 9 9 5
4 When the users search for positional information, they are not provided with functions to 7 6 8
expand their search range. They are also provided with insufficient options.
5 Sometimes explanation about a menu is located in other place like ‘options’ and sometimes there 3 8 5
is a lack of explanation about the menu. These problems make beginner users uncomfortable.
6 When the application is expected to be operated by one hand (especially the left hand), menus 2 5 8
are located on one side (e.g., left side) and icons are too small to operate.
7 Complicated technical terms make the users frustrated and cause misunderstanding when they — 10 —
are trying to set the options.
8 When the users want to obtain additional information by adjusting their smartphones to the — — 10
special object, their smartphones don’t react as the users intended or they require a deliberate
process to obtain additional information.
9 A method of going back to the previous screen without selecting any themes in the theme menu — — 9
does not exist.
10 When selecting the category for search location, items which could be plurally selected could be — — 8
mixed up with items which could not be plurally selected. This problem can lead to a
cancellation of the users’ operation and cause confusion to the users.
 Downloaded by [Escuela Sup Pol del Ejercito ] at 06:40 02 September 2013

TABLE 7
Detailed Information: Common Factors in the Top 10 Problems
The Reason for the Related Usability
Problem Description Expected Difficulties Problem Context Problem Principles
Problem 1 – Shows the duplicated It is difficult for users to When there are more than Providing information Context-based, Error
Duplicated results on the screen. confirm or tab the two results in the same without considering the management, Hierarchy,
expression of hidden information. direction. complexity of Low physical effort,
information information Visibility
Problem 2 – Only limited name Tabbing is required to When the users require Insufficient options or Context-based,
Providing limited information is provided access the essential additional information functions for indicating Personalization,
information when the users see the information and from the search result additional information Visibility
result of searching for difficulties are expected (e.g., distance) (e.g., distance, grade)
positional information. when moving.
Problem 3 – Absence of explanation The users should operate When the users want to Developers used Consistency, Error
Unfamiliar icon and familiarity of icons. the application step by operate the menus or unfamiliar icons or management,
expression step because they don’t functions (such as didn’t consider the Familiarity, Help and
understand the meaning selecting search touching environment documentation,
of icons. category) which doesn’t have the Learnability,
mouse-over function. Navigation,
Recognition, Visibility

509
Problem 4 – The problem of It is difficult for the users When expanding the The application is not Availability,
Expanding the expression and function to expand the searching search range because provided with adequate Context-based,
range of searching supporting the range, find a function, or the users are not icons, functions, Familiarity,
expansion of the operate the function. satisfied with the result. expression methods or Learnability,
searching range. function for expanding Personalization,
searching range Recognition,
Responsiveness,
Visibility,
Problem 5 – Help Misallocation of Help The users have trouble When the users are Contents are insufficient Error management, Help
menu menu and problem of with finding the location curious about or want to or Help menu is located and documentation,
content. of Help menu and try a specific function. in the ‘option’ menu. Hierarchy, Learnability,
comprehending the
contents of it.
Problem 6 – Absence of functions and Menus are tilted on the When user wants to The function of moving or Context-based, Defaults,
One-handed options for one-handed one side. That makes it operate with only one magnifying the menu is Low physical effort,
operation operation. difficult to operate the hand (especially the left not provided. Personalization,
menus and makes the hand) in mobile Visibility
icons too small to environment.
operate.
 510 S. M. KO ET AL.

TABLE 8
Guidelines for Improving the Top 10 Problems
No. Problem Guideline
1 Duplicated expression Use an active way of expressing information to obtain hidden information by only
of information operating the camera.
2 Providing limited Provide additional information about what users want from the result of positional search.
information Provide it according to the number of search results or according to the user option.
3 Unfamiliar icon Design the icons with familiarity and consistency based on the analysis of widely used
expression icons. Support the users with a simple explanation about the icons.
4 Expanding the range of The function of expanding searching range on the initial screen should be provided
searching intuitively and should react promptly to user actions. It also has to provide option
storage function about what users want on the range search.
5 Help menu Considering the unfamiliarity of augmented reality, Help menu should be provided with
1 level menu and should be divided into many steps for the users to remember easily.
Downloaded by [Escuela Sup Pol del Ejercito ] at 06:40 02 September 2013

6 One-handed operation Location and size of the menus and icons should make one-handed operation possible.
Initial menus and icons should be placed where they are back-camera reachable while
adjusting the location and size of the menus and icons.

development. The other role will be to evaluate the design and aspect: how they express information to users. This prob-
usability of smartphone AR applications. lem occurs when providing too much information on a small
We conducted heuristic evaluations of the three most-used smartphone screen. For the usability testing, we developed a
smartphone AR applications in Korea by applying the devel- user interface improved prototype. This prototype was based on
oped usability principles. As a result we found problems cor- guidelines designed to solve the problem of duplicate informa-
responding with the properties of the AR applications. The tion (Problem 1) and to solve the problem of providing limited
problems we found usually involved operation of the device information (Problem 2).
with one hand or displaying too much information on a small To reduce duplicate information (Problem 1) we designed
screen. We concluded that a lack of systemic and flexible the prototype to not require any additional tapping opera-
information and options for users brought out these problems. tions. We provided bar-shaped duplicate search results as the
We also found a lack of explanation in help menus and a lack of camera spontaneously moves around the center of the screen.
cognitive support for users. We concluded that these problems To improve the problem of providing limited information
occurred because the AR applications were designed based on (Problem 2), we made a distance indicator on the bar as shown
previous application development design principles. in Figure 4. In addition, we provided a distance-indication
setting on the basic search results screen. We provided a man-
ual distance indicator to users so that they can use prototypes
Improving AR Application Prototype With
Heuristic Evaluation Results
We then attempted to validate our smartphone AR appli-
cation usability principles. We developed an improved AR
application prototype of an Android-based smartphone based on
the developed design guidelines in order to conduct a usability
testing. Twenty participants were recruited for this test using
both iOS and Android applications.
Some limitations existed with regard to developing a new
prototype including cost, time, and the proficiency level of
developers with Android SDK. Specialists drew two represen-
tative problems from the heuristic evaluation and developed
an improved prototype to accept the realistic problems previ-
ously shown and to reflect the result of the heuristic evaluation.
Among the common problems of smartphone AR applica-
tions, Problem 1 (duplicated expression of information) and FIG. 4. Distance display function of the prototype (color figure available
Problem 2 (providing limited information) have one common online).
 USABILITY FOR AUGMENTED REALITY APPLICATIONS 511

TABLE 9
Applications for the Usability Testing
Operating Screen

Providing Information About the

App Results Providing Detailed Information
A
Downloaded by [Escuela Sup Pol del Ejercito ] at 06:40 02 September 2013

Prototype

whenever they want and provide distance information when the
search result is not sufficient (Figure 4).
As shown in Table 9, we selected iOS application A, Android
application B, and the developed prototype in the environment
of Android OS for use in this research.
Usability Testing With AR Application Prototype
There were two types of task scenarios for the participants
(see Table 10). Each was designed for Problems 1 and 2, which
were drawn from the heuristic evaluation. Each scenario design
considered frequently used tasks when using the place-based
TABLE 10
Tasks for the Usability Testing
Task 1 Search the nearest () from here. Then input text of
the name and distance.
Task 2 Find () and make a phone call after confirming the
location on the map by using the detailed
information.
service AR application in a mobile environment. To control for
the ordering and learning effects, we designated different search
objects for each application. We also designed the number of
search results to be similar to make the evaluation fair for both
applications.
The questionnaire for the usability testing was divided into
two phases. The first phase of this questionnaire was admin-
istered before the experiment took place. We developed a
pretask questionnaire regarding participants’ personal back-
grounds with regard to smartphones. The second phase of the
questionnaire was designed to be asked after processing each
of two tasks. In this phase, the definitions of usability princi-
ples were established. These usability principles were related
to Problems 1 and 2 and included visibility, personalization,
context-based, error management, hierarchy, and low physical
effort. We also developed an additional questionnaire composed
of eight questions. This questionnaire included two questions
from Lewis’s After Scenario Questionnaire, which should be
processed after finishing the usability testing scenario (Lewis,
1991).
The usability testing was conducted by providing a scenario-
based task to the participant. After providing the scenario-based
 512 S. M. KO ET AL.

TABLE 11
Results Obtained From the Usability Testing
M of Rating Score (SD)

Dimension App A App B Prototype

Usability Principles Context-based 3.05 (1.19) 3.00 (0.97) 3.65 (1.35)
Error management∗ 3.10 (0.97)a,b 2.75 (1.20)a 3.90 (1.33)b
Hierarchy∗ 3.20 (1.28)a,b 2.90 (0.97)a 4.10 (1.71)b
Low physical effort∗∗ 3.25 (1.29)a 2.95 (1.05)a 4.80 (1.20)b
Personalization∗∗ 2.55 (1.23)a 2.40 (0.88)a 4.45 (0.81)b
Visibility∗∗ 3.25 (1.37)a 2.80 (0.89)a 4.70 (1.08)b
Total∗∗ 3.07 (0.90)a 2.80 (0.61)a 4.25 (0.99)b
ASQ Amount of time∗∗ 2.70 (1.22)a 2.95 (1.15)a 5.05 (1.23)b
Ease of completing∗∗ 3.50 (1.32)a 3.20 (1.20)a 5.00 (1.52)b
Downloaded by [Escuela Sup Pol del Ejercito ] at 06:40 02 September 2013

Note. ASQ = after-scenario questionnaire.

a
Mean scores are significantly different at the a = 0.01(∗ ) and a = 0.001(∗∗ ). b Mean scores are
significantly different at the a = 0.05 when the superscripts (a, b) are different.

task to the participant, the test was conducted by observa-
tion of and the questionnaire from the participant. Twenty
people participated in this test. The participants were under-
graduate students, graduate students, and company employees
(10 male, 10 female; average age = 27.05). These participants
included 12 who had previously used AR applications on their
smartphone.
Each participant was asked to fill out the pretask question-
naire on their personal smartphone background after listening to
the outline and the method of the experiment. The participants
were then asked to answer six questions based on the usability
principles related to Problems 1 and 2. They were then asked to
answer eight questions including two questions from the After
Scenario Questionnaire. While conducting this experiment, the
observer was asked to record the elapsed time and any errors
(Lewis, 1991).
The explanation of the experiment required about 10 min
on average. Progression and response took 5 min each. It took
about 40 min to finish the entire experiment. Because this was
a comparison experiment, we randomly chose the sequence of
the task progression order of application.
Results of Usability Testing
We conducted a repeated analysis of variance to analyze the
results of the questionnaire. The Greenhouse-Geisser method
was applied to revise the assumption violation of sphericity.
As a result of applying a one-way analysis of variance by using
application as an independent variable, there was a meaningful
difference in visibility, F(2, 38) = 24.51, p < .001; personal-
ization, F(2, 38) = 28.35, p < .001; error management, F(1.53,
29.323) = 6.496, p = .007; hierarchy, F(2, 38) = 6.58, p = .004;
low physical effort, F(2, 38) = 18.17, p < .001, of usability
principles and total sum, F(2, 38) = 24.93, p < .001, that
depended on application. No difference was found in the case
of context based. There were significant differences both on the
satisfaction about the usefulness of the task, F(1.56, 29.540) =
15.777, p < .001, and satisfaction about the elapsed time to
finish the task, F(2, 38) = 27.91, p < .001. After analyzing
the effects of sex and possession of smartphones, no meaning-
ful main effect was found. The average, standard deviation and
the result of analysis through the Tukey examination of each
question item are presented in Table 11.
As a result of postexamination using the Tukey test, error
management and hierarchy appeared to have meaningful differ-
ences between application B and the prototype using average
scores. However, there was no significant difference between
application A and application B or between application A
and the prototype. This result can be interpreted to mean
that the composition of application B, which is based on an
Android environment, is lower than application A or the pro-
totype, which are both based on the iOS environment. In the
case of context based, there was no significant difference
between application A and application B. In terms of task
time, satisfaction with application B appeared to be higher
than with application A. This result was expected because
there is no function to indicate distance on the search result in
application A.
For all the question items except satisfaction about the
elapsed time to complete the task, it appeared that users were
most highly satisfied with the applications in the order of
prototype, application A, and application B. There were signif-
icant differences between prototype and applications A and B
except from context based, which showed no significant differ-
ence between the three aspects. In addition, difference between
application A and B was not encounter.
 USABILITY FOR AUGMENTED REALITY APPLICATIONS
Downloaded by [Escuela Sup Pol del Ejercito ] at 06:40 02 September 2013
FIG. 5. The average time to complete task (color figure available online).
Figure 5 shows the result of the quantitative evaluation of
the time elapsed while working on tasks. On the same task,
the improved prototype showed shorter elapsed time when
compared to application A and application B. We also found
a statistically meaningful difference in the average elapsed time
between the elapsed task time of the prototype and the elapsed
task time of applications A and B.
As seen in the analysis of the usability testing, the evalua-
tion result of the prototype was better than the evaluation result
of the existing applications A and B. We applied improved
guidelines drawn from the heuristic evaluation to the develop-
ment of the prototype. In this research, we also conducted a
usability testing based on the developed smartphone AR appli-
cation usability principles. As a result, we found that existing
problems were solved and user satisfaction was improved.
4. DISCUSSION AND CONCLUSIONS
In this study, 22 usability principles were developed for
user-centered design and evaluation of AR applications on
smartphones. Heuristic evaluation was also conducted based
on these principles. We also gathered previously published
user principles and integrated and selected principles suit-
able for AR applications on smartphones. Expert evalua-
tion showed that previous design principles were inadequate
for newly developed technologies. Significant improvement
occurred through usability testing of an Android-based proto-
type using updated design guidelines. It is expected that the
usability principles suggested in this study could be used for
developing and evaluating AR applications in the future
In this research, consideration of the exterior environment
of the smartphone was excluded because the usability test-
ing was conducted only in a laboratory environment. As a
result, research on the principle of context based was limited
because the user environment wasn’t relevant to the principle.
513
Improvement was shown for all items except context based
compared to the available applications. Through this result, we
confirmed that the usability principles developed in this study
and the methods of heuristic evaluation are appropriate for AR
application evaluation. It is expected that the usability principles
that were developed in this research will be used in the process
and evaluation of development of AR applications.
This study has some limitations. First, we drew problems
from three popularly used smartphone AR applications which
are currently used in Korea. We selected the applications by
considering their download times. A limitation existed in that
the applications are only popularly used in Korea and don’t offer
English versions. Future work could conduct a heuristic evalu-
ation and usability testing on various and globally used appli-
cations. Second, we developed a prototype based on Android
circumstance and conducted a usability testing in this study.
Future work could include usability testing and heuristic eval-
uation of various kinds of smartphone device OS such as iOS
and Windows Mobile. Third, this usability testing was con-
ducted on a fixed display of smartphone. This can be a limitation
of this study because smart device circumstance is changing
rapidly. The next study should reflect various display sizes and
device types. Fourth, we only studied AR application that uses
basic information of the user location information. However,
the future work should be conducted on various types of AR
applications.
REFERENCES
Allen, M., Currie, L. M., Bakken, S., Patel, V. L., & Cimino, J. J.
(2006). Heuristic evaluation of paper-based Web pages: A simplified
inspection usability methodology. Journal of Biomedical Informatics, 39,
412–423.
Atkinson, B. F. W., Bennett, T. O., Bahr, G. S., & Nelson, M. M. W. (2007).
Development of a multiple heuristics evaluation table (MHET) to sup-
port software development and usability analysis. Paper presented at the
Proceedings of the 4th International conference on Universal Access in
Human–Computer Interaction: Coping With Diversity (pp. 563–572). July
22–27, 2007, Beijing, China.
Azuma, R. T. (1997). A survey of augmented reality. Presence-Teleoperators
and Virtual Environments, 6, 355–385.
Azuma, R., Baillot, Y., Behringer, R., Feiner, S., Julier, S., & MacIntyre, B.
(2001). Recent advances in augmented reality. Computer Graphics and
Applications, IEEE, 21, 34–47.
Billinghurst, M., Belcher, D., Gupta, A., & Kiyokawa, K. (2003).
Communication behaviors in colocated collaborative AR interfaces.
International Journal of Human–Computer Interaction, 16, 395–423.
Bruns, E., Brombach, B., & Bimber, O. (2008). Mobile phone-enabled
museum guidance with adaptive classification. IEEE Computer Graphics
and Applications, 28, 98–102.
Caudell, T. P., & Mizell, D. W. (1992). Augmented reality: An applica-
tion of heads-up display technology to manual manufacturing processes.
Proceedings of the Twenty-Fifth Hawaii International Conference on the
System Sciences, 1992, 2, 659–669.
Chincholle, D., Goldstein, M., Nyberg, M., & Eriksson, M. (2002). Lost
or found? A usability evaluation of a mobile navigation and location-
based service. Paper presented at the Proceedings of the 4th International
Symposium on Mobile Human–Computer Interaction (pp. 211–224).
September 18–20, 2002, Pisa, Italy.
 514 S. M. KO ET AL.
Cockton, G., & Woolrych, A. (2001). Understanding inspection methods:
Lessons from an assessment of heuristic evaluation. People and Computers,
15, 171–192.
Desurvire, H., Caplan, M., & Toth, J. A. (2004). Using heuristics to evaluate
the playability of games. Paper presented at the CHI’04 extended abstracts
on Human Factors in Computing Systems (pp. 1509–1512). April 24–29,
2004, Vienna, Austria.
Dünser, A., Grasset, R., & Billinghurst, M. (2008). A survey of evaluation
techniques used in augmented reality studies. Paper presented at the ACM
SIGGRAPH ASIA 2008 courses, Singapore.
Dünser, A., Grasset, R., Seichter, H., Billinghurst, M. (2007). Applying HCI
principles to AR systems design. Proceedings of the 2nd International
workshop on Mixed Reality User Interfaces: Specification, Authoring,
Adaptation (MRUI 2007) (pp. 37–42). March 11, 2007, Charlotte, NC,
USA.
Faulkner, L. (2003). Beyond the five-user assumption: Benefits of increased
sample sizes in usability testing. Behavior Research Methods, 35, 379–383.
Föckler, P., Zeidler, T., Brombach, B., Bruns, E., & Bimber, O. (2005).
PhoneGuide: Museum guidance supported by on-device object recogni-
Downloaded by [Escuela Sup Pol del Ejercito ] at 06:40 02 September 2013
tion on mobile phones. Proceedings of the 4th international conference
on Mobile and Ubiquitous Multimedia (pp. 3–10). December 8–10, 2005,
Christchurch, New Zealand.
Gabbard, J. L., & Swan Ii, J. E. (2008). Usability engineering for augmented
reality: Employing user-based studies to inform design. IEEE Transactions
on Visualization and Computer Graphics, 14, 513–525.
Gong, J., & Tarasewich, P. (2004). Guidelines for handheld mobile device inter-
face design. Proceedings of the 2004 DSI Annual Meeting (pp. 3751–3756).
November 20–23, 2004, Boston, Massachusetts, USA.
Henrysson, A. (2007). Bringing augmented reality to mobile phones. Linköping,
Sweden: Linköping University.
Höllerer, T., Feiner, S., Terauchi, T., Rashid, G., & Hallaway, D. (1999).
Exploring MARS: Developing indoor and outdoor user interfaces to a
mobile augmented reality system. Computers and Graphics (Pergamon), 23,
779–785.
Hvannberg, E. T., Law, E. L.-C., & Lárusdóttir, M. K. (2007). Heuristic
evaluation: Comparing ways of finding and reporting usability problems.
Interacting with Computers, 19, 225–240.
Ishii, H. (2008). The tangible user interface and its evolution. Communications
of the ACM, 51(6), 32–36.
Jeon, J. H., & Lee, S. Y. (2011). Standardizations for Mobile Augmented Reality
Technology Electronics and telecommunications trends. Electronics and
Telecommunications Research Institute, 26, 61–74.
Jones, M. (2006). Mobile interaction design. Hoboken, NJ: Wiley & Sons.
Kim, H. J., Kim, M. H., Chio, J. K., & Ji, Y. G. (2008). A study of evaluation
framework for tangible user interface. Proceedings of the 2nd International
Conference on Applied Human Factors and Ergonomics AHFE2008. July
14–17, 2008, Las Vegas, USA.
Kostaras, N., & Xenos, M. (2007). Assessing educational web-site usability
using heuristic evaluation rules. Paper presented at the 11th Panhellenic
Conference on Informatics with International Participation, Patras, Greece.
Kurniawan, S., & Zaphiris, P. (2005). Research-derived web design guidelines
for older people. Proceedings of the 7th International ACM SIGACCESS
conference on Computers and Accessibility, 129–135.
Law, E. L.-C., & Hvannberg, E. T. (2004). Analysis of strategies for improving
and estimating the effectiveness of heuristic evaluation. Proceedings of the
Third Nordic Conference on Human–Computer Interaction, 241–250.
Lewis, J. R. (1991). Psychometric evaluation of an after-scenario questionnaire
for computer usability studies: the ASQ. SIGCHI Bulletin, 23, 78–81.
Livingston, M. A. (2005). Evaluating human factors in augmented reality
systems. Computer Graphics and Applications, IEEE, 25(6), 6–9.
Livingston, M. A., Rosenblum, L. J., Julier, S. J., Brown, D., Baillot, Y., II,
J. E. S., . . . Hix, D. (2002). An augmented reality system for military
operations in urban terrain. Proceedings of Interservice/Industry Training,
Simulation & Education Conference, 89.
Milgram, P., & Colquhoun, H. (1999). A taxonomy of real and virtual world
display integration. Mixed Reality: Merging Real and Virtual Worlds, pp.
5–30.
Milgram, P., & Kishino, F. (1994). Taxonomy of mixed reality visual displays.
IEICE Transactions on Information and Systems, E77–D, 1321–1329.
Miyashita, T., Meier, P., Tachikawa, T., Orlic, S., Eble, T., Scholz,
V., . . . Lieberknecht, S. (2008). An Augmented Reality museum guide.
Proceedings of the 7th IEEE/ACM International Symposium on Mixed and
Augmented Reality, 103–106.
Morrison, A., Oulasvirta, A., Peltonen, P., Lemmelä, S., Jacucci, G., Reitmayr,
G., . . . Juustila, A. (2009). Like bees around the hive: A comparative study
of a mobile augmented reality map. Proceedings of the 27th international
conference on Human Factors in Computing Systems, 1889–1898.
Nassar, M. A., & Meawad, F. (2010). An Augmented Reality exhibition guide
for the iPhone. Proceedings of the 2010 International Conference on User
Science and Engineering (i–USEr), 157–162.
Nielsen, J., & Landauer, T. K. (1993). A mathematical model of the find-
ing of usability problems. Proceedings of the INTERACT  93 and CHI
 93 conference on Human Factors in Computing Systems, 206–213.
Nielsen, J., & Mack, R. L. (1994). Usability inspection methods (pp. 25–64).
New York, NY: Wiley.
Nilsson, S., & Johansson, B. (2006). User experience and acceptance of a mixed
reality system in a naturalistic setting—A case study. Proceedings of the 5th
IEEE and ACM International Symposium on Mixed and Augmented Reality,
247–248.
Park, H., & Park, J.-I. (2010). Invisible marker–based augmented reality.
International Journal of Human–Computer Interaction, 26, 829–848.
Rau, P.-L. P., & Liang, S.-F. M. (2003). Internationalization and localiza-
tion: evaluating and testing a Website for Asian users. Ergonomics, 46,
255–270.
Shin, C., Oh, Y., Suh, Y., & Woo, H. Y. (2010). A survey on mobile augmented
reality services and an outlook toward a sustainable content ecosystem.
Korea Information Science Society Review, 28, 43–50.
Shneiderman, B. (2010). Designing the user interface: Strategies for effective
human–computer interaction (5th ed.). Boston, MA: Addison-Wesley.
Sutcliffe, A., & Gault, B. (2004). Heuristic evaluation of virtual reality applica-
tions. Interacting with Computers, 16, 831–849.
Sutherland, I. E. (1968). A head-mounted three dimensional display.
Proceedings of the December 9–11, 1968, fall joint computer conference,
Part I, 757–764.
Swan, J. E., & Gabbard, J. L. (2005). Survey of user-based experimentation
in augmented reality. Proceedings 1st International Conference on Virtual
Reality (pp. 1–9). July 22–27, Las Vegas, Nevada, USA.
Tan, W.-s., Liu, D., & Bishu, R. (2009). Web evaluation: Heuristic evaluation vs.
user testing. International Journal of Industrial Ergonomics, 39, 621–627.
Tang, Z., Johnson, T. R., Tindall, R. D., & Zhang, J. (2006). Applying heuristic
evaluation to improve the usability of a telemedicine system. Telemedicine
Journal & E-Health, 12, 24–34.
Thomas, B., Close, B., Donoghue, J., Squires, J., Bondi, P. D., & Piekarski,
W. (2002). First person indoor/outdoor augmented reality application:
ARQuake. Personal Ubiquitous Computing, 6, 75–86.
Ullmer, B., & Ishii, H. (2000). Emerging frameworks for tangible user inter-
faces. IBM Systems Journal, 39, 915–931.
Vaughan-Nichols, S. J. (2009). Augmented Reality: No longer a novelty?
Computer, 42, 19–22.
Virrantaus, K., Markkula, J., Garmash, A., Terziyan, V., Veijalainen,
J., Katanosov, A., & Tirri, H. (2001). Developing GIS-supported
location-based services. Proceedings of the Second International
Conference on Web Information Systems Engineering, 2001, 66–75.
Wiedenmaier, S., Oehme, O., Schmidt, L., & Luczak, H. (2003). Augmented
Reality (AR) for assembly processes design and experimental eval-
uation. International Journal of Human–Computer Interaction, 16,
497–514.
ABOUT THE AUTHORS
Sang Min Ko is a Ph.D. student in the Department of
Information and Industrial Engineering at Yonsei University,
Korea. His research interests include Haptic Interface,
 USABILITY FOR AUGMENTED REALITY APPLICATIONS 515

Multisensory Interaction, Eye-Tracking, and Augmented
Reality.
Won Suk Chang received a master’s degree from the
Department of Information and Industrial Engineering at
Yonsei University, Korea, in 2011. His research interests
include Augmented Reality and Mobile Interaction Design.
Downloaded by [Escuela Sup Pol del Ejercito ] at 06:40 02 September 2013
Yong Gu Ji is an Associate Professor in the Department
of Information and Industrial Engineering at Yonsei University,
where he directs the Interaction Design Laboratory. He received
his Ph.D. in Industrial Engineering from Purdue University.
His research interests include Usability, Emotional Design,
Accessibility, and Elderly in HCI.