692 PROCEEDINGS of the HUMAN FACTORS AND ERGONOMICS SOCIETY 45th ANNUAL MEETING- 2001

MEASURING ENGAGEMENT IN VIDEO GAMES: A QUESTIONNAIRE

Daniel K. Mayes and James E. Cotton

University of Central Florida

Video games continue to grown in popularity and presently account for annual revenues in the billions of
dollars. Although the technologies underlying a modern video game are well understood, the
characteristics of a game that make it a success are not. The Engagement Questionnaire (EQ) is introduced
in an attempt to capture those dimensions that are thought to influence the degree to which a user becomes
engaged while playing a video game. Our goal was to develop a metric that could be applied to a broad
range of video game genres (e.g., action, adventure, strategy, sports) and video game users (e.g.,
expert/novice, male/female, young/old). Factor analyses of data representing 243 participants suggested
the existence of five stable factors among 46 questionnaire items. These dimensions were labeled Interest,
Authenticity, Curiosity, Involvement and Fidelity. Though further empirical testing is necessary, we expect
that the questionnaire will prove to be a useful tool for the appraisal of video games, as well as for the
identification of critical differences between those who play them.

INTRODUCTION Usoh, & Steed, 1994). Unfortunately, presence metrics were

Background
Owing to vastly improved computer hardware and
software, modern video games are capable of delivering
highly realistic, compelling experiences to their players. As a
result, game production has grown into a multi-billion dollar
industry - one that presently eclipses the motion-picture
industry. While consumer demand for video games remains
strong, an explanation for the demand is not so clear. What
causes people to choose one game over another and what
elements of a game contribute to its appeal? Is it simply a
function of the improved realism (e.g., graphics and
interactivity), or are there other, equally important variables?
While successful marketing sometimes predicts success, an
understanding of the nature of games, and the psychological
and behavioral qualities of those who choose to play them,
may be even more relevant. For example, reliable data
suggesting relationships between player and game variables
would be useful to the entertainment industry as a means to
enhance success rates of new releases. In addition, for
military agencies that are presently considering the use of
commercial video games to support their training agendas,
research relating game design to transfer-of-training and
soldier performance could be equally valuable. Unfortunately,
the manner in which a game's entertainment value is currently
assessed is limited to items such as expert feedback, marketing
surveys and game sales figures. What is lacking are
systematic, empirical measures that would provide more direct
and reliable data about a game's characteristics,
Instead of relying on intuition-based designs, what is
needed is an empirical method of predicting the success of a
particular game design, across a range of unique users. In
order to make predictions, however, we must first be able to
measure the quality of a particular game along a variety of
dimensions. There are currently no published metrics to serve
this purpose. The nearest equivalents are questionnaires
developed to measure the construct of presence, or the
subjective sense of being physically situated in the
environment specified by a simulation (Heeter, 1992; Slater,
not developed with video games in mind. For example, the
Presence Questionnaire (Witmer & Singer, 1998), a
commonly used metric, was developed by Army research
scientists for the purpose of measuring the quality of a virtual
environment (VE) in military training scenarios. As a result,
the Presence Questionnaire focuses largely on how natural, or
realistic the VE and VE-interface is. This approach, however,
is not necessarily relevant for an entertainment simulation
such as a video game. For example, many game themes are
founded in fantasy, with a story occurring in a distant world
with weapons and creatures only known by science fiction.
Measurement for such attributes would not be found on the
Presence Questionnaire.
Engagement
Mayes, Cotton, and Sims (2000) have proposed a
construct labeled "engagement" as appropriate for determining
the quality of a video game experience. Engagement is
defined as how fun, involving, and motivating a task is. The
construct was conceived in consideration of the outcome that
is desired most by game developers: an enjoyable game. It is
hypothesized that enjoyable games lead to strong product
sales, recommendation of the game to others, and longer and
more frequent play by the end user.
The Engagement Questionnaire (EQ) introduced here
was created in an attempt to capture those dimensions that are
thought to influence the degree to which a user becomes
engaged while playing a video game. The initial goal was to
develop a metric that could be applied to a broad range of
video game genres (e.g., action, adventure, strategy, sports)
and video game users (e.g., expert/novice, male/female,
young/old).
Constructs Targeted for the Questionnaire
Upon its initial administration, the EQ questionnaire
consisted of 55 items that were derived from various sources.
While the construct "engagement" incorporates many of the
elements thought to contribute to presence (such as
 PROCEEDINGS of the HUMAN FACTORS AND ERGONOMICS SOCIETY 45th ANNUAL MEETING- 2001 693

"authenticity"), it also considers those dimensions considered
to be relevant to intrinsic motivation and reward. According
to Malone (1981), these include the dimensions of challenge,
fantasy, and curiosity. These additional dimensions help to
address those content items (e.g., story, theme, objectives) that
are integral to most modem video games, yet are absent in
existingpresencemeasures.
The majority of questions relating to "authenticity"
(e.g., level of interactivity, naturalness, ease of use, and
consistency) were adapted from the Presence Questionnaire
(Witmer & Singer, 1998). Other questions, including several
relating to "involvement" and "fidelity" were also adapted
from the Presence Questionnaire. The remaining questions
related to "curiosity," "fantasy," and "interest" were
developed exclusively by the authors based on a review of the
psychological literature, leading publications that track the
latest video game design techniques, pilot data from focus
groups, and personal experiences with playing video games. It
is proposed that a relationship exists between and among these
dimensions, andengagement.
They responded to each EQ item on a 7-point Likert scale with
labels for the extreme and mid points of the scale. In addition,
each participant also provided biographical data.
Procedure
Participantscompletedthe materialnearthe endof a
class period and returned the completed questionnaire
immediately. Two EQs and a biographical data form were
provided for each participant. The respondents were asked to
recall both their favorite and least favorite video game and
then answer identical questions about each game.
Respondents were also instructed to provide the title of each
game. Separating the two copies of the EQ was a biographical
data form. Question and form order (either for favorite or
least favorite game) were counterbalanced to control for order
effects. Participants were debriefed and received extra credit
upon return of the survey material.
RESULTS

METHOD Before factor analysis could proceed, a dependent

Participants
All participants were undergraduate students enrolled
in introductory psychology courses at a large, southeastern-
state university. Extra credit was exchanged for their
participation. Data from the 243 participants (94 males and
149 females) were used for the analyses,
Measures
All participants completed two versions of the EQ,
one each for their favorite and least favorite game.
samples t-test was performed to determine which of the 55
questionsdidnotdistinguishbetweenfavoriteandleast
favorite game. The results indicated that nine questions did
not meet the criterion alpha level of.001. These questions
were excluded from further analyses but are described in the
discussion.
Exploratory factor analysis using varimax rotation
was performed on the 46 remaining items using SPSS
Windows 10.0. Checks for outliers, multicollinearity, and
factorability of the correlation matrices did not reveal
deviations from acceptable limits. Tabachnick and Fidell
(1996), among others, point out the benefits, limitations and
applications of the statistical techniques utilized in this study.

Table 1. Factor Ioadings for exploratory factor analysis using varimax

rotation
Item F1
a F2 F3 F4 F5
23 .541
28 .637
29 .666
30 .657
32 .623
33 .554
37 .577
38 .583
41 .679
46 .595
54 .523
694 PROCEEDINGS of the HUMAN FACTORS AND ERGONOMICS SOCIETY 45th ANNUAL MEETING- 2001

Table I (Continued). FactorIoadings for exploratoryfactor analysis using

varimax rotation
Item F1a F2 F3 F4 F5
1 .768
2 .747
3 .831
4 .607
7 .657
12 °563
14 .619
5O .609
10 .5O9
24 .673
25 ,587
34 .645
35 .679
36 .719
39 .644
40 ,699
15 .648
16 .603
17 ,703
18 .672
20 ,581
21 .556
27 -.662
55 .512
5 .719
6 ,776
8 .644
9 .674
10" .547
11 .626
19 .599
% of Variance 14,67 14.27 13,83 12.61 11.55

* Item appears in F3 and F5

a Suggested factor labels: F1 = Interest, F2 =Control, F3 = Curiosity, F4 = Involvement, and F5 - Fidelity

Six factors were extractedfollowinganalyses. After DISCUSSION

rotation, one factor consisting of two seemingly unrelated
questions was excluded from further investigation. Overall,
variables were well defined by this factor solution,
Communality values tended to be at reasonable levels. A cut
of o50was used for inclusion of a variable in interpretation of
a factor. As such, three variables failed to load on any factor,
and one variable loaded highly on two factors. A second
review of the factors using a cut of.40 resulted in multiple
complex variables that were less defined by the factor
solution. Because of this, the original cutoff of .50 was used
for interpretation. Loadings on factors and percent of variance
accounted for are shown in Table 1o Loadings under .50 are
not shown. Interpretive labels are suggested for each factor in
a foomote and are discussed further below.
The present factor-analytic approach appears to have
provided the information necessary to initiate an empirical
approach to video game engagement. Nine questions of the
original EQ did not discriminate between favorite and least
favorite games and were not included as a part of the factor
analysis. The results of the factor analysis on the remaining
46 items suggested the existence of five stable factors.
Additionally, one factor was dropped because it was under-
identified and unstable across solutions. Below is a description
of each of the factors as well as a brief account of the
questions that did not contribute to engagement.
 PROCEEDINGS of the HUMAN FACTORS AND ERGONOMICS SOCIETY 45th ANNUAL MEETING- 2001
Factor 1. Labeled "Interest"
Items in this factor addressed subjective impressions
of how interesting, appealing, and exciting the content of the
game was. Questions focused on the general theme,
objective(s), characters, social structure, and objects in the
computer-generated environment,
Factor 2. Labeled "Authenticity"
The majority of items that loaded heavily on this
factor had been derived and modified from the Presence
Questionnaire (Witmer & Singer, 1998). The central theme of
this factor included the degree to which events in the game
could be controlled, the responsiveness of the game to actions
that were initiated by the user, and the ease/naturalness with
which one could move or manipulate objects in the game.
Usability of hardware interfaces, as well as the degree of
interactivity built into the game would likely contribute to
scores in this factor.
Factor 3. Labeled "Curiosity"
This factor included items about one's interest with
exploration of the scene and its characters. These items
seemed to cluster around the concept of curiosity, which can
be defined as the intrinsically motivating desire for
information (Lowenstein, 1994). Since previous research
indicated that humans seek out environmental variability, the
desire for novel and changing stimuli in a video game
interaction is not surprising (Jones, Wilkinson, & Braden,
1961).
Factor 4. Labeled "Involvement"
This factor was comprised of items that asked
directly, "Was the game fun?" and "How involved were you in
the game?" Other items inquired about the degree of focus a
user experienced, as well as preferences about length of play.
It is interesting that questions conceming focus and
involvement were grouped with others asking about how
much fun and excitement was experienced.
Factor 5. Labeled "Fidelity"
The fidelity factor consisted of questions regarding
the graphical and auditory components of the game. The
realism, familiarity, and consistency of the graphical and
auditory components were important aspects of this factor,
Items included in this factor asked whether one could you
easily identify sounds and graphics, as well as how consistent
were the game's images and sounds. Hardware issues such as
image resolution and sound quality, as well as
design/modeling issues such as level of scene detail, are
expected to be determinants of a game's perceived fidelity.
695
Questions Not Included in the Factor Analysis
As previously mentioned, nine questions failed to
reach the significance criterion of 12< .001 for the t-test
between liked and disliked games and were excluded from the
factor analysis. The majority of these items included
questions regarding how difficult/challenging the game was
(e.g., "How would you rate the game's difficulty level?,"
"Were there times that you felt overwhelmed by the amount of
information that you had to remember in order to successfully
complete some objective?").
Several other excluded items were related to what
Witmer and Singer (1998) referred to as control factors (e.g.,
"How much did the game's interface (i.e., the controls,
display, layout, etc.) interfere with or distract you from
playing the game?, and "How much delay did you experience
between your actions and expected outcomes?"). Finally, one
question of a general nature did not satisfy the inclusion
criterion: "How interested are you in video games?"
FUTURE RESEARCH
The purpose of the developing the Engagement
Questionnaire was to create a tool that could be used to
quickly and inexpensively assess salient information about
video game interactions. Five factors were identified that
could be labeled along the dimensions of Interest,
Authenticity, Curiosity, Involvement, and Fidelity.
Interestingly, "challenge" was not represented although we
had hypothesized that this construct would be particularly
important. Though it may be reasonable to expect that these
factorsaffectvideogameengagement, considerable empirical
research is necessary prior to drawing such a conclusion.
In addition to assessing existing and future video
games, the questionnaire can also be used to identify critical
differences between those who buy and play video games
frequently, and those with little video game interest and/or
experience. Finally, a number of gender issues can be studied
using the questionnaire, e.g., "What makes a game interesting
to females, and to males?" We have already begun this line of
research, and will present those results at future meetings.
REFERENCES
Heeter, C. (1992). Being There: The subjective
experience of presence. Presence, l(2), 262-271.
Jones, A., Wilkinson, J.J., & Braden, I. (1961). Information
deprivation as a motivational variable. Journal of
Experimental Psychology, 62, 126-137.
Lowenstein, G. (1994). The psychology of curiosity: A
review and reinterpretations. Psychological Bulletin,
116(1), 75-98.
Malone, T.W. (1981). Toward a theory of intrinsically
motivating instruction. Cognitive Science, 5(4),
333-369.
696 PROCEEDINGS of the HUMAN FACTORS AND ERGONOMICS SOCIETY 45th ANNUAL MEETING- 2001

Mayes, D., Cotton, J., & Sims, V. (2000). A conceptual model

of engagement in video games. Presented at the First
Annual Florida Human Factors and Ergonomics
Student Meeting, Orlando, FL.

Slater, M., Usoh, M., & Steed, A. (1994). Depth of presence

in virtual environments. Presence, 3(2), 130-144.

Tabachnick, B.G. & Fidell, L.S. (1996). Using multivariate

statistics (3ra ed.). New York: Harper Collins.

Witmer, B.G., & Singer, M.J. (1998). Measuring presence in

virtual environments: A presence questionnaire.
Presence, 7(3), 225-240.