RESEARCH AND
Exploring the Use of an Online Quiz
Game to Provide Formative Feedback
in a Large-Enrollment, Introductory
Biochemistry Course
By Rachel Milner, Jonathan Parrish, Adrienne Wright, Judy Gnarpe, and Louanne Keenan
In our large-enrollment,
introductory biochemistry course for
nonmajors, we provide students with
formative feedback through practice
questions in PDF format. Recently,
we investigated possible benefits
of providing the practice questions
via an online game (Brainspan).
Participants were randomly
assigned to either the online game
group or the PDF-only group. Both
groups received identical practice
questions and correct/incorrect
feedback only, with no additional
detail. We believed that the game
might increase engagement with the
material and improve performance
on the examination. However,
we found that the exam scores of
the two groups did not differ. The
attitude of participants toward the
practice questions was similar for
both formats, with agreement that
the questions were appropriate
for course objectives and helped
in preparation for examinations.
However, the game users were less
likely to agree that the questions
provided useful feedback on
learning, suggesting that they had
greater expectations for the practice
questions delivered as a game.
Game users were also more than
twice as likely to state that they
wanted the other or both formats.
86
Journal of College Science Teaching
TEACHING
L
earning outcomes in higher
education may be improved
through formative feedback,
at least partly because learn-
ing is enhanced through experience,
reflection, and practice (Ericsson &
Kintsch, 1995; Kolb & Kolb, 2009;
Thatcher, 1990). That is, learning
outcomes may be influenced by
activities provided, such as a quiz
(Biggs, 2012). It is argued that an-
swering study questions creates
more robust memory traces (Glass,
Brill, & Ingate, 2008) and that re-
peated retrieval during learning is
key to long-term retention (Karpicke
& Roediger, 2007). Many studies on
the use of quizzes have reported im-
proved performance on exams (Gos-
per, 2010; Nicol & Macfarlane-Dick,
2006; Shute, Hansen, & Almond,
2008). Given this, it is increasingly
common that instructors in higher
education use quizzes as tools to
support learning. These quizzes take
a variety of forms, including paper-
based and online homework quiz-
zes or in-class Jeopardy-style games
(Benek-Rivera & Mathews, 2004).
Notably, much of the use of quizzes
in formative assessment is electronic
(Peat & Franklin, 2002) because of
the large class sizes, which are an in-
creasingly common feature in higher
education.
Although quizzes can be use-
ful learning tools, it is important to
consider the variety of factors that
influence whether they are effective
(Akl et al., 2010; Biggs, 2012; B. R.
Crisp, 2007; G. T. Crisp, 2012; Shute
et al., 2008). These include timing of
the feedback relative to the delivery
of the content; the number of attempts
to answer that a student is permitted;
and the amount of detail provided in
the feedback (Peat, Franklin, Devlin,
& Charles, 2005). Gosper (2010)
suggested that the nature of feedback
depends on the level of knowledge
required; immediate feedback is most
important for memorization and recall
of factual knowledge, and for this,
feedback should be simply correct/
incorrect. In contrast, for higher order
cognitive tasks, delaying feedback
might be more important, and when
given, informative and elaborate
feedback is more effective (Jaehnig
& Miller, 2007), especially feedback
including an explanation of how one
would determine the correct answer
(Narciss & Huth, 2006). Other factors
that might influence the effective-
ness of feedback include the length
of time that a quiz is available, the
type of quiz, and student preferences
and motivation (Chickering, 2006;
Dermo, 2009; Glass et al., 2008; Max-
well, 2010). Most critically, feedback
must be engaged with by students to
be effective, and so we must always
consider factors that affect student
engagement, including outcomes and
rewards (Anderson et al., 2001; G. T.
Crisp, 2012).
Students are generally motivated
to use online quizzes, perceiving
them as effective because they pro-
vide autonomy and fast feedback
(Dermo, 2009; Glass et al., 2008;
Maxwell, 2010). Motivation and
engagement may also be improved
through educational games (Horizon
Report, 2006) because these “can
be fun” (Hudson & Bristow, 2006).
Despite this, the available evidence
to date “neither confirms nor refutes
the utility of educational games as
an effective teaching strategy for
medical students” (Akl et al., 2010),
and there is room for further inves-
tigation of their impact. Therefore,
in this study we explored the effec-
tiveness of an online quiz game as
an alternative means for providing
students with practice questions.
Our context was a large-enrollment,
introductory biochemistry class with
a pedagogical focus on recall of basic
principles and terminology. Approxi-
mately half of the study participants
had access to practice questions via
a game, Brainspan, and the remain-
ing participants were provided with
identical practice questions via
PDF, as usual. We hypothesized that
providing the practice questions as
a quiz game might increase engage-
ment with the material, compared
with PDF only, and thereby improve
performance on the examination.
Because engagement is influenced by
individual differences (Anderson et
al., 2001; Biggs, 2012; Chickering,
2006; Krathwohl & Anderson, 2010),
we also explored student experiences
with the game. Specifically, we asked
two major research questions:
1. Is student performance in university in Canada. The course is
examinations improved by access a large-class lecture format and is
to an online quiz game? offered in each term in multiple sec-
2. What are student attitudes toward tions with different instructors. It is
the practice questions delivered team taught by experienced instruc-
in an online game? tors who follow a common syllabus
with shared materials to achieve
Methods consistent learning outcomes re-
Context (Introductory gardless of section. The course is
Biochemistry, BIOCH200) focused on the “vocabulary” of the
The context was an introductory bio- subject. Students are introduced to
chemistry course (BIOCH200) of- terms and basic concepts that they
fered at a large, research-intensive need to understand and memorize
FIGURE 1
One page of a feedback file, with the correct responses underlined.
The question files were identical to the feedback files except for the
underlining of correct answer.
Biological Membranes
1. Which of the following statements about passive and primary active transporter
proteins is FALSE?
A. They are both integral membrane proteins.
B. They both show a high degree of selectivity.
C. Both require a concentration gradient to function.
D. They both change conformation during transport.
2. Which one of the following statements is TRUE for passive transport across a bio-
logical membrane?
A. Passive transport is driven by a solute concentration gradient.
B. Passive transport is driven by ATP.
C. Passive transport is irreversible.
D. Passive transport is endergonic (requires an input of energy to occur).
E. Passive transport is not specific with respect to the substrate.
3. Which of the following determines the force that “drives” an ion through an ion
channel in a membrane?
A. The size and shape of the channel.
B. The size of the ion.
C. The properties of the selectivity filter.
D. The size of the concentration gradient across the membrane.
4. Which of the following statements about biological membranes is TRUE?
A. The composition of membrane lipid bilayers may be varied slightly, to main-
tain it in the gel-crystalline state.
B. The bilayer is stabilized by hydrophobic interactions between the polar lipid
head groups and the aqueous environment.
C. Integral membrane proteins penetrate or span the lipid bilayer, interacting
with the hydrophobic lipid acyl chains.
D. Peripheral membrane proteins are covalently bound with the polar lipid head
groups of the bilayer.
Vol. 45, No. 2, 2015 87
RESEARCH AND TEACHING
prior to enrolling in more advanced
courses. Learning is evaluated by
two multiple-choice examinations,
and all students in a given term
take identical examinations concur-
rently. Questions are selected from a
large database of pretested questions
that were created by instructors for
BIOCH200. The question database
is not available to students, but they
are provided with practice questions
from it as PDF documents. Only cor-
rect/incorrect feedback is given, with
no elaboration (Figure 1).
Study design
The study was conducted in the five
sections of BIOCH200 that were of-
fered in the fall, winter, and spring
terms of a single academic year.
In all five sections, students were
randomly assigned to one of two
groups: One group received prac-
tice questions as normal, via PDF,
and the other received password-
protected access to the quiz game,
Brainspan. During the study, many
students assigned to the game in-
formed us that they had obtained
copies of the PDFs from their class-
mates. Given this, the comparison
groups discussed here are students
who had access to the game (game
users) and those who did not (PDF
only).
Each PDF was generated from
a Word document (Figure 1) and
e-mailed to the PDF-only group at
the same time as game users were
given access to a game. A separate
“answer” PDF in which correct
responses were underlined was
emailed simultaneously. Students
were encouraged to use the question
file to self-test prior to viewing the
answer in the other file, but they had
access to both files at the same time
and could view or print them without
restriction.
88
Journal of College Science Teaching
Brainspan is an asynchronous
multiplayer online quiz generator
(Gnarpe, 2009). Instructors create
multiple-choice questions and load
these into the “games” they are cre-
ating. The appearance of the ques-
tions online is variable, with several
distinct backgrounds available for
selection, including a game show
background. The background used
in BIOCH200 was simple (Figure
2A). Students log on to the site and
answer questions sequentially. For
each question, there is immediate
correct/incorrect feedback (Figures
2B and 2C). Students can check
their progress in the game by gen-
erating a report that shows them the
questions and categories they made
errors on. The game saves scores so
that the student can exit and reenter
without losing points. While students
are answering questions, they can
challenge other students to answer
questions that they have answered
correctly. Challenges are associ-
ated with betting points, adding
an element of competition. A chal-
lenge may be ignored. Questions in
Brainspan can be linked to additional
information files, and instructors can
add feedback via a [Learn More] but-
ton, which becomes available after
the student has answered a question
(Figures 2B and 2C). Students can
also send a message to their instruc-
tors, and instructors can monitor
progress. However, in this study, no
additional information or feedback
was provided because we wanted to
compare the use of the online game
as directly as possible with the PDF
delivery of simple correct/incorrect
feedback.
This study used a mixed meth-
odology to answer the two research
questions. To determine whether
student performance was improved
by access to the game, we calculated
and compared average examina-
tion scores for the game-users and
PDF-only groups. To explore stu-
dent attitudes, we used a survey.
Toward the end of term, the survey
was distributed at the beginning of
a class period. Surveys were color
coded and included a check box so
that the respondents could confirm
which group they were assigned
to. The survey included a series
of specific questions (Table 1) to
which participants responded using
a Likert scale (1 = strongly agree, 2
= agree, 3 = neutral, 4 = disagree,
and 5 = strongly disagree), and an
open-ended question: “Please add
any comments or suggestions about
the use of the practice questions/
learning game in this course in the
space below.”
Participants
Participants were the students in any
of the five sections who voluntarily
completed and returned a survey (N
= 549). Of these, 252 reported being
in the game-users group and 297 re-
ported being in the PDF-only group.
A total of 400 participants included
a written response to the open-end-
ed question.
Students from diverse university
programs enroll in BIOCH200, in-
cluding general science, kinesiol-
ogy, medical laboratory sciences,
nutrition, dental hygiene, physiology,
chemistry, cell biology, and educa-
tion. However, students enrolled in
each section comprise equivalent
random samples from the same
population of students. Each section
of the course is approximately the
same size, with approximately 950
students enrolled in the five sections
included in this study. This research
complies with the University Stan-
dards for the Protection of Human
Research Participants.
FIGURE 2
The appearance of the question and feedback screens in the quiz game. (A) The question was presented at
the top of the screen with a series of responses listed below. Students selected a response by clicking on the
[A], [B], [C] or [D] button. (B) The screen as it appeared if the response selected was correct. (C) The screen as it
appeared if the response selected was incorrect.

Vol. 45, No. 2, 2015 89

RESEARCH AND TEACHING

TABLE 1
Student perceptions of the value of questions in game and PDF format.
Game users PDF users
1 The questions used were appropriate for accomplishing the objectives of the Mean 1.53 1.52
course.
N 252 297
SD .694 .678
2 The questions helped me to build a solid understanding of core concepts. Mean 1.76 1.65
N 251 297
SD .815 .757
3* The game or questions provided useful feedback to me on my learning. Mean 2.28 1.75
(r = .371)
N 250 294
SD 1.193 .940
4* The game or questions took too much time away from my studying. Mean 3.69 4.09
(r = .406) 241 294
N
1.135 .893
SD
5* The questions helped me to quickly identify concepts that I didn’t Mean 2.16 1.90
understand. (r = .429)
N 249 295
SD .948 .833
6 Use of the game or questions resulted in less course stress. Mean 2.41 2.27
N 248 294
SD 1.080 1.049
7 The game or questions helped me to “think like a biochemist.” Mean 2.83 2.76
N 245 286
SD 1.006 .993
8 The questions were designed well for learning this subject. Mean 1.83 1.76
N 249 295
SD .764 .774
9 I enjoyed the course more because of the use of the learning game or Mean 2.53 2.38
questions.
N 247 291
SD 1.107 .987
10 The game or questions allowed me to engage directly with the content being Mean 1.93 1.88
presented.
N 249 285
SD .917 .775
11* I felt better prepared for exams due to usage of the game/practice Mean 1.75 1.45
questions. (r = .406)
N 248 286
SD .928 .722

Note: Responses were collected using a Likert scale: 1= strongly agree; 2 = agree; 3 = neutral; 4 = disagree; 5 = strongly disagree.
Mean item scores for each question were calculated for the game-users and PDF-only groups. A mean item score of ≤2.0 is defined
as the agree range, a score of >2.0 to <4.0 is defined as the neutral range, and a score of ≥4.0 is defined as the disagree range.

*For items 3, 4, 5, and 11 (in bold), a Mann–Whitney test indicated a significant difference in response by the two groups. Item
3: U = 27331.0, z = –5.469, p < .001. Item 4: U = 28786.0, z = –3.971, p < .001. Item 5: U = 31502.5, z = –3.074, p < .01. Item 11: U =
28821.5, z = –4.229, p < .001. The effect size (r) was calculated as recommended by Grissom and Kim (2012) and is equal to the
Mann–Whitney U statistic divided by the product of the two sample sizes.

90
Journal of College Science Teaching
Data analysis
Examination scores and responses to
the survey items were analyzed using
SPSS. Mean scores for each question
were compared for PDF-only and
game-users groups. Mean scores on
the final examination were also com-
pared, for PDF-only and game-user
groups in each class.
The open-ended responses were
analyzed by categorization and cod-
ing. Many of the written responses
contained multiple codes, so the sum
of the responses by category is greater
than the number of participants
(400 written responses included 547
distinct comments). The comments
were analyzed first by two of the
researchers who categorized each
comment as either negative or positive
and then identified subcodes. These
researchers ensured consistency in
coding through regular discussions
of their analysis, which led to agree-
ment on the definitions of the codes.
Each researcher then completed
coding of the responses separately.
The coding was cross-checked by a
third researcher (Gibbs, 2007), who
coded the responses herself, using the
codes and definitions agreed on by the
others, and then compared her own
coding with that achieved by the first
two researchers. The third researcher
found complete agreement.
notably, did not differ from mean We found that the game-users and
scores achieved in other sections of PDF-only groups agreed to the same
BIOCH 200 over a period of at least extent that the questions were well
4 years prior to the study (data not designed and appropriate for learning,
shown). and this was unaffected by the format
(Items 1, 2, 8, and 10). Also, both
Question 2: What are student groups responded neutrally to Items
attitudes towards the practice 6, 7 and 9, suggesting that neither for-
questions delivered in an online mat was perceived as affecting stress
game? levels or enjoyment of the course.
Table 1 compares mean scores for Only the scores for Items 3, 4, 5, and
game-users and PDF-only groups 11 appeared to differ significantly
for each of the survey items. Be- for the two groups (p < .01, Table 1),
cause the Likert scale responses had and there was a medium effect size
a nonnormal distribution, we used the (≥0.3) for all four of these differences
nonparametric Mann–Whitney test (≥0.5 is a large effect size). Effect size
to determine whether there were any was determined as recommended for
differences between the groups. Mann–Whitney tests by Grissom and
FIGURE 3
Performance on the final examination: Game versus PDF-only groups.
Mean final examination scores for game-user and PDF-only groups in
five sections of BIOCH 200 offered in the academic year 2008–2009.
The graph shows mean scores + SD for game-user and PDF-only in each
of five sections. (The –SD error is assumed and not shown.)

Results
Question 1: Is student
performance in examinations
improved by access to an online
quiz game?
Figure 3 shows mean examination
scores for PDF-only and game-users
groups in each of the five sections of
BIOCH 200. We found no difference
in examination scores for the game
users compared with the PDF-only
group. The scores were highly con-
sistent from section to section and,

Vol. 45, No. 2, 2015 91

RESEARCH AND TEACHING

Kim (2012; Table 1). Interestingly,
game users were less likely to agree
that the questions provided useful
feedback on learning (Item 3), less
likely to agree that the questions
helped them to quickly identify
concepts that they didn’t understand
(Item 5), and less likely to agree that
the questions helped them to feel
better prepared for exams (Item 11).
In contrast, game users were more
likely to agree that the practice ques-
tions took too much time away from
their studying (Item 4).
We explored the difference in
response for Item 3 further, by com-
paring the distribution of the scores
for the two groups (Figure 4), and we
found that whereas 85.0% of the PDF
users agreed (score = 1 or 2) that the
questions provided useful feedback
on their learning, only 63.6% of
game users agreed with this state-
ment. Notably, in both cases feed-
back was limited to correct/incorrect
only: in the game this was displayed
on screen immediately after each
question (Figure 2), whereas in the
PDF it was displayed in a separate
file (Figure 1).
Comments and suggestions
Students were also invited to pro-
vide a written response concerning
their overall impression of the game
or PDF. Table 2 summarizes the cat-
egorization of these comments.
Game users submitted more nega-
tive comments: Overall, there were
somewhat more negative than posi-
tive comments (308 negative and
239 positive). However, game users
submitted more than twice as many
negative comments than positive
(211 negative and 97 positive),
whereas PDF users submitted almost
twice as many positive comments
than negative (154 positive and 85
negative).
Questions in PDF format were
more valued in preparation for
exams: Overwhelmingly, the major-
ity of positive comments for both
the game and the PDF were in the
category exam preparation (56% of
positive comments about the game
and 71% of positive comments about
the PDF). Indeed, 29% of all com-
ments returned (157) were catego-
rized as being positive with respect
to preparation for the examination.
These data suggest that regardless
of format, the students perceived
practice questions as helpful to
them. Interestingly, however, 110
of the positive comments regarding
preparation for examinations were
from PDF users compared with only
47 from game users. Apparently, the

FIGURE 4
The distribution of responses for Item 3: The game or questions provided useful feedback to me on my
learning. Responses were collected using a Likert scale: 1 = strongly agree; 2 = agree; 3 = neutral; 4 = disagree;
5 = strongly disagree. For each group, the number of responses in each category is presented as a percentage
of the total responses received. PDF users: mean = 1.75; SD = 0.940; n = 294. Game users: mean = 2.28; SD =
1.193; n = 250.

92
Journal of College Science Teaching
game users did not feel quite as well
prepared for examinations as their
classmates who were assigned the
PDF. This finding was surprising
because the game and PDF provided
students with access to an identi-
cal set of questions and equivalent,
limited, correct/incorrect feedback.
Some of the comments provided at
least a partial explanation for this,
suggesting that game users felt at a
disadvantage in preparation for the
exams because the game format was
different from the exam:
It needs to be more similar to the
exams, I do not want them to be
exactly the same but the exams
have some completely new parts
that I did not see in practice.
(P3-275)
Questions presented as a game
were perceived as engaging and
fun: In keeping with other research
exploring the use and benefits of
learning games, 24% of the posi-
tive comments about the game were
categorized as indicating that the re-
spondents “liked” the game and 16%
were categorized as indicating it was
“engaging and fun.” Unsurprisingly,
there were no positive comments
suggesting that the practice ques-
tions in PDF format were “engaging
and fun.”
Learning game was fun. (G3-78)
Overall it was fun to have the
questions in that format . . . (G4-
86)
Students expressed frustration with
technical issues and format incon-
venience: A number of game users
provided negative comments, which
were related to technical difficul-
ties (37) and difficulties in studying
(29). In contrast, there were almost
no comments in this category from
PDF users.
I hated how the images had to
be open separately and could not
see the questions and image/dia-
gram on same page. This made it
difficult to engage in questions.

TABLE 2
Number of comments by category from game-user and PDF-only groups.

Comment category Game users PDF users Fraction of total

(547)
Negative Feedback explanations 69 20 16%
Wanted other format or both 43 21 12%
Question content 18 29 9%
Technical difficulties 37 2 7%
Difficulties in studying 29 2 6%
Errors in questions/answers 5 17 4%
Boring, uninteresting 7 6 3%
Disliked 3 0 1%
Total 211 97 56%
Positive Exam preparation 47 110 29%
Liked 20 27 9%
Format convenience 2 16 3%
Engaging and fun 14 0 3%
Immediate feedback/instant 2 1 1%
Total 85 154 44%
Note: Written responses to the open-ended question were categorized as shown. Responses from game users (n =195) and PDF
users (n =205) were categorized separately. In some cases, a single response addressed several issues that were coded under
more than one category. Therefore, there are more comments recorded (n = 547) than written responses received (n = 400). The
table shows the number of comments in each category received from the game-users and PDF-only groups. The total number of
comments in each category (from both groups) is also given as a fraction (%) of the total number of comments received.

Vol. 45, No. 2, 2015 93

RESEARCH AND TEACHING
Website was sometimes down,
due to expiration of site certifi-
cate? I think students should be
provided with both the PDF and
game, to study over the questions
after playing games. (G3-79)
It was frustrating that you could
not see the question when you
clicked on the diagram, pop-up
window. You should be able
to move the pop-up window.
The window with the question
should be larger so you do not
have to scroll a lot. (G3-37)
I was not able to view any im-
ages on the game on my com-
puter nor was I able to download
any software that allowed me to
view them. (G4-86)
Bothersome to go back while
studying, cannot print the notes
and use them to study mate-
rial and have to go through all
the questions to review them
for tests. Maybe a print option
would be helpful for the ability
to have a hard copy. (G3-78)
Game users were more likely to
want the other format or both for-
mats: A notable proportion of the
comments (12%) expressed a desire
for access to the other format or to
both formats. However, twice as many
of these comments were provided by
game users (43) than by PDF users
(21), suggesting that the desire for the
“other format” was more pronounced
among the game users. Many of the
respondents noted that they had asked
classmates for a copy of the questions
in PDF format:
I completely admit to asking
my friend to send me one PDF,
much better, I like having a hard
94
Journal of College Science Teaching
copy. (G4-97)
I used the PDF questions of
my classmates who got them
emailed to them and I know I
am not the only one who did this
all term. (G3-32)
This desire is likely related to
the observation that students re-
ported feeling “better prepared”
for examinations when using the
PDF.
Students wanted feedback expla-
nations to the questions: The largest
category of negative comments, 16%
of all comments received, consisted
of complaints about the lack of feed-
back explanations.
Sometimes I did not understand
why my answers were wrong so
a way to explain how the mecha-
nism works would be extremely
helpful. (G1-1)
There should be an explanation
for why the option chosen was
wrong and there should be a
footnote describing where the
topic discussed can be found in
the book or in notes, the dif-
ficulty should be easier as well.
(G1-6)
It would be really nice if the an-
swers were explained, otherwise
the game is no different than a
simple PDF of questions, waste
of time. (G4-154)
Notably, more than three times
as many game users submitted com-
ments in this category than PDF us-
ers (69 compared with 20). That is,
game users felt more strongly that
they should have been provided with
feedback explanations.
The “learn more” link never had
anything posted; it should have
extra information to help you
further understand the questions,
regardless if you got it right or
wrong. (G3-37)
Discussion
Question 1: Is student
performance in examinations
improved by access to an online
quiz game?
We hypothesized that providing
practice questions as an online
game might increase motivation
and engagement and thus improve
performance on exams. However,
we found no difference in perfor-
mance between students who had
access to practice questions in game
format and those who had access
only in PDF format.
Much educational research lit-
erature demonstrates the difficulty
of measuring “improvements” in
learning resulting from specific in-
terventions. Even if we are confident
that we measure learning outcomes
well, those outcomes are influenced
by many factors, including the stu-
dent’s inherent ability, individual
differences (Chickering, 2006), mo-
tivation (Biggs, 2012), and learning
approach (Entwistle & Peterson,
2004; Entwistle, Tait, & McCune,
2000). By offering practice ques-
tions in game format, we intended to
increase engagement with the course
material. However, students differed
notably in their level of engagement
with the game; for example, while
approximately one third of students
in Sections A1, A2, and B2 reported
using the challenge feature of the
game regularly, another one third
reported using it never. That is, stu-
dents were not equally engaged by
the game; many expressed a desire
for “the other format” and indicated
that they had acquired the PDF from
classmates.
There is substantial evidence that
learner differences can affect learn-
ing outcomes. In BIOCH200, we
have previously found that learning
approach (Entwistle & Peterson,
2004) is correlated with performance
on the final examination (Milner,
2014). Also, in studying the impact
of student personal response systems
(clickers) on learning outcomes, we
found that only the highest perform-
ers did better on the final examina-
tion (Addison, Wright, & Milner,
2009). Overall, it is possible that
certain learners did benefit from us-
ing the game in the current study and
that we simply did not measure it. It
also remains possible that the game
could improve longer term retention
through extended practice, but this
was not measured here either.
To address some of these issues,
and to further explore the impact of
Brainspan on student learning in
BIOCH 200, we are currently ana-
lyzing data from a more controlled
study in which individual examina-
tion performances were measured,
rather than class averages, and each
student’s learning approach was as-
sessed.
Question 2: What are student
attitudes towards the practice
questions delivered in an online
game?
Student engagement with a quiz de-
termines its effectiveness in learn-
ing (Anderson et al., 2001; Chick-
ering, 2006). Because engagement
is determined by individual prefer-
ences (Biggs, 2012), it is important
to take student experiences into ac-
count when choosing or designing
tools for learning (Gosper, 2010). In
this study, we used a survey to test
our assumption that students would
engage with the material more ex-
tensively through the online game,
Brainspan.
Overall, the survey responses
revealed that students appreciated
access to the questions in game for-
mat, with a large majority (84%) of
game users agreeing that the use of
the game should be continued. Game
users reported that the questions
were engaging and fun, and they
valued the learning tool. Despite
this, however, game users submitted
many more negative comments than
PDF users.
In general, game users seemed to
feel strongly that detailed feedback
explanations for the questions were
missing. This was unexpected, be-
cause both the PDF and the game
provided identical, limited correct/
incorrect feedback. We suspect that
this sentiment resulted from the
[Learn More] button, an additional
feature of the game that had been
used in other courses. Unfortunately,
in the study reported here, we had
neither the time nor the resources
to populate the field. In general, the
open-ended comments revealed that
students expected more from the
game than from the PDF, and the
[Learn More] button was certainly a
factor in this.
Both PDF and game users agreed
that the practice questions were
helpful in preparation for exams.
Again, however, the game users
were less likely to agree, and the
open-ended comments revealed that
the game users felt that they had
been denied a useful tool, the PDF.
Many more game users than PDF
users expressed a desire for the other
format or both, and this is probably
related to frustration with technical
issues and format inconvenience,
both of which were expressed by
game users. It may also be related
to the similarity in format between
the PDF and the examination. Had
the examination been in an online
format, the reverse might well have
been reported. Of course, the stu-
dents’ reported only a perception,
and there is no evidence that PDF
users were, in fact, better prepared
for the examination.
Conclusions and
implications for practice
In this study, we observed no ben-
efit of delivering practice questions
in an online game compared with
the PDF. This does not mean that
there were no benefits, however, as
impacts of learning tools on learn-
ing outcomes are difficult to mea-
sure (Biggs, 1996). Overall, Brain-
span was received by students as a
“fun” addition to BIOCH200, and
the participants expressed a clear
desire for the use of the game to be
continued. We believe that online
games like Brainspan have the po-
tential to make worthwhile contri-
butions in similar, large-enrollment,
introductory-level science classes,
particularly when provided through
tried-and-tested quiz systems that
require a minimal investment of
time, money, and resources.
We were surprised by our stu-
dents’ perception of disadvantage
and format inconvenience with the
game. We believe that these percep-
tions reflect (a) our decision not
to populate the [Learn More] field
for this study and (b) the students’
desire to have the format of practice
questions match the format of the
examination. We were reminded of
the importance of soliciting feedback
from students when implementing a
new learning tool, because students’
perceptions of a tool will affect their
engagement with it, and no tool can
be useful if it is not used. ■
Vol. 45, No. 2, 2015 95
RESEARCH AND TEACHING
References
Addison, S., Wright, A., & Milner,
R. (2009). Using clickers to
improve student engagement and
performance in an introductory
biochemistry class. Biochemistry
& Molecular Biology Education,
37(2), 84–91.
Akl, E. A., Pretorius, R. W., Sackett,
K., Erdley, W. S., Bhoopathi, P. S.,
Alfarah, Z., & Schünemann, H. J.
(2010). The effect of educational
games on medical students’ learning
outcomes: A systematic review:
BEME guide no. 14. Medical
Teacher, 32, 16–27.
Anderson, L. W., Krathwohl, D. R.,
Airasian, P. W., Cruikshank, K. A.,
Mayer, R. E., Pintrich, P. R., . . .
Wittrock, M. C. (Eds.). (2001). A
taxonomy for learning, teaching,
and assessing: A revision of Bloom’s
taxonomy of educational objectives
(Complete ed.). New York, NY:
Longman.
Benek-Rivera, J., & Mathews, V.
E. (2004). Active learning with
“jeopardy”: Students ask the
questions. Journal of Management
Education, 28, 104–118.
Biggs, J. (1996). Enhancing teaching
through constructive alignment.
Higher Education, 32, 347–364.
Biggs, J. (2012). What the student
does: Teaching for enhanced
learning. Higher Education
Research and Development, 31,
39–55.
Chickering, A. W. (2006). Every
student can learn—if. About
Campus, 11(2), 9–15.
Crisp, B. R. (2007). Is it worth the
effort? How feedback influences
students’ subsequent submission
of assessable work. Assessment &
Evaluation in Higher Education, 32,
571–581.
Crisp, G. T. (2012). Integrative
assessment: Reframing assessment
96
Journal of College Science Teaching
practice for current and future
learning. Assessment & Evaluation
in Higher Education, 37, 33–43.
Dermo, J. (2009). e-Assessment and
the student learning experience:
A survey of student perceptions
of e-assessment. British Journal
of Educational Technology, 40,
203–214.
Entwistle, N. J., & Peterson, E. R.
(2004). Conceptions of learning
and knowledge in higher education:
Relationships with study behaviour
and influences of learning
environments. International Journal
of Educational Research, 41,
407–428.
Entwistle, N., Tait, H., & McCune,
V. (2000). Patterns of response
to an approaches to studying
inventory across contrasting groups
and contexts. European Journal
of Psychology of Education, 15,
33–48.
Ericsson, K. A., & Kintsch, W. (1995).
Long-term working memory.
Psychological Review, 102, 211–
245.
Gibbs, G. R. (2007). Analyzing
qualitative data (Series: The SAGE
Qualitative Research Kit). London,
England: Sage.
Glass, A. L., Brill, G., & Ingate, M.
(2008). Combined online and in-
class pretesting improves exam
performance in general psychology.
Educational Psychology, 28, 483–
503.
Gnarpe, J. (2009). A multiplayer
learning game for medical
education [Letter to the editor].
Medical Teacher, 31, 483–503.
Gosper, M. (2010). Designing online
quizzes: A whole of curriculum
approach. In Proceedings of the
IADIS International Conference
on Cognition and Exploratory
Learning in Digital Age (pp.
11–18).
Grissom, R. J., & Kim, J. J. (2012).
Effect sizes for research: Univariate
and multivariate applications. New
York, NY: Routledge.
Horizon Report. (2006). EDUCAUSE
Learning Initiative and New Media
Consortium. Available at https://
net.educause.edu/ir/library/pdf/
CSD4387.pdf
Hudson, J. N., & Bristow, D. R.
(2006). Formative assessment
can be fun as well as educational.
Advances in Physiology Education,
30(1-4), 33–37.
Jaehnig, W., & Miller, M. L. (2007).
Feedback types in programmed
instruction: A systematic review.
Psychological Record, 57, 219–232.
Karpicke, J. D., & Roediger, H. L.
(2007). Repeated retrieval during
learning is the key to long-term
retention. Journal of Memory and
Language, 57, 151–162.
Kolb, A. Y., & Kolb, D. A. (2009).
The learning way: Meta-cognitive
aspects of experiential learning.
Simulation & Gaming, 40, 297–327.
Krathwohl, D. R., & Anderson, L. W.
(2010). Merlin C. Wittrock and
the revision of Bloom’s taxonomy.
Educational Psychologist, 45,
64–65.
Maxwell, A. (2010). Assessment
strategies for a history exam, or,
why short-answer questions are
better than in-class essays. History
Teacher, 43, 233–245.
Milner, R. E. (2014). Learner
differences and learning outcomes
in an introductory biochemistry
class: Attitude toward images,
visual cognitive skills, and learning
approach. Biochemisty and
Molecular Biology Education, 42,
285–298.
Narciss, S., & Huth, K. (2006).
Fostering achievement and
motivation with bug-related tutoring
feedback in a computer-based
 training for written subtraction.
Learning and Instruction, 16,
310–322.
Nicol, D. J., & Macfarlane-Dick, D.
(2006). Formative assessment and
self-regulated learning: A model
and seven principles of good
feedback practice. Studies in Higher
Education, 31, 199–218.
Peat, M., & Franklin, S. (2002).
Supporting student learning: The
use of computer-based formative
assessment modules. British
Journal of Educational Technology,
33, 515–523.
Peat, M., Franklin, S., Devlin, M., &
Charles, M. (2005). Revisiting the
impact of formative assessment
opportunities on student
learning. Australasian Journal
of Educational Technology, 21,
102–117.
Rocco, S. (2007). Online assessment
and evaluation. New Directions
for Adult & Continuing Education,
2007(113), 75–86.
Shute, V. J., Hansen, E. G., &
Almond, R. G. (2008). You can’t
fatten a hog by weighing it—or
can you? Evaluating an assessment
for learning system called ACED.
International Journal of Artificial
Intelligence in Education, 18,
289–316.
Thatcher, D. C. (1990). Promoting
learning through games and
simulations. Simulation & Gaming,
21, 262–273.
Rachel Milner (rmilner@ualberta.ca) is
a teaching professor in the Department
of Biochemistry, Jonathan Parrish is a
teaching associate professor in the Depart-
ment of Biochemistry, Adrienne Wright is
a teaching professor in the Department of
Biochemistry, Judy Gnarpe is a teaching
professor in the Department of Medical Mi-
crobiology and Immunology, and Louanne
Keenan is an associate professor in the
Department of Family Medicine and direc-
tor of community engaged research, Divi-
sion of Community Engagement, all at the
University of Alberta in Edmonton, Alberta,
Canada.

Ordering is simple
Why would you pay more for the same top quality product?
Here you will find discounts of up to 20% on anatomy models,
medical simulators, biology products and science supplies.

Apply Coupon Code

for an additional 5% discount on orders over $400.00

Discounted prices for

Vol. 45, No. 2, 2015 97

Copyright of Journal of College Science Teaching is the property of National Science
Teachers Association and its content may not be copied or emailed to multiple sites or posted
to a listserv without the copyright holder's express written permission. However, users may
print, download, or email articles for individual use.