Education Tech Research Dev (2009) 57:79–97

DOI 10.1007/s11423-008-9088-3

RESEARCH ARTICLE

Computer-supported aids to making sense of scientific

articles: cognitive, motivational, and attitudinal effects

Julie A. Gegner Æ Donald H. J. Mackay Æ Richard E. Mayer

Published online: 21 March 2008

Ó Association for Educational Communications and Technology 2008

Abstract High school students can access original scientific research articles on the
Internet, but may have trouble understanding them. To address this problem of online
literacy, the authors developed a computer-based prototype for guiding students’ com-
prehension of scientific articles. High school students were asked to read an original
scientific research article about the effects of drug or alcohol abuse, which was presented
on a computer screen either with computer-based comprehension aids or without aids. The
aids provided background text and animations including answers given by the article
authors. Across two experiments involving different schools and different research articles,
the comprehension aids group performed better than the control group on a comprehension
test concerning the article (d = +0.79 in Experiment 1 and d = +0.82 in Experiment 2)
and rated the content of article as less difficult to read (d = +0.96 in Experiment 1 and
d = +0.69 in Experiment 2). In addition, the comprehension aids group reported more
positive motivational beliefs than the control group concerning understanding what sci-
entists do (d = +0.51 in Experiment 1 and d = +0.56 in Experiment 2), appraising the
article as worthwhile (d = +0.80 for Experiment 1 and d = +0.70 for Experiment 2), and
being interested in reading another research article (d = +0.19 in Experiment 1 and
d = +0.58 in Experiment 2). The groups did not differ on their attitudes concerning a
career as a scientist or about alcohol and drug abuse. Overall, computer-based compre-
hension aids can help guide students’ comprehension processes for an original scientific
research article, which in turn can improve their appreciation of what scientists do.

Keywords Online literacy
Science education
Reading comprehension

J. A. Gegner
D. H. J. Mackay
ScienceMedia, Inc., E206, 6450 Lusk Blvd., San Diego, CA 92121, USA

R. E. Mayer (&)
Department of Psychology, University of California, Santa Barbara, CA 93106, USA
e-mail: mayer@psych.ucsb.edu

123
80 J. A. Gegner et al.

Introduction

Suppose a high school student is interested in what happens to a developing fetus when a
pregnant woman consumes alcohol. The student goes to an online browser and finds a
primary source article from a scientific journal. Making sense of the article is a difficult
task, so she skims the article without deeply comprehending the study. She comes away
without fully understanding the article and begins to develop negative beliefs about reading
scientific articles.

Goals of the study

How can we avoid this kind of scenario? This study examines the cognitive and motiva-
tional consequences of using online comprehension aids intended to help high school
biology students make sense of a scientific research article they read online in class.
Making sense of online documents—such as a scientific research article—represents a
form of literacy for which many students are under-prepared (Leu et al. 2008; Pailliotet and
Mosenthal 2000; Willoughby and Wood 2008). Another important aspect of science
education involves the development of a fruitful disposition towards science (Shapka and
Keating 2003). In short, this study is a response to the need to help students make sense of
the scientific research articles that they encounter on the Internet. Our primary research
questions are: Can we improve students’ comprehension of an online scientific research
article by adding annotations intended to guide cognitive processing? Does using the
comprehension aids also improve students’ motivational beliefs about reading online
scientific articles and attitudes about pursuing a career in science and about alcohol and
drug abuse?
Researchers have suggested ways to use features of computers as cognitive tools that
enhance learning (Lajoie 2000; O’Neil 2005; O’Neil and Perez 2006), including as tools
for teaching of online reading strategies (Azevedo and Cromley 2004), but research is
needed to determine how best to incorporate comprehension aids within a computer-based
online reading environment. For example, Wilson and Landoni (2001) describe criteria for
evaluating electronic textbooks. We have developed our comprehension aids in light of
Clark’s (2001) admonition that educational media do not cause learning, but rather edu-
cational methods cause learning. In short, rather than relying on educational technology per
se, we designed a cognitive tool that would assist learners in the way that they process
multimedia instructional messages (Mayer 2001, 2005a).
We sought to develop online comprehension aids that are consistent with how learners
process information during learning, based on Mayer’s (2005a) cognitive theory of mul-
timedia learning. We assumed that learners have separate channels for processing words
and pictures, that the processing capacity of each channel is limited in capacity, and that
meaningful learning requires appropriate cognitive processing during learning including
attending to relevant information, organizing the information into a coherent representa-
tion, and integrating it with other knowledge. The aids were intended to guide the students’
cognitive processing during learning while avoiding the creation of extraneous process-
ing—cognitive processing that wastes cognitive capacity but does not promote learning. In
particular, the aids were intended to accomplish three cognitive goals—to direct the
learner’s attention through the use of highlighting which Mayer (2005b) calls the signaling
principle, to provide pretraining in the names and characteristics of the key concepts which
Mayer (2005c) calls the pretraining principle, and to encourage learners to engage in deep

123
Computer-supported aids to making sense of scientific articles 81

processing by using conversational style and making the author visible which Mayer
(2005d) calls the personalization principle.

Examining the effectiveness of online comprehension aids

Although our ultimate goal is to develop an automated computer-based system for anno-
tating scientific research articles in ways that guide student comprehension, our goal in the
present study is more modest. We seek to test the effectiveness of an online prototype
aimed at guiding student comprehension for an article about the effects of prenatal
exposure to alcohol on children’s brain development (Riley et al. 1995) in Experiment 1
and an article about the biology of drug abuse (Ahmed and Koob 1998) in Experiment 2.
The online prototype was designed in accord with research-based principles of multimedia
design (Mayer 2001, 2005a), with particular focus on directing the student’s attention to
relevant material (i.e., signaling principle), providing background information about key
concepts (i.e., pretraining principle), and offering first-person conversational text from the
author (i.e., personalization principle). Overall, our goal was to create aids that would
prime appropriate cognitive processing during learning, such as attending to relevant
material, organizing the material into a coherent structure, and relating the material to
existing knowledge.
The prototype consisted of a Flash interface to facilitate student access to content in
either a linear (Experiment 1) or exploratory (Experiment 2) manner. In the exploratory

Fig. 1 Experiment 1 prototype interface for reading a primary research article. Content areas were accessed
by clicking on the tabs along the top. Other buttons, such as those displayed on the right, and pull down
menus (not shown) allowed students to move through the comprehension aids. Highlighted text and
underlined glossary terms provided learning assistance within the context of the paper itself

123
82 J. A. Gegner et al.

interface, students have more control over the order of presentation of comprehension aids
than in the linear interface. The prototype was populated with narrated animations, glos-
sary terms, and motivational content (e.g. why was this paper chosen by the author to
annotate). The visuals focused on the journal article to encourage students to learn the
material within the context of the paper itself. Self-check questions were scattered
throughout to promote active student participation in the learning process. An image of the
interface used in Experiment 1 is provided in Fig. 1.

Experiment 1

The purpose of Experiment 1 was to determine the cognitive and attitudinal effects of
asking high school students to use computer-based comprehension aids while reading an
original scientific article (Riley et al. 1995). Experiment 1 compared the learning outcomes
of students who read the Riley paper with and without comprehension aids, thereby
allowing a direct test of the effectiveness of the comprehension aids.

Method

Participants and design

The participants were 122 high school students from a high school in the San Diego area.
The students came from five sections of Advanced Placement (AP) Biology all taught by
the same instructor. Seventy-eight students served in the comprehension aids group and 44
served in the control group. The distribution of students between the two groups was
unequal because three sections participated in the comprehension aids group whereas only
two sections participated in the control group. The assignment of the sections to the groups
was random. The mean age of students in the comprehension aids group was 16.6 years
(SD = 0.77) and the mean age of the students in the control group was 16.7 years
(SD = 0.77). The mean grade level was 11.3 (SD = 0.53) for the comprehension aid
group and 11.4 (SD = 0.54) for the control group. There were 49% boys and 51% girls in
the comprehension aids group and 37% boys and 63% girls in the control group. None of
these differences was statistically significant (at p \ .05) based on t-tests (for age and
grade level) and a chi square test (for sex), indicating that the groups were equivalent on
several basic demographic measures.

Materials and apparatus

The paper-based materials consisted of an instructional sheet for each treatment group and
a questionnaire printed on 8.5 9 11-inch sheets of paper. The instruction sheet for the
control group contained the URL for the Riley article and the instruction sheet for the
comprehension aids group contained the URL for the Riley article with comprehension
aids. The students were asked to go to the assigned website, read the paper, and try to
understand as much of it as they could.
The computer-based materials for the control group consisted of a website that con-
tained an electronic version (i.e., PDF) of an original scientific research article. The article
was selected by its author (Riley), who was recruited to participate in this project based on
his expertise in the field of Fetal Alcohol Syndrome. The criteria for selecting the article

123
Computer-supported aids to making sense of scientific articles 83

was that it was an authentic research article published in a scientific journal, that it reported
original research of high scientific significance, and that it addressed the topic of alcohol or
drug abuse, which is considered to be interesting to high school students. The article used
in this study was ‘‘Abnormalities of the corpus callosum in children prenatally exposed to
alcohol’’ by Riley et al. (1995). The article contained approximately 4000 words and five
graphics (in four figures) and ran for five journal pages.
The computer-based materials for the comprehension aids group consisted of an online
computer program written in Flash, comprised of an interface with an electronic version of
the Riley article and tools for assisting the reader in comprehending the article. The tools
incorporated into the computer program are summarized in Table 1. The tools were orga-
nized into three sections. In section one, called the Paper Tutorial, text-based interview
questions with the paper author (‘‘Q & A’’) introduced students to the paper topic and
included questions such as ‘‘How did the idea for this research develop?’’ and ‘‘What was
the biggest obstacle to completing this work?’’ Section one also contained background
animations and article animations. Background animations covered the prerequisite infor-
mation most AP Biology students would need to better understand the article’s content. The
background animations were two to four minutes in length and explained the medical
condition being studied (Fetal Alcohol Syndrome), what was being studied (the corpus
callosum), the evaluation technique (magnetic resonance imaging), and how the data were

Table 1 Comprehension aids incorporated into interface

Experiment 1 Experiment 2
Section Learning Tools Section Title Learning Tools
Title
Paper Tutorial About the Article interview Investigational About the Article interview questions
questions (Q&A) Mode (Q&A)
Background animations: FAS, Background animations: Drug
Brain Structure, MRI, Statistics Dependence, Animal Models in Addiction
Research, Drugs of Abuse
Research article animations: Research article animations (embedded
Introduction, Methods, Results, within the context of the paper)
Discussion
Self-check questions (embedded Self-check questions (embedded within
within the animations) the context of the paper)
Investigational Text highlights Text highlights
Mode Glossary terms Glossary terms
Access to tutorials About the Scientist interview questions
About the About the Scientist interview (Q&A)
Scientist questions (Q&A)

Self-Check
Questions

123
84 J. A. Gegner et al.

analyzed (statistics). The article animations were one to three minutes in length and dis-
cussed each section of the paper: Introduction, Methods, Results, and Discussion. The
article animations highlighted paper text and graphics to an audio description. For example,
when brain structures were mentioned in the results section, the text within the context of
the article was highlighted while a brain image from the paper was shown with the relevant
brain anatomy highlighted and labeled. Self-check questions were dispersed throughout the
animations. Examples of the self-check questions are provided in Table 1. Students
received feedback giving the correct answer. The material in the Paper Tutorial did not
provide answers to the test questions, but rather helped guide the students’ reading of the
article––which did contain relevant material. Students advanced through section one con-
tent by clicking on Next Chapter or Next Background Topic buttons. The background and
article animations and supporting content were designed to provide a high school science
student with prerequisite information, guidance, and active learning situations. A goal of the
comprehension aids was to remove barriers to understanding for the student.
In section two (Investigational Mode), article text was supplemented with pop-up
explanations and definitions, and the previous animations could be replayed by clicking on
the buttons on the right of the interface under the heading ‘‘Relevant Background Tuto-
rials’’ (Fig. 1).
In section three (About the Scientist), ‘‘Q & A’’ introduced the students to the paper
author with questions such as ‘‘How did you end up in science?’’ and ‘‘What is it about
your research you find most interesting?’’
The questionnaire, which was given to all students, consisted of several sections: (a) a
section that included three rating questions concerning difficulty and attitudes as shown in
the top portion of Table 2, (b) a section containing 12 multiple-choice questions that were
intended to cover the content of the article as shown in the second portion of Table 2, (c) a
section that included three rating questions concerning attitudes, as shown in the third
section of Table 2, (d) a section that included 18 rating items concerning level of disap-
proval of various forms of drug abuse and alcohol abuse (on a 3-point scale with ‘‘1 as
‘‘don’t disapprove’’ and 3 as ‘‘strongly disapprove’’) and 18 rating items concerning level
of risk of various forms of alcohol and drug abuse (on a 4-point scale with 1 as ‘‘no risk’’
and 4 as ‘‘great risk’’), as shown in the fourth section of Table 2, and (e) demographic
questions including age, sex, and ethnicity, as shown in the fifth section of Table 2. The
questionnaire also included open-ended questions and a scientific thinking survey that were
not used for the present study. The questionnaire was designed to evaluate the effectiveness
of the comprehension aids and to elicit comments from students for the empirical revision
and improvement of this learning approach. Whereas the rating items used for the alcohol
and drug abuse were based on a questionnaire developed for the ‘‘Monitoring the Future’’
study conducted at the University of Michigan (Johnston et al. 2007), the other components
of the assessment described in this paper were developed specifically for this study. No
validity data for the measures are available, although the assessment items were designed
to have face validity.
The apparatus consisted of Dell Optiplex 150 computers in the school library media
room of a high school campus.

Procedure

Students were recruited with the offer of a $1,000 stipend to the school’s AP Biology
program. Because the study took place during regularly scheduled class time, student

123
Computer-supported aids to making sense of scientific articles 85

Table 2 Questionnaire description with examples*

Description Example Questions

Rating (3 items) I found reading this paper worthwhile.
Strongly agree Agree Neither agree or disagree Disagree Strongly disagree

Reading this paper helped me to better understand what scientists do.

Strongly agree Agree Neither agree or disagree Disagree Strongly disagree

How would you rate the content level of this exercise?

Too easy Easy About right Difficult Too difficult

Comprehension Children in this study were evaluated using:

Test (12 items) ACC
FAS
PET
MRI
I don’t recall seeing this content in the paper.
What did the study compare?
Brain size of children exposed to alcohol with brain size of children not exposed
to alcohol.
Brain size of children exposed to alcohol with brain size of children with ADHD.
Brain size of children prenatally exposed to alcohol with brain size of children
exposed to alcohol postnatally.
Brain size as a function of the amount of alcohol consumed.
I don’t recall seeing this content in the paper.
What can be concluded from data presented in THIS study?
Children exposed to alcohol have irreversible brain damage.
Children with FAS have ADHD.
Children with FAS have cognitive and behavioral problems.
Alcohol affects specific regions of the developing brain.
I don’t recall seeing this content in the paper.
Rating (3 items) How desirable of a career is being a scientist?
Very Desirable Sort of Not Really not
desirable desirable desirable desirable

After reading this paper, are you more or less interested in reading another research
article?
Much A little No change A little Much less
more more in interest less interested
interested interested interested

After reading this paper, are you more or less interested in a career in science?
Much A little No change A little Much less
more more in interest less interested
interested interested interested

Drug and Alcohol Trying cocaine in powder form once or twice

Attitudes (36 Don’t disapprove Disapprove Strongly disapprove
items)
Taking cocaine powder occasionally
Don’t disapprove Disapprove Strongly disapprove

Trying cocaine in powder form once or twice

No risk Slight risk Moderate risk Great risk Can’t say

Taking cocaine powder occasionally

No risk Slight risk Moderate risk Great risk Can’t say

Are pregnant and take one or two drinks occasionally

No risk Slight risk Moderate risk Great risk Can’t say

Are pregnant and take one or two drinks regularly

No risk Slight risk Moderate risk Great risk Can’t say

Demographics What is your age?

What is your sex?
Male Female
What is your race/ethnicity?

* The overall score for the drug and alcohol disapproval subscale was summed across all items

123
86 J. A. Gegner et al.

attendance was required so all students participated. Students were not graded on their
participation or performance. They were told they were taking part in a research study and
that the AP Biology program at their school would receive a stipend for their participation
but no additional motivation was provided.
Students from five sections of AP Biology all taught by the same instructor were
randomly assigned to groups based on class periods, with students in periods 2 and 6
assigned to the control group and students in periods 3, 4, and 5 assigned to the com-
prehension aids group. Students were assigned to a particular section of AP Biology
independent of this study. Student assignment to a particular section was the result of
availability within their schedule of classes and for the purposes of this study, assumed
random. This assumption is supported by the demographic data presented above. Students
participated in groups of approximately 25 per session as part of their regular class
meeting, but worked individually on the learning and testing phase of the study.
First, the learning phase took place in the school library media room of the high school
campus and lasted 50 minutes. Each student was seated at a workstation with a computer.
Students were told they were taking part in a research project that involved reading a
primary research article and filling out a questionnaire, but they were not told about the
differences between treatment groups or the nature of the questionnaire. Students were
asked not to discuss the study with anyone until after its completion. Students in the control
group were given an instruction sheet that included the URL for the Riley paper and asked
to go the site and read the journal article. Students in the comprehension aids group were
given a similar instruction sheet that included the URL for the Riley paper with com-
prehension aids and asked to go to that site and follow the instructions at the site.
When students in the control group first logged onto the website, a text-based interface
introduced them to the study. The interface contained a ‘‘Click here to begin’’ button that
opened a PDF of the journal article.
When students in the comprehension aids group first logged onto the website, an
interface with an introductory page of text and a ‘‘Click here to begin’’ button appeared.
When clicked on, a narrated introduction instructed the students to maximize their screen.
The narration then described the comprehension aids embedded within the online proto-
type and demonstrated how they could be used. Students moved through the
comprehension aids by clicking Next buttons. ‘‘Q & A’’ text opened followed by
approximately 30 min of background and article animations. Students were not required to
view all animations but could navigate freely through the material by clicking Next or
Previous buttons. At the end of each section, students were instructed to move on to the
next section by clicking on buttons within the Flash interface. Additional navigational
options, such as a slider bar and dropdown menu also were accessible to the student.
Students, on average, completed the tutorial with comprehension aids in about 45 min.
Students in the control group generally took less time averaging about 25 min. Although
time-on-task was not our focus in this study, the cognitive aids treatment was intended to
improve the way that comprehension aid students use their time. Although we are unable to
disentangle time on task with treatment in this study, our main goal was to determine
whether the cognitive aids treatment would help student learning. In short, both groups
were given the same amount of time, although the controls appear to have not needed to
use as much time as students in the comprehension aids group. All students appeared to
complete the exercise within the 50-minute class period.
Upon completion of the computer portion of the study, students returned to their
classroom to complete the questionnaire. The questionnaire took approximately 30 min to
complete.

123
Computer-supported aids to making sense of scientific articles 87

Results

Statistical analysis

To assess the differences between the two groups of students on article comprehension,
perceived difficulty, motivational beliefs, and attitudes about science and drug and alcohol
abuse, t-tests and effect size estimates (Cohen’s d) were calculated independently for each
variable. We used individual t-tests in order to examine specific a priori research questions
based on different data sets. We used an alpha level of .01 to reduce the possibility of Type
1 error.

Does the comprehension aids treatment help students comprehend the article?

The primary goal of the comprehension tool is to help students learn from an original
scientific article. The top row of Table 3 shows the mean comprehension score (out of 12
possible) and standard deviation for the two groups. As shown, the comprehension aid
group scored significantly higher than the control group on the comprehension test,
t(120) = 4.11, p \ 0.0001. The effect size, favoring the comprehension aid group was
d = +0.79, which is considered a medium-to-large effect. The comprehension group thus
had a clear advantage in comprehending the article.

Table 3 Mean scores (M) and standard deviations (SD) for two groups on five kinds of test measures—
Experiment 1
Comprehension aids Control p-value Cohen’s d Test measure

M SD M SD

8.44 2.08 6.93 1.70 \0.0001 +0.79 Comprehension: Number correct on

comprehension test (12 questions).
2.96* 0.55 3.73* 1.00 \0.0001 +0.96 Difficulty: How would you rate the content
level of this exercise?
2.16** 0.83 2.87** 0.94 \0.0001 +0.80 Affect: I found reading this paper
worthwhile.
2.27** 0.91 2.76** 1.00 0.0067 +0.51 Motivation: Reading this paper helped me
better understand what scientists do.
2.60§ 0.98 2.80§ 1.07 0.32 +0.19 Motivation: After reading this paper, are
you more or less interested in reading
another research article?
2.48§§ 1.11 2.82§§ 1.26 0.12 +0.29 Attitudes about scientists: How desirable
of a career is being a scientist?
2.74
0.88 2.89
0.78 0.36 +0.18 Attitudes about scientists: After reading
this paper, are you more or less
interested in a career in science?
46.10

5.97 45.70

6.86 0.70 +0.06 Attitudes about drug/alcohol use: Overall
disapproval rating.
61.40 6.54 61.40 4.97 0.96 +0.00 Attitudes about drug/alcohol use: Overall
risk assessment.

* 1 = too easy, 5 = too difficult; ** 1 = strongly agree, 5 = strongly disagree; § 1 = much more inter-
ested, 5 = much less interested; §§ 1 = very desirable, 5 = really not desirable;
1 = much more
interested, 5 = much less interested;

don’t disapprove, 3 = strongly disapprove;  1 = no risk, 4 = great
risk

123
88 J. A. Gegner et al.

Does the comprehension aids treatment reduce the perceived difficulty of reading
a scientific article?

A related goal of the comprehension aids is to reduce the students’ perceived difficulty in
reading a scientific article. The second row of Table 3 shows the mean rating (with 1 = too
easy and 5 = too difficult) and standard deviation for a question about difficulty, ‘‘How
would you rate the content level of this exercise?’’ As can be seen, the comprehension aids
group rated the exercise as easier than did the control group, t(120) = 5.46, p \ 0.0001.
The effect size favoring the comprehension aid group was d = +0.96, which is considered
a large effect. Overall, the comprehension aids treatment made the reading task feel easier
to the students.

Does the comprehension aids treatment improve students’ motivational beliefs about
reading scientific articles?

In addition to these cognitive outcomes, we were also interested in improving students’

affect and motivation concerning reading scientific articles. The third row of Table 3
shows the mean level of agreement (with 1 = strongly agree and 5 = strongly disagree)
and standard deviation for a question about affect, ‘‘I found reading this paper worth-
while.’’ As shown, the comprehension aid group agreed more with this statement than did
the control group, t(120) = 4.32, p \ 0.0001. The effect size favoring the comprehension
aid group was d = +0.80, which is considered a large effect.
The fourth row of Table 3 shows the mean level of agreement (with 1 = strongly agree
and 5 = strongly disagree) and standard deviation by both groups for a statement about a
motivational belief, ‘‘Reading this paper helped me better understand what scientists do.’’
The difference between the groups was statistically significant, t(120) = 2.76, p = 0.0067.
The effect size was d = +0.51, which is considered a medium-sized effect.
The fifth row of Table 3 shows the mean rating (with 1 = much more interested and
5 = much less interested) and standard deviation by both groups for a question about
motivation, ‘‘After reading this paper, are you more or less interested in reading another
research article?’’ The difference between the groups did not reach statistical significance,
t(120) = 1.00, p = 0.32. The effect size was d = +0.19, which is considered a small effect.

Does the comprehension aids treatment improve students’ attitudes about science
and about alcohol and drug abuse?

Finally, we were interested in whether the comprehension aids could affect students’
attitudes about science and about alcohol and drug abuse. The sixth row of Table 3 shows
the mean rating (with 1 = very desirable and 5 = really not desirable) and standard
deviation for the question, ‘‘How desirable of a career is being a scientist?’’ The difference
between the groups did not reach statistical significance, t(120) = 1.57, p = 0.12,
d = +0.29. The seventh row of Table 3 shows the mean rating (with 1 = much more
interested and 5 = much less interested) and standard deviation for the question, ‘‘After
reading this paper are you more or less interested in a career in science?’’ The difference
between the groups did not reach statistical significance, t(120) = 0.919, p = 0.36,
d = +0.18. Overall, there is no statistically significant evidence that the comprehension aid
treatment had a positive effect on students’ attitudes about pursuing a career as a scientist.

123
Computer-supported aids to making sense of scientific articles 89

The eighth row of Table 3 gives the mean rating (with 1 = don’t disapprove and
3 = strongly disapprove) and standard deviation across all items on the drug and alcohol
disapproval subscale of the questionnaire. Results indicated no significant difference
between the groups, t(120) = 0.386, p = 0.70, d = +0.06. The ninth row of Table 3 gives
the mean rating (with 1 = no risk and 5 = great risk) and standard deviation across all
items on the drug and alcohol risk subscale of the questionnaire, t(120) = 0.044, p = 0.96,
d = 0.00. The comprehension aids treatment did not affect students’ attitudes about drug
and alcohol abuse.
Overall, the comprehension aids treatment resulted in better comprehension of the
research article, less reported difficulty in reading the article, and more positive motiva-
tional beliefs about reading scientific articles relative to the control group. On the other
hand, the comprehension aids treatment did not affect the students’ attitudes about scientist
careers and alcohol/drug abuse.

Experiment 2

To establish the generality of the results of Experiment 1, Experiment 2 was conducted to

determine if the results of Experiment 1 would be replicated in a context differing in the
participating school, treatment assignment, scientific article, lag time in testing, computer
equipment, and interface for delivering the comprehension aids. Students read a research
article about the biology of drug abuse (Ahmed and Koob 1998) either with or without
comprehension aids. The article was selected by its author (Koob), who was given the
selection criteria defined in Experiment 1.

Method

Participants and design

Students were recruited as described in Experiment 1. The participants were 97 high school
students from a high school in the San Diego area that was different from the one studied in
Experiment 1. The students came from three sections of Advanced Placement (AP)
Biology all taught by the same instructor. Student attendance was required so all students
participated; students were not graded on their participation and no additional motivation
for the task outside of the school stipend was provided. Students within each section were
randomly assigned to the groups, as in Experiment 1. Fifty students served in the com-
prehension aids group and 47 served in the control group. The mean age of students in the
comprehension aids group was 16.2 years (SD = 0.81) and the mean age of the students in
the control group was 16.0 years (SD = 0.82). The mean grade level was 10.6
(SD = 0.72) for the comprehension aid group and 10.7 (SD = 0.72) for the control group.
There were 52% boys and 48% girls in the comprehension aids group and 49% boys and
51% girls in the control group. None of these differences was statistically significant,
indicating that the groups were equivalent on several basic demographic measures.

Materials and apparatus

The paper-based materials consisted of an instructional sheet for each treatment group and
a six-part questionnaire printed on 8.5 9 11-inch sheets of paper. The instruction sheet for
the control group contained the URL for the Ahmed and Koob article, and the instruction

123
90 J. A. Gegner et al.

sheet for the comprehension aids group contained the URL for the Ahmed and Koob article
with comprehension aids. The students were asked to go to the assigned website, read the
paper, and try to understand as much of it as they could.
The computer-based materials for the control group consisted of a website that con-
tained an electronic version (i.e., PDF) of an original scientific research article, chosen with
the same criteria as in Experiment 1. The article used in this study was ‘‘Transition from
Moderate to Excessive Drug Intake: Change in Hedonic Set Point’’ by Ahmed and Koob
(1998). The article contained approximately 3000 words and 10 graphics (in 3 figures) and
ran for 3 journal pages.
The computer-based materials for the comprehension aids group consisted of an online
computer program written in Flash, comprised of an electronic version of the Ahmed and
Koob article along with tools for assisting the reader in comprehending the article. The
tools were comparable to those created for Experiment 1 although they were organized into
a single section as defined in Table 1. As Fig. 2 illustrates, the ‘‘Q & A’’ and background
animations were accessed by clicking on buttons on the lower right of the interface.
Background animations described the human medical condition being studied (drug
dependence), the experimental technique (animal models), and the mechanism of action
(drugs of abuse). Article animations and self-check questions could be viewed by clicking
on icons embedded within the margins of the research article. These comprehension aids

Fig. 2 Experiment 2 prototype interface for reading a primary research article. Background animations and
author ‘‘Q&A’’ comprehension aids were accessed by clicking on buttons in the lower right of the interface.
Highlighted text and underlined glossary terms provided learning assistance within the context of the paper
itself. Self-check questions and article animations were accessed by clicking on red question marks and
arrowheads near the article content they described. A zoom feature allowed students to adjust font size for
improved readability

123
Computer-supported aids to making sense of scientific articles 91

were located near the article content they discussed. The rationale for using multiple
comprehension aids was to encourage active cognitive processing in which learners related
the presented material to their prior knowledge and in which they attempted to focus on
and organize on relevant material.
The questionnaire was the same as that used in Experiment 1 with the exception that the
12 comprehension questions were replaced with questions that pertained to the content in
the Ahmed and Koob article. For example, three of the comprehension questions were: (1)
The animals in this study were evaluated by––––––; (2) The scientists monitored the
animals’ behavior to determine the relationship between––––––; (3) Changes in drug
consumption were attributed to––––––. As in Experiment 1, the comprehension multiple
choice questions were written to obtain a wide distribution of responses and included a
range of questions requiring various levels of abstraction.
The apparatus consisted of OS 9.1 iMac computers in the school library media room of
a high school campus for the control participants and OS 10 iBooks in a classroom for the
comprehension aids group.

Procedure

Students from three sections of AP Biology all taught by the same instructor were ran-
domly assigned to groups, with half the students in each class randomly assigned to one
treatment and the other half assigned to the other treatment. Students participated in groups
of approximately 10–20 per session as part of their regular class meeting, but worked
individually on the learning and testing phase of the study.
Students in the control group were asked to go to one classroom whereas students in the
comprehension aids group were asked to go to another room. Both classrooms contained
workstations with computers and each student was seated at a workstation with a computer.
Students were told they were taking part in a research project that involved reading a
primary research article and filling out a questionnaire, but they were not told about the
differences between treatment groups or the nature of the questionnaire. Students were
asked not to discuss the study with anyone until after its completion. Students in the control
group were given an instruction sheet that included the URL for the Ahmed and Koob
paper, and asked to go to the site and read the journal article. Students in the compre-
hension aids group were given a similar instruction sheet that included the URL for the
Ahmed and Koob paper with comprehension aids, and asked to go to that site and follow
the instructions at the site. Class time was 55 minutes. Students in the control group logged
onto a website with a text introduction and link to a PDF of the journal article.
Students in the comprehension aids group logged onto the website and viewed an
introductory page of text with a ‘‘Click here to begin’’ button. When clicked on, a narrated
introduction instructed the students to maximize their screen. This information was fol-
lowed by a second narration that demonstrated how the tools located within the article
interface could be used. When the narration finished, students had immediate access to all
of the comprehension aids and they could select their own path through the material with
no additional guidance. For example, students had the option to read the article text, click
on animations and self-check questions pertaining to that text, click on text highlights and
glossary terms, and click on background animations in any order. This differed from
Experiment 1 where the background animations and article animations played before
students could navigate freely through the article content. Students completed this exercise
in approximately the same amount of time as the students in Experiment 1. The students

123
92 J. A. Gegner et al.

completed the questionnaire a day after reading the article. The questionnaire took
approximately 30 minutes to complete.

Results

Does the comprehension aids treatment help students comprehend the article?

The primary goal of the comprehension tool is to help students learn from an original
scientific article. The top row of Table 4 shows the mean comprehension score (out of 12
possible) and standard deviation for the two groups. As in Experiment 1, the compre-
hension aid group scored significantly higher than the control group on the comprehension
test, t(95) = 4.02, p \ 0.0001. The effect size, favoring the comprehension aid group was
d = + 0.82, which is considered a large effect.

Does the comprehension aids treatment reduce the perceived difficulty of reading
a scientific article?

A related goal is to reduce the students’ perceived difficulty in reading a scientific article.
The second row of Table 4 shows the mean rating (with 1 = too easy and 5 = too diffi-
cult) and standard deviation for a question about difficulty, ‘‘How would you rate the

Table 4 Mean scores (M) and standard deviations (SD) for two groups on five kinds of test measures—
Experiment 2
Comprehension Control p Cohen’s d Test measure
aids

M SD M SD

7.68 2.13 5.94 2.14 \0.0001 +0.82 Comprehension: Number correct on

comprehension test.
3.24* 0.76 3.72* 0.62 0.0008 +0.69 Difficulty: How would you rate the content level of
this exercise?
2.35** 0.63 2.96** 1.07 0.0008 +0.70 Affect: I found reading this paper worthwhile.
2.29** 0.76 2.81** 1.07 0.0061 +0.56 Motivation: Reading this paper helped me better
understand what scientists do.
2.48§ 0.65 2.98§ 1.03 0.0056 +0.58 Motivation: After reading this paper, are you more
or less interested in reading another research
article?
2.49§§ 1.03 2.39§§ 1.11 0.63 +0.09 Attitudes about scientists: How desirable of a
career is being a scientist?
2.70
0.65 2.79
0.74 0.52 +0.13 Attitudes about scientists: After reading this paper,
are you more or less interested in a career in
science?
44.70

5.70 44.80

6.80 0.95 +0.02 Attitudes about drug/alcohol use: Overall
disapproval rating.
58.90 6.42 58.10 8.02 0.62 +0.11 Attitudes about drug/alcohol use: Overall risk
assessment.

* 1 = too easy, 5 = too difficult; ** 1 = strongly agree, 5 = strongly disagree; § 1 = much more inter-
ested, 5 = much less interested; §§ 1 = very desirable, 5 = really not desirable;
1 = much more
interested, 5 = much less interested;

don’t disapprove, 3 = strongly disapprove;  1 = no risk, 4 = great
risk

123
Computer-supported aids to making sense of scientific articles 93

content level of this exercise?’’ As in Experiment 1, the comprehension aid group rated the
exercise as easier than did the control group, t(95) = 3.47, p = 0.0008. The effect size
favoring the comprehension aid group was d = +0.69, which is considered a medium-to-
large effect. Overall, the comprehension aids treatment made the reading task feel easier to
the students.

Does the comprehension aids treatment improve students’ motivational beliefs

about reading scientific articles?

In addition to these cognitive outcomes, we were also interested in improving students’

affect and motivation concerning reading scientific articles. The third row of Table 4
shows the mean level of agreement (with 1 = strongly agree and 5 = strongly disagree)
and standard deviation for a question about affect, ‘‘I found reading this paper worth-
while.’’ As can be seen, the comprehension aid group expressed stronger agreement with
this statement than did the control group, t(95) = 3.46, p = 0.0008, consistent with the
results of Experiment 1. The effect size favoring the comprehension aid group was
d = +0.70, which is considered a medium-to-large effect.
The fourth row of Table 4 shows the mean level of agreement (with 1 = strongly agree
and 5 = strongly disagree) and standard deviation by both groups for a statement about a
motivational belief, ‘‘Reading this paper helped me better understand what scientists do.’’
The difference between the groups reached statistical significance, t(95) = 2.81,
p = 0.0061, also consistent with the results of Experiment 1. The effect size was
d = +0.56, which is considered a medium-sized effect.
The fifth row of Table 4 shows the mean level of agreement (with 1 = much more
interested and 5 = much less interested) and standard deviation by both groups for a
question about motivation, ‘‘After reading this paper, are you more or less interested in
reading another research article?’’ The comprehension aids group reported more interest in
reading another article than did the control group, t(95) = 2.87, p = 0.0056, although the
effect was not significant in Experiment 1. The effect size was d = +0.58, which is
considered a medium effect. The comprehension aids treatment resulted in improvements
in students’ motivational beliefs.

Does the comprehension aids treatment improve students’ attitudes about science
and about alcohol and drug abuse?

Finally, we were interested in whether the comprehension aids could affect students’
attitudes about science and about alcohol and drug abuse. The sixth row of Table 4 shows
the mean rating (with 1 = very desirable and 5 = really not desirable) and standard
deviation for the question, ‘‘How desirable of a career is being a scientist?’’ As in
Experiment 1, the difference between the groups did not reach statistical significance,
t(95) = 0.478, p = 0.63, d = +0.09. The seventh row of Table 4 shows the mean rating
(with 1 = much more interested and 5 = much less interested) and standard deviation for
the question, ‘‘After reading this paper are you more or less interested in a career in
science?’’ As in Experiment 1, the difference between the groups did not reach statistical
significance, t(95) = 0.652, p = 0.52, d = +0.13. Overall, there is no statistically signif-
icant evidence that the comprehension aid treatment had a positive effect on students’
attitudes about a pursuing a career as a scientist.

123
94 J. A. Gegner et al.

The eighth row of Table 4 gives the mean rating (with 1 = don’t disapprove and
3 = strongly disapprove) and standard deviation across all items on the drug and alcohol
abuse disapproval subscale of the questionnaire. There was no significant difference
between the groups. The group comparison was nonsignificant: t(95) = 0.067, p = 0.95,
d = +0.02. The ninth row of Table 3 gives the mean rating (with 1 = no risk and
5 = great risk) and standard deviation across all items on the drug abuse subscale of the
questionnaire, and again there was no significant effect, t(95) = 0.495, p = 0.62,
d = 0.11. Consistent with the results of Experiment 1, the comprehension aids treatment
did not affect students’ attitudes about drug and alcohol abuse.
The pattern of results closely corresponds to that in Experiment 1: Overall, the com-
prehension aids treatment resulted in better comprehension of the research article, less
reported difficulty with the material, and higher liking of the article, but did not affect the
students’ attitudes about scientists or about alcohol and drug abuse.

Conclusion

Empirical contribution

In two controlled experiments, high school students who read a scientific article
accompanied by online comprehension aids, learned more from the article, found the
article easier to read, and developed more positive beliefs about reading scientific articles
than did students who read the article without comprehension aids. A variety of online
comprehension aids were used in the comprehension aids group so future research is
needed to determine the individual contribution of each comprehension aid. The com-
prehension aids appear to be effective for both linear and exploratory interfaces, although
additional research is needed to determine whether certain aids work best for certain
kinds of interfaces. It is possible that the comprehension aid treatment made it easier for
students to the read the article, which is reflected in the difference in difficulty ratings
between the groups. In addition, the comprehension aids group made more use of the
available learning time than did the control group so future research is needed to pin-
point the role of time on task as well as how to motivate students to apply more effort in
learning.

Theoretical implications

The results are consistent with the cognitive theory of multimedia learning (Mayer 2001,
2005a), which posits that learners may need guidance in how to process text and graphical
material. In particular, students need guidance in how to avoid extraneous processing (that
is, processing that does not support the instructional goal), how to manage essential pro-
cessing (that is, processing that is needed to mentally represent the content material), and
how to foster generative processing (that is, mentally organizing the material and relating it
to other knowledge).
The comprehension aids treatment sought to accomplish these three cognitive goals.
First, signaling—highlighting of the essential material—was used to direct the student’s
attention toward to core material and thereby reduce extraneous processing. According to
the signaling principle, students learn better when cues are added that highlight the
essential material (Mayer 2005b).

123
Computer-supported aids to making sense of scientific articles 95

Second, pretraining—training in the names and meanings of the key concepts—was

used to help the student be able to engage in essential processing. According to the
pretraining principle, students learn better when they know the names and characteristics of
the key concepts (Mayer 2005c).
Third, personalization—such as making the author more visible and using conversa-
tional style—helps encourage the student to process more deeply, that is, engage in
generative processing. According to the personalization principle, students learn better
when the author speaks directly to them using first and second personal conversational
style rather than formal style (Mayer 2005d). Although the present study was not intended
to determine the independent contribution of each of these three factors, an instructional
method that incorporated them caused improvements in learning. The results offer general
support for the three research-based principles of signaling, pretraining, and personaliza-
tion, but cannot pinpoint which principles produced the strongest effects.

Practical implications

Although students have access to important scientific articles through online searches, they
may lack adequate comprehension skills to make sense of the articles they find. The
present results encourage the development of online comprehension aids to help students
make sense of important scientific articles they access during online searches. Online
comprehension aids can allow students to benefit from the content of original source
scientific articles while also developing positive believes about reading the articles.
Finally, students may not have been highly motivated to learn because they were not
graded on this exercise.

Limitations and future directions

This project was intended to test the potential value of online comprehension aids for
helping students make sense of scientific articles. Although the techniques used in the
present studies were successful (yielding medium and large effect sizes), it is possible that
other techniques would be as effective or more effective. Thus, we can only conclude that
adding comprehension aids can be helpful. Further research is needed to identify which
aids are most helpful for which students for which kinds of learning objectives. In par-
ticular, it would be useful to pinpoint whether the question-and-answer section with the
authors is helpful, whether the multimedia materials are helpful, and whether user control
is helpful.
The development of comprehension aids for individual articles can be a time-consuming
task, so work is needed to help instructors create comprehension aids with more efficiency.
The design and development of the aids used in this study took approximately 200 hours
per paper. For their development to be cost effective, this value would need to be reduced
about 10-fold. In short, there is a need for a computer-based tool for creating useful
annotations to key scientific papers. Such a tool has the potential to automate the input and
display of content within a template based on the instructional design principles used in
this study.
The effects of the treatment were limited to strong effects on learning and on moti-
vational beliefs, as the treatment did not affect students’ attitudes about a career in science
or alcohol and drug abuse. If the instructional goal is to influence attitudes about scientific
careers and alcohol and drug abuse, other persuasive methods are needed.

123
96 J. A. Gegner et al.

The students were taking Advanced Placement courses so they might not be typical of
all high school students. AP students may be more motivated than other students to use
comprehension aids. Further research is needed to determine whether the comprehension
aids are effective for other types of students.
While we used intact classes in Experiment 1 but not in Experiment 2, both designs led
to similar results. We treated individual students as our units of analysis because each
student worked individually at a computer. We used somewhat different testing conditions
and different computers between the experiments, but still obtained a similar pattern of
results in both experiments.
Overall, the present study encourages the development of a system for creating anno-
tations to scientific research articles, consistent with research-based principles of
multimedia instructional design. Providing online comprehension aids––in the form of
annotations to an article––can help improve online literacy by guiding students’ cognitive
processing of scientific articles they find in online searches (Leu et al. 2008).

Acknowledgments This study was supported by Grant 1R43HD049931-01A1 from the National Institute
of Child Health and Human Development to ScienceMedia, Inc. In the interests of full disclosure, we note
that the computer-based comprehension system tested in this study is the property of ScienceMedia, Inc.,
which holds all proprietary rights. Richard E. Mayer served as a paid consultant on the project.

References

Ahmed, S. H., & Koob, G. F. (1998). Transition from moderate to excessive drug intake: change in hedonic
set point. Science, 282, 298–300.
Azevedo, R., & Cromley, J. G. (2004). Does training in self-regulated learning facilitate students’ learning
with hypermedia? Journal of Educational Psychology, 96, 523–535.
Clark, R. E. (2001). Learning from media. Greenwich, CT: Information Age Publishing.
Johnston, L. D., O’Malley, P. M., Bachman, J. G., & Schulenberg, J. E. (2007). Monitoring the future
national survey results on drug use, 1975–2006. Secondary school students (Vol. 1). Bethesda, MD:
National Institute on Drug Abuse (NIH Publication No. 07-6205).
Lajoie, S. P. (Ed.). (2000). Computers as cognitive tools: No more walls. Mahwah, NJ: Erlbaum.
Leu, D. J., Coiro, J., Knobel, M., & Lankshear, C. (Eds.). (2008). The handbook of research in new
literacies. Mahwah, NJ: Erlbaum.
Mayer, R. E. (2001). Multimedia learning. New York: Cambridge University Press.
Mayer, R. E. (2005a). Cognitive theory of multimedia learning. In R. E. Mayer (Ed.), The Cambridge
handbook of multimedia learning (pp. 31–48). New York: Cambridge University Press.
Mayer, R. E. (2005b). Principles for reducing extraneous processing in multimedia learning: Coherence,
signaling, redundancy, spatial contiguity, temporal contiguity principles. In R. E. Mayer (Ed.), The
Cambridge handbook of multimedia learning (pp. 183–200). New York: Cambridge University Press.
Mayer, R. E. (2005c). Principles for managing essential processing in multimedia learning: Segmenting,
pretraining, modality principles. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia
learning (pp. 169–182). New York: Cambridge University Press.
Mayer, R. E. (2005d). Principles of multimedia learning based on social cues: Personalization, voice, and
image principles. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (pp. 201–
212). New York: Cambridge University Press.
O’Neil, H. F. (Ed.). (2005). What works in distance learning: Guidelines. Greenwich, CT: Information Age
Publishing.
O’Neil, H. F., & Perez, R. S. (Eds.). (2006). Web-based learning: Theory, research, and practice. Mahwah,
NJ: Erlbaum.
Pailliotet, A. W., & Mosenthal, P. B. (Eds.). (2000). Reconceptualizing literacy in the media age. Stamford,
CT: JAI Press.
Riley, E., Mattson, S. N., Sowell, E. R., Jernigan, T. L., Sobel, D. F., & Jones, K. L. (1995). Abnormalities
of the corpus callosum in children prenatally exposed to alcohol. Alcohol Clinical Experimental
Research, 19, 1198–1202.

123
Computer-supported aids to making sense of scientific articles 97

Shapka, J. D., & Keating, D. P. (2003). Effects of a girls-only curriculum during adolescence: Performance,
persistence, and engagement in mathematics and science. American Educational Research Journal, 40,
929–960.
Willoughby, T., & Wood, E. (Eds.). (2008). Children’s learning in a digital world. Oxford, UK: Blackwell
Publishing.
Wilson, R., & Landoni, M. (2001). Evaluating electronic textbooks: A methodology. In Research and
advanced technologies for digital libraries (pp. 1–12). London: Springer-Verlag.

Julie A. Gegner has a Ph.D. in Biochemistry. She is a Program Manager at ScienceMedia, Inc.

Donald H. J. Mackay has a Ph.D. in Chemical Physics. He is a Founder of ScienceMedia, Inc. Dr. Mackay
teaches high school chemistry at Southwest High School, Imperial Beach, CA.

Richard E. Mayer is Professor of Psychology at the University of California, Santa Barbara where he has
served since 1975. He is Vice President for Division C (Learning and Instruction) of the American Edu-
cational Research Association, past-President of the Division of Educational Psychology of the American
Psychological Association, and recipient of the E. L. Thorndike Award for career achievement in educa-
tional psychology. In 2008 he received the American Psychological Association’s Distinguished
Contributions of Applications of Psychology to Education and Training Award, and in 2004 he was ranked
#1 as the most productive educational psychologist by Contemporary Educational Psychology. He is the
author of 20 books and more than 250 articles and chapters, including The Promise of Educational
Psychology: Vols. 1 and 2 (1999, 2002), Multimedia Learning (2001), Learning and Instruction (2003,
2008), e-Learning and the Science of Instruction (with R. Clark, 2003, 2008), and The Cambridge Handbook
of Multimedia Learning (editor, 2005).

123