Designing Websites For ESL Learners: A Usability Testing Study

See discussions, stats, and author profiles for this publication at: https://www.researchgate.
net/publication/228375221
Designing websites for ESL learners: A usability testing study
Article · January 2008
CITATIONS READS
6 406
6 authors, including:
Min Liu Alison McGregor

University of Texas at Austin Princeton University
113 PUBLICATIONS 1,918 CITATIONS 10 PUBLICATIONS 16 CITATIONS
SEE PROFILE SEE PROFILE
Some of the authors of this publication are also working on these related projects:
SNS and MOOC View project
Alien Rescue View project
All content following this page was uploaded by Alison McGregor on 09 June 2015.
The user has requested enhancement of the downloaded file.

CALICO Journal, 25 (2) M. Liu, T. Traphagan, J. Huh, Y. I. Koh, G. Choi, and A. McGregor
Designing Websites for ESL Learners:

A Usability Testing Study
Min Liu
Tomoko Traphagan
Jin Huh
Young Ihn Koh
Gilok Choi
Allison McGregor
The University of Texas at Austin
ABSTRACT
The purpose of this paper is to report on a usability study for ESL websites conducted
to gain insights from learners of English as a second language (ESL) as they interacted
with specific sites. Five carefully selected ESL sites were tested by 10 different users
generating a total of fifty testing sessions. Two major research questions guided the
study: (a) what are intermediate-level ESL learners’ perceptions of the design features
for the specific ESL websites? and (b) what design features contribute to learning from
the ESL websites? Participant reactions to site specific learning tasks and evaluative
questionnaires provided evidence of design features relevant to the ESL context. The
findings address information, interface, and interaction design issues as well as highlight
the importance of (a) organizing the site meaningfully, (b) recognizing the users’ prior
knowledge to identify unambiguous labels and categories, (c) making the site easy to
navigate, and (d) aligning learning activities or tasks with how ESL learners learn. Re-
sults of the study provide implications of the usefulness of ESL websites, the effective-
ness of site design features, and the degree to which sites meet ESL learners’ needs.
KEYWORDS
English as a Second Language (ESL), Website Design, Usability Testing, User-centered Design, Second
Language Learning and Teaching
INTRODUCTION
The world wide web has become an established tool for language learning and teaching (Suss-
er & Robb, 2004; Warschauer, 2005). Given the vast number of websites related to English
as a Second Language (ESL) and the proliferation of their use, questions have been raised
regarding the effectiveness of this evolving technology for learning English. Specifically, from
an evaluation standpoint, Susser and Robb (2004) assert that “the vast quantity and variety
of Internet resources for language study is welcome but leaves us with the problem of evalu-
ating the quality of what is available” (p. 279).
Investigating design effectiveness of ESL websites requires establishing good design

criteria for the context of second language acquisition (SLA). One window into understanding
the design factors is by obtaining information directly from users. Usability testing provides
an opportunity for understanding users’ experience with a particular website. According to
CALICO Journal, 25 (2), p-p 207-240. © 2008 CALICO Journal
207
CALICO Journal, 25 (2) Designing Websites for ESL Learners: A Usability Testing Study
Nielsen, Snyder, Molich, and Farrell (2000, p. 7) usability testing “involves watching a person
use the site in order to discover the ways the site aids or hinders people from reaching their
goals.” Usability testing results not only highlight user preferences but also their behaviors
while interacting with sites. The objective of this research is to contribute to design guidelines
for ESL websites based on such directly observed user preferences and behaviors.
REVIEW OF THE LITERATURE

Websites can be useful resources in fostering English language learning. In particular, their
capacity to offer authentic language tasks (e.g., reading authentic materials, participating in
online chat, listening to native speaker recordings, and communicating with native speakers)
and to boost students’ motivation holds promise for facilitating the acquisition of English.
Some studies have shown positive effects of using web-based activities on the students’ con-
versational, reading, and writing skills (Gu, 2002; Kung & Chuo, 2002; Lin, 2003). Other stud-
ies have reported that web-enhanced activities fostered students’ development of in-depth
cultural understanding while improving linguistic proficiency, as well as generating increased
engagement in the learning activities (Hoecherl-Alden, 2000; Osuna & Meskill, 1998).
Literature on using websites in ESL classrooms, however, also suggests that simply
providing technologically enhanced learning tools to students does not itself foster learning
but that, instead, positive results require carefully designed materials based on a critical
understanding of the relationship between the language-learning processes and technology
(Warschauer, Turbee, & Roberts, 1996). According to Plass (1998), design should support the
cognitive processes involved in linguistic and pragmatic skill development for SLA. His argu-
ment is echoed by Peterson (1998, p. 350) who stated that “any general design rationale for
the creation of web-based second language learning environments will be an integrative con-
struct encompassing both a high-level theoretical view of how students learn and lower-level
issues of interface design and site construction.” Therefore, an effective ESL web learning site
should embody not only good design with technological enhancements but also connect to
SLA theory.
Good design creates a positive experience for users and can lead to more desirable
outcomes (Phyo, 2003). In human-computer interaction literature, much emphasis has been
placed on the importance of user-centered design (Norman & Draper, 1986; Norman, 2002).
Understanding how users think can help create websites that meet users’ goals and objec-
tives (Krug, 2005). Design should address the three essential components of information,
interface, and interaction. Information design is “the art of organizing data so that it is easy
to find and use” the information on a site (Phyo, 2003, p. 2). Interface design refers to those
elements that affect the appearance of a site (e.g., font, font size, colors, spacing, visuals,
and icons) and the manipulation of these elements to both enhance the presentation of in-
formation and make the site easy to navigate (Phyo, 2003). In other words, information and
interface should be purposefully implemented so that they enrich the overall experience of
using a site. Accordingly, the result of a well organized site is that users know at all times
what they can do in the site, where they are, where they can go, and where they have been
(Krug, 2005). Interaction design encompasses the elements that create a meaningful and
enjoyable experience (Shedroff, 1994) for the purpose of enhancing the way people learn,
work, communicate, and interact (Preece, Rogers, & Sharp, 2002). Clearly, the three design
aspects are related to each other and, hence, many design features involve more than one
aspect. Together, information, interface and interaction make up the basic tenets of a sound
design and should, therefore, be incorporated in well designed ESL sites.
208
Given the scarcity of empirical research addressing such design considerations for ESL
sites, additional research is needed on how ESL learners actually use sites as a tool to facili-
tate their learning. The objective of this ESL websites usability study was to gain evaluative
information from ESL learners for the purpose of informing design practices. As stated above,
usability testing involves observing target audience participants using sites so researchers can
note how the sites help, confuse, or defeat users. Then, design principles drawn from the us-
ers’ perspective (both impressionistic and behavioral feedback) gained from usability testing
should help build effective and attractive websites (Preece et al., 2002). Two primary research
questions guided the study:
1. What are intermediate-level ESL learners’ perceptions of the design fea-

tures for specific ESL websites?
2. What design features contribute to learning from the ESL websites?
METHODOLOGY
Collection and Selection of ESL Websites for Usability Testing
This usability research study consisted of two phases: identifying potential ESL websites for
usability testing and actual testing of five websites by ESL learners. Initially, a total of 80
websites were collected via solicitation from ESL instructors affiliated with a large university in
the Southwest as well as Google searches for ESL-related websites. After reviewing each site’s
goals, content, target skill levels, and media, a list of 37 websites was generated for poten-
tial use in the study (see list of websites in Appendix A). A manageable number of candidate
websites is recommended for evaluation (Susser & Robb, 2004), and according to Nielsen (n.
d.), five evaluators are sufficient to locate most problems in a usability study. Given these
recommendations, our goal was to select five sites from the list of 37 and test each site with
10 different ESL users to ensure both a manageable and rigorous research project.
To reduce the number of websites to five, we first excluded those websites that had
broken links, charged a fee to use, or were part of other URLs on our list. Second, we elimi-
nated sites that did not have a main learning component because our usability testing in-
tended to evaluate websites from an ESL perspective in learning. At this point, the research
team evaluated and rated each site using a 3-point scale based on how useful the site seemed
to be. We retained the websites that received the highest ratings and two types of websites
emerged: those that addressed more than one language skill and those whose primary focus
was on grammar. Given this finding, we decided to focus our testing on the more comprehen-
sive sites (sites addressing more than one language skill) and excluded those covering only
English grammar. This resulted in a total of nine sites. The final selections were based on the
type of media used, language skills covered, popularity, and appropriate activities for our in-
termediate-level participants. The final set of five sites for the usability testing were
1. Stuff for English Learners (http://www.Stuff.co.uk/wicked.htm)

2. English Club (http://www.englishclub.com)
3. Grammar & Writing (http://grammar.ccc.commnet.edu/grammar)
4. ESL Cafe (http://www.eslcafe.com)
5. Study Zone (http://web2.uvcs.uvic.ca/elc/studyzone/index.htm)
Each site will be referred to by the name listed above in later sections of this paper. A brief
description of each site follows.
209
Website Descriptions
Stuff for English learners
The Stuff for English Learners (hereafter Stuff) site includes vocabulary, pronunciation, TOEFL,
reading, and word games. It uses animation and interactivity supported by Shockwave and
Flash technologies. For example, the Phrasal Verb Dispenser has a vocabulary activity for two-
part verb combinations in which users first choose a topic (e.g. computers, emotions, etc.)
from a side menu. Next, by clicking on arrows to change the verbs, particles, or meanings,
users try to find the correct combination. Through another feature called the Phonetic Calcula-
tor, learners hear English vowel sounds after clicking on the corresponding symbols based on
the International Phonetic Alphabet.
English Club
The English Club site offers over 5,000 pages and provides numerous resources for both
learners and teachers. These can be accessed through a tab menu across the top of the home
page (Learning Center, ESL Clubhouse, Teacher’s Lounge, ESL Resources, and ESL World) or
found through the site’s search function. Activities for learners include quizzes (some with
automatic scoring), timed typing practice with automatic scoring, games, synchronous chat-
ting, as well as English jokes, and articles for reading. The site includes lessons not only for
general adult learners (from beginning to advanced levels), but also English for business and
young learners.
Grammar & Writing

The Grammar & Writing site focuses on issues pertaining to writing, including the writing pro-
cess, organization, and common problems for ESL learners such as primer style and sentence
variety. The home page is organized around six themes (Word & Sentence Level; Paragraph
Level; Essay & Research Paper Level; Ask Grammar, Quizzes, Search Devices; Peripherals &
PowerPoint; and GrammarPoll, Guestbook, Awards) with pulldown menus to navigate to the
information or activities included in each section. Lessons are mainly presented through text,
but there are over 170 interactive quizzes and crossword puzzles with feedback created in
JavaScript and CGI.
ESL Cafe
Users enter the ESL Cafe site by clicking on one of the following labels: Stuff for Students,
Stuff for Teachers, or Stuff for Everyone. The Stuff for Students section includes Help Center,
Hint-of-the-Day, Idioms, Phrasal Verbs, Pronunciation Power, Quizzes, Slang, and Student Fo-
rums. Within this section, there are extensive lists of vocabulary (idioms, phrasal verbs, and
slang) and over 40 interactive quizzes with automatic feedback/scoring on grammar, vocabu-
lary, culture, reading, and writing.
Study Zone
The Study Zone site is designed for various levels of ESL learners: the lessons and practices
are organized by five TOEFL score ranges (200, 330, 410, 490, and 570) with level specific
lessons, grammar quizzes, reading, and vocabulary. At the 200 level, the vocabulary quiz-
210
zes are particularly extensive and guessing game activities are included. At the 490 and 570
levels, various activities such as timed reading, reading comprehension exercises, grammar
exercises, crossword puzzles, and writing exercises are offered based on reading passages.
The exercises have hints and/or immediate feedback.
Participants and Site Order

The participants in this usability study were intermediate-level ESL learners who volunteered
to participate in the project in order to practice their English. Most were affiliated with a large
research university in the southwestern part of the US and enrolled in ESL classes at the time
of this research. Each participant was asked to test two sites with one hour allocated for each
site. A short familiarity survey was given to the participants prior to the testing to determine
whether they had had any previous experience using any of the websites being tested. Given
their responses on the survey, the participants were randomly assigned to test a site with
which they were not familiar. A total of 50 testing sessions were generated from 25 partici-
pants. In other words, each of the five sites was tested by 10 different participants. We also
counterbalanced the order in which the participants tested the sites to minimize any bias that
could possibly be introduced through the site sequence.
Instruments
To address our research questions, participants’ information and feedback were collected
through: (a) a demographic questionnaire, (c) an observation of how participants actually
used the site while performing site related tasks, and 3) a feedback questionnaire evaluating
the site. Construction of the instruments closely followed Nielsen et al.’s (2000) methodology
and previous research on usability testing (Krug, 2005; Mayhew, 1999; Preece et al., 2002;
Rubin, 1994) (see instruments used in Appendixes B and C).
Demographic questionnaire
The purpose of the demographic questionnaire was to collect participant information relevant
to the usability testing of ESL websites. The questionnaire included three sections: (a) per-
sonal information (e.g. age, gender), (b) level of computer and web experience, and (c) Eng-
lish learning history including experience using the web for English training.
Site-related tasks
Based on Nielson et al.’s (2000) usability testing method, three types of tasks—first impres-
sions, specific tasks, and open-ended tasks—were designed for each of the five websites
tested. The research team developed these tasks to simulate activities learners would do in
self-study. After each task, a think-aloud elicitation question was used to prompt verbal feed-
back from the participants. Sample think-aloud questions included “Is it easy/difficult to find
phrasal verbs in this site? Why? Do you like to learn this way (by trial and error)?” Each task
is described in more detail below.
Task 1: First impressions. In order to elicit participants’ general first impression of the
sites, free-form exploration was used for the initial task. The participants could look at any
part of the website and were then asked to describe their first impression of the site.
211
Task 2: Specific tasks. The aim of the specific tasks was to observe participants’ in-
teraction with the site in semistructured tasks. Because the five websites covered different
content, specific tasks were uniquely created for each site. For example, the following is a
specific task used for English Club:
You want to improve your English speaking skill. Today, you have decided to
study words and expressions for telephone English. Review the Telephone Terms
section and answer the following questions:
1. Name one word or term that you newly learned from this site.
2. How long did it take to find the Telephone Terms section?
Task 3: Open-ended tasks. We asked open-ended questions which required the partici-
pants to browse the site in any way they chose and find information or express their opinions
about the effectiveness of the activities. For example, the following are open-ended questions
for Stuff:
Listen to a poem. Click Hints on Pronunciation for Foreigners and work on the
activity for three minutes.
1. What did you learn from this activity about pronunciation?

2. Do you think this information is useful?
For tasks 2 and 3, the topics were chosen carefully from different skill areas including
grammar, listening, writing, reading, speaking, chatting, and games (see sample tasks in Ap-
pendix B). One researcher created tasks for one site while other researchers from the team
performed and evaluated the tasks. Modifications were made based on the feedback, and
then two pilot-testing sessions were held to try out the tasks with two ESL participants from
the target population of the study. The objectives of the pilot testing were to verify that (a)
the tasks were understandable and doable, (b) the questionnaires made sense to the users,
(c) the testing environment was functional, (d) the users understood the pretest introduction
and instructions, and finally (e) the time allotted for each test was adequate. After pilot test-
ing, several of the task explanations and survey questions were revised to make them more
comprehensible. Also, to prevent user fatigue, the total number of tasks was reduced from
five to four and the number of the required written responses was lessened.
Site feedback questionnaire

The purpose of the site feedback questionnaire was to assess the participants’ overall satis-
faction with their website experience. First, based on a Likert scale of 1 (very unsatisfied) to
5 (very satisfied), each website was evaluated in nine aspects including ease of finding the
information/activities, quality of the information/activities, and appearance of the site. In the
second part of the questionnaire, participants were asked to rate how useful each site was to
improve different English skills (i.e. speaking, listening, reading, writing, grammar, and cul-
ture) on a scale of 1 (useless) to 5 (very useful). Finally, the third section included questions
such as the likelihood of returning to the site and what they liked most or least about the site
(See Appendix C for the questionnaire).
212
Procedure and Testing Facility

The usability testing was conducted with one researcher as a moderator for each partici-
pant. To ensure the consistency of the testing procedures, all the researchers followed the
same testing protocol. When a participant arrived, the assigned researcher briefly explained
the purpose of the test, the procedure, and instructions for each task. The participant then
completed the demographic questionnaire. Afterwards, the participant was given the site fa-
miliarity survey to identify previous exposure to any of the five selected sites. Based on this
information, the researcher randomly assigned a site the participant had never seen or used
before. Next, the participant went to the assigned website and completed the specific learning
tasks described above. Each participant was observed while he or she navigated the site and
performed the given tasks. During the observation, the researcher took notes and captured
screen shots of the web page(s) in which the participant seemed to have difficulty in perform-
ing the tasks.1 After each of the specific tasks, the researcher asked think-aloud elicitation
questions. Finally, the participant was asked to complete the site feedback questionnaire.
The majority of the users, 23 out of 25, did their usability testing in one of the com-
puter labs on the university campus. They used DELL PCs equipped with high-speed Internet
connection with Internet Explorer and all plug-ins required to use the sites (i.e., Shockwave,
Windows Media Player, and Flash Player). One user participated in this study in a university
office with a Gateway PC laptop with the same software setup and Ethernet connection. The
final participant completed the study at the home of one of the researchers using a PC desktop
with the same software setup and high-speed cable connection.
RESULTS AND DISCUSSION

About the Participants
The ESL learners participating in this study ranged in age from 16 to 40, were intermediate-
level in English-language learning, and equally represented males and females. All of them
had experience in using computers, had access to the Internet, and daily surfed the web for
various purposes. Three fourths of them had experience of using the web for studying English
and reported doing so for about an average of 2 hours weekly (See Table 1).
Table 1
Results of the Demographic Questionnairea
Age Gender Years of Eng- Level of Years of Computer Web surfing
lish learning Englishb computer use experiencec experience
16-30 male <= 5 intermediate 2-5 basic basic
72% 48% 41% 91% 12% 52% 60%
(n = 18) (n = 12) (n = 9) (n = 20) (n = 3) (n = 13) (n = 15)
31-40 female >5 intermediate 5-10 experienced experienced
28% 52% 59% high 60% 48% 40%
(n = 7) (n = 13) (n = 13) 9% (n = 15) (n = 12) (n = 10)
(n = 2)
advanced over 10
0% 28%
(n = 0) (n = 7)
213
Table 1 (continued)
Internet Hours of web Preference for Experience in Hours of studying
access use per day using ESL webd using web to English using web
study English (per week)e
home <2 low yes <= 2
76% 52% 70% 76% 68%
(n = 19) (n = 13) (n = 14) (n = 19) (n = 13)
home and work >2 high no >2
24% 48% 30% 24% 32%
(n = 6) (n = 12) (n = 6) (n = 6) (n = 6)
a
Missing responses were excluded.
b
The Likert scale for this item ranged from 1 (Intermediate) through 5 (Advanced).
c
“Basic” represents levels 1 - 3 and “Experienced” represents levels 4 -5 on Item 9a of the questionnaire
with 1 being “Not at All Experienced” and 5 being “Very Experienced” (see questionnaire in Appendix
C).
d
“Low” represents levels 1-3 and “High” represents levels 4-5 on Item 14 of the questionnaire with 1
being “Not at All” and 5 being “Very Much.”
e
Only respondents who use the web for studying answered this question.
Participants’ Responses in the Site Feedback Questionnaire

As shown in Table 2, the compiled ratings from the site feedback questionnaire indicated the
highest general satisfaction with English Club followed by ESL Cafe. For these two sites, all
items were rated 3 or higher on the 5-point scale (1-very unsatisfactory to 5-very satisfied).
Participants reported viewing English Club and ESL Cafe positively in terms of information,
interaction, and interface designs, as well as their value in helping them practice language
skills. Participants also reported that they would be likely to use these sites again or recom-
mend them to other people.
Table 2
Results of the Site Feedback Questionnairea
Usability Aspect Stuff English Club Grammar & ESL Cafe Study Zone
Writing
Interfaceb 3.08 3.94 3.18 3.91 3.37
Interaction c
3.15 4.05 2.55 3.65 3.10
Information d
2.80 4.05 3.35 3.90 3.55
Perceived Usefulness for 2.66 3.57 2.55 3.29 2.87
Practicing Linguistic Skillse
Worth Visiting Againf 2.55 4.20 2.90 4.00 3.15
a
The results of this questionnaire are based on a 5-point Likert scale.
b
Interface includes items 1, 3, 4, 5, and 9 of the questionnaire (see Appendix C).
c
Interaction includes items 6 and 7.
d
Information includes items 2 and 8.
e
This aspect includes item 10.
f
This aspect includes items 11 and 12.
In examining how participants with different demographic characteristics rated their
experiences using the five sites, we found that higher ratings were given by (a) females,
(b) those with fewer years of English learning, (c) intermediate-level learners (rather than
high-intermediate-level learners), (d) novice computer users, and (e) participants with no
214
experience in using the web to study English (see Tables 3 and 4 below). Other demographic
differences were not noticeable.
The finding that female participants gave higher ratings than male participants is con-
sistent with other research results indicating females tend to have more positive attitudes
toward learning using computers than males (Hao & Liu, 2006). The results also indicated
that less experienced English learners and computer users found these ESL sites more helpful
than the more experienced users. Presumably, less experienced English learners were aided
more or found more value in using the websites. Users with more computer and web experi-
ence were perhaps more evaluative in their feedback indicating that those users with more
technogy savvy may have different expectations than those with less experience.
Table 3
Site Feedback Means Based on Demographic Questions
Demographic question Interface Interaction Information Linguistic Worth
skills visiting
Age group (n = 25)
16-30 (n = 18) 3.51 3.24 3.49 3.04 3.30
31-50 (n = 7) 3.45 3.46 3.64 3.04 3.54
Gender (n = 25)
Male (n = 12) 2.67 2.57 2.71 2.29 2.34
Female (n = 13) 3.27 3.06 3.29 2.94 3.10
Years of learning English (n = 22)
<= 5 (n = 9) 3.80 3.56 3.81 3.37 3.84
> 5 (n = 13) 3.33 3.15 3.37 2.93 3.15
Levels of English (n = 22)
Intermediate (n = 20) 3.60 3.36 3.60 3.18 3.46
Intermediate high (n = 2) 2.80 2.88 3.00 2.54 3.13
Years of computer use (n = 25)
2-5 (n = 3) 3.63 3.42 3.75 3.22 3.50
5-10 (n = 15) 3.53 3.32 3.58 3.20 3.41
Over 10 (n = 7) 3.35 3.21 3.32 2.65 3.21
Computer experience (n = 25)
Basic (n = 13) 3.58 3.40 3.69 3.19 3.38
Experienced (n = 12) 3.39 3.19 3.35 2.86 3.35
Web surfing experience (n = 25)
Basic (n = 15) 3.56 3.28 3.58 3.08 3.36
Experienced (n = 10) 3.39 3.33 3.45 2.96 3.38
215
Table 4
Site Feedback Based on Web-surfing Behaviors
Demographic question Interface Interaction Information Linguistic Worth
skills visiting
Internet access (n = 25)
Home (n = 19) 3.42 3.22 3.51 3.04 3.23
Home and work (n = 6) 3.74 3.54 3.58 3.05 3.79
Hours of web use per day (n = 25)
<= 2 (n = 13) 3.59 3.37 3.69 3.23 3.54
> 2 (n = 12) 3.38 3.23 3.35 2.87 3.19
Preference for using ESL websites (n
= 20)
Low (n = 14) 3.53 3.21 3.45 3.18 3.38
High high (n = 6) 3.27 3.33 3.46 2.85 3.21
Experience in using web to study
English (n = 25)
Yes (n = 19) 3.42 3.20 3.42 3.04 3.30
No (n = 6) 3.73 3.63 3.88 3.04 3.58
Hours of studying English using web
(n = 19)
<= 2 (n = 13) 3.41 3.21 3.48 3.22 3.29
>2 (n = 6) 3.43 3.17 3.29 2.63 3.33
Findings from Site Task Performance Data1

Analyses of the participants’ reactions to the learning tasks took four steps. First, the obser-
vation notes for each testing session were written up and compiled. Second, in order to make
sense of the data in a consistent and systematic manner, we compiled observation notes
from one site. The observation notes were organized into a table with three columns: (a) the
participants’ reactions to the tasks as observed in their behaviors or their comments, (b) the
researcher’s interpretation and description of what he or she observed, and (c) whether what
was observed related to the information, interface, or interaction design of the site. As themes
related to site design emerged, a template for organizing the data was established. Using this
template, we organized the data for the other four sites and summarized the findings for each
site. We attempted to extrapolate findings reflecting opinions of most participants and identi-
fied design consideration across the sites. Finally, by drawing comparisons across all five sites,
themes emerged. The following is a discussion of the results.
Overall site design issues

Organization of information. ESL websites often contain a rich amount of information. For
users to access site content easily, organization in well thought out categories is imperative
(Silver, 2005). Clearly presenting the site hierarchy or in layers of information is particularly
important for the home page in which users attempt to understand how the information is
organized in the site (Krug, 2005). For example, most participants favorably perceived the
home page of ESL Cafe where the information was presented in three major categories (Stuff
for Teachers, Stuff for Students, and Stuff for Everyone) and could identify how to enter the
site through the navigation system (see Figure 1).
216
Figure 1
ESL cafe: Home Page
Similarly for Study Zone, the participants liked the idea of dividing the content of the
website according to users’ English levels (see Figure 2).
Figure 2
Study Zone: Home Page
They pointed out, however, that the TOEFL scores on the home page might not be
meaningful to those users who have not taken a TOEFL test. Since there was no explicit expla-
nation about the levels of the scores, some users might have a hard time understanding the
significance of these numbers. Using more meaningful labels such as beginner, intermediate,
and advanced together with (or without) TOEFL scores or providing a short explanation about
the TOEFL scores would therefore have improved the usability of the site.
Participants found other aspects of ESL Cafe’s organization of information less clear.
217
For example, on the Complete List page of the Idiom section, some participants stated that
the alphabetical listing was not helpful and suggested listing the idioms by themes. Addition-
ally, under Hint-of-the-Day, after clicking List All Entries, a long list of date, topic, number, and
contributor’s name sorted by date was presented (see Figure 3).
Figure 3
ESL Cafe: List All Entries Under Hint-of-the-Day
The participants were overwhelmed by the large amount of information and stated
they could not easily find the information they were interested in. Similarly, on the Forum
page, postings were listed without categorization, and some participants expressed frustra-
tion in how much time it took to find information (see Figure 4). In all these cases, a theme-
based categorization would have made the organization of information more transparent and
accessible to the users.
Figure 4
ESL Cafe: Forum Page
218
Previous research on the organization of information has highlighted the importance

of constructing a website’s content in a meaningful and relevant way from the users’ per-
spectives (Preece et al., 2002). Our findings suggest that information on ESL websites can
be meaningfully organized by category of user (e.g., teachers or students), skill level (e.g.,
beginner, intermediate, or advanced), task (e.g., grammar, reading, writing, listening, and
speaking), or themes/topics.
Layout. Users have varying degrees of tolerance for visual noise, and some users are
easily distracted by a complex layout (Krug, 2005). In general, participants in this study dis-
liked busy layouts, those which included too much information or formats which required a lot
of scrolling. Pages in the Grammar & Writing site, for example, tended to be long and wide
because there were too many words on a page. The participants needed to scroll down to
locate the information they were looking for (see Figure 5). As one user stated, “It was very
difficult to find information …”.
Figure 5
Grammar & Writing: Typical Layout of Content Page
Although there was an Index button on Grammar & Writing’s home page that led to an
overview of all the topics covered in the site, it was placed near the bottom of the page. This
potentially useful navigation tool, therefore, was not utilized by many participants.
Participants also disliked the long pages in ESL Cafe, which required a lot of scrolling
(see Figures 3 and 4 above). In Study Zone, the vocabulary practice page was so long that us-
ers had to scroll down repeatedly to view all the questions on the page. Participants suggested
that the page could be designed with a different layout such as using pulldown menus or mul-
tiple columns that could utilize the space on either side of the pictures. They also pointed out
that the timer, which showed the remaining time for reading comprehension, was not visible
after scrolling down in the window.
In general, pages crowded with words tended to overwhelm users, and long pages re-
quiring a lot of scrolling made it difficult to find information quickly and easily. The challenge
to present a rich amount of information in limited screen space necessitates efficient layout
designs. For example, the reading comprehension section in Study Zone provided clearly
defined task areas (Krug, 2005) in different frames so users could view the text passages,
questions, and feedback to their answers simultaneously without feeling overwhelmed (see
Figure 6). The participants particularly liked this feature.
219
Figure 6
Study Zone: Three Separate Frames
The participants also reported they could easily find information they were seeking in
English Club, even with the large amount of information on the home page, because of the
left-hand side navigation menu and a multiple-column layout (see Figure 7).
Figure 7
English Club: Left-hand Side Navigation and Multiple-column Layout
Search function. A search engine is the first place where many web users go to find
what they need (Krug, 2005; Silver, 2005). In English Club, all participants found the search
engine function on the home page very helpful. However, it took a few participants some time
to locate the search function (see Figure 8). The search box could have been displayed more
prominently.
220
Figure 8
English Club: Search Engine Function
Use of pulldown menus. The use of pulldown menus are an effective way to save space
when there is a lot of information to present in a website. However, they have a tradeoff: the
items in the pulldown menus are not visible to users until they click on the pulldown menu or
submenus in the pulldown menu (Krug, 2005). In the Grammar & Writing site, multiple pull-
down menus were used as the main navigation tool, but participants could not see the items
in the pulldown menus nor the hierarchy of the information without clicking on each one of
the menus (see Figure 9).
Figure 9
Grammar & Writing: Pulldown Menu
221
One user mentioned “[pulldown] menus in both the main and other pages are … very
difficult and tedious [to use].” Pulldown menus can be useful, yet they should be used only
for certain types of lists, such as alphabetized lists of known items (e.g., the 50 states) (Krug,
2005). Using them as the main navigation device is not effective as shown by the results of
this study.
Home buttons and breadcrumbs. Breadcrumbs refer to a form of navigation that in-
dicates the current location within the website by a list of pages from the current location up
to the home page. Home buttons and breadcrumbs are among the most powerful navigation
tools to indicate to users where they are in the site, especially for hierarchically organized
sites (Silver, 2005). Breadcrumbs were not present or conspicuous in most sites we exam-
ined. For example, in Grammar & Writing, navigating through content pages relied solely on
the pulldown menus, and this made it hard for users to navigate back to the home page or
other pages. The lack of site ID linked directly to the home page and the inconspicuous home
button, placed at the bottom of each subpage (requiring users to scroll down to see it), in-
creased participants’ frustration in navigation (see Figure 10).
Figure 10
Grammar & Writing: Link to Home Page
In fact, two participants failed to find the Home buttons, and said, “I hope there is an
icon that directly leads to the Home page somewhere because I had to click on the back but-
ton of the browser several times.” This difficulty in navigation could have been reduced if the
website had contained breadcrumbs.
The Study Zone site also provided go back buttons (to the previous or home page) as
a main navigation tool, but these buttons were placed on the bottom of the page, and were
hard for users to notice. Similarly, participants recommended a home button on Stuff and ESL
Cafe to avoid getting lost. Such findings emphasized the need for using home buttons and
breadcrumbs as essential navigation tools for sites containing a large amount of information
with a complex hierarchical organization (Krug, 2005; Phyo, 2003).
Labels and word choices. Users decide which part of a website to click on based on the
labels they see. The mapping between labels and the expected results from clicking them,
therefore, should be clear to the users (Silver, 2005). As mentioned before, the TOEFL score
222
labels found on the home page of Study Zone could have been made more meaningful to us-
ers if terms such as beginner, intermediate, and advanced had been used.
In the Idiom section of ESL Cafe, there were hyperlinks to Complete List, Meanings and
Examples, and Idiom (see Figure 11).
Figure 11
ESL Cafe: Labels
The Idiom hyperlink randomly brings up an idiom from the collection. However, only
the word “Idiom” was hyperlinked, not the word “random” in front of it. One participant clicked
on the link to Idiom without seeing “random” before it and got very confused to find one idiom
presented to him. Also, another participant was very confused when he clicked on “E” on the
Complete List page because the pulldown menu showed “F” instead of “E” (See Figure 12). In
summary, when labels were not used properly, they misled and confused the participants.
Figure 12
ESL Cafe: Menu of “E” Page
In the Grammar & Writing site, labels contained a number of grammatical terms, such
as "conjunctions," "determiners," and "interjections," which were daunting to the intermedi-
223
ate-level ESL users. For example, “He [The participant] didn’t know the word plural and I [the
moderator] had to explain it to him,” stated one moderator. Some participants experienced
navigation problems simply because they were not familiar with the terms used in the menus
and did not know where to click for the information they wanted, while the participants who
happened to know such terms easily located their desired information. Although definitions
of the terms were given at the beginning of each subpage, they did not appear to help the
participants. This was perhaps because the participants were not likely to click on a menu
item that they did not understand. Similarly, in using Stuff, some participants had trouble
getting to the Phonetic Calculator page because they did not know what “phonetic” meant.
Also in the Think of a Number activity in Stuff (see Figure 17 below), four participants failed
to complete the task because they did not know the words “multiply” and “digits” used in the
instructions.
Poor mapping diverts users’ attention from the task to the interface (Silver, 2005),
thereby sacrificing learning efficiency. Labels and other language elements used in websites
should be selected so that they are easily understandable by users. For ESL websites in par-
ticular, users’ language proficiency and their prior knowledge should be taken into consider-
ation in deciding which labels to use.
Color and font size. A home page is a preview into a website. Users often determine
whether they want to look into the site further based on the feeling they receive from a home
page. Choice of colors for a home page is critical in this respect. The yellow-based colorful de-
sign of ESL Cafe (see Figure 1 above) was well received by most participants. One participant
mentioned, “I like the interface design. They are colorful so that they make me interested in
the things inside.”
In Stuff, five participants found the home page unattractive, and four of them specifi-
cally attributed it to its dark background color, suggesting the need for the use of a lighter
color (see Figures 13, 15, and 16 below). Similarly, participants perceived the grey back-
ground color of the Study Zone site as gloomy. A dark background color makes pages not only
unattractive but also makes the text hard to read (Vest, Crowson, & Pochran, 2005). Hence
dark colors should be avoided for web page background.
Participants in general did no like the small font sizes used for some websites. For
example, in English Club, a few participants complained that the home page was busy with
numerous small-lettered links (see Figures 7 and 8 above). Similarly, in Study Zone, the small
font size and narrow spacing in the Reading Comprehension section were mentioned as a sig-
nificant design problem. Designers of these sites may want to consider increasing the size of
the fonts.
Design of learning activities

Activity design and instruction. Activity designs for some sites were not intuitive, and partici-
pants had difficulties in figuring out how to use the activities. This was particularly the case
with Stuff. While exploring the site freely, four participants commented that they did not know
how to do the activities. One participant, for example, said, “even though this website shows
a new attempt to teach English using various multimedia tools, it is too difficult and compli-
cated for me to figure out how to use each web page.” For example, in Phrasal Verb Drinks
Dispenser (see Figure 13), the participants were supposed to change verbs, particles, and
meanings until they found the corresponding combination of the three components using the
arrows. Two participants simply clicked the buttons on the right to change the content areas
and got lost. Another participant kept clicking the external links at the top of the page. Later,
224
this participant could not figure out how to check the right answers. There appeared to be
too many operational steps for the users to perform the activity, and the operation was not
intuitive enough. Although brief instruction was provided, many participants did not notice it
because the instruction appeared below the activity box.
Figure 13
Stuff: Phrasal Verb Drink Dispenser
In using ESL Cafe, a few participants got lost not knowing what they were supposed to
do for the Hint of the Day activity. They explained that they did not know what the page was
offering and/or how to use the page, specifically the search box. The title of the page, Hint of
the Day, was not descriptive enough of what this section was about, nor did the search box
or buttons clearly indicate what one was supposed to do on the page (see Figure 14). A brief
description of what to do would be beneficial for this activity.
Figure 14
ESL Cafe: Hint of the Day
In using Grammar & Writing, participants’ comments focused on navigation difficulties

and the lack of instructions on how to use the website. For instance, one participant com-
225
plained, “I don’t know how to use … no introduction or instruction on how to get to the gram-
mar part … it was not very clear to me.”
Ideally, the interface of an activity should be designed intuitively enough for users to
easily navigate the site and complete the learning activities without instruction. If necessary,
brief instruction should be provided in noticeable places for users to follow.
Animation and audio. One clear advantage of websites over textbooks or workbooks
is their capacity to include multimedia such as graphics, animations, videos, and audios. In
particular, it is desirable to provide audio in language learning websites since ample listening
is considered important for language acquisition. In using ESL Cafe, one participant, who was
very eager to learn how to pronounce the idioms presented in the site, said he sometimes
had no clue how to pronounce the words without listening to them; there was no audio in ESL
Cafe. The Pronunciation Calculator page in Stuff was one example that exploited animation
and audio to facilitate language learning (see Figure 15).
Figure 15
Stuff: Phonetic Calculator
In this activity, users could click the vowel buttons or combinations of buttons to play
the vowels or diphthongs and also activate the animation that showed the lip movements to
produce those sounds. Some participants mentioned that the animation was helpful to under-
stand how to pronounce these sounds correctly. Unfortunately, the lips moved too fast and the
animation was too small for some users to observe how the lips were moving. Many partici-
pants mentioned that the audio in this activity was helpful. They said, “It was helpful because
it gives actual pronunciation unlike book-type dictionaries” and “I could hear the difference
between two sounds.” Also, for Hints on Pronunciation for Foreigners in Stuff (see Figure 16),
one participant mentioned, “hearing the words helped me understand the words because I
could compare some sounds that were similar to each other.”
While animation and audio seem to make activities look fun, users may not necessarily
appreciate these features if they are not presented in a clearly defined learning framework. In
the animation-based Think of a Number activity in Stuff, six participants said it was fun but
that they did not consider it as helpful for learning. One participant said, “This is just for fun,
not for learning English. If I have free time, I would do it just for fun, but not for studying.”
Learning objectives should be defined before considering the inclusion of animation and audio,
not vice versa. Also, it is important to consider loading time because a long loading time is
likely to discourage users. Some activities in Stuff took a long time to load due to animations
and audio.
226
Figure 16
Stuff: Hints on Pronunciation for Foreigners
Feedback. Complex activities may not be effective for learning because they demand a
lot of time from users for relatively little learning. In the Phrasal Verb Drinks Dispenser activity
in the Stuff site, participants were to change verbs, particles, and meanings until they arrived
at the right combination of the three components (see Figure 13 above). All participants ex-
cept for one stated that they disliked learning through this lengthy trial and error process. As
one participant said, “Learning phrasal verbs this way is not easy, because you have to spend
a lot of time, and it is boring. It would be better if they give you answers, after a couple of
tries.” Another participant gave up on the task after spending a fairly long time on it and said
she did not like this type of learning because finding a right combination was too difficult for
her. She added, “If the website provided two of the three components and asked you to find
the other one, it would have worked.” Still another participant said that one may not learn
anything after spending so much time because the activity did not provide answers. Explor-
atory learning needs to be designed with care. If users have to explore an activity too long
without receiving any concrete feedback, they may lose interest in it or become frustrated.
For Think of a Number activity in the Stuff site (see Figure 17), very few participants actually
finished the task because the exercise required a long chain of responses, and making one
mistake in the process resulted in entire failure in completing the task. If hints were provided
for getting back to the right track after making a mistake, the activity may have been per-
ceived as more useful by the participants.
Figure 17
Stuff: Think of a Number
227
In all the above cases, task complexity could have been adjusted by incorporating
some type of feedback. Potentially effective feedback for these activities could include provid-
ing hints or answers on demand from users and providing a dictionary tool. Hints for Pronun-
ciation for Foreigners in Stuff provided hints (see Figure 16 above), which alleviated some
participants’ frustration when they could not find answers easily, although the participants still
performed poorly due to the genre of the passage (poem) and the reading speed. Similarly,
the Listen to Poetry activity in English Club allowed the participants to read the script along
with the audio clip (see Figure 18), which was perceived as helpful. The participants, however,
expressed that they would like to have an option of choosing whether they wanted to listen
to the audio clip before actually looking at the script. When providing hints or answers, it is
important to consider the ways for users to control the amount of help they receive, depend-
ing on their language ability.
Figure 18
English Club: Listen to Poetry
Many participants liked the Help Center in ESL Cafe in which learners could post ques-
tions and receive answers. One participant explained that he was not good enough to com-
municate with his teachers because of his pronunciation, but, if he used this online function,
it would be much easier for him to ask his questions. A few participants, however, commented
that they did not like to wait to receive answers via posting and preferred instead asking
somebody they knew (e.g., their teachers) or finding answers promptly by using search tools.
A few others expressed their desire to have quiz functions; one participant mentioned that
information in ESL Cafe was good but not exciting and suggested including quizzes so that
users could check their learning themselves. For Study Zone, participants stated that it would
be more helpful if the website provided explanations for wrong answers.
When providing feedback, it is important to do so in a way not to discourage learners.

One participant was annoyed with the feedback given in Phrasal Verb Drinks Dispenser in
the Stuff site. Whenever she made a mistake, the program showed a pink object falling from
the top with a dull colliding sound and displayed the word NO!, which she considered to be
unpleasant (see Figure 19). Feedback should be provided to encourage learners to continue
learning, and feedback to wrong answers should focus on the learners’ future improvement
rather than their current mistake.
228
Figure 19
Stuff: Negative Feedback in Phrasal Verb Drinks Dispenser
SUMMARY AND CONCLUSIONS

Because learners’ ultimate goal in using ESL websites is to acquire knowledge and skills in
English, their cognitive capacity should be oriented towards learning tasks rather than figuring
out how to use the sites. In this respect, the usability of ESL websites is particularly impor-
tant, and good design of a website increases its usability.
The findings of this usability study as revealed in the site feedback questionnaire (see
Table 2 above) showed that overall the participants viewed the design of some sites (i.e., Eng-
lish Club and ESL Cafe) more favorably than the others (i.e., Stuff and Grammar & Writing).
The English Club site received the highest ratings of all five sites in terms of its information,
interface, and interaction designs, and participants had the fewest problems in navigating
the site while performing their assigned tasks as indicated in the site specific tasks discus-
sion above. Such results indicate that English Club and ESL Cafe met ESL learners’ needs
to a large extent. The results also showed that the participants were computer literate (see
Table 3 above). All had web surfing experience and access to the Internet, and many of them
used the web as a vehicle to learn English. This indicates a trend that as the web becomes a
common tool at home and in classrooms, more ESL learners will rely on websites to develop
their language skills. The design of ESL websites will become a significant factor to ensure
their success and desirable learning outcomes. We have also found that more experienced
computer and web users tended to have more critical views in evaluating websites (see Table
4 above). Creators of websites need to pay more attention to design practices as web technol-
ogy matures and more people become experienced users.
As we observed our participants using the five sites, many usability issues surfaced.
These issues, manifested less in some sites (e.g., English Club) but more in others (e.g., Stuff
and Grammar & Writing), highlighted the importance of three design aspects: information,
interface, and interaction. The insights we gained point to design features that are important
to include in ESL sites to facilitate learning. Three main issues are important to consider for
designing ESL sites.
First, ESL sites tend to carry a large amount of content because the development of
language skills and knowledge requires exposure to a great amount of information. This com-
plicates the design of content organization and navigation in multiple ways. In order for us-
ers to locate desired information or exercises, content needs to be presented in an organized
structure with meaningful categories recognizable to the users. Although one may be tempted
to squeeze as much information as possible into a limited space, long pages crowded with a
lot of information written in small fonts with narrow line spacing overwhelm users, and exces-
sive scrolling hinders the visibility of information. Using common and established navigational
devices such as search functions, links to the home page, and breadcrumbs will help users
locate information quickly and enable them to identify where they are in the site and to move
efficiently through the site.
229
Second, users of ESL websites are trying to understand how to use the site in their
second language. Special caution is required in deciding what words to use for labels and
instructions. Technical terms should be avoided as much as possible, and labels and other
language elements that are meaningful from a learner’s perspective should be used. Because
users’ background knowledge and proficiency are likely to vary significantly, reference tools
such as a glossary or links to dictionaries may also be useful.
Third, ESL websites often include learning activities such as quizzes, puzzles, and
games. The design of learning activities should be guided by the learning objectives and SLA
theory rather than the use of technology. Multimedia and feedback should be provided in ways
to maximize learning.
Given the findings of this study, we recommend designers of ESL websites consider the
following questions:
1. Is your site organized by level (e.g., beginner, intermediate, and advanced),

skills (e.g., reading, writing, grammar, and speaking), themes or topics, and
types of tasks (e.g., practice of irregular verbs)?
2. Are the pages easy to view in terms of page length, layout, text density,
color, and font size?
3. How many clicks does it take to get to a specific place in your site, and can
users find their way back to the main page easily? Should you add search
functions, links to the home page, and/or breadcrumbs to your site?
4. What prior knowledge and level of English proficiency are assumed to make
use of the labels and categories?
5. Will a learning activity or task on the site support SLA? Are the objectives of
an activity and how to accomplish these objectives clearly communicated?
Websites are potentially powerful learning resources to a virtually limitless number of

ESL learners. The design of ESL learning sites is a critical factor in how websites enhance their
language-learning experience. Increasing the usability of an ESL website can lead to a more
positive user experience and ultimately increase learning outcomes.
NOTE
1
After this article was accepted for publication, a number of the pages from the evaluated sites have
been changed.
REFERENCES
Gu, P. (2002). Web-based project learning and EFL learners: A Chinese example. Teaching English with
Technology: A Journal for Teachers of English, 2 (4). Retrieved October 24, 2005, from http://
www.iatefl.org.pl/call/j_article10.htm
Hao, Y. W., & Liu, M. (2006). Students’ attitudes toward online interaction. Academic Exchange Quar-
terly, 10 (4), 74-79.
Hoecherl-Alden, G. (2000). Turning professional: Content-based communication and the evolution of a

cross-cultural language curriculum. Foreign Language Annals, 33 (6), 614-621.
230
Krug, S. (2005). Don’t make me think: A common sense approach to web usability (2nd ed.). India-
napolis, IN: New Riders Press.
Kung, S. C., & Chuo, T. W. (2002). Students’ perceptions of English learning through ESL/EFL websites.
TESL-EJ, 6 (1). Retrieved July 10, 2006, from http://www-writing.berkeley.edu/tesl-ej/ej21/
a2.html
Lin, A. (2003). An initial study on EFL learners’ attitude towards multimedia application in language
learning. Teaching English with Technology: A Journal for Teachers of English, 3 (2). Retrieved
October 24, 2005, from http://www.iatefl.org.pl/call/j_prev.htm
Mayhew, D. J. (1999). The usability engineering lifecycle: A practitioner’s handbook for user interface
design. San Francisco: Morgan Kaufmann.
Nielsen, J. (n. d.). How to conduct a heuristic evaluation. Retrieved October 24, 2005, from http://www.
useit.com/papers/heuristic_evaluation.html
Nielsen, J., Snyder, C., Molich, R., & Farrell, S. (2000). E-Commerce user experience: Methodology of the
study. Retrieved October 24, 2005, from http://www.nngroup.com/reports/ecommerce/meth
odology.html
Norman, D. A. (2002). The Design of everyday things. New York: Basic Books.
Norman, D. A., & Draper, S. (1986). User centered system design: New perspectives on human-com-
puter interaction. Hillsdale, NJ: Lawrence Erlbaum Associates.
Osuna, M., & Meskill, C. (1998). Using the world wide web to integrate Spanish language and culture:
A pilot study. Language Learning & Technology, 1 (2), 71-92. Retrieved October 29, 2007, from
http://llt.msu.edu/vol1num2/article4/default.html
Peterson, M. (1998). The virtual learning environment: The design of a website for language learning.
Computer Assisted Language Learning, 11 (4), 349-361.
Phyo, A. (2003). Return on design: Smarter web design that works. Indianapolis, IN: New Riders.
Plass, J. L. (1998). Design and evaluation of the user interface of foreign language multimedia software:
A cognitive approach. Language Learning & Technology, 2 (1), 35-45. Retrieved October 29,
2007, from http://llt.msu.edu/vol2num1/article2
Preece, J., Rogers, Y., & Sharp, H. (2002). Interaction design: Beyond human-computer interaction.
Hoboken, NJ: John Wiley & Sons.
Rubin, J. (1994). Handbook of usability testing: How to plan, design, and conduct effective tests. Hobo-
ken, NJ: John Wiley & Sons.
Shedroff, N. (1994). Information interaction design: A unified field theory of design. Retrieved March 15,
2006, from http://www.nathan.com/thoughts/unified/index.html
Silver, M. (2005). Exploring interface design. Clifton Park, NY: Delmar Learning.
Susser, B., & Robb, T. N. (2004). Evaluation of ESL/EFL instructional web sites. In S. Fotos & C. Browne
(Eds.), New perspective on CALL for second language classrooms (3rd ed.) (pp. 279-295). Mah-
wah, NJ: Lawrence Erlbaum Associates.
Vest, J., Crowson, W., & Pochran, S. (2005). Exploring web design. Clifton Park, NY: Delmar Learning.
Warschauer, M. (2005). Socio-cultural Perspectives in CALL. In J. Egbert & G. M. Petrie (Eds.), CALL
research perspectives (pp. 41-51). Mahwah, NJ: Lawrence Erbaum Associates.
Warschauer, M., Turbee, L., & Roberts, B. (1996). Computer learning networks and student empower-
ment. System, 24 (1), 1-14.
231
Appendix A
Websites on ESL
Websites Instruction/ Target Skill area(s) Media

Resource audience
01 http://www.stuff.co.uk/ instruction student comprehensive (gram- text, audio,
wicked.htm mar, vocabulary, writing, quizzes with
pronunciation, reading, feedback,
listening, spelling) graphics
02* http://doug-j.org/index. instruction/ student vocabulary, reading text, interac-
html resource tive quizzes
03 http://www.geocities. instruction/ student/ grammar, vocabulary text, graph-
com/pccprep/qz.html resource teacher ics, interactive
quizzes
04 http://www.projec instruction student comprehensive (gram- text, interac-
tx2002.org/frameset. mar, culture, pronuncia- tive quizzes
html?interactive_english. tion, reading, vocabulary)
htm
05 http://www.elfs.com/ instruction student comprehensive (vocabu- text, interac-
ELFsX.html lary, grammar, writing, tive quizzes,
pronunciation, reading, audio, graph-
listening), TOEFL test ics
preparation
06 http://www.englishclub. instruction/ student/ comprehensive (speak- text, audio,
com resource teacher ing, listening, reading, graphics
writing, vocabulary,
grammar, pronunciation)
07 http://www.manythings. instruction student comprehensive (vocabu- text, interac-
org lary, grammar, reading, tive quizzes,
writing, listening, spell- audio, graph-
ing, culture) ics
08 http://iteslm.org/ESL/ resource student/ comprehensive text, audio,
html teacher video
09 http://grammar.ccc.com instruction student writing, grammar text, graph-
mnet.edu/grammar/ ics, interactive
quizzes
10 http://www.study.com instruction/ student/ listening, writing text, graphics,
resources teacher audio
11 http://towerofenglish. instruction/ student comprehensive text, graphics
com resources
12 http://www.eslcafe.com instruction/ student/ comprehensive (gram- text, graphics,
resources teacher mar, vocabulary, idiom, sound, inter-
phrasal verbs, words, active quizzes
slang, culture, reading,
writing, punctuation)
13 http://web2.uvcs.uvic. instruction student comprehensive (reading, text, graph-
ca/elc/studyzone/index. writing, grammar, vocab- ics, interactive
htm ulary, puzzles) quizzes
14* http://lc.byuh.edu/CNN- instruction student listening, grammar, vo- text, audio,
N/CNN-N.html cabulary interactive
quizzes,
232
15 http://a4esl.org/q/h instruction student/ grammar, vocabulary, text, interac-

teacher games tive quizzes
16* http://deil.lang.uiuc. instruction/ student grammar, reading text
edu/web.pages/gram resource
marsafari.html
17* http://www.hut.fi/ instruction student grammar text, graph-
~rvilmi/help/grammar_ ics, interactive
help quizzes
18 http://smccd.net/ac instruction student grammar, writing, listen- text, graphics,
counts/sevas/esl/gram ing audio, interac-
check/index.html tive quizzes
19 http://www.edunet.com/ instruction/ student grammar text, audio,
english/sounds/a-e.au resources quizzes with
feedback
20 http://owl.english/pur instruction/ student grammar text
due.edu/handouts/esl/ resources
index.html
21* http://ilcw.doshisha. instruction student grammar text, quizzes
ac.jp/users/kkitao/class/ with feedback
material/quiz
22 http://online.ohlone. instruction student grammar text, quizzes
cc.ca.us/~mlieu with feedback
23 http://www.esl-lab.com instruction student listening text, audio
24 http://www.americanac instruction student pronunciation text
cent.com/analysis.html
25 http://www.intranet. instruction student vocabulary text
csupomona.edu/
~jasanders/judi2.html
26 http://www.geocities. instruction student idioms text
com/athens/aege
an/6720
27 http://www-personal. instruction/ student vocabulary text
umich.edu/~cellis/an resource
tagonym.html
28 http://www.intranet. instruction student slang text
csupomona.edu/
~jasanders/judi2.html
29 http://www.aitech.ac.jp/ resources teacher articles, teaching tech- text
~iteslj niques, lesson plans
related to ESL
30 http://www.eslpartyland. instruction/ student/ grammar, idioms text, graphics
com resources teacher
31 http://www.npr.org resources student/ listening text, audio
teacher
32 http://www.toefl.org resources student general information on text
TOEFL test
33 http://esl.miningco.com resources/ student comprehensive (gram- text
instruction mar, vocabulary, reading,
writing, pronunciation),
exam preparation
233
34* http://lrs.ed.uiuc.edu/ resource teacher reading, vocabulary text, audio

Impact
35* http://www.ql.umbc. resource/ student/ comprehensive text
edu/~kpokoy1/gram instruction teacher
mar1.htm
36 http://www.eslbears. resources/ student/ comprehensive (gram- text, sound,
homestead.com/Index. instruction teacher mar, vocabulary, reading, quizzes with
html writing, listening, games) feedback
37* http://www.bbc.co.uk/ resource student comprehensive text
worldservice/learnin
genglish/compuspeak
* These URLs were no longer available at the time of the submission of this manuscript.
Appendix B
Tasks for http://www.stuff.co.uk/wicked.htm
(One of the five sites tested in the study)
Task 1: Specific Questions1

1. Learn a phrasal verb
You decided to learn one phrasal verb (combinations of verbs and particles, for example, “give
up”) every day.
(a) Using this site, find one phrasal verb that you did not know well before.
(b) How long did it take you to find the phrasal verbs on the Web site?
!!!STOP and tell the assistant that you’ve finished this task!!!
2. Learn Pronunciation
Now you are working on your pronunciation.
(a) Find out from the Web site how to pronounce [æ]and [ɘ]. Please pronounce the sounds to
the observer.
(b) How long did it take for you to find the answer?
* The first task (Task 1: First impressions), in which users are asked to explore the site freely and tell
their impression about the site, is not included on these task sheets for users because it was orally
conducted. Therefore, on these task sheets, the second task is labeled as Task 1 (Specific Questions),
and the third task is labeled as Task 2 (Open-Ended Tasks).
234
(c) In your dictionary it said that you can pronounce “amino” either as [æmi:nou] or as
[ɘmi:nou]. Please pronounce the word to the observer in the two ways.
Task 2: Open-ended Tasks
1. Listen to a poem.
Click “Hints on Pronunciation for foreigners” and work on the activity for 3 minutes.
(a) What did you learn from this activity about pronunciation?
(b) Do you think this information is useful?
2. Think of a number
Click “Think of a number” and go through the activity.
(a) Did you get the animal you were thinking of at the end?
(b) Was it fun?
235
Appendix C
Survey Instruments used in the Study
Background Questionnaire
User ID #
1. Demographic Information
1) Your name:
2) Your age:
16-20 21-25 26-30 31-35 36-40 41-50 Over 50
3) Your gender:
Male Female
4) Do you prefer using a Mac or PC ?
2. Computer Experience
5) Where do you access the Internet?
home work both other (specify )
6) How much do you use the Internet per day?
less than an hour 1-2 hours 2-4 hours
4-7 hours 8 hours or more
7) What do you generally use the Internet for (check all that apply)?
Study Research Education Entertainment News
Shopping Email Other
If other, please specify:
8) How long have you been using the computer?
less than 2 years 2-5 years 5-10 years
10-15 years more than 15 years
236
9a) Please rate your level of computer experience by drawing a circle.
Not experienced at all I-----------I-----------I-----------I-----------I very experienced

1 2 3 4 5
9b) Please, rate your web surfing experience.
Not experienced at all I-----------I-----------I-----------I-----------I very experienced

1 2 3 4 5
3. Website Usage for English
10) Have you used the Internet to study English? yes no
If yes, go to question 11. If no, this is the end of the questionnaire.
11) For what specific purpose do you use the Internet when you study English?
12) How long per week do you use the Internet when you study English?
less than an hour 1-2 hours 2-4 hours
4-7 hours 8 hours or more
13) Name 1 or 2 most frequently visited sites
14) Rate how much you like using the web to learn English.
Not at all I-----------I-----------I-----------I-----------I very much

1 2 3 4 5
15) How long have you been learning English?
less than 2 years 2-5 years 5-10 years
10-15 years more than 15 years
16) Rate your English skills.
intermediate I-----------I-----------I-----------I-----------I advanced

1 2 3 4 5
237
Site Feedback Questionnaire
User ID: Date: Tested site:
This questionnaire is for understanding how you feel about the site you tested. Please rate the
site by circling the appropriate number. As you evaluate other sites, you may change ratings
for this site later.
Very Unsatisfied Neutral Very Satisfied

1. Ease of finding the information/activities 1 2 3 4 5
Comment:
2. Quality of information/activities 1 2 3 4 5
Comment:
3. Ease of reading the text 1 2 3 4 5
Comment:
4. Appearance of site, including colors and graphics 1 2 3 4 5
Comment:
5. Speed of pages displaying 1 2 3 4 5
Comment:
6. Fun, entertainment value 1 2 3 4 5
Comment:
7. Overall learning experience 1 2 3 4 5
Comment:
8. Ease of understanding the instruction for activities 1 2 3 4 5
Comment:
9. Ease of moving around the site without getting lost 1 2 3 4 5
Comment:
10. Mark how useful this site is for improving the English skills you practiced.
Check the skills

that you practiced Useless Neutral Very Useful
Speaking 1 2 3 4 5
Listening 1 2 3 4 5
Reading 1 2 3 4 5
Writing 1 2 3 4 5
Grammar 1 2 3 4 5
Culture 1 2 3 4 5
11. How likely are you to return to this site on your own?
No way -2 -1 0 1 2 I’ll probably return the next time

I sit down at my computer.
238
Explain why you are or are not likely to return to this site.
12. Would you recommend using this site to your friends who are learning English?
No way -2 -1 0 1 2 I’ll definitely recommend this site.
Explain why you would or you would not recommend this site.
13. What do you like best about this site?
14. What do you like least about this site?
AUTHORS’ BIODATA
Min Liu, Ed.D., is Professor in the Instructional Technology (IT) Program at the University of
Texas at Austin. She is also the Coordinator and Graduate Advisor for the IT Program. She has
been teaching courses on new media design, production, and research for over 15 years. Her
teaching and research interests center on the impact of new media technology on learning
and the design of engaging and interactive learning environments for all age groups. She also
has strong interests in technology use for second language learning. She publishes widely in
leading instructional technology research journals and serves on a number of editorial/manu-
script review boards. She is active in professional organizations such as AACE, AERA, and
ISTE. She has directed and managed both CD-ROM and web-based development projects,
including award winning ones.
Tomoko Traphagan is a research associate in the Division of Instructional Innovation and

Assessment at the University of Texas at Austin. She has been involved in the research and
evaluation of various instructional technology programs at the university, including webcast-
ing, Second Life, digital media services, course management systems, web-based inquiry-
based learning, and usability of instructional websites. Her research interests lie in the impact
of technology on students’ learning behaviors and performance.
Jin Huh is Assistant Professor in the Department of English at the Korea Military Academy. He
is a graduate of the Foreign Language Program, Curriculum and Instruction at the University
of Texas at Austin. His areas of interest include teaching foreign language pronunciation, ac-
cent modification, multimedia-aided language teaching, and teaching writing.
239
Young-Ihn Koh recently received her Ph.D. degree from the Foreign Language Education Pro-
gram at the University of Texas at Austin and is now a part-time lecturer at Dongduk Women’s
University and Hanyang University in Seoul, Korea. Her research interests include teaching
and learning with technology, especially the use of computer-mediated communication with
young and adult ESL learners, teaching culture, and bilingual children’s codeswitching pat-
terns.
Gilok Choi is a Ph.D. student in School of Information at the University of Texas at Austin.
Her area of specialization is in human computer interaction (HCI), user interface design, and
usability studies and methods.
Alison McGregor is a graduate from the Educational Psychology Department at the University
of Texas at Austin. Her research interests include technology in language learning, skill devel-
opment, and prosodic features of American English. She has worked in the ESL field for over
10 years and has taught English in Taiwan, Japan, and the United States. She specializes in
accent modification for nonnative English speakers.
AUTHORS’ ADDRESSES
Min Liu
Instructional Technology Program
Department of Curriculum and Instruction
1 University Station D5700
Austin, TX 78712
Phone: 512 471 5211
Fax: 512 471 8460
MLiu@mail.utexas.edu
Tomoko Traphagan
Division of Instructional Innovation and Assessment, Continuing and Innovative Education
P.O.Box 7246
2616 Wichita Street
Austin, TX 78713-7246
Email: tomoko.traphagan@austin.utexas.edu
240
View publication stats

Designing Websites For ESL Learners: A Usability Testing Study

Загружено:

Сведения о документе

Оригинальное название

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

Designing Websites For ESL Learners: A Usability Testing Study

Загружено:

Авторское право:

Доступные форматы

See discussions, stats, and author profiles for this publication at: https://www.researchgate.

Designing websites for ESL learners: A usability testing study

Article · January 2008

Min Liu Alison McGregor

SEE PROFILE SEE PROFILE

SNS and MOOC View project

Alien Rescue View project

The user has requested enhancement of the downloaded file.

Designing Websites for ESL Learners:

Investigating design effectiveness of ESL websites requires establishing good design

CALICO Journal, 25 (2), p-p 207-240. © 2008 CALICO Journal

REVIEW OF THE LITERATURE

1. What are intermediate-level ESL learners’ perceptions of the design fea-

1. Stuff for English Learners (http://www.Stuff.co.uk/wicked.htm)

Grammar & Writing

Participants and Site Order

1. What did you learn from this activity about pronunciation?

Site feedback questionnaire

Procedure and Testing Facility

RESULTS AND DISCUSSION

Participants’ Responses in the Site Feedback Questionnaire

Findings from Site Task Performance Data1

Overall site design issues

Previous research on the organization of information has highlighted the importance

Design of learning activities

In using Grammar & Writing, participants’ comments focused on navigation difficulties

When providing feedback, it is important to do so in a way not to discourage learners.

SUMMARY AND CONCLUSIONS

1. Is your site organized by level (e.g., beginner, intermediate, and advanced),

Websites are potentially powerful learning resources to a virtually limitless number of

Hoecherl-Alden, G. (2000). Turning professional: Content-based communication and the evolution of a

Websites Instruction/ Target Skill area(s) Media

15 http://a4esl.org/q/h instruction student/ grammar, vocabulary, text, interac-

34* http://lrs.ed.uiuc.edu/ resource teacher reading, vocabulary text, audio

Task 1: Specific Questions1

Task 2: Open-ended Tasks

(b) Do you think this information is useful?

(b) Was it fun?

16-20 21-25 26-30 31-35 36-40 41-50 Over 50

4) Do you prefer using a Mac or PC ?

5) Where do you access the Internet?

home work both other (specify )

6) How much do you use the Internet per day?

less than an hour 1-2 hours 2-4 hours

4-7 hours 8 hours or more

Study Research Education Entertainment News

Shopping Email Other

If other, please specify:

8) How long have you been using the computer?

less than 2 years 2-5 years 5-10 years

10-15 years more than 15 years

9a) Please rate your level of computer experience by drawing a circle.

Not experienced at all I-----------I-----------I-----------I-----------I very experienced

9b) Please, rate your web surfing experience.

Not experienced at all I-----------I-----------I-----------I-----------I very experienced

3. Website Usage for English

10) Have you used the Internet to study English? yes no

If yes, go to question 11. If no, this is the end of the questionnaire.

less than an hour 1-2 hours 2-4 hours

4-7 hours 8 hours or more

13) Name 1 or 2 most frequently visited sites

Not at all I-----------I-----------I-----------I-----------I very much

15) How long have you been learning English?