EVALUATION CHECKLIST IMPLEMENTATION 1

Evaluation Checklist Implementation

Sarah van Nostrand

Colorado State University

EVALUATION CHECKLIST IMPLEMENTATION 2

Scenario
In creating my evaluation checklist, I had a specific scenario in mind (please see

Appendix 1 to view my evaluation checklist). I imagined being an ESL teacher for a listening

and speaking class in a public high school in or around the Burlington, Vermont area. Since I am

familiar with the general demographic of Burlington, and I plan to return to teach there someday,

I decided this would be a practical scenario to evaluate websites and software for. Burlington has

a growing refugee population, largely African, with many people continuing to arrive from

countries like Somalia and Sudan annually. Burlington hosts a sizeable immigrant population as

well, with many non-native English speakers from countries such as Vietnam settling in

Burlington every year. Lastly, there are many migrant dairy farm workers around the Burlington

area, mostly, if not exclusively, from Mexico. My high school ESL class will reflect this diverse

population. I will have 19 students, and they will have all been lumped into my intermediate

level ESL listening and speaking class, however realistically their proficiency levels will range

from mid-beginner to high intermediate. Most of these students have not received the level of

formal public education that their American classmates have, and therefore many are ill-prepared

to take on the workload that the high school expects of them. Unfortunately, many of the students

are not even considering applying for college, as it seems unattainable. A more manageable goal

for many of these students is to work toward having strong enough language skills to graduate

high school. See Table 1 on the following page for a summary of student characteristics.

Table 1
Characteristics of ESL Students
Characteristics Descriptions
EVALUATION CHECKLIST IMPLEMENTATION 3

Age Range Ages vary from 14-19

Native Languages Somali, Sudanese, Spanish, & Vietnamese
Proficiency Level Ranges from mid-beginner to high-intermediate
Previous Education Ranges from no formal public education to some high school
Education Goals Majority only plan to receive GED; three or four students considering
higher education

Technology Use Varies, but all students are familiar with using the internet in some
capacity; most have smart phones

Values

I am a strong proponent of the Communicative Language Teaching (CLT) approach.

Larson-Freeman (2011) describes CLT as working toward the goal of communicative

competence. Particularly in regard to my scenario as a high school ESL teacher, my focus will be

learner-centered, with communicative competence a driving force in my teaching. I believe in

using language to learn language, which can be accomplished in part through ample interaction,

feedback, and use of authentic materials. With that, I am in support of the constructivist view of

learning. On the opposite end of the spectrum, in the cognitivist view, learning is teacher-

centered, whereas on the other end of the continuum, constructivism is learner-centered and more

clear like CLT. I want to continuously challenge and engage my students with material that is

relevant and of use to them. Another reason I lean heavily toward the constructivist view of

learning is, according to Reinhardt and Thorne (2011) a constructivist interface better

facilitates collaboration at the computer (p. 163). I believe it is important to incorporate these

various Second Language Acquisition (SLA) theories into planning CALL activities, as

discussed by Kessler and Bikowski (2011). The following section is a justification for the five

categories that serve as the framework for the evaluation form, as they relate to my theoretical

and pedagogical values.

Interactivity
EVALUATION CHECKLIST IMPLEMENTATION 4

Interaction is a vital component of my CLT-based, constructivist view of language

learning. Particularly in such a diverse high school listening and speaking class context where

native languages and comfort levels vary greatly, I feel it is very important for the students to be

able interact with their web-based materials. With interaction comes collaboration, and Reinhardt

and Thorne (2011) discuss that a positive and effective computer-based learning environment

requires good collaborative learning practices that focus on learner centeredness and learner

interaction. There is a lot of software available that provides interactive material and activities

that can effectively supplement my ESL class.

Feedback
Feedback is a natural follow-up to interaction and an important value to consider. It is

absolutely vital to me that my students receive adequate feedback from the software that they use

in positive and useful way. Feedback is a type of formative assessment that serves as a type of

interaction between the student, teacher, and software. The feedback should be constructive and

positive in its delivery, allowing students to use it interactively for remediation purposes

throughout the course of instruction.

Authenticity
Having access to authentic materials is yet another important value. Jamieson and

Chapelle (2010) believe that using language for realistic purposes is the best way to learn.

Especially with high school ESL students, having easy, quick, and abundant access to authentic

materials online is so important. Whether they are searching for GED study guides, the latest

news headlines, music videos, or sample resumes and job applications, this type of real-life

language is far more valuable to them than the contrived pedagogical materials that is so often

found in traditional textbooks.

Four Strands of Language Learning
Nation (2007) discusses the importance of well-balanced language learning. This is

achieved through the implementation of the Four Strands of Language Learning. These include
EVALUATION CHECKLIST IMPLEMENTATION 5

learning through meaning-focused input, meaning-focused-output, language-focused learning,

and fluency development. Since every activity in an ESL class fits at least one of the four

strands, no matter the context, it is a useful value to consider while using computer software as

well, even if the web-based resources act as supplemental to the actual lesson.
Practicality and Ease of Use
Practicality is an important value. Students, especially teenagers, are drawn to the

internet, but if a website or piece of software is confusing, cluttered, expensive, boring, or

irrelevant in any way, they will be quick to dismiss it. Students should be able to quickly and

easily access material. If the software does in fact cost money, the students should not be the

ones to pay for it. Ideally, they should also be able to access online materials inside and outside

of the classroom.
Relevance to CALL programs
The software that I chose to evaluate for my checklist implementation largely reflects my

theoretical teaching scenario, as well as my values. Since I am teaching an intermediate ESL

listening and speaking class to high school students, there are a number of things I had to keep in

mind when choosing the software to evaluate. The success or failure of software is determined

largely by contextual factors and how those factors interact with a specific class or learner

setting, Hubbard (2006). So, keeping the high school context in mind, the software has to be

easily accessed by the students. This means, in part, that it must be free for them. The school

may have to pay a fee, like with the 8-in-1 Dictionary, but it should be readily available to the

students, preferably in multiple settings. All of the software is intended to serve as supplemental

material. Therefore, the classroom should not be the only appropriate place for students to access

and utilize them. Apps like Duolingo provide a free, fun, and quick way for students to practice

various listening and speaking skills wherever they are. Google Translate is a helpful translation

app that students can use anytime and virtually anywhere, whether it is to aid in a homework
EVALUATION CHECKLIST IMPLEMENTATION 6

assignment such as a selfie video, or in ordering food at a restaurant. Therefore, ease of use and

practicality was key in identifying the right kind of software to evaluate.

I included sites like BrainPop, Kahoot!, and StoryBird (see Appendix 2) because I want

my students to be creative and promote interaction, not only with the technology itself but also

via student forums and shared online projects. Ultimately, I hope that this motivates the students

to learn. Steele and Levy (2013) note that activities that are not well liked were not nearly as

useful as those activities that are well liked. These programs allow for students to create their

own accounts and profiles, work at their own pace, and practice and produce something original.

Asmentionedaboveundervalues,interaction,feedback,authenticity,theFourStrands,and

practicalityandeaseofusewerethemajorcategoriesIincludedonmyevaluationform,aswell

asafinalcommentssectionattheendtosummarizefinalthoughtsaboutthesoftware.

Checklist Function and Revision

After completing my evaluations, I discovered that there are many areas on my

evaluation checklist that need to be improved. Some of these changes are not all that surprising

to me, while others had not even crossed my mind until after I had implemented the checklist

myself. The following section will outline the several shortcomings of my evaluation checklist

and how I plan to resolve them for future use.

First, it became clear early on in my evaluations that I like to write a lot of comments. I

have a hard time simply checking a box or committing to a single number; I want to expand on

that point by something specific I noticed, and I feel the need to justify my decisions with

copious comments and random facts about the software. With that said, I did not have enough

space to write (neatly) all of the comments and notes that I wanted to make. The other problem

that comes along with this is that by writing so much in the comments section (and all along the

margins throughout the checklist) it does not make it very appealing to read. Someone could be
EVALUATION CHECKLIST IMPLEMENTATION 7

easily tempted to skip over it altogether. The solution I came up with for this is to keep the

comments section at the end the same size, but perhaps put four or five bullet points on the side

to encourage less scribbling and to be more concise. Another method to minimize, or at the very

least organize, the comments at the end is to move that information to each individual category.

In other words, I plan to add a line or two of space under each category, or section, so that I can

write notes as I go, and that will hopefully make it easier to keep track of the points I want to

make.
A second problem that I ran into frequently had to do with my Likert scale itself. I had

thought that a simple five-point scale would be fitting for these types of evaluations, but I often

caught myself wondering if it would make any difference if I checked the first or second box.

What really is the difference between disagree and strongly disagree? Both, to me, mean that the

software did not meet the criteria in question. The same goes with the fourth and fifth boxes,

although I found it slightly easier to check the fifth box if I was really excited about the software.

I do not have a definite answer to this problem, but I think a good place to start would be to

change the wording of my Likert scale and the wording of the statements, so that, for example,

each number on the scale represents a clear feeling or description that is unique from the others.
Another problem I found with my evaluation form is that it caters heavily toward all-

encompassing language learning sites, as opposed to specific language learning tools, for

example. This made it difficult to evaluate applications like Google Translate, where the majority

of my criteria did not even apply, even though overall I gave it the highest score possible because

it is incredibly useful for what it is. I am not sure if it makes sense to develop different checklists

depending on the type of software being evaluated, or if there is a way to develop one that better

caters to all types of computer-assisted language learning.

The final problem I had with my checklist was, depending on the type of resource that is

being evaluated, the price as well as type of payment varies. For example, English Central is a
EVALUATION CHECKLIST IMPLEMENTATION 8

software that is paid via monthly subscription, whereas the 8-in-1 Dictionary is a one-time fee.

Additionally, some software offers limited access to a free version while charging for a premium

version. In revising my checklist, I will plan to add a line at the very top that asks for the

evaluator to specify whether the software is totally free, subscription based, requires a one-time

fee, or offers limited free access. To me, this is very important information that is a large

determining factor in whether or not the software will be utilized. A single box toward the end of

the form simply asking is the software is free is not adequate.

The aforementioned points stand out as possible revisions for the checklist. However, I

also want to make note of what went well with the checklist. In a general sense, I was pleased

with the overall simplicity of the form itself, which made the process of evaluating software very

straightforward. Although I have some possible changes I would like to make to the Likert Scale,

I liked that the scale was simple and consistent throughout the criteria. This made it easy to get a

sense of the software by just taking a quick glance at where the checkmarks were placed since

the values never changed. One other aspect of the checklist that I was happy about was the

section designated for comments at the end of the form. I mentioned not having enough space to

write, but the placement of the comments section was good. I also liked this because it gave the

evaluator a chance to make general comments about the software, as opposed to scattering one or

two comments specific to each criteria throughout the form. This also allows readers of the

evaluation forms to see all of the comments in one place.

EVALUATION CHECKLIST IMPLEMENTATION 9

References

Kessler, G., & Bikowski, D. (2011). The influence of SLA training in curricular
design among teachers in preparation. CALICO Journal, 28(2), 522-545.

Jamieson, J., & Chapelle, C. A. (2010, June 3). Evaluating CALL use across multiple contexts.
ScienceDirect, 38, 357-369.

Larsen-Freeman, D., & Anderson, M. (2011). Techniques & principles in language teaching.
Oxford: Oxford University Press.

Nation, I. S., & Newton, J. (2009). Teaching ESL/EFL listening and speaking. New York:
Routledge.

Reinhardt, J. & Thorne, S. (2011). Beyond comparisons: Frameworks for developing digital L2
literacies. In N. Arnold and L. Ducate (Eds.), Present and Future Promises of CALL: From
Theory and Research to New Directions in Language Teaching, 257-280. San Marcos, TX:
CALICO.

Steel, C. H., & Levy, M. (2013). Language students and their technologies: Charting the
evolution 20062011. ReCALL, 25(03), 306-320.

Appendix 1
Evaluation Checklist

CALL Software Evaluation

Directions:
Fill in pertinent information. Use the Likert scale below to check the box that best
answers/describes how you feel regarding each statement in the leftmost column. There is space
for final comments at the end of this form.
EVALUATION CHECKLIST IMPLEMENTATION 10

Name of evaluator: ____________________________________

Name of software being evaluated: ____________________________________

Date of evaluation: ____________________________________

URL address of software being evaluated: ____________________________________

1 2 3 4 5
Strongly Disagree Undecided Agree Strongly
Disagree N/A Agree
or No or Maybe or Yes

Part 1: LANGUAGE LEARNING POTENTIAL AND USEFULNESS

Interactivity________________________
________________________
Language materials, games and activities
simulate real communicative events.
Games/activities/materials promote
discussion (written and/or oral).
Learners are given opportunities to express
ideas and opinions regarding content
(written and/or oral).
Learners have opportunities to practice
language beyond the sentence level.
Learners have access to interactions using
this software outside of the classroom.
Learners have access to both individualized
and group interactions using this software.

Feedback_________________________________________________ 1 2 3 4 5
_
The amount of feedback is appropriate.
The feedback received is useful and constructive.
The feedback is immediate.
Learners have opportunities to use feedback for remediation.
Feedback provides individualized suggestions and/or examples for
learners.
EVALUATION CHECKLIST IMPLEMENTATION 11

Authenticity____________________________________________ 1 2 3 4 5
_____
Realistic (if not authentic) language is used consistently throughout
games/activities.
A variety of authentic materials (such as newspaper articles or
weather forecasts, for example) are used.
Learners have opportunities to build language skills as theyre used
in authentic communication.

Four Strands of Language Learning*________________________ 1 2 3 4 5

Sufficient opportunities for meaning-focused input are presented (through
listening and reading activities).
Sufficient opportunities for meaning-focused output are presented
(through speaking and writing activities).
Sufficient opportunities for fluency development are presented (using all
four different skills [Reading, Writing, Listening, and Speaking]).
Sufficient opportunities for language-focused learning are presented
(learning language form through repetitious activities such as drills).

Part 2: PRACTICALITY & EASE OF USE OF SOFTWARE

Practicality_&_Ease_of_Use_________________________________ 1 2 3 4 5
Software is logically organized and easy to navigate.
Instructions are clear and easy to follow.
Software is visually appealing and engaging.
This software is free to use.
This software requires learners to enter an active email account.
Software has ability save and track learner progress/work.

Final comments:

Rate this software overall. Circle the most appropriate number, with 1 being very unsatisfactory
and 5 being very satisfactory.

1 2 3 4 5
EVALUATION CHECKLIST IMPLEMENTATION 12

* Paul Nation, 2007.

Appendix 2
List of evaluated software
Table 2
List of evaluated software
Name of software evaluated URL Address
StoryBird storybird.com
Kahoot! getkahoot.com
Daves ESL Caf eslcafe.com
English Central englishcentral.com
Randalls ESL Cyber Listening Lab esl-lab.com
Brain Pop brainpop.com
Duolingo App duolingo.com
Grammarly grammarly.com
8-in-1 English Dictionary englishlearning.com
Google Translate translate.google.com