Task-Related Variation in Computer-Assisted Language Learning 2012

Task-Related Variation in Computer-Assisted Language Learning
Author(s): TRUDE HEIFT and ANNE RIMROTT

Source: The Modern Language Journal, Vol. 96, No. 4 (Winter 2012), pp. 525-543
Published by: Wiley on behalf of the National Federation of Modern Language Teachers
Associations
Stable URL: https://www.jstor.org/stable/23361714
Accessed: 08-04-2019 19:30 UTC
JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide
range of content in a trusted digital archive. We use information technology and tools to increase productivity and
facilitate new forms of scholarship. For more information about JSTOR, please contact support@jstor.org.
Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at
https://about.jstor.org/terms
National Federation of Modern Language Teachers Associations, Wiley are collaborating

with JSTOR to digitize, preserve and extend access to The Modern Language Journal
This content downloaded from 201.131.90.36 on Mon, 08 Apr 2019 19:30:21 UTC
All use subject to https://about.jstor.org/terms
Task-Related Variation in
Computer-Assisted Language
Learning
TRUDE HEIFT ANNE RIMROTT
Simon Fraser University Simon Fraser University

Linguistics Department Linguistics Department
University Drive 8888 University Drive 8888
Bumaby, BC Burnaby, BC
Canada V5A1S6 Canada V5A1S6
Email: heifi@sfu.ca Email: arimrott@sfu.ca
This study investigates task-related variation in learner performance in a computer-assisted

language learning (GALL) environment. For our study, we collected data from 15 beginner and
then intermediate second language (L2) learners of German who worked on 3 distinct activity
types over 16 months: free composition, translation, and sentence building. Study results reveal
that grammatical accuracy with respect to German word order was significantly higher with the
meaning-focused task type (i.e., free composition) for both the beginner and intermediate levels.
Moreover, proficiency level also had a significant effect on L2 word order accuracy: Beginner
students performed significandy better than intermediate learners on the two form-focused task
types (i.e., translation and sentence building). With the ultimate goal of understanding learner
performance as it relates to different task types and success in CALL, this article provides possible
explanations of these study results and suggests areas for future development of task design in
CALL.
ONE IMPORTANT DECISION WHEN DESIGN deviations from the target grammar) given that the
ing a computer-assisted language learning search
(CALL)space of possible errors is much reduced.
program is the choice of learning activities From
that thea pedagogical perspective, however, CALL
program should include for students toactivities
explore, with tightly restricted input allow for little
practice, and test target language vocabularyflexibility for the learner to explore aspects of
and grammar. Task types that are morethe tightly
target language, which is an important part
restricted with respect to the nature ofoflearner second language (L2) acquisition (see, e.g.,
input are popular among CALL program design2001; Felix, 2003). Moreover, different
Chapelle,
ers because the computational mechanisms for reflect different learning goals, and this
task types
input analysis and feedback generation are in simple
itself warrants a variety of task types. However,
provided a variety of restricted to more open
compared to those required for less constrained
input, such as free compositions. For instance,
ended CALL a activities, does learner performance
simple string match that evaluates learneractually
input vary
in across these activities, in particular, in
a one-word fill-in-the-blank exercise can, with little
form-focused as opposed to meaning-focused tasks
computational effort, provide fairly specificand, if and
so, what do these distinct learning outcomes
accurate feedback on a variety of mistakes mean(i.e.,
for researchers, teachers, and learners?
Many questions that center around the effects of
task-related variation on learner performance are
The Modem Language Journal, 96, 4, (2012) still unanswered. This applies in particular to
DOI: 10.1111/j. 1540-4781.2012.01392.x CALL environments and to investigations that
0026-7902/12/525-543 $1.50/0
require learner observations at different points in
©2012 The Modem Language Journal
time. Chapelle (2001, 2004, 2007), for instance,
526 The Modem Language Journal 96 (2012)
stresses the importance of investigating different and proficiency-related performance differences

CALL tasks to gain further insight into second as well as learner variability by inspecting these task
language acquisition (SLA) and interlanguage types at two proficiency levels, beginner and
development. Similarly, Skehan (2003a) notes that intermediate. Learner performance is measured
"the claims emerging from the second language in terms of linguistic accuracy, as one of the three
literature need to be matched by research with learner proficiency measures next to complexity
computer and Web-based materials to confirm, and fluency (see, e.g., Larsen-Freeman, 2006,
or not, that similar effects do in fact operate in 2009).1
this changed context" (p. 409). More generally, In the following, we first provide a definition of
Samuda and Bygate (2008) state that: task. We then situate our research in the literature
by examining previous work on task-related

the impact of task type performance is a topic of
variation both conducted in the regular classroom
considerable interest to second language task re as well as in a CALL environment. We then
searchers. This should not be surprising: If tasks are a
introduce our by describing its participants, ta
pedagogical tool for generating language work, then
the kind of language that arises as a result of their use is and methodology. Our results section provide
bound to be of central interest to teachers, learners, examination of task-related variation at both the
designers, and others who consider their use to beginner and intermediate level, followed by a
promote language development, (p. 47) discussion of our findings. The article concludes
with a discussion of the limitations of our study and
Accordingly, questions surrounding task-related pedagogical implications of our findings.
variation are important from more than just a
CALL design perspective. By observing learner
TASK-RELATED VARIATION
performance on several task types in a conven
tional language course as opposed to a laboratory What Is a Task1?
setting, we can also examine what learners (can)
do in the context of their usual habits and In SLA research, a number of definitions of task
practices. This knowledge can thenhave
inform
been provided, and they typically include
teachers on a number of pedagogical decisions,
language learning activities ranging from those
such as task selection with respect towith language
a primary focus on meaning to mainly form
practice materials, learner guidance, and oriented
evaluaones. For example, working in the
tion. For instance, if task type has an framework
effect onof task-based instruction, Bygate,
learner performance, then teachers may Skehan,
wantand to Swain (2001) provide a general
employ a variety of task types not only todefinition
provide of a task as "an activity which requires
learnerscan
rich practice environment in which learners to use language, with emphasis on
explore L2 constructions in a number ofmeaning,
contexts, to attain an objective" (p. 11) (for a
detailedthem
but also to assess learners fairly and to guide overview of task definitions, see Samuda &
in their L2 language development. From a CALL
Bygate, 2008, pp. 62-70). Along the same lines, for
Ellis perfor
design perspective, if individual learner (2003), tasks are activities that emphasize
mance, for example, varies across task types,
meaningthen
over form, while exercises primarily refer
an individualized CALL program that canto form-focused
accom language learning activities. In
contrast
modate learner variability would justify to emphasizing meaning, Breen's (1989)
the addi
tional computational effort. definition of a task entails all materials designed
To address some of these open research
for language teaching. This includes "a simple and
questions, we conducted a longitudinal brief
study on exercise" as well as "more complex...
practice
the acquisition of German word order workplans
by collect which require spontaneous communi
cation of meaning" (p. 187) (see also Tarone,
ing data from 15 beginner and then intermediate
learners of German who worked on three distinct 1987). Bachman and Palmer (1996), by also
activity types in a CALL environment over the placing emphasis on testing environments, share
course of 16 months: free composition (Essay), this broader definition of task. However, the
translation (Translation) and sentence buildingauthors distinguish test tasks such as multiple
choice items or compositions from language use
(S-Building). Our study investigates learner per
formance on word order in these three CALL task tasks. A test task is completed by language learners
types, which were embedded in a conventional as part of a language test, while a language use task
is defined more generally as "an activity that
language course as opposed to a laboratory, thus
providing a more authentic reflection of the involves individuals in using language for the
learners' performance. We investigate task-related
purpose of achieving a particular goal or objective
Trude Heifl and Anne Rimrott 527
in a particular situation" (Bachman & Palmer, Studies have shown that variation cannot be
1996, p. 44). attributed to the differences in task alone but that
For this study, we also adopt a wider view by consideration must also be given to the grammati
defining a task as any learning activity designed for cal constructions in question. For instance,
language learning, which can be either form examining data from 20 ESL learners in an
focused, meaning-focused, or some combination advanced university English class, Tarone (1985;
of these. Accordingly, our definition of task is not see also Tarone & Parrish, 1988) found task
restricted to activities where a focus on meaning is related variation in morphological and grammati
primary. Instead, and following others, we also cal forms (e.g., articles, 3rd person -v) in three
include what Ellis (2003) refers to as exercises, that different tasks that were not part of the students'
is, activity types with a primary focus on form. This regular curriculum: a written grammar test, an
definition is prompted by the fact that we "need to oral interview, and a narration. While in some
recognize that the overall purpose of tasks is cases, the more form-focused grammar test
the same as exercises—learning a language—the improved morpheme accuracy, in other instances,
difference lying in the means by which this grammatical accuracy was significantly better
purpose is to be achieved" (Ellis, 2003, p. 3). in the more meaning-focused oral tasks than
the grammar test. Some researchers (see, e.g.,
Adamson, 2009, pp. 139-140) believe this finding
Studies of Task-Related Variation With
Classroom Learners to be compatible with Krashen's Monitor Model in
that simple grammar rules (e.g., 3rd person -s)
Pienemann (1998) notes that "there is over can be "monitored" successfully in form-focused
whelming evidence that the shape of an interlan tasks whereas difficult rules (e.g., articles), if
guage varies within one and the same learner on monitored in a form-focused task, might result
one and the same day depending on which in decreased accuracy.2
linguistic task the learner performs in which In addition, Ellis (1994) notes that task-related
context" (p. 273). In fact, a large amount of variation is "best considered as a blanket term to
research exists on task-related grammatical and/or cover the variability evident when learners perform
phonological variation in interlanguage. Review different tasks, and is ultimately traceable to one of
ing an array of early studies on the topic, Tarone the other sources (the linguistic, situational, or
(1988) states that many studies have observed a psycholinguistic context)" (p. 135) (see also
"general 'task effect' on learner accuracy at many Samuda & Bygate, 2008; Van den Branden,
different linguistic levels" (p. 13). For instance, 2006). Linguistic context refers to the linguistic
Larsen-Freeman (1975) investigated the acquisi elements that shape or are in the environment of
tion order of 10 English morphemes (e.g., the form in question. For example, Dickerson
progressive -ing) in 24 beginner English-as-a (1975) found that Japanese learners of English
second-language (ESL) learners. Ranking the produced English /z/ more accurately when it was
morphemes in order of accuracy for each of five followed by a vowel as opposed to, for instance, by a
tasks (reading, writing, listening, imitating, and consonant such as /m/. Situational context can
speaking), the study found that morpheme also induce interlanguage variation and covers a
accuracy varied considerably across tasks. range of variables, such as the setting, topic, or
As one of the primary causes of such task-related purpose of an instance of language use, or the
variation, researchers have postulated that elicita participants involved. For example, in a letter
tion tasks that draw attention to form elicit more writing task with 26 female Japanese university
grammatically accurate performance than more learners of English, Porter and O'Sullivan (1999)
meaning-focused tasks (Tarone, 1987). In fact, found systematic variability in both the linguistic
Krashen's Monitor Model (1981) claims that tasks content and the overall letter score depending
favoring activation of the monitor enable learners on the age of the letter's intended addressee.
to pay more attention to form, and thus produce Psycholinguistic factors that affect interlanguage
more correct language. For instance, Dickerson's variation include the time available for planning a
(1974) findings support an accuracy advantage language task and whether monitoring is encour
for form-focused tasks compared to meaning aged or discouraged. In general, if a language task
focused tasks. Studying the pronunciation of /r/ is planned, it results in higher accuracy rates for
by Japanese ESL students, Dickerson found that target-language forms (see, e.g., Ellis, 1994; Foster
learners produced /r/ with 100% accuracy in word & Skehan, 1996; Skehan & Foster, 1997).
lists but only with 50% accuracy in free conversa The studies cited previously provide evidence that
tion (see also Adamson, 2009; LoCoco, 1976). a number of factors can affect the interlanguage
forms a learner uses to carry out specific language types, suggesting that the input environment also
tasks. These factors are inherently embedded affects learner behavior and learning outcomes.
and, to some extent, reflected in the task learners In contrast, Thouësny (2010) conducted a case
perform. For this reason, a language task is best study of an Irish learner of French by focusing on
viewed as a collection of characteristics rather meaning-focused tasks only. Over the course of a
than a single indivisible entity (Bachman &
semester, the student submitted 32 texts for
Palmer, 1996; Purpura, 2004), with changes correction
in in the form of a discussion forum,
the parameters of the characteristics generating
email correspondence, a few wiki texts, and one
task-related variation. essay. Each of her written texts was error encoded
and part-of-speech tagged, and extracted from the
corpus to investigate intralearner variability. Study
CALL Studies of Task-Related Variation
results showed no significant differences among
In CALL, the number of studies that have tasks for this learner. According to the author, the
lack of significant findings may be due to the
examined the impact of different task types on
learner performance is more limited, althoughlearner's level of carefulness when writing in
there are several studies that have employed
French, paying attention to grammatical and
different tasks to obtain a more comprehensive lexical features in all types of documents.
assessment of their participants' linguistic perfor Ayoun (2001) employed a mixture of form
mance (e.g., Doughty, 1991).3 For instance, Hill focused and meaning-focused tasks in a 5-week
and Laufer (2003) investigated the effects of tasklong CALL study investigating the role of feedback
type on incidental vocabulary with 96 advanced on the L2 acquisition of two French past tenses.
Chinese learners of English. In one session that University learners of French (N = 145) partici
was not part of the regular curriculum, each pated in the study outside classroom time. The
participant read a text passage containing 12 learners underwent three learning sessions, each
target words and subsequently performed one of session followed by a different task type (e.g., a
three tasks (depending on his/her treatment translation task) and finally a posttest task (essay).
group) : a meaning-focused task, a form-oriented The participants, divided into three levels of
comprehension task, or a form-oriented produc competency, performed differently on the four
tion task. Study results indicate that the two form
task types in terms of the correct application of the
oriented tasks generally enabled better word two past tense forms, with their performance
learning and retention scores than the meaning improving steadily with each testing occasion. Task
focused task and also generated a significantlyperformance also varied with the participants'
increased amount of dictionary activity. The competency level, suggesting that task effects
authors conclude that the amount of word-relatedmight differ for learners of varying L2 proficiency.
DeKeyser (1995) investigated the learning of
activity induced by a task is an important factor in
grammar rules of a pseudolanguage, Implexan.
determining task effectiveness in incidental vocab
ulary learning (see also, Laufer & Girsai, 2008;
Two treatment groups, consisting of a total of 61
Laufer & Roitblat-Rozovski, 2011). university students, were exposed to Implexan in
Focusing on form-focused tasks only, Heift 20 sessions, after which they completed a judg
(2003) also studied task-related variation in ment task (judging the correctness of an Implexan
CALL with 27 second-semester university learnerssentence) and a production task (typing an
of German in a 2-week-long study integrated intoImplexan sentence). Both treatment groups
the curriculum. Splitting the participants into(explicit-deductive learners and implicit-induc
three groups, she examined the effects of three tive learners) generally showed strong evidence of
CALL tasks (multiple choice, drag and drop, andlearning on the production task, particularly the
typed completion) on learner behavior and explicit-deductive group. On the judgment task,
learning outcomes with respect to German word however, both groups' performance was virtually
order. Regarding learner behavior during identical
a to what would have been expected by
practice phase, most errors occurred with drag chance. The study once more draws attention to
and drop. However, in terms of learning outcomesthe importance of considering the test tasks in
as measured by a posttest, drag and drop was interpreting one's findings.
significantly better than multiple choice and mar Finally, and in contrast to studying aspects of
ginally better than typed completion. AUTHOR grammar, Reinders (2010) studied the relative
speculates that one explanation for these resultseffects of listening tasks on the intake and
lies in the greater flexibility and ease of use of
acquisition of negative adverbs and adverb place
drag and drop compared to the other two taskment with 50 intermediate and upper intermediate
Trude Heift and Anne Rimrott 529
L2 learners of English. The study employed three 1. Do accuracy rates in German word order va
different tasks: a dictation, an individual recon across different CALL task types?
struction, and a collaborative reconstruction. Study 2. If so, is that variation consistent for begin
results indicate that the dictation task resulted in and intermediate L2 learners of German?
high intake but less acquisition than the individual

reconstruction task, which resulted in greaterStudy Participants
acquisition but low intake, while the results for
The data were collected from 15 L2 learners of
the collaborative reconstruction task were most
German who took three consecutive language
consistent. These findings again demonstrate task
courses at a Canadian university over the course of
related variability in learner performance.
16 months: German 100, German 101, and
German 200. Each course consisted of 13 weeks
Summary of Previous Findings of instruction. For this study, we consider data

from the beginner (German 100) and intermedi
In sum, the limited number of CALL studies that
ate course (German 200) of the 15 learners. The
investigated task-related variation found that task
participants were selected because they completed
type has an impact on learner performance. Atall
the
three German courses and submitted their
same time, the literature review identifies an homework regularly. The students had no prior
apparent need for further investigation of ques knowledge of German when first registering in
tions surrounding task-related variation in CALL German 100. Their language level was determined
because previous studies examined form-focused by university course placement.
tasks without considering meaning-focused tasks According to a questionnaire that elicited
and different proficiency levels (e.g., Heift, 2003), background information on our learners, the 9
and/or focused on vocabulary acquisition or female and 6 male students were native speakers
writing as opposed to other L2 skills (e.g., Hill & of English (9), Polish (1), Russian (1), Cantonese
Laufer, 2003; Reinders, 2010; Thouësny, 2010), (1), Mandarin (1), Tagalog (1), and Arabic (1).
and/or investigated short-term effects of non Their mean age when they enrolled in German
curricular tasks (e.g., Ayoun, 2001) and/or 100 was 19.1 years, with a range of 17 to 23 years.
employed a pseudolanguage (e.g., DeKeyser, The participants indicated that they were com
1995). fortable using computers and generally enjoyed
Accordingly, our study contributes to previous the use of computers for foreign language
research in various ways. First, it considers a practice. Moreover, given that our university
mixture of form-focused and meaning-focused has no language requirement for undergradu
task types and assesses their impact on learner ates, students who choose to learn German do so
performance with German word order. Second, for reasons of heritage, travel, business or general
our study considers task types that are integrated interest and are thus assumed to be highly
into the language curriculum as homework assign motivated.
ments, thus providing learner data that were At the beginning of each semester, the partic
collected over a total of three semesters as part ipants consented to a possible anonymous analysis
of regular L2 instruction. Third, our research of their data for research purposes.
examines learner performance at two distinct
proficiency levels by studying the same L2 learners Data Collection
performing the same three task types first at a

To investigate the impact of task type on the
beginner and then at an intermediate level. The
performance of L2 learners of German, our study
data thus allow us to examine task-related variation
relied on three different task types that students
from L2 learners of German not only in terms of
course-specific performance at two proficiency
completed in an online CALL program for
German during their beginner and intermediate
levels but also with regard to learner variability
semesters. Students' interaction with the CALL
across task types and time.
system was recorded in order to examine their
responses in view of errors in word order.
METHODOLOGY
Descriptive and inferential statistics were applied
to examine the data. A questionnaire that elicited
Research Questions
background information on our study participants
Based on the research gaps identifiedcompleted
in the the data set.
previous section, our study investigates theThe data for this study were collected by the
follow
ing research questions: E-Tutor (www.e-tutor.org), an online CALL
530 The Modern Language Journal 96 (2012)
program that is integrated into the students' chapter 4, learners had to provide the correct
regular curriculum. The E-Tutor is a comprehen vocabulary (e.g., 'begin' = anfangen), morpholo
sive language learning environment for German gy (e.g., 3rd person singular), graphology and
with a multitude of exercises for L2 learners of word order (e.g., verb stem fängt, second position,
all levels, from beginner to advanced. Each
separable verb prefix an: final position).
chapter of the E-Tutor corresponds to a chapter
of Deutsch: Na klar! (Di Donato, Clyde, & EXAMPLE 2
Vansant, 2008), a textbook commonly used in
North America for L2 university learners of a) Prompt:
German. The exercises cover vocabulary, gram 'The play begins at eight on Fridays.'
mar, writing, listening and reading comprehen b) Sample expected response:
sion as well as cultural information about Das Theaterstück fängt freitags um acht an.
Germany.
Along with several other task types, the three
In terms of a comparison of English-German
task types considered in this study (S-Building,
word order, there was usually a mix of exercises in
Translation, and Essay) were part of students'
each chapter, with some English prompts having
regular homework assignments, and all task types
the same word order as the German response (e.g.,
for each chapter were available to the learners
"Our niece is planning a party in October,"
from their first login. Students completed the tasks
chapter 3) and others having a word order that
independently and outside class time by the
is different from the German response (e.g.,
stipulated deadline for each chapter, which was
communicated to them in the first week of classes
"Unfortunately, you are going to celebrate his
birthday without him," chapter 3) ,5
and again before each chapter's due date. The
students were free to choose the order in which
For the Essay, while there was no space limit in
the CALL environment, students were asked to
they completed the different task types.
write a minimum of 50 words per essay in the
Three Task Types. For this study, we consider beginner course and a minimum of 100 words in
the three E-Tutorta.sk types that allowed students
the to
intermediate course. Similar to the S-Building
practice word order rules productively: S-Building,
and Translation tasks, the essay topics varied by
Translation, and Essay. The students completed chapter and were always closely related to the
the three task types for each of four chapters in theme and the vocabulary and grammar
chapter
each course. taught in each chapter. For instance, in chapter 10,
students learned to talk about travel plans, means
For the S-Building task type, students were asked
to construct a sentence from words that were of transportation, and travel preparation by also
provided in their base forms and grammatical learningcues
the past tense forms. Accordingly, the
(for function words). For instance, for the Essay instructions, shown in Figure 1, asked
example exercise from chapter 10 given in students to write about a trip they took some
Example 1, students needed to provide the correct time ago. Other Essay instructions included, for
definite article das (neuter, nominative, singular), instance, describing guests and sending an invita
conjugate the verb sein ('tobe') as war (3rd person, tion for a birthday party (chapter 3) and giving
singular, simple past), and apply the correct word directions to a friend in the city of Cologne
order (verb second). (chapter 9).
EXAMPLE 1
Differences Among the Three Task Types. The
three task types chosen for our study vary in a
a) Prompt: number of ways. To capture these differences, we
(def. article) / Wasser / warm / sein (simple past). draw on Bachman and Palmer's (1996) framework
(def. article) / water / warm / to be (simple for the description of tasks because the framework
past), [translation not part of prompt] operationalizes the notion that a task is a collection
b) Expected response: of different characteristics rather than an indivisi
Das Wasser war warm.
ble holistic entity. For instance, a multiple-choice
'The water was warm.' [translation not part of grammar task might be based on a picture or a
response]
reading passage, and the instructions might be
given in the learners' LI or the L2. It is especially
For the Translation task,4 students translated an important to capture such detailed distinctions for
English sentence into German, as illustrated in our task types because our tasks are all embedded
Example 2. For this example sentence from in the same CALL environment, but each task has
FIGURE 1
Sample Essay Task From German 200, Chapter 106
tutor
: jj|5| iNTHOOUCTfON
Kapitel 10
INSTRUCTIONS
mtffl > H N H Q._f.

Write a story about a trip that you took some time ago using the
following questions as guidelines:
Where did you go?

For how long?
What did you do?
Did you meet any other travellers on your trip?
Was your trip expensive?
Use your imagination.
£3 Submit your work

Kapttel 11
Kaptei12
CD ® CD ® ® ® ®
Kapltei 13 e ah war, gin^en war nach OeutKhiand. Meirws Freyndinnen und kh spanen unstf Celd far ein Jahr. Wsr
Kapttel 14 Wteben in Deutschatnd for xwel Woch«. W!r besuchten metaer Freundins Oma in Munchen. Auth gingen ri»ch Frankfurt.
Berlin und Dresden. Wr btieben in MSnchen ond Berlin fur vier Tag* und in Franfurt und Dresden fur drei Tag*. In Munches*
LOGOUT ubemachteten <rtr m)t mtitstr Freundins Oma, W>i Ubernachmen in jugendherbefaen, weil es i« b»l8g«r als Motels. Wtr
eriebten die Kultur in jede Stadt Auch wir trafen vtel Menschen, Letder urtser Urlaub mat zu kurz.
distinct characteristics that may impact learner Bachman and Palmer's framework for our three
performance. task types.

To pinpoint differences among different task Although Bachman and Palmer's framework
types that might lead to performance variation, was mainly designed for testing environments, in
Bachman and Palmer's framework outlines a set of contrast to the language practice environment of
our study, the framework readily captures the
characteristics grouped under five aspects of tasks:
different characteristics with respect to the learn
setting, test rubric, input, expected response, and
relationship between input and response. Table ing 1 goals of our task types (for other approaches to
summarizes the aspects and subcomponents oftask classification, see, e.g., Robinson, 2007). For
TABLE 1
Differences Among the Three Task Types

Characteristic S-Building Translation Essay
Setting CALL task CALL task CALL task

Test Rubric Prompt: 20 exercises Prompt: 1 essay per chapter
Prompt; 5 to 8 exercises
per chapter per chapter
Input Language L2 (German) LI (English) LI (English)
Input Organizational Vocabulary, Spelling N/A [no L2 input] N/A [no L2 input]
Language
Response Length 1 sentence 1 sentence 1+ paragraphs
Type of Response Limited production
Limited production Extended production
Response Topical Language Generic Generic Generic (+ cultural, personal)
Response Organizational Syntax, MorphologySyntax, Morphology, Syntax, Morphology,
Language Vocabulary, Spelling Vocabulary, Spelling,
Textual organization
Form-focused Form-focused Meaning-focused
Relationship Input-Response Direct Direct Direct (indirect)
instance, the most prominent difference among course instructor via the E-Tutor interface. Course
our task types, as indicated in Table 1, lies in the instructors then provided feedback to the learner
distinction between form-focused (i.e., S-Building, on a print copy.
Translation) and meaning-focused task types From a pedagogical perspective, the E-Tutor was
(i.e., Essay). The goal of our form-focused primarily conceived as an opportunity for students
exercises was to draw the learners' attention to to practice course-related L2 structures in a
nonthreatening and flexible environment by
the lexical and grammatical forms that correspond
to the chapter topic. Accordingly, for the formencouraging learner autonomy and self-motivated
focused task types, students were given a restricted
practice. For this reason, for all task types, students
elicitation task consisting of one sentence as graded on task completion and not on their
were
opposed to an extended production task as foundlinguistic performance, that is, the number of
with the Essay. Moreover, while the topics covered
errors they made was not taken into consideration.
by each task type were fairly generic, although Students were repeatedly made aware of this
their content was derived from the main chapter grading criterion from the beginning of the
semester.6
content, the topics for the Essay task also referred
to cultural content or elicited personal informa
Target Language Features. Word order in Ger
tion (see Table 1). For instance, one essay was
about giving directions in the city of Cologne,is highly complex depending on the mor
man
phosyntactic components of the sentence. The
another asked students to relate a past vacation.
With respect to differences between the twodifferent types of word orders are introduced in
restricted elicitation tasks, most notably, S-Build
our courses according to the chapters of the
course book (see Table 2), and the E-Tutor tasks
ing provided the German vocabulary items as well
as the spelling of their base forms that studentsfollow
had the same grammatical sequence.
to use to form sentences. The Translation task only
displayed the English words that needed to be
Data Analysis
translated. Accordingly, Translation also required
lexical and graphological knowledge in additionWith
to respect to data collection and analysis, the
morphology and syntax as the only challenges E-Tutor logged student interaction with the system
posed by the S-Building task (see Table 1). for all task types, keeping track of each submission
and saving it for research purposes (e.g., student
Feedback. For both the S-Building and Transla
ID, task type, student input, time submitted). For
tion task, the E-Tutor provided immediate, error
both the S-Building and Translation task type, only
the first student submission for each exercise was
specific and individualized feedback by perform
ing a linguistic analysis of student input, thereby
considered, that is, the first input that the learner
identifying grammar (e.g., subject-verb agree provided prior to receiving feedback from the
ment), spelling, and word order errors. Along CALL program on potential word order (or other)
errors for that sentence. For the Essay task, the
with the feedback messages, the /Î-Tutorfrequently
displayed context-sensitive help through hyper
£-7 ittor logged a copy of the Essay that the student
submitted to the instructor, which the researchers
links, providing inflectional paradigms, or expla
nations of word order rules, for instance. subsequently analyzed.
For each S-Building and Translation exercise, Two human coders independently tagged the
students could resubmit their answer as many submissions for word order errors in all three task
times as they wished, receiving immediate, error

types. We chose the sentence as the unit of analysis
and applied a binary judgment, that is, students
specific feedback anew every time. Once an entire
exercise set of a task type was completed, the
either provided a correct or an incorrect word
course instructor received a summary perfor order for a sentence. For our analysis, all sentences
mance report for each student, containing containing
the word order errors were recorded
equally
number of exercises the student completed with a for all chapters, ignoring at which point
breakdown of the number of correct responses a particular grammatical construction pertaining
to word
and the kinds of mistakes that were made (e.g., 5 order was introduced. In all instances, the
spelling mistakes). However, the course instructor
two coders agreed on the word order judgment for
neither saw the actual student input noreach
thesentence.
number of times the student repeated an exercise We employed both descriptive and inferential
set before having submitted it. statistics. For the descriptive statistics, we first
In contrast, the Essay task was not evaluatedcounted
by the total number of sentences for all
the CALL program; instead, it was submitted tostudents
the in each course and for each task type, and
Trude Heijt and Anne Rimrott
TABLE 2
Grammar Topics in Deutsch: Na Klar!
Course Chapter Grammar Content
German 100 Chapter 1 Basic sentences and questions

Chapter 2 Negation, nominative case
Chapter 3 Accusative case
Chapter 4 Separable prefix verbs, modals
German 101 Chapter 5 Dative case
Chapter 6 Simple Past of modals, time expressions

Chapter 7 Present perfect
Chapter 8 Coordinating/subordinating conjunctions,
reflexive pronouns
German 200 Chapter 9 Genitive case
Chapter 10 Simple past, past perfect

Chapter 11 Relative clauses
Chapter 12 Subjunctive
then divided those into the number of sentences

the two courses combined was 83.1%. Considering
with correct and incorrect word order. The each task type separately, our students were most
inferential statistics required a calculation of the with the Essay task with an accuracy
successful
rate of 88.9%, followed by Translation (81.3%)
average rate of correct sentences for each student.
and S-Building (79.2%). The same order was
To allow for cross-comparisons among learners
echoed
and task types, we normalized the working in German 100, while in German 200 the
style
data by dividing for each student the number
highestof
accuracy rate also occurred with the Essay
task by
sentences that contained correct word order (88.4%), but the average accuracy rates
forwith
the total number of sentences (i.e., sentences Translation and S-Building were identical
(75.8%).
correct and incorrect word order) produced by
each student for each task type. For the inferential
The data displayed in Table 3 further show that
for each task type, grammatical accuracy was
statistics, the alpha level to determine statistical
significance was set to .05. higher in the beginner course than in the inter
mediate course. The largest difference in accuracy
RESULTS was evident in the Translation task (86.9%,
75.8%), followed by S-Building (82.5%, 75.8%),
The data for our study were taken fromwhile the the accuracy rates for the Essay task were
three tasks that our L2 learners of German fairly consistent for the two proficiency levels
completed in the two courses, German 100 (89.3%,
and88.4%).
200, and then normalized as described above. For an empirical comparison of the impact of
Table 3 displays the means and standard devia task type on learner performance for the two
tions of the learners' accuracy rates for the threedistinct proficiency levels, we normalized the data
task types and two courses. by calculating for each student, course, and task
The data in Table 3 indicate that the averagetype the means of sentences with correct word
accuracy rate for word order for all three tasks inorder and subsequently applied a two-way analysis
TABLE 3
Means (M) and Standard Deviation (SD) of Word Order Accuracy Rates by Task and Course
German 100 German 200 Both Courses
M SD M SD M SD
S-Building 82.5% .07 75.8% .10 79.2% .08

Translation 86.9% .09 75.8% .11 81.3% .07
Essay 89.3% .09 88.4% .07 88.9% .06
All Task Types 86.3% .06 80.0% .06 83.1% .05
534 The Modem, Language Journal 96 (2012)
of variance (ANOVA). Results indicate that task Hill and Laufer (2003) examined incidental voca
type has a significant effect on L2 word order bulary learning.
accuracy (*1(2,42) = 9.674; p = .000; t)2 = .178). The following discussion focuses on three
Pairwise comparisons conducted to examine the aspects that all appear to have had some influence
significant main effect of task type revealed that on the fact that students achieved the highest word
the two form-focused task types, S-Building and order accuracy in the meaning-focused task: the
Translation, both result in significantly more error avoidance phenomenon, task sequence, and
errors than the Essay task (see Table 4, statistically attention to task types.
significant results indicated in bold). Proficiency
Error Avoidance Phenomenon. A commonly ob
level also has a significant effect on L2 word order
served L2 learner behavior when studying aspects
accuracy (F( 1,28) = 11.022;p = .001;nq2 = .116),
of learner interlanguage and error analysis is the
meaning that beginner students perform signifi
avoidance phenomenon, first noted by Schachter
cantly better than intermediate learners. Finally,
(1974). In her study with L2 learners of English,
no interaction effect between task type and Schachter concluded that avoidance behavior is
proficiency level was found (p = .093).
an L2 strategy whereby learners tend to avoid
complex target language structures if they present
challenges for them. Research has further shown
DISCUSSION that the only factor that consistently predicts
avoidance is the L1-L2 difference variable (see
With an overall mean of 83.1 % correct also
responses
Dagut & Laufer, 1985; Gass & Selinker, 1994,
for all task types and a minimum mean1992;score of
Hulstijn & Marchena, 1989; Kleinmann,
75.8% correct responses for S-Building and Laufer & Eliasson, 1993; Waibel, 2007).
1977,1978;
Translation in German 200 (see Table 3), word
The avoidance phenomenon might have had
order accuracy for our participants is generally
some impact on our finding that the meaning
quite high. The following sections discuss thetask produced higher accuracy rates in
focused
performance in the meaning-oriented task and the
word order than the two form-oriented tasks for
two form-focused tasks. An examination of learner
both German 100 and German 200.7 Interestingly,
variability in our data follows. in considering learner performance in each
chapter, Table 5 indicates that the lowest word
order accuracy rates for the two form-focused task
Best Performance With the Meaning-Focused Task
types occurred in chapter 11 (relative clauses). In
Unlike Hill and Laufer's (2003) study, whichcontrast, for the Essay task type, we find the lowest
also compared form-focused tasks with meaning accuracy rate in chapter 4 (separable prefix verbs).
focused tasks in a CALL environment, our data Thus, for both S-Building and Translation,
reveal that for both courses, German 100 and chapter 11, which focuses on relative clauses, was
German 200, our learners performed best with the the most challenging chapter for our learners with
meaning-focused task type (i.e., Essay). Given that respect to word order. Given the rigidity of the two
previous classroom-based studies have found thatform-focused task types, students had no choice
the teaching objective can have an impact on but to use the relative clauses in their submissions.
learner performance (see, e.g., Tarone & Parrish, In contrast, the Essay prompts generally did not
1988), the difference in findings might at least force students to use a particular grammatical
partly be due to the fact that our study consideredconstruction, and thus students could avoid the
the acquisition of syntax (i.e., word order), while more syntactically complex ones. For instance, the
TABLE 4
Results for Pairwise Comparison for the Three Task Types (bold = p < 0.05)
Task Type (I) Task Type (J) Mean Difference (I - J) Std. error Sig.
S-building Translation -.022 .023 .352
Essay -.097 .023 .000

Translation S-building .022 .023 .352
Essay -.075 .023 .002
Essay S-building .097 .023 .000
Translation .075 .023 .002
TABLE 5
Means of Word Order Accuracy Rates for German 100 and 200 by Task and Chapter
Chapter 1 Chapter 2 Chapter 3 Chapter 4 Chapter 9 Chapter 10 Chapter 11 Chap
S-Building 89.3% 83.0% 80.8% 77.0% 84.0% 75.0% 68.7% 75.6%

Translation 98.1% 89.2% 77.3% 82.2% 84.4% 74.1% 64.9% 79.7%
Essay 91.4% 91.2% 90.3% 82.4% 92.8% 83.8% 85.8% 92.0%
All Task types 92.7% 87.9% 82.7% 80.8% 87.1% 77.6% 73.1% 82.4%
content of a relative clause could also be stated in a For both chapters 4 and 11, of those students
separate sentence. who did use the constructions under investigation,
However, higher syntactic complexity is some only a small percentage of the sentences contained
a separable prefix verb (31.1%) but even fewer
times closely tied to lexical choice and thus cannot
be avoided even in less rigid, meaning-focused contained a relative clause (12.4%). This contrasts
tasks such as the Essay. Chapter 4 on separable
with the form-focused exercises, which prescribe
the usage of certain grammatical constructions
prefix verbs triggered the most errors for the Essay
task (see Table 5). Here, students could not avoid and thus no choice is given.
the more demanding word order prompted byIn summary, the Essay task in general lends itself
separable prefix verbs because these verbs are to error avoidance because it affords flexibility
primarily a lexical choice. For example, students in the content and structure of the learners'
wanting to express the meaning of "invite" in response. In contrast, the S-Building and Transla
German generally need to use the separable prefix tion tasks are constrained elicitation techniques
that predetermine a limited set of correct
verb einladen ('to invite'). Accordingly, to write on
the essay topic, students must generally use responses:
at Only very slight variation in word order
least some separable prefix verbs. The additional is allowed but the specific vocabulary, morphology,
challenge presented by their syntax is a byproduct grammatical construction, and graphology is
of this lexical choice that the students cannot already determined by the task prompt. These
circumvent. differences in task characteristics, as illustrated
This is supported by our data displayed inearlier by Bachman and Palmer's (1996) frame
Table 6, which indicate that, in their essays, all work, are reflected in our results and may also, at
learners used separable prefix verbs (100.0%)least partly, explain the discrepancies in results
while much fewer students used relative clauses between Hill and Laufer's (2003) study and ours.
(53.3%). In chapter 4, the Essay task askedAt the same time, our results underscore that
changes in the teaching objective of a language
students to describe how they spent a day after
winning a prize of 2,500 Euros. Every participant
task (e.g., relative clauses versus separable prefix
used the separable prefix verb ankaufen ('toverbs)
go can lead to task-related variation, thus
shopping') and 8 of the 15 participants (53.3%)
further supporting the claim that language tasks
used the separable prefix verbs aufwachen ('toare best examined as collections of individual
features rather than as holistic entities (Bachm
wake up') and/or aufstehen ('to get up') in their
& Palmer, 1996; Purpura, 2004). This is als
essay, demonstrating how the essay topic can guide
lexical choice of separable prefix verbs and thus
evident in the variation in learner accuracy we fi
syntax. within each of our three tasks. For instance, for
TABLE 6
Usage of Grammatical Constructions in Chapters 4 and 11 for the Essay Task
% of Learners % of Sentences
Using the Containing the % of Constructions % of Constructions
Construction Construction Used Correctly Used Incorrectly
Chapter 4:
Separable Prefix Verbs 100.0% 31.1% 93.5% 6.5%
Chapter 11:
Relative Clauses 53.3% 12.4% 85.7% 14.3%
S-Building the accuracy rate ranges from 89.3% in importance of accurate performance," p. 49) i
chapter 1 to 68.7% in chapter 11. This suggests that "tasks perceived as high stakes are likely t
prompt more attention to form" (p. 49). Fo
that, in addition to other task complexity meas
instance, previous research (see Ellis, 1994;
ures, the teaching objective that the task addresses
must be considered. Skehan & Foster, 1997), although based on ora
An interesting question is whether practiceperformances,
in has found that the amount o
planning that goes into the task has an effect o
form-focused instruction leads to the development
of more advanced L2 structures and thus to less
learners' performance in that a planned languag
avoidance in free composition tasks, a topic task
of generally results in higher accuracy rates fo
investigation that is beyond the scope of target-language
the forms.
current study. For both the S-Building and Translation tasks
the CALL program allowed learners to explore
Task Sequence. Another factor to consider with
L2 structures by providing feedback on their
respect to our findings, however, is the order in
input. Students could choose how often the
which the tasks were completed. For all chapters,
redid an exercise set before submitting the fin
our students generally first worked on S-Building,
report to their instructor. Even more importantly
then Translation and, last, the Essay.8 Accordingly,
they were aware that instructors only saw
one may speculate that the practice students
summarized final report and not their actu
obtained with the more form-focused tasks may
language input or the in-between tries. Accor
have led to a higher accuracy rate for the Essay task
ingly, for both S-Building and Translation
type, given also that the CALL program provided
immediate feedback on the learners' word order
without fear of "losing face," students could pa
less attention to their response in their first
mistakes for the S-Building and Translation tasks.
submission (which we took as the basis of our dat
This is certainly a valid consideration, in particu
analysis) and/or guess at an answer because the
lar, given Heift's (2003) findings that the input
environment of the task affects learner behavior
were still able to submit perfect scores to thei
instructors and hide the fact that it might hav
and learning outcomes. For the form-focused taken them several tries.10
tasks, for instance, the CALL environment provid
With the Essay, however, students did no
ed learners with the opportunity to explore
receive computer feedback along the way but
language and guess at an answer because a
sentence could be submitted several times and
rather submitted their essay to their instructor for
error correction. For this reason, students migh
system feedback was provided.9 This is distinct
have been more inclined to pay close attention t
from the Essay task where no ongoing feedback
form in the Essay to "save face" in front of thei
was received and once the Essay was completed
instructor than in their first submission in
and submitted, no further changes could be made.
S-Building and Translation of the CALL progr
At the same time, our findings cannot be
Thus, knowing what is at stake in the Essay,
attributed to practice alone because this, in turn,
learners might have devoted more planning to
would suggest that students should have per
Essay task, which might have led to higher w
formed better with Translation than S-Building
order accuracy rates in that task.
given that Translation was generally completed
after S-Building in both courses. However, for
three of the eight chapters, students performed
better on the S-Building task (see Table 5). Accuracy in
Higher Beginner Course for Both
Form-Focused, Tasks
Moreover, and in addition to considering the
overall results of all learners, our data on learner
Although in this regard our study was n
variability indicate that 7 students scored better on
controlled enough to allow for a reliable inter
the S-Building than on the Translation task (see
tation of learner interlanguage development fr
Table 8 below).
German 100 to German 200, it is nevertheless
Attention to Task Types: What's at Stake?. noteworthy
In addi that for both form-focused tasks (see
tion to task sequence, another issue to consider
Table 3),11 word order accuracy is significantly
highertoin the beginner than in the intermediate
is the potential difference in learners' attention
the three tasks, which might have led tocourse,differ while the accuracy rate for the Essay task
ential results in word order accuracy for thefairly constant. The most likely reason for
remains
form-focused versus meaning-focused taskthese results is an increase in task complexity in
types.
Chapelle (2001) notes that learner attention addition
is to developmental readiness, both of
affected by stakes (i.e., "learners' perceptionwhich we discuss in the following.
of the
Task Complexity. In support form-focused oftasks and the relative
previous re st
search (see, e.g., Adamson, accuracy2009,
for the pp.
Essay139-140),
task might also
one likely explanation for issues
theof developmental
decrease in readiness
accuracy (Pie
rates for the two form-focused tasks is that in form 1998). According to Pienemann's Proc
focused tasks monitoring of simple word order Theory, L2 acquisition follows a devel
rules can be successfully accomplished, whereas sequence, which gradually unfolds
monitoring of difficult rules might result in acquisition of procedural skills needed
decreased accuracy. The increase in difficulty in essing the second language. The pr
word order is reflected in the change of sentence procedures follow a set order, wherein e
length from the beginner to the intermediate level level procedure is a necessary prerequ
for the form-focused tasks, as one indicator of task the following procedure. Accordingly, t
complexity (see Ortega, 2003).12 predicts that stages of acquisition
More specifically, Table 7 indicates that there "skipped" through formal instruction,
was an increase of 73% in sentence length from learner's developmental sequence is no
German 100 to 200 for Translation and an increase ily in line with the instructional sequenc
of 51% for S-Building, while the increase for the s/he is exposed to target language stru
Essay task was much smaller (9% ). This increase in instructed SLA.
sentence length partly reflects the more complex The relatively low accuracy in German 200 for
sentence structures that students were forced to both S-Building and Translation might indicate
use at an intermediate level with the S-Building that the word order concepts practiced in these
and Translation tasks (e.g., embedded clauses). form-focused tasks in the more advanced course
This greater complexity in the intermediate were beyond the learners' developmental readi
course in turn may have resulted in the signifi ness. However, given the rigidity of the task types,
cantly lower word order accuracy rates in the two students had no choice but to (try to) apply them.
form-focused tasks in the intermediate compared In contrast, the Essay task allowed learners to
to the beginner course.13 In contrast, for the Essaychoose the complexity of the word order structures
students had more freedom in choosing the that they employed to a much larger extent. This
grammatical constructions and sentence length enabled learners to remain within their develop
as long as they focused on the essay topic of each mental stage and thus might explain the consisten
chapter. Thus the sentence length for the two cy in the word order accuracy rate for both German
courses varied little, which may have been an 100 and German 200 for this task type.14
additional factor leading to higher accuracy ratesThe concept of developmental readiness might
also explain some of the learner variability we
for the Essay task despite the fact that, at least for
German 100, the Essay task contained on average found in our data in that, for instance, the
the longest sentences. accuracy rates of the form-focused task types did
This suggests, however, that even if students not drop from German 100 to German 200 for
have been taught the more complex construc all learners. Accordingly, learner variability also
appears to be relevant to the investigation of task
tions, they tend to stick to shorter and thus simpler
sentence constructions unless they are forced related
to variation in CALL. This is discussed in
use longer/more complex constructions as in the more detail in the following.

case of the two form-focused task types.
A Developmental Perspective. The decrease Learner

in Variability
accuracy from German 100 to German 200 for the
In addition to the results for the two groups an
courses as a whole, learner variability must als
TABLE 7 be considered, in particular, given that previou
Mean Number of Words/Sentence for the Three research has shown that some learners excel on
Task Types some tasks more than on others (see, e.g., Heift,
2005; Purpura, 2004; Skehan, 2003b; Thouësny,
Mean Number of Words per
2010). Our sample also displays learner variability
Sentence (Standard Deviation)
with respect to word order accuracy in the three
German 100 German 200 Increase tasks.
Table 8 displays some of our data that relates

S-Building 4.5 (1.2) 6.8 (1.7) +51% to the performance of our 15 learners across
Translation 5.6 (1.7) 9.7 (2.5) +73%
task types. For instance, the results displayed in
Essay 6.7 (3.2) 7.3 (3.1) +9%
Table 5 show that, in considering each learner's
The Modem Language Journal 96 (2012)
TABLE 8
Interlearner Variability: Overall Accuracy Rates by Task Type
S-Building Translation Essay All Task Types
Student A 87.7% 100.0% 95.1% 94.3%

Student B 71.5% 76.8% 78.1% 75.5%
Student C 66.3% 80.9% 91.7% 79.6%
Student D 80.0% 73.8% 85.9% 79.9%
Student E 79.0% 84.0% 90.0% 84.4%
Student F 87.0% 93.0% 87.0% 88.8%
Student G 69.0% 83.0% 90.0% 80.8%
Student H 82.0% 78.0% 85.0% 81.6%
Student I 87.0% 79.0% 87.0% 84.3%
Student J 69.0% 76.0% 97.0% 80.7%
Student K 85.0% 85.0% 90.0% 86.6%
Student L 83.0% 76.0% 83.0% 80.8%
Student M 72.0% 80.0% 96.0% 82.6%
Student N 78.0% 71.0% 79.0% 76.1%
Student O 89.0% 84.0% 97.0% 90.0%
79.2%, range: 7.3%) whereas student C's perfor

performance for the two courses combined, there
mance
were in fact 4 students who, in contrast to the mean was most varied (S-Building: 66.3%,
Translation: 80.9%, Essay: 91.7%, range: 25.4%).
accuracy rates for the learners as a whole, scored
Furthermore, student A always produced the
better or equally well on one or both of the form
focused tasks compared to the meaning-focused correct word order in Translation (100.0%) but
had lower accuracy rates in Essay (95.1%) and
task (students A, F, I, and L). Also distinct from the
findings of the whole group (see Table 3), S-Building (87.7%).
7 students scored better on the S-Building than In considering the two courses independently,
on the Translation task (students D, H, I, L, N, the
andresults in Table 9 also show learner variability.
O). For example, student I achieved 87.0% For instance, unlike the results for all students,
accuracy in both S-Building and Essay but had a were in fact 3 students who achieved a higher
there
lower accuracy rate in Translation (79.0%). The accuracy rate for S-Building in German 200 than in
range of intralearner variation across the three German 100 (students B, D, and L) and 3 students
tasks varied, as well. For instance, student B's who achieved a higher or equally high accuracy
performance was most consistent across tasks rate in Translation in German 200 compared to
(S-Building: 71.9%, Translation: 75.1%, Essay:German 100 (students A, E, and L). In addition, 6
TABLE 9
Interlearner Variability: Accuracy Rates by Course and Task Type
S-Building Translation Essay
100 200 100 200 100 200
Student A 93.7% 81.7% 100.0% 100.0% 96.1% 93.8%

Student B 68.8% 75.0% 90.2% 60.0% 69.5% 88.9%
Student C 78.8% 53.8% 86.9% 75.0% 92.6% 91.2%
Student D 77.5% 82.5% 83.1% 64.6% 100.0% 75.3%
Student E 82.5% 76.3% 82.7% 85.0% 85.0% 94.6%
Student F 93.8% 80.0% 95.0% 90.0% 94.4% 79.9%
Student G 76.3% 62.5% 96.9% 70.0% 89.9% 89.2%
Student H 82.5% 82.4% 81.7% 73.8% 77.9% 93.5%
Student I 87.5% 86.1% 83.5% 75.0% 85.2% 88.6%
Student J 77.2% 61.1% 86.7% 64.6% 97.7% 96.9%
Student K 89.9% 80.5% 87.7% 81.7% 95.8% 84.2%
Student L 80.0% 86.1% 67.1% 85.0% 82.3% 84.4%
Student M 73.4% 71.0% 91.9% 68.3% 97.4% 93.8%
Student N 83.8% 72.0% 72.3% 70.0% 81.0% 77.4%
Student O 91.3% 86.7% 95.0% 73.8% 95.3% 97.9%
Trude Heifl and Anne Rimrott 539
students also performed better on the Essay task in CONCLUSION
German 200 than in German 100 (students B, E,

H, I, L, and O). Student L was the only student who Our results indicate that learner perfor
achieved higher accuracy rates in German 200 on German word order varies across different
than in German 100 in all three tasks (while 7 CALL tasks. Our learners performed more
students—C, F, G, J, K, M, and N—showed the accurately with the meaning-focused task than
opposite trend). with the two form-focused task types, and this
In addition to interlearner variability, Figure 2, variation was consistent for both the beginner
by way of example, provides the mean accuracy and intermediate proficiency levels. Moreover,
rates for student B for each chapter of the two grammatical accuracy was significantly higher
courses (German 100: chapter 1-4; German 200: in the beginner than the intermediate course
chapters 9-12). for the form-focused task types, while learner
The data in Figure 2 show that, unlike a linear, performance in both courses was fairly
systematic progression, student B's performance constant for the Essay task. Finally, our study
varied not only by chapter but also for all three task also confirmed previous findings of learner
types and for both courses. For example, for the variability found across task types and time (e.g.,
S-building task, he achieved the highest word Thouësny, 2011).
order accuracy (95%) in chapter 9 and the Our results can likely be attributed to a
lowest accuracy (60%) in chapters 4 and 10. For confluence of factors, such as error avoidance,
Translation, he achieved 100% accuracy in increased practice, and more attention to task in
chapters 3, 4, and 11, while his lowest accuracy the meaning-focused Essay task (see section Best
rates for all task types and chapters was for Performance with the Meaning-Focused Task). In
Translation in chapter 10 (20%), followed by the addition, the increase in task difficulty from the
Essay task in chapter 2 (47%). Moreover, unlike beginner to the intermediate proficiency level
the results of all 15 participants, student B and, concomitantly a possible mismatch of task
achieved a higher mean accuracy rate for Essay demands and developmental readiness in the
and S-Building in German 200 than in German 100 intermediate course with respect to the two form
while also scoring the lowest for Translation in focused S-Building and Translation tasks may have
German 200. impacted our findings. Finally, our findings also
FIGURE 2
Intralearner Variability (Student B)
Means Means
1 2 3 4 for 9 10 11 12 for
100 200
—■—S-Building 80% 65% 70% 60% 68.8% 95% 60% 75% 70% 75.0%
û.
A Translation
Translation 86% 75% 100% 100% 90.2% 60% 20% 100% 60% 60.0%
—♦—Essay 71% 47% 74% 86% 69.5% 100% 67% 89% 100% 88.9%
540 The Modern Language Journal 96 (2012)
appear to be influenced by interlearner and that it depends on the particular GALL env
intralearner variability. ment (e.g., task type) to which the accur
However, given the complex issues involved in measure is applied. This has implications for
this topic, further research is clearly required to guidance and evaluation of students in tha
more reliably identify the factors that contribute to need to provide a wide variety of CALL tasks
task-related variation in CALL. To name a few, allow learners to explore and practice the L2 g
previous classroom-based research has shown that that their accuracy rates differ depending
psycholinguistic factors may also be relevant which task they perform. Generally, our st
(Ellis, 1994). For the Essay task type, for example, suggests that meaning-focused CALL tasks r
students could express their own ideas and in a higher accuracy rate than form-focused
thoughts, which was likely more motivating than independent of the learners' proficiency le
completing the more rigid elicitation tasks. This Our study further demonstrates that word o
may have also contributed to the higher accuracy accuracy not only varies between form-foc
rate for the Essay task. Moreover, the connection and meaning-focused tasks but also for diff
and impact of form-focused versus meaning types of form-focused tasks such as S-Buil
focused task types on the acquisition of German and Translation. Teachers, CALL designers,
word order remain to be investigated. In addition, students need to be aware that the accuracy ra
while our study focused mainly on accuracy in one CALL task differs from that in another and
learner performance across tasks, task-related that a fair evaluation can only be reached by
variation in the complexity and fluency of learner considering different task types. As stated by
language in CALL is yet to be explored. Skehan (2003a):
These additional research topics also point to
The choice of task that a teacher makes is not a neutral
some of the limitations of our study. Most impor
tantly, the sample size of our study was limited to 15 affair at all, and that to choose a particular task type
may well mean that learners are being pushed to
students who were selected because they complet
advantage some areas of language performance and
ed three consecutive German language courses
disadvantage others. Some tasks tend to produce more
and submitted their homework regularly. None complex language, others greater accuracy, while
theless, the limited sample size was large enough to others favour fluency, (p. 395)
gain some insight into task-related variation by not
only investigating learners' overall performance in For instance, our results indicate that students
two courses but also individual performance across tend not to use the more challenging grammatical
task types and time. However, our results are based constructions, if a choice is given (i.e., in less
on one aspect of L2 acquisition of German only, constrained, meaning-focused tasks). However,
namely the acquisition of word order, and it given that more challenging constructions (e.g.,
remains to be investigated whether, and to what relative clauses) form an essential part of the
extent these findings are applicable to other native speaker's linguistic competence, form
languages and other aspects of grammar. Further focused CALL tasks such as sentence construction
more, due to the nature of the task types chosen for or translation tasks can provide reliable and
our study, the tasks differed in the number of extensive practice with these constructions that
sentences that students wrote, and this may have may be lacking in meaning-focused tasks such
affected our results. For instance, Translation as essays otherwise. This possibly leads to the
contained fewer individual exercises than S-Build learners' eventual utilization of these structures by
ing for all chapters. Finally, this study did not choice. Moreover, CALL ideally lends itself to
form-focused instruction that can be carried out
examine the acquisition of word order concepts
over time. Accordingly, it is still unclear whether
independently and meaningfully. As an additional
intermediate L2 learners of German, for example,
advantage, assigning form-focused CALL tasks for
independent study in turn frees up classroom time
still commit errors with word order concepts taught
for more meaning-focused tasks in instructed SLA
at a beginner level and, if so, for which task types.
Moreover, it remains to be investigated which
settings. For this, however, it is important that
grammatical constructions pose persistent prob
the CALL tasks are integrated into the overall
lems and thus deserve special attention. curriculum, thus effectively augmenting classroom
instruction.
PEDAGOGICAL IMPLICATIONS Our study also shows that even within the same
task (e.g., Essay), learner performance varies ac
From a pedagogical perspective, our cording
findingsto the teaching objective (e.g., grammati
emphasize that the notion of accuracy is cal content
relative inor topic) of the instructional unit
Trude Heifl and Anne Rimrott
(see section Error Avoidance Phenomenon). 3 Note that there are a vast number of studies that
Accordingly, instructors should also take this investigate issues in technology and Task-Based Lan
guageTeaching
possible variation into account, for instance, by (TBLT) more generally but their focus is
not on task-related variation. For an overview, see, for
aggregating performance scores from different
instance, Thomas and Reinders (2010) and Lai and Li
samples of the same task type for a more accurate
(2011).
reflection of learner performance.
4 For all chapters, the E-Tutor contains some transla
We also need to address the fact that, as
tion exercises although, in following the more promi
suggested by our study, learner performance nent current approaches to L2 teaching, this task type
manifests itself in distinct ways not only forrepresents a small percentage of the tasks contained in
learners at different proficiency levels but alsothe E-Tutor, performed in regular classroom instruction
for each individual learner. CALL, with its and on exams. However, and as noted by Schjoldager
potential for adaptive instruction, might offer(2004), translation "involves a beneficial constraint on
the writing process: The learner is not free to choose the
some solutions here. For example, learners could
receive scaffolding (e.g., feedback) based on themeanings that s/he must express and therefore may
be forced to venture into unknown areas of the L2 system
task they are completing, their proficiency level,
(e.g., Duff 1989/1992, p. 7; Cook, 1998, p. 119; see also
and/or their individual past performance.
Campbell 1998, p. 58)" (p. 135). Moreover, we believe
In sum, our study suggests that languagethat providing a vast range of task types further assists in
instructors, CALL designers, and language learnobtaining a more complete picture of the learner's
ers should be aware that a more accurate assess
interlanguage performance (see also Laufer & Girsai,
2008).
ment of a learner's performance can be achieved
° However, we believe that this did not influence the
by evaluating data drawn from a variety of CALL
task types, preferably a mix of form-focusedresults
and because, first, the learners did not know which
sentences required the same or a different word order
meaning-focused task types and at different points
and, second, as stated by Pienemann (1998), while "there
in time. In addition, adaptive instruction in CALL
is no guarantee that the learner will not attempt to
can address some of the variability in interlan
transfer LI word order onto L2... it is equally likely that
guage accuracy stemming from task types, learner
he or she will initially express the relationship between
proficiency levels, and/or the individual learner.
conceptual structures and surface form in a computa
However, the extent to which CALL leads to tionally simpler way" (p. 75).
successful mastering and use of the grammatical 6 The picture shows the central railway station in
construction in meaning-focused tasks still has toCologne, Germany. The caption Kölner Hauptbahnhof
be determined. For now, our findings suggest that means 'central railway station Cologne.'
due attention must be given to task design in 7 We believe that this provided students with an
opportunity to genuinely practice the L2 constructions
CALL, that is, the careful selection and sequencing
that they had been taught in class without penalty.
of a wide variety of tasks in order to provide a
Moreover, for research purposes, we believe that this
practice environment most conducive to L2reflects a more authentic environment in that students
acquisition.
had no need to consult external sources when perform
ing the tasks and thus their performance more accurately
represents their current knowledge state.
8 While we did not elicit from our learners through
ACKNOWLEDGMENTS
retrospective interviews, for instance, whether they
consciously chose to avoid certain word order concepts,
our data for the meaning-focused task indicates that
This research was supported by Social Sciences
and Humanities Research Council (SSHRC), some types of word order were used less than others.
Canada, grant number 632209. We would like to 9 This might be due to the order in which the tasks
thank Marie van Aarsen for her assistance with the were displayed in the E-Tutor given that students were
not instructed that the tasks had to be completed in a
data coding. fixed order.
10 Note, however, that for our data analysis, only the

first submission for each sentence was considered for
S-Building and Translation, that is, the submission that
NOTES
was provided before receiving error feedback from the
CALL program for that sentence.
11 Interestingly, a study by Heift (2005) showed that
1 The term interlanguage performance or learner perfor
70% of the students repeated E-Tutor exercise sets to
mance thus refers to the accuracy measure in this article.
2 For studies that investigated task-related variation in
improve their performance before submitting them to
meaning-focused tasks only, see, for example, Geeslin
their instructor, thus suggesting that learner practice
(2006) and Newton and Kennedy (1996). behavior is influenced by their performance despite the
The Modern Language Journal 96 (2012)
fact that learners were not graded on their performance fluency of L2 student writing. Journal of Second
but only on exercise completion. This, in effect, might be Language Writing, 12, 267-296.
a beneficial side effect of grading for completion because Chapelle, C. A. (2001). Computer applications in second
learners will engage in additional self-motivated practice language acquisition: Foundations for teaching, testing
without teachers having to overtly prompt them. and research. Cambridge: Cambridge University
12 Please note that this would have required a more Press.
laboratory-like design with students known to be at Chapelle, C. A. (2004). Technology and second language
different levels performing the same set of tasks in a learning: Expanding methods and agendas. System,
counterbalanced design to control for order effects. In 32, 593-601.
addition, a close examination of the particular word Chapelle, C. A. (2007). Technology and second language
order concepts acquired at each stage would have been acquisition. Annual Review of Applied Linguistics, 27,
necessary. 98-114.
13 Note that text length (e.g., words per sentence) is Dagut, M., & Laufer, B. (1985). Avoidance of phrasal
commonly employed to assess fluency which, in addition verbs: A case for contrastive analysis. Studies in
to accuracy and complexity, forms part of a learner's Second Language Acquisition, 7, 73-79.
language proficiency (see e.g., Chander, 2003; Larsen DeKeyser, R. M. (1995). Learning second language
Freeman, 1978; Leal, 2005). grammar rules: an experiment with a miniature
14 Note, however, that in both courses, the mean linguistic system. Studies in Second Language Acquisi
sentence length in the S-Building task was smaller than tion, 17, 379-410.
the mean sentence length in the Translation task (see Di Donato, R., Clyde, M. D., & Vansant.J. (2008). Deutsch:
Table 7) but students achieved higher word order Na Klar! An introductory German course. New York:
accuracy in Translation than in S-Building in German McGraw-Hill.
100 (86.9% Translation, 82.5% S-Building, Table 3) and Dickerson, L. B. (1974). Internal and external patterning of
equal accuracy in both tasks in German 200 (75.8%, phonological variability in the speech of Japanese learners
Table 3). This suggests that the findings cannot be solely of English: Toward a theory of second language
attributed to task complexity as measured by sentence acquisition. Unpublished doctoral dissertation,
length. University of Illinois.
15 Interestingly, Pienemann (1998) reports on a Dickerson, L.J. (1975). The learner's interlanguage as a
similar case on the acquisition of German word order system of variable rules. TESOL Quarterly, 9, 401
by one learner, Guy, in a 1-year longitudinal study. 407.
Although Guy was explicitly taught the word order of Doughty, C.J. (1991). Second language instruction does
separable prefix verbs early on, for instance, he started make a difference: Evidence from an empirical
applying the structure much later, leading Pienemann study of SL relativization. Studies in Second Language
to conclude that "over a period of 10 weeks, Guy's Acquisition, 13, 431-469.
interlanguage structure contrasted sharply with the input Ellis, R. (1994). The study of second language acquisition.
and the learning objectives of his German course" Oxford: Oxford University Press.
(p. 119). Ellis, R. (2003). Task-based language learning and teaching.
Oxford: Oxford University Press.
Felix, U. (Ed.), (2003). Language learning online: Towards
REFERENCES
best practice. Lisse: Swets & Zeitlinger.
Foster, P., & Skehan, P. (1996). The influence of
Adamson, H. D. (2009). Interlanguage variation in
planning and task type on second language
theoretical and pedagogical perspective. New York:
performance. Studies in Second Language Acquisition,
Routledge. 18, 299-323.
Ayoun, D. (2001). The role of negative and positive Gass, S. M., & Selinker, L. (1994). Second language
feedback in the second language acquisition of the acquisition. Hillsdale, NJ: Lawrence Erlbaum.
passé composé and imparfait. Modem Language Gass, S. M., & Selinker, L., (Eds.). (1992). Language
Journal, 85, 226-243. transfer in language learning. Philadelphia/Amster
Bachman, L. F., & Palmer, A. S. (1996). Language testing in dam: John Benjamins.
practice: Designing and developing useful language tests. Geeslin, K. L. (2006). Task design, discourse context and
Oxford: Oxford University Press. variation in second language data elicitation. In C.
Breen, M. (1989). The evaluation cycle for language A. Klee & T. L. Face (Eds.), Selected proceedings of the
learning tasks. In R. K. Johnson (Ed.), The second 7th Conference on the Acquisition of Spanish and
language curriculum (pp. 187-206). Cambridge: Portuguese as First and Second Languages (pp. 74-85).
Cambridge University Press. Somerville, MA: Cascadilla Proceedings Project.
Bygate, M., Skehan, P., & Swain, M. (2001). Introduction. Heift, T. (2003). Drag or type, but don't click: A study
In M. Bygate, P. Skehan, & M. Swain (Eds.), on the effectiveness of different CALL exercise
Researching pedagogic tasks: Second language learning, types. Canadian Journal of Applied Linguistics / Revue
teaching and testing (pp. 1-22). New York: Pearson. canadienne de linguistique appliquée, 6(1), 69-85.
Chander, J. (2003). The efficacy of various kinds of error Heift, T. (2005). Inspectable learner reports for web
feedback for improvement in the accuracy and based language learning. ReCALL, 17( 1), 32-46.
Trude Heifl and Anne Rimrott
Hill, M., & Laufer, B. (2003). Type of task, time-on-task Pienemann, M. (1998). Language processing and seco
and electronic dictionaries in incidental vocabu language development: Processability theory. Amster
lary acquisition. International Review of Applied dam: John Benjamin.
Linguistics, 41, 87-106. Porter, D., & O'Sullivan, B. (1999). The effect of
Hulstijn, J. H., & Marchena, E. (1989). Avoidance: audience age on measured written performance.
Grammatical or semantic causes? Studies in Second System, 27, 65-77.
Language Acquisition, 11, 241—255. Purpura, J. E. (2004). Assessing grammar. Cambridge:
Kleinmann, H. H. (1977). Avoidance behaviour in adult Cambridge University Press.
second language acquisition. Language Learning,Reinders, H. (2010). The effects of task type and instruction
27, 93-107. on second language acquisition. Newcastle, UK:
Kleinmann, H. H. (1978). The strategy of avoidance in Cambridge Scholars.
adult second language acquisition. In W. C. Ritchie Robinson, P. (2007). Criteria for classifying and sequenc
(Ed.), Second language acquisition research: issues and ing pedagogic tasks. In M. Garcia Mayo (Ed.),
implications (pp. 157-174). New York: Academic Investigating tasks in formal language learning (pp. 7
Press. 27). Clevedon, UK: Multilingual Matters.
Krashen, S. (1981). Second language acquisition and second Samuda, V., & Bygate, M. (2008). Tasks in second language
language learning. Oxford, UK Pergamon Press. learning. Basingstoke, UK Palgrave Macmillan.
Lai, C., & Li, G. (2011). Technology and task-based Schachter, J. (1974). An error in error analysis. Language
language teaching: A critical Review. CALICO, 28, Learning, 24, 205-214.
498-521. Schjoldager, A. (2004). Are L2 learners more prone
Larsen-Freeman, D. E. (1975). The acquisition of to err when they translate? In K Malmkjaer
grammatical morphemes by adult ESL students. (Ed.), Translation in undergraduate degree programmes
TESOL Quarterly, 9, 409-419. (pp. 127-150). Philadelphia/Amsterdam: John
Larsen-Freeman, D. (1978). An ESL index of develop Benjamins.
ment. TESOL Quarterly, 12, 439-448. Skehan, P. (2003a). Focus on form, tasks, and technology.
Larsen-Freeman, D. (2006). The emergence of complex Computer Assisted Language Learning, 16, 391-411.
Skehan, P. (2003b). Task-based instruction. Language
ity, fluency, and accuracy in the oral and written
production of five Chinese learners of English. Teaching, 36, 1—14.
Applied Linguistics, 27, 590-619. Skehan, P., & Foster, P. (1997). The influence of
Larsen-Freeman, D. (2009). Adjusting expectations: The planning and post-task activities on accuracy and
study of complexity, accuracy, and fluency in complexity in task-based learning. Language Teach
second language acquisition. Applied Linguistics, ing Research, 1, 16-33.
30, 579-589. Tarone, E. (1985). Variability in interlanguage use: A
Laufer, B., & Eliasson, S. (1993). What causes avoidance study of style-shifting in morphology and syntax.
in L2 learning? Studies in Second Language Acquisi Language Learning, 35, 373-403.
tion, 15, 35—48. Tarone, E. (1987). Methodologies for studying variability
Laufer, B., & Roitblat-Rozovski, B. (2011). Incidental in second language acquisition. In R. Ellis, (Ed.),
vocabulary acquisition: The effects of task type, Second language acquisition in context (pp. 35-46).
word occurrence and their combination. Language Englewood Cliffs, NJ: Prenctice Hall.
Teaching Research, 15(1), 391-411. Tarone, E. (1988). Variation in interlanguage. London:
Laufer, B., & Girsai, N. (2008). Form-focused instruction Edward Arnold.
in second language vocabulary learning: A case for Tarone, E., & Parrish, B. (1988). Task-related variation in
contrastive analysis and translation. Applied Linguis interlanguage: The case of articles. Language
tics, 29, 694-716. Learning, 38, 21-44.
Leal, D. J. (2005). The word writing CAFE: Assessing Thomas, M., & Reinders, H. (2010). Task-based languag
student writing for complexity, accuracy, and teaching and technology. London: Continuum.
fluency. Reading Teacher, 59, 340-350. Thouësny, S. (2010). Investigating learner variability:
LoCoco, V. G.-M. (1976). A comparison of three The impact of task type on language learners'
methods for the collection of L2 data: Free errors and mistakes. CALICO foumal, 28, 21-34.
composition, translation, and picture description.
Thouësny, S. (2011). Modeling second language learner
Working Papers on Bilingualism, 8, 59-86. interlanguage and its variability: A computer-based
Lyster, R. (2004). Research on form-focused instruction
dynamic assessment approach to distinguishing between
in immersion classrooms: Implications for theory
errors and mistakes. Unpublished doctoral disserta
and practice. French Language Studies, 1, 321-341.
tion. Dublin City University, Dublin, Ireland.
Newton, J., & Kennedy, G. (1996). Effects of communi
Van den Branden, K (Ed.). (2006). Task-based languag
cation tasks on the grammatical relations marked
education: From theory to practice. Cambridge: Cam
by second language learners. System, 24, 309-322.
bridge University Press.
Ortega, L. (2003). Syntactic complexity measures and B. (2007). Phrasal verbs in learner English: A
Waibel,
their relationship to L2 proficiency: A research
corpus-based study of German and Italian students.
synthesis of college-level L2 writing. Applied
Unpublished doctoral dissertation, Albert-Lud
Linguistics, 24, 492-518. wigs-University, Freiburg i. Br., Germany.

Task-Related Variation in Computer-Assisted Language Learning 2012

Загружено:

Сведения о документе

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

Task-Related Variation in Computer-Assisted Language Learning 2012

Загружено:

Авторское право:

Доступные форматы

Task-Related Variation in Computer-Assisted Language Learning

Author(s): TRUDE HEIFT and ANNE RIMROTT

National Federation of Modern Language Teachers Associations, Wiley are collaborating

Simon Fraser University Simon Fraser University

Email: heifi@sfu.ca Email: arimrott@sfu.ca

This study investigates task-related variation in learner performance in a computer-assisted

stresses the importance of investigating different and proficiency-related performance differences

by examining previous work on task-related

1996, p. 44). attributed to the differences in task alone but that

high intake but less acquisition than the individual

Summary of Previous Findings of instruction. For this study, we consider data

performing the same three task types first at a

Sample Essay Task From German 200, Chapter 106

mtffl > H N H Q._f.

Where did you go?

Use your imagination.

£3 Submit your work

performance. task types.

Differences Among the Three Task Types

Setting CALL task CALL task CALL task

times as they wished, receiving immediate, error

Course Chapter Grammar Content

German 100 Chapter 1 Basic sentences and questions

Chapter 6 Simple Past of modals, time expressions

Chapter 10 Simple past, past perfect

then divided those into the number of sentences

German 100 German 200 Both Courses

S-Building 82.5% .07 75.8% .10 79.2% .08

S-building Translation -.022 .023 .352

Essay -.097 .023 .000

Chapter 1 Chapter 2 Chapter 3 Chapter 4 Chapter 9 Chapter 10 Chapter 11 Chap

S-Building 89.3% 83.0% 80.8% 77.0% 84.0% 75.0% 68.7% 75.6%

Usage of Grammatical Constructions in Chapters 4 and 11 for the Essay Task

use longer/more complex constructions as in the more detail in the following.

A Developmental Perspective. The decrease Learner

Table 8 displays some of our data that relates

Interlearner Variability: Overall Accuracy Rates by Task Type

S-Building Translation Essay All Task Types

Student A 87.7% 100.0% 95.1% 94.3%

79.2%, range: 7.3%) whereas student C's perfor

Interlearner Variability: Accuracy Rates by Course and Task Type

S-Building Translation Essay

100 200 100 200 100 200

Student A 93.7% 81.7% 100.0% 100.0% 96.1% 93.8%

students also performed better on the Essay task in CONCLUSION

German 200 than in German 100 (students B, E,

10 Note, however, that for our data analysis, only the

Вам также может понравиться