Академический Документы
Профессиональный Документы
Культура Документы
, Edith TESTING SKILLS AND COMPONENTS: A Practical Guide for EFL Teacher Trainees
CHAPTER VI
PRODUCTIVE SKILL
In a very general way, oral communication can be defined as an interactive process in which
an individual alternately takes the roles of speaker and listener, and which includes both
verbal and nonverbal components. This definition clearly states the fact that in many tests of
oral production, it is neither possible nor desirable to separate the speaking skills from the
listening skills. In normal speech situations the two skills are interdependent. As Heaton
1979 claims, it is impossible to hold any meaningful conversation without understanding
what is being said and without making oneself understood at the same time.
Yet this is not the only reason why it is so difficult to assess with precision the speaking
ability. This complex skill requires the simultaneous use of a number of different abilities and
components which often develop at different rates. Pronunciation, grammar and vocabulary
are language components that are difficult to over look when testing oral production. There
are still other factors such as fluency, comprehension and appropriateness that also play
important roles in oral communication. In brief, the elements of speaking are numerous and
the test administrator will have to decide how to evaluate so many things at once.
Two methods are used for assessing speaking skills. In the observational approach, the
student’s behaviour is observed and assessed unobtrusively. In the structured approach, the
student is asked to perform one or more specific oral communication tasks. His performance
on the task is then evaluated using a rating system. The task can be administered in one-on-
one setting (with the administrator and one student) or in a group or class setting. It is
important to consider that in either setting, students should feel that they are communicating
meaningful content to real audience.
In this chapter, I am going to deal mainly with the structured approach since the purpose of
this material is to introduce the teacher trainees into testing techniques that they will employ
when planning their formal evaluations.
Scoring the test is as important and challenging as getting the student to say something
during his performance in a speaking test. So in the last part of this chapter, the scoring
procedures, holistic and objectified will be explained.
Pictures Production
Paraphrase Production
Explanation Production
CONTROLLED TECHNIQUES:
Some of these techniques are used with students with limited speaking skills since the
directions of the item instruct in a deliberately way, what specifically the students must do.
They restrict the performance of the student to a very specific kind of answer. Even though,
they are generally used with beginners, the level of difficulty may be increased when
introducing items that demand more student initiative in answering those propositions.
Malagarriga G., Edith TESTING SKILLS AND COMPONENTS: A Practical Guide for EFL Teacher Trainees. 106
DIRECTED RESPONSES.
This technique requires of a specific direction or cue for every item. The cues or instructions
go from very simple or quite artificial cues to directions which require more creativity and
initiative on the part of the participants. Here there are some examples:
Examples:
The student’s response would be, “She lives in Canada,” this is just a repetition of
the last part of the tester’s cue.
The student’s response would be, “I was born in Scotland.” In this particular case,
the student must be aware that a modification is required, the personal pronoun
changes and so the verb to be.
6.3. (cue) “Tell him politely that he can come this afternoon.”
The student’s answer could be, “Excuse me sir, the doctor says that you can
come this afternoon.” Here the student, depending on his level, will provide
different kinds of answers.
The answer could be, “It’s very kind of you to invite me, but I’m not sure if I can”
or “Thank you very much for the invitation, but I don’t really think I can. In this
case the student, depending on his level, will refuse the invitation in a particular way.
6.5. (cue) “Express a personal opinion about the last conference you attended.”
The student’s response could be, “Personally, I think It was excellent” or “As far
as I’m able to judge, it was a waste of time.” Here the student, depending on his
level, will provide different opinions.
6.6. (cue) “Show concern and give advice to a friend who is telling you about her
terrible cold”
The answer could be, “I’m sorry to hear that. Maybe you should go home, get
some rest and drink plenty of liquids.” In this case the participant, according to his
experience, will react in quite different ways.
PICTURES.
The use of pictures, maps and diagrams is very useful for testing oral production since they
serve as stimuli for descriptions, narrations, comparisons, and conversational exchanges.
Malagarriga G., Edith TESTING SKILLS AND COMPONENTS: A Practical Guide for EFL Teacher Trainees. 107
When applying this technique, the student is given a picture to look at and study for a minute
or a few minutes; he is then required to describe the picture or narrate the series of events in
a given time. Finally the examiner will score the student’s performance by using the most
appropriate scoring procedure.
The careful selection of the pictures will help the examiner control the vocabulary and the
structures used for the description of a particular sketch. There are many sources where to
find good pictures, like: books, magazines, newspapers, posters, comic strips, etc. Look at
the following examples:
INSTRUCTIONS.
Look at this sketch for a minute in order to answer the questions the examiner is
going to ask you.
Example:
6.7.
Even though this is a controlled technique, the sketch itself and the difficulty level of the
questions will determine the type of students who are being tested.
INSTRUCTIONS.
Look at these pictures for a minute in order to narrate in a sequential way the
events that are taking place.
Example:
6.8.
Malagarriga G., Edith TESTING SKILLS AND COMPONENTS: A Practical Guide for EFL Teacher Trainees. 109
INSTRUCTIONS.
Look at this chart about popular sports for a minute in order to explain the
popularity of these sports during those years.
Example:
6.9.
Note: Besides pictures, the examiner can also use real objects for the students to describe.
For example the tester may give the student a handbag, a watch, a wallet, a book or any
other object for the student to make a detailed description.
READING ALOUD.
The examinee is given several minutes to read a passage silently, after which he is
instructed to read it aloud at normal speed and with appropriate expression. This technique
offers some advantages as well as significant disadvantages. It provides the opportunity to
check if the students have mastered the sound-symbol relationships in English, through their
pronunciation. It also gives the teacher the chance of determining fluency and grammar
control since errors in reading aloud may reflect confusion about structural patterns in the
foreign language.
There are quite a few limitations that discourage the use of this technique for testing
speaking skills. First of all, reading aloud can not be employed with young children who
haven’t yet reach the reading skill, or with students, young or adults, whose speaking skills
are much more advanced than their reading skills. In second place, the emphasis that many
people place on pronunciation constitutes a secondary point when testing speaking at
intermediate and advanced levels. Here I am more concerned with interaction, fluency and
appropriateness of response. Finally, the fact that a student reads well may not guarantee a
good performance in his speaking skill.
There are essentially two ways to approach this technique. A very punctual one that consists
of giving the student a sentence with the specific aspects, the teacher wants to assess. The
other type is to give the student a passage that contains several points the teacher has to
score. This last one is useful for diagnostic purposes since the student is going to be faced
to a large variety aspects that will allow the tester to identify the student errors. Look at the
examples that follow:
INSTRUCTIONS.
Malagarriga G., Edith TESTING SKILLS AND COMPONENTS: A Practical Guide for EFL Teacher Trainees. 110
Examples:
6.9. The student reads: Hello George, you don’t look too well !
6.10. The student reads: Have you got anything fixed up for this evening?
6.11. The student reads: Well, I’m awfully sorry but I don’t think I’ll be
able to come after all.
INSTRUCTIONS.
Read this passage silently for a minute then read it aloud at a normal speed.
Example:
It’s really never too early to start reading to a child. If a child is old enough to
talk toand parents talk to their children from Day 1then he or she is old
enough to be read to.
It doesn’t matter if they cannot understand the words; the language inside the
covers of books is frequently a whole lot more organized, colourful and coherent
than “ koochie, koochie, koochie.”
Holding even a tiny child in front of a book and reading to him or her is
intellectual stimulation. A visual competency is developing, too, because the child
is being taught to focus attention on a picture on a page.
This visual literacy is just as important as print literacy, and it is usually
achieved before print literacy. At 18 months old, a child can identify a picture
of a puppy and understand the wordand that’s long before he/she can read it.
(Taken from Milan, 1988)
The preceding passage allows the teacher to diagnose errors that the student may present in
speaking. Sometimes, the student has internalised a mistake in such a way, that even when
reading a text, he produces that error automatically
GUIDED TECHNIQUES.
These techniques, as the name suggests, provide a kind of guidance or help for the student
to follow. Even when they supply a specific instruction, they do not limit the response of the
Malagarriga G., Edith TESTING SKILLS AND COMPONENTS: A Practical Guide for EFL Teacher Trainees. 111
student, they allow the student to freely produce an answer that may also evidence some
creativity.
PARAPHRASE.
Although paraphrase is a technique that incorporates another skill to the one that is being
tested, it can be used at almost any level. In order to make the student paraphrase an idea,
the examiner must present the input with either a listening or a reading text. As has been
indicated in preceding chapters, in the implementation of this technique, the student is given
a text that he will later say in his own words. Take a look at the following items:
INSTRUCTIONS.
Listen to the following statement as I read it. Then tell me its content in your own
words.
Example:
America, the land of opportunities, the dream land. For some people America is
the land of opportunities, but for others it is the land of self-destruction.
INSTRUCTIONS.
Read the following story silently. Then tell me its content in your own words.
Example:
HER LUCK
by Parastoo Khorramian
She was a little girl who lived in a small town in Iran. In her small town people
were separated into two groups, lucky and unlucky. No matter how hard they
worked, the results were the same. Some could improve their lives by working
hard and some could not. In the little girl’s case, her family could not improve
their life. Her family wasn’t lucky. Her mother and her father worked hard, but
still there wasn’t enough money at home. Thinking of being unlucky forever would
make her sad, but one day everything changed. (Taken from Oxford and Scarcella,
1994)
girl’s family was unsuccessful, her parents were unable to support the family and this
made the girl feel very unhappy until the day that a transformation occurred.
EXPLANATION.
This technique can be used with students with at least an intermediate level in the language.
You can either ask a student to explain something he has just read or you may ask that
student to explain a special celebration in his country, a festivity, a particular custom, a
traditional feast, a particular event, a special recipe, how to get from one place to another,
how to use an automatic money machine, etc. The following are just some ideas:
Examples:
6.15. INSTRUCTIONS.
You are going to have three minutes approximately to explain how Venezuelan
people celebrate “Carnival.”
6.16. INSTRUCTIONS.
You are going to have three minutes approximately to explain how to make a collect
call to your country.
6.17. INSTRUCTIONS.
I am going to give you three minutes approximately to describe a traditional recipe
for Christmas celebrations in your country.
INSTRUCTIONS.
I am going to give you a situation in order to create a conversation between you and
me. Try to respond naturally using the appropriate structures and vocabulary.
Example:
6.18. Situation: Mrs. Morgan goes to the grocery store to buy the things she needs at home.
Mr. Grant , the clerk, helps her to choose the groceries. Now I am going to play the role of
the clerk and you are going to be Mrs. Morgan.
The fixed role of the teacher or the teacher assistant should be performed as natural as
possible, at a normal speed and with only a brief look at the outline. It is also important
to mention that the person taking the fixed role must adapt his part to the student’s
responses.
OTHER TECHNIQUES.
This part includes those techniques which even when they allow more freedom and creativity
to the participants, they can be used with students with different levels of performance. In
this section, I am going to deal with only a few of the many techniques which can be used
for this purpose.
ORAL INTERVIEW.
It could be claimed that this is the most common way to assess the ability to interact with
other people. However, it has a severe disadvantage related to the relationship between the
tester and the participant. No matter how special their rapport is, there will always be a
superiority dependency that may affect the performance of the student during the interview.
There is also another drawback that has to do with the scoring of the interview where the
subjectivity plays an important role. This can be controlled, in a way, by using a reliable
scoring procedure.
The level of the student involved in the interview will determine the elicitation techniques that
the interviewer will apply. This means that all the controlled and guided techniques
introduced in this chapter may be used, as elicitation procedures.
Conduct the interviews in some quiet place with suitable acoustics. Echoes, street
noises, conversation in an adjacent room will greatly reduce the reliability and
validity of the ratings.
Reserve sufficient time for each interview. Ten to fifteen minutes would seem
essential as the minimum for each interview, though the time required will vary
somewhat from one student to another.
Use at least two raters for each student. At least two independent ratings are
necessary if satisfactory rater reliability is to be obtained. The two (or more) raters
may either interview separately, one after the other in different rooms, or they may
independently score the same performance. If the latter is used, it is generally
advisable for one of the raters to conduct the interview while the other observes from
the back of the room.
Score the student immediately after he leaves the room. The procedure for bringing
in the participants should be so arranged that the next examinee will not enter until
the marking has been completed.
Obtain each candidate’s final score by pooling or averaging the ratings that have
been given him. If the difference between the ratings is clearly excessive, the
student should be called back for a second evaluation. (Adapted from Harris, 1969)
The interview should begin with some social questions: “How are you today?”
“Where do you live?” “Do you have brothers and sisters?” Such questioning will
serve as a warming up and also as a way to determine how well he can function in a
social situation.
Avoid highly personal items related to: religion, marital status, weight, age, etc.
During the initial period the interviewer should speak at normal conversational
speed. If it is immediately apparent that the candidate cannot comprehend what is
being said to him, it may be necessary for the interviewer to modify his speech
somewhat, speaking more slowly and with some simplification of sentence structure
and vocabulary –while making a mental note to score the candidate accordingly. If
the candidate continues to have extreme difficulty, it is best to make the interview
brief. It is valueless to attempt to carry on a prolonged interview in what can easily
degenerate into non-English on the part of both participants.
After the preliminary questions, the interviewer should move on to other areas of
discussion that reveal the candidate’s proficiency in handling the aspects that are
being tested.
Malagarriga G., Edith TESTING SKILLS AND COMPONENTS: A Practical Guide for EFL Teacher Trainees. 115
It is absolutely essential that the questioning should not always follow a set pattern.
Some students immediately start with a prepared speech on a subject for which the
previous interviewees have obviously prepared him.
Sometimes it would be a good idea that the interviewer occasionally inserts his own
ideas on the subject being discussed, thus permitting him to observe how well the
participant can comprehend, and respond to expository speech.
Special Care should be taken not to give the student the impression that he is being
cut off in the middle of the discussion.
Care must be taken when selecting the topics. In this sense, subjects about which a
participant knows very little should be avoided. On the contrary, candidates talk better when
they have something valuable to say and can incorporate particular experiences and some
imaginative situations.
The topics may be presented as either claims or questions. Look at these examples:
Examples:
1.19. INSTRUCTIONS.
You are going to have fifteen minutes to prepare a ten-minute talk about “The
Pernicious influence of examinations in education”
1.20. INSTRUCTIONS.
I am going to give you twenty minutes for you to prepare a fifteen-minute talk that
answers the following question: “Should compulsory military service be abolished in
all countries?”
1.21. INSTRUCTIONS.
Prepare a ten-minute talk to develop the following topic: “Parents are too permissive
with their children nowadays.” You are going to have half an hour to prepare the
speech. Here you have some material you may use for the preparation.
ROLE PLAYING.
Malagarriga G., Edith TESTING SKILLS AND COMPONENTS: A Practical Guide for EFL Teacher Trainees. 116
Candidates are asked to assume a role in a particular situation. Thus two or three students
are given a situation and then assigned roles. The performance of a role play involves
several aspects: the use of appropriate register of language, the application of adequate
structures and the utilization of extra-linguistic factors. The following is an illustration of this
technique:
1.22. INSTRUCTIONS.
Listen to the following situation where three people are going to be involved in the
conversation. Thomas is going to be Steve; Catherine will be Jane and Michael is
going to perform the role of the landlord, Mr. Lee.
Steve and Jane are looking for a place to live. They have seen many apartments, but
they haven’t found anything they like. They have just arrived to see another
apartment. They have met the landlord , and he is showing them the apartment.
Example:
Jane: The kitchen is wonderful, isn’t it, Steve? Look at all the cabinets and drawers.
Steve: The counter space couldn’t be better, and I really like the work area in the middle. The cabinets
aren’t in the best condition, but I can restore them myself. The appliances are supplied, aren’t
they, Mr. Lee?
Mr. Lee: No. This is an unfurnished apartment. The stove and the refrigerator belong to the present
tenant.
Jane: What do you think, honey? The bedrooms aren’t very big.
Steve: No, but there are two of them, and they have great closets. The living room/dining room area is
tremendous. There’s room for the couch under that window, and my recliner can go right here.
Steve: It ‘s definitely the best place we’ve seen so far. Excuse me, Mr. Lee. What did you say the rent
was?
Mr. Lee: $ 900 a month. If you decide to rent it, I’d like three month’s rent as a security deposit.
Mr. Lee: By the end of this month. You can move in on the first.
SCORING PROCEDURES.
Malagarriga G., Edith TESTING SKILLS AND COMPONENTS: A Practical Guide for EFL Teacher Trainees. 117
Scoring a speaking test is a complex task that needs to be simplified by the use of a specific
scoring procedure that will provide a measure of quality of performance on the basis of
established criteria. The examiner will have to select the scoring system he is going to use
and this decision will depend either on the examiner’s experience in the evaluation of oral
communication or on the aspects he pretends to evaluate. In this sense, the tester will
choose between applying a holistic, an objectified or an analytic scoring procedure.
HOLISTIC SCORING.
This kind of procedure responds to language performance as a whole. Raters using holistic
scales are trained not to think about the individual components of the oral skill or to count the
number of errors that the student make. Each score on the scale represents an overall
impression that serves as a standard against which production is judge. Normally the scores
are on scales of 1 to 4, or 1 to 6, or even 1 to 10. Each level on the scale is accompanied by
a verbal description of the performance required to achieve that score. The emphasis of this
procedure is on what the student does well.
It is important to mention that the use of holistic scoring is recommended for experienced
teachers, since it is rather difficult to provide a consistent and an accurate holistic grading
without having had a proper training and practice in this kind of evaluation. Look at this five
point scale developed by the Foreign Service Institute:
On the next pages you will find samples of the following assessment instruments:
OBJECTIFIED SCORING.
This kind of scoring refers to a system where points are assigned to every answer.
Depending on the difficulty level of the propositions, the tester will allow a full credit or a
partial credit to each student response. An illustration for this procedure could be: 2 points
for fully correct answers; 1 point for partially correct responses; and 0 for unacceptable
answers.
According to Bachman 1990, objectified scoring can be used by teachers with little
specialized training and by highly trained teachers who prefer evaluation which is consistent
and easy to use.
The scoring criteria for the determination of what is right and what is partially right should be
carefully discussed in advance. A suggestion could be to give half credit when serious
language errors occur, since it would be very difficult to get a full credit on any item if minor
errors are penalized.
Malagarriga G., Edith TESTING SKILLS AND COMPONENTS: A Practical Guide for EFL Teacher Trainees.
comprehensible in its has no unnatural includes all relevant employs consistent and is mostly correct with only Varied and accurate
entirety with a few pauses; sounds like information accurate use of structures; minor flaws
5 minor flaws natural speech. may contain a few minor
errors that don’t interfere with
the communication
generally fairly smooth with a few includes most relevant generally uses correct influenced by first appropriate
comprehensible unnatural pauses; information structures with some errors language
slight choppiness
4 and/or occasional error
in information.
somewhat occasionally halting includes a fair amount of demonstrates an inconsistent shows strong influence simple with some
comprehensible and fragmentary with relevant information; may use of correct structures from first language inappropriate use
some unnatural include
pauses, choppiness, or contradictions,informational
3 inappropriate gaps, or redundancies
intonation
difficult to understand very halting and little relevant information is shows many errors in use is dominated be first limited or incorrect
fragmentary with many presented and structure language
2 unnatural pauses;
speech sounds
mechanical
incomprehensible very halting and vague or confusing has no apparent interferes with very poor or
fragmentary with information is presented understanding of structures comprehension inaccurate for topic;
excessive unnatural first language words
1 pauses may be used;
speaker may create a
target language form
from first language
ANALYTIC SCORING.
This scoring procedure requires a separate score for each aspect of the scale. Then a
candidate’s total mark is the sum of the weighted scores. It is important to mention that the fact
that the examiner has to assign several scores to each student will tend to make the scoring
more reliable. Look at the following scale for an analytic scoring:
PROFICIENCY DESCRIPTION
ACCENT:
1. Pronunciation frequently unintelligible.
2. Frequent gross errors and very heavy accent make understanding difficult,
require frequent repetition.
3. “Foreign accent” requires concentrated listening, and mispronunciations
lead to occasional misunderstanding and apparent errors in grammar or
vocabulary.
4. Marked “foreign accent” and occasional mispronunciations which do not
interfere with understanding.
5. No conspicuous mispronunciations, but would not be taken for a native
speaker.
6. Native pronunciation, with no trace of “foreign accent.”
GRAMMAR:
1. Grammar almost entirely inaccurate except in stock phrases.
2. Constant errors showing control of very few major patterns and frequently
preventing communication.
3. Frequent errors showing some major patterns uncontrolled and causing
occasional irritation and misunderstanding.
4. Occasional errors showing imperfect control of some patterns but not
weakness that causes misunderstanding.
5. Few errors, with no patterns of failure.
6. No more than two errors during the interview.
VOCABULARY:
1. Vocabulary inadequate for even the simplest conversation.
2. Vocabulary limited to basic personal and survival areas (time, food,
transportation, family, etc.).
3. Choice of words sometimes inaccurate, limitations of vocabulary prevent
discussion of some common professional and social topics.
4. Professional vocabulary adequate to discuss special interests; general
vocabulary permits discussion of any non-technical subject with some
circumlocutions.
5. Professional vocabulary broad and precise; general vocabulary adequate
to cope with complex practical problems and varied social situations.
6. Vocabulary apparently as accurate and extensive as that of an educated
native speaker.
Malagarriga G., Edith TESTING SKILLS AND COMPONENTS: A Practical Guide for EFL Teacher Trainees. 124
PROFICIENCY DESCRIPTION
FLUENCY:
1. Speech is so halting and fragmentary that conversation is virtually
impossible.
2. Speech is very slow and uneven except for short or routine sentences.
3. Speech is frequently hesitant and jerky; sentences may be left
uncompleted.
4. Speech is occasionally hesitant, with some unevenness caused by
rephrasing and groping for words.
5. Speech is effortless and smooth, but perceptively non-native in speed and
evenness.
6. Speech on all professional and general topics as effortless and smooth as
a native speaker’s.
COMPREHENSION:
1. Understands too little for the simplest type of conversation.
2. Understands only slow, very simple speech on common social and
touristic topics; requires constant repetition and rephrasing.
3. Understands careful, somewhat simplified speech when engaged in a
dialogue, but may require considerable repetition and rephrasing.
4. Understands quite well normal educated speech when engaged in a
dialogue, but requires occasional repetition and rephrasing.
5. Understands everything in normal educated conversation except for very
colloquial or low-frequency items, or exceptionally rapid or slurred speech.
6. Understands everything in both formal and colloquial speech to be
expected of an educated native speaker.
CONTROLLED TECHNIQUES.
A. DIRECTED RESPONSES.
1. Prepare a battery of six items with specific directions or cues for students with a low level of
proficiency in English.
2. Write suitable instructions and directed-request items that require your students to produce
the following statements:
2.1. This is my next-door neighbour.
2.2. When do you usually see the patients?
2.3. Well, we could go to a pub. There’s a good one not far from the office.
Malagarriga G., Edith TESTING SKILLS AND COMPONENTS: A Practical Guide for EFL Teacher Trainees. 125
2.4. I think the government failed to realise how serious the situation was.
2.5. Sorry for being late, but there was this important meeting.
B. PICTURES.
1. Find a sketch and prepare questions to be asked by the examiner. Specify the students’
level.
2. Find or draw a picture with a sequence of events for the students to narrate. Prepare
questions that help the students tell the story. Specify the students’ level.
C. READING ALOUD.
Find a passage for your students to read aloud that allows you diagnose the errors that your
students may present in speaking. Specify the students’ level.
GUIDED TECHNIQUES.
A. PARAPHRASE.
1. Write down two complex ideas for the students to hear in order to tell you, in their own words
the whole message.
2. Find a not very long story for your students to read silently. Then ask them to tell you its
content in their own words.
B. EXPLANATION.
Provide a list of special celebrations, festivities, particular customs or the like for your students to
produce a clear explanation about them.
Prepare a guided role play to be used to test your students oral production on a particular topic.
Supply samples of students responses. Specify the students’ level.
OTHER TECHNIQUES.
A. ORAL INTERVIEW.
Prepare twenty items to be used in a guided interview. Specify the students’ level and the
scoring procedure. Include items from different techniques, such as: directed response,
paraphrase, explanation, pictures, etc.
Malagarriga G., Edith TESTING SKILLS AND COMPONENTS: A Practical Guide for EFL Teacher Trainees. 126
Provide a list of topics for your students to give a ten-minute talk. Include detailed instructions
for each topic.
C. ROLE PLAYING.
Prepare a situation for your students to produce a role playing. Include in the directions the
distribution of the roles and any other special aspects you are interested in. Specify the
students’ level.
CHALLENGE.
Design an item that shows your inventiveness and creativity. Specify the level and kind of
students.