Language Teaching Research 13,3 (2009); pp.

259278

The evaluation of foreign-language-teacher education programmes

Matthew Peacock City University of Hong Kong

This article presents a new procedure for the evaluation of EFL teacher-training programmes based on principles of programme evaluation and foreign-languageteacher (FLT) education. The procedure focuses on programme strengths and weaknesses and how far the programme meets the needs of students. I tested the procedure through an evaluation of a TEFL programme, collecting data from students, teachers, and alumni through interviews, questionnaires, essays and analysis of programme materials. The programme had many strengths, including the teaching of pedagogic skills and promoting reflection and self-evaluation, but also shortcomings. It should increase the amount of practice teaching and increase input in certain areas such as knowledge of teaching within the local sociocultural context and classroom management. I suggest that the procedure may be useful for other contexts. Keywords: teacher education, programme evaluation

I Introduction
While the literature on programme evaluation and foreign-language-teacher (FLT) education is extensive, the literature seems to contain very few if any descriptions of a procedure for the overall evaluation of FLT education programmes. It was the aim of the research on which this paper reports to design and test such a procedure. The method was to review the literature on programme evaluation and the recommended content and procedures of FLT education programmes, then design a procedure that uses recognized methods of programme evaluation to assess programmes in terms of how far they match those recommendations. I tested the procedure by using it to evaluate an existing programme, and assessed its value for use in other contexts.

II Background to the research

The present research was based on two main premises. First, it is important for every teacher-training programme to have a system for regular internal
Address for correspondence: Matthew Peacock, Department of English, City University of Hong Kong, 83 Tat Chee Avenue, Kowloon, Hong Kong; email: enmatt@cityu.edu.hk
The Author(s), 2009. Reprints and permissions: http://www.sagepub.co.uk/journalspermissions.nav 10.1177/1362168809104698

260

FLT education programmes

evaluation. There is a lot of support for this in the literature on language-teacher preparation (much of which dates from the 1990s): Rea-Dickins and Germaine (1998, p. 8) stress the importance of having systematic evaluation at the heart of a programme. Many others also emphasize this, e.g. Richards (1990), Wallace (1991), Reid (1996) and Lynch (2003). Second, the field of teacher education remains under-researched. In 1996 Freeman noted (p. 351) that scant attention has been paid still an unstudied problem. He calls for the development of the area as an important field of enquiry and study subject to little critical scrutiny and less organized study (p. 374). Freeman and Johnson (1998, p. 397) say research in the area is noticeably missing. A number of authors make recommendations regarding the content and procedures of FLT education programmes. Wallace (1991, p. 141) suggests they need a clearly stated philosophy, and that programme content should reflect it. He also asserts (p. 1415) that programmes should balance received versus experiential knowledge. Wedell (1992, p. 344) proposes that programmes balance the required components of FLT training linguistic, pedagogic and managerial competence. Linguistic competence means language proficiency; Cots and Arno (2005) note that we must distinguish this from knowledge about language. Crandall (1993, p. 507) says programmes should promote trainee flexibility in using different teaching approaches for different situations, and Wallace (1991, p. 145) adds the ability to use and adapt teaching materials. Another suggestion (Wallace, 1991) is that programmes incorporate and encourage trainee reflection on the experiences and values they have when they enter; in particular, they should encourage trainee reflection on the effects of Lorties (1975, p. 26) apprenticeship of observation. This refers to preservice teachers having spent 10 000+ hours watching teachers in school. Reflection may be absent from many TEFL programmes: Grosse (1991, p. 43), after posting a questionnaire on 94 TESOL methods courses (not programmes), says too many lack a reflective component. Wu (1996, p. 20), describing the Chinese view of TEFL training, notes that trainees are passive receivers seeing what to do is more important than understanding why it is done and that for trainees, experienced teachers are models to be observed and followed (p. 20). Liou (2001, p. 198) says this view still holds in Taiwan. Wallace (1991) asserts that programmes need to promote the skill of reflection and self-evaluation as a teacher. A connected suggestion (Stoynoff, 1999, p. 148; Lo, 2005) is that programmes need to promote the long-term and developmental character of learning to teach (that is, post-qualification teacher growth and development) and others add that they must promote future reflective practice (Crandall, 1993; Bartolome, 1994; Lynch, 2003). Further proposals for programme content are that they should embody coherence among courses in linkage and avoid overlaps (Weir and Roberts, 1994, pp. 1124); be up-to-date; and balance teacher- and student-centred

Matthew Peacock 261

learning (Bartels, 2005, p. 416; Wallace, 1991, pp. 14, 1134). Bartolome (1994, p. 179) adds that it is important to evaluate programmes in terms of how well they prepare EFL teachers to function in the sociocultural context in which they will work. Freeman and Johnson (1998, pp. 400, 406, 4089) develop this and suggest it is essential to expand views of the knowledge-base of foreign-language teachers to include knowledge of the social context of learning (i.e. classrooms), because learning cannot be fully understood without it. They add (pp. 4012) that some programmes may place too much emphasis on theoretical and teaching skills. Johnson (2006, pp. 2367) adds that this knowledge, and the fact that learning is a dynamic social activity, is an important part of teacher cognition and that its importance has not been adequately recognized in teacher education (pp. 237, 243). However, Yates and Muchisky (2003, pp. 13640, 144) raise the objection that their arguments might marginalize the more central study of language and of secondlanguage acquisition, and I feel they have a very strong point. Freeman and Johnson in a response article (2004, pp. 120, 122) say they did not do this; Muchisky and Yates (2004, pp. 135, 137, 139), in a further response, emphasize that while they do recognize a cultural component to language, they wish to stress the critical and central role in language-teacher education of the nature of language and how it is learned. Further suggestions for evaluation are to ask students whether programmes meet and are relevant to their needs (Wallace, 1991, p. 147), and how well the programmes prepare them for classroom teaching (Reid, 1996, p. 3). Roberts (1998) also makes valuable contributions. He notes (p. 235) that evaluations are never neutral, as they expose stresses and competition in institutions, creating a need to identify clear questions and plan evaluations carefully (p. 236). He adds that they increase teacher feelings of ownership if teachers share the information gathered, and he also provides a detailed history and examination of Certificate in English Language Teaching to Adults programmes (pp. 198203). His reflective description of one such programme in Tokyo (pp. 2035) is particularly useful, providing a balanced view of programme strengths and weaknesses. For this research, I adopted Robinsons definition of programme evaluation (2003, p. 199): the collection, analysis, and interpretation of information for forming judgments about the value of a particular programme. Among the aims of evaluation she describes are providing information on perceptions of a programmes value, measuring how far they meet their objectives, and giving feedback to course providers on necessary improvements. She adds that we need to build evaluation plans in to programmes.

1 Rationale for creating a new evaluation procedure

There seem to be very few detailed descriptions of how to conduct overall evaluations of FLT education programmes. Grosse (1991) notes a lack

262

FLT education programmes

of research in the area. I agree that teacher-training programmes must embody systems for internal programme evaluation and wish to design such a system. Their use increases the accountability of the programme to stakeholders (White, 1998; Lynch, 2003). Another important reason for conducting programme evaluation is to contribute to programme improvement (Candlin, 1998; Rea-Dickins and Germaine, 1998). I suggest there is a need for an evaluation procedure for FLT-training programmes, including a mechanism for obtaining and using feedback on whole programmes, not individual courses, from students, teachers and others. This would be a step towards the professionalization of the field of English-language teaching and make a useful contribution to theory. I also suggest that the new procedure will be of value in helping to answer Zhongs (1985, p. 61) question What constitutes adequate training of a foreign language teacher?, and that the procedure will facilitate and encourage the evaluation of other teacher-education programmes elsewhere. The model could be useful for other programmes.

2 The evaluation procedure

I created the procedure by reviewing literature on what others recommend regarding the content and procedures of FLT education programmes. It aims to collect appropriate data in order to assess programmes in terms of how far they match those recommendations. The procedure is: Review the literature and produce a set of questions. Establish appropriate sources of data in your setting. Choose and design data collection methods and instruments. Collect and analyse each set of data against your questions. Construct an account by relating each interpretation to the others.

3 Testing the procedure

I tested the system by evaluating the effectiveness of one TEFL programme, and then evaluated the system. The programme stakeholders in the University are the students and teachers on the BA in TEFL, those responsible for education in Hong Kong, the Government, and the University. Bartolomes sociocultural context here means EFL classrooms in Hong Kong, where English is regarded as a foreign, not second language. The questions in this case were as follows: Does the programme: 1) have a clearly stated philosophy? 2) reflect programme philosophy? 3) promote trainee flexibility in using different teaching approaches for different situations?

Matthew Peacock 263

4) promote the ability to use, and to adapt, foreign-language-teaching materials? 5) balance received versus experiential knowledge? 6) incorporate and encourage trainee reflection on the experiences and values they have when they enter the programme? In particular, does it encourage trainee reflection on their apprenticeship of observation? 7) promote the skill of reflection and self-evaluation as a teacher? 8) promote future reflective practice? 9) promote the long-term, developmental nature of learning to teach does it promote post-qualification teacher growth and development? 10) have good linkage among courses, avoiding overlaps? 11) Is the programme up-to-date? 12) balance teacher- and student-centred learning? 13) prepare EFL teachers to function in the sociocultural context in which they will work? 14) Do students believe the programme meets their needs, is relevant to their needs, and adequately prepares them for classroom teaching? 15) incorporate and balance linguistic, pedagogic, and managerial competence to an appropriate degree? Linguistic competence here means L2 proficiency. Pedagogic competence refers to teaching skills plus knowledge of language and second language acquisition.

III Research method

This section, section 4 Results and part of section 5 Discussion and Conclusions describe the evaluation of an existing programme. The research design included qualitative and quantitative research methods. Aksamit et al. stress qualitative methods (1990, p. 217; also see Ross, 2003; Kogan and Shea, 2007), and they suggest that evaluators should be subjective partners with stakeholders in reflecting on the programme (p. 219). I used multiple methods and triangulation of data, integrating the views of different participants at different stages; Wang (2007, pp. 101416), Rea-Dickins and Germaine (1992; 1998, p. 8), and Roberts (1998, p. 235) all stress the importance of this for programme evaluation. I gathered the views of various participants, e.g. students and teachers, about the whole programme, as Wallace (1991) and Weir and Roberts (1994) recommend. Baartman et al. stress in particular the importance of teacher opinions (2007, pp. 85860).

1 Subjects, school and programme

The subjects were 166 third-year students (all trainee teachers, 98% Hong Kong Chinese, 2% Westerners; average age 21; 20% male, 80% female), eight programme teachers, the coordinators of all 35 courses on the programme, and those programme alumni who responded to a posted questionnaire.

264

FLT education programmes

The students were all full-time undergraduates on the TEFL programme in the Department of English at the City University of Hong Kong, a three-year full-time preservice EFL teacher-training programme for local students who wish to teach English in secondary schools. The degree is an initial teaching qualification, though graduates need a postgraduate Diploma in Education for promotion, and enjoys a high degree of international recognition; it has a yearly intake of 38. Trainees complete a large number of courses e.g. The practice of language teaching and How languages are learned. Each course is three hours a week for 13 weeks. The minimum starting level of English for programme students is TOEFL 525/IELTS 5.5. Students do not take either test but enter on the results of local EFL tests. The programme has a procedure for regular external evaluation: an external examiner evaluates it biannually, though visits last only three days. The Programme Committee meets regularly but there is no system for evaluation of the whole programme, as opposed to individual courses. I suggest that the current mechanisms for internal evaluation of the whole programme are inadequate.

2 Aims of the project

The aims of this project arose out of Section II.1 Rationale for creating a new evaluation procedure: 1) Design a procedure for evaluating FLT training programmes based on existing principles of programme evaluation and models of teacher education, and answer Zhongs question What constitutes adequate training of a foreign-language teacher? 2) Test the new procedure through a global internal evaluation of one programme. Programme effectiveness was assessed regarding the extent to which it meets the needs of the students, i.e. preparing them to teach in the local context. 3) Evaluate the new procedure and assess its value for use in other contexts.

3 Data-collection instruments and procedure

This research took several years and used six data-collection methods. I collected and analysed most data; exceptions were help from trained assistants with student interviews, another programme teacher with evaluating course materials, and the programme leader who collected alumni questionnaires. a Student interviews (n = 101): The assistants conducted individual 1015 minute interviews of 101 third-year students, asking them (1) What are the

Matthew Peacock 265

strengths and weaknesses of the programme? (2) How can it be improved? Interviews lasted 1015 minutes, and the interviewers took notes. b Student questionnaires (n = 65): I collected questionnaires over several years from 65 third-year students. The questionnaire has 22 items (all may be seen in Table 1 below, section 4.2). It collects student opinions on a number of the above 15 questions. I constructed the questionnaire and piloted it before use. Questionnaires collected quantitative data and students answered on a fivepoint scale, strongly agree-agree-neither agree nor disagree-disagree-strongly disagree. Students did not write their names on the questionnaires. c Teacher interviews (n = 8): I interviewed eight randomly selected programme teachers to elicit their views and comments, showing them the list of 15 questions and asking them to give comments, particularly on programme strengths and weaknesses and how to improve the programme. Interviews lasted 20 minutes. d Student essays programme philosophy (n = 35): I asked 35 third-year TEFL students to write a 100150 word essay on the question Does the programme reflect programme philosophy? The aim was to collect opinions on question (2) above; it also gathers some information on questions (5), (12), (13) and (15). Students were given a copy of the philosophy (Appendix), and one month to write. e Evaluation of course materials: This was done in two stages, independently by me and another programme teacher: (1) we contacted the coordinators of all 35 courses on the programme and gathered full written descriptions of the goals, objectives, assignments, and assessment criteria. (2) We evaluated courses in terms of the balance between (i) linguistic competence/L2 proficiency, (ii) pedagogic competence, and (iii) managerial competence. Pedagogic competence refers to both teaching theory, and to teaching practices. Theory involves teaching skills plus the essential knowledge of language and language acquisition. Practices means teaching, planning for teaching, and reflecting on it afterwards. The aim was to gather data on question (15) above. f Alumni questionnaires: The programme leader designed and posted short questionnaires to all alumni, asking for suggestions on how to improve the programme and on problems they have teaching.

4 Data analysis
I searched the qualitative data from student interviews, teacher interviews and alumni questionnaires for useful information and extracted comments, and quantified data from student essays. I calculated percentages of

266

FLT education programmes

agreement/disagreement for all student questionnaire items and collapsed the data into three categories for clearer presentation, i.e. combining strongly agree/agree and disagree/strongly disagree. I checked correlations among questionnaire items using Pearsons r in the Statistical Package for the Social Sciences with statistical significance set at p < .01. As noted above, two programme teachers independently evaluated course materials and assessed the proportion for each area; after this we calculated percentages for each area and assessed inter-rater reliability by comparing the results from the both teachers.

IV Results
I completed the three aims of the project. In this section I present the results of the application of the new procedure.

1 Student interviews (n = 101)

The individual interviews of 101 final-year students were very useful. The data fell naturally into two categories, following the interview questions: strengths of the programme, and weaknesses of the programme with suggestions for improvement. I will use these categories to present the results. a Strengths of the programme The most common comment, made by a third of students, was that the programme is very practical in preparing them to become teachers. Typical comments were The programme is very professional in training English teachers and I have learned a lot of teaching skills. Two other top strengths, mentioned by almost a third of students, were that practice teaching is very useful and that teachers are very helpful and easy to contact. Students said Practice Teaching is a very good opportunity for us to apply the learned teaching theories in practice and I feel that many lecturers and tutors are very helpful ... we can easily contact them. Many students also praised the amount of theory on the programme e.g. It is good that the programme includes many teaching theories and I have learned many teaching theories and skills, while others said there is a good balance between theory and practice. Other common comments were that the programme improved their English, that it is professional, that grammar and phonetics are very useful courses, that the small tutorial groups allow them to receive more attention, and that the programme has been carefully designed. b Weaknesses of the programme, and suggestions for improvement Students highlighted a number of weaknesses and made helpful suggestions for improvement. No less than half the students said the programme should include a postgraduate diploma in education, and more practice teaching. Typical comments were It would be better if the programme could be

Matthew Peacock 267

changed from three years to four years like Hong Kong University which includes the educational diploma and It would be better if the period of practice teaching could be extended [we need this to learn how to teach, to get a job]. Another common suggestion, made by nearly half the students was to increase time for English proficiency development e.g. I think it would be better if more English enhancement courses could be added into the programme to consolidate our English proficiency. Many students found the Year 3 assignment load too heavy and pace too fast. A typical comment was Since some courses in year 3 are very heavy, such as Practice Teaching and the Final Year Project, it allows very little time for us to focus on these courses and the assignments. Other common suggestions were to add educational technology/I.T. courses, that input on classroom management was insufficient and should be taught before we go out to do practice teaching, to provide a workroom/computer room for students, and to let them do the Final Project in pairs.

2 Student questionnaires (n = 65)

I computed the internal consistency reliability of the third-year student questionnaire at.8687 using Cronbach alpha based on the entire sample. Results are below in Table 1: Items 8, 11, 7 and 9 attracted the highest levels of student agreement: 71% of students agreed with 8 The TEFL programme encouraged me to be a reflective teacher (when I start teaching), 60% with 11 taught me how to teach English, 58% with 7 and 9, encouraged me to reflect on my past experiences as a language learner and promotes flexibility in using different teaching practices for different situations, and 55% with 12 taught me how to evaluate myself as a teacher. Items 19, 3, 5, 13, 18, 21, 15, 14, and 2 attracted the lowest levels of student agreement: only 13% of students agreed with 19 The TEFL programme has a good balance between the teaching of: English, teaching skills, and classroom management skills; 29% with 3, 5, and 13 gave me adequate training in English, gave me adequate training for the needs of the local context (teaching in Hong Kong schools), and taught me classroom management skills; 32% with 18 and 21 is relevant to my needs and met my needs; and 33%/36% respectively with 15 and 14, taught me how to adapt, and use, foreign-language-teaching materials. Finally, item 2 shows that only 35% agreed that the programme avoids overlapping information between different courses. I found modest correlations between some questionnaire items: 0.55 between items 3 and 11, 0.55 (items 4 and 13), 0.48 (11 and 13), 0.50 (11 and 17), 0.41 (12 and 18), 0.41 (11 and 19), 0.50 (18 and 19), 0.49 (17 and 21), and 0.44 (19 and 21). Modest refers to relationship strength. Below 0.2 is very low, 0.20.39 low, 0.40.69 modest, 0.70.89 high, 0.90.99 very high (Bryman and Cramer, 1990, pp. 168, 171). All correlations were statistically significant

268

FLT education programmes

Table 1 Student (n = 65) questionnaire results Number The TEFL programme Agree or strongly Agree 42 35 29 52 29 48 58 71 58 Neither agree nor disagree 32 29 26 22 39 39 32 23 19 Disagree or strongly disagree 26 36 45 26 32 13 10 6 23

1 2 3 4 5

6 7

8 9

10

11 12 13 14 15 16 17 18 19

20 21 22

has good linkage between different courses. avoids overlapping information between different courses. gave me adequate training in English. gave me adequate training in teaching skills. gave me adequate training for the needs of the local context (teaching in Hong Kong schools). is up-to-date. encouraged me to reflect on my past experiences as a language learner. encouraged me to be a reflective teacher (when I start teaching). promotes flexibility in using different teaching practices for different situations. balances teacher-centred and student-centred learning on its courses. taught me how to teach English. taught me how to evaluate myself as a teacher. taught me classroom management skills. taught me how to use foreign language teaching materials. taught me how to adapt foreign language teaching materials. increased my powers of self-evaluation. taught me foreign language testing and evaluation skills. is relevant to my needs. has a good balance between the teaching of: English,teaching skills, and classroom management skills. prepared me to teach English in the classroom. met my needs. By the end of the TEFL Programme, I will be ready to teach English.

39 60 55 29 36 33 51 49 32 13

42 33 32 26 45 47 39 32 52 29

19 7 13 45 19 20 10 19 16 58

45 32 35

42 39 42

13 29 23

Matthew Peacock 269

(p < .01), indicating they did not happen by chance. I suggest that they reveal student opinions about FLT education and being a teacher, and interpret them as follows: items 4, 11, 18 and 21 are all concerned with student opinions on whether their needs were met concerning their training to be English teachers. These items correlated with items 3, 12, 13, and 17. This indicates that students believe that important parts of learning how to teach English are receiving adequate training in the English language, learning how to evaluate themselves as teachers, learning classroom management, and learning foreign-language testing skills. The correlations between items 11, 18 and 21 and item 19 are a student comment on the programme: they indicate that students believe that it should have a good balance between the teaching of English, teaching skills, and classroom management skills.

3 Teacher interviews (n = 8)
Teacher interviews were valuable and provided a different perspective on the programme. Regarding programme strengths, several teachers said balance is a strength. Comments included: Nice balance between language proficiency, the nature of language, and methodology we give our students exactly what they need and Nice balance of teaching theories and teaching experience. Several also said: We have a lot of emphasis [on] reflection raising awareness in students of issues of learning and teaching and Very strong on pedagogical knowledge. Others said that the programme promotes the spirit of being a teacher, meaning that it promotes the fundamental essence or real meaning of being a teacher. Three teachers responded to question 9 and said the programme does promote the long-term, developmental nature of learning to teach, promote post-qualification teacher growth and development, and that it is careeroriented. Two teachers responded to question 13 and said that the programme does well in preparing students to function in the local context in which they will work, and that it is sensitive to that context, noting our close links with local schools. Other comments were that the programme is well-organized, fosters creativity, and that staffstudent relationships are good. Regarding weaknesses, several teachers said there is too little practice teaching, e.g. Practice teaching, gaining experience in schools, is limited. Five weeks is insufficient. Others said that more knowledge of linguistics and language awareness was needed, e.g. We focus too much on the teaching side not the language side though as they said pedagogical knowledge is strong, teaching side must refer to knowledge of language and second language acquisition. Teachers also said We need more on language awareness, especially about the wider sociological setting. Three teachers said student workload is unbalanced, with too much at the end of semesters. Two said staff do not know enough about the Hong Kong education system and are sometimes out of touch with developments there, and that there is not

270

FLT education programmes

enough focus on being a member of a school community. They added that the programme needs to promote the culture of teaching and what it means to be a teacher.

4 Student essays: programme philosophy (n = 35)

The programme has a philosophy (Appendix 1), which is given in the student handbook. Essay analysis shows that 77% of students think the balance is poor among the three sources of subject matter competence, skills and awareness on the programme. 64% said it needs more on awareness, meaning understanding learning and teaching and the sociocultural context, because they need this once they start teaching. 32% said the programme needs more on subject matter competence or knowledge of and proficiency in English. 38% said it needs more on skills (teaching methods), and another 21% said courses in this area are too theoretical and/or impractical. 86% of students said the programme does engage students as active participants in learning. However, 66% of students said lecturers do not use a variety of teaching modes to stimulate real learning. Most said the main mode was lecture plus handouts, and 14% said there was little or no student-directed learning. Only 20% said the programme uses a variety of teaching modes.

5 Evaluation of course materials

The two raters had very similar results: we assessed inter-rater reliability at above 0.9, an acceptable figure. Evaluation revealed the following: 1) Linguistic competence makes up 18% of the programme, mostly in year 1. It makes up 42% of the year. 2) Pedagogic competence makes up 60%. Teaching theory makes up 36%, and teaching practices, 24% (actual teaching, 5%; planning for that teaching, 12%; and reflecting on it afterwards, 7%). Teaching practices are covered mostly in the last two years. 3) Managerial competence makes up 1% of the programme. 4) I will address the question of what an appropriate balance is among these three areas below, in Discussion and conclusions.

6 Alumni questionnaires
This part of the research was not successful. The questionnaire return rate was only 20%, despite further efforts to contact alumni. Few respondents wrote comments on the questionnaire; most only circled a number on the job satisfaction scale. The teachers rated their job satisfaction at a mean of 4.8 on a scale of 1 very unsatisfactory to 7 very satisfactory. Their suggestions, ranked commonest first, for improving the programme were to

Matthew Peacock 271

add components on improving student motivation, classroom management, counselling students, information technology, teaching grammar and teaching English literature.

V Discussion and conclusions

Student interviews, teacher interviews, student questionnaires, student essays and the evaluation of course materials all provided valuable data. They allow us to evaluate the programme from multiple perspectives, and try to answer the 15 questions from Section II.3. I will first discuss the case study results and their implications, and then, more importantly, comment on and assess the new evaluation procedure.

1 The case study: assessing the programme

The programme does have a clearly stated philosophy (Wallace, 1991, questions 1 and 2). However, programme content does not reflect programme philosophy well. Student essays indicate an imbalance among the three areas of subject matter competence, skills and awareness. Most students said the programme needs more on awareness (understanding learning and teaching and the sociocultural context). Teacher interview comments back up this finding, e.g. We need more on language awareness, especially about the wider sociological setting, as do student answers on questionnaire item 5. These comments are somewhat surprising, considering students apprenticeship of observation, and I therefore conclude that the comments refer to teaching in the sociocultural context, not learning in it (i.e. being part of it as a learner). A majority of students also wrote that lecturers do not use a variety of teaching modes to stimulate real learning, as the philosophy claims, and answers on questionnaire item 10 support this. The programme does appear to promote trainee flexibility adequately in using different teaching approaches for different situations (Crandall, 1993, question 3). Student and teacher interview responses, e.g. very professional very strong, indicate satisfaction with the teaching skills components of the programme, as do the 58% and 60% student agreement with questionnaire items 9 and 11. However, the programme does poorly in promoting the ability to use and adapt FLT materials (Wallace, 1991, question 4): only about a third of students agreed with questionnaire items 14 and 15. Teacher interview comments indicate that the programme successfully promotes teacher reflection and self-evaluation, and incorporates experiential knowledge to a satisfactory degree (Wallace, 1991; Grosse, 1991; Liou, 2001; Bartolome, 1994; Lynch, 2003 and others, questions 5, 6, 7 and 8), as do levels of student agreement with questionnaire items 7, 8 and 12. Teachers also said it promotes the long-term, developmental nature of learning to teach and teacher development (Stoynoff, 1999; Lo, 2005, question 9). On the other

272

FLT education programmes

hand, the level of student agreement (42% and 35%) with questionnaire items 1 and 2 suggest the programme does not do well in linking courses and avoiding overlaps (Weir and Roberts, 1994, question 10). And less than half of students think the programme is up-to-date (Bartels, 2005; Wallace, 1991, question 11, questionnaire item 6). Student essays suggest that the programme does not balance teacher-centred and student-centred learning (Bartels, 2005; Wallace, 1991, question 12), and this is backed up by only 39% of students agreeing with questionnaire item 10. In their essays most students implied it is weak in preparing students to function in the sociocultural context in which they will work (Bartolome, 1994, question 13). Teacher interview data, and only 29% of students agreeing with questionnaire item 5, both corroborate this. Function here means teach in EFL classrooms. The question of whether the programme meets and is relevant to student needs, and adequately prepares them for teaching (Wallace, 1991; Reid, 1996, question 14), is fairly broad. Student interview data indicate it does not adequately meet their needs in three areas: gaining a postgraduate diploma in education; sufficient practice teaching; and consolidating English proficiency. Teachers agreed with the second point during interviews. Additionally, only 32% of students felt the programme is relevant to and met their needs (questionnaire items 18 and 21). However, 52% were not sure whether it is relevant, leading us to speculate that students may not be aware of what their needs will be when they start teaching. Regarding the question of whether the programme adequately prepares them for teaching, students seem satisfied with certain teacher preparation aspects. During interviews students said its top strength is its practicality in preparing them to teach; teachers also named the teaching of pedagogic knowledge as a strength. This is backed up by student agreement with questionnaire items 4 adequate training in teaching skills and 11 taught me how to teach English, which at 52% and 60% respectively was higher than for most items. On the other hand, only 35% of students agreed with student questionnaire item 22, indicating that they would not feel ready to teach English by the end of the programme; yet this conflicts to some extent with question 11. It is difficult to know why only 35% of students agreed with this item, but I speculate that this was for two reasons: students do not yet feel ready to teach in the relevant sociocultural context, and year 3 students lack confidence in their teaching abilities. The modest correlations I found between certain questionnaire items are also relevant to the question of whether the programme meets and is relevant to student needs, and adequately prepares them for classroom teaching. They appear to reveal what students believe some of their needs to be, i.e. receiving adequate training in English, learning how to evaluate themselves as teachers and learning classroom management and foreign-language testing skills. Regarding an appropriate balance among linguistic, pedagogic and managerial competence (Wedell, 1992, question 15), I conclude that the

Matthew Peacock 273

programme incorporates but does not balance these three areas. It is difficult to say what an appropriate balance is, and to some extent I have to rely on participants opinions. Students certainly perceive a misbalance; only 13% of students agreed that it balances these three (questionnaire item 19). I suggest that the stronger focus on language skills in year 1, and increase in teaching practices by year 3, are desirable features of the programme. However, only 24% of the programme engages students in actual teaching practices, and only 5% in actual teaching. Half the students we interviewed said this is not enough, and a weakness; interviewed teachers agreed. Regarding pedagogic competence, I conclude that the programme is stronger in promoting teaching skills than the vital knowledge of language and second language acquisition. Additionally, students said the programme needs more on English proficiency. Almost half the students we interviewed said they wanted more time here; and student essays, where 32% said they wanted more, and questionnaire item 3, back this up to some extent. Only 29% of students feel it has enough. On the other hand, I note that teachers in interviews said that there was an appropriate balance among English proficiency, the nature of language, and teaching methods, and I speculate that many other experts would feel that 42% of year 1 is sufficient. Regarding managerial competence, I suggest that few experts would consider that 1% is appropriate. Student interviewees and alumni mentioned lack of input on classroom management skills as a programme weakness; and on questionnaire item 13, only 29% of students agree the programme taught them this. Lack of management skills may be a problem. Richards (1990, p. 11) stresses that discipline problems are fewer, motivation higher and learning more efficient in a well-managed class and, as I said above, many experts emphasize its importance. Other weaknesses emerged: not enough on testing or educational technology; the heavy and unbalanced workload (particularly in year 3); and teaching methods courses being too theoretical and/or impractical. Other strengths were teachers being very helpful and easy to contact, and the programme having plenty of theory and a good theorypractice balance. It is possible to use these results to make suggestions for programme improvement, and they also have implications for EFL teacher training. Robinson (2003, p. 199) called on evaluators to form judgments about a programmes value and how far it meets its objectives. Regarding the programmes value for preservice EFL teacher training, I note that the latest external examiners report called it a high-quality professional preparation programme of high academic standard that stands comparison with similar programmes elsewhere. While I do not disagree with the external examiner, this evaluation revealed certain areas where improvements are possible and desirable. The programme appears strong in teaching certain aspects of pedagogic skills (including flexibility in using different teaching approaches for different situations) and promoting teacher reflection and self-evaluation.

274

FLT education programmes

Staffstudent relationships are good, and the programme also seems sound at promoting the long-term, developmental nature of learning to teach and postqualification teacher development, and in having sufficient theory and a good theorypractice balance. However, I suggest it needs strengthening in various areas: more input on classroom management and the ability to use/adapt teaching materials; more teaching practice; better linkage among courses (avoiding overlaps); and more instruction regarding teaching within the sociocultural context (strengthening this may give students more confidence in their teaching ability) and in the important area of language and second language acquisition. It also needs to balance the workload more carefully. Regarding the philosophy, the programme should adhere to it; I also question whether it is adequate. It could be strengthened, as it talks more about programme content than philosophy (beliefs and values). Finally, the programme does not have a built-in procedure for internal evaluation of the whole programme, as Robinson (2003) recommends. We can add one. In addition to these areas, students say the programme is not up-to-date and does not balance teacher- and student-centred learning, and that it needs more input on English proficiency, foreign-language testing and educational technology. However, I cannot confirm the need for more in these areas. That needs further evaluation. Several writers (e.g. Robinson, 2003, p. 210) note that an important problem with programme evaluation is getting the information across to decisionmakers, who can then act on it. Detailed evaluation results must be sent in writing to programme committees. However, a serious and difficult problem for programme planners may well be that adding components involves adding student hours and de-emphasizing or removing other components. We must communicate these evaluation results to other stakeholders: students, teachers, the Government and departmental and university planners and administrators. Programme committees must set up a clear mechanism for doing this; methods can include reports and presentations.

2 Assessing and improving the programme evaluation procedure

While this procedure evaluated a programme in one context, I suggest others can adapt it for other contexts. To assist this I will generalize out from the specifics of the case study and assess the validity, reliability, strengths and weaknesses of the procedure. I will discuss how well the data-collection instruments worked and what I recommend others to do. I do not propose that this evaluation procedure is valid for all contexts. Clearly, informants and situations differ. However, as I said above, I suggest it will facilitate and encourage the evaluation of other teacher-education programmes. I also suggest the procedure is useful in that it has a comprehensive list of 15 questions that arose from relevant literature, and collects a lot of valuable information from various programme stakeholders. Other key factors in the procedure are collecting data from participants in sufficient numbers,

Matthew Peacock 275

e.g. entire student cohorts, teachers and alumni, and collecting a variety of data, e.g. from student interviews, teacher interviews, alumni questionnaires, student essays and course materials. I propose that the student and teacher interviews are very valuable. I concluded that one helpful feature was that participants were not constrained by overstructured questions, but asked to comment on programme strengths and weaknesses. This, particularly for students, allowed them to reflect and comment on areas of particular and practical importance to them, e.g. the lack of a PGD in education, practice teaching, and work on English proficiency. The additional step with teachers of showing the list of 15 questions also helped them provide a more comprehensive analysis, because it focused their thoughts on a wide range of relevant areas. I found the student questionnaire to be of value and it provided a lot of useful data. The internal consistency reliability of the questionnaire was an acceptable 0.87. However, possible differences between student and teacher understanding of some concepts (e.g. adequate in questions 3, 4, and 5, good in question 19, and reflect/reflective in questions 7 and 8) slightly restrict the value of the questionnaire. Also, item 5 could be clearer; items 11, 18, 20 and 21 are rather broad and generalized; and students may not be fully competent to answer on item 6, and possibly item 19. The student essays were also valuable, but a programme teacher collecting them could possibly have constrained students. The evaluation of course materials provided crucial information and inter-rater reliability was above 0.9, an acceptable figure. The alumni questionnaires, on the other hand, were not at all successful. The return rate was 20%, normal for postal questionnaires. This was probably because of changing addresses and lack of interest, but more efforts to contact alumni would help. A weakness in this research was not contacting employers of programme graduates, which was beyond the scope of this study. However, this missed step is an important one for future improvement of the process. Other suggestions for improving further research (not carried out during this project, some because of lack of time and access to classrooms, others because they emerged only on later reflection) are: to collect questionnaires from more students; to give similar questionnaires to teachers and compare student and teacher results; to interview a larger number of teachers; to conduct lesson observations focusing on the content and process of lessons; and to collect more data from alumni, e.g. by school visits, extensive interviews, essays and a better questionnaire.

Finally, checking course readings would help answer the question of whether the programme is up-to-date.

276

FLT education programmes

This article provides an account of the design and test of a procedure for evaluating FLT education programmes, assesses its strengths and weaknesses, and makes suggestions for improvement. I believe that all teacher-education programmes should incorporate a built-in procedure for overall internal evaluation, and use it regularly. I hope that the new procedure will facilitate and encourage this. Critical review of programmes is a beneficial and essential exercise.

VI References
Aksamit, D.L., Hall, S.P. and Ryan, L. (1990). Naturalistic inquiry applied to the evaluation of a teacher education program. Teaching and Teacher Education, 6(3), 21526. Baartman, L.K.J., Bastiaens, T.J., Kirschner, P.A. and Van der Vleuten, C.P.M. (2007). Teachers opinions on quality criteria for Competency Assessment Programs. Teaching and Teacher Education, 23(6), 85767. Bartels, N. (2005). Applied linguistics and language teacher education: What we know. In Bartels, N. (Ed.), Applied linguistics and language teacher education (pp. 40524). New York: Springer. Bartolome, L.I. (1994). Beyond the methods fetish: toward a humanizing pedagogy. Harvard Educational Review, 64(2), 17394. Bryman, A. and Cramer, D. (1990). Quantitative data analysis for social sciences. London: Routledge. Candlin, C.N. (1998). General editors preface. In Rea-Dickins, P. and Germaine, K.P. (Eds.), Managing evaluation and innovation in language teaching: building bridges (pp. xivxxi). London: Longman. Cots, J.M. and Arno, E. (2005). Integrating language teachers discipline knowledge in a language course. In Bartels, N. (Ed.), Applied linguistics and language teacher education (pp. 5978). New York: Springer. Crandall, J. (1993). Professionalism and professionalization of adult ESL literacy. TESOL Quarterly, 27(3), 497515. Freeman, D. (1996). The unstudied problem: research on teacher learning in language teaching. In Freeman, D. and Richards, J.C. (Eds.), Teacher learning in language teaching (pp. 35178). Cambridge: Cambridge University Press. Freeman, D. and Johnson, K. (1998). Reconceptualizing the knowledge-base of language teacher education. TESOL Quarterly, 38(1), 397417. Freeman, D. and Johnson, K.E. (2004). Readers react Common misconceptions about the quiet revolution. TESOL Quarterly, 38(1), 11927. Grosse, C.U. (1991). The TESOL methods course. TESOL Quarterly, 25(1), 2949. Johnson, K.E. (2006). The sociocultural turn and its challenges for second language teacher education. TESOL Quarterly, 40(1), 23557. Kogan, J.R. and Shea, J.A. (2007). Course evaluation in medical education. Teaching and Teacher Education, 23(3), 25164. Liou, H.-C. (2001). Reflective practice in a pre-service teacher education program for high school English teachers in Taiwan, ROC. System, 29(2), 197208.

Matthew Peacock 277

Lo, Y.-H.G. (2005). Relevance of knowledge of second language acquisition. In Bartels, N. (Ed.), Applied linguistics and language teacher education (pp. 13557). New York: Springer. Lortie, D. (1975). Schoolteacher: a sociological study. Chicago, IL: University of Chicago Press. Lynch, B.K. (2003). Language assessment and programme evaluation. Edinburgh: Edinburgh University Press. Muchisky, D. and Yates, R. (2004). The authors respond Defending the discipline, field, and profession. TESOL Quarterly, 38(1), 13440. Rea-Dickins, P. and Germaine, K. (1992). Evaluation. Oxford: Oxford University Press. Rea-Dickins, P. and Germaine, K.P. (1998). The price of everything and the value of nothing: trends in language programme evaluation. In Rea-Dickins, P. and Germaine, K.P. (Eds.), Managing evaluation and innovation in language teaching: building bridges (pp. 319). London: Longman. Reid, J.M. (1996). Lets put the T back in TESL/TEFL programs. TESOL Matters, 5(6), 3. Richards, J.C. (1990). The dilemma of teacher education in second language teaching. In Richards, J.C. and Nunan, D. (Eds.), Second language teacher education (pp. 315). Cambridge: Cambridge University Press. Roberts, J. (1998). Language teacher education. London: Edward Arnold. Robinson, B. (2003). Evaluation, research and quality. In Robinson, B. and Latchem, C. (Eds.), Teacher education through open and distance learning (pp. 193211). London: Routledge/Farmer. Ross, S.J. (2003). A diachronic coherence model for language program evaluation. Language Learning, 53(1), 133. Stoynoff, S. (1999). The TESOL practicum: An integrated model in the US. TESOL Quarterly, 33(1), 14551. Wallace, M.J. (1991). Training foreign language teachers. Cambridge: Cambridge University Press. Wang, W. (2007). Evaluation of 2 + 2 alternative teacher performance appraisal program in Shanxi, Peoples Republic of China. Teaching and Teacher Education, 23(7), 101223. Wedell, M. (1992). Pre/in service training of ELT teacher trainers: Planning the regional MATEFL project. In Flowerde, J., Brock, M. and Hsia, S. (Eds.), Perspectives on second language teacher education (pp. 33750). Hong Kong: City University of Hong Kong. Weir, C. and Roberts, J. (1994). Evaluation in ELT. Oxford: Blackwell. White, R. (1998). What is quality in English language teacher education? English Language Teaching Journal, 52(2), 13339. Wu, X. (1996). Train ourselves first? The Teacher Trainer, 10(3), 2022. Yates, R. and Muchisky, D. (2003). On reconceptualizing teacher education. TESOL Quarterly, 37(1), 13547. Zhong, Y. (1985). Towards adequate E.F.L. teacher education in China: an analysis of the problem. Australian Review of Applied Linguistics, 8(1), 6169.

278

FLT education programmes

Appendix 1: Programme philosophy (Student handbook)

The teaching of English as a second language draws on three sources: subject matter competence, skills and awareness [emphasis in the original]. Subject matter competence relates to knowledge and proficiency in the subject of teaching, i.e. the English language. Skills relates to methodologies of teaching and course development as well as basic competence in the classroom. Awareness relates to an understanding of learning and teaching processes and of the sociocultural contexts in which learning and teaching take place. The programme seeks to engage students in the learning process as active participants in learning. Lecturers use a variety of teaching modes to stimulate real learning. Students often work in pairs or groups on class projects, and student-directed learning forms the core of many activities.