Program Evaluation FRIT 8435 Fall 2011

Group 8 Alexis Scranton Laura Houston

Pilot program used for instructing teachers on the methodology to instruct English Language Learners

Public School in Johor Bahru, Malaysia

Executive Summary

The purpose of this evaluation is to describe the effectiveness of a pilot program used for instructing teachers on the methodology to instruct English Language Learners. The program is implemented by the Ministry of Education in a school in Malaysia.

Introduction A new pilot program has been introduced in the Malaysian primary public schools by the Ministry of Education aiming to enhance the level of English instruction nation-wide. The purpose of the evaluation is to determine the effects the components of the program are having on the teachers instruction of English. Based on teachers observations and reviews of lesson plans, this report will allow stakeholders to determine the effectiveness of the program thus far and provide suggestions to consider for improvement. There are many stakeholders involved in this project, from the Prime Minister himself down to the teachers, students and the parents of students affected by this program. This report, however, being that the program is in developing stages, is intended for key stakeholders. Included amongst these stakeholders are the employees in the Ministry of Education, the district level English language officers and the project manager responsible for overseeing the foreign, native English speaking teacher mentors. Due to the overwhelming span of program and the lack of resources and access to participants nation-wide for the evaluation, this report will focus on schools from one district. It is not to be assumed that this sample of participants is necessarily representative of all participants and conditions nationwide. It is a good baseline to see if positive changes are being made and to gather feedback from

participants. Seeing how each district is unique in how they manage certain aspects of the schools and their curriculum, it is reasonable to do this report on a district-level basis. Changes found to be necessary should be made according to the study done for that district and should not be imposed on participants in other districts and in other states, as the necessary changes recommended may not apply elsewhere. Taking this into consideration, the report is useful, however, as an outline that may be used in other locations to conduct a similar evaluation. Found in this report are the following contents; I. Focus of the Evaluation Includes a description of the evaluation object utilized, the questions created for the study and required information necessary in order to complete the study. II. Brief Overview of the Evaluation Plan & Procedures III. Presentation of Evaluation Results Includes a summary and interpretation of the findings IV. Conclusions and Recommendations Includes the criteria and standards used to judge the evaluation object, strengths and weaknesses discovered about the program along with suggested recommendations for improvement. V. Appendix

Focus of the Evaluation The multi-faceted programs main aim is to enhance the English language teaching and learning in the country to help aid in the national vision of gaining developed country status. The Teaching English Language and Literacy (TELL) project has rolled out across the nation in attempts to enhance the language and literacy instruction implemented by the local English teachers, in hopes of sustainability once the project comes to an end. Currently in the first year of the three year pilot phase, this

evaluation serves primarily as a formative evaluation of the entire contract, but may be summative in terms of the first year. Those chosen to participate in the pilot program attend professional development sessions twice a month for two hours each on new and current teaching methodologies and are visited by teacher mentors at least once every two weeks. These visitations may include observations, co-teaching experiences, or lesson-planning and materials creation. The hope is to introduce more new age, student-centered, engaging, and communicative approaches to language learning to the teachers and have them to implement these instructional techniques with their students Each school district involved across the nations has a cluster of five 15 participating schools divided into 3 sub-clusters of 5 schools each. Each sub-cluster has a maximum of 25 teachers involved, with one teacher mentor per sub-cluster. Mentors are responsible for visiting each of their teachers in the actual school setting. Then, on varying days, all three mentors will come together to present an afternoon course to teachers, one sub-cluster at a time. The English Language Officers, employed at the district office, conduct observations of all English teachers in the entire district as a major responsibility of their position. They are the local point of contact for those directly involved in this project. Project managers oversee the teacher mentors and have a direct line of contact with the staff in the Ministry of Education. Though there are several key components included in the overall goal of this T.E.L.L. program, this particular evaluation was conducted to address the priority question; Does the program enhance the teachers knowledge and use of strategies that encourage communicative approaches to learning, as introduced during training and coaching sessions? This question was assessed based on the following criteria: Teachers provide students with opportunities to practice the English language in authentic tasks and activities on a regular basis. Teachers consistently incorporate a variety of communicative activities into their teaching plans

and instruction. These criteria were evaluated by the following standards: Teachers attend 80% of training sessions provided. Students are provided with opportunities to practice their language skills with classroom peers multiple times per week. A minimum of 2-3 lessons per week must incorporate this component. In order to carry out an evaluation based on these criteria and standards, evaluators had to have an attendance record kept of all professional development sessions for the year. Regular observations had to be done, with record keeping. These notes would be obtained by both the teacher trainers and the English Language Officers. The language officers have the ability to provide information on the teaching abilities and techniques of those teachers not enrolled in this program, as the comparison group. They are also key in helping to obtain lesson plans of participating teachers from previous year(s) before the onset of this program. Brief Overview of Evaluation Plan and Procedures Comparison group and cross-sectional designs were both used in this examination. The comparison groups were comprised of students from the same school and grade level, with the difference being those taught by teachers involved in this pilot program and those who were not. Observations of teaching techniques, in addition to collection of lesson plans were the modes of information collection. Cross-sectional methods were used via surveys and interviews, conducted only with those involved in the program. Surveys were completed by all 75 participating teachers, while interviews were conducted with only one fourth of them. Teacher trainers and district-level English language officers all partook in the observations of teachers as well as the analysis of their submitted lesson plans. Each teacher was seen by a mentor or language officer at least once every two weeks for a minimum of a one hour class. These observations took place throughout the entire school year. Surveys and interviews were distributed and conducted over the course of the last two months of the program. The information collected from surveys and interviews were used as part of the analysis

process. In addition, any change in the improvement of teachers teaching methodologies and resources, as well as the documentation of such, submitted as formal lesson plans, were all also included in this task. The frequency of communicative language approaches carried out in the classroom was an ideal three times per week minimum. Teachers were expected to attend and participate in 80% of the professional development sessions required by the program.

Presentation of Evaluation Results Many conclusions were made from the data collected from the English language learners program offered by the Ministry of Education in Malaysia. Conclusions were made from surveys, interviews, observations, and teacher lesson plans. All teacher participants within the program participated in the survey. Eighty-seven percent of teachers surveyed, said that the program and training helped them learn new strategies in teacher English language learners. The thirteen percent of teachers that said the program was not beneficial stated that they had already been using the same strategies that were displayed in the trainings within the program. In surveys teachers were also asked to rate the trainings offered by the program. Teachers were given the following ratings: Above Average, Average, Below Average. Sixty percent of teachers said that the training was above average. Forty percent stated that the training was average. Teachers were able to leave comments. One of the comments stated, More hands on activities to show us how to use and implement the strategies would be helpful for future trainings. Trainers of the program, also called Mentors and Coaches, conducted observations of teacher lessons twice per month. The data collected from the trainers showed that sixty percent of the time, teachers were incorporating techniques from the trainings to instruct students. The trainers also observed student opportunities during their visits into classroom. Teachers were to provide students with opportunities to practice their language skills with classroom peers with a minimum of 2 to 3 times per week. This was measured by teacher lesson plans. One hundred percent of teachers had the

minimum of 2 to 3 strategic lessons in their lesson plan books. The evaluation plan results indicate that teachers are gaining useful strategies and techniques in teacher English language learners. Teachers are using the strategies within their lesson plans at least 2 to 3 times a week. When observed, most teachers were using the techniques provided in trainings and giving students opportunities to practice language skills with their peers. Teachers that were surveyed agreed that the program trainings were effective.

Conclusions and Recommendations Based on the results from the evaluation relating to enhancing teachers' knowledge and use of strategies, this plan concludes the following: Program Strengths: All teachers within the program received trainings to learn new strategies. 100% of teachers included the strategies in their lesson plan books. All teachers were evaluated and observed by Coaches. 87% of teachers surveyed, said that the program and training helped them learn new strategies in teacher English language learners. Observations concluded that 60% of teachers were implementing strategies and techniques from trainings. 60% of teachers agreed that the training sessions were above average. Program Weaknesses: The 13% of teachers that said the program was not beneficial stated that they had already been using the same strategies that were displayed in the trainings within the program. Teachers comments on the program training stated, More hands on activities to show us how to use and implement the strategies would be helpful for future trainings. The data collected from the trainers showed that 60% of the time, teachers were incorporating techniques from the trainings to instruct students. This indicates that 40%

of the time teachers were not implementing the strategies and techniques from the trainings. Recommendations: Appoint a full-time program supervisor for each school. Provide mandatory hands on trainings for teachers to learn how to implement effective strategies and techniques to enhance student learning with English language learners. Provide optional training for teachers that are already using the strategies and techniques in the classroom. These trainings should include other optional or advanced techniques to use in the classroom. Use advanced teachers to mentor and help teachers that are struggling with using information from the trainings. Incorporate tiers of trainings: beginner, intermediate, and advanced

Following these recommendations can help make the program from the Ministry of Education in Malaysia more successful. Data from the evaluation plan suggests that the program trainings are helping teachers become more successful at teaching English language learners. The data indicates that teachers may need more hands on training on how to implement strategies into every day lessons. As with any new program or initiative, it has room for improvement. Overall the program is showing signs of success in training teachers to implement strategies and techniques to effectively educate English language learners.