Fall

08

An Evaluation Plan for an iPad Entrepreneur Programming Class

Shane Wheeler
EDTech 505 Section 4172 Professor Perkin August 1st, 2011

Table of Contents
LEARNING REFLECTION EXECUTIVE SUMMARY PURPOSES OF EVALUATION QUESTIONS
TO

3 4 5 5 5 6 6 6

ADDRESS

STAKEHOLDERS BACKGROUND INFORMATION ORIGINS PRIOR PROGRAMS PEOPLE INVOLVED CHARACTERISTICS GOALS OBJECTIVE 1 OBJECTIVE 2 OBJECTIVE 3 OBJECTIVE 4 DESCRIPTION OF EVALUATION DESIGN CONCLUSIONS SOURCES APPENDIX A: EVALUATOR PROGRAM DESCRIPTION APPENDIX B: EVALUATOR PROGRAM TIMELINE APPENDIX C: BEGINNING OF THE YEAR STUDENT SURVEY APPENDIX D: END OF THE YEAR STUDENT SURVEY APPENDIX E: HARMONY STUDENT PORTFOLIO RUBRIC APPENDIX F: INTERVIEW QUESTIONS FOR TEACHER APPENDIX G: INTERVIEW QUESTIONS FOR STUDENTS
IN THE

PROGRAM

6 7 7 7 7 8 8 8 9 10 11 12 14 17 19 20 21

Learning Reflection
EDTech 505 has taught me a great many things about evaluation. Evaluation is used to make decisions, compare items objectively, refine processes, or eliminate processes. Evaluation is not research and research is not evaluation. Research is stricter in its application. The parameters that need to be set up for research concern keeping the outside world from affecting variables in the program, where evaluation is not so much concerned with outside influence. Evaluation is more concerned with the key questions and decisions regarding the program. Evaluation is more practical in the real world application of improving education at the level I teach at because conducting research at my level is not practical for a working educator. Im not discounting research, because it is vital and not nearly read or debated enough by educators. Im actually surprised by how much we try to change our systems all the time, but there is little real debate on the research or evaluations about research. I have learned the proper steps for conducting an evaluation, but it seems that every evaluation is going to be unique given that each evaluation has its own unique goals, processes, funding, and variables. I found reading the different evaluations interesting, because they read so differently. While there is a set process, the authors voice is present in the material. The last thing that I hope to take away from the process of taking EDTech 505 is a good, solid core evaluation process for my grant program. It is going to probably be more than what the Iowa School Board Foundation is going to expect, but I personally am more interested in evaluating the success and failures of the program. I look forward to the feedback I will receive of my evaluation setup, but Im also nervous because my project will be different from everyone elses. Having worked on this final project, I have come to realize that I should have sought more clarity on how to change the final project (totally my fault). I realized too late that I have research, but feels clumsy to inject into the project. I also realize that it is impossible to be objective in a program evaluation if your are the major figure that needs to be evaluated.

Executive Summary
This report is a justification of the evaluation plan for the iPad Entrepreneur Programming Class (IEPC) during the 2011-2012 school year. Since the program cannot be evaluated during the time frame of summer session of EDTech 505, my report is a defense of my chosen methods for evaluation. As a requirement of receiving the grant for Qwest and the Iowa School Board Foundation, I have to report the outcomes of the program to the Iowa School Board Foundation by the end of May of 2012. The grant was received based on the four guiding principles: Improve student learning through the innovative use of mobile technology Enhance the development of students higher order thinking and problem solving skills Increase educators collaborative study of their content, practice, and student effects Increase educators understanding of best practices related to integration of technology for improving student learning. The evaluation will follow the goal-based model (Boulmetis & Dutwin, 2005). The chosen methods for data collection will be surveys and interviews with students in the class, co-teachers of the class, and administration. A checklist of student accomplishments on a pass/fail basis is also one of data sources. The surveys will be administrated through Google Docs, and both the curriculum director and myself will conduct the interviews. Student portfolios will be collected and analyzed by the curriculum director based on a predetermined rubric. Based on the results of the evaluation, the class will be refined to best meet the intended goals. The results for the evaluation will be made public in the Iowa School Foundation newsletter in May and also published in the Cardinal newsletter.

Purposes of Evaluation
The reason for this evaluation is to fulfill the requirements of the Qwest Technology grant as judged by the Iowa School Board Foundation (ISBF). The grant requires that author of the grant report the results they found during the first year of the program. The evaluation is to hold the district, administration, and author accountable for fulfilling the grant, and also the research branch of ISBF are interested at looking at the results of the program.

Questions to Address
After having conferred with Mary Delagardelle, the executive director of ISBF, and discussing the goals of the grant and of the course as it relates to the Earlham curriculum, the following questions are the main area of concern for both the ISBF and the author of the grant:

Has student learning improved through the innovative use of mobile technology? Have students higher order thinking and problem solving skills improved because of this course? How has the co-teachers collaborative study of their content, practice, and student effects been affected? How has understanding of best practices related to integration of technology for improving student learning changed for the co-teachers?

Stakeholders
The students are the more invested stakeholders of this program because they will be receiving the instruction in a newly designed class by instructors who are practicing a new style of instruction. Future students of the class will be impacted because through the evaluation the class should see improvement in the delivery of instruction and content. The co-teachers are the second most impacted stakeholders in this new program. This is a departure into a new content area, and a new style of delivery for both teachers. The administration is also affected by this program because they are responsible for helping evaluate the program, but more importantly they are also responsible for the learning of the district which the course takes place in. They too are responsibility to make sure the program is carried out in a way that benefits the students who take the class.

Background Information
Origins
In the fall of 2010, I applied for a 2011 Teachers and Technology Grant Program. After being one of 6 accepted grants, the program received funding in May of 2011. The Earlham School Board then approved the addition of the new class called Entrepreneur Programming into the business section of the curriculum. The offering of the new class in business marketing section of the curriculum in the high school that will focus on creating, marketing, and testing applications for the Apple iPad 2. The class is an attempt to start to bring back more computer-based programming classes, which had been eliminated the year before within the high school because of downsizing. While the class deals with both marketing and programming content, programming will take the majority of content instruction within the program.

Prior Programs
While Earlham Community School District has not had a previous program like this in the past, the idea for the program came to me when I was reading about research of programs that are similar in nature to this one or about the use of iPads. Numerous articles point to the future of hand held device revolutionizing the world student live in (Dale, 2008; Johnson, 2005; The Consortium for School Networking, 2006). The research on the use of show that the technology itself makes now difference in learning, but shows that it can reduce barriers to education (Vesisenaho et al.; Hussain & Aslam, 2009). The most interesting research came from Crispin Dales article iPods and Creativity in Learning and Teaching: An Instructional Perpective. In the article states that while using mobile devices is risky there are perceived benefits in the growth of creativity in the students work. Most research shows that technology with not cause growth in students learning, so the improvement must come from the changes in instruction of this new program.

People Involved in the Program

The program will be co-taught by the business teacher, Mr. Rogers, and myself (Shane Wheeler). It will be a yearlong course and will meet during one 43-minute class period a day. A curriculum will be developed based on the Iowa Cores 21st Century Skills and the International Society for Technology in Education Standards. The curriculum director, Kristin Sheffield, will oversee the development by the two co-teachers, and will play a key role in evaluating the class as an outside observer. The principle, Jean Fletcher was responsible for changing teacher schedule and the master class schedule to make room for the new class. The superintendent, Michael Wright, was responsible for approving 6

the purchases made with the grant money and any additional money that is authorized to the program. Fifteen students will make up the first class of students to take the class. The students range in age from 14 to 17 year of age. The students need to have taken a math class of Algebra 1 or higher to be accepted into the class.

Characteristics
The students from the entrepreneurial/programming class will use the iPads to replace traditional textbooks with the enhanced offerings of the e-reader with Internet access, which will serve as a platform to create original content related to class. The class will use iPhone Programming: The Big Nerd Ranch Guide by Joe Conway and Aaron Hillegass, and The Business of iPhone App Development: Making and Marketing Apps that Succeed by Dave Wooldridge and Michael Schneider as e-textbooks. Students will learn C+ language. In addition, students will create applications for the school district, such as a school district calendar application or simple learning games for elementary students. Teachers opinions will be sought on applications that would best address the learning needs of their students. For the summation of the course, students will be required to have for sale at least one application of his/her own original design. To develop applications for iPads, iPhones, and iPods, the students will be grouped in teams of three that best balance the skill sets needed to take their products to completion.

Goals
Objective 1: Improve student learning through the innovative use of mobile technology The grant was rewarded based on trying something new and different for the school district. The Earlham community School District will not be the first high school to teach application programming for mobile devices, but it is new for the school. Does these new class and change instruction improve learning? Objective 2: Enhance the development of students higher order thinking and problem solving skills The idea of the program is to face students with the real life problem of marketing, programming, testing, and getting people to download an app from Apple iTunes store. Do these tasks required by students to accomplish this force them to develop higher order thinking skills and know how to face complex problems? Objective 3: Increase educators collaborative study of their content, practice, and student effects

The grant requires that collaboration between teachers be a big point of concern. Does the creation of this course cause the co-teachers to collaborate more? Does this collaboration benefit their ability to understand their content and instructional delivery? What effect does their collaboration have on the students learning? Objective 4: Increase educators understanding of best practices related to integration of technology for improving student learning. The class will use technology, but does the integration of technology enhance the delivery of instruction? Does the class help the co-teachers develop a better understanding of how best to integrate technology to enhance and not just because it is there?

Description of Evaluation Design

The design used for this evaluation will be a goal-based model
(Boulmetis & Dutwin, 2005). The goals for this evaluation are quite straightforward and laid out by the ISBF and the grant application process. The Earlham Community School administration and co-teachers have all signed documents to fulfill and carry out the intentions of the goals. The goals are addressed in the Background Information under the subsection on Goals. While the goals are straightforward in their approach, the design of methods and measurements of these goals are more difficult given the limited number of staff responsible for the program and the fact that the main evaluator of the program is also the person most responsible for the program. Certain features need to be designed into the evaluation to make sure objectivity is maintained as much as possible. This evaluation is both qualitative and quantitative in its collection of data. The quantitative portion will be a pre and post survey given to students in the class. The qualitative portion is a pre and post interview with both students and co-teachers and an evaluation of student portfolios using a rubric. Two surveys were designed to address the use of mobile technology of students daily life and for educational purposes. The first survey will be given to students at the beginning of the school year, with the second survey with similar questions given at the end of the year. The survey utilizes the Likert scale (never, rarely, sometimes, most of time, all the time) and open-end questions. The surveys are intended to track the changes in use of mobile devices for educational purposes and compare changes in regular daily use. If there are no changes in regular daily use, but changes in educational use, a correlation could possibly be drawn from the data to indicate the impact the program is having on how students use mobile devices for education. The survey is not designed to measure the improvement of learning by using mobile devices inside and outside of class, but to monitor habitual changes in student use.

The curriculum director at the beginning and the end of the school year will interview students. Because the instructor is also responsible for the program and evaluation, it is in the best interest of the program that an outside person conducts interviews for the program. This person is also able to observe the classroom on a regular basis to further collect data on the overriding objectives of the program. The curriculum director will ask open-ended questions to students about how they perceive the class, their ability to deal with complex problems, how well they deal with higher order thinking processes, and questions regarding how the different instructional strategies the co-teacher uses benefits or hinders their learning. The curriculum director will also interview both co-teachers and ask them a series of open-ended questions with prompts. The questions will be focused on the topic of their experiences teaching, learning new content, developing coteaching instruction strategies, and integrating technology into the program for best practice. The curriculum director will also rate the students portfolios of work for the school based on a rubric used by the Harmon High School of the Kansas City School District. The curriculum director will analyze work form the beginning of the school year to that of the end of the school year using a rubric, and report out the results. The curriculum director will also be responsible for observing every classroom once a week within the district. The curriculum director has accepted the responsibility of reporting back observations of the class to the leader of the program to include in the report.

Conclusions
There is no way to be completely objective if Im in charge of every evaluative measure. I have enlisted the help of the curriculum director, in an effort to increase objectivity into the evaluation. I would also recommend the principal be more involved, but that is not realistic in this case because he has just been introduced to the school district 10 days before the start of the year and knows very little about the district, let alone one specific class. He has too much on his plate to be involved in this process. It hard not to think about how to answer the questions that you have written about the evaluation, because you start to address your own concerns about the program. Thus, you have already basis the evaluation. I believe now that the best course of action would have been to adopt a goal-free model, and had the principal and curriculum director do all the observations, not knowing the goals of the grant.

Sources
Dale, C. (2008). IPods and creativity in learning and teaching: an instructional perspective. International Journal of Teaching and Learning in Higher Education, 20(1), 1-9. Johnson, D. (2005, October). A vision for the net generation media center. International Society for Technology and Education. Retrieved July 19, 2011, from http://www.eric.ed.gov/ERICWebPortal/contentdelivery/servlet/ER ICServlet?accno=EJ719951 Hussain, I., & Aslam, M. (2009). Role of mobile technology in promoting campus-wide learning environment. The Turkish Online Journal of Educational Technology, 8(3), 48-140. Retrieved July 13, 2011, from http://www.eric.ed.gov/ERICWebPortal/contentdelivery/servlet/ER ICServlet?accno=EJ859492 The consortium for school networking. (2006). Hot technologies for education: what's happening now and later. Middle Ground, 9(4), 12-15. Vesisenaho, M., Valtonen, T., Kukkonen, J., Havu-Vuutinen, S., Hartikainen, A., & Karkkainen, S. (2010). Blended learning with everyday technologies to activate students' collaborative learning. Science Education International, 21(4), 272-283.

10

Appendix A
Evaluator's Program Description
Evaluation Question Has student learning improved through the innovative use of technology? Program Objective/Goals Student will demonstrate an increase in quality of work from the beginning of the year to the end of the school year. Activity(ies) to Observe survey responses, interview responses, review of students' portfolio, analysis of students' success rate of placing an app on iTunes survey responses, interview responses, review of students' portfolio, analysis of students' success rate of placing an app on iTunes interview responses, survey responses Data Source surveys, interviews, portfolio, observation Population/Sam ple students in the business/app programming class Design surveys created and distributed by Google forms; interviews pre, and post; creating a comparison rubric to collect data on student portfolios; observation data collection on submitted and accepted apps surveys created and distributed by Google forms; interviews pre, and post; creating a comparison rubric to collect data on student portfolios; observation data collection on submitted and accepted apps interviews pre, and post by coteachers, administrators, and curriculum directors; interviews pre, and post by coteachers, administrators, and curriculum directors; Data Analysis Descriptive analysis Audience Administration, curriculum director, school board, and Iowa School Board Foundation

Have students higher order thinking and problem solving skills improved because of this course?

Students will be able to analyze and plan for complex problems in order to reach a resolution point. Students will demonstrate growth in higher order thinking skills and problem solving. Co-teacher will present their views on a collaborative work environment and the correlating changes to their teaching practices. Co-teacher will demonstrate the best-practice learned through the process of teaching the class for an entire school year. Co-teachers will present student works as examples of how to apply best-practices for use of technology in the classroom.

surveys, interviews, portfolio, observation

students in the business/app programming class

Descriptive analysis and chart

Administration, curriculum director, school board, and Iowa School Board Foundation

How has the coteachers collaborative study of their content, practice, and student effects been affected? How has the understanding of best practices related to integration of technology for improving learning changed for the coteachers?

surveys, interviews

co-teachers

Descriptive analysis

Administration, curriculum director, school board, and Iowa School Board Foundation

interview responses,

interviews

co-teachers

Descriptive analysis

Administration, curriculum director, school board, and Iowa School Board Foundation

11

Appendix B
EDP Schedule Date
6/27/2011 6/29/2011 6/30/2011 7/1/2011 7/2/2011 7/3/2011

Activity

Create EDP and timeline Gather research on best practices for technology Gather research on best practices for technology Gather research on collaborating teachers in a classroom Gather research on best practices for technology Gather research on higher order thinking skills and problem solving for high school students 7/5/2011 Gather research on higher order thinking skills and problem solving for high school students 7/6/2011 Revise EDP if necessary 7/7/2011 Work on survey questions, interview questions 7/8/2011 Work on survey questions, interview questions 7/9/2011 Work on survey questions, interview questions 7/10/2011 Work on survey questions, interview questions 7/11/2011 Work on rubrics for portfolios 7/12/2011 Work on rubrics for portfolios 7/13/2011 Work on rubrics for portfolios 7/14/2011 Make spreadsheet for recording submitted apps, and approvals share with co-teacher 7/15/2011 Review survey questions, and interview questions 7/16/2011 Start rough draft of final project. Focus on the research proving the effectiveness of technology in the classroom compared to the objectives. The methodology section and background section. 7/17/2011 Work on rough draft, and search for additional research 7/18/2011 Work on rough draft, and search for additional research 7/19/2011 Finish up purpose and background section of the final report 7/20/2011 Revise the background and background section

12

7/21/2011

Finish up the methodology section. This should be the research piece that proves the methods chosen will work. 7/22/2011 Revise methodology section 7/23/2011 Result section? Will not have any 7/24/2011 Write Discussion and conclusion section. Focus on the research and methods. 7/25/2011 Revise the discussion and conclusion sections. 7/26/2011 Write the summary section. 7/27/2011 Revise the summary section. 7/28/2011 Work on revisions and corrections. 7/29/2011 Work on revisions and corrections. 8/1/2011 Hand in Final Project to Dr. Perkins

13

Appendix C

14

15

Appendix D

16

17

Appendix E

Appendix F
Overall Evaluation Questions: 1. Has student learning improved through the innovative use of mobile technology? 2. Have students higher order thinking and problem solving skills improved because of this course? 3. How has the co-teachers collaborative study of their content, practice, and student effects been affected? 4. How has understanding of best practices related to integration of technology for improving student learning changed for the co-teachers? Interview Protocol for Teachers 1. What is your name? 2. What is your previous experience co-teaching? 3. What was your experience like co-teaching this year? 4. What was your experience level with the different content of this class at the beginning of the year? a. What was your experience level with the specific content (app programming & marketing) of this class when you started teaching?

18

2. a. 2. 3. 4. 5. a. 2. 3. 4. a. 2. 3. 4. 5. 6. 7.

1. 2. 3.

b. How has that changed throughout the year? How did you choose the materials for the class? What would you change about the How do you communicate with your co-teacher? Have you explored other ways to work together? What things have you done to try to promote higher order thinking skills in students? How do you think that this class has innovated using mobile technology? What proof do you have that it has been a positive in the classroom? How have you worked with your co-teacher to understand and develop your content? How have you worked with your co-teacher to improve your practice? How do you think your co-teaching has affected student achievement? Can you give explains? b. What data tells you that this is happening? How has co-teaching affected your understanding of best practices? What ways have you worked to understand what best practices work for technology? What data do you have that your best practices affected student learning? How have you integrated technology into your co-teaching? How have the student reacted to the integration? How have you shared your experience with other teachers outside of your coteacher? a. What impact do you think that it has had on other teachers? b. What impact do you think it had on the district as a whole? Having co-taught for a year, what are the positives and negatives of co-teaching? What would be your recommendations to others that might start co-teaching? After teaching this course for a year, what changes would you make in curriculum, instruction, assessment and/or technology integration?

Appendix G
Overall Evaluation Questions: 1. Has student learning improved through the innovative use of mobile technology? 2. Have students higher order thinking and problem solving skills improved because of this course? 3. How has the co-teachers collaborative study of their content, practice, and student effects been affected? 4. How has understanding of best practices related to integration of technology for improving student learning changed for the co-teachers? Interview Questions for Students at the End of the Year 1. What background knowledge did you have on the subject area? a. Can you give specific examples for both programming and marketing? 2. What experience do you have with mobile devices? a. Have you used them for educational purposes? i. If so how? b. How has mobile technology changed your daily life? c. How has mobile technology changed your education?

19

2. Has the use of an iPad changed from the beginning of the year to the end? 1. If so how? 2. Has it benefited you learning? 2. Has your outlook at mobile devices changed at all because of this class? 3. Have you ever taken a class with either teacher before? 1. If so, have they or both changed in the delivery of their instruction? 2. Do you think these changes have improved you learning? i. Could give a specific example? 2. Has the outlook on solving problems changed because of this class? 1. If so, how and why? 2. How do you apply this to other classes, if at all? 2. Do you feel that you have been challenged by this class to think at a higher level? 1. Can you give an example? 2. What challenges could the teacher presented to you to make you think a high level? 2. What were the biggest challenges or problems you faced in this class? 1. How did you deal with it? i. Did it work? 2. Would you take this class again, if you had the chance? 3. Did Apple accept your app?

20