Program Evaluation Report

Evaluation of senior online program: RVOP at Happy Valley High

School.
Submitted to: Ralston Valley High School administration
Submitted by: Terry Oakley: RVOP teacher and Boise State MET
student
Date: 12/10/2014

Summary
The program being evaluated is a senior online class that was implemented by Happy Valley high
school three years ago. It is a mandatory class for all seniors and requires 15 clock hours of time
commitment during one semester. The 15 hours make up for a deficit in clock hours required for
graduation as determined by the state of Colorado. Over the past few years, concerns have risen
regarding the effectiveness of the program and the attitudes of stakeholders involved. The
purpose of the this evaluation, detailed within the report, is to determine attitudes of stakeholders
and to determine whether or not the program is preparing students for life after high school and
for taking an online class.
The evaluator met with the school instructional coach to determine specific concerns about the
program regarding attitudes of all stakeholders and effectiveness of the program. In addition,
goals for of the evaluation project were determined. Results from the interview with the
instructional coach indicated concerns regarding student and teacher attitudes about the class as
well as concerns about the amount of effort students commit to the class. In order to receive the
points associated with activities, student need to achieve a grade of 70% or higher on each
activity. Students can attempt each assignment three times. It is rumored that many students are
putting little effort into the class, sharing results with each other to cheat and complete the course
quickly, or in some cases having their parents complete the course for them. This brings up
ethical as well as liability concerns.
The survey results determine that the goals of the RVOP course are generally not being met.
Students see little value in the content of the course, 57% of them do not feel that the course is
preparing them for life after high school, 21% strongly disagreed with the statement. Of
particular concern is the amount of time and effort students are putting into the course. 76% of
students admitted to putting little time and effort into the course and 75% of the teachers
surveyed agree with them. If students are not meeting the time requirements of the course, the
Colorado Department of Education could prevent them from graduating, even if they have met
course credit requirements.
In a positive finding, students and teachers agree that Schoology is a positive platform for an
online course and the operations aspect of the course are favorable. 75% of students felt that the
course was easy to navigate and did not have any technical issues with the course, 51% of them
feel prepared to take online courses in the future. Teachers responded in agreement with students
in regards to the technical execution of the course. While only 28% of them enjoy teaching the
RVOP class, 63% agreed that Schoology is a user friendly platform and they seldom had issues
with administering the course.
If the program is to be continued, an overhaul of the content and accountability procedures in
place is recommended. A pre assessment would help determine whether or not students have
prior knowledge on the content and allow content developers to better attune the goals and
objectives of the course. A longitudinal follow up could be administered to determine what
aspects of the content indeed prepared students for life after high school.

Description of Program
Happy Valley High School is a traditional high school with a population of 1840 students grades
9-12. The senior online course was developed to fill a void of minutes that seniors needed to
meet state graduation requirements. Due to the complexity of the daily bell schedule, students
were not meeting this basic requirement. Rather than forcing students to attend a mandatory
study hall, the administration decided to use this dilemma as a way to fulfill other goals
determined by the administrative team.
With the increase in the numbers of students choosing to attend online college programs, the
administration felt it was important to help prepare students for success in this type of
environment. Traditional brick and mortar schools and job training programs are increasingly
incorporating online components. At some point in their lives most students will need to be
familiar with online learning. Prior to this program, the only students who received education in
a strictly online environment were students who took an online credit recovery class through the
district virtual academy.
In addition, the administrative team wanted to design the course curriculum to meet the needs of
teenagers preparing to move out on their own. The content of the course focuses on career
pathways, internet safety, personal finance, and nutrition. Within the course, students have the
ability to choose between various online activities, mostly quizzes from outside sites. Each
activity is worth a set amount of hours predetermined by the course designers, the school
instructional coach, teacher librarian, and a social studies teacher. Students must complete the
activities on their own time. All students have off blocks during the school day and technology
resources that allow them to complete the assignments on computers in the school library. In
addition, 99% of the students in the school have access to the internet at home as discovered in
the tech use survey given to students in August, 2014 (see Appendix A).
Program Objectives:

To deliver quality content that will improve student preparedness for living on their own.
Instilling confidence through knowledge and the aptitude to use developed life skills
independently when needed.

To familiarize students with the process of taking an online course. Introduce students to
classroom discussions, online etiquette, self-discipline and time management.

Educating all stake holders on the importance of students learning how to be successful in
an online class and learn content to prepare them for life outside of high school.

The evaluation questions built upon the objectives of this program are:

Did the course activities better prepare students and instill confidence for life
beyond high school?
3

Did students gain experience on how to conduct and self-guide themselves through an
online course, meeting all deadlines and expectations?

Did the online class foster an environment that increased positive attitudes among all
stake holders? Was value determined by stake holders in dedicating the time and effort
into successfully completion and administration of this course?

Program Components
The senior online course is administered through Schoology. Schoology is supported at the
district level and available to every student. Although Schoology is supported by the district and
available to all students and staff, very few teachers use it as a class learning platform. RVOP is
many students first experience with Schoology. Teachers in the school district have received very
little training on Schoology. Some have taken personal time to learn how to administer a class in
the platform, but most have not had the necessary resources to use Schoology to its full potential.
Many teachers administering the senior online course have volunteered in efforts to better
understand the benefits of using Schoology in traditional classes and receive further training on
the resource.

Evaluation Method
Participants
The participants of this program include the 435 seniors at Happy Valley High School and 29
teachers who administer their progress in the course. A sample population of 200 students was
chosen randomly with no order or formula involved. Within this sample group 75% of the
students responded to the survey. This type of probability sampling was necessary because
classes were determined based on 12th grade English teachers who were willing to participate. As
suggested by Boulmetis and Dutwin, 2011 a sample size of 50% is appropriate for populations of
500 or under. There were a few choices in determining the sample for this evaluation. There was
an option to survey the entire population through the online course and offering students credit
for completing the activity, or the option to physically administer the survey through 12th grade
English classes. The chosen method was physical participation from students in their 12th grade
English classes. This decision was made based on the interim interview with the instructional
coach. At the mid semester interview it was determined that many students had not begun
working on the course. In past years a large population of the participants for this course have
waited until the last week of the semester to log into the class and complete the activities
required. Due to the time constraints of this project, if this delayed participation occurred, the
evaluator would not have the opportunity to analyze data within the time restraints of this
evaluation project, therefore missing the deadline.
Procedures
The evaluation was designed using the transactional model with goal based orientation. This
model was selected because the evaluator plays a part in the project and has the perspective of an
active participant. Findings are shared with staff in attempts to improve the program (Boulmetis
and Dutwin, 2011). Since the surveys and interpretations of results are subjective in nature and
open to interpretation by the evaluator, this project will yield data that is qualitative in nature.
4

The evaluator first met with the school instructional coach, a key stakeholder and developer of
the course. At this time, program history was obtained and goals and intentions of the program
were discussed.
Next, the evaluator sent out the survey to the teachers administering the course and a sample of
200 students who are currently taking the course. The main goal of the survey was to determine
attitudes about the program and examine the level of effort and commitment currently being
dedicated to the project by both sets of stakeholders.
Data Sources
The data sources selected for this evaluation were designed to address the program objectives
and evaluation questions.
Interview:
Near the midpoint of the semester, the evaluator met with the instructional coach to clarify the
goals of the project and determine attitudes of stakeholders at this point in the semester. The
questions addressed in this interview can be found in Appendix B, responses in Appendix C.
Surveys:
The primary data source for this evaluation was administered through socrative.com. The survey,
which can be found in Appendix C, was given to students in their 12th grade English classes. The
evaluator randomly selected a group of 200 students who make up 7, 12th grade English classes
and sent out an email to teachers willing to participate. The email contained a class link for
students to enter into the socrative.com website. Students entered the class code using their smart
phones and took the quiz. The evaluator needed to make the quiz as short as possible so it did not
cut into instructional time, therefore the quiz only addresses the most pertinent information
needed to answer the overarching questions and goals of the evaluation. Since determining
attitudes is main goal of the evaluation, a Likert scale was used to develop the survey. 75% of the
sample group responded to the survey.
The second data source for this evaluation was also delivered through socrative.com. The link to
the survey was delivered through email to the 29 teachers who currently teach the senior online
course. Like the student survey, this survey was also developed using a Likert scale in attempts
to determine attitudes about the program. This survey can be found in Appendix D. 83% of the
teachers responded to the survey.

Results
Table 1.1 Student Survey
5

The assignments in the RVOP class were useful. They helped prepare me for
life after high school.
A-Strongly Agree
0%
B-Agree
16%
C-Neither Agree or Disagree
27%
D-Disagree
27%
E-Strongly Disagree
36%
Results are based on a Likert scale ranging from A (Strongly Agree) to E
(Strongly Disagree). This item received 149 responses. Values were rounded
to the nearest whole number.
Table 1.2
I put considerable time and effort into the RVOP course. I took my time completing assignments
and made sure I did things correctly the first time around. If an assignment was worth 3 hours, I
spent around three hours completing the work.

A-Strongly Agree
B-Agree
C-Neither Agree or Disagree
D-Disagree
E-Strongly Disagree

1%
8%
14%
48%
28%

Results are based on a Likert scale ranging from A (Strongly Agree) to E

(Strongly Disagree). This item received 147 responses. Values were rounded
to the nearest whole number.
Table 1.3
Taking RVOP has made me more comfortable with the idea of taking an online class. I
understand how to participate in online discussions, practice online etiquette, and feel
comfortable uploading assignments into Schoology.
A-Strongly Agree
B-Agree
C-Neither Agree or Disagree
D-Disagree
E-Strongly Disagree

10%
41%
23%
21%
5%

Results are based on a Likert scale ranging from A (Strongly Agree) to E

(Strongly Disagree). 146 students responded to the survey. This item
received 146 responses. Values were rounded to the nearest whole number.

Table 1.4

After taking the RVOP course, I now feel that I have the self-discipline and time management
6

skills to complete an online college course.

A-Strongly Agree
B-Agree
C-Neither Agree or Disagree
D-Disagree
E-Strongly Disagree

2%
30%
37%
36%
7%

Results are based on a Likert scale ranging from A (Strongly Agree) to E

(Strongly Disagree). This item received 151 responses. Values were rounded
to the nearest whole number.
Table 1.5

The RVOP class was easy to navigate. I understood what I needed to do and didn't have
technical problems taking the course.
A-Strongly Agree
16%
B-Agree
57%
C-Neither Agree or Disagree
11%
D-Disagree
11%
E-Strongly Disagree
5%
Results are based on a Likert scale ranging from A (Strongly Agree) to E
(Strongly Disagree). This item received 152 responses. Values were rounded
to the nearest whole number.
Table 2.1

I enjoy teaching the RVOP class at Ralston Valley HS and feel it is a good use of my time and
effort.
A-Strongly Agree
0%
B-Agree
28%
C-Neither Agree or Disagree
44%
D-Disagree
20%
E-Strongly Disagree
8%
Results are based on a Likert scale ranging from A (Strongly Agree) to E
(Strongly Disagree). This item received 25 responses. Values were rounded to
the nearest whole number.

Table 2.2

I feel that the RVOP class effectively helps to prepare students for life after high school.
7

A-Strongly Agree
B-Agree
C-Neither Agree or Disagree
D-Disagree
E-Strongly Disagree

0%
67%
17%
13%
4%

Results are based on a Likert scale ranging from A (Strongly Agree) to E

(Strongly Disagree). This item received 24 responses. Values were rounded to
the nearest whole number.
Table 2.3

I feel that the RVOP class effectively helps to prepare students for taking an online course.
A-Strongly Agree
B-Agree
C-Neither Agree or Disagree
D-Disagree
E-Strongly Disagree

8%
79%
8%
4%
0%

Results are based on a Likert scale ranging from A (Strongly Agree) to E

(Strongly Disagree). This item received 24 responses. Values were rounded to
the nearest whole number.
Table 2.4

I feel that most students put considerable effort into the activities in the RVOP class.
A-Strongly Agree
B-Agree
C-Neither Agree or Disagree
D-Disagree
E-Strongly Disagree

0%
8%
4%
75%
13%

Results are based on a Likert scale ranging from A (Strongly Agree) to E

(Strongly Disagree). This item received 24 responses. Values were rounded to
the nearest whole number.

Table 2.5

I feel that Schoology is a user friendly platform for administering the RVOP course. I seldom
have frustrations finding the information I need to grade assignments in the course.
A-Strongly Agree
B-Agree
C-Neither Agree or Disagree
D-Disagree
E-Strongly Disagree

8%
63%
8%
13%
8%

Results are based on a Likert scale ranging from A (Strongly Agree) to E

(Strongly Disagree). This item received 24 responses. Values were rounded to
the nearest whole number.

Discussion
The purpose of this evaluation was to determine attitudes about the senior online program from
both a teacher and a student viewpoint and to determine the perceived effectiveness of its
content. An initial concern is a contradiction on the opinion of effectiveness of the content
between teachers and students. When surveyed 57% of students did not feel that the course
prepared them for life after high school, 21% strongly disagreed with the statement. On the
contrary, 67% of teachers felt that the activities in the course are better preparing students for life
after high school. If the program continues a review of the content and the opposing viewpoints
between teachers and students is recommended. It would be necessary to determine students
reasons for not finding value in the material, do they already know the material covered or do
they not fully understand the relevance as they are still in high school and living with their
parents? A student pre assessment is recommended to determine prior knowledge as well as a
potential summative exam upon completion. Ultimately, a longitudinal follow up is
recommended to determine effectiveness of the material once students are truly living on their
own.
As discovered in the survey results, 76% of students disagreed with the statement that they put
considerable time and effort into the course. This suggests that the majority of students cut
corners and put little effort into the course assignments. In agreement, 75% of the teachers
disagreed with the survey statement. This is clearly a problem that needs to be addressed. An
accountability plan is recommended to better determine strategies for increasing student effort.
This also brings up a liability concern. Schoology allows teachers and administrators to see the
amount of time students spend on some of the course assignments. If actual clock hours
were audited by the state, the school could potentially receive disciplinary action if the stated
student clock hours are not being met through the efforts of the RVOP class. Seniors could
potentially not graduate if they do not meet the clock hour requirements.

Students and teachers do agree that the course better prepares students for taking future online
courses. Though only 32% of the sample felt that taking the RVOP course improved their time
management and self-discipline skills needed to take and online college course, 75% of students
felt that the course was easy to navigate and did not have any technical issues with the course.
Consequently, having this positive technology experience 51% of students now feel prepared for
the process of taking an online course and the procedures involved.
While only 28% of teachers enjoy teaching the RVOP class, 63% of them agreed that Schoology
is a user friendly platform and seldom had issues with the administration of the course. Attitudes
of using Schoology as a platform for online class administration are a positive outcome of the
RVOP course. Teachers may consider adding more online elements to their traditional classes
because of this positive experience.
If the program is continued in the future, recommended actions are to overhaul of content of the
program and examine ways to increase student accountability. This would demand an increased
staffing and financial commitment to execute properly. If the funds and resources are not
available for this type of attention, it is recommended that the program is discontinued.

Project Cost
Primary Evaluator

Rates or unit
cost
10 days at $500.00/day
2 hours at $40.00/hour

Terry Oakley (Evaluator)

Meeting time with instructional
coach.
Supplies (Copies, Internet access etc.) $50.00

subtotal costs
$5000.00
$80.00
$50.00

Total Budget $5130.00

Works Cited
Boulmetis, J.,& Dutwin, P. (2011). The ABCs of evaluation 93rd ed.). San Fransisco, CA:
Jossey-Bass

10

Appendix A:

Exert from the 2014 Tech Use Survey at Ralston

Valley HS

Appendix B: Interview Questions

1. What were the initial goals of the RVOP program?
2. Were there any additional goals determined when the program
began?
3. What improvements or changes have been made over the first
two years of the program?
4. Is there support from administration to continue with the
program?
5. What are current concerns about the program?
6. How would you describe student attitudes about the program?
7. How would you describe teacher attitudes about the program?

11

Appendix C: Interview Question Response

1. What were the initial goals of the RVOP program?
The initial goals of the RVOP program were to make up for a class time discrepancy for
graduating seniors. While students meet the credit requirements as determined by the state,
they do not meet the seat time requirements. The school needed to come up with a solution to
meet this requirement with as little disruption to the current schedule as possible.
2. Were there any additional goals determined when the program began?
Many conversations were arising about students not having the necessary skills to
successfully live on their own after high school. In addition, the school was receiving
pressure from the district to use Schoology, an online learning platform that the district
implemented. Many teachers had voiced concerns about students having little or no
experience taking online classes and their ability to succeed in this type of class in college or
future job training.
3. What improvements or changes have been made over the first two years of the
program?
When the program was developed the development team had to pull together the program
very quickly with minimal support. The first few years posed many problems that needed to
be addressed. An initial concern was the initial attitude of the senior class during the first
year of execution. They felt that the class was a punishment to them since they were the first
class to do it. They had negative attitudes and were not very patient with the quirks that
needed to be worked out. Teachers shared frustration as well because there was a learning
curve involved with administering the class, many teachers were unsure about what they
were responsible for doing. Through additional training, teachers became more comfortable
with the program. When students found out that a mandatory study hall was the alternative to
the class, they were more cooperative. Another concern was students waiting until the last
minute to complete the class. They would simply guess on quizzes and retake them until they
passed, spending as little time as possible on the tasks. The following year, students were
limited to taking each quiz 3 times.
4. Is there support from administration to continue with the program?
This year the administration team at the high school is almost completely new. The program
was already in place at the beginning of the school year. At that point there was no choice but
to continue the RVOP course for the 2014-2015 school year.
5. What are current concerns about the program?
A big concern is that students are still cutting corners and not putting adequate time or effort
into the program to receive benefits or gains. Another concern is that teachers are over
worked and this program puts additional pressure on them and their work load, ultimately
taking away from the impact they have on students in their traditional classes.
6. How would you describe student attitudes about the program?
Since this is the third year that students are taking the RVOP class, the attitudes have
improved. They no longer think of the class as a personal punishment to them because it is
12

something that has been done in the past. Students in general are still put minimal time into
the course. At the time of this interview, few students had performed the required activities. It
is the administrators hope that students will spend time over Thanksgiving break completing
the program.
7. How would you describe teacher attitudes about the program?
Teachers have a somewhat negative attitude about the program because it is an extra duty
they are not being paid for. In addition, teachers responsibilities and accountability reporting
responsibilities have increased in the past few years and many feel that they are being
stretched too far.

13

Appendix C: Student Survey:

Link to survey with results

14

Appendix D: Teacher Survey:

Link to survey with results

15