2

Table of Contents
Overview.3
Goals Analysis4
Figure 1: Goal Statement & Goals Analysis Diagram.....4
Subordinate Skills Analysis..5
Figure 2: Subordinate Skills Analysis for Step 6 in Public Program Evaluation ....6
Learner Analysis....6
Table 1: Learner Analysis for PAD6327: Public Program Evaluation..7
Context Analysis...12
Performance Context Analysis.12
Table 2: Analysis of Performance Context for PAD6327: Public Program Evaluation... 13
Learner Context Analysis16
Table 3: Analysis of Learner Context Analysis for PAD6327: Public Program
Evaluation.16
Appendix A: PAD6327 Student Interview Questions..19
Appendix B: PAD6327 Student Interview Questions Part II......20
Appendix C: PAD6327: End of Course Evaluation Survey....21















3

Overview

PAD6327 Public Program Evaluation is a Graduate course offered at the University of
Central Florida. During this course students will concentrate on concepts and issues of
public programs. This course focuses on developing an evaluation process and designing
questions that will aid in the plan and growth of the program. Students will apply principles
to assess the needs of the platform, discover issues that affect the outcome of the process,
analyze questions, and evaluate ethical issues. Students will focus on the evaluation
process and the related results.
This course is one of the major elective courses for the Master of Public Administration
program and is one of the core courses for the Graduate Certificate in Nonprofit
Management program.
PAD6327 will be delivered on the World Wide Web. Students will be required to contribute
to discussions and complete assignments as scheduled. This is a 16 week course.
Below is an Analysis Report, which summarizes the goals of the course along with the
subordinate skills analysis for evaluation issues and questions.





















4

Goals Analysis
Students in PAD6327 will explain major concepts in program evaluation and apply the
concepts in developing a practical evaluation proposal (Figure 1). Additionally, students will
apply principles of research design to evaluation questions. Students will also identify
useful, feasible, ethical, accurate, and culturally sensitive methods of conducting process
and outcome evaluation for research administration
students.
The goals statement was identified by Dr. Chen and the subject matter expert (SME) for this
course, Dr. Jo Smith of the Department of Public Administration. It states that:
Working with a service-learning partner, Research Administration graduate
students will formulate an action plan to evaluate a public or nonprofit
organization using a logic model following established research methods.
The statement was classified as an Intellectual Skill based on Gagnes
taxonomy. According to the Dick and Carey model, intellectual skills require learners to
learn concepts, follow rules and solve problems in order to master goals (2009, p.44). The
goals statement identifies eight major steps that are necessary for students to successfully
meet the course goal statement. The subsequent sections of this analysis report will focus
on Step 6: Formulate Evaluation Questions, where the focus will be on creating evaluation
questions for Public Program Evaluation Research.
Figure 1: Goal Statement & Goals Analysis Diagram









5
Subordinate Skills
Figure 2 depicts the subordinate skills and procedural tasks for the sixth step: Formulate
Evaluation Questions. To begin the process, learners will need to identify who the decision
makers are in regards to current evaluation issues at hand. Once the decision makers are
identified, the purpose of the evaluation needs to be determined by asking the decision
makers what they feel is the purpose of the evaluation.
After the purpose of the evaluation has been determined, a program logic model will need to
be constructed. Following the construction of the program logic model, summary questions
for the evaluation should be drafted. The learner should develop both formative and
summative questions. Examples of some formative questions are:
What is the definition and scope of the problem or issue?
Where or how serious is the problem?
How should the program or technology be delivered to address the problem?
How well is the program or technology delivered?
Examples of summative questions include:
What type of evaluation is feasible?
What is the effectiveness of the problem or technology?
What is the net impact of the program?

Once those questions have been drafted, the last step is to verify that the evaluation
questions meet the required criteria to be included in the evaluation plan.
It should be noted that the entry skills required for this step are the completion of steps 1-5
of the Goals Analysis Diagram (see Figure 1) prior to beginning step 6. Since these skills
apply to the entire unit or course goal, it does not have any direct connection with any
specific subordinate skill.














6

Figure 2: Subordinate Skills Analysis for Step 6 in Public Program Evaluation in
Research Administration



Learner Analysis
One significant reason analysis of learners is very important is to ensure that design and
implementation of instruction is geared to the specific needs of the intended target group. It
is significant to note that Instructional Analysis and Learner Analysis are parallel steps, not
sequential, in systematic design. The size and complexity of a goal will frequently depend
on any number of characteristics of those who will perform the tasks IDs identify and
describe.
Below is a table that explains across nine separate categories, the data sources, learner
characteristics and implications for design, delivery and evaluation of the target audience for
PAD 6327. Data was collected from four major sources: a verbal interview with Dr. Jo
Smith, the professor for the course and the SME, the UCF Graduate schools current
catalog description of the MRA masters program, a written interview with April Heyward
(graduated from the UCF MRA program in August 2013 and was enrolled in PAD 6327






7
during the Summer 2013 semester), and finally anonymous responses to a PAD 6327
Summer 2013 course evaluation filled out by three students. The last two items can be
found in Appendices A and B respectively.
Table 1. Learner Analysis for PAD6327: Public Program Evaluation
Information
Categories
Data
Sources
Learner
Characteristics
Implications:
Design, Delivery and/or
Evaluation
Entry
Skills and
knowledge
Interviewed:
Dr. Jo Smith
Records:
UCF
Graduate Catalog
2013-14, Research
Administration MRA
Interviewed:
April Heyward, a
student enrolled in
PAD6327 in 2013

















Relative to course content:

PAD6327 is the culminating course for
the UCF Master Program in Research
Administration (MRA), as such, learners
will normally have already taken 33 hours
in:

PAD 6742 Introduction to Research
Administration

PAD 6XXX Governance and Regulatory
Issues for Sponsored Programs

PAD 5850 Grant and Contact
Management

PAD 6743 Leadership and Organization
Models in Research Administration

PAD 6744 Financial Management in
Research Administration

PAD 6417 Human Resource
Management

PAD 6745 Contracting for Sponsored
Programs

PAD 6335 Strategic Planning and
Management

PAD 6746 Intellectual Property,
Technology Transfer and
Commercialization

PAD 6747 Audits in Research
Administration

PAD 6237 Research Integrity for
Research Administrators
While most students have are highly
experienced in the RA field by the time
they enter PAD 6327, they are not
familiar with the course content.

Ms. Heyward reported that she was
exposed to some concepts pertinent to
Relative to course content:

PAD6327 is the last in a sequence
of 12 courses required to earn the
MRA. It may be necessary to
provide a brief review of
information covered in the earliest
courses, particularly for those skills
necessary for successful
completion of Public Program
Evaluation.

Given that there is variability in the
entry skills of incoming students
(due to undergraduate or work for
certifications, prior to being
accepted into the UCF MRA) the
professor may want to administer a
pre-test or even do a brief survey of
all students in order to introduce
any concepts that are basic
building blocks for the course.

Relative to Delivery System:

The MRA is given entirely online,
therefore students are thoroughly
fluent in all skills needed for a
positive learning experience.
Additionally, UCF has numerous
tools readily available to assist
students who are taking courses
online such as videos, tutorials,
and technicians are available by
phone and email. Last, the
professor is available for help with
any glitches that may be creating
difficulty for the student.






8
PAD 6327, while taking the course
PAD 6335 - Strategic Planning and
Management. In particular, she
mentioned that she learned Formative
evaluations monitor the progress of the
strategic plan implementation on an
ongoing basis and provide feedback for
improvements. That same student also
reported that an undergraduate course in
Quantitative Methods which taught such
skills as probability, tests of significance,
linear programming, and Markov
analysis, queuing theory, inventory
models, basic calculus with business
applications, was very helpful in
preparing her for PAD 6327.

Relative to Delivery System:

The MRA is given completely online
indicating that by the time they are taking
PAD6327 students are sophisticated
users of technology, experienced
practitioners of internet research and
excellent independent workers who are
highly self-motivated (students must
maintain a B average to stay in the
program).
Prior knowledge
of topic area
Interviewed:
Dr. Jo Smith
Interviewed:
April Heyward, a
student enrolled in
PAD6327 in 2013

The interview with Dr. Smith revealed that
students entering this course have not
been introduced to the concepts in Public
Program Evaluation in any prior MRA
courses. Given the newness of the
subject, and the fact that it is known in
relatively small academic circles,
learners prior knowledge is likely little to
none.

Ms. Heywards responses reinforced
information found in the previous
category: that there is a wide variation of
prior knowledge in students taking
PAD6327, due to classes taken and
certifications earned as undergraduates
before entering the UCF MRA program.
Given that entering students are
unfamiliar with the concepts
necessary for an understanding of
what Public Program Evaluation is,
designers should start with more
basic information than they
otherwise would with such an
intellectually sophisticated
group. Given the caliber of
students, their ability to learn will
quickly make the learning curve
less steep. It is likely that they will
rapidly gain momentum and be
able to demonstrate the pragmatic
use of the concepts, rules and
problem-solving skills in the field.

Despite the fact that some students
may have already acquired skills
helpful in performing public
program evaluations, the professor
is only safe in assuming those
concepts and problem solving
methods taught in the first eleven
courses of the MRA program.
Attitude toward
content
Interviewed:
Dr. Jo Smith


Dr. Smith, as the instructor, felt that the
unfamiliarity with the topic led to an initial
struggling with the content and thus, a
somewhat negative attitude toward
content. Once they were assured of its
relevancy, and fact that these skills will
be expected of them if they want to
Being aware of the students
intimidation by these particular
aspects of research methodology is
important for the instructional
designer. Once they learn that
concepts such as quasi-
experimental, meta-analysis,






9

Survey:
3 Students taking
PAD6327 during
Summer 2013.

Interviewed:
April Heyward, a
student enrolled in
PAD6327 in 2013


advance in their careers, their attitudes
improved.

Ms. Heyward reported that, the course
content is beneficial for, but not limited to:

Developing a proposal for funding

Developing & evaluating a new or
existing program/service/technology

Working with faculty and staff principal
investigators in developing their
proposals for funding

Reviewing proposals for funding as a
Peer Reviewer

There were eight questions in the course
evaluation dealing with matters of course
content that were relevant to
Implications: Design, Delivery and/or
Evaluation. Those questions were 2, 11,
12, 14, 18, 19, 21 and 22. These
answers can be found in Appendix B and
are described below.

The most positive aspect to be identified
in the course evaluations was the
students appreciation of Power Point
presentations and video lectures that
explained in a practical way how to apply
concepts to their evaluation plans.

On the negative end, 1 or 2 students (of
the 3 who submitted their evaluations)
mentioned they did not find the course
contents relevant, useful in understanding
Research Administration, nor were
satisfied with it. Also, 1 student found the
textbook to be intense and stodgy, a lot
to read and not easy to conceptualize the
information in the way we would need it
for our own plans.

Finally, one student found the course to
be moderately easy and two found it to
be not easy at all.
statistical effect sizes (powers and
errors) are similar in difficulty level
than other concepts they have
learned, students proceed without
unusual difficulty. It would be
important to introduce them to the
lingo of their field at this time,
possibly in an instructor provided
glossary or perhaps in one created
by the students as part of an
assignment. Rapid improvement
in attitude toward content can then
be comfortably anticipated.
IDs should inform their work with
the knowledge that students learn
most readily when they have
videos or presentations that remain
stable and can be reviewed as
many times as possible for
understanding, and that they have
a stronger preference for electronic
media over printed materials.

Only one student felt negatively
enough about the textbook to
mention it in the survey. It would be
difficult to know from that one
response if it is worth the time to
investigate other texts.

Attitude toward
delivery system
Interviewed:
Dr. Jo Smith
Records:
UCF Graduate
Catalog 2013-14,
Research
Administration MRA

Interviewed:
April Heyward, a
student enrolled in
The MRA program is given completely
online. The students are employed in full-
time positions as research administrators
and many of them have families. The
online system, while often challenging,
makes it possible for these students to
earn their degrees. As has been stated
previously, this is a culminating course in
the program and as such students are
familiar with the technology necessary to
do well in an online course. Given the
above, their attitudes toward the delivery
The completely online delivery
system of the UCF MRA attracts
many of its students precisely
because it is extremely beneficial
for them to work at their own pace,
place and time.

Instructional designers can take full
advantage of the fact that students
are so familiar with having their
course completely supplied to them






10
PAD6327 in 2013 system are extremely positive.

Ms. Heyward concurs with the above
adding that the Canvas-Instructure
System which allowed me to take the
required courses in the program. The
online system was my general learning
preference while she was taking her
degree courses. She elaborated, It
[Canvas] allows you to work
collaboratively with fellow cohort
members around the country. You can
post and view videos developed by
faculty and fellow cohort members. The
system notifies you of your progress
during each course.

through technological means. The
IDs can use their expertise with all
areas of online learning and vary
instructional delivery through the
use of videos, podcasts, and
information accessible by devices
other than computers such as
tablets, phones and PDAs.

It is very important for IDs to keep
in mind while developing curriculum
and designing instruction that they
should take advantage of the ease
with which classmates can work
collaboratively. This is especially
imperative because research
administrators generally work in
teams on the job.
Academic
motivation
Interviewed:
Dr. Jo Smith
Records:
UCF Graduate
Catalog 2013-14,
Research
Administration MRA

Interviewed:
April Heyward, a
student enrolled in
PAD6327 in 2013

UCF requires that a student, in order to
remain in the program, cannot earn lower
than a B in any of the 12 MRA
courses. This alone motivates students
to achieve to the best of their ability.
The caliber of professionalism and
independence required to prepare for and
enter the field of Research Administration
is significant enough to assure
Instructional Designers that they also
possess considerable degrees of self-
generated incentive to consistently work
in a diligent and effective manner.

It is safe to say that Ms. Heyward spoke
for herself and her classmates when she
said My primary motivation during the
semester, the last semester of graduate
school, when I took PAD 6327 was
graduating with my MRA degree on
Saturday, August 3, 2013 at the
University of Central Florida.
Instructional designers need to take
full advantage of the fact that all
students are currently employed in
Research Administration. Their
work-circumstances augment their
academic motivation, which
provides them with the context in
which to do their best work. It is
reasonable for IDs to expect that
these students can process
sophisticated material efficiently as
they design curriculum.
Educational and
ability levels
Interviewed:
Dr. Jo Smith
Records:
UCF Graduate
Catalog 2013-14,
Research
Administration MRA
Surveyed:
Students who enter the MRA program are
already college graduates. At the time
they take PAD6327 they have almost two
additional years of post-secondary
education.
To quote the UCF graduate school
catalog: The Master in Research
Administration is a highly competitive
program that admits only thirty (30)
students once a year in the fall
semester.
In order to be accepted into the MRA
program students must submit three
letters of recommendation: letters of
recommendation should be from
professors or professional research
administrators who can attest to the
applicants ability to succeed in graduate
coursework and his or her work ethic.
Finally prospective students must
accompany their applications with
Based on the stringent
qualifications for being accepted
into the program and remaining in
it, it is safe to say the educational
and ability levels are fairly
heterogeneous and unquestionably
solid. Instructional design for such
a group should keep this in mind
when deciding how large individual
goals should be and when they
make decisions regarding how
much detail this class of learners
will need for each step.
The ID can safely expect a high
level of intuitive, abstract, critical
and concrete thinking when
creating learning materials for this
course.






11
mission statements and sophisticated
descriptions of why they want to be in this
program, what their areas of special
interest are, what their expectations are
for this program and what they feel is
special about the qualifications they will
bring to the program.
General learning
preferences
Interviewed:
Dr. Jo Smith
Surveyed:
According to Dr. Smith this is a
heterogeneous group when it comes to
learning preferences. A certain
percentage prefers working in groups,
others like to work individually and a third
group is comfortable with both. Most
students have a strong appreciation for
instructional videos because they are
valuable for reinforcement of clarifying
initial confusion with new material.
It is important to note that as these
students progress in the field of Research
Administration their need to work in
groups will continue to grow.
Instructional designers should
provide material that supports
students who are the most
reluctant and have the greatest
difficulty working in groups.
Perhaps it would be helpful to
make sure that every student rates
their own and each group
members work in order to allay
concerns about grading fairness
with group work.
Attitudes
towards UCF
and the
organization
evaluated during
PAD 6327
Interviewed:
Dr. Jo Smith

Interviewed:
April Heyward, a
student enrolled in
PAD6327 in 2013
Dr. Smith explained that students are
initially concerned about whether or not
the MRA is accredited at UCF because it
is such a new field and very few
institutions offer majors. Once they are
assured that accreditation is covered by
UCFs university-wide umbrella
accreditation, their attitude is one of
satisfaction.
Additionally, the students are appreciative
that UCF was awarded a grant from the
National Council of University Research
Administrators to start a two-year
Masters Program.
Ms. Heyward concurred with the above
stating that The University of Central
Florida is the premier institution in the
country and one of the few institutions
that offered a Masters program in
Research Administration. The MRA
program is comprehensive and covers all
areas of Research Administration.

Heyward was employed by the University
of South Carolina during the MRA
program (and is currently) thus, that is the
organization she evaluated during PAD
6327. She stated that the University of
Central Florida is more progressive
regarding Research Administration as
compared to the University of South
Carolina such as technology, policies,
procedures, etc.
There are, fortunately, no negative
attitudes that instructional
designers need to address.

All of Ms. Heywards remarks
support the observation that IDs do
not need to adjust design, delivery
or evaluation due to any negative
attitudes on the part of students
towards UCF.

While the students attitude towards
the organization they evaluate for
this course is relevant, because of
the variation due to each student
potentially working on a different
evaluation, it is reasonable that ID
cannot anticipate the needs of the
individual learner in this
regard. The professor may have
to start with a checklist and work
one on one with students hopefully
giving each one reason to see
positive aspects in situations that
are not apparently encouraging.







12

Group
Characteristics

Interviewed:
Dr. Jo Smith
Records:
UCF Graduate
Catalog 2013-14,
Research
Administration MRA

Interviewed:
April Heyward, a
student enrolled in
PAD6327 in 2013

The MRA is limited to 30 students. As a
result of the competitiveness necessary
to become one of the 30, the group as a
whole consistently demonstrates strong
motivation and superior diligence in
independent work.

Ms. Heyward observed that the
inaugural graduating class of the Master
of Research Administration (MRA)
program consisted of females mostly and
three males. I believe one male had a
Master degree prior to entering the MRA
program at UCF. Everyone else had
Bachelor degrees prior to entering the
MRA program at UCF. The inaugural
graduating class had diverse
backgrounds and experiences relating to
Research Administration. There was a
mix of students in the MRA program that
held management positions and non-
management positions at their respective
institutions.

Since there are only 30 students or
less in this course, designers can
capitalize on the fact that the
instructor can gear her teaching to
smaller groups and not have to
deal with the impersonality of
communicating with a larger
population. She has the luxury of
tailoring the course delivery to
student needs more-so than if the
group were larger. Since PAD
6327 is a culminating class,
designers can take advantage of
the fact that these MRA candidates
are well-acquainted with the
standard techniques and
vocabulary of Research
Administrators each having
worked in the field for 3-5 years
and having received a B average
or higher in the first 11 courses of
the program.

Additional Information Requirements
The interviews with Dr. Jo Smith and Ms. Heyward provided substantial details for the
purpose of the learner analysis, as did the pertinent descriptions of the MRA requirements
in the graduate school catalog. Only the sparse number of students who filled out the class
evaluations shows any promise from further investigation.
Context Analysis
The Context Analysis report is comprised of information pertaining to the performance and
learning environment of students in the Public Program Evaluation course. The information
provided is meant to enhance the learning experience for students as well as provide insight
for students to enhance their motivation and apply their newly acquired skills in the
workplace. Information was obtained through interviews conducted with the SME, Dr. Smith
who is also the instructor of this course, and a survey sent to a former student, April
Heyward, who completed this course previously.
Performance Context Analysis
Table 2 has the results of four information categories relevant to performance context
analysis. Data was contributed by the SME, Dr. Smith, and by a former student of PAD632,
April Heyward.
The following table contains four categories. The first category defines the level of
managerial or supervisory provisions in the work setting.
The second category analyzes the physical facets of the location; we collected information
based on four areas: the setting, the equipment, the time and the resources.






13
In the third category, we measured the social facets of the work setting that are often
determined by working in the field of Public Evaluation.
The last category in the performance context analysis shows how the skills learned in the
instructional unit will be pertinent to the work environment. Based on feedback from April
Heyward, (see the performance context box under Relevance of new skills to workplace)
one sees how well prepared Dr. Smiths students were to function in the real world of
Research Administration. This is an indication that former students are an excellent source
of information that can impact future instructional design.

We measured the social facets of the work setting that are often determined by the
spectrum of the research. The last deliberation in the performance context analysis is to
guarantee the skills learned in the instructional unit will be pertinent to the work
environment.
Table 2: Analysis of Performance Context for PAD6327: Public Program Evaluation
Information
Categories
Data Sources Performance Context Implications Design, Delivery
and/or Evaluation
Managerial/supervisory
support
Interviewed:
Dr. Smith

Interviewed:
April Heyward
a student
enrolled in
PAD6327 in
2013
Although a limited number of students
were able to complete the survey,
students who have been employed in
the Public Research Administration
field will need to be examined in order
to determine the level of
managerial/supervisory support, for
the purpose of this analysis, it is
assumed that supervisory support will
vary based on the companys
environment and the supervisors
personality.

According to Ms. Heyward, I selected
to evaluate the USCeRA (University of
South Carolina Electronic Research
Administration) System. I met with the
IT & Data Management Director who
developed the USCeRA system in
2002. The IT & Data Management
Director reports directly to the Vice
President for Research. The
stakeholders that I identified for the
evaluation plan were:
IT & Data Management Director
Involved in USCeRA Operations
Category
Research Awards Budget Manager
Involved in USCeRA Operations
Category
Senior Applications Analyst
Involved in USCeRA Operations
Category
Research Awards Budget Assistant
Involved in USCeRA Operations
Category
IT & Administrative Support
Involved in USCeRA Operations
Category
USC Faculty Served or Affected
Instruction should cover complications
and potential pushback from
managers/supervisors that may arise
during the evaluation process so
students will be better equipped to
handle various levels of resistance
from managers/supervisors that might
occur in their future work.


Ms. Heyward met with over 10
members of the USCeRA Operations
team, and an unspecified number of
USC staff and students, she availed
herself of an optimal managerial and
supervisory support
experience. Making herself known to
such a wide variety of workers that
she would interact with gave her a
massive advantage over any other
Research Administrators that may
have limited themselves to a select
few.

The implications for design and
delivery are huge. A worker who does
the work upfront to learn as much as
he/she can about all those who may
have an impact on his/her work opens
up avenues that can be accessed
when necessary in order to create
products that are the result of being
able to reach the most ambitious and
sophisticated goals of instruction.







14
by USCeRA Category
USC Staff Served or Affected by
USCeRA Category
USC Students Served or Affected
by USCeRA Category
Vice President for Research
Intended Users of Evaluation Findings
Category
Associate Vice President for
Research Intended Users of
Evaluation Findings Category


Physical aspects of site Interviewed:
Dr. Smith

Interviewed:
April
Heyward, a
student
enrolled in
PAD6327 in
2013

Facilities: Each facility will vary. It is
assumed that each student will have
access to an appropriate workspace
and necessary equipment needed to
complete job requirements.

Resources: It is assumed that
students will be given access to the
necessary data needed for completing
program evaluations.

Time: While Ms. Heyward worked on
her evaluation plan she was a student,
not an employee of the University of
South Carolina (which she now is).
Students were given 15 weeks to
complete their evaluation plan.

Equipment: It is assumed that most
companies will have the necessary
computer equipment and software
required to complete program
evaluations.
Instruction will mirror real workplace
settings as close as
possible. Students will be required to
complete service-learning hours to
gain real world experience. Students
will also have access to the necessary
hardware and software required to
complete a public evaluation.

Opportunities will be provided through
the service-learning hours to practice
writing public evaluations in a real
world environment.

In the first part of Ms. Heywards
student interview questions, she made
the comments: The University of
Central Florida is more progressive
regarding Research Administration as
compared to the University of South
Carolina such as technology, policies,
procedures, etc. For the purposes of
design and delivery, one can deduce
that there is not an unlimited amount
of time available to accomplish an
evaluation in real world at the
University of South Carolina. Thus,
the design and delivery of instruction
can describe optimal conditions under
which to produce an evaluation and
give students options for the best way
to cut corners when necessary.

Social aspects of the
performance site
Interviewed:
Dr. Smith

Interviewed:
April
Heyward, a
student
enrolled in
PAD6327 in
2013
The social aspects of a workplace
have a significant impact on worker
performance. IDs must determine
whether the student will work alone or
as part of a team when demonstrating
what has been learned as a result of
instruction.

Will the student ultimately be
supervising a number of other
workers; will she be alone in the
performance of her work obligations;
will she be required to present to
audiences - these are among the
questions that need to be answered by
Ms. Heyward responded in her
Interview Questions Part II that I
assigned the following roles and
responsibilities of the Evaluation Team
Members:
IT & Data Management Director
Lead Evaluator Role
Research Awards Budget Manager
Data Collection and Data Analysis
Role
Senior Applications Analyst Data
Collection and Data Analysis Role
Research Awards Budget Assistant
Data Collection and Data Analysis
Role






15


Additional Information Requirements
Additional information from students who are enrolled or have been previously enrolled in
the Public Evaluation program is required to complete the Performance Context
Analysis. Online surveys could be conducted to obtain a fuller depiction from the students
perspective of what is required to complete an online course in Research Administration.


instructional designers.



IT & Administrative Support
Evaluation Support Role
Vice President for Research
Stakeholder/Advisor Role
Associate Vice President for
Research Stakeholder/Advisor Role
The implication for design and delivery
that come out of that response are
huge. A public program evaluation is
best served and created by someone
who has been taught how to bring out
the best and most cooperation from
team members and others working on
the same project. Instruction needs to
be geared to the skills and tools that
prepare students on how to interact
effectively with a wide range of other
workers.
Relevance of new skills
to workplace
Interviewed:
April
Heyward, a
student
enrolled in
PAD6327 in
2013

As research administrators,
evaluation is important in order to
measure work performance as it
relates to efficiency, training program
impact, and processes.

Ms. Heyward answered the question
how did PAD 6327 prepare students
for real world experiences? In the
following manner:
The Public Program Evaluation
course content prepared me to be able
to:
Develop a full proposal for funding
(i.e., project narrative, measurements,
indicators, evaluation plan)

Develop and evaluate a new or
existing program, service, or
technology

Work with faculty and staff principal
investigators in developing their
proposals for funding

Review proposals for funding as a
Peer Reviewer

Clearly the instructional designers of
Ms. Heywards course well knew the
implications for design, delivery and
evaluation. Based on her response,
her class was launched rather fully
prepared with skills, tools and a
sophisticated understanding of the
requirements of her field.

For any future evaluations of the
design and delivery of the course, it
would be logical to continue surveying
graduates and using their real world
experiences as guides to any
instructional changes.






16
Learning Context
Table 3 illustrates the four major categories of learning context analysis. These categories
include: number and nature of the sites, site compatibility with instructional needs, site
compatibility with learner needs, and feasibility for simulating workplace. These categories
provide information about the learning environment and the information obtained was
compiled by interviews conducted with the SME, Dr. Smith as well as with a survey that was
previously sent former student, April Heyward. Additionally implications for design, deliver
and/or evaluation are discussed.


Table 3: Analysis of Learner Context Analysis for PAD6327: Public Program
Information
Categories
Data
Sources
Performance Context Implications Design, Delivery and/or
Evaluation
Number and
nature of the
learning site
Interviewed:
Dr. Smith
Class size has never gone above 23 students
in the class.

Facilities: This class is an online class.
Student learning will take place on UCF
Webcourses. Students can work at their
speed. The learning process takes place
through Canvas.

Equipment: It is an online class, so the only
equipment required is a computer. Students
must have the ability to work on the computer
and have some information about
Webcourses. They also need internet access,
Microsoft office, Adobe, and different types of
video and audio players. Students also need to
have a Knights email account.

Resources: One textbook is required. A full-
time instructor monitors the students progress
throughout the course semester. The instructor
reviews weekly postings for student
understanding and provides timely feedback in
areas that students may be struggling in.

Constraints: Students may face issues
accessing and downloading the materials.
They may have some difficulties accessing
Webcourses during maintenance time.

Facilities: Since this class is an online
class, students need to be familiar with
Canvas and online learning. Due dates
and requirements are stated in the
syllabus.

Equipment: Students should have the
ability to work with the software needed
for this class.

Resources: All required material will be
posted on Canvas.

Constraints: The course syllabus
should state where students can obtain
access to the software required to
successfully access and download
course material. Additionally, 24-hour
notice from Webcourses about
maintenance scheduling would help the
students avoid submitting late
assignments.
Site
compatibility
with
instructional
needs
Interviewed:
Dr. Smith
Instructional strategies: Canvas is
compatible with many different instructional
tools that are used. The course uses traditional
modules, and videos.
Group functionality provides students with
access to a wiki doc function so students can
share and work on projects together online.
A discussion board is also required.

Delivery Approaches: Students have the
Instructional strategies:
Weekly postings are aligned with the
sections of the Evaluation Plan paper
due at the end of the course. Two exams
are given in the middle and at the end of
the course to measure student
comprehension and understanding of the
reading materials and concepts.

Delivery Approaches: Students should






17
ability to contact one another online using the
conference area, course emails, or the
discussion boards.
Groups are also formed to provide instructional
tools and resources for group projects. Most
students are in the same profession and have
been working together as a cohort over the
past 16 months.

Time: Students are given one semester to
complete the evaluation assignment for the
course.

Personnel: One instructor is available to
present the course.
be able to exchange ideas online and
use the appropriate method to manage
their groups.

Time: The syllabus indicates the
required timeframe for students to
complete all coursework, assignments,
and tests.

Personnel: Since there is just one
instructor presenting the class, a clear
syllabus needs to be provided.
Site
compatibility
with learner
needs
Interviewed:
Dr. Smith
Location: The learning portion of this course
occurs entirely through Canvas. All course
materials are accessed online. The course is
designed to serve students by being
completely asynchronous.

Convenience: Learners can access
Webcourses from anywhere they would like.

Space: Since it is an online class no physical
space required.

Equipment: The campus has many computer
labs for the students to access Webcourses.
Students are provided with multiple ways
to access the necessary materials
required to complete this instructional
unit as well as the course.
Feasibility for
simulating
workplace
Interviewed:
Dr. Smith

Supervisory Characteristics:
As research administrators, evaluation is
important in order to measure work
performance as it relates to efficiency, training
program impact, and processes.

Physical Characteristics: This course is a
service-learning course. Students are required
to work directly with a real organization to
develop an evaluation plan. They must interact
with stakeholders within the organization to
complete the final product.

Social Characteristics:
Students are expected to maintain
professionalism while working with
stakeholders to complete their evaluations.
Additionally, the course is designed for
students to complete the evaluation process
while working with an organization. Therefore,
the service-learning portion of this course will
prepare students for real-world applications.
Supervisory Characteristics:
Evaluation performance is often
conducted in research administration
offices annually and when special
projects are to be reviewed. For
example, training programs for
administrators or faculty, and evaluating
the impact of in-house grant programs to
encourage external funding
competitiveness for young faculty
researchers.

Physical Characteristics: Going to a
real organization to work will help the
students to develop an idea about the
evaluation plan. Further action from the
instructor is required to give some advice
from time to time.
Social Characteristics:
The social characteristics are easily
simulated, as the students are required
to work directly with an organization to
complete their evaluation. Students are
expected to maintain professionalism
throughout the evaluation process while
working with their selected organization,
which simulates a workplace.








18
Additional Information Requirements
Additional information from students who are enrolled or have been previously enrolled in
the Public Evaluation program is required to complete the Learner Context Analysis. Online
surveys could be conducted to obtain a fuller depiction from the students perspective of
what is required to complete an online course in Public Evaluation.































19
Appendix A: PAD6327 Student Interview Questions
This questionnaire was administered via an interview to students who have already taken
PAD6327 to gain a better understanding of student perceptions of the course.
1. Prior to taking PAD6327 Public Program Evaluation, how would you describe your entry
skills and knowledge of the topic - Public Program Evaluation; please address both course
content and delivery system.
2. Prior to taking PAD6327 how would you describe your educational and ability levels?
3. Would you please describe your general learning preferences during the time you were
taking PAD6327.
4. Would you please describe your attitudes toward both UCF and the organization you
evaluated during the semester you took PAD6327.
5. Would you please describe group characteristics of your classmates during the semester you
took PAD6327. This would include but not be limited to such areas as:

Subgroups of audience members (rural/urban,
grade/educational level, age and gender)
Background
Culture (social, economic)
Educational expectations
PAD6327 is the last in a
sequence of 12 courses.
Heterogeneity
Overall impressions

6. Would you please describe your academic motivation during the semester you took
PAD6327
7. Would you please describe your prior knowledge of public program evaluation at the
beginning of the semester you took PAD6327.
8. Would you please describe your attitude toward the content of PAD6327 before, during and
after taking the class?
9. Would you please describe your attitude toward the delivery system of PAD6327 before,
during and after taking the class?










20
Appendix B: PAD6327 Student Interview Questions Part II
This questionnaire was administered via an interview to students who have already taken
PAD6327 to gain a better understanding of student perceptions of the course.
1. In the workplace, how much time is given to complete a public research admin evaluation?

2. What are a Public Research Admin's job requirements?

3. Which employees do Public Research admin interact with while completing a program
evaluation? (stakeholders, managers?)

4. How did PAD6327 prepare students for real world experiences?

5. How many service learning hours were you required to complete for PAD6327?
























21








Appendix C: PAD6327 End of Course Evaluation Survey
This survey was administered to students who completed PAD6327. The purpose of this survey
was to provide feedback to the instructor about the overall course and instruction.

Q1 How well did your instructor meet the
course objectives?
Answ ered: 3 Skipped: 0




Comple
tely met
obj ectiv
e






Met the
obj ectiv
e w ell






Just
met the
obj ectiv
e






Barely
met the
obj ectiv
e






DId not
meet
the obj
ectiv e



0% 20% 40% 60% 80% 100%



Students will be able to explain major c onc epts of program evaluation,






22

Studients will be to apply the c onc epts in developing a prac tic al evaluation
proposal pape



Students w ill be able to explain maj or
concepts of program ev aluation,
Studients w ill be to apply the concepts in dev eloping
a practical ev aluation proposal paper.
Total
Respondents
Completely 100% 100%


4 met objec tive 2 2
Met the 100% 100%


2 objec tive well 1 1
Just met the 0% 0%


0 objec tive 0 0
Barely met the 0% 0%


0 objec tive 0 0


DId not meet the
objec tive
0%
0
0%
0


0






23







Q2 How relevant was the course material?
Answ ered: 3 Skipped: 0


Extremely w orthw hile



Very w orthw hile



Moderately w orthw hile



Slightly w orthw hile



Not at all w orthw hile


0% 20% 40% 60% 80% 100%



Answ er Choices Responses
Extremely worthwhile
33.33% 1
Very worthwhile
0% 0
Moderately worthwhile
66.67% 2
Slightly worthwhile
0% 0
Not at all worthwhile
0% 0
Total 3






24







Q3 How organized was the course content?
Answ ered: 3 Skipped: 0


Extremely organized



Very organized



Moderately organized



Slightly organized



Not at all organized


0% 20% 40% 60% 80% 100%



Answ er Choices Responses
Extremely organized
66.67% 2
Very organized
0% 0
Moderately organized
33.33% 1
Slightly organized
0% 0
Not at all organized
0% 0
Total 3






25







Q4 How knowledgeable in the course content was your
instructor?
Answ ered: 3 Skipped: 0


Extremely know ledgeable



Very know ledgeable



Moderately know ledgeable



Slightly know ledgeable



Not at all know ledgeable


0% 20% 40% 60% 80% 100%



Answ er Choices Responses
Extremely knowledgeable
66.67% 2
Very knowledgeable
33.33% 1
Moderately knowledgeable
0% 0
Slightly knowledgeable
0% 0
Not at all knowledgeable
0% 0
Total 3






26







Q5 How clearly did your instructor explain difficult material?
Answ ered: 3 Skipped: 0


Extremely clearly



Very clearly




Moderately clearly



Slightly clearly



Not at all clearly


0% 20% 40% 60% 80% 100%



Answ er Choices Responses
Extremely c learly
66.67% 2
Very c learly
33.33% 1
Moderately c learly
0% 0
Slightly c learly
0% 0
Not at all c learly
0% 0
Total 3






27




Q6 How concerned was your instructor with how well
students were learning?
Answ ered: 3 Skipped: 0


Extremely concerned



Very concerned



Moderately concerned



Slightly concerned



Not at all concerned


0% 20% 40% 60% 80% 100%



Answ er Choices Responses
Extremely c onc erned
100% 3
Very c onc erned
0% 0
Moderately c onc erned
0% 0
Slightly c onc erned
0% 0
Not at all c onc erned
0% 0
Total 3






28




Q7 How easy was it to contact and receive feedback from
your instructor for clarification or questions?
Answ ered: 3 Skipped: 0


Extremely easy



Very easy




Moderately easy



Slightly easy




Not at all easy


0% 20% 40% 60% 80% 100%



Answ er Choices Responses
Extremely easy
100% 3
Very easy
0% 0
Moderately easy
0% 0
Slightly easy
0% 0
Not at all easy
0% 0
Total 3






29




Q8 How clearly did your instructor explain how students
would be graded?
Answ ered: 3 Skipped: 0


Extremely clearly



Very clearly




Moderately clearly



Slightly clearly



Not at all clearly


0% 20% 40% 60% 80% 100%



Answ er Choices Responses
Extremely c learly
100% 3
Very c learly
0% 0
Moderately c learly
0% 0
Slightly c learly
0% 0
Not at all c learly
0% 0
Total 3






30




Q9 How fair was your instructor's grading?
Answ ered: 3 Skipped: 0


Extremely fair



Very fair




Moderately fair



Slightly fair




Not at all fair


0% 20% 40% 60% 80% 100%



Answ er Choices Responses
Extremely fair
100% 3
Very fair
0% 0
Moderately fair
0% 0
Slightly fair
0% 0
Not at all fair
0% 0
Total 3






31





Q10 Were you given too many assignments, too few
assignments, or about the right amount?
Answ ered: 3 Skipped: 0


Much too many






Slightly too many


About the right amount


Slightly too few







Much too few


0% 20% 40% 60% 80% 100%



Answ er Choices Responses
Muc h too many
0% 0
Somewhat too many
0% 0
Slightly too many
0% 0
About the right amount
100% 3
Slightly too few
0% 0
Somewhat too few
0% 0
Muc h too few
0% 0
Total 3






32







Q11 How useful were the assignments in helping you
understand research administration?
Answ ered: 3 Skipped: 0


Extremely useful



Very useful




Moderately useful



Slightly useful



Not at all useful


0% 20% 40% 60% 80% 100%



Answ er Choices Responses
Extremely useful
33.33% 1
Very useful
0% 0
Moderately useful
66.67% 2
Slightly useful
0% 0
Not at all useful
0% 0
Total 3






33






Q12 How easy were the assignments in this course?
Answ ered: 3 Skipped: 0


Extremely easy



Very easy




Moderately easy



Slightly easy




Not at all easy


0% 20% 40% 60% 80% 100%



Answ er Choices Responses
Extremely easy
0% 0
Very easy
0% 0
Moderately easy
33.33% 1
Slightly easy
0% 0
Not at all easy
66.67% 2
Total 3






34









Q13 How much did success in the course depend upon
understanding ideas, rather than memorizing facts?
Answ ered: 3 Skipped: 0



A great deal




A lot




A moderate amount



A little




None at all


0% 20% 40% 60% 80% 100%



Answ er Choices Responses
A great deal
33.33% 1
A lot
33.33% 1
A moderate amount
33.33% 1
A l ittl e
0% 0
None at all
0% 0
Total 3






35






Q14 Were you satisfied with the course content, neither
satisfied nor dissatisfied with it, or dissatisfied with it?
Answ ered: 3 Skipped: 0


Extremely satisfied


Moderately satisfied


Slightly satisfied






Slightly dissatisfied


Moderately dissatisfied


Extremely dissatisfied

0% 20% 40% 60% 80% 100%



Answ er Choices Responses
Extremely satisfied
33.33% 1
Moderately satisfied
0% 0
Slightly satisfied
66.67% 2
Neither satisfied nor dissatisfied
0% 0
Slightly dissatisfied
0% 0
Moderately dissatisfied
0% 0
Extremely dissatisfied
0% 0
Total 3






36






Q15 Were you satisfied with your instructor's teaching,
neither satisfied nor dissatisfied with it, or dissatisfied with it?
Answ ered: 3 Skipped: 0


Extremely satisfied


Moderately satisfied


Slightly satisfied






Slightly dissatisfied


Moderately dissatisfied


Extremely dissatisfied

0% 20% 40% 60% 80% 100%



Answ er Choices Responses
Extremely satisfied
66.67% 2
Moderately satisfied
33.33% 1
Slightly satisfied
0% 0
Neither satisfied nor dissatisfied
0% 0
Slightly dissatisfied
0% 0
Moderately dissatisfied
0% 0
Extremely dissatisfied
0% 0
Total 3






37

Q16 Which assignments were most useful?
Why?
Answ ered: 1 Skipped: 2



# Responses Date
1 It was helpful to work on the piec es of the evaluation planf 8/8/2013 8:32 AM






38

Q17 Which assignments were least useful?
Why?
Answ ered: 0 Skipped: 3



# Responses Date
T here are no responses.






39

Q18 Which course materials were most useful? Why?
Answ ered: 2 Skipped: 1



# Responses Date
1 PowerPoint and video lec tures explaining the evaluation plan. 8/8/2013 2:51 PM
2 instruc tor video and powerpoint were most useful - they explained the c onc epts in a prac tic al way,
as they needed to be applied to the evaluation plan
8/8/2013 8:32 AM






40

Q19 Which course materials were least useful? Why?
Answ ered: 1 Skipped: 2



# Responses Date
1 T he textbook was intense and stodgy. It was a lot to read, and not easy to c onc eptualize the
information in the way we would need it for our own plans
8/8/2013 8:32 AM




20

Q20 What are the strengths of this course?
Answ ered: 1 Skipped: 2



# Responses Date
1 I do think understanding evaluations is important, whether you are pre-, post-, or other area of researc
h administration. It is a key c onc ept to understand.
8/8/2013 8:32 AM




21

Q21 How could the instructor improve this course?
Answ ered: 1 Skipped: 2



# Responses Date
1 use the modules to better explain the parts of the evaluation plan / give the video & powerpoint
toward the beginning of the c ourse
8/8/2013 8:32 AM





Q22 Other comments.
Answ ered: 1 Skipped: 2



# Responses Date
1 T his c ourse presented interesting material, but I think it will probably be the least useful c ourse for
me in my role as a researc h administrator.
8/8/2013 8:32 AM