Program Evaluation Project: Paradigm Health and Wellness Elite Leadership Program

Katherine J. Chartier

Tracy L. Yother

Purdue University
Chartier & Yother Paradigm Program Evaluation Plan 2

Executive Summary

Paradigm Health and Wellness is a local company with the aim of promoting holistic

health and wellness in a business and industry setting. Paradigm offers several programs

including: Wellness Coaching (SHIFT), smoking cessation, and Elite Leadership Development.

The Elite Leadership program is the newest program the company has created. During our

investigation, it was discovered that there are currently no plans for program evaluation. Given

the competitive nature of the company and their commitment to program growth, developing an

evaluation plan for the leadership program became our primary goal. To accomplish this goal,

we addressed the following questions:

1. What are the evaluation needs of the leadership program?

2. What evaluation models best fit the needs of the leadership program?

3. What recommendations and resources would help Paradigm create an

evaluation plan for the leadership program?

Interviews with the client, artifact evaluation of program brochures and the Paradigm

website, and an onsite observation of a program session were the primary methods of data

collection. Analysis consisted of document cross-comparison and resulted in the following main

themes: (1) there is no formal mission/vision statement for Paradigm and (2) the leadership

program currently lacks a program goal and individual lesson objectives. Thus, we determined

an objectives-oriented approach (Fitzpatrick, Sanders, & Worthen, 2011) to program evaluation

would be appropriate to help the company begin aligning goals with assessment strategies.

Given the results of our analysis and using Stufflebeam's (2003) CIPP model, we propose

a program evaluation plan for Paradigm which incorporates an objectives-oriented approach.

Other deliverables including the decision tree and example leadership plan will help Paradigm
Chartier & Yother Paradigm Program Evaluation Plan 3

have the necessary resources to create evaluation plans for the leadership program and

subsequent programs in the future. Our goal is to provide recommendations and resources

necessary for sustainable program evaluation planning at Paradigm.

Chartier & Yother Paradigm Program Evaluation Plan 4

Table of Contents

Introduction 6
Initial Client Meeting 6
Program Evaluation Details 8
Context 8
Purpose and Scope 9
Evaluation Questions 9
Information Sources 9
Resources 9
Methods and Techniques 10
Target Audience 10
Instrumentation 10
Project Management Tentative Plan 11
Data Collection 11
Onboarding meeting notes. 11
Artifact Evaluation 14
Observation – Lesson 3 Wheel of Life 14
Observation notes. 15
Data Analysis 16
Evaluation Report: Results and Recommendations 16
What are the evaluation needs of the leadership program? 16
What evaluation models best fit the needs of the leadership program? (CIPP Model) 17
Recommendations. 19
Context 20
Input 20
Process 20
Product 20
What methods would best meet the evaluation needs of the leadership program? 21
Additional Recommendations: Website 23
Paradigm Evaluation Plan – Decision Tree 25
Example Evaluation Plan: Elite Leadership Development (ELD) 26
Chartier & Yother Paradigm Program Evaluation Plan 5

Program Goal 26
Questions 26
Research Questions 26
Research Design 26
Approach 27
Data Collection 27
Data Analysis 28
Results and Recommendations 28
Dissemination and Use of Results 29
Appendix A: Website Analysis 32
Appendix B: Leadership Program Series Topic Outline. 39
Appendix C: Elite Leadership Development Brochure 41
Appendix D: Wheel of Life Handout 50
Appendix E: Meta Evaluation Survey 52
Appendix F: Memorandum of Understanding 55
Chartier & Yother Paradigm Program Evaluation Plan 6

Introduction

This program evaluation project was completed in cooperation with Paradigm Health and

Wellness as partial fulfillment of the requirements for EDCI 65000 Program Evaluation in

Career and Technical Education. This report will detail our evaluation process, including the

purpose and scope, evaluation questions, methodology, findings, conclusions, and

recommendations. Findings and recommendations were disseminated to the client on Tuesday,

November 14th, 2017 and to our class on November 29th, 2017.

Initial Client Meeting

An informational client meeting was conducted on Tuesday, August 19th. The overall goal

for our course program evaluation project was described and the needs for the project

(connection with stakeholders, data, etc.) were discussed. Paradigm’s representative Kathy

Saker, a coach and administrative assistant, expressed interest and committed to talk with Kristin

Stapleton, program developer. After email follow-up, Kristin responded affirmatively that the

company had a need and interest in our evaluation project. She responded on Wednesday,

September 13th to the following questions:

What is one problem or goal Paradigm would like to learn more about?

“We would like to evaluate the impact of our leadership series curriculum on employee

engagement and well-being – this could include improvements in overall health/health

risks.”

Why is this particular goal of importance/priority?

“We need to prove value to the client based on their needs to improve the performance

and outputs of their employees and lower their overall health costs.”
Chartier & Yother Paradigm Program Evaluation Plan 7

What factors would need to be explored to find a helpful solution

“Metrics being collected, metrics that could be collected, engagement surveys, well-being

surveys, etc., potentially some medical claim cost analysis.”

What data is already existing that needs to be analyzed?

“We have approx. 200 surveys to be analyzed now. We are working on the best solution

going forward. We have a completion survey as well.”

What data is desirable to gather?

“For clients with interest, we would hope to gather some internal data related to payroll –

absenteeism, lost days due to illness, family/medical leave, etc.”

Onboarding Meeting

A face-to-face meeting was held on September 19th with both Kristin and Kathy. We

(Chartier/Yother) learned about the organization and the programs they offer. Paradigm offers

three programs to local companies: Wellness Coaching (SHIFT), smoking cessation, and Elite

Leadership Development. The leadership program is their newest offering which begun in 2014.

This is the program with the greatest evaluation need in the company.

We talked about current evaluation plans. Currently, they are partnering with Bart

Collins, Ph.D. from the Brian Lamp School of Communication, Regenstrief Center for

Healthcare Engineering at Purdue University. They are working with Dr. Collins in order to

evaluate the SHIFT program.

The greatest benefit of this meeting was being able to hear about the company's aims and

vision. In preparing for this meeting Chartier explored the website and noticed there was not a

mission or vision statement. While the leadership had a strong vision for where they want to go,

it was discussed at the meeting that there is currently no formal mission for a vision statement.
Chartier & Yother Paradigm Program Evaluation Plan 8

As we discussed the needs of the leadership program, Kristin mentioned: "they didn't have a

good evaluation plan moving forward" (personal communication). This quickly became a goal

we felt was within our ability to adequately address given the timeframe and considering the

breadth of other evaluation and instrument development needs. Thus, our goal for this project is

to develop an informed, program evaluation plan for the leadership series. While this plan will

be focused on the leadership series, it is our aim to create a sustainable model or template that

Paradigm can use for future program evaluation needs. The final benefit from this initial

meeting was building rapport with our client, hearing their passion for what they do, and being

able to help them see the value of what we could do to help meet their needs.

Program Evaluation Details

Context

Paradigm Health and Wellness is a local organization that partners with business across

the area with the goal to improve the bottom line by improving health. They have 3 primary

programs: (1) a wellness coaching program (SHIFT) designed to reduce healthcare costs to

employers, (2) a smoking cessation course, and (3) the Elite Leadership Development course.

Elite is the newest course designed to hook clients with a shorter, more focused and cost-

effective version of their wellness coaching program targeted at leadership. This program is

intended to serve as a gateway to establish buy-in for further contracts for the wellness program.

The specific context of our evaluation is Paradigm. However, to get an idea of evaluation needs,

we are conducting a needs assessment for the Elite Leadership Development course (e.g.

attending leadership classes, and interviewing client/stakeholders). This will allow us to see how

the program looks in context and give us information to make evaluation recommendations.
Chartier & Yother Paradigm Program Evaluation Plan 9

Purpose and Scope

Purpose: To create an evaluation plan for the Elite Leadership Development course.

Scope: The leadership course is the newest program and does not currently have a plan for

evaluation. Our project will focus only on this one program and develop a strategic plan or

model for evaluation that can guide future evaluations or iterations of the ongoing evaluation.

Limitations in scope include not formatively or summatively evaluating the leadership program,

creating evaluation plans for any other program, or creating an assessment tool/metric for

evaluating the leadership program.

Evaluation Questions

1. What are the evaluation needs of the leadership program?

2. What evaluation models best fit the needs of the leadership program?

3. What recommendations and resources would help Paradigm create an evaluation plan for

the leadership program?

Information Sources

Information sources include the client (Kristin) and other stakeholders (Kathy), health

coaches, leadership program participants, the Paradigm website, evaluation experts (Bart

Collins) and other relevant academic research. In addition, our EDCI 650 course and textbook

provide relevant information.

Resources

Resources to aid in the effectiveness and completion of our project include: Dr. Greenan

and our EDCI 650 peers, Program Evaluation (Fitzpatrick et al., 2011), other relevant program

evaluation research, program evaluation experts (Bart Collins), the Paradigm team, instructional
Chartier & Yother Paradigm Program Evaluation Plan 10

materials and handouts from the “Wheel of Life” lesson of the leadership series, the Paradigm

website, and printing resources at Purdue.

Methods and Techniques

Target Audience

The audience for our specific project is the client, Kristin Stapleton, and other

stakeholders (coaches).

Instrumentation

An unstructured interview protocol was used at the client meeting on Tuesday, September

19th, 2017. While the research was done by the company via evaluating the website, the goal of

the meeting was to determine what the client identified as their biggest need. Thus, no

prescriptive questions were used during the meeting. Questions discussed resembled the

following:

 Tell us about the overall goal of your company.

 Describe the different programs you offer.

 Is there a formal mission statement for your company?

 What evaluation needs do you have and which are most important?

 What evaluations have been done in the past?

Chartier & Yother Paradigm Program Evaluation Plan 11

Project Management Tentative Plan

Mon Mon MonMonMonMonMonMonMonMonMonMonMon

10/16
10/23
10/30

11/13
Task

8/28

8/14

9/11
9/18
9/25
10/2
10/9

11/6
Task Name Start Date End Date

9/4
Duration
Preliminary Information 0 9/1/2017 9/20/2017
Informal Client Meeting 0 8/19/2017 8/19/2017
Phone Conversation 0 9/1/2017 9/1/2017
Develop Preliminary Documents 19 9/1/2017 9/20/2017
Data Collection and Analysis 25 9/20/2017 10/15/2017
First Formal Client Meeting 9/13/2017 9/13/2017
Receive Data 11 9/20/2017 10/1/2017
Observation 0 9/27/2017 9/27/2017
Type up notes from observation 14 9/26/2017 10/10/2017
Type up notes from interview 7 10/3/2017 10/10/2017
Analyze Data 13 10/2/2017 10/15/2017
Creating draft documents 22 10/3/2017 10/25/2017
1st draft of final document 8 10/3/2017 10/11/2017
Follow-Up Questions 8 10/17/2017 10/25/2017
Develop Preliminary Conclusions 7 10/16/2017 10/23/2017
Customer Deliverables 32 10/13/2017 11/14/2017
Draft documents 0 10/13/2017 10/13/2017
Walk-through video 0 10/27/2017 10/27/2017
Final Document Presentation 0 11/14/2017 11/14/2017

Data Collection

Data collection consisted of (1) initial client meeting notes (Chartier/Yother) (2) artifact

evaluation of the Paradigm website, leadership brochure, and leadership series content outline

and (3) field notes from leadership lecture series observation

Onboarding meeting notes.

Date: September 19th, 2017

Location: Paradigm Consulting

Participants: Kristin (Client), Kathy (Coach, Administrator), Chartier, Yother

Chartier notes. Paradigm – experts in behavior change. Holistic change model.

Programs: shift, quit smart, leadership. Education intervention. Product works well, participants

have no idea it was the product – internal change process – makes it hard to sell externally (and

evaluate). Qualitative – get the data – participants own the change themselves. Measurable
Chartier & Yother Paradigm Program Evaluation Plan 12

outcomes: lower medical cost, higher productivity, etc. Challenge: Hard to measure the benefit

of the leadership program for management – management wants quick change but behavior

change is not quick. Have worked with Bart Collins previously to evaluate SHIFT. Needs: to

educate the market, create a metric to measure performance, human performance improvement,

individual focus, and accountability. Target audience: athletes and executives. Leadership

serves as a gateway to SHIFT – should it? Goals: systematic change and empowerment. Is there

a vision/mission statement? Not a formal one. There used to be but not one now.

Yother notes. The meeting began with introductions and moved quickly into asking

questions. One of the first questions was on what programs do they offer. They offer three

programs Leadership program, Wellness, and Smoking cessation. Some of the programs are

offered onsite.

There are 4-5 employees at Paradigm. Kristin is the director of operations, Kathy is a coach.

There has not been a business case since 2010. Paradigm has offered wellness coaching since

2010.

They use Twine as an online coaching program/interface platform for SHIFT, the

wellness program, but not for the leadership program.

Paradigm coaches are behavior change experts in medical and clinical and look at wellness as a

holistic picture.

They feel their greatest challenge is their internal process. Creating a one size fits all

process is difficult because the individual owns their change. It's an internal process. The idea is

to change the way you think so you change the way you act.

Another challenge is because their experience is that until someone has experienced the

program themselves it is difficult to package and sell to others.

Chartier & Yother Paradigm Program Evaluation Plan 13

Paradigm can measure some improvements to clients after completing the Wellness

program.

 lower medical costs

 higher productivity

 lower absenteeism

 higher presentism

To clarify, their customer is not the individual, but the company.

For their wellness program, SHIFT, they have 36 months of collected data. Examples of

data include:

 medical costs

 self-report of results

 interviews with coaches

 surveys of clients

Their greatest success tied to outcomes is:

 behavior change

 goal progress

Their most common customers for SHIFT are companies or schools with at least 300

people.

The customers for their Leadership program are generally smaller and more progressive.

The leadership program meets in groups of 20-25 people, though they have had groups as

large as 40.
Chartier & Yother Paradigm Program Evaluation Plan 14

They have done a survey and follow-up on the leadership program. They found

improvement in:

 work/life balance

 nutrition

 stress

Paradigm is working with Bart Collins at Purdue on evaluating their SHIFT program.

We were given a copy of the preliminary results of his report.

Paradigm has questions on what and how to measure metrics for the leadership program.

Though they are open to using the Net Performer score.

The leadership program has a lower cost of investment for companies, and Paradigm

hopes to leverage success in the short-term leadership program into a longer-term commitment in

the wellness program, SHIFT.

Artifact Evaluation

During the initial client meeting, we were given a brochure and content outline for the

leadership program (see Appendix C). These two documents and the Paradigm website represent

the artifacts evaluated to determine what information is provided, and what kind of vision and

goals are established, formally or informally, for the leadership program.

Observation – Lesson 3 Wheel of Life

On September 27th, Chartier attended a leadership workshop held at Wabash Nationals

Learning Center. Lesson three: Wheel of Life was presented by Kathy and Kristin.

Unstructured observation focused on observing the relationship among group members and

Paradigm leadership, instructional strategies and techniques, and the overall impact of the lesson.
Chartier & Yother Paradigm Program Evaluation Plan 15

Observation notes.

As I walked in the participants were having a good time chatting, joking, and teasing

each other. It was clear right off the bat this team had a strong rapport and a good working

relationship. They knew each other’s likes/dislikes and referred to spending time together

outside of the work setting. Various members of the team were present from leadership/manager

to HR. I didn't catch all their roles but there was a diverse group, but you could tell Joe was the

leader.

As the training began Kristin and Kathy demonstrated good rapport with the group and

were at ease presenting. They used real-life examples to tailor the relevance of the content to the

target population. The format of the presentation was mostly lecturing with examples and

stories. One activity, see the wheel of life handout (Appendix D), was completed towards the

beginning of the lesson. This handout walked participants through the basic content of the lesson

while personalizing specific goals as individuals answered the questions on the handout. After

this activity, the remainder of the lesson elaborated the main points and focused on how

attending to each of the different elements of health (personal, career, social, etc.) has both a

professional and personal impact. At the end of the lesson, the instructors had each individual

review their handout and set one goal to work on. SMART goals were mentioned but there was

no formal goal setting – participants mostly reflected. There was brief discussion about goals

and one participant, who had seemed a little more distant than the others during the lesson,

shared how he was really committed to working on his goal – that this particular goal had been a

challenge for him for a while – but that he was inspired and motived to work towards improving

his health.
Chartier & Yother Paradigm Program Evaluation Plan 16

As the session ended I talked with Kristin about my thoughts. Given our recent focus and

finding there were no specific program goals, plus being an instructional designer, I mentioned to

her that clarifying goals for the program and objectives for each lesson was a top priority. I

mentioned I would follow up with her about those – but upon later reflection realized I needed to

contain my instructional design tendencies and keep a close focus on our goal – to create an

evaluation plan – not to conduct an instructional analysis.

Data Analysis

Comparing both evaluators notes after the first client meeting revealed the following

themes: (1) While no formal missing statement is presented, Paradigm's mission is focused on

improving business practices by increasing the holistic health and wellness of employees. (2)

The leadership program is new and currently does not have a plan for program evaluation

moving forward. (3) One goal is to use the leadership program as a gateway to longer-term

commitment to the SHIFT wellness program.

Analysis and comparison of the leadership series brochure, content outline, and company

website (See annotations in Appendix C) further corroborated our findings from client meetings

and observation. Results and recommendations are discussed in relation to our evaluation

questions below.

Evaluation Report: Results and Recommendations

What are the evaluation needs of the leadership program?

Results. Analysis of our meetings, observation, and analysis of artifacts revealed the

following findings. First, there is no vision or mission statement for the company. Second, there

is no existing evaluation plan for evaluating the program or assessing the desired outcomes.

Third, there is not an overall program goal or learning objectives for leadership series lessons.
Chartier & Yother Paradigm Program Evaluation Plan 17

Recommendations. First, create a mission or vision statement and add it to the company

website. All existing, or future, programs should strategically align to accomplish the mission of

the company. Second, use informed models (e.g. CIPP see model below) to create an evaluation

plan for the leadership program (see ELD evaluation example below). Third, create an overall

program goal for the leadership program based on the desired outcomes. Additional

recommendations include developing goals and objectives should clearly align with assessment

strategies. Make sure to differentiate the goals of SHIFT from the goals of the leadership

program. Determine what goals are unique to the leadership program and how you can

accomplish your goals via the instructional intervention. Consider doing an instructional

analysis to assess the needs of diverse learner populations, the efficacy of training, and what

additional instructional strategies would maximize the impact of the leadership program.

What evaluation models best fit the needs of the leadership program? (CIPP Model)

Results. Stufflebeam’s (2003) CIPP model (See Figure 1) would be an appropriate

model for Paradigm to use moving forward. It focuses on assessing the context and setting

goals, planning for instruction and evaluation, and taking action to improve instruction and

evaluation by conducting formative and summative evaluations. This model would produce the

high-quality results Paradigm is looking for.

As a formative assessment tool, each step of the evaluation should consider the following

questions:

 Context: What should be done?

 Input: How should it be done?

 Process: Is it actually being done?

 Product: Is it succeeding?
Chartier & Yother Paradigm Program Evaluation Plan 18
Chartier & Yother Paradigm Program Evaluation Plan 19

As a summative evaluation assessment tool, each step of the evaluation should consider

these related, but different questions.

 Context: Were important needs address?

 Input: Was the effort well guided?

 Process: Was service designed and executed as required?

 Product: Did the effort succeed?

Figure 1. CIPP Model

Recommendations.

An example of possible formative evaluation questions to be explored for the Elite Leadership
Program following the CIPP model could include the following.
Chartier & Yother Paradigm Program Evaluation Plan 20

Context

What should we do? Set objectives and determines needs.

 What is the goal of the program?

 Who is intended customer?

 What are the desired outcomes for the participants?

 What do the customers need?

 Are the customers the same as the participants?

Input

How should we do it? Create or improve plans.

 Design a program that meets the goals of the Elite Leadership program.

 Design a program that meets the needs of the participants.

 Design a program that meets the needs of the customer.

Process

Are we doing it as planned? Monitoring process.

 Summative (or formative) assess whether the program meets the program goals

and company mission statement.

 Assess whether the needs of the participants and customers are the same.

Product

Did the program work? Judge the accomplishments.

 Develop an on-going process to assess the quality of the program

implementation’s ability to meet the goals of the program.

 Develop a process to use the results to feedback and improve the program.
Chartier & Yother Paradigm Program Evaluation Plan 21

What methods would best meet the evaluation needs of the leadership program?

Results. From our experience with the leadership team, our analysis of artifacts, and

synthesis of findings, we recommend the following steps. First, objectives oriented approach to

program evaluation. The lack of goals and objectives from the top (company mission) down

(program and lesson level) was identified early on as an element of objectives oriented approach

that would work well to begin to develop an evaluation plan for Paradigm. Second, determining

what type of evaluation needs to be conducted is important (See Table 1). Finally, creating an

intentional, informed plan for program evaluation will help Paradigm accomplish the vision and

goals they have for their program and company.

Chartier & Yother Paradigm Program Evaluation Plan 22

Table 1. How to determine the type of evaluation.

Formative Summative

Use To improve the program To make decisions about the

program’s future or adoption

Audience Program managers and staff Administrators, consumers

By Whom Internal evaluators External evaluators

Major Characteristics Program improvement Information to enable

feedback so personnel can decision makers to decide to

improve it continue or adopt program

Design Constraints What information is needed? What standards or criteria

When? will be used to make

decisions?

Purpose of Data Collection Diagnostic Judgmental

Frequency of Data Frequent Infrequent

Collection During Program After Program

Sample Size Small Large

Questions Asked What is working? What needs What are the results; with

to be improved? How can it what training?

be improved? Under what conditions?

What audience?

What are the costs?

(Fitzpatrick, Sanders, & Worthen, 2011 p. 24)

Chartier & Yother Paradigm Program Evaluation Plan 23

Recommendations.

A brief overview of key decision points was created to walk the client through the

program evaluation process (see decision tree below). Then a more thorough evaluation plan

was created with examples pulled from existing data and our current evaluation (see evaluation

plan below). It is our hope that these two resources can serve as a program evaluation model

Paradigm can use for evaluating the leadership program and as a sustainable resource for future

program evaluation planning needs.

Additional Recommendations: Website

Recommendations for the website include adding the mission statement. Additionally,

there should be increased visibility on the website for the leadership program. Adding a link to

the Elite Leadership Program at the same level as the SHIFT Wellness Program adds more

opportunity for people to find the program on your website.

Chartier & Yother Paradigm Program Evaluation Plan 24

Strengths Recommendations

Material is clear, engaging, informative

Claims well supported by evidence

Several cues to action

Overview of SHIFT program demonstrates Determine how SHIFT and the leadership

benefits program are similar/different? What is the

unique goal of each? What are the desired

outcomes of each? Complement but

differentiate.

Implicit overall program goal: develop Make this goal more explicit

holistic strategies by enhancing 5 core areas Does the literature/SHIFT link holistic health

will create leaders who are engaged, resilient, to engagement, resilience, and productivity?

and productive. Determine how to measure each of these

constructs.

How do these constructs equate to strong

leaders/leadership?

How will these outcomes translate into

competitive advantage and exponential

growth?
Chartier & Yother Paradigm Program Evaluation Plan 25

Paradigm Evaluation Plan – Decision Tree

Mission and
Vision
Statement

Program Goal(s) Type of

and Objective(s) Evaluation

Research Evaluation
Question & Approach
Design

Data Collection

Data Analysis

Results

Recommendations
Chartier & Yother Paradigm Program Evaluation Plan 26

Example Evaluation Plan: Elite Leadership Development (ELD)

Program Goal

Create leaders who are engaged, resilient, and productive.

Questions

What is desirable to know?

Research Questions

Formative Are the leadership series lessons effective and engaging?

What activities do participants find most relevant in promoting change?

Summative Does participating in ELD have significant, long-term benefits for

participant’s levels of engagement, resiliency, and productivity? (Quantitative)

How do participants describe the impact of ELD on their levels of

engagement, resilience, and productivity? (Qualitative)

Research Design

Depends on overall goal and research questions.

Consider:

 Participants, are there significant differences in characteristics of participants between

sites? What are personal barriers to adoption of new habits/change?

 Context – how does the context support the intervention and behavior/attitudinal change?

What are the barriers to implementation within the context?

Chartier & Yother Paradigm Program Evaluation Plan 27

 Measures: Align measures with goal and research questions, and instructional

content/activities.

o For examples, if you want to measure productivity, resiliency, and engagement

the goals of the overall program, series content, and research questions should

target these constructs.

 Intervention: Education, training

o Holistic Wellness Topics: Career, Physical, Community, Financial, Social

o How do these topics align with the constructs you want to measure?

Approach

In our analysis one gap, we identified was there is not a clear overall program goal or

objectives for each lesson in the series. We recommend taking an objectives oriented approach to

program evaluation which includes aligning goals, objectives, and assessment strategies.

Data Collection

Will depend on research questions and type of evaluation (formative, or summative)

 Stage One: Pre-test, entry interviews for summative evaluation – does the program have

a measurable impact?

 Stage Two: Program evaluation - is the training effective?

 Stage Three: Post-test, exit interviews for summative evaluation – does the program have

a measurable impact?
Chartier & Yother Paradigm Program Evaluation Plan 28

Data Analysis

 Quantitative: Statistical Tests

 Qualitative: Coding of interviews or observations, analyzing artifacts

 Mixed Methods: Do participant self-reports corroborate or expand statistical findings?

Results and Recommendations

 Summative Evaluation: Program adoption, revision, or elimination

 Formative Evaluation: Training development

 Formative Evaluation: Content Development

Chartier & Yother Paradigm Program Evaluation Plan 29

Dissemination and Use of Results

In order to help the client understand the progress the program was taking, and the

deliverables they would be receiving, on October 27, 2017, a video was prepared and sent to the

client. This video walks the viewer through the results and recommendations of the report. This

was helpful to the client in understanding the progress and direction this evaluation was taking.

On Tuesday, November 14th, we presented our evaluation plan at Paradigm Health and

Wellness. In attendance were Kristen Stapleton, Kathy Saker, and Jim Gothard, owner. This

was the first meeting Jim had attended. He had also previously not participated in any email

discussions.

At that meeting, there was a review of materials and a brief hands-on workshop. The

workshop time was used to help the clients develop initial mission and goal statements. This

effort is to help the clients begin creating goals and an evaluation plan after the evaluation is

complete.

Results focused on practical program improvements and will not be disseminated as

research results, shared with other parties outside of the client and our class presentation, or

generalized to other programs.

Meta-Evaluation

At the end of the final report meeting with the clients, a brief survey evaluating our

process was given (see Appendix E). Results indicated the recommendations we provided were

helpful/valuable or extremely helpful/valuable. Further qualitative feedback indicated the

workshop time was the most helpful/valuable deliverable from our project. When describing the

workshop time participants cited it “left me wanted to do more of it” and “it really helped us start

putting all the pieces together and formalizing a great company mission.” When asked to
Chartier & Yother Paradigm Program Evaluation Plan 30

consider the overall process and what went well, participants described feedback from instructors

being helpful, communication, and tangible deliverables that could be used were cited. When

asked what could be done to make improvements, having time to work towards solutions and

involving more stakeholders were cited. In sum, the participants found our deliverables

meaningful and something tangible they can use to improve future program planning and

evaluation. While we experienced some constraints in communication and access to the client,

overall communication was effective which contributed in large part to our success. It was a

pleasure to work with a local company and be able to see our efforts result in real-world impact.
Chartier & Yother Paradigm Program Evaluation Plan 31

References

Fitzpatrick, J.L., Sanders, J.R., & Worthen,B.R. (2011). Program evaluation: Alternative

approaches and practical guidelines (4th ed.). Boston: Pearson.

Stufflebeam D.L. (2003) The CIPP Model for Evaluation. In: Kellaghan T., Stufflebeam D.L.

(eds) International Handbook of Educational Evaluation. Kluwer International

Handbooks of Education, vol 9. Springer, Dordrecht

Media Credit and Additional CIPP Resource

http://arcmit01.uncw.edu/jonesi/Evaluation.html
Chartier & Yother Paradigm Program Evaluation Plan 32

Appendix A: Website Analysis

Website Main Page

Chartier & Yother Paradigm Program Evaluation Plan 33

Website Home Page – Leadership Program

Chartier & Yother Paradigm Program Evaluation Plan 34

Claims

Implicit
program
goal

Cue to
action
Chartier & Yother Paradigm Program Evaluation Plan 35
Chartier & Yother Paradigm Program Evaluation Plan 36

Website What We Do – About Us

Chartier & Yother Paradigm Program Evaluation Plan 37
Chartier & Yother Paradigm Program Evaluation Plan 38

Goal of
SHIFT

How are SHIFT and

the Leadership
program
similar/different?
Chartier & Yother Paradigm Program Evaluation Plan 39

Appendix B: Leadership Program Series Topic Outline.

How does
this relate to
leadership?
Chartier & Yother Paradigm Program Evaluation Plan 40

Goal?
Chartier & Yother Paradigm Program Evaluation Plan 41

Appendix C: Elite Leadership Development Brochure

Chartier & Yother Paradigm Program Evaluation Plan 42
Chartier & Yother Paradigm Program Evaluation Plan 43
Chartier & Yother Paradigm Program Evaluation Plan 44
Chartier & Yother Paradigm Program Evaluation Plan 45
Chartier & Yother Paradigm Program Evaluation Plan 46

Desired
outcomes

Program
Objectives

Claims – how
will you
measure this?
Chartier & Yother Paradigm Program Evaluation Plan 47

Desired
outcome.
How will
you measure
and
evaluate?
Chartier & Yother Paradigm Program Evaluation Plan 48

Cue to
action.
Chartier & Yother Paradigm Program Evaluation Plan 49
Chartier & Yother Paradigm Program Evaluation Plan 50

Appendix D: Wheel of Life Handout

Chartier & Yother Paradigm Program Evaluation Plan 51
Chartier & Yother Paradigm Program Evaluation Plan 52

Appendix E: Meta Evaluation Survey

Chartier & Yother Paradigm Program Evaluation Plan 53
Chartier & Yother Paradigm Program Evaluation Plan 54
Chartier & Yother Paradigm Program Evaluation Plan 55

Appendix F: Memorandum of Understanding

Chartier & Yother Paradigm Program Evaluation Plan 56