1

Training Upgrade Initiative (TUI)

Evaluation Plan

Prepared by:

Lindsey Simmons
ECI 502 (601) Continuous Improvement of School Technology
Initiatives, Spring 2016

April 11, 2016

 2

1. Introduction
In the Patient Services department at Biogen, the new hire training program has
traditionally been 6 weeks long, with 90% of the required content facilitated in an on-site,
instructor-led format, and the remaining 10% facilitated online via self-paced eLearning
Modules. New hires are trained on customer service skills, product knowledge, how to use
multiple systems, problem-solving skills, and how to follow complex business processes in a
highly regulated call center environment. Patient Services’ trainers facilitate and oversee
the learning process for all Patient Services new hire classes. New hire classes are typically
hired 2-3 times per year, and each class is comprised of 15-20 new hires. Once new hires
successfully complete training, they report to a Supervisor, who then manages their
performance moving forward in their role in the department.

The Training Upgrade Initiative (TUI) was implemented for all 2016 new hire
classes. TUI incorporates use of enhanced learning methods, such as mobile learning, self-
study, use of Web 2.0 tools, and increased on-the-job training. The goal of TUI is to increase
training effectiveness by better preparing new hires for their role in the Patient Services
department. Table 1 shows a comparison of the traditional training model against the TIU
model.

Table 1: Traditional New Hire Training Model Compared to TUI Model

Traditional Model (prior to 2016) TUI Model (starting in 2016)

Product Training was taught using Product Training is taught using mobile learning courses in
Instructor-led training which the new hires review product information and
participate in polling and quizzing events (with use of tools
such as Poll Everywhere)

Systems Training was taught using Systems Training is taught by means of job observation,
both eLearning and Instructor-led followed by the new hire completing tasks in the system
training (with a mentor or Trainer available to give coaching and
feedback)

Customer Service Skills was taught
using instructor-led training
Customer Services skill training continues to be taught in an
instructor-led setting, but Web 2.0 tools (such as Voice
Thread and Evernote) are incorporated to allow increased
collaboration with peers and Subject Matter Experts on this
topic

Problem Solving Skills were
acquired during job observation
and role play/practice, which
currently take up approximately
10-15% of the entire course
Job observation and role play/practice take up 50% of the
entire course, with new hires allowed to increase problem
solving skills earlier and more frequently in training

Complex business processes, as
well as compliance guidelines,
were taught using Instructor-led
training, group activities, job
observation and role play/practice
Complex business processes are more heavily taught using
group activities, job observation and role play/practice.
These activities take place earlier and more frequently in
training. To ensure key learning objectives are met, use of
 3

Web 2.0 tools and mobile learning are used to assess

learning and allow for collaboration.

2. Overview of Evaluation
The evaluation of the Training Upgrade Initiative (TUI) will be completed at the end
of 2016 and will include data from three new hire classes (45-60 employees) who have
been trained according to the TUI method in 2016 as compared to three previous new hire
classes (45-60 employees) who were trained according to traditional methods in 2015.

The primary objectives of the TUI evaluation are to describe the overall
effectiveness and impact of the initiative, determine which learning methods were found to
be the most effective, and to identify areas for improving the New Hire Training program.

New resources, such as mobile learning technologies and Web 2.0 tools have been
funded by the department business leaders and are being used to implement the initiative.
Existing resources, such as the written assessment tool, learning management system,
online survey platform, and business metrics system will be used to evaluate the
effectiveness of the initiative.

Projected outcomes for the initiative include increased new hire confidence and job
readiness. Ultimately, as a result of the initiative, new hires will be better prepared for their
role after successfully completing training using the enhanced learning methodologies.

Figure 1 shows a Logic Model that diagrams the resources, activities, outputs,
outcomes and impact for the Training Upgrade Initiative.

Figure 1: Training Upgrade Initiative Logic Model

 4

3. Evaluation Design
Evaluation Questions

The Training Upgrade Initiative evaluation is guided by the following evaluation questions:

1. To what extent, if any, is the TUI model more effective than the traditional training
model?

2. In what ways, and to what extent, if any, did the TUI model improve the following
components of New Hire training?
o Product training
o Systems training
o Customer service skills training
o Problem solving skills training
o Training on complex business processes

Stakeholder Needs

The Patient Services training team will use the evaluation results in order to further
improve the New Hire training program. Based on evaluation results, the training team will
elect to continue or modify the enhanced learning methodology for the specific components
of New Hire training: product training, systems training, customer service skills training,
problem solving skills training, and training on complex business processes.
 5

The Patient Services business leaders will use the evaluation results to determine
whether to continue, stop, or expand funding for existing and new learning technologies,
such as mobile learning and Web 2.0 tools.

Evaluation Approach
This evaluation is being conducted using various approaches, including use of
qualitative and quantitative data. Quantitative methods will be used to analyze learning
outcomes such as written test scores, performance-based role play assessment scores, and
role metrics taken 30, 60 and 90 days after new hires have completed training and have
started in their role. Qualitative data will include survey feedback from new hires during
and after training as well as survey feedback from Supervisors and mentors after new hires
have successfully completed training.

4. Data Collection
Data Collection Methods
On-line written tests are administered for all courses within the New Hire training
program. In addition to written tests, new hires complete five comprehensive performance-
based role play assessments, which are administered by a mentor and scored on a rubric.
Test scores and role play assessment scores for new hires who have completed the Training
Upgrade Initiative will be compiled and compared to scores from new hires that have
completed the traditional training model.
In addition, role-based metrics are calculated 30, 60, and 90 days once new hires
successfully complete training. These role-based metrics are compared to new hires who
have completed the traditional training model as well as to current employees.
Online surveys are administered to new hires as well as to their Supervisors and
mentors. New Hires complete weekly online surveys to rate course effectiveness as well as
their confidence and job readiness. Once new hires have successfully completed training,
their Supervisors and mentors complete a similar online survey that rates new hire
confidence and job readiness.

Data Collection Method – Evaluation Question Link

To determine if the TUI model is considered more effective than the traditional
training model, qualitative and quantitative data will be collected using a variety of
methods. New Hires’ test scores, role play assessment scores, and role metrics taken 30, 60,
and 90 days after completion of training will be compiled and compared to results from the
previous traditional new hires classes from 2015.
Online survey feedback from New Hires, Supervisors and mentors will also be
compared to the results from previous traditional new hires classes from 2015. The New
Hire survey includes questions for which the New Hire can rate components of the New
Hire training (such as product training, systems training, customer service skills training,
problem solving skills training, and training on complex business processes). They also rate
their confidence and job readiness every week during training and upon completion of
training. In addition, the survey includes open-ended questions for which they can provide
general feedback and comments. The Supervisor and mentor surveys include questions that
also allow them to rate the New Hires’ knowledge and ability as it relates to specific
 6

components of the New Hire training (such as the New Hires’ product knowledge, ability to
use systems, customer service skills, problem solving skills, and ability to complete complex
business processes). In addition, the Supervisors and mentors rate New Hires’ job readiness
upon completion of training. Lastly, their survey also includes open-ended questions for
which they can provide general feedback and comments.
To determine in what ways, and to what extent, the TUI method improved the New
Hire training components, much of the same qualitative and quantitative data will be used.
However, the data will be looked at for individual components rather than comprehensively
for the entire program. For example, Product training test scores will be reviewed to
determine if scores are higher than for the traditional New Hire training methods taught in
2015. New Hire survey feedback on the Product training will be reviewed to determine the
learners’ reaction for this component of training and will provide guidance on ways to
further improve the learning methods that were used for this component of training.
Table 2 shows the evaluation questions and associated data collection methods.

Table 2: Evaluation Questions and Associated Data Collection Methods

Evaluation Question Data Collection Method

1. To what extent, if any, is the TUI model more
effective than the traditional training model?
Comprehensive (for entire program):
 New Hire test scores
 New Hire role play assessment scores
 New Hire role metrics taken 30, 60 and 90
days after completion of training
 New Hire survey feedback
 Supervisor survey feedback
 Mentor survey feedback

2. In what ways, and to what extent, if any, did For each component of New Hire training:
the TUI method improve the following  New Hire Test scores
components of New Hire training?  New Hire survey feedback
 Product training  Supervisor survey feedback
 Systems training  Mentor survey feedback
 Customer service skills training
 Problem solving skills training
 Training on complex business
processes
 7

5. Data Analysis and Interpretation

Indicators and Standards
When comparing the 2016 TUI new hire classes to the 2015 traditional new hire
classes, an increase in written test scores, performance-based role play assessment scores,
and role metrics will demonstrate that the Training Upgrade Initiative is effective.
In addition, a trend of more favorable New Hire, Supervisor, and mentor survey
results for the 2016 TUI new hire classes (as compared to the 2015 traditional new hire
classes) will demonstrate that the Training Upgrade Initiative is effective.
The criteria and standards that define success for the initiative are displayed in
more detail in Table 3.

Table 3: Indicators and Success

Evaluation Question
1. To what extent, if any, is
the TUI model more
effective than the
traditional training model?
Criteria or Indicator
When comparing average
New Hire test scores for 2016
(TUI model) versus 2015 test
scores (traditional model),
2016 test scores are higher
Standards
(What Constitutes
“Success”?)
An increase in average
New Hire test scores of at
least 5% or greater

When comparing average An increase in average

New Hire performance-based New Hire performance-
role play assessment scores based role play
for 2016 (TUI model) versus assessment scores of at
2015 role play assessment least 5% or greater
scores (traditional model),
2016 role play assessment
scores are higher

When comparing average 30, Average 30, 60, and 90 day

60, and 90 day role-based
New Hire metrics for 2016
(TUI model) versus 2015
metrics (traditional model),
2016 metrics are better
role-based New Hire
metrics are at least 5% (or
more) better

When comparing summative Overall trend for 2016

New Hire survey feedback for feedback is more positive
2016 (TUI model) versus
2015 survey feedback
(traditional model), 2016
feedback is more positive
 8

When comparing summative Overall trend for 2016

Supervisor survey feedback feedback is more positive
on 2016 New Hire
performance (TUI model)
versus 2015 Supervisor
survey feedback (traditional
model), 2016 feedback is
more favorable

When comparing summative Overall trend for 2016

mentor survey feedback on feedback is more positive
2016 New Hire performance
(TUI model) versus 2015
mentor survey feedback
(traditional model), 2016
feedback is more favorable

2. In what ways, and to When comparing each An increase in each

what extent, if any, did the component of (average) New component of (average)
TUI method improve the Hire test scores for 2016 (TUI New Hire test scores of at
following components of model) versus 2015 test least 5% or greater
New Hire training? scores (traditional model),
 Product training each component’s 2016 test
 Systems training scores are higher:
 Customer service  Product training test
skills training scores are higher
 Problem solving  System training test
skills training scores are higher
 Training on  Customer service skills
complex business training test scores are
processes higher
 Problem solving skills
training test scores are
higher
 Complex business
processes test scores are
higher
 9

When comparing New Hire Overall trend for 2016

survey feedback on each feedback is more positive
component of New Hire
training for 2016 (TUI model)
versus 2015 survey feedback
(traditional model), 2016
feedback is more positive:
 Product training feedback
is more positive
 System training feedback
is more positive
 Customer service skills
training feedback is more
positive
 Problem solving skills
training feedback is more
positive
 Training on complex
business processes
feedback is more positive

When comparing Supervisor Overall trend for 2016

survey feedback on 2016 New feedback is more positive
Hire performance on specific
components of New Hire
training (TUI model) versus
2015 Supervisor survey
feedback (traditional model),
2016 feedback is more
favorable:
 New Hires’ product
knowledge is rated higher
 New Hires’ ability to use
the system is rated higher
 New Hires’ customer
service skills are rated
higher
 New Hires’ problem
solving skills are rated
higher
 New Hires’ ability to
navigate complex
business processes is
rated higher
 10

When comparing mentor Overall trend for 2016

survey feedback on 2016 New feedback is more positive
Hire performance on specific
components of New Hire
training (TUI model) versus
2015 mentor survey
feedback (traditional model),
2016 feedback is more
favorable:
 New Hires’ product
knowledge is rated higher
 New Hires’ ability to use
the system is rated higher
 New Hires’ customer
service skills are rated
higher
 New Hires’ problem
solving skills are rated
higher
 New Hires’ ability to
navigate complex
business processes is
rated higher
 11

Analysis
Quantitative data will be analyzed using statistical methods necessary to determine
comparison of average scores and metrics. Table 4 and Figure 1 display how the
quantitative data will be summarized and displayed, upon completion of the 2016 TUI New
Hire classes.

Table 4: Quantitative Data Analysis of Scores and Metrics

Test Scores 2015 New Hires 2016 New Hires Comparison (%)

Product training test

score average
Systems training test
score average
Customer service
skills training test
score average
Problem solving skills
training test score
average
Training on complex
business processes
test score average
Total average test
score

Performance-based 2015 New Hires 2016 New Hires Comparison (%)

Role Play Scores
Role Play #1 average
score
Role Play #2 average
score
Role Play #3 average
score
Role Play #4 average
score
Role Play #5 average
score
Total average role play
score

Role-Based Metrics 2015 New Hires 2016 New Hires Comparison (%)

Average Calls per

Staffed Hour at 30
days
 12

Average Calls per

Staffed Hour at 60
days
Average Calls per
Staffed Hour at 90
days
Average Call Handling
Time at 30 days
Average Call Handling
Time at 60 days
Average Call Handling
Time at 90 days

Figure 2: Quantitative Data Analysis of New Hire Survey

6

2
2015 Average
1 2016 Average

0
Product Systems Customer Problem Training on
Training rated Training rated Service Skills Solving Skills Complex
as Somewhat as Somewhat Training rated Training rated Business
Effective (or Effective (or as Somewhat as Somewhat Processes
higher) higher) Effective (or Effective (or rated as
higher) higher) Somewhat
Effective (or
higher)

Qualitative data will be analyzed by looking for patterns and themes in survey
feedback. Figure 2 shows how the quantitative data will be summarized and displayed,
upon completion of the 2016 TUI New Hire classes.

Table 5: Qualitative Data Analysis of Surveys

New Hire Survey 2015 New Hires 2016 New Hires

Positive Trends

Negative Trends

Average Rating on Confidence

 13

Average Rating on Job Readiness

Supervisor Survey 2015 New Hires 2016 New Hires

Positive Trends

Negative Trends

Average Rating on Confidence

Average Rating on Job Readiness

Mentor Survey 2015 New Hires 2016 New Hires

Positive Trends

Negative Trends

Average Rating on Confidence

Average Rating on Job Readiness