2013

On the Job Training Project Evaluation

SCENTSY FAMILY
BRUCE YOUNGER

Summary
This evaluation is to determine the successfulness of a training program that was designed to enhance a new employees initial experience with a local company and the training experience that is delivered to the new employee within their first few days of employment. The evaluation is also expected to determine if those new training practices have been adopted by the current training team. In the past Scentsy Family has always used individuals in the companys operations group who are top performers and the most knowledgeable to train new employees. An issue that has been discovered is that these individuals while great at their respective positions are not necessarily the best at training. During needs analysis it was discovered that the trainers were using several different tricks of the trade to prepare new hires for their new jobs, in some cases trainers would unload so much knowledge that the new employee would become overwhelmed and in a few instances become frustrated and quit. The evaluation begins with interviewing key stakeholders and recipients of the new program, additionally there is a qualitative survey that will be completed by individuals who received the initial training. This survey is designed to find how the learners perceive the new OJT program is performing and how they are using it to train. It was also discovered that because the methods used to perform a job skill varied so greatly that the quality of performance was being negatively affected. The purpose for creating an OJT program was to create a standard method of training for each job skill in hopes that employee frustration would be reduced and production quality would increase. The purpose for evaluating the OJT program will be to discover if this training program will improving the operations area with a program that has not been tested. The evaluation will be used to discover if the program is helping employees reach their productivity goals and whether or not the program itself is meeting those objectives. There is a need to analyze whether or not the program is meeting its objectives using evaluation as a checks and balance for improvement is a need for the OJT process. If the program proves to be affective Scentsy would use the evaluation data to work toward improving the OJT process and to discover areas that need adjustment. The goal in this case for the evaluation tool is to determine whether and to what degree the objectives are being met. Additionally the evaluation will also be able to supply recommendations as to where adjustments need to be made in the OJT Program. This evaluation found several issues with the implementation and execution of the OJT program. After discussion with stake holders and subject matter experts, and review of quantitative data the evaluation found that the program was unsuccessful, primarily due to the timing of the program implementation. This finding does not intimate that the program itself was unsuccessful in design, given the proper environment the program has the ability to thrive. During discussions there were indicators that several components had been implemented with success.

Description of the program evaluated

Scentsy Family Operations OJT Program: In the past Scentsy Family has always used individuals in the companys operations group who are top performers and the most knowledgeable to train new employees. An issue that has been discovered is that these individuals while great at their respective positions are not necessarily the best at training. During needs analysis it was discovered that the trainers were using several different tricks of the trade to prepare new hires for their new jobs, in some cases trainers would unload so much knowledge that the new employee would become overwhelmed and in a few instances become frustrated and quit. It was also discovered that because the methods used to perform a job skill varied so greatly that the quality of performance was being negatively affected. The purpose for creating an OJT program was to create a standard method of training for each job skill in hopes that employee frustration will go down and production quality would increase.

Program Objectives
The Scentsy Family On-the-Job Training program was developed to bring continuity to the area of training for production operators and to establish a form of consistent training for operators specifically in the shipping area of operations. Future plans include deploying the program and its objectives to areas such as manufacturing and warehouse. The primary purpose of the OJT Program: To increase productivity- Instructions can help employees increase their level of performance on their present assignment. Increased human performance often leads to increased operational productivity and increased company profit. To improve quality- Better informed worked are less likely to make operational mistakes. Quality increases may be in relationship to company product or service or in reference to the intangible organizational employment atmosphere. To help a company fulfill its future personnel needs- Organizations that have a good internal program for development will have to make less drastic manpower changes and adjustments in the event of sudden personnel alterations. Personal growth- Employees on a personal basis gain individually from their exposure to educational expressions. Management development program seems to give participants a wider awareness, an enlarged skill and enlightens realistic philosophy and make personal growth possible.

Program Components
The OJT program has very few components. Initially there was the instructional design component where the program was planned and developed and eventually the training plan moved forward to delivery. The OJT Program was modeled after a training program designed and developed by the Boy Scouts of America (BSA). The Model is called EDGE and has four components:

EXPLAIN, what is going to be done, which is typically a trainerled activity. DEMONSTRATE the concept or skill correctly so the learner has a clear image in his or her mind of what success looks like. GUIDE the learner through the skill being learned. the learner gets fully engaged by working under the watchful eye of the trainer, who provides instant feedback to Guide him or her toward success. EMPOWER the learnergiving over control and supporting the learner by giving him or her a chance to fly solo. This means that the learner can successfully use the new knowledge and skills.

Initially the OJT Program was delivered in a classroom environment to current trainers and sectional supervisors. This was necessary to have management support the new model that would be deployed on the training floor. In the classroom environment items delivered along with facilitated training were: Construct a training module and deliver to facilitated classroom environment Observation check-of forms which had never previously been used Job Aids in the form of a wallet sized laminated cards that reinforced the EDGE model for training.

Evaluation Method
Participants o Dena Kinney Shipping Operations Manager Dena is responsible for the oversight of the entire shipping program. Dena is a key stakeholder in the development and implementation of the OJT Program o Kaya Refaey Shipping Department Manager Kaya is one of the individuals responsible for personal performance. Kaya is also one of the key players in getting the OJT Program created and launched. Kaya is considered one of the key stakeholders of the OJT Program o Kenna Clark Lead Instructional Designer Kenna oversaw the development of the OJT Program and was also a facilitator of one of the initial OJT courses that was delivered to the process trainers in the shipping department. As a lead on this project Kenna is considered a stakeholder. o The evaluation included three subject matter experts who are process supervisors from the shipping department that work on the ground level with trainers and trainees Chaurise Brost Day Shift Supervisor

Barry Baringer Swing Shift Supervisor (for new employees) Gary Wartchow Swing Shift Supervisor

o The evaluation included qualitative data from ten (10) shipping operators who are level three operator trainers that completed a survey designed to capture how the trainers are using the OJT program and how they feel about the implementation and use of the OJT program.

Procedures
During the evaluation of this program, it was difficult to get accurate data. The company ramps up its temporary staff to manage the seasonal surge in orders. After thoughtful consideration the evaluation moved focus from the order picking process where the majority of seasonal temporary worker are assign to the order packing stations where they moved full-time hires to work. Over all the evaluation was able to gain credible data in regard to the effectiveness of the OJT program because the staff that was moved from picking to packing were more familiar with how the company operates and are also familiar with the product being packed and can tell if the orders are correct or not. The evaluation considered data mined from order packing audits, a group discussion with stake holders and subject matter experts, as well as data gathered from a survey that was completed by 10 individuals who fulfill roles as process trainers and who had participated in the initial OJT Program training event.

Data Sources
The quantitative data that was provided for the evaluation is statistical data that is collected on each employee on a weekly basis. This data tracks the employees performance. The data collected will be a measure of performance prior to the implementation of the OJT program and a time period approximately two months after implementation. This data is collected by designated auditors on a daily basis and is a random sampling of 10 orders that have been packed at various times within the auditors eight hour shift. The qualitative data was provided via a survey that was delivered to each of the individuals listed in the Evaluation Method section under Participants. The survey was designed to gather information that would allow the evaluator to understand whether or not the OJT program is considered to be successful in the eyes of the management team that desired to implement the OJT Program. Finally the evaluator held a discussion session with the management group to discuss their observation of employee performance as a result of the OJT program. The information from that discussion will be shared in the Discussion section of this report.

Results
Quantitative Data (source Scentsy Family Order Packing Audits spread sheet for the year 2013) After receiving the quantitative data from the shipping operations manager, the data reveals exactly what management and the SMEs were expressing during the discussions with them. o The data tallies errors in the following areas: Missing Items Incorrect Items Incorrect packaging procedures o That data shows on the first glance that numbers of orders that were not packed correctly increased from September 2013 to November of 2013. The number if incorrectly packaged orders more than doubled from 11 failures in 128 audited packages to 30 failures in that time frame. o The data does however show that there was a lower number of orders packaged which makes the margin of error even greater. o The data however does not show however why there was a decrease in the number of orders nor the increase in errors. Management and SMEs discussed that there was a large increase in new seasonal employees. Qualitative Data (source Scentsy Family OJT Program Evaluation Survey conducted in November of 2013) o Link to OJT Experience Survey

Discussion
The Scentsy Family Corporation and the OJT program stake holders feel that the program has not been as successful as originally intended, as a result Scentsy Family contracted a third party evaluator to conduct an evaluation of the OJT Program. The evaluation discovered that while there are components of the program that are being used extensively, the OJT program itself has been unsuccessful in meeting the goals that were intended such as reducing picking and packing errors, in fact the quantitative data shows that errors dramatically increased. Qualitative data tells another story, one of trainers falling back into old training practices because it was easier to train the old way when the trainers were placed under pressure to train a large number of seasonal hires in a short amount of time. When analyzing the results of the survey of the packing operator level 3 trainers, the evaluator found that most of the trainers felt the implementation of the OJT program was not well timed. When comparing quantitative data in regard to the order packaging department, data shows that for each month since the implementation of the OJT program, packing errors have increased. The data shows two possible reasons for the increase of errors. First because of a large influx of seasonal employees the number of errors increased due to a

learning curve. However the OJT programs primary purpose was to provide better training so that errors would decrease. This did not happen. According to the qualitative data provided through Qualtrics Survey engine, it reveals that the trainers who were surveyed feel that the OJT program was unsuccessful based on this question: o Would you agree that the OJT program has been successful? Result, only one participant said they Agree Result, six out of ten either Disagree or Completely Disagree Result, three participants Neither agree or disagree As the evaluation progress and discussions, surveys, and other information began to compile, it has been noted that it is possible that the OJT program was not entirely and of its own design a failure. During discussions with stake holders and subject matter experts, it came to light that while everyone appreciates what the program can possibly do, they feel that when the program was implemented it fell victim to bad timing. The issue is that the OJT program was implemented a week prior to Scentsys busiest time of year as a result the trainers became overwhelmed by the need to train the nearly 80 seasonal employees the company had hired for the holiday season. One stake holder, Kaya Refaey remarked during a group discussion that we launched it (the OJT program) at the worst possible time (Discussions, 2013). Ms. Refaey goes on to say at first we thought this is tough we are not going to be able to do it (the OJT program). That said it explains some of the quantitative data that show a more than double increase in packing errors. The process was overwhelmed with new employees and as a result trainers as revealed in the OJT Survey 10 out of 10 trainers surveyed either agreed or completely agreed to the question that asked Is it easier to use the old ways of training when under pressure? Given this information the evaluation would have to agree that the underlying reason that the OJT program did not meet the needs of the stake holders was because the timing of introduction was not right. The quantitative data clearly shows that the new OJT program was unsuccessful due to many factors. The quantitative data showed a greater than 100% increase in packing errors. One item of note, only seasoned Level 2 operators are allowed to pack orders. The increase in errors was not due to new or seasonal employees, the errors were made due to issues with training. The qualitative data appears to echo the findings whether in discussions with the stake holders and subject matter experts or the results of the OJT survey; however, there are areas of data that indicate that the program can be successful. In order for a program to be successful there has to be early adoption, the qualitative data shows that there was not early adoption of the program. Scentsy has a training staff of 20 individuals within the order picking and packing department. Of those 20 individuals 50% were surveyed and of those trainers that participated in that survey, nine of them or 90% did not feel that the OJT program had been successful or were neutral in their response neither agreeing nor disagreeing.

While we can discuss what the data shows, there are indicators that the program can be successful if the timing of implementation was planned better. Even though not all facets of the program where adopted there are some areas that have been well received and are being used on a daily basis. A reintroduction of the OJT program at a less busy time may prove to help the program find success.

Project Cost
Budget and Fees (The budget and fees listed in for this evaluation are all inclusive and is based on standard government rates for travel and per diem rates) As with any project this budget are for fees for services rendered. Those fees are based on the skill level of those we intend to include in the evaluation of your project. Each individual is highly qualified in their respective areas and their compensation package reflects their level of expertise. Each person involved in the evaluation of the Scentsy OJT project were allotted a specific number of days to work on the project and each of those day is accounted for in a more detail Task Schedule. A schedule of fees is attached below. The evaluation includes a survey to collect qualitative data. The cost of the survey is included in the breakout of costs. Quantitative data is proprietary data provided by Scentsy Family Corporation and was used by permission from the listed stake holders. Additionally we have included fees related to on-site visits which include airfare, mileage for driving expenses off-site as well as a per diem which is intended to cover lodging and meals. Since this evaluation was local, there were no fees associated with per diem. Personnel Professional salaries: Bruce Younger M.E.T. 20 days at $500/day $10,000 TOTAL PERSONNEL Estimated miscellaneous mileage: 230 miles at $.54 per mile Survey/Analysis Qualtrics Survey TOTAL TRAVEL COSTS TOTAL BUDGET FOR EVALUATION $1,500 $10,000 $10,000 $124.20 $1,500 $124.20 $11,624.20

Appendices
Quantitative Data

Quantitative data supplied by Scentsy Family Corporation Order Packing Department Data represents number of packages audited in September when OJT program was implemented. The bright blue cells represent number of pass/fail audits performed each month. Septembers numbers indicate that 127 packages were audited and 16 failed. November indicates that 128 packages were audited and 30 failed. The data certainly indicates that the OJT program has failed to meet the needs of the department.

Figure 1: Quantitative Data suggests that training is not affective notice the bright blue cells

The qualitative data suggests that the OJT program was ineffective primarily due to the timing of implementation.

Qualitative Data

The evaluation employed a survey created using Qualtrics Survey software and server. The qualitative survey was provided to 10 trainers who are the primary users of the OJT program. The qualitative survey points out several issues with the OJT program ranging from effective use under pressure to pointing to time of implementation of the program.

Figure 2: Qualitative Data indicates that program is not being used effectively and that trainers are reverting to old habits rather than learning how to use the new training tool

Discussing OJT (2013). Scentsy OJT program Discussion - DR0000_0170.mp3 [Recorded by D. McKinny, K. Refaey, C. Brost, B. Baringer, G. Wartchow, & B. Younger]. Meridian, Idaho.