Training Best Practices:

Needs Assessment and Evaluation

Presented January 25, 2008 By Kristi L. Thompson, M Ed

Sierra Associates www.KristiThompson.com

Objectives
Participants will be provided the information and tools needed to conduct a basic Needs Assessment for course development. Participants will be provided the information and tools needed to conduct a Course Evaluation. Participants will be provided some tips on structuring budget requests.

Sierra Associates 2008

Learning is provided in order to improve performance on the present job. (Nadler, 1984)

Sierra Associates 2008

What have you done today to enhance (or at least insure against the decline of) the relative overall useful-skill level of your work force vis-a-vis competitors? - Tom Peters Thriving on Chaos

Sierra Associates 2008

To Train or Not
High

Problem: Low Motivation Method Assess personal consequences/ rewards system

Problem: Systemic

Job Knowledge

Method Consider system issues, problem is out of control of the employee

Problem: Bad Fit Method Consider improper placement of employee in the position
Low

Problem: Lack of Knowledge or Tools Method Training

Sierra Associates 2008

My Favorite Two

Sierra Associates 2008

Training Approaches
Intervention Approach Training is an intervention for solving problems involving employees. Focus is on performance and/or organizational results as corrections to problems. Steps taken when performance issue is identified. System Approach Training is a part of a continuous improvement process, fully integrated in the regular process of organizational improvement. Some steps done only once.

Sierra Associates 2008

Instructional Design Our Focus

Analyze Analyze system (department, job, etc.) to gain a complete understanding of it Compile an inventory of all tasks associated with each job (if needed) Select tasks that need to be trained Build performance measures for the tasks to be trained Choose instructional setting for the tasks to be trained Estimate what is going to cost to train the tasks Design Develop Implement Evaluate Review and evaluate during each phase (analyze, design, develop, implement) Perform evaluations Full ISD model in Appendix Revise training system based on evaluations
Sierra Associates 2008 8

What is an Needs Assessment?

Needs assessment is the systematic effort that we make to gather opinions and ideas from a variety of sources on performance problems or new systems and technologies. Allison Rossett (1987)

Sierra Associates 2008

Why do a Needs Assessment?

To make sure we are applying the right solution to the problem To identify what learning will be accomplished To identify what changes in behavior and performance are expected To determine the expected economic costs and benefits

Sierra Associates 2008

10

Steps of a Needs Assessment

Conduct Task Analysis and compile a Task Inventory Perform a Gap Analysis Select which tasks will be addressed Determine performance measures for the trained task Select Training Method Estimate training costs
Sierra Associates 2008 11

Task Analysis/Inventory
What The breakdown of performance into detailed levels of specificity Statements of
what will be done how it will be done for what end result Ex: Produce weekly summary Help Desk reports for all calls using a provided Excel template to be delivered to IT Director on Fridays.

Why To determine the operational components of a job, skill, goal, or objective To describe what and how they are performed To describe the sequence and scope
Task Inventory

When Whenever there are new processes or equipment, when job performance is below standards, or when requests for changes to current training or for new training are received
12

Sierra Associates 2008

Task Inventory Example

Department: IT Help Desk Analyst: Barty Crouch Job Title: Administrative Assistant Date: July 7, 2007 Department Supervisor: Susan Malfoy Job Code: 0742

Task Number and Task: 0742-1 Types orders received by mail, telephone, or in person at a minimum rate of 45 WPM into a computer database under general supervision of the Help Desk Manager to document calls. 0742-2 Receives and answers customer questions under close supervision of the Help Desk Manager in order to provide good customer relations. 0742-3 Delivers triage and delivery information received by mail, telephone, or in person under general supervision of the Help Desk Manager to fulfill customer requests. 0742-4 Acts as liaison between customer and various departments under close supervision of the Help Desk Manager in order to provide good customer satisfaction. 0742-5 Post and maintains records in a computer database without supervision to provide the department with historical records for statistical needs. .
Sierra Associates 2008 13

Gap Analysis Performance Behaviors

What Comparison of actual performance against new or existing standards Why To identify the performance gap between what is actually done and what is required or expected When

An intervention is required New processes and/or procedures New equipment/hardware New Applications New technologies Change in staffing Reduction in productivity Governmental mandates Security breaches Routinely, as part of continuous improvement process
14

Sierra Associates 2008

Gap Analysis Questions

For each identified task: Determine current performance level on task Determine desired performance level Task Current Performance Level Desired Performance Level

Produce weekly Emp. often makes Reports are fully summary Help Desk mistakes, missing info complete and accurate reports for all calls using approx 50% of the time every week a provided Excel template to be delivered Emp. repeatedly asks forEmp. is able to produce to IT Director on Fridays. assistance on producing report with no assistance report, particularly on recalculating Pivot tables

Sierra Associates 2008

15

Tasks Selection
What A determination of which tasks will be addressed in the training Directs course objectives Why To determine the scope and content of the training When

An intervention is required New processes and/or procedures New equipment/hardware New Applications New technologies Change in staffing Reduction in productivity Governmental mandates Security breaches Routinely, as part of continuous improvement process
16

Sierra Associates 2008

Tasks Selection
Required The task/topic is required by law or for safety Risk There is a high risk if task is not done correctly The task is critical Complexity Task is difficult or complex Task is done frequently Task is time-consuming Task is critical to the performance of the role/project Team Considerations Task requires coordination with other staff or with other tasks Task is part of a collective set of tasks Performance Task is required for acceptable role performance Task distinguishes star performers Task Selection
17

Sierra Associates 2008

Task Selection - example

Task Current Performance Level
Emp. often makes mistakes, missing info approx 50% of the time

Desired Performance Level

Reports are fully complete and accurate every week

Select - Criteria

Produce weekly summary Help Desk reports for all calls using a provided Excel template to be delivered to IT Director on Fridays.

Include Performance

Emp. repeatedly Emp. is able to asks for assistance produce report on producing with no assistance report, particularly on recalculating Pivot tables

Sierra Associates 2008

18

Select Training Format

What A determination of training delivery mode Why Identify the best tool for the job Determine training materials and format Begin to understand possible costs When For every training intervention or project Can be done at the task level or just generally
Sierra Associates 2008 19

Training Format Options

One-on-one/small group coaching Boot Camp small, very specialized audience Job Aid Classroom Vendor course OJT Mentoring CBT e-Learning Text-based Video Active Learning

Training Format flowchart

Sierra Associates 2008

20

Budget
What do They want? The greatest needs addressed with the least cost (ROI) Structure your requests by Projects supported (the bigger the better) Productivity increase expected (SWAG!) Number of employees to affected by a particular course/product (one product, many people) Problems solved with metrics

Training Costs Worksheet

Sierra Associates 2008

21

No money, no time, no resources

Step 1. Identify performance problems Method Questionnaire/Survey Focus group (see Lunchtime Assessment)

2. Select which tasks 3. Develop performance measures 4. Select method of instruction/setting 5. Estimate costs/budget

Use the most reported from the survey You decide based on time and resource limits for course Lecture/classroom

On location with internal SME/trainer

Sierra Associates 2008

22

Lunchtime Assessment
1. Gather a possible audience (Pizza works well.) Ask each person to write down their ten most important training needs in a particular area. Emphasize specific needs. A good format is I need to know how to 2. Capture the training needs on the white board or flip chart. Avoid duplicate answers by questioning. 3. Use a weighted voting process to prioritize the training needs across the group (sticky dots) to prioritize the list. Give half as many dots as there are items. Tell needs assessment participants to place their dots on the chart to vote for their priorities. 4. List the training needs in order of importance, with the number of points assigned as votes determining priority, as determined by the sticky dot voting process. A Step Beyond Have each person mark their top two needs on their lists of ten most important and then collect the lists. Share these Top Twos with the manager for personal development work.
23

Sierra Associates 2008

Basic Data Gathering

TYPE OF INFO SOURCES OF INFORMATION needed OBSERVATION INTERVIEW DOCUMENTATION

AUDIENCE Observe in work Interview/survey Review employee files

environment

audience or supervisors

or personnel records

TASKS

Observe Interview expert audience or or other expert performing performers

Review job descriptions, policy statements, and trouble reports specifications, and marketing guidelines

CONTENT Observe expert or Interview SMEs, Review product plans,

creators of policymakers, product/process marketers, or managers

Sierra Associates 2008

24

IT Training Needs Assessment?

IT Trait Obsolescence Process Modification Use vendor training for very specialized skills/tasks Define supported tools/apps/skills and limit training to these Define supported tools/apps/skills and limit training to these For static info develop documentation For active info consider the fastest Training method Identify audience groupings related to skills/knowledge/performance needed. Tweak content/materials for each audience. Train the Trainer use department IT stars
25

Enormous amt of possible subject matter: New products daily New services daily CHANGES daily! Entire campus as possible audience with different needs/levels/application for same tasks

Sierra Associates 2008

The linkage of Assessment and Evaluation

From Pamphlet 350-70-6 http://www-tradoc.monroe.army.mil/tpubs/pams/p350-70-6.pdf

Sierra Associates 2008

26

Evaluation Along the Way

Phase 1 - Analysis phase Determining if it really is a training problem Make sure there isnt already a solution available Phase 2 - Design phase Ensure the course content link directly to the objectives Phase 3 - Development phase Perform rapid-prototyping (implement-evaluate-implementevaluate-etc) or Have another training specialist check the solutions Phase 4 Implementation phase Use Level 1 evaluations (next slides)

Sierra Associates 2008

27

Four Levels of Evaluation

Reaction Learning Behavior or performance Outcomes or results

Sierra Associates 2008

28

Level One - Reaction

Measures and evaluates the learners reactions and opinions about training program itself Does not measure what new skills the learners have acquired or what they have learned that will transfer back to the working environment. Method - Attitude questionnaires after classes

COWC Level One

Level One Questionaire

29

Sierra Associates 2008

Level One - Questions to Ask

Rate the instructor Rate the course Estimate how much you have learned in the course Rate the effectiveness of the instructor in stimulating your interest in the subject Rate the effectiveness of this course in challenging you intellectually Minimally What major conclusion did you draw from today's session? What major questions remain in your mind?

Sierra Associates 2008

30

Level Two - Learning

Measures and evaluates the changes in the participants skills, knowledge, or attitudes as a result of the training: did they learn anything?

Methods Pre- and post-testing Questions should be directly linked to the selected tasks and the course content

Sierra Associates 2008

31

Level Three - Performance (behavior)

Measures and evaluates the transfer of the learning to the job or organization: can and do people use their newly acquired learning on the job? Methods Formal (testing or Manager interview) Informal (observation) Must take place after the learners have returned to their jobs

Manager Level 3 Evaluation

Employee Level 3 Evaluation

Sierra Associates 2008

32

Level Four - Results

Measures and evaluates the impact of the training on the productivity and profitability of the organization: what "return" has the organization received from the training? Methods Develop a "balanced scorecard with four perspectives Financial Customer Satisfaction Internal Perspective Innovation and Learning Linked to Vision/Mission statement on these aspects Allow time for the behavior change to take place Evaluate both before and after the program

Sierra Associates 2008

33

Balanced Score Card

Financial A measurement, such as an ROI, that shows a monetary return, or the impact itself, such as how the output is affected (Can be either soft or hard results.) Customer Improving an area in which the organization differentiates itself from competitors to attract, retain, and deepen relationships with its targeted customers Internal Achieve excellence by improving such processes as supplychain management, production process, or support process Innovation and Learning Ensuring the learning package supports a climate for organizational change, innovation, and the growth of individuals
Sierra Associates 2008
VinFin Balanced Scorecard example

34

Big Picture

Sierra Associates 2008

35

References
Rossett, Allison, (1987). Training Needs Assessment (Techniques in Training and Performance Development Series. Educational Technology Pubns (August 1987) Kirkpatrick, Donald, (1994). Evaluating Training Programs The Four Levels. San Francisco, CA: Berrett-Koehler Publishers, Inc. Multimedia Courseware Development Guide http://www.tradoc.army.mil/tpubs/pams/p350-70-2.doc

Writing a learning objective standard that measures Learning Objective (LO) performance. http://www.asat.army.mil/briefings/Learning%20objective%20standard%20writing.
Big Dog's Performance Coaching Page http://nwlink.com/~donclark/perform/coach.html Design Training Needs Assessment Surveys http://adulted.about.com/od/trngneedsasst/a/TNAsurveys.htm
36

Sierra Associates 2008

Resources
First Things Fast: A Handbook for Performance Analysis by Allison Rossett (Nov 6, 1998)

Job Aids and Performance Support: Moving From Knowledge in the Classroom to Kno by Allison Rossett and Lisa Schafer (Nov 3, 2006) A Handbook of Job Aids by Allison Rossett and Jeannette Gautier-Downes (Jun 15, 1991) Beyond the Podium: Delivering Training and Performance to a Digital World by Allison Rossett and Kendra Sheldon (May 23, 2001) Article What is a Balanced Scorecard http://www.balancedscorecard.org/BSCResources/AbouttheBalancedScorecard/tabid/ 55/Default.aspx

Digital Tools Survey

Sierra Associates 2008 37

Training Requirements

Sierra Associates 2008

38

Kristi Thompson is the Principal of Sierra Associates. She has over 20 years of experience in the development of stronger work environments. Kristi works with groups on workplace and relationship processes to redirect the focus and perspective, resulting in higher morale, focus and productivity. She has applied her expertise in the high-performance manufacturing, service industry, hi-tech companies, academic institutions and the medical field. She has presented on numerous topics such as Change, Conflict, Team Development and Quality Improvement. Kristi received her Masters of Education from the University of Massachusetts Amherst and her Bachelors of Science in Chemistry from Worcester Polytechnic Institute. Sierra Associates 8 Munn Road Monson, MA 01057 413 267 3585 Cell: 508 789 4112 www.KristiThompson.com sierra@samnet.net

Sierra Associates 2008

39

Data gathering methods 1

Method
Questionnaire/ Survey population

Pros
Can customize per audience Can be anonymous Can and should ask different style questions Efficient in collecting lots of data The easiest way of collecting needs assessment data. -Provides management/HR perspective

Cons
% return is often less than 20% Takes time and skill to develop well Directed/limited answers Should be limited to 10 minute completion time What people tell you and what they really do may differ Cant probe deeper May be difficult or impossible to obtain desired documents May require interpretation/ explanation

Documentation review (i.e., job descriptions, performance appraisals)

Interview managers/ Good for complex or supervisors undefined problems/areas Can be modified on the fly Personal touch
Sierra Associates 2008

Time and resource intensive

40

Data gathering methods 2

Method Observations Pros The most direct method of collecting needs assessment data Time efficient Cons Very resource consuming (time and people resources) Dependent upon the quality of documentation and honesty in documents Best used in conjunction with conversations with Mgr Scheduling issues Wide range of participants in attendance what people tell you and what they really do may differ Sometimes important to verify the results with observations and document analysis
41

Review performance appraisals of lesser performers Focus groups

Good for complex or undefined problems/areas Can be modified on the fly Personal touch Ideas can build on each other Relatively effective and efficient

Sierra Associates 2008

Steps for a full Analysis

Step Method Toolbox Template

1. Understand the clients business (Analyze system) 2. Conduct a Task Inventory*

Interview manager/department head See next slides

System Overview Instrument

System Overview example

Task Inventory
Task Inventory

3. Select which tasks will Discuss with be addressed manager/department head/client 4. Build performance measures for the trained tasks Review performance appraisals Interview manager/department head Interview manager/department head Ask HR/EAP

Task Selection
Task Selection

Task Performance Measure

Task Performance Template example

Sierra Associates 2008

42

Steps for a Needs Assessment

5. Select method Lockstep of instruction/ Classroom (In-house or Vendor) instruction Boot Camp setting Video Self-Paced Personalized System of Instruction (PSI) Computer Based Training (CBT) Text Instruction e-learning or Internet Distance Learning (IDL) Job Job Performance Aid (JPA) On-The-Job (OJT) 6. Estimate costs Training Method Decision Flow

Training Cost Worksheet

43

Sierra Associates 2008