Training Analytics

Market Analysis:

How Do Companies Measure

Training and ROI?

© Bersin & Associates

Bersin & Associates

March, 2005
© Bersin & Associates and LTI Magazine 1 August, 2003
Bersin & Associates Market Research

Training Analytics: A Market Analysis

Date of survey: October, 2004

Surveys sent: 5,999
Respondent 67% Corporate
Profile: 18% Education,
15% Government

In October of 2004, Bersin & Associates conducted an on-line survey of nearly 6,000
subscribers to understand how they are currently measuring the effectiveness of their
training programs. Why did we choose this topic? We are keenly interest in
measurement practices because we believe that measurement, analytics, and
benchmarking are critical tools to help training organizations optimize their programs to
drive higher and higher business value.

In general what we found is that companies are keenly interested in measuring business
impact from training (Kirkpatrick Level 3 and 4) yet struggle with a clear lack of
methodologies and tools. Most companies spend 1%-3% of their overall training budget
on measurement but have a strong interest in spending more. LMS systems, while often
purchased for the purpose of measuring training operations, are falling short in providing
this capability.

Training Analytics Survey: Key Findings

1. Most organizations routinely measure training volumes (enrollments,

completions, hours delivered), Learner satisfaction, and total training costs.
These measures are the easiest to obtain and are widely used.
2. The majority of organizations consider the business impact of training to be
the most valuable information to obtain, but very few firms are actually
measuring it. The reason for this is a clear lack of tools and methodologies.
3. The top business impacts which companies measure are sales-related, quality-
related, retention, and overall cost reduction.
4. Most organizations are not satisfied with their Learning Management Systems’
ability to provide the requisite measurements and reporting information.
5. Spending on tracking and measurement is small today, but there is a strong
demand for higher spending and resource allocation in this area.

What do Companies Measure?

© Bersin & Associates 2 March, 2005

Bersin & Associates Market Research

The first question we asked was “what areas do companies measure today?” As the
chart shows below, training volumes (enrollments, completions, and student hours) are
the top measure, followed closely by satisfaction. This shows that companies still
benchmark their operation by “How much training did we do” and “how well did our
audience like our programs.” Unfortunately, these are not business-related metrics nor
can they be used to rank business impact across training programs.

When asked about measuring costs (an Efficiency measure), only about half of
companies routinely measure the cost of individual programs, and only about a quarter
actually compare costs between programs. Again we see an opportunity for
improvement here. Cost comparisons across programs can be valuable information in
scoring program efficiency and ROI. Companies that do not measure costs are clearly
not measuring ROI.

When asked about measuring true business impacts, the numbers are even lower. Only
14% have any regular way of measuring job impact (Level 3) and 11% measure business
impact (Level 4). There are several reasons for this. First, respondents clearly told us
that they are lacking tools and methodologies to perform these correlations. Second,
many organizations are not given the budget or resources to perform these types of
measurement projects.

Information Routinely Tracked

Business Impact 11%

Job Impact 14%
Cost comparisons 28%
Cost per student/hr. 39%
Student Scores 50%
Certifications/Compliance 52%
Total cost per program 57%
Total hrs. delivered 74%

Completions 76%
Learner Satisfaction 76%
Enrollments 82%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%

Figure 1: What information do companies track today? (All organizations)

Our Measurement Taxonomy

In our research in this area we have created a taxonomy or way of characterizing all the
possible things that can be measured. We group all measurements into three
categories:

© Bersin & Associates 3 March, 2005

Bersin & Associates Market Research

1. Effectiveness. How well does this program actually

impact learning, job results, business results, or cost Effectiveness
savings.

2. Efficiency. How cost-effectively did we reach the Efficiency

audience? What was our cost per student or student-hour
and how does it compare with other programs? Compliance
3. Compliance. Did we achieve our compliance targets? Who is in compliance and
how is not? What level of compliance did we achieve? What is our risk of falling out of
compliance?

You can read more about our measurement approaches in our whitepaper, Training
Analytics, the Time is Now.

Learning Results: Scores and Certifications

A critical goal of training is to improve human performance; hence one would expect a
very high percentage of firms would measure scores or other learning-related metrics.
This was not true. In our survey only about half of the respondents stated that they
routinely track scores for training programs. Our qualitative surveys show that sores
are used and tracked when the program has a “mandatory certification” nature.

Why is this? If you go to the trouble of training people, shouldn’t you always measure
whether or not they learned anything? There are many reasons why programs are not
scored: lack of infrastructure, the program did not have the budget, inability to decide
what to measure, or just lack of time. These are not “excuses” but rather real business
limitations. We would hope this number to go higher over time as more and more
technology is applied to training, but as described above, many training departments do
not have the budget or focus on this area.

There is also some confusion about whether assessments can adequately measure actual
learning and job performance. One participant commented, “Pre-testing has come and
gone because we are not clear that is best use of time and that it is accurate representation of
learning. We always have some degree of post-testing - as an effort to reinforce key messages
rather than to measure learning.” This comment indicates that scoring is more of a
motivational tool than a measurement tool.

Another aspect to scoring is the need to measure workers on-the-job. One participant
described his company’s technique in this way: “We regularly track observed performance
of skills. We are a high tech company and train technicians to fix and repair our products.
Physical skills are observed prior to certification.” As any experienced training professional
knows, demonstration of on-the-job skills is the most valuable way to truly test Level 2
learning.

Compliance or Certification Driven Programs

Many training programs are “compliance” or “certification” driven. 52% of respondents
told us that they regularly run these types of programs. We define these as programs
that must be strictly monitored for completion, where completion may be defined as
simply enrolling, actually completed, or passing a series of assessments. These programs
are often driven by actual learning needs – or may be driven by government regulation
needs.

© Bersin & Associates 4 March, 2005

Bersin & Associates Market Research

In some cases learning is not the driver, hence scores are not used. A good example
is sexual harassment training. If a company certifies that all its managers have
completed sexual harassment training, they can no-longer be sued for certain types of
sexual harassment claims. The score or learning results of this program are not
relevant to the business need. The business driver is simply to make sure that
everyone has completed.

Measuring E-Learning vs. Traditional Approaches

With the surge in e-learning over recent years, a number of firms are evaluating their e-
learning programs relative to their classroom-based programs. One participant
commented, “We track why students elect to take web-based training when classroom
training is available, the hours when students want to take web-based training, if the costs of
the web-based training were reasonable, and what element (trainer, presentation, graphics, etc.)
of the training was the most and least valuable.”

Analysis: This is a very healthy trend. In the early days of e-learning companies were just
happy to get courses online. We have reached the maturity level now where organizations are
starting to try to benchmark the effectiveness of their e-learning vs. other approaches, and often
find that e-learning does fall short. As our “Blended Learning: What Works™” study found,
companies are quickly finding that for certain high-impact programs, blended delivery is needed.

What Job Impact do Companies Want to Measure?

Very few firms, less than 15%, are measuring learners’ behavioral changes or the job
impact of training programs, and only one in ten measures the business impact of their
training programs on a routine basis. For those that do go to this level, what job
impacts do they measure?

Analysis: This problem is one of the big reasons why we are very positive and working hard to
help companies understand the value of “Training Analytics” systems, which make this problem
much easier to solve.

When asked “what types of business measures do you try to correlate to training, the
list was as follows:

Business Measures
Companies Want to Correlate to Training

Measure % of Organizations who

want to use It

Employee Turnover 11%

Employee Productivity 19%

© Bersin & Associates 5 March, 2005

Bersin & Associates Market Research

Sales Revenue 16%

Sales Cycle Time (time to close a sale) 10%
Time to Market 13%
Product or Service Quality 29%
Operational Cost Savings 24%

Figure 2: What business measures do companies want to correlate?

We were surprised by the number of comments from companies who want to measure
other business impacts. Some other areas include:

• Quality of service done the first time (rework)

• Service call surveys (customer satisfaction)
• Accidents (worker compensation incidents)
• Increased sales of trained VARs vs. untrained VARs (value-added resellers)
• Speed of answer (technical support).

Cost Savings Frequently Used as a Measure

As cited previously, costs are a primary concern and the most common way of assessing
training’s impact on the business. When asked about specific data being measured, cost
savings was most often cited. One participant explained his company’s measurement
activities:

“I do an annual survey to determine Level 3 (Job impact of training). In the same

survey, I ask managers/employees to tell me how the program has saved time/cost
and to be specific in terms of dollar amount.”

The Trap of Cost Reduction (Commentary)

The most popular measure of business performance we found (and continuously hear) is
cost reduction. “This particular program reduced costs by xyz.” If these cost
reductions are actual cost reductions in business operations, these are excellent
measures. However, if companies are trying to score their programs by the amount of
money saved on training, this becomes a trap.

Consider this. Most companies spend 1-3% of their payroll on training. If the training
organization saves 20 or 30% of this, the overall company savings will be a very small
fraction of overall company expenses. There is usually not that much budget in the
training organization to save. Moreover, most e-learning savings are a one-time effect.
The first year things look cheaper, but what do you do for the second year?

Ultimately if you really want to save money on training, you can eliminate it altogether.
We think people should seriously consider whether this is a valid measure at all. The
business impact of training is 10-100X higher when viewed as a performance enhancer,
so if you save $10 in training delivery but potentially lose $100-1000 in company
performance, the savings are clearly nil.

© Bersin & Associates 6 March, 2005

Bersin & Associates Market Research

Smaller Companies More Easily Measure Impact

Small to medium-sized organizations (those with less than 1,000 employees) were
somewhat more likely than large enterprises to measure the job or business impact of
training. Seventeen percent (17%) of Small to medium-sized organizations reported that
they routinely measure the job and business impact of their training programs. Just
eleven percent (11%) of large enterprises measures the job impact and only 5% measure
the business impact of training. We believe the reason for this is that smaller companies
just have an easier time obtaining data and talking with managers about what impact a
training intervention had.

Large enterprises find it difficult to obtain good data due to the complexity of their
internal structures and systems, as shown below. Some participants believe it is too
difficult to assess the business impact of training. As one individual commented:

“The only relevant measures are employee-specific. More general business measures
are confounded by too many other factors; it is and would be impossible to identify
the training impact, per se, except in the broadest overall way.”

A Clear Lack of Tools and Processes

We asked people what they wanted to measure more closely and what issues they face
that holds them back. The greatest “need” companies have is to understand how to
better measure business impact.

Why do so few do it (less than 11%)? Companies clearly lack the tools and processes
to make such measurement easy. 70% of respondents said that “they would measure
the business impact of training programs more routinely if they had better tools”, and
the same number said “they would be helped by having more formal processes or
methodologies in place.”

Needs for Measuring Business Impact of Training

Better process or 71%

methodology

Better measurement
tools 70%

Higher priority by
63%
mgmt.

More resources or
63%
budget
All Organizations
Better reporting
61%
system

0% 10% 20% 30% 40% 50% 60% 70% 80%

Figure 3: What do companies need to improve measurement processes and results?

Large enterprises were more likely than small to medium-sized organizations to cite a
lack of tools and processes in preventing them from measuring the business impact of
their training programs. We believe this is because small companies can more easily
talk to managers and obtain information on their audiences.

© Bersin & Associates 7 March, 2005

Bersin & Associates Market Research

LMS Systems Fall Short

We regularly study the Learning Management Systems (LMS) market, and recently
identified that “centralizing information about all my training investments” is now the #1
reason companies give for purchasing an LMS. Despite this need, the promise appears
not to be met.

While large enterprises are more likely to have an LMS in place, of the companies who
had an LMS, only 3% gave their systems top marks for being able to provide the needed
measurements and reporting. In fact, 22% rated their LMS “poor” in providing
reporting and measurement, and 39% rated their LMS “fair.” Our findings say that after
going to the expense of purchasing an LMS, only about a third of companies are satisfied
with the reporting and measurement solution they have obtained.

There could be many reasons for this and our whitepaper “Training Analytics: The
Time is Now” describes some of these in detail. We believe that this is an important
new area for LMS systems, and strongly recommend that companies learn about
Training Analytics before selecting or purchasing an LMS system.

Rating of LMS's Ability to Provide Necessary

Measurements & Reporting

Poor 22%

Fair 39%

Good 36%

All organizations
Excellent 3%

0% 10% 20% 30% 40% 50% 60% 70% 80%

Figure 4: How would you rate your LMS in its ability to provide you the measurement and
reporting you need?

© Bersin & Associates 8 March, 2005

Bersin & Associates Market Research

Demand for Greater Spending

Although measuring the impact of training is considered to be very valuable by most

firms, very few have put significant resources behind it. The majority (51%) of
participants who were knowledgeable of their departmental spending said they spend
less than 2% of their total annual training budget on tracking and measurement. Large
enterprises spend more than smaller organizations, but the spending is still extremely
limited.

Spending on Measurement
as % of total training budget

57%
less than 2%
44%

22%
2-5%
33%

18%
5-9% Small/Medium-Size
13% Organizations
Large enterprises
4%
10% or more
9%

0% 10% 20% 30% 40% 50% 60%

Figure 5: What percentage of your training budget do you spend on measurement and analytics
today?

Most participants recognize that more needs to be spent on tracking and measurement.
One in four said they should be spending much more on tracking and measurement activities
and half said they should spend slightly more. We see a latent demand for better analytics
solutions.

Conclusions and Analysis: Opportunity for Impact

The subject of training measurement is one filled with research, opinions,
methodologies, and case studies. Despite this body of knowledge, we find over and
over that our clients and research participants struggle with the lack of tools and
methodologies to measure, benchmark, and score their training investments. Hence in
most organizations, there is a tremendous opportunity to more effectively measure
training.

As you have probably read, “If you can’t measure it, you can’t manage it.” We are very
enthusiastic that companies understand the problems they face and are working hard to
improve this critical part of the training organization.

© Bersin & Associates 9 March, 2005

Bersin & Associates Market Research

Additional Resources Available

We have invested a lot of our research into this market, and have many resources to
help companies further understand the why’s, how’s, and what’s of training
measurements and analytics. These include our whitepaper “Training Analytics, The
Time is Now” and our recent 3-part webinar series on Training Analytics. Both are
available on our website by going to
http://www.bersin.com/research/analytics_measure.htm . If you become a research
subscriber, you can receive additional methodologies and detailed case studies on best
practices.

About Us

Bersin & Associates is a leading provider of corporate and vendor consulting services in
enterprise learning technology and implementation. With more than 20 years of
experience in e-learning, training, and enterprise technology, Bersin & Associates
provides a wide range of services including product development, product marketing,
industry research, corporate workshops, corporate implementation plans, and sales and
marketing programs. Some of Bersin & Associates’ innovations include a complete
methodology for LMS selection and application usage, an end-to-end architecture and
solution for e-learning analytics, and one of the industry’s largest research studies on
blended learning implementations. Bersin & Associates can be reached at
www.bersin.com or at (510) 654-8500.

© Bersin & Associates 10 March, 2005