Measurement and Evaluation in Corporate Universities

By Mark Allen, Ph.D. and Philip McGee, Ed.D.

Although corporate universities in America can be

traced back to Pre-World War II times, the phenomenon truly
exploded in the 1990s. Among the reasons cited for the
proliferation of corporate universities was dissatisfaction
with the education that was provided in both degree and
non-degree programs by traditional universities. Yet many
corporate universities have adopted some of the trappings
of traditional universities: deans, course catalogues,
numerous schools within a university, and even a few
sprawling campuses. Yet one phrase from traditional academe
has not worked its way into corporate university jargon:
institutional research.
Many corporate universities do, however, invest
considerable resources in measurement and evaluation.
Corporate training managers have a long history of
measuring the effectiveness of training programs, and
corporate university deans and directors have devised
numerous clever and creative means for evaluating the
overall value of a corporate university.
In this chapter, we will first define corporate
universities and discuss how they differ from both

traditional universities and traditional training

departments. The rest of the chapter is devoted to a
discussion of a variety of methods that are used to measure
both individual programs as well as entire corporate
universities.
Definition of Corporate Universities
Just as many state colleges underwent name changes in
the past two decades and became universities, corporate
colleges, as they were called in the 1980s (Eurich, 1985)
transformed into corporate universities by the 1990s.
Entities that called themselves corporate universities
ranged from very large organizations (such as Motorola
University) to small training departments that sought
greater status through a name change. For many years, there
was little agreement about what constituted a corporate
university. Many companies (such as General Electric)
avoided using the term, and in Europe, where the phenomenon
is growing rapidly, the term is almost never used
corporate academies is the preferred appellation, and the
word university in many countries is exclusively reserved
for degree-granting institutions (Renaud-Coulon, 2002). In
this chapter, we define a corporate university as,
an educational entity that is a strategic tool
designed to assist its parent organization in

achieving its mission by conducting activities that

cultivate individual and organizational learning,
knowledge, and wisdom (Allen, 2002, p. 9).
While the definition stipulates that corporate
universities are educational in nature, the most important
word in this definition is strategic. A corporate
university, in order to fit this definition, must be truly
strategic in its intent and activities. As the definition
implies, the corporate university exists to help its parent
organization (and a corporate university, by definition, is
not a stand-alone entityit is a part of a larger
organization) achieve its mission. The activities of a
corporate university are related to the overall strategy
and mission of the organization it serves.
The definition does not specify what sort of
activities a corporate university undertakes. The most
frequently asked question we receive is how does a
corporate university differ from a training department?
The best answer is that a training department does
training, while a corporate university does training, plus
many other things. Among the activities that corporate
universities are responsible for or assist with are:

needs assessment

managerial and executive development

the design, delivery, and management of elearning programs

internal and possibly external marketing of

programs

university partnerships

strategic hiring

new employee orientation

career planning

succession planning

knowledge management

research and development

culture change

strategic change, and, of course

measurement and evaluation.

This list goes well beyond the scope of most training

departments. While training is generally highly tactical,
this list of corporate university activities is much more
strategic.
The final piece of the definition discusses
individual and organizational learning, knowledge, and
wisdom. The first key distinction is between the
individual and the organization. While learning activities
at traditional universities are generally focused on
individuals, learning in corporate universities is designed

to benefit both the individuals in the classroom and the

organization itself. As the concept of the learning
organization (Senge, 1990) grew in popularity throughout
the 1990s, organizations began focusing their efforts on
organizational learning. Of course an organization cannot
learn unless the people that comprise the organization are
learning, so corporate universities must focus on both
individual and organizational learning. This is a crucial
distinction between corporate universities and traditional
universities, where the main locus of educational efforts
is individual students. While much of the classroom and elearning activity in corporate universities is directed at
individuals, it is important to remember that the raison
dtre of corporate universities is organizational
betterment.
Learning is a process of changing, both for
individuals and organizations. Usually, that change process
involves the acquisition of knowledge, which may be
declarative (facts, or things you know) or procedural
(skills, or things you know how to do). The final piece of
the definition, wisdom, refers to the intelligent
application of knowledge. The organization does not benefit
if knowledge is acquired, but not used. We will elaborate
on this concept in our discussion of behavior change and

organizational impact below, but without the wisdom to

apply knowledge for positive results, the organization does
not receive any benefits from learning for its own sake.
This distinction marks another area of difference
between corporate and traditional universities. Academe has
long valued learning for its own sake, but that is not part
of the strategic nature of corporate universities.
This leads to another important distinction between
corporate and traditional universities: degree programs.
Nowhere in our definition or list of corporate university
activities does the granting of degrees appear. There are
thousands of accredited degree granting universities in the
United Statescorporate universities do not need to get
into the business of establishing degree programs. As
corporate universities proliferated into the hundreds and
perhaps thousands over the past decade, the great fear in
the halls of traditional academe was that corporate
universities would encroach on their turf. It turns out
that corporate universities had a very ambitious to-do
list, but the granting of degrees was not on that list. In
2000, Gordon Thompson went in search of corporate
universities that were granting degrees. In a study of
corporate universities in the United States and Canada, he
found fewer than ten corporate universities that were

granting degrees in a population that was estimated to

exceed 2000 entities that called themselves corporate
universities (Thompson, 2000).
The other notable omission from the definition of
corporate universities is the term corporation. While
most corporate universities are in the for-profit world,
there are numerous corporate universities that are not
corporate at all--they exist in not-for-profit
corporations, municipalities, and governmental agencies.
Traditional Training Measures: Kirkpatricks Four Levels
Since corporate universities are direct descendants
from training departments, it is not surprising that the
basic model that many corporate universities use for
measurement is the Kirkpatrick Four Level Model
(Kirkpatrick, 1994) that has been favored by training
departments for more than four decades. Developed by Donald
Kirkpatrick as his doctoral dissertation in the late 1950s,
this model still serves as the basic template for
measurement in many training departments and corporate
universities.
In Kirkpatricks model, the first level of measurement
is reaction. Immediately after a training program,
participants are asked for their reactions to the classdid
you like the teacher? Did you find the content useful? Were

the meals tasty and the chairs comfortable? While this type
of measurement does not yield too much in the way of indepth data, it does have the advantage of being inexpensive
and easy to administer.
Of course the whole point of education is to create
learning, and Level One evaluation does not measure
learning. Kirkpatricks second level addresses learning.
Learning can be measured in a variety of ways: short answer
tests, essays, demonstrations, and even on the job.
A major difference between traditional and corporate
universities is in the area of measuring the
accomplishments of students. While most traditional
universities will measure using Kirkpatricks first two
levels, that is where they stop. It is customary after a
college course for students to fill out surveys about the
course and the professorin Kirkpatricks world, this is
Level I evaluation. College courses also attempt to measure
student learning, both throughout the course and upon its
completion. But most university evaluation ends when
individual courses end. Kirkpatricks final two levels
extend over periods of time after the completion of the
course. Measurement that occurs months or years after a
course ends is generally beyond the scope of traditional

universities, but is at the core of meaningful evaluation

in corporate settings.
The third level in Kirkpatricks taxonomy is behavior
change. It is not enough, in corporate learning
initiatives, to merely measure learning. Employees need to
do something differently in order for that learning to have
relevance for the organization. Learning without behavior
change is merely learning for its own sake, which is
generally not the province of corporate universities. In
order to measure behavior change, employees behaviors need
to be tracked prior to the training so a baseline is
established. Then, post-training, behavior needs to be
observed in order to establish the extent of change.
Measurement of behavior change, especially across a large
number of students, is not a simple task and requires a
high level of commitment by the organization and an
investment of time and energy.
Kirkpatrickss fourth level is results or
organizational impact. Not only do you want learning and
behavior change to happen, you also want those changes to
yield positive results for the organization. While the
locus of measurement for learning and behavior change is
typically the individual employee, results are typically
measured at the organizational level. There is often a

lengthy lag between the time of the training and the

measurement of results. Moreover, the organization needs to
determine what sort of results it is looking for prior to
the training (always a good idea), and track correlation
over time. Needless to say, causality is tricky here, but
most organizations that aspire to Level IV measurement (and
not all do), are satisfied with a correlation between
training and positive organizational results.
ROI and Human Capital Measures
While Kirkpatrick ended his model at Level IV, Jack
Phillips has done considerable work in the area of
measuring Return on Investment (ROI) for training programs
(Phillips, 1997). ROI has sometimes been called Level Five
evaluation. Like standard ROI measures, Phillips work
involves examining the ratio of costs to benefits. However
in the world of corporate universities, costs are not as
easily measured as most areas, and the quantification of
benefits can be extremely tricky.
Direct costs for a training class consist of payment
to the instructor, the cost of the room, course materials
and handouts, and catering. However the company also incurs
a cost for having a number of employees out of the
workplace. Most companies figure this cost as the salary

plus taxes and benefits for each worker in the room for as
many days as the class lasts.
Calculating the benefits is even more problematical.
First, the company needs a pretty clear picture of what it
is trying to accomplish. Then, it needs to figure out what
a successful outcome would look like and how to measure it.
These calculations can be quite complicated and timeconsuming. Many organizations find ROI to be desirable, but
too complicated or costly to calculate. Others have
successfully implemented ROI programs.
TVA University, at the Tennessee Valley Authority, has
developed a method for calculating the ROI of each of the
hundreds of courses it offers each year. After calculating
costs through some pretty standard measures, TVA University
then calculates benefits by measuring the extent to which
each worker uses the material taught in the class in the
course of his or her job, the amount of improvement in that
area as a result of the course, and the workers value to
the organization (as measured by annual salary). Needless
to say, some of the measures are hard to get at, but the
statisticians at TVA University have devised ways of
measuring each of these elements and the university can
assess the ROI for each of its courses.

Motorola University uses human capital as the locus of

measurement at the heart of its evaluation methods (Barney,
2002). Eschewing satisfaction measures and volume measures
(such as the number of student days of training provided)
Motorola Universitys measurements focus on the value of
human capital. By analyzing performance gaps, employee
performance, and business opportunities, Motorola
University can calculate the value of closing the
performance gaps and therefore determine the ROI of
investing in improving human capital.
Organizational Metrics for Corporate University
Effectiveness
While Kirkpatricks model, ROI measures, and human
capital metrics all serve as good tools for measuring
individual programs, when it comes to evaluating a
corporate university in its entirety, we advocate a more
holistic approach. As organizations invest more and more
money in corporate universities, it becomes increasingly
important for corporate universities to justify their
existence. In order to do this, we recommend mission-based
metrics that are organization-specific.
As we mentioned in our definition, corporate
universities exist to help their parent organizations
achieve their mission. Each corporate universities should

have its own mission statement, and that mission should be

based on the specific needs of the organization. Not every
organization has a corporate university, nor does every
organization need one. But those that have chosen to
establish a corporate university presumably have done so
for a reason. The answer to the question, Why do you have
a corporate university? should drive the measurement of
the effectiveness of the corporate university. If you can
articulate an answer to that question clearly and
specifically, that should serve as a roadmap for what to
measure. And if you cannot answer that question clearly and
specifically, then measurement is not your greatest
problem.
This raises the most crucial issue in corporate
university evaluation: measurement is not something to be
undertaken after the completion of educational activities
it must be baked in up front during the planning process.
As a corporate university and its activities are being
planned, the question of why? must be asked at all times.
All corporate university activities must be goal-directed.
If you can articulate goals for each activity and for the
university as a whole, then achievement of those goals is
what should be measured. The question of what to measure is
simply the answer to the question, What were you trying to

accomplish? Whatever it is that you were trying to do is

what you measure. And the close connection between goals
and measurement reinforces the notion that measurement is a
part of the planning process, not something to be
considered after the fact.
Because of the goals-based focus of corporate
university measurement, often the evaluation process does
not look at traditional measures of learning, but rather it
focuses more on organizational goals. While learning-based
metrics are targeted at individuals, the locus of
measurement for many corporate universities is the
organization. For this reason, we often advocate that
organizational measures (such as profitability, growth,
employee retention, employee satisfaction, productivity,
and many others) are the appropriate metrics for gauging
corporate university success.
For example, the MGM Grand Hotel and Casino in Las
Vegas created its corporate university, the University of
Oz, prior to opening its doors in December of 1993. The
hotel was planning to open with 5,005 guest rooms, which
would make it the largest hotel in the world from its first
day in business. The MGM Grand needed to hire 8,000
employees by its opening day. In addition to the monumental
task of locating, interviewing, and hiring such a large

number of employees, the management at MGM knew that one of

the biggest challenges in the gaming industry is employee
turnover. There are so many large hotel-casinos in Las
Vegas that employees have a great deal of choice about
where to work and change jobs frequently. As the largest
hotel-casino in the city, the MGM Grand knew that high
turnover would be quite costly.
Early in the planning process, the management at the
MGM Grand decided to make a conscious effort to focus on
employee retention. The University of Oz was at the heart
of the retention efforts. Through a thoughtful
acculturation process, the University of Oz created a
culture where none existed before and worked hard to make
all newly hired employees feel that they were a part of
that culture.
Moreover, the University of Oz developed an internal
degree program, the Th.D. (it stands for Doctor of
Thinkology, the same degree the Wizard of Oz bestowed on
the Scarecrow at the end of The Wizard of Oz). A sequence
of courses spanning many months was required to earn the
degree. Although the Th.D. designation was quite
prestigious within the walls of the MGM Grand, it was
meaningless outside of the company, so employees who earned
the degree were less likely to change jobs.

This is

another example of an employee retention effort spearheaded

by the University of Oz.
When it came time to determine if the University of Oz
was doing a good job, management did not need any complex
formulas. Since the primary focus of the University of Oz
was employee retention, that was the organizational metric
that was used as the yardstick for the University of Oz.
Since the MGM Grand had low turnover and high retention,
the University of Oz was deemed a success. While
statisticians might quibble that it is difficult to prove a
causal relationship between the efforts of the University
and the organizational results, we would argue that it is
not necessary in a case like this to prove causality. The
organization had a specific result that it wanted to
achieve, it set up a mechanism to help achieve those
results, and the ultimate goals were met. By any measure,
this is evidence of a successful outcome.
This example demonstrates several important points.
First, there is no single method that is best for corporate
university evaluation. Every corporate university exists
for different reasons, and therefore every corporate
university should have different metrics. Secondly, by
focusing on the specific mission and goals of the corporate
university, it is easy to determine what should be

measured. Moreover, it shows that evaluation need not be

complicated or costly. A highly focused mission can lead to
simple metrics.
The Productivity Model
As described earlier, Donald Kirkpatrick created a
four level of approach for evaluating training programs.
Building upon this foundation, Jack Phillips added a fifth
level, i.e., measuring the Return On Investment (ROI)
produced by training programs. However neither the
Kirkpatrick model nor Phillips ROI formula provide the
type of information needed by instructional designers and
corporate university leaders to improve the productivity of
instruction and related activities. In response for an
alternative evaluation model, McGee has developed the
Productivity Model.
To begin, let us recognize that corporate universities
value and implement a variety of instructional systems to
achieve predetermined goals and outcomes. It is possible
describe any system as being composed of three basic
elements: Resources + Activity = Results
First, let us recognize that all systems produce
results, even though the results they produce may at times
be difficult to predict. Secondly, to achieve results,
something must happen; something must be done. In other

words, activity must take place. Third, in order to have

activity, resources must be used. These three elements of a
system hold true whether the system is mechanical,
electrical, biological, financial, social or educational.
The next step to understanding Productivity Model is
to examine the dynamic relationships that exist between the
elements of a system. We will start with the relationship
between results and activity.
Effectiveness
A system is effective only when an activity creates or
produces a predetermined result with a high degree of
predictability, in other words, effectiveness is concerned
with "how well" something works. This concern for "how
well" is the basis for another concept known as quality. As
designers and managers of systems, we must strive first for
effectiveness. For without it, there is little reason to
proceed with the design or implementation of any system.
Efficiency
Efficiency is the dynamic relationship that exists
between resources and an activity. Efficiency asks the
question, "How much?" However, this is a dangerous question
to ask. It is dangerous in that if we cut resources too
much, we run the risk of producing poor results. This is
not to say that we should not be concerned with resources

and their associated costs. We should, because within every

system there is an optimum balance between resources and
activity, and activity and results. This optimum balance is
known as productivity.

Figure 1. The Productivity Model

Productivity
We can see this balance between effectiveness and
efficiency in Figure 2 below. System (A), which is neither
effective nor efficient, cannot be said to be productive.
System (B), while very effective, cannot be said to be
productive, because it is not efficient. System (C), while
efficient, is not effective. Therefore it cannot be
considered productive. Only system (D), which is both
effective and efficient, can be said to be productive. In
application, it is possible to establish metrics

(performance standards) for the system being evaluated in

terms of effectiveness and efficiency, and furthermore to
determine corrective courses of action when evaluative data
is generated and graphed.

Figure 2. Effectiveness vs. Efficiency

Applying the Model to Improving the Productivity of

Training Systems
If we are to measure and improve the productivity of a
training system, we must focus our attention on two
distinct measures. The first is the quality of the
instruction, i.e., what should be taught? And the second is
the quantity of resources needed to deliver the

instruction, i.e., by what means should the curriculum be

taught?

Figure 3. Curriculum Development and Instructional Strategy

To expand on this concept, instructional quality is

the major concern of curriculum development. Curriculum
development answers the question, What should be taught?

It is during the development of the instructional

curriculum that a performance standard for the curriculum
is established and program content is identified.
Instructional quality is often measured by how well
participants are able to predictably achieve the objectives
of the program. This data is most often gained through
traditional testing procedures. For example, a particular
curriculum may produce results, wherein an average
participant achieves a 95% level of competency on the
material presented.
Instructional quantity, on the other hand, falls into
the domain of instructional strategy, which is the process
of determining and selecting the most efficient method and
media for delivering a program of instruction (curriculum).
The goal of instructional strategy is to answer the
question, By what means should the curriculum be taught?
Again, a performance standard must be established by which
to measure this dimension. Common standards are money,
time, instructional staff, equipment required, i.e.,
instructional resources.
In order to determine the productivity of an
instructional system, we must consider both the results
produced by the curriculum and the instructional resources
required to deliver the curriculum. Keep in mind that

productivity is a ratio or composite measure of both the

effectiveness and efficiency of a system.
The power of the Productivity Model is that it enables
trainers and instructional developers to identify where
they should take corrective action.
There are a number of factors that influence the
effectiveness of an instructional system (curriculum
development). Among these are:

Needs Assessments

Assessment of Learners

Analysis of Work Settings

Job, Task, or Content Analysis

Statements of Performance Objectives

Performance Measurements (test items)

Sequence Performance Objectives

The factors that influence the efficiency of an

instructional system (instructional strategy)are:

Instructional Techniques

Designs for Instructional Materials

Instructional Resources, i.e., money, time,

instructional staff, or required equipment.

Where and When to use the Productivity Model

Because the Productivity Model is a logical construct,
it may be used to evaluate and fine-tune the development
activities within any phase of the ADDIE model, i.e.,
analysis, design, development, implementation or
evaluation. Furthermore, the Model may be used at all
levels of evaluation as described by Kirkpatrick (1994) and
used in conjunction with the concepts espoused by Phillips
(1997) and his work related to measuring the ROI of
training systems.
A Grounded Model
The Productivity Modeled described in this paper is
based upon established principles from a variety of
disciplines.
The basic elements of a system: resources, activity
and results, are described throughout the literature on
systems theory and are sometimes referred to simply as an
input/output model.
Accountants and financial people, who often speak of
return on investment, have known the relationship between
resources and results: (ROI). ROI is where returns are
results and investments are costs, and in this way, are
able to determine the health of a business enterprise
(activity).

Physicists and engineers recognized long ago that in

order to describe the performance of various phenomena and
systems, they had to be described in terms of dynamic
relationships between two variables. For example, miles per
gallon or feet per second.
Thomas Gilbert (1978), a founding father of the human
performance field, developed the First Leisurely Theorem,
which says that worth is equal to value divided by costs.
In other words, activity adds value to resources (cost) and
results in something of greater worth.
Peter Drucker (1974) pointed out that effectiveness
was doing right things, while efficiency was doing things
right. In the field of education, these concepts were
expanded upon by Ivor Davies (1981) and applied to
decision-making concerning instructional methods.
Conclusion
We illustrated that corporate universities, despite
eschewing the term institutional research, have developed
a variety of useful and creative methods for evaluation.
The menu of methodologies presented in this chapter are not
meant to be exhaustive, merely instructive. Some
organizations might choose to duplicate or adapt some of
the techniques highlighted in this chapter. If not, the
lesson to be learned is that evaluation is mission-

specific, and the thoughtful application of the techniques

in this chapter and some common sense can lead to the
development of evaluation methods that match the goals of
the parent organization, the mission of the corporate
university, and the specific informational needs that drive
the desire to evaluate.

References
Allen, M. (2002). The corporate university handbook. New
York: Amacom.
Barney, M. (2002). Measuring ROI in corporate
universities: Death of the student day and birth of
human capital. In M. Allen (Ed.), The corporate
university handbook. New York: Amacom.
Davies, I. K. (1981). Instructional techniques. New York:
McGraw-Hill Book Company.
Drucker, P. (1974). Management tasks, responsibilities,
practices. New York: Harper & Row Publishers.
Eurich, N. P. (1985). Corporate classrooms. Princeton, NJ:
The Carnegie Foundation for the Advancement of
Teaching.
Gilbert, T. F. (1978). Human competence. New York: McGrawHill Book Company.
Kirkpatrick, D. (1994). Evaluating training programs. San
Francisco, CA: Berrett-Koehler Publishers.
Phillips, J.J. (1997). Return on investment in training and
performance improvement programs. Boston, MA:
Butterworth-Heinemann.
Renaud-Coulon, A. (2002). Corporate universities in
Europe. In M. Allen (Ed.), The corporate university
handbook. New York: Amacom.

Senge, P. M. (1990). The fifth discipline. New York:

Doubleday.
Thompson, G. (2000). Unfulfilled prophecy: The evolution
of corporate colleges. The Journal of Higher
Education, 71, (3), 322-341.

About the Authors

Mark Allen, Ph.D. is an author, consultant, and
educator who has specialized in corporate universities. He
is the editor and lead author of The Corporate University
Handbook, published in 2002 by Amacom. Mark is also the
Director of Executive Education at Pepperdine Universitys
Graziadio School of Business and Management, where he is
responsible for designing and delivering executive
education programs to corporate clients. He is also an
adjunct professor at the Graziadio School and at
Pepperdines Graduate School of Education and Psychology.
He lives in Redondo Beach, California with his wife Dayna
and two sons, Skyler and Dylan.
For seventeen years, Philip McGee, Ed.D. was involved
in the design and implementation of instructional and
organizational systems within education, government,
industry and business through his company, Instructional
Designs, Inc. During this time he also served as the

Regional Program Director for the Doctor of Business

Administration program offered by the School of Business
and Entrepreneurship, Nova Southeastern University, and as
an Adjunct Professor of HRD for Webster University. Dr.
McGee is currently an Assistant Professor of Technology and
HRD at Clemson University.