Tata Institute of Social Sciences, Mumbai

M. A. in Organisation Development, Change and Leadership Program

ODCL 19 - Evaluation of OD and Change: New Developments

Case Study Assignment no.2

Date:​ ​11​th​ February, 2020

Case Analysis: ​Eyes of Janus: Evaluating Learning & Development at Tata Motors

Submitted by: ​Shruti, Shreyansi, Harsh, Sanjay & Vaseem

This case details a revamping effort of the L&D function at Tata Motors.

1) What is your assessment of the way Tata Motors Academy (TMA) was
built and the way the programs are being delivered? Is there a better
way of doing this? (30%)

Ans. 1

Organization Context

● Employee strength: 49000

● Major strength is customer orientation- reflected in 3500 distributors and service
centres.
● Enthusiastic in introducing locally innovative products to cater to the emerging needs of
the community.
● Tata Motors Academy (TMA) was conceived as a corporate university to realize the
human capital strategy. The university was established in 2011 in Mumbai.
● TMA was created to cater to the learning needs of all employees and other key
stakeholders such as dealers, vendors and contract employees.
 ● In TMA, 60-70% of the training was focused on developing the functional and technical
capabilities, both designed and customized by L&D to meet local needs but the
management and global programs designed centrally for the entire organization.
● Guiding principle of TMA

1. Alignment of business strategy

2. Leadership involvement in TMA
3. Efficiency in training delivery
4. Training effectiveness
5. Appropriate L&D investment
6. Broad based learning opportunities
7. Competency of the L&D
8. Measurement of training outcomes

● ​The Academy was structured in six centres of excellence (COEs).

1. Manufacturing Excellence and Innovation

2. Operator Training
3. Commercial Training
4. Dealer Training
5. Corporate Training
6. Behavioural Training

The possible way to assess the TMA and also the delivery of the programs which are being
delivered is to understand how TMA performed on the guiding principles while highlighting,
what they did on short term and also in the long term to ensure sustainability.

1. Alignment of business strategy:

● Learning programs made relevant to business by measuring outcomes

● Governance structure through a learning advisory council to design and create critical
training programs
● Emphasis on company specific and practice oriented issues

2. Leadership involvement in TMA

● Leaders were involved to create a culture of Leaders as Teachers- to develop internal

trainers and subject matters experts.
● Programs like ‘Iteach’ and certifications to motivate employees to make sure employees
participate and be involved in the learning.
 ● It was evident in the governing structure of learning advisory that training was to be
driven by business and not just by HR and L&D and the way TMA designed it in such a
way that an ‘Iteach’ trainer would receive 200 Iteach reward point and that can be
converted in to non-monetary rewards with the external parties.

3. Efficiency in training delivery

● Shift in focus from volume training conducted to efficiency of the programs.

● The efficiency data were gathered and then interventions made to achieve the efficiency
of the programs.
● Ensuring learning outside India: connecting their overseas locations though TMA online
and offering e-learning courses.
● Common dashboards used for all locations and business units.

4. Training effectiveness

● L&D teams used five standard criteria (delivery quality, knowledge gain, application to
job, business impact and value) to measure effectiveness.
● Business impact data were the KPI linked to the desired outcomes of investing linked to
L&D.

5. L&D Investment

● Increased investment from 2010 onwards.

● With close to 50% of the training done internally.
● The belief that inducting business HR colleagues into the methodology would help when
training programs were designed and delivered.

6. Broad based learning opportunities

● Several initiatives were taken for building a culture of learning.

● Knowledge portals, creating diverse portfolio of e-learning modules, coaching and
mentoring programs etc.
● Focused investment made to facilitate online learning with the development of the TMA
online e-learning platform.

7. Competency of the Learning & Development Team

● T​MA made significant investments in both infrastructure and learning resources.

● The team also attended focused programs on the 6D process, enhancing business acumen,
and becoming better facilitators under the Iteach process.
8. Measurement of Training Outcomes

● Though the outcome of training investment could be seen at the organizational level, as
the change was felt by one and all.
● Senior management appeared to be more interested in quantifiable outcomes and metrics.

Is there a better way of doing this?

OD is a long term change effort that requires as compared to short term training, which has a
very different focus but also requires an evaluation component.

● It is mentioned that L&D team was in the initial stages of developing metrics to measure
the impact of training programs on business outcomes and the way the team was
approaching was linking participants' collective opinions of the value of the program and
its predictive impact on job performance. So it were the participants who were guiding
and being autonomous of the intended achievement of the outcome of the program
instead of the senior management whose say was not mentioned, how they wanted to be
accomplished in clear stated terms. If L&D and LAC had developed the right metrics to
capture what measurable business outcome, senior management wanted to achieve
throughout this whole change process, it would have given more intended result.
● Almost 50% of the training was done internally despite having other parties during the
whole process which help us to draw an assumption that the program would have
achieved a higher success rate if the mix of both internal and externally were linked to
overall program charter.
● Also organization appeared to be weaker on the feedback side. Despite the success of
‘Iteach’ and other programs, there does not seem to making a forward step to add new
material and also new behavioural interventions after achieving this impressive success
of the program.
● The gap between the investment made and investment needed in L&D for the effective
delivery of the TMA charter and to secure the additional investment during the tough
business environment faced by TML justify the lack of acknowledging the investment
which has already been made by the organization because of not linking the direct
performance outcomes which are important to deliver the real time growth which is
evident from the profit and loss statement. From March03(12) to March12(12), there is
an increase in total income and also there is a net increase in profit also but there was no
significant increase in the total profit post introduction of the program considering the
increasing cost of raw material and overall employee cost.
● The metric developed by Bector only evaluated the preliminary level of factors. The
organization needed an advanced level of metrics to evaluate training effectiveness at
individual level and group level. The performance dashboard would have been more
 advanced by including other factors e.g. which functional/ technical/behavioural training
were covered most and which gave the direct result related to business outcome. Also
what were the training areas that needed the more significant interventions? For example:
Pantnagar’s consumption of annual budget was 96% but turn out rate for the functional/
technical training is 56%.

Though organization seemed to be at the forefront while devising the strategy and understanding
the importance of the program, the scope of improvement could be identified from the below
table also.

Demands of Evaluation Resources of Evaluation

Accountability of the result Time for designing, data gathering,

analyzing, and reporting

Deciding further investments Time for employees participating in the

evaluation

Revising the OD intervention strategy Professionals from HR, finance, social

research, and so on

Finding a success story Consulting costs

Organization power and political Opportunity costs (e.g., deferred decision)

situations

Significance of results for employees’ Technology

health and safety

2) Comment on the current evaluation practices pursued at TMA. (35%)

Ans. 2

It is evident from the case, Tata Motors faced many ups and downs in its given tenure and
despite incurring losses during economic slowdown, the organisation managed and survived in
the automobile industry. Tata motors wanted to differentiate themselves from the automobile
industry. They setup the Tata Motors Academy to train their employees. They worked on their
capabilities and also with a close focus on customer centricity and innovation to sustain and
grow in the market, they setup the Tata Motors academy to train their employees and to make
them future ready.

Currently the Learning & Development teams use the following criteria to measure ​efficiency in
training delivery:

i.​
​ The number of training programs delivered
​

ii.​ No of participants
​

iii.​ Cost per participant etc.

​

There are centrally driven programs focused on global leadership management development and
other induction programs.

Learning & Development teams used five standard criteria to measure effectiveness:

i.​
​ Delivery quality
​

ii.​ Knowledge gain

​

iii.​ Application to job

​

iv.​ Business impact

​

v.​ Creation of values

​
 The objectiveness of this measurement was to link training effectiveness with the tangible
performance improvements & business outcomes. (E.g. revenue, productivity, contribution of
profit per employee)

They were also in the initial stages of developing metrics to measure impact training
programs on business outcomes; they were also incorporating the participant’s feedback (Exhibit
8) and internal metrics which was developed by Bector (Exhibit 10) as a practice of evaluation.
They struggled to develop metrics that could evaluate training effectiveness at the group and
individual level. As finally they need to prove it to senior management in terms of quantified
outcomes and metrics which are further than these. Their current practices seem very basic. They
could use the Kirkpatrick model to have an in depth evaluation. They seem to apply maybe just 1
part of the model … its opportunity apply (Exhibit 8).

Evaluation of a training event should be carefully planned and include conversations that
establish the time frames for evaluation, type of data needed to complete the evaluation, and
what participants will be included in the sample. Another critical consideration would be to
define the training outcomes and then consider what evaluation metrics will ascertain if those
objectives have been met.

Another useful frame- work for developing a training evaluation is the Philips ROI. It is best to
conduct a summative evaluation at the end of the program to assess the four levels of evaluation,
including Phillips’ fifth level, and return on investment (ROI) (Head & Sorensen, 2005;
Rothwell et al., 2010). ROI seeks to connect interventions with outcome metrics that makes
improvement clearly definable.

The complexity of the task or skill will also influence the type of evaluation used. In addition,
the evaluation of a training event will focus only on the training activities and does not consider
wider organizational change efforts or other variables that fall outside the time frame or episode.
Usually, a training evaluation falls into the short-term category and is focused on the one
intervention/event.
ROI calculation (Adapted from Phillips, J. J. (1997). ​Handbook of Training Evaluation and
Measurement Methods,​ 3rd ed. Houston, TX: Gulf Publishing Company.)

Overall there is a need at TMA to measure the effectiveness of TMA training programmes. This
is a daunting task due to the large number of courses being offered, the multi-site specific
training, and the dealer training. The challenge is to see what is working, what is contributing to
the increase in profits, and how best to utilize the limited HR resources. Also finding relevant
metrics to demonstrate and measure the impact of L&D at the bottom line of the company and
how TMA has impacted talent management of TML and related businesses is also a task for
TMA now. It is better to follow a model which is all inclusive and measures performance as well
as business impact which have been suggested above.

3) How do you measure the effectiveness of the various initiatives taken by

TMA? (35%)

Ans. 3

PRE INTERVENTION PHASE:

Firstly let’s understand the various initiatives taken by Tata Motors Ltd. w.r.t Tata Motors
Academy which were a response to the emerging business dynamics, these were as follows:

1.​ New Human Capital Strategy​ – A renewed focus on developing and retaining talent by
​

building a learning culture.

○ Led to the emergence of vision of talent management and learning & development
function as a strategic partner to drive the culture change.(Functional strategy)
○ To cater to the TMLs vision of change & growth (Vision of being the - “Most
admired by our customers, employees, business partners and shareholders for the
experience and value they enjoy from being with us” & Mission – “to be
passionate in anticipating and providing the best vehicles and experiences that
excite our customers globally”).
 ○ By getting employees updated, upskilled & equipped with necessary
competencies to face challenges of increasing competition and dynamic business
conditions.
○ Objective of the New Human Capital Strategy was to create a culture of high
performance through a programme called ACES.
○ Three planks of the New Human Capital Strategy aimed at building sustained
customer satisfaction and business performance were:
■ Talent and Leadership
■ Organizational Renewal
■ Organizational Enablers
2. Tata Motors Academy, a corporate university,​ was established in early 2011 to realise
the new Human Capital Strategy (drive learning agenda).
○ Overarching objective was to transform TML into a world class destination for
top talent. (Catered to the "learning" needs of employees, shareholders, and
dealers.)
○ CHROs 3 main criteria for evaluating TMAs performance (L&Ds Internal metrics
had internal measures and lacked external measures to track performance):
■ The number and quality of internal specialists TMA could attract as
faculty for its programs
■ The self – learning and career development created by TMA’s
interventions, especially through e-learning programs
■ Impact of TMA programs on leadership development and performance
improvement
○ However, the three criteria failed to explain concerns regarding the financial
returns on the investment (ROI) made in TMAs (in terms of quantifiable business
outcomes & metrics) key initiatives.
○ Clearly in the case it was seen that the TMA and CHRO were not able to clearly
address the challenges of evaluation of their Intervention due to a lack of properly
designed evaluation strategy (in terms of scope and rigor they were in the
‘Dynamic phase’ leading to a misinterpretations).

INTRA IMPLEMENTATION PHASE:

TMA was created to take care of the learning needs of all employees and other stakeholders
(Contract employees, vendors & dealers).
 ·​ This was compromised as one of the guiding principles of TMA i.e. the
​

measurement of training outcomes could easily see the outcome of training

investment at organization level and struggled to capture and explain training
effectiveness at the group and individual level.

1. Learning programs offered in four broad areas:

·​ Functional and technical skills

​

·​ Management skills
​

·​ Mandatory programs (new recruit induction, compliance training etc.)

​

·​ Global leadership skills

​

2. Structured in Six CENTRES of Excellence (COEs) focusing on target groups or types of

activity. Major COEs were:

·​ Manufacturing excellence and innovation

​

·​ Operator training
​

·​ Commercial training
​

·​ Dealer training
​

·​ Corporate training
​

·​ Behavioural training
​

They were further divided into sub-centres. Learning support services (LSS) dept. provided
common services to the COEs. COEs and LSS operated in a matrix structure. This could also
have been a potential pain area considering the complexity that was built into structure, function
and processes making it difficult to create a robust evaluation design.

3. Program designs

·​ 60- 70 % of TMAs programmes focused on developing functional and technical

​

capabilities & designed & customized to local L&D needs by the local L&D teams.
 ·​ Management and Global programs were designed centrally for the entire
​

organisation.

·​ All behavioural training programs were aligned to the leadership and Tata Leadership
​

Practice competency frame work.(common to Tata group companies)

What seemed to work for programs designed and conducted by TMA was the evolution of
process of program design and revisions followed by TMA

E.g. the popular Executive Development Program on global leadership.

·​ The process redesign was guided by 6 disciplines of learning (6D Approach – US

​

based Fort Hill co.)

·​ Its shift from conventional launch to a judicious mix of e learning, classroom learning
​

and on field internships based on the feedback received on the program.

·​ The important feature of this new design a 4-6 day module on cross domain
​

orientation allowed participants to spend time with departments other than their own.

·​ The net effect of all these reforms – course dealt with issues of Individual specific,
​

Company specific and practice oriented and also providing understanding of other
departments.

4. Therefore, ​5 features of TMAs internal reviews​ did bring success and broad based
acceptance at TML but the features lacked an external metric to get a holistic view of the
impact created by the programs & training. Therefore, was the vision of TMA truly
inclusive?

5. The training and variety of groups, content and regions involved made it massive and
complex operation. The task was to quantify the magnitude of the coverage required by
TMA with nearly 49,000 employees and 2000 dealers having thousands of salesmen. To
ensure uniform implementation of BU aligned learning programs for operation of large
magnitude while catering to local needs – involved 3 criteria of evaluation &
improvement:

·​ Efficiency
​
 ·​ Effectiveness
​

·​ Business outcomes
​

This was a weak evaluation design considering the scale they were operating on.

6. 8 Guiding Principles of TMA ​(tried to focus on business strategy, implementation,

outcomes and evaluation):

·​ Alignment of business strategy

​

(Paradigm shift was created with the governance structure driven through Learning
Advisory Council i.e. training was to be driven by business rather than HR & L&D –
changing their focus to L&D team’s performance metrics and were being evolved.
The question was what would be the reference for new metrics)

·​ Leadership involvement in TMA

​

(The use of 200 Iteach reward points weren’t the primary drivers rather was the
recognition accorded to the Iteach trainers as it was both an aspirational and
motivational factor for employees. Therefore, rewards system used weren’t relevant
and the question of retention of this tacit knowledge)

·​ Efficiency in training delivery

​

(Brought in efficiency data and yet the functional and technical programs with
60-70% of total offerings were conducted locally at BU level with LACs approval did
not take into account cross collaboration opportunity needs)

·​ Training effectiveness
​

(The learning impact calculator was still in the making thus the programs were
conducted in the absence of a robust evaluation design & metrics)

·​ Appropriate L&D investment

​

·​ Broad basing learning opportunities

​

·​ Competency of the L&D team

​
 (Continued existence of large gaps in terms of investment made and investment
needed in L&D for effective delivery of TMA charter. Therefore, questioning the
additional investment in a tough business environment faced by TML)

·​ Measurement of training outcomes

​

(Outcome and Impact seen and felt at organisational level but could not account for
group and individual level)

These guidelines did help in uncovering a number of gaps and shortcoming of TMA & the L&D
team. However, they could not ensure timely corrective measures being taken at TML.

7. TMA plan to bring efficiency in Training delivery:

Criteria for effective training

● Delivery quality
● Knowledge gain
● Application to job
● Business impact
● Value

Key performance indicators

● Business impact data
● Learning impact calculator

Alternative ways to measure the effectiveness of Training program can be used from OD theory
of evaluating any change process.

We can evaluate any implementation at different stages using various tools:

1) Formative evaluation: ​In this part of evaluation, the implementation is in the middle of
intervention. This type of evaluation is a dipstick which is done to measure if the process going
are aligned with the required outcome. Data gathering and the feedback process for evaluation
can serve as a communication during the implementation to ensure that participants are
constantly engaged and learning are aligned with the Business outcome.
 2) Summative evaluation:​ This type of evaluation is conducted immediately after training to
ensure that the training session offered values, merit, or worth on the levels of individuals, teams,
or organizations.

Checklist for starting up an evaluation plan:

Sr. no. Questions Check when completed

1) What needs to be achieved through the Learning

module?

2) What would the desired outcome look like?

3) What metric will be used to measure success for the

desired learning intervention?

4) Who/which are the targets of change (e.g.,

people/group/units)? What level of change do we
need?

5) What are the specific evaluation expectations at each

level?

6) Are the evaluation expectations aligned with the aims

and levels of the Training outcome?

7) Are you in agreement with each level of targets, with

regard to evaluation goals and objectives?

OD change evaluation theory also explains various advantages of taking an external evaluator or
an external evaluator.

Instructions:​ Place a check on the left column for all the areas accomplished to guide you in
determining the OD evaluation criteria (to be completed by evaluator).
 Questions/Statements for Consideration

1. Have you had a preliminary meeting with different stakeholders regarding

determining the evaluation criteria?

2. Is senior management in support of determining evaluation criteria and initiating

an evaluation process?

3. Have the stakeholders and/or the senior management given feedback about what
outcomes they expect from the Training evaluation?

4. Have the stakeholders and/or senior management given feedback how the results
of this evaluation will be used in the future?

5. Has the required format of the evaluation outcomes been identified?

6. Has a specific time period been identified for providing a summary of feedback
and results based on the required format established?

7. Is there a consensus among the stakeholders and senior management on who

would have access to the outcomes of this evaluation?

8. Are the criteria focused on the identified problem and these interventions, rather
than on the effectiveness of the overall Training & Development effort? (to ensure
validity and evaluation of different levels of the OD process)

9. Are the evaluation criteria in alignment with the Business outcome and goals?

POST INTERVENTION PHASE

○ Consider the low and high rigor options their focus entailed – Knowledge,
Behaviour & Attitude. However, did not clearly focus on their rigour w.r.t Impact
and Return.
 ○ The process variables and outcome variables w.r.t to Measurement of Training
Outcomes had developed metrics that focused and could evaluate outcomes of
training effectiveness at the organizational level and had undermined/ struggled to
develop metrics that could evaluate at the group and individual level. The strategy
alignment did focus on Primary/external metrics but struggled with Advanced/
internal metrics for measuring outcomes. Another area of concern was the
frequency of change in the existing performance metrics.
○ The strategy Alignment w.r.t strategy and organizational competence and capacity
in fragmentation phase considering the explanation of financial return on
investment. Due to lack of connect between strategy and performance in finite
and measurable terms to track the outcomes.
○ The data used for evaluation were mostly- professional data, soft data, energy
data, readiness and capability data; and lacked hard data, political data,
competency & competitor data.
○ Running too many programs at TMA led to an inability of evaluators to
demonstrate the benefit of participation of the programs run by them. A
possible reason could be the experience of the evaluators in driving such a
large scale change thus making it difficult to make evaluation decisions
and comprising the impact of an evaluators role.

Important Instructions:
When analysing and responding to the case questions please remember that you are
not simply being assessed on how well you understood the case. You are also graded
on how well you use and incorporate the course material discussed in class regarding
evaluation. A high-quality report will meaningfully relate the case situations with the
course material.
Please remember that this case was not developed exclusively for a course on Change
Evaluation. You will in some cases need to be creative to see how the concepts
discussed in class may apply.

This is a team assignment. There will be one submission per team. You will need to
submit your team’s report in a Dropbox created in Moodle.

-------------------------------------------------------------------------------------------------------------------------------
Tata Motors
This case details a revamping effort of the L&D function at Tata Motors.
1) What is your assessment of the way Tata Motors Academy (TMA) was built and
the way the programs are being delivered? Is there a better way of doing this? (30%)

Organization Context

● ​Employee strength: 49000

● ​Major strength is customer orientation- reflected in 3500 distributors and service

centres.
● ​Enthusiastic in introducing locally innovative products to cater to the emerging

needs of the community.

● ​Tata Motors Academy (TMA) was conceived as a corporate university to realize

the human capital strategy. The university was established in 2011 in Mumbai.
● TMA was created to cater to the learning needs of all employees and other key
stakeholders such as dealers, vendors and contract employees.
● ​In TMA, 60-70% of the training was focused on developing the functional and

technical capabilities, both designed and customized by L&D to meet local needs
but the management and global programs designed centrally for the entire
organization.
● Guiding principle of TMA
1. Alignment of business strategy
2. Leadership involvement in TMA
3. Efficiency in training delivery
4. Training effectiveness
5. Appropriate L&D investment
6. Broad based learning opportunities
7. Competency of the L&D
8. Measurement of training outcomes

● The Academy was structured in six centres of excellence (COEs).

​

1. Manufacturing Excellence and Innovation

2. Operator Training
3. Commercial Training
4. Dealer Training
5. Corporate Training
6. Behavioural Training
The possible way to assess the TMA and also the delivery of the programs which are being
delivered is to understand how TMA performed on the guiding principles while highlighting,
what they did on short term and also in the long term to ensure sustainability.

1. Alignment of business strategy:

● Learning programs made relevant to business by measuring outcomes

​

● ​Governance structure through a learning advisory council to design and create critical
training programs
● Emphasis on company specific and practice oriented issues

2. Leadership involvement in TMA

● Leaders were involved to create a culture of Leaders as Teachers- to develop internal

trainers and subject matters experts.
● Programs like ‘Iteach’ and certifications to motivate employees to make sure employees
participate and be involved in the learning.
● It was evident in the governing structure of learning advisory that training was to be
driven by business and not just by HR and L&D and the way TMA designed it in such a
way that an ‘iteach’ trainer would receive 200 iteach reward point and that can be
converted in to non-monetary rewards with the external parties.

3. Efficiency in training delivery

● Shift in focus from volume training conducted to efficiency of the programs.

● The efficiency data were gathered and then interventions made to achieve the efficiency
of the programs.
● Ensuring learning outside India: connecting their overseas locations though TMA online
and offering e-learning courses.
● Common dashboards used for all locations and business units.

4. Training effectiveness

● L&D teams used five standard criteria (delivery quality, knowledge gain, application to
job, business impact and value) to measure effectiveness.
● Business impact data were the KPI linked to the desired outcomes of investing linked to
L&D.

5. L&D Investment

● Increased investment from 2010 onwards.

 ● With close to 50% of the training done internally.
● The belief that inducting business HR colleagues into the methodology would help when
training programs were designed and delivered.

6. Broad based learning opportunities

● Several initiatives were taken for building a culture of learning.

● Knowledge portals, creating diverse portfolio of e-learning modules, coaching and
mentoring programs etc.
● Focused investment made to facilitate online learning with the development of the TMA
online e-learning platform.

7. Competency of the Learning & Development Team

● T​MA made significant investments in both infrastructure and learning resources.

● The team also attended focused programs on the 6D process, enhancing business acumen,
and becoming better facilitators under the iteach process.

8. Measurement of Training Outcomes

● Though the outcome of training investment could be seen at the organizational level, as
the change was felt by one and all.
● Senior management appeared to be more interested in quantifiable outcomes and metrics.

Is there a better way of doing this?

OD is a long term change effort that requires as compared to short term training, which has a
very different focus but also requires an evaluation component.

● It is mentioned that L&D team was in the initial stages of developing metrics to measure
​

the impact of training programs on business outcomes and the way the team was
approaching was linking participants' collective opinions of the value of the program and
its predictive impact on job performance. So it were the participants who were guiding
and being autonomous of the intended achievement of the outcome of the program
instead of the senior management whose say was not mentioned, how they wanted to be
accomplished in clear stated terms. If L&D and LAC had developed the right metrics to
capture what measurable business outcome, senior management wanted to achieve
throughout this whole change process, it would have given more intended result.
● Almost 50% of the training was done internally despite having other parties during the
whole process which help us to draw an assumption that the program would have
 achieved a higher success rate if the mix of both internal and externally were linked to
overall program charter.
● ​Also organization appeared to be weaker on the feedback side. Despite the success of
‘iteach’ and other programs, there does not seem to making a forward step to add new
material and also new behavioral interventions after achieving this impressive success of
the program.
● The gap between the investment made and investment needed in L&D for the effective
delivery of the TMA charter and to secure the additional investment during the tough
business environment faced by TML justify the lack of acknowledging the investment
which has already been made by the organization because of not linking the direct
performance outcomes which are important to deliver the real time growth which is
evident from the profit and loss statement. From March03(12) to March12(12), there is
an increase in total income and also there is a net increase in profit also but there was no
significant increase in the total profit post introduction of the program considering the
increasing cost of raw material and overall employee cost.
● The metric developed by Bector only evaluated the preliminary level of factors. The
organization needed an advanced level of metrics to evaluate training effectiveness at
individual level and group level. The performance dashboard would have been more
advanced by including other factors e.g which functional/ technical/behavioral training
were covered most and which gave the direct result related to business outcome. Also
what were the training areas that needed the more significant interventions? For example:
Pant-nagar’s consumption of annual budget was 96% but turn out rate for the functional/
technical training is 56%.

· Though organization seemed to be at the forefront while devising the strategy and
​

understanding the importance of the program, the scope of improvement could be identified
from the below table also.

Demands of Evaluation Resources of Evaluation

Accountability of the result Time for designing, data gathering,

analyzing, and reporting

Deciding further investments Time for employees participating in the

evaluation
 Revising the OD intervention strategy Professionals from HR, finance, social
research, and so on

Finding a success story Consulting costs

Organization power and political situations Opportunity costs (e.g., deferred decision)

Significance of results for employees’ Technology

health and safety

2) Comment on the current evaluation practices pursued at TMA. (35%)

It is evident from the case, Tata Motors faced many ups and downs in its given
tenure and despite incurring losses during economic slowdown, the organisation
managed and survived in the automobile industry.Tata motors wanted to differentiate
themselves from the automobile industry. They setup the Tata Motors Academy to train
their employees. They worked on their capabilities and also with a close focus on
customer centricity and innovation to sustain and grow in the market, they setup the
Tata Motors academy to train their employees and to make them future ready.
Currently the Learning & Development teams uses the following criteria to
measure ​efficiency in training delivery:
i. The number of training programs delivered
ii. No of participants
iii. Cost per participant etc.

There are centrally driven programs focused on global leadership management development
and other induction programs.

Learning & Development teams used five standard criteria to measure

effectiveness:
i. Delivery quality
ii. Knowledge gain
iii. Application to job
iv. Business impact
v. Creation of values
 The objectiveness of this measurement was to link training effectiveness with the
tangible performance improvements & business outcomes.
(e.g.revenue,productivity,contribution,of profit per employee)

They were also in the initial stages of developing metrics to impact of training
programs on business outcomes, they were also incorporating the participants
feedback(Exhibit 8) and internal metrics which was developed by Bector (Exhibit 10)
as a practice of evaluation. They struggled to develop metrics that could evaluate
training effectiveness at the group and individual level. As finally they need to prove it to
senior management in terms of quantified outcomes and metrics which are further than
these. Their current practices seem very basic. They could use the Kirkpatrick model to
have an in depth evaluation. They seem to apply maybe just 1 part of the model … its
opportunity apply (Exhibit 8).

Evaluation of a training event should be carefully planned and include conversations

that establish the time frames for evaluation, type of data needed to complete the
evaluation, and what participants will be included in the sample. Another critical con-
sideration would be to define the training outcomes and then consider what evaluation
metrics will ascertain if those objectives have been met.
Another useful frame- work for developing a training evaluation is the Philips ROI. It is
best to conduct a summative evaluation at the end of the program to assess the four
levels of evaluation, including Phillips’ fifth level, and return on invest- ment (ROI) (Head
& Sorensen, 2005; Rothwell et al., 2010). ROI seeks to connect interventions with
outcome metrics that makes improvement clearly definable.

The complexity of the task or skill will also influence the type of evaluation used. In
addition, the evaluation of a training event will focus only on the training activities and
does not consider wider organizational change efforts or other variables that fall outside
the time frame or episode. Usually, a training evaluation falls into the short-term
category and is focused on the one intervention/event.
ROI calculation. (Adapted from Phillips, J. J. (1997). ​Handbook of Training Evaluation and
Measurement Methods,​ 3rd ed. Houston, TX: Gulf Publishing Company.)

Overall there is a need at TMA to measure the effectiveness of TMA training

programmes. This is a daunting task due to the large number of courses being offered,
the multi-site specific training, and the dealer training. The challenge is to see what is
working, what is contributing to the increase in profits , and how best to utilize the
limited HR resources. Also finding relevant metrics to demonstrate and measure the
impact of L&D at the bottom line of the company and how TMA has impacted talent
management of TML and related businesses is also a task for TMA now. It is better to
follow a model which is all inclusive and measures performance as well as business
impact which have been suggested above.

3)How do you measure the effectiveness of the various initiatives taken by TMA?(35%)

Ans . 3

Firstly lets understand the various initiatives taken by Tata Motors Ltds. w.r.t Tata Motors
Academy which were as follows:
● TMA a corporate university to realise the Human Capital Strategy.
● Overarching objective was to transform TML into a world class destination for top
talent.Catered to the "learning" needs of employees, shareholders, and dealers.
 ● CHROs 3 criteria for

Structured in Six CENTRES of Excellence (COEs)

● Functional and Technical
● Management Programs
● Mandatory Programs
● Global Leadership Program

TMA plan to bring efficiency in Training delivery:

Criteria for effective training

● Delivery quality
● Knowledge gain
● Application to job
● Business impact
● Value

Key performance indicators

● Business impact data
● Learning impact calculator

Alternative ways to measure the effectiveness of Training program can be used from OD theory
of evaluating any change process.

We can evaluate any implementation at different stages using various tools:

1) Formative evaluation: In this part of evaluation, the implementation is in the mid of

intervention. This type of evaluation is a dipstick which is done to measure if the process going
are aligned with the required outcome. Data gathering and the feedback process for evaluation
can serve as a communication during the implementation to ensure that participants are
constantly engaged and learning are aligned with the Business outcome.

2) Summative evaluation: This type of evaluation is conducted immediately after training to

ensure that the training session offered values, merit, or worth on the levels of individuals,
teams, or organizations.
Checklist for starting up an evaluation plan:

Questions Check when completed

1) What needs to be achieved through the Learning

module?

2) What would the desired outcome look like?

3) What metric will be used to measure success for the

desired learning intervention?

4) Who/which are the targets of change (e.g.,

people/group/units)? What level of change do we need?

5) What are the specific evaluation expectations at each

level?

6) Are the evaluation expectations aligned with the aims

and levels of the Training outcome?

7) Are you in agreement with each level of targets, with

regard to evaluation goals and objectives?

OD change evaluation theory also explains various advantages of taking an external evaluator
or an external evaluator.

Instructions: Place a check on the left column for all the areas accomplished to guide you in
determining the OD evaluation criteria (to be completed by evaluator).
Questions/Statements for Consideration

1. Have you had a preliminary meeting with different stakeholders

regarding determining the evaluation criteria?

2. Is senior management in support of determining evaluation criteria

and initiating an evaluation process?

3. Have the stakeholders and/or the senior management given feedback

about what outcomes they expect from the Training evaluation?

4. Have the stakeholders and/or senior management given feedback

how the results of this evaluation will be used in the future?

5. Has the required format of the evaluation outcomes been identified?

6. Has a specific time period been identified for providing a summary of

feedback and results based on the required format established?

7. Is there a consensus among the stakeholders and senior

management on who would have access to the outcomes of this
evaluation?

8. Are the criteria focused on the identified problem and this

interventions, rather than on the effectiveness of the overall Training &
Development effort? (to ensure validity and evaluation of different levels
of the OD process)

9. Are the evaluation criteria in alignment with the Business outcome

and goals?
Important Instructions:
When analyzing and responding to the case questions please remember that you are not simply
being assessed on how well you understood the case. You are also graded on how well you use
and incorporate the course material discussed in class regarding evaluation. A high-quality
report will meaningfully relate the case situations with the course material.
Please remember that this case was not developed exclusively for a course on Change
Evaluation. You will in some cases need to be creative to see how the concepts discussed in
class may apply.

This is a team assignment. There will be one submission per team. You will need to submit your
team’s report in a Dropbox created in Moodle.