The Level 1 of Training Evaluation/Awab

The Level 1 of Training Evaluation

A Case of Training the HEIs Staff in Pakistan
M. Awab, M&E Specialist, TESP-HEC, Islamabad

Abstract
The case study is about the experience of the TESP M&E Unit of the
evaluation of the trainings of HEIs staff in Pakistan. The case study
explains how the M&E Unit standardized the Level-1 evaluation format
to analyze the contribution of different elements in making the training
a success. The Model produced surprisingly interesting results for
improved accountability and planning for future trainings.

Background
My recent assignment was with the World Bank funded Tertiary Education Support Project (TESP), as
Monitoring & Evaluation (M&E) Specialist. One of the important tasks of the Project was to build the
capacity of the senior management of higher education institutions (HEIs) in Pakistan. The target was to
train 1600 participants, ranging from vice chancellors to rectors, registrars, treasures and directors etc.
The Project started late in 2011. The end of the project was in December, 2015. The trainings could not
be kicked off until March 2015. There was hardly any time left for a formal training needs assessment
(TNA). The Project team decided to offer the trainings through two implementation partners (IPs),
selected from the public sector institutions, having repute and experience in the field of training.
Each IP had to conduct the training on 8 modules such as Leadership and Change Management, Human
Resource Management, and Quality Management etc. In view of the big number of the staff members to
be trained (800 per IP), for each module the participants were divided into three batches. Each batch
consisted of around 35 participants.

Training Evaluation: The SOPs

It was decided that the trainings will be evaluated on the first two levels: i.e. Reaction and Learning,
using the Kirkpatrick 4-level Model of training evaluation. As for the remaining two levels, the project life
did not allow sufficient time.
The IPs were supposed to design a format to assess the reaction of the participants to the various
elements of training such as the knowledge and delivery of facilitators, the contents of course materials,
food, accommodation and venue of the training. The tests were to be shared with the M&E team of the
TESP so that they could ensure that the tests were designed to fully capture the feedback of the
participants. Once the tests were approved by the M&E team of the TESP, the IPs could use them for
evaluation. The results of the test were to be made a part of the training report. The payment to the IP
was conditional to the acceptance of the report.

The Challenge
Since both the IPs used different forms, their information was not mutually comparable. So, we were
unable to decide who was performing better.

The Level 1 of Training Evaluation/Awab

From the analysis of Level-1 evaluation, we expected to find out the answers to the following
questions:
1. How satisfied were the participants with the knowledge, experience, and delivery of the
facilitators?
2. How much were the training contents relevant to the needs of the participants?
3. How effective was the course material in improving the understanding of the
participants?
4. How much satisfied were the participants with the food, accommodation and the venue
of the training?
5. What were the areas for improvement?
Though the reports of the IPs contained answers to some of the above-cited questions, we were
unable to compare the results provided by one IP with the other one.

The Solution
In order to meet the above challenge, we decided to design a standard format to capture
mutually comparable results. The format is attached at the end as Annexure 1.
Some Salient Feature of the Level-1 Evaluation Format:
iA Mix of Quantitative and Qualitative Information
While the scores obtained against each variable provide crisp data to assess the success of the
training, the qualitative information provides breadth to the information by putting things in
appropriate context. For instance, if a participant rates the training venue low in his/her
assessment, he/she can narrate things such as the place was noisy or the chair was not
comfortable etc.
iiFlexibility in Analysis
The format is flexible enough to accommodate different levels of analyses: e.g. assessment of a
facilitator or comparison of multiple facilitators within a training or comparing the overall
feedback on a training batch with another batch.
In Table 1, given below, the cumulative assessment of participants about a trainer has been
analyzed. The rating given by participants has been marked on a Likert scale and then the scores
obtained against each variable have been calculated in terms of average and percentage. This
makes the comparison very easy.
Table 1: Assessment of a Trainer

Trainer 3
Questions

Rating
2

The Level 1 of Training Evaluation/Awab

Strongly Agree Disagree Strongly Mean

Agree
Disagree

Key
Areas
Facilitator

Delivery

Course
Contents

Course
Materials

1. The facilitator had

sufficient knowledge on
the topic.
2. The facilitator had
sufficient experience
related to the topic.
3. There were sufficient
activities and exercises to
generate my interest in
the training.
4. I was well engaged with
what was going on during
the training.
5. The information in this
training was relevant and
applicable to my work.
6. I sufficiently learned new
things from this training.
7. The course materials were
very helpful in
understanding the course
contents.

Ev. %

12

14

3.20

80.00

10

15

3.14

78.45

15

2.63

65.83

14

2.63

65.83

18

3.00

75.00

15

2.93

73.33

16

2.93

73.33

2.92

73.11

Average:-

Figure 1, given below, illustrates how in a training on IT management, the participants expressed
their satisfaction with different trainers. Also, it illustrates how the perception of learning1 varies
in accordance with different attributes of the trainers. From this chart, we also learn the strongest
areas and the weakest areas of the training. In terms of knowledge and experience of the
facilitators, the participants were most satisfied with the trainers. As for the weakest areas, lack
of engagement with participants and inability of the trainers to initiate different activities and
exercises of interest have been identified as the weakest areas of the training. It is interesting to
discover that the perception of learning is more correlated with the ability of the facilitator to
engage the participants in activities and exercises rather than with the knowledge and experience
of the facilitators.

It is worth noting here that what we can measure at level 1 of evaluation is only the perception of learning. The
actual learning is measured at level-2, through pre & post-tests.

The Level 1 of Training Evaluation/Awab

Performance of Training Facilitators

100.00

100.00

90.00

90.00

80.00

80.00

70.00

70.00

60.00

60.00

50.00

50.00

40.00

40.00

30.00

30.00

20.00

20.00

10.00

10.00

0.00

0.00
Trainer 1

Trainer 2

Trainer 3

Trainer 4

Trainer 5

Trainer 6

Trainer 7

Trainer 8

The facilitator had sufficient knowledge on the topic.

The facilitator had sufficient experience related to the topic.
There were sufficient activities and exercises to generate my interest in the training.
I was well engaged with what was going on during the training.
The information in this training was relevant and applicable to my work.
The course materials were very helpful in understanding the course contents.
I sufficiently learned new things from this training.
Figure 1: Comparison of Multiple Trainers

Similarly, the format can inform us about other elements of training where improvement is
needed the most. As shown in the Figure 2 below, the participants expressed their satisfaction
with all logistic arrangements except for the food quality which is identified as an area for
improvement.

The Level 1 of Training Evaluation/Awab

Overall Rating on Different Elements of Training

100.00
90.00
80.00
70.00
60.00
50.00
40.00
30.00
20.00
10.00
0.00
Venue

Accommodation

Food

Coordination

Figure 2: Overall Rating on Different Elements of Training

Although, the form had provided the option to explain how the food quality did not meet the
expectations of the participants, none of the participants gave any specific comments on food.
This calls for further investigation and documenting the oral comments by the training team, so
that it can be taken care of, in future.
iiiAccountability
The format also calculates the overall rating of the training, which can be compared to already
established benchmarks. These benchmarks can be absolute or relative. Absolute bench marks
are based on best practices of the industry, while relative benchmarks can be the scores obtained
by a competitor or at least can be based on previous best practices of the same trainer. Thus, the
trainer can be rewarded for meeting the benchmark or can also be held accountable for not being
able to meet the benchmarks.
ivPlanning for Improved Quality
The format can also help in planning for future trainings. Once the problem areas are surfaced, it
is easy to reflect on solutions, in consultation with the stakeholders such as the trainees, the
trainers and different service providers for food, accommodation and the venue of the training.

The Level 1 of Training Evaluation/Awab

To view the data analysis on Excel sheet, please follow the link: https://goo.gl/i4m3EG

The Level 1 of Training Evaluation/Awab

Annexure-1
Training Evaluation Form (Level 1)
Estimated time required to fill this form: 15 minutes
Date:

Training Name:

Dear Participant,
Filling out this questionnaire is very important. Your candid feedback will help us in our continued
learning and improving our future endeavors.
Please answer the questions as fully as possible.
Please indicate to what degree you agree with each statement using this rating scale:
1=Strongly Disagree
2= Disagree
3= Agree
4= Strongly Agree
Thank you very much for your support!
Section A
Name of the Facilitator:
Key Areas
Questions
Facilitator

1.
2.

Rating

The facilitator had sufficient knowledge on the topic.

The facilitator had sufficient experience related to the topic.

1 2 3 4
1 2 3 4

There were sufficient activities and exercises to generate my

interest in the training.
I was well engaged with what was going on during the training.

1 2 3 4

The information in this training was relevant and applicable to

my work.
I sufficiently learned new things from this training.

1 2 3 4

Additional Comments:

Delivery

3.
4.

1 2 3 4

Additional Comments:

Course
Contents

5.
6.

1 2 3 4

Additional Comments:

7. The course materials were very helpful in understanding the

Course
course contents.
Materials
Additional Comments:

(Please repeat the section A if there are multiple facilitators)

1 2 3 4

The Level 1 of Training Evaluation/Awab

Section B
Please write N/A where not applicable.
Key Areas
Questions
8.

Venue

9.

Rating

I found the training venue comfortable and conducive for

learning.
I felt comfortable with the accommodation.

1 2 3 4
1 2 3 4

Accommodation
Food
Coordination
Overall

10. I am satisfied with the food quality.

1 2 3 4

11. I am satisfied with how the host institution staff coordinated

and communicated with me regarding the training.

1 2 3 4

12. Overall, the training met my expectations.

13. I am clear on how to apply what I learned.

1 2 3 4
1 2 3 4

14.

15.

16.

17.

The most important thing I liked about the training

The most important thing I disliked about the training

If you are going to bring some changes, learned during the training, what results do you expect to
achieve?

Please share any other suggestions or comments you may have.

(If you authorize us to use your comments in marketing materials, please mention below your
name, designation and institution)