HRD 13 Evaluation

13.
CHAPTER 13: EVALUATION

• MISCONCEPTIONS ABOUT EVALUATION
• ASSESSMENT OF LEARNING
• KIRKPATRICK’S FOUR LEVELS
• THE PRESAGE FACTORS
• THE SCIENTIFIC MODELS
• COST-BENEFIT ANALYSIS
• THE EVALUATION PLAN
• THE EVALUATION REPORT
• THE LEGITIMATE SYSTEM
• THE SHADOW SYSTEM
• THE NEED FOR DIALECTIC THINKING
Human Resource Development: Principles and Practice By Brian Delahaye
13.2

13.3
MISCONCEPTIONS ABOUT
EVALUATION
While evaluation is seen as professionally and theoretically
desirable, some have doubts about its worth.
1. It is too costly. This misconception is based on a very short-
term view of costs. The cost of risk is also part of the equation.
These risks include not knowing if mistakes were made and not
knowing how to avoid these mistakes in the future.
2. The measures are not exact. This misconception does not

recognise that any complex decision is based on less than perfect
information.
3. It is too difficult. True, most evaluation recommendations

apply more obviously to the legitimate system. However, the
developmental efforts of the shadow system can be evaluated.
13.4
THE ROLE OF EVALUATION

The role of the evaluation stage is four-fold:
1. to measure what change has occurred
2. to improve the other three stages — investigation, design and

implementation
3. to see if the change is attributable to the learning episode
4. to see if the amount of change is worthwhile.

13.5

13.6
ASSESSMENT OF LEARNING
Assessment addresses the question ‘What changes have occurred
in the mind of the learner as a result of the learning episode?’. To
measure this change, the learning has to be converted into a
behaviour that can be observed and this has some weaknesses:
 The behaviour is only a sample of the learning.

 The behaviour represents explicit knowledge only.
 The learning generation processes may take some time.
 The test has to tap into the appropriate knowledge.
 The test has to tap into the same potential knowledge each
time.
There are several issues that need to be considered when
planning assessment.

13.7
TYPES OF ASSESSMENT
Assessment can be divided into six types:
 skill tests — are used for procedural skills

 objective written tests — e.g. multiple choice
 subjective written tests — e.g. essays
 performance tests — used to assess complex processes. The
examiner may asses the process itself or the product often using a
pre-designed observation form (see Fig.13.1 on page 346).
 learning diaries — Journals written by the learner; good for
assessing externalisation
 portfolio assessment — collections of the learner’s work;
takes a long term view, focuses on quality and the learner reflects
on own work.

13.8
THE MEANING OF SCORES
When a test has been examined and given a quantitative score the
result is called a raw score.
This raw score can be interpreted in a number of ways:

 criterion-referenced scoring — the learner’s score is
compared to a ‘pass’ mark
 norm-referenced scoring — the raw score is compared to the
average of a nominated group
 formative assessment — the score is used for developmental
feedback
 summative assessment — the score is used for evidence of
learning.

13.9
OVERLAP OF THE FOUR HRD

PROCESSES
The assessment of learning plays a dual role — being critical to
both the implementation stage and the evaluation stage of HRD.
In the implementation stage, assessment provides dynamic

feedback. Firstly, this feedback is often immediate. Secondly, it
provides irrefutable evidence that the learner finds difficult to
ignore.
In the evaluation stage, assessment provides initial evidence of

the success or otherwise of the learning experience. This
feedback may be taken into account immediately (e.g. the HR
developer changes the learning processes) or later (e.g.
recommendations in the Evaluation Report).

13.10

13.11
KIRKPATRICK’S FOUR LEVELS

Kirkpatrick’s model is based on four levels of ascending order:
1. reaction — measures the reactions of the learners to the

learning episode
2. learning — measures the knowledge gained by the learners.
Has been discussed under ‘Assessment of Learning’.
3. behaviour — examines the change of behaviour of the learner
on the job
4. results — measures the impact of the learning on the
organisation as a whole
Several comments can be made about Kirkpatrick’s model.

13.12
COMMENTS ON KIRKPATRICK’S
MODEL
The strengths of Kirkpatrick’s model include:
 simple and seemingly robust model
 recognises the impact of a learning episode should extend
beyond the individual learner
 provides an easily remembered checklist
 each level provides a unique examination of HRD.
The criticisms of Kirkpatrick’s model include:

 the levels are not co-dependent
 the model concentrates on processes and outputs, not inputs
 the initial level is easier but measurement of the deeper levels
are more susceptible to contamination.

13.13

13.14
THE PRESAGE FACTORS

Brinkerhoff’s model proposes six levels:
Stage I: Evaluate needs and goals ( this is similar to the HRDNI).

Stage II: Evaluate the HR design
Stage III: Evaluate implementation
Stage IV: Evaluate learning (similar to Kirkpatrick’s level 1)
Stage V: Evaluate usage and endurance of learning (similar to
Kirkpatrick’s level 3)
Stage VI: Evaluate pay-off (similar to Kirkpatrick’s level 4).
Brinkerhoff, then, adds two that occur prior to the learning

episode — evaluate HR design and evaluate implementation.
These are called presage factors.

13.15
THE PRESAGE FACTORS (Continued)

Stage II, evaluating the HR design focuses on the quality of the
HR plan (i.e. the training or workshop program).
A stage II evaluation should occur when:

 the design is unique or experimental
 the costs of mounting the program are high
 the HRD issues are crucial
 the participant groups are volatile, influential or demanding.
The design should be reviewed by a variety of stakeholders.

13.16
THE PRESAGE FACTORS (Continued)
Stage III, evaluating during the implementation provides data

that will help ‘shepherd’ the learning event to a successful
conclusion.
This highlights the covert activity of the conducting HR

developer where the program is continually monitored to
compare reality with the program plan.
The HR developer can also be monitored, usually by another HR

developer. However, the ‘ big brother’ syndrome has to be
avoided.

13.17
TIME OUT
To this stage of the discussion, we have examined assessment of
learning, Kirkpatrick’s model and Brinkerhoff’s model. This has
answered two of the roles of evaluation:
1. to identify what change has occurred

2. to improve the other three stages.
The other two roles will now be addressed:
3. to see if the change is attributable to the learning episode. This

will be examined by looking at the SCIENTIFIC MODELS.
4. to see if the amount of change was worthwhile. This will be
examined by looking at COST BENEFIT ANALYSIS.

13.18

13.19
THE SCIENTIFIC MODELS

The scientific models are based on the experimental methods
used in research laboratories and are used to demonstrate
causality. They also assume that changes can be measured.
 Post-test = learning experience/evaluation

 Pre-test - Post-test = evaluation/learning experience/evaluation
 Time series evaluation = pre-test/pre-test/pre-test/learning
experience/post-test/post-test/post-test
 Control group =
Experimental group — pre-test/learning experience/post-test
Control group — pre-test/usual duty/ post-test
 The Solomon four uses three control groups and the
experimental group
The HR developer has to compare the costs of the designs

compared to the expected benefits gained.
13.20

13.21
COST-BENEFIT ANALYSIS
Cost-benefit analysis (CBA) is used to see if the amount of
change was worthwhile. It is based on the premise:
 identify the costs, in dollar terms, for the learning experience
 identify the the benefits accruing from the learning
experience, in dollar terms
 the ratio between the costs and benefits should be in favour
of the benefits.
There are two problems often encountered:

 converting the benefits into monetary values
 deciding on the cut-off points.
There are two good reasons for conducting a CBA. Firstly, the
survival of the HR section may depend on proving the value of
development. Secondly, it helps in the decision of selecting the
learning programs that have the most impact.
13.22

13.23
THE EVALUATION PLAN

Planning for evaluation commences during the design stage. This
ensures that the appropriate evaluation occurs at the appropriate
time. It also allows the coordination of developmental and
judgemental evaluation. Planning or evaluation should include:
 develop the assessment for developmental purposes first
 incorporate this into the evaluation plus any further
assessment required
 decide what presage variables will be evaluated and when
 incorporate appropriate HRDNI investigation instruments
 design daily and course/workshop reaction sheets
 design pre-test and post-test instruments, if appropriate
 identify and plan methods to be used at Stage V and VI
 if CBA is to be used, arrange collection of costs and benefits
 prepare a budget
 send evaluation plan to staff who are affected.
13.24

13.25
THE EVALUATION REPORT

Once the evaluation has been completed a report should be
completed for the key stakeholders. The evaluation plan has a
communication role, decision making role and becomes an
historical document. This report should include:
 an executive summary
 a findings/recommendations section
 a contents list
 the main body, which should include:
 reasons for the evaluation
 list of personnel involved
 discussion of the various types of evaluation undertaken
 a discussion on the findings, the options and the
conclusions
 a list of, and discussion of, the recommendations
 appendices.
13.26

13.27
THE LEGITIMATE SYSTEM

Most of what has been discussed in this chapter applies directly
to the legitimate system. The methods mainly rely on negative
feedback systems. The methods also emphasise the legitimate
systems focus on the efficient use of resources and the
immediate survival of the organisation.
For the legitimate system, one of the advantages of competency-

based learning has been the recognition of the strong link
between learning in a planned program and the application of
that learning in the workplace.
It should also be recognised that, if the assessment process fails

to detect a lack of comprehension on the part of the learners, the
HR developer could be held liable.

13.28

13.29
THE SHADOW SYSTEM

Evaluation in the shadow system is a very delicate affair. The
use, by a third party, of the methods described in this chapter
results in negative feedback loops — and this kills the creativity
needed in the shadow system.
The main evaluation method for the shadow system is the use of
values systems. The manger instills a particular value in the
system and checks for evidence for this value.
The actors in the shadow system, however, should use the

evaluation methods on themselves and their own endeavours.

13.30

13.31
THE NEED FOR DIALECTIC

THINKING
Some of the misperceptions about evaluation can be explained
by the concept of dialectic thinking (see Chapter 2).
Evaluation has a number of opposing characteristics:

 evaluation can be both developmental and judgemental
 the more objective the measure used, the less rich the insights
— and vice versa
 liberal education vs. vocational education.
Evaluation serves and supports a number of dual roles.

Balancing these conflicting roles is the new challenge as
knowledge is a valuable but delicate resource.

HRD 13 Evaluation

Загружено:

Сведения о документе

Оригинальное название

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

HRD 13 Evaluation

Загружено:

Авторское право:

Доступные форматы

13.

CHAPTER 13: EVALUATION

CHAPTER 13: EVALUATION

• MISCONCEPTIONS ABOUT EVALUATION

2. The measures are not exact. This misconception does not

3. It is too difficult. True, most evaluation recommendations

THE ROLE OF EVALUATION

1. to measure what change has occurred

2. to improve the other three stages — investigation, design and

3. to see if the change is attributable to the learning episode

4. to see if the amount of change is worthwhile.

Human Resource Development: Principles and Practice By Brian Delahaye

CHAPTER 13: EVALUATION

• MISCONCEPTIONS ABOUT EVALUATION

 The behaviour is only a sample of the learning.

Human Resource Development: Principles and Practice By Brian Delahaye

 skill tests — are used for procedural skills

Human Resource Development: Principles and Practice By Brian Delahaye

THE MEANING OF SCORES

This raw score can be interpreted in a number of ways:

Human Resource Development: Principles and Practice By Brian Delahaye

OVERLAP OF THE FOUR HRD

In the implementation stage, assessment provides dynamic

In the evaluation stage, assessment provides initial evidence of

Human Resource Development: Principles and Practice By Brian Delahaye

CHAPTER 13: EVALUATION

• MISCONCEPTIONS ABOUT EVALUATION

KIRKPATRICK’S FOUR LEVELS

1. reaction — measures the reactions of the learners to the

Several comments can be made about Kirkpatrick’s model.

Human Resource Development: Principles and Practice By Brian Delahaye

The criticisms of Kirkpatrick’s model include:

Human Resource Development: Principles and Practice By Brian Delahaye

CHAPTER 13: EVALUATION

• MISCONCEPTIONS ABOUT EVALUATION

THE PRESAGE FACTORS

Stage I: Evaluate needs and goals ( this is similar to the HRDNI).

Brinkerhoff, then, adds two that occur prior to the learning

Human Resource Development: Principles and Practice By Brian Delahaye

THE PRESAGE FACTORS (Continued)

A stage II evaluation should occur when:

The design should be reviewed by a variety of stakeholders.

Human Resource Development: Principles and Practice By Brian Delahaye

THE PRESAGE FACTORS (Continued)

Stage III, evaluating during the implementation provides data

This highlights the covert activity of the conducting HR

The HR developer can also be monitored, usually by another HR

Human Resource Development: Principles and Practice By Brian Delahaye

1. to identify what change has occurred

The other two roles will now be addressed:

3. to see if the change is attributable to the learning episode. This

Human Resource Development: Principles and Practice By Brian Delahaye

CHAPTER 13: EVALUATION

• MISCONCEPTIONS ABOUT EVALUATION

THE SCIENTIFIC MODELS

 Post-test = learning experience/evaluation

The HR developer has to compare the costs of the designs

CHAPTER 13: EVALUATION

• MISCONCEPTIONS ABOUT EVALUATION

There are two problems often encountered:

CHAPTER 13: EVALUATION

• MISCONCEPTIONS ABOUT EVALUATION

THE EVALUATION PLAN