Monitoring and Evaluation

“Why did the chicken

cross the road?”

ICHD 08-09/ HPF Module

Jurrien Toonen,
09/08/20 1
 Learning objectives

At the end of this session the participants will be able

to:
 Explain and discuss the purpose of and approaches
to evaluation in relation to public health
programmes.
 Indicate and select relevant stakeholders to be
involved in the process
 Specify the elements of a public health programme
to be evaluated

03-06-05 KIT-Dev for “evaluation society” “Setting the scene” 2

 The issue is not that we need
more data ………..

……… but better use may be made of existing data

03-06-05 KIT-Dev for “evaluation society” “Setting the scene” 3

 Set-up of session

 What are we talking about?

 Clarification of terms
 Principles behind MandE
 Changes in context (project to program)
 Current challenges in MandE
 Casestudy: set-up of a M&E system on health
reform in Egypt
 Synthesis

03-06-05 KIT-Dev for “evaluation society” “Setting the scene” 4

 Definitions

Talk 5 mins with your neighbour

 What is monitoring?
 What is evalation?
 (What is the difference between them?)
 What types of evaluation are there?

03-06-05 KIT-Dev for “evaluation society” “Setting the scene” 5

 Definitions (1)

Monitoring:
 EU 2005: Continuous and systematic process carried
during the duration of an intervention, generates data
on implementation, in order to correct

 WHO: Mechanism of data collection, processing and

interpretation with the purpose to provide information
about progress of activities and to enable management
decisions.

03-06-05 KIT-Dev for “evaluation society” “Setting the scene” 6

 Definitions (2)

Evaluation
 DFID: Systematic and objective assessment of the
relevance, efficiency, effectiveness, impact and
sustainability of development activities

 EU (2005): Judgement of interventions according to

their results/ impacts and needs they aim to satisfy

03-06-05 KIT-Dev for “evaluation society” “Setting the scene” 7

 Definitions (3)

Audit
 Originally process by which probity of operations and activities
of an organisation was examined (internal audit) a report on
the annual accounts produced (external audit).
 Now more widely, e.g., clinical audits, management audit
effectiveness and efficiency of organisational and management
arrangements, etc.
In the context of donor relationships:
 Determining whether, to what extent, organizational
procedures of financing agency have been implemented in
accordance to norms and criteria set out in advance.

03-06-05 KIT-Dev for “evaluation society” “Setting the scene” 8

 Some basic questions –
for general discussion

 Why M&E
 Who
 What
 For whom
 How: methodology
 How: tools

03-06-05 KIT-Dev for “evaluation society” “Setting the scene” 9

 The results chain:

Results chain

OBJECTIVES
TARGETS in country

INPUTS ACTIVITIES
PROJECTS
PROGRAMS
OUTPUTS
INTERMEDIATE
OUTCOMES
EFFICIENCY FINAL
OUTCOMES
IMPACT

EFFECTIVENESS

03-06-05 KIT-Dev for “evaluation society” “Setting the scene” 10

 The results chain (2)

 Relevance: extent to which objectives of the

development intervention were consistent with
beneficiaries' requirements, country needs, global
priorities and partners' and donors' policies
 Efficiency: how economically resources/inputs
were converted to results
 Effectiveness: extent to which development
intervention's objectives were achieved, taking into
account their relative importance

03-06-05 KIT-Dev for “evaluation society” “Setting the scene” 11

 The results chain (3)

 Impact: the positive and negative, primary and

secondary long term effects produced by a
development intervention, directly or indirectly,
intended or unintended
 Sustainability: the continuation of benefits from
a development intervention after major
development assistance has been completed

03-06-05 KIT-Dev for “evaluation society” “Setting the scene” 12

 the NEW context: effect on M&E?

 SWAp:
 Nat program = heart, then M&E = backbone
 Pooled funding: attribution?
 Impact? Learning oriented monitoring loop
Resource
allocatorsl
 counterfactual – experimental? Control
oriented
 Quasi- or Non- experimental? monitoring
loop

 Value for donor funding? Resource users

 Public Expenditure Review/ MTEF

 Accountability or learning?

03-06-05 KIT-Dev for “evaluation society” “Setting the scene” 13

 The NEW challenges:
Complex stakeholder setting

 Partnerships:
 Effectiveness/ added value? Or:
 Quality of the partnership?
 Effect on the “classic M&E framework”?

 Complexity of relationships
 Typology of partnerships

03-06-05 KIT-Dev for “evaluation society” “Setting the scene” 14

 The NEW challenges:
Effect on the M&E framework
Roles of WHO and DFID towards achieving the MDGs
Relevance Scope for added value by DFID
The ISP as a tool of the relationship
Appropriateness ??
Partnership structures
Partnership unity! Values and goals, coherence with corporate objectives
Trust

Management – coordination, technical and financial support

Efficiency Communication
Accessibility and mobilisation of resources

Partnership processes – stability, M&E and capacity building

Effectiveness Effectiveness through technical and policy programme tracers – pro-poor
orientation, coherence and interaction

03-06-05 KIT-Dev for “evaluation society” “Setting the scene” 15

 The NEW challenges:
Complexity
 Partnership & learning vs accountability
 Complex systems:
 collection of parts at different levels
 Interconnected/ interdependent
 Power relations
 Organizational changes –
 both in donors as in countries
 Overloaded wish-lists rather than strategies
 Difference between “problems” (known structure);
“puzzles” (structured with a specific solution) “messes” (no
defined structure, no specific solution);

03-06-05 KIT-Dev for “evaluation society” “Setting the scene” 16

 The NEW challenges:
Complex architecture
Outline of the DFID-WHO Relationship
HMG
Other
like-minded
UK
countries
Dept. of Health

DFID UK W HA
•UNCD – First Secretary Health Rep in Geneva
•Chief Human Development Adviser UN Agencies &
•DFID Health Policy Team s Programmes
•CHAD
(e.g. UNICEF;
UNAIDS; UNDG)
WHO
• Resource management
• Technical teams
•DFID country-based advisers
• DFID TA & Secondees in W HO

Partner countries W HO Regional Offices

Ministries of Health
NGOs W HO Country Offices
Communities

03-06-05 KIT-Dev for “evaluation society” “Setting the scene” 17

 The NEW challenges:
Many levels of info-users

Info needs at different levels changed

• Global Initiatives & UN bodies
• “New” roles of central Government:
• stewardship & resource allocation
• Policy development & performance of the
sector
•Decentralisation
• Deconcentration vs devolution
• Responsiveness to community needs: SIEC
• Privatization and out-sourcing
03-06-05 KIT-Dev for “evaluation society” “Setting the scene” 18
 8. Accountability to taxpayers

7. Accountability to donors

6. Accountability to higher
administration

5. Improving internal management

4. Enhancing internal learning

3. Enhancing learning in a
multi stakeholder landscape

2. Accountability to patients,
clients, members

1. Accountability to society

03-06-05 KIT-Dev for “evaluation society” “Setting the scene” 19

 The NEW challenges:
the effect on poverty (reduction)

 Poverty Reduction – M&E

 Sector program or PRSP
 MDG or “MDG-friendly PRSP”?
 MDG-indicators or “underlying factors”
 Effect on the poor
 Criteria for equitable distribution resources
 Disaggregated data for poverty – DHS or?

03-06-05 KIT-Dev for “evaluation society” “Setting the scene” 20

 New tools/ methods

Staying busy
“

is not the same as achieving results –

the challenge is
to demonstrate outcomes and impact”

03-06-05 KIT-Dev for “evaluation society” “Setting the scene” 21

 New tools/ methods
Making choices

03-06-05 KIT-Dev for “evaluation society” “Setting the scene” 22

 New tools/ methods
Type of Model
making choices (1)
Result Models
a. Goal-attainment Model
B. System Model
c. Effects Model
Explanatory process Model
Economic Model
a. Cost-efficiency
b. Cost-effectiveness
c. Cost-benefit
System Questions
To what degree have the goal(s) and/or objectives been
realized?
How has the performance functioned as a whole?
Were inputs, structure, outputs, outputs and expected
results realized?
Which outcome can be uncovered?
What effects / impact (results that can not all be controlled
by the intervention alone) can be uncovered?
What were other consequences that were not aimed for?
Is the level of activity satisfactory?
Are there implementation problems?
Did processes, needed to come to the desired results, work?
Did organisational forms in a multi-stakeholder environment
work?
Is productivity in relation to inputs satisfactory
Is effectiveness (outputs and/or objectives) in relation to
inputs satisfactory?
Is effectiveness (impact) in relation to their inputs
satisfactory?
Is utility in relation to inputs satisfactory?

03-06-05 KIT-Dev for “evaluation society” “Setting the scene” 23

 New tools/ methods
making choices (2)
Actor Model
a. Client-oriented Are clients satisfied?
b. Stakeholder Are stakeholders satisfied?
Is the system open or closed?
Which stakeholder is the principle information user?
c. Partnership What was the quality of the partnership, unity of direction?
d. Peer review Is professional quality in order?
What can others learn from this organization?
Programme-theory Model
(theory-based model) What works why – for whom in what context?
What were strengths, weaknesses, opportunities and
threats?
Is it possible to ascertain errors in program theory?

03-06-05 KIT-Dev for “evaluation society” “Setting the scene” 24

 New tools/ methods
making choices (3)

Criteria for selection

 Is the information system aiming at monitoring or at
evaluation?
 What is the leitmotiv
 Which models are best for learning purposes.
 Who is the information user:
 Adapted to the type of intervention level:
 Characteristics of the system or organisation to be
evaluated or monitored:
 Is it feasible to appreciate the results:

03-06-05 KIT-Dev for “evaluation society” “Setting the scene” 25

 New tools/ methods (1)

 Log-frame
 Results Based Management (RBM) and
 Results Based Budgeting (RBF)
 Accountability at decentralised level
 Poverty monitoring systems
 Poverty Social Impact Assessment (PSIA)
 Public Finance Management instruments (PFM)
 Value for Money analysis
 Service Delivery Surveys

03-06-05 KIT-Dev for “evaluation society” “Setting the scene” 26

 New tools/ methods (2)

 Qualitative/ process orientation

 OECD tool for budget support
 Outcome Mapping
 Most Significant Change (MSC) technique
 Result Based Management/ Budgeting
 Stakeholder analysis/ SWOT
 CORMAC and SPR-tool for policy-M&E
 Impact evaluation (?!)

03-06-05 KIT-Dev for “evaluation society” “Setting the scene” 27

 MandE the Egypt Health Reform:
criteria for indicators

 1. only be selected if the health reform may influence it ;

 2. a “minimal package of essential information”;
 3. adapted to the information users (specific needs per
level)
 4. should enable to take evidence-based decisions;
 5. should address the pace, the effects and the direction of
the HSRP;
 6. impact of the HSRef should be measured (use proxy
indicators, tracer diseases );
 7. SMART indicators.

03-06-05 KIT-Dev for “evaluation society” “Setting the scene” 28

 HIO- Central level
Information flow Non-reformed HIO Facilities
HIO intermediate level Reformed HIO Facilities Vertical
FH Fund- Central FHF Programs
Reformed MOHP facilities
TSO Non reformed PHC-Fac.

District Hospitals

HIS (District Level)

MIS- Moderrya
Central level Operational Level
Data Data
Data
provision
(TSO-Produces) (TST-Produces)

Health Reform
M&E system
MIS- MOHP
(Data base)

Data
processing
Operational Level Central level
Indicators Indicators
(TST Construct) (TSO Construct)

HIO- Central level

Health System Research FH Fund- Central

impact indicators
Demographic Health Survey TSO-users
Data
Users
FH Fund-operational

HIO-operational

Governorate HA

District

Data producers
Reformed Facilities HIO-Ref. Facilities
Data users: Non-reformed facilities HIO Non-ref. facilities

03-06-05 KIT-Dev for “evaluation society” “Setting the scene” 29

 Egypt HDS Reform M&E system

Inputs

C o n d i ti o n s B

Health Facility
Performance
Q u a l i ty o f c a re

U ti l is a ti o n Impact
!?

03-06-05 KIT-Dev for “evaluation society” “Setting the scene” 30

 Support LIC & MIC in building systems

Enable decision-makers to take evidence-based decisions

Support decision-makers in defining information needs

Adapt Indicators to the information users (situation, level)

•Simple, clear and feasible indicators and SMART
• Address the pace, the effects and the direction

• Not unnecessarily increase the workload.

• Development of systems: info-flow etc
• Linking M&E
03-06-05 KIT-Dev for “evaluation society” “Setting the scene” 31
 Further reading:

 http://www.ausaid.gov.au/ausguide/pdf/ausguidelin
e4.3.pdf: Monitoring guidelines by AusAid
 http://www.ausaid.gov.au/ausguide/part5.cfm:
Evaluation guidlines by Ausaid
 http://www.dfid.gov.uk/aboutdfid/performance/files
/guidance-evaluation.pdf: DFID with loads of extra
websites
 www.oecd.org/dac/evaluation of the OECD

03-06-05 KIT-Dev for “evaluation society” “Setting the scene” 32