Session 1 Overview of

Monitoring and Evaluation

M&E Training Workshop
Rathnapura
15 June 2012
Overview
Project planning and M&E, and learning cycle
The logical framework
Purpose of M&E
Elements of M&E system
Principles

The project planning M&E and
learning cycle

Annual Cycles within overall Project cycle

Project period
Annual plan M&E Annual plan M&E Annual plan M&E Annual plan M&E
The logical framework as basic
planning and M&E tool
Objective
Goal - Impact
Outcome
Output
Inputs and activities
Example
Change in poverty rate
Improved services, adoption of
new technology
Number of persons well trained,
completion of infrastructure
Resources provided, training
activity
Within
project
control
Outside
project
control
Definitions of M&E
Monitoring is the continuous collection of data on
specified indicators to implementation in relation to
activity schedules and
expenditure of allocated funds, and
its progress and achievements in relation to its objectives.
Evaluation is the periodic assessment of the design,
implementation, outcomes and impact of a
development intervention. It should assess
the relevance and
achievement of objectives,
implementation performance in terms of effectiveness and
efficiency, and
the nature, distribution and sustainability of impacts
Source: OECD, 2002; Casley & Kumar, 1987
Linking project design and Evaluation
criteria

Monitoring focuses mainly inputs and
outputs

Monitoring
Evaluation focuses mainly on Impact
and outcomes

EVALUATION
Purpose of M&E and Learning
At project level monitoring and evaluation should:
provide regular and timely feedback on project
results and performance and need for
corrections;
identify problems early and propose solutions;
monitor access to project services and outcomes
by the target population;
evaluate achievement of project operational and
strategic management objectives;
Support learning and adaptation;
incorporate stakeholder views and promote
participation, ownership and accountability.

Purpose (continued)
At a sector level monitoring and evaluation can:
Improve programme design from MTR and
evaluation and knowledge of what works,
what does not and why
provide the evidence for building consensus
between stakeholders
Elements of an M&E system

4
3
2
1
The main components of a project M&E system
1. Logframe and targets:
Clear statements of measurable objectives for the project and its
components.
A structured set of indicators covering
A sound regular target setting /planning system
2. Data collection mechanisms
That can record progress over time, including baselines
A means to compare progress and achievements against targets.
With evaluation framework and methodology that helps explain cause
effect
3-4. Reporting and use of information, feeding into
action:
Process for analyzing findings and agreeing on progress and lessons
Clear mechanisms for reporting and use of M&E results in decision making
Using information for learning and strengthening future actions with
stakeholders.
Organizational mechanisms
Organizational arrangements for data collection, management, analysis
and reporting. Management Information System (MIS) is part of the above
Principles
It must reflect real and important results
Keep it as simple as possible
Should be meaningful and user friendly to
ministries and villagers
Consider if M&E system should be
mainstreamed and sustained, not only for
project
Build it up in participatory way, so it suits local
condition

Workshop Background
and Structure
Rationale
M&E staff different levels of training and experience
Need to have complete overview of M&E system and
concerns
There have been continuing challenges with regards to
information flows in the project (e.g. MIS)
Heavy burden of work for officers
How to engage other technical staff and communities
Need to streamline and organize the operations of the
M&E system

Some expected training workshop
outputs
Increase knowledge and skills of M&E officers
Greater understanding and clarity of SCDLIP
logical framework
Upgraded tools, such as results frameworks
and indicators, and formats
Recommendations on improved effectives and
efficiency of information flow
Action plan for implementing strengthened
project M&E system

What would be good outcomes of the
M&E training workshop
M&E staff can better respond to and improve
M&E system
Rapid and accurate reporting
Participatory M&E system
More effective reporting for managers
Improved feedback to communities and
stakeholders



Strengthen and build the M&E team !!
Day 1
Introduction
Session 1 Setting goals and objectives.
Part 1: Deepening understanding of the logframe
Part 2: The SCDLIP logframe and results framework
Lunch
Part 3: Group work - to review and refine SCDLIP
logframe


Day 2
Session 2: Indicators
Part 1: Theoretical approaches
Part 2. Revisit SCDLIP indicators: Results framework
(RF), QIMM and MPR
Lunch
Part 3: Group work - reviewing indicators and
formats of RF, QIMM and MPR
Part 4: Developing common reporting format
Day 3
Session 3. Data Collection and Management
Part 1: Theoretical
Part 2: Planning and measuring results
Part 3: Developing and M&E system
Part 4: Capturing good practices and lessons
Lunch
Part 5: Group work - Making the information flow
work
Day 4
Session 4: Preparing and Action Plan
Part 1: Review of previous days results and
outstanding concerns
Part 2: Group planning
Lunch
Part 3: Consolidation of action plan
Part 4: Feedback to senior panel and commitments
Closing ceremony


Results in relation to SCDLIP Village
development process
The SCDLIP village development
process as in COMS
Mobilization and
formation of VO
Planning stage
Implementation of
VDP: CBF, LSF and
Infrastructure
Monitoring stage:
O&M, continued
inclusion and
liveihoods
Mobilization and
formation of VO
Planning stage
Implementation of
VDP: CBF, LSF and
Infrastructure
Monitoring stage:
O&M, continued
inclusion and
liveihoods
Training
Training,
agreement
s
Resources,
training
Follow
up
VO
registered
VDP
Infra-
structure
built,
VSCO
established,
etc
VDP 2,
O&M system
established
Convergence
Better road
access
Empowered
community
leaders
Improved use
of loans
Value added
from market
Sustained
organizations
INPUTS ACTIVITIES OUTPUTS OUTCOMES
INPUTS ACTIVITIES OUTPUTS OUTCOMES PDO Outcome /
Impact
Mobilization and
formation of VO
Planning stage
Implementation of
VDP: CBF, LSF and
Infrastructure
Monitoring stage:
O&M, continued
inclusion and
liveihoods
Training
Training,
agreements
Resources,
training
Follow up
VO
registered
VDP
Infra-
structure
built,
VSCO
established,
etc
VDP 2,
O&M system
established
Convergence
Better road
access
Empowered
community
leaders
Improved use
of loans
Value added
from market
Sustained
organizations
Enhance
quality
of life of
the
poor
Increase
incomes
of the
poor
Working group:
Developing component wise logframe
Looking at results framework
Long term impact
Project development
outcome
Changes in peoples lives
Intermediate outcome Sustainability
Output
Inputs - activities
World Bank
Results
framework
focus
QIMM
National impact
Project development
outcome
Changes in peoples lives
Intermediate outcome Sustainability
Output Key milestones
Inputs - activities
Project
management
Output
indicators
By major component
- At end of project -
National impact
Project development
outcome
Changes in peoples lives
Intermediate outcome Changes in practices, and
Sustainability
Output Key milestones
Inputs - activities
Working Groups
1. Village organization
Possible outcomes: Maturity, sustainability , services to members
2. Intra-village Infrastructure
Possible outcomes: Operation and maintenance
Providing water, easier access
3. Microfinance and VSCO
Possible outcomes: Sustainability
Increased loans to more people
4. Livelihoods and PPP
Mature producer groups
Continued operation and stronger livelihoods activities
5. PS and interconnectivity
Sustained PS support
Better acces
Operation and maintenance
6. Convergence
Mainstreaming
Continued support programmes



Session 2 - Indicators
Revisiting SCDLIP Results Framework
Indicators
World Bank Results Framework

Developing a Village Development
Process Progress-card
Mobilization and
formation of VO
Planning stage
Implementation of
VDP: CBF, LSF and
Infrastructure
Monitoring stage:
O&M, continued
inclusion and
liveihoods
Training
Training,
agreement
s
Resources,
training
Follow
up
VO
registered
VDP
Infra-
structure
built,
VSCO
established,
etc
VDP 2,
O&M system
established
Convergence
Better road
access
Empowered
community
leaders
Improved use
of loans
Value added
from market
Sustained
organizations
INPUTS ACTIVITIES OUTPUTS OUTCOMES
Using the
QIMM
Mobilization and
formation of VO
Planning stage
Implementation of
VDP: CBF, LSF and
Infrastructure
Monitoring stage:
O&M, continued
inclusion and
liveihoods
VO establishment
and operations
Community
Infrastructure
VSCO and
livelihoods
Sustainability
Basic VO Capacity building

Community infrastructure

Livelihoods Progress

Second Generation institutions

PS and Convergence

Developing a village progress scorecard
Key indicator progress
VO establishment
and operations
1
2
3
Community
Infrastructure

1
2
3
Microfinance and
VSCO

1
2
3
Livelihoods 1
2
3
Handover and
Convergence
1
2
3
QIMM Progress monitoring
Using the Progress-card at Village level
Key indicator
progress
VO establishment and
operations
1
2
3
Community
Infrastructure

1
2
3
Microfinance and VSCO

1
2
3
Livelihoods 1
2
3
Handover and
Convergence
1
2
3
Progress
against VDP
Progress against
Quarter target
Progress-card to assist grading and
quarterly progress
Progress
against overall
targets
Quarterly
progress
Overview consolidated progress-card
Villages
Consolidated
District
Comparing Batches
0 3 2 1 4
Comparing Districts
District 1 District 4 District 3 District 2 District 5 District 6
SESSION 3 - PART 2
Planning and measuring outcomes
Annual Cycles within overall Project cycle

Project period
Annual plan M&E Annual plan M&E Annual plan M&E Annual plan M&E
WB project cycle
Project
Appraisal
Document
Imple-
mentation
completion
and results
report (ICRR)
Mid term
review
Impact
assessment
study

Review
study

Baseline study


Project
Plan


Revised
Project
Plan

Logframe
Next
phase
New
projec
t
New
policy
Annual Project Planning and reporting cycle
VO Annual
Plan (part of
VDP)
District Plan
Consolidated
National
Annual Plan
Project
budget and
thematic
priorities
District
allocation
and
thematic
priorities
AUGUST SEPTEMBER OCTOBER
QPR3 &
DRAFT
ANNUAL
REPORT
QPR 1 &2

Focus of reporting
Objective
Monthly Quarterly Annual Baseline,
MTR, End
Project
development
outcome
Emerging
impacts
Yes
Lessons
Intermediate
outcome

Yes
innovations
issues
Yes
Lessons
learned
Yes
lessons
Output


Yes Yes Yes Summarise
Good
process
Inputs activities


Yes (yes)
Expenditure
budget/
expenditure
Project
efficiency
EFA
Presenting indicators
From 1
st
Quarter Progress Report
- Capacity building fund -

Presenting data
Phase 1 Phase II Phase 1 Phase II
Actual progress ( ) against
total expected budget (Rs
million) ( )
% Progress against
quarter ( ) and 2012
target ( )
500
97%
54%
100%
Percent of VSCOs with Portfolio at Risk (PAR) 30
days, which is less than 50%
Phase I
Phase II
100%
50%
2010 2011 2012 2013 2014
Minimum
VO operational cost recovery (income/expenses)
1
2010 2011 2012 2013 2014
Debt
Profit
Batch 1
Batch 3
Batch 2
TARGET ?
Human Resources (2012 1
st
QPR)
Average progress of the % Physical
completion in Village Infrastructure

>95% >80%
Target
Measuring outcomes and impacts
Basic methods Mainly Primary data
Household condition and income
Household survey
Household panel survey, longitudinal time study
Oral testimonies
Organizational assessment
Survey and interviews with businesses
Focus groups with stakeholders
Participatory assessment
Service delivery
Thematic studies e.g. for infrastructure interventions
User surveys
Case studies
Livelihoods and economic
Farm monitoring and survey
Market survey and interviews
Economic and financial analysis
Key informant interviews
Stakeholder workshops for consensus on findings and lessons
Measuring project outcome change:
Attribution: how do we know result is because of
project not other effect?
Using control!
2009 2010 2011 2012 2013 2014
Project
beneficiary
HH
Non-
Project HH
CONTROL
Incremental
difference
Sampling: districts, batches, villages and households
District 1
District 3
District 2
Batch 1
Batch 1
Batch 1
Batch 3
Batch 3
Non-
project
Non-
project
Non-
project
Non-
project
Randomly sampled HH
Measuring difference in difference
2009 2010 2011 2012 2013 2014
Project
beneficiary
HH
Non-
Project HH
Incremental
Difference in
difference
Some comments
Need to design baseline and impact assessment when
designing project - ideally
Samples sizes have to be big to be significantly
meaningful have to use formula
Questionnaires need to be constructed carefully and
tested specialist task
Analysis needs econometric and statistical tools
Specialized independent teams often best because it
takes training and time
Reports need to be short and full of user friendly
graphs! (Tables in Annexes)

Capturing lessons and good (and bad)
practices
Facts, findings, lessons, recommendations
What does a good practice or case
study provide?
Provide context
Why is it important
Provide facts about outcomes (and impacts?) - who
benefited how?
Describe key processes which led to result
What are critical factors which contribute to result
Get oral testimony, beneficiary focus group and key
informant interviews
Discuss the findings with stakeholders
Draw lessons which are more widely applicable, and
Applicable under what conditions? May be special case!
Provide recommendations
Lessons
Can be for future activities in the project
For other projects and programmes:
in same sector,
in different sector but similar processes
For policy change
For other initiatives in other countries

CHECK 1: is there a consensus on lessons based on
facts?
CHECK 2: are the conditions and capacities elsewhere
sufficient to take up lessons
SESSION 3 Data Collection and
Information Flow
Group Work:
Making the information flow work two way process!
Group 1: Village <> PIU


Group 2: PIU <> District


Group 3: District <> National

Groups will look at
What Issues comments,

Especially on roles
Improvements and
changes
Data collection
Information flow
Analysis Reporting

Use and feedback of
information and
reporting
Some issues?
Roles and responsibilities - who should be doing what, so
that they find it useful for themselves
Whose information?
Data quality, accuracy checks and balances, how to know
what is up to date
Efficient and effective data gathering, data entry process
Who is information for? User friendliness and ownership
Reporting both ways, not just one way
Who analyses what
What reporting matters: results and trends, progress
against targets, appropriate comparisons
Capacity and staffing concerns