PM&E for PRSPs

Challenges and Lessons

World Bank Workshop: Voices and Choices at the Macro Level Action Learning Programme on Participatory Processes for PRSPs 3-5 April, 2001. Washington DC

Presentation by Rosemary McGee and John Gaventa Institute for Development Studies Brighton, UK

The Context of PM&E

The growing importance of monitoring and evaluation
Moving participation down the project cycle: from whose reality counts? To who counts reality?

Who measures matters

From Projects to Policies Increasingly PM&E Is Used

By citizens, to hold larger programmes and institutions to account

As a tool for learning, for continuous improvement for programmes and policy implementation
As a tool for dialogue, amongst differing stakeholders across power and other differences

Recent Examples of PM&E of Largescale Policies and Programmes

In the US, citizen learning teams monitored the federal empowerment zone programme in rural areas In the UK, a number of local organisations are monitoring the affect of national neighborhood renewal programmes on local people

Examples
In the Philippines, the BATMAN coalition is using PM&E to develop indicators of good local governance, and to evaluate the impacts of the decentralisation

In Colombia, ACIN, an association of indigenous councils, uses PM&E as part of a regional planning and development process The NGO working group of the world bank, involved its members globally to monitor and evaluate the banks policies of participation

In Differing Parts of the World PM&E Processes Often Use Similar Steps

Decide who Participates Take Action Establish Goals PM&E Learning Cycle Develop Analyse Indicators Results Gather Information

Steps of the Learning Cycle

Selecting participants: who will be involved?
Intended beneficiaries? Primary stakeholders? Other stakeholders?

Choosing goals
Which goals of the policy are most important and to whom? Whats the goal of the PME process itself: accountability? Learning? Proving impact?

Developing indicators
Indicators of success, even of similar goals, will vary by stakeholders. Negotiation and flexibility if diverse voices are to be included

 Gathering information
A range of methods may be used, from qualitative to quantitative, participatory to conventional, depending on the key purpose

Analyzing and reflecting on what has been learned

A key part of the process, which if done together, can build dialogue and learning across stakeholders

Taking action
Who needs to know for change to occur? How will the results be communicated and used?

Applying the Learning Cycle: Key Lessons for PM&E of Policy

Who measures matters - involving more stakeholders will open up questions of what is most important and to whom PM&E is a social learning process, as much as a technical process, involving trust building, negotiation and dialogue, all of which take time Possibilities for PM&E of policy will vary according to the space for participation in a given context. The PM&E process itself can create space by bringing differing stakeholders together

Key Lessons (Continued)

The methods for PM&E must fit the purpose and the audience - is the PM&E for:
Learning lessons for improvement? Building dialogue and capacities for stakeholders? Proving impacts? Tracking accountability?

Norms of validity will vary by purpose Capacity to do PM&E requires not only individual skills but institutional change: flexibility, ability to listen and learn, openness to change

How Do These Lessons Apply to PM&E of the PRSP Process?

World Bank PRSP Sourcebook States:

Outcome monitoring can be achieved through household surveys

Impact monitoring is a "demanding and complex methodology generally carried out by researchers outside the agencies that monitor outcomes
Civil society is seen to have a role in helping set goals and disseminate information But: Monitoring can be undertaken by civil society groups Appropriate goals, indicators and methods can allow civil society a more significant role in monitoring

Decide who Participates Take Action Establish Goals

PM&E Learning Cycle

Analyse Results

Develop Indicators

Gather Information

Step One: Who Should Be Involved in Monitoring and Evaluation?

Mapping the population: stakeholder and institutional analysis Analysis of whether the required skills, relationships and resources for PM&E currently exist

Stakeholder Analysis
The poor

Organisations representing poor people: community groups, religious leaders, trade unions, farmers unions, traditional authorities The better-off
Central and local government Politicians and political parties Academic researchers and analysts The press and broadcast media Donor agencies

Institutional Analysis
What existing relationships and networks exist between stakeholder institutions? A. Formally institutionalised groups B. Livelihood/ interest/ identity groups

Which institutionalised groups (A) represent which livelihood and identity groups (B)?
What groups, networks and communities should be involved in the PM&E process to make it as inclusive and workable as possible?

PM&E of PRSP Policy Processes Requires:

Civil society
Skills Collaborative networks within & between civil society and state and other responsible institutions Relationships of trust, accountability & learning between stakeholders Spaces and policies which enable engagement and monitoring Resources and time Structures for representation

PM&E of PRSP Policy Processes Requires:

Government
Transparency and openness Responsiveness and accountability Openness to institutional learning to enable participation Relationships of trust, listening and accountability with civil society Time

What form of PM&E is possible given the existing capacities, relationships and resources?

Strong Capacities

Weak Capacities

PM&E of PRSP by multiple civil society stakeholders

PM&E of PRSP by 'external' civil society stakeholders, such as International NGOs

What form of PM&E is possible given the existing capacities, relationships and resources?

Strong Capacities

Weak Capacities

PM&E of PRSP by multiple civil society stakeholders

PM&E of PRSP by 'external' civil society stakeholders, such as International NGOs

Forming a Steering Committee and Structure for PM&E

PM&E by multiple civil society stakeholders involves many monitoring activities being undertaken in different areas and issues and at different levels It is vital that these different activities are co-ordinated to enable: Analysis of findings A formal mechanism for engagement with responsible institutions Scope for public debate on findings A vision of the national picture: inclusions & exclusions Communication and information

Zambia: Developing a Monitoring Steering Committee

Civil society groups were invited by government to participate in the process of formulating the PRSP.
Co-ordinated by the jesuit centre, civil society groups established a network and steering committee of 14 csos: 'civil society for poverty reduction' (CSPR). The responsiveness of government to the CSPR monitoring and evaluation is enabled through a CSPR seat in the government PRSP working group.

Tanzania: Structure for Monitoring and Evaluation

Poverty monitoring steering committee

Surveys & census group

Routine data collection group

Research & analysis group

Dissemination group

All of these groups have civil society representation. In addition, wider scale participation of civil society and poor people in monitoring through PPA mechanism facilitated through the Research and Analysis group

Key Lessons:
Successful multi-stakeholder PM&E requires certain skills and capacities There are a range of different forms & structures for civil society engagement in monitoring PRSPs It is important to allow space for different stakeholders to participate in monitoring in different ways Appropriate and accountable institutional mechanisms linking civil society monitoring to state and other monitoring processes are crucial to ensure that civil society monitoring is not sidelined

Decide who Participates Take Action Establish Goals

PM&E Learning Cycle

Analyse Results

Develop Indicators

Gather Information

Step two: Establishing goals & intermediate goals

Key Questions:
What type of goals are to be monitored? Reduction in income poverty? Improvements in service delivery? Principles? Processes?
Who defines the goals and intermediate goals? Ideally, goals are defined through dialogue between multiple stakeholders. The defining of goals through partnership is central to PM&E. What mechanisms are there to hold responsible parties accountable for particular goals? Each goal should be backed up with agreed accountability mechanisms

World Bank PRSP Sourcebook: Guidelines on Monitoring

Intermediate goals Final goals
Monitoring inputs Where are resources being allocated? Public expenditure reviews Monitoring outputs What goods and services are generated by the project? What is the quality and scope of these outputs?

Monitoring outcomes Who are the beneficiaries? Do the poor have improved access to goods and services? Are these goods and services appropriate, & for whom? Monitoring impacts What is the impact on poverty & meeting different needs? Who is healthier, better educated, better employed?

These are all poverty monitoring goals but do not address PRSP principles or the participatory process. By most contemporary definitions, poverty is more than material wellbeing (income levels, health, education provision). Powerlessness and voicelessness are also dimensions of poverty. Monitoring of the PRSPs principles and of the participatory process, focuses attention on empowerment, voice and influence as goals and indicators of poverty reduction .

What type of goals to monitor?

Monitoring Poverty

Monitoring Principles

Monitoring the participator y process

What type of goals to monitor?

Monitoring Poverty

Monitoring Principles

Monitoring the participator y process

Tanzania: Poverty monitoring goals

Through regular PPAs, civil society is offered opportunities to express its perspectives on:
Nature of changes in poverty Causes behind changes in level & nature of poverty Impact of different policies and strategies on the poor

Tanzanias inclusion of PPAs in the overall Poverty Monitoring Strategy ensures that civil society participates in monitoring progress towards PRSP targets

What type of goals to monitor?

Monitoring Poverty

Monitoring Principles

Monitoring the participator y process

Monitoring PRSP principles

Aim: To hold government institutions and International Financial Institutions to account for the principles underpinning the PRSP
Country ownership Results-orientated Comprehensive Long-term in approach Based on partnership

Example: Monitoring country ownership principle

Broad: Multiple sets of stakeholders feel that they share ownership of the processes and outcomes

Goals:

Government & donors engage in a systematic learning process, through which they consciously step back from their traditional dominant positions
Different sections of civil society perceive policy to be a domain over which they can and have a right to own and influence

Intermediate: Capacity of civil society & government to jointly own the Goals PRSP is strengthened through institutional development, (3-5 years) relationship-building & reflection processes The meaning of 'country ownership', and attaining this goal, is agreed between different stakeholders, and mechanisms for accountability to this goal are defined

What type of goals to monitor?

Monitoring Poverty

Monitoring Principles

Monitoring the participator y process

Monitoring the participatory process

Aim:
To evaluate whether participatory PRSP processes lead to the assumed benefits

Monitoring the participatory process

Focusing on:
The process itself Form, level, quality and inclusiveness of participation

The developmental benefits of the process The impact of participation on self-development

The instrumental benefits The impact of participation on policy, decision-making and other broader changes

Good quality and inclusive participation itself contributes to poverty reduction by:

Making better policy Empowering participants & enabling them to demand more responsive institutions Building participants' capacities to dialogue, negotiate & advocate

Indicators for monitoring policy process- COGS Framework, Yorkshire Council UK.
Principles Influence
Ensuring that participation leads to real influence

BENCHMARKS
Local stakeholders: Valued as equal partners Represented on decision-making bodies Opportunities to participate Access and control over resources Evaluation includes local stakeholder agenda Diversity of local stakeholder interests reflected in policy and process Two-way information strategy Policy & strategy are clear & accessible

Inclusivity
Valuing diversity & addressing inequality

Communication
Clear, transparent & accessible policies & procedures

Capacity
Understanding different partners & their capacities

Local stakeholders - resources to participate Understanding, knowledge & skills are developed to make a working partnership

Key lessons:
Different stakeholders are interested in different goals and have different capacities It is crucial that space is given for different stakeholder groups to define different goals If goals differ among stakeholders, processes of negotiation over goals are required at the outset to identify a number which can feasibly be monitored, and to ensure that the outputs will be carried through to action

Decide who Participates Take Action Establish Goals

PM&E Learning Cycle

Analyse Results

Develop Indicators

Gather Information

Step three: Developing indicators

What goals can be measured, through which indicators?
What indicator data is it practicable to collect and analyse? Is quantitative or qualitative information needed? Which stakeholders have the capacity to collect & analyse? Who chooses the indicators? Different stakeholders may choose different indicators to monitor the same goal Will these indicators be effective for holding government and civil society stakeholders accountable? If monitoring is to have influence, different stakeholders must agree on the validity of indicators, and be willing to account for their actions as measured by them

Key lessons:
Not everything can be monitored

Indicators should be carefully chosen and related to priority goals Identifying indicators for intangible goals is not an exact science
If people attach high priority to intangibles, they can probably devise ways of measuring their attainment

UNDP Tanzania statement on choosing indicators: "Agreement has to be reached among the stakeholders on a priority list of
indicators for poverty monitoring. There is a balance to be struck between comprehensiveness and affordability. In the case of Tanzania, middle ground has to be found between the extensive list of poverty and welfare indicators and the much shorter core list presented in the PRSP".

Example: Indicators to monitor principle of country ownership

INPUT Indicators State resources allocated to increase quality & scope of participation, transparency and accountability. Civil society seeks resources to support their participation and advocacy Initiation of meaningful participation before PRSP goals are set Civil society demands responsiveness and accountability as conditions for their participation Meaningful participatory arenas opened by government across country and include all stakeholders who want to participate Civil society facilitation of participatory analysis & monitoring with confidence that government will accountable for the results PRSP includes needs and priorities of wide range of stakeholders Increased civil society and government capabilities and opportunities to lead PRSP and negotiate with donors and creditors over it

PROCESS Indicators

OUTPUT Indicators

OUTCOME Indicators

Decide who Participates Take Action Establish Goals

PM&E Learning Cycle

Analyse Results

Develop Indicators

Gather Information

Step four: Gathering information

An information-gathering strategy needs to be developed, including how resource needs will be met A baseline is needed point of comparison for tracking changes over time Qualitative or quantitative information? Which information gathering tools? Formal or informal approaches?

Decide who Participates Take Action Establish Goals

PM&E Learning Cycle

Analyse Results

Develop Indicators

Gather Information

Step five: Analysing and reflecting on what has been learnt

Analysis of findings feeds into critical review of whatever is being monitored reflection on PM&E process feeds into a better PM&E process next time round The PRSP was conceived as a dynamic, iterative framework in which both content and process could be continually reviewed and improved on This places considerable, ongoing demands on leaders and participants of PRSPs and implies a need for sustained investment in capacity-building for diverse stakeholders

Decide who Participates Take Action Establish Goals

PM&E Learning Cycle

Analyse Results

Develop Indicators

Gather Information

Step Six: Taking action for improvement

Improvements to PRSPs content, the participatory process or adherence to underlying principles Links between the civil society PM&E process and national PRSP Steering Committee will be crucial for achieving this Improvements in PM&E process may take various forms:Indigenous civil society in a more central role More skilled application of information-gathering tools More strategic ways found to spread and apply the lessons learnt Civil society actors can take action themselves but cannot force Governments or donors to do so How committed are government and donors to learning and changing as a result of PM&E?

Example: Bolivia
One proposal: to establish a civil society monitoring system, parallel to the Governments (concern over accountability and transparency of government institutions) Alternative proposal: broad, long-term effort to address these concerns by gradually building capacity in participatory approaches among government and civil society Proposed 3-year programme of support to National Working Group on Participatory Approaches:
To strengthen the capacity of the National Working Group to facilitate and support participation of local people and other actors in planning, decisionmaking and project activities related to poverty reduction at the local, regional and national levels

Slow, expensive, work-intensive

Key lesson:
There is no quick fix to establishing PM&E of PRSPs participatory processes take time, cost money and demand skills If poverty reduction is about increasing poor peoples voice, empowering them to address their own problems, and making government and donor institutions more accountable to them, the benefits of PM&E outweigh the costs and go beyond what conventional M&E can offer