Академический Документы
Профессиональный Документы
Культура Документы
EVALUATION
1. Introduction 3
• Indicators 3
• Monitoring and Evaluation System
• Monitoring and Evaluation providing 3
options 4
for decision-makers
• Structuring the M&E plan 5
5
1. Qualities of an Indicator 6
3. Types of Indicators 8
• Proxy Indicators 8
• Indicators for Monitoring
9
• Process Indicators
• Output Indicators 9
• Economic Indicators
• Quality of Life Indicators 10
10
10
1. Development of Indicators 11
• Basic steps in developing indicators as part of an a 11
M&E system
1
1. Role of Indicators in the Planning Process 12
• Selecting Indicators 13
2. Conclusion 13
3. References 14
Introduction
Indicators are measures of ‘change’, progress and achievement in a project or programme. A
good monitoring and evaluation system should support improvement and adaptation at several
different levels. At the project management level, a monitoring and evaluation system should
provide information needed to improve the efficiency of the implementation process and the
performance of those involved. At a strategic level, it should also support regular reviews of the
strategy itself—to revaluate chosen courses of action and take into account changing contexts.
The monitoring and evaluation system itself should also be subject to regular reviews.
Indicators
An indicator is a piece of information which communicates a certain state, trend, warning or
progress to the audience. Indicators can help to answer the questions where are we now,
where do we want to go, are we taking the right path to get there, and are we there yet?
The ‘change’ or ‘difference’ that a project makes or brings about has to be kept track of to ensure
that projects are implemented successfully so that intended results are obtained. In order o keep
track of the progress of implementation and also the results, we need some devices or
information to measure the changes or achievements at different levels i.e. input delivery,
performance of activities, and realization of outputs, effects and inputs. Such a measure used by
project management is called Indicators.
Indicators provide insight into matters of larger significance and make perceptible trends that are
not immediately detectable. Indicators reflect the status of a system, for example an oil pressure
gauge on an engine or the number of owls in a forest. Indicators are measures selected to assess
progress towards the targets associated with goals and objectives and the accomplishment of
actions. For example, the prevalence of underweight children under five years of age and the
proportion of population below the minimum level of dietary energy consumption are used as
indicators for the Millennium Development target on hunger.
2
Monitoring and Evaluation System
Monitoring and Evaluation System is explained as below:
• Monitoring is a continuous assessment of project implementation in relation to agreed
schedules, use of inputs, infrastructure and services provided by project beneficiaries.
Evaluation is a periodic assessment of the relevance, performance efficiency and impact (both
expected and unexpected) of the project in relation to stated objectives.
Monitoring and Evaluation involves:
• Monitoring the process of implementation—to ensure that the actions outlined in the
strategy are being taken and that resources are being allocated and used effectively.
• Monitoring the outcomes of those actions—in terms of investments in infrastructure and
changes in policies, institutional frameworks, and management instruments.
• Evaluating the progress towards the achievement of goals and objectives in relation to
actions.
• Using the information gained to refine the strategy and to inform decision making at
different levels—from national planning to water user behavior.
Although normally used in conjunction, the words monitoring and evaluation represent two
separate concepts with different meanings and purposes. While monitoring and evaluation are
both concerned with the collection, analysis and use of information to evaluate the relevance of
the work done within the project and support informed decision-making, it is useful to
understand the differences between the two: What are the general purposes? Who is responsible?
When and why are they carried out? This distinction is shown in Table 1.
completion, ex post
3
Why? To check the physical (input provision, Learn broad and generic lessons,
capacity development)
Monitoring refers to a continuous analysis of the progress towards achieving the planned results.
According to UNDP (1997) it “aims primarily to provide project management and the main
stakeholders of an ongoing programme or project with early indications of progress, or lack
thereof, in the achievement of programme or project objectives”. Its main purpose is the
improvement of informed management decisions including corrective actions if needed. It
usually is part of the internal management responsibility and refers to all levels involved in the
capacity programme. It provides information on the physical progress in terms of input provision
and execution of activities and results accomplished. It also refers to the quality of the
established capacity development process with regard to the level of stakeholder participation,
communication and acquired skills and knowledge.
Evaluation refers to a time-bound review of the effectiveness, efficiency, impact, relevance and
sustainability of an ongoing or completed programme. It encapsulates a broader view on the way
the capacity programme is designed and formulated, and to what extent the assumptions are
realistic and valid. Evaluations serve the purpose of policy makers and planners as well as the
accountability needs of lending agencies and governments. They involve key stakeholders with
direct responsibilities for implementation on the ground (i.e. the project management team) but
an external view is also required for more objectivity. They provide a structured opportunity to
discuss and agree on the content and build a common understanding of key issues/concerns and
of actions that need to be undertaken.
Monitoring Evaluation
4
Measuring efficiency • Confirming project expectations
• Measuring impacts
Question “Is the project doing things • Question “ Is the project doing the right
right?” things?”
Source: Alex and Byeriee, Monitoring and Evaluation for AKIS Projects, World Bank, 2000
an M&E system
Table 3: Purposes and benefits of M&E
Purpose Benefit
5
• Local and staff empowerment
Qualities of an Indicator
Every indicator that is identified to measure something should fulfill four criteria that include
basic purpose, quantity, quality and a time frame. Only when all these four criteria are ensured
the indicator would be able to objectively verify the changes that are claimed to have been
brought about. The illustrations below in Table -1 provide examples of indicators possessing the
criteria one lined.
1. Identify Basic Small farmers increase rice yields Improving access to safe
Purpose drinking
2. Set a Quantity 30,000 small farmers increase rice 50% of households in five
villages
yields by 50% (from X to Y)
per block of a district
6
3. Define Quality 30,000 small farmers increase rice Water from a covered
well/ hand
yields by 50% while maintaining
same pump/ piped water supply
not
quality existing in 1979 harvest
more than 500 meters
away
4. Specify Time 30,000 small farmers increase Within a year after the
beginning
production between 1990 and
1995 of the project
Before proceeding to consider appropriate indicators for measuring project progress towards
each objective separately, introductory comments on these questions serve to set the scene.
(d) Bias
7
Clearly, it is desirable that an indicator should, whenever it is measured, reflect the true value of
the condition it is representing. For example, the volume of a crop purchased by a marketing co-
operative will usually be a biased estimate of the total crop marketed, but changes in the volume
purchased by the board may be a good indicator of change in the total volume marketed so long
as the co-operative’s share of the total market remains constant. In years of large harvests, the
price on the open market may drop and farmers may sell most of their crop to the marketing co-
operative at a guaranteed stable price. In lean years, the farmer may take advantage of high
prices in his nearby market and the sales to the co-operative may decline proportionately much
more than the decline in total marketed volumes.
.
(e) Contribution to explanation of Variation
In selecting indicators for monitoring and evaluation system, the objective is to put together a set
wherein each indicator, individually and separately, explains a part of the overall variation in the
condition being studied; and, together, explain a substantial proportion of the total variation.
Previous studies or even intuition may identify an indicator that is so closely correlated with
another indicator that the inclusion of both contributes little more than the inclusion of their one
alone. Examples of pairs that are sometimes, but not invariably, highly correlated are
• income - expenditure
• area planted - quantity of seed used
• literacy - years of formal education
• cash crop production - cash crop sales.
Types of Indicators
• Proxy Indicators
• Indicators for Monitoring
• Process Indicators
• Output Indicators
• Economic Indicators
• Quality of Life Indicators
1. Proxy Indicators
Of the criteria listed above, the one of most practical importance from the designer’s viewpoint
8
is the ability to measure the indicator accurately, at reasonable cost on the scale required.
Inability to do this may require a search for a proxy indicator that can be measured more easily
and which appears to meet, or partly meet, the other criteria. Consider first the example of
indicators of the area under maize. The questions asked of a farmer can include the following in
ascending order of difficulty:
• How many plots did the farmer plant?
• How much seed did the farmer use?
• What was the area of maize (e.g., to the nearest tenth of a hectare)?
Either (a) or (b) may serve as a rapid assessment aimed at producing an indicator of whether the
incidence of maize planting is above or below previous seasons. For many purposes, however, a
more precise area may be required; but if the farmers do not know the area in these terms, a
massive quantum jump in survey scale is required in order to physically measure the land using
one of the acceptable techniques.
A different example is provided by certain social indicators. Consider indicators of nutritional
status as a proxy measure of the general quality of life. Simple measures of age, height and
weight may be used to calculate indicators, particularly for children, that reflect not only the
nutritional status of the individual but may, in appropriate circumstances, be used as a proxy for
general wellbeing. (Table – 5)
(Proxy or Substitutes)
To increase per capita Crop Sales and Food Purchase of Typical Consumer
Consumption
income of small farmers Items. Tin roofs on huts
To Improve Nutrition of Amounts and Types of Food Height and Weight of Children:
1. Process Indicators
Process indicators, which monitor the basic progress of implementing the actions outlined in the
strategy. This includes monitoring implementation processes and also the tracking of inputs—the
people, money, equipment needed to achieve actions. Processes are the ways in which activities
are conducted. For example group formats is an activity and it involves certain processes and
procedures (rapport building, awareness generation, training etc.) unless these processe are
followed systematically, desired results may not be forthcoming. We need to evolve indicators to
keep track of the process.
2. Output Indicators
In agriculture and rural development projects, outputs are expressed in terms of production;
whether crops, livestock, forest products fish, etc. In general, quantification of production maybe
through direct measurement or the producer’s stated estimate. If direct measurement is adopted,
the following common categories of production can be identified:
• seasonal crops—areas and yields per unit area;
• tree crops (including forestry) – numbers of trees and yield per tree (or area and yield per
unit area);
• animals—live numbers, and off-take number; and
• animal products (including fish)—volumes, weights.
The indicators, therefore, are well defined; numbers, areas, yields, volumes, and weights. The
estimation of crop production deserves special mention. A well-documented and popular method
is to conduct a crop-cut of a small portion of a known area of the plot in order to estimate the
10
average yield per hectare for the crop. The alternative to direct measurements of production is to
obtain estimates from the producer.
1. Economic Indicators
The postulated economic benefits of a project are usually expressed in income terms; so income
is the obvious choice as the indicator to measure. However, as indicated earlier, in the case of
small farmers, the accurate definition and measurement of income is difficult. The measurement
of expenditure does not present some of these definitional and response problems. The
experience of many budget surveys is that expenditure is recorded more accurately and
completely than income. Expenditure can be used as a proxy for income in evaluating change.
c) Published statistics.
In large scale programmes of longer duration like agricultural development, health project etc,
indicators can be generated/ obtained from state/ national statistics which are brought out
periodically. For instance, Agriculture Department brings out Area, Production and Yield of
majorcrops every year. Similarly Health Department brings out data relating to Infant mortality
Rate, Maternalmortality rate etc. Such information, if available on regular basis could be used.
Many a times such publications are not regular or it may be time lagged.
There are various approaches to defining indicators. The model below is one example:
Step 1: Make sure that targets associated with strategy goals, objectives and actions are clearly
11
defined and agreed upon; and that the inputs necessary carry out actions are identified.
Step 2: Define indicators for each target based on stakeholder consultations and on criteria, such
as relevance, reliability, and cost-effectiveness. Involve stakeholders who cause or are affected
by the problem or issue addressed in the target; who have relevant information or expertise; and
who will be responsible for implementing indicators.
Step 3: Select indicators to track human and financial resources and ensure that they are being
disbursed and used efficiently.
Step 4: Check to make sure that indicators relate clearly to targets, and that these in turn
support the achievement of actions, objectives and goals. Identify and fill gaps. Refine indicators
and/or targets as necessary. This step may involve taking an inventory of indicators that are
already in use in the country to eliminate redundancies, and also considering the relationship of
national M&E efforts to international monitoring programs—such as the World Water
Assessment Programme.
Step 5: Calculate human and financial resources needed to apply the indicator package.
Evaluate whether the package is a good investment, i.e. the human and financial resources
required are commensurate with the value of the various indicators employed.
Step 6: Agree on the agencies/institutions that will be responsible for applying the different
indicators, how, and how often.
Step 7: Determine how the information resulting from the different indicators will be managed:
how it fits into decision-making processes, both specifically related to the strategy but also
ongoing policy and planning processes; how information will be amalgamated to get a more
comprehensive picture of progress; and how it will be communicated to stakeholders.
Step 8: Include requirements for M&E package in capacity building plan, budget, and staff
allocation.
M & E studies have identified six steps in designing an M & E system. These are:
a. Establishing the purpose and scope - why do we need M & E and how comprehensive
should our M & E system be?
b. Identifying performance questions, information needs and indicators - what do we need
to know to monitor and evaluate the project in order to manage it well?
c. Planning information gathering and organisation - how will the required information be
gathered and organised?
d. Planning critical reflection processes and events - how will we make sense of the
information gathered and use it to make improvements?
e. Planning for quality communication and reporting how and to whom do we want to
communicate what in terms of our project activities and processes?
f. Planning for the necessary conditions and capacities - what is needed to ensure our M&E
system actually works?
Where are we now? As part of a baseline assessment at the beginning of the strategy process,
indicators help to determine what the problems are, where they are, and their level of severity.
12
Where do we want to go? Information provided by the baseline assessment indicators can help
decide on priorities and can serve as a useful input into stakeholder dialogues during the strategy
formulation process. Indicators may also be used in identifying the necessary actions by helping
to assess the effectiveness of existing institutions, policies, regulations, etc. Once priorities and
basic goals, objectives, and actions have been agreed upon, the process of defining indicators for
monitoring and evaluation can help to set and refine specific targets.
Core indicators are the ways we come to understand the inputs and outcomes of a program or
project that we may or may not be able to observe directly. A strong monitoring and evaluation
system helps ensure that a project meets its main objective of fostering positive change, and also
that the strategy can adapt to evolving needs and conditions. Although a vital component in the
success of any strategy, development and implementation of a M&E system is often allocated
insufficient time, thought, and human and financial resources.
Selecting Indicators
The question of ‘what are the right/best indicators’ dominates many discussions about M&E.
This only intensifies when more participatory forms of working are pursued, as perceptions of
what is ‘right’ or what are the ‘best’ indicators will inevitably differ between stakeholders. What
is sometimes forgotten is that an indicator is simply a means to help communicate complex
changes to a wider audience. Indicators describe and express conditions and represent some kind
of simplification or approximation of a situation. The most crucial question when selecting
indicators is clarity about the end-users and end-uses of the information. It is also paramount to
minimize the number of indicators to keep the M&E process manageable. Resolving these
questions requires considerable discipline, compromise and hard decisions.
Decision makers at every level and scale will find very different kinds of indicators relevant to
their decisions, therefore, getting consensus about objectives and indicators will usually require
negotiation. For example, municipal level agricultural research in Brazil involves many different
kinds of farmers, farmer experimentation groups, community associations, university
researchers, government extension staff, local NGOs and even international funding agencies.
Each operates at a different scale with more immediate and local or longer-term and broader
objectives. If these objectives are clear, then the easier it is to develop indicators.
A M&E system also has to be responsive to changing information needs as contexts change, to
the changing skills of those involved, and indeed to changing levels of participation as new
partners join and others leave. Indicators must be reviewed regularly to ensure that they are
providing information that is relevant. However, any change to an indicator means reducing the
possibility of producing time-series data. One approach is to limit indicator-based monitoring to
longer-term, broader objectives that are less likely to change, but the less specific the objective,
the more difficult it is to be clear about the cause–effect linkages that indicators represent.
Distinguishing between more immediate and longer-term objectives is important when selecting
indicators. Monitoring often focuses on the immediate, more tangible, and easily accessible
information like ‘the number of farmers trained’. By comparison, evaluation will focus on
assessing whether, for example, the training efforts are worthwhile and the effect of those trained
farmers on their fields and households.
Monitoring and evaluation needs a strong information system. Data from grassroots and various
segments need to be collected on a regular basis. Such data need to be compiled, collected and
analysed and converted into concise and precise pieces of information. Such a concise form of
information/ data is call an Indicator which a re measured of change, progress and achievement.
There are certain basic qualities for an indicator and there different types of indicators.
Developing indicators is an expert job and should be done collectively through a participation
process. Indicators serve like a barometer, road sign or gauges in the dash board of car for
project management.
References
1.CSD (2001). Indicators of Sustainable Development: Guidelines and Methodologies.
http://www.un.org/esa/sustdev/publications/indisd-mg2001
2.Davis-Case, Community Forestry. Participatory Assessment, Monitoring and Evaluation.
Rome:FAO. 1990.
3.Dennis J. Casley & Krishna Kumar, Project Monitoring and Evaluation in Agriculture, The
World Bank. 1990.
4.Guijt, I.. Participatory Monitoring and Impact Assessment of Sustainable Agriculture
Initiatives. 1998.
5.EC. Aid delivery methods. Vol.1: Project cycle management operational guidelines. European
Commission. 2004.
6.Gulati, Ashok., and Gary Pursell, Trade Policies, Incentives and Resource Allocation in Indian
Agriculture, MIMEO, The World Bank, Washington, D.C. 1996.
7.International Food Policy Research Institute, WTO, Agriculture and Developing Countries: A
survey of Issues, TMD Discussion paper 81. Washington, D.C. 2002.
8.Parris, T. and Kates, R., Characterizing and Managing Sustainable Development. Annual
Review of Environment and Resources vol. 28, pp. 559-86. http://arjournals.annualreviews.org,
2003
14
9.Monitoring & Evaluation Manual, National Agricultural Innovation Project (NAIP), Project
Implementation Unit (PIU), NAIP, Indian Council of Agricultural Research, 2007.
10.World Bank,Monitoring & Evaluation – Some Tools, Methods and Approaches, 2004.
15