Results-based Management

What is results-based management (RBM)?

RBM is a program/project life-cycle approach to management that integrates strategy, people, resources, processes and measurements to improve decision-making, transparency, and accountability. The approach focuses on achieving outcomes, implementing performance measurement, learning, and adapting, as well as reporting performance.

How is RBM used at CIDA?

RBM has a long history at CIDA and has been used, in one form or another, for over 30 years. CIDA's first official RBM policy was released in 1996 and a revised and updated policy was approved in 2008. RBM is not a tool, but rather a way of working that looks beyond activities and outputs to focus on actual results; the outcomes of projects and programs. CIDA uses RBM to effectively and efficiently manage Canada's international development aid. Performance Management Division of CIDA's Strategic Policy and Performance Branch supports the Agency for a sound practice of results-based management. For further information, please contact Performance Management Division. RBM - Key Policy Documents: CIDA's 2008 RBM Policy and supporting documents

RBM - Guides: For understanding RBM at CIDA - from theory to practice RBM - Templates: For working with CIDA's three main RBM tools; Logic Model (LM), Performance Measurement Framework (PMF) and Risk Register.

RBM - Related Sites

Results-Based Management Tools at CIDA: A How-to Guide

Instructions for the use of CIDA's three main results-based management working tools: the logic model, performance measurement framework, and risk register Introduction Using CIDA's results-based management tools The logic model The performance measurement framework Risk analysis

Introduction
What is results-based management (RBM)? Results-based management (RBM) is a life-cycle approach to management that integrates strategy, people, resources, processes, and measurements to improve decision making, transparency, and accountability. The approach focuses on achieving outcomes, implementing performance measurement, learning, and adapting, as well as reporting performance.

RBM

Page 1

RBM is: defining realistic expected results based on appropriate analysis; clearly identifying program beneficiaries and designing programs to meet their needs; monitoring progress toward results and resources consumed with the use of appropriate indicators; identifying and managing risk while bearing in mind the expected results and necessary resources; increasing knowledge by learning lessons and integrating them into decisions; and reporting on the results achieved and resources involved. Why results-based management? Historically, government departments-and implementing organizations (IOs)- focused their attention on inputs (what they spent), activities (what they did), and outputs (what they produced). Although accurate information at this level is important, they discovered it did not tell them whether or not they were making progress toward solving the problem they had set out to resolve, and that the problems often remained once projects were completed. Modern management requires that we look beyond activities and outputs to focus on actual results: the changes created, and contributed to, by our programming. By establishing clearly defined expected results, collecting information to assess progress toward them on a regular basis, and taking timely corrective action, practitioners can manage their projects or investments in order to maximize achievement of development results: a sustained improvement in the lives of people in developing countries. How is results-based management used at CIDA? RBM has a long history at CIDA: it has been used, in one form or another, for more than thirty years. CIDA's first official RBM policy was released in 1996, and a revised and updated policy was approved in June 2008. CIDA uses RBM to better manage its international development programming from start (investment or project analysis, planning, design, implementation, monitoring, adjusting, and reporting) to finish (final evaluations and reports, and integrating lessons learned into future programming). CIDA has developed three main RBM working tools to make managing for results throughout the entire life cycle of an investment or project easier for CIDA staff, partners, and executing agencies: thelogic model (LM), performance measurement framework (PMF), and risk register. As the name implies, these tools are meant to be flexible working documents that remain evergreen throughout the lifecycle of the investment, and can be adjusted and modified under certain circumstances. The LM and PMF are usually at least partially completed during the planning and design stages of an investment, and refined during the development of an implementation plan (this will vary depending on the type of programming in question). The risk register is completed during project design, and updated on a regular basis.

Using CIDA's results-based management tools

Participatory approach Developing the LM, PMF, and risk register in a participatory fashion is an integral part of investment design and planning. In RBM, investments must be designed, planned, and implemented using a participatory approach whereby all stakeholders (including beneficiaries) are involved throughout the investment's life cycle. By using a consensus-building process to define and agree upon the information in the LM and the PMF, local stakeholders are given a sense of ownership that enhances subsequent commitment throughout the investment and beyond.

RBM

Page 2

The logic model

What is a logic model? Sometimes also called a "results chain," a LM is a depiction of the causal or logical relationships between inputs, activities, outputs, and the outcomes of a given policy, program or investment. The LM is divided into six levels: inputs, activities, outputs, immediate outcomes, intermediate outcomes, and ultimate outcome. Each of these represents a distinct step in the causal logic of a policy, program, or investment. The bottom three levels (inputs, activities, and outputs) address the howof an investment, whereas the top three levels (the various outcomes) constitute the actual changesthat take place: the development results. Logic Model with definitions Standard template - logic model (LM) CIDA has a standard template for the logic model Please note that CIDA's LM template does not include inputs; rather, it starts at the activity level. To complete a logic model template, you need to write clear and concise result statements. Drafting or assessing result statements What is a result? A result is a describable or measurable change that is derived from a cause-and-effect relationship. At CIDA, results are the same as outcomes, and are further qualified as immediate (short term), intermediate (medium term), or ultimate (long term). What is a result statement? A result statement outlines what a policy, program, or investment is expected to achieve or contribute to. It describes the change stemming from CIDA's contribution to a development activity in cooperation with others. Outcomes = result statements Hint # 1: Questions to ask yourself when drafting or assessing a result statement: Is the statement simply worded, and does it contain only one idea? The logic model is a snapshot of your investment, and the result statements should be clearly stated and easy to understand. Would the Canadian public be able to understand this result statement? Does the statement contain more than one idea? If so, can it be split into separate statements? Was the result statement drafted in an inclusive, participatory fashion? RBM is a participatory process. The process and methodology for the selection of outcomes and drafting of result statements should be as participatory as possible, involving a wide representation of key stakeholders. It is essential to ensure that all the voices are heard and that your expected outcomes are shared with all involved. Were key stakeholders, including analysts and specialists (working in gender equality and environment), partners, and implementers involved? Make sure that the IO has mechanisms in place to ensure that leaders, decision makers, women and men, minorities and direct beneficiaries are involved.

A. Stakeholder involvement Has a stakeholder analysis been done? Has adequate consultation been performed? Is there participation of both male and female stakeholders? Are there mechanisms for participation in the design and decision making throughout the life cycle of the investment?

RBM

Page 3

B. Gender analysis Are the results truly gender sensitive? Do they address the concerns, priorities, and needs of women and men, and girls and boys?

C. Environmental analysis Have you taken environmental implications into consideration? Will results be sustainable? Does the result statement include an adjective and tell you:

WHAT? Does the result statement describe the type of change expected using an adjective that is drawn from a verb and that indicates direction (e.g. increased, improved, strengthened, reduced, enhanced)? WHO? Does the result statement specify the target population or beneficiary of the intervention? Does it specify the unit of change (e.g. individual, organization, group)? WHERE? Is the location or site where the result will occur specified?

Can the result be measured? Can the result be measured by either quantitative or qualitative performance indicators? Can performance indicators that will measure the result be easily found, collected, and analyzed? Is the result realistic and achievable? Is the result within the scope of the project's control or sphere of influence? Is there an adequate balance between the time, resources allocated, and expected reach and depth of change expected? Are the results at the immediate and intermediate levels achievable within the funding levels and time period for the project? Is the result (immediate and intermediate outcome levels) achievable during the life cycle of the investment? In other words, can the expected changes (immediate and intermediate outcome levels) be realistically achieved by the end of the intervention? Is the result relevant? Does the result reflect country ownership and needs, and will it support higher-level developmental change in the strategies or programs it supports? Is the result aligned to the country partner's national development strategy? Does the result reflect needs and priorities among the beneficiaries that were identified in a participatory fashion? Does the result take into account the culture of the local population? Is the result aligned to CIDA program and corporate priorities?

Examples of weak and strong result statements Result Increased literacy Issue Does not identify for whom or where the expected change will occur Is it a strong result statement? Not strong

Increased literacy among men and women in region X of country Y More women can get maternal health-care services

Strong Doesn't use an adjective drawn from Not strong a verb that clearly indicates direction of change Does not identify where the expected change will occur Strong

Improved access to maternal health-care services for women in

RBM

Page 4

country X Peace in country X Does not specify direction of expected Not strong change, nor whom, specifically, it will affect. Not achievable Strong

Increased stability in country X Developing a logic model

Here are the steps that need to be taken to create a LM. The order in which they are undertaken will depend on the status, scope, and size of the investment. Identify ultimate beneficiaries, intermediaries, and stakeholders. Hint #2: Useful definitions Beneficiary: The set of individuals and/or organizations that experience the change of state at the ultimate outcome level of a LM although they could also be targeted in the immediate and intermediate outcome levels. Also referred to as "reach" or "target population" (Treasury Board Secretariat lexicon) Intermediary: An individual, group, institution or government who are not the ultimate beneficiary of an investment but who are the target of select activities that will lead, via the associated immediate and intermediate outcomes, to a change in state (ultimate outcome) for the ultimate beneficiaries. Stakeholder: An individual, group, institution, or government with an interest or concern, either economic, societal, or environmental, in a particular measure, proposal, or event. (Termium Plus) Partner: The individuals and/or organizations that collaborate to achieve mutually agreed upon expected results. (OECD-DAC Glossary of Key Terms in Evaluation and Results Based Management, with slight modification) Implementing organization: An IO is any organization or agency, whether governmental, non-governmental, intergovernmental, specialized, multilateral, or in the private sector, that implements an investment (project or program) for which CIDA provides funding. Ensure that the right people (e.g. development officer; specialists in environment, governance, and gender equality; executing agency representative; local stakeholders; and beneficiaries) are at the table. Remember that this is a participatory exercise. This can be done via brainstorming, focus groups, meetings, consultative emails, etc. Note that the "right people" may vary depending on the type of programming. For directive programming, ensure that country-partner organizations, beneficiaries, and stakeholders (including women, men, and children) are at the table during the design/development of the LM. For responsive programming, ensure that the right CIDA team is at the table during the review and assessment of the LM. The review team should include the development officer or project team lead; branch specialists in environment, governance, and gender equality; other corresponding sector specialists; and performance management advisors. You should also validate, as part of your due diligence, that the proponent developed the LM through a participatory approach. Identify the ultimate outcome. Start by identifying the problem your investment intends to address. The ultimate outcome of an investment is its raison d'tre: the highest level of change we want to see to solve that problem. Make sure to analyze the context (cultural, socio-political, economic, and environmental) surrounding the problem). Example: Problem: Poor health among inhabitants of region Y of country X due to water-borne illness.

RBM

Page 5

The ultimate outcome is the highest level of change that can be achieved, a change of state for the target population. Ultimate outcome: Improved health among people living in region Y of country X.

Identify main activities (for both CIDA and partners). Brainstorm the main or key activities of the investment, making sure to address contributing contextual factors. If possible, group activities into broad categories or work packages to avoid duplication. Identify outputs for each activity package. Hint #3: Make sure activity statements begin with a verb in the imperative form and that outputs are written as completed actions. Outputs are usually things you can count. Example: To achieve the ultimate outcome of "Improved health among people living in region Y of country X," stakeholders in country X (e.g. ministry of health, regional health authority, local community organizations), our IO and CIDA staff have decided to concentrate on three groups of activities: building wells, offering training in well maintenance, and rehabilitating and staffing regional health centres. Activities:

Build wells in region Y Develop and deliver training on well maintenance to people living in region Y Rehabilitate and staff regional health centres in region Y

Outputs:

Wells built in region Y Training on well maintenance developed and delivered to people living in region Y Regional health centres in region Y rehabilitated and staffed

Identify logical outcomes for immediate and intermediate levels Hint #4: A logic model is like a pyramid: it gets smaller the closer you move toward the highest level. Three or four changes at the immediate level (changes in access, ability, awareness) may lead to only two changes at the intermediate level (practice, behaviour). Similarly, two changes at the intermediate level will lead to only one change at the ultimate level (change in state). The logic model template is flexible: it will allow you to change the number of boxes at each level to reflect the logic of your investment. Make sure the number of outcomes decreases as you move upward toward the ultimate outcome. Try also to have only one outcome per box. Example: Immediate level results flow logically from the activities and outputs. They represent the change brought about by the existence of goods and/or services created through the activities. Thus, the provision of wells = increased access to clean water. Intermediate-level results represent a change in behaviour. They are the next logical step from the immediate level and lead logically to the ultimate outcome. Immediate outcomes (a change in access, ability, or skills):

Increased access to clean drinking water for people living in region Y. Increased ability to maintain wells among people living in region Y.

RBM

Page 6

Increased access to health services for people living in region Y.

Intermediate outcomes (a change in behaviour or practice):

Increased use of clean drinking water by people living in region Y. Increased use of health services by people living in region Y. Identify linkages. Check back and forth through the levels (from activities to ultimate outcome and from ultimate outcome to activities) to make sure everything flows in a logical manner. Make sure there is nothing in your outcomes that you do not have an activity to support. Similarly, make sure that all your activities contribute to the outcomes listed.
Validate with stakeholders/partners. Share your draft logic model with your colleagues, specialists, stakeholders, and partners, etc. to ensure that the outcomes meet their needs and that the investment will actually work the way you have envisioned it. Where required, write the narrative text to illustrate linkages and explain logic model. The narrative should speak to the arrows in the logic model: between the levels and how we see the activities proposed leading to the The most compelling narratives are those that are succinct and use brief, based examples to support these explanations. the causality of the the causal relationship expected changes. concrete, evidence-

Note: Targets, although necessary for the establishment of a budget, are not displayed in the LM; rather, they appear in the PMF. These will be discussed in further detail in the PMF section. Example: Ultimate outcome Intermediate outcomes Immediate outcomes Outputs Improved health among people living in region Y of country X. Increased use of clean drinking water by people living in region Y. Increased access to Increased ability to maintain clean drinking water for wells among people living in people living in region Y. region Y. Wells built in region Y. Increased use of health services by people living in region Y. Increased access to health services for people living in region Y.

Training on well maintenance Regional health centres developed and delivered to in region Y rehabilitated people living in region Y. and staffed. Develop and deliver training on well maintenance to people living in region Y. Rehabilitate and staff regional health centres in region Y.

Activities

Build wells in region Y.

The performance measurement framework

What is performance measurement? Measuring performance is a vital component of the RBM approach. It is important to establish a structured plan for the collection and analysis of performance information. At CIDA, as in other departments and agencies of the Government of Canada, the PMF is the RBM tool used for this purpose. Use of the PMF is not limited to the Government of Canada, however: other organizations and donors use similar tools to plan the collection and analysis of performance information for their programming as well. Why performance measurement? Performance measurement is undertaken on a continuous basis during the implementation of investments so as to empower managers and stakeholders with "real time" information (e.g. use of resources, extent of reach, and progress toward the achievement of outputs and outcomes). This helps identify strengths, weaknesses, and problems as they occur, and enables project managers to take timely corrective action during the investment's life cycle. This in turn increases the chance of

RBM

Page 7

achieving the expected outcomes. What is a performance measurement framework? A performance measurement framework is a plan to systematically collect relevant data over the lifetime of an investment to assess and demonstrate progress made in achieving expected results. It documents the major elements of the monitoring system and ensures that performance information is collected on a regular basis. It also contains information on baseline, targets, and the responsibility for data collection. As with the LM, the PMF should be developed and/or assessed in a participatory fashion with the inclusion of local partners, beneficiaries, stakeholders, etc. Standard template - performance measurement framework (PMF) CIDA has a standard Performance measurement Framework template The PMF is divided into eight columns: expected results, indicators, baseline date, targets, data sources, data-collection methods, frequency, and responsibility. To complete a PMF, you will need to fill in each of the columns accurately. Definitions Expected results column The expected results column is divided into four rows: outputs, immediate outcomes, intermediate outcomes, and ultimate outcome. To complete this column, simply copy and paste the result statements from your LM into the appropriate row. What are performance indicators? The performance indicators are what you will use to measure your actual results. A performance indicator is a quantitative or qualitative unit of measurement that specifies what is to be measured along a scale or dimension. However, it is neutral: it neither indicates a direction or change nor embeds a target. It is important that the stakeholders agree beforehand on the indicators that will be used to measure the performance of the investment. Quantitative performance indicators are discrete measures such as a number, frequency, percentile, and ratio, (e.g. number of human rights violations, ratio of women-to-men in decisionmaking positions in the government). Qualitative performance indicators are measures of an individual or group's judgement and/or perception of congruence with established standards, the presence or absence of specific conditions, the quality of something, or the opinion about something (e.g. the client's opinion of the timeliness of service). Qualitative indicators can be expressed concretely when used to report on achievement of results. They should convey specific information that shows progress towards results, and is useful for project management and planning. Example: Our investment has, as one of its immediate outcomes, "Increased ability to maintain wells among people living in region Y." Through consultation, it was decided that this would be measured by tracking "confidence of women and men who took training in their ability to maintain wells." The pre-training survey of women and men participating in the training showed that 3 percent felt that they were capable of maintaining wells. A survey conducted directly after training showed that 80 percent of participants felt that they were capable of maintaining the wells. As well, a follow-up survey at the midpoint of the investment showed that 75 percent of women and men who received training still felt that they were capable of maintaining the wells in their communities. Hint #5: The criteria of a strong performance indicator are as follows: Validity: Does the performance indicator actually measure the result? Your result statement is: "Increased use of clean drinking water by people living in community X." A valid performance indicator would be "percentage of households using drinking water drawn from clean source." An invalid performance indicator would be something like "number of wells in community X," because although this performance indicator would measure the availability of clean

RBM

Page 8

water (i.e., wells) it would not actually tell us whether people were using them and whether the number of people using them had increased. This would not actually measure your result. Reliability: Is the performance indicator a consistent measure over time? Your result statement is: "Increased access to health services for people living in region Y." A reliable performance indicator would be "percentage of population living within a two-hour walk of a health clinic." An unreliable performance indicator could be "gross mortality rate for region Y of country X." This performance indicator would not be reliable because it may not change consistently along with the result: a change in access to health care is not a change in usage, and thus may not be reflected in a change in mortality rates. Similarly, mortality rates can be affected by external and unpredictable circumstances (e.g. drought, natural disaster) that can change independently of the result. Sensitivity: When the result changes, will the performance indicator be sensitive to those changes? The example above for reliability also applies here. The performance indicator "gross mortality rate of region Y of country X" may not always be sensitive to a change in the availability of health care, or may be sensitive to other factors that are not directly linked to the result. "Percentage of population living within a two-hour walk of a health clinic," on the other hand, is sensitive to access to health services, and will change when it changes. Simplicity: How easy will it be to collect and analyze the data? An indicator may provide a good measure of the expected outcome but present too many challenges (such as complexity, technical expertise, local capacity, and shared understanding) for easy use. If your result statement is: "Increased ability to maintain wells among people living in region Y," the indicator "number of women and men in region Y who receive a passing grade on a practical well-maintenance exam" would provide an accurate measure, but would also require a complex and time-consuming data-collection process. The indicator "Confidence of women and men who took training in their ability to maintain wells," on the other hand, could be incorporated as pre- and post-exercises in the training activities, and be collected through a number of methods such as a written survey for training participants, a verbal response, or a response using the body (position in the room or height of a raised hand) to indicate level of confidence. Utility: Will the information be useful for investment management (decision making, learning, and adjustment)? If your result statement is: "Increased use of clean water among people living in region Y," various performance indicators could be used to measure this result. Some may be more useful for decision-making purposes than others. A performance indicator such as "Percentage of households using drinking water drawn from clean source" would provide information that could be used to take corrective action, if need be (e.g. to make adjustments to the project during implementation to ensure that the expected results will be achieved), or for planning subsequent phases of the investment (e.g. do there need to be more wells to facilitate greater use?). A performance indicator such as "Number of times well used daily" would indeed measure the result, but wouldn't provide a lot of useful information such as who was using it or how widespread that use was over the community. Affordability: Can the program/investment afford to collect the information? A household-by-household survey of the inhabitants of region Y to ascertain their opinion of the new wells and the training they received on how to maintain them may provide excellent performance data on the investment, but may also be too costly for the executing agency to conduct. Choose performance indicators that provide the best possible measurement of the results achieved within the budget available, and wherever possible, use existing sources and datacollection methods. Look for a balance between rigour and realism. What is baseline data? Baseline data form a set of conditions existing at the outset of a program/investment-the quantitative and qualitative data collected to establish a profile. Baseline data is collected at one point in time, and is used as a point of reference against which results will be measured or assessed. A baseline is needed for each performance indicator that will be used to measure results during the investment.

RBM

Page 9

What are targets? A target specifies a particular value for a performance indicator to be accomplished by a specific date in the future. It is what the investment would like to achieve within a certain period of time in relation to one of its expected results. Targets provide tangible and meaningful points of discussion with beneficiaries, stakeholders, and partners, and allow us to add further specificity to the outcomes from the LM. Targets however, belong only in the PMF; they should not appear in outcomes themselves, for a number of reasons. First, when targets are included in a result statement, they limit the ability to report against the achievement of that result by restricting success to an overly narrow window: the target itself. Reporting, in this context, becomes an exercise of justifying why the target was not met or was exceeded (both could be seen as poor planning) instead of comparing expected outcomes to actual outcomes and discussing the variance. In addition, outcomes need to be measurable statements that capture, with simplicity and specificity, an expected change. The inclusion of targets makes this impossible: a statement that includes its own measurement cannot then be assessed or measured. Hint # 6: Developing strong targets Targets must be realistic and reviewed regularly. Beneficiaries and stakeholders should be involved in establishing targets. Time lines for targets can vary from short to long term (e.g. monthly, midway, end of project). A strong target consists of a clear statement of desired performance against an expected outcome, and it is developed using an established baseline.

Example: Indicator: Percentage of households in region Y living within X distance of a well. Baseline: At the moment, 5 percent of households in region Y live within x distance of a well. Target: For the first year of the health initiative for region Y of country Z, the target is to have 25 percent of households living within X distance of a well. The target for the end of the initiative is to have 65 percent of households living within X distance of a well. This target is realistic because it takes into account the low percentage established during the baseline study and the fact that some communities in region Y are very remote and potentially difficult to work in.

What is a data source? Data sources are the individuals, organizations, or documents from which data about your indicators will be obtained. Performance data on some indicators can be found in existing sources, such as a land registry, appointment logs, tracking sheets, or the reports and studies carried out annually by actors in the international development community. Other data can be obtained through indicators tracked by governments and partner organizations, and reported in annual reports to donors. Finally CIDA staff and/partners may need to identify their own sources of data to track performance against expected results. The source of performance data is very important to the credibility of reported results. Try to incorporate data from a variety of sources to validate findings. Some examples of data sources: Beneficiaries Partner organizations (local and international) Government documents Government statistical reports Human Development Report

RBM

Page 10

What is a data-collection method? Data-collection methods represent how data about indicators is collected. Choosing a datacollection method depends on the type of indicator and the purpose of the information being gathered. It also depends on how often this information will be gathered. Hint # 7: Selecting appropriate data-collection methods: Determine which data-collection methods best suit the indicators in question. Use multiple lines of evidence. Consider the practicality and costs of each method. Weigh the pros and cons of each data-collection method (accuracy, difficulty, reliability, time).

Some examples of data-collection methods: Observation Analysis (of records or documents) Literature review Survey Interview Focus group Comparative study Collection of anecdotal evidence Questionnaire Pre- and post-intervention survey

Hint # 8: The identification of data-collection methods and data sources can help with the selection and validation of realistic indicators. Data sources and collection methods should be established in collaboration with partners, IOs, stakeholders and evaluation specialists.

What is ''frequency''? Frequency looks at the timing of data collection: how often will information about each indicator be collected and/or validated? Will information about a performance indicator be collected regularly (quarterly or annually) as part of ongoing performance management and reporting, or periodically, for baseline, midterm, or final evaluations? It is important to note that data on some indicators will need to be collected early in the investment to establish the baseline. What is ''responsibility''? Responsibility looks at who is responsible for collecting and/or validating the data. In the case of responsive or core programming, the implementing organization (IO) or multilateral institution, (together with its local partner(s) and potentially beneficiaries) has the primary responsibility for collecting and validating the data, and for using the data to report to CIDA on project performance. CIDA will use the information received as one of many potential tools to assess the performance of the organization's investment against the plan (their proposal; LM; PMF; risk register; project implementation plan, if relevant; and work plans). CIDA may also monitor and evaluate ongoing performance through exercises such as site visits and the inclusion of performance among criteria for evaluation in audit activities. Although the IO is responsible for the management of the investment, CIDA is responsible for ensuring adequate due diligence in the expenditure of Canadian funds. In the case of directive programming, CIDA has designed, in collaboration with the partner-country government or other partner organization, the project and is responsible (in a participatory fashion)

RBM

Page 11

for the LM and the PMF. CIDA retains an IO under contract to implement the project. This IO is responsible for collecting data in accordance with the PMF and for reporting results to CIDA. CIDA may also engage in independent data collection through monitoring or evaluation activities to validate the information being provided by the IO. CIDA is accountable for ensuring that the data is collected and the reporting is undertaken through the project steering committee and in conjunction with the recipient government partner. This performance information is used to assess overall progress, evaluate annual work plans, and take corrective action as required. Some examples of actors responsible for data collection/validation : Beneficiaries Local professionals Partner organizations Consultants External monitoring and evaluation specialists CIDA staff

Steps to complete a Performance Measurement Framework The development of a PMF starts at the planning-and-design phase. Remember that some elements of the PMF may be established after or during project implementation (e.g. collection of baseline data and setting some targets). 1. Ensure that the information for your PMF is developed in a participatory fashion, including key local stakeholders, partners, beneficiaries, and the appropriate specialists. 2. Cut and paste the ultimate outcome, intermediate outcomes, immediate outcomes, and outputs from your LM into the appropriate boxes in the PMF template. 3. Establish performance indicators for your expected outcomes and outputs, and enter the performance indicators for the ultimate, intermediate, and immediate outcomes and the outputs. Validate and check the quality of your performance indicators. Do they have: validity, reliability, sensitivity, utility, and affordability? 4. Establish the data sources and data-collection methods for your chosen performance indicators. Look to include multiple lines of evidence wherever possible to increase the reliability of your performance data. 5. Fill in the Frequency and Responsibility columns for each performance indicator. Decide whether information on each performance indicator needs to be collected on an ongoing basis as part of performance monitoring or periodically through evaluations. 6. Fill in baseline data where possible. If reliable historical data on your performance indicators exists (in the form of government data, information from a previous phase of the investment, or information gathered during a needs analysis), then it should be used; otherwise, you will have to collect a set of baseline data at the first opportunity. If you will be gathering the data later, indicate this in your PMF with a statement such as: "Baseline data to be collected at investment inception" or "Data to be provided by IO after communities identified." If possible, set the date by when this will be completed. This should be done within the first year. 7. Establish realistic targets for each indicator in relation to the baseline data you have identified. This sets the expectations for performance over a fixed period of time. Key targets, based on gaps and priorities identified during initial analysis, are necessary to establish budgets and allocate resources, and play an important role in project planning and design. Others may be established latter, once a baseline study had been conducted.

Example: Community X, CIDA, and the IO are working together to create a logic model and a PMF for their health investment.

Risk analysis
What is a risk register? A risk register lists the most important risks, the results of their analysis, and a summary of riskresponse strategies. Information on the status of the risk is included over a regular reporting

RBM

Page 12

schedule. The risk register should be continuously updated and reviewed throughout the course of a project. Integrated risk management Integrated risk management is a continuous, proactive, and systematic process to understand, manage, and communicate risk across the organization. Other government departments, other donors, and private-sector companies use similar frameworks. Elements of integrated risk management: 1. 2. 3. 4. Development of a corporate risk profile Establishment of an integrated risk-management framework Practicing integrating risk management at all levels Ensuring continuous risk-management learning

Drivers: Aid effectiveness - Integrated risk management supports a consistent approach to risk management both vertically and horizontally. CIDA is recognized as working effectively in a high-risk environment. By providing a common and consistent platform, we can reduce uncertainty for staff and managers and allow them to better understand and manage their risks. As a result, they will be in a position to make informed decisions and take responsible risks where appropriate. Good management - The Treasury Board Secretariat (TBS), Management Accountability Framework (MAF), Office of the Auditor General of Canada, and Organisation for Economic Co-operation and Development Development Assistance Committee (OECD-DAC) all highlight the importance of ranking and rating risks; identifying accountabilities for managing, reporting, and monitoring risks, scanning for new risks; and assessing risk collaboratively with other donors. The development of an integrated risk management approach is a TBS expectation under the MAF.

Key objectives: Integrated risk management helps make more informed decisions in managing risks that are within our control and will position us to better respond to risks that are beyond our control. Specific objectives are: Develop a systematic approach to risk management Contribute to a risk-aware culture Propose simpler, more effective practices Provide an ongoing scan of key risks.

Corporate risk profile: Producing a corporate risk profile (CRP) is a key step in developing an integrated risk management approach. At CIDA, the CRP was validated and approved in June 2008, and it takes into account CIDA's evolving working environment. The process used was iterative and based on a structure of 4 key risk areas and 12 key risks. The CRP is an evergreen document for risk management, and therefore must undergo annual reviews. Basic model (adapted from the World Bank; similar to those of AusAID, DFID, and others):

RBM

Page 13

Conclusions: CIDA understands and manages risk well. However, the Agency needs consistency, simplification, accountabilities, risk rating and ranking, and collaboration with other donors. A continuous learning culture exists. There is significant staff demand. Infrastructure is in place and accountable. Management of risks and due diligence is embedded in CIDA practices. Useful risk terminology: Risk refers to the effect of uncertainty on results (ISO 31000). Impactis the effect of the risk on the achievement of results Likelihood is the perceived probability of occurrence of an event or circumstance Risk level is Impact multiplied by Likelihood Risk response is the plan to manage a risk (by avoiding, reducing, sharing, transferring or accepting it) Risk owner is the person who owns the process of coordinating, responding to and gathering information about the specific risk as opposed to the person who enacts the controls. Stated otherwise, it is the person or entity with the accountability and authority to resolve a risk incident (ISO 31000) Operational risk is the potential impact on CIDA's ability to operate effectively or efficiently Financial risk is the potential impact on the ability to properly protect public funds Development risk is the potential impact on the ability to achieve expected development results Reputation risk is the potential impact arising from a reduction in CIDA's reputation and in stakeholder confidence in the Agency's ability to fulfill its mandate Standard template - risk register CIDA has a standardized investment risk register template Steps to complete a risk register: Step 1. Under Risk Definition, write down the key risks to the project. There should be at least two risks each for the categories Operational Risks, Financial Risks, and Development Risks, and at least one risk in the category of Reputation Risks. Step 2. For each risk selected, establish the current risk level, i.e. the intensity of the risk. A risk map or some other tool may be useful for determining the level. Identify the risk on the four-point scale below, and transfer the colour under Initial rating.

RBM

Page 14

Step 3. Over a regular monitoring schedule, re-rate the risk and add the colour under Date 2, and so on. Monitoring periods will vary according to the project, but a typical period is three months. Step 4. Indicate if the risk is the same as one found in the program risk assessment (if one exists). Step 5. A risk is an uncertainty about a result. Indicate the level of the result as found on your Logic Model. Step 6. Give a brief summary of the risk response strategies that will be used to manage the risk or to prevent a risk event. Step 7. Indicate the risk owner. If possible, there should only be one person per box. The owner will vary according who is the person that actually has to deal with a given risk event. Monitoring. In the real world of development, the risk profile will change constantly during the life of the project. As risks arise or disappear, change the corresponding risk definitions and risk level. Also track the use and the effectiveness of the risk response strategies, and change the Risk Response column as necessary. Note: Please do not hesitate to rate risks as "Very likely" if that is their real level. Four-point rating scale Criteria Likelihood of occurrence Potential impact on CIDA ability to meet objectives Very low (1) Very unlikely Routine procedures sufficient to deal with consequences Low (2) Unlikely High (3) Likely Very high (4) Very likely Would prevent achievement of goals and objectives

Could threaten Would threaten goals and goals and objectives, and thus objectives, and may require thus may require monitoring review

PDF Format Note: to read the PDF version you need Adobe Acrobat Reader on your system. If the Adobe download site is not accessible to you, you can download Acrobat Reader from an accessible page. If you choose not to use Acrobat Reader you can have the PDF file converted to HTML or ASCII text by using one of the conversion services offered by Adobe. Logic Model with definitions (PDF - 21 KB, 1 page) RTF To read the RTF version, use the document conversion features available in most modern word processing software, or use a file viewer capable of reading RTF. Logic Model Template (RTF - 80 KB, 1 page) Performance Measurement Framework Template (RTF - 70 KB, 1 page) Investment Risk Management Template (RTF - 250 KB, 4 page)

RBM

Page 15