NGO Policy Briefing Paper No.

3, March For the NGO Sector Analysis 2001

Programme

NGOS AND IMPACT ASSESSMENT 1. Introduction

The role and approach of Non-Governmental Organisations (NGOs) in development has changed radically over the last fifteen to twenty years. NGOs are now accepted as significant contributors to the development process by governments and official agencies. In the early 1980s it was assumed that NGOs would have an impact because of who they were and their relationship and closeness to the beneficiaries. This unsubstantiated assumption has increasingly come into question; throughout the 1990s the issue of assessing NGO impact and the need for appropriate methodologies to do so came to the Definitions fore. As the of Impact profile of NGOs hasof increased, too has the forFor them to assess the impact of their There are a number different so definitions of need impact. example, impact is work. The rise in improvements in the lives and popularity of NGOs and the increase in funding channelled through them byby governments livelihoods of beneficiaries (OECD/DAC, 1997). The following definition Blankenberg has had Impact concerns long-term and sustainable changes introduced by a given (1995) is consequences in terms of performance accountability (Edwards and Hulme In intervention in particularly helpful because it is relatedand to the key concepts surrounding impact 1995). assessment: addition, theNGOs lives of beneficiaries. Impact can be related either to the specific objectives have become of an more critically aware themselves of the need to assess their impact, both for organisational intervention or to unanticipated changes caused by an intervention; such learning and strategy development and in order to inform an increasingly discerning public unanticipated The key concepts in Blankenbergs definition are: supporter base.may also changes occur in the lives of people not tocomes the beneficiary Impact assessment is about sustainable change. This is belonging change that about as a result This Policy Briefing Paper explores the current state of the debate on impact assessment of group. Impact of project development can be either positive or negative, the latter being equally important to be or programme activities. interventions. It moves on be to unanticipated. review the current state of and has methodologies and it aware of. can even These changes A project orpractice programme objectives which concludes with a (Blankenberg, 1995.) is hoped will summary of critical issues and implications NGOs. be achieved through the planned activities.for It may be that through links with other projects or the catalytic effect of the project orat programme changes to the onesPaper. proposed have Jerry Adams, Senior Consultant INTRAC, additional prepared this Policy Briefing come about. Comments on the paper It essential to remember that change can be negative. Negative change can befor attributable are welcome and canproduced be sent to: j.adams@intrac.org This paper has been as part of the NGO Sector Analysis Programme the to an following European intervention or be due to wider circumstances beyond the control of those managing a organisations: APSO, Cordaid, Concern Worldwide, DanChurchAid, MS Denmark, project or Church Aid, Norwegian programme. Novib, Rdda Barnen, Redd Barna and Save the Children Fund - UK. For further 1 information about the programme please contact Vicky Brehm at INTRAC (v.brehm@intrac.org)

Definitions of Impact Assessment Impact assessment describes an assessment of the longer term and sustainable changes that are planned to occur from development interventions. The term is relatively new to the development community but has The clarifies the difference between outputs, outcomes and impact. beendiagram in use inbelow environmental impact and social impact assessment for many years. Impact (Fowler, 1997). assessment has Point of MeasurementWhat isevaluation MeasuredIndicators often been confused with the of a projects or programmes immediate OutputsEffortImplementation of activities objectives. OutcomesEffectivenessUse of outputs and sustained production of benefits ImpactChangeDifference from the original problem situation

2. The Complexity of Impact Assessment There are three main difficulties in relation to the development of an appropriate

methodology for impact assessment: Firstly, there is often confusion between the evaluation of a project or programmes objectives and an assessment of its long-term impact. An OECD/DAC study tools on impact (Kruse et al, 1997) concluded there was a lack of in Secondly, evaluation and methodologies are often poorlythat developed. This results firm and reliable weaknesses in the chain of evidence from data collection through to analysis and evidence on the impact of NGO development projects and programmes, related to the conclusions. paucity of data andof assessing impact qualitatively is very complex. Thirdly, the issue weakness of evaluation methodologies. Often project evaluations consisted of descriptions of activities carried out rather than any analysis of the relevance of what had been done. This was partly due to a lack of appropriate evaluation and assessment methodologies. A further reason was that The essential starting point in developing a methodology for assessing impact is the need to insufficient attention take a primarily had been paid to the development and use of appropriate monitoring processes and qualitative procedures.perspective. This was It is therefore important to draw a clear distinction between a qualitative and a by the fact that many projects evolve and change over time. Whist this is often compounded quantitative approach. Firestones definition (left) is helpful not a negative Qualitative research attempts tobecause it shows that the scope of an assessment defines factor per se, it made tracking a projects progress and impact almost impossible when the coupled with persuade through rich systems. depiction andmethodologies to be used, application and to the insufficient monitoring The DAC Study concluded that their it was very difficult amount make anof assessment strategic comparison across cases,time to the reports assessment. of impact because having reviewed manygiven evaluation there Where was the project or programme There was not a clear chain a)thereby Face validityAre wethe measuring what we assessed think we is are measuring? overcoming abstractionbeing very large, clearly a very weak link between the information gathered and theit is not necessary to b) BiasIs there a tendency to make errors in one direction? assess all of evidence (Yin 1989). c)inherent Convergent validity Is the information gained through one method by that in quantitative studiesareas of the work but enough to gainvalidated a clear picture. conclusions drawn. gained through (Firestone 1987). another method? The scope of a specific assessment visit needs to be such that d) Internal validityAre the results this group? it will enable the assessment to begenuine reliable,for valid and credible. Validity and credibility are e) External validityAre the results applicable to other situations? taken to mean are the results believable? There are different understandings of how to assess validity. 2 For example Cohen and Manion (1989) take a very positivistic stance and see validity as follows: