Im going to reflect on two very different kinds of evidence-making in a single INGO,

considering their claims to truth, and asking evidence for what, and whom?
(Youll see some evidence in this of the fact that I spent the summer researching right
wing populism.)

The INGO is Oxfam. Perhaps inevitably, given its size and reach, there are many different
evidence-making practices within it. Im going to focus on two of the most visible
impact evaluation and research for advocacy.

Since 2011, in addition to routine monitoring and evaluation, Oxfam GB has been
conducting rigorous impact evaluations effectiveness reviews on a random sample
of its projects worldwide, stratified by theme. The methods used match both the themes
and the number and kind of intended beneficiaries. Where, for example, were looking
at impact on a large enough number of households, we use quasi-experimental research
designs. When assessing policy influencing weve experimented with process tracing
and so on.
(My role in this has been undertaking a number of deep dives, in-depth follow-up
studies on a purposive sample of quantitative effectiveness reviews.)

The effectiveness reviews are evolving. When Im being charitable and I usually am I
think of this, and impact evaluation in general, as a progressive research programme.
As this audience is aware, its not without its problems, methodological and otherwise.
In my own work Ive been particularly struck by mistakes in implementation (some
stemming from lack of good researchers, both quantitative and qualitative).

Theres more at stake here than just improving the evidence. Impact evaluation hasnt
been evolving in a moral vacuum, but has been co-opted and promoted as part of
particular political agendas. I think you can see this in one of its major fault lines
accountability vs. learning. In Oxfam were nowhere near getting all the learning that we
might from impact evaluation and that its proponents hope.
(We are, however, trying hard, and are currently producing a series of meta-analyses of
our effectiveness reviews.)

Indeed the truth-claims of some evaluation evangelists are unjustified they bear a
distinct resemblance to parts of this wonderful spoof (see the slide).
(Recall here to the definition of evidence shown in an earlier presentation, with its
reference to validity and truth.)

That kind of epistemological arrogance is certainly unjustified when it comes to the

evaluation of campaigns.
Making change happen in this way is now the major focus of Oxfams work (in addition
to humanitarian response and long-term development).
(Plug for Duncan Greens new book.)

Research is just one component of advocacy, and it can play smaller and larger
roles. Ten years ago one of my colleagues produced a guide or aide memoire to
research for advocacy in the form of a diagram, shown here.
It makes some audiences quite uncomfortable (from university professors to
Bank of England economists), as though they were not trying to influence people
themselves with evidence, even if its only a journal editor or assessors in the
REF (Research Excellence Framework).

The key is the way in which we analyse power, identify potential audiences who can
and should we try to influence and think through the kinds of evidence that would
most likely achieve that (the categories in the table are indicative they might include
RCTs, randomised controlled trials, in some contexts).

This can work spectacularly well, as it has in our inequality campaign and advocacy
around Davos.

10

Does it mean that to coin a phrase were speaking post-truth to power?(!) Theres
always some danger that evidence-based policy-making can morph into policy-based
evidence-making, and internal arguments around this are not unknown. But we do
check our killer facts, engage with and employ experts, and aim to use evidence that
will stand up to critical scrutiny.

11

The evidence from impact evaluations (our own and others) may be one part of this,
thought at present Id say that it is only a small part, with a potential to grow.

12