5/21/2019 Selecting evaluation questions and types - Learning for Sustainability

(http://learningforsustainability.net/)

(http://learningforsustainability.net/)

Search … Search

Menu 

Selecting evaluation questions and types

Evaluations provide an opportunity for the initiative’s overall progress to be considered, including focused
consideration of specific aspects of the initiative. A small number of Key Evaluation Questions (KEQs)
(http://learningforsustainability.net/post/key-evaluation-questions/) help provide this focus.These are not
questions that are asked in an interview or questionnaire, but high level research questions that will be
answered by combining data from several sources and methods. Using a Theory of change
(http://learningforsustainability.net/theory-of-change/) approach with an accompanying project logic model
(http://learningforsustainability.net/logic-models/) provides an outline that helps develop measures of success
that trace the projects development and impact over time. These, in turn, need to be focused with appropriate
evaluation questions that are driven by funders, project participants and other key stakeholders. The five criteria
to evaluate development interventions (relevance, effectiveness, efficiency, impact, and sustainability) outlined
in the OEDC/DAC evaluation guidelines
(http://www.oecd.org/dac/evaluation/daccriteriaforevaluatingdevelopmentassistance.htm) provide a good starting
framework. A useful starting set of key evaluation questions (KEQs) (http://learningforsustainability.net/post/key-
evaluation-questions/) based around these criteria could, for example, include:

Is the initiative delivering on outputs and outcomes as planned? (efficiency and effectiveness)
Are the (or were the) activities and their delivery methods been effective? Are there aspects that could
have been done differently? (process effectiveness)
Is the wider project story being told? What range of outcomes (intended and unintended) has the project
contributed to – taking account of each of social, economic, environmental and cultural considerations

learningforsustainability.net/evaluation-questions/ 1/6
5/21/2019 Selecting evaluation questions and types - Learning for Sustainability

(relevance and impact)

How has the initiative influenced the appropriate stakeholder community, and what capacities has it built?
(relevance and impact)
Has the initiative being delivered on budget? (efficiency)
Is the project impacting positively on key groups and issues that have been identified as important in
project design – particularly gender, indigenous, youth and environment? (relevance and impact)
Is there evidence that the initiative is likely to grow – scaling up and out – beyond the project life?
(sustainability)

Evaluation types (and/or methods) are distinguished by the nature of the questions they attempt to answer. It is
important to begin an evaluation by being clear on what is wanted from the evaluation. A logic model helps by
providing a project outline that helps develop different measures of success that trace the projects development
and impact over time. There are other frameworks which can provide ideas as to the scale and levels of
programme intensity to be considered by stakeholders. As these perspectives illustrate, all but very simple
projects involve a number of elements, and so many evaluation projects will involve using a number of methods
to look at different questions. The diagram below shows how different evaluation approaches and methods can
be used to measure different parts of the overall project or change intitiative.

A project logic model showing how different evaluation types and approaches can be used to measure progress
through different stages of implementation

Different project stages require different evaluation questions to assess progress:

Needs Assessments: These evaluations verify and map the extent of a problem. They answer questions about
the number and characteristics of the individuals or institutions who would constitute the targets of a program to
address the problem. Needs assessments can help design a new program or justify continuation of an existing
program.
Accountability: Monitoring activities produce regular, ongoing information that answers questions about
whether a program or project is being implemented as planned, and identifies problems and facilitates their
resolution in a timely way.
Formative evaluations: These evaluations answer questions about how to improve and refine a developing or
ongoing program. Formative evaluation usually is undertaken during the initial, or design, phase of a project.
However, it also can be helpful for assessing the ongoing activities of an established program. formative
evaluation may include process and impact studies. Typically, the findings from formative evaluations are
learningforsustainability.net/evaluation-questions/ 2/6
5/21/2019 Selecting evaluation questions and types - Learning for Sustainability

provided as feedback to the programs evaluated.

Process evaluations: Studies of this kind are directed toward understanding and documenting program
implementation. They answer questions about the types and quantities of services delivered, the beneficiaries of
those services, the resources used to deliver the services, the practical problems encountered, and the ways
such problems were resolved. Information form process evaluations is useful for understanding how program
impact and outcome were achieved and for program replication. Process evaluations are usually undertaken for
projects that are innovative service delivery models, where the technology and the feasibility of implementation
are not well known in advance.
Impact or Outcome Evaluations: These evaluations assess the effectiveness of a program in producing
change. They focus on the difficult questions of what happened to program participants and how much of a
difference the program made. Impact or outcome evaluations are undertaken when it is important to know how
well a grantee’s or foundation’s objectives for a program were met, or when a program is an innovative model
whose effectiveness has not yet been demonstrated.
Summative Evaluations: Summative evaluations answer questions about program quality and impact for the
purposes of accountability and decision making. They are conducted at a project’s or program’s end and usually
include a synthesis of process and impact or outcome evaluation components.

A practical guide for engaging stakeholders in developing evaluation questions

(http://www.fsg.org/tools-and-resources/practical-guide-engaging-stakeholders-developing-evaluation-
questions-0) In this report Hallie Preskill and Nathalie Jones describes a five-step process for engaging
stakeholders in developing evaluation questions, and includes four worksheets and a case example to further
facilitate the planning and implementation of your stakeholder engagement process. The report recognises that
one way to ensure the relevance and usefulness of an evaluation is to develop a set of evaluation questions that
reflect the perspectives, experiences and insights of as many relevant individuals, groups, organizations, and
communities as possible. As potential users of the evaluation findings, their input is essential to establishing the
focus and direction of the evaluation. By soliciting the opinions, interests, concerns and priorities of stakeholders
early in the evaluation process, the results are more likely to address stakeholders’ specific information needs
and be useful for a range of purposes, among them to improve program effectiveness, to affect policy decisions
and/or to instigate behavioral change.

Developing a monitoring and evaluation plan (http://evaluationtoolbox.net.au/index.php?

option=com_content&view=article&id=20&Itemid=159). This page from the Australian-based
Community Sustainability Engagement Evaluation Toolbox provides guidance for developing a monitoring and
evaluation plan. The plan outlines the key evaluation questions and the detailed monitoring questions that help
answer the evaluation questions. The site was developed and is maintained by Damien Sweeney and Martin
Pritchard from Pacific Research & Evalauation Associates (http://prea.com.au/) (PREA (http://prea.com.au/)).

Specify the Key Evaluation Questions

(http://betterevaluation.org/plan/engage_frame/decide_evaluation_questions). This BetterEvaluation
post reminds us that having an agreed set of Key Evaluation Questions (KEQs) makes it easier to decide what
data to collect, how to analyze it, and how to report it. Try not to have too many Key Evaluation Questions – a
maximum of 5-7 main questions will be sufficient. It might also be useful to have some more specific questions
under the KEQs.

Key evaluation questions

(https://evaluationinpractice.files.wordpress.com/2008/01/keyquestionschoices.pdf) – This presentation
illustrates what is meant by Key evaluation questions (KEQs). It reminds us to involve stakeholders in
identifying appropriate and relevant questions. The authors point out not to start with evaluation methods – the
aim & purpose of an evaluation should always decide the method, rather than the reverse.

learningforsustainability.net/evaluation-questions/ 3/6
5/21/2019 Selecting evaluation questions and types - Learning for Sustainability

(/#facebook) (/#twitter) (/#google_plus)

(https://www.addtoany.com/share#url=http%3A%2F%2Flearningforsustainability.net%2Fevaluation-
questions%2F&title=Selecting%20evaluation%20questions%20and%20types)

Site information

About this site (http://learningforsustainability.net/about/)

Will Allen & Associates (http://learningforsustainability.net/willallen/)
Contact (http://learningforsustainability.net/contact/)
Subscribe to site update (http://learningforsustainability.net/subscribe/)

Recent Posts

An introduction to systems thinking and tools for systems thinking

(http://learningforsustainability.net/post/intro-systems-thinking/)
Six ways that agencies can influence behaviour through public policy
(http://learningforsustainability.net/post/6es-policy-framework/)
Design and facilitation of collaborative processes (http://learningforsustainability.net/post/design-
facilitation-collaborative-processes/)
2017 in review – your favorite LfS content (http://learningforsustainability.net/post/2017-in-review/)
Social learning – what it looks like (http://learningforsustainability.net/post/social-learning/)
After action reviews – and how they can be linked with ToCs to support strategic thinking
(http://learningforsustainability.net/post/after-action-reviews/)
Key evaluation questions (KEQs) (http://learningforsustainability.net/post/key-evaluation-questions/)
More about outcomes – why they are important … and elusive!
(http://learningforsustainability.net/post/outcomes-important-elusive/)
Diagramming a theory of change (http://learningforsustainability.net/post/diagramming-theory-change/)
Using a theory of change (ToC) to better understand your program
(http://learningforsustainability.net/post/theory-of-change/)

Twitter feed

learningforsustainability.net/evaluation-questions/ 4/6
5/21/2019 Selecting evaluation questions and types - Learning for Sustainability

Tweets by @will_allen
Will Allen Retweeted

Dr Sandra Velarde
@sandrinha2021

Jack Craw - Biosecurity NOT losing the challenge. Climate change an opportunity for
funding biosecurity? - Biosecurity, an opportunity to share control. #CA2 @BioHeritageNZ

20h

Will Allen Retweeted

Andrea Byrom

Embed View on Twitter

(/#facebook) (/#twitter) (/#google_plus)

(https://www.addtoany.com/share#url=http%3A%2F%2Flearningforsustainability.net%2Fevaluation-
questions%2F&title=Selecting%20evaluation%20questions%20and%20types%20-
%20Learning%20for%20Sustainability)

About this site

This site is compiled and maintained by Will Allen (http://learningforsustainability.net/willallen/) (PhD) For more
information you can visit the site background (http://learningforsustainability.net/about/) page. (c) 2005-2017
Learning for Sustainability [Formerly NRM-changelinks (1998-2006)]

Subscribe to site update

learningforsustainability.net/evaluation-questions/ 5/6
5/21/2019 Selecting evaluation questions and types - Learning for Sustainability

Newsletter postings advise of major changes and additions to this site, and will be sent out as an occasional LfS
site update e-mail to those of you who wish - 2 to 3 times/year. You can subscribe here
(http://learningforsustainability.net/subscribe/).

Contact
Thanks in advance for your comments, ideas or feedback. Its good to make contact with others interested in this
area, Feel free to suggest additional content to link to. Just use the site contact form
(http://learningforsustainability.net/contact/).

About this site (http://learningforsustainability.net/about/) | Disclaimer (http://learningforsustainability.net/about/) |

Privacy policy (http://learningforsustainability.net/privacy-policy/) | About me (http://learningforsustainability.net/willallen/) |
Contact me (http://learningforsustainability.net/contact/) |
Subscribe to LfS site update email (http://learningforsustainability.net/subscribe/)

 
(ht (ht
tps tps
:// ://p
 w lus
(ht ww .go
tps .lin ogl
://t ke e.c
wit din om
ter. .co /u/
co m/i 0/+
m/ n/ Wil
will will lAll
_al all en
len en NZ
) nz) )

© 2019 Learning for Sustainability (http://learningforsustainability.net/)

Except where otherwise noted, this work is licensed under a Creative Commons Attribution 3.0 License
(http://creativecommons.org/licenses/by/3.0/).

learningforsustainability.net/evaluation-questions/ 6/6