VITAL TRANSPORT AND EDUCATION

INFRASTRUCTURE (VTEI) PROGRAMME:

IMPACT EVALUATION

Proposed Methodology for Impact Assessment of Vital Transport &

Education Network (VTEI) Program (Revised Draft)

Authors:
Team Lead: Noaman Saeed,
Team members: Fida Muhammad & Khurram Jilani

4th June 2014

Table of Contents
1.0 Background: ....................................................................................................................................................3
2.0 Purpose: ..........................................................................................................................................................3
3.0 Primary Goal ...................................................................................................................................................3
4.0 Constraints with availability and quality of baseline data & attribution........................................................3
5.0 Definition of Impact Assessment - IE ..............................................................................................................4
6.0 Suggested route for conducting impact evaluation .......................................................................................4
7.0 A brief on mixed-method approach ...............................................................................................................4
8.0 Broad objectives of Phase-I ............................................................................................................................5
9.0 Matrix of proposed activities for Phase -1 .....................................................................................................5
10.0 Data Quality Assurance activities .................................................................................................................6
11.0 Proposed investigative themes ....................................................................................................................7
Inputs ................................................................................................................................................................8
12.0 Expected deliverable ....................................................................................................................................9
12.1 Phase – 1 activity: .....................................................................................................................................9
12.2 Phase 2: (Implementation) .................................................................................................................... 10
13.0 Proposed Preliminary Tasks................................................................................................................... 10
14.0 Suggested evaluation approach: ............................................................................................................... 11
14.1 Making use of the “Participatory Impact Assessment”, contextual approach– PIA technique, ........... 11
14.2 Rationale ................................................................................................................................................ 11
14.3 A key requirement ............................................................................................................................. 12
14.4 Community-defined impact indicators of project impact ..................................................................... 12
14.5 Sampling for Qualitative analysis .......................................................................................................... 12
14.6 Proposed step by step implementation plan and tools to be used. ..................................................... 13
14.7 Quantifying data from CSC technique ................................................................................................... 13
14.8 Complimentary evaluation techniques to be explored. ........................................................................ 14
15.0 Stakeholder wise Implementation Methodology and proposed tools of evaluation ............................... 15
1.0 Background:
As is clear from the ToRs, the contractor clearly requires assessment of both intermediate ‘outcomes’
and ultimate ‘outcomes and impact’ of the VTEI initiative. The situation at hand provides three
Annual Reviews – reporting on the numerous successes of the programme while identifying the
multitude of challenges faced during the implementation process. The findings of these reviews
include recommendations for how the programme can be improved. The gap in DFID’s understanding
is the extent to which the funding has contributed to social and economic development in the
Malakand Region.

2.0 Purpose:
The purpose of the intended evaluation is to provide an estimate of programme’s impact for
accountability purposes and to make suitable recommendations for further improving the benefits
of on-going phases of the programme. The evaluation is intended to:

 Provide DFID and its partners, with an initial assessment of the contributions that the VTEI
Programme has made towards bringing about a Socio-Economic Development of the
Malakand Region, in-line with its stipulated Goal1 & Purpose2;
 Suggest any improvements that can be made to inform the implementation of remaining
bridges to be built in the region; and,
 Introduce application of mixed method approach to Impact Evaluation of the VTEI program;
thereby amplifying the beneficial impacts of the programme interventions.

3.0 Primary Goal

The primary goal that DFID seeks is not an impact evaluation that directly attributes any changes
observed at the “outcome” level to the programme, it in fact seek to estimate the contribution that
the programme has made to changes observed in the region (i.e. socio-economic development)
including increased public confidence in the state

4.0 Constraints with availability and quality of baseline data & attribution
1. Some baseline surveys were undertaken before the launch of the programme and regular
beneficiary feedback surveys as well as traffic counts are being undertaken to monitor key
indicators for the programme. There is not, however, a comprehensive assessment of
baseline data in terms of its coverage of all programme sites (schools and bridges) or its
quality (in terms of reliability and validity).The availability and establishing of robustness of
baseline data could potentially become an issue.

2. Secondly, it is deemed difficult to create a valid control group of citizens that would not have
access to the rebuilding benefits of schools’ and bridges’ (at the locations where they have
already been provided). The locations where bridges are yet to go up or where schools still
don’t exist (or have not been repaired / replaced post the floods’ damage) would need to be
assessed for the extent to which they are comparable and could therefore act as an Ante -
construction scenario. It is likely, however, that they will be systematically different across
the range of socio-economic indicators making comparisons invalid.

1
Goal is the term DFID used at the time when the VTEI programme was designed, prior to adopting ‘Impact’ in current Log frames and Theories of
Change.
2
Purpose is the term DFID used at the time when the VTEI programme was designed, prior to adopting ‘Outcome’ in current Log frames and Theories
of Change.
5.0 Definition of Impact Assessment - IE
3The OECD-DAC (Organisation for Economic Co-operation and Development – Development
Assistance Committee) definition of impact with its emphasis on long-term effects can be taken as
the basis for defining Impact Evaluation for this report as well. The definition of IE looks to
establishing cause and effect relationships between development interventions and development
results, although not exclusively through counterfactual-based methods.

Emphasis on ‘contributory’ causes can be assumed to be a major theme in this study which is
consistent with the broad consensus that development aid interventions work best in combination
with other non-aid factors. This contribution-based logic is also consistent with the complex and
multi-dimensional nature of VTEI development interventions as specified in the ToRs.

Contributory causality is relevant when it is likely that there is more than one possible cause, i.e. the
intervention is just one part of a causal package. If there are several different but similar programmes
operating in the area then various methodologies coupled with the comparative ‘Case’ design
approach may also be considered relevant. If VTEI is the only programme operating in that area, then
unpacking various case details through within-(VTEI) case analysis by reviewing the existing ‘Theory
of Change’ and using ‘process tracing’ can be applied and finally eliminating alternative explanations
may be considered appropriate4.

Note: This approach is to be considered subject to the literature review and in conjunction with the
analysis of existing qualitative and quantitative data based on ground realities.

As the ToRs for this study emphasise, with the growing importance of ‘qualitative’ methods the
distinction between quantitative and qualitative approaches is both clarified and challenged. This
provides the opportunity of combining different methods – quantitative and qualitative (E.g.
quantifying participatory survey results through ranking based perception matrices or similar scales
encompassing investigative variables and triangulation of survey results through findings of Focus
Group Discussions).

6.0 Suggested route for conducting impact evaluation

Given the data and attribution constraints (difficulty in establishing a counterfactual etc.) mentioned
in the ToRs, strategy would be to undertake an impact evaluation that:

 Does not rely on ONLY measuring quantitative changes in indicators over time - and
 Does not rely on having comparison groups alone to establish a counterfactual. (Although the
option of their formation will still be investigated wherever possible)

7.0 A brief on mixed-method approach

i- “An impact evaluation that combines qualitative and quantitative methods can
generate both a statistically reliable measure of the magnitude of the impact as well as
a greater depth of understanding of how and why a programme was or was not effective
and how it might be adapted in future to make it more effective (DFID, 2008).

3
“Broadening the range of designs and methods for impact evaluations” - Report of a study commissioned by the Department for
International Development - APRIL 2012; Stern et, al
4
“Eliminating alternative explanations is a standard way to leverage causal inference in science”. - Formalised by Michael Scriven
(1976) in his General Eliminative Method (GEM) approach
 ii- Combining designs and methods – even within the same design ‘approach’ –
strengthens causal claims

Good evaluations are almost invariably “mixed method evaluations”. Qualitative information informs
both the design and interpretation of quantitative data. In a “Theory-based approach”, as being
rightly favoured in the ToRs for this study, reliance on qualitative data provides vital context and a
means to verify quantitative data or to strengthen it in case it is not robust or representative enough
as seems to be the case for VTEI study so far. The “mixed methodology” approach being suggested
will have to match the activities as mentioned in the ToRs to ensure line by line compliance to all the
required objectives and constraints thus identified both for Phases 1 & 2 i.e.

8.0 Broad objectives of Phase-I

 To establish first contact with the stakeholders of VTEI initiative and form a consensus on
methodology for impact assessment based on existing evidence and data in hand.
 To address uncertainties over the most appropriate indicators to be undertaken based on
compliance with an internationally recognized set of standardized indicators (E.g. UKAID &
UNGASS socio-economic indicators) in order to enrich the existing list of indicators used to
measure the programme’s impact.
 Verify linkages of existing “theory of change” and suggest improvements to enrich its
potential to deliver the envisaged outcomes and impacts through a carefully designed
“implementation plan” having consensus of all concerned.

9.0 Matrix of proposed activities for Phase -1

PHASE - 1: Research into outcome measures and data quality - (An activity spanning one month common for both components of Bridges and
Main Activities
Inception meetings with key
stakeholders for both Schools'
and Bridges' component
Collection of existing data for
both Schools' and Bridges'
component
Review of data/documentation
and collation of key information core National and
Building a consensus and
getting agreement from DFID
& GoKP on the overall
evaluation methodology &
variables of investigation
Inception Report
Schools)
Type of
Stakeholders Assessment Methodology Literature Review
methodology
1) Three Annual reports and
DFID, Ministry of
One on One interviews and FGDs where possible with baseline surveys, available
Education - GoKP, (Participative)
relevant GoKP members, TA team & Provincial C&W review of school and bridges
TA and Provincial Qualitative
Dept. staff construction work records
C&W Dept. teams
2) NEMIS record
(1) Desk reviews, Literature review on existing material 3) "Pakistan Social and Living
DFID, Ministry of
& (2) Baseline data where available, (3) NEMIS data Standards Measurement Survey
Education - GoKP, (PSLM)", 2010-11 data with
for Education data and (3) If permitted, preperation of Qualitative
TA and Provincial 2006-07 data and see the
GIS base map marking for location of bridges &
C&W Dept. teams changes in these indicators
beneficiary communities
4) District budgets allocations
Desk Reveiews & Discussions to collate gathered and expenditures to compare
DFID - CRHR team
existing data with proposed survey methodology and field the Percentage of budgetary
IMC Worldwide (Participative)
activities. (Situational Analysis). Reviewing the Theory support in school construction to
Qualitative
of change indicators with international socio-economic overall government education
International team
indicators to enrich them sector development budge
DFID - CRHR team 5) Education sector KPIs
Sharing of findings and suggesting a proposed "mixed- 6) Review of Malakand
, GoKP, IMC
method" methodology approach with the relevant Comprehensive Stabilization (Participative)
Worldwide core
stakeholders to be adopted at the implementation phase - and Socio-economic Qualitative
National and
2 development Strategy
International team
DFID - CRHR team 7) Reviewing UNAID &
, GoKP, IMC UNGASS socio-economic
Inception Workshop to present Draft Report outline, indicators for enrichment of
Worldwide core Qualitative
Revision & submission of finalized draft inception report existing variables.
National and
International team

As specified in the detailed action plan, the Phase -1 activity would span a period of one month in
which the team would endeavour to establish first contact with all the mentioned stakeholders
relevant to the VTEI initiative. That would include DFID-CRHR team, members of the GoKP relevant
ministries/departments of education and construction/ works along with the TA team responsible
for construction of bridges.

One of the major objectives of the start-up meetings is to address the concept of “Attribution” and
improve the understanding of situation on ground. The consultants will sit down with stakeholders,
primarily DFID-CRHR team to identify/verify a set of ‘attributes’ that could likely to have design
implications. These attributes could include identifying:

 Overlap with other interventions with similar aims

 Multiple and diverse activities and projects within the purview of VTEI
 If the likely impacts of programmes have long term or medium term implications
 Challenges faced due to working in areas of high risk or uncertainty as in the case of VTEI
 To identify if the stated impacts are difficult to measure, possibly intangible
 Others

This phase will also see the team designing the data collection tools that would ensure collection of
data as per the investigation themes and going through available data for the two components of the
project.

10.0 Data Quality Assurance activities

The objective of this exercise will also be to investigate the quality and reliability of existing data
through internationally recognized QA tool or any other tool as agreed upon by the DFID-VRHR team.

The team will also look into the available literature regarding various reported variables to establish
an investigative “baseline”. The main purposes of any QA system should be to ensure that Impact
evaluations can be defended and justified; whilst at the same time encouraging good practice among
evaluators. As highlighted in the ToRs, the quality and relevance of the available baseline data lacks
comprehensive evaluation. Under these circumstances the following options can be considered
which are in line with the two major activities identified for phase -1:
 Identify core data points relevant to the links of the existing “theory of change” diagram and
investigate relevance and robustness of available data (utilising the three Annual Reports,
regular beneficiary/stakeholder feedback, baselines conducted earlier at inception of the
program and the on-going traffic counts). All qualitative and quantitative data will need to be
evaluated as part of the initial “desk review” to identify gaps and areas of improvement.

 Ensure that various aspects relevant to data quality assurance norms are fulfilled. For that,
Lincoln & Guba’s naturalistic criteria could be a good starting point as evidenced by the
following matrix:

Aspect Scientific term Naturalistic term

Truth Value Internal Validity Credibility
Applicability External validity / generalisability Transferability
Consistencey Reliability Dependability
Neutrality Objectivity Confirmabilty

(Source: Spencer et al 2003)

  In order to ensure that various above mentioned QA aspects are catered for, it is
recommended to utilise an established data quality assessment tool5 to establish the
robustness and perform QA check on the entire existing stream of data prevalent at VTEI. The
QA Tool focuses exclusively on
o Verifying the quality of reported data, and
o Assessing the underlying data management and
o A reporting systems for standard program-level output indicators

Preferred tools to be recommended for use include OECD-DAC Quality Standards and Europe Aid
‘checklist’ both aim to ‘improve the quality of development evaluation processes and products’ and
reflect a wider concern to improve practice by influencing what evaluators do.

The objectives of any of the selected QA Tool will be to:

 VERIFY rapidly 1) the quality of reported data for key indicators at selected sites; and 2) the
ability of data-management systems to collect, 3) manage and report quality data.
 IMPLEMENT corrective measures with action plans for strengthening the data management
and reporting system and improving data quality.
 MONITOR capacity improvements and performance of the data management and reporting
system to produce quality data

11.0 Proposed investigative themes

Those are themes the team aims to cover in the Phase – 1 of the activity will gravitate around the
relevance of concepts and linkages provided in the theory of change. The line of investigation will
follow a set sequence in-sync with the theory of change format segregating queries according to
Inputs, Activities, Outputs, Outcomes and Impact. This sequence will ensure uniformity of
questioning as well as help in avoiding any probability to leave any area of interest. This list is not
exhaustive and is based on the given information of the ToRs.

The team of consultants intend to refine the line of questioning during their meetings with identified
stakeholders for Phase -1 activity where it is expected that more areas of investigation might be
identified and a more thorough investigation will be required for newly identified areas of interest.

A graphical representation of the intended queries is elaborated in the following matrix:

5
This tool was developed with input from a number of people from various organizations. Those most directly involved in development of the tool
included teams from The World Health Organization, Global Fund to Fight AIDS, Tuberculosis and Malaria, MEASURE Evaluation and Office of the
Global AIDS Coordinator, PEPFAR, USAID and UNAIDS. This tool directly benefited from feedback on the DQA for Auditing from a number of
participants in workshops and meetings held in Nigeria, Ghana, Senegal, South Africa, Vietnam, Switzerland and the U.S.
Inputs
Considerations - to review the assumptions that underpin the theory of change
Theory of VTEI Programme (June 2010 to Numbers/ Process for the Review of
Consideration during existing
Change October 2014) Value Questions for Inception Meetings documents/data
data collection
DFID original Contribution (million 1. Are the inputs (£ value grant, condition for the
23 use of grant, TA, implementation and supervision
£)
partner selection) relevant and sufficient to carry Compare allocation of resources with
66 Steel Bridge Kits for Malakand out the VTEI Programme activities? actual distributions
12.9
Division
2. Was the programme launched timely i.e. June
20 more bridges in the Malakand 2010 and was needed at that time? Compare timelines for distribution of
3.5 Identify rationale and
region 3. Is/Was the proportion of support w.r.t other resources and inputs with actual time
documented evidence of
Inputs Government of Khyber Pakhtunkhwa
3.5
rehabilitation and reconstruction programmes
responses during inception
followed
(GoKP) - civil works for the bridges significant?
meetings.
Engineering Services TA (Halcrow and 4. Are the programme inputs matches the Review Aid support and Government
Sarhad Rural Support Programme 2.1 expectations i.e. activities, outputs, outcomes and budgerty allocations for reconstruction and
(SRSP) impact? rehabilitation of education and transport
5. Is the programme experienced any delays or infrastructure in Malakand division
40 semi-permanent schools 1 inefficiencies in the allocation and distribution of
resources/inputs?

Activities
Theory of VTEI Programme (June 2010 to Numbers/ Consideration during existing Process for the Review of
Questions for Inception Meetings
Change October 2014) Value data collection documents/data

Semi-permanent schools for girls and

boys were set up in Swat District of KP
Activities - Schools 40
and handed over to the education
(Schools rebuilt and
authorities Identify evidence of effective
completed) 1. Is the VTEI Programme Implementation workplan
programme management and
- SRSP PTCs were established 40 (with detail of activities, responsibilities and Compare progress against workplan
monitoring practices.
timelines)
PTCs members trained 320 developed and followed? Review Programme management and
Identify if activities were
2. Have the programme implementation guidelines monitoring system
demand driven and not donor
and processes been developed?
Activities - Bridges driven.
3. How were the progress against workplan, and Review the rationale for picking the
(Bridges rebuilt and
ensure progress data quality assurance managed? workplan activities to generate programme
completed) Assess if activities are properly
4. How was the activity of implementation process outputs
- Provincial C&W implemented and managed to
Bridges in Malakand Region 86 monitired and supervised?
Department of KP achieve results (outputs)
province (civil works)
- Mott MacDonald /
Halcrow (supervision)

Outputs
Theory of VTEI Programme (June 2010 to Numbers/ Consideration during existing Process for the Review of
Questions for Inception Meetings
Change October 2014) Value data collection documents/data
Improved access and better Assess if progress on outputs
Outputs - Education Review the strength of indicators used to
educational facilities 1. Which indicators have been considered/used to is properly reflected through
measure progress on these outputs
assess the changes in these output? indicators
Employment during construction
2. Do you have baseline value for these output
phase Assess if the indicators are good
indicators? If yes, what is the source of data on Asses if output indicators are
reflection of outputs
this, and are these sources verifiable? properly tracked and monitoring
Cost of transport reduced
Indicators and 3. Have you tracked the progress on these during programme
Outputs - Bridges Assess the quality and reliability of data on
baseline value indicators during the programme life? if yes, what it implementation
Increased vehicle traffic output indicators
shows?
4. Do you think there could be any other Identify if indicator tracking and
Identify if there are any other more
Increased pedestrian traffic indicator(s) (currently not part of result framework- monitoring guided the
appropriate indicators to reflect the
ToC) that reflects/ evident the achievements of the programme - course correction
Output - Schools & Schools and Bridges maintained and progress on these output indicators; and
programme and managing the programme
Bridges protected readily data is available on those
priorities
Outcomes
Theory of VTEI Programme (June 2010 to Numbers/ Consideration during existing Process for the Review of
Questions for Inception Meetings
Change October 2014) Value data collection documents/data
Immediate Identify the relationship of
More responsible and gainfully
Outcomes - outputs with the immediate Review the correlation b/w outputs and
employed citizens of the future
Education outcomes outcomes
1. How strongly, significantly and effectively the
Cheaper access to markets, schools outputs contributed to changes in these outcomes?
Assess if progress on Review the strength of indicators used to
and clinics 2. Which indicators have been considered/used
immediate outcomes is measure progress on these immediate
by the programme to assess the changes in these
properly reflected through outcomes
outcomes?
Cheaper and better availability of food indicators
3. Are there baseline value for these outcome
staples Assess if the indicators are good
Immediate Indicators and indicators recorded? If yes, what is the source of
Asses if outcome indicators are reflection of immediate outcomes
Outcomes - Bridges baseline value data on this, and are these sources verifiable?
properly tracked and monitoring
Increased profitability of local products 4. Have the progress on these indicator been
during programme Assess the quality and reliability of data on
tracked during the programme life? if yes, what it
implementation outcome indicators
shows?
Quicker access to markets, schools 5. Could there be any other indicator(s) (currently
and clinics Identify if indicator tracking and Identify if there are any other more
not part of result framework- ToC) that reflects/
monitoring guided the appropriate indicators to reflect the
evident the achievements of the programme
Accelerated reconstruction and programme - course correction progress on these outcome indicators;
Immediate Outcome -
service delivery improvements in all and managing the programme and readily data is available on those
Overall
sectors priorities

Outcome & Overall Impact

Theory of VTEI Programme (June 2010 to Numbers/ Consideration during existing Process for the Review of
Questions for Inception Meetings
Change October 2014) Value data collection documents/data
Review Theory of Change and Result
1. Same questions apply to this section, with Matrix
Particular benefits to poor and addition of these:
Assess if Theory of Change
vulnerable, including women 2. Do the changes in these programme level Review assumptions and risk matrixs
reflects the timelines for change
outcome and impact indicators are/could be
at outcome and impact level
Outcomes - Socio- observed during the life of the programme? Identify if verifiable primary and/or
economic 3. What are the underlying assumptions and risks secondary data used to track the outcome
Assess if risks and assumptions
development Reduction in poverty for VTEI programme to contribute (or not to and impact level indicators
in reaching to outcome and
Indicators and contribute) to socio-economic development of
impact level results considered
baseline value Malakand division? Review and validate the findings of
and properly managed
4. Is there any formal mechanism through which Independent programme evaluation
Increased pro-poor growth
assumptions, risks and expectations are properly conducted by DFID
Assess the attribution of VTEI
managed and updated?
programme to the intended
5. Are there other programmes aiming socio- Identify if timeline for bringing the change
outcomes and impact; and if
economic development (outcome & impact) of at outcome and impact level in the ToC is
Improved perception of Government ToC effective outlined that
Overall Impact Malakand division, if yes, what is the significance rationale and still valid
ability to provide basic infrastructure of VTEI programme over other programmes?

12.0 Expected deliverable

12.1 Phase – 1 activity:

A clear understanding of the program attributes and dynamics would be established vis-à-vis
situation at hand and ground realities factoring in various constraints and challenges. This phase aims
to assess, test and challenge the assumptions with the theory of change and would include

 Through the synthesis of existing research evidence and reviews of project documentation
 Review of existing literature on socio-economic development related to infrastructure
projects (particularly in fragile states).
 Identifying and determining appropriate indicators to assess progress on socio-economic
development.
 Reviewing the coverage and quality of existing baseline and trend data on the appropriate
indicators of socio-economic development.
  Reviewing the evaluation questions in conjunction with the evaluation users and
stakeholders.
 Identifying and getting consensus on detailed evaluation method that sets out a final set of
questions.
 Designing of data collection tools to meet the evaluation requirements and balanced against
the availability and quality of the existing data
 Reviewing the options for establishing a counterfactual, including comparison sites without
schools or bridges as well as methods that could be used without any comparison groups.
 Identifying a mechanism to identify unexpected outcomes.
 Using the findings from the above to produce an inception report document that sets out the
feasibility of evaluating the programme. The inception report would be able to demonstrate
how the different forms of data will be integrated into a single coherent analysis. A clear cost
and resources’ plan would be included with a timetable for delivery

12.2 Phase 2: (Implementation)

Although the detailed design of phase 2 is not currently possible since this activity is contingent upon
the findings of Phase – 1. As such the team proposes certain steps that it envisages would be
instrumental in providing an insight of the expected activities and approach that it intends to adopt
to accomplish the implementation phase through a suggested approach.

13.0 Proposed Preliminary Tasks

Certain preliminary tasks are envisioned that need to take place before the team starts field
operations. It is subjective at this point in time to attribute them to either phase as some of the
activities mentioned can be undertaken in Phase - 1, but since the nature of activates being
mentioned require some field information as well, it is being suggested to be undertaken at the
initiation stage of implementation phase.

A- Developing a qualitative module maybe after conducting a small survey for bridges' & schools'
target areas to select a "Qualitative Investigation sample" of beneficiaries. (Random stratified
sample of beneficiary sites can be taken).
a. Dividing beneficiaries into “target groups” from within the sample where ever possible
to achieve disaggregation based on gender and ethnicity or any other entity groups to
identify the conflict sensitive delivery of the program.

B- Re-verifying "priority issues" & comparing them with existing "Theory of change" linkages to
check for relevance and identification of knowledge gaps to be filled by surveys

C- Refining indicators and variables of investigation to compliment or replace existing indicators

in "theory of change" matrix. Eliciting community participation in verifying these "Impact &
Process indicators" before survey activity as a participatory method. The participatory
methods are another way of ensuring investigating "Unexpected outcomes" through FGDs

D- Incorporating recalibrated "Impact & process" indicators in thematic questionnaires for FGDs
and various ranking matrices to be designed for perception mapping activities later in field
surveys

The idea of conducting a small preliminary survey allows would allow the team to identify priority
issues to be covered in detail by the qualitative methods that are to follow and to target beneficiaries
based on gender, ethnicity or groups, to ensure all are catered.
 It will also allow the team to address any knowledge gaps as well as identifying categories of
responses be administered through a mix of perception ranking matrices and observable change
approach.

14.0 Suggested evaluation approach:

The evaluation approach to be adopted in the implementation phase will cover a number of tasks as
highlighted in the ToRs. The main aim is to address uncertainties over the most appropriate indicators
and review assumptions that underpin the theory of change

It is proposed that this section of the impact assessment would be undertaken using the mixed-
methodology approach with a heavy tilt towards the “participatory approach” with contribution from
the “quantitative methods” wherever necessary so that both are complimentary to each other. What
is being suggested here is:

Quantitative Data as point of Departure for Qualitative Research, i.e. A quantitative data set serves
as a starting point for framing a study that is primarily qualitative, which is relevant in case of VTEI
ToRs.

14.1 Making use of the “Participatory Impact Assessment”, contextual approach– PIA
technique6, it will be possible to evaluate the sample of beneficiaries without having to form the
control groups to establish a counterfactual. This fact will be established in Phase- 1 deliberations,
but the techniques being proposed can also deliver sound evaluations in the absence of control
groups.

Option of forming comparison groups from within the beneficiaries will still be investigated in detail
and is based on the type and quality of data available and will be majorly affected by the possibility
of outreach to those community based on various challenges pertaining to ground realities such as
security situation, availability of enough numbers to form a group, willingness to participate and
relevance to the area of intervention etc.

Note: Participatory evaluation is usually a module within an overall design rather than an overarching
principle7

14.2 Rationale - In common with qualitative research, participatory research tends to employ more
contextual methods and elicit more qualitative and interpretive information, but brings an important
additional commitment to respect local (emic) knowledge and facilitate local ownership and control
of data generation and analysis (Chambers, 1994, 1997). This aspect of ownership and control in
participatory research is intended to provide space for local people to establish quantitative and
qualitative methods in impact evaluation and measuring results their own analytical framework and
to be in a position to challenge ‘development from above’ (Mukherjee, 1995, 27). Participatory
methods generate both qualitative and quantitative data. ‘Participatory numbers’ can be generated
and used in context, but have also been taken to scale, most notably through participatory surveys
or through aggregation of group-based scoring and ranking activities. Participatory methods can be
quick and efficient, producing data in a timely fashion for evidence-based analysis and action.

6
Participatory Impact Assessment (PIA) is an extension of Participatory Rural Appraisal (PRA) and involves the adaptation of participatory tools
combined with more conventional statistical approaches specifically to measure the impact of humanitarian assistance and development projects on
people’s lives - Participatory Impact Assessment – A guide for practitioners; Cately et al, 2010) – Feinstein Int’l Centre; Tufts University.
7
Stern et, al
14.3 A key requirement is to produce results from a representative sample, which can be
generalised in order to reach conclusions for the population of interest. This implies working in a
larger number of sites than is common for most studies that use participatory methods.

This raises the question as to “who” will participate. In VTEI case those participating will definitely
include beneficiaries but may also include country-based officials and decision makers. There will be
different implications from different patterns of participation. For example the participation of
decision makers may have implications for implementation efficiency and sustainability. In order for
the measurement of qualitative impacts not to become too reductionist it will be sequenced with
qualitative analysis. In this way the evaluation and subsequent policy learning can be enriched by
qualitative analytical studies.

Quantification from the findings of PIA would involve developing and/or applying indicators or
indexes that measure changes in qualitative impacts, including both perception scoring data and
observable changes in behaviour. These indicators will allow for measurement and aggregation of
non-material and often complex multi-dimensional impacts.

To address the difficulty of establishing “attribution”, in the context of VTEI, it is recommended to

field a small survey using “recall methodology” on the variables of interest for identified participants.
There has been some critique as to how much can one rely on recall but all survey questions are
recall, so it is a question of degree. The evaluators will use judgment as to what it is reasonable to
expect a respondent to remember. It is reasonable to expect people to recall major life changing
events that are relevant to quantifying our indicators of interest, when people do recall there may
be telescoping (thinking things were more recent than they were, so it is useful to refer to some
widely known event as a time benchmark for recall questions.

14.4 Community-defined impact indicators of project impact - As far as possible, impact

indicators which are identified by the community or intended project participants will be utilised by
asking a selected member of each target community giving due consideration to gender stratification,
preferably to be assisting the field team to make it more “participatory”. The idea is to empower the
community through the research process and make them a part of it ensuring representation as well
as enrichment of data based on real impacts.

Communities have their own priorities for improving their lives, and their own ways of identifying
impact indicators and measuring change. Oftentimes these priorities and indicators are different
from those identified by external actors. Traditional M&E systems tend to over emphasize ‘our
indicators’ not ‘their indicators’.

14.5 Sampling for Qualitative analysis

The indicators can be collected in more than one way and will be finalized after discussion with the
CRHR team at DFID.

Firstly, a qualitative module can be developed that will add to an existing longitudinal survey
instrument (existing baseline surveys and data from the three Annual Reviews and other baseline
data), that was applied at the initial stage of the project presumably to a relatively large sample of
the targeted population and, later to a comparator population.

If enough similarities are found out between the now selected comparison group and the earlier
targeted beneficiary group formation of a comparison group can be investigated. This can be done
for infrastructural interventions that are yet to take off.
This sample will be the basis of all outreach and investigation in the target areas and will be
formulated using advice of all stakeholders and availability of resources and ease of out-reach
keeping in mind community nuances and sensitivities (e.g. gender exposure etc.)

14.6 Proposed step by step implementation plan and tools to be used.

Contextual methods (Participatory approach) is the main theme of conducting this research as
mentioned above and could be even more feasible if the existing data to prepare a qualitative module
is not feasible or the option of establishing a comparator is found to be difficult due to non-similarities
of sample conditions.

A random stratified sample of sites will be taken to generate and aggregate indicator data on
qualitative impacts. This data can be collected using a mix of individual and group based scoring—
often described as a community score card (CSC). A community score card was designed and
implemented in Jamaica, for example, as part of a community based monitoring and evaluation of
social policy impacts on police-youth relations in a cross section of communities. The score card
included indicators of empowerment, designed to measure the existence of choice, exercise of choice
and impact of choice for young people in their interactions with the police. This tool is found suitable
for gauging perceptions hence will be helpful for the purpose of finding out the perception of the
targeted communities’ perception about Govt service delivery in the aftermath of a disaster.

This community score card can be further enriched using the additional investigative variables
relevant to the “theory of change” map as well as some taken from the UNAID and UNGASS indicators
list for socio-economic development.

It can alternatively be collected through a locally conducted beneficiary or household survey—

sometimes referred to as a citizen report card (CRC). The CRC will be conducted in a focus group
setting with a stratified sample of 6-12 service users. CSC trustworthiness can be increased by
triangulating the CSC data with equivalent data generated by bigger sample-size survey instrument.
8The CSC is described as a ‘mixed-method’ tool because it generates both quantitative and qualitative

data and analysis due to the presence of qualitative module that can be quantified using various
ranking and scoring methods to investigate perceptions.

The quantitative data thus generated will comprise perception scores of specific qualities of service
provision in the project so far, usually scored on a 4 or 5 point scale. These can be complimented by

These scores will then be aggregated from all the focus group discussions held and can be compared
across groups and over time. The key to a successful CSC session, in contrast with a survey module,
is that the scores are not simply elicited as an end in themselves but feed qualitative discussion. The
scoring will be used to prompt a discussion of three questions: (a) Defining the problem/issue; (b)
Diagnosing the problem; and (c) Identifying solutions. Follow up action might involve service users
taking action or engaging with service providers to resolve some of the problems identified during
the CSC session

14.7 Quantifying data from CSC technique

In order to strengthen confidence in the findings and to sequence the group based scoring with
qualitative analysis, findings from the CSC techniques can be merged with a longitudinal survey, if
found feasible to be conducted. The group based activity in the CSC will generate perception data

8
Holland et al, 2007
through “Ranking Matrices” involving investigative variables that can be triangulated with the survey
data.

In other words, scoring is intrinsically useful to the qualitative exercise because the act of being
required to score something that is subjective sharpens the qualitative analysis that follows.
Participants have to justify their (relatively precise) scores. This process shows variance in the opinion
and the facilitator can call upon those people who have scored differently than the majority and can
ask for the reasons why they have done so.

One on One interviews and Key informant interviews with identified stakeholders will also be utilised
as they are an important part of the research. Each sector team (Bridges / Schools) would undertake
a thorough review of available documentation prior to the visit to the area of interest. It aims to
engage with a full range of stakeholders while minimising the transaction costs of their involvement

14.8 Complimentary evaluation techniques to be explored.

Simultaneous to the above detailed methods, a hybrid of Theory based evaluation –contribution
analysis as well as Case study based techniques to validate within group findings can also be utilised
using comparable cases from the given population of beneficiary population where a common set of
causes are present and evidence exists as to their potency. This is necessitated owing to the nature
of the VTEI program where there are several relevant causes that need to be disentangled.
Interventions are just one part of a causal package.

Experiments and Statistical studies are still favoured as complimentary to contextual methods being
proposed, but that would mean presence of sufficient numbers (beneficiaries, households etc.) for
statistical analysis. The possibility of their inclusion and percentage contribution to the overall
evaluation methodology will largely depend on the data furnished after the Phase -1 feedback.
Similarly, critique on the “Theory of change” at this point will be a bit premature, owing to the lack
of availability of data collected earlier that identified gaps and areas of improvements that can be
explored in the context of designing this new impact assessment exercise. A lot of permutations exist
as to the selection and hybridization of techniques available to conduct a QCA (qualitative
comparative analysis), and are subject to debate regarding their relevance for the job in hand.
15.0 Stakeholder wise Implementation Methodology and proposed tools of
evaluation
Phase II: Implemtation phase based on "proposed mixed-method methodologies". Subject to
findings of Phase-1
Type of
Stakeholders Assessment Methodology Intended Benefit
methodology
(1) Review the option of setting up a counter (Qualitative) 1) Increased enrolment /
factual, or else utilise primary data from A- Participatory retention in schools
"Memory Recall methodology" asking Assessments + Key 2) Reduction in the costs of
beneficiary children & teachers to deliberate Informant interviews + getting access to schools
on various Impact variables based on before FGDs 3) Improved access to schools/
/ after scenario. (Quantitative education services for children
(2) Conducting Key Informant Interviews of B- Closed ended 4) Reduction in the time taken
representatives of all beneficiary groups. questionnaire to be to get to schools
(3) Making use of "Community Scorecard" administered in 5) Increased savings from
method can be used for quantification of identified survey areas + children education for other
Students, before/after findings using ranking & scoring Community Score household expenses
Teachers & methods for quantification. Cards and developing
Parents, 4) Conducting Focus Group Discussion Perception ranking
(FGDs) with beneficiary parents on various matirces for perception
overall
enriched variables of investigation. mapping
community 5) Using "Perception scoring" to quantify C- Participatory/ group
members. observable changes during exercises (1-4). based data to be
6) Based on conveneance of outreach, a triangulated with survey
"closed ended" questionnaire based on the data
findings of Steps (1-5) can be administered
to quantify findings using SPSS.
7) Triangulation of steps (1-6) to converge
findings

1) Review and compare the Percentage of 1) Improved literacy rate

budgetary support in school construction to 2) Reduced number of "out of
overall government education sector school children"
development budget 3) Increased/ improved access
2) Review and compare education sector and quality of education
Key performance Indicators (KPIs) before services to the citizens
and after intervention (Qualitative) 4) Budgetary relief - education
(Governement) 3) Investigate options for establishing Control Participatory utilising
sector development budget
Education Vs. treatment - compare progress of DFID Interviews and possibility 5) Support in achievement of
Dept. focal VTEI schools with rest of the Malakand of conducting FGDs Key performance Indicators
persons division and compare the progress efficiency coupled with Perception (KPIs) in education sector
surveys 6) Accelerated speed of
reconstruction and service
delivery in education sectors

1) Construction Support
(Qualitative) 2) Introduction to a model of
Key Informant Interviews with C&W Participatory utilising community based construction
(Government)
department staff to assess their learning and Interviews and possibility model in education
Construction & experience from VTEI programme school 3) Improved accountability
of conducting FGDs
Works - CW construction activities coupled with Perception system
surveys
 1) Conduct perception survey of (Qualitative) Case 1) Improved employment
communities that have experience Studies of sample opportunities during the
reconstruction and provision of education schools Focused construction phase
services interviews at Chamber 2) Cost of transport reduced
2) Review through "case study" of of commerce and 3) Increased vehicle traffic
sample school construction activities to industries 4) Increased pedestrian traffic
Overall calculate number of labor days employed (Quantitative) 5) Cheaper & better avaialbility
General during school infrastructure development Perception Surveys & of food staples
population of activities quantification through 6) Improved access to markets
3) Stability Index measurement - to identify perception 7) Increased profitability of local
the target area communities' perception about economic measurement indices & produce
(Bridges & revival improved because of school & changes in behaviour. 8) Increased evidence in the
Schools) bridges construction, new shops opened, Triangulation for the incidence of poverty reduction
new commercial brands introduced, private two sets of data in the 9) Accelerated speed of
sector investment in education sector (private end reconstruction and service
schools and learning academies) etc. delivery in all sectors
10) Increased level of overall
economic activity

Perception survey of beneficiary (Qualitative) 1) Communities experience the

communities on Key Informant benefit of reconstruction and
1) broader construction efforts interviews / FGDs of have positive perception of the
2) responsiveness and capacity of members of community, broader reconstruction effort
government institutions at all levels to restore politicians, community 2) Enhanced public’s
and improve vital infrastructure elites etc. perception of the
3) Key Informant Interviews with (Quantitative) responsiveness and capacity of
Community leaders, politicians and elites on Perception Surveys & government institutions at all
Political future prospects of Malakand in terms of quantification through levels to restore and improve
Government peace, stability and economic growth measurement indices vital infrastructure
and administering 3) Increased public
Community based confidence in the state in
score cards - CSCc reducing tendency to support
based on investigative insurgency and encourage them
variables to support peace building and
conflict reduction process in
region

The above proposed methodology is a mixed-method approach primarily making use of “contextual
methods” and relying on participatory approach wherever applicable to qualitatively assess the
impacts of the VTEI program. Quantification of various qualitative methods adopted will be
attempted through utilising the Community based scorecard approach and designing ranking
matrices containing indicators for investigation.

Ample use of point contact methodologies like Focal point interviews, Focus group discussions will
be carried out to substantiate the findings from the perception matrices as well as data from any
longitudinal “quantitative survey” if the team finds it feasible to conduct it in the targeted areas.

It is expected that in order to quantify data on gender and individual entities or groups based on
ethnicity that the project benefited, target groups will be identified from within the randomly
stratified sample of target areas established earlier to capture various dimensions affecting those
entities.

To sum it all up, all findings will be triangulated to identify any “unexpected outcomes” that are
expected to arise, especially with the participation of identified community members as part of the
research team.