The Asian Development Bank’s Engagement with Middle-Income Countries, Linked Document 1

EVALUATION APPROACH AND METHODOLOGY

Key Evaluation Questions

Q1. What are the key development challenges of client middle-income countries (MICs) and which challenges has the
Asian Development Bank (ADB) supported?
(i) Given that, although they are diverse, many MICs share one or more development challenges, is it possible to
cluster MICs according to their characteristics or attributes?a
(ii) What are the underlying factors that contribute to the development challenges? For example, to what extent are
country systems developed (e.g., public financial management, safeguards, following good international
practices)?

Q2. What support has ADB offered to client MICs and to what extent has this support been responsive to client needs?
(i) What type of support has ADB offered to MICs to enable them to address their development challenges and
generate positive externalities?
(a) What has been the mix of investment and technical assistance support and the knowledge solutions that are
embedded in these forms of support?
(b) What has been the mix of sovereign and nonsovereign (including private sector) operations?
(c) What regional cooperation and integration initiatives have complemented country-centered approaches
through country partnership strategies?
(d) How has ADB supported MICs to reduce their vulnerability to external shocks?
(e) How do ADB products and services compare with those offered by other multilateral development banks?
(ii) What is MICs’ perspective on ADB support?
(a) Do they prefer certain types of ADB products and services and why?
(b) How ADB’s on-the-ground presence, skill-sets, systems and procedures, and comparative advantages (real or
perceived) have influenced the type of support they have actively sought from ADB?
(c) Are there any noticeable differences between ADB and other multilateral development banks, in terms of a
responsive and beneficial engagement?

Q3. What results have been achieved through ADB support to client MICs?
(i) What benefits do MIC stakeholders consider they have received from ADB support?
(ii) What efforts have MICs made to evolve from being recipients to becoming collaborators?
(iii) How does ADB compare with alternative financing sources?
(a) What has been ADB’s share in the MIC’s external debt profile?
(b) What are the relative benefits of ADB financing vis-à-vis alternative financing sources?

Q4. Is there a rationale for ADB to reconsider or modify its graduation policy?
(i) To what extent has ADB applied the graduation policy to date to upper middle-income countries (UMICs) that
have crossed the income threshold for regular support?
(ii) What are the implications for ADB if UMICs eligible for graduation were to stop receiving regular support?
(iii) What is the graduation policy and practice in other multilateral development banks?
(iv) Have countries “reverse graduated” and sought assistance from ADB or another multilateral development bank
after a long gap?
(v) To what extent have multilaterals continued to provide support to countries after they have achieved high-income
country status?

__________________________________
a
The evaluation attempts to distinguish noticeable differences in the mix of development challenges between, for instance,
(i) small island developing states and other MICs, (ii) lower and upper MICs, (iii) resource-rich countries and others, and (iv)
landlocked countries and others.

Source: ADB Independent Evaluation Department.

2 Linked Document 1

1. A two-tiered approach was adopted. The evaluation focused on 19 of 37 middle-income

countries (MICs) for which country-specific issues were investigated. These 19 MICs accounted for more
than 77% of all sovereign plus sovereign operations (including cofinancing) during the study period
2006–2015, which included more than 80% of all sovereign lending operations, 68% of nonsovereign
operations, and 40% of technical assistance operations from 2006 to 2015. Excluding cofinancing, the
share of these 19 MICs for sovereign and nonsovereign lending operations exceeded 87% and 81%
respectively.

2. Issues related to ADB engagement for these 19 MICs were examined through desk studies and
consultations with relevant ADB staff. Case studies, including consultations with in-country
stakeholders, were conducted for 5 of these 19 MICs. Given that, for the first time, ADB is going to
formulate a corporate strategy when a significant number of MICs are in the upper middle-income
category, and that many of these upper middle-income countries (UMICs) have pressed ADB to rethink
its manner of engagement, the case study countries were mostly UMICs. The five case study countries
were (i) the People’s Republic of China, a UMIC that has received a large and steady stream of ADB
support, and has set up a fund for regional cooperation and poverty reduction;
(ii) Kazakhstan, a large resource-rich UMIC with widely varying levels of ADB support from year to year;
(iii) Malaysia, a UMIC with a low level of ADB support since at least the mid–2000s; (iv) Thailand, a
UMIC with mostly nonsovereign support; and (v) Indonesia, a lower middle-income country (LMIC)
which has preferred policy-based loans in recent years, although ADB did provide a results-based loan
in late 2015.

3. The remaining 18 MICs were covered mostly through a literature review that spanned MICs
with various features and characteristics (such as landlocked and small island countries). For these
MICs, general development related issues were identified through a literature review. Further details are
provided in Table 1.

Table 1: Evaluation Approach

Middle-Income Countries
19 of 37 MICs for detailed desk studies and consultation
with ADB staff:
(i) 7 UMICs (Azerbaijan, the PRC, Kazakhstan, Malaysia,
Maldives, Thailand, and Turkmenistan)
(ii) 12 LMICs (Bangladesh, India, Indonesia, Mongolia,
Pakistan, Papua New Guinea, Philippines, Sri Lanka,
Tonga, Uzbekistan, Vanuatu, and Viet Nam)
Notes:
(i) Mongolia and Uzbekistan are landlocked.
(ii) Kazakhstan is referred to as landlocked in the CPS
2012–2016.
(iii) Azerbaijan and Turkmenistan have ports on the
Caspian Sea. Baku (Azerbaijan) is a significant port;
Turkmenbashy (Turkmenistan) is now expanding—
neither is considered landlocked.
5 of 19 MICs for detailed case studies
(i) 4 UMICs (the PRC, Kazakhstan, Malaysia, Thailand)
(ii) 1 LMIC (Indonesia)
Note:
Kazakhstan is referred to as landlocked in the CPS (2012–
2016), as the Caspian Sea is an inland water body which
provides limited transportation opportunities.
Approach
Desk studies and ADB staff consultations focused on
understanding countries’ development challenges,
their access to non-ADB sources of finance for
development and investment, sovereign guaranteed
investment and TA support from ADB, nonsovereign
(including private sector) operations from ADB, social
and other development indicators. Conducted
extensive literature review.
The case studies examined government perspectives
on continuing regular ADB support, their views on
ADB’s graduation policy, the value addition expected
from ADB versus what was actually delivered, the
engagement with ADB for subregional interventions
versus country support, problems, and prospects for
ADB in developing relationships with subnational
governments, issues, and options available for support
to private and public sector entities, types of products
and services offered by ADB versus those offered by
other development partners (notably other multilateral
development banks), competition from other sources
 Evaluation Approach and Methodology 3

Middle-Income Countries Approach

of finance etc.

The case studies were informed by desk studies and

consultation with ADB staff.
18 Other MICs: Review of non-country-specific reports, strategies, and
(i) 5 UMICs and Pacific island states (Fiji, Marshall other literature to understand development
Islands, Nauru, Palau, and Tuvalu) constraints, the implicit assumption being that ADB
(ii) 5 LMICs and Pacific island states (Kiribati, Federated will remain relevant to these countries’ economic
States of Micronesia, Samoa, Solomon Islands, and development and growth through 2030.
Timor-Leste)
(iii) 1 other UMIC (Georgia) and 7 other LMICs (Armenia,
Bhutan, Cambodia, Georgia, , Kyrgyz Republic, Lao
People’s Democratic Republic, Myanmar, and
Tajikistan)

Notes:
(i) Armenia, Bhutan, Kyrgyz Republic, Lao People’s
Democratic Republic, Tajikistan are landlocked.
(ii) 4 of 5 LMIC Pacific island states are FCAS (Kiribati,
Federated States of Micronesia, Solomon Islands, and
Timor-Leste) and 3 of 5 UMIC Pacific island states are
FCAS (Marshall Islands, Nauru, and Tuvalu).
(iii) 1 of the 8 other LMICs is FCAS (Myanmar).
ADB = Asian Development Bank, CPS = country partnership strategy, FCAS = fragile and conflict-affected situations, LMIC =
lower middle-income country, MIC = middle-income country, PRC = People’s Republic of China, UMIC = upper middle-income
country.
Source: Independent Evaluation Department.

4. Data and information were obtained through the following:

(i) A literature review of ADB documents that included working papers from the Economic
Research and Regional Cooperation Department and other knowledge departments,
country partnerships strategies, ADB’s sector and thematic policies and strategies,
relevant corporate and thematic evaluation studies, relevant country program
evaluations and validations, relevant economics literature on development issues in
fragile and conflict-affected situations and landlocked MICs, and countries in transition.
(ii) A literature review of ADB and non-ADB sources formed the basis for understanding
key external factors that influence MIC development priorities. This included a review of
the global agreements on the post-2015 global development agenda and climate
change, along with a review of commentaries and insights prepared by selected think-
tanks and others to gain a comprehensive understanding of how such agreements
could influence development priorities of MICs.
(iii) Online databases and data gathered from the Bank of International Settlements,
International Monetary Fund, the World Bank, and other sources, were used to analyze
options available to country governments for raising finance from diverse sources.
(iv) A review of ADB policies regarding products and modalities, and selected ongoing and
completed interventions using those modalities. This was used to assess the need to
align the products and services with evolving country priorities and needs.
(v) Interviews with selected high-level and senior ADB management and staff to gain a
better understanding of ADB’s perspective on partnerships, value addition,
relationships, products and services, and areas of operations that can be strengthened
to help ADB achieve its objectives.
(vi) Interviews with ADB personnel in ADB headquarters and resident missions in the
People's Republic of China, Indonesia, Kazakhstan, Malaysia, and Thailand, and
interviews with government ministries and other in-country stakeholders, and other
development partners to gather inputs for case studies.