Validation Report

Reference Number: PCV: PAK 2010-69 Project Number: 32058 Loan Number: 1891 December 2010

Pakistan: Road Sector Development Program

Independent Evaluation Department

ABBREVIATIONS
ADB ADF IED MOC NHA OCR PCR RRP RSDP TA Asian Development Bank Asian Development Fund Independent Evaluation Department Ministry of Communications National Highway Authority ordinary capital resources program completion report report and recommendation of the President Road Sector Development Program technical assistance

NOTE In this report, $ refers to US dollars. Key Words pakistan, adb, asian development bank, highways, roads, lessons, independent evaluation department, performance evaluation, sector development program Director H. Hettige, Independent Evaluation Division 2, Independent Evaluation Department (IED) R. Bolt, Advisor, Office of the Director General, Southeast Asia Department R. Lumain, Senior Evaluation Officer, Independent Evaluation Division 2, IED C. Roldan, Assistant Operations Evaluation Analyst, Independent Evaluation Division 2, IED

Team leader Team members

In preparing any evaluation report, or by making any designation of or reference to a particular territory or geographic area in this document, the Independent Evaluation Department does not intend to make any judgments as to the legal or other status of any territory or area.

PROGRAM COMPLETION REPORT VALIDATION

A. Basic Program Data PCR Validation Date: Program Number: 32058 1891 (program loan component only) Loan Number: Road Sector Development Program Name: Total Program Cost Program ($ million): Islamic Republic of Pakistan Loan/Grant ($ million) Country: Transport (Roads Transport) Total Cofinancing ($ million): Sector: and Information and Communications Technology ADF: 0.0 Borrower ($ million): ADB Financing ($ million): OCR: 50.0 Beneficiaries ($ million): None Others ($ million): Cofinanciers: 19 December 2001 Approval Date: Effectiveness Date: 1 February 2002 Signing Date: Closing Date: Project Officers: Name: Location: H. Carlsson ADB headquarters H. Masood ADB headquarters T. Nakazaki ADB headquarters S. Muramoto ADB headquarters O. Norojono ADB headquarters T. F. Jones III, Consultant Validator: Director: R. Bolt, Advisor, Office of the Quality Control/ Director General, Southeast Peer Reviewers: Asia Department1 R. Lumain, Senior Evaluation Officer, IED2 December 2010 Approved Actual 83.2 50.00 0.00 50.76 0.00

0.00 0.00 0.00 0.00 0.00 0.00 2 May 2002 27 Jun 2002 30 Jun 2007 30 Jun 2007 From To 2002 2003 2003 2004 2004 2005 2005 2007 2007 2008 H. Hettige, IED2

ADB = Asian Development Bank, ADF = Asian Development Fund, OCR = ordinary capital resources, PCR = program completion report.

B. Program Description (summarized from the report and recommendation of the President)2 (i) Rationale. Policy reforms were critically needed to increase the efficiency of Pakistans road sector at the national and provincial levels. Reforms initiated at the national level provided the framework for overall sector development involving (a) increased capital investments to develop an efficient transport network; (b) an efficient internal network to support regional cooperation; (c) increased resources and improved institutional mechanisms to address inadequate road rehabilitation and maintenance; (d) strengthened road network institutions in professional and management skills, and assistance in establishing an effective policy and regulatory enabling environment; (e) initiatives to improve traffic management and road safety; and (f) support for measures to control excessive axle loads and vehicle weights. In this context, the Asian Development Bank was to finance a portion of the adjustment costs associated with the national road sector policy reforms. Impact. The Road Sector Development Program (RSDP) was to improve management of the provincial and rural access roads network. Policy and institutional changes in the National Highway Authority (NHA) would result in the effective delivery of services in the road sector. This was then expected to help reduce rural poverty and contribute to national economic growth. Objectives or expected outcomes. The expected outcomes were (a) preservation of road assets, (b) governance and institutional development, (c) operational efficiency improvements, (d) technical

(ii)

(iii)

1 2

Formerly with the Independent Evaluation Department at the time this validation was carried out. ADB. 2001. Report and Recommendation of the President to the Board of Directors: Proposed Loans to the Islamic Republic of Pakistan for the Road Sector Development Program. Manila (Loans 1891-PAK,1892-PAK, and 1893PAK(SF), for $200 million, approved on 19 December).

2
efficiency improvements, (e) road safety, (f) private sector participation including involvement of knowledge-based industries, and (g) poverty reduction and community development. Specific program targets include (a) improved network condition through better institutional management and maintenance practices, (b) improved local government capacity to become involved in the management of road networks within their jurisdiction, (c) reduced burden on public sector resources by alternative financing mechanisms, (d) improved access and lower transport costs, and (e) reduced incidence and severity of road accidents. (iv) Components and/or outputs. The RSDP comprised two components to address structural inefficiencies in the national and provincial road network: national road sector policy reform, and a provincial sector development project. On the former, the RSDP was to support the continuation of the reform process with the following elements: (a) rationalizing staffing; (b) developing the NHA Code, financial manual, and standard operating procedures; (c) enhancing staff accountability in financial management; (d) prioritizing investment based on economic principles; (e) self-financing maintenance needs through toll receipts; (f) preserving assets through an axle-load control program involving installation of weigh stations and enhancing enforcement procedures; (g) ensuring transparency in procurement processes; and (h) adopting a comprehensive approach to road safety, including public awareness and stakeholder consultations.

C. Evaluation of Design and Implementation (program completion report assessment and validation) (i) Relevance of design and formulation. Conception of the RSDP was generally appropriate. The Asian Development Bank (ADB) assistance comprised three loans using a sector development program modality. One at $50 million equivalent (Loan 1891-PAK) was for development of the policy reform program, while the second and third (footnote 2) were for the improvement of provincial roads in Pakistans Sindh Province. This validation covers the program completion report (PCR) on the national policy component of the RSDP, which covered the implementation of the national road sector reform program.3 The PCR adequately assesses the relevance of program design and formulation. The policy reform component of the RSDP (and its timing) is very relevant. Its stated aim and purpose are consistent with the governments development strategy and the country assistance program of ADB for Pakistan from 2001 to 2003. This ADB-financed policy loan was mainly intended to support continuing efforts of the NHA to adapt business processes to those of contemporary road agencies. 4 Its program interventions were built around a comprehensive set of five policy themes involving (a) transport policy, (b) road sector resource management, (c) road asset preservation, (d) institutional efficiency improvement, and (e) road safety. These themes, which were consistent with the policy themes of the national policy reform program, were highly relevant and remain so. While the design and formulation of the RSDP was suitable, there was room for improvement. In particular, the PCR frankly comments that the policy loan should have been implemented with accompanying technical assistance (TA), particularly for the transport policy and road safety themes (section C[iv]). (ii) Program outputs. The accomplishments of the RSDP policy loan relative to the five themes above are summarized in Table 1 (see also PCR, paras. 1021). The final status of associated policy actions (for tranche releases) relative to the national policy reform program policy matrix are described in Appendix 4 of the PCR. Of the twenty-four policy actions for tranche release, twenty (83.3% of total) were either complied or generally complied with. All policy actions for transport policy (1), road sector resource management (8), and road asset preservation (5) themes were carried out. For the institutional efficiency improvement theme, three were complied with while a fourth is judged to be generally complied with. Implementation of the road safety theme was an issue, with only two of
ADB. 2008. Completion Report: Road Sector Development Program in the Islamic Republic of Pakistan. Manila. The closing date for Loans 1892 and 1893 was extended from 30 June 2007 to 30 June 2009. An ongoing national policy reform program largely reflected the commitment of the NHA to change and adapt. It is aimed at building the capacity of the NHA and transforming it into an efficient service provider to road users (i.e., to ensure safe and comfortable travel on the national highways).

3 4

the six tranche release conditions either fully complied with or generally complied with.5 Two were partly complied with, while the balance was not complied with because of lack of resources. Table 1: Assessing Program Outputs
Policy Theme Transport Policy Road Sector Resource Management Assessment Achieved. Transport policy reform involved the development of a long-term national transport policy framework that will combine strategic vision and ensure sustainability. According to the PCR, policy actions were carried out and objectives achieved. Achieved. The policys objectives are being achieved and actions have been implemented. The NHA has prepared a long-term financial business plan for 20072020 for approval by the government. To help ensure that national highway maintenance is sustainable, most of the national highways have been brought under a tolling regime, which generates revenues for this purpose but this is still inadequate. The toll rate is under review with a view to help increase revenues to the needed level. Generally achieved. Some progress has been achieved with the enactment of a national highway safety ordinance. The National Highway and Motorway Police was created under this ordinance. Of 112 planned vehicle weighbridges, 42 have been constructed and are operational with noticeable effect on the motorways. Further actions regarding axle weight control are being considered for implementation in the future. Generally achieved. These were focused on the NHA, which operates as a complete business entity that executes two areas of operations: (a) development of the national highway network, and (b) maintenance and upgrading of the developed network. The NHA outsources functions such as (a) maintenance, (b) construction, (c) toll collection, and (d) design and feasibility studies. The PCR relates that policy actions have been carried out and objectives achieved, while the institutional reform and corporate strengthening program are under way. The NHA initiated rightsizing and reduced staff in 2003 and later increased it in accordance with increasing work demands.

Road Asset Preservation

Institutional Efficiency Improvement

Road Safety

Partly achieved. The National Highway Safety Council was established in 2001 as part of the road safety theme, but it has become a dormant organization without a permanent secretariat because funds were not available. A road safety cell was established and road design safety audits were introduced as required by external funding agencies. Efforts by the National Highway and Motorway Police were partly responsible for the reduced number of road accidents. Inadequate funding retarded some policy actions from being implemented, as did a lack of commitment at higher government levels and a lack of coordination among institutions involved. NHA = National Highway Authority, PCR = program completion report. Source: Independent Evaluation Department staff based on Asian Development Bank. 2008. Completion Report: Road Sector Development Program in the Islamic Republic of Pakistan. Manila.

(iii)

Program cost, disbursements, borrower contribution, and conformance to schedule (as relevant to project performance). As this was a program loan, it was disbursed in two tranches based on fulfillment of tranche conditions set out in the national policy reform program policy matrix (section C[ii]). As the ADB loan portion was fixed at loan approval, the total amount was disbursed with two adjustments. A total of $486,172 was debited to ADB as the front-end fee, while the total loan value (denominated in Y6,079.3 million or $50 million equivalent) increased to $50,763,300 because of depreciation of the US dollar versus the yen. Since the loan was to be used to pay for approved imports, none of the loan went directly to financing the reform program. The total appraised program cost was estimated at $83.2 million (PCR, Table 1). The PCR does not report on the actual costs incurred by the government, as much of this would have been in kind. The loan was approved on 19 December 2001 and closed on 30 June 2007. Although disbursement of the first and second tranches was delayed, the loan was closed as originally scheduled. The first

Three policy actions (on the road safety theme) required for the release of the first tranche were reportedly not fully met at program completion. On the other hand, an ADB communication with the government on the status of the national reform program matrix (dated August 2002) do indicate these were complied with at the time. To some extent, the release criteria seem to have been subject to meeting release schedules rather than tranche release requirements.

4
tranche in the amount of $34,065,602 equivalent was disbursed on 28 June 2002, the day after the loan was made effective. Loan release was delayed by 4 months compared with the original program period.6 The second and final tranche was disbursed in two parts. The first part in the amount of $15,000,000 equivalent was disbursed on 14 November 2003, while the second part in the amount of $1,210,986 equivalent was disbursed on 28 November 2003, thus completing full disbursement under the program. There was a 17-month delay in disbursing the second tranche because of delayed government action on release conditions. (iv) Implementation arrangements, conditions and covenants, related technical assistance, and procurement and consultant performance. As identified at appraisal, the program was implemented through the executing agency, the Ministry of Finance (MOF), with the support of the Ministry of Communications (MOC) and the NHA. In the conditions and covenants section, section II.G of the PCR states that the delays in disbursing the two tranches were due to the governments delay in completing requirements for loan effectiveness (footnote 6) and completing the second tranche policy actions. Conditions for tranche release are given in Appendix 5 of the PCR. A review of the covenants reveals that only 1 of the 18 program-specific covenants was partly complied with. The partly complied with covenant involved the provision of funds in a timely manner for carrying out the program. Funds were not always readily available, particularly for implementing the road safety actions. The rest of the covenants were either complied with (8 covenants) or generally complied with (9 covenants). From this validations point of view, the assessment of generally complied with seems to mean that while attempts were made to be in compliance, the actual compliance was less than the intended spirit of the covenant. For example, coordination among agencies was often poor, target dates were often missed, actions met at one point in time were not continued, and some actions were not initiated during the program period. There was no TA attached to the loan and no consultants were required. In hindsight, suitable TA should have been included, particularly for the transport policy and road safety themes (section C[i]). For example, transport policy TA was later sponsored upon government request, and this contributed to the national transport policy document (PCR, para. 59).7 Procurement of the various goods and materials purchased by the government followed normal government procurement procedures. (v) Performance of the borrower and executing agency. In general, the program loan was implemented as conceived. ADBs program performance ratings for development objectives and implementation progress from June 2002 to June 2007 were satisfactory. The performance of the borrower and the MOF was satisfactory in terms of using the loan amount for the purchase of supplies and materials. The three agencies involved (MOF, MOC, and NHA) performed satisfactorily in establishing and implementing policy reforms. The NHA remains committed to continuing improvements in institutional efficiency and reforms in the management of the national road network (PCR, para. 63). As explained above, only one of the 18 loan covenants was partly complied with. Eight of the loan covenants were complied with, while the balance was judged generally complied with (section C[iv]). To a certain extent, this observation implies that some of the agencies responsible for implementing the program were not fully committed to instituting reforms over the 5-year implementation period. Several factors also detract from borrower performance. First, implementation of some of the road safety components had not had the expected impact and sustainability. This was partly caused by lack of funds. In general, the PCR rightfully points out that the borrower did not fully analyze the resources
6

The report and recommendation of the President stated that the period of the policy loan was from January 2002 to June 2007. The PCR indicated that the original schedule of first tranche release was February 2002 and the second tranche release on June 2002. Compliance with the requirements for loan effectiveness, as a condition for the first tranche release, was delayed. ADB. 2004. Technical Assistance to the Islamic Republic of Pakistan for Transport Policy Support. Manila (TA 4400-PAK, for $290,000, approved on 30 September).

5
required for implementation of some components (PCR, para. 58). These were started but could not be continued because of lack of funds. Their continuity is now dependent on establishing a stable source of funds to finance the components. Second, some basic problems in the sector related to poor interministerial coordination and lack of needed legislation prevailed during implementation. From a short-term perspective, this validation agrees with the PCRs overall performance rating of satisfactory. However, in the longer term, the government must ensure that the policy reforms are continued for sustainable operations. As cited above, continued implementation of some of the reforms have already suffered from lack of funds (section D[iv]). (vi) Performance of the Asian Development Bank. The PCR assessment of ADB performance is contradictory. In para. 33 of the PCR, it states The Program was administered by different officers in ADB and monitoring could have been better. Although the tranche conditions were satisfied, ADBs review could have been more periodic to avoid partial compliance actions by the EA. Para. 67 follows these up with The program administration and review by ADB should emphasize close monitoring of the executing agency and ensure continuity of staff in the administration team. The PCR goes on to emphasize that ADB provided assistance in a timely manner, which helped to facilitate actions taken by the government agencies in complying with tranche conditions required for the release of loan tranches (PCR, para. 33). This leaves questions as to whether ADB provided adequate assistance during program implementation, especially in terms of field supervision. While ADB headquarters support appears to have been sufficient to ensure that project management issues were addressed, it is unclear from the PCR whether field-level dialogue was adequate. The PCR assessed ADB performance satisfactory. This validation considers ADB headquarters-based administrative support to have been adequate before and after tranche release. However, supervision missions fielded were seemingly not sufficient. During the 5-year implementation period, ADB undertook only three review missions led by two different staff members. 8 This validation considers this level of supervision inadequate for a program. The program loan mission leader had no part in program implementation. Overall, taking into consideration the programs timely completion and successes in a challenging reform process, this validation agrees with the PCR rating on ADB performance of satisfactory. D. Evaluation of Performance (program completion report assessment and validation) (i) Relevance. The PCR rated the program very relevant, as the aims and purposes were consistent with the governments development strategy and ADBs country assistance plan (PCR, para. 34). All components were appropriate, and program inputs were sufficient to generate the planned outputs. This validation agrees that the program was satisfactorily conceived and is reasonably comprehensive (section C[i]). In addition, to be effective, program-related achievements need to be sustainable. In this sense, the program remains important and very relevant to the countrys road sector. The NHA is said to be committed to continuing improvements in institutional efficiency and reforms in the management of the national road network. The PCR rightfully points out that this policy program should have been implemented with TA, particularly for the transport policy and road safety components (section C[iv]). In particular, implementation of the road safety components did not meet expectations, while the transport policy theme reportedly benefited from a subsequent stand-alone transport policy TA project. This observed design weakness detracts from a full relevance rating at completion. Taking into consideration the overall program quality-at-entry at both approval and completion, this validation rates the program relevant. Effectiveness in achieving outcome. The PCR states that the lack of resources brings the continued implementation of all aspects of the policy reform into doubt. Therefore, while the initial efforts to set up new policies might have been largely successful, since they are largely on paper or the responsibility of newly established agencies, any possible positive effect might be lost as if nothing
These comprised a consultation mission (February 2002), review mission (November 2002), and a special program administration mission (November 2003). The November 2002 mission was a multiproject mission covering three other loans and a TA project.

(ii)

6
ever happened without proper follow-up. The validation agrees with the PCR rating of effective inasmuch as the desired effect is still in progress. At completion, the result for each of the major themes is summarized in Table 2. Table 2: Assessing Effectiveness in Achieving Program Outcomes
Policy Theme Transport Policy Rating Effective Assessment ADB intervention was successful and effective in developing a draft transport policy document for submission to the cabinet after interministerial discussion. While effectiveness is relatively positive up to program completion, the proof will come after the cabinet and further approvals and adequate funding. However, its approval is lingering for a long time due to evaluation required by many ministries and departments.a The road asset management system has been developed and helps ensure that the road sector agencies carry out asset management and preservation effectively. However, budget allocations need to be rationalized through revenue sources, so that an appropriate balance between revenue sources and appropriate allocation for the roads sector is maintained. The NHA, as one of the major road agencies, can now supplement government revenues through off-budget initiatives such as tolls on the national highway system. Thus, the NHA is more capable of undertaking both highway upgrading and maintenance. An axle load control program was introduced to protect and increase the service life of road assets. While legislation was enacted to restrict maximum axle loads on the national highway network, there is no such legislation on the provincial highways. The National Highway and Motorway Police (NH&MP) were responsible for identifying the operational parameters for enforcement. The policy reform is being monitored through the development of awareness programs and measuring axle load levels at regular intervals on the network. An enforcement procedure and institutional framework were established. Associated penalties for violation of load levels were set and are being levied. The NHA has moved toward becoming a commercial agency with a view to economy and efficiency in operations by operating on a fee-for-service basis. More recently, the NHA changed its organizational structure and is involved in institutional capacity building under b follow-on ADB-financed programs. Achieved outcomes here will probably be sustained considering follow-on assistance.

Road Asset Management

Effective

Road Asset Preservation

Effective

Institutional Efficiency Improvement

Effective

Road Safety

Concerning the road safety theme, various measures were introduced to help reduce road accidents in Pakistan. The NHA and NH&MP are improving their knowledge, capability, and personnel training. Community awareness and stakeholder participation is being pursued, as well as professional-level accident recognition and implementation of remedial measures such as accident blackspotc identification and rectification. The PCR states several positive efforts in this section, but in section C(v) shortcomings are identified such as the implementation of some of the road safety policy components, which have not had the expected impact and sustainability. This was due to lack of funds, poor interministerial coordination, and lack of needed legislation. ADB = Asian Development Bank, NHA = National Highway Authority, PCR = program completion report. a A comprehensive National Transport Policy for Pakistan was completed on 30 May 2008. The final document was submitted to the executing agency for further processing and approval by the government. A departmental review was conducted until early 2009. The executing agency is still coordinating with the Planning Commission and the Cabinet for their final review and approval. Previously, several attempts were made by the government to articulate a comprehensive transport policy that encompasses all subsectors. The draft policy document was presented to the interministerial National Trade Corridor (NTC) Task Force. The NTC core group will take up the policy development process further and to formalize it as the government document. Implementation of the policy will be coordinated by the Planning Commission. b ADB. 2005. Report and Recommendation of the President to the Board of Directors: Proposed Multitranche Financing Facility and Loan to the Islamic Republic of Pakistan: National Highway Development Sector Investment Program. Manila (Loan 2210-PAK[SF], for $3 million, approved on 13 December and Loan 2231-PAK [Tranche 1], for $180 million, approved on 5 February 2006); and ADB. 2007. Report and Recommendation of the President to the Board of Directors: Proposed Multitranche Financing Facility and Technical Assistance Grant to the Islamic Republic of Pakistan: National Trade Corridor Highway investment Program. Manila (Loan 2400-PAK, for $545 million, approved on 17 December and Loan 2401-PAK[SF], for $10 million, approved on 17 December). c In terms of road safety management, this refers to a place where road accidents have historically been concentrated. Source: Independent Evaluation Department staff based on ADB. 2008. Completion Report: Road Sector Development Program in the Islamic Republic of Pakistan. Manila.

Less Effective

7
(iii) Efficiency in achieving outcome and outputs. In terms of implementation efficiency, the policy loan cost was in line with approved loan amount while tranche releases were delayed.9 The first tranche was delayed by only 4 months while the second tranche disbursements were late by 17 months because of delayed government action on release conditions. Despite these delays, the loan was closed as originally scheduled. The PCR rated the program efficient. Based on the PCR findings at completion, this validation agrees with that assessment. Table 3 briefly summarizes PCR assessments on program efficiency under the reform program. Table 3: Assessing Efficiency in Achieving Program Outcomes
Policy Theme Transport Policy Rating Efficient Assessment The transport policy has gone through several drafts, the first to qualify for tranche release, and subsequent revisions have been made to reflect the views of the various agencies involved. MOC has finalized the national transport policy, but final approval has not been achieved. The National Highway Council has approved a medium-term road resource management plan. This initiative seems to be working efficiently as evidenced by the condition of the national highway and motorway network, which has improved steadily since 2001. Actions have been undertaken to improve road asset preservation. Weigh stations have been installed at strategic locations throughout the national highway system and on motorways and were integrated with the fine collection system. According to the PCR, enforcing road safety and axle load control on national highways is being extended regularly in phases. The PCR does not suggest exactly how far these phases have progressed. To achieve institutional efficiency improvement, the NHA has undertaken rightsizing exercises but with additions and subtractions, there has been little effect. NHA standard operating procedures have been revised and updated. The NHA has acquired financial management expertise, which should improve the efficiency of in-house financial management capability.

Road Asset Management

Highly Efficient

Road Asset Preservation

Efficient

Institutional Efficiency Improvement

Less Efficient

The lack of achievements under the road safety theme is discussed in section C of the PCR. Lack of funds is cited as the main reason for inaction. This initiative was reportedly being pursued under the Northwest Frontier Province Road Development Sector Project, but was dropped by the government citing difficult conditions.a MOC = Ministry of Communications, NHA = National Highway Authority, PCR = program completion report. a Asian Development Bank (ADB). 2004. Report and Recommendation of the President to the Board of Directors: Proposed Loan to the Islamic Republic of Pakistan for the Northwest Frontier Province Road Development Sector and Subregional Connectivity Project. Manila (Loan 2103-PAK, for $296.2 million, approved on 18 November and Loan 2104-PAK[SF], for $5 million, approved on 28 November). Source: Independent Evaluation Department staff based on Asian Development Bank. 2008. Completion Report: Road Sector Development Program in the Islamic Republic of Pakistan. Manila.

Road Safety

Inefficient

(iv)

Preliminary assessment of sustainability. While the PCR does not specifically rate sustainability in the sustainability section (PCR, paras. 4950), the impacts section (PCR, para. 54) states that the policy program is expected to result in a sustainable impact over the long term (considering the effectiveness and efficiency of the outputs and outcomes). In broad terms, the PCR explains sustainability this way (para. 50): The government is trying its best to sustain the actions initiated under this Program. Because of the lack of resources and TA, some actions were not implemented and some objectives could not be achieved. Additional external assistance would be necessary for complete implementation of all policy actions and establishment of alternative funding source for their continuity. This would improve the Programs performance, sustainability, and intended benefits. The government committed itself to supporting this program before loan approval, but has apparently failed in meeting its obligations (section C[v]). Aside from external financial assistance, there seems to be little case for an expectation of high sustainability. The NHA and other concerned agencies will likely sustain actions where they see a clear benefit and will generally eschew actions deemed less beneficial or politically unacceptable.

Program loans do not fit the typical measure of efficiency as indicated by the economic internal rate of return of an infrastructure project.

Accordingly, this validation sees mixed results in the long-term sustainability of the program actions, i.e., depending on the interest of the donors in pursuing specific policy actions. In the likelihood that program sustainability is to become supply-driven or donor-dependent, this validation rates the sustainability less likely to be sustainable. (v) Impact (both intended and unintended). The PCR states many positive impacts such as an increased maintenance allocation now covering about 50% of the development budget, up from 38% (PCR, para. 51 and Appendix 4). The installation of weigh stations at strategic locations on national highways has been partly accomplished. However, as stated in the PCR, the main issue outstanding is full enforcement of legal loading, which is not fully operational. Maintenance planning is better executed and a maintenance manual developed. On the negative side, the government did not create the necessary organizations for the efficient treatment of road accident data. Regarding road infrastructure, stakeholder consultations are being carried out. The PCR concludes that the policy programs socioeconomic impact can be perceived as moderate. This validation agrees with this rating.

E. Overall Assessment, Lessons, and Recommendations (validation of program completion report assessment) (i) Overall assessment. The government and ADB recognized for some time that a comprehensive program of policy development and reform was necessary within the transport sector, particularly within the highways and roads subsectors. Understanding the existing policies or lack thereof takes considerable time and interaction between ADB staff and concerned government officials. This sector development program was well conceived and at the beginning was readily agreed by the government. At program completion, the objectives of the policies were achieved with most agreed policy actions being carried out, except for road safety. Accordingly, the PCR rated the policy loan satisfactory. However, such a program is only as effective as the government wishes it to be. The PCR continually refers to policy objectives as having been generally achieved. The conditional generally is not defined in the PCR but may imply something less than full achievement (section C[iv]).10 For the program to be fully effective, the PCR states that the government should ensure that these initiatives are continued in the long term to ensure sustainable operations. Likewise, continued donor engagement in pursuing program actions will be critical for proper follow-up. Based on program information at completion, the overall progress made under difficult conditions, and the level of donor interest, the validation rates the program at the low end of successful.11 Lessons. There are several important lessons cited in the PCR. Considering their importance, they are highlighted below. (a) In the implementation of an interministerial policy, the capacity of the executing agency to coordinate with all the involved ministries is crucial. The implementation structure should reflect the organizational arrangement existing within the government. One of the basic problems in the sector is the lack of coordination, collaboration, and cooperation among the government agencies involved. Since this is an important requirement for success, the reasons for this failure must be understood and ways to ensure coordination developed. (b) Support for the implementation of the policy program could have been achieved with TA, particularly for transport policy and road safety. ADB is discussing ways to support various components. Recommendations. If the national transport policy document is endorsed by the government, it would require additional assistance in the development of implementation strategy and development of subsector specific guidelines and regulations. ADBs support in the endorsement process and implementation would go further in achieving the goals set out in this intervention. The PCR also made a number of sound recommendations, and these are presented and/or summarized below.

(ii)

(iii)

10

For example, at program completion, the national transport policy formulated by MOC with assistance from ADB is being submitted for cabinet approval at some future unspecified date. 11 The program is assessed to be relevant, effective, efficient, and less likely to be sustainable. Its equivalent overall weighted average of 1.8 falls within the range of 2.7<S1.6 for scoring successful projects/programs.

9
(a) (b) (c) (d) (e) Three first tranche conditions require further actions to ensure full compliance. The government should accelerate efforts to establish the accident reporting center, driver instructor/training school, and accident evaluation laboratory (PCR, para. 60). There should be a set of mechanisms associated with covenants to guide and ensure their compliance. There is a need to coordinate, through relevant government organizations, implementation of the road safety program on the entire road network. To improve the programs performance and help make it sustainable, additional assistance is needed in the form of loans, grants, and TA. For such a complex and interactive initiative to be undertaken comprehensively, it is essential that the executing agency keep involved staff engaged through the duration of the program. Similarly, there should be continuity of ADB staff involvement both in review frequency and staff.

F. Monitoring and Evaluation Design, Implementation, and Utilization (program completion report assessment and validation) Monitoring was left to a consultation mission in 2002, a review mission in 2002, and a special program administration mission in 2003. To maintain an active dialogue, this input on the part of ADB was inadequate. The two missions in 2002 focused on tranche approval and had one mission leader, while the one in 2003 was led by another ADB staff. No missions followed, as the two tranches had been released upon general compliance with the covenanted reforms. G. Other (e.g., safeguards, including governance and anticorruption; fiduciary aspects; government assessment of the program, as applicable) (program completion report assessment and validation) Specific safeguard measures were not needed as part of the program. Better coordination was required at the highest levels of government. Corruption was not reported to be an issue. The government did not provide the necessary operation and maintenance funds for the many activities required to sustain reform. Poor governance must be concluded. H. Ratings Relevance: Effectiveness in Achieving Outcome: Efficiency in Achieving Outcome and Outputs: Preliminary Assessment of Sustainability: PCR Very Relevant Effective Efficient Not Rated IED Review Relevant Effective Efficient Less Likely Reason for Disagreement/Comments A weakness in program design relates to use of attached TA projects to support the transport policy and road safety themes (sections C[iv] and D[i]). At program completion, initial efforts to set up the new policies appear successful (although largely on paper). While tranche disbursements were delayed, the loan was financially closed as originally scheduled. Given funding constraints, sustainability of policy actions will be mixed (i.e., actions will be where government see a clear benefit and will generally eschew actions deemed less beneficial or politically unacceptable). In the absence of external assistance, there seems to be little case for optimistic expectations on sustainability (section D[iv]). PCR rating at completion confirmed based on program achievements amidst a challenging reform process. To sustain this, longer term government performance must ensure the policy reforms are continued. The reform process would have also benefited further from closer field supervision by ADB staff (section C[vi]). The policy loan is rated at the low end of successful.

Borrower and Executing Agency: Performance of ADB: Impact: Overall Assessment: Quality of PCR:

Satisfactory

Satisfactory

Satisfactory Moderate Satisfactory

Satisfactory Moderate Successful Satisfactory

10
I. Comments on PCR Quality The PCR appears thorough and comprehensive. It generally conformed to IED guidelines. To a certain extent, this validation appreciates the PCRs candid approach in pointing out some of the programs shortcomings during implementation. In particular, this validation also takes note of the good lessons identified by the PCR and some of the good recommendations made. However, the lack of easily understood criteria for meeting reform requirements as covenanted caused some confusion in the PCR, as general compliance rather than specific compliance was acceptable. Where assessment of the program and the PCR proved difficult when general compliance was not clear, some questions emerged on program effectiveness. J. Recommendation for Independent Evaluation Department Follow-Up IED may want to perform a detailed evaluation of the sector development program given the many difficulties the policy loan faced and the difficulty that sustainability seems to be facing. K. Data Sources for Validation The sources for this validation report included the PCR; report and recommendation of the President; minutes of the Management Review Meeting, Staff Review Committee, and discussions of ADBs Board of Directors; as well as back-to-office reports of supervisory missions.

REGIONAL DEPARTMENTS RESPONSE TO THE PROGRAM COMPLETION REPORT VALIDATION REPORT

On 14 December 2009, Director, IED2, Independent Evaluation Department, received the following response from the Transport and Communications Division, Central and West Asia Department. We have no comment.