Methodology for the EU-Ukraine Association Agenda monitoring

Expert: Dr. Volodymyr Pyzhov

Kyiv, 2011

Table of Contents
Introduction.............................................................................................................................................4 Developing the AA monitoring...............................................................................................................5 1.1 The purpose of monitoring...........................................................................................................5 1.2 The subject of monitoring.............................................................................................................6 1.3 How does a monitoring subject look like?....................................................................................6 1.4 What methodological approaches can be used?.........................................................................7 1.4.1 Objectively Verifiable Indicators (OVI) based approach.......................................................7 1.4.2 Expert assessment................................................................................................................8 1.4.3 Proposal.................................................................................................................................8 1.5 AA monitoring criteria...................................................................................................................8 OVI template for AA monitoring.............................................................................................................9 Table 1: OVI template for AA monitoring.............................................................................................10 1.Identifying objectively verifiable indicators.......................................................................................12 3.1 Why do we need indicators? .....................................................................................................12 3.2 Indicators should be SMART ....................................................................................................13 3.3 Objectively verifiable What does it mean?..............................................................................14 3.4 Recommendations ....................................................................................................................14 Algorithm of defining indicators .......................................................................................................15 How to define verification sources for an indicator?............................................................................17 Assumptions.........................................................................................................................................18 Monitoring reporting.............................................................................................................................20

Annexes Annex 1 Template for OVIs for the implementation of AA priorities (Policy Dialogue)

Introduction One of the tasks of the project Enhanced civic engagement with reforms in Ukraine in the framework of European Neighbourhood Policy implemented by the European Partnership for Democracy and the Civil Network OPORA is to develop a methodology for monitoring of the fulfilling of the Ukrainian Governments public obligations regarding the implementation of the reforms envisaged in the EU-Ukraine Association Agenda. According to the project specification, the methodology can be used by nongovernmental organizations for AA monitoring in the future as well. Some of the methodology aspects where previously used by the expert in recommendations for development of actions to implement the Association Agenda under the EU-funded project Support to the PCA Implementation".

Developing the AA monitoring 1.1 The purpose of monitoring Monitoring according to EU Project Cycle Management terminology is The systematic and continuous collecting, analysis and using of information for the purpose of management and decision-making. The key element of this definition is its purpose-oriented function i.e. monitoring is a component of the management system. Civil society is not a party that is directly involved in the management of AA implementation and decision making. However, NGOs can and have to externally influence the process of management and decision making by main actors - the European Commission and the Ukrainian Government. So, civic monitoring is an external independent monitoring. The overall purpose of the monitoring is formulated in response to the following questions:

How can we know whether or not what was planned during AA adoption is actually happening or has happened? How do we verify the success in the implementation of AA priorities?

The question can be applied to each of the AA parties: Ukrainian Government, European Union, civil society. At the same time project specifications imply the development of a methodology for monitoring of the commitments of the Ukrainian Governments commitments in the context of the European Neighbourhood Policy regarding the implementation of the reforms in the area of democracy, rule of law and human rights, and which are publically available. This is a more specific purpose compared to the overall one. We should not forget that AA is a joint document, and each of the parties committed their responsibilities for AA implementation. Thus, while monitoring the commitments of the Ukrainian Government, we should also keep in mind the EU actions in this direction, for instance its contribution to the
5

implementation of priorities through relevant technical assistance projects or sector budget support programmes. 1.2 The subject of monitoring At the beginning, we should define the subject and object of monitoring i.e. what is actually to be monitored. What exactly should be observed and what basic information about the subject do we have? Generally, the subject is AA implementation, in particular each of its priorities. That means we have to consider both each AA priority and respective sub-chapters and areas of cooperation. Publications or internet media are not a subject of monitoring. Analysis of media is only one of the instruments of the AA monitoring 1.3 How does a monitoring subject look like? Potentially, there are several packages of information about the subject of monitoring.
1.

One-two sentences in the text of the Association Agenda (a priority)

Having a look at the priority we do not find any information about: What should be done to implement the priority? Who should implement it? When should it be done? What resources are needed? Of course, the governmental institutions responsible for the implementation of the specific AA priorities should establish a certain action plan for the AA priorities implementation and develop these actions in detail. Within the project Support to the PCA Implementation, the specific guidelines to develop actions for AA implementation were proposed, and respective trainings for the Cabinet of Ministers of Ukraine, ministries and other institutions were delivered. Specifically, it was recommended to develop relevant quantitative and qualitative indicators of AA implementation. However, for the time being it is not known whether such approach is being used by the Ukrainian Government.

2. AA text and Action Plan of the Cabinet for AA implementation In this case we can get more detailed information. 3. AA text and Action Plan of the Cabinet and Action Plans of concrete bodies responsible for AA priorities implementation There is a detailed description of the object of monitoring. Competences of central bodies of the governmental executive bodies of Ukraine as well as of the EU-Ukraine sub-committees responsible for AA implementation can be found in the Annex 1. However this information is not valid for the years 2011-2012. The general conclusion is the following:

The less detailed the Action Plans for AA implementation, the more difficult it will be to conduct monitoring and the more efforts by monitoring experts will be needed.

What should be done if there are no detailed Action Plans for AA implementation? In this case, monitoring experts would need to develop their own description of the monitoring subject i.e. to develop a pseudo-plan of measures to implement the specific AA priority. 1.4 What methodological approaches can be used? 1.4.1 Objectively Verifiable Indicators (OVI) based approach This approach is a widely used and approbated instrument for policy analysis, planning, management, monitoring and reporting in EU programmes and projects. It is an integral component of the general approach of Logical Framework Analysis (LFA). More detailed description of this approach can be found in the EU Project Cycle Management Guidelines (see Annex 2). A key advantage of this approach is its objectiveness and argumentation. This approach can be used by the monitoring experts of different qualification, resulting in the similar observations and conclusions. The developed indicators, if agreed with all interested stakeholders, do not leave much room for different, including biased interpretation of achieved results.

Some weakness of this approach is that it is not sufficiently effective if the policies, programmes or projects are not properly developed, unspecific, nonlogical or badly structured. As a rule, this approach is quite formalised and provides little space for experts analytical fantasy. 1.4.2 Expert assessment This approach is based on professional experience of individual experts or of a group of experts. The quality monitoring will fully depend on the quality of the expertise. Of course, each expert bases his/her conclusions on certain indicators. However, these indicators are not always objectively verifiable, and subjective opinion may influence the assessment. Moreover, every expert has his/her own style of describing observations and conclusions as well as writing reports, which can complicate the drafting of the final AA monitoring reports. An advantage of this method is its effectiveness in case the lack of publically available information about the actions/ measures which are being implemented by the Ukrainian Government to meet AA priorities. In addition, this method allows using information from informal sources. 1.4.3 Proposal It is proposed to use the OVI based approach. However, identification of indicators should be done taking into consideration individual professional experiences of the AA monitoring experts. 1.5 AA monitoring criteria For AA civic monitoring we need to define monitoring criteria i.e. what exactly should be assessed within a priority for implementation. The EU and other donors use the following monitoring and evaluation criteria: relevance efficiency effectiveness impact sustainability

Maybe, not all of them are useful for AA monitoring. Previous discussions defined the following EU evaluation criteria as relevant for AA monitoring: 1. Progress assessment of priority implementation The activities done in the reporting period should be evaluated here, as well as how these activities contributed to the achievement of AA priority. This criterion is similar to the criterion of efficiency. Forecast assessment It should be assessed what is remaining to be done to achieve/ implement AA priority, and what is the likelihood of this achievement. This criterion is similar to the criterion of effectiveness.
2.

Impact assessment This is a general assessment of the impact of AA priority fulfilment on the development of a certain policy area of the policy dialogue and/or reforms that take place in the respective sector of AA. This criterion coincides with the criterion of impact.
3.

All expert observations and conclusions must be focused on these three criteria. OVI template for AA monitoring If the OVIs based approach is accepted, we need to propose an instrument for identifying indicators i.e. a template (indicators matrix) for each of the AA priorities. The example of a template for AA priority OVIs is presented in the Table 1 below. It is similar to a well-known Logframe Matrix, which is widely used for developing, monitoring and managing EU-funded programmes and projects. In particular, the methodology for Logframe preparation and utilisation is described in the Project Cycle Management Guidelines1 (see also Annex 2). In comparison with the normal Logframe Matrix for a project or action, the OVI table does not have the logical levels (overall objective, purpose, expected results and planned activities).
1

http://ec.europa.eu/europeaid/multimedia/publications/documents/tools/europeaid_adm_pcm_guid elines_2004

A table includes information about: No. and title of AA Chapter No. and title of AA sub-chapter No. and name of AA dialogue and cooperation area This information is needed to see the place and role of each of the priorities in the overall system and to facilitate the consolidation in a final report of the monitoring conclusions. Unfortunately, numeration of priorities changed or may change in the lists of AA priorities adopted for different time periods. To allow monitoring and tracking of the AA as a whole and avoid possible misunderstanding, it is recommended to stick to the numeration used in the original text of AA. The table itself consists of five columns: 1. No. of AA priority 2. Name of AA priority 3. Objectively Verifiable Indicator of achievement 4. Sources of Verification 5. Assumption The number and name of the AA priority can be taken form the AA text. Indicators, Sources of Verification and Assumptions are identified before the monitoring starts. Recommendations on how to fill in columns 3, 4 and 5 are given in the next chapters. An OVI overall template for all AA priorities (Chapter 2 Policy Dialogue) is given in Annex 3. It only deals with the priorities adopted for 2011-2012. Other priorities, which are not agreed for the list 2011-2012, are also included in the table with respective references, to have a complete picture of the AA implementation. Table 1: OVI template for AA monitoring No. and title of AA e.g. 2. Chapter: No. and title of sub-chapter: No. and name of dialogue and Policy Dialogue

e.g. 2.1 Democracy, rule of law, human rights and fundamental freedoms e.g. (i) Strengthen the stability, independence and effectiveness of institutions guaranteeing
10

cooperation area: democracy and the rule of law No. of AA Name of AA Objectively Sources of priority priority Verifiable Indicator Verification of achievement

Assumption

11

1. Identifying objectively verifiable indicators 3.1 Why do we need indicators? Fulfilment of AA priorities or progress in their implementation should be measurable (estimated). The measurement is to be done through indicators. Objectively verifiable indicators are the basis for development of a proper AA monitoring methodology. They are formulated as an answer to a question: How would we know whether or not what has been planned during AA adoption is actually happening or happened? How can we assess activity results? Indicators establish and measure specific characteristics of the achievement as follows: Quantity How many? Quality How successful? Time When? Additional characteristics: Place Where? Target group For whom? For example, the planned result of a project/action is A new line of bottles production is put into operation. Even if we know for sure that the line was launched, to properly assess the achievement of this result we have to answer the following questions: What is the production capacity? quantity What is the line productivity compared to already existed lines? quantity What is the material (glass or plastic)? quality Shape, colour? quality Does it correspond to healthcare standards? quality

When was the line launched? time Where are the bottles produced? place Purpose of the product (for babies, pharmaceutics, alcohol etc)? target group

The information gathered through the indicators measuring characteristics of the achievements should provide answers to such questions. 3.2 Indicators should be SMART The Logframe approach requires the fulfilment of several conditions, which can be retained using the abbreviation SMART as a memory hook: S Specific Indicators should refer to one specific AA priority. M Measurable Indicators should be measured in quantity and quality. For example, the aim of the project/activity is to achieve progress in cooperation within the EU Seventh Framework Programme for research. One of the proposed indicators is the number of joint research projects. However, the indicator itself does not measure the progress in cooperation. Therefore it should be put in other words, for example: the number of joint research projects in 2011 increased by 20% compared to the year 2008. A Available That means that information required to measure indicators can be collected and cost to get this information is reasonable. R Relevant Indicators must correspond to the needs of information recipients. Project Terms of Reference define the following target groups of monitoring: 1. Authorities
13

2. Civil society 3. Media 4. Population Different target groups as well as recipients at different levels within one target group require different information. For example, ministries and other institutions need information about the progress in the implementation of the AA specific priorities in the areas they are responsible for, while the Cabinet of Ministers is interested in general aggregated information about the implementation of all AA priorities. T Time-bound Indicators should be time-bound which means that we have to know, when the AA priorities can be implemented. 3.3 Objectively verifiable What does it mean? The meaning Objectively Verifiable is that information collected should be the same if collected by different people. In other words, assessment of indicators has not to be dependent on the subjective opinion or bias of any one person. The same is applicable to different institutions which may be involved in monitoring and evaluation. The number of indicators Indicators should be independent and correspond to concrete AA priorities. Usually, one AA priority requires more than one indicator. For example, an indicator providing information on quantitative characteristics should be supplemented with another one providing information on quantity. At the same time, we should not choose too many indicators. The key approach is to collect as little information as needed by monitoring experts (authorities, other target groups) to see whether the AA priority is being/was implemented or not. 3.4 Recommendations

14

Summing up, we can give the following recommendations for identifying objectively verifiable indicators:

Identify indicators before the start of the monitoring; Check if indicators are SMART; Do not define too many indicators;

When defining indicators, we should as well think about the sources needed to verify them; If we choose to use a comparative indicator, we should establish a proper data base during the action or as soon as it is adopted/implemented.

Objectively verifiable indicators defined for the specific AA priorities are to be put into column 3 in OVI table (see Table 1). Algorithm of defining indicators An algorithm of defining indicators can be described with the following scheme:

15

Defining OVIs

Indicator 1

Indicator n

OVI correction/clarification Is indicator specific: corresponds to only one AA priority?

NO YES

Is indicator measurable: quantatively and qualitatively?

NO

YES

Is indicator available?

NO

YES

Is indicator relevant to societys information needs?

NO

YES

Is the indicator time-bound?

NO

YES

Any known verification sources?

NO YES

Indicators defined

16

How to define verification sources for an indicator? In the fourth column of the OVIs table for AA priorities implementation (see Table 1) we have to provide verification sources for each of the indicators. Verification sources should be defined once the indicators are formulated. This step is crucial, as it helps us to understand if there is a real possibility to measure the identified indicators, given a number of constrains in the given context (time, money and efforts). Verification sources should define: What are the information sources? Information sources can be:

Official statistics Decrees and other documents issued by the ministries and other structures reports web-pages certificates and other technical documents legislative and normative acts publications etc How can the information be collected?

Information needed to measure the indicators can be collected by:

analyzing existed documents concrete researches monitoring reports surveys interviews etc Who should collect the information?

17

Within this project, monitoring experts are responsible for collecting information. However, if there is a need to conduct e.g. a survey or different kind of long-term analysis, additional project resources may be required. When do we need to collect information (how often)? It is necessary to decide what is a frequency of monitoring. On the one hand, the timing depends on the project specification and the available resources. On the other, the timeline should be agreed with the monitoring team. The proposed reporting frequency is once per month. Usually, the higher level of monitoring targets (the specific AA priority, sub-area of cooperation, area of the policy dialogue, AA as a whole) the more difficult, and time and resources consuming the verification process will be. Assumptions Assumptions are external factors, which can potentially have an impact on (or even determine) the success of AA priority implementation, and which, however, are beyond the direct control of the governmental body and staff responsible for certain elements of the implementation process. The assumptions answer the question: What external factors can influence implementation of AA priority? Of course, identification of assumptions is the responsibility of the governmental bodies in charge of AA priority implementation at the operational level. At the same time results of some efforts made at the operational level may be affected by actions or lack of actions at the higher policy level. This can be considered as external factors for the AA priority implementing agencies and may create a basis for identification of assumptions. Identification of assumptions is useful for AA monitoring experts, because it allows analysing and taking into account external environment of the specific AA priority.

18

Connection between AA priorities and appropriate activities/measures The responsible governmental institutions have to implement some relevant activities/measures to achieve the specific AA priority, they are responsible for. In this context, the link between AA priorities, related activities and assumptions can be formulated as follows: IF the activities/measures are implemented, and IF the assumptions are come true, THEN the AA priority is achieved.

Identification of assumptions Assumptions may be of a different kind and nature i.e.: political institutional technical economic legal social financial In any case they should be external factors which might have an influence on priority implementation. We should also keep in mind, that an assumption is a positively formulated risk. For instance, the measure should result in publishing a new textbook for schools, which meets EU approaches and standards of studying foreign language. Obviously, there can be a financial risk if the Ministry of Education, Science, Youth and Sport did not receive funds from the Ministry of Finances in a due time for publishing the textbook. In this case, an assumption can be identified as Funds for publishing a textbook are available.

19

As it was mentioned earlier, realization of assumptions or non-realization of risks can lead to the fulfilment of a priority. This means that for good quality monitoring of AA achievements, assumptions should be also assessed together with respective OVIs. Assumptions make up the fifth column of the OVI template for AA monitoring (see Table 1). Monitoring reporting Monitoring for a certain period should result in a respective report. Reports can be of different format and meet different objectives. The following reporting levels can be defined.
1.

Interim Expert reports on specific AA priority for a certain period These reports are based on a detailed assessment of indicators (OVI) during the period. The report may have both narrative and tabular parts, depending on agreed format. They should focus on agreed monitoring criteria (chapter 1.5). All experts should use a unified reporting format and submit it periodically (once per month). The properly addressed and targeted recommendations for the improvement of the AA implementation are mandatory.

2. Final reports for interested parties These reports are for EU institutions and the Ukrainian government, as well as for interested NGOs. They may include conclusions for concrete priorities and areas or AA chapters (expert articles). These reports should always include properly addressed and targeted recommendations for the improvement of the AA implementation.
3.

Final reports for the general public Should be concise, but at the same time informative, properly supported by factual materials, as well as media and reader-friendly.

With regards to the reporting format for expert reports, it is recommended to use an OVI-based approach including assessment of the current state of indicators.

20

Reporting format Report data Report ref. number Date of report Reporting period Name of monitoring expert Start date of monitoring of AA priority Ref. number and date of previous report Planned date of the next report Report content 1. 2. Title and No. of the priority Priority implementation plan Is there an adopted plan for priority implementation? Has the responsible body been identified? What was planned to be done in order to meet the obligations within the priority? - Up to a half page. Priority at the beginning of monitoring Please describe what has been done earlier for priority implementation Up to a half page. Activities (changes)* What happened within the monitoring period? Were the indicators achieved? If yes, which of them? (description of dd.mm.yyyy from dd.mm.yyyy to dd.mm.yyyy dd.mm.yyyy dd.mm.yyyy dd.mm.yyyy

3.

4.

21

the main changes that happened within the reporting period adopted documents, initiatives etc) up to 1 page. 5. Results* What was achieved within the reporting period? (based on the quantitative and qualitative indicators) Quality of those results? up to 1 page. Forecast assessment* What else should be done in order to implement the priority? Is implementation possible? up to a half page. Impact evaluation (positive and negative) A general evaluation of the impact of the priority implementation on the policy area and reforms in this area up to 1 page. Expert recommendations Please indicate what the government should do in order to fullfill the priority (Who should do this? What should be done?, When should it be done?) up to 1 page? - what should be done to achieve results? - what should be done to improve their quality? - what should be done to enhance positive impact/reduce negative impact? Short commentary (for a webpage)
22

6.

7.

8.

9.

On the overall AA priority implementation for the reporting period up to 10 sentences. * - based on the developed OVIs (individual tables).

23