This article was downloaded by: [Arif Setya]

On: 27 January 2015, At: 06:30

Publisher: Taylor & Francis
Informa Ltd Registered in England and Wales Registered Number: 1072954
Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

Environmental Sciences
Publication details, including instructions for authors and
subscription information:
http://www.tandfonline.com/loi/nens19

General Policy Questions Need

Specific Policy Effectiveness
Indicators: An Example for
Emission Reduction Targets
a

Tinus Pulles & Toon van Harmelen

TNO Environment, Energy and Process Innovation ,

Apeldoorn, The Netherlands
Published online: 02 Feb 2010.

To cite this article: Tinus Pulles & Toon van Harmelen (2004) General Policy Questions
Need Specific Policy Effectiveness Indicators: An Example for Emission Reduction Targets,
Environmental Sciences, 1:4, 351-367, DOI: 10.1080/15693430412331314845
To link to this article: http://dx.doi.org/10.1080/15693430412331314845

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all the information
(the Content) contained in the publications on our platform. However, Taylor
& Francis, our agents, and our licensors make no representations or warranties
whatsoever as to the accuracy, completeness, or suitability for any purpose
of the Content. Any opinions and views expressed in this publication are the
opinions and views of the authors, and are not the views of or endorsed by
Taylor & Francis. The accuracy of the Content should not be relied upon and
should be independently verified with primary sources of information. Taylor and
Francis shall not be liable for any losses, actions, claims, proceedings, demands,
costs, expenses, damages, and other liabilities whatsoever or howsoever caused
arising directly or indirectly in connection with, in relation to or arising out of the
use of the Content.
This article may be used for research, teaching, and private study purposes.
Any substantial or systematic reproduction, redistribution, reselling, loan, sublicensing, systematic supply, or distribution in any form to anyone is expressly

Downloaded by [Arif Setya] at 06:30 27 January 2015

forbidden. Terms & Conditions of access and use can be found at http://
www.tandfonline.com/page/terms-and-conditions

Environmental Sciences
2003/2004, Vol. 1, No. 4, pp. 351367

RESEARCH ARTICLE

General Policy Questions Need Specific Policy

Effectiveness Indicators: An Example for Emission
Reduction Targets
Tinus Pulles and Toon van Harmelen

Downloaded by [Arif Setya] at 06:30 27 January 2015

TNO Environment, Energy and Process Innovation, Apeldoorn, The Netherlands

ABSTRACT
This paper describes the importance of different policy perspectives and contexts on the
numerical values of policy effectiveness indicators, developed to inform the policy makers and
the public on the progress of environmental policies towards the targets set. We concentrate on
indicators for progress towards emission reduction targets for greenhouse gases (Kyoto) and
air pollutants (EU National Emission Ceilings Directive) within the European Union. It is
shown that both the policy perspective (monitoring, planning and strategy) and the context
(individual EU Member State or the European Union as a whole) have a strong influence on the
relative score of the Member States. Scientific and technical policy advisors should be aware of
this when developing indicators to answer relatively simple policy questions. The choices they
make on perspective and context have a strong influence on the answers they produce.
Keywords: indicators; International targets, emission reduction

1. INTRODUCTION
1.1. Information for Environmental Policy
Interactions and processes in the environment are very complicated and,
hence, environmental information is in most cases rather complex. Moreover,
many of the interactions are not fully understood or still subject to scientific
debate, resulting in conflicting and mutually inconsistent data interpretations.
Science in many cases responds to this by adding more and more data and
information to the existing body of knowledge.
Address correspondence to: Tinus Pulles, TNO Environment, Energy and Process Innovation,
PO Box 342, 7300 AH Apeldoorn, The Netherlands. E-mail: M.P.J.Pulles@mep.tno.nl
10.1080/15693430412331314845$16.00 # Taylor & Francis Ltd.

Downloaded by [Arif Setya] at 06:30 27 January 2015

352

TINUS PULLES & TOON VAN HARMELEN

Environmental decision makers, however, need quick and simple overviews

that inform them on both the causes of environmental problems, the state of
the environment and likely consequences of the decisions they are preparing
and taking. To this end, scientific and technical policy advisors design socalled indicators to answer policy questions. This approach is often taken in
the framework of international conventions like the United Nations Framework Convention on Climate Change (UNFCCC) where indicators are used to
make data in different regions comparable. Such indicators necessarily
simplify the complex environmental information. The indicator developers
will apply specific simplifications and aggregations, related to the exact policy
questions to be answered. Choices on simplifications and aggregations might
influence the developed indicators and hence the answers the indicators
produce to specific questions. Moreover, different indicator developers might
use different simplifications and aggregations in answering the same policy
question. The consequences of the implicit choices in the indicator development are not frequently communicated to the indicator users and the
limitations caused by the simplifications are not always seen and understood
by the users.
In this study, we investigate the consequences of different simplification
and aggregation algorithms to convert data on progress towards an environmental target to indicators used in the policy process. As an example,
we develop different indicators to answer the same policy question: How
far are individual European Union Member States and the European Union
as a whole on their road towards meeting the emission reduction targets of
Kyoto (greenhouse gases) and Gothenburg (long range trans-boundary air
pollution)?.
1.2. The Concept of Indicators
As indicated above, an indicator is developed to simplify a complex state
or development in the real world in such a way that a specific policy makers
question is answered. This is reflected in the various definitions listed in
Table 1. The definitions as given in the table all clearly recognize that the
indicator is not the real world itself. It is an approximation, produced for a
specific purpose.
In environmental policy, many indicators are purely meant to support
the reporting of progress in the state of the environment, either in State
of the Environment Reports like those of the European Environment
Agency (e.g. EEA, 2003b) or in reports in response to obligation of in-

POLICY EFFECTIVENESS INDICATORS

353

Table 1. Some Definitions of the Concept Indicator.

Source

Downloaded by [Arif Setya] at 06:30 27 January 2015

1. The Oxford Dictionary

of English

2. The Free Dictionaryy

3. European
Environment Agencyz

4. A. Joyce Furfero (2000)

Definition
1. A thing that indicates a state or level.
2. A gauge or meter of a specified kind.
3. A flashing light on a vehicle to show
that it is about to change lanes or turn.
4. (Brit.) An information board or screen
in a railway station, airport, etc.
5. (Chemistry) A compound which changes
colour at a specific pH value or in the presence
of a particular substance, and can be used
to monitor a chemical change.
Indicator a number or ratio (a value on a
scale of measurement) derived from a series
of observed facts; can reveal relative changes
as a function of time.
A parameter or a value derived from parameters
that describe the state of the environment
and its impact on human beings, ecosystems
and materials, the pressures on the environment,
the driving forces and the responses steering
that system. An indicator has gone through
a selection and/or aggregation process to enable
it to steer action.
An indicator is a variable or group of variables
that provide policy makers with information.
This information has two different types
of value. Ex ante, before a policy measure
is implemented, an indicator has predictive
value. Ex post, after a policy measure
has been implemented, an indicator has
effectiveness value.

Sources:
http://www.askoxford.com/concise_oed/indicator?view uk
y
http://www.thefreedictionary.com/indicator
z
http://glossary.eea.eu.int/EEAGlossary/E/environmental_indicator

http://www.drfurfero.com/books/231book/ch13a.html

ternational conventions and protocols. An example of this is the national

total emissions of greenhouse gases, expressed as CO2 equivalents,
annually reported by parties to the UNFCCC (UNFCCC, 2003). Other
examples are emissions of acidifying pollutants, total mass of certain
waste streams, per capita waste production and many other indicators used

354

TINUS PULLES & TOON VAN HARMELEN

Downloaded by [Arif Setya] at 06:30 27 January 2015

in the EEAs report Europes Environment: The Third Assessment (EEA,

2003b).
This type of indicator is relatively unambiguous. Although they are sometimes still under discussion (for instance: the values of global warming
potential to be used in calculating CO2 equivalents (Ramaswamy, 2001)),
calculation is rather straightforward. Such indicators are the ones covered by
definitions 1, 2 and 3 in Table 1.
1.3. Policy Effectiveness Indicators
Definition 4 in Table 1 adds a specific dimension to the indicator: the issue of
effectiveness of policy decisions, either ex ante while preparing decisions
or ex post when monitoring the results of a decision. The concept of effectiveness assumes some kind of reference or comparison to other situations or
possibilities. In many cases, different alternatives could be chosen to develop
specific indicators to answer the policy effectiveness question. Obviously this
might lead to different answers and both indicator developers and indicator
users should be aware of this.
As an example, we will derive in this paper different answers to the same
policy question: How far are individual European Union Member States and
the European Union as a whole on their road towards meeting emission
reduction targets? in two policy fields:
 Both the European Union (EU) and its Member States are parties to
UNFCCC and have ratified the Conventions first protocol on reducing the
emission of greenhouse gases, the so-called Kyoto Protocol. (The UNFCCC
and its Kyoto Protocol were negotiated when the EU had 15 members.
Since June 2004, the EU has 25 Member States. The original 15 EU
Member States took part in the EU burden sharing agreement. That is why
in this study we use data for these 15 EU members only.) So, both the EU
and individual Member States have committed themselves to implement
policies, ensuring that they will meet these Kyoto targets for the first
commitment period 20082012.
 Both the EU and all of its present and future Member States are parties to
the convention on Long Range Transboundary Air Pollutants (LRTAP) and
as a consequence, the EU has developed legislation to ensure reduction of air
pollutants according to the Gothenburg protocol by implementing national
emission reduction targets in the National Emissions Ceiling Directive
(EC, 2001) to be reached by 2010.

355

POLICY EFFECTIVENESS INDICATORS

An indicator developed to answer this policy question, could have an

ex ante and an ex post perspective: it is looking back at the recent past
and it looks forward towards the commitment period. In addition it could
have a strategic perspective, aimed at adjusting existing policies when
needed. These perspectives coincide with the different stages of the policy
process:

Downloaded by [Arif Setya] at 06:30 27 January 2015

 the monitoring perspective: what has been done?

 the planning perspective: what is still to do? and
 the policy strategy perspective: are we doing well enough to meet the
targets?
Apart from these three perspectives, a second dimension is important
while developing an indicator to answer the question above in a complicated
multinational situation as in the EU. The indicator could be developed within

Table 2. Different Context and Perspectives on Policy Effectiveness Indicators for Emission
Reduction Targets.
Context
Member State
(MS) level

European Union
(EU) level

Perspective

Indicator

MS monitoring

Done MS

MS planning

To do MS

MS policy
strategy

Extrapolated
gap MS

EU monitoring

Done EU

EU planning

To do EU

EU policy
strategy

Extrapolated
gap EU

Meaning
The effort already delivered
by each Member State
The remaining emission
reduction requirement
for each Member State
The expectation (chance)
of each Member State
not meeting its target
The contribution of each
Member State to the total
emission reduction
achieved in the EU
The dependency of the EU
on the emission reduction
still to be achieved by each
Member State
The sensitivity of the EU
meeting its target to the chance
that a Member State will
not meet its target

356

TINUS PULLES & TOON VAN HARMELEN

different scopes or contexts:

Downloaded by [Arif Setya] at 06:30 27 January 2015

(1) The Member State context: (a) What are the efforts undertaken by the
Member States? (b) What are the requirements the Member States still
need to accommodate? (c) What are the expectations or chances that the
Member States will not meet the target?
(2) The EU context: (a) What is the contribution of Member States to meeting
the EU target? (b) How dependent is compliance at the EU level on the
efforts the Member States still have to do? (c) What is the sensitivity of EU
compliance or what risk does the EU run with respect to Member States
not meeting the national targets?
Table 2 summarizes these different contexts and perspectives. This paper
designs six different indicators (listed in the third column of Table 2) for each
of the three perspectives in both contexts.

2. MATERIALS AND METHODS

2.1. Data Used
The greenhouse gas data used in this study are obtained from a report,
produced by the European Environment Agency (EEA, 2003a), to describe the
progress of EU Member States towards the targets of the Kyoto Protocol.
These data are summarized in Table 3.
The EU agreed to a reduction of 8% for the averaged annual emissions during the first commitment period (20082012) relative to the 1990 emissions.
(In our analysis we set the target year to 2010. The base year for the
fluorinated gases is 1995 in many of the 15 original EU Member States.) Each
of the Member States in Kyoto agreed on the same emission reduction target.
The EU Member States however have since Kyoto made use of a provision
of the protocol (article 3, part 1 of the Protocol (UNFCCC, 1997)) that allows
parties to meet the targets jointly. In Council Decision 2002/358/EC (EC,
2002), the EU and its Member States agreed in 2002 on different emission
limitation and/or reduction targets for each Member State according to
economic circumstances, called the burden-sharing agreement. Due to
specific circumstances, some countries could relatively easily accept a large
reduction target (Luxembourg, Germany, Denmark, UK), whereas others
committed to limiting their emission increases to enable economic growth

357

POLICY EFFECTIVENESS INDICATORS

Downloaded by [Arif Setya] at 06:30 27 January 2015

Table 3. EU and Member States Greenhouse Gas Emission Trends (1990 and 2001) and Targets
(Burden-Sharing) for 20082012.

EU total
Austria
Belgium
Denmark
Finland
France
Germany
Greece
Ireland
Italy
Luxembourg
Netherlands
Portugal
Spain
Sweden
UK

Base year (1990)

(million ton of CO2
equivalents)

2001 (million ton

of CO2 equivalents)

EU Burden
sharing target
(% of 1990 emission)

4203.9
78.3
141.2
69.5
77.2
558.4
1216.2
107.0
53.4
509.3
10.9
211.1
61.4
289.9
72.9
747.2

4108.4
85.9
150.2
69.4
80.9
560.8
993.5
132.2
70.0
545.4
6.1
219.7
83.8
382.8
70.5
657.2

8.0%
13.0%
7.5%
21.0%
0.0%
0.0%
21.0%
25.0%
13.0%
6.5%
28.0%
6.0%
27.0%
15.0%
4.0%
12.5%

From: Table C4 in Greenhouse Gas Emission Trends and Projections in Europe 2003 (EEA,
2003a).

(Portugal, Greece, Spain, Ireland). The emission reduction targets as given in

Table 3 are the ones after the burden sharing.
The data on SO2 and NOx of the NEC Directive (Table 4) used in this study
have been downloaded from the electronic Data Service on the web site of the
European Environment Agency (EEA, 2004a). The most recent complete data
in this source are the emissions for the year 2000. Also the emissions for the
base year 1990 are taken from this data source.
The NEC Directive requests the EU Member States to reduce the national
emission totals of SO2, NOx, volatile organic compounds (VOC) and NH3 by
2010 to the levels given in the Annex I of the Directive. In this study we use the
SO2 and NOx data only. Table 4 also presents the 2010 national targets according
to the NEC Directive. Different from the climate case, the location where emissions of the NEC Directive pollutants take place is quite relevant for the environmental impact. This is reflected by the fact that no joint emission reduction
target for the EU is defined and a burden-sharing mechanism is not negotiated.

358

TINUS PULLES & TOON VAN HARMELEN

Table 4. EU and Member States Emission Trends (1990 and 2000) and Targets (National
Emissions Ceiling Directive NECD) for 2010 for NOx and SO2.

Downloaded by [Arif Setya] at 06:30 27 January 2015

NOx (thousand ton/year)

Austria
Belgium
Denmark
Finland
France
Germany
Greece
Ireland
Italy
Luxembourg
The Netherlands
Portugal
Spain
Sweden
UK

SO2 (thousand ton/year)

Base year
(1990)

2000

NECD
2010

Base year
(1990)

2000

NECD
2010

204
334
277
300
1900
2730
290
118
1940
23
570
286
1270
334
2760

196
329
209
236
1440
1580
321
125
1370
17
413
405
1430
252
1740

103
176
127
170
810
1051
344
65
990
11
260
250
847
148
1167

79
362
180
260
1320
5320
493
186
1650
15
202
288
2180
106
3720

38
165
28
74
654
638
483
131
758
3
92
288
1520
57
1190

39
99
55
110
375
520
523
42
475
4
50
160
746
67
585

From: EEA Data Service (EEA, 2004a); NECD targets from Handley and Taylor (2003).

2.2. Algorithms
Figure 1 explains the calculation for a certain year (current year in the
graph) of the progress towards meeting the target according to the three
following perspectives:
 Monitoring perspective: the effort that has been delivered from base year to
the current year is indicated by the arrow marked Done that gives simply
the decrease in emissions from base year to the current year.
 Planning perspective: the arrow marked To do indicates the emission
reduction required from the current year to reach the target emission in the
target year.
 Strategy perspective: the arrow marked Extrapolated gap indicates the
linearly extrapolated over- or undercompliance in the target year. It is in
fact a linear combination of the indicator To do and Done, as is
graphically depicted in Figure 1. Only if Done is on the straight line
from base year emission to target emission, the extrapolation of Done
for the period current year to target year equals To do resulting in an
Extrapolated gap of 0.

Downloaded by [Arif Setya] at 06:30 27 January 2015

POLICY EFFECTIVENESS INDICATORS

359

Fig. 1. Calculating indicators for the different perspectives.

To express the two different contexts, we calculate the emission values

relative to the national base year emission for each Member States separately
(MS context) or the aggregated EU base year emission (EU context). Table 5
lists the equations used to calculate the indicators.
Please note that indicators calculated in the EU context for each
individual Member State will add up to the same indicator for the EU as
a total. The indicators in the Member State context will not. In this case
the EU indicator will be the weighted average of the Member State indicators; the indicator Distance to Target used in the EEAs report Greenhouse Gas Emission Trends and Projections in Europe 2003 (EEA, 2003a)
is equivalent to our Extrapolated gap MS indicator; the difference is
only that we relate the gap to the 2010 target, whereas the EEAs report
is relating the gap to the interpolated 2001 target. The numerical values
in the EEAs report (EEA, 2003a) therefore are a factor of 0.55
((20011990)/(20101990)) smaller. The interpolation in both the distance to target indicator and our extrapolated gap indicator implicitly
assumes that the target is reached along a linear time path. This obviously
does not need to be the case. Reduction possibilities might have been
partly exhausted in the reporting year or the country might have policies
in place or preparation that take effect later in time. More sophisticated
approaches might be developed, but these will always need more data and
information.

360

Table 5. The Algorithms Used to Calculate the Six Different Policy Effectiveness Indicators.
Algorithm

Done MS

Idone;MS

EMS;year EMS;base
EMS;base

To do MS

Ito do;MS

EMS;target EMS;year
EMS;base

Extrapolated
gap MS



tt
Igap;MS  Ito do;MS  tbasetarget
t  Idone;MS

Done EU

Idone;EU

EMS;year EMS;base
EEU;base

To do EU

Ito do;EU

EMS;target EMS;year
EEU;base

Extrapolated
gap EU



tt
Igap;EU  Ito do;EU  tbasetarget
t  Idone;EU

Explanation
Calculates the reduction achieved relative
to the Member State base year emission
Calculates the reduction still to be achieved,
relative to the Member State base
year emission
Calculates the linearly extrapolated
over- or undercompliance in the target
year, relative to the Member State
base year emission
Calculates the reduction achieved
relative to the EU base year emission
Calculates the reduction still
to be achieved, relative to the EU
base year emission
Calculates the linearly extrapolated
over- or undercompliance in the target
year, relative to the EU base year emission

Note. I, with appropriate suffixes, stands for the indicator, E stands for the emission of MS or EU in, respectively, the base year (tbase; 1990
in the Kyoto Protocol), the current year (t) and the target year (ttarget; we use 2010 as the average of 20082012).

TINUS PULLES & TOON VAN HARMELEN

Downloaded by [Arif Setya] at 06:30 27 January 2015

Indicator

POLICY EFFECTIVENESS INDICATORS

361

3. RESULTS

Downloaded by [Arif Setya] at 06:30 27 January 2015

Figure 2 graphically presents for climate change the values for the six
indicators, calculated by the algorithms of Table 5. The indicators in the left
part of the graph are reflecting the Member State context while the indicators
on the right part of the graph represent the EU context. The indicators
Done, To do and Extrapolated gap are presented from the top of the
graph downwards. Each indicator is sorted from the worst to the best.
These results show that not only the pattern of numerical values is sensitive for
the different indicators, but also the ranking of the individual Member States is
quite different for each indicator:
 For the indicator Done representing the effort delivered by each Member
State, Luxembourg appears as the best, whereas Portugal is the worst; the
indicator for the contribution given to the total EU achievement, however,
results in Germany as the best and Spain as the worst.
 The To do indicator expressing the reduction requirements still remaining for each individual Member State points to Luxembourg again
as the best, whereas Austria now is the worst; in the similar
indicator in the European context (the dependency of the EU on each
Member State meeting the targets), Sweden is the best and Italy is
the worst.
 The Extrapolated gap indicator in the Member States context shows
again Luxembourg as the best and this time Ireland as the worst; the
corresponding indicator in the EU context now shows Germany as the best
and Spain as the worst.
The differences between the indicators can sometimes easily be understood. The difference between the Member State and the EU context is rooted
in the relative size of the base year emission, resulting in relatively high
(positive or negative) impacts for large (emitting) countries compared with
small (in terms of emission level) Member States. For example, the
Netherlands, having the sixth largest emission out of 15 countries, scores
relatively low in the EU context compared with the MS context, resulting in a
worst score at the Extrapolated gap indicator with a third worst place after
Spain and Italy. In fact, the middle scores on the indicators according MS
context can be explained by the fact that the Netherlands scores better than
relatively small countries such as Portugal, Ireland, Austria and Greece. But
this is not very relevant in reaching the EU target.

TINUS PULLES & TOON VAN HARMELEN

Downloaded by [Arif Setya] at 06:30 27 January 2015

362

Fig. 2. The six different indicators for distance to target of greenhouse gas emissions in 2001
for the 15 EU Member States separately (dark bars) and the EU as a total (light bars).

By this example it becomes clear that using specific indicators for different
contexts can acknowledge and clarify differences in opinion and sense of
urgency on Member States progress in policy.

Downloaded by [Arif Setya] at 06:30 27 January 2015

POLICY EFFECTIVENESS INDICATORS

363

Fig. 3. The rankings for individual Member States on the six different indicators for climate
change policies; countries are sorted according to increasing averaged ranking.

However, the differences between scores on indicators according different

perspectives are less easy to understand. Although one might expect that
scoring high on Done would lead to scoring low on To do, this is not
always the case since emission reduction targets differ substantially from
country to country. The Extrapolated gap indicator is adding more
complexity since it is a linear combination of the other two indicators.
In Figure 3 the highest, lowest and average ranking for each Member State
are given. For six countries (Finland, France, Belgium. Luxembourg, Spain,
Sweden and the UK) the ranking is not very sensitive to the choice of the
indicator. The highest ranking differs three places or less from their lowest
ranking. For other countries (Germany, Greece, Denmark and Portugal) the
highest and lowest ranking differ eight or more places. For the other countries
(Italy, the Netherlands, Austria and Ireland) the highest and lowest rank differ
between five, six or seven places.
To establish whether or not this behaviour is typical for the greenhouse gas
emissions and the Kyoto targets, a similar analysis is presented in Figure 4 for
the NEC Directive reduction targets for NOx and SO2 in the 15 EU Member
States. In this case, the EU Context is not relevant, since there is no joint target

TINUS PULLES & TOON VAN HARMELEN

Downloaded by [Arif Setya] at 06:30 27 January 2015

364

Fig. 4. Rankings of EU Members States on the three MS context distance to target indicators
for NOx and SO2 against the NEC Directive targets for 2010.

or burden sharing mechanism in place. The figure therefore only presents the
results for the three perspectives in the Member State context. The figure
shows that also for these pollutants the ranking of each Member State differs

POLICY EFFECTIVENESS INDICATORS

365

quite a lot for the three indicators used here. The differences seem to be
smaller than in the case of the greenhouse gases in Figure 3, but this is mainly
caused by the fact that the EU context is not used here.
So, even in these straightforward examples, six (greenhouse gases) or three
(NEC Directive) simple indicators generate a variety of scores for a majority
of countries. Because of this complexity, indicators have to be defined and
explained explicitly in order to be able to address specific aspects of policy
effectiveness.

Downloaded by [Arif Setya] at 06:30 27 January 2015

4. DISCUSSION AND CONCLUSIONS

Indicators to support policy decisions are usually quite straightforward and
easy to understand. This is in fact the purpose of developing such indicators: to
simplify a complex state or development in the real world in such a way that a
specific policy makers question is answered.
Nevertheless, we have shown that a certain type of policy indicators
addressing the effectiveness of policies is subject to a more ambiguous
interpretation. This is due to the variety of possible (and thus subjective)
definitions regarding the issue at stake here: policy effectiveness. What is
meant exactly by policy effectiveness? To this end, we put forward three
definitions of effectiveness according to three so-called perspectives. Do we
address effectiveness in terms of what has been done, what is still to be
done or what is expected to be the result in the target year? Each
perspective stands for a different stage of the policy cycle, respectively
monitoring, planning and strategy. For each of these perspectives, we
developed three accompanying indicators. In addition to this, we argued
that the used definition of policy effectiveness can be dependent on the
government (level). Hence, the indicators need to be specific to contexts as
well.
We illustrated this with the cases of greenhouse gas policies and the air
pollutant policies within the EU. The policy question How far are the EU and
its Member States on its road to meeting the emission reduction targets? for
both policy areas has been translated into different indicators, depending on
the perspective and context.
The indicators as discussed in this paper assess the total effect of all policy
measures together, since they are based upon the national total emissions only.
Moreover, favourable national indicator values do not guarantee the existence
of a conscious and effective policy, since many general (e.g. energy prices)

Downloaded by [Arif Setya] at 06:30 27 January 2015

366

TINUS PULLES & TOON VAN HARMELEN

and specific (e.g. unification) factors influence national emission developments. Therefore, policy makers will also need indicators that show the
efficiency of individual policy measures. This adds complexity to the data and
the analysis that goes beyond the scope of this paper. In a further study we will
investigate this issue further, based on the results of another recent EEA report
on air pollution (EEA, 2004b).
The present paper showed that even with simple examples (15 actual
reported national totals to be related with 15 different targets and, in the case
of greenhouse gases, capped by a EU total), completely different rankings for
EU Member States occur. This is surprising since the use of a single policy
effectiveness indicator is internationally widespread and accepted to make a
simple and convincing point. It is even more surprising that hardly any
literature exists on the proper use of policy effectiveness indicators. We
conclude that policy effectiveness is such an ambiguous term that the use of
only one indicator should always cause suspicion. Only when several
interpretations related to different perspectives and contexts are addressed
explicitly, preferably in the form of explicit indicators, robust conclusions can
be drawn on the effectiveness of policies.
It would create an understanding between parties if different contexts and
perspectives are acknowledged. Hence, general policy questions, usually
covering several parties, can only be addressed effectively by more than one
specific policy effectiveness indicator. Because in the real world, scapegoats
mostly come in herds, just as ministering angels do.

REFERENCES
EC. (2001). Directive 2001/81/EC of the European Parliament and of the Council of 23 October
2001 on national emission ceilings for certain atmospheric pollutants. Official Journal L
309, 27 November 2001, pp. 2230.
EC. (2002). Council Decision of 25 April 2002 concerning the approval, on behalf of the
European Community, of the Kyoto Protocol to the United Nations Framework
Convention on Climate Change and the joint fulfilment of commitments thereunder
(2002/358/CE). Official Journal L130, 15 May 2002, p. 1.
EEA. (2003a). Greenhouse gas emission trends and projections in Europe 2003, Environmental
issue Report No. 36, European Environment Agency. URL: http://reports.eea.eu.int/
environmental_issue_report_2003_36/en
EEA. (2003b). Europes environment: The third assessment, Environmental assessment Report
No. 10, European Environment Agency. URL: http://reports.eea.eu.int/assessment_
report_2003_10/en

Downloaded by [Arif Setya] at 06:30 27 January 2015

POLICY EFFECTIVENESS INDICATORS

367

EEA. (2004a). Trends in emissions of ozone precursors (CLRTAP/EMEP), European Environment Agencys data service. URL: http://dataservice.eea.eu.int/dataservice/metadetails.
asp?id 620, data downloaded in April 2004.
EEA. (2004b). Air pollution in Europe 19902000, Topic Report No. 4/2003, European
Environment Agency. URL: http://reports.eea.eu.int/topic_report_2003_4/en
Furfero, A.J. (2000). Macroeconomic stabilization policies: Goals, institutions, and theories
(8th ed.). URL: http://www.drfurfero.com/books/231book/book.html
Handley, C. & Taylor, P. (2003). An initial assessment of Member States national programmes
and projections under the National Emissions Ceiling Directive (2001/81/EC). ETC/
ACC Technical Paper 2003/8, ETC-ACC 2003. Available: http://air-climate.
eionet.eu.int/reports/ETCACC_TechPaper2003_8_NECD_InitAss
Ramaswamy, V. (2001). Radiative forcing of climate change. In J.T. Houghton (Ed.), Climate
Change 2001: The Scientific Basis. Cambridge: Cambridge University Press.
UNFCCC. (1997). Report of the conference of the parties on its third session, Held at Kyoto
from 1 to 11 December 1997; Addendum, Part Two: Action Taken by the Conference of
the Parties at its Third Session, FCCC/CP/1997/7/Add.1, 18 March 1998.
UNFCCC. (2003). All national submissions of Greenhouse Gas Emission inventories and
National inventory Reports in 2003 can be found at UNFCCCs web site at URL: http://
unfccc.int/program/mis/ghg/submis2003.html