Impl icati ons of Cl imate

Ch ang e
for Eval uation

Per Mickwitz
Finnish Environment Institute (SYKE)
Environmental Evaluator’s Networking
Forum

13.6.2008
Structure of the presentation

 Categorization of the implications of climate

change on evaluation

 European and Finnish examples of the different

implication categories

 Implications of climate change from an evaluation

methodology perspective
Implications of climate change for
evaluation

1. Implications through evaluation of climate

programs and policies

3. Implications of climate change on the evaluation

of "non-climate" programs and policies.
a) implications through changes of the climate;
b) implications due to interactions with climate programs
and policies (both mitigation and adaptation);
c) implications because other policies and programs, e.g.
traffic or agricultural programs, integrate climate aims.
1. Implications through evaluation of
climate programs and policies (1/3)

 Program and policy evaluation has not yet been a central

issue in climate policy
• monitoring of emissions, verification of emission reductions of
CDM projects, assessing additionality have been much more
central
• IPCC’s 4th assessment talks about evaluation, but at a very
general level
 The evaluation community (e.g. AEA, EES), on the other
hand, has not yet discussed issues related to climate
change a lot.
• Some recent exceptions, Alexandria conference & ADAM

 The role of evaluation will increase, when the end of the

commitment period of the Kyoto protocol (2008-12) is
approaching and when targets are becoming more
stringent (e.g. EU 20 20 to 2020)
1. Implications through evaluation of
climate programs and policies (2/3)

 The need for evaluation will be very different for:

• Programs and policies directed at activities/sectors covered by
emission trading
• Other programs and policies

 In the case of activities covered by emission trading

schemes
• Evaluation is not required to establish impacts or
effectiveness in reducing greenhouse gas emissions
• Evaluation largely about costs
• Side-effects (air, biodiversity; development)
• Distribution of costs and side-effects (geographic, industries,
income groups …)
1. Implications through evaluation of
climate programs and policies (3/3)
Sectors not covered by emission trading: transport

 The EU 20 20 by 2020
proposal includes legally-
binding reduction target in
sectors not covered by the
EU ETS for 2020 compared
to 2005:
Finland -16 %
 1. Implications through evaluation of
climate programs and policies (3/3)
Sectors not covered by emission trading: transport
16

 The EU 20 20 by 2020
14
proposal includes legally-
12 binding reduction target in
sectors not covered by the
10
EU ETS for 2020 compared
to 2005:
million t CO2

8
Finland -16 %
From 1990 + 8.7 %
6

From 1995 +16.1 %

 Implication: CO2 emissions
4 from transport have to be
reduced substantially
2

0
1990 1995 2000 2005
 1. Implications through evaluation of
climate programs and policies (3/3)
Sectors not covered by emission trading: transport
Developm ent of public and passanger car transport

70 000
Million person kilometres  The EU 20 20 by 2020
proposal includes legally-
60 000 binding reduction target in
Passenger cars
sectors not covered by the
50 000
EU ETS for 2020 compared
to 2005:
40 000
Finland -16 %
30 000
 Implication: CO2 emissions
20 000
from transport have to be
Public transport reduced substantially
10 000

0
1980 1985 1990 1995 2000 2005
1. Implications through evaluation of
climate programs and policies (3/3)
Sectors not covered by emission trading: transport

Examples of explanations:
 The share of public transport
has gone down
 Urbane structure has
changed, e.g. people
commute longer distances to
work
 1. Implications through evaluation of
climate programs and policies (3/3)
Sectors not covered by emission trading: transport
16
Policies and programs:
14
 EU legislation on CO2 from new
12 cars
 National fuel taxation
10
 CO2 based taxation of cars
million t CO2

8
 Review of income tax rebate for
commuting to work
6
 Promoting eco-driving
4
 Urban planning (shopping malls)
2  All these jointly influence
emissions
0
1990 1995 2000 2005
 Huge need for effectiveness
evaluations
2 a. Implications of climate change on the
evaluation of "non-climate" programs and
policies through changes of the climate

 Example: eutrophication of
the Baltic sea due to
nutrient loading
 The main Finnish source
of nutrients is agriculture
• 63 % of Phosphorus 2004
• 51 % of Nitrogen 2004
 In the Finnish Government
decision-in-principle on
Water Protection Policy
Outlines to 2015: goal
- 30% compared to 2001-5
2 a. Implications of climate change on the
evaluation of "non-climate" programs and
policies through changes of the climate

 Many options to reduce

nutrient loads from
agriculture:
• Reduced use of fertilizers
• Changed use of fertilizers
• Wetlands & buffer zones
• Reduced farming on steep
fields close to waters
• Changed crops
 Programs and policies
• Education
• Support conditions
• Fertilizer tax proposal
• Wetland agreements
 2 a. Implications of climate change on the
evaluation of "non-climate" programs and
policies through changes of the climate

 The Winter 2007-8 was the

warmest ever recorded in Finland
• Hardly no snow or frost
 Huge effects on run-offs of
nutrients from fields
 Should be considered in
evaluations
• Confounding factor
• Attribution
 Should be taken into account
when planning evaluations
 Programs and policies
• Education
Only artificial snow on the ski track • Support conditions
• Fertilizer tax proposal
• wetland agreements
2 a. Implications of climate change on the
evaluation of "non-climate" programs and
policies through changes of the climate

 Long term climate change

expected to have much larger
impacts on the temperature
on rainfall than last winter
and thus on farming practices
 A fixed indicator approach
(too common in Europe)
misleading
 Implications on program
theories (even science based
part) & on results chains
 Sensitivity of a program or
policy under climate change
as an evaluation criteria
2 b. Implications of climate change on the
evaluation of "non-climate" programs and
policies due to interactions with climate
programs and policies – Example bio-energy

 The EU 20 20 by 2020
proposal based on 20 %
renewables by 2020, include
specific legally-binding share
of renewables in the final
energy demand by 2020:
Finland 38 %
 The European Council
decided on a 10% biofuel
target for transport
 In Finland 20 % of all energy
is already bio-energy
2 b. Implications of climate change on the
evaluation of "non-climate" programs and
policies due to interactions with climate
programs and policies – Example bio-energy
“Efforts to preserve forest diversity
have been intensified recently,
through increases in the areas of
forest under protection, improvements
in the quality of protected forests
achieved though habitat restoration
schemes, and the development of
natural forest management practices
for commercially managed forests.
… without further measures the
numbers of forest species becoming
threatened or extinct in Finland will go
on rising in the near future. … With
regard to the structural features of
natural forests, it is particularly
important for the sake of biodiversity
to increase the amounts of decaying
wood in the forests.”
2 b. Implications of climate change on the
evaluation of "non-climate" programs and
policies due to interactions with climate
programs and policies – Example bio-energy
“Efforts to preserve forest diversity
have been intensified recently,
through increases in the areas of
forest under protection, improvements
in the quality of protected forests
achieved though habitat restoration
schemes, and the development of
natural forest management practices
for commercially managed forests.
… without further measures the
numbers of forest species becoming
threatened or extinct in Finland will go
Changed practices due to on rising in the near future. … With
increased demand of bio- regard to the structural features of
energy: far less “residuals” natural forests, it is particularly
important for the sake of biodiversity
are left in the forests, even to increase the amounts of decaying
the stubs are removed. wood in the forests.”
2 b. Implications of climate change on the
evaluation of "non-climate" programs and
policies due to interactions with climate
programs and policies – Example bio-energy
“Efforts to preserve forest diversity
have been intensified recently,
Should be considered through increases in the areas of
forest under protection, improvements
in evaluations in the quality of protected forests
• Confounding factor achieved though habitat restoration
• Attribution schemes, and the development of
Implications on natural forest management practices
for commercially managed forests.
program theories
… without further measures the
numbers of forest species becoming
threatened or extinct in Finland will go
Changed practices due to on rising in the near future. … With
increased demand of bio- regard to the structural features of
energy: far less “residuals” natural forests, it is particularly
important for the sake of biodiversity
are left in the forests, even to increase the amounts of decaying
the stubs are removed. wood in the forests.”
2 c. Implications of climate change on the
evaluation of "non-climate" programs and
policies because other policies and programs
integrate climate aims.

 Climate integration needs to be evaluated – It is

far easier to declare intention to integrate than to
carry it out
• We have just finished a study on Mainstreaming Climate Policy
for the Finnish Prime Minister’s Office
• Mickwitz P. and P. Kivimaa 2007. Evaluating Policy Integration:
The Case of Policies for Environmentally Friendlier
Technological Innovations, Evaluation 13 (1), 68-86.

 Climate integration will influence the evaluation

of the other aims of the programs and policies
Implications of climate change from an
evaluation methodology perspective

 Side-effects  The evaluation of climate

evaluation programs and policies
requires a strong focus on
 Evaluation criteria side-effects
 Attribution • Positive e.g. air emissions

 Program theory • Negative e.g. biodiversity

• Non-environmental e.g.
 Data development, income
distribution, innovations
 Uncertainty and technological
development
 IPCC 2007 (chapter 11)
“Spillover effects”
Implications of climate change from an
evaluation methodology perspective

 IPCC (2007, WG3, 13.1.2) Criteria for

 Side-effects policy choice:
evaluation Environmental effectiveness – the extent
to which a policy meets its intended
 Evaluation criteria environmental objective or realizes
positive environmental outcomes.
 Attribution Cost-effectiveness – the extent to which
the policy can achieve its objectives at a
 Program theory minimum cost to society.
Distributional considerations – the
 Data incidence or distributional
consequences of a policy, which
includes dimensions such as fairness
 Uncertainty and equity, although there are others.
Institutional feasibility – the extent to
which a policy instrument is likely to be
viewed as legitimate, gain acceptance,
adopted and implemented.
Ex ante & Climate policies
Implications of climate change from an
evaluation methodology perspective

 Side-effects Other criteria & non-climate

evaluation programs and policies :
 Flexibility – can the program
 Evaluation criteria cope with changing climate or
 changing climate policies?
Attribution
 Climate robustness - will the
 Program theory program be effective under
different climate scenarios
 Data
• e.g. protecting a specific habitat
 Uncertainty vs. creating ecological corridors
 Persistence – are the impacts
lasting even when the climate
change
Implications of climate change from an
evaluation methodology perspective

 Side-effects When attributing impacts to

evaluation programs and policies the
 Evaluation criteria contribution of:
• Climate change
 Attribution • Climate programs and
 Program theory policies
• Climate policy integration
 Data
should be considered.
 Uncertainty
Implications of climate change from an
evaluation methodology perspective

 Side-effects Program theories of climate as well

as other programs need to take
evaluation into account:
 Evaluation criteria • Climate change
• Climate policies and programs as
 Attribution well as the social and economic
processes affected by them
 Program theory
• Climate policy integration
 Data  Requires:
 Uncertainty • Increased awareness
• Utilization of natural sciences
(adaptation & impacts)
• Utilization of social sciences
• More research, especially on
social & economic processes
Implications of climate change for
evaluation: a methods perspective

 Side-effects  Climate change may interrupt

evaluation data series “no-ice effect”
“The flood took the samples”
 Evaluation criteria  Interpretation of data and
 Attribution especially “indicators”
 Program theory
 Data
 Uncertainty
Implications of climate change from an
evaluation methodology perspective

 Side-effects  Many aspects of climate

evaluation change are VERY uncertain
 Technological development,
 Evaluation criteria social and economic
 Attribution processes, attitudes are
among the most uncertain
 Program theory ones
 Data  Thresholds and non-linear
 Uncertainty changes are possible

 Relying on linear models may

be misleading
 I believe that climate change will have huge
implications for evaluation.
 The huge uncertainties involved will have
implications for the evaluation processes –
unrealistic expectations may influence the
legitimacy of evaluation.
 How influential should evaluations be (Hans
Bruyninckx & Jared Hardner)?