The evaluation of climate change mitigation policy is crucial for understanding how well policies and measures work. Policy evaluation offers analysts insights in the functioning of policies and provides policy makers with much-needed information on how to improve them. In addition, policy evaluations can enhance the transparency of policy implementation, which is essential to gain citizens’ support for those policies.

Meta-analysis

The CARISMA project team carried out a meta-analysis of climate policy evaluations, of which results were published in CARISMA Discussion Paper No. 4. The team opted to include the three largest EU Member States (France, Germany, and the UK) as well as smaller Member States from Central and South-Eastern Europe (Austria, Czech Republic, and Greece). In addition, since important climate policy evaluations for the EU had been carried out at the EU level, the EU was included as a separate jurisdiction.

To offer a recent overview, reflecting climate policy developments in the period after the COP15 in Copenhagen and after the enactment of the EU’s 2020 climate and energy package, both in 2009, it was decided to cover the period from January 2010 to December 2016.

The number of climate policy evaluations is potentially large, and the project team sought to limit the number of evaluations in several ways: only ex-post evaluations; only mitigation policies; excluding purely academic articles; excluding non-systematic analyses such as position papers; excluding subnational evaluations; and only including publicly available documents.

Results

The total number of sampled evaluations is 236. The three major jurisdictions contributing to the evaluations are the EU (70 evaluations, 30%), the UK (64 evaluations, 27%) and Germany (59 evaluations, 25%).

Figure 1 shows the affiliation of authors, highlighting that universities or research institutes, followed by government bodies, were responsible for the clear majority of evaluations.

Most evaluations were commissioned by government bodies.

It was found that the energy sector is dominant in policy evaluations, with 171 evaluation entries covering the sector (multiple answers were possible). This was followed by cross-sectoral evaluations (61 entries), industry/industrial processes (51) and transport (48). This pattern is common to most of the jurisdictions covered, except the EU, which had a relatively higher share of cross-sectoral approaches than Member States. Moreover, some sectors, such as the agricultural, waste and land use, land-use change and forestry (LULUCF) sectors, are clearly under-represented.

Figure 2. Sectors covered, by jurisdiction

The strong focus on the energy sector in climate change mitigation policies corresponds to observations made in the CARISMA analysis (pdf) of information available in international climate change mitigation policy databases.

Using the evaluation methods and evaluation criteria as a proxy for reflexivity, the majority of evaluations were found to be non-reflexive. The largest number of evaluation methods applied (multiple answers were allowed) belonged to the category of documentary analysis (153 entries), followed by modelling, regression analysis or time series analysis (88), public opinion polls, user surveys, stakeholder analysis, feasibility assessments or expert interviews (85) and cost benefit analysis, cost-effectiveness, multi-criteria analysis, feasibility analysis or risk analysis (44). In terms of stakeholder involvement, it can be assumed that most of the methods applied are neither participatory nor interactive, except for the second category.

Recommendations

What insights does this meta-analysis provide for policymakers? First, the meta-analysis reported here shows that a wealth of climate policy evaluations is being produced on an annual basis as a subset of policy evaluations in Europe. While the evaluation practice has not been compared across case studies (e.g. how does Germany evaluate mitigation policies compared to the United Kingdom), the large and increasing number of evaluations might harbour some redundancies and overlaps. The exercise of identifying, coding and categorising 236 evaluations for this seven-year period helps to provide insights into the potential use of ex-post evaluations in support of future EU legislative proposals and accompanying impact assessments.

Addressing the evaluation criteria such as fairness and legitimacy in more Member States could enhance understanding of climate change mitigation policies across the EU. Having a good understanding on how a certain policy performed particularly according to these criteria might form the basis for more ambitious climate change mitigation policies in the future. More inclusive and participatory approaches to policy evaluations could enhance their usefulness by giving space to a variety of actors (civil society organizations, businesses, citizens, etc.) to voice their views and share their experiences when analysing policies.

It is crucial and urgent to allocate sufficient resources to the coverage of relatively under-represented sectors, such as LULUCF and waste. On the sectoral coverage, the scope of evaluations is narrowed down from climate policies in general to those eligible for climate change mitigation under the UNFCCC reporting system. This study reaffirms that in the areas of mitigation policies, there is a mismatch between the sectors targeted by the majority of policy evaluations and those responsible for emissions.

The full results of the meta-analysis, as well as more detailed recommendations, are available in CARISMA Discussion Paper No. 4, titled “Climate change policy evaluations in the EU and Member States: Results from a meta-analysis”, prepared by Noriko Fujiwara, Stefan Bößner, and Harro van Asselt.

This article is based on a Dicussion Paper published by the EU-funded CARISMA project on innovation for climate change mitigation.