Blog

Why Carry Out Economic Evaluation?

Impact blog

Dr Ulrike Hotopp, Chief Economist, Simetrica and member of the Council of the UK Evaluation Society

I have only recently joined the small economic research consultancy Simetrica. Before this I spent 16 years in the Government Economic Service, starting as an economic advisor in DTI in 2000 (now known as BEIS). I first worked on employment policy and one of my main tasks was to produce Impact Assessments for new employment regulation using the tools of Cost Benefit Analysis (CBA). These can also be described as ex-ante evaluation – forecasting the impact of new regulation.

Having done a few of these, an important question is: how realistic was the assessment? Was it really true that improved maternity rights had no impact on female employment but instead improved the rate of women returning to work after having a child? Were there any wider micro and macro economic effects? On other groups in the labour market, company profits?

Access to Data is Crucial

data

Emma Uprichard and Robert MacKay, University of Warwick

Interviewed by Candice Howarth, University of Surrey

CECAN is exploring how evaluation of policy can better inform the impact those policies have and assess the extent to which these have been successful. In order to do this, access to data is crucial, yet can at times be problematic. CECAN’s Knowledge Integrator, Candice Howarth met Emma Uprichard and Robert MacKay from the Centre and based at the University of Warwick and asked them over a series of emails to explain what the implications of some of these challenges are.

Why is data needed in evaluations? 

Aligning Policy and Evidence for the Age of Complexity

Aligning Policy and Evidence for the Age of Complexity

Gary Kass, Visiting Professor, Centre for Environment and Sustainability, University of Surrey

As the world changes in complex and unpredictable ways, Government is changing too.  As it does so, the need grows for policy-making and the evidence that informs it to be alive and responsive to the increasing pervasiveness of complexity.  In public service systems the increase in complexity often means that  no single institution is ever ‘in charge’ or has direct control over how changes unfold.  In response, as policy-makers and service deliverers begin to recognise the new reality of pervasive complexity, we see a fading of inadequate notions of one-size-fits all, predict-and-provide and command-and-control approaches.  Rather, we see a brightening of newer notions such as open policy-making; outcomes-focus; and service-driven adaptive approaches.

One key driver of these shifts is the growing recognition that complexity and uncertainty are pervasive and dominant features in the world with many of the certainties and stabilities of the past being eroded or removed.  A key requirement, then, is to build capacity to set and deliver public policy as these changes unfold.

Complexity High on the Agenda at the EES 2016 Biannual Conference

EES conference

Dione Hills, Tavistock Institute

Maastricht was the location of this year’s European Evaluation Society (EES) conference over a sunny week in late September. At the end of first day, we were treated to a civic reception in the building in which the Maastricht treaty was signed in 1992, bringing up mixed emotions for some of us.

700 participants from across the globe attended over 200 presentations and workshops that spanned high quality discussions about the current state of evaluation theory to practical accounts of interesting – and innovative - evaluations ‘on the ground’.

Likelihoods

Blog by Stuart Astill IOD Parc

By Stuart Astill, Principal Consultant, IOD PARC

Likelihoods

The scope of the CECAN project runs wide as well as deep; complexity in the energy, water, environment and food domain would most immediately be thought to arise from the physical systems at the nexus core. Yes, complexity in weather systems, biological populations spring quickly and easily to mind as do the ‘user level’ human interactions with these systems and other local and global physical systems. However, when we are addressing remedial interventions, or even ongoing management of nexus concerns, we quickly realise a double dose of complexity if, as we certainly should, we look towards governance.

Is Policy Evaluation Fit For Purpose?

cecan eevent

By Roger Highfield, Director of External Affairs of the Science Museum, member of the Royal Society’s Science Policy Advisory Group.
 
To tackle climate change, ecosystem destruction and the many daunting issues facing humanity we need not only to draw on science and engineering but also develop policies that can change the behaviour of 7.5 billion people.
 
That means we need ways to evaluate which policies work and which don’t, and figure out how to hone them. However, there is still some way to go to make our existing institutional machinery fit for purpose, according to a far-ranging discussion on Policy Evaluation for a Complex World I chaired this month, at St Martin’s in the Fields, Trafalgar Square, London, for the Centre for the Evaluation of Complexity Across the Nexus (CECAN).

After the Brexit vote: What Next for the UK’s Environment?

After the Brexit Vote

This post is by Andy Jordan, Charlotte Burns and Viviane Gravey.  They recently co-led an expert reviewof the environmental implications of Brexit funded by the UK in a Changing Europe Initiative.

After a deeply divisive campaign, UK voters have opted by a small majority to leave the European Union.  Environmentalists are accustomed to most policy being made jointly with the EU. The shock result flips that assumption completely on its head. The referendum process may be over, but the hard political debate over policy starts now.

How Should Academics Interact with Policy Makers? Lessons on Building a Long-term Advocacy Strategy.

how should academics interact

What can academics learn from how civil society organisations and NGOs approach policy impact? Julia Himmrich argues that academics have a lot to gain from embracing the practices of long-term advocacy. Advocacy is about establishing relationships and creating a community of experts both in and outside of government who can give informed input on policies. Being more aware of the political aspects of research can help academics understand and re-evaluate their own arguments about the impact of research.

This piece is based on an article written for the Dahrendorf Forum at LSE IDEAS.

Reflections on Language and Complexity

nexus

Blog by Anne Liddon, Scientific Communications Manager, Newcastle University

When I turned up at the CECAN Evaluation and Complexity workshop this week it was my first day back at work after a holiday in Crete, an experience that I thought might have put me in the right frame of mind.   Being the non-scientist in a roomful of scientists often seems like being in a foreign country where I only speak a few words of the language.  Sometimes it also feels as though I’m wading through treacle with a paper bag over my head, because I can’t see what’s going on, I keep getting stuck and I don’t know which way to go, then I bump into people I don’t recognise.  Finally I find out who they are, (sometimes we have been emailing each other for months) and they feel like old friends after all. 

Should Academics be Expected to Change Policy? Six Reasons Why it is Unrealistic for Research to Drive Policy Change.

Should academics drive policy change

UK social scientists feel a growing pressure to achieve policy change. In reality, this process is more complex than it sounds. James Lloyd looks at six reasons that limit the impact research can have on policy change. None of this should suggest that academic researchers shouldn’t seek to influence policymaking. But more consideration is needed on how best academic evidence can leverage the real-world nature of policymaking.

There has been understandable relief at indications from the government that academic researchers will be exempt from anti-advocacy clauses in research grant contracts. The possibility that academics with publicly funded research grants would not be able to press the government for policy change was clearly unacceptable and anti-democratic.

The Science of Using Research...

the science of using research

Governments all over the world invest large sums of public money into producing knowledge that helps them understand their countries’ complex socioeconomic issues. This knowledge, in the form of research, can be used to formulate potential solutions through public policies and programmes.

But it’s not enough just to produce research. It must also be considered and drawn from when policies are being created. However, a range of barriers might prevent policymakers from accessing and using evidence in their work. To understand the use of evidence, then, it’s important to understand the policymaker. Who is she? What are her incentives and biases? What is her professional and institutional context?

Blog article by Lawrence Langer and Ruth Stewart

Clearing The Fog

Complex Policy Evaluation

Development actors facing pressure to provide more rigorous assessments of their impact on policy and practice need new methods to deliver them.

There is now a broad consensus that the traditional counterfactual analysis leading to the assessment of the net effect of an intervention is incapable of capturing the complexity of factors at play in any particular policy change.

We suggest that evaluations focus instead on establishing whether a clearly-defined process of change has taken place, and improve the validity and credibility of qualitative impact statements.

IIED research in Uganda shows that the methods of process tracing and Bayesian updating facilitate a dialogue between theory and evidence that allows us to assess our degree of confidence in ‘contribution claims’ in a transparent and replicable way.

Full article from CECAN's Barbara Befani

Global Challenges Require Cross-Cutting Solutions

Complexity in Policy Evaluation

New research led by the Tyndall Centre for Climate Change Research suggests that current UK policies on water, energy and food are too fragmented to effectively tackle global challenges. Issues such as climate change, resource constraints and the increasing population cut across several sectors and need similarly cross-sectoral policies. Future research must meet this challenge by focusing on the nexus between sectors, scales and timeframes.

To ensure resilience to complex global challenges, it is necessary to evaluate whether policies and underlying research are working effectively across a range of sectors. It is also important to analyse how existing policies affect the water-energy-food nexus at different scales (local, national, global) and times (short- and long-term). Our new research does exactly that, exploring how the policies could better function across the water, energy and food-related sectors of the economy.

Tools, Tools Everywhere and not a Hammer in Sight!

Complexity in Policy Evaluation

By Pete Barbrook-Johnson (né Johnson)

Members of the Sociology department, alongside colleagues from across the University of Surrey, have been working on the ERIE project for the past six years. One of the main outputs of the project is the development of a suite of software tools designed to help anyone and everyone make decisions and think strategically. The tools have been designed with government, industry and charity sector activities in mind, but they can be applied to a very wide range of topics, from managing your health and exercise, to understanding how we might better collaborate with colleagues across the University, and even bringing together students to share and build knowledge.

Have a look, get involved…