Blog

Complexity and Evaluation Failure

Complexity and Evaluation Failure

 

By Dione Hills, Principal Researcher Consultant (The Tavistock Institute of Human Relations) 

24th July 2019   

 

Why complex evaluations fail: 

People attending CECAN trainings and events often want to hear about evaluations that ‘failed’ because the wrong (i.e. not complexity appropriate) methods or approach were used. Providing such examples is not easy: accounts of evaluations that ‘fail’ are rarely published, and when we have examples from our own practice, confidentiality – or embarrassment – can make it hard to talk about these in public. 

 

Slipping into simplicity

Slipping into Simplicity

By Pete Barbrook-Johnson, (UKRI Innovation Fellow and Research Fellow, University of Surrey, @bapeterj) and Alexandra Penn (Senior Research Fellow, University of Surrey, @DrAlexPenn)

24th June 2019   

 

We and our colleagues often talk a good game when it comes to complexity. We want to understand nuance and context, and we know things are messier than we realise. Yet, we still struggle to grapple with complex issues and can feel frustrated with our lack of progress, or others’ seemingly foggy thinking. We groan as we think we see yet another slick report, high-profile journal article, or politician, being openly simple and linear in their thinking, and proud of it too! 

 

How to evaluate complex research impact

How to evaluate complex research impact

From 'fasttrackimpact.com' 18th April 2019

By Pete Barbrook-Johnson, Innovation Fellow (Research Fellow) (University of Surrey) 

 

We know research impact unfolds in complex and unpredictable ways, so how on earth do we learn from and evaluate it? In this blog, I will take a look at some of the approaches we have been developing and using in CECAN - a research centre set up to tackle the issue of complexity in evaluation. I will explain how you can use these approaches to do a quick and effective evaluation of complex research impacts, helping you understand what works and why. 

 

Complexity and Evaluative Thinking

Complexity & Evaluative Thinking

By Dione Hills, CECAN Fellow (evaluating the ‘capacity building’ side of CECAN’s work) 

21st January 2019

 

There has recently been an upsurge of interest in what constitutes ‘Evaluative Thinking’ (ET). One frequently quoted definition of this term (Buckley 2015) refers to ET being “critical thinking applied to contexts of evaluation”[i]. This blog reflects on ways in which ‘evaluative thinking’ and the application of an understanding of complexity to evaluation can be mutually supportive. It also considers ways in which a complexity perspective may need to be enhanced in order to promote true ‘evaluative thinking’.

 

A recent issue of New Directions for Evaluation(Official Journal of the American Evaluation Association)[ii]brought together a number of papers on this topic. An interesting theme running through these papers (of particularly relevance to those of us involved in CECAN activities) is that evaluative thinking can help evaluators (and their clients) think more clearly about complexity, uncertainty and ambiguity. 

What has economic pluralism got to do with understanding complexity?

PEP Logo

By Teresa Linzner, Research and Communications Officer, Promoting Economic Pluralism

17th December 2018

 

Please note, the views and opinions expressed in this blog are those of the authors and do not necessarily reflect the views of CECAN.

 

To understand the complex systems we inhabit, surely, we must recognise a plurality of perspectives. After all everyone gets a different view. However mainstream economists, the dominant social scientists, only accept one perspective on how economies work drowning out others. Unless this is challenged, new understandings of complex systems will remain marginalised in policy and practice. Outside the mainstream, though, there are many around the world who understand the importance of plural perspectives in economics. Join us in working with them to raise their legitimacy and profile.

 

The Story of the Pluralistic Evaluation Framework

Pluralistic Framework

By Richard Gunton, CECAN Fellow

19th November 2018

 

The pluralistic evaluation framework is a new tool for policymakers that has gradually taken shape during the last 12 months of my CECAN fellowship. It is now ready to be presented at a webinar on 20 November, where I will be explaining the rationale in the space of 45 minutes. Here I want to share a little of the journey that it has been on, building on what I wrote here last January and June

 

Power and Hope: Using Evaluation in Volatile Contexts

Migration

By Caroline Oliver, Reader in Sociology, Roehampton University and Research Affiliate at the Centre on Migration, Policy & Society, University of Oxford

24th October 2018

 

As a social scientist, I find myself increasingly caught up in the ‘impact’ agenda, since universities are audited in the Research Excellence Framework on how the work of researchers has economic and social benefits. While there is merit in looking critically at this agenda (Holmwood 2011) it also provokes researchers to actively imagine a role for the social sciences in influencing social policy and practice, informing social reform through using robust evidence and social theory (Pawson 2000; Reed and Chubb 2018). The need for this is great; as Brian Castellani put it in his recent blog, we are witnessing an increasingly widespread ‘culture of cruelty’ and backlash against the establishment of a more just and equitable civil society. Engaged governance and just social policies become important means of challenging negative psychologies based on escalating fears, conflicts and resentments, while knowledge from research and evaluation provides vital ingredients in the search for something different.

 

The Nexus: A New Approach to Sustainability Transformations – What, Why and How

Pollution photo

By Adam Hejnowicz (CECAN, University of York), Pete Barbrook-Johnson (CECAN, University of Surrey), Kirsty Blackstock (The James Hutton Institute), and Chad Staddon (University of West England).

12th September 2018

This blog reports on recent presentations and discussions held at the Royal Geographical Society annual conference in Cardiff, at a special session on adaptive management and governance of the food-energy-water-environment nexus. The session was jointly organised by researchers from CECAN and the James Hutton Institute.

 

What is a Nexus Approach?

A Pluralistic Evaluation Framework

Richard Gunton
 
Richard Gunton, CECAN Fellow
 
I wrote at the start of this year about my work on articulating values in evaluation.  This has crystallised into a pluralistic evaluation framework, which I will shortly be presenting to policymakers from Defra and other agencies (in London on 12 June and in York on 3 July).  This workshop is intended to offer fresh ideas and a simple template to assist at all stages of the policy cycle - from appraisal and impact assessment through to full evaluation.
 

Value pluralism is the technical term for an ethical approach that recognises multiple kinds of values that cannot be collapsed down to a single value.

Looking Through a Glass Darkly: Public Goods and Agricultural Policy

Brexit & Environment

From Brexit & Environment, 9th March 2018

By Dr Adam Hejnowicz and Prof. Sue Hartley

This blogpost highlights some of the points raised in a policy brief, New Directions: A public goods approach to agricultural policy post-Brexit.

Following the Brexit referendum, the mantra “public monies for public goods” has been increasingly heard, especially in relation to agricultural and environmental policy. This public goods agenda represents, in part, a reaction against the well described failings of the Common Agricultural Policy (CAP), which despite greening reforms continues to oversee declines in European environmental quality.

Humility and courage: strategy and evidence in our complex world

Humility and courage

Stuart Astill, IOD Parc

Here in CECAN, a lot of time is spent considering and analysing policies, projects and interventions. We also think about methodologies, approaches and systems that we can adopt to do this analysis in the best way possible.

At the same time, we all recognise that there is a particular way of thinking that is valuable when working in the complex world – and certainly other ways of thinking that can trip you up quickly.

It is clear, if we take a sideways look at much of what CECAN does, that having the mindset of an explorer, a navigator, an illuminator, a learner and a satisfier is valuable.

Trying to be a ‘finisher’ or seeker of certainty would be inadvisable; being a ‘taker’ of ‘results’ rather than a learner would be foolish and to be a ‘maximiser’ would lead to a lifetime spent collecting information that still turned out to be incomplete. By definition.

Dynamic Pattern Synthesis for Modeling Complex Systems. An Interview with Phil Haynes

complexity

Courtesy of Brian Castellani's 'Sociology & Complexity Science' Blog.

Phil Haynes is Professor of Public Policy and researches and teaches public policy and management, as applied to a variety of contemporary circumstances. His research focuses on the application of complex systems theory, often using applied statistical methods. His research has been funded by the ESRC and the government and voluntary sector. He has published in a wider variety of journals including Social Policy and Administration and Public Management Review.  He is author of several books including Managing Complexity in the Public Services (2015) now in its second edition.

His most recent book, which is part of our complexity in social sciences series at Routledge, is aptly titled, SOCIAL SYNTHESIS: Finding Dynamic Patterns in Complex Social Systems.