Annual Report

One of the key insights from CECAN’s first phase was that complexity appropriate methods have value and should be used across the policy cycle in policy development and appraisal, not just in evaluation. Complexity also requires evaluation to be conducted earlier in the policy process with more frequent cycles of ongoing data collection and action being used to inform policy implementation.

Centre for Connected and Autonomous Vehicles (CCAV) case study: a systems map of ‘Mobility as a Service’ across the policy cycle, covering policy design and evaluation

We have raised awareness of CECAN and complexity appropriate methods of analysis across Government and beyond our original partners (i.e., the Department for Environment, Food and Rural Affairs (Defra) and the Department for Business, Energy and Industrial Strategy (BEIS)). We receive regular requests for assistance in policy development and evaluation. With the BEIS / Department for Transport (DfT) Centre for Connected and Autonomous Vehicles (CCAV), we have developed a new case study to conduct a systems mapping of ‘Mobility as a Service’ across the policy cycle, covering policy design and evaluation. The CCAV case study outputs will feed directly into a regulatory review and consultation on the Future of Mobility. This work is part of the wider Industrial Strategy: a key area for CECAN to build expertise and awareness of its work, and is further described in CECAN Evaluation Policy and Practice Note (EPPN) No. 18.

CECAN’s Complexity Evaluation Framework for Defra, January 2020

In the first half of 2019, CECAN was commissioned (via its spin out company, CECAN Ltd) to develop a Complexity Evaluation Framework (CEF) for Defra. The framework is intended to guide analysts and policy-makers across Defra in ensuring complexity thinking is appropriately embedded into evaluation with the aim of producing robust evaluation evidence to inform policy. In January 2020, following a process of a peer review, Defra published the CEF and subsequently commissioned an independent evaluation of it. This evaluation was undertaken by Steer-ED Consultancy, with positive findings.

Publication of CECAN’s annex on complexity in HM Treasury’s Magenta Book, March 2020

CECAN’s guidance, Handling Complexity in Policy Evaluation, in HM Treasury’s Magenta Book (the central guidance for all UK government departments on evaluation) was published in March 2020. More than 200 policy analysts, evaluation professionals and other stakeholders participated in the interactive launch events that CECAN ran in London, Birmingham and online through March and April 2020. CECAN has plans for a series of future engagement activities that will further promote and support the use of the guidance.

CECAN Special Issue for Evaluation journal, January 2021

CECAN was invited to compile a Special Issue for the journal Evaluation (Sage) and this was published in January 2021. The CECAN Special Issue, ‘Policy Evaluation for a Complex World: practical methods and reflections from the UK Centre for the Evaluation of Complexity Across the Nexus’ consists of six papers plus an editorial, which draw not only on our case study work but also on our broader learnings, e.g. reflections on developing our guidance for HM Treasury’s Magenta Book, our work on Systems-based Theory of Change, and the application of our COMPLEX-IT app.

The case studies to date and methods used within them have had a significant impact on policy-makers and analysts in Government and have informed the development of major areas of policy in conceptual, methodological and practical terms. Publication of CECAN’s guidance, Handling Complexity in Policy Evaluation, in HM Treasury’s Magenta Book and related resources on methods on the CECAN website combined with the capacity building activities of Workstream 4 are likely to make a significant further contribution to the awareness and use of complexity appropriate methods in Government and the wider evaluation community.

Participatory Systems Mapping

Our Participatory Systems Mapping approach – developed and used in the first phase of CECAN case studies – has continued to grow in popularity, leaving a lasting impression on those who engage with it. Through CECAN Ltd (CECAN’s spin-out company that was launched in Summer 2018) we have worked with Mott Macdonald on a Systems Analysis of Water Resources project funded by Defra. The project used CECAN’s Participatory Systems Mapping approach to explore how whole systems and complexity-appropriate approaches can be used to provide insight on, and inform management of, river catchments. Defra described CECAN as “pivotal to the success of the project” and Mott Macdonald requested repeat engagement with CECAN when they were commissioned to work on a different project with Water Resources South East. Through ongoing engagement with government and industry partners, CECAN is becoming established as ‘the place to go’ for providing technical advice in complex evaluation.

CECAN has also been addressing the broader challenge of turning systems maps into something actionable for evaluation purposes. A team from CECAN – including complexity and system mapping experts and evaluation practitioners – has developed a methodology that plots a path through from systems mapping to a Theory of Change (ToC), which evaluators can use to frame their evaluations. This allows evaluators to develop practical ToC maps that take into account feedbacks, wider context, and potential negative or unexpected outcomes. A paper has been included in the Special Edition of the journal Evaluation. The approach has generated particular interest with the team at the Medical Research Council and Scottish Government Chief Scientist Office, Social and Public Health Sciences Unit (SPHSU)at the University of Glasgow. CECAN is developing a collaboration with SPHSU and we are exploring a range of topics of mutual interest, including system based ToCs and innovative tools designed to promote collaboration on mapping, understanding and communicating complex systems. Via CECAN Ltd, we are also working with consultancy partner Technopolis to develop a Theory of Change tool for Defra.

A book about Participatory Systems Mapping will be forthcoming in 2021, authored by CECAN researchers, Dr Alex Penn and Dr Pete Barbrook-Johnson. The book will explore a range of new and old systems mapping methods, focussing on practical insights for mapping in real world contexts. Each method covered in the book will be accompanied by a podcast and a detailed guide on the realities and challenges of building and using systems maps.

New and emerging methods

CECAN is looking to bring a wider range of methods into our work with a focus on further innovations in systems mapping approaches, exploring ‘rapid methods’ suitable for addressing complexity quickly and for non-experts. In addition, CECAN has been developing a relationship with the Alan Turing Institute, looking at work on Digital Twins. Work in this area is also underway with CECAN researchers based at UCL.

Tools

CECAN has developed a number of practical tools to facilitate uptake of our methods. They are all open source and free to run online or download. They have all been successfully used in real-world situations.

COMPLEX-IT

In November 2019, we launched our COMPLEX-IT software. Developed by CECAN researchers at Durham and Surrey, COMPLEX-IT is designed to increase access to the tools of computational social science (e.g., artificial intelligence, micro-simulation, predictive analytics) through its visually intuitive learning environment. The Covid-19 Community Health and Social Care Modelling Team at Durham University were able to use the COMPLEX-IT software to create a model that provided weekly reports on Covid-19 for the local authorities in the North East of England to support decisions and planning accordingly.

PRSM

We have also developed a Participatory Systems Mapping app, PRSM, which enables groups of people, each using their own computer (or tablet) to collaborate in the drawing of a system map (a network of factor nodes and presumed causal links). The app includes a built-in chat feature and every edit is broadcast to all the other participants in real time. Development of the PRSM app has drawn on CECAN work at Warwick on trophic analysis (quantifying how far up or downstream each node is to indicate the extent to which the network fails to line up). The PRSM app is being used in our work with the Health and Safety Executive developing a map, with other stakeholders, of how a wide range of workplace and other factors influence mental health. The great strength of the app for this application is the ability it provides for groups to work collaboratively on maps in virtual workshops.

“Choosing Appropriate Evaluation Methods” tool

CECAN Researcher, Dr Barbara Befani, has updated the “Choosing Appropriate Evaluation Methods” tool that she originally developed in 2016 with Bond Consultancy. The tool is an accessible aid to help users to understand evaluation methods and choose the right ones for their purposes. It has already proven popular in CECAN’s case study work.

Through most of 2018-19, we continued to work on existing CECAN case studies (e.g. with BEIS (Renewable Heat Incentive), Defra (Rural Development Programme England) and the Food Standards Agency) while also developing new ones, e.g. with the Centre for Connected and Autonomous Vehicles (DfT/BEIS). As of 2020, CECAN has largely completed its case study work, the findings from which continue to be presented via a range of mechanisms including the publication of reports and conference presentations. For example, in June 2020 CECAN researcher, Dr Adam Hejnowicz, gave an invited presentation describing CECAN’s case study work at the York Environmental Sustainability Institute (YESI)’s Summer Social Programme: “Opportunities for transformation: Bridging divides at the science-policy interface”.

Case studies with Defra

CECAN has been engaged at both a local and national level in Defra’s Tests and Trials (T&T) projects to develop and inform policy under the Environmental Land Management Scheme (ELMS). CECAN researchers at Newcastle’s Centre for Rural Economy are involved in the Curlew Contracts T&T with Northumberland National Park, acting as a ‘critical friend’, evaluating activity and processes, offering internal challenge, drawing together lessons, and reporting to Defra. Through CECAN Ltd, CECAN is working in collaboration with ICF Consultancy to advise on complexity appropriate methods that could be applied in the national evaluation. The impact of our work with Defra through our involvement with the ELMS programme is to bolster CECAN’s role as a ‘critical friend’ rather than a consultancy partner. We have strengthened our identity as an advocate for complexity thinking and approaches in policy making. We have also learned about the importance of building not only capacity but complexity competence within stakeholder groups.

The use of the CECAN approach to Participatory Systems Mapping has spread well beyond the initial future farming / rural development case study group at Defra. A joint CECAN/Defra report capturing lessons and impacts from this work was published internally at Defra in December 2019 (Penn, A.S, Jones, C. et al (2019) Understanding Complexity in the Rural System in England: Analysis and Insights from the FF&C Systems Mapping Project, CECAN & Department for Environment, Food and Rural Affairs. UK Government Internal Report) and this has been widely circulated across Defra by the Future Farming team. Nonetheless, converting this into academic publications is difficult as the report contains programme information that is highly classified.

Case studies with BEIS

CECAN was authorised to publish a report about our Energy Trilemma case study with BEIS (2018) and a subsequent paper based on our two BEIS case studies – ‘Participatory systems mapping in energy evaluation’ – is included in the CECAN Special Issue in Evaluation. Our most recent case study culminated in a final report that was submitted in August 2020 to the Centre for Connected and Autonomous Vehicles (CCAV) and a CECAN Evaluation Policy and Practice Note (EPPN).

Other case studies / new areas of work

In developing new case studies, we have wanted to ensure that they use approaches that we have not used to date or where there is a clear opportunity for development of an existing approach. For example, there is a clear demand for further case studies based on systems mapping approaches. However, we have resisted developing these unless there is an opportunity for methodological development of contextual novelty, as with the CCAV case study.

CECAN’s case study work, particularly with Defra, continues to be extremely well valued by stakeholders. Stemming from our systems mapping work with Defra’s rural development team, CECAN researcher Dr Alex Penn was invited by the head of evidence at the Marine Management Organisation to give a keynote presentation at the Defra MOOSC conference (an internal conference for Defra and all arm’s length bodies involved in Marine policy) in early November 2020. Her presentation covered our use of complexity approaches in a large, complex, rapidly changing policy area to inspire those working in the Marine policy area, who are keen to use our approaches as they tackle the challenges presented by Brexit. This is a great endorsement of CECAN’s work and presents a strategic opportunity for us to open up work in Defra Marine and to do something of a similar scale as happened in Future Farming. CECAN is eager to capitalise on the high-level endorsement of and clear appetite for our work that we have been creating but the lack of funding available for our future will present a challenging obstacle.

CECAN Ltd consultancy

CECAN continues to engage heavily in real-world testing and innovation, aided by its ability to bid for government contracts through CECAN Ltd. Examples of this include our Water Systems Mapping work at Defra mentioned previously; our work with Technopolis Ltd to evaluate UKRI’s Strategic Priorities Fund; our evaluation of the Forces Connect South East Veterans’ Hub Project for Surrey County Council; our work with the Food Standards Agency to design and apply a methodology to compare and evaluate the impact of their research outputs; our work with the Climate Change Committee on the interacting risks of climate change; and our work with the Health and Safety Executive to explore how stakeholders can work together to improve mental health in the workplace. Further information about CECAN Ltd’s work is available here.

Workshops and seminars

Our programme of workshops, seminars and webinars is intended to promote the work and outputs from CECAN along with the work of related researchers and evaluators, and build capacity to use complexity appropriate approaches. Our webinars regularly reach audiences of 100 or more with many participants joining from outside the UK. In May 2019, CECAN researchers were filmed by SAGE discussing their research on complex systems using participatory systems mapping and the film is included in the SAGE Research Methods website. We held a CECAN showcase for BEIS analysts and researchers in October 2019 to raise awareness of complexity concepts, methods and practices. Feedback from our seminars, webinars and workshops demonstrates that these are received positively. This is further illustrated by requests for repeat engagement, for example the Department for Education (DfE) requested CECAN to provide another bespoke training course, on a new subject, in November 2019. CECAN also provided a one-day bespoke training session for the £5m SIPHER project team: an ambitious systems science project focussed on public health issues at different geographical scales in the UK, funded by the UK Prevention Research Partnership.

CECAN Ltd training provision

CECAN Ltd has facilitated the further development of CECAN’s bespoke Continuing Professional Development (CPD) training provision, enabling CECAN to offer organisations a more cost effective and focused short course delivery method. For example, in 2019 the company provided a two-day course, ‘Advanced Evaluation’, to over 20 Department for Education (DfE) delegates at a location of their choice. Based on CECAN’s Syllabus (2017), the course was tailored to the requirements of the DfE and received extremely positive feedback. CECAN Ltd was later commissioned to deliver bespoke training in complex evaluation to Public Health England between November 2019 and March 2020. CECAN has also continued working with the UK Evaluation Society – via CECAN Ltd – to develop and deliver a CPD Masterclass series. Due to its sell-out success, the series was extended beyond its initial scope.

CECAN PhD studentships and Fellowships

CECAN’s PhD students continue their projects building their own and wider partners’ knowledge and capacity to deal with complexity. Current projects include:

• “Interfacing scientific data and complexity policy” (Lisa Stoltz, Warwick)
• “Building policy-makers’ capacity to evaluate complexity” (Anna Kaxira, Westminster)

Completed projects include:

• “A case study of the Marine Pioneer: investigating local management and natural capital valuation” (Betheney Wills, Surrey): Betheney was awarded a PhD in November 2020 following a viva in which her thesis was approved without corrections. Her doctoral research was a case study of the Marine Pioneer, a UK government pilot testing natural resource governance and management principles from the 25-Year Environment Plan. The aim of the research was to investigate the effects of testing local scale governance alongside the experimental implementation of the Natural Capital Approach in the Marine Pioneer. Betheney has since been successful in securing the position of Marine and Fisheries Senior Research Officer in the Social Science Team at Defra.

Following on from the funded Fellowship programme in the first phase of CECAN, we have been able to support three ‘no cost’ or alternatively funded Fellowships, and these are:

• “Evaluating the impact of adaptation measures and resilience building by examining the complexity nexus using a systems-thinking complexity approach” (November 2018 – January 2019, funded by DFID, via IOD Parc)
• “Evaluating the Potential for Natural Capital Investment to Reverse Soil Degradation: A Dynamic Simulation Approach Exploring Connections between Soil Health and Money in England” (from February 2019, No-cost Fellowship)
• “How do organizations learn from evaluations? Highlighting the role of social complexity” (from February 2019, No-cost Fellowship)

Although the Fellowship programme is now closed, outputs from the work continue to be produced. An example of this is CECAN Fellow, Dr Richard Gunton’s Pluralistic Evaluation Framework, which was presented at a joint meeting of the British Ecological Society journal, People and Nature, and the Valuing Nature Programme, in March 2020. Jonathan Nichols’ fellowship led to a presentation to Defra and a CECAN webinar, with the results from his work being written up in his Masters’ thesis: “Evaluating the potential for natural capital investment to reverse soil degradation: a dynamic simulation approach exploring connections between soil health and money in England”. Other publications arising from CECAN Fellowship projects include:

• Barbrook-Johnson, P. et al., (2020), How do policy evaluators understand complexity?, Evaluation, Vol. 26(3) 315–332. Sage. DOI: 1177/1356389020930053
• Kirsop-Taylor, N., Hejowicz, A. P., Scott, K., (2020), Four cultural narratives for managing social-ecological complexity in Public Natural Resource Management. Environmental Management 66, 419–434. Springer. DOI: 1007/s00267-020-01320-6
• Kirsop-Taylor, N., Hejowicz, A. P., (2020), Designing public agencies for 21st century water-energy-food nexus complexity: the case of Natural Resources Wales. Public Policy and Administration, 0(0) 1–21. Sage. DOI: 1177/0952076720921444

Formal guidance documents

Concepts, methods and frameworks for complexity appropriate evaluation are being incorporated into government guidance and practice, with CECAN taking a leading role in this. As previously mentioned, CECAN developed a Complexity Evaluation Framework (CEF) for Defra that was published in January 2020. The CEF has provided Defra with a clear and accessible description of what complexity is and why it matters to them, along with the rationale for why complexity-appropriate is crucial to help, and what this looks like. Furthermore, through interviews, workshops and presentations over the course of the CEF development project, the CEF work has contributed to capacity building in Defra and the Defra Group by raising awareness of and fluency in complexity concepts and complexity-appropriate evaluation. The CEF was used by Defra almost straight away to inform the early stages of an evaluation strategy for forthcoming evaluations in waste and resources. Other government departments have expressed interest in similar work tailored to their specific policy challenges.

CECAN’s work on producing guidance documents has been extremely well received. Anecdotal feedback from participants at our Magenta Book Guidance events confirms that users can really see the value of applying the guidance to their own work. With regard to Defra’s Complexity Evaluation Framework, the findings of Steer-ED Consultancy’s independent impact evaluation were published in October 2020 and conclude that: “Overall, there is no question that the main content of the CEF, in terms of the key considerations and questions, is proving useful and is already contributing to adoption of some complexity-aware evaluation practices in Defra.” and “Although still in early stages of use, there is already some evidence that the CEF is changing behaviour – for example by encouraging a more holistic or nuanced approach to evaluation, the adoption of complexity-appropriate methods, or more stakeholder-driven approaches.”

New areas of work

The diversity of requests made to CECAN to conduct work and the inclusion of complexity appropriate methods in tender specifications indicates that CECAN is having an impact on the practice of policy evaluation amongst commissioners, providers and users of evaluation across Government and beyond. New work funded by the charitable foundation Lankelly Chase is exploring the third sector’s ability and needs in conducting complexity appropriate and systems-based evaluation in the area of severe and multiple deprivation. This is broadening the remit of CECAN from Westminster and Whitehall policy making to the local delivery of services and the relationship between central policy and local service delivery.

CECAN researchers are engaged in a number of projects that draw on the expertise we have developed. Some examples of these are:

• ANTICIPATE: Actively anticipating the unintended consequences on air quality of future public policies (NERC, February 2019 – August 2022), Prof. Nigel Gilbert
• LIPSIT: Local Institutions, Productivity, Sustainability and Inclusivity Trade-offs (ESRC, October 2019 – September 2021), Prof. Nigel Gilbert
• “Implementing innovative legal and organisational structures for a just transition: a catchment management beacon project” (University of Surrey ESRC Impact Accelerator Account), Dr Pete Barbrook-Johnson
• PURE: Peri-Urban Resilient Ecosytems (GCRF Pump Priming Grant, January – July 2020, extended due to Covid-19 pandemic), Dr Adam Hejnowicz and Dr Richard Gunton
• SysRisk: Systemic environmental risk analysis for threats to UK recovery from COVID-19 (NERC, January -December 2021), Prof. Nigel Gilbert