The first of two blogs following this event, from the perspective of the lead facilitator, Dr Paul Brand
‘The Complexity in Evaluation Workshop: What we did and what we learned’
This was my first experience of facilitating for CECAN. I’d heard about it from colleagues who are involved and it all sounded fascinating, but perhaps a little ‘technical’ – possibly even a tad ‘dry’.
This event was publicised as:
“This 2 day residential workshop, conducted under the Chatham House Rule, will bring together evidence teams, policy makers, policy analysts, complexity scientists, evaluation experts and experts in Nexus subjects.”
Fascinating I thought, but not necessarily a hotbed of creativity and passion. How wrong I was!
The post event feedback included comments like:
‘A journey’, ‘worthwhile’, ‘enjoyable’, ‘dynamism’, ‘energy’ and ‘larding’ (the latter I will explain later).
As a facilitator I was privileged to see people from all the groups listed above, engaging with the discussion and interchange not just as a matter of academic or theoretical interest (though there was much of that) but with a sense of urgency. These 40-50 experts engaged because it matters that we find pathways, methods and approaches to evaluation that will serve our society better, by managing the true complexity of our world with open eyes.
So with that in mind, let’s look at
- What did we do at the workshop?
- What topics were discussed?
- What did we learn / achieve?
- What happens next?
WHAT WE DID and the FLOW
One of the key principles in the workshop design was to provide space for the participants to work in two very different mind-sets, ‘Exploration’ and ‘Building’. The workshop programme interleaved sessions targeting these two perspectives.
Exploration Sessions These allowed space for groups to explore ideas, learn from others and exchange experiences. Over the course of the programme, the level of pre-defined structure and content for these sessions reduced, starting from formal presentations at the start of day 1, and gradually allowing the topics to be defined by the participants as they emerged.
Building Sessions These sessions were designed to guide the participants to talk about practical solutions to key challenges and develop initial propositions of how CECAN could work to deliver those ideas. Working in 4 teams, the initial activities used a very open question to start the team debate, working towards a structured presentation of propositions by each team at the close of Day 2
What topics were discussed?
DAY 1
Exploration: Evaluation, Complexity and the relationship between them
Discussion groups were led by CECAN experts in complexity and evaluation using a method based on the World Café approach. (www.theworldcafe.com)
Building: Developing an evaluation timeline with complexity at its heart
A key lesson emerging here was that good evaluation, especially where the policy is addressing a complex problem, and may be complex in its own right, is the importance of planning for evaluation very early in the policy design process and of the key role formative evaluation can play in optimising policy intervention design and implementation.
DAY 2
Exploration: Discussion group proposed and led by participants
The discussions took place ‘Open Mic’ and ‘Open Space’ format (www.openspaceworld.org)
Four topic areas emerged:
- Complexity methods and approaches: including discussion of the value of specific methods and approaches such as cognitive task analysis and agent based modelling and methodological challenges – such as tackling uncertainty. It was agreed that is would be useful to collate and share best practice examples of evaluations tackling complexity regardless of the sector.
- Communication and capacity building: this remains a challenge how can complexity be rendered graspable, what vocabulary is most accessible. Can we make more use of narrative and metaphor? Is the language of risk more familiar and engaging than the language of complexity? Methods of building capacity and sharing knowledge that are accessible to busy people were discussed – including the use of IT solutions such as MOOCs
- Framing and process: in parallel to developing and communicating methods, participants recognised the need to establish complexity as a norm, and evaluation as a part of the holistic policy making process. Barriers to effective commissioning need to be removed and guidance on when and how to tackle complexity in evaluation developed and tested.
Some fundamental question about CECAN’s scope emerged from these discussions. What should CECAN be tackling?
- Just ex-post evaluations or also ex-ante (policy appraisal)
- How can we extend our activities effectively outside Westminster, and at what different scales (central vs. local government)
- How can we ensure NEXUS specific challenges remain front and centre, while also learning from other sectors and providing value more broadly?
Building: Solution development:
Topics chosen by the team for particular attention in the coming months included:
- Developing and testing, critically with the client audience, complexity friendly approaches and methods
- Developing guidance to help people decide when they need to consider complexity, what methods are appropriate and effective commissioning
- Methods for building competence and capacity, and
- The more fundamental need to raise the profile of complexity in both policy appraisal and evaluation and to ensure that these two activities are more closely integrated
The value and need to develop CECAN’s working method to include policy makers and analysts effectively in the centre’s on-going work was also recognised, as was the need for NEXUS considerations to be kept front and centre.
What did we learn / achieve?
The outputs from the workshop are still being processed by CECAN, but as facilitator I saw the following key topics emerging and repeating throughout the two days:
- The inherent need for approaches and methods and policy and decision making processes, to transform in parallel. Complexity is not confined only to the technical or managerial solution to the issue, but is present in the entire NEXUS system, which crosses departmental and geographic boundaries and operates at many levels: nationally and regionally, across urban and rural contexts, and through large industrial players, family farms and individual members of the public.Established processes for defining a problem, establishing solutions and evaluating their operation and effectiveness, appear unsuited to the breadth of engagement and fluid re-design of approaches required that is inherent in complex problems.
- The tension between the competing values of:
- Integrated, formative evaluation commissioned as part of the work of implementing and refining policy interventions as they are piloted and fully implemented
vs. - Independent, summative evaluation, commissioned as an unbiased view of the outcomes and impact of the work or policy.
- The power of policy makers’ and analysts’ presence in forums like this that bring together complexity and evaluation theorists and modelling and evaluation practitioners and the importance and power of face to face contact in building a community of interest around the CECAN core. CECAN have made a good start, beginning to build layers of engagement out into the wide range of organisations and networks who have experience and insight to bring to the discussion and those who will benefit from the outcomes.
What next?
The CECAN team will be digesting the outputs from the workshop in their October Project Meeting and are already cracking on with some of the activities. Most importantly they are launching into the case studies with the co-funding departments. These will provide one of the key mechanism for testing methodologies, approaches and processes.
Paul Brand – Coordinating Designer and Facilitator