Dr. Clare Twigger-Ross, Collingwood Environmental Planning
I first heard of Qualitative Comparative Analysis from Prof David Byrne at an early CECAN meeting and it seemed to address key issues we faced when evaluating 13 very different interventions across England designed to improve community resilience to flooding. That project was for Defra from 2013 – 2015 (report can be found here) and was called the Flood Resilience Community Pathfinder. The evaluation focussed on unpacking the concept of community resilience to flooding in order to provide an assessment of its improvement through the period of the project in 13 very different places. Each had their own configurations of institutions, actors and events that were integral to the success or failure of their different approaches and as such need to be examined as part of the analysis rather than factored out as “noise”. Also, what was clear at the end of the two years was that different factors needed to be combined in different ways for similar outcomes to be achieved. What was lacking was an approach to enable a more systematic analysis of those factors. Finally, many of the proposed outcomes of the pathfinders did not come to fruition during the period of the project, since the time period was too short, and for many the real test of improved resilience would not come until the next flood.
Given this, my fellowship is focussed on learning a new method: QCA, re-analysing data collected during the pathfinder project and collecting new data by returning to the pathfinders to find out what has worked well and what has not worked well and the circumstances surrounding those successes and failures.
One of the strengths of QCA is that it is a method which embraces complexity and the heterogeneity of the real world context, and is helpful for addressing “wicked issues” (that is, persistent highly complex problems that resist ‘silo’ approaches to policy solutions). It moves away from a linear approach to understanding change and acknowledges that a configuration of factors contribute to change occurring. By using a more nuanced approach, QCA identifies multiple paths and combinations of ‘ingredients’ which are different but equally relevant for a successful outcome. This is exactly what is needed for the pathfinder projects – understanding how in one place it might be the combination of an institutional champion, community engagement expertise, a persistent flood issue, local social networks that combine to lead to sustained activity by a flood group whereas in another place it could be the combination of a local champion, a persistent flood issue, local social networks, and links with the local council that lead to sustained activity.
So how has that worked in practice so far? In terms of analysis to begin with a spreadsheet with numerous variables all of which were known to have an impact on successful outcomes has been developed. But there was the first hurdle, how to define success? And it needed to be subtle enough to show failure as well, since for QCA to be useful it defines what configurations of factors go together for success and what go together for failure. With colleagues Paula Orr (CEP) and Stacy Sharman (Defra) our outcome variables have been refined to focus on whether specific flood groups set up under the pathfinder were still active two years after the end of the pathfinder. This is one of the outcomes we are looking at, but encouraged by our QCA advisor Prof Dave Byrne (Durham University) we will be adding to that with other indicators of success.
From August – October we have been visiting the pathfinders and conducting in depth interviews with pathfinder project managers (if they are still in post) together with members of community groups. Returning to interventions some years after they have finished is not something that is done very often, but we have been met with much enthusiasm from those who were involved in the pathfinders and it has been a real privilege for us to hear what has and hasn’t happened: as a result of, catalysed by, and alongside the pathfinder projects. Seeing what has flourished and what has not since 2015 is fascinating and we are collecting invaluable data on the processes of resilience building by communities largely in collaboration with more formal organisations. We have plunged into the complexity of context, organisations, communities and floods with narratives of success, reductions in flood risk as a result of local knowledges, new networks, new ways of working and new activities and commitment. Also of failures where things have not worked or have just finished. Sometimes in all the talk of evaluation, complexity and methods it can be easy to forget the pleasure and privilege of hearing about real lives and the possibility of transformation.
The next challenge is to take this rich, nuanced data and using QCA systematise it and analyse it so that it can be more than the sum of 13 stories. We need to do that without losing the subtleties, complexity and voices of those engaged in the community resilience processes. So watch this space!