Likelihoods - by Stuart Astill, IOD Parc

Blog by Stuart Astill IOD Parc

By Stuart Astill, Principal Consultant, IOD PARC


The scope of the CECAN project runs wide as well as deep; complexity in the energy, water, environment and food domain would most immediately be thought to arise from the physical systems at the nexus core. Yes, complexity in weather systems, biological populations spring quickly and easily to mind as do the ‘user level’ human interactions with these systems and other local and global physical systems. However, when we are addressing remedial interventions, or even ongoing management of nexus concerns, we quickly realise a double dose of complexity if, as we certainly should, we look towards governance.

We see just as clearly that complex systems feature in governance, with multiple stakeholders, competing objectives, uncertainty over outcomes and organisational/institutional (rule based) and human interactions. When such a governance system, already complex, has as its object of interest a complex physical system then things are doubly tricky. Perhaps complex-times-complex, or complex-squared, is the same thing mathematically – but when we are trying to understand the system in the real world we can’t get away with definitional mathematical tricks!

One approach to help us along the path towards talking meaningfully about complexity is the language of likelihood. For example, often discussed is the role of climate change in increasing the expected number of extreme weather events in any given period (1); this is the same thing as saying there is an increase in the likelihood of an extreme weather event occurring.

The language and notion of likelihoods is vital when we talk about such issues. Without having to say that we will see a given number of extreme events this year when we would previously have only seen fewer, we can instead say that the likelihood of an extreme event is some factor higher than it was before.

This, in itself, is something that some who work in public policy may take a bit of getting used to. Traditional policy interventions would be expected to be able to claim a reduction in the number of bad things happening, or the number of good things happening increased. Definitively.

For many people it is a considerable step to start talking about “the likelihood of something bad happening has fallen” (they know as well as we do it may not have happened anyway!) It is an even bigger step to spend money on reducing the likelihood of something bad happening. Loosening the purse strings of government when all we can promise is shifting the estimated chance of a thing happening is tough: the estimates of likelihoods can only ever be that. But the money is real…

And then we come onto governance. In the governance world of the nexus it is often the case that improved governance is a critical component of changing the world for the better. However, the governance structures around nexus issues tend towards multi-stakeholder (possibly international) and conflicting or overlapping objectives as well as mixing in technical, administrative, community and political stakeholders.

It is nearly always true, in any field that matters, that effective governance emerges best through support and is achieved often by small, but difficult, steps forward that may not happen quickly. The support required could be called an ‘intervention’ and the intervention requires resources, which can often be, and will often be called, money.

And so, around governance, we start to perceive a mirror of the way we were thinking about the physical nexus system itself. Interventions to improve governance only have a ‘likelihood’ of making a breakthrough – will these people sit down together and work effectively if we support them with programmes/resources/knowledge of the type A, B and C? Clearly, we cannot say: all we can do is estimate the increase in the likelihood that this will happen within a given timeframe.

Exactly as with the extreme weather events, we are not able to tell those who oversee public policy whether we expect that they will get less of this, or more of that, as a countable outcome. Instead we have to talk to about a shift in the likelihood – that’s what they will get for their money. As before we are all very aware that the event/outcome at the centre of the conversation is still, after all the work, something which still may not happen at all, or might have happened anyway.

We shape our evaluation to assess how much (if at all) the odds shift in favour of the outcome we are looking for. We can have a discussion about, and give our best estimates on, the changed odds, but cannot say for sure that any particular horse will or will not win the race.

All of the discussion above about likelihoods in systems and governance is true even in the absence of complexity – however in the face of real complexity the estimates become less precise, the caveats around our pronouncements become greater and the uncertainty starts to compete for the upper hand with certainty.

This is an opportunity as well as a challenge for good complexity-aware evaluators. It is true of all modelling that the exploratory adventure of building, specifying and populating a model * is where the greater part of learning about the question lies, rather than in the answer that pops out of the model  (which it never does anyway, it must always be interpreted).

We know that the process that we go through with our clients and their stakeholders is where much of the value lies. So when evaluators and the evaluated can accept the nature of complex problems and learn from the journey, as much or more than from the ‘answer’ then evaluation becomes the valuable activity that it was deserves to be.

Perhaps in this cross-fertilisation we find the essence of CECAN – exchanging knowledge in the realm of complex governance of a complex issue: methodological innovation, higher awareness and the broader aim of meaningful and fruitful interaction between those carrying out evaluation and the public policy practitioners who would learn from it.

1. Link to The Guardian


* When I use the word model, I do not limit myself to talking about formal mathematical or economic models but any qualitative or quantitative approach that we may use to simplify the vast reality of the world in order to draw generalised conclusions.

CECAN - funded and supported by the following organisations