Contact:
Amy Proctor, Newcastle University
CECAN Case Study: Enforcement on Waste Crime – Environment Agency
Between 2016 and 2018, the CECAN case study team, led by Newcastle University, worked with the Environment Agency to develop a new approach to their evaluation of waste crime. This involved a sustained and ongoing programme of advice and support from CECAN experts, who helped to guide the work at key stages, advising on the Agency’s methodology and acting as a critical friend. In particular, two bespoke activities were developed with the Agency which provided opportunities to co-design and co-develop a practical methodology and evaluation framework with CECAN experts in a highly participative way.
Why is waste crime an important issue?
- Waste crime is estimated to cost the UK economy between £300m and £800m per annum. It has been referred to as ‘the new narcotics’ – a way of making quick, community damaging profit.
- It is a multi-faceted problem and includes a raft of activities including the operation of illegal waste sites (unpermitted sites taking multiple loads of waste), illegal dumping (large scale one off illegal disposal), illegal waste exports, and the misdescription of waste in order to avoid Landfill Tax.
- Waste crime is important because it has serious impacts on people and the environment and also on the economy and legitimate industry. Illegal waste sites can blight local communities, whilst illegal export can lead to far flung human and environmental impacts as waste is mis-handled at a final overseas destination.
- The Environment Agency typically spends around £15m per annum on waste crime enforcement.
Why was the evaluation challenging?
The evaluation was challenging because:
- Existing approaches to evaluation of interventions on waste crime were seen to be too narrowly econometric. The Environment Agency had information on the overall Benefit : Cost ratio of its enforcement work, but wanted evidence that it could use at a more detailed level to help improve its operational effectiveness.
- These crimes involve a large number of variables, often interacting with the additional complexity of links to organised crime. Interventions are often used in combination or sequence so understanding the unique contribution of each and multiple outcomes had proven difficult for the Agency.
What activities were employed?
CECAN worked with the Environment Agency to help them explore a more behavioural approach to their evaluation in order to capture more fully the complex nature of the problem and help understand the impact of their interventions.
CECAN Activity 1: Methods scoping workshop
In November 2016, a scoping workshop brought together CECAN practitioners from Risk Solutions and the Tavistock Institute and personnel from across the Agency’s Evaluation, Prevention and Disruption and Enforcement teams. CECAN team members:
- Listened to Agency staff outline the evaluation problems and challenges they faced and discussed at length possible approaches.
- Proposed a methodology combining systems mapping, realist approaches and Qualitative Comparative Analysis (QCA).
- Found that the Environment Agency was particularly keen on the use of QCA as a way to bridge the gap between their need for quantitative evidence and rich and detailed qualitative description.
What is Qualitative Comparative Analysis (QCA)?
QCA:
- Is a methodological approach which allows the user to make systematic comparisons across cases in order to explore what causes differences between the characters of cases.
- Starts from a careful qualitative consideration of each case when the number of cases is small enough for this to be done.
- Moves into a quantitative mode by describing each case in relation to a set of attributes which are considered to be of significance.
- In this case was used to conduct a systematic assessment of the effectiveness of waste crime interventions across different cases to understand and determine success factors.
CECAN Activity 2: QCA design workshop
The Agency specifically wanted to explore critical success factors across their range of interventions on waste crime, including what works well, in what contexts and how different interventions work in combination.
In March 2017, a one-day design workshop:
- Brought together CECAN experts in QCA from Durham University and Surrey University and Environment Agency personnel from their evaluation, prevention and disruption and enforcement teams, as well as their economist, plus representatives from the Welsh and Northern Irish counterpart agencies.
- Gave participants an introduction to complexity thinking and the application of QCA.
- Used facilitated groups to define outcomes and causal factors to feed into a QCA model.
- Helped the Environment Agency to design a data collection protocol which they can use to build an evidence base for a QCA analysis.
How is the Environment Agency developing this work for the future?
The Environment Agency are now taking forward this work within the organisation, and refining their data collection processes to enable them to use QCA for the first time as part of their evaluation toolkit for assessing their waste crime interventions. This work has included:
- An initial exercise harvesting data about interventions used, and site context from internal databases
- A questionnaire survey of some of our enforcement staff to provide supplementary information
- Initial QCA analysis
Having piloted the method, the Environment Agency is about to embark on more detailed data collection to allow the method to perform more effectively.
What learning and insights did this case study produce?
There were a number of positive outcomes including:
- The independent, external advice the CECAN case study team provided gave members of the Environment Agency’s Research, Assessment and Evaluation team the confidence to explore alternative evaluation methods.
- The scoping workshop provided a safe space for the Agency to discuss and consider new ideas and approaches to evaluation, guided by experienced practitioners with expertise in their application.
- The design workshop enabled the Agency to understand the QCA method in more detail and begin to test its application, in a highly participative way, within their specific context.
- The design workshop setting encouraged expertise exchange between CECAN and the Agency and also between the operational and strategic teams within the Agency and between the different devolved agencies, leading to a more nuanced understanding of the evaluation challenges and how to tackle them.
What are the implications for future policy evaluation?
The project has changed future evaluation of approaches to waste crime because:
- The Environment Agency’s work with CECAN on the evaluation of waste crime is underpinning a refreshed approach to evaluation of regulatory activity across the wider remit of the Agency.
- There is the potential for the lessons the Environment Agency has learnt from this experimental work to feed into evaluations of waste crime in the future in Wales, Scotland and Northern Ireland.