Emma Uprichard and Robert MacKay, University of Warwick
Interviewed by Candice Howarth, University of Surrey
CECAN is exploring how evaluation of policy can better inform the impact those policies have and assess the extent to which these have been successful. In order to do this, access to data is crucial, yet can at times be problematic. CECAN’s Knowledge Integrator, Candice Howarth met Emma Uprichard and Robert MacKay from the Centre and based at the University of Warwick and asked them over a series of emails to explain what the implications of some of these challenges are.
Why is data needed in evaluations?
Evaluation of policy must be based on some form of evidence, for us that means data. Data don’t necessarily represent exactly what has happened or what is happening, but they certainly represent what David Byrne (2008) calls a ‘variate trace’ of what has happened. This means that policy decisions can be made based on something rather than nothing – the idea being that evidence in the form of data is more likely to provide better and more meaningful evaluations. For example, in order to evaluate whether policies on waste crime have been effective in particular areas, then having access to data about the kinds of waste crimes that might be on file, when and where and how these may or may not have changed over time would provide valuable insights to assess the success of policies. Of course, having data to evaluate policies doesn’t necessarily mean that all decisions based on that data will be correct – but one would hope they at least would be. It would be irresponsible to evaluate policies without knowledge of what has actually taken place, so there is also an ethical dimension to needing data in evaluations too.
How is data used?
Data should be used to evaluate the evidence that shows whether a specific policy achieved its desired effects; or if it didn’t, it may produce clues as to why it that is. An intermediate step is to evaluate the evidence that the actual effects were a result of the policy or would have happened anyway. Ultimately, data helps to show what has happened and this in turn helps to know whether a particular policy has worked or not, or which parts of the policy have been successful and which parts haven’t.
Can methods for evaluation be designed without data or without knowing what type of data is available?
Robert: I’d say no: the design of evaluation should say what data are required and what to do with them.
Emma: I’d say, it depends! To really test the method/s, data are needed. But it is possible, that the first stages of methodological development might be achieved without data; indeed, it may be a necessary part of the methodological development. The problem is, ‘methods for evaluation’ very much depend on what the data actually are. So, although hypothetical methods might be developed, knowing whether or not the methods are appropriate rests on actually accessing the data. It is also worth noting, that the methods may be different for the same data over time.
What are some of the challenges to accessing data?
There are a number of challenges, for example the data holder may not want to make it available or alternatively, the data holder may require a more or less lengthy memorandum of use detailing intellectual property rights; this may take any number of months; in some situations, even over a year. Once the data has been accessed it may be in a form that requires standardising for use by software, or it may be too big to transfer by internet, and may be accessible only on the premises.
Why might holders of data be reluctant to share it?
Again, there may be a number of reasons for this. One may be fear that you’d find out and reveal something adverse to their business or that you’d reveal information to competitors. Another, which has a legal dimension is that there are contractual issues to follow which may be out of the data holder and data analysts’ hands.
What are challenges with using multiple datasets?
There are a number of challenges associated with using data; when it comes to using multiple data sets there could be issues around making them compatible and how they are classified; for example how things are counted over time changes, so any longitudinal data even within the same dataset usually need some work to make the measures meaningful over time.
How are findings/analysis of data then disseminated?
It’s important to ensure findings based on analysis of data and evidence is communicated clearly and in a way that can effectively inform policy making. For example this might be in reduced form with some key messages or graphs, or journal articles and blogs (including video) may be most accessible. We’d often present some of our findings at conferences, talks and seminars where we’d use PowerPoint slides that can be circulated to various audiences
What suggestions do you have on overcoming challenges of accessing data?
We have a way to go before we can overcome many of these issues but there are a range of ways that can help address some of these. One may be to build in at the earliest stage of planning evaluation a requirement to provide specified data to specified bodies. You need to allow for a lot of time for negotiations to be made and all those involved need to be prepared to compromise! It’s important to be both persistent and patient to ensure the needs of all those involved are met