The CECAN team would like to extend thanks to Professor Trish Greenhalgh from the University of Oxford, who kindly dedicated her lunch hour to give us an energetic, thought provoking and highly logical seminar on evaluating e-health research.
Trish began by discussing what we mean by e-health research and that research of technologies that only focuses on ‘technologies which will change the world’, is aligned with a vague modernist vision, where technology is meant to solve many of society’s problems. Why? Because it does not consider the complex elements of peoples’ everyday lives.
Of course, e-health technologies have not changed the world, as highlighted by Tony Blair’s ‘National Programme for IT’. This £12.8billion programme was meant to improve transparency, efficiency and accountability within the health service. It didn’t happen and this became a paradigm case of inflated hopes and underestimated challenges in e-health. Trish was involved in evaluating the programme and fears that Jeremy Hunt is now going down a similar route, with a recent headline of ‘£4billion NHS tech investment’, suggesting the same lack of attention is being paid to the system elements in evaluation.
Trish described the research she has been carrying out into ‘smart homes’ including e-health technologies for health, wellness and assisted living. She was quick to point out that in reality nobody lives in a smart home or has room for social assistance robots and the biggest determinant of an individual’s health remains their social position, with poverty, poor housing, unsafe streets, ethnicity/culture, low health literacy, social exclusion and digital exclusion being additional contributory factors.
So, the vision of e-health does not match up with reality.
Through the case study of the ‘Whole Systems Demonstrator Programme’, a multi-million pound randomised control trial of telehealth and telecare, Trish explained how evaluations of such trials were biased because they could never capture the ‘whole’ system of influences on individuals’ lives. She also explained that Government were quick to use very early data in order to ‘demonstrate’ the positive outcomes of the programme before the scientists had finished evaluating and analysing. Scientists eventually produced a very different picture to that which had been made public highlighting the struggle of conducting randomised control trials that are politically driven.
Trish suggests that RCT evidence is of limited value in e-health research, because ‘today’s trial relates to yesterday’s technology’. Technologies are becoming cheaper and faster all the time. Also, neither people nor technologies can be meaningfully studied in isolation from the socio-technical system in which they are embedded.
She also discussed the importance of case studies in social research, suggesting a social science discipline that lacks in-depth case studies is impoverished because we need detailed narratives of what happened and why.
Trish talked us through various models of evaluation including those of Professor Saville Kushner and Professor Barry Macdonald, and suggested we need more democratic (deliberative) evaluation, where we critically question what is going on, who is not being heard, where are the hidden agendas.
Despite the fact that funding panels seem intent on recruiting more RCTs, Trish suggests RCTs reduce complex questions to over simplified scientific questions, particularly the question of ‘what works.’
Trish summarised her seminar by suggesting that:
- E-health research is dominated by a naïve, modernist mindset that envisages a rational technological utopia.
- The RCT, borrowed from drug experiments and promising simple answers about ‘what works’, aligns with this vision and is strongly favoured by funders.
- E-health programmes are complex, multifaceted and politically situated, they fail chiefly because no one fully understands what’s going on.
- In depth case studies offer rich understanding but such approaches are misunderstood.
You will soon be able to watch the film of Trish’s seminar and you can access her slides here.