Tuesday 6th July 2021, 15:00 – 16:00 BST
Presenters: Dr Corey Schimpf, Dr Brian Castellani and Dr Pete Barbrook-Johnson
Webinar Overview:
This seminar introduces case-based scenario simulation (CBSS) to evaluators. CBSS is well aligned with existing evaluation approaches, most directly realist evaluation which emphasizes ‘what works for whom in which circumstances’. CBSS’s approach enables users to explore in a simplified simulation environment the possibility of driving different clusters of outcomes in a desired evaluation direction, as well as explore counterfactuals. CBSS starts with the notion that interventions or policies may themselves not always be complex in nature, but that such interventions always happen in complex systems and that policy evaluators and analysts are in need of accessible tools or methods for studying how interventions unfold in these systems. In light of this challenge, we introduce a new approach that integrates two key methodologies for modeling and studying complexity: case-based methods and scenario simulation. The integrated approach of CBSS enables the identification and modeling of underlying groups of cases within a study and simulating hypothetical or ‘what-if’ scenarios to further explore how the system under study may respond to additional or alternative interventions. CBSS achieves this by modelling unique case types, their trajectories and the heterogeneous effects interventions may have on them. This work is grounded in a pluralist view of methodology, meaning that CBSS is not meant to replace other evaluation methods but rather to complement, support and extend them.
More concretely, for case-based modelling CBSS uses k-means and the self-organizing map, a type of neural network, together to uncover underlying case groups within the data so that the unique profiles of case groups and their dynamics or responses can be modelled. The scenario simulation part of CBSS maps the cases to an n x m grid representing different configurational possibilities for the case groups and allows the analyst to make targeted interventions on case groups, representing ‘what if’ or counterfactual scenarios, to explore if this improves, worsens, or alters case groups trajectories. We briefly present COMPLEX-IT, which is a lightweight open-source web application that supports the use of CBSS regardless of an analyst’s or researcher’s experience with these methods. In this presentation we discuss CBSS in relation to evaluation purpose (learning or accountability) and evaluation design (cross-sectional or longitudinal). From these dimensions we highlight four evaluation situations as combinations of each dimension and identify four documented evaluation cases that map to each situation. We discuss how CBSS could complement and extend evaluations in each situation, noting the ways in which CBSS can help attend to differences within cases, alternative scenarios, and attention to complex dynamics operative in the systems under study. We conclude with a discussion of the possibilities CBSS opens up for different types of evaluation as well as the commonalities across these situations and the other methodological implications it has for evaluation research.
Presenter Biographies:
Dr Corey Schimpf is an Assistant Professor of Engineering Education at the University at Buffalo, SUNY. Before rejoining academia, he spent five years at a small research and development nonprofit developing and examining innovative technologies for supporting secondary students in the United States to learn complex skills including engineering design thinking and science inquiry. A major strand of his work focuses on researching and developing lightweight applications to support practitioners, policy evaluators and other analysts to leverage complexity-appropriate methods, such as case-based modelling and agent-based modelling, into their toolkits. He is the primary developer of COMPLEX-IT. Much of his work looks at advancing research methods and more recently has sought to better leverage computer-logged data streams (such as through the web or MOOCs) to analyse complex process skills and to develop artificial intelligence systems within digital environments to respond dynamically to users interacting with the system.
Dr Brian Castellani Trained as a sociologist, clinical psychologist and methodologist (statistics and computational social science), Brian has spent the past ten years developing a new case-based, data-mining approach to modelling complex social systems – called the SACS Toolkit – which my colleagues and I have used to help researchers, policy makers and service providers address and improve complex public health issues such as public health and well-being; infrastructure and grid reliability; mental health and inequality; big data and data mining; and globalization and global civil society. We have also recently developed the COMPLEX-IT R-studio software app, which allows everyday users seamless access to such high-powered techniques as machine intelligence, neural nets, and agent-based modelling to make better sense of the complex world(s) in which they live and work.
Dr Pete Barbrook-Johnson is a Senior Research Associate at the Institute for New Economic Thinking and the Smith School for Enterprise and the Environment at the University of Oxford. Pete’s core research interests sit at the crossroads of social science and economics, complexity science, and environmental and energy policy. He uses a range of methods in his research including agent-based modelling, network analysis, and systems mapping. He regularly uses these, and other methods, to explore applied social, economic, and policy questions, and to support complexity-appropriate policy evaluation, but is equally interested in more theoretical aspects of complex adaptive systems. Pete is also a member of the Centre for the Evaluation of Complexity Across the Nexus (CECAN), a Visiting Fellow at the Centre for Research in Social Simulation (CRESS) and Department of Sociology at the University of Surrey, and a Research Associate at St Catherine’s College, Oxford. Pete is on twitter @bapeterj and his personal website is https://www.barbrookjohnson.com/
How to Join:
This talk will take place via a Zoom Webinar (registration now closed).
After registering, you will receive a confirmation email containing information about joining the webinar. In case you are unable to attend, a recording of the webinar will be uploaded to our website following the event.
Link to Webinar Recording & Blog:
If you were unable to join the webinar, you can watch the session via our YouTube channel below. A link to Brian’s post on the Sociology and Complexity Science Blog is included here. We are also pleased to be able to share a PDF version of the PowerPoint slides, please click here to view.