Responsible Data Science and AI: Accounting for Dataset Shifts and Designing Fair Machine Learning Systems
Recently, CDS PhD student, Harvineet Singh, had the opportunity to present in the University of Michigan’s Future Leaders Summit. The theme of the summit? Responsible Data Science and AI.
As we know, data science and AI have a tremendous impact on how we make decisions in society, and it can be extremely beneficial by assisting in disease detection, optimizing efficiency of non-profits, and identifying solutions for a multitude of social issues. However, data science and AI also have the ability to be incredibly detrimental by automating discriminatory lending, swaying elections, and amplifying racist precedent in the justice system.
So how do we address this problem? By practicing responsible data science. This can take place in many forms such as analyzing equity and fairness in automated decision making, identifying the reproducibility and replication of research results, and considering systemic issues when conducting research. Harvineet specifically addresses how to design fair machine learning systems and address dataset shifts in his research. We reached out to Harvineet to learn more about his presentation and research.
Tell us about your talk at the Future Leaders Summit!
H: First, I would like to thank the MIDAS institute at University of Michigan for the opportunity to attend the Future Leaders Summit. The Future Leaders Summit this year was organized around the theme of Responsible Data Science and AI. The summit included short talks by graduate students and postdocs on this theme. My talk was on the aspects of fairness and robustness of machine learning (ML) systems. The use of ML systems to make decisions that impact humans raises many important questions. What are the societal implications of discrimination inherent in the ML systems? How do we design such systems to make fair decisions? What are the challenges to ensuring that systems make fair decisions in the real world?
In my talk, I presented our work on a challenge to designing fair ML systems in the real world — the challenge of dataset shifts. Characteristics of datasets resulting from the real world rarely remain the same over time. “Change is the only constant” (in ML systems), paraphrasing Heraclitus. For example in the context of COVID-19, we witnessed the changing prevalence of the disease and its symptoms as new variants arrived and vaccinations ramped up. As a result, a ML model trained on data at a given time in such a changing world is expected to be deployed in a world where the data looks very different. This shift in data from the time model is trained to the time it is deployed may result in serious decay in its performance.
What would you say was the impetus for the research covered in your talk?
H: We were motivated mainly by the healthcare domain in which dataset shifts are prevalent [2]. Moreover, issues such as systemic racism that acutely affect the health and access to proper healthcare for marginalized groups result in unfairness of the models derived from health data. Thus, fairness interventions at the model design stage (but not just restricted to modeling) are needed to ensure fair medical decisions in an ever-changing world. A recent report by U.S. Food and Drug Administration (FDA) [3] aptly makes a call for approaches to address robustness and bias in the use of ML to make medical decisions. Thus, the problem of fair and robust ML is being recognized as an important one. We hope that our work highlights some of the underlying issues in addressing this problem and motivates the importance of causal inference to come up with a viable solution.
What are the implications of your research?
H: Dataset shifts pose a threat to the validity of model fairness. In brief, model fairness means that the model is trained in a way that the resulting predictions satisfy some desired criteria such as error rates for two demographic groups should be equal. Since the model was designed to be fair on a different dataset, its fairness when deployed is going to be suspect. So, how do we design fair ML systems that are robust to dataset shifts? I presented our recent work [1] addressing this question. We propose methods that are inspired from earlier work in causal inference to build robust models. We also discuss limitations of our methods in terms of the types of dataset shifts, which we presently cannot address. There is much work to be done on this important problem! Fittingly, the issue of dataset shifts was also highlighted in a couple of talks and a panel discussion at the Summit.
How do you approach the topic of “Responsible Data Science” within your research?
H: My approach to responsible data science, as exemplified in the above work, centers around thinking critically about the data that we use in our models and how it relates to the objectives that we have for the model outputs. The ability to specify how the available data reflects the real world and what we want out of the models is important for using data science responsibly. For instance, depending on the type of mismatch that exists between training and deployment data, and the fairness objectives for the model output, the solution to the above problem will look different. Tools and, more importantly, the language of causal inference enables reasoning about the data and objectives. A by-product is that by being specific in describing our inputs and outputs, we are more transparent about the limitations of the models. I believe that the intersection of causal inference and machine learning has much to offer towards a responsible conduct of data science.
The work Harvineet presented at the Summit was done with Professor Rumi Chunara, Vishwali Mhasawade, and Dr. Rina Singh.
Resources
