Making AI Work Reliably for Everyone: Tim G. J. Rudner’s Research Earns Multiple Honors
The University of Massachusetts Amherst named CDS Instructor Tim G. J. Rudner a Rising Star in Generative AI in 2024, recognizing him as one of the field’s most promising researchers preparing to enter the academic job market. The honor marked a breakthrough year for Rudner, whose research on making AI systems more robust and transparent earned additional recognition from two major machine learning conferences.
The Rising Star award included participation in an intensive career development workshop focused on launching successful academic careers in AI. Selected Rising Stars gave research presentations and received mentoring from established faculty on topics ranging from building research collaborations to creating impactful partnerships with industry.
Rudner also earned a Notable Paper Award at AISTATS 2024 for his work with CDS Silver Professor of Computer Science, Mathematics, and Data Science Julia Kempe and CDS Professor of Computer Science and Data Science Andrew Gordon Wilson on improving machine learning fairness. Their paper, “Mind the GAP: Improving Robustness to Subpopulation Shifts with Group-Aware Priors,” put forth a novel probabilistic approach called Group-Aware Priors (GAP) that helps neural networks maintain reliable performance across groups even when certain groups are highly underrepresented in training data.
The problem they tackled extends beyond simple statistical underrepresentation. When collecting training data, minority groups often appear even less frequently than their population numbers would suggest, creating what’s known in the literature as a subpopulation shift. That is, due to biased data collection procedures, minority groups might make up an even smaller percentage of the training data than they do in the real population. This compounds the challenge of building models that work equally well for everyone.
“We wanted to ensure that machine learning models never perform very poorly on any groups in the population,” Rudner said. “Rather than just optimizing for average performance across groups, we created a probabilistic method for training neural networks that explicitly favors predictive models that work well across all groups.”
The collaboration grew from previous work with Kempe on adversarial robustness — published earlier this year in the Transactions on Machine Learning Research (TMLR) — where they had identified errors in existing research. This time, they wanted to focus on developing solutions rather than just identify problems. Their approach leverages the fact that many different machine learning models can achieve similar average performance on a dataset. By carefully designing what’s called a prior distribution over neural network parameters, they could guide the training process toward neural networks that maintain consistent performance across all groups.
A third honor came when Rudner’s paper “Pre-trained Text-to-Image Diffusion Models Are Versatile Representation Learners for Control” with researchers at the University of Oxford and Georgia Tech received a Spotlight at NeurIPS 2024. This work challenged conventional wisdom about which types of pre-trained AI models work best for language-guided control tasks. While most researchers use CLIP-like models as their standard approach, Rudner and his colleagues demonstrated that text-to-image diffusion models could serve as more effective building blocks for AI systems that interact with the physical world.
The paper’s message that “pre-trained text-to-image diffusion models are versatile representation learners for control” could reshape how researchers and practitioners approach problems in vision-language-guided control. Their “stable control representations,” derived from Stable Diffusion models, achieved leading results on challenging robotics tasks, particularly excelling at what’s known as OVMM, a particularly difficult open-vocabulary navigation benchmark.
The research team conducted a careful empirical investigation of which components of diffusion models contain the most useful information for control tasks. They found that diffusion models, which are trained to generate images from text descriptions, naturally learn to represent fine-grained visual details that are crucial for robotic control but often missed by CLIP models.
This latter collaboration brought together researchers across continents, including PhD students from Oxford and Georgia Tech, alongside junior and senior faculty members. “Everyone brought their specific expertise to the table,” Rudner said. “Having team members with diverse technical backgrounds helped us tackle these complex problems from multiple angles.”
Prior to joining CDS, Rudner completed his PhD at the University of Oxford, where he focused on probabilistic machine learning methods and maintains several collaborations. His current research aims to make AI systems more robust and transparent, particularly for applications in generative AI, healthcare, and scientific discovery.
By Stephen Thomas
