Building Human-Compatible AI Through Cognitive Science: A New Path Forward
Current AI systems act more like tools than true partners, often failing to understand human intentions or adapt to individual needs. A new perspective paper in Nature Human Behaviour, “Building machines that learn and think with people,” led in part by CDS Faculty Fellows Ilia Sucholutsky and Umang Bhatt, proposes a fundamentally different approach to creating AI systems that can genuinely collaborate with humans.
The research team, which also includes former CDS Faculty Fellow and current NYU Assistant Professor of Psychology Mark Ho and researchers from Cambridge, Princeton, MIT, Microsoft, the University of Chicago, and the Alan Turing Institute, laid out a vision for “thought partners” — AI systems designed to complement human cognition rather than simply execute tasks. These systems would understand users’ goals and limitations, communicate clearly about their own capabilities, and share a common understanding of the world with humans.
“We’re reaching a point where AI systems are getting smarter and more capable, and being employed more broadly for a much wider range of tasks,” said Sucholutsky. “Historically, AI was like a tool for us to use — basically glorified calculators. However, increasingly, people have started to think about AI-based automation in terms of replacing humans. This paper is advocating for a middle role: AI systems as thought partners.”
For Sucholutsky, who is building on previous work in which he led a large consortium of cognitive scientists, neuroscientists, ML researchers, and roboticists in unifying the field of representational alignment, “Getting aligned on representational alignment,” this looks like agents that can learn and think with us rather than instead of us.
“We need AI systems that don’t just mimic human behavior, but actually build explicit models of human cognition and the world,” said Sucholutsky. “This allows them to adapt to individual users’ needs and constraints while maintaining clear communication about their own limitations.”
The work also builds on Bhatt’s research exploring how different cultures interact with AI systems. “A lot of our research has focused on understanding when AI systems are actually used, versus when we’ve trained them to be used,” Bhatt noted. “It’s not necessarily true that you need a large language model for a task that humans are very good at already. If you want to align AI systems well, you need to understand the cultural context of the people who are actually interacting with those tools.”
The paper identifies several modes of collaborative thought where humans and AI could work together effectively. These range from planning and decision-making to learning and creative tasks. In programming, for example, an AI system could not only suggest code but understand a programmer’s intentions and misconceptions, helping to explain why code behaves in certain ways.
The researchers propose using frameworks from cognitive psychology and Bayesian modeling to create these more sophisticated partnerships. This approach differs significantly from current AI development methods that rely primarily on training models on large datasets. Instead, it emphasizes building systems that can reason about human minds and intentions while maintaining transparency about their own capabilities.
One key innovation the team explores is “cooperative language-guided inverse plan search” (CLIPS), which allows AI systems to understand human intentions from both actions and language. This could enable more natural collaboration in physical tasks, from cooking to search-and-rescue missions.
The paper also addresses potential challenges in medical settings, where AI systems need to work alongside doctors while accounting for both technical expertise and patient care considerations. The researchers envision systems that could help doctors process large amounts of medical evidence while adapting to individual physicians’ time constraints and working styles.
The research team emphasizes that their approach requires rethinking not just AI capabilities but the entire ecosystem of human-AI interaction. This includes developing ways to prevent over-reliance on AI systems and ensuring they remain complementary to human intelligence rather than attempting to replace it.
The research has significant implications for the future of AI development. As these systems become more integrated into human work across various professional fields, the ability to create true partnerships between humans and AI becomes increasingly crucial. The team’s framework offers a path toward AI systems that enhance human capabilities while remaining transparent and trustworthy.
Looking ahead, the researchers identify several key challenges, including evaluating AI thought partners effectively and ensuring these tools remain accessible to diverse users. They emphasize the importance of continuing to draw on insights from cognitive science as this field develops.
“The goal isn’t just to make AI systems more powerful, but to make them better partners,” Sucholutsky explained. “This means understanding how humans think and learn, and using that knowledge to build systems that can truly work alongside us.”
By Stephen Thomas
