How LLMs Could Transform the Study of Human Groups
One of the most commonly tweeted phrases today is “@grok, is this real?” — a stark indicator of how people are turning to AI systems with a tendency to hallucinate as their primary source of truth. CDS Faculty Fellow Ilia Sucholutsky and his colleagues argue that this phenomenon, along with other emerging patterns of human-AI interaction, demands new scientific approaches to understanding how groups of people think and work together.
In a new perspective published in Nature Computational Science titled “Using LLMs to advance the cognitive science of collectives,” Sucholutsky and his co-authors lay out a framework for how large language models could revolutionize the study of collective cognition — the cognitive phenomena that emerge when multiple people interact. The journal’s editors commissioned Sucholutsky to lead a team to create this paper, which is part of a special issue exploring how LLMs will impact various scientific fields.
The research team, which also includes Katherine M. Collins (Cambridge), Nori Jacoby (Cornell), Bill D. Thompson (UC Berkeley), Robert D. Hawkins (Stanford) — all early-career researchers — identified three major axes along which cognitive science has struggled to scale its studies of human groups. The first is structural complexity — the challenge of studying large, intricate social networks where people have varying connections and access to each other. The second is interactional complexity, which involves moving beyond simple yes-or-no responses to capture the richness of natural conversation and dynamic scenarios. The third is individual complexity, accounting for the vast differences in language, culture, and background that exist both within and across societies.
“What we’re saying is that the greatest impact is going to be and where it’s a little bit underexplored still is using these LLMs to advance our study of groups of people at much larger scales — society-level, potentially,” Sucholutsky said.
The paper outlines five distinct roles that LLMs can play in studying collective behavior: as participants embedded in social networks alongside humans, as interviewers capable of conducting conversations in multiple languages, as dynamic environments that adapt to participant actions, as facilitators that can summarize and route information between large groups, and as analysts that can process massive amounts of unstructured conversational data.
Recent studies have already begun demonstrating this potential. In one experiment, researchers used an LLM to help over 5,000 people reach consensus on contested political issues like abortion and immigration by iteratively summarizing and ranking human statements. Another study placed 625 humans and LLM agents in an experimental social network where participants selected and creatively rewrote stories, revealing that hybrid human-AI groups showed greater diversity than purely human or purely AI societies.
These applications matter because much of human intelligence emerges not from individual brainpower alone, but from interaction and collaboration. Yet studying these interactions at realistic scales has always been methodologically challenging. Observational studies of existing social networks lack control, while laboratory experiments with small groups lack the scale and naturalness of real-world social dynamics.
The authors also identified critical risks that need addressing. LLMs often produce a narrower distribution of responses than actual human populations would, potentially missing the linguistic and cross-cultural diversity of participants from around the world. There are also concerns about reproducibility — as Sucholutsky noted, when OpenAI releases new models, they often deprecate older ones, raising questions about what happens to studies built on models that suddenly disappear.
Sucholutsky’s own research is already exploring some of these frontiers. His team is investigating what happens when humans increasingly rely on language models for creative and productive work, and those models are then trained on the outputs of that human-AI collaboration. “Is there some kind of homogenization effect that happens over a few generations, where everyone is producing similar looking data, the LLMs are trained on that similar looking data, and it narrows that distribution even further?” he asked.
Another major study his group has launched examines patterns of over-reliance on AI systems across different cultures and domains, from medical diagnostics to fact-checking. The research spans multiple universities and countries, moving beyond the typical focus on Western populations to understand how people from diverse cultural backgrounds decide when to outsource their cognitive labor to these models.
The paper represents what Sucholutsky described as “five young researchers who have these exciting, big ideas about the future of the field, trying to write a framework” that could enable the kinds of large-scale studies of human interaction that were previously impossible, allowing us to understand how AI systems are transforming not just individual thinking, but the very fabric of how humans communicate and collaborate as societies.
By Stephen Thomas
Have feedback on our content? Help us improve our blog by completing our (super quick) survey.
