Open Research on Discord: A New Model for Scientific Collaboration
Scientific research traditionally requires institutional affiliations and formal commitments. However, CDS Research Scientist Ravid Shwartz-Ziv is experimenting with a different approach, coordinating multiple research projects through a Discord server where anyone interested can contribute to exploring connections between large language models and information theory.
The initiative began with a Twitter post. “I think Twitter is a great tool for science and development of science,” Shwartz-Ziv said. “So far, most scientists have been sharing our research and ideas after we do the work. Let’s try to do it before. Let’s try to make it collaborative research.”
The Discord server now hosts four to five concurrent projects investigating various aspects of large language models through an information theory lens. Two have already produced papers accepted to upcoming workshops. One, “Learning to Compress: Local Rank and Information Compression in Deep Neural Networks,” co-authored by Shwartz-Ziv and Niket Patel, a master’s student at UCLA, examines how neural networks compress information during training, demonstrating that they naturally reduce the dimensionality of their learned representations — a phenomenon that connects to Information Bottleneck theory.
Another paper that resulted from the Discord, “Does Representation Matter? Exploring Intermediate Layers in Large Language Models,” has been accepted to this year’s NeurIPS workshop. Authored by Shwartz-Ziv, Oscar Skean, a PhD student at the University of Kentucky, and Md Rifat Arefin, a PhD student at Mila/University of Montréal, the paper investigates how different factors affect model representations in large language models.
Additional ongoing projects dive deeper into specific aspects of language model behavior. One team is examining in-context learning, where models adapt their behavior based on examples provided in their input. “There is not a lot of knowledge about what in-context learning actually does,” Shwartz-Ziv said. “We are trying to look at this topic from an information-theoretic perspective — how you can analyze this phenomena. Does it compress information? Does it keep relevant information?”
The participants meet weekly or biweekly to discuss experiments, theoretical progressions, and empirical findings. The collaborators come from diverse backgrounds, including PhD students like Denis Janiak at Wrocław University of Science and Technology and Rohan Pandey at UMass Amherst, and also people in industry, like Abi Aryan, an LLMOps Advisor and MLE Mentor. This diversity of backgrounds and perspectives enables unique collaborations, though the voluntary nature of participation creates both opportunities and challenges.
“No one is bound to come by any commitment,” Shwartz-Ziv said. “They need to want to be involved and to want to do things. Many of them came one, two, three times and then just quit. There are a lot of efforts and time without results.”
The structure differs markedly from traditional academic research, where advisors can direct their students’ work and ensure consistent progress. Here, projects succeed only through sustained mutual interest and commitment. While this leads to some initiatives fading away, it also ensures that completed work represents genuine collaborative enthusiasm.
Despite these challenges, Shwartz-Ziv plans to continue this experimental approach to research coordination, though likely with fewer concurrent projects. “I think this is a great kind of research — more collaborative, more open,” he said. “You can get a lot of results. You can answer a lot of interesting questions from this kind of research.”
The initiative represents a new model for scientific collaboration, one that leverages modern communication tools to enable voluntary, interest-driven research across institutional and geographical boundaries.
If you’re interested in getting involved, contact Ravid at rs8020@nyu.edu or find him on X/Twitter.
By Stephen Thomas
