As artificial intelligence develops, how do we stay in charge?
Professors Vasant Dhar and Sam Bowman discuss the control problem in Episode 58 of the Brave New World Podcast
The question at the center of Sam Bowman’s career has been: what can neural networks do with language? The CDS Associate Professor of Linguistics, Data Science & Computer Science studies neural network models for natural language understanding and is currently working as a visiting researcher for the San Francisco-based AI company Anthropic. Bowman met with Vasant Dhar, CDS Director of Industry Relations and Professor of Technology, Operations, and Statistics at Stern School of Business, on a recent Brave New World podcast to discuss large language models (LLMs), oversight, and the potential risks and benefits that technologies like ChatGPT have in store for humanity. Hosted by Dhar, Brave New World explores how machine learning technologies are transforming society in the post-COVID world.
The podcast starts off with some history. Bowman walks through the research communities’ interest in sentence analysis, textual similarity, and exploring if models “understand” what sentences mean. He discusses the shift away from applying human reasoning to technology towards letting models learn through observation.
Bowman also breaks down how LLMs work. “It’s the same thing your phone keyboard does when it suggests the next word,” said Bowman, explaining that after being trained on vast amounts of text, the model is essentially predicting what words come next. Dhar brings up the creativity of models like ChatGPT that have recently grabbed the world’s attention. “How is it able to do that behind the scenes,” asks Dhar. “Where’s the magic?”
In response, Bowman says as models become bigger, they become qualitatively more impressive, creative, and flexible. This brings the professors into a discussion over a major question facing the AI research community: whether the current foundations of LLMs are strong enough to form the basis of a more general-purpose intelligence. While there is significant potential, Bowman explains current models face limitations.
There is also concern over the technology’s harm to society, especially when deployed on a large scale. Dhar points to the control problem described by Norbert Wiener in his 1950 book The Human Use of Human Beings, “If we use, to achieve our purposes, a mechanical agency with whose operations we cannot efficiently interfere once we have started it, then we had better be quite sure the purpose we had put into the machine is the purpose which we really desire and not merely a colorful imitation of it.”
Bowman breaks down three categories of concern, ranging from students cheating on tests, stereotype-based harms, and the ease of sidestepping the technologies set guardrails. The third category toes into a more science-fiction realm with the potential for AI to develop sub-goals in pursuit of their programmed goals, leading to unintended consequences. “How do we incentivize systems,” asks Bowman. “How do we show systems what we want when they are, in some sense, more capable than we are?”
Even with safety as a major concern, the technology has the potential to improve humanity through enhancing accessibility, providing assistance to humans, and acting as a “high-level tutoring” device for many. Bowman discusses his recent work on a test case “Measuring Progress on Scalable Oversight for Large Language Models”. Researchers found after testing a person on medical questions they had no background knowledge of, as expected, they performed poorly. When test subjects had access to a model like ChatGPT, they were able to learn more quickly and provide more accurate answers.
To listen to the full podcast episode, please visit “Episode 58 of the Brave New World Podcast: Sam Bowman on ChatGPT & Controlling AI.”
By Meryl Phair
