From Rule-Based Systems to General Intelligence: The Evolution of AI Through Four Major Paradigm Shifts
The artificial intelligence field has been shaped not by steady progress, but by dramatic transformations in how machines process knowledge. In a newly published article in Communications of the ACM, “The Paradigm Shifts in Artificial Intelligence,” CDS and Stern Professor Vasant Dhar traces AI’s development through four major paradigm shifts — a perspective that emerged from an unexpected source: his PhD classroom.
“Many people in the field think that AI is deep learning, and nothing else is worthwhile,” Dhar said. “I realized while teaching a PhD seminar on AI that the students would benefit from a historical perspective on the field. To take a Bob Marley line, “if you don’t know your history, then you won’t know where you’re coming from.”
The first paradigm centered on expert systems in the 1960s–80s, where human knowledge was explicitly encoded as rules. While this worked for narrow domains, it proved extremely difficult to capture the full complexity of human expertise. The field then shifted toward machine learning in the late 1980s, letting algorithms automatically discover patterns in structured data. However, this still required extensive human curation of features. Deep learning emerged as the third paradigm, finally allowing machines to process raw sensory data like images and speech directly. But the most dramatic shift came with the recent rise of large language models and general AI systems that can leverage knowledge across domains.
“What’s remarkable about the current paradigm is that these systems weren’t explicitly trained for most of what they can do,” Dhar noted. “In learning to communicate fluently, they had to acquire broad knowledge about the world as a side effect.”
However, this new paradigm has introduced unexpected challenges. Dhar’s recent work on a “Damodaran bot” that emulates the stock valuation process of his NYU Stern colleague, Aswath Damodaran, notes troubling variability in its outcomes that are not trivial to address.
“What I’m finding now is a really vexing problem — every time you run it, it gives you something different,” Dhar explained. “I’m just surprised that people aren’t bothered by that. We’ve gone from machines that give you the same output every time to these systems where we have no idea what’s going to come back. What makes it challenging is that sometimes the variability is legit.”
He illustrated this with an example: “Two months ago when evaluating BYD, the system was really focused on the company’s battery technology and said the company was undervalued. Three weeks ago, it started emphasizing tariffs and global trade risks, concluding the company was closer to being fairly valued. Maybe it had started picking up on the possibility of a Trump victory.”
But this unpredictability points to deeper questions about how modern AI systems reason. While previous paradigms struggled with explicit knowledge engineering, today’s challenge centers on controlling attention and reasoning paths. “The one thing we’ve come to expect from machines is consistency,” Dhar noted. “And that’s ironically what we’re currently failing on.”
Despite these challenges, Dhar remains optimistic about the field’s trajectory. “There’s no choice — we have to solve it,” he said. “The only question is how well and how soon.”
Looking ahead, he sees the field potentially coming full circle to revisit questions that dominated early AI research. “We’ve come back to thinking about thinking and reasoning and meaning — all those good things we used to worry about in the expert systems era. We just forgot about them in the machine learning and deep learning eras.”
CACM also produced a video of Vasant talking about his article.
By Stephen Thomas
