CDS Professor and Student Create AI Course Assistant More Accurate Than ChatGPT

Over 95% of students at NYU used AI tools for coursework in 2024, a dramatic shift that transformed how students approach learning. In “Towards a Personalized AI Assistant to Learn Machine Learning,” published in Nature Machine Intelligence, CDS Clinical Associate Professor of Data Science and Psychology Pascal Wallisch and his former student Ibrahim Sheikh developed an innovative solution: an AI course assistant called Plutarch that reduces AI hallucinations by incorporating validated course materials.

Rather than fight against AI use in education, Wallisch and Sheikh created a system that harnesses AI’s capabilities while addressing its shortcomings. Their tool, called Plutarch, uses a technique called retrieval-augmented generation (RAG) to enhance the accuracy of AI responses by incorporating specific lecture content and course materials.

When students ask questions about course concepts, Plutarch searches through a database of verified lecture materials to find relevant content, which it then uses to generate accurate responses. This approach significantly reduces the “hallucinations” — incorrect or contradictory information — that often plague traditional AI models.

In their paper, Wallisch and Sheikh demonstrated Plutarch’s superiority over standard AI models. When asked whether adding a poor predictor to a regression model could decrease the R-squared value, ChatGPT incorrectly said yes. Plutarch, drawing from lecture materials, correctly explained that R-squared can only increase or stay the same.

An illustration of how Plutarch improves upon traditional AI systems. While standard AI models (shown in panel a) simply take a query and generate a response, Plutarch (shown in panel b) adds several crucial steps, with information flow highlighted in blue. When a learner submits a query (like “Can R² be negative?”), the system performs a similarity search to find relevant material in its database. This validated content is then combined with the original query before generating a response, ensuring accuracy by grounding answers in the professor’s actual course materials.

The system has been deployed in multiple CDS courses, including Principles of Data Science and Fundamentals of Machine Learning. Students can generate practice quizzes, ask questions about specific slides, and even interact with the system in their native language. For visually impaired students, Plutarch can describe elements of slides or read content aloud.

Initial results showed promise. About half the students who used Plutarch reported spending more time actively engaging with course material than those who didn’t use the tool. These students also reported better understanding of course content and achieved higher grades.

“The idea was to channel this urge of the students to use AI into more controlled and positive channels,” Wallisch said.

The team has made Plutarch’s code freely available on GitHub, aiming to help other educators adapt to the new reality of AI in education.

By Stephen Thomas