Think Your Code Model Is Smart? Interactive Benchmarks Might Say Otherwise
Letting models receive human-style feedback changed which ones ranked best by up to four spots. That is what Courant PhD student Jane Pan, CDS PhD student Jacob Pfau, CDS Assistant Professor He He, and colleagues showed in “When Benchmarks Talk: Re-Evaluating Code LLMs with Interactive Feedback,” which described a way to replace static coding benchmarks like HumanEval, MBPP, APPS and CodeXGLUE with a multi-step, human-in-the-loop evaluation. Valerie Chen, who spent last summer and fall as a visiting researcher at CDS before returning to Carnegie Mellon University for her PhD, was the senior author on this project, which will be presented at this summer’s Association for Computational Linguistics conference (Findings of ACL 2025).
In real-world use, programmers rarely hand off perfect, paragraph-long prompts to AI assistants. What they actually do is write a short comment or paste in code and then ask the assistant to fix or change it.
This study copied that process. It hid key details of each problem and then fed back simulated user responses — code edits, brief instructions or short paragraphs — into ten different language models across three benchmarks. The added feedback reshuffled model standings; one model rose from 7th place to 3rd when given paragraph-style comments. Another dropped from 2nd to 6th with only a single-sentence prompt. “We found that testing on new problems the models had never seen made feedback even more powerful,” Chen said. The shuffled rankings showed that code models — relative to each other — perform differently in the interactive setting than they do on standard (static) benchmarks. As Pan pointed out, “This indicates that standard benchmarks don’t fully capture how models perform in a more ‘realistic’ setting.”
Longer comments drove the biggest gains and the most code changes. On average, they fixed 22% more failed test cases and added hundreds of edits each time. Simpler prompts led to smaller improvements and fewer changes.
To circumvent the cost and delay of real user studies, the researchers built an automated system that acted like a developer. The system used language models to generate feedback in the form of code edits, rephrased input queries, single-sentence directions or short paragraphs. “Our simulations let us run these tests at scale without real users,” Chen said.
Chen added that naming specific code lines directly often matched or beat long text instructions. This method lets users point to exactly what to change in the code, she explained. Chen described this approach as “meeting the model where it was.”
The team also measured steerability, which quantifies how much a model’s output shifts after feedback. They found that interaction amplified behavioral differences across models. This suggests that the kind of static benchmark that currently dominates LLM evaluations paints an incomplete picture of real-world performance, especially now that coding assistants are integral to software development workflows.
By mimicking real developer-AI exchanges, the study made clear that evaluations must match actual workflows. Without interactive benchmarks, researchers could overstate a model’s strengths or overlook flaws. As coding assistants become more common, testing them with feedback loops will be key for both creators and outside evaluators who want a clear, unbiased view of AI performance.
By Stephen Thomas
