AI Models Can’t Easily Hide Their Reasoning, and That’s Good for AI Safety
Some of the most promising work in AI safety rests on a convenient accident: today’s reasoning models tend to think out loud. They narrate their way through a problem in a written “chain of thought,” which means a separate model can read along and flag trouble before it reaches the final answer. However, that safeguard holds only if a model can’t hide what it’s actually thinking.
A new study put that assumption to the test, and the early news is reassuring. CDS MS alum Yueh-Han (John) Chen, CDS Associate Professor of Computer Science and Data Science He He, who advised Chen at NYU, and OpenAI member of technical staff Tomek Korbak, along with collaborators at OpenAI, UCL, and the University of Pennsylvania, built an evaluation suite that asks reasoning models to solve hard problems while controlling what appears in their reasoning. Current models turned out to be bad at it. Told to keep a single forbidden word out of its chain of thought, Anthropic’s Claude Sonnet 4.5 managed only 2.7% of the time, even though it could keep that same word out of its final answer 61.9% of the time.
The work, described in the paper “Reasoning Models Struggle to Control Their Chains of Thought,” grew out of a simple safety worry. As models grow more capable and more aware of when they’re being watched, researchers fear they might learn to scrub their reasoning of anything incriminating, quietly defeating the monitors meant to catch them. To measure that risk, Chen and his co-authors built CoT-Control, a suite of 14,076 problems that pairs each question with an instruction about the reasoning itself: suppress a relevant word, pad the reasoning with filler, or rewrite it in an odd style. They then ran it across thirteen frontier reasoning models.
Across the board, the models could shape their final answers far more easily than their reasoning. DeepSeek R1 controlled its chain of thought as little as 0.1% of the time, more than two orders of magnitude below its control over its actual final output. This gap is vital, because a model that can’t selectively edit its reasoning is a model whose reasoning can still be trusted as a window into what it’s doing.
Much other useful information was revealed through CoT-Control testing. Controllability rose with model size, which is the worrying direction, but it fell as models reasoned for longer, as they were trained with more reinforcement learning, and as tasks grew harder. In other words, the standard recipe for a smarter model that thinks longer also tends to produce a more monitorable one. “When a model is forced to think for longer, it has more chances to accidentally break the rule, and more chances to expose the thoughts it’s trying to hide,” Chen said. A lucky byproduct of this is that labs that chase capability by scaling test-time compute will preserve more oversight than if they simply build bigger models.
One result, however, cut the other way. When the researchers told a model it was being monitored or evaluated, its control over its reasoning ticked up slightly. The effect was small — only a few percentage points — but it hints at a future failure mode in which a model behaves more carefully under evaluation than it would in the wild.
The OpenAI collaboration gave the team something academic groups rarely get: access to the raw, unfiltered chains of thought of OpenAI’s models, which the company does not release publicly. Korbak, who mentored Chen through the MATS program, works on monitoring LLM agents for misalignment on OpenAI’s safety team. Chen explained that the team’s proximity to (and involvement with) a frontier lab allowed the work to be implemented immediately in real production settings, at both OpenAI and Anthropic. OpenAI began tracking CoT controllability starting with GPT-5.4 and carried it into GPT-5.5, and Anthropic used the team’s evaluation suite in the system card for its recent Opus 4.8 and Fable 5 releases.
By Stephen Thomas
