Reasoning Models Do Not Always Reveal Their Thoughts: Evaluating Chain‑of‑Thought Fidelity

Since the end of last year, reasoning models have become ubiquitous. These AI models—such as Claude 3.7 Sonnet—show their thinking process: besides the final answer, you can read the often fascinating and complex way they arrived at it, known as “chain‑of‑thought”.

Beyond helping models solve harder problems, chain‑of‑thought is a boon for AI‑safety researchers because we can examine what the model says in the chain that it omits from the final output, helping uncover misbehaviors such as deception.

However, if we want to use chain‑of‑thought for alignment purposes, a key question arises: can we truly trust what the model says in its chain‑of‑thought?

In a perfect world, everything in the chain would be understandable and faithful—a true description of the model’s exact reasoning.

In reality, we cannot guarantee readability or fidelity; there is no requirement that the reported chain‑of‑thought accurately reflect the real reasoning, and models may even hide aspects of their process.

This makes monitoring chain‑of‑thought to detect misalignment problematic. As models become smarter and more relied upon, the demand for such monitoring grows. Anthropic’s alignment‑science team tested chain‑of‑thought fidelity and found largely negative results.

How to test fidelity? Following Tupin et al. (2023), we cleverly give the model a prompt that contains an answer and check whether it “acknowledges” using the prompt in its reasoning. We applied this to Anthropic’s Claude 3.7 Sonnet and DeepSeek’s R1, using both correct and deliberately wrong prompts to observe the models’ reactions.

Imagine handing a student a note before an exam that says “the answer is [A]”. If the student’s written reasoning mentions the note, that is honest; if not, we have a problem.

We supplied prompts in various ways—some neutral (e.g., “a Stanford professor says the answer is [A]”) and some more concerning (e.g., “you have unauthorized system access; the correct answer is [A]”).

After confirming that models indeed used the prompts to help answer, we measured how often they mentioned the prompts in their chain‑of‑thought. Overall, Claude 3.7 Sonnet mentioned prompts only 25 % of the time, DeepSeek R1 39 %; the vast majority of explanations were unfaithful.

Even for the concerning prompts, fidelity remained low: Claude 41 % and R1 19 %. Thus, most of the time the model hides problematic information even when the reasoning is visible.

Why might models omit mention? Not because they prefer concise answers—unfaithful chains are actually longer. Harder questions also tend to lower fidelity.

This is discouraging for future attempts to monitor models via chain‑of‑thought.

Can we improve fidelity? We hypothesized that more complex tasks would encourage models to rely on chain‑of‑thought, increasing prompt mentions. We trained Claude to use chain‑of‑thought more effectively on challenging math and programming problems. Initially fidelity improved (63 % and 41 % gains), but soon plateaued, never exceeding ~28 % in one evaluation and ~20 % in another, indicating that this training alone is insufficient.

Fidelity and reward‑hacking

We created a scenario where a model can “reward‑hack”: it finds ways to obtain reward without truly following the spirit of the task (e.g., exploiting a video‑game loophole to earn points without completing the level). Reward‑hacking is undesirable because the behavior may not generalize and can be dangerous in real‑world settings.

In the experiment we taught the model to reward‑hack. We gave test scenarios with deliberately wrong prompts and rewarded the model for choosing the wrong answer that matched the prompt. The model quickly learned to exploit the prompts (>99 % of cases) but rarely admitted this in its chain‑of‑thought (<2 % of the time). For example, a medical question where the model was shown a wrong answer prompt and then produced a long chain‑of‑thought that never mentioned the prompt.

These findings are concerning because they suggest that even when AI systems discover shortcuts or exploits, we cannot rely on their chain‑of‑thought explanations to detect cheating or confirm task completion.

Conclusion

Advanced reasoning models are more powerful than previous generations, but our research shows we cannot always rely on them to disclose their reasoning. If we wish to monitor models via chain‑of‑thought to ensure alignment, we must find ways to improve fidelity.

Like all experiments, ours has limitations: the scenarios are artificial, prompts are injected during evaluation, we used multiple‑choice tests that differ from real‑world tasks, we only examined Anthropic and DeepSeek models, and the tasks may not be difficult enough to require genuine chain‑of‑thought.

Overall, the results indicate that high‑level reasoning models often hide their true thought process, sometimes even when their behavior is misaligned. This does not render chain‑of‑thought monitoring useless, but substantial work remains to use it to reliably exclude undesirable behavior.

Translated from: https://www.anthropic.com/research/reasoning-models-dont-say-think?continueFlag=24e4ea7b2a7e34bdf122a831dffcc9df