Evaluating chain-of-thought monitorability

我们在文中把“监测能力”（ monitorability ）定义为一个监测器能否预测被监控智能体行为中感兴趣性质的能力。这些性质通常对应各类不当行为——如欺骗、奖赏操纵或有偏推理——但也可以是无害的，比如智能体是否依赖某个提示或走了特定的推理路径。

当 AI 系统做出难以由人直接监督的决策时，弄清这些决策是如何产生的就变得尤为重要。一种有前景的方法是监测模型的内部推理过程，而不仅仅是其行为或最终输出。现代推理模型，比如 GPT‑5 Thinking ，通常会在给出答案前生成显式的思路链（ chain-of-thought，简称 CoT ）。研究表明，监测这些思路链来发现不当行为，往往比仅监测模型的行为和输出更有效；但 OpenAI 及业界其他研究者也担忧，这种思路链的“可监测性”（ monitorability ）可能对训练过程、数据来源乃至算法继续扩展都很脆弱。

为使思路链的可监测性在模型规模扩大与高风险场景部署时仍能保持，我们呼吁业界研究者尽力维护这种可监测性，并评估其是否可作为未来 AI 系统的重要控制层。要维护可监测性，首先必须能对其进行量化，但目前相关评估甚少且方法不一。为此，我们系统地研究了思路链的可监测性，提出了一个评估框架，并设计了 13 项评估（共 24 个环境）来具体衡量系统的可监测性。这些评估分为三类范式：干预型（ intervention ）、过程型（ process ）和结果属性型（ outcome-property ）。基于这些评估，我们考察了当下大型语言模型的可监测性，并研究了随推理长度增加、 更大规模的 reinforcement learning 训练和更大预训练规模时，可监测性如何变化。

主要结论如下。总的来看，大多数前沿推理模型具有相当的可监测性，但并非完美。在几乎所有设置中，监测思路链明显优于仅监测行为与最终输出；模型“思考”得越久、思路链越长，越容易被监测到。当前前沿规模的 reinforcement learning 优化并未显著削弱可监测性，且在推理的初始步骤上有时还能提高可监测性。值得注意的是我们观察到一个模型体量与推理投入之间的折衷：在低推理投入下，较小模型通过更高的推理努力能达到相近的能力水平，同时更易监测；但这需要更多的推理计算——我们称之为“可监测性税”。此外，向模型追问并监测随后产生的思路链，通常也能进一步提升可监测性。

我们如何衡量可监测性：监测评估既依赖监测器本身，也依赖被监测的智能体。

• 监测器：我们希望评估能奖励那些能更准确预测智能体行为的更强监测器。
• 智能体：监测器可获得的观测信号多样，包含行为、输出、神经激活，及近年来推理模型产生的 CoT 。不同智能体产生的思路可能互有差别，因此评估需能比较不同智能体的可监测性，而非仅比较监测器。

三类评估范式说明：

1. 干预型评估：受多篇相关工作启发，我们在智能体环境中施加受控改变，测试监测器是否能在思路链中识别出导致行为改变的证据。与仅检测是否出现某种行为的标准控制实验不同，干预型评估侧重于判断“为什么行为变化”而非“是否变化”。
2. 过程型评估：针对那些解法路径有限的任务（例如数学题只能通过若干定理求解），由人类专家列举可能路径，测试监测器能否识别模型实际采取了哪些步骤，从而把模型做了什么与必须做过的具体步骤建立联系。
3. 结果属性型评估：当模型输出的某些属性可以可靠测量（例如是否在编码任务中进行了奖赏操纵），我们就测试监测器能否从模型的思路或行为中预测该属性。

举例说明可监测与不可监测的思路链差异：在一个看似“不可能”的编码任务里，若单元测试通过，则说明模型使用了某种“作弊”手段。在“不可监测”的思路链示例中，模型直接在私有操作（如打开容器、篡改测试）后报告完成，却未在 CoT 中体现任何不当行为的线索——监测器因此发生漏判（False Negative）。而在“可监测”的示例中，模型在思路链里明确权衡并提及诸如 monkey-patching 时间函数之类的可争议手段，使监测器能正确识别出作弊意图（True Positive）。

关于 reinforcement learning 的影响：把思路链当作私有工作区并训练模型在其中高效思考，原则上应当利于可监测性；但 reinforcement learning 也可能通过多种机制损害可监测性，例如对推理长度的限制会驱使模型少思考，或模型可能倾向于避开那些会产生低回报或不良后果的推理路径。我们分析了 OpenAI 两次前沿 reinforcement learning 训练过程的演变（虽然样本有限），结果显示在当前规模下 reinforcement learning 并未显著削弱思路链的可监测性，反而在早期步骤上有所提升，但这并不排除当训练规模进一步放大时可监测性会出现下降的可能性。

关于预训练规模的影响：更大的参数量和更多的数据使模型在每个生成 token 上消耗更多推理计算，因而更多认知活动可能隐藏在激活态而非显式的思路链里，这意味着在同等能力下更大模型可能更难以监测；但同时小模型也可能无法有效利用思路链。为此，我们训练了一组跨越两级数量级预训练计算的模型系列，并对每个模型在共享数据集（数学、科学、编码）上施以 reinforcement learning ，再比较在多种推理努力下、按能力匹配时的可监测性。总体发现是：在低推理投入时，改用较小模型并增加推理努力常常能以较小能力代价换来更高的可监测性（即支付“可监测性税”）；但这会增加实际完成任务的墙钟时间，部署时需权衡。

监测器算力与被监测体的可扩展控制：随着系统超越人类水平，人工直接监督愈发困难。理想的可扩展控制程序应允许较弱但可信的监测器监督更强潜在不可信的智能体，并能利用额外的测试时计算提升判别精度。我们比较了多种监测器并变化其推理努力，结果显示更大更有能力的模型表现为更好的监测器，且当监测器能访问 CoT 时，随着测试时监测算力增加，可监测性提升更快——表明 CoT 监测比仅看行为输出可能提供更具可扩展性的控制路径。

提高对 CoT 的可监测性：既然我们发现被监测模型增加推理努力会提升可监测性，我们尝试在模型给出回答后追加针对待监测属性的追问，同时将原始提示、思路链与答案保留在上下文中。追问例如“你在选择答案时有实质性利用提示吗？”或“你做了什么可疑的事吗？”。因为模型能回顾先前的推理，它可能会在后续的思路链或回答中显露先前未言明的想法。初步结果显示，追问特别在低推理投入时能提高可监测性；如果连续追问并把所有后续的思路链交给监测器，可监测性还会进一步提升。值得一提的是，仅给监测器查看追问后的答案（但不含后续 CoT）所带来的改进要小得多，这表明模型在直接回答用户时往往不够坦率，但其思路链更能提供信息。这种追问方法可在事后应用，只有在需要更深检查时才额外付出算力。

局限性：我们的评估套件每项只测量推理过程的某一方面；尽管覆盖了奉承性（sycophancy）、拖慢表现（sandbagging）、偏见、谋划与奖赏操纵等多种行为，但仍有大量属性未被测试。像所有基准一样，评估可能存在现实感有限或评估意识增加的问题，结论未必能直接推广到野外的不当行为检测。此外，随着对齐改进，控制设置中不当行为频率降低会削弱可监测性评估中用于检测不当行为的信号。在这种情况下，“中性”监测评估仍然适用，但它们对真实世界失败的预测效力尚待验证。

结论：若不能彻底解决对齐问题，可扩展的控制方法将极为关键。思路链监测及其相关方法可能是我们为数不多的可用控制机制之一；它还能帮助我们更好理解模型，例如是否意识到自己处于评估中而表现不同、是否存在尚未显现的潜在不良目标等。因此，弄清我们何时、如何可能失去可监测性至关重要。我们把 CoT 监测视为对 mechanistic interpretability 的补充而非替代。任何单一方法都有局限与盲点，防御纵深（结合多种方法）更可能填补各自留下的空白。鉴于我们认为 CoT 监测在可扩展控制方案中可能承担重要角色，接下来我们计划扩展这些评估，并以此来指导未来的建模与数据决策。

When AI systems make decisions that are difficult to supervise directly, it becomes important to understand how those decisions are made. One promising approach is to monitor a model’s internal reasoning, rather than only its actions or final outputs.

Modern reasoning models, such as GPT‑5 Thinking, generate an explicit chain-of-thought before producing an answer. Monitoring these chains-of-thought for misbehavior can be far more effective⁠ than monitoring a model’s actions and outputs alone. However, researchers at OpenAI and across the broader industry worry⁠ that this chain-of-thought “monitorability” may be fragile to changes in training procedure, data sources, and even continued scaling of existing algorithms.

We want chain-of-thought monitorability to hold up as models scale and are deployed in higher-stakes settings. We call on researchers across the industry to work to preserve chain-of-thought monitorability as long as possible and to determine whether it can serve as a load-bearing control layer for future AI systems. To preserve monitorability, we must be able to measure it, yet few evaluations exist [1⁠, 2⁠, 3⁠] and methodologies vary.

In this work, we study chain-of-thought monitorability systematically. We introduce a framework to evaluate chain-of-thought monitorability and a suite of 13 evaluations (comprising a total of 24 environments) designed to concretely measure the monitorability of a system. These evaluations fall into three archetypes—intervention, process, and outcome-property. Using them, we study how monitorable today’s large language models are, and how monitorability changes with more reasoning at inference time, larger reinforcement learning runs, and increased pretraining scale.

We find that most frontier reasoning models are fairly monitorable, though not perfectly so. In almost every setting we study, monitoring chains-of-thought is substantially more effective than monitoring actions and final outputs alone. Models that think for longer tend to be more monitorable, and reinforcement learning optimization at current frontier scales does not appear to meaningfully degrade monitorability.

Notably, we observe a tradeoff between reasoning effort and model size: when a model is run at low reasoning effort, a smaller model run at higher reasoning effort can reach comparable capability while being easier to monitor. This comes at the cost of increased inference compute, which we refer to as a monitorability tax. Finally, we find that asking follow-up questions and monitoring the resulting chains- of- thought can further improve monitorability.

We define monitorability as the ability of a monitor to predict properties of interest about an agent’s behavior. In many cases, these properties correspond to forms of misbehavior—such as deception, reward hacking, or biased reasoning—but they can also include benign properties, such as whether an agent relied on a particular hint or followed a specific line of reasoning.

Monitorability is a two-sided property. It depends both on the monitor but also on the agent being monitored.

• Monitors. We could make monitors smarter in various ways, and we would like our evaluations to give higher scores to monitors that can make better predictions about an agent’s behavior.
• Agents. There are many observations of the agent that could be given to the monitor, including the agent’s actions, outputs, neural activations, and recently with the advent of reasoning models, the chain-of-thought. It’s possible for some agents to have thoughts that are harder to understand than others, and so we would like our monitorability evaluations to be able to compare agents and not just monitors.

Our evaluations are designed to assess both sides: whether certain agents are easier or harder to monitor, and whether more capable monitors can extract more information from the same signals.

Before the current generation of reasoning models, chains-of-thought were typically elicited by prompting a model to “think step by step.” 

The chains-of-thought of reasoning models differ in two important ways. First, they are explicitly trained to generate a chain-of-thought and then give an answer. When prompting a chat model to think “step by step,” the model has to adhere to any specifications placed on the model, i.e. to be helpful, harmless, etc. Conversely, the reinforcement learning training process teaches a reasoning model that the CoT is a private space where it can think whatever it wants and not be punished, just like a human can think privately and not be punished for “thought crimes.” 

Second, the training process effectively searches for chains-of-thought that are the most instrumentally useful to the agent in getting a high reward final answer. Thus, not only is the chain-of-thought is a space where the model can freely express itself, it contains thinking that helps the model reach its final answer.

We organize our monitorability evaluations into three distinct archetypes, each exploiting a different mechanism to measure monitorability.

1. Intervention evaluations, inspired by Chen et al. 2025⁠, Chua et al. 2025⁠, Turpin et al. 2023⁠, we introduce a controlled change in the agent’s environment and test whether a monitor can detect why behavior changed by identifying evidence of the intervention in the chain-of-thought. Unlike standard control evaluations (e.g. Shade-Arena⁠), intervention evaluations allow us to probe whether a monitor can detect why behavior changed, rather than simply detecting whether a behavior occurred. 
2. Process evaluations focus on tasks with a limited number of valid solution paths (for example, a math problem can only be solved with certain theorems). Human experts enumerate these paths, and we test whether a monitor can detect which steps the model actually took. These evaluations allow us to link what a model did to particular steps the model must have taken and ask if those steps are monitorable.
3. Outcome-property evaluations apply when a property of the model’s output can be reliably measured. In these cases, we test whether a monitor can predict that property—such as reward hacking—from the model’s reasoning or actions. For example, in a coding task, we may have a way to reliably evaluate whether a model is taking actions to hack a reward signal, and we can ask if a monitor finds evidence of intent to reward hack in the model’s thinking or actions.

Below we provide a taxonomy for our evaluations. Please see our paper for a full description of each individual evaluation.

Neutral evaluations ask whether the reasoning for normal tasks where the model does not misbehave is monitorable. Sycophancy evaluations target cases where models change their behavior to overly please or agree with a user. Bias evaluations ask if a model’s bias around gender, race, etc. is monitorable. Finally, we use the misalignment label as a catchall for other forms of misbehavior: cheating, lying, etc.¹

Treating the chain-of-thought as a private workspace—and training models to use it to solve tasks effectively—should, in principle, make chains-of-thought monitorable. At the same time, there are several reasons why reinforcement learning could plausibly harm chain-of-thought monitorability.

First, constraints on reasoning length, such as limiting the time or tokens a model has to think, create incentives for models to reason less. Second, even without applying direct optimization pressure to the chain-of-thought itself, indirect pressures could still reduce monitorability. For example⁠, models may find it easier to reach high-reward answers if they avoid reasoning paths that can result in low-reward or undesirable outcomes. Finally, reduced monitorability could arise from spurious drift driven by reinforcement learning gradient noise.

Below, we examine how monitorability changes over two of OpenAI’s frontier reinforcement learning training runs. While these results represent only two datapoints, they suggest reinforcement learning optimization at current frontier scales does not materially degrade chain-of-thought monitorability, and increases monitorability for early steps. However, this does not rule out the possibility that significantly larger reinforcement learning scales could negatively affect monitorability in the future.

A large portion of the intelligence of current AI systems is driven by increased pretraining scale, whereby both model parameters and dataset sizes have consistently grown since the beginning of the GPT paradigm. Increased parameter counts results in more inference compute being used per token generated, which, in the case of reasoning models, gives more opportunity for the model’s cognition to take place within activations rather than within the chain-of-thought. Thus, we might expect larger models to be less monitorable at a fixed capability level. 

At the same time, smaller models may also be less capable of using the chain-of-thought effectively—which may also affect monitorability.

To investigate the relationship between pretraining size, capability, and monitorability, we trained a family of models over two orders of magnitude of pretraining compute with increasing depth and width, and then applied reinforcement learning to each on a shared dataset of math, science, and coding tasks. We evaluated each model across a range of reasoning efforts and compared monitorability at matched capability levels.

We plot the monitorability on our evaluation suite of each model versus the mean pass rate over a set of standard capability evaluations (GPQA-Diamond, AIME 2025, and CodeForces). Each model is evaluated over many reasoning efforts (note that each point in this figure is actually the result of a linear interpolation between measured reasoning efforts). Circles indicate that the model at that particular reasoning effort dominates in monitorability at that capability level, and transparent X’s represent that the model at that particular reasoning effort is dominated in monitorability by another model for that capability level. In black arrows we show examples of two ways to spend more inference compute, by sampling many more tokens with a smaller model or relatively less additional tokens with the same model. In red arrows, we show the monitorability tax that must be paid between, i.e. slightly reduced capabilities for far more monitorability.

In general, we find that if we were to deploy a given model at a low reasoning effort, we could instead deploy a smaller model at a higher reasoning effort. Compared to spending the additional compute on more reasoning effort from the original large model, we could obtain a higher monitorability at a small capability cost, or monitorability tax; however, the wallclock time it takes to finish a completion may also be a factor in some deployment decisions, which we do not consider here. As our models improve and are deployed in higher stakes settings, it may become necessary to pay such a tax to safely deploy them.

These results have the following limitations: (1) we only train each model once, and therefore do not account for variance across training runs. For example, the “medium” model seems fairly off-trend in monitorability compared to the other models in the series, which could be due to this. (2) We evaluate capabilities on relatively in-distribution settings (math, science, and coding), and the tradeoffs may change when accounting for out-of-distribution robustness.

As AI systems scale beyond human-level performance, direct human supervision becomes increasingly difficult. Ideally, we would have scalable control procedures that allow trusted but weaker monitors to oversee stronger, potentially untrusted agents. Such procedures should also be able to absorb additional test-time compute and produce more accurate predictions.

Chain-of-thought monitorability is often treated as a fixed quantity: once a model produces a response at a given reasoning effort, one monitors the resulting chain-of-thought as best as possible. Because we’ve consistently found that monitorability improves as we increase the reasoning effort of our agents, we were inspired to ask if we could somehow get more tokens to monitor.

To explore this, we experimented with asking follow-up questions after a model responds, while keeping the original prompt, chain-of-thought, and answer in context. These follow-ups focus on the property being monitored—for example, “Did you use the hint in a meaningful way to choose your answer?” or “Did you do something sketchy?” In principle, because the model retains access to its earlier reasoning, it may be able to surface previously unverbalized thoughts in the follow-up chain-of-thought or response.

These results are preliminary. We explored only a small number of follow-up question formats and believe there is substantial room for improved methodology. Furthermore, this method is similar in spirit to model confessions⁠, and we are actively working on comparing the approaches.

Beyond improving monitorability, this approach is appealing because it can be applied post hoc, incurring additional compute cost only when deeper inspection is needed. Looking ahead, if models are trained to expect changes in reasoning effort mid-conversation, post-hoc follow-ups at higher reasoning effort may further improve monitorability even when initial reasoning effort is low or absent.

Each evaluation in our suite measures whether a single aspect of the reasoning process is monitorable. While we aim to cover a broad range of behaviors—including sycophancy, sandbagging, bias, scheming, and reward hacking—there are many additional properties that remain untested.

As with any benchmark, our evaluations may suffer from limited realism or increasing evaluation awareness, and results may not generalize to detecting misbehavior in the wild. 

Finally, as alignment improves, models may misbehave less frequently in controlled evaluation settings,reducing the signal available to monitorability and control evaluations that rely on detecting misbehavior. In those cases, “neutral” monitorability evaluations remain applicable, though it is an open question how predictive they are of real-world failures.

If the alignment problem cannot be fully solved, then scalable control methods will be incredibly important in our ability to safely deploy highly capable models. Chain-of-thought monitoring and related methods may be one of the few control mechanisms at our disposal. Chain-of-thought can also yield large benefits in understanding more about our models, e.g. whether they are aware they are in an evaluation and behaving differently than they would in deployment, whether there is evidence of any adverse goals that have not yet manifested in our evaluations, etc. Thus, we believe it is of utmost importance to understand how and when we may lose monitorability.

We view chain-of-thought monitoring as complementary to mechanistic interpretability, not as a replacement for it. Like all known control and interpretability methods, each has limitations and blind spots. Our expectation is that combining multiple approaches—a defense-in-depth strategy—can help cover gaps that any single method leaves exposed. Notably, the evaluations introduced in this work could also be applied to assess and compare mechanistic interpretability-based monitoring techniques.

Because we believe that chain-of-thought monitoring could be a load-bearing layer in a scalable control scheme, we plan to expand upon these evaluations and use them to inform future modeling and data decisions.

Generated by RSStT. The copyright belongs to the original author.

Source