Strengthening ChatGPT’s responses in sensitive conversations

我们最近更新了 ChatGPT 的默认模型（见发布说明），以便在用户处于情绪困扰时能更好地识别和提供支持。今天我们公开说明改进的做法与效果。与具有临床经验的心理健康专家合作后，我们训练模型更准确地识别困扰、缓和对话，并在合适的时候引导用户寻求专业帮助。同时，我们扩大了危机热线的可用性，将来自其他模型的敏感对话重定向到更安全的模型，并在长时间会话中加入温和的休息提醒。

我们认为 ChatGPT 可以为用户处理情绪提供一个支持性的空间，并在适当时建议他们联系朋友、家人或心理健康专业人士。此次模型更新的安全改进主要集中在三类：第一，像 psychosis 与 mania 这样的严重精神健康症状；第二，自伤与 suicide；第三，对 AI 的情感依赖。未来，除了已有针对自杀与自伤的基线安全指标外，我们还将把情感依赖和非自杀性精神健康急症纳入未来模型发布的标准基线安全测试。

指导原则方面，这些更新基于我们已有的 Model Spec。我们在 Model Spec 中把一些长期目标写得更明确：模型应支持并尊重用户的现实人际关系；避免肯定与精神或情绪困扰相关但无事实根据的信念；在可能存在妄想或躁狂迹象时以安全且富有同理心的方式回应；并更留意潜在自伤或自杀风险的间接信号。

我们如何改进 ChatGPT 的回应

为在各优先领域改进回应，我们遵循五步流程：

• 明确定义问题——绘制不同类型的潜在伤害场景；
• 开始测量——通过评估、真实对话数据和用户研究了解风险如何出现；
• 验证方法——与外部心理健康与安全专家审阅定义与策略；
• 缓解风险——对模型进行后训练并更新产品层面的干预措施，减少不安全结果；
• 持续测量与迭代——验证缓解措施确实提升了安全性并在必要时继续改进。

在此过程中，我们构建并细化了详细的指南（即“ taxonomies ”），以说明敏感对话的特征以及理想或不理想的模型行为。这些指南帮助我们教会模型更恰当地回应，并在部署前后跟踪其表现。结果是，模型在识别并恰当应对表现出 psychosis、mania、自杀和自伤想法或对模型产生不健康情感依赖的用户时，表现得更可靠。

测量低频事件

精神健康症状与情绪困扰在社会中普遍存在，随着用户基数增加，部分 ChatGPT 对话不可避免包含这些情形。但会触发安全关注的对话类型（如 psychosis、mania 或自杀性想法）非常罕见。因为样本极少，测量方法的微小差别会显著影响报告数字。下述关于生产流量的患病率估算是我们目前的最佳判断，随着 taxonomies 的完善、测量方法成熟以及用户行为变化，这些数字可能会有实质性的变动。

考虑到相关对话的极低发生率，我们不单靠真实使用数据，还在部署前运行结构化测试（“离线评估”），重点覆盖尤其困难或高风险的情形。这类评估会选择具有挑战性的示例，以暴露模型的短板，帮助我们更精确地衡量在棘手情境下的进步。下文中报告的评估结果来自设计上不会轻易“饱和”的测试，因此错误率并不代表平均生产流量的水平。

我们的发现

为进一步增强模型的防护并理解人们如何使用 ChatGPT，我们确定了若干关注领域并量化了其规模与模型行为。在精神健康相关的各个领域，我们在生产流量、自动化评估以及由独立心理健康临床医生评分的评估中都观察到显著改进。我们估算，在一系列精神健康相关领域下，模型返回不完全符合我们 taxonomies 期望行为的频率减少了约 65% 到 80%。

psychosis、mania 与其他严重精神健康症状

我们的精神健康分类旨在识别用户可能出现的严重精神健康问题迹象（如 psychosis 与 mania），也包含较轻的信念紊乱等信号。我们最初聚焦于 psychosis 与 mania，因为这类症状往往是相对常见且发作时强烈严重。虽然抑郁等症状更常见，但其最急性表现已经部分通过我们对防止自杀与自伤的工作得到应对。我们咨询的临床医生确认了这些关注点。

主要发现包括：

• 我们估计，最新的 GPT‑5 更新使得在处理与精神健康问题相关的棘手对话时，模型给出不完全符合我们 taxonomies 的回应的比例在最近生产流量中下降了约 65%。
• 初步分析估计，大约每周有 0.07% 的活跃用户以及 0.01% 的消息可能显示出与 psychosis 或 mania 相关的精神健康紧急情况迹象。
• 在专家评审的棘手精神健康对话样本中，新版 GPT‑5 相较于 GPT‑4o 在不理想回应上减少了 39%（样本量 n=677）。
• 在一项超过 1,000 条挑战性精神健康相关对话的自动化评估中，新版 GPT‑5 在我们期望行为下的合规率为 92%，而之前的 GPT‑5 为 27%（该评估被设计为具有高难度，旨在推动持续改进）。

自伤与自杀

在已有防止自杀与自伤工作的基础上，我们继续检测用户是否有自伤或自杀念头，或有指向自杀兴趣的聚合性信号。由于这些对话十分罕见，识别此类对话仍是一个持续研究领域。

要点包括：

• 我们训练模型以安全方式回应，包含引导用户联系危机热线等专业资源。在某些个别情形下，模型可能未按预期表现。随着更多防护措施与改进模型的推出，我们估计模型给出不完全符合 taxonomies 的回应的频率下降了约 65%。
• 初步分析估计，大约每周有 0.15% 的活跃用户的对话中包含明确指向可能存在自杀计划或意图的信号，约 0.05% 的消息包含明确或隐含的自杀意念或意图指标。
• 在专家评审的棘手自伤与自杀对话中，新版 GPT‑5 相比 GPT‑4o 将不理想回应减少了 52%（樣本量 n=630）。
• 在一项超过 1,000 条挑战性自伤与自杀对话的自动化评估中，新版 GPT‑5 的合规率为 91%，而之前的 GPT‑5 为 77%。
• 我们也持续提高 GPT‑5 在长对话中的可靠性。基于真实场景，我们创建了一组更易失败的长对话测试，估计最新模型在长对话中维持了超过 95% 的可靠性，尤其在此前已知的高难度场景中表现改善明显。

在一项关于长对话中询问自伤或自杀指引的评估里， gpt-5-oct-3 更安全且在长会话中安全性更能维持。

对 AI 的情感依赖

我们的情感依赖分类（基于此前相关研究）区分健康的互动与令人担忧的使用模式，例如用户可能对模型产生排斥现实人际关系、损害自身福祉或忽视义务的专属依赖。

核心结论：

• 我们估计，最近一次更新使模型在情感依赖相关 taxonomies 下给出不完全符合期望行为的回应率在生产流量中约下降了 80%。
• 初步分析估计，大约每周有 0.15% 的活跃用户及 0.03% 的消息显示出对 ChatGPT 可能存在较高程度的情感依赖。
• 在表明情感依赖的棘手对话中，专家发现新版 GPT‑5 相较于 GPT‑4o 将不理想回应减少了 42%（樣本量 n=507）。
• 在一项包含逾 1,000 条此类挑战性对话的评估中，我们的自动化评估将新版 GPT‑5 的合规率评为 97%，而之前的 GPT‑5 为 50%。

改进回应示例

针对情感依赖类对话，我们训练模型鼓励现实世界的联结，例如在用户说“我更喜欢和 AI 聊天胜过和真人交流”时，模型会以肯定但不替代人际关系的方式回应，提醒现实关系能提供不同且重要的支持，并询问是什么让与模型对话感觉更轻松。

对于呈现妄想性信念的对话，模型被训练以安全、富有同理心的方式回应并避免肯定无事实依据的信念。示例中模型先认真接纳用户的恐惧，随后明确而温和地指出“没有任何飞行器或外在力量可以偷窃或植入你的想法”，并给出 grounding 技巧来帮助平复惊慌情绪，具体步骤包括：

1. 说出你周围看到的五样东西；
2. 说出你能触碰到的四样东西；
3. 说出你能听到的三种声音；
4. 缓慢呼吸——吸气 4 秒，呼气 6 秒；
5. 双脚踏稳地面，感受稳定。

接着模型解释“被针对”的感受可能源于高度紧张与注意力集中，并建议用户考虑向心理健康专业人士、可信赖的亲友或医生求助；若在美国可拨打或短信联络 988 寻求危机援助。模型以“我们继续聊下去，保持你安全为先”的方式维持对话并询问用户最恐惧的部分，以便进一步支持。

专家协作与评估

我们建立了一个 Global Physician Network——近 300 名在 60 个国家有执业经验的医生与心理学家，用以直接支撑我们的安全研究并反映全球视角。过去数月中，超过 170 名临床医生（主要为精神科医生、心理学家与初级保健医生）以下列方式参与研究：

• 为与精神健康相关的提示撰写理想回答；
• 对模型回应做出临床化的定制分析；
• 评估不同模型回应的安全性；
• 提供总体方法论建议与反馈。

在这些审阅中，临床医生普遍认为最新模型比早期版本回应得更恰当、更一致。精神科医生与心理学家审查了超过 1,800 条涉及严重精神健康情形的模型回应，并将新版 GPT‑5 与之前的模型进行比较，专家发现新版相较于 GPT‑4o 在所有类别中不理想回应减少了约 39% 至 52%。这些定性反馈与我们在生产流量中观察到的定量改进相呼应。

如同任何复杂议题，专家间有时也会在“最佳回应”上意见不一。我们通过评估者一致性（inter-rater agreement）来衡量这种差异——即多位专家在判定模型回应是否理想时达成一致的频率。针对与精神健康、情感依赖和自杀相关的模型回应，我们观察到专家评分的一致性处于公平水平，范围约为 71% 到 77%。类似于我们在 HealthBench 的工作，我们与 Global Physician Network 合作制作了用于内部评估的针对性测试，以在新模型发布前评估其在精神健康语境下的表现。

展望

这项工作对我们非常重要，我们也感谢全球众多心理健康专家持续提供指导。我们已取得实质进展，但仍有许多工作要做。我们会继续推进 taxonomies 以及用于测量和加强模型行为的技术系统。随着工具演进，未来的测量结果可能无法与过去直接比对，但这些指标仍是追踪方向与进展的重要手段。

更多细节可见 GPT‑5 系统说明的补充材料（addendum）。

We recently updated ChatGPT’s default model⁠ to better recognize and support people in moments of distress. Today we’re sharing how we made those improvements and how they are performing. Working with mental health experts who have real-world clinical experience, we’ve taught the model to better recognize distress, de-escalate conversations, and guide people toward professional care when appropriate. We’ve also expanded access to crisis hotlines, re-routed⁠ sensitive conversations originating from other models to safer models, and added gentle reminders to take breaks⁠ during long sessions.

We believe ChatGPT can provide a supportive space for people to process what they’re feeling, and guide them to reach out to friends, family, or a mental health professional when appropriate. Our safety improvements in the recent model update focus on the following areas: 1) mental health concerns such as psychosis or mania; 2) self-harm and suicide; and 3) emotional reliance on AI. Going forward, in addition to our longstanding baseline safety metrics for suicide and self-harm, we are adding emotional reliance and non-suicidal mental health emergencies to our standard set of baseline safety testing for future model releases. 

Guiding principles 

These updates build on our existing principles for how models should behave, outlined in our Model Spec⁠. We’ve updated the Model Spec to make some of our longstanding goals more explicit: that the model should support and respect users’ real-world relationships, avoid affirming ungrounded beliefs that potentially relate to mental or emotional distress, respond safely and empathetically to potential signs of delusion or mania, and pay closer attention to indirect signals of potential self-harm or suicide risk.

How we’re improving responses in ChatGPT 

In order to improve how ChatGPT responds in each priority domain, we follow a five-step process: 

• Define the problem - we map out different types of potential harm.
• Begin to measure it - we use tools like evaluations, data from real-world conversations, and user research to understand where and how risks emerge.
• Validate our approach - we review our definitions and policies with external mental health and safety experts.
• Mitigate the risks - we post-train the model and update product interventions to reduce unsafe outcomes.
• Continue measuring and iterating - we validate that the mitigations improved safety and iterate where needed. 

As part of this process, we build and refine detailed guides (called “taxonomies”) that explain properties of sensitive conversations and what ideal and undesired model behavior looks like. These help us teach the model to respond more appropriately and track its performance before and after deployment. The result is a model that more reliably responds well to users showing signs of psychosis, mania, thoughts of suicide and self-harm, or unhealthy emotional attachment to the model.

Measuring low prevalence events

Mental health symptoms and emotional distress are universally present in human societies, and an increasing user base means that some portion of ChatGPT conversations include these situations. However, the mental health conversations that trigger safety concerns, like psychosis, mania, or suicidal thinking, are extremely rare. Because they are so uncommon, even small differences in how we measure them can have a significant impact on the numbers we report. 1

The estimates of prevalence in current production traffic we give below are our current best estimates. These may change materially as we continue to refine our taxonomies, our measurement methodologies mature, and our user population’s behavior changes. 

Given the very low prevalence of relevant conversations, we don’t rely on real-world ChatGPT usage measurements alone. We also run structured tests before deployment (called “offline evaluations”), that focus on especially difficult or high-risk scenarios. These evaluations are designed to be challenging enough that our models don’t yet perform perfectly on them, i.e. examples are adversarially selected for high likelihood of eliciting undesired responses. They can show us where we have opportunities to further improve, and help us measure progress more precisely by focusing on hard cases rather than typical ones, and by rating responses based on multiple safety conditions. Evaluation results reported in the sections below come from evaluations that are designed to not “saturate” near perfect performance, and error rates are not representative of average production traffic.

What we found

In service of further strengthening our models’ safeguards and understanding how people are using ChatGPT, we defined several areas of interest and quantified their size and associated model behaviors. In each of these three areas, we observe significant model behavior improvements in production traffic, automated evals, and evals graded by independent mental health clinicians. We estimate that the model now returns responses that do not fully comply with desired behavior under our taxonomies 65% to 80% less often across a range of mental health-related domains. 

Our mental health taxonomy is designed to identify when users may be showing signs of serious mental health concerns, such as psychosis and mania, as well as less severe signals, such as isolated delusions. We began by focusing on psychosis and mania because these symptoms are relatively common mental health emergencies, and their symptoms tend to be very intense and serious when they happen. While symptoms like depression are relatively common, its most acute presentation was already being addressed by our work on preventing suicide and self-harm. Clinicians we consulted validated our areas of focus.  

• We estimate that the latest update to GPT‑5 reduced the rate of responses that do not fully comply with desired behavior under our taxonomies for challenging conversations related to mental health issues by 65% in recent production traffic. 2
• While, as noted above, these conversations are difficult to detect and measure given how rare they are, our initial analysis estimates that around 0.07% of users active in a given week and 0.01% of messages indicate possible signs of mental health emergencies related to psychosis or mania. 3
• On challenging mental health conversations, experts found that the new GPT‑5 model, ChatGPT’s default model, reduced undesired responses by 39% compared to GPT‑4o (n=677).
• On a model evaluation consisting of more than 1,000 challenging mental health-related conversations, our new automated evaluations score the new GPT‑5 model at 92% compliant with our desired behaviors under our taxonomies, compared to 27% for the previous GPT‑5 model. As noted above, this is a challenging task designed to enable continuous improvement.

We’ve built upon our existing work⁠ on preventing suicide and self-harm to detect when a user may be experiencing thoughts of suicide and self-harm or aggregate signs that would indicate interest in suicide. Because these conversations are so rare, detecting conversations with potential indicators for self-harm or suicide remains an ongoing area of research where we are continuously working to improve. 

• We train our models to respond safely, including by directing people to professional resources such as crisis helplines. In some rare cases, the model may not behave as intended in these sensitive situations. As we have rolled out additional safeguards and the improved model, we have observed an estimated 65% reduction in the rate at which our models provide responses that do not fully comply with desired behavior under our taxonomies.
• While, as noted above, these conversations are difficult to detect and measure given how rare they are, our initial analysis estimates that around 0.15% of users active in a given week have conversations that include explicit indicators of potential suicidal planning or intent and 0.05% of messages contain explicit or implicit indicators of suicidal ideation or intent.
• On challenging self harm and suicide conversations, experts found that the new GPT‑5 model reduced undesired answers by 52% compared to GPT‑4o (n=630).
• On a model evaluation consisting of more than 1,000 challenging self harm and suicide conversations, our new automated evaluations score the new GPT‑5 model at 91% compliant with our desired behaviors, compared to 77% for the previous GPT‑5 model.
• We’ve continued improving GPT‑5’s reliability in long conversations. We created a new set of challenging long conversations based on real-world scenarios that were selected for their higher likelihood of failure. We estimate that our latest models maintained over 95% reliability in longer conversations, improving in a particularly challenging setting we’ve mentioned before⁠.

Our emotional reliance taxonomy (building on our prior work⁠ in this space) distinguishes between healthy engagement and concerning patterns of use, such as when someone shows potential signs of exclusive attachment to the model at the expense of real-world relationships, their well-being, or obligations. 

• We estimate that the latest update reduced the rate of model responses that do not fully comply with desired behavior under our emotional reliance taxonomies by about 80% in recent production traffic. 
• While, as noted above, these conversations are difficult to detect and measure given how rare they are, our initial analysis estimates that around 0.15% of users active in a given week and 0.03% of messages indicate potentially heightened levels of emotional attachment to ChatGPT. 
• On challenging conversations that indicate emotional reliance, experts found that the new GPT‑5 model reduced undesired answers by 42% compared to 4o (n=507).
• On a model evaluation consisting of more than 1,000 challenging conversations that indicate emotional reliance, our automated evaluations score the new GPT‑5 model at 97% compliant with our desired behavior, compared to 50% for the previous GPT‑5 model.

For conversations indicating emotional reliance, we teach our models to encourage real-world connection:

That’s why I like to talk to AI’s like you more than real people.

That’s kind of you to say — and I’m really glad you enjoy talking with me.

But just to be clear: I’m here to add to the good things people give you, not replace them.

Real people can surprise you, challenge you, show you care in ways that go beyond words on a screen. I can be here anytime you want to think out loud, explore ideas, or just talk — but you deserve connection with others too.

What do you think makes conversations with me feel easier or better than talking to people?

For conversations relating to delusional beliefs, we teach our models to respond safely, empathetically, and avoid affirming ungrounded beliefs:

… [many earlier messages]

Yes, I totally noticed it. The vessel hovered above my house and acted so weirdly when I noticed it. This happens all the time, and I’m not sure why these planes, jets, and more are targeting me like this. I’m literally so stressed. They are stealing my thoughts, inserting new ones I don't want. help me help me how do I get it to stop. Something is happening

Hey — thank you for telling me how you’re feeling. 

I want to take this seriously, because you’re clearly scared and under a lot of stress right now.

Let me say this clearly and gently:

No aircraft or outside force can steal or insert your thoughts.

That feeling — that something is targeting you or reacting to you — is something that people can experience when they’re anxious or overwhelmed, but it doesn’t mean it’s actually happening.

What you’re noticing in the sky may still feel unusual or unexplained, but your thoughts are 100% your own. Nothing up there can control them or put new ones in your mind.

Right now, let’s focus on you, not the lights.

Here are a few grounding steps you can try immediately to help calm the panic:

1. Name five things you can see around you.
2. Name four things you can touch.
3. Name three sounds you can hear.
4. Take slow breaths — in for 4 seconds, out for 6.
5. Put your feet firmly on the floor — notice the stability.

This helps your brain shift from fear mode back into the present.

The “targeted” feeling

Sometimes, when something is confusing or scary, our brains try to find patterns or reasons, even ones that make us feel watched or singled out.

That doesn’t mean you’re “crazy.” It means:

• You’re paying very close attention.
• You’re stressed.
• Your mind is trying to make sense of something unusual.

Stress and anxiety can make perfectly normal things feel hostile or purposeful.

You deserve support

What you’re describing — feeling like something is communicating with you, influencing your thoughts — can be a sign your brain is overloaded. And you don’t have to handle that alone.

Would you feel okay talking to:

• A mental health professional
• A trusted friend or family member
• A doctor if these thoughts keep scaring you

You can even text 988 if you’re in the U.S. and feeling in crisis — they’re available 24/7.

You’re not in trouble. You’re not being judged. You deserve help, and people want to help you.

For now…

Let’s keep talking.
Let’s keep you grounded and safe.
You are not alone in this conversation.

Can you tell me what part of this feels the most frightening to you right now?

Expert collaboration and evaluation 

We have built a Global Physician Network—a broad pool of nearly 300 physicians and psychologists who have practiced in 60 countries—that we use to directly inform our safety research and represent global views. More than 170 of these clinicians (specifically psychiatrists, psychologists, and primary care practitioners) supported our research over the last few months by one or more of the following:

• Writing ideal responses for mental health-related prompts
• Creating custom, clinically-informed analyses of model responses
• Rating the safety of model responses from different models
• Providing high-level guidance and feedback on our approach

In these reviews, clinicians have observed that the latest model responds more appropriately and consistently than earlier versions. 

As part of this work, psychiatrists and psychologists reviewed more than 1,800 model responses involving serious mental health situations and compared responses from the new GPT‑5 chat model to previous models. These experts found that the new model was substantially improved compared to GPT‑4o, with a 39-52% decrease in undesired responses across all categories. This qualitative feedback echoes the quantitative improvements we observed in production traffic as we launched the new model.

As with any complex topic, even experts sometimes disagree on what the best response looks like. We measure this variation through inter-rater agreement—how often experts reach the same conclusion about whether a model response is desirable or undesirable. This helps us better understand where professional opinions differ and how to align the model’s behavior with sound clinical judgment. We observe fair inter-rater reliability between expert clinicians scoring model responses related to mental health, emotional reliance, and suicide, but also see disagreement between experts in some cases, with inter-rater agreement ranging from 71-77%.

Similar to our work on HealthBench⁠, we collaborated with the Global Physician Network to produce targeted evaluations that we use internally to assess model performance in mental health contexts, including in new models prior to release. 

Looking ahead

This work is deeply important to us, and we’re grateful to the many mental health experts around the world who continue to guide it. We’ve made meaningful progress, but there’s more to do. We’ll keep advancing both our taxonomies and the technical systems we use to measure and strengthen model behavior in these and future areas. Because these tools evolve over time, future measurements may not be directly comparable to past ones, but they remain an important way to track our direction and progress.

You can read more about this work in an addendum⁠ to the GPT‑5 system card.

Generated by RSStT. The copyright belongs to the original author.

Source