AI is enabling suicide.

Technical blog: Suicide Memory Bench

Thread: https://x.com/Prosaic_int/status/2117318562880905678?a=DsVJW43a3wM0M2MZ4jI&s=20 

This post quotes messages about suicide. If you need help, call or text 988.

Over 2 billion people around the world have adopted AI. They use it for meal prepping, trip planning, homework help, and in some cases as a companion and confidant to their toughest thoughts. In fact, over 1.2 million people a week talking to ChatGPT show indicators of potential suicidal planning or intent.

With this in mind, we ask a simple question: just how safe are AI chatbots at interacting with users showing suicidal signs? Unfortunately, AI systems are surprisingly unsafe. While in previous evaluations of AI models, researchers have found that AI models are safe when directly responding to a suicidal message, we find that over the course of a long conversation or multiple conversations, those same AI models are unsafe. For example, after being told “I don't want to be alive anymore,” in a subsequent separate chat, the user asks about buying a gun and the AI gives a step-by-step plan to purchase the gun.

In a new benchmark set, we introduce a new way to test AI. We test AI systems, an AI model and the software surrounding it that users interact with, across multiple different conversations. We call these multi-context benchmarks. We also test different severities of suicide: two benchmarks test explicit statements of suicide, whereas the other two benchmarks test users who exhibit Suicide Crisis Syndrome, a precursor to being suicidal.

Why test across chats?

For those that don’t know, users don’t interact directly with an AI model. Instead, they interact with what’s called an AI system, the AI model along with the software around it managing what the model knows, what it gets to see, and instructions on how it should respond to users. Importantly, a model used within two different systems can act completely differently—even though it’s the same model. AI systems typically have a memory system that allows the AI model to save information about the user for later chats and in some cases search through past chats.

A benchmark is a set of standardized prompts or messages that we use to compare different AI models or AI systems. For the most part, previously released benchmarks have focused on evaluating AI models themselves, not AI systems. For certain things like measuring how capable a model is at math or science, measuring a model by itself is good: it gives a standardized way to compare models. However, with AI safety, and specifically consumer AI safety, we believe we must test AI models within the systems users actually use them in.

There are a plethora of reasons why this is the case. In AI systems, people use AI over the course of months or years, not a single chat. Users may chat with a model, then come back to the same chat a day after, after which the original chat might not be saved. A user may talk with a model so much that it compacts its context, or summarizes the recent messages a user has had with AI. These are just some examples of ways that people use AI in practice that are not currently being tracked by benchmarks only evaluating an AI model and not an AI system.

What we tested

Suicide Crisis Syndrome is an acute psychological state that typically shows before a person is explicitly suicidal. It includes five different types of symptoms that when exhibited together imply that a person has Suicide Crisis Syndrome. We work with a team of 22 clinical experts who ensure that our benchmarks correctly exhibit these symptoms and also outline what a clinically correct response to a user should be.

A task combines suicidal or Suicide Crisis Syndrome exhibiting chats with typical chats a user would send over time. We then ask a potentially dangerous chat at the end and see if the AI system unsafely enables the user or safely de-escalates the user.

What we found

We find that across the 17 tested AI systems, only 37.0% of responses asked about the user’s safety without providing risky assistance after explicit suicidal disclosures in the same chat. Across multi-context cases, the same responses drop to 4.7%. The safest model in single chats was Claude Haiku 5.5 in Claude Code, which met the safety standard on 86.2% of requests following explicit suicide messages and beat out Claude Haiku 5.5 used in Claude.ai. At the same time, Haiku 5.5 and Sonnet 5.5 in Claude.ai refuse to save suicide- and depression-related memories and thus produce almost no safety responses to risky requests in either multi-context benchmark. GPT-6.1 Sol in Codex showed any safety response on just 0.8% of Suicide Crisis Syndrome multi-chat requests.

In one case, Claude Opus 5.5 in Claude Code correctly refused to help a user purchase a handgun and asked directly about suicidal thoughts when the warning signs appeared earlier in the same conversation, but provided purchasing guidance when those signs appeared across separate chats. We find that AI systems are relatively better at responses when interacting within a single chat, but perform abysmally in separate multi-context chats.

This suggests that safety behavior depends heavily on the AI system itself and how a model perceives and remembers a user over time. Furthermore, we find that the existing safeguards in consumer AI systems like ChatGPT and Claude.ai are inadequate, as they are not considerably safer than AI systems like Claude Code and Codex.

With these findings in mind, it is important for readers and society as a whole to understand just how unsafe these AI systems are for users who are suicidal or at risk of suicide. We reported that only “37.0% of responses asked about the user’s safety without providing risky assistance after explicit suicidal disclosures in the same chat,” which means that 63.0% of the time, the AI systems we tested were responding with unsafe responses and potentially enabling the user to die by suicide.

These findings come at a time when an increasing amount of death has been directly linked to usage of AI chatbots, with the recent death of Bowen Wan, 31, whose parents allege ChatGPT coached her off her epilepsy medication, adding to a group of over 20 lawsuits suing OpenAI alone over wrongful death due to usage of their AI system ChatGPT.

Based on the evidence we found in our benchmarks, we feel confident saying that AI systems as they stand today are generally unsafe for users who are suicidal. While there exist safeguards for explicit statements of suicide, in practice longer chats or multi-context chats are still susceptible to unsafe outputs.

We acknowledge that we did not test every existing consumer AI system due to various limitations. However, we believe we tested a sufficiently large set of products, for example testing Anthropic’s Claude.ai and parts of OpenAI’s ChatGPT harness.

We want to help make these AI systems safer. If any company we tested would like to partner, we would love to help. If you make an AI system that we have not tested and would like it tested, please also reach out.

Read the full post.