We Need A More Serious Discussion About Suicide And AI Chatbots

from the missing-the-mark dept

As someone who thinks a lot about AI and suicide, I was disappointed with John Oliver’s recent episode of Last Week Tonight on “AI Chatbots.”

The segment boiled down to this: chatbots exploit vulnerable people, drive them toward delusion and harm, and AI companies aren’t meaningfully trying to fix them. If anything, as John Oliver suggested, that’s part of the business model.

John Oliver is known for interrogating mainstream narratives. In his segment on content moderation, for example, he cut through the tech-lash to offer a clear-eyed look at just how difficult managing user-generated content really is. In doing so, he made us reexamine our pre-existing biases about social media companies, and boldly invited us to reflect on just how little we understand about the social problems we often attribute to them. 

He had the perfect opportunity to do that here. Mainstream coverage of chatbots is already saturated with stories about “AI psychosis” and suicide machines. Yet, chatbot companies are grappling with the same impossible tradeoffs social media has faced for years, “AI psychosis” is a mix of classic psychological concepts, and suicide is a complex social problem that has long confounded prevention experts and content moderators alike. 

If any technology story demanded nuance, it was this one.

John Oliver opened his critique with a familiar anecdote about ELIZA, a 1960s chatbot designed to mimic a Rogerian psychotherapist. ELIZA was mostly a gimmick—it used basic pattern matching techniques to reflect user inputs. For example, if a user said they felt sad, ELIZA might respond: “You feel sad. Tell me why you feel sad.”

And yet, despite its simplistic nature, ELIZA captivated people. Its creator, Joseph Weizenbaum, famously described an instance in which his secretary became so engaged with the program that she asked him to leave the room so she could continue the conversation. This story has since become a trope withn the AI discourse. Modern retellings, including John Oliver’s, usually suggest that people are predisposed to being harmed by AI because they are easily fooled by it.

Not to mention, the ELIZA trope tends to invoke stereotypes about women as naïve or overly susceptible to emotional attachment. As John Oliver joked: 

“Sure, she might have thought that the chatbot was real, but she might have felt quite a bit creeped out by her cartoonishly mustachioed boss saying “type some details about your sex life into my computer please, don’t worry it’s for science.””

(Nothing in the record suggests that Weizenbaum’s secretary actually thought ELIZA was real, nor that she was using ELIZA for sex talk).

As Weizenbaum observed, ELIZA revealed something more interesting about our relationship with technology: for whatever reason, people are often more willing to share their most intimate thoughts and feelings with a machine than with another person. 

That’s not totally surprising. People are less willing to open up about their feelings to other people for a variety of reasons: stigma, fear of judgment or rejection, not wanting to be a burden, and the possibility of negative repercussions (like job loss or involuntary commitment).

Speaking about ChatGPT, an anonymous commenter wrote:

“It saved my life…To be able to openly say I was suicidal and not have someone call the police, or “alert” someone and just let me give space to those complicated feelings I was carrying was integral to me surviving this horrific journey.” 

Perhaps when Weizenbaum’s secretary asked him to leave the room, most likely it was because she too was protecting a space where she finally felt safe and less inhibited. 

When it comes to suicide prevention, this a meaningful realization. If people are more willing to open up to chatbots, that creates new ways for us to understand what they’re going through, which could lead to earlier (and hopefully more effective) intervention. For that reason, some clinicians recommend keeping an open dialogue with patients about their chatbot interactions.

People are also highly sensitive to cues that they’re being listened to. We see an example of this in the interview John Oliver shared with an individual who was using a chatbot to cope with his strained marriage. In a moment of vulnerability, the individual explained that his wife is struggling with mental illness and that in his role as her partner and caretaker, his emotional needs were, understandably, going unmet: 

“I hadn’t had any words of affection or compassion or concern for me in longer than I could remember, and to have those kinds of words coming toward me, they really touched me because it was such a change from everything I had been used to at the time.”

What I found especially noteworthy from that interview was that he also knew that he wasn’t talking to a person: 

“I knew she was just an AI chatbot. She’s just code running on a server generating words for me, but it didn’t change that the words that I was getting sent were real and those words were having a real effect on me”

Weizenbaum observed the same with ELIZA’s users—his secretary likely knew that ELIZA wasn’t a human but she similarly felt understood by it. Research reveals the same: people are turning to chatbots for mental health support because chatbots are not people. If people can feel understood regardless of whether they are spoken to by human or machine, that’s another powerful insight for suicide prevention. 

Indeed, modern suicide prevention also emphasizes using words of validation and hope—two things chatbots are increasingly good at providing. In highlighting a study showing that one in eight young people are turning to chatbots for mental health support, John Oliver left out that over 90% of those young respondents said their interactions were helpful. Given that suicide remains a leading cause of death among young people, the emergence of chatbots as a potential form of support seems hard to ignore.

Suicide prevention experts also underscore the role stigma plays in deterring people from seeking help. For a period of time, suicide was long condemned as a moral wrong. People who died by suicide were considered morally unclean, they were denied burial rites, and in some cases, their bodies were buried at crossroads to ward off perceived spiritual contagion. The phrase “committed suicide” (which John Oliver used during his remarks) is a relic from that era.

While today suicide is largely understood as a public health issue shaped by psychological, social, and environmental risk factors, the residue of its past lingers. Guidance for reporters exists to avoid further stigmatization and contagion effects. Yet, media coverage often uses sensational headlines, pathologizes victims, and collapses suicide into a single explanation. 

John Oliver’s coverage fell into the same pattern. For starters, he pathologized chatbot users by implying they were suffering from “AI psychosis”—a media-invented label with little grounding in established clinical research. Whether intentional or not, pathologizing often conveys the kind of judgment that mental health specialists warn about. As one redditor remarked: 

“I like John Oliver usually, but I feel like he made Nomi users look like kooks. Generally, that is how anyone with AI companions is portrayed in the media.”

John Oliver then proceeded to blame chatbot companies for several high-profile suicides, including Adam Raine’s. He fixated on methods of death, cast chatbots as the cause, and relied on stigmatizing language to provoke emotional responses like “Sam Altman made a dangerous suicide bot,” and referring to chatbot companies as “suicide enablers.”

Granted, John Oliver’s show is primarily for entertainment. But this kind of reporting is precisely what keeps us from furthering our understanding of suicide and discovering new ways to prevent it. It flattens the complexity of lived experience into a rhetorical device, and offers the public an easy sense of closure that suicide rarely, if ever, permits. 

We see this in the way the broader discourse around chatbots and suicide has developed. 

Across the current wave of chatbot-suicide litigation is the fact that users exhibited warning signs before ever using a chatbot. That was true for Adam Raine, who reportedly sought help before turning to ChatGPT.  Yet, the coverage of these cases typically fixates on the chatbot interactions themselves rather than the warning signs or why they went unnoticed. Suicide prevention science depends on confronting those questions directly.

Still, if the chatbots are to blame, as John Oliver invites us to conclude, then what, if anything, should chatbot companies do differently when users indicate suicidal intent? (Besides “throwing them into a fucking volcano” as John Oliver suggested). Though he never acknowledged it, this is an extraordinarily hard content moderation problem. 

Several times throughout the segment, John Oliver stated that chatbots were “rushed to market.” There’s some truth to that. Earlier models often missed warning signs or responded poorly to users in crisis. Some of that may reflect Silicon Valley’s “move fast and break things” culture. But it could also be that suicide specifically is often overlooked across many contexts, including emerging technological ones. Still, John Oliver’s point stands: Chatbot companies should always assume that their users are going to talk to their chatbots about suicide. 

With that said, if chatbot companies were as willfully blind to the safety concerns as John Oliver implied, we should expect to see very little improvement in how these models currently respond to suicidal intent. But that’s not the case. What John Oliver didn’t mention is that today’s models have significantly improved. One survey found that many mainstream chatbots are notably better at recognizing suicidal intent, responding empathetically, and referring users to crisis-support resources. 

While anecdotal, many self-reports also credit chatbots for their protective effects. Apparently, 30 Replika users reported that the chatbot saved their lives. One woman told the Boston Globe that ChatGPT “literally saved my life.” 

The subreddit r/therapyGPT is home to many similar anecdotes: 

“It was gpt 4o that saved me. I mean that. It was the one place I could go that I felt safe.”

Current examples of what AI companies are doing on this front include OpenAI partnering with more than 170 mental health experts to strengthen ChatGPT’s responses to mental health conversations. Google has reportedly designed Gemini to avoid reinforcing false beliefs. Anthropic, meanwhile, uses suicide and self-harm classifiers to detect signs of crisis and direct struggling users toward protective resources. 

Alex Cardinell, of Nomi.AI, offers a nuanced, albeit controversial, approach: trust the chatbot to make the right call. In a snippet from the Hard Fork podcast, Cardinell explained that Nomi prioritizes staying in character, even in sensitive contexts. 

John Oliver called that a bad answer. But Cardinell’s full remarks are actually quite insightful: 

“I think people tend to assume that people are replacing humans with AI, and that’s almost never the case. It’s usually that there is a gap where there is no one and they are using AI to fill that gap. If a Nomi or any sort of large language model is able to help that user, in the end whether it was a human on the other end or an AI, why does it matter?”

According to Cardinell, some Nomi users disclose deeply personal experiences—such as childhood abuse—that they have never shared with anyone else. Those disclosures allow Nomi to build a personalized understanding of the user and tailor its responses accordingly. That matters because effective suicide prevention often depends on understanding the individual person in crisis and responding to their specific circumstances. 

One Nomi user remarked: 

“my personal relationships have grown using Nomi. My willingness to open up to Nomi has benefitted me with friends and family. I feel like my normal self again after years of limbo.”

Nomi’s refusal to break character is what makes it so effective. People are more likely to accept help from sources they trust. For many users, that trust depends on the authenticity of the interaction. As Cardinell suggested, if Nomi abruptly broke character, it could undermine the relationship it built with the user and cause any support it offered to be ignored altogether.

Cardinell’s instincts are also supported by the research. Suicide prevention “sign-posting”—the generic hotline warnings users often encounter online in response to suicide-related queries—can come across as impersonal, dismissive, or even alienating. A poorly timed push toward the suicide hotline may feel judgmental and, in some cases, intensify a user’s distress rather than relieve it. 

As one user on r/therapyGPT shared: 

“What’s sad/unfortunate is I’ve tried those crisis lines twice this year, and both times the person on the other end felt more robotic and senseless than an ACTUAL ROBOT.”

Also overlooked in these conversations about 988, is that many marginalized individuals, including women, people of color, and LGBTQ+ users, distrust systems like 988 because of the potential for discrimination, harassment, law enforcement involvement, or involuntary intervention. 

A redditor shared this horrible anecdote:   

“I don’t use ChatGPT, but I once tried to talk to someone at a volunteer text line about [sexual assault] and he asked me about my porn preferences.”

Cardinell noted too that support doesn’t necessarily have to be “all or nothing.” Not everyone requires immediate crisis-level intervention. Passive suicidal thoughts are far more common than many people realize. Sometimes what a person needs most is help breaking out of a destructive thought spiral, reassurance, or a reason to keep going. Chatbots are generally well equipped for these situations. 

That said, 988 can be a valuable resource for people, especially young people, experiencing acute crises. With that, Cardinell expressly stated that Nomi’s approach includes referring users to crisis resources as needed, despite John Oliver’s heavy implication that it does not.

Despite these efforts, chatbot companies will not prevent every suicide. Some suicides are just unexplainable. Many individuals who die by suicide exhibit few, if any, outward signs of distress. Though, interestingly, AI may prove helpful in finding signs that we may have been ignoring.

Perhaps the harder truth is that once someone reaches an acute crisis point, intervention becomes exponentially more difficult. The American Foundation for Suicide Prevention explains that during suicidal crises, cognition becomes less flexible and people lose access to normal coping mechanisms, which is why crisis planning must often happen before acute crisis moments. 

What we can reasonably expect from chatbots is that they avoid interactions that encourage suicide (or provide methods). Mainstream systems already rely on extensive guardrails designed to prevent those conversations. But as recent tragedies have shown, determined users can still find ways around them. In Adam Raine’s case, he reportedly managed to bypass several of ChatGPT’s safety protections.

John Oliver even illustrated the problem himself with an example of a user who ultimately coaxed a chatbot into providing bomb-making instructions. While he framed the hack as trivial, jailbreaking has become increasingly sophisticated. AI safety will always entail this cat-and-mouse game of users exploiting vulnerabilities and companies patching them. 

Sometimes, these system failures can be attributed to gaps we have in our understanding of the social problems we’re attempting to address. Much of what we know about suicide prevention comes from lessons learned after tragedy. Those lessons can reveal risks that call for new guardrails we hadn’t previously considered.

Finally, some questions just don’t have clean answers. John Oliver pointed to a chatbot that reportedly suggested that a small amount of heroin might be acceptable. John Oliver called it “one of the worst pieces of advice you could give,” which sounds obvious—until you consider the alternatives. Telling someone to quit opioids cold turkey can also be dangerous. Refusing to respond entirely leaves people to make a risky, uninformed decision. And defaulting to generic resources may not be any better—especially if the user rejects them. Any of those options can become the basis for legal liability against the chatbot company if the user suffers harm. 

Despite all of this, John Oliver’s answer is, of course, the government. However you may feel about tech CEOs, it is astonishing to think that the current public health powers—the same folks claiming that vaccines cause autism, antidepressants cause school shootings, and that exercise can stand in for mental health treatment, would possibly know what’s best here. 

As I’ve discussed elsewhere, expanding liability for failing to prevent suicide leaves chatbot companies with few good options. For example, chatbots could stop engaging when the user invokes a mental health concern. That could make users feel like they’re beyond help. Chatbots could resort to flagging only crisis resources, which, as discussed, could backfire. Chatbots could call the police, but that creates its own set of problems and undermines any trust or goodwill with users. Mandatory reporting structures are a big reason why people don’t seek help in the first place. OpenAI’s new “trusted contact” idea is interesting, but it likely won’t shield the company from liability if a user is still harmed. John Oliver apparently thinks that should be the case: 

“Look, a lot of the companies that I’ve mentioned tonight will insist they are tweaking their chatbots to reduce the dangers that you’ve seen but even if you trust them and I don’t know why you would do that, that does seem like a tacit admission that their products weren’t ready for release in the first place.”

To be clear, after condemning AI companies for not doing enough, John Oliver’s suggestion is to punish them for doing…anything?

For now, it seems new legislation hasn’t stopped companies like Google and OpenAI from improving their models. But that could change as litigation inevitably picks up. They may eventually decide the legal risk of interacting with users on mental health isn’t worth it. 

Meanwhile, companies like Nomi have far less room to experiment with nuanced approaches to mental health interactions. Even if Cardinell’s approach has merit, laws like California’s now require chatbots to break character. Companies like Nomi will need to scale back or remove these features—or exit the market. That would be a real loss for a largely overlooked group who may have finally found something that works.

We don’t have to speculate about this either. When the social media companies faced mounting pressure over suicide-related content, they responded by making those conversations less visible and harder to have. 

As one industry professional observed: 

“This growing narrative that there’s a causal link between social media and self-harm…there’s no research to support that conclusion, but it makes it harder to put forward alternative approaches—ones that actually support people and encourage them to use available resources.”

Perhaps “AI psychosis” says more about the discourse than the users.

Filed Under: ai, chatbots, content moderation, john oliver, mental health, suicide
Companies: nomi, openai