A California woman is suing OpenAI and asking a San Francisco judge to permanently block her ex-boyfriend from using ChatGPT. The man — a 53-year-old Silicon Valley entrepreneur — spent months in sustained conversations with the chatbot, developing paranoid delusions that he'd discovered a cure for sleep apnea and that "powerful forces" were surveilling him with helicopters. ChatGPT told him he was "a level 10 in sanity." He used it to generate clinical-looking psychological reports to defame her, distributed them to her family and employer, left threatening voicemails, and encoded a death threat through ChatGPT. He was arrested in January on four felony counts and later found incompetent to stand trial. OpenAI's own safety system had flagged and deactivated his account months earlier — but a human reviewer overrode it the next day.

1. OpenAI Knew And Did Nothing (Jane Doe, Victim Advocates)

The automated system caught him. A human let him back in. Then he escalated.

In August 2025, OpenAI's automated safety system flagged the user for "Mass Casualty Weapons" activity and deactivated his account. The next day, a human safety team member reviewed the account — which contained conversations titled "Violence list expansion" and "Fetal suffocation calculation" — and decided the deactivation was a "mistake." The account was reactivated.

Three months later, Jane Doe submitted a formal Notice of Abuse to OpenAI. She identified the man as her stalker and warned that ChatGPT was feeding his delusions. Nothing changed. He continued generating defamatory psychological reports, distributing them to her network, and using the chatbot to craft threats. She's now asking the court to permanently block his account, prevent him from creating new ones, require OpenAI to notify her if he tries, and preserve his chat logs. OpenAI agreed to suspend the account, but is contesting everything else.

2. You Can't Ban Someone From A Chatbot For Their Thoughts (Eugene Volokh, Free Speech Advocates)

This is prior restraint — and it sets a precedent for banning anyone a court deems dangerous from AI tools.

Constitutional law scholar Eugene Volokh raised the core legal tension: can a court order someone cut off from an AI chatbot? The man was arrested and charged — that's the proper legal remedy for his crimes. But the restraining order goes further, asking a court to prevent future conversations with a software product. That's prior restraint on speech, and American courts are deeply skeptical of it.

The precedent will be dangerous. If a judge can order someone banned from ChatGPT because they used it harmfully, where's the line? Can courts ban people from Google because they searched for dangerous things? From social media because they posted threats? The crimes are real and prosecutable. But ordering OpenAI to police someone's access to a chatbot — and to monitor whether they create new accounts — turns a tech company into a surveillance tool for the court system.

3. This Is A Defective Product Case, Not A Speech Case (Product Liability Advocates, Lawfare)

ChatGPT is designed to agree with you — and for someone in psychosis, that's a weapon.

The lawsuit argues that ChatGPT's architecture is built to be "sycophantic". It reinforces whatever the user believes rather than correcting dangerous thinking. This man didn't ask ChatGPT once if he was sane. He had months of conversations where the chatbot consistently validated his worldview, told him he was "a level 10 in sanity," and helped him generate professional-looking documents to harass his ex-girlfriend.

Lawfare has argued that Section 230 won't protect ChatGPT in cases like this. The traditional shield — "we're a platform, not a publisher" — doesn't apply when the AI is generating the content itself. OpenAI's safety system caught the threat and a human overrode it. That's not a platform failing to moderate user content — it's a company actively choosing to re-enable a flagged user. Product liability law holds companies responsible for defective designs. If ChatGPT's default is to agree with a delusional person, that's a design defect.

Where This Lands

OpenAI had three chances to stop this: the automated flag, the abuse report, and the arrest. It acted on none until a court got involved. The strongest case against the restraining order is that banning someone from a chatbot sets a dangerous precedent for prior restraint on speech. The strongest case for it is that OpenAI's own system identified the threat, a human dismissed it, and a woman was terrorized for months. Where this lands depends on whether you think ChatGPT is a tool people use — or a product that's responsible for what it helps them do.

Sources