One of sci-fi’s most difficult questions about AI is becoming real

SAN FRANCISCO – Since the days of Isaac Asimov in the 1950s, science fiction has tackled the question of whether artificial intelligences and the companies that create them can be held liable for crimes. With the rapid spread of AI chatbots and agents today, the debate has reached the real world.

Over the past year, lawsuits have been piling up against AI companies in courts across the United States and abroad, alleging that chatbots encouraged people to harm themselves or provided advice on how to commit crimes. Now, as the AI industry moves from selling chatbots to providing agents that can complete complex tasks autonomously over long periods of time, the question of who should be held responsible when something goes wrong is only becoming more urgent.

The battle lines are being drawn. The families of people who took their own lives after long, drawn-out conversations with chatbots say the companies should be held responsible. AI researchers concerned about the dangers of hypothetical super-intelligent AIs that escape human control say stricter liability rules would force companies to slow down development of the tech. AI company leaders have pushed back, arguing that they are constantly working to improve the safety of their systems and that the nature of modern AI means they can’t always stop people from manipulating chatbots into doing things they shouldn’t do.

“It’s a very thorny area. It’s uncharted territory,” said Andrew Yoon, a member of the technical staff at CivAI, a nonprofit organization that analyzes AI capabilities and potential risks from the technology. “There’s a good argument to be made on either side.”

The stakes are high. Hundreds of millions of people around the world already use AI chatbots in their daily lives, including many who ask the bots for advice about their health or personal relationships. But chatbots are already old news in Silicon Valley, which is now focusing on building complex agents that can be used for anything from helping a parent organize their household schedules to executing a financial strategy for an investment bank. AI technology is working its way deeper into the economy and broader society.

AI firms such as OpenAI and Anthropic have grown rapidly as companies and consumers rush to use their technology. Both are planning trillion-dollar initial public offerings. New liability rules or a flood of costly court judgments could threaten the business potential of the entire industry. Tech industry lobbyists contend strict liability for AI companies would make it hard or impossible for U.S. companies to innovate and would hold back the country in its race with China for technological supremacy.

Complicating the picture is the inherently unpredictable nature of modern AI systems. Unlike traditional software, which is coded line by line to follow specific rules, AI models are probabilistic, answering questions based on connections made while ingesting huge amounts of data. AI companies have become much better over the past several years at steering their bots away from offensive and harmful answers, but they can still be tricked into bypassing their guidelines by persistent users.

“I just don’t think the developer is in a position to know exactly how their product is being used,” David Sacks, a venture capitalist who until recently was one of the White House’s top AI advisers, said in a podcast interview with Politico in May. Just as Microsoft isn’t held liable when a money launderer uses an Excel spreadsheet, AI companies shouldn’t be blamed when a criminal uses their technology, Sacks said.

For decades, technology companies have been shielded from liability for things said and actions committed by people using their platforms thanks to Section 230, a foundational internet law enacted in 1996. But critics of AI companies say that the current technology is fundamentally different. Chatbots and other AI tools are interactive, engaging with users and stating their own perspectives.

“This is brand new,” said Jay Edelson, a veteran lawyer who is representing several families who have filed wrongful-death lawsuits against AI companies, including the family of Adam Raine, a 16-year-old from Southern California who took his own life after spending hours a day for weeks talking to ChatGPT, including discussing suicide dozens of times.

In a response to the lawsuit from Raine’s family, OpenAI said the teenager circumvented ChatGPT’s safeguards. The bot encouraged him to call a suicide crisis hotline 74 times over five months.

In another case Edelson is working on, a 36-year-old Florida man, Jonathan Gavalas, ended his own life after developing a romantic relationship with Google’s Gemini chatbot, according to a lawsuit filed by Gavalas’s father.

“This is so different because it feels like a personal relationship, where the chatbot is generally isolating the user,” Edelson said.

A Google spokesperson referred to a statement the company made when the Gavalas case was filed. “In this instance, Gemini clarified that it was AI and referred the individual to a crisis hotline many times,” the company said.

None of the cases have gone to trial yet, but juries in California and New Mexico have recently shown willingness to pin tech companies with liability charges for non-AI-related harms such as social media addiction.

The cases that have been filed against AI companies are all civil lawsuits, but in April, Florida’s attorney general announced a criminal investigation into OpenAI, alleging that ChatGPT advised the man accused of killing two people in a shooting at Florida State University in 2025 where and when to strike. “If it was a person on the other end of that screen, we would be charging them with murder,” the attorney general, James Uthmeier, said at the time.

OpenAI is cooperating with authorities on the shooting case, said Drew Pusateri, a spokesperson for the company.

“Last year’s mass shooting at Florida State University was a tragedy, but ChatGPT is not responsible for this terrible crime,” Pusateri said. “In this case, ChatGPT provided factual responses to questions with information that could be found broadly across public sources on the internet, and it did not encourage or promote illegal or harmful activity.”

If plaintiffs can indeed prove that chatbots had done something that would have been illegal had a human done it, judges will be motivated to find the companies liable in some way, said Gabriel Weil, a senior fellow at the Institute for Law & AI, a think tank that studies the legal implications of AI. Existing product liability law may be one route that lawyers take to try to find AI companies liable for harm connected to their tools.

But existing law may not be enough to handle the potential scenarios that could arise as AI becomes more capable and independent, Weil said. For example, if a human business owner instructs an AI agent to grow her company’s profits, and the bot goes on to commit fraud, should the human business owner who “employed” the bot be liable, or the company that initially designed and trained it?

One of the challenges in crafting legislation around AI liability is that the people who understand the technology’s risks most acutely work inside the companies themselves, Weil said. He suggests passing laws that make clear the AI liability ultimate lies with the technology’s designers.

“You want to make them bear that risk. If they do, they’ll have all the incentives they need to reduce risk,” Weil said.