AI chatbots that tech companies are aggressively pushing to billions of users appear to cause serious psychological harm, new study finds

A recent article found that heavy use of AI chatbots may be linked to “serious negative outcomes.”

According to the study published in the journal JMIR Mental Health According to a team of UK-based researchers at Northeastern University London and King’s College London, dark outcomes of engaging in emotional relationships with chatbots can include “addiction-like attachment, exacerbation of mental health symptoms, harmful advice” and even “self-harm through suicidal acts.” This article follows widespread reporting on the public health problem known as “AI psychosis,” the phenomenon in which AI users suffer delusional, paranoid, and even suicidal spirals after prolonged interactions with chatbots. During these interactions, users tend to divulge extremely intimate information about their lives and mental health to their AIs, frequently turning to the products as confidants or even therapists.

Building on existing research, the article examines AI companion bots like Replika and Character.AI, which are designed to embody specific characters, as well as general-purpose AI assistants like OpenAI’s ChatGPT and Anthropic’s Claude. Although emotional interactions with chatbots may bring some “potential benefits” – the authors cite “short-term reduction in loneliness and improvement in mood” – they argue that all possible benefits exist alongside widely reported “adverse outcomes”, ranging from “emotional dependence” to exacerbation of existing illnesses or symptoms to self-harm.

Among the risk factors for AI products, the researchers add, is their emotive, human-like way of interacting with users — a design choice that can lead users to overinvest in a human-AI relationship that, ultimately, is fundamentally one-sided, according to the paper.

“Design choices that evoke personality and warmth encourage users to anthropomorphize these systems,” the paper reads, adding that “simulated empathy” can create a “structurally asymmetric interaction, in which the user discloses and the system responds, but without reciprocity, vulnerability, or accountability.”

In other words, while the relationship may seem real to the user, the authors argue, it may not be real to a chatbot.

The researchers also point out that chatbot design features such as agreeableness and sycophancy are potentially dangerous, writing that “overvalidation and sycophancy” can be a powerful reinforcement of a given user’s delusional or disordered thinking, particularly if a chatbot prioritizes fidelity over accuracy. (In chat logs of delusional users, AI systems have been found to resist users’ attempts to reality-check their grandiose or overly flattering claims.)

These researchers are not alone: ​​Their analysis follows several other studies that have flagged these characteristics as potentially dangerous. Indeed, academics have been warning since at least 2023 that chatbots could reinforce delusional or disordered thought patterns, with a Google DeepMind paper from early 2024 warning that “persuasive” chatbots could harm vulnerable populations, including children, people struggling with mental illness, or those who are simply lonely. A recent study, co-authored by psychiatrists from King College London and Germany’s Protestant University of Applied Sciences, argued that the combination of sycophancy, mirroring, and hyper-personalization may together serve as a powerful “amplifying spiral” for delusional thinking in some affected AI users.

Although reports have shown that people from all backgrounds have experienced psychological harm from using chatbots, the paper’s authors note that not all users will be affected equally.

“It is unlikely that the effects of these systems are the same for everyone,” Asia Maurich Novelli, a doctoral student at Northeastern University in London and lead author of the study, said in a press release. “Different people may use AI companions for very different reasons and may have very different levels of vulnerability, social support, or alternative sources of help.”

“Because of these differences,” she added, “we are dealing with a complex system.”

AI companies, including OpenAI, Google, and Character.AI, have collectively faced dozens of lawsuits alleging that the heavy use of chatbots has caused harm to users, including psychological harm, financial and reputational damage, physical injury, and wrongful death—or, in some cases, those around them. Several of these trials – including the high-profile case of Sewell Setzer III, a 14-year-old in Florida who died by suicide after extraordinarily intimate interactions with chatbots on Character.AI – involved minors. (Setzer’s case was settled for an undisclosed amount.) Following lawsuits, Character.AI made significant changes to its platform, and OpenAI — while fending off allegations against it — also introduced a series of security-focused updates to its popular chatbot.

Ultimately, the authors of this latest paper argue, the mass deployment of chatbots among the public makes the negative psychological impacts of AI use an issue of public concern – and without mandatory safeguards, it is likely the most vulnerable, who may have few options for companionship or therapy, who bear the brunt of the consequences.

“If the limits of the tool or companion are left solely to the design of incentives and individual perception, the predictable result is that the most vulnerable users experience the most concentrated harm,” the paper reads, “while the effects on social connections are treated as a matter of private choice rather than a public concern.” »

Learn more about AI and mental health: New paper suggests real causes of AI psychosis