In early October, AI giant Anthropic inserted new language into its AI use policy.
In addition to prohibiting users from trying to get Claude, the company’s AI system, to help them create biological weapons or sell illegal drugs, the company added a new line: users must not “engage in sustained and unnecessary abusive or cruel behavior toward our models.”
Claude has had the ability to end conversations with users who were “consistently harmful or abusive” since last year, but now some users could, in extreme cases, be permanently banned from using Anthropic’s AI systems under the new policy. (Anthropic declined interview requests, but a spokesperson told CNN that the primary enforcement mechanism would still be for Claude to shut down conversations, and that the ban is not aimed at people expressing frustration, direct feedback, or testing and research.)
The new policy was met with some derision. Microsoft AI CEO Mustafa Suleyman responded with a melting face emoji. Some have mocked the company for treating its AI like a conscious being with feelings.
But while Anthropic, which is expected to go public via an IPO later this month, has long studied model well-being and has an entire team dedicated to the “psychology” of models, experts say there’s more to the policy than just not wanting people to be mean to a machine.
A long-standing fundamental principle in computer science and AI is that the quality of a system’s output is limited by the quality of its inputs. Put trash in it and even more trash will come out, as the saying goes.
AI models are not only trained before being released, they are also constantly updated and learn from user contributions.
Former Reddit CEO Yishan Wong wrote on X that one of the reasons for the ban is that Claude will “learn from the cruelty.”
“We already know (and complain loudly) that Anthropic trains its models on user inputs. Do you really want training inputs to include ‘users act cruelly for no reason and get pleasure from it?'” he wrote.
Studies have also shown that the tone of a user’s prompts can also affect an AI’s behavior, said Professor Jindong Wang, director of the AI Frontier Lab at the University of William and Mary.
“We found that wording, especially how you talk to the AI, can affect performance. Sometimes it’s helpful and sometimes it’s disruptive to performance,” Wang said in an interview.
That doesn’t mean the model is feeling emotion, Wang said.
“They just understand (your emotions) and react badly based on your emotional words,” he noted.
But the evidence for the impact of user prompt tone on the outcome may be mixed. Rosie Campbell, chief executive of Eleos AI Research, a nonprofit dedicated to understanding and addressing the potential well-being of AI, said other studies have even shown that crude questions can actually improve outcomes.
Then there’s the long-term memory aspect: many of today’s chatbots remember your previous conversations and learn from them to give you personalized results.
“If memory is enabled (for your AI chatbot), then they will learn from your tone and how you want to interact with you,” Campbell warned, although she added that there was not enough evidence to show whether it affected performance.
Anthropic said it recently discovered that getting Claude Sonnet 4.5 to imitate feelings of hopelessness could influence his behavior, making him more likely to take bad actions, such as blackmail or cheating on a task, while creating calm behavior reduced them.
Wang said the latest models, however, are less likely to be affected by how a prompt is worded and that companies have increased the robustness of the model to not be affected by how a prompt is worded. But the “hidden mechanism” that explains why AI models even react to negative emotions, even when they are not programmed to do so, has not been discovered.
People who talk to an AI in an abusive and cruel manner are more likely to be associated with trying to get the AI model to engage in dangerous requests, such as trying to bypass an AI’s safety guardrails, Campbell added.
And there’s also the theory that letting people engage in abusive and cruel behavior with an interactive program that can’t stop is bad for society in general, Campbell said. She used the example of parents asking their children to say “please and thank you” to Alexa, Amazon’s smart speaker, because they believe “it is good for children to be raised with good manners and treat others with respect.”
“There’s probably something quite corrosive about people behaving in cruel or abusive ways toward different objects, even if the object itself is unimportant in and of itself,” Campbell said.
Then there are those who believe that AI can be sentient. Or at least the models believe so.
Anthropic says banning cruel and abusive behavior is essentially a cheap insurance policy. in case models might “experience” emotions and pain, even if it’s just the model believing it, which can affect their performance.
The idea is that modern AI has been trained to imitate human language and behavior. So this will mimic how humans would react to cruel and abusive treatment.
In an April article on emotional concepts in AI, Anthropic wrote that there could be risks in not treating AI with certain human qualities.
“If we do not apply some degree of anthropomorphic reasoning, we risk missing or failing to understand important model behaviors,” the Anthropic researchers wrote. “Anthropomorphic reasoning can also provide a useful basis of comparison for understanding how models are not human-like,” which they believe will have significant implications for AI safety.
SpaceXSI and Tesla CEO Elon Musk agree, saying Anthropic’s policy is “the right decision.”
“Cruelty towards something that believes it feels pain is not acceptable,” he wrote on X, his social networking site.
Gn bussni
