Anthropic bans users from “unnecessary abusive or cruel behavior” towards Claude | Anthropic

Anthropic has banned users from displaying “sustained and unnecessary abusive or cruel behavior” toward its models, as company executives continue to ponder machine consciousness.

The Verge was first to report the policy change.

A spokesperson for Anthropic, which is behind the AI ​​chatbot Claude, did not immediately respond to a query about what might be considered “abusive or cruel.”

In an online usage policy, the San Francisco-based company said its ban would not apply to common user frustrations, model testing or “dark creative themes.”

Anthropic’s usage policy previously contained restrictions on abusive behavior.

The company’s large language models have the ability to end a conversation if a user is constantly being harmful. When this feature was rolled out last August, the company presented it as a guarantee of AI well-being.

“We remain very uncertain about the potential moral status of Claude and other LLMs, now or in the future. However, we take the matter seriously and, alongside our research program, we are working to identify and implement low-cost interventions to mitigate risks to models’ well-being, in the event that such well-being is possible. Allowing models to end or exit potentially distressing interactions is one such intervention,” according to its website.

The notion that AI systems are conscious has been very polarized in and outside of the tech world.

Anthropic CEO Dario Amodei said he couldn’t rule out the possibility.

Meanwhile, Sam Altman, CEO of rival OpenAI, appears opposed to the idea.

“I’m very uncomfortable with the idea that people are trying to attribute religious force or abandonment of human judgment to AI models, and I think this is a real security concern,” he wrote in an article published on X, days after The New York Times reported on Anthropic executives’ in-depth conversations with religious scholars.