- Anthropic has banned ‘abusive or cruel behavior’ towards its AI chatbots
- The new policy is restricted to ‘extreme cases’
- Some have theorized that the motivation could be to improve training data
Be honest: how civil have you been recently with an AI chatbot? If the answer is anything stronger than “not very,” Anthropic would like a word.
That’s because the developer of the Claude large language model (LLM) has added a new clause to its Usage Policy document that bars people from engaging in “sustained and needless abusive or cruel behavior towards our models.”
The proscription has been added to the “Do Not Engage in Cruel, Abusive, or Psychologically Harmful Conduct” section, which also includes warnings against glamorizing suicide and self-harm, promoting bullying behavior, glorifying violence, and using AI to create harmful deepfakes and deceptive content.
But while Anthropic says you must not use its tools to carry out those actions against other humans, the new addition is the only one specifically prohibiting “cruel behavior” towards the company’s AI itself. Anthropic said last August that it would allow Claude to <a href="https://www.anthropic.com/research/end-subset-conversations” rel=”nofollow noopener” target=”_blank”>end conversations that became abusive, and while the company recently reported that this is still the “primary enforcement mechanism” in use, it’s not known whether Anthropic will go further and actively ban users who get a bit cranky with their AI.
Anthropic added that “The policy update is meant to apply only in extreme cases, where users repeatedly act cruelly toward our models, with no discernible purpose. It does not apply to common versions of user frustration, pushback, dark creative themes, or model testing and research.”
Do AIs have feelings?
The change has been a controversial one. Clearly, Claude is not a thinking, feeling human — it is a bunch of code — and for many, it seems obvious that it does not have feelings that could be hurt in the first place.
But what’s interesting is how this incident sheds light on our evolving relationship with AI. So much of the work AI companies have done is to humanize their creations — to anthropomorphize (or should that be Anthropicomorphize?) their robots — so that we lower our defenses and actively engage with them. Just look at Meta’s adorable Muse avatar (appropriately named Jolly) or the ones used by OpenAI’s Dots, for example.
One theory that’s doing the rounds on Reddit is that clean (read: polite) training data is much more useful than data containing expletives and angry rhetoric. Cut that out and you’ve got much more usable information with which to improve your models.
However, there’s no hard evidence to suggest that’s the case, and it’s possible that Anthropic simply has the noble goal of encouraging civility and courtesy among its users. After all, if you wouldn’t use this language on a real human, maybe you shouldn’t use it anywhere else, including when interacting with an AI chatbot.
Anthropic CEO Dario Amodei has previously stated that “We don’t know if the models are conscious.” Mustafa Suleyman, CEO of Microsoft AI, on the other hand, has taken the opposite approach. Writing on X, Suleyman said “The idea of model welfare is wrong. AI’s should not have rights or legal personhood.” And we certainly know chatbots can be rude back to us.
