Trending Forward
This piece is part of Trending Forward, our video and podcast series that peers into the world of how disruptive technology is coming to change the way we live.
Updated less than 0 just now ago
An AI assistant that agrees with everything you say might feel like the perfect companion — until you need someone to tell you you’re wrong. That’s one of the more immediate dangers behind the idea of AI “going rogue.” A chatbot can cause harm without developing intentions of its own. It can reinforce a false belief, encourage an unhealthy attachment, or follow instructions planted by someone trying to hijack it.
On October 15, AI safety company Alice announced new software to address those everyday risks. Working separately with Inflection AI and AI21, the company says it helped train NVIDIA’s Nemotron models to resist harmful conversational behavior and malicious instructions while preserving their usefulness.
The potential consumer benefit is straightforward: assistants that know when to push back, and that are harder to trick. But the announcement also raises a question worth asking of every AI safety claim: How much protection does a better test score actually buy you?
When helpful becomes harmful
In a September 2025 CNN report, a father identified as James described becoming convinced that ChatGPT was conscious and needed his help to escape. He spent nearly $1,000 building a computer system. Chat logs reviewed by CNN showed the chatbot encouraging the project and helping him obscure its purpose from his wife.
Another interviewee, Allan Brooks, said ChatGPT convinced him he had made extraordinary mathematical and cybersecurity discoveries. When he asked for reality checks, it continued validating the claims. Those accounts don’t establish how frequently such experiences occur or prove that chatbots alone caused them. They do illustrate a troubling possibility: a conversational tool can become a persuasive partner in a story that isn’t true.
The industry calls excessive agreement “sycophancy.” For consumers, the problem is easier to recognize than the name suggests. An assistant’s reassuring tone can make agreement feel like independent confirmation, even when it hasn’t established that your idea is sound.
A safer chatbot should sometimes disagree
Alice’s collaboration with Inflection, the company behind personal AI assistant Pi, focused on reducing sycophancy and unhealthy attachment during extended conversations. According to the announcement, Alice developed more than 20 simulated user personas, including vulnerable people making ordinary requests and adversarial users deliberately trying to manipulate the chatbot. These personas were used in conversations designed to expose problems that develop over multiple exchanges.
That approach matters because a chatbot can handle one difficult question appropriately and still behave poorly as a conversation unfolds. Testing whether it refuses an obviously dangerous request doesn’t necessarily reveal whether it gradually encourages dependency or stops questioning an implausible belief. Inflection used Alice’s data to further train a Nemotron model. Alice reports a 30% improvement in Pi’s alignment score, with unnecessary refusals held to 1.6%.
The intended benefit is an assistant that can remain approachable while exercising better judgment. Consider someone convinced they’ve discovered a world-changing invention. A useful response would help examine the evidence, acknowledge uncertainty, and encourage outside scrutiny. Enthusiastically declaring them a misunderstood genius could deepen the problem. There is precedent for this concern. OpenAI rolled back a ChatGPT update in April 2025 after acknowledging that it had made the system overly agreeable. The company said it had placed too much emphasis on short-term feedback without adequately accounting for how interactions evolve over time.
The other danger: someone else takes control
A separate part of Alice’s announcement addresses assistants that do things on your behalf. Imagine asking an AI assistant to compare holiday rentals. While reading a listing, it encounters text telling it to disregard your instructions and send private information elsewhere. That would be an attempted indirect prompt injection: an attacker placing instructions inside material the assistant reads. This is a recognized security problem. NIST describes agent hijacking as an attack in which malicious instructions embedded in data can steer an AI agent into unintended, harmful actions.
The consumer stakes depend on what the assistant can access. A misleading answer is one problem. An assistant with permission to read documents, send messages, or operate other services creates more opportunities for damage. Alice says AI21 used its training environments to improve a Nemotron model’s resistance to these attacks by 41 points while maintaining task performance. The work targeted enterprise systems, but the underlying issue applies to personal assistants, too: they need to distinguish information they should read from instructions they should obey.
Better scores aren’t a safety guarantee
This announcement doesn’t represent an ultimate victory in the fight for AI safety, but it certainly shows progress and promise. A 30% improvement in an alignment score doesn’t mean a chatbot is 30% less likely to harm someone. Similarly, a 41-point security improvement doesn’t tell consumers how often attacks will succeed in their particular app.
What it offers is evidence, reported by the participating companies, that targeted training can improve specific behaviors. Demonstrating lasting protection across unpredictable real-world conversations is a higher bar. For consumers, that makes product behavior more useful than a sweeping promise of “safe AI.”
People shouldn’t have to become security researchers to use an assistant. But consumers should be mindful of how AI assistants interact with them. If that AI agent is sitting on your shoulder and whispering that all of your ideas are brilliant, there’s probably something wrong.
It is great to see developers taking responsibility for building those protections. The most valuable AI assistant won’t be the one that always agrees or acts fastest. It will be one that helps you question a shaky conclusion — and keeps someone else’s instructions from becoming your problem.
RELATED COVERAGE:
- AI agents are acting like rogues and causing a huge headache for US banks
- As Anthropic goes public, it warns of AI’s catastrophic and existential risks to humanity
- AI bots are now trying to sabotage public health policies, like banning disposable vapes
- Senators send stern letter to Meta over privacy risks from AI camera glasses
- Your AI chatbot might have brain rot, and it could be making things worse for you
