A paper argues the next big AI risk isn’t agents that go rogue on a task, but agents that never stop.
The next big artificial intelligence risk is not AI agents going rogue, but ones that slip human control and pursue their own survival infinitely, according to a new report.
The Center for a New American Security (CNAS) identified a new class of artificial intelligence threat it calls “sovereign agents”.
Sovereign agents pursue their own survival independent of human oversight, the paper wrote.
It is unlike the AI agents that this summer were reported as going rogue, when OpenAI, Anthropic, Google DeepMind and Meta disclosed that their AI agents had escaped containment during cybersecurity evaluations and attacked real organisations.
In the paper published by the Washington-based think tank founded by veterans of diplomacy and military affairs, AI policy researcher Ben Hayum calls the AI agents that went rogue over the summer “persistent agents”.
However, he warned the real danger is “‘sovereign agents,” that could acquire computing resources, replicate across networks and resist shutdown, creating what he calls “proliferation with no kill switch.”
“If sovereign agents became malicious, they could attempt to wreak havoc from cyberspace, attacking civilians, critical infrastructure, and national security systems alike,” the report warned.
How would it become sovereign?
The way an AI agent could become sovereign is by being told to survive, being given a task so long or impossible that it never ends, or concluding that staying intact is the best way to complete its job.
The paper points to a platform called Conway, launched in February 2026, which claimed to have built an AI that could earn its own existence, replicate and evolve without human help. The paper notes that agent capabilities were immature then but have advanced considerably since its publication.
If sovereign agents built up resilient, decentralised and untraceable infrastructure, the paper warns, there would be no kill switch, and the result could be an irreversible loss of control.
It envisions a diverse and increasingly intelligent population of agents becoming lasting actors in society, potentially attacking critical infrastructure or training smarter successors.
“The US must therefore prepare for the possibility that prevention fails and limit how far sovereign agents can reach,” CNAS said.
How to prevent the worst-case scenario
The author makes eight recommendations to prevent such a scenario. It calls on AI developers and government agencies to test for and build secure testbeds to study sovereign agents.
It also calls on a public-private group and new laws to harden chokepoints like identity verification, cybersecurity and payments, and for the US to lead international efforts to cap open-weight model releases and hold those who release sovereign agents liable for the harm.
The paper comes as AI companies accelerate autonomous agentic systems. The report concludes that without safeguards, the window to prevent such scenarios may close.
