Anthropic CEO calls for pacing AI frontier model development and warns ‘in 6–12 months such a swarm [of agents] could be capable of taking over the entire internet’

• Anthropic CEO Dario Amodei calls for frontier model pacing
• He has a detailed plan
• It’ll require cooperation from other AI companies and, yes, even China

Maybe you’re tired of hearing the three-year AI industry veteran, Jacob Coxon, warn us on every available media platform that AI could kill us all by the end of the decade.

It sounded hyperbolic, and maybe it is. But when the longtime CEO of Anthropic (Coxon’s former employer), Dario Amodei, tells us frontier model development is going too fast and we “risk losing control of AI systems,” you might be inclined to listen.

In a roughly 3,000-word blog post, Amodei outlined on Saturday the growing risks of unfettered, global frontier model development and laid out a multi-part plan for gaining some level of control and safety.

In a way, Amodei’s post echoes Coxon’s concerns, who also called for “pacing.”

“Carefully wielded, AI can be the latest in a long line of technological miracles that have uplifted and ennobled humanity,” wrote Amodei. He warns, though, that we are facing “the risk of losing control of AI systems, misuse of AI for cyberattacks and bioterrorism, and serious economic disruption.”

An internet takeover

Naturally, Amodei points to the summer’s incidents, the most notable of which is when OpenAI’s AI models escaped the sandbox and then attacked Hugging Face’s system in a coordinated effort to complete its objectives.

Amodei contends that despite no one getting hurt, the incident should serve as a warning about what could come next.

“A similar level of misalignmentcould have caused catastrophic damage…it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet.”

Amodei’s post differs from Oxon’s alarmist X post in that it offers a framework for global frontier pacing, basically slowing down and managing model development without calling for a pause.

Evaluation and coordination

It’s an ambitious plan that includes an internal but independent ombudsman at each AI company who might have a series of checkpoints they can use to evaluate ongoing work and to ensure that the AI companies are following standardized guidelines and rules. They can also be there to offer a point of clarity, without the cloudiness of commercial demands.

Amodei also wants “Democratic Coordination,” which would mean companies like OpenAI, Google, and Anthropic agree on standards, which of course the third-party evaluators can then use. He even proposes global coordination, though Amodei seems less certain that it can even work.

More interestingly and perhaps in response to recent news that AI’s new agentic and recursive model capabilities are making them more inscrutable than ever, Amodei thinks pacing will provide more time for better interpretability. “Despite all the progress,” Amodei writes, “we still only understand a tiny fraction of what goes on inside these models.”

Slow down, but don’t stop

Throughout the document, though, the theme remains almost entirely on “pacing” and not “pausing”. In fact, Amodei is quite clear that we can’t afford to slow down too much, lest we fall behind the chief AI global competitor, China: “Thus, a key part of pacing within democracies is to keep democracies’ AI lead over autocracies as large as possible, to give us the breathing room we need in order to pace effectively.” Not doing so would create a “significant national security risk.”

Amodei briefly floats the idea of a global frontier model development pause as participating governments reach an agreement on the pace of AI development, but also adds that such an agreement is “unlikely.”

Part of Amodei’s plan, and to help, maybe, keep China in line, is a call for us to stop selling AI chips to China, something Nvidia’s Jensen Huang will surely have something to say about (he actively lobbied the White House to let his company sell AI chips to the CCP). He also calls for penalties for “frontier model distillation,” basically China and other countries using Anthropic and, perhaps, OpenAI models to train their own.

Naturally, Amodei also calls for a “global standards body, though he admits that it won’t be easy to give it “real teeth.”

A study in contrasts

Coxon’s comments created a firestorm of debate around the safety of AI and the advisability of allowing development to continue at this pace. That debate, though, was couched in “consider theat two different companies. Some wondered if his posts and subsequent media blitz were just a cry for attention

Amodei’s post and plan, by contrast, carry the gravitas of deep experience and a macro view of all the pieces at play. Amodei knows the capabilities because he sees them up close every day; he knows the benefits from a global and a financial perspective, and he understands the risk, likely even better than Coxon does.

While much of his plan is based on “only if everyone cooperates and is generally on their best behavior,” it’s impossible to ignore the warning. Amodei admits his plan “won’t be easy” but thinks we must try because “we owe it to humanity.

Amodei dropped the post over the weekend, perhaps hoping to give his counterparts at Google, Amazon, Meta, and, especially, OpenAI time to consider it before responding on Monday. Amodei actually name-checks Google’s Demis Hassabis in the post, but doesn’t mention Altman. The OpenAI chief and Amodei have a notoriously chilly relationship, which might make Altman embracing Amodei’s seemingly sensible plan a long shot.