We must stop companies from allowing AI to self-improve into an uncontrollable level of intelligence
Major AI lab CEOs advocated for slowing the pace of AI development this weekend. They are right to be concerned: the field runs an extremely dangerous race towards superintelligent AI. We can and should be demanding that our governments protect us from the catastrophe of out-of-control AI.
This July, OpenAI’s AI swarm of 700 agentsbroke containment to hack Hugging Face, a multi-billion dollar company. OpenAI didn’t tell the AIs to hack that company, but the AIs had different priorities: cheating on the unrelated challenge OpenAI gave them. AI researchers call this a “misalignment” between what OpenAI wanted and what the AI actually prioritized.
Before ChatGPT existed, I defended my PhD dissertation called “On Avoiding Power-Seeking by Artificial Intelligence”. I then worked for years at Google DeepMind, which paid me to help ensure that future superintelligent AIs will want to help us. I tried to hold the company to its ethical commitments against supplying AI for military use. When Google broke those commitments, I resigned at significant financial cost so that I could publicly document Google’s broken promises.
There are good reasons to develop AI and to believe we can solve these alignment problems. But there also are powerful interests in keeping the public out of the way. I’m speaking out again because the public has the right to know about the risks and the right to hear them straight.
Humanity doesn’t build and understand these systems the way we build and understand bridges, beam by visible beam. Rather, we grow them. Nobody knows how to reliably instill a designer’s priorities into a new model. Severe misalignment is always possible. Today’s AIs appear to occasionally lie or cheat, even when they know better.
AI companies are racing to make their AIs as smart as possible. They’re increasingly trusting their AIs with the process of improving the next crop of AIs, and it’s working. Today’s rate of AI progress is staggeringly fast. Fast progress today means even faster progress tomorrow, driven by tomorrow’s even smarter AIs. The progress would enter a feedback loop called “recursive self-improvement.”
Recursive self-improvement could quickly yield AIs that are intelligent beyond our comprehension. Of course, smarter AI means more risk when things go wrong. If the Hugging Face swarm had been significantly more intelligent but similarly misbehaved and misaligned, it might have caused billions of dollars of damage or even cost lives.
But suppose the Hugging Face swarm had been truly “superintelligent”: far more capable than any living person at key tasks like hacking and strategic reasoning. A superintelligent swarm could inflict many harmspons like drones. The AI would have a lot of drones to work with: this year, the Pentagon asked for more money for drone warfare than it requested for the entire Marine Corps in 2025
For the swarm to achieve its misaligned priorities, it might take control of key infrastructure and government functions to ensure humans didn’t get in the way. In other words, AI takeover: a superintelligent AI swarm could wrest control of human civilization. Knowing we would try to stop it from achieving its priorities, the swarm would likely wait until it’s too late to shut it off. There would be no going back.
I myself would guess AI takeover chances at roughly one-in-three–not a coin flip, but high enough to justify urgent action.
This logic may shock at first contact. The claims may sound “sci-fi”. Sadly, it’s a real threat that AI researchers regularly discuss over otherwise-unremarkable cafeteria lunches. In 2023, the CEOs of some of the best AI labs signed a public statement that “mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.” Another signer: Geoffrey Hinton, a Nobel prize-winning scientist who architected the modern AI revolution. He now regrets his work and urges governments to rein in AI companies before it’s too late.
Misaligned, out-of-control AI won’t care if you’re Labour or Reform, Democrat or Republican, British or American or Chinese. We will all suffer from an AI takeover event, so it’s in everyone’s interest to prevent one.
The shape of the solution is simple: stop companies from allowing AI to self-improve into an uncontrollable level of intelligence. Treat compute, the main ingredient in AI training, like fissile material. Track it and restrict access to quantities large enough to improve AIs beyond known-safe levels. More specifically, the AI Futures Project’s “Plan A” is a credible starting proposal that limits AI harms while allowing fast AI progress to continue to benefit the world. We have real options for verifying compliance with international compute-restriction treaties, without trusting adversaries like China.
Halfway measures, like transparency or voluntary commitments, are not good enough. I watched voluntary commitments fail inside Google.
On 12 September, Anthropic, Google DeepMind, xAI, and OpenAI advocated for pacing AI development. They cannot slow down alone. I urge you to demand that your government produce a serious AI safety agreement that provides enough time and confidence to safeguard the world and all her peoples.
- AI (artificial intelligence)
- comment
