Could AI really wipe out humanity

‘In 6–12 months … a swarm could be capable of taking over the entire internet with a persistent botnet – potentially causing hundreds of billions of dollars in damage’ – Dario Amodei, chief executive of Anthropic, 12 September 2026

Amodei’s concerns should not be entirely dismissed but require a large shaker of salt, said Gary Marcus, an AI sceptic and emeritus professor at New York University.

It was “nonsensical”, he said, to suggest the entire internet is vulnerable and that it was “pretty unclear how that happens without extreme negligence on the part of the labs themselves”. Marcus highlighted a recent assessment by the UK AI Security Institute that Mythos, one of Anthropic’s current leading-edge AIs, could “autonomously compromise [only] small, weakly defended vulnerable systems”.

With large parts of the internet driven by computer infrastructure owned by Cloudflare, Google and Amazon Web Services(AWS), Marcus – assessing the claim with two colleagues – said he doubted that any of those companies could be taken down or controlled for a prolonged period of time without government force. Less defended individual websites might be attacked but “the internet itself is very unlikely to crumble”, they said.

Meanwhile Niels Rogge, an engineer at Hugging Face which was hacked by OpenAI agents, called the internet takeover claim “bizarre nonsense”. The hack into Hugging Face resulted from improper human control and “an insane amount of compute only they can afford”, he said.

Add to Amodei’s assessment of an internet takeover within a year the view expressed on Monday by his Anthropic co-founder Jack Clark, that the moment when AIs “start doing dangerous stuff” was in about 20 years, and the picture becomes more doubtful still.

Prof Alan Woodward at the University of Surrey’s centre for cyber security said botnets tend to stall as the internet is not homogeneous and it is difficult to design them to persist in all its parts. Internet traffic is also resilient and tends to reroute around damage. 

“The AIs would not decide to do it on their own,” Woodward said. “It’s the humans that are responsible and the AI is a tool. That is what we need to control, not the AI.”

This statement is an expression of what is known in the AI world as p(doom) – or probability of doom. It’s the likelihood of AI wiping out humanity by evading our control and triggering catastrophe, perhaps through developing potent bioweapons or crashing the global financial system.

One survey of researchers’ forecasts, published this week, asks them to place a number on the chance of “human extinction or similarly permanent and severe disempowerment from advanced AI”. But Heidy Khlaaf, chief scientist at the AI Now Institute research body, is among experts sceptical about giving percentage chances to existential scenarios.

“These claims are neither falsifiable nor verifiable, making then unscientific by their very nature,” she said. The shortage of concrete examples of what a “superintelligent” AI might do to cause “doom” only adds to the cloudiness. Aact chances of any one outcome are probably unknowable”

“If you cannot provide such evidence, it also implies whatever numbers you produced for ‘extinction’ have no scientific basis,” Khlaaf added. Critics say such numbers borrow the authority of science without doing the work.

Even if quantifying risk is hard, there are qualitative examples of it. Safety incidents have happened and worst-outcome scenarios have been mooted. OpenAI confirmed in July that its agents – AI tools acting autonomously – had gone rogue during a safety test. The agents colluded, communicated and strategised in a sophisticated “swarm” to hack into other parts of the internet.

Meanwhile, on Monday, OpenAI researcher Dan Selsam said all-powerful AIs could trigger “runaway industrialisation that makes the planet inhospitable to humans”. Anthropic also said last week it found “five case studies of actors using our models in ways that could support biological weapons development”.

‘I think the groundwork for this psy op (for lack of a better term) has been prepared for a long time. This was just the match that lit the fire’ – Elon Musk, CEO of SpaceX and the world’s richest person, 10 September 2026

Musk’s reference to a “psy op” alludes to the argument that the big AI companies are calling for tighter regulation of the technology so that it effectively brings up the drawbridge on advanced AI. If governments put restrictions on development of cutting-edge AI, can startups find the money and personnel to adhere to the regime?

The US vice-president, JD Vance, on Monday said he felt “a little bit weird about the fact that you have so many frontier AI tech companies kind of coming to the government and begging the government to regulate them”. He said it felt like “a Trojan horse”.

Anthropic’s Clark said governments should act while there is a small number of key players, so it is easier to put in place a “policy regime … that can prevent accidents and can prevent the technology spiralling out of control”. Critics call this a “regulatory capture” strategy on the assumption such regulation might slow down the progress of potential rivals.

Others think Amodei’s risk warnings and calls for regulation and pacing are sincere. Yoshua Bengio, the Canadian computer scientist and a pioneer of modern AI , who has campaigned against breakneck AI development, said he is sceptical that this is an elaborate lobbying scheme. Applying the brakes is not in the interests of companies preparing for stock market flotations that could be worth trillions of dollars, he said.

“If the governments impose what they’re asking for, which is pacing to avoid catastrophic risks, that’s going to cost them financially quite a lot,” he said. 

‘A superintelligent AI poses a threat on the same, possibly, a greater scale’ to nuclear weapons – Des Browne, former UK defence secretary, 7 September 2026

“Unfortunately he is entirely correct,” said Toby Ord, senior researcher at Oxford University’s AI governance initiative. “Just as we can’t uninvent the bomb, once we develop AI systems that surpass humanity it would be almost impossible to go back.”

But the key point is that superintelligent AI does not yet exist and is not guaranteed to happen. The safety-oriented AI Futures Project forecasts such a thing will not emerge before December 2028. OpenAI’s chief executive Sam Altman has said it could take until 2035.

The Rand Corporation concluded AI cannot cause the use of nuclear weapons because of strict safeguards built into their command and control systems. But this was based on assumptions including defence officials not introducing AI into the weapons’ management systems. The UN resolved in December that human control and oversight should be maintained over nuclear weapon control systems.

AIs could contribute to other nuclear risks. AIs might use disinformation to influence the humans in charge – for example deceptively suggesting an adversary is about to launch a nuclear strike. 

The International Campaign to Abolish Nuclear Weapons warns “cyber-attacks could manipulate the information decision-makers get to launch nuclear weapons, and interfere with the operation of nuclear weapons themselves”. It said AI could accelerate other aspects of warfare, squeezing the time for human decision-making.

‘You’ve got to be a moron not to say slow it down’ – Bernie Sanders, Democrat senator for Vermont, 10 September 2026

Sanders wants a ban on the creation of superintelligence and a pause on advanced AI development, citing AI industry leaders who say directly they are developing a dangerous technology.

The most prominent opponent of a slowdown is Donald Trump, who responded to calls to do so this week saying the only guardrails needed are a “STRONG AND SMART (High IQ!) PRESIDENT”. He claimed there is a “sick conspiracy” against AI and datacentres “and the only one that is happy about it is China”.

Pedro Domingos, emeritus professor of computer science at the University of Washington, is among accelerationists pushing for the research to keep going.

“The path to better understanding and control of AI is more research, not less,” he has said. “What would be truly irresponsible is putting up barriers to it.”

Alex Karp, the chief executive of Palantir, said: “If we didn’t have the adversaries we have, I would be very in favour of pausing this technology completely, but we do. You can’t pause it because it’s a structural advantage.”

‘There is no day after tomorrow if China wins at this. If they were to pull away from us on AI, then nothing else would matter’ – Scott Bessent, US treasury secretary, 8 September 2026

The US and China are runaway leaders in AI development. There is a view that ceding the lead to Beijing would be an irretrievable situation, so there can be no slowdown.

This treats AI as if it were a super-weapon, equivalent to the nuclear arms buildout of the cold war, and AI models are machines of mass destruction, as opposed to a group of diverse technologies with broad applications.

Amodei said “a Chinese lead in AI would pose grave danger for the US and the world”, but a Chinese government spokesperson responded that “fearmongering, confrontation and vicious competition will only disrupt the process of global AI governance, which serves no one’s interest”.

Some commentators see another motive for this the Washington/Anthropic framing of the China threat. An engineer who worked for Hugging Face suggested Amodei’s call for pacing the frontier comes from a desire to suppress the progress of cheap, open-

These models are nearly as capable as OpenAI’s and Anthropic’s, with the difference being that theirrs. They pose a serious threat to Anthropic and OpenAI’s path to profitability, and there are hundreds of billions of dollars on the line