Toby Walshfor the Conversation
From a dangerous new bioweapon to total societal breakdown, is a ‘superintelligent’ AI capable of wiping out humanity?
Earlier this month, artificial intelligence researcher Jacob Coxon resigned from Anthropicafter just four months. In an announcement on X, he stated: “The people building AI earnestly believe that it could kill us all by the end of the decade.”
A senior member of Anthropic’s staff, Evan Hubinger, actually agreed with Coxon, adding he personally thinks the chance of this happening in the next decade is more than 10%.
Understandably, these statements made waves. There’s now lots of talk about slowing down AI research and increasing “human control” over the technology.
But how exactly might AI kill us all? There’s no shortage of fantastical scenarios, and most of them involve the concept of “superintelligent” AI – that is, AI that’s more capable than humans.
I’ve distilled these scenarios down to the top five, ordering them roughly from most vague to most precise. And I’d argue the list is also ordered from least to most probable.
1. We’ll never know
AI doomers often justify their concerns by means of an annoying catch-22 paradox: how can we possibly imagine what a superintelligence might do to take out less intelligent beings like us?
We’d have to be superintelligent to predict what a superintelligence would be able to do. It’s like asking your family dog to imagine thermonuclear war.
The good news here is that superintelligence is still perhaps some distance away. Current AI models are really good at solving particular problems, but that’s not the same as being more intelligent than a human in all domains.
However, AI did recently solve one of the seven most challenging maths problems known. It’s apparently closing in on others, which might leave you feeling less optimistic here.
2. Paperclips
A superintelligent AI would probably be extraordinarily competent at achieving its goals. But it might be indifferent to human survival.
A classic example of such indifference comes from Oxford philosopher Nick Bostrom’s imagined superintelligent AI that has been designed to optimise paperclip production. To produce its preferred form of office supplies, it quickly converts all available matter – including humans, planets and stars – into paperclips.
What we have here is the perfect execution of improperly specified objectives. The AI doesn’t hate humanity; it simply recognises we’re composed of atoms that could be better utilised for paperclips. It’s not personal.
The good news here is that this scenario confuses intelligence with power. A superintelligent AI doesn’t necessarily have the power to achieve its goals. Turning the planet into paperclip factories would require planning permissions.
Even if it got the permissions, building too many paperclip factories would lead to inevitable public outcry. Interest groups would block the proceedings in the courts. Environmental activists would block the bulldozers.
There’s a lot of friction in the world that prevents even the very intelligent from imposing their will on the rest of us. In fact, you could think of datacentres as a current embodiment of the theoretical paperclip scenario. And humans are increasingly pushing back against turning the planet over to datacentres.
3. Bioweapons
Humanity could be killed by a superintelligent AI making and releasing some dangerous new bioweapon into the atmosphere. This is, in fact, one outcome of the AI 2027 scenario by the AI Futures Project, a non-profit dedicated to forecasting the impacts of advanced AI.
This risk was made more concrete last month, when researchers at Stanford University announced they’d used a genetic language AI model to synthesise 16 new viruses.
Worryingly, they just sent the genetic sequences off to a mail-order lab and it sent the viruses back in test tubes. The whole experiment cost a couple of hundred thousand dollars at most.
The good news here is that it’s remarkably hard to killeveryonewith a new virus. To do that, you need a virus that’s very transmissible, so it spreads far and wide. But it’s a rule of biology – viruses that spread easily are typically less fatal. By contrast, if a virus is very fatal, transmissibility tends to go down, as most people infected die before there’s time to spread the infection.
Covid killed less than 1% of humanity. The deadliest pandemic in recorded history was the Black Death, when the plague killed more than one-third of Europe’s population in the 14th century. However, even the plague would be much less deadly today due to our increased medical knowledge and better sanitation.
4. Nuclear war
What if AI got into the nuclear command and control chain and started a nuclear war? We’ve come close to nuclear war by mistake several times in the past 50 years.
We’re told that nuclear command and control is completely disconnected from the internet. But, as we saw in 2010, Iran’s nuclear centrifuges got taken out by a computer worm called Stuxnet, thought to have been brought in on a USB stick. AI can also give the military false intelligence, which could lead to irreparable actions.
The good news here is that nuclear stockpiles are down. But they are still enough perhaps to take out half of us. And it wouldn’t be by the nuclear blast itself, but the famine in the nuclear winter that would follow.
5. Other humans
Perhaps the most likely risk is that we take ourselves out. And AI might precipitate this.
Imagine – and it doesn’t take a lot of imagination – that AI causes massive job losses, pollutes the information space with misinformation, fractures our politics and destroys human relationships with fake synthetic companionship.
Society might easily break. Slowly but surely, we’d stop being able to support human life at any scale.
What then to take away from all these scenarios? There are some things to be worried about for sure. But not to be too worried, I hope.
Toby Walsh is professor of AI, research group leader at the University of New South Wales. He is the author of God AI: Boom or Doom? What to Expect When Machines Outsmart Us. This article originally appeared in the Conversation
Explore more on these topicsReuse this content
