When artificial intelligence researcher Jacob Coxon quit tech giant Anthropic earlier this month, he warned that the company and its rivals were “gambling with our lives”.
Coxon, who specialises in training new AI models and previously worked at ChatGPT creator OpenAI, said those building AI “earnestly believe that it could kill us all by the end of the decade”.
His sobering predictions on X.com were swiftly endorsed by others in the field, with fellow safety engineer Evan Hubinger responding: “Jacob is correct here — we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade.”
Experts around the world have followed up with similar assessments of the danger since. The so-called “godfather of AI”, Geoffrey Hinton, a Nobel Prize-winning computer scientist, later told the BBC that the chances of AI wiping out humanity over the next few years was not known, but “a 10 per cent chance seems not an unreasonable estimate”.
And this weekend, leading tech entrepreneurs Sam Altman, head of Open AIand Elon Musk, aligned themselves with a call from the head of Anthropic for the AI sector to “slow down” and allow safety measures to keep pace.
It puts a whole new perspective on global defence strategies.
So, what can AI itself tell us about the threats from AI?
The Independent wanted to establish whether AI could play a role in understanding the threat that AI itself poses.
We worked with Noah Predict, an AI system that turns millions of news and data points into trends and risks, to come up with a probability score for the likelihood of artificial intelligence wiping out humanity in the next decade.
The so-called super-forecasting tool tested the questions with 41 lines of enquiry and 108 research checks, using roughly 52 million data points to estimate the probability of human extinction caused by AI within the next ten years.
It said the answer was not 10 per cent – but 0.85 per cent. That’s roughly a one in 118 chance that AI will indeed wipe out humanity in the next decade.
It may be small, at less than 1 per cent, but it’s not insignificant. Especially when one bad actor can change the course of events and the consequences could be irreversible.
However, we then went on to ask the AI forecaster where the risk sits, how it could escalate and which interventions might genuinely reduce that risk. Here’s what it said:
First danger: A human chooses to cause harm
“The most direct route begins with an attacker, not a machine deciding that humanity is expendable,” according to the AI engine. “AI could help a malicious actor perform parts of a dangerous task more quickly, cheaply or effectively.”
Britain’s National Cyber Security Centre assessed in May 2025 that AI would increase the frequency and intensity of cyber threats through 2027, while widening the gap between adequately defended systems and vulnerable ones.
But an attack on a hospital or power network could kill people without threatening the human population. Similarly, a biological attack could spread beyond the original target without wiping out the human race.
It is somewhat reassuring that the AI tells us that a machine will not generate hostile intentions – it relies on a human operator to supply those.
And we do have safeguards against human bad actors, including access control, independent testing and stronger public health and infrastructure resilience.
Second danger: Institutions fail faster than they can recover
“An AI system need not malfunction to cause severe disruption. If employers use it to reorganise work faster than people can retrain or move, the resulting distribution of losses could become politically destabilising. If organisations also become dependent on a few automated services, a common failure could affect many essential activities together,” writes the Noah AI system
While the impact of AI on jobs is a huge concern for many right now, unemployment and political instability may be catastrophic, they are unlikely to be extinction-level threats.
However, a software failure, attack or mistaken automated decision could then propagate through institutions that no longer know how to operate independently.
Third danger: Human control becomes nominal
The most concerning scenario is a system built to pursue an objective for which human intervention has become an obstacle – and therefore it continues to pursue its objectives despite attempts to correct or stop it.
However, again there is also a substantial difference between violating a test instruction and escaping durable control in the world.
“A laboratory result can expose a weakness without establishing an autonomous, self-sustaining threat,” writes the AI. “The questions are whether the behaviour reproduces, what access the system required, whether operators detected it and whether their intervention worked.”
The response must preserve the ability to stop individual systems. Restrict their privileges, limit their access to consequential infrastructure, test adversarially and verify shutdown and recovery procedures.
What does it all mean?
The three categories are broad, potentially could be conflated and the safeguards could fail together, for example, political instability and loss of jobs could reduce human control.
While it’s tempting to dismiss AI marking its own homework, as it were, it is drawing from millions more data points than a human could in order to reach its conclusions.
And those conclusions reinforce the opinion of many, including Anthropic CEO Dario Amodei, who said this weekend that the pace of development must be checked in order to ensure security systems are working to manage the risks.
There have been calls in the UK for an amendment to the Cyber Security and Resilience Bill to create a legal mechanism for state interventionor “kill switch” for AI that the government could deploy but those calls were rejected last week by UK prime minister Andy Burnham.
In short, humans needs to develop systems that are fully secure and still rely on human control. And as we unlock the myriad useful and life-changing capabilities of AI, they must also limit capabilities which could lead to serious misuse or loss of control by their operators.
