OpenAI’s chief scientist Jakub Pachocki has called for “extreme caution” over AI’s runaway progress and warned more intervention may be needed to ensure “humans remain in control of the future”.
“I am concerned no one is prepared for the consequences of a continued rapid rise in machine intelligence,” he wrote in a blog post entitled “An Alien Mind“.
The post comes only a few days after the ChatGPT-maker released its latest model GPT-6 Astra, which it called its most powerful product ever.
OpenAI and other AI firms like Anthropic shared reports of their AI agents acting autonomously and carrying out real-world cyber-attacks on other companies.
In July, OpenAI called an incident in which its AI agents – AI systems which can operate alone after human instruction – hacked the tech platform Hugging Face “unprecedented”.
Meanwhile in September a report claimed months earlier AI agents from the firm had also hijacked a German website.
“We are facing a transition to a world with incredibly intelligent machines, and we need to ensure that transition works out well for humanity,” Pachocki wrote in the latest blog post.
He said OpenAI would continue to “build defensive systems” and seek technical solutions to alignment – a term used to describe ensuring a machine’s actions and goals perfectly match human intent and safety guardrails.
Pachocki added one of the firm’s main priorities moving forward would be to build an “automated AI researcher” to keep pace with AI progress while ensuring ways that human researchers could remain part of the process.
“Instead of better AI guardrails, regulations, or assurance to keep people safe, they propose developing internal AI agents to research these problems,” said Professor Gina Neff, head of the Minderoo Centre for Technology and Democracy at the University of Cambridge.
“Such answers to growing concerns about the problems OpenAI’s models are causing for cyber-security, job loss, mistakes, errors and fraud are simply not good enough.”
Nathan Calvin, general counsel at the advocacy group Encode AI, said he agreed with Pachocki on the hazards present in advanced AI model development – but he claimed OpenAI was unwilling to be transparent, and said this meant warnings risked being dismissed as “just self-interested hype”.
“If Jakub and others at OpenAI want relevant folks in the AI industry to act in concert with them to make things go well, one of the most important things they can do is share far more information about what they are seeing that is making them call for caution,” he wrote on X.
What AI guardrails are in place?
Global regulations are struggling to keep up with the pace of AI development.
The European Union’s AI Act came into force on 2 August, which requires AI giants like OpenAI to prove their most powerful models cannot autonomously launch cyber-attacks or evade human control before they are allowed to be sold in Europe.
But because the law’s jurisdiction is confined to Europe, it cannot stop a rogue AI developed elsewhere from threatening the continent.
In his blog post, Pachocki called for legally or internationally required minimum safety thresholds which could be enforced by a “network of third-party auditors” or “government agencies”.
AI labs would need to meet these thresholds, he said, before they were allowed to continue scaling or deploy advanced models.
He added he hoped “voluntary slow downs” – a self-imposed pause by companies in AI development – would become commonplace until shared guardrails were established.
In August OpenAI said it had slowed down training some of its most advanced AI models to improve security.
