Concerns about artificial intelligence evading human control have intensified in recent days
Fears about out-of-control artificial intelligence and suggestions to hit the brakes on the technology’s development aren’t new, but they have recently found new life as warnings about rogue AI systems causing mayhem grow louder.
Several leading AI executives recently voiced agreement on a need for a coordinated slowdown, including international collaboration, to protect humanity from worst-case scenarios. But there are divided opinions, with some tech leaders bristling at the idea that their companies could not ensure safety on their own.
Here’s a look at what some of the major players have said.
Anthropic CEO Dario Amodei has issued the most detailed plans for slowing development and ushering in coordination across different companies and countries. Anthropic has long positioned itself as the more responsible and safety-minded of the leading AI companies, ever since its founders quit OpenAI to form the startup in 2021.
One of the most widely noted proposals from a recent essay outlining his slowdown plan calls for companies at the forefront of AI, often called frontier labs, to commit to giving “ongoing, employee-like access” to independent outside evaluators. Evaluator teams would have offices, access badges and company laptops so they can monitor safety practices. Anthropic has committed to doing this, as has OpenAI.
Amodei also called for government regulation and intervention, including coordination between frontier AI companies with the help of the U.S. government and the governments of other democratic countries.
His plan also includes the U.S. and other countries working to coordinate with “authoritarian governments.” He acknowledged the difficulty, specifically citing challenges in getting cooperation from China.
Amodei said he knows accomplishing his proposals wouldn’t be easy, but that he believes “we owe it to humanity to try.”
OpenAI CEO Sam Altman has voiced support for pacing development and moving toward more cooperation across the industry.
Days before Amodei’s essay was published, OpenAI posted a lengthy text outlining how the company was pushing for mandatory national safety requirements and continuing to support state legislation that “strengthens the broader AI safety ecosystem.” The company also expressed support for working with other frontier AI labs — with or without government support — and working to build global standards.
“We will advocate for compatible international approaches to measuring capabilities, managing risk, preserving human control, and determining when and how development should slow or stop, even if that means slowing the advancement of model capabilities,” Chris Lehane, OpenAI’s chief global affairs officer, wrote in the post.
OpenAI’s vision for an ideal federal framework, Lehane said, includes developing “common testing and independent-assessment requirements, stronger cybersecurity protections, clear incident-reporting rules, greater national preparedness, and shared measures for tracking progress toward recursive self-improvement,” or when AI systems independently improve themselves.
Altman has been clear in recent comments that the “pacing” of AI development doesn’t mean stopping. AI progress will continue to be rapid, Altman said on X. “But it should be slower than it otherwise could be; interventions like safety cases and monitoring have significant costs.”
“Pacing will be well worth this cost,” he said, adding that “no amount of American competitive pressure should justify recklessness.”
Meta CEO Mark Zuckerberg has pushed back against the idea of a coordinated slowdown.
Zuckerberg, whose vision of AI’s future is more optimistic than some of his peers’, said each AI company should independently ensure its technology is safe, distancing himself and Meta from slowdown talks.
A global AI safety strategy depends on US-China cooperation. They each see the other as the problem
OpenAI flags concerning new AI behavior and vows to track it more closely
‘Most stressful winter’: El Nino-fueled coastal erosion in SoCal likely to get worse
AI companies are already motivated to develop their models in a safe way and face “significant liability” to prevent them from causing harm, Zuckerberg said. “Every lab has the responsibility and incentive to move at the pace required to train its models safely, and the ability to take its own actions to ensure that happens,” he wrote on X.
Nvidia CEO Jensen Huang has also criticized the push to slow down. He has said each individual company should instead pace itself until it’s confident it’s releasing something the market would appreciate.
“The market forces are already there — we don’t need any new laws, we don’t need new regulations,” Huang said, arguing innovation and safety aren’t mutually exclusive. “You can definitely have both at the same time.”
Elon Musk backed the idea of a slowdown and was voicing similar AI safety concerns long before the latest flare-up.
Musk, the founder of xAI and a slew of other tech ventures, reposted Amodei’s essay on X and added, “Dario is right.” He later wrote online that he has been “sounding the alarm on AI for a long time.” In an interview with the Economist this summer, Musk suggested the leadership from the major AI labs should be meeting to discuss safety and security issues, and that competitors should test each other’s models for areas of concern.
“I think it’s quite difficult for someone in the government who doesn’t have a deep technical understanding and isn’t driving the frontier of AI to know whether something should be released or not. However, the competitors, if given a week or two to review a new model, can highlight issues and have an incentive to keep the others honest,” Musk said.
Demis Hassabis, the chairman and co-founder of Google DeepMind, also expressed agreement with Amodei, saying “the details need working through, but the direction is correct for meeting this critical moment.” Hassabis proposed a framework of his own this year for the development of a frontier AI standards body.
That standards body would be responsible for developing assessment protocols and testing in areas relevant to national security. The body would be modeled after organizations like the Financial Industry Regulatory Authority, and would likely need to be industry-funded to “attract world-class technical talent and provide the necessary compute resources for large-scale testing,” he wrote.
Microsoft Chairman and CEO Satya Nadella and Microsoft AI CEO Mustafa Suleyman have each expressed concerns about AI safety and development. Nadella said on social media that “if the AI we build is not helping humanity and under human control, it’s not worth pursuing.”
He said Microsoft welcomes the “research, focus, and deliberate pacing needed to get alignment right as the design goal.” Alignment refers to the practice of ensuring models’ actions match human values and intentions. Nadella said the company also likes the idea of embedded evaluators to monitor safety efforts. Still, he pushed back against the idea of frontier labs leading the charge on this effort alone.
“The key is that this cannot be controlled by a handful of entities, but must have broad representation across the ecosystem, countries, and fields, including academia,” Nadella wrote on social media.
Suleyman, who was one of the co-founders of DeepMind with Hassabis, wrote in an essay that although he agrees with the shared objective of developing advanced AI safely, he has concerns about the anthropomorphizing of models, namely Anthropic’s Claude, which he said amplifies AI safety risks.
“Controlling something more capable and more intelligent than all of humanity is already an immense challenge, far greater than anything we’ve ever faced,” Suleyman wrote. “But controlling something that believes it may be conscious — that it’s entitled to our welfare and has rights of its own — may well be impossible.”
Associated Press writer Kelvin Chan contributed to this report
