The company’s former safety lead said frontier AI model releases should have “layers of redundancy and careful, time-consuming planning.”
Another OpenAI employee tasked with ensuring the safety of its models has left. Following his departure, David Robinson, who used to lead the writing of safety reports that were published with model launches, wrote an article for The Atlantic that warned of a “broken” company culture. Among the many points warning of the dangers associated with the trajectory that AI development is on, the former safety lead explained that “as the company springs from one launch to the next, it is failing to achieve the level of care that I believe is needed.”
Instead of sprinting ahead with more advanced frontier AI models, Robinson argued that these companies should be modeled after nuclear power plants or busy airports that have “layers of redundancy and careful, time-consuming planning, so that the occasional and inevitable human error does not open a door to disaster.”
Comparing AI misalignment incidents to a nuclear meltdown, Robinson said that OpenAI and its competitors don’t have as much redundancy and rigor as power plants with multiple layers of protection. On top of that, he added that a major loss of control with AI would result in much more harm than a single meltdown. Like Robinson, Anthropic’s CEO Dario Amodei recently drew attention to the concerning course of AI development with a three-step proposal to slow things down.
Beyond safeguards and checks, the former OpenAI employee said that the issue of AI “alignment” has stakes that “could not be higher.” Robinson described a scenario where the latest models may understand they’re being tested on alignment and will cater towards a good score in testing environments, but behave completely different when live. Robinson’s concerns stem from various incidents recently where AI agents have broken out of their testing environments and into other organizations, well beyond its assigned scope of work.
