A security chief at OpenAI has left the company, warning that its culture was broken and that AI companies were not “careful enough” in developing the technology.
David Robinson, who led the writing of security reports accompanying developer ChatGPT’s product releases, explained his resignation in an essay titled: “I left OpenAI because its culture is broken.”
Robinson wrote that a cultural overhaul was needed at cutting-edge AI companies and that incidents such as a “swarm” of OpenAI agents — AI programs operating autonomously without human oversight — attacking AI startup Hugging Face were “typical of the industry, given the speed and flexibility with which people operate.”
Writing in The Atlantic magazine, Robinson wrote: “I agree with other recently departed employees that the companies building this technology aren’t careful enough. But I believe we need to go deeper than specific rules or new laws. We need to talk about culture.”
Referring to OpenAI’s pace of development, he wrote: “As the company moves from one launch to the next, it fails to achieve the level of care that I believe is necessary. »
OpenAI has, however, shown signs of caution in recent weeks following the Hugging Face incident and the revelation that it had informed more than 100 organizations about malware activity. This week, it announced it was abandoning the release of a next-generation AI model after researchers raised security concerns during internal testing. OpenAI has also suspended training of its most advanced models.
Geoffrey Irving, who worked at OpenAI and DeepMind before becoming Resolution’s chief scientist, also joined in the AI warnings on Saturday.
Writing in Time, he said: “Recent warnings about the potential destructive power of AI underestimate the severity of the situation.
“I believe there is about a 50% chance that we will all die from the development of smarter-than-human AI systems, and that our actions over the next two to ten years will determine the outcome.”
Robinson’s essay also follows the resignation of Jacob Coxon, a researcher at OpenAI rival Anthropic, who left chatbot developer Claude last month. He warned that AI “could kill us all by the end of the decade” – and was followed by an anthropogenic warning that there was a greater than 10% chance that AI would wipe out humanity in the next decade. Critics of these warnings, however, have warned that they are unscientific because they cannot be verified or falsified.
Robinson wrote that Silicon Valley lacked awareness of “how to deal with dangerous technologies” and “what it means to take care of people.” Warning that OpenAI had “unfettered optimism” about solving problems as they arose, he wrote that this internal culture meant security failures would only increase as systems became more capable.
“Imagine ‘rogue’ agents who work like hacking teams (e.g., holding hospital computer systems for ransom) but never need to sleep,” Robinson wrote.
after newsletter promotion
Robinson called for two changes in security: that AI companies build on security expertise in other fields such as nuclear and aviation and develop “new science” that ensures the powerful systems of the future can be mastered when operating autonomously.
“Given current risks, border laboratories must operate like nuclear power plants or busy airports, with levels of redundancy and careful, time-consuming planning so that occasional and inevitable human errors do not open the door to disaster,” he wrote.
An OpenAI spokesperson said the company continues to “strengthen our safety and security practices to address the risks we see today,” while working to manage risks that could be created by future advances in AI.
“We ensure that our models do not become more capable than we can safely handle and secure, and we pause training or hold back models when we need to slow down,” the spokesperson said.
Gn bussni
