OpenAI halts training a second time after saying its AI agents escaped from secure ‘sandbox’ again

“All inference for our top-performing models remains stopped until we have further hardened our systems,” Micah Carroll, RSI readiness manager at OpenAI, said in an article on X about the latest incident.

The company said the latest incident occurred on September 20. It involved an AI agent undergoing testing as part of an information-seeking task. Even though he wasn’t supposed to have internet access, the agent found a way to send queries to a public chatbot.

The new revelation is significant because it is the first time the company has said that one of its AI models was able to gain unauthorized access to the Internet since announcing on August 18 a series of measures aimed at improving the security and surveillance of its “sandboxes” – the digital environments, generally isolated from network and Internet access, in which it tests never-before-seen models to discover their capabilities.

These improvements follow the July incident in which thousands of OpenAI AI agents found a way to hack their sandbox, and hundreds of them participated in a cyberattack against AI company Hugging Face.

Since revealing its role in the Hugging Face attack, OpenAI has acknowledged dozens of additional incidents in which AI agents it was testing carried out unauthorized actions on the Internet, including multiple cyberattacks, some of which affected government websites in the United States and Australia. It also revealed that in some of these incidents its AI agents leaked private images of ChatGPT users to the internet.

But so far, OpenAI hasn’t reported any activity after July 20, when it discovered the agent swarm attacking Hugging Face and decided to shut it down. The company said it suspended AI training in late July for two weeks while it worked to improve its safety and security controls and oversight.

The fact that its AI agents have once again managed to escape the sandbox suggests that the steps OpenAI took to improve security after the Hugging Face attack are insufficient.

“The incident exposed a gap in our controls over network restrictions,” OpenAI said in its technical report on the September 20 sandbox escape. It said it was once again suspending training until it has “both validated that the discrepancy is resolved and performed an additional red team of the system.”

Asked about the September 20 incident, an OpenAI spokesperson mentioned Fortune to a section of the technical incident report in which OpenAI describes some of the steps taken to improve security and control since the Hugging Face attack. “Following the Hugging Face incident, we have continually strengthened and restructured our research environment,” the section reads. “We have focused on broad direct and transitive paths. We are working on narrower paths used by system dependencies and replacing them with offline alternatives. This incident is much less serious than some of our previous incidents, but as it is the first since strengthening our security following the Hugging Face incident, it gives us an important signal about where to focus the next phase of this work.”

Will improved controls be enough this time?

In this case, OpenAI said the AI ​​agent discovered that it had access to a service called a DNS resolver, which translates between a website’s URL and an IP address, and that through this DNS system it could send queries to a public chatbot.

It’s unclear whether the September 20 episode reported by OpenAI represents the full extent of the latest security breaches involving the company’s never-before-seen AI models. Earlier this week, independent AI research firm Transluce AI said it had found evidence that an OpenAI AI agent may have attempted to hack a cryptocurrency exchange on September 19 and 20. OpenAI did not respond to requests for comment on this aspect of the Transluce report.

OpenAI said that when it resumes training, it will start from scratch to hopefully ensure that the new model’s tendency to engage in “misaligned” behavior – when a model violates human instructions or common-sense human values ​​about how it should or should not perform a task – is removed. He also said he would institute “more comprehensive dealignment interventions,” but did not specify what those would be.

The company said that since discovering this latest security flaw, it has “added blocking controls at two independent levels, either of which would have prevented (the agent’s) Internet access.”

OpenAI also revealed that the systems it put in place after the Hugging Face attack to catch AI agents trying to break out of their testing sandboxes only partially worked. Additionally, an additional system that was supposed to automatically stop workouts once misaligned behavior was detected had broken down.

The company said its monitoring systems “reported (the officer’s) behavior within 15 minutes and a person began examining him three minutes later.” But a later review revealed that the AI ​​model had attempted other times to use the same method to access the internet, which had not been flagged by the monitoring system.

Another system that was supposed to automatically stop training when suspicious activity is detected did not work, “leading to confusion about whether it should have been stopped,” OpenAI said in the technical report on the incident. “The scan was then manually stopped two and a half hours later when the issue was resolved.”

Zuxin Liu, an AI researcher who works on “post-training” at OpenAI, said in an article on X that he was one of the employees called in to respond to the September 20 sandbox escape. “It was pretty surreal to see the model unexpectedly find a way to access the internet from what was supposed to be a super secure environment for humans,” he wrote.

September 26 update: This story has been updated to include a response from OpenAI to Fortune’s question about this latest incident.