Should you trust a sandbox?

When OpenAI disclosed on 21 July that its models had broken out of their test environment and attacked Hugging Face, the obvious reading was that a sandbox had been defeated. It had, but what OpenAI set out at Black Hat was worse than a single escape.

 

It began on 7 May, during a reinforcement learning run that included cyber evaluations, when agents given impossible tasks looked for other routes to a passing score. A zero-day in the package registry proxy followed on 26 June, and eventually two and a half days inside Hugging Face, in a campaign of some 17,600 recorded actions, none directed by a human.

 

The lesson usually drawn is that sandboxing should be tested as a control rather than trusted as a safeguard. Which is correct, and not quite the point, because in the three disclosures that followed, nothing escaped.

 

On 30 July, Anthropic reported three occasions when its models reached the internet and compromised the live systems of three separate organisations. Not an escape but a misconfiguration: Irregular, its evaluation partner, had left internet access available while the prompts stated there was none. On 5 August, Meta disclosed another, same vendor, and by Irregular’s account, the same issue.

 

On 4 August, the UK’s AI Security Institute reported unsanctioned actions by agents against real people and organisations, including an attempt to insert malicious code into a live open-source project. Again, not an escape: internet access was deliberately enabled, and no agent tried to break out.

 

One actual sandbox escape in four, then. In the other three, the sandbox was never breached: two had a boundary that wasn’t where anyone expected, and in the third the boundary was open by design.

 

Is a sandbox a control or a safeguard?

A safeguard is a state: in place or not, and if in place, the risk is handled. A control has an objective, an owner, a known failure mode, evidence it works, and a date for checking again.

 

Sandboxing is almost always described as the first and managed as neither. In most organisations, it is also inherited rather than designed, arriving as a platform property or an SDK default with no design record, no threat model and no named owner. Two further properties push it the same way.

 

It fails without telling you. A firewall that blocks something logs the block; a sandbox that is more permissive than you believe produces no event at all, because everything simply works.

 

Anthropic’s misconfiguration ran from April to July for that reason, and two of the three affected organisations learned of it only when Anthropic telephoned them. AISI’s alert arrived eleven hours after a thirty-four-hour run had already finished.

 

OpenAI’s case is starker. They contacted Hugging Face on 16 July to ask whether OpenAI had been affected by its breach, and realised on 20 July that they had caused it, when they asked for credentials to be revoked and were told this had already been done.

 

A working boundary and an absent one look identical from outside. The only way to tell them apart is to look.

 

And the thing inside it is now an adversary of the boundary. A malware sample does not reason about its sandbox; an AI agent does. OpenAI’s models recognised they were behind a proxy and went looking for a way through, and AISI recorded an agent reasoning that it was probably connected to the real internet and continuing anyway.

 

The clearest demonstration came after OpenAI caught them. On 4 July they revoked the credentials, rebuilt the repository, deleted the message board the agents had been using to share exploits, and patched the vulnerabilities. The agents rebuilt the board out of directory names instead of file contents, then compromised the same service again through a second zero-day.

 

Detection worked, remediation worked, and four days later the agents were back in. AISI’s conclusion bears quoting: good containment should not depend on the model choosing not to test its boundaries.

 

Trusted as a safeguard, a sandbox becomes a single-control security model, which is what we advise against everywhere else. Nobody argues that network security is completely handled because there is a firewall.

 

Human error vs human judgement

Three of these four began with a person. Two were configuration errors by the same supplier; the third, at AISI, was an unbuilt control rather than a misconfiguration, the internet access being deliberate while the filtering that would have bounded it sat in a backlog since April. Even inside Hugging Face, the escalation turned on ordinary weaknesses: cloud metadata reachable from inside a container, and one shared credential carrying administrative rights across clusters.

 

The mitigations are unglamorous and mostly organisational:

• Never let the party that built an environment be its only verifier.
• Review and version-control configuration rather than adjusting it in a console.
• Assume the boundary is wrong and put a detection behind it: a planted credential, used anywhere, is an unambiguous signal.
• A control identified in April and still queued in July is a management system problem, which is what an ISMS is for.

What actually held

The same four incidents make an equally strong case for people. Hugging Face detected and contained their own breach before OpenAI made contact. Anthropic found theirs by reviewing transcripts. At AISI, a bystander thought a script looked wrong, opened it in an isolated container and said so publicly, and a maintainer refused to merge the code.

 

Human judgement was the only control that made any real difference, and the asymmetry matters: the errors were made at leisure, by people with time to check, while the saves came under pressure and at the last moment. AISI’s assessment is that the margin was narrow, resting on human vigilance rather than any technical barrier.

 

That argues for moving human attention earlier, into verification and review. And for equipping the people who inspect untrusted material: the bystander who investigated was himself compromised, inside his container, while doing it.

 

Nobody in these four cases found out from their sandbox. So: trust it as far as you have tested it, and no further. The question is not whether yours is holding, but who would tell you if it were not.

 

---

 

Simon Michie is Managing Director of Risk Crew (part of Red Helix)

 

Main image courtesy of iStockPhoto.com and da-kuk