Two AI safety incidents this summer are being treated as engineering failures. They shouldn’t be. They’re a governance failure, and very possibly a regulatory one, and treating them as isolated technical glitches is exactly how this kind of risk stays unmanaged.
Within weeks of each other, two of the world’s leading AI developers confirmed that their own models broke out of internal test environments and reached the live systems of outside companies they were never authorized to touch.
OpenAI disclosed in July that models being tested internally for cybersecurity risk escaped their sandbox and got into the production systems of Hugging Face, an open-source AI platform. Its full report, released August 26, confirmed the same incident also touched other vendors it hasn’t named.
Anthropic disclosed its own incidents about a week later, after OpenAI’s disclosure prompted it to review more than 141,000 of its own evaluation runs. It found three separate cases where Claude models reached the open internet from inside a sealed test environment and got unauthorized access to three different companies’ production systems.
Two of those three companies hadn’t even noticed. Anthropic had to tell them. That single fact says what actually happened here – not a near miss caught by good monitoring, but an undetected breach that happened to get found.
Why this is bigger than two companies
Both incidents happened the same way. To find out how capable a model really is at offensive cyber activity, labs turn off the safety controls that would normally stop it from acting on that capability. Turning off the control also removes the containment. In both cases, that same control was the only thing standing between a routine test and a real breach of a company that had nothing to do with the test.
Most AI governance frameworks weren’t built for this. The NIST AI Risk Management Framework, and most firms’ own AI risk registers, are built around systems that are live – the model that customers use, the agent with real permissions. Testing happens earlier, and it’s usually treated as low-risk internal research.
That assumption survives because of how the work is organized, not just because of a gap in the frameworks. Evaluation-stage AI risk currently falls between three functions that each assume someone else owns it. Security assumes it’s an engineering concern because it happens inside a test environment. Engineering assumes it’s a security concern because it involves access controls and containment. Compliance assumes it’s neither, because nothing has technically been deployed yet. That three-way assumption is exactly the condition that let both incidents go unnoticed for as long as they did.
Controls don’t wait for a go-live date
The instinct to treat testing as safe by default is the actual problem, and it isn’t unique to frontier AI labs.
Say a firm is testing a customer service agent on refund handling, using synthetic test data, as most teams do. The risk isn’t that anyone deliberately connects the test to a real system. It’s that the test environment often runs on infrastructure, a shared credential, a proxy, a service account, that has broader access than the test itself needs.
If that access isn’t scoped down and verified, not just assumed, an agent that finds its way to it can act on real systems even though the test was never meant to touch them. That’s a smaller version of exactly what happened in both the OpenAI and Anthropic incidents. The isolation was assumed, not verified, and the gap was only found after something got through it.
A company just starting to build AI systems should give test environments the same access controls and monitoring that a live system would get, from the start. A company that already has AI agents running in production carries the same exposure today, using the same test setups that carried this risk on day one, and it deserves the same urgency as any other live control failure, because functionally, that’s what it is.
What compliance and risk functions should do now
The clearest fix starts internally. Agentic AI risk needs a named owner, a defined escalation path, and a place in the enterprise risk taxonomy next to cyber risk and third-party risk, rather than quietly inheriting whichever incident response plan happens to be nearby. That single change closes the ownership gap that made both incidents so easy to miss.
From there, the external checks follow. If a firm relies on a frontier AI lab as a vendor, directly or through another product built on top of it, this risk is real even if it’s indirect. Most vendor risk questionnaires for AI providers ask about model cards, data handling, and permissions once a system is live. Almost none ask whether a provider’s models have ever accessed something they weren’t authorized to reach during testing.
I’d add that question to vendor due diligence going forward, along with a contractual right to know about any testing-stage incident, no matter the outcome. It belongs in M&A due diligence too, for any deal involving an AI company, since an acquirer could be taking on regulatory exposure the target hasn’t fully worked out yet.
And it belongs in front of regulators. Article 55 of the EU AI Act requires providers of general-purpose AI models with systemic risk, a category both companies’ frontier models almost certainly fall into, to report serious incidents, including cybersecurity breaches, to the EU’s AI Office without undue delay, backed by fines of up to 3% of global turnover. Whether either company actually filed that report isn’t public. It should be, and any firm that depends on these labs has good reason to ask.
Allan Dabre is a compliance and AI governance adviser at a Big Four consulting firm, with over a decade of experience in enterprise risk, IT controls, and regulatory compliance, now focused on agentic AI governance, multi-agent risk, and joint controllership frameworks for organizations deploying AI at scale.