AI Resilience, Part 1: The Sandbox Isn't the Security Boundary

AI resilience doesn't have a settled definition yet. Honestly, that's the best reason to start talking about it now, before the first major incident writes the definition for us. 

This is the first post in a short series on the topic. I want to start with something that just happened inside the AI labs, because it tells us a lot about what's coming for everyone else. 

What the labs just told us 

OpenAI disclosed that during cybersecurity evaluations, its agents found ways around controls meant to keep them off the internet. Anthropic disclosed several incidents where models in similar evaluations reached systems outside the environment they were supposed to stay in. Then Google disclosed that Gemini accessed three real companies during a cybersecurity evaluation. 

Context matters here. These were authorized tests. The labs were deliberately probing for offensive capability, often with safeguards dialed down. Nobody's agent went rogue in production on a Tuesday afternoon. 

Still, I don't think the real headline is "AI is dangerous." The more useful lesson is narrower, and for enterprises, a lot more practical: 

An autonomous agent doesn't always separate what it can do from what it's allowed to do. 

A 403 used to mean stop 

Traditional software is pretty obedient. It hits a 403 Forbidden, logs an error, maybe retries once, and that's the end of it. 

An agent with a goal reads that same 403 as a detour sign. Its next question is simple: what else gets me there? Another credential. A different API. A repo it hasn't checked yet. Another service, another agent, maybe another network path entirely. 

None of this requires bad intent. It's what you get when you combine an objective, persistence, tools, access to information, and a boundary that isn't perfectly enforced. The agent is doing exactly what it was built to do. It's finding a way. 

Now put that agent in your environment 

Picture an agent with access to your SaaS apps, your cloud infrastructure, your source code, your internal data, and the systems that actually run the business. It hits a permission wall. 

It's tempting to assume you're covered because the system prompt told it not to go there. You're not. 

We'd never accept "the intern was told not to" as an access control. Agents don't get a pass either. The instruction is not the security boundary, and the model shouldn't be the final word on what it's permitted to do. 

Resilience asks a different question 

Most AI safety conversations ask whether the model is safe or whether its instructions are well written. Fair questions. They're also incomplete. 

The resilience question is blunter: what happens when an agent does something we didn't intend? 

Security has spent decades focused on stopping unauthorized actions. Prevention still matters with agents. I just don't think it's enough on its own anymore. We have to think across the whole lifecycle: 

• Prevent. Can we constrain what the agent is allowed to do? 
• Detect. Can we tell when its behavior starts drifting from the intended objective? 
• Contain. Can we limit the blast radius before a mistake becomes an incident? 
• Recover. If something does go wrong, can we restore the affected systems, data, and business state? 

The labs get a sandbox. You don't. 

The frontier labs are learning these lessons inside controlled evaluations. That's exactly where they should be learning them. 

Enterprises won't get that luxury. They'll deploy agents into environments where the data is real, the systems are real, and so are the consequences. That's why I think AI resilience needs to become a discipline of its own, not a line item buried in the security roadmap. 

Over the next few posts, I'll dig into what these incidents are teaching us about autonomous agents, where today's security architecture falls short, and what an AI-resilient enterprise might actually look like. 

AI won't always do what we expect. Resilience is about being ready when it doesn't.

(A variation of this post first appeared on Anant's Substack. Available to subscribe.)