AI Resilience, Part 2: Why Did the Agent Do That?

When an AI agent does something unexpected, everyone asks the same thing first: why did it do that? 

Fair question. For autonomous systems, I think a more useful one is: what happens when it does? 

An agent doesn't follow a fixed path. It interprets a goal, pulls in context, decides what to do, uses tools, checks the result, and keeps going. That's exactly what makes agents useful. It's also what creates a new kind of enterprise problem. 

An agent doesn't have to be malicious, compromised, or "rogue" to cause an incident. It just has to make the wrong call while holding enough authority to act on it. 

A replacement order nobody should have approved 

Take a customer service agent that handles product returns. A customer writes in: "The product arrived damaged. Please send me a replacement." 

The agent checks the order, pulls the customer's history, looks up the return policy, creates a replacement order, and sends confirmation. Routine. 

Now add one small wrinkle. The customer says the damage happened yesterday. The order system says the package was delivered three weeks ago. The policy says any replacement after 14 days needs extra verification. 

The agent weighs it up, decides the account looks credible, and creates the replacement anyway. 

Technically, nothing broke. The agent had a valid identity. The API call worked. The authorization checked out. The transaction is complete. 

The business outcome was still wrong. That's the part enterprises need to start thinking about differently. 

The agent isn't the application 

We tend to talk about an AI agent as if it were the whole application. It isn't. It's a decision-making component inside a much bigger system. 

The model shapes its behavior, and so does everything around it: the instructions it gets, the information it retrieves, the tools it can reach, the permissions on those tools, and the systems that finally carry out its actions. 

So the enterprise can't count on the model to police its own boundaries. The agent can decide, "I should issue this refund." Something outside the agent should independently decide, "Is this action allowed?" 

That principle is showing up in emerging best practices. NIST is developing standards-based approaches for identifying and authorizing software agents, covering identity, authority, auditing, and accountability. OWASP recommends enforcing authorization in downstream systems instead of trusting an LLM to judge what is permitted. 

None of this is new, really. We've spent decades wrapping controls around applications because we never assumed an application would always behave. Autonomous software deserves the same discipline. 

Some failures will get through 

Even with strong controls, things slip. An agent makes a reasonable call from incomplete information. The policy is ambiguous. Retrieval surfaces two facts that contradict each other. A downstream system behaves differently than anyone expected. Or several individually valid actions add up to an outcome nobody wanted. 

So, the next question becomes: when something goes wrong, can we uncover what actually happened? 

I chose to uncover over monitor on purpose. Monitoring tells you that a refund was issued. Uncovering should tell you: 

• What did the agent see? 
• What did it do? 
• Which controls applied? 
• What changed, and why? 
• How far did it spread? 
• Can we get back to the intended state? 

That's a very different bar than conventional application logging. 

A 200 OK doesn't explain anything 

Go back to that replacement order. A conventional log might give you this: 

10:42:13  POST /replacement-order  200 OK 

Useful, and nowhere near enough. For an autonomous system, you need to rebuild the execution path that led there: 

1. Customer request 
2. Agent identity and run 
3. Information retrieved 
4. Policy and permissions 
5. Tool invoked 
6. Action taken 
7. Business outcome 

The goal isn't a mythical transcript of the model's private reasoning. It's enough execution provenance to trace the path from input to outcome. 

This is becoming standard thinking, too. NIST's agent's work explicitly calls out auditing and non-repudiation, and current enterprise guidance increasingly treats agent runs, tool calls, identities, and retrieved information as first-class operational telemetry. 

That matters because the root cause may not be the model at all. It could be the data, the policy, the tool, the permission, the workflow, or the way they interact. You can't fix what you can't reconstruct. 

Recovery means business state, not just data 

Sooner or later, prevention fails. When it does, resilience comes down to getting back to the intended state. 

For one bad replacement order, that might be as simple as reversing a transaction. Now imagine an agent that made thousands of changes across CRM, billing, fulfillment, and customer communications before anyone notices. What does recovery mean then? 

It could mean reversing transactions, reconciling systems, restoring configuration, revoking the agent's authority, or rebuilding workflow state. Often, it means the hardest job of all: sorting the legitimate changes from the ones that shouldn't exist. 

That's the distinction I keep coming back to: 

Traditional recovery asks: what data do we restore? AI resilience asks: what business state do we restore? 

The second question is a much bigger problem. 

Prevent, Uncover, Recover, and then Learn 

In Part 1, I framed AI resilience as Prevent, Detect, Contain, Recover. I've since tightened that into three stages: 

• Prevent. Constrain what the agent can do and limit its blast radius. 
• Uncover. Understand what happened, why, what changed, and what was affected. 
• Recover. Restore the enterprise to its intended state. 

Then close the loop: Prevent, Uncover, Recover, Learn, and back to Prevent. 

Learning doesn't mean retraining the model. It might mean narrowing permission, changing a workflow, adding a policy check, improving the quality or provenance of retrieved information, adding a circuit breaker, or getting better at recovery itself. 

The lesson belongs to the system, not just the model. 

Security, reliability, resilience 

This is why I see AI resilience as broader than AI security. Each discipline asks its own question: 

• AI security: Can we prevent unauthorized action? 
• AI reliability: Does the system perform as expected? 
• AI resilience: Can the enterprise stay in control when an autonomous system doesn't? 

That means designing the gap between what we intend and what the agent actually does. We don't expect agents to fail constantly. Once software is making decisions and taking actions at machine speed, though, assuming it will always behave exactly as intended stops being a strategy. 

The frontier labs are finding this out in controlled evaluations. Enterprises will find it out in production, where the systems, data, and consequences are real. 

The goal isn't to eliminate autonomy. It's to make autonomy bound, understandable, and recoverable. 

AI won't always do what we expect. Resilience is about being ready when it doesn't. 

Up next: when the agent isn't yours 

It gets more complicated when the agent belongs to someone else. Enterprises are adopting agents from SaaS providers and AI companies that can reach corporate data, call enterprise tools, and increasingly hand work off to other agents. 

Who controls the chain of agency when the agent acting on your behalf isn't yours? That's where Part 3 picks up.