AI Resilience, Part 4: When Agents Are Locally Right, Globally Wrong

Image
why agentic workflows need impact budgets enforced outside the agent.

It's the end of the quarter and the forecast is due. A sales leader asks an enterprise agent to tighten up forecast accuracy using the latest engagement, opportunity, and activity data. 

The agent hands off part of the job to a specialist: an opportunity-scoring agent whose only task is estimating how likely each deal is to close. It looks at a strategic opportunity, sees almost no recent CRM activity, and recommends lowering the close probability. 

The primary agent takes that advice and moves the deal to a lower forecast category. The CRM recalculates. Every downstream report picks up the new number. 

Each step looks fine. The scoring agent applied its model to the data in front of it. The primary agent stayed inside its permissions. The CRM processed a valid update. 

Meanwhile, that strategic deal is moving forward in executive conversations that never made it into any connected system. The forecast is now materially wrong, and nothing flagged it. 

Locally right, globally wrong 

That scenario is hypothetical. The pattern behind it isn't. An agentic system can be locally right and globally wrong. 

The scoring agent did its job. The primary agent trusted a reasonable recommendation. The CRM executed the update correctly. Together, they still produced a picture of the business that didn't match reality, which was the one thing the sales leader actually wanted. 

No permission was violated. Nobody slipped in a malicious prompt. No agent was obviously broken. The failure came from the interaction of legitimate actions, incomplete context, and a business goal bigger than any single agent task. 

That shifts the resilience question. "Is this agent allowed to act?" isn't enough anymore. We also have to ask how much consequence it can create, how far that consequence can travel, and what has to happen before it takes a bigger step. 

The risk lives in the sequence 

Traditional authorization checks one action at a time. Can this identity update this record? Call this API? Reach this system? Those checks still matter. They just can't see what a string of individually approved actions adds up to. 

One CRM field change is usually harmless. The same change might shift a forecast, kick off a resource-planning workflow, nudge another agent's recommendations, or land in front of the executive team as a decision input. Add more systems and more agents, and the eventual impact can sit a long way from where it started. 

I've been calling this compositional blast radius: the impact that builds as authorized actions pile up, interact, and spread across systems. It's my own working term, not an industry standard. 

The point is that impact isn't set by the permissions on each tool call. It depends on the sequence, the scale, how reversible the change is, and what downstream processes it sets in motion. 

In the forecast example, permission to update opportunity records doesn't mean the agent should, on its own, make a material change to the quarterly forecast based on a delegated model's opinion. Making each agent more accurate helps. The bigger job is limiting what the whole workflow can do when an agent is wrong, its context is thin, or its objective misses the real business intent. 

Give every workflow an impact budget 

One practical design principle: give each agentic workflow an impact budget. That's an enforceable policy defining what kinds of consequences, and how much of them, a workflow can create on its own before it has to pause, get stronger authorization, or clear an extra validation step. 

Like compositional blast radius, that's a working term, not a standard mechanism. It also doesn't mean boiling every action down to one risk score. In most enterprise workflows, that would mislead more than it helps. Financial exposure, business criticality, records touched, reversibility, and downstream reach are separate dimensions, and a budget should treat them that way. 

In practice, building one looks like this: 

  1. Define what the workflow can change. Updating meeting dates or activity notes is probably fine under normal permissions. Changing a strategic deal's forecast category needs stronger evidence. Publishing a materially revised quarterly forecast needs validation at the aggregate level. 
  2. Bound the scale and reach. A workflow might update a handful of records on its own and need sign-off before touching hundreds or triggering a major downstream process. The right threshold depends on business context, not on a count of API calls. 
  3. Raise the gate as the stakes rise. Material changes might need corroborating evidence, an independent check, or a human approval. The goal isn't a person in the loop for everything. It's saving the heavier controls for actions that warrant them. 
  4. Enforce it outside the agent. The agent can propose a change. A separate policy layer decides whether it can be executed under current constraints. 

Run the forecast scenario again with a budget in place. The scoring agent recommends the downgrade. The primary agent proposes a CRM update. The policy layer recognizes the change as material, sees the supporting evidence may fall short, and stages it for validation instead of letting it flow straight into the published forecast. 

Notice what the policy layer doesn't do. It doesn't judge whether the scoring agent's reasoning was right. It just enforces the business rule: changes this consequential need enough evidence or extra authorization. 

The agent recommends it. The budget decides whether that recommendation can be executed without a check. 

The budget has to follow the workflow 

Delegation complicates this. If the primary agent hands analysis to a specialist, and the specialist calls another tool or agent, each one may carry its own permissions and local limits. The business consequence still belongs to the workflow as a whole. 

So the primary agent shouldn't get a fresh allowance every time it delegates. And a chain of agents shouldn't be able to dodge a limit just because each link stays under its own quota. 

That means evaluating every proposed action in context. What has already changed? What else is pending? What downstream effects are known? Where systems expose their dependencies, policy can account for them. Where dependencies are unknown, the system should default to conservative limits or extra validation. 

You don't need a universal impact meter to do this. Existing authorization checks, workflow orchestration, transaction controls, and policy gates can get you there. The hard part is applying them consistently across the whole chain of execution. 

It's worth being honest about the limits, too. No budget can reliably contain every consequence when critical integrations are invisible, dependencies are unmapped, or the policy itself is badly calibrated. An impact budget guards against unacceptable autonomous action. It doesn't guarantee the right outcome. 

The boundary has to hold when the reasoning doesn't 

You can tell an agent to stay within a limit. You can ask it to double-check its work before acting. Both are good habits. Neither is a real enforcement boundary. 

The same reasoning that produced a bad decision may not notice it just crossed a meaningful business line. Monitoring catches some of this, often after the change has already spread. 

For consequential actions, the policy has to be enforced independently of the agent's judgment. That could mean blocking the action, asking for stronger evidence, staging the change for review, or separating the proposal from the final commit. 

The friction should scale with the consequences. A low-risk, reversible update shouldn't face the same hurdles as a change that could move a financial forecast, a customer commitment, or a business-critical workflow. 

Counting actions isn't enough either. Ten harmless updates might be fine. One consequential update followed by a chain of automated decisions might not be. 

The aim isn't to kill autonomy. It's to make autonomy conditional on what the action could do and how strong the controls around it are. 

Stopping the damage isn't the same as fixing it 

Even a well-built impact budget won't catch everything. Some mistakes only surface after a change has been executed. Others have consequences nobody could see when the action was approved. 

Halting execution limits further damage. It doesn't put the business back where it should be. In the forecast example, the CRM record may already be changed, the forecast recalculated, and decisions already made on the new number. 

That's why bounded autonomy is one piece of AI resilience, not the whole thing: 

  • Prevent sets the policies and boundaries that constrain what agents can do. 
  • Uncover shows what happened, what changed, and how far the effects spread. 
  • Recover restores the intended business state, which is a lot more than restarting the agent or undoing the last API call. 

They only work together. Without preventive limits, a workflow can set too much in motion before anyone steps in. Without visibility, you can't size the damage. Without recovery, stopping the workflow just leaves the consequences sitting there. 

The real test isn't whether every agent makes the right call. It's whether one wrong call can cascade unchecked through the enterprise, and whether the business can put things right when it does. 

Up next: rebuilding what actually happened 

That leaves a harder question for Part 5. When a chain of agents has already changed the business, can we reconstruct how it happened, understand the consequences, and recover the state the business actually intended? 

(A variation of this post first appeared on Anant's Substack. Available to subscribe.)