AI agents do not need more autonomy — they need better boundaries

The usual direction of travel in AI is easy to describe.

Give the model more tools. Give it more permissions. Let it handle more steps without interruption. Reduce the number of times a person has to approve its work.

The result is called autonomy.

Sometimes it is progress. Often it is only a larger blast radius.

An agent that can do more is not necessarily an agent that creates more value. It may simply be able to make a wrong decision in more places, more quickly, with less opportunity for anyone to notice.

The useful question is not how autonomous can this agent become? It is:

What is the most valuable action this agent can take safely, and where must control return to a human?

That question leads to a less impressive architecture. It also leads to systems that are more likely to survive contact with real work.

Autonomy is not a product requirement

Autonomy describes how much of a workflow a system can execute without asking for help. It does not describe whether the workflow should be automated, whether the result is correct, or what happens when the system is uncertain.

Those are separate properties.

Consider an agent that handles customer refunds. It can read a ticket, inspect an order, apply a policy, issue a refund, and send a reply. That sounds like a single task, but it contains several different kinds of authority:

• access to customer data;
• authority to interpret a policy;
• authority to move money;
• authority to communicate a final decision;
• authority to create a record that other systems will trust.

Giving one agent all of these powers creates a clean demo. It also makes every mistake a transaction, every ambiguous input a policy decision, and every tool failure a potential customer incident.

The workflow is not one action. It is a chain of actions with different costs of error. Treating them as one permission called handle_refund hides the risk.

Good agent design starts by separating capability from authority.

The boundary is part of the system

A boundary tells an agent where it can act, what it can change, and when it must stop.

It is not a warning in the system prompt. It is an executable property of the system around the model.

An agent should have boundaries at least across four dimensions.

Scope

What objects and cases can it touch?

An internal support agent may be allowed to work on open tickets from a specific product line. That does not mean it should be able to search every customer record or modify a closed dispute.

Scope limits the set of situations in which the agent is allowed to operate.

Action

What can it do to those objects?

Reading an invoice, drafting a reply, changing a shipping address, and issuing a refund are different actions. They should not inherit the same level of trust because they happen to appear in the same workflow.

Conditions

What must be true before an action is allowed?

An agent might issue a refund only when the order is within the return period, the amount is below a threshold, and the payment provider confirms the original charge. The model can help interpret the case. The system should enforce the conditions.

Exit

What causes the agent to stop and hand the case over?

Conflicting records, missing evidence, an unfamiliar request, a high-value transaction, or repeated tool failures should produce a controlled escalation. They should not invite the agent to improvise.

These boundaries turn autonomy into a defined operating envelope. Outside that envelope, the system has a known response: stop, preserve the evidence, and ask for help.

Confidence is not a boundary

Many systems use model confidence as a proxy for permission.

If the model is sufficiently confident, the workflow continues. If confidence falls below a threshold, a human reviews the case.

This can be useful as one signal. It is a poor foundation for authority.

A model can be confident because the prompt is familiar, while the underlying data is incomplete. It can produce a fluent answer when two sources disagree. It can be certain about the wrong interpretation of a policy because the ambiguity was never represented in the input.

Confidence describes the model’s output. A boundary describes the system’s exposure.

The two questions are different:

• How likely is this answer to be correct?
• What happens if this answer is wrong?

An uncertain draft may be harmless if a person must approve it. A confident instruction to delete data is unacceptable if the action cannot be reversed.

The cost and reversibility of the action should determine the control level, not the tone of the response.

Design permissions around reversibility

The safest path to useful autonomy is usually progressive authority.

Start with actions that are easy to inspect and undo. Increase the agent’s permissions only when the evidence supports it.

A practical sequence looks like this:

1. The agent observes the case and collects relevant information.
2. It proposes an action with the evidence and policy it used.
3. It performs a reversible change in a constrained environment.
4. It executes low-risk actions automatically within explicit limits.
5. It requests approval for actions that create an external or durable effect.

This structure gives the team several useful things at once: a record of the agent’s reasoning, a smaller failure surface, and a way to measure performance before expanding the scope.

The order matters. Starting with a fully autonomous workflow makes failures hard to classify. Starting with observation and proposals creates evidence about where the system is reliable and where it needs a boundary.

Stop conditions need to be concrete

“Escalate when uncertain” sounds responsible. It is also too vague to implement.

Uncertainty needs an observable trigger. For example:

• two authoritative systems return different values;
• a required field is missing;
• the requested action exceeds a financial or operational threshold;
• the case belongs to a category outside the evaluation set;
• a tool returns an unexpected status;
• the agent has already retried the same action twice;
• the proposed change cannot be rolled back automatically.

These conditions do not need to predict every failure. They need to catch the failures that are expensive, difficult to detect, or difficult to reverse.

The escalation should carry useful state with it: what the agent observed, which rules it applied, which tools it called, what it tried, and what decision remains open. Handing a person a blank queue labelled “AI failed” only moves the work.

A good handoff reduces the cost of human judgment.

Tool access is a security model

An agent’s tools are often treated as implementation details. They are closer to an access control system.

Every tool answers three questions:

• what information can the agent read?
• what state can it change?
• what side effects can it trigger?

The tool interface should make those answers visible. A function called update_customer is broad enough to hide several unrelated powers. Separate operations such as draft_address_change, validate_address_change, and commit_address_change make the control points explicit.

The distinction also improves review. A reviewer can assess whether the agent should be allowed to validate a change without granting it permission to commit one.

The model should not be the only component deciding whether a tool call is allowed. A policy layer can check identity, scope, thresholds, rate limits, and current system state before execution. The model can request an action. The system decides whether that request is valid.

That separation is one of the few architectural decisions that becomes more valuable as models improve.

Measure the boundary, not just the completion rate

Teams often report how many tasks an agent completes without human intervention. That metric rewards broader autonomy, even when the additional cases create more downstream work.

A better scorecard measures the quality of the operating envelope:

• successful completion inside the allowed scope;
• correct escalation outside the scope;
• unsafe actions prevented;
• reversals and rework created by the agent;
• time required to resolve an escalated case;
• cost and latency per completed case;
• policy violations and unexplained tool calls.

An agent that completes 60% of cases safely and routes the rest with complete evidence may be more valuable than one that completes 85% while creating hidden rework.

The goal is not to maximize the percentage of work done by the model. The goal is to maximize useful work per unit of risk.

The right level of autonomy changes over time

Boundaries are not permanent. They should change as the team learns.

If an agent handles a narrow class of cases reliably, the scope can expand. If a new policy, tool, or data source changes the environment, permissions may need to contract until the system is evaluated again.

This makes deployment an operating loop rather than a one-time launch:

observe → evaluate → constrain or expand → monitor

The loop should include the failures that did not become incidents. Near misses are valuable evidence. They show that a boundary worked, or that it was missing one condition that a future version may need.

The best agent teams treat permissions as a living part of the product. They do not grant authority once and assume the model’s improvement will solve the rest.

Autonomy is an output of good system design. It is not a substitute for it.

An agent becomes dependable when its capabilities are matched by clear scope, limited permissions, concrete stop conditions, and useful escalation paths. The model may produce the decision, but the surrounding system determines whether that decision is allowed to matter.

The future of agentic software will not be decided by which system can act alone for the longest time.

It will be decided by which systems know exactly when acting alone is safe.