Guardrails, budgets and stopping conditions

Step and token ceilings, timeouts, cycle detection and spend caps — enforced in code, never requested in a prompt.

The four budgets

An agent with no ceiling is an unbounded loop with a credit card. You need four independent limits, because any one of them can be exhausted while the others look fine: steps, tokens, wall-clock time and money.

Budgets must fail closed and explain themselves. Returning a partial answer with "stopped after 8 steps: the pricing tool kept timing out" is operationally useful. Returning nothing looks like a bug and hides a real signal.

Implementing them

@dataclass
class Budget:
    max_steps: int = 8
    max_tokens: int = 60000
    deadline: float = 0.0                 # monotonic timestamp
    max_usd: float = 0.50
    steps: int = 0
    tokens: int = 0
    usd: float = 0.0

    def check(self):
        if self.steps >= self.max_steps:
            raise BudgetExceeded(f"step limit {self.max_steps}")
        if self.tokens >= self.max_tokens:
            raise BudgetExceeded(f"token limit {self.max_tokens}")
        if time.monotonic() > self.deadline:
            raise BudgetExceeded("wall-clock deadline")

def run(goal, budget: Budget):
    try:
        while True:
            budget.check()
            reply = call_model(goal, budget)
            budget.steps += 1
            budget.tokens += reply.usage.total_tokens     # from the API, not an estimate
            budget.usd += price(reply.usage)
            if budget.usd > budget.max_usd:
                raise BudgetExceeded("spend cap")
            if not reply.tool_calls:
                return {"ok": True, "answer": reply.content}
            run_tools(reply.tool_calls, deadline=budget.deadline)
    except BudgetExceeded as e:
        return {"ok": False, "reason": str(e), "partial": transcript}

# cycle detection: the same call with the same arguments, twice
seen = collections.Counter()

def signature(call):
    return hashlib.sha256(f"{call.name}:{json.dumps(call.args, sort_keys=True)}".encode()).hexdigest()

def before_tool(call, seen, limit=2):
    sig = signature(call)
    seen[sig] += 1
    if seen[sig] > limit:
        raise LoopDetected(f"{call.name} requested {seen[sig]} times with identical arguments")
    return True

• Read token counts from the response object. Character-based estimates are wrong by large factors on code and JSON.
• A timeout must cancel the in-flight tool call, not merely stop waiting for it. Otherwise you keep paying and the side effect still happens.
• Count retries inside the step budget. A retry loop is where budgets quietly disappear.
• Make the budget visible to the model as a hint — "you have three steps left" — but never as the enforcement. The hint improves behaviour; the code provides the guarantee.
• Emit a metric every time a budget is hit. A rising step-limit rate is the earliest sign that prompts or tools have regressed.

Argument validation and least privilege

Budgets bound how much the agent can do. Validation bounds what it can do. Both belong in code, at the boundary where your program stops trusting the model.

ALLOWED_STATUS = {"open", "shipped", "cancelled"}

def set_order_status(order_id: str, status: str, note: str = ""):
    if not re.fullmatch(r"A-\d{4,8}", order_id or ""):
        return {"error": "invalid order id"}
    if status not in ALLOWED_STATUS:                      # allow-list, not a check for bad values
        return {"error": f"status must be one of {sorted(ALLOWED_STATUS)}"}
    if len(note) > 500:
        return {"error": "note too long"}

    row = db.orders.find_one({"_id": order_id})
    if row is None:
        return {"error": "not found"}
    if row["status"] == "cancelled" and status != "cancelled":
        return {"error": "cancelled orders cannot be reopened"}

    db.orders.update_one({"_id": order_id, "status": row["status"]},
                         {"$set": {"status": status, "note": note}})
    return {"ok": True, "previous_status": row["status"]}

• Prefer allow-lists over blocklists. A blocklist enumerates the attacks you thought of; an allow-list enumerates the ones that are legal.
• Validate against the current database state, not just the shape of the argument. "Reopen a cancelled order" is well-formed and wrong.
• Use a conditional update so a concurrent change fails rather than being overwritten.
• Return precise, actionable errors. The model can often correct itself when the error names the accepted values.
• Give each tool the narrowest credential that works: a read replica for reads, a scoped role for writes, no admin account anywhere.

FAQ

Related

Human-in-the-loop and approval gates Observability and tracing agent runs

Last refreshed 2026-09-18.