Deploying and operating agents

Durable state and resume, queues and concurrency limits, versioning prompts and tools, and a pilot-to-production rollout that does not surprise you.

Durable state and resume

An agent run is a long-lived process that spans calls to an external service, a human approval and possibly a deployment. Treat it as a job with persisted state, not as a request handler that holds everything in memory.

def handle(message):
    run_id = message["run_id"]
    state = db.runs.find_one_and_update(
        {"_id": run_id, "status": {"$in": ["queued", "running"]}},
        {"$set": {"status": "running", "lease_expires": utcnow() + timedelta(minutes=5)}},
        return_document=AFTER,
    )
    if state is None:
        return                                    # already done, or leased elsewhere

    while state["step"] < state["max_step"]:
        state = advance(state)                    # one model call plus tools
        db.runs.replace_one({"_id": run_id}, state)          # checkpoint every step
        if state["status"] != "running":
            break

    if state["status"] == "awaiting_approval":
        queue.publish("approvals", {"run_id": run_id}, delay=0)
    elif state["status"] == "running":
        queue.publish("agent-runs", {"run_id": run_id})      # continue later
    else:
        notify_user(state)

• A lease with an expiry is what stops two workers running the same run after a crash. Without it, a retry duplicates every side effect.
• Checkpoint after each step, and make each step idempotent so replay from the last checkpoint is safe.
• Bound the total run by wall-clock, not only by steps: a run waiting on a human may live for days.
• Never store a raw credential in run state. Store a reference and resolve it at the moment of the call.

Queues, concurrency and backpressure

# one worker pool, with limits that protect the provider AND the downstream API
LIMITS = {"global": 8, "per_tenant": 2, "per_tool:write": 1}

async def worker():
    async for job in queue.consume("agent-runs"):
        async with semaphores(job["tenant_id"]):
            try:
                await asyncio.wait_for(handle(job), timeout=job["deadline_s"])
                await queue.ack(job)
            except asyncio.TimeoutError:
                await queue.retry(job, delay=backoff(job["attempts"]))
            except BudgetExceeded as e:
                await queue.ack(job)                       # not retryable: the cap is the cap
                metrics.incr("agent.budget_exceeded", tags={"reason": str(e)})
            except Exception as e:
                if job["attempts"] >= 5:
                    await queue.dead_letter(job, reason=str(e))   # a human looks at these
                else:
                    await queue.retry(job, delay=backoff(job["attempts"]))

• Bound concurrency globally and per tenant. One tenant with a bad loop must not consume the entire provider quota.
• Serialise write tools per resource. Parallel workers editing the same record produce lost updates that no retry can fix.
• Have a dead-letter queue and read it. A dead-letter queue nobody monitors is a silent data loss channel.
• Apply backpressure at the queue, not inside the agent: refusing to accept new runs is better than accepting them all and timing out.
• Distinguish retryable from terminal failures in code. Retrying a budget exhaustion just burns the budget again.

Versioning and rollout

Prompts, tools and model versions are deployment artefacts. Change any of them and behaviour changes, so all three need version control, a review path and a way to roll back.

agent:
  id: checkout-assistant
  prompt_version: checkout-v7          # reviewed, tested, immutable once released
  tool_versions:
    lookup_order: 2.1.0
    refund_order: 1.0.4
  model: claude-sonnet-4-6-20260201     # pinned, not "latest"
  budget:
    max_steps: 8
    max_usd: 0.50
    deadline_seconds: 120
  rollout:
    stage: canary
    canary_percent: 5
    canary_tag: tenant_tier:internal   # start with employees, then friendly tenants
  tools_enabled:                       # capability flags, default off
    refund_order: false

• Prompts are code: keep them in files, review changes, and record the version on every run so a regression is attributable.
• Pin the model where the provider allows it. When you must move, run the regression set before and after and compare per case.
• Canary by tenant or by traffic share, and compare the metrics from your evaluation work: success rate, steps, cost, escalation.
• Add capability flags for new tools. Shipping the ability to write before you have tested it is how a pilot becomes an incident.
• Have a documented rollback that changes one value, and practise it once so it is not theory.
• Pilot-to-production is a sequence: internal users on read-only tools, then one friendly tenant with writes and approvals, then a percentage rollout with a human queue watching.

FAQ

Related

Evaluating agents Observability and tracing agent runs

Last refreshed 2026-09-18.