Observability and tracing agent runs

Step-level traces of reasoning, tool calls and results, structured logs, and catching behavioural drift after a change.

What a trace must contain

You cannot debug an agent from its final answer. A single run is a sequence of decisions, and the interesting one is usually the third tool call whose result was misread. A trace is the run rendered as data.

Note what is deliberately absent: the full prompt is not in every span. Store a prompt hash on every step and sample the full text, or your trace store becomes a copy of your most sensitive data.

Instrumenting the loop

import uuid, structlog

log = structlog.get_logger()

def run(goal, tools, max_steps=8):
    ctx = {
        "run_id": f"run_{uuid.uuid4().hex[:12]}",
        "goal_hash": sha(goal),
        "prompt_version": PROMPT_VERSION,
        "model": MODEL_ID,
    }
    for i in range(max_steps):
        reply = call_model(goal, tools)
        log.info("step", **ctx, step=i, phase="think",
                 tokens=reply.usage.total_tokens, ms=reply.ms,
                 tool_calls=[c.name for c in reply.tool_calls])

        if not reply.tool_calls:
            log.info("run.end", **ctx, decision="answered", steps=i + 1)
            return reply.content

        for call in reply.tool_calls:
            started = time.monotonic()
            try:
                result = execute(call, tools)
                err = None
            except Exception as e:
                result, err = None, f"{type(e).__name__}: {e}"
            log.info("step", **ctx, step=i, phase="act",
                     tool=call.name, args=redact(call.args),
                     result_id=sha(result), error=err,
                     ms=int((time.monotonic() - started) * 1000))

    log.warning("run.end", **ctx, decision="step_limit", steps=max_steps)
    return None

• One run_id across every step, including each tool call. Without it you have unlinked lines, not a trace.
• Log the arguments and a content hash of the result, not the whole result. Full payloads belong in sampled storage with a retention rule.
• Redact secrets and personal data at the logging call, not in the log pipeline. The pipeline is shared and someone else configures it.
• Record the prompt version and tool version on every run. Comparing two versions is impossible if the trace omits which one ran.
• Never log raw model output that you have not treated as untrusted text: log viewers render HTML and terminals interpret escape sequences.

Catching behavioural drift

Agents fail quietly. A provider updates a model, a tool's upstream API changes its error format, a page the agent reads gets rewritten, and the success rate slips without any exception being raised. Aggregate metrics over a rolling window are how you notice.

ROLLING_METRICS = {
    "success_rate":      lambda r: r.ok,
    "avg_steps":         lambda r: len(r.steps),
    "tool_error_rate":   lambda r: any(s.error for s in r.steps),
    "loop_rate":         lambda r: r.decision == "loop_detected",
    "budget_hit_rate":   lambda r: r.decision in ("step_limit", "token_limit"),
    "refusal_rate":      lambda r: r.decision == "model_refused",
    "escalation_rate":   lambda r: r.escalated,
}

def compare_windows(current, previous, metric, threshold=0.10):
    a, b = mean_metric(previous, metric), mean_metric(current, metric)
    if a and abs(b - a) / a > threshold:
        alert(f"{metric} moved {a:.3f} -> {b:.3f} over the last window")

• Watch the shape of runs, not just their success: step count, tool error rate, loop detection and budget exhaustion all move before success does.
• Segment by prompt version and model version. A mixed aggregate hides exactly the change you are looking for.
• Keep the regression set from your evaluation work and re-run it after any model or provider update, then compare with the live window.
• Sample full prompts at a fixed rate so a change in prompt assembly is visible without storing everything.
• Alert on the absence of data as well as on bad values: a silent drop in run volume often means a broken integration.

FAQ

Related

Evaluating agents Agent security and permissions

Last refreshed 2026-09-18.