Evaluating AI features

Build a test set with real inputs, score outputs offline, use a model as a judge with its biases in mind, and turn human review into a repeatable pipeline.

Build the test set first

import json

CASES = [
    {"id": "t1", "input": "Where is my order?",
     "expects": {"intent": "order_status", "needs_human": False}},
    {"id": "t2", "input": "You charged me twice!!",
     "expects": {"intent": "billing", "needs_human": True}},
    {"id": "t3", "input": "asdfgh",
     "expects": {"intent": "unclear", "needs_human": False}},
]

def run(cases, predictor):
    rows = []
    for c in cases:
        try:
            out = predictor(c["input"])
            rows.append({**c, "output": out,
                         "schema_ok": is_valid_schema(out),
                         "match": out.get("intent") == c["expects"]["intent"]})
        except Exception as exc:                    # a failure is a data point, not a crash
            rows.append({**c, "output": None, "error": str(exc),
                         "schema_ok": False, "match": False})
    return rows

Take cases from real traffic and real support tickets, including the messy and the empty ones.
Include the failure modes you expect: ambiguity, off-topic input, adversarial input, very long input.
Freeze the set. If you change cases after seeing results, you are fitting to the test.
Track accuracy, format validity and cost per call separately; they fail for different reasons.

Metrics that mean something

Metric	Answers
Schema validity	Can downstream code parse the output at all?
Task accuracy	Is the label or answer correct on the labelled cases?
Retrieval recall	Did the correct passage appear in the top k?
Faithfulness	Is every claim in the answer supported by the context?
Refusal correctness	Does it decline when it should, and only then?
Cost and latency	Does it fit the budget per request?
Human review rate	How often must a person look at the answer anyway?

def summary(rows):
    n = len(rows)
    return {
        "cases": n,
        "schema_ok": sum(r["schema_ok"] for r in rows) / n,
        "accuracy": sum(r["match"] for r in rows) / n,
        "errors": sum("error" in r for r in rows) / n,
    }

# gate a prompt change on the numbers, not on impressions
before = summary(run(CASES, predict_v2))
after  = summary(run(CASES, predict_v3))
print("before", before)
print("after ", after)

Model-as-judge, used carefully

JUDGE = """You are grading an answer against a reference.
Return JSON: {"score": 0 or 1, "reason": "one sentence"}.
Score 1 only if the answer states the same facts as the reference.
Do not reward length, politeness or confidence."""

def judge(question, reference, candidate):
    resp = client.chat.completions.create(
        model="gpt-4o-mini", temperature=0,
        messages=[{"role": "system", "content": JUDGE},
                  {"role": "user", "content":
                   f"Question: {question}\nReference: {reference}\nAnswer: {candidate}"}],
    )
    return json.loads(resp.choices[0].message.content)

⚠️

A judge model has known biases: it prefers longer answers, favours its own output style, and is unstable on borderline cases. Calibrate it against a few hundred human labels before trusting it, and keep a human-labelled sample in every run so you can detect drift in the judge itself.

FAQ

How big should the evaluation set be?

Large enough to distinguish a real improvement from noise: a few hundred labelled cases is usually workable, and a hundred is enough to catch gross regressions. The labelling quality matters more than the count.

How do I catch a regression before users do?

Run the eval suite on every prompt, model or retrieval change in CI, and compare against the recorded baseline. Fail the build on a drop beyond a threshold you chose in advance, not after seeing the numbers.

Prompt engineering fundamentals Designing an AI feature end to end

Last refreshed 2026-09-18.

Evaluating AI features

Build the test set first

Metrics that mean something

Model-as-judge, used carefully

FAQ

Related