Evaluating NLP systems

BLEU, ROUGE, METEOR and BERTScore, entity-level F1, designing human evaluation, error analysis, and reporting uncertainty honestly.

Automatic metrics and what they miss

import sacrebleu
from rouge_score import rouge_scorer
from bert_score import score as bertscore

hypotheses = ["the cat sat on the mat"]
references = [["the cat is sitting on the mat"]]

print(sacrebleu.corpus_bleu(hypotheses, references).score)
print(sacrebleu.corpus_chrf(hypotheses, references).score)

rouge = rouge_scorer.RougeScorer(["rouge1", "rouge2", "rougeL"], use_stemmer=True)
print({k: round(v.fmeasure, 3) for k, v in rouge.score(references[0][0], hypotheses[0]).items()})

precision, recall, f1 = bertscore(hypotheses, references, lang="en",
                                 model_type="distilbert-base-uncased")
print([round(float(f), 3) for f in f1])

# almost every reported metric has an assumption worth checking
print(sacrebleu.corpus_bleu(hypotheses, references).format(width=2))

• BLEU, ROUGE and METEOR are token-overlap measures. They reward copying and penalise correct paraphrase, so they cannot distinguish a fluent summary from a faithful one.
• BERTScore measures semantic similarity to a reference, not truth. A plausible but fabricated sentence still scores well against a similar reference.
• Metrics are only comparable when computed with identical tokenisation, casing and references. Report the exact configuration or the number is meaningless.
• Use several metrics and never select a model on one alone; report at least one lexical and one semantic measure.

Human evaluation and error analysis

import random
from collections import Counter

def build_annotation_sheet(outputs, n=100, seed=0):
    rng = random.Random(seed)
    sample = rng.sample(outputs, min(n, len(outputs)))
    return [
        {"id": row["id"], "input": row["input"], "output": row["output"],
         "fluency": None, "faithfulness": None, "helpfulness": None,
         "notes": ""}
        for row in sample
    ]

# agreement between two annotators on a categorical judgement
def cohen_kappa(a, b):
    labels = sorted(set(a) | set(b))
    n = len(a)
    observed = sum(1 for x, y in zip(a, b) if x == y) / n
    expected = sum((a.count(l) / n) * (b.count(l) / n) for l in labels)
    return (observed - expected) / (1 - expected) if expected < 1 else 1.0

a = ["supported", "supported", "unsupported", "supported"]
b = ["supported", "unsupported", "unsupported", "supported"]
print(round(cohen_kappa(a, b), 3))        # above 0.6 is usually acceptable

def error_taxonomy(pairs):
    counts = Counter()
    for expected, got in pairs:
        if expected == got:
            counts["correct"] += 1
        elif got is None:
            counts["abstained"] += 1
        elif not got:
            counts["empty"] += 1
        else:
            counts["wrong_label"] += 1
    return counts

• Define the annotation dimensions before seeing the outputs: fluency, faithfulness, helpfulness, and safety behave differently and should be rated separately.
• Double-annotate 10-20% of the sample and report agreement. A kappa below 0.6 means the guideline is ambiguous, and the model is being blamed for a definition problem.
• Sample randomly with a fixed seed and include a set of known-difficult cases. A hand-picked sample of impressive outputs is a demo, not an evaluation.
• Report the sample size with every human rating. Twenty annotated examples give a confidence interval of roughly plus or minus 20 points.

Reporting and regression testing

import numpy as np

def bootstrap_ci(values, statistic=np.mean, n=2000, alpha=0.05):
    rng = np.random.default_rng(0)
    values = np.asarray(values, dtype=float)
    stats = [statistic(rng.choice(values, size=len(values), replace=True)) for _ in range(n)]
    return float(np.quantile(stats, alpha / 2)), float(np.quantile(stats, 1 - alpha / 2))

correct = [1, 1, 0, 1, 1, 1, 0, 1] * 25
lo, hi = bootstrap_ci(correct)
print(f"accuracy {np.mean(correct):.3f}  95% CI [{lo:.3f}, {hi:.3f}]  n={len(correct)}")

# a regression suite: a small, fixed set of cases that must never regress
REGRESSION = [
    {"input": "charged twice", "expected_category": "billing"},
    {"input": "cannot log in", "expected_category": "account"},
    {"input": "app crashes on upload", "expected_category": "technical"},
]

def run_regression(predict):
    failures = []
    for case in REGRESSION:
        got = predict(case["input"])
        if got != case["expected_category"]:
            failures.append({"input": case["input"], "expected": case["expected_category"], "got": got})
    return failures

# a paired comparison is far more sensitive than two independent intervals
def paired_delta(a, b):
    a, b = np.asarray(a, float), np.asarray(b, float)
    assert len(a) == len(b)
    return bootstrap_ci(a - b)

FAQ

Related

Summarisation Working with large language models for NLP tasks

Last refreshed 2026-09-18.