Reading machine learning papers

Notation conventions, tensor shape bookkeeping, reading an ablation table critically, and reproducing a result from the maths alone.

Notation and shape bookkeeping

Papers are written for readers who already know the conventions. Learn them once and most equations become mechanical: the rest of the difficulty is usually an unstated shape.

• Subscript indices are the fastest way to find a bug: W_{ij} is row i, column j, and PyTorch's nn.Linear stores the transpose of the textbook matrix.
• The symbol * in a paper is often elementwise, while juxtaposition or @ is a matrix product. Getting this backwards produces code that runs and is completely wrong.
• When an equation will not typecheck, write the shape of every symbol before touching code. Most misunderstandings are a missing batch axis or a transposed weight.
• Bold lowercase is a vector, bold uppercase a matrix, calligraphic letters a set or distribution: these are near-universal conventions.

Reading claims and ablation tables

# a typical ablation table, read as deltas from the full model
ablation = {
    "full model":             {"acc": 0.884, "params": 110_000_000},
    "without component A":    {"acc": 0.879, "params": 92_000_000},
    "without component B":    {"acc": 0.851, "params": 109_000_000},
    "with A but not the rest":{"acc": 0.860, "params": 95_000_000},
    "smaller model retrained": {"acc": 0.872, "params": 92_000_000},
}

full = ablation["full model"]
for name, row in ablation.items():
    drop = full["acc"] - row["acc"]
    print(f"{name:28s} delta={drop:+.4f}  params={row['params']:,}")

• Look for the parameter-matched baseline. If removing a component also removes a fifth of the parameters, the ablation measures size, not the component.
• Check how many seeds were run. A 0.3-point difference with no variance reported and a single run is not evidence.
• The best number in the table is usually the last row, chosen after tuning on the test set. Prefer the results with a held-out split procedure described explicitly.
• Reproduced improvements are frequently smaller than the headline. A gain that survives a careful reimplementation is worth far more than a large one that does not.

Read the method section twice: once for the idea, once for the details. Attention masks, warmup steps, weight decay and the exact tokeniser are where reproduction actually lives, and they are usually in an appendix rather than the main text.

Reproducing from the maths

# a minimal reproduction checklist encoded as an executable sanity script
checks = {
    "parameter count matches the paper": None,
    "forward pass shape matches the described tensors": None,
    "loss decreases on a tiny overfit batch": None,
    "gradient check passes for the custom layer": None,
    "reported metric reproduced within 1 point": None,
}

# step 1: overfit a handful of examples until the loss is near zero
def overfit_check(model, batch, steps=200):
    for step in range(steps):
        loss = train_step(model, batch)
        if step % 50 == 0:
            print(step, round(float(loss), 6))
    return float(loss)      # should be near zero if the model can express the data

• Start with the smallest version of the method that can work: one layer, ten examples, one attention head. Get that exact before scaling.
• Reproduce a table cell by cell. If a single cell is far off, that is a bug; if the first cell matches, you can debug the rest with confidence.
• Keep an implementation log of every deviation from the paper. Afterwards you will not remember which line you changed, and the deviation is often the reason your numbers differ.
• If a detail is genuinely absent, note the assumption you made. Reading a paper is a dialogue with the authors, and an unstated assumption is the most valuable thing to record.

FAQ

Related

Matrix decompositions and PCA Evaluation and overfitting

Last refreshed 2026-09-18.