Transformers and fine-tuning basics
What self-attention computes, how encoder and decoder stacks differ, and when fine-tuning is worth the cost compared with prompting.
What self-attention computes
A transformer layer projects each token into a query, a key and a value. Every token compares its query against all keys, turns those scores into weights, and takes a weighted sum of the values. That weighted sum is the token's new representation, so information moves between positions in one step rather than through a recurrent chain.
import torch
import torch.nn.functional as F
def self_attention(x, Wq, Wk, Wv):
# x: (batch, seq_len, d_model)
q, k, v = x @ Wq, x @ Wk, x @ Wv
d_k = q.size(-1)
scores = q @ k.transpose(-2, -1) / d_k ** 0.5 # scale keeps softmax stable
weights = F.softmax(scores, dim=-1) # each row sums to 1
return weights @ v
x = torch.randn(2, 16, 64)
out = self_attention(x, *(torch.randn(64, 64) for _ in range(3)))
print(out.shape) # torch.Size([2, 16, 64]) - same shape as the input| Stack | Examples | Attention | Typical use |
|---|---|---|---|
| Encoder-only | BERT, DeBERTa | Bidirectional | Embeddings, classification, token extraction |
| Decoder-only | GPT family, LLaMA | Causal, left to right | Generation, chat, code completion |
| Encoder-decoder | T5, BART | Both, plus cross-attention | Translation, summarisation, rewriting |
Because every token attends to every other, attention cost grows with the square of the sequence length. Doubling the context roughly quadruples the attention work, which is why long-context inference is expensive and why positional information has to be injected explicitly.
When fine-tuning is worth it
Fine-tuning changes how a model behaves; it is a poor way to teach facts. Reach for it when the task is stable, your labels are consistent, and a prompt plus a small classifier has already plateaued.
| Approach | Labels needed | Compute | Risk |
|---|---|---|---|
| Prompting / few-shot | A handful of examples | None | No training risk; behaviour drifts with prompt wording |
| Embeddings plus a linear classifier | Hundreds | Minutes on CPU | Very low; a strong first baseline |
| LoRA / adapters | Thousands of pairs | One GPU, hours | Low; base weights stay frozen |
| Full fine-tuning | Tens of thousands | Several GPUs | High; can damage general ability |
from transformers import (
AutoModelForSequenceClassification, AutoTokenizer, Trainer, TrainingArguments,
)
name = "distilbert-base-uncased"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForSequenceClassification.from_pretrained(name, num_labels=2)
def encode(batch):
return tok(batch["text"], truncation=True, max_length=256, padding="max_length")
train_ds, val_ds = train_raw.map(encode), val_raw.map(encode)
args = TrainingArguments(
output_dir="out",
learning_rate=2e-5, # above roughly 5e-5 this task diverges
per_device_train_batch_size=16,
num_train_epochs=3, # 2-4 is usually enough; more overfits
eval_strategy="epoch",
save_strategy="epoch",
load_best_model_at_end=True,
metric_for_best_model="f1",
)
Trainer(model=model, args=args, train_dataset=train_ds, eval_dataset=val_ds).train()Evaluate before you believe it
- Split by entity or by time, not randomly. Random splits leak near-duplicate examples into the test set and inflate every score.
- Report per-class precision and recall, not just accuracy, whenever classes are imbalanced.
- Keep a frozen set of negative and adversarial examples: empty strings, sarcasm, mixed languages, text you already misclassified.
- Compare against the simplest baseline you have - most-frequent class, TF-IDF, or a prompted model. A fine-tuned model that barely beats them is not worth the maintenance.
- Inspect the largest error cluster by hand before you tune anything. What you find usually points to label noise, not to the architecture.
FAQ
Can I fine-tune to add new factual knowledge?
How much data do I need?
Related
Word and sentence embeddings Choosing and evaluating models
Last refreshed 2026-09-18.