Text preprocessing and tokens
Turn raw text into clean, tokenised input: normalisation, a sparse baseline, and how subword tokenizers actually split words.
Cleaning without destroying signal
Most failures that look like a bad model are really inconsistent input. Normalise first: unify Unicode forms, collapse whitespace, and decide deliberately what to remove. Every removal is a lossy choice, so keep the transformations minimal, documented and reversible.
import re
import unicodedata
def normalise(text):
# NFKC folds full-width characters, ligatures and compatibility forms
text = unicodedata.normalize("NFKC", text)
text = re.sub(r"https?://\S+", " ", text) # URLs are noise for most tasks
text = re.sub(r"[ \t]+", " ", text) # collapse horizontal whitespace
return text.strip()
print(normalise("Cafe\u0301 costs 5 pounds")) # "Cafe costs 5 pounds"- Lowercase only when case carries no meaning:
Applethe company andapplethe fruit are different things. - Do not strip punctuation before sentiment or negation analysis - removing
notflips the label. - Remove stop words only for keyword-style search, and never before a transformer, which was trained on natural text.
- Keep a raw copy of every record. You will want to re-run preprocessing with different rules, and the raw text is your only way back.
From strings to tokens
A tokenizer maps text to integers from a fixed vocabulary. Subword tokenizers split rare words into common pieces, which is why a model can handle a word it has never seen as a whole.
| Tokenizer | Used by | How it splits |
|---|---|---|
| BPE | GPT family, RoBERTa | Merges the most frequent character pairs; rare words become a sequence of common pieces |
| WordPiece | BERT, DistilBERT | Like BPE but merges by likelihood gain; continuation pieces carry a ## marker |
| Unigram / SentencePiece | T5, LLaMA, many multilingual models | Language-agnostic, runs on raw text with no assumption about spaces |
| Whitespace / regex | Classic pipelines, TF-IDF | Splits on spaces and punctuation; fast, but the vocabulary grows without bound |
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("bert-base-uncased")
ids = tok("Tokenization isn't trivial.", add_special_tokens=True)["input_ids"]
print(tok.convert_ids_to_tokens(ids))
# ['[CLS]', 'token', '##ization', 'isn', "'", 't', 'trivial', '.', '[SEP]']
print(len(tok("unbelievable")["input_ids"]) - 2) # pieces inside the special tokens⚠️
Cost, context limits and truncation are measured in tokens, not words. English averages roughly 4 characters per token, but code, numbers and non-Latin scripts are far more expensive. Always count with the tokenizer of the exact model you are calling, and never pair a tokenizer from one checkpoint with the weights of another.
FAQ
Should I remove stop words before training a classifier?
Usually no. Modern models learn that frequent words carry little weight on their own, and removing them also removes negation and question cues. Test it: on short text the loss often outweighs the speed gain.
My model truncates long documents. What should I do?
Chunk the document and aggregate per-chunk predictions, or retrieve only the relevant passages. Truncating at the token limit silently discards the end of the input, which is often where the answer lives.
Related
Word and sentence embeddings Machine learning in one page
Last refreshed 2026-09-18.