Translation and multilingual pipelines

Sequence-to-sequence and multilingual models, tokenisation for non-Latin scripts, quality estimation, and the post-processing translation actually needs.

Choosing a translation model

from transformers import pipeline, AutoTokenizer, AutoModelForSeq2SeqLM

# a bilingual model: one pair, usually higher quality
en_de = pipeline("translation_en_to_de", model="Helsinki-NLP/opus-mt-en-de")
print(en_de("The deployment failed because the database was unreachable.")[0]["translation_text"])

# a multilingual model: many pairs, one artefact to operate
tok = AutoTokenizer.from_pretrained("facebook/nllb-200-distilled-600M")
model = AutoModelForSeq2SeqLM.from_pretrained("facebook/nllb-200-distilled-600M")

def translate(text, src="eng_Latn", tgt="fra_Latn", max_new_tokens=256):
    tok.src_lang = src
    inputs = tok(text, return_tensors="pt", truncation=True, max_length=512)
    generated = model.generate(
        **inputs,
        forced_bos_token_id=tok.convert_tokens_to_ids(tgt),
        max_new_tokens=max_new_tokens,
        num_beams=4,
    )
    return tok.batch_decode(generated, skip_special_tokens=True)[0]

print(translate("The deployment failed because the database was unreachable."))

• Set forced_bos_token_id for the target language; without it a multilingual model picks the target from the prompt and can silently translate into the wrong language.
• Beam search (4-5 beams) is standard for translation. Sampling makes output non-reproducible, which is unacceptable for a cached or reviewed artefact.
• Check the language code table for your checkpoint. NLLB uses BCP-47-like codes (eng_Latn), OPUS uses two-letter codes, and mixing them produces garbage rather than an error.

Tokenisation and non-Latin scripts

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("facebook/nllb-200-distilled-600M")

samples = {
    "english": "Deployment failed.",
    "german": "Die Bereitstellung ist fehlgeschlagen.",
    "japanese": "Deployment failed.",
    "arabic": "Deployment failed.",
}

for name, text in samples.items():
    ids = tok(text)
    print(f"{name:10s} tokens={len(ids['input_ids']):3d} chars={len(text):3d}")

# a token budget is not a character budget: plan capacity per language
FRAGMENTS = {"eng_Latn": 1.0, "deu_Latn": 1.3, "jpn_Jpan": 1.8, "ara_Arab": 2.2}
def estimated_tokens(character_count, lang_code):
    return int(character_count / 3.5 * FRAGMENTS.get(lang_code, 1.5))

• A token in English is roughly four characters; Japanese, Chinese, Thai and Arabic often need one to three tokens per character. Capacity planning based on English will overflow on other languages.
• SentencePiece and BPE vocabularies under-represent low-resource languages, so the same meaning costs more tokens and more compute. This is a real cost asymmetry, not a rounding error.
• Normalise Unicode before tokenising but preserve it afterwards: NFC normalisation merges combining marks, and some scripts need them intact for correct rendering.
• Never strip diacritics to simplify a language. In Arabic and Hebrew they can be semantically meaningful, and in Vietnamese they change the word entirely.

Quality estimation and post-processing

from transformers import pipeline

qe = pipeline("text-classification", model="Unbabel/wmt22-cometkiwi-da")
# a reference-free score: how good is this translation likely to be?
score = qe({"src_text": source, "tgt_text": translation})
print(score)

def postprocess(text, target_lang):
    import re
    # remove the model's stray language tokens and duplicated whitespace
    text = re.sub(r"\b(eng_Latn|fra_Latn|deu_Latn)\b", "", text)
    text = re.sub(r"\s+([.,;:!?])", r"\1", text)
    text = re.sub(r"[ \t]{2,}", " ", text).strip()
    if target_lang in {"fra_Latn", "deu_Latn", "spa_Latn"}:
        text = text.replace(" ,", ",").replace(" .", ".")
    return text

print(postprocess("The build failed , please retry . eng_Latn", "eng_Latn"))

# protect placeholders and code before translating, restore after
def protect(text, pattern=r"\{[a-z_]+\}|<[^>]+>|\b[A-Z_]{3,}\b"):
    import re
    found = re.findall(pattern, text)
    for i, token in enumerate(found):
        text = text.replace(token, f"__PH{i}__")
    return text, found

def restore(text, found):
    for i, token in enumerate(found):
        text = text.replace(f"__PH{i}__", token)
    return text

FAQ

Related

Evaluating NLP systems Text preprocessing and tokens

Last refreshed 2026-09-18.