HowToCodePage

Mojibake: diagnosing and fixing broken text

Recognising the classic UTF-8-read-as-Latin-1 damage, recovering the original bytes, finding the wrong decode point, and preventing the bug from returning.

Recognising the damage

You seeIntendedWhat happened
cafécaféUTF-8 bytes read as Latin-1
’right single quoteUTF-8 apostrophe read as Latin-1
�U+FFFDA replacement character was itself mis-decoded
cafécaféHTML entity never decoded
cafécaféDouble-encoded: UTF-8 applied twice
?? or boxesanything non-LatinDecoded with a lossy 7-bit encoding
intended : cafe + U+0301 combining acute  (or e-acute U+00E9)
UTF-8    : 63 61 66 C3 A9
read as  : Latin-1 -> A-tilde, copyright sign
displayed: "café"

The à family is the signature. Whenever a text is full of A-tilde characters, UTF-8 bytes have been interpreted as a single-byte code page.

Recovering the original bytes

broken = "café"

# the fix: re-encode with the wrong encoding, then decode with the right one
fixed = broken.encode("latin-1").decode("utf-8")
print(fixed)      # café

# validate before trusting the result
def repair(s):
    for wrong, right in (("latin-1", "utf-8"), ("cp1252", "utf-8")):
        try:
            candidate = s.encode(wrong).decode(right)
            if "\ufffd" not in candidate:
                return candidate
        except (UnicodeEncodeError, UnicodeDecodeError):
            continue
    return s

print(repair(broken))             # café
print(repair("already fine"))     # already fine
  • The repair only works while the wrong encoding preserved every byte: Latin-1 and cp1252 are lossless, so no information was destroyed.
  • If the text passed through a lossy decode such as ascii with replacement, the original bytes are gone and the text cannot be recovered.
  • Always check for U+FFFD after repairing; a clean result with no replacement character is a strong signal.
  • Store the fixed value only after confirming a sample by eye; an automated repair on already-correct text can corrupt it.

Finding the wrong decode point

  1. Find the outermost layer: check the raw bytes on the wire before any parsing.
  2. Declare the encoding for every boundary — HTTP header, database connection, file open, template render.
  3. Send Content-Type: text/html; charset=utf-8 and set the same charset in the document metadata.
  4. Configure the database client and the table collation to UTF-8, not just the server default.
  5. Add a test fixture with an accented letter and a non-Latin script, and assert on the exact bytes.
  6. Round-trip every value at the boundary: encode then decode and compare before writing.
# locate the layer by inspecting the raw response
curl -sI https://example.com/page | grep -i content-type
curl -s https://example.com/page | xxd | head -4

# and the terminal is a layer too: check that it renders UTF-8
echo $LANG ; locale | head -3
⚠️
The database is the most common hiding place. A column declared with a legacy character set can accept UTF-8 bytes and return them mangled, and the damage happens at write time — so fixing the connection later does not repair the rows already stored.

FAQ

Can mojibake be reversed?
Usually yes, if the wrong decoding was lossless such as Latin-1 or cp1252. Once a value has been decoded with replacement characters, or transcoded through a lossy single-byte set, the original bytes are gone.
Why does it look correct in my terminal but wrong in the database?
Your terminal is probably being lenient or is configured as UTF-8 while the connection is not. Inspect the bytes with a hex dump rather than trusting a renderer.

Character sets: ASCII, Latin-1 and Windows-1252 Unicode in depth: planes, properties and normalisation

Last refreshed 2026-09-18.