Unicode in depth: planes, properties and normalisation

BMP and supplementary planes, surrogate ranges, combining marks, and the four normalisation forms that decide whether two visually identical strings are equal.

Code points, planes and surrogates

s = "A\U0001F600"          # A plus a smiley
len(s)                     # 2 code points
[len(ch) for ch in s]      # [1, 1] — Python indexes by code point

import unicodedata
hex(ord(s[1]))             # '0x1f600'
unicodedata.name(s[1])     # 'GRINNING FACE'

A lone surrogate in a UTF-8 document is invalid data. Encoders reject it; some languages silently substitute U+FFFD, which is why broken emoji often appear as a replacement character.

Combining marks and the same string twice

import unicodedata

a = "caf\u00e9"            # e with acute, one code point  (NFC-like)
b = "cafe\u0301"           # e followed by combining acute  (NFD-like)

a == b                      # False — different code points
len(a), len(b)              # (4, 5)

unicodedata.normalize("NFC", a) == unicodedata.normalize("NFC", b)   # True
unicodedata.normalize("NFD", a) == unicodedata.normalize("NFD", b)   # True

Compatibility normalisation loses information: １ (full-width one) becomes 1, and fi as a ligature becomes two letters. That is desirable for search and dangerous for passwords or display.

Comparing, hashing and storing

• Normalise to NFC before storing, unless you have a reason not to.
• Normalise to NFC (or NFKC for identifiers) before comparing or hashing.
• Normalise and case-fold before building a username uniqueness index.
• Never normalise a password except for the documented form; changing it later locks users out.
• Unicode data changes between releases, so normalisation is not perfectly stable across versions — pin the runtime for reproducibility.

import unicodedata

def canonical_key(s: str) -> str:
    # casefold handles more than lower(), including the German sharp s
    return unicodedata.normalize("NFC", s).casefold()

canonical_key("STRASSE") == canonical_key("strasse")   # True
canonical_key("Stra\u00dfe")                            # 'strasse'

FAQ

Related

UTF-16, UTF-32, BOM and endianness Mojibake: diagnosing and fixing broken text

Last refreshed 2026-09-18.