UTF-16, UTF-32, BOM and endianness

Code units versus code points, surrogate pairs, what a byte order mark really is, why Windows and Java default to UTF-16 internally, and when a BOM helps or hurts.

Code units are not code points

UTF-16 stores most characters in one 16-bit code unit. Characters above U+FFFF need two code units, a surrogate pair, which is why string length in UTF-16 is not the number of characters.

const s = "A\u{1F600}";
s.length;                   // 3 — UTF-16 code units, not code points
[...s].length;              // 2 — code points, thanks to the iterator
s.charAt(1);                // a lone high surrogate — a broken half

// correct iteration
for (const ch of s) console.log(ch.codePointAt(0).toString(16));
// 41
// 1f600

Character	Code point	UTF-8 bytes	UTF-16 units
A	U+0041	1	1
e acute	U+00E9	2	1
Euro sign	U+20AC	3	1
Grinning face	U+1F600	4	2 (surrogate pair)
Rare ideograph	U+20000	4	2

The byte order mark

UTF-8        EF BB BF              (a marker, not needed for order)
UTF-16 BE    FE FF                 (big endian)
UTF-16 LE    FF FE                 (little endian)
UTF-32 BE    00 00 FE FF
UTF-32 LE    FF FE 00 00

In UTF-16 the BOM is a real byte-order indicator, because the file could legitimately be either.
In UTF-8 there is only one byte order, so the BOM carries no information — it is only a signature.
Some tools treat a UTF-8 BOM as part of the first line, which is why the first CSV header can come back with an invisible prefix.
The Unicode consortium recommends against a UTF-8 BOM, but Windows tooling and Excel emit one anyway.

open("f.csv", encoding="utf-8")       # leaves the BOM in the first field
open("f.csv", encoding="utf-8-sig")   # strips it if present, harmless if absent

# check for a BOM before trusting the first byte
raw = open("f.csv", "rb").read(3)
print(raw == b"\xef\xbb\xbf")        # True when a UTF-8 BOM is present

Where each encoding lives

Platform	Internal string	File default	Note
JavaScript / DOM	UTF-16	UTF-8 over the wire	`length` counts code units
Java	UTF-16	Platform default, historically	Use explicit charsets on every stream
Windows APIs	UTF-16 (`wchar_t` is 16-bit)	UTF-16 LE with BOM	Legacy code pages elsewhere
Linux / macOS	`wchar_t` is 32-bit	UTF-8	The BOM is unusual
Python 3	Code points	UTF-8 source	Indexing is by code point
Go / Rust	UTF-8 bytes	UTF-8	Indexing is by byte; iterate runes or chars

⚠️

Slicing a UTF-16 or UTF-8 string by index can split a surrogate pair or a multi-byte sequence, producing invalid text. Always slice on code point boundaries or use a library that understands grapheme clusters, and remember that one visible character can be several code points.

FAQ

Should I write a UTF-8 BOM?

Generally no. It breaks naive parsers and adds nothing, since there is only one UTF-8 byte order. Accept and strip it on read for interoperability with tools that emit one.

Why is my string length wrong for emoji?

You are counting UTF-16 code units or UTF-8 bytes. Count code points, or better, count grapheme clusters if the number must match what a user perceives.

Unicode in depth: planes, properties and normalisation Bytes, bits, hex and number bases

Last refreshed 2026-09-18.

UTF-16, UTF-32, BOM and endianness

Code units are not code points

The byte order mark

Where each encoding lives

FAQ

Related