Character Encodings cheat sheet

A scannable Character Encodings reference: 32 short snippets across 14 topics, each linking back to the lesson it came from.

At a glance

Quick snippets

Base64 encoding

What Base64 is

Man → TWFu

M a n
01001101 01100001 01101110   (3 bytes / 24 bits)
|||||||| |||||||| ||||||||
T  W  F  u  (split into four 6-bit groups → 4 Base64 chars)

Full lesson: Base64 encoding →

URL encoding (percent-encoding)

What percent-encoding is

hello world      → hello%20world
c++ & c#        → c%2B%2B%20%26%20c%23
price=€10        → price%3D%E2%82%AC10   (€ is 3 UTF-8 bytes: E2 82 AC)

Encoding is over bytes, not characters

// JavaScript
const s = 'café';
const enc = encodeURIComponent(s); // "caf%C3%A9"
const dec = decodeURIComponent(enc);   // "café"

Full lesson: URL encoding (percent-encoding) →

UTF-8 and character sets

How UTF-8 works

A      → 0x41            (1 byte)
é      → 0xC3 0xA9      (2 bytes)
中      → 0xE4 0xB8 0xAD  (3 bytes)
😀  → 0xF0 0x9F 0x98 0x80 (4 bytes)

Full lesson: UTF-8 and character sets →

Bytes, bits, hex and number bases

Three notations for the same value

n = 0x41              # hex literal, value 65
bin(n)                # '0b1000001'
hex(n)                # '0x41'
f"{n:08b}"            # '01000001'  eight bits, zero padded
f"{n:02X}"            # '41'        two hex digits, uppercase

int("ff", 16)         # 255
int("11111111", 2)    # 255

# shifting is how you pack and unpack bytes
high, low = (n >> 4) & 0xF, n & 0xF
packed = (high << 4) | low            # 0x41 again

Reading a hex dump

00000000  50 4b 03 04 14 00 00 00  08 00 21 8a 4c 5d 19 3c  |PK........!.L].<|
00000010  d4 03 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  68 65 6c 6c 6f 2e 74 78  74 0a 77 6f 72 6c 64 0a  |hello.txt.world.|
            ^                                               ^
            byte offset (hex)                               ASCII rendering

Reading a hex dump

xxd file.bin | head
hexdump -C file.bin | head
xxd -s 0x20 -l 16 file.bin           # 16 bytes starting at offset 32
printf 'hello' | xxd                  # 68656c6c6f

Full lesson: Bytes, bits, hex and number bases →

Character sets: ASCII, Latin-1 and Windows-1252

ASCII is only 7 bits

b"A".decode("ascii")        # 'A'
chr(65)                     # 'A'
ord("A")                    # 65

b"\x80".decode("ascii")     # UnicodeDecodeError — outside the 7-bit range
b"\x80".decode("latin-1")   # '\x80' — Latin-1 maps every byte to a code point

The high half became a collision zone

# the same bytes, three different stories
raw = b"caf\xe9 \x92quoted\x92 \x80 10"

raw.decode("latin-1")        # 'café ’quoted’ € 10'
raw.decode("cp1252")         # 'café ’quoted’ € 10'  — the intended text
raw.decode("utf-8", "replace")  # 'caf��quoted...' — bytes are not valid UTF-8

Living with legacy text

def decode_legacy(raw: bytes) -> str:
    for enc in ("utf-8", "cp1252", "latin-1"):
        try:
            return raw.decode(enc)
        except UnicodeDecodeError:
            continue
    raise ValueError("no candidate encoding matched")

Full lesson: Character sets: ASCII, Latin-1 and Windows-1252 →

HTML entities and escaping in markup

Three ways to write the same character

<p>Tom &amp; Jerry &lt;script&gt; &copy; 2026 &#8364;10 &#x1F600;</p>

<!-- the five characters that must always be escaped in text content -->
<!-- &  ->  &amp;      <  ->  &lt;      >  ->  &gt;
     "  ->  &quot;     '  ->  &#39; -->

The context decides the rule

// the safe approach: set text, let the platform escape
el.textContent = userInput;                 // never innerHTML for untrusted text
el.setAttribute("data-note", userInput);    // the DOM escapes the value for you

// if you must build a string, escape every context separately
const esc = (s) => s.replace(/[&<>"']/g, (c) => ({
  "&": "&amp;", "<": "&lt;", ">": "&gt;", '"': "&quot;", "'": "&#39;",
}[c]));
el.innerHTML = "<b>" + esc(userInput) + "</b>";

Double escaping and double decoding

user types        : Tom & Jerry
escaped once      : Tom &amp; Jerry          <- correct
escaped twice     : Tom &amp;amp; Jerry      <- renders as "Tom &amp; Jerry"

decoded twice     : Tom & Jerry              <- fine
                    but &lt;b&gt; becomes <b>  <- now live markup

Full lesson: HTML entities and escaping in markup →

Form encoding vs percent-encoding

application/x-www-form-urlencoded

POST /search HTTP/1.1
Content-Type: application/x-www-form-urlencoded

q=rock+%26+roll&page=2&tag=a%2Fb

  q     = "rock & roll"    (+ means space, %26 is a literal ampersand)
  page  = "2"
  tag   = "a/b"            (%2F is an escaped slash, so it is not a path separator)

The decoding bugs that reach production

from urllib.parse import parse_qs, unquote, quote

body = "q=rock+%26+roll&tag=a%2Fb"
params = parse_qs(body, keep_blank_values=True)
# {'q': ['rock & roll'], 'tag': ['a/b']}   plus and %26 both resolved

unquote("a+b")        # 'a+b'  unquote does NOT treat plus as space
unquote_plus("a+b")   # 'a b'  the form rule

Full lesson: Form encoding vs percent-encoding →

Unicode in depth: planes, properties and normalisation

Code points, planes and surrogates

s = "A\U0001F600"          # A plus a smiley
len(s)                     # 2 code points
[len(ch) for ch in s]      # [1, 1] — Python indexes by code point

import unicodedata
hex(ord(s[1]))             # '0x1f600'
unicodedata.name(s[1])     # 'GRINNING FACE'

Combining marks and the same string twice

import unicodedata

a = "caf\u00e9"            # e with acute, one code point  (NFC-like)
b = "cafe\u0301"           # e followed by combining acute  (NFD-like)

a == b                      # False — different code points
len(a), len(b)              # (4, 5)

unicodedata.normalize("NFC", a) == unicodedata.normalize("NFC", b)   # True
unicodedata.normalize("NFD", a) == unicodedata.normalize("NFD", b)   # True

Comparing, hashing and storing

import unicodedata

def canonical_key(s: str) -> str:
    # casefold handles more than lower(), including the German sharp s
    return unicodedata.normalize("NFC", s).casefold()

canonical_key("STRASSE") == canonical_key("strasse")   # True
canonical_key("Stra\u00dfe")                            # 'strasse'

Full lesson: Unicode in depth: planes, properties and normalisation →

UTF-16, UTF-32, BOM and endianness

Code units are not code points

const s = "A\u{1F600}";
s.length;                   // 3 — UTF-16 code units, not code points
[...s].length;              // 2 — code points, thanks to the iterator
s.charAt(1);                // a lone high surrogate — a broken half

// correct iteration
for (const ch of s) console.log(ch.codePointAt(0).toString(16));
// 41
// 1f600

The byte order mark

UTF-8        EF BB BF              (a marker, not needed for order)
UTF-16 BE    FE FF                 (big endian)
UTF-16 LE    FF FE                 (little endian)
UTF-32 BE    00 00 FE FF
UTF-32 LE    FF FE 00 00

The byte order mark

open("f.csv", encoding="utf-8")       # leaves the BOM in the first field
open("f.csv", encoding="utf-8-sig")   # strips it if present, harmless if absent

# check for a BOM before trusting the first byte
raw = open("f.csv", "rb").read(3)
print(raw == b"\xef\xbb\xbf")        # True when a UTF-8 BOM is present

Full lesson: UTF-16, UTF-32, BOM and endianness →

Escape sequences across languages

JSON escapes are a closed set

{
  "quote":   "she said \"hi\"",
  "path":    "C:\\Users\\dev",
  "newline": "line one\nline two",
  "tab":     "col1\tcol2",
  "unicode": "\u20ac  \ud83d\ude00",
  "control": "\u0007"
}

JSON escapes are a closed set

intended JSON   : {"re": "\d+"}
written as JS   : const s = '{"re": "\\d+"}';      // four slashes in source
written as JSON : {"re": "\\d+"}                    // two slashes in the document

Per-language escapes

import re, json

# a backslash-hungry regular expression
pattern = r"\d{4}-\d{2}-\d{2}"        # raw string: no escape processing
re.fullmatch(pattern, "2026-09-18")

# the same value as JSON text needs the backslash doubled
json.dumps({"re": pattern})   # '{"re": "\\d{4}-\\d{2}-\\d{2}"}'

Full lesson: Escape sequences across languages →

Mojibake: diagnosing and fixing broken text

Recognising the damage

intended : cafe + U+0301 combining acute  (or e-acute U+00E9)
UTF-8    : 63 61 66 C3 A9
read as  : Latin-1 -> A-tilde, copyright sign
displayed: "café"

Finding the wrong decode point

# locate the layer by inspecting the raw response
curl -sI https://example.com/page | grep -i content-type
curl -s https://example.com/page | xxd | head -4

# and the terminal is a layer too: check that it renders UTF-8
echo $LANG ; locale | head -3

Full lesson: Mojibake: diagnosing and fixing broken text →

Quoted-printable, MIME and email encoding

Email was designed for 7-bit text

Subject: =?UTF-8?Q?Caf=C3=A9_meeting?=
From: =?UTF-8?B?QW5hIFBlem5pY2s=?= <[email protected]>
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

The caf=C3=A9 is open until 18:00 =
on weekdays.

Full lesson: Quoted-printable, MIME and email encoding →

Punycode and internationalised domain names

Two forms of the same name

import idna

u_label = "münchen"
a_label = idna.encode(u_label).decode()
print(a_label)                  # 'xn--mnchen-3ya'

print(idna.decode(a_label))     # 'münchen'

# a full domain, label by label
print(idna.encode("café.example").decode())   # 'xn--caf-dma.example'

What IDNA actually normalises away

# see what the wire actually carries
python -c "import idna; print(idna.encode('münchen.de').decode())"
curl -sv https://xn--mnchen-3ya.de/ 2>&1 | grep -i '^> Host'

# resolve the ASCII form directly
dig +short xn--mnchen-3ya.de

Homographs and why browsers hide some names

# two different strings, identical on screen in many fonts
a = "apple"                  # Latin small a ... (all Latin)
b = "\u0430pple"             # Cyrillic small a, then Latin pple

a == b                       # False
print(a, b)                  # they look the same

Full lesson: Punycode and internationalised domain names →

Compression vs encoding: gzip, deflate and Brotli

Negotiating a content coding

GET /app.js HTTP/1.1
Accept-Encoding: br, gzip, deflate
Host: example.com

HTTP/1.1 200 OK
Content-Encoding: br
Content-Type: application/javascript
Vary: Accept-Encoding
Content-Length: 48213

<compressed bytes>

Negotiating a content coding

# compare algorithms on a real file
for a in gzip br zstd; do
  printf '%-6s ' "$a"
  case $a in
    gzip) gzip -9 -c app.js | wc -c ;;
    br)   brotli -q 11 -c app.js | wc -c ;;
    zstd) zstd -19 -c app.js | wc -c ;;
  esac
done
wc -c app.js

Full lesson: Compression vs encoding: gzip, deflate and Brotli →

FAQ

Related cheat sheets

Algorithms Data Structures Computer Networks Operating Systems Hashing & Checksums Data Formats

Last refreshed 2026-09-27.