HowToCodePage

Files, the codecs module and text vs binary I/O

Read and write non-ASCII text without corrupting it, handle byte-order marks, and know why the Python 2 open() is always binary.

open() is binary

# Python 2: open() reads bytes. 'r' does newline translation, not decoding.
f = open("notes.txt", "rb")
raw = f.read()                 # str (bytes)
f.close()

print type(raw)                # <type 'str'>
text = raw.decode("utf-8")     # unicode
print type(text)               # <type 'unicode'>

In Python 2 a file opened in text mode still gives you bytes; the mode only affects newline handling on Windows. Decoding is entirely your responsibility, and str is bytes while unicode is text.

  • Put # -*- coding: utf-8 -*- at the top of every source file that contains non-ASCII literals, and from __future__ import unicode_literals so string literals are unicode.
  • No encoding declaration means Python 2 assumes ASCII for the source file, and non-ASCII bytes raise SyntaxError.
  • The same trap applies to output: print of a unicode object encodes with ascii and fails on anything outside it.

The codecs module

# -*- coding: utf-8 -*-
from __future__ import unicode_literals
import codecs

# read and write text with an explicit encoding
with codecs.open("notes.txt", "r", encoding="utf-8") as f:
    text = f.read()

with codecs.open("out.txt", "w", encoding="utf-8") as f:
    f.write(text)

# and for stdout, which is the usual source of UnicodeEncodeError
import sys
sys.stdout = codecs.getwriter("utf-8")(sys.stdout)

# strip a byte-order mark that Excel likes to add
with codecs.open("export.csv", "r", encoding="utf-8-sig") as f:
    rows = [line.rstrip("\r\n").split(",") for line in f]
CodecBehaviour
utf-8No BOM; the correct choice for data interchange
utf-8-sigConsumes a leading BOM on read, writes one on write
utf-16Writes a BOM and uses native byte order; common from Windows tools
latin-1Never fails, maps bytes one to one; useful for recovering unknown data
asciiRaises on the first byte above 127; the default that causes most bugs
# 'latin-1' is the escape hatch when you must not lose bytes
data = open("unknown.bin", "rb").read()
guessed = data.decode("utf-8", "replace")     # lossy but readable
safe = data.decode("latin-1")                 # lossless, may look wrong

Decoding with errors="replace" or "ignore" is fine for logs and search, but never for data you will write back: information is destroyed. Use latin-1 when you need a lossless round trip.

Round-trip discipline

# The rule: decode at the boundary, work in unicode, encode at the boundary.
def read_lines(path):
    with codecs.open(path, "r", encoding="utf-8-sig") as f:
        return [line.rstrip(u"\r\n") for line in f]     # unicode in memory

def write_lines(path, lines):
    with codecs.open(path, "w", encoding="utf-8") as f:
        for line in lines:
            f.write(line + u"\n")

# and prove the round trip, which is the test that catches encoding bugs
import os
os.rename("out.txt", "tmp.txt")
assert read_lines("tmp.txt") == read_lines("notes.txt")
💡
The UnicodeDecodeError and UnicodeEncodeError you see in Python 2 almost always come from an implicit conversion: concatenating str with unicode, comparing them, or str() on a unicode object. Track down the implicit conversion rather than sprinkling encode and decode everywhere.

FAQ

Why does print fail with UnicodeEncodeError only in some terminals?
The output encoding depends on the terminal's locale. A pipe to a file often uses ASCII, so the same code works interactively and fails in a cron job. Set PYTHONIOENCODING=utf-8, or wrap sys.stdout with a codecs writer.
Should I open files as 'rb' or 'r' in Python 2?
Use 'rb' for anything binary and codecs.open(..., encoding=...) for text. A bare 'r' gives you bytes with newline translation, which is rarely what you want and differs from Python 3.

unicode and str are different things Testing legacy Python 2 code before you change it

Last refreshed 2026-09-18.