HowToCodePage

unicode and str are different things

Python 2 has a bytes type called str and a text type called unicode, plus hidden ASCII conversions that fail on any non-ASCII input.

Two string types

In Python 3 there is one text type, str, and one byte type, bytes. In Python 2 the names are reused: str holds bytes and unicode holds text. Everything confusing about legacy Python strings follows from that.

# -*- coding: utf-8 -*-
name = "caf\xc3\xa9"       # str: these are raw UTF-8 bytes, not text
text = u"caf\xe9"          # unicode: a sequence of code points

print type(name), len(name)      # <type 'str'> 5   (c, a, f, then two UTF-8 bytes)
print type(text), len(text)      # <type 'unicode'> 4

print name.decode("utf-8")       # u'caf\xe9'  - bytes in, text out
print text.encode("utf-8")       # 'caf\xc3\xa9' - text in, bytes out
print text.encode("ascii")    # UnicodeEncodeError: 'ascii' codec can't encode ...
print name + text             # UnicodeDecodeError: the str is decoded as ASCII first
print str(text)               # usually raises, and is never the right conversion
⚠️
When you see UnicodeDecodeError, the answer is almost never a different encode call. Decode the incoming bytes once, at the boundary where they enter your program, keep everything as unicode internally, and encode once on the way out. Mixing the two types makes Python 2 insert hidden ASCII conversions that pass every test written in English and fail on the first real user name.

Rules that keep a legacy codebase sane

OperationResult in Python 2Trap
type("abc")str, meaning byteslen() counts bytes, not characters
type(u"abc")unicodeOnly unicode literals interpret backslash-u escapes
"abc" + u"abc"unicodeThe byte string is decoded with ASCII and raises if it is not ASCII
u"...".encode("utf-8")A str of bytesDo this once, at the I/O boundary
str(u"...")Often raisesUses ASCII; never treat it as a converter
open(path).read()A str of bytesPython 2 text mode performs no decoding at all
io.open(path, encoding="utf-8")unicodeGives Python 3 file semantics in Python 2 today
u"a" in "\xc3\xa9"Either works or raisesComparisons also trigger the implicit decode
  • Use io.open instead of open for text files, and always pass encoding=.
  • Prefer u"..." for every literal you control, or add from __future__ import unicode_literals and get that behaviour for plain quotes too.
  • Remember len(text) counts code points, not bytes and not grapheme clusters — combining accents still count separately.
  • When a library returns bytes, decode immediately rather than passing them around and decoding at random call sites later.

FAQ

Is u"..." the same as Python 3 str?
Semantically yes: both are sequences of code points. The practical differences are that Python 2 repr adds a u prefix and that print encodes to the terminal encoding, which may raise on a non-UTF-8 console.
What is basestring?
An abstract superclass of both str and unicode, so isinstance(x, basestring) is the Python 2 way to ask "is this a string?". In Python 3 the equivalent is str, and six.string_types picks the right one for you.

The print statement and integer division UTF-8 and character sets

Last refreshed 2026-09-18.