Categoricals, dtypes and memory optimisation

The category dtype, nullable integers and booleans, deliberate downcasting, and how to measure what a frame actually costs in memory.

Knowing the dtypes

dtype	Stores	Missing value
`int64`	Whole numbers	Impossible - a single `NaN` forces `float64`
`Int64`	Nullable whole numbers	`pd.NA`
`float64`	Decimals and any number with gaps	`NaN` (a float, which is why sums ignore it)
`boolean`	Nullable booleans	`pd.NA`
`object`	Anything: text, mixed types, lists	`None` or `NaN`
`string`	Text with a declared type	`pd.NA`
`category`	A small set of repeated labels plus codes	`NaN`
`datetime64[ns, tz]`	Timestamps with a zone	`NaT`

df.dtypes
df.select_dtypes(include="number").columns
df.select_dtypes(include="category").columns
df["year"].astype("Int64")            # capital I: nullable
df["ok"].astype("boolean")
df["name"].astype("string")

⚠️

One missing value in an integer column silently promotes the whole column to float64, and account numbers become 1234.0. If a numeric column is missing data, decide explicitly between a nullable Int64 and filling the gap.

The category dtype

df["team"] = df["team"].astype("category")
df["team"].cat.categories          # Index(['A', 'B'], dtype='object')
df["team"].cat.codes               # the integer codes actually stored

df["size"] = pd.Categorical(
    df["size"], categories=["s", "m", "l"], ordered=True
)
df["size"] > "m"                   # meaningful only because ordered=True

df["team"] = df["team"].cat.add_categories(["bench"]).cat.remove_unused_categories()
df["team"].value_counts(dropna=False)

A category column stores small integers plus one shared list of labels, which is a large saving on repeated text.
Grouping and sorting by a categorical column use the code order - define it with ordered=True when the order carries meaning.
observed=True in groupby stops the combination of every category pair from exploding the result.
Adding a value that is not in the category list turns the column into object, silently losing the memory win.

Worked example: measuring before and after

import pandas as pd

before = df.memory_usage(deep=True).sum() / 1024 ** 2
print(f"{before:.1f} MB")        # f-strings here are Python, not the outer template

df["team"]    = df["team"].astype("category")
df["country"] = df["country"].astype("category")
df["year"]    = pd.to_numeric(df["year"], downcast="integer")
df["amount"]  = pd.to_numeric(df["amount"], downcast="float")
df["note"]    = df["note"].astype("string")

after = df.memory_usage(deep=True).sum() / 1024 ** 2
print(df.memory_usage(deep=True).sort_values(ascending=False).head(10))
print(f"{after:.1f} MB, {100 * after / before:.0f}% of the original")

Always measure with deep=True: without it, an object column is reported as 8 bytes per row because only the pointers are counted, which hides exactly the columns you are trying to fix.

FAQ

When is category a bad idea?

When the column is nearly unique, such as identifiers or free text. You pay for the label list with no reuse, and operations on categories can be slower than on plain strings.

Why did my merge turn categories back into object?

Merging two categorical columns whose category lists differ produces an object column. Unify the categories with astype("category") after the merge, or set them explicitly on both sides first.

apply, map and writing fast pandas code String operations with .str

Last refreshed 2026-09-18.

Categoricals, dtypes and memory optimisation

Knowing the dtypes

The category dtype

Worked example: measuring before and after

FAQ

Related