HowToCodePage

Debugging pandas: SettingWithCopy, dtype and index surprises

The three failures that waste the most time - chained assignment, silent dtype coercion and misaligned indexes - and how to catch them early.

Chained indexing and SettingWithCopyWarning

# warns, and may write to a throwaway copy: nothing downstream changes
sub = df[df["a"] > 1]
sub["b"] = 0

# correct: one .loc, one assignment
df.loc[df["a"] > 1, "b"] = 0

# correct: an independent frame you intend to modify
sub = df.loc[df["a"] > 1].copy()
sub["b"] = 0

# legacy, avoid in new code
pd.set_option("mode.chained_assignment", "raise")
  • The warning is about whether the assignment lands, not about performance: a slice may be a view or a copy, and pandas cannot promise which.
  • The fix is structural - select and assign in one .loc call, or copy deliberately.
  • Turning the warning into an error in development ("raise") catches the pattern before it silently corrupts a pipeline.
  • inplace=True does not make this safe; it hides the same question behind a different syntax.

Silent dtype coercion

df["n"].dtype                      # int64
df.loc[df["n"] > 0, "n"] = None    # a gap appears
df["n"].dtype                      # float64 - 1234 became 1234.0

df["amount"] = pd.to_numeric(df["amount"], errors="coerce")   # bad text -> NaN
df["id"]     = df["id"].astype("string")                      # keep leading zeros

df["score"].sum()                  # NaN is skipped, so a total can look fine
df["score"].count(), len(df)       # compare: the difference is the missing rows
df["score"].mean(skipna=False)     # NaN if any value is missing - the honest read
SymptomLikely cause
Numbers ending in .0An integer column was promoted to float by a missing value
Leading zeros lost in an idThe column was parsed as a number instead of text
Totals that look too highNaN rows skipped silently by sum
object dtype on a numeric columnCommas, currency symbols or stray text in the source
Comparison that never matchesString "1" compared with integer 1

Index and alignment bugs

s = pd.Series([100, 200], index=["x", "y"])
df["bonus"] = s                    # aligned by label: "x" and "y" match, everything else NaN
df["bonus"] = s.to_numpy()         # positional: length must match exactly

df = df.reset_index(drop=True)     # when the labels carry no meaning

# duplicate labels break reindex and make .loc return a frame, not a row
df.index.is_unique                 # False?
df = df[~df.index.duplicated(keep="first")]

pd.testing.assert_frame_equal(expected, actual, check_dtype=False)
pd.testing.assert_series_equal(s1, s2, rtol=1e-6, check_names=False)
⚠️
A wrong answer that looks plausible is worse than an exception. After every merge, pivot or reindex, assert the shape and the null counts - assertframe.shape == (expected_rows, expected_cols) - and use pd.testing to pin a known-good result as a regression test.

FAQ

How do I silence SettingWithCopyWarning?
Do not. Change the code: assign through .loc in a single call, or take an explicit .copy() when you intend to modify a subset. The warning disappears as a consequence of the fix.
Why do two frames that look identical fail assert_frame_equal?
Usually the dtype or the index. Inspect .dtypes and .index on both; check_dtype=False isolates whether the difference is only in types.

The index in depth: MultiIndex, set_index and reindex String operations with .str

Last refreshed 2026-09-18.