Robust XPath for scraping and testing

Paths built from structure break the day a designer adds a wrapper. Anchor selectors to content and attributes instead, and detect breakage early.

Why structural paths break

# A path generated by a browser devtools "Copy XPath"
/html/body/div[3]/div[2]/table/tbody/tr[4]/td[2]

# What the same thing looks like after one wrapper div is added
/div[1]/div[3]/div[2]/table/tbody/tr[4]/td[2]   # now broken

Every positional step is a promise that nothing will ever be inserted before it. Live pages break that promise constantly. A tbody element, for example, is inserted by the browser's HTML parser but not present in the source, so a path copied from devtools will not run against the raw HTML you fetch.

Prefer a stable attribute: //*[@data-testid="price"] or //input[@name="email"].
Fall back to text: //label[normalize-space(.)="Total"]/following::span[1].
Use semantic containers: //main//article//h2 survives layout changes better than index chains.
Combine conditions so one change does not shift everything: //button[@type="submit" and contains(@class,"primary")].

Writing selectors that fail loudly

from lxml import html

PAGE = """
<main>
  <form>
    <label for="em">Email</label><input id="em" name="email" type="email">
    <button type="submit" class="btn primary">Continue</button>
  </form>
</main>
"""

def field(doc, name):
    nodes = doc.xpath("//input[@name=$n]", n=name)
    if not nodes:
        raise LookupError("input named " + name + " not found")
    return nodes[0]

doc = html.fromstring(PAGE)
el = field(doc, "email")
print(el.get("type"))       # email

Symptom	Likely cause	Fix
Selector matched 0 nodes after a release	Wrapper added	Anchor to an attribute or text
Matched 3 nodes instead of 1	Template repeated for mobile and desktop	Add a visibility or container condition
Text includes unexpected spacing	Whitespace in the markup	`normalize-space()`
Works in devtools, fails in code	devtools sees parsed DOM, code sees source	Use the same parser in both
Values swapped between fields	Positional order changed	Select by name or label, never by index

# A contract test: assert the shape you depend on, per fixture, in CI
import glob, lxml.html

REQUIRED = {
    "price": '//*[@data-testid="price"]',
    "title": '//h1[@itemprop="name"]',
}

for path in glob.glob("fixtures/*.html"):
    doc = lxml.html.parse(path).getroot()
    for name, expr in REQUIRED.items():
        assert doc.xpath(expr), path + " lost " + name
print("selectors ok")

⚠️

Do not test that a selector returns a specific string from a live site — pages change copy for reasons that have nothing to do with you. Assert that the element exists and that its text is non-empty, then assert content rules on fixture files you control.

FAQ

Should I just let devtools generate the XPath?

Use it as a starting point, then replace positional steps with attribute or text anchors. A generated path is a snapshot of one version of the page, not a selector.

How many fallback selectors should one field have?

Two at most, and log which one matched. A long fallback chain hides the fact that the primary selector has been broken for months.

XPath in the browser: document.evaluate Selecting attributes, text and special nodes

Last refreshed 2026-09-18.

Robust XPath for scraping and testing

Why structural paths break

Writing selectors that fail loudly

FAQ

Related