Selecting attributes, text and special nodes
Attributes, text nodes, comments and processing instructions each behave differently in a predicate, and whitespace makes text nodes trickier than they look.
Attributes
# Select an attribute value
xmllint --xpath '//a/@href' page.html
# Filter elements by an attribute
xmllint --xpath '//a[@rel="nofollow"]' page.html
xmllint --xpath '//a[starts-with(@href, "https://")]' page.html
# Existence, not value
xmllint --xpath '//img[@alt]' page.html
xmllint --xpath '//img[not(@alt)]' page.html # the accessibility bug hunt
# Attributes are not children: this selects nothing
xmllint --xpath '//a/@href/child::node()' page.html- An attribute node is a leaf. It has no children, no parent in the child axis, and no string-value beyond its own text.
@idinside a predicate tests existence;@id="x"tests the value.- Attribute selection with
@*on//*is a common but expensive way to fingerprint a page. - In XPath, attribute names are case sensitive, like all XML names.
Text, comments and processing instructions
| Goal | Expression | Catch |
|---|---|---|
| Text of an element | //h1/text() | Returns a node-set; use string() for one string |
| All text under an element | //h1//text() | Includes nested spans, so markup boundaries vanish |
| Trimmed text | normalize-space(//h1) | Collapses internal runs of whitespace too |
| Match by text | //h2[text()="Overview"] | Fails if the heading contains a nested element |
| Match by loose text | //h2[contains(., "Overview")] | Matches descendants as well |
| Comments | //comment() | Also grabs conditional-comment payloads |
| Processing instructions | //processing-instruction("xml-stylesheet") | Target name is the argument |
from lxml import etree, html
doc = html.fromstring("<div><h2>Over<span>view</span></h2></div>")
# text() sees the heading and its span as separate text nodes
print(doc.xpath("//h2/text()")) # ['Over']
print(doc.xpath("//h2//text()")) # ['Over', 'view']
print(doc.xpath("string(//h2)")) # 'Overview'
print(doc.xpath("//h2[text() = 'Overview']")) # [] -- the trap
print(doc.xpath("//h2[normalize-space(.) = 'Overview']")) # [<Element h2>]The example above is the single most common scraping bug in XPath: text()="Overview" only compares the first text node of the element if the predicate is written that way, and any nested markup splits the string. Comparing against . uses the element's full string-value instead.
Whitespace-only text nodes
# Indented markup produces text nodes that look empty
xmllint --xpath 'count(//ul/li/text())' page.html
# Inspect what is actually there
xmllint --xpath '//ul/li/text()' page.html
# XML parsers can drop them for you
xmllint --noblanks --xpath 'count(//ul/li/text())' page.html⚠️
Whitespace-only text nodes are why
[position()=2] behaves differently on HTML parsed from a pretty-printed file than on the same markup minified. Do not build index-based paths against formatted markup — select by position within the element children using li[2], not node()[2].FAQ
Should I use text() or a dot in predicates?
Use a dot when the element may contain nested markup, because it compares the full string-value. Use text() only when you really mean a single direct text node.
Why does selecting @class return a class with several words?
class is a single attribute whose value may contain several space-separated tokens. Match a token with contains(concat(' ', normalize-space(@class), ' '), ' active ') if needed.
Related
The XPath data model: nodes, order and values Robust XPath for scraping and testing
Last refreshed 2026-09-18.