Regular expressions and text processing

re syntax, groups, greedy versus lazy quantifiers, findall against finditer, compiled patterns, substitution, and parsing logs.

The syntax you actually use

A regular expression describes a shape of text. Always write patterns as raw strings (r"...") so backslashes reach the engine instead of being eaten by Python's string escapes.

import re

re.search(r"\d{4}-\d{2}-\d{2}", text)        # first match anywhere
re.match(r"^ERROR", line)                    # anchored at position 0 only
re.fullmatch(r"[A-Z]{2}\d{4}", code)         # the whole string must match

# a greedy quantifier takes as much as it can
re.search(r"<(.+)>", "<a><b>").group(1)      # 'a><b'
# a lazy one takes as little as possible
re.search(r"<(.+?)>", "<a><b>").group(1)     # 'a'

# groups, optional groups and named groups
m = re.search(r"(\w+)@(\w+)\.(\w+)", "mail [email protected]")
m.group(0), m.group(1), m.groups()           # ('[email protected]', 'ada', ('ada', 'example', 'com'))

m = re.search(r"(?P<user>\w+)@(?P<domain>\w+)", "[email protected]")
m.group("domain")                            # 'example'
m.groupdict()                                # {'user': 'ada', 'domain': 'example'}

re.split(r"\s*,\s*", "a, b ,c")             # ['a', 'b', 'c']

Searching and extracting

The function you choose changes the output shape: findall returns strings, finditer returns match objects you can inspect.

text = "id=41 status=ok id=42 status=fail"

re.findall(r"id=(\d+)", text)            # ['41', '42'] - strings only

for m in re.finditer(r"id=(?P<id>\d+) status=(?P<status>\w+)", text):
    print(m.group("id"), m.group("status"), m.span())
# 41 ok (0, 15)
# 42 fail (16, 33)

# one capture group returns strings, several return tuples
re.findall(r"(\w+)=(\w+)", text)        # [('id', '41'), ('status', 'ok'), ...]

# compile once when a pattern is used in a loop
EMAIL = re.compile(r"[\w.+-]+@[\w-]+\.[\w.]+", re.IGNORECASE)
EMAIL.findall("[email protected], [email protected]")
EMAIL.sub("<email>", "write to [email protected]")   # 'write to <email>'

# never build a pattern from raw user input
def contains(user_text, needle):
    return re.search(re.escape(needle), user_text) is not None

• findall for a quick list of strings, finditer when you need positions or named groups.
• re.escape any literal fragment you interpolate into a pattern.
• re.compile caches internally, but an explicit compiled pattern documents intent and saves a lookup per call.
• Use verbose mode (re.VERBOSE) to spread a long pattern over lines with comments.

Substitution and log parsing

The replacement argument to sub can be a string with back-references (\1) or a function that receives the match — the function form lets you transform values rather than just reshape text.

import re

re.sub(r"\s+", " ", "  a   b  ")            # ' a b ' - collapses runs
re.sub(r"(\w+), (\w+)", r"\2 \1", "Doe, Ada")   # 'Ada Doe' - swap via back-refs

def double(match):
    return str(int(match.group(0)) * 2)

re.sub(r"\d+", double, "cost 7 and 3")        # 'cost 14 and 6'

LOG = re.compile(
    r"(?P<ip>\d{1,3}(?:\.\d{1,3}){3}) - - "
    r"\[(?P<ts>[^\]]+)\] "
    r'"(?P<method>[A-Z]+) (?P<path>\S+) HTTP/[\d.]+" '
    r"(?P<status>\d{3}) (?P<size>\d+|-)"
)

SLOW = {"", "-"}

def parse_line(line):
    m = LOG.search(line)
    if m is None:
        return None                    # always handle the no-match case
    d = m.groupdict()
    d["size"] = int(d["size"]) if d["size"] not in SLOW else 0
    return d

# a compiled pattern is reusable and thread-safe
hits = [parse_line(l) for l in open("access.log", encoding="utf-8")]
errors = [h for h in hits if h and h["status"].startswith("5")]
print(len(errors))

For structured formats the standard library often wins: json.loads for JSON, csv for comma-separated data, fromisoformat for timestamps. Reach for a regular expression when the text is genuinely irregular or when you only need to pull a few fields out of a large line.

FAQ

Related

Command-line tools: argparse and logging Testing with pytest

Last refreshed 2026-09-18.