HowToCodePage

Aggregating feeds: OPML, merging and republishing

Aggregation is mostly bookkeeping: keeping track of which items you have seen, in what order to present them, and what you are allowed to republish.

OPML as a subscription list

<?xml version="1.0" encoding="UTF-8"?>
<opml version="2.0">
  <head>
    <title>Engineering blogs</title>
    <dateCreated>Thu, 18 Sep 2026 09:00:00 GMT</dateCreated>
  </head>
  <body>
    <outline text="Databases" title="Databases">
      <outline type="rss" text="Release Notes"
               title="Release Notes" xmlUrl="https://example.com/feed.xml"
               htmlUrl="https://example.com/releases"/>
      <outline type="rss" text="Query Weekly"
               title="Query Weekly" xmlUrl="https://db.example.org/rss"
               htmlUrl="https://db.example.org/"/>
    </outline>
  </body>
</opml>
  • xmlUrl is the feed and htmlUrl is the site. Confusing them is the single most common OPML import failure.
  • Nested outlines are categories, and flat lists are also valid. Import tools handle nesting to varying degrees, so keep depth shallow.
  • text is the human-readable label; title is the formal one. Set both to the same value for maximum compatibility.
  • OPML is also the export format for most reader applications, so supporting import and export makes migration between tools possible.
import lxml.etree as etree

def subscriptions(path: str) -> list[dict]:
    root = etree.parse(path).getroot()
    out = []
    for o in root.iter("outline"):
        url = o.get("xmlUrl")
        if not url:
            continue                     # a category, not a feed
        out.append({
            "title": o.get("title") or o.get("text") or url,
            "feed_url": url,
            "site_url": o.get("htmlUrl"),
            "category": o.getparent().get("text") if o.getparent() is not None else None,
        })
    return out

Merging without duplicates

from dataclasses import dataclass
from datetime import datetime
from urllib.parse import urlsplit, urlunsplit

def canonical(url: str) -> str:
    """Normalise for comparison, but never for display or attribution."""
    u = urlsplit(url)
    return urlunsplit((u.scheme.lower(), u.netloc.lower(),
                       u.path.rstrip("/"), "", ""))

@dataclass
class Entry:
    key: str
    source: str
    url: str
    title: str
    published: datetime | None
    html: str

def merge(entries: list[Entry]) -> list[Entry]:
    seen: set[str] = set()
    unique: list[Entry] = []

    for e in entries:
        # Two independent keys: the declared id, and the canonical link.
        # Aggregators rewrite ids, so the link is the more durable identity.
        keys = [e.key, canonical(e.url)]
        if any(k in seen for k in keys):
            continue
        seen.update(keys)
        unique.append(e)

    # Undated items sort last rather than being dropped or guessed at
    unique.sort(key=lambda e: (e.published is None,
                               e.published or datetime.min.replace(tzinfo=None)),
                reverse=False)
    return unique
  1. Collect items from every source with a per-source timeout, and record the failure of one source rather than abandoning the whole refresh.
  2. Normalise identifiers, then deduplicate on both the declared id and the canonical link.
  3. Sort by publication date descending, placing undated items at the end.
  4. Apply a per-source rate limit, at most one item in N, so a prolific blog does not bury everything else.
  5. Store what you have seen so the next refresh does not re-notify or re-republish.
  6. Record attribution with every stored item: source title, source URL and the original link.
⚠️
Republishing someone else's full article without permission is a copyright problem regardless of technical feasibility, and many publishers specifically forbid it in their feed terms. Link out, quote a short excerpt, and store the licence terms you found rather than assuming the feed's existence is consent.

FAQ

Should I deduplicate on the guid or the link?
Both. Aggregators sometimes rewrite guids, and a site sometimes republishes the same article at a new URL, so each key catches a different class of duplicate.
How often should I poll?
Respect the feed's own hints in sy:updatePeriod and sy:updateFrequency when present, otherwise poll at most hourly per source, and honour 429 and Retry-After without exception.

Parsing and consuming feeds in code Feeds at scale: pagination, archives and pipelines

Last refreshed 2026-09-18.