Feeds at scale: pagination, archives and pipelines

A feed with a hundred thousand items is useless to a reader. Paged and archive feeds, generated incrementally, are what keep large catalogues usable.

Splitting a large catalogue

<!-- Paged feeds: the specification is RFC 5005 -->
<channel>
  <title>Release Notes, page 4</title>
  <atom:link rel="self"   href="https://example.com/feed.xml?page=4" type="application/rss+xml"/>
  <atom:link rel="first"  href="https://example.com/feed.xml?page=1" type="application/rss+xml"/>
  <atom:link rel="prev"   href="https://example.com/feed.xml?page=3" type="application/rss+xml"/>
  <atom:link rel="next"   href="https://example.com/feed.xml?page=5" type="application/rss+xml"/>
  <atom:link rel="last"   href="https://example.com/feed.xml?page=17" type="application/rss+xml"/>
</channel>

<!-- An archive feed is marked so readers do not treat the history as new -->
<channel>
  <title>Release Notes archive</title>
  <atom:link rel="current" href="https://example.com/feed.xml" type="application/rss+xml"/>
</channel>

• Full-content feeds should be complete: RFC 5005 requires that an item appears in full in exactly one feed, not truncated across pages.
• Mark an archive with rel="current" pointing at the live feed, so a reader does not treat the back catalogue as a flood of new posts.
• Advertise the recent feed from the page head; paged and archive feeds are discovered through the links, not from the site.

Generating and serving incrementally

from datetime import datetime, timezone
from email.utils import format_datetime

def render_feed(items, etag_store, path) -> tuple[str, str]:
    """Write the feed only when the content actually changed, and report an ETag."""
    newest = items[0].updated if items else None
    version = str(len(items)) + ":" + (newest.isoformat() if newest else "empty")

    cached = etag_store.get(path)
    if cached and cached["version"] == version:
        return cached["etag"], "not-modified"

    xml_parts = ['<?xml version="1.0" encoding="utf-8"?>',
                 '<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">',
                 "<channel>",
                 "<title>Release Notes</title>",
                 "<link>https://example.com/releases</link>",
                 "<description>Product changes, weekly.</description>",
                 "<lastBuildDate>" + format_datetime(newest or datetime.now(timezone.utc))
                 + "</lastBuildDate>",
                 "</channel></rss>"]
    xml = "\n".join(xml_parts)

    etag = '"' + version + '"'
    etag_store.set(path, {"version": version, "etag": etag})
    return etag, xml

GET /feed.xml HTTP/1.1
If-None-Match: "412:2026-09-18T09:00:00+00:00"

HTTP/1.1 304 Not Modified
ETag: "412:2026-09-18T09:00:00+00:00"
Cache-Control: public, max-age=300
Content-Type: application/rss+xml; charset=utf-8

• Serve feeds from a CDN with a short max-age, and generate them on write rather than on read for a large catalogue.
• Aggressive feed readers poll far more often than the content changes, so a 304 is the cheapest response you can send.
• Serve the correct content type: application/rss+xml for RSS, application/atom+xml for Atom. A generic text/xml works but is less informative.
• Compress the response, and set a cap on the number of items regardless of the requested page size.
• Two-tier generation works well: the recent feed rebuilt on every publish, and archive pages generated once and stored as static files.
• Alert when generation fails, because a silently stale feed looks identical to a blog that stopped publishing.

FAQ

Related

Full-text versus summary feeds Validating and troubleshooting feeds

Last refreshed 2026-09-18.