HowToCodePage

How search engines crawl and index

The three-stage pipeline - crawl, index, serve - and the controls that decide whether a page enters it at all.

Crawling: how URLs are discovered

A search engine keeps a queue of URLs waiting to be fetched. It learns about them from links on pages it already knows, from sitemaps you submit, and from redirects. The crawler fetches a URL, reads the response, and follows the links it finds. Nothing else exists as a discovery mechanism - a page with no links and no sitemap entry is effectively invisible.

# robots.txt - must be served at the root of the host
User-agent: *
Disallow: /admin/
Disallow: /search?
Allow: /

Sitemap: https://example.com/sitemap.xml
  • robots.txt controls crawling, not indexing: a disallowed URL can still appear in results when other sites link to it.
  • Blocking a page in robots.txt also stops the crawler from ever reading a noindex tag on that page, so the tag has no effect.
  • Spend crawl budget where it matters: block faceted search URLs, internal result pages and staging hosts so real content is fetched first.
  • A sitemap is a hint about URLs and priority, never a guarantee of indexing.
💡
If you want a page out of the index, use noindex (and let the crawler reach it). If you want to save server load, use robots.txt. Choose one deliberately - combining both achieves neither.

Indexing: what actually gets stored

After fetching, the engine renders the page much like a browser does, extracts the visible text plus the metadata in the head, and writes it into a huge inverted index. Client-side JavaScript is rendered too, but it is queued separately and can be delayed - content that exists only after a script runs is indexed later and less reliably than content in the HTML.

<head>
  <title>Wireless Headphones Guide - Buyer Checklist</title>
  <meta name="description" content="How to choose wireless headphones: battery, codecs, comfort and noise cancelling.">
  <link rel="canonical" href="https://example.com/guides/wireless-headphones">
  <meta name="robots" content="index, follow">
</head>
Index statusWhat it usually means
IndexedEligible to appear in results (not a ranking guarantee)
Excluded by noindexDeliberate removal - the tag is respected
Discovered - currently not indexedQueued but never crawled; often low-value or duplicate content
Crawled - currently not indexedFetched and judged not worth storing; usually thin or duplicate
Duplicate, Google chose a different canonicalTwo URLs say the same thing and the signals were not consolidated
Blocked by robots.txtNever fetched, so never evaluated

Serving: how a result is chosen

Ranking happens per query, not per page. The same URL can rank first for one phrasing and be absent for another, and results are personalised by language, location, device and history. There is no single knob that lifts a site; there are signals that consistently correlate with visibility.

  • Topical match: does the page answer the query, at the right depth and in the right format?
  • Links: how many other sites point at it, and how trusted those sites are.
  • Freshness: matters a lot for news and prices, barely at all for definitions.
  • Page experience: mobile usability, intrusive interstitials and Core Web Vitals.
  • Clarity: one clear topic per URL, and a title that reads like an answer.

Practical consequence: write for one intent per page and let each page be the obvious answer to that intent. Pages that try to cover five different intents usually rank for none of them.

FAQ

How long until a new page is indexed?
Hours for a well-linked site with a submitted sitemap, weeks for a new domain. You can nudge it with internal links from existing pages and a manual URL inspection, but you cannot force it.
Do I need a sitemap if my site is small?
It rarely hurts and often helps. For a handful of pages, internal links are enough, but a sitemap makes new URLs discoverable immediately and gives you a checklist to verify in Search Console.

On-page essentials

Last refreshed 2026-09-18.