Main area
Agency
technisches-seo

Pagination and Faceted Navigation — solving crawl traps cleanly

Redaktion ·

Faceted navigation — the filter and facet navigation in shops and listings — is by far the biggest crawl trap a technical SEO needs to know. A single category tree with filters for colour, size, brand, price and sorting can generate millions of crawlable URL combinations, all showing almost the same content. If you don’t tame this, you burn crawl budget, bloat the index and produce duplicate content at scale. This article shows how the trap forms — and which patterns defuse it cleanly.

Why facets generate URLs exponentially

The problem is combinatorics. With five filters of four options each, you already get 4⁵ = 1024 combinations — per category. Add sorting parameters, pagination and multi-select and the number explodes. Each of these URLs looks new to the crawler, and it can only decide after fetching whether it’s useful. That’s Google’s own description of the problem: crawlers typically access a very large number of faceted URLs before their processes determine the URLs are useless.

Three harms arise in parallel:

  • Crawl budget trap — Googlebot wastes its fetch capacity on filter variants instead of discovering genuinely new content. The result: new products and articles get indexed more slowly.
  • Index bloat — thin, near-identical combination pages land in the index and dilute the signals of the actually strong category pages.
  • Duplicate content — dozens of URLs with near-identical content compete for the same rankings and fragment relevance.

The core decision: which facets should rank?

Before thinking about technology, make a content decision: for which filter combinations is there real search demand? “Men’s running shoes” or “red sneakers” have search volume — such combinations deserve an indexable, optimised landing page. “Running shoes size 43 colour blue price 80–90 € sorted by popularity” has none and belongs consistently out of the index.

The rule of thumb: a few deliberately chosen facets indexable, all others blocked. You maintain this list editorially, not by gut feel — ideally based on keyword research. Anything not on the list gets tamed with the patterns below.

The solution patterns in detail

robots.txt disallow for filter parameters

The strongest lever when indexing isn’t wanted at all: block the filter parameters via robots.txt disallow so the crawler never fetches them. Example: Disallow: /*?*color= blocks all URLs with a color parameter while keeping the base category crawlable. This is the most effective method against overcrawling because it protects the budget before it’s spent. Important: a URL blocked by robots.txt can’t be crawled by Google — so it can’t see its noindex either. The two means are mutually exclusive.

noindex for thin combination pages

If the pages should stay crawlable (e.g. so product links remain reachable) but not land in the index, set noindex on the thin combinations. The crawler may visit the page but doesn’t add it to the index. This costs more crawl budget than a disallow but keeps the internal link flow intact.

Canonical to the base category

For variants that show the same content under parameters (sorting, view toggles), the canonical tag points to the clean base category without parameters. Google consolidates the signals there. Note: canonical is a hint, not a command — Google may ignore it if the pages are too different. It works well for true duplicates, less reliably for merely similar pages. Google itself calls canonical and nofollow less effective long-term than a robots.txt disallow.

JavaScript filters without URL change or URL fragments

The cleanest trick: filters that swap the content via JavaScript without creating a new crawlable URL — or that store the state in a URL fragment (#color=red). Fragments are ignored by the crawler and have neither a positive nor negative effect on crawling. That way no separate URL even exists for non-indexable filters that could cost budget.

Pagination today: rel=next/prev is history

For a long time, rel="next" and rel="prev" were considered mandatory best practice for paginated series. That’s over: in 2019 Google publicly confirmed it no longer uses these tags for indexing — and hadn’t for years before. Today the rules are:

  • Every paginated page is self-canonical. Page 2 canonicals to itself, not to page 1. Canonicalising all pages to page 1 hides the products of the following pages from Google.
  • Real, clickable links between the pages. Google follows visible anchor links, not bare rel attributes. Pagination that only works via JavaScript without crawlable links can mean deep pages are never discovered.
  • Don’t remove rel=next/prev if already present. Bing still uses the tags and they help accessibility. They do no harm — Google simply ignores them.

FAQ

FAQ

Does Google still use rel=next and rel=prev?
No. Google confirmed in 2019 it no longer uses the tags for indexing — and hadn't for years before. Bing still uses them and they help accessibility, so keeping existing tags does no harm. But don't rely on them for Google.
Should I canonicalise paginated pages to page 1?
No. Every paginated page gets a self-referencing canonical to itself. Pointing them all to page 1 hides the products of the following pages from Google and risks deep content not being indexed.
robots.txt disallow or noindex for filter pages?
Disallow if the pages should be neither crawled nor indexed — that protects crawl budget most. noindex if they must stay crawlable (e.g. for internal links) but should not be indexed. Both together don't work: a page blocked via robots.txt can't have its noindex read by Google.
Which facets should be indexable?
Only those with real search demand, ideally determined by keyword research — like brand or a core attribute. Deep combinations of four or five filters almost never have search volume and belong consistently out of the index. A few deliberately chosen facets indexable, the rest blocked.
How do I prevent duplicates from different parameter order?
Keep the logical order of the filters in the URL always the same and prevent duplicate filters, so the same selection doesn't produce several URL variants. Use the standard separator and return a 404 for empty filter combinations instead of a thin empty page.

Conclusion

Faceted navigation isn’t harmful per se — poorly managed faceted navigation is. The key lies in a deliberate decision: make a few facets with real search demand indexable and tame all other combinations with the right means — robots.txt disallow against overcrawling, noindex for crawlable-but-not-indexable pages, canonical for true duplicates, JavaScript or URL fragments to avoid creating a URL in the first place. For pagination: self-canonical per page, real clickable links, and rel=next/prev as a nice extra with no Google effect. Set this up cleanly and you protect your crawl budget and keep the index lean.

How crawl budget works overall and how to steer it is deepened in the sister article on crawl budget.

Themenuebersicht

Technisches SEOCrawling & Indexierung