Main area
Agency
technisches-seo

robots.txt vs. noindex vs. Canonical — Three Signals, Three Phases, One Common Bug

Redaktion ·

robots.txt vs. noindex vs. Canonical — Three Signals, Three Phases, One Common Bug

Three tools that get constantly confused: robots.txt, noindex and rel=canonical. At first glance they all do “something with visibility in Google”. In reality they act in three different phases — crawling, indexing, consolidating — and confusing them builds setups that do the exact opposite of what you wanted. The best-known mistake regularly costs visibility because two signals block each other. This article sorts out which signal acts when, and gives you a decision tree for everyday use.

Three phases, three signals

Google processes a URL in a fixed order: first it is crawled (the content fetched), then indexed (added to the searchable index), and duplicates are consolidated (several similar URLs merged onto one preferred URL). Each of the three signals sits in exactly one of these phases:

robots.txt → controls crawling. A Disallow tells the crawler: do not fetch this URL. It is a door in front of the server, not a command to the index.

noindex → controls indexing. A noindex (as a meta tag or X-Robots header) says: this page may be crawled but not put in the index. It acts one phase later than robots.txt.

rel=canonical → controls consolidation. A canonical says: this page is a variant, the actual preferred URL is over there. It removes nothing from the index, it only bundles duplicates.

Internalise this and you avoid most mistakes outright. The question is never “how do I hide a page” but “which phase do I want to intervene in”.

robots.txt controls crawling — not indexing

The most important and most misunderstood point: robots.txt is not a mechanism for keeping a page out of Google. Google puts it plainly: robots.txt “is not a mechanism for keeping a web page out of Google” (source: Google Search Central, accessed 2026-06-06).

The reason is mechanical. A URL blocked by Disallow is not crawled — but it can still end up in the index if other pages link to it. The result is the characteristic search entry without a description: Google shows the URL (and possibly the anchor text of the links) but no snippet text, because it never saw the content (source: Google Search Central, accessed 2026-06-06).

So robots.txt is a crawl lever, useful against crawl-budget waste on masses of unimportant URLs — but the wrong tool when the goal is “not in the index”.

noindex controls indexing — but must be crawlable

When a page should reliably not appear in the index, noindex is the right signal. Two ways:

  • Meta tag in the <head>: <meta name="robots" content="noindex"> — for HTML pages.
  • X-Robots-Tag as an HTTP header: X-Robots-Tag: noindex — also for non-HTML like PDFs or images (see X-Robots-Tag).

The condition is decisive: for noindex to work, the page must not be blocked by robots.txt and must be accessible to the crawler. Google is unambiguous here: if the page is blocked by robots.txt or otherwise inaccessible, “the crawler will never see the noindex rule, and the page can still appear in search results” (source: Google Search Central, accessed 2026-06-06).

That is the core of the whole topic — and the next section shows why it goes wrong so often.

The classic bug: Disallow plus noindex

The logic seems tempting: “I want to keep the page safely out — so I block it in robots.txt and set noindex.” That is exactly what does not work.

If robots.txt forbids crawling, Google never fetches the page. So Google never reads the noindex tag in the HTML either — it sits inside the page that is not fetched. The noindex is invisible to Google. Result: the page can keep showing in the index (e.g. via incoming links), and the one signal that would reliably remove it never takes effect.

The correct order for a page that is already indexed and should come out: first set noindex, remove the robots.txt disallow for that URL (so Google sees the tag), wait until the page falls out of the index — and only then optionally block it again if you want to save crawl budget.

Decision tree — what do I want to achieve?

Start from the goal, not the tool:

  • “This URL should not even be crawled” (e.g. endless filter/sort parameters, internal search). → robots.txt Disallow. Accept that the URL can in theory appear in the index without a snippet if it is linked.
  • “This page should not appear in the index / should come out” (e.g. thank-you pages, test pages, thin tag archives). → noindex, keep the page crawlable. No Disallow.
  • “There are several URL variants of the same content, one should be authoritative” (e.g. tracking parameters, print view, sort variants). → rel=canonical to the preferred URL. All variants stay crawlable, the link value is bundled.
  • “Fully private, no one but the authorised” (e.g. customer areas). → password protection, not robots.txt — that is the only hard block.

Common conflicts and how to resolve them

Disallow + noindex at the same time. The classic described above. Fix: remove the Disallow for that URL so the noindex can be read.

Canonical to a URL blocked by robots.txt. Google cannot crawl the target URL and cannot verify the canonical signal — the consolidation is ignored. Fix: always keep the canonical target crawlable.

Canonical and noindex on the same page. Contradictory: canonical says “bundle me with that page” (i.e. treat us as belonging together in the index), noindex says “take me out”. Google gets mixed signals. Fix: decide — either consolidate (canonical) or remove (noindex), not both.

FAQ

Does robots.txt keep a page out of the Google index?

No. robots.txt only controls crawling. A URL blocked by Disallow can still be indexed if it is linked — it then appears without a description in the results. To keep a page reliably out of the index you need noindex (and the page must be crawlable) or password protection.

Why isn’t my noindex working?

Most likely because the page is simultaneously blocked by robots.txt. Then Google cannot fetch the page, never reads the noindex tag, and consequently does not remove the page. Fix: lift the Disallow for that URL so Google sees the noindex — then the page falls out of the index.

What is the difference between noindex and rel=canonical?

noindex removes a page from the index. rel=canonical removes nothing; it bundles several URL variants of the same content onto one preferred URL and consolidates their signals. Use noindex against pages that should not appear at all; canonical against duplicates where one variant should appear.

Can I set canonical and noindex together?

Better not — the signals contradict each other. Canonical treats the page as part of a cluster, noindex wants to remove it. Google receives mixed instructions and decides itself, which is not reliable. Decide per page: consolidate or remove.

In what order do I switch signals when an indexed page should come out?

First set noindex and make sure the page is not blocked by robots.txt. Then wait until Google re-crawls the page, reads the noindex and removes it from the index. Only after that can you optionally block it in robots.txt to save crawl budget — never before.

Themenuebersicht

Technisches SEOCrawling & Indexierung