Term
robots.txt
robots.txt is a text file at a domain's root telling search-engine crawlers which URLs may be crawled and which must not.
robots.txt — explained in more detail
The robots.txt lives unconditionally at https://example.com/robots.txt and follows the Robots Exclusion Protocol. Per user agent (e.g. Googlebot, Bingbot or * for all), Allow: and Disallow: rules can be defined. Crawlers from reputable providers honour these directives; technical enforcement isn’t possible.
Important distinction: robots.txt only controls crawling, not indexing. A URL blocked via Disallow can still end up in the index if it’s linked from outside — Google then shows it without a snippet. To reliably keep something out of the index, use a noindex meta tag or HTTP header instead — but the page must remain crawlable, otherwise Googlebot won’t see the tag.
Example / practical context
User-agent: *
Disallow: /admin/
Disallow: /search?
Allow: /
Sitemap: https://example.com/sitemap.xmlThis configuration blocks the admin area and parameterised search-result pages, allows the rest, and links the sitemap.
Distinction from similar terms
robots.txt = crawl control. The noindex meta tag = index control. Canonical tag = duplicate control. Three different tools for three different problems — never mix them up.
Entdecke mehr
301 Redirect
A 301 redirect is an HTTP status code permanently redirecting one URL to another — the standard for URL changes, domain moves and site migrations.
LexikonTechnical SEO — what Google actually has to crawl, render and index
How crawl budget, robots, sitemap, JS rendering, indexing, canonical and Core Web Vitals fit together — the full arc for production sites.
NewsGoogle Search Central Live Toronto 2026 — what Google officially said about the future of search
Information Gain as the new guideline, Google-Extended clarified, AI Mode with 93% zero-click. The key statements from Toronto.