Term
X-Robots-Tag
HTTP header that sets robots directives (e.g. noindex, nofollow) at the server level — also works for non-HTML resources like PDFs, images, or JSON responses.
X-Robots-Tag — explained in more detail
The X-Robots-Tag HTTP header is the server-side equivalent of <meta name="robots">. Example: X-Robots-Tag: noindex, nofollow. It can be sent per file, per URL pattern, or globally — wherever <meta> tags don’t apply because the resource isn’t HTML.
Typical use cases: removing PDF files from the index, marking generated images as “noindex”, controlling entire directories via web-server configuration. The header supports all standard directives, including noimageindex, unavailable_after: <DATE>, noarchive, nosnippet.
Example / practical use
Apache: Header set X-Robots-Tag "noindex, nofollow" for <FilesMatch "\.pdf$">. Nginx: add_header X-Robots-Tag "noindex" always; in the matching location block. The directive then lands in every response matching that pattern — without touching the HTML.
Distinction from related terms
X-Robots-Tag and meta robots are functionally equivalent, only the transport differs (HTTP header vs. HTML meta). robots.txt, by contrast, blocks crawling, not indexing — to remove an already indexed URL from the index you need noindex via meta or header, and you must not block it in robots.txt at the same time (or the bot will never see the directive).
Entdecke mehr
301 Redirect
A 301 redirect is an HTTP status code permanently redirecting one URL to another — the standard for URL changes, domain moves and site migrations.
LexikonTechnical SEO — what Google actually has to crawl, render and index
How crawl budget, robots, sitemap, JS rendering, indexing, canonical and Core Web Vitals fit together — the full arc for production sites.
NewsGoogle Search Central Live Toronto 2026 — what Google officially said about the future of search
Information Gain as the new guideline, Google-Extended clarified, AI Mode with 93% zero-click. The key statements from Toronto.