LLM Crawlers / AI Bots

LLM Crawlers / AI Bots — explained in more detail

LLM crawlers are a distinct category of web crawlers operating separately from classic search crawlers. Two main use cases: training (one-off or periodic content for model training, e.g. GPTBot from OpenAI, ClaudeBot from Anthropic, Google-Extended for Google models) and live retrieval (at answer time for chat or search responses, e.g. PerplexityBot, OAI-SearchBot, ChatGPT-User).

Control works the same way as for classic crawlers — via robots.txt with the corresponding User-Agent. Most major providers publicly respect robots.txt. To block AI training, list the User-Agents explicitly.

Example / In practice

robots.txt entry for full AI blocking:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Granular: block training, allow live retrieval (e.g. allow OAI-SearchBot and ChatGPT-User, but block GPTBot). The live-retrieval bots are the ones that can surface your site as a source in AI answers.

Distinction from similar terms

Googlebot ≠ Google-Extended: Googlebot crawls for classic Google Search and is separate from Google-Extended. Blocking Google-Extended does not remove you from the search index — but it does remove you from Gemini and Vertex AI training corpora.

Entdecke mehr

Glossar

301 Redirect

A 301 redirect is an HTTP status code permanently redirecting one URL to another — the standard for URL changes, domain moves and site migrations.

Lexikon

Technical SEO — what Google actually has to crawl, render and index

How crawl budget, robots, sitemap, JS rendering, indexing, canonical and Core Web Vitals fit together — the full arc for production sites.

News

Google Search Central Live Toronto 2026 — what Google officially said about the future of search

Information Gain as the new guideline, Google-Extended clarified, AI Mode with 93% zero-click. The key statements from Toronto.

Themenuebersicht

Technisches SEOCrawling & Indexierung