Main area
Agency
geo

Controlling AI Crawlers (GPTBot, ClaudeBot, PerplexityBot)

Redaktion ·

Controlling AI Crawlers (GPTBot, ClaudeBot, PerplexityBot)

Ever since ChatGPT, Claude, and Perplexity started pulling answers from the web, a whole set of new bots visits your site. And unlike the classic Googlebot, where the only question is “index or not,” here it’s about two very different things: is your content used to train an AI model? Or is it fetched live to make you visible in an AI answer?

That’s the central distinction in this article — and the reason “block all AI bots” is a worse idea than it sounds. This text is the bridge between technical SEO and GEO: you steer via robots.txt who gets in — but you need to know what you’re giving up.

Training crawlers vs. retrieval crawlers

The most important line runs not between providers but between two purposes:

Training crawlers collect publicly reachable content to train or improve AI models. This happens in the background, often months before any user query. Block them and you’re essentially saying: “my content should not go into future training data.” It has no direct effect on your visibility in live AI answers.

Retrieval crawlers fetch content at answer time — when a user is asking a question right now and the AI searches the web to answer with current information. Block these and you risk no longer appearing in exactly those AI answers. That’s the decisive point: blocking a retrieval bot can cost you visibility, often within hours.

Mixing these up and blocking everything wholesale may protect your training data — but cuts you off from AI search at the same time. The clean stance is a deliberate per-bot decision, not a blanket ban.

The key bots and their purpose (as of June 2026)

The three big providers now each run multiple bots with separate jobs:

OpenAI (ChatGPT) — three bots:

  • GPTBot — training. Collects content for model training. Respects robots.txt.
  • OAI-SearchBot — search. Indexes for the search features in ChatGPT. Respects robots.txt; exclusion means: no longer in ChatGPT search results.
  • ChatGPT-User — user-triggered fetching. Fetches a page when a user prompts it in ChatGPT. Per OpenAI, robots.txt rules do not necessarily apply here, since it’s a direct user action.

Anthropic (Claude) — three bots:

  • ClaudeBot — training. Respects robots.txt.
  • Claude-SearchBot — search. Builds the search results within Claude. Respects robots.txt.
  • Claude-User — user-triggered fetching for Claude. Also respects robots.txt per Anthropic. (Note: the older anthropic-ai token is obsolete — today it’s ClaudeBot and the three-bot framework.)

Google — Google-Extended. A pure control token: block it and your content stays out of Gemini training — but you remain fully indexed in Google Search. Normal crawling runs via Googlebot, separately.

Perplexity — PerplexityBot (indexing) and Perplexity-User (real-time fetch at answer time).

Common Crawl — CCBot. Not an AI provider in the narrow sense, but the Common Crawl dataset is a central training source for many AI companies.

ByteDance — Bytespider. Collects training data for ByteDance models.

Steering via robots.txt

Control runs through User-agent directives in robots.txt. One block per bot:

# Block training
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

# Allow search and retrieval (no entry = allowed)
# OAI-SearchBot, Claude-SearchBot, PerplexityBot permitted

This “surgical” pattern is the most recommended strategy (as of 2026): block training crawlers, allow search and retrieval bots. That keeps your content out of training pipelines while preserving your visibility in AI search.

The strategic trade-off

There’s no universally right answer — only a deliberate decision aligned with your goals:

  • Want maximum visibility in AI answers? Then at least allow all retrieval and search bots. Whether you permit training is a separate question.
  • Want to prevent your content from training models for free? Block the training crawlers — but let the retrieval bots through, or you vanish from the answers.
  • Have unique, proprietary content? Here blocking training crawlers can make sense, so you don’t become part of a model unasked.

The most common mistake is the blanket ban: a User-agent: * with Disallow: / for “all AI” — and with it the unintended severing of your own visibility in ChatGPT search, Claude, and Perplexity.

Not every bot honors robots.txt

A sober caveat to close on: robots.txt is a request, not a fence. Reputable providers like OpenAI, Anthropic, and Google honor their own directives. But:

  • Perplexity was observed by Cloudflare using undeclared crawlers with generic user-agent strings to fetch pages that had blocking directives in place (source: ALM Corp / Cloudflare research, as of 2026). For such cases robots.txt is not enough — you need blocking at the server or CDN level.
  • Bots like Bytespider have not reliably respected robots.txt in the past.

In practice: robots.txt is the first line of defense and entirely sufficient for the compliant providers. Anyone who must hard-protect content supplements it with server-side rules. For voluntarily steering visibility and training, it’s the right tool.

If you want to go a step further — actively providing content for AI systems rather than just regulating access — the starting point is llms.txt.

FAQ

Should I block all AI crawlers? Almost never wholesale. Blocking training crawlers is a legitimate decision if you don’t want your content training models. But blocking retrieval and search bots along with them costs you visibility in AI answers — which in a GEO context you usually want to keep.

What’s the difference between GPTBot and OAI-SearchBot? GPTBot collects content for training OpenAI’s models — that happens in advance, independent of individual queries. OAI-SearchBot indexes for the search features in ChatGPT, so it governs your visibility in ChatGPT search results. Both respect robots.txt, but blocking each has quite different consequences.

Do I disappear from ChatGPT if I block GPTBot? No, not directly. GPTBot is the training crawler. Your visibility in ChatGPT search is governed by OAI-SearchBot — and live fetches run via ChatGPT-User. Block only GPTBot and you protect your training data while remaining fundamentally findable in ChatGPT search.

Is robots.txt enough to reliably keep AI crawlers out? For the reputable, compliant providers (OpenAI, Anthropic, Google), yes. But robots.txt is a voluntary convention, not a technical fence. Individual actors have been observed circumventing it. Anyone who must hard-protect content supplements robots.txt with blocking at the server or CDN level.

What’s the point of blocking Google-Extended? Google-Extended controls only whether your content flows into Gemini training. Block it and you stay fully indexed in normal Google Search — handled by the separate Googlebot. So it’s a targeted opt-out of AI training without touching your search visibility.

Themenuebersicht

GEOAI-CrawlerGEO-Grundlagen