LEARN

Decide Retrieval and Training Crawlers Separately

Two independently activatable perforated plates directing particles to a glowing display or dense core, symbolizing separate decisions for retrieval and training crawlers.
Retrieval crawlers determine citeability, training crawlers determine content entry into model training—these two should be decided separately.

IN ONE SENTENCE

Retrieval crawlers decide whether you can be cited; training crawlers decide whether your content trains the model — decide them separately rather than with one blanket rule.

Retrieval crawlers fetch pages to answer a user's current question and determine whether you can be cited. Training crawlers collect content for model training and have no direct bearing on citation. Decide them separately.

OUR POSITION

'Don't train on me, but do cite me' is coherent in principle, but see the cost: blocking training crawlers does nothing for citation and only forfeits long-term corpus presence. For brands optimising exposure, allowing both is usually the better trade. **Blocking retrieval while allowing training is the worst combination** — contributing corpus while switching off exposure.

01

Which identifiers fall where

Retrieval (must be allowed for citation): OpenAI's OAI-SearchBot and ChatGPT-User, Anthropic's Claude-SearchBot and Claude-User, Perplexity's PerplexityBot and Perplexity-User, and Googlebot on Google's side.

Training (a policy choice): GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Bytespider, CCBot and others.

⚠️ Some platforms publish no dedicated identifier — DeepSeek among them. No identifier does not mean no crawling: look in server logs rather than assuming absence because no matching agent appears in robots.txt.

02

What each vendor has stated

OpenAI: blocking OAI-SearchBot removes pages from citation candidacy.

Anthropic: blocking Claude-User may reduce visibility for user-directed web search.

Perplexity: PerplexityBot indexes similarly to other search engines, and content is not used for model pre-training.

Google: Google-Extended controls only training and grounding and is explicitly not a ranking signal; blocking Googlebot exits both Search and Gemini's retrievable set.

03

One more step after deciding

Editing robots.txt is only the first step. A CDN or edge layer can block outside your site config, and Cloudflare has blocked AI crawlers by default for new domains since 1 July 2025 with a rule that does not separate retrieval from training.

So verify after allowing: check server logs for the requests, confirm the status codes, and confirm the response body contains the content. An unverified allow is not an allow.

Data behind this page

97%

Share of published llms.txt files with zero requests in May 2026

Source：Ahrefs, across 137,000 domains，2026

Sources

  1. [1]Optimizing your website for generative AI features on Google Search．Google Search Central．2026-05-15

Updated 2026-08-10