GLOSSARY
Retrieval Crawler
IN ONE SENTENCE
A retrieval crawler fetches pages to answer a user's current question, and determines whether content can be cited at all.
A retrieval crawler fetches pages to answer a user's current question, and determines whether content can be cited at all.
01
In detail
Decide it separately from the training crawler: citation requires allowing retrieval crawlers, and that is a switch-level prerequisite. Identifiers include OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, and on Google's side Googlebot.
Updated 2026-08-06