LEARN

CDN-Level Blocks Are Not in Your robots.txt

An open lattice with large gaps and a denser front sheet featuring smaller misaligned apertures controlling particle flow, representing how CDN/edge layers override robots.txt declarations
robots.txt is a declaration, CDN and edge layers are network-level switches; the latter takes effect when they conflict

IN ONE SENTENCE

robots.txt is a declaration and the CDN is a network-level switch; when they disagree, the network wins.

robots.txt is a declaration for well-behaved crawlers; the CDN and edge layer are the actual network-level switch. When they disagree, the network wins — an allow in robots.txt does not stop the edge from blocking.

OUR POSITION

Any conclusion that AI crawlers are allowed needs log evidence behind it. Declaring the job done after editing robots.txt is the most common false verification in this chain.

01

Known default policies

Cloudflare has blocked AI crawlers by default for new domains since 1 July 2025, and its 'Block AI bots' rule does not distinguish retrieval from training — meaning it also blocks the crawlers that create visibility.

Network-level reputation blocking also exists: after the August 2025 dispute between Cloudflare and Perplexity over stealth crawlers, requests in some zones may be blocked at the CDN even where the site allows PerplexityBot.

02

How to confirm whether you are affected

Read server logs: if requests from an agent never reach the origin, something in front of it is blocking.

Read the CDN's rules and logs: security rules, bot management and rate limiting are all candidates.

Only after checking both layers, allowing in both, and seeing 200s with complete response bodies in the logs is it confirmed.

03

Keeping it proportionate

Allowing AI crawlers can conflict with scraping and price-scraping defences. A workable compromise is per-path: allow content pages while keeping APIs and admin blocked.

Do not switch off bot protection wholesale to achieve it — that trades a small problem for a larger one.

Sources

  1. [1]Optimizing your website for generative AI features on Google Search．Google Search Central．2026-05-15

Updated 2026-08-10