PLATFORMS

DeepSeek Crawler Access

IN ONE SENTENCE

DeepSeek's retrieval crawlers are no published identifier and its training crawler is not separately identified; citation requires allowing the retrieval ones, while training access is a separate brand decision.

Retrieval crawlers decide whether you can be cited; training crawlers decide whether your content trains the model. Decide them separately. For DeepSeek: retrieval is no published identifier; training is not separately identified.

OUR POSITION

No published crawler identifier does not mean no crawling. Look for it in server logs rather than assuming absence because no matching user agent appears in robots.txt.

01

Setting the access policy

Retrieval crawlers must be allowed for citation to be possible. This is a switch-level prerequisite that no amount of content quality can work around.

Training crawlers are a brand policy choice. Some argue for 'don't train on me, but do cite me'; for brands optimising for exposure, allowing both is usually the better trade.

⚠️ Allowing them is not only a robots.txt edit. A CDN or edge layer can block outside your site config — Cloudflare has blocked AI crawlers by default for new domains since 1 July 2025, and its 'Block AI bots' rule does not separate retrieval from training.

02

Verifying that crawling happens

Server logs are the primary source. Requests that do not execute JavaScript never reach front-end analytics, so almost none of this is visible there.

Checklist: are there requests from the expected user agent, do they get 200s, does the returned HTML contain the body copy rather than a shell, and has request volume dropped sharply.

⚠️ Watch specifically for serving an empty shell to crawlers while serving full HTML to conventional search engines. Nothing but a crawl-layer check reveals it, and it zeroes out every piece of content work.

Data behind this page

97%

Share of published llms.txt files receiving zero requests in May 2026

Source：Ahrefs, across 137,000 domains，2026

Sources

  1. [1]Optimizing your website for generative AI features on Google Search．Google Search Central．2026-05-15

Updated 2026-08-06