PLATFORMS

Doubao Crawler Access

IN ONE SENTENCE

Doubao's retrieval crawlers are Bytespider and its training crawler is Bytespider; citation requires allowing the retrieval ones, while training access is a separate brand decision.

Retrieval crawlers decide whether you can be cited; training crawlers decide whether your content trains the model. Decide them separately. For Doubao: retrieval is Bytespider; training is Bytespider.

OUR POSITION

⚠️ Working the Chinese ecosystem requires allowing Bytespider. Note also that naive string matching produces a high false-positive rate here — figures reported to clients need manual review.

01

Setting the access policy

Retrieval crawlers must be allowed for citation to be possible. This is a switch-level prerequisite that no amount of content quality can work around.

Training crawlers are a brand policy choice. Some argue for 'don't train on me, but do cite me'; for brands optimising for exposure, allowing both is usually the better trade.

⚠️ Allowing them is not only a robots.txt edit. A CDN or edge layer can block outside your site config — Cloudflare has blocked AI crawlers by default for new domains since 1 July 2025, and its 'Block AI bots' rule does not separate retrieval from training.

02

Verifying that crawling happens

Server logs are the primary source. Requests that do not execute JavaScript never reach front-end analytics, so almost none of this is visible there.

Checklist: are there requests from the expected user agent, do they get 200s, does the returned HTML contain the body copy rather than a shell, and has request volume dropped sharply.

⚠️ Watch specifically for serving an empty shell to crawlers while serving full HTML to conventional search engines. Nothing but a crawl-layer check reveals it, and it zeroes out every piece of content work.

Data behind this page

97%

Share of published llms.txt files receiving zero requests in May 2026

Source：Ahrefs, across 137,000 domains，2026

Sources

  1. [1]Optimizing your website for generative AI features on Google Search．Google Search Central．2026-05-15

Updated 2026-08-06