ByteDance · AI training crawlers · json
User-agent: Bytespider Disallow: /
| robots.txt token | Bytespider |
| User-agent contains | Bytespider |
| Operator | ByteDance |
| Category | AI training crawlers |
| robots.txt | compliance disputed |
| Verify by | no published verification method |
ByteDance's crawler, associated with training data collection for Doubao and related models. Repeatedly reported by CDNs and site operators as the highest-volume AI crawler on the web and as inconsistent about robots.txt.
Little to lose. If you want it gone, expect to block by user-agent at the edge rather than to ask politely in robots.txt.
Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; spider-feedback@bytedance.com)
User-agent: Bytespider Allow: /
Operator documentation: https://www.bytespider.net/
Machine copies: json ·
markdown
Policies that name this crawler:
allow-all · block-ai-training · block-all-ai · block-disputed · maximum-ai-visibility