# Bytespider

> ByteDance's crawler, associated with training data collection for Doubao and related models. Repeatedly reported by CDNs and site operators as the highest-volume AI crawler on the web and as inconsistent about robots.txt.

| field | value |
|---|---|
| operator | ByteDance |
| category | AI training crawlers |
| robots.txt token | `Bytespider` |
| user-agent contains | `Bytespider` |
| robots.txt | compliance disputed |
| verify by | no published verification method |
| published IP ranges | none published |
| prefixes mirrored | 0 IPv4 / 0 IPv6 |
| operator docs | https://www.bytespider.net/ |
| last reviewed | 2026-09-01 |

## What blocking it costs you

Little to lose. If you want it gone, expect to block by user-agent at the edge rather than to ask politely in robots.txt.

## Full user-agent string

```
Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; spider-feedback@bytedance.com)
```

## Block it

```
User-agent: Bytespider
Disallow: /
```

## Allow it

```
User-agent: Bytespider
Allow: /
```

JSON: https://www.pathwren.workers.dev/crawler/bytespider.json · index: https://www.pathwren.workers.dev/llms.txt