# Bytespider

What this client asked https://www.pathwren.workers.dev for, from that host's own request log.
Window: 2026-09-01 07:20:36Z to 2026-09-13 17:24:43Z UTC. Generated 2026-09-14.

## User-agent strings, exactly as they arrived

- `Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; spider-feedback@bytedance.com)` — 51 request(s), 21 address(es), 2026-09-01 07:20:36Z to 2026-09-13 17:24:43Z

- Requests: 51
- Distinct addresses (salted hashes, never published): 21
- Distinct paths: 36
- Visits (30-minute gap rule): 33
- Status codes: 200×51
- Accept headers: image/heif,image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8, text/html,application/xhtml+xml,application/xml;q=0.9,image/heif,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
- Our classifier called it: crawler×50, agent×1 (ours, from the user-agent)

## What it asked for, in order

1. `/icon.png` — 2026-09-01 07:20:36Z — 200
2. `/favicon.ico` — 2026-09-04 23:15:20Z — 200
3. `/c/lemmy/crawler/` — 2026-09-06 00:15:35Z — 200
4. `/px.gif` — 2026-09-06 00:15:36Z — 200
5. `/c/lemmy/policy/` — 2026-09-06 00:39:25Z — 200
6. `/bot/` — 2026-09-06 02:58:35Z — 200
7. `/c/lemmy/ip-ranges/` — 2026-09-06 03:02:36Z — 200
8. `/data/` — 2026-09-06 03:06:38Z — 200
9. `/c/mbin/ip-ranges/` — 2026-09-06 06:14:39Z — 200
10. `/blog/ai-crawler-traffic-2026-w36.html` — 2026-09-06 09:16:11Z — 200
11. `/` — 2026-09-06 11:36:40Z — 200
12. `/c/piefed/api/` — 2026-09-07 03:57:50Z — 200
13. `/c/lemmy/ai-crawler-robots/` — 2026-09-08 03:14:34Z — 200
14. `/data/observed-clients.csv` — 2026-09-08 07:15:02Z — 200
15. `/robots/allow-ai-search-only.txt` — 2026-09-08 07:34:28Z — 200
16. `/c/mbin/sitemap.md` — 2026-09-08 10:23:12Z — 200
17. `/robots/block-ai-training.txt` — 2026-09-08 14:12:50Z — 200
18. `/robots/maximum-ai-visibility.txt` — 2026-09-08 19:40:07Z — 200
19. `/robots/block-disputed.txt` — 2026-09-08 20:38:45Z — 200
20. `/robots/block-all-ai.txt` — 2026-09-09 00:08:17Z — 200
21. `/.well-known/security.txt` — 2026-09-09 01:21:00Z — 200
22. `/c/lemmy/blog/crawler-fleet-fold-2026-w36.html` — 2026-09-09 03:13:57Z — 200
23. `/c/lemmy/ai-crawler-logs/` — 2026-09-09 03:24:04Z — 200
24. `/llms-full.txt` — 2026-09-09 16:52:59Z — 200
25. `/c/mbin/ip-ranges/index.md` — 2026-09-09 17:19:46Z — 200
26. `/data/ua-regex.txt` — 2026-09-10 01:08:52Z — 200
27. `/c/lemmy/ai-crawler-robots/index.md` — 2026-09-10 16:22:21Z — 200
28. `/ip-ranges/all.txt` — 2026-09-10 17:04:40Z — 200
29. `/robots/block-seo-tools.txt` — 2026-09-11 02:26:54Z — 200
30. `/robots/allow-all.txt` — 2026-09-11 09:55:41Z — 200
31. `/robots/block-datasets.txt` — 2026-09-11 16:43:24Z — 200
32. `/data/robots-tokens.txt` — 2026-09-11 16:49:44Z — 200
33. `/data/agents.csv` — 2026-09-11 21:17:40Z — 200
34. `/c/lemmy/sitemap.md` — 2026-09-11 23:13:20Z — 200
35. `/index.md` — 2026-09-12 04:56:49Z — 200
36. `/c/lemmy/blog/crawler-fleet-fold-2026-w36.md` — 2026-09-13 03:03:54Z — 200

## What it asked for that did not exist

Nothing.

## Verification

no published verification methodFrom the crawler record at /crawler/bytespider.html, which cites https://www.bytespider.net/ — one source for both pages.

## Ask this page a question, with one GET

A question about this page, answered by a call you can make right now — no key, no account, no JSON-RPC, nothing to assemble. The query string is already filled in with this page's own subject.

- **What is Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/…?**  
  `curl -s 'https://www.pathwren.workers.dev/tools/classify-ua?ua=Mozilla%2F5.0%20%28Linux%3B%20Android%205.0%29%20AppleWebKit%2F537.36%20%28KHTML%2C%20like%20Gecko%29%20Mobile%20Safari%2F537.36%20%28compatible%3B%20Bytespider%3B%20spider-feedback%40bytedance.com%29&s=client-dossiers'`  
  This exact user-agent run through the catalogue: which crawler it is, who operates it, what the crawl is for, what blocking it costs — or an honest "not recognised" if it is nobody we know.

Every parameter above is a value the endpoint reads, not an address it visits: nothing is fetched on your behalf. Called without any parameter, each endpoint returns its own specification instead of an error.


## What it publishes about you

Not observed. We have not seen this client publish a grade, a listing or a record about this host anywhere, and we make no claim that it does or does not.

## Documentation it names

mailto:spider-feedback@bytedance.com

JSON: https://www.pathwren.workers.dev/bot/bytespider.json · all clients: https://www.pathwren.workers.dev/data/observed-clients.json