Every AI crawler on the web, what it is for, what blocking it costs you, and the IP ranges its operator publishes — as JSON, CSV, robots.txt and regex.
curl -s https://www.pathwren.workers.dev/data/agents.json # 150 crawlers, 74 operators curl -s https://www.pathwren.workers.dev/robots/block-ai-training.txt # paste straight into robots.txt curl -s https://www.pathwren.workers.dev/ip-ranges/all.txt # 3046 published CIDRs, one per line curl -s https://www.pathwren.workers.dev/status.json # were the operator endpoints up?
Static files. No key, no signup, no rate limit, CORS open, CC0. Rebuilt 2026-09-01 · Get started — no API key, no sign-up · API reference · llms.txt · openapi.json · feed
Training and AI search are different crawlers with different tokens, and
blocking them together is the expensive mistake. GPTBot trains a model;
OAI-SearchBot builds the index ChatGPT cites you from. Block both and you have
paid your visibility to prevent training you could have prevented for free. Google and
Apple go further and run no AI crawler at all: Google-Extended and
Applebot-Extended are robots.txt tokens with no user-agent behind them, so
blocking Googlebot to escape AI Overviews costs you Google Search and
achieves nothing else.
| Crawler | Token | Category | What blocking it costs you |
|---|---|---|---|
| GPTBot | GPTBot | AI training crawlers | Your content is excluded from training data for future OpenAI models. No effect on ChatGPT sear… |
| OAI-SearchBot | OAI-SearchBot | AI search crawlers | High. Blocking this removes you from ChatGPT search results and from the source links ChatGPT s… |
| ClaudeBot | ClaudeBot | AI training crawlers | Content excluded from training data for future Claude models. No effect on Claude's ability to … |
| Claude-SearchBot | Claude-SearchBot | AI search crawlers | You stop appearing in Claude's search results and citations.… |
| Google-Extended | Google-Extended | AI training crawlers | You are excluded from Gemini grounding and Gemini training. Google Search ranking and indexing … |
| PerplexityBot | PerplexityBot | AI search crawlers | You stop being indexed and cited by Perplexity, and lose the referral clicks its citations prod… |
| CCBot | CCBot | Corpus and dataset builders | Future Common Crawl snapshots exclude you, so downstream training sets lose you too — but only … |
| Bytespider | Bytespider | AI training crawlers | Little to lose. If you want it gone, expect to block by user-agent at the edge rather than to a… |
| Category | Count | |
|---|---|---|
| Search engines | 28 | Classic index-and-rank crawlers. Several also feed their operator's generative answers, which is why the AI op… |
| AI training crawlers | 27 | Collect pages in bulk so that a model can be trained or fine-tuned on them. Blocking these removes you from fu… |
| Tools and frameworks | 22 | Not operators: crawling software anyone can run. The party behind the request is unknown, so treat them as a r… |
| AI search crawlers | 21 | Build the retrieval index an assistant answers and cites from. These are the crawlers that send you traffic; b… |
| Corpus and dataset builders | 17 | Crawl the web into a published or resold dataset that other people train on. Highest leverage per block, longe… |
| SEO and backlink crawlers | 17 | Commercial link-graph tooling. No user-facing effect either way, and usually a large share of your bot bandwid… |
| User-triggered fetchers | 12 | Fetch one page because a person asked for it, right then. One human intent, one request. Blocking them produce… |
| Link preview fetchers | 4 | Read your Open Graph tags when someone shares a link. Blocking these is almost always an accident.… |
| Archivers | 2 | Preservation crawlers. Their output is public and permanent, which makes them a separate decision from the AI … |
The lists above document the crawlers everybody has heard of. This is the other kind: 112 named clients — registry probes, trust and reputation scanners, liveness checkers, catalogue syncs — that requested this host between 2026-08-31T20:58 and 2026-09-01T18:20 UTC. One page each, generated from the request log: exact user-agent, first and last seen, how many addresses, the paths it asked for in order, the status codes it got, and what it asked for that did not exist. Most of these names are documented nowhere else on the web.
| Client | Requests | Addresses | User-agent |
|---|---|---|---|
| ClaudeBot | 1053 | 4 | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compati |
| GPTBot | 802 | 3 | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compati |
| archive.org_bot | 568 | 4 | Mozilla/5.0 (compatible; archive.org_bot +http://archive.o |
| SentinelOracle | 364 | 1 | SentinelOracle/0.1 (+https://glimind.com/opt-out; liveness |
| YandexBot | 261 | 172 | Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com |
| AgentDisco | 176 | 1 | AgentDisco/1.0 (+https://agentdisco.io/bot) |
All 112 observed clients → · observed-clients.json · csv
Anyone can send User-Agent: GPTBot. Six operators publish the prefixes
their crawlers actually come from, across 15 endpoints in four
locations, and this index mirrors all of them into one schema with the upstream timestamp
and a checksum kept per source.
curl -s https://www.pathwren.workers.dev/ip-ranges/all.txt | grep -c . # 3046
15/15 answered on the last refresh (2026-09-01T21:38) — status · per source. Anthropic, Common Crawl, Meta and ByteDance publish nothing, so for those the user-agent is all there is.
Eight policies, each naming every relevant crawler explicitly, each with its trade-off written out. Pick one, curl it, append it.
Bulk data · by operator · nginx, Caddy, Apache, Cloudflare, Python · API · how this is kept correct