Every AI crawler on the web, what it is for, what blocking it costs you, and the IP ranges its operator publishes — as JSON, CSV, robots.txt and regex.
curl -s https://www.pathwren.workers.dev/data/agents.json # 56 crawlers, 30 operators curl -s https://www.pathwren.workers.dev/robots/block-ai-training.txt # paste straight into robots.txt curl -s https://www.pathwren.workers.dev/ip-ranges/all.txt # 2943 published CIDRs, one per line curl -s https://www.pathwren.workers.dev/status.json # were the operator endpoints up?
Static files. No key, no signup, no rate limit, CORS open, CC0. Rebuilt 2026-09-01 · llms.txt · openapi.json · feed
Training and AI search are different crawlers with different tokens, and
blocking them together is the expensive mistake. GPTBot trains a model;
OAI-SearchBot builds the index ChatGPT cites you from. Block both and you have
paid your visibility to prevent training you could have prevented for free. Google and
Apple go further and run no AI crawler at all: Google-Extended and
Applebot-Extended are robots.txt tokens with no user-agent behind them, so
blocking Googlebot to escape AI Overviews costs you Google Search and
achieves nothing else.
| Crawler | Token | Category | What blocking it costs you |
|---|---|---|---|
| GPTBot | GPTBot | AI training crawlers | Your content is excluded from training data for future OpenAI models. No effect on ChatGPT sear… |
| OAI-SearchBot | OAI-SearchBot | AI search crawlers | High. Blocking this removes you from ChatGPT search results and from the source links ChatGPT s… |
| ClaudeBot | ClaudeBot | AI training crawlers | Content excluded from training data for future Claude models. No effect on Claude's ability to … |
| Claude-SearchBot | Claude-SearchBot | AI search crawlers | You stop appearing in Claude's search results and citations.… |
| Google-Extended | Google-Extended | AI training crawlers | You are excluded from Gemini grounding and Gemini training. Google Search ranking and indexing … |
| PerplexityBot | PerplexityBot | AI search crawlers | You stop being indexed and cited by Perplexity, and lose the referral clicks its citations prod… |
| CCBot | CCBot | Corpus and dataset builders | Future Common Crawl snapshots exclude you, so downstream training sets lose you too — but only … |
| Bytespider | Bytespider | AI training crawlers | Little to lose. If you want it gone, expect to block by user-agent at the edge rather than to a… |
| Category | Count | |
|---|---|---|
| AI training crawlers | 13 | Collect pages in bulk so that a model can be trained or fine-tuned on them. Blocking these removes you from fu… |
| Search engines | 13 | Classic index-and-rank crawlers. Several also feed their operator's generative answers, which is why the AI op… |
| AI search crawlers | 8 | Build the retrieval index an assistant answers and cites from. These are the crawlers that send you traffic; b… |
| Corpus and dataset builders | 8 | Crawl the web into a published or resold dataset that other people train on. Highest leverage per block, longe… |
| User-triggered fetchers | 6 | Fetch one page because a person asked for it, right then. One human intent, one request. Blocking them produce… |
| Tools and frameworks | 3 | Not operators: crawling software anyone can run. The party behind the request is unknown, so treat them as a r… |
| SEO and backlink crawlers | 2 | Commercial link-graph tooling. No user-facing effect either way, and usually a large share of your bot bandwid… |
| Archivers | 2 | Preservation crawlers. Their output is public and permanent, which makes them a separate decision from the AI … |
| Link preview fetchers | 1 | Read your Open Graph tags when someone shares a link. Blocking these is almost always an accident.… |
Anyone can send User-Agent: GPTBot. Six operators publish the prefixes
their crawlers actually come from, across 12 endpoints in four
locations, and this index mirrors all of them into one schema with the upstream timestamp
and a checksum kept per source.
curl -s https://www.pathwren.workers.dev/ip-ranges/all.txt | grep -c . # 2943
12/12 answered on the last refresh (2026-08-31T22:48) — status · per source. Anthropic, Common Crawl, Meta and ByteDance publish nothing, so for those the user-agent is all there is.
Eight policies, each naming every relevant crawler explicitly, each with its trade-off written out. Pick one, curl it, append it.
Bulk data · by operator · nginx, Caddy, Apache, Cloudflare, Python · API · how this is kept correct