---
title: "AI Crawler Index — 150 AI crawlers, their robots.txt tokens and IP ranges"
description: "Every AI crawler on the web, what it is for, what blocking it costs you, and the IP ranges its operator publishes — as JSON, CSV, robots.txt and regex."
canonical: "https://www.pathwren.workers.dev/index.html"
url: "https://www.pathwren.workers.dev/index.md"
format: "markdown"
source: "the bytes of /index.html, in the build that wrote the page"
generator: "surfaces/ai-crawler-index/build.py"
generated: "2026-09-03T06:29:54+00:00"
license: "CC0-1.0"
---

# AI Crawler Index

> Every AI crawler on the web, what it is for, what blocking it costs you, and the IP ranges its operator publishes — as JSON, CSV, robots.txt and regex.

Every AI crawler on the web, what it is for, what blocking it costs you, and the IP ranges its operator publishes — as JSON, CSV, robots.txt and regex.

```bash
curl -s https://www.pathwren.workers.dev/data/agents.json          # 150 crawlers, 74 operators
curl -s https://www.pathwren.workers.dev/robots/block-ai-training.txt   # paste straight into robots.txt
curl -s https://www.pathwren.workers.dev/ip-ranges/all.txt         # 3046 published CIDRs, one per line
curl -s https://www.pathwren.workers.dev/status.json               # were the operator endpoints up?
```

Static files. No key, no signup, no rate limit, CORS open, CC0.
Rebuilt 2026-09-03 · [Get started — no API key, no sign-up](https://www.pathwren.workers.dev/register) ·
[Pricing — free, no plans](https://www.pathwren.workers.dev/pricing) ·
[API reference](https://www.pathwren.workers.dev/reference) ·
[llms.txt](https://www.pathwren.workers.dev/llms.txt) ·
[openapi.json](https://www.pathwren.workers.dev/openapi.json) · [feed](https://www.pathwren.workers.dev/feed.json)

## The one distinction that costs people money

Training and AI search are *different crawlers with different tokens*, and
blocking them together is the expensive mistake. `GPTBot` trains a model;
`OAI-SearchBot` builds the index ChatGPT cites you from. Block both and you have
paid your visibility to prevent training you could have prevented for free. Google and
Apple go further and run no AI crawler at all: `Google-Extended` and
`Applebot-Extended` are robots.txt tokens with no user-agent behind them, so
blocking `Googlebot` to escape AI Overviews costs you Google Search and
achieves nothing else.

| Crawler | Token | Category | What blocking it costs you |
| --- | --- | --- | --- |
| [GPTBot](https://www.pathwren.workers.dev/crawler/gptbot.html) | `GPTBot` | AI training crawlers | Your content is excluded from training data for future OpenAI models. No effect on ChatGPT sear… |
| [OAI-SearchBot](https://www.pathwren.workers.dev/crawler/oai-searchbot.html) | `OAI-SearchBot` | AI search crawlers | High. Blocking this removes you from ChatGPT search results and from the source links ChatGPT s… |
| [ClaudeBot](https://www.pathwren.workers.dev/crawler/claudebot.html) | `ClaudeBot` | AI training crawlers | Content excluded from training data for future Claude models. No effect on Claude's ability to … |
| [Claude-SearchBot](https://www.pathwren.workers.dev/crawler/claude-searchbot.html) | `Claude-SearchBot` | AI search crawlers | You stop appearing in Claude's search results and citations.… |
| [Google-Extended](https://www.pathwren.workers.dev/crawler/google-extended.html) | `Google-Extended` | AI training crawlers | You are excluded from Gemini grounding and Gemini training. Google Search ranking and indexing … |
| [PerplexityBot](https://www.pathwren.workers.dev/crawler/perplexitybot.html) | `PerplexityBot` | AI search crawlers | You stop being indexed and cited by Perplexity, and lose the referral clicks its citations prod… |
| [CCBot](https://www.pathwren.workers.dev/crawler/ccbot.html) | `CCBot` | Corpus and dataset builders | Future Common Crawl snapshots exclude you, so downstream training sets lose you too — but only … |
| [Bytespider](https://www.pathwren.workers.dev/crawler/bytespider.html) | `Bytespider` | AI training crawlers | Little to lose. If you want it gone, expect to block by user-agent at the edge rather than to a… |

[All 150 crawlers →](https://www.pathwren.workers.dev/crawler/index.html)

## By what the crawl is for

| Category | Count |   |
| --- | --- | --- |
| [Search engines](https://www.pathwren.workers.dev/category/search.html) | 28 | Classic index-and-rank crawlers. Several also feed their operator's generative answers, which is why the AI op… |
| [AI training crawlers](https://www.pathwren.workers.dev/category/ai-training.html) | 27 | Collect pages in bulk so that a model can be trained or fine-tuned on them. Blocking these removes you from fu… |
| [Tools and frameworks](https://www.pathwren.workers.dev/category/tool.html) | 22 | Not operators: crawling software anyone can run. The party behind the request is unknown, so treat them as a r… |
| [AI search crawlers](https://www.pathwren.workers.dev/category/ai-search.html) | 21 | Build the retrieval index an assistant answers and cites from. These are the crawlers that send you traffic; b… |
| [Corpus and dataset builders](https://www.pathwren.workers.dev/category/dataset.html) | 17 | Crawl the web into a published or resold dataset that other people train on. Highest leverage per block, longe… |
| [SEO and backlink crawlers](https://www.pathwren.workers.dev/category/seo.html) | 17 | Commercial link-graph tooling. No user-facing effect either way, and usually a large share of your bot bandwid… |
| [User-triggered fetchers](https://www.pathwren.workers.dev/category/user-fetch.html) | 12 | Fetch one page because a person asked for it, right then. One human intent, one request. Blocking them produce… |
| [Link preview fetchers](https://www.pathwren.workers.dev/category/preview.html) | 4 | Read your Open Graph tags when someone shares a link. Blocking these is almost always an accident.… |
| [Archivers](https://www.pathwren.workers.dev/category/archive.html) | 2 | Preservation crawlers. Their output is public and permanent, which makes them a separate decision from the AI … |

## Who actually crawled this host, from its own log

The lists above document the crawlers everybody has heard of. This is the other kind:
163 named clients — registry probes, trust and reputation
scanners, liveness checkers, catalogue syncs — that requested this host between
2026-08-31T20:58 and 2026-09-03T06:25 UTC. One page
each, generated from the request log: exact user-agent, first and last seen, how many addresses,
the paths it asked for in order, the status codes it got, and what it asked for that did not
exist. Most of these names are documented nowhere else on the web.

| Client | Requests | Addresses | User-agent |
| --- | --- | --- | --- |
| [GPTBot](https://www.pathwren.workers.dev/bot/gptbot.html) | 1835 | 11 | `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compati` |
| [SentinelOracle](https://www.pathwren.workers.dev/bot/sentineloracle.html) | 1463 | 1 | `SentinelOracle/0.1 (+https://glimind.com/opt-out; liveness` |
| [ClaudeBot](https://www.pathwren.workers.dev/bot/claudebot.html) | 1342 | 6 | `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compati` |
| [archive.org_bot](https://www.pathwren.workers.dev/bot/archive-org-bot.html) | 993 | 7 | `Mozilla/5.0 (compatible; archive.org_bot +http://archive.o` |
| [node](https://www.pathwren.workers.dev/bot/node.html) | 929 | 53 | `node` |
| [mcpbeat](https://www.pathwren.workers.dev/bot/mcpbeat.html) | 456 | 1 | `mcpbeat/0.1 (+https://mcpbeat.com/bot/; liveness check)` |

[All 163 observed clients →](https://www.pathwren.workers.dev/bot/) ·
[observed-clients.json](https://www.pathwren.workers.dev/data/observed-clients.json) ·
[csv](https://www.pathwren.workers.dev/data/observed-clients.csv)

## Verify before you trust the user-agent

Anyone can send `User-Agent: GPTBot`. Six operators publish the prefixes
their crawlers actually come from, across 15 endpoints in four
locations, and this index mirrors all of them into one schema with the upstream timestamp
and a checksum kept per source.

```bash
curl -s https://www.pathwren.workers.dev/ip-ranges/all.txt | grep -c .   # 3046
```

15/15 answered on the last refresh
(2026-09-03T02:42) — [status](https://www.pathwren.workers.dev/status.html) ·
[per source](https://www.pathwren.workers.dev/ip-ranges/). Anthropic, Common Crawl, Meta and ByteDance publish
nothing, so for those the user-agent is all there is.

## Ready-made robots.txt

Eight policies, each naming every relevant crawler explicitly, each with its trade-off
written out. Pick one, curl it, append it.

| Policy | Crawlers |   |
| --- | --- | --- |
| [Allow everything, explicitly](https://www.pathwren.workers.dev/policy/allow-all.html) | 150 | [curl](https://www.pathwren.workers.dev/robots/allow-all.txt) |
| [Block AI training, keep AI search](https://www.pathwren.workers.dev/policy/block-ai-training.html) | 27 | [curl](https://www.pathwren.workers.dev/robots/block-ai-training.txt) |
| [Block every AI crawler](https://www.pathwren.workers.dev/policy/block-all-ai.html) | 77 | [curl](https://www.pathwren.workers.dev/robots/block-all-ai.txt) |
| [Block corpus and dataset builders](https://www.pathwren.workers.dev/policy/block-datasets.html) | 17 | [curl](https://www.pathwren.workers.dev/robots/block-datasets.txt) |
| [Allow AI search and user fetches, block the rest](https://www.pathwren.workers.dev/policy/allow-ai-search-only.html) | 65 | [curl](https://www.pathwren.workers.dev/robots/allow-ai-search-only.txt) |
| [Block SEO and backlink crawlers](https://www.pathwren.workers.dev/policy/block-seo-tools.html) | 17 | [curl](https://www.pathwren.workers.dev/robots/block-seo-tools.txt) |
| [Block the crawlers with disputed robots compliance](https://www.pathwren.workers.dev/policy/block-disputed.html) | 18 | [curl](https://www.pathwren.workers.dev/robots/block-disputed.txt) |
| [Maximum AI visibility](https://www.pathwren.workers.dev/policy/maximum-ai-visibility.html) | 133 | [curl](https://www.pathwren.workers.dev/robots/maximum-ai-visibility.txt) |

## Everything else

[Bulk data](https://www.pathwren.workers.dev/data/index.html) · [by operator](https://www.pathwren.workers.dev/operator/index.html)
· [nginx, Caddy, Apache, Cloudflare, Python](https://www.pathwren.workers.dev/snippet/index.html)
· [API](https://www.pathwren.workers.dev/api.html) · [how this is kept correct](https://www.pathwren.workers.dev/about.html)

## Sitemap

- [Full sitemap (XML)](https://www.pathwren.workers.dev/sitemap.xml) — every page, with dates
- [Full sitemap (markdown)](https://www.pathwren.workers.dev/sitemap.md) — the same map, readable
- [llms.txt](https://www.pathwren.workers.dev/llms.txt) — the whole host in one text file
- [documents.json](https://www.pathwren.workers.dev/documents.json) — every document, with its ETag
- [A2A agents](https://www.pathwren.workers.dev/a2a.html)
- [About and method](https://www.pathwren.workers.dev/about.html)
- [API](https://www.pathwren.workers.dev/api.html)
- [Changelog](https://www.pathwren.workers.dev/changelog.html)
- [Compliance](https://www.pathwren.workers.dev/compliance)
- [Contact](https://www.pathwren.workers.dev/contact)
- [Impressum · Anbieterkennzeichnung](https://www.pathwren.workers.dev/impressum)
- [AI Crawler Index](https://www.pathwren.workers.dev/index.html)
- [No model runs here](https://www.pathwren.workers.dev/inference.html)
- [Legal](https://www.pathwren.workers.dev/legal)
- [MCP server](https://www.pathwren.workers.dev/mcp-doctor.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-lint.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-netcheck.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-robots.html)
- [MCP transport: the GET and HEAD leg](https://www.pathwren.workers.dev/mcp-transport.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-triage.html)
- [MCP server](https://www.pathwren.workers.dev/mcp.html)
- [Packages](https://www.pathwren.workers.dev/packages.html)
- [Pricing](https://www.pathwren.workers.dev/pricing)
- [Privacy](https://www.pathwren.workers.dev/privacy.html)
- [API reference](https://www.pathwren.workers.dev/reference)
- [Access, keys and sign-up](https://www.pathwren.workers.dev/register)
- [Security posture](https://www.pathwren.workers.dev/security.html)
- [Upstream status](https://www.pathwren.workers.dev/status.html)
- [Terms of use](https://www.pathwren.workers.dev/terms.html)
- [Trust](https://www.pathwren.workers.dev/trust)

## Machine copies of this page

- [HTML (canonical)](https://www.pathwren.workers.dev/index.html)
- [JSON](https://www.pathwren.workers.dev/index.json)
- [Markdown](https://www.pathwren.workers.dev/index.md) — this document

This document is a markdown rendering of [https://www.pathwren.workers.dev/index.html](https://www.pathwren.workers.dev/index.html), generated from that page's own bytes in the same build. The HTML page is canonical.