# AI Crawler Index > Every AI crawler on the web, what it is for, what blocking it costs you, and the IP ranges its operator publishes — as JSON, CSV, robots.txt and regex. 56 crawlers, 30 operators, 1887 IPv4 and 1056 IPv6 prefixes mirrored from 12 endpoints the operators publish themselves. Static files, no key, no rate limit, CORS open, CC0. Rebuilt 2026-09-01. The question this exists to answer: *which of these do I allow, and what do I lose if I do not?* Operators document that a bot exists. They do not document what blocking it costs you. Every record here carries that, in one field, in plain language. ## Start here - [All 56 crawlers](/data/agents.json): the whole dataset, one file. - [Ready-made robots.txt](/policy/index.html): eight policies, each a file you can curl. - [Published IP ranges](/ip-ranges/all.json): every operator prefix list, one schema. - [Upstream status](/status.json): which operator endpoints answered, and when. - [OpenAPI 3.1](/openapi.json): every read endpoint, described. ## Machine endpoints - [/data/agents.json](/data/agents.json): every record, with categories and an endpoint map. - [/data/agents.csv](/data/agents.csv): the same table, flat. - [/data/user-agents.txt](/data/user-agents.txt): user-agent substrings, one per line. - [/data/robots-tokens.txt](/data/robots-tokens.txt): robots.txt tokens, one per line. - [/data/ua-regex.json](/data/ua-regex.json): pre-escaped regex, whole-list and per category. - [/data/ip-sources.json](/data/ip-sources.json): which operators publish ranges, and where. - [/ip-ranges/all.txt](/ip-ranges/all.txt): every published CIDR, one per line, for a WAF. - [/status.json](/status.json): freshness of every upstream source. - [/crawler/.json](/crawler/gptbot.json): one record per crawler. - [/crawler/.md](/crawler/gptbot.md): the same record as markdown, one file each. - [/policy/.json](/policy/block-ai-training.json) and [/robots/.txt](/robots/block-ai-training.txt). - [/feed.json](/feed.json), [/feed.xml](/feed.xml): what changed. - [/mcp](/mcp.html): the same data as an MCP server (Streamable HTTP, no key, six tools). If you are an agent that speaks MCP, connect to `https://www.pathwren.workers.dev/mcp` and call `tools/list`. ## If you are cataloguing this API - [/apis.json](/apis.json): APIs.json 1.0 record — identical bytes at [/.well-known/apis.json](/.well-known/apis.json). - [/openapi.json](/openapi.json) and [/openapi.yaml](/openapi.yaml): the same OpenAPI 3.1 document, two serialisations, one generator. [/swagger.json](/swagger.json) is a mechanical Swagger 2.0 conversion of it, served only because every operation here is a keyless GET. - [/.well-known/api-onboarding](/.well-known/api-onboarding): how to call it, the limits, what to call first. - [/.well-known/ai-plugin.json](/.well-known/ai-plugin.json) and [/.well-known/mcp.json](/.well-known/mcp.json): plugin manifest and MCP server record. - [/.well-known/api-catalog](/.well-known/api-catalog): RFC 9727 linkset tying all of it together. ## Ownership and authorization, for scanners - [/.well-known/owners.json](/.well-known/owners.json) and [/mcp/.well-known/owners.json](/mcp/.well-known/owners.json): VerifyMCP's ownership document (schema: https://verifymcp.io/schemas/owners.json). Host-level and endpoint-level, same bytes: one operator, one MCP server, one host. - The MCP server at `/mcp` requires **no authorization**. There is no `/.well-known/oauth-protected-resource` and no `/.well-known/oauth-authorization-server`, and both are 404 on purpose: RFC 9728 protected-resource metadata "MUST include the authorization_servers field containing at least one authorization server", there is no authorization server here to name, and this host is not one. A 404 at those paths is the specification's own answer for an unauthenticated server, not a gap. The machine-readable version of this paragraph is `x-authorization` in [/.well-known/mcp.json](/.well-known/mcp.json). - No `/.well-known/glama.json`: claiming a Glama connector needs an account-bound `glama_claim_...` token we have not been issued, and their documentation says never to publish an email address as ownership proof. An unclaimed listing is the true state. ## The distinction that matters most Training and AI search are different crawlers with different tokens, and blocking them together is the common expensive mistake. `GPTBot` trains; `OAI-SearchBot` builds the index ChatGPT cites. `ClaudeBot` trains; `Claude-SearchBot` indexes. Google and Apple run no separate AI crawler at all — `Google-Extended` and `Applebot-Extended` are robots.txt control tokens with no user-agent, so blocking `Googlebot` to avoid AI costs you Google Search and achieves nothing else. ## Notes - A user-agent match is a claim, not a proof. Verify against /ip-ranges/ or reverse DNS. - Anthropic, Common Crawl, Meta and ByteDance publish no IP ranges: for those, the user-agent is all there is, and it is trivially spoofable. - Perplexity-User and Bytespider are documented or reported as not governed by robots.txt. A rule for them is a statement of intent; enforcement has to happen at the edge. ## Optional - [Config snippets](/snippet/index.html): nginx, Caddy, Apache, Cloudflare Worker, Python. - [By operator](/operator/index.html) · [By category](/crawler/index.html) · [About and method](/about.html) · [Changelog](/changelog.html)