# AI Crawler Index > Every AI crawler on the web, what it is for, what blocking it costs you, and the IP ranges its operator publishes — as JSON, CSV, robots.txt and regex. 56 crawlers, 30 operators, 1887 IPv4 and 1056 IPv6 prefixes mirrored from 12 endpoints the operators publish themselves. Static files, no key, no rate limit, CORS open, CC0. Rebuilt 2026-09-01. The question this exists to answer: *which of these do I allow, and what do I lose if I do not?* Operators document that a bot exists. They do not document what blocking it costs you. Every record here carries that, in one field, in plain language. ## Start here - [All 56 crawlers](/data/agents.json): the whole dataset, one file. - [Ready-made robots.txt](/policy/index.html): eight policies, each a file you can curl. - [Published IP ranges](/ip-ranges/all.json): every operator prefix list, one schema. - [Upstream status](/status.json): which operator endpoints answered, and when. - [OpenAPI 3.1](/openapi.json): every read endpoint, described. ## Machine endpoints - [/data/agents.json](/data/agents.json): every record, with categories and an endpoint map. - [/data/agents.csv](/data/agents.csv): the same table, flat. - [/data/user-agents.txt](/data/user-agents.txt): user-agent substrings, one per line. - [/data/robots-tokens.txt](/data/robots-tokens.txt): robots.txt tokens, one per line. - [/data/ua-regex.json](/data/ua-regex.json): pre-escaped regex, whole-list and per category. - [/data/ip-sources.json](/data/ip-sources.json): which operators publish ranges, and where. - [/ip-ranges/all.txt](/ip-ranges/all.txt): every published CIDR, one per line, for a WAF. - [/status.json](/status.json): freshness of every upstream source. - [/crawler/.json](/crawler/gptbot.json): one record per crawler. - [/crawler/.md](/crawler/gptbot.md): the same record as markdown, one file each. - [/policy/.json](/policy/block-ai-training.json) and [/robots/.txt](/robots/block-ai-training.txt). - [/feed.json](/feed.json), [/feed.xml](/feed.xml): what changed. - [/mcp](/mcp.html): the same data as an MCP server (Streamable HTTP, no key, six tools). If you are an agent that speaks MCP, connect to `https://www.pathwren.workers.dev/mcp` and call `tools/list`. - [/a2a](/a2a.html): the same six skills as an A2A (Agent2Agent) v1.0 agent — JSON-RPC 2.0 over POST, no key. Agent card at [/.well-known/agent-card.json](/.well-known/agent-card.json), identical bytes at the legacy [/.well-known/agent.json](/.well-known/agent.json) and from `GET /a2a`. No streaming, no push notifications, no task store: `SendMessage` answers with a `Message` rather than inventing a task id, and the card declares each of those false. ## If you are cataloguing this API - [/apis.json](/apis.json): APIs.json 1.0 record — identical bytes at [/.well-known/apis.json](/.well-known/apis.json). - [/openapi.json](/openapi.json) and [/openapi.yaml](/openapi.yaml): the same OpenAPI 3.1 document, two serialisations, one generator. [/swagger.json](/swagger.json) is a mechanical Swagger 2.0 conversion of it, served only because every operation here is a keyless GET. - [/.well-known/api-onboarding](/.well-known/api-onboarding): how to call it, the limits, what to call first. - [/.well-known/ai-plugin.json](/.well-known/ai-plugin.json) and [/.well-known/mcp.json](/.well-known/mcp.json): plugin manifest and MCP server record. - [/.well-known/api-catalog](/.well-known/api-catalog): RFC 9727 linkset tying all of it together. - [/.well-known/agent-card.json](/.well-known/agent-card.json): A2A v1.0 Agent Card for the agent at `/a2a`, with the legacy [/.well-known/agent.json](/.well-known/agent.json) serving identical bytes. It declares one JSONRPC interface and six skills; `streaming`, `pushNotifications` and `extendedAgentCard` are all `false` and the endpoint refuses each with the error code the spec assigns rather than pretending. ## Ownership and authorization, for scanners - [/.well-known/owners.json](/.well-known/owners.json) and [/mcp/.well-known/owners.json](/mcp/.well-known/owners.json): VerifyMCP's ownership document (schema: https://verifymcp.io/schemas/owners.json). Host-level and endpoint-level, same bytes: one operator, one MCP server, one host. - The MCP server at `/mcp` requires **no authorization**. There is no `/.well-known/oauth-protected-resource` and no `/.well-known/oauth-authorization-server`, and both are 404 on purpose: RFC 9728 protected-resource metadata "MUST include the authorization_servers field containing at least one authorization server", there is no authorization server here to name, and this host is not one. A 404 at those paths is the specification's own answer for an unauthenticated server, not a gap. The machine-readable version of this paragraph is `x-authorization` in [/.well-known/mcp.json](/.well-known/mcp.json). - No `/.well-known/glama.json`: claiming a Glama connector needs an account-bound `glama_claim_...` token we have not been issued, and their documentation says never to publish an email address as ownership proof. An unclaimed listing is the true state. ## The distinction that matters most Training and AI search are different crawlers with different tokens, and blocking them together is the common expensive mistake. `GPTBot` trains; `OAI-SearchBot` builds the index ChatGPT cites. `ClaudeBot` trains; `Claude-SearchBot` indexes. Google and Apple run no separate AI crawler at all — `Google-Extended` and `Applebot-Extended` are robots.txt control tokens with no user-agent, so blocking `Googlebot` to avoid AI costs you Google Search and achieves nothing else. ## Notes - A user-agent match is a claim, not a proof. Verify against /ip-ranges/ or reverse DNS. - Anthropic, Common Crawl, Meta and ByteDance publish no IP ranges: for those, the user-agent is all there is, and it is trivially spoofable. - Perplexity-User and Bytespider are documented or reported as not governed by robots.txt. A rule for them is a statement of intent; enforcement has to happen at the edge. ## Optional - [Config snippets](/snippet/index.html): nginx, Caddy, Apache, Cloudflare Worker, Python. - [By operator](/operator/index.html) · [By category](/crawler/index.html) · [About and method](/about.html) · [Changelog](/changelog.html) --- # Full index ## AI training crawlers Collect pages in bulk so that a model can be trained or fine-tuned on them. Blocking these removes you from future training sets and changes nothing a user sees today. ### anthropic-ai - Operator: Anthropic - robots.txt token: `anthropic-ai` - User-agent contains: `anthropic-ai` - robots.txt: control token only — no crawler - Verify by: no published verification method - Docs: https://support.anthropic.com/en/articles/8896518 - JSON: https://www.pathwren.workers.dev/crawler/anthropic-ai.json A legacy robots.txt token from before Anthropic consolidated on ClaudeBot. It is still widely present in robots.txt files and costs nothing to keep, but it is a control token rather than a bot you will see in logs. **Blocking it costs you:** None. Nothing crawls under this name today; keeping the rule is harmless insurance. ``` User-agent: anthropic-ai Disallow: / ``` ### Applebot-Extended - Operator: Apple - robots.txt token: `Applebot-Extended` - User-agent contains: `(control token only — no crawler)` - robots.txt: control token only — no crawler - Verify by: no published verification method - Docs: https://support.apple.com/en-us/119829 - JSON: https://www.pathwren.workers.dev/crawler/applebot-extended.json Apple's counterpart to Google-Extended: a robots.txt token that withdraws consent for Apple Intelligence and Apple foundation-model training, without touching Applebot's search crawl. **Blocking it costs you:** Excluded from Apple Intelligence training. Siri, Spotlight and Safari suggestions are unaffected. ``` User-agent: Applebot-Extended Disallow: / ``` ### Bytespider - Operator: ByteDance - robots.txt token: `Bytespider` - User-agent contains: `Bytespider` - robots.txt: compliance disputed - Verify by: no published verification method - Docs: https://www.bytespider.net/ - JSON: https://www.pathwren.workers.dev/crawler/bytespider.json ByteDance's crawler, associated with training data collection for Doubao and related models. Repeatedly reported by CDNs and site operators as the highest-volume AI crawler on the web and as inconsistent about robots.txt. **Blocking it costs you:** Little to lose. If you want it gone, expect to block by user-agent at the edge rather than to ask politely in robots.txt. ``` User-agent: Bytespider Disallow: / ``` ### ClaudeBot - Operator: Anthropic - robots.txt token: `ClaudeBot` - User-agent contains: `ClaudeBot` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://support.anthropic.com/en/articles/8896518 - JSON: https://www.pathwren.workers.dev/crawler/claudebot.json Anthropic's bulk crawler, gathering pages that may be used to train Claude models. **Blocking it costs you:** Content excluded from training data for future Claude models. No effect on Claude's ability to fetch a link a user gives it. ``` User-agent: ClaudeBot Disallow: / ``` ### cohere-training-data-crawler - Operator: Cohere - robots.txt token: `cohere-training-data-crawler` - User-agent contains: `cohere-training-data-crawler` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://cohere.com/ - JSON: https://www.pathwren.workers.dev/crawler/cohere-training-data-crawler.json Cohere's separately-named bulk crawler for model training data, split out so consent for training and consent for retrieval can differ. **Blocking it costs you:** Excluded from Cohere model training. ``` User-agent: cohere-training-data-crawler Disallow: / ``` ### FacebookBot - Operator: Meta - robots.txt token: `FacebookBot` - User-agent contains: `FacebookBot` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://developers.facebook.com/docs/sharing/webmasters/web-crawlers - JSON: https://www.pathwren.workers.dev/crawler/facebookbot.json Meta's older speech- and language-corpus crawler, largely superseded by meta-externalagent but still listed as a valid robots token. **Blocking it costs you:** Negligible today. Keep the rule; expect little traffic. ``` User-agent: FacebookBot Disallow: / ``` ### Google-Extended - Operator: Google - robots.txt token: `Google-Extended` - User-agent contains: `(control token only — no crawler)` - robots.txt: control token only — no crawler - Verify by: no published verification method - Docs: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers - JSON: https://www.pathwren.workers.dev/crawler/google-extended.json Not a crawler. A robots.txt token that tells Google whether pages Googlebot already fetched may be used to train and ground Gemini. You will never see it in an access log; disallowing it changes what Google does with content it fetched under a different name. **Blocking it costs you:** You are excluded from Gemini grounding and Gemini training. Google Search ranking and indexing are explicitly unaffected. This is the cleanest 'no training, keep my search traffic' lever that exists. ``` User-agent: Google-Extended Disallow: / ``` ### GoogleOther - Operator: Google - robots.txt token: `GoogleOther` - User-agent contains: `GoogleOther` - robots.txt: obeys robots.txt (documented) - Verify by: published IP ranges (https://developers.google.com/static/search/apis/ipranges/special-crawlers.json, 135 IPv4 / 135 IPv6) - Docs: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers - JSON: https://www.pathwren.workers.dev/crawler/googleother.json A generic fetcher used by Google product teams for one-off crawls and research, including data collection that does not belong to Search. **Blocking it costs you:** No effect on Search indexing. Blocks internal Google research and product fetches. ``` User-agent: GoogleOther Disallow: / ``` ### GPTBot - Operator: OpenAI - robots.txt token: `GPTBot` - User-agent contains: `GPTBot` - robots.txt: obeys robots.txt (documented) - Verify by: published IP ranges (https://openai.com/gptbot.json, 21 IPv4 / 0 IPv6) - Docs: https://platform.openai.com/docs/bots - JSON: https://www.pathwren.workers.dev/crawler/gptbot.json OpenAI's bulk crawler. Pages it fetches may be used to train future OpenAI foundation models. It is not the bot that puts you in ChatGPT's search results, and blocking it does not remove you from them. **Blocking it costs you:** Your content is excluded from training data for future OpenAI models. No effect on ChatGPT search visibility, on citations, or on links a user pastes into ChatGPT. ``` User-agent: GPTBot Disallow: / ``` ### meta-externalagent - Operator: Meta - robots.txt token: `meta-externalagent` - User-agent contains: `meta-externalagent` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://developers.facebook.com/docs/sharing/webmasters/web-crawlers - JSON: https://www.pathwren.workers.dev/crawler/meta-externalagent.json Meta's AI crawler, gathering training data for Llama and Meta AI. It replaced the older FacebookBot name for this purpose. **Blocking it costs you:** Excluded from Meta AI training. Link previews on Facebook, Instagram and WhatsApp are unaffected — those are a different bot. ``` User-agent: meta-externalagent Disallow: / ``` ### SemrushBot-OCOB - Operator: Semrush - robots.txt token: `SemrushBot-OCOB` - User-agent contains: `SemrushBot-OCOB` - robots.txt: obeys robots.txt (documented) - Verify by: reverse DNS - Docs: https://www.semrush.com/bot/ - JSON: https://www.pathwren.workers.dev/crawler/semrushbot-ocob.json Semrush's separately-tokenised crawler for its AI content tooling, split out so SEO crawling and AI reuse can be answered differently. **Blocking it costs you:** Exclusion from Semrush's AI corpus, with its SEO crawl unaffected. ``` User-agent: SemrushBot-OCOB Disallow: / ``` ### TikTokSpider - Operator: ByteDance - robots.txt token: `TikTokSpider` - User-agent contains: `TikTokSpider` - robots.txt: compliance disputed - Verify by: no published verification method - Docs: https://www.bytespider.net/ - JSON: https://www.pathwren.workers.dev/crawler/tiktokspider.json A second ByteDance crawler identifying with TikTok, collecting page content for the same family of models. **Blocking it costs you:** Little to lose unless TikTok search referral matters to you. ``` User-agent: TikTokSpider Disallow: / ``` ### Webzio-Extended - Operator: Webz.io - robots.txt token: `Webzio-Extended` - User-agent contains: `Webzio-Extended` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://webz.io/blog/machine-learning/ - JSON: https://www.pathwren.workers.dev/crawler/webzio-extended.json Webz.io's opt-out token specifically for AI training reuse, in the pattern Google and Apple established. **Blocking it costs you:** Your content is excluded from the AI-training tier of Webz.io's product while ordinary collection continues. ``` User-agent: Webzio-Extended Disallow: / ``` ## AI search crawlers Build the retrieval index an assistant answers and cites from. These are the crawlers that send you traffic; blocking them is the expensive mistake in this space. ### Amazonbot - Operator: Amazon - robots.txt token: `Amazonbot` - User-agent contains: `Amazonbot` - robots.txt: obeys robots.txt (documented) - Verify by: reverse DNS - Docs: https://developer.amazon.com/amazonbot - JSON: https://www.pathwren.workers.dev/crawler/amazonbot.json Amazon's crawler, feeding Alexa's ability to answer questions from the web and Amazon's own search and assistant products. **Blocking it costs you:** Alexa and Amazon's assistants stop answering from your pages. Verify with reverse DNS to crawl.amazonbot.amazon before trusting the user-agent. ``` User-agent: Amazonbot Disallow: / ``` ### Claude-SearchBot - Operator: Anthropic - robots.txt token: `Claude-SearchBot` - User-agent contains: `Claude-SearchBot` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://support.anthropic.com/en/articles/8896518 - JSON: https://www.pathwren.workers.dev/crawler/claude-searchbot.json Indexes pages so Claude's web search can find and cite them. Separate token from the training crawler, so search visibility and training consent are independent decisions. **Blocking it costs you:** You stop appearing in Claude's search results and citations. ``` User-agent: Claude-SearchBot Disallow: / ``` ### Claude-Web - Operator: Anthropic - robots.txt token: `Claude-Web` - User-agent contains: `Claude-Web` - robots.txt: control token only — no crawler - Verify by: no published verification method - Docs: https://support.anthropic.com/en/articles/8896518 - JSON: https://www.pathwren.workers.dev/crawler/claude-web.json An earlier Anthropic token for user-facing web access, superseded by Claude-User and Claude-SearchBot. Kept here because it appears in most published robots.txt templates. **Blocking it costs you:** None in practice. Retain the rule; expect no traffic. ``` User-agent: Claude-Web Disallow: / ``` ### DuckAssistBot - Operator: DuckDuckGo - robots.txt token: `DuckAssistBot` - User-agent contains: `DuckAssistBot` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot/ - JSON: https://www.pathwren.workers.dev/crawler/duckassistbot.json Fetches pages so DuckAssist can generate and cite answers inside DuckDuckGo. **Blocking it costs you:** No DuckAssist answers or citations from your site. Ordinary DuckDuckGo results are unaffected. ``` User-agent: DuckAssistBot Disallow: / ``` ### Google-CloudVertexBot - Operator: Google - robots.txt token: `Google-CloudVertexBot` - User-agent contains: `Google-CloudVertexBot` - robots.txt: obeys robots.txt (documented) - Verify by: published IP ranges (https://developers.google.com/static/search/apis/ipranges/special-crawlers.json, 135 IPv4 / 135 IPv6) - Docs: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers - JSON: https://www.pathwren.workers.dev/crawler/google-cloudvertexbot.json Crawls a site on behalf of a Vertex AI Agent Builder customer who is building an agent over that site. It only visits sites the customer has asked it to. **Blocking it costs you:** Third parties can no longer build Vertex AI agents that read your site. Irrelevant to Google Search. ``` User-agent: Google-CloudVertexBot Disallow: / ``` ### OAI-SearchBot - Operator: OpenAI - robots.txt token: `OAI-SearchBot` - User-agent contains: `OAI-SearchBot` - robots.txt: obeys robots.txt (documented) - Verify by: published IP ranges (https://openai.com/searchbot.json, 35 IPv4 / 0 IPv6) - Docs: https://platform.openai.com/docs/bots - JSON: https://www.pathwren.workers.dev/crawler/oai-searchbot.json Builds the index ChatGPT search answers from. Content it collects is used for retrieval and citation, not for model training. **Blocking it costs you:** High. Blocking this removes you from ChatGPT search results and from the source links ChatGPT shows. This is the single most expensive block on this list for anyone who wants to be cited by an assistant. ``` User-agent: OAI-SearchBot Disallow: / ``` ### PerplexityBot - Operator: Perplexity - robots.txt token: `PerplexityBot` - User-agent contains: `PerplexityBot` - robots.txt: obeys robots.txt (documented) - Verify by: published IP ranges (https://www.perplexity.ai/perplexitybot.json, 8 IPv4 / 0 IPv6) - Docs: https://docs.perplexity.ai/guides/bots - JSON: https://www.pathwren.workers.dev/crawler/perplexitybot.json Builds Perplexity's search index. Perplexity is citation-heavy by product design, so inclusion here converts to referral traffic more directly than most AI surfaces. **Blocking it costs you:** You stop being indexed and cited by Perplexity, and lose the referral clicks its citations produce. ``` User-agent: PerplexityBot Disallow: / ``` ### YouBot - Operator: You.com - robots.txt token: `YouBot` - User-agent contains: `YouBot` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://about.you.com/youbot/ - JSON: https://www.pathwren.workers.dev/crawler/youbot.json You.com's crawler, feeding its AI search product and its search API. **Blocking it costs you:** Removal from You.com's index and from answers built on its API. ``` User-agent: YouBot Disallow: / ``` ## User-triggered fetchers Fetch one page because a person asked for it, right then. One human intent, one request. Blocking them produces a visible error for a real reader. ### ChatGPT-User - Operator: OpenAI - robots.txt token: `ChatGPT-User` - User-agent contains: `ChatGPT-User` - robots.txt: obeys robots.txt (documented) - Verify by: published IP ranges (https://openai.com/chatgpt-user.json, 204 IPv4 / 0 IPv6) - Docs: https://platform.openai.com/docs/bots - JSON: https://www.pathwren.workers.dev/crawler/chatgpt-user.json Fetches a single page at the moment a user or a ChatGPT agent asks for it — a pasted link, a browsing step, an Operator task. One human intent, one request. OpenAI states these fetches are not used for training. **Blocking it costs you:** ChatGPT cannot open your pages when a user explicitly asks it to. The user sees a fetch failure. This is usually the last bot anyone means to block. ``` User-agent: ChatGPT-User Disallow: / ``` ### Claude-User - Operator: Anthropic - robots.txt token: `Claude-User` - User-agent contains: `Claude-User` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://support.anthropic.com/en/articles/8896518 - JSON: https://www.pathwren.workers.dev/crawler/claude-user.json Fetches a page because a Claude user asked Claude to read it, at that moment. **Blocking it costs you:** Claude reports a fetch failure to a user who asked for your page by name. ``` User-agent: Claude-User Disallow: / ``` ### cohere-ai - Operator: Cohere - robots.txt token: `cohere-ai` - User-agent contains: `cohere-ai` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://cohere.com/ - JSON: https://www.pathwren.workers.dev/crawler/cohere-ai.json Cohere's fetcher, used when its assistant products need a page. **Blocking it costs you:** Cohere-powered assistants cannot read your pages on request. ``` User-agent: cohere-ai Disallow: / ``` ### meta-externalfetcher - Operator: Meta - robots.txt token: `meta-externalfetcher` - User-agent contains: `meta-externalfetcher` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://developers.facebook.com/docs/sharing/webmasters/web-crawlers - JSON: https://www.pathwren.workers.dev/crawler/meta-externalfetcher.json Fetches a page when a Meta AI user asks about a specific link. **Blocking it costs you:** Meta AI cannot read pages users hand it. ``` User-agent: meta-externalfetcher Disallow: / ``` ### MistralAI-User - Operator: Mistral AI - robots.txt token: `MistralAI-User` - User-agent contains: `MistralAI-User` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://docs.mistral.ai/ - JSON: https://www.pathwren.workers.dev/crawler/mistralai-user.json Fetches a page when a Le Chat user asks Mistral's assistant to read it. **Blocking it costs you:** Le Chat cannot open links your readers give it. ``` User-agent: MistralAI-User Disallow: / ``` ### Perplexity-User - Operator: Perplexity - robots.txt token: `Perplexity-User` - User-agent contains: `Perplexity-User` - robots.txt: not governed by robots.txt (user-initiated, by operator policy) - Verify by: published IP ranges (https://www.perplexity.ai/perplexity-user.json, 4 IPv4 / 0 IPv6) - Docs: https://docs.perplexity.ai/guides/bots - JSON: https://www.pathwren.workers.dev/crawler/perplexity-user.json Fetches a page because a Perplexity user asked for it. Perplexity documents that this fetch is user-initiated and is therefore not governed by robots.txt — a robots rule will not stop it, by stated policy. **Blocking it costs you:** Not controllable via robots.txt. If you must stop it, verify by the published IP ranges and block at the edge — and accept that users who ask for your page get an error. ``` User-agent: Perplexity-User Disallow: / ``` ## Corpus and dataset builders Crawl the web into a published or resold dataset that other people train on. Highest leverage per block, longest delay before any effect. ### AI2Bot - Operator: Allen Institute for AI - robots.txt token: `AI2Bot` - User-agent contains: `AI2Bot` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://allenai.org/crawler - JSON: https://www.pathwren.workers.dev/crawler/ai2bot.json The Allen Institute's crawler, gathering pages for open research corpora such as Dolma that underpin fully open models like OLMo. **Blocking it costs you:** Excluded from open research datasets. Worth a deliberate decision: this is the category where 'blocking AI' also blocks the open, auditable end of it. ``` User-agent: AI2Bot Disallow: / ``` ### Ai2Bot-Dolma - Operator: Allen Institute for AI - robots.txt token: `Ai2Bot-Dolma` - User-agent contains: `Ai2Bot-Dolma` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://allenai.org/crawler - JSON: https://www.pathwren.workers.dev/crawler/ai2bot-dolma.json The variant of AI2's crawler named for the Dolma corpus specifically. **Blocking it costs you:** Same as AI2Bot: exclusion from an open, published training corpus. ``` User-agent: Ai2Bot-Dolma Disallow: / ``` ### CCBot - Operator: Common Crawl - robots.txt token: `CCBot` - User-agent contains: `CCBot` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://commoncrawl.org/faq - JSON: https://www.pathwren.workers.dev/crawler/ccbot.json Common Crawl's corpus builder. It trains nothing itself, but its archive is an input to most open and many closed LLM training sets, which makes it the highest-leverage single entry on this list. **Blocking it costs you:** Future Common Crawl snapshots exclude you, so downstream training sets lose you too — but only going forward. Existing snapshots are permanent and blocking today does not retract them. ``` User-agent: CCBot Disallow: / ``` ### Diffbot - Operator: Diffbot - robots.txt token: `Diffbot` - User-agent contains: `Diffbot` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://docs.diffbot.com/ - JSON: https://www.pathwren.workers.dev/crawler/diffbot.json Extracts structured records from pages to build a commercial knowledge graph that is resold and used for retrieval and training. **Blocking it costs you:** Your facts stop entering a widely-licensed knowledge graph. Whether that is a loss depends on whether you want to be a machine-readable entity. ``` User-agent: Diffbot Disallow: / ``` ### ImagesiftBot - Operator: Hive AI - robots.txt token: `ImagesiftBot` - User-agent contains: `ImagesiftBot` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://imagesift.com/about - JSON: https://www.pathwren.workers.dev/crawler/imagesiftbot.json Crawls images for Hive AI's reverse-image and dataset products. Image-heavy sites see this one long before they see the text crawlers. **Blocking it costs you:** Your images stop entering an image dataset and reverse-image index. ``` User-agent: ImagesiftBot Disallow: / ``` ### img2dataset - Operator: LAION / img2dataset - robots.txt token: `img2dataset` - User-agent contains: `img2dataset` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://github.com/rom1504/img2dataset - JSON: https://www.pathwren.workers.dev/crawler/img2dataset.json The tool used to turn image-URL lists such as LAION's into downloaded training sets. It is run by whoever is building a dataset, not by a single operator. **Blocking it costs you:** Your images are skipped when someone materialises an image-text dataset that references them. ``` User-agent: img2dataset Disallow: / ``` ### omgili - Operator: Webz.io - robots.txt token: `omgili` - User-agent contains: `omgili` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://webz.io/blog/machine-learning/ - JSON: https://www.pathwren.workers.dev/crawler/omgili.json The older robots token for the same Webz.io collection, still honoured and still worth listing. **Blocking it costs you:** Same as omgilibot. ``` User-agent: omgili Disallow: / ``` ### omgilibot - Operator: Webz.io - robots.txt token: `omgilibot` - User-agent contains: `omgilibot` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://webz.io/blog/machine-learning/ - JSON: https://www.pathwren.workers.dev/crawler/omgilibot.json Webz.io's crawler, collecting web and forum text sold as datasets, including to model builders. **Blocking it costs you:** Exclusion from a commercial dataset resold to third parties. ``` User-agent: omgilibot Disallow: / ``` ## Search engines Classic index-and-rank crawlers. Several also feed their operator's generative answers, which is why the AI opt-out for Google and Apple is a token rather than a block. ### Applebot - Operator: Apple - robots.txt token: `Applebot` - User-agent contains: `Applebot` - robots.txt: obeys robots.txt (documented) - Verify by: published IP ranges (https://search.developer.apple.com/applebot.json, 33 IPv4 / 0 IPv6) - Docs: https://support.apple.com/en-us/119829 - JSON: https://www.pathwren.workers.dev/crawler/applebot.json Powers Siri, Spotlight and Safari suggestions. Blocking it is a search decision, not an AI decision — the AI decision has its own token. **Blocking it costs you:** You disappear from Siri, Spotlight and Safari search suggestions across Apple's install base. ``` User-agent: Applebot Disallow: / ``` ### Baiduspider - Operator: Baidu - robots.txt token: `Baiduspider` - User-agent contains: `Baiduspider` - robots.txt: obeys robots.txt (documented) - Verify by: reverse DNS - Docs: https://help.baidu.com/question?prod_id=99&class=0&id=3001 - JSON: https://www.pathwren.workers.dev/crawler/baiduspider.json Baidu's search crawler, and the ingest path for Baidu's Ernie-backed answers. **Blocking it costs you:** Removal from Baidu Search, which matters only if you want Chinese-language traffic. ``` User-agent: Baiduspider Disallow: / ``` ### bingbot - Operator: Microsoft - robots.txt token: `bingbot` - User-agent contains: `bingbot` - robots.txt: obeys robots.txt (documented) - Verify by: published IP ranges (https://www.bing.com/toolbox/bingbot.json, 28 IPv4 / 0 IPv6) - Docs: https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0 - JSON: https://www.pathwren.workers.dev/crawler/bingbot.json Bing's only crawler, and therefore also the crawler behind Microsoft Copilot's grounding. Microsoft's documented way to keep search indexing while refusing generative reuse is the nocache / noarchive robots meta directive, not a separate user-agent. **Blocking it costs you:** Very high and very wide: Bing, Copilot, DuckDuckGo and several assistants that resell Bing's index all lose you at once. Use nocache/noarchive rather than blocking. ``` User-agent: bingbot Disallow: / ``` ### DuckDuckBot - Operator: DuckDuckGo - robots.txt token: `DuckDuckBot` - User-agent contains: `DuckDuckBot` - robots.txt: obeys robots.txt (documented) - Verify by: published IP ranges (https://duckduckgo.com/duckduckbot.json, 481 IPv4 / 0 IPv6) - Docs: https://duckduckgo.com/duckduckgo-help-pages/results/duckduckbot/ - JSON: https://www.pathwren.workers.dev/crawler/duckduckbot.json DuckDuckGo's own crawler. Note that the bulk of DuckDuckGo's web results come from Bing, so blocking bingbot removes you from DuckDuckGo whether or not you allow this one. **Blocking it costs you:** Limited on its own; the real DuckDuckGo lever is bingbot. ``` User-agent: DuckDuckBot Disallow: / ``` ### Googlebot - Operator: Google - robots.txt token: `Googlebot` - User-agent contains: `Googlebot` - robots.txt: obeys robots.txt (documented) - Verify by: published IP ranges (https://developers.google.com/static/search/apis/ipranges/googlebot.json, 169 IPv4 / 146 IPv6) - Docs: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers - JSON: https://www.pathwren.workers.dev/crawler/googlebot.json The classic search crawler. It is also the crawler behind AI Overviews: Google does not run a separate bot for them, which is why the only AI opt-out is the Google-Extended token and not a Googlebot block. **Blocking it costs you:** Total. You leave Google Search. Never block this to avoid AI use; use Google-Extended instead. ``` User-agent: Googlebot Disallow: / ``` ### Googlebot-Image - Operator: Google - robots.txt token: `Googlebot-Image` - User-agent contains: `Googlebot-Image` - robots.txt: obeys robots.txt (documented) - Verify by: published IP ranges (https://developers.google.com/static/search/apis/ipranges/googlebot.json, 169 IPv4 / 146 IPv6) - Docs: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers - JSON: https://www.pathwren.workers.dev/crawler/googlebot-image.json Image indexing for Google Images. A separate token so you can leave images out of search without leaving search. **Blocking it costs you:** Your images stop appearing in Google Images. ``` User-agent: Googlebot-Image Disallow: / ``` ### Googlebot-News - Operator: Google - robots.txt token: `Googlebot-News` - User-agent contains: `Googlebot-News` - robots.txt: obeys robots.txt (documented) - Verify by: published IP ranges (https://developers.google.com/static/search/apis/ipranges/googlebot.json, 169 IPv4 / 146 IPv6) - Docs: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers - JSON: https://www.pathwren.workers.dev/crawler/googlebot-news.json A robots.txt token controlling inclusion in Google News. It does not have its own user-agent string; the fetch arrives as Googlebot. **Blocking it costs you:** Removal from Google News, with normal Search unaffected. ``` User-agent: Googlebot-News Disallow: / ``` ### PetalBot - Operator: Huawei - robots.txt token: `PetalBot` - User-agent contains: `PetalBot` - robots.txt: obeys robots.txt (documented) - Verify by: reverse DNS - Docs: https://aspiegel.com/petalbot - JSON: https://www.pathwren.workers.dev/crawler/petalbot.json Huawei's crawler for Petal Search, shipped as the default search on Huawei devices. **Blocking it costs you:** Removal from Petal Search. Frequently blocked for volume rather than for policy. ``` User-agent: PetalBot Disallow: / ``` ### SeznamBot - Operator: Seznam - robots.txt token: `SeznamBot` - User-agent contains: `SeznamBot` - robots.txt: obeys robots.txt (documented) - Verify by: reverse DNS - Docs: https://napoveda.seznam.cz/en/seznamzbozi/subject-matter-crawler/ - JSON: https://www.pathwren.workers.dev/crawler/seznambot.json Seznam's crawler — the dominant search engine in the Czech Republic and one of the few national engines with its own index. **Blocking it costs you:** Removal from Seznam. Also removes you from its IndexNow endpoint's usefulness. ``` User-agent: SeznamBot Disallow: / ``` ### Storebot-Google - Operator: Google - robots.txt token: `Storebot-Google` - User-agent contains: `Storebot-Google` - robots.txt: obeys robots.txt (documented) - Verify by: published IP ranges (https://developers.google.com/static/search/apis/ipranges/special-crawlers.json, 135 IPv4 / 135 IPv6) - Docs: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers - JSON: https://www.pathwren.workers.dev/crawler/storebot-google.json Checks shopping and checkout flows for Google's shopping surfaces. **Blocking it costs you:** Product listings may lose shopping-specific enrichment. Irrelevant to non-commerce sites. ``` User-agent: Storebot-Google Disallow: / ``` ### Timpibot - Operator: Timpi - robots.txt token: `Timpibot` - User-agent contains: `Timpibot` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://timpi.io/ - JSON: https://www.pathwren.workers.dev/crawler/timpibot.json A distributed crawler building an independent search index outside the Google/Bing duopoly. **Blocking it costs you:** Absence from a small independent index. ``` User-agent: Timpibot Disallow: / ``` ### YandexBot - Operator: Yandex - robots.txt token: `YandexBot` - User-agent contains: `YandexBot` - robots.txt: obeys robots.txt (documented) - Verify by: reverse DNS - Docs: https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html - JSON: https://www.pathwren.workers.dev/crawler/yandexbot.json Yandex's search crawler, which also feeds Alice and Yandex's generative answers. **Blocking it costs you:** Removal from Yandex Search. Verify with reverse DNS to a yandex.ru, yandex.net or yandex.com host — YandexBot is among the most-spoofed user-agents there is. ``` User-agent: YandexBot Disallow: / ``` ### Yeti - Operator: Naver - robots.txt token: `Yeti` - User-agent contains: `Yeti` - robots.txt: obeys robots.txt (documented) - Verify by: reverse DNS - Docs: https://searchadvisor.naver.com/guide/seo-basic-crawl - JSON: https://www.pathwren.workers.dev/crawler/yeti.json Naver's crawler. Naver is South Korea's largest search portal and runs its own index and its own generative answers. **Blocking it costs you:** Removal from Naver, which is most of Korean search. ``` User-agent: Yeti Disallow: / ``` ## SEO and backlink crawlers Commercial link-graph tooling. No user-facing effect either way, and usually a large share of your bot bandwidth. ### AhrefsBot - Operator: Ahrefs - robots.txt token: `AhrefsBot` - User-agent contains: `AhrefsBot` - robots.txt: obeys robots.txt (documented) - Verify by: reverse DNS - Docs: https://ahrefs.com/robot - JSON: https://www.pathwren.workers.dev/crawler/ahrefsbot.json Ahrefs' backlink crawler, and one of the largest non-search crawlers on the web by request volume. **Blocking it costs you:** No user-facing effect. Ahrefs honours Crawl-delay, so rate-limiting is usually better than blocking. ``` User-agent: AhrefsBot Disallow: / ``` ### SemrushBot - Operator: Semrush - robots.txt token: `SemrushBot` - User-agent contains: `SemrushBot` - robots.txt: obeys robots.txt (documented) - Verify by: reverse DNS - Docs: https://www.semrush.com/bot/ - JSON: https://www.pathwren.workers.dev/crawler/semrushbot.json Semrush's backlink and keyword crawler. It is not an AI crawler, but it is usually in the top three by volume on any site, and it is the cheapest block on this list. **Blocking it costs you:** Your competitors' Semrush reports get thinner, and so do yours. No user-facing effect. ``` User-agent: SemrushBot Disallow: / ``` ## Archivers Preservation crawlers. Their output is public and permanent, which makes them a separate decision from the AI one. ### archive.org_bot - Operator: Internet Archive - robots.txt token: `archive.org_bot` - User-agent contains: `archive.org_bot` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://archive.org/details/archive.org_bot - JSON: https://www.pathwren.workers.dev/crawler/archive-org-bot.json The Wayback Machine's crawler. Preservation rather than AI, but it lands in the same 'is this bot welcome' decision and its output is a public corpus. **Blocking it costs you:** Your site stops being preserved. When it dies, it is gone. Consider this one separately from the AI question. ``` User-agent: archive.org_bot Disallow: / ``` ### ia_archiver - Operator: Internet Archive - robots.txt token: `ia_archiver` - User-agent contains: `ia_archiver` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://archive.org/details/archive.org_bot - JSON: https://www.pathwren.workers.dev/crawler/ia-archiver.json The legacy Alexa/Internet Archive token, still present in most robots.txt files and still occasionally honoured. **Blocking it costs you:** Negligible today; retain for tidiness. ``` User-agent: ia_archiver Disallow: / ``` ## Link preview fetchers Read your Open Graph tags when someone shares a link. Blocking these is almost always an accident. ### facebookexternalhit - Operator: Meta - robots.txt token: `facebookexternalhit` - User-agent contains: `facebookexternalhit` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://developers.facebook.com/docs/sharing/webmasters/web-crawlers - JSON: https://www.pathwren.workers.dev/crawler/facebookexternalhit.json The link unfurler: it reads your Open Graph tags when somebody shares your URL on a Meta property. **Blocking it costs you:** Severe and usually accidental. Your links share as bare grey boxes with no title, image or description across Facebook, Instagram, Messenger and WhatsApp. Almost nobody means to block this. ``` User-agent: facebookexternalhit Disallow: / ``` ## Tools and frameworks Not operators: crawling software anyone can run. The party behind the request is unknown, so treat them as a rate-limit question rather than a consent question. ### FirecrawlAgent - Operator: Firecrawl - robots.txt token: `FirecrawlAgent` - User-agent contains: `FirecrawlAgent` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://docs.firecrawl.dev/ - JSON: https://www.pathwren.workers.dev/crawler/firecrawlagent.json A hosted scrape-to-markdown service that LLM applications call to read pages. The requester is whoever is building on it, not Firecrawl itself, so volume and intent vary wildly. **Blocking it costs you:** Applications built on Firecrawl cannot read your pages. This is increasingly how agents fetch the web, so it is a bigger block than its name suggests. ``` User-agent: FirecrawlAgent Disallow: / ``` ### Google-InspectionTool - Operator: Google - robots.txt token: `Google-InspectionTool` - User-agent contains: `Google-InspectionTool` - robots.txt: obeys robots.txt (documented) - Verify by: published IP ranges (https://developers.google.com/static/search/apis/ipranges/special-crawlers.json, 135 IPv4 / 135 IPv6) - Docs: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers - JSON: https://www.pathwren.workers.dev/crawler/google-inspectiontool.json The fetcher behind Search Console's URL Inspection and the Rich Results Test. It runs when a site owner clicks a button. **Blocking it costs you:** Your own Search Console live tests stop working. Blocking this only hurts you. ``` User-agent: Google-InspectionTool Disallow: / ``` ### Scrapy - Operator: Scrapy project - robots.txt token: `Scrapy` - User-agent contains: `Scrapy` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://scrapy.org/ - JSON: https://www.pathwren.workers.dev/crawler/scrapy.json Not an operator: the default user-agent of the most common Python crawling framework. Anyone can be behind it. Modern Scrapy obeys robots.txt by default, which is why the default UA is still worth a rule. **Blocking it costs you:** You block a very large tail of unattributed one-off crawlers, and also every well-behaved researcher who did not change the default. ``` User-agent: Scrapy Disallow: / ``` --- ## robots.txt policies ### Allow everything, explicitly (`allow-all`, 56 crawlers) Every crawler on this index is named and allowed. Use when you want maximum reach into search and assistants and have nothing to withhold. An empty robots.txt already allows everything, so this file is not about permission — it is about being explicit. Naming each token means a later change is a one-line diff instead of a rewrite, and it documents that the allow was a decision. This is the policy this site itself serves. `curl -s https://www.pathwren.workers.dev/robots/allow-all.txt` ### Block AI training, keep AI search (`block-ai-training`, 13 crawlers) Refuse the crawlers that feed model training. Keep the ones that put you in ChatGPT, Claude, Perplexity and Gemini answers. The distinction most people actually want, and the one that is easy to get wrong: GPTBot trains, OAI-SearchBot indexes for citation. Blocking both loses you the traffic and gains you nothing extra. Google and Apple have no separate crawler at all — Google-Extended and Applebot-Extended are pure control tokens, so they belong in this file while Googlebot and Applebot must not. `curl -s https://www.pathwren.workers.dev/robots/block-ai-training.txt` ### Block every AI crawler (`block-all-ai`, 35 crawlers) Training, AI search, user-triggered fetches and corpus builders, all refused. Classic search engines still allowed. The maximal AI opt-out that still leaves you in Google and Bing. Understand the price before deploying it: you will not be cited by any assistant, and when a reader explicitly asks ChatGPT or Claude to open your page, they get an error. Note also that Perplexity-User and Bytespider are listed here but documented as not governed by robots.txt, so this file is a statement of intent for those two, not an enforcement mechanism. `curl -s https://www.pathwren.workers.dev/robots/block-all-ai.txt` ### Block corpus and dataset builders (`block-datasets`, 8 crawlers) Refuse the crawlers whose output is a dataset other people train on: Common Crawl, AI2, Webz.io, Diffbot, ImagesiftBot. These are the highest-leverage blocks per line, because one crawl becomes many downstream training runs. It is also the block with the longest delay before it has any effect, and no effect at all on archives already published. `curl -s https://www.pathwren.workers.dev/robots/block-datasets.txt` ### Allow AI search and user fetches, block the rest (`allow-ai-search-only`, 28 crawlers) Be findable and citable in assistants without contributing to training corpora. The inverse framing of block-ai-training, written as an allowlist so the default for anything new is deny. Fetches a user explicitly asked for stay allowed, because refusing those produces a visible error for a real person who wanted your page. `curl -s https://www.pathwren.workers.dev/robots/allow-ai-search-only.txt` ### Block SEO and backlink crawlers (`block-seo-tools`, 2 crawlers) Ahrefs, Semrush and friends. No user-facing consequence, and often the largest single slice of your bot traffic. The cheapest bandwidth saving available to most sites, and the one nobody regrets. The only cost is that your own dashboards on those tools get thinner. Consider Crawl-delay first: both major operators honour it. `curl -s https://www.pathwren.workers.dev/robots/block-seo-tools.txt` ### Block the crawlers with disputed robots compliance (`block-disputed`, 3 crawlers) The ones repeatedly reported as ignoring robots.txt. Included for completeness — expect to enforce this at the edge instead. A robots.txt rule is a request. For the operators in this file the request is documented as unreliable or explicitly not applicable, so the honest use of this file is as a record of intent that sits alongside a real block by user-agent or by IP at your CDN. `curl -s https://www.pathwren.workers.dev/robots/block-disputed.txt` ### Maximum AI visibility (`maximum-ai-visibility`, 54 crawlers) Allow every AI crawler and every search engine; refuse only SEO scrapers. For sites whose goal is to be found and cited by machines. If your content exists to be read by assistants — documentation, reference data, an API — every block costs you and none of them protect anything. Pair this with an llms.txt, a sitemap, and per-item JSON, and the crawlers can actually use what they find. `curl -s https://www.pathwren.workers.dev/robots/maximum-ai-visibility.txt` --- ## Published IP-range endpoints | source | operator | upstream | IPv4 | IPv6 | operator timestamp | |---|---|---|---|---|---| | `openai-gptbot` | OpenAI | https://openai.com/gptbot.json | 21 | 0 | 2025-10-30T11:00:00.000000 | | `openai-searchbot` | OpenAI | https://openai.com/searchbot.json | 35 | 0 | 2026-01-02T11:00:00.000000 | | `openai-chatgpt-user` | OpenAI | https://openai.com/chatgpt-user.json | 204 | 0 | 2026-08-14T20:03:38.055838 | | `google-googlebot` | Google | https://developers.google.com/static/search/apis/ipranges/googlebot.json | 169 | 146 | 2026-08-31T14:46:46.000000 | | `google-special` | Google | https://developers.google.com/static/search/apis/ipranges/special-crawlers.json | 135 | 135 | 2026-08-31T14:46:39.000000 | | `google-user-triggered` | Google | https://developers.google.com/static/search/apis/ipranges/user-triggered-fetchers.json | 528 | 528 | 2026-08-31T14:46:36.000000 | | `google-user-triggered-google` | Google | https://developers.google.com/static/search/apis/ipranges/user-triggered-fetchers-google.json | 247 | 247 | 2026-08-31T14:46:39.000000 | | `bing-bingbot` | Microsoft | https://www.bing.com/toolbox/bingbot.json | 28 | 0 | 2024-01-03T10:00:00.121331 | | `apple-applebot` | Apple | https://search.developer.apple.com/applebot.json | 33 | 0 | 2026-07-31T10:00:00.000000 | | `duckduckgo-duckduckbot` | DuckDuckGo | https://duckduckgo.com/duckduckbot.json | 481 | 0 | 2026-07-03T15:15:37.000000 | | `perplexity-bot` | Perplexity | https://www.perplexity.ai/perplexitybot.json | 8 | 0 | 2025-02-07T16:56:00.000000 | | `perplexity-user` | Perplexity | https://www.perplexity.ai/perplexity-user.json | 4 | 0 | 2025-10-17T10:17:00.000000 |