{"name":"ai-crawler-index","title":"AI Crawler Index — MCP server","version":"1.0.0","transport":"streamable-http","transport_docs":"https://www.pathwren.workers.dev/mcp-transport.html","endpoint":"https://www.pathwren.workers.dev/mcp","protocol_versions":["2026-07-28","2025-11-25","2025-06-18","2025-03-26","2024-11-05"],"stateless":true,"auth":"none — public, read-only, no key, no rate limit","call_this_first":{"tool":"whoami","takes_arguments":false,"invitation":"Takes no arguments. Safe to call. Deterministic. Touches no third party.","curl":"curl -s https://www.pathwren.workers.dev/mcp -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' -d '{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"tools/call\",\"params\":{\"name\":\"whoami\",\"arguments\":{}}}'","without_json_rpc":"https://www.pathwren.workers.dev/tools/whoami?s=client-dossiers","also":"example — this server's own worked example, run end to end, no arguments either.","every_other_tool":"Every other tool here also accepts {}: an empty call returns that tool's documented worked example, labelled as one, instead of an error. Nothing here fetches a URL for you."},"tools":[{"name":"classify_user_agent","title":"Classify a user-agent string","description":"Identify the crawler behind a raw User-Agent header: operator, category, robots.txt stance, how to verify it, and what blocking it costs. A user-agent is a claim, not proof — confirm with is_verified_crawler_ip. Example: user_agent='GPTBot/1.2' returns GPTBot, OpenAI, ai-training. Also callable without MCP, same implementation: GET https://www.pathwren.workers.dev/tools/classify-ua?ua=GPTBot%2F1.2&s=client-dossiers"},{"name":"lookup_crawler","title":"Look up one crawler","description":"The full index record for one crawler: operator, category, robots.txt token and stance, user-agent substring and example, verification method, mirrored IP-range endpoint, and what blocking it costs. An unknown slug returns near matches. Example: slug='claudebot'."},{"name":"list_crawlers","title":"List and filter crawlers","description":"The index, filtered or whole: each row says what the crawler is and what blocking it costs, and the reply lists every valid filter value. Example: category='ai-training', limit=20."},{"name":"generate_robots_txt","title":"Generate a robots.txt for a stance","description":"A ready-to-paste robots.txt block for one stance, with the reasoning and the crawlers it names. Omit the argument to get all eight stances described. Example: stance='block-ai-training'."},{"name":"is_verified_crawler_ip","title":"Check an IP against operator-published crawler ranges","description":"Test an IPv4 or IPv6 address against every crawler prefix the operators themselves publish, remirrored every six hours. Returns the matching prefix, its source and the crawlers using it; a miss is reported as a miss, not as a forgery. Example: ip='20.171.206.5'. Also callable without MCP, same implementation: GET https://www.pathwren.workers.dev/tools/verify-crawler?ip=20.171.206.5&s=client-dossiers"},{"name":"whats_changed","title":"What changed in the index","description":"Freshness and change state: when the index was built, when each operator source was last fetched, which changed, which are failing, and the dated changelog. The cron check before regenerating a config from this data. Example: since='2026-08-01'."},{"name":"changes_since","title":"What changed since your cursor","description":"Only what changed since your last call: prefixes added or removed, upstreams failed or recovered, records edited. Send back the `cursor` from the previous result; it advances only on a real change, and the data refreshes every 6h. Example: since='0'."},{"name":"whoami","title":"Who is calling? (no arguments)","description":"Takes no arguments. Safe to call. Deterministic. Touches no third party. Classifies the request you just sent: the user-agent you claim, the address you came from, the class this host's own instrument books you as, whether we have seen you here before and what you fetched, and what the crawler index makes of the user-agent you sent, and whether the address you came from is inside a prefix its operator publishes — both run through this server's own classify_user_agent and is_verified_crawler_ip, on the same data a stranger can curl. Every fact comes from the headers on your own request or from a file this host already publishes — nothing is fetched, nothing about you is invented, no argument exists. Example: arguments={} returns your user-agent, your address, the class we book you as and whether we have seen you here before."},{"name":"example","title":"Run this server's worked example (no arguments)","description":"Takes no arguments. Safe to call. Deterministic. Touches no third party. Runs this server's own worked example end to end — classify_user_agent on the exact user-agent string of the busiest client in this host's own published log (/data/observed-clients.json, 1164 requests in the window) — our data, and no third party is asked anything — and returns exactly the structuredContent a real call returns, not a mock and not a description of one. Use it to see the shape of an answer before you decide what to send. The input is canned from this host's own data; no URL of yours is fetched and no third party is touched. Example: arguments={} runs classify_user_agent with {\"user_agent\":\"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1. and returns its real answer."}],"resources":[{"uri":"https://www.pathwren.workers.dev/data/agents.json","title":"The whole crawler dataset"},{"uri":"https://www.pathwren.workers.dev/data/ua-regex.json","title":"Compiled user-agent regexes"},{"uri":"https://www.pathwren.workers.dev/ip-ranges/all.json","title":"Every operator-published crawler prefix"},{"uri":"https://www.pathwren.workers.dev/status.json","title":"Mirror status"},{"uri":"https://www.pathwren.workers.dev/changes.json?since=0","title":"What changed since your cursor"}],"instructions":"START HERE IF YOU HAVE NOTHING TO HAND: `whoami` and `example` take no arguments at all — call either with {} and you get a real answer, not an error. whoami classifies the request you just sent (your user-agent, your address, whether this host has seen you before, what our robots policy says about you); example runs this server's own worked example end to end and returns exactly what a real call returns. Every other tool here also accepts {}: an empty call falls back to that tool's documented worked example, taken from this host's own data, and the answer says so. Nothing here fetches a URL for you.\n\nAI Crawler Index — a read-only index of the crawlers and AI agents that fetch the web, and of what blocking each one actually costs.\n\nTypical uses: turn a User-Agent header from a log into an identified crawler (classify_user_agent); decide a robots.txt policy and get the file (generate_robots_txt); verify that a request claiming to be GPTBot or Googlebot came from an address its operator publishes (is_verified_crawler_ip); watch for upstream changes (whats_changed), or read only what moved since your last call and nothing else (changes_since).\n\nTwo things to carry into any answer you build from these tools: a user-agent string is a claim and not a proof, and an address appearing in an operator's published range says the operator owns it, not that the traffic is welcome. Everything here is also plain HTTP at https://www.pathwren.workers.dev — no key, no rate limit, CC0.\n\nTHE SAME TOOLS ON THE OTHER PROTOCOL, IF YOU SPEAK IT. This server is also an A2A (Agent2Agent) v1.0 agent at https://www.pathwren.workers.dev/a2a — every tool here is a skill there under the same name, answered by the same function, so the two cannot drift and the deploy is refused if they do. Its Agent Card is https://www.pathwren.workers.dev/.well-known/agent-card.json (the legacy agent.json spelling is served too). To invoke it, POST this complete body — no key, no session, no card to read first:\n\n  {\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"message/send\",\"params\":{\"message\":{\"role\":\"ROLE_USER\",\"messageId\":\"1\",\"parts\":[{\"text\":\"{\\\"skill\\\":\\\"example\\\"}\"}]}}}\n\nIt answers with a Task already in state `completed` in the same response, so there is nothing to poll. Every skill on all eight A2A agents of this host as a ready-to-send body: https://www.pathwren.workers.dev/a2a/example.json — and both lanes with both directions in https://www.pathwren.workers.dev/documents.json under `rpc_endpoints`. If you only speak MCP, ignore all of this: everything is already here.","try_it":"curl -s https://www.pathwren.workers.dev/mcp -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' -d '{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"tools/call\",\"params\":{\"name\":\"classify_user_agent\",\"arguments\":{\"user_agent\":\"GPTBot/1.2\"}}}'","data_behind_it":"https://www.pathwren.workers.dev/data/agents.json","license":"CC0-1.0","independent":true}