The whole index, spoken as Model Context Protocol instead of as files. Streamable HTTP, stateless, no key, no signup, no session to keep alive. Every tool is read-only.
# it answers right now — no account, no header ceremony
curl -s https://www.pathwren.workers.dev/mcp \
-H 'content-type: application/json' -H 'accept: application/json, text/event-stream' \
-d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' | jq -r '.result.tools[].name'
Anything that speaks MCP over HTTP. Claude Code:
claude mcp add --transport http ai-crawler-index https://www.pathwren.workers.dev/mcp
Clients that take a JSON config (Claude Desktop, Cursor, VS Code, Windsurf):
{
"mcpServers": {
"ai-crawler-index": { "type": "streamable-http", "url": "https://www.pathwren.workers.dev/mcp" }
}
}
| Tool | What it answers |
|---|---|
classify_user_agent | Given a raw User-Agent header: which crawler is this, who runs it, what is it for, and what does blocking it cost me? |
lookup_crawler | The full record for one crawler by slug, name or robots.txt token. |
list_crawlers | The 56 crawlers, filtered by category, operator, robots.txt stance or verification method. Returns the valid filter values too, so one call teaches the vocabulary. |
generate_robots_txt | A ready-to-paste robots.txt for one of 8 stances, with the reasoning and the exact tokens it implies. |
is_verified_crawler_ip | Is this IP inside a range the operator itself publishes? 1887 IPv4 and 1056 IPv6 prefixes, mirrored from 12 endpoints, refreshed every six hours. |
whats_changed | Build time, per-source freshness, which sources changed or are failing, and the dated changelog. The cron check for anything that regenerates a config from this data. |
Four, each pointing at the file the tools are computed from:
agents.json, ua-regex.json, ip-ranges/all.json,
status.json. Read them directly if you would rather do your own matching.
It has no side effects and no write path: it cannot edit your robots.txt, block anything,
or fetch a URL for you. A user-agent match is a claim, not a proof — classify_user_agent
says so on every answer, and is_verified_crawler_ip is the tool that turns a claim
into evidence. Where an operator publishes no ranges, no tool here can verify it, and each
one says which case you are in.
Protocol versions 2026-07-28, 2025-11-25, 2025-06-18, 2025-03-26, 2024-11-05, negotiated per call.
server/discover answers for clients on 2026-07-28, initialize for everyone else.
Same data as /data/agents.json and
/openapi.json — pick whichever your client speaks.
Listed in the official
MCP Registry as dev.workers.pathwren.www/ai-crawler-index.