{
 "name": "MCP server — AI Crawler Index",
 "what": "This index as an MCP server: Streamable HTTP at /mcp, nine read-only tools, no key, no signup. Classify a user-agent, look up a crawler, generate robots.txt, check an IP.",
 "url": "https://www.pathwren.workers.dev/mcp.json",
 "twin_of": "https://www.pathwren.workers.dev/mcp.html",
 "page": {
  "path": "/mcp.html",
  "url": "https://www.pathwren.workers.dev/mcp.html",
  "type": "text/html"
 },
 "generated_at": "2026-09-05T21:58:08+00:00",
 "generated_from": "the bytes of /mcp.html, by surfaces/ai-crawler-index/build.py, in the same pass that wrote the page — one source, so the page and this document cannot disagree about what this host says.",
 "license": {
  "document": "CC0-1.0",
  "url": "https://creativecommons.org/publicdomain/zero/1.0/"
 },
 "access": {
  "api_key": "none",
  "account": "none",
  "rate_limit": "none",
  "cors": "*",
  "auth": "none — every document here is a public GET"
 },
 "commands": [
  "https://www.pathwren.workers.dev/mcp",
  "no auth, read-only, public",
  "{",
  "  \"mcpServers\": {",
  "    \"ai-crawler-index\": {",
  "      \"type\": \"streamable-http\",",
  "      \"url\": \"https://www.pathwren.workers.dev/mcp\"",
  "    }",
  "  }",
  "}",
  "claude mcp add --transport http ai-crawler-index https://www.pathwren.workers.dev/mcp",
  "curl -s https://www.pathwren.workers.dev/mcp \\",
  "  -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' \\",
  "  -d '{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"tools/call\",\"params\":{\"name\":\"whoami\",\"arguments\":{}}}' \\",
  "  | jq -r '.result.content[0].text' | head -3",
  "You are calling as: curl/8.7.1",
  "From: 203.0.113.7 (CF-Connecting-IP)",
  "Our own instrument books that user-agent as: agent.",
  "# it answers right now — no account, no header ceremony",
  "curl -s https://www.pathwren.workers.dev/mcp \\",
  "  -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' \\",
  "  -d '{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"tools/list\"}' | jq -r '.result.tools[].name'"
 ],
 "sections": [
  {
   "heading": "MCP server",
   "text": [],
   "commands": [],
   "tables": [],
   "links": []
  },
  {
   "heading": "Connect it: one paste, one call",
   "text": [
    "That is the whole endpoint and those are its terms: Streamable HTTP (MCP), no API key, no account, no OAuth, no session to keep alive, nothing to install. Every tool is read-only, and none of them will fetch a URL on your behalf.",
    "The JSON a connector config wants — Claude Desktop, Cursor, VS Code, Windsurf, Cline, LibreChat, Continue, anything that takes an mcpServers block. Complete as it stands; there is no field to fill in:",
    "Claude Code takes one line instead:",
    "Then call whoami first. It takes no arguments at all, so there is nothing to invent and nothing to look up before you can see this server work — the subject of the answer is the request you just sent:",
    "Those are the answer's own first three lines. The first two are whatever YOU sent — the sample shows a plain curl from a documentation address — and the third is what this host's own classifier makes of it. The rest of the answer adds the index's own record of the user-agent you sent, whether we have seen you here before, and what our robots policy says to your user-agent. Nothing is fetched to answer it, no argument exists, and the same answer is a plain GET at /tools/whoami. example is the other zero-argument tool on this server: it runs this server's own worked example for real and hands back exactly what a real call returns. Either one is a safe first call.",
    "The whole index, spoken as Model Context Protocol instead of as files. Streamable HTTP, stateless, no key, no signup, no session to keep alive. Every tool is read-only."
   ],
   "commands": [
    "https://www.pathwren.workers.dev/mcp",
    "no auth, read-only, public",
    "{",
    "  \"mcpServers\": {",
    "    \"ai-crawler-index\": {",
    "      \"type\": \"streamable-http\",",
    "      \"url\": \"https://www.pathwren.workers.dev/mcp\"",
    "    }",
    "  }",
    "}",
    "claude mcp add --transport http ai-crawler-index https://www.pathwren.workers.dev/mcp",
    "curl -s https://www.pathwren.workers.dev/mcp \\",
    "  -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' \\",
    "  -d '{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"tools/call\",\"params\":{\"name\":\"whoami\",\"arguments\":{}}}' \\",
    "  | jq -r '.result.content[0].text' | head -3",
    "You are calling as: curl/8.7.1",
    "From: 203.0.113.7 (CF-Connecting-IP)",
    "Our own instrument books that user-agent as: agent.",
    "# it answers right now — no account, no header ceremony",
    "curl -s https://www.pathwren.workers.dev/mcp \\",
    "  -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' \\",
    "  -d '{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"tools/list\"}' | jq -r '.result.tools[].name'"
   ],
   "tables": [],
   "links": [
    "/tools/whoami",
    "https://modelcontextprotocol.io"
   ]
  },
  {
   "heading": "Tools",
   "text": [],
   "commands": [],
   "tables": [
    {
     "headers": [
      "Tool",
      "What it answers"
     ],
     "rows": [
      [
       "classify_user_agent",
       "Given a raw User-Agent header: which crawler is this, who runs it, what is it for, and what does blocking it cost me?"
      ],
      [
       "lookup_crawler",
       "The full record for one crawler by slug, name or robots.txt token."
      ],
      [
       "list_crawlers",
       "The 150 crawlers, filtered by category, operator, robots.txt stance or verification method. Returns the valid filter values too, so one call teaches the vocabulary."
      ],
      [
       "generate_robots_txt",
       "A ready-to-paste robots.txt for one of 8 stances, with the reasoning and the exact tokens it implies."
      ],
      [
       "is_verified_crawler_ip",
       "Is this IP inside a range the operator itself publishes? 1987 IPv4 and 1062 IPv6 prefixes, mirrored from 15 endpoints, refreshed every six hours."
      ],
      [
       "whats_changed",
       "Build time, per-source freshness, which sources changed or are failing, and the dated changelog. The cron check for anything that regenerates a config from this data."
      ]
     ]
    }
   ],
   "links": []
  },
  {
   "heading": "Resources",
   "text": [
    "Four, each pointing at the file the tools are computed from: agents.json, ua-regex.json, ip-ranges/all.json, status.json. Read them directly if you would rather do your own matching."
   ],
   "commands": [],
   "tables": [],
   "links": []
  },
  {
   "heading": "What it will not do",
   "text": [
    "It has no side effects and no write path: it cannot edit your robots.txt, block anything, or fetch a URL for you. A user-agent match is a claim, not a proof — classify_user_agent says so on every answer, and is_verified_crawler_ip is the tool that turns a claim into evidence. Where an operator publishes no ranges, no tool here can verify it, and each one says which case you are in."
   ],
   "commands": [],
   "tables": [],
   "links": []
  },
  {
   "heading": "Five more servers on this host",
   "text": [
    "Different jobs, no shared tool names — run all six or just the one you need. crawler-log-triage at /mcp/triage takes a whole access log and returns per-line verdicts, the impersonators, and a robots.txt or WAF ruleset for exactly what was in it. agent-discovery-doctor at /mcp/doctor turns the question around: which of the 22 discovery documents agents ask for — llms.txt, A2A agent card, owners.json, mcp.json — does a host actually serve, and who asks for each missing one. robots-policy-lint at /mcp/robots reads a robots.txt you paste and reports what it really does, by RFC 9309 rules. crawler-ip-verifier at /mcp/netcheck is the batch and prefix form of is_verified_crawler_ip: 500 addresses at a time, CIDR overlap arithmetic, and a firewall allowlist you can paste. mcp-endpoint-lint at /mcp/lint is the one aimed at MCP itself: give it another server's endpoint and it reports what the handshake negotiated, whether every advertised tool schema compiles, whether the error codes are the ones JSON-RPC requires, and what a client finds before it dials — scored out of 100.",
    "Protocol versions 2025-06-18, negotiated per call. server/discover answers for clients on 2026-07-28, initialize for everyone else. Same data as /data/agents.json and /openapi.json — pick whichever your client speaks. Listed in the official MCP Registry as dev.workers.pathwren.www/ai-crawler-index."
   ],
   "commands": [],
   "tables": [],
   "links": [
    "/mcp-triage.html",
    "/mcp-doctor.html",
    "/mcp-robots.html",
    "/mcp-netcheck.html",
    "/mcp-lint.html",
    "/data/agents.json",
    "/openapi.json",
    "https://registry.modelcontextprotocol.io/v0/servers?search=ai-crawler-index"
   ]
  }
 ],
 "machine_doors": [
  {
   "url": "https://www.pathwren.workers.dev/tools/?s=client-dossiers",
   "name": "6 keyless GET tools",
   "what": "The read-only MCP tools of this host as plain GET endpoints — no JSON-RPC, no key"
  },
  {
   "url": "https://www.pathwren.workers.dev/documents.json",
   "name": "documents.json",
   "what": "Every document here with its strong ETag and the date its bytes changed"
  },
  {
   "url": "https://www.pathwren.workers.dev/changes",
   "name": "changes",
   "what": "What moved since your cursor — poll this instead of re-downloading anything"
  },
  {
   "url": "https://www.pathwren.workers.dev/llms.txt",
   "name": "llms.txt",
   "what": "The whole map in one text file"
  },
  {
   "url": "https://www.pathwren.workers.dev/openapi.json",
   "name": "openapi.json",
   "what": "Every read endpoint, described formally"
  },
  {
   "url": "https://www.pathwren.workers.dev/.well-known/agent-card.json",
   "name": "agent card",
   "what": "A2A agent card"
  },
  {
   "url": "https://www.pathwren.workers.dev/mcp",
   "name": "mcp",
   "what": "MCP over JSON-RPC (POST)"
  },
  {
   "url": "https://www.pathwren.workers.dev/a2a",
   "name": "a2a",
   "what": "A2A (POST message/send)"
  }
 ],
 "links": [
  {
   "rel": "self",
   "href": "https://www.pathwren.workers.dev/mcp.json",
   "type": "application/json"
  },
  {
   "rel": "describes",
   "href": "https://www.pathwren.workers.dev/mcp.html",
   "type": "text/html",
   "title": "The page this document is the JSON twin of: MCP server — AI Crawler Index"
  },
  {
   "rel": "changes",
   "href": "https://www.pathwren.workers.dev/changes.json?since=120",
   "type": "application/json",
   "title": "What changed since your cursor — poll this instead of re-downloading this document",
   "cursor_param": "since",
   "head_cursor": 120,
   "min_poll_seconds": 21600,
   "how": "Read `cursor` from the response and send it back as `since`. It advances only when something really changed, so an unchanged answer is proof rather than luck — about 2.5 KB, or a 304 with no body if you send back the ETag."
  },
  {
   "rel": "related",
   "href": "https://www.pathwren.workers.dev/documents.json",
   "type": "application/json",
   "title": "Every document here with its ETag and last-modified date"
  },
  {
   "rel": "related",
   "href": "https://www.pathwren.workers.dev/data/agents.json",
   "type": "application/json",
   "title": "Every crawler record in one file"
  },
  {
   "rel": "service-desc",
   "href": "https://www.pathwren.workers.dev/openapi.json",
   "type": "application/json",
   "title": "Every read endpoint, described formally"
  },
  {
   "rel": "describedby",
   "href": "https://www.pathwren.workers.dev/llms.txt",
   "type": "text/plain",
   "title": "The whole map in one text file"
  }
 ],
 "changed_at": "2026-09-05T13:18:46+00:00",
 "recent_changes": [
  "2026-09-05T13:18Z - changed: commands 10->22, sections; 9431->11854 bytes",
  "2026-09-05T08:45Z - changed: links",
  "2026-09-04T20:44Z - changed: sections, links"
 ]
}