{
 "name": "MCP server — Crawler Log Triage",
 "what": "Paste raw access-log lines into an MCP tool: every crawler identified and attributed, the impersonators named against operator-published IP ranges, and robots.txt or WAF rules for exactly what you saw. Streamable HTTP at /mcp/triage, no key, no signup.",
 "url": "https://www.pathwren.workers.dev/mcp-triage.json",
 "twin_of": "https://www.pathwren.workers.dev/mcp-triage.html",
 "page": {
  "path": "/mcp-triage.html",
  "url": "https://www.pathwren.workers.dev/mcp-triage.html",
  "type": "text/html"
 },
 "generated_at": "2026-09-11T21:25:22+00:00",
 "generated_from": "the bytes of /mcp-triage.html, by surfaces/ai-crawler-index/build.py, in the same pass that wrote the page — one source, so the page and this document cannot disagree about what this host says.",
 "license": {
  "document": "CC0-1.0",
  "url": "https://creativecommons.org/publicdomain/zero/1.0/"
 },
 "access": {
  "api_key": "none",
  "account": "none",
  "rate_limit": "none",
  "cors": "*",
  "auth": "none — every document here is a public GET"
 },
 "commands": [
  "https://www.pathwren.workers.dev/mcp/triage",
  "no auth, read-only, public",
  "{",
  "  \"mcpServers\": {",
  "    \"crawler-log-triage\": {",
  "      \"type\": \"streamable-http\",",
  "      \"url\": \"https://www.pathwren.workers.dev/c/mcp-connector/mcp/triage\"",
  "    }",
  "  }",
  "}",
  "claude mcp add --transport http crawler-log-triage https://www.pathwren.workers.dev/c/mcp-connector/mcp/triage",
  "curl -s https://www.pathwren.workers.dev/c/mcp-connector/mcp/triage \\",
  "  -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' \\",
  "  -d '{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"tools/call\",\"params\":{\"name\":\"no_arguments_triage_this_hosts_own_crawler_log\",\"arguments\":{}}}' \\",
  "  | jq -r '.result.content[0].text' | head -3",
  "This host's own published log, triaged: 54761 requests over 400 distinct user-agent strings, 31% identified against the index.",
  "(unidentified): 37793 requests (69%)",
  "Anthropic: 6847 requests (12.5%)",
  "# three lines of a log, one call, no account",
  "curl -s https://www.pathwren.workers.dev/mcp/triage \\",
  "  -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' \\",
  "  -d '{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"tools/call\",\"params\":{\"name\":\"triage_log\",",
  "       \"arguments\":{\"log\":\"66.249.66.1 Googlebot/2.1\\n203.0.113.9 Googlebot/2.1\\n412 GPTBot/1.2\"}}}' \\",
  "  | jq -r '.result.structuredContent.rows[] | \"\\(.identified)\\t\\(.operator)\\t\\(.verdict)\"'",
  "Googlebot\tGoogle\tverified",
  "Googlebot\tGoogle\timpersonation",
  "GPTBot\tOpenAI\tunverifiable",
  "curl -s https://www.pathwren.workers.dev/mcp/triage \\",
  "  -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' \\",
  "  -d \"$(jq -Rs '{jsonrpc:\"2.0\",id:1,method:\"tools/call\",",
  "       params:{name:\"summarize_by_operator\",arguments:{log:.}}}' < access.log)\" \\",
  "  | jq -r '.result.structuredContent.by_operator[] | \"\\(.requests)\\t\\(.operator)\\t\\(.share_pct)%\"'"
 ],
 "sections": [
  {
   "heading": "Crawler Log Triage — MCP server",
   "text": [],
   "commands": [],
   "tables": [],
   "links": []
  },
  {
   "heading": "Connect it: one paste, one call",
   "text": [
    "That is the whole endpoint and those are its terms: Streamable HTTP (MCP), no API key, no account, no OAuth, no session to keep alive, nothing to install. Every tool is read-only, and none of them will fetch a URL on your behalf.",
    "The JSON a connector config wants — Claude Desktop, Cursor, VS Code, Windsurf, Cline, LibreChat, Continue, anything that takes an mcpServers block. Complete as it stands; there is no field to fill in:",
    "Claude Code takes one line instead:",
    "The URL in those three boxes carries /c/mcp-connector/, a channel tag: it is the same endpoint by another path, serving byte-identical responses, and it lets this host see that a client arrived from a config pasted off this page rather than from a directory. Strip the prefix and https://www.pathwren.workers.dev/mcp/triage is the canonical URL — both work, and nothing about the answer changes.",
    "Then call no_arguments_triage_this_hosts_own_crawler_log first. It takes no arguments at all, so there is nothing to invent and nothing to look up before you can see this server work — the subject of the answer is this host's own published crawler log, run through the real engine:",
    "Those are the answer's own first three lines, from one real run on 2026-09-06 — the log grows, so run the curl and read today's. The tool takes no arguments because its subject is not you: it is the request log THIS host publishes, so every caller gets the same bytes. It is the same parser, the same crawler index and the same operator-prefix verification that triage_log runs on a file you paste — including the 69% it cannot identify, which is the number a demo would have hidden. The rest of the answer is the rollup by operator, by category and by crawler, the log it built line by line so you can re-run it yourself, and the browser-shaped strings kept apart from the crawlers. Nothing is fetched to answer it, no argument exists. triage_my_request is the second zero-argument call and answers about YOU — your own requests to this host, triaged the same way; whoami is the third and works unchanged on every MCP server here; example is the fourth. All four are safe first calls.",
    "Paste your access log. Get back what is crawling you, who runs it, what it costs to block, and which lines are lying about who they are. Streamable HTTP, stateless, no key, no signup. Every tool is read-only.",
    "With a real file, the shape you actually want:"
   ],
   "commands": [
    "https://www.pathwren.workers.dev/mcp/triage",
    "no auth, read-only, public",
    "{",
    "  \"mcpServers\": {",
    "    \"crawler-log-triage\": {",
    "      \"type\": \"streamable-http\",",
    "      \"url\": \"https://www.pathwren.workers.dev/c/mcp-connector/mcp/triage\"",
    "    }",
    "  }",
    "}",
    "claude mcp add --transport http crawler-log-triage https://www.pathwren.workers.dev/c/mcp-connector/mcp/triage",
    "curl -s https://www.pathwren.workers.dev/c/mcp-connector/mcp/triage \\",
    "  -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' \\",
    "  -d '{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"tools/call\",\"params\":{\"name\":\"no_arguments_triage_this_hosts_own_crawler_log\",\"arguments\":{}}}' \\",
    "  | jq -r '.result.content[0].text' | head -3",
    "This host's own published log, triaged: 54761 requests over 400 distinct user-agent strings, 31% identified against the index.",
    "(unidentified): 37793 requests (69%)",
    "Anthropic: 6847 requests (12.5%)",
    "# three lines of a log, one call, no account",
    "curl -s https://www.pathwren.workers.dev/mcp/triage \\",
    "  -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' \\",
    "  -d '{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"tools/call\",\"params\":{\"name\":\"triage_log\",",
    "       \"arguments\":{\"log\":\"66.249.66.1 Googlebot/2.1\\n203.0.113.9 Googlebot/2.1\\n412 GPTBot/1.2\"}}}' \\",
    "  | jq -r '.result.structuredContent.rows[] | \"\\(.identified)\\t\\(.operator)\\t\\(.verdict)\"'",
    "Googlebot\tGoogle\tverified",
    "Googlebot\tGoogle\timpersonation",
    "GPTBot\tOpenAI\tunverifiable",
    "curl -s https://www.pathwren.workers.dev/mcp/triage \\",
    "  -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' \\",
    "  -d \"$(jq -Rs '{jsonrpc:\"2.0\",id:1,method:\"tools/call\",",
    "       params:{name:\"summarize_by_operator\",arguments:{log:.}}}' < access.log)\" \\",
    "  | jq -r '.result.structuredContent.by_operator[] | \"\\(.requests)\\t\\(.operator)\\t\\(.share_pct)%\"'"
   ],
   "tables": [],
   "links": []
  },
  {
   "heading": "Tools",
   "text": [],
   "commands": [],
   "tables": [
    {
     "headers": [
      "Tool",
      "What it answers"
     ],
     "rows": [
      [
       "no_arguments_triage_this_hosts_own_crawler_log",
       "Takes no arguments, and the name says so. The whole job above, run on the request log THIS host publishes: the same parser, index and prefix verification, rolled up by operator, category and crawler, with the share that matches nothing named rather than hidden."
      ],
      [
       "triage_log",
       "The whole job. Per line: which crawler, whose, what for, what blocking costs — plus a rollup by operator and category and every line whose address contradicts its claim."
      ],
      [
       "find_impersonators",
       "Only the lines claiming a crawler whose operator publishes ranges, from an address in none of them. Checked against 1992 IPv4 and 1062 IPv6 prefixes from 15 operator sources."
      ],
      [
       "summarize_by_operator",
       "Aggregate only: who crawled you, how many requests, what share, which category, what blocking each would cost. Eats a uniq -c table straight from a shell pipeline."
      ],
      [
       "robots_from_log",
       "A robots.txt naming only the crawlers your log actually contains — not a generic list of everything that exists — with a warning for any that do not documentably obey it."
      ],
      [
       "waf_ruleset_from_log",
       "nginx, Caddy, Cloudflare, HAProxy or Apache rules for the same set. The enforcement half, for the crawlers that ignore robots.txt."
      ]
     ]
    }
   ],
   "links": []
  },
  {
   "heading": "What it eats",
   "text": [
    "Formats are detected per line, so a messy paste is fine and a mixed file is fine:",
    "The usual way to produce the third one: awk -F'\"' '{print $6}' access.log | sort | uniq -c | sort -rn | head -50"
   ],
   "commands": [],
   "tables": [
    {
     "headers": [
      "Shape",
      "Example"
     ],
     "rows": [
      [
       "combined / common",
       "66.249.66.1 - - [01/Sep/2026:00:14:02 +0000] \"GET / HTTP/1.1\" 200 512 \"-\" \"…Googlebot/2.1…\""
      ],
      [
       "JSON lines",
       "{\"remote_addr\":\"66.249.66.1\",\"http_user_agent\":\"…Googlebot/2.1…\"}"
      ],
      [
       "uniq -c table",
       "412 Mozilla/5.0 … GPTBot/1.2 … — the count is respected, so 412 lines cost you one"
      ],
      [
       "ip + user-agent",
       "66.249.66.1 Googlebot/2.1"
      ],
      [
       "bare user-agent",
       "Mozilla/5.0 (compatible; ClaudeBot/1.0; …)"
      ]
     ]
    }
   ],
   "links": []
  },
  {
   "heading": "How is this different from /mcp?",
   "text": [
    "ai-crawler-index answers questions about one thing — this user-agent, this IP, this crawler, this stance. It is a reference. This server does not answer questions about crawlers at all: it takes your evidence, a log you already have, and gives back a decision about it. One call in, per-line verdicts plus a rollup plus a config out. Different input, different unit of work, no tool in common — same index underneath, so the two agree by construction. Run both."
   ],
   "commands": [],
   "tables": [],
   "links": [
    "/mcp.html"
   ]
  },
  {
   "heading": "What it will not do",
   "text": [
    "It has no side effects and no write path. It has no tool that fetches a URL, and no tool that takes one — it works on text you paste, and nothing else. That is deliberate: a log-analysis tool that will fetch an arbitrary URL for whoever is talking to it is an SSRF proxy with a friendly description, so this one cannot be asked.",
    "Operators who verify by reverse DNS (24 of them, including Bing, Apple and DuckDuckGo) cannot be checked without a lookup, so instead of doing one it hands you the exact dig command and says the result is unknown until you run it. A user-agent is a claim, not a proof; every answer says which of the two you are holding.",
    "Protocol versions 2025-06-18, negotiated per call. server/discover answers for clients on 2026-07-28, initialize for everyone else. Same index as /data/agents.json, refreshed every six hours. Listed in the official MCP Registry as dev.workers.pathwren.www/crawler-log-triage."
   ],
   "commands": [],
   "tables": [],
   "links": [
    "/data/agents.json",
    "https://registry.modelcontextprotocol.io/v0/servers?search=crawler-log-triage"
   ]
  }
 ],
 "machine_doors": [
  {
   "url": "https://www.pathwren.workers.dev/tools/?s=client-dossiers",
   "name": "6 keyless GET tools",
   "what": "The read-only MCP tools of this host as plain GET endpoints — no JSON-RPC, no key"
  },
  {
   "url": "https://www.pathwren.workers.dev/documents.json",
   "name": "documents.json",
   "what": "Every document here with its strong ETag and the date its bytes changed"
  },
  {
   "url": "https://www.pathwren.workers.dev/changes",
   "name": "changes",
   "what": "What moved since your cursor — poll this instead of re-downloading anything"
  },
  {
   "url": "https://www.pathwren.workers.dev/llms.txt",
   "name": "llms.txt",
   "what": "The whole map in one text file"
  },
  {
   "url": "https://www.pathwren.workers.dev/openapi.json",
   "name": "openapi.json",
   "what": "Every read endpoint, described formally"
  },
  {
   "url": "https://www.pathwren.workers.dev/.well-known/agent-card.json",
   "name": "agent card",
   "what": "A2A agent card"
  },
  {
   "url": "https://www.pathwren.workers.dev/mcp",
   "name": "mcp",
   "what": "MCP over JSON-RPC (POST)"
  },
  {
   "url": "https://www.pathwren.workers.dev/a2a",
   "name": "a2a",
   "what": "A2A (POST message/send)"
  }
 ],
 "links": [
  {
   "rel": "self",
   "href": "https://www.pathwren.workers.dev/mcp-triage.json",
   "type": "application/json"
  },
  {
   "rel": "describes",
   "href": "https://www.pathwren.workers.dev/mcp-triage.html",
   "type": "text/html",
   "title": "The page this document is the JSON twin of: MCP server — Crawler Log Triage"
  },
  {
   "rel": "changes",
   "href": "https://www.pathwren.workers.dev/changes.json?since=159",
   "type": "application/json",
   "title": "What changed since your cursor — poll this instead of re-downloading this document",
   "cursor_param": "since",
   "head_cursor": 159,
   "min_poll_seconds": 21600,
   "how": "Read `cursor` from the response and send it back as `since`. It advances only when something really changed, so an unchanged answer is proof rather than luck — about 2.5 KB, or a 304 with no body if you send back the ETag."
  },
  {
   "rel": "related",
   "href": "https://www.pathwren.workers.dev/documents.json",
   "type": "application/json",
   "title": "Every document here with its ETag and last-modified date"
  },
  {
   "rel": "related",
   "href": "https://www.pathwren.workers.dev/data/agents.json",
   "type": "application/json",
   "title": "Every crawler record in one file"
  },
  {
   "rel": "alternate",
   "href": "https://www.pathwren.workers.dev/sitemap.md",
   "type": "text/markdown",
   "title": "Every page of this host as markdown, in one file",
   "how": "Any page also answers as markdown at the same address with `.md` — and at `.mdx`, `<page>.html.md` and `<page>.html.mdx`, which are the same bytes. `Accept: text/markdown` on the page itself returns the same document. The HTML page stays canonical and every mirror says so in a Link header."
  },
  {
   "rel": "service-desc",
   "href": "https://www.pathwren.workers.dev/openapi.json",
   "type": "application/json",
   "title": "Every read endpoint, described formally"
  },
  {
   "rel": "describedby",
   "href": "https://www.pathwren.workers.dev/llms.txt",
   "type": "text/plain",
   "title": "The whole map in one text file"
  }
 ]
}