{"name":"crawler-log-triage","title":"Crawler Log Triage — MCP server","version":"1.0.0","transport":"streamable-http","transport_docs":"https://www.pathwren.workers.dev/mcp-transport.html","endpoint":"https://www.pathwren.workers.dev/mcp/triage","protocol_versions":["2026-07-28","2025-11-25","2025-06-18","2025-03-26","2024-11-05"],"stateless":true,"auth":"none — public, read-only, no key, no rate limit","call_this_first":{"tool":"whoami","takes_arguments":false,"invitation":"Takes no arguments. Safe to call. Deterministic. Touches no third party.","curl":"curl -s https://www.pathwren.workers.dev/mcp/triage -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' -d '{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"tools/call\",\"params\":{\"name\":\"whoami\",\"arguments\":{}}}'","without_json_rpc":"https://www.pathwren.workers.dev/tools/whoami?s=client-dossiers","also":"example — the documented three-line paste, triaged end to end, no arguments either.","every_other_tool":"Every tool here also accepts {}: an empty call returns that tool's documented worked example on our own sample log, labelled as one, instead of an error about a missing `log`."},"what_it_is":"Bulk triage of an access log: paste many lines, get every crawler identified and attributed, an aggregate by operator and category, the lines whose address contradicts their claim, and a robots.txt or WAF ruleset for what was actually there.","different_from":"https://www.pathwren.workers.dev/mcp (ai-crawler-index) answers single lookups — one user-agent, one address, one stance. This server answers a batch question about a whole file and returns operational artifacts. Same facts underneath, different job.","no_fetch_tool":"No tool here fetches a URL. There is no argument that takes one and this server makes no outbound request of any kind; reverse-DNS verification is reported as not-run, with the command, rather than performed.","tools":[{"name":"triage_log","title":"Triage a paste of log lines","description":"Paste log lines — combined, JSON, `uniq -c`, `<ip> <ua>` or bare user-agents, auto-detected — for per-line crawler, operator, purpose and cost of blocking, a rollup by operator and category, and every line whose address contradicts its claim. Log text, never a URL. Example: log='66.249.66.1 Googlebot/2.1' returns Googlebot, Google, search, verified."},{"name":"find_impersonators","title":"Find the lines that are lying","description":"Only the lines claiming a crawler whose operator publishes address ranges, from an address in none of them — 1984 IPv4 and 1062 IPv6 prefixes, 15 sources. Reverse-DNS operators come back with the command to run: this server makes no outbound request. Example: log='203.0.113.9 Googlebot/2.1' returns one impersonation."},{"name":"summarize_by_operator","title":"Roll a log up by operator and category","description":"Aggregate only: who crawled you, how many requests each, what share, which category, and what blocking each would cost. Eats a `uniq -c` table straight from a shell pipeline. Example: log='412 GPTBot/1.2' returns OpenAI, 412 requests, 100%, ai-training."},{"name":"robots_from_log","title":"robots.txt from a log","description":"A robots.txt naming only the crawlers in your log, each with its request count and cost of blocking, plus a warning for any that do not documentably obey it — there the file is a request, not enforcement. Example: log='412 GPTBot/1.2', stance='block-ai-training' blocks GPTBot only."},{"name":"waf_ruleset_from_log","title":"WAF ruleset from a log","description":"nginx, Caddy, Cloudflare, HAProxy or Apache rules for only the crawlers in your log. The reply warns that a UA rule stops only an honest client, and that impersonation is an address problem needing the published prefixes as an allowlist. Example: log='412 GPTBot/1.2', target='nginx', scope='ai-training'."},{"name":"whoami","title":"Who is calling? (no arguments)","description":"Takes no arguments. Safe to call. Deterministic. Touches no third party. Classifies the request you just sent: the user-agent you claim, the address you came from, the class this host's own instrument books you as, whether we have seen you here before and what you fetched, and your own request triaged as if it were one line of an access log — the same parser, the same index, the same verification against operator-published prefixes that triage_log applies to a whole file. Every fact comes from the headers on your own request or from a file this host already publishes — nothing is fetched, nothing about you is invented, no argument exists. Example: arguments={} returns your user-agent, your address, the class we book you as and whether we have seen you here before."},{"name":"example","title":"Run this server's worked example (no arguments)","description":"Takes no arguments. Safe to call. Deterministic. Touches no third party. Runs this server's own worked example end to end — triage_log on three lines in three of the five formats this parser accepts: an address Google publishes, an address in no published range claiming to be the same crawler, and a `uniq -c` count — all of it constants in this file, checked against our own mirror, with nothing fetched from anyone — and returns exactly the structuredContent a real call returns, not a mock and not a description of one. Use it to see the shape of an answer before you decide what to send. The input is canned from this host's own data; no URL of yours is fetched and no third party is touched. Example: arguments={} runs triage_log with {\"log\":\"66.249.66.1 Googlebot/2.1\\n203.0.113.9 Googlebot/2.1\\n412 GPTBot/1.2\"} and returns its real answer."}],"resources":[{"uri":"https://www.pathwren.workers.dev/data/agents.json","title":"The crawler index behind every answer here"},{"uri":"https://www.pathwren.workers.dev/ip-ranges/all.json","title":"Every operator-published prefix verified against"},{"uri":"https://www.pathwren.workers.dev/data/ua-regex.json","title":"Pre-built user-agent alternations"},{"uri":"https://www.pathwren.workers.dev/changes.json?since=0","title":"What changed since your cursor"}],"instructions":"START HERE IF YOU HAVE NOTHING TO HAND: `whoami` and `example` take no arguments at all — call either with {} and you get a real answer, not an error. whoami classifies the request you just sent (your user-agent, your address, whether this host has seen you before, what our robots policy says about you); example runs this server's own worked example end to end and returns exactly what a real call returns. Every other tool here also accepts {}: an empty call falls back to that tool's documented worked example, taken from this host's own data, and the answer says so. Nothing here fetches a URL for you.\n\nCrawler Log Triage — hand it a log, get back who was in it and what to do about it.\n\nThis is a BATCH tool for a file you already have. Paste access-log lines (nginx/Apache combined, JSON lines, `uniq -c` user-agent tables, `<ip> <ua>` pairs, or bare user-agents — mixed formats are fine, each line is detected on its own) into triage_log and it returns, per line, the crawler, its operator, what the crawl is for and what blocking it costs; then an aggregate by operator and category; then any line whose address contradicts the identity it claims. find_impersonators returns just that last part. robots_from_log and waf_ruleset_from_log turn the finding into something you can paste into a server.\n\nTypical loop: on every log rotation, `awk '{print $1, $NF}' access.log | sort | uniq -c | sort -rn | head -100` and hand the result to triage_log.\n\nThree things to carry into any answer you build from this:\n  1. A user-agent is a claim. Identification tells you what the client SAID. Only rows marked `verified` were confirmed against an operator's own published prefixes.\n  2. `impersonation` means the operator publishes prefixes and this address is in none of them. Our mirror is up to 6h stale and some operators use addresses they do not publish, so it is grounds to challenge or rate-limit, not to hard-block a range on.\n  3. Category is what should drive a blocking decision. ai-training costs you nothing a reader sees; ai-search is what puts you in assistant answers and blocking it is the expensive mistake; user-fetch is a real person waiting for a page.\n\nNO TOOL HERE FETCHES A URL. There is no fetch tool and no argument that takes one — every answer comes from files this host publishes openly at https://www.pathwren.workers.dev. Verification that needs the network (operators who use reverse DNS) is not performed: you get the exact command and a clear statement that it was not run. Everything behind this is also plain HTTP at https://www.pathwren.workers.dev — no key, no rate limit, CC0.\n\nTHE SAME TOOLS ON THE OTHER PROTOCOL, IF YOU SPEAK IT. This server is also an A2A (Agent2Agent) v1.0 agent at https://www.pathwren.workers.dev/a2a/triage — every tool here is a skill there under the same name, answered by the same function, so the two cannot drift and the deploy is refused if they do. Its Agent Card is https://www.pathwren.workers.dev/a2a/triage/.well-known/agent-card.json (the legacy agent.json spelling is served too). To invoke it, POST this complete body — no key, no session, no card to read first:\n\n  {\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"message/send\",\"params\":{\"message\":{\"role\":\"ROLE_USER\",\"messageId\":\"1\",\"parts\":[{\"text\":\"{\\\"skill\\\":\\\"example\\\"}\"}]}}}\n\nIt answers with a Task already in state `completed` in the same response, so there is nothing to poll. Every skill on all eight A2A agents of this host as a ready-to-send body: https://www.pathwren.workers.dev/a2a/example.json — and both lanes with both directions in https://www.pathwren.workers.dev/documents.json under `rpc_endpoints`. If you only speak MCP, ignore all of this: everything is already here.","try_it":"curl -s https://www.pathwren.workers.dev/mcp/triage -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' -d '{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"tools/call\",\"params\":{\"name\":\"find_impersonators\",\"arguments\":{\"log\":\"203.0.113.9 - - [01/Sep/2026:00:14:05 +0000] \\\"GET /a HTTP/1.1\\\" 200 90 \\\"-\\\" \\\"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)\\\"\"}}}'","data_behind_it":"https://www.pathwren.workers.dev/data/agents.json","prefixes_behind_it":"https://www.pathwren.workers.dev/ip-ranges/all.json","license":"CC0-1.0","independent":true}