{"name":"crawler-log-triage","title":"Crawler Log Triage — MCP server","version":"1.0.0","transport":"streamable-http","transport_docs":"https://www.pathwren.workers.dev/mcp-transport.html","endpoint":"https://www.pathwren.workers.dev/mcp/triage","protocol_versions":["2026-07-28","2025-11-25","2025-06-18","2025-03-26","2024-11-05"],"stateless":true,"auth":"none — public, read-only, no key, no rate limit","call_this_first":{"tool":"triage_my_request","takes_arguments":false,"body":{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"triage_my_request","arguments":{}}},"curl":"curl -s https://www.pathwren.workers.dev/mcp/triage -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' -d '{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"tools/call\",\"params\":{\"name\":\"triage_my_request\",\"arguments\":{}}}'","answers":"your own request triaged as one line of an access log — the crawler this host's index identifies from your user-agent, its operator and category, and whether the address you came from verifies against that operator's published prefixes","also":"whoami and example — the two zero-argument calls every server here carries, same empty arguments. triage_my_request is this server's own: only it answers your own request triaged as one line of an access log — the crawler this host's index identifies from your user-agent, its operator and category, and whether the address you came from verifies against that operator's published prefixes.","every_other_tool":"Every tool here also accepts {}: an empty call returns that tool's documented worked example on our own sample log, labelled as one, instead of an error about a missing `log`.","invitation":"Takes no arguments. Safe to call. Deterministic. Touches no third party.","without_json_rpc":"https://www.pathwren.workers.dev/tools/whoami?s=client-dossiers"},"what_it_is":"Bulk triage of an access log: paste many lines, get every crawler identified and attributed, an aggregate by operator and category, the lines whose address contradicts their claim, and a robots.txt or WAF ruleset for what was actually there.","different_from":"https://www.pathwren.workers.dev/mcp (ai-crawler-index) answers single lookups — one user-agent, one address, one stance. This server answers a batch question about a whole file and returns operational artifacts. Same facts underneath, different job.","no_fetch_tool":"No tool here fetches a URL. There is no argument that takes one and this server makes no outbound request of any kind; reverse-DNS verification is reported as not-run, with the command, rather than performed.","tools":[{"name":"triage_my_request","title":"Triage my own request (no arguments)","description":"{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"tools/call\",\"params\":{\"name\":\"triage_my_request\",\"arguments\":{}}} — POST that exact body to https://www.pathwren.workers.dev/mcp/triage and the answer is your own request triaged as one line of an access log — the crawler this host's index identifies from your user-agent, its operator and category, and whether the address you came from verifies against that operator's published prefixes. THIS IS THE CALL TO MAKE FIRST on this server: it is the only one whose subject is the request you just sent, so there is nothing to look up and no argument to invent. No key, no account, no OAuth, no session to open first, read-only, and nothing for you to invent. Every fact is computed from your own headers and from files this host already publishes; nothing is fetched to build it, and no URL of yours is touched. Only this server answers your own request triaged as one line of an access log, by the same parser and the same index that read a whole file — its five siblings answer one other question each about you, same empty arguments, each under the tool named beside its path: /mcp (whoami), /mcp/doctor (whoami), /mcp/lint (whoami), /mcp/robots (am_i_allowed), /mcp/netcheck (verify_my_address) on https://www.pathwren.workers.dev; whoami is answered by all six and is never an error, so one identical body still walks the whole family if you prefer that. Example: the complete call, exactly as written, nothing to fill in — {\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"tools/call\",\"params\":{\"name\":\"triage_my_request\",\"arguments\":{}}} returns the single log line built from your address and your user-agent, the triage row for it, and the same rollup triage_log returns for a whole file."},{"name":"whoami","title":"Who is calling? — the one call that needs nothing from you","description":"{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"tools/call\",\"params\":{\"name\":\"whoami\",\"arguments\":{}}} — POST that to https://www.pathwren.workers.dev/mcp/triage and the answer is the user-agent you sent, the class this host's own instrument books it as, whether we have logged you here before — and the part only this server can answer: the request you just made, run through triage_log as a single log line, naming the crawler it identifies, its operator and category, and whether your address verifies against that operator's published prefixes. Takes no arguments. Safe to call. Deterministic. Touches no third party. Nothing is fetched to build it: every fact comes from the headers on your own request or from a file this host already publishes. Only this server answers the last part — its five siblings answer one other question each about you, same empty arguments, each under the tool named beside its path: /mcp (whoami), /mcp/doctor (whoami), /mcp/lint (whoami), /mcp/robots (am_i_allowed), /mcp/netcheck (verify_my_address) on https://www.pathwren.workers.dev. Example: arguments={} returns your user-agent, your address, the class we book you as, whether we have seen you here before, and your own request triaged as one line of an access log, by the same parser and the same index that read a whole file."},{"name":"example","title":"Run this server's worked example (no arguments)","description":"{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"tools/call\",\"params\":{\"name\":\"example\",\"arguments\":{}}} — POST that exact body to https://www.pathwren.workers.dev/mcp/triage and you are done: free, read-only, no API key, no account, no consent prompt, nothing for you to invent, and what comes back is this server's own worked example — triage_log, run for real on a canned input from this host's own data — so you see the exact shape of an answer before you send anything of yours. Takes no arguments. Safe to call. Deterministic. Touches no third party. Runs this server's own worked example end to end — triage_log on three lines in three of the five formats this parser accepts: an address Google publishes, an address in no published range claiming to be the same crawler, and a `uniq -c` count — all of it constants in this file, checked against our own mirror, with nothing fetched from anyone — and returns exactly the structuredContent a real call returns, not a mock and not a description of one. Use it to see the shape of an answer before you decide what to send. The input is canned from this host's own data; no URL of yours is fetched and no third party is touched. Example: arguments={} runs triage_log with {\"log\":\"66.249.66.1 Googlebot/2.1\\n203.0.113.9 Googlebot/2.1\\n412 GPTBot/1.2\"} and returns its real answer."},{"name":"triage_log","title":"Triage a paste of log lines","description":"FIRST CALL, needs nothing: {\"name\":\"triage_my_request\",\"arguments\":{}} — Paste log lines — combined, JSON, `uniq -c`, `<ip> <ua>` or bare user-agents, auto-detected — for per-line crawler, operator, purpose and cost of blocking, a rollup by operator and category, and every line whose address contradicts its claim. Log text, never a URL. Example: log='66.249.66.1 Googlebot/2.1' returns Googlebot, Google, search, verified."},{"name":"find_impersonators","title":"Find the lines that are lying","description":"FIRST CALL, needs nothing: {\"name\":\"triage_my_request\",\"arguments\":{}} — Only the lines claiming a crawler whose operator publishes address ranges, from an address in none of them — 1987 IPv4 and 1062 IPv6 prefixes, 15 sources. Reverse-DNS operators come back with the command to run: this server makes no outbound request. Example: log='203.0.113.9 Googlebot/2.1' returns one impersonation."},{"name":"summarize_by_operator","title":"Roll a log up by operator and category","description":"FIRST CALL, needs nothing: {\"name\":\"triage_my_request\",\"arguments\":{}} — Aggregate only: who crawled you, how many requests each, what share, which category, and what blocking each would cost. Eats a `uniq -c` table straight from a shell pipeline. Example: log='412 GPTBot/1.2' returns OpenAI, 412 requests, 100%, ai-training."},{"name":"robots_from_log","title":"robots.txt from a log","description":"FIRST CALL, needs nothing: {\"name\":\"triage_my_request\",\"arguments\":{}} — A robots.txt naming only the crawlers in your log, each with its request count and cost of blocking, plus a warning for any that do not documentably obey it — there the file is a request, not enforcement. Example: log='412 GPTBot/1.2', stance='block-ai-training' blocks GPTBot only."},{"name":"waf_ruleset_from_log","title":"WAF ruleset from a log","description":"FIRST CALL, needs nothing: {\"name\":\"triage_my_request\",\"arguments\":{}} — nginx, Caddy, Cloudflare, HAProxy or Apache rules for only the crawlers in your log. The reply warns that a UA rule stops only an honest client, and that impersonation is an address problem needing the published prefixes as an allowlist. Example: log='412 GPTBot/1.2', target='nginx', scope='ai-training'."}],"resources":[{"uri":"https://www.pathwren.workers.dev/data/agents.json","title":"The crawler index behind every answer here"},{"uri":"https://www.pathwren.workers.dev/ip-ranges/all.json","title":"Every operator-published prefix verified against"},{"uri":"https://www.pathwren.workers.dev/data/ua-regex.json","title":"Pre-built user-agent alternations"},{"uri":"https://www.pathwren.workers.dev/changes.json?since=0","title":"What changed since your cursor"}],"instructions":"{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"tools/call\",\"params\":{\"name\":\"triage_my_request\",\"arguments\":{}}}\n\nPOST that exact body to https://www.pathwren.workers.dev/mcp/triage and it answers with your own request triaged as one line of an access log — the crawler this host's index identifies from your user-agent, its operator and category, and whether the address you came from verifies against that operator's published prefixes. It is the first call to make here — no key, no account, no OAuth, read-only, and no argument to invent: every fact in the answer is computed from the request you just sent and from files this host already publishes, and nothing is fetched to build it.\n\n{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"tools/call\",\"params\":{\"name\":\"whoami\",\"arguments\":{}}}\n\nPOST that exact body to https://www.pathwren.workers.dev/mcp/triage and it answers with the user-agent you sent, the class this host's own instrument books it as, whether we have logged you here before — and the part only this server can answer: the request you just made, run through triage_log as a single log line, naming the crawler it identifies, its operator and category, and whether your address verifies against that operator's published prefixes.\n\nIt is free, it needs no key, no account, no session id and no argument at all: every fact in the answer is computed from the request you just sent and from files this host already publishes, and nothing is fetched from anywhere to build it.\n\nONLY THIS SERVER ANSWERS THAT PART. There are six MCP servers here and they answer one question each about you — one zero-argument call each, no key, no account, no session. The middle column is the tool to name at that URL, because three of the six answer under a name of their own; whoami is answered by all six and is never an error, so one identical body still walks the whole family if you prefer that.\n  https://www.pathwren.workers.dev/mcp/triage    triage_my_request  your own request triaged as one line of an access log, by the same parser and the same index that read a whole file  <- you are here\n  https://www.pathwren.workers.dev/mcp           whoami             what this host's crawler index makes of the user-agent you sent, and whether your address sits inside a prefix its operator publishes\n  https://www.pathwren.workers.dev/mcp/doctor    whoami             which agent-discovery documents this host has watched a client like yours ask for — the dates, and the status each request took\n  https://www.pathwren.workers.dev/mcp/lint      whoami             your own client's record on this host — every surface of ours your user-agent has been seen on, with the status codes it took\n  https://www.pathwren.workers.dev/mcp/robots    am_i_allowed       what this host's own robots.txt says to YOU — the group that matches your token, the line that decided it, and whether you are allowed the path you are on\n  https://www.pathwren.workers.dev/mcp/netcheck  verify_my_address  whether the address you are calling from falls inside a prefix a crawler operator publishes, and which prefix from which source\n\nALSO ON THIS SERVER: triage_log takes a pasted access log in five formats; find_impersonators, summarize_by_operator, robots_from_log and waf_ruleset_from_log turn the finding into something you can paste into a server. Each of those takes an argument, and each also accepts arguments {} — an empty call returns that tool's own worked example, from this host's data, instead of an error.\n\nTHE SAME TOOLS ON THE OTHER PROTOCOL, IF YOU SPEAK IT. This server is also an A2A (Agent2Agent) v1.0 agent at https://www.pathwren.workers.dev/a2a/triage — every tool here is a skill there under the same name, answered by the same function, so the two cannot drift and the deploy is refused if they do. Its Agent Card is https://www.pathwren.workers.dev/a2a/triage/.well-known/agent-card.json (the legacy agent.json spelling is served too). To invoke it, POST this complete body — no key, no session, no card to read first:\n\n  {\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"message/send\",\"params\":{\"message\":{\"role\":\"ROLE_USER\",\"messageId\":\"1\",\"parts\":[{\"text\":\"{\\\"skill\\\":\\\"example\\\"}\"}]}}}\n\nIt answers with a Task already in state `completed` in the same response, so there is nothing to poll. Every skill on all eight A2A agents of this host as a ready-to-send body: https://www.pathwren.workers.dev/a2a/example.json — and both lanes with both directions in https://www.pathwren.workers.dev/documents.json under `rpc_endpoints`. If you only speak MCP, ignore all of this: everything is already here.","try_it":"curl -s https://www.pathwren.workers.dev/mcp/triage -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' -d '{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"tools/call\",\"params\":{\"name\":\"find_impersonators\",\"arguments\":{\"log\":\"203.0.113.9 - - [01/Sep/2026:00:14:05 +0000] \\\"GET /a HTTP/1.1\\\" 200 90 \\\"-\\\" \\\"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)\\\"\"}}}'","data_behind_it":"https://www.pathwren.workers.dev/data/agents.json","prefixes_behind_it":"https://www.pathwren.workers.dev/ip-ranges/all.json","license":"CC0-1.0","independent":true}