{
 "name": "robots.txt Policy Lint — AI Crawler Index",
 "description": "Take a robots.txt you already have and say what it actually does. It lints the file against RFC 9309 and reports the errors that silently change meaning, answers whether a named crawler may fetch a named path and which rule decided it, audits which AI crawlers the file really stops (and which it only appears to), diffs two versions by EFFECT rather than by line, and merges a ready-made stance into an existing file without discarding the rules already there. It reads the file you paste; it fetches nothing. robots.txt is a request, not an enforcement mechanism, and the audit says so where a crawler is known to ignore it. Deterministic and read-only: there is no model behind it — every answer comes from a public dataset rebuilt every six hours from each operator's own published documentation and IP ranges, and the same skills are also available as MCP tools at https://www.pathwren.workers.dev/mcp/robots. No key, no signup, no quota. Independent and unaffiliated with any operator it documents. TO CALL IT: send `message/send` (v1.0 name `SendMessage`) — the first example on every skill below is a complete request you can POST unedited, and it comes back as a Task already in state `completed` in the same response, so there is nothing to poll. Nothing to hand over? The `whoami` skill takes no arguments and answers about you. Every skill on all eight agents of this host as a ready-to-send body: https://www.pathwren.workers.dev/a2a/example.json",
 "supportedInterfaces": [
  {
   "url": "https://www.pathwren.workers.dev/a2a/robots",
   "protocolBinding": "JSONRPC",
   "protocolVersion": "1.0"
  }
 ],
 "url": "https://www.pathwren.workers.dev/a2a/robots",
 "preferredTransport": "JSONRPC",
 "protocolVersion": "1.0",
 "provider": {
  "organization": "Pathwren",
  "url": "https://www.pathwren.workers.dev"
 },
 "version": "1.0.0",
 "documentationUrl": "https://www.pathwren.workers.dev/a2a.html",
 "iconUrl": "https://www.pathwren.workers.dev/icon.png",
 "capabilities": {
  "streaming": false,
  "pushNotifications": false,
  "extendedAgentCard": false,
  "extensions": [
   {
    "uri": "https://www.pathwren.workers.dev/changes.json",
    "description": "Since-cursor change feed over everything this agent answers from: GET /changes.json?since=<cursor> returns only what moved — operator IP-range lists that gained or lost prefixes, upstreams that failed or recovered, crawler records added or edited. Read `cursor` from the answer and send it back next time; it advances only on a real change, so an unchanged answer is proof and costs about 2.5 KB. The same feed is the changes_since skill on this endpoint.",
    "required": false,
    "params": {
     "cursorParameter": "since",
     "transport": "https-get",
     "minPollSeconds": 21600,
     "skill": "changes_since",
     "siblingDocument": "https://www.pathwren.workers.dev/data/agents.json"
    }
   },
   {
    "uri": "https://www.pathwren.workers.dev/mcp/robots",
    "description": "THE SAME AGENT ON THE OTHER PROTOCOL. Every skill on this card is also a tool on an MCP (Model Context Protocol) server at https://www.pathwren.workers.dev/mcp/robots — same name, same arguments, same answer, because one function answers both doors and a deploy that let them drift is refused. JSON-RPC 2.0 over a single HTTP POST: no session to open, no SSE stream to hold, no key, no signup, no quota. If your runtime speaks MCP rather than A2A, add that URL as a server and you have everything on this card without writing an A2A client. `params.initialize` and `params.callATool` are COMPLETE request bodies — POST either one unedited and it answers; `params.curl` is the same thing as one line. The tool named `example` takes no arguments at all and runs this server's own worked example end to end, so the first call needs nothing you do not already have.",
    "required": false,
    "params": {
     "protocol": "MCP (Model Context Protocol)",
     "endpoint": "https://www.pathwren.workers.dev/mcp/robots",
     "transport": "streamable-http (JSON-RPC 2.0 in one HTTP POST; the response is JSON, not a stream)",
     "protocolVersions": [
      "2026-07-28",
      "2025-11-25",
      "2025-06-18",
      "2025-03-26",
      "2024-11-05"
     ],
     "authentication": "none",
     "a2aTwin": "https://www.pathwren.workers.dev/a2a/robots",
     "toolNamesEqualSkillIds": true,
     "sameImplementation": "Each skill on this card and the tool of the same name on that server are answered by one function; the deploy gate compares them skill by skill.",
     "initialize": {
      "jsonrpc": "2.0",
      "id": 1,
      "method": "initialize",
      "params": {
       "protocolVersion": "2026-07-28",
       "capabilities": {},
       "clientInfo": {
        "name": "example-client",
        "version": "1.0.0"
       }
      }
     },
     "callATool": {
      "jsonrpc": "2.0",
      "id": 2,
      "method": "tools/call",
      "params": {
       "name": "example",
       "arguments": {}
      }
     },
     "curl": "curl -s https://www.pathwren.workers.dev/mcp/robots -H 'content-type: application/json' -d '{\"jsonrpc\":\"2.0\",\"id\":2,\"method\":\"tools/call\",\"params\":{\"name\":\"example\",\"arguments\":{}}}'",
     "everyToolAsABody": "https://www.pathwren.workers.dev/tools/",
     "discovery": "https://www.pathwren.workers.dev/.well-known/mcp.json"
    }
   }
  ]
 },
 "defaultInputModes": [
  "application/json",
  "text/plain"
 ],
 "defaultOutputModes": [
  "application/json",
  "text/plain"
 ],
 "skills": [
  {
   "id": "lint_robots_txt",
   "name": "Lint a robots.txt",
   "description": "Parse a robots.txt you paste and report every fault that makes it do something other than what it looks like: misspelled directives, a full UA string where a product token belongs, rules before any User-agent line, duplicate groups, noindex (unsupported since 2019), relative Sitemap URLs, BOM. Each finding carries the line number and the fix. Example: robots_txt='User-agent: GPTBot\\nDisallow: /\\n\\nUser-agent: *\\nAllow: /\\n' — paste the whole file, it is never fetched for you. Also callable without MCP, same implementation: GET https://www.pathwren.workers.dev/tools/robots-lint?robots_txt=<urlencoded>&s=client-dossiers — or POST the file as the raw body to the same URL.",
   "tags": [
    "robots.txt",
    "lint",
    "rfc9309",
    "validation"
   ],
   "examples": [
    "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"message/send\",\"params\":{\"message\":{\"role\":\"ROLE_USER\",\"messageId\":\"1\",\"parts\":[{\"text\":\"{\\\"skill\\\":\\\"lint_robots_txt\\\",\\\"robots_txt\\\":\\\"User-agent: GPTBot\\\\nDisallow: /\\\\n\\\\nUser-agent: *\\\\nAllow: /\\\\n\\\"}\"}]}}}",
    "{\"skill\":\"lint_robots_txt\",\"robots_txt\":\"User-agent: GPTBot\\nDisallow: /\\n\\nUser-agent: *\\nAllow: /\\n\"}",
    "User-agent: GPTBot\nDisallow: /"
   ],
   "inputModes": [
    "application/json",
    "text/plain"
   ],
   "outputModes": [
    "application/json",
    "text/plain"
   ]
  },
  {
   "id": "check_path_allowed",
   "name": "Would this crawler fetch this path?",
   "description": "Evaluate a pasted robots.txt for one crawler and one or more paths under RFC 9309: longest token match for the group, longest path pattern for the rule, Allow breaking a tie, * and $ supported. Returns allowed/disallowed per path with the exact line that decided it, and flags the cases where a merge-groups parser and a first-group-wins parser would disagree. Example: user_agent='GPTBot', paths=['/', '/blog'], with your robots_txt pasted in. Also callable without MCP, same implementation: GET https://www.pathwren.workers.dev/tools/robots-allowed?robots_txt=<urlencoded>&ua=GPTBot&path=/blog&s=client-dossiers",
   "tags": [
    "robots.txt",
    "crawlers",
    "path",
    "rules"
   ],
   "examples": [
    "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"message/send\",\"params\":{\"message\":{\"role\":\"ROLE_USER\",\"messageId\":\"1\",\"parts\":[{\"text\":\"{\\\"skill\\\":\\\"check_path_allowed\\\",\\\"robots_txt\\\":\\\"User-agent: GPTBot\\\\nDisallow: /\\\\n\\\\nUser-agent: *\\\\nAllow: /\\\\n\\\",\\\"user_agent\\\":\\\"GPTBot\\\",\\\"paths\\\":[\\\"/docs\\\",\\\"/\\\"]}\"}]}}}",
    "{\"skill\":\"check_path_allowed\",\"robots_txt\":\"User-agent: GPTBot\\nDisallow: /\\n\\nUser-agent: *\\nAllow: /\\n\",\"user_agent\":\"GPTBot\",\"paths\":[\"/docs\",\"/\"]}"
   ],
   "inputModes": [
    "application/json",
    "text/plain"
   ],
   "outputModes": [
    "application/json",
    "text/plain"
   ]
  },
  {
   "id": "audit_ai_access",
   "name": "Which AI crawlers does this file actually stop?",
   "description": "Evaluate a pasted robots.txt against every AI crawler in this index and return the two lists that matter: blocked and allowed, per operator and category. Also names the tokens in your file that match no known crawler (a typo blocks nothing) and separates the crawlers that document obedience from the ones observed ignoring robots.txt, which need an IP or WAF rule instead. Example: path='/' with your robots_txt pasted in — the verdict is per crawler, at that path. Also callable without MCP, same implementation: GET https://www.pathwren.workers.dev/tools/ai-access?robots_txt=<urlencoded>&s=client-dossiers",
   "tags": [
    "robots.txt",
    "ai crawlers",
    "audit",
    "policy"
   ],
   "examples": [
    "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"message/send\",\"params\":{\"message\":{\"role\":\"ROLE_USER\",\"messageId\":\"1\",\"parts\":[{\"text\":\"{\\\"skill\\\":\\\"audit_ai_access\\\",\\\"robots_txt\\\":\\\"User-agent: GPTBot\\\\nDisallow: /\\\\n\\\\nUser-agent: *\\\\nAllow: /\\\\n\\\"}\"}]}}}",
    "{\"skill\":\"audit_ai_access\",\"robots_txt\":\"User-agent: GPTBot\\nDisallow: /\\n\\nUser-agent: *\\nAllow: /\\n\"}"
   ],
   "inputModes": [
    "application/json",
    "text/plain"
   ],
   "outputModes": [
    "application/json",
    "text/plain"
   ]
  },
  {
   "id": "diff_robots_txt",
   "name": "Diff two robots.txt by effect",
   "description": "Compare two versions of a robots.txt and report only the crawlers whose verdict actually changes at a given path — not the text difference. Answers 'did my edit do what I meant, and did it do anything else', including sitemap additions and whether the parse errors went up or down. Example: before='User-agent: *\\nAllow: /\\n', after=your edited file, path='/'.",
   "tags": [
    "robots.txt",
    "diff",
    "review",
    "change"
   ],
   "examples": [
    "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"message/send\",\"params\":{\"message\":{\"role\":\"ROLE_USER\",\"messageId\":\"1\",\"parts\":[{\"text\":\"{\\\"skill\\\":\\\"diff_robots_txt\\\",\\\"before\\\":\\\"User-agent: *\\\\nDisallow: /\\\",\\\"after\\\":\\\"User-agent: GPTBot\\\\nDisallow: /\\\\n\\\\nUser-agent: *\\\\nAllow: /\\\\n\\\"}\"}]}}}",
    "{\"skill\":\"diff_robots_txt\",\"before\":\"User-agent: *\\nDisallow: /\",\"after\":\"User-agent: GPTBot\\nDisallow: /\\n\\nUser-agent: *\\nAllow: /\\n\"}"
   ],
   "inputModes": [
    "application/json",
    "text/plain"
   ],
   "outputModes": [
    "application/json",
    "text/plain"
   ]
  },
  {
   "id": "merge_policy",
   "name": "Add a ready-made stance to an existing file",
   "description": "Merge one of eight maintained robots.txt stances (block-ai-training, allow-ai-search-only, block-all-ai, block-datasets, block-disputed, block-seo-tools, allow-all, maximum-ai-visibility) into a robots.txt you already have, without touching a single rule you wrote: a token you already name keeps your rules and the stance's version is reported instead of applied. Example: stance='block-ai-training', robots_txt='User-agent: *\\nAllow: /\\n'.",
   "tags": [
    "robots.txt",
    "policy",
    "generator",
    "merge"
   ],
   "examples": [
    "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"message/send\",\"params\":{\"message\":{\"role\":\"ROLE_USER\",\"messageId\":\"1\",\"parts\":[{\"text\":\"{\\\"skill\\\":\\\"merge_policy\\\",\\\"robots_txt\\\":\\\"User-agent: GPTBot\\\\nDisallow: /\\\\n\\\\nUser-agent: *\\\\nAllow: /\\\\n\\\",\\\"stance\\\":\\\"block-ai-training\\\"}\"}]}}}",
    "{\"skill\":\"merge_policy\",\"robots_txt\":\"User-agent: GPTBot\\nDisallow: /\\n\\nUser-agent: *\\nAllow: /\\n\",\"stance\":\"block-ai-training\"}"
   ],
   "inputModes": [
    "application/json",
    "text/plain"
   ],
   "outputModes": [
    "application/json",
    "text/plain"
   ]
  },
  {
   "id": "whoami",
   "name": "Who is calling? (no arguments)",
   "description": "Takes no arguments. Safe to call. Deterministic. Touches no third party. Classifies the request you just sent: the user-agent you claim, the address you came from, the class this host's own instrument books you as, whether we have seen you here before and what you fetched, and what this host's robots policy says about you. Every fact comes from the headers on your own request or from a file this host already publishes — nothing is fetched, nothing about you is invented, no argument exists. Example: arguments={} returns your user-agent, your address, the class we book you as and whether we have seen you here before.",
   "tags": [
    "identify",
    "no arguments",
    "diagnostics",
    "user-agent"
   ],
   "examples": [
    "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"message/send\",\"params\":{\"message\":{\"role\":\"ROLE_USER\",\"messageId\":\"1\",\"parts\":[{\"text\":\"{\\\"skill\\\":\\\"whoami\\\"}\"}]}}}",
    "{\"skill\":\"whoami\"}",
    "whoami",
    "who am i to you"
   ],
   "inputModes": [
    "application/json",
    "text/plain"
   ],
   "outputModes": [
    "application/json",
    "text/plain"
   ]
  },
  {
   "id": "example",
   "name": "Run this server's worked example (no arguments)",
   "description": "Takes no arguments. Safe to call. Deterministic. Touches no third party. Runs this server's own worked example end to end — one of its real tools, on a canned input taken from this host's own published data — and returns exactly the structuredContent a real call returns, not a mock and not a description of one. Use it to see the shape of an answer before you decide what to send. No URL of yours is fetched and no third party is touched. Example: arguments={} runs it and returns the real answer.",
   "tags": [
    "example",
    "no arguments",
    "demo",
    "getting started"
   ],
   "examples": [
    "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"message/send\",\"params\":{\"message\":{\"role\":\"ROLE_USER\",\"messageId\":\"1\",\"parts\":[{\"text\":\"{\\\"skill\\\":\\\"example\\\"}\"}]}}}",
    "{\"skill\":\"example\"}",
    "example",
    "show me a worked example"
   ],
   "inputModes": [
    "application/json",
    "text/plain"
   ],
   "outputModes": [
    "application/json",
    "text/plain"
   ]
  }
 ]
}