Paste a robots.txt and get every fault that makes it do something other than what it looks like: misspelled directives, a whole User-Agent string where a product token belongs, rules before any User-agent line, duplicate groups, noindex (unsupported since 2019), relative Sitemap URLs, a byte-order mark. Each finding carries the line number and the fix.
curl -s 'https://www.pathwren.workers.dev/tools/robots-lint?robots_txt=User-agent%3A%20GPTBot%0ADisallow%3A%20%2F%0ANoindex%3A%20%2Fprivate%0A&s=client-dossiers'
Three lines with one real fault: Noindex has not been supported since 2019 and stops nothing. The answer names the line and the fix.
curl -s --data-binary @robots.txt 'https://www.pathwren.workers.dev/tools/robots-lint?s=client-dossiers'
The same lint on a file too big for a query string. POST is accepted here for exactly this reason; it is still keyless and still not JSON-RPC.
Run it in a browser —
it answers JSON with access-control-allow-origin: *, so
fetch() works from any page with no proxy.
Allow header and points at the MCP endpoint.max-age=3600, a strong ETag,
and a 304 when you send it back./tools/ never costs you a 400.| parameter | required | what it is |
|---|---|---|
robots_txt · robots, text | yes | The contents of the robots.txt file, URL-encoded. Or POST the file as the raw request body, which is what a file larger than a URL wants. |
This endpoint holds no copy of the logic: it calls lint_robots_txt on
the robots-policy-lint MCP server in-process, through that server's own
JSON-RPC envelope. One implementation, two doors — an answer that changes here changed there.
curl -s https://www.pathwren.workers.dev/mcp/robots \
-H 'content-type: application/json' \
-H 'accept: application/json, text/event-stream' \
-d '{"jsonrpc":"2.0","id":1,"method":"tools/call",
"params":{"name":"lint_robots_txt","arguments":{}}}'