Evaluate a pasted robots.txt against every AI crawler in this index and get the two lists that matter — blocked and allowed, per operator and category — plus the tokens in your file that match no known crawler (a typo blocks nothing), and the separation between crawlers that document obedience and the ones observed ignoring robots.txt, which need an IP or WAF rule instead.
curl -s 'https://www.pathwren.workers.dev/tools/ai-access?robots_txt=User-agent%3A%20GPTBot%0ADisallow%3A%20%2F%0A&path=/&s=client-dossiers'
One rule against the whole index: GPTBot blocked, every other AI crawler still allowed, and the count of what that leaves open.
Run it in a browser —
it answers JSON with access-control-allow-origin: *, so
fetch() works from any page with no proxy.
Allow header and points at the MCP endpoint.max-age=1800, a strong ETag,
and a 304 when you send it back./tools/ never costs you a 400.| parameter | required | what it is |
|---|---|---|
robots_txt · robots, text | yes | The contents of the robots.txt file, URL-encoded, or the raw POST body. |
path | no | Path to test the verdict at. Defaults to /. |
This endpoint holds no copy of the logic: it calls audit_ai_access on
the robots-policy-lint MCP server in-process, through that server's own
JSON-RPC envelope. One implementation, two doors — an answer that changes here changed there.
curl -s https://www.pathwren.workers.dev/mcp/robots \
-H 'content-type: application/json' \
-H 'accept: application/json, text/event-stream' \
-d '{"jsonrpc":"2.0","id":1,"method":"tools/call",
"params":{"name":"audit_ai_access","arguments":{}}}'