Would this crawler fetch this path?

Evaluate a pasted robots.txt for one crawler and one or more paths under RFC 9309 — longest token match for the group, longest pattern for the rule, Allow breaking a tie, * and $ supported — and get allowed/disallowed per path with the exact line that decided it, plus a flag on the cases where a merge-groups parser and a first-group-wins parser would disagree.

Copy this

curl -s 'https://www.pathwren.workers.dev/tools/robots-allowed?robots_txt=User-agent%3A%20*%0ADisallow%3A%20%2F%0AAllow%3A%20%2Fblog%2F%0A&ua=GPTBot&path=/blog/post&path=/private&s=client-dossiers'

One file, one crawler, two paths: /blog/post allowed by the longer Allow, /private disallowed, each with the line number that decided it.

Run it in a browser — it answers JSON with access-control-allow-origin: *, so fetch() works from any page with no proxy.

No setup at all

No account, no key, no token, no handshake, no session.
GET, or POST the file as the raw request body. Anything else answers 405 with an Allow header and points at the MCP endpoint.
Cacheable: max-age=3600, a strong ETag, and a 304 when you send it back.
Asked with no parameters at all it answers 200 with its own specification, so walking /tools/ never costs you a 400.
CC0-1.0. Nothing is logged about you beyond the request line this host publishes about every client, including itself.

Parameters

parameter	required	what it is
`robots_txt` · `robots`, `text`	yes	The contents of the robots.txt file, URL-encoded, or the raw POST body.
`ua` · `user_agent`, `token`	yes	The crawler's product token — GPTBot, Googlebot, ClaudeBot — not a whole User-Agent header.
`path` · `paths`	no	Path to test. Repeat for several, or send a comma separated list. Defaults to /. Max 200.

The same tool over MCP

This endpoint holds no copy of the logic: it calls check_path_allowed on the robots-policy-lint MCP server in-process, through that server's own JSON-RPC envelope. One implementation, two doors — an answer that changes here changed there.

curl -s https://www.pathwren.workers.dev/mcp/robots \
  -H 'content-type: application/json' \
  -H 'accept: application/json, text/event-stream' \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/call",
       "params":{"name":"check_path_allowed","arguments":{}}}'

What it will not do

robots.txt is a request, not an enforcement mechanism. This says what a compliant crawler would do; it does not say what an arbitrary client will do.
It fetches nothing. No argument here or on the MCP tool behind it takes a URL, and neither makes any outbound request: every answer comes out of files this host already publishes.

Also here

The other 7 endpoints, one page each.
123 clients that actually crawled this host, 2026-08-31 20:58:11Z to 2026-09-02 04:57:09Z UTC, one page each.
/data/observed-clients.json — that whole set in one request.