A robots.txt with a typo is not an error anywhere. No status code changes, no log line appears; the file simply stops meaning what its author thought, and the crawler it was written to stop keeps arriving. This server reads the file you already have and says what it actually does — to 56 named AI crawlers, by RFC 9309 matching rules rather than by eye.
# which AI crawlers does my robots.txt really block?
curl -s https://www.pathwren.workers.dev/mcp/robots \
-H 'content-type: application/json' -H 'accept: application/json, text/event-stream' \
-d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"audit_ai_access",
"arguments":{"robots_txt":"User-agent: GPTBot\nDisallow: /\n"}}}' \
| jq -r '.result.structuredContent.blocked_crawlers[].name'
GPTBot
| Tool | What it does |
|---|---|
lint_robots_txt | Parse a robots.txt you paste and report every fault that makes it do something other than what it looks like: misspelled directives, a full UA string where a product token belongs, rules before any User-agent line, duplicate groups, noindex (unsupported since 2019), relative Sitemap URLs, BOM. Each finding carries the line number and the fix. |
check_path_allowed | Evaluate a pasted robots.txt for one crawler and one or more paths under RFC 9309: longest token match for the group, longest path pattern for the rule, Allow breaking a tie, * and $ supported. Returns allowed/disallowed per path with the exact line that decided it, and flags the cases where a merge-groups parser and a first-group-wins parser would disagree. |
audit_ai_access | Evaluate a pasted robots.txt against every AI crawler in this index and return the two lists that matter: blocked and allowed, per operator and category. Also names the tokens in your file that match no known crawler (a typo blocks nothing) and separates the crawlers that document obedience from the ones observed ignoring robots.txt, which need an IP or WAF rule instead. |
diff_robots_txt | Compare two versions of a robots.txt and report only the crawlers whose verdict actually changes at a given path — not the text difference. Answers 'did my edit do what I meant, and did it do anything else', including sitemap additions and whether the parse errors went up or down. |
merge_policy | Merge one of eight maintained robots.txt stances (block-ai-training, allow-ai-search-only, block-all-ai, block-datasets, block-disputed, block-seo-tools, allow-all, maximum-ai-visibility) into a robots.txt you already have, without touching a single rule you wrote: a token you already name keeps your rules and the stance's version is reported instead of applied. |
Each finding carries the line number, the text as written, and what the consequence is — not a severity number. The ones that cost the most, in the order they cost it:
Useragent:, Disallow :,
User agent:. RFC 9309 §2.2 says a parser must ignore a line it does not
recognise, so the group silently does not exist and every rule under it is inert.User-agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://...) matches nothing:
§2.2.1 matches a product token, not the whole header.noindex: in robots.txt. Google stopped supporting it on
1 September 2019. A file that relies on it is not doing what its author believes.Sitemap: that is not
absolute, a Crawl-delay aimed at a crawler that documents ignoring it, rules
after a blank line inside a group, and a group that repeats a token already claimed above.check_path_allowed implements §2.2.2 in full: * and $
wildcards, longest-match wins, and Allow beats Disallow on an exact tie — the rule people get
backwards most often. Every verdict names the rule that decided it and the line it is on, so a
surprising answer is traceable to a line rather than to this server's opinion.
It does not fetch your robots.txt, or anyone's. Every tool takes the file as text you paste. That is the same refusal crawler-log-triage makes and for the same reason: a tool that fetches an arbitrary URL for whoever is talking to it is an SSRF proxy with a friendly description, and one published by a host that counts its own requests is also a way to manufacture traffic. Fetch the file with your own client, paste the bytes.
It also never edits anything. merge_policy returns new text and a list of what
changed; what you do with it is your decision, and rules you already wrote are preserved
untouched — a group naming a crawler you already name is reported as skipped, never
overwritten.
ai-crawler-index and crawler-log-triage both write a robots.txt from a policy or from a log. This one is the only server here that reads the one you have and reports on it, and the 8 maintained stances it can merge in are the same ones behind /policy/. No tool name is shared with any of the other four.
Protocol versions 2026-07-28, 2025-11-25, 2025-06-18, 2025-03-26, 2024-11-05, negotiated per call.
server/discover answers for clients on 2026-07-28, initialize for everyone else.
Read-only, stateless, no key. Same crawler index as
/data/agents.json, refreshed every six hours. Listed in the
official
MCP Registry as dev.workers.pathwren.www/robots-policy-lint.