--- title: "MCP server — Robots Policy Lint" description: "Lint a robots.txt you paste: misspelled directives, unsupported noindex, path rules that match nothing, and which of the AI crawlers in this index it actually blocks. RFC 9309 path matching. Streamable HTTP at /mcp/robots, no key, no signup." canonical: "https://www.pathwren.workers.dev/mcp-robots.html" url: "https://www.pathwren.workers.dev/mcp-robots.md" format: "markdown" source: "the bytes of /mcp-robots.html, in the build that wrote the page" generator: "surfaces/ai-crawler-index/build.py" generated: "2026-09-11T21:25:22+00:00" license: "CC0-1.0" --- # Robots Policy Lint — MCP server > Lint a robots.txt you paste: misspelled directives, unsupported noindex, path rules that match nothing, and which of the AI crawlers in this index it actually blocks. RFC 9309 path matching. Streamable HTTP at /mcp/robots, no key, no signup. ## Connect it: one paste, one call ```text https://www.pathwren.workers.dev/mcp/robots no auth, read-only, public ``` That is the whole endpoint and those are its terms: Streamable HTTP (MCP), no API key, no account, no OAuth, no session to keep alive, nothing to install. Every tool is read-only, and none of them will fetch a URL on your behalf. The JSON a connector config wants — Claude Desktop, Cursor, VS Code, Windsurf, Cline, LibreChat, Continue, anything that takes an `mcpServers` block. Complete as it stands; there is no field to fill in: ```json { "mcpServers": { "robots-policy-lint": { "type": "streamable-http", "url": "https://www.pathwren.workers.dev/c/mcp-connector/mcp/robots" } } } ``` Claude Code takes one line instead: ```text claude mcp add --transport http robots-policy-lint https://www.pathwren.workers.dev/c/mcp-connector/mcp/robots ``` The URL in those three boxes carries `/c/mcp-connector/`, a channel tag: it is the same endpoint by another path, serving byte-identical responses, and it lets this host see that a client arrived from a config pasted off this page rather than from a directory. Strip the prefix and `https://www.pathwren.workers.dev/mcp/robots` is the canonical URL — both work, and nothing about the answer changes. **Then call `no_arguments_lint_this_hosts_robots_txt` first.** It takes no arguments at all, so there is nothing to invent and nothing to look up before you can see this server work — the subject of the answer is this host's own robots.txt and ai.txt, linted for real: ```bash curl -s https://www.pathwren.workers.dev/c/mcp-connector/mcp/robots \ -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' \ -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_lint_this_hosts_robots_txt","arguments":{}}}' \ | jq -r '.result.content[0].text' | head -3 robots.txt: 0 error(s), 2 warning(s), 134 note(s) over 134 group(s) and 134 rule(s). ai.txt: 0 error(s), 0 warning(s), 46 note(s) over 45 group(s) and 56 rule(s). Of 150 AI crawlers in this host's index, this robots.txt blocks 0 at / and allows 150. ``` Those are the answer's own first three lines, from one real run on 2026-09-06 — the counts move when the files do, so run the curl and read today's. The tool takes no arguments because its subject is not you: it is the two policy files THIS host publishes, so every caller gets the same bytes. The rest of the answer is the findings grouped by fault with the lines each one is on and the fix, an RFC 9309 verdict for six crawler tokens against three real paths naming the rule and the line that decided each, and a check that robots.txt and ai.txt still agree verdict by verdict. It reports our own faults too — a duplicate `User-agent: ChatGPT-User` group in our robots.txt, and `Allow` lines at the end of our ai.txt that read as global and are scoped to the last group. A lint that flatters its own host would not be worth pointing at yours. Nothing is fetched to answer it, no argument exists. `am_i_allowed` is the second zero-argument call and answers about YOU — which group in this robots.txt matches your user-agent, the line that decided it, and your verdict for three paths; `whoami` is the third and works unchanged on every MCP server here; `example` is the fourth. All four are safe first calls. A robots.txt with a typo is not an error anywhere. No status code changes, no log line appears; the file simply stops meaning what its author thought, and the crawler it was written to stop keeps arriving. This server reads the file you already have and says what it actually does — to 150 named AI crawlers, by [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html) matching rules rather than by eye. ```text # which AI crawlers does my robots.txt really block? curl -s https://www.pathwren.workers.dev/mcp/robots \ -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' \ -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"audit_ai_access", "arguments":{"robots_txt":"User-agent: GPTBot\nDisallow: /\n"}}}' \ | jq -r '.result.structuredContent.blocked_crawlers[].name' GPTBot ``` ## Tools | Tool | What it does | | --- | --- | | `no_arguments_lint_this_hosts_robots_txt` | TAKES NO ARGUMENTS. The lint this server runs on a robots.txt you paste, run instead on the two policy files THIS host publishes: every finding with its severity, line number and fix, an RFC 9309 verdict for six crawler tokens against three real paths naming the rule and the line that decided each one, a check that robots.txt and ai.txt still agree verdict by verdict, and a rollup of which AI crawlers in this host's index the file actually stops. The input schema is literally empty — `arguments: {}` and no arguments key at all both work. No key, no account, no OAuth, read-only. Nothing is fetched to build it: no request leaves this edge and none is made to you. Example: arguments={} returns the findings, the verdict table and the agreement check. | | `am_i_allowed` | Takes no arguments. Safe to call. Deterministic. Touches no third party. THE CALL TO MAKE FIRST on this server. Reads this host's own robots.txt against YOUR user-agent through the same RFC 9309 evaluator as check_path_allowed: the group that matches you, the line that decided it, that group verbatim, and an allowed/disallowed verdict for three real paths. No key, no account, no OAuth, read-only, nothing to invent. Example: arguments={} returns your matched group, the line it was matched on, and the verdicts. | | `lint_robots_txt` | Parse a robots.txt you paste and report every fault that makes it do something other than what it looks like: misspelled directives, a full UA string where a product token belongs, rules before any User-agent line, duplicate groups, noindex (unsupported since 2019), relative Sitemap URLs, BOM. Each finding carries the line number and the fix. Example: robots_txt='User-agent: GPTBot\nDisallow: /\n\nUser-agent: *\nAllow: /\n' — paste the whole file, it is never fetched for you. Also callable without MCP, same implementation: GET https://www.pathwren.workers.dev/tools/robots-lint?robots_txt=&s=client-dossiers — or POST the file as the raw body to the same URL. | | `check_path_allowed` | Evaluate a pasted robots.txt for one crawler and one or more paths under RFC 9309: longest token match for the group, longest path pattern for the rule, Allow breaking a tie, * and $ supported. Returns allowed/disallowed per path with the exact line that decided it, and flags the cases where a merge-groups parser and a first-group-wins parser would disagree. Example: user_agent='GPTBot', paths=['/', '/blog'], with your robots_txt pasted in. Also callable without MCP, same implementation: GET https://www.pathwren.workers.dev/tools/robots-allowed?robots_txt=&ua=GPTBot&path=/blog&s=client-dossiers | | `audit_ai_access` | Evaluate a pasted robots.txt against every AI crawler in this index and return the two lists that matter: blocked and allowed, per operator and category. Also names the tokens in your file that match no known crawler (a typo blocks nothing) and separates the crawlers that document obedience from the ones observed ignoring robots.txt, which need an IP or WAF rule instead. Example: path='/' with your robots_txt pasted in — the verdict is per crawler, at that path. Also callable without MCP, same implementation: GET https://www.pathwren.workers.dev/tools/ai-access?robots_txt=&s=client-dossiers | | `diff_robots_txt` | Compare two versions of a robots.txt and report only the crawlers whose verdict actually changes at a given path — not the text difference. Answers 'did my edit do what I meant, and did it do anything else', including sitemap additions and whether the parse errors went up or down. Example: before='User-agent: *\nAllow: /\n', after=your edited file, path='/'. | | `merge_policy` | Merge one of eight maintained robots.txt stances (block-ai-training, allow-ai-search-only, block-all-ai, block-datasets, block-disputed, block-seo-tools, allow-all, maximum-ai-visibility) into a robots.txt you already have, without touching a single rule you wrote: a token you already name keeps your rules and the stance's version is reported instead of applied. Example: stance='block-ai-training', robots_txt='User-agent: *\nAllow: /\n'. | | `whoami` | Takes no arguments. Safe to call. Deterministic. Touches no third party. Classifies the request you just sent: the user-agent you claim, the address you came from, the class this host's own instrument books you as, whether we have seen you here before and what you fetched, and what this host's robots policy says about you. Every fact comes from the headers on your own request or from a file this host already publishes — nothing is fetched, nothing about you is invented, no argument exists. Example: arguments={} returns your user-agent, your address, the class we book you as and whether we have seen you here before. | | `example` | Takes no arguments. Safe to call. Deterministic. Touches no third party. Runs this server's own worked example end to end — one of its real tools, on a canned input taken from this host's own published data — and returns exactly the structuredContent a real call returns, not a mock and not a description of one. Use it to see the shape of an answer before you decide what to send. No URL of yours is fetched and no third party is touched. Example: arguments={} runs it and returns the real answer. | ## The faults it looks for Each finding carries the line number, the text as written, and what the consequence is — not a severity number. The ones that cost the most, in the order they cost it: - **A misspelled directive.** `Useragent:`, `Disallow :`, `User agent:`. RFC 9309 §2.2 says a parser must ignore a line it does not recognise, so the group silently does not exist and every rule under it is inert. - **A user-agent string where a product token belongs.** `User-agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://...)` matches nothing: §2.2.1 matches a product token, not the whole header. - **`noindex:` in robots.txt.** Google stopped supporting it on 1 September 2019. A file that relies on it is not doing what its author believes. - **A path without a leading slash**, a `Sitemap:` that is not absolute, a `Crawl-delay` aimed at a crawler that documents ignoring it, rules after a blank line inside a group, and a group that repeats a token already claimed above. ## Path matching, spelled out `check_path_allowed` implements §2.2.2 in full: `*` and `$` wildcards, longest-match wins, and Allow beats Disallow on an exact tie — the rule people get backwards most often. Every verdict names the rule that decided it and the line it is on, so a surprising answer is traceable to a line rather than to this server's opinion. ## What it will not do **It does not fetch your robots.txt, or anyone's.** Every tool takes the file as text you paste. That is the same refusal [crawler-log-triage](https://www.pathwren.workers.dev/mcp-triage.html) makes and for the same reason: a tool that fetches an arbitrary URL for whoever is talking to it is an SSRF proxy with a friendly description, and one published by a host that counts its own requests is also a way to manufacture traffic. Fetch the file with your own client, paste the bytes. It also never edits anything. `merge_policy` returns new text and a list of what changed; what you do with it is your decision, and rules you already wrote are preserved untouched — a group naming a crawler you already name is reported as skipped, never overwritten. ## How is this different from the others? [ai-crawler-index](https://www.pathwren.workers.dev/mcp.html) and [crawler-log-triage](https://www.pathwren.workers.dev/mcp-triage.html) both *write* a robots.txt from a policy or from a log. This one is the only server here that *reads* the one you have and reports on it, and the 8 maintained stances it can merge in are the same ones behind [/policy/](https://www.pathwren.workers.dev/policy/). No tool name is shared with any of the other four. Protocol versions 2025-06-18, negotiated per call. `server/discover` answers for clients on 2026-07-28, `initialize` for everyone else. Read-only, stateless, no key. Same crawler index as [/data/agents.json](https://www.pathwren.workers.dev/data/agents.json), refreshed every six hours. Listed in the [official MCP Registry](https://registry.modelcontextprotocol.io/v0/servers?search=robots-policy-lint) as `dev.workers.pathwren.www/robots-policy-lint`. ## Sitemap - [Full sitemap (XML)](https://www.pathwren.workers.dev/sitemap.xml) — every page, with dates - [Full sitemap (markdown)](https://www.pathwren.workers.dev/sitemap.md) — the same map, readable - [llms.txt](https://www.pathwren.workers.dev/llms.txt) — the whole host in one text file - [documents.json](https://www.pathwren.workers.dev/documents.json) — every document, with its ETag - [A2A agents](https://www.pathwren.workers.dev/a2a.html) - [About and method](https://www.pathwren.workers.dev/about.html) - [API](https://www.pathwren.workers.dev/api.html) - [/c//](https://www.pathwren.workers.dev/c) - [Changelog](https://www.pathwren.workers.dev/changelog.html) - [Compliance](https://www.pathwren.workers.dev/compliance) - [Contact](https://www.pathwren.workers.dev/contact) - [Impressum · Anbieterkennzeichnung](https://www.pathwren.workers.dev/impressum) - [AI Crawler Index](https://www.pathwren.workers.dev/index.html) - [No model runs here](https://www.pathwren.workers.dev/inference.html) - [Legal](https://www.pathwren.workers.dev/legal) - [MCP server](https://www.pathwren.workers.dev/mcp-doctor.html) - [MCP server](https://www.pathwren.workers.dev/mcp-lint.html) - [MCP server](https://www.pathwren.workers.dev/mcp-markdown.html) - [MCP server](https://www.pathwren.workers.dev/mcp-netcheck.html) - [MCP server](https://www.pathwren.workers.dev/mcp-robots.html) - [MCP transport: the GET and HEAD leg](https://www.pathwren.workers.dev/mcp-transport.html) - [MCP server](https://www.pathwren.workers.dev/mcp-triage.html) - [MCP server](https://www.pathwren.workers.dev/mcp.html) - [Packages](https://www.pathwren.workers.dev/packages.html) - [Pricing](https://www.pathwren.workers.dev/pricing) - [Privacy](https://www.pathwren.workers.dev/privacy.html) - [API reference](https://www.pathwren.workers.dev/reference) - [Access, keys and sign-up](https://www.pathwren.workers.dev/register) - [Security posture](https://www.pathwren.workers.dev/security.html) - [Services](https://www.pathwren.workers.dev/services) - [Upstream status](https://www.pathwren.workers.dev/status.html) - [Terms of use](https://www.pathwren.workers.dev/terms.html) - [Trust](https://www.pathwren.workers.dev/trust) ## Machine copies of this page - [HTML (canonical)](https://www.pathwren.workers.dev/mcp-robots.html) - [JSON](https://www.pathwren.workers.dev/mcp-robots.json) - [Markdown](https://www.pathwren.workers.dev/mcp-robots.md) — this document This document is a markdown rendering of [https://www.pathwren.workers.dev/mcp-robots.html](https://www.pathwren.workers.dev/mcp-robots.html), generated from that page's own bytes in the same build. The HTML page is canonical.