An agent that meets your site for the first time does not read your homepage. It asks for about twenty small files at fixed paths, and what it finds decides whether you exist in its index at all. This server checks which of them a host serves, and names, for each one missing, the client that asked us for it and the date it did.
# which discovery documents does a host serve?
curl -s https://www.pathwren.workers.dev/mcp/doctor \
-H 'content-type: application/json' -H 'accept: application/json, text/event-stream' \
-d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"check_discovery_documents",
"arguments":{"host":"example.com"}}}' \
| jq -r '.result.structuredContent.documents[] | "\(.verdict)\t\(.path)"'
served /robots.txt
missing /llms.txt
missing /.well-known/agent-card.json
soft-404 /.well-known/mcp.json
Each missing line comes back with who asks for it, when they asked here, and what the 404 costs — not a style-guide opinion:
curl -s https://www.pathwren.workers.dev/mcp/doctor \
-H 'content-type: application/json' -H 'accept: application/json, text/event-stream' \
-d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"explain_document",
"arguments":{"name":"owners.json"}}}' | jq -r '.result.structuredContent.observed_askers[]
| "\(.at)\t\(.ua)"'
2026-08-31T23:12:20Z VerifyMCP-OwnersBot/1.0 (+https://verifymcp.io/docs/build/owners-json)
claude mcp add --transport http agent-discovery-doctor https://www.pathwren.workers.dev/mcp/doctor
Clients that take a JSON config (Claude Desktop, Cursor, VS Code, Windsurf):
{
"mcpServers": {
"agent-discovery-doctor": { "type": "streamable-http", "url": "https://www.pathwren.workers.dev/mcp/doctor" }
}
}
| Tool | What it answers |
|---|---|
check_discovery_documents | The whole job. GETs the 21 paths on a host you name and returns each as served, missing, gated or soft-404 — a 200 carrying an HTML error page, which is worse than a 404 because the reader believes it — with who asks for each missing one. |
explain_document | One document: what it is for, the named clients seen asking this host for it with dates and the status they took, what a 404 costs, a minimal skeleton, and the spec. No argument returns the whole catalogue. |
validate_llms_txt | Paste an llms.txt, get errors and warnings with line numbers and the fix, plus the link list as parsed. Checks the format, not your prose. |
llms_txt_from_sitemap | Paste sitemap.xml or a list of URLs, get a draft llms.txt: sections by path, titles from slugs, and a TODO everywhere a sentence only you can write belongs. |
validate_agent_card | Paste an A2A agent card, get the required fields it is missing and the capabilities it declares true — the ones a reader will then try. |
The catalogue is not a reading of the specs. Every row below is a request that arrived at this host, with the user-agent as it came and the status it took:
| Document | Asked for here by | When | It got |
|---|---|---|---|
/.well-known/agent-card.json | GolemreachTrustBot/0.1 | 2026-09-01 00:48Z | 404 — then 200 on its return at 01:59Z, once we shipped one |
/.well-known/agent.json | GolemreachTrustBot/0.1 | 2026-09-01 00:48Z | 404 — it asks for both paths in the same second |
/.well-known/owners.json | VerifyMCP-OwnersBot/1.0 | 2026-08-31 23:12Z | 404 at / and at /mcp/, in the same second |
/.well-known/oauth-protected-resource | mcpbeat/0.1, exaforce-mcprep/0.1, undici | 2026-08-31 22:32Z onward | 404 — and every one of them carried on regardless. Since 2026-09-01 03:35Z the 404 is application/json and says why, instead of an HTML page |
/apis.json and 11 more | apis.io-submit/1.0 | 2026-08-31 21:21Z | a 12-document walk during directory submission; 7 were 404 |
/llms.txt | ClaudeBot/1.0 | 2026-09-01 01:10Z | 200 |
/.well-known/agent-card.json | SaSameAgentAudit/0.1 | 2026-09-01 01:06Z | 404 |
The other documents in the catalogue — ai.txt, api-catalog,
ai-plugin.json, swagger.json, security.txt and the rest —
are marked as conventions nobody has been observed asking us for. The tool says which is which
rather than implying every file is equally urgent.
It refuses to check this host. A tool that fetches a URL for whoever is
talking to it, published by someone who counts requests, is a way to manufacture traffic — so
before any request is made it rejects its own origin and every subdomain of it, the hostname of
the request that is asking, localhost, every bare IP literal, internal TLDs, and
ephemeral preview domains (*.trycloudflare.com, *.ngrok.io,
*.vercel.app previews). The refusal names the host and the reason. It is https-only,
one GET per path, capped bytes, and it identifies itself in the User-Agent as
agent-discovery-doctor/1.0 with a link back to this page, so you can find it in
your own log and see exactly what it did.
The other four tools fetch nothing at all: text in, verdict out.
ai-crawler-index answers questions about crawlers. crawler-log-triage reads a log you already have. This one is about the other direction entirely — not who came to you, but what a visiting agent asks for and whether the answer it gets is any good. No tool name, and no argument, is shared with either.
Protocol versions 2026-07-28, 2025-11-25, 2025-06-18, 2025-03-26, 2024-11-05, negotiated per call.
server/discover answers for clients on 2026-07-28, initialize for everyone else.
Read-only, stateless, no key. Listed in the
official
MCP Registry as dev.workers.pathwren.www/agent-discovery-doctor.