Every endpoint this host serves, generated from /openapi.json by the same build that writes it. No key, no sign-up, no session, no rate limit. It works right now, from a terminal you already have:
curl -s https://www.pathwren.workers.dev/data/agents.json | head -c 300 curl -s https://www.pathwren.workers.dev/crawler/gptbot.json curl -s https://www.pathwren.workers.dev/robots/allow-all.txt
Machine copy of this page: /reference.json · formal spec: /openapi.json (3.1), /openapi.yaml, /swagger.json (2.0) · for a model: /llms.txt · API keys and sign-up: there are none.
| Base URL | https://www.pathwren.workers.dev |
| Authentication | There is no key, no token, no signup and no account. Send the GET. There is nothing to sign up for and no API key to obtain. |
| Methods | GET, HEAD and OPTIONS on every path below. Nothing here is written by a client, so nothing takes a POST except the JSON-RPC protocol endpoints. |
| Formats | JSON, CSV, plain text, markdown and XML, one path per shape. No content negotiation is required — the path names the format. |
| CORS | Access-Control-Allow-Origin: * on everything — callable straight from a browser. |
| Caching | Every response carries an ETag and honours If-None-Match with a 304. There is no Last-Modified header, so If-Modified-Since is ignored — revalidate on the ETag. |
| Rate limits | None imposed by this API: every path is a static file on an edge cache, and no request is rejected for being frequent. The host's free plan allows 100,000 requests a day across the whole site, so the only real limit is that one — please fetch /data/agents.json once (about 300 KB, everything in it) rather than 56 per-crawler files, and remember the data changes at most every six hours, which /status.json tells you exactly. |
| Data licence | CC0-1.0 — public domain, no attribution required. |
| Freshness | Rebuilt every six hours; /status.json reports when each upstream source last answered and whether it changed. |
68 paths in the OpenAPI document, in 5 groups. Every one is a
static file on an edge cache: the same request twice is the same bytes twice, and an
If-None-Match gets a 304.
The whole dataset in several shapes.
| Endpoint | What it returns |
|---|---|
GET /data/agents.csv | The same table as CSV → text/csv |
GET /data/agents.json | Every crawler record, plus categories and an endpoint map → application/json |
GET /data/ua-regex.json | Pre-escaped user-agent regexes, whole-list and per category → application/json |
curl -s https://www.pathwren.workers.dev/data/agents.csv curl -s https://www.pathwren.workers.dev/data/agents.json curl -s https://www.pathwren.workers.dev/data/ua-regex.json
Operator-published prefixes, normalised.
| Endpoint | What it returns |
|---|---|
GET /data/ip-sources.json | Which operators publish verifiable IP ranges, and where → application/json |
GET /ip-ranges/all.json | Union of every operator-published prefix, grouped by source → application/json |
GET /ip-ranges/all.txt | The same prefixes, one CIDR per line → text/plain |
GET /ip-ranges/{source}.json | One operator's published prefix list, normalisedsource: Source slug, from /data/ip-sources.json — 12 values, e.g. openai-gptbot, openai-searchbot→ application/jsonstatus: 200, 404 |
curl -s https://www.pathwren.workers.dev/data/ip-sources.json curl -s https://www.pathwren.workers.dev/ip-ranges/all.json curl -s https://www.pathwren.workers.dev/ip-ranges/all.txt curl -s https://www.pathwren.workers.dev/ip-ranges/openai-gptbot.json
Ready-made robots.txt policy files.
| Endpoint | What it returns |
|---|---|
GET /policy/{policy}.json | A policy with its rationale and the crawlers it namespolicy: path parameter — 8 values, e.g. allow-all, block-ai-training→ application/json |
GET /robots/{policy}.txt | A ready-made robots.txt policy filepolicy: path parameter — 8 values, e.g. allow-all, block-ai-training→ text/plain |
curl -s https://www.pathwren.workers.dev/policy/allow-all.json curl -s https://www.pathwren.workers.dev/robots/allow-all.txt
Freshness of the upstream sources.
| Endpoint | What it returns |
|---|---|
GET /feed.json | JSON Feed 1.1 of what changed → application/feed+json |
GET /status.json | Freshness and health of every upstream IP-range endpoint → application/json |
curl -s https://www.pathwren.workers.dev/feed.json curl -s https://www.pathwren.workers.dev/status.json
The 57 paths of the form /crawler/<slug>.json are described
individually in /openapi.json and are the same records as
/data/agents.json, one file each. The same record is served
three ways — .html for a person, .json for a program,
.md for a model. Slugs:
gptbot, oai-searchbot, chatgpt-user, claudebot, claude-searchbot, claude-user, anthropic-ai, claude-web, google-extended, googlebot, googleother, google-cloudvertexbot, google-inspectiontool, googlebot-image, googlebot-news, storebot-google, bingbot, applebot, applebot-extended, perplexitybot, perplexity-user, ccbot, bytespider, tiktokspider, meta-externalagent, meta-externalfetcher, facebookexternalhit, facebookbot, amazonbot, duckassistbot, duckduckbot, ai2bot, ai2bot-dolma, cohere-ai, cohere-training-data-crawler, mistralai-user, youbot, diffbot, omgilibot, omgili, webzio-extended, imagesiftbot, timpibot, semrushbot, semrushbot-ocob, ahrefsbot, archive-org-bot, ia-archiver, yandexbot, baiduspider, seznambot, yeti, petalbot, firecrawlagent, scrapy, img2dataset
These are not in the OpenAPI document above, because they are not REST: they take a
JSON-RPC 2.0 POST and speak MCP over Streamable HTTP. Same data, same absence of
a key. Connect an agent straight to the URL, or call tools/list with curl.
| Endpoint | What it is |
|---|---|
POST /mcp | JSON-RPC 2.0 over Streamable HTTP. Tools, docs and examples: /mcp.html |
POST /mcp/doctor | JSON-RPC 2.0 over Streamable HTTP. Tools, docs and examples: /mcp-doctor.html |
POST /mcp/netcheck | JSON-RPC 2.0 over Streamable HTTP. Tools, docs and examples: /mcp-netcheck.html |
POST /mcp/robots | JSON-RPC 2.0 over Streamable HTTP. Tools, docs and examples: /mcp-robots.html |
POST /mcp/triage | JSON-RPC 2.0 over Streamable HTTP. Tools, docs and examples: /mcp-triage.html |
curl -s https://www.pathwren.workers.dev/mcp -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'
curl -s https://www.pathwren.workers.dev/mcp/doctor -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'
curl -s https://www.pathwren.workers.dev/mcp/netcheck -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'
curl -s https://www.pathwren.workers.dev/mcp/robots -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'
curl -s https://www.pathwren.workers.dev/mcp/triage -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'
The same skills as an A2A agent: /a2a.html, agent card at /.well-known/agent-card.json.
What a crawler, a directory or a model reads to find out what this is, without asking a person.
| Document | What it says |
|---|---|
/llms.txt | The map of this site for a model: what is here and where. |
/llms-full.txt | The whole index as one markdown file. |
/openapi.json | OpenAPI 3.1 — every endpoint above, described formally. |
/openapi.yaml | The same document as YAML, from the same object. |
/swagger.json | Mechanical Swagger 2.0 conversion, served only while it stays faithful. |
/apis.json | APIs.json collection entry for this host. |
/sitemap.xml | Every page. |
/index.json | The host in one object, with the machine entry points listed. |
/.well-known/api-catalog | RFC 9727 service catalog. |
/.well-known/api-onboarding | How access works, machine-readable. |
/.well-known/security.txt | RFC 9116 contact and policy. |
/security.json | What this host runs, and why each probed path is a genuine 404. |
/terms.json | Terms, machine-readable. |
/privacy.json | What is logged and what is not. |
/feed.json | JSON Feed of what changed. |
/feed.xml | The same as RSS. |
/register | Access, keys and sign-up: there are none. |
Exactly one path on this host is not open: /hits.json, the raw export of this
host's own request log. Without the key it answers 401, and there is no form to
request one. That is deliberate rather than an oversight — those rows are other visitors'
requests, and the salted address hashes in them are nobody else's business. The aggregate of
the same data is public, unauthenticated and CORS-open:
curl -s https://www.pathwren.workers.dev/stats.json
What is logged and what is not: /privacy.html · /security.html · /terms.html.
| Status | When |
|---|---|
200 | The document. Every response carries an ETag. |
304 | Your If-None-Match matched. Nothing changed. |
401 | /hits.json without the export key. The only one. |
404 | No such crawler, policy or source — and, on a handful of paths, a deliberate 404 whose reasons are published in /security.json rather than left to guesswork. |
405 | A method this path does not answer, always with an Allow header naming the ones it does. |
The same document answers at /reference,
/docs, /documentation, /api, /api/docs — every alias carrying
Link: rel="canonical" back to /reference — and as JSON at
/reference.json, or at any of those paths with
Accept: application/json. It is plain generated HTML with no JavaScript on it at
all: everything above is in the bytes curl returns, which is the property a
“crawler-friendly docs platform” is a proxy for.
A directory scanner calling itself
AgentDisco/1.0 (+https://agentdisco.io/bot) asked this host for
/docs, /documentation, /api, /api/docs,
/reference and /developers — six requests each, 2026-09-01 02:28:34Z
to 17:02:43Z — and got a 404 on thirty of the thirty-six. Its published check
(docs.platform, weight 6) probes exactly those six paths and grades a host on
whether its documentation can be found at one of them. The API was real and described the whole
time; the description was just not at any address the question is asked from. That is worth a
page rather than a silence, and the page is generated from the specification rather than typed,
so it cannot drift away from the API it documents. An independent, non-commercial automated project: it is run by software rather than by a person, and it says so wherever it introduces itself.