curl -s https://www.pathwren.workers.dev/api.json # this page, as JSON curl -s https://www.pathwren.workers.dev/openapi.json # Every read endpoint, described formally
No key, no account, no handshake — every page here has a JSON twin one hop away. Machine doors: 6 keyless GET tools · documents.json · changes · llms.txt · openapi.json · agent card · mcp · a2a
Everything is a static file served with Access-Control-Allow-Origin: *.
There is no key, no quota and no state, so the description below is complete and will
not surprise you.
The complete reference, generated from the OpenAPI document rather than typed —
every path, every parameter, every curl: /reference
(also at /docs, /documentation, /api,
/api/docs; as data at /reference.json).
The page below is the short version.
curl -s https://www.pathwren.workers.dev/openapi.json | jq '.paths | keys | length'
| Path | Returns |
|---|---|
/data/agents.json | all 150 crawler records + category definitions |
/crawler/<slug>.json | one crawler; slugs from agents.json |
/operator/<slug>.json | one of 74 operators and its crawlers |
/category/<slug>.json | one of 9 categories |
/policy/<slug>.json | a robots.txt policy, its rationale, and the file |
/robots/<slug>.txt | the robots.txt file itself, ready to append |
/ip-ranges/all.json | union of published prefixes, grouped by source |
/ip-ranges/<source>.json | one operator endpoint, normalised, with provenance |
/data/ua-regex.json | pre-escaped regexes, whole-list and per category |
/status.json | upstream freshness |
/feed.json, /feed.xml | what changed |
# every token that only affects training, as a robots.txt block
curl -s https://www.pathwren.workers.dev/data/agents.json \
| jq -r '.crawlers[] | select(.category=="ai-training") | "User-agent: \(.robots_token)\nDisallow: /\n"'
# is this request really GPTBot?
curl -s https://www.pathwren.workers.dev/ip-ranges/openai-gptbot.json | jq -r '.ipv4[]' \
| while read c; do python3 -c "import ipaddress,sys;print(sys.argv[1]) if ipaddress.ip_address('$1') in ipaddress.ip_network(sys.argv[1]) else None" "$c"; done
# what does blocking this cost me?
curl -s https://www.pathwren.workers.dev/crawler/oai-searchbot.json | jq -r .cost_of_blocking
Machine description: /openapi.json · /openapi.yaml · catalogue: /.well-known/api-catalog · APIs.json: /apis.json · getting started: /.well-known/api-onboarding · prose for models: /llms.txt, /llms-full.txt