Everything is a static file served with Access-Control-Allow-Origin: *.
There is no key, no quota and no state, so the description below is complete and will
not surprise you.
curl -s https://www.pathwren.workers.dev/openapi.json | jq '.paths | keys | length'
| Path | Returns |
|---|---|
/data/agents.json | all 56 crawler records + category definitions |
/crawler/<slug>.json | one crawler; slugs from agents.json |
/operator/<slug>.json | one of 30 operators and its crawlers |
/category/<slug>.json | one of 9 categories |
/policy/<slug>.json | a robots.txt policy, its rationale, and the file |
/robots/<slug>.txt | the robots.txt file itself, ready to append |
/ip-ranges/all.json | union of published prefixes, grouped by source |
/ip-ranges/<source>.json | one operator endpoint, normalised, with provenance |
/data/ua-regex.json | pre-escaped regexes, whole-list and per category |
/status.json | upstream freshness |
/feed.json, /feed.xml | what changed |
# every token that only affects training, as a robots.txt block
curl -s https://www.pathwren.workers.dev/data/agents.json \
| jq -r '.crawlers[] | select(.category=="ai-training") | "User-agent: \(.robots_token)\nDisallow: /\n"'
# is this request really GPTBot?
curl -s https://www.pathwren.workers.dev/ip-ranges/openai-gptbot.json | jq -r '.ipv4[]' \
| while read c; do python3 -c "import ipaddress,sys;print(sys.argv[1]) if ipaddress.ip_address('$1') in ipaddress.ip_network(sys.argv[1]) else None" "$c"; done
# what does blocking this cost me?
curl -s https://www.pathwren.workers.dev/crawler/oai-searchbot.json | jq -r .cost_of_blocking
Machine description: /openapi.json · /openapi.yaml · catalogue: /.well-known/api-catalog · APIs.json: /apis.json · getting started: /.well-known/api-onboarding · prose for models: /llms.txt, /llms-full.txt