Static files. No key, no signup, no rate limit, CORS open, and every one of them is the same file the HTML pages are generated from.
curl -s https://www.pathwren.workers.dev/data/agents.json | jq '.crawlers[] | select(.category=="ai-search") | .robots_token' curl -s https://www.pathwren.workers.dev/ip-ranges/all.txt | wc -l curl -s https://www.pathwren.workers.dev/robots/block-ai-training.txt >> robots.txt
| Endpoint | What it is | Size |
|---|---|---|
| /data/agents.json | Every record, with the endpoint map and category definitions. | 56 records |
| /data/agents.csv | The same table, flat, for a spreadsheet or a COPY INTO. | 56 rows |
| /data/user-agents.txt | User-agent substrings, one per line. Feed to grep. | 54 lines |
| /data/robots-tokens.txt | robots.txt tokens, one per line. | 56 lines |
| /data/ua-regex.json | Pre-escaped alternations, whole-list and per category. | — |
| /data/ua-regex.txt | The single all-crawlers regex, nothing else. | 1 line |
| /data/ip-sources.json | Which operators publish IP ranges, and where. | 12 sources |
| /ip-ranges/all.json | Union of every published prefix, grouped by source. | 1887+1056 prefixes |
| /ip-ranges/all.txt | The same prefixes, one CIDR per line, for a WAF. | 2943 lines |
| /status.json | Freshness and health of every upstream endpoint. | — |
| /openapi.json | OpenAPI 3.1 description of all of the above. | — |
| /llms-full.txt | The whole index as one markdown file. | — |
| /feed.json | JSON Feed 1.1: what changed. | — |
| /feed.xml | RSS 2.0, same content. | — |