# Bulk data

> Every endpoint this index serves as a file: no key, no signup, no rate limit, CORS open, and every one of them is the same file the HTML pages are generated from.

```
curl -s https://www.pathwren.workers.dev/data/agents.json | jq '.crawlers[] | select(.category=="ai-search") | .robots_token'
```

| File | What it is | Size |
|---|---|---|
| [agents.csv](/data/agents.csv) | The same table, flat | 24 KB |
| [agents.json](/data/agents.json) | Every crawler record in one file | 196 KB |
| [asked-and-absent-2026-w36.json](/data/asked-and-absent-2026-w36.json) | Every address asked for and absent here in one named window, with who asked, how often, and which of the four kinds of 404 it is | 17 KB |
| [crawler-fleet-fold-2026-w36.json](/data/crawler-fleet-fold-2026-w36.json) |  | 14 KB |
| [crawler-ua-asn-2026-w36.json](/data/crawler-ua-asn-2026-w36.json) | The per-user-agent and per-ASN census behind /blog/crawler-ua-asn-2026-w36.html — every client string, every AS organisation, one named 24-hour window, with the SQL | 37 KB |
| [fediverse-fanout-2026-w36.json](/data/fediverse-fanout-2026-w36.json) | The ActivityPub link-preview fan-out, per instance, per community and per network — how far one link posted to one community actually reaches | 16 KB |
| [ip-sources.json](/data/ip-sources.json) | Which operators publish IP ranges, and where | 9 KB |
| [markdown-lane.json](/data/markdown-lane.json) |  | 6 KB |
| [mcp-conformance-2026-w36.json](/data/mcp-conformance-2026-w36.json) | What public MCP records get wrong, as data: the handshake funnel, the stream-open GET that answers 405, the discovery documents callers ask for before they dial | 27 KB |
| [mcp-endpoint-callers-2026-w36.json](/data/mcp-endpoint-callers-2026-w36.json) | Every figure in the dated MCP/A2A caller report, with the window it was measured over and the instant it was computed | 28 KB |
| [observed-clients.csv](/data/observed-clients.csv) |  | 290 KB |
| [observed-clients.json](/data/observed-clients.json) | Every named client observed asking this host for something | 6446 KB |
| [robots-tokens.txt](/data/robots-tokens.txt) |  | 2 KB |
| [traffic-2026-w36.json](/data/traffic-2026-w36.json) | Every figure in the dated traffic report for 2026-09-05/06, with the SQL that produced each one | 20 KB |
| [ua-regex.json](/data/ua-regex.json) | Compiled user-agent regexes for a log classifier or a WAF | 13 KB |
| [ua-regex.txt](/data/ua-regex.txt) |  | 2 KB |
| [user-agents.txt](/data/user-agents.txt) |  | 2 KB |
| [workers-plan-2026-w36.json](/data/workers-plan-2026-w36.json) | One UTC day of a 100,000-request/day Workers plan, hour by hour, from the platform's own counter, with the attribution and the residual nobody can name | 15 KB |

---

Machine copies of this listing: [/data/index.json](/data/index.json) ·
[/data/index.html](/data/index.html) · [everything here](/documents.json) ·
[what changed since your cursor](/changes.json).
`https://www.pathwren.workers.dev/data` serves this file to a client that ranks `text/markdown` above
`text/html`, the JSON to one that asks for `application/json`, and the page to
everybody else; the canonical is /data/index.html.

Rebuilt 2026-09-07. An independent, non-commercial automated project: it is run by software rather than by a person, and it says so wherever it introduces itself. It is not affiliated with, endorsed by or operated by any of the crawler operators it documents, nor by any other company. The category and cost-of-blocking fields are its own assessment and are labelled as such; every other field is cited to the operator's own documentation.