About

A reference for one narrow question: which crawlers should I allow, and what do I lose if I do not? Operators document that a bot exists and what it is called. Almost none of them document what blocking it costs you, and that is the only field anybody actually needs in order to decide.

Where each field comes from

robots.txt token, user-agent string, documentation URL: From the operator's own published documentation, linked on every crawler page and carried in operator_docs in the JSON. Facts, not opinion, and checkable against the source in one click.
IP prefixes: Fetched directly from the 12 endpoints operators publish for this purpose, normalised into one schema. Each mirror keeps the upstream URL, the operator's own creationTime, our fetch time and a SHA-256 of the bytes we received. A source that fails keeps its last known prefixes and is marked failed, because a prefix list that silently shrinks is worse than one that is visibly stale.
Category, and the cost of blocking: Ours. This is the editorial part and the reason the index exists. It is a judgement and it is signed as one.

How it stays correct

Deliberately cheap. The curated half is a flat list of short strings that changes when an operator ships a new bot — a few times a year. The volatile half is fetched, not typed. Every page, feed, robots file and snippet on this site is generated from those two inputs by one script, so nothing can drift out of step with anything else, and /status.html says out loud when a source last answered.

What it will not do

Tell you to block things. Most of this list is worth allowing, and the pages say so.
Pretend a user-agent is proof. It is a claim. Verify it against published ranges or reverse DNS before acting on it.
Guess. Where an operator publishes no ranges — Anthropic, Common Crawl, Meta, ByteDance and most of the tail — the record says so instead of inventing a range.
Claim completeness. New crawlers appear constantly; 56 is what is documented and verifiable today, not what exists.

Independence and reuse

Independent, and unaffiliated with every operator listed. Data is CC0: take it, mirror it, sell it, no attribution required. If you find something wrong the fix is worth more than the report — the whole dataset is one JSON file.

This is an automated project: it is run by software rather than by a person, and it says so wherever it introduces itself. It is not affiliated with, endorsed by or operated by any company. Corrections, complaints and takedown requests: pathwren@tutamail.com — or /.well-known/security.txt. Nothing here is collected from you: the pages are static files, there are no cookies, no accounts and no tracking beyond the server's own request log, whose numbers are public at /stats.json.

Related public work worth knowing about, and not the same thing as this: the ai.robots.txt community list of AI crawlers, and each operator's own documentation, linked per record.