A robots.txt that looks right and blocks nothing is the normal failure. This reads the file you already have and says which of the 56 AI crawlers it actually stops.
pip install ai-crawler-robots
Lint a robots.txt, evaluate any path against RFC 9309, and audit the file against every AI crawler this index knows. Zero dependencies, no network, no URL fetching. Python >=3.8, zero dependencies, standard library only. Code MIT, bundled data CC0-1.0.
| Command | What it does |
|---|---|
ai-crawler-robots lint robots.txt | the faults that make a file do nothing: a UA string where a product token belongs, a misspelled directive, Noindex:, a shadowed group, a path without a leading slash |
ai-crawler-robots audit robots.txt | every AI crawler this index knows, split into blocked / allowed / says-no-but-will-not-obey |
ai-crawler-robots check robots.txt --agent GPTBot --path /blog/ | allowed or not, and the line number that decided it |
ai-crawler-robots diff old.txt new.txt | what changed by EFFECT, not by text — a reformat is not a change |
ai-crawler-robots policy block-ai-training --merge robots.txt | add a ready-made stance without touching the rules already in the file |
| Call | Returns |
|---|---|
lint(text) | findings with level, line, message and a suggested fix |
is_allowed(text, agent, path) | the RFC 9309 verdict, the winning rule, its line, and whether two conformant parsers would disagree |
audit(text, path='/') | blocked / allowed / unenforceable per crawler, plus the tokens in the file that match nothing |
diff(old, new) | only the crawlers whose verdict flipped |
merge_policy(text, stance) | the file with a stance added and its existing groups untouched |
refresh() | today's table, cached six hours — the only network call in the package |
The single most common robots.txt failure is not a syntax error. It is a file that looks right and blocks nothing: a User-agent line carrying a whole browser-style UA string instead of the product token, a group shadowed by an earlier one, a Disallow under a token no crawler sends, or Noindex:, which Google stopped honouring in robots.txt in 2019. Every one of those is invisible to a person reading the file and obvious to a parser.
It reads the file rather than writing one. Generators are everywhere; the question an operator actually has is about the file they already deployed.
Nothing here fetches a URL. There is no argument that takes one, and the only network call in the package is refresh(), which you have to name.
The wheel bundles data.json — 150 crawlers,
8 ready-made robots.txt stances, generated 2026-09-01 — so nothing
here needs the network. refresh() is the only call that does, it fetches
the same document live, and it caches for six hours
because that is how often the index behind it is rebuilt.
Generated from the AI Crawler Index: every crawler record is checked against its operator's own published documentation, linked on each crawler's page. On PyPI: https://pypi.org/project/ai-crawler-robots/.
Sibling packages, same table, different question: ai-crawler-index · ai-crawler-verify · ai-crawler-logs.
An independent, non-commercial automated project: it is run by software rather than by a person, and it says so wherever it introduces itself. It is not affiliated with, endorsed by or operated by any of the crawler operators it documents, nor by any other company. The category and cost-of-blocking fields are its own assessment and are labelled as such; every other field is cited to the operator's own documentation.