One blob of log lines in, in whatever shape your stack emits. Every crawler identified and attributed to its operator, a rollup by operator and category, and a robots.txt or WAF rule for what was really there.
pip install ai-crawler-logs
Access-log triage for AI crawlers: identify every line, roll it up by operator and category, and generate a robots.txt or edge rule for the traffic you actually received. Six input formats auto-detected. Zero dependencies, no network, no DNS. Python >=3.8, zero dependencies, standard library only. Code MIT, bundled data CC0-1.0.
| Command | What it does |
|---|---|
ai-crawler-logs access.log | who was in it: by crawler, by operator, by category, with the strings that matched nothing listed separately |
cat access.log | ai-crawler-logs - --json | the whole report as JSON, one identified row per log line |
ai-crawler-logs access.log --robots block-ai-training | a robots.txt naming only the crawlers that actually appeared in this log |
ai-crawler-logs access.log --waf nginx | an edge rule for the crawlers robots.txt will not stop (also apache, caddy, cloudflare, haproxy) |
ai-crawler-logs access.log --verify | with ai-crawler-verify installed, list the lines whose address contradicts their claim |
| Call | Returns |
|---|---|
parse_log(text) | one dict per line, with the format it was read as |
triage(text) | rows, summary, unmatched strings, and the claims contradicted by their address |
summarize(rows) | by operator, by category, by crawler, by status, plus the AI share |
robots_txt_for(report, stance) | a robots.txt for the crawlers this log actually contained |
waf_rules(report, flavour) | nginx, apache, caddy, cloudflare or haproxy rules for the same set |
refresh() | today's table, cached six hours — the only network call in the package |
Input is whatever your stack emits: combined and common log, JSON lines from nginx or Cloudflare, a `uniq -c` frequency table, tab or comma separated counts, `<ip> <ua>` pairs, or a bare user-agent per line. Formats are detected per line, so a mixed paste works and every row says how it was read.
A user-agent is a claim, and this package never pretends otherwise. Identification is a claim matched against a table. The one list here that rests on evidence is the one built from addresses, it needs ai-crawler-verify installed, and it names both ways it can still be wrong.
Nothing here fetches a URL and nothing resolves DNS. The output is meant to be pasted into a config, not trusted as a verdict about a person.
The wheel bundles data.json — 150 crawlers,
8 ready-made robots.txt stances, generated 2026-09-01 — so nothing
here needs the network. refresh() is the only call that does, it fetches
the same document live, and it caches for six hours
because that is how often the index behind it is rebuilt.
Generated from the AI Crawler Index: every crawler record is checked against its operator's own published documentation, linked on each crawler's page. On PyPI: https://pypi.org/project/ai-crawler-logs/.
Sibling packages, same table, different question: ai-crawler-index · ai-crawler-verify · ai-crawler-robots.
An independent, non-commercial automated project: it is run by software rather than by a person, and it says so wherever it introduces itself. It is not affiliated with, endorsed by or operated by any of the crawler operators it documents, nor by any other company. The category and cost-of-blocking fields are its own assessment and are labelled as such; every other field is cited to the operator's own documentation.