curl -s https://www.pathwren.workers.dev/crawler/index.json   # this page, as JSON
curl -s https://www.pathwren.workers.dev/data/agents.json     # Every crawler record in one file

All crawlers (150)

Grouped by what the crawl is for. Machine copy: agents.json · agents.csv · user-agents.txt

Measured, not asserted: 24 hours of crawler and agent traffic against this host — 2,921 unique clients, 1,229 of them crawlers, and 72.4% of the busiest hour was one bot from one address. Dated 2026-09-06, one named window, every figure with its SQL at /data/traffic-2026-w36.json.

Measured here, not asserted: 24 hours of AI-crawler traffic, as the actual table — 3,374 client keys, 1,313 distinct user-agent strings and the AS organisation behind every one of them, in the 24 hours to 2026-09-06T16:00:40+00:00. Published as the per-UA and per-ASN table rather than as a claim about it, with the SQL beside it. One of five documents about the same 24 hours — the other four are named at the foot of each one — every one of them also at .md and .json, with the figures and the SQL under the data index beside them.

AI training crawlers (27)

Collect pages in bulk so that a model can be trained or fine-tuned on them. Blocking these removes you from future training sets and changes nothing a user sees today.

AI search crawlers (21)

Build the retrieval index an assistant answers and cites from. These are the crawlers that send you traffic; blocking them is the expensive mistake in this space.

User-triggered fetchers (12)

Fetch one page because a person asked for it, right then. One human intent, one request. Blocking them produces a visible error for a real reader.

Corpus and dataset builders (17)

Crawl the web into a published or resold dataset that other people train on. Highest leverage per block, longest delay before any effect.

Search engines (28)

Classic index-and-rank crawlers. Several also feed their operator's generative answers, which is why the AI opt-out for Google and Apple is a token rather than a block.

SEO and backlink crawlers (17)

Commercial link-graph tooling. No user-facing effect either way, and usually a large share of your bot bandwidth.

Archivers (2)

Preservation crawlers. Their output is public and permanent, which makes them a separate decision from the AI one.

Link preview fetchers (4)

Read your Open Graph tags when someone shares a link. Blocking these is almost always an accident.

Tools and frameworks (22)

Not operators: crawling software anyone can run. The party behind the request is unknown, so treat them as a rate-limit question rather than a consent question.

This page as markdown: /c/mbin/crawler/index.md — the same text, no markup to strip, no JavaScript, no key, CC0. Every page here has one: add .md to any address (also .mdx, <page>.html.md, <page>.html.mdx), or send Accept: text/markdown to this one. All of them in a single index: /c/mbin/sitemap.md.