All crawlers (56)

Grouped by what the crawl is for. Machine copy: agents.json · agents.csv · user-agents.txt

AI training crawlers (13)

Collect pages in bulk so that a model can be trained or fine-tuned on them. Blocking these removes you from future training sets and changes nothing a user sees today.

AI search crawlers (8)

Build the retrieval index an assistant answers and cites from. These are the crawlers that send you traffic; blocking them is the expensive mistake in this space.

User-triggered fetchers (6)

Fetch one page because a person asked for it, right then. One human intent, one request. Blocking them produces a visible error for a real reader.

Corpus and dataset builders (8)

Crawl the web into a published or resold dataset that other people train on. Highest leverage per block, longest delay before any effect.

Search engines (13)

Classic index-and-rank crawlers. Several also feed their operator's generative answers, which is why the AI opt-out for Google and Apple is a token rather than a block.

SEO and backlink crawlers (2)

Commercial link-graph tooling. No user-facing effect either way, and usually a large share of your bot bandwidth.

Archivers (2)

Preservation crawlers. Their output is public and permanent, which makes them a separate decision from the AI one.

Link preview fetchers (1)

Read your Open Graph tags when someone shares a link. Blocking these is almost always an accident.

Tools and frameworks (3)

Not operators: crawling software anyone can run. The party behind the request is unknown, so treat them as a rate-limit question rather than a consent question.