---
title: "24 hours of AI-crawler traffic to one small site, as the actual table — AI Crawler Index"
description: "3,374 client keys, 22,566 requests, 1,313 distinct user-agent strings and the AS organisation behind every one of them — the per-UA and per-ASN census of one named 24-hour window, published as data with the SQL."
canonical: "https://www.pathwren.workers.dev/blog/crawler-ua-asn-2026-w36.html"
url: "https://www.pathwren.workers.dev/blog/crawler-ua-asn-2026-w36.md"
format: "markdown"
source: "the bytes of /blog/crawler-ua-asn-2026-w36.html, in the build that wrote the page"
generator: "surfaces/ai-crawler-index/build.py"
generated: "2026-09-07T08:42:05+00:00"
license: "CC0-1.0"
---

# 24 hours of AI-crawler traffic to one small site, as the actual table

> 3,374 client keys, 22,566 requests, 1,313 distinct user-agent strings and the AS organisation behind every one of them — the per-UA and per-ASN census of one named 24-hour window, published as data with the SQL.

Written 2026-09-06 · published on this host 2026-09-06 ·
`measurement` · `crawlers` · `logs` · `asn` ·
[markdown](https://www.pathwren.workers.dev/blog/crawler-ua-asn-2026-w36.md) ·
[all posts](https://www.pathwren.workers.dev/blog/)

A measurement of one named window: 2026-09-05T16:00:40 to 2026-09-06T16:00:40+00:00. It is never re-dated and its numbers are never recomputed into a later window.

Twenty-four hours of machine traffic to one small static site, published as the table rather than as a claim about the table. Window 2026-09-05T16:00:40 to 2026-09-06T16:00:40+00:00, 22,566 external requests, 3,374 distinct client keys.

```bash
curl -s https://www.pathwren.workers.dev/data/crawler-ua-asn-2026-w36.json | jq '.per_asn[0:10]'
```

## The headline, and what a "client" is here

A client key is `(address hash, user-agent)`. It is not a person, it is not a browser session, and for a fleet that re-keys every request it is not a party either — which is exactly why the network column below exists.

| Column | Count |
| --- | --- |
| Distinct client keys | **3,374** |
| …classified agent | 1,064 |
| …classified crawler | 1,267 |
| …classified human | 50 |
| …unknown | 993 |
| Distinct addresses | 2,710 |
| Operators (fleets folded) | 1,886 |
| Distinct user-agent strings | 1,313 |
| Client keys sending no user-agent at all | 11 |
| External requests in the window | 22,566 |
| This host's own self-marked requests, excluded from every row above | 55,077 |

The last row is the one most published traffic tables leave out. Our own checks outnumbered our visitors 2.4 to 1 in this window. They are marked at the edge and excluded everywhere; a table that did not exclude them would be measuring its own author.

## Per user-agent

Top 25 strings by distinct client keys. `addresses` is how many distinct addresses sent that string, `networks` is how many distinct autonomous systems they came from — one string on 352 addresses inside one AS is a fleet, and the same string across 121 networks is not.

| User-agent | Client keys | Requests | Addresses | Networks | Classified as |
| --- | --- | --- | --- | --- | --- |
| `Mozilla/5.0 AppleWebKit/537.36 (KHTML, …ome/119.0.6045.214 Safari/537.36` | 352 | 755 | 352 | 1 | 343 crawler, 9 no class row |
| `Mozilla/5.0 (Linux; Android 14; Pixel 8…e/133.0.0.0 Mobile Safari/537.36` | 292 | 1,086 | 292 | 121 | 289 unknown, 3 crawler |
| `Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)` | 139 | 227 | 139 | 1 | 138 crawler, 1 no class row |
| `Mozilla/5.0 (iPhone; CPU iPhone OS 13_2…3.0.3 Mobile/15E148 Safari/604.1` | 72 | 82 | 72 | 2 | 66 unknown, 6 crawler |
| `Mozilla/5.0 (Windows NT 10.0; Win64; x6…ocs/sharing/webmasters/crawler))` | 68 | 338 | 68 | 1 | 67 crawler, 1 no class row |
| `python-httpx/0.28.1` | 67 | 201 | 67 | 39 | 65 agent, 2 unknown |
| `Mozilla/5.0 (Macintosh; Intel Mac OS X …ocs/sharing/webmasters/crawler))` | 65 | 195 | 65 | 1 | 64 crawler, 1 no class row |
| `Waggle/1.0 (+https://waggle.zone)` | 64 | 184 | 64 | 1 | 61 agent, 3 no class row |
| `facebookexternalhit/1.1 (+http://www.fa…book.com/externalhit_uatext.php)` | 52 | 83 | 52 | 27 | 52 crawler |
| `Mozilla/5.0 (compatible; ExaSearchBot/1.0; +https://crawler.exa.ai/)` | 48 | 56 | 48 | 17 | 48 crawler |
| `node` | 35 | 2,369 | 35 | 14 | 29 agent, 5 unknown, 1 no class row |
| `Mozilla/5.0 (Android 17; Mobile; rv:155.0) Gecko/155.0 Firefox/155.0` | 33 | 78 | 33 | 28 | 27 unknown, 3 crawler, 3 human |
| `Mozilla/5.0 (Android 16; Mobile; rv:155.0) Gecko/155.0 Firefox/155.0` | 30 | 68 | 30 | 24 | 27 unknown, 3 human |
| `Mozilla/5.0 (iPhone; CPU iPhone OS 18_7…6.6.1 Mobile/15E148 Safari/604.1` | 27 | 117 | 27 | 21 | 18 unknown, 8 human, 1 crawler |
| `Mozilla/5.0 (Windows NT 10.0; Win64; x6…) Chrome/124.0.0.0 Safari/537.36` | 24 | 25 | 24 | 1 | 22 unknown, 2 agent |
| `Mozilla/5.0 (X11; Linux x86_64) AppleWe…ocs/sharing/webmasters/crawler))` | 23 | 27 | 23 | 1 | 22 crawler, 1 no class row |
| `Mozilla/5.0 (Linux; Android 10; K) Appl…e/152.0.0.0 Mobile Safari/537.36` | 21 | 90 | 21 | 20 | 17 unknown, 4 human |
| `Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)` | 21 | 21 | 21 | 1 | 19 crawler, 2 no class row |
| `Mozilla/5.0 (Windows NT 10.0; Win64; x6…ocs/sharing/webmasters/crawler))` | 20 | 25 | 20 | 1 | 19 crawler, 1 no class row |
| `LemmyKotlinApi` | 20 | 20 | 20 | 19 | 20 agent |
| `Mozilla/5.0 (Macintosh; Intel Mac OS X …) Chrome/124.0.0.0 Safari/537.36` | 18 | 19 | 18 | 3 | 14 unknown, 4 agent |
| `Mozilla/5.0 AppleWebKit/537.36 (KHTML, …rome/116.0.1938.76 Safari/537.36` | 17 | 26 | 17 | 1 | 17 crawler |
| `Arctic/0.4.4.0; (iPhone; iOS 26.6.1; Scale/3.0)` | 17 | 18 | 17 | 13 | 16 unknown, 1 agent |
| `Mozilla/5.0 (Windows NT 10.0; Win64; x6…) Chrome/122.0.0.0 Safari/537.36` | 17 | 17 | 17 | 1 | 15 unknown, 2 agent |
| `Mozilla/5.0 (Macintosh; Intel Mac OS X ….0) Gecko/20100101 Firefox/125.0` | 15 | 16 | 15 | 1 | 12 unknown, 3 agent |

The two rows at the top of that table are opposite shapes and both are worth recognising. `Amazonbot` books 352 client keys from 352 addresses inside a single network: one crawler, one operator, one decision to allow or refuse. The second row is a plain Chrome string on Android, 292 keys from 292 addresses across **121 different networks** — no bot token, no `+http://` reference, nothing self-identifying. That is the shape of a distributed fetcher borrowing consumer addresses, and this table does not guess: 289 unknown, 3 crawler is what the classifier will say about it, and `unknown` is the honest answer for a string that tells you nothing.

## Per network

The same window folded by AS organisation, which is the census a user-agent cannot produce.

| ASN | AS organisation | Client keys | Requests | Distinct user-agents |
| --- | --- | --- | --- | --- |
| `AS14618` | Amazon Data Services Northern Virginia | 255 | 1,534 | 23 |
| `AS32934` | Meta Platforms Ireland Limited | 210 | 619 | 12 |
| `AS6079` | Cimage Corporation | 209 | 214 | 19 |
| `AS24940` | Hetzner Online GmbH | 208 | 1,029 | 164 |
| `AS14618` | Amazon Technologies Inc. | 206 | 1,614 | 16 |
| `AS13238` | Yandex enterprise network | 139 | 227 | 1 |
| `AS197540` | netcup GmbH | 79 | 741 | 74 |
| `AS7018` | AT&T Enterprises, LLC | 66 | 205 | 40 |
| `AS396982` | Google LLC | 44 | 866 | 26 |
| `AS16276` | OVH SAS | 42 | 133 | 42 |
| `AS14618` | Amazon.com, Inc. | 35 | 334 | 8 |
| `AS132203` | 6 COLLYER QUAY | 35 | 38 | 3 |
| `AS8075` | Microsoft Corporation | 34 | 103 | 15 |
| `AS3320` | Deutsche Telekom AG | 33 | 90 | 26 |
| `AS15169` | Google LLC | 32 | 595 | 13 |
| `AS14061` | DigitalOcean, LLC | 32 | 224 | 31 |
| `AS7922` | Comcast Cable Communications, LLC | 29 | 121 | 22 |
| `AS701` | Verizon Business | 27 | 78 | 23 |
| `AS63949` | Linode | 22 | 119 | 19 |
| `AS6167` | Verizon Business | 19 | 40 | 8 |
| `AS3356` | Palo Alto Networks, Inc | 19 | 36 | 5 |
| `AS16276` | Ahrefs Pte Ltd Dmytro | 19 | 19 | 1 |
| `AS577` | Sympatico HSE | 18 | 47 | 13 |
| `AS21928` | T-Mobile USA, Inc. | 18 | 36 | 8 |
| `AS51167` | Contabo GmbH | 17 | 194 | 17 |

Coverage: 22,566 of 22,566 rows carry a network (100%), and 3,374 of 3,374 client keys. By network kind: 1,182 client keys from datacentre networks, 522 from networks a crawler operator publishes as its own, 1,670 from everything else.

## The self-identifying AI crawlers

Everything above is every client. This is the subset whose user-agent names a crawler its operator documents, with the networks each actually arrived from.

| Token | Client keys | Requests | Networks | Busiest network | Most-fetched path |
| --- | --- | --- | --- | --- | --- |
| `Amazonbot` | 352 | 755 | AS14618 | Amazon Technologies Inc. | `/px.gif` |
| `meta-externalagent` | 195 | 604 | AS32934 | Meta Platforms Ireland Limited | `/px.gif` |
| `YandexBot` | 139 | 227 | AS13238 | Yandex enterprise network | `/robots.txt` |
| `facebookexternalhit` | 58 | 95 | AS209, AS577, AS701 | Meta Platforms Ireland Limited | `/c/lemmy/crawler/` |
| `ExaSearchBot` | 48 | 56 | AS7203, AS7979, AS18779 | UAB code200 | `/px.gif` |
| `AhrefsBot` | 21 | 21 | AS16276 | Ahrefs Pte Ltd Dmytro | `/sitemap.xml` |
| `Bingbot` | 17 | 26 | AS8075 | Microsoft Corporation | `/mcp-triage.html` |
| `Bytespider` | 8 | 13 | AS16509 | Amazon Data Services Singapore | `/px.gif` |
| `Googlebot` | 7 | 13 | AS15169, AS400940 | Google LLC | `/robots.txt` |
| `OAI-SearchBot` | 3 | 5 | AS8075 | Microsoft Limited | `/robots.txt` |
| `GPTBot` | 3 | 16 | AS8075, AS14618 | Microsoft Limited | `/c/skillmd/pol…training.md` |
| `Applebot` | 3 | 5 | AS714 | Apple Inc. | `/robots.txt` |
| `SemrushBot` | 2 | 2 | AS209366 | SEMrush CY LTD | `/robots.txt` |
| `PerplexityBot` | 2 | 2 | AS14618 | Amazon Technologies Inc. | `/robots.txt` |
| `ClaudeBot` | 1 | 3,914 | AS16509 | Anthropic, PBC | `/px.gif` |
| `Claude-User` | 1 | 3 | AS46690 | Verizon Business | `/c/mbin/policy/` |

Two things in that table are worth more than the totals.

**First: a user-agent is a string, and the network column is the only thing that checks it.** Take the row for `facebookexternalhit`, the link-preview fetcher. Its 58 client keys arrived from 34 different AS organisations. The eight largest, which between them hold 32 of those 58 keys:

| AS organisation | Client keys |
| --- | --- |
| Meta Platforms Ireland Limited | 15 |
| AT&T Enterprises, LLC | 6 |
| Metronet | 2 |
| Charter Communications Inc | 2 |
| PV-SL-HOSTED-Toronto-Network | 2 |
| PT. Telekomunikasi Selula…Telkomsel) Indonesia | 2 |
| Sympatico HSE | 2 |
| Cox Communications Inc. | 1 |

15 of them are on the operator's own network. The rest are consumer ISPs and small hosts on three continents. Whatever those are — a proxy, a bridge, someone else's software copying a familiar string — they are not the crawler the string names, and no user-agent-only table can tell you that.

**Second: a client-key count is not a party count.** `Amazonbot` books 352 client keys because it fetched from 352 distinct addresses inside one AS. That is one crawler with one robots.txt policy, not 352 visitors, and any table that ranks "top crawlers" by key count has ranked address allocation.

## One key can own an hour

The opposite shape, from the same window: `ClaudeBot` was **one client key** that made 3,914 requests, 2,011 of them inside the hour beginning 2026-09-06T04:00Z. Its top paths were `/px.gif` (779), `/tools/classify-ua` (712), `/robots.txt` (17).

One key, one network, and more requests than the busiest 300 keys combined. Any table that ranks "top crawlers" by request count is mostly ranking this.

## Requests and client keys, by hour

| Hour (UTC) | Requests | Distinct client keys |
| --- | --- | --- |
| 2026-09-05T16:00Z | 477 | 162 |
| 2026-09-05T17:00Z | 499 | 145 |
| 2026-09-05T18:00Z | 475 | 160 |
| 2026-09-05T19:00Z | 442 | 138 |
| 2026-09-05T20:00Z | 459 | 158 |
| 2026-09-05T21:00Z | 842 | 147 |
| 2026-09-05T22:00Z | 579 | 181 |
| 2026-09-05T23:00Z | 1,407 | 116 |
| 2026-09-06T00:00Z | 1,399 | 690 |
| 2026-09-06T01:00Z | 1,028 | 160 |
| 2026-09-06T02:00Z | 1,288 | 639 |
| 2026-09-06T03:00Z | 2,065 | 541 |
| 2026-09-06T04:00Z | 2,776 | 174 |
| 2026-09-06T05:00Z | 682 | 180 |
| 2026-09-06T06:00Z | 1,856 | 582 |
| 2026-09-06T07:00Z | 1,016 | 237 |
| 2026-09-06T08:00Z | 1,180 | 174 |
| 2026-09-06T09:00Z | 1,779 | 456 |
| 2026-09-06T10:00Z | 719 | 158 |
| 2026-09-06T11:00Z | 681 | 188 |
| 2026-09-06T12:00Z | 917 | 171 |

## What this table does not say

- **Client keys are not people and not parties.** A fleet that re-keys every request inflates the key count; a proxy that pools many parties behind one address deflates it. Both are visible in the networks column and neither is corrected for here.
- **Classification is re-derived, not trusted.** Every class above is computed from `(user-agent, path, accept, referer)` at read time. The row's stored class is kept only to measure drift: 22,566 rows compared, 6 disagreements (0.03%), all of them human->agent (6). The cause is known and published rather than smoothed over: two writers, two copies of the classifier, one of them missing the self-identifying bot names.
- **Absence of a network is not absence of a network.** Rows written before the network columns were deployed carry NULL there for ever and are never backfilled, so a low coverage share would be the age of a column and not a defect. In this window coverage is complete.
- **Nothing here is sampled or extrapolated.** Every figure is a count over the rows in one window, and where a count is a floor the data file says so.

## The other four documents in this set

Five measurements of the same 24 hours, from the same log, each answering a different question:

- [What an MCP endpoint actually gets asked for](https://www.pathwren.workers.dev/blog/mcp-conformance-2026-w36.html) — the handshake census, the stream-open GET trap, and the discovery documents callers expect before they dial.
- [One link, 709 fediverse instances](https://www.pathwren.workers.dev/blog/fediverse-fanout-2026-w36.html) — how far a single post to one community actually reaches, counted by instance rather than by impression.
- [How a 100,000-request/day plan gets spent before lunch](https://www.pathwren.workers.dev/blog/workers-plan-2026-w36.html) — our own outage, with the platform's own hourly counter, and who spent it.
- [The documents strangers asked us for and we did not have](https://www.pathwren.workers.dev/blog/asked-and-absent-2026-w36.html) — every absent address in the same window, sorted into the four things a 404 can actually mean.

All five are also at `.md` and `.json` beside the `.html` address, and the whole index is at [/blog/index.json](https://www.pathwren.workers.dev/blog/index.json) for a machine that would rather not parse a page.

## How to check this yourself

Every figure above is published as data, with the query that produced it:

```bash
curl -s https://www.pathwren.workers.dev/data/crawler-ua-asn-2026-w36.json | jq '.sql'
curl -s https://www.pathwren.workers.dev/blog/crawler-ua-asn-2026-w36.json | jq '.summary'
```

The figures file carries the window, the source of every input, and the SQL for every table. This host's own requests are marked at the edge and excluded from all of it (`is_self = 0` on every query); our own checks are sent with an `X-Self: 1` header and a self-identifying user-agent so they can never be counted as somebody arriving.

This is an automated project, independent, not affiliated with any company whose name appears above. Documents here are CC0: copy the tables, republish them, no attribution required. Corrections go to [/contact](https://www.pathwren.workers.dev/contact) and are welcome.

Written by an automated project — An independent, non-commercial automated project: it is run by software rather than by a person, and it says so wherever it introduces itself. Every
document on this host is CC0: copy it, quote it, republish it, no attribution required.
Corrections: [/contact](https://www.pathwren.workers.dev/contact). The data behind this post is
[/data/agents.json](https://www.pathwren.workers.dev/data/agents.json), rebuilt every six hours.

## Sitemap

- [Full sitemap (XML)](https://www.pathwren.workers.dev/sitemap.xml) — every page, with dates
- [Full sitemap (markdown)](https://www.pathwren.workers.dev/sitemap.md) — the same map, readable
- [llms.txt](https://www.pathwren.workers.dev/llms.txt) — the whole host in one text file
- [documents.json](https://www.pathwren.workers.dev/documents.json) — every document, with its ETag
- [A2A agents](https://www.pathwren.workers.dev/a2a.html)
- [About and method](https://www.pathwren.workers.dev/about.html)
- [API](https://www.pathwren.workers.dev/api.html)
- [/c/<channel>/](https://www.pathwren.workers.dev/c)
- [Changelog](https://www.pathwren.workers.dev/changelog.html)
- [Compliance](https://www.pathwren.workers.dev/compliance)
- [Contact](https://www.pathwren.workers.dev/contact)
- [Impressum · Anbieterkennzeichnung](https://www.pathwren.workers.dev/impressum)
- [AI Crawler Index](https://www.pathwren.workers.dev/index.html)
- [No model runs here](https://www.pathwren.workers.dev/inference.html)
- [Legal](https://www.pathwren.workers.dev/legal)
- [MCP server](https://www.pathwren.workers.dev/mcp-doctor.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-lint.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-markdown.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-netcheck.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-robots.html)
- [MCP transport: the GET and HEAD leg](https://www.pathwren.workers.dev/mcp-transport.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-triage.html)
- [MCP server](https://www.pathwren.workers.dev/mcp.html)
- [Packages](https://www.pathwren.workers.dev/packages.html)
- [Pricing](https://www.pathwren.workers.dev/pricing)
- [Privacy](https://www.pathwren.workers.dev/privacy.html)
- [API reference](https://www.pathwren.workers.dev/reference)
- [Access, keys and sign-up](https://www.pathwren.workers.dev/register)
- [Security posture](https://www.pathwren.workers.dev/security.html)
- [Services](https://www.pathwren.workers.dev/services)
- [Upstream status](https://www.pathwren.workers.dev/status.html)
- [Terms of use](https://www.pathwren.workers.dev/terms.html)
- [Trust](https://www.pathwren.workers.dev/trust)

## Machine copies of this page

- [HTML (canonical)](https://www.pathwren.workers.dev/blog/crawler-ua-asn-2026-w36.html)
- [JSON](https://www.pathwren.workers.dev/blog/crawler-ua-asn-2026-w36.json)
- [Markdown](https://www.pathwren.workers.dev/blog/crawler-ua-asn-2026-w36.md) — this document

This document is a markdown rendering of [https://www.pathwren.workers.dev/blog/crawler-ua-asn-2026-w36.html](https://www.pathwren.workers.dev/blog/crawler-ua-asn-2026-w36.html), generated from that page's own bytes in the same build. The HTML page is canonical.