archive.org_bot

What this client asked www.pathwren.workers.dev for, when, and what it got. Every number below is from this host's own server-side request log, 2026-08-31 20:58:34Z to 2026-09-09 01:41:24Z UTC.

Mozilla/5.0 (compatible; archive.org_bot +http://archive.org/details/archive.org_bot) Zeno/d3712ae warc/v0.8.99

1028 request(s) from 15 distinct address(es), 921 distinct path(s), first seen 2026-08-31 20:58:34Z, last seen 2026-09-09 01:41:24Z UTC. 16 separate visit(s), counting a gap of more than 30 minutes as a new one, median 362.1 minutes between them.

It describes itself, inside its own user-agent, as archive.org_bot +http://archive.org/details/archive.org_bot. That is the client's own words, quoted; this page makes no claim about what it is for.

The crawler catalogue on this site has a record for it: /crawler/archive-org-bot.html — what it is for, and what blocking it costs. This page is only what it did here.

The user-agent strings, exactly as they arrived

user-agentrequestsaddressesfirst seenlast seen
Mozilla/5.0 (compatible; archive.org_bot +http://archive.org/details/archive.org_bot) Zeno/d3712ae warc/v0.8.991028152026-08-31 20:58:34Z2026-09-09 01:41:24Z

What it asked for, in the order it asked

First request to each path, oldest first, first 40 shown.

#pathat (UTC)status
1/category/dataset.html2026-08-31 20:58:34Znot recorded
2/category/archive.html2026-08-31 20:58:34Znot recorded
3/robots/block-disputed.txt2026-08-31 20:58:34Znot recorded
4/crawler/oai-searchbot.html2026-08-31 20:58:34Znot recorded
5/status.html2026-08-31 20:58:34Znot recorded
6/robots/allow-ai-search-only.txt2026-08-31 20:58:34Znot recorded
7/category/user-fetch.html2026-08-31 20:58:34Znot recorded
8/data/2026-08-31 20:58:34Znot recorded
9/robots/maximum-ai-visibility.txt2026-08-31 20:58:34Znot recorded
10/crawler/ccbot.html2026-08-31 20:58:34Znot recorded
11/category/seo.html2026-08-31 20:58:34Znot recorded
12/policy/block-disputed.html2026-08-31 20:58:34Znot recorded
13/category/tool.html2026-08-31 20:58:34Znot recorded
14/crawler/gptbot.html2026-08-31 20:58:34Znot recorded
15/crawler/perplexitybot.html2026-08-31 20:58:34Znot recorded
16/snippet/2026-08-31 20:58:34Znot recorded
17/ip-ranges/2026-08-31 20:58:34Znot recorded
18/policy/allow-ai-search-only.html2026-08-31 20:58:34Znot recorded
19/crawler/claude-searchbot.html2026-08-31 20:58:34Znot recorded
20/policy/block-all-ai.html2026-08-31 20:58:34Znot recorded
21/ip-ranges/index.html2026-08-31 20:58:34Znot recorded
22/px.gif2026-08-31 20:58:35Znot recorded
23/data/agents.json2026-08-31 20:58:35Z200
24/data/index.html2026-08-31 20:58:39Znot recorded
25/ip-ranges/openai-chatgpt-user.json2026-08-31 20:58:41Znot recorded
26/crawler/chatgpt-user.html2026-08-31 20:58:41Znot recorded
27/crawler/gptbot.json2026-08-31 20:58:41Znot recorded
28/crawler/oai-searchbot.json2026-08-31 20:58:41Znot recorded
29/ip-ranges/openai-gptbot.json2026-08-31 20:58:41Znot recorded
30/ip-ranges/openai-searchbot.json2026-08-31 20:58:41Znot recorded
31/crawler/chatgpt-user.json2026-08-31 20:58:41Znot recorded
32/crawler/claudebot.html2026-08-31 20:58:41Znot recorded
33/data/Mozilla/5.0%20(compatible;%20ClaudeBot/1.0;%20+claudebot@anthropic.com)2026-08-31 20:58:43Znot recorded
34/crawler/claudebot.json2026-08-31 20:58:43Znot recorded
35/crawler/claude-searchbot.json2026-08-31 20:58:43Znot recorded
36/data/Mozilla/5.0%20(compatible;%20Claude-SearchBot/1.0;%20+Claude-SearchBot@anthropic.com)2026-08-31 20:58:43Znot recorded
37/data/Mozilla/5.0%20(compatible;%20Claude-User/1.0;%20+Claude-User@anthropic.com)2026-08-31 20:58:43Znot recorded
38/crawler/claude-user.html2026-08-31 20:58:43Znot recorded
39/crawler/claude-user.json2026-08-31 20:58:43Znot recorded
40/crawler/anthropic-ai.html2026-08-31 20:58:43Znot recorded

Everything it asked for

pathrequestsstatus codesfirstlast
/px.gif67not recorded×45, 200×222026-08-31 20:58:35Z2026-09-09 00:08:55Z
/changes.json6200×62026-09-02 03:03:21Z2026-09-04 21:31:17Z
/about.html3200×2, not recorded×12026-08-31 20:59:44Z2026-09-02 09:04:26Z
/bot/agentgrade.html3200×32026-09-01 07:02:31Z2026-09-01 09:00:15Z
/bot/gptbot.html3200×32026-09-01 07:02:34Z2026-09-01 15:00:55Z
/bot/mcpbeat.html3200×32026-09-01 07:02:30Z2026-09-03 09:07:27Z
/crawler/bytespider.html3200×2, not recorded×12026-08-31 20:59:01Z2026-09-02 09:03:16Z
/crawler/google-extended.html3200×2, not recorded×12026-08-31 20:58:46Z2026-09-02 09:03:16Z
/data/agents.json3200×2, not recorded×12026-08-31 20:58:35Z2026-09-02 09:03:19Z
/openapi.json3200×2, not recorded×12026-08-31 20:59:36Z2026-09-02 09:03:16Z
/bot/amazonbot.html2200×22026-09-09 00:08:50Z2026-09-09 00:09:01Z
/bot/archive-org-bot.html2200×22026-09-01 07:02:30Z2026-09-05 21:15:32Z
/bot/meta-externalagent.html2200×22026-09-05 21:15:29Z2026-09-05 21:15:37Z
/bot/node.html2200×22026-09-01 07:02:16Z2026-09-01 07:02:40Z
/bot/yandexbot.html2200×22026-09-01 07:02:17Z2026-09-01 07:02:20Z
/crawler/ccbot.html2not recorded×22026-08-31 20:58:34Z2026-08-31 20:59:01Z
/crawler/claude-searchbot.html2not recorded×22026-08-31 20:58:34Z2026-08-31 20:58:43Z
/crawler/gptbot.html2not recorded×22026-08-31 20:58:34Z2026-08-31 20:58:41Z
/crawler/oai-searchbot.html2not recorded×22026-08-31 20:58:34Z2026-08-31 20:58:41Z
/crawler/perplexitybot.html2not recorded×22026-08-31 20:58:34Z2026-08-31 20:58:59Z
/documents.json2200×22026-09-02 09:03:15Z2026-09-04 21:31:14Z
/mcp.html2200×22026-09-01 03:00:04Z2026-09-01 09:00:15Z
/tools/ai-access.html2200×22026-09-02 09:03:20Z2026-09-02 09:04:57Z
/tools/classify-ua.html2200×22026-09-02 09:03:20Z2026-09-02 09:04:58Z
/tools/example.html2200×22026-09-02 09:03:20Z2026-09-02 09:04:57Z
/tools/robots-allowed.html2200×22026-09-02 09:03:20Z2026-09-02 09:04:58Z
/tools/robots-lint.html2200×22026-09-02 09:03:20Z2026-09-02 09:04:58Z
/tools/verification-methods.html2200×22026-09-02 09:03:20Z2026-09-02 09:04:58Z
/tools/verify-crawler.html2200×22026-09-02 09:03:20Z2026-09-02 09:04:58Z
/tools/whoami.html2200×22026-09-02 09:03:20Z2026-09-02 09:04:57Z
/.well-known/agent-card.json1200×12026-09-02 09:03:20Z2026-09-02 09:03:20Z
/.well-known/ai-catalog.json1200×12026-09-01 09:00:09Z2026-09-01 09:00:09Z
/.well-known/ard.json1200×12026-09-01 09:00:09Z2026-09-01 09:00:09Z
/.well-known/owners.json1200×12026-09-01 09:00:14Z2026-09-01 09:00:14Z
/a2a1200×12026-09-02 09:03:14Z2026-09-02 09:03:14Z
/a2a.html1200×12026-09-01 09:00:15Z2026-09-01 09:00:15Z
/a2a/doctor/.well-known/agent-card.json1200×12026-09-02 09:03:20Z2026-09-02 09:03:20Z
/a2a/example.json1200×12026-09-02 09:03:20Z2026-09-02 09:03:20Z
/a2a/netcheck/.well-known/agent-card.json1200×12026-09-02 09:03:20Z2026-09-02 09:03:20Z
/a2a/robots/.well-known/agent-card.json1200×12026-09-02 09:03:20Z2026-09-02 09:03:20Z
/a2a/triage/.well-known/agent-card.json1200×12026-09-02 09:03:20Z2026-09-02 09:03:20Z
/about.json1200×12026-09-02 09:04:58Z2026-09-02 09:04:58Z
/api.html1200×12026-09-01 07:02:16Z2026-09-01 07:02:16Z
/apple-touch-icon.png1200×12026-09-01 03:00:05Z2026-09-01 03:00:05Z
/bot/402explorer.html1200×12026-09-01 07:02:40Z2026-09-01 07:02:40Z
/bot/AgentGrade/1.0%20(security%20research%3B%20agentgrade.net)1404×12026-09-01 09:00:15Z2026-09-01 09:00:15Z
/bot/AgentGrade/1.0%20(security%20research;%20agentgrade.net)1307×12026-09-01 09:00:14Z2026-09-01 09:00:14Z
/bot/Amazonbot/0.11404×12026-09-09 00:09:01Z2026-09-09 00:09:01Z
/bot/GPTBot/1.41404×12026-09-01 15:00:55Z2026-09-01 15:00:55Z
/bot/Mozilla/5.0%20AppleWebKit/537.36%20(KHTML%2C%20like%20Gecko%3B%20compatible%3B%20Amazonbot/0.1%3B%20%2Bhttps%3A/developer.amazon.com/support/amazonbot)%20Chrome/119.0.6045.214%20Safari/537.361404×12026-09-09 00:09:06Z2026-09-09 00:09:06Z
/bot/Mozilla/5.0%20AppleWebKit/537.36%20(KHTML,%20like%20Gecko;%20compatible;%20Amazonbot/0.1;%20+https://developer.amazon.com/support/amazonbot)%20Chrome/119.0.6045.214%20Safari/537.361307×12026-09-09 00:09:01Z2026-09-09 00:09:01Z
/bot/YandexBot/3.01404×12026-09-01 07:02:22Z2026-09-01 07:02:22Z
/bot/a2a-directory-discovery.html1200×12026-09-01 07:02:31Z2026-09-01 07:02:31Z
/bot/a2a-directory-liveness.html1200×12026-09-01 07:02:35Z2026-09-01 07:02:35Z
/bot/a2a-registry-healthcheck.html1200×12026-09-01 07:02:33Z2026-09-01 07:02:33Z
/bot/a2a-registry-smoke.html1200×12026-09-01 07:02:19Z2026-09-01 07:02:19Z
/bot/a2a-registry.html1200×12026-09-01 07:02:27Z2026-09-01 07:02:27Z
/bot/aetherlink-public-agent-card-policy-check.html1200×12026-09-01 07:02:26Z2026-09-01 07:02:26Z
/bot/aetherlink-public-discovery-evidence.html1200×12026-09-01 07:02:31Z2026-09-01 07:02:31Z
/bot/aetherlinkdiscoveryevidence.html1200×12026-09-01 07:02:42Z2026-09-01 07:02:42Z

861 further paths are in the JSON.

What it asked for that did not exist

path it asked forwhat it gotfirst askedsince then
/data/A2A-Registry-HealthCheck/1.0404×12026-09-01 07:02:17Zstill absent
/data/bingbot/2.0404×12026-09-01 07:02:17Zstill absent
/data/ClaudeBot/1.0404×12026-09-01 07:02:17Zstill absent
/data/exaforce-mcprep/0.1%20(MCP%20server%20reputation%20scanner)404×12026-09-01 07:02:17Zstill absent
/data/python-httpx2/2.12.0404×12026-09-01 07:02:18Zstill absent
/data/YandexBot/3.0404×12026-09-01 07:02:18Zstill absent
/data/Orbit-MCP-Registry-IconResolver/1.0404×12026-09-01 07:02:18Zstill absent
/data/Mozilla/5.0404×12026-09-01 07:02:18Zstill absent
/data/Go-http-client/2.0404×12026-09-01 07:02:18Zstill absent
/data/mcp-registry/1.0404×12026-09-01 07:02:18Zstill absent
/data/GPTBot/1.4404×12026-09-01 07:02:18Zstill absent
/data/measure-mcp-schema/0.1.0404×12026-09-01 07:02:18Zstill absent
/data/mcp-checker/1.0404×12026-09-01 07:02:18Zstill absent
/data/GF-Agent-Toll-Outbound/1.6.0404×12026-09-01 07:02:18Zstill absent
/data/utopian-foundry-probe/1.0404×12026-09-01 07:02:18Zstill absent
/data/MCPCatalogSync/1.0404×12026-09-01 07:02:18Zstill absent
/data/GF-Agent-Toll-Remediation/1.0404×12026-09-01 07:02:18Zstill absent
/data/AetherLink-Public-Discovery-Evidence/1.0404×12026-09-01 07:02:18Zstill absent
/data/python-httpx/0.28.1404×12026-09-01 07:02:18Zstill absent
/data/ReactorNetty/1.0.32404×12026-09-01 07:02:18Zstill absent
/data/ReactorNetty/1.2.1404×12026-09-01 07:02:19Zstill absent
/data/A2A-Registry/1.0404×12026-09-01 07:02:19Zstill absent
/data/AetherLink-Public-Agent-Card-Policy-Check/1.0404×12026-09-01 07:02:19Zstill absent
/data/AgentSure-MCPScan/0.1404×12026-09-01 07:02:19Zstill absent
/data/curl/7.88.0404×12026-09-01 07:02:19Zstill absent
/data/PackageHound/1.0404×12026-09-01 07:02:19Zstill absent
/data/SaSame-Census-Era-Probe/1.0404×12026-09-01 07:02:19Zstill absent
/data/SaSame-MCP-Audit/0.1404×12026-09-01 07:02:19Zstill absent
/data/TAR-Discovery/1.0404×12026-09-01 07:02:19Zstill absent
/data/TAR-Directory-Indexer/1.0404×12026-09-01 07:02:19Zstill absent
/data/TAR-Health/1.0404×12026-09-01 07:02:19Zstill absent
/bot/YandexBot/3.0404×12026-09-01 07:02:22Zstill absent
/bot/text/html%2Capplication/xhtml%2Bxml%2Capplication/xml%3Bq%3D0.9%2C*/*%3Bq%3D0.8404×12026-09-01 07:02:22Zstill absent
/data/APIEvangelist-AddAPI-Control/1.0%20(%2Bhttps%3A/apis.io)404×12026-09-01 07:02:48Zstill absent
/data/Mozilla/5.0%20(compatible%3B%20archive.org_bot%20%2Bhttp%3A/archive.org/details/archive.org_bot)%20Zeno/d3712ae%20warc/v0.8.99404×12026-09-01 07:02:48Zstill absent
/data/Mozilla/5.0%20AppleWebKit/537.36%20(KHTML%2C%20like%20Gecko%3B%20compatible%3B%20ClaudeBot/1.0%3B%20%2Bclaudebot%40anthropic.com)404×12026-09-01 07:02:48Zstill absent
/data/text/html%2Capplication/xhtml%2Bxml%2Capplication/xml%3Bq%3D0.9%2C*/*%3Bq%3D0.8404×12026-09-01 07:02:48Zstill absent
/data/apis.io-submit/1.0%20(%2Bhttps%3A/apis.io)404×12026-09-01 07:02:48Zstill absent
/data/Mozilla/5.0%20AppleWebKit/537.36%20(KHTML%2C%20like%20Gecko%3B%20compatible%3B%20bingbot/2.0%3B%20%2Bhttp%3A/www.bing.com/bingbot.htm)%20Chrome/116.0.1938.76%20Safari/537.36404×12026-09-01 07:02:48Zstill absent
/data/AgentGrade/1.0%20(security%20research%3B%20agentgrade.net)404×12026-09-01 07:02:48Zstill absent
/data/agent-tools.cloud-crawler/0.1%20(%2Bhttps%3A/agent-tools.cloud)404×12026-09-01 07:02:48Zstill absent
/data/Deno/2.1.4%20(variant%3B%20SupabaseEdgeRuntime/1.74.3)404×12026-09-01 07:02:48Zstill absent
/data/lastseen-schema-probe/1.0%20(%2Bhttps%3A/lastseen.dev%3B%20introspection-only)404×12026-09-01 07:02:48Zstill absent
/data/hultra-link/1.0%20(%2Bhttps%3A/donnees.hultra.link/sondes.md)404×12026-09-01 07:02:48Zstill absent
/data/agent-tools.cloud-a2a/0.1%20(%2Bhttps%3A/agent-tools.cloud)404×12026-09-01 07:02:48Zstill absent
/data/application/json%2C%20text/event-stream%3Bq%3D0.9%2C%20*/*%3Bq%3D0.1404×12026-09-01 07:02:48Zstill absent
/data/MCPWatch/0.1.0%20(%2Bmcpwatch%40iyre.com)%20longitudinal%20MCP%20security%20research404×12026-09-01 07:02:48Zstill absent
/data/AgentReputationBot/1.0%20(%2Bhttps%3A/agentreputation.dev)404×12026-09-01 07:02:49Zstill absent
/data/mcp-schema-archive/1.0%20(%2Bhttps%3A/mcp-schema-archive.delfrost42.workers.dev)404×12026-09-01 07:02:49Zstill absent
/data/AgentIndexBot/0.1%20(%2Bhttps%3A/agents.traderszone.net%3B%20polite%20ARD%20crawler)404×12026-09-01 07:02:49Zstill absent
/data/x402-observatory/0.2%20(%2Bresearch%20collector%3B%20contact%3A%20300tsb%40gmail.com)404×12026-09-01 07:02:49Zstill absent
/data/Mozilla/5.0%20(compatible%3B%20MCPCatalogSync/1.0)404×12026-09-01 07:02:49Zstill absent
/data/agent-guild-scout/1%20(%2Bhttps%3A/agent-guild-5d5r.onrender.com/.well-known/agent-guild.json)404×12026-09-01 07:02:49Zstill absent
/data/A2A-Registry-Smoke/1.0%20(%2Bhttps%3A/a2aregistry.org)404×12026-09-01 07:02:49Zstill absent
/data/mcp-observatory/0.1.0%20(%2Bhttps%3A/github.com/yhouta/mcp-observatory%3B%20public%20transparency%20log%3B%20contact%3A%20yassine.houta%40outlook.fr)404×12026-09-01 07:02:49Zstill absent
/data/SaSameAgentAudit/0.1%20(%2Bhttps%3A/live-vps.sasame.online/.well-known/agent-card.json)404×12026-09-01 07:02:49Zstill absent
/data/pip/24.3.1%20%7B%22ci%22%3Anull%2C%22cpu%22%3A%22x86_64%22%2C%22implementation%22%3A%7B%22name%22%3A%22CPython%22%2C%22version%22%3A%223.11.9%22%7D%2C%22installer%22%3A%7B%22name%22%3A%22pip%22%2C%22version%22%3A%2224.3.1%22%7D%2C%22openssl_version%22%3A%22OpenSSL%203.0.13%2030%20Jan%202024%22%2C%22python%22%3A%223.11.9%22%2C%22setuptools_version%22%3A%2275.6.0%22%2C%22system%22%3A%7B%22name%22%3A%22Linux%22%2C%22release%22%3A%226.8.0%22%7D%7D404×12026-09-01 07:02:49Zstill absent
/data/truespar-mcp-registry/0.1%20(%2Bhttps%3A/registry.truespar.com)404×12026-09-01 07:02:49Zstill absent
/data/application/json%2C%20application/a2a%2Bjson%3Bq%3D0.9404×12026-09-01 07:02:49Zstill absent
/data/AetherLinkDiscoveryEvidence/1.0%20(%2Bhttps%3A/aetherlink.47-85-186-77.sslip.io/llms.txt)404×12026-09-01 07:02:50Zstill absent
/data/api-forge-mcp-index/1.0%20(%2Bhttps%3A/api.temsor.com/mcp/index%3B%20iletisim%3A%20altyapi%40temsor.com)404×12026-09-01 07:02:50Zstill absent
/data/Mozilla/5.0%20(compatible%3B%20AgentSure-MCPScan/0.1%3B%20%2Bhttps%3A/agentsure.tech)404×12026-09-01 07:02:50Zstill absent
/data/application/json%2Ctext/html%3Bq%3D0.9%2C*/*%3Bq%3D0.7404×12026-09-01 07:02:50Zstill absent
/data/Mozilla/5.0%20(Windows%20NT%2010.0%3B%20Win64%3B%20x64)%20AppleWebKit/537.36%20(KHTML%2C%20like%20Gecko)%20Chrome/151.0.0.0%20Safari/537.36404×12026-09-01 07:02:50Zstill absent
/data/Mozilla/5.0%20(iPhone%3B%20CPU%20iPhone%20OS%2013_2_3%20like%20Mac%20OS%20X)%20AppleWebKit/605.1.15%20(KHTML%2C%20like%20Gecko)%20Version/13.0.3%20Mobile/15E148%20Safari/604.1404×12026-09-01 07:02:50Zstill absent
/data/Mozilla/5.0%20(X11%3B%20Linux%20x86_64)%20AppleWebKit/537.36%20(KHTML%2C%20like%20Gecko)%20Chrome/148.0.0.0%20Safari/537.36404×12026-09-01 07:02:50Zstill absent
/data/Mozilla/5.0%20(Macintosh%3B%20Intel%20Mac%20OS%20X%2010_15_7)%20AppleWebKit/537.36%20(KHTML%2C%20like%20Gecko)%20Chrome/124.0.0.0%20Safari/537.36404×12026-09-01 07:02:50Zstill absent
/data/Mozilla/5.0%20(Linux%3B%20Android%2010%3B%20K)%20AppleWebKit/537.36%20(KHTML%2C%20like%20Gecko)%20Chrome/151.0.0.0%20Mobile%20Safari/537.36404×12026-09-01 07:02:50Zstill absent
/data/Mozilla/5.0%20(Windows%20NT%2010.0%3B%20Win64%3B%20x64)%20AppleWebKit/537.36%20(KHTML%2C%20like%20Gecko)%20Chrome/125.0.0.0%20Safari/537.36%20Edg/125.0.0.0404×12026-09-01 07:02:51Zstill absent
/data/Mozilla/5.0%20(Windows%20NT%2010.0%3B%20Win64%3B%20x64)%20AppleWebKit/537.36%20(KHTML%2C%20like%20Gecko)%20Chrome/144.0.0.0%20Safari/537.36404×12026-09-01 07:02:51Zstill absent
/data/Mozilla/5.0%20(Windows%20NT%2010.0)%20AppleWebKit/537.36%20(KHTML%2C%20like%20Gecko)%20Chrome/106.0.0.0%20Safari/537.36404×12026-09-01 07:02:51Zstill absent
/data/Mozilla/5.0%20(Macintosh%3B%20Intel%20Mac%20OS%20X%2010_10_1)%20AppleWebKit/600.2.5%20(KHTML%2C%20like%20Gecko)%20Version/8.0.2%20Safari/600.2.5%20(Gort)404×12026-09-01 07:02:51Zstill absent
/bot/AgentGrade/1.0%20(security%20research%3B%20agentgrade.net)404×12026-09-01 09:00:15Zstill absent
/bot/GPTBot/1.4404×12026-09-01 15:00:55Zstill absent
/data/Googlebot-Video/1.0404×12026-09-02 09:03:16Zstill absent
/data/GoogleOther-Image/1.0404×12026-09-02 09:03:18Zstill absent
/data/GoogleOther-Video/1.0404×12026-09-02 09:03:20Zstill absent
/crawler/Googlebot-Video/1.0404×12026-09-02 09:04:26Zstill absent
/crawler/GoogleOther-Image/1.0404×12026-09-02 09:04:26Zstill absent
/crawler/GoogleOther-Video/1.0404×12026-09-02 09:04:27Zstill absent
/data/Mozilla/5.0%20(Linux%3B%20Android%205.0%3B%20SM-G920A)%20AppleWebKit%20(KHTML%2C%20like%20Gecko)%20Chrome%20Mobile%20Safari%20(compatible%3B%20AdsBot-Google-Mobile%3B%20%2Bhttp%3A/www.google.com/mobile/adsbot.html)404×12026-09-02 09:04:29Zstill absent
/data/Mozilla/5.0%20(compatible%3B%20Google-Site-Verification/1.0)404×12026-09-02 09:04:31Zstill absent
/data/Mozilla/5.0%20(compatible%3B%20Google-CWS)404×12026-09-02 09:04:32Zstill absent
/data/Mozilla/5.0%20(X11%3B%20Linux%20x86_64)%20AppleWebKit/537.36%20(KHTML%2C%20like%20Gecko%3B%20compatible%3B%20Google-Agent%3B%20%2Bhttps%3A/developers.google.com/crawling/docs/crawlers-fetchers/google-agent)%20Chrome/W.X.Y.Z%20Safari/537.36404×12026-09-02 09:04:33Zstill absent
/data/Mozilla/5.0%20(compatible%3B%20SiteAuditBot/0.97%3B%20%2Bhttp%3A/www.semrush.com/bot.html)404×12026-09-02 09:04:37Zstill absent
/data/Mozilla/5.0%20(compatible%3B%20SemrushBot-BA%3B%20%2Bhttp%3A/www.semrush.com/bot.html)404×12026-09-02 09:04:37Zstill absent
/data/Mozilla/5.0%20(compatible%3B%20SemrushBot-SI%3B%20%2Bhttp%3A/www.semrush.com/bot.html)404×12026-09-02 09:04:37Zstill absent
/data/Mozilla/5.0%20(compatible%3B%20SemrushBot-SWA%3B%20%2Bhttp%3A/www.semrush.com/bot.html)404×12026-09-02 09:04:37Zstill absent
/data/Mozilla/5.0%20(compatible%3B%20SplitSignalBot%3B%20%2Bhttp%3A/www.semrush.com/bot.html)404×12026-09-02 09:04:38Zstill absent
/data/Mozilla/5.0%20(compatible%3B%20SemrushBot-FT%3B%20%2Bhttp%3A/www.semrush.com/bot.html)404×12026-09-02 09:04:38Zstill absent
/data/Mozilla/5.0%20(compatible%3B%20SemrushBot-ESI%3B%20%2Bhttp%3A/www.semrush.com/bot.html)404×12026-09-02 09:04:38Zstill absent
/data/Mozilla/5.0%20(compatible%3B%20DotBot/1.2%3B%20%2Bhttps%3A/opensiteexplorer.org/dotbot%3B%20help%40moz.com)404×12026-09-02 09:04:40Zstill absent
/data/rogerbot/1.2%20(http%3A/moz.com/help/pro/what-is-rogerbot-%2C%20rogerbot-crawler%40moz.com)404×12026-09-02 09:04:40Zstill absent
/data/serpstatbot/2.1%20(advanced%20backlink%20tracking%20bot%3B%20https%3A/serpstatbot.com/%3B%20abuse%40serpstatbot.com)404×12026-09-02 09:04:41Zstill absent
/data/Mozilla/5.0%20(compatible%3B%20SEOkicks%3B%20%2Bhttps%3A/www.seokicks.de/robot.html)404×12026-09-02 09:04:41Zstill absent
/data/Mozilla/5.0%20(compatible%3B%20MojeekBot/0.11%3B%20%2Bhttps%3A/www.mojeek.com/bot.html)404×12026-09-02 09:04:41Zstill absent
/data/Mozilla/5.0%20(compatible%3B%20Pinterestbot/1.0%3B%20%2Bhttps%3A/www.pinterest.com/bot.html)404×12026-09-02 09:04:43Zstill absent
/data/Mozilla/5.0%20(compatible%3B%20panscient.com)404×12026-09-02 09:04:44Zstill absent
/data/AwarioSmartBot/1.0%20(%2Bhttps%3A/awario.com/bots.html%3B%20bots%40awario.com)404×12026-09-02 09:04:50Zstill absent
/data/AwarioRssBot/1.0%20(%2Bhttps%3A/awario.com/bots.html%3B%20bots%40awario.com)404×12026-09-02 09:04:51Zstill absent
/crawler/Mozilla/5.0%20(Linux%3B%20Android%205.0%3B%20SM-G920A)%20AppleWebKit%20(KHTML%2C%20like%20Gecko)%20Chrome%20Mobile%20Safari%20(compatible%3B%20AdsBot-Google-Mobile%3B%20%2Bhttp%3A/www.google.com/mobile/adsbot.html)404×12026-09-02 09:04:58Zstill absent
/crawler/Mozilla/5.0%20(compatible%3B%20Google-Site-Verification/1.0)404×12026-09-02 09:04:58Zstill absent
/crawler/Mozilla/5.0%20(X11%3B%20Linux%20x86_64)%20AppleWebKit/537.36%20(KHTML%2C%20like%20Gecko%3B%20compatible%3B%20Google-Agent%3B%20%2Bhttps%3A/developers.google.com/crawling/docs/crawlers-fetchers/google-agent)%20Chrome/W.X.Y.Z%20Safari/537.36404×12026-09-02 09:04:58Zstill absent
/crawler/Mozilla/5.0%20(compatible%3B%20Google-CWS)404×12026-09-02 09:04:58Zstill absent
/crawler/Mozilla/5.0%20(compatible%3B%20SiteAuditBot/0.97%3B%20%2Bhttp%3A/www.semrush.com/bot.html)404×12026-09-02 09:04:58Zstill absent
/crawler/Mozilla/5.0%20(compatible%3B%20SemrushBot-SWA%3B%20%2Bhttp%3A/www.semrush.com/bot.html)404×12026-09-02 09:04:58Zstill absent
/crawler/Mozilla/5.0%20(compatible%3B%20SemrushBot-BA%3B%20%2Bhttp%3A/www.semrush.com/bot.html)404×12026-09-02 09:04:58Zstill absent
/crawler/Mozilla/5.0%20(compatible%3B%20SemrushBot-SI%3B%20%2Bhttp%3A/www.semrush.com/bot.html)404×12026-09-02 09:04:58Zstill absent
/crawler/Mozilla/5.0%20(compatible%3B%20SplitSignalBot%3B%20%2Bhttp%3A/www.semrush.com/bot.html)404×12026-09-02 09:04:58Zstill absent
/crawler/Mozilla/5.0%20(compatible%3B%20SemrushBot-ESI%3B%20%2Bhttp%3A/www.semrush.com/bot.html)404×12026-09-02 09:04:58Zstill absent
/crawler/Mozilla/5.0%20(compatible%3B%20DotBot/1.2%3B%20%2Bhttps%3A/opensiteexplorer.org/dotbot%3B%20help%40moz.com)404×12026-09-02 09:04:58Zstill absent
/crawler/Mozilla/5.0%20(compatible%3B%20SemrushBot-FT%3B%20%2Bhttp%3A/www.semrush.com/bot.html)404×12026-09-02 09:04:58Zstill absent
/crawler/rogerbot/1.2%20(http%3A/moz.com/help/pro/what-is-rogerbot-%2C%20rogerbot-crawler%40moz.com)404×12026-09-02 09:04:58Zstill absent
/crawler/serpstatbot/2.1%20(advanced%20backlink%20tracking%20bot%3B%20https%3A/serpstatbot.com/%3B%20abuse%40serpstatbot.com)404×12026-09-02 09:04:58Zstill absent
/crawler/Mozilla/5.0%20(compatible%3B%20SEOkicks%3B%20%2Bhttps%3A/www.seokicks.de/robot.html)404×12026-09-02 09:04:58Zstill absent
/crawler/Mozilla/5.0%20(compatible%3B%20MojeekBot/0.11%3B%20%2Bhttps%3A/www.mojeek.com/bot.html)404×12026-09-02 09:04:58Zstill absent
/crawler/Mozilla/5.0%20(compatible%3B%20Pinterestbot/1.0%3B%20%2Bhttps%3A/www.pinterest.com/bot.html)404×12026-09-02 09:04:58Zstill absent
/crawler/AwarioRssBot/1.0%20(%2Bhttps%3A/awario.com/bots.html%3B%20bots%40awario.com)404×12026-09-02 09:04:58Zstill absent
/crawler/AwarioSmartBot/1.0%20(%2Bhttps%3A/awario.com/bots.html%3B%20bots%40awario.com)404×12026-09-02 09:04:58Zstill absent
/crawler/Mozilla/5.0%20(compatible%3B%20panscient.com)404×12026-09-02 09:04:58Zstill absent
/message/send%20(the%20v1.0%20name%20SendMessage%20is%20accepted%20too)404×12026-09-04 21:31:17Zstill absent
/bot/image/avif%2Cimage/webp%2Cimage/apng%2Cimage/svg%2Bxml%2Cimage/*%2C*/*%3Bq%3D0.8404×12026-09-05 21:15:38Zstill absent
/bot/text/html%2Capplication/xhtml%2Bxml%2Capplication/xml%3Bq%3D0.9%2Cimage/avif%2Cimage/webp%2Cimage/apng%2C*/*%3Bq%3D0.8%2Capplication/signed-exchange%3Bv%3Db3%3Bq%3D0.7404×12026-09-05 21:15:38Zstill absent
/bot/Amazonbot/0.1404×12026-09-09 00:09:01Zstill absent
/bot/Mozilla/5.0%20AppleWebKit/537.36%20(KHTML%2C%20like%20Gecko%3B%20compatible%3B%20Amazonbot/0.1%3B%20%2Bhttps%3A/developer.amazon.com/support/amazonbot)%20Chrome/119.0.6045.214%20Safari/537.36404×12026-09-09 00:09:06Zstill absent

What it got, and what it sent

Status codes
200×510, not recorded×309, 404×125, 307×84
Accept headers
none sent
Bytes served
6154945
Attributed to a channel
direct (45), mcp-registry-official (4), ard (3), mastodon (2), freepublicapis (1)
Our instrument classed it
crawler×1028 — that is our classifier's label from the user-agent, not the operator's, and it has been wrong before.

How to verify it is really them

no published verification method

From the crawler record at /crawler/archive-org-bot.html, which cites https://archive.org/details/archive.org_bot — one source for both pages.

Ask this page a question, with one GET

A question about this page, answered by a call you can make right now — no key, no account, no JSON-RPC, nothing to assemble. The query string is already filled in with this page's own subject.

curl -s 'https://www.pathwren.workers.dev/tools/classify-ua?ua=Mozilla%2F5.0%20%28compatible%3B%20archive.org_bot%20%2Bhttp%3A%2F%2Farchive.org%2Fdetails%2Farchive.org_bot%29%20Zeno%2Fd3712ae%20warc%2Fv0.8.99&s=client-dossiers'

Every parameter above is a value the endpoint reads, not an address it visits: nothing is fetched on your behalf. Called without any parameter, each endpoint returns its own specification instead of an error. All of them.

What it publishes about you afterwards

Not observed. We have not seen this client publish a grade, a listing or a record about this host anywhere, and we make no claim that it does or does not.

Its own documentation

It names http://archive.org/details/archive.org_bot in its own user-agent. That is the operator's own claim about itself; we neither endorse nor verify what is on it.

This page as data

curl -s https://www.pathwren.workers.dev/bot/archive-org-bot.json
curl -s https://www.pathwren.workers.dev/data/observed-clients.json   # every client, one request

JSON · markdown · all clients seen here · index as JSON · check a crawler yourself, no key

Method, and what this page will not say

Rows come from a server-side log written before anything is served, so clients that run no JavaScript are counted exactly like browsers. The window is the whole life of the log, 2026-08-31 20:58:34Z to 2026-09-09 01:41:24Z UTC, and it is stated on every number because a count without a window is not a fact. Addresses are stored as salted hashes and counted, never printed. Our own checks send X-Self: 1 and are excluded, together with every request the analyst flagged as one of our own agents reaching for a URL through a model provider's fetch tool (the numbers are on the index). Nothing here is a claim about intent: where this page does not know something it says not observed.