# AI Crawler Index — robots.txt user-agent tokens, one per line, 2026-09-01
# https://www.pathwren.workers.dev/data/robots-tokens.txt
AhrefsBot
AI2Bot
Ai2Bot-Dolma
Amazonbot
anthropic-ai
Applebot
Applebot-Extended
archive.org_bot
Baiduspider
bingbot
Bytespider
CCBot
ChatGPT-User
Claude-SearchBot
Claude-User
Claude-Web
ClaudeBot
cohere-ai
cohere-training-data-crawler
Diffbot
DuckAssistBot
DuckDuckBot
FacebookBot
facebookexternalhit
FirecrawlAgent
Google-CloudVertexBot
Google-Extended
Google-InspectionTool
Googlebot
Googlebot-Image
Googlebot-News
GoogleOther
GPTBot
ia_archiver
ImagesiftBot
img2dataset
meta-externalagent
meta-externalfetcher
MistralAI-User
OAI-SearchBot
omgili
omgilibot
Perplexity-User
PerplexityBot
PetalBot
Scrapy
SemrushBot
SemrushBot-OCOB
SeznamBot
Storebot-Google
TikTokSpider
Timpibot
Webzio-Extended
YandexBot
Yeti
YouBot