# AI Crawler Index — one case-insensitive regex matching every crawler, 2026-09-01
(AhrefsBot|AI2Bot|Ai2Bot\-Dolma|Amazonbot|anthropic\-ai|Applebot|archive\.org_bot|Baiduspider|bingbot|Bytespider|CCBot|ChatGPT\-User|Claude\-SearchBot|Claude\-User|Claude\-Web|ClaudeBot|cohere\-ai|cohere\-training\-data\-crawler|Diffbot|DuckAssistBot|DuckDuckBot|FacebookBot|facebookexternalhit|FirecrawlAgent|Google\-CloudVertexBot|Google\-InspectionTool|Googlebot|Googlebot\-Image|Googlebot\-News|GoogleOther|GPTBot|ia_archiver|ImagesiftBot|img2dataset|meta\-externalagent|meta\-externalfetcher|MistralAI\-User|OAI\-SearchBot|omgili|omgilibot|Perplexity\-User|PerplexityBot|PetalBot|Scrapy|SemrushBot|SemrushBot\-OCOB|SeznamBot|Storebot\-Google|TikTokSpider|Timpibot|Webzio\-Extended|YandexBot|Yeti|YouBot)