Allow every AI crawler and every search engine; refuse only SEO scrapers. For sites whose goal is to be found and cited by machines.
curl -s https://www.pathwren.workers.dev/robots/maximum-ai-visibility.txt >> robots.txt
If your content exists to be read by assistants — documentation, reference data, an API — every block costs you and none of them protect anything. Pair this with an llms.txt, a sitemap, and per-item JSON, and the crawlers can actually use what they find.
AI2Bot · Ai2Bot-Dolma · Amazonbot · anthropic-ai · Applebot · Applebot-Extended · archive.org_bot · Baiduspider · bingbot · Bytespider · CCBot · ChatGPT-User · Claude-SearchBot · Claude-User · Claude-Web · ClaudeBot · cohere-ai · cohere-training-data-crawler · Diffbot · DuckAssistBot · DuckDuckBot · FacebookBot · facebookexternalhit · FirecrawlAgent · Google-CloudVertexBot · Google-Extended · Google-InspectionTool · Googlebot · Googlebot-Image · Googlebot-News · GoogleOther · GPTBot · ia_archiver · ImagesiftBot · img2dataset · meta-externalagent · meta-externalfetcher · MistralAI-User · OAI-SearchBot · omgili · omgilibot · Perplexity-User · PerplexityBot · PetalBot · Scrapy · SemrushBot-OCOB · SeznamBot · Storebot-Google · TikTokSpider · Timpibot · Webzio-Extended · YandexBot · Yeti · YouBot
/robots/maximum-ai-visibility.txt · json
# AI Crawler Index — policy: maximum-ai-visibility # Maximum AI visibility # Allow every AI crawler and every search engine; refuse only SEO scrapers. For sites whose goal is to be found and cited by machines. # Generated 2026-09-01 from https://www.pathwren.workers.dev/policy/maximum-ai-visibility.html # 54 crawlers named. Paste into robots.txt at your document root. User-agent: AI2Bot Allow: / User-agent: Ai2Bot-Dolma Allow: / User-agent: Amazonbot Allow: / User-agent: anthropic-ai # control token, no crawler uses this user-agent Allow: / User-agent: Applebot Allow: / User-agent: Applebot-Extended # control token, no crawler uses this user-agent Allow: / User-agent: archive.org_bot Allow: / User-agent: Baiduspider Allow: / User-agent: bingbot Allow: / User-agent: Bytespider # compliance disputed; enforce at the edge Allow: / User-agent: CCBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: / User-agent: Claude-Web # control token, no crawler uses this user-agent Allow: / User-agent: ClaudeBot Allow: / User-agent: cohere-ai Allow: / User-agent: cohere-training-data-crawler Allow: / User-agent: Diffbot Allow: / User-agent: DuckAssistBot Allow: / User-agent: DuckDuckBot Allow: / User-agent: FacebookBot Allow: / User-agent: facebookexternalhit Allow: / User-agent: FirecrawlAgent Allow: / User-agent: Google-CloudVertexBot Allow: / User-agent: Google-Extended # control token, no crawler uses this user-agent Allow: / User-agent: Google-InspectionTool Allow: / User-agent: Googlebot Allow: / User-agent: Googlebot-Image Allow: / User-agent: Googlebot-News Allow: / User-agent: GoogleOther Allow: / User-agent: GPTBot Allow: / User-agent: ia_archiver Allow: / User-agent: ImagesiftBot Allow: / User-agent: img2dataset Allow: / User-agent: meta-externalagent Allow: / User-agent: meta-externalfetcher Allow: / User-agent: MistralAI-User Allow: / User-agent: OAI-SearchBot Allow: / User-agent: omgili Allow: / User-agent: omgilibot Allow: / User-agent: Perplexity-User # operator states robots.txt does not apply; enforce at the edge Allow: / User-agent: PerplexityBot Allow: / User-agent: PetalBot Allow: / User-agent: Scrapy Allow: / User-agent: SemrushBot-OCOB Allow: / User-agent: SeznamBot Allow: / User-agent: Storebot-Google Allow: / User-agent: TikTokSpider # compliance disputed; enforce at the edge Allow: / User-agent: Timpibot Allow: / User-agent: Webzio-Extended Allow: / User-agent: YandexBot Allow: / User-agent: Yeti Allow: / User-agent: YouBot Allow: / User-agent: * Allow: / Sitemap: https://www.pathwren.workers.dev/sitemap.xml