Cloudflare · AI search crawlers · json
User-agent: Cloudflare-AutoRAG Disallow: /
| robots.txt token | Cloudflare-AutoRAG |
| User-agent contains | Cloudflare-AutoRAG |
| Operator | Cloudflare |
| Category | AI search crawlers |
| robots.txt | obeys robots.txt (documented) |
| Verify by | no published verification method |
The crawler behind Cloudflare's AI Search / AutoRAG, which indexes a website into a retrieval index for an application. Cloudflare's own documentation warns that a bot-blocking rule on your zone will also stop this crawler and tells you to allow-list it.
Applications built on Cloudflare AI Search cannot retrieve your pages. If you are the one building the index over your own site, blocking it breaks your own product.
Cloudflare-AutoRAG
User-agent: Cloudflare-AutoRAG Allow: /
Operator documentation: https://developers.cloudflare.com/ai-search/
Machine copies: json ·
markdown
Policies that name this crawler:
allow-all · block-all-ai · allow-ai-search-only · maximum-ai-visibility