# Cloudflare-AutoRAG

> The crawler behind Cloudflare's AI Search / AutoRAG, which indexes a website into a retrieval index for an application. Cloudflare's own documentation warns that a bot-blocking rule on your zone will also stop this crawler and tells you to allow-list it.

| field | value |
|---|---|
| operator | Cloudflare |
| category | AI search crawlers |
| robots.txt token | `Cloudflare-AutoRAG` |
| user-agent contains | `Cloudflare-AutoRAG` |
| robots.txt | obeys robots.txt (documented) |
| verify by | no published verification method |
| published IP ranges | none published |
| prefixes mirrored | 0 IPv4 / 0 IPv6 |
| operator docs | https://developers.cloudflare.com/ai-search/ |
| last reviewed | 2026-09-01 |

## What blocking it costs you

Applications built on Cloudflare AI Search cannot retrieve your pages. If you are the one building the index over your own site, blocking it breaks your own product.

## Full user-agent string

```
Cloudflare-AutoRAG
```

## Block it

```
User-agent: Cloudflare-AutoRAG
Disallow: /
```

## Allow it

```
User-agent: Cloudflare-AutoRAG
Allow: /
```

JSON: https://www.pathwren.workers.dev/crawler/cloudflare-autorag.json · index: https://www.pathwren.workers.dev/llms.txt