# bedrockbot

> The web crawler an AWS customer points at URLs they chose, to build a knowledge base for a Bedrock application. AWS documents that it respects robots.txt and that the user-agent carries a per-customer suffix, so you can allow or refuse one customer's crawl by naming bedrockbot-UUID.

| field | value |
|---|---|
| operator | Amazon |
| category | AI search crawlers |
| robots.txt token | `bedrockbot` |
| user-agent contains | `bedrockbot` |
| robots.txt | obeys robots.txt (documented) |
| verify by | no published verification method |
| published IP ranges | none published |
| prefixes mirrored | 0 IPv4 / 0 IPv6 |
| operator docs | https://docs.aws.amazon.com/bedrock/latest/userguide/webcrawl-data-source-connector.html |
| last reviewed | 2026-09-01 |

## What blocking it costs you

Companies building retrieval applications on Bedrock cannot include your pages. This is a RAG block, not a training block: nothing is being trained, but nothing can cite you either.

## Full user-agent string

```
bedrockbot-UUID
```

## Block it

```
User-agent: bedrockbot
Disallow: /
```

## Allow it

```
User-agent: bedrockbot
Allow: /
```

JSON: https://www.pathwren.workers.dev/crawler/bedrockbot.json · index: https://www.pathwren.workers.dev/llms.txt