Amazon · AI search crawlers · json
User-agent: bedrockbot Disallow: /
| robots.txt token | bedrockbot |
| User-agent contains | bedrockbot |
| Operator | Amazon |
| Category | AI search crawlers |
| robots.txt | obeys robots.txt (documented) |
| Verify by | no published verification method |
The web crawler an AWS customer points at URLs they chose, to build a knowledge base for a Bedrock application. AWS documents that it respects robots.txt and that the user-agent carries a per-customer suffix, so you can allow or refuse one customer's crawl by naming bedrockbot-UUID.
Companies building retrieval applications on Bedrock cannot include your pages. This is a RAG block, not a training block: nothing is being trained, but nothing can cite you either.
bedrockbot-UUID
User-agent: bedrockbot Allow: /
Operator documentation: https://docs.aws.amazon.com/bedrock/latest/userguide/webcrawl-data-source-connector.html
Machine copies: json ·
markdown
Policies that name this crawler:
allow-all · block-all-ai · allow-ai-search-only · maximum-ai-visibility