curl -s https://www.pathwren.workers.dev/crawler/bedrockbot.json   # this page, as JSON

bedrockbot

Amazon · AI search crawlers · json

User-agent: bedrockbot
Disallow: /

What it is

The web crawler an AWS customer points at URLs they chose, to build a knowledge base for a Bedrock application. AWS documents that it respects robots.txt and that the user-agent carries a per-customer suffix, so you can allow or refuse one customer's crawl by naming bedrockbot-UUID.

What blocking it costs you

Companies building retrieval applications on Bedrock cannot include your pages. This is a RAG block, not a training block: nothing is being trained, but nothing can cite you either.

Full user-agent string

bedrockbot-UUID

Allow it instead

User-agent: bedrockbot
Allow: /

Ask this page a question, with one GET

A question about this page, answered by a call you can make right now — no key, no account, no JSON-RPC, nothing to assemble. The query string is already filled in with this page's own subject.

curl -s 'https://www.pathwren.workers.dev/tools/verification-methods?crawler=bedrockbot&s=client-dossiers'

• /tools/verification-methods?crawler=bedrockbot
  How do I prove a request really is bedrockbot? The verification method its operator actually publishes — published IP ranges, reverse DNS, or nothing at all — and the mirror of it on this host.

Every parameter above is a value the endpoint reads, not an address it visits: nothing is fetched on your behalf. Called without any parameter, each endpoint returns its own specification instead of an error. All of them.

Operator documentation: https://docs.aws.amazon.com/bedrock/latest/userguide/webcrawl-data-source-connector.html
Machine copies: json · markdown
Policies that name this crawler: allow-all · block-all-ai · allow-ai-search-only · maximum-ai-visibility