Build the retrieval index an assistant answers and cites from. These are the crawlers that send you traffic; blocking them is the expensive mistake in this space.
| Crawler | Token | Operator | Cost of blocking |
|---|---|---|---|
| Amazonbot | Amazonbot | Amazon | Alexa and Amazon's assistants stop answering from your pages. Verify with reverse DNS to c… |
| Claude-SearchBot | Claude-SearchBot | Anthropic | You stop appearing in Claude's search results and citations.… |
| Claude-Web | Claude-Web | Anthropic | None in practice. Retain the rule; expect no traffic.… |
| DuckAssistBot | DuckAssistBot | DuckDuckGo | No DuckAssist answers or citations from your site. Ordinary DuckDuckGo results are unaffec… |
| Google-CloudVertexBot | Google-CloudVertexBot | Third parties can no longer build Vertex AI agents that read your site. Irrelevant to Goog… | |
| OAI-SearchBot | OAI-SearchBot | OpenAI | High. Blocking this removes you from ChatGPT search results and from the source links Chat… |
| PerplexityBot | PerplexityBot | Perplexity | You stop being indexed and cited by Perplexity, and lose the referral clicks its citations… |
| YouBot | YouBot | You.com | Removal from You.com's index and from answers built on its API.… |