Fetch one page because a person asked for it, right then. One human intent, one request. Blocking them produces a visible error for a real reader.
| Crawler | Token | Operator | Cost of blocking |
|---|---|---|---|
| ChatGPT-User | ChatGPT-User | OpenAI | ChatGPT cannot open your pages when a user explicitly asks it to. The user sees a fetch fa… |
| Claude-User | Claude-User | Anthropic | Claude reports a fetch failure to a user who asked for your page by name.… |
| cohere-ai | cohere-ai | Cohere | Cohere-powered assistants cannot read your pages on request.… |
| meta-externalfetcher | meta-externalfetcher | Meta | Meta AI cannot read pages users hand it.… |
| MistralAI-User | MistralAI-User | Mistral AI | Le Chat cannot open links your readers give it.… |
| Perplexity-User | Perplexity-User | Perplexity | Not controllable via robots.txt. If you must stop it, verify by the published IP ranges an… |