Collect pages in bulk so that a model can be trained or fine-tuned on them. Blocking these removes you from future training sets and changes nothing a user sees today.
| Crawler | Token | Operator | Cost of blocking |
|---|---|---|---|
| anthropic-ai | anthropic-ai | Anthropic | None. Nothing crawls under this name today; keeping the rule is harmless insurance.… |
| Applebot-Extended | Applebot-Extended | Apple | Excluded from Apple Intelligence training. Siri, Spotlight and Safari suggestions are unaf… |
| Bytespider | Bytespider | ByteDance | Little to lose. If you want it gone, expect to block by user-agent at the edge rather than… |
| ClaudeBot | ClaudeBot | Anthropic | Content excluded from training data for future Claude models. No effect on Claude's abilit… |
| cohere-training-data-crawler | cohere-training-data-crawler | Cohere | Excluded from Cohere model training.… |
| FacebookBot | FacebookBot | Meta | Negligible today. Keep the rule; expect little traffic.… |
| Google-Extended | Google-Extended | You are excluded from Gemini grounding and Gemini training. Google Search ranking and inde… | |
| GoogleOther | GoogleOther | No effect on Search indexing. Blocks internal Google research and product fetches.… | |
| GPTBot | GPTBot | OpenAI | Your content is excluded from training data for future OpenAI models. No effect on ChatGPT… |
| meta-externalagent | meta-externalagent | Meta | Excluded from Meta AI training. Link previews on Facebook, Instagram and WhatsApp are unaf… |
| SemrushBot-OCOB | SemrushBot-OCOB | Semrush | Exclusion from Semrush's AI corpus, with its SEO crawl unaffected.… |
| TikTokSpider | TikTokSpider | ByteDance | Little to lose unless TikTok search referral matters to you.… |
| Webzio-Extended | Webzio-Extended | Webz.io | Your content is excluded from the AI-training tier of Webz.io's product while ordinary col… |