Classic index-and-rank crawlers. Several also feed their operator's generative answers, which is why the AI opt-out for Google and Apple is a token rather than a block.
| Crawler | Token | Operator | Cost of blocking |
|---|---|---|---|
| Applebot | Applebot | Apple | You disappear from Siri, Spotlight and Safari search suggestions across Apple's install ba… |
| Baiduspider | Baiduspider | Baidu | Removal from Baidu Search, which matters only if you want Chinese-language traffic.… |
| bingbot | bingbot | Microsoft | Very high and very wide: Bing, Copilot, DuckDuckGo and several assistants that resell Bing… |
| DuckDuckBot | DuckDuckBot | DuckDuckGo | Limited on its own; the real DuckDuckGo lever is bingbot.… |
| Googlebot | Googlebot | Total. You leave Google Search. Never block this to avoid AI use; use Google-Extended inst… | |
| Googlebot-Image | Googlebot-Image | Your images stop appearing in Google Images.… | |
| Googlebot-News | Googlebot-News | Removal from Google News, with normal Search unaffected.… | |
| PetalBot | PetalBot | Huawei | Removal from Petal Search. Frequently blocked for volume rather than for policy.… |
| SeznamBot | SeznamBot | Seznam | Removal from Seznam. Also removes you from its IndexNow endpoint's usefulness.… |
| Storebot-Google | Storebot-Google | Product listings may lose shopping-specific enrichment. Irrelevant to non-commerce sites.… | |
| Timpibot | Timpibot | Timpi | Absence from a small independent index.… |
| YandexBot | YandexBot | Yandex | Removal from Yandex Search. Verify with reverse DNS to a yandex.ru, yandex.net or yandex.c… |
| Yeti | Yeti | Naver | Removal from Naver, which is most of Korean search.… |