Not operators: crawling software anyone can run. The party behind the request is unknown, so treat them as a rate-limit question rather than a consent question.
| Crawler | Token | Operator | Cost of blocking |
|---|---|---|---|
| FirecrawlAgent | FirecrawlAgent | Firecrawl | Applications built on Firecrawl cannot read your pages. This is increasingly how agents fe… |
| Google-InspectionTool | Google-InspectionTool | Your own Search Console live tests stop working. Blocking this only hurts you.… | |
| Scrapy | Scrapy | Scrapy project | You block a very large tail of unattributed one-off crawlers, and also every well-behaved … |