Ahrefs, Semrush and friends. No user-facing consequence, and often the largest single slice of your bot traffic.
curl -s https://www.pathwren.workers.dev/robots/block-seo-tools.txt >> robots.txt
The cheapest bandwidth saving available to most sites, and the one nobody regrets. The only cost is that your own dashboards on those tools get thinner. Consider Crawl-delay first: both major operators honour it.
/robots/block-seo-tools.txt · json
# AI Crawler Index — policy: block-seo-tools # Block SEO and backlink crawlers # Ahrefs, Semrush and friends. No user-facing consequence, and often the largest single slice of your bot traffic. # Generated 2026-09-01 from https://www.pathwren.workers.dev/policy/block-seo-tools.html # 2 crawlers named. Paste into robots.txt at your document root. User-agent: AhrefsBot Disallow: / User-agent: SemrushBot Disallow: / User-agent: * Allow: / Sitemap: https://www.pathwren.workers.dev/sitemap.xml