# AI Crawler Index — policy: block-seo-tools
# Block SEO and backlink crawlers
# Ahrefs, Semrush and friends. No user-facing consequence, and often the largest single slice of your bot traffic.
# Generated 2026-09-01 from https://www.pathwren.workers.dev/policy/block-seo-tools.html
# 2 crawlers named. Paste into robots.txt at your document root.

User-agent: AhrefsBot
Disallow: /

User-agent: SemrushBot
Disallow: /

User-agent: *
Allow: /

Sitemap: https://www.pathwren.workers.dev/sitemap.xml