robots.txt policy: Block the crawlers with disputed robots compliance

The ones repeatedly reported as ignoring robots.txt. Included for completeness — expect to enforce this at the edge instead. A robots.txt rule is a request. For the operators in this file the request is documented as unreliable or explicitly not applicable, so the honest use of this file is as a record of intent that sits alongside a real block by user-agent or by IP at your CDN. curl -s https://www.pathwren.workers.dev/robots/block-disputed.txt