Each file names every relevant crawler explicitly, so a later change is a one-line diff and the intent stays readable. Pick the trade-off, curl the file.
curl -s https://www.pathwren.workers.dev/robots/block-ai-training.txt