robots.txt policy: Block corpus and dataset builders

Refuse the crawlers whose output is a dataset other people train on: Common Crawl, AI2, Webz.io, Diffbot, ImagesiftBot. These are the highest-leverage blocks per line, because one crawl becomes many downstream training runs. It is also the block with the longest delay before it has any effect, and no effect at all on archives already published. curl -s https://www.pathwren.workers.dev/robots/block-datasets.txt