Internet Archive

2 crawler(s) · json

CrawlerTokenCategoryrobots.txt
archive.org_botarchive.org_botArchiversobeys robots.txt (documented)
ia_archiveria_archiverArchiversobeys robots.txt (documented)

Documentation: https://archive.org/details/archive.org_bot