Webz.io · Corpus and dataset builders · json
User-agent: omgilibot Disallow: /
| robots.txt token | omgilibot |
| User-agent contains | omgilibot |
| Operator | Webz.io |
| Category | Corpus and dataset builders |
| robots.txt | obeys robots.txt (documented) |
| Verify by | no published verification method |
Webz.io's crawler, collecting web and forum text sold as datasets, including to model builders.
Exclusion from a commercial dataset resold to third parties.
omgilibot/0.4; +http://omgili.com
User-agent: omgilibot Allow: /
Operator documentation: https://webz.io/blog/machine-learning/
Machine copies: json ·
markdown
Policies that name this crawler:
allow-all · block-all-ai · block-datasets · maximum-ai-visibility