aiHit · Corpus and dataset builders · json
User-agent: aiHitBot Disallow: /
| robots.txt token | aiHitBot |
| User-agent contains | aiHitBot |
| Operator | aiHit |
| Category | Corpus and dataset builders |
| robots.txt | obeys robots.txt (documented) |
| Verify by | no published verification method |
aiHit's automated collector, building a company dataset from public company websites.
Your company record in a B2B dataset goes stale. Documented as respecting robots.txt, so the rule works.
aiHitBot
User-agent: aiHitBot Allow: /
Operator documentation: https://www.aihitdata.com/about
Machine copies: json ·
markdown
Policies that name this crawler:
allow-all · block-all-ai · block-datasets · maximum-ai-visibility