Thinkbot · Corpus and dataset builders · json
User-agent: Thinkbot Disallow: /
| robots.txt token | Thinkbot |
| User-agent contains | Thinkbot |
| Operator | Thinkbot |
| Category | Corpus and dataset builders |
| robots.txt | compliance disputed |
| Verify by | no published verification method |
Collects pages for analysis of how sites are adopting AI and automation. The ai.robots.txt dataset records the operator as not respecting robots.txt.
Exclusion from a market-research dataset. Expect to enforce this at the edge rather than in robots.txt.
Thinkbot
User-agent: Thinkbot Allow: /
Operator documentation: https://www.thinkbot.agency
Machine copies: json ·
markdown
Policies that name this crawler:
allow-all · block-all-ai · block-datasets · block-disputed · maximum-ai-visibility