Lyrenth · AI search crawlers · json
User-agent: AIWebIndex Disallow: /
| robots.txt token | AIWebIndex |
| User-agent contains | AIWebIndex |
| Operator | Lyrenth |
| Category | AI search crawlers |
| robots.txt | obeys robots.txt (documented) |
| Verify by | no published verification method |
Builds an index of public pages and serves them to AI agents as extracted readable text, with attribution and a link back. Lyrenth publishes a crawler policy stating it does not train foundation models on what it collects and that it obeys robots.txt.
Agents reading through this index stop seeing you — including the attribution and link back that make it a referral rather than a summary.
AIWebIndex
User-agent: AIWebIndex Allow: /
Operator documentation: https://lyrenth.com/crawler-policy
Machine copies: json ·
markdown
Policies that name this crawler:
allow-all · block-all-ai · allow-ai-search-only · maximum-ai-visibility