Crawl4AI project · Tools and frameworks · json
User-agent: Crawl4AI Disallow: /
| robots.txt token | Crawl4AI |
| User-agent contains | Crawl4AI |
| Operator | Crawl4AI project |
| Category | Tools and frameworks |
| robots.txt | operator publishes no robots.txt statement |
| Verify by | no published verification method |
An open-source LLM-oriented crawler and scraper library, run by whoever installs it. Like Scrapy, the default user-agent identifies the software and says nothing about who is behind the request.
You block a library, not an operator: the rule catches a researcher and a bulk scraper equally, and anyone who edits one config line is not caught at all.
Crawl4AI
User-agent: Crawl4AI Allow: /
Operator documentation: https://github.com/unclecode/crawl4ai
Machine copies: json ·
markdown
Policies that name this crawler:
allow-all · maximum-ai-visibility