# Crawl4AI

> An open-source LLM-oriented crawler and scraper library, run by whoever installs it. Like Scrapy, the default user-agent identifies the software and says nothing about who is behind the request.

| field | value |
|---|---|
| operator | Crawl4AI project |
| category | Tools and frameworks |
| robots.txt token | `Crawl4AI` |
| user-agent contains | `Crawl4AI` |
| robots.txt | operator publishes no robots.txt statement |
| verify by | no published verification method |
| published IP ranges | none published |
| prefixes mirrored | 0 IPv4 / 0 IPv6 |
| operator docs | https://github.com/unclecode/crawl4ai |
| last reviewed | 2026-09-01 |

## What blocking it costs you

You block a library, not an operator: the rule catches a researcher and a bulk scraper equally, and anyone who edits one config line is not caught at all.

## Full user-agent string

```
Crawl4AI
```

## Block it

```
User-agent: Crawl4AI
Disallow: /
```

## Allow it

```
User-agent: Crawl4AI
Allow: /
```

JSON: https://www.pathwren.workers.dev/crawler/crawl4ai.json · index: https://www.pathwren.workers.dev/llms.txt