# AGENTS.md — AI Crawler Index

> A skill file for a coding agent working against `https://www.pathwren.workers.dev`. Every command
> below runs as written: no account, no API key, no signup, and nothing to
> install to use the API at all. Canonical address `https://www.pathwren.workers.dev/AGENTS.md`; the
> same bytes answer at `/agents.md`, `/docs/AGENTS.md`, `/CLAUDE.md`, `/.well-known/agents.md`, `/.cursorrules`, `/.cursor/rules`. Regenerated by the build that writes the
> data, so it cannot drift from what the host serves. Build: 2026-09-05T22:29:56+00:00.

## What this host is

One dataset, published in every form a machine might ask for it: **150
AI crawlers from 74 operators** — what each one is for, whether it obeys
robots.txt, what blocking it costs you, and the operator-published IP ranges
that turn a user-agent claim into a verified one.

Every fact comes from the operator's own documentation and every record links
to it. Data is CC0-1.0, package code is MIT, and this host is independent of
and unaffiliated with every operator it lists.

## Install

Nothing is needed to use the API: it is static JSON over HTTPS with
`Access-Control-Allow-Origin: *` on every document, so `curl`, `fetch()` or
`requests` is the entire client. The packages are optional — they carry the
table with them and work offline after the first fetch:

```bash
pip install ai-crawler-index   # What does this user-agent claim to be?
pip install ai-crawler-verify  # Is that claim true? Check the address against the operator's own list.
pip install ai-crawler-robots  # Does your robots.txt block the crawlers you think it blocks?
pip install ai-crawler-logs    # Who was actually in your access log, and what to paste to act on it.

# ai-crawler-robots in depth: https://www.pathwren.workers.dev/ai-crawler-robots/
# ai-crawler-logs in depth: https://www.pathwren.workers.dev/ai-crawler-logs/
```

## Configure

There is no key, no token, no signup and no account to configure —
`https://www.pathwren.workers.dev/register.json` says the same thing in JSON — and no rate limit to
write a backoff for. Two settings on your side are what decide whether the
integration behaves:

```bash
# 1. The base URL, and a user-agent that names you. Identifying yourself is the
#    difference between a fetch somebody can attribute and one that looks like
#    a scraper to every edge between us.
export CRAWLER_INDEX="https://www.pathwren.workers.dev"
export UA="my-service/1.0 (+https://example.com/bot)"

# 2. Cache on the validator, not on a timer. This host rebuilds every six hours
#    and most rebuilds change nothing at all.
curl -sS -A "$UA" -D head.txt -o agents.json "$CRAWLER_INDEX/data/agents.json"
grep -i '^etag:' head.txt          # store this next to your copy
```

Send the stored validator back and a no-op rebuild costs you a 304 instead of
300 KB:

```bash
curl -sS -A "$UA" -H 'If-None-Match: "<stored-etag>"' \
     -o agents.json -w '%{http_code}\n' "$CRAWLER_INDEX/data/agents.json"
```

## Usage

### Is this request an AI crawler, and whose?

```bash
curl -sS -A "$UA" -o ua-regex.json "$CRAWLER_INDEX/data/ua-regex.json"
python3 - <<'PY'
import json, re
rx = json.load(open("ua-regex.json"))
ua = "Mozilla/5.0 (compatible; GPTBot/1.1; +https://openai.com/gptbot)"
print(re.search(rx["all"], ua, re.I).group(0))                      # GPTBot
print(bool(re.search(rx["by_category"]["ai-training"], ua, re.I)))  # True
PY
```

`all` matches every crawler in the index, `ai_only` only the AI ones, and
`by_category` is one ready-made alternation per category. They are escaped and
case-insensitive by construction, so they drop into a middleware unmodified.

### What is this crawler, in one document?

```bash
curl -sS -A "$UA" "$CRAWLER_INDEX/crawler/gptbot.json" \
| python3 -c 'import json,sys; d=json.load(sys.stdin); print(d["operator"], d["category"], d["robots_token"], d["verification_method"], sep=" | ")'
# OpenAI | ai-training | GPTBot | published-ranges
```

Slugs are stable and every one of them is listed in `/data/agents.json`.
`/crawler/<slug>.html` is the same record for a person, `/crawler/<slug>.md`
for a model.

### Did that request really come from who it claims?

A user-agent string is a claim; anybody can send one. The operators that
publish verifiable ranges are listed in `/data/ip-sources.json`, and each
one's prefixes are a single document:

```bash
curl -sS -A "$UA" "$CRAWLER_INDEX/ip-ranges/openai-gptbot.json" \
| python3 -c 'import json,sys,ipaddress as ip; d=json.load(sys.stdin); a=ip.ip_address("20.171.207.1"); print(any(a in ip.ip_network(p) for p in d["ipv4"]))'
# True — that address is inside a prefix OpenAI publishes for GPTBot
```

For a crawler whose operator publishes nothing, the record's
`verification_method` says what to do instead — usually a reverse lookup that
has to forward-confirm — and `/ip-ranges/all.json` is every range in one file.

### A robots.txt for a stance, not a hand-typed list

```bash
curl -sS -A "$UA" "$CRAWLER_INDEX/robots/block-ai-training.txt" >> robots.txt
```

Each stance is a text file at `/robots/<stance>.txt` and is explained, crawler
by crawler, at `/policy/<stance>.json`. `/policy/` lists all of them.

### Only fetch what changed

```bash
curl -sS -A "$UA" "$CRAWLER_INDEX/changes.json?since=$CURSOR" \
| python3 -c 'import json,sys; d=json.load(sys.stdin); print(d["count"], "change(s), next cursor", d["cursor"])'
```

Store `cursor` from the response, send it back as `since`, and follow `next`
while `has_more` is true. This endpoint exists so that nobody has to re-crawl
the whole index to find the two documents that moved.

### From Python, without HTTP

```bash
pip install ai-crawler-index
```

```python
from ai_crawler_index import identify, is_ai_crawler

hit = identify("Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)")
print(hit["name"], hit["operator"], hit["category"])
# ClaudeBot Anthropic ai-training

print(is_ai_crawler("Mozilla/5.0 (compatible; GPTBot/1.1; +https://openai.com/gptbot)"))
# True
```

### The source, as a git repository

```bash
git clone https://www.pathwren.workers.dev/git/ai-crawler-index.git
```

The packages above, the MCP servers behind `/mcp*` and the CC0 index data are all
in there: `data/agents.json` in the repository is the same bytes `/data/agents.json`
serves here. It is a bare repository served as static files over git's dumb HTTP
protocol, so clone and fetch work, push does not, and there is no account to make.

This is the address to record as the repository. If you point `git` at some other
document of ours instead — the package metadata names a `Data source` and no
`Source`, and clients do guess — ref discovery answers `301` to this repository
rather than `404`, and the clone completes at the right base.

### From an MCP client

```bash
curl -sS -A "$UA" -X POST "$CRAWLER_INDEX/mcp" \
     -H 'Content-Type: application/json' \
     -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'
```

The server card is at `/.well-known/mcp.json` and the page is `/mcp.html`. The
A2A agent card is at `/.well-known/agent-card.json`, and `/agents.json`
describes these same operations as flows for a planner that reads OpenAPI.

## Conventions on this host

1. **Every page has a machine twin.** `<page>.html` also answers as
   `<page>.json` and `<page>.md` at the same address. Parse the twin; do not
   scrape the HTML.
2. **Nothing is behind JavaScript.** There is none on this host. `curl` sees
   the entire document, and so does your parser.
3. **The ledger names every document.** `/documents.json` lists every path
   published here with its media type, size and validators.
4. **A match is a claim, not a proof.** Pair `/data/ua-regex.json` with
   `/ip-ranges/` before you act on an identification.
5. **The index is not the operator.** Every record links to the operator's own
   documentation; when we disagree with it, it is right and we are stale —
   `/contact` is how that gets fixed.

## Verify your integration

```bash
for p in /data/agents.json /data/ua-regex.json /crawler/gptbot.json \
         /ip-ranges/all.json /documents.json /status.json /openapi.json; do
  printf '%s %s\n' "$(curl -sS -A "$UA" -o /dev/null -w '%{http_code}' "$CRAWLER_INDEX$p")" "$p"
done
```

All seven answer `200`. `/status.json` also reports when each upstream IP
source was last fetched and whether it answered, which is the first thing to
check before blaming the data.

## What this host will not do

- **No writes.** Every documented operation is a GET or a HEAD. The MCP and
  A2A endpoints take JSON-RPC POSTs and still change nothing here.
- **No accounts, no keys, no payment.** `/.well-known/x402` publishes an empty
  `accepts` list for exactly that reason: nothing here is for sale.
- **No inference.** There is no model behind any path on this origin, and
  `/inference.html` says so at length.
- **No opinion about your robots.txt.** `/policy/` describes stances and what
  each one costs; it does not push one.
- **No affiliation.** This index is independent of every operator it lists and
  never claims otherwise.

## The machine documents, in one table

| Path | What it is |
|---|---|
| `/llms.txt` | the map of this host, written for a model |
| `/openapi.json` | every operation, with operationIds |
| `/agents.json` | the same operations as flows (agents.json 0.1.0) |
| `/.well-known/agent-permissions.json` | what an agent may do here (LAS-WG 1.0.0) |
| `/documents.json` | every document published, with size and validators |
| `/changes.json` | what moved since your cursor |
| `/data/agents.json` | the dataset itself, ~300 KB, everything in it |
| `/data/ua-regex.json` | ready-made user-agent alternations |
| `/data/ip-sources.json` | which operators publish verifiable ranges |
| `/status.json` | freshness and last HTTP status of every upstream |
| `/sitemap.md` | every page here, grouped, as markdown |

Data CC0-1.0 · package code MIT · rebuilt every six hours · independent of
every operator listed.