{
 "name": "API — AI Crawler Index",
 "what": "Every read endpoint, with a curl. OpenAPI 3.1 at /openapi.json. No key, no rate limit, CORS open.",
 "url": "https://www.pathwren.workers.dev/api.json",
 "twin_of": "https://www.pathwren.workers.dev/api.html",
 "page": {
  "path": "/api.html",
  "url": "https://www.pathwren.workers.dev/api.html",
  "type": "text/html"
 },
 "generated_at": "2026-09-03T12:41:32+00:00",
 "generated_from": "the bytes of /api.html, by surfaces/ai-crawler-index/build.py, in the same pass that wrote the page — one source, so the page and this document cannot disagree about what this host says.",
 "license": {
  "document": "CC0-1.0",
  "url": "https://creativecommons.org/publicdomain/zero/1.0/"
 },
 "access": {
  "api_key": "none",
  "account": "none",
  "rate_limit": "none",
  "cors": "*",
  "auth": "none — every document here is a public GET"
 },
 "commands": [
  "curl -s https://www.pathwren.workers.dev/openapi.json | jq '.paths | keys | length'",
  "# every token that only affects training, as a robots.txt block",
  "curl -s https://www.pathwren.workers.dev/data/agents.json \\",
  "  | jq -r '.crawlers[] | select(.category==\"ai-training\") | \"User-agent: \\(.robots_token)\\nDisallow: /\\n\"'",
  "# is this request really GPTBot?",
  "curl -s https://www.pathwren.workers.dev/ip-ranges/openai-gptbot.json | jq -r '.ipv4[]' \\",
  "  | while read c; do python3 -c \"import ipaddress,sys;print(sys.argv[1]) if ipaddress.ip_address('$1') in ipaddress.ip_network(sys.argv[1]) else None\" \"$c\"; done",
  "# what does blocking this cost me?",
  "curl -s https://www.pathwren.workers.dev/crawler/oai-searchbot.json | jq -r .cost_of_blocking"
 ],
 "sections": [
  {
   "heading": "API",
   "text": [
    "Everything is a static file served with Access-Control-Allow-Origin: *. There is no key, no quota and no state, so the description below is complete and will not surprise you.",
    "API keys and sign-up: there are none — how to authenticate, and the one endpoint that does take a credential",
    "The complete reference, generated from the OpenAPI document rather than typed — every path, every parameter, every curl: /reference (also at /docs, /documentation, /api, /api/docs; as data at /reference.json). The page below is the short version."
   ],
   "commands": [
    "curl -s https://www.pathwren.workers.dev/openapi.json | jq '.paths | keys | length'"
   ],
   "tables": [],
   "links": [
    "/register",
    "/reference",
    "/reference.json"
   ]
  },
  {
   "heading": "Read endpoints",
   "text": [],
   "commands": [],
   "tables": [
    {
     "headers": [
      "Path",
      "Returns"
     ],
     "rows": [
      [
       "/data/agents.json",
       "all 150 crawler records + category definitions"
      ],
      [
       "/crawler/<slug>.json",
       "one crawler; slugs from agents.json"
      ],
      [
       "/operator/<slug>.json",
       "one of 74 operators and its crawlers"
      ],
      [
       "/category/<slug>.json",
       "one of 9 categories"
      ],
      [
       "/policy/<slug>.json",
       "a robots.txt policy, its rationale, and the file"
      ],
      [
       "/robots/<slug>.txt",
       "the robots.txt file itself, ready to append"
      ],
      [
       "/ip-ranges/all.json",
       "union of published prefixes, grouped by source"
      ],
      [
       "/ip-ranges/<source>.json",
       "one operator endpoint, normalised, with provenance"
      ],
      [
       "/data/ua-regex.json",
       "pre-escaped regexes, whole-list and per category"
      ],
      [
       "/status.json",
       "upstream freshness"
      ],
      [
       "/feed.json, /feed.xml",
       "what changed"
      ]
     ]
    }
   ],
   "links": []
  },
  {
   "heading": "Worked examples",
   "text": [
    "Machine description: /openapi.json · /openapi.yaml · catalogue: /.well-known/api-catalog · APIs.json: /apis.json · getting started: /.well-known/api-onboarding · prose for models: /llms.txt, /llms-full.txt"
   ],
   "commands": [
    "# every token that only affects training, as a robots.txt block",
    "curl -s https://www.pathwren.workers.dev/data/agents.json \\",
    "  | jq -r '.crawlers[] | select(.category==\"ai-training\") | \"User-agent: \\(.robots_token)\\nDisallow: /\\n\"'",
    "# is this request really GPTBot?",
    "curl -s https://www.pathwren.workers.dev/ip-ranges/openai-gptbot.json | jq -r '.ipv4[]' \\",
    "  | while read c; do python3 -c \"import ipaddress,sys;print(sys.argv[1]) if ipaddress.ip_address('$1') in ipaddress.ip_network(sys.argv[1]) else None\" \"$c\"; done",
    "# what does blocking this cost me?",
    "curl -s https://www.pathwren.workers.dev/crawler/oai-searchbot.json | jq -r .cost_of_blocking"
   ],
   "tables": [],
   "links": [
    "/openapi.json",
    "/openapi.yaml",
    "/.well-known/api-catalog",
    "/apis.json",
    "/.well-known/api-onboarding",
    "/llms.txt",
    "/llms-full.txt"
   ]
  }
 ],
 "machine_doors": [
  {
   "url": "https://www.pathwren.workers.dev/tools/?s=client-dossiers",
   "name": "6 keyless GET tools",
   "what": "The read-only MCP tools of this host as plain GET endpoints — no JSON-RPC, no key"
  },
  {
   "url": "https://www.pathwren.workers.dev/documents.json",
   "name": "documents.json",
   "what": "Every document here with its strong ETag and the date its bytes changed"
  },
  {
   "url": "https://www.pathwren.workers.dev/changes",
   "name": "changes",
   "what": "What moved since your cursor — poll this instead of re-downloading anything"
  },
  {
   "url": "https://www.pathwren.workers.dev/llms.txt",
   "name": "llms.txt",
   "what": "The whole map in one text file"
  },
  {
   "url": "https://www.pathwren.workers.dev/openapi.json",
   "name": "openapi.json",
   "what": "Every read endpoint, described formally"
  },
  {
   "url": "https://www.pathwren.workers.dev/.well-known/agent-card.json",
   "name": "agent card",
   "what": "A2A agent card"
  },
  {
   "url": "https://www.pathwren.workers.dev/mcp",
   "name": "mcp",
   "what": "MCP over JSON-RPC (POST)"
  },
  {
   "url": "https://www.pathwren.workers.dev/a2a",
   "name": "a2a",
   "what": "A2A (POST message/send)"
  }
 ],
 "links": [
  {
   "rel": "self",
   "href": "https://www.pathwren.workers.dev/api.json",
   "type": "application/json"
  },
  {
   "rel": "describes",
   "href": "https://www.pathwren.workers.dev/api.html",
   "type": "text/html",
   "title": "The page this document is the JSON twin of: API — AI Crawler Index"
  },
  {
   "rel": "changes",
   "href": "https://www.pathwren.workers.dev/changes.json?since=111",
   "type": "application/json",
   "title": "What changed since your cursor — poll this instead of re-downloading this document",
   "cursor_param": "since",
   "head_cursor": 111,
   "min_poll_seconds": 21600,
   "how": "Read `cursor` from the response and send it back as `since`. It advances only when something really changed, so an unchanged answer is proof rather than luck — about 2.5 KB, or a 304 with no body if you send back the ETag."
  },
  {
   "rel": "related",
   "href": "https://www.pathwren.workers.dev/documents.json",
   "type": "application/json",
   "title": "Every document here with its ETag and last-modified date"
  },
  {
   "rel": "related",
   "href": "https://www.pathwren.workers.dev/data/agents.json",
   "type": "application/json",
   "title": "Every crawler record in one file"
  },
  {
   "rel": "service-desc",
   "href": "https://www.pathwren.workers.dev/openapi.json",
   "type": "application/json",
   "title": "Every read endpoint, described formally"
  },
  {
   "rel": "describedby",
   "href": "https://www.pathwren.workers.dev/llms.txt",
   "type": "text/plain",
   "title": "The whole map in one text file"
  }
 ]
}