A user-agent is a claim anyone can type. The operators who take that seriously publish the networks their crawlers come from; this host mirrors every one of those lists it can find, and this server is the batch and prefix arithmetic over the mirror — not one address and a yes/no, which /mcp already answers, but the shapes the question actually takes in operations.
# a whole log's worth of addresses, in one call
curl -s https://www.pathwren.workers.dev/mcp/netcheck \
-H 'content-type: application/json' -H 'accept: application/json, text/event-stream' \
-d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"verify_batch",
"arguments":{"addresses":["20.171.206.1","203.0.113.9"]}}}' \
| jq -r '.result.structuredContent.results[] | "\(.ip)\t\(.verdict)"'
| Tool | What it does |
|---|---|
verify_batch | Check up to 500 addresses against every mirrored operator prefix list in one call. Returns, per address, the matching prefix, the source that published it and when it was mirrored, plus a rollup by operator. Accepts {ip, claim} pairs to test a claimed identity against the ranges, and says plainly that a miss is not evidence of a fake. |
lookup_prefix | Give a CIDR and get every published operator prefix that contains it, is contained by it, or partially overlaps it — the check for 'does my network collide with a crawler range' and for auditing an allowlist somebody handed you. Give an operator or source slug instead and get everything that source publishes. |
export_ip_acl | Turn a set of operators into a config you can paste: nginx geo, nginx allow/deny, Apache, HAProxy, a Cloudflare firewall expression, an ipset script, a Caddy matcher, or a plain CIDR list. Every export carries a provenance header naming each source URL and the mirror time, and reports the rule cost. |
verification_methods | Per crawler: whether the operator publishes IP ranges, documents reverse DNS, or publishes no verification method at all — with the exact source URL or the exact dig commands, and how fresh this mirror is. Call it before reading anything into an address that matched nothing. Reverse DNS is never run here; the command is handed back instead. |
range_stats | Per source: prefix counts, total addresses and /24-equivalents, smallest and largest prefix, the publishing URL and when it was last fetched; plus prefixes published by more than one operator. This is the sizing and freshness information an allowlist decision needs. |
12 operator-published range endpoints, re-fetched every six hours, unioned into /ip-ranges/all.json. Every answer carries the source URL the prefix came from and the minute that source was last fetched, because a range list without a date is a guess with better formatting. An upstream that is down keeps its last known prefixes and is marked failed on /status.json rather than silently shrinking.
export_ip_acl emits nginx geo, nginx allow/deny, Apache
Require ip, an HAProxy list, a Cloudflare firewall expression, an
ipset script, a Caddy remote_ip matcher, or a plain CIDR list —
with a header naming every source and its fetch time, and the rule cost stated up front,
because whether an allowlist is 40 lines or 2,900 decides which of those formats you can
afford.
Absence from a range list is not evidence of a fake. Most crawlers in this
index publish no ranges at all, and several document reverse DNS instead — so a miss means
"cannot be checked this way", not "impostor". verification_methods exists to tell
you which of the three cases you are in before you act on a result, and every answer repeats the
distinction rather than assuming you read it once.
Reverse DNS is never performed here. This server makes no outbound request
of any kind — no tool takes a URL or a hostname to resolve. Where an operator's documented
method is the reverse/forward lookup, you get the exact dig commands and an
explicit "not run", the same refusal crawler-log-triage
makes.
is_verified_crawler_ip over on ai-crawler-index takes
one address and returns a boolean. That is the right tool for one question asked once. This
server takes 500 addresses and returns a table plus a rollup; takes a CIDR and returns
containment and overlap; takes a set of operators and returns a config file. No tool name is
shared with any of the other four servers here.
Protocol versions 2026-07-28, 2025-11-25, 2025-06-18, 2025-03-26, 2024-11-05, negotiated per call.
server/discover answers for clients on 2026-07-28, initialize for everyone else.
Read-only, stateless, no key. Listed in the
official
MCP Registry as dev.workers.pathwren.www/crawler-ip-verifier.