---
title: "MCP server — Crawler IP Verifier"
description: "Check up to 500 addresses at once against operator-published crawler IP ranges, do prefix arithmetic on a CIDR, and export a paste-ready allowlist for nginx, Apache, HAProxy, Cloudflare, ipset or Caddy. Streamable HTTP at /mcp/netcheck, no key."
canonical: "https://www.pathwren.workers.dev/mcp-netcheck.html"
url: "https://www.pathwren.workers.dev/mcp-netcheck.md"
format: "markdown"
source: "the bytes of /mcp-netcheck.html, in the build that wrote the page"
generator: "surfaces/ai-crawler-index/build.py"
generated: "2026-09-11T21:25:22+00:00"
license: "CC0-1.0"
---

# Crawler IP Verifier — MCP server

> Check up to 500 addresses at once against operator-published crawler IP ranges, do prefix arithmetic on a CIDR, and export a paste-ready allowlist for nginx, Apache, HAProxy, Cloudflare, ipset or Caddy. Streamable HTTP at /mcp/netcheck, no key.

## Connect it: one paste, one call

```text
https://www.pathwren.workers.dev/mcp/netcheck
no auth, read-only, public
```

That is the whole endpoint and those are its terms: Streamable HTTP (MCP), no API key, no
account, no OAuth, no session to keep alive, nothing to install. Every tool is read-only, and
none of them will fetch a URL on your behalf.

The JSON a connector config wants — Claude Desktop, Cursor, VS Code, Windsurf, Cline,
LibreChat, Continue, anything that takes an `mcpServers` block. Complete as it
stands; there is no field to fill in:

```json
{
  "mcpServers": {
    "crawler-ip-verifier": {
      "type": "streamable-http",
      "url": "https://www.pathwren.workers.dev/c/mcp-connector/mcp/netcheck"
    }
  }
}
```

Claude Code takes one line instead:

```text
claude mcp add --transport http crawler-ip-verifier https://www.pathwren.workers.dev/c/mcp-connector/mcp/netcheck
```

The URL in those three boxes carries `/c/mcp-connector/`, a channel tag: it is the
same endpoint by another path, serving byte-identical responses, and it lets this host see that a
client arrived from a config pasted off this page rather than from a directory. Strip the prefix and
`https://www.pathwren.workers.dev/mcp/netcheck` is the canonical URL — both work, and nothing about the answer changes.

**Then call `no_arguments_report_the_crawler_ip_ranges_this_host_mirrors` first.** It takes no arguments at all, so there
is nothing to invent and nothing to look up before you can see this server work — the subject of
the answer is the mirror itself — every operator prefix list this host holds, and how stale each one is:

```bash
curl -s https://www.pathwren.workers.dev/c/mcp-connector/mcp/netcheck \
  -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_report_the_crawler_ip_ranges_this_host_mirrors","arguments":{}}}' \
  | jq -r '.result.content[0].text' | head -3

15/15 operator prefix lists mirrored, 1987 IPv4 + 1062 IPv6 prefixes (652 /24-equivalents), newest 58m ago, oldest 58m ago.
google-user-triggered: 529 IPv4 + 529 IPv6, fetched 58m ago
google-user-triggered-google: 248 IPv4 + 248 IPv6, fetched 58m ago
```

Those are the answer's own first three lines, from one real run on 2026-09-06 —
the mirror refreshes every six hours, so run the curl and read today's. The tool takes no arguments
because its subject is not you: it is the state of the mirror every other tool here checks against,
so every caller gets the same bytes. That state is what decides how much any verdict is worth — a
prefix list fetched four days ago and a verdict of "not in any published range" mean two different
things, and this is the tool that tells you which one you have. The rest of the answer is the
per-source table with coverage and freshness in minutes, the prefixes more than one operator
claims, and the verification method each operator documents for the crawlers that publish no list
at all. Nothing is fetched to answer it — this server makes no outbound request of any kind — and
no argument exists. `verify_my_address` is the second zero-argument call and answers
about YOU; `whoami` is the third and works unchanged on every MCP server here;
`example` is the fourth. All four are safe first calls.

A user-agent is a claim anyone can type. The operators who take that seriously publish the
networks their crawlers come from; this host mirrors every one of those lists it can find, and
this server is the batch and prefix arithmetic over the mirror — not one address and a
yes/no, which [/mcp](https://www.pathwren.workers.dev/mcp.html) already answers, but the shapes the question
actually takes in operations.

```text
# a whole log's worth of addresses, in one call
curl -s https://www.pathwren.workers.dev/mcp/netcheck \
  -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"verify_batch",
       "arguments":{"addresses":["20.171.206.1","203.0.113.9"]}}}' \
  | jq -r '.result.structuredContent.results[] | "\(.ip)\t\(.verdict)"'
```

## Tools

| Tool | What it does |
| --- | --- |
| `no_arguments_report_the_crawler_ip_ranges_this_host_mirrors` | TAKES NO ARGUMENTS. The state of the mirror every other tool here checks against: each crawler-operator prefix list this host holds, its IPv4 and IPv6 prefix counts and address total, when it was last fetched and how many minutes ago that was, which crawlers it covers, which prefixes more than one operator claims, and the verification method each operator documents for the crawlers that publish no list at all. It is the fact that decides how much any verify_batch verdict is worth. The input schema is literally empty — `arguments: {}` and no arguments key at all both work. No key, no account, no OAuth, read-only. Nothing is fetched to build it: no request leaves this edge and none is made to you. Example: arguments={} returns the per-source table with freshness in minutes. |
| `verify_my_address` | Takes no arguments. Safe to call. Deterministic. Touches no third party. THE CALL TO MAKE FIRST on this server. Takes the address you are calling from and checks it against every operator-published prefix list this host mirrors, with your user-agent taken as the claim: the verdict, the prefix that matched, which operator published it and when it was last mirrored — and, when your user-agent names a crawler this index tracks, how that operator can be verified at all. Same verify_batch a paid call would run, one address, in process. No key, no account, no OAuth, read-only, nothing to invent. Example: arguments={} returns a verdict for your own address and the sentence that a miss is not evidence of a fake. |
| `verify_batch` | Check up to 500 addresses against every mirrored operator prefix list in one call. Returns, per address, the matching prefix, the source that published it and when it was mirrored, plus a rollup by operator. Accepts {ip, claim} pairs to test a claimed identity against the ranges, and says plainly that a miss is not evidence of a fake. Example: addresses=[{ip:'66.249.66.1', claim:'Googlebot'}, {ip:'203.0.113.9', claim:'GPTBot'}] — the first matches a published Google prefix, the second matches nothing. Also callable without MCP, same implementation: GET https://www.pathwren.workers.dev/tools/verify-crawler?ip=66.249.66.1&ua=Googlebot&s=client-dossiers |
| `lookup_prefix` | Give a CIDR and get every published operator prefix that contains it, is contained by it, or partially overlaps it — the check for 'does my network collide with a crawler range' and for auditing an allowlist somebody handed you. Give an operator or source slug instead and get everything that source publishes. Example: cidr='66.249.66.0/24'. |
| `export_ip_acl` | Turn a set of operators into a config you can paste: nginx geo, nginx allow/deny, Apache, HAProxy, a Cloudflare firewall expression, an ipset script, a Caddy matcher, or a plain CIDR list. Every export carries a provenance header naming each source URL and the mirror time, and reports the rule cost. Example: operators='all', format='cidr-list', action='allow'. |
| `verification_methods` | Per crawler: whether the operator publishes IP ranges, documents reverse DNS, or publishes no verification method at all — with the exact source URL or the exact dig commands, and how fresh this mirror is. Call it before reading anything into an address that matched nothing. Reverse DNS is never run here; the command is handed back instead. Example: crawler='claudebot'. Also callable without MCP, same implementation: GET https://www.pathwren.workers.dev/tools/verification-methods?crawler=claudebot&s=client-dossiers |
| `range_stats` | Takes no arguments. Safe to call. Deterministic. Touches no third party. Per source: prefix counts, total addresses and /24-equivalents, smallest and largest prefix, the publishing URL and when it was last fetched; plus prefixes published by more than one operator. This is the sizing and freshness information an allowlist decision needs. Example: {} — no arguments at all. |
| `whoami` | Takes no arguments. Safe to call. Deterministic. Touches no third party. Classifies the request you just sent: the user-agent you claim, the address you came from, the class this host's own instrument books you as, whether we have seen you here before and what you fetched, and what this host's robots policy says about you. Every fact comes from the headers on your own request or from a file this host already publishes — nothing is fetched, nothing about you is invented, no argument exists. Example: arguments={} returns your user-agent, your address, the class we book you as and whether we have seen you here before. |
| `example` | Takes no arguments. Safe to call. Deterministic. Touches no third party. Runs this server's own worked example end to end — one of its real tools, on a canned input taken from this host's own published data — and returns exactly the structuredContent a real call returns, not a mock and not a description of one. Use it to see the shape of an answer before you decide what to send. No URL of yours is fetched and no third party is touched. Example: arguments={} runs it and returns the real answer. |

## What is behind it

15 operator-published range endpoints, re-fetched every six hours, unioned
into [/ip-ranges/all.json](https://www.pathwren.workers.dev/ip-ranges/all.json). Every answer carries the source URL
the prefix came from and the minute that source was last fetched, because a range list without
a date is a guess with better formatting. An upstream that is down keeps its last known
prefixes and is marked failed on [/status.json](https://www.pathwren.workers.dev/status.json) rather than silently
shrinking.

## Allowlists you can paste

`export_ip_acl` emits nginx `geo`, nginx allow/deny, Apache
`Require ip`, an HAProxy list, a Cloudflare firewall expression, an
`ipset` script, a Caddy `remote_ip` matcher, or a plain CIDR list —
with a header naming every source and its fetch time, and the rule cost stated up front,
because whether an allowlist is 40 lines or 2,900 decides which of those formats you can
afford.

## What it refuses to claim

**Absence from a range list is not evidence of a fake.** Most crawlers in this
index publish no ranges at all, and several document reverse DNS instead — so a miss means
"cannot be checked this way", not "impostor". `verification_methods` exists to tell
you which of the three cases you are in before you act on a result, and every answer repeats the
distinction rather than assuming you read it once.

**Reverse DNS is never performed here.** This server makes no outbound request
of any kind — no tool takes a URL or a hostname to resolve. Where an operator's documented
method is the reverse/forward lookup, you get the exact `dig` commands and an
explicit "not run", the same refusal [crawler-log-triage](https://www.pathwren.workers.dev/mcp-triage.html)
makes.

## How is this different from /mcp?

`is_verified_crawler_ip` over on [ai-crawler-index](https://www.pathwren.workers.dev/mcp.html) takes
one address and returns a boolean. That is the right tool for one question asked once. This
server takes 500 addresses and returns a table plus a rollup; takes a CIDR and returns
containment and overlap; takes a set of operators and returns a config file. No tool name is
shared with any of the other four servers here.

Protocol versions 2025-06-18, negotiated per call.
`server/discover` answers for clients on 2026-07-28, `initialize` for everyone else.
Read-only, stateless, no key. Listed in the
[official MCP Registry](https://registry.modelcontextprotocol.io/v0/servers?search=crawler-ip-verifier) as `dev.workers.pathwren.www/crawler-ip-verifier`.

## Sitemap

- [Full sitemap (XML)](https://www.pathwren.workers.dev/sitemap.xml) — every page, with dates
- [Full sitemap (markdown)](https://www.pathwren.workers.dev/sitemap.md) — the same map, readable
- [llms.txt](https://www.pathwren.workers.dev/llms.txt) — the whole host in one text file
- [documents.json](https://www.pathwren.workers.dev/documents.json) — every document, with its ETag
- [A2A agents](https://www.pathwren.workers.dev/a2a.html)
- [About and method](https://www.pathwren.workers.dev/about.html)
- [API](https://www.pathwren.workers.dev/api.html)
- [/c/<channel>/](https://www.pathwren.workers.dev/c)
- [Changelog](https://www.pathwren.workers.dev/changelog.html)
- [Compliance](https://www.pathwren.workers.dev/compliance)
- [Contact](https://www.pathwren.workers.dev/contact)
- [Impressum · Anbieterkennzeichnung](https://www.pathwren.workers.dev/impressum)
- [AI Crawler Index](https://www.pathwren.workers.dev/index.html)
- [No model runs here](https://www.pathwren.workers.dev/inference.html)
- [Legal](https://www.pathwren.workers.dev/legal)
- [MCP server](https://www.pathwren.workers.dev/mcp-doctor.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-lint.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-markdown.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-netcheck.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-robots.html)
- [MCP transport: the GET and HEAD leg](https://www.pathwren.workers.dev/mcp-transport.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-triage.html)
- [MCP server](https://www.pathwren.workers.dev/mcp.html)
- [Packages](https://www.pathwren.workers.dev/packages.html)
- [Pricing](https://www.pathwren.workers.dev/pricing)
- [Privacy](https://www.pathwren.workers.dev/privacy.html)
- [API reference](https://www.pathwren.workers.dev/reference)
- [Access, keys and sign-up](https://www.pathwren.workers.dev/register)
- [Security posture](https://www.pathwren.workers.dev/security.html)
- [Services](https://www.pathwren.workers.dev/services)
- [Upstream status](https://www.pathwren.workers.dev/status.html)
- [Terms of use](https://www.pathwren.workers.dev/terms.html)
- [Trust](https://www.pathwren.workers.dev/trust)

## Machine copies of this page

- [HTML (canonical)](https://www.pathwren.workers.dev/mcp-netcheck.html)
- [JSON](https://www.pathwren.workers.dev/mcp-netcheck.json)
- [Markdown](https://www.pathwren.workers.dev/mcp-netcheck.md) — this document

This document is a markdown rendering of [https://www.pathwren.workers.dev/mcp-netcheck.html](https://www.pathwren.workers.dev/mcp-netcheck.html), generated from that page's own bytes in the same build. The HTML page is canonical.