curl -s https://www.pathwren.workers.dev/crawler/crawlspace.json   # this page, as JSON

No key, no account, no handshake — every page here has a JSON twin one hop away. Machine doors: 6 keyless GET tools · documents.json · changes · llms.txt · openapi.json · agent card · mcp · a2a

Crawlspace

Crawlspace · Tools and frameworks · json

User-agent: Crawlspace
Disallow: /


robots.txt token	`Crawlspace`
User-agent contains	`Crawlspace`
Operator	Crawlspace
Category	Tools and frameworks
robots.txt	obeys robots.txt (documented)
Verify by	no published verification method

What it is

A crawling platform: customers run their own crawls on it to feed agents, RAG pipelines and structured-data workflows. Like Firecrawl, the party behind any given request is the customer, not the platform.

What blocking it costs you

Whatever any Crawlspace customer was building over your pages stops working. Volume and intent vary per customer, so this is a rate-limit decision more than a consent one.

Full user-agent string

Crawlspace

Allow it instead

User-agent: Crawlspace
Allow: /

Ask this page a question, with one GET

A question about this page, answered by a call you can make right now — no key, no account, no JSON-RPC, nothing to assemble. The query string is already filled in with this page's own subject.

curl -s 'https://www.pathwren.workers.dev/tools/verification-methods?crawler=crawlspace&s=client-dossiers'

/tools/verification-methods?crawler=crawlspace
How do I prove a request really is Crawlspace? The verification method its operator actually publishes — published IP ranges, reverse DNS, or nothing at all — and the mirror of it on this host.

Every parameter above is a value the endpoint reads, not an address it visits: nothing is fetched on your behalf. Called without any parameter, each endpoint returns its own specification instead of an error. All of them.

Operator documentation: https://crawlspace.dev
Machine copies: json · markdown
Policies that name this crawler: allow-all · maximum-ai-visibility