Crawlspace · Tools and frameworks · json
User-agent: Crawlspace Disallow: /
| robots.txt token | Crawlspace |
| User-agent contains | Crawlspace |
| Operator | Crawlspace |
| Category | Tools and frameworks |
| robots.txt | obeys robots.txt (documented) |
| Verify by | no published verification method |
A crawling platform: customers run their own crawls on it to feed agents, RAG pipelines and structured-data workflows. Like Firecrawl, the party behind any given request is the customer, not the platform.
Whatever any Crawlspace customer was building over your pages stops working. Volume and intent vary per customer, so this is a rate-limit decision more than a consent one.
Crawlspace
User-agent: Crawlspace Allow: /
Operator documentation: https://crawlspace.dev
Machine copies: json ·
markdown
Policies that name this crawler:
allow-all · maximum-ai-visibility