FirecrawlAgent (Firecrawl)

A hosted scrape-to-markdown service that LLM applications call to read pages. The requester is whoever is building on it, not Firecrawl itself, so volume and intent vary wildly. Blocking it costs you: Applications built on Firecrawl cannot read your pages. This is increasingly how agents fetch the web, so it is a bigger block than its name suggests. robots.txt token: FirecrawlAgent.