Diffbot · Corpus and dataset builders · json
User-agent: Diffbot Disallow: /
| robots.txt token | Diffbot |
| User-agent contains | Diffbot |
| Operator | Diffbot |
| Category | Corpus and dataset builders |
| robots.txt | obeys robots.txt (documented) |
| Verify by | no published verification method |
Extracts structured records from pages to build a commercial knowledge graph that is resold and used for retrieval and training.
Your facts stop entering a widely-licensed knowledge graph. Whether that is a loss depends on whether you want to be a machine-readable entity.
Mozilla/5.0 (compatible; Diffbot/0.1; +http://www.diffbot.com)
User-agent: Diffbot Allow: /
Operator documentation: https://docs.diffbot.com/
Machine copies: json ·
markdown
Policies that name this crawler:
allow-all · block-all-ai · block-datasets · maximum-ai-visibility