SemrushBot-OCOB

Semrush · AI training crawlers · json

User-agent: SemrushBot-OCOB
Disallow: /
robots.txt tokenSemrushBot-OCOB
User-agent containsSemrushBot-OCOB
OperatorSemrush
CategoryAI training crawlers
robots.txtobeys robots.txt (documented)
Verify byreverse DNS

What it is

Semrush's separately-tokenised crawler for its AI content tooling, split out so SEO crawling and AI reuse can be answered differently.

What blocking it costs you

Exclusion from Semrush's AI corpus, with its SEO crawl unaffected.

Full user-agent string

Mozilla/5.0 (compatible; SemrushBot-OCOB/1.0; +http://www.semrush.com/bot.html)

Allow it instead

User-agent: SemrushBot-OCOB
Allow: /

Operator documentation: https://www.semrush.com/bot/
Machine copies: json · markdown
Policies that name this crawler: allow-all · block-ai-training · block-all-ai · maximum-ai-visibility