ClaudeBot
Operated by Anthropic · Powers Claude model training
ClaudeBot collects publicly available web content that may contribute to training and improving Anthropic's generative AI models. Blocking it signals that your site's future materials should be excluded from Anthropic's AI training datasets — it's unrelated to Claude's search indexing or live user requests.
What it does
ClaudeBot crawls publicly available web pages to collect content that Anthropic says may contribute to training and improving its generative AI models. It operates independently of Claude-User and Claude-SearchBot — blocking ClaudeBot has no effect on whether Claude can fetch your pages for a live user question or find them in Claude's search results.
It is the training-data counterpart to Claude-SearchBot's indexing crawl, mirroring the same three-bot pattern OpenAI uses with GPTBot and OAI-SearchBot.
Why this is optional to block
- Only affects whether your content contributes to Anthropic's future AI training datasets
- Has no effect on Claude-User's live fetches or Claude-SearchBot's indexing
- Many publishers block ClaudeBot specifically to opt out of training while keeping search/assistant access open
Identification & robots.txt
ClaudeBot
Additional verification information is available in Anthropic's official documentation.
User-agent: ClaudeBot
Disallow: /