Claude: Unblock AI Crawlers from Your Website
How to fix the problem of ai crawlers blocked by robots.txt specifically for Claude. Platform-specific solutions and prevention strategies.
Why This Happens in Claude
Claude uses ClaudeBot to discover and index content. Blanket 'Disallow: /' rules affecting AI bots. To fix this, you need to ensure Claude's crawler can access your content and that your pages are structured in a way Claude can understand.
Signs You Have This Problem in Claude
ClaudeBot and anthropic-ai are blocked in robots.txt
Claude's crawler can't index your content
Server logs show zero visits from Claude-Web
Claude can't access your site to build knowledge
How to Fix This in Claude
Follow these Claude-specific solutions:
Add to robots.txt: User-agent: Claude-Web\nAllow: /
Also allow: User-agent: anthropic-ai\nAllow: /
Remove overly restrictive Disallow rules
Monitor server logs for ClaudeBot visits
Test that Claude can access key pages
Create an llms.txt file with AI-specific guidance
Claude Technical Details
Crawler Name
ClaudeBot
User Agent
Claude-Web
Required robots.txt configuration:
User-agent: Claude-Web
Allow: /Prevent This Problem in Claude
Audit robots.txt (for Claude-Web) quarterly
Stay updated on new Claude crawler user agents
Test Claude crawler access after any robots.txt (for Claude-Web) changes
Balance security needs with Claude visibility goals