Perplexity: Unblock AI Crawlers from Your Website
How to fix the problem of ai crawlers blocked by robots.txt specifically for Perplexity. Platform-specific solutions and prevention strategies.
Why This Happens in Perplexity
Perplexity uses PerplexityBot to discover and index content. Blanket 'Disallow: /' rules affecting AI bots. To fix this, you need to ensure Perplexity's crawler can access your content and that your pages are structured in a way Perplexity can understand.
Signs You Have This Problem in Perplexity
PerplexityBot is blocked from crawling your site
Perplexity's real-time search can't access your pages
Server logs show no PerplexityBot visits
Perplexity can't cite you because it can't reach your content
How to Fix This in Perplexity
Follow these Perplexity-specific solutions:
Add to robots.txt: User-agent: PerplexityBot\nAllow: /
Ensure real-time crawling isn't blocked
Remove restrictions affecting PerplexityBot
Monitor server logs for PerplexityBot activity
Test Perplexity search to verify access
Keep robots.txt rules permissive for AI crawlers
Perplexity Technical Details
Crawler Name
PerplexityBot
User Agent
PerplexityBot
Required robots.txt configuration:
User-agent: PerplexityBot
Allow: /Prevent This Problem in Perplexity
Audit robots.txt (for PerplexityBot) quarterly
Stay updated on new Perplexity crawler user agents
Test Perplexity crawler access after any robots.txt (for PerplexityBot) changes
Balance security needs with Perplexity visibility goals