All Tools

Robots.txt Generator

Build a robots.txt file with an interactive editor. Configure rules for search engines and AI crawlers with presets and syntax validation.

Quick Add Bot Rules

Common paths:

Additional Directives

Generated robots.txt

User-agent: *
Disallow: /admin/

About AI Crawlers

  • GPTBot: OpenAI's crawler for training data. Block to prevent content use in AI training.
  • Google-Extended: Google's AI training crawler. Separate from regular Googlebot.
  • CCBot: Common Crawl's bot, used by many AI companies for training data.
  • Claude-Web/anthropic-ai: Anthropic's crawlers for Claude.
  • PerplexityBot: Perplexity AI's crawler for search results.

Note: Blocking AI crawlers prevents training but may reduce AI visibility. Consider your goals carefully.

Features

  • Interactive rule builder
  • AI crawler presets
  • Search engine presets
  • Sitemap reference
  • Syntax validation
  • Copy or download

SEO Tips

  • Keep title tags under 60 characters
  • Meta descriptions should be 150-160 characters
  • Include your primary keyword naturally
  • Make each page's meta unique