All Templates
text template

Robots.txt Template for AI Crawler Access

A ready-to-use robots.txt configuration that allows all major AI crawlers (GPTBot, ClaudeBot, PerplexityBot, etc.) to index your site.

Template

# Robots.txt - AI Crawler Optimized
# Generated by Teoraspace

# Allow all standard crawlers
User-agent: *
Allow: /

# ===================
# AI Crawlers - ALLOW
# ===================

# OpenAI / ChatGPT
User-agent: GPTBot
Allow: /

User-agent: ChatGPT-User
Allow: /

# Anthropic / Claude
User-agent: Claude-Web
Allow: /

User-agent: anthropic-ai
Allow: /

User-agent: ClaudeBot
Allow: /

# Perplexity
User-agent: PerplexityBot
Allow: /

# Google AI (Gemini)
User-agent: Google-Extended
Allow: /

# Microsoft / Copilot (uses Bing)
User-agent: bingbot
Allow: /

# Other AI Crawlers
User-agent: Cohere-ai
Allow: /

User-agent: YouBot
Allow: /

# ===================
# Blocked Paths
# ===================
# Block admin and private areas from all crawlers

User-agent: *
Disallow: /admin/
Disallow: /api/
Disallow: /private/
Disallow: /login/
Disallow: /account/

# ===================
# Sitemap
# ===================
Sitemap: https://www.yoursite.com/sitemap.xml

How to Use This Template

1

Replace 'yoursite.com' with your actual domain

2

Add any additional private paths to the Disallow section

3

Upload to your website root directory

4

Test with Google's robots.txt tester

5

Monitor server logs to verify AI crawlers are accessing your site

When to Use This

New websites setting up AI crawler access

Existing sites that have blocked AI crawlers

Sites migrating to AI-optimized configuration

Need Help Implementing?

Templates are a great starting point. For custom implementation tailored to your specific needs, we can help.

More Templates

llms.txt File Template

The llms.txt file provides direct instructions to AI systems about your site. This emerging standard helps AI understand how to represent your brand.

Organization Schema (JSON-LD)

Structured data template for your organization/company. Helps AI systems understand your brand, social profiles, and contact information.

Article Schema (JSON-LD)

Structured data template for blog posts and articles. Helps AI systems understand your content's author, date, and topic.