Robots.txt Template for AI Crawler Access
A ready-to-use robots.txt configuration that allows all major AI crawlers (GPTBot, ClaudeBot, PerplexityBot, etc.) to index your site.
템플릿
# Robots.txt - AI Crawler Optimized
# Generated by Teoraspace
# Allow all standard crawlers
User-agent: *
Allow: /
# ===================
# AI Crawlers - ALLOW
# ===================
# OpenAI / ChatGPT
User-agent: GPTBot
Allow: /
User-agent: ChatGPT-User
Allow: /
# Anthropic / Claude
User-agent: Claude-Web
Allow: /
User-agent: anthropic-ai
Allow: /
User-agent: ClaudeBot
Allow: /
# Perplexity
User-agent: PerplexityBot
Allow: /
# Google AI (Gemini)
User-agent: Google-Extended
Allow: /
# Microsoft / Copilot (uses Bing)
User-agent: bingbot
Allow: /
# Other AI Crawlers
User-agent: Cohere-ai
Allow: /
User-agent: YouBot
Allow: /
# ===================
# Blocked Paths
# ===================
# Block admin and private areas from all crawlers
User-agent: *
Disallow: /admin/
Disallow: /api/
Disallow: /private/
Disallow: /login/
Disallow: /account/
# ===================
# Sitemap
# ===================
Sitemap: https://www.yoursite.com/sitemap.xml사용 방법
Replace 'yoursite.com' with your actual domain
Add any additional private paths to the Disallow section
Upload to your website root directory
Test with Google's robots.txt tester
Monitor server logs to verify AI crawlers are accessing your site
이럴 때 쓰세요
New websites setting up AI crawler access
Existing sites that have blocked AI crawlers
Sites migrating to AI-optimized configuration
다른 템플릿
llms.txt File Template
The llms.txt file provides direct instructions to AI systems about your site. This emerging standard helps AI understand how to represent your brand.
Organization Schema (JSON-LD)
Structured data template for your organization/company. Helps AI systems understand your brand, social profiles, and contact information.
Article Schema (JSON-LD)
Structured data template for blog posts and articles. Helps AI systems understand your content's author, date, and topic.