Technical

Robots Meta Tag

HTML meta tags that provide page-level instructions to search engine and AI crawler bots.

Understanding Robots Meta Tag

Robots meta tags control crawler behavior per page: <meta name="robots" content="noindex, nofollow">. Options include index/noindex (include in index), follow/nofollow (follow links), and directives like max-snippet, max-image-preview, and max-video-preview. These tags give granular control over how AI crawlers interact with specific pages.

See Also

Noindex

A robots meta tag or HTTP header directive that prevents search engines and AI crawlers from indexing specific pages.

Robots.txt

A file that tells web crawlers and AI bots which pages they can and cannot access on your website.

Related Resources

Want to Improve Your AI Visibility?

Get a comprehensive audit of your current AI visibility and learn how to improve.

Browse More Terms

301 RedirectAI CrawlersAI OverviewsAI SEOAI VisibilityCanonical URLCDN (Content Delivery Network)Citation AuthorityContent ClustersContext WindowConversational SearchCore Web VitalsCrawl BudgetDisavow ToolE-E-A-TEdge ComputingEmbeddings (Vector Embeddings)Featured SnippetsGEO (Generative Engine Optimization)HreflangInternal LinkingJSON-LDKnowledge GraphLLM OptimizationLong-Tail KeywordsMeta TagsNoindexOpen Graph ProtocolPerplexity AIProgrammatic SEOPrompt InjectionRAG (Retrieval-Augmented Generation)Rich SnippetsRobots.txtSchema.orgSearch IntentSemantic HTMLServer-Side Rendering (SSR)Static Site Generation (SSG)Structured DataTemperature (AI Parameter)Token (LLM)Topical AuthorityUser AgentXML SitemapZero-Click Search