Noindex
A robots meta tag or HTTP header directive that prevents search engines and AI crawlers from indexing specific pages.
Understanding Noindex
The noindex directive tells crawlers not to include a page in their index. Use noindex for duplicate content, admin pages, or low-value pages to preserve crawl budget. However, be cautious with AI crawlers - pages you want AI systems to learn from should remain indexable. Check your noindex tags don't accidentally block valuable content from AI visibility.
See Also
AI Crawlers
Automated bots used by AI companies to index web content for training and retrieval in AI systems.
Crawl Budget
The number of pages a search engine or AI crawler will crawl on your site within a given timeframe.
Robots Meta Tag
HTML meta tags that provide page-level instructions to search engine and AI crawler bots.