Meta robots tag
What is a meta robots tag?
A meta robots tag is an HTML <meta> element in a page’s <head> that gives search engines instructions for that page, such as noindex (don’t show it in results), nofollow (don’t follow its links) or nosnippet (don’t show a text snippet). Without it, the default is index, follow. You can target all crawlers with name="robots" or one, such as name="googlebot".
Common values include noindex, nofollow, none (both), nosnippet, max-snippet:[number], max-image-preview:[none|standard|large], max-video-preview:[number], noimageindex, notranslate, indexifembedded and unavailable_after:[date]. For non-HTML files such as PDFs, the same rules can be sent in an HTTP header with the X-Robots-Tag.
A crawler has to fetch the page to see the tag. If robots.txt blocks the URL, Google never reads the noindex, and the URL can still be indexed from links. To remove a page from results, allow crawling and use noindex. See robots.txt for how the two interact.
Nofollow in the meta robots tag applies to every link on the page. To mark a single link, use the rel attribute on that link instead; see nofollow. Google treats nofollow as a hint rather than a strict rule.
Why meta robots tag matters for SEO
The meta robots tag is how you keep thin, private or duplicate pages, such as internal search results, thank-you pages and staging sites, out of search results.
It is also a common cause of disaster: a noindex left on after a redesign or copied from staging can remove a whole site from Google.
How to use the meta robots tag
- Add noindex to pages that shouldn’t appear in search, and leave them crawlable.
- Don’t put noindex on pages you list in your XML sitemap or point canonicals to.
- Use max-snippet or nosnippet if you need to limit how much text appears in results.
- After any launch or migration, check key templates for unwanted noindex tags.
Meta robots tag example
<!-- Keep out of search results, but follow links -->
<meta name="robots" content="noindex, follow">
<!-- Limit snippets for Google only -->
<meta name="googlebot" content="max-snippet:50, max-image-preview:large">Meta robots tag FAQ
What is the difference between robots.txt and the meta robots tag?
Robots.txt controls crawling across URLs or folders. The meta robots tag controls indexing and display of a single page, and only works if the page can be crawled.
Do I need a meta robots tag on every page?
No. The default is index, follow, so you only need the tag when you want something different.
Related SEO terms
Noindex is a robots directive, set in a meta tag or HTTP header, that tells search engines not to show a page in search results.
Nofollow is a link attribute or meta directive asking search engines not to follow or credit links. Google treats it as a hint.
X-Robots-Tag is an HTTP response header that gives search engines robots directives such as noindex, including for PDFs and images.
Robots.txt is a plain-text file at the root of a site that tells crawlers which URLs they may and may not request.
Indexing is when a search engine analyses a crawled page and stores it in its index, so it becomes eligible to appear in search results.