X-Robots-Tag
What is X-Robots-Tag?
X-Robots-Tag is an HTTP response header that carries the same robots directives as the robots meta tag, such as noindex, nofollow and nosnippet. Because it is sent by the server rather than written in HTML, it works for any file type, including PDFs, images, videos and other documents that cannot hold a meta tag.
Any directive valid in a meta robots tag is valid in X-Robots-Tag, for example noindex, nofollow, none, nosnippet, max-snippet, max-image-preview, noimageindex and unavailable_after. You can combine them with commas, send several headers, and target one crawler by naming it first: X-Robots-Tag: googlebot: noindex.
As with the meta tag, crawlers only see the header when they fetch the URL. A file blocked in robots.txt is never fetched, so its X-Robots-Tag is never read.
Why X-Robots-Tag matters for SEO
It is the only way to keep non-HTML files out of search. If you have PDFs of price lists, internal documents or duplicates of web pages that you do not want ranking, X-Robots-Tag is how you noindex them.
It also scales. One server rule can noindex a whole folder, a file type or a staging host without touching any templates.
That power cuts both ways. A header added at the CDN or server level is invisible when you view the page source, so an accidental noindex there can remove a site from Google without anyone noticing. Check response headers, not just HTML, when you audit indexing.
How to set an X-Robots-Tag header
- Decide which URLs need it: a file type (all PDFs), a path (/internal/) or a whole host (staging).
- Add the header in your server, CDN or framework config. The example shows Nginx and Apache.
- Make sure those URLs are not disallowed in robots.txt.
- Check the live response with your browser’s developer tools (Network tab) or
curl -I https://example.com/file.pdf, and confirm in Search Console’s URL Inspection tool.
X-Robots-Tag example
# Nginx: keep all PDFs out of search
location ~* \.pdf$ {
add_header X-Robots-Tag "noindex, nofollow";
}
# Apache (.htaccess)
<FilesMatch "\.pdf$">
Header set X-Robots-Tag "noindex, nofollow"
</FilesMatch>
# The resulting response header
X-Robots-Tag: noindex, nofollowX-Robots-Tag FAQ
What is the difference between X-Robots-Tag and the meta robots tag?
They accept the same directives. The meta tag lives in a page’s HTML; X-Robots-Tag is an HTTP header, so it also works for PDFs, images and other non-HTML files.
How do I check if a page has an X-Robots-Tag?
Look at the response headers in your browser’s developer tools, run curl -I on the URL, or use URL Inspection in Google Search Console, which shows whether indexing is allowed.
Related SEO terms
Noindex is a robots directive, set in a meta tag or HTTP header, that tells search engines not to show a page in search results.
A meta robots tag is an HTML tag that tells search engines whether to index a page, follow its links and how to show it in results.
Robots.txt is a plain-text file at the root of a site that tells crawlers which URLs they may and may not request.
Nofollow is a link attribute or meta directive asking search engines not to follow or credit links. Google treats it as a hint.
Indexing is when a search engine analyses a crawled page and stores it in its index, so it becomes eligible to appear in search results.
HTTP status codes are three-digit numbers a server returns with each response to say whether a request worked, moved or failed.