Robots.txt

What is robots.txt?

Robots.txt is a plain-text file at the root of a website, such as example.com/robots.txt, that tells search engine crawlers which URLs they may request. It follows the Robots Exclusion Protocol. It controls crawling, not indexing: a blocked URL can still appear in search results if other pages link to it.

The file is made of groups. Each group starts with one or more `User-agent` lines naming the crawler it applies to, followed by `Disallow` and `Allow` rules that match URL paths. Google supports four fields: `user-agent`, `allow`, `disallow` and `sitemap`. Other fields, such as `crawl-delay`, are ignored by Google (Bing does read `crawl-delay`).

Rules match from the start of the path and are case-sensitive. Google supports two wildcards: `*` matches any run of characters and `$` marks the end of the URL. When an `Allow` and a `Disallow` rule both match a URL, Google uses the most specific rule, meaning the one with the longest matching path. If they are equally specific, Google uses the least restrictive one, the `Allow`.

A robots.txt file only covers the exact host, protocol and port it is served from. The file at https://example.com/robots.txt does not apply to https://shop.example.com or to http://example.com. Google caches the file for up to 24 hours and ignores anything after the first 500 KiB.

Robots.txt is a request, not a lock. Googlebot and other well-behaved crawlers obey it, but scrapers and malicious bots can ignore it, and anyone can read the file. Never use it to hide private content; use authentication instead.

Why robots.txt matters for SEO

Robots.txt is the first file a crawler fetches on your site, so a single bad line can stop search engines crawling everything. `Disallow: /` left over from a staging site is one of the most common causes of a site vanishing from Google.

Used well, it keeps crawlers out of URL spaces that waste their time: internal search results, endless filter combinations from faceted navigation, cart and checkout pages. On large sites that protects crawl budget for the pages you want found.

It is also how you opt out of some AI uses. Tokens such as `GPTBot` (OpenAI) and `Google-Extended` (Google’s token for Gemini training and grounding) are controlled with robots.txt groups; see AI crawlers.

The big misunderstanding: blocking a page in robots.txt does not remove it from Google. If the page is linked from elsewhere, Google can index the URL without its content and show it with no description. To keep a page out of results, let it be crawled and add noindex. Google stopped supporting a `noindex` rule inside robots.txt in September 2019.

How to write a robots.txt file

1. Create a UTF-8 plain-text file named exactly robots.txt and serve it at the root of each host, for example https://example.com/robots.txt. It must return a 200 status.
2. Start a group with `User-agent: *` for rules that apply to every crawler. Add separate groups only for crawlers that need different rules. A crawler follows only the most specific group that names it, so it ignores the `*` group if it has its own.
3. Disallow URL patterns that have no search value: internal search, cart, checkout, account pages and parameter combinations that create near-endless duplicates.
4. Do not block CSS, JavaScript or image files that pages need to render. Google renders pages, and blocked resources can make a page look broken or empty to it.
5. Do not disallow pages you want removed from search. Use a noindex tag or X-Robots-Tag header, and leave the page crawlable so Google can see it.
6. Add a `Sitemap:` line with the absolute URL of your XML sitemap. It can go anywhere in the file and is not tied to a group.
7. Test before you ship. Check the robots.txt report in Google Search Console for fetch errors, and check a handful of important URLs against the rules.
8. Watch the server. If robots.txt returns a 5xx error, Google stops crawling the site for up to 12 hours, then falls back to the last good copy for up to 30 days. A 404 for robots.txt is fine: Google treats it as no restrictions.

Robots.txt example

# Rules for every crawler
User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /search
Allow: /search/help
Disallow: /*?sort=

# Opt out of Gemini training and grounding
User-agent: Google-Extended
Disallow: /

Sitemap: https://example.com/sitemap.xml