Indexing

What is indexing?

Indexing is the step where a search engine processes a crawled page and stores information about it in its index, a huge database of web content. Only indexed pages can appear in search results. During indexing, the engine reads the text, titles, links, images and structured data, and decides which URL is the canonical version.

Indexing happens after crawling and, for pages that use JavaScript, after rendering. Google analyses the page’s content and meaning, identifies its language and the queries it might answer, and groups duplicates together. From a group of similar pages it chooses one canonical URL to show; the others are treated as duplicates. Your canonical tag is a strong hint, not a command.

You can control indexing in a few ways. A meta robots tag or X-Robots-Tag header with noindex keeps a page out. A canonical tag suggests which duplicate to keep. A 301 redirect tells Google to index the destination instead. A 404 or 410 status tells it the page is gone, and Google drops it after recrawling. An XML sitemap lists the URLs you want indexed, but listing a URL doesn’t force Google to index it.

Indexing is not guaranteed. Google states it does not index every page it finds. A page may be left out because it carries a noindex rule, duplicates another page, is low quality or thin, returns an error, or because Google simply hasn’t got to it yet. Search Console’s Page indexing report lists the reasons, such as “Crawled - currently not indexed” (Google fetched it and chose not to index it) and “Discovered - currently not indexed” (Google knows the URL but hasn’t crawled it yet).

Since mobile-first indexing became the default, Google mainly indexes the mobile version of each page. Content that only exists on your desktop version may not be indexed at all.

Why indexing matters for SEO

If a page isn’t indexed, it gets zero search traffic, whatever its quality. Indexing problems are some of the most common reasons important pages go missing from Google, and they are often caused by a single tag or setting.

The opposite problem matters too. Index bloat, thousands of thin, duplicate or parameter URLs in the index, can dilute your site’s quality signals and waste crawl budget. Good SEO controls what is indexed, not just whether pages are.

Indexing is also where Google forms its view of what each page is about. Clear titles, headings, internal links and structured data help it understand and file the page correctly.

How to get a page indexed

1. Inspect the URL in Google Search Console. The URL Inspection tool shows whether it is indexed, the canonical Google chose, and any blocking issue.
2. Remove anything blocking it: a noindex meta tag or X-Robots-Tag header, a robots.txt disallow rule, or a login requirement.
3. Make sure the page returns a 200 status code and that its canonical tag points to itself (or to the version you want indexed).
4. Link to it from relevant, already-indexed pages on your site, and include it in your XML sitemap.
5. Make the content substantial and distinct. Pages that duplicate others or add little are often crawled but not indexed.
6. Click “Request indexing” in URL Inspection for new or updated important pages. Use it sparingly; it queues a crawl but doesn’t guarantee indexing.
7. Check the Page indexing report every few weeks for important URLs moving into excluded states.

Indexing example

<!-- Lets Google index this page (the default; no tag needed) -->
<link rel="canonical" href="https://example.com/guides/indexing">

<!-- Keeps a page out of the index -->
<meta name="robots" content="noindex">