Crawling

What is crawling?

Crawling is the process by which search engines discover pages on the web. Automated programs called crawlers or bots, such as Googlebot, request URLs, download the content and follow the links they find to reach new pages. Crawling is the first step: a page that is never crawled can’t be indexed or ranked.

A crawler starts from URLs it already knows, from earlier crawls, links on other sites and XML sitemaps you submit. For each URL it checks robots.txt to see whether it is allowed in, requests the page, notes the HTTP status code, and passes the content on for processing. Links found on the page are added to a queue to crawl later.

Google decides how often and how much to crawl a site. Its crawl budget depends on how much your server can handle without slowing down (the crawl capacity limit) and how much Google wants to crawl your URLs, based on their popularity and how often they change (crawl demand). Popular, frequently updated pages are crawled more often than old, rarely linked ones.

Crawling and indexing are different steps. A crawled page may still not be indexed, for example because it has a noindex tag, is a duplicate of another URL, or Google judged it not useful enough. Equally, robots.txt stops crawling, not indexing: a blocked URL can still appear in results, without a description, if other sites link to it.

Google uses a mobile crawler (Googlebot Smartphone) for most sites, because indexing is mobile-first. After fetching, Google renders pages that rely on JavaScript in a recent version of Chromium, which can happen some time after the first fetch. Links that only exist after JavaScript runs can still be found, but plain HTML links with an href are the most reliable way to be discovered.

Why crawling matters for SEO

If a search engine can’t crawl a page, nothing else you do for that page matters. Crawl problems are common and often invisible: a stray robots.txt rule, a broken internal link, a login wall, a server that times out, or navigation built only with JavaScript click handlers.

For small sites, Google says crawl budget is rarely a concern. For large sites, with hundreds of thousands or millions of URLs, endless faceted navigation, URL parameters and duplicate URLs can waste crawling on unimportant pages while new or updated ones wait.

Crawl health also affects freshness. When you update prices or publish news, you want the change picked up quickly. Fast servers, clean internal linking and accurate sitemaps help crawlers find what changed.

How to make your site easy to crawl

1. Check robots.txt and make sure it doesn’t block pages, CSS or JavaScript that you want indexed.
2. Link to every important page from at least one other crawlable page, with normal <a href> links. Avoid orphan pages.
3. Keep important pages within a few clicks of the homepage; a flat site architecture is easier to crawl.
4. Submit an XML sitemap that lists only canonical, indexable URLs returning 200, with accurate lastmod dates.
5. Fix broken links, redirect chains and 5xx server errors, which waste crawls.
6. Keep the server fast. Slow responses and errors make Google slow its crawl rate.
7. On large sites, stop crawlers wandering into infinite URL spaces: limit crawlable filter combinations and parameters.
8. Watch the Crawl stats report in Search Console and, on big sites, your server logs, to see what Googlebot actually requests.

Crawling example

Googlebot fetches example.com/, reads robots.txt, which allows it, and finds links to /blog/ and /products/. It fetches /blog/ and finds links to ten articles, then queues them. One article links to /products/blue-mug, which is not in the sitemap but is now discovered through that link. A filter link like /products/?colour=blue&size=s&sort=price creates thousands of near-identical URLs; if left open, Googlebot may spend many requests there instead of on new articles.