Crawl budget
What is crawl budget?
Crawl budget is the number of URLs a search engine can and wants to crawl on a site in a given period. Google defines it as the combination of crawl capacity, how much crawling your server can handle without slowing down, and crawl demand, how much Google wants to crawl based on the site’s size, update frequency, quality and popularity.
Crawl capacity rises when your server responds quickly and falls when it slows down or returns errors: 5xx and 429 responses make Google’s crawlers back off. Crawl demand rises for popular pages and content that changes often, and falls for URLs Google sees as stale, duplicate or low value.
Google says crawl budget is something most sites do not need to think about. Its guidance is aimed at large sites, roughly 1 million or more unique pages that change weekly, or 10,000 or more pages that change daily, and at sites with many URLs Google has discovered but not crawled. Google calls those numbers rough estimates.
Why crawl budget matters for SEO
If Google spends its visits on junk URLs, new and updated pages wait longer to be crawled and indexed. On a large ecommerce site, that can mean new products or price changes taking weeks to show up.
The usual culprits are faceted navigation and URL parameters that create near-endless combinations, duplicate content, soft 404s, long redirect chains and slow servers.
Crawl budget does not directly affect rankings. More crawling does not make a page rank higher; it only means Google sees changes sooner.
How to optimise crawl budget
- Check whether you have a problem first. In Search Console, look at the Crawl stats report and the page indexing report. Many "Discovered – currently not indexed" URLs on a big site is a sign.
- Block URL spaces with no search value, such as sort and filter combinations or internal search, in robots.txt. Noindex does not save crawling, because Google still has to fetch the page to see it.
- Consolidate duplicates with redirects or canonicals, and link internally to canonical URLs only.
- Return real 404 or 410 codes for removed pages, and fix soft 404s.
- Shorten redirect chains so each old URL takes one hop.
- Speed up server responses so Google can crawl more without overloading you.
- Keep your XML sitemap clean and its lastmod dates accurate, so Google knows what changed.
Crawl budget FAQ
Does crawl budget matter for small sites?
Rarely. Google says sites with fewer than a few thousand pages are usually crawled efficiently. If new pages on a small site are not indexed, the cause is more likely quality or discovery than crawl budget.
Does noindex save crawl budget?
No. Google has to crawl a page to see the noindex, so noindexed pages still use crawl budget. Use robots.txt to stop crawling of URLs you never want fetched.
How can I see my crawl budget?
Google does not publish a number. The Crawl stats report in Search Console shows how many requests Googlebot makes per day and how fast your server responds, and server log file analysis shows exactly which URLs it fetches.
Related SEO terms
Crawling is how search engines discover pages: automated bots fetch URLs and follow the links on them to find more pages.
Googlebot is Google’s web crawler. It fetches pages so Google Search can index them, using a smartphone crawler for most sites.
Faceted navigation lets visitors filter a list by attributes such as size, colour or price, and can create huge numbers of crawlable URLs.
URL parameters are key-value pairs after a ? in a URL, used for tracking, sorting, filtering and pagination. Each variation can look like a new page.
Log file analysis is reviewing your server’s access logs to see exactly which URLs search engine bots requested, when and with what result.
Index bloat is when a search engine indexes many more of a site’s pages than it should, most of them low-value, duplicate or unintended URLs.
Robots.txt is a plain-text file at the root of a site that tells crawlers which URLs they may and may not request.