Googlebot
What is Googlebot?
Googlebot is the web crawler Google uses to find and fetch pages for Google Search. It comes in two main types, Googlebot Smartphone and Googlebot Desktop, and because of mobile-first indexing most sites are crawled mainly by the smartphone crawler. Googlebot respects robots.txt rules aimed at it.
Googlebot identifies itself in the user agent string, but anyone can fake that, so Google recommends verifying real Googlebot traffic with a reverse DNS lookup (the host should end in googlebot.com or google.com) or by checking against Google’s published IP ranges. Google also runs other crawlers, such as Googlebot Image and Googlebot Video, plus special-purpose fetchers. Google-Extended is not a separate crawler but a robots.txt token that controls whether content may be used for training and grounding Gemini models; it doesn’t affect Search.
Googlebot fetches up to the first 15MB of an HTML or supported text file, then renders the page with an evergreen Chromium.
Why Googlebot matters for SEO
If Googlebot can’t reach a page, Google can’t index it. Blocking it by mistake, through robots.txt, a firewall, a bot-protection service or rate limiting, is a quick way to lose rankings.
Watching what Googlebot crawls tells you whether Google spends its time on the pages you care about. See log file analysis.
How to check Googlebot can access your site
- Run a live test in Search Console’s URL Inspection tool to see if Googlebot can fetch and render the page.
- Check the Crawl stats report for errors, response times and the share of requests by Googlebot type.
- Make sure your firewall or CDN bot protection allows verified Googlebot.
- Verify suspicious “Googlebot” hits in your logs with reverse DNS before trusting them.
Googlebot example
# Rules for Googlebot only
User-agent: Googlebot
Disallow: /internal-search/
# Rules for everyone else
User-agent: *
Disallow: /admin/Googlebot FAQ
How do I know if a visit is really Googlebot?
Do a reverse DNS lookup on the IP address; it should resolve to googlebot.com or google.com, and a forward lookup should return the same IP. Google also publishes its crawler IP ranges.
Does blocking Google-Extended affect my rankings?
No. Google says Google-Extended doesn’t affect inclusion or ranking in Google Search. It only controls use in Gemini models.
Related SEO terms
A web crawler is an automated program that visits web pages, downloads them and follows their links, used by search engines and SEO tools.
Crawling is how search engines discover pages: automated bots fetch URLs and follow the links on them to find more pages.
Robots.txt is a plain-text file at the root of a site that tells crawlers which URLs they may and may not request.
A user agent is the software making a web request, identified by a user-agent string such as Googlebot’s that servers and robots.txt can read.
Crawl budget is the set of URLs Google can and wants to crawl on a site. It only becomes a constraint on very large or fast-changing sites.
Mobile-first indexing means Google crawls and indexes the mobile version of your pages and uses it for ranking, on mobile and desktop alike.
AI crawlers (GPTBot, ClaudeBot)
AI crawlers are bots that AI companies use to collect web content for model training or for AI search answers, such as GPTBot and ClaudeBot.