User agent
What is a user agent?
A user agent is the software that makes a request to a web server on someone’s behalf, such as a browser or a crawler. It identifies itself with a user-agent string sent in the User-Agent HTTP header, for example a version of Chrome on Android, or Googlebot.
In SEO the term has two related uses. Servers and analytics read the user-agent string to tell browsers from bots. And robots.txt uses user-agent tokens, short names such as Googlebot, Bingbot, GPTBot or ClaudeBot, to set rules for particular crawlers. Some tokens control use rather than crawling: Google-Extended, for example, lets sites opt out of their content being used to train Google’s Gemini models, but it is not a separate crawler.
User-agent strings are easy to fake, so a request claiming to be Googlebot is not proof. Verify important bots by reverse DNS or published IP ranges.
Why user agent matters for SEO
Knowing user agents lets you write the right robots.txt rules, read your logs correctly in log file analysis, and make sure your server, firewall or CDN is not accidentally blocking search engine crawlers. Serving different content to search engine user agents than to people, on purpose, is cloaking and breaks Google’s spam policies.
How to target a user agent in robots.txt
- Find the crawler’s token in its owner’s documentation, such as Googlebot, Bingbot or GPTBot.
- Start a group with User-agent: followed by the token, then add Allow and Disallow rules beneath it.
- Remember that a crawler follows only the most specific group that matches it, so rules in the * group do not also apply to a crawler that has its own group.
- Test changes before you publish them, as one wrong line can block a whole site.
User agent example
# Googlebot Smartphone (W.X.Y.Z is the current Chrome version)
# Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
User-agent: *
Disallow: /search/
User-agent: GPTBot
Disallow: /User agent FAQ
What is Googlebot’s user agent?
Googlebot has a smartphone and a desktop crawler, both identified by “Googlebot/2.1” in their strings. Google crawls most sites with the smartphone version. The robots.txt token for both is Googlebot.
Can I block bots by user agent?
Well-behaved bots follow robots.txt rules for their user agent. Bad bots ignore them and fake their strings, so blocking those needs firewall or server-level rules.
Related SEO terms
Googlebot is Google’s web crawler. It fetches pages so Google Search can index them, using a smartphone crawler for most sites.
A web crawler is an automated program that visits web pages, downloads them and follows their links, used by search engines and SEO tools.
Robots.txt is a plain-text file at the root of a site that tells crawlers which URLs they may and may not request.
AI crawlers (GPTBot, ClaudeBot)
AI crawlers are bots that AI companies use to collect web content for model training or for AI search answers, such as GPTBot and ClaudeBot.
Log file analysis is reviewing your server’s access logs to see exactly which URLs search engine bots requested, when and with what result.
CloakingBlack hat
Cloaking is showing search engines different content from what users see in order to manipulate rankings, which breaks Google’s spam policies.