AI crawlers (GPTBot, ClaudeBot)
What are AI crawlers?
AI crawlers are automated bots run by AI companies that fetch web pages, either to collect training data for language models or to find and read pages for AI search answers. Examples include OpenAI’s GPTBot and OAI-SearchBot, Anthropic’s ClaudeBot and Claude-SearchBot, and PerplexityBot. Each identifies itself with a user agent you can allow or block in robots.txt.
The key distinction is training versus search. Training crawlers collect content that may be used to train future models; blocking them is an opt-out from training and doesn’t affect whether you are cited in live answers. Search crawlers index pages so an assistant can show and cite them; blocking them can keep you out of that assistant’s answers. A third kind, user-triggered fetchers, visit a page because a user asked the assistant about it, and providers say robots.txt may not apply to them in the same way.
The main ones, as documented by their operators: **GPTBot** (OpenAI, training); **OAI-SearchBot** (OpenAI, ChatGPT search; blocking it keeps your pages out of ChatGPT search answers); **ChatGPT-User** (OpenAI, fetches pages for a user’s request); **ClaudeBot** (Anthropic, training); **Claude-SearchBot** (Anthropic, search results in Claude); **Claude-User** (Anthropic, fetches pages for a user’s request); **PerplexityBot** (Perplexity, its search index; Perplexity says it isn’t used for training); **Perplexity-User** (Perplexity, live fetches for users).
Google works differently. It doesn’t have a separate AI crawler: Googlebot crawls for Search, including AI Overviews and AI Mode. **Google-Extended** is a robots.txt product token, not a separate bot. Disallowing it stops Google using your content to train Gemini models and to ground answers in the Gemini apps and its developer grounding service, but Google says it has no effect on inclusion or ranking in Google Search. To keep content out of AI Overviews you would have to use snippet controls such as nosnippet, which also affect normal results.
Why AI crawlers matter for SEO
Blocking the wrong bot is one of the easiest ways to lose AI visibility by accident. Many sites added blanket blocks on AI bots in 2023-2024 and, in doing so, also blocked the search bots that decide whether they appear in ChatGPT, Claude or Perplexity answers. CDN and firewall “block AI bots” settings can do the same without anyone editing robots.txt.
The opposite choice is also legitimate. A publisher may want to be cited in answers but not used for training. Because the bots are separate, robots.txt lets you make that choice per company and per purpose.
How to control AI crawlers
- Decide your policy for each purpose: training, AI search, and user-triggered fetches.
- Write separate robots.txt groups for each user agent. A block on one bot does not block the others.
- Check your CDN, WAF or hosting “bot protection” settings, which may block AI bots regardless of robots.txt.
- Check your server logs to see which bots actually visit and whether they get 200 responses or 403s. Verify bots by the IP ranges their operators publish, since user agents can be faked.
- Review the policy every few months. Operators add and rename bots, so check their documentation.
AI crawlers example
# OpenAI
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
# Anthropic
User-agent: Claude-SearchBot
Allow: /
User-agent: ClaudeBot
Disallow: /
# Perplexity (search index, not training)
User-agent: PerplexityBot
Allow: /
# Google: opt out of Gemini training and grounding (Search is unaffected)
User-agent: Google-Extended
Disallow: /AI crawlers FAQ
Should I block GPTBot?
Blocking GPTBot opts your content out of OpenAI model training. It doesn’t remove you from ChatGPT search, which uses OAI-SearchBot, so it is a choice about training, not visibility.
Does blocking Google-Extended remove me from AI Overviews?
No. AI Overviews and AI Mode use content crawled by Googlebot for Search. Google-Extended only covers Gemini model training and grounding outside Search.
Do AI crawlers respect robots.txt?
OpenAI, Anthropic, Google and Perplexity say their crawlers follow robots.txt. User-triggered fetchers such as ChatGPT-User may be treated differently, and some bots don’t follow the rules, so check server logs.
Related SEO terms
Robots.txt is a plain-text file at the root of a site that tells crawlers which URLs they may and may not request.
A user agent is the software making a web request, identified by a user-agent string such as Googlebot’s that servers and robots.txt can read.
A web crawler is an automated program that visits web pages, downloads them and follows their links, used by search engines and SEO tools.
ChatGPT search is the feature that lets ChatGPT search the web and answer with up-to-date information and links to its sources.
Grounding is tying an AI model’s answer to real, retrievable sources, such as search results, so it is current and can be checked.
Log file analysis is reviewing your server’s access logs to see exactly which URLs search engine bots requested, when and with what result.