If you want to be cited by AI search engines, allow their search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot) and decide separately whether to allow their training crawlers (GPTBot, ClaudeBot, Google-Extended). The two are independent. Blocking training doesn’t remove you from AI search, but blocking search crawlers does.
This is the most misunderstood part of AI search, so here’s every major bot and what it does.

OpenAI
According to OpenAI’s documentation (OpenAI):
- OAI-SearchBot: surfaces websites in ChatGPT search. Block it and your site won’t appear in ChatGPT search answers (though it may still appear as a navigational link).
- GPTBot: collects content that may be used to train OpenAI’s models. Blocking it signals your content shouldn’t be used for training.
- ChatGPT-User: visits pages when a user asks ChatGPT to. It isn’t used for automatic crawling, and because users trigger it, robots.txt may not apply.
OpenAI notes each setting is independent, and search-related changes take about 24 hours.
Perplexity
From Perplexity’s documentation (Perplexity):
- PerplexityBot: surfaces and links websites in Perplexity results. Perplexity says it isn’t used to crawl content for AI foundation models.
- Perplexity-User: fetches pages when a user asks. Generally ignores robots.txt because a person triggered it.
Anthropic (Claude)
From Anthropic’s support article (Anthropic):
- ClaudeBot: collects content that may contribute to model training.
- Claude-SearchBot: indexes content to improve Claude’s search results. Blocking it may reduce your visibility in Claude’s search answers.
- Claude-User: fetches pages when a Claude user asks a question that needs them.
Anthropic also advises against IP blocking because it can stop the bot reading your robots.txt.
- Googlebot: crawls for Google Search, including AI Overviews and AI Mode. Block it and you disappear from Google.
- Google-Extended: a robots.txt token (not a separate crawler) controlling whether Google can use your content for Gemini training and grounding in Gemini Apps and Vertex AI. Google states it does not affect inclusion or ranking in Google Search (Google).
Three robots.txt setups
Setup A: Maximum AI visibility
Allow everything (the default if you don’t mention these bots). Nothing to add.
Setup B: Be cited, don’t be trained on
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
This is the setup I recommend most often for publishers who are uneasy about training but want AI search traffic.
Setup C: Opt out of AI entirely
Disallow all the bots above except Googlebot. Understand the trade-off: you lose ChatGPT, Claude and Perplexity search citations.
Don’t forget the firewall
The most common problem I see isn’t robots.txt. It’s a CDN or firewall setting that blocks AI bots before they ever read robots.txt. In Cloudflare, check bot management and any “block AI scrapers” options. Both OpenAI and Perplexity publish IP ranges you can allowlist.
How to check what’s actually happening
- Search your server logs for each bot name.
- Note the date of the last visit from each.
- If a bot you allow never appears, look at your firewall.
Related guides
- How to Run an AI Visibility Audit in One Afternoon
- Answer Engine Optimization Tools: The Stack I’d Build
- Best AI Visibility Tools: How to Choose One
FAQ
Should I block GPTBot?
That depends on whether you’re comfortable with your content being used for training. Blocking it doesn’t remove you from ChatGPT search; OAI-SearchBot controls that.
Does blocking Google-Extended hurt my SEO?
Google says it doesn’t affect inclusion or ranking in Google Search.
Can I block AI from summarising my pages in Google?
Only with snippet controls like nosnippet or max-snippet, which also affect normal search snippets.
Do all AI bots respect robots.txt?
The documented search and training crawlers say they do. User-triggered fetchers (ChatGPT-User, Perplexity-User) may not, because a person requested the page.
Sources
- OpenAI, Overview of OpenAI crawlers
- Perplexity, Perplexity crawlers
- Anthropic, Does Anthropic crawl data from the web?
- Google, Google’s common crawlers

