Guides

Robots.txt for AI Crawlers: GPTBot, OAI-SearchBot, PerplexityBot and More

Robots.txt for AI Crawlers: GPTBot, OAI-SearchBot, PerplexityBot and More

If you want to be cited by AI search engines, allow their search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot) and decide separately whether to allow their training crawlers (GPTBot, ClaudeBot, Google-Extended). The two are independent. Blocking training doesn’t remove you from AI search, but blocking search crawlers does.

This is the most misunderstood part of AI search, so here’s every major bot and what it does.

Table separating AI search and citation bots from AI training bots for OpenAI, Anthropic, Perplexity and Google

OpenAI

According to OpenAI’s documentation (OpenAI):

  • OAI-SearchBot: surfaces websites in ChatGPT search. Block it and your site won’t appear in ChatGPT search answers (though it may still appear as a navigational link).
  • GPTBot: collects content that may be used to train OpenAI’s models. Blocking it signals your content shouldn’t be used for training.
  • ChatGPT-User: visits pages when a user asks ChatGPT to. It isn’t used for automatic crawling, and because users trigger it, robots.txt may not apply.

OpenAI notes each setting is independent, and search-related changes take about 24 hours.

Perplexity

From Perplexity’s documentation (Perplexity):

  • PerplexityBot: surfaces and links websites in Perplexity results. Perplexity says it isn’t used to crawl content for AI foundation models.
  • Perplexity-User: fetches pages when a user asks. Generally ignores robots.txt because a person triggered it.

Anthropic (Claude)

From Anthropic’s support article (Anthropic):

  • ClaudeBot: collects content that may contribute to model training.
  • Claude-SearchBot: indexes content to improve Claude’s search results. Blocking it may reduce your visibility in Claude’s search answers.
  • Claude-User: fetches pages when a Claude user asks a question that needs them.

Anthropic also advises against IP blocking because it can stop the bot reading your robots.txt.

Google

  • Googlebot: crawls for Google Search, including AI Overviews and AI Mode. Block it and you disappear from Google.
  • Google-Extended: a robots.txt token (not a separate crawler) controlling whether Google can use your content for Gemini training and grounding in Gemini Apps and Vertex AI. Google states it does not affect inclusion or ranking in Google Search (Google).

Three robots.txt setups

Setup A: Maximum AI visibility

Allow everything (the default if you don’t mention these bots). Nothing to add.

Setup B: Be cited, don’t be trained on

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

This is the setup I recommend most often for publishers who are uneasy about training but want AI search traffic.

Setup C: Opt out of AI entirely

Disallow all the bots above except Googlebot. Understand the trade-off: you lose ChatGPT, Claude and Perplexity search citations.

Don’t forget the firewall

The most common problem I see isn’t robots.txt. It’s a CDN or firewall setting that blocks AI bots before they ever read robots.txt. In Cloudflare, check bot management and any “block AI scrapers” options. Both OpenAI and Perplexity publish IP ranges you can allowlist.

How to check what’s actually happening

  • Search your server logs for each bot name.
  • Note the date of the last visit from each.
  • If a bot you allow never appears, look at your firewall.

Related guides

FAQ

Should I block GPTBot?
That depends on whether you’re comfortable with your content being used for training. Blocking it doesn’t remove you from ChatGPT search; OAI-SearchBot controls that.

Does blocking Google-Extended hurt my SEO?
Google says it doesn’t affect inclusion or ranking in Google Search.

Can I block AI from summarising my pages in Google?
Only with snippet controls like nosnippet or max-snippet, which also affect normal search snippets.

Do all AI bots respect robots.txt?
The documented search and training crawlers say they do. User-triggered fetchers (ChatGPT-User, Perplexity-User) may not, because a person requested the page.

Sources

Shubham Khurana

Written by

Shubham Khurana

Founder, DigiBounce · Performance Marketing & AEO Strategist

Shubham Khurana is the founder of DigiBounce and an experienced digital marketing & SEO consultant with over 9 years of experience. He specialises in Answer Engine Optimisation (AEO), Generative Engine Optimisation (GEO), performance marketing, and scaling high-growth brands across modern AI and search engines.

Related Articles

Leave a Reply

Your email address will not be published. Required fields are marked *