GEO Expert
Let’s talk

AI crawler robots.txt generator

Choose which AI crawlers can read your site, separately for search, training and user-requested visits, and get a robots.txt block you can paste in. Nothing you enter leaves your browser.

Sai Siddesh2 min
Start from a preset
robots.txt

Which AI crawlers should a business allow?

For most businesses that want to be named in AI answers, allow the search crawlers and decide separately about training. Search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot build the indexes that AI search answers draw on, so blocking them removes you from those answers. Training crawlers such as GPTBot and ClaudeBot collect content that may be used to train future models; blocking them does not remove you from search, so it is a business decision rather than a visibility one.

TokenOperatorUsed for
OAI-SearchBotOpenAIChatGPT search results
GPTBotOpenAIContent that may train OpenAI models
ChatGPT-UserOpenAIPages fetched when a ChatGPT user asks
Claude-SearchBotAnthropicSearch results quality for Claude
ClaudeBotAnthropicContent that may train Claude models
Claude-UserAnthropicPages fetched when a Claude user asks
PerplexityBotPerplexityPerplexity search and answers
Perplexity-UserPerplexityPages fetched when a Perplexity user asks
Google-ExtendedGoogleGemini training and some grounding; not Google Search
Applebot-ExtendedAppleWhether Apple may train models on content
meta-externalagentMetaContent for Meta AI training and products
CCBotCommon CrawlOpen web archive widely used for training

Sources: OpenAI crawler documentation, Anthropic crawler documentation and Google Search Central. Operators add and rename crawlers, so check their pages before relying on a rule.

How do I use the generated file?

  1. Pick a preset, then adjust any crawler individually.
  2. Paste your current robots.txt into “Your existing rules” so nothing you already block is lost.
  3. Copy or download the result and replace the file at https://yourdomain/robots.txt.
  4. Check your server or firewall is not blocking the same crawlers. A robots.txt allow cannot override a firewall block.
  5. Allow about a day: OpenAI notes that search-related robots.txt changes can take around 24 hours to take effect.

Questions about AI crawlers

Will blocking GPTBot remove my site from ChatGPT?

No. GPTBot collects content that may be used for training. Appearing in ChatGPT search is controlled by OAI-SearchBot, which OpenAI recommends allowing. A site can block GPTBot and still be eligible for ChatGPT search answers.

Does Google-Extended affect AI Overviews?

No. AI Overviews and AI Mode are part of Google Search and use Googlebot. Google-Extended governs whether content can be used for Gemini training and some grounding, and blocking it does not change Google Search inclusion. To limit what Google shows from a page, use snippet controls such as nosnippet.

Do user-triggered fetchers obey robots.txt?

Not always. OpenAI says robots.txt rules may not apply to ChatGPT-User because it acts on a user's request rather than crawling automatically. Anthropic says its bots, including Claude-User, honour robots.txt. If you must block a fetcher reliably, do it at your server or firewall.

Is my data sent anywhere?

No. The generator runs entirely in your browser. Nothing you type is uploaded or stored.

What this means for you

See what the assistants say about you.

The free visibility sweep puts forty questions a real client would ask to five answer engines and records every answer and cited source, with three competitors on the same run.

Keep going

Cite this page

Siddesh, S., AI crawler robots.txt generator: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Headnote (GEO Expert, 23 Oct 2026). https://geoexpert.online/tools/ai-crawler-robots-txt/