It depends which ones, and it should be a decision, not an accident. "AI crawlers" are several different bots doing different jobs. Blocking the ones that collect content for training is a reasonable choice for many agencies. Blocking the ones that power AI search usually isn't, because it takes your pages out of the answers people are reading. Allowing them doesn't guarantee your pages are used or cited, but blocking them rules it out.
Three jobs, three kinds of crawler
The main AI companies now run separate crawlers for separate purposes, and each one is switched on or off on its own in your robots.txt file.
| Company | Training | Search | Fetches for a user |
|---|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User |
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User |
| Perplexity | (none) | PerplexityBot | Perplexity-User |
| Google-Extended | Googlebot |
Training crawlers collect content that may be used to train future models. OpenAI says blocking GPTBot means your content shouldn't be used to train its foundation models.
Search crawlers build the index that AI search tools answer from and link to. OpenAI says sites that block OAI-SearchBot won't be shown in ChatGPT search answers, though they can still appear as plain navigational links.
User-triggered fetchers visit a page because a person asked the assistant to. Because a person started it, OpenAI says robots.txt rules "may not apply" to ChatGPT-User, and Perplexity says its user agent generally ignores them.
Sources: OpenAI, Anthropic, Perplexity and Google.
The common mistake
The most common mix-up is treating them as one switch. An agency blocks GPTBot to keep its content out of training, and assumes it has also kept itself out of ChatGPT. It hasn't: ChatGPT search uses OAI-SearchBot. The reverse happens too, and it's more costly: blocking OAI-SearchBot to stop training does nothing about training, and takes the site out of ChatGPT's search answers.
To stay out of training
User-agent: GPTBot
Disallow: /Your pages stay out of OpenAI's model training, and still show in ChatGPT search answers.
The common mix-up
User-agent: OAI-SearchBot
Disallow: /Training carries on through GPTBot, and your pages drop out of ChatGPT search answers.
AI tools get this wrong themselves. When I asked Google's AI Mode who could help a council with outdated ChatGPT answers, part of its advice was to allow GPTBot. That's the training crawler, not the search one.
Google works differently. Its AI Overviews and AI Mode use the normal Search index, so the controls are the normal Search ones, such as noindex and nosnippet. Google-Extended covers training Gemini models and grounding in some other Google products, and Google says it "does not impact a site's inclusion in Google Search".
robots.txt is a request, not a lock
robots.txt tells well-behaved crawlers what you'd like. Google is plain about it: robots.txt "cannot enforce crawler behavior", and it isn't a way to keep a page out of search.
The opposite problem is just as common. Firewalls and content delivery networks can block AI crawlers with a single setting, whatever robots.txt says. So a site can publish a robots.txt that welcomes OAI-SearchBot while its firewall turns it away with an error. Often nobody decided that: two teams made two settings at different times.
The only way to know is to check what each crawler actually receives when it asks for your pages.
Why blocking on purpose can be right
Some agencies block AI crawlers deliberately, and they have good reasons. Aggressive crawlers can load servers and push up hosting costs. Some content shouldn't be reused for training. Those are legitimate calls, and nobody should tell you you're "doing it wrong" for making them.
What matters is that each setting is chosen, written down and consistent across robots.txt and the firewall, and that whoever made it knows what it costs.
A sensible starting point
- Allow the search crawlers. If you want your pages, not someone else's copy, to be the source of AI answers about your services, allow OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot.
- Decide on the training crawlers. GPTBot, ClaudeBot and Google-Extended are a policy question for you and your privacy and legal people, not a server setting.
- Make the firewall match. If robots.txt allows a crawler, check the firewall and CDN don't block it, and the other way round.
- Write the decision down. Note who made it and why, so the next migration or vendor change doesn't undo it.
- Check again every few months. The companies add and rename crawlers.
In robots.txt, allowing a search crawler while blocking a training one looks like this:
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /Blocking search crawlers by accident is one of the reasons AI tools give wrong answers about an agency: they answer from whatever copies of your content they can read.
The findability audit checks each crawler against both robots.txt and what your firewall actually returns, and says who needs to act on each finding. I also work alongside teams on findability, AI search and AI implementation: see AI project consulting.