Which AI Crawlers Should You Allow?

One robots.txt decision cannot cleanly mean “show my pages in AI answers, but never use them for training.” Each vendor separates those jobs differently. OpenAI and Anthropic publish distinct training, search, and user-fetch agents. Google-Extended is a policy token, not a crawler name you will see in request logs. If you block names from a copied list without checking their roles, you can cut off the citations you wanted or opt out of less than you assumed.

As of mid-2026, the practical goal is to decide separately whether you want model training, search indexing, and user-requested retrieval. Then express each choice using the user-agent token the operator documents.

Which AI crawlers do what?

These six names are often grouped together, but they do not perform the same job:

• GPTBot: OpenAI’s training crawler; it collects public content that may be used to train generative AI foundation models.
• ClaudeBot: Anthropic’s model-development crawler; it collects web content that may contribute to future Claude training.
• Claude-Web: A legacy Anthropic user-agent token; it is absent from Anthropic’s current three-bot lineup, so keep an old rule for compatibility but do not treat it as the current Claude search control.
• PerplexityBot: Perplexity’s search crawler; it indexes pages so Perplexity can surface and link them in answers, and Perplexity says it is not used to collect foundation-model training data.
• Google-Extended: Google’s robots control for Gemini model training and some Gemini grounding uses; it has no effect on inclusion or ranking in Google Search.
• CCBot: Common Crawl’s crawler; it builds an open web dataset used for research, analysis, and downstream model training rather than powering one named answer engine.

The missing names matter. OpenAI uses OAI-SearchBot to surface pages in ChatGPT search. Anthropic uses Claude-SearchBot for search indexing and Claude-User when a person asks Claude to retrieve a page. Perplexity also has Perplexity-User for user-initiated fetches. Those retrieval agents are not interchangeable with the training crawlers above.

Check the operator documentation before copying any rule: OpenAI crawlers, Anthropic bots, Perplexity crawlers, Google-Extended, and CCBot.

Can you allow answers but decline training?

Often, yes, but the result depends on the platform.

For OpenAI, you can disallow GPTBot while allowing OAI-SearchBot. Blocking GPTBot asks OpenAI not to use future crawls of your site for foundation-model training. That may keep your brand out of answers that rely on stored model knowledge, but it does not directly remove you from live ChatGPT search. Blocking OAI-SearchBot is the rule that prevents your pages from appearing in ChatGPT search answers, apart from possible navigational links.

Anthropic offers a similar split. Disallow ClaudeBot for future training collection, then allow Claude-SearchBot and Claude-User if you want Claude to index or retrieve your public pages for users. A Claude-Web rule can remain as defensive coverage for older configurations, but the current controls are the three named in Anthropic’s documentation.

PerplexityBot is already on the answer side of the line. Blocking it can reduce your visibility in Perplexity search results. Perplexity-User is different: Perplexity says these user-triggered requests generally ignore robots.txt, so do not assume a crawler rule will stop every fetch initiated by a person.

Google is the common source of confusion. Disallowing Google-Extended opts your content out of specified Gemini training and grounding uses. It does not remove the site from Google Search, and it does not opt the site out of AI Overviews. AI Overviews are a Search feature governed through Google Search controls. Blocking Googlebot would also affect ordinary search visibility, while supported preview controls such as nosnippet can limit how content is used in Search features.

CCBot offers no answer-versus-training switch. If you allow it, your pages may enter Common Crawl’s open dataset, where downstream uses are broader than a single product. If you block it, you reduce access to future Common Crawl snapshots, not copies already collected or held elsewhere.

What should selective robots.txt look like?

This example allows the documented answer-discovery agents while declining the named training and broad-dataset crawlers:

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-Web
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Put the file at https://example.com/robots.txt, replace the sitemap URL, and publish a separate file on every subdomain you care about. Named groups take precedence for those agents; the wildcard keeps other crawlers open. Do not paste this policy blindly. A documentation site seeking broad model familiarity may allow training bots, while a publisher protecting licensed archives may block them.

After publishing, fetch the file from the public URL and check the response body. A CMS plugin, CDN rule, staging deployment, or malformed group can make the live file differ from the version you edited. You can check these six tokens in a free Suede Signal audit, then confirm important traffic in your own server or CDN logs.

Where does robots.txt stop protecting you?

robots.txt is a public request to crawler operators, not authentication or access control. Reputable operators document that their automated crawlers honor it, but the file cannot make public content private, erase material already collected, or bind an unrelated scraper.

User-agent spoofing also exists. Any client can put GPTBot, ClaudeBot, or another trusted label in an HTTP header. If traffic volume or content protection matters, compare requests with the operator’s published IP ranges or reverse-DNS guidance. Configure your WAF carefully so verified search crawlers can pass while obvious impersonators, abusive rates, and unwanted automation are blocked.

Keep confidential, paid, or account-only material behind authentication. Do not list secret paths in robots.txt; the file is visible to everyone. Use server authorization, signed URLs, rate limits, and contractual controls where the risk requires enforcement.

What to do next

• Write a three-column policy for each vendor: training, search indexing, and user-requested retrieval.
• Update robots.txt on every relevant host, keeping Googlebot and other ordinary search crawlers open unless you intend to leave search.
• Fetch the live file, test representative paths, and inspect logs for the exact user agents you allowed or blocked.
• Recheck the operators’ official crawler pages periodically, because bot names and product boundaries change.