🤖 AI & Agents · 10 min read

AI Crawlers in robots.txt: Which to Allow, Which to Block

Allow the retrieval bots whose answers you want to be part of, and treat the training bots as a separate decision. A vendor like OpenAI gives you two independent switches for exactly that. Before you write the list, know one rule that catches almost every hand-written file: consecutive User-agent lines share a single group of rules. Then comes the part nobody enjoys, which is that a robots.txt file is a request and not a wall. RFC 9309 says so in its second paragraph, and the wording is worth keeping in mind while reading the rest of this page.

Which AI crawler tokens exist, and what does each one do?

The naming is not decorative. Vendors split their traffic by intent, and the intent decides whether blocking costs you anything. The table below only lists tokens whose purpose we could read on the vendor's own page, with the token in the form it takes in the file.

Two of those rows deserve a longer look, because they are the ones people misread. Google documents Google-Extended as a token with no user agent string of its own: the crawl happens under ordinary Google user agents and the token acts as a control switch. That means you cannot find it in a log, and you cannot treat its appearance there as evidence of anything. Apple makes a similar distinction from the other direction, noting that disallowing Applebot-Extended does not stop Applebot from crawling, and that content stays discoverable in Spotlight, Siri and Safari either way.

OpenAI frames it as two independent tags rather than one crawler with a mood. Their overview says each setting works separately: a site can allow OAI-SearchBot to appear in search results while disallowing GPTBot to indicate the content should stay out of training data, and if a site allows both, one crawl may serve both purposes. They also note that a robots.txt change takes roughly 24 hours to show up in search behaviour. Anthropic splits its traffic three ways, with training, search indexing and user-initiated fetches on separate names, and adds the consequence that disabling user-initiated fetching may reduce how often your site surfaces when someone asks Claude a question.

Cloudflare's bot documentation is a useful second opinion on the categories, since it sits between sites and this traffic at scale. Under the taxonomy the company introduced on 1 July 2026 it labels AI Assistant as a bot driven by a user action, giving Perplexity-User and DuckAssistBot as examples, AI Crawler as traffic that gathers content for training, and AI Search as the bots behind AI-driven search experiences, with OAI-SearchBot as the example.

The grouping rule, and the file that gets it wrong

Here is the part that turns a tidy list of names into a surprise. RFC 9309 defines a group as one or more user-agent lines followed by one or more rules, and its grammar repeats the user-agent line within a group rather than after it. The example in the specification prints barbot and bazbot on two consecutive lines above a single rule, and describes that as a group relevant for more than one user agent. So this:

User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /private

is not two rules. Both names get the same one. The same document covers the other half of the behaviour: if more than one group matches a product token, the matching groups' rules are combined, and * applies to user agents that have no explicit match at all. Nothing about group order changes the outcome.

Longest match is the last piece. Where an allow and a disallow both match a path, the more specific pattern wins, which is how you exempt one folder from a broader block.

What we measured on our own robots.txt

We wrote a small parser against the specification's grammar and ran it on our live file rather than on a copy, because a copy drifts. On 19 September 2026 our robots.txt came back as two groups: a wildcard group with one user-agent line and six rules, and a second group carrying sixteen user-agent lines that share the same six rules. Among the sixteen are GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-Web, anthropic-ai, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, meta-externalagent, cohere-ai, Bytespider and Amazonbot.

The count is identical in every row, and that is the finding worth passing on. Our private paths are closed to everything by the wildcard group already, so listing sixteen names beneath the same rules changes nothing about what any of them may fetch. The list is documentation rather than control, which is a fair thing to be honest about. It exists so that a person reading the file can see which bots we thought about.

Running the same parser over a synthetic file turned up the second finding, and it is the reason the trap above is worth printing. With GPTBot and ClaudeBot stacked above a single disallow, the parser assigned that rule to both, exactly as the specification requires. Anyone who intended two different policies has written one policy twice.

Two details of our own file are less flattering. We carry Claude-Web and anthropic-aialongside Anthropic's current names, and the page we read describes the company's crawlers as three: ClaudeBot, Claude-SearchBot and Claude-User. Legacy tokens are harmless in a list, since an unused name simply never matches, but a file that documents a set which partly no longer exists is a file that has stopped being maintained. We also noticed that our wildcard block comes first in the file and the named block second, which is legal and irrelevant to matching, and would confuse anyone who assumed the order carried meaning.

So should you block GPTBot?

The useful version of this question separates two things that get discussed as one. Being cited in an answer is a retrieval path. Being included in a training set is a separate use of the same content, and the vendors have given you separate switches for them, which suggests they consider the distinction real. Blocking the training token while allowing the retrieval tokens is the documented combination, not a hack.

What we would not do is claim this guarantees anything. No vendor has promised that allowing a bot produces a citation, and a file that asks for a behaviour is a long way from a system that enforces it. The honest summary is that the retrieval switches are cheap to leave open and the training switches are yours to argue about.

Do AI crawlers actually obey robots.txt?

Varies by bot, and the two clearest statements come from opposite sides of the line. Google's documentation states that its common crawlers always obey robots.txt rules when crawling automatically. OpenAI, describing the fetches made on a user's behalf, notes that ChatGPT-User is not used for crawling the web in an automatic fashion and that robots.txt rules may not apply, because a person asked for that page.

RFC 9309 already told you where this lands. Its introduction states that the rules are not a form of access authorization. If the difference matters to you, handle it where requests actually arrive, at the edge or in the application, and treat the file as a published statement of intent that most well-behaved crawlers will read.

How do you write the file so each bot gets its own rules?

1. Decide by intent, not by name. Retrieval, training, and private paths. Three decisions, three blocks.
2. One group per policy. Put the names that share a policy on consecutive lines, and separate policies with their own user-agent line followed by their own rules.
3. Keep a wildcard block. Whatever has no explicit match falls back to *, so this is where your private paths belong.
4. Parse it after you edit it. A grammar of this size is worth checking with a script rather than an eye, especially when your list grows past a handful of names.
5. Pair it with a reading list. A robots.txt states what may be fetched. An llms.txt states what is worth opening, and the two answer different questions.

One more file belongs in the same afternoon's work if your site offers tools rather than text. Declaring those tools is what WebMCP covers, and it is the difference between a site an assistant can read and one it can use.

Frequently asked questions

I build this layer for other sites: robots.txt, llms.txt, agent-tools.json and WebMCP declarations, verified against the live pages. Details at /agent-ready.