Should You Block AI Crawlers? It Depends Which One

A publisher adds four lines to robots.txt — User-agent: GPTBot, Disallow: / — and considers the matter settled. Six weeks later the same site is still being summarised inside ChatGPT, still has its training content excluded, and nobody can explain why both are true at once.

Both are true because the question should you block AI crawlers compresses three separate decisions into one. Training crawlers, search crawlers and user-triggered fetchers are distinct user agents with independent controls, and the cost of blocking them varies by an order of magnitude. Answering the compressed question produces the wrong robots.txt roughly half the time.

Decompressed, the answer is tractable: the training opt-out is nearly free and reversible, the search opt-out is the expensive one, and the user-triggered fetch is largely outside robots.txt anyway. What follows is the evidence for each, taken from the crawler operators' own documentation.

"AI crawler" names three jobs, not one

OpenAI runs four documented crawlers with four purposes, and its crawler documentation is unambiguous that the controls are separable: "Each setting is independent of the others – for example, a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training."

That sentence is the whole article in miniature. Blocking the training crawler and appearing in AI search answers are compatible positions, and most sites that think they have opted out of AI have in fact only opted out of training.

User agent	Operator	Job	What blocking it costs you
GPTBot	OpenAI	Crawls content for model training	Nothing in ChatGPT search, provided OAI-SearchBot stays allowed
OAI-SearchBot	OpenAI	Indexes pages for ChatGPT search	Removal from ChatGPT search answers
ChatGPT-User	OpenAI	Fetches a page a user asked ChatGPT to visit	Little — robots.txt rules may not apply to it
OAI-AdsBot	OpenAI	Validates submitted ad landing pages	Ad review coverage; its data is not used for foundation-model training
Google-Extended	Google	Controls Gemini training and grounding use	Nothing in Google Search inclusion or ranking
ClaudeBot	Anthropic	Crawls content for model training	Training corpus inclusion
PerplexityBot	Perplexity	Indexes pages for Perplexity answers	Visibility in Perplexity results

The user-triggered row deserves a second look because it defeats the intuition behind blanket blocking. OpenAI's documentation says of ChatGPT-User: "Because these actions are initiated by a user, robots.txt rules may not apply." A person who pastes your URL into ChatGPT and asks for a summary is not a crawl campaign, and robots.txt was never the mechanism for governing that.

Auditing which of these appear in a site's robots.txt takes a minute and is worth doing before any policy discussion — the illustrative MendMySEO demo reads robots.txt for a set of named AI user agents and reports each as allowed or blocked, which turns an assumption about your own configuration into an observation.

The training switch is the cheap one

Google documents the training opt-out and its consequence in the same breath. In the entry for Google-Extended in Google's common crawlers reference, the token is described as the control for "whether content Google crawls from their sites may be used for training future generations of Gemini models" — followed by the sentence that settles the risk question: "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search."

"Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search."

— Google Search Central, crawler documentation

OpenAI's independence clause makes the same guarantee on its side of the fence. So the training decision, for the two largest operators, costs a site no search visibility in either Google or ChatGPT. It is also trivially reversible: delete two lines, wait for the next robots.txt fetch.

What the training opt-out actually buys is harder to price. It excludes your content from future model weights, which matters to publishers whose archive is the product and matters much less to a services business whose pages exist to be found. Nobody can currently show you a measurable revenue effect in either direction, and any vendor claiming to has invented the number.

Given a free, reversible switch with an unclear benefit, the defensible position is to decide it on principle rather than on projected traffic — and to keep it separate from the decision that follows.

The search switch is where the bill arrives

OpenAI's wording on the search crawler leaves no ambiguity: "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links." A site that blocks this crawler has removed itself from an answer surface, which is the opposite of what most teams intend when they set out to "block AI."

The case for accepting that cost rests on traffic economics, and Cloudflare has published the cleanest measurement of it. Its crawl-to-refer ratio, introduced on 1 July 2025, divides the HTML requests a platform's crawlers make by the HTML requests arriving with that platform in the referrer header. For the week of 19–26 June 2025, Cloudflare put Anthropic at 70,900:1 and Mistral at 0.1:1 — four orders of magnitude apart on the same metric.

Cloudflare framed the business consequence on the same day in Content Independence Day, announcing that new domains would default to blocking AI crawlers unless the operator pays, and putting the position bluntly: "With OpenAI, it's 750 times more difficult to get traffic than it was with the Google of old. With Anthropic, it's 30,000 times more difficult."

Two cautions before anyone quotes those figures in a strategy memo. They are a snapshot of one week in mid-2025, and the ratios move — Cloudflare Radar publishes them live, so the current number is the one to cite. More importantly, a ratio measures the platform's behaviour across the whole network, not your site's outcome. A publisher monetising pageviews and a B2B site whose pages exist to produce ten qualified enquiries a month read the same ratio to opposite conclusions.

If your site is…	Training crawlers	Search crawlers	Reasoning
Ad or subscription funded publishing	Block	Judgement call	Archive is the asset; referral volume decides whether search access earns its keep
B2B services or SaaS	Optional	Allow	Being cited in an answer is the acquisition channel; training inclusion is close to irrelevant
Ecommerce	Optional	Allow	Product and category pages benefit from answer-surface presence
Documentation or support	Optional	Allow	Deflected support tickets are the win; being quoted accurately serves that
Paywalled or licensed archive	Block	Block	Content is licensed per use; free indexing undercuts the licence

The pattern across that table is that blocking the search crawler is defensible mainly when the content itself is the revenue line. For everyone else, it trades an acquisition channel for a principle that the training switch already satisfies at no cost.

Whatever the decision, write it into robots.txt per user agent, record the date and the reason, and re-check the file after the next deployment — templated robots.txt files quietly drift. Reviewing the current release status before scheduling that re-check is worth a minute — commercial access remains evidence-gated.

Frequently Asked Questions

Does blocking GPTBot remove my site from ChatGPT?

No. GPTBot governs training. ChatGPT search visibility is governed by OAI-SearchBot, and OpenAI documents the two settings as independent — a site can disallow GPTBot and still appear in ChatGPT search answers as long as OAI-SearchBot is allowed.

Does blocking Google-Extended hurt my Google rankings?

Google states directly that Google-Extended has no impact on inclusion in Google Search and is not used as a ranking signal. It controls whether crawled content may be used for training and grounding in Gemini products.

Can robots.txt stop ChatGPT from reading a page a user pasted in?

Largely no. That fetch uses ChatGPT-User, and OpenAI's documentation notes that because the action is initiated by a user, robots.txt rules may not apply. Access control at the server or an authentication wall is the mechanism for that case.

Is there any downside to blocking every AI crawler?

The main one is removal from AI answer surfaces that send referral traffic, since a blanket block catches the search crawlers alongside the training crawlers. Blanket rules also age badly — new user agents appear regularly, and a wildcard written in 2024 will not describe the 2026 crawler population.

How often should AI crawler rules be re-checked?

Whenever robots.txt is redeployed, and on a fixed schedule otherwise. Operators add and rename user agents, and a rule written for one crawler generation silently stops matching the next one.