What AI engines can read decides what they can cite. See also: can AI crawlers read JavaScript-rendered pages, does llms.txt help you get cited by AI and should you block GPTBot in robots.txt.
What are the three kinds of AI crawler?
The three kinds of AI crawler are search-indexing bots, user-triggered fetchers and training crawlers, and robots.txt lets you allow or block each one separately. GPTBot is OpenAI's training crawler; OpenAI's documentation says that "Disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models." The volume is real: a Vercel analysis found GPTBot made 569 million requests across its network in one month in 2024, and Anthropic's crawler 370 million. Anthropic draws the same line between bots in its crawler help article, which says "Claude-SearchBot navigates the web to improve search result quality for users." Perplexity says in its bot documentation that "PerplexityBot is designed to surface and link websites in search results on Perplexity."
| Vendor | Search indexing (keep allowed) | User-triggered fetch | Training |
|---|---|---|---|
| OpenAI | OAI-SearchBot | ChatGPT-User | GPTBot |
| Anthropic | Claude-SearchBot | Claude-User | ClaudeBot |
| Perplexity | PerplexityBot | Perplexity-User | |
| Googlebot | Google-Extended | ||
| Microsoft | Bingbot |
Other training crawlers include Applebot-Extended, CCBot (Common Crawl), Bytespider (ByteDance) and meta-externalagent (Meta). A training crawler is a bot that collects pages to train future AI models, not to answer today's searches. Google's crawler documentation states that Google-Extended controls whether crawled content "may be used for training future generations of Gemini models", separate from the data Search uses.
What does a robots.txt that blocks training but keeps search look like?
A robots.txt that blocks training but keeps search names each training crawler with Disallow: / and leaves everything else allowed, as in this example:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: *
Allow: /OpenAI states that "For search results, please note it can take ~24 hours from a site’s robots.txt update for our systems to adjust", so expect a change to apply within 24 hours. Google's introduction to robots.txt explains the file's syntax and limits, and its data on crawl control applies to every bot above: robots.txt controls crawling, not whether a page is indexed.
How does GeoTest score robots.txt?
GeoTest scores robots.txt with up to 10 site-layer points, checking 5 sample URLs on each scan: 2 for each of the five search-indexing crawlers (Googlebot, Bingbot, OAI-SearchBot, PerplexityBot and Claude-SearchBot), 1 if a crawler is blocked from part of the site, and 0 if it is blocked entirely. Training crawlers are not scored, so blocking GPTBot costs nothing. Blocking two or more AI search crawlers caps the site score at 70, and blocking Googlebot or Bingbot site-wide caps it at 30.
Does blocking GPTBot remove your site from ChatGPT?
Blocking GPTBot does not remove your site from ChatGPT search, because ChatGPT search results come from a different bot. OpenAI's documentation states that "OAI-SearchBot is used to surface websites in search results in ChatGPT's search features", while GPTBot collects content for training. Leave OAI-SearchBot allowed and a GPTBot block costs no search visibility; OpenAI says a robots.txt change takes about 24 hours to reach ChatGPT search.
Is robots.txt the same as a noindex tag?
Robots.txt is not the same as a noindex tag: robots.txt tells a crawler which paths it may fetch, while a noindex tag tells a search engine not to list the page. The two fail in different ways. In the GeoTest Benchmark Report 2026, 5.1% of the 4,999 sites, about one in twenty, had a noindex setting on their homepage, usually left on from when the site was built. GeoTest caps a noindexed page at 30, because a page search engines are told to skip cannot be retrieved for an answer.
What do you lose by blocking AI search crawlers?
Blocking AI search crawlers loses the citations those engines would have given you, and citations are increasingly where the attention goes. Similarweb data shows zero-click searches rose from 56% to 69% between May 2024 and May 2025, and Pew Research Center data from 2025 shows users clicked a link inside a Google AI summary in just 1% of visits. In GeoTest, blocking two or more AI search crawlers caps the site score at 70.
How do you see which AI crawlers visit your site?
Server logs show which AI crawlers visit your site, because each bot names itself in the user-agent string: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and the rest. Many hosting dashboards and CDNs can filter requests by user agent. Expect wasted requests as well as useful ones: in the same 2024 Vercel data, 34.82% of ChatGPT's crawler fetches hit 404 pages, so a clean sitemap and working links help every crawler spend its visits on pages that exist.
Should you block AI training crawlers?
Whether to block AI training crawlers is a business decision, not a GEO one. Blocking a crawler means adding a Disallow rule for its user agent, and blocking GPTBot, ClaudeBot or Google-Extended keeps new content out of future model training. It does not remove what is already there, and it does not affect live search citations. User-triggered fetchers sit outside the rules too: OpenAI's crawler notes for site owners report that for ChatGPT-User, "Because these actions are initiated by a user, robots.txt rules may not apply." If you want AI search engines to cite you, keep the five search-indexing bots allowed.
