Pay-Per-Crawl and the New Bot Web

Key concepts

Training crawler

A bot that collects public web content that may be used to train generative models. GPTBot and ClaudeBot are the named examples. Both can be restricted through robots.txt. S1 · theadfirm.netS3 · triomize.com

Search crawler

A bot that indexes or surfaces pages in an AI search product. Examples are OAI-SearchBot for ChatGPT Search and Claude-SearchBot. Perplexity describes PerplexityBot as primarily a search crawler. S1 · theadfirm.netS3 · triomize.com

User-initiated fetcher

A bot that retrieves a page at the moment a user asks about it, such as ChatGPT-User or Claude-User. OpenAI says robots.txt may not apply to ChatGPT-User requests. S1 · theadfirm.netS6 · web-alert.io

Google-Extended token

A robots.txt product token for Gemini training and grounding in some non-Search systems. It is not a request user agent and does not control AI Overviews or AI Mode. S3 · triomize.comS6 · web-alert.io

Pay per crawl

A Cloudflare feature that the company covered in early July 2025, alongside trends in AI crawlers consuming web data while sending few visitors back. S5 · blog.cloudflare.com

How an AI crawler request is handled

1. DiscoveryThe crawler finds a URL through links, sitemaps, search indexes, user prompts or known web datasets.
2. Permission checkIt requests /robots.txt and checks the rules that apply to its user agent.
3. FetchingIf allowed, it downloads HTML, media or other public resources.
4. ParsingIt extracts visible text, links, metadata, headings, schema and page structure.
5. Use case routingThe content may support training, search indexing, live answer retrieval or citation selection.
6. LoggingThe request appears in server logs with a user agent, status code, path, IP address and timestamp.

One vendor, several bots, separate controls

Each major AI company splits its crawling by purpose. OpenAI documents GPTBot for training, OAI-SearchBot for ChatGPT Search and ChatGPT-User for user-directed requests; one source also lists OAI-AdsBot as a fourth role. Anthropic documents ClaudeBot, Claude-SearchBot and Claude-User. Perplexity documents PerplexityBot and Perplexity-User, and says PerplexityBot surfaces and links sites in search results rather than crawling for foundation model training. Because the agents have independent robots.txt settings, a rule aimed at training can remove a site from AI search if written too broadly. OpenAI specifically tells publishers to block the GPTBot user agent on pages they want excluded from potential training, and blocking GPTBot does not automatically keep a site out of ChatGPT Search. Monitoring guidance adds that names and IP ranges change, so vendor documentation, not a copied list, should be the source of truth.

The crawl-to-click gap and why it matters

Cloudflare reports that crawling by search engines and AI services surged in the first half of 2025, up 24% year-over-year in June, before slowing to 4% year-over-year growth in July. Training-related crawling led the way. At the same time, in Cloudflare's dataset of news-related customers, Google referrals declined from February 2025: March 2025 referral traffic was 9% lower than January, and April was down 15% compared with January. Cloudflare links this to Google's AI Overviews and AI Mode changes. The result for publishers is many bot hits but far fewer readers clicking through, which means fewer ad views and subscription chances. Other sources describe AI referrals as growing from a small base: ChatGPT accounts for 87.4% of AI referral traffic, and AI search visits grew 42.8% year over year. That tension between giving content to bots and getting visitors back is the context for pay per crawl.

Operating in the bot web: logs, redirects, tooling

Standard analytics such as Google Analytics do not show AI bot traffic, so server logs or a bot-management layer are needed. Options listed include Cloudflare Bot Management, which is available on Business and Enterprise plans, raw server logs, Screaming Frog log analysis and specialist tools. Monitoring guidance says to verify each request where the vendor provides a method, map it to its documented purpose and compare it with your access policy, rather than treating all AI bots as one bucket. High volume can also appear as latency spikes, 429s and 5xx storms on under-provisioned sites. Redirects matter too: none of the three vendors publicly states how many hops its crawler tolerates, and observed chain tolerance is 3 to 5 hops. The advice is for the final URL to return 200 OK in 1 to 3 hops maximum.

Documented crawler roles by vendor

Timeline

1. 2024-05
   Google AI Overviews launch in the U.S.blog.cloudflare.com
2. 2025-03
   Google upgrades AI Overviews with Gemini 2.0 and introduces AI Mode in Labs.blog.cloudflare.com
3. 2025-05
   AI Mode rolls out broadly in the U.S. with Gemini 2.5.blog.cloudflare.com
4. 2025-07
   Cloudflare covers AI crawler trends and features such as pay per crawl.blog.cloudflare.com
5. 2025-09
   ChatGPT reaches 700 million weekly active users.anagram.ai

Common pitfalls

• Blocking one bot does not block the others. An all-or-nothing robots.txt rule meant to stop training can also remove content from AI search.
• Google-Extended is a robots.txt token, not a user agent, and it does not control inclusion in AI Overviews or AI Mode, which use Googlebot.
• Redirect chains that pass for Google can drop a page from AI answers. The loss is silent because analytics tools do not reflect it.
• Treating user-initiated fetchers like training crawlers is risky: OpenAI says robots.txt may not apply to ChatGPT-User requests.

Key takeaways

• Set robots.txt per agent by purpose: training, search and user fetch are separate decisions for each vendor.
• Measure bot traffic with server logs or bot management, and verify user agents and IPs against vendor documentation.
• Cloudflare data shows heavy AI crawling with weak referral return, the gap that pay per crawl-style controls address.

FAQ

How this page is made. Built from the full text of 7 sources found with the AISA Tavily API (advanced search), refreshed weekly. A language model organizes what those sources state into the blocks above; each block links the sources it draws on, and figures that do not appear in the sources are removed.