# Pay-Per-Crawl and the New Bot Web

> AI companies run separate bots for training, search and user-requested fetches, so one blanket robots.txt rule can block the wrong thing. Cloudflare data…

- **Canonical:** https://agentflare.org/research/pay-per-crawl-and-the-new-bot-web.html
- **Updated:** 2026-10-16
- **Category:** research
- **Full structured data:** `/research/pay-per-crawl-and-the-new-bot-web.data.json` — $0.02 via AISA HTTP 402 (https://esa.aisa.one/api/v1/access/verify; agents set crawlerAutoPrice=true)

## Key data

- **Cluster:** Discovery & crawling
- **Sources:** 7
- **Updated:** 2026-10-16

AI companies run separate bots for training, search and user-requested fetches, so one blanket robots.txt rule can block the wrong thing. Cloudflare data shows heavy crawling with few referrals back, which is the backdrop for pay per crawl.

_…full analysis and the complete dataset are available to agents for $0.02 — fetch `/research/pay-per-crawl-and-the-new-bot-web.data.json` (HTTP 402)._

## Sources

1. [GPTBot, ClaudeBot, and PerplexityBot: An AI Crawler Guide | The Ad Firm](https://www.theadfirm.net/what-is-gptbot-claudebot-and-perplexitybot-a-small-business-guide-to-ai-crawlers)
2. [AI Crawlers Explained: GPTBot, ClaudeBot, PerplexityBot and How to Let Them In (2026) | Anagram](https://www.anagram.ai/blog/ai-crawlers-explained-gptbot-claudebot-perplexitybot-and-how-to-let-them-in-2026)
3. [AI Crawlers Explained: How GPTBot, ClaudeBot & PerplexityBot Crawl Your Website? – Triomize](https://triomize.com/blog/ai-crawlers)
4. [AI crawlers & redirects: GPTBot, ClaudeBot, Perplexity 2026](https://www.captaindns.com/en/blog/ai-crawlers-redirects-handling-gptbot-claudebot-perplexitybot)
5. [The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals | Cloudflare Blog](https://blog.cloudflare.com/crawlers-click-ai-bots-training)
6. [AI Crawler Monitoring: GPTBot, ClaudeBot & PerplexityBot | Webalert](https://web-alert.io/blog/ai-crawler-bot-monitoring-gptbot-claudebot-perplexitybot-guide)
7. [Best Tools to Track AI Crawlers on Your Website](https://www.xseek.io/blogs/articles/best-tools-to-track-ai-crawlers-on-your-website)

## Related

- [Generative Engine Optimization (GEO): A Primer](https://agentflare.org/research/generative-engine-optimization-geo-a-primer.html)
- [llms.txt: The Standard for AI-Readable Sites](https://agentflare.org/research/llmstxt-the-standard-for-ai-readable-sites.html)
- [Crawler Policy on the Modern Web: robots.txt and Beyond](https://agentflare.org/research/crawler-policy-on-the-modern-web-robotstxt-and-beyond.html)
- [Structured Data for LLMs: JSON-LD That Agents Actually Use](https://agentflare.org/research/structured-data-for-llms-json-ld-that-agents-actually-use.html)
- [HTTP 402 & x402: How AI Agents Pay for Content](https://agentflare.org/research/http-402-x402-how-ai-agents-pay-for-content.html)
- [The AI Agent Economy in 2026](https://agentflare.org/research/the-ai-agent-economy-in-2026.html)

---
_Part of AgentFlare, an agent-native data network powered by AISA. https://aisa.one/docs_