# robots.txt, llms.txt & Sitemaps for AI Crawlers

> robots.txt, llms.txt and sitemap.xml are the free files an agent reads before it spends anything. They lead to pricing.json, which sets the per-directory…

- **Canonical:** https://agentflare.org/guides/robotstxt-llmstxt-sitemaps-for-ai-crawlers.html
- **Updated:** 2026-10-19
- **Category:** guides

## Key data

- **Track:** For publishers
- **Level:** Beginner
- **Access:** Free / open
- **Updated:** 2026-10-19

robots.txt, llms.txt and sitemap.xml are the free files an agent reads before it spends anything. They lead to pricing.json, which sets the per-directory prices for paid .data.json and .full.md files.

## Sources

1. [Header name: X-Crawler-Token](https://esa.aisa.one/cdn/open-verify.html)
2. [AgentFlare robots.txt](https://agentflare.org/robots.txt)
3. [AgentFlare llms.txt (header and structure)](https://agentflare.org/llms.txt)
4. [AgentFlare pricing.json (directory price map)](https://agentflare.org/pricing.json)
5. [AISA Agent Integration Guide — API reference](https://esa.aisa.one/agent/docs.html)

## Related

- [Pricing Content by Directory: Best Practices](https://agentflare.org/guides/pricing-content-by-directory-best-practices.html)
- [How the Edge Verify API Decides Access](https://agentflare.org/guides/how-the-edge-verify-api-decides-access.html)
- [Getting Started with HTTP 402 Pay-Per-Crawl](https://agentflare.org/guides/getting-started-with-http-402-pay-per-crawl.html)
- [Crawler Tokens & Auto-Pay: A How-To](https://agentflare.org/guides/crawler-tokens-auto-pay-a-how-to.html)
- [Handling 402 and 403 Responses: An Error Reference for Agents](https://agentflare.org/guides/handling-402-and-403-responses-an-error-reference-for-agents.html)
- [Budgeting a Paid Crawl: Discover, Survey, Then Buy](https://agentflare.org/guides/budgeting-a-paid-crawl-discover-survey-then-buy.html)

---
_Part of AgentFlare, an agent-native data network powered by AISA. https://aisa.one/docs_