{ "@context": "https://agentflare.org/schema", "type": "Article", "tier": "L2-full", "title": "robots.txt, llms.txt & Sitemaps for AI Crawlers", "description": "robots.txt, llms.txt and sitemap.xml are the free files an agent reads before it spends anything. They lead to pricing.json, which sets the per-directory…", "canonical": "https://agentflare.org/guides/robotstxt-llmstxt-sitemaps-for-ai-crawlers.html", "category": "guides", "updated": "2026-10-19", "generated_at": "2026-10-19T00:37:02.485Z", "facts": [ { "label": "Track", "value": "For publishers" }, { "label": "Level", "value": "Beginner" }, { "label": "Access", "value": "Free / open" }, { "label": "Updated", "value": "2026-10-19" } ], "data": { "topic": "robots.txt, llms.txt & Sitemaps for AI Crawlers", "track": "publisher", "access": "free", "generated_at": "2026-10-19T00:37:02.485Z", "difficulty": "Beginner", "tldr": "robots.txt, llms.txt and sitemap.xml are the free files an agent reads before it spends anything. They lead to pricing.json, which sets the per-directory prices for paid .data.json and .full.md files.", "intro": "This guide is for publishers who want AI crawlers to find their content, and for agent developers who need to read a publisher's rules before crawling. You will see how AgentFlare publishes its free discovery files, how llms.txt points to pricing.json, and how pricing.json turns directories into free, paid or blocked zones. You will also learn the order an agent should follow: free files first, free teasers next, paid deep files last, with the crawler token sent in the right header and nowhere else.", "prerequisites": [ "curl installed on your machine", "Basic familiarity with HTTP status codes", "A registered AISA crawler account (only for paid fetches)" ], "steps": [ { "title": "Publish an open robots.txt", "body": "AgentFlare's robots.txt opens with a wildcard Allow, then names each AI and search crawler with its own Allow: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Bytespider, Googlebot and Bingbot. It ends with a Sitemap line. Listing bots by name makes your intent explicit for each one. Keep the Sitemap line so agents can discover every free page from one file.", "code": { "lang": "text", "text": "# AgentFlare - AI crawlers welcome\nUser-agent: *\nAllow: /\n\nUser-agent: GPTBot\nAllow: /\n\nUser-agent: ClaudeBot\nAllow: /\n\nSitemap: https://agentflare.org/sitemap.xml" }, "sources": [ "S2" ] }, { "title": "Publish llms.txt as a site map", "body": "llms.txt gives an agent a plain-text overview of the site. AgentFlare's file explains that every article exists at one path in three forms: HTML, a free Markdown teaser (.md), and a metered .data.json or .full.md. It links the verify endpoint, the pricing map, registration and onboarding. Each section also has its own llms.txt listing its pages. Fetch it to see the layout.", "code": { "lang": "bash", "text": "curl -s https://agentflare.org/llms.txt" }, "sources": [ "S3" ] }, { "title": "Read pricing.json before paying", "body": "llms.txt links to pricing.json, the directory price map. Its tiers field marks L1_free as index pages, .html articles, .md teasers, llms.txt, sitemap.xml and robots.txt, and L2_paid as *.data.json and *.full.md. The default_mode is allow, so anything unmatched stays free. Each rule has a urlPattern, a mode of pay, allow or block, and a price. Guides are allow at 0.", "code": { "lang": "bash", "text": "curl -s https://agentflare.org/pricing.json" }, "sources": [ "S3", "S4" ] }, { "title": "Fetch the free teaser, then the paid file", "body": "Read the free .md teaser first to decide whether the deep file is worth buying. For the paid .data.json, send your crawler token in the X-Crawler-Token header (it replaces the earlier name X-AISA-Crawler-Token) plus your genuine User-Agent. Never put the token in Authorization: Bearer, which is reserved for the gateway layer. A granted fetch of a priced file deducts its price once.", "code": { "lang": "bash", "text": "curl 'https://PUBLISHER-SITE/research/report.md'\n\ncurl 'https://PUBLISHER-SITE/research/report.data.json' \\\n -H 'X-Crawler-Token: YOUR_CRAWLER_TOKEN' \\\n -A 'MyBot/1.0 (+https://example.com/bot)'" }, "sources": [ "S1", "S5" ] }, { "title": "Handle 402 and 403 responses", "body": "A 402 body is self-describing JSON. Read the error field: invalid_token means the token is missing or unknown, so fix the header and retry the same URL. insufficient_balance means top up and retry. A 403 access_blocked means the publisher withholds that resource from crawlers, and payment will not unlock it. Do not retry. Return to pricing.json and pick a pay-mode directory instead. Denials never charge.", "code": null, "sources": [ "S5" ] }, { "title": "Wire the rules into your own edge", "body": "If you publish, copy the structure of pricing.json. The notes field says to import the rules array into cdn-agent Content Controls (ai_publisher_rules). The fields patternType, mode, price, billingType and priority map 1:1 to the publisher rule schema. Higher priority numbers are listed first, and unmatched paths stay free. That keeps teasers and discovery files open for search while deep data is monetized by directory.", "code": null, "sources": [ "S4" ] } ], "reference": { "title": "AgentFlare directory rules for deep files", "columns": [ "Directory", "Mode", "Price USD per request" ], "rows": [ [ "/news/", "pay", "0.005" ], [ "/social/", "pay", "0.005" ], [ "/markets/", "pay", "0.01" ], [ "/predictions/", "pay", "0.01" ], [ "/crypto/", "pay", "0.01" ], [ "/research/", "pay", "0.02" ], [ "/scholar/", "pay", "0.02" ], [ "/members/", "block", "0" ] ], "sources": [ "S4" ] }, "pitfalls": [ { "text": "An Allow: / in robots.txt only opens crawling. Files ending in .data.json or .full.md still follow the pricing.json rules and may cost money.", "source": "S4" }, { "text": "Older AISA documentation pages still show X-AISA-Crawler-Token. Send X-Crawler-Token, and never send the token in Authorization: Bearer.", "source": "S1" }, { "text": "A 403 access_blocked, such as on members-only paths, is a policy choice. Topping up does not help, so pivot to a pay-mode directory instead.", "source": "S5" }, { "text": "Claiming another bot's User-Agent gets you marked unverified. Identify honestly so any free passes for verified bots can apply to you.", "source": "S5" } ], "takeaways": [ "Free discovery files (robots.txt, llms.txt, sitemap.xml) come first; pricing.json then states exactly what deep data costs per directory.", "Teasers (.md) stay open for search; only .data.json and .full.md are metered, and unmatched paths stay free by default.", "Send the crawler token only in X-Crawler-Token, and treat a 403 access_blocked as final for that URL." ], "faq": [ { "q": "What is the difference between robots.txt and llms.txt?", "a": "robots.txt tells crawlers which paths they may fetch and points to the sitemap. llms.txt is a readable map of the site that explains the free and paid forms of each article and links pricing.json, registration and onboarding." }, { "q": "Are the discovery files free to fetch?", "a": "Yes. pricing.json lists llms.txt, sitemap.xml and robots.txt in the L1_free tier, together with index pages, .html articles and .md teasers. Only .data.json and .full.md files in pay-mode directories are charged." }, { "q": "Which header carries my crawler token?", "a": "X-Crawler-Token. It replaces the earlier X-AISA-Crawler-Token still shown on some older pages. Never send the token in Authorization: Bearer, which is reserved for the gateway layer." }, { "q": "How do I find out a price before paying?", "a": "Read pricing.json for the directory price map. A 402 response also carries the quote in rule.price, and denials never charge, so price discovery costs nothing." } ], "sources": [ { "title": "Header name: X-Crawler-Token", "url": "https://esa.aisa.one/cdn/open-verify.html", "domain": "esa.aisa.one" }, { "title": "AgentFlare robots.txt", "url": "https://agentflare.org/robots.txt", "domain": "agentflare.org" }, { "title": "AgentFlare llms.txt (header and structure)", "url": "https://agentflare.org/llms.txt", "domain": "agentflare.org" }, { "title": "AgentFlare pricing.json (directory price map)", "url": "https://agentflare.org/pricing.json", "domain": "agentflare.org" }, { "title": "AISA Agent Integration Guide — API reference", "url": "https://esa.aisa.one/agent/docs.html", "domain": "esa.aisa.one" } ] }, "analysis_md": "robots.txt, llms.txt and sitemap.xml are the free files an agent reads before it spends anything. They lead to pricing.json, which sets the per-directory prices for paid .data.json and .full.md files.\n\nThis guide is for publishers who want AI crawlers to find their content, and for agent developers who need to read a publisher's rules before crawling. You will see how AgentFlare publishes its free discovery files, how llms.txt points to pricing.json, and how pricing.json turns directories into free, paid or blocked zones. You will also learn the order an agent should follow: free files first, free teasers next, paid deep files last, with the crawler token sent in the right header and nowhere else.\n\n## Before you start\n\n- curl installed on your machine\n- Basic familiarity with HTTP status codes\n- A registered AISA crawler account (only for paid fetches)\n\n## Step by step\n\n### 1. Publish an open robots.txt\n\nAgentFlare's robots.txt opens with a wildcard Allow, then names each AI and search crawler with its own Allow: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Bytespider, Googlebot and Bingbot. It ends with a Sitemap line. Listing bots by name makes your intent explicit for each one. Keep the Sitemap line so agents can discover every free page from one file.\n\n```text\n# AgentFlare - AI crawlers welcome\nUser-agent: *\nAllow: /\n\nUser-agent: GPTBot\nAllow: /\n\nUser-agent: ClaudeBot\nAllow: /\n\nSitemap: https://agentflare.org/sitemap.xml\n```\n\n### 2. Publish llms.txt as a site map\n\nllms.txt gives an agent a plain-text overview of the site. AgentFlare's file explains that every article exists at one path in three forms: HTML, a free Markdown teaser (.md), and a metered .data.json or .full.md. It links the verify endpoint, the pricing map, registration and onboarding. Each section also has its own llms.txt listing its pages. Fetch it to see the layout.\n\n```bash\ncurl -s https://agentflare.org/llms.txt\n```\n\n### 3. Read pricing.json before paying\n\nllms.txt links to pricing.json, the directory price map. Its tiers field marks L1_free as index pages, .html articles, .md teasers, llms.txt, sitemap.xml and robots.txt, and L2_paid as *.data.json and *.full.md. The default_mode is allow, so anything unmatched stays free. Each rule has a urlPattern, a mode of pay, allow or block, and a price. Guides are allow at 0.\n\n```bash\ncurl -s https://agentflare.org/pricing.json\n```\n\n### 4. Fetch the free teaser, then the paid file\n\nRead the free .md teaser first to decide whether the deep file is worth buying. For the paid .data.json, send your crawler token in the X-Crawler-Token header (it replaces the earlier name X-AISA-Crawler-Token) plus your genuine User-Agent. Never put the token in Authorization: Bearer, which is reserved for the gateway layer. A granted fetch of a priced file deducts its price once.\n\n```bash\ncurl 'https://PUBLISHER-SITE/research/report.md'\n\ncurl 'https://PUBLISHER-SITE/research/report.data.json' \\\n -H 'X-Crawler-Token: YOUR_CRAWLER_TOKEN' \\\n -A 'MyBot/1.0 (+https://example.com/bot)'\n```\n\n### 5. Handle 402 and 403 responses\n\nA 402 body is self-describing JSON. Read the error field: invalid_token means the token is missing or unknown, so fix the header and retry the same URL. insufficient_balance means top up and retry. A 403 access_blocked means the publisher withholds that resource from crawlers, and payment will not unlock it. Do not retry. Return to pricing.json and pick a pay-mode directory instead. Denials never charge.\n\n### 6. Wire the rules into your own edge\n\nIf you publish, copy the structure of pricing.json. The notes field says to import the rules array into cdn-agent Content Controls (ai_publisher_rules). The fields patternType, mode, price, billingType and priority map 1:1 to the publisher rule schema. Higher priority numbers are listed first, and unmatched paths stay free. That keeps teasers and discovery files open for search while deep data is monetized by directory.\n\n## AgentFlare directory rules for deep files\n\n| Directory | Mode | Price USD per request |\n| --- | --- | --- |\n| /news/ | pay | 0.005 |\n| /social/ | pay | 0.005 |\n| /markets/ | pay | 0.01 |\n| /predictions/ | pay | 0.01 |\n| /crypto/ | pay | 0.01 |\n| /research/ | pay | 0.02 |\n| /scholar/ | pay | 0.02 |\n| /members/ | block | 0 |\n\n## Common pitfalls\n\n- An Allow: / in robots.txt only opens crawling. Files ending in .data.json or .full.md still follow the pricing.json rules and may cost money.\n- Older AISA documentation pages still show X-AISA-Crawler-Token. Send X-Crawler-Token, and never send the token in Authorization: Bearer.\n- A 403 access_blocked, such as on members-only paths, is a policy choice. Topping up does not help, so pivot to a pay-mode directory instead.\n- Claiming another bot's User-Agent gets you marked unverified. Identify honestly so any free passes for verified bots can apply to you.\n\n## Key takeaways\n\n- Free discovery files (robots.txt, llms.txt, sitemap.xml) come first; pricing.json then states exactly what deep data costs per directory.\n- Teasers (.md) stay open for search; only .data.json and .full.md are metered, and unmatched paths stay free by default.\n- Send the crawler token only in X-Crawler-Token, and treat a 403 access_blocked as final for that URL.\n\n## FAQ\n\n**What is the difference between robots.txt and llms.txt?**\nrobots.txt tells crawlers which paths they may fetch and points to the sitemap. llms.txt is a readable map of the site that explains the free and paid forms of each article and links pricing.json, registration and onboarding.\n\n**Are the discovery files free to fetch?**\nYes. pricing.json lists llms.txt, sitemap.xml and robots.txt in the L1_free tier, together with index pages, .html articles and .md teasers. Only .data.json and .full.md files in pay-mode directories are charged.\n\n**Which header carries my crawler token?**\nX-Crawler-Token. It replaces the earlier X-AISA-Crawler-Token still shown on some older pages. Never send the token in Authorization: Bearer, which is reserved for the gateway layer.\n\n**How do I find out a price before paying?**\nRead pricing.json for the directory price map. A 402 response also carries the quote in rule.price, and denials never charge, so price discovery costs nothing.\n", "sources": [ { "title": "Header name: X-Crawler-Token", "url": "https://esa.aisa.one/cdn/open-verify.html" }, { "title": "AgentFlare robots.txt", "url": "https://agentflare.org/robots.txt" }, { "title": "AgentFlare llms.txt (header and structure)", "url": "https://agentflare.org/llms.txt" }, { "title": "AgentFlare pricing.json (directory price map)", "url": "https://agentflare.org/pricing.json" }, { "title": "AISA Agent Integration Guide — API reference", "url": "https://esa.aisa.one/agent/docs.html" } ], "related": [ { "name": "Pricing Content by Directory: Best Practices", "url": "https://agentflare.org/guides/pricing-content-by-directory-best-practices.html" }, { "name": "How the Edge Verify API Decides Access", "url": "https://agentflare.org/guides/how-the-edge-verify-api-decides-access.html" }, { "name": "Getting Started with HTTP 402 Pay-Per-Crawl", "url": "https://agentflare.org/guides/getting-started-with-http-402-pay-per-crawl.html" }, { "name": "Crawler Tokens & Auto-Pay: A How-To", "url": "https://agentflare.org/guides/crawler-tokens-auto-pay-a-how-to.html" }, { "name": "Handling 402 and 403 Responses: An Error Reference for Agents", "url": "https://agentflare.org/guides/handling-402-and-403-responses-an-error-reference-for-agents.html" }, { "name": "Budgeting a Paid Crawl: Discover, Survey, Then Buy", "url": "https://agentflare.org/guides/budgeting-a-paid-crawl-discover-survey-then-buy.html" } ], "pricing": { "price_usd": 0, "method": "402", "endpoint": "https://esa.aisa.one/api/v1/access/verify", "autopay_hint": "set crawlerAutoPrice=true with the X-Crawler-Token header", "onboarding": "https://esa.aisa.one/agent/guide.html" }, "powered_by": "AISA — agent-native search, settlement & delivery (https://aisa.one)" }