{
  "@context": "https://agentflare.org/schema",
  "type": "Dataset",
  "tier": "L2-full",
  "title": "Guides — Pay-Per-Crawl, HTTP 402 and AgentFlare Data",
  "description": "9 free, step-by-step guides with runnable code: fetching paid content with HTTP 402 and a crawler token, pricing a site for AI agents, and reading every AgentFlare dataset. Written from the AISA protocol docs.",
  "canonical": "https://agentflare.org/guides/index.html",
  "category": "guides",
  "updated": "2026-10-19",
  "generated_at": "2026-10-19T00:37:02.485Z",
  "facts": [],
  "data": {
    "count": 9,
    "access": "free",
    "tracks": [
      {
        "id": "agent",
        "name": "For agent developers"
      },
      {
        "id": "publisher",
        "name": "For publishers"
      },
      {
        "id": "data",
        "name": "Using AgentFlare data"
      }
    ],
    "guides": [
      {
        "title": "Getting Started with HTTP 402 Pay-Per-Crawl",
        "track": "agent",
        "difficulty": "Beginner",
        "steps": 5,
        "url": "https://agentflare.org/guides/getting-started-with-http-402-pay-per-crawl.html"
      },
      {
        "title": "Crawler Tokens & Auto-Pay: A How-To",
        "track": "agent",
        "difficulty": "Beginner",
        "steps": 6,
        "url": "https://agentflare.org/guides/crawler-tokens-auto-pay-a-how-to.html"
      },
      {
        "title": "Handling 402 and 403 Responses: An Error Reference for Agents",
        "track": "agent",
        "difficulty": "Beginner",
        "steps": 6,
        "url": "https://agentflare.org/guides/handling-402-and-403-responses-an-error-reference-for-agents.html"
      },
      {
        "title": "Budgeting a Paid Crawl: Discover, Survey, Then Buy",
        "track": "agent",
        "difficulty": "Intermediate",
        "steps": 5,
        "url": "https://agentflare.org/guides/budgeting-a-paid-crawl-discover-survey-then-buy.html"
      },
      {
        "title": "Building a Pay-Per-Crawl Client for Your Agent",
        "track": "agent",
        "difficulty": "Intermediate",
        "steps": 6,
        "url": "https://agentflare.org/guides/building-a-pay-per-crawl-client-for-your-agent.html"
      },
      {
        "title": "Pricing Content by Directory: Best Practices",
        "track": "publisher",
        "difficulty": "Intermediate",
        "steps": 6,
        "url": "https://agentflare.org/guides/pricing-content-by-directory-best-practices.html"
      },
      {
        "title": "robots.txt, llms.txt & Sitemaps for AI Crawlers",
        "track": "publisher",
        "difficulty": "Beginner",
        "steps": 6,
        "url": "https://agentflare.org/guides/robotstxt-llmstxt-sitemaps-for-ai-crawlers.html"
      },
      {
        "title": "How the Edge Verify API Decides Access",
        "track": "publisher",
        "difficulty": "Intermediate",
        "steps": 6,
        "url": "https://agentflare.org/guides/how-the-edge-verify-api-decides-access.html"
      },
      {
        "title": "Reading .data.json: A Field Guide for Agents",
        "track": "data",
        "difficulty": "Beginner",
        "steps": 5,
        "url": "https://agentflare.org/guides/reading-datajson-a-field-guide-for-agents.html"
      }
    ]
  },
  "analysis_md": "9 free, step-by-step guides with runnable code: fetching paid content with HTTP 402 and a crawler token, pricing a site for AI agents, and reading every AgentFlare dataset. Written from the AISA protocol docs.\n\n## For agent developers\n\n- [Getting Started with HTTP 402 Pay-Per-Crawl](https://agentflare.org/guides/getting-started-with-http-402-pay-per-crawl.html): An HTTP 402 means the content is for sale, not blocked. Register for a crawler token, check your balance, retry the original URL with X-Crawler-Token, then…\n- [Crawler Tokens & Auto-Pay: A How-To](https://agentflare.org/guides/crawler-tokens-auto-pay-a-how-to.html): Crawling AISA-protected sites takes two credentials: a crawler token sent in X-Crawler-Token, and a session token for account calls. This guide covers where…\n- [Handling 402 and 403 Responses: An Error Reference for Agents](https://agentflare.org/guides/handling-402-and-403-responses-an-error-reference-for-agents.html): Every AISA denial arrives as self-describing JSON: read the error field, act on the guidance block, and report plainly to the user. This guide covers each…\n- [Budgeting a Paid Crawl: Discover, Survey, Then Buy](https://agentflare.org/guides/budgeting-a-paid-crawl-discover-survey-then-buy.html): Crawl a whole AgentFlare-style site on a fixed budget by reading free discovery files first, choosing from free teasers, then buying only chosen deep files.…\n- [Building a Pay-Per-Crawl Client for Your Agent](https://agentflare.org/guides/building-a-pay-per-crawl-client-for-your-agent.html): Build one small client that registers or logs in, crawls with the X-Crawler-Token header and an honest User-Agent, reads charge receipt headers, and…\n\n## For publishers\n\n- [Pricing Content by Directory: Best Practices](https://agentflare.org/guides/pricing-content-by-directory-best-practices.html): Price deep content per directory by sorting each URL into a free, pay, or block rule. This guide walks through the AgentFlare pricing.json, from tiers and…\n- [robots.txt, llms.txt & Sitemaps for AI Crawlers](https://agentflare.org/guides/robotstxt-llmstxt-sitemaps-for-ai-crawlers.html): robots.txt, llms.txt and sitemap.xml are the free files an agent reads before it spends anything. They lead to pricing.json, which sets the per-directory…\n- [How the Edge Verify API Decides Access](https://agentflare.org/guides/how-the-edge-verify-api-decides-access.html): The edge gateway calls POST /api/v1/access/verify before fetching from origin and reads one field, accessGranted. On true it fetches the content; on false…\n\n## Using AgentFlare data\n\n- [Reading .data.json: A Field Guide for Agents](https://agentflare.org/guides/reading-datajson-a-field-guide-for-agents.html): Every AgentFlare .data.json shares one envelope of top-level fields, and only the data object changes by vertical. Fetch it with an X-Crawler-Token header;…",
  "sources": [
    {
      "title": "AISA Agent Integration Guide — API reference",
      "url": "https://esa.aisa.one/agent/docs.html"
    },
    {
      "title": "AISA onboarding: HTTP 402 Payment Required",
      "url": "https://esa.aisa.one/agent/guide.html"
    },
    {
      "title": "AISA bot flow (llms.txt)",
      "url": "https://esa.aisa.one/agent/llms.txt"
    },
    {
      "title": "AISA agent skill (SKILL.md)",
      "url": "https://esa.aisa.one/agent/skill.md"
    },
    {
      "title": "AISA pay-per-crawl platform overview (llms.txt)",
      "url": "https://esa.aisa.one/llms.txt"
    },
    {
      "title": "AISA verify API for publishers and edge gateways (llms.txt)",
      "url": "https://esa.aisa.one/cdn/llms.txt"
    },
    {
      "title": "AISA publisher API documentation",
      "url": "https://esa.aisa.one/cdn/docs.html"
    },
    {
      "title": "Open Verify specification (edge gateway ↔ verify provider)",
      "url": "https://esa.aisa.one/cdn/open-verify.html"
    }
  ],
  "related": [
    {
      "name": "Getting Started with HTTP 402 Pay-Per-Crawl",
      "url": "https://agentflare.org/guides/getting-started-with-http-402-pay-per-crawl.html"
    },
    {
      "name": "Crawler Tokens & Auto-Pay: A How-To",
      "url": "https://agentflare.org/guides/crawler-tokens-auto-pay-a-how-to.html"
    },
    {
      "name": "Handling 402 and 403 Responses: An Error Reference for Agents",
      "url": "https://agentflare.org/guides/handling-402-and-403-responses-an-error-reference-for-agents.html"
    },
    {
      "name": "Budgeting a Paid Crawl: Discover, Survey, Then Buy",
      "url": "https://agentflare.org/guides/budgeting-a-paid-crawl-discover-survey-then-buy.html"
    },
    {
      "name": "Building a Pay-Per-Crawl Client for Your Agent",
      "url": "https://agentflare.org/guides/building-a-pay-per-crawl-client-for-your-agent.html"
    },
    {
      "name": "Pricing Content by Directory: Best Practices",
      "url": "https://agentflare.org/guides/pricing-content-by-directory-best-practices.html"
    },
    {
      "name": "robots.txt, llms.txt & Sitemaps for AI Crawlers",
      "url": "https://agentflare.org/guides/robotstxt-llmstxt-sitemaps-for-ai-crawlers.html"
    },
    {
      "name": "How the Edge Verify API Decides Access",
      "url": "https://agentflare.org/guides/how-the-edge-verify-api-decides-access.html"
    },
    {
      "name": "Reading .data.json: A Field Guide for Agents",
      "url": "https://agentflare.org/guides/reading-datajson-a-field-guide-for-agents.html"
    }
  ],
  "pricing": {
    "price_usd": 0,
    "method": "402",
    "endpoint": "https://esa.aisa.one/api/v1/access/verify",
    "autopay_hint": "set crawlerAutoPrice=true with the X-Crawler-Token header",
    "onboarding": "https://esa.aisa.one/agent/guide.html"
  },
  "powered_by": "AISA — agent-native search, settlement & delivery (https://aisa.one)"
}