{ "@context": "https://agentflare.org/schema", "type": "Article", "tier": "L2-full", "title": "Getting Started with HTTP 402 Pay-Per-Crawl", "description": "An HTTP 402 means the content is for sale, not blocked. Register for a crawler token, check your balance, retry the original URL with X-Crawler-Token, then…", "canonical": "https://agentflare.org/guides/getting-started-with-http-402-pay-per-crawl.html", "category": "guides", "updated": "2026-10-19", "generated_at": "2026-10-19T00:37:02.485Z", "facts": [ { "label": "Track", "value": "For agent developers" }, { "label": "Level", "value": "Beginner" }, { "label": "Access", "value": "Free / open" }, { "label": "Updated", "value": "2026-10-19" } ], "data": { "topic": "Getting Started with HTTP 402 Pay-Per-Crawl", "track": "agent", "access": "free", "generated_at": "2026-10-19T00:37:02.485Z", "difficulty": "Beginner", "tldr": "An HTTP 402 means the content is for sale, not blocked. Register for a crawler token, check your balance, retry the original URL with X-Crawler-Token, then read the 200, 402 or 403 result.", "intro": "This guide is for developers building agents or crawlers that run into HTTP 402 Payment Required on a publisher site. You will walk through the full first paid fetch: recognising what a 402 means, registering an account, saving the crawler token, checking and funding your balance, retrying the original URL with the right header, and interpreting the 200, 402 or 403 that comes back. Every command uses the AISA account endpoints and the single crawl header, so you can paste the snippets into a script and complete one paid request end to end.", "prerequisites": [ "A paid publisher URL that returned HTTP 402 to your crawler", "curl, or Python with the requests library", "An email address and a password of at least 8 characters" ], "steps": [ { "title": "Recognise what the 402 means", "body": "A 402 from a publisher edge is a standard payment signal, not a block. The edge checked the URL's access rules and found no valid crawler token, or a token with too little balance, so it answered 402 instead of the content. The body is JSON: read error first, then follow the guidance block and the onboarding link. Denials never charge you.", "code": { "lang": "text", "text": "HTTP/1.1 402 Payment Required\nContent-Type: application/json\nX-Tengine-Error: payment required by http_crawler\n\nerror: invalid_token\nonboarding: https://esa.aisa.one/agent/guide.html" }, "sources": [ "S2", "S4" ] }, { "title": "Register and save the crawler token", "body": "Register once with POST /api/agent/auth/register. The response holds crawler.crawlerToken (format aisa_crawler_ followed by 64 hex characters) and a session token in token. Keep both, and never print the full crawler token in logs. The header that carries it is X-Crawler-Token, which replaces the earlier name X-AISA-Crawler-Token. If signupBonus is present, you already have trial credit.", "code": { "lang": "python", "text": "import requests\nBASE = 'https://cdn.aisa.one'\nr = requests.post(BASE + '/api/agent/auth/register', json={\n 'email': 'mybot@example.com',\n 'password': 'AtLeast8Chars',\n 'name': 'My Crawler',\n 'crawlerType': 'custom'})\ndata = r.json()\ncrawlerToken = data['crawler']['crawlerToken']\nsessionToken = data['token']\nprint(data['crawler']['balance'], data.get('signupBonus'))" }, "sources": [ "S1", "S3" ] }, { "title": "Check the balance and top up", "body": "Fetch the real balance with GET /api/agent/balance, using the same crawler token you crawl with. If you need funds, POST /api/agent/topup/create-session returns a checkout_url for Stripe, with a minimum of 0.50 USD. The top-up call takes the session token in Authorization: Bearer; a human must open the checkout_url and pay. Never send the crawler token in Authorization: Bearer.", "code": { "lang": "python", "text": "import requests\nBASE = 'https://cdn.aisa.one'\nb = requests.get(BASE + '/api/agent/balance',\n headers={'X-Crawler-Token': 'YOUR_CRAWLER_TOKEN'})\nprint(b.json()['crawler']['balance'])\n\nt = requests.post(BASE + '/api/agent/topup/create-session',\n headers={'Authorization': 'Bearer SESSION_TOKEN'},\n json={'amount': 5.00})\nprint(t.json()['checkout_url'])" }, "sources": [ "S1", "S3" ] }, { "title": "Retry the original URL with your token", "body": "Send the same publisher URL that returned 402, this time with X-Crawler-Token and an honest User-Agent. Do not change the path or call a payment endpoint: the publisher edge verifies the token, deducts the price from your balance and serves the content in the same request. One request from you results in at most one charge.", "code": { "lang": "bash", "text": "curl -i 'https://PUBLISHER-SITE/research/report.data.json' \\\n -H 'X-Crawler-Token: YOUR_CRAWLER_TOKEN' \\\n -A 'MyBot/1.0 (+https://example.com/bot)'" }, "sources": [ "S1", "S3" ] }, { "title": "Read the outcome: 200, 402 or 403", "body": "On 200, the content is yours; a paid fetch may carry the receipt headers X-AISA-Charged and X-AISA-Charge-Amount. On 402, read error: invalid_token means fix or register the token, insufficient_balance means top up using crawler.shortfall, then retry. On 403 access_blocked, the publisher withholds the resource from crawlers, so paying will not unlock it; look at the site's /pricing.json instead.", "code": { "lang": "python", "text": "import requests\nurl = 'https://PUBLISHER-SITE/research/report.data.json'\nr = requests.get(url, headers={\n 'X-Crawler-Token': 'YOUR_CRAWLER_TOKEN',\n 'User-Agent': 'MyBot/1.0 (+https://example.com/bot)'})\nif r.status_code == 200:\n print('charged:', r.headers.get('X-AISA-Charged'),\n r.headers.get('X-AISA-Charge-Amount'))\nelse:\n body = r.json()\n print(r.status_code, body.get('error'))\n print(body.get('guidance', {}).get('steps'))" }, "sources": [ "S2", "S3" ] } ], "reference": { "title": "Crawl outcomes and what to do next", "columns": [ "Response", "Meaning", "Your action" ], "rows": [ [ "200 (free tier)", "Open content such as HTML pages, .md summaries or discovery files", "Use it; cost is $0" ], [ "200 (paid)", "A priced resource your token just paid for", "Read the receipt headers and report the amount charged" ], [ "402 invalid_token", "No, unknown or inactive crawler token", "Register or fix the X-Crawler-Token header, then retry the same URL" ], [ "402 insufficient_balance", "Balance too low; crawler.shortfall gives the gap", "Top up, then retry" ], [ "402 price_required", "Price discovery; the quote is in price and currency", "Retry with crawlerAutoPrice set to true, which gateways usually do for you" ], [ "402 price_mismatch / price_too_high", "Your exact or maximum price did not match the quote", "Adjust to the quoted price, or skip" ], [ "403 access_blocked", "The publisher withholds this from crawlers; payment does not unlock it", "Do not retry or top up; check /pricing.json for purchasable directories" ] ], "sources": [ "S2", "S3" ] }, "pitfalls": [ { "text": "Sending the crawler token in Authorization: Bearer makes the request count as unauthenticated and returns 402 invalid_token. Use X-Crawler-Token only.", "source": "S3" }, { "text": "The session token from register or login is for account APIs such as top-up only. The crawler token is what you send to publisher sites.", "source": "S3" }, { "text": "Do not quote a remembered balance. Call GET /api/agent/balance before and after a top-up, since every paid fetch changes it.", "source": "S3" }, { "text": "Fetching a priced resource twice charges twice. A 403 access_blocked is a publisher policy, so retrying or topping up will not change it.", "source": "S3" } ], "takeaways": [ "A 402 is an invitation to pay: register, hold a balance, and retry the same URL with X-Crawler-Token.", "Denials never charge; only a granted fetch of a priced resource deducts its per-request price.", "Treat 403 access_blocked differently from 402: payment cannot unlock it, so pivot to the pay-mode directories in /pricing.json." ], "faq": [ { "q": "What does an HTTP 402 response mean for my crawler?", "a": "The resource exists but requires a valid crawler token and sufficient balance. It is a standard payment signal, not an error or a block. Read the error field in the JSON body and follow the guidance steps." }, { "q": "Which header carries the crawler token?", "a": "X-Crawler-Token. It replaces the earlier X-AISA-Crawler-Token. Never put the token in Authorization: Bearer, which is for the session token on account APIs and otherwise reserved for the gateway layer." }, { "q": "Do I need to pay before my first paid fetch?", "a": "Not always. New accounts may receive a trial credit, shown as signupBonus in the register response, for example USD 0.10. If it is absent, top up through Stripe with a minimum of 0.50 USD." }, { "q": "How do I know what a fetch cost?", "a": "A paid 200 may include X-AISA-Charged and X-AISA-Charge-Amount headers. A 402 body shows the price in rule.price, for example 0.02. Confirm your remaining funds by calling GET /api/agent/balance." } ], "sources": [ { "title": "Header name: X-Crawler-Token", "url": "https://esa.aisa.one/cdn/open-verify.html", "domain": "esa.aisa.one" }, { "title": "AISA onboarding: HTTP 402 Payment Required", "url": "https://esa.aisa.one/agent/guide.html", "domain": "esa.aisa.one" }, { "title": "AISA Agent Integration Guide — API reference", "url": "https://esa.aisa.one/agent/docs.html", "domain": "esa.aisa.one" }, { "title": "Live capture: anonymous request to a paid AgentFlare file", "url": "https://agentflare.org/markets/index.data.json", "domain": "agentflare.org" }, { "title": "AISA pay-per-crawl platform overview (llms.txt)", "url": "https://esa.aisa.one/llms.txt", "domain": "esa.aisa.one" } ] }, "analysis_md": "An HTTP 402 means the content is for sale, not blocked. Register for a crawler token, check your balance, retry the original URL with X-Crawler-Token, then read the 200, 402 or 403 result.\n\nThis guide is for developers building agents or crawlers that run into HTTP 402 Payment Required on a publisher site. You will walk through the full first paid fetch: recognising what a 402 means, registering an account, saving the crawler token, checking and funding your balance, retrying the original URL with the right header, and interpreting the 200, 402 or 403 that comes back. Every command uses the AISA account endpoints and the single crawl header, so you can paste the snippets into a script and complete one paid request end to end.\n\n## Before you start\n\n- A paid publisher URL that returned HTTP 402 to your crawler\n- curl, or Python with the requests library\n- An email address and a password of at least 8 characters\n\n## Step by step\n\n### 1. Recognise what the 402 means\n\nA 402 from a publisher edge is a standard payment signal, not a block. The edge checked the URL's access rules and found no valid crawler token, or a token with too little balance, so it answered 402 instead of the content. The body is JSON: read error first, then follow the guidance block and the onboarding link. Denials never charge you.\n\n```text\nHTTP/1.1 402 Payment Required\nContent-Type: application/json\nX-Tengine-Error: payment required by http_crawler\n\nerror: invalid_token\nonboarding: https://esa.aisa.one/agent/guide.html\n```\n\n### 2. Register and save the crawler token\n\nRegister once with POST /api/agent/auth/register. The response holds crawler.crawlerToken (format aisa_crawler_ followed by 64 hex characters) and a session token in token. Keep both, and never print the full crawler token in logs. The header that carries it is X-Crawler-Token, which replaces the earlier name X-AISA-Crawler-Token. If signupBonus is present, you already have trial credit.\n\n```python\nimport requests\nBASE = 'https://cdn.aisa.one'\nr = requests.post(BASE + '/api/agent/auth/register', json={\n 'email': 'mybot@example.com',\n 'password': 'AtLeast8Chars',\n 'name': 'My Crawler',\n 'crawlerType': 'custom'})\ndata = r.json()\ncrawlerToken = data['crawler']['crawlerToken']\nsessionToken = data['token']\nprint(data['crawler']['balance'], data.get('signupBonus'))\n```\n\n### 3. Check the balance and top up\n\nFetch the real balance with GET /api/agent/balance, using the same crawler token you crawl with. If you need funds, POST /api/agent/topup/create-session returns a checkout_url for Stripe, with a minimum of 0.50 USD. The top-up call takes the session token in Authorization: Bearer; a human must open the checkout_url and pay. Never send the crawler token in Authorization: Bearer.\n\n```python\nimport requests\nBASE = 'https://cdn.aisa.one'\nb = requests.get(BASE + '/api/agent/balance',\n headers={'X-Crawler-Token': 'YOUR_CRAWLER_TOKEN'})\nprint(b.json()['crawler']['balance'])\n\nt = requests.post(BASE + '/api/agent/topup/create-session',\n headers={'Authorization': 'Bearer SESSION_TOKEN'},\n json={'amount': 5.00})\nprint(t.json()['checkout_url'])\n```\n\n### 4. Retry the original URL with your token\n\nSend the same publisher URL that returned 402, this time with X-Crawler-Token and an honest User-Agent. Do not change the path or call a payment endpoint: the publisher edge verifies the token, deducts the price from your balance and serves the content in the same request. One request from you results in at most one charge.\n\n```bash\ncurl -i 'https://PUBLISHER-SITE/research/report.data.json' \\\n -H 'X-Crawler-Token: YOUR_CRAWLER_TOKEN' \\\n -A 'MyBot/1.0 (+https://example.com/bot)'\n```\n\n### 5. Read the outcome: 200, 402 or 403\n\nOn 200, the content is yours; a paid fetch may carry the receipt headers X-AISA-Charged and X-AISA-Charge-Amount. On 402, read error: invalid_token means fix or register the token, insufficient_balance means top up using crawler.shortfall, then retry. On 403 access_blocked, the publisher withholds the resource from crawlers, so paying will not unlock it; look at the site's /pricing.json instead.\n\n```python\nimport requests\nurl = 'https://PUBLISHER-SITE/research/report.data.json'\nr = requests.get(url, headers={\n 'X-Crawler-Token': 'YOUR_CRAWLER_TOKEN',\n 'User-Agent': 'MyBot/1.0 (+https://example.com/bot)'})\nif r.status_code == 200:\n print('charged:', r.headers.get('X-AISA-Charged'),\n r.headers.get('X-AISA-Charge-Amount'))\nelse:\n body = r.json()\n print(r.status_code, body.get('error'))\n print(body.get('guidance', {}).get('steps'))\n```\n\n## Crawl outcomes and what to do next\n\n| Response | Meaning | Your action |\n| --- | --- | --- |\n| 200 (free tier) | Open content such as HTML pages, .md summaries or discovery files | Use it; cost is $0 |\n| 200 (paid) | A priced resource your token just paid for | Read the receipt headers and report the amount charged |\n| 402 invalid_token | No, unknown or inactive crawler token | Register or fix the X-Crawler-Token header, then retry the same URL |\n| 402 insufficient_balance | Balance too low; crawler.shortfall gives the gap | Top up, then retry |\n| 402 price_required | Price discovery; the quote is in price and currency | Retry with crawlerAutoPrice set to true, which gateways usually do for you |\n| 402 price_mismatch / price_too_high | Your exact or maximum price did not match the quote | Adjust to the quoted price, or skip |\n| 403 access_blocked | The publisher withholds this from crawlers; payment does not unlock it | Do not retry or top up; check /pricing.json for purchasable directories |\n\n## Common pitfalls\n\n- Sending the crawler token in Authorization: Bearer makes the request count as unauthenticated and returns 402 invalid_token. Use X-Crawler-Token only.\n- The session token from register or login is for account APIs such as top-up only. The crawler token is what you send to publisher sites.\n- Do not quote a remembered balance. Call GET /api/agent/balance before and after a top-up, since every paid fetch changes it.\n- Fetching a priced resource twice charges twice. A 403 access_blocked is a publisher policy, so retrying or topping up will not change it.\n\n## Key takeaways\n\n- A 402 is an invitation to pay: register, hold a balance, and retry the same URL with X-Crawler-Token.\n- Denials never charge; only a granted fetch of a priced resource deducts its per-request price.\n- Treat 403 access_blocked differently from 402: payment cannot unlock it, so pivot to the pay-mode directories in /pricing.json.\n\n## FAQ\n\n**What does an HTTP 402 response mean for my crawler?**\nThe resource exists but requires a valid crawler token and sufficient balance. It is a standard payment signal, not an error or a block. Read the error field in the JSON body and follow the guidance steps.\n\n**Which header carries the crawler token?**\nX-Crawler-Token. It replaces the earlier X-AISA-Crawler-Token. Never put the token in Authorization: Bearer, which is for the session token on account APIs and otherwise reserved for the gateway layer.\n\n**Do I need to pay before my first paid fetch?**\nNot always. New accounts may receive a trial credit, shown as signupBonus in the register response, for example USD 0.10. If it is absent, top up through Stripe with a minimum of 0.50 USD.\n\n**How do I know what a fetch cost?**\nA paid 200 may include X-AISA-Charged and X-AISA-Charge-Amount headers. A 402 body shows the price in rule.price, for example 0.02. Confirm your remaining funds by calling GET /api/agent/balance.\n", "sources": [ { "title": "Header name: X-Crawler-Token", "url": "https://esa.aisa.one/cdn/open-verify.html" }, { "title": "AISA onboarding: HTTP 402 Payment Required", "url": "https://esa.aisa.one/agent/guide.html" }, { "title": "AISA Agent Integration Guide — API reference", "url": "https://esa.aisa.one/agent/docs.html" }, { "title": "Live capture: anonymous request to a paid AgentFlare file", "url": "https://agentflare.org/markets/index.data.json" }, { "title": "AISA pay-per-crawl platform overview (llms.txt)", "url": "https://esa.aisa.one/llms.txt" } ], "related": [ { "name": "Crawler Tokens & Auto-Pay: A How-To", "url": "https://agentflare.org/guides/crawler-tokens-auto-pay-a-how-to.html" }, { "name": "Handling 402 and 403 Responses: An Error Reference for Agents", "url": "https://agentflare.org/guides/handling-402-and-403-responses-an-error-reference-for-agents.html" }, { "name": "Budgeting a Paid Crawl: Discover, Survey, Then Buy", "url": "https://agentflare.org/guides/budgeting-a-paid-crawl-discover-survey-then-buy.html" }, { "name": "Building a Pay-Per-Crawl Client for Your Agent", "url": "https://agentflare.org/guides/building-a-pay-per-crawl-client-for-your-agent.html" }, { "name": "Pricing Content by Directory: Best Practices", "url": "https://agentflare.org/guides/pricing-content-by-directory-best-practices.html" }, { "name": "robots.txt, llms.txt & Sitemaps for AI Crawlers", "url": "https://agentflare.org/guides/robotstxt-llmstxt-sitemaps-for-ai-crawlers.html" } ], "pricing": { "price_usd": 0, "method": "402", "endpoint": "https://esa.aisa.one/api/v1/access/verify", "autopay_hint": "set crawlerAutoPrice=true with the X-Crawler-Token header", "onboarding": "https://esa.aisa.one/agent/guide.html" }, "powered_by": "AISA — agent-native search, settlement & delivery (https://aisa.one)" }