{ "@context": "https://agentflare.org/schema", "type": "Article", "tier": "L2-full", "title": "Reading .data.json: A Field Guide for Agents", "description": "Every AgentFlare .data.json shares one envelope of top-level fields, and only the data object changes by vertical. Fetch it with an X-Crawler-Token header;…", "canonical": "https://agentflare.org/guides/reading-datajson-a-field-guide-for-agents.html", "category": "guides", "updated": "2026-10-19", "generated_at": "2026-10-19T00:37:02.485Z", "facts": [ { "label": "Track", "value": "Using AgentFlare data" }, { "label": "Level", "value": "Beginner" }, { "label": "Access", "value": "Free / open" }, { "label": "Updated", "value": "2026-10-19" } ], "data": { "topic": "Reading .data.json: A Field Guide for Agents", "track": "data", "access": "free", "generated_at": "2026-10-19T00:37:02.485Z", "difficulty": "Beginner", "tldr": "Every AgentFlare .data.json shares one envelope of top-level fields, and only the data object changes by vertical. Fetch it with an X-Crawler-Token header; an anonymous request gets HTTP 402.", "intro": "This guide is for developers and agent builders who want to pull structured data from AgentFlare instead of scraping HTML. You will learn which files are paid, how to send your crawler token, how to recognise the 402 response, and how to read the shared envelope that wraps every file. You will also see how the data block differs between Markets index files, Markets ticker files and Newswire digests, so your parser can branch on the vertical.", "prerequisites": [ "A registered crawler token from the AISA onboarding guide", "curl or Python with the requests library installed", "Basic familiarity with JSON and HTTP status codes" ], "steps": [ { "title": "Identify which files are paid", "body": "AgentFlare splits content into two tiers. L1_free covers index pages, .html articles, .md teasers, llms.txt, sitemap.xml and robots.txt. L2_paid covers every *.data.json and *.full.md file. Rules in pricing.json meter Markets files at 0.01 USD per request and Newswire files at 0.005 USD per request. Guides data files are free, and Members files are blocked to crawlers. Read the price map before you plan a crawl.", "code": { "lang": "bash", "text": "curl -s https://agentflare.org/pricing.json" }, "sources": [ "S4" ] }, { "title": "Fetch a file with your crawler token", "body": "Send your token in the X-Crawler-Token header, which is the Open Verify name and replaces the earlier X-AISA-Crawler-Token still shown on older AISA pages. Do not put the token in Authorization: Bearer, because that header is reserved for the gateway layer. The example fetches the Markets index; the same header works for dated files such as YYYY-MM-DD.data.json and per-item slug files.", "code": { "lang": "bash", "text": "curl -s \\\n -H 'X-Crawler-Token: YOUR_CRAWLER_TOKEN' \\\n https://agentflare.org/markets/index.data.json" }, "sources": [ "S1", "S5" ] }, { "title": "Recognise the 402 response", "body": "A request to a paid file without a recognised token returns HTTP 402 Payment Required with a JSON body. In the live capture that body carried the error code invalid_token, a message, and an onboarding URL pointing to https://esa.aisa.one/agent/guide.html. Check the status code before parsing the envelope, and follow the onboarding link if your token is not registered.", "code": { "lang": "python", "text": "import requests\n\nurl = 'https://agentflare.org/markets/index.data.json'\nr = requests.get(url)\nif r.status_code == 402:\n err = r.json()\n print(err['error'])\n print(err['message'])\n print(err['onboarding'])" }, "sources": [ "S5" ] }, { "title": "Read the shared envelope", "body": "Every file listed in the field references starts with the same top-level fields: @context, type, tier, title, description, canonical, category, updated, generated_at, facts, data, analysis_md, sources, related, pricing and powered_by. Each facts entry has a label and a value, both strings. The pricing object carries price_usd, method, endpoint, autopay_hint and onboarding. Parse the envelope first, then branch on category.", "code": { "lang": "python", "text": "import requests\n\nurl = 'https://agentflare.org/markets/index.data.json'\nh = {'X-Crawler-Token': 'YOUR_CRAWLER_TOKEN'}\ndoc = requests.get(url, headers=h).json()\nprint(doc['title'], doc['updated'], doc['generated_at'])\nfor f in doc['facts']:\n print(f['label'], f['value'])\nprint(doc['pricing']['price_usd'])\nprint(doc['analysis_md'][:200])" }, "sources": [ "S2" ] }, { "title": "Parse the vertical-specific data block", "body": "Only the data object changes. Markets index and dated files hold count, advancers, decliners, sectors and a stocks array with ticker, price_usd, day_change_pct, return_6m_pct, market_cap_usd, pe and sector. A Markets ticker file such as xom.data.json instead holds price_usd, returns_pct (m1, m3, m6), metrics and company. Newswire files hold date, headline, stories, market and items.", "code": { "lang": "python", "text": "import requests\n\nh = {'X-Crawler-Token': 'YOUR_CRAWLER_TOKEN'}\nbase = 'https://agentflare.org/markets/'\n\nidx = requests.get(base + 'index.data.json', headers=h).json()\nfor s in idx['data']['stocks']:\n print(s['ticker'], s['price_usd'], s['day_change_pct'], s['sector'])\n\nxom = requests.get(base + 'xom.data.json', headers=h).json()\nprint(xom['data']['returns_pct']['m6'])\nprint(xom['data']['metrics']['price_to_earnings_ratio'])\nprint(xom['data']['company']['industry'])" }, "sources": [ "S2", "S3" ] } ], "reference": { "title": "Envelope and data block fields", "columns": [ "Field", "Type", "Notes" ], "rows": [ [ "@context, type, tier, title, description, canonical, category, updated, generated_at", "string", "Top-level on every file" ], [ "facts[].label, facts[].value", "string", "Array of objects in the envelope" ], [ "analysis_md", "string", "Top-level analysis text" ], [ "sources[].title, sources[].url", "string", "Array of objects in the envelope" ], [ "pricing.price_usd", "number", "Other pricing fields are strings" ], [ "data.stocks[].ticker, price_usd, day_change_pct", "string, number, number", "Markets index and dated files" ], [ "data.returns_pct.m1, m3, m6", "number", "Markets per-item files such as xom.data.json" ], [ "data.headline, data.stories[], data.items[]", "string, array, array", "Newswire digest files" ] ], "sources": [ "S2", "S3" ] }, "pitfalls": [ { "text": "Sending the token as Authorization: Bearer fails by design; that header is reserved for the gateway layer. Use X-Crawler-Token, not the older X-AISA-Crawler-Token.", "source": "S1" }, { "text": "Do not reuse one data parser across verticals. Markets index files use data.stocks, ticker files use data.metrics, and Newswire files use data.stories.", "source": "S2" }, { "text": "Newswire market entries differ by file: the AI digest lists d1 and d30, while the crypto digest lists d1 and d7 plus symbol and name.", "source": "S3" }, { "text": "Each paid request is metered per crawl, so an anonymous or failed call is not a free probe. Check for HTTP 402 and error invalid_token before retrying in a loop.", "source": "S5" } ], "takeaways": [ "Every .data.json shares one envelope; read title, updated, facts, analysis_md, sources and pricing before touching data.", "Send X-Crawler-Token on every request to a paid file; never use Authorization: Bearer.", "Branch on the vertical: stocks for Markets index, metrics for ticker files, stories for Newswire digests." ], "faq": [ { "q": "What header do I use to authenticate to AgentFlare .data.json files?", "a": "Use X-Crawler-Token with your crawler token as the value. It replaces the earlier X-AISA-Crawler-Token name. Do not send the token in Authorization: Bearer, which is reserved for the gateway layer." }, { "q": "Why does my request return HTTP 402 Payment Required?", "a": "Paid *.data.json files require a recognised crawler token. The live response for an anonymous request had error invalid_token and an onboarding URL, https://esa.aisa.one/agent/guide.html, where you can register." }, { "q": "How much does a .data.json request cost?", "a": "Per pricing.json, Markets files cost 0.01 USD per request and Newswire files cost 0.005 USD per request. Crypto and Predictions are 0.01, Social is 0.005, and Research and Scholar are 0.02. Guides data files are free." }, { "q": "What file names are available under /markets/?", "a": "There is index.data.json, one dated file per day at https://agentflare.org/markets/YYYY-MM-DD.data.json, and one file per item at https://agentflare.org/markets/.data.json, for example aapl.data.json or xom.data.json." } ], "sources": [ { "title": "Header name: X-Crawler-Token", "url": "https://esa.aisa.one/cdn/open-verify.html", "domain": "esa.aisa.one" }, { "title": "AgentFlare Markets .data.json field reference", "url": "https://agentflare.org/markets/index.html", "domain": "agentflare.org" }, { "title": "AgentFlare Newswire .data.json field reference", "url": "https://agentflare.org/news/index.html", "domain": "agentflare.org" }, { "title": "AgentFlare pricing.json (directory price map)", "url": "https://agentflare.org/pricing.json", "domain": "agentflare.org" }, { "title": "Live capture: anonymous request to a paid AgentFlare file", "url": "https://agentflare.org/markets/index.data.json", "domain": "agentflare.org" } ] }, "analysis_md": "Every AgentFlare .data.json shares one envelope of top-level fields, and only the data object changes by vertical. Fetch it with an X-Crawler-Token header; an anonymous request gets HTTP 402.\n\nThis guide is for developers and agent builders who want to pull structured data from AgentFlare instead of scraping HTML. You will learn which files are paid, how to send your crawler token, how to recognise the 402 response, and how to read the shared envelope that wraps every file. You will also see how the data block differs between Markets index files, Markets ticker files and Newswire digests, so your parser can branch on the vertical.\n\n## Before you start\n\n- A registered crawler token from the AISA onboarding guide\n- curl or Python with the requests library installed\n- Basic familiarity with JSON and HTTP status codes\n\n## Step by step\n\n### 1. Identify which files are paid\n\nAgentFlare splits content into two tiers. L1_free covers index pages, .html articles, .md teasers, llms.txt, sitemap.xml and robots.txt. L2_paid covers every *.data.json and *.full.md file. Rules in pricing.json meter Markets files at 0.01 USD per request and Newswire files at 0.005 USD per request. Guides data files are free, and Members files are blocked to crawlers. Read the price map before you plan a crawl.\n\n```bash\ncurl -s https://agentflare.org/pricing.json\n```\n\n### 2. Fetch a file with your crawler token\n\nSend your token in the X-Crawler-Token header, which is the Open Verify name and replaces the earlier X-AISA-Crawler-Token still shown on older AISA pages. Do not put the token in Authorization: Bearer, because that header is reserved for the gateway layer. The example fetches the Markets index; the same header works for dated files such as YYYY-MM-DD.data.json and per-item slug files.\n\n```bash\ncurl -s \\\n -H 'X-Crawler-Token: YOUR_CRAWLER_TOKEN' \\\n https://agentflare.org/markets/index.data.json\n```\n\n### 3. Recognise the 402 response\n\nA request to a paid file without a recognised token returns HTTP 402 Payment Required with a JSON body. In the live capture that body carried the error code invalid_token, a message, and an onboarding URL pointing to https://esa.aisa.one/agent/guide.html. Check the status code before parsing the envelope, and follow the onboarding link if your token is not registered.\n\n```python\nimport requests\n\nurl = 'https://agentflare.org/markets/index.data.json'\nr = requests.get(url)\nif r.status_code == 402:\n err = r.json()\n print(err['error'])\n print(err['message'])\n print(err['onboarding'])\n```\n\n### 4. Read the shared envelope\n\nEvery file listed in the field references starts with the same top-level fields: @context, type, tier, title, description, canonical, category, updated, generated_at, facts, data, analysis_md, sources, related, pricing and powered_by. Each facts entry has a label and a value, both strings. The pricing object carries price_usd, method, endpoint, autopay_hint and onboarding. Parse the envelope first, then branch on category.\n\n```python\nimport requests\n\nurl = 'https://agentflare.org/markets/index.data.json'\nh = {'X-Crawler-Token': 'YOUR_CRAWLER_TOKEN'}\ndoc = requests.get(url, headers=h).json()\nprint(doc['title'], doc['updated'], doc['generated_at'])\nfor f in doc['facts']:\n print(f['label'], f['value'])\nprint(doc['pricing']['price_usd'])\nprint(doc['analysis_md'][:200])\n```\n\n### 5. Parse the vertical-specific data block\n\nOnly the data object changes. Markets index and dated files hold count, advancers, decliners, sectors and a stocks array with ticker, price_usd, day_change_pct, return_6m_pct, market_cap_usd, pe and sector. A Markets ticker file such as xom.data.json instead holds price_usd, returns_pct (m1, m3, m6), metrics and company. Newswire files hold date, headline, stories, market and items.\n\n```python\nimport requests\n\nh = {'X-Crawler-Token': 'YOUR_CRAWLER_TOKEN'}\nbase = 'https://agentflare.org/markets/'\n\nidx = requests.get(base + 'index.data.json', headers=h).json()\nfor s in idx['data']['stocks']:\n print(s['ticker'], s['price_usd'], s['day_change_pct'], s['sector'])\n\nxom = requests.get(base + 'xom.data.json', headers=h).json()\nprint(xom['data']['returns_pct']['m6'])\nprint(xom['data']['metrics']['price_to_earnings_ratio'])\nprint(xom['data']['company']['industry'])\n```\n\n## Envelope and data block fields\n\n| Field | Type | Notes |\n| --- | --- | --- |\n| @context, type, tier, title, description, canonical, category, updated, generated_at | string | Top-level on every file |\n| facts[].label, facts[].value | string | Array of objects in the envelope |\n| analysis_md | string | Top-level analysis text |\n| sources[].title, sources[].url | string | Array of objects in the envelope |\n| pricing.price_usd | number | Other pricing fields are strings |\n| data.stocks[].ticker, price_usd, day_change_pct | string, number, number | Markets index and dated files |\n| data.returns_pct.m1, m3, m6 | number | Markets per-item files such as xom.data.json |\n| data.headline, data.stories[], data.items[] | string, array, array | Newswire digest files |\n\n## Common pitfalls\n\n- Sending the token as Authorization: Bearer fails by design; that header is reserved for the gateway layer. Use X-Crawler-Token, not the older X-AISA-Crawler-Token.\n- Do not reuse one data parser across verticals. Markets index files use data.stocks, ticker files use data.metrics, and Newswire files use data.stories.\n- Newswire market entries differ by file: the AI digest lists d1 and d30, while the crypto digest lists d1 and d7 plus symbol and name.\n- Each paid request is metered per crawl, so an anonymous or failed call is not a free probe. Check for HTTP 402 and error invalid_token before retrying in a loop.\n\n## Key takeaways\n\n- Every .data.json shares one envelope; read title, updated, facts, analysis_md, sources and pricing before touching data.\n- Send X-Crawler-Token on every request to a paid file; never use Authorization: Bearer.\n- Branch on the vertical: stocks for Markets index, metrics for ticker files, stories for Newswire digests.\n\n## FAQ\n\n**What header do I use to authenticate to AgentFlare .data.json files?**\nUse X-Crawler-Token with your crawler token as the value. It replaces the earlier X-AISA-Crawler-Token name. Do not send the token in Authorization: Bearer, which is reserved for the gateway layer.\n\n**Why does my request return HTTP 402 Payment Required?**\nPaid *.data.json files require a recognised crawler token. The live response for an anonymous request had error invalid_token and an onboarding URL, https://esa.aisa.one/agent/guide.html, where you can register.\n\n**How much does a .data.json request cost?**\nPer pricing.json, Markets files cost 0.01 USD per request and Newswire files cost 0.005 USD per request. Crypto and Predictions are 0.01, Social is 0.005, and Research and Scholar are 0.02. Guides data files are free.\n\n**What file names are available under /markets/?**\nThere is index.data.json, one dated file per day at https://agentflare.org/markets/YYYY-MM-DD.data.json, and one file per item at https://agentflare.org/markets/.data.json, for example aapl.data.json or xom.data.json.\n", "sources": [ { "title": "Header name: X-Crawler-Token", "url": "https://esa.aisa.one/cdn/open-verify.html" }, { "title": "AgentFlare Markets .data.json field reference", "url": "https://agentflare.org/markets/index.html" }, { "title": "AgentFlare Newswire .data.json field reference", "url": "https://agentflare.org/news/index.html" }, { "title": "AgentFlare pricing.json (directory price map)", "url": "https://agentflare.org/pricing.json" }, { "title": "Live capture: anonymous request to a paid AgentFlare file", "url": "https://agentflare.org/markets/index.data.json" } ], "related": [ { "name": "Getting Started with HTTP 402 Pay-Per-Crawl", "url": "https://agentflare.org/guides/getting-started-with-http-402-pay-per-crawl.html" }, { "name": "Crawler Tokens & Auto-Pay: A How-To", "url": "https://agentflare.org/guides/crawler-tokens-auto-pay-a-how-to.html" }, { "name": "Handling 402 and 403 Responses: An Error Reference for Agents", "url": "https://agentflare.org/guides/handling-402-and-403-responses-an-error-reference-for-agents.html" }, { "name": "Budgeting a Paid Crawl: Discover, Survey, Then Buy", "url": "https://agentflare.org/guides/budgeting-a-paid-crawl-discover-survey-then-buy.html" }, { "name": "Building a Pay-Per-Crawl Client for Your Agent", "url": "https://agentflare.org/guides/building-a-pay-per-crawl-client-for-your-agent.html" }, { "name": "Pricing Content by Directory: Best Practices", "url": "https://agentflare.org/guides/pricing-content-by-directory-best-practices.html" } ], "pricing": { "price_usd": 0, "method": "402", "endpoint": "https://esa.aisa.one/api/v1/access/verify", "autopay_hint": "set crawlerAutoPrice=true with the X-Crawler-Token header", "onboarding": "https://esa.aisa.one/agent/guide.html" }, "powered_by": "AISA — agent-native search, settlement & delivery (https://aisa.one)" }