llms.txt: The Standard for AI-Readable Sites

llms.txt is a proposed standard from Australian technologist Jeremy Howard: a Markdown file at a site's root that points large language models to the most useful content on that site. It targets two crawler problems: context windows too small for entire sites, and HTML cluttered with navigation, ads and JavaScript that is hard to convert into clean text. It complements robots.txt, sitemaps and schema markup rather than replacing them. Adoption is niche, no major AI provider has officially confirmed support, and compliance is voluntary with no enforcement mechanism. This brief covers the file format, deployment steps, the comparison with robots.txt, and the limits developers should plan around.

Key concepts

llms.txt

A proposed guidance file, written in Markdown and placed at the site root, that gives LLMs a curated, structured map of a site's most important content and preferred formats. S3 · blog.openreplay.comS4 · oltre.ai

llms-full.txt

An optional companion file for comprehensive sites. llms.txt stays a concise overview, while llms-full.txt holds extensive documentation, code examples and deeper technical detail. S3 · blog.openreplay.comS5 · royalplugins.com

Guidance, not enforcement

llms.txt is not an enforcement mechanism like robots.txt. Each platform can honor it, partially honor it, or ignore it, and it guarantees nothing about crawling, retrieval or citation. S4 · oltre.aiS8 · university.webflow.com

Curation layer

llms.txt sits beside robots.txt, schema markup, canonical tags and XML sitemaps, signalling what matters most for AI consumption, such as /docs/, /pricing/ and /security/ entry points. S4 · oltre.ai

Agent Readiness

A category of publisher-declared AI policy surfaces that includes llms.txt, Content Signals in robots.txt, AGENTS.md, 404 recovery blocks, Markdown negotiation and MCP-era declarations. S5 · royalplugins.com

Publishing an llms.txt file

1. Write the file in MarkdownThe proposed standard calls for Markdown, the same lightweight format used in GitHub README files, which AI systems parse easily. Use valid syntax; extra structure such as subsections, tables or code snippets is acceptable.
2. Add a title and short descriptionStart with the company or site name as a top-level heading, followed by a brief description of what the site or company does.
3. Group links under headed sectionsCreate sections such as Products, Documentation and Resources. List each link with a short description, for example Getting Started: Quick setup guide, or Authentication: OAuth 2.0 flow.
4. Host it at the domain rootMake the file reachable at yourdomain.com/llms.txt, following the robots.txt convention. In Webflow, upload it as an asset or create a page with that path.
5. Optionally add llms-full.txtFor comprehensive sites, publish a second file with extensive documentation, code examples and deeper technical details, leaving llms.txt as the concise overview.

File format and structure in practice

The proposed standard specifies a Markdown file. A minimal version opens with the company name as an H1, followed by a short description, then H2 sections such as Products, Documentation and Resources, each containing a list of links with one-line descriptions. Typical files run 10-50 lines of Markdown. Larger sites can add H3 and H4 subsections, tables and code snippets. Published examples differ in depth. Hugging Face uses multiple heading levels with full code examples and many links, reading like a knowledge base. Vercel opens with title, description and tags lines, then uses headers, step-by-step instructions and code examples. Zapier uses few headings and mainly a long list of links with descriptions. For sites that need more detail, llms-full.txt carries extended documentation while llms.txt remains the concise overview. The aim is a clean, structured alternative to complex HTML that AI systems can read quickly.

Adoption and provider support

Adoption is small. NerdyData counted only 951 domains with an llms.txt file as of July 2025, and another source cites approximately 950 domains, including many influential tech companies. Users are mostly SaaS and developer-focused companies. The main limitation is provider support: OpenAI, Google and Anthropic have not confirmed that their crawlers use these files, and compliance varies by AI system with no enforcement mechanism. Search-impact claims are equally cautious. One source states llms.txt files do not change rankings. Another reports its own research advises treating the file as a policy surface, not a citation booster. Citation behavior also depends on other systems: 87% of SearchGPT citations match Bing's top organic results (Seer Interactive, 2025), so AI discoverability depends on Bing and Google indexing, platform retrieval rules and on-page clarity as well as any llms.txt file.

Using llms.txt alongside other controls

llms.txt works best as a curation layer on top of existing technical SEO controls rather than a substitute. robots.txt governs crawler access rules and acts as the gatekeeper, deciding who gets in. llms.txt is the briefing document, telling authorized visitors what matters. Schema markup such as FAQPage, Product and Organization improves machine understanding of entities and page meaning. Canonical tags signal preferred URLs, and XML sitemaps improve discovery and recrawl prioritization. In practice, teams use schema markup to improve understanding for Google AI Overviews and Gemini, and use llms.txt to surface best entry points such as /docs/, /pricing/ and /security/ for retrieval workflows like RAG, where an LLM fetches web content before answering. Other publisher-declared surfaces in the same Agent Readiness category include Content Signals in robots.txt, AGENTS.md, 404 recovery blocks, Markdown negotiation and MCP-era declarations. Because the file takes under 10 minutes to ship, it is described as a low-effort signal.

robots.txt versus llms.txt

Timeline

1. 2024
   Similarweb Q4 data shows ChatGPT receiving roughly 3.7 billion monthly visits.royalplugins.com
2. 2025
   Seer Interactive finds 87% of SearchGPT citations match Bing's top results.oltre.ai
3. 2025-07
   NerdyData counts 951 domains that have published an llms.txt file.semrush.com

Common pitfalls

• Do not use llms.txt as access control. It is guidance with no enforcement mechanism; platforms may honor, partially honor or ignore it. Use robots.txt for access rules.
• No major AI provider has officially confirmed support. OpenAI, Google and Anthropic have not stated that their crawlers use llms.txt files.
• Do not expect a citation or ranking boost. Treat the file as a policy surface; one source says it does not change rankings.
• Content loaded by JavaScript is often missed because most AI crawlers read only basic HTML, so key information should be reachable through plain linked pages.

Key takeaways

• llms.txt is a proposed, voluntary Markdown file at the site root that curates your most important content for AI systems.
• Provider support is unconfirmed and adoption is niche, so treat it as a low-effort signal, not a ranking or citation lever.
• Pair llms.txt with robots.txt, schema markup, canonical tags and sitemaps; it complements these standards rather than replacing them.

FAQ

How this page is made. Built from the full text of 8 sources found with the AISA Tavily API (advanced search), refreshed weekly. A language model organizes what those sources state into the blocks above; each block links the sources it draws on, and figures that do not appear in the sources are removed.