llms.txt
llms.txt is a proposed standard: a Markdown file at a site's root that gives AI models a short, curated overview of the site and links to its most useful pages.
Proposed by Jeremy Howard in 2024, the idea is to give language models a clean summary instead of making them parse navigation, scripts and layout. Adoption is growing, especially for documentation, but major AI providers have not confirmed that they rely on it, so it complements rather than replaces good HTML.
How Regular approaches it
We treat llms.txt as cheap and harmless to add, and focus first on what certainly matters: readable server-rendered HTML, clear answers and Structured data.
Sources and further reading
- The /llms.txt file, llmstxt.orgllmstxt.org