🤖 AI & Agents · 9 min read
llms.txt Explained: What to Put in the File
llms.txt is a Markdown file served at your site root that tells an AI assistant which pages of yours are worth opening: an H1 with your site name, a one-line summary, then links with short notes. You write it by hand, it takes about ten minutes, and it is a proposal rather than an official standard. That is the whole answer to the question people arrive with, usually phrased as “what is llms.txt” or “llms txt example”. The rest of this page is the format, real files measured by size, and what happens when you keep three of them accurate over time.
What is llms.txt?
A web page is built for a person. It wraps the answer in navigation, banners and JavaScript, and a model that has to convert all of that back into text spends tokens on things that do not matter. The llms.txt proposal asks owners to write the important part down once, in Markdown, at a predictable URL. An agent fetches that file, sees a short list of destinations, and opens only the pages it needs.
The proposal was written by Jeremy Howard and published at llmstxt.org. The version on the site today is v2, dated Modified August 10, 2026, and the changes page lists what moved since the first draft. The parts relevant to anyone writing a file are unchanged: most of it is just Markdown with one structural rule about headings.
What goes inside the file?
The spec is short, and the order matters. In sequence, a file contains an optional byte order mark, an H1 with the project or site name, a blockquote with a short summary, then free-form Markdown sections such as paragraphs or lists, then any number of sections introduced by H2 headings. The H1 is the only required section. A file with a title, a summary and two links is a valid file.
Each H2 section holds what the proposal calls a file list: a Markdown list where every entry is a link, optionally followed by a colon and a note explaining the destination. One convention is worth copying. A section named Optional is reserved for secondary links that an agent may skip when its context is tight, which is a way of saying “here is the rest, if you have room”.
# Your Site Name
> One or two sentences on what this site is and who it is for.
## Docs
- [Getting started](https://example.com/docs/start): Install and first run.
- [Pricing](https://example.com/pricing): Plans, limits and what is free.
## Optional
- [Changelog](https://example.com/changelog)Two rules from the spec that people miss. The file can live at a subpath and then covers only the pages beneath it, so /docs/llms.txt describes the documentation rather than the whole domain, and when several files apply an agent should use the most specific one. And the filename is fixed: it is llms.txt, at the root of whatever it covers.
What do real llms.txt files look like?
The proposal's own page says the AI labs publish files for their developer documentation, so we fetched four of them on 18 September 2026 and measured the size. All four returned HTTP 200:
| File | Size | What it covers |
|---|---|---|
| docs.anthropic.com/llms.txt | 68,036 bytes | API documentation for the whole developer site |
| platform.openai.com/docs/llms.txt | 41,794 bytes | Guides and conceptual documentation under /docs |
| cloud.google.com/llms.txt | 41,007 bytes | Product documentation across the cloud platform |
| ai.google.dev/gemini-api/docs/llms.txt | 33,708 bytes | The Gemini API docs, under the path it describes |
One detail from that table is more instructive than the sizes. The URL at the domain root, ai.google.dev/llms.txt, returned 404 the same day, while the file under /gemini-api/docs/ returned the 33 KB document. That is the subpath rule in production: a large organisation does not publish one file for everything, it publishes one per section and lets the most specific file win.
What we measured across three of our own sites
Nobody can tell you what length to aim for from a spec. So we counted our own files, which sit on three very different sites.
| Site | Words | Links | Sections |
|---|---|---|---|
| nocodecsv.com | 1,203 | 49 | Docs, Blog, For AI agents, Key Facts, Contact |
| buildex-1dm.pages.dev | 382 | 2 | What we supply, Key product data, Documentation, For AI agents, Contact |
| carecosttool.pages.dev | 393 | 6 | Tools, Data source, For AI agents, Contact |
The link count is what drove the length, not the other way round. The first site is a product with dozens of documented pages, so its file lists 49 of them. The other two are small business sites where only a handful of pages carry information an agent would need, and their files are under 400 words each. Padding either one out to 1,000 words would mean inventing destinations.
Two findings from the same afternoon are less flattering, and worth passing on. First, a static count inside a file goes stale. Ours states a number of published guides that no longer matches the blog, which now runs to 61 article directories against a sitemap of 75 URLs. Avoid hard-coded totals; describe the categories instead, or plan to update them. Second, v2 also proposes machine-readable link relations for Markdown copies of pages, and we checked our own headers: no Link header is being sent on the home page or on a tool page. Publishing the file and declaring the alternate versions are separate jobs, and we have only done the first.
Is llms.txt an official standard?
No, and saying so plainly is more useful than hedging. There is no RFC, no W3C specification and no IETF draft behind the filename. What exists is a proposal with real adoption behind it. The v2 page states that thousands of sites publish a file, that documentation platforms generate one automatically, that Chrome's Lighthouse audits sites for one as part of its agentic browsing checks, and that OpenAI, Anthropic and Gemini publish files for their own developer documentation. Those are adoption claims made by the proposal itself, and the four files we fetched are consistent with them.
The practical consequence: publish it because it is cheap, honest and reusable, not because a vendor has promised to read it. No major assistant vendor has committed publicly to consuming llms.txt, which is exactly the same status that makes it low risk. A file that helps an assistant costs you a few hundred words; a file that does not costs you nothing but the time.
How is it different from robots.txt and sitemap.xml?
A site usually ends up with all three, and they do not overlap. The proposal puts the distinction this way: robots.txt tells automated tools what access is acceptable, while llms.txt is used on demand when an agent needs information about a topic.
| File | Audience | Content | When it is read |
|---|---|---|---|
| robots.txt | Crawlers | Allow and disallow rules per user agent | Before crawling, on a schedule |
| sitemap.xml | Search engines | Every URL you want indexed | On the crawler's schedule |
| llms.txt | Assistants and agents | A curated subset, with notes | During a task, when a model needs detail |
That difference explains the sizing too. Our sitemap carries 75 URLs because a sitemap should be complete. The same site's llms.txt carries 49 links because a reading list should be selective. If the two files list the same thing, the llms.txt one is not doing its job.
How do you write one in ten minutes?
- List the pages that answer something. Open your sitemap, then cross out everything that exists for navigation, legal boilerplate or marketing copy. What remains is your reading list.
- Write the title and the summary line. One H1, one blockquote. Say what the site is and who it is for, in a sentence or two. If you cannot summarise the site in two sentences, the file is not the problem.
- Group the links under H2 headings. Docs, pricing, guides, whatever matches your structure. Add a short note after each link when the title alone is not clear, in the form
[name](url): note. - Serve it and check it. A plain text response at
/llms.txtis enough. Fetch it in a browser to confirm it is the file you think it is, and put the matching entry in sitemap.xml so it is discoverable.
One more file is worth writing while you are in there. An llms.txt describes what you have; a tool declaration such as agent-tools.json describes what your site can do, with selectors and expected output, and the browser-side version of that idea is what WebMCP standardises. If an assistant can read both, it can describe your site and use it.
Frequently asked questions
What is llms.txt?
llms.txt is a Markdown file you serve at /llms.txt, or at any subpath you want it to cover, that gives a language model a short curated map of your site: a title, a one or two sentence summary, and links to the pages worth reading. The proposal comes from llmstxt.org and describes it as a complement to sitemap.xml rather than a replacement: the sitemap lists everything for crawlers, while llms.txt picks the destinations an assistant should actually open.
What goes in an llms.txt file?
In the order the proposal gives: an optional byte order mark, then an H1 with the project or site name, which is the only required part, then a blockquote summary, then any free-form Markdown paragraphs, then sections introduced by H2 headings where each section is a list of markdown links written as [name](url) followed optionally by a colon and a note. A section titled Optional is used by convention for links an agent can skip when context is short.
Is llms.txt an official standard?
No. It is a community proposal published by Jeremy Howard at llmstxt.org, first written in 2024 and revised in a v2 that the site dates to August 2026. There is no W3C, IETF or RFC backing for the filename or the format. The strongest signals are adoption ones: the v2 page states that thousands of sites publish a file, that documentation platforms generate one automatically, that Chrome's Lighthouse audits for one during agentic browsing checks, and that OpenAI, Anthropic and Gemini publish files for their own developer documentation.
How is llms.txt different from robots.txt and sitemap.xml?
Different jobs, and they can coexist without changes to each other. robots.txt states which automated access a site considers acceptable, which is a crawling rule. sitemap.xml lists every URL you want indexed, which is a discovery feed. llms.txt is a curated reading list fetched on demand when an agent needs information about a topic. A file also covers only the path it sits under, so /docs/llms.txt describes the pages beneath /docs rather than the whole domain.
Where should the llms.txt file live?
At the site root as /llms.txt for a whole site, or at the root of the section it describes. The proposal explicitly allows subpaths and says that when more than one file applies, an agent should use the most specific one. It also explains why the file is not placed under the well-known prefix reserved by RFC 8615: those URIs exist only at an origin root, and an author who controls just one directory on a shared host could never publish there.
How long should llms.txt be?
Long enough to list the destinations that matter and short enough to keep in a context window. Our three sites came in at 382, 393 and 1,203 words, and the length followed the number of real destinations rather than any word target: the smallest file has two links because the site has two pages worth an agent's attention. If your lines are filler, delete them; every saved token is the point of the exercise.
Does llms.txt improve SEO rankings?
Nobody has published evidence that it moves rankings, and it is not a ranking factor you can point at in a search console. Its purpose is to help an assistant answer questions about your site accurately, which is a different channel from the blue links. Publish it because it is cheap and reusable, keep it accurate, and do not treat it as a substitute for pages that answer a real question.
Should I also publish Markdown copies of my pages?
That is the second half of the v2 proposal, and it is optional. It suggests serving a clean Markdown version of a page at the same URL with .md appended or substituted, and pointing at both files with link relations: rel=alternate with type text/markdown for the Markdown copy, and rel=describedby for the llms.txt that covers the page. The same relations can be sent as HTTP Link response headers, which works for non-HTML files too. We measured our own site and found no Link headers yet, so this part of the proposal is still ahead of us.
Tools mentioned in this guide
Writing the file is manual, but keeping it current as a site grows is not, which is where these earn their place:
- OpenCode Go — a file that lists 49 links goes stale the moment you publish, and a small script that reads your sitemap and re-generates the list is faster to write with an assistant than to fix by hand each month. Try OpenCode Go
- Stack AI — when the same summary has to feed several channels, a workflow can generate the file from your own content instead of you retyping descriptions in three places. Try Stack AI
- Softr — if your site is a directory or a catalogue, publishing the collection as a no-code app gives you pages with stable URLs, which is exactly what a curated link list needs to point at. Try Softr
Some links above are affiliate links — if you buy through them we may earn a commission at no extra cost to you. OpenCode Go uses our referral link; the other two currently point to each vendor's official page until our tracking links are approved.
Read a Real llms.txt, Then Get Yours Published
The file we describe in the measurements above is live and public. If you would rather have someone write and verify yours, that is the work I do.
I build this layer for other sites: llms.txt, agent-tools.json and WebMCP declarations, verified against the live pages. Details at /agent-ready.
Related reading
AI & Analysis — other guides that pair well with this one.
- Free ChatGPT Code Interpreter Alternative
- AI for CSV Files
- Convert CSV to Excel Without Excel
- Convert Excel to CSV Free Online
Browse all guides in the NoCodeCSV blog.