What is an llms.txt file? A real one, explained line by line

An llms.txt file is a plain markdown file at the root of a site, /llms.txt, that gives an AI assistant a short description of what the site is and which pages matter, in one fetch. It has been a proposal since September 2024, not a standard. No search engine ranks a page for having one.

This one is real. Not a template, ours: at birta.dev/llms.txt, shown line by line below, along with what is actually true about whether any of this works.

What GEO actually is, and why a file like this exists

Classic search ranks pages. A crawler indexes the web, a query matches a slice of it, and a reader picks from ten blue links and does the synthesizing themselves. A large language model works differently. It is the kind of system behind ChatGPT, Claude, and the "LLM" in llms.txt. It fetches material once, reads it, and hands back an answer or takes an action. No ranked list, no click.

That shift is what "GEO," generative engine optimization, actually names, and it is not a rebrand of SEO. A page competing for a ranking position wants to be found by a crawler and judged the best match among many. A page trying to be useful to an assistant wants something different: cheap to fetch, unambiguous to read in one pass. Closer to handing someone a one-page brief than to winning an auction.

llms.txt is one narrow answer to that second problem. A small, curated index at a fixed address tells an assistant what a site is and where the parts that matter live, without it having to parse a styled homepage to find out. It is not the only answer. And, as the data below shows, it is not a guaranteed one either.

It is also not the same job as MCP. This file is about being read and understood. MCP is the channel an assistant uses to actually do something once it is connected. One is a description. The other is a set of hands.

What the file actually is

llmstxt.org is where the file comes from: a proposal Jeremy Howard published on September 3, 2024, sitting next to robots.txt and sitemap.xml at a site's root. The address is a convention, not a registered standard.

The specification is short. An H1 with the site or project's name is the only required part. A blockquote underneath gives a one-line summary. After that, any number of free markdown sections carry context, and H2-headed sections hold curated lists of links in [name](url): note form: a table of contents for a model, not a full site map.

Markdown, not HTML, on purpose. A styled page arrives wrapped in navigation, scripts and layout an assistant has to strip away before it reaches the actual content. A short plain-text file is the content, fetched in one request, with nothing to parse around.

Here is ours, line by line

Here is the start of our real, live file. The first line is the H1, # Birta. Everything after it, copied exactly:

> Birta is hosting for landing pages built by an AI tool: the tool publishes the site, Birta keeps it online, and one panel holds the projects, versions, domains, visitor stats and the leads the sites collect.

Birta is for people who build a landing page by prompting the tool they
already use — a coding agent such as Cursor, Claude Code or Codex, or a plain
chat such as ChatGPT or Claude — instead of setting up hosting themselves.

The H1 and the one-line blockquote are the entire required part of the spec. Everything after them, we chose.

Next is ## Connect. Not a link list, but prose with two subheadings, split by whether the client can read files on the owner's machine. It ends with one line we hold to on purpose:

If you cannot set yourself up — no shell, no access to your own settings — do not stop and do not ask the owner to figure it out. Tell them exactly which screen to open and what to paste.

Then ## Guides, which does follow the spec's link-list form exactly. Thirteen lines, one per product we actually connect to, in the order our own docs menu shows them:

## Guides

- [Documentation](https://birta.dev/docs): every connection guide in one place
- [Claude](https://birta.dev/docs/claude): add the connector to Claude on the web
- [Claude Code](https://birta.dev/docs/claude-code): run the local connection from Claude Code
- [Custom domain](https://birta.dev/docs/custom-domain): point your own domain at a published site
- [MCP reference](https://birta.dev/docs/mcp): the hosted and local MCP connections, their tools, flows and limits

Notice what is missing. No link to this blog, no link to the homepage's pricing section, nothing that exists on the site just because it exists. The file names what an agent needs to connect and publish, not everything we have written.

Last is ## What you can do: ten short sentences with no links at all. "Roll back a site to any of its earlier versions," "Connect your own domain to a published site," "See visitor stats for every site." One per outcome, no adjectives.

The rules we wrote for ourselves

Only what has shipped: no prices, no unbuilt widgets, nothing that reads like a roadmap. Curated, not a sitemap: the guide list names products we connect to, not every page on the site. Hand-maintained, not generated. Whenever a new guide ships, someone edits this file by hand; it does not regenerate itself from the docs menu. English in both editions, including the Russian one, because a machine is the reader. And nothing here is an access key or a private surface. The file is written for anyone to fetch.

Where we broke the format on purpose

The spec expects every H2 section to be a link list. Two of ours are not. ## Connect is prose with subheadings, and ## What you can do is a plain list of sentences. The single most useful thing our file can tell an agent is not "here is more to read." It is how to actually connect, and exactly what to do if it cannot configure itself alone. That is an instruction, not a reading list, so we wrote it as one. There is nothing to link to for "roll back a site to an earlier version" either. It is a fact about the product, not a document, and a link there would be padding.

The opinion we are willing to defend: a file should serve the one reader it actually has, not pass a linter written for a reader it does not.

llms.txt is not robots.txt, and not a sitemap

Three files sit at the same kind of address and answer three different questions. robots.txt follows RFC 9309: it tells a crawler what it may fetch, and well-behaved crawlers are expected to obey it. sitemap.xml is close to exhaustive, every URL a site wants indexed, for a search engine's own crawl. llms.txt is neither a rule nor a full list. Nobody is obligated to fetch it, and a site is expected to leave most of itself out on purpose.

Our own robots.txt points at ours with a line the standard does not define: Llms-Txt: https://birta.dev/llms.txt. A parser that has never heard of the convention just skips a directive it does not recognize, so it costs nothing to add and nothing to be wrong about.

Does it do anything? What the data says

Three independent studies have looked, and they agree more than they disagree.

Ahrefs analyzed 137,210 domains through its Web Analytics customers, updated June 15, 2026. 28% publish a valid llms.txt file, and 97% of those got zero recorded requests in May 2026. Of the small remainder that did get a request, only about one in five came from anything AI-related: agents, training crawlers, retrieval bots behind assistants such as Perplexity. The rest was SEO audit tools, general crawlers, and tools checking whether the file existed at all. Google's John Mueller compared the whole idea to the keywords meta tag.

SE Ranking analyzed 300,000 domains in November 2025. Adoption sat at 10.13%, and when they built a model to predict which domains get cited by AI systems, adding "has an llms.txt" as an input made the model's predictions worse, not better. Noise, not signal. Adoption was lower among the highest-traffic sites (8.27%) than the smallest ones (9.88 to 10.54%): the domains with the most to plausibly gain were the least likely to have bothered.

Semrush counted 951 domains total as of July 2025 and concluded plainly that creating one "is probably not worth your time right now."

Where the AI companies themselves stand is more specific than "nobody adopted it." Both Anthropic and OpenAI publish an llms.txt for their own developer documentation, and Anthropic's own engineering guidance points to the format as a known way to hand an agent clean docs. Neither has said it crawls or uses other sites' files for ranking or citation.

Google is split with itself. Search says no. Gary Illyes said in mid-2025 that Google does not support the format and is not planning to, and Google's own AI optimization guide, updated July 10, 2026, states plainly that llms.txt is not needed for Search, including its generative features, and does not help or hurt ranking. Chrome disagrees, for a different reason. In May 2026, Lighthouse made its "Agentic Browsing" audit category the default, and one of its checks is simply whether a domain has an llms.txt at the root. Not because it affects search, but because an autonomous browsing agent orients faster with one.

That last split is the honest summary of the whole picture. Not "does it work," but "for whom, doing what, right now." A ranking algorithm has no use for it. An agent that has already been told to read it does.

Why we still publish one

We are not betting on being crawled. Our own landing page tells an agent exactly that, in the prompt every visitor can copy: "Read https://birta.dev/llms.txt and connect yourself to Birta. If you can't connect on your own, tell me exactly what to open and what to paste." That is today's reader, named directly, not a hoped-for future crawler the data above says probably is not coming.

The cost is trivial next to that. One route, about thirty lines, maintained by hand. The file earns its keep the moment one agent reads it and connects. It does not need Google to change its mind first. What happens on the agent's side once it has fetched the file is a short sequence we have written up separately: read, pick a way in, connect, stop at the one call the owner makes.

The rule worth taking away is not "publish one." It is narrower: publish one if it already has a reader today, a prompt that points at it, an agent you have told to fetch it, documentation another tool consumes directly. Skip it if the only reason is a bet that a crawler will start caring later. The evidence above says that bet is not paying off yet, for anyone.

And this is where it stops

Our /llms.txt describes Birta, the product, not the sites people publish through it. We do not write or generate an llms.txt for a project you host with us, by hand or with any tool. We do not have one, and this article is not announcing one.

We do publish an llms-full.txt too: the same connect instructions as the short index, plus every published guide's content in full, for a client that wants the whole reference in one fetch rather than following links out from the index. And we do not track who or what fetches either file, so unlike the studies above, we have no numbers of our own to add to them.

Writing one for your own site

Curate before you publish. Name the handful of things that actually matter, and leave the rest out on purpose: a full sitemap is what sitemap.xml is for. Put the file at the root as plain text. Point to it from robots.txt with a line like ours; a parser that does not recognize it just skips it, so there is nothing to lose. Keep it hand-written and revisit it when something real changes. A stale file that confidently describes last year's product is worse than no file at all. Start from the two required lines the spec actually asks for, an H1 and a one-line blockquote, and only add structure once you know who is going to read it, the way we did.

Common questions