What is an llms.txt file? A real one, explained line by line

An llms.txt file is a plain markdown file at the root of a site, /llms.txt, that gives an AI assistant a short description of what the site is and which pages matter, in one fetch. A proposal since September 2024, not a standard, and no search engine ranks a page for having one.

This one is real. Ours, at birta.dev/llms.txt, line by line below, with what is actually true about whether any of it works.

What GEO actually is, and why a file like this exists

Classic search ranks pages: a crawler indexes the web, a query matches a slice of it, and a reader picks from ten blue links. A large language model fetches material once, reads it, and hands back an answer or takes an action. No ranked list, no click.

That shift is what "GEO," generative engine optimization, names, and it is not a rebrand of SEO. A page competing for a ranking position wants to be judged the best match among many. A page trying to be useful to an assistant wants to be cheap to fetch and unambiguous in one pass. Closer to handing someone a one-page brief than to winning an auction.

llms.txt is one narrow answer to the second problem: a small curated index at a fixed address, so an assistant does not have to parse a styled homepage to learn what a site is. Not the only answer, and, as the data below shows, not a guaranteed one.

It is also not MCP. This file is about being read. MCP is the channel an assistant uses to do something once connected. One is a description, the other a set of hands.

What the file actually is

llmstxt.org is where the file comes from: a proposal Jeremy Howard published on September 3, 2024, sitting next to robots.txt and sitemap.xml at a site's root. The address is a convention, not a registered standard.

The specification is short. An H1 with the site's name is the only required part, a blockquote underneath gives a one-line summary, and after that any number of free markdown sections carry context, with H2-headed sections holding curated links in [name](url): note form. A table of contents for a model, not a full site map.

Markdown rather than HTML, on purpose. A styled page arrives wrapped in navigation, scripts and layout an assistant has to strip away. A short plain-text file is the content, fetched in one request.

Here is ours, line by line

The start of our live file. The first line is the H1, # Birta. Everything after it, copied exactly:

> Birta is hosting for landing pages built by an AI tool: the tool publishes the site, Birta keeps it online, and one panel holds the projects, versions, domains, visitor stats and the leads the sites collect.

Birta is for people who build a landing page by prompting the tool they
already use, a coding agent such as Cursor, Claude Code or Codex, or a plain
chat such as ChatGPT or Claude, instead of setting up hosting themselves.

The H1 and the blockquote are the whole required part of the spec. Everything after them we chose.

Next is ## Connect. Not a link list but prose with two subheadings, split by whether the client can read files on the owner's machine. It ends with one line we hold to on purpose:

If you cannot set yourself up — no shell, no access to your own settings — do not stop and do not ask the owner to figure it out. Tell them exactly which screen to open and what to paste.

Then ## Guides, which follows the spec's link-list form exactly. Thirteen lines, one per product we connect to, in the order our docs menu shows them:

## Guides

- [Documentation](https://birta.dev/docs): every connection guide in one place
- [Claude](https://birta.dev/docs/claude): add the connector to Claude on the web
- [Claude Code](https://birta.dev/docs/claude-code): run the local connection from Claude Code
- [Custom domain](https://birta.dev/docs/custom-domain): point your own domain at a published site
- [MCP reference](https://birta.dev/docs/mcp): the hosted and local MCP connections, their tools, flows and limits

Notice what is missing: no link to this blog, no pricing section, nothing that exists on the site just because it exists. The file names what an agent needs to connect and publish.

Last is ## What you can do: ten short sentences with no links. "Roll back a site to any of its earlier versions," "Connect your own domain to a published site." One per outcome, no adjectives.

The rules we wrote for ourselves

Only what has shipped: no prices, no unbuilt widgets, nothing that reads like a roadmap. Curated rather than a sitemap. Hand-maintained: when a new guide ships someone edits this file, it does not regenerate from the docs menu. English in both editions, including the Russian one, because a machine is the reader. And nothing here is an access key, because the file is written for anyone to fetch.

Where we broke the format on purpose

The spec expects every H2 section to be a link list. Two of ours are not. The single most useful thing our file can tell an agent is not "here is more to read," it is how to connect and what to do if it cannot configure itself. That is an instruction, so we wrote it as one. There is nothing to link to for "roll back a site to an earlier version" either: a fact about the product, not a document, and a link there would be padding.

The opinion we will defend: a file should serve the one reader it has, not pass a linter written for a reader it does not.

llms.txt is not robots.txt, and not a sitemap

Three files sit at the same kind of address and answer different questions. robots.txt follows RFC 9309 and tells a crawler what it may fetch, and well-behaved crawlers are expected to obey. sitemap.xml is close to exhaustive, every URL a site wants indexed. llms.txt is neither a rule nor a full list: nobody is obligated to fetch it, and a site is expected to leave most of itself out.

Our own robots.txt points at ours with a line the standard does not define: Llms-Txt: https://birta.dev/llms.txt. A parser that has never heard of the convention skips a directive it does not recognize, so it costs nothing to add.

Does it do anything? What the data says

Three independent studies have looked, and they agree more than they disagree.

Ahrefs analyzed 137,210 domains through its Web Analytics customers, updated June 15, 2026. 28% publish a valid llms.txt, and 97% of those got zero recorded requests in May 2026. Of the remainder that did get a request, about one in five came from anything AI-related. The rest was SEO audit tools, general crawlers, and tools checking whether the file existed. Google's John Mueller compared the whole idea to the keywords meta tag.

SE Ranking analyzed 300,000 domains in November 2025. Adoption sat at 10.13%, and when they modelled which domains get cited by AI systems, adding "has an llms.txt" as an input made the predictions worse. Noise, not signal. Adoption was lower among the highest-traffic sites (8.27%) than the smallest ones (9.88 to 10.54%): the domains with the most to plausibly gain were the least likely to have bothered.

Semrush counted 951 domains total as of July 2025 and concluded that creating one "is probably not worth your time right now."

Where the AI companies stand is more specific than "nobody adopted it." Both Anthropic and OpenAI publish an llms.txt for their own developer documentation, and Anthropic's engineering guidance points to the format as a known way to hand an agent clean docs. Neither has said it crawls or uses other sites' files.

Google is split with itself. Search says no: Gary Illyes said in mid-2025 that Google does not support the format and is not planning to, and its AI optimization guide, updated July 10, 2026, states that llms.txt is not needed for Search, including its generative features, and does not help or hurt ranking. Chrome disagrees for a different reason. In May 2026, Lighthouse made its "Agentic Browsing" audit category the default, and one check is whether a domain has an llms.txt at the root. Not because it affects search, but because an autonomous browsing agent orients faster with one.

That split is the honest summary. Not "does it work," but "for whom, doing what." A ranking algorithm has no use for it. An agent already told to read it does.

Why we still publish one

We are not betting on being crawled. Our own landing page tells an agent exactly that, in the prompt every visitor can copy: "Read https://birta.dev/llms.txt and connect yourself to Birta. If you can't connect on your own, tell me exactly what to open and what to paste." Today's reader, named directly.

The cost is trivial next to that: one route, about thirty lines, maintained by hand. The file earns its keep the moment one agent reads it and connects, without Google changing its mind first. What happens next on the agent's side is a short sequence written up separately.

So the rule is not "publish one." It is: publish one if it already has a reader today, a prompt that points at it, an agent you have told to fetch it, documentation another tool consumes. Skip it if the only reason is a bet that a crawler will start caring later.

And this is where it stops

Our /llms.txt describes Birta, not the sites people publish through it. We do not write or generate one for a project you host with us, and this article is not announcing one.

We do publish an llms-full.txt: the same connect instructions plus every published guide in full, for a client that wants the whole reference in one fetch. And we do not track who fetches either file, so unlike the studies above we have no numbers of our own.

Writing one for your own site

Curate before you publish. Name the handful of things that matter and leave the rest out; a full sitemap is what sitemap.xml is for. Put the file at the root as plain text. Point to it from robots.txt, since a parser that does not recognize the line skips it. Keep it hand-written and revisit it when something real changes, because a stale file that confidently describes last year's product is worse than no file. Start from the two required lines, an H1 and a blockquote, and add structure once you know who is reading.

Common questions