llms.txt – a proposed standard for LLM-readable content, and why we are sceptical for now

Definition

llms.txt is a proposed file format that is placed in the root of a website and describes its content in markdown for AI systems. The format is a community proposal, not an official standard adopted by any AI provider.

llms.txt is a proposed text-based file format that sits in the root of a website – much like robots.txt – and is intended to help large language models (LLMs) and AI systems understand what the website is about, which pages matter most and how the content fits together. The idea is good. The evidence, so far, is thin.

At SEOS Design we naturally take an interest in llms.txt and follow how it develops. We document the format because our clients ask about it, and because we think you should know what you are passing up before you pass it up. But we want to be straight: the independent experiments we have seen do not currently suggest that an llms.txt file affects how content is selected or cited by AI systems.

What llms.txt is meant to be

The proposal comes from Jeremy Howard (Answer.AI) and was published in September 2024. The basic idea is that a website should be able to put an llms.txt in its root, for example https://example.com/llms.txt, which describes in plain markdown:

• What the website is (a short summary)
• Which sections matter most
• Links to key resources, preferably in a markdown version
• Optionally an llms-full.txt with all the content gathered in one place, so that an LLM can take in the whole website in a single context window

The concept is a cross between a sitemap, a robots.txt file and a summary "about us" page – but written for a machine that reads markdown, which carries less noise than raw HTML.

What does Google say about llms.txt?

The clearest signal on this from a major search engine comes from Google's John Mueller. In a public post on Bluesky in summer 2025, he compared llms.txt to the outdated keywords meta tag: "It's like the keywords meta tag — a site owner's claim about their content." Mueller added that, as far as he knows, no major AI service uses llms.txt. According to Search Engine Journal he also recommended setting noindex on the file so that it does not show up in search results.

It is not an official Google policy statement, but it is as close to a "no" as you are likely to get from a Google spokesperson today. OpenAI, Anthropic and Perplexity have not publicly committed to reading or prioritising the file either.

Why we are sceptical – for now

For a file like this to have an effect, someone has to actually read it and act on it. robots.txt works because Googlebot, Bingbot and the like are implemented to follow the specification and have clear rules. For llms.txt there is no such commitment – and, as noted above, Google has if anything actively played down its value.

The experiments we have seen from the SEO and AEO community consistently struggle to show an effect. All else being equal, pages with an llms.txt appear in AI answers about as often as pages without one.

Two things are worth pointing out:

• Absence of evidence is not evidence of absence. It is entirely possible that future systems will start respecting llms.txt if the file becomes widespread enough.
• The comparison with schema markup is tempting but flawed. JSON-LD became standard because Google actively rewarded it with rich results – a clear incentive. llms.txt currently has no equivalent mechanism. Historically, Mueller's comparison with the keywords meta tag is actually more apt: a proposal driven by website owners that never gained a foothold with the search engines.

So our position is this: we do not recommend building a strategy around llms.txt, but we have nothing against putting up a simple version – the cost is close to zero.

What an llms.txt looks like

The proposal defines a very lightweight structure in markdown:

# Company name

> A short, clear summary of what the website is about
> and who it is for.

## Important pages

- [Services](https://example.com/services.md): What we offer
- [Case studies](https://example.com/case-studies.md): Results we have delivered
- [Glossary](https://example.com/glossary.md): Terms we explain

## Secondary

- [About us](https://example.com/about.md)
- [Contact](https://example.com/contact.md)

Note the markdown links in the example. Part of the idea is to pair the .html version (for people) with a .md version (for LLMs) – markdown is more token-efficient and contains less boilerplate than HTML with navigation, ads and scripts. The argument was stronger in 2024 than it is in 2026: modern context windows in Claude and Gemini already handle large HTML websites without any problem.

For larger websites, the proposal also includes an llms-full.txt that gathers all the body content in a single markdown file. In practice it becomes an LLM-friendly export of the whole website.

How it differs from robots.txt and sitemap.xml

llms.txt does not replace either of them. It complements them.

• robots.txt controls which parts of the website may be crawled. It says whether you may read.
• sitemap.xml lists all the URLs that should be indexed. It says what exists.
• llms.txt summarises and prioritises content in markdown. It says what matters and how it fits together.

Of the three, robots.txt is mandatory in practice, sitemap.xml the de facto standard, and llms.txt experimental.

What should you do as a website owner?

We break it down into three levels:

• Level 1 – do nothing about llms.txt yet. Put your energy into what we know works: well-structured content, schema markup, semantic heading structure, internal linking and E-E-A-T signals.
• Level 2 – publish a minimal llms.txt. If you already have a clear information architecture and a sitemap, creating an llms.txt is about half an hour's work. Include the 10–20 most important pages with short descriptions. Consider setting noindex on the file, in line with Mueller's recommendation, so that it does not show up in search results. It does no harm. It might help.
• Level 3 – generate a markdown version of each page. This is where we start to doubt the ROI. Maintaining parallel .md versions of every important page takes time and commitment.

How we approach it ourselves

We are following developments and will update this page when solid evidence appears – either that a major AI provider officially starts respecting the format, or that controlled experiments show a consistent effect on citation rate or share of voice in AI answers.

A small or medium-sized company with limited time for digital content usually has more to gain from basic technical SEO, faster loading, better headings and clearer entities than from chasing an experimental text file format. Once that list is ticked off, we can talk about llms.txt.