Every so often the search and content world produces a new file, tag or protocol that promises to make your website legible to AI. llms.txt is the current one. It has a memorable name, a plausible rationale, and a growing number of people willing to sell you one.

It is worth understanding what it is, because the underlying idea is sensible. It is also worth being clear about what the evidence currently supports, which is considerably less than the enthusiasm suggests.

What llms.txt actually is

llms.txt is a proposed convention, not a standard. The idea is that a website publishes a plain Markdown file at the root of its domain that lists the pages which matter most, each with a one-line description of what it contains. Instead of an AI system crawling the whole site and working out for itself which pages are substantive, it can read one short, curated index.

The name deliberately echoes robots.txt, and that comparison is useful as long as you notice where it breaks down. Robots.txt is restrictive: it tells crawlers where they may not go. llms.txt is suggestive: it tries to point AI systems towards the material you consider worth reading. One is a fence. The other is a signpost — and signposts only work if someone is looking at them.

Why the idea is appealing

AI crawlers spend a lot of effort working out what a site is about. They wade through navigation, cookie banners, footers, near-duplicate pages and templated boilerplate before they get to anything that answers a question. A clean, machine-readable summary of the pages that actually carry meaning removes most of that guesswork.

In principle, that should make a site easier to understand and easier to cite. The argument is strongest for documentation-heavy products and content-heavy publishers, where there is a genuine hierarchy of importance and a real risk that the good material is buried beneath hundreds of thin or procedural pages. Markdown is also cheap to produce and readable by humans, so the proposal asks very little of anyone adopting it.

None of that is unreasonable. The problem is the gap between a sensible proposal and a working one.

The honest reality check

No major AI provider has officially committed to supporting llms.txt. Not OpenAI, not Google, not Anthropic, not Perplexity. There have been occasional comments from people inside these companies, but no documented commitment, no published specification of how the file would be used, and no guarantee that publishing one changes anything about how a site is read.

Independent research has not found evidence that the file measurably improves AI visibility or citation rates. Studies looking at whether sites with an llms.txt file are cited more often by AI assistants than comparable sites without one have not produced a convincing effect. That is not proof that it does nothing. It does mean nobody has yet demonstrated that it does something.

Adoption also remains thin — roughly one in ten of the sites examined in published surveys. And unlike robots.txt, which has decades of practice behind it and a formal specification that crawler operators actually implement, llms.txt is a suggestion that a group of interested people happen to like. Those are very different things, and the similar filename makes them look more alike than they are.

To be fair to the idea, it is not entirely ignored. Several companies, mostly developer-tooling businesses with large documentation sets, have reported seeing AI crawler requests for the file in their server logs after publishing one. Something out there is fetching it. But a request for a file is not the same as improved visibility. A crawler retrieving a document tells you it was downloaded, not that it was used, weighted, or that it influenced a single answer.

Warning

If an agency offers to sell you an llms.txt file as an AI visibility service, treat the pitch with suspicion. It is a short Markdown file that takes under an hour to write, and there is currently no independent evidence that having one improves how often AI systems cite you.

So should you publish one?

Probably, if it is genuinely cheap for you — and it usually is. The file costs very little to write, does no harm, and carries no technical risk. If you have substantial documentation, a large content library, or a technical audience, there is a reasonable case for adding one and forgetting about it. Think of it as a cheap option on a convention that might become useful, not as an investment with an expected return.

There is no good case for any of the following:

  • Paying someone a meaningful sum to produce one for you.
  • Treating it as a ranking factor, or reporting on it as though it were one.
  • Prioritising it over work that demonstrably affects how your site is read and understood.

What actually helps AI systems understand your site today

The measures that reliably make a site legible to AI systems are unglamorous and mostly predate the current wave of interest. They are worth doing properly before anything experimental.

  • Clean semantic HTML. Proper heading structure, real lists, real tables. The document outline should tell the story even with the styling stripped away.
  • Server-rendered content. AI crawlers generally do not execute client-side JavaScript. If your content only appears after hydration, there is a fair chance it is invisible to them.
  • Schema markup. An established standard with documented support, which is precisely what llms.txt lacks. It tells machines what an entity is rather than leaving them to infer it.
  • Clear, direct answers to specific questions. Content that states the answer plainly near the top is far easier to quote than content that circles it for six paragraphs.
  • A working XML sitemap. Accurate, current, and free of redirects and dead entries.
  • A robots.txt that does not block the crawlers you want. This catches people out regularly. It is worth checking before worrying about any new file.

If a site is weak on those six points, no amount of curated Markdown at the domain root will compensate.

If you do want to publish one

The format is straightforward. Create a Markdown file named llms.txt and place it at the root of your domain, alongside robots.txt. Start with a top-level heading containing your site or business name, follow it with a short blockquote summarising what you do, then add sections of links with a one-line description of each important page.

llms.txtmarkdown
        # Example Company

> We build accounting software for small UK construction firms.
> This file lists the pages most useful for understanding what we do.

## Documentation

- [Getting started](/docs/getting-started): Install, connect a bank feed, run a first report.
- [VAT and CIS](/docs/vat-cis): How we handle Construction Industry Scheme deductions.

## About

- [Pricing](/pricing): Plans, limits and what is included at each tier.
- [Security](/security): Data handling, hosting location and certifications.

Keep it short and genuinely curated. The value of the file, if it has any, lies entirely in the editorial judgement behind it. An llms.txt that lists every page on the site is just a sitemap in a less useful format, and it discards the one thing that made the idea interesting.

The question to ask of every new AI tactic

llms.txt will not be the last proposal of its kind. The pattern is already familiar: a plausible idea, an enthusiastic early community, a wave of agency packages, and very little published evidence either way. Some of these conventions will be adopted and become genuinely useful. Most will quietly disappear.

Before investing real time in any emerging AI-SEO tactic, ask what independent evidence exists that it works — and notice who benefits from telling you that it does.

Applied to llms.txt, that question gives a clear answer. Spend twenty minutes on it if you have documentation worth signposting. Do not spend money on it, do not report on it as a result, and do not let it displace the structural work that measurably determines whether machines can read your site at all.