I Said Don’t Bother With llms.txt. Then I Looked at Anthropic’s. Here Is What Changed and What Didn’t.

What changed when major AI companies started publishing llms.txt themselves, and what stayed exactly the same for everyone else.

A few weeks ago I wrote that llms.txt is mostly a waste of time. The crawler logs backed me up, the studies backed me up, and Google said on the record it would not use the file. Then I went looking and found that several major AI companies publish one. That stopped me. If the AI labs themselves are doing this, was I wrong? The honest answer is no, but the full picture is more interesting than either post alone.

Key Takeaways

Wait, So Was the First Article Wrong?

No, but it was incomplete, and I would rather say that plainly than pretend I had the whole thing the first time. The original post argued that llms.txt does not measurably increase AI citations for a typical website. That claim is still true and still supported by the data.

What the first post did not cover is that the major AI companies publish llms.txt files of their own. I found that out after publishing, which is the honest order of events. It looked at first like a contradiction, a reason to walk the whole thing back.

It is not a contradiction. It is a different use case wearing the same filename, and the difference is the entire point of this follow-up.

Which AI Companies Actually Publish an llms.txt?

Several of the most visible AI and developer platforms do. Anthropic, OpenAI, Perplexity, Stripe, Cloudflare, and Zapier all expose an llms.txt or llms-full.txt on their documentation surfaces, with platforms like Vercel and Hugging Face in the same lane.

Anthropic’s setup is the cleanest example. It publishes a slim llms.txt index that links to a much larger llms-full.txt file containing its full documentation, so a tool can pick the level of depth it needs. The slim file is a table of contents. The full file is designed to provide the documentation in one larger Markdown-friendly format.

That sounds like a strong endorsement of the standard. Until you look at where the file lives and what it is for.

Why Do AI Labs Publish a File They Don’t Confirm Reading?

Because the file is for their developer documentation, not their search behavior. These llms.txt files are overwhelmingly attached to developer docs, API references, or product documentation surfaces, not ordinary marketing sites, and their job appears to be helping coding assistants and agent tools load documentation cleanly on demand.

Here is the use case in plain terms. This is the kind of file that makes sense for coding agents, IDE assistants, and developer workflows, where a user or tool intentionally loads documentation as context. The llms.txt and llms-full.txt give the tool a clean, structured copy of the docs to pull in, instead of scraping a JavaScript-heavy website.

That is a completely separate thing from an AI search engine deciding, on its own, to read your llms.txt when answering a stranger’s question. One writer put it best: the existence of an llms.txt on docs.anthropic.com does not prove that Anthropic reads your llms.txt during search. Confusing the two is like assuming a restaurant that prints a menu also reads every other restaurant’s menu before cooking.

So Does Publishing One Help My Business Get Cited?

For most businesses, still no, and that is the part that did not change. If your site is a coaching business, a course, or a local service, your content is probably not being loaded into Cursor by developers. It is being retrieved by AI search, and AI search has not committed to reading llms.txt.

The evidence from the first article still stands. SE Ranking found no correlation between having llms.txt and how often a site gets cited, and Google confirmed it does not use the file for AI Overviews. A documentation platform publishing one for its developer audience does nothing to change that for a Kajabi site.

There is one nuance worth adding, because it makes the whole split visible inside a single company. Google Search says llms.txt is not needed and is ignored for Search visibility, but Chrome’s Lighthouse now includes an optional llms.txt audit for agentic browsing. That is not a contradiction. It is the same divide one more time: llms.txt may help agents understand a site’s structure, but that is not the same as helping a page get cited in AI search.

The lesson is to match the tactic to the situation. llms.txt earns its place when your content is technical documentation that gets loaded into AI coding tools. It does not earn its place as a general citation booster, no matter how many developer platforms publish one.

Is There Any Honest Argument for Adding It Anyway?

Yes, and I want to give it fairly, because the all-or-nothing version of this debate is the part everyone gets wrong. The strongest case is historical: web standards often get published by sites first and supported by platforms later. Robots.txt was a convention before any search engine formally committed to it.

So an auto-generated llms.txt costs almost nothing and gives you optionality if adoption shifts. That was true in the first article and it is true now. The mistake was never adding the file. The mistake is believing the file is doing work it has not been shown to do.

What changed for me is the precision of the recommendation. Before, I would have said skip it unless it is free. Now I would say skip it for citation purposes, but if you run a documentation-heavy or developer-facing site, an llms.txt is a real, working tool for getting your docs into AI coding assistants. Same file. Two very different reasons to care.

Frequently Asked Questions About llms.txt and AI Companies

Q: Does Anthropic publish an llms.txt file?
A: Yes. Anthropic publishes a slim llms.txt index on its documentation site that links to a larger llms-full.txt file containing its full docs, so AI coding tools can load the reference material on demand.

Q: Do OpenAI and Perplexity publish LLM-readable docs files too?
A: Yes. OpenAI’s developer docs publish an llms.txt, while Perplexity’s docs currently expose an llms-full.txt. More broadly, several AI and developer platforms now publish machine-readable documentation files for agent and coding-tool workflows, alongside companies like Stripe, Cloudflare, and Zapier.

Q: If AI companies publish llms.txt, does that mean it works for SEO?
A: No. Those files primarily make their documentation easier for AI tools, coding agents, and developer workflows to consume, not to influence search rankings. Publishing a file for your own docs is different from an AI search engine reading your file when answering questions.

Q: Should a normal business add an llms.txt to get cited by AI?
A: There is still no evidence it increases citations for a typical business site. It is cheap to add as optionality, but it is not a citation lever, and on-page structure and schema do far more.

Q: When is llms.txt actually worth adding?
A: When your site is documentation-heavy or developer-facing and your content gets loaded into AI coding tools like Cursor or Claude Code. In that case the file is a real, working way to give those tools clean reference material.

Q: What is the difference between llms.txt and llms-full.txt?
A: llms.txt is a slim index that lists and links to your key documentation. llms-full.txt is a single large Markdown file containing the full documentation text, for tools that want to ingest everything at once.

I almost did not write this, because admitting a post was incomplete feels worse than it should. But the whole point of learning in public is that the second look is allowed to sharpen the first one. The companies publishing llms.txt are not proving you should. They are proving they had documentation to feed a coding tool, which is a problem most of us do not have.

AI Visibility Studio helps websites structure content so AI systems can find it, understand it, cite it, and actually use it when generating answers. aivisibilitystudio.com

References