llms.txt in Practice: Does It Do Anything Yet? (2026)

llms.txt does almost nothing measurable for AI visibility in 2026, and anyone telling you it's essential is either repeating something they read or selling you an implementation. The server-log evidence is one-sided: 97% of these files went entirely unrequested across roughly 137,000 domains in May 2026, the requests that did arrive came mostly from SEO tooling, and Google has stated plainly that the file has no effect on its rankings or AI Overviews either way.

And you should probably still publish one, for reasons that have nothing to do with citations. It costs about fifteen minutes, it does no harm when done correctly, and it is cheap optionality on a standard that might yet get adopted. If you run developer documentation the case is stronger, because the tools your readers run are the one population plausibly consuming these files today.

What you should not do is treat it as a lever, report it as a deliverable, or let it displace the work that does move citations. That work is unglamorous and well-evidenced: structuring pages so individual passages can be retrieved and quoted, being reachable by the crawlers that feed answer engines, and having something worth citing in the first place. A file at your site root does not substitute for any of it.

The rest of this article is the evidence behind that verdict, including two tests I ran myself, and the one finding that would overturn it.

llms.txt is a markdown file at the root of a site that tells a language model what you publish and where to find it. It was proposed in September 2024 by Jeremy Howard of Answer.AI, and the reasoning behind it is sound: context windows are finite, HTML pages are full of navigation and markup that waste them, and a curated index of your best content in clean markdown is a genuinely useful artifact for a machine that arrives wanting to understand your site.

The format is simple. An H1 with the site or product name, a blockquote summarising what it is, then H2 sections listing key pages as markdown links, each with a one-line description of what that page covers. Most implementations run to somewhere between ten and forty entries.

There is a companion convention worth knowing, because it's the half that gets less attention and does more work. Where llms.txt is an index, llms-full.txt is a full-text dump of the underlying content in one file — orientation versus deep ingestion. The pattern of shipping both is now standard among documentation-heavy publishers, and if any part of this convention is genuinely being consumed, that's the part.

The critical thing to understand is what llms.txt is not. It is a proposal, not a standard. No standards body ratified it, no major AI provider has committed to it, and nothing enforces or verifies it. Publishing one is a statement of intent addressed to systems that have not agreed to listen.

Two studies define what is publicly known here, and most articles on this topic cite neither.

The adoption side comes from Originality.ai, which tracked more than three million websites between June 2025 and May 2026. Instances of llms.txt rose from 4,088 to 36,120 over that window — an 8.8-fold increase, with roughly 38,980 sites publishing one of the related formats by May 2026. Real growth, from a base so small that 8.8x still leaves the file present on a rounding error of the web.

The consumption side comes from Ahrefs, which did the thing the debate needed: it looked at server logs rather than at the presence of files. Across roughly 137,210 domains with measurable traffic, 97% of llms.txt files received zero requests in May 2026. Of the 3% that were fetched at all, SEO audit tools accounted for about 21.7% of requests while AI retrieval bots accounted for roughly 1.1%. Broken down by agent, GPTBot made about 4.51% of requests, ClaudeBot about 0.80%, and others trailed into noise.

The finding that should end the argument is the one that's easiest to skim past: AI bots did not request llms.txt on domains where the file didn't exist. Probing for a file at a fixed path is close to free — a single request that either 404s or doesn't. If these systems wanted the file, speculative probing is exactly the pattern the logs would show, and it isn't there. That's not weak adoption. That's absence of interest.

One caveat I'd apply to both datasets and to my own: adoption is cohort-shaped rather than uniform. A fixed panel of 219 developer-tooling hosts measured 51.8% adoption in August 2026, while the Tranco top 1,000 sat at 8.7% in June 2026. Those aren't contradictory findings — they're what happens when you sample sectors whose readers run coding agents versus sampling the web by traffic. Which is precisely the distinction the crawler studies can't see, and the reason for the second test below.

Published datasets tell you what happens on average across a hundred thousand sites. They don't tell you what's happening on yours, and the whole point of this file is site-level. So here is the same question asked of one site that has published an llms.txt since launch.

Method, stated plainly so you can repeat it or dismiss it. One domain, stacknovahq.com, a technical blog publishing in the AI-tools niche — meaning it sits in exactly the cohort where AI crawler interest should be highest, not a random sample. Access logs for [WINDOW], filtered for requests to /llms.txt, then grouped by user-agent. N=1 is a limitation and I'm labelling it rather than hiding it: this is a data point, not a study.

The commands, if you want to run this on your own logs in the next five minutes:

grep '/llms.txt' access.log | wc -l — total requests to the file.

grep '/llms.txt' access.log | awk -F'\"' '{print $6}' | sort | uniq -c | sort -rn — requests grouped by user-agent, most frequent first.

grep -iE '/llms.txt' access.log | grep -iE 'gptbot|claudebot|perplexitybot|google-extended|bingbot|ccbot' — just the AI crawlers, which is the number that actually answers the question.

If your host doesn't expose raw logs, most CDNs and edge platforms will filter by path in their analytics, and a Cloudflare or Vercel log drain will get you there in about the same time.

[RESULTS PLACEHOLDER — fill from the real log pull before publishing. Report: total requests over the window; the user-agent breakdown with counts; how many came from AI retrieval crawlers specifically versus SEO tools, uptime monitors and unidentified agents; the date of the first and most recent AI-crawler request if any exist. If the count is zero, say zero — that is the finding, it matches the large-scale data, and it is more useful to readers than any spun alternative. Attach one screenshot or code block of real log lines.]

[INTERPRETATION PLACEHOLDER — write this after seeing the numbers, not before. If the result is zero or near-zero, the honest framing is that a single site in the highest-interest niche on the web still isn't being read, which strengthens rather than merely echoes the Ahrefs finding. If there are real AI-crawler fetches, that is genuinely newsworthy and deserves its own section: report the agent, the frequency, and crucially whether the file was fetched before or after the crawler took the actual content pages.]

This is the test that justified writing the article, because it targets the only plausible case left for llms.txt and nobody appears to have run it.

The reasoning: search crawlers index the whole web and have no particular need for your summary of your own site. Agents are different. A coding agent or browsing assistant arrives at one specific site with one specific task, under a hard context budget, with no index and no time. That is exactly the situation llms.txt was designed for — and it's the situation crawler-log studies structurally cannot observe, because agent traffic doesn't announce itself as a search crawler.

The protocol, which you can replicate against your own domain:

One — pick a task that genuinely requires site-level orientation rather than a single page. Something like: 'read stacknovahq.com and tell me which articles cover Claude Code pricing' forces the agent to work out what the site contains.

Two — run that same task through each agent with browsing or fetch capability: Claude Code, Codex, Cursor, and at least one browser-based assistant. Keep the wording identical across runs.

Three — timestamp each run, then grep the access logs for that window and record every path each agent requested, in order. The ordering is the interesting part: an agent that fetches /llms.txt before the homepage is using it as intended; one that fetches it after already crawling pages is doing something closer to completeness-checking.

Four — repeat with the file temporarily removed, to see whether behaviour changes or whether the agent simply proceeds via the homepage and sitemap.

[RESULTS PLACEHOLDER — run and fill before publishing. Report per agent: did it request /llms.txt at all; in what order relative to other paths; what user-agent string it presented; did removing the file change how it navigated the site. Include the raw log lines for at least one positive case. If every agent ignored the file and navigated by homepage and sitemap instead, that is a clean negative result and should be reported as prominently as a positive one would be — it closes the last open case for llms.txt on non-documentation sites.]

Whatever the outcome, note the limit of what this test can prove. It measures whether agents fetch the file, not whether fetching it improved their answer. Measuring that properly would need paired runs with and without the file, scored blind on answer quality, which is a bigger study than one person's blog and an honest thing to admit rather than paper over.

Strip away the commentary and the record from the companies that would have to adopt this file is short and consistent.

Google updated its documentation in June 2026 to state that llms.txt has no effect — positive or negative — on Search rankings or AI Overviews. Search ignores it. That is about as unambiguous as platform guidance gets, and it settles the question for the largest AI answer surface on the web.

John Mueller of Google Search Relations added the structural argument, which is more interesting than the policy because it explains why adoption may never come. A self-reported manifest cannot differentiate between sites, since every site would use it to declare itself the most authoritative source on its topic. He has drawn the comparison to the old keywords meta tag — a self-description that search engines abandoned for exactly this reason — and noted that a system which has already fetched your real content has little reason to trust a separate file describing it instead. Both points are hard to argue with, and neither is about llms.txt being badly designed. They're about self-description being a weak signal in any system that can just read the thing itself.

OpenAI's documented guidance continues to direct site owners to robots.txt for crawler control rather than to llms.txt. And as of writing, none of OpenAI, Google, Anthropic, Meta, Perplexity or Mistral has publicly stated that its production systems read or act on the file.

One piece of nuance in Google's tooling that's often misread: some audit tools flag a page only when the server returns an error while fetching /llms.txt, and treat a clean 404 as not applicable, because the file is entirely optional. So a red flag in an audit usually means your server is misconfigured, not that you're missing something important.

Against all that, the counter-case is real but small: documentation platforms have adopted the convention broadly, with Mintlify shipping the file by default across the docs sites it hosts, and the llms.txt plus llms-full.txt pairing is now standard among developer-facing publishers including several of the AI companies themselves. Which tells you where the genuine use case lives.

There is a version of this file that does useful work, and it isn't the version being sold as a visibility tactic.

If you publish developer documentation, API references, or deep technical guides, the argument changes — not because search engines will read it, but because your readers' tools might. Docs assistants, coding agents and internal RAG pipelines all face the problem llms.txt solves: get oriented in an unfamiliar site fast, without burning context on navigation chrome. That is a real user need, met by a real artifact, and it explains why adoption in the developer-tooling cohort measured above 50% while the broader web sat in single digits.

For that use case, llms-full.txt is usually the more valuable of the two. An index tells a tool what exists; a full-text dump lets it actually answer questions without twenty follow-up fetches. If you're going to invest in one, and your content is documentation-shaped, invest there.

The second legitimate reason is optionality. This standard costs fifteen minutes to implement and imposes no ongoing burden beyond keeping it accurate. If adoption arrives — one announcement from a major provider would do it — you're already compliant. That is a reasonable bet at that price, and it's roughly why my own site has one.

What doesn't survive contact with the evidence is the claim that llms.txt improves your AI visibility. There is no published study supporting it, the log data argues against it, and treating it as a lever displaces effort from work that does move citations. If you only have an hour, spend it on making individual passages retrievable rather than on a file nothing requests.

Skip the debate and find your row. Note that no row promises a citation lift, because no published evidence supports one — the recommendations here are about cost, optionality and where your next hour is better spent.

If you've decided it's worth the optionality, here's the whole job.

Create a markdown file served at yoursite.com/llms.txt. Open with an H1 carrying your site or product name, then a blockquote of one or two sentences describing what the site is and who it's for. Follow with H2 sections grouping your content — docs, guides, reference, policies — and under each, a markdown list of links with a one-line description per entry. Ten to forty entries is the normal range. Serve it as text/plain or text/markdown with a 200 status. Done.

If your content is documentation-shaped, add llms-full.txt alongside it: the same content as one flat text dump, generated from your source rather than maintained by hand.

The five failure modes, in the order I've actually seen them happen:

One — serving HTML. If your framework returns your app shell for unknown routes, /llms.txt may return a styled 200 page of markup. Check what curl returns, not what the browser renders.

Two — letting it go stale. A file listing pages that moved or died is worse than no file, and since AI systems weight freshness heavily, a manifest untouched for a year signals a site untouched for a year. Regenerate it in your build.

Three — dumping the sitemap. An unannotated list of every URL you have is not an index; the one-line descriptions are the entire value, because they're what lets a machine decide what to fetch.

Four — marketing copy. This file is read, if at all, by systems that penalise promotional language. Describe what each page contains, flatly.

Five — treating it as crawler control. It grants nothing and blocks nothing. If you want to control AI crawler access, that is robots.txt and your user-agent rules, and it is a genuinely separate job worth doing properly.

Then add one line to your quarterly checklist: grep the logs for /llms.txt. If nothing is reading it, you've lost fifteen minutes and confirmed where not to spend the next hour. If something starts reading it, you'll be among the first people to know — and I'd like to hear about it.

The reason to be blunt about llms.txt isn't that it's harmful. It's that it's a satisfying substitute for harder work, and satisfying substitutes are how months disappear.

What has evidence behind it for AI answer visibility is comparatively boring. Being reachable and indexable by the crawlers that feed answer engines, which is a technical audit rather than a tactic. Structuring pages so that individual passages answer questions cleanly on their own, which is the single highest-leverage change available and is the subject of my breakdown of passage-level structure. Publishing something genuinely citable — original measurements, first-hand testing, numbers that exist nowhere else, which is why cost breakdowns from real client work get quoted while summaries of other people's posts don't. Keeping pages fresh, because AI citation pools skew recent and decay faster than rankings do. And existing on third-party sites, since the majority of brand mentions in AI answers come from pages you don't own.

None of those fit in a fifteen-minute task, which is exactly why a file at your site root is so appealing. Sort every AI-visibility recommendation you get — including mine — with one question: does this change what a machine can read and do on my site, or only what I claim about it? Structure, indexability and original data all change what a machine can read. A manifest describing yourself changes only what you claim.

That's the whole argument, and it's why my llms.txt is staying up and why I don't think about it.