Analysis of over 137,000 domains found that most of the traffic hitting llms.txt files comes from SEO audit tools checking whether the file exists. The industry is largely studying itself — and that detail tells you almost everything about where this stands.
What the evidence actually shows
llms.txt is a Markdown file at your domain root offering AI systems a curated map of your most important pages. Proposed in late 2024, it has been promoted heavily as "the new robots.txt." The data doesn't support that framing.
Adoption grew; usage didn't. One study found llms.txt adoption rising roughly 8.8 times over — while around 97% of those files received no AI requests at all, and statistical modelling showed no citation effect.
Crawlers aren't looking for it. The most telling finding: AI bots were not observed requesting llms.txt on domains that didn't have one. If retrieval systems wanted the file, cheap speculative probing is exactly what you'd expect to see in server logs. It isn't there.
Google has been unambiguous. Search Central documentation states that llms.txt is not used for Google Search — not for rankings, not for AI Overviews, not for AI Mode. Google representatives have compared it to the discredited keywords meta tag, and that comparison has a specific logic worth understanding: a file where a site declares what it's about, with no verification mechanism, is structurally the same kind of thing.
The finding that says it all AI crawlers don't probe for llms.txt on sites that lack it. The files that do get fetched are mostly being fetched by SEO tools checking whether you've made one.
The conflation powering the hype
You'll see it claimed that major AI companies "use" llms.txt, often citing Anthropic and OpenAI. That claim rests on eliding two entirely different things.
| Claim | Status |
|---|---|
| AI companies publish llms.txt for their own documentation | True and verifiable — several do |
| Their crawlers consume third-party llms.txt as a citation signal | Not publicly committed to by any major provider |
Anthropic, OpenAI, Stripe, Vercel, Cloudflare and others do ship the file. They're doing it so that coding assistants can retrieve their API documentation cleanly — a decision about their own developer experience, aimed at their own users.
That is not evidence that their retrieval systems read your file when deciding what to cite. Those are separate claims requiring separate evidence, and the second one doesn't have any. Crawler documentation from the major providers points to robots.txt for crawler management; llms.txt appears in their developer docs, not their crawler specifications.
Once you separate the two, most of the promotional content on this topic falls apart.
So what is it genuinely for?
The binary "useless" position doesn't hold either. There's a real use case — it's just narrower than the marketing suggests, and it's a different audience.
Developer documentation consumed by coding agents. When someone uses an AI coding assistant to write an integration against your API, the assistant retrieves your docs in real time. A curated Markdown file means fewer tokens spent parsing HTML navigation and a lower chance of the tool inventing an endpoint that doesn't exist. For API products this is a developer-experience requirement rather than a marketing tactic.
Agentic browsing readiness. Autonomous agents that navigate sites to complete tasks — booking, retrieving documentation, comparing options — benefit from a clean structural map before they start. Chrome's Lighthouse added an optional agentic browsing audit checking for the file in 2026, which signals the direction of travel without establishing present-day citation value.
Internal AI infrastructure. If you're building retrieval over your own content, a curated manifest is straightforwardly useful. That's you consuming your own file, which nobody disputes works.
The honest summary: llms.txt is real infrastructure aimed at the wrong audience relative to how it's being sold. It's agent plumbing, not a citation lever.
Nightwatch
Accurate daily rank tracking tool with beautiful visual reporting dashboards
Best for: Daily Rank Tracking
The implementation mistake that does cause harm
One popular approach genuinely damages sites, and it's worth naming specifically.
Some guides recommend generating an individual Markdown copy of every page alongside the llms.txt manifest. If those Markdown files are indexable, you've created duplicate content at scale — every page on your site now exists twice. That dilutes crawl budget and can suppress rankings for the originals.
If you implement anything beyond a simple manifest, make sure the generated files are excluded from indexing. This is the one way a "harmless experiment" stops being harmless.
What actually gets you cited
Less exciting than a new file, considerably better evidenced.
Be in the index the assistant retrieves from
Several AI assistants lean heavily on existing search infrastructure rather than crawling independently — analysis has found a large majority of one major assistant's citations matching the top results of a conventional search index. Which means classic search visibility remains the primary route into AI answers, and the two aren't the separate disciplines they're often presented as. That's the ground covered in generative engine optimisation and in our mid-year state of SEO.
Serve content in the HTML
The most consequential technical factor, and the one most often broken accidentally. Many AI crawlers do not render JavaScript. Content assembled client-side may be fully visible to a browser, visible to Google, and effectively blank to the systems generating AI answers.
This is a common casualty of a platform change — a site can migrate successfully by every conventional measure while quietly becoming invisible to AI retrieval, which is why it belongs on the checklist in any site migration. Test it by viewing source rather than inspecting the rendered DOM: if your main content isn't in the raw HTML, no manifest file will help.
Write things that can be lifted
A citation requires something quotable. Specific claims, named figures with sources, plainly stated definitions, clearly labelled frameworks — these survive being separated from their surroundings. Atmospheric brand prose contains nothing to extract, however well the site is structured.
This is the content-side half of the work and it's where the actual leverage sits, as covered in content marketing in the age of AI answers. No technical file compensates for a page with nothing specific in it.
Use structured data properly
Schema is the machine-readable layer that actually has adoption, verification and years of tooling behind it — the substantive version of what llms.txt gestures at. Accurate markup describing what a page is and what it contains does the job people hope the manifest will do, and the practicalities are in making your site machine-readable.
Demonstrate depth on the subject
Systems assembling answers weigh whether a source is substantive on a topic. Coverage from many angles, consistently, reads as a source rather than a passing mention — which is the argument for topic clusters, and it operates at a level no file can shortcut.
Why this keeps happening
Worth a moment, because llms.txt won't be the last one.
The pattern is recognisable. A genuine shift creates real anxiety. The underlying response is slow, expensive and unglamorous — better content, cleaner technical foundations, depth built over quarters. So a cheap, concrete, checkable artefact appears and gets adopted as a proxy for the hard work, because shipping a file is satisfying in a way that "keep publishing genuinely specific content for a year" is not.
The keywords meta tag comparison is apt for exactly this reason. That tag also let sites declare what they were about, with no verification, and it also felt like doing something.
The practical filter for the next one: ask who verifies the claim. If a mechanism lets you assert something about yourself with nothing checking it, expect the systems consuming it to discount it — because the people building those systems know that too.
A note on the sources
Worth flagging, because researching this topic produces genuinely contradictory material.
Some 2026-dated articles state confidently that major AI providers use llms.txt as a signal. Others, drawing on log analysis and provider documentation, state the opposite. The disagreement is largely explained by the conflation described earlier — companies publishing the file for their own docs being reported as companies consuming it.
The strongest evidence available is behavioural rather than declarative: what crawlers actually request, measured across large domain samples. Statements of intent from anyone with a product to sell in this space deserve more scepticism than server logs. And it's worth noting that a large share of writing on llms.txt comes from tools that generate llms.txt files.
That includes this article, eventually. If a major provider commits to consuming third-party files, the calculus changes — check current documentation rather than relying on any guide, including this one.
What to do
- Check whether your content is in your raw HTML. View source on your three most important pages. If the content isn't there, fix that before anything else — it's the difference between being retrievable and not.
- Verify your structured data. The machine-readable layer with actual adoption.
- Ship llms.txt if you have developer documentation. Half a day, genuinely useful to your users, no downside.
- Ship it anyway if you like, but budget zero effect on citations and don't count it as progress.
- Exclude any generated Markdown copies from indexing if your implementation creates them.
- Spend the remaining effort on specificity — claims worth quoting, figures with sources, definitions stated plainly. That's what a citation is made of.
- Track whether you're actually cited. Query your priority questions in a few assistants monthly. That measurement matters more than any file.
If the underlying problem is that your site is technically fine and still never quoted, the constraint is usually the content rather than the plumbing — the point at which an SEO and content partner is more useful than another technical file.
The short version
Around 97% of llms.txt files get no AI requests, crawlers don't probe for the file on sites that lack it, and Google states plainly it isn't used for Search or AI Overviews. The claim that major AI companies use it conflates publishing a file for their own docs with consuming yours — different things, and only the first is established. The genuine use case is narrow and real: developer documentation retrieved by coding agents. Ship it if that's you, ship it anyway if you like since it's cheap, but budget no citation effect. What actually earns citations is being in the index assistants retrieve from, serving content in raw HTML rather than assembling it client-side, accurate structured data, and writing specific enough claims that there's something worth quoting.
Technically sound and still never quoted in AI answers?
We fix what actually drives citation — retrievability, structure, and content specific enough to quote.
Explore SEO Services →Frequently asked questions
Does llms.txt actually help you get cited by AI?
There is no good evidence that it does. Google's Search documentation states plainly that llms.txt is not used for Search, including AI Overviews and AI Mode, and Google representatives have compared it to the discredited keywords meta tag. Large-scale analysis found around 97% of llms.txt files received no AI requests at all, with statistical models showing no citation effect. Notably, AI crawlers were not observed probing for the file on domains that did not have one — which is what you would expect to see if they wanted it.
Why do Anthropic and OpenAI publish llms.txt files if crawlers don't use them?
Because publishing a file for your own documentation is a completely different thing from your crawler consuming other people's files, and the two get conflated constantly. Several AI companies ship llms.txt for their developer documentation so that coding assistants can retrieve their API references cleanly. That is a documentation decision aimed at their own users. None of them has publicly committed to their production retrieval systems reading third-party llms.txt files as a citation or ranking signal, and those are separate claims that require separate evidence.
Who should actually create an llms.txt file?
Primarily sites whose content is consumed programmatically: developer documentation, API references, and technical product docs. AI coding assistants retrieve these in real time, and a curated file materially improves token efficiency and reduces the chance of a tool generating code against an endpoint that does not exist. For a typical marketing site, blog or local business, the file is cheap and harmless but should not be expected to influence AI citations, because the evidence for that effect does not currently exist.
What actually gets content cited in AI answers?
Unglamorous fundamentals rather than a file. Being indexed in the sources the answer engines retrieve from matters most, since several assistants lean heavily on existing search indexes. Serving clean, crawlable HTML matters because many AI crawlers do not render JavaScript, so client-side assembled content can be invisible to them. And publishing specific, self-contained statements that answer a question directly makes a passage quotable, whereas atmospheric prose contains nothing extractable regardless of how well the site is structured.
Is there any downside to creating an llms.txt file?
The file itself is low risk — roughly half a day to produce, no ongoing maintenance, and no known penalty. The real costs are indirect. One is opportunity cost, where a team believes it has addressed AI visibility and stops doing the work that actually matters. The other is a specific implementation mistake: generating individual indexable Markdown copies of every page creates duplicate content at scale, which can dilute crawl budget and suppress rankings for the original pages.
