Back in 2025, llms.txt was the file everyone told you to build. Add it, the advice went, and ChatGPT and Perplexity would finally understand your site and start citing you. So teams built them. Then Google put out its own AI guidance in 2026 and said, in effect: this does nothing for us. Cue the confusion. AI crawling itself is exploding. Cloudflare’s 2025 Radar Year in Review clocked AI user-action crawling — bots fetching live pages to answer real user queries — growing more than 15x in a single year, so the instinct to prepare for machine readers wasn’t wrong. The file, though? Different question. So should you bother with one or not?
Does llms.txt work? The short answer
No. Not the way most people hoped. For Google Search and AI Overviews, llms.txt has no effect. Google confirmed that directly in 2026. It has never been a ranking input and it isn’t one now.
There’s one real exception: developer documentation. If your audience is people using AI coding tools, a clean llms.txt genuinely helps. For everyone else chasing AI visibility, it’s a low-stakes bet. Maybe 30 to 60 minutes of work, no harm done, and probably no payoff either. That’s the honest read, and the rest of this piece backs it up.
What llms.txt actually is (in 30 seconds)
It’s a plain-text file you drop at the root of your domain, like yoursite.com/llms.txt. Written in Markdown. An H1 with your name, a blockquote summarizing what you do, then sections of links to your most important pages with a one-line description each. Jeremy Howard proposed it in September 2024.
Think of it as robots.txt’s more helpful cousin. Where robots.txt tells crawlers what they can’t touch, llms.txt tells language models what’s worth reading first. That’s the theory, anyway. Whether models actually read it is where things get interesting.
What Google and the data actually say
Here’s where the “should I bother” question gets an evidence-based answer instead of a vibe.
Google’s official position
In May 2026, Google published its first official guide on optimizing for AI features in Search. One month later, on June 15, 2026, it added a section specifically about llms.txt. The SEO community was confused enough that Google felt the need to clear it up directly. The wording was blunt for Google: llms.txt has never been a Search ranking input, and it has zero effect on visibility in Google Search or AI Overviews.
Google’s John Mueller went further. He called the file speculative for now and pointed people toward WebMCP instead (more on that below). He’s also compared llms.txt to the old keywords meta tag, the one everyone eventually stopped using, noting that AI services don’t even check for it in server logs.
The independent studies
The data lines up with Google’s position. SE Ranking analyzed 300,000 domains and found no correlation between having an llms.txt file and getting cited by AI. In their test, the model actually did slightly better without it.
Then there are the server logs, which are brutal. One analysis by researcher Kai Spriestersbach found that out of 62,100 AI bot requests, exactly 84 went to the llms.txt file. That’s 0.1%. The only bot reliably fetching it was BuiltWith, a service that catalogs which files exist rather than one that uses them. So the file was mostly talking to itself.
Why the gap between the promise and the reality? Mueller’s own objection is the clearest explanation. A model that has already crawled and downloaded your real pages has no particular reason to trust a separate file describing them. And there’s a trust problem baked in: nothing stops a site from listing one polished set of pages in llms.txt while serving something else to actual visitors. That’s cloaking, just aimed at language models instead of search crawlers. A signal that’s trivial to game is a signal engines learn to ignore. Which is roughly where we are.
When llms.txt IS worth building
Now the other side, because “it does nothing” is too blunt.
There’s one setting where llms.txt earns its keep: developer-facing content. API references, SDK docs, technical guides. AI coding assistants like Claude Code and Cursor parse documentation constantly, and a curated Markdown map saves them from wading through cluttered HTML. This isn’t hypothetical. Anthropic explicitly recommends llms.txt in its guidance for writing for agents, OpenAI uses it for its Agents SDK, and adoption is concentrated among the Stripes, Cloudflares, Vercels, and Supabases of the world. Developer tools, basically.
The reason it works here and nowhere else is worth understanding, because it tells you whether your own site qualifies. Coding agents are already visiting docs — that’s the whole job. When a file has a reader that shows up anyway, a clean index genuinely saves tokens and cuts down on the model guessing at your API. The file has a consumer. For a plumber’s website or a fashion ecommerce store, no agent is sitting there parsing your pages for fun, so the same file points at an empty room. The value isn’t in the file. It’s in whether anything reads it.
So the real question isn’t “does llms.txt work” in the abstract. It’s “does it work for a site like mine?” Here’s the quick version:
| Your site | Build an llms.txt? |
|---|---|
| Developer docs, API refs, SDKs | Yes. This is the genuine use case |
| SaaS with a real docs section | Optional. Worth it if AI agents already visit your docs |
| Ecommerce, publisher, local business chasing Google | Skip it, or do it last. Your effort belongs elsewhere |
If you’re in that bottom row, the file isn’t hurting you. It’s just not the thing that’ll move your numbers.
llms.txt vs WebMCP — where Google is actually investing
Here’s the part most llms.txt coverage misses while it argues about a text file.
If Google isn’t betting on llms.txt, where is it betting? On WebMCP, the Web Model Context Protocol. It showed up in Chrome Canary in February 2026, got featured at Google I/O 2026, and is running an origin trial in Chrome 149. It was published as a W3C Draft Community Group Report on February 10, 2026, built by engineers at Google and Microsoft.
The difference matters more than the acronyms suggest. llms.txt describes a site. It’s a static map an agent reads to orient itself. WebMCP lets an agent operate a site, declaring functional tool contracts in HTML or JavaScript so an agent can actually do things, like check availability or fill out a form, on a live session. One hands the agent a menu. The other hands it the kitchen.
That distinction is exactly why it’s the more interesting bet for anyone thinking past this quarter. An agent browsing on a user’s behalf doesn’t just want to know your pricing page exists. It wants to filter your catalog, add something to a cart, book a slot, submit a form. A flat text file can’t offer any of that. A declared tool contract can. So the moment agentic browsing goes from demo to default, the sites that already speak that language are the ones agents can actually transact with.
You can see Google’s hand in Chrome’s own tooling. Lighthouse 13.3 added an experimental Agentic Browsing audit that checks WebMCP registration, accessibility, layout stability, and llms.txt presence too. Notice that llms.txt made the list — it’s not being written off entirely. But it’s one line item among several, and the ones above it are about function, not description. That ordering is the tell. As agents start doing tasks instead of just reading, a functional interface is a more durable investment than a flat descriptive file sitting quietly at your root.
None of this means you need to ship WebMCP tomorrow. It’s still early — an origin trial, not a finished standard. The point is calibration: if you were about to pour a weekend into a perfect llms.txt because it felt like the future, that instinct is aimed one layer too shallow. Keep an eye on WebMCP, and spend the weekend on your actual content instead.
If you’re building one anyway (the fast version)
Decided your site’s in the “yes” or “optional” camp? Fine. Here’s how to build one that actually has a job to do, rather than an auto-generated sitemap wearing a new name.
Curate hard. List 10 to 40 pages that answer real questions, like docs, pricing, and key guides, not every blog post you’ve ever published. Write a real one-sentence description for each link, because that’s the metadata a model uses to judge relevance. Make sure the destination pages are clean and answer-first, since the file only points; the pages do the work. Got deep documentation? Add an llms-full.txt with the full text. Then check your server logs monthly for GPTBot, ClaudeBot, and PerplexityBot. If they’re reading it, expand it. If not, you’ve lost an hour and learned where to spend instead.
What actually earns AI citations
If llms.txt is a tiny slice of the job, what’s the rest? The same fundamentals that have driven good SEO for years, with a couple of AI-era twists.
Topical authority comes first. AI engines favor sources that cover a subject thoroughly, not one clever post floating alone. A cluster of interlinked, genuinely expert content beats a single optimized page. This is the same logic behind how SEO for LLMs works generally: depth and coverage over tricks.
Then entity clarity. Name your brand, product, people, and location explicitly. Models use named entities as anchors when they build citations, and they recommend brands they can verify from multiple places.
Structure matters too. Question-style H2s, clear hierarchy, specific factual claims. AI extractors pull from headers and clean paragraphs, not dense walls of text. If you want a sense of what kinds of content LLMs actually cite, structure is a recurring theme.
And then freshness, the one that quietly decides everything. Perplexity and other engines retrieve from the live web, and they lean toward recently updated pages. Outdated content gets deprioritized even with a strong backlink profile. That’s the trap. A page that ranked well two years ago can bleed traffic and citations without anyone noticing, because the data inside it went stale. This is content decay, and it’s exactly the problem WordPattern was built to catch. It watches Google Search Console for pages losing momentum and flags what needs a refresh before the damage compounds. Keeping your best pages current with fresh statistics and data does more for AI visibility than any file at your root. Weighing tools for this? Our rundown of generative engine optimization tools covers the landscape.
Conclusion
So, does llms.txt work? Only as part of a system, and only for the right site. Publishing a text file doesn’t make you citable any more than printing a menu makes the food taste better. For developer docs, it’s a reasonable, low-cost addition. Build it and move on. For most other sites chasing AI visibility, it’s optional at best, and the time is better spent on freshness, structure, and authority.
The web is genuinely shifting toward machine readers. That instinct is right. Just don’t mistake a static file for the strategy. The sites that get cited in 2026 are the ones keeping their best work sharp and current, file or no file.
FAQs
No. Google confirmed in its 2026 AI guidance that llms.txt is not a Search ranking input and has no effect on Google Search or AI Overviews. John Mueller has compared it to the old keywords meta tag and said AI services don’t even request the file in server logs. Treat it as a low-cost bet, not a ranking tactic.
No, they’re near opposites. robots.txt tells crawlers what they may not access, and crawlers respect it as an enforcement standard. llms.txt is a voluntary suggestion telling AI systems what to read first. There’s no enforcement and no formal commitment from Google or OpenAI to consume it.
Not on its own. Citations come from entity strength, answer-first content, third-party mentions, and freshness. The file only makes your content slightly easier to ingest if an agent already visits you. A study of 300,000 domains found no citation uplift from having the file at all.
Probably not first. If you run a local or ecommerce site chasing Google and AI visibility, your time is better spent on content quality, freshness, and clear entity signals. The file does no harm, so add it if a plugin generates one for free. Just don’t treat it as a growth lever.
