If you’ve spent any time in an SEO forum or on LinkedIn this year, someone has told you to add an llms.txt file before you fall behind on AI search. It’s a reasonable-sounding pitch: hand AI models a clean, curated map of your site instead of letting them guess at what matters. At Indexed Zone SEO, we field this question on nearly every technical audit call now, and the honest answer is more nuanced than the hype suggests. This guide covers what an llms.txt file actually does, what the current adoption and crawler data shows, and a practical framework for deciding whether your site needs one.
What Is an llms.txt File, Exactly?
An llms.txt file is a plain Markdown document you publish at yourdomain.com/llms.txt. Think of it as a table of contents written for a machine reader instead of a human one: a short, curated list of your most important pages, each with a one-line description, so an AI model doesn’t have to wade through navigation menus, ads, and JavaScript to figure out what your site is actually about.
The idea came from Jeremy Howard, co-founder of Answer.AI and fast.ai, who proposed adding an llms.txt file to websites designed for reading by language models, not just humans, describing it as a file that outlines the information a model may want to retrieve when assembling context for prompts related to that site. The original problem he was solving was narrower than most people realize: context windows were too small to hold entire websites, and converting cluttered HTML into clean text for a model to read was slow and error-prone, particularly for developers who needed AI tools to reference API documentation quickly.
Here’s a simple llms.txt example:
# Acme Docs
> Acme is a developer platform for building payment workflows. This file
> helps AI models find our most important documentation quickly.
## Docs
– [Getting Started](https://acme.dev/docs/start): Install the SDK and make your first API call
– [Authentication](https://acme.dev/docs/auth): API keys, OAuth, and token refresh
– [Webhooks](https://acme.dev/docs/webhooks): Event types and payload formats
## Optional
– [Changelog](https://acme.dev/changelog): Recent API changes
Some sites also publish a companion llms-full.txt, a longer version that includes the full text of key pages rather than just links, giving a model everything in one fetch instead of requiring it to follow each link separately.
llms.txt vs. Robots.txt: They’re Not Solving the Same Problem
This is where most of the confusion starts. robots.txt is a decades-old, universally respected standard that tells crawlers which URLs they’re allowed to request. It’s a permissions file: allow this, disallow that.
llms.txt doesn’t grant or deny access to anything. It fills a gap robots.txt can’t address: robots.txt controls crawler access at the URL level but says nothing about which pages matter most or how a brand should be described, so llms.txt adds editorial curation on top of access control rather than competing with it. In practice, that means:
- robots.txt answers: “Are you allowed to crawl this page?”
- llms.txt answers: “If you’re going to read our site, here’s what actually matters and why.”
You need robots.txt regardless of what you decide about llms.txt. It’s still your main tool for AI crawler control, since it’s what determines whether you allow or block bots like GPTBot, ClaudeBot, or Google-Extended in the first place. llms.txt only matters once a model has already decided to visit.
The Data: Does llms.txt for AI Search Actually Work?
This is the part most llms.txt guides gloss over, and it’s the part that should actually drive your decision.
Adoption Is Still Low
In the biggest look at this so far, Ahrefs checked 137,000 sites and found that 28% publish an llms.txt file. That sounds like real momentum, until you look at who those sites are: Ahrefs’ own analytics users skew technical and SEO-aware, so 28% is the ceiling, not the norm for the web. The more telling number comes next. In May 2026, 97% of those llms.txt files got zero traffic. Nothing fetched them at all. Of the small share that saw any requests, 96% came from bots, not the AI assistants the file is meant to feed. Two years after Jeremy Howard proposed llms.txt, plenty of sites have added one, and almost nothing is reading it.
Major AI Crawlers Mostly Aren’t Fetching It
One firm that monitors AI bot traffic at scale reported that across more than 500 million logged crawler visits, GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, and Google-Extended overwhelmingly skipped the file and crawled HTML pages directly instead. Only 408 requests targeted llms.txt directly across a 90-day window.
Google Has Said It Doesn’t Use It
Google’s Gary Illyes confirmed the company doesn’t support llms.txt and has no plans to, and John Mueller has compared it to the discontinued keywords meta tag, the same fate that befell a signal that once mattered until search engines simply stopped looking at it. Separately, Mueller has told site owners directly that no AI crawler has claimed to extract information via llms.txt.
No Measurable Citation Lift
A machine-learning model built to predict which domains get cited in AI answers actually improved in accuracy when the llms.txt variable was removed, meaning the file’s presence added noise rather than useful signal to the prediction.
None of this means llms.txt is a scam or a wasted afternoon. It means the specific claim driving most of the hype, that publishing the file gets you cited more often in ChatGPT or AI Overviews, isn’t supported by the data available right now. In our experience running technical audits, the sites asking about llms.txt are almost always really asking how to get cited more by AI search, and llms.txt is rarely the lever that answer needs.
So, Do You Need One? A Practical Framework
The right answer depends heavily on what kind of site you run, not on industry hype cycles.
You should probably build one if:
- You run developer-facing tools, an API, or technical documentation. AI coding assistants like Cursor, Windsurf, and Claude Code already use llms.txt-style files as a starting map when developers ask them to work against your docs. This is closer to the file’s original intended use case than general AI search visibility.
- You have a large, messy site where a curated index genuinely helps orient any reader, human or machine.
- Implementation cost is genuinely low for you (a developer’s afternoon), and you want the file in place on the off chance a major provider starts weighting it later.
You can deprioritize it if:
- You’re a local business, ecommerce store, or service company whose main goal is showing up in AI Overviews or chatbot answers for commercial queries. The data above suggests this isn’t where llms.txt earns its keep.
- Your resources are better spent on the content and structural work that does correlate with AI citation: clear, well-organized answers to real questions, accurate and specific information, and pages that are easy for any crawler, human-built or AI, to parse without a curated shortcut.
- You’re weighing it against higher-impact technical SEO fixes, like crawlability issues, thin content, or missing schema markup.
How to Create an llms.txt File
If you land on “worth building,” the process is straightforward:
- Inventory your priority pages. Pull together the 10–30 pages you’d want an AI model or agent to see first: core documentation, pricing, key product pages, or foundational guides.
- Write one-line descriptions. For each link, write a plain-language summary of what the page covers and why it matters. Skip marketing language; describe the content the way you’d describe it to a colleague.
- Structure it in Markdown. Start with an H1 for your site or product name, a short blockquote summary underneath, then H2 sections grouping your links logically (Docs, Guides, Pricing, and so on).
- Host it at the root domain. The file needs to live at yourdomain.com/llms.txt as plain text, not behind a redirect or a JavaScript-rendered page.
- Consider an llms-full.txt for documentation-heavy sites. If you have extensive technical docs, a full-text companion file can save an AI agent multiple fetches.
- Monitor your server logs. Check whether anything is actually requesting the file. This tells you, for your specific site, whether the investment is paying off in real crawler behavior rather than theory.
What Actually Moves the Needle for AI Search Visibility
If AI citation and traffic are the real goal, the more reliable way to optimize your site for LLMs sits outside llms.txt entirely. Generative search systems tend to cite sources that provide clear facts, original data, and verifiable expertise, and they pick up on consistent semantic structure, including clean heading hierarchies and well-organized entities, during the regular crawling process, without needing a separate file to point the way. That means the fundamentals haven’t changed much: answer real questions directly, structure pages so the key answer isn’t buried under three paragraphs of preamble, keep your robots.txt current for AI bot user-agents, and make sure your site is technically crawlable in the first place. That’s the work we spend most of our time on with clients, and it’s a better use of a marketing team’s hours than debating a file that most AI crawlers currently skip.
FAQ
Does adding llms.txt improve my SEO or AI search rankings?
Not based on current evidence. A large-scale study testing whether llms.txt presence correlates with AI citation frequency found the file added noise rather than a genuine signal to the prediction. It’s not a ranking factor, and Google has said directly that it doesn’t use it.
Do ChatGPT, Claude, and Perplexity actually read llms.txt files?
Crawler traffic data shows they mostly don’t, at least not yet. Major AI bots overwhelmingly fetch and parse regular HTML pages rather than requesting the llms.txt file, even on sites that publish one. AI coding tools that operate inside a developer’s editor are the more consistent users of llms.txt-style files today.
What’s the difference between llms.txt and robots.txt?
robots.txt controls crawler access: it tells bots which URLs they can or can’t request. llms.txt doesn’t control access at all; it’s a curated content index meant to help a model that’s already decided to visit understand what matters most on your site. You need robots.txt regardless of what you decide about llms.txt.
Should every website have an llms.txt file?
No. It’s most useful for developer tools, API documentation, and technical sites that AI coding agents actively reference. For most business and commercial sites, the time is better spent on content quality, page structure, and technical crawlability, which have a clearer connection to AI visibility.
Will publishing an llms.txt file hurt my SEO?
There’s no evidence it causes harm. The main tradeoff is that a tidy, curated list of your best content can make it marginally easier for scrapers or competitors to find and copy it. For most sites, that risk is minor and doesn’t outweigh the low cost of publishing the file if you decide it’s worth doing.
So, to add, or not to add llms.txt?
llms.txt isn’t a scam, and it isn’t a growth hack either. It’s a low-cost, low-certainty bet: genuinely useful for developer-facing sites and AI coding workflows, unproven for general AI search visibility, and not something Google or the major AI crawlers currently rely on. If you run documentation or dev tools, build one. If you’re chasing AI Overviews or chatbot citations for a typical business site, your time is better spent on the content and technical fundamentals that the data actually supports. If you’re not sure which category you fall into, or want a second opinion before you spend a developer’s afternoon on it, talk to our team at Indexed Zone SEO. We can walk through what’s actually worth prioritizing for your site’s AI visibility.
