llms.txt: what it is and whether your site needs it
llms.txt is a text file some sites publish at the root of their domain to give language models an ordered summary of their content. It is often described as "robots.txt for AI". It is not, and Google has been clear that Search does not use it. Here is what is known and what deserves your time.
Key takeaways
- llms.txt is a proposal, not a standard adopted by search engines.
- Google states no AI-specific files are needed to appear in Search or its generative AI features.
- It may make sense for technical documentation consumed by coding assistants and agents.
- Before spending time on it, settle the basics: crawling, indexing and useful content.
What the file is
A Markdown file at `/llms.txt` that briefly describes the site and links to its most important pages, often in clean text versions. The idea appeared in 2024 as an open proposal: models have a limited context window, and a page's HTML carries menus, scripts and other material that adds nothing. A curated index would make reading easier.
How it differs from robots.txt and sitemaps
| File | Purpose | Who uses it |
|---|---|---|
| robots.txt | Tell crawlers what they may crawl | Search engines and most known crawlers |
| sitemap.xml | List URLs you want discovered | Search engines |
| llms.txt | Offer a summary and selected links to language models | Varies by service; not used by Google |
Swipe the table sideways to see all columns.
The key difference: robots.txt gives access instructions; llms.txt only offers information. It blocks nothing and permits nothing.
What Google says
The Search Central guide on optimising for generative AI features states that you do not need to create machine-readable files, AI text files, special markup or Markdown to appear in Google Search, including its generative AI features, because Search does not use them. The AI features documentation adds that there are no additional requirements to appear in AI Overviews or AI Mode.
Google also notes that site owners are free to maintain such files for other services that do use them. In short: it does not help at Google, and having one does not hurt there either.
What about other AI assistants?
Here the picture is thinner. Some companies publish an llms.txt on their own documentation sites, and some coding tools and agents read it when present. We have not found official documentation from the major assistants confirming they use it to choose which sources to cite. Treat it as an experiment, not a visibility lever. What is documented for ChatGPT is in the article on ChatGPT SEO.
When it may make sense
- Technical documentation or API references developers query through coding assistants.
- Large knowledge bases, where a curated index helps an agent reach the right page.
- Products with their own agents that consume the site's content.
For a services site, a local business or a company blog, the demonstrated benefit today is small.
How to create one, if you decide to
- Create a text file called `llms.txt` at the root of the domain.
- Open with a title (`# Site name`) and a one- or two-sentence description.
- Add sections (`## Services`, `## Guides`) with link lists and one explanatory line each.
- Include only pages that are already public and indexable. It is not a place for hidden content.
- Keep it short and update it when the site structure changes.
- Check it loads with a 200 status code as plain text.
Do not use it to stack keywords: it is a declarative file anyone can write in their own favour, which is exactly why systems have little reason to trust it.
What matters more
- Allow crawling for the robots you want, with no accidental block at the CDN.
- Check indexing with the technical SEO checklist.
- Content as text, with clear answers at the start of each section.
- Your own information: data, cases and experience not found elsewhere.
- Structured data consistent with what is visible.
- Presence off your site: reviews, mentions, current profiles.
Common mistakes
- Believing llms.txt controls AI crawler access; robots.txt does that.
- Expecting better Google rankings from it.
- Publishing different content there from what visitors see.
- Mirroring the whole site in Markdown while neglecting the real pages.
Frequently asked questions
Does llms.txt improve Google rankings?
No. Google states Search does not use this type of file, including for generative AI features.
Does it block AI crawlers?
No. It is purely informational. Access is managed through robots.txt and your server or CDN settings.
Can having one hurt me?
According to Google, no. You may maintain it for other services that use it.
Do ChatGPT and other assistants read it?
We have found no official confirmation for source selection. Some coding tools and agents do fetch it.
Should I create one now?
Only if you have technical documentation or a large knowledge base. Otherwise prioritise crawling, indexing and content.
Sources
About this article. Produced by the MetaEX team with AI assistance, from the official documentation listed under Sources. Examples marked as illustrative are not client data. Last updated on 11 November 2026.
Want a read on your digital operation?
Talk to us: we first understand your situation, then tell you what makes sense.
Book a meeting