llms.txt, Explained: What It Is, Whether It Works, and When You Need One

By Minel Gunesoglu — I build Is My Brand in AI, a tool that tracks how brands show up across ChatGPT, Perplexity, Gemini and Google AI Overviews. I added an llms.txt file to this site, submitted it to the public directories, and have been watching the server logs and engine answers ever since. Last updated June 2, 2026.

TL;DR: llms.txt is a Markdown file you place at your site's root that gives AI models a curated map of your most important content. It was proposed by Answer.AI's Jeremy Howard in 2024 and is documented at llmstxt.org. It is advisory, not enforced, and as of 2026 most AI engines barely read it. So it's worth the effort for documentation and developer sites but usually skippable for small brochure sites.

You have probably seen the advice ("add an llms.txt file so ChatGPT can find you"), then, a week later, the opposite: "llms.txt does nothing." Both camps are partly right. This page settles it: what the file actually is, a real annotated example, the public evidence on whether AI engines read it, and a plain rule for whether you need one. The honest answer is more useful than the optimistic one.

What is llms.txt?

llms.txt is a plain-text Markdown file that lives at the root of your site, at yoursite.com/llms.txt. Its job is to hand large language models a short, curated map of your most important pages, in language a machine can parse cleanly.

The format was proposed by Jeremy Howard, co-founder of the research lab Answer.AI, in September 2024, and the full specification lives at llmstxt.org. It solves a real problem: when an AI model reads a modern web page, it wades through navigation menus, cookie banners, ads, and JavaScript before reaching the content. A llms.txt file skips that noise and points the model straight at the pages you most want understood.

The structure is deliberately simple:

An H1 line with your site or project name.
A blockquote with a one- or two-sentence summary of what the site is.
One or more sections of links, each followed by a short note on what that page covers.

That's the whole format: human-readable and machine-readable at once, which is the point.

One thing matters most: llms.txt is advisory. It is a suggestion to AI models, not a command: a curated reading list you offer. Whether a given engine picks it up, and most do not yet, is up to that engine. Hold that thought; it shapes the rest of this page.

llms.txt vs robots.txt (and sitemap.xml)

The most common mix-up is treating llms.txt like robots.txt. They do different jobs. robots.txt is a gate: it tells crawlers where they may and may not go. llms.txt is a map: it highlights the content you want read. sitemap.xml sits in between: a complete machine-readable inventory of your URLs, built for search-engine crawlers rather than language models. Here is the side-by-side:

File	Purpose	Audience	Enforced?	Format
llms.txt	Curated map of your best content for AI models	Large language models (ChatGPT, Perplexity, Gemini)	No — advisory suggestion	Markdown
robots.txt	Allow or block crawler access to paths	Search and AI crawlers	Yes — widely honored convention	Plain-text directives
sitemap.xml	Full inventory of every indexable URL	Search-engine crawlers	No — a discovery aid	XML

The takeaway: llms.txt does not replace either file. Keep robots.txt for access and sitemap.xml for search crawlers, and optionally add llms.txt to guide AI models toward your strongest pages.

An llms.txt example (annotated)

The fastest way to understand the file is to read one. Here is a short, well-formed example for a fictional documentation site, with notes on each part:

# Acme Docs

> Acme is an open-source toolkit for building data pipelines.
> This file points AI models to the docs that matter most.

## Core documentation

- [Getting started](https://acme.dev/docs/start.md): Install Acme and run your first pipeline.
- [API reference](https://acme.dev/docs/api.md): Every public function, with arguments and return types.
- [Configuration](https://acme.dev/docs/config.md): All settings and environment variables.

## Guides

- [Deploy to production](https://acme.dev/docs/deploy.md): Production checklist and scaling notes.

## Optional

- [Changelog](https://acme.dev/docs/changelog.md): Release history.

Reading it top to bottom:

The H1 (# Acme Docs) names the project (required).
The blockquote is your elevator pitch: one or two concrete sentences telling a model what the site is.
The ## sections group related links. The spec gives Optional a special meaning — links under an Optional heading can be skipped by a model short on space.
Each link points to a clean page (ideally a Markdown version), followed by a colon and a plain-language description.

The links point to .md files by convention: many docs platforms publish a Markdown copy of each page so models get content without the page chrome. Optional, but it helps.

You don't need to hand-write this. We built a free tool that scans your site and produces a valid file for you. You can generate a valid llms.txt free, no signup and edit it from there. This page stays focused on the concept; the tool handles the building.

Does llms.txt actually work?

The "does llms.txt work" question started the 2026 arguments, so let's be straight. The honest summary: llms.txt is easy to adopt, but the evidence that AI engines actually read it is thin. Adoption has raced ahead of consumption. Here is what the public record shows as of mid-2026:

Server-log tests show near-zero AI-bot traffic on the file. Semrush ran its own crawler-log analysis and reported that AI bots were essentially not requesting /llms.txt. It sat on the server while the major engines ignored it. Several independent log studies in early 2026 reached the same conclusion.
Google says it does not use it for ranking. John Mueller has commented publicly that, to his knowledge, no AI system was using llms.txt, comparing it to the long-dead keywords meta tag: present, but unread. Google has not announced support in Search.
No major engine has confirmed it reads the file for answers. Neither OpenAI, Anthropic, nor Google has documented that their answer engines fetch llms.txt at query time. Some agentic and developer tools do read it; consumer answer engines, by and large, do not yet.
But the signal is being formalized. In May 2026, Google added a check for llms.txt to the Chrome Lighthouse "Agentic Browsing" audit (more below). That's not consumption, but it's the first official nod the format has received.

So is it useless? No. It's early. The file does real work in two places today: documentation and developer tools that explicitly support it (where it shortens the path to your content), and as low-cost insurance for the day consumer engines start reading it. What it does not do today is move you up ChatGPT's answers. If someone says a llms.txt file will get your brand cited, they are ahead of the evidence.

A note on our own work, since you'll see this claimed both ways online and I want to be honest about what we have and haven't measured. Is My Brand in AI tracks how brands appear across ChatGPT, Perplexity, Gemini and Google AI Overviews. We are running an ongoing study to isolate whether adding a llms.txt file changes how often a site gets surfaced, engine by engine — methodology and early reads are coming, and I'll update this page as the data lands. I won't hand you a tidy percentage I can't yet stand behind. The public log studies above are the best evidence right now, and they all point one way: adoption high, consumption low.

Does ChatGPT use llms.txt?

There is no public confirmation that ChatGPT reads llms.txt to build its answers. OpenAI's crawlers fetch normal web pages and respect robots.txt, but OpenAI has not documented llms.txt support. Influencing ChatGPT today works the same as for any answer engine: clear, well-structured pages with genuine information a model can extract. A llms.txt file may help an agent that already supports it navigate your site, but it is not a switch that turns on ChatGPT visibility. For the full picture, see how to get cited by ChatGPT.

llms.txt and SEO / GEO (not a shortcut)

llms.txt is sometimes sold as a search shortcut. It isn't one. It carries no ranking weight in Google Search, and no engine rewards its mere presence. Where it fits into generative engine optimization is more modest: a tidy, machine-friendly summary of your best content, sitting alongside the real work: pages that answer questions well, structured so models can lift a clean passage. Treat it as good hygiene that may pay off as engines mature, not a lever you pull for traffic. The bigger picture on AI citations does not run through this one file.

llms.txt vs llms-full.txt

There are two files in the spec, and the difference trips people up. llms.txt is the index: the short curated list of links above. llms-full.txt is the full text: one large file with the content of your key pages concatenated, so a model can ingest everything in a single fetch without following links.

Use llms.txt for a lightweight map. Use llms-full.txt when you want your whole documentation set in a model's context window in one request — popular with developer-tool docs, since a user can paste the entire file into an assistant. The trade-off is size: llms-full.txt can get very large, and a smaller context window may not swallow all of it. Many sites publish both.

llms.txt on WordPress

On WordPress you don't need to touch the server by hand. Several plugins now generate and serve a llms.txt file for you, pulling from your published pages and refreshing it as you add content. The convenience is double-edged: an auto-generated file often lists everything rather than a curated few. If you use a plugin, prune the output so the file points at your genuinely important pages, not your whole archive. A focused map is the whole value.

Chrome Lighthouse "Agentic Browsing" audit (May 2026)

The most concrete development of 2026: in May, Google added a llms.txt check to the Chrome Lighthouse "Agentic Browsing" audit. Lighthouse is the open auditing tool built into Chrome's developer tools; the new "Agentic Browsing" category scores how ready a site is for automated AI agents, and one of its checks looks for an llms.txt file served at your root.

This matters for one reason: it's the first time llms.txt moved from a community proposal toward an officially recognized, testable signal. A Lighthouse check is not the same as an engine consuming the file in production. It is also a light check: it reports whether an llms.txt file is being served at your root, but a missing file is marked "not applicable" rather than failed, and the audit does not validate the file's structure. Even so, none of the major guides on this topic have caught up to it yet, so if your team runs Lighthouse, expect the line item and know what it means.

How to create and host an llms.txt file

Creating the file is straightforward. The work is in curating it well, not in the syntax:

List your most important pages. Not every page, just your best ten or twenty. Docs, key product pages, your strongest guides. If a model could read only a handful of your pages, which would you pick? Those go in.
Write the file in Markdown. Start with an H1 (your site name), add a blockquote summary, then group the links under ## sections with a short description after each, as in the example above.
Save it as llms.txt and upload it to your site root so it resolves at yoursite.com/llms.txt, the same level as robots.txt. On most hosts that means your public web directory; on WordPress, a plugin handles placement.
Verify it's live. Open yoursite.com/llms.txt in a browser. You should see raw Markdown served as plain text. A 404 or your site's HTML template means the file is misplaced or your host is rewriting the URL.

If hand-writing Markdown isn't your idea of a good afternoon, our free tool does steps one and two. Point it at your site and it returns a clean, valid file to review, then host via step three. You can create your llms.txt there. For inspiration, Anthropic, Cloudflare, and Vercel all publish llms.txt files you can view at their roots.

Do you actually need one? (decision tree)

Here is the honest decision rule. Skip the guesswork:

You run documentation, developer tools, or an API. Yes, add one. This is where llms.txt does real work today. Users genuinely paste docs into AI assistants, agentic tools in your ecosystem may read the file, and a curated map (or a llms-full.txt) shortens the path to your content.
You publish a large, content-heavy site (a knowledge base, big blog, or SaaS with deep docs). Probably yes. Curation forces you to identify your best pages, and you're ready when consumer engines start reading the file. Low cost, plausible upside.
You run a small brochure site (a few pages, a simple landing site, a portfolio). Skip it, or treat it as a five-minute nice-to-have. With near-zero consumption today and only a handful of pages, the file adds little a model couldn't grasp from your homepage. Spend the time on clear page content.
You're doing it purely to rank in ChatGPT. Stop. Wrong tool. Nothing in the current evidence supports that. Put the effort into pages that answer real questions well.

The short version: high value for docs and developer sites, low value for small brochure sites, never a substitute for good content. Between categories, "yes, but keep it curated" is a fine default.

Frequently asked questions

Is llms.txt an official standard? No. llms.txt is a community proposal published by Answer.AI's Jeremy Howard in 2024, documented at llmstxt.org. No standards body has ratified it and no AI company is obligated to support it. Adoption is voluntary on both sides — sites choose to publish it, engines choose whether to read it.

Where do I put the llms.txt file? At the root of your site, so it is reachable at yoursite.com/llms.txt, the same level as robots.txt and your sitemap. If it resolves anywhere deeper, models following the convention won't find it.

Does llms.txt help with Google rankings? No. Google has said it does not use llms.txt to decide rankings, and John Mueller has publicly likened its status to the unused keywords meta tag. It carries no weight in Google Search today.

Is llms.txt the same as robots.txt? No. robots.txt controls crawler access: an enforced gate. llms.txt is an advisory content map for AI models, not access control. Different purposes; you can run both.

Do I need llms-full.txt as well? Only if you want models to ingest your full content in one fetch. llms.txt is the lightweight index of links; llms-full.txt is the complete concatenated text. Documentation sites often publish both; most others are fine with just llms.txt.

Will llms.txt get my brand cited in ChatGPT? No public evidence says it does today — no major answer engine has confirmed it reads the file to build responses. The reliable path to being surfaced is clear, well-structured content a model can extract; that's the bigger picture on AI citations.

llms.txt is a small, honest idea: hand AI models a clean map of your best content. In 2026 it is early: widely adopted, barely consumed, freshly recognized by Google's Lighthouse audit, but not yet a lever for visibility. Add one if you run docs or a developer tool, keep it curated, and don't expect it to do work that good content does. When the engines start reading it in earnest, you'll be ready, and I'll update this page the moment our cross-engine tracking has something solid to report.

This guide is part of our series on how to rank on ChatGPT and AI search visibility. Written and maintained by Minel Gunesoglu (LinkedIn), founder of Is My Brand in AI. Reviewed June 2, 2026; updated monthly as the evidence changes.