Do any AI companies actually read llms.txt?

Not officially. As of mid-2026, no major vendor — OpenAI, Anthropic, Google, or Meta — has committed to consuming it, and Ahrefs found 97% of published llms.txt files received zero requests in a month. The exception is coding assistants like Cursor and Claude Code, which fetch it from documentation sites when developers point them there.

Should I block AI crawlers instead of courting them?

For most startups, no. Blocking search crawlers like OAI-SearchBot, Claude-SearchBot, and PerplexityBot removes you from AI assistants' cited answers — exactly where B2B buyers increasingly research vendors. Blocking training bots is a defensible choice for content publishers, but a company whose problem is obscurity has more to lose than to protect.

Is llms.txt the same as robots.txt?

No. robots.txt controls which crawlers may access your pages and is verifiably fetched by AI bots every day; llms.txt merely suggests which pages are worth reading, and on current evidence almost nothing fetches it. If you only maintain one of the two files carefully, make it robots.txt.

What should a startup put in its llms.txt file?

Your company name as a heading, a one-line factual description, then five to ten key pages — product, pricing, comparisons, docs — each with a one-sentence summary. Write the summaries as concrete facts rather than marketing slogans, keep the whole file under a page, and host it at your domain root.

llms.txt: What It Is and Whether It Works (2026)

llms.txt has become the "one weird trick" of AI search. It shows up in every checklist, agencies sell it as a deliverable, and Shopify quietly switched it on for every store in the spring of 2026. It's also the rare tactic where the skeptics have server logs and the believers have vibes. This post lays out both sides with the receipts, then shows you the version of the 15-minute job that's actually worth doing.

What is llms.txt?

llms.txt is a plain markdown file at your domain root that offers AI systems a curated map of your site: your key pages, each described in one line. The idea, proposed by Jeremy Howard of Answer.AI in late 2024, is that language models work with small context windows and messy HTML, so a clean, human-written index at yourdomain.com/llms.txt would help them find and understand your best content.

Two things it is not, because the confusion is everywhere. It is not robots.txt: robots.txt controls which crawlers may access your pages, while llms.txt merely suggests which pages are worth reading — one is a lock, the other is a brochure. And it is not a ranking lever: no search engine or AI assistant has ever said the file influences whether you get ranked, cited, or recommended.

Does any AI system actually read llms.txt?

No — as of mid-2026, no major AI vendor, not OpenAI, Anthropic, Google, or Meta, has publicly committed to consuming llms.txt in production. That's the verified picture, and it hasn't budged since Google's John Mueller said it plainly in June 2025: "no AI system currently uses llms.txt" — he compared it to the keywords meta tag, a signal sites supply and nobody consumes.

The server logs back him up. Ahrefs studied 137,000 domains and found that of the 38,000-plus publishing a valid llms.txt, 97% received zero requests for the file in May 2026 — no bots, no humans, nothing. A separate crawl of roughly 300,000 domains found no statistically significant correlation between having the file and how often AI assistants cite you. GPTBot, ClaudeBot, and PerplexityBot skip it and crawl your HTML directly, exactly as they always have.

There is one honest exception: coding assistants. IDE agents like Cursor and Claude Code do fetch llms.txt when a developer points them at a documentation site, which is why Stripe, Vercel, Cloudflare, and Anthropic all publish one. If your product has developer docs, the file has a real, if narrow, audience today. For a marketing site, it does not.

So is publishing one a waste of 15 minutes?

Not quite. It costs 15 minutes, carries zero downside, and if any major assistant starts consuming the file, early publishers inherit the benefit for free. That's the whole hedge case, and it's worth stating as modestly as that: llms.txt is a cheap lottery ticket, not a strategy. Adoption on the publishing side keeps climbing — roughly one site in ten now has one — so if consumption ever follows, the standard is already lying in wait.

There's also a quiet side benefit. Writing the file forces you to name your ten most important pages and describe each in one sentence — an exercise that regularly reveals a site has no clear answer. Just hold the honest frame: publish it, forget it, and expect nothing. If someone is charging you real money for "llms.txt optimization," they are selling you a file that, on today's evidence, nothing reads.

What should a small B2B site put in llms.txt?

Keep it under one page: your company name as a heading, a one-line description of what you do, then your key pages with a one-line summary each. The format is ordinary markdown. For a small B2B site, the whole thing looks like this:

# Acme Analytics

> Acme Analytics gives seed-stage SaaS teams
> revenue dashboards without a data engineer.

## Key pages

- [Product](https://acme.com/product): What the
  dashboards do and who they're for
- [Pricing](https://acme.com/pricing): Plans from
  $49/month, no annual lock-in
- [vs. Looker](https://acme.com/compare/looker):
  How we differ from enterprise BI tools
- [Docs](https://acme.com/docs): Setup guides and
  API reference
- [About](https://acme.com/about): Who builds Acme
  and why

Save it as plain UTF-8 text at yourdomain.com/llms.txt. Prioritize the pages a buyer would need to evaluate you — product, pricing, comparisons, docs — over blog posts. Write the one-liners as facts, not slogans; "plans from $49/month" is useful to a model in a way "flexible pricing for growing teams" never will be. Fifteen minutes, done, move on.

What matters more in the same 15 minutes?

Your robots.txt. Unlike llms.txt, AI crawlers verifiably request robots.txt every day, and one careless block there removes you from AI answers entirely. This is the file with actual consequences, and plenty of startups have shot themselves in the foot with it — usually by copying a "block all AI bots" template in 2023 and forgetting about it.

The key fact: most AI vendors now run two kinds of crawlers, and they respect robots.txt rules separately. Training crawlers — GPTBot, ClaudeBot, Google-Extended — collect content for building future models. Search crawlers — OAI-SearchBot, Claude-SearchBot, PerplexityBot — fetch pages so assistants can cite you in live answers. Blocking a training bot keeps your content out of the next model's memory; blocking a search bot removes you from ChatGPT's citations today. (One wrinkle: Google's AI Overviews use the ordinary Googlebot, so Google-Extended only governs Gemini's training and grounding, not your visibility in AI Overviews.)

If you want AI visibility — and the numbers on where B2B buyers now research say you do — the policy for a startup is simple: allow them all. GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended. The IP-protection argument for blocking training bots is real for publishers who sell content; for a startup whose entire problem is being unknown, being absent from the models is the bigger risk. Check your robots.txt right now — it takes two minutes, and if a template blocked these bots, that single fix outweighs a hundred llms.txt files.

Where does llms.txt rank on a founder's priority list?

Third, distantly. Answer-shaped content is what earns citations, crawler access is what makes citations possible, and llms.txt is a hedge you write once and forget. In order:

Answer-shaped content

Pages that directly answer the questions your buyers ask AI assistants. This is 90% of the game — our full GEO guide covers how to write it.

Crawler access

A robots.txt that lets the AI search bots in. Two minutes to check, and unlike everything else on this list, getting it wrong is fatal.

llms.txt

The 15-minute hedge. Publish it, expect nothing, and let it sit there in case the standard ever gets its consumers.

The uncomfortable truth about AI search is that there's no shortcut file. Assistants recommend companies whose sites clearly answer real questions — that's the pattern across ChatGPT, and it's the same pattern for Claude and Gemini. llms.txt is a fine thing to do on a Tuesday afternoon. The content it points to is the actual work — the same answer-shaped content our Search specialist is built to keep producing.

llms.txt Takes 15 Minutes. Does It Actually Matter?