Main area
Agency
geo

llms.txt — what it is and what it does (not) deliver

Redaktion ·

llms.txt is a proposed format meant to give LLMs a curated, readable overview of a website’s most important content — a Markdown file at /llms.txt, intended as a signpost for AI systems. The idea sounds plausible and is being heavily promoted right now. The honest assessment up front: it is not a ratified standard, and to date there is no solid evidence that the major providers actually evaluate the file. This article explains what llms.txt is, where it comes from — and why you should weigh the cost-benefit soberly.

What llms.txt wants to be

llms.txt is a Markdown file at a domain’s root (https://example.com/llms.txt). Unlike most machine files, it deliberately uses Markdown instead of XML — a format that both humans and language models read well. The thinking: instead of feeding an LLM the full HTML page, often clogged with navigation, ads and boilerplate, llms.txt provides a curated short version with background and links to the most important content.

The structure is leanly prescribed:

  • An H1 with the name of the project or site — the only mandatory part.
  • An optional blockquote with a short summary.
  • Any number of H2 sections, each containing lists of links (with a brief explanation) to detailed Markdown versions of individual pages.

A variant llms-full.txt is also common, which doesn’t just link but embeds the full content directly — intended for cases where the model should ingest everything at once.

Origin: a proposal, not a resolution

The idea comes from Jeremy Howard (co-founder of Answer.AI), who proposed it in September 2024. Important for context: it was a proposal from an individual or a small team, not a resolution by a standards body like the W3C or IETF. The specification on llmstxt.org remains open and informal to this day, with development running through GitHub discussions and a Discord. That’s no flaw — robots.txt began similarly — but it is the crucial difference from the reality the AI providers operate in.

The honest status check

Here it gets uncomfortable, and it’s exactly the point most llms.txt guides stay silent on: no major LLM provider demonstrably evaluates the file.

  • Google does not support llms.txt and, by its own statements, doesn’t plan to. John Mueller publicly compared it to the long-ignored keywords meta tag and noted that no AI service had requested the file (as of late 2025).
  • OpenAI points site operators to robots.txt for crawler control. Server-log analyses show OpenAI crawlers don’t request /llms.txt during normal visits.
  • Anthropic, OpenAI and Perplexity do have an llms.txt of their own — but on their developer documentation sites, as a structured entry point for coding assistants into the API docs. The fact that these companies maintain a file of their own does not prove they consider your llms.txt in training or retrieval.

That’s the central confusion: seeing an llms.txt on docs.anthropic.com doesn’t mean your llms.txt on the company website feeds into any answer process. No standards body has ratified the format, no model provider has formally committed to evaluating it.

Distinguishing it from robots.txt and sitemap

llms.txt is often lumped together with two established files, but serves a different purpose:

  • robots.txt governs crawl permission — which bot may fetch which paths. It is access control, not a content offering. Anyone wanting to block or allow AI crawlers does it here (or via the LLM-crawler-specific user agents).
  • Sitemap serves discovery — it lists all indexable URLs so search engines miss nothing. It says nothing about content or importance.
  • llms.txt wants to provide curation — an editorially selected, readable overview of the most important content. It replaces neither robots.txt nor sitemap; it would be a third, complementary layer — if it were actually used.

Weigh the cost-benefit realistically

The file is cheap to create: for a manageable site you write it in an hour; for a documentation-heavy site it can be generated. The risk is low, the direct harm essentially zero. But “cheap” isn’t “useful”. As long as no major provider demonstrably reads the file, the expected benefit for AI visibility is speculative. A sensible stance: if you already have clean Markdown versions of your content, add an llms.txt as a cheap bet on the future — but don’t prioritise it over things with proven effect like structured data, clear content and robots.txt hygiene for AI crawlers.

FAQ

FAQ

Do ChatGPT, Claude or Google really use llms.txt?
As of late 2025/early 2026, there is no solid evidence for it. Google explicitly does not support it, OpenAI points to robots.txt, and no provider having its own llms.txt proves they evaluate yours. The benefit for AI visibility is currently speculative.
Is llms.txt an official standard?
No. It is an informal format proposed in 2024 by Jeremy Howard (Answer.AI). No standards body has ratified it, and the specification continues to evolve openly via GitHub and Discord.
Does llms.txt replace my robots.txt or sitemap?
No. robots.txt controls crawl permission, the sitemap serves URL discovery. llms.txt is meant to provide a curated content overview for LLMs — a different, complementary function; none replaces the others.
Does an llms.txt hurt if nobody reads it anyway?
Directly no — the risk is low. Google does recommend setting the file to noindex so it does not itself appear as a search result. The effort is small, but the expected benefit is currently unproven.
Where does llms.txt already deliver something today?
With coding assistants and documentation tools that deliberately load an llms.txt as a structured entry into API documentation when the user provides the URL. That is on-demand use — not automatic harvesting by search AIs.

Conclusion

llms.txt is a charming idea with an honest goal: give AI systems a clean, curated view of a website instead of making them wade through HTML clutter. Technically the file is trivial and low-risk. What’s missing is the demand side: no major provider demonstrably evaluates it, Google rejects it, and the frequently cited provider-owned files prove nothing about yours. Treat llms.txt accordingly — as a cheap, optional bet on the future, not a mandatory lever for AI visibility. Anyone wanting impact should first invest in what demonstrably works: good content, structured data and clean crawler control.

Themenuebersicht

GEOAI-CrawlerGEO-Grundlagen