Skip to main content
METRIC VAULT
← All posts

The llms.txt Setup That AI Engines Respect

Most llms.txt files are guesswork. Here's what a correct one actually contains, why AI engines use it, and the misconfigurations that quietly waste the file entirely.

·

A handwritten outline on white paper sits on an oak desk beside a fountain pen, reading glasses, and water glass, bathed in warm morning sunlight.

A good llms.txt is a short Markdown file at your domain root. It has an H1 name, a one-line blockquote summary, and H2 sections of hand-picked links with real descriptions pointing to server-rendered pages. Keep it selective, don't treat it like robots.txt, and update it whenever your site structure changes.

If you run a site that cares about showing up in ChatGPT, Perplexity, Claude, or Google's AI Overviews, you've probably heard about llms.txt. The trouble is that most files out there are either copied from a template that doesn't fit the site, or built on wrong assumptions about how AI engines actually behave.

This guide covers what a correct llms.txt really contains, how it fits into a wider GEO technical setup, and the mistakes that quietly waste all your effort. The aim is a file that AI engines can read, trust, and use — not one that just sits in your root directory doing nothing.

What llms.txt Is Actually For

llms.txt is a plain-text Markdown file placed at the root of your domain (/llms.txt). Its job is to give large language models a hand-picked, machine-readable map of the content on your site that matters most — the pages you want an AI crawler to understand and cite.

It is not a replacement for robots.txt. robots.txt controls access and crawling permissions. llms.txt is about understanding and priority. Think of it as a sitemap written for reasoning models instead of indexers — a short table of contents with links and brief descriptions, not a dump of every URL you own.

This matters because a lot of early adopters treat llms.txt like a second robots.txt, stuffing it with disallow rules and crawl directives it was never meant to carry. That's the first mistake we'll clear up.

The format AI engines expect

The spec is simple on purpose. A valid file is Markdown with a specific loose structure:

  • An H1 with the site or project name
  • A short blockquote summary of what the site is
  • Optional detail paragraphs
  • H2 sections with lists of links, each written as Title: short description

Here's a small, correct example:

# Acme Analytics

> Acme Analytics is a product analytics platform for SaaS teams. These docs cover setup, the query API, and billing.

Acme helps teams track events, build funnels, and export data. The links below point to the canonical documentation.

## Core Docs

- [Getting Started](https://acme.com/docs/start): Install the SDK and send your first event.
- [Query API](https://acme.com/docs/api): Full REST reference with authentication and rate limits.
- [Billing](https://acme.com/docs/billing): Plans, invoices, and how usage is metered.

## Optional

- [Changelog](https://acme.com/changelog): Release notes going back to launch.

That structure is the whole point: easy for a human to read, trivial for a model to parse, and strict about what it includes.


What a Correct llms.txt Should Contain

A file that AI engines respect does a few things well. It curates, it describes, and it links to clean source content.

1. A clear, honest summary

The blockquote under your H1 is the single most important line. It's the context an LLM uses to decide what your site is about. Write it the way you'd explain the site to a colleague in one sentence. Skip the marketing language — "the world's leading platform" tells a model nothing useful, while "a product analytics platform for SaaS teams" tells it everything.

2. Only your highest-value pages

The temptation is to include everything. Don't. The value of llms.txt is signal, not coverage. Include documentation, key product pages, pricing, foundational guides, and canonical reference material. Leave out tag archives, paginated lists, thin pages, and anything you wouldn't want quoted in an AI answer.

3. Descriptions that add information

Each link should have a description explaining why the page matters or what specific thing it covers. "Pricing page" is weak. "Pricing: three tiers, per-seat billing, and the free-tier limits" is strong. These descriptions are often what a model reads before deciding whether to fetch the full page.

4. Grouping by intent

Use H2 sections to group links logically — Docs, Guides, API, Company. Some setups treat a section literally named ## Optional as content that can be skipped when context is tight, so use that convention on purpose if you want lower-priority links to drop off gracefully.

5. Links to clean, parseable targets

This is the part people forget. An llms.txt is only as good as the pages it points to. If a linked page is a JavaScript-rendered shell with no server-side content, the AI crawler that follows the link gets nothing. Pair your llms.txt with HTML pages that render real content without client-side execution — and ideally offer a plain Markdown version at the same path with an .md extension, which is a common companion pattern these days.


How AI Crawlers Actually Use the File

Let's be realistic about adoption. No AI engine has promised it reads llms.txt, and support varies. Treat it as a low-cost, forward-looking signal, not a switch that turns on citations.

Here's what we do know about how an AI crawler handles structured hints like this:

  • Models and retrieval systems prefer content they can read cheaply and confidently.
  • A curated list of canonical URLs cuts the chance a crawler grabs a duplicate or thin version of a page.
  • Clear descriptions help a retrieval layer match your pages to user queries.

In short, llms.txt doesn't force an engine to cite you. It lowers the friction and confusion that cause engines to skip or misread your content. That's the honest way to put it when stakeholders ask whether it "works."

Where it sits in your GEO technical setup

A complete GEO technical setup has several layers, and llms.txt is just one piece. The others still carry most of the weight:

LayerPurposePriority
Server-rendered HTMLContent an AI crawler can read without JSCritical
robots.txt / AI bot rulesAccess control for GPTBot, ClaudeBot, etc.Critical
Structured data (schema)Explicit entity and fact markupHigh
XML sitemapFull URL inventory for indexersHigh
llms.txtCurated map for LLMsSupporting

If your HTML isn't crawlable, or you're accidentally blocking AI user agents in robots.txt, a perfect llms.txt won't save you. Fix the foundation first, then add the curated layer on top.


Common llms.txt Misconfigurations

Most broken files fail in a few predictable ways. Here are the ones worth checking for.

Treating it like robots.txt

Adding User-agent and Disallow lines to llms.txt makes the file invalid as Markdown and useless as a content map. Crawl control belongs in robots.txt. Keep the two files separate in your head.

Wrong location or content type

The file must live at https://yourdomain.com/llms.txt, be served as text/plain or text/markdown, and return a 200 status. Common failures:

  • Placing it in a subdirectory
  • Serving it with an HTML content type wrapped in page chrome
  • Returning a soft 404 or a redirect chain

You can check this quickly from the command line:

curl -sI https://yourdomain.com/llms.txt
# Expect: HTTP/2 200 and content-type: text/plain or text/markdown

Dumping the entire sitemap into it

An llms.txt with 4,000 links isn't curation — it's noise. It defeats the purpose and buries your important pages. If you need full URL coverage, that's what the XML sitemap is for. Keep llms.txt to the pages that truly represent your expertise.

Missing or useless descriptions

Bare link lists with no descriptions throw away the file's main advantage. A model reading API learns far less than it would from a one-line summary. Always describe.

Linking to pages that render empty

If the targets need JavaScript to show text, an AI crawler following the link finds a hollow page. Check that each linked URL serves real content on the first HTML response. Spot-check with:

curl -s https://acme.com/docs/start | grep -i "getting started"

If the content you expect isn't in the raw response, the page needs server-side rendering or a static alternative.

Letting it rot

A stale llms.txt is a liability. If it points to pages you've deleted, restructured, or moved behind a login, it actively misleads crawlers. Add it to whatever process you use for sitemap upkeep so it updates when your site does.


A Practical Setup Checklist

Use this to go from nothing to a file AI engines can respect:

  1. List your best 20–50 pages. Docs, guides, pricing, canonical references.
  2. Write a one-sentence site summary as a blockquote under the H1.
  3. Group links into logical H2 sections. Put lower-priority links under ## Optional.
  4. Write a real description for every link. Specific, not promotional.
  5. Check each target renders content server-side. Fix any JS-only pages.
  6. Place the file at /llms.txt and confirm a 200 with a plain-text content type.
  7. Keep robots.txt separate and make sure you aren't blocking legitimate AI user agents you want to reach you.
  8. Schedule a review whenever your site structure changes.

The Bottom Line

A good llms.txt is small, honest, and selective. It gives AI engines a clean path to the content you most want understood and cited, and it clears up the confusion that makes crawlers grab the wrong page. It's not magic, and it's not a ranking lever you can pull on its own — it's the curated top layer of a solid GEO technical setup that starts with crawlable HTML and correct access rules.

Get the foundation right, keep the file tight, avoid the mistakes above, and you'll have an llms.txt that earns its place in your root directory instead of quietly doing nothing.