Skip to main content
METRIC VAULT
← All posts

How to Validate Your llms.txt Without Guessing

Your llms.txt might look fine and still be broken. Here's a repeatable check to confirm it's syntactically correct, reachable and actually being read by AI crawlers.

·

A developer works intently at a cluttered desk late at night, their face lit by a glowing monitor as warm lamplight and cool screen glow illuminate scattered coffee, cables, and sticky notes.

Validate llms.txt in four steps. Confirm it returns a clean 200 at your domain root. Check the Markdown syntax and that every linked URL resolves. Inspect server logs for AI crawler hits like GPTBot. And verify the linked content is current. Automate the first two steps in CI.

Publishing an llms.txt file is easy. Knowing it works is another matter. Most site owners upload the file, see a 200 response, and call it done. Proper llms.txt validation goes further. It confirms the file is syntactically correct, reachable at the right path, and actually parsed by the tools that read it. This guide walks through each check in order, so you can stop guessing and start verifying.

Why llms.txt validation matters

Unlike robots.txt, there's no universally enforced spec police for llms.txt. A malformed file rarely throws a loud error. It just gets ignored. That makes silent failure the default outcome of a sloppy llms.txt setup. Validation is the only way to catch problems before they cost you visibility in AI-generated answers.


Step 1: Confirm the file is reachable

Your AI crawler file needs to live at the root of your domain and return a clean 200 status. Test it from the command line:

curl -I https://example.com/llms.txt

Check three things in the response:

  • Status is 200, not a redirect or 404
  • Content-Type is text/plain or text/markdown
  • No authentication wall or geo-block gets in the way

If you get a 301/302, fix the canonical path. Redirect chains are a common reason an otherwise valid file never gets read.

Step 2: Validate the syntax

The llms.txt format is Markdown-based with a defined structure: an H1 title, an optional blockquote summary, and sections of linked resources. A quick manual checklist catches most syntax errors:

ElementRequirement
TitleSingle H1 (#) at the top
SummaryOptional blockquote (>) right after the title
SectionsH2 (##) headings grouping links
LinksMarkdown link format name
URLsAbsolute, not relative

Parse it to catch what the eye misses. A minimal script confirms every link resolves:

grep -oP '\(https?://[^)]+\)' llms.txt | tr -d '()' | while read url; do
  echo "$(curl -s -o /dev/null -w '%{http_code}' "$url") $url"
done

Any line returning 404 or 500 is a broken reference an LLM won't be able to follow.

Step 3: Check that it's actually used

A syntactically perfect file is worthless if nothing reads it. This is where most llms.txt validation stops short. To confirm real usage, look through your server logs for AI crawler user agents requesting the file:

grep "llms.txt" access.log | grep -iE "gptbot|claudebot|perplexitybot|google-extended"

Seeing those hits confirms the file is in the fetch path of the agents you care about. If you see no requests over several weeks, the file may be published but undiscovered. Or your audience's tools simply don't read it yet.


Step 4: Verify content accuracy

Validation isn't only structural. Open the file and confirm every linked page still exists, titles match current content, and no stale sections point to deleted resources. An accurate AI crawler file keeps earning trust each time it's fetched.

Build validation into your workflow

Run these four checks whenever you change your llms.txt setup or ship major site updates. Automate steps 1 and 2 in CI, so a broken link or wrong status fails the build. Treat the file like any other production asset: tested, monitored, and never assumed correct.

Do that, and your llms.txt validation becomes a reliable signal rather than a hopeful guess.