Validate llms.txt in four steps. Confirm it returns a clean 200 at your domain root. Check the Markdown syntax and that every linked URL resolves. Inspect server logs for AI crawler hits like GPTBot. And verify the linked content is current. Automate the first two steps in CI.
Publishing an llms.txt file is easy. Knowing it works is another matter. Most site owners upload the file, see a 200 response, and call it done. Proper llms.txt validation goes further. It confirms the file is syntactically correct, reachable at the right path, and actually parsed by the tools that read it. This guide walks through each check in order, so you can stop guessing and start verifying.
Why llms.txt validation matters
Unlike robots.txt, there's no universally enforced spec police for llms.txt. A malformed file rarely throws a loud error. It just gets ignored. That makes silent failure the default outcome of a sloppy llms.txt setup. Validation is the only way to catch problems before they cost you visibility in AI-generated answers.
Step 1: Confirm the file is reachable
Your AI crawler file needs to live at the root of your domain and return a clean 200 status. Test it from the command line:
curl -I https://example.com/llms.txtCheck three things in the response:
- Status is
200, not a redirect or404 Content-Typeistext/plainortext/markdown- No authentication wall or geo-block gets in the way
If you get a 301/302, fix the canonical path. Redirect chains are a common reason an otherwise valid file never gets read.
Step 2: Validate the syntax
The llms.txt format is Markdown-based with a defined structure: an H1 title, an optional blockquote summary, and sections of linked resources. A quick manual checklist catches most syntax errors:
| Element | Requirement |
|---|---|
| Title | Single H1 (#) at the top |
| Summary | Optional blockquote (>) right after the title |
| Sections | H2 (##) headings grouping links |
| Links | Markdown link format name |
| URLs | Absolute, not relative |
Parse it to catch what the eye misses. A minimal script confirms every link resolves:
grep -oP '\(https?://[^)]+\)' llms.txt | tr -d '()' | while read url; do
echo "$(curl -s -o /dev/null -w '%{http_code}' "$url") $url"
doneAny line returning 404 or 500 is a broken reference an LLM won't be able to follow.
Step 3: Check that it's actually used
A syntactically perfect file is worthless if nothing reads it. This is where most llms.txt validation stops short. To confirm real usage, look through your server logs for AI crawler user agents requesting the file:
grep "llms.txt" access.log | grep -iE "gptbot|claudebot|perplexitybot|google-extended"Seeing those hits confirms the file is in the fetch path of the agents you care about. If you see no requests over several weeks, the file may be published but undiscovered. Or your audience's tools simply don't read it yet.
Step 4: Verify content accuracy
Validation isn't only structural. Open the file and confirm every linked page still exists, titles match current content, and no stale sections point to deleted resources. An accurate AI crawler file keeps earning trust each time it's fetched.
Build validation into your workflow
Run these four checks whenever you change your llms.txt setup or ship major site updates. Automate steps 1 and 2 in CI, so a broken link or wrong status fails the build. Treat the file like any other production asset: tested, monitored, and never assumed correct.
Do that, and your llms.txt validation becomes a reliable signal rather than a hopeful guess.
