robots.txt Auditor

Summary#

robots.txt Auditor fetches your live robots.txt, parses every directive in it, and renders a crawler-access matrix showing whether Google, Bing and the major AI crawlers can reach your site. It also verifies that every sitemap the file declares actually loads. It costs nothing.

Purpose#

robots.txt is a plain text file that most sites set up once and never read again, and it is the single most dangerous file in a website. One line — Disallow: / under the wrong user-agent — removes a site from search results, and nothing on the site looks different afterwards.

The file has also become the place where you decide whether AI systems may read your content. GPTBot, ClaudeBot, PerplexityBot, CCBot and Google-Extended all obey it, and most site owners have never checked what theirs currently says.

This tool exists so you can read that decision at a glance, in a table, instead of parsing directive precedence in your head. The decision it helps you make is who is allowed to see my content, and is that what I intended.

Overview#

You give it a domain. Metric Vault requests https://yourdomain.com/robots.txt directly, strips comments, and parses the file line by line into user-agent blocks with their Allow, Disallow and Crawl-delay directives, plus every Sitemap: line.

It then does two things a text editor cannot. First, it resolves effective access for each named crawler: a crawler's own block if it has one, otherwise the wildcard * block, otherwise "not addressed". Second, it sends a request to each declared sitemap URL — up to five — to confirm it actually returns HTTP 200, because a robots.txt that points at a dead sitemap is worse than one that points at none.

Benefits#

• Reads directive precedence for you. No more guessing whether the wildcard block or the named block applies to Googlebot.
• Covers AI crawlers, not just search engines — the access decision most sites have never audited.
• Verifies the sitemaps, so a broken Sitemap: line cannot hide.
• A score with a fix list, not just a dump of the file.
• Free on every plan, with the raw file shown syntax-highlighted underneath.

Use Cases#

• After a launch or migration. The staging Disallow: / that never got removed is the most common catastrophic SEO mistake, and this finds it in one run.
• Deciding your AI policy. You want to know, today, whether ChatGPT, Claude, Perplexity and Common Crawl can read your content.
• Debugging a sudden indexing drop. Before anything else, confirm crawlers can still reach the site.
• Checking a client's site on day one. The crawler matrix is a one-screen answer to "is anything blocked that should not be?"
• Confirming a sitemap is discoverable. Search engines find your sitemap through this file more reliably than any other route.

Requirements#

• A signed-in Metric Vault account.
• Any plan, including Free.
• A publicly reachable domain. Private, internal and loopback hosts are refused.
• No credits, no integration, no ownership check.

Permissions#

No role restriction applies. See Roles and what each can do.

Cost#

0 credits. The button reads Audit robots.txt · FREE. The run never touches your monthly credit allowance and counts only toward the fair-use limit of 100 light-tool calls per hour. See How credits work and What you get on Free.

There is no result cache and no "You already ran this" prompt: every run fetches the file again, which is what you want from a file that changes on deploy.

Navigation Path#

Dashboard → Site Health → robots.txt Auditor

Inputs#

Step-by-Step Guide#

1. Open Dashboard → Site Health → robots.txt Auditor.
2. Optionally click See example to preview a full report for free.
3. Type your domain into the field marked e.g. nike.com.
4. Click Audit robots.txt · FREE, or press Enter.
5. Read the SEO Score and the verdict line first — that tells you whether anything is on fire.
6. Check the Crawler Access Matrix row by row against what you intended.
7. Work through Priority Fix Breakdown, which converts each detected issue into a titled instruction.
8. Export from the row at the top of the screen: Export PDF, Export Excel, Export CSV, Export JSON or Share public link.

Reading the Results#

The header and verdict. Your domain with a status pill, and one plain-English verdict chosen from these:

The KPI tiles. User-agents (how many blocks the file defines), Allow rules, Disallow rules, and Sitemaps with three sub-figures — Declared, HTTP 200 and Broken. Broken should always be 0.

SEO Score. A 0-100 grade labelled HEALTHY (85+), NEEDS WORK (65-84) or CRITICAL (below 65). It starts at 100 and subtracts: 40 if the file is unreachable, 30 if Googlebot is blocked, 15 if no sitemap is declared, 12 per critical issue, 10 per broken sitemap and 5 per warning. The floor is 20, so a score of 20 means several of those fired at once.

Rule Distribution. A donut of Allow, Disallow, Sitemaps and Crawl-delay directives with the total in the center. A file that is overwhelmingly Disallow with no Allow rules is worth reading closely — broad blocks catch pages you did not mean to hide.

Sitemap Status. Every declared sitemap with the HTTP status it returned. Any non-200 here is a real problem: search engines discover most sitemaps through this file, so a dead link there slows discovery of every new page you publish. Only the first five declared sitemaps are checked.

Crawler Access Matrix. The most valuable panel. Eight rows, each showing the rule that actually applies to that crawler:

Each row resolves to one of four verdicts:

• Full access (green) — no Disallow rules apply.
• Partial access (amber) — N paths blocked, and any crawl-delay in effect.
• Blocked (red) — Disallow: / blocks the entire site for that crawler.
• Not addressed (grey) — no specific rule and no wildcard rule; the crawler will assume full access.

What good looks like: Googlebot and Bingbot on Full access or a deliberate Partial access, and the AI rows matching a decision you actually made. What bad looks like: Googlebot Blocked — stop reading and fix that now. There is no wrong answer on the AI rows, only an unintentional one.

Priority Fix Breakdown. Each issue turned into a titled instruction with a severity badge — for example "Remove site-wide Disallow: /", "Make robots.txt reachable", "Declare a Sitemap directive" (with the exact line to paste for your domain), "Populate robots.txt", "Add User-agent rules" and "Broken sitemap URL" with the failing URL and status. If nothing is wrong you get a single card: "No fixes needed".

Parsed Rule Blocks and the raw file. Every user-agent block with its Allow, Disallow and Crawl-delay directives listed, followed by your actual robots.txt rendered with syntax highlighting so you can confirm the parse against the source. The raw view shows the first 8,000 characters.

Examples#

Screenshots#

Tips#

• Audit both example.com and www.example.com if both resolve. They are separate origins and can serve different files.
• If you want AI systems to stay out, block them explicitly and include Google-Extended — blocking GPTBot alone does not cover Google's AI training.
• robots.txt controls crawling, not indexing. To keep a page out of the index, use a noindex robots meta tag, which On-Page SEO Checker reports.
• A blocked page cannot be re-crawled, so a noindex on a blocked page is never read. Never combine the two.
• Run this immediately after any launch, migration or CDN change.

Best Practices#

• Declare your sitemap in robots.txt. It is the most reliable discovery route, and its absence costs you 15 points here for a reason.
• Prefer narrow Disallow paths over broad ones, and add explicit Allow rules for anything inside a blocked directory that should stay crawlable.
• Never block CSS or JavaScript directories — Google needs them to render and judge your pages.
• Re-audit after every deploy that touches infrastructure, and keep an export so you can see what changed.
• Decide your AI-crawler policy once, write it down, and make the matrix match it.

Common Mistakes#

• Shipping the staging file. Disallow: / survives a launch far more often than anyone expects.
• Assuming a missing file is safe. It is treated as "crawl everything" and costs you the sitemap declaration.
• Using robots.txt to hide private content. It is a public file that lists exactly which paths you consider sensitive. Use authentication.
• Blocking a page you also want de-indexed. Blocked pages can still appear in results as a bare URL, because the noindex can never be read.
• Checking only the apex domain when your site also serves a www host or regional subdomains.

Limitations#

• One origin per run. Subdomains each have their own robots.txt and need their own audit.
• Only the first five declared sitemaps are status-checked.
• The raw view shows the first 8,000 characters of the file.
• The matrix covers eight crawlers. Other bots are represented by the wildcard row.
• It reports what the file says, not what a crawler did. For actual crawl behavior, use Google Search Console.
• Path-level matching is not simulated — the matrix answers "full, partial or blocked", not "is this specific URL crawlable".

Troubleshooting#

FAQs#

Does this cost credits? No. It is one of the zero-credit technical tools and runs on every plan, including Free.

What is the difference between blocking a crawler here and using noindex? robots.txt stops a crawler fetching a page. noindex lets it fetch the page and tells it not to list it. They solve different problems, and combining them breaks both — a blocked page's noindex is never read.

If I block GPTBot, am I safe from all AI training? No. Each system uses its own user-agent. The matrix shows GPTBot, ClaudeBot, PerplexityBot, CCBot and Google-Extended separately precisely because blocking one does nothing to the others.

My score is 78 but nothing looks broken. Why? The most common cause is a missing Sitemap: directive, which costs 15 points on its own. Warnings cost 5 each and a broken sitemap costs 10.

Does it check whether one specific URL is crawlable? No. It resolves access at the crawler level — full, partial, blocked or not addressed — and lists the blocked paths. For a single-URL verdict, use Google Search Console's URL Inspection.

Should I be worried that my file has 40 Disallow lines? Not necessarily. Volume is not the signal; scope is. Check the Crawler Access Matrix verdicts and the blocked paths in Parsed Rule Blocks, and confirm nothing you want indexed sits underneath them.