robots.txt checker

Summary#

The Free Robots.txt Checker fetches robots.txt from a site's root, parses every User-agent, Allow, Disallow and Crawl-delay directive, checks that the sitemaps declared in it actually respond, and flags the mistakes that quietly cost traffic. It runs at /free-tools/robots-checker with no account.

Purpose#

robots.txt is four lines of plain text that can remove an entire site from search. Nobody is warned when it goes wrong. A staging config ships to production with Disallow: /, a developer pastes a snippet that blocks AI crawlers, a sitemap line points at a URL that 404s after a migration. Each of those is silent and each is expensive.

This checker turns the file into a readable report in one fetch so those mistakes are visible before they cost you a quarter.

Overview#

The tool requests /robots.txt at the origin of the domain you enter. It strips comments, parses each directive line, and groups Allow, Disallow and Crawl-delay under the User-agent they follow. Every Sitemap: line is collected, and the first five are checked with a request to confirm they respond.

It then applies five issue rules and shows the raw file so you can read it yourself.

Benefits#

• Turns an unformatted text file into grouped, readable rules per crawler.
• Verifies the sitemaps you declare rather than trusting the line exists.
• Catches a site-wide Disallow: / immediately and marks it critical.
• Shows the raw file, so you can confirm the parse against the source.
• No account, no credits.

Use Cases#

• Post-deploy verification. The single highest-value thirty seconds after a release to a new environment.
• Diagnosing an indexing drop. Check this before anything else.
• AI-crawler policy review. See which of the AI user agents are named and what they are allowed.
• Auditing a site you are about to take over. No access needed.

Requirements#

A public domain. The file must be served from /robots.txt at the origin, which is the only place crawlers look for it.

Permissions#

Anyone. No account, plan or role. A shared hourly fair-use limit applies per IP address across the free tools.

Cost#

Free. Zero credits.

Navigation Path#

metricvaultai.com → Free tools → Robots.txt Checker

The page is at /free-tools/robots-checker.

Inputs#

Helper line: No signup · No credit card · Tests live robots.txt from the root.

Step-by-Step Guide#

1. Open /free-tools/robots-checker.
2. Enter the domain in the field marked example.com.
3. Select Check robots.txt. The button reads Checking… and the result area shows Fetching robots.txt and verifying sitemaps….
4. Read HTTP Status first. Anything other than 200 means crawlers are not getting the file you think they are.
5. Work the Issues Found list, then confirm the parse against Raw robots.txt.

Reading the Results#

The four headline numbers.

Issues Found. Each entry is prefixed CRITICAL: or WARNING:.

When the file is reachable and no issue applies, the section shows No issues detected. Your robots.txt looks healthy.

Parsed Rules (N). One block per User-agent, headed User-agent: <name>, listing each Allow: and Disallow: path and any Crawl-delay: value. This is the fastest way to see what a specific crawler is allowed to do.

Sitemaps Declared. One chip per Sitemap: line, each showing the URL and the result of the reachability check as HTTP 200, another status, or FAIL. Only the first five declared sitemaps are checked.

Raw robots.txt. The file as served, up to the first 8,000 characters. Read it when the parse surprises you.

Examples#

Screenshots#

![screenshot](placeholder: The "Parsed Rules (N)" blocks with per-user-agent Allow and Disallow directives, and the "Sitemaps Declared" chips with their HTTP status.)

Tips#

• Read Parsed Rules for the specific crawler you care about. A permissive User-agent: * block does not help a bot that has its own stricter block.
• Add a Sitemap: line even if you also submit the sitemap in Search Console. It costs one line and every crawler reads it.
• Disallow is not noindex. Blocking a URL stops it being crawled; it can still appear in results if other pages link to it. Use a noindex meta tag to keep a page out of the index.
• Do not block CSS and JavaScript directories. Search engines render pages, and blocking assets makes your page look broken to them.

Best Practices#

• Check robots.txt as part of every production deploy, not just when something looks wrong.
• Be explicit about AI crawlers rather than leaving it to a copied snippet. Decide whether you want to appear in AI answers, then write the rules to match. See AI visibility checker for what that decision costs or earns.
• Keep the file short. Long, heavily commented robots.txt files accumulate contradictions.
• Pair this with Sitemap validator. This tool confirms the sitemap line responds; that one confirms the sitemap itself is valid.

Common Mistakes#

• Blocking a whole site with Disallow: /. Marked critical for a reason.
• Using robots.txt to hide a page. It prevents crawling, not indexing, and a blocked page cannot even be read to discover its noindex tag.
• Declaring a sitemap that 404s. The chip shows the real status; trust it.
• Blocking AI crawlers by default and then wondering why AI never mentions you. Both are legitimate choices; make it deliberately.

Limitations#

• One domain per run, and only the root robots.txt. Subdomains have their own file and need their own run.
• Only the first five declared sitemaps are reachability-checked.
• The raw file is shown up to 8,000 characters.
• The parser reads directives literally. It does not simulate a specific crawler's precedence rules, so it will not tell you which of two conflicting rules Google would apply to a given URL.
• Crawl-delay is reported as written. Google ignores the directive; other crawlers may honour it.
• No export, history, scheduling or recommendations panel. See Free tools, no account needed.

Troubleshooting#

FAQs#

Does this tell me whether a specific URL is blocked? Not directly. It shows every rule grouped by crawler so you can read the answer, but it does not simulate a crawler's precedence logic against a single path.

Does it check subdomains? No. Each subdomain serves its own robots.txt. Run the check once per subdomain.

Why does it check my sitemaps? Because a Sitemap: line pointing at a dead URL is one of the most common and least noticed faults. Only the first five are checked.

Is Disallow: / always wrong? No. It is correct on a staging site and catastrophic on a production one. The tool marks it critical so you notice; you decide whether it belongs.

Is it really free? Yes. No account, no card, no credits.

What does the in-app version add? robots.txt Auditor runs the same audit inside your workspace, where the result is saved to your history and can be exported, and sits alongside the rest of the technical checks in Site Audit and Advanced Technical Checks.