Free Robots.txt
Checker
Catch the misconfigurations that block Googlebot from your site. Check Disallow rules, sitemap directives, and crawl directives on any domain in seconds.
One line can wave crawlers off your whole site. Catch it first.
Paste any domain and read its live robots.txt the way a crawler does: every rule grouped by user-agent, every sitemap checked, and any site-wide block flagged. Fetched from the root, never cached.
A file nobody reads can quietly shut crawlers out.
robots.txt is the first file Google looks for, and it is only a few lines long. That is exactly why a single wrong line can tell every crawler to stay off your site, and why it can go unnoticed for weeks. The checker reads the live file and turns it into a short, honest punch list.
One domain in. A plain report out.
Paste a domain
Just the domain, like example.com. We add https and find its /robots.txt for you.
We fetch and parse it live
We pull the real file from the site root, group every rule under its user-agent, and follow each Sitemap line to check it responds.
You read the result
Four headline numbers, every parsed rule, each sitemap verified, and any critical block flagged, with the raw file underneath.
What these words mean
robots.txt
A plain-text file at your domain's root that tells crawlers which URLs they are allowed to fetch.
example.com/robots.txtUser-agent
The name of the crawler a block of rules applies to. A star means the rules apply to every crawler.
Googlebot, Bingbot, starDisallow
A path a crawler is asked not to fetch. A lone slash means the whole site, which is the line to watch for.
Disallow: /cart/Allow
A path a crawler may fetch, even when it sits inside a folder you disallowed. The longest matching path wins.
Allow: /cart/successSitemap directive
A line pointing crawlers at your sitemap so new pages get discovered sooner. The checker follows it to see if it responds.
Sitemap: /sitemap.xmlCrawling vs indexing
robots.txt controls crawling, not indexing. It asks crawlers not to fetch a URL. To keep a page out of results, use a noindex tag instead.
block equals not fetchedRead your rules the way a crawler does.
The file is grouped under each user-agent with its Allow, Disallow, and Crawl-delay lines, so you can see exactly what Googlebot, Bingbot, or GPTBot is allowed to fetch, without parsing it in your head.
- Every rule grouped under the crawler it applies to
- Allow, Disallow, and Crawl-delay shown per user-agent
- Star rules apply to any crawler without its own group
Why it works: when two paths match the same URL, the most specific, longest one wins, not the first line. Reading the groups this way is how you spot the rule that actually applies.
Parsed rules · example.com
The site-wide block, flagged before it costs you.
The checker sorts what it finds into critical issues and warnings. A Disallow slash on any user-agent is flagged critical, because it can wave a crawler off your entire site. Missing, empty, or unreachable files land in the same tier, and a missing Sitemap line comes back as a warning.
- A site-wide Disallow slash flagged critical, per crawler
- An unreachable, empty, or rule-less file caught
- A missing Sitemap line surfaced as a warning
Pro tip: before you panic at a Disallow slash, read which user-agent it sits under. On AhrefsBot it is probably deliberate. Under Googlebot or a star, it is the emergency.
- robots.txt reachable, HTTP 200
- File is present, not empty, with rules defined
- Sitemap line found and both sitemaps respond
Not just found. Fetched.
For each Sitemap line in the file, the checker follows the URL and reports whether it responds, so a sitemap that returns a 404 does not sit unnoticed while your new pages wait to be discovered.
- Every Sitemap line collected from the file
- Each URL fetched and its HTTP status shown
- Up to five sitemaps verified on every check
Rule of thumb: a missing sitemap line is a warning, not an emergency. It will not take you offline, but adding it helps crawlers find new pages sooner.
- Each Sitemap line followed to its live URL
- HTTP status reported next to each one
- A broken sitemap stands out instead of hiding
How to read a result to a verdict.
Four numbers only help if you know what to do with them. Here is the exact read on our example domain, example.com, the way an experienced SEO would think it through.
One paste, before it costs you.
Verify the deploy
Confirm a release did not ship a blocking robots.txt from staging. Paste the live URL and see exactly what crawlers read right now.
Audit before it hurts
Catch a stray Disallow slash on Googlebot, or a broken sitemap, before pages quietly drop out of the crawl.
Sanity-check a migration
Make sure a new platform's default robots.txt did not lock crawlers out of the whole site the day you launched.
Speed up discovery
Confirm your Sitemap line is present and responding, so fresh posts get found sooner instead of by chance.
Questions, answered.
It controls crawling, not indexing. It asks well-behaved crawlers which URLs they may fetch. It does not remove a page from Google's results; a noindex meta tag or header does that. If you block a page here, Google cannot see its noindex, so the URL can still appear as a bare link. To keep a page out of results, allow crawling and add noindex instead.
No. A site-wide block is flagged critical because it is high-stakes, but it is often deliberate on crawlers like AhrefsBot, where owners hide their backlinks from competitors. Under Googlebot or a star, it is usually a mistake. The checker flags every one so you can read the user-agent and decide.
We fetch the live robots.txt from the domain's root when you search, then follow each Sitemap line to check it responds. Nothing is cached, so you see what crawlers see at this moment.
This reads your robots.txt and confirms each declared sitemap URL responds. The Sitemap Validator opens a sitemap and inspects the URLs inside it. Use this to find and verify the sitemap line, and that to audit the sitemap's contents.
Yes. Paste a domain and read the four headline numbers. Green is healthy, and any critical issue is explained in plain words with what it means and what to do next. No account, no credit card, and you can check as many domains as you like.
Not quite, and the difference matters. Google retired the robots.txt tester that lived in the old Search Console, and what replaced it is a report showing the robots.txt files Google has already fetched for a property you own and have verified. That is useful for auditing your own site after the fact. A google robots.txt checker of that kind cannot be pointed at a competitor, and it will not tell you anything about crawlers other than Google. This one fetches the live file for any domain and reports how it reads to Googlebot, Bingbot and the AI crawlers -- GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot -- which is where most accidental blocks turn up.
Paste a domain and see what crawlers see.
It takes one paste. You will get every rule grouped by crawler, every sitemap verified, and any site-wide block flagged, before it quietly costs you. Free, no signup, one of 30-plus SEO tools in Metric Vault AI.
See if AI actually recommends you
The checker tells you if AI can read your site. Metric Vault tracks whether ChatGPT, Gemini, Perplexity and Claude actually mention you, how you stack up against competitors, and how it trends over time, plus the rest of the platform's 60+ SEO and AI tools.