robots.txt Security Checker

What this robots.txt checker looks for

Enter a URL and this tool fetches /robots.txt from that site's root — never a path you typed, always the origin — and analyses it from an angle most robots.txt validators skip entirely: security. Most tools check whether the syntax parses. This one also asks what the file tells an attacker, because that question matters more than whether a crawler can technically read the rules.

robots.txt is not an access control mechanism

This is the single most misunderstood thing about robots.txt, and the whole reason this checker exists. A Disallow line does not block a request — it asks polite crawlers not to make one. Google, Bing and every other reputable search engine will honour it. A script written by someone probing your site for weaknesses will not, because nothing stops it from ignoring a text file and requesting the path directly. The file has no enforcement mechanism at all; it is a request, not a wall.

That distinction matters because robots.txt is served to anyone who asks, with no authentication. It is one of the first files an attacker fetches when reconnoitring a target, precisely because site owners keep putting useful information in it.

Sensitive paths disclosed

This is the headline check. Every Disallow entry is compared against patterns commonly found in admin panels, backup directories, configuration files and internal APIs — things like /admin, /wp-admin, /.git, /.env, /backup, /config, and /phpmyadmin. If a match is found, the tool flags it as medium severity and lists every matching path in one finding.

The failure mode is specific: someone adds Disallow: /admin-panel-v2/ to keep it out of Google, believing this hides it. Instead they have published, in a world-readable file, the exact path of their admin panel — including the "-v2" that tells an attacker there was a v1 they might also try. Hiding a path this way is strictly worse than not mentioning it at all.

Whole-site disallow

User-agent: * paired with Disallow: / blocks every crawler from the entire site. This is normal for a staging environment that should never appear in search results. On a production domain it is usually accidental — a rule left over from before launch, silently keeping the site out of Google indefinitely. We flag this as informational either way, since only you know which case applies.

Syntax problems

robots.txt has a loose, decades-old grammar that most parsers are forgiving about, but a few mistakes reliably break it:

• A directive that appears before any User-agent line has no group to belong to, so crawlers ignore it entirely.
• Allow and Disallow values must start with / — a value like admin instead of /admin matches nothing.
• Unrecognised directive names are usually typos (Disalow, Useragent) that silently do nothing.
• Doubled wildcards (**) behave identically to a single * and almost always indicate confusion about how the matching works.

Crawl-delay and Sitemap

Crawl-delay asks crawlers to wait a number of seconds between requests. It is worth knowing that Google does not support it at all — Search Console has its own crawl-rate controls — so this directive only affects the subset of crawlers that honour it. A Sitemap directive is the opposite: unambiguously useful, pointing crawlers straight at your sitemap XML instead of making them discover pages by following links. Its absence is not an error, just a missed opportunity.

File size

A robots.txt over 10KB or more than 100 rules is flagged as low severity — not because size itself is dangerous, but because files this large accumulate stale rules nobody remembers to remove, and are exactly where forgotten sensitive paths tend to survive for years.

A safe baseline

For most production sites, the safest robots.txt says as little as possible:

User-agent: *
Disallow:

Sitemap: https://example.com/sitemap.xml

An empty Disallow allows everything, which is correct for a public marketing site. If specific paths genuinely need to stay out of search results, prefer an HTTP-level X-Robots-Tag header or a page-level noindex meta tag over listing the path in robots.txt — both achieve the same search-visibility goal without publishing the path to everyone who reads the file.