Skip to content

Free tools

robots.txt tester

Paste the live robots.txt and a path. The tester applies the longest matching rule. Allow wins a tie. No match means allowed.

    What this implements

    User-agent groups, Allow, Disallow, * wildcards, and the $ end anchor. A more specific agent beats *. Empty Disallow lines are ignored because they do not block a path. Crawl-delay and sitemap lines are not access rules.

    What still surprises people

    A disallow of /blog also blocks /blog-old and /blogger if you meant a directory. Trailing slashes matter. Google's indexer and a random bot are different agents. Test the agent you care about, not only *.

    The file is not the whole story

    A page can be allowed in robots.txt and still carry noindex. The site check reads both. This box only answers the robots question.

    • Test the directory

      Include the slash you meant.

    • Test a money URL

      Home, product, and one article.

    • Then view source

      Look for noindex after the path is allowed.

    FAQ

    Does allowed mean indexed?

    No. Allowed means the agent may fetch it. Indexing is a later decision.

    Is this Google's tester?

    No. It is a readable subset of the matching rules. Confirm edge cases in Search Console if a disallow is business-critical.

    Why is Fix SEO Today cheaper?

    A robots.txt test buried in an extremely expensive suite answers one question: which rule matched. Fix SEO Today is cheaper because you pay for the diagnosis. A free peek, then $19 a month for one project or $99 a month for unlimited projects. That covers the crawl and the plain-English fix list, including GEO. It leaves out the backlink index, the rank-tracker seats, and the ads module.