Robots.txt Tester

Robots.txt Tester fetches a site's robots.txt and parses it into its user-agent groups, showing exactly which Allow and Disallow rules apply to each crawler along with any sitemap URLs it references.

Reach for this after changing a robots.txt to confirm it parses the way you intended, when a page is not being indexed and a crawling rule is a suspect, or simply to see what a site is telling crawlers without downloading the raw file yourself.

This shows the rules as written, not a judgment on whether a specific crawler obeys them. Reputable search engines follow robots.txt closely, but the standard is voluntary and nothing stops a misbehaving bot from ignoring it entirely.