How Crawlers Find and Are Guided Through Your Site

robots.txt and a sitemap sound like they might do the same thing, but they answer two different questions. robots.txt tells a crawler where it is and is not allowed to go. A sitemap tells a crawler what exists in the first place. A site can have a perfect sitemap and still block crawling entirely with robots.txt, and a site with no robots.txt at all is fully crawlable by default, sitemap or not.

A robots.txt file is organized into groups, each starting with one or more User-agent lines followed by Allow and Disallow rules that apply to those specific crawlers. A bare Disallow: / blocks everything under that group; an empty Disallow: means nothing is blocked at all. Different groups can target different crawlers by name, so a site can, for example, allow a specific search engine while blocking others.

Listing a URL in a sitemap tells a search engine it exists and may be worth crawling, but it is not a promise that the page will actually be indexed. A sitemap entry for a page that robots.txt blocks, or one marked noindex, is a contradiction that crawlers generally resolve in favor of not indexing it, since the more restrictive signal usually wins.

Nothing about robots.txt is technically enforced. Reputable search engines follow it closely because ignoring it would damage their relationship with site owners, but a robots.txt file is not a security control and should never be relied on to keep a page genuinely private, since a misbehaving crawler or a person who simply requests the URL directly will see it regardless of what the file says.

Robots.txt Tester parses a site's robots.txt into its user-agent groups so you can see exactly which rules apply to which crawler, and Sitemap Checker validates a sitemap's structure, counts its URLs, and spot-checks a sample of them for a real response. Run both together after changing either one to confirm they say what you actually intended.