Tutorial
Step-by-step guide
What to know about Robots Preview Tool
robots.txt uses Allow, Disallow and Sitemap directives to guide crawlers. It is public, easy to misread and powerful enough to block an entire site by accident.
Robots Preview Tool runs on public inputs and is suitable for VPS-side checks using HTTP requests, DNS lookups, HTML parsing, validation logic or generated output. It does not need a paid SEO API.
The result should be treated as a practical webmaster report: read the status, confirm the affected signal and retest after you change the source.
Common problems
Common robots.txt failures include Disallow: / left from staging, missing sitemap hints, blocked assets needed for rendering and rules applied to the wrong user-agent.
On real sites, these issues often appear after CMS updates, DNS migrations, CDN changes, template edits, plugin installs or rushed launch work.
Do not check only the homepage. Run the tool against the exact URL, domain, record or file that matters.
How to fix and retest
Keep production robots rules narrow. Remove staging blocks, include the sitemap directive and use page-level noindex when the goal is index control rather than crawl control.
Change one thing at a time, clear any relevant CDN or application cache, then run Robots Preview Tool again from the public Frabs page.
If the result differs between your machine and Frabs, check DNS propagation, CDN edge behavior, bot filtering and whether the URL redirects to a different final page.
Copy and paste checks
Use these examples when you want a second opinion from a terminal. Replace example.com with your own domain or URL.
The command output is not a replacement for the Frabs report, but it helps confirm the raw public signal.
curl -L https://example.com/robots.txtcurl -I https://example.com/robots.txtBest practice checklist
Run Robots Preview Tool before major changes, immediately after deployment and again once caches or DNS propagation have settled.
Save the result with your launch notes if the page, domain or configuration is important to search, email, security or revenue.
Pair this check with related Frabs tools so you can see whether the problem is isolated or part of a wider technical pattern.