Skip to main content

Free Robots.txt Checker

Check the file that tells Google which pages to skip, and make sure it is not hiding pages you want found.

SEO tools

What a passing result looks like

Your robots.txt exists, doesn't block important content, and includes a Sitemap directive.

How to fix this

Create or fix your robots.txt file at the domain root with proper directives.

  1. 1Visit yoursite.com/robots.txt to check the current file.
  2. 2Ensure it does NOT contain "Disallow: /" (this blocks everything).
  3. 3Add a Sitemap line: Sitemap: https://yoursite.com/sitemap.xml
  4. 4Use Google Search Console's robots.txt tester to validate.

Tips by website platform

WordPress: Yoast SEO → Tools → File editor → edit robots.txt directly.
Wix: Settings → SEO → SEO Tools → Robots.txt Editor.
Shopify: Shopify manages robots.txt automatically. Use the robots.txt.liquid template for customisation.

Frequently asked questions

What does robots.txt do?
It tells search engine crawlers which parts of your site they can and cannot access. It's a text file at your domain root (e.g., yoursite.com/robots.txt).
Can robots.txt block my site from Google?
Yes. A misconfigured robots.txt with "Disallow: /" will prevent all search engines from crawling your site.
Is robots.txt required?
Not strictly, but it's best practice. Without one, search engines will crawl everything, which may include admin pages or duplicate content you'd rather hide.

Related tools

Fix guides that use this tool

Browse fix library

Want the full picture?

This tool checks one thing. Our full audit runs 100+ checks across 8 areas, and it is free.

Run a full audit (free)