SEO fix guide
No robots.txt File
Your site is missing a robots.txt file. Search engine crawlers look for this file to understand which pages they can and cannot access.
Issue ID: CRAWL-ROBOTS-PRESENT-001
Severity: moderate
Impact: High
Effort: S
Use this article when
- You need deeper remediation guidance than the issue card can show.
- You want CMS-specific steps before handing the fix to a developer.
- You want a repeatable re-check path after shipping the change.
What this issue is
Your site is missing a robots.txt file. Search engine crawlers look for this file to understand which pages they can and cannot access.
Why it matters
Your site is missing a robots.txt file. Search engine crawlers look for this file to understand which pages they can and cannot access. This affects how clearly search engines understand the page and how persuasive it looks in search results.
How we detect it
- FreeSiteAudit flags this issue when the rule for CRAWL-ROBOTS-PRESENT-001 fails and the page evidence points to Http headers.
- You can usually confirm this by checking the page source or the relevant page settings inside your CMS.
Evidence examples
Check the affected page source, rendered output, or relevant CMS setting to confirm the missing or incorrect element.
How to fix it
- 1Create a robots.txt file in your site root
- 2Add User-agent: * and appropriate Allow/Disallow rules
- 3Reference your sitemap with a Sitemap: directive
How to re-check it
- Navigate to /robots.txt and confirm it returns a 200 response with valid directives
Related tools
This issue is best verified with the full FreeSiteAudit crawl rather than a single-point mini tool.