Technical fix guide
No robots.txt File Found
No robots.txt file detected. While not strictly required, a robots.txt file helps guide search engine crawlers and is considered best practice.
Issue ID: TECH-ROBOTSTXT-001
Severity: medium
Impact: Med
Effort: S
Use this article when
- You need deeper remediation guidance than the issue card can show.
- You want CMS-specific steps before handing the fix to a developer.
- You want a repeatable re-check path after shipping the change.
What this issue is
No robots.txt file detected. While not strictly required, a robots.txt file helps guide search engine crawlers and is considered best practice.
Why it matters
No robots.txt file detected. While not strictly required, a robots.txt file helps guide search engine crawlers and is considered best practice. This affects crawlability, rendering, and whether the site can be interpreted correctly by search engines.
How we detect it
- FreeSiteAudit flags this issue when the rule for TECH-ROBOTSTXT-001 fails and the page evidence points to Http headers.
- You can usually confirm this by checking the page source or the relevant page settings inside your CMS.
Evidence examples
Check the affected page source, rendered output, or relevant CMS setting to confirm the missing or incorrect element.
How to fix it
- 1Create a robots.txt file in your site root
- 2At minimum, include: User-agent: *\nAllow: /
- 3Add your sitemap URL: Sitemap: https://yourdomain.com/sitemap.xml
How to re-check it
- Visit /robots.txt and confirm it returns a valid response
Related tools
This issue is best verified with the full FreeSiteAudit crawl rather than a single-point mini tool.