Robots.txt Generator
Create custom robots.txt files to control search engine crawlers.
📖 About robots.txt
The robots.txt file tells search engine crawlers which pages or files they can or can't request from your site. It's placed in the root directory of your website (e.g., example.com/robots.txt).
Common Directives
- User-agent - Specifies which crawler the rules apply to
- Allow - Allows crawling of a specific path
- Disallow - Blocks crawling of a specific path
- Sitemap - Points to your XML sitemap location
- Crawl-delay - Sets delay between crawler requests
💡 Pro Tip: Remember that robots.txt is publicly accessible. Don't use it to hide sensitive content - use password protection or noindex meta tags instead.
❓ Frequently Asked Questions
Upload the robots.txt file to the root directory of your website. It should be accessible at yourdomain.com/robots.txt. Use FTP or your hosting file manager to upload it.
No, robots.txt only prevents crawling, not indexing. Pages can still appear in search results if other sites link to them. Use the "noindex" meta tag to prevent indexing.
Crawl delay tells crawlers to wait a specified number of seconds between requests. This can help reduce server load, but may slow down indexing. Google ignores crawl delay - use Google Search Console instead.