Robots.txt Generator
Create a robots.txt file with crawl rules and your sitemap URL.
1. Add your input2. Run the tool3. Review and download
How to use this tool
Enter your website URL, add any paths crawlers should avoid, and download the file.
Limits and useful notes
Upload robots.txt to your domain root. Robots.txt is not access control and does not keep private information secure. Blocking crawling does not guarantee removal from search results.
Files and inputs are processed in your browser. This tool does not upload them to the AllgoTool server.