AI Bot Robots.txt Tester
Compare robots.txt permissions for a selection of AI crawler product tokens.
1. Enter a URL or sample2. Run the check3. Review the results
How to use this tool
Enter your page information. Fetch mode checks public content using the AllgoTool server; paste mode processes the text without fetching that website. Review warnings before changing your website.
What this result means
Results describe declared robots.txt permissions for the listed tokens, not actual bot traffic or enforced access. This does not test firewall rules, IP allowlists, bot identity, training usage or indexing.