Robots.txt Tester
Check which robots.txt rules allow or disallow a URL for a chosen crawler.
1. Enter a URL or sample2. Run the check3. Review the results
How to use this tool
Enter your page information. Fetch mode checks public content using the AllgoTool server; paste mode processes the text without fetching that website. Review warnings before changing your website.
What this result means
This evaluates robots.txt rules only. Allowed does not mean indexed; disallowed does not guarantee removal from search. Firewalls, authentication, cached rules and crawler-specific behaviour can differ.