Website checks · Free tool
Free Robots.txt Tester
Check whether a crawler is allowed to request a path. Read the published robots.txt and test the URL against the matching user-agent group.
100% freePublic website lookup
Your report will appear here after you run a check.
Getting started
How to use the robots.txt tester
- Enter the page URL and crawler name.
- Run the test to fetch the site’s robots.txt.
- Review the matching rule, decision and sitemap declarations.
Put the results to work
Understand your report
The tester applies matching user-agent groups, wildcard rules and the longest matching path. An Allow rule wins an equal-length tie. The result describes crawling permission for this URL, while the page’s own meta robots and HTTP headers control separate indexing instructions.
A little more clarity
Frequently asked questions
Does robots.txt control whether a page is indexed?
Robots.txt controls crawling. To inspect indexing directives in a page response, use the Website SEO Checker.
What if there is no robots.txt?
A 404 or 410 robots.txt response is treated as no published crawl restrictions. Server errors are shown as inconclusive.