SEO generators · Free tool
Free Robots.txt Generator
Build a simple robots.txt file you can review and download. Choose a crawler, add path rules and include your sitemap location.
100% freeRuns in your browser
Add your details or try an example to get started.
Getting started
How to use the robots.txt generator
- Set the user-agent or use * for all crawlers.
- Enter one allowed or disallowed path per line.
- Download the file and publish it at your domain’s /robots.txt URL.
Put the results to work
Understand your report
Robots rules should describe the parts of a site that crawlers may request. The generator validates path formatting and outputs a plain-text file. After publishing, use the Robots.txt Tester to check important page URLs against the actual file served by your website.
A little more clarity
Frequently asked questions
Where must robots.txt be hosted?
At /robots.txt on the same hostname and protocol as the website you want it to apply to.
Can robots.txt protect private information?
No. Use authentication for private pages. Robots rules are public crawl instructions.