Robots.txt Generator
Choose what crawlers may visit, add your sitemap, and download a correct robots.txt. A built-in tester shows whether a given path is allowed for a given crawler under the rules.
How to use this tool
- Start from a preset or an empty file.
- Add rules per crawler (user agent) and your sitemap address.
- Test paths, then download robots.txt and upload it to the root of your site.
Understanding the results
Robots.txt controls crawling, not indexing. A disallowed page can still appear in search results if other sites link to it. To keep a page out of search results, let it be crawled and use a noindex robots meta tag or X-Robots-Tag header, or protect it with a password.
How it works
Rules follow RFC 9309: crawlers use the most specific group that names them (otherwise the * group), and within it the longest matching rule wins, with Allow winning a tie. * and $ wildcards are supported. The tester uses the same logic as our website checker.
Limitations
- Robots.txt is a request, not a lock: well-behaved crawlers follow it, others may not. Never rely on it to hide private content.
- Crawler names change; AI-crawler user agents listed here are those published by their operators and may need updating.
Privacy and data
Everything happens in your browser. Nothing you type is sent to our server.