Robots.txt Generator
Build a valid robots.txt file to control how search engines crawl your site.
About Robots.txt Generator
Generate a correct robots.txt file to tell search engine crawlers which parts of your site they may or may not access. Set the user-agent, allow or disallow specific paths, add an optional crawl delay, and link your sitemap. Copy or download the file and place it at your domain root.
How to use Robots.txt Generator
- 1
Choose which crawler the rules apply to (use * for all).
- 2
Decide whether to allow crawling by default, then list paths to allow or disallow.
- 3
Optionally add a crawl-delay and your sitemap URL.
- 4
Copy or download the file and upload it to your site root.
Frequently Asked Questions
Where does robots.txt go?
It must live at the root of your domain, e.g. https://example.com/robots.txt. Crawlers always look for it there.
Does Disallow guarantee a page won't be indexed?
No. Disallow blocks crawling, but a page can still be indexed if linked elsewhere. Use a noindex meta tag to keep a page out of results.
More SEO Tools
Keyword Density Checker
Analyze how often each keyword and phrase appears in your content, with density percentages.
Meta Tag Generator
Generate clean HTML meta tags — title, description, keywords, robots and more.
Open Graph Generator
Generate Open Graph and Twitter Card meta tags with a live social-share preview.