Robots.txt Generator
Create a robots.txt to control what crawlers can access.
How it works
Build a robots.txt file to tell search-engine crawlers which parts of your site they may or may not crawl. Choose a preset or set custom rules, then save the result to your site root.
User-agent: * Disallow: /admin
🔒 Generated entirely in your browser — nothing is uploaded.
🔒 Runs in your browser: tags, sitemaps and minification are generated locally from what you type — nothing is uploaded, no signup, no limits.
About the Robots.txt Generator
This free robots.txt generator builds the file that tells search-engine crawlers which parts of your site they may and may not crawl. Pick a preset or set custom allow/disallow rules per user-agent, add your sitemap, and save the result to your site root.
It runs entirely in your browser.
How to use it
- Choose a preset (allow all, block all, WordPress) or go custom.
- Add Disallow paths for areas crawlers should skip, and Allow paths for exceptions.
- Add your sitemap URL so crawlers can find it.
- Save the output as robots.txt at your site root.
Frequently asked questions
What is robots.txt?
A plain-text file at your site root that gives crawlers rules about which URLs they may request. It's the first thing most search engines check when they visit.
Does robots.txt keep a page out of Google?
Not reliably. Disallow stops crawling, but a blocked URL can still be indexed if linked elsewhere. To keep a page out of results, use a noindex meta robots tag and allow crawling so it can be seen.
Where does the file go?
It must be at the root: https://example.com/robots.txt. Crawlers won't look for it anywhere else.
Is my configuration uploaded?
No. The file is generated in your browser.