Robots.txt Generator
🔒 In your browserCreate a robots.txt to control what crawlers can access.
How it works
Build a robots.txt file to tell search-engine crawlers which parts of your site they may or may not crawl. Choose a preset or set custom rules, then save the result to your site root.
User-agent: * Disallow: /admin
🔒 Generated entirely in your browser — nothing is uploaded.
🔒 Runs in your browser: tags, sitemaps and minification are generated locally from what you type — nothing is uploaded, no signup, no limits.
About the Robots.txt Generator
This free robots.txt generator builds the file that tells search-engine crawlers which parts of your site they may and may not crawl. Pick a preset or set custom allow/disallow rules per user-agent, add your sitemap, and save the result to your site root.
It runs entirely in your browser.
How to use it
- Choose a preset (allow all, block all, WordPress) or go custom.
- Add Disallow paths for areas crawlers should skip, and Allow paths for exceptions.
- Add your sitemap URL so crawlers can find it.
- Save the output as robots.txt at your site root.
What robots.txt does — and doesn't
robots.txt controls crawling, not indexing. Disallowing a path stops compliant crawlers from fetching it, but a blocked URL can still appear in search results if other sites link to it. To truly keep a page out of Google, allow crawling and add a noindex meta tag so the instruction can be seen. Disallowing a page you also want de-indexed is a common mistake.
Frequently asked questions
What is robots.txt?
A plain-text file at your site root that gives crawlers rules about which URLs they may request. It's the first thing most search engines check when they visit.
Does robots.txt keep a page out of Google?
Not reliably. Disallow stops crawling, but a blocked URL can still be indexed if linked elsewhere. To keep a page out of results, use a noindex meta robots tag and allow crawling so it can be seen.
Where does the file go?
It must be at the root: https://example.com/robots.txt. Crawlers won't look for it anywhere else.
Is my configuration uploaded?
No. The file is generated in your browser.
Where does the robots.txt file go?
At your site root — https://example.com/robots.txt. Crawlers only look there, so it won't work at a subpath. Add a Sitemap: line pointing to your sitemap.xml.
Related searches
Learn more
Related SEO tools
🔗 Embed this tool on your website — free
Copy this and paste it into your page's HTML. The tool runs in the visitor's browser, just like here. Change height to fit, or add the optional auto-resize snippet below. Add ?theme=dark to the URL for dark mode.
<iframe src="https://toolhq.dev/embed/robots-txt-generator/" title="Robots.txt Generator — ToolsHub" width="100%" height="520" style="border:1px solid #e5e7eb;border-radius:12px;max-width:680px" loading="lazy"></iframe>Optional: auto-resize the iframe height
Add this once on the same page so the iframe grows to fit the tool:
<script>
addEventListener("message", function (e) {
if (e.data && e.data.type === "toolshub:resize") {
document.querySelectorAll('iframe[src*="/embed/"]').forEach(function (f) {
if (f.contentWindow === e.source) f.style.height = e.data.height + "px";
});
}
});
</script>