AI Crawler Blocker (robots.txt)
🔒 In your browserBlock ChatGPT, Claude & other AI bots from your site — without hurting SEO.
How it works
Choose which AI crawlers to block from training on or fetching your site, and get a ready-to-use robots.txt. Normal search engines (Google, Bing) stay allowed, so your SEO is unaffected. Put the file at your site root. Generated in your browser.
robots.txt
# AI crawler rules — generated by ToolsHub # Blocks the selected AI/LLM crawlers. Normal search engines are unaffected. # OpenAI — Trains OpenAI models User-agent: GPTBot Disallow: / # OpenAI — Fetches pages when a user asks ChatGPT User-agent: ChatGPT-User Disallow: / # OpenAI — Indexes for ChatGPT Search User-agent: OAI-SearchBot Disallow: / # Anthropic — Trains Claude models User-agent: ClaudeBot Disallow: / # Anthropic — Fetches pages for Claude when a user asks User-agent: Claude-User Disallow: / # Anthropic — Indexes for Claude search User-agent: Claude-SearchBot Disallow: / # Google — Trains Gemini / Vertex AI (does not affect Search) User-agent: Google-Extended Disallow: / # Apple — Trains Apple AI (does not affect Siri/Search index) User-agent: Applebot-Extended Disallow: / # Perplexity — Indexes for Perplexity answers User-agent: PerplexityBot Disallow: / # Common Crawl — Open crawl used to train many models User-agent: CCBot Disallow: / # ByteDance — Trains ByteDance / Doubao models User-agent: Bytespider Disallow: / # Amazon — Powers Alexa & Amazon AI User-agent: Amazonbot Disallow: / # Meta — Trains Meta AI / Llama User-agent: meta-externalagent Disallow: / # Cohere — Fetches pages for Cohere User-agent: cohere-ai Disallow: / # All other crawlers (Google, Bing, etc.) — allowed User-agent: * Allow: /
robots.txt is honoured voluntarily — reputable AI crawlers (GPTBot, ClaudeBot, Google-Extended…) respect it, but it can't force compliance. Google-Extended and Applebot-Extended only opt you out of AI training, not search indexing. For general crawl rules, see the Robots.txt Generator. Built in your browser.
🔒 Runs in your browser: tags, sitemaps and minification are generated locally from what you type — nothing is uploaded, no signup, no limits.
About the AI Crawler Blocker (robots.txt)
AI companies crawl the web to train models and to answer questions in tools like ChatGPT, Claude and Perplexity. This free AI crawler blocker builds a robots.txt that opts your site out of the crawlers you choose — while leaving normal search engines like Google and Bing fully allowed, so your SEO is unaffected.
The file is generated in your browser; you just place it at your site root.
How to use it
- Tick the AI crawlers you want to block — or use Block all / Allow all.
- Each bot shows who runs it and what it's used for, so you can decide case by case.
- Optionally add your sitemap URL for the search engines you still allow.
- Copy or download the robots.txt and place it at https://yoursite.com/robots.txt.
Training vs. crawling
Some directives only affect AI training, not access. Google-Extended and Applebot-Extended opt you out of model training but do not change how Google Search or Siri index your pages. Others, like GPTBot, ChatGPT-User and PerplexityBot, control whether those services fetch your pages at all. The tool notes which is which so you can choose deliberately.
Frequently asked questions
Will blocking AI crawlers hurt my Google ranking?
No. This blocks AI-specific user-agents only. Googlebot, Bingbot and other search crawlers are left allowed, so normal search indexing and ranking are unaffected.
Does robots.txt actually stop AI bots?
It's honoured voluntarily. Reputable crawlers such as GPTBot, ClaudeBot and Google-Extended respect robots.txt, but it can't force compliance from bots that choose to ignore it.
What's the difference between GPTBot and ChatGPT-User?
GPTBot crawls pages to train OpenAI's models; ChatGPT-User fetches a page on demand when a user asks ChatGPT about it. You can block them independently.
How do I block AI training but keep search working?
Block the training user-agents (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot) and leave search crawlers allowed — which is exactly what this generator produces.
Related searches
Related SEO tools
🔗 Embed this tool on your website — free
Copy this and paste it into your page's HTML. The tool runs in the visitor's browser, just like here. Change height to fit, or add the optional auto-resize snippet below. Add ?theme=dark to the URL for dark mode.
<iframe src="https://toolhq.dev/embed/ai-crawler-blocker/" title="AI Crawler Blocker (robots.txt) — ToolsHub" width="100%" height="520" style="border:1px solid #e5e7eb;border-radius:12px;max-width:680px" loading="lazy"></iframe>Optional: auto-resize the iframe height
Add this once on the same page so the iframe grows to fit the tool:
<script>
addEventListener("message", function (e) {
if (e.data && e.data.type === "toolshub:resize") {
document.querySelectorAll('iframe[src*="/embed/"]').forEach(function (f) {
if (f.contentWindow === e.source) f.style.height = e.data.height + "px";
});
}
});
</script>