What is a robots.txt file?

robots.txt is a text file at your site's root that tells crawlers which parts they may or may not access. It guides well-behaved bots but doesn't enforce security.

Does robots.txt stop a page from being indexed?

Not reliably. Disallow stops crawling, but a blocked URL can still be indexed if linked elsewhere. To keep a page out of search, use a noindex meta tag instead.

Where does robots.txt go?

It must sit at the root of your domain, at yourdomain.com/robots.txt. Crawlers look for it there before visiting other pages.

How do I create a robots.txt file?

List user-agent groups with Allow and Disallow rules plus a link to your sitemap. Our Robots.txt Generator builds a valid file from simple options.

SEO · 3 min read

robots.txt Explained

What a robots.txt file controls, how allow and disallow rules work, and the crucial thing it does NOT do — keep a page out of Google.

By the ToolsHub team · Updated May 1, 2026

robots.txt is a plain-text file at your site root that tells search-engine crawlers which URLs they may and may not request. It's the first thing most crawlers check when they visit. Create one with our Robots.txt Generator.

How the rules work

You group rules by crawler with User-agent, then list Disallow paths to keep them out and Allow paths for exceptions. A common file blocks an admin area while allowing everything else, and points to your sitemap with a Sitemap: line. It must live at example.com/robots.txt — crawlers won't look anywhere else.

The crucial thing it does NOT do

robots.txt controls crawling, not indexing. A page you disallow can still appear in Google if other sites link to it — Google just won't have crawled its contents. To truly keep a page out of results, allow crawling and add a noindex meta robots tag, so Google can see the instruction. Disallowing a page you also want de-indexed is a common mistake, because Google then can't read the noindex.

Common robots.txt mistakes

A small slip in robots.txt can hide your whole site or, worse, expose pages you meant to keep private. Watch for these:

Using Disallow: / in a template and shipping it to production — it blocks everything.
Expecting Disallow to hide a page from search; it only stops crawling, so use a noindex tag to de-index.
Forgetting to link your sitemap, which helps crawlers find your pages.

Build a correct file from simple options with our Robots.txt Generator, and remember it must live at your domain's root.

Frequently asked questions

What is a robots.txt file?: robots.txt is a text file at your site's root that tells crawlers which parts they may or may not access. It guides well-behaved bots but doesn't enforce security.
Does robots.txt stop a page from being indexed?: Not reliably. Disallow stops crawling, but a blocked URL can still be indexed if linked elsewhere. To keep a page out of search, use a noindex meta tag instead.
Where does robots.txt go?: It must sit at the root of your domain, at yourdomain.com/robots.txt. Crawlers look for it there before visiting other pages.
How do I create a robots.txt file?: List user-agent groups with Allow and Disallow rules plus a link to your sitemap. Our Robots.txt Generator builds a valid file from simple options.

Try the tool

Robots.txt Generator →