robots.txt Explained
What a robots.txt file controls, how allow and disallow rules work, and the crucial thing it does NOT do — keep a page out of Google.
By the ToolsHub team · Updated May 1, 2026
robots.txt is a plain-text file at your site root that tells search-engine crawlers which URLs they may and may not request. It's the first thing most crawlers check when they visit. Create one with our Robots.txt Generator.
How the rules work
You group rules by crawler with User-agent, then list Disallow paths to keep them out and Allow paths for exceptions. A common file blocks an admin area while allowing everything else, and points to your sitemap with a Sitemap: line. It must live at example.com/robots.txt — crawlers won't look anywhere else.
The crucial thing it does NOT do
robots.txt controls crawling, not indexing. A page you disallow can still appear in Google if other sites link to it — Google just won't have crawled its contents. To truly keep a page out of results, allow crawling and add a noindex meta robots tag, so Google can see the instruction. Disallowing a page you also want de-indexed is a common mistake, because Google then can't read the noindex.