What Is Robots.txt?

Robots.txt is a text file at the root of your website that tells search engine crawlers which parts of your site they can and can’t access. It’s a simple but powerful file: used correctly, it guides crawlers efficiently; used carelessly, it can accidentally block important pages from being crawled. It controls crawling — not indexing — which is a distinction that matters.

Here’s what robots.txt does, the critical thing it does not do, and how to avoid the common mistakes.

What robots.txt does

The robots.txt file lives at yoursite.com/robots.txt and gives crawlers instructions before they crawl. Using simple Allow and Disallow rules, it tells bots which paths they may or may not crawl. Common uses:

• Keeping crawlers out of admin areas, internal search results, or duplicate/parameter URLs.
• Preventing wasted crawling on low-value sections.
• Pointing crawlers to your sitemap (a common line in the file).

It’s a set of polite instructions that legitimate crawlers like Googlebot respect.

The critical distinction: crawling ≠ blocking from Google

Here’s the misunderstanding that causes real damage: robots.txt controls crawling, not indexing. Disallowing a page in robots.txt tells crawlers not to fetch it — but the page can still get indexed (and appear in results, often with no description) if it’s linked from elsewhere, because Google knows it exists even without crawling it. So robots.txt is the wrong tool for keeping a page out of search results. To do that, you use a noindex tag — which requires the page to be crawlable so the tag can be seen. Blocking in robots.txt and adding noindex is self-defeating: the crawler can’t see the noindex because it’s blocked.

Common robots.txt mistakes

• Accidentally disallowing the whole site (Disallow: /) — a catastrophic, surprisingly common error that de-crawls everything.
• Using robots.txt to hide pages from search — wrong tool; use noindex instead.
• Blocking CSS/JS that Google needs to render the page properly.
• Blocking a page you also noindex — the crawler never sees the noindex.
• Assuming it’s security — robots.txt is public and only requests compliance; it doesn’t protect private content. Never rely on it to hide sensitive data.

How to use robots.txt well

Keep it simple and deliberate: disallow only what genuinely shouldn’t be crawled (admin paths, true duplicates, low-value parameter URLs), reference your sitemap, don’t block resources Google needs to render pages, and test it in search-engine tools before trusting it. Because a single wrong line can hide your whole site, it’s worth double-checking — a tiny file with outsized power, and a standard checkpoint in any technical SEO audit. (SEO consulting covers robots.txt as part of the technical foundation.)

Frequently asked questions

What is robots.txt in simple terms? Robots.txt is a text file at the root of your website that tells search engine crawlers which parts of your site they can and can’t access. Using simple allow and disallow rules, it guides bots on where to crawl. It controls crawling, not indexing, and a single wrong line can accidentally block important pages, so it’s powerful but easy to misuse.

What does robots.txt do? It gives instructions to search engine crawlers before they crawl your site, telling them which paths they may or may not access using allow and disallow rules. It’s commonly used to keep crawlers out of admin areas, duplicate URLs, or low-value sections, and to point them to your sitemap. Legitimate crawlers like Googlebot respect these instructions.

Does robots.txt stop a page from being indexed? No — this is a common and costly misunderstanding. Robots.txt controls crawling, not indexing. A page disallowed in robots.txt can still be indexed and appear in results (often without a description) if it’s linked from elsewhere. To keep a page out of search results, use a noindex tag instead, which requires the page to remain crawlable.

What is the difference between robots.txt and noindex? Robots.txt controls whether crawlers can fetch a page, while a noindex tag tells search engines not to store a page in their index. To keep a page out of search results, use noindex — and don’t also block it in robots.txt, because then the crawler can’t see the noindex tag. They serve different purposes and shouldn’t be combined for the same page.

Where is the robots.txt file located? The robots.txt file must be at the root of your domain, accessible at yoursite.com/robots.txt. Search engine crawlers check that specific location before crawling. If it’s placed anywhere else, crawlers won’t find it, and your rules won’t apply. There’s one robots.txt file per site, controlling crawler access across the whole domain.

Can robots.txt hide sensitive information? No. Robots.txt is publicly accessible — anyone can read it — and it only requests that compliant crawlers stay away; it doesn’t enforce anything or protect content. Listing a private path in robots.txt can even draw attention to it. To protect sensitive content, use proper authentication or server-level restrictions, never robots.txt.

What happens if I make a mistake in robots.txt? A mistake can have big consequences — for example, “Disallow: /” blocks crawlers from your entire site, which can cause pages to drop from search over time. Blocking CSS or JavaScript can stop Google from rendering pages properly. Because a single line has such reach, always test robots.txt in search-engine tools before relying on it.

Written by Bryan Collins, SEO & AEO strategist. Want your crawl directives right? See SEO strategy work or run a free search presence audit.