What is a meta robots tag? Page-level instructions like noindex and nofollow that control how search engines handle a page — and how it differs from
What Is a Meta Robots Tag?
A meta robots tag is a snippet of HTML in a page’s <head> that tells search engines how to handle that specific page — most commonly whether to index it (noindex) and whether to follow its links (nofollow). Unlike robots.txt, which controls crawling at the site level, the meta robots tag controls indexing and link-following at the page level. Getting that distinction right is critical, because confusing the two is a classic, costly SEO mistake.
Here’s what the meta robots tag does, its main values, and how it differs from robots.txt.
What a meta robots tag does
The meta robots tag lives in the HTML <head> of a page and gives page-specific instructions to search engine crawlers. A typical example looks like <meta name="robots" content="noindex, nofollow">. It lets you control, for that one page, whether it should appear in search results and whether search engines should follow the links on it.
The main values
index(default) — this page may be included in search results.noindex— do not include this page in search results. The key tool for keeping specific pages out of the index (thank-you pages, internal search results, thin utility pages).follow(default) — follow the links on this page.nofollow— don’t follow the links on this page.- Others — like
noarchive(no cached copy) andnosnippet(no snippet in results).
You can combine them, e.g. noindex, follow (keep it out of the index but still follow its links).
The critical distinction: meta robots vs. robots.txt
This is the trap that catches people, so it’s worth being precise:
- robots.txt controls crawling — whether search engines are allowed to access a page or section.
- Meta robots
noindexcontrols indexing — whether a crawled page is allowed into the search results.
Here’s the crucial consequence: to reliably noindex a page, Google must be able to crawl it and see the tag. If you block a page in robots.txt, Google can’t crawl it — which means it may never see your noindex tag, and the page can still end up in the index (as a bare URL). So blocking in robots.txt and expecting that to remove a page from search is a common, self-defeating error. If your goal is to keep a page out of search results, use noindex and let the page be crawled — don’t block it in robots.txt.
When to use the meta robots tag
noindex— for pages you don’t want in search: thank-you/confirmation pages, login pages, internal search result pages, thin or duplicate utility pages, staging content.nofollow(page-level) — rarely needed sitewide; usually link-level attributes are more appropriate.- Leave the default (
index, follow) for all the pages you do want ranking — which is most of them.
Why it matters
The meta robots tag gives you precise, page-level control over what appears in search — useful for keeping low-value or private-but-public pages out of results without hurting the rest of your site. The main thing is understanding it correctly: noindex (meta robots) removes a page from results; robots.txt controls crawling, not indexing. Use each for its actual job. (SEO consulting sets indexing controls correctly so the right pages rank and the wrong ones stay out.)
Frequently asked questions
What is a meta robots tag in simple terms? A meta robots tag is a snippet of HTML in a page’s head section that tells search engines how to handle that specific page — most commonly whether to index it (using noindex) and whether to follow its links (using nofollow). It gives page-level control over whether a page appears in search results and how its links are treated, unlike robots.txt, which controls crawling at the site level.
What does the noindex tag do? The noindex value in a meta robots tag tells search engines not to include that page in search results. It’s the main tool for keeping specific pages out of the index, such as thank-you pages, login pages, internal search results, or thin utility pages. For noindex to work, search engines must be able to crawl the page and see the tag, so the page shouldn’t be blocked in robots.txt.
What is the difference between meta robots and robots.txt? Robots.txt controls crawling — whether search engines are allowed to access a page or section — while the meta robots noindex tag controls indexing, meaning whether a crawled page can appear in search results. They govern different stages. The key consequence is that to reliably noindex a page, Google must be able to crawl it to see the tag, so you shouldn’t block it in robots.txt.
Why is my noindexed page still in Google? The most common reason is that the page is also blocked in robots.txt, so Google can’t crawl it and therefore can’t see the noindex tag — meaning it may still appear as a bare URL in results. To fix this, remove the robots.txt block so Google can crawl the page and read the noindex instruction, which then removes it from search results properly.
When should I use noindex? Use noindex for pages you don’t want appearing in search results but that still need to exist for users, such as thank-you and confirmation pages, login pages, internal search result pages, thin or duplicate utility pages, and staging content. Leave the default index setting on all the pages you do want to rank, which is the majority of your important content.
Can I combine noindex and follow? Yes. A common combination is “noindex, follow,” which keeps a page out of search results while still allowing search engines to follow the links on it and pass through to other pages. You can mix the values depending on your goal — for example, noindex with follow is useful for utility pages you don’t want indexed but that link to pages you do want crawled.
Does a meta robots tag affect rankings? Indirectly. The tag itself doesn’t boost rankings, but using noindex correctly keeps low-value or duplicate pages out of search results, which helps focus your site’s indexed content on the pages that matter. Misusing it — accidentally noindexing important pages — can remove valuable pages from search entirely. So its impact comes from controlling what’s indexed, not from any direct ranking signal.
Written by Bryan Collins, SEO & AEO strategist. Want your indexing controls set correctly? See SEO strategy work or run a free Google presence audit.