What is Meta Robots
Meta robots is an HTML tag in the <head> section that controls how search engine crawlers behave toward a specific page. Unlike robots.txt (which controls crawling across the entire site), meta robots works at the individual page level.
<meta name="robots" content="noindex, nofollow">Main Meta Robots directives
| Directive | Meaning |
|---|---|
index | The page can be indexed (default) |
noindex | Do not add the page to the index |
follow | Follow links on the page (default) |
nofollow | Do not follow links on the page |
noarchive | Do not save a cached copy |
nosnippet | Do not show a snippet in search results |
noimageindex | Do not index images on the page |
none | Equivalent to noindex, nofollow |
all | Equivalent to index, follow |
Combining directives
<!-- Не индексировать, но следовать ссылкам -->
<meta name="robots" content="noindex, follow">
<!-- Индексировать, но не следовать ссылкам -->
<meta name="robots" content="index, nofollow">
<!-- Не индексировать, не следовать, не кешировать -->
<meta name="robots" content="noindex, nofollow, noarchive">Directives for specific crawlers
<!-- Только для Googlebot -->
<meta name="googlebot" content="noindex">
<!-- Только для Yandex -->
<meta name="yandex" content="noindex">
<!-- Для всех роботов -->
<meta name="robots" content="noindex">The X-Robots-Tag HTTP header
For files without HTML (PDFs, images), use the HTTP header:
# Nginx: noindex для PDF файлов
location ~* \.pdf$ {
add_header X-Robots-Tag "noindex, nofollow";
}HTTP/1.1 200 OK
X-Robots-Tag: noindex
Content-Type: application/pdfWhen to use noindex
Add noindex to pages that should not appear in search results:
Страницы входа и регистрации → /login, /register
Административные панели → /admin/*
Страницы корзины/оформления → /cart, /checkout
Страницы «Спасибо» → /thank-you
Технические страницы → /api-test, /debug
Дублированные URL с параметрами → /?sort=price&color=red
Предварительные версии → /staging/*
Robots.txt vs Meta Robots: which to choose
| Aspect | robots.txt | meta robots |
|---|---|---|
| Action | Blocks crawling | Blocks indexing |
| Level | Entire site/section | One page |
| Crawling | Blocks it | Does not block it |
| Precision | URL patterns | Specific page |
To block indexing, use meta robots. To save crawl budget, use robots.txt Disallow.
Check with reChecker
Use the robots.txt analyzer to check your site's indexing settings. The tool analyzes robots.txt and shows blocked sections.
FAQ
What takes precedence if a page is blocked by robots.txt Disallow and has meta noindex? robots.txt Disallow blocks crawling—the crawler will not visit the page or read the meta noindex. If you want Google to see noindex, remove Disallow from robots.txt. Crawling must be allowed for noindex pages.
Does noindex remove a page from the index immediately? No. Google must first revisit the page and read noindex. The page will remain in the index until the next crawl (days or weeks). To request this directly: Google Search Console → URL Inspection → Request Indexing (paradoxically, this also speeds up deindexing).
Is noindex needed for duplicates if there is a canonical? Canonical is a “hint” to Google. Noindex is a directive. For important duplicates, you can use both for reliability, but canonical is usually sufficient.