The most common technical SEO confusion: “I blocked the page in robots.txt, but it’s still in the index.” This isn’t a search engine bug—it’s a misunderstanding of what robots.txt actually does. It controls crawling only, not indexing. Let’s look at all three mechanisms separately and put them into a clear framework for when to use each.
Three mechanisms, three different tasks
Confusion arises because all three tools seem to be “about the same thing”—controlling a page’s visibility in search. In fact, they solve different problems at different levels:
| Mechanism | Level | What it controls | Where it is applied |
|---|---|---|---|
| robots.txt | Entire site | Whether a bot is allowed to crawl specific paths | File in the domain root |
| meta robots | Individual HTML page | Whether that page can be indexed and its links followed | Tag <meta> in <head> |
| X-Robots-Tag | Individual HTTP response | The same as meta robots, but for any file type | HTTP response header from the server |
robots.txt: about crawling, not indexing
robots.txt is a file in the site root that tells bots which sections they can crawl and which they cannot. The key point many people miss is that a robots.txt block does not guarantee that a page won’t be indexed.
If external links point to a page blocked in robots.txt, a search engine may index the URL itself (without the content it couldn’t crawl). In search results, it may appear as a snippet without a description, with a note such as “no information is available for this page.” The bot couldn’t access the page to read noindex because it was prohibited from visiting it.
User-agent: *
Disallow: /admin/
Disallow: /search
Allow: /robots.txt is the right tool when:
- you need to save crawl budget on sections that are clearly unimportant (internal search, filters, cart);
- you need to completely block crawling of service directories;
- you do NOT need to guarantee that a page is removed from the index—there are other tools for that.
You can check the current configuration and find potential errors with the robots.txt analyzer.
meta robots: about indexing an individual HTML page
The <meta name="robots"> tag is placed in the <head> of an HTML document and explicitly specifies what to do with that particular page:
<meta name="robots" content="noindex, follow">The most common values:
index, follow— index the page and follow its links (the default behavior; you don’t need to add the tag at all);noindex, follow— don’t index the page, but follow its links (pass link equity onward);noindex, nofollow— neither index the page nor follow its links;index, nofollow— index the page, but don’t pass link equity through its links (a rare case).
Unlike robots.txt, meta robots works only if the bot was able to crawl the page—that is, its path must not be blocked in robots.txt. This is the source of a classic mistake: a page is blocked both in robots.txt and with noindex in meta, but the desired effect doesn’t occur because the bot never reaches the tag that should have told it to exclude the page.
That’s exactly how we recently prevented low-quality, automatically generated pages on our own project from being indexed—read more in our article about scaled content abuse. The pages remained accessible through direct links (important for users who may already have had them bookmarked), but were given noindex, follow so they wouldn’t remain in search results.
X-Robots-Tag: the same thing, but for anything that isn’t HTML
The meta robots tag works only within HTML. PDFs, images, XML feeds, and API responses don’t have a <head> where it can be inserted. For these cases, the HTTP header X-Robots-Tag is used to set the same rules at the server response level:
HTTP/1.1 200 OK
Content-Type: application/pdf
X-Robots-Tag: noindexThe header is configured at the server or application level, for example in Nginx:
location ~* \.pdf$ {
add_header X-Robots-Tag "noindex, noarchive";
}X-Robots-Tag is the only option when you need to prevent the following from being indexed:
- PDF files, documents, price lists;
- images and media files;
- generated non-HTML responses (for example, JSON endpoints that have accidentally been indexed).
You can check which headers the server actually returns for a specific URL with the HTTP headers analyzer—sometimes a header is set in the code but doesn’t reach the client because of a proxy or CDN.
How these three mechanisms work together
The right mental model isn’t “choose one of the three,” but to understand at which stage each one comes into play:
- robots.txt determines whether the bot can access the URL at all.
- If it can, it downloads the content and sees meta robots (for HTML) or X-Robots-Tag (for everything else), which determine whether to index it and whether to pass link equity through its links.
- If the URL is blocked in robots.txt, steps 2 and 3 don’t happen at all—the bot sees neither the meta tag nor the header.
So here’s the practical rule: if a page definitely shouldn’t be included in the index, use noindex (meta robots or X-Robots-Tag), not a robots.txt block. Reserve robots.txt for cases where you need to save crawl budget on sections that don’t cause harm by being in the index, but simply don’t need to be crawled by the bot (for example, internal search pages with thousands of query variations).
Also, don’t confuse noindex with canonical: these are different tools for different situations. noindex says “this page shouldn’t be indexed at all,” while canonical says “here are several similar pages; treat this one as the main page.” If the goal is to consolidate duplicates rather than remove a page from search results entirely, the right tool is canonical, not noindex.
Common mistakes
Blocking with robots.txt instead of using noindex. The page remains in the index without its content because the bot can’t crawl it and read the directive.
Using noindex on pages blocked in robots.txt. The tag will never be read, so this combination is pointless.
X-Robots-Tag lost on the CDN. The header is configured on the backend, but the CDN or proxy overwrites or strips it. The page continues to be indexed even though everything looks correct in the code.
Leaving noindex in place after a test period. A common scenario: a page is temporarily blocked during development, and the tag is forgotten after launch, so the page simply never appears in search even though it should.
Checklist
- Make sure pages with
noindexaren’t also blocked in robots.txt - Check that X-Robots-Tag is used for PDFs and other non-HTML files if they need to be excluded
- Check canonical and noindex—don’t use them together on the same page with conflicting signals
- Check whether X-Robots-Tag is being lost at the CDN/proxy level
- Find pages that were accidentally left with
noindexafter development or testing - Use robots.txt only to save crawl budget, not as a way to remove a page from the index
Conclusion
All three mechanisms solve different problems at different levels, and most indexing problems arise from using the wrong tool for a given situation. It’s easy to remember: robots.txt is about crawling; meta robots and X-Robots-Tag are about indexing; and the difference between those two is simply whether the content is HTML. You can check how these mechanisms are configured on your site right now with the robots.txt analyzer and HTTP headers checker—inconsistencies are usually found in a few minutes.