Indexability: what it is and how to check it

What is indexability

Indexability is the ability of search engine crawlers not only to crawl a page but also to add it to the search index. A page must be crawlable AND indexable to appear in search results.

The difference between crawling and indexing:

• Crawling — a crawler visits a page
• Indexing — a page is added to a search engine's database

What prevents indexing

1. Meta robots noindex

<!-- Страница не попадёт в индекс -->
<meta name="robots" content="noindex">

<!-- noindex только для Google -->
<meta name="googlebot" content="noindex">

2. HTTP header X-Robots-Tag

X-Robots-Tag: noindex
X-Robots-Tag: noindex, nofollow

It works like meta robots, but at the HTTP level — making it suitable for PDFs and other non-text files.

3. Canonical pointing to another page

<!-- Эта страница отдаёт «вес» другой — может не индексироваться самостоятельно -->
<link rel="canonical" href="https://example.com/main-page" />

4. Blocked in robots.txt

The crawler cannot crawl the page → it does not index it.

5. Server errors (5xx)

If the server continually returns 500/503, Google eventually removes the page from the index.

6. Duplicate content

Google may choose the “best” version of a page to index and deindex the others.

How to check indexing

Google Search Console

Enter a URL in the “URL Inspection” field — the tool will show:

• Whether the page is indexed
• When Googlebot last visited
• The canonical status
• Robots directives

The site: search operator

site:example.com/page  → проверить конкретную страницу
site:example.com       → общее количество страниц в индексе

Controlling indexing with robots directives

<!-- Полный контроль через meta robots -->
<meta name="robots" content="index, follow">   <!-- по умолчанию -->
<meta name="robots" content="noindex, follow"> <!-- не индексировать, но краулить ссылки -->
<meta name="robots" content="index, nofollow"> <!-- индексировать, не следовать ссылкам -->
<meta name="robots" content="noindex, nofollow"> <!-- ничего -->

Pages that should not be indexed

• Login pages (/login, /register)
• Admin panels (/admin)
• Search pages with parameters (?q=...)
• Technical pages (thank-you pages)
• Testing and development pages

Check with reChecker

Use the robots.txt Analyzer to check your website's indexing settings. The tool will show:

• The contents of robots.txt and blocked sections
• noindex directives on the page
• Whether a sitemap is present

FAQ

Can a page blocked by robots.txt Disallow be indexed? No. If a page is blocked in robots.txt, Google will not crawl or index it. Remove the block in robots.txt if the page needs to be indexed.

A page was in the index but disappeared — what happened? Possible reasons include a noindex directive being added, the server returning errors, the content becoming duplicated, or the page receiving a manual action. Check in Google Search Console.

How quickly does Google index new pages? For authoritative websites, it takes hours or days. For new websites, it takes weeks. Speed this up with a sitemap and Google URL Inspection Tool → “Request indexing.”