A page that is absent from search can be in several different states. A crawler may not have discovered one URL, may have failed to load another, or may have processed a third successfully but consolidated it with a duplicate. Changing the text, requesting a recrawl, and rebuilding the sitemap all at once will muddy the evidence about the cause. It is more reliable to follow the page’s path in the same order as the search engine.
Distinguish between three verification results:
| What has been established | What the conclusion is based on | What still needs to be determined |
|---|---|---|
The page returns 200 | The server’s current GET response | Whether the correct document was retrieved and its resources are accessible |
| No technical blocks were found | robots.txt, meta directives, headers, and content | How the search engine processed the URL and which canonical it selected |
| The URL is in the index | Search engine data about the specific URL | When it crawled the page and which version is stored |
The first result does not replace the second, and the second does not prove the third. Under Google’s technical requirements, an accessible page with 200 and indexable content may be eligible for indexing; inclusion in the index is not guaranteed.
First, establish exactly what the problem is
The query site: is useful for a quick, approximate check, but it does not prove that a specific URL is absent from the index. For Google, open the URL Inspection tool in Search Console: it shows information about the last known indexed version, while a separate live test checks current accessibility. These are two different states and should not be conflated—Google’s official help explicitly distinguishes the indexed and live versions.
For Yandex, use the page check in Webmaster, and compare the date of the last crawl with information about whether the page appears in search. A check of the current response does not replace this information. Record the full URL, including the protocol, subdomain, path capitalization, and parameters. Sometimes the team discusses https://example.ru/page, while the sitemap contains the version with a trailing slash, the canonical points to the version without one, and internal links use HTTP. In that case, the question is no longer “why is the page not being indexed?” but “which of these versions does the system consider primary?”
Record the publication date or date of a significant update, how the URL was discovered, and the expected search demand. A new document needs to be crawled; for an older document that has suddenly dropped out of the index, investigate a change or failure.
Check the server response outside a browser session
Open the URL in a private window, then save the headers and body of the GET response, following redirects:
curl -sS -L -D /tmp/index-check-headers.txt \
-o /tmp/index-check-page.html \
-w 'HTTP %{http_code}\nURL %{url_effective}\n' \
https://example.com/pageThe command does not execute JavaScript. Use the saved headers to examine the redirect chain, and look for the main page text in the HTML. A HEAD request curl -I can also be used: sometimes a server handles HEAD and GET differently. The final response should have code 200, a reasonable redirect chain, and the same document that an unauthenticated user sees. A redirect to the home page, a city selection page, a login form, or a security placeholder needs to be investigated separately.
Check several user-agent variants. Changing the user-agent string in curl can reveal differences in responses, but does not confirm access by a real crawler from its IP address; that requires logs and verification that the address belongs to the crawler. A CDN, WAF, or anti-bot system may sometimes serve a search crawler 403, 429, empty HTML, or a JavaScript check, even though a regular browser receives the page. A random 5xx should not be assessed with a single request either: repeat the check at different times and review the origin server logs. An explanation of status codes and common redirects is available in the article about HTTP status codes.
The criterion for this step is simple: the crawler must be able to reliably retrieve meaningful HTML at the final URL. If it cannot, meta tags and text quality do not matter yet.
Rule out direct blocks on crawling and indexing
Check robots.txt for the specific search crawler and path. The rule Disallow limits crawling, but is not a way to remove a URL already known to search engines. That is why combining a crawl block with noindex is particularly problematic: the crawler may be unable to load the page and see the instruction to exclude it. This distinction is described in Google’s robots.txt rules.
Next, inspect the source HTML and HTTP headers. Look for meta name="robots", separate directives for Googlebot or Yandex, as well as X-Robots-Tag. The value noindex may be added not by the page template, but by a proxy, security plugin, or test environment setting. Make sure the directive does not change unexpectedly after JavaScript runs.
To exclude a page from Google, the crawler must be able to retrieve noindex—in the HTML or response header. Check both locations using the official instructions.
The result should be unambiguous: the URL is allowed to be retrieved, and the actual response contains no instruction not to index it. If a block was removed, record when it was changed and request a recrawl only after checking the response again.
Compare the canonical, sitemap, and actual URL
A page with 200 and without noindex may still not appear as a standalone result if the search engine considers it a duplicate. Check rel="canonical" in the source and rendered HTML. The canonical target should return 200, be available for indexing, and match the meaning of the document. Pointing to a category, the home page, or a product with a different variant does not fix duplicates—it suggests consolidating different entities.
Compare four places: the page address, the canonical, the URL in the sitemap, and the final redirect destination. For the primary indexable version, these should usually point to the same variant. Google describes redirects and rel="canonical" as strong signals, and sitemap inclusion as a weaker one; when signals conflict, the final choice is up to the system. Details are available in the official documentation and our canonical guide.
If the search console selected a different canonical, compare the documents: are the main text, title, images, product parameters, and language the same? Fix the source of duplication or conflicting markup. Resubmitting the same URL will not fix the cause.
Make sure the crawler can discover the page
A sitemap helps communicate a URL, but does not replace site navigation. Find the incoming internal links to the URL. They should be standard HTML links with href, located on accessible pages and leading directly to the canonical version. A link that appears only after a site search, filter selection, or click may remain inaccessible to crawlers.
Orphan pages often occur after a catalog migration, the removal of a recommendations block, or a URL change. Add the page to a relevant category, breadcrumb trail, or topical article. Do not create dozens of mechanical links in the footer: the goal is to integrate the document into a clear structure. The principles for choosing source pages and anchor text are covered in the article about internal linking.
After the change, the crawler should be able to find the URL from the home page or section map within a limited number of hops. The sitemap should contain only the canonical indexable version, with the correct modification date.
Assess the content from a search engine’s perspective
Once technical blocks have been ruled out, compare the source HTML, rendered DOM, and screen visible to the user. The main text, heading, product attributes, and links should not depend on an API request that fails for the crawler. An empty shell with a loading indicator formally returns 200, but provides no material to index.
Google can execute JavaScript, so the absence of text in the source alone does not prove that the content is inaccessible to Google. Check the rendered page and resource accessibility in Search Console. If the source response contains noindex, Google may skip rendering: removing the block with a script is unreliable. These cases are covered in Google’s JavaScript SEO documentation. SSR or static HTML containing the main content makes it easier to retrieve, including for crawlers that do not execute JavaScript.
Next, ask specific questions. Is the page different from adjacent pages? Does it address a distinct user need? Are there hundreds of URLs that differ only by city, one sentence, or product order? Do the title and H1 match the actual content? A search engine may crawl a weak or nearly duplicated document without selecting it to appear in results.
This is not a reason to increase the text to some arbitrary length. Add missing attributes, terms, first-hand expertise, comparisons between options, or answers to real questions—only what makes the page stand on its own. You can deliberately assign noindex to a utility page with no search purpose; there is no need to try to get it indexed.
Separate the fix from confirmation of the result
Change one group of causes at a time. First restore accessibility, then remove the block, consolidate the URL to one version, add a discovery path, and only then improve the content. Record the initial state, the change, the deployment date, and how it was checked. This makes it possible to distinguish a recrawl delay from a fix that did not work a week later.
After deployment, repeat the HTTP check, live test, and crawl with an internal crawler. An indexing request speeds up the submission of a signal, but does not override technical or quality requirements. reChecker’s technical audit checks the page response, meta directives, canonical, and links; its report can be attached to the log alongside search console data. These signals can indicate possible obstacles. They do not confirm that a specific URL is already in the Google or Yandex index, and do not measure the number of indexed pages. Check Search Console and Webmaster for confirmation. The search engine decides whether to include a URL.
In the technical report, examine the server response and redirect chain, robots.txt rules, meta tags, and canonical separately. The value of an allowing meta directive applies to the retrieved document. It does not mean that Google has already selected this URL for indexing.
A site’s absence from the search results checked for a query also does not prove that the page is absent from the index. Rank monitoring records the site’s position within the scope of the measurement taken. Confirm the indexing status of a specific page separately.
Before the next indexing request, check the URL record:
- the final response consistently returns
200; - there are no indexing blocks;
- the canonical and sitemap point to the selected version;
- an internal link is accessible to the crawler.
If the page is still not indexed after a confirmed crawl, compare it with the canonical selected by the search engine and with neighboring documents. Record the new observation separately rather than repeating the entire set of changes at once.