Service pages open and show the correct headings, but an audit finds noindex. The cause may not be on the page itself. The setting may have been left in a shared layout.tsx, added in a closer route segment, or sent as a separate HTTP header by the server.
Below is a custom educational example of a Next.js 15 project using the App Router. It has public services and a private account area. We want to make only the services available to search while preserving the existing rules for the account area and test environment. The example does not mean that every found noindex directive is wrong.
First, identify which page you intend to make available
Record the exact service URL, the final address after redirects, and the environment: production, preview, or local development. Each needs its own expected result. A https://example.ru/services/design page may be public, while a similar page on a preview domain may be intentionally private.
Open the URL without signing in. If the content depends on a user session, you may be checking a personal version of the page that a crawler will never receive. Find out whether the application responds the same way to an ordinary guest and a search crawler. Do not turn technical troubleshooting into an attempt to access customer documents or administrative sections.
Save the full directive you find, including the tag name. robots applies to the general policy; googlebot applies to a specific crawler. You also need the X-Robots-Tag HTTP header, if present. A screenshot showing “Indexing disallowed” is not enough to identify its source.
Trace the layout and page hierarchy
In the App Router, page settings are combined across route segments. Start with the root app/layout.tsx, then inspect nested layouts and the relevant page.tsx. Route groups in parentheses can also have layouts, even though their names do not appear in the public URL.
Check the metadata and generateMetadata exports. For example, a shared file may contain robots: { index: false, follow: false } left over from development. If a nested public page does not set its own policy, the setting you found will continue to affect the result. If it does set one, read the entire definition.
Next.js performs a shallow merge of metadata: a nested robots object defined closer to the page replaces the corresponding object from the previous segment. This does not automatically merge all the inner fields one by one. Next.js inheritance rules 15 matter when a developer changes only one field and expects the others to be preserved.
| Setting location | What to check |
|---|---|
| Root layout | The general policy and environment-dependent settings |
| Public-group layout | The settings specifically for services and the catalog |
| Account-area layout | Separate restrictions for private pages |
| Page or generateMetadata | The final robots definition for a specific URL |
| Server and CDN | Additional X-Robots-Tag |
Define the public group’s policy explicitly
In the educational project, services are under app/(public)/services. Their layout sets the complete required robots object; shared design settings remain in the parent layout. This is an example of where to place the policy, not a universal file to copy into any site.
// app/(public)/layout.tsx
import type { Metadata } from 'next'
import type { ReactNode } from 'react'
export const metadata: Metadata = {
robots: {
index: true,
follow: true,
googleBot: { index: true, follow: true },
},
}
export default function PublicLayout({ children }: {
children: ReactNode
}) {
return <>{children}</>
}For app/(private)/account/layout.tsx, the same example sets a separate private policy. Check the actual structure: if the account area is accidentally nested inside the public group, the group’s setting may apply more broadly than intended.
Do not change index: false to true using a mass replacement across the repository. In addition to the test environment, restrictions may be needed for site search, service results, filtered-page duplicates, or private pages. First make a list of routes and specify the expected result for each. Document exceptions so the next developer understands their purpose.
Check the final response, not just the source file
After making the change, run a production build and open the page with a direct request. Dev mode is convenient for development, but search directives should be verified in conditions close to those of the published site. Save the full HTML response and its headers.
In some Next.js configurations, metadata is streamed. So not finding the tag in the initial part of head does not prove that it is absent from the completed document. Inspect the full response and the final browser DOM. When checking as a crawler, record the User-Agent used: different retrieval methods may produce a different delivery sequence.
In our example, three checks are required: a public service page, an account page, and a preview version. The service page no longer contains an indexing restriction. The account area retains its agreed policy. The preview remains restricted using the method chosen by the team. If only the first check passes and the other two were opened accidentally, the work is not complete.
If noindex remains, check other layers
The application may have the correct tag while the response carries a restrictive X-Robots-Tag. It may be set by a reverse proxy, CDN, middleware, or Route Handler. Identify at which step the header appears. Requests to the local server and public domain can help narrow down where the problem is, but they may not reveal every intermediate hosting setting.
Also check whether the route is returning a notFound() state because of a data-source failure. That can add noindex independently of your static metadata object. In that case, changing the page’s regular policy will not fix the cause. See an example of distinguishing an API failure from a missing resource.
Google explains that noindex must be accessible to the crawler during crawling so it can process it. Blocking access with robots.txt and an indexing directive solve different problems. Do not add a crawl block to “clean up noindex”—it may prevent the crawler from seeing the fix. Google documentation on noindex.
What to include in the task and how to accept the fix
A good task includes the URL, the tag or header found, the expected policy, and a list of nearby routes that must not be changed. The developer should identify the source of the restriction and show the final response after the change. A summary saying “indexing enabled” without the URL and verification result is not enough.
Check several services in different nested sections. Their policies may differ because of their own layouts or dynamic generateMetadata. Then repeat the audit of a specific page and compare the saved directive with a manual inspection of the response. For the application structure, see the Next.js guide 15.
Removing noindex does not guarantee that a URL will appear in search. After technical acceptance, the page still needs to be accessible to crawlers, have a correct canonical, contain useful content, and have its status checked in the webmaster tools. The acceptance report confirms the specific directive fix, while indexing is monitored separately using search engine data.