Disallow in robots.txt prevents a crawler from reading a page. Noindex on the page itself asks search engines to remove it from the index. If a section is blocked with Disallow, the crawler won’t read the noindex directive. For a public archive, first add noindex, then allow crawling; protect personal data with account sign-in.
Public archives and account areas require different actions
For Google to read noindex, the page must be crawlable. Official documentation.
A section such as /archive/ with public collections differs from /account/ with personal information. You can leave a public archive accessible to crawlers and add noindex to it. The latter requires access control: robots.txt and noindex do not hide data from a person making a direct request. Make a list of public URLs that should be excluded from search, separately from private ones. Allow crawling only for public pages. If the archive and private documents are under the same path, first check authentication and separate the access settings. Do not open the entire path just to add noindex.
What to check before removing Disallow
| Page | What is currently getting in the way | What to change | How to confirm the result |
|---|---|---|---|
| Public section blocked by Disallow | The crawler cannot read the page | Check for noindex | The directive must be accessible |
| Public section with noindex | Crawling is allowed | Wait for processing | Timing is not guaranteed |
| Private account area | Private data | Authentication | Do not open it to crawlers |
| noindex added only after a script runs | The script may not run | Check the response | Prefer server-side output |
Find noindex in the HTML and headers
Use a standard request (GET) to retrieve the page’s source HTML before scripts run. Check the meta robots tag and the X-Robots-Tag response header. Then request the robots.txt file for the relevant host. The presence of noindex only in a saved setting in the content management system (CMS) does not confirm that it is publicly output. Do not rely only on a script adding the directive later, especially if the source page is already inaccessible to crawlers. Check the exact path, capitalization, and specific User-agent groups. In Search Console, check the date of the last crawl. A blocked message may have appeared before the file was changed; it does not automatically describe the current response.
Use the command below in a terminal to retrieve the archive page; pass it to a developer if needed.
curl -sS -D headers.txt -o response.html 'https://example.com/archive/old/'In headers.txt, look for X-Robots-Tag, and in response.html, look for meta robots. The public archive must return noindex in the actual server response before you remove Disallow. A regular service page should not receive the same restriction. Replace example.com with your domain; this GET request does not change the site settings. curl documentation.
In the HTML, look for <meta name="robots" content="noindex">; in the headers, look for X-Robots-Tag: noindex. In /robots.txt, find the rule that matches this path. Remove it only after noindex appears on the public pages.
The order for changing the archive section
Prepare an archive page with server-side noindex in a test environment, but do not change the production site’s robots.txt yet. Check the new response, then deploy it to the target public pages. Once you confirm that noindex is present, remove the Disallow rule that blocks the relevant crawler and public path. Request the archive page and robots.txt again. Then monitor for a new crawl in Search Console: the page is excluded from search after it has been processed, and you cannot set an exact date in advance.
Remove the archive from the sitemap
Remove the public URLs you want to exclude from the list of pages you submit for indexing through the sitemap. Assess internal links based on their usefulness to visitors: noindex does not mean the archive must be unavailable in navigation. Do not automatically change all links to nofollow. Checks should confirm that people can open a useful archive, crawlers can access the directive, and current public service pages have not accidentally received noindex. After updating the SEO plugin, check one archive URL, one regular URL, and one private URL again.
Check the archive, service page, and account area
The archive opens and returns noindex, the regular service page does not have it, and the account area requires sign-in. In Search Console, check the date of the last crawl for the archive URL: an old Disallow message may refer to the version before the change. The time it takes to disappear from search is not guaranteed in advance.
Read also: robots.txt, meta robots, and X-Robots-Tag: what’s the difference and when should you use each? and 7 robots.txt mistakes that kill indexing. Site checks are available on the reChecker technical audit page.