A test copy correctly blocks all crawling, but the same robots.txt reaches the production domain during release. The problem lies in the production and test site settings, not in the file's formatting. Check the public URL after the build and after clearing the relevant cache.
Rules each domain needs
The rules in robots.txt apply to a specific host, protocol, and port. Official documentation.
A small project may have three addresses: example.com, staging.example.com, and a temporary preview domain. The production site is for visitors and search engines; the test copy is for the team. List the production site's sections that should be accessible to crawlers, along with the test copy's restrictions. A preview may contain client data, so decide how to handle authorization first. Don't turn the check into a blanket rule that says “robots must not contain Disallow”: a restriction may be intentional in a test environment. Check both addresses: the production site should remain open to crawling, while the test copy should retain its chosen restrictions.
Production site, test copy, and cache
| Where to check | What's hosted there | Restrictions needed | What to check after release |
|---|---|---|---|
| Production site | Public pages | Allow crawling of the required paths | Check the response after release |
| Test site | Test content | Separate restrictions | Don't copy these to the production site |
| Preview | Temporary build | Check the access policy | Don't treat the domain as production |
| CDN cache | Old configuration | Compare the response body | Clear the relevant key |
Where the server gets robots.txt
Ask the developer to check possible sources one at a time: a project file, the content management system (CMS), a proxy server in front of the application, and the CDN—a content delivery network for pages and files. Don't rename files at random. First compare the public response body with the output of the suspected source. In some systems, the environment setting is applied during the build, so a runtime variable won't change a file that's already been built in. In others, robots.txt is generated on request and cached by the CDN. If the file is built in, a new build is needed after changing the setting. If the CMS generates it on request, change its setting and clear the cache for that specific /robots.txt.
A developer can save the public robots.txt using the following command in a terminal.
curl -sS -D headers.txt -o response.html 'https://example.com/robots.txt'The saved response.html here contains the text of robots.txt, despite its file extension. Compare it with the build and the CMS setting. The status in headers.txt must refer to the file itself, not a login page or an error. Replace example.com with your domain; this GET request does not change the site's settings. curl documentation.
Repeat the request for the test domain, saving the response to different files. On the production site, check for an unexpected Disallow: /; on the test site, check that its presence is intentional and that access is actually protected.
Set the environment in the generator's settings
Ask the developer to explicitly list the production and test domains in the generator's settings and check this rule during publishing. The presence of the word staging in an arbitrary Host header must not select the mode by itself: this header passes the name of the requested domain. For an unknown name, the build should stop or use a separately selected rule for the test copy, so the production site doesn't accidentally get blocked. If there are several production domains, specify each one separately. When adding a domain, choose its mode first, then check the resulting file.
Check the response after clearing the cache
Clearing the CDN cache helps you get a new version, but it doesn't change the generator itself. After clearing it, request the file again and compare its contents with what you expect. Check Age, Cache-Control, and any available cache diagnostic headers. Don't assume that the absence of Age means there is no cache: the infrastructure may not expose that header. If the response is still old, check where the file comes from and the cache key. Clearing all site caches is rarely necessary to fix a single text route and makes diagnosis harder.
Check both domains after publishing
Open the regular URLs /robots.txt without extra parameters. The production file allows the required public sections and contains the correct Sitemap address if that directive is used; the test file retains its restrictions. A new build must not copy the test file to the production domain again.
Also read: What to check on a site immediately after publishing changes and 7 robots.txt mistakes that kill indexing. Site checks are available on the reChecker technical audit page.